2024-08-02

Title: Replication in Visual Diffusion Models: A Survey and Outlook

• Authors: Wenhao Wang, Yifan Sun, Zongxin Yang, Zhengdong Hu, Zhentao Tan, Yi Yang
• Subjects: cs.CV, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2408.00001
• Pdf URL: https://arxiv.org/pdf/2408.00001
• Copy Paste: [[2408.00001]] Replication in Visual Diffusion Models: A Survey and Outlook(https://arxiv.org/abs/2408.00001)
• Keywords: security, privacy, robust, diffusion
• Abstract: Visual diffusion models have revolutionized the field of creative AI, producing high-quality and diverse content. However, they inevitably memorize training images or videos, subsequently replicating their concepts, content, or styles during inference. This phenomenon raises significant concerns about privacy, security, and copyright within generated outputs. In this survey, we provide the first comprehensive review of replication in visual diffusion models, marking a novel contribution to the field by systematically categorizing the existing studies into unveiling, understanding, and mitigating this phenomenon. Specifically, unveiling mainly refers to the methods used to detect replication instances. Understanding involves analyzing the underlying mechanisms and factors that contribute to this phenomenon. Mitigation focuses on developing strategies to reduce or eliminate replication. Beyond these aspects, we also review papers focusing on its real-world influence. For instance, in the context of healthcare, replication is critically worrying due to privacy concerns related to patient data. Finally, the paper concludes with a discussion of the ongoing challenges, such as the difficulty in detecting and benchmarking replication, and outlines future directions including the development of more robust mitigation techniques. By synthesizing insights from diverse studies, this paper aims to equip researchers and practitioners with a deeper understanding at the intersection between AI technology and social good. We release this project at this https URL.

Title: Evaluating Transfer Learning in Deep Learning Models for Classification on a Custom Wildlife Dataset: Can YOLOv8 Surpass Other Architectures?

• Authors: Subek Sharma, Sisir Dhakal, Mansi Bhavsar
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.00002
• Pdf URL: https://arxiv.org/pdf/2408.00002
• Copy Paste: [[2408.00002]] Evaluating Transfer Learning in Deep Learning Models for Classification on a Custom Wildlife Dataset: Can YOLOv8 Surpass Other Architectures?(https://arxiv.org/abs/2408.00002)
• Keywords: protect
• Abstract: Biodiversity plays a crucial role in maintaining the balance of the ecosystem. However, poaching and unintentional human activities contribute to the decline in the population of many species. Hence, active monitoring is required to preserve these endangered species. Current human-led monitoring techniques are prone to errors and are labor-intensive. Therefore, we study the application of deep learning methods like Convolutional Neural Networks (CNNs) and transfer learning, which can aid in automating the process of monitoring endangered species. For this, we create our custom dataset utilizing trustworthy online databases like iNaturalist and ZooChat. To choose the best model for our use case, we compare the performance of different architectures like DenseNet, ResNet, VGGNet, and YOLOv8 on the custom wildlife dataset. Transfer learning reduces training time by freezing the pre-trained weights and replacing only the output layer with custom, fully connected layers designed for our dataset. Our results indicate that YOLOv8 performs better, achieving a training accuracy of 97.39 % and an F1 score of 96.50 %, surpassing other models. Our findings suggest that integrating YOLOv8 into conservation efforts could revolutionize wildlife monitoring with its high accuracy and efficiency, potentially transforming how endangered species are monitored and protected worldwide.

Title: On the Perturbed States for Transformed Input-robust Reinforcement Learning

• Authors: Tung M. Luu, Haeyong Kang, Tri Ton, Thanh Nguyen, Chang D. Yoo
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.00023
• Pdf URL: https://arxiv.org/pdf/2408.00023
• Copy Paste: [[2408.00023]] On the Perturbed States for Transformed Input-robust Reinforcement Learning(https://arxiv.org/abs/2408.00023)
• Keywords: defense, attack, robust
• Abstract: Reinforcement Learning (RL) agents demonstrating proficiency in a training environment exhibit vulnerability to adversarial perturbations in input observations during deployment. This underscores the importance of building a robust agent before its real-world deployment. To alleviate the challenging point, prior works focus on developing robust training-based procedures, encompassing efforts to fortify the deep neural network component's robustness or subject the agent to adversarial training against potent attacks. In this work, we propose a novel method referred to as \textit{Transformed Input-robust RL (TIRL)}, which explores another avenue to mitigate the impact of adversaries by employing input transformation-based defenses. Specifically, we introduce two principles for applying transformation-based defenses in learning robust RL agents: \textit{(1) autoencoder-styled denoising} to reconstruct the original state and \textit{(2) bounded transformations (bit-depth reduction and vector quantization (VQ))} to achieve close transformed inputs. The transformations are applied to the state before feeding it into the policy network. Extensive experiments on multiple \mujoco environments demonstrate that input transformation-based defenses, \ie, VQ, defend against several adversaries in the state observations.

Title: Temporal Subspace Clustering for Molecular Dynamics Data

• Authors: Anna Beer, Martin Heinrigs, Claudia Plant, Ira Assent
• Subjects: cs.LG, cs.IR, physics.chem-ph
• Abstract URL: https://arxiv.org/abs/2408.00056
• Pdf URL: https://arxiv.org/pdf/2408.00056
• Copy Paste: [[2408.00056]] Temporal Subspace Clustering for Molecular Dynamics Data(https://arxiv.org/abs/2408.00056)
• Keywords: segmentation
• Abstract: We introduce MOSCITO (MOlecular Dynamics Subspace Clustering with Temporal Observance), a subspace clustering for molecular dynamics data. MOSCITO groups those timesteps of a molecular dynamics trajectory together into clusters in which the molecule has similar conformations. In contrast to state-of-the-art methods, MOSCITO takes advantage of sequential relationships found in time series data. Unlike existing work, MOSCITO does not need a two-step procedure with tedious post-processing, but directly models essential properties of the data. Interpreting clusters as Markov states allows us to evaluate the clustering performance based on the resulting Markov state models. In experiments on 60 trajectories and 4 different proteins, we show that the performance of MOSCITO achieves state-of-the-art performance in a novel single-step method. Moreover, by modeling temporal aspects, MOSCITO obtains better segmentation of trajectories, especially for small numbers of clusters.

Title: Localized Gaussian Splatting Editing with Contextual Awareness

• Authors: Hanyuan Xiao, Yingshu Chen, Huajian Huang, Haolin Xiong, Jing Yang, Pratusha Prasad, Yajie Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.00083
• Pdf URL: https://arxiv.org/pdf/2408.00083
• Copy Paste: [[2408.00083]] Localized Gaussian Splatting Editing with Contextual Awareness(https://arxiv.org/abs/2408.00083)
• Keywords: robust, diffusion
• Abstract: Recent text-guided generation of individual 3D object has achieved great success using diffusion priors. However, these methods are not suitable for object insertion and replacement tasks as they do not consider the background, leading to illumination mismatches within the environment. To bridge the gap, we introduce an illumination-aware 3D scene editing pipeline for 3D Gaussian Splatting (3DGS) representation. Our key observation is that inpainting by the state-of-the-art conditional 2D diffusion model is consistent with background in lighting. To leverage the prior knowledge from the well-trained diffusion models for 3D object generation, our approach employs a coarse-to-fine objection optimization pipeline with inpainted views. In the first coarse step, we achieve image-to-3D lifting given an ideal inpainted view. The process employs 3D-aware diffusion prior from a view-conditioned diffusion model, which preserves illumination present in the conditioning image. To acquire an ideal inpainted image, we introduce an Anchor View Proposal (AVP) algorithm to find a single view that best represents the scene illumination in target region. In the second Texture Enhancement step, we introduce a novel Depth-guided Inpainting Score Distillation Sampling (DI-SDS), which enhances geometry and texture details with the inpainting diffusion prior, beyond the scope of the 3D-aware diffusion prior knowledge in the first coarse step. DI-SDS not only provides fine-grained texture enhancement, but also urges optimization to respect scene lighting. Our approach efficiently achieves local editing with global illumination consistency without explicitly modeling light transport. We demonstrate robustness of our method by evaluating editing in real scenes containing explicit highlight and shadows, and compare against the state-of-the-art text-to-3D editing methods.

Title: From Attributes to Natural Language: A Survey and Foresight on Text-based Person Re-identification

• Authors: Fanzhi Jiang, Su Yang, Mark W. Jones, Liumei Zhang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.00096
• Pdf URL: https://arxiv.org/pdf/2408.00096
• Copy Paste: [[2408.00096]] From Attributes to Natural Language: A Survey and Foresight on Text-based Person Re-identification(https://arxiv.org/abs/2408.00096)
• Keywords: security, extraction
• Abstract: Text-based person re-identification (Re-ID) is a challenging topic in the field of complex multimodal analysis, its ultimate aim is to recognize specific pedestrians by scrutinizing attributes/natural language descriptions. Despite the wide range of applicable areas such as security surveillance, video retrieval, person tracking, and social media analytics, there is a notable absence of comprehensive reviews dedicated to summarizing the text-based person Re-ID from a technical perspective. To address this gap, we propose to introduce a taxonomy spanning Evaluation, Strategy, Architecture, and Optimization dimensions, providing a comprehensive survey of the text-based person Re-ID task. We start by laying the groundwork for text-based person Re-ID, elucidating fundamental concepts related to attribute/natural language-based identification. Then a thorough examination of existing benchmark datasets and metrics is presented. Subsequently, we further delve into prevalent feature extraction strategies employed in text-based person Re-ID research, followed by a concise summary of common network architectures within the domain. Prevalent loss functions utilized for model optimization and modality alignment in text-based person Re-ID are also scrutinized. To conclude, we offer a concise summary of our findings, pinpointing challenges in text-based person Re-ID. In response to these challenges, we outline potential avenues for future open-set text-based person Re-ID and present a baseline architecture for text-based pedestrian image generation-guided re-identification(TBPGR).

Title: ReLiK: Retrieve and LinK, Fast and Accurate Entity Linking and Relation Extraction on an Academic Budget

• Authors: Riccardo Orlando, Pere-Lluis Huguet-Cabot, Edoardo Barba, Roberto Navigli
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.00103
• Pdf URL: https://arxiv.org/pdf/2408.00103
• Copy Paste: [[2408.00103]] ReLiK: Retrieve and LinK, Fast and Accurate Entity Linking and Relation Extraction on an Academic Budget(https://arxiv.org/abs/2408.00103)
• Keywords: extraction
• Abstract: Entity Linking (EL) and Relation Extraction (RE) are fundamental tasks in Natural Language Processing, serving as critical components in a wide range of applications. In this paper, we propose ReLiK, a Retriever-Reader architecture for both EL and RE, where, given an input text, the Retriever module undertakes the identification of candidate entities or relations that could potentially appear within the text. Subsequently, the Reader module is tasked to discern the pertinent retrieved entities or relations and establish their alignment with the corresponding textual spans. Notably, we put forward an innovative input representation that incorporates the candidate entities or relations alongside the text, making it possible to link entities or extract relations in a single forward pass and to fully leverage pre-trained language models contextualization capabilities, in contrast with previous Retriever-Reader-based methods, which require a forward pass for each candidate. Our formulation of EL and RE achieves state-of-the-art performance in both in-domain and out-of-domain benchmarks while using academic budget training and with up to 40x inference speed compared to competitors. Finally, we show how our architecture can be used seamlessly for Information Extraction (cIE), i.e. EL + RE, and setting a new state of the art by employing a shared Reader that simultaneously extracts entities and relations.

Title: WAS: Dataset and Methods for Artistic Text Segmentation

• Authors: Xudong Xie, Yuzhe Li, Yang Liu, Zhifei Zhang, Zhaowen Wang, Wei Xiong, Xiang Bai
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.00106
• Pdf URL: https://arxiv.org/pdf/2408.00106
• Copy Paste: [[2408.00106]] WAS: Dataset and Methods for Artistic Text Segmentation(https://arxiv.org/abs/2408.00106)
• Keywords: diffusion, generative, segmentation
• Abstract: Accurate text segmentation results are crucial for text-related generative tasks, such as text image generation, text editing, text removal, and text style transfer. Recently, some scene text segmentation methods have made significant progress in segmenting regular text. However, these methods perform poorly in scenarios containing artistic text. Therefore, this paper focuses on the more challenging task of artistic text segmentation and constructs a real artistic text segmentation dataset. One challenge of the task is that the local stroke shapes of artistic text are changeable with diversity and complexity. We propose a decoder with the layer-wise momentum query to prevent the model from ignoring stroke regions of special shapes. Another challenge is the complexity of the global topological structure. We further design a skeleton-assisted head to guide the model to focus on the global structure. Additionally, to enhance the generalization performance of the text segmentation model, we propose a strategy for training data synthesis, based on the large multi-modal model and the diffusion model. Experimental results show that our proposed method and synthetic dataset can significantly enhance the performance of artistic text segmentation and achieve state-of-the-art results on other public datasets.

Title: Automated Sperm Morphology Analysis Based on Instance-Aware Part Segmentation

• Authors: Wenyuan Chen, Haocong Song, Changsheng Dai, Aojun Jiang, Guanqiao Shan, Hang Liu, Yanlong Zhou, Khaled Abdalla, Shivani N Dhanani, Katy Fatemeh Moosavi, Shruti Pathak, Clifford Librach, Zhuoran Zhang, Yu Sun
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.00112
• Pdf URL: https://arxiv.org/pdf/2408.00112
• Copy Paste: [[2408.00112]] Automated Sperm Morphology Analysis Based on Instance-Aware Part Segmentation(https://arxiv.org/abs/2408.00112)
• Keywords: segmentation
• Abstract: Traditional sperm morphology analysis is based on tedious manual annotation. Automated morphology analysis of a high number of sperm requires accurate segmentation of each sperm part and quantitative morphology evaluation. State-of-the-art instance-aware part segmentation networks follow a "detect-then-segment" paradigm. However, due to sperm's slim shape, their segmentation suffers from large context loss and feature distortion due to bounding box cropping and resizing during ROI Align. Moreover, morphology measurement of sperm tail is demanding because of the long and curved shape and its uneven width. This paper presents automated techniques to measure sperm morphology parameters automatically and quantitatively. A novel attention-based instance-aware part segmentation network is designed to reconstruct lost contexts outside bounding boxes and to fix distorted features, by refining preliminary segmented masks through merging features extracted by feature pyramid network. An automated centerline-based tail morphology measurement method is also proposed, in which an outlier filtering method and endpoint detection algorithm are designed to accurately reconstruct tail endpoints. Experimental results demonstrate that the proposed network outperformed the state-of-the-art top-down RP-R-CNN by 9.2% [AP]_vol^p, and the proposed automated tail morphology measurement method achieved high measurement accuracies of 95.34%,96.39%,91.2% for length, width and curvature, respectively.

Title: Measuring Progress in Dictionary Learning for Language Model Interpretability with Board Game Models

• Authors: Adam Karvonen, Benjamin Wright, Can Rager, Rico Angell, Jannik Brinkmann, Logan Smith, Claudio Mayrink Verdun, David Bau, Samuel Marks
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2408.00113
• Pdf URL: https://arxiv.org/pdf/2408.00113
• Copy Paste: [[2408.00113]] Measuring Progress in Dictionary Learning for Language Model Interpretability with Board Game Models(https://arxiv.org/abs/2408.00113)
• Keywords: interpretability
• Abstract: What latent features are encoded in language model (LM) representations? Recent work on training sparse autoencoders (SAEs) to disentangle interpretable features in LM representations has shown significant promise. However, evaluating the quality of these SAEs is difficult because we lack a ground-truth collection of interpretable features that we expect good SAEs to recover. We thus propose to measure progress in interpretable dictionary learning by working in the setting of LMs trained on chess and Othello transcripts. These settings carry natural collections of interpretable features -- for example, "there is a knight on F3" -- which we leverage into $\textit{supervised}$ metrics for SAE quality. To guide progress in interpretable dictionary learning, we introduce a new SAE training technique, $\textit{p-annealing}$, which improves performance on prior unsupervised metrics as well as our new metrics.

Title: Certifying Robustness of Learning-Based Keypoint Detection and Pose Estimation Methods

• Authors: Xusheng Luo, Tianhao Wei, Simin Liu, Ziwei Wang, Luis Mattei-Mendez, Taylor Loper, Joshua Neighbor, Casidhe Hutchison, Changliu Liu
• Subjects: cs.CV, cs.LG, cs.RO, eess.SY
• Abstract URL: https://arxiv.org/abs/2408.00117
• Pdf URL: https://arxiv.org/pdf/2408.00117
• Copy Paste: [[2408.00117]] Certifying Robustness of Learning-Based Keypoint Detection and Pose Estimation Methods(https://arxiv.org/abs/2408.00117)
• Keywords: robust
• Abstract: This work addresses the certification of the local robustness of vision-based two-stage 6D object pose estimation. The two-stage method for object pose estimation achieves superior accuracy by first employing deep neural network-driven keypoint regression and then applying a Perspective-n-Point (PnP) technique. Despite advancements, the certification of these methods' robustness remains scarce. This research aims to fill this gap with a focus on their local robustness on the system level--the capacity to maintain robust estimations amidst semantic input perturbations. The core idea is to transform the certification of local robustness into neural network verification for classification tasks. The challenge is to develop model, input, and output specifications that align with off-the-shelf verification tools. To facilitate verification, we modify the keypoint detection model by substituting nonlinear operations with those more amenable to the verification processes. Instead of injecting random noise into images, as is common, we employ a convex hull representation of images as input specifications to more accurately depict semantic perturbations. Furthermore, by conducting a sensitivity analysis, we propagate the robustness criteria from pose to keypoint accuracy, and then formulating an optimal error threshold allocation problem that allows for the setting of a maximally permissible keypoint deviation thresholds. Viewing each pixel as an individual class, these thresholds result in linear, classification-akin output specifications. Under certain conditions, we demonstrate that the main components of our certification framework are both sound and complete, and validate its effects through extensive evaluations on realistic perturbations. To our knowledge, this is the first study to certify the robustness of large-scale, keypoint-based pose estimation given images in real-world scenarios.

Title: Gemma 2: Improving Open Language Models at a Practical Size

• Authors: Gemma Team: Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, Nino Vieillard, Piotr Stanczyk, Sertan Girgin, Nikola Momchev, Matt Hoffman, Shantanu Thakoor, Jean-Bastien Grill, Behnam Neyshabur, Alanna Walton, Aliaksei Severyn, Alicia Parrish, Aliya Ahmad, Allen Hutchison, Alvin Abdagic, Amanda Carl, Amy Shen, Andy Brock, Andy Coenen, Anthony Laforge, Antonia Paterson, Ben Bastian, Bilal Piot, Bo Wu, Brandon Royal, Charlie Chen, Chintu Kumar, Chris Perry, Chris Welty, Christopher A. Choquette-Choo, Danila Sinopalnikov, David Weinberger, Dimple Vijaykumar, Dominika Rogozińska, Dustin Herbison, Elisa Bandy, Emma Wang, Eric Noland, Erica Moreira, Evan Senter, Evgenii Eltyshev, Francesco Visin, Gabriel Rasskin, Gary Wei, Glenn Cameron, Gus Martins, Hadi Hashemi, Hanna Klimczak-Plucińska, Harleen Batra, Harsh Dhand, Ivan Nardini, Jacinda Mein, Jack Zhou, James Svensson, Jeff Stanway, Jetha Chan, Jin Zhou, Joana Carrasqueira, Joana Iljazi, Jocelyn Becker, Joe Fernandez, Joost van Amersfoort, Josh Gordon, Josh Lipschultz, Josh Newlan, Ju-yeong Ji, Kareem Mohamed, Kartikeya Badola, Kat Black, Katie Millican, Keelin McDonell, Kelvin Nguyen, Kiranbir Sodhia, Kish Greene, Lars Lowe Sjoesund, Lauren Usui, Laurent Sifre, Lena Heuermann, Leticia Lago, Lilly McNealus, Livio Baldini Soares
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.00118
• Pdf URL: https://arxiv.org/pdf/2408.00118
• Copy Paste: [[2408.00118]] Gemma 2: Improving Open Language Models at a Practical Size(https://arxiv.org/abs/2408.00118)
• Keywords: transformer
• Abstract: In this work, we introduce Gemma 2, a new addition to the Gemma family of lightweight, state-of-the-art open models, ranging in scale from 2 billion to 27 billion parameters. In this new version, we apply several known technical modifications to the Transformer architecture, such as interleaving local-global attentions (Beltagy et al., 2020a) and group-query attention (Ainslie et al., 2023). We also train the 2B and 9B models with knowledge distillation (Hinton et al., 2015) instead of next token prediction. The resulting models deliver the best performance for their size, and even offer competitive alternatives to models that are 2-3 times bigger. We release all our models to the community.

Title: A Course Shared Task on Evaluating LLM Output for Clinical Questions

• Authors: Yufang Hou, Thy Thy Tran, Doan Nam Long Vu, Yiwen Cao, Kai Li, Lukas Rohde, Iryna Gurevych
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.00122
• Pdf URL: https://arxiv.org/pdf/2408.00122
• Copy Paste: [[2408.00122]] A Course Shared Task on Evaluating LLM Output for Clinical Questions(https://arxiv.org/abs/2408.00122)
• Keywords: large language model
• Abstract: This paper presents a shared task that we organized at the Foundations of Language Technology (FoLT) course in 2023/2024 at the Technical University of Darmstadt, which focuses on evaluating the output of Large Language Models (LLMs) in generating harmful answers to health-related clinical questions. We describe the task design considerations and report the feedback we received from the students. We expect the task and the findings reported in this paper to be relevant for instructors teaching natural language processing (NLP) and designing course assignments.

Title: Vera Verto: Multimodal Hijacking Attack

• Authors: Minxing Zhang, Ahmed Salem, Michael Backes, Yang Zhang
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.00129
• Pdf URL: https://arxiv.org/pdf/2408.00129
• Copy Paste: [[2408.00129]] Vera Verto: Multimodal Hijacking Attack(https://arxiv.org/abs/2408.00129)
• Keywords: attack
• Abstract: The increasing cost of training machine learning (ML) models has led to the inclusion of new parties to the training pipeline, such as users who contribute training data and companies that provide computing resources. This involvement of such new parties in the ML training process has introduced new attack surfaces for an adversary to exploit. A recent attack in this domain is the model hijacking attack, whereby an adversary hijacks a victim model to implement their own -- possibly malicious -- hijacking tasks. However, the scope of the model hijacking attack is so far limited to the homogeneous-modality tasks. In this paper, we transform the model hijacking attack into a more general multimodal setting, where the hijacking and original tasks are performed on data of different modalities. Specifically, we focus on the setting where an adversary implements a natural language processing (NLP) hijacking task into an image classification model. To mount the attack, we propose a novel encoder-decoder based framework, namely the Blender, which relies on advanced image and language models. Experimental results show that our modal hijacking attack achieves strong performances in different settings. For instance, our attack achieves 94%, 94%, and 95% attack success rate when using the Sogou news dataset to hijack STL10, CIFAR-10, and MNIST classifiers.

Title: Correcting Negative Bias in Large Language Models through Negative Attention Score Alignment

• Authors: Sangwon Yu, Jongyoon Song, Bongkyu Hwang, Hoyoung Kang, Sooah Cho, Junhwa Choi, Seongho Joe, Taehee Lee, Youngjune L. Gwon, Sungroh Yoon
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.00137
• Pdf URL: https://arxiv.org/pdf/2408.00137
• Copy Paste: [[2408.00137]] Correcting Negative Bias in Large Language Models through Negative Attention Score Alignment(https://arxiv.org/abs/2408.00137)
• Keywords: large language model
• Abstract: A binary decision task, like yes-no questions or answer verification, reflects a significant real-world scenario such as where users look for confirmation about the correctness of their decisions on specific issues. In this work, we observe that language models exhibit a negative bias in the binary decisions of complex reasoning tasks. Based on our observations and the rationale about attention-based model dynamics, we propose a negative attention score (NAS) to systematically and quantitatively formulate negative bias. Based on NAS, we identify attention heads that attend to negative tokens provided in the instructions as answer candidate of binary decisions, regardless of the question in the prompt, and validate their association with the negative bias. Additionally, we propose the negative attention score alignment (NASA) method, which is a parameter-efficient fine-tuning technique to address the extracted negatively biased attention heads. Experimental results from various domains of reasoning tasks and large model search space demonstrate that NASA significantly reduces the gap between precision and recall caused by negative bias while preserving their generalization abilities. Our codes are available at \url{this https URL}.

Title: Distributed In-Context Learning under Non-IID Among Clients

• Authors: Siqi Liang, Sumyeong Ahn, Jiayu Zhou
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.00144
• Pdf URL: https://arxiv.org/pdf/2408.00144
• Copy Paste: [[2408.00144]] Distributed In-Context Learning under Non-IID Among Clients(https://arxiv.org/abs/2408.00144)
• Keywords: large language model
• Abstract: Advancements in large language models (LLMs) have shown their effectiveness in multiple complicated natural language reasoning tasks. A key challenge remains in adapting these models efficiently to new or unfamiliar tasks. In-context learning (ICL) provides a promising solution for few-shot adaptation by retrieving a set of data points relevant to a query, called in-context examples (ICE), from a training dataset and providing them during the inference as context. Most existing studies utilize a centralized training dataset, yet many real-world datasets may be distributed among multiple clients, and remote data retrieval can be associated with costs. Especially when the client data are non-identical independent distributions (non-IID), retrieving from clients a proper set of ICEs needed for a test query presents critical challenges. In this paper, we first show that in this challenging setting, test queries will have different preferences among clients because of non-IIDness, and equal contribution often leads to suboptimal performance. We then introduce a novel approach to tackle the distributed non-IID ICL problem when a data usage budget is present. The principle is that each client's proper contribution (budget) should be designed according to the preference of each query for that client. Our approach uses a data-driven manner to allocate a budget for each client, tailored to each test query. Through extensive empirical studies on diverse datasets, our framework demonstrates superior performance relative to competing baselines.

Title: Generative Learning of the Solution of Parametric Partial Differential Equations Using Guided Diffusion Models and Virtual Observations

• Authors: Han Gao, Sebastian Kaltenbach, Petros Koumoutsakos
• Subjects: cs.LG, physics.comp-ph, physics.flu-dyn
• Abstract URL: https://arxiv.org/abs/2408.00157
• Pdf URL: https://arxiv.org/pdf/2408.00157
• Copy Paste: [[2408.00157]] Generative Learning of the Solution of Parametric Partial Differential Equations Using Guided Diffusion Models and Virtual Observations(https://arxiv.org/abs/2408.00157)
• Keywords: robust, diffusion, generative
• Abstract: We introduce a generative learning framework to model high-dimensional parametric systems using gradient guidance and virtual observations. We consider systems described by Partial Differential Equations (PDEs) discretized with structured or unstructured grids. The framework integrates multi-level information to generate high fidelity time sequences of the system dynamics. We demonstrate the effectiveness and versatility of our framework with two case studies in incompressible, two dimensional, low Reynolds cylinder flow on an unstructured mesh and incompressible turbulent channel flow on a structured mesh, both parameterized by the Reynolds number. Our results illustrate the framework's robustness and ability to generate accurate flow sequences across various parameter settings, significantly reducing computational costs allowing for efficient forecasting and reconstruction of flow dynamics.

Title: Automatic Generation of Behavioral Test Cases For Natural Language Processing Using Clustering and Prompting

• Authors: Ying Li, Rahul Singh, Tarun Joshi, Agus Sudjianto
• Subjects: cs.CL, cs.AI, cs.ET, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.00161
• Pdf URL: https://arxiv.org/pdf/2408.00161
• Copy Paste: [[2408.00161]] Automatic Generation of Behavioral Test Cases For Natural Language Processing Using Clustering and Prompting(https://arxiv.org/abs/2408.00161)
• Keywords: large language model
• Abstract: Recent work in behavioral testing for natural language processing (NLP) models, such as Checklist, is inspired by related paradigms in software engineering testing. They allow evaluation of general linguistic capabilities and domain understanding, hence can help evaluate conceptual soundness and identify model weaknesses. However, a major challenge is the creation of test cases. The current packages rely on semi-automated approach using manual development which requires domain expertise and can be time consuming. This paper introduces an automated approach to develop test cases by exploiting the power of large language models and statistical techniques. It clusters the text representations to carefully construct meaningful groups and then apply prompting techniques to automatically generate Minimal Functionality Tests (MFT). The well-known Amazon Reviews corpus is used to demonstrate our approach. We analyze the behavioral test profiles across four different classification algorithms and discuss the limitations and strengths of those models.

Title: Non-convolutional Graph Neural Networks

• Authors: Yuanqing Wang, Kyunghyun Cho
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.00165
• Pdf URL: https://arxiv.org/pdf/2408.00165
• Copy Paste: [[2408.00165]] Non-convolutional Graph Neural Networks(https://arxiv.org/abs/2408.00165)
• Keywords: robust
• Abstract: Rethink convolution-based graph neural networks (GNN) -- they characteristically suffer from limited expressiveness, over-smoothing, and over-squashing, and require specialized sparse kernels for efficient computation. Here, we design a simple graph learning module entirely free of convolution operators, coined \textit{random walk with unifying memory} (RUM) neural network, where an RNN merges the topological and semantic graph features along the random walks terminating at each node. Relating the rich literature on RNN behavior and graph topology, we theoretically show and experimentally verify that RUM attenuates the aforementioned symptoms and is more expressive than the Weisfeiler-Lehman (WL) isomorphism test. On a variety of node- and graph-level classification and regression tasks, RUM not only achieves competitive performance, but is also robust, memory-efficient, scalable, and faster than the simplest convolutional GNNs.

Title: Strike the Balance: On-the-Fly Uncertainty based User Interactions for Long-Term Video Object Segmentation

• Authors: Stéphane Vujasinović, Stefan Becker, Sebastian Bullinger, Norbert Scherer-Negenborn, Michael Arens
• Subjects: cs.CV, cs.HC, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.00169
• Pdf URL: https://arxiv.org/pdf/2408.00169
• Copy Paste: [[2408.00169]] Strike the Balance: On-the-Fly Uncertainty based User Interactions for Long-Term Video Object Segmentation(https://arxiv.org/abs/2408.00169)
• Keywords: robust, segmentation
• Abstract: In this paper, we introduce a variant of video object segmentation (VOS) that bridges interactive and semi-automatic approaches, termed Lazy Video Object Segmentation (ziVOS). In contrast, to both tasks, which handle video object segmentation in an off-line manner (i.e., pre-recorded sequences), we propose through ziVOS to target online recorded sequences. Here, we strive to strike a balance between performance and robustness for long-term scenarios by soliciting user feedback's on-the-fly during the segmentation process. Hence, we aim to maximize the tracking duration of an object of interest, while requiring minimal user corrections to maintain tracking over an extended period. We propose a competitive baseline, i.e., Lazy-XMem, as a reference for future works in ziVOS. Our proposed approach uses an uncertainty estimation of the tracking state to determine whether a user interaction is necessary to refine the model's prediction. To quantitatively assess the performance of our method and the user's workload, we introduce complementary metrics alongside those already established in the field. We evaluate our approach using the recently introduced LVOS dataset, which offers numerous long-term videos. Our code is publicly available at this https URL.

Title: CC-SAM: SAM with Cross-feature Attention and Context for Ultrasound Image Segmentation

• Authors: Shreyank N Gowda, David A. Clifton
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.00181
• Pdf URL: https://arxiv.org/pdf/2408.00181
• Copy Paste: [[2408.00181]] CC-SAM: SAM with Cross-feature Attention and Context for Ultrasound Image Segmentation(https://arxiv.org/abs/2408.00181)
• Keywords: transformer, segmentation
• Abstract: The Segment Anything Model (SAM) has achieved remarkable successes in the realm of natural image segmentation, but its deployment in the medical imaging sphere has encountered challenges. Specifically, the model struggles with medical images that feature low contrast, faint boundaries, intricate morphologies, and small-sized objects. To address these challenges and enhance SAM's performance in the medical domain, we introduce a comprehensive modification. Firstly, we incorporate a frozen Convolutional Neural Network (CNN) branch as an image encoder, which synergizes with SAM's original Vision Transformer (ViT) encoder through a novel variational attention fusion module. This integration bolsters the model's capability to capture local spatial information, which is often paramount in medical imagery. Moreover, to further optimize SAM for medical imaging, we introduce feature and position adapters within the ViT branch, refining the encoder's representations. We see that compared to current prompting strategies to fine-tune SAM for ultrasound medical segmentation, the use of text descriptions that serve as text prompts for SAM helps significantly improve the performance. Leveraging ChatGPT's natural language understanding capabilities, we generate prompts that offer contextual information and guidance to SAM, enabling it to better understand the nuances of ultrasound medical images and improve its segmentation accuracy. Our method, in its entirety, represents a significant stride towards making universal image segmentation models more adaptable and efficient in the medical domain.

Title: S-SYNTH: Knowledge-Based, Synthetic Generation of Skin Images

• Authors: Andrea Kim, Niloufar Saharkhiz, Elena Sizikova, Miguel Lago, Berkman Sahiner, Jana Delfino, Aldo Badano
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.00191
• Pdf URL: https://arxiv.org/pdf/2408.00191
• Copy Paste: [[2408.00191]] S-SYNTH: Knowledge-Based, Synthetic Generation of Skin Images(https://arxiv.org/abs/2408.00191)
• Keywords: segmentation
• Abstract: Development of artificial intelligence (AI) techniques in medical imaging requires access to large-scale and diverse datasets for training and evaluation. In dermatology, obtaining such datasets remains challenging due to significant variations in patient populations, illumination conditions, and acquisition system characteristics. In this work, we propose S-SYNTH, the first knowledge-based, adaptable open-source skin simulation framework to rapidly generate synthetic skin, 3D models and digitally rendered images, using an anatomically inspired multi-layer, multi-component skin and growing lesion model. The skin model allows for controlled variation in skin appearance, such as skin color, presence of hair, lesion shape, and blood fraction among other parameters. We use this framework to study the effect of possible variations on the development and evaluation of AI models for skin lesion segmentation, and show that results obtained using synthetic data follow similar comparative trends as real dermatologic images, while mitigating biases and limitations from existing datasets including small dataset size, lack of diversity, and underrepresentation.

Title: Resilience and Security of Deep Neural Networks Against Intentional and Unintentional Perturbations: Survey and Research Challenges

• Authors: Sazzad Sayyed, Milin Zhang, Shahriar Rifat, Ananthram Swami, Michael De Lucia, Francesco Restuccia
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.00193
• Pdf URL: https://arxiv.org/pdf/2408.00193
• Copy Paste: [[2408.00193]] Resilience and Security of Deep Neural Networks Against Intentional and Unintentional Perturbations: Survey and Research Challenges(https://arxiv.org/abs/2408.00193)
• Keywords: secure, security, robust
• Abstract: In order to deploy deep neural networks (DNNs) in high-stakes scenarios, it is imperative that DNNs provide inference robust to external perturbations - both intentional and unintentional.Although the resilience of DNNs to intentional and unintentional perturbations has been widely investigated, a unified vision of these inherently intertwined problem domains is still this http URL this work, we fill this gap by providing a survey of the state of the art and highlighting the similarities of the proposed approaches.We also analyze the research challenges that need to be addressed to deploy resilient and secure this http URL there has not been any such survey connecting the resilience of DNNs to intentional and unintentional perturbations, we believe this work can help advance the frontier in both domains by enabling the exchange of ideas between the two communities.

Title: Automated Software Vulnerability Static Code Analysis Using Generative Pre-Trained Transformer Models

• Authors: Elijah Pelofske, Vincent Urias, Lorie M. Liebrock
• Subjects: cs.CR, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.00197
• Pdf URL: https://arxiv.org/pdf/2408.00197
• Copy Paste: [[2408.00197]] Automated Software Vulnerability Static Code Analysis Using Generative Pre-Trained Transformer Models(https://arxiv.org/abs/2408.00197)
• Keywords: transformer, generative
• Abstract: Generative Pre-Trained Transformer models have been shown to be surprisingly effective at a variety of natural language processing tasks -- including generating computer code. We evaluate the effectiveness of open source GPT models for the task of automatic identification of the presence of vulnerable code syntax (specifically targeting C and C++ source code). This task is evaluated on a selection of 36 source code examples from the NIST SARD dataset, which are specifically curated to not contain natural English that indicates the presence, or lack thereof, of a particular vulnerability. The NIST SARD source code dataset contains identified vulnerable lines of source code that are examples of one out of the 839 distinct Common Weakness Enumerations (CWE), allowing for exact quantification of the GPT output classification error rate. A total of 5 GPT models are evaluated, using 10 different inference temperatures and 100 repetitions at each setting, resulting in 5,000 GPT queries per vulnerable source code analyzed. Ultimately, we find that the GPT models that we evaluated are not suitable for fully automated vulnerability scanning because the false positive and false negative rates are too high to likely be useful in practice. However, we do find that the GPT models perform surprisingly well at automated vulnerability detection for some of the test cases, in particular surpassing random sampling, and being able to identify the exact lines of code that are vulnerable albeit at a low success rate. The best performing GPT model result found was Llama-2-70b-chat-hf with inference temperature of 0.1 applied to NIST SARD test case 149165 (which is an example of a buffer overflow vulnerability), which had a binary classification recall score of 1.0 and a precision of 1.0 for correctly and uniquely identifying the vulnerable line of code and the correct CWE number.

Title: OmniParser for Pure Vision Based GUI Agent

• Authors: Yadong Lu, Jianwei Yang, Yelong Shen, Ahmed Awadallah
• Subjects: cs.CV, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.00203
• Pdf URL: https://arxiv.org/pdf/2408.00203
• Copy Paste: [[2408.00203]] OmniParser for Pure Vision Based GUI Agent(https://arxiv.org/abs/2408.00203)
• Keywords: robust
• Abstract: The recent success of large vision language models shows great potential in driving the agent system operating on user interfaces. However, we argue that the power multimodal models like GPT-4V as a general agent on multiple operating systems across different applications is largely underestimated due to the lack of a robust screen parsing technique capable of: 1) reliably identifying interactable icons within the user interface, and 2) understanding the semantics of various elements in a screenshot and accurately associate the intended action with the corresponding region on the screen. To fill these gaps, we introduce \textsc{OmniParser}, a comprehensive method for parsing user interface screenshots into structured elements, which significantly enhances the ability of GPT-4V to generate actions that can be accurately grounded in the corresponding regions of the interface. We first curated an interactable icon detection dataset using popular webpages and an icon description dataset. These datasets were utilized to fine-tune specialized models: a detection model to parse interactable regions on the screen and a caption model to extract the functional semantics of the detected elements. \textsc{OmniParser} significantly improves GPT-4V's performance on ScreenSpot benchmark. And on Mind2Web and AITW benchmark, \textsc{OmniParser} with screenshot only input outperforms the GPT-4V baselines requiring additional information outside of screenshot.

Title: Sentence-wise Speech Summarization: Task, Datasets, and End-to-End Modeling with LM Knowledge Distillation

• Authors: Kohei Matsuura, Takanori Ashihara, Takafumi Moriya, Masato Mimura, Takatomo Kano, Atsunori Ogawa, Marc Delcroix
• Subjects: cs.CL, eess.AS
• Abstract URL: https://arxiv.org/abs/2408.00205
• Pdf URL: https://arxiv.org/pdf/2408.00205
• Copy Paste: [[2408.00205]] Sentence-wise Speech Summarization: Task, Datasets, and End-to-End Modeling with LM Knowledge Distillation(https://arxiv.org/abs/2408.00205)
• Keywords: transformer
• Abstract: This paper introduces a novel approach called sentence-wise speech summarization (Sen-SSum), which generates text summaries from a spoken document in a sentence-by-sentence manner. Sen-SSum combines the real-time processing of automatic speech recognition (ASR) with the conciseness of speech summarization. To explore this approach, we present two datasets for Sen-SSum: Mega-SSum and CSJ-SSum. Using these datasets, our study evaluates two types of Transformer-based models: 1) cascade models that combine ASR and strong text summarization models, and 2) end-to-end (E2E) models that directly convert speech into a text summary. While E2E models are appealing to develop compute-efficient models, they perform worse than cascade models. Therefore, we propose knowledge distillation for E2E models using pseudo-summaries generated by the cascade models. Our experiments show that this proposed knowledge distillation effectively improves the performance of the E2E model on both datasets.

Title: A Prior Embedding-Driven Architecture for Long Distance Blind Iris Recognition

• Authors: Qi Xiong, Xinman Zhang, Jun Shen
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.00210
• Pdf URL: https://arxiv.org/pdf/2408.00210
• Copy Paste: [[2408.00210]] A Prior Embedding-Driven Architecture for Long Distance Blind Iris Recognition(https://arxiv.org/abs/2408.00210)
• Keywords: robust, generative
• Abstract: Blind iris images, which result from unknown degradation during the process of iris recognition at long distances, often lead to decreased iris recognition rates. Currently, little existing literature offers a solution to this problem. In response, we propose a prior embedding-driven architecture for long distance blind iris recognition. We first proposed a blind iris image restoration network called Iris-PPRGAN. To effectively restore the texture of the blind iris, Iris-PPRGAN includes a Generative Adversarial Network (GAN) used as a Prior Decoder, and a DNN used as the encoder. To extract iris features more efficiently, we then proposed a robust iris classifier by modifying the bottleneck module of InsightFace, which called Insight-Iris. A low-quality blind iris image is first restored by Iris-PPRGAN, then the restored iris image undergoes recognition via Insight-Iris. Experimental results on the public CASIA-Iris-distance dataset demonstrate that our proposed method significantly superior results to state-of-the-art blind iris restoration methods both quantitatively and qualitatively, Specifically, the recognition rate for long-distance blind iris images reaches 90% after processing with our methods, representing an improvement of approximately ten percentage points compared to images without restoration.

Title: Load Balancing in Federated Learning

• Authors: Alireza Javani, Zhiying Wang
• Subjects: cs.LG, cs.IT
• Abstract URL: https://arxiv.org/abs/2408.00217
• Pdf URL: https://arxiv.org/pdf/2408.00217
• Copy Paste: [[2408.00217]] Load Balancing in Federated Learning(https://arxiv.org/abs/2408.00217)
• Keywords: privacy, federate, fair
• Abstract: Federated Learning (FL) is a decentralized machine learning framework that enables learning from data distributed across multiple remote devices, enhancing communication efficiency and data privacy. Due to limited communication resources, a scheduling policy is often applied to select a subset of devices for participation in each FL round. The scheduling process confronts significant challenges due to the need for fair workload distribution, efficient resource utilization, scalability in environments with numerous edge devices, and statistically heterogeneous data across devices. This paper proposes a load metric for scheduling policies based on the Age of Information and addresses the above challenges by minimizing the load metric variance across the clients. Furthermore, a decentralized Markov scheduling policy is presented, that ensures a balanced workload distribution while eliminating the management overhead irrespective of the network size due to independent client decision-making. We establish the optimal parameters of the Markov chain model and validate our approach through simulations. The results demonstrate that reducing the load metric variance not only promotes fairness and improves operational efficiency, but also enhances the convergence rate of the learning models.

Title: A Survey on the Applications of Zero-Knowledge Proofs

• Authors: Ryan Lavin, Xuekai Liu, Hardhik Mohanty, Logan Norman, Giovanni Zaarour, Bhaskar Krishnamachari
• Subjects: cs.CR, cs.CC
• Abstract URL: https://arxiv.org/abs/2408.00243
• Pdf URL: https://arxiv.org/pdf/2408.00243
• Copy Paste: [[2408.00243]] A Survey on the Applications of Zero-Knowledge Proofs(https://arxiv.org/abs/2408.00243)
• Keywords: secure, security, privacy
• Abstract: Zero-knowledge proofs (ZKPs) represent a revolutionary advance in computational integrity and privacy technology, enabling the secure and private exchange of information without revealing underlying private data. ZKPs have unique advantages in terms of universality and minimal security assumptions when compared to other privacy-sensitive computational methods for distributed systems, such as homomorphic encryption and secure multiparty computation. Their application spans multiple domains, from enhancing privacy in blockchain to facilitating confidential verification of computational tasks. This survey starts with a high-level overview of the technical workings of ZKPs with a focus on an increasingly relevant subset of ZKPs called zk-SNARKS. While there have been prior surveys on the algorithmic and theoretical aspects of ZKPs, our work is distinguished by providing a broader view of practical aspects and describing many recently-developed use cases of ZKPs across various domains. These application domains span blockchain privacy, scaling, storage, and interoperability, as well as non-blockchain applications like voting, authentication, timelocks, and machine learning. Aimed at both practitioners and researchers, the survey also covers foundational components and infrastructure such as zero-knowledge virtual machines (zkVM), domain-specific languages (DSLs), supporting libraries, frameworks, and protocols. We conclude with a discussion on future directions, positioning ZKPs as pivotal in the advancement of cryptographic practices and digital privacy across many applications.

Title: Enhanced Structured State Space Models via Grouped FIR Filtering and Attention Sink Mechanisms

• Authors: Tian Meng, Yang Tao, Wuliang Yin
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.00244
• Pdf URL: https://arxiv.org/pdf/2408.00244
• Copy Paste: [[2408.00244]] Enhanced Structured State Space Models via Grouped FIR Filtering and Attention Sink Mechanisms(https://arxiv.org/abs/2408.00244)
• Keywords: transformer
• Abstract: Structured State Space Models (SSMs) have emerged as compelling alternatives to Transformer architectures, offering linear-time complexity and superior performance in various sequence modeling tasks. Despite their advantages, SSMs like the original Mamba-2 face training difficulties due to the sensitivities introduced by the extended series of recurrent matrix multiplications. In this paper, we propose an advanced architecture that mitigates these challenges by decomposing A-multiplications into multiple groups and optimizing positional encoding through Grouped Finite Impulse Response (FIR) filtering. This new structure, denoted as Grouped FIR-enhanced SSM (GFSSM), employs semiseparable matrices for efficient computation. Furthermore, inspired by the "attention sink" phenomenon identified in streaming language models, we incorporate a similar mechanism to enhance the stability and performance of our model over extended sequences. Our approach further bridges the gap between SSMs and Transformer architectures, offering a viable path forward for scalable and high-performing sequence modeling.

Title: Task-Adapter: Task-specific Adaptation of Image Models for Few-shot Action Recognition

• Authors: Congqi Cao, Yueran Zhang, Yating Yu, Qinyi Lv, Lingtong Min, Yanning Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.00249
• Pdf URL: https://arxiv.org/pdf/2408.00249
• Copy Paste: [[2408.00249]] Task-Adapter: Task-specific Adaptation of Image Models for Few-shot Action Recognition(https://arxiv.org/abs/2408.00249)
• Keywords: extraction
• Abstract: Existing works in few-shot action recognition mostly fine-tune a pre-trained image model and design sophisticated temporal alignment modules at feature level. However, simply fully fine-tuning the pre-trained model could cause overfitting due to the scarcity of video samples. Additionally, we argue that the exploration of task-specific information is insufficient when relying solely on well extracted abstract features. In this work, we propose a simple but effective task-specific adaptation method (Task-Adapter) for few-shot action recognition. By introducing the proposed Task-Adapter into the last several layers of the backbone and keeping the parameters of the original pre-trained model frozen, we mitigate the overfitting problem caused by full fine-tuning and advance the task-specific mechanism into the process of feature extraction. In each Task-Adapter, we reuse the frozen self-attention layer to perform task-specific self-attention across different videos within the given task to capture both distinctive information among classes and shared information within classes, which facilitates task-specific adaptation and enhances subsequent metric measurement between the query feature and support prototypes. Experimental results consistently demonstrate the effectiveness of our proposed Task-Adapter on four standard few-shot action recognition datasets. Especially on temporal challenging SSv2 dataset, our method outperforms the state-of-the-art methods by a large margin.

Title: Revocable Backdoor for Deep Model Trading

• Authors: Yiran Xu, Nan Zhong, Zhenxing Qian, Xinpeng Zhang
• Subjects: cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2408.00255
• Pdf URL: https://arxiv.org/pdf/2408.00255
• Copy Paste: [[2408.00255]] Revocable Backdoor for Deep Model Trading(https://arxiv.org/abs/2408.00255)
• Keywords: attack, robust
• Abstract: Deep models are being applied in numerous fields and have become a new important digital product. Meanwhile, previous studies have shown that deep models are vulnerable to backdoor attacks, in which compromised models return attacker-desired results when a trigger appears. Backdoor attacks severely break the trust-worthiness of deep models. In this paper, we turn this weakness of deep models into a strength, and propose a novel revocable backdoor and deep model trading scenario. Specifically, we aim to compromise deep models without degrading their performance, meanwhile, we can easily detoxify poisoned models without re-training the models. We design specific mask matrices to manage the internal feature maps of the models. These mask matrices can be used to deactivate the backdoors. The revocable backdoor can be adopted in the deep model trading scenario. Sellers train models with revocable backdoors as a trial version. Buyers pay a deposit to sellers and obtain a trial version of the deep model. If buyers are satisfied with the trial version, they pay a final payment to sellers and sellers send mask matrices to buyers to withdraw revocable backdoors. We demonstrate the feasibility and robustness of our revocable backdoor by various datasets and network architectures.

Title: Mobility-Aware Federated Self-supervised Learning in Vehicular Network

• Authors: Xueying Gu, Qiong Wu, Pingyi Fan, Qiang Fan
• Subjects: cs.LG, cs.NI
• Abstract URL: https://arxiv.org/abs/2408.00256
• Pdf URL: https://arxiv.org/pdf/2408.00256
• Copy Paste: [[2408.00256]] Mobility-Aware Federated Self-supervised Learning in Vehicular Network(https://arxiv.org/abs/2408.00256)
• Keywords: privacy, protect, federate
• Abstract: Federated Learning (FL) is an advanced distributed machine learning approach, that protects the privacy of each vehicle by allowing the model to be trained on multiple devices simultaneously without the need to upload all data to a road side unit (RSU). This enables FL to handle scenarios with sensitive or widely distributed data. However, in these fields, it is well known that the labeling costs can be a significant expense, and models relying on labels are not suitable for these rapidly evolving fields especially in vehicular networks, or mobile internet of things (MIoT), where new data emerges constantly. To handle this issue, the self-supervised learning paves the way for training without labels. Additionally, for vehicles with high velocity, owing to blurred images, simple aggregation not only impacts the accuracy of the aggregated model but also reduces the convergence speed of FL. This paper proposes a FL algorithm based on image blur level to aggregation, called FLSimCo, which does not require labels and serves as a pre-training stage for self-supervised learning in the vehicular environment. Simulation results demonstrate that the proposed algorithm exhibits fast and stable convergence.

Title: Improving Image De-raining Using Reference-Guided Transformers

• Authors: Zihao Ye, Jaehoon Cho, Changjae Oh
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.00258
• Pdf URL: https://arxiv.org/pdf/2408.00258
• Copy Paste: [[2408.00258]] Improving Image De-raining Using Reference-Guided Transformers(https://arxiv.org/abs/2408.00258)
• Keywords: robust, transformer
• Abstract: Image de-raining is a critical task in computer vision to improve visibility and enhance the robustness of outdoor vision systems. While recent advances in de-raining methods have achieved remarkable performance, the challenge remains to produce high-quality and visually pleasing de-rained results. In this paper, we present a reference-guided de-raining filter, a transformer network that enhances de-raining results using a reference clean image as guidance. We leverage the capabilities of the proposed module to further refine the images de-rained by existing methods. We validate our method on three datasets and show that our module can improve the performance of existing prior-based, CNN-based, and transformer-based approaches.

Title: Clover-2: Accurate Inference for Regressive Lightweight Speculative Decoding

• Authors: Bin Xiao, Lujun Gui, Lei Su, Weipeng Chen
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.00264
• Pdf URL: https://arxiv.org/pdf/2408.00264
• Copy Paste: [[2408.00264]] Clover-2: Accurate Inference for Regressive Lightweight Speculative Decoding(https://arxiv.org/abs/2408.00264)
• Keywords: robust, transformer, large language model
• Abstract: Large Language Models (LLMs) frequently suffer from inefficiencies, largely attributable to the discord between the requirements of auto-regressive decoding and the architecture of contemporary GPUs. Recently, regressive lightweight speculative decoding has garnered attention for its notable efficiency improvements in text generation tasks. This approach utilizes a lightweight regressive draft model, like a Recurrent Neural Network (RNN) or a single transformer decoder layer, leveraging sequential information to iteratively predict potential tokens. Specifically, RNN draft models are computationally economical but tend to deliver lower accuracy, while attention decoder layer models exhibit the opposite traits. This paper presents Clover-2, an advanced iteration of Clover, an RNN-based draft model designed to achieve comparable accuracy to that of attention decoder layer models while maintaining minimal computational overhead. Clover-2 enhances the model architecture and incorporates knowledge distillation to increase Clover's accuracy and improve overall efficiency. We conducted experiments using the open-source Vicuna 7B and LLaMA3-Instruct 8B models. The results demonstrate that Clover-2 surpasses existing methods across various model architectures, showcasing its efficacy and robustness.

Title: QUITO: Accelerating Long-Context Reasoning through Query-Guided Context Compression

• Authors: Wenshan Wang, Yihang Wang, Yixing Fan, Huaming Liao, Jiafeng Guo
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.00274
• Pdf URL: https://arxiv.org/pdf/2408.00274
• Copy Paste: [[2408.00274]] QUITO: Accelerating Long-Context Reasoning through Query-Guided Context Compression(https://arxiv.org/abs/2408.00274)
• Keywords: large language model
• Abstract: In-context learning (ICL) capabilities are foundational to the success of large language models (LLMs). Recently, context compression has attracted growing interest since it can largely reduce reasoning complexities and computation costs of LLMs. In this paper, we introduce a novel Query-gUIded aTtention cOmpression (QUITO) method, which leverages attention of the question over the contexts to filter useless information. Specifically, we take a trigger token to calculate the attention distribution of the context in response to the question. Based on the distribution, we propose three different filtering methods to satisfy the budget constraints of the context length. We evaluate the QUITO using two widely-used datasets, namely, NaturalQuestions and ASQA. Experimental results demonstrate that QUITO significantly outperforms established baselines across various datasets and downstream LLMs, underscoring its effectiveness. Our code is available at this https URL.

Title: DMESA: Densely Matching Everything by Segmenting Anything

• Authors: Yesheng Zhang, Xu Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.00279
• Pdf URL: https://arxiv.org/pdf/2408.00279
• Copy Paste: [[2408.00279]] DMESA: Densely Matching Everything by Segmenting Anything(https://arxiv.org/abs/2408.00279)
• Keywords: robust
• Abstract: We propose MESA and DMESA as novel feature matching methods, which utilize Segment Anything Model (SAM) to effectively mitigate matching redundancy. The key insight of our methods is to establish implicit-semantic area matching prior to point matching, based on advanced image understanding of SAM. Then, informative area matches with consistent internal semantic are able to undergo dense feature comparison, facilitating precise inside-area point matching. Specifically, MESA adopts a sparse matching framework and first obtains candidate areas from SAM results through a novel Area Graph (AG). Then, area matching among the candidates is formulated as graph energy minimization and solved by graphical models derived from AG. To address the efficiency issue of MESA, we further propose DMESA as its dense counterpart, applying a dense matching framework. After candidate areas are identified by AG, DMESA establishes area matches through generating dense matching distributions. The distributions are produced from off-the-shelf patch matching utilizing the Gaussian Mixture Model and refined via the Expectation Maximization. With less repetitive computation, DMESA showcases a speed improvement of nearly five times compared to MESA, while maintaining competitive accuracy. Our methods are extensively evaluated on five datasets encompassing indoor and outdoor scenes. The results illustrate consistent performance improvements from our methods for five distinct point matching baselines across all datasets. Furthermore, our methods exhibit promise generalization and improved robustness against image resolution variations. The code is publicly available at this https URL.

Title: Navigating Text-to-Image Generative Bias across Indic Languages

• Authors: Surbhi Mittal, Arnav Sudan, Mayank Vatsa, Richa Singh, Tamar Glaser, Tal Hassner
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2408.00283
• Pdf URL: https://arxiv.org/pdf/2408.00283
• Copy Paste: [[2408.00283]] Navigating Text-to-Image Generative Bias across Indic Languages(https://arxiv.org/abs/2408.00283)
• Keywords: diffusion, generative
• Abstract: This research investigates biases in text-to-image (TTI) models for the Indic languages widely spoken across India. It evaluates and compares the generative performance and cultural relevance of leading TTI models in these languages against their performance in English. Using the proposed IndicTTI benchmark, we comprehensively assess the performance of 30 Indic languages with two open-source diffusion models and two commercial generation APIs. The primary objective of this benchmark is to evaluate the support for Indic languages in these models and identify areas needing improvement. Given the linguistic diversity of 30 languages spoken by over 1.4 billion people, this benchmark aims to provide a detailed and insightful analysis of TTI models' effectiveness within the Indic linguistic landscape. The data and code for the IndicTTI benchmark can be accessed at this https URL.

Title: Bailing-TTS: Chinese Dialectal Speech Synthesis Towards Human-like Spontaneous Representation

• Authors: Xinhan Di, Zihao Chen, Yunming Liang, Junjie Zheng, Yihua Wang, Chaofan Ding
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2408.00284
• Pdf URL: https://arxiv.org/pdf/2408.00284
• Copy Paste: [[2408.00284]] Bailing-TTS: Chinese Dialectal Speech Synthesis Towards Human-like Spontaneous Representation(https://arxiv.org/abs/2408.00284)
• Keywords: transformer
• Abstract: Large-scale text-to-speech (TTS) models have made significant progress recently.However, they still fall short in the generation of Chinese dialectal speech. Toaddress this, we propose Bailing-TTS, a family of large-scale TTS models capable of generating high-quality Chinese dialectal speech. Bailing-TTS serves as a foundation model for Chinese dialectal speech generation. First, continual semi-supervised learning is proposed to facilitate the alignment of text tokens and speech tokens. Second, the Chinese dialectal representation learning is developed using a specific transformer architecture and multi-stage training processes. With the proposed design of novel network architecture and corresponding strategy, Bailing-TTS is able to generate Chinese dialectal speech from text effectively and efficiently. Experiments demonstrate that Bailing-TTS generates Chinese dialectal speech towards human-like spontaneous representation. Readers are encouraged to listen to demos at \url{this https URL}.

Title: Diff3DETR:Agent-based Diffusion Model for Semi-supervised 3D Object Detection

• Authors: Jiacheng Deng, Jiahao Lu, Tianzhu Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.00286
• Pdf URL: https://arxiv.org/pdf/2408.00286
• Copy Paste: [[2408.00286]] Diff3DETR:Agent-based Diffusion Model for Semi-supervised 3D Object Detection(https://arxiv.org/abs/2408.00286)
• Keywords: diffusion, transformer
• Abstract: 3D object detection is essential for understanding 3D scenes. Contemporary techniques often require extensive annotated training data, yet obtaining point-wise annotations for point clouds is time-consuming and laborious. Recent developments in semi-supervised methods seek to mitigate this problem by employing a teacher-student framework to generate pseudo-labels for unlabeled point clouds. However, these pseudo-labels frequently suffer from insufficient diversity and inferior quality. To overcome these hurdles, we introduce an Agent-based Diffusion Model for Semi-supervised 3D Object Detection (Diff3DETR). Specifically, an agent-based object query generator is designed to produce object queries that effectively adapt to dynamic scenes while striking a balance between sampling locations and content embedding. Additionally, a box-aware denoising module utilizes the DDIM denoising process and the long-range attention in the transformer decoder to refine bounding boxes incrementally. Extensive experiments on ScanNet and SUN RGB-D datasets demonstrate that Diff3DETR outperforms state-of-the-art semi-supervised 3D object detection methods.

Title: Multi-Modal Parameter-Efficient Fine-tuning via Graph Neural Network

• Authors: Bin Cheng, Jiaxuan Lu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.00290
• Pdf URL: https://arxiv.org/pdf/2408.00290
• Copy Paste: [[2408.00290]] Multi-Modal Parameter-Efficient Fine-tuning via Graph Neural Network(https://arxiv.org/abs/2408.00290)
• Keywords: large language model
• Abstract: With the advent of the era of foundation models, pre-training and fine-tuning have become common paradigms. Recently, parameter-efficient fine-tuning has garnered widespread attention due to its better balance between the number of learnable parameters and performance. However, some current parameter-efficient fine-tuning methods only model a single modality and lack the utilization of structural knowledge in downstream tasks. To address this issue, this paper proposes a multi-modal parameter-efficient fine-tuning method based on graph networks. Each image is fed into a multi-modal large language model (MLLM) to generate a text description. The image and its corresponding text description are then processed by a frozen image encoder and text encoder to generate image features and text features, respectively. A graph is constructed based on the similarity of the multi-modal feature nodes, and knowledge and relationships relevant to these features are extracted from each node. Additionally, Elastic Weight Consolidation (EWC) regularization is incorporated into the loss function to mitigate the problem of forgetting during task learning. The proposed model achieves test accuracies on the OxfordPets, Flowers102, and Food101 datasets that improve by 4.45%, 2.92%, and 0.23%, respectively. The code is available at this https URL.

Title: RDP: Ranked Differential Privacy for Facial Feature Protection in Multiscale Sparsified Subspace

• Authors: Lu Ou, Shaolin Liao, Shihui Gao, Guandong Huang, Zheng Qi
• Subjects: cs.CV, cs.IR
• Abstract URL: https://arxiv.org/abs/2408.00294
• Pdf URL: https://arxiv.org/pdf/2408.00294
• Copy Paste: [[2408.00294]] RDP: Ranked Differential Privacy for Facial Feature Protection in Multiscale Sparsified Subspace(https://arxiv.org/abs/2408.00294)
• Keywords: privacy, protect
• Abstract: With the widespread sharing of personal face images in applications' public databases, face recognition systems faces real threat of being breached by potential adversaries who are able to access users' face images and use them to intrude the face recognition systems. In this paper, we propose a novel privacy protection method in the multiscale sparsified feature subspaces to protect sensitive facial features, by taking care of the influence or weight ranked feature coefficients on the privacy budget, named "Ranked Differential Privacy (RDP)". After the multiscale feature decomposition, the lightweight Laplacian noise is added to the dimension-reduced sparsified feature coefficients according to the geometric superposition method. Then, we rigorously prove that the RDP satisfies Differential Privacy. After that, the nonlinear Lagrange Multiplier (LM) method is formulated for the constraint optimization problem of maximizing the utility of the visualization quality protected face images with sanitizing noise, under a given facial features privacy budget. Then, two methods are proposed to solve the nonlinear LM problem and obtain the optimal noise scale parameters: 1) the analytical Normalization Approximation (NA) method with identical average noise scale parameter for real-time online applications; and 2) the LM optimization Gradient Descent (LMGD) numerical method to obtain the nonlinear solution through iterative updating for more accurate offline applications. Experimental results on two real-world datasets show that our proposed RDP outperforms other state-of-the-art methods: at a privacy budget of 0.2, the PSNR (Peak Signal-to-Noise Ratio) of the RDP is about ~10 dB higher than (10 times as high as) the highest PSNR of all compared methods.

Title: Contrastive Graph Representation Learning with Adversarial Cross-view Reconstruction and Information Bottleneck

• Authors: Yuntao Shou, Haozhi Lan, Xiangyong Cao
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.00295
• Pdf URL: https://arxiv.org/pdf/2408.00295
• Copy Paste: [[2408.00295]] Contrastive Graph Representation Learning with Adversarial Cross-view Reconstruction and Information Bottleneck(https://arxiv.org/abs/2408.00295)
• Keywords: robust
• Abstract: Graph Neural Networks (GNNs) have received extensive research attention due to their powerful information aggregation capabilities. Despite the success of GNNs, most of them suffer from the popularity bias issue in a graph caused by a small number of popular categories. Additionally, real graph datasets always contain incorrect node labels, which hinders GNNs from learning effective node representations. Graph contrastive learning (GCL) has been shown to be effective in solving the above problems for node classification tasks. Most existing GCL methods are implemented by randomly removing edges and nodes to create multiple contrasting views, and then maximizing the mutual information (MI) between these contrasting views to improve the node feature representation. However, maximizing the mutual information between multiple contrasting views may lead the model to learn some redundant information irrelevant to the node classification task. To tackle this issue, we propose an effective Contrastive Graph Representation Learning with Adversarial Cross-view Reconstruction and Information Bottleneck (CGRL) for node classification, which can adaptively learn to mask the nodes and edges in the graph to obtain the optimal graph structure representation. Furthermore, we innovatively introduce the information bottleneck theory into GCLs to remove redundant information in multiple contrasting views while retaining as much information as possible about node classification. Moreover, we add noise perturbations to the original views and reconstruct the augmented views by constructing adversarial views to improve the robustness of node feature representation. Extensive experiments on real-world public datasets demonstrate that our method significantly outperforms existing state-of-the-art algorithms.

Title: Towards Flexible Evaluation for Generative Visual Question Answering

• Authors: Huishan Ji, Qingyi Si, Zheng Lin, Weiping Wang
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2408.00300
• Pdf URL: https://arxiv.org/pdf/2408.00300
• Copy Paste: [[2408.00300]] Towards Flexible Evaluation for Generative Visual Question Answering(https://arxiv.org/abs/2408.00300)
• Keywords: fair, generative, large language model
• Abstract: Throughout rapid development of multimodal large language models, a crucial ingredient is a fair and accurate evaluation of their multimodal comprehension abilities. Although Visual Question Answering (VQA) could serve as a developed test field, limitations of VQA evaluation, like the inflexible pattern of Exact Match, have hindered MLLMs from demonstrating their real capability and discourage rich responses. Therefore, this paper proposes the use of semantics-based evaluators for assessing unconstrained open-ended responses on VQA datasets. As characteristics of VQA have made such evaluation significantly different than the traditional Semantic Textual Similarity (STS) task, to systematically analyze the behaviour and compare the performance of various evaluators including LLM-based ones, we proposes three key properties, i.e., Alignment, Consistency and Generalization, and a corresponding dataset Assessing VQA Evaluators (AVE) to facilitate analysis. In addition, this paper proposes a Semantically Flexible VQA Evaluator (SFVE) with meticulous design based on the unique features of VQA evaluation. Experimental results verify the feasibility of model-based VQA evaluation and effectiveness of the proposed evaluator that surpasses existing semantic evaluators by a large margin. The proposed training scheme generalizes to both the BERT-like encoders and decoder-only LLM.

Title: ABC Align: Large Language Model Alignment for Safety & Accuracy

• Authors: Gareth Seneque, Lap-Hang Ho, Ariel Kuperman, Nafise Erfanian Saeedi, Jeffrey Molendijk
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2408.00307
• Pdf URL: https://arxiv.org/pdf/2408.00307
• Copy Paste: [[2408.00307]] ABC Align: Large Language Model Alignment for Safety & Accuracy(https://arxiv.org/abs/2408.00307)
• Keywords: large language model
• Abstract: Alignment of Large Language Models (LLMs) remains an unsolved problem. Human preferences are highly distributed and can be captured at multiple levels of abstraction, from the individual to diverse populations. Organisational preferences, represented by standards and principles, are defined to mitigate reputational risk or meet legislative obligations. In this paper, we present ABC Align, a novel alignment methodology for LLMs that enables integration of the standards and preferences of a large media organisation into the LLM itself. We combine a set of data and methods that build on recent breakthroughs in synthetic data generation, preference optimisation, and post-training model quantisation. Our unified approach mitigates bias and improves accuracy, while preserving reasoning capability, as measured against standard benchmarks.

Title: Translating Imaging to Genomics: Leveraging Transformers for Predictive Modeling

• Authors: Aiman Farooq, Deepak Mishra, Santanu Chaudhury
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.00311
• Pdf URL: https://arxiv.org/pdf/2408.00311
• Copy Paste: [[2408.00311]] Translating Imaging to Genomics: Leveraging Transformers for Predictive Modeling(https://arxiv.org/abs/2408.00311)
• Keywords: transformer
• Abstract: In this study, we present a novel approach for predicting genomic information from medical imaging modalities using a transformer-based model. We aim to bridge the gap between imaging and genomics data by leveraging transformer networks, allowing for accurate genomic profile predictions from CT/MRI images. Presently most studies rely on the use of whole slide images (WSI) for the association, which are obtained via invasive methodologies. We propose using only available CT/MRI images to predict genomic sequences. Our transformer based approach is able to efficiently generate associations between multiple sequences based on CT/MRI images alone. This work paves the way for the use of non-invasive imaging modalities for precise and personalized healthcare, allowing for a better understanding of diseases and treatment.

Title: ADBM: Adversarial diffusion bridge model for reliable adversarial purification

• Authors: Xiao Li, Wenxuan Sun, Huanran Chen, Qiongxiu Li, Yining Liu, Yingzhe He, Jie Shi, Xiaolin Hu
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2408.00315
• Pdf URL: https://arxiv.org/pdf/2408.00315
• Copy Paste: [[2408.00315]] ADBM: Adversarial diffusion bridge model for reliable adversarial purification(https://arxiv.org/abs/2408.00315)
• Keywords: defense, attack, robust, diffusion
• Abstract: Recently Diffusion-based Purification (DiffPure) has been recognized as an effective defense method against adversarial examples. However, we find DiffPure which directly employs the original pre-trained diffusion models for adversarial purification, to be suboptimal. This is due to an inherent trade-off between noise purification performance and data recovery quality. Additionally, the reliability of existing evaluations for DiffPure is questionable, as they rely on weak adaptive attacks. In this work, we propose a novel Adversarial Diffusion Bridge Model, termed ADBM. ADBM directly constructs a reverse bridge from the diffused adversarial data back to its original clean examples, enhancing the purification capabilities of the original diffusion models. Through theoretical analysis and experimental validation across various scenarios, ADBM has proven to be a superior and robust defense mechanism, offering significant promise for practical applications.

Title: OTAD: An Optimal Transport-Induced Robust Model for Agnostic Adversarial Attack

• Authors: Kuo Gai, Sicong Wang, Shihua Zhang
• Subjects: cs.LG, cs.AI, math.OC, stat.ML
• Abstract URL: https://arxiv.org/abs/2408.00329
• Pdf URL: https://arxiv.org/pdf/2408.00329
• Copy Paste: [[2408.00329]] OTAD: An Optimal Transport-Induced Robust Model for Agnostic Adversarial Attack(https://arxiv.org/abs/2408.00329)
• Keywords: secure, defense, attack, robust, transformer
• Abstract: Deep neural networks (DNNs) are vulnerable to small adversarial perturbations of the inputs, posing a significant challenge to their reliability and robustness. Empirical methods such as adversarial training can defend against particular attacks but remain vulnerable to more powerful attacks. Alternatively, Lipschitz networks provide certified robustness to unseen perturbations but lack sufficient expressive power. To harness the advantages of both approaches, we design a novel two-step Optimal Transport induced Adversarial Defense (OTAD) model that can fit the training data accurately while preserving the local Lipschitz continuity. First, we train a DNN with a regularizer derived from optimal transport theory, yielding a discrete optimal transport map linking data to its features. By leveraging the map's inherent regularity, we interpolate the map by solving the convex integration problem (CIP) to guarantee the local Lipschitz property. OTAD is extensible to diverse architectures of ResNet and Transformer, making it suitable for complex data. For efficient computation, the CIP can be solved through training neural networks. OTAD opens a novel avenue for developing reliable and secure deep learning systems through the regularity of optimal transport maps. Empirical results demonstrate that OTAD can outperform other robust models on diverse datasets.

Title: "Patriarchy Hurts Men Too." Does Your Model Agree? A Discussion on Fairness Assumptions

• Authors: Marco Favier, Toon Calders
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.00330
• Pdf URL: https://arxiv.org/pdf/2408.00330
• Copy Paste: [[2408.00330]] "Patriarchy Hurts Men Too." Does Your Model Agree? A Discussion on Fairness Assumptions(https://arxiv.org/abs/2408.00330)
• Keywords: fair
• Abstract: The pipeline of a fair ML practitioner is generally divided into three phases: 1) Selecting a fairness measure. 2) Choosing a model that minimizes this measure. 3) Maximizing the model's performance on the data. In the context of group fairness, this approach often obscures implicit assumptions about how bias is introduced into the data. For instance, in binary classification, it is often assumed that the best model, with equal fairness, is the one with better performance. However, this belief already imposes specific properties on the process that introduced bias. More precisely, we are already assuming that the biasing process is a monotonic function of the fair scores, dependent solely on the sensitive attribute. We formally prove this claim regarding several implicit fairness assumptions. This leads, in our view, to two possible conclusions: either the behavior of the biasing process is more complex than mere monotonicity, which means we need to identify and reject our implicit assumptions in order to develop models capable of tackling more complex situations; or the bias introduced in the data behaves predictably, implying that many of the developed models are superfluous.

Title: DECIDER: Leveraging Foundation Model Priors for Improved Model Failure Detection and Explanation

• Authors: Rakshith Subramanyam, Kowshik Thopalli, Vivek Narayanaswamy, Jayaraman J.Thiagarajan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.00331
• Pdf URL: https://arxiv.org/pdf/2408.00331
• Copy Paste: [[2408.00331]] DECIDER: Leveraging Foundation Model Priors for Improved Model Failure Detection and Explanation(https://arxiv.org/abs/2408.00331)
• Keywords: large language model
• Abstract: Reliably detecting when a deployed machine learning model is likely to fail on a given input is crucial for ensuring safe operation. In this work, we propose DECIDER (Debiasing Classifiers to Identify Errors Reliably), a novel approach that leverages priors from large language models (LLMs) and vision-language models (VLMs) to detect failures in image classification models. DECIDER utilizes LLMs to specify task-relevant core attributes and constructs a ``debiased'' version of the classifier by aligning its visual features to these core attributes using a VLM, and detects potential failure by measuring disagreement between the original and debiased models. In addition to proactively identifying samples on which the model would fail, DECIDER also provides human-interpretable explanations for failure through a novel attribute-ablation strategy. Through extensive experiments across diverse benchmarks spanning subpopulation shifts (spurious correlations, class imbalance) and covariate shifts (synthetic corruptions, domain shifts), DECIDER consistently achieves state-of-the-art failure detection performance, significantly outperforming baselines in terms of the overall Matthews correlation coefficient as well as failure and success recall. Our codes can be accessed at~\url{this https URL}

Title: DistillGrasp: Integrating Features Correlation with Knowledge Distillation for Depth Completion of Transparent Objects

• Authors: Yiheng Huang, Junhong Chen, Nick Michiels, Muhammad Asim, Luc Claesen, Wenyin Liu
• Subjects: cs.CV, cs.IR
• Abstract URL: https://arxiv.org/abs/2408.00337
• Pdf URL: https://arxiv.org/pdf/2408.00337
• Copy Paste: [[2408.00337]] DistillGrasp: Integrating Features Correlation with Knowledge Distillation for Depth Completion of Transparent Objects(https://arxiv.org/abs/2408.00337)
• Keywords: robust
• Abstract: Due to the visual properties of reflection and refraction, RGB-D cameras cannot accurately capture the depth of transparent objects, leading to incomplete depth maps. To fill in the missing points, recent studies tend to explore new visual features and design complex networks to reconstruct the depth, however, these approaches tremendously increase computation, and the correlation of different visual features remains a problem. To this end, we propose an efficient depth completion network named DistillGrasp which distillates knowledge from the teacher branch to the student branch. Specifically, in the teacher branch, we design a position correlation block (PCB) that leverages RGB images as the query and key to search for the corresponding values, guiding the model to establish correct correspondence between two features and transfer it to the transparent areas. For the student branch, we propose a consistent feature correlation module (CFCM) that retains the reliable regions of RGB images and depth maps respectively according to the consistency and adopts a CNN to capture the pairwise relationship for depth completion. To avoid the student branch only learning regional features from the teacher branch, we devise a distillation loss that not only considers the distance loss but also the object structure and edge information. Extensive experiments conducted on the ClearGrasp dataset manifest that our teacher network outperforms state-of-the-art methods in terms of accuracy and generalization, and the student network achieves competitive results with a higher speed of 48 FPS. In addition, the significant improvement in a real-world robotic grasping system illustrates the effectiveness and robustness of our proposed system.

Title: Advancing Medical Image Segmentation: Morphology-Driven Learning with Diffusion Transformer

• Authors: Sungmin Kang, Jaeha Song, Jihie Kim
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.00347
• Pdf URL: https://arxiv.org/pdf/2408.00347
• Copy Paste: [[2408.00347]] Advancing Medical Image Segmentation: Morphology-Driven Learning with Diffusion Transformer(https://arxiv.org/abs/2408.00347)
• Keywords: robust, diffusion, transformer, segmentation
• Abstract: Understanding the morphological structure of medical images and precisely segmenting the region of interest or abnormality is an important task that can assist in diagnosis. However, the unique properties of medical imaging make clear segmentation difficult, and the high cost and time-consuming task of labeling leads to a coarse-grained representation of ground truth. Facing with these problems, we propose a novel Diffusion Transformer Segmentation (DTS) model for robust segmentation in the presence of noise. We propose an alternative to the dominant Denoising U-Net encoder through experiments applying a transformer architecture, which captures global dependency through self-attention. Additionally, we propose k-neighbor label smoothing, reverse boundary attention, and self-supervised learning with morphology-driven learning to improve the ability to identify complex structures. Our model, which analyzes the morphological representation of images, shows better results than the previous models in various medical imaging modalities, including CT, MRI, and lesion images.

Title: Securing the Diagnosis of Medical Imaging: An In-depth Analysis of AI-Resistant Attacks

• Authors: Angona Biswas, MD Abdullah Al Nasim, Kishor Datta Gupta, Roy George, Abdur Rashid
• Subjects: cs.CR, cs.AI, eess.IV
• Abstract URL: https://arxiv.org/abs/2408.00348
• Pdf URL: https://arxiv.org/pdf/2408.00348
• Copy Paste: [[2408.00348]] Securing the Diagnosis of Medical Imaging: An In-depth Analysis of AI-Resistant Attacks(https://arxiv.org/abs/2408.00348)
• Keywords: security, attack, robust
• Abstract: Machine learning (ML) is a rapidly developing area of medicine that uses significant resources to apply computer science and statistics to medical issues. ML's proponents laud its capacity to handle vast, complicated, and erratic medical data. It's common knowledge that attackers might cause misclassification by deliberately creating inputs for machine learning classifiers. Research on adversarial examples has been extensively conducted in the field of computer vision applications. Healthcare systems are thought to be highly difficult because of the security and life-or-death considerations they include, and performance accuracy is very important. Recent arguments have suggested that adversarial attacks could be made against medical image analysis (MedIA) technologies because of the accompanying technology infrastructure and powerful financial incentives. Since the diagnosis will be the basis for important decisions, it is essential to assess how strong medical DNN tasks are against adversarial attacks. Simple adversarial attacks have been taken into account in several earlier studies. However, DNNs are susceptible to more risky and realistic attacks. The present paper covers recent proposed adversarial attack strategies against DNNs for medical imaging as well as countermeasures. In this study, we review current techniques for adversarial imaging attacks, detections. It also encompasses various facets of these techniques and offers suggestions for the robustness of neural networks to be improved in the future.

Title: A Simple Background Augmentation Method for Object Detection with Diffusion Model

• Authors: Yuhang Li, Xin Dong, Chen Chen, Weiming Zhuang, Lingjuan Lyu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.00350
• Pdf URL: https://arxiv.org/pdf/2408.00350
• Copy Paste: [[2408.00350]] A Simple Background Augmentation Method for Object Detection with Diffusion Model(https://arxiv.org/abs/2408.00350)
• Keywords: robust, diffusion, generative, segmentation
• Abstract: In computer vision, it is well-known that a lack of data diversity will impair model performance. In this study, we address the challenges of enhancing the dataset diversity problem in order to benefit various downstream tasks such as object detection and instance segmentation. We propose a simple yet effective data augmentation approach by leveraging advancements in generative models, specifically text-to-image synthesis technologies like Stable Diffusion. Our method focuses on generating variations of labeled real images, utilizing generative object and background augmentation via inpainting to augment existing training data without the need for additional annotations. We find that background augmentation, in particular, significantly improves the models' robustness and generalization capabilities. We also investigate how to adjust the prompt and mask to ensure the generated content comply with the existing annotations. The efficacy of our augmentation techniques is validated through comprehensive evaluations of the COCO dataset and several other key object detection benchmarks, demonstrating notable enhancements in model performance across diverse scenarios. This approach offers a promising solution to the challenges of dataset enhancement, contributing to the development of more accurate and robust computer vision models.

Title: Hierarchically Structured Neural Bones for Reconstructing Animatable Objects from Casual Videos

• Authors: Subin Jeon, In Cho, Minsu Kim, Woong Oh Cho, Seon Joo Kim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.00351
• Pdf URL: https://arxiv.org/pdf/2408.00351
• Copy Paste: [[2408.00351]] Hierarchically Structured Neural Bones for Reconstructing Animatable Objects from Casual Videos(https://arxiv.org/abs/2408.00351)
• Keywords: interpretability
• Abstract: We propose a new framework for creating and easily manipulating 3D models of arbitrary objects using casually captured videos. Our core ingredient is a novel hierarchy deformation model, which captures motions of objects with a tree-structured bones. Our hierarchy system decomposes motions based on the granularity and reveals the correlations between parts without exploiting any prior structural knowledge. We further propose to regularize the bones to be positioned at the basis of motions, centers of parts, sufficiently covering related surfaces of the part. This is achieved by our bone occupancy function, which identifies whether a given 3D point is placed within the bone. Coupling the proposed components, our framework offers several clear advantages: (1) users can obtain animatable 3D models of the arbitrary objects in improved quality from their casual videos, (2) users can manipulate 3D models in an intuitive manner with minimal costs, and (3) users can interactively add or delete control points as necessary. The experimental results demonstrate the efficacy of our framework on diverse instances, in reconstruction quality, interpretability and easier manipulation. Our code is available at this https URL.

Title: Autonomous LLM-Enhanced Adversarial Attack for Text-to-Motion

• Authors: Honglei Miao, Fan Ma, Ruijie Quan, Kun Zhan, Yi Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.00352
• Pdf URL: https://arxiv.org/pdf/2408.00352
• Copy Paste: [[2408.00352]] Autonomous LLM-Enhanced Adversarial Attack for Text-to-Motion(https://arxiv.org/abs/2408.00352)
• Keywords: security, attack, steal, generative, large language model
• Abstract: Human motion generation driven by deep generative models has enabled compelling applications, but the ability of text-to-motion (T2M) models to produce realistic motions from text prompts raises security concerns if exploited maliciously. Despite growing interest in T2M, few methods focus on safeguarding these models against adversarial attacks, with existing work on text-to-image models proving insufficient for the unique motion domain. In the paper, we propose ALERT-Motion, an autonomous framework leveraging large language models (LLMs) to craft targeted adversarial attacks against black-box T2M models. Unlike prior methods modifying prompts through predefined rules, ALERT-Motion uses LLMs' knowledge of human motion to autonomously generate subtle yet powerful adversarial text descriptions. It comprises two key modules: an adaptive dispatching module that constructs an LLM-based agent to iteratively refine and search for adversarial prompts; and a multimodal information contrastive module that extracts semantically relevant motion information to guide the agent's search. Through this LLM-driven approach, ALERT-Motion crafts adversarial prompts querying victim models to produce outputs closely matching targeted motions, while avoiding obvious perturbations. Evaluations across popular T2M models demonstrate ALERT-Motion's superiority over previous methods, achieving higher attack success rates with stealthier adversarial prompts. This pioneering work on T2M adversarial attacks highlights the urgency of developing defensive measures as motion generation technology advances, urging further research into safe and responsible deployment.

Title: DNTextSpotter: Arbitrary-Shaped Scene Text Spotting via Improved Denoising Training

• Authors: Yu Xie, Qian Qiao, Jun Gao, Tianxiang Wu, Shaoyao Huang, Jiaqing Fan, Ziqiang Cao, Zili Wang, Yue Zhang, Jielei Zhang, Huyang Sun
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.00355
• Pdf URL: https://arxiv.org/pdf/2408.00355
• Copy Paste: [[2408.00355]] DNTextSpotter: Arbitrary-Shaped Scene Text Spotting via Improved Denoising Training(https://arxiv.org/abs/2408.00355)
• Keywords: transformer
• Abstract: More and more end-to-end text spotting methods based on Transformer architecture have demonstrated superior performance. These methods utilize a bipartite graph matching algorithm to perform one-to-one optimal matching between predicted objects and actual objects. However, the instability of bipartite graph matching can lead to inconsistent optimization targets, thereby affecting the training performance of the model. Existing literature applies denoising training to solve the problem of bipartite graph matching instability in object detection tasks. Unfortunately, this denoising training method cannot be directly applied to text spotting tasks, as these tasks need to perform irregular shape detection tasks and more complex text recognition tasks than classification. To address this issue, we propose a novel denoising training method (DNTextSpotter) for arbitrary-shaped text spotting. Specifically, we decompose the queries of the denoising part into noised positional queries and noised content queries. We use the four Bezier control points of the Bezier center curve to generate the noised positional queries. For the noised content queries, considering that the output of the text in a fixed positional order is not conducive to aligning position with content, we employ a masked character sliding method to initialize noised content queries, thereby assisting in the alignment of text content and position. To improve the model's perception of the background, we further utilize an additional loss function for background characters classification in the denoising training part.Although DNTextSpotter is conceptually simple, it outperforms the state-of-the-art methods on four benchmarks (Total-Text, SCUT-CTW1500, ICDAR15, and Inverse-Text), especially yielding an improvement of 11.3% against the best approach in Inverse-Text dataset.

Title: DeliLaw: A Chinese Legal Counselling System Based on a Large Language Model

• Authors: Nan Xie, Yuelin Bai, Hengyuan Gao, Feiteng Fang, Qixuan Zhao, Zhijian Li, Ziqiang Xue, Liang Zhu, Shiwen Ni, Min Yang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.00357
• Pdf URL: https://arxiv.org/pdf/2408.00357
• Copy Paste: [[2408.00357]] DeliLaw: A Chinese Legal Counselling System Based on a Large Language Model(https://arxiv.org/abs/2408.00357)
• Keywords: large language model
• Abstract: Traditional legal retrieval systems designed to retrieve legal documents, statutes, precedents, and other legal information are unable to give satisfactory answers due to lack of semantic understanding of specific questions. Large Language Models (LLMs) have achieved excellent results in a variety of natural language processing tasks, which inspired us that we train a LLM in the legal domain to help legal retrieval. However, in the Chinese legal domain, due to the complexity of legal questions and the rigour of legal articles, there is no legal large model with satisfactory practical application yet. In this paper, we present DeliLaw, a Chinese legal counselling system based on a large language model. DeliLaw integrates a legal retrieval module and a case retrieval module to overcome the model hallucination. Users can consult professional legal questions, search for legal articles and relevant judgement cases, etc. on the DeliLaw system in a dialogue mode. In addition, DeliLaw supports the use of English for counseling. we provide the address of the system: this https URL.

Title: Few-shot Defect Image Generation based on Consistency Modeling

• Authors: Qingfeng Shi, Jing Wei, Fei Shen, Zhengtao Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.00372
• Pdf URL: https://arxiv.org/pdf/2408.00372
• Copy Paste: [[2408.00372]] Few-shot Defect Image Generation based on Consistency Modeling(https://arxiv.org/abs/2408.00372)
• Keywords: diffusion
• Abstract: Image generation can solve insufficient labeled data issues in defect detection. Most defect generation methods are only trained on a single product without considering the consistencies among multiple products, leading to poor quality and diversity of generated results. To address these issues, we propose DefectDiffu, a novel text-guided diffusion method to model both intra-product background consistency and inter-product defect consistency across multiple products and modulate the consistency perturbation directions to control product type and defect strength, achieving diversified defect image generation. Firstly, we leverage a text encoder to separately provide consistency prompts for background, defect, and fusion parts of the disentangled integrated architecture, thereby disentangling defects and normal backgrounds. Secondly, we propose the double-free strategy to generate defect images through two-stage perturbation of consistency direction, thereby controlling product type and defect strength by adjusting the perturbation scale. Besides, DefectDiffu can generate defect mask annotations utilizing cross-attention maps from the defect part. Finally, to improve the generation quality of small defects and masks, we propose the adaptive attention-enhance loss to increase the attention to defects. Experimental results demonstrate that DefectDiffu surpasses state-of-the-art methods in terms of generation quality and diversity, thus effectively improving downstream defection performance. Moreover, defect perturbation directions can be transferred among various products to achieve zero-shot defect generation, which is highly beneficial for addressing insufficient data issues. The code are available at this https URL.

Title: On the Limitations and Prospects of Machine Unlearning for Generative AI

• Authors: Shiji Zhou, Lianzhe Wang, Jiangnan Ye, Yongliang Wu, Heng Chang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.00376
• Pdf URL: https://arxiv.org/pdf/2408.00376
• Copy Paste: [[2408.00376]] On the Limitations and Prospects of Machine Unlearning for Generative AI(https://arxiv.org/abs/2408.00376)
• Keywords: security, privacy, diffusion, generative
• Abstract: Generative AI (GenAI), which aims to synthesize realistic and diverse data samples from latent variables or other data modalities, has achieved remarkable results in various domains, such as natural language, images, audio, and graphs. However, they also pose challenges and risks to data privacy, security, and ethics. Machine unlearning is the process of removing or weakening the influence of specific data samples or features from a trained model, without affecting its performance on other data or tasks. While machine unlearning has shown significant efficacy in traditional machine learning tasks, it is still unclear if it could help GenAI become safer and aligned with human desire. To this end, this position paper provides an in-depth discussion of the machine unlearning approaches for GenAI. Firstly, we formulate the problem of machine unlearning tasks on GenAI and introduce the background. Subsequently, we systematically examine the limitations of machine unlearning on GenAI models by focusing on the two representative branches: LLMs and image generative (diffusion) models. Finally, we provide our prospects mainly from three aspects: benchmark, evaluation metrics, and utility-unlearning trade-off, and conscientiously advocate for the future development of this field.

Title: What comes after transformers? -- A selective survey connecting ideas in deep learning

• Authors: Johannes Schneider
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.00386
• Pdf URL: https://arxiv.org/pdf/2408.00386
• Copy Paste: [[2408.00386]] What comes after transformers? -- A selective survey connecting ideas in deep learning(https://arxiv.org/abs/2408.00386)
• Keywords: transformer
• Abstract: Transformers have become the de-facto standard model in artificial intelligence since 2017 despite numerous shortcomings ranging from energy inefficiency to hallucinations. Research has made a lot of progress in improving elements of transformers, and, more generally, deep learning manifesting in many proposals for architectures, layers, optimization objectives, and optimization techniques. For researchers it is difficult to keep track of such developments on a broader level. We provide a comprehensive overview of the many important, recent works in these areas to those who already have a basic understanding of deep learning. Our focus differs from other works, as we target specifically novel, alternative potentially disruptive approaches to transformers as well as successful ideas of recent deep learning. We hope that such a holistic and unified treatment of influential, recent works and novel ideas helps researchers to form new connections between diverse areas of deep learning. We identify and discuss multiple patterns that summarize the key strategies for successful innovations over the last decade as well as works that can be seen as rising stars. Especially, we discuss attempts on how to improve on transformers covering (partially) proven methods such as state space models but also including far-out ideas in deep learning that seem promising despite not achieving state-of-the-art results. We also cover a discussion on recent state-of-the-art models such as OpenAI's GPT series and Meta's LLama models and, Google's Gemini model family.

Title: Deepfake Media Forensics: State of the Art and Challenges Ahead

• Authors: Irene Amerini, Mauro Barni, Sebastiano Battiato, Paolo Bestagini, Giulia Boato, Tania Sari Bonaventura, Vittoria Bruni, Roberto Caldelli, Francesco De Natale, Rocco De Nicola, Luca Guarnera, Sara Mandelli, Gian Luca Marcialis, Marco Micheletto, Andrea Montibeller, Giulia Orru', Alessandro Ortis, Pericle Perazzo, Davide Salvi, Stefano Tubaro, Claudia Melis Tonti, Massimo Villari, Domenico Vitulano
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.00388
• Pdf URL: https://arxiv.org/pdf/2408.00388
• Copy Paste: [[2408.00388]] Deepfake Media Forensics: State of the Art and Challenges Ahead(https://arxiv.org/abs/2408.00388)
• Keywords: security, diffusion, generative
• Abstract: AI-generated synthetic media, also called Deepfakes, have significantly influenced so many domains, from entertainment to cybersecurity. Generative Adversarial Networks (GANs) and Diffusion Models (DMs) are the main frameworks used to create Deepfakes, producing highly realistic yet fabricated content. While these technologies open up new creative possibilities, they also bring substantial ethical and security risks due to their potential misuse. The rise of such advanced media has led to the development of a cognitive bias known as Impostor Bias, where individuals doubt the authenticity of multimedia due to the awareness of AI's capabilities. As a result, Deepfake detection has become a vital area of research, focusing on identifying subtle inconsistencies and artifacts with machine learning techniques, especially Convolutional Neural Networks (CNNs). Research in forensic Deepfake technology encompasses five main areas: detection, attribution and recognition, passive authentication, detection in realistic scenarios, and active authentication. Each area tackles specific challenges, from tracing the origins of synthetic media and examining its inherent characteristics for authenticity. This paper reviews the primary algorithms that address these challenges, examining their advantages, limitations, and future prospects.

Title: A Zero-Knowledge Proof of Knowledge for Subgroup Distance Problem

• Authors: Cansu Betin Onur
• Subjects: cs.CR, math.GR
• Abstract URL: https://arxiv.org/abs/2408.00395
• Pdf URL: https://arxiv.org/pdf/2408.00395
• Copy Paste: [[2408.00395]] A Zero-Knowledge Proof of Knowledge for Subgroup Distance Problem(https://arxiv.org/abs/2408.00395)
• Keywords: secure, security, robust
• Abstract: In this study, we introduce a novel zero-knowledge identification scheme based on the hardness of the subgroup distance problem in the Hamming metric. The proposed protocol, named Subgroup Distance Zero Knowledge Proof (SDZKP), employs a cryptographically secure pseudorandom number generator to mask secrets and utilizes a Stern-type algorithm to ensure robust security properties.

Title: In-Context Example Selection via Similarity Search Improves Low-Resource Machine Translation

• Authors: Armel Zebaze, Benoît Sagot, Rachel Bawden
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.00397
• Pdf URL: https://arxiv.org/pdf/2408.00397
• Copy Paste: [[2408.00397]] In-Context Example Selection via Similarity Search Improves Low-Resource Machine Translation(https://arxiv.org/abs/2408.00397)
• Keywords: generative, large language model
• Abstract: The ability of generative large language models (LLMs) to perform in-context learning has given rise to a large body of research into how best to prompt models for various natural language processing tasks. In this paper, we focus on machine translation (MT), a task that has been shown to benefit from in-context translation examples. However no systematic studies have been published on how best to select examples, and mixed results have been reported on the usefulness of similarity-based selection over random selection. We provide a study covering multiple LLMs and multiple in-context example retrieval strategies, comparing multilingual sentence embeddings. We cover several language directions, representing different levels of language resourcedness (English into French, German, Swahili and Wolof). Contrarily to previously published results, we find that sentence embedding similarity can improve MT, especially for low-resource language directions, and discuss the balance between selection pool diversity and quality. We also highlight potential problems with the evaluation of LLM-based MT and suggest a more appropriate evaluation protocol, adapting the COMET metric to the evaluation of LLMs. Code and outputs are freely available at this https URL.

Title: Towards Reliable Advertising Image Generation Using Human Feedback

• Authors: Zhenbang Du, Wei Feng, Haohan Wang, Yaoyu Li, Jingsen Wang, Jian Li, Zheng Zhang, Jingjing Lv, Xin Zhu, Junsheng Jin, Junjie Shen, Zhangang Lin, Jingping Shao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.00418
• Pdf URL: https://arxiv.org/pdf/2408.00418
• Copy Paste: [[2408.00418]] Towards Reliable Advertising Image Generation Using Human Feedback(https://arxiv.org/abs/2408.00418)
• Keywords: diffusion, generative
• Abstract: In the e-commerce realm, compelling advertising images are pivotal for attracting customer attention. While generative models automate image generation, they often produce substandard images that may mislead customers and require significant labor costs to inspect. This paper delves into increasing the rate of available generated images. We first introduce a multi-modal Reliable Feedback Network (RFNet) to automatically inspect the generated images. Combining the RFNet into a recurrent process, Recurrent Generation, results in a higher number of available advertising images. To further enhance production efficiency, we fine-tune diffusion models with an innovative Consistent Condition regularization utilizing the feedback from RFNet (RFFT). This results in a remarkable increase in the available rate of generated images, reducing the number of attempts in Recurrent Generation, and providing a highly efficient production process without sacrificing visual appeal. We also construct a Reliable Feedback 1 Million (RF1M) dataset which comprises over one million generated advertising images annotated by human, which helps to train RFNet to accurately assess the availability of generated images and faithfully reflect the human feedback. Generally speaking, our approach offers a reliable solution for advertising image generation.

Title: MPT-PAR:Mix-Parameters Transformer for Panoramic Activity Recognition

• Authors: Wenqing Gan, Yan Sun, Feiran Liu, Xiangfeng Luo
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.00420
• Pdf URL: https://arxiv.org/pdf/2408.00420
• Copy Paste: [[2408.00420]] MPT-PAR:Mix-Parameters Transformer for Panoramic Activity Recognition(https://arxiv.org/abs/2408.00420)
• Keywords: transformer
• Abstract: The objective of the panoramic activity recognition task is to identify behaviors at various granularities within crowded and complex environments, encompassing individual actions, social group activities, and global activities. Existing methods generally use either parameter-independent modules to capture task-specific features or parameter-sharing modules to obtain common features across all tasks. However, there is often a strong interrelatedness and complementary effect between tasks of different granularities that previous methods have yet to notice. In this paper, we propose a model called MPT-PAR that considers both the unique characteristics of each task and the synergies between different tasks simultaneously, thereby maximizing the utilization of features across multi-granularity activity recognition. Furthermore, we emphasize the significance of temporal and spatial information by introducing a spatio-temporal relation-enhanced module and a scene representation learning module, which integrate the the spatio-temporal context of action and global scene into the feature map of each granularity. Our method achieved an overall F1 score of 47.5\% on the JRDB-PAR dataset, significantly outperforming all the state-of-the-art methods.

Title: MonoMM: A Multi-scale Mamba-Enhanced Network for Real-time Monocular 3D Object Detection

• Authors: Youjia Fu, Zihao Xu, Junsong Fu, Huixia Xue, Shuqiu Tan, Lei Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.00438
• Pdf URL: https://arxiv.org/pdf/2408.00438
• Copy Paste: [[2408.00438]] MonoMM: A Multi-scale Mamba-Enhanced Network for Real-time Monocular 3D Object Detection(https://arxiv.org/abs/2408.00438)
• Keywords: robust, transformer
• Abstract: Recent advancements in transformer-based monocular 3D object detection techniques have exhibited exceptional performance in inferring 3D attributes from single 2D images. However, most existing methods rely on resource-intensive transformer architectures, which often lead to significant drops in computational efficiency and performance when handling long sequence data. To address these challenges and advance monocular 3D object detection technology, we propose an innovative network architecture, MonoMM, a Multi-scale \textbf{M}amba-Enhanced network for real-time Monocular 3D object detection. This well-designed architecture primarily includes the following two core modules: Focused Multi-Scale Fusion (FMF) Module, which focuses on effectively preserving and fusing image information from different scales with lower computational resource consumption. By precisely regulating the information flow, the FMF module enhances the model adaptability and robustness to scale variations while maintaining image details. Depth-Aware Feature Enhancement Mamba (DMB) Module: It utilizes the fused features from image characteristics as input and employs a novel adaptive strategy to globally integrate depth information and visual information. This depth fusion strategy not only improves the accuracy of depth estimation but also enhances the model performance under different viewing angles and environmental conditions. Moreover, the modular design of MonoMM provides high flexibility and scalability, facilitating adjustments and optimizations according to specific application needs. Extensive experiments conducted on the KITTI dataset show that our method outperforms previous monocular methods and achieves real-time detection.

Title: An Experimental Evaluation of TEE technology Evolution: Benchmarking Transparent Approaches based on SGX, SEV, and TDX

• Authors: Luigi Coppolino, Salvatore D'Antonio, Davide Iasio, Giovanni Mazzeo, Luigi Romano
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.00443
• Pdf URL: https://arxiv.org/pdf/2408.00443
• Copy Paste: [[2408.00443]] An Experimental Evaluation of TEE technology Evolution: Benchmarking Transparent Approaches based on SGX, SEV, and TDX(https://arxiv.org/abs/2408.00443)
• Keywords: protect
• Abstract: Protection of data-in-use is a key priority, for which Trusted Execution Environment (TEE) technology has unarguably emerged as a, possibly the most, promising solution. Multiple server-side TEE offerings have been released over the years, exhibiting substantial differences with respect to several aspects. The first comer was Intel SGX, which featured Process-based TEE protection, an efficient yet difficult to use approach. Some SGX limitations were (partially) overcome by runtimes, notably: Gramine, Scone, and Occlum. A major paradigm shift was later brought by AMD SEV, with VM-based TEE protection, which enabled lift-and-shift deployment of legacy applications. This new paradigm has been implemented by Intel only recently, in TDX. While the threat model of the aforementioned TEE solutions has been widely discussed, a thorough performance comparison is still lacking in the literature. This paper provides a comparative evaluation of TDX, SEV, Gramine-SGX, and Occlum-SGX. We study computational overhead and resource usage, under different operational scenarios and using a diverse suite of legacy applications. By doing so, we provide a reliable performance assessment under realistic conditions. We explicitly emphasize that, at the time of writing, TDX was not yet available to the public. Thus, the evaluation of TDX is a unique feature of this study.

Title: Multi-label Sewer Pipe Defect Recognition with Mask Attention Feature Enhancement and Label Correlation Learning

• Authors: Xin Zuo, Yu Sheng, Jifeng Shen, Yongwei Shan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.00489
• Pdf URL: https://arxiv.org/pdf/2408.00489
• Copy Paste: [[2408.00489]] Multi-label Sewer Pipe Defect Recognition with Mask Attention Feature Enhancement and Label Correlation Learning(https://arxiv.org/abs/2408.00489)
• Keywords: interpretability
• Abstract: The coexistence of multiple defect categories as well as the substantial class imbalance problem significantly impair the detection of sewer pipeline defects. To solve this problem, a multi-label pipe defect recognition method is proposed based on mask attention guided feature enhancement and label correlation learning. The proposed method can achieve current approximate state-of-the-art classification performance using just 1/16 of the Sewer-ML training dataset and exceeds the current best method by 11.87\% in terms of F2 metric on the full dataset, while also proving the superiority of the model. The major contribution of this study is the development of a more efficient model for identifying and locating multiple defects in sewer pipe images for a more accurate sewer pipeline condition assessment. Moreover, by employing class activation maps, our method can accurately pinpoint multiple defect categories in the image which demonstrates a strong model interpretability. Our code is available at \href{this https URL}{\textcolor{black}{this https URL.}

Title: Graph Representation Learning via Causal Diffusion for Out-of-Distribution Recommendation

• Authors: Chu Zhao, Enneng Yang, Yuliang Liang, Pengxiang Lan, Yuting Liu, Jianzhe Zhao, Guibing Guo, Xingwei Wang
• Subjects: cs.LG, cs.AI, cs.IR, cs.SI
• Abstract URL: https://arxiv.org/abs/2408.00490
• Pdf URL: https://arxiv.org/pdf/2408.00490
• Copy Paste: [[2408.00490]] Graph Representation Learning via Causal Diffusion for Out-of-Distribution Recommendation(https://arxiv.org/abs/2408.00490)
• Keywords: diffusion
• Abstract: Graph Neural Networks (GNNs)-based recommendation algorithms typically assume that training and testing data are drawn from independent and identically distributed (IID) spaces. However, this assumption often fails in the presence of out-of-distribution (OOD) data, resulting in significant performance degradation. In this study, we construct a Structural Causal Model (SCM) to analyze interaction data, revealing that environmental confounders (e.g., the COVID-19 pandemic) lead to unstable correlations in GNN-based models, thus impairing their generalization to OOD data. To address this issue, we propose a novel approach, graph representation learning via causal diffusion (CausalDiffRec) for OOD recommendation. This method enhances the model's generalization on OOD data by eliminating environmental confounding factors and learning invariant graph representations. Specifically, we use backdoor adjustment and variational inference to infer the real environmental distribution, thereby eliminating the impact of environmental confounders. This inferred distribution is then used as prior knowledge to guide the representation learning in the reverse phase of the diffusion process to learn the invariant representation. In addition, we provide a theoretical derivation that proves optimizing the objective function of CausalDiffRec can encourage the model to learn environment-invariant graph representations, thereby achieving excellent generalization performance in recommendations under distribution shifts. Our extensive experiments validate the effectiveness of CausalDiffRec in improving the generalization of OOD data, and the average improvement is up to 10.69% on Food, 18.83% on KuaiRec, 22.41% on Yelp2018, and 11.65% on Douban datasets.

Title: SegStitch: Multidimensional Transformer for Robust and Efficient Medical Imaging Segmentation

• Authors: Shengbo Tan, Zeyu Zhang, Ying Cai, Daji Ergu, Lin Wu, Binbin Hu, Pengzhang Yu, Yang Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.00496
• Pdf URL: https://arxiv.org/pdf/2408.00496
• Copy Paste: [[2408.00496]] SegStitch: Multidimensional Transformer for Robust and Efficient Medical Imaging Segmentation(https://arxiv.org/abs/2408.00496)
• Keywords: robust, transformer, segmentation
• Abstract: Medical imaging segmentation plays a significant role in the automatic recognition and analysis of lesions. State-of-the-art methods, particularly those utilizing transformers, have been prominently adopted in 3D semantic segmentation due to their superior performance in scalability and generalizability. However, plain vision transformers encounter challenges due to their neglect of local features and their high computational complexity. To address these challenges, we introduce three key contributions: Firstly, we proposed SegStitch, an innovative architecture that integrates transformers with denoising ODE blocks. Instead of taking whole 3D volumes as inputs, we adapt axial patches and customize patch-wise queries to ensure semantic consistency. Additionally, we conducted extensive experiments on the BTCV and ACDC datasets, achieving improvements up to 11.48% and 6.71% respectively in mDSC, compared to state-of-the-art methods. Lastly, our proposed method demonstrates outstanding efficiency, reducing the number of parameters by 36.7% and the number of FLOPS by 10.7% compared to UNETR. This advancement holds promising potential for adapting our method to real-world clinical practice. The code will be available at this https URL

Title: How Effective are Self-Supervised Models for Contact Identification in Videos

• Authors: Malitha Gunawardhana, Limalka Sadith, Liel David, Daniel Harari, Muhammad Haris Khan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.00498
• Pdf URL: https://arxiv.org/pdf/2408.00498
• Copy Paste: [[2408.00498]] How Effective are Self-Supervised Models for Contact Identification in Videos(https://arxiv.org/abs/2408.00498)
• Keywords: robust
• Abstract: The exploration of video content via Self-Supervised Learning (SSL) models has unveiled a dynamic field of study, emphasizing both the complex challenges and unique opportunities inherent in this area. Despite the growing body of research, the ability of SSL models to detect physical contacts in videos remains largely unexplored, particularly the effectiveness of methods such as downstream supervision with linear probing or full fine-tuning. This work aims to bridge this gap by employing eight different convolutional neural networks (CNNs) based video SSL models to identify instances of physical contact within video sequences specifically. The Something-Something v2 (SSv2) and Epic-Kitchen (EK-100) datasets were chosen for evaluating these approaches due to the promising results on UCF101 and HMDB51, coupled with their limited prior assessment on SSv2 and EK-100. Additionally, these datasets feature diverse environments and scenarios, essential for testing the robustness and accuracy of video-based models. This approach not only examines the effectiveness of each model in recognizing physical contacts but also explores the performance in the action recognition downstream task. By doing so, valuable insights into the adaptability of SSL models in interpreting complex, dynamic visual information are contributed.

Title: To Change Or To Stick: Unveiling The Consistency Of Cyber Criminal Signatures Through Statistical Analysis

• Authors: Ronan Mouchoux, François Moerman
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.00499
• Pdf URL: https://arxiv.org/pdf/2408.00499
• Copy Paste: [[2408.00499]] To Change Or To Stick: Unveiling The Consistency Of Cyber Criminal Signatures Through Statistical Analysis(https://arxiv.org/abs/2408.00499)
• Keywords: security, attack
• Abstract: This study unveils the elusive presence of criminal signatures in cyberspace, validating for the first time their existence through statistical evidence. By applying the A priori algorithm to the modus operandi of Advanced Persistent Threats, extracted from an extensive corpus of over 17,000 articles spanning 2007 to 2020, we highlight the enduring patterns leveraged by sophisticated cyber criminals. Our findings verify the existence of unique signatures associated with advanced cybercriminals, bridging a crucial gap in current understanding of human behavior in cyber-attacks. This pivotal research sets the foundation for an entirely new academic intersection in cybersecurity and computational criminology.

Title: If It Looks Like a Rootkit and Deceives Like a Rootkit: A Critical Examination of Kernel-Level Anti-Cheat Systems

• Authors: Christoph Dorner, Lukas Daniel Klausner
• Subjects: cs.CR, cs.CY
• Abstract URL: https://arxiv.org/abs/2408.00500
• Pdf URL: https://arxiv.org/pdf/2408.00500
• Copy Paste: [[2408.00500]] If It Looks Like a Rootkit and Deceives Like a Rootkit: A Critical Examination of Kernel-Level Anti-Cheat Systems(https://arxiv.org/abs/2408.00500)
• Keywords: security, privacy, protect
• Abstract: Addressing a critical aspect of cybersecurity in online gaming, this paper systematically evaluates the extent to which kernel-level anti-cheat systems mirror the properties of rootkits, highlighting the importance of distinguishing between protective and potentially invasive software. After establishing a definition for rootkits (making distinctions between rootkits and simple kernel-level applications) and defining metrics to evaluate such software, we introduce four widespread kernel-level anti-cheat solutions. We lay out the inner workings of these types of software, assess them according to our previously established definitions, and discuss ethical considerations and the possible privacy infringements introduced by such programs. Our analysis shows two of the four anti-cheat solutions exhibiting rootkit-like behaviour, threatening the privacy and the integrity of the system. This paper thus provides crucial insights for researchers and developers in the field of gaming security and software engineering, highlighting the need for informed development practices that carefully consider the intersection of effective anti-cheat mechanisms and user privacy.

Title: Hacked in Translation -- from Subtitles to Complete Takeover

• Authors: Omri Herscovici, Omer Gull
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.00502
• Pdf URL: https://arxiv.org/pdf/2408.00502
• Copy Paste: [[2408.00502]] Hacked in Translation -- from Subtitles to Complete Takeover(https://arxiv.org/abs/2408.00502)
• Keywords: security, attack
• Abstract: Check Point researchers revealed a new attack vector which threatens millions of users worldwide - attack by subtitles. By crafting malicious subtitle files, which are then downloaded by a victim's media player, attackers can take complete control over any type of device via vulnerabilities found in many popular streaming platforms, including VLC, Kodi (XBMC), Popcorn-Time and this http URL. We estimate there are approximately 200 million video players and streamers that currently run the vulnerable software, making this one of the most widespread, easily accessed and zero-resistance vulnerability reported in recent years. Our research reveals a new possible attack vector, using a completely overlooked technique in which the cyberattack is delivered when movie subtitles are automatically loaded from online repositories by the user's media player. These subtitles repositories are, in practice, treated as a trusted source by the user or media player; our research also reveals that those repositories can be manipulated and be made to award the attacker's malicious subtitles a high score, which results in those specific subtitles being served to the user. This method requires little or no deliberate action on the part of the user, making it all the more dangerous. Unlike traditional attack vectors, which security firms and users are widely aware of, movie subtitles are perceived as nothing more than benign text files. This means users, Anti-Virus software, and other security solutions vet them without trying to assess their real nature, leaving millions of users exposed to this risk.

Title: Block-Operations: Using Modular Routing to Improve Compositional Generalization

• Authors: Florian Dietz, Dietrich Klakow
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.00508
• Pdf URL: https://arxiv.org/pdf/2408.00508
• Copy Paste: [[2408.00508]] Block-Operations: Using Modular Routing to Improve Compositional Generalization(https://arxiv.org/abs/2408.00508)
• Keywords: transformer
• Abstract: We explore the hypothesis that poor compositional generalization in neural networks is caused by difficulties with learning effective routing. To solve this problem, we propose the concept of block-operations, which is based on splitting all activation tensors in the network into uniformly sized blocks and using an inductive bias to encourage modular routing and modification of these blocks. Based on this concept we introduce the Multiplexer, a new architectural component that enhances the Feed Forward Neural Network (FNN). We experimentally confirm that Multiplexers exhibit strong compositional generalization. On both a synthetic and a realistic task our model was able to learn the underlying process behind the task, whereas both FNNs and Transformers were only able to learn heuristic approximations. We propose as future work to use the principles of block-operations to improve other existing architectures.

Title: VecAug: Unveiling Camouflaged Frauds with Cohort Augmentation for Enhanced Detection

• Authors: Fei Xiao, Shaofeng Cai, Gang Chen, H. V. Jagadish, Beng Chin Ooi, Meihui Zhang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.00513
• Pdf URL: https://arxiv.org/pdf/2408.00513
• Copy Paste: [[2408.00513]] VecAug: Unveiling Camouflaged Frauds with Cohort Augmentation for Enhanced Detection(https://arxiv.org/abs/2408.00513)
• Keywords: robust
• Abstract: Fraud detection presents a challenging task characterized by ever-evolving fraud patterns and scarce labeled data. Existing methods predominantly rely on graph-based or sequence-based approaches. While graph-based approaches connect users through shared entities to capture structural information, they remain vulnerable to fraudsters who can disrupt or manipulate these connections. In contrast, sequence-based approaches analyze users' behavioral patterns, offering robustness against tampering but overlooking the interactions between similar users. Inspired by cohort analysis in retention and healthcare, this paper introduces VecAug, a novel cohort-augmented learning framework that addresses these challenges by enhancing the representation learning of target users with personalized cohort information. To this end, we first propose a vector burn-in technique for automatic cohort identification, which retrieves a task-specific cohort for each target user. Then, to fully exploit the cohort information, we introduce an attentive cohort aggregation technique for augmenting target user representations. To improve the robustness of such cohort augmentation, we also propose a novel label-aware cohort neighbor separation mechanism to distance negative cohort neighbors and calibrate the aggregated cohort information. By integrating this cohort information with target user representations, VecAug enhances the modeling capacity and generalization capabilities of the model to be augmented. Our framework is flexible and can be seamlessly integrated with existing fraud detection models. We deploy our framework on e-commerce platforms and evaluate it on three fraud detection datasets, and results show that VecAug improves the detection performance of base models by up to 2.48\% in AUC and 22.5\% in R@P$_{0.9}$, outperforming state-of-the-art methods significantly.

Title: Jailbreaking Text-to-Image Models with LLM-Based Agents

• Authors: Yingkai Dong, Zheng Li, Xiangtao Meng, Ning Yu, Shanqing Guo
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.00523
• Pdf URL: https://arxiv.org/pdf/2408.00523
• Copy Paste: [[2408.00523]] Jailbreaking Text-to-Image Models with LLM-Based Agents(https://arxiv.org/abs/2408.00523)
• Keywords: attack, generative, large language model
• Abstract: Recent advancements have significantly improved automated task-solving capabilities using autonomous agents powered by large language models (LLMs). However, most LLM-based agents focus on dialogue, programming, or specialized domains, leaving gaps in addressing generative AI safety tasks. These gaps are primarily due to the challenges posed by LLM hallucinations and the lack of clear guidelines. In this paper, we propose Atlas, an advanced LLM-based multi-agent framework that integrates an efficient fuzzing workflow to target generative AI models, specifically focusing on jailbreak attacks against text-to-image (T2I) models with safety filters. Atlas utilizes a vision-language model (VLM) to assess whether a prompt triggers the T2I model's safety filter. It then iteratively collaborates with both LLM and VLM to generate an alternative prompt that bypasses the filter. Atlas also enhances the reasoning abilities of LLMs in attack scenarios by leveraging multi-agent communication, in-context learning (ICL) memory mechanisms, and the chain-of-thought (COT) approach. Our evaluation demonstrates that Atlas successfully jailbreaks several state-of-the-art T2I models in a black-box setting, which are equipped with multi-modal safety filters. In addition, Atlas outperforms existing methods in both query efficiency and the quality of the generated images.

Title: Contrastive Learning with Dynamic Localized Repulsion for Brain Age Prediction on 3D Stiffness Maps

• Authors: Jakob Träuble, Lucy Hiscox, Curtis Johnson, Carola-Bibiane Schönlieb, Gabriele Kaminski Schierle, Angelica Aviles-Rivero
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.00527
• Pdf URL: https://arxiv.org/pdf/2408.00527
• Copy Paste: [[2408.00527]] Contrastive Learning with Dynamic Localized Repulsion for Brain Age Prediction on 3D Stiffness Maps(https://arxiv.org/abs/2408.00527)
• Keywords: robust, generative
• Abstract: In the field of neuroimaging, accurate brain age prediction is pivotal for uncovering the complexities of brain aging and pinpointing early indicators of neurodegenerative conditions. Recent advancements in self-supervised learning, particularly in contrastive learning, have demonstrated greater robustness when dealing with complex datasets. However, current approaches often fall short in generalizing across non-uniformly distributed data, prevalent in medical imaging scenarios. To bridge this gap, we introduce a novel contrastive loss that adapts dynamically during the training process, focusing on the localized neighborhoods of samples. Moreover, we expand beyond traditional structural features by incorporating brain stiffness, a mechanical property previously underexplored yet promising due to its sensitivity to age-related changes. This work presents the first application of self-supervised learning to brain mechanical properties, using compiled stiffness maps from various clinical studies to predict brain age. Our approach, featuring dynamic localized loss, consistently outperforms existing state-of-the-art methods, demonstrating superior performance and laying the way for new directions in brain aging research.

Title: Intermittent Semi-working Mask: A New Masking Paradigm for LLMs

• Authors: Mingcong Lu, Jiangcai Zhu, Wang Hao, Zheng Li, Shusheng Zhang, Kailai Shao, Chao Chen, Nan Li, Feng Wang, Xin Lu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.00539
• Pdf URL: https://arxiv.org/pdf/2408.00539
• Copy Paste: [[2408.00539]] Intermittent Semi-working Mask: A New Masking Paradigm for LLMs(https://arxiv.org/abs/2408.00539)
• Keywords: large language model
• Abstract: Multi-turn dialogues are a key interaction method between humans and Large Language Models (LLMs), as conversations extend over multiple rounds, keeping LLMs' high generation quality and low latency is a challenge. Mainstream LLMs can be grouped into two categories based on masking strategy: causal LLM and prefix LLM. Several works have demonstrated that prefix LLMs tend to outperform causal ones in scenarios that heavily depend on historical context such as multi-turn dialogues or in-context learning, thanks to their bidirectional attention on prefix sequences. However, prefix LLMs have an inherent inefficient training problem in multi-turn dialogue datasets. In addition, the attention mechanism of prefix LLM makes it unable to reuse Key-Value Cache (KV Cache) across dialogue rounds to reduce generation latency. In this paper, we propose a novel masking scheme called Intermittent Semi-working Mask (ISM) to address these problems. Specifically, we apply alternate bidirectional and unidirectional attention on queries and answers in the dialogue history. In this way, ISM is able to maintain the high quality of prefix LLM and low generation latency of causal LLM, simultaneously. Extensive experiments illustrate that our ISM achieves significant performance.

Title: Alleviating Hallucination in Large Vision-Language Models with Active Retrieval Augmentation

• Authors: Xiaoye Qu, Qiyuan Chen, Wei Wei, Jishuo Sun, Jianfeng Dong
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2408.00555
• Pdf URL: https://arxiv.org/pdf/2408.00555
• Copy Paste: [[2408.00555]] Alleviating Hallucination in Large Vision-Language Models with Active Retrieval Augmentation(https://arxiv.org/abs/2408.00555)
• Keywords: large language model
• Abstract: Despite the remarkable ability of large vision-language models (LVLMs) in image comprehension, these models frequently generate plausible yet factually incorrect responses, a phenomenon known as hallucination.Recently, in large language models (LLMs), augmenting LLMs by retrieving information from external knowledge resources has been proven as a promising solution to mitigate hallucinations.However, the retrieval augmentation in LVLM significantly lags behind the widespread applications of LVLM. Moreover, when transferred to augmenting LVLMs, sometimes the hallucination degree of the model is even exacerbated.Motivated by the research gap and counter-intuitive phenomenon, we introduce a novel framework, the Active Retrieval-Augmented large vision-language model (ARA), specifically designed to address hallucinations by incorporating three critical dimensions: (i) dissecting the retrieval targets based on the inherent hierarchical structures of images. (ii) pinpointing the most effective retrieval methods and filtering out the reliable retrieval results. (iii) timing the retrieval process to coincide with episodes of low certainty, while circumventing unnecessary retrieval during periods of high certainty. To assess the capability of our proposed ARA model in reducing hallucination, we employ three widely used LVLM models (LLaVA-1.5, Qwen-VL, and mPLUG-Owl2) across four benchmarks. Our empirical observations suggest that by utilizing fitting retrieval mechanisms and timing the retrieval judiciously, we can effectively mitigate the hallucination problem. We hope that this study can provide deeper insights into how to adapt the retrieval augmentation to LVLMs for reducing hallucinations with more effective retrieval and minimal retrieval occurrences.

Title: MUFASA: Multi-View Fusion and Adaptation Network with Spatial Awareness for Radar Object Detection

• Authors: Xiangyuan Peng, Miao Tang, Huawei Sun, Kay Bierzynski, Lorenzo Servadei, Robert Wille
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.00565
• Pdf URL: https://arxiv.org/pdf/2408.00565
• Copy Paste: [[2408.00565]] MUFASA: Multi-View Fusion and Adaptation Network with Spatial Awareness for Radar Object Detection(https://arxiv.org/abs/2408.00565)
• Keywords: robust, extraction
• Abstract: In recent years, approaches based on radar object detection have made significant progress in autonomous driving systems due to their robustness under adverse weather compared to LiDAR. However, the sparsity of radar point clouds poses challenges in achieving precise object detection, highlighting the importance of effective and comprehensive feature extraction technologies. To address this challenge, this paper introduces a comprehensive feature extraction method for radar point clouds. This study first enhances the capability of detection networks by using a plug-and-play module, GeoSPA. It leverages the Lalonde features to explore local geometric patterns. Additionally, a distributed multi-view attention mechanism, DEMVA, is designed to integrate the shared information across the entire dataset with the global information of each individual frame. By employing the two modules, we present our method, MUFASA, which enhances object detection performance through improved feature extraction. The approach is evaluated on the VoD and TJ4DRaDSet datasets to demonstrate its effectiveness. In particular, we achieve state-of-the-art results among radar-based methods on the VoD dataset with the mAP of 50.24%.

Title: Non Verbis, Sed Rebus: Large Language Models are Weak Solvers of Italian Rebuses

• Authors: Gabriele Sarti, Tommaso Caselli, Malvina Nissim, Arianna Bisazza
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.00584
• Pdf URL: https://arxiv.org/pdf/2408.00584
• Copy Paste: [[2408.00584]] Non Verbis, Sed Rebus: Large Language Models are Weak Solvers of Italian Rebuses(https://arxiv.org/abs/2408.00584)
• Keywords: large language model
• Abstract: Rebuses are puzzles requiring constrained multi-step reasoning to identify a hidden phrase from a set of images and letters. In this work, we introduce a large collection of verbalized rebuses for the Italian language and use it to assess the rebus-solving capabilities of state-of-the-art large language models. While general-purpose systems such as LLaMA-3 and GPT-4o perform poorly on this task, ad-hoc fine-tuning seems to improve models' performance. However, we find that performance gains from training are largely motivated by memorization. Our results suggest that rebus solving remains a challenging test bed to evaluate large language models' linguistic proficiency and sequential instruction-following skills.

Title: Closing the gap between open-source and commercial large language models for medical evidence summarization

• Authors: Gongbo Zhang, Qiao Jin, Yiliang Zhou, Song Wang, Betina R. Idnay, Yiming Luo, Elizabeth Park, Jordan G. Nestor, Matthew E. Spotnitz, Ali Soroush, Thomas Campion, Zhiyong Lu, Chunhua Weng, Yifan Peng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.00588
• Pdf URL: https://arxiv.org/pdf/2408.00588
• Copy Paste: [[2408.00588]] Closing the gap between open-source and commercial large language models for medical evidence summarization(https://arxiv.org/abs/2408.00588)
• Keywords: large language model
• Abstract: Large language models (LLMs) hold great promise in summarizing medical evidence. Most recent studies focus on the application of proprietary LLMs. Using proprietary LLMs introduces multiple risk factors, including a lack of transparency and vendor dependency. While open-source LLMs allow better transparency and customization, their performance falls short compared to proprietary ones. In this study, we investigated to what extent fine-tuning open-source LLMs can further improve their performance in summarizing medical evidence. Utilizing a benchmark dataset, MedReview, consisting of 8,161 pairs of systematic reviews and summaries, we fine-tuned three broadly-used, open-sourced LLMs, namely PRIMERA, LongT5, and Llama-2. Overall, the fine-tuned LLMs obtained an increase of 9.89 in ROUGE-L (95% confidence interval: 8.94-10.81), 13.21 in METEOR score (95% confidence interval: 12.05-14.37), and 15.82 in CHRF score (95% confidence interval: 13.89-16.44). The performance of fine-tuned LongT5 is close to GPT-3.5 with zero-shot settings. Furthermore, smaller fine-tuned models sometimes even demonstrated superior performance compared to larger zero-shot models. The above trends of improvement were also manifested in both human and GPT4-simulated evaluations. Our results can be applied to guide model selection for tasks demanding particular domain knowledge, such as medical evidence summarization.

Title: Downstream bias mitigation is all you need

• Authors: Arkadeep Baksi, Rahul Singh, Tarun Joshi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.00612
• Pdf URL: https://arxiv.org/pdf/2408.00612
• Copy Paste: [[2408.00612]] Downstream bias mitigation is all you need(https://arxiv.org/abs/2408.00612)
• Keywords: transformer, large language model
• Abstract: The advent of transformer-based architectures and large language models (LLMs) have significantly advanced the performance of natural language processing (NLP) models. Since these LLMs are trained on huge corpuses of data from the web and other sources, there has been a major concern about harmful prejudices that may potentially be transferred from the data. In many applications, these pre-trained LLMs are fine-tuned on task specific datasets, which can further contribute to biases. This paper studies the extent of biases absorbed by LLMs during pre-training as well as task-specific behaviour after fine-tuning. We found that controlled interventions on pre-trained LLMs, prior to fine-tuning, have minimal effect on lowering biases in classifiers. However, the biases present in domain-specific datasets play a much bigger role, and hence mitigating them at this stage has a bigger impact. While pre-training does matter, but after the model has been pre-trained, even slight changes to co-occurrence rates in the fine-tuning dataset has a significant effect on the bias of the model.

Title: Harnessing Uncertainty-aware Bounding Boxes for Unsupervised 3D Object Detection

• Authors: Ruiyang Zhang, Hu Zhang, Hang Yu, Zhedong Zheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.00619
• Pdf URL: https://arxiv.org/pdf/2408.00619
• Copy Paste: [[2408.00619]] Harnessing Uncertainty-aware Bounding Boxes for Unsupervised 3D Object Detection(https://arxiv.org/abs/2408.00619)
• Keywords: robust
• Abstract: Unsupervised 3D object detection aims to identify objects of interest from unlabeled raw data, such as LiDAR points. Recent approaches usually adopt pseudo 3D bounding boxes (3D bboxes) from clustering algorithm to initialize the model training, and then iteratively updating both pseudo labels and the trained model. However, pseudo bboxes inevitably contain noises, and such inaccurate annotation accumulates to the final model, compromising the performance. Therefore, in an attempt to mitigate the negative impact of pseudo bboxes, we introduce a new uncertainty-aware framework. In particular, Our method consists of two primary components: uncertainty estimation and uncertainty regularization. (1) In the uncertainty estimation phase, we incorporate an extra auxiliary detection branch alongside the primary detector. The prediction disparity between the primary and auxiliary detectors is leveraged to estimate uncertainty at the box coordinate level, including position, shape, orientation. (2) Based on the assessed uncertainty, we regularize the model training via adaptively adjusting every 3D bboxes coordinates. For pseudo bbox coordinates with high uncertainty, we assign a relatively low loss weight. Experiment verifies that the proposed method is robust against the noisy pseudo bboxes, yielding substantial improvements on nuScenes and Lyft compared to existing techniques, with increases of 6.9% in AP$_{BEV}$ and 2.5% in AP$_{3D}$ on nuScenes, and 2.2% in AP$_{BEV}$ and 1.0% in AP$_{3D}$ on Lyft.

Title: Are Bigger Encoders Always Better in Vision Large Models?

• Authors: Bozhou Li, Hao Liang, Zimo Meng, Wentao Zhang
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2408.00620
• Pdf URL: https://arxiv.org/pdf/2408.00620
• Copy Paste: [[2408.00620]] Are Bigger Encoders Always Better in Vision Large Models?(https://arxiv.org/abs/2408.00620)
• Keywords: large language model
• Abstract: In recent years, multimodal large language models (MLLMs) have shown strong potential in real-world applications. They are developing rapidly due to their remarkable ability to comprehend multimodal information and their inherent powerful cognitive and reasoning capabilities. Among MLLMs, vision language models (VLM) stand out for their ability to understand vision information. However, the scaling trend of VLMs under the current mainstream paradigm has not been extensively studied. Whether we can achieve better performance by training even larger models is still unclear. To address this issue, we conducted experiments on the pretraining stage of MLLMs. We conduct our experiment using different encoder sizes and large language model (LLM) sizes. Our findings indicate that merely increasing the size of encoders does not necessarily enhance the performance of VLMs. Moreover, we analyzed the effects of LLM backbone parameter size and data quality on the pretraining outcomes. Additionally, we explored the differences in scaling laws between LLMs and VLMs.

Title: Empowering Snapshot Compressive Imaging: Spatial-Spectral State Space Model with Across-Scanning and Local Enhancement

• Authors: Wenzhe Tian, Haijin Zeng, Yin-Ping Zhao, Yongyong Chen, Zhen Wang, Xuelong Li
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2408.00629
• Pdf URL: https://arxiv.org/pdf/2408.00629
• Copy Paste: [[2408.00629]] Empowering Snapshot Compressive Imaging: Spatial-Spectral State Space Model with Across-Scanning and Local Enhancement(https://arxiv.org/abs/2408.00629)
• Keywords: transformer
• Abstract: Snapshot Compressive Imaging (SCI) relies on decoding algorithms such as CNN or Transformer to reconstruct the hyperspectral image (HSI) from its compressed measurement. Although existing CNN and Transformer-based methods have proven effective, CNNs are limited by their inadequate modeling of long-range dependencies, while Transformer ones face high computational costs due to quadratic complexity. Recent Mamba models have demonstrated superior performance over CNN and Transformer-based architectures in some visual tasks, but these models have not fully utilized the local similarities in both spatial and spectral dimensions. Moreover, the long-sequence modeling capability of SSM may offer an advantage in processing the numerous spectral bands for HSI reconstruction, which has not yet been explored. In this paper, we introduce a State Space Model with Across-Scanning and Local Enhancement, named ASLE-SSM, that employs a Spatial-Spectral SSM for global-local balanced context encoding and cross-channel interaction promoting. Specifically, we introduce local scanning in the spatial dimension to balance the global and local receptive fields, and then propose our across-scanning method based on spatial-spectral local cubes to leverage local similarities between adjacent spectral bands and pixels to guide the reconstruction process. These two scanning mechanisms extract the HSI's local features while balancing the global perspective without any additional costs. Experimental results illustrate ASLE-SSM's superiority over existing state-of-the-art methods, with an inference speed 2.4 times faster than Transformer-based MST and saving 0.12 (M) of parameters, achieving the lowest computational cost and parameter count.

Title: Privacy-preserving datasets by capturing feature distributions with Conditional VAEs

• Authors: Francesco Di Salvo, David Tafler, Sebastian Doerrich, Christian Ledig
• Subjects: cs.LG, cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2408.00639
• Pdf URL: https://arxiv.org/pdf/2408.00639
• Copy Paste: [[2408.00639]] Privacy-preserving datasets by capturing feature distributions with Conditional VAEs(https://arxiv.org/abs/2408.00639)
• Keywords: privacy, robust, generative
• Abstract: Large and well-annotated datasets are essential for advancing deep learning applications, however often costly or impossible to obtain by a single entity. In many areas, including the medical domain, approaches relying on data sharing have become critical to address those challenges. While effective in increasing dataset size and diversity, data sharing raises significant privacy concerns. Commonly employed anonymization methods based on the k-anonymity paradigm often fail to preserve data diversity, affecting model robustness. This work introduces a novel approach using Conditional Variational Autoencoders (CVAEs) trained on feature vectors extracted from large pre-trained vision foundation models. Foundation models effectively detect and represent complex patterns across diverse domains, allowing the CVAE to faithfully capture the embedding space of a given data distribution to generate (sample) a diverse, privacy-respecting, and potentially unbounded set of synthetic feature vectors. Our method notably outperforms traditional approaches in both medical and natural image domains, exhibiting greater dataset diversity and higher robustness against perturbations while preserving sample privacy. These results underscore the potential of generative models to significantly impact deep learning applications in data-scarce and privacy-sensitive environments. The source code is available at this https URL .

Title: Enhancing Ethereum Fraud Detection via Generative and Contrastive Self-supervision

• Authors: Chenxiang Jin, Jiajun Zhou, Chenxuan Xie, Shanqing Yu, Qi Xuan, Xiaoniu Yang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.00641
• Pdf URL: https://arxiv.org/pdf/2408.00641
• Copy Paste: [[2408.00641]] Enhancing Ethereum Fraud Detection via Generative and Contrastive Self-supervision(https://arxiv.org/abs/2408.00641)
• Keywords: generative
• Abstract: The rampant fraudulent activities on Ethereum hinder the healthy development of the blockchain ecosystem, necessitating the reinforcement of regulations. However, multiple imbalances involving account interaction frequencies and interaction types in the Ethereum transaction environment pose significant challenges to data mining-based fraud detection research. To address this, we first propose the concept of meta-interactions to refine interaction behaviors in Ethereum, and based on this, we present a dual self-supervision enhanced Ethereum fraud detection framework, named Meta-IFD. This framework initially introduces a generative self-supervision mechanism to augment the interaction features of accounts, followed by a contrastive self-supervision mechanism to differentiate various behavior patterns, and ultimately characterizes the behavioral representations of accounts and mines potential fraud risks through multi-view interaction feature learning. Extensive experiments on real Ethereum datasets demonstrate the effectiveness and superiority of our framework in detecting common Ethereum fraud behaviors such as Ponzi schemes and phishing scams. Additionally, the generative module can effectively alleviate the interaction distribution imbalance in Ethereum data, while the contrastive module significantly enhances the framework's ability to distinguish different behavior patterns. The source code will be released on GitHub soon.

Title: Towards End-to-End Explainable Facial Action Unit Recognition via Vision-Language Joint Learning

• Authors: Xuri Ge, Junchen Fu, Fuhai Chen, Shan An, Nicu Sebe, Joemon M. Jose
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.00644
• Pdf URL: https://arxiv.org/pdf/2408.00644
• Copy Paste: [[2408.00644]] Towards End-to-End Explainable Facial Action Unit Recognition via Vision-Language Joint Learning(https://arxiv.org/abs/2408.00644)
• Keywords: interpretability
• Abstract: Facial action units (AUs), as defined in the Facial Action Coding System (FACS), have received significant research interest owing to their diverse range of applications in facial state analysis. Current mainstream FAU recognition models have a notable limitation, i.e., focusing only on the accuracy of AU recognition and overlooking explanations of corresponding AU states. In this paper, we propose an end-to-end Vision-Language joint learning network for explainable FAU recognition (termed VL-FAU), which aims to reinforce AU representation capability and language interpretability through the integration of joint multimodal tasks. Specifically, VL-FAU brings together language models to generate fine-grained local muscle descriptions and distinguishable global face description when optimising FAU recognition. Through this, the global facial representation and its local AU representations will achieve higher distinguishability among different AUs and different subjects. In addition, multi-level AU representation learning is utilised to improve AU individual attention-aware representation capabilities based on multi-scale combined facial stem feature. Extensive experiments on DISFA and BP4D AU datasets show that the proposed approach achieves superior performance over the state-of-the-art methods on most of the metrics. In addition, compared with mainstream FAU recognition methods, VL-FAU can provide local- and global-level interpretability language descriptions with the AUs' predictions.

Title: Disentangling Dense Embeddings with Sparse Autoencoders

• Authors: Charles O'Neill, Christine Ye, Kartheik Iyer, John F. Wu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.00657
• Pdf URL: https://arxiv.org/pdf/2408.00657
• Copy Paste: [[2408.00657]] Disentangling Dense Embeddings with Sparse Autoencoders(https://arxiv.org/abs/2408.00657)
• Keywords: interpretability, large language model
• Abstract: Sparse autoencoders (SAEs) have shown promise in extracting interpretable features from complex neural networks. We present one of the first applications of SAEs to dense text embeddings from large language models, demonstrating their effectiveness in disentangling semantic concepts. By training SAEs on embeddings of over 420,000 scientific paper abstracts from computer science and astronomy, we show that the resulting sparse representations maintain semantic fidelity while offering interpretability. We analyse these learned features, exploring their behaviour across different model capacities and introducing a novel method for identifying ``feature families'' that represent related concepts at varying levels of abstraction. To demonstrate the practical utility of our approach, we show how these interpretable features can be used to precisely steer semantic search, allowing for fine-grained control over query semantics. This work bridges the gap between the semantic richness of dense embeddings and the interpretability of sparse representations. We open source our embeddings, trained sparse autoencoders, and interpreted features, as well as a web app for exploring them.

Title: AutoM3L: An Automated Multimodal Machine Learning Framework with Large Language Models

• Authors: Daqin Luo, Chengjian Feng, Yuxuan Nong, Yiqing Shen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.00665
• Pdf URL: https://arxiv.org/pdf/2408.00665
• Copy Paste: [[2408.00665]] AutoM3L: An Automated Multimodal Machine Learning Framework with Large Language Models(https://arxiv.org/abs/2408.00665)
• Keywords: large language model
• Abstract: Automated Machine Learning (AutoML) offers a promising approach to streamline the training of machine learning models. However, existing AutoML frameworks are often limited to unimodal scenarios and require extensive manual configuration. Recent advancements in Large Language Models (LLMs) have showcased their exceptional abilities in reasoning, interaction, and code generation, presenting an opportunity to develop a more automated and user-friendly framework. To this end, we introduce AutoM3L, an innovative Automated Multimodal Machine Learning framework that leverages LLMs as controllers to automatically construct multimodal training pipelines. AutoM3L comprehends data modalities and selects appropriate models based on user requirements, providing automation and interactivity. By eliminating the need for manual feature engineering and hyperparameter optimization, our framework simplifies user engagement and enables customization through directives, addressing the limitations of previous rule-based AutoML approaches. We evaluate the performance of AutoM3L on six diverse multimodal datasets spanning classification, regression, and retrieval tasks, as well as a comprehensive set of unimodal datasets. The results demonstrate that AutoM3L achieves competitive or superior performance compared to traditional rule-based AutoML methods. Furthermore, a user study highlights the user-friendliness and usability of our framework, compared to the rule-based AutoML methods.

Title: An effect analysis of the balancing techniques on the counterfactual explanations of student success prediction models

• Authors: Mustafa Cavus, Jakub Kuzilek
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2408.00676
• Pdf URL: https://arxiv.org/pdf/2408.00676
• Copy Paste: [[2408.00676]] An effect analysis of the balancing techniques on the counterfactual explanations of student success prediction models(https://arxiv.org/abs/2408.00676)
• Keywords: robust
• Abstract: In the past decade, we have experienced a massive boom in the usage of digital solutions in higher education. Due to this boom, large amounts of data have enabled advanced data analysis methods to support learners and examine learning processes. One of the dominant research directions in learning analytics is predictive modeling of learners' success using various machine learning methods. To build learners' and teachers' trust in such methods and systems, exploring the methods and methodologies that enable relevant stakeholders to deeply understand the underlying machine-learning models is necessary. In this context, counterfactual explanations from explainable machine learning tools are promising. Several counterfactual generation methods hold much promise, but the features must be actionable and causal to be effective. Thus, obtaining which counterfactual generation method suits the student success prediction models in terms of desiderata, stability, and robustness is essential. Although a few studies have been published in recent years on the use of counterfactual explanations in educational sciences, they have yet to discuss which counterfactual generation method is more suitable for this problem. This paper analyzed the effectiveness of commonly used counterfactual generation methods, such as WhatIf Counterfactual Explanations, Multi-Objective Counterfactual Explanations, and Nearest Instance Counterfactual Explanations after balancing. This contribution presents a case study using the Open University Learning Analytics dataset to demonstrate the practical usefulness of counterfactual explanations. The results illustrate the method's effectiveness and describe concrete steps that could be taken to alter the model's prediction.

Title: Improving Text Embeddings for Smaller Language Models Using Contrastive Fine-tuning

• Authors: Trapoom Ukarapol, Zhicheng Lee, Amy Xin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.00690
• Pdf URL: https://arxiv.org/pdf/2408.00690
• Copy Paste: [[2408.00690]] Improving Text Embeddings for Smaller Language Models Using Contrastive Fine-tuning(https://arxiv.org/abs/2408.00690)
• Keywords: large language model
• Abstract: While Large Language Models show remarkable performance in natural language understanding, their resource-intensive nature makes them less accessible. In contrast, smaller language models such as MiniCPM offer more sustainable scalability, but often underperform without specialized optimization. In this paper, we explore the enhancement of smaller language models through the improvement of their text embeddings. We select three language models, MiniCPM, Phi-2, and Gemma, to conduct contrastive fine-tuning on the NLI dataset. Our results demonstrate that this fine-tuning method enhances the quality of text embeddings for all three models across various benchmarks, with MiniCPM showing the most significant improvements of an average 56.33\% performance gain. The contrastive fine-tuning code is publicly available at this https URL.

Title: Accelerating Full Waveform Inversion By Transfer Learning

• Authors: Divya Shyam Singh, Leon Herrmann, Qing Sun, Tim Bürchner, Felix Dietrich, Stefan Kollmannsberger
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.00695
• Pdf URL: https://arxiv.org/pdf/2408.00695
• Copy Paste: [[2408.00695]] Accelerating Full Waveform Inversion By Transfer Learning(https://arxiv.org/abs/2408.00695)
• Keywords: robust
• Abstract: Full waveform inversion (FWI) is a powerful tool for reconstructing material fields based on sparsely measured data obtained by wave propagation. For specific problems, discretizing the material field with a neural network (NN) improves the robustness and reconstruction quality of the corresponding optimization problem. We call this method NN-based FWI. Starting from an initial guess, the weights of the NN are iteratively updated to fit the simulated wave signals to the sparsely measured data set. For gradient-based optimization, a suitable choice of the initial guess, i.e., a suitable NN weight initialization, is crucial for fast and robust convergence. In this paper, we introduce a novel transfer learning approach to further improve NN-based FWI. This approach leverages supervised pretraining to provide a better NN weight initialization, leading to faster convergence of the subsequent optimization problem. Moreover, the inversions yield physically more meaningful local minima. The network is pretrained to predict the unknown material field using the gradient information from the first iteration of conventional FWI. In our computational experiments on two-dimensional domains, the training data set consists of reference simulations with arbitrarily positioned elliptical voids of different shapes and orientations. We compare the performance of the proposed transfer learning NN-based FWI with three other methods: conventional FWI, NN-based FWI without pretraining and conventional FWI with an initial guess predicted from the pretrained NN. Our results show that transfer learning NN-based FWI outperforms the other methods in terms of convergence speed and reconstruction quality.

Title: Granular-Balls based Fuzzy Twin Support Vector Machine for Classification

• Authors: Lixi Zhao, Weiping Ding, Duoqian Miao, Guangming Lang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.00699
• Pdf URL: https://arxiv.org/pdf/2408.00699
• Copy Paste: [[2408.00699]] Granular-Balls based Fuzzy Twin Support Vector Machine for Classification(https://arxiv.org/abs/2408.00699)
• Keywords: robust
• Abstract: The twin support vector machine (TWSVM) classifier has attracted increasing attention because of its low computational complexity. However, its performance tends to degrade when samples are affected by noise. The granular-ball fuzzy support vector machine (GBFSVM) classifier partly alleviates the adverse effects of noise, but it relies solely on the distance between the granular-ball's center and the class center to design the granular-ball membership function. In this paper, we first introduce the granular-ball twin support vector machine (GBTWSVM) classifier, which integrates granular-ball computing (GBC) with the twin support vector machine (TWSVM) classifier. By replacing traditional point inputs with granular-balls, we demonstrate how to derive a pair of non-parallel hyperplanes for the GBTWSVM classifier by solving a quadratic programming problem. Subsequently, we design the membership and non-membership functions of granular-balls using Pythagorean fuzzy sets to differentiate the contributions of granular-balls in various regions. Additionally, we develop the granular-ball fuzzy twin support vector machine (GBFTSVM) classifier by incorporating GBC with the fuzzy twin support vector machine (FTSVM) classifier. We demonstrate how to derive a pair of non-parallel hyperplanes for the GBFTSVM classifier by solving a quadratic programming problem. We also design algorithms for the GBTSVM classifier and the GBFTSVM classifier. Finally, the superior classification performance of the GBTWSVM classifier and the GBFTSVM classifier on 20 benchmark datasets underscores their scalability, efficiency, and robustness in tackling classification tasks.

Title: You Can't Ignore Either: Unifying Structure and Feature Denoising for Robust Graph Learning

• Authors: Tianmeng Yang, Jiahao Meng, Min Zhou, Yaming Yang, Yujing Wang, Xiangtai Li, Yunhai Tong
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.00700
• Pdf URL: https://arxiv.org/pdf/2408.00700
• Copy Paste: [[2408.00700]] You Can't Ignore Either: Unifying Structure and Feature Denoising for Robust Graph Learning(https://arxiv.org/abs/2408.00700)
• Keywords: attack, robust
• Abstract: Recent research on the robustness of Graph Neural Networks (GNNs) under noises or attacks has attracted great attention due to its importance in real-world applications. Most previous methods explore a single noise source, recovering corrupt node embedding by reliable structures bias or developing structure learning with reliable node features. However, the noises and attacks may come from both structures and features in graphs, making the graph denoising a dilemma and challenging problem. In this paper, we develop a unified graph denoising (UGD) framework to unravel the deadlock between structure and feature denoising. Specifically, a high-order neighborhood proximity evaluation method is proposed to recognize noisy edges, considering features may be perturbed simultaneously. Moreover, we propose to refine noisy features with reconstruction based on a graph auto-encoder. An iterative updating algorithm is further designed to optimize the framework and acquire a clean graph, thus enabling robust graph learning for downstream tasks. Our UGD framework is self-supervised and can be easily implemented as a plug-and-play module. We carry out extensive experiments, which proves the effectiveness and advantages of our method. Code is avalaible at this https URL.

Title: Point-supervised Brain Tumor Segmentation with Box-prompted MedSAM

• Authors: Xiaofeng Liu, Jonghye Woo, Chao Ma, Jinsong Ouyang, Georges El Fakhri
• Subjects: cs.CV, cs.AI, cs.LG, eess.IV, physics.med-ph
• Abstract URL: https://arxiv.org/abs/2408.00706
• Pdf URL: https://arxiv.org/pdf/2408.00706
• Copy Paste: [[2408.00706]] Point-supervised Brain Tumor Segmentation with Box-prompted MedSAM(https://arxiv.org/abs/2408.00706)
• Keywords: segmentation
• Abstract: Delineating lesions and anatomical structure is important for image-guided interventions. Point-supervised medical image segmentation (PSS) has great potential to alleviate costly expert delineation labeling. However, due to the lack of precise size and boundary guidance, the effectiveness of PSS often falls short of expectations. Although recent vision foundational models, such as the medical segment anything model (MedSAM), have made significant advancements in bounding-box-prompted segmentation, it is not straightforward to utilize point annotation, and is prone to semantic ambiguity. In this preliminary study, we introduce an iterative framework to facilitate semantic-aware point-supervised MedSAM. Specifically, the semantic box-prompt generator (SBPG) module has the capacity to convert the point input into potential pseudo bounding box suggestions, which are explicitly refined by the prototype-based semantic similarity. This is then succeeded by a prompt-guided spatial refinement (PGSR) module that harnesses the exceptional generalizability of MedSAM to infer the segmentation mask, which also updates the box proposal seed in SBPG. Performance can be progressively improved with adequate iterations. We conducted an evaluation on BraTS2018 for the segmentation of whole brain tumors and demonstrated its superior performance compared to traditional PSS methods and on par with box-supervised methods.

Title: Synthetic dual image generation for reduction of labeling efforts in semantic segmentation of micrographs with a customized metric function

• Authors: Matias Oscar Volman Stern, Dominic Hohs, Andreas Jansche, Timo Bernthaler, Gerhard Schneider
• Subjects: cs.CV, cs.CE, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.00707
• Pdf URL: https://arxiv.org/pdf/2408.00707
• Copy Paste: [[2408.00707]] Synthetic dual image generation for reduction of labeling efforts in semantic segmentation of micrographs with a customized metric function(https://arxiv.org/abs/2408.00707)
• Keywords: robust, generative, segmentation
• Abstract: Training of semantic segmentation models for material analysis requires micrographs and their corresponding masks. It is quite unlikely that perfect masks will be drawn, especially at the edges of objects, and sometimes the amount of data that can be obtained is small, since only a few samples are available. These aspects make it very problematic to train a robust model. We demonstrate a workflow for the improvement of semantic segmentation models of micrographs through the generation of synthetic microstructural images in conjunction with masks. The workflow only requires joining a few micrographs with their respective masks to create the input for a Vector Quantised-Variational AutoEncoder model that includes an embedding space, which is trained such that a generative model (PixelCNN) learns the distribution of each input, transformed into discrete codes, and can be used to sample new codes. The latter will eventually be decoded by VQ-VAE to generate images alongside corresponding masks for semantic segmentation. To evaluate the synthetic data, we have trained U-Net models with different amounts of these synthetic data in conjunction with real data. These models were then evaluated using non-synthetic images only. Additionally, we introduce a customized metric derived from the mean Intersection over Union (mIoU). The proposed metric prevents a few falsely predicted pixels from greatly reducing the value of the mIoU. We have achieved a reduction in sample preparation and acquisition times, as well as the efforts, needed for image processing and labeling tasks, are less when it comes to training semantic segmentation model. The approach could be generalized to various types of image data such that it serves as a user-friendly solution for training models with a small number of real images.

Title: MotionFix: Text-Driven 3D Human Motion Editing

• Authors: Nikos Athanasiou, Alpár Ceske, Markos Diomataris, Michael J. Black, Gül Varol
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2408.00712
• Pdf URL: https://arxiv.org/pdf/2408.00712
• Copy Paste: [[2408.00712]] MotionFix: Text-Driven 3D Human Motion Editing(https://arxiv.org/abs/2408.00712)
• Keywords: diffusion
• Abstract: The focus of this paper is 3D motion editing. Given a 3D human motion and a textual description of the desired modification, our goal is to generate an edited motion as described by the text. The challenges include the lack of training data and the design of a model that faithfully edits the source motion. In this paper, we address both these challenges. We build a methodology to semi-automatically collect a dataset of triplets in the form of (i) a source motion, (ii) a target motion, and (iii) an edit text, and create the new MotionFix dataset. Having access to such data allows us to train a conditional diffusion model, TMED, that takes both the source motion and the edit text as input. We further build various baselines trained only on text-motion pairs datasets, and show superior performance of our model trained on triplets. We introduce new retrieval-based metrics for motion editing and establish a new benchmark on the evaluation set of MotionFix. Our results are encouraging, paving the way for further research on finegrained motion generation. Code and models will be made publicly available.

Title: SAM 2: Segment Anything in Images and Videos

• Authors: Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross Girshick, Piotr Dollár, Christoph Feichtenhofer
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.00714
• Pdf URL: https://arxiv.org/pdf/2408.00714
• Copy Paste: [[2408.00714]] SAM 2: Segment Anything in Images and Videos(https://arxiv.org/abs/2408.00714)
• Keywords: transformer, segmentation
• Abstract: We present Segment Anything Model 2 (SAM 2), a foundation model towards solving promptable visual segmentation in images and videos. We build a data engine, which improves model and data via user interaction, to collect the largest video segmentation dataset to date. Our model is a simple transformer architecture with streaming memory for real-time video processing. SAM 2 trained on our data provides strong performance across a wide range of tasks. In video segmentation, we observe better accuracy, using 3x fewer interactions than prior approaches. In image segmentation, our model is more accurate and 6x faster than the Segment Anything Model (SAM). We believe that our data, model, and insights will serve as a significant milestone for video segmentation and related perception tasks. We are releasing a version of our model, the dataset and an interactive demo.

Title: A Natural Language Processing Framework for Hotel Recommendation Based on Users' Text Reviews

• Authors: Lavrentia Aravani, Emmanuel Pintelas, Christos Pierrakeas, Panagiotis Pintelas
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.00716
• Pdf URL: https://arxiv.org/pdf/2408.00716
• Copy Paste: [[2408.00716]] A Natural Language Processing Framework for Hotel Recommendation Based on Users' Text Reviews(https://arxiv.org/abs/2408.00716)
• Keywords: transformer
• Abstract: Recently, the application of Artificial Intelligence algorithms in hotel recommendation systems has become an increasingly popular topic. One such method that has proven to be effective in this field is Deep Learning, especially Natural Language processing models, which are able to extract semantic knowledge from user's text reviews to create more efficient recommendation systems. This can lead to the development of intelligent models that can classify a user's preferences and emotions based on their feedback in the form of text reviews about their hotel stay experience. In this study, we propose a Natural Language Processing framework that utilizes customer text reviews to provide personalized recommendations for the most appropriate hotel based on their preferences. The framework is based on Bidirectional Encoder Representations from Transformers (BERT) and a fine-tuning/validation pipeline that categorizes customer hotel review texts into "Bad," "Good," or "Excellent" recommended hotels. Our findings indicate that the hotel recommendation system we propose can significantly enhance the user experience of booking accommodations by providing personalized recommendations based on user preferences and previous booking history.

Title: Pathway to Secure and Trustworthy 6G for LLMs: Attacks, Defense, and Opportunities

• Authors: Sunder Ali Khowaja, Parus Khuwaja, Kapal Dev, Hussam Al Hamadi, Engin Zeydan
• Subjects: cs.CR, cs.AI, cs.DC
• Abstract URL: https://arxiv.org/abs/2408.00722
• Pdf URL: https://arxiv.org/pdf/2408.00722
• Copy Paste: [[2408.00722]] Pathway to Secure and Trustworthy 6G for LLMs: Attacks, Defense, and Opportunities(https://arxiv.org/abs/2408.00722)
• Keywords: secure, security, privacy, defense, attack, membership infer, large language model
• Abstract: Recently, large language models (LLMs) have been gaining a lot of interest due to their adaptability and extensibility in emerging applications, including communication networks. It is anticipated that 6G mobile edge computing networks will be able to support LLMs as a service, as they provide ultra reliable low-latency communications and closed loop massive connectivity. However, LLMs are vulnerable to data and model privacy issues that affect the trustworthiness of LLMs to be deployed for user-based services. In this paper, we explore the security vulnerabilities associated with fine-tuning LLMs in 6G networks, in particular the membership inference attack. We define the characteristics of an attack network that can perform a membership inference attack if the attacker has access to the fine-tuned model for the downstream task. We show that the membership inference attacks are effective for any downstream task, which can lead to a personal data breach when using LLM as a service. The experimental results show that the attack success rate of maximum 92% can be achieved on named entity recognition task. Based on the experimental analysis, we discuss possible defense mechanisms and present possible research directions to make the LLMs more trustworthy in the context of 6G networks.

Title: Improving Retrieval-Augmented Generation in Medicine with Iterative Follow-up Questions

• Authors: Guangzhi Xiong, Qiao Jin, Xiao Wang, Minjia Zhang, Zhiyong Lu, Aidong Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.00727
• Pdf URL: https://arxiv.org/pdf/2408.00727
• Copy Paste: [[2408.00727]] Improving Retrieval-Augmented Generation in Medicine with Iterative Follow-up Questions(https://arxiv.org/abs/2408.00727)
• Keywords: large language model
• Abstract: The emergent abilities of large language models (LLMs) have demonstrated great potential in solving medical questions. They can possess considerable medical knowledge, but may still hallucinate and are inflexible in the knowledge updates. While Retrieval-Augmented Generation (RAG) has been proposed to enhance the medical question-answering capabilities of LLMs with external knowledge bases, it may still fail in complex cases where multiple rounds of information-seeking are required. To address such an issue, we propose iterative RAG for medicine (i-MedRAG), where LLMs can iteratively ask follow-up queries based on previous information-seeking attempts. In each iteration of i-MedRAG, the follow-up queries will be answered by a vanilla RAG system and they will be further used to guide the query generation in the next iteration. Our experiments show the improved performance of various LLMs brought by i-MedRAG compared with vanilla RAG on complex questions from clinical vignettes in the United States Medical Licensing Examination (USMLE), as well as various knowledge tests in the Massive Multitask Language Understanding (MMLU) dataset. Notably, our zero-shot i-MedRAG outperforms all existing prompt engineering and fine-tuning methods on GPT-3.5, achieving an accuracy of 69.68\% on the MedQA dataset. In addition, we characterize the scaling properties of i-MedRAG with different iterations of follow-up queries and different numbers of queries per iteration. Our case studies show that i-MedRAG can flexibly ask follow-up queries to form reasoning chains, providing an in-depth analysis of medical questions. To the best of our knowledge, this is the first-of-its-kind study on incorporating follow-up queries into medical RAG.

Title: CERT-ED: Certifiably Robust Text Classification for Edit Distance

• Authors: Zhuoqun Huang, Neil G Marchant, Olga Ohrimenko, Benjamin I. P. Rubinstein
• Subjects: cs.CL, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.00728
• Pdf URL: https://arxiv.org/pdf/2408.00728
• Copy Paste: [[2408.00728]] CERT-ED: Certifiably Robust Text Classification for Edit Distance(https://arxiv.org/abs/2408.00728)
• Keywords: defense, attack, robust
• Abstract: With the growing integration of AI in daily life, ensuring the robustness of systems to inference-time attacks is crucial. Among the approaches for certifying robustness to such adversarial examples, randomized smoothing has emerged as highly promising due to its nature as a wrapper around arbitrary black-box models. Previous work on randomized smoothing in natural language processing has primarily focused on specific subsets of edit distance operations, such as synonym substitution or word insertion, without exploring the certification of all edit operations. In this paper, we adapt Randomized Deletion (Huang et al., 2023) and propose, CERTified Edit Distance defense (CERT-ED) for natural language classification. Through comprehensive experiments, we demonstrate that CERT-ED outperforms the existing Hamming distance method RanMASK (Zeng et al., 2023) in 4 out of 5 datasets in terms of both accuracy and the cardinality of the certificate. By covering various threat models, including 5 direct and 5 transfer attacks, our method improves empirical robustness in 38 out of 50 settings.

Title: TurboEdit: Text-Based Image Editing Using Few-Step Diffusion Models

• Authors: Gilad Deutch, Rinon Gal, Daniel Garibi, Or Patashnik, Daniel Cohen-Or
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2408.00735
• Pdf URL: https://arxiv.org/pdf/2408.00735
• Copy Paste: [[2408.00735]] TurboEdit: Text-Based Image Editing Using Few-Step Diffusion Models(https://arxiv.org/abs/2408.00735)
• Keywords: diffusion
• Abstract: Diffusion models have opened the path to a wide range of text-based image editing frameworks. However, these typically build on the multi-step nature of the diffusion backwards process, and adapting them to distilled, fast-sampling methods has proven surprisingly challenging. Here, we focus on a popular line of text-based editing frameworks - the ``edit-friendly'' DDPM-noise inversion approach. We analyze its application to fast sampling methods and categorize its failures into two classes: the appearance of visual artifacts, and insufficient editing strength. We trace the artifacts to mismatched noise statistics between inverted noises and the expected noise schedule, and suggest a shifted noise schedule which corrects for this offset. To increase editing strength, we propose a pseudo-guidance approach that efficiently increases the magnitude of edits without introducing new artifacts. All in all, our method enables text-based image editing with as few as three diffusion steps, while providing novel insights into the mechanisms behind popular text-based editing approaches.

Title: Virchow 2: Scaling Self-Supervised Mixed Magnification Models in Pathology

• Authors: Eric Zimmermann, Eugene Vorontsov, Julian Viret, Adam Casson, Michal Zelechowski, George Shaikovski, Neil Tenenholtz, James Hall, Thomas Fuchs, Nicolo Fusi, Siqi Liu, Kristen Severson
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.00738
• Pdf URL: https://arxiv.org/pdf/2408.00738
• Copy Paste: [[2408.00738]] Virchow 2: Scaling Self-Supervised Mixed Magnification Models in Pathology(https://arxiv.org/abs/2408.00738)
• Keywords: transformer
• Abstract: Foundation models are rapidly being developed for computational pathology applications. However, it remains an open question which factors are most important for downstream performance with data scale and diversity, model size, and training algorithm all playing a role. In this work, we present the result of scaling both data and model size, surpassing previous studies in both dimensions, and introduce two new models: Virchow 2, a 632M parameter vision transformer, and Virchow 2G, a 1.85B parameter vision transformer, each trained with 3.1M histopathology whole slide images. To support this scale, we propose domain-inspired adaptations to the DINOv2 training algorithm, which is quickly becoming the default method in self-supervised learning for computational pathology. We achieve state of the art performance on twelve tile-level tasks, as compared to the top performing competing models. Our results suggest that data diversity and domain-specific training can outperform models that only scale in the number of parameters, but, on average, performance benefits from domain-tailoring, data scale, and model scale.

Title: Collaborative Vision-Text Representation Optimizing for Open-Vocabulary Segmentation

• Authors: Siyu Jiao, Hongguang Zhu, Jiannan Huang, Yao Zhao, Yunchao Wei, Humphrey Shi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.00744
• Pdf URL: https://arxiv.org/pdf/2408.00744
• Copy Paste: [[2408.00744]] Collaborative Vision-Text Representation Optimizing for Open-Vocabulary Segmentation(https://arxiv.org/abs/2408.00744)
• Keywords: segmentation
• Abstract: Pre-trained vision-language models, e.g. CLIP, have been increasingly used to address the challenging Open-Vocabulary Segmentation (OVS) task, benefiting from their well-aligned vision-text embedding space. Typical solutions involve either freezing CLIP during training to unilaterally maintain its zero-shot capability, or fine-tuning CLIP vision encoder to achieve perceptual sensitivity to local regions. However, few of them incorporate vision-text collaborative optimization. Based on this, we propose the Content-Dependent Transfer to adaptively enhance each text embedding by interacting with the input image, which presents a parameter-efficient way to optimize the text representation. Besides, we additionally introduce a Representation Compensation strategy, reviewing the original CLIP-V representation as compensation to maintain the zero-shot capability of CLIP. In this way, the vision and text representation of CLIP are optimized collaboratively, enhancing the alignment of the vision-text feature space. To the best of our knowledge, we are the first to establish the collaborative vision-text optimizing mechanism within the OVS field. Extensive experiments demonstrate our method achieves superior performance on popular OVS benchmarks. In open-vocabulary semantic segmentation, our method outperforms the previous state-of-the-art approaches by +0.5, +2.3, +3.4, +0.4 and +1.1 mIoU, respectively on A-847, A-150, PC-459, PC-59 and PAS-20. Furthermore, in a panoptic setting on ADE20K, we achieve the performance of 27.1 PQ, 73.5 SQ, and 32.9 RQ. Code will be available at this https URL .

Title: Leaf Angle Estimation using Mask R-CNN and LETR Vision Transformer

• Authors: Venkat Margapuri, Prapti Thapaliya, Trevor Rife
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.00749
• Pdf URL: https://arxiv.org/pdf/2408.00749
• Copy Paste: [[2408.00749]] Leaf Angle Estimation using Mask R-CNN and LETR Vision Transformer(https://arxiv.org/abs/2408.00749)
• Keywords: transformer, segmentation
• Abstract: Modern day studies show a high degree of correlation between high yielding crop varieties and plants with upright leaf angles. It is observed that plants with upright leaf angles intercept more light than those without upright leaf angles, leading to a higher rate of photosynthesis. Plant scientists and breeders benefit from tools that can directly measure plant parameters in the field i.e. on-site phenotyping. The estimation of leaf angles by manual means in a field setting is tedious and cumbersome. We mitigate the tedium using a combination of the Mask R-CNN instance segmentation neural network, and Line Segment Transformer (LETR), a vision transformer. The proposed Computer Vision (CV) pipeline is applied on two image datasets, Summer 2015-Ames ULA and Summer 2015- Ames MLA, with a combined total of 1,827 plant images collected in the field using FieldBook, an Android application aimed at on-site phenotyping. The leaf angles estimated by the proposed pipeline on the image datasets are compared to two independent manual measurements using ImageJ, a Java-based image processing program developed at the National Institutes of Health and the Laboratory for Optical and Computational Instrumentation. The results, when compared for similarity using the Cosine Similarity measure, exhibit 0.98 similarity scores on both independent measurements of Summer 2015-Ames ULA and Summer 2015-Ames MLA image datasets, demonstrating the feasibility of the proposed pipeline for on-site measurement of leaf angles.

Title: Segment anything model 2: an application to 2D and 3D medical images

• Authors: Haoyu Dong, Hanxue Gu, Yaqian Chen, Jichen Yang, Maciej A. Mazurowski
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.00756
• Pdf URL: https://arxiv.org/pdf/2408.00756
• Copy Paste: [[2408.00756]] Segment anything model 2: an application to 2D and 3D medical images(https://arxiv.org/abs/2408.00756)
• Keywords: segmentation
• Abstract: Segment Anything Model (SAM) has gained significant attention because of its ability to segment a variety of objects in images given a prompt. The recently developed SAM 2 has extended this ability to video inputs. This opens an opportunity to apply SAM to 3D images, one of the fundamental tasks in the medical imaging field. In this paper, we provide an extensive evaluation of SAM 2's ability to segment both 2D and 3D medical images. We collect 18 medical imaging datasets, including common 3D modalities such as computed tomography (CT), magnetic resonance imaging (MRI), and positron emission tomography (PET) as well as 2D modalities such as X-ray and ultrasound. We consider two evaluation pipelines of SAM 2: (1) multi-frame 3D segmentation, where prompts are provided to one or multiple slice(s) selected from the volume, and (2) single-frame 2D segmentation, where prompts are provided to each slice. The former is only applicable to 3D modalities, while the latter applies to both 2D and 3D modalities. We learn that SAM 2 exhibits similar performance as SAM under single-frame 2D segmentation, and has variable performance under multi-frame 3D segmentation depending on the choices of slices to annotate, the direction of the propagation, the predictions utilized during the propagation, etc.

Title: Text-Guided Video Masked Autoencoder

• Authors: David Fan, Jue Wang, Shuai Liao, Zhikang Zhang, Vimal Bhat, Xinyu Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.00759
• Pdf URL: https://arxiv.org/pdf/2408.00759
• Copy Paste: [[2408.00759]] Text-Guided Video Masked Autoencoder(https://arxiv.org/abs/2408.00759)
• Keywords: robust
• Abstract: Recent video masked autoencoder (MAE) works have designed improved masking algorithms focused on saliency. These works leverage visual cues such as motion to mask the most salient regions. However, the robustness of such visual cues depends on how often input videos match underlying assumptions. On the other hand, natural language description is an information dense representation of video that implicitly captures saliency without requiring modality-specific assumptions, and has not been explored yet for video MAE. To this end, we introduce a novel text-guided masking algorithm (TGM) that masks the video regions with highest correspondence to paired captions. Without leveraging any explicit visual cues for saliency, our TGM is competitive with state-of-the-art masking algorithms such as motion-guided masking. To further benefit from the semantics of natural language for masked reconstruction, we next introduce a unified framework for joint MAE and masked video-text contrastive learning. We show that across existing masking algorithms, unifying MAE and masked video-text contrastive learning improves downstream performance compared to pure MAE on a variety of video recognition tasks, especially for linear probe. Within this unified framework, our TGM achieves the best relative performance on five action recognition and one egocentric datasets, highlighting the complementary nature of natural language for masked video modeling.

Title: Smoothed Energy Guidance: Guiding Diffusion Models with Reduced Energy Curvature of Attention

• Authors: Susung Hong
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.00760
• Pdf URL: https://arxiv.org/pdf/2408.00760
• Copy Paste: [[2408.00760]] Smoothed Energy Guidance: Guiding Diffusion Models with Reduced Energy Curvature of Attention(https://arxiv.org/abs/2408.00760)
• Keywords: diffusion
• Abstract: Conditional diffusion models have shown remarkable success in visual content generation, producing high-quality samples across various domains, largely due to classifier-free guidance (CFG). Recent attempts to extend guidance to unconditional models have relied on heuristic techniques, resulting in suboptimal generation quality and unintended effects. In this work, we propose Smoothed Energy Guidance (SEG), a novel training- and condition-free approach that leverages the energy-based perspective of the self-attention mechanism to enhance image generation. By defining the energy of self-attention, we introduce a method to reduce the curvature of the energy landscape of attention and use the output as the unconditional prediction. Practically, we control the curvature of the energy landscape by adjusting the Gaussian kernel parameter while keeping the guidance scale parameter fixed. Additionally, we present a query blurring method that is equivalent to blurring the entire attention weights without incurring quadratic complexity in the number of tokens. In our experiments, SEG achieves a Pareto improvement in both quality and the reduction of side effects. The code is available at \url{this https URL}.

Title: Tamper-Resistant Safeguards for Open-Weight LLMs

• Authors: Rishub Tamirisa, Bhrugu Bharathi, Long Phan, Andy Zhou, Alice Gatti, Tarun Suresh, Maxwell Lin, Justin Wang, Rowan Wang, Ron Arel, Andy Zou, Dawn Song, Bo Li, Dan Hendrycks, Mantas Mazeika
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2408.00761
• Pdf URL: https://arxiv.org/pdf/2408.00761
• Copy Paste: [[2408.00761]] Tamper-Resistant Safeguards for Open-Weight LLMs(https://arxiv.org/abs/2408.00761)
• Keywords: security, attack, robust, large language model
• Abstract: Rapid advances in the capabilities of large language models (LLMs) have raised widespread concerns regarding their potential for malicious use. Open-weight LLMs present unique challenges, as existing safeguards lack robustness to tampering attacks that modify model weights. For example, recent works have demonstrated that refusal and unlearning safeguards can be trivially removed with a few steps of fine-tuning. These vulnerabilities necessitate new approaches for enabling the safe release of open-weight LLMs. We develop a method, called TAR, for building tamper-resistant safeguards into open-weight LLMs such that adversaries cannot remove the safeguards even after thousands of steps of fine-tuning. In extensive evaluations and red teaming analyses, we find that our method greatly improves tamper-resistance while preserving benign capabilities. Our results demonstrate that tamper-resistance is a tractable problem, opening up a promising new avenue to improve the safety and security of open-weight LLMs.

Title: AgentGen: Enhancing Planning Abilities for Large Language Model based Agent via Environment and Task Generation

• Authors: Mengkang Hu, Pu Zhao, Can Xu, Qingfeng Sun, Jianguang Lou, Qingwei Lin, Ping Luo, Saravan Rajmohan, Dongmei Zhang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.00764
• Pdf URL: https://arxiv.org/pdf/2408.00764
• Copy Paste: [[2408.00764]] AgentGen: Enhancing Planning Abilities for Large Language Model based Agent via Environment and Task Generation(https://arxiv.org/abs/2408.00764)
• Keywords: large language model
• Abstract: Large Language Model (LLM) based agents have garnered significant attention and are becoming increasingly popular. Furthermore, planning ability is a crucial component of an LLM-based agent, involving interaction with the environment and executing actions to complete a planning task, which generally entails achieving a desired goal from an initial state. This paper investigates enhancing the planning abilities of LLMs through instruction tuning, referred to as agent training. Recent studies have demonstrated that utilizing expert-level trajectory for instruction-tuning LLMs effectively enhances their planning capabilities. However, existing work primarily focuses on synthesizing trajectories from manually designed planning tasks and environments. The labor-intensive nature of creating these environments and tasks impedes the generation of sufficiently varied and extensive trajectories. To address this limitation, this paper explores the automated synthesis of diverse environments and a gradual range of planning tasks, from easy to difficult. We introduce a framework, AgentGen, that leverages LLMs first to generate environments and subsequently generate planning tasks conditioned on these environments. Specifically, to improve environmental diversity, we propose using an inspiration corpus composed of various domain-specific text segments as the context for synthesizing environments. Moreover, to increase the difficulty diversity of generated planning tasks, we propose a bidirectional evolution method, Bi-Evol, that evolves planning tasks from easier and harder directions to synthesize a task set with a smoother difficulty curve. The evaluation results derived from AgentBoard show that AgentGen greatly improves LLMs' planning ability, e.g., the AgentGen instruction-tuned Llama-3 8B surpasses GPT-3.5 in overall performance. Moreover, in certain tasks, it even outperforms GPT-4.

Title: Optimizing Diffusion Models for Joint Trajectory Prediction and Controllable Generation

• Authors: Yixiao Wang, Chen Tang, Lingfeng Sun, Simone Rossi, Yichen Xie, Chensheng Peng, Thomas Hannagan, Stefano Sabatini, Nicola Poerio, Masayoshi Tomizuka, Wei Zhan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.00766
• Pdf URL: https://arxiv.org/pdf/2408.00766
• Copy Paste: [[2408.00766]] Optimizing Diffusion Models for Joint Trajectory Prediction and Controllable Generation(https://arxiv.org/abs/2408.00766)
• Keywords: diffusion, generative
• Abstract: Diffusion models are promising for joint trajectory prediction and controllable generation in autonomous driving, but they face challenges of inefficient inference steps and high computational demands. To tackle these challenges, we introduce Optimal Gaussian Diffusion (OGD) and Estimated Clean Manifold (ECM) Guidance. OGD optimizes the prior distribution for a small diffusion time $T$ and starts the reverse diffusion process from it. ECM directly injects guidance gradients to the estimated clean manifold, eliminating extensive gradient backpropagation throughout the network. Our methodology streamlines the generative process, enabling practical applications with reduced computational overhead. Experimental validation on the large-scale Argoverse 2 dataset demonstrates our approach's superior performance, offering a viable solution for computationally efficient, high-quality joint trajectory prediction and controllable generation for autonomous driving. Our project webpage is at this https URL.