2024-03-27

Title: HEAL-ViT: Vision Transformers on a spherical mesh for medium-range weather forecasting

• Authors: Vivek Ramavajjala
• Subjects: cs.CV, cs.LG, physics.ao-ph
• Abstract URL: https://arxiv.org/abs/2403.17016
• Pdf URL: https://arxiv.org/pdf/2403.17016
• Copy Paste: [[2403.17016]] HEAL-ViT: Vision Transformers on a spherical mesh for medium-range weather forecasting(https://arxiv.org/abs/2403.17016)
• Keywords: transformer
• Abstract: In recent years, a variety of ML architectures and techniques have seen success in producing skillful medium range weather forecasts. In particular, Vision Transformer (ViT)-based models (e.g. Pangu-Weather, FuXi) have shown strong performance, working nearly "out-of-the-box" by treating weather data as a multi-channel image on a rectilinear grid. While a rectilinear grid is appropriate for 2D images, weather data is inherently spherical and thus heavily distorted at the poles on a rectilinear grid, leading to disproportionate compute being used to model data near the poles. Graph-based methods (e.g. GraphCast) do not suffer from this problem, as they map the longitude-latitude grid to a spherical mesh, but are generally more memory intensive and tend to need more compute resources for training and inference. While spatially homogeneous, the spherical mesh does not lend itself readily to be modeled by ViT-based models that implicitly rely on the rectilinear grid structure. We present HEAL-ViT, a novel architecture that uses ViT models on a spherical mesh, thus benefiting from both the spatial homogeneity enjoyed by graph-based models and efficient attention-based mechanisms exploited by transformers. HEAL-ViT produces weather forecasts that outperform the ECMWF IFS on key metrics, and demonstrate better bias accumulation and blurring than other ML weather prediction models. Further, the lowered compute footprint of HEAL-ViT makes it attractive for operational use as well, where other models in addition to a 6-hourly prediction model may be needed to produce the full set of operational forecasts required.

Title: Continuous, Subject-Specific Attribute Control in T2I Models by Identifying Semantic Directions

• Authors: Stefan Andreas Baumann, Felix Krause, Michael Neumayr, Nick Stracke, Vincent Tao Hu, Björn Ommer
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.17064
• Pdf URL: https://arxiv.org/pdf/2403.17064
• Copy Paste: [[2403.17064]] Continuous, Subject-Specific Attribute Control in T2I Models by Identifying Semantic Directions(https://arxiv.org/abs/2403.17064)
• Keywords: robust, diffusion
• Abstract: In recent years, advances in text-to-image (T2I) diffusion models have substantially elevated the quality of their generated images. However, achieving fine-grained control over attributes remains a challenge due to the limitations of natural language prompts (such as no continuous set of intermediate descriptions existing between ``person'' and ``old person''). Even though many methods were introduced that augment the model or generation process to enable such control, methods that do not require a fixed reference image are limited to either enabling global fine-grained attribute expression control or coarse attribute expression control localized to specific subjects, not both simultaneously. We show that there exist directions in the commonly used token-level CLIP text embeddings that enable fine-grained subject-specific control of high-level attributes in text-to-image models. Based on this observation, we introduce one efficient optimization-free and one robust optimization-based method to identify these directions for specific attributes from contrastive text prompts. We demonstrate that these directions can be used to augment the prompt text input with fine-grained control over attributes of specific subjects in a compositional manner (control over multiple attributes of a single subject) without having to adapt the diffusion model. Project page: https://compvis.github.io/attribute-control. Code is available at https://github.com/CompVis/attribute-control.

Title: Semantic Ranking for Automated Adversarial Technique Annotation in Security Text

• Authors: Udesh Kumarasinghe, Ahmed Lekssays, Husrev Taha Sencar, Sabri Boughorbel, Charitha Elvitigala, Preslav Nakov
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.17068
• Pdf URL: https://arxiv.org/pdf/2403.17068
• Copy Paste: [[2403.17068]] Semantic Ranking for Automated Adversarial Technique Annotation in Security Text(https://arxiv.org/abs/2403.17068)
• Keywords: security, large language model
• Abstract: We introduce a new method for extracting structured threat behaviors from threat intelligence text. Our method is based on a multi-stage ranking architecture that allows jointly optimizing for efficiency and effectiveness. Therefore, we believe this problem formulation better aligns with the real-world nature of the task considering the large number of adversary techniques and the extensive body of threat intelligence created by security analysts. Our findings show that the proposed system yields state-of-the-art performance results for this task. Results show that our method has a top-3 recall performance of 81\% in identifying the relevant technique among 193 top-level techniques. Our tests also demonstrate that our system performs significantly better (+40\%) than the widely used large language models when tested under a zero-shot setting.

Title: Enhancing UAV Security Through Zero Trust Architecture: An Advanced Deep Learning and Explainable AI Analysis

• Authors: Ekramul Haque, Kamrul Hasan, Imtiaz Ahmed, Md. Sahabul Alam, Tariqul Islam
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2403.17093
• Pdf URL: https://arxiv.org/pdf/2403.17093
• Copy Paste: [[2403.17093]] Enhancing UAV Security Through Zero Trust Architecture: An Advanced Deep Learning and Explainable AI Analysis(https://arxiv.org/abs/2403.17093)
• Keywords: security, interpretability
• Abstract: In the dynamic and ever-changing domain of Unmanned Aerial Vehicles (UAVs), the utmost importance lies in guaranteeing resilient and lucid security measures. This study highlights the necessity of implementing a Zero Trust Architecture (ZTA) to enhance the security of unmanned aerial vehicles (UAVs), hence departing from conventional perimeter defences that may expose vulnerabilities. The Zero Trust Architecture (ZTA) paradigm requires a rigorous and continuous process of authenticating all network entities and communications. The accuracy of our methodology in detecting and identifying unmanned aerial vehicles (UAVs) is 84.59\%. This is achieved by utilizing Radio Frequency (RF) signals within a Deep Learning framework, a unique method. Precise identification is crucial in Zero Trust Architecture (ZTA), as it determines network access. In addition, the use of eXplainable Artificial Intelligence (XAI) tools such as SHapley Additive exPlanations (SHAP) and Local Interpretable Model-agnostic Explanations (LIME) contributes to the improvement of the model's transparency and interpretability. Adherence to Zero Trust Architecture (ZTA) standards guarantees that the classifications of unmanned aerial vehicles (UAVs) are verifiable and comprehensible, enhancing security within the UAV field.

Title: Attribute First, then Generate: Locally-attributable Grounded Text Generation

• Authors: Aviv Slobodkin, Eran Hirsch, Arie Cattan, Tal Schuster, Ido Dagan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.17104
• Pdf URL: https://arxiv.org/pdf/2403.17104
• Copy Paste: [[2403.17104]] Attribute First, then Generate: Locally-attributable Grounded Text Generation(https://arxiv.org/abs/2403.17104)
• Keywords: large language model
• Abstract: Recent efforts to address hallucinations in Large Language Models (LLMs) have focused on attributed text generation, which supplements generated texts with citations of supporting sources for post-generation fact-checking and corrections. Yet, these citations often point to entire documents or paragraphs, burdening users with extensive verification work. In this paper, we introduce a locally-attributable text generation approach, prioritizing concise attributions. Our method, named ``Attribute First, then Generate'', breaks down the conventional end-to-end generation process into three intuitive steps: content selection, sentence planning, and sequential sentence generation. By initially identifying relevant source segments (``select first'') and then conditioning the generation process on them (``then generate''), we ensure these segments also act as the output's fine-grained attributions (``select'' becomes ``attribute''). Tested on Multi-document Summarization and Long-form Question-answering, our method not only yields more concise citations than the baselines but also maintains - and in some cases enhances - both generation quality and attribution accuracy. Furthermore, it significantly reduces the time required for fact verification by human assessors.

Title: Stochastic Gradient Langevin Unlearning

• Authors: Eli Chien, Haoyu Wang, Ziang Chen, Pan Li
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2403.17105
• Pdf URL: https://arxiv.org/pdf/2403.17105
• Copy Paste: [[2403.17105]] Stochastic Gradient Langevin Unlearning(https://arxiv.org/abs/2403.17105)
• Keywords: privacy
• Abstract: ``The right to be forgotten'' ensured by laws for user data privacy becomes increasingly important. Machine unlearning aims to efficiently remove the effect of certain data points on the trained model parameters so that it can be approximately the same as if one retrains the model from scratch. This work proposes stochastic gradient Langevin unlearning, the first unlearning framework based on noisy stochastic gradient descent (SGD) with privacy guarantees for approximate unlearning problems under convexity assumption. Our results show that mini-batch gradient updates provide a superior privacy-complexity trade-off compared to the full-batch counterpart. There are numerous algorithmic benefits of our unlearning approach, including complexity saving compared to retraining, and supporting sequential and batch unlearning. To examine the privacy-utility-complexity trade-off of our method, we conduct experiments on benchmark datasets compared against prior works. Our approach achieves a similar utility under the same privacy constraint while using $2\%$ and $10\%$ of the gradient computations compared with the state-of-the-art gradient-based approximate unlearning methods for mini-batch and full-batch settings, respectively.

Title: The Strong Pull of Prior Knowledge in Large Language Models and Its Impact on Emotion Recognition

• Authors: Georgios Chochlakis, Alexandros Potamianos, Kristina Lerman, Shrikanth Narayanan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.17125
• Pdf URL: https://arxiv.org/pdf/2403.17125
• Copy Paste: [[2403.17125]] The Strong Pull of Prior Knowledge in Large Language Models and Its Impact on Emotion Recognition(https://arxiv.org/abs/2403.17125)
• Keywords: large language model
• Abstract: In-context Learning (ICL) has emerged as a powerful paradigm for performing natural language tasks with Large Language Models (LLM) without updating the models' parameters, in contrast to the traditional gradient-based finetuning. The promise of ICL is that the LLM can adapt to perform the present task at a competitive or state-of-the-art level at a fraction of the cost. The ability of LLMs to perform tasks in this few-shot manner relies on their background knowledge of the task (or task priors). However, recent work has found that, unlike traditional learning, LLMs are unable to fully integrate information from demonstrations that contrast task priors. This can lead to performance saturation at suboptimal levels, especially for subjective tasks such as emotion recognition, where the mapping from text to emotions can differ widely due to variability in human annotations. In this work, we design experiments and propose measurements to explicitly quantify the consistency of proxies of LLM priors and their pull on the posteriors. We show that LLMs have strong yet inconsistent priors in emotion recognition that ossify their predictions. We also find that the larger the model, the stronger these effects become. Our results suggest that caution is needed when using ICL with larger LLMs for affect-centered tasks outside their pre-training domain and when interpreting ICL results.

Title: Benchmarking Video Frame Interpolation

• Authors: Simon Kiefhaber, Simon Niklaus, Feng Liu, Simone Schaub-Meyer
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.17128
• Pdf URL: https://arxiv.org/pdf/2403.17128
• Copy Paste: [[2403.17128]] Benchmarking Video Frame Interpolation(https://arxiv.org/abs/2403.17128)
• Keywords: fair
• Abstract: Video frame interpolation, the task of synthesizing new frames in between two or more given ones, is becoming an increasingly popular research target. However, the current evaluation of frame interpolation techniques is not ideal. Due to the plethora of test datasets available and inconsistent computation of error metrics, a coherent and fair comparison across papers is very challenging. Furthermore, new test sets have been proposed as part of method papers so they are unable to provide the in-depth evaluation of a dedicated benchmarking paper. Another severe downside is that these test sets violate the assumption of linearity when given two input frames, making it impossible to solve without an oracle. We hence strongly believe that the community would greatly benefit from a benchmarking paper, which is what we propose. Specifically, we present a benchmark which establishes consistent error metrics by utilizing a submission website that computes them, provides insights by analyzing the interpolation quality with respect to various per-pixel attributes such as the motion magnitude, contains a carefully designed test set adhering to the assumption of linearity by utilizing synthetic data, and evaluates the computational efficiency in a coherent manner.

Title: MetaAligner: Conditional Weak-to-Strong Correction for Generalizable Multi-Objective Alignment of Language Models

• Authors: Kailai Yang, Zhiwei Liu, Qianqian Xie, Tianlin Zhang, Nirui Song, Jimin Huang, Ziyan Kuang, Sophia Ananiadou
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.17141
• Pdf URL: https://arxiv.org/pdf/2403.17141
• Copy Paste: [[2403.17141]] MetaAligner: Conditional Weak-to-Strong Correction for Generalizable Multi-Objective Alignment of Language Models(https://arxiv.org/abs/2403.17141)
• Keywords: large language model
• Abstract: Recent advancements in large language models (LLMs) aim to tackle heterogeneous human expectations and values via multi-objective preference alignment. However, existing methods are parameter-adherent to the policy model, leading to two key limitations: (1) the high-cost repetition of their alignment algorithms for each new target model; (2) they cannot expand to unseen objectives due to their static alignment objectives. In this work, we propose Meta-Objective Aligner (MetaAligner), a model that performs conditional weak-to-strong correction for weak responses to approach strong responses. MetaAligner is the first policy-agnostic and generalizable method for multi-objective preference alignment, which enables plug-and-play alignment by decoupling parameter updates from the policy models and facilitates zero-shot preference alignment for unseen objectives via in-context learning. Experimental results show that MetaAligner achieves significant and balanced improvements in multi-objective alignments on 11 policy models with up to 63x more parameters, and outperforms previous alignment methods with down to 22.27x less computational resources. The model also accurately aligns with unseen objectives, marking the first step towards generalizable multi-objective preference alignment.

Title: Guided Distant Supervision for Multilingual Relation Extraction Data: Adapting to a New Language

• Authors: Alistair Plum, Tharindu Ranasinghe, Christoph Purschke
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.17143
• Pdf URL: https://arxiv.org/pdf/2403.17143
• Copy Paste: [[2403.17143]] Guided Distant Supervision for Multilingual Relation Extraction Data: Adapting to a New Language(https://arxiv.org/abs/2403.17143)
• Keywords: extraction
• Abstract: Relation extraction is essential for extracting and understanding biographical information in the context of digital humanities and related subjects. There is a growing interest in the community to build datasets capable of training machine learning models to extract relationships. However, annotating such datasets can be expensive and time-consuming, in addition to being limited to English. This paper applies guided distant supervision to create a large biographical relationship extraction dataset for German. Our dataset, composed of more than 80,000 instances for nine relationship types, is the largest biographical German relationship extraction dataset. We also create a manually annotated dataset with 2000 instances to evaluate the models and release it together with the dataset compiled using guided distant supervision. We train several state-of-the-art machine learning models on the automatically created dataset and release them as well. Furthermore, we experiment with multilingual and cross-lingual experiments that could benefit many low-resource languages.

Title: Outcome-Constrained Large Language Models for Countering Hate Speech

• Authors: Lingzi Hong, Pengcheng Luo, Eduardo Blanco, Xiaoying Song
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.17146
• Pdf URL: https://arxiv.org/pdf/2403.17146
• Copy Paste: [[2403.17146]] Outcome-Constrained Large Language Models for Countering Hate Speech(https://arxiv.org/abs/2403.17146)
• Keywords: transformer, large language model
• Abstract: Counterspeech that challenges or responds to hate speech has been seen as an alternative to mitigate the negative impact of hate speech and foster productive online communications. Research endeavors have been directed to using language models for the automatic generation of counterspeech to assist efforts in combating online hate. Existing research focuses on the generation of counterspeech with certain linguistic attributes, such as being polite, informative, and intent-driven. However, it remains unclear what impact the counterspeech might have in an online environment. We first explore methods that utilize large language models (LLM) to generate counterspeech constrained by potential conversation outcomes. We build two conversation outcome classifiers that predict the incivility level and the hater reentry behavior following replies to hate with Reddit data, then propose four methods to incorporate the desired outcomes, i.e., low conversation incivility and non-hateful hater reentry, into the text generation process, including Prompt with Instructions, Prompt and Select, LLM finetune, and LLM transformer reinforcement learning (TRL). Evaluation results show effective strategies to generate outcome-constrained counterspeech and the linguistic characteristics of texts generated by different methods.

Title: Task-Agnostic Detector for Insertion-Based Backdoor Attacks

• Authors: Weimin Lyu, Xiao Lin, Songzhu Zheng, Lu Pang, Haibin Ling, Susmit Jha, Chao Chen
• Subjects: cs.CL, cs.CR
• Abstract URL: https://arxiv.org/abs/2403.17155
• Pdf URL: https://arxiv.org/pdf/2403.17155
• Copy Paste: [[2403.17155]] Task-Agnostic Detector for Insertion-Based Backdoor Attacks(https://arxiv.org/abs/2403.17155)
• Keywords: security, attack
• Abstract: Textual backdoor attacks pose significant security threats. Current detection approaches, typically relying on intermediate feature representation or reconstructing potential triggers, are task-specific and less effective beyond sentence classification, struggling with tasks like question answering and named entity recognition. We introduce TABDet (Task-Agnostic Backdoor Detector), a pioneering task-agnostic method for backdoor detection. TABDet leverages final layer logits combined with an efficient pooling technique, enabling unified logit representation across three prominent NLP tasks. TABDet can jointly learn from diverse task-specific models, demonstrating superior detection efficacy over traditional task-specific methods.

Title: Less Is More - On the Importance of Sparsification for Transformers and Graph Neural Networks for TSP

• Authors: Attila Lischka, Jiaming Wu, Rafael Basso, Morteza Haghir Chehreghani, Balázs Kulcsár
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.17159
• Pdf URL: https://arxiv.org/pdf/2403.17159
• Copy Paste: [[2403.17159]] Less Is More - On the Importance of Sparsification for Transformers and Graph Neural Networks for TSP(https://arxiv.org/abs/2403.17159)
• Keywords: transformer
• Abstract: Most of the recent studies tackling routing problems like the Traveling Salesman Problem (TSP) with machine learning use a transformer or Graph Neural Network (GNN) based encoder architecture. However, many of them apply these encoders naively by allowing them to aggregate information over the whole TSP instances. We, on the other hand, propose a data preprocessing method that allows the encoders to focus on the most relevant parts of the TSP instances only. In particular, we propose graph sparsification for TSP graph representations passed to GNNs and attention masking for TSP instances passed to transformers where the masks correspond to the adjacency matrices of the sparse TSP graph representations. Furthermore, we propose ensembles of different sparsification levels allowing models to focus on the most promising parts while also allowing information flow between all nodes of a TSP instance. In the experimental studies, we show that for GNNs appropriate sparsification and ensembles of different sparsification levels lead to substantial performance increases of the overall architecture. We also design a new, state-of-the-art transformer encoder with ensembles of attention masking. These transformers increase model performance from a gap of $0.16\%$ to $0.10\%$ for TSP instances of size 100 and from $0.02\%$ to $0.00\%$ for TSP instances of size 50.

Title: CYGENT: A cybersecurity conversational agent with log summarization powered by GPT-3

• Authors: Prasasthy Balasubramanian, Justin Seby, Panos Kostakos
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.17160
• Pdf URL: https://arxiv.org/pdf/2403.17160
• Copy Paste: [[2403.17160]] CYGENT: A cybersecurity conversational agent with log summarization powered by GPT-3(https://arxiv.org/abs/2403.17160)
• Keywords: security, attack, generative, large language model
• Abstract: In response to the escalating cyber-attacks in the modern IT and IoT landscape, we developed CYGENT, a conversational agent framework powered by GPT-3.5 turbo model, designed to aid system administrators in ensuring optimal performance and uninterrupted resource availability. This study focuses on fine-tuning GPT-3 models for cybersecurity tasks, including conversational AI and generative AI tailored specifically for cybersecurity operations. CYGENT assists users by providing cybersecurity information, analyzing and summarizing uploaded log files, detecting specific events, and delivering essential instructions. The conversational agent was developed based on the GPT-3.5 turbo model. We fine-tuned and validated summarizer models (GPT3) using manually generated data points. Using this approach, we achieved a BERTscore of over 97%, indicating GPT-3's enhanced capability in summarizing log files into human-readable formats and providing necessary information to users. Furthermore, we conducted a comparative analysis of GPT-3 models with other Large Language Models (LLMs), including CodeT5-small, CodeT5-base, and CodeT5-base-multi-sum, with the objective of analyzing log analysis techniques. Our analysis consistently demonstrated that Davinci (GPT-3) model outperformed all other LLMs, showcasing higher performance. These findings are crucial for improving human comprehension of logs, particularly in light of the increasing numbers of IoT devices. Additionally, our research suggests that the CodeT5-base-multi-sum model exhibits comparable performance to Davinci to some extent in summarizing logs, indicating its potential as an offline model for this task.

Title: Engagement Measurement Based on Facial Landmarks and Spatial-Temporal Graph Convolutional Networks

• Authors: Ali Abedi, Shehroz S. Khan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.17175
• Pdf URL: https://arxiv.org/pdf/2403.17175
• Copy Paste: [[2403.17175]] Engagement Measurement Based on Facial Landmarks and Spatial-Temporal Graph Convolutional Networks(https://arxiv.org/abs/2403.17175)
• Keywords: privacy
• Abstract: Engagement in virtual learning is crucial for a variety of factors including learner satisfaction, performance, and compliance with learning programs, but measuring it is a challenging task. There is therefore considerable interest in utilizing artificial intelligence and affective computing to measure engagement in natural settings as well as on a large scale. This paper introduces a novel, privacy-preserving method for engagement measurement from videos. It uses facial landmarks, which carry no personally identifiable information, extracted from videos via the MediaPipe deep learning solution. The extracted facial landmarks are fed to a Spatial-Temporal Graph Convolutional Network (ST-GCN) to output the engagement level of the learner in the video. To integrate the ordinal nature of the engagement variable into the training process, ST-GCNs undergo training in a novel ordinal learning framework based on transfer learning. Experimental results on two video student engagement measurement datasets show the superiority of the proposed method compared to previous methods with improved state-of-the-art on the EngageNet dataset with a %3.1 improvement in four-class engagement level classification accuracy and on the Online Student Engagement dataset with a %1.5 improvement in binary engagement classification accuracy. The relatively lightweight ST-GCN and its integration with the real-time MediaPipe deep learning solution make the proposed approach capable of being deployed on virtual learning platforms and measuring engagement in real time.

Title: LOTUS: Evasive and Resilient Backdoor Attacks through Sub-Partitioning

• Authors: Siyuan Cheng, Guanhong Tao, Yingqi Liu, Guangyu Shen, Shengwei An, Shiwei Feng, Xiangzhe Xu, Kaiyuan Zhang, Shiqing Ma, Xiangyu Zhang
• Subjects: cs.CV, cs.CR
• Abstract URL: https://arxiv.org/abs/2403.17188
• Pdf URL: https://arxiv.org/pdf/2403.17188
• Copy Paste: [[2403.17188]] LOTUS: Evasive and Resilient Backdoor Attacks through Sub-Partitioning(https://arxiv.org/abs/2403.17188)
• Keywords: security, attack
• Abstract: Backdoor attack poses a significant security threat to Deep Learning applications. Existing attacks are often not evasive to established backdoor detection techniques. This susceptibility primarily stems from the fact that these attacks typically leverage a universal trigger pattern or transformation function, such that the trigger can cause misclassification for any input. In response to this, recent papers have introduced attacks using sample-specific invisible triggers crafted through special transformation functions. While these approaches manage to evade detection to some extent, they reveal vulnerability to existing backdoor mitigation techniques. To address and enhance both evasiveness and resilience, we introduce a novel backdoor attack LOTUS. Specifically, it leverages a secret function to separate samples in the victim class into a set of partitions and applies unique triggers to different partitions. Furthermore, LOTUS incorporates an effective trigger focusing mechanism, ensuring only the trigger corresponding to the partition can induce the backdoor behavior. Extensive experimental results show that LOTUS can achieve high attack success rate across 4 datasets and 7 model structures, and effectively evading 13 backdoor detection and mitigation techniques. The code is available at https://github.com/Megum1/LOTUS.

Title: Strategies to Improve Real-World Applicability of Laparoscopic Anatomy Segmentation Models

• Authors: Fiona R. Kolbinger, Jiangpeng He, Jinge Ma, Fengqing Zhu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.17192
• Pdf URL: https://arxiv.org/pdf/2403.17192
• Copy Paste: [[2403.17192]] Strategies to Improve Real-World Applicability of Laparoscopic Anatomy Segmentation Models(https://arxiv.org/abs/2403.17192)
• Keywords: segmentation
• Abstract: Accurate identification and localization of anatomical structures of varying size and appearance in laparoscopic imaging are necessary to leverage the potential of computer vision techniques for surgical decision support. Segmentation performance of such models is traditionally reported using metrics of overlap such as IoU. However, imbalanced and unrealistic representation of classes in the training data and suboptimal selection of reported metrics have the potential to skew nominal segmentation performance and thereby ultimately limit clinical translation. In this work, we systematically analyze the impact of class characteristics (i.e., organ size differences), training and test data composition (i.e., representation of positive and negative examples), and modeling parameters (i.e., foreground-to-background class weight) on eight segmentation metrics: accuracy, precision, recall, IoU, F1 score, specificity, Hausdorff Distance, and Average Symmetric Surface Distance. Based on our findings, we propose two simple yet effective strategies to improve real-world applicability of image segmentation models in laparoscopic surgical data: (1) inclusion of negative examples in the training process and (2) adaptation of foreground-background weights in segmentation models to maximize model performance with respect to specific metrics of interest, depending on the clinical use case.

Title: Extracting Social Support and Social Isolation Information from Clinical Psychiatry Notes: Comparing a Rule-based NLP System and a Large Language Model

• Authors: Braja Gopal Patra, Lauren A. Lepow, Praneet Kasi Reddy Jagadeesh Kumar, Veer Vekaria, Mohit Manoj Sharma, Prakash Adekkanattu, Brian Fennessy, Gavin Hynes, Isotta Landi, Jorge A. Sanchez-Ruiz, Euijung Ryu, Joanna M. Biernacka, Girish N. Nadkarni, Ardesheer Talati, Myrna Weissman, Mark Olfson, J. John Mann, Alexander W. Charney, Jyotishman Pathak
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.17199
• Pdf URL: https://arxiv.org/pdf/2403.17199
• Copy Paste: [[2403.17199]] Extracting Social Support and Social Isolation Information from Clinical Psychiatry Notes: Comparing a Rule-based NLP System and a Large Language Model(https://arxiv.org/abs/2403.17199)
• Keywords: extraction, large language model
• Abstract: Background: Social support (SS) and social isolation (SI) are social determinants of health (SDOH) associated with psychiatric outcomes. In electronic health records (EHRs), individual-level SS/SI is typically documented as narrative clinical notes rather than structured coded data. Natural language processing (NLP) algorithms can automate the otherwise labor-intensive process of data extraction. Data and Methods: Psychiatric encounter notes from Mount Sinai Health System (MSHS, n=300) and Weill Cornell Medicine (WCM, n=225) were annotated and established a gold standard corpus. A rule-based system (RBS) involving lexicons and a large language model (LLM) using FLAN-T5-XL were developed to identify mentions of SS and SI and their subcategories (e.g., social network, instrumental support, and loneliness). Results: For extracting SS/SI, the RBS obtained higher macro-averaged f-scores than the LLM at both MSHS (0.89 vs. 0.65) and WCM (0.85 vs. 0.82). For extracting subcategories, the RBS also outperformed the LLM at both MSHS (0.90 vs. 0.62) and WCM (0.82 vs. 0.81). Discussion and Conclusion: Unexpectedly, the RBS outperformed the LLMs across all metrics. Intensive review demonstrates that this finding is due to the divergent approach taken by the RBS and LLM. The RBS were designed and refined to follow the same specific rules as the gold standard annotations. Conversely, the LLM were more inclusive with categorization and conformed to common English-language understanding. Both approaches offer advantages and are made available open-source for future testing.

Title: CADGL: Context-Aware Deep Graph Learning for Predicting Drug-Drug Interactions

• Authors: Azmine Toushik Wasi, Taki Hasan Rafi, Raima Islam, Serbetar Karlo, Dong-Kyu Chae
• Subjects: cs.LG, cs.AI, cs.IR, q-bio.BM, q-bio.MN
• Abstract URL: https://arxiv.org/abs/2403.17210
• Pdf URL: https://arxiv.org/pdf/2403.17210
• Copy Paste: [[2403.17210]] CADGL: Context-Aware Deep Graph Learning for Predicting Drug-Drug Interactions(https://arxiv.org/abs/2403.17210)
• Keywords: robust, extraction
• Abstract: Examining Drug-Drug Interactions (DDIs) is a pivotal element in the process of drug development. DDIs occur when one drug's properties are affected by the inclusion of other drugs. Detecting favorable DDIs has the potential to pave the way for creating and advancing innovative medications applicable in practical settings. However, existing DDI prediction models continue to face challenges related to generalization in extreme cases, robust feature extraction, and real-life application possibilities. We aim to address these challenges by leveraging the effectiveness of context-aware deep graph learning by introducing a novel framework named CADGL. Based on a customized variational graph autoencoder (VGAE), we capture critical structural and physio-chemical information using two context preprocessors for feature extraction from two different perspectives: local neighborhood and molecular context, in a heterogeneous graphical structure. Our customized VGAE consists of a graph encoder, a latent information encoder, and an MLP decoder. CADGL surpasses other state-of-the-art DDI prediction models, excelling in predicting clinically valuable novel DDIs, supported by rigorous case studies.

Title: AnimateMe: 4D Facial Expressions via Diffusion Models

• Authors: Dimitrios Gerogiannis, Foivos Paraperas Papantoniou, Rolandos Alexandros Potamias, Alexandros Lattas, Stylianos Moschoglou, Stylianos Ploumpis, Stefanos Zafeiriou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.17213
• Pdf URL: https://arxiv.org/pdf/2403.17213
• Copy Paste: [[2403.17213]] AnimateMe: 4D Facial Expressions via Diffusion Models(https://arxiv.org/abs/2403.17213)
• Keywords: diffusion, generative
• Abstract: The field of photorealistic 3D avatar reconstruction and generation has garnered significant attention in recent years; however, animating such avatars remains challenging. Recent advances in diffusion models have notably enhanced the capabilities of generative models in 2D animation. In this work, we directly utilize these models within the 3D domain to achieve controllable and high-fidelity 4D facial animation. By integrating the strengths of diffusion processes and geometric deep learning, we employ Graph Neural Networks (GNNs) as denoising diffusion models in a novel approach, formulating the diffusion process directly on the mesh space and enabling the generation of 3D facial expressions. This facilitates the generation of facial deformations through a mesh-diffusion-based model. Additionally, to ensure temporal coherence in our animations, we propose a consistent noise sampling method. Under a series of both quantitative and qualitative experiments, we showcase that the proposed method outperforms prior work in 4D expression synthesis by generating high-fidelity extreme expressions. Furthermore, we applied our method to textured 4D facial expression generation, implementing a straightforward extension that involves training on a large-scale textured 4D facial expression database.

Title: Ontology Completion with Natural Language Inference and Concept Embeddings: An Analysis

• Authors: Na Li, Thomas Bailleux, Zied Bouraoui, Steven Schockaert
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.17216
• Pdf URL: https://arxiv.org/pdf/2403.17216
• Copy Paste: [[2403.17216]] Ontology Completion with Natural Language Inference and Concept Embeddings: An Analysis(https://arxiv.org/abs/2403.17216)
• Keywords: large language model
• Abstract: We consider the problem of finding plausible knowledge that is missing from a given ontology, as a generalisation of the well-studied taxonomy expansion task. One line of work treats this task as a Natural Language Inference (NLI) problem, thus relying on the knowledge captured by language models to identify the missing knowledge. Another line of work uses concept embeddings to identify what different concepts have in common, taking inspiration from cognitive models for category based induction. These two approaches are intuitively complementary, but their effectiveness has not yet been compared. In this paper, we introduce a benchmark for evaluating ontology completion methods and thoroughly analyse the strengths and weaknesses of both approaches. We find that both approaches are indeed complementary, with hybrid strategies achieving the best overall results. We also find that the task is highly challenging for Large Language Models, even after fine-tuning.

Title: DiffusionAct: Controllable Diffusion Autoencoder for One-shot Face Reenactment

• Authors: Stella Bounareli, Christos Tzelepis, Vasileios Argyriou, Ioannis Patras, Georgios Tzimiropoulos
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.17217
• Pdf URL: https://arxiv.org/pdf/2403.17217
• Copy Paste: [[2403.17217]] DiffusionAct: Controllable Diffusion Autoencoder for One-shot Face Reenactment(https://arxiv.org/abs/2403.17217)
• Keywords: diffusion
• Abstract: Video-driven neural face reenactment aims to synthesize realistic facial images that successfully preserve the identity and appearance of a source face, while transferring the target head pose and facial expressions. Existing GAN-based methods suffer from either distortions and visual artifacts or poor reconstruction quality, i.e., the background and several important appearance details, such as hair style/color, glasses and accessories, are not faithfully reconstructed. Recent advances in Diffusion Probabilistic Models (DPMs) enable the generation of high-quality realistic images. To this end, in this paper we present DiffusionAct, a novel method that leverages the photo-realistic image generation of diffusion models to perform neural face reenactment. Specifically, we propose to control the semantic space of a Diffusion Autoencoder (DiffAE), in order to edit the facial pose of the input images, defined as the head pose orientation and the facial expressions. Our method allows one-shot, self, and cross-subject reenactment, without requiring subject-specific fine-tuning. We compare against state-of-the-art GAN-, StyleGAN2-, and diffusion-based methods, showing better or on-par reenactment performance.

Title: Making Sentence Embeddings Robust to User-Generated Content

• Authors: Lydia Nishimwe, Benoît Sagot, Rachel Bawden
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.17220
• Pdf URL: https://arxiv.org/pdf/2403.17220
• Copy Paste: [[2403.17220]] Making Sentence Embeddings Robust to User-Generated Content(https://arxiv.org/abs/2403.17220)
• Keywords: robust
• Abstract: NLP models have been known to perform poorly on user-generated content (UGC), mainly because it presents a lot of lexical variations and deviates from the standard texts on which most of these models were trained. In this work, we focus on the robustness of LASER, a sentence embedding model, to UGC data. We evaluate this robustness by LASER's ability to represent non-standard sentences and their standard counterparts close to each other in the embedding space. Inspired by previous works extending LASER to other languages and modalities, we propose RoLASER, a robust English encoder trained using a teacher-student approach to reduce the distances between the representations of standard and UGC sentences. We show that with training only on standard and synthetic UGC-like data, RoLASER significantly improves LASER's robustness to both natural and artificial UGC data by achieving up to 2x and 11x better scores. We also perform a fine-grained analysis on artificial UGC data and find that our model greatly outperforms LASER on its most challenging UGC phenomena such as keyboard typos and social media abbreviations. Evaluation on downstream tasks shows that RoLASER performs comparably to or better than LASER on standard data, while consistently outperforming it on UGC data.

Title: DreamPolisher: Towards High-Quality Text-to-3D Generation via Geometric Diffusion

• Authors: Yuanze Lin, Ronald Clark, Philip Torr
• Subjects: cs.CV, cs.AI, cs.GR
• Abstract URL: https://arxiv.org/abs/2403.17237
• Pdf URL: https://arxiv.org/pdf/2403.17237
• Copy Paste: [[2403.17237]] DreamPolisher: Towards High-Quality Text-to-3D Generation via Geometric Diffusion(https://arxiv.org/abs/2403.17237)
• Keywords: diffusion
• Abstract: We present DreamPolisher, a novel Gaussian Splatting based method with geometric guidance, tailored to learn cross-view consistency and intricate detail from textual descriptions. While recent progress on text-to-3D generation methods have been promising, prevailing methods often fail to ensure view-consistency and textural richness. This problem becomes particularly noticeable for methods that work with text input alone. To address this, we propose a two-stage Gaussian Splatting based approach that enforces geometric consistency among views. Initially, a coarse 3D generation undergoes refinement via geometric optimization. Subsequently, we use a ControlNet driven refiner coupled with the geometric consistency term to improve both texture fidelity and overall consistency of the generated 3D asset. Empirical evaluations across diverse textual prompts spanning various object categories demonstrate the efficacy of DreamPolisher in generating consistent and realistic 3D objects, aligning closely with the semantics of the textual instructions.

Title: Manufacturing Service Capability Prediction with Graph Neural Networks

• Authors: Yunqing Li, Xiaorui Liu, Binil Starly
• Subjects: cs.LG, cs.SI
• Abstract URL: https://arxiv.org/abs/2403.17239
• Pdf URL: https://arxiv.org/pdf/2403.17239
• Copy Paste: [[2403.17239]] Manufacturing Service Capability Prediction with Graph Neural Networks(https://arxiv.org/abs/2403.17239)
• Keywords: robust
• Abstract: In the current landscape, the predominant methods for identifying manufacturing capabilities from manufacturers rely heavily on keyword matching and semantic matching. However, these methods often fall short by either overlooking valuable hidden information or misinterpreting critical data. Consequently, such approaches result in an incomplete identification of manufacturers' capabilities. This underscores the pressing need for data-driven solutions to enhance the accuracy and completeness of manufacturing capability identification. To address the need, this study proposes a Graph Neural Network-based method for manufacturing service capability identification over a knowledge graph. To enhance the identification performance, this work introduces a novel approach that involves aggregating information from the graph nodes' neighborhoods as well as oversampling the graph data, which can be effectively applied across a wide range of practical scenarios. Evaluations conducted on a Manufacturing Service Knowledge Graph and subsequent ablation studies demonstrate the efficacy and robustness of the proposed approach. This study not only contributes a innovative method for inferring manufacturing service capabilities but also significantly augments the quality of Manufacturing Service Knowledge Graphs.

Title: A Hybrid Approach To Aspect Based Sentiment Analysis Using Transfer Learning

• Authors: Gaurav Negi, Rajdeep Sarkar, Omnia Zayed, Paul Buitelaar
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.17254
• Pdf URL: https://arxiv.org/pdf/2403.17254
• Copy Paste: [[2403.17254]] A Hybrid Approach To Aspect Based Sentiment Analysis Using Transfer Learning(https://arxiv.org/abs/2403.17254)
• Keywords: extraction, large language model
• Abstract: Aspect-Based Sentiment Analysis (ABSA) aims to identify terms or multiword expressions (MWEs) on which sentiments are expressed and the sentiment polarities associated with them. The development of supervised models has been at the forefront of research in this area. However, training these models requires the availability of manually annotated datasets which is both expensive and time-consuming. Furthermore, the available annotated datasets are tailored to a specific domain, language, and text type. In this work, we address this notable challenge in current state-of-the-art ABSA research. We propose a hybrid approach for Aspect Based Sentiment Analysis using transfer learning. The approach focuses on generating weakly-supervised annotations by exploiting the strengths of both large language models (LLM) and traditional syntactic dependencies. We utilise syntactic dependency structures of sentences to complement the annotations generated by LLMs, as they may overlook domain-specific aspect terms. Extensive experimentation on multiple datasets is performed to demonstrate the efficacy of our hybrid method for the tasks of aspect term extraction and aspect sentiment classification. Keywords: Aspect Based Sentiment Analysis, Syntactic Parsing, large language model (LLM)

Title: Diffusion-based Negative Sampling on Graphs for Link Prediction

• Authors: Trung-Kien Nguyen, Yuan Fang
• Subjects: cs.LG, cs.SI
• Abstract URL: https://arxiv.org/abs/2403.17259
• Pdf URL: https://arxiv.org/pdf/2403.17259
• Copy Paste: [[2403.17259]] Diffusion-based Negative Sampling on Graphs for Link Prediction(https://arxiv.org/abs/2403.17259)
• Keywords: robust, diffusion
• Abstract: Link prediction is a fundamental task for graph analysis with important applications on the Web, such as social network analysis and recommendation systems, etc. Modern graph link prediction methods often employ a contrastive approach to learn robust node representations, where negative sampling is pivotal. Typical negative sampling methods aim to retrieve hard examples based on either predefined heuristics or automatic adversarial approaches, which might be inflexible or difficult to control. Furthermore, in the context of link prediction, most previous methods sample negative nodes from existing substructures of the graph, missing out on potentially more optimal samples in the latent space. To address these issues, we investigate a novel strategy of multi-level negative sampling that enables negative node generation with flexible and controllable ``hardness'' levels from the latent space. Our method, called Conditional Diffusion-based Multi-level Negative Sampling (DMNS), leverages the Markov chain property of diffusion models to generate negative nodes in multiple levels of variable hardness and reconcile them for effective graph link prediction. We further demonstrate that DMNS follows the sub-linear positivity principle for robust negative sampling. Extensive experiments on several benchmark datasets demonstrate the effectiveness of DMNS.

Title: Automate Knowledge Concept Tagging on Math Questions with LLMs

• Authors: Hang Li, Tianlong Xu, Jiliang Tang, Qingsong Wen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.17281
• Pdf URL: https://arxiv.org/pdf/2403.17281
• Copy Paste: [[2403.17281]] Automate Knowledge Concept Tagging on Math Questions with LLMs(https://arxiv.org/abs/2403.17281)
• Keywords: large language model
• Abstract: Knowledge concept tagging for questions plays a crucial role in contemporary intelligent educational applications, including learning progress diagnosis, practice question recommendations, and course content organization. Traditionally, these annotations have been conducted manually with help from pedagogical experts, as the task requires not only a strong semantic understanding of both question stems and knowledge definitions but also deep insights into connecting question-solving logic with corresponding knowledge concepts. In this paper, we explore automating the tagging task using Large Language Models (LLMs), in response to the inability of prior manual methods to meet the rapidly growing demand for concept tagging in questions posed by advanced educational applications. Moreover, the zero/few-shot learning capability of LLMs makes them well-suited for application in educational scenarios, which often face challenges in collecting large-scale, expertise-annotated datasets. By conducting extensive experiments with a variety of representative LLMs, we demonstrate that LLMs are a promising tool for concept tagging in math questions. Furthermore, through case studies examining the results from different LLMs, we draw some empirical conclusions about the key factors for success in applying LLMs to the automatic concept tagging task.

Title: Not All Federated Learning Algorithms Are Created Equal: A Performance Evaluation Study

• Authors: Gustav A. Baumgart, Jaemin Shin, Ali Payani, Myungjin Lee, Ramana Rao Kompella
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2403.17287
• Pdf URL: https://arxiv.org/pdf/2403.17287
• Copy Paste: [[2403.17287]] Not All Federated Learning Algorithms Are Created Equal: A Performance Evaluation Study(https://arxiv.org/abs/2403.17287)
• Keywords: federate
• Abstract: Federated Learning (FL) emerged as a practical approach to training a model from decentralized data. The proliferation of FL led to the development of numerous FL algorithms and mechanisms. Many prior efforts have given their primary focus on accuracy of those approaches, but there exists little understanding of other aspects such as computational overheads, performance and training stability, etc. To bridge this gap, we conduct extensive performance evaluation on several canonical FL algorithms (FedAvg, FedProx, FedYogi, FedAdam, SCAFFOLD, and FedDyn) by leveraging an open-source federated learning framework called Flame. Our comprehensive measurement study reveals that no single algorithm works best across different performance metrics. A few key observations are: (1) While some state-of-the-art algorithms achieve higher accuracy than others, they incur either higher computation overheads (FedDyn) or communication overheads (SCAFFOLD). (2) Recent algorithms present smaller standard deviation in accuracy across clients than FedAvg, indicating that the advanced algorithms' performances are stable. (3) However, algorithms such as FedDyn and SCAFFOLD are more prone to catastrophic failures without the support of additional techniques such as gradient clipping. We hope that our empirical study can help the community to build best practices in evaluating FL algorithms.

Title: Hawk: Accurate and Fast Privacy-Preserving Machine Learning Using Secure Lookup Table Computation

• Authors: Hamza Saleem, Amir Ziashahabi, Muhammad Naveed, Salman Avestimehr
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.17296
• Pdf URL: https://arxiv.org/pdf/2403.17296
• Copy Paste: [[2403.17296]] Hawk: Accurate and Fast Privacy-Preserving Machine Learning Using Secure Lookup Table Computation(https://arxiv.org/abs/2403.17296)
• Keywords: secure, security, privacy
• Abstract: Training machine learning models on data from multiple entities without direct data sharing can unlock applications otherwise hindered by business, legal, or ethical constraints. In this work, we design and implement new privacy-preserving machine learning protocols for logistic regression and neural network models. We adopt a two-server model where data owners secret-share their data between two servers that train and evaluate the model on the joint data. A significant source of inefficiency and inaccuracy in existing methods arises from using Yao's garbled circuits to compute non-linear activation functions. We propose new methods for computing non-linear functions based on secret-shared lookup tables, offering both computational efficiency and improved accuracy. Beyond introducing leakage-free techniques, we initiate the exploration of relaxed security measures for privacy-preserving machine learning. Instead of claiming that the servers gain no knowledge during the computation, we contend that while some information is revealed about access patterns to lookup tables, it maintains epsilon-dX-privacy. Leveraging this relaxation significantly reduces the computational resources needed for training. We present new cryptographic protocols tailored to this relaxed security paradigm and define and analyze the leakage. Our evaluations show that our logistic regression protocol is up to 9x faster, and the neural network training is up to 688x faster than SecureML. Notably, our neural network achieves an accuracy of 96.6% on MNIST in 15 epochs, outperforming prior benchmarks that capped at 93.4% using the same architecture.

Title: InternLM2 Technical Report

• Authors: Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong, Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye Ge, Chenya Gu, Yuzhe Gu, Tao Gui, Aijia Guo, Qipeng Guo, Conghui He, Yingfan Hu, Ting Huang, Tao Jiang, Penglong Jiao, Zhenjiang Jin, Zhikai Lei, Jiaxing Li, Jingwen Li, Linyang Li, Shuaibin Li, Wei Li, Yining Li, Hongwei Liu, Jiangning Liu, Jiawei Hong, Kaiwen Liu, Kuikun Liu, Xiaoran Liu, Chengqi Lv, Haijun Lv, Kai Lv, Li Ma, Runyuan Ma, Zerun Ma, Wenchang Ning, Linke Ouyang, Jiantao Qiu, Yuan Qu, Fukai Shang, Yunfan Shao, Demin Song, Zifan Song, Zhihao Sui, Peng Sun, Yu Sun, Huanze Tang, Bin Wang, Guoteng Wang, Jiaqi Wang, Jiayu Wang, Rui Wang, Yudong Wang, Ziyi Wang, Xingjian Wei, Qizhen Weng, Fan Wu, Yingtong Xiong, et al. (31 additional authors not shown)
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.17297
• Pdf URL: https://arxiv.org/pdf/2403.17297
• Copy Paste: [[2403.17297]] InternLM2 Technical Report(https://arxiv.org/abs/2403.17297)
• Keywords: large language model
• Abstract: The evolution of Large Language Models (LLMs) like ChatGPT and GPT-4 has sparked discussions on the advent of Artificial General Intelligence (AGI). However, replicating such advancements in open-source models has been challenging. This paper introduces InternLM2, an open-source LLM that outperforms its predecessors in comprehensive evaluations across 6 dimensions and 30 benchmarks, long-context modeling, and open-ended subjective evaluations through innovative pre-training and optimization techniques. The pre-training process of InternLM2 is meticulously detailed, highlighting the preparation of diverse data types including text, code, and long-context data. InternLM2 efficiently captures long-term dependencies, initially trained on 4k tokens before advancing to 32k tokens in pre-training and fine-tuning stages, exhibiting remarkable performance on the 200k ``Needle-in-a-Haystack" test. InternLM2 is further aligned using Supervised Fine-Tuning (SFT) and a novel Conditional Online Reinforcement Learning from Human Feedback (COOL RLHF) strategy that addresses conflicting human preferences and reward hacking. By releasing InternLM2 models in different training stages and model sizes, we provide the community with insights into the model's evolution.

Title: Decoding Probing: Revealing Internal Linguistic Structures in Neural Language Models using Minimal Pairs

• Authors: Linyang He, Peili Chen, Ercong Nie, Yuanning Li, Jonathan R. Brennan
• Subjects: cs.CL, q-bio.NC
• Abstract URL: https://arxiv.org/abs/2403.17299
• Pdf URL: https://arxiv.org/pdf/2403.17299
• Copy Paste: [[2403.17299]] Decoding Probing: Revealing Internal Linguistic Structures in Neural Language Models using Minimal Pairs(https://arxiv.org/abs/2403.17299)
• Keywords: robust, transformer
• Abstract: Inspired by cognitive neuroscience studies, we introduce a novel `decoding probing' method that uses minimal pairs benchmark (BLiMP) to probe internal linguistic characteristics in neural language models layer by layer. By treating the language model as the `brain' and its representations as `neural activations', we decode grammaticality labels of minimal pairs from the intermediate layers' representations. This approach reveals: 1) Self-supervised language models capture abstract linguistic structures in intermediate layers that GloVe and RNN language models cannot learn. 2) Information about syntactic grammaticality is robustly captured through the first third layers of GPT-2 and also distributed in later layers. As sentence complexity increases, more layers are required for learning grammatical capabilities. 3) Morphological and semantics/syntax interface-related features are harder to capture than syntax. 4) For Transformer-based models, both embeddings and attentions capture grammatical features but show distinct patterns. Different attention heads exhibit similar tendencies toward various linguistic phenomena, but with varied contributions.

Title: Physical 3D Adversarial Attacks against Monocular Depth Estimation in Autonomous Driving

• Authors: Junhao Zheng, Chenhao Lin, Jiahao Sun, Zhengyu Zhao, Qian Li, Chao Shen
• Subjects: cs.CV, cs.CR
• Abstract URL: https://arxiv.org/abs/2403.17301
• Pdf URL: https://arxiv.org/pdf/2403.17301
• Copy Paste: [[2403.17301]] Physical 3D Adversarial Attacks against Monocular Depth Estimation in Autonomous Driving(https://arxiv.org/abs/2403.17301)
• Keywords: attack, robust
• Abstract: Deep learning-based monocular depth estimation (MDE), extensively applied in autonomous driving, is known to be vulnerable to adversarial attacks. Previous physical attacks against MDE models rely on 2D adversarial patches, so they only affect a small, localized region in the MDE map but fail under various viewpoints. To address these limitations, we propose 3D Depth Fool (3D$^2$Fool), the first 3D texture-based adversarial attack against MDE models. 3D$^2$Fool is specifically optimized to generate 3D adversarial textures agnostic to model types of vehicles and to have improved robustness in bad weather conditions, such as rain and fog. Experimental results validate the superior performance of our 3D$^2$Fool across various scenarios, including vehicles, MDE models, weather conditions, and viewpoints. Real-world experiments with printed 3D textures on physical vehicle models further demonstrate that our 3D$^2$Fool can cause an MDE error of over 10 meters.

Title: Two Birds with One Stone: Differential Privacy by Low-power SRAM Memory

• Authors: Jianqing Liu, Na Gong, Hritom Das
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.17303
• Pdf URL: https://arxiv.org/pdf/2403.17303
• Copy Paste: [[2403.17303]] Two Birds with One Stone: Differential Privacy by Low-power SRAM Memory(https://arxiv.org/abs/2403.17303)
• Keywords: secure, privacy, attack
• Abstract: The software-based implementation of differential privacy mechanisms has been shown to be neither friendly for lightweight devices nor secure against side-channel attacks. In this work, we aim to develop a hardware-based technique to achieve differential privacy by design. In contrary to the conventional software-based noise generation and injection process, our design realizes local differential privacy (LDP) by harnessing the inherent hardware noise into controlled LDP noise when data is stored in the memory. Specifically, the noise is tamed through a novel memory design and power downscaling technique, which leads to double-faceted gains in privacy and power efficiency. A well-round study that consists of theoretical design and analysis and chip implementation and experiments is presented. The results confirm that the developed technique is differentially private, saves 88.58% system power, speeds up software-based DP mechanisms by more than 10^6 times, while only incurring 2.46% chip overhead and 7.81% estimation errors in data recovery.

Title: JMultiWOZ: A Large-Scale Japanese Multi-Domain Task-Oriented Dialogue Dataset

• Authors: Atsumoto Ohashi, Ryu Hirai, Shinya Iizuka, Ryuichiro Higashinaka
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.17319
• Pdf URL: https://arxiv.org/pdf/2403.17319
• Copy Paste: [[2403.17319]] JMultiWOZ: A Large-Scale Japanese Multi-Domain Task-Oriented Dialogue Dataset(https://arxiv.org/abs/2403.17319)
• Keywords: large language model
• Abstract: Dialogue datasets are crucial for deep learning-based task-oriented dialogue system research. While numerous English language multi-domain task-oriented dialogue datasets have been developed and contributed to significant advancements in task-oriented dialogue systems, such a dataset does not exist in Japanese, and research in this area is limited compared to that in English. In this study, towards the advancement of research and development of task-oriented dialogue systems in Japanese, we constructed JMultiWOZ, the first Japanese language large-scale multi-domain task-oriented dialogue dataset. Using JMultiWOZ, we evaluated the dialogue state tracking and response generation capabilities of the state-of-the-art methods on the existing major English benchmark dataset MultiWOZ2.2 and the latest large language model (LLM)-based methods. Our evaluation results demonstrated that JMultiWOZ provides a benchmark that is on par with MultiWOZ2.2. In addition, through evaluation experiments of interactive dialogues with the models and human participants, we identified limitations in the task completion capabilities of LLMs in Japanese.

Title: Don't Listen To Me: Understanding and Exploring Jailbreak Prompts of Large Language Models

• Authors: Zhiyuan Yu, Xiaogeng Liu, Shunning Liang, Zach Cameron, Chaowei Xiao, Ning Zhang
• Subjects: cs.CR, cs.CL
• Abstract URL: https://arxiv.org/abs/2403.17336
• Pdf URL: https://arxiv.org/pdf/2403.17336
• Copy Paste: [[2403.17336]] Don't Listen To Me: Understanding and Exploring Jailbreak Prompts of Large Language Models(https://arxiv.org/abs/2403.17336)
• Keywords: security, protect, generative, large language model
• Abstract: Recent advancements in generative AI have enabled ubiquitous access to large language models (LLMs). Empowered by their exceptional capabilities to understand and generate human-like text, these models are being increasingly integrated into our society. At the same time, there are also concerns on the potential misuse of this powerful technology, prompting defensive measures from service providers. To overcome such protection, jailbreaking prompts have recently emerged as one of the most effective mechanisms to circumvent security restrictions and elicit harmful content originally designed to be prohibited. Due to the rapid development of LLMs and their ease of access via natural languages, the frontline of jailbreak prompts is largely seen in online forums and among hobbyists. To gain a better understanding of the threat landscape of semantically meaningful jailbreak prompts, we systemized existing prompts and measured their jailbreak effectiveness empirically. Further, we conducted a user study involving 92 participants with diverse backgrounds to unveil the process of manually creating jailbreak prompts. We observed that users often succeeded in jailbreak prompts generation regardless of their expertise in LLMs. Building on the insights from the user study, we also developed a system using AI as the assistant to automate the process of jailbreak prompt generation.

Title: Language Models are Free Boosters for Biomedical Imaging Tasks

• Authors: Zhixin Lai, Jing Wu, Suiyao Chen, Yucheng Zhou, Anna Hovakimyan, Naira Hovakimyan
• Subjects: cs.CV, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.17343
• Pdf URL: https://arxiv.org/pdf/2403.17343
• Copy Paste: [[2403.17343]] Language Models are Free Boosters for Biomedical Imaging Tasks(https://arxiv.org/abs/2403.17343)
• Keywords: transformer, large language model
• Abstract: In this study, we uncover the unexpected efficacy of residual-based large language models (LLMs) as part of encoders for biomedical imaging tasks, a domain traditionally devoid of language or textual data. The approach diverges from established methodologies by utilizing a frozen transformer block, extracted from pre-trained LLMs, as an innovative encoder layer for the direct processing of visual tokens. This strategy represents a significant departure from the standard multi-modal vision-language frameworks, which typically hinge on language-driven prompts and inputs. We found that these LLMs could boost performance across a spectrum of biomedical imaging applications, including both 2D and 3D visual classification tasks, serving as plug-and-play boosters. More interestingly, as a byproduct, we found that the proposed framework achieved superior performance, setting new state-of-the-art results on extensive, standardized datasets in MedMNIST-2D and 3D. Through this work, we aim to open new avenues for employing LLMs in biomedical imaging and enriching the understanding of their potential in this specialized domain.

Title: TRAM: Global Trajectory and Motion of 3D Humans from in-the-wild Videos

• Authors: Yufu Wang, Ziyun Wang, Lingjie Liu, Kostas Daniilidis
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.17346
• Pdf URL: https://arxiv.org/pdf/2403.17346
• Copy Paste: [[2403.17346]] TRAM: Global Trajectory and Motion of 3D Humans from in-the-wild Videos(https://arxiv.org/abs/2403.17346)
• Keywords: robust, transformer
• Abstract: We propose TRAM, a two-stage method to reconstruct a human's global trajectory and motion from in-the-wild videos. TRAM robustifies SLAM to recover the camera motion in the presence of dynamic humans and uses the scene background to derive the motion scale. Using the recovered camera as a metric-scale reference frame, we introduce a video transformer model (VIMO) to regress the kinematic body motion of a human. By composing the two motions, we achieve accurate recovery of 3D humans in the world space, reducing global motion errors by 60% from prior work. https://yufu-wang.github.io/tram4d/

Title: Chain-of-Action: Faithful and Multimodal Question Answering through Large Language Models

• Authors: Zhenyu Pan, Haozheng Luo, Manling Li, Han Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.17359
• Pdf URL: https://arxiv.org/pdf/2403.17359
• Copy Paste: [[2403.17359]] Chain-of-Action: Faithful and Multimodal Question Answering through Large Language Models(https://arxiv.org/abs/2403.17359)
• Keywords: large language model
• Abstract: We present a Chain-of-Action (CoA) framework for multimodal and retrieval-augmented Question-Answering (QA). Compared to the literature, CoA overcomes two major challenges of current QA applications: (i) unfaithful hallucination that is inconsistent with real-time or domain facts and (ii) weak reasoning performance over compositional information. Our key contribution is a novel reasoning-retrieval mechanism that decomposes a complex question into a reasoning chain via systematic prompting and pre-designed actions. Methodologically, we propose three types of domain-adaptable `Plug-and-Play' actions for retrieving real-time information from heterogeneous sources. We also propose a multi-reference faith score (MRFS) to verify and resolve conflicts in the answers. Empirically, we exploit both public benchmarks and a Web3 case study to demonstrate the capability of CoA over other methods.

Title: Activity-Biometrics: Person Identification from Daily Activities

• Authors: Shehreen Azad, Yogesh Singh Rawat
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.17360
• Pdf URL: https://arxiv.org/pdf/2403.17360
• Copy Paste: [[2403.17360]] Activity-Biometrics: Person Identification from Daily Activities(https://arxiv.org/abs/2403.17360)
• Keywords: biometric
• Abstract: In this work, we study a novel problem which focuses on person identification while performing daily activities. Learning biometric features from RGB videos is challenging due to spatio-temporal complexity and presence of appearance biases such as clothing color and background. We propose ABNet, a novel framework which leverages disentanglement of biometric and non-biometric features to perform effective person identification from daily activities. ABNet relies on a bias-less teacher to learn biometric features from RGB videos and explicitly disentangle non-biometric features with the help of biometric distortion. In addition, ABNet also exploits activity prior for biometrics which is enabled by joint biometric and activity learning. We perform comprehensive evaluation of the proposed approach across five different datasets which are derived from existing activity recognition benchmarks. Furthermore, we extensively compare ABNet with existing works in person identification and demonstrate its effectiveness for activity-based biometrics across all five datasets. The code and dataset can be accessed at: \url{https://github.com/sacrcv/Activity-Biometrics/}

Title: Bridging Textual and Tabular Worlds for Fact Verification: A Lightweight, Attention-Based Model

• Authors: Shirin Dabbaghi Varnosfaderani, Canasai Kruengkrai, Ramin Yahyapour, Junichi Yamagishi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.17361
• Pdf URL: https://arxiv.org/pdf/2403.17361
• Copy Paste: [[2403.17361]] Bridging Textual and Tabular Worlds for Fact Verification: A Lightweight, Attention-Based Model(https://arxiv.org/abs/2403.17361)
• Keywords: extraction
• Abstract: FEVEROUS is a benchmark and research initiative focused on fact extraction and verification tasks involving unstructured text and structured tabular data. In FEVEROUS, existing works often rely on extensive preprocessing and utilize rule-based transformations of data, leading to potential context loss or misleading encodings. This paper introduces a simple yet powerful model that nullifies the need for modality conversion, thereby preserving the original evidence's context. By leveraging pre-trained models on diverse text and tabular datasets and by incorporating a lightweight attention-based mechanism, our approach efficiently exploits latent connections between different data types, thereby yielding comprehensive and reliable verdict predictions. The model's modular structure adeptly manages multi-modal information, ensuring the integrity and authenticity of the original evidence are uncompromised. Comparative analyses reveal that our approach exhibits competitive performance, aligning itself closely with top-tier models on the FEVEROUS benchmark.

Title: ChatGPT Rates Natural Language Explanation Quality Like Humans: But on Which Scales?

• Authors: Fan Huang, Haewoon Kwak, Kunwoo Park, Jisun An
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.17368
• Pdf URL: https://arxiv.org/pdf/2403.17368
• Copy Paste: [[2403.17368]] ChatGPT Rates Natural Language Explanation Quality Like Humans: But on Which Scales?(https://arxiv.org/abs/2403.17368)
• Keywords: large language model
• Abstract: As AI becomes more integral in our lives, the need for transparency and responsibility grows. While natural language explanations (NLEs) are vital for clarifying the reasoning behind AI decisions, evaluating them through human judgments is complex and resource-intensive due to subjectivity and the need for fine-grained ratings. This study explores the alignment between ChatGPT and human assessments across multiple scales (i.e., binary, ternary, and 7-Likert scale). We sample 300 data instances from three NLE datasets and collect 900 human annotations for both informativeness and clarity scores as the text quality measurement. We further conduct paired comparison experiments under different ranges of subjectivity scores, where the baseline comes from 8,346 human annotations. Our results show that ChatGPT aligns better with humans in more coarse-grained scales. Also, paired comparisons and dynamic prompting (i.e., providing semantically similar examples in the prompt) improve the alignment. This research advances our understanding of large language models' capabilities to assess the text explanation quality in different configurations for responsible AI development.

Title: AIDE: An Automatic Data Engine for Object Detection in Autonomous Driving

• Authors: Mingfu Liang, Jong-Chyi Su, Samuel Schulter, Sparsh Garg, Shiyu Zhao, Ying Wu, Manmohan Chandraker
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.17373
• Pdf URL: https://arxiv.org/pdf/2403.17373
• Copy Paste: [[2403.17373]] AIDE: An Automatic Data Engine for Object Detection in Autonomous Driving(https://arxiv.org/abs/2403.17373)
• Keywords: robust, large language model
• Abstract: Autonomous vehicle (AV) systems rely on robust perception models as a cornerstone of safety assurance. However, objects encountered on the road exhibit a long-tailed distribution, with rare or unseen categories posing challenges to a deployed perception model. This necessitates an expensive process of continuously curating and annotating data with significant human effort. We propose to leverage recent advances in vision-language and large language models to design an Automatic Data Engine (AIDE) that automatically identifies issues, efficiently curates data, improves the model through auto-labeling, and verifies the model through generation of diverse scenarios. This process operates iteratively, allowing for continuous self-improvement of the model. We further establish a benchmark for open-world detection on AV datasets to comprehensively evaluate various learning paradigms, demonstrating our method's superior performance at a reduced cost.

Title: Self-Rectifying Diffusion Sampling with Perturbed-Attention Guidance

• Authors: Donghoon Ahn, Hyoungwon Cho, Jaewon Min, Wooseok Jang, Jungwoo Kim, SeonHwa Kim, Hyun Hee Park, Kyong Hwan Jin, Seungryong Kim
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.17377
• Pdf URL: https://arxiv.org/pdf/2403.17377
• Copy Paste: [[2403.17377]] Self-Rectifying Diffusion Sampling with Perturbed-Attention Guidance(https://arxiv.org/abs/2403.17377)
• Keywords: diffusion
• Abstract: Recent studies have demonstrated that diffusion models are capable of generating high-quality samples, but their quality heavily depends on sampling guidance techniques, such as classifier guidance (CG) and classifier-free guidance (CFG). These techniques are often not applicable in unconditional generation or in various downstream tasks such as image restoration. In this paper, we propose a novel sampling guidance, called Perturbed-Attention Guidance (PAG), which improves diffusion sample quality across both unconditional and conditional settings, achieving this without requiring additional training or the integration of external modules. PAG is designed to progressively enhance the structure of samples throughout the denoising process. It involves generating intermediate samples with degraded structure by substituting selected self-attention maps in diffusion U-Net with an identity matrix, by considering the self-attention mechanisms' ability to capture structural information, and guiding the denoising process away from these degraded samples. In both ADM and Stable Diffusion, PAG surprisingly improves sample quality in conditional and even unconditional scenarios. Moreover, PAG significantly improves the baseline performance in various downstream tasks where existing guidances such as CG or CFG cannot be fully utilized, including ControlNet with empty prompts and image restoration such as inpainting and deblurring.

Title: Transcribing Bengali Text with Regional Dialects to IPA using District Guided Tokens

• Authors: S M Jishanul Islam, Sadia Ahmmed, Sahid Hossain Mustakim
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.17407
• Pdf URL: https://arxiv.org/pdf/2403.17407
• Copy Paste: [[2403.17407]] Transcribing Bengali Text with Regional Dialects to IPA using District Guided Tokens(https://arxiv.org/abs/2403.17407)
• Keywords: transformer
• Abstract: Accurate transcription of Bengali text to the International Phonetic Alphabet (IPA) is a challenging task due to the complex phonology of the language and context-dependent sound changes. This challenge is even more for regional Bengali dialects due to unavailability of standardized spelling conventions for these dialects, presence of local and foreign words popular in those regions and phonological diversity across different regions. This paper presents an approach to this sequence-to-sequence problem by introducing the District Guided Tokens (DGT) technique on a new dataset spanning six districts of Bangladesh. The key idea is to provide the model with explicit information about the regional dialect or "district" of the input text before generating the IPA transcription. This is achieved by prepending a district token to the input sequence, effectively guiding the model to understand the unique phonetic patterns associated with each district. The DGT technique is applied to fine-tune several transformer-based models, on this new dataset. Experimental results demonstrate the effectiveness of DGT, with the ByT5 model achieving superior performance over word-based models like mT5, BanglaT5, and umT5. This is attributed to ByT5's ability to handle a high percentage of out-of-vocabulary words in the test set. The proposed approach highlights the importance of incorporating regional dialect information into ubiquitous natural language processing systems for languages with diverse phonological variations. The following work was a result of the "Bhashamul" challenge, which is dedicated to solving the problem of Bengali text with regional dialects to IPA transcription https://www.kaggle.com/competitions/regipa/. The training and inference notebooks are available through the competition link.

Title: Neural Clustering based Visual Representation Learning

• Authors: Guikun Chen, Xia Li, Yi Yang, Wenguan Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.17409
• Pdf URL: https://arxiv.org/pdf/2403.17409
• Copy Paste: [[2403.17409]] Neural Clustering based Visual Representation Learning(https://arxiv.org/abs/2403.17409)
• Keywords: extraction, interpretability
• Abstract: We investigate a fundamental aspect of machine vision: the measurement of features, by revisiting clustering, one of the most classic approaches in machine learning and data analysis. Existing visual feature extractors, including ConvNets, ViTs, and MLPs, represent an image as rectangular regions. Though prevalent, such a grid-style paradigm is built upon engineering practice and lacks explicit modeling of data distribution. In this work, we propose feature extraction with clustering (FEC), a conceptually elegant yet surprisingly ad-hoc interpretable neural clustering framework, which views feature extraction as a process of selecting representatives from data and thus automatically captures the underlying data distribution. Given an image, FEC alternates between grouping pixels into individual clusters to abstract representatives and updating the deep features of pixels with current representatives. Such an iterative working mechanism is implemented in the form of several neural layers and the final representatives can be used for downstream tasks. The cluster assignments across layers, which can be viewed and inspected by humans, make the forward process of FEC fully transparent and empower it with promising ad-hoc interpretability. Extensive experiments on various visual recognition models and tasks verify the effectiveness, generality, and interpretability of FEC. We expect this work will provoke a rethink of the current de facto grid-style paradigm.

Title: On permutation-invariant neural networks

• Authors: Masanari Kimura, Ryotaro Shimizu, Yuki Hirakawa, Ryosuke Goto, Yuki Saito
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2403.17410
• Pdf URL: https://arxiv.org/pdf/2403.17410
• Copy Paste: [[2403.17410]] On permutation-invariant neural networks(https://arxiv.org/abs/2403.17410)
• Keywords: transformer
• Abstract: Conventional machine learning algorithms have traditionally been designed under the assumption that input data follows a vector-based format, with an emphasis on vector-centric paradigms. However, as the demand for tasks involving set-based inputs has grown, there has been a paradigm shift in the research community towards addressing these challenges. In recent years, the emergence of neural network architectures such as Deep Sets and Transformers has presented a significant advancement in the treatment of set-based data. These architectures are specifically engineered to naturally accommodate sets as input, enabling more effective representation and processing of set structures. Consequently, there has been a surge of research endeavors dedicated to exploring and harnessing the capabilities of these architectures for various tasks involving the approximation of set functions. This comprehensive survey aims to provide an overview of the diverse problem settings and ongoing research efforts pertaining to neural networks that approximate set functions. By delving into the intricacies of these approaches and elucidating the associated challenges, the survey aims to equip readers with a comprehensive understanding of the field. Through this comprehensive perspective, we hope that researchers can gain valuable insights into the potential applications, inherent limitations, and future directions of set-based neural networks. Indeed, from this survey we gain two insights: i) Deep Sets and its variants can be generalized by differences in the aggregation function, and ii) the behavior of Deep Sets is sensitive to the choice of the aggregation function. From these observations, we show that Deep Sets, one of the well-known permutation-invariant neural networks, can be generalized in the sense of a quasi-arithmetic mean.

Title: PCToolkit: A Unified Plug-and-Play Prompt Compression Toolkit of Large Language Models

• Authors: Jinyi Li, Yihuai Lan, Lei Wang, Hao Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.17411
• Pdf URL: https://arxiv.org/pdf/2403.17411
• Copy Paste: [[2403.17411]] PCToolkit: A Unified Plug-and-Play Prompt Compression Toolkit of Large Language Models(https://arxiv.org/abs/2403.17411)
• Keywords: large language model
• Abstract: Prompt compression is an innovative method for efficiently condensing input prompts while preserving essential information. To facilitate quick-start services, user-friendly interfaces, and compatibility with common datasets and metrics, we present the Prompt Compression Toolkit (PCToolkit). This toolkit is a unified plug-and-play solution for compressing prompts in Large Language Models (LLMs), featuring cutting-edge prompt compressors, diverse datasets, and metrics for comprehensive performance evaluation. PCToolkit boasts a modular design, allowing for easy integration of new datasets and metrics through portable and user-friendly interfaces. In this paper, we outline the key components and functionalities of PCToolkit. We conducted evaluations of the compressors within PCToolkit across various natural language tasks, including reconstruction, summarization, mathematical problem-solving, question answering, few-shot learning, synthetic tasks, code completion, boolean expressions, multiple choice questions, and lies recognition.

Title: The Privacy Policy Permission Model: A Unified View of Privacy Policies

• Authors: Maryam Majedi, Ken Barker
• Subjects: cs.CR, cs.CY
• Abstract URL: https://arxiv.org/abs/2403.17414
• Pdf URL: https://arxiv.org/pdf/2403.17414
• Copy Paste: [[2403.17414]] The Privacy Policy Permission Model: A Unified View of Privacy Policies(https://arxiv.org/abs/2403.17414)
• Keywords: privacy
• Abstract: Organizations use privacy policies to communicate their data collection practices to their clients. A privacy policy is a set of statements that specifies how an organization gathers, uses, discloses, and maintains a client's data. However, most privacy policies lack a clear, complete explanation of how data providers' information is used. We propose a modeling methodology, called the Privacy Policy Permission Model (PPPM), that provides a uniform, easy-to-understand representation of privacy policies, which can accurately and clearly show how data is used within an organization's practice. Using this methodology, a privacy policy is captured as a diagram. The diagram is capable of highlighting inconsistencies and inaccuracies in the privacy policy. The methodology supports privacy officers in properly and clearly articulating an organization's privacy policy.

Title: InterHandGen: Two-Hand Interaction Generation via Cascaded Reverse Diffusion

• Authors: Jihyun Lee, Shunsuke Saito, Giljoo Nam, Minhyuk Sung, Tae-Kyun Kim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.17422
• Pdf URL: https://arxiv.org/pdf/2403.17422
• Copy Paste: [[2403.17422]] InterHandGen: Two-Hand Interaction Generation via Cascaded Reverse Diffusion(https://arxiv.org/abs/2403.17422)
• Keywords: diffusion, generative
• Abstract: We present InterHandGen, a novel framework that learns the generative prior of two-hand interaction. Sampling from our model yields plausible and diverse two-hand shapes in close interaction with or without an object. Our prior can be incorporated into any optimization or learning methods to reduce ambiguity in an ill-posed setup. Our key observation is that directly modeling the joint distribution of multiple instances imposes high learning complexity due to its combinatorial nature. Thus, we propose to decompose the modeling of joint distribution into the modeling of factored unconditional and conditional single instance distribution. In particular, we introduce a diffusion model that learns the single-hand distribution unconditional and conditional to another hand via conditioning dropout. For sampling, we combine anti-penetration and classifier-free guidance to enable plausible generation. Furthermore, we establish the rigorous evaluation protocol of two-hand synthesis, where our method significantly outperforms baseline generative models in terms of plausibility and diversity. We also demonstrate that our diffusion prior can boost the performance of two-hand reconstruction from monocular in-the-wild images, achieving new state-of-the-art accuracy.

Title: Robust and Scalable Model Editing for Large Language Models

• Authors: Yingfa Chen, Zhengyan Zhang, Xu Han, Chaojun Xiao, Zhiyuan Liu, Chen Chen, Kuai Li, Tao Yang, Maosong Sun
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.17431
• Pdf URL: https://arxiv.org/pdf/2403.17431
• Copy Paste: [[2403.17431]] Robust and Scalable Model Editing for Large Language Models(https://arxiv.org/abs/2403.17431)
• Keywords: robust, large language model
• Abstract: Large language models (LLMs) can make predictions using parametric knowledge--knowledge encoded in the model weights--or contextual knowledge--knowledge presented in the context. In many scenarios, a desirable behavior is that LLMs give precedence to contextual knowledge when it conflicts with the parametric knowledge, and fall back to using their parametric knowledge when the context is irrelevant. This enables updating and correcting the model's knowledge by in-context editing instead of retraining. Previous works have shown that LLMs are inclined to ignore contextual knowledge and fail to reliably fall back to parametric knowledge when presented with irrelevant context. In this work, we discover that, with proper prompting methods, instruction-finetuned LLMs can be highly controllable by contextual knowledge and robust to irrelevant context. Utilizing this feature, we propose EREN (Edit models by REading Notes) to improve the scalability and robustness of LLM editing. To better evaluate the robustness of model editors, we collect a new dataset, that contains irrelevant questions that are more challenging than the ones in existing datasets. Empirical results show that our method outperforms current state-of-the-art methods by a large margin. Unlike existing techniques, it can integrate knowledge from multiple edits, and correctly respond to syntactically similar but semantically unrelated inputs (and vice versa). The source code can be found at https://github.com/thunlp/EREN.

Title: Expectations Versus Reality: Evaluating Intrusion Detection Systems in Practice

• Authors: Jake Hesford, Daniel Cheng, Alan Wan, Larry Huynh, Seungho Kim, Hyoungshick Kim, Jin B. Hong
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.17458
• Pdf URL: https://arxiv.org/pdf/2403.17458
• Copy Paste: [[2403.17458]] Expectations Versus Reality: Evaluating Intrusion Detection Systems in Practice(https://arxiv.org/abs/2403.17458)
• Keywords: attack
• Abstract: Our paper provides empirical comparisons between recent IDSs to provide an objective comparison between them to help users choose the most appropriate solution based on their requirements. Our results show that no one solution is the best, but is dependent on external variables such as the types of attacks, complexity, and network environment in the dataset. For example, BoT_IoT and Stratosphere IoT datasets both capture IoT-related attacks, but the deep neural network performed the best when tested using the BoT_IoT dataset while HELAD performed the best when tested using the Stratosphere IoT dataset. So although we found that a deep neural network solution had the highest average F1 scores on tested datasets, it is not always the best-performing one. We further discuss difficulties in using IDS from literature and project repositories, which complicated drawing definitive conclusions regarding IDS selection.

Title: LaRE^2: Latent Reconstruction Error Based Method for Diffusion-Generated Image Detection

• Authors: Yunpeng Luo, Junlong Du, Ke Yan, Shouhong Ding
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.17465
• Pdf URL: https://arxiv.org/pdf/2403.17465
• Copy Paste: [[2403.17465]] LaRE^2: Latent Reconstruction Error Based Method for Diffusion-Generated Image Detection(https://arxiv.org/abs/2403.17465)
• Keywords: security, privacy, extraction, diffusion
• Abstract: The evolution of Diffusion Models has dramatically improved image generation quality, making it increasingly difficult to differentiate between real and generated images. This development, while impressive, also raises significant privacy and security concerns. In response to this, we propose a novel Latent REconstruction error guided feature REfinement method (LaRE^2) for detecting the diffusion-generated images. We come up with the Latent Reconstruction Error (LaRE), the first reconstruction-error based feature in the latent space for generated image detection. LaRE surpasses existing methods in terms of feature extraction efficiency while preserving crucial cues required to differentiate between the real and the fake. To exploit LaRE, we propose an Error-Guided feature REfinement module (EGRE), which can refine the image feature guided by LaRE to enhance the discriminativeness of the feature. Our EGRE utilizes an align-then-refine mechanism, which effectively refines the image feature for generated-image detection from both spatial and channel perspectives. Extensive experiments on the large-scale GenImage benchmark demonstrate the superiority of our LaRE^2, which surpasses the best SoTA method by up to 11.9%/12.1% average ACC/AP across 8 different image generators. LaRE also surpasses existing methods in terms of feature extraction cost, delivering an impressive speed enhancement of 8 times.

Title: DiffGaze: A Diffusion Model for Continuous Gaze Sequence Generation on 360° Images

• Authors: Chuhan Jiao, Yao Wang, Guanhua Zhang, Mihai Bâce, Zhiming Hu, Andreas Bulling
• Subjects: cs.CV, cs.HC
• Abstract URL: https://arxiv.org/abs/2403.17477
• Pdf URL: https://arxiv.org/pdf/2403.17477
• Copy Paste: [[2403.17477]] DiffGaze: A Diffusion Model for Continuous Gaze Sequence Generation on 360° Images(https://arxiv.org/abs/2403.17477)
• Keywords: diffusion, transformer
• Abstract: We present DiffGaze, a novel method for generating realistic and diverse continuous human gaze sequences on 360{\deg} images based on a conditional score-based denoising diffusion model. Generating human gaze on 360{\deg} images is important for various human-computer interaction and computer graphics applications, e.g. for creating large-scale eye tracking datasets or for realistic animation of virtual humans. However, existing methods are limited to predicting discrete fixation sequences or aggregated saliency maps, thereby neglecting crucial parts of natural gaze behaviour. Our method uses features extracted from 360{\deg} images as condition and uses two transformers to model the temporal and spatial dependencies of continuous human gaze. We evaluate DiffGaze on two 360{\deg} image benchmarks for gaze sequence generation as well as scanpath prediction and saliency prediction. Our evaluations show that DiffGaze outperforms state-of-the-art methods on all tasks on both benchmarks. We also report a 21-participant user study showing that our method generates gaze sequences that are indistinguishable from real human sequences.

Title: DGoT: Dynamic Graph of Thoughts for Scientific Abstract Generation

• Authors: Xinyu Ning, Yutong Zhao, Yitong Liu, Hongwen Yang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.17491
• Pdf URL: https://arxiv.org/pdf/2403.17491
• Copy Paste: [[2403.17491]] DGoT: Dynamic Graph of Thoughts for Scientific Abstract Generation(https://arxiv.org/abs/2403.17491)
• Keywords: large language model
• Abstract: The method of training language models based on domain datasets has obtained significant achievements in the task of generating scientific paper abstracts. However, such models face problems of generalization and expensive training costs. The use of large language models (LLMs) to solve the task of generating paper abstracts saves the cost of model training. However, due to the hallucination problem of LLM, it is often necessary to improve the reliability of the results through multi-round query prompt approach such as Graph of Thoughts (GoT), which also brings additional reasoning costs. In this paper, we propose a Dynamic Graph of Thought (DGoT). It not only inherits the advantages of the existing GoT prompt approach, but also dynamically adjust the graph structure according to data characteristics while reducing model reasoning cost. Experimental results show that our method's cost-effectiveness in abstract generation tasks is only 43.7% to 56.4% of other multi-round query prompt approaches. Our code is available at https://github.com/JayceNing/DGoT.

Title: FaultGuard: A Generative Approach to Resilient Fault Prediction in Smart Electrical Grids

• Authors: Emad Efatinasab, Francesco Marchiori, Alessandro Brighente, Mirco Rampazzo, Mauro Conti
• Subjects: cs.CR, eess.SP
• Abstract URL: https://arxiv.org/abs/2403.17494
• Pdf URL: https://arxiv.org/pdf/2403.17494
• Copy Paste: [[2403.17494]] FaultGuard: A Generative Approach to Resilient Fault Prediction in Smart Electrical Grids(https://arxiv.org/abs/2403.17494)
• Keywords: security, attack, robust, generative
• Abstract: Predicting and classifying faults in electricity networks is crucial for uninterrupted provision and keeping maintenance costs at a minimum. Thanks to the advancements in the field provided by the smart grid, several data-driven approaches have been proposed in the literature to tackle fault prediction tasks. Implementing these systems brought several improvements, such as optimal energy consumption and quick restoration. Thus, they have become an essential component of the smart grid. However, the robustness and security of these systems against adversarial attacks have not yet been extensively investigated. These attacks can impair the whole grid and cause additional damage to the infrastructure, deceiving fault detection systems and disrupting restoration. In this paper, we present FaultGuard, the first framework for fault type and zone classification resilient to adversarial attacks. To ensure the security of our system, we employ an Anomaly Detection System (ADS) leveraging a novel Generative Adversarial Network training layer to identify attacks. Furthermore, we propose a low-complexity fault prediction model and an online adversarial training technique to enhance robustness. We comprehensively evaluate the framework's performance against various adversarial attacks using the IEEE13-AdvAttack dataset, which constitutes the state-of-the-art for resilient fault prediction benchmarking. Our model outclasses the state-of-the-art even without considering adversaries, with an accuracy of up to 0.958. Furthermore, our ADS shows attack detection capabilities with an accuracy of up to 1.000. Finally, we demonstrate how our novel training layers drastically increase performances across the whole framework, with a mean increase of 154% in ADS accuracy and 118% in model accuracy.

Title: Dr.Hair: Reconstructing Scalp-Connected Hair Strands without Pre-training via Differentiable Rendering of Line Segments

• Authors: Yusuke Takimoto, Hikari Takehara, Hiroyuki Sato, Zihao Zhu, Bo Zheng
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2403.17496
• Pdf URL: https://arxiv.org/pdf/2403.17496
• Copy Paste: [[2403.17496]] Dr.Hair: Reconstructing Scalp-Connected Hair Strands without Pre-training via Differentiable Rendering of Line Segments(https://arxiv.org/abs/2403.17496)
• Keywords: robust
• Abstract: In the film and gaming industries, achieving a realistic hair appearance typically involves the use of strands originating from the scalp. However, reconstructing these strands from observed surface images of hair presents significant challenges. The difficulty in acquiring Ground Truth (GT) data has led state-of-the-art learning-based methods to rely on pre-training with manually prepared synthetic CG data. This process is not only labor-intensive and costly but also introduces complications due to the domain gap when compared to real-world data. In this study, we propose an optimization-based approach that eliminates the need for pre-training. Our method represents hair strands as line segments growing from the scalp and optimizes them using a novel differentiable rendering algorithm. To robustly optimize a substantial number of slender explicit geometries, we introduce 3D orientation estimation utilizing global optimization, strand initialization based on Laplace's equation, and reparameterization that leverages geometric connectivity and spatial proximity. Unlike existing optimization-based methods, our method is capable of reconstructing internal hair flow in an absolute direction. Our method exhibits robust and accurate inverse rendering, surpassing the quality of existing methods and significantly improving processing speed.

Title: Random-coupled Neural Network

• Authors: Haoran Liu, Mingzhe Liu, Peng Li, Jiahui Wu, Xin Jiang, Zhuo Zuo, Bingqi Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.17512
• Pdf URL: https://arxiv.org/pdf/2403.17512
• Copy Paste: [[2403.17512]] Random-coupled Neural Network(https://arxiv.org/abs/2403.17512)
• Keywords: robust, segmentation
• Abstract: Improving the efficiency of current neural networks and modeling them in biological neural systems have become popular research directions in recent years. Pulse-coupled neural network (PCNN) is a well applicated model for imitating the computation characteristics of the human brain in computer vision and neural network fields. However, differences between the PCNN and biological neural systems remain: limited neural connection, high computational cost, and lack of stochastic property. In this study, random-coupled neural network (RCNN) is proposed. It overcomes these difficulties in PCNN's neuromorphic computing via a random inactivation process. This process randomly closes some neural connections in the RCNN model, realized by the random inactivation weight matrix of link input. This releases the computational burden of PCNN, making it affordable to achieve vast neural connections. Furthermore, the image and video processing mechanisms of RCNN are researched. It encodes constant stimuli as periodic spike trains and periodic stimuli as chaotic spike trains, the same as biological neural information encoding characteristics. Finally, the RCNN is applicated to image segmentation, fusion, and pulse shape discrimination subtasks. It is demonstrated to be robust, efficient, and highly anti-noised, with outstanding performance in all applications mentioned above.

Title: Boosting Adversarial Training via Fisher-Rao Norm-based Regularization

• Authors: Xiangyu Yin, Wenjie Ruan
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.17520
• Pdf URL: https://arxiv.org/pdf/2403.17520
• Copy Paste: [[2403.17520]] Boosting Adversarial Training via Fisher-Rao Norm-based Regularization(https://arxiv.org/abs/2403.17520)
• Keywords: robust
• Abstract: Adversarial training is extensively utilized to improve the adversarial robustness of deep neural networks. Yet, mitigating the degradation of standard generalization performance in adversarial-trained models remains an open problem. This paper attempts to resolve this issue through the lens of model complexity. First, We leverage the Fisher-Rao norm, a geometrically invariant metric for model complexity, to establish the non-trivial bounds of the Cross-Entropy Loss-based Rademacher complexity for a ReLU-activated Multi-Layer Perceptron. Then we generalize a complexity-related variable, which is sensitive to the changes in model width and the trade-off factors in adversarial training. Moreover, intensive empirical evidence validates that this variable highly correlates with the generalization gap of Cross-Entropy loss between adversarial-trained and standard-trained models, especially during the initial and final phases of the training process. Building upon this observation, we propose a novel regularization framework, called Logit-Oriented Adversarial Training (LOAT), which can mitigate the trade-off between robustness and accuracy while imposing only a negligible increase in computational overhead. Our extensive experiments demonstrate that the proposed regularization strategy can boost the performance of the prevalent adversarial training algorithms, including PGD-AT, TRADES, TRADES (LSE), MART, and DM-AT, across various network architectures. Our code will be available at https://github.com/TrustAI/LOAT.

Title: Provably Secure Disambiguating Neural Linguistic Steganography

• Authors: Yuang Qi, Kejiang Chen, Kai Zeng, Weiming Zhang, Nenghai Yu
• Subjects: cs.CR, cs.CL
• Abstract URL: https://arxiv.org/abs/2403.17524
• Pdf URL: https://arxiv.org/pdf/2403.17524
• Copy Paste: [[2403.17524]] Provably Secure Disambiguating Neural Linguistic Steganography(https://arxiv.org/abs/2403.17524)
• Keywords: secure, security, segmentation
• Abstract: Recent research in provably secure neural linguistic steganography has overlooked a crucial aspect: the sender must detokenize stegotexts to avoid raising suspicion from the eavesdropper. The segmentation ambiguity problem, which arises when using language models based on subwords, leads to occasional decoding failures in all neural language steganography implementations based on these models. Current solutions to this issue involve altering the probability distribution of candidate words, rendering them incompatible with provably secure steganography. We propose a novel secure disambiguation method named SyncPool, which effectively addresses the segmentation ambiguity problem. We group all tokens with prefix relationships in the candidate pool before the steganographic embedding algorithm runs to eliminate uncertainty among ambiguous tokens. To enable the receiver to synchronize the sampling process of the sender, a shared cryptographically-secure pseudorandom number generator (CSPRNG) is deployed to select a token from the ambiguity pool. SyncPool does not change the size of the candidate pool or the distribution of tokens and thus is applicable to provably secure language steganography methods. We provide theoretical proofs and experimentally demonstrate the applicability of our solution to various languages and models, showing its potential to significantly improve the reliability and security of neural linguistic steganography systems.

Title: Equipping Sketch Patches with Context-Aware Positional Encoding for Graphic Sketch Representation

• Authors: Sicong Zang, Zhijun Fang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.17525
• Pdf URL: https://arxiv.org/pdf/2403.17525
• Copy Paste: [[2403.17525]] Equipping Sketch Patches with Context-Aware Positional Encoding for Graphic Sketch Representation(https://arxiv.org/abs/2403.17525)
• Keywords: protect
• Abstract: The drawing order of a sketch records how it is created stroke-by-stroke by a human being. For graphic sketch representation learning, recent studies have injected sketch drawing orders into graph edge construction by linking each patch to another in accordance to a temporal-based nearest neighboring strategy. However, such constructed graph edges may be unreliable, since a sketch could have variants of drawings. In this paper, we propose a variant-drawing-protected method by equipping sketch patches with context-aware positional encoding (PE) to make better use of drawing orders for learning graphic sketch representation. Instead of injecting sketch drawings into graph edges, we embed these sequential information into graph nodes only. More specifically, each patch embedding is equipped with a sinusoidal absolute PE to highlight the sequential position in the drawing order. And its neighboring patches, ranked by the values of self-attention scores between patch embeddings, are equipped with learnable relative PEs to restore the contextual positions within a neighborhood. During message aggregation via graph convolutional networks, a node receives both semantic contents from patch embeddings and contextual patterns from PEs by its neighbors, arriving at drawing-order-enhanced sketch representations. Experimental results indicate that our method significantly improves sketch healing and controllable sketch synthesis.

Title: Boosting Few-Shot Learning with Disentangled Self-Supervised Learning and Meta-Learning for Medical Image Classification

• Authors: Eva Pachetti, Sotirios A. Tsaftaris, Sara Colantonio
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.17530
• Pdf URL: https://arxiv.org/pdf/2403.17530
• Copy Paste: [[2403.17530]] Boosting Few-Shot Learning with Disentangled Self-Supervised Learning and Meta-Learning for Medical Image Classification(https://arxiv.org/abs/2403.17530)
• Keywords: robust
• Abstract: Background and objective: Employing deep learning models in critical domains such as medical imaging poses challenges associated with the limited availability of training data. We present a strategy for improving the performance and generalization capabilities of models trained in low-data regimes. Methods: The proposed method starts with a pre-training phase, where features learned in a self-supervised learning setting are disentangled to improve the robustness of the representations for downstream tasks. We then introduce a meta-fine-tuning step, leveraging related classes between meta-training and meta-testing phases but varying the granularity level. This approach aims to enhance the model's generalization capabilities by exposing it to more challenging classification tasks during meta-training and evaluating it on easier tasks but holding greater clinical relevance during meta-testing. We demonstrate the effectiveness of the proposed approach through a series of experiments exploring several backbones, as well as diverse pre-training and fine-tuning schemes, on two distinct medical tasks, i.e., classification of prostate cancer aggressiveness from MRI data and classification of breast cancer malignity from microscopic images. Results: Our results indicate that the proposed approach consistently yields superior performance w.r.t. ablation experiments, maintaining competitiveness even when a distribution shift between training and evaluation data occurs. Conclusion: Extensive experiments demonstrate the effectiveness and wide applicability of the proposed approach. We hope that this work will add another solution to the arsenal of addressing learning issues in data-scarce imaging domains.

Title: Sparse Logistic Regression with High-order Features for Automatic Grammar Rule Extraction from Treebanks

• Authors: Santiago Herrera, Caio Corro, Sylvain Kahane
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.17534
• Pdf URL: https://arxiv.org/pdf/2403.17534
• Copy Paste: [[2403.17534]] Sparse Logistic Regression with High-order Features for Automatic Grammar Rule Extraction from Treebanks(https://arxiv.org/abs/2403.17534)
• Keywords: extraction
• Abstract: Descriptive grammars are highly valuable, but writing them is time-consuming and difficult. Furthermore, while linguists typically use corpora to create them, grammar descriptions often lack quantitative data. As for formal grammars, they can be challenging to interpret. In this paper, we propose a new method to extract and explore significant fine-grained grammar patterns and potential syntactic grammar rules from treebanks, in order to create an easy-to-understand corpus-based grammar. More specifically, we extract descriptions and rules across different languages for two linguistic phenomena, agreement and word order, using a large search space and paying special attention to the ranking order of the extracted rules. For that, we use a linear classifier to extract the most salient features that predict the linguistic phenomena under study. We associate statistical information to each rule, and we compare the ranking of the model's results to those of other quantitative and statistical measures. Our method captures both well-known and less well-known significant grammar rules in Spanish, French, and Wolof.

Title: ILLUMINER: Instruction-tuned Large Language Models as Few-shot Intent Classifier and Slot Filler

• Authors: Paramita Mirza, Viju Sudhi, Soumya Ranjan Sahoo, Sinchana Ramakanth Bhat
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.17536
• Pdf URL: https://arxiv.org/pdf/2403.17536
• Copy Paste: [[2403.17536]] ILLUMINER: Instruction-tuned Large Language Models as Few-shot Intent Classifier and Slot Filler(https://arxiv.org/abs/2403.17536)
• Keywords: large language model
• Abstract: State-of-the-art intent classification (IC) and slot filling (SF) methods often rely on data-intensive deep learning models, limiting their practicality for industry applications. Large language models on the other hand, particularly instruction-tuned models (Instruct-LLMs), exhibit remarkable zero-shot performance across various natural language tasks. This study evaluates Instruct-LLMs on popular benchmark datasets for IC and SF, emphasizing their capacity to learn from fewer examples. We introduce ILLUMINER, an approach framing IC and SF as language generation tasks for Instruct-LLMs, with a more efficient SF-prompting method compared to prior work. A comprehensive comparison with multiple baselines shows that our approach, using the FLAN-T5 11B model, outperforms the state-of-the-art joint IC+SF method and in-context learning with GPT3.5 (175B), particularly in slot filling by 11.1--32.2 percentage points. Additionally, our in-depth ablation study demonstrates that parameter-efficient fine-tuning requires less than 6% of training data to yield comparable performance with traditional full-weight fine-tuning.

Title: NeRF-HuGS: Improved Neural Radiance Fields in Non-static Scenes Using Heuristics-Guided Segmentation

• Authors: Jiahao Chen, Yipeng Qin, Lingjie Liu, Jiangbo Lu, Guanbin Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.17537
• Pdf URL: https://arxiv.org/pdf/2403.17537
• Copy Paste: [[2403.17537]] NeRF-HuGS: Improved Neural Radiance Fields in Non-static Scenes Using Heuristics-Guided Segmentation(https://arxiv.org/abs/2403.17537)
• Keywords: robust, segmentation
• Abstract: Neural Radiance Field (NeRF) has been widely recognized for its excellence in novel view synthesis and 3D scene reconstruction. However, their effectiveness is inherently tied to the assumption of static scenes, rendering them susceptible to undesirable artifacts when confronted with transient distractors such as moving objects or shadows. In this work, we propose a novel paradigm, namely "Heuristics-Guided Segmentation" (HuGS), which significantly enhances the separation of static scenes from transient distractors by harmoniously combining the strengths of hand-crafted heuristics and state-of-the-art segmentation models, thus significantly transcending the limitations of previous solutions. Furthermore, we delve into the meticulous design of heuristics, introducing a seamless fusion of Structure-from-Motion (SfM)-based heuristics and color residual heuristics, catering to a diverse range of texture profiles. Extensive experiments demonstrate the superiority and robustness of our method in mitigating transient distractors for NeRFs trained in non-static scenes. Project page: https://cnhaox.github.io/NeRF-HuGS/.

Title: Large Language Models Are State-of-the-Art Evaluator for Grammatical Error Correction

• Authors: Masamune Kobayashi, Masato Mita, Mamoru Komachi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.17540
• Pdf URL: https://arxiv.org/pdf/2403.17540
• Copy Paste: [[2403.17540]] Large Language Models Are State-of-the-Art Evaluator for Grammatical Error Correction(https://arxiv.org/abs/2403.17540)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have been reported to outperform existing automatic evaluation metrics in some tasks, such as text summarization and machine translation. However, there has been a lack of research on LLMs as evaluators in grammatical error correction (GEC). In this study, we investigate the performance of LLMs in GEC evaluation by employing prompts designed to incorporate various evaluation criteria inspired by previous research. Our extensive experimental results demonstrate that GPT-4 achieved Kendall's rank correlation of 0.662 with human judgments, surpassing all existing methods. Furthermore, in recent GEC evaluations, we have underscored the significance of the LLMs scale and particularly emphasized the importance of fluency among evaluation criteria.

Title: Naive Bayes-based Context Extension for Large Language Models

• Authors: Jianlin Su, Murtadha Ahmed, Wenbo, Luo Ao, Mingren Zhu, Yunfeng Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.17552
• Pdf URL: https://arxiv.org/pdf/2403.17552
• Copy Paste: [[2403.17552]] Naive Bayes-based Context Extension for Large Language Models(https://arxiv.org/abs/2403.17552)
• Keywords: transformer, large language model
• Abstract: Large Language Models (LLMs) have shown promising in-context learning abilities. However, conventional In-Context Learning (ICL) approaches are often impeded by length limitations of transformer architecture, which pose challenges when attempting to effectively integrate supervision from a substantial number of demonstration examples. In this paper, we introduce a novel framework, called Naive Bayes-based Context Extension (NBCE), to enable existing LLMs to perform ICL with an increased number of demonstrations by significantly expanding their context size. Importantly, this expansion does not require fine-tuning or dependence on particular model architectures, all the while preserving linear efficiency. NBCE initially splits the context into equal-sized windows fitting the target LLM's maximum length. Then, it introduces a voting mechanism to select the most relevant window, regarded as the posterior context. Finally, it employs Bayes' theorem to generate the test task. Our experimental results demonstrate that NBCE substantially enhances performance, particularly as the number of demonstration examples increases, consistently outperforming alternative methods. The NBCE code will be made publicly accessible. The code NBCE is available at: https://github.com/amurtadha/NBCE-master

Title: RuBia: A Russian Language Bias Detection Dataset

• Authors: Veronika Grigoreva, Anastasiia Ivanova, Ilseyar Alimova, Ekaterina Artemova
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.17553
• Pdf URL: https://arxiv.org/pdf/2403.17553
• Copy Paste: [[2403.17553]] RuBia: A Russian Language Bias Detection Dataset(https://arxiv.org/abs/2403.17553)
• Keywords: fair, large language model
• Abstract: Warning: this work contains upsetting or disturbing content. Large language models (LLMs) tend to learn the social and cultural biases present in the raw pre-training data. To test if an LLM's behavior is fair, functional datasets are employed, and due to their purpose, these datasets are highly language and culture-specific. In this paper, we address a gap in the scope of multilingual bias evaluation by presenting a bias detection dataset specifically designed for the Russian language, dubbed as RuBia. The RuBia dataset is divided into 4 domains: gender, nationality, socio-economic status, and diverse, each of the domains is further divided into multiple fine-grained subdomains. Every example in the dataset consists of two sentences with the first reinforcing a potentially harmful stereotype or trope and the second contradicting it. These sentence pairs were first written by volunteers and then validated by native-speaking crowdsourcing workers. Overall, there are nearly 2,000 unique sentence pairs spread over 19 subdomains in RuBia. To illustrate the dataset's purpose, we conduct a diagnostic evaluation of state-of-the-art or near-state-of-the-art LLMs and discuss the LLMs' predisposition to social biases.

Title: Enhancing Privacy in Federated Learning through Local Training

• Authors: Nicola Bastianello, Changxin Liu, Karl H. Johansson
• Subjects: cs.LG, math.OC
• Abstract URL: https://arxiv.org/abs/2403.17572
• Pdf URL: https://arxiv.org/pdf/2403.17572
• Copy Paste: [[2403.17572]] Enhancing Privacy in Federated Learning through Local Training(https://arxiv.org/abs/2403.17572)
• Keywords: privacy, federate
• Abstract: In this paper we propose the federated private local training algorithm (Fed-PLT) for federated learning, to overcome the challenges of (i) expensive communications and (ii) privacy preservation. We address (i) by allowing for both partial participation and local training, which significantly reduce the number of communication rounds between the central coordinator and computing agents. The algorithm matches the state of the art in the sense that the use of local training demonstrably does not impact accuracy. Additionally, agents have the flexibility to choose from various local training solvers, such as (stochastic) gradient descent and accelerated gradient descent. Further, we investigate how employing local training can enhance privacy, addressing point (ii). In particular, we derive differential privacy bounds and highlight their dependence on the number of local training epochs. We assess the effectiveness of the proposed algorithm by comparing it to alternative techniques, considering both theoretical analysis and numerical results from a classification task.

Title: Towards a Zero-Data, Controllable, Adaptive Dialog System

• Authors: Dirk Väth, Lindsey Vanderlyn, Ngoc Thang Vu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.17582
• Pdf URL: https://arxiv.org/pdf/2403.17582
• Copy Paste: [[2403.17582]] Towards a Zero-Data, Controllable, Adaptive Dialog System(https://arxiv.org/abs/2403.17582)
• Keywords: large language model
• Abstract: Conversational Tree Search (V\"ath et al., 2023) is a recent approach to controllable dialog systems, where domain experts shape the behavior of a Reinforcement Learning agent through a dialog tree. The agent learns to efficiently navigate this tree, while adapting to information needs, e.g., domain familiarity, of different users. However, the need for additional training data hinders deployment in new domains. To address this, we explore approaches to generate this data directly from dialog trees. We improve the original approach, and show that agents trained on synthetic data can achieve comparable dialog success to models trained on human data, both when using a commercial Large Language Model for generation, or when using a smaller open-source model, running on a single GPU. We further demonstrate the scalability of our approach by collecting and testing on two new datasets: ONBOARD, a new domain helping foreign residents moving to a new city, and the medical domain DIAGNOSE, a subset of Wikipedia articles related to scalp and head symptoms. Finally, we perform human testing, where no statistically significant differences were found in either objective or subjective measures between models trained on human and generated data.

Title: Forest-ORE: Mining Optimal Rule Ensemble to interpret Random Forest models

• Authors: Haddouchi Maissae, Berrado Abdelaziz
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.17588
• Pdf URL: https://arxiv.org/pdf/2403.17588
• Copy Paste: [[2403.17588]] Forest-ORE: Mining Optimal Rule Ensemble to interpret Random Forest models(https://arxiv.org/abs/2403.17588)
• Keywords: security, robust, interpretability
• Abstract: Random Forest (RF) is well-known as an efficient ensemble learning method in terms of predictive performance. It is also considered a Black Box because of its hundreds of deep decision trees. This lack of interpretability can be a real drawback for acceptance of RF models in several real-world applications, especially those affecting one's lives, such as in healthcare, security, and law. In this work, we present Forest-ORE, a method that makes RF interpretable via an optimized rule ensemble (ORE) for local and global interpretation. Unlike other rule-based approaches aiming at interpreting the RF model, this method simultaneously considers several parameters that influence the choice of an interpretable rule ensemble. Existing methods often prioritize predictive performance over interpretability coverage and do not provide information about existing overlaps or interactions between rules. Forest-ORE uses a mixed-integer optimization program to build an ORE that considers the trade-off between predictive performance, interpretability coverage, and model size (size of the rule ensemble, rule lengths, and rule overlaps). In addition to providing an ORE competitive in predictive performance with RF, this method enriches the ORE through other rules that afford complementary information. It also enables monitoring of the rule selection process and delivers various metrics that can be used to generate a graphical representation of the final model. This framework is illustrated through an example, and its robustness is assessed through 36 benchmark datasets. A comparative analysis of well-known methods shows that Forest-ORE provides an excellent trade-off between predictive performance, interpretability coverage, and model size.

Title: Dual Memory Networks: A Versatile Adaptation Approach for Vision-Language Models

• Authors: Yabin Zhang, Wenjie Zhu, Hui Tang, Zhiyuan Ma, Kaiyang Zhou, Lei Zhang
• Subjects: cs.CV, cs.AI, cs.LG, cs.MM
• Abstract URL: https://arxiv.org/abs/2403.17589
• Pdf URL: https://arxiv.org/pdf/2403.17589
• Copy Paste: [[2403.17589]] Dual Memory Networks: A Versatile Adaptation Approach for Vision-Language Models(https://arxiv.org/abs/2403.17589)
• Keywords: robust
• Abstract: With the emergence of pre-trained vision-language models like CLIP, how to adapt them to various downstream classification tasks has garnered significant attention in recent research. The adaptation strategies can be typically categorized into three paradigms: zero-shot adaptation, few-shot adaptation, and the recently-proposed training-free few-shot adaptation. Most existing approaches are tailored for a specific setting and can only cater to one or two of these paradigms. In this paper, we introduce a versatile adaptation approach that can effectively work under all three settings. Specifically, we propose the dual memory networks that comprise dynamic and static memory components. The static memory caches training data knowledge, enabling training-free few-shot adaptation, while the dynamic memory preserves historical test features online during the testing process, allowing for the exploration of additional data insights beyond the training set. This novel capability enhances model performance in the few-shot setting and enables model usability in the absence of training data. The two memory networks employ the same flexible memory interactive strategy, which can operate in a training-free mode and can be further enhanced by incorporating learnable projection layers. Our approach is tested across 11 datasets under the three task settings. Remarkably, in the zero-shot scenario, it outperforms existing methods by over 3\% and even shows superior results against methods utilizing external training data. Additionally, our method exhibits robust performance against natural distribution shifts. Codes are available at \url{https://github.com/YBZh/DMN}.

Title: Fake or JPEG? Revealing Common Biases in Generated Image Detection Datasets

• Authors: Patrick Grommelt, Louis Weiss, Franz-Josef Pfreundt, Janis Keuper
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.17608
• Pdf URL: https://arxiv.org/pdf/2403.17608
• Copy Paste: [[2403.17608]] Fake or JPEG? Revealing Common Biases in Generated Image Detection Datasets(https://arxiv.org/abs/2403.17608)
• Keywords: robust, generative
• Abstract: The widespread adoption of generative image models has highlighted the urgent need to detect artificial content, which is a crucial step in combating widespread manipulation and misinformation. Consequently, numerous detectors and associated datasets have emerged. However, many of these datasets inadvertently introduce undesirable biases, thereby impacting the effectiveness and evaluation of detectors. In this paper, we emphasize that many datasets for AI-generated image detection contain biases related to JPEG compression and image size. Using the GenImage dataset, we demonstrate that detectors indeed learn from these undesired factors. Furthermore, we show that removing the named biases substantially increases robustness to JPEG compression and significantly alters the cross-generator performance of evaluated detectors. Specifically, it leads to more than 11 percentage points increase in cross-generator performance for ResNet50 and Swin-T detectors on the GenImage dataset, achieving state-of-the-art results. We provide the dataset and source codes of this paper on the anonymous website: https://www.unbiased-genimage.org

Title: MMVP: A Multimodal MoCap Dataset with Vision and Pressure Sensors

• Authors: He Zhang, Shenghao Ren, Haolei Yuan, Jianhui Zhao, Fan Li, Shuangpeng Sun, Zhenghao Liang, Tao Yu, Qiu Shen, Xun Cao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.17610
• Pdf URL: https://arxiv.org/pdf/2403.17610
• Copy Paste: [[2403.17610]] MMVP: A Multimodal MoCap Dataset with Vision and Pressure Sensors(https://arxiv.org/abs/2403.17610)
• Keywords: robust
• Abstract: Foot contact is an important cue not only for human motion capture but also for motion understanding and physically plausible motion generation. However, most of the foot-contact annotations in existing datasets are estimated by purely visual matching and distance thresholding, which results in low accuracy and coarse granularity. Even though existing multimodal datasets synergistically capture plantar pressure (foot contact) and visual signals, they are specifically designed for small-range and slow motion such as Taiji Quan and Yoga. Therefore, there is still a lack of a vision-pressure multimodal dataset with large-range and fast human motion, as well as accurate and dense foot-contact annotation. To fill this gap, we propose a Multimodal MoCap Dataset with Vision and Pressure sensors, named MMVP. MMVP provides accurate and dense plantar pressure signals synchronized with RGBD observations, which is especially useful for both plausible shape estimation, robust pose fitting without foot drifting, and accurate global translation tracking. To validate the dataset, we propose an RGBD-P SMPL fitting method and also a monocular-video-based baseline framework, VP-MoCap, for human motion capture. Experiments demonstrate that our RGBD-P SMPL Fitting results significantly outperform pure visual motion capture. Moreover, VP-MoCap outperforms SOTA methods in foot-contact and global translation estimation accuracy. We believe the configuration of the dataset and the baseline frameworks will stimulate the research in this direction and also provide a good reference for MoCap applications in various domains. Project page: https://haolyuan.github.io/MMVP-Dataset/.

Title: "You are an expert annotator": Automatic Best-Worst-Scaling Annotations for Emotion Intensity Modeling

• Authors: Christopher Bagdon, Prathamesh Karmalker, Harsha Gurulingappa, Roman Klinger
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.17612
• Pdf URL: https://arxiv.org/pdf/2403.17612
• Copy Paste: [[2403.17612]] "You are an expert annotator": Automatic Best-Worst-Scaling Annotations for Emotion Intensity Modeling(https://arxiv.org/abs/2403.17612)
• Keywords: transformer, large language model
• Abstract: Labeling corpora constitutes a bottleneck to create models for new tasks or domains. Large language models mitigate the issue with automatic corpus labeling methods, particularly for categorical annotations. Some NLP tasks such as emotion intensity prediction, however, require text regression, but there is no work on automating annotations for continuous label assignments. Regression is considered more challenging than classification: The fact that humans perform worse when tasked to choose values from a rating scale lead to comparative annotation methods, including best-worst scaling. This raises the question if large language model-based annotation methods show similar patterns, namely that they perform worse on rating scale annotation tasks than on comparative annotation tasks. To study this, we automate emotion intensity predictions and compare direct rating scale predictions, pairwise comparisons and best-worst scaling. We find that the latter shows the highest reliability. A transformer regressor fine-tuned on these data performs nearly on par with a model trained on the original manual annotations.

Title: AniArtAvatar: Animatable 3D Art Avatar from a Single Image

• Authors: Shaoxu Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.17631
• Pdf URL: https://arxiv.org/pdf/2403.17631
• Copy Paste: [[2403.17631]] AniArtAvatar: Animatable 3D Art Avatar from a Single Image(https://arxiv.org/abs/2403.17631)
• Keywords: diffusion
• Abstract: We present a novel approach for generating animatable 3D-aware art avatars from a single image, with controllable facial expressions, head poses, and shoulder movements. Unlike previous reenactment methods, our approach utilizes a view-conditioned 2D diffusion model to synthesize multi-view images from a single art portrait with a neutral expression. With the generated colors and normals, we synthesize a static avatar using an SDF-based neural surface. For avatar animation, we extract control points, transfer the motion with these points, and deform the implicit canonical space. Firstly, we render the front image of the avatar, extract the 2D landmarks, and project them to the 3D space using a trained SDF network. We extract 3D driving landmarks using 3DMM and transfer the motion to the avatar landmarks. To animate the avatar pose, we manually set the body height and bound the head and torso of an avatar with two cages. The head and torso can be animated by transforming the two cages. Our approach is a one-shot pipeline that can be applied to various styles. Experiments demonstrate that our method can generate high-quality 3D art avatars with desired control over different motions.

Title: Intrinsic Subgraph Generation for Interpretable Graph based Visual Question Answering

• Authors: Pascal Tilli, Ngoc Thang Vu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.17647
• Pdf URL: https://arxiv.org/pdf/2403.17647
• Copy Paste: [[2403.17647]] Intrinsic Subgraph Generation for Interpretable Graph based Visual Question Answering(https://arxiv.org/abs/2403.17647)
• Keywords: interpretability, explainability
• Abstract: The large success of deep learning based methods in Visual Question Answering (VQA) has concurrently increased the demand for explainable methods. Most methods in Explainable Artificial Intelligence (XAI) focus on generating post-hoc explanations rather than taking an intrinsic approach, the latter characterizing an interpretable model. In this work, we introduce an interpretable approach for graph-based VQA and demonstrate competitive performance on the GQA dataset. This approach bridges the gap between interpretability and performance. Our model is designed to intrinsically produce a subgraph during the question-answering process as its explanation, providing insight into the decision making. To evaluate the quality of these generated subgraphs, we compare them against established post-hoc explainability methods for graph neural networks, and perform a human evaluation. Moreover, we present quantitative metrics that correlate with the evaluations of human assessors, acting as automatic metrics for the generated explanatory subgraphs. Our implementation is available at https://github.com/DigitalPhonetics/Intrinsic-Subgraph-Generation-for-VQA.

Title: Healthcare Data Governance, Privacy, and Security - A Conceptual Framework

• Authors: Amen Faridoon, M. Tahar Kechadi
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.17648
• Pdf URL: https://arxiv.org/pdf/2403.17648
• Copy Paste: [[2403.17648]] Healthcare Data Governance, Privacy, and Security - A Conceptual Framework(https://arxiv.org/abs/2403.17648)
• Keywords: security, privacy, protect
• Abstract: The abundance of data has transformed the world in every aspect. It has become the core element in decision making, problem solving, and innovation in almost all areas of life, including business, science, healthcare, education, and many others. Despite all these advances, privacy and security remain critical concerns of the healthcare industry. It is important to note that healthcare data can also be a liability if it is not managed correctly. This data mismanagement can have severe consequences for patients and healthcare organisations, including patient safety, legal liability, damage to reputation, financial loss, and operational inefficiency. Healthcare organisations must comply with a range of regulations to protect patient data. We perform a classification of data governance elements or components in a manner that thoroughly assesses the healthcare data chain from a privacy and security standpoint. After deeply analysing the existing literature, we propose a conceptual privacy and security driven healthcare data governance framework.

Title: Exploring Dynamic Transformer for Efficient Object Tracking

• Authors: Jiawen Zhu, Xin Chen, Haiwen Diao, Shuai Li, Jun-Yan He, Chenyang Li, Bin Luo, Dong Wang, Huchuan Lu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.17651
• Pdf URL: https://arxiv.org/pdf/2403.17651
• Copy Paste: [[2403.17651]] Exploring Dynamic Transformer for Efficient Object Tracking(https://arxiv.org/abs/2403.17651)
• Keywords: transformer
• Abstract: The speed-precision trade-off is a critical problem for visual object tracking which usually requires low latency and deployment on constrained resources. Existing solutions for efficient tracking mainly focus on adopting light-weight backbones or modules, which nevertheless come at the cost of a sacrifice in precision. In this paper, inspired by dynamic network routing, we propose DyTrack, a dynamic transformer framework for efficient tracking. Real-world tracking scenarios exhibit diverse levels of complexity. We argue that a simple network is sufficient for easy frames in video sequences, while more computation could be assigned to difficult ones. DyTrack automatically learns to configure proper reasoning routes for various inputs, gaining better utilization of the available computational budget. Thus, it can achieve higher performance with the same running speed. We formulate instance-specific tracking as a sequential decision problem and attach terminating branches to intermediate layers of the entire model. Especially, to fully utilize the computations, we introduce the feature recycling mechanism to reuse the outputs of predecessors. Furthermore, a target-aware self-distillation strategy is designed to enhance the discriminating capabilities of early predictions by effectively mimicking the representation pattern of the deep model. Extensive experiments on multiple benchmarks demonstrate that DyTrack achieves promising speed-precision trade-offs with only a single model. For instance, DyTrack obtains 64.9% AUC on LaSOT with a speed of 256 fps.

Title: CANOS: A Fast and Scalable Neural AC-OPF Solver Robust To N-1 Perturbations

• Authors: Luis Piloto, Sofia Liguori, Sephora Madjiheurem, Miha Zgubic, Sean Lovett, Hamish Tomlinson, Sophie Elster, Chris Apps, Sims Witherspoon
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.17660
• Pdf URL: https://arxiv.org/pdf/2403.17660
• Copy Paste: [[2403.17660]] CANOS: A Fast and Scalable Neural AC-OPF Solver Robust To N-1 Perturbations(https://arxiv.org/abs/2403.17660)
• Keywords: secure, security, robust
• Abstract: Optimal Power Flow (OPF) refers to a wide range of related optimization problems with the goal of operating power systems efficiently and securely. In the simplest setting, OPF determines how much power to generate in order to minimize costs while meeting demand for power and satisfying physical and operational constraints. In even the simplest case, power grid operators use approximations of the AC-OPF problem because solving the exact problem is prohibitively slow with state-of-the-art solvers. These approximations sacrifice accuracy and operational feasibility in favor of speed. This trade-off leads to costly "uplift payments" and increased carbon emissions, especially for large power grids. In the present work, we train a deep learning system (CANOS) to predict near-optimal solutions (within 1% of the true AC-OPF cost) without compromising speed (running in as little as 33--65 ms). Importantly, CANOS scales to realistic grid sizes with promising empirical results on grids containing as many as 10,000 buses. Finally, because CANOS is a Graph Neural Network, it is robust to changes in topology. We show that CANOS is accurate across N-1 topological perturbations of a base grid typically used in security-constrained analysis. This paves the way for more efficient optimization of more complex OPF problems which alter grid connectivity such as unit commitment, topology optimization and security-constrained OPF.

Title: Language Models for Text Classification: Is In-Context Learning Enough?

• Authors: Aleksandra Edwards, Jose Camacho-Collados
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.17661
• Pdf URL: https://arxiv.org/pdf/2403.17661
• Copy Paste: [[2403.17661]] Language Models for Text Classification: Is In-Context Learning Enough?(https://arxiv.org/abs/2403.17661)
• Keywords: large language model
• Abstract: Recent foundational language models have shown state-of-the-art performance in many NLP tasks in zero- and few-shot settings. An advantage of these models over more standard approaches based on fine-tuning is the ability to understand instructions written in natural language (prompts), which helps them generalise better to different tasks and domains without the need for specific training data. This makes them suitable for addressing text classification problems for domains with limited amounts of annotated instances. However, existing research is limited in scale and lacks understanding of how text generation models combined with prompting techniques compare to more established methods for text classification such as fine-tuning masked language models. In this paper, we address this research gap by performing a large-scale evaluation study for 16 text classification datasets covering binary, multiclass, and multilabel problems. In particular, we compare zero- and few-shot approaches of large language models to fine-tuning smaller language models. We also analyse the results by prompt, classification type, domain, and number of labels. In general, the results show how fine-tuning smaller and more efficient language models can still outperform few-shot approaches of larger language models, which have room for improvement when it comes to text classification.

Title: DiffFAE: Advancing High-fidelity One-shot Facial Appearance Editing with Space-sensitive Customization and Semantic Preservation

• Authors: Qilin Wang, Jiangning Zhang, Chengming Xu, Weijian Cao, Ying Tai, Yue Han, Yanhao Ge, Hong Gu, Chengjie Wang, Yanwei Fu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.17664
• Pdf URL: https://arxiv.org/pdf/2403.17664
• Copy Paste: [[2403.17664]] DiffFAE: Advancing High-fidelity One-shot Facial Appearance Editing with Space-sensitive Customization and Semantic Preservation(https://arxiv.org/abs/2403.17664)
• Keywords: diffusion
• Abstract: Facial Appearance Editing (FAE) aims to modify physical attributes, such as pose, expression and lighting, of human facial images while preserving attributes like identity and background, showing great importance in photograph. In spite of the great progress in this area, current researches generally meet three challenges: low generation fidelity, poor attribute preservation, and inefficient inference. To overcome above challenges, this paper presents DiffFAE, a one-stage and highly-efficient diffusion-based framework tailored for high-fidelity FAE. For high-fidelity query attributes transfer, we adopt Space-sensitive Physical Customization (SPC), which ensures the fidelity and generalization ability by utilizing rendering texture derived from 3D Morphable Model (3DMM). In order to preserve source attributes, we introduce the Region-responsive Semantic Composition (RSC). This module is guided to learn decoupled source-regarding features, thereby better preserving the identity and alleviating artifacts from non-facial attributes such as hair, clothes, and background. We further introduce a consistency regularization for our pipeline to enhance editing controllability by leveraging prior knowledge in the attention matrices of diffusion model. Extensive experiments demonstrate the superiority of DiffFAE over existing methods, achieving state-of-the-art performance in facial appearance editing.

Title: How Private is DP-SGD?

• Authors: Lynn Chua, Badih Ghazi, Pritish Kamath, Ravi Kumar, Pasin Manurangsi, Amer Sinha, Chiyuan Zhang
• Subjects: cs.LG, cs.CR, cs.DS
• Abstract URL: https://arxiv.org/abs/2403.17673
• Pdf URL: https://arxiv.org/pdf/2403.17673
• Copy Paste: [[2403.17673]] How Private is DP-SGD?(https://arxiv.org/abs/2403.17673)
• Keywords: privacy
• Abstract: We demonstrate a substantial gap between the privacy guarantees of the Adaptive Batch Linear Queries (ABLQ) mechanism under different types of batch sampling: (i) Shuffling, and (ii) Poisson subsampling; the typical analysis of Differentially Private Stochastic Gradient Descent (DP-SGD) follows by interpreting it as a post-processing of ABLQ. While shuffling based DP-SGD is more commonly used in practical implementations, it is neither analytically nor numerically amenable to easy privacy analysis. On the other hand, Poisson subsampling based DP-SGD is challenging to scalably implement, but has a well-understood privacy analysis, with multiple open-source numerically tight privacy accountants available. This has led to a common practice of using shuffling based DP-SGD in practice, but using the privacy analysis for the corresponding Poisson subsampling version. Our result shows that there can be a substantial gap between the privacy analysis when using the two types of batch sampling, and thus advises caution in reporting privacy parameters for DP-SGD.

Title: Depending on yourself when you should: Mentoring LLM with RL agents to become the master in cybersecurity games

• Authors: Yikuan Yan, Yaolun Zhang, Keman Huang
• Subjects: cs.CR, cs.AI, cs.MA
• Abstract URL: https://arxiv.org/abs/2403.17674
• Pdf URL: https://arxiv.org/pdf/2403.17674
• Copy Paste: [[2403.17674]] Depending on yourself when you should: Mentoring LLM with RL agents to become the master in cybersecurity games(https://arxiv.org/abs/2403.17674)
• Keywords: security
• Abstract: Integrating LLM and reinforcement learning (RL) agent effectively to achieve complementary performance is critical in high stake tasks like cybersecurity operations. In this study, we introduce SecurityBot, a LLM agent mentored by pre-trained RL agents, to support cybersecurity operations. In particularly, the LLM agent is supported with a profile module to generated behavior guidelines, a memory module to accumulate local experiences, a reflection module to re-evaluate choices, and an action module to reduce action space. Additionally, it adopts the collaboration mechanism to take suggestions from pre-trained RL agents, including a cursor for dynamic suggestion taken, an aggregator for multiple mentors' suggestions ranking and a caller for proactive suggestion asking. Building on the CybORG experiment framework, our experiences show that SecurityBot demonstrates significant performance improvement compared with LLM or RL standalone, achieving the complementary performance in the cybersecurity games.

Title: Hierarchical Light Transformer Ensembles for Multimodal Trajectory Forecasting

• Authors: Adrien Lafage, Mathieu Barbier, Gianni Franchi, David Filliat
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.17678
• Pdf URL: https://arxiv.org/pdf/2403.17678
• Copy Paste: [[2403.17678]] Hierarchical Light Transformer Ensembles for Multimodal Trajectory Forecasting(https://arxiv.org/abs/2403.17678)
• Keywords: transformer
• Abstract: Accurate trajectory forecasting is crucial for the performance of various systems, such as advanced driver-assistance systems and self-driving vehicles. These forecasts allow to anticipate events leading to collisions and, therefore, to mitigate them. Deep Neural Networks have excelled in motion forecasting, but issues like overconfidence and uncertainty quantification persist. Deep Ensembles address these concerns, yet applying them to multimodal distributions remains challenging. In this paper, we propose a novel approach named Hierarchical Light Transformer Ensembles (HLT-Ens), aimed at efficiently training an ensemble of Transformer architectures using a novel hierarchical loss function. HLT-Ens leverages grouped fully connected layers, inspired by grouped convolution techniques, to capture multimodal distributions, effectively. Through extensive experimentation, we demonstrate that HLT-Ens achieves state-of-the-art performance levels, offering a promising avenue for improving trajectory forecasting techniques.

Title: Not All Similarities Are Created Equal: Leveraging Data-Driven Biases to Inform GenAI Copyright Disputes

• Authors: Uri Hacohen, Adi Haviv, Shahar Sarfaty, Bruria Friedman, Niva Elkin-Koren, Roi Livni, Amit H Bermano
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2403.17691
• Pdf URL: https://arxiv.org/pdf/2403.17691
• Copy Paste: [[2403.17691]] Not All Similarities Are Created Equal: Leveraging Data-Driven Biases to Inform GenAI Copyright Disputes(https://arxiv.org/abs/2403.17691)
• Keywords: protect, fair, diffusion, generative
• Abstract: The advent of Generative Artificial Intelligence (GenAI) models, including GitHub Copilot, OpenAI GPT, and Stable Diffusion, has revolutionized content creation, enabling non-professionals to produce high-quality content across various domains. This transformative technology has led to a surge of synthetic content and sparked legal disputes over copyright infringement. To address these challenges, this paper introduces a novel approach that leverages the learning capacity of GenAI models for copyright legal analysis, demonstrated with GPT2 and Stable Diffusion models. Copyright law distinguishes between original expressions and generic ones (Sc\`enes \`a faire), protecting the former and permitting reproduction of the latter. However, this distinction has historically been challenging to make consistently, leading to over-protection of copyrighted works. GenAI offers an unprecedented opportunity to enhance this legal analysis by revealing shared patterns in preexisting works. We propose a data-driven approach to identify the genericity of works created by GenAI, employing "data-driven bias" to assess the genericity of expressive compositions. This approach aids in copyright scope determination by utilizing the capabilities of GenAI to identify and prioritize expressive elements and rank them according to their frequency in the model's dataset. The potential implications of measuring expressive genericity for copyright law are profound. Such scoring could assist courts in determining copyright scope during litigation, inform the registration practices of Copyright Offices, allowing registration of only highly original synthetic works, and help copyright owners signal the value of their works and facilitate fairer licensing deals. More generally, this approach offers valuable insights to policymakers grappling with adapting copyright law to the challenges posed by the era of GenAI.

Title: Manifold-Guided Lyapunov Control with Diffusion Models

• Authors: Amartya Mukherjee, Thanin Quartz, Jun Liu
• Subjects: cs.CV, cs.LG, math.DG, math.OC, stat.CO
• Abstract URL: https://arxiv.org/abs/2403.17692
• Pdf URL: https://arxiv.org/pdf/2403.17692
• Copy Paste: [[2403.17692]] Manifold-Guided Lyapunov Control with Diffusion Models(https://arxiv.org/abs/2403.17692)
• Keywords: diffusion
• Abstract: This paper presents a novel approach to generating stabilizing controllers for a large class of dynamical systems using diffusion models. The core objective is to develop stabilizing control functions by identifying the closest asymptotically stable vector field relative to a predetermined manifold and adjusting the control function based on this finding. To achieve this, we employ a diffusion model trained on pairs consisting of asymptotically stable vector fields and their corresponding Lyapunov functions. Our numerical results demonstrate that this pre-trained model can achieve stabilization over previously unseen systems efficiently and rapidly, showcasing the potential of our approach in fast zero-shot control and generalizability.

Title: AniPortrait: Audio-Driven Synthesis of Photorealistic Portrait Animation

• Authors: Huawei Wei, Zejun Yang, Zhisheng Wang
• Subjects: cs.CV, cs.GR, eess.IV
• Abstract URL: https://arxiv.org/abs/2403.17694
• Pdf URL: https://arxiv.org/pdf/2403.17694
• Copy Paste: [[2403.17694]] AniPortrait: Audio-Driven Synthesis of Photorealistic Portrait Animation(https://arxiv.org/abs/2403.17694)
• Keywords: robust, diffusion
• Abstract: In this study, we propose AniPortrait, a novel framework for generating high-quality animation driven by audio and a reference portrait image. Our methodology is divided into two stages. Initially, we extract 3D intermediate representations from audio and project them into a sequence of 2D facial landmarks. Subsequently, we employ a robust diffusion model, coupled with a motion module, to convert the landmark sequence into photorealistic and temporally consistent portrait animation. Experimental results demonstrate the superiority of AniPortrait in terms of facial naturalness, pose diversity, and visual quality, thereby offering an enhanced perceptual experience. Moreover, our methodology exhibits considerable potential in terms of flexibility and controllability, which can be effectively applied in areas such as facial motion editing or face reenactment. We release code and model weights at https://github.com/scutzzj/AniPortrait

Title: PlainMamba: Improving Non-Hierarchical Mamba in Visual Recognition

• Authors: Chenhongyi Yang, Zehui Chen, Miguel Espinosa, Linus Ericsson, Zhenyu Wang, Jiaming Liu, Elliot J. Crowley
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.17695
• Pdf URL: https://arxiv.org/pdf/2403.17695
• Copy Paste: [[2403.17695]] PlainMamba: Improving Non-Hierarchical Mamba in Visual Recognition(https://arxiv.org/abs/2403.17695)
• Keywords: segmentation
• Abstract: We present PlainMamba: a simple non-hierarchical state space model (SSM) designed for general visual recognition. The recent Mamba model has shown how SSMs can be highly competitive with other architectures on sequential data and initial attempts have been made to apply it to images. In this paper, we further adapt the selective scanning process of Mamba to the visual domain, enhancing its ability to learn features from two-dimensional images by (i) a continuous 2D scanning process that improves spatial continuity by ensuring adjacency of tokens in the scanning sequence, and (ii) direction-aware updating which enables the model to discern the spatial relations of tokens by encoding directional information. Our architecture is designed to be easy to use and easy to scale, formed by stacking identical PlainMamba blocks, resulting in a model with constant width throughout all layers. The architecture is further simplified by removing the need for special tokens. We evaluate PlainMamba on a variety of visual recognition tasks including image classification, semantic segmentation, object detection, and instance segmentation. Our method achieves performance gains over previous non-hierarchical models and is competitive with hierarchical alternatives. For tasks requiring high-resolution inputs, in particular, PlainMamba requires much less computing while maintaining high performance. Code and models are available at https://github.com/ChenhongyiYang/PlainMamba

Title: MEP: Multiple Kernel Learning Enhancing Relative Positional Encoding Length Extrapolation

• Authors: Weiguo Gao
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.17698
• Pdf URL: https://arxiv.org/pdf/2403.17698
• Copy Paste: [[2403.17698]] MEP: Multiple Kernel Learning Enhancing Relative Positional Encoding Length Extrapolation(https://arxiv.org/abs/2403.17698)
• Keywords: transformer
• Abstract: When the predicted sequence length exceeds the length seen during training, the transformer's inference accuracy diminishes. Existing relative position encoding methods, such as those based on the ALiBi technique, address the length extrapolation challenge exclusively through the implementation of a single kernel function, which introduces a constant bias to every post-softmax attention scores according to their distance. These approaches do not investigate or employ multiple kernel functions to address the extrapolation challenge. Drawing on the ALiBi approach, this study proposes a novel relative positional encoding method, called MEP, which employs a weighted average to combine distinct kernel functions(such as the exponential kernel and the Gaussian kernel) to generate a bias that is applied to post-softmax attention scores. Initially, the framework utilizes various kernel functions to construct multiple kernel functions. Each kernel function adheres to a consistent mean weight coefficient, harnessing the synergistic advantages of different kernels to formulate an innovative bias function. Subsequently, specific slopes are tailored for each kernel function, applying penalties at varying rates, to enhance the model's extrapolation capabilities. Finally, this bias is seamlessly incorporated as a penalty to the post-softmax scores. We present two distinct versions of our method: a parameter-free variant that requires no new learnable parameters, which enhances length extrapolation capabilities without compromising training efficiency, and a parameterized variant capable of integrating state-of-the-art techniques. Empirical evaluations across diverse datasets have demonstrated that both variants of our method achieve state-of-the-art performance, outperforming traditional parameter-free and parameterized approaches.

Title: Enhanced Short Text Modeling: Leveraging Large Language Models for Topic Refinement

• Authors: Shuyu Chang, Rui Wang, Peng Ren, Haiping Huang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.17706
• Pdf URL: https://arxiv.org/pdf/2403.17706
• Copy Paste: [[2403.17706]] Enhanced Short Text Modeling: Leveraging Large Language Models for Topic Refinement(https://arxiv.org/abs/2403.17706)
• Keywords: large language model
• Abstract: Crafting effective topic models for brief texts, like tweets and news headlines, is essential for capturing the swift shifts in social dynamics. Traditional topic models, however, often fall short in accurately representing the semantic intricacies of short texts due to their brevity and lack of contextual data. In our study, we harness the advanced capabilities of Large Language Models (LLMs) to introduce a novel approach termed "Topic Refinement". This approach does not directly involve itself in the initial modeling of topics but focuses on improving topics after they have been mined. By employing prompt engineering, we direct LLMs to eliminate off-topic words within a given topic, ensuring that only contextually relevant words are preserved or substituted with ones that fit better semantically. This method emulates human-like scrutiny and improvement of topics, thereby elevating the semantic quality of the topics generated by various models. Our comprehensive evaluation across three unique datasets has shown that our topic refinement approach significantly enhances the semantic coherence of topics.

Title: Groupwise Query Specialization and Quality-Aware Multi-Assignment for Transformer-based Visual Relationship Detection

• Authors: Jongha Kim, Jihwan Park, Jinyoung Park, Jinyoung Kim, Sehyung Kim, Hyunwoo J. Kim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.17709
• Pdf URL: https://arxiv.org/pdf/2403.17709
• Copy Paste: [[2403.17709]] Groupwise Query Specialization and Quality-Aware Multi-Assignment for Transformer-based Visual Relationship Detection(https://arxiv.org/abs/2403.17709)
• Keywords: transformer
• Abstract: Visual Relationship Detection (VRD) has seen significant advancements with Transformer-based architectures recently. However, we identify two key limitations in a conventional label assignment for training Transformer-based VRD models, which is a process of mapping a ground-truth (GT) to a prediction. Under the conventional assignment, an unspecialized query is trained since a query is expected to detect every relation, which makes it difficult for a query to specialize in specific relations. Furthermore, a query is also insufficiently trained since a GT is assigned only to a single prediction, therefore near-correct or even correct predictions are suppressed by being assigned no relation as a GT. To address these issues, we propose Groupwise Query Specialization and Quality-Aware Multi-Assignment (SpeaQ). Groupwise Query Specialization trains a specialized query by dividing queries and relations into disjoint groups and directing a query in a specific query group solely toward relations in the corresponding relation group. Quality-Aware Multi-Assignment further facilitates the training by assigning a GT to multiple predictions that are significantly close to a GT in terms of a subject, an object, and the relation in between. Experimental results and analyses show that SpeaQ effectively trains specialized queries, which better utilize the capacity of a model, resulting in consistent performance gains with zero additional inference cost across multiple VRD models and benchmarks. Code is available at https://github.com/mlvlab/SpeaQ.

Title: Optimization-based Prompt Injection Attack to LLM-as-a-Judge

• Authors: Jiawen Shi, Zenghui Yuan, Yinuo Liu, Yue Huang, Pan Zhou, Lichao Sun, Neil Zhenqiang Gong
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.17710
• Pdf URL: https://arxiv.org/pdf/2403.17710
• Copy Paste: [[2403.17710]] Optimization-based Prompt Injection Attack to LLM-as-a-Judge(https://arxiv.org/abs/2403.17710)
• Keywords: security, attack, robust, large language model
• Abstract: LLM-as-a-Judge is a novel solution that can assess textual information with large language models (LLMs). Based on existing research studies, LLMs demonstrate remarkable performance in providing a compelling alternative to traditional human assessment. However, the robustness of these systems against prompt injection attacks remains an open question. In this work, we introduce JudgeDeceiver, a novel optimization-based prompt injection attack tailored to LLM-as-a-Judge. Our method formulates a precise optimization objective for attacking the decision-making process of LLM-as-a-Judge and utilizes an optimization algorithm to efficiently automate the generation of adversarial sequences, achieving targeted and effective manipulation of model evaluations. Compared to handcraft prompt injection attacks, our method demonstrates superior efficacy, posing a significant challenge to the current security paradigms of LLM-based judgment systems. Through extensive experiments, we showcase the capability of JudgeDeceiver in altering decision outcomes across various cases, highlighting the vulnerability of LLM-as-a-Judge systems to the optimization-based prompt injection attack.

Title: Deep Learning for Segmentation of Cracks in High-Resolution Images of Steel Bridges

• Authors: Andrii Kompanets, Gautam Pai, Remco Duits, Davide Leonetti, Bert Snijder
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2403.17725
• Pdf URL: https://arxiv.org/pdf/2403.17725
• Copy Paste: [[2403.17725]] Deep Learning for Segmentation of Cracks in High-Resolution Images of Steel Bridges(https://arxiv.org/abs/2403.17725)
• Keywords: robust, segmentation
• Abstract: Automating the current bridge visual inspection practices using drones and image processing techniques is a prominent way to make these inspections more effective, robust, and less expensive. In this paper, we investigate the development of a novel deep-learning method for the detection of fatigue cracks in high-resolution images of steel bridges. First, we present a novel and challenging dataset comprising of images of cracks in steel bridges. Secondly, we integrate the ConvNext neural network with a previous state- of-the-art encoder-decoder network for crack segmentation. We study and report, the effects of the use of background patches on the network performance when applied to high-resolution images of cracks in steel bridges. Finally, we introduce a loss function that allows the use of more background patches for the training process, which yields a significant reduction in false positive rates.

Title: Leave No Patient Behind: Enhancing Medication Recommendation for Rare Disease Patients

• Authors: Zihao Zhao, Yi Jing, Fuli Feng, Jiancan Wu, Chongming Gao, Xiangnan He
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.17745
• Pdf URL: https://arxiv.org/pdf/2403.17745
• Copy Paste: [[2403.17745]] Leave No Patient Behind: Enhancing Medication Recommendation for Rare Disease Patients(https://arxiv.org/abs/2403.17745)
• Keywords: robust, fair, transformer
• Abstract: Medication recommendation systems have gained significant attention in healthcare as a means of providing tailored and effective drug combinations based on patients' clinical information. However, existing approaches often suffer from fairness issues, as recommendations tend to be more accurate for patients with common diseases compared to those with rare conditions. In this paper, we propose a novel model called Robust and Accurate REcommendations for Medication (RAREMed), which leverages the pretrain-finetune learning paradigm to enhance accuracy for rare diseases. RAREMed employs a transformer encoder with a unified input sequence approach to capture complex relationships among disease and procedure codes. Additionally, it introduces two self-supervised pre-training tasks, namely Sequence Matching Prediction (SMP) and Self Reconstruction (SR), to learn specialized medication needs and interrelations among clinical codes. Experimental results on two real-world datasets demonstrate that RAREMed provides accurate drug sets for both rare and common disease patients, thereby mitigating unfairness in medication recommendation systems.

Title: Can multiple-choice questions really be useful in detecting the abilities of LLMs?

• Authors: Wangyue Li, Liangzhi Li, Tong Xiang, Xiao Liu, Wei Deng, Noa Garcia
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.17752
• Pdf URL: https://arxiv.org/pdf/2403.17752
• Copy Paste: [[2403.17752]] Can multiple-choice questions really be useful in detecting the abilities of LLMs?(https://arxiv.org/abs/2403.17752)
• Keywords: large language model
• Abstract: Multiple-choice questions (MCQs) are widely used in the evaluation of large language models (LLMs) due to their simplicity and efficiency. However, there are concerns about whether MCQs can truly measure LLM's capabilities, particularly in knowledge-intensive scenarios where long-form generation (LFG) answers are required. The misalignment between the task and the evaluation method demands a thoughtful analysis of MCQ's efficacy, which we undertake in this paper by evaluating nine LLMs on four question-answering (QA) datasets in two languages: Chinese and English. We identify a significant issue: LLMs exhibit an order sensitivity in bilingual MCQs, favoring answers located at specific positions, i.e., the first position. We further quantify the gap between MCQs and long-form generation questions (LFGQs) by comparing their direct outputs, token logits, and embeddings. Our results reveal a relatively low correlation between answers from MCQs and LFGQs for identical questions. Additionally, we propose two methods to quantify the consistency and confidence of LLMs' output, which can be generalized to other QA evaluation benchmarks. Notably, our analysis challenges the idea that the higher the consistency, the greater the accuracy. We also find MCQs to be less reliable than LFGQs in terms of expected calibration error. Finally, the misalignment between MCQs and LFGQs is not only reflected in the evaluation performance but also in the embedding space. Our code and models can be accessed at https://github.com/Meetyou-AI-Lab/Can-MC-Evaluate-LLMs.

Title: CCDSReFormer: Traffic Flow Prediction with a Criss-Crossed Dual-Stream Enhanced Rectified Transformer Model

• Authors: Zhiqi Shao, Michael G.H. Bell, Ze Wang, D. Glenn Geers, Xusheng Yao, Junbin Gao
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.17753
• Pdf URL: https://arxiv.org/pdf/2403.17753
• Copy Paste: [[2403.17753]] CCDSReFormer: Traffic Flow Prediction with a Criss-Crossed Dual-Stream Enhanced Rectified Transformer Model(https://arxiv.org/abs/2403.17753)
• Keywords: transformer
• Abstract: Accurate, and effective traffic forecasting is vital for smart traffic systems, crucial in urban traffic planning and management. Current Spatio-Temporal Transformer models, despite their prediction capabilities, struggle with balancing computational efficiency and accuracy, favoring global over local information, and handling spatial and temporal data separately, limiting insight into complex interactions. We introduce the Criss-Crossed Dual-Stream Enhanced Rectified Transformer model (CCDSReFormer), which includes three innovative modules: Enhanced Rectified Spatial Self-attention (ReSSA), Enhanced Rectified Delay Aware Self-attention (ReDASA), and Enhanced Rectified Temporal Self-attention (ReTSA). These modules aim to lower computational needs via sparse attention, focus on local information for better traffic dynamics understanding, and merge spatial and temporal insights through a unique learning method. Extensive tests on six real-world datasets highlight CCDSReFormer's superior performance. An ablation study also confirms the significant impact of each component on the model's predictive accuracy, showcasing our model's ability to forecast traffic flow effectively.

Title: Constructions Are So Difficult That Even Large Language Models Get Them Right for the Wrong Reasons

• Authors: Shijia Zhou, Leonie Weissweiler, Taiqi He, Hinrich Schütze, David R. Mortensen, Lori Levin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.17760
• Pdf URL: https://arxiv.org/pdf/2403.17760
• Copy Paste: [[2403.17760]] Constructions Are So Difficult That Even Large Language Models Get Them Right for the Wrong Reasons(https://arxiv.org/abs/2403.17760)
• Keywords: large language model
• Abstract: In this paper, we make a contribution that can be understood from two perspectives: from an NLP perspective, we introduce a small challenge dataset for NLI with large lexical overlap, which minimises the possibility of models discerning entailment solely based on token distinctions, and show that GPT-4 and Llama 2 fail it with strong bias. We then create further challenging sub-tasks in an effort to explain this failure. From a Computational Linguistics perspective, we identify a group of constructions with three classes of adjectives which cannot be distinguished by surface features. This enables us to probe for LLM's understanding of these constructions in various ways, and we find that they fail in a variety of ways to distinguish between them, suggesting that they don't adequately represent their meaning or capture the lexical properties of phrasal heads.

Title: Makeup Prior Models for 3D Facial Makeup Estimation and Applications

• Authors: Xingchao Yang, Takafumi Taketomi, Yuki Endo, Yoshihiro Kanamori
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2403.17761
• Pdf URL: https://arxiv.org/pdf/2403.17761
• Copy Paste: [[2403.17761]] Makeup Prior Models for 3D Facial Makeup Estimation and Applications(https://arxiv.org/abs/2403.17761)
• Keywords: robust
• Abstract: In this work, we introduce two types of makeup prior models to extend existing 3D face prior models: PCA-based and StyleGAN2-based priors. The PCA-based prior model is a linear model that is easy to construct and is computationally efficient. However, it retains only low-frequency information. Conversely, the StyleGAN2-based model can represent high-frequency information with relatively higher computational cost than the PCA-based model. Although there is a trade-off between the two models, both are applicable to 3D facial makeup estimation and related applications. By leveraging makeup prior models and designing a makeup consistency module, we effectively address the challenges that previous methods faced in robustly estimating makeup, particularly in the context of handling self-occluded faces. In experiments, we demonstrate that our approach reduces computational costs by several orders of magnitude, achieving speeds up to 180 times faster. In addition, by improving the accuracy of the estimated makeup, we confirm that our methods are highly advantageous for various 3D facial makeup applications such as 3D makeup face reconstruction, user-friendly makeup editing, makeup transfer, and interpolation.

Title: Secure Aggregation is Not Private Against Membership Inference Attacks

• Authors: Khac-Hoang Ngo, Johan Östman, Giuseppe Durisi, Alexandre Graell i Amat
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2403.17775
• Pdf URL: https://arxiv.org/pdf/2403.17775
• Copy Paste: [[2403.17775]] Secure Aggregation is Not Private Against Membership Inference Attacks(https://arxiv.org/abs/2403.17775)
• Keywords: secure, privacy, attack, membership infer, federate
• Abstract: Secure aggregation (SecAgg) is a commonly-used privacy-enhancing mechanism in federated learning, affording the server access only to the aggregate of model updates while safeguarding the confidentiality of individual updates. Despite widespread claims regarding SecAgg's privacy-preserving capabilities, a formal analysis of its privacy is lacking, making such presumptions unjustified. In this paper, we delve into the privacy implications of SecAgg by treating it as a local differential privacy (LDP) mechanism for each local update. We design a simple attack wherein an adversarial server seeks to discern which update vector a client submitted, out of two possible ones, in a single training round of federated learning under SecAgg. By conducting privacy auditing, we assess the success probability of this attack and quantify the LDP guarantees provided by SecAgg. Our numerical results unveil that, contrary to prevailing claims, SecAgg offers weak privacy against membership inference attacks even in a single training round. Indeed, it is difficult to hide a local update by adding other independent local updates when the updates are of high dimension. Our findings underscore the imperative for additional privacy-enhancing mechanisms, such as noise injection, in federated learning.

Title: GenesisTex: Adapting Image Denoising Diffusion to Texture Space

• Authors: Chenjian Gao, Boyan Jiang, Xinghui Li, Yingpeng Zhang, Qian Yu
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2403.17782
• Pdf URL: https://arxiv.org/pdf/2403.17782
• Copy Paste: [[2403.17782]] GenesisTex: Adapting Image Denoising Diffusion to Texture Space(https://arxiv.org/abs/2403.17782)
• Keywords: diffusion
• Abstract: We present GenesisTex, a novel method for synthesizing textures for 3D geometries from text descriptions. GenesisTex adapts the pretrained image diffusion model to texture space by texture space sampling. Specifically, we maintain a latent texture map for each viewpoint, which is updated with predicted noise on the rendering of the corresponding viewpoint. The sampled latent texture maps are then decoded into a final texture map. During the sampling process, we focus on both global and local consistency across multiple viewpoints: global consistency is achieved through the integration of style consistency mechanisms within the noise prediction network, and low-level consistency is achieved by dynamically aligning latent textures. Finally, we apply reference-based inpainting and img2img on denser views for texture refinement. Our approach overcomes the limitations of slow optimization in distillation-based methods and instability in inpainting-based methods. Experiments on meshes from various sources demonstrate that our method surpasses the baseline methods quantitatively and qualitatively.

Title: Improving Text-to-Image Consistency via Automatic Prompt Optimization

• Authors: Oscar Mañas, Pietro Astolfi, Melissa Hall, Candace Ross, Jack Urbanek, Adina Williams, Aishwarya Agrawal, Adriana Romero-Soriano, Michal Drozdzal
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2403.17804
• Pdf URL: https://arxiv.org/pdf/2403.17804
• Copy Paste: [[2403.17804]] Improving Text-to-Image Consistency via Automatic Prompt Optimization(https://arxiv.org/abs/2403.17804)
• Keywords: robust, generative, large language model
• Abstract: Impressive advances in text-to-image (T2I) generative models have yielded a plethora of high performing models which are able to generate aesthetically appealing, photorealistic images. Despite the progress, these models still struggle to produce images that are consistent with the input prompt, oftentimes failing to capture object quantities, relations and attributes properly. Existing solutions to improve prompt-image consistency suffer from the following challenges: (1) they oftentimes require model fine-tuning, (2) they only focus on nearby prompt samples, and (3) they are affected by unfavorable trade-offs among image quality, representation diversity, and prompt-image consistency. In this paper, we address these challenges and introduce a T2I optimization-by-prompting framework, OPT2I, which leverages a large language model (LLM) to improve prompt-image consistency in T2I models. Our framework starts from a user prompt and iteratively generates revised prompts with the goal of maximizing a consistency score. Our extensive validation on two datasets, MSCOCO and PartiPrompts, shows that OPT2I can boost the initial consistency score by up to 24.9% in terms of DSG score while preserving the FID and increasing the recall between generated and real data. Our work paves the way toward building more reliable and robust T2I systems by harnessing the power of LLMs.

Title: Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanisms

• Authors: Michael Hanna, Sandro Pezzelle, Yonatan Belinkov
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2403.17806
• Pdf URL: https://arxiv.org/pdf/2403.17806
• Copy Paste: [[2403.17806]] Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanisms(https://arxiv.org/abs/2403.17806)
• Keywords: interpretability
• Abstract: Many recent language model (LM) interpretability studies have adopted the circuits framework, which aims to find the minimal computational subgraph, or circuit, that explains LM behavior on a given task. Most studies determine which edges belong in a LM's circuit by performing causal interventions on each edge independently, but this scales poorly with model size. Edge attribution patching (EAP), gradient-based approximation to interventions, has emerged as a scalable but imperfect solution to this problem. In this paper, we introduce a new method - EAP with integrated gradients (EAP-IG) - that aims to better maintain a core property of circuits: faithfulness. A circuit is faithful if all model edges outside the circuit can be ablated without changing the model's performance on the task; faithfulness is what justifies studying circuits, rather than the full model. Our experiments demonstrate that circuits found using EAP are less faithful than those found using EAP-IG, even though both have high node overlap with circuits found previously using causal interventions. We conclude more generally that when using circuits to compare the mechanisms models use to solve tasks, faithfulness, not overlap, is what should be measured.

Title: Are Compressed Language Models Less Subgroup Robust?

• Authors: Leonidas Gee, Andrea Zugarini, Novi Quadrianto
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2403.17811
• Pdf URL: https://arxiv.org/pdf/2403.17811
• Copy Paste: [[2403.17811]] Are Compressed Language Models Less Subgroup Robust?(https://arxiv.org/abs/2403.17811)
• Keywords: robust, large language model
• Abstract: To reduce the inference cost of large language models, model compression is increasingly used to create smaller scalable models. However, little is known about their robustness to minority subgroups defined by the labels and attributes of a dataset. In this paper, we investigate the effects of 18 different compression methods and settings on the subgroup robustness of BERT language models. We show that worst-group performance does not depend on model size alone, but also on the compression method used. Additionally, we find that model compression does not always worsen the performance on minority subgroups. Altogether, our analysis serves to further research into the subgroup robustness of model compression.

Title: DN-Splatter: Depth and Normal Priors for Gaussian Splatting and Meshing

• Authors: Matias Turkulainen, Xuqian Ren, Iaroslav Melekhov, Otto Seiskari, Esa Rahtu, Juho Kannala
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.17822
• Pdf URL: https://arxiv.org/pdf/2403.17822
• Copy Paste: [[2403.17822]] DN-Splatter: Depth and Normal Priors for Gaussian Splatting and Meshing(https://arxiv.org/abs/2403.17822)
• Keywords: extraction
• Abstract: 3D Gaussian splatting, a novel differentiable rendering technique, has achieved state-of-the-art novel view synthesis results with high rendering speeds and relatively low training times. However, its performance on scenes commonly seen in indoor datasets is poor due to the lack of geometric constraints during optimization. We extend 3D Gaussian splatting with depth and normal cues to tackle challenging indoor datasets and showcase techniques for efficient mesh extraction, an important downstream application. Specifically, we regularize the optimization procedure with depth information, enforce local smoothness of nearby Gaussians, and use the geometry of the 3D Gaussians supervised by normal cues to achieve better alignment with the true scene geometry. We improve depth estimation and novel view synthesis results over baselines and show how this simple yet effective regularization technique can be used to directly extract meshes from the Gaussian representation yielding more physically accurate reconstructions on indoor scenes. Our code will be released in https://github.com/maturk/dn-splatter.

Title: DiffH2O: Diffusion-Based Synthesis of Hand-Object Interactions from Textual Descriptions

• Authors: Sammy Christen, Shreyas Hampali, Fadime Sener, Edoardo Remelli, Tomas Hodan, Eric Sauser, Shugao Ma, Bugra Tekin
• Subjects: cs.CV, cs.AI, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.17827
• Pdf URL: https://arxiv.org/pdf/2403.17827
• Copy Paste: [[2403.17827]] DiffH2O: Diffusion-Based Synthesis of Hand-Object Interactions from Textual Descriptions(https://arxiv.org/abs/2403.17827)
• Keywords: diffusion
• Abstract: Generating natural hand-object interactions in 3D is challenging as the resulting hand and object motions are expected to be physically plausible and semantically meaningful. Furthermore, generalization to unseen objects is hindered by the limited scale of available hand-object interaction datasets. We propose DiffH2O, a novel method to synthesize realistic, one or two-handed object interactions from provided text prompts and geometry of the object. The method introduces three techniques that enable effective learning from limited data. First, we decompose the task into a grasping stage and a text-based interaction stage and use separate diffusion models for each. In the grasping stage, the model only generates hand motions, whereas in the interaction phase both hand and object poses are synthesized. Second, we propose a compact representation that tightly couples hand and object poses. Third, we propose two different guidance schemes to allow more control of the generated motions: grasp guidance and detailed textual guidance. Grasp guidance takes a single target grasping pose and guides the diffusion model to reach this grasp at the end of the grasping stage, which provides control over the grasping pose. Given a grasping motion from this stage, multiple different actions can be prompted in the interaction phase. For textual guidance, we contribute comprehensive text descriptions to the GRAB dataset and show that they enable our method to have more fine-grained control over hand-object interactions. Our quantitative and qualitative evaluation demonstrates that the proposed method outperforms baseline methods and leads to natural hand-object motions. Moreover, we demonstrate the practicality of our framework by utilizing a hand pose estimate from an off-the-shelf pose estimator for guidance, and then sampling multiple different actions in the interaction stage.

Title: Assessment of Multimodal Large Language Models in Alignment with Human Values

• Authors: Zhelun Shi, Zhipin Wang, Hongxing Fan, Zaibin Zhang, Lijun Li, Yongting Zhang, Zhenfei Yin, Lu Sheng, Yu Qiao, Jing Shao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.17830
• Pdf URL: https://arxiv.org/pdf/2403.17830
• Copy Paste: [[2403.17830]] Assessment of Multimodal Large Language Models in Alignment with Human Values(https://arxiv.org/abs/2403.17830)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) aim to serve as versatile assistants aligned with human values, as defined by the principles of being helpful, honest, and harmless (hhh). However, in terms of Multimodal Large Language Models (MLLMs), despite their commendable performance in perception and reasoning tasks, their alignment with human values remains largely unexplored, given the complexity of defining hhh dimensions in the visual world and the difficulty in collecting relevant data that accurately mirrors real-world situations. To address this gap, we introduce Ch3Ef, a Compreh3ensive Evaluation dataset and strategy for assessing alignment with human expectations. Ch3Ef dataset contains 1002 human-annotated data samples, covering 12 domains and 46 tasks based on the hhh principle. We also present a unified evaluation strategy supporting assessment across various scenarios and different perspectives. Based on the evaluation results, we summarize over 10 key findings that deepen the understanding of MLLM capabilities, limitations, and the dynamic relationships between evaluation levels, guiding future advancements in the field.

Title: GPFL: A Gradient Projection-Based Client Selection Framework for Efficient Federated Learning

• Authors: Shijie Na, Yuzhi Liang, Siu-Ming Yiu
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2403.17833
• Pdf URL: https://arxiv.org/pdf/2403.17833
• Copy Paste: [[2403.17833]] GPFL: A Gradient Projection-Based Client Selection Framework for Efficient Federated Learning(https://arxiv.org/abs/2403.17833)
• Keywords: federate
• Abstract: Federated learning client selection is crucial for determining participant clients while balancing model accuracy and communication efficiency. Existing methods have limitations in handling data heterogeneity, computational burdens, and independent client treatment. To address these challenges, we propose GPFL, which measures client value by comparing local and global descent directions. We also employ an Exploit-Explore mechanism to enhance performance. Experimental results on FEMINST and CIFAR-10 datasets demonstrate that GPFL outperforms baselines in Non-IID scenarios, achieving over 9\% improvement in FEMINST test accuracy. Moreover, GPFL exhibits shorter computation times through pre-selection and parameter reuse in federated learning.

Title: GTA-HDR: A Large-Scale Synthetic Dataset for HDR Image Reconstruction

• Authors: Hrishav Bakul Barua, Kalin Stefanov, KokSheik Wong, Abhinav Dhall, Ganesh Krishnasamy
• Subjects: cs.CV, cs.GR, cs.LG, cs.MM, eess.IV
• Abstract URL: https://arxiv.org/abs/2403.17837
• Pdf URL: https://arxiv.org/pdf/2403.17837
• Copy Paste: [[2403.17837]] GTA-HDR: A Large-Scale Synthetic Dataset for HDR Image Reconstruction(https://arxiv.org/abs/2403.17837)
• Keywords: segmentation
• Abstract: High Dynamic Range (HDR) content (i.e., images and videos) has a broad range of applications. However, capturing HDR content from real-world scenes is expensive and time- consuming. Therefore, the challenging task of reconstructing visually accurate HDR images from their Low Dynamic Range (LDR) counterparts is gaining attention in the vision research community. A major challenge in this research problem is the lack of datasets, which capture diverse scene conditions (e.g., lighting, shadows, weather, locations, landscapes, objects, humans, buildings) and various image features (e.g., color, contrast, saturation, hue, luminance, brightness, radiance). To address this gap, in this paper, we introduce GTA-HDR, a large-scale synthetic dataset of photo-realistic HDR images sampled from the GTA-V video game. We perform thorough evaluation of the proposed dataset, which demonstrates significant qualitative and quantitative improvements of the state-of-the-art HDR image reconstruction methods. Furthermore, we demonstrate the effectiveness of the proposed dataset and its impact on additional computer vision tasks including 3D human pose estimation, human body part segmentation, and holistic scene segmentation. The dataset, data collection pipeline, and evaluation code are available at: https://github.com/HrishavBakulBarua/GTA-HDR.

Title: ReMamber: Referring Image Segmentation with Mamba Twister

• Authors: Yuhuan Yang, Chaofan Ma, Jiangchao Yao, Zhun Zhong, Ya Zhang, Yanfeng Wang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.17839
• Pdf URL: https://arxiv.org/pdf/2403.17839
• Copy Paste: [[2403.17839]] ReMamber: Referring Image Segmentation with Mamba Twister(https://arxiv.org/abs/2403.17839)
• Keywords: transformer, segmentation
• Abstract: Referring Image Segmentation (RIS) leveraging transformers has achieved great success on the interpretation of complex visual-language tasks. However, the quadratic computation cost makes it resource-consuming in capturing long-range visual-language dependencies. Fortunately, Mamba addresses this with efficient linear complexity in processing. However, directly applying Mamba to multi-modal interactions presents challenges, primarily due to inadequate channel interactions for the effective fusion of multi-modal data. In this paper, we propose ReMamber, a novel RIS architecture that integrates the power of Mamba with a multi-modal Mamba Twister block. The Mamba Twister explicitly models image-text interaction, and fuses textual and visual features through its unique channel and spatial twisting mechanism. We achieve the state-of-the-art on three challenging benchmarks. Moreover, we conduct thorough analyses of ReMamber and discuss other fusion designs using Mamba. These provide valuable perspectives for future research.

Title: Mechanistic Design and Scaling of Hybrid Architectures

• Authors: Michael Poli, Armin W Thomas, Eric Nguyen, Pragaash Ponnusamy, Björn Deiseroth, Kristian Kersting, Taiji Suzuki, Brian Hie, Stefano Ermon, Christopher Ré, Ce Zhang, Stefano Massaroli
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.17844
• Pdf URL: https://arxiv.org/pdf/2403.17844
• Copy Paste: [[2403.17844]] Mechanistic Design and Scaling of Hybrid Architectures(https://arxiv.org/abs/2403.17844)
• Keywords: transformer
• Abstract: The development of deep learning architectures is a resource-demanding process, due to a vast design space, long prototyping times, and high compute costs associated with at-scale model training and evaluation. We set out to simplify this process by grounding it in an end-to-end mechanistic architecture design (MAD) pipeline, encompassing small-scale capability unit tests predictive of scaling laws. Through a suite of synthetic token manipulation tasks such as compression and recall, designed to probe capabilities, we identify and test new hybrid architectures constructed from a variety of computational primitives. We experimentally validate the resulting architectures via an extensive compute-optimal and a new state-optimal scaling law analysis, training over 500 language models between 70M to 7B parameters. Surprisingly, we find MAD synthetics to correlate with compute-optimal perplexity, enabling accurate evaluation of new architectures via isolated proxy tasks. The new architectures found via MAD, based on simple ideas such as hybridization and sparsity, outperform state-of-the-art Transformer, convolutional, and recurrent architectures (Transformer++, Hyena, Mamba) in scaling, both at compute-optimal budgets and in overtrained regimes. Overall, these results provide evidence that performance on curated synthetic tasks can be predictive of scaling laws, and that an optimal architecture should leverage specialized layers via a hybrid topology.

Title: ArabicaQA: A Comprehensive Dataset for Arabic Question Answering

• Authors: Abdelrahman Abdallah, Mahmoud Kasem, Mahmoud Abdalla, Mohamed Mahmoud, Mohamed Elkasaby, Yasser Elbendary, Adam Jatowt
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2403.17848
• Pdf URL: https://arxiv.org/pdf/2403.17848
• Copy Paste: [[2403.17848]] ArabicaQA: A Comprehensive Dataset for Arabic Question Answering(https://arxiv.org/abs/2403.17848)
• Keywords: large language model
• Abstract: In this paper, we address the significant gap in Arabic natural language processing (NLP) resources by introducing ArabicaQA, the first large-scale dataset for machine reading comprehension and open-domain question answering in Arabic. This comprehensive dataset, consisting of 89,095 answerable and 3,701 unanswerable questions created by crowdworkers to look similar to answerable ones, along with additional labels of open-domain questions marks a crucial advancement in Arabic NLP resources. We also present AraDPR, the first dense passage retrieval model trained on the Arabic Wikipedia corpus, specifically designed to tackle the unique challenges of Arabic text retrieval. Furthermore, our study includes extensive benchmarking of large language models (LLMs) for Arabic question answering, critically evaluating their performance in the Arabic language context. In conclusion, ArabicaQA, AraDPR, and the benchmarking of LLMs in Arabic question answering offer significant advancements in the field of Arabic NLP. The dataset and code are publicly accessible for further research https://github.com/DataScienceUIBK/ArabicaQA.

Title: Counterfactual Fairness through Transforming Data Orthogonal to Bias

• Authors: Shuyi Chen, Shixiang Zhu
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2403.17852
• Pdf URL: https://arxiv.org/pdf/2403.17852
• Copy Paste: [[2403.17852]] Counterfactual Fairness through Transforming Data Orthogonal to Bias(https://arxiv.org/abs/2403.17852)
• Keywords: fair
• Abstract: Machine learning models have shown exceptional prowess in solving complex issues across various domains. Nonetheless, these models can sometimes exhibit biased decision-making, leading to disparities in treatment across different groups. Despite the extensive research on fairness, the nuanced effects of multivariate and continuous sensitive variables on decision-making outcomes remain insufficiently studied. We introduce a novel data pre-processing algorithm, Orthogonal to Bias (OB), designed to remove the influence of a group of continuous sensitive variables, thereby facilitating counterfactual fairness in machine learning applications. Our approach is grounded in the assumption of a jointly normal distribution within a structural causal model (SCM), proving that counterfactual fairness can be achieved by ensuring the data is uncorrelated with sensitive variables. The OB algorithm is model-agnostic, catering to a wide array of machine learning models and tasks, and includes a sparse variant to enhance numerical stability through regularization. Through empirical evaluation on simulated and real-world datasets - including the adult income and the COMPAS recidivism datasets - our methodology demonstrates its capacity to enable fairer outcomes without compromising accuracy.

Title: Using Domain Knowledge to Guide Dialog Structure Induction via Neural Probabilistic Soft Logic

• Authors: Connor Pryor, Quan Yuan, Jeremiah Liu, Mehran Kazemi, Deepak Ramachandran, Tania Bedrax-Weiss, Lise Getoor
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.17853
• Pdf URL: https://arxiv.org/pdf/2403.17853
• Copy Paste: [[2403.17853]] Using Domain Knowledge to Guide Dialog Structure Induction via Neural Probabilistic Soft Logic(https://arxiv.org/abs/2403.17853)
• Keywords: generative
• Abstract: Dialog Structure Induction (DSI) is the task of inferring the latent dialog structure (i.e., a set of dialog states and their temporal transitions) of a given goal-oriented dialog. It is a critical component for modern dialog system design and discourse analysis. Existing DSI approaches are often purely data-driven, deploy models that infer latent states without access to domain knowledge, underperform when the training corpus is limited/noisy, or have difficulty when test dialogs exhibit distributional shifts from the training domain. This work explores a neural-symbolic approach as a potential solution to these problems. We introduce Neural Probabilistic Soft Logic Dialogue Structure Induction (NEUPSL DSI), a principled approach that injects symbolic knowledge into the latent space of a generative neural model. We conduct a thorough empirical investigation on the effect of NEUPSL DSI learning on hidden representation quality, few-shot learning, and out-of-domain generalization performance. Over three dialog structure induction datasets and across unsupervised and semi-supervised settings for standard and cross-domain generalization, the injection of symbolic knowledge using NEUPSL DSI provides a consistent boost in performance over the canonical baselines.

Title: Verbing Weirds Language (Models): Evaluation of English Zero-Derivation in Five LLMs

• Authors: David R. Mortensen, Valentina Izrailevitch, Yunze Xiao, Hinrich Schütze, Leonie Weissweiler
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.17856
• Pdf URL: https://arxiv.org/pdf/2403.17856
• Copy Paste: [[2403.17856]] Verbing Weirds Language (Models): Evaluation of English Zero-Derivation in Five LLMs(https://arxiv.org/abs/2403.17856)
• Keywords: large language model
• Abstract: Lexical-syntactic flexibility, in the form of conversion (or zero-derivation) is a hallmark of English morphology. In conversion, a word with one part of speech is placed in a non-prototypical context, where it is coerced to behave as if it had a different part of speech. However, while this process affects a large part of the English lexicon, little work has been done to establish the degree to which language models capture this type of generalization. This paper reports the first study on the behavior of large language models with reference to conversion. We design a task for testing lexical-syntactic flexibility -- the degree to which models can generalize over words in a construction with a non-prototypical part of speech. This task is situated within a natural language inference paradigm. We test the abilities of five language models -- two proprietary models (GPT-3.5 and GPT-4), three open-source models (Mistral 7B, Falcon 40B, and Llama 2 70B). We find that GPT-4 performs best on the task, followed by GPT-3.5, but that the open source language models are also able to perform it and that the 7B parameter Mistral displays as little difference between its baseline performance on the natural language inference task and the non-prototypical syntactic category task, as the massive GPT-4.

Title: ChroniclingAmericaQA: A Large-scale Question Answering Dataset based on Historical American Newspaper Pages

• Authors: Bhawna Piryani, Jamshid Mozafari, Adam Jatowt
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.17859
• Pdf URL: https://arxiv.org/pdf/2403.17859
• Copy Paste: [[2403.17859]] ChroniclingAmericaQA: A Large-scale Question Answering Dataset based on Historical American Newspaper Pages(https://arxiv.org/abs/2403.17859)
• Keywords: large language model
• Abstract: Question answering (QA) and Machine Reading Comprehension (MRC) tasks have significantly advanced in recent years due to the rapid development of deep learning techniques and, more recently, large language models. At the same time, many benchmark datasets have become available for QA and MRC tasks. However, most existing large-scale benchmark datasets have been created predominantly using synchronous document collections like Wikipedia or the Web. Archival document collections, such as historical newspapers, contain valuable information from the past that is still not widely used to train large language models. To further contribute to advancing QA and MRC tasks and to overcome the limitation of previous datasets, we introduce ChroniclingAmericaQA, a large-scale dataset with 485K question-answer pairs created based on the historical newspaper collection Chronicling America. Our dataset is constructed from a subset of the Chronicling America newspaper collection spanning 120 years. One of the significant challenges for utilizing digitized historical newspaper collections is the low quality of OCR text. Therefore, to enable realistic testing of QA models, our dataset can be used in three different ways: answering questions from raw and noisy content, answering questions from cleaner, corrected version of the content, as well as answering questions from scanned images of newspaper pages. This and the fact that ChroniclingAmericaQA spans the longest time period among available QA datasets make it quite a unique and useful resource.

Title: Exploring LLMs as a Source of Targeted Synthetic Textual Data to Minimize High Confidence Misclassifications

• Authors: Philip Lippmann, Matthijs Spaan, Jie Yang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.17860
• Pdf URL: https://arxiv.org/pdf/2403.17860
• Copy Paste: [[2403.17860]] Exploring LLMs as a Source of Targeted Synthetic Textual Data to Minimize High Confidence Misclassifications(https://arxiv.org/abs/2403.17860)
• Keywords: large language model
• Abstract: Natural Language Processing (NLP) models optimized for predictive performance often make high confidence errors and suffer from vulnerability to adversarial and out-of-distribution data. Existing work has mainly focused on mitigation of such errors using either humans or an automated approach. In this study, we explore the usage of large language models (LLMs) for data augmentation as a potential solution to the issue of NLP models making wrong predictions with high confidence during classification tasks. We compare the effectiveness of synthetic data generated by LLMs with that of human data obtained via the same procedure. For mitigation, humans or LLMs provide natural language characterizations of high confidence misclassifications to generate synthetic data, which are then used to extend the training set. We conduct an extensive evaluation of our approach on three classification tasks and demonstrate its effectiveness in reducing the number of high confidence misclassifications present in the model, all while maintaining the same level of accuracy. Moreover, we find that the cost gap between humans and LLMs surpasses an order of magnitude, as LLMs attain human-like performance while being more scalable.

Title: Boosting Diffusion Models with Moving Average Sampling in Frequency Domain

• Authors: Yurui Qian, Qi Cai, Yingwei Pan, Yehao Li, Ting Yao, Qibin Sun, Tao Mei
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2403.17870
• Pdf URL: https://arxiv.org/pdf/2403.17870
• Copy Paste: [[2403.17870]] Boosting Diffusion Models with Moving Average Sampling in Frequency Domain(https://arxiv.org/abs/2403.17870)
• Keywords: diffusion, generative
• Abstract: Diffusion models have recently brought a powerful revolution in image generation. Despite showing impressive generative capabilities, most of these models rely on the current sample to denoise the next one, possibly resulting in denoising instability. In this paper, we reinterpret the iterative denoising process as model optimization and leverage a moving average mechanism to ensemble all the prior samples. Instead of simply applying moving average to the denoised samples at different timesteps, we first map the denoised samples to data space and then perform moving average to avoid distribution shift across timesteps. In view that diffusion models evolve the recovery from low-frequency components to high-frequency details, we further decompose the samples into different frequency components and execute moving average separately on each component. We name the complete approach "Moving Average Sampling in Frequency domain (MASF)". MASF could be seamlessly integrated into mainstream pre-trained diffusion models and sampling schedules. Extensive experiments on both unconditional and conditional diffusion models demonstrate that our MASF leads to superior performances compared to the baselines, with almost negligible additional complexity cost.

Title: Empowering Data Mesh with Federated Learning

• Authors: Haoyuan Li, Salman Toor
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2403.17878
• Pdf URL: https://arxiv.org/pdf/2403.17878
• Copy Paste: [[2403.17878]] Empowering Data Mesh with Federated Learning(https://arxiv.org/abs/2403.17878)
• Keywords: security, privacy, federate
• Abstract: The evolution of data architecture has seen the rise of data lakes, aiming to solve the bottlenecks of data management and promote intelligent decision-making. However, this centralized architecture is limited by the proliferation of data sources and the growing demand for timely analysis and processing. A new data paradigm, Data Mesh, is proposed to overcome these challenges. Data Mesh treats domains as a first-class concern by distributing the data ownership from the central team to each data domain, while keeping the federated governance to monitor domains and their data products. Many multi-million dollar organizations like Paypal, Netflix, and Zalando have already transformed their data analysis pipelines based on this new architecture. In this decentralized architecture where data is locally preserved by each domain team, traditional centralized machine learning is incapable of conducting effective analysis across multiple domains, especially for security-sensitive organizations. To this end, we introduce a pioneering approach that incorporates Federated Learning into Data Mesh. To the best of our knowledge, this is the first open-source applied work that represents a critical advancement toward the integration of federated learning methods into the Data Mesh paradigm, underscoring the promising prospects for privacy-preserving and decentralized data analysis strategies within Data Mesh architecture.

Title: Deepfake Generation and Detection: A Benchmark and Survey

• Authors: Gan Pei, Jiangning Zhang, Menghan Hu, Guangtao Zhai, Chengjie Wang, Zhenyu Zhang, Jian Yang, Chunhua Shen, Dacheng Tao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.17881
• Pdf URL: https://arxiv.org/pdf/2403.17881
• Copy Paste: [[2403.17881]] Deepfake Generation and Detection: A Benchmark and Survey(https://arxiv.org/abs/2403.17881)
• Keywords: privacy, attack
• Abstract: In addition to the advancements in deepfake generation, corresponding detection technologies need to continuously evolve to regulate the potential misuse of deepfakes, such as for privacy invasion and phishing attacks. This survey comprehensively reviews the latest developments in deepfake generation and detection, summarizing and analyzing the current state of the art in this rapidly evolving field. We first unify task definitions, comprehensively introduce datasets and metrics, and discuss the development of generation and detection technology frameworks. Then, we discuss the development of several related sub-fields and focus on researching four mainstream deepfake fields: popular face swap, face reenactment, talking face generation, and facial attribute editing, as well as foreign detection. Subsequently, we comprehensively benchmark representative methods on popular datasets for each field, fully evaluating the latest and influential works published in top conferences/journals. Finally, we analyze the challenges and future research directions of the discussed fields. We closely follow the latest developments in https://github.com/flyingby/Awesome-Deepfake-Generation-and-Detection.

Title: Superior and Pragmatic Talking Face Generation with Teacher-Student Framework

• Authors: Chao Liang, Jianwen Jiang, Tianyun Zhong, Gaojie Lin, Zhengkun Rong, Jiaqi Yang, Yongming Zhu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.17883
• Pdf URL: https://arxiv.org/pdf/2403.17883
• Copy Paste: [[2403.17883]] Superior and Pragmatic Talking Face Generation with Teacher-Student Framework(https://arxiv.org/abs/2403.17883)
• Keywords: robust
• Abstract: Talking face generation technology creates talking videos from arbitrary appearance and motion signal, with the "arbitrary" offering ease of use but also introducing challenges in practical applications. Existing methods work well with standard inputs but suffer serious performance degradation with intricate real-world ones. Moreover, efficiency is also an important concern in deployment. To comprehensively address these issues, we introduce SuperFace, a teacher-student framework that balances quality, robustness, cost and editability. We first propose a simple but effective teacher model capable of handling inputs of varying qualities to generate high-quality results. Building on this, we devise an efficient distillation strategy to acquire an identity-specific student model that maintains quality with significantly reduced computational load. Our experiments validate that SuperFace offers a more comprehensive solution than existing methods for the four mentioned objectives, especially in reducing FLOPs by 99\% with the student model. SuperFace can be driven by both video and audio and allows for localized facial attributes editing.

Title: Compressed Multi-task embeddings for Data-Efficient Downstream training and inference in Earth Observation

• Authors: Carlos Gomes, Thomas Brunschwiler
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.17886
• Pdf URL: https://arxiv.org/pdf/2403.17886
• Copy Paste: [[2403.17886]] Compressed Multi-task embeddings for Data-Efficient Downstream training and inference in Earth Observation(https://arxiv.org/abs/2403.17886)
• Keywords: segmentation
• Abstract: As repositories of large scale data in earth observation (EO) have grown, so have transfer and storage costs for model training and inference, expending significant resources. We introduce Neural Embedding Compression (NEC), based on the transfer of compressed embeddings to data consumers instead of raw data. We adapt foundation models (FM) through learned neural compression to generate multi-task embeddings while navigating the tradeoff between compression rate and embedding utility. We update only a small fraction of the FM parameters (10%) for a short training period (1% of the iterations of pre-training). We evaluate NEC on two EO tasks: scene classification and semantic segmentation. Compared with applying traditional compression to the raw data, NEC achieves similar accuracy with a 75% to 90% reduction in data. Even at 99.7% compression, performance drops by only 5% on the scene classification task. Overall, NEC is a data-efficient yet performant approach for multi-task EO modelling.

Title: The Unreasonable Ineffectiveness of the Deeper Layers

• Authors: Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, Daniel A. Roberts
• Subjects: cs.CL, cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2403.17887
• Pdf URL: https://arxiv.org/pdf/2403.17887
• Copy Paste: [[2403.17887]] The Unreasonable Ineffectiveness of the Deeper Layers(https://arxiv.org/abs/2403.17887)
• Keywords: robust
• Abstract: We empirically study a simple layer-pruning strategy for popular families of open-weight pretrained LLMs, finding minimal degradation of performance on different question-answering benchmarks until after a large fraction (up to half) of the layers are removed. To prune these models, we identify the optimal block of layers to prune by considering similarity across layers; then, to "heal" the damage, we perform a small amount of finetuning. In particular, we use parameter-efficient finetuning (PEFT) methods, specifically quantization and Low Rank Adapters (QLoRA), such that each of our experiments can be performed on a single A100 GPU. From a practical perspective, these results suggest that layer pruning methods can complement other PEFT strategies to further reduce computational resources of finetuning on the one hand, and can improve the memory and latency of inference on the other hand. From a scientific perspective, the robustness of these LLMs to the deletion of layers implies either that current pretraining methods are not properly leveraging the parameters in the deeper layers of the network or that the shallow layers play a critical role in storing knowledge.

Title: ELGC-Net: Efficient Local-Global Context Aggregation for Remote Sensing Change Detection

• Authors: Mubashir Noman, Mustansar Fiaz, Hisham Cholakkal, Salman Khan, Fahad Shahbaz Khan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.17909
• Pdf URL: https://arxiv.org/pdf/2403.17909
• Copy Paste: [[2403.17909]] ELGC-Net: Efficient Local-Global Context Aggregation for Remote Sensing Change Detection(https://arxiv.org/abs/2403.17909)
• Keywords: robust, extraction, transformer
• Abstract: Deep learning has shown remarkable success in remote sensing change detection (CD), aiming to identify semantic change regions between co-registered satellite image pairs acquired at distinct time stamps. However, existing convolutional neural network and transformer-based frameworks often struggle to accurately segment semantic change regions. Moreover, transformers-based methods with standard self-attention suffer from quadratic computational complexity with respect to the image resolution, making them less practical for CD tasks with limited training data. To address these issues, we propose an efficient change detection framework, ELGC-Net, which leverages rich contextual information to precisely estimate change regions while reducing the model size. Our ELGC-Net comprises a Siamese encoder, fusion modules, and a decoder. The focus of our design is the introduction of an Efficient Local-Global Context Aggregator module within the encoder, capturing enhanced global context and local spatial information through a novel pooled-transpose (PT) attention and depthwise convolution, respectively. The PT attention employs pooling operations for robust feature extraction and minimizes computational cost with transposed attention. Extensive experiments on three challenging CD datasets demonstrate that ELGC-Net outperforms existing methods. Compared to the recent transformer-based CD approach (ChangeFormer), ELGC-Net achieves a 1.4% gain in intersection over union metric on the LEVIR-CD dataset, while significantly reducing trainable parameters. Our proposed ELGC-Net sets a new state-of-the-art performance in remote sensing change detection benchmarks. Finally, we also introduce ELGC-Net-LW, a lighter variant with significantly reduced computational complexity, suitable for resource-constrained settings, while achieving comparable performance. Project url https://github.com/techmn/elgcnet.

Title: LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning

• Authors: Rui Pan, Xiang Liu, Shizhe Diao, Renjie Pi, Jipeng Zhang, Chi Han, Tong Zhang
• Subjects: cs.LG, cs.AI, cs.CL, math.OC
• Abstract URL: https://arxiv.org/abs/2403.17919
• Pdf URL: https://arxiv.org/pdf/2403.17919
• Copy Paste: [[2403.17919]] LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning(https://arxiv.org/abs/2403.17919)
• Keywords: large language model
• Abstract: The machine learning community has witnessed impressive advancements since the first appearance of large language models (LLMs), yet their huge memory consumption has become a major roadblock to large-scale training. Parameter Efficient Fine-Tuning techniques such as Low-Rank Adaptation (LoRA) have been proposed to alleviate this problem, but their performance still fails to match full parameter training in most large-scale fine-tuning settings. Attempting to complement this deficiency, we investigate layerwise properties of LoRA on fine-tuning tasks and observe an uncommon skewness of weight norms across different layers. Utilizing this key observation, a surprisingly simple training strategy is discovered, which outperforms both LoRA and full parameter training in a wide range of settings with memory costs as low as LoRA. We name it Layerwise Importance Sampled AdamW (LISA), a promising alternative for LoRA, which applies the idea of importance sampling to different layers in LLMs and randomly freeze most middle layers during optimization. Experimental results show that with similar or less GPU memory consumption, LISA surpasses LoRA or even full parameter tuning in downstream fine-tuning tasks, where LISA consistently outperforms LoRA by over $11\%$-$37\%$ in terms of MT-Bench scores. On large models, specifically LLaMA-2-70B, LISA achieves on-par or better performance than LoRA on MT-Bench, GSM8K, and PubMedQA, demonstrating its effectiveness across different domains.

Title: The Need for Speed: Pruning Transformers with One Recipe

• Authors: Samir Khaki, Konstantinos N. Plataniotis
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.17921
• Pdf URL: https://arxiv.org/pdf/2403.17921
• Copy Paste: [[2403.17921]] The Need for Speed: Pruning Transformers with One Recipe(https://arxiv.org/abs/2403.17921)
• Keywords: transformer, segmentation
• Abstract: We introduce the $\textbf{O}$ne-shot $\textbf{P}$runing $\textbf{T}$echnique for $\textbf{I}$nterchangeable $\textbf{N}$etworks ($\textbf{OPTIN}$) framework as a tool to increase the efficiency of pre-trained transformer architectures $\textit{without requiring re-training}$. Recent works have explored improving transformer efficiency, however often incur computationally expensive re-training procedures or depend on architecture-specific characteristics, thus impeding practical wide-scale adoption. To address these shortcomings, the OPTIN framework leverages intermediate feature distillation, capturing the long-range dependencies of model parameters (coined $\textit{trajectory}$), to produce state-of-the-art results on natural language, image classification, transfer learning, and semantic segmentation tasks $\textit{without re-training}$. Given a FLOP constraint, the OPTIN framework will compress the network while maintaining competitive accuracy performance and improved throughput. Particularly, we show a $\leq 2$% accuracy degradation from NLP baselines and a $0.5$% improvement from state-of-the-art methods on image classification at competitive FLOPs reductions. We further demonstrate the generalization of tasks and architecture with comparative performance using Mask2Former for semantic segmentation and cnn-style networks. OPTIN presents one of the first one-shot efficient frameworks for compressing transformer architectures that generalizes well across different class domains, in particular: natural language and image-related tasks, without $\textit{re-training}$.

Title: AID: Attention Interpolation of Text-to-Image Diffusion

• Authors: Qiyuan He, Jinghao Wang, Ziwei Liu, Angela Yao
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.17924
• Pdf URL: https://arxiv.org/pdf/2403.17924
• Copy Paste: [[2403.17924]] AID: Attention Interpolation of Text-to-Image Diffusion(https://arxiv.org/abs/2403.17924)
• Keywords: diffusion, generative
• Abstract: Conditional diffusion models can create unseen images in various settings, aiding image interpolation. Interpolation in latent spaces is well-studied, but interpolation with specific conditions like text or poses is less understood. Simple approaches, such as linear interpolation in the space of conditions, often result in images that lack consistency, smoothness, and fidelity. To that end, we introduce a novel training-free technique named Attention Interpolation via Diffusion (AID). Our key contributions include 1) proposing an inner/outer interpolated attention layer; 2) fusing the interpolated attention with self-attention to boost fidelity; and 3) applying beta distribution to selection to increase smoothness. We also present a variant, Prompt-guided Attention Interpolation via Diffusion (PAID), that considers interpolation as a condition-dependent generative process. This method enables the creation of new images with greater consistency, smoothness, and efficiency, and offers control over the exact path of interpolation. Our approach demonstrates effectiveness for conceptual and spatial interpolation. Code and demo are available at https://github.com/QY-H00/attention-interpolation-diffusion.

Title: Track Everything Everywhere Fast and Robustly

• Authors: Yunzhou Song, Jiahui Lei, Ziyun Wang, Lingjie Liu, Kostas Daniilidis
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.17931
• Pdf URL: https://arxiv.org/pdf/2403.17931
• Copy Paste: [[2403.17931]] Track Everything Everywhere Fast and Robustly(https://arxiv.org/abs/2403.17931)
• Keywords: robust
• Abstract: We propose a novel test-time optimization approach for efficiently and robustly tracking any pixel at any time in a video. The latest state-of-the-art optimization-based tracking technique, OmniMotion, requires a prohibitively long optimization time, rendering it impractical for downstream applications. OmniMotion is sensitive to the choice of random seeds, leading to unstable convergence. To improve efficiency and robustness, we introduce a novel invertible deformation network, CaDeX++, which factorizes the function representation into a local spatial-temporal feature grid and enhances the expressivity of the coupling blocks with non-linear functions. While CaDeX++ incorporates a stronger geometric bias within its architectural design, it also takes advantage of the inductive bias provided by the vision foundation models. Our system utilizes monocular depth estimation to represent scene geometry and enhances the objective by incorporating DINOv2 long-term semantics to regulate the optimization process. Our experiments demonstrate a substantial improvement in training speed (more than \textbf{10 times} faster), robustness, and accuracy in tracking over the SoTA optimization-based method OmniMotion.

Title: AiOS: All-in-One-Stage Expressive Human Pose and Shape Estimation

• Authors: Qingping Sun, Yanjun Wang, Ailing Zeng, Wanqi Yin, Chen Wei, Wenjia Wang, Haiyi Mei, Chi Sing Leung, Ziwei Liu, Lei Yang, Zhongang Cai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.17934
• Pdf URL: https://arxiv.org/pdf/2403.17934
• Copy Paste: [[2403.17934]] AiOS: All-in-One-Stage Expressive Human Pose and Shape Estimation(https://arxiv.org/abs/2403.17934)
• Keywords: transformer
• Abstract: Expressive human pose and shape estimation (a.k.a. 3D whole-body mesh recovery) involves the human body, hand, and expression estimation. Most existing methods have tackled this task in a two-stage manner, first detecting the human body part with an off-the-shelf detection model and inferring the different human body parts individually. Despite the impressive results achieved, these methods suffer from 1) loss of valuable contextual information via cropping, 2) introducing distractions, and 3) lacking inter-association among different persons and body parts, inevitably causing performance degradation, especially for crowded scenes. To address these issues, we introduce a novel all-in-one-stage framework, AiOS, for multiple expressive human pose and shape recovery without an additional human detection step. Specifically, our method is built upon DETR, which treats multi-person whole-body mesh recovery task as a progressive set prediction problem with various sequential detection. We devise the decoder tokens and extend them to our task. Specifically, we first employ a human token to probe a human location in the image and encode global features for each instance, which provides a coarse location for the later transformer block. Then, we introduce a joint-related token to probe the human joint in the image and encoder a fine-grained local feature, which collaborates with the global feature to regress the whole-body mesh. This straightforward but effective model outperforms previous state-of-the-art methods by a 9% reduction in NMVE on AGORA, a 30% reduction in PVE on EHF, a 10% reduction in PVE on ARCTIC, and a 3% reduction in PVE on EgoBody.

Title: OmniVid: A Generative Framework for Universal Video Understanding

• Authors: Junke Wang, Dongdong Chen, Chong Luo, Bo He, Lu Yuan, Zuxuan Wu, Yu-Gang Jiang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.17935
• Pdf URL: https://arxiv.org/pdf/2403.17935
• Copy Paste: [[2403.17935]] OmniVid: A Generative Framework for Universal Video Understanding(https://arxiv.org/abs/2403.17935)
• Keywords: generative
• Abstract: The core of video understanding tasks, such as recognition, captioning, and tracking, is to automatically detect objects or actions in a video and analyze their temporal evolution. Despite sharing a common goal, different tasks often rely on distinct model architectures and annotation formats. In contrast, natural language processing benefits from a unified output space, i.e., text sequences, which simplifies the training of powerful foundational language models, such as GPT-3, with extensive training corpora. Inspired by this, we seek to unify the output space of video understanding tasks by using languages as labels and additionally introducing time and box tokens. In this way, a variety of video tasks could be formulated as video-grounded token generation. This enables us to address various types of video tasks, including classification (such as action recognition), captioning (covering clip captioning, video question answering, and dense video captioning), and localization tasks (such as visual object tracking) within a fully shared encoder-decoder architecture, following a generative framework. Through comprehensive experiments, we demonstrate such a simple and straightforward idea is quite effective and can achieve state-of-the-art or competitive results on seven video benchmarks, providing a novel perspective for more universal video understanding. Code is available at https://github.com/wangjk666/OmniVid.

Title: ConvoFusion: Multi-Modal Conversational Diffusion for Co-Speech Gesture Synthesis

• Authors: Muhammad Hamza Mughal, Rishabh Dabral, Ikhsanul Habibie, Lucia Donatelli, Marc Habermann, Christian Theobalt
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.17936
• Pdf URL: https://arxiv.org/pdf/2403.17936
• Copy Paste: [[2403.17936]] ConvoFusion: Multi-Modal Conversational Diffusion for Co-Speech Gesture Synthesis(https://arxiv.org/abs/2403.17936)
• Keywords: diffusion
• Abstract: Gestures play a key role in human communication. Recent methods for co-speech gesture generation, while managing to generate beat-aligned motions, struggle generating gestures that are semantically aligned with the utterance. Compared to beat gestures that align naturally to the audio signal, semantically coherent gestures require modeling the complex interactions between the language and human motion, and can be controlled by focusing on certain words. Therefore, we present ConvoFusion, a diffusion-based approach for multi-modal gesture synthesis, which can not only generate gestures based on multi-modal speech inputs, but can also facilitate controllability in gesture synthesis. Our method proposes two guidance objectives that allow the users to modulate the impact of different conditioning modalities (e.g. audio vs text) as well as to choose certain words to be emphasized during gesturing. Our method is versatile in that it can be trained either for generating monologue gestures or even the conversational gestures. To further advance the research on multi-party interactive gestures, the DnD Group Gesture dataset is released, which contains 6 hours of gesture data showing 5 people interacting with one another. We compare our method with several recent works and demonstrate effectiveness of our method on a variety of tasks. We urge the reader to watch our supplementary video at our website.

Title: Efficient Video Object Segmentation via Modulated Cross-Attention Memory

• Authors: Abdelrahman Shaker, Syed Talal Wasim, Martin Danelljan, Salman Khan, Ming-Hsuan Yang, Fahad Shahbaz Khan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.17937
• Pdf URL: https://arxiv.org/pdf/2403.17937
• Copy Paste: [[2403.17937]] Efficient Video Object Segmentation via Modulated Cross-Attention Memory(https://arxiv.org/abs/2403.17937)
• Keywords: transformer, segmentation
• Abstract: Recently, transformer-based approaches have shown promising results for semi-supervised video object segmentation. However, these approaches typically struggle on long videos due to increased GPU memory demands, as they frequently expand the memory bank every few frames. We propose a transformer-based approach, named MAVOS, that introduces an optimized and dynamic long-term modulated cross-attention (MCA) memory to model temporal smoothness without requiring frequent memory expansion. The proposed MCA effectively encodes both local and global features at various levels of granularity while efficiently maintaining consistent speed regardless of the video length. Extensive experiments on multiple benchmarks, LVOS, Long-Time Video, and DAVIS 2017, demonstrate the effectiveness of our proposed contributions leading to real-time inference and markedly reduced memory demands without any degradation in segmentation accuracy on long videos. Compared to the best existing transformer-based approach, our MAVOS increases the speed by 7.6x, while significantly reducing the GPU memory by 87% with comparable segmentation performance on short and long video datasets. Notably on the LVOS dataset, our MAVOS achieves a J&F score of 63.3% while operating at 37 frames per second (FPS) on a single V100 GPU. Our code and models will be publicly available at: https://github.com/Amshaker/MAVOS.