2024-01-19

Title: RoleCraft-GLM: Advancing Personalized Role-Playing in Large Language Models

• Authors: Meiling Tao, Xuechen Liang, Tianyu Shi, Lei Yu, Yiting Xie
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.09432
• Pdf URL: https://arxiv.org/pdf/2401.09432
• Copy Paste: [[2401.09432]] RoleCraft-GLM: Advancing Personalized Role-Playing in Large Language Models(https://arxiv.org/abs/2401.09432)
• Keywords: large language model
• Abstract: This study presents RoleCraft-GLM, an innovative framework aimed at enhancing personalized role-playing with Large Language Models (LLMs). RoleCraft-GLM addresses the key issue of lacking personalized interactions in conversational AI, and offers a solution with detailed and emotionally nuanced character portrayals. We contribute a unique conversational dataset that shifts from conventional celebrity-centric characters to diverse, non-celebrity personas, thus enhancing the realism and complexity of language modeling interactions. Additionally, our approach includes meticulous character development, ensuring dialogues are both realistic and emotionally resonant. The effectiveness of RoleCraft-GLM is validated through various case studies, highlighting its versatility and skill in different scenarios. Our framework excels in generating dialogues that accurately reflect characters' personality traits and emotions, thereby boosting user engagement. In conclusion, RoleCraft-GLM marks a significant leap in personalized AI interactions, and paves the way for more authentic and immersive AI-assisted role-playing experiences by enabling more nuanced and emotionally rich dialogues

Title: Object Attribute Matters in Visual Question Answering

• Authors: Peize Li, Qingyi Si, Peng Fu, Zheng Lin, Yan Wang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.09442
• Pdf URL: https://arxiv.org/pdf/2401.09442
• Copy Paste: [[2401.09442]] Object Attribute Matters in Visual Question Answering(https://arxiv.org/abs/2401.09442)
• Keywords: robust
• Abstract: Visual question answering is a multimodal task that requires the joint comprehension of visual and textual information. However, integrating visual and textual semantics solely through attention layers is insufficient to comprehensively understand and align information from both modalities. Intuitively, object attributes can naturally serve as a bridge to unify them, which has been overlooked in previous research. In this paper, we propose a novel VQA approach from the perspective of utilizing object attribute, aiming to achieve better object-level visual-language alignment and multimodal scene understanding. Specifically, we design an attribute fusion module and a contrastive knowledge distillation module. The attribute fusion module constructs a multimodal graph neural network to fuse attributes and visual features through message passing. The enhanced object-level visual features contribute to solving fine-grained problem like counting-question. The better object-level visual-language alignment aids in understanding multimodal scenes, thereby improving the model's robustness. Furthermore, to augment scene understanding and the out-of-distribution performance, the contrastive knowledge distillation module introduces a series of implicit knowledge. We distill knowledge into attributes through contrastive loss, which further strengthens the representation learning of attribute features and facilitates visual-linguistic alignment. Intensive experiments on six datasets, COCO-QA, VQAv2, VQA-CPv2, VQA-CPv1, VQAvs and TDIUC, show the superiority of the proposed method.

Title: Incorporating Riemannian Geometric Features for Learning Coefficient of Pressure Distributions on Airplane Wings

• Authors: Liwei Hu, Wenyong Wang, Yu Xiang, Stefan Sommer
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.09452
• Pdf URL: https://arxiv.org/pdf/2401.09452
• Copy Paste: [[2401.09452]] Incorporating Riemannian Geometric Features for Learning Coefficient of Pressure Distributions on Airplane Wings(https://arxiv.org/abs/2401.09452)
• Keywords: attack
• Abstract: The aerodynamic coefficients of aircrafts are significantly impacted by its geometry, especially when the angle of attack (AoA) is large. In the field of aerodynamics, traditional polynomial-based parameterization uses as few parameters as possible to describe the geometry of an airfoil. However, because the 3D geometry of a wing is more complicated than the 2D airfoil, polynomial-based parameterizations have difficulty in accurately representing the entire shape of a wing in 3D space. Existing deep learning-based methods can extract massive latent neural representations for the shape of 2D airfoils or 2D slices of wings. Recent studies highlight that directly taking geometric features as inputs to the neural networks can improve the accuracy of predicted aerodynamic coefficients. Motivated by geometry theory, we propose to incorporate Riemannian geometric features for learning Coefficient of Pressure (CP) distributions on wing surfaces. Our method calculates geometric features (Riemannian metric, connection, and curvature) and further inputs the geometric features, coordinates and flight conditions into a deep learning model to predict the CP distribution. Experimental results show that our method, compared to state-of-the-art Deep Attention Network (DAN), reduces the predicted mean square error (MSE) of CP by an average of 8.41% for the DLR-F11 aircraft test set.

Title: Voila-A: Aligning Vision-Language Models with User's Gaze Attention

• Authors: Kun Yan, Lei Ji, Zeyu Wang, Yuntao Wang, Nan Duan, Shuai Ma
• Subjects: cs.CV, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.09454
• Pdf URL: https://arxiv.org/pdf/2401.09454
• Copy Paste: [[2401.09454]] Voila-A: Aligning Vision-Language Models with User's Gaze Attention(https://arxiv.org/abs/2401.09454)
• Keywords: interpretability
• Abstract: In recent years, the integration of vision and language understanding has led to significant advancements in artificial intelligence, particularly through Vision-Language Models (VLMs). However, existing VLMs face challenges in handling real-world applications with complex scenes and multiple objects, as well as aligning their focus with the diverse attention patterns of human users. In this paper, we introduce gaze information, feasibly collected by AR or VR devices, as a proxy for human attention to guide VLMs and propose a novel approach, Voila-A, for gaze alignment to enhance the interpretability and effectiveness of these models in real-world applications. First, we collect hundreds of minutes of gaze data to demonstrate that we can mimic human gaze modalities using localized narratives. We then design an automatic data annotation pipeline utilizing GPT-4 to generate the VOILA-COCO dataset. Additionally, we innovate the Voila Perceiver modules to integrate gaze information into VLMs while preserving their pretrained knowledge. We evaluate Voila-A using a hold-out validation set and a newly collected VOILA-GAZE Testset, which features real-life scenarios captured with a gaze-tracking device. Our experimental results demonstrate that Voila-A significantly outperforms several baseline models. By aligning model attention with human gaze patterns, Voila-A paves the way for more intuitive, user-centric VLMs and fosters engaging human-AI interaction across a wide range of applications.

Title: Offline Handwriting Signature Verification: A Transfer Learning and Feature Selection Approach

• Authors: Fatih Ozyurt, Jafar Majidpour, Tarik A. Rashid, Canan Koc
• Subjects: cs.CV, cs.AI, eess.IV
• Abstract URL: https://arxiv.org/abs/2401.09467
• Pdf URL: https://arxiv.org/pdf/2401.09467
• Copy Paste: [[2401.09467]] Offline Handwriting Signature Verification: A Transfer Learning and Feature Selection Approach(https://arxiv.org/abs/2401.09467)
• Keywords: security, biometric
• Abstract: Handwritten signature verification poses a formidable challenge in biometrics and document authenticity. The objective is to ascertain the authenticity of a provided handwritten signature, distinguishing between genuine and forged ones. This issue has many applications in sectors such as finance, legal documentation, and security. Currently, the field of computer vision and machine learning has made significant progress in the domain of handwritten signature verification. The outcomes, however, may be enhanced depending on the acquired findings, the structure of the datasets, and the used models. Four stages make up our suggested strategy. First, we collected a large dataset of 12600 images from 420 distinct individuals, and each individual has 30 signatures of a certain kind (All authors signatures are genuine). In the subsequent stage, the best features from each image were extracted using a deep learning model named MobileNetV2. During the feature selection step, three selectors neighborhood component analysis (NCA), Chi2, and mutual info (MI) were used to pull out 200, 300, 400, and 500 features, giving a total of 12 feature vectors. Finally, 12 results have been obtained by applying machine learning techniques such as SVM with kernels (rbf, poly, and linear), KNN, DT, Linear Discriminant Analysis, and Naive Bayes. Without employing feature selection techniques, our suggested offline signature verification achieved a classification accuracy of 91.3%, whereas using the NCA feature selection approach with just 300 features it achieved a classification accuracy of 97.7%. High classification accuracy was achieved using the designed and suggested model, which also has the benefit of being a self-organized framework. Consequently, using the optimum minimally chosen features, the proposed method could identify the best model performance and result validation prediction vectors.

Title: Plug-in for visualizing 3D tool tracking from videos of Minimally Invasive Surgeries

• Authors: Shubhangi Nema, Abhishek Mathur, Leena Vachhani
• Subjects: cs.CV, eess.IV, eess.SY
• Abstract URL: https://arxiv.org/abs/2401.09472
• Pdf URL: https://arxiv.org/pdf/2401.09472
• Copy Paste: [[2401.09472]] Plug-in for visualizing 3D tool tracking from videos of Minimally Invasive Surgeries(https://arxiv.org/abs/2401.09472)
• Keywords: segmentation
• Abstract: This paper tackles instrument tracking and 3D visualization challenges in minimally invasive surgery (MIS), crucial for computer-assisted interventions. Conventional and robot-assisted MIS encounter issues with limited 2D camera projections and minimal hardware integration. The objective is to track and visualize the entire surgical instrument, including shaft and metallic clasper, enabling safe navigation within the surgical environment. The proposed method involves 2D tracking based on segmentation maps, facilitating creation of labeled dataset without extensive ground-truth knowledge. Geometric changes in 2D intervals express motion, and kinematics based algorithms process results into 3D tracking information. Synthesized and experimental results in 2D and 3D motion estimates demonstrate negligible errors, validating the method for labeling and motion tracking of instruments in MIS videos. The conclusion underscores the proposed 2D segmentation technique's simplicity and computational efficiency, emphasizing its potential as direct plug-in for 3D visualization in instrument tracking and MIS practices.

Title: Triamese-ViT: A 3D-Aware Method for Robust Brain Age Estimation from MRIs

• Authors: Zhaonian Zhang, Richard Jiang
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.09475
• Pdf URL: https://arxiv.org/pdf/2401.09475
• Copy Paste: [[2401.09475]] Triamese-ViT: A 3D-Aware Method for Robust Brain Age Estimation from MRIs(https://arxiv.org/abs/2401.09475)
• Keywords: robust, interpretability, transformer
• Abstract: The integration of machine learning in medicine has significantly improved diagnostic precision, particularly in the interpretation of complex structures like the human brain. Diagnosing challenging conditions such as Alzheimer's disease has prompted the development of brain age estimation techniques. These methods often leverage three-dimensional Magnetic Resonance Imaging (MRI) scans, with recent studies emphasizing the efficacy of 3D convolutional neural networks (CNNs) like 3D ResNet. However, the untapped potential of Vision Transformers (ViTs), known for their accuracy and interpretability, persists in this domain due to limitations in their 3D versions. This paper introduces Triamese-ViT, an innovative adaptation of the ViT model for brain age estimation. Our model uniquely combines ViTs from three different orientations to capture 3D information, significantly enhancing accuracy and interpretability. Tested on a dataset of 1351 MRI scans, Triamese-ViT achieves a Mean Absolute Error (MAE) of 3.84, a 0.9 Spearman correlation coefficient with chronological age, and a -0.29 Spearman correlation coefficient between the brain age gap (BAG) and chronological age, significantly better than previous methods for brian age estimation. A key innovation of Triamese-ViT is its capacity to generate a comprehensive 3D-like attention map, synthesized from 2D attention maps of each orientation-specific ViT. This feature is particularly beneficial for in-depth brain age analysis and disease diagnosis, offering deeper insights into brain health and the mechanisms of age-related neural changes.

Title: A Framework for Agricultural Food Supply Chain using Blockchain

• Authors: Sudarssan N
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.09476
• Pdf URL: https://arxiv.org/pdf/2401.09476
• Copy Paste: [[2401.09476]] A Framework for Agricultural Food Supply Chain using Blockchain(https://arxiv.org/abs/2401.09476)
• Keywords: security
• Abstract: The main aim of the paper is to create a trust and transparency in the food supply chain system, ensuring food safety for everyone with the help of Blockchain Technology. Food supply chain is the process of tracing a crop from the farmer or producer to the buyer. With the advent of blockchain, providing a safe and fraud-free environment for the provision of numerous agricultural necessities has become much easier. Because of the globalization of trade, the present supply chain market today includes various companies involving integration of data, complex transactions and distribution. Information tamper resistance, supply-demand relationships, and traceable oversight are all difficulties that arise as a result of this. Blockchain is a distributed ledger technology that can provide information that is resistant to tampering. This strategy can eliminate the need for a centralized trusted authority, intermediaries, and business histories, allowing for increased production and security while maintaining the highest levels of integrity, liability, and safety. In order to have an integrity and transparency in food supply chain in the agricultural sector, a framework is proposed here based on block chain and IoT.

Title: Uncertainty-Aware Hardware Trojan Detection Using Multimodal Deep Learning

• Authors: Rahul Vishwakarma, Amin Rezaei
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.09479
• Pdf URL: https://arxiv.org/pdf/2401.09479
• Copy Paste: [[2401.09479]] Uncertainty-Aware Hardware Trojan Detection Using Multimodal Deep Learning(https://arxiv.org/abs/2401.09479)
• Keywords: security, generative
• Abstract: The risk of hardware Trojans being inserted at various stages of chip production has increased in a zero-trust fabless era. To counter this, various machine learning solutions have been developed for the detection of hardware Trojans. While most of the focus has been on either a statistical or deep learning approach, the limited number of Trojan-infected benchmarks affects the detection accuracy and restricts the possibility of detecting zero-day Trojans. To close the gap, we first employ generative adversarial networks to amplify our data in two alternative representation modalities, a graph and a tabular, ensuring that the dataset is distributed in a representative manner. Further, we propose a multimodal deep learning approach to detect hardware Trojans and evaluate the results from both early fusion and late fusion strategies. We also estimate the uncertainty quantification metrics of each prediction for risk-aware decision-making. The outcomes not only confirms the efficacy of our proposed hardware Trojan detection method but also opens a new door for future studies employing multimodality and uncertainty quantification to address other hardware security challenges.

Title: LoMA: Lossless Compressed Memory Attention

• Authors: Yumeng Wang, Zhenyang Xiao
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2401.09486
• Pdf URL: https://arxiv.org/pdf/2401.09486
• Copy Paste: [[2401.09486]] LoMA: Lossless Compressed Memory Attention(https://arxiv.org/abs/2401.09486)
• Keywords: large language model
• Abstract: The ability to handle long texts is one of the most important capabilities of Large Language Models (LLMs), but as the text length increases, the consumption of resources also increases dramatically. At present, reducing resource consumption by compressing the KV cache is a common approach. Although there are many existing compression methods, they share a common drawback: the compression is not lossless. That is, information is inevitably lost during the compression process. If the compression rate is high, the probability of losing important information increases dramatically. We propose a new method, Lossless Compressed Memory Attention (LoMA), which allows for lossless compression of information into special memory token KV pairs according to a set compression ratio. Our experiments have achieved remarkable results, demonstrating that LoMA can be efficiently trained and has very effective performance.

Title: IPR-NeRF: Ownership Verification meets Neural Radiance Field

• Authors: Win Kent Ong, Kam Woh Ng, Chee Seng Chan, Yi Zhe Song, Tao Xiang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.09495
• Pdf URL: https://arxiv.org/pdf/2401.09495
• Copy Paste: [[2401.09495]] IPR-NeRF: Ownership Verification meets Neural Radiance Field(https://arxiv.org/abs/2401.09495)
• Keywords: protect, attack, robust, watermark, diffusion
• Abstract: Neural Radiance Field (NeRF) models have gained significant attention in the computer vision community in the recent past with state-of-the-art visual quality and produced impressive demonstrations. Since then, technopreneurs have sought to leverage NeRF models into a profitable business. Therefore, NeRF models make it worth the risk of plagiarizers illegally copying, re-distributing, or misusing those models. This paper proposes a comprehensive intellectual property (IP) protection framework for the NeRF model in both black-box and white-box settings, namely IPR-NeRF. In the black-box setting, a diffusion-based solution is introduced to embed and extract the watermark via a two-stage optimization process. In the white-box setting, a designated digital signature is embedded into the weights of the NeRF model by adopting the sign loss objective. Our extensive experiments demonstrate that not only does our approach maintain the fidelity (\ie, the rendering quality) of IPR-NeRF models, but it is also robust against both ambiguity and removal attacks compared to prior arts.

Title: Learning to Generalize over Subpartitions for Heterogeneity-aware Domain Adaptive Nuclei Segmentation

• Authors: Jianan Fan, Dongnan Liu, Hang Chang, Weidong Cai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.09496
• Pdf URL: https://arxiv.org/pdf/2401.09496
• Copy Paste: [[2401.09496]] Learning to Generalize over Subpartitions for Heterogeneity-aware Domain Adaptive Nuclei Segmentation(https://arxiv.org/abs/2401.09496)
• Keywords: segmentation
• Abstract: Annotation scarcity and cross-modality/stain data distribution shifts are two major obstacles hindering the application of deep learning models for nuclei analysis, which holds a broad spectrum of potential applications in digital pathology. Recently, unsupervised domain adaptation (UDA) methods have been proposed to mitigate the distributional gap between different imaging modalities for unsupervised nuclei segmentation in histopathology images. However, existing UDA methods are built upon the assumption that data distributions within each domain should be uniform. Based on the over-simplified supposition, they propose to align the histopathology target domain with the source domain integrally, neglecting severe intra-domain discrepancy over subpartitions incurred by mixed cancer types and sampling organs. In this paper, for the first time, we propose to explicitly consider the heterogeneity within the histopathology domain and introduce open compound domain adaptation (OCDA) to resolve the crux. In specific, a two-stage disentanglement framework is proposed to acquire domain-invariant feature representations at both image and instance levels. The holistic design addresses the limitations of existing OCDA approaches which struggle to capture instance-wise variations. Two regularization strategies are specifically devised herein to leverage the rich subpartition-specific characteristics in histopathology images and facilitate subdomain decomposition. Moreover, we propose a dual-branch nucleus shape and structure preserving module to prevent nucleus over-generation and deformation in the synthesized images. Experimental results on both cross-modality and cross-stain scenarios over a broad range of diverse datasets demonstrate the superiority of our method compared with state-of-the-art UDA and OCDA methods.

Title: Technical Report: On the Convergence of Gossip Learning in the Presence of Node Inaccessibility

• Authors: Tian Liu, Yue Cui, Xueyang Hu, Yecheng Xu, Bo Liu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.09498
• Pdf URL: https://arxiv.org/pdf/2401.09498
• Copy Paste: [[2401.09498]] Technical Report: On the Convergence of Gossip Learning in the Presence of Node Inaccessibility(https://arxiv.org/abs/2401.09498)
• Keywords: federate
• Abstract: Gossip learning (GL), as a decentralized alternative to federated learning (FL), is more suitable for resource-constrained wireless networks, such as FANETs that are formed by unmanned aerial vehicles (UAVs). GL can significantly enhance the efficiency and extend the battery life of UAV networks. Despite the advantages, the performance of GL is strongly affected by data distribution, communication speed, and network connectivity. However, how these factors influence the GL convergence is still unclear. Existing work studied the convergence of GL based on a virtual quantity for the sake of convenience, which fail to reflect the real state of the network when some nodes are inaccessible. In this paper, we formulate and investigate the impact of inaccessible nodes to GL under a dynamic network topology. We first decompose the weight divergence by whether the node is accessible or not. Then, we investigate the GL convergence under the dynamic of node accessibility and theoretically provide how the number of inaccessible nodes, data non-i.i.d.-ness, and duration of inaccessibility affect the convergence. Extensive experiments are carried out in practical settings to comprehensively verify the correctness of our theoretical findings.

Title: Enhancing Surveillance Camera FOV Quality via Semantic Line Detection and Classification with Deep Hough Transform

• Authors: Andrew C. Freeman, Wenjing Shi, Bin Hwang
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.09515
• Pdf URL: https://arxiv.org/pdf/2401.09515
• Copy Paste: [[2401.09515]] Enhancing Surveillance Camera FOV Quality via Semantic Line Detection and Classification with Deep Hough Transform(https://arxiv.org/abs/2401.09515)
• Keywords: security
• Abstract: The quality of recorded videos and images is significantly influenced by the camera's field of view (FOV). In critical applications like surveillance systems and self-driving cars, an inadequate FOV can give rise to severe safety and security concerns, including car accidents and thefts due to the failure to detect individuals and objects. The conventional methods for establishing the correct FOV heavily rely on human judgment and lack automated mechanisms to assess video and image quality based on FOV. In this paper, we introduce an innovative approach that harnesses semantic line detection and classification alongside deep Hough transform to identify semantic lines, thus ensuring a suitable FOV by understanding 3D view through parallel lines. Our approach yields an effective F1 score of 0.729 on the public EgoCart dataset, coupled with a notably high median score in the line placement metric. We illustrate that our method offers a straightforward means of assessing the quality of the camera's field of view, achieving a classification accuracy of 83.8\%. This metric can serve as a proxy for evaluating the potential performance of video and image quality applications.

Title: Dimensional Neuroimaging Endophenotypes: Neurobiological Representations of Disease Heterogeneity Through Machine Learning

• Authors: Junhao Wen, Mathilde Antoniades, Zhijian Yang, Gyujoon Hwang, Ioanna Skampardoni, Rongguang Wang, Christos Davatzikos
• Subjects: cs.LG, eess.IV, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2401.09517
• Pdf URL: https://arxiv.org/pdf/2401.09517
• Copy Paste: [[2401.09517]] Dimensional Neuroimaging Endophenotypes: Neurobiological Representations of Disease Heterogeneity Through Machine Learning(https://arxiv.org/abs/2401.09517)
• Keywords: robust, generative
• Abstract: Machine learning has been increasingly used to obtain individualized neuroimaging signatures for disease diagnosis, prognosis, and response to treatment in neuropsychiatric and neurodegenerative disorders. Therefore, it has contributed to a better understanding of disease heterogeneity by identifying disease subtypes that present significant differences in various brain phenotypic measures. In this review, we first present a systematic literature overview of studies using machine learning and multimodal MRI to unravel disease heterogeneity in various neuropsychiatric and neurodegenerative disorders, including Alzheimer disease, schizophrenia, major depressive disorder, autism spectrum disorder, multiple sclerosis, as well as their potential in transdiagnostic settings. Subsequently, we summarize relevant machine learning methodologies and discuss an emerging paradigm which we call dimensional neuroimaging endophenotype (DNE). DNE dissects the neurobiological heterogeneity of neuropsychiatric and neurodegenerative disorders into a low dimensional yet informative, quantitative brain phenotypic representation, serving as a robust intermediate phenotype (i.e., endophenotype) largely reflecting underlying genetics and etiology. Finally, we discuss the potential clinical implications of the current findings and envision future research avenues.

Title: Privacy Engineering in Smart Home (SH) Systems: A Comprehensive Privacy Threat Analysis and Risk Management Approach

• Authors: Emmanuel Dare Alalade, Mohammed Mahyoub, Ashraf Matrawy
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2401.09519
• Pdf URL: https://arxiv.org/pdf/2401.09519
• Copy Paste: [[2401.09519]] Privacy Engineering in Smart Home (SH) Systems: A Comprehensive Privacy Threat Analysis and Risk Management Approach(https://arxiv.org/abs/2401.09519)
• Keywords: privacy
• Abstract: Addressing trust concerns in Smart Home (SH) systems is imperative due to the limited study on preservation approaches that focus on analyzing and evaluating privacy threats for effective risk management. While most research focuses primarily on user privacy, device data privacy, especially identity privacy, is almost neglected, which can significantly impact overall user privacy within the SH system. To this end, our study incorporates privacy engineering (PE) principles in the SH system that consider user and device data privacy. We start with a comprehensive reference model for a typical SH system. Based on the initial stage of LINDDUN PRO for the PE framework, we present a data flow diagram (DFD) based on a typical SH reference model to better understand SH system operations. To identify potential areas of privacy threat and perform a privacy threat analysis (PTA), we employ the LINDDUN PRO threat model. Then, a privacy impact assessment (PIA) was carried out to implement privacy risk management by prioritizing privacy threats based on their likelihood of occurrence and potential consequences. Finally, we suggest possible privacy enhancement techniques (PETs) that can mitigate some of these threats. The study aims to elucidate the main threats to privacy, associated risks, and effective prioritization of privacy control in SH systems. The outcomes of this study are expected to benefit SH stakeholders, including vendors, cloud providers, users, researchers, and regulatory bodies in the SH systems domain.

Title: BERTologyNavigator: Advanced Question Answering with BERT-based Semantics

• Authors: Shreya Rajpal (1,2), Ricardo Usbeck (1) ((1) Universität Hamburg, Hamburg, Germany,(2) Vellore Institute of Technology, Vellore, Tamil Nadu, India)
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.09553
• Pdf URL: https://arxiv.org/pdf/2401.09553
• Copy Paste: [[2401.09553]] BERTologyNavigator: Advanced Question Answering with BERT-based Semantics(https://arxiv.org/abs/2401.09553)
• Keywords: extraction
• Abstract: The development and integration of knowledge graphs and language models has significance in artificial intelligence and natural language processing. In this study, we introduce the BERTologyNavigator -- a two-phased system that combines relation extraction techniques and BERT embeddings to navigate the relationships within the DBLP Knowledge Graph (KG). Our approach focuses on extracting one-hop relations and labelled candidate pairs in the first phases. This is followed by employing BERT's CLS embeddings and additional heuristics for relation selection in the second phase. Our system reaches an F1 score of 0.2175 on the DBLP QuAD Final test dataset for Scholarly QALD and 0.98 F1 score on the subset of the DBLP QuAD test dataset during the QA phase.

Title: Improving Classification Performance With Human Feedback: Label a few, we label the rest

• Authors: Natan Vidra, Thomas Clifford, Katherine Jijo, Eden Chung, Liang Zhang
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2401.09555
• Pdf URL: https://arxiv.org/pdf/2401.09555
• Copy Paste: [[2401.09555]] Improving Classification Performance With Human Feedback: Label a few, we label the rest(https://arxiv.org/abs/2401.09555)
• Keywords: large language model
• Abstract: In the realm of artificial intelligence, where a vast majority of data is unstructured, obtaining substantial amounts of labeled data to train supervised machine learning models poses a significant challenge. To address this, we delve into few-shot and active learning, where are goal is to improve AI models with human feedback on a few labeled examples. This paper focuses on understanding how a continuous feedback loop can refine models, thereby enhancing their accuracy, recall, and precision through incremental human input. By employing Large Language Models (LLMs) such as GPT-3.5, BERT, and SetFit, we aim to analyze the efficacy of using a limited number of labeled examples to substantially improve model accuracy. We benchmark this approach on the Financial Phrasebank, Banking, Craigslist, Trec, Amazon Reviews datasets to prove that with just a few labeled examples, we are able to surpass the accuracy of zero shot large language models to provide enhanced text classification performance. We demonstrate that rather than needing to manually label millions of rows of data, we just need to label a few and the model can effectively predict the rest.

Title: Sharing Knowledge in Multi-Task Deep Reinforcement Learning

• Authors: Carlo D'Eramo, Davide Tateo, Andrea Bonarini, Marcello Restelli, Jan Peters
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.09561
• Pdf URL: https://arxiv.org/pdf/2401.09561
• Copy Paste: [[2401.09561]] Sharing Knowledge in Multi-Task Deep Reinforcement Learning(https://arxiv.org/abs/2401.09561)
• Keywords: extraction
• Abstract: We study the benefit of sharing representations among tasks to enable the effective use of deep neural networks in Multi-Task Reinforcement Learning. We leverage the assumption that learning from different tasks, sharing common properties, is helpful to generalize the knowledge of them resulting in a more effective feature extraction compared to learning a single task. Intuitively, the resulting set of features offers performance benefits when used by Reinforcement Learning algorithms. We prove this by providing theoretical guarantees that highlight the conditions for which is convenient to share representations among tasks, extending the well-known finite-time bounds of Approximate Value-Iteration to the multi-task setting. In addition, we complement our analysis by proposing multi-task extensions of three Reinforcement Learning algorithms that we empirically evaluate on widely used Reinforcement Learning benchmarks showing significant improvements over the single-task counterparts in terms of sample efficiency and performance.

Title: Aligning Large Language Models with Counterfactual DPO

• Authors: Bradley Butcher
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.09566
• Pdf URL: https://arxiv.org/pdf/2401.09566
• Copy Paste: [[2401.09566]] Aligning Large Language Models with Counterfactual DPO(https://arxiv.org/abs/2401.09566)
• Keywords: large language model
• Abstract: Advancements in large language models (LLMs) have demonstrated remarkable capabilities across a diverse range of applications. These models excel in generating text completions that are contextually coherent and cover an extensive array of subjects. However, the vast datasets required for their training make aligning response styles during the pretraining and instruction tuning phases challenging. Consequently, an additional alignment phase is typically employed, wherein the model is further trained with human preference data to better align its outputs with human expectations. While this process doesn't introduce new capabilities per se, it does accentuate generation styles innate to the model. This paper explores the utilization of counterfactual prompting within the framework of Direct Preference Optimization (DPO) to align the model's style without relying on human intervention. We demonstrate that this method effectively instils desirable behaviour, mitigates undesirable ones, and encourages the model to disregard inappropriate instructions. Our findings suggest that counterfactual prompting with DPO presents a low-resource way to fine-tune LLMs to meet the demands for responsible and ethically aligned AI systems.

Title: Towards Scalable and Robust Model Versioning

• Authors: Wenxin Ding, Arjun Nitin Bhagoji, Ben Y. Zhao, Haitao Zheng
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2401.09574
• Pdf URL: https://arxiv.org/pdf/2401.09574
• Copy Paste: [[2401.09574]] Towards Scalable and Robust Model Versioning(https://arxiv.org/abs/2401.09574)
• Keywords: protect, attack, robust
• Abstract: As the deployment of deep learning models continues to expand across industries, the threat of malicious incursions aimed at gaining access to these deployed models is on the rise. Should an attacker gain access to a deployed model, whether through server breaches, insider attacks, or model inversion techniques, they can then construct white-box adversarial attacks to manipulate the model's classification outcomes, thereby posing significant risks to organizations that rely on these models for critical tasks. Model owners need mechanisms to protect themselves against such losses without the necessity of acquiring fresh training data - a process that typically demands substantial investments in time and capital. In this paper, we explore the feasibility of generating multiple versions of a model that possess different attack properties, without acquiring new training data or changing model architecture. The model owner can deploy one version at a time and replace a leaked version immediately with a new version. The newly deployed model version can resist adversarial attacks generated leveraging white-box access to one or all previously leaked versions. We show theoretically that this can be accomplished by incorporating parameterized hidden distributions into the model training data, forcing the model to learn task-irrelevant features uniquely defined by the chosen data. Additionally, optimal choices of hidden distributions can produce a sequence of model versions capable of resisting compound transferability attacks over time. Leveraging our analytical insights, we design and implement a practical model versioning method for DNN classifiers, which leads to significant robustness improvements over existing methods. We believe our work presents a promising direction for safeguarding DNN services beyond their initial deployment.

Title: Zero Trust Implementation in the Emerging Technologies Era: Survey

• Authors: Abraham Itzhak Weinberg, Kelly Cohen
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2401.09575
• Pdf URL: https://arxiv.org/pdf/2401.09575
• Copy Paste: [[2401.09575]] Zero Trust Implementation in the Emerging Technologies Era: Survey(https://arxiv.org/abs/2401.09575)
• Keywords: secure, security, attack
• Abstract: This paper presents a comprehensive analysis of the shift from the traditional perimeter model of security to the Zero Trust (ZT) framework, emphasizing the key points in the transition and the practical application of ZT. It outlines the differences between ZT policies and legacy security policies, along with the significant events that have impacted the evolution of ZT. Additionally, the paper explores the potential impacts of emerging technologies, such as Artificial Intelligence (AI) and quantum computing, on the policy and implementation of ZT. The study thoroughly examines how AI can enhance ZT by utilizing Machine Learning (ML) algorithms to analyze patterns, detect anomalies, and predict threats, thereby improving real-time decision-making processes. Furthermore, the paper demonstrates how a chaos theory-based approach, in conjunction with other technologies like eXtended Detection and Response (XDR), can effectively mitigate cyberattacks. As quantum computing presents new challenges to ZT and cybersecurity as a whole, the paper delves into the intricacies of ZT migration, automation, and orchestration, addressing the complexities associated with these aspects. Finally, the paper provides a best practice approach for the seamless implementation of ZT in organizations, laying out the proposed guidelines to facilitate organizations in their transition towards a more secure ZT model. The study aims to support organizations in successfully implementing ZT and enhancing their cybersecurity measures.

Title: Efficient generative adversarial networks using linear additive-attention Transformers

• Authors: Emilio Morales-Juarez, Gibran Fuentes-Pineda
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.09596
• Pdf URL: https://arxiv.org/pdf/2401.09596
• Copy Paste: [[2401.09596]] Efficient generative adversarial networks using linear additive-attention Transformers(https://arxiv.org/abs/2401.09596)
• Keywords: diffusion, transformer, generative
• Abstract: Although the capacity of deep generative models for image generation, such as Diffusion Models (DMs) and Generative Adversarial Networks (GANs), has dramatically improved in recent years, much of their success can be attributed to computationally expensive architectures. This has limited their adoption and use to research laboratories and companies with large resources, while significantly raising the carbon footprint for training, fine-tuning, and inference. In this work, we present LadaGAN, an efficient generative adversarial network that is built upon a novel Transformer block named Ladaformer. The main component of this block is a linear additive-attention mechanism that computes a single attention vector per head instead of the quadratic dot-product attention. We employ Ladaformer in both the generator and discriminator, which reduces the computational complexity and overcomes the training instabilities often associated with Transformer GANs. LadaGAN consistently outperforms existing convolutional and Transformer GANs on benchmark datasets at different resolutions while being significantly more efficient. Moreover, LadaGAN shows competitive performance compared to state-of-the-art multi-step generative models (e.g. DMs) using orders of magnitude less computational resources.

Title: Rethinking FID: Towards a Better Evaluation Metric for Image Generation

• Authors: Sadeep Jayasumana, Srikumar Ramalingam, Andreas Veit, Daniel Glasner, Ayan Chakrabarti, Sanjiv Kumar
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.09603
• Pdf URL: https://arxiv.org/pdf/2401.09603
• Copy Paste: [[2401.09603]] Rethinking FID: Towards a Better Evaluation Metric for Image Generation(https://arxiv.org/abs/2401.09603)
• Keywords: robust
• Abstract: As with many machine learning problems, the progress of image generation methods hinges on good evaluation metrics. One of the most popular is the Frechet Inception Distance (FID). FID estimates the distance between a distribution of Inception-v3 features of real images, and those of images generated by the algorithm. We highlight important drawbacks of FID: Inception's poor representation of the rich and varied content generated by modern text-to-image models, incorrect normality assumptions, and poor sample complexity. We call for a reevaluation of FID's use as the primary quality metric for generated images. We empirically demonstrate that FID contradicts human raters, it does not reflect gradual improvement of iterative text-to-image models, it does not capture distortion levels, and that it produces inconsistent results when varying the sample size. We also propose an alternative new metric, CMMD, based on richer CLIP embeddings and the maximum mean discrepancy distance with the Gaussian RBF kernel. It is an unbiased estimator that does not make any assumptions on the probability distribution of the embeddings and is sample efficient. Through extensive experiments and analysis, we demonstrate that FID-based evaluations of text-to-image models may be unreliable, and that CMMD offers a more robust and reliable assessment of image quality.

Title: MedBlindTuner: Towards Privacy-preserving Fine-tuning on Biomedical Images with Transformers and Fully Homomorphic Encryption

• Authors: Prajwal Panzade, Daniel Takabi, Zhipeng Cai
• Subjects: cs.CR, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.09604
• Pdf URL: https://arxiv.org/pdf/2401.09604
• Copy Paste: [[2401.09604]] MedBlindTuner: Towards Privacy-preserving Fine-tuning on Biomedical Images with Transformers and Fully Homomorphic Encryption(https://arxiv.org/abs/2401.09604)
• Keywords: secure, privacy, transformer
• Abstract: Advancements in machine learning (ML) have significantly revolutionized medical image analysis, prompting hospitals to rely on external ML services. However, the exchange of sensitive patient data, such as chest X-rays, poses inherent privacy risks when shared with third parties. Addressing this concern, we propose MedBlindTuner, a privacy-preserving framework leveraging fully homomorphic encryption (FHE) and a data-efficient image transformer (DEiT). MedBlindTuner enables the training of ML models exclusively on FHE-encrypted medical images. Our experimental evaluation demonstrates that MedBlindTuner achieves comparable accuracy to models trained on non-encrypted images, offering a secure solution for outsourcing ML computations while preserving patient data privacy. To the best of our knowledge, this is the first work that uses data-efficient image transformers and fully homomorphic encryption in this domain.

Title: Robustness Evaluation of Machine Learning Models for Robot Arm Action Recognition in Noisy Environments

• Authors: Elaheh Motamedi, Kian Behzad, Rojin Zandi, Hojjat Salehinejad, Milad Siami
• Subjects: cs.CV, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2401.09606
• Pdf URL: https://arxiv.org/pdf/2401.09606
• Copy Paste: [[2401.09606]] Robustness Evaluation of Machine Learning Models for Robot Arm Action Recognition in Noisy Environments(https://arxiv.org/abs/2401.09606)
• Keywords: robust
• Abstract: In the realm of robot action recognition, identifying distinct but spatially proximate arm movements using vision systems in noisy environments poses a significant challenge. This paper studies robot arm action recognition in noisy environments using machine learning techniques. Specifically, a vision system is used to track the robot's movements followed by a deep learning model to extract the arm's key points. Through a comparative analysis of machine learning methods, the effectiveness and robustness of this model are assessed in noisy environments. A case study was conducted using the Tic-Tac-Toe game in a 3-by-3 grid environment, where the focus is to accurately identify the actions of the arms in selecting specific locations within this constrained environment. Experimental results show that our approach can achieve precise key point detection and action classification despite the addition of noise and uncertainties to the dataset.

Title: Land Cover Image Classification

• Authors: Antonio Rangel, Juan Terven, Diana M. Cordova-Esparza, E.A. Chavez-Urbiola
• Subjects: cs.CV, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2401.09607
• Pdf URL: https://arxiv.org/pdf/2401.09607
• Copy Paste: [[2401.09607]] Land Cover Image Classification(https://arxiv.org/abs/2401.09607)
• Keywords: transformer
• Abstract: Land Cover (LC) image classification has become increasingly significant in understanding environmental changes, urban planning, and disaster management. However, traditional LC methods are often labor-intensive and prone to human error. This paper explores state-of-the-art deep learning models for enhanced accuracy and efficiency in LC analysis. We compare convolutional neural networks (CNN) against transformer-based methods, showcasing their applications and advantages in LC studies. We used EuroSAT, a patch-based LC classification data set based on Sentinel-2 satellite images and achieved state-of-the-art results using current transformer models.

Title: Learning Shortcuts: On the Misleading Promise of NLU in Language Models

• Authors: Geetanjali Bihani, Julia Taylor Rayz
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.09615
• Pdf URL: https://arxiv.org/pdf/2401.09615
• Copy Paste: [[2401.09615]] Learning Shortcuts: On the Misleading Promise of NLU in Language Models(https://arxiv.org/abs/2401.09615)
• Keywords: robust, large language model
• Abstract: The advent of large language models (LLMs) has enabled significant performance gains in the field of natural language processing. However, recent studies have found that LLMs often resort to shortcuts when performing tasks, creating an illusion of enhanced performance while lacking generalizability in their decision rules. This phenomenon introduces challenges in accurately assessing natural language understanding in LLMs. Our paper provides a concise survey of relevant research in this area and puts forth a perspective on the implications of shortcut learning in the evaluation of language models, specifically for NLU tasks. This paper urges more research efforts to be put towards deepening our comprehension of shortcut learning, contributing to the development of more robust language models, and raising the standards of NLU evaluation in real-world scenarios.

Title: Physics-Informed Calibration of Aeromagnetic Compensation in Magnetic Navigation Systems using Liquid Time-Constant Networks

• Authors: Favour Nerrise (1 and 2), Andrew Sosa Sosanya (2), Patrick Neary (2) ((1) Department of Electrical Engineering, Stanford University, CA, USA, (2) SandboxAQ, Palo Alto, CA, USA)
• Subjects: cs.LG, eess.SY, physics.comp-ph, physics.geo-ph
• Abstract URL: https://arxiv.org/abs/2401.09631
• Pdf URL: https://arxiv.org/pdf/2401.09631
• Copy Paste: [[2401.09631]] Physics-Informed Calibration of Aeromagnetic Compensation in Magnetic Navigation Systems using Liquid Time-Constant Networks(https://arxiv.org/abs/2401.09631)
• Keywords: attack
• Abstract: Magnetic navigation (MagNav) is a rising alternative to the Global Positioning System (GPS) and has proven useful for aircraft navigation. Traditional aircraft navigation systems, while effective, face limitations in precision and reliability in certain environments and against attacks. Airborne MagNav leverages the Earth's magnetic field to provide accurate positional information. However, external magnetic fields induced by aircraft electronics and Earth's large-scale magnetic fields disrupt the weaker signal of interest. We introduce a physics-informed approach using Tolles-Lawson coefficients for compensation and Liquid Time-Constant Networks (LTCs) to remove complex, noisy signals derived from the aircraft's magnetic sources. Using real flight data with magnetometer measurements and aircraft measurements, we observe up to a 64% reduction in aeromagnetic compensation error (RMSE nT), outperforming conventional models. This significant improvement underscores the potential of a physics-informed, machine learning approach for extracting clean, reliable, and accurate magnetic signals for MagNav positional estimation.

Title: ClimateGPT: Towards AI Synthesizing Interdisciplinary Research on Climate Change

• Authors: David Thulke, Yingbo Gao, Petrus Pelser, Rein Brune, Rricha Jalota, Floris Fok, Michael Ramos, Ian van Wyk, Abdallah Nasir, Hayden Goldstein, Taylor Tragemann, Katie Nguyen, Ariana Fowler, Andrew Stanco, Jon Gabriel, Jordan Taylor, Dean Moro, Evgenii Tsymbalov, Juliette de Waal, Evgeny Matusov, Mudar Yaghi, Mohammad Shihadah, Hermann Ney, Christian Dugast, Jonathan Dotan, Daniel Erasmus
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2401.09646
• Pdf URL: https://arxiv.org/pdf/2401.09646
• Copy Paste: [[2401.09646]] ClimateGPT: Towards AI Synthesizing Interdisciplinary Research on Climate Change(https://arxiv.org/abs/2401.09646)
• Keywords: large language model
• Abstract: This paper introduces ClimateGPT, a model family of domain-specific large language models that synthesize interdisciplinary research on climate change. We trained two 7B models from scratch on a science-oriented dataset of 300B tokens. For the first model, the 4.2B domain-specific tokens were included during pre-training and the second was adapted to the climate domain after pre-training. Additionally, ClimateGPT-7B, 13B and 70B are continuously pre-trained from Llama~2 on a domain-specific dataset of 4.2B tokens. Each model is instruction fine-tuned on a high-quality and human-generated domain-specific dataset that has been created in close cooperation with climate scientists. To reduce the number of hallucinations, we optimize the model for retrieval augmentation and propose a hierarchical retrieval strategy. To increase the accessibility of our model to non-English speakers, we propose to make use of cascaded machine translation and show that this approach can perform comparably to natively multilingual models while being easier to scale to a large number of languages. Further, to address the intrinsic interdisciplinary aspect of climate change we consider different research perspectives. Therefore, the model can produce in-depth answers focusing on different perspectives in addition to an overall answer. We propose a suite of automatic climate-specific benchmarks to evaluate LLMs. On these benchmarks, ClimateGPT-7B performs on par with the ten times larger Llama-2-70B Chat model while not degrading results on general domain benchmarks. Our human evaluation confirms the trends we saw in our benchmarks. All models were trained and evaluated using renewable energy and are released publicly.

Title: Mobility Accelerates Learning: Convergence Analysis on Hierarchical Federated Learning in Vehicular Networks

• Authors: Tan Chen, Jintao Yan, Yuxuan Sun, Sheng Zhou, Deniz Gündüz, Zhisheng Niu
• Subjects: cs.LG, cs.AI, cs.DC
• Abstract URL: https://arxiv.org/abs/2401.09656
• Pdf URL: https://arxiv.org/pdf/2401.09656
• Copy Paste: [[2401.09656]] Mobility Accelerates Learning: Convergence Analysis on Hierarchical Federated Learning in Vehicular Networks(https://arxiv.org/abs/2401.09656)
• Keywords: privacy, federate
• Abstract: Hierarchical federated learning (HFL) enables distributed training of models across multiple devices with the help of several edge servers and a cloud edge server in a privacy-preserving manner. In this paper, we consider HFL with highly mobile devices, mainly targeting at vehicular networks. Through convergence analysis, we show that mobility influences the convergence speed by both fusing the edge data and shuffling the edge models. While mobility is usually considered as a challenge from the perspective of communication, we prove that it increases the convergence speed of HFL with edge-level heterogeneous data, since more diverse data can be incorporated. Furthermore, we demonstrate that a higher speed leads to faster convergence, since it accelerates the fusion of data. Simulation results show that mobility increases the model accuracy of HFL by up to 15.1% when training a convolutional neural network on the CIFAR-10 dataset.

Title: Artwork Protection Against Neural Style Transfer Using Locally Adaptive Adversarial Color Attack

• Authors: Zhongliang Guo, Kaixuan Wang, Weiye Li, Yifei Qian, Ognjen Arandjelović, Lei Fang
• Subjects: cs.CV, cs.CR, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2401.09673
• Pdf URL: https://arxiv.org/pdf/2401.09673
• Copy Paste: [[2401.09673]] Artwork Protection Against Neural Style Transfer Using Locally Adaptive Adversarial Color Attack(https://arxiv.org/abs/2401.09673)
• Keywords: security, protect, attack
• Abstract: Neural style transfer (NST) is widely adopted in computer vision to generate new images with arbitrary styles. This process leverages neural networks to merge aesthetic elements of a style image with the structural aspects of a content image into a harmoniously integrated visual result. However, unauthorized NST can exploit artwork. Such misuse raises socio-technical concerns regarding artists' rights and motivates the development of technical approaches for the proactive protection of original creations. Adversarial attack is a concept primarily explored in machine learning security. Our work introduces this technique to protect artists' intellectual property. In this paper Locally Adaptive Adversarial Color Attack (LAACA), a method for altering images in a manner imperceptible to the human eyes but disruptive to NST. Specifically, we design perturbations targeting image areas rich in high-frequency content, generated by disrupting intermediate features. Our experiments and user study confirm that by attacking NST using the proposed method results in visually worse neural style transfer, thus making it an effective solution for visual artwork protection.

Title: Curriculum Recommendations Using Transformer Base Model with InfoNCE Loss And Language Switching Method

• Authors: Xiaonan Xu, Bin Yuan, Yongyao Mo, Tianbo Song, Shulin Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.09699
• Pdf URL: https://arxiv.org/pdf/2401.09699
• Copy Paste: [[2401.09699]] Curriculum Recommendations Using Transformer Base Model with InfoNCE Loss And Language Switching Method(https://arxiv.org/abs/2401.09699)
• Keywords: transformer
• Abstract: The Curriculum Recommendations paradigm is dedicated to fostering learning equality within the ever-evolving realms of educational technology and curriculum development. In acknowledging the inherent obstacles posed by existing methodologies, such as content conflicts and disruptions from language translation, this paradigm aims to confront and overcome these challenges. Notably, it addresses content conflicts and disruptions introduced by language translation, hindrances that can impede the creation of an all-encompassing and personalized learning experience. The paradigm's objective is to cultivate an educational environment that not only embraces diversity but also customizes learning experiences to suit the distinct needs of each learner. To overcome these challenges, our approach builds upon notable contributions in curriculum development and personalized learning, introducing three key innovations. These include the integration of Transformer Base Model to enhance computational efficiency, the implementation of InfoNCE Loss for accurate content-topic matching, and the adoption of a language switching strategy to alleviate translation-related ambiguities. Together, these innovations aim to collectively tackle inherent challenges and contribute to forging a more equitable and effective learning journey for a diverse range of learners. Competitive cross-validation scores underscore the efficacy of sentence-transformers/LaBSE, achieving 0.66314, showcasing our methodology's effectiveness in diverse linguistic nuances for content alignment prediction. Index Terms-Curriculum Recommendation, Transformer model with InfoNCE Loss, Language Switching.

Title: P2Seg: Pointly-supervised Segmentation via Mutual Distillation

• Authors: Zipeng Wang, Xuehui Yu, Xumeng Han, Wenwen Yu, Zhixun Huang, Jianbin Jiao, Zhenjun Han
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.09709
• Pdf URL: https://arxiv.org/pdf/2401.09709
• Copy Paste: [[2401.09709]] P2Seg: Pointly-supervised Segmentation via Mutual Distillation(https://arxiv.org/abs/2401.09709)
• Keywords: segmentation
• Abstract: Point-level Supervised Instance Segmentation (PSIS) aims to enhance the applicability and scalability of instance segmentation by utilizing low-cost yet instance-informative annotations. Existing PSIS methods usually rely on positional information to distinguish objects, but predicting precise boundaries remains challenging due to the lack of contour annotations. Nevertheless, weakly supervised semantic segmentation methods are proficient in utilizing intra-class feature consistency to capture the boundary contours of the same semantic regions. In this paper, we design a Mutual Distillation Module (MDM) to leverage the complementary strengths of both instance position and semantic information and achieve accurate instance-level object perception. The MDM consists of Semantic to Instance (S2I) and Instance to Semantic (I2S). S2I is guided by the precise boundaries of semantic regions to learn the association between annotated points and instance contours. I2S leverages discriminative relationships between instances to facilitate the differentiation of various objects within the semantic map. Extensive experiments substantiate the efficacy of MDM in fostering the synergy between instance and semantic information, consequently improving the quality of instance-level object representations. Our method achieves 55.7 mAP$_{50}$ and 17.6 mAP on the PASCAL VOC and MS COCO datasets, significantly outperforming recent PSIS methods and several box-supervised instance segmentation competitors.

Title: SkyEyeGPT: Unifying Remote Sensing Vision-Language Tasks via Instruction Tuning with Large Language Model

• Authors: Yang Zhan, Zhitong Xiong, Yuan Yuan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.09712
• Pdf URL: https://arxiv.org/pdf/2401.09712
• Copy Paste: [[2401.09712]] SkyEyeGPT: Unifying Remote Sensing Vision-Language Tasks via Instruction Tuning with Large Language Model(https://arxiv.org/abs/2401.09712)
• Keywords: large language model
• Abstract: Large language models (LLMs) have recently been extended to the vision-language realm, obtaining impressive general multi-modal capabilities. However, the exploration of multi-modal large language models (MLLMs) for remote sensing (RS) data is still in its infancy, and the performance is not satisfactory. In this work, we introduce SkyEyeGPT, a unified multi-modal large language model specifically designed for RS vision-language understanding. To this end, we meticulously curate an RS multi-modal instruction tuning dataset, including single-task and multi-task conversation instructions. After manual verification, we obtain a high-quality RS instruction-following dataset with 968k samples. Our research demonstrates that with a simple yet effective design, SkyEyeGPT works surprisingly well on considerably different tasks without the need for extra encoding modules. Specifically, after projecting RS visual features to the language domain via an alignment layer, they are fed jointly with task-specific instructions into an LLM-based RS decoder to predict answers for RS open-ended tasks. In addition, we design a two-stage tuning method to enhance instruction-following and multi-turn dialogue ability at different granularities. Experiments on 8 datasets for RS vision-language tasks demonstrate SkyEyeGPT's superiority in image-level and region-level tasks, such as captioning and visual grounding. In particular, SkyEyeGPT exhibits encouraging results compared to GPT-4V in some qualitative tests. The online demo, code, and dataset will be released in https://github.com/ZhanYang-nwpu/SkyEyeGPT.

Title: HCVP: Leveraging Hierarchical Contrastive Visual Prompt for Domain Generalization

• Authors: Guanglin Zhou, Zhongyi Han, Shiming Chen, Biwei Huang, Liming Zhu, Tongliang Liu, Lina Yao, Kun Zhang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.09716
• Pdf URL: https://arxiv.org/pdf/2401.09716
• Copy Paste: [[2401.09716]] HCVP: Leveraging Hierarchical Contrastive Visual Prompt for Domain Generalization(https://arxiv.org/abs/2401.09716)
• Keywords: transformer, generative
• Abstract: Domain Generalization (DG) endeavors to create machine learning models that excel in unseen scenarios by learning invariant features. In DG, the prevalent practice of constraining models to a fixed structure or uniform parameterization to encapsulate invariant features can inadvertently blend specific aspects. Such an approach struggles with nuanced differentiation of inter-domain variations and may exhibit bias towards certain domains, hindering the precise learning of domain-invariant features. Recognizing this, we introduce a novel method designed to supplement the model with domain-level and task-specific characteristics. This approach aims to guide the model in more effectively separating invariant features from specific characteristics, thereby boosting the generalization. Building on the emerging trend of visual prompts in the DG paradigm, our work introduces the novel \textbf{H}ierarchical \textbf{C}ontrastive \textbf{V}isual \textbf{P}rompt (HCVP) methodology. This represents a significant advancement in the field, setting itself apart with a unique generative approach to prompts, alongside an explicit model structure and specialized loss functions. Differing from traditional visual prompts that are often shared across entire datasets, HCVP utilizes a hierarchical prompt generation network enhanced by prompt contrastive learning. These generative prompts are instance-dependent, catering to the unique characteristics inherent to different domains and tasks. Additionally, we devise a prompt modulation network that serves as a bridge, effectively incorporating the generated visual prompts into the vision transformer backbone. Experiments conducted on five DG datasets demonstrate the effectiveness of HCVP, outperforming both established DG algorithms and adaptation protocols.

Title: Large Language Model Lateral Spear Phishing: A Comparative Study in Large-Scale Organizational Settings

• Authors: Mazal Bethany, Athanasios Galiopoulos, Emet Bethany, Mohammad Bahrami Karkevandi, Nishant Vishwamitra, Peyman Najafirad
• Subjects: cs.CR, cs.CL
• Abstract URL: https://arxiv.org/abs/2401.09727
• Pdf URL: https://arxiv.org/pdf/2401.09727
• Copy Paste: [[2401.09727]] Large Language Model Lateral Spear Phishing: A Comparative Study in Large-Scale Organizational Settings(https://arxiv.org/abs/2401.09727)
• Keywords: security, attack, large language model
• Abstract: The critical threat of phishing emails has been further exacerbated by the potential of LLMs to generate highly targeted, personalized, and automated spear phishing attacks. Two critical problems concerning LLM-facilitated phishing require further investigation: 1) Existing studies on lateral phishing lack specific examination of LLM integration for large-scale attacks targeting the entire organization, and 2) Current anti-phishing infrastructure, despite its extensive development, lacks the capability to prevent LLM-generated attacks, potentially impacting both employees and IT security incident management. However, the execution of such investigative studies necessitates a real-world environment, one that functions during regular business operations and mirrors the complexity of a large organizational infrastructure. This setting must also offer the flexibility required to facilitate a diverse array of experimental conditions, particularly the incorporation of phishing emails crafted by LLMs. This study is a pioneering exploration into the use of Large Language Models (LLMs) for the creation of targeted lateral phishing emails, targeting a large tier 1 university's operation and workforce of approximately 9,000 individuals over an 11-month period. It also evaluates the capability of email filtering infrastructure to detect such LLM-generated phishing attempts, providing insights into their effectiveness and identifying potential areas for improvement. Based on our findings, we propose machine learning-based detection techniques for such emails to detect LLM-generated phishing emails that were missed by the existing infrastructure, with an F1-score of 98.96.

Title: Instance Brownian Bridge as Texts for Open-vocabulary Video Instance Segmentation

• Authors: Zesen Cheng, Kehan Li, Hao Li, Peng Jin, Chang Liu, Xiawu Zheng, Rongrong Ji, Jie Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.09732
• Pdf URL: https://arxiv.org/pdf/2401.09732
• Copy Paste: [[2401.09732]] Instance Brownian Bridge as Texts for Open-vocabulary Video Instance Segmentation(https://arxiv.org/abs/2401.09732)
• Keywords: segmentation
• Abstract: Temporally locating objects with arbitrary class texts is the primary pursuit of open-vocabulary Video Instance Segmentation (VIS). Because of the insufficient vocabulary of video data, previous methods leverage image-text pretraining model for recognizing object instances by separately aligning each frame and class texts, ignoring the correlation between frames. As a result, the separation breaks the instance movement context of videos, causing inferior alignment between video and text. To tackle this issue, we propose to link frame-level instance representations as a Brownian Bridge to model instance dynamics and align bridge-level instance representation to class texts for more precisely open-vocabulary VIS (BriVIS). Specifically, we build our system upon a frozen video segmentor to generate frame-level instance queries, and design Temporal Instance Resampler (TIR) to generate queries with temporal context from frame queries. To mold instance queries to follow Brownian bridge and accomplish alignment with class texts, we design Bridge-Text Alignment (BTA) to learn discriminative bridge-level representations of instances via contrastive objectives. Setting MinVIS as the basic video segmentor, BriVIS surpasses the Open-vocabulary SOTA (OV2Seg) by a clear margin. For example, on the challenging large-vocabulary VIS dataset (BURST), BriVIS achieves 7.43 mAP and exhibits 49.49% improvement compared to OV2Seg (4.97 mAP).

Title: Measuring the Discrepancy between 3D Geometric Models using Directional Distance Fields

• Authors: Siyu Ren, Junhui Hou, Xiaodong Chen, Hongkai Xiong, Wenping Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.09736
• Pdf URL: https://arxiv.org/pdf/2401.09736
• Copy Paste: [[2401.09736]] Measuring the Discrepancy between 3D Geometric Models using Directional Distance Fields(https://arxiv.org/abs/2401.09736)
• Keywords: robust
• Abstract: Qualifying the discrepancy between 3D geometric models, which could be represented with either point clouds or triangle meshes, is a pivotal issue with board applications. Existing methods mainly focus on directly establishing the correspondence between two models and then aggregating point-wise distance between corresponding points, resulting in them being either inefficient or ineffective. In this paper, we propose DirDist, an efficient, effective, robust, and differentiable distance metric for 3D geometry data. Specifically, we construct DirDist based on the proposed implicit representation of 3D models, namely directional distance field (DDF), which defines the directional distances of 3D points to a model to capture its local surface geometry. We then transfer the discrepancy between two 3D geometric models as the discrepancy between their DDFs defined on an identical domain, naturally establishing model correspondence. To demonstrate the advantage of our DirDist, we explore various distance metric-driven 3D geometric modeling tasks, including template surface fitting, rigid registration, non-rigid registration, scene flow estimation and human pose optimization. Extensive experiments show that our DirDist achieves significantly higher accuracy under all tasks. As a generic distance metric, DirDist has the potential to advance the field of 3D geometric modeling. The source code is available at \url{https://github.com/rsy6318/DirDist}.

Title: Hijacking Attacks against Neural Networks by Analyzing Training Data

• Authors: Yunjie Ge, Qian Wang, Huayang Huang, Qi Li, Cong Wang, Chao Shen, Lingchen Zhao, Peipei Jiang, Zheng Fang, Shenyi Zhang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2401.09740
• Pdf URL: https://arxiv.org/pdf/2401.09740
• Copy Paste: [[2401.09740]] Hijacking Attacks against Neural Networks by Analyzing Training Data(https://arxiv.org/abs/2401.09740)
• Keywords: defense, attack, robust
• Abstract: Backdoors and adversarial examples are the two primary threats currently faced by deep neural networks (DNNs). Both attacks attempt to hijack the model behaviors with unintended outputs by introducing (small) perturbations to the inputs. Backdoor attacks, despite the high success rates, often require a strong assumption, which is not always easy to achieve in reality. Adversarial example attacks, which put relatively weaker assumptions on attackers, often demand high computational resources, yet do not always yield satisfactory success rates when attacking mainstream black-box models in the real world. These limitations motivate the following research question: can model hijacking be achieved more simply, with a higher attack success rate and more reasonable assumptions? In this paper, we propose CleanSheet, a new model hijacking attack that obtains the high performance of backdoor attacks without requiring the adversary to tamper with the model training process. CleanSheet exploits vulnerabilities in DNNs stemming from the training data. Specifically, our key idea is to treat part of the clean training data of the target model as "poisoned data," and capture the characteristics of these data that are more sensitive to the model (typically called robust features) to construct "triggers." These triggers can be added to any input example to mislead the target model, similar to backdoor attacks. We validate the effectiveness of CleanSheet through extensive experiments on 5 datasets, 79 normally trained models, 68 pruned models, and 39 defensive models. Results show that CleanSheet exhibits performance comparable to state-of-the-art backdoor attacks, achieving an average attack success rate (ASR) of 97.5% on CIFAR-100 and 92.4% on GTSRB, respectively. Furthermore, CleanSheet consistently maintains a high ASR, when confronted with various mainstream backdoor defenses.

Title: Image Translation as Diffusion Visual Programmers

• Authors: Cheng Han, James C. Liang, Qifan Wang, Majid Rabbani, Sohail Dianat, Raghuveer Rao, Ying Nian Wu, Dongfang Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.09742
• Pdf URL: https://arxiv.org/pdf/2401.09742
• Copy Paste: [[2401.09742]] Image Translation as Diffusion Visual Programmers(https://arxiv.org/abs/2401.09742)
• Keywords: explainability, diffusion
• Abstract: We introduce the novel Diffusion Visual Programmer (DVP), a neuro-symbolic image translation framework. Our proposed DVP seamlessly embeds a condition-flexible diffusion model within the GPT architecture, orchestrating a coherent sequence of visual programs (i.e., computer vision models) for various pro-symbolic steps, which span RoI identification, style transfer, and position manipulation, facilitating transparent and controllable image translation processes. Extensive experiments demonstrate DVP's remarkable performance, surpassing concurrent arts. This success can be attributed to several key features of DVP: First, DVP achieves condition-flexible translation via instance normalization, enabling the model to eliminate sensitivity caused by the manual guidance and optimally focus on textual descriptions for high-quality content generation. Second, the framework enhances in-context reasoning by deciphering intricate high-dimensional concepts in feature spaces into more accessible low-dimensional symbols (e.g., [Prompt], [RoI object]), allowing for localized, context-free editing while maintaining overall coherence. Last but not least, DVP improves systemic controllability and explainability by offering explicit symbolic representations at each programming stage, empowering users to intuitively interpret and modify results. Our research marks a substantial step towards harmonizing artificial image translation processes with cognitive intelligence, promising broader applications.

Title: Applications of Machine Learning to Optimizing Polyolefin Manufacturing

• Authors: Niket Sharma, Y.A. Liu
• Subjects: cs.LG, cs.CE
• Abstract URL: https://arxiv.org/abs/2401.09753
• Pdf URL: https://arxiv.org/pdf/2401.09753
• Copy Paste: [[2401.09753]] Applications of Machine Learning to Optimizing Polyolefin Manufacturing(https://arxiv.org/abs/2401.09753)
• Keywords: transformer
• Abstract: This chapter is a preprint from our book by , focusing on leveraging machine learning (ML) in chemical and polyolefin manufacturing optimization. It's crafted for both novices and seasoned professionals keen on the latest ML applications in chemical processes. We trace the evolution of AI and ML in chemical industries, delineate core ML components, and provide resources for ML beginners. A detailed discussion on various ML methods is presented, covering regression, classification, and unsupervised learning techniques, with performance metrics and examples. Ensemble methods, deep learning networks, including MLP, DNNs, RNNs, CNNs, and transformers, are explored for their growing role in chemical applications. Practical workshops guide readers through predictive modeling using advanced ML algorithms. The chapter culminates with insights into science-guided ML, advocating for a hybrid approach that enhances model accuracy. The extensive bibliography offers resources for further research and practical implementation. This chapter aims to be a thorough primer on ML's practical application in chemical engineering, particularly for polyolefin production, and sets the stage for continued learning in subsequent chapters. Please cite the original work [169,170] when referencing.

Title: Universally Robust Graph Neural Networks by Preserving Neighbor Similarity

• Authors: Yulin Zhu, Yuni Lai, Xing Ai, Kai Zhou
• Subjects: cs.LG, cs.CR, cs.SI
• Abstract URL: https://arxiv.org/abs/2401.09754
• Pdf URL: https://arxiv.org/pdf/2401.09754
• Copy Paste: [[2401.09754]] Universally Robust Graph Neural Networks by Preserving Neighbor Similarity(https://arxiv.org/abs/2401.09754)
• Keywords: attack, robust
• Abstract: Despite the tremendous success of graph neural networks in learning relational data, it has been widely investigated that graph neural networks are vulnerable to structural attacks on homophilic graphs. Motivated by this, a surge of robust models is crafted to enhance the adversarial robustness of graph neural networks on homophilic graphs. However, the vulnerability based on heterophilic graphs remains a mystery to us. To bridge this gap, in this paper, we start to explore the vulnerability of graph neural networks on heterophilic graphs and theoretically prove that the update of the negative classification loss is negatively correlated with the pairwise similarities based on the powered aggregated neighbor features. This theoretical proof explains the empirical observations that the graph attacker tends to connect dissimilar node pairs based on the similarities of neighbor features instead of ego features both on homophilic and heterophilic graphs. In this way, we novelly introduce a novel robust model termed NSPGNN which incorporates a dual-kNN graphs pipeline to supervise the neighbor similarity-guided propagation. This propagation utilizes the low-pass filter to smooth the features of node pairs along the positive kNN graphs and the high-pass filter to discriminate the features of node pairs along the negative kNN graphs. Extensive experiments on both homophilic and heterophilic graphs validate the universal robustness of NSPGNN compared to the state-of-the-art methods.

Title: A Comparative Study on Annotation Quality of Crowdsourcing and LLM via Label Aggregation

• Authors: Jiyi Li
• Subjects: cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2401.09760
• Pdf URL: https://arxiv.org/pdf/2401.09760
• Copy Paste: [[2401.09760]] A Comparative Study on Annotation Quality of Crowdsourcing and LLM via Label Aggregation(https://arxiv.org/abs/2401.09760)
• Keywords: large language model
• Abstract: Whether Large Language Models (LLMs) can outperform crowdsourcing on the data annotation task is attracting interest recently. Some works verified this issue with the average performance of individual crowd workers and LLM workers on some specific NLP tasks by collecting new datasets. However, on the one hand, existing datasets for the studies of annotation quality in crowdsourcing are not yet utilized in such evaluations, which potentially provide reliable evaluations from a different viewpoint. On the other hand, the quality of these aggregated labels is crucial because, when utilizing crowdsourcing, the estimated labels aggregated from multiple crowd labels to the same instances are the eventually collected labels. Therefore, in this paper, we first investigate which existing crowdsourcing datasets can be used for a comparative study and create a benchmark. We then compare the quality between individual crowd labels and LLM labels and make the evaluations on the aggregated labels. In addition, we propose a Crowd-LLM hybrid label aggregation method and verify the performance. We find that adding LLM labels from good LLMs to existing crowdsourcing datasets can enhance the quality of the aggregated labels of the datasets, which is also higher than the quality of LLM labels themselves.

Title: CLIP Model for Images to Textual Prompts Based on Top-k Neighbors

• Authors: Xin Zhang, Xin Zhang, YeMing Cai, Tianzhi Jia
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.09763
• Pdf URL: https://arxiv.org/pdf/2401.09763
• Copy Paste: [[2401.09763]] CLIP Model for Images to Textual Prompts Based on Top-k Neighbors(https://arxiv.org/abs/2401.09763)
• Keywords: generative
• Abstract: Text-to-image synthesis, a subfield of multimodal generation, has gained significant attention in recent years. We propose a cost-effective approach for image-to-prompt generation that leverages generative models to generate textual prompts without the need for large amounts of annotated data. We divide our method into two stages: online stage and offline stage. We use a combination of the CLIP model and K-nearest neighbors (KNN) algorithm. The proposed system consists of two main parts: an offline task and an online task. Our method owns the highest metric 0.612 among these models, which is 0.013, 0.055, 0.011 higher than Clip, Clip + KNN(top 10) respectively.

Title: SEINE: Structure Encoding and Interaction Network for Nuclei Instance Segmentation

• Authors: Ye Zhang, Linghan Cai, Ziyue Wang, Yongbing Zhang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.09773
• Pdf URL: https://arxiv.org/pdf/2401.09773
• Copy Paste: [[2401.09773]] SEINE: Structure Encoding and Interaction Network for Nuclei Instance Segmentation(https://arxiv.org/abs/2401.09773)
• Keywords: segmentation
• Abstract: Nuclei instance segmentation in histopathological images is of great importance for biological analysis and cancer diagnosis but remains challenging for two reasons. (1) Similar visual presentation of intranuclear and extranuclear regions of chromophobe nuclei often causes under-segmentation, and (2) current methods lack the exploration of nuclei structure, resulting in fragmented instance predictions. To address these problems, this paper proposes a structure encoding and interaction network, termed SEINE, which develops the structure modeling scheme of nuclei and exploits the structure similarity between nuclei to improve the integrality of each segmented instance. Concretely, SEINE introduces a contour-based structure encoding (SE) that considers the correlation between nuclei structure and semantics, realizing a reasonable representation of the nuclei structure. Based on the encoding, we propose a structure-guided attention (SGA) that takes the clear nuclei as prototypes to enhance the structure learning for the fuzzy nuclei. To strengthen the structural learning ability, a semantic feature fusion (SFF) is presented to boost the semantic consistency of semantic and structure branches. Furthermore, a position enhancement (PE) method is applied to suppress incorrect nuclei boundary predictions. Extensive experiments demonstrate the superiority of our approaches, and SEINE achieves state-of-the-art (SOTA) performance on four datasets. The code is available at \href{https://github.com/zhangye-zoe/SEINE}{https://github.com/zhangye-zoe/SEINE}.

Title: Controllable Decontextualization of Yes/No Question and Answers into Factual Statements

• Authors: Lingbo Mo, Besnik Fetahu, Oleg Rokhlenko, Shervin Malmasi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.09775
• Pdf URL: https://arxiv.org/pdf/2401.09775
• Copy Paste: [[2401.09775]] Controllable Decontextualization of Yes/No Question and Answers into Factual Statements(https://arxiv.org/abs/2401.09775)
• Keywords: transformer
• Abstract: Yes/No or polar questions represent one of the main linguistic question categories. They consist of a main interrogative clause, for which the answer is binary (assertion or negation). Polar questions and answers (PQA) represent a valuable knowledge resource present in many community and other curated QA sources, such as forums or e-commerce applications. Using answers to polar questions alone in other contexts is not trivial. Answers are contextualized, and presume that the interrogative question clause and any shared knowledge between the asker and answerer are provided. We address the problem of controllable rewriting of answers to polar questions into decontextualized and succinct factual statements. We propose a Transformer sequence to sequence model that utilizes soft-constraints to ensure controllable rewriting, such that the output statement is semantically equivalent to its PQA input. Evaluation on three separate PQA datasets as measured through automated and human evaluation metrics show that our proposed approach achieves the best performance when compared to existing baselines.

Title: Leveraging Biases in Large Language Models: "bias-kNN'' for Effective Few-Shot Learning

• Authors: Yong Zhang, Hanzhang Li, Zhitao Li, Ning Cheng, Ming Li, Jing Xiao, Jianzong Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.09783
• Pdf URL: https://arxiv.org/pdf/2401.09783
• Copy Paste: [[2401.09783]] Leveraging Biases in Large Language Models: "bias-kNN'' for Effective Few-Shot Learning(https://arxiv.org/abs/2401.09783)
• Keywords: robust, large language model
• Abstract: Large Language Models (LLMs) have shown significant promise in various applications, including zero-shot and few-shot learning. However, their performance can be hampered by inherent biases. Instead of traditionally sought methods that aim to minimize or correct these biases, this study introduces a novel methodology named ``bias-kNN''. This approach capitalizes on the biased outputs, harnessing them as primary features for kNN and supplementing with gold labels. Our comprehensive evaluations, spanning diverse domain text classification datasets and different GPT-2 model sizes, indicate the adaptability and efficacy of the ``bias-kNN'' method. Remarkably, this approach not only outperforms conventional in-context learning in few-shot scenarios but also demonstrates robustness across a spectrum of samples, templates and verbalizers. This study, therefore, presents a unique perspective on harnessing biases, transforming them into assets for enhanced model performance.

Title: Instant Answering in E-Commerce Buyer-Seller Messaging

• Authors: Besnik Fetahu, Tejas Mehta, Qun Song, Nikhita Vedula, Oleg Rokhlenko, Shervin Malmasi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.09785
• Pdf URL: https://arxiv.org/pdf/2401.09785
• Copy Paste: [[2401.09785]] Instant Answering in E-Commerce Buyer-Seller Messaging(https://arxiv.org/abs/2401.09785)
• Keywords: federate
• Abstract: E-commerce customers frequently seek detailed product information for purchase decisions, commonly contacting sellers directly with extended queries. This manual response requirement imposes additional costs and disrupts buyer's shopping experience with response time fluctuations ranging from hours to days. We seek to automate buyer inquiries to sellers in a leading e-commerce store using a domain-specific federated Question Answering (QA) system. The main challenge is adapting current QA systems, designed for single questions, to address detailed customer queries. We address this with a low-latency, sequence-to-sequence approach, MESSAGE-TO-QUESTION ( M2Q ). It reformulates buyer messages into succinct questions by identifying and extracting the most salient information from a message. Evaluation against baselines shows that M2Q yields relative increases of 757% in question understanding, and 1,746% in answering rate from the federated QA system. Live deployment shows that automatic answering saves sellers from manually responding to millions of messages per year, and also accelerates customer purchase decisions by eliminating the need for buyers to wait for a reply

Title: PatchAD: Patch-based MLP-Mixer for Time Series Anomaly Detection

• Authors: Zhijie Zhong, Zhiwen Yu, Yiyuan Yang, Weizheng Wang, Kaixiang Yang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.09793
• Pdf URL: https://arxiv.org/pdf/2401.09793
• Copy Paste: [[2401.09793]] PatchAD: Patch-based MLP-Mixer for Time Series Anomaly Detection(https://arxiv.org/abs/2401.09793)
• Keywords: extraction
• Abstract: Anomaly detection stands as a crucial aspect of time series analysis, aiming to identify abnormal events in time series samples. The central challenge of this task lies in effectively learning the representations of normal and abnormal patterns in a label-lacking scenario. Previous research mostly relied on reconstruction-based approaches, restricting the representational abilities of the models. In addition, most of the current deep learning-based methods are not lightweight enough, which prompts us to design a more efficient framework for anomaly detection. In this study, we introduce PatchAD, a novel multi-scale patch-based MLP-Mixer architecture that leverages contrastive learning for representational extraction and anomaly detection. Specifically, PatchAD is composed of four distinct MLP Mixers, exclusively utilizing the MLP architecture for high efficiency and lightweight architecture. Additionally, we also innovatively crafted a dual project constraint module to mitigate potential model degradation. Comprehensive experiments demonstrate that PatchAD achieves state-of-the-art results across multiple real-world multivariate time series datasets. Our code is publicly available.\footnote{\url{https://github.com/EmorZz1G/PatchAD}}

Title: Wavelet-Guided Acceleration of Text Inversion in Diffusion-Based Image Editing

• Authors: Gwanhyeong Koo, Sunjae Yoon, Chang D. Yoo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.09794
• Pdf URL: https://arxiv.org/pdf/2401.09794
• Copy Paste: [[2401.09794]] Wavelet-Guided Acceleration of Text Inversion in Diffusion-Based Image Editing(https://arxiv.org/abs/2401.09794)
• Keywords: diffusion
• Abstract: In the field of image editing, Null-text Inversion (NTI) enables fine-grained editing while preserving the structure of the original image by optimizing null embeddings during the DDIM sampling process. However, the NTI process is time-consuming, taking more than two minutes per image. To address this, we introduce an innovative method that maintains the principles of the NTI while accelerating the image editing process. We propose the WaveOpt-Estimator, which determines the text optimization endpoint based on frequency characteristics. Utilizing wavelet transform analysis to identify the image's frequency characteristics, we can limit text optimization to specific timesteps during the DDIM sampling process. By adopting the Negative-Prompt Inversion (NPI) concept, a target prompt representing the original image serves as the initial text value for optimization. This approach maintains performance comparable to NTI while reducing the average editing time by over 80% compared to the NTI method. Our method presents a promising approach for efficient, high-quality image editing based on diffusion models.

Title: A Fast, Performant, Secure Distributed Training Framework For Large Language Model

• Authors: Wei Huang, Yinggui Wang, Anda Cheng, Aihui Zhou, Chaofan Yu, Lei Wang
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2401.09796
• Pdf URL: https://arxiv.org/pdf/2401.09796
• Copy Paste: [[2401.09796]] A Fast, Performant, Secure Distributed Training Framework For Large Language Model(https://arxiv.org/abs/2401.09796)
• Keywords: secure, security, steal, federate
• Abstract: The distributed (federated) LLM is an important method for co-training the domain-specific LLM using siloed data. However, maliciously stealing model parameters and data from the server or client side has become an urgent problem to be solved. In this paper, we propose a secure distributed LLM based on model slicing. In this case, we deploy the Trusted Execution Environment (TEE) on both the client and server side, and put the fine-tuned structure (LoRA or embedding of P-tuning v2) into the TEE. Then, secure communication is executed in the TEE and general environments through lightweight encryption. In order to further reduce the equipment cost as well as increase the model performance and accuracy, we propose a split fine-tuning scheme. In particular, we split the LLM by layers and place the latter layers in a server-side TEE (the client does not need a TEE). We then combine the proposed Sparsification Parameter Fine-tuning (SPF) with the LoRA part to improve the accuracy of the downstream task. Numerous experiments have shown that our method guarantees accuracy while maintaining security.

Title: All in How You Ask for It: Simple Black-Box Method for Jailbreak Attacks

• Authors: Kazuhiro Takemoto
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2401.09798
• Pdf URL: https://arxiv.org/pdf/2401.09798
• Copy Paste: [[2401.09798]] All in How You Ask for It: Simple Black-Box Method for Jailbreak Attacks(https://arxiv.org/abs/2401.09798)
• Keywords: security, attack, large language model
• Abstract: Large Language Models (LLMs) like ChatGPT face `jailbreak' challenges, where safeguards are bypassed to produce ethically harmful prompts. This study introduces a simple black-box method to effectively generate jailbreak prompts, overcoming the limitations of high complexity and computational costs associated with existing methods. The proposed technique iteratively rewrites harmful prompts into non-harmful expressions using the target LLM itself, based on the hypothesis that LLMs can directly sample safeguard-bypassing expressions. Demonstrated through experiments with ChatGPT (GPT-3.5 and GPT-4) and Gemini-Pro, this method achieved an attack success rate of over 80% within an average of 5 iterations and remained effective despite model updates. The jailbreak prompts generated were naturally-worded and concise, suggesting they are less detectable. The results indicate that creating effective jailbreak prompts is simpler than previously considered, and black-box jailbreak attacks pose a more serious security threat.

Title: Enhancing Small Object Encoding in Deep Neural Networks: Introducing Fast&Focused-Net with Volume-wise Dot Product Layer

• Authors: Ali Tofik, Roy Partha Pratim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.09823
• Pdf URL: https://arxiv.org/pdf/2401.09823
• Copy Paste: [[2401.09823]] Enhancing Small Object Encoding in Deep Neural Networks: Introducing Fast&Focused-Net with Volume-wise Dot Product Layer(https://arxiv.org/abs/2401.09823)
• Keywords: transformer
• Abstract: In this paper, we introduce Fast&Focused-Net, a novel deep neural network architecture tailored for efficiently encoding small objects into fixed-length feature vectors. Contrary to conventional Convolutional Neural Networks (CNNs), Fast&Focused-Net employs a series of our newly proposed layer, the Volume-wise Dot Product (VDP) layer, designed to address several inherent limitations of CNNs. Specifically, CNNs often exhibit a smaller effective receptive field than their theoretical counterparts, limiting their vision span. Additionally, the initial layers in CNNs produce low-dimensional feature vectors, presenting a bottleneck for subsequent learning. Lastly, the computational overhead of CNNs, particularly in capturing diverse image regions by parameter sharing, is significantly high. The VDP layer, at the heart of Fast&Focused-Net, aims to remedy these issues by efficiently covering the entire image patch information with reduced computational demand. Experimental results demonstrate the prowess of Fast&Focused-Net in a variety of applications. For small object classification tasks, our network outperformed state-of-the-art methods on datasets such as CIFAR-10, CIFAR-100, STL-10, SVHN-Cropped, and Fashion-MNIST. In the context of larger image classification, when combined with a transformer encoder (ViT), Fast&Focused-Net produced competitive results for OpenImages V6, ImageNet-1K, and Places365 datasets. Moreover, the same combination showcased unparalleled performance in text recognition tasks across SVT, IC15, SVTP, and HOST datasets. This paper presents the architecture, the underlying motivation, and extensive empirical evidence suggesting that Fast&Focused-Net is a promising direction for efficient and focused deep learning.

Title: Boosting Few-Shot Semantic Segmentation Via Segment Anything Model

• Authors: Chen-Bin Feng, Qi Lai, Kangdao Liu, Houcheng Su, Chi-Man Vong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.09826
• Pdf URL: https://arxiv.org/pdf/2401.09826
• Copy Paste: [[2401.09826]] Boosting Few-Shot Semantic Segmentation Via Segment Anything Model(https://arxiv.org/abs/2401.09826)
• Keywords: segmentation
• Abstract: In semantic segmentation, accurate prediction masks are crucial for downstream tasks such as medical image analysis and image editing. Due to the lack of annotated data, few-shot semantic segmentation (FSS) performs poorly in predicting masks with precise contours. Recently, we have noticed that the large foundation model segment anything model (SAM) performs well in processing detailed features. Inspired by SAM, we propose FSS-SAM to boost FSS methods by addressing the issue of inaccurate contour. The FSS-SAM is training-free. It works as a post-processing tool for any FSS methods and can improve the accuracy of predicted masks. Specifically, we use predicted masks from FSS methods to generate prompts and then use SAM to predict new masks. To avoid predicting wrong masks with SAM, we propose a prediction result selection (PRS) algorithm. The algorithm can remarkably decrease wrong predictions. Experiment results on public datasets show that our method is superior to base FSS methods in both quantitative and qualitative aspects.

Title: Enhanced Automated Quality Assessment Network for Interactive Building Segmentation in High-Resolution Remote Sensing Imagery

• Authors: Zhili Zhang, Xiangyun Hu, Jiabo Xu
• Subjects: cs.CV, cs.HC
• Abstract URL: https://arxiv.org/abs/2401.09828
• Pdf URL: https://arxiv.org/pdf/2401.09828
• Copy Paste: [[2401.09828]] Enhanced Automated Quality Assessment Network for Interactive Building Segmentation in High-Resolution Remote Sensing Imagery(https://arxiv.org/abs/2401.09828)
• Keywords: robust, extraction, segmentation
• Abstract: In this research, we introduce the enhanced automated quality assessment network (IBS-AQSNet), an innovative solution for assessing the quality of interactive building segmentation within high-resolution remote sensing imagery. This is a new challenge in segmentation quality assessment, and our proposed IBS-AQSNet allievate this by identifying missed and mistaken segment areas. First of all, to acquire robust image features, our method combines a robust, pre-trained backbone with a lightweight counterpart for comprehensive feature extraction from imagery and segmentation results. These features are then fused through a simple combination of concatenation, convolution layers, and residual connections. Additionally, ISR-AQSNet incorporates a multi-scale differential quality assessment decoder, proficient in pinpointing areas where segmentation result is either missed or mistaken. Experiments on a newly-built EVLab-BGZ dataset, which includes over 39,198 buildings, demonstrate the superiority of the proposed method in automating segmentation quality assessment, thereby setting a new benchmark in the field.

Title: Exploring Latent Cross-Channel Embedding for Accurate 3D Human Pose Reconstruction in a Diffusion Framework

• Authors: Junkun Jiang, Jie Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.09836
• Pdf URL: https://arxiv.org/pdf/2401.09836
• Copy Paste: [[2401.09836]] Exploring Latent Cross-Channel Embedding for Accurate 3D Human Pose Reconstruction in a Diffusion Framework(https://arxiv.org/abs/2401.09836)
• Keywords: diffusion
• Abstract: Monocular 3D human pose estimation poses significant challenges due to the inherent depth ambiguities that arise during the reprojection process from 2D to 3D. Conventional approaches that rely on estimating an over-fit projection matrix struggle to effectively address these challenges and often result in noisy outputs. Recent advancements in diffusion models have shown promise in incorporating structural priors to address reprojection ambiguities. However, there is still ample room for improvement as these methods often overlook the exploration of correlation between the 2D and 3D joint-level features. In this study, we propose a novel cross-channel embedding framework that aims to fully explore the correlation between joint-level features of 3D coordinates and their 2D projections. In addition, we introduce a context guidance mechanism to facilitate the propagation of joint graph attention across latent channels during the iterative diffusion process. To evaluate the effectiveness of our proposed method, we conduct experiments on two benchmark datasets, namely Human3.6M and MPI-INF-3DHP. Our results demonstrate a significant improvement in terms of reconstruction accuracy compared to state-of-the-art methods. The code for our method will be made available online for further reference.

Title: MatSciRE: Leveraging Pointer Networks to Automate Entity and Relation Extraction for Material Science Knowledge-base Construction

• Authors: Ankan Mullick, Akash Ghosh, G Sai Chaitanya, Samir Ghui, Tapas Nayak, Seung-Cheol Lee, Satadeep Bhattacharjee, Pawan Goyal
• Subjects: cs.CL, cs.CE, cs.IR
• Abstract URL: https://arxiv.org/abs/2401.09839
• Pdf URL: https://arxiv.org/pdf/2401.09839
• Copy Paste: [[2401.09839]] MatSciRE: Leveraging Pointer Networks to Automate Entity and Relation Extraction for Material Science Knowledge-base Construction(https://arxiv.org/abs/2401.09839)
• Keywords: extraction
• Abstract: Material science literature is a rich source of factual information about various categories of entities (like materials and compositions) and various relations between these entities, such as conductivity, voltage, etc. Automatically extracting this information to generate a material science knowledge base is a challenging task. In this paper, we propose MatSciRE (Material Science Relation Extractor), a Pointer Network-based encoder-decoder framework, to jointly extract entities and relations from material science articles as a triplet ($entity1, relation, entity2$). Specifically, we target the battery materials and identify five relations to work on - conductivity, coulombic efficiency, capacity, voltage, and energy. Our proposed approach achieved a much better F1-score (0.771) than a previous attempt using ChemDataExtractor (0.716). The overall graphical framework of MatSciRE is shown in Fig 1. The material information is extracted from material science literature in the form of entity-relation triplets using MatSciRE.

Title: Enhancing the Fairness and Performance of Edge Cameras with Explainable AI

• Authors: Truong Thanh Hung Nguyen, Vo Thanh Khang Nguyen, Quoc Hung Cao, Van Binh Truong, Quoc Khanh Nguyen, Hung Cao
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.09852
• Pdf URL: https://arxiv.org/pdf/2401.09852
• Copy Paste: [[2401.09852]] Enhancing the Fairness and Performance of Edge Cameras with Explainable AI(https://arxiv.org/abs/2401.09852)
• Keywords: fair
• Abstract: The rising use of Artificial Intelligence (AI) in human detection on Edge camera systems has led to accurate but complex models, challenging to interpret and debug. Our research presents a diagnostic method using Explainable AI (XAI) for model debugging, with expert-driven problem identification and solution creation. Validated on the Bytetrack model in a real-world office Edge network, we found the training dataset as the main bias source and suggested model augmentation as a solution. Our approach helps identify model biases, essential for achieving fair and trustworthy models.

Title: Temporal Insight Enhancement: Mitigating Temporal Hallucination in Multimodal Large Language Models

• Authors: Li Sun, Liuan Wang, Jun Sun, Takayuki Okatani
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.09861
• Pdf URL: https://arxiv.org/pdf/2401.09861
• Copy Paste: [[2401.09861]] Temporal Insight Enhancement: Mitigating Temporal Hallucination in Multimodal Large Language Models(https://arxiv.org/abs/2401.09861)
• Keywords: large language model
• Abstract: Recent advancements in Multimodal Large Language Models (MLLMs) have significantly enhanced the comprehension of multimedia content, bringing together diverse modalities such as text, images, and videos. However, a critical challenge faced by these models, especially when processing video inputs, is the occurrence of hallucinations - erroneous perceptions or interpretations, particularly at the event level. This study introduces an innovative method to address event-level hallucinations in MLLMs, focusing on specific temporal understanding in video content. Our approach leverages a novel framework that extracts and utilizes event-specific information from both the event query and the provided video to refine MLLMs' response. We propose a unique mechanism that decomposes on-demand event queries into iconic actions. Subsequently, we employ models like CLIP and BLIP2 to predict specific timestamps for event occurrences. Our evaluation, conducted using the Charades-STA dataset, demonstrates a significant reduction in temporal hallucinations and an improvement in the quality of event-related responses. This research not only provides a new perspective in addressing a critical limitation of MLLMs but also contributes a quantitatively measurable method for evaluating MLLMs in the context of temporal-related questions.

Title: Improving fine-grained understanding in image-text pre-training

• Authors: Ioana Bica, Anastasija Ilić, Matthias Bauer, Goker Erdogan, Matko Bošnjak, Christos Kaplanis, Alexey A. Gritsenko, Matthias Minderer, Charles Blundell, Razvan Pascanu, Jovana Mitrović
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.09865
• Pdf URL: https://arxiv.org/pdf/2401.09865
• Copy Paste: [[2401.09865]] Improving fine-grained understanding in image-text pre-training(https://arxiv.org/abs/2401.09865)
• Keywords: segmentation
• Abstract: We introduce SPARse Fine-grained Contrastive Alignment (SPARC), a simple method for pretraining more fine-grained multimodal representations from image-text pairs. Given that multiple image patches often correspond to single words, we propose to learn a grouping of image patches for every token in the caption. To achieve this, we use a sparse similarity metric between image patches and language tokens and compute for each token a language-grouped vision embedding as the weighted average of patches. The token and language-grouped vision embeddings are then contrasted through a fine-grained sequence-wise loss that only depends on individual samples and does not require other batch samples as negatives. This enables more detailed information to be learned in a computationally inexpensive manner. SPARC combines this fine-grained loss with a contrastive loss between global image and text embeddings to learn representations that simultaneously encode global and local information. We thoroughly evaluate our proposed method and show improved performance over competing approaches both on image-level tasks relying on coarse-grained information, e.g. classification, as well as region-level tasks relying on fine-grained information, e.g. retrieval, object detection, and segmentation. Moreover, SPARC improves model faithfulness and captioning in foundational vision-language models.

Title: Boosting Few-Shot Segmentation via Instance-Aware Data Augmentation and Local Consensus Guided Cross Attention

• Authors: Li Guo, Haoming Liu, Yuxuan Xia, Chengyu Zhang, Xiaochen Lu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.09866
• Pdf URL: https://arxiv.org/pdf/2401.09866
• Copy Paste: [[2401.09866]] Boosting Few-Shot Segmentation via Instance-Aware Data Augmentation and Local Consensus Guided Cross Attention(https://arxiv.org/abs/2401.09866)
• Keywords: segmentation
• Abstract: Few-shot segmentation aims to train a segmentation model that can fast adapt to a novel task for which only a few annotated images are provided. Most recent models have adopted a prototype-based paradigm for few-shot inference. These approaches may have limited generalization capacity beyond the standard 1- or 5-shot settings. In this paper, we closely examine and reevaluate the fine-tuning based learning scheme that fine-tunes the classification layer of a deep segmentation network pre-trained on diverse base classes. To improve the generalizability of the classification layer optimized with sparsely annotated samples, we introduce an instance-aware data augmentation (IDA) strategy that augments the support images based on the relative sizes of the target objects. The proposed IDA effectively increases the support set's diversity and promotes the distribution consistency between support and query images. On the other hand, the large visual difference between query and support images may hinder knowledge transfer and cripple the segmentation performance. To cope with this challenge, we introduce the local consensus guided cross attention (LCCA) to align the query feature with support features based on their dense correlation, further improving the model's generalizability to the query image. The significant performance improvements on the standard few-shot segmentation benchmarks PASCAL-$5^i$ and COCO-$20^i$ verify the efficacy of our proposed method.

Title: GA-SmaAt-GNet: Generative Adversarial Small Attention GNet for Extreme Precipitation Nowcasting

• Authors: Eloy Reulen, Siamak Mehrkanoon
• Subjects: cs.LG, physics.ao-ph
• Abstract URL: https://arxiv.org/abs/2401.09881
• Pdf URL: https://arxiv.org/pdf/2401.09881
• Copy Paste: [[2401.09881]] GA-SmaAt-GNet: Generative Adversarial Small Attention GNet for Extreme Precipitation Nowcasting(https://arxiv.org/abs/2401.09881)
• Keywords: generative
• Abstract: In recent years, data-driven modeling approaches have gained considerable traction in various meteorological applications, particularly in the realm of weather forecasting. However, these approaches often encounter challenges when dealing with extreme weather conditions. In light of this, we propose GA-SmaAt-GNet, a novel generative adversarial architecture that makes use of two methodologies aimed at enhancing the performance of deep learning models for extreme precipitation nowcasting. Firstly, it uses a novel SmaAt-GNet built upon the successful SmaAt-UNet architecture as generator. This network incorporates precipitation masks (binarized precipitation maps) as an additional data source, leveraging valuable information for improved predictions. Additionally, GA-SmaAt-GNet utilizes an attention-augmented discriminator inspired by the well-established Pix2Pix architecture. Furthermore, we assess the performance of GA-SmaAt-GNet using real-life precipitation dataset from the Netherlands. Our experimental results reveal a notable improvement in both overall performance and for extreme precipitation events. Furthermore, we conduct uncertainty analysis on the proposed GA-SmaAt-GNet model as well as on the precipitation dataset, providing additional insights into the predictive capabilities of the model. Finally, we offer further insights into the predictions of our proposed model using Grad-CAM. This visual explanation technique generates activation heatmaps, illustrating areas of the input that are more activated for various parts of the network.

Title: Question-Answer Cross Language Image Matching for Weakly Supervised Semantic Segmentation

• Authors: Songhe Deng, Wei Zhuo, Jinheng Xie, Linlin Shen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.09883
• Pdf URL: https://arxiv.org/pdf/2401.09883
• Copy Paste: [[2401.09883]] Question-Answer Cross Language Image Matching for Weakly Supervised Semantic Segmentation(https://arxiv.org/abs/2401.09883)
• Keywords: segmentation
• Abstract: Class Activation Map (CAM) has emerged as a popular tool for weakly supervised semantic segmentation (WSSS), allowing the localization of object regions in an image using only image-level labels. However, existing CAM methods suffer from under-activation of target object regions and false-activation of background regions due to the fact that a lack of detailed supervision can hinder the model's ability to understand the image as a whole. In this paper, we propose a novel Question-Answer Cross-Language-Image Matching framework for WSSS (QA-CLIMS), leveraging the vision-language foundation model to maximize the text-based understanding of images and guide the generation of activation maps. First, a series of carefully designed questions are posed to the VQA (Visual Question Answering) model with Question-Answer Prompt Engineering (QAPE) to generate a corpus of both foreground target objects and backgrounds that are adaptive to query images. We then employ contrastive learning in a Region Image Text Contrastive (RITC) network to compare the obtained foreground and background regions with the generated corpus. Our approach exploits the rich textual information from the open vocabulary as additional supervision, enabling the model to generate high-quality CAMs with a more complete object region and reduce false-activation of background regions. We conduct extensive analysis to validate the proposed method and show that our approach performs state-of-the-art on both PASCAL VOC 2012 and MS COCO datasets. Code is available at: https://github.com/CVI-SZU/QA-CLIMS

Title: Cooperative Edge Caching Based on Elastic Federated and Multi-Agent Deep Reinforcement Learning in Next-Generation Network

• Authors: Qiong Wu, Wenhua Wang, Pingyi Fan, Qiang Fan, Huiling Zhu, Khaled B. Letaief
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.09886
• Pdf URL: https://arxiv.org/pdf/2401.09886
• Copy Paste: [[2401.09886]] Cooperative Edge Caching Based on Elastic Federated and Multi-Agent Deep Reinforcement Learning in Next-Generation Network(https://arxiv.org/abs/2401.09886)
• Keywords: privacy, protect, federate
• Abstract: Edge caching is a promising solution for next-generation networks by empowering caching units in small-cell base stations (SBSs), which allows user equipments (UEs) to fetch users' requested contents that have been pre-cached in SBSs. It is crucial for SBSs to predict accurate popular contents through learning while protecting users' personal information. Traditional federated learning (FL) can protect users' privacy but the data discrepancies among UEs can lead to a degradation in model quality. Therefore, it is necessary to train personalized local models for each UE to predict popular contents accurately. In addition, the cached contents can be shared among adjacent SBSs in next-generation networks, thus caching predicted popular contents in different SBSs may affect the cost to fetch contents. Hence, it is critical to determine where the popular contents are cached cooperatively. To address these issues, we propose a cooperative edge caching scheme based on elastic federated and multi-agent deep reinforcement learning (CEFMR) to optimize the cost in the network. We first propose an elastic FL algorithm to train the personalized model for each UE, where adversarial autoencoder (AAE) model is adopted for training to improve the prediction accuracy, then {a popular} content prediction algorithm is proposed to predict the popular contents for each SBS based on the trained AAE model. Finally, we propose a multi-agent deep reinforcement learning (MADRL) based algorithm to decide where the predicted popular contents are collaboratively cached among SBSs. Our experimental results demonstrate the superiority of our proposed scheme to existing baseline caching schemes.

Title: Skeleton-Guided Instance Separation for Fine-Grained Segmentation in Microscopy

• Authors: Jun Wang, Chengfeng Zhou, Zhaoyan Ming, Lina Wei, Xudong Jiang, Dahong Qian
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.09895
• Pdf URL: https://arxiv.org/pdf/2401.09895
• Copy Paste: [[2401.09895]] Skeleton-Guided Instance Separation for Fine-Grained Segmentation in Microscopy(https://arxiv.org/abs/2401.09895)
• Keywords: segmentation
• Abstract: One of the fundamental challenges in microscopy (MS) image analysis is instance segmentation (IS), particularly when segmenting cluster regions where multiple objects of varying sizes and shapes may be connected or even overlapped in arbitrary orientations. Existing IS methods usually fail in handling such scenarios, as they rely on coarse instance representations such as keypoints and horizontal bounding boxes (h-bboxes). In this paper, we propose a novel one-stage framework named A2B-IS to address this challenge and enhance the accuracy of IS in MS images. Our approach represents each instance with a pixel-level mask map and a rotated bounding box (r-bbox). Unlike two-stage methods that use box proposals for segmentations, our method decouples mask and box predictions, enabling simultaneous processing to streamline the model pipeline. Additionally, we introduce a Gaussian skeleton map to aid the IS task in two key ways: (1) It guides anchor placement, reducing computational costs while improving the model's capacity to learn RoI-aware features by filtering out noise from background regions. (2) It ensures accurate isolation of densely packed instances by rectifying erroneous box predictions near instance boundaries. To further enhance the performance, we integrate two modules into the framework: (1) An Atrous Attention Block (A2B) designed to extract high-resolution feature maps with fine-grained multiscale information, and (2) A Semi-Supervised Learning (SSL) strategy that leverages both labeled and unlabeled images for model training. Our method has been thoroughly validated on two large-scale MS datasets, demonstrating its superiority over most state-of-the-art approaches.

Title: Meme-ingful Analysis: Enhanced Understanding of Cyberbullying in Memes Through Multimodal Explanations

• Authors: Prince Jha, Krishanu Maity, Raghav Jain, Apoorv Verma, Sriparna Saha, Pushpak Bhattacharyya
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.09899
• Pdf URL: https://arxiv.org/pdf/2401.09899
• Copy Paste: [[2401.09899]] Meme-ingful Analysis: Enhanced Understanding of Cyberbullying in Memes Through Multimodal Explanations(https://arxiv.org/abs/2401.09899)
• Keywords: protect, explainability
• Abstract: Internet memes have gained significant influence in communicating political, psychological, and sociocultural ideas. While memes are often humorous, there has been a rise in the use of memes for trolling and cyberbullying. Although a wide variety of effective deep learning-based models have been developed for detecting offensive multimodal memes, only a few works have been done on explainability aspect. Recent laws like "right to explanations" of General Data Protection Regulation, have spurred research in developing interpretable models rather than only focusing on performance. Motivated by this, we introduce {\em MultiBully-Ex}, the first benchmark dataset for multimodal explanation from code-mixed cyberbullying memes. Here, both visual and textual modalities are highlighted to explain why a given meme is cyberbullying. A Contrastive Language-Image Pretraining (CLIP) projection-based multimodal shared-private multitask approach has been proposed for visual and textual explanation of a meme. Experimental results demonstrate that training with multimodal explanations improves performance in generating textual justifications and more accurately identifying the visual evidence supporting a decision with reliable performance improvements.

Title: XAI-Enhanced Semantic Segmentation Models for Visual Quality Inspection

• Authors: Tobias Clement, Truong Thanh Hung Nguyen, Mohamed Abdelaal, Hung Cao
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.09900
• Pdf URL: https://arxiv.org/pdf/2401.09900
• Copy Paste: [[2401.09900]] XAI-Enhanced Semantic Segmentation Models for Visual Quality Inspection(https://arxiv.org/abs/2401.09900)
• Keywords: segmentation
• Abstract: Visual quality inspection systems, crucial in sectors like manufacturing and logistics, employ computer vision and machine learning for precise, rapid defect detection. However, their unexplained nature can hinder trust, error identification, and system improvement. This paper presents a framework to bolster visual quality inspection by using CAM-based explanations to refine semantic segmentation models. Our approach consists of 1) Model Training, 2) XAI-based Model Explanation, 3) XAI Evaluation, and 4) Annotation Augmentation for Model Enhancement, informed by explanations and expert insights. Evaluations show XAI-enhanced models surpass original DeepLabv3-ResNet101 models, especially in intricate object segmentation.

Title: Probabilistic Truly Unordered Rule Sets

• Authors: Lincen Yang, Matthijs van Leeuwen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.09918
• Pdf URL: https://arxiv.org/pdf/2401.09918
• Copy Paste: [[2401.09918]] Probabilistic Truly Unordered Rule Sets(https://arxiv.org/abs/2401.09918)
• Keywords: interpretability
• Abstract: Rule set learning has recently been frequently revisited because of its interpretability. Existing methods have several shortcomings though. First, most existing methods impose orders among rules, either explicitly or implicitly, which makes the models less comprehensible. Second, due to the difficulty of handling conflicts caused by overlaps (i.e., instances covered by multiple rules), existing methods often do not consider probabilistic rules. Third, learning classification rules for multi-class target is understudied, as most existing methods focus on binary classification or multi-class classification via the ``one-versus-rest" approach. To address these shortcomings, we propose TURS, for Truly Unordered Rule Sets. To resolve conflicts caused by overlapping rules, we propose a novel model that exploits the probabilistic properties of our rule sets, with the intuition of only allowing rules to overlap if they have similar probabilistic outputs. We next formalize the problem of learning a TURS model based on the MDL principle and develop a carefully designed heuristic algorithm. We benchmark against a wide range of rule-based methods and demonstrate that our method learns rule sets that have lower model complexity and highly competitive predictive performance. In addition, we empirically show that rules in our model are empirically ``independent" and hence truly unordered.

Title: BlenDA: Domain Adaptive Object Detection through diffusion-based blending

• Authors: Tzuhsuan Huang, Chen-Che Huang, Chung-Hao Ku, Jun-Cheng Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.09921
• Pdf URL: https://arxiv.org/pdf/2401.09921
• Copy Paste: [[2401.09921]] BlenDA: Domain Adaptive Object Detection through diffusion-based blending(https://arxiv.org/abs/2401.09921)
• Keywords: diffusion, transformer
• Abstract: Unsupervised domain adaptation (UDA) aims to transfer a model learned using labeled data from the source domain to unlabeled data in the target domain. To address the large domain gap issue between the source and target domains, we propose a novel regularization method for domain adaptive object detection, BlenDA, by generating the pseudo samples of the intermediate domains and their corresponding soft domain labels for adaptation training. The intermediate samples are generated by dynamically blending the source images with their corresponding translated images using an off-the-shelf pre-trained text-to-image diffusion model which takes the text label of the target domain as input and has demonstrated superior image-to-image translation quality. Based on experimental results from two adaptation benchmarks, our proposed approach can significantly enhance the performance of the state-of-the-art domain adaptive object detector, Adversarial Query Transformer (AQT). Particularly, in the Cityscapes to Foggy Cityscapes adaptation, we achieve an impressive 53.4% mAP on the Foggy Cityscapes dataset, surpassing the previous state-of-the-art by 1.5%. It is worth noting that our proposed method is also applicable to various paradigms of domain adaptive object detection. The code is available at:https://github.com/aiiu-lab/BlenDA

Title: Biases in Expected Goals Models Confound Finishing Ability

• Authors: Jesse Davis, Pieter Robberechts
• Subjects: cs.LG, stat.AP
• Abstract URL: https://arxiv.org/abs/2401.09940
• Pdf URL: https://arxiv.org/pdf/2401.09940
• Copy Paste: [[2401.09940]] Biases in Expected Goals Models Confound Finishing Ability(https://arxiv.org/abs/2401.09940)
• Keywords: attack, fair
• Abstract: Expected Goals (xG) has emerged as a popular tool for evaluating finishing skill in soccer analytics. It involves comparing a player's cumulative xG with their actual goal output, where consistent overperformance indicates strong finishing ability. However, the assessment of finishing skill in soccer using xG remains contentious due to players' difficulty in consistently outperforming their cumulative xG. In this paper, we aim to address the limitations and nuances surrounding the evaluation of finishing skill using xG statistics. Specifically, we explore three hypotheses: (1) the deviation between actual and expected goals is an inadequate metric due to the high variance of shot outcomes and limited sample sizes, (2) the inclusion of all shots in cumulative xG calculation may be inappropriate, and (3) xG models contain biases arising from interdependencies in the data that affect skill measurement. We found that sustained overperformance of cumulative xG requires both high shot volumes and exceptional finishing, including all shot types can obscure the finishing ability of proficient strikers, and that there is a persistent bias that makes the actual and expected goals closer for excellent finishers than it really is. Overall, our analysis indicates that we need more nuanced quantitative approaches for investigating a player's finishing ability, which we achieved using a technique from AI fairness to learn an xG model that is calibrated for multiple subgroups of players. As a concrete use case, we show that (1) the standard biased xG model underestimates Messi's GAX by 17% and (2) Messi's GAX is 27% higher than the typical elite high-shot-volume attacker, indicating that Messi is even a more exceptional finisher than people commonly believed.

Title: HGAttack: Transferable Heterogeneous Graph Adversarial Attack

• Authors: He Zhao, Zhiwei Zeng, Yongwei Wang, Deheng Ye, Chunyan Miao
• Subjects: cs.LG, cs.CR, cs.IR
• Abstract URL: https://arxiv.org/abs/2401.09945
• Pdf URL: https://arxiv.org/pdf/2401.09945
• Copy Paste: [[2401.09945]] HGAttack: Transferable Heterogeneous Graph Adversarial Attack(https://arxiv.org/abs/2401.09945)
• Keywords: attack, robust
• Abstract: Heterogeneous Graph Neural Networks (HGNNs) are increasingly recognized for their performance in areas like the web and e-commerce, where resilience against adversarial attacks is crucial. However, existing adversarial attack methods, which are primarily designed for homogeneous graphs, fall short when applied to HGNNs due to their limited ability to address the structural and semantic complexity of HGNNs. This paper introduces HGAttack, the first dedicated gray box evasion attack method for heterogeneous graphs. We design a novel surrogate model to closely resemble the behaviors of the target HGNN and utilize gradient-based methods for perturbation generation. Specifically, the proposed surrogate model effectively leverages heterogeneous information by extracting meta-path induced subgraphs and applying GNNs to learn node embeddings with distinct semantics from each subgraph. This approach improves the transferability of generated attacks on the target HGNN and significantly reduces memory costs. For perturbation generation, we introduce a semantics-aware mechanism that leverages subgraph gradient information to autonomously identify vulnerable edges across a wide range of relations within a constrained perturbation budget. We validate HGAttack's efficacy with comprehensive experiments on three datasets, providing empirical analyses of its generated perturbations. Outperforming baseline methods, HGAttack demonstrated significant efficacy in diminishing the performance of target HGNN models, affirming the effectiveness of our approach in evaluating the robustness of HGNNs against adversarial attacks.

Title: CustomVideo: Customizing Text-to-Video Generation with Multiple Subjects

• Authors: Zhao Wang, Aoxue Li, Enze Xie, Lingting Zhu, Yong Guo, Qi Dou, Zhenguo Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.09962
• Pdf URL: https://arxiv.org/pdf/2401.09962
• Copy Paste: [[2401.09962]] CustomVideo: Customizing Text-to-Video Generation with Multiple Subjects(https://arxiv.org/abs/2401.09962)
• Keywords: diffusion
• Abstract: Customized text-to-video generation aims to generate high-quality videos guided by text prompts and subject references. Current approaches designed for single subjects suffer from tackling multiple subjects, which is a more challenging and practical scenario. In this work, we aim to promote multi-subject guided text-to-video customization. We propose CustomVideo, a novel framework that can generate identity-preserving videos with the guidance of multiple subjects. To be specific, firstly, we encourage the co-occurrence of multiple subjects via composing them in a single image. Further, upon a basic text-to-video diffusion model, we design a simple yet effective attention control strategy to disentangle different subjects in the latent space of diffusion model. Moreover, to help the model focus on the specific object area, we segment the object from given reference images and provide a corresponding object mask for attention learning. Also, we collect a multi-subject text-to-video generation dataset as a comprehensive benchmark, with 69 individual subjects and 57 meaningful pairs. Extensive qualitative, quantitative, and user study results demonstrate the superiority of our method, compared with the previous state-of-the-art approaches.

Title: Sketch-Guided Constrained Decoding for Boosting Blackbox Large Language Models without Logit Access

• Authors: Saibo Geng, Berkay Döner, Chris Wendler, Martin Josifoski, Robert West
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.09967
• Pdf URL: https://arxiv.org/pdf/2401.09967
• Copy Paste: [[2401.09967]] Sketch-Guided Constrained Decoding for Boosting Blackbox Large Language Models without Logit Access(https://arxiv.org/abs/2401.09967)
• Keywords: extraction, large language model
• Abstract: Constrained decoding, a technique for enforcing constraints on language model outputs, offers a way to control text generation without retraining or architectural modifications. Its application is, however, typically restricted to models that give users access to next-token distributions (usually via softmax logits), which poses a limitation with blackbox large language models (LLMs). This paper introduces sketch-guided constrained decoding (SGCD), a novel approach to constrained decoding for blackbox LLMs, which operates without access to the logits of the blackbox LLM. SGCD utilizes a locally hosted auxiliary model to refine the output of an unconstrained blackbox LLM, effectively treating this initial output as a "sketch" for further elaboration. This approach is complementary to traditional logit-based techniques and enables the application of constrained decoding in settings where full model transparency is unavailable. We demonstrate the efficacy of SGCD through experiments in closed information extraction and constituency parsing, showing how it enhances the utility and flexibility of blackbox LLMs for complex NLP tasks.

Title: Better Explain Transformers by Illuminating Important Information

• Authors: Linxin Song, Yan Cui, Ao Luo, Freddy Lecue, Irene Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.09972
• Pdf URL: https://arxiv.org/pdf/2401.09972
• Copy Paste: [[2401.09972]] Better Explain Transformers by Illuminating Important Information(https://arxiv.org/abs/2401.09972)
• Keywords: transformer
• Abstract: Transformer-based models excel in various natural language processing (NLP) tasks, attracting countless efforts to explain their inner workings. Prior methods explain Transformers by focusing on the raw gradient and attention as token attribution scores, where non-relevant information is often considered during explanation computation, resulting in confusing results. In this work, we propose highlighting the important information and eliminating irrelevant information by a refined information flow on top of the layer-wise relevance propagation (LRP) method. Specifically, we consider identifying syntactic and positional heads as important attention heads and focus on the relevance obtained from these important heads. Experimental results demonstrate that irrelevant information does distort output attribution scores and then should be masked during explanation computation. Compared to eight baselines on both classification and question-answering datasets, our method consistently outperforms with over 3\% to 33\% improvement on explanation metrics, providing superior explanation performance. Our anonymous code repository is available at: https://github.com/LinxinS97/Mask-LRP

Title: WorldDreamer: Towards General World Models for Video Generation via Predicting Masked Tokens

• Authors: Xiaofeng Wang, Zheng Zhu, Guan Huang, Boyuan Wang, Xinze Chen, Jiwen Lu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.09985
• Pdf URL: https://arxiv.org/pdf/2401.09985
• Copy Paste: [[2401.09985]] WorldDreamer: Towards General World Models for Video Generation via Predicting Masked Tokens(https://arxiv.org/abs/2401.09985)
• Keywords: large language model
• Abstract: World models play a crucial role in understanding and predicting the dynamics of the world, which is essential for video generation. However, existing world models are confined to specific scenarios such as gaming or driving, limiting their ability to capture the complexity of general world dynamic environments. Therefore, we introduce WorldDreamer, a pioneering world model to foster a comprehensive comprehension of general world physics and motions, which significantly enhances the capabilities of video generation. Drawing inspiration from the success of large language models, WorldDreamer frames world modeling as an unsupervised visual sequence modeling challenge. This is achieved by mapping visual inputs to discrete tokens and predicting the masked ones. During this process, we incorporate multi-modal prompts to facilitate interaction within the world model. Our experiments show that WorldDreamer excels in generating videos across different scenarios, including natural scenes and driving environments. WorldDreamer showcases versatility in executing tasks such as text-to-video conversion, image-tovideo synthesis, and video editing. These results underscore WorldDreamer's effectiveness in capturing dynamic elements within diverse general world environments.

Title: FLex&Chill: Improving Local Federated Learning Training with Logit Chilling

• Authors: Kichang Lee, Songkuk Kim, JeongGil Ko
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.09986
• Pdf URL: https://arxiv.org/pdf/2401.09986
• Copy Paste: [[2401.09986]] FLex&Chill: Improving Local Federated Learning Training with Logit Chilling(https://arxiv.org/abs/2401.09986)
• Keywords: federate
• Abstract: Federated learning are inherently hampered by data heterogeneity: non-iid distributed training data over local clients. We propose a novel model training approach for federated learning, FLex&Chill, which exploits the Logit Chilling method. Through extensive evaluations, we demonstrate that, in the presence of non-iid data characteristics inherent in federated learning systems, this approach can expedite model convergence and improve inference accuracy. Quantitatively, from our experiments, we observe up to 6X improvement in the global federated learning model convergence time, and up to 3.37% improvement in inference accuracy.

Title: Developing an AI-based Integrated System for Bee Health Evaluation

• Authors: Andrew Liang
• Subjects: cs.LG, cs.CV, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2401.09988
• Pdf URL: https://arxiv.org/pdf/2401.09988
• Copy Paste: [[2401.09988]] Developing an AI-based Integrated System for Bee Health Evaluation(https://arxiv.org/abs/2401.09988)
• Keywords: robust
• Abstract: Honey bees pollinate about one-third of the world's food supply, but bee colonies have alarmingly declined by nearly 40% over the past decade due to several factors, including pesticides and pests. Traditional methods for monitoring beehives, such as human inspection, are subjective, disruptive, and time-consuming. To overcome these limitations, artificial intelligence has been used to assess beehive health. However, previous studies have lacked an end-to-end solution and primarily relied on data from a single source, either bee images or sounds. This study introduces a comprehensive system consisting of bee object detection and health evaluation. Additionally, it utilized a combination of visual and audio signals to analyze bee behaviors. An Attention-based Multimodal Neural Network (AMNN) was developed to adaptively focus on key features from each type of signal for accurate bee health assessment. The AMNN achieved an overall accuracy of 92.61%, surpassing eight existing single-signal Convolutional Neural Networks and Recurrent Neural Networks. It outperformed the best image-based model by 32.51% and the top sound-based model by 13.98% while maintaining efficient processing times. Furthermore, it improved prediction robustness, attaining an F1-score higher than 90% across all four evaluated health conditions. The study also shows that audio signals are more reliable than images for assessing bee health. By seamlessly integrating AMNN with image and sound data in a comprehensive bee health monitoring system, this approach provides a more efficient and non-invasive solution for the early detection of bee diseases and the preservation of bee colonies.

Title: BPDO:Boundary Points Dynamic Optimization for Arbitrary Shape Scene Text Detection

• Authors: Jinzhi Zheng, Libo Zhang, Yanjun Wu, Chen Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.09997
• Pdf URL: https://arxiv.org/pdf/2401.09997
• Copy Paste: [[2401.09997]] BPDO:Boundary Points Dynamic Optimization for Arbitrary Shape Scene Text Detection(https://arxiv.org/abs/2401.09997)
• Keywords: segmentation
• Abstract: Arbitrary shape scene text detection is of great importance in scene understanding tasks. Due to the complexity and diversity of text in natural scenes, existing scene text algorithms have limited accuracy for detecting arbitrary shape text. In this paper, we propose a novel arbitrary shape scene text detector through boundary points dynamic optimization(BPDO). The proposed model is designed with a text aware module (TAM) and a boundary point dynamic optimization module (DOM). Specifically, the model designs a text aware module based on segmentation to obtain boundary points describing the central region of the text by extracting a priori information about the text region. Then, based on the idea of deformable attention, it proposes a dynamic optimization model for boundary points, which gradually optimizes the exact position of the boundary points based on the information of the adjacent region of each boundary point. Experiments on CTW-1500, Total-Text, and MSRA-TD500 datasets show that the model proposed in this paper achieves a performance that is better than or comparable to the state-of-the-art algorithm, proving the effectiveness of the model.

Title: Distantly Supervised Morpho-Syntactic Model for Relation Extraction

• Authors: Nicolas Gutehrlé, Iana Atanassova
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.10002
• Pdf URL: https://arxiv.org/pdf/2401.10002
• Copy Paste: [[2401.10002]] Distantly Supervised Morpho-Syntactic Model for Relation Extraction(https://arxiv.org/abs/2401.10002)
• Keywords: extraction
• Abstract: The task of Information Extraction (IE) involves automatically converting unstructured textual content into structured data. Most research in this field concentrates on extracting all facts or a specific set of relationships from documents. In this paper, we present a method for the extraction and categorisation of an unrestricted set of relationships from text. Our method relies on morpho-syntactic extraction patterns obtained by a distant supervision method, and creates Syntactic and Semantic Indices to extract and classify candidate graphs. We evaluate our approach on six datasets built on Wikidata and Wikipedia. The evaluation shows that our approach can achieve Precision scores of up to 0.85, but with lower Recall and F1 scores. Our approach allows to quickly create rule-based systems for Information Extraction and to build annotated datasets to train machine-learning and deep-learning based classifiers.

Title: Advancing Large Multi-modal Models with Explicit Chain-of-Reasoning and Visual Question Generation

• Authors: Kohei Uehara, Nabarun Goswami, Hanqin Wang, Toshiaki Baba, Kohtaro Tanaka, Tomohiro Hashimoto, Kai Wang, Rei Ito, Takagi Naoya, Ryo Umagami, Yingyi Wen, Tanachai Anakewat, Tatsuya Harada
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2401.10005
• Pdf URL: https://arxiv.org/pdf/2401.10005
• Copy Paste: [[2401.10005]] Advancing Large Multi-modal Models with Explicit Chain-of-Reasoning and Visual Question Generation(https://arxiv.org/abs/2401.10005)
• Keywords: robust, large language model
• Abstract: The increasing demand for intelligent systems capable of interpreting and reasoning about visual content requires the development of Large Multi-Modal Models (LMMs) that are not only accurate but also have explicit reasoning capabilities. This paper presents a novel approach to imbue an LMM with the ability to conduct explicit reasoning based on visual content and textual instructions. We introduce a system that can ask a question to acquire necessary knowledge, thereby enhancing the robustness and explicability of the reasoning process. Our method comprises the development of a novel dataset generated by a Large Language Model (LLM), designed to promote chain-of-thought reasoning combined with a question-asking mechanism. We designed an LMM, which has high capabilities on region awareness to address the intricate requirements of image-text alignment. The model undergoes a three-stage training phase, starting with large-scale image-text alignment using a large-scale datasets, followed by instruction tuning, and fine-tuning with a focus on chain-of-thought reasoning. The results demonstrate a stride toward a more robust, accurate, and interpretable LMM, capable of reasoning explicitly and seeking information proactively when confronted with ambiguous visual input.

Title: Attack tree metrics are operad algebras

• Authors: Milan Lopuhaä-Zwakenberg
• Subjects: cs.CR, math.CT
• Abstract URL: https://arxiv.org/abs/2401.10008
• Pdf URL: https://arxiv.org/pdf/2401.10008
• Copy Paste: [[2401.10008]] Attack tree metrics are operad algebras(https://arxiv.org/abs/2401.10008)
• Keywords: security, attack
• Abstract: Attack Trees (ATs) are a widely used tool for security analysis. ATs can be employed in quantitative security analysis through metrics, which assign a security value to an AT. Many different AT metrics exist, and there exist multiple general definitions that aim to study a wide variety of AT metrics at once. However, these all have drawbacks: they do not capture all metrics, and they do not easily generalize to extensions of ATs. In this paper, we introduce a definition of AT metrics based on category theory, specifically operad algebras. This encompasses all previous definitions of AT metrics, and is easily generalized to extensions of ATs. Furthermore, we show that under easily expressed operad-theoretic conditions, existing metric calculation algorithms can be extended in considerable generality.

Title: Towards Hierarchical Spoken Language Dysfluency Modeling

• Authors: Jiachen Lian, Gopala Anumanchipalli
• Subjects: cs.CL, eess.AS
• Abstract URL: https://arxiv.org/abs/2401.10015
• Pdf URL: https://arxiv.org/pdf/2401.10015
• Copy Paste: [[2401.10015]] Towards Hierarchical Spoken Language Dysfluency Modeling(https://arxiv.org/abs/2401.10015)
• Keywords: robust
• Abstract: Speech dysfluency modeling is the bottleneck for both speech therapy and language learning. However, there is no AI solution to systematically tackle this problem. We first propose to define the concept of dysfluent speech and dysfluent speech modeling. We then present Hierarchical Unconstrained Dysfluency Modeling (H-UDM) approach that addresses both dysfluency transcription and detection to eliminate the need for extensive manual annotation. Furthermore, we introduce a simulated dysfluent dataset called VCTK++ to enhance the capabilities of H-UDM in phonetic transcription. Our experimental results demonstrate the effectiveness and robustness of our proposed methods in both transcription and detection tasks.

Title: Gender Bias in Machine Translation and The Era of Large Language Models

• Authors: Eva Vanmassenhove
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2401.10016
• Pdf URL: https://arxiv.org/pdf/2401.10016
• Copy Paste: [[2401.10016]] Gender Bias in Machine Translation and The Era of Large Language Models(https://arxiv.org/abs/2401.10016)
• Keywords: fair, transformer, generative, large language model
• Abstract: This chapter examines the role of Machine Translation in perpetuating gender bias, highlighting the challenges posed by cross-linguistic settings and statistical dependencies. A comprehensive overview of relevant existing work related to gender bias in both conventional Neural Machine Translation approaches and Generative Pretrained Transformer models employed as Machine Translation systems is provided. Through an experiment using ChatGPT (based on GPT-3.5) in an English-Italian translation context, we further assess ChatGPT's current capacity to address gender bias. The findings emphasize the ongoing need for advancements in mitigating bias in Machine Translation systems and underscore the importance of fostering fairness and inclusivity in language technologies.

Title: Text Region Multiple Information Perception Network for Scene Text Detection

• Authors: Jinzhi Zheng, Libo Zhang, Yanjun Wu, Chen Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.10017
• Pdf URL: https://arxiv.org/pdf/2401.10017
• Copy Paste: [[2401.10017]] Text Region Multiple Information Perception Network for Scene Text Detection(https://arxiv.org/abs/2401.10017)
• Keywords: robust, segmentation
• Abstract: Segmentation-based scene text detection algorithms can handle arbitrary shape scene texts and have strong robustness and adaptability, so it has attracted wide attention. Existing segmentation-based scene text detection algorithms usually only segment the pixels in the center region of the text, while ignoring other information of the text region, such as edge information, distance information, etc., thus limiting the detection accuracy of the algorithm for scene text. This paper proposes a plug-and-play module called the Region Multiple Information Perception Module (RMIPM) to enhance the detection performance of segmentation-based algorithms. Specifically, we design an improved module that can perceive various types of information about scene text regions, such as text foreground classification maps, distance maps, direction maps, etc. Experiments on MSRA-TD500 and TotalText datasets show that our method achieves comparable performance with current state-of-the-art algorithms.

Title: R-Judge: Benchmarking Safety Risk Awareness for LLM Agents

• Authors: Tongxin Yuan, Zhiwei He, Lingzhong Dong, Yiming Wang, Ruijie Zhao, Tian Xia, Lizhen Xu, Binglin Zhou, Fangqi Li, Zhuosheng Zhang, Rui Wang, Gongshen Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.10019
• Pdf URL: https://arxiv.org/pdf/2401.10019
• Copy Paste: [[2401.10019]] R-Judge: Benchmarking Safety Risk Awareness for LLM Agents(https://arxiv.org/abs/2401.10019)
• Keywords: large language model
• Abstract: Large language models (LLMs) have exhibited great potential in autonomously completing tasks across real-world applications. Despite this, these LLM agents introduce unexpected safety risks when operating in interactive environments. Instead of centering on LLM-generated content safety in most prior studies, this work addresses the imperative need for benchmarking the behavioral safety of LLM agents within diverse environments. We introduce R-Judge, a benchmark crafted to evaluate the proficiency of LLMs in judging safety risks given agent interaction records. R-Judge comprises 162 agent interaction records, encompassing 27 key risk scenarios among 7 application categories and 10 risk types. It incorporates human consensus on safety with annotated safety risk labels and high-quality risk descriptions. Utilizing R-Judge, we conduct a comprehensive evaluation of 8 prominent LLMs commonly employed as the backbone for agents. The best-performing model, GPT-4, achieves 72.29% in contrast to the human score of 89.38%, showing considerable room for enhancing the risk awareness of LLMs. Notably, leveraging risk descriptions as environment feedback significantly improves model performance, revealing the importance of salient safety risk feedback. Furthermore, we design an effective chain of safety analysis technique to help the judgment of safety risks and conduct an in-depth case study to facilitate future research. R-Judge is publicly available at https://github.com/Lordog/R-Judge.

Title: Framing Analysis of Health-Related Narratives: Conspiracy versus Mainstream Media

• Authors: Markus Reiter-Haas, Beate Klösch, Markus Hadler, Elisabeth Lex
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2401.10030
• Pdf URL: https://arxiv.org/pdf/2401.10030
• Copy Paste: [[2401.10030]] Framing Analysis of Health-Related Narratives: Conspiracy versus Mainstream Media(https://arxiv.org/abs/2401.10030)
• Keywords: extraction
• Abstract: Understanding how online media frame issues is crucial due to their impact on public opinion. Research on framing using natural language processing techniques mainly focuses on specific content features in messages and neglects their narrative elements. Also, the distinction between framing in different sources remains an understudied problem. We address those issues and investigate how the framing of health-related topics, such as COVID-19 and other diseases, differs between conspiracy and mainstream websites. We incorporate narrative information into the framing analysis by introducing a novel frame extraction approach based on semantic graphs. We find that health-related narratives in conspiracy media are predominantly framed in terms of beliefs, while mainstream media tend to present them in terms of science. We hope our work offers new ways for a more nuanced frame analysis.

Title: LOCALINTEL: Generating Organizational Threat Intelligence from Global and Local Cyber Knowledge

• Authors: Shaswata Mitra, Subash Neupane, Trisha Chakraborty, Sudip Mittal, Aritran Piplai, Manas Gaur, Shahram Rahimi
• Subjects: cs.CR, cs.AI, cs.IR, cs.LO
• Abstract URL: https://arxiv.org/abs/2401.10036
• Pdf URL: https://arxiv.org/pdf/2401.10036
• Copy Paste: [[2401.10036]] LOCALINTEL: Generating Organizational Threat Intelligence from Global and Local Cyber Knowledge(https://arxiv.org/abs/2401.10036)
• Keywords: security, large language model
• Abstract: Security Operations Center (SoC) analysts gather threat reports from openly accessible global threat databases and customize them manually to suit a particular organization's needs. These analysts also depend on internal repositories, which act as private local knowledge database for an organization. Credible cyber intelligence, critical operational details, and relevant organizational information are all stored in these local knowledge databases. Analysts undertake a labor intensive task utilizing these global and local knowledge databases to manually create organization's unique threat response and mitigation strategies. Recently, Large Language Models (LLMs) have shown the capability to efficiently process large diverse knowledge sources. We leverage this ability to process global and local knowledge databases to automate the generation of organization-specific threat intelligence. In this work, we present LOCALINTEL, a novel automated knowledge contextualization system that, upon prompting, retrieves threat reports from the global threat repositories and uses its local knowledge database to contextualize them for a specific organization. LOCALINTEL comprises of three key phases: global threat intelligence retrieval, local knowledge retrieval, and contextualized completion generation. The former retrieves intelligence from global threat repositories, while the second retrieves pertinent knowledge from the local knowledge database. Finally, the fusion of these knowledge sources is orchestrated through a generator to produce a contextualized completion.

Title: Depth Over RGB: Automatic Evaluation of Open Surgery Skills Using Depth Camera

• Authors: Ido Zuckerman, Nicole Werner, Jonathan Kouchly, Emma Huston, Shannon DiMarco, Paul DiMusto, Shlomi Laufer
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.10037
• Pdf URL: https://arxiv.org/pdf/2401.10037
• Copy Paste: [[2401.10037]] Depth Over RGB: Automatic Evaluation of Open Surgery Skills Using Depth Camera(https://arxiv.org/abs/2401.10037)
• Keywords: privacy, robust, segmentation
• Abstract: Purpose: In this paper, we present a novel approach to the automatic evaluation of open surgery skills using depth cameras. This work is intended to show that depth cameras achieve similar results to RGB cameras, which is the common method in the automatic evaluation of open surgery skills. Moreover, depth cameras offer advantages such as robustness to lighting variations, camera positioning, simplified data compression, and enhanced privacy, making them a promising alternative to RGB cameras. Methods: Experts and novice surgeons completed two simulators of open suturing. We focused on hand and tool detection, and action segmentation in suturing procedures. YOLOv8 was used for tool detection in RGB and depth videos. Furthermore, UVAST and MSTCN++ were used for action segmentation. Our study includes the collection and annotation of a dataset recorded with Azure Kinect. Results: We demonstrated that using depth cameras in object detection and action segmentation achieves comparable results to RGB cameras. Furthermore, we analyzed 3D hand path length, revealing significant differences between experts and novice surgeons, emphasizing the potential of depth cameras in capturing surgical skills. We also investigated the influence of camera angles on measurement accuracy, highlighting the advantages of 3D cameras in providing a more accurate representation of hand movements. Conclusion: Our research contributes to advancing the field of surgical skill assessment by leveraging depth cameras for more reliable and privacy evaluations. The findings suggest that depth cameras can be valuable in assessing surgical skills and provide a foundation for future research in this area.

Title: Large Language Models for Scientific Information Extraction: An Empirical Study for Virology

• Authors: Mahsa Shamsabadi, Jennifer D'Souza, Sören Auer
• Subjects: cs.CL, cs.AI, cs.DL, cs.IT
• Abstract URL: https://arxiv.org/abs/2401.10040
• Pdf URL: https://arxiv.org/pdf/2401.10040
• Copy Paste: [[2401.10040]] Large Language Models for Scientific Information Extraction: An Empirical Study for Virology(https://arxiv.org/abs/2401.10040)
• Keywords: robust, extraction, large language model
• Abstract: In this paper, we champion the use of structured and semantic content representation of discourse-based scholarly communication, inspired by tools like Wikipedia infoboxes or structured Amazon product descriptions. These representations provide users with a concise overview, aiding scientists in navigating the dense academic landscape. Our novel automated approach leverages the robust text generation capabilities of LLMs to produce structured scholarly contribution summaries, offering both a practical solution and insights into LLMs' emergent abilities. For LLMs, the prime focus is on improving their general intelligence as conversational agents. We argue that these models can also be applied effectively in information extraction (IE), specifically in complex IE tasks within terse domains like Science. This paradigm shift replaces the traditional modular, pipelined machine learning approach with a simpler objective expressed through instructions. Our results show that finetuned FLAN-T5 with 1000x fewer parameters than the state-of-the-art GPT-davinci is competitive for the task.

Title: ContextMix: A context-aware data augmentation method for industrial visual inspection systems

• Authors: Hyungmin Kim, Donghun Kim, Pyunghwan Ahn, Sungho Suh, Hansang Cho, Junmo Kim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.10050
• Pdf URL: https://arxiv.org/pdf/2401.10050
• Copy Paste: [[2401.10050]] ContextMix: A context-aware data augmentation method for industrial visual inspection systems(https://arxiv.org/abs/2401.10050)
• Keywords: robust, segmentation
• Abstract: While deep neural networks have achieved remarkable performance, data augmentation has emerged as a crucial strategy to mitigate overfitting and enhance network performance. These techniques hold particular significance in industrial manufacturing contexts. Recently, image mixing-based methods have been introduced, exhibiting improved performance on public benchmark datasets. However, their application to industrial tasks remains challenging. The manufacturing environment generates massive amounts of unlabeled data on a daily basis, with only a few instances of abnormal data occurrences. This leads to severe data imbalance. Thus, creating well-balanced datasets is not straightforward due to the high costs associated with labeling. Nonetheless, this is a crucial step for enhancing productivity. For this reason, we introduce ContextMix, a method tailored for industrial applications and benchmark datasets. ContextMix generates novel data by resizing entire images and integrating them into other images within the batch. This approach enables our method to learn discriminative features based on varying sizes from resized images and train informative secondary features for object recognition using occluded images. With the minimal additional computation cost of image resizing, ContextMix enhances performance compared to existing augmentation techniques. We evaluate its effectiveness across classification, detection, and segmentation tasks using various network architectures on public benchmark datasets. Our proposed method demonstrates improved results across a range of robustness tasks. Its efficacy in real industrial environments is particularly noteworthy, as demonstrated using the passive component dataset.

Title: DiffusionGPT: LLM-Driven Text-to-Image Generation System

• Authors: Jie Qin, Jie Wu, Weifeng Chen, Yuxi Ren, Huixia Li, Hefeng Wu, Xuefeng Xiao, Rui Wang, Shilei Wen
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.10061
• Pdf URL: https://arxiv.org/pdf/2401.10061
• Copy Paste: [[2401.10061]] DiffusionGPT: LLM-Driven Text-to-Image Generation System(https://arxiv.org/abs/2401.10061)
• Keywords: diffusion, generative, large language model
• Abstract: Diffusion models have opened up new avenues for the field of image generation, resulting in the proliferation of high-quality models shared on open-source platforms. However, a major challenge persists in current text-to-image systems are often unable to handle diverse inputs, or are limited to single model results. Current unified attempts often fall into two orthogonal aspects: i) parse Diverse Prompts in input stage; ii) activate expert model to output. To combine the best of both worlds, we propose DiffusionGPT, which leverages Large Language Models (LLM) to offer a unified generation system capable of seamlessly accommodating various types of prompts and integrating domain-expert models. DiffusionGPT constructs domain-specific Trees for various generative models based on prior knowledge. When provided with an input, the LLM parses the prompt and employs the Trees-of-Thought to guide the selection of an appropriate model, thereby relaxing input constraints and ensuring exceptional performance across diverse domains. Moreover, we introduce Advantage Databases, where the Tree-of-Thought is enriched with human feedback, aligning the model selection process with human preferences. Through extensive experiments and comparisons, we demonstrate the effectiveness of DiffusionGPT, showcasing its potential for pushing the boundaries of image synthesis in diverse domains.

Title: Code Prompting Elicits Conditional Reasoning Abilities in Text+Code LLMs

• Authors: Haritz Puerto, Martin Tutek, Somak Aditya, Xiaodan Zhu, Iryna Gurevych
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.10065
• Pdf URL: https://arxiv.org/pdf/2401.10065
• Copy Paste: [[2401.10065]] Code Prompting Elicits Conditional Reasoning Abilities in Text+Code LLMs(https://arxiv.org/abs/2401.10065)
• Keywords: large language model
• Abstract: Reasoning is a fundamental component for achieving language understanding. Among the multiple types of reasoning, conditional reasoning, the ability to draw different conclusions depending on some condition, has been understudied in large language models (LLMs). Recent prompting methods, such as chain of thought, have significantly improved LLMs on reasoning tasks. Nevertheless, there is still little understanding of what triggers reasoning abilities in LLMs. We hypothesize that code prompts can trigger conditional reasoning in LLMs trained on text and code. We propose a chain of prompts that transforms a natural language problem into code and prompts the LLM with the generated code. Our experiments find that code prompts exhibit a performance boost between 2.6 and 7.7 points on GPT 3.5 across multiple datasets requiring conditional reasoning. We then conduct experiments to discover how code prompts elicit conditional reasoning abilities and through which features. We observe that prompts need to contain natural language text accompanied by high-quality code that closely represents the semantics of the instance text. Furthermore, we show that code prompts are more efficient, requiring fewer demonstrations, and that they trigger superior state tracking of variables or key entities.

Title: Communication-Efficient Personalized Federated Learning for Speech-to-Text Tasks

• Authors: Yichao Du, Zhirui Zhang, Linan Yue, Xu Huang, Yuqing Zhang, Tong Xu, Linli Xu, Enhong Chen
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2401.10070
• Pdf URL: https://arxiv.org/pdf/2401.10070
• Copy Paste: [[2401.10070]] Communication-Efficient Personalized Federated Learning for Speech-to-Text Tasks(https://arxiv.org/abs/2401.10070)
• Keywords: privacy, protect, federate
• Abstract: To protect privacy and meet legal regulations, federated learning (FL) has gained significant attention for training speech-to-text (S2T) systems, including automatic speech recognition (ASR) and speech translation (ST). However, the commonly used FL approach (i.e., \textsc{FedAvg}) in S2T tasks typically suffers from extensive communication overhead due to multi-round interactions based on the whole model and performance degradation caused by data heterogeneity among clients.To address these issues, we propose a personalized federated S2T framework that introduces \textsc{FedLoRA}, a lightweight LoRA module for client-side tuning and interaction with the server to minimize communication overhead, and \textsc{FedMem}, a global model equipped with a $k$-nearest-neighbor ($k$NN) classifier that captures client-specific distributional shifts to achieve personalization and overcome data heterogeneity. Extensive experiments based on Conformer and Whisper backbone models on CoVoST and GigaSpeech benchmarks show that our approach significantly reduces the communication overhead on all S2T tasks and effectively personalizes the global model to overcome data heterogeneity.

Title: A locally statistical active contour model for SAR image segmentation can be solved by denoising algorithms

• Authors: Guangming Liu, Quanying Sun, Jing Liang, Qi Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.10083
• Pdf URL: https://arxiv.org/pdf/2401.10083
• Copy Paste: [[2401.10083]] A locally statistical active contour model for SAR image segmentation can be solved by denoising algorithms(https://arxiv.org/abs/2401.10083)
• Keywords: diffusion, segmentation
• Abstract: In this paper, we propose a novel locally statistical variational active contour model based on I-divergence-TV denoising model, which hybrides geodesic active contour (GAC) model with active contours without edges (ACWE) model, and can be used to segment images corrupted by multiplicative gamma noise. By adding a diffusion term into the level set evolution (LSE) equation of the proposed model, we construct a reaction-diffusion (RD) equation, which can gradually regularize the level set function (LSF) to be piecewise constant in each segment domain and gain the stable solution. We further transform the proposed model into classic ROF model by adding a proximity term. Inspired by a fast denoising algorithm proposed by Jia-Zhao recently, we propose two fast fixed point algorithms to solve SAR image segmentation question. Experimental results for real SAR images show that the proposed image segmentation model can efficiently stop the contours at weak or blurred edges, and can automatically detect the exterior and interior boundaries of images with multiplicative gamma noise. The proposed FPRD1/FPRD2 models are about 1/2 (or less than) of the time required for the SBRD model based on the Split Bregman technique.

Title: Cross-Modality Perturbation Synergy Attack for Person Re-identification

• Authors: Yunpeng Gong, others
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.10090
• Pdf URL: https://arxiv.org/pdf/2401.10090
• Copy Paste: [[2401.10090]] Cross-Modality Perturbation Synergy Attack for Person Re-identification(https://arxiv.org/abs/2401.10090)
• Keywords: security, attack, robust
• Abstract: In recent years, there has been significant research focusing on addressing security concerns in single-modal person re-identification (ReID) systems that are based on RGB images. However, the safety of cross-modality scenarios, which are more commonly encountered in practical applications involving images captured by infrared cameras, has not received adequate attention. The main challenge in cross-modality ReID lies in effectively dealing with visual differences between different modalities. For instance, infrared images are typically grayscale, unlike visible images that contain color information. Existing attack methods have primarily focused on the characteristics of the visible image modality, overlooking the features of other modalities and the variations in data distribution among different modalities. This oversight can potentially undermine the effectiveness of these methods in image retrieval across diverse modalities. This study represents the first exploration into the security of cross-modality ReID models and proposes a universal perturbation attack specifically designed for cross-modality ReID. This attack optimizes perturbations by leveraging gradients from diverse modality data, thereby disrupting the discriminator and reinforcing the differences between modalities. We conducted experiments on two widely used cross-modality datasets, namely RegDB and SYSU, which not only demonstrated the effectiveness of our method but also provided insights for future enhancements in the robustness of cross-modality ReID systems.

Title: Power in Numbers: Robust reading comprehension by finetuning with four adversarial sentences per example

• Authors: Ariel Marcus
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.10091
• Pdf URL: https://arxiv.org/pdf/2401.10091
• Copy Paste: [[2401.10091]] Power in Numbers: Robust reading comprehension by finetuning with four adversarial sentences per example(https://arxiv.org/abs/2401.10091)
• Keywords: attack, robust
• Abstract: Recent models have achieved human level performance on the Stanford Question Answering Dataset when using F1 scores to evaluate the reading comprehension task. Yet, teaching machines to comprehend text has not been solved in the general case. By appending one adversarial sentence to the context paragraph, past research has shown that the F1 scores from reading comprehension models drop almost in half. In this paper, I replicate past adversarial research with a new model, ELECTRA-Small, and demonstrate that the new model's F1 score drops from 83.9% to 29.2%. To improve ELECTRA-Small's resistance to this attack, I finetune the model on SQuAD v1.1 training examples with one to five adversarial sentences appended to the context paragraph. Like past research, I find that the finetuned model on one adversarial sentence does not generalize well across evaluation datasets. However, when finetuned on four or five adversarial sentences the model attains an F1 score of more than 70% on most evaluation datasets with multiple appended and prepended adversarial sentences. The results suggest that with enough examples we can make models robust to adversarial attacks.

Title: Marrying Adapters and Mixup to Efficiently Enhance the Adversarial Robustness of Pre-Trained Language Models for Text Classification

• Authors: Tuc Nguyen, Thai Le
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.10111
• Pdf URL: https://arxiv.org/pdf/2401.10111
• Copy Paste: [[2401.10111]] Marrying Adapters and Mixup to Efficiently Enhance the Adversarial Robustness of Pre-Trained Language Models for Text Classification(https://arxiv.org/abs/2401.10111)
• Keywords: attack, robust
• Abstract: Existing works show that augmenting training data of neural networks using both clean and adversarial examples can enhance their generalizability under adversarial attacks. However, this training approach often leads to performance degradation on clean inputs. Additionally, it requires frequent re-training of the entire model to account for new attack types, resulting in significant and costly computations. Such limitations make adversarial training mechanisms less practical, particularly for complex Pre-trained Language Models (PLMs) with millions or even billions of parameters. To overcome these challenges while still harnessing the theoretical benefits of adversarial training, this study combines two concepts: (1) adapters, which enable parameter-efficient fine-tuning, and (2) Mixup, which train NNs via convex combinations of pairs data pairs. Intuitively, we propose to fine-tune PLMs through convex combinations of non-data pairs of fine-tuned adapters, one trained with clean and another trained with adversarial examples. Our experiments show that the proposed method achieves the best trade-off between training efficiency and predictive performance, both with and without attacks compared to other baselines on a variety of downstream tasks.

Title: Towards Principled Graph Transformers

• Authors: Luis Müller, Christopher Morris
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.10119
• Pdf URL: https://arxiv.org/pdf/2401.10119
• Copy Paste: [[2401.10119]] Towards Principled Graph Transformers(https://arxiv.org/abs/2401.10119)
• Keywords: transformer
• Abstract: Graph learning architectures based on the k-dimensional Weisfeiler-Leman (k-WL) hierarchy offer a theoretically well-understood expressive power. However, such architectures often fail to deliver solid predictive performance on real-world tasks, limiting their practical impact. In contrast, global attention-based models such as graph transformers demonstrate strong performance in practice, but comparing their expressive power with the k-WL hierarchy remains challenging, particularly since these architectures rely on positional or structural encodings for their expressivity and predictive performance. To address this, we show that the recently proposed Edge Transformer, a global attention model operating on node pairs instead of nodes, has at least 3-WL expressive power. Empirically, we demonstrate that the Edge Transformer surpasses other theoretically aligned architectures regarding predictive performance while not relying on positional or structural encodings.

Title: Spatial-Temporal Large Language Model for Traffic Prediction

• Authors: Chenxi Liu, Sun Yang, Qianxiong Xu, Zhishuai Li, Cheng Long, Ziyue Li, Rui Zhao
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2401.10134
• Pdf URL: https://arxiv.org/pdf/2401.10134
• Copy Paste: [[2401.10134]] Spatial-Temporal Large Language Model for Traffic Prediction(https://arxiv.org/abs/2401.10134)
• Keywords: robust, large language model
• Abstract: Traffic prediction, a critical component for intelligent transportation systems, endeavors to foresee future traffic at specific locations using historical data. Although existing traffic prediction models often emphasize developing complex neural network structures, their accuracy has not seen improvements accordingly. Recently, Large Language Models (LLMs) have shown outstanding capabilities in time series analysis. Differing from existing models, LLMs progress mainly through parameter expansion and extensive pre-training while maintaining their fundamental structures. In this paper, we propose a Spatial-Temporal Large Language Model (ST-LLM) for traffic prediction. Specifically, ST-LLM redefines the timesteps at each location as tokens and incorporates a spatial-temporal embedding module to learn the spatial location and global temporal representations of tokens. Then these representations are fused to provide each token with unified spatial and temporal information. Furthermore, we propose a novel partially frozen attention strategy of the LLM, which is designed to capture spatial-temporal dependencies for traffic prediction. Comprehensive experiments on real traffic datasets offer evidence that ST-LLM outperforms state-of-the-art models. Notably, the ST-LLM also exhibits robust performance in both few-shot and zero-shot prediction scenarios.

Title: Model Compression Techniques in Biometrics Applications: A Survey

• Authors: Eduarda Caldeira, Pedro C. Neto, Marco Huber, Naser Damer, Ana F. Sequeira
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.10139
• Pdf URL: https://arxiv.org/pdf/2401.10139
• Copy Paste: [[2401.10139]] Model Compression Techniques in Biometrics Applications: A Survey(https://arxiv.org/abs/2401.10139)
• Keywords: biometric, fair
• Abstract: The development of deep learning algorithms has extensively empowered humanity's task automatization capacity. However, the huge improvement in the performance of these models is highly correlated with their increasing level of complexity, limiting their usefulness in human-oriented applications, which are usually deployed in resource-constrained devices. This led to the development of compression techniques that drastically reduce the computational and memory costs of deep learning models without significant performance degradation. This paper aims to systematize the current literature on this topic by presenting a comprehensive survey of model compression techniques in biometrics applications, namely quantization, knowledge distillation and pruning. We conduct a critical analysis of the comparative value of these techniques, focusing on their advantages and disadvantages and presenting suggestions for future work directions that can potentially improve the current methods. Additionally, we discuss and analyze the link between model bias and model compression, highlighting the need to direct compression research toward model fairness in future works.

Title: Explicitly Disentangled Representations in Object-Centric Learning

• Authors: Riccardo Majellaro, Jonathan Collu, Aske Plaat, Thomas M. Moerland
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.10148
• Pdf URL: https://arxiv.org/pdf/2401.10148
• Copy Paste: [[2401.10148]] Explicitly Disentangled Representations in Object-Centric Learning(https://arxiv.org/abs/2401.10148)
• Keywords: robust
• Abstract: Extracting structured representations from raw visual data is an important and long-standing challenge in machine learning. Recently, techniques for unsupervised learning of object-centric representations have raised growing interest. In this context, enhancing the robustness of the latent features can improve the efficiency and effectiveness of the training of downstream tasks. A promising step in this direction is to disentangle the factors that cause variation in the data. Previously, Invariant Slot Attention disentangled position, scale, and orientation from the remaining features. Extending this approach, we focus on separating the shape and texture components. In particular, we propose a novel architecture that biases object-centric models toward disentangling shape and texture components into two non-overlapping subsets of the latent space dimensions. These subsets are known a priori, hence before the training process. Experiments on a range of object-centric benchmarks reveal that our approach achieves the desired disentanglement while also numerically improving baseline performance in most cases. In addition, we show that our method can generate novel textures for a specific object or transfer textures between objects with distinct shapes.

Title: Multi-Agent Reinforcement Learning for Maritime Operational Technology Cyber Security

• Authors: Alec Wilson, Ryan Menzies, Neela Morarji, David Foster, Marco Casassa Mont, Esin Turkbeyler, Lisa Gralewski
• Subjects: cs.LG, cs.CR, cs.MA
• Abstract URL: https://arxiv.org/abs/2401.10149
• Pdf URL: https://arxiv.org/pdf/2401.10149
• Copy Paste: [[2401.10149]] Multi-Agent Reinforcement Learning for Maritime Operational Technology Cyber Security(https://arxiv.org/abs/2401.10149)
• Keywords: security, attack
• Abstract: This paper demonstrates the potential for autonomous cyber defence to be applied on industrial control systems and provides a baseline environment to further explore Multi-Agent Reinforcement Learning's (MARL) application to this problem domain. It introduces a simulation environment, IPMSRL, of a generic Integrated Platform Management System (IPMS) and explores the use of MARL for autonomous cyber defence decision-making on generic maritime based IPMS Operational Technology (OT). OT cyber defensive actions are less mature than they are for Enterprise IT. This is due to the relatively brittle nature of OT infrastructure originating from the use of legacy systems, design-time engineering assumptions, and lack of full-scale modern security controls. There are many obstacles to be tackled across the cyber landscape due to continually increasing cyber-attack sophistication and the limitations of traditional IT-centric cyber defence solutions. Traditional IT controls are rarely deployed on OT infrastructure, and where they are, some threats aren't fully addressed. In our experiments, a shared critic implementation of Multi Agent Proximal Policy Optimisation (MAPPO) outperformed Independent Proximal Policy Optimisation (IPPO). MAPPO reached an optimal policy (episode outcome mean of 1) after 800K timesteps, whereas IPPO was only able to reach an episode outcome mean of 0.966 after one million timesteps. Hyperparameter tuning greatly improved training performance. Across one million timesteps the tuned hyperparameters reached an optimal policy whereas the default hyperparameters only managed to win sporadically, with most simulations resulting in a draw. We tested a real-world constraint, attack detection alert success, and found that when alert success probability is reduced to 0.75 or 0.9, the MARL defenders were still able to win in over 97.5% or 99.5% of episodes, respectively.

Title: Motion-Zero: Zero-Shot Moving Object Control Framework for Diffusion-Based Video Generation

• Authors: Changgu Chen, Junwei Shu, Lianggangxu Chen, Gaoqi He, Changbo Wang, Yang Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.10150
• Pdf URL: https://arxiv.org/pdf/2401.10150
• Copy Paste: [[2401.10150]] Motion-Zero: Zero-Shot Moving Object Control Framework for Diffusion-Based Video Generation(https://arxiv.org/abs/2401.10150)
• Keywords: diffusion, generative
• Abstract: Recent large-scale pre-trained diffusion models have demonstrated a powerful generative ability to produce high-quality videos from detailed text descriptions. However, exerting control over the motion of objects in videos generated by any video diffusion model is a challenging problem. In this paper, we propose a novel zero-shot moving object trajectory control framework, Motion-Zero, to enable a bounding-box-trajectories-controlled text-to-video diffusion model.To this end, an initial noise prior module is designed to provide a position-based prior to improve the stability of the appearance of the moving object and the accuracy of position. In addition, based on the attention map of the U-net, spatial constraints are directly applied to the denoising process of diffusion models, which further ensures the positional and spatial consistency of moving objects during the inference. Furthermore, temporal consistency is guaranteed with a proposed shift temporal attention mechanism. Our method can be flexibly applied to various state-of-the-art video diffusion models without any training process. Extensive experiments demonstrate our proposed method can control the motion trajectories of objects and generate high-quality videos.

Title: VMamba: Visual State Space Model

• Authors: Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, Yunfan Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.10166
• Pdf URL: https://arxiv.org/pdf/2401.10166
• Copy Paste: [[2401.10166]] VMamba: Visual State Space Model(https://arxiv.org/abs/2401.10166)
• Keywords: transformer
• Abstract: Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) stand as the two most popular foundation models for visual representation learning. While CNNs exhibit remarkable scalability with linear complexity w.r.t. image resolution, ViTs surpass them in fitting capabilities despite contending with quadratic complexity. A closer inspection reveals that ViTs achieve superior visual modeling performance through the incorporation of global receptive fields and dynamic weights. This observation motivates us to propose a novel architecture that inherits these components while enhancing computational efficiency. To this end, we draw inspiration from the recently introduced state space model and propose the Visual State Space Model (VMamba), which achieves linear complexity without sacrificing global receptive fields. To address the encountered direction-sensitive issue, we introduce the Cross-Scan Module (CSM) to traverse the spatial domain and convert any non-causal visual image into order patch sequences. Extensive experimental results substantiate that VMamba not only demonstrates promising capabilities across various visual perception tasks, but also exhibits more pronounced advantages over established benchmarks as the image resolution increases. Source code has been available at https://github.com/MzeroMiko/VMamba.

Title: SHINOBI: Shape and Illumination using Neural Object Decomposition via BRDF Optimization In-the-wild

• Authors: Andreas Engelhardt, Amit Raj, Mark Boss, Yunzhi Zhang, Abhishek Kar, Yuanzhen Li, Deqing Sun, Ricardo Martin Brualla, Jonathan T. Barron, Hendrik P. A. Lensch, Varun Jampani
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2401.10171
• Pdf URL: https://arxiv.org/pdf/2401.10171
• Copy Paste: [[2401.10171]] SHINOBI: Shape and Illumination using Neural Object Decomposition via BRDF Optimization In-the-wild(https://arxiv.org/abs/2401.10171)
• Keywords: robust
• Abstract: We present SHINOBI, an end-to-end framework for the reconstruction of shape, material, and illumination from object images captured with varying lighting, pose, and background. Inverse rendering of an object based on unconstrained image collections is a long-standing challenge in computer vision and graphics and requires a joint optimization over shape, radiance, and pose. We show that an implicit shape representation based on a multi-resolution hash encoding enables faster and robust shape reconstruction with joint camera alignment optimization that outperforms prior work. Further, to enable the editing of illumination and object reflectance (i.e. material) we jointly optimize BRDF and illumination together with the object's shape. Our method is class-agnostic and works on in-the-wild image collections of objects to produce relightable 3D assets for several use cases such as AR/VR, movies, games, etc. Project page: https://shinobi.aengelhardt.com Video: https://www.youtube.com/watch?v=iFENQ6AcYd8&feature=youtu.be

Title: Beyond Reference-Based Metrics: Analyzing Behaviors of Open LLMs on Data-to-Text Generation

• Authors: Zdeněk Kasner, Ondřej Dušek
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.10186
• Pdf URL: https://arxiv.org/pdf/2401.10186
• Copy Paste: [[2401.10186]] Beyond Reference-Based Metrics: Analyzing Behaviors of Open LLMs on Data-to-Text Generation(https://arxiv.org/abs/2401.10186)
• Keywords: large language model
• Abstract: We investigate to which extent open large language models (LLMs) can generate coherent and relevant text from structured data. To prevent bias from benchmarks leaked into LLM training data, we collect Quintd-1: an ad-hoc benchmark for five data-to-text (D2T) generation tasks, consisting of structured data records in standard formats gathered from public APIs. We leverage reference-free evaluation metrics and LLMs' in-context learning capabilities, allowing us to test the models with no human-written references. Our evaluation focuses on annotating semantic accuracy errors on token-level, combining human annotators and a metric based on GPT-4. Our systematic examination of the models' behavior across domains and tasks suggests that state-of-the-art open LLMs with 7B parameters can generate fluent and coherent text from various standard data formats in zero-shot settings. However, we also show that semantic accuracy of the outputs remains a major issue: on our benchmark, 80% of outputs of open LLMs contain a semantic error according to human annotators (91% according to GPT-4). Our code, data, and model outputs are available at https://d2t-llm.github.io.

Title: Chem-FINESE: Validating Fine-Grained Few-shot Entity Extraction through Text Reconstruction

• Authors: Qingyun Wang, Zixuan Zhang, Hongxiang Li, Xuan Liu, Jiawei Han, Heng Ji, Huimin Zhao
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.10189
• Pdf URL: https://arxiv.org/pdf/2401.10189
• Copy Paste: [[2401.10189]] Chem-FINESE: Validating Fine-Grained Few-shot Entity Extraction through Text Reconstruction(https://arxiv.org/abs/2401.10189)
• Keywords: extraction
• Abstract: Fine-grained few-shot entity extraction in the chemical domain faces two unique challenges. First, compared with entity extraction tasks in the general domain, sentences from chemical papers usually contain more entities. Moreover, entity extraction models usually have difficulty extracting entities of long-tailed types. In this paper, we propose Chem-FINESE, a novel sequence-to-sequence (seq2seq) based few-shot entity extraction approach, to address these two challenges. Our Chem-FINESE has two components: a seq2seq entity extractor to extract named entities from the input sentence and a seq2seq self-validation module to reconstruct the original input sentence from extracted entities. Inspired by the fact that a good entity extraction system needs to extract entities faithfully, our new self-validation module leverages entity extraction results to reconstruct the original input sentence. Besides, we design a new contrastive loss to reduce excessive copying during the extraction process. Finally, we release ChemNER+, a new fine-grained chemical entity extraction dataset that is annotated by domain experts with the ChemNER schema. Experiments in few-shot settings with both ChemNER+ and CHEMET datasets show that our newly proposed framework has contributed up to 8.26% and 6.84% absolute F1-score gains respectively.

Title: Eclectic Rule Extraction for Explainability of Deep Neural Network based Intrusion Detection Systems

• Authors: Jesse Ables, Nathaniel Childers, William Anderson, Sudip Mittal, Shahram Rahimi, Ioana Banicescu, Maria Seale
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.10207
• Pdf URL: https://arxiv.org/pdf/2401.10207
• Copy Paste: [[2401.10207]] Eclectic Rule Extraction for Explainability of Deep Neural Network based Intrusion Detection Systems(https://arxiv.org/abs/2401.10207)
• Keywords: extraction, explainability
• Abstract: This paper addresses trust issues created from the ubiquity of black box algorithms and surrogate explainers in Explainable Intrusion Detection Systems (X-IDS). While Explainable Artificial Intelligence (XAI) aims to enhance transparency, black box surrogate explainers, such as Local Interpretable Model-Agnostic Explanation (LIME) and SHapley Additive exPlanation (SHAP), are difficult to trust. The black box nature of these surrogate explainers makes the process behind explanation generation opaque and difficult to understand. To avoid this problem, one can use transparent white box algorithms such as Rule Extraction (RE). There are three types of RE algorithms: pedagogical, decompositional, and eclectic. Pedagogical methods offer fast but untrustworthy white-box explanations, while decompositional RE provides trustworthy explanations with poor scalability. This work explores eclectic rule extraction, which strikes a balance between scalability and trustworthiness. By combining techniques from pedagogical and decompositional approaches, eclectic rule extraction leverages the advantages of both, while mitigating some of their drawbacks. The proposed Hybrid X-IDS architecture features eclectic RE as a white box surrogate explainer for black box Deep Neural Networks (DNN). The presented eclectic RE algorithm extracts human-readable rules from hidden layers, facilitating explainable and trustworthy rulesets. Evaluations on UNSW-NB15 and CIC-IDS-2017 datasets demonstrate the algorithm's ability to generate rulesets with 99.9% accuracy, mimicking DNN outputs. The contributions of this work include the hybrid X-IDS architecture, the eclectic rule extraction algorithm applicable to intrusion detection datasets, and a thorough analysis of performance and explainability, demonstrating the trade-offs involved in rule extraction speed and accuracy.

Title: MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizer

• Authors: Changyao Tian, Xizhou Zhu, Yuwen Xiong, Weiyun Wang, Zhe Chen, Wenhai Wang, Yuntao Chen, Lewei Lu, Tong Lu, Jie Zhou, Hongsheng Li, Yu Qiao, Jifeng Dai
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2401.10208
• Pdf URL: https://arxiv.org/pdf/2401.10208
• Copy Paste: [[2401.10208]] MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizer(https://arxiv.org/abs/2401.10208)
• Keywords: generative
• Abstract: Developing generative models for interleaved image-text data has both research and practical value. It requires models to understand the interleaved sequences and subsequently generate images and text. However, existing attempts are limited by the issue that the fixed number of visual tokens cannot efficiently capture image details, which is particularly problematic in the multi-image scenarios. To address this, this paper presents MM-Interleaved, an end-to-end generative model for interleaved image-text data. It introduces a multi-scale and multi-image feature synchronizer module, allowing direct access to fine-grained image features in the previous context during the generation process. MM-Interleaved is end-to-end pre-trained on both paired and interleaved image-text corpora. It is further enhanced through a supervised fine-tuning phase, wherein the model improves its ability to follow complex multi-modal instructions. Experiments demonstrate the versatility of MM-Interleaved in recognizing visual details following multi-modal instructions and generating consistent images following both textual and visual conditions. Code and models are available at \url{https://github.com/OpenGVLab/MM-Interleaved}.

Title: AutoFT: Robust Fine-Tuning by Optimizing Hyperparameters on OOD Data

• Authors: Caroline Choi, Yoonho Lee, Annie Chen, Allan Zhou, Aditi Raghunathan, Chelsea Finn
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.10220
• Pdf URL: https://arxiv.org/pdf/2401.10220
• Copy Paste: [[2401.10220]] AutoFT: Robust Fine-Tuning by Optimizing Hyperparameters on OOD Data(https://arxiv.org/abs/2401.10220)
• Keywords: robust
• Abstract: Foundation models encode rich representations that can be adapted to a desired task by fine-tuning on task-specific data. However, fine-tuning a model on one particular data distribution often compromises the model's original performance on other distributions. Current methods for robust fine-tuning utilize hand-crafted regularization techniques to constrain the fine-tuning process towards the base foundation model. Yet, it is hard to precisely specify what characteristics of the foundation model to retain during fine-tuning, as this depends on how the pre-training, fine-tuning, and evaluation data distributions relate to each other. We propose AutoFT, a data-driven approach for guiding foundation model fine-tuning. AutoFT optimizes fine-tuning hyperparameters to maximize performance on a small out-of-distribution (OOD) validation set. To guide fine-tuning in a granular way, AutoFT searches a highly expressive hyperparameter space that includes weight coefficients for many different losses, in addition to learning rate and weight decay values. We evaluate AutoFT on nine natural distribution shifts which include domain shifts and subpopulation shifts. Our experiments show that AutoFT significantly improves generalization to new OOD data, outperforming existing robust fine-tuning methods. Notably, AutoFT achieves new state-of-the-art performance on the WILDS-iWildCam and WILDS-FMoW benchmarks, outperforming the previous best methods by $6.0\%$ and $1.5\%$, respectively.

Title: Supervised Fine-tuning in turn Improves Visual Foundation Models

• Authors: Xiaohu Jiang, Yixiao Ge, Yuying Ge, Chun Yuan, Ying Shan
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.10222
• Pdf URL: https://arxiv.org/pdf/2401.10222
• Copy Paste: [[2401.10222]] Supervised Fine-tuning in turn Improves Visual Foundation Models(https://arxiv.org/abs/2401.10222)
• Keywords: transformer
• Abstract: Image-text training like CLIP has dominated the pretraining of vision foundation models in recent years. Subsequent efforts have been made to introduce region-level visual learning into CLIP's pretraining but face scalability challenges due to the lack of large-scale region-level datasets. Drawing inspiration from supervised fine-tuning (SFT) in natural language processing such as instruction tuning, we explore the potential of fine-grained SFT in enhancing the generation of vision foundation models after their pretraining. Thus a two-stage method ViSFT (Vision SFT) is proposed to unleash the fine-grained knowledge of vision foundation models. In ViSFT, the vision foundation model is enhanced by performing visual joint learning on some in-domain tasks and then tested on out-of-domain benchmarks. With updating using ViSFT on 8 V100 GPUs in less than 2 days, a vision transformer with over 4.4B parameters shows improvements across various out-of-domain benchmarks including vision and vision-linguistic scenarios.

Title: ChatQA: Building GPT-4 Level Conversational QA Models

• Authors: Zihan Liu, Wei Ping, Rajarshi Roy, Peng Xu, Mohammad Shoeybi, Bryan Catanzaro
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.10225
• Pdf URL: https://arxiv.org/pdf/2401.10225
• Copy Paste: [[2401.10225]] ChatQA: Building GPT-4 Level Conversational QA Models(https://arxiv.org/abs/2401.10225)
• Keywords: large language model
• Abstract: In this work, we introduce ChatQA, a family of conversational question answering (QA) models, that obtain GPT-4 level accuracies. Specifically, we propose a two-stage instruction tuning method that can significantly improve the zero-shot conversational QA results from large language models (LLMs). To handle retrieval in conversational QA, we fine-tune a dense retriever on a multi-turn QA dataset, which provides comparable results to using the state-of-the-art query rewriting model while largely reducing deployment cost. Notably, our ChatQA-70B can outperform GPT-4 in terms of average score on 10 conversational QA datasets (54.14 vs. 53.90), without relying on any synthetic data from OpenAI GPT models.

Title: Towards Language-Driven Video Inpainting via Multimodal Large Language Models

• Authors: Jianzong Wu, Xiangtai Li, Chenyang Si, Shangchen Zhou, Jingkang Yang, Jiangning Zhang, Yining Li, Kai Chen, Yunhai Tong, Ziwei Liu, Chen Change Loy
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.10226
• Pdf URL: https://arxiv.org/pdf/2401.10226
• Copy Paste: [[2401.10226]] Towards Language-Driven Video Inpainting via Multimodal Large Language Models(https://arxiv.org/abs/2401.10226)
• Keywords: diffusion, large language model
• Abstract: We introduce a new task -- language-driven video inpainting, which uses natural language instructions to guide the inpainting process. This approach overcomes the limitations of traditional video inpainting methods that depend on manually labeled binary masks, a process often tedious and labor-intensive. We present the Remove Objects from Videos by Instructions (ROVI) dataset, containing 5,650 videos and 9,091 inpainting results, to support training and evaluation for this task. We also propose a novel diffusion-based language-driven video inpainting framework, the first end-to-end baseline for this task, integrating Multimodal Large Language Models to understand and execute complex language-based inpainting requests effectively. Our comprehensive results showcase the dataset's versatility and the model's effectiveness in various language-instructed inpainting scenarios. We will make datasets, code, and models publicly available.

Title: A Simple Latent Diffusion Approach for Panoptic Segmentation and Mask Inpainting

• Authors: Wouter Van Gansbeke, Bert De Brabandere
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.10227
• Pdf URL: https://arxiv.org/pdf/2401.10227
• Copy Paste: [[2401.10227]] A Simple Latent Diffusion Approach for Panoptic Segmentation and Mask Inpainting(https://arxiv.org/abs/2401.10227)
• Keywords: diffusion, generative, segmentation
• Abstract: Panoptic and instance segmentation networks are often trained with specialized object detection modules, complex loss functions, and ad-hoc post-processing steps to handle the permutation-invariance of the instance masks. This work builds upon Stable Diffusion and proposes a latent diffusion approach for panoptic segmentation, resulting in a simple architecture which omits these complexities. Our training process consists of two steps: (1) training a shallow autoencoder to project the segmentation masks to latent space; (2) training a diffusion model to allow image-conditioned sampling in latent space. The use of a generative model unlocks the exploration of mask completion or inpainting, which has applications in interactive segmentation. The experimental validation yields promising results for both panoptic segmentation and mask inpainting. While not setting a new state-of-the-art, our model's simplicity, generality, and mask completion capability are desirable properties.

Title: RAP-SAM: Towards Real-Time All-Purpose Segment Anything

• Authors: Shilin Xu, Haobo Yuan, Qingyu Shi, Lu Qi, Jingbo Wang, Yibo Yang, Yining Li, Kai Chen, Yunhai Tong, Bernard Ghanem, Xiangtai Li, Ming-Hsuan Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.10228
• Pdf URL: https://arxiv.org/pdf/2401.10228
• Copy Paste: [[2401.10228]] RAP-SAM: Towards Real-Time All-Purpose Segment Anything(https://arxiv.org/abs/2401.10228)
• Keywords: transformer, segmentation
• Abstract: Advanced by transformer architecture, vision foundation models (VFMs) achieve remarkable progress in performance and generalization ability. Segment Anything Model (SAM) is one remarkable model that can achieve generalized segmentation. However, most VFMs cannot run in realtime, which makes it difficult to transfer them into several products. On the other hand, current real-time segmentation mainly has one purpose, such as semantic segmentation on the driving scene. We argue that diverse outputs are needed for real applications. Thus, this work explores a new real-time segmentation setting, named all-purpose segmentation in real-time, to transfer VFMs in real-time deployment. It contains three different tasks, including interactive segmentation, panoptic segmentation, and video segmentation. We aim to use one model to achieve the above tasks in real-time. We first benchmark several strong baselines. Then, we present Real-Time All Purpose SAM (RAP-SAM). It contains an efficient encoder and an efficient decoupled decoder to perform prompt-driven decoding. Moreover, we further explore different training strategies and tuning methods to boost co-training performance further. Our code and model are available at https://github.com/xushilin1/RAP-SAM/.

Title: OMG-Seg: Is One Model Good Enough For All Segmentation?

• Authors: Xiangtai Li, Haobo Yuan, Wei Li, Henghui Ding, Size Wu, Wenwei Zhang, Yining Li, Kai Chen, Chen Change Loy
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.10229
• Pdf URL: https://arxiv.org/pdf/2401.10229
• Copy Paste: [[2401.10229]] OMG-Seg: Is One Model Good Enough For All Segmentation?(https://arxiv.org/abs/2401.10229)
• Keywords: transformer, segmentation
• Abstract: In this work, we address various segmentation tasks, each traditionally tackled by distinct or partially unified models. We propose OMG-Seg, One Model that is Good enough to efficiently and effectively handle all the segmentation tasks, including image semantic, instance, and panoptic segmentation, as well as their video counterparts, open vocabulary settings, prompt-driven, interactive segmentation like SAM, and video object segmentation. To our knowledge, this is the first model to handle all these tasks in one model and achieve satisfactory performance. We show that OMG-Seg, a transformer-based encoder-decoder architecture with task-specific queries and outputs, can support over ten distinct segmentation tasks and yet significantly reduce computational and parameter overhead across various tasks and datasets. We rigorously evaluate the inter-task influences and correlations during co-training. Code and models are available at https://github.com/lxtGH/OMG-Seg.

Title: ParaHome: Parameterizing Everyday Home Activities Towards 3D Generative Modeling of Human-Object Interactions

• Authors: Jeonghwan Kim, Jisoo Kim, Jeonghyeon Na, Hanbyul Joo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.10232
• Pdf URL: https://arxiv.org/pdf/2401.10232
• Copy Paste: [[2401.10232]] ParaHome: Parameterizing Everyday Home Activities Towards 3D Generative Modeling of Human-Object Interactions(https://arxiv.org/abs/2401.10232)
• Keywords: generative
• Abstract: To enable machines to learn how humans interact with the physical world in our daily activities, it is crucial to provide rich data that encompasses the 3D motion of humans as well as the motion of objects in a learnable 3D representation. Ideally, this data should be collected in a natural setup, capturing the authentic dynamic 3D signals during human-object interactions. To address this challenge, we introduce the ParaHome system, designed to capture and parameterize dynamic 3D movements of humans and objects within a common home environment. Our system consists of a multi-view setup with 70 synchronized RGB cameras, as well as wearable motion capture devices equipped with an IMU-based body suit and hand motion capture gloves. By leveraging the ParaHome system, we collect a novel large-scale dataset of human-object interaction. Notably, our dataset offers key advancement over existing datasets in three main aspects: (1) capturing 3D body and dexterous hand manipulation motion alongside 3D object movement within a contextual home environment during natural activities; (2) encompassing human interaction with multiple objects in various episodic scenarios with corresponding descriptions in texts; (3) including articulated objects with multiple parts expressed with parameterized articulations. Building upon our dataset, we introduce new research tasks aimed at building a generative model for learning and synthesizing human-object interactions in a real-world room setting.