2024-01-16

Title: DFU: scale-robust diffusion model for zero-shot super-resolution image generation

• Authors: Alex Havrilla, Kevin Rojas, Wenjing Liao, Molei Tao
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.06144
• Pdf URL: https://arxiv.org/pdf/2401.06144
• Copy Paste: [[2401.06144]] DFU: scale-robust diffusion model for zero-shot super-resolution image generation(https://arxiv.org/abs/2401.06144)
• Keywords: robust, diffusion, generative
• Abstract: Diffusion generative models have achieved remarkable success in generating images with a fixed resolution. However, existing models have limited ability to generalize to different resolutions when training data at those resolutions are not available. Leveraging techniques from operator learning, we present a novel deep-learning architecture, Dual-FNO UNet (DFU), which approximates the score operator by combining both spatial and spectral information at multiple resolutions. Comparisons of DFU to baselines demonstrate its scalability: 1) simultaneously training on multiple resolutions improves FID over training at any single fixed resolution; 2) DFU generalizes beyond its training resolutions, allowing for coherent, high-fidelity generation at higher-resolutions with the same model, i.e. zero-shot super-resolution image-generation; 3) we propose a fine-tuning strategy to further enhance the zero-shot super-resolution image-generation capability of our model, leading to a FID of 11.3 at 1.66 times the maximum training resolution on FFHQ, which no other method can come close to achieving.

Title: AAMDM: Accelerated Auto-regressive Motion Diffusion Model

• Authors: Tianyu Li, Calvin Qiao, Guanqiao Ren, KangKang Yin, Sehoon Ha
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2401.06146
• Pdf URL: https://arxiv.org/pdf/2401.06146
• Copy Paste: [[2401.06146]] AAMDM: Accelerated Auto-regressive Motion Diffusion Model(https://arxiv.org/abs/2401.06146)
• Keywords: diffusion
• Abstract: Interactive motion synthesis is essential in creating immersive experiences in entertainment applications, such as video games and virtual reality. However, generating animations that are both high-quality and contextually responsive remains a challenge. Traditional techniques in the game industry can produce high-fidelity animations but suffer from high computational costs and poor scalability. Trained neural network models alleviate the memory and speed issues, yet fall short on generating diverse motions. Diffusion models offer diverse motion synthesis with low memory usage, but require expensive reverse diffusion processes. This paper introduces the Accelerated Auto-regressive Motion Diffusion Model (AAMDM), a novel motion synthesis framework designed to achieve quality, diversity, and efficiency all together. AAMDM integrates Denoising Diffusion GANs as a fast Generation Module, and an Auto-regressive Diffusion Model as a Polishing Module. Furthermore, AAMDM operates in a lower-dimensional embedded space rather than the full-dimensional pose space, which reduces the training complexity as well as further improves the performance. We show that AAMDM outperforms existing methods in motion quality, diversity, and runtime efficiency, through comprehensive quantitative analyses and visual comparisons. We also demonstrate the effectiveness of each algorithmic component through ablation studies.

Title: Image Classifier Based Generative Method for Planar Antenna Design

• Authors: Yang Zhong, Weiping Dou, Andrew Cohen, Dia'a Bisharat, Yuandong Tian, Jiang Zhu, Qing Huo Liu
• Subjects: cs.CV, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2401.06149
• Pdf URL: https://arxiv.org/pdf/2401.06149
• Copy Paste: [[2401.06149]] Image Classifier Based Generative Method for Planar Antenna Design(https://arxiv.org/abs/2401.06149)
• Keywords: generative
• Abstract: To extend the antenna design on printed circuit boards (PCBs) for more engineers of interest, we propose a simple method that models PCB antennas with a few basic components. By taking two separate steps to decide their geometric dimensions and positions, antenna prototypes can be facilitated with no experience required. Random sampling statistics relate to the quality of dimensions are used in selecting among dimension candidates. A novel image-based classifier using a convolutional neural network (CNN) is introduced to further determine the positions of these fixed-dimension components. Two examples from wearable products have been chosen to examine the entire workflow. Their final designs are realistic and their performance metrics are not inferior to the ones designed by experienced engineers.

Title: FRED: Towards a Full Rotation-Equivariance in Aerial Image Object Detection

• Authors: Chanho Lee, Jinsu Son, Hyounguk Shon, Yunho Jeon, Junmo Kim
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.06159
• Pdf URL: https://arxiv.org/pdf/2401.06159
• Copy Paste: [[2401.06159]] FRED: Towards a Full Rotation-Equivariance in Aerial Image Object Detection(https://arxiv.org/abs/2401.06159)
• Keywords: robust
• Abstract: Rotation-equivariance is an essential yet challenging property in oriented object detection. While general object detectors naturally leverage robustness to spatial shifts due to the translation-equivariance of the conventional CNNs, achieving rotation-equivariance remains an elusive goal. Current detectors deploy various alignment techniques to derive rotation-invariant features, but still rely on high capacity models and heavy data augmentation with all possible rotations. In this paper, we introduce a Fully Rotation-Equivariant Oriented Object Detector (FRED), whose entire process from the image to the bounding box prediction is strictly equivariant. Specifically, we decouple the invariant task (object classification) and the equivariant task (object localization) to achieve end-to-end equivariance. We represent the bounding box as a set of rotation-equivariant vectors to implement rotation-equivariant localization. Moreover, we utilized these rotation-equivariant vectors as offsets in the deformable convolution, thereby enhancing the existing advantages of spatial adaptation. Leveraging full rotation-equivariance, our FRED demonstrates higher robustness to image-level rotation compared to existing methods. Furthermore, we show that FRED is one step closer to non-axis aligned learning through our experiments. Compared to state-of-the-art methods, our proposed method delivers comparable performance on DOTA-v1.0 and outperforms by 1.5 mAP on DOTA-v1.5, all while significantly reducing the model parameters to 16%.

Title: Scissorhands: Scrub Data Influence via Connection Sensitivity in Networks

• Authors: Jing Wu, Mehrtash Harandi
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2401.06187
• Pdf URL: https://arxiv.org/pdf/2401.06187
• Copy Paste: [[2401.06187]] Scissorhands: Scrub Data Influence via Connection Sensitivity in Networks(https://arxiv.org/abs/2401.06187)
• Keywords: security, privacy
• Abstract: Machine unlearning has become a pivotal task to erase the influence of data from a trained model. It adheres to recent data regulation standards and enhances the privacy and security of machine learning applications. Most existing machine unlearning methods perform well, however, they typically necessitate access to the entirety of the remaining data, which might not be feasible in certain scenarios. In this work, we present a new machine unlearning approach Scissorhands, which operates effectively with only a subset of the training data. Initially, Scissorhands identifies the most pertinent parameters in the given model relative to the forgetting data via connection sensitivity. This process involves reinitializing the most influential top-$k$ percent of these parameters, resulting in a trimmed model for erasing the influence of the forgetting data. Subsequently, Scissorhands retrains the trimmed model through a min-max optimization process, seeking parameters that preserve information on the remaining data while discarding information related to the forgetting data. Our experimental results, conducted across five distinct datasets and utilizing both CNN and ViT, demonstrate that Scissorhands, despite utilizing only a limited portion of the training data, showcases competitive performance when compared to existing methods.

Title: TriNeRFLet: A Wavelet Based Multiscale Triplane NeRF Representation

• Authors: Rajaei Khatib, Raja Giryes
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.06191
• Pdf URL: https://arxiv.org/pdf/2401.06191
• Copy Paste: [[2401.06191]] TriNeRFLet: A Wavelet Based Multiscale Triplane NeRF Representation(https://arxiv.org/abs/2401.06191)
• Keywords: diffusion
• Abstract: In recent years, the neural radiance field (NeRF) model has gained popularity due to its ability to recover complex 3D scenes. Following its success, many approaches proposed different NeRF representations in order to further improve both runtime and performance. One such example is Triplane, in which NeRF is represented using three 2D feature planes. This enables easily using existing 2D neural networks in this framework, e.g., to generate the three planes. Despite its advantage, the triplane representation lagged behind in its 3D recovery quality compared to NeRF solutions. In this work, we propose TriNeRFLet, a 2D wavelet-based multiscale triplane representation for NeRF, which closes the 3D recovery performance gap and is competitive with current state-of-the-art methods. Building upon the triplane framework, we also propose a novel super-resolution (SR) technique that combines a diffusion model with TriNeRFLet for improving NeRF resolution.

Title: CrisisKAN: Knowledge-infused and Explainable Multimodal Attention Network for Crisis Event Classification

• Authors: Shubham Gupta, Nandini Saini, Suman Kundu, Debasis Das
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2401.06194
• Pdf URL: https://arxiv.org/pdf/2401.06194
• Copy Paste: [[2401.06194]] CrisisKAN: Knowledge-infused and Explainable Multimodal Attention Network for Crisis Event Classification(https://arxiv.org/abs/2401.06194)
• Keywords: robust, extraction
• Abstract: Pervasive use of social media has become the emerging source for real-time information (like images, text, or both) to identify various events. Despite the rapid growth of image and text-based event classification, the state-of-the-art (SOTA) models find it challenging to bridge the semantic gap between features of image and text modalities due to inconsistent encoding. Also, the black-box nature of models fails to explain the model's outcomes for building trust in high-stakes situations such as disasters, pandemic. Additionally, the word limit imposed on social media posts can potentially introduce bias towards specific events. To address these issues, we proposed CrisisKAN, a novel Knowledge-infused and Explainable Multimodal Attention Network that entails images and texts in conjunction with external knowledge from Wikipedia to classify crisis events. To enrich the context-specific understanding of textual information, we integrated Wikipedia knowledge using proposed wiki extraction algorithm. Along with this, a guided cross-attention module is implemented to fill the semantic gap in integrating visual and textual data. In order to ensure reliability, we employ a model-specific approach called Gradient-weighted Class Activation Mapping (Grad-CAM) that provides a robust explanation of the predictions of the proposed model. The comprehensive experiments conducted on the CrisisMMD dataset yield in-depth analysis across various crisis-specific tasks and settings. As a result, CrisisKAN outperforms existing SOTA methodologies and provides a novel view in the domain of explainable multimodal event classification.

Title: Efficient Deformable ConvNets: Rethinking Dynamic and Sparse Operator for Vision Applications

• Authors: Yuwen Xiong, Zhiqi Li, Yuntao Chen, Feng Wang, Xizhou Zhu, Jiapeng Luo, Wenhai Wang, Tong Lu, Hongsheng Li, Yu Qiao, Lewei Lu, Jie Zhou, Jifeng Dai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.06197
• Pdf URL: https://arxiv.org/pdf/2401.06197
• Copy Paste: [[2401.06197]] Efficient Deformable ConvNets: Rethinking Dynamic and Sparse Operator for Vision Applications(https://arxiv.org/abs/2401.06197)
• Keywords: robust, diffusion, generative, segmentation
• Abstract: We introduce Deformable Convolution v4 (DCNv4), a highly efficient and effective operator designed for a broad spectrum of vision applications. DCNv4 addresses the limitations of its predecessor, DCNv3, with two key enhancements: 1. removing softmax normalization in spatial aggregation to enhance its dynamic property and expressive power and 2. optimizing memory access to minimize redundant operations for speedup. These improvements result in a significantly faster convergence compared to DCNv3 and a substantial increase in processing speed, with DCNv4 achieving more than three times the forward speed. DCNv4 demonstrates exceptional performance across various tasks, including image classification, instance and semantic segmentation, and notably, image generation. When integrated into generative models like U-Net in the latent diffusion model, DCNv4 outperforms its baseline, underscoring its possibility to enhance generative models. In practical applications, replacing DCNv3 with DCNv4 in the InternImage model to create FlashInternImage results in up to 80% speed increase and further performance improvement without further modifications. The advancements in speed and efficiency of DCNv4, combined with its robust performance across diverse vision tasks, show its potential as a foundational building block for future vision models.

Title: EASYTOOL: Enhancing LLM-based Agents with Concise Tool Instruction

• Authors: Siyu Yuan, Kaitao Song, Jiangjie Chen, Xu Tan, Yongliang Shen, Ren Kan, Dongsheng Li, Deqing Yang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.06201
• Pdf URL: https://arxiv.org/pdf/2401.06201
• Copy Paste: [[2401.06201]] EASYTOOL: Enhancing LLM-based Agents with Concise Tool Instruction(https://arxiv.org/abs/2401.06201)
• Keywords: large language model
• Abstract: To address intricate real-world tasks, there has been a rising interest in tool utilization in applications of large language models (LLMs). To develop LLM-based agents, it usually requires LLMs to understand many tool functions from different tool documentation. But these documentations could be diverse, redundant or incomplete, which immensely affects the capability of LLMs in using tools. To solve this, we introduce EASYTOOL, a framework transforming diverse and lengthy tool documentation into a unified and concise tool instruction for easier tool usage. EasyTool purifies essential information from extensive tool documentation of different sources, and elaborates a unified interface (i.e., tool instruction) to offer standardized tool descriptions and functionalities for LLM-based agents. Extensive experiments on multiple different tasks demonstrate that EasyTool can significantly reduce token consumption and improve the performance of tool utilization in real-world scenarios. Our code will be available at \url{https://github.com/microsoft/JARVIS/} in the future.

Title: An Exploratory Assessment of LLM's Potential Toward Flight Trajectory Reconstruction Analysis

• Authors: Qilei Zhang, John H. Mott
• Subjects: cs.LG, cs.AI, eess.SP
• Abstract URL: https://arxiv.org/abs/2401.06204
• Pdf URL: https://arxiv.org/pdf/2401.06204
• Copy Paste: [[2401.06204]] An Exploratory Assessment of LLM's Potential Toward Flight Trajectory Reconstruction Analysis(https://arxiv.org/abs/2401.06204)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) hold transformative potential in aviation, particularly in reconstructing flight trajectories. This paper investigates this potential, grounded in the notion that LLMs excel at processing sequential data and deciphering complex data structures. Utilizing the LLaMA 2 model, a pre-trained open-source LLM, the study focuses on reconstructing flight trajectories using Automatic Dependent Surveillance-Broadcast (ADS-B) data with irregularities inherent in real-world scenarios. The findings demonstrate the model's proficiency in filtering noise and estimating both linear and curved flight trajectories. However, the analysis also reveals challenges in managing longer data sequences, which may be attributed to the token length limitations of LLM models. The study's insights underscore the promise of LLMs in flight trajectory reconstruction and open new avenues for their broader application across the aviation and transportation sectors.

Title: Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs

• Authors: Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, Saining Xie
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.06209
• Pdf URL: https://arxiv.org/pdf/2401.06209
• Copy Paste: [[2401.06209]] Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs(https://arxiv.org/abs/2401.06209)
• Keywords: large language model
• Abstract: Is vision good enough for language? Recent advancements in multimodal models primarily stem from the powerful reasoning abilities of large language models (LLMs). However, the visual component typically depends only on the instance-level contrastive language-image pre-training (CLIP). Our research reveals that the visual capabilities in recent multimodal LLMs (MLLMs) still exhibit systematic shortcomings. To understand the roots of these errors, we explore the gap between the visual embedding space of CLIP and vision-only self-supervised learning. We identify ''CLIP-blind pairs'' - images that CLIP perceives as similar despite their clear visual differences. With these pairs, we construct the Multimodal Visual Patterns (MMVP) benchmark. MMVP exposes areas where state-of-the-art systems, including GPT-4V, struggle with straightforward questions across nine basic visual patterns, often providing incorrect answers and hallucinated explanations. We further evaluate various CLIP-based vision-and-language models and found a notable correlation between visual patterns that challenge CLIP models and those problematic for multimodal LLMs. As an initial effort to address these issues, we propose a Mixture of Features (MoF) approach, demonstrating that integrating vision self-supervised learning features with MLLMs can significantly enhance their visual grounding capabilities. Together, our research suggests visual representation learning remains an open challenge, and accurate visual grounding is crucial for future successful multimodal systems.

Title: LEGOBench: Leaderboard Generation Benchmark for Scientific Models

• Authors: Shruti Singh, Shoaib Alam, Mayank Singh
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.06233
• Pdf URL: https://arxiv.org/pdf/2401.06233
• Copy Paste: [[2401.06233]] LEGOBench: Leaderboard Generation Benchmark for Scientific Models(https://arxiv.org/abs/2401.06233)
• Keywords: large language model
• Abstract: The ever-increasing volume of paper submissions makes it difficult to stay informed about the latest state-of-the-art research. To address this challenge, we introduce LEGOBench, a benchmark for evaluating systems that generate leaderboards. LEGOBench is curated from 22 years of preprint submission data in arXiv and more than 11,000 machine learning leaderboards in the PapersWithCode portal. We evaluate the performance of four traditional graph-based ranking variants and three recently proposed large language models. Our preliminary results show significant performance gaps in automatic leaderboard generation. The code is available on https://github.com/lingo-iitgn/LEGOBench and the dataset is hosted on https://osf.io/9v2py/?view_only=6f91b0b510df498ba01595f8f278f94c .

Title: YOLO-Former: YOLO Shakes Hand With ViT

• Authors: Javad Khoramdel, Ahmad Moori, Yasamin Borhani, Armin Ghanbarzadeh, Esmaeil Najafi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.06244
• Pdf URL: https://arxiv.org/pdf/2401.06244
• Copy Paste: [[2401.06244]] YOLO-Former: YOLO Shakes Hand With ViT(https://arxiv.org/abs/2401.06244)
• Keywords: transformer
• Abstract: The proposed YOLO-Former method seamlessly integrates the ideas of transformer and YOLOv4 to create a highly accurate and efficient object detection system. The method leverages the fast inference speed of YOLOv4 and incorporates the advantages of the transformer architecture through the integration of convolutional attention and transformer modules. The results demonstrate the effectiveness of the proposed approach, with a mean average precision (mAP) of 85.76\% on the Pascal VOC dataset, while maintaining high prediction speed with a frame rate of 10.85 frames per second. The contribution of this work lies in the demonstration of how the innovative combination of these two state-of-the-art techniques can lead to further improvements in the field of object detection.

Title: AGSPNet: A framework for parcel-scale crop fine-grained semantic change detection from UAV high-resolution imagery with agricultural geographic scene constraints

• Authors: Shaochun Li, Yanjun Wang, Hengfan Cai, Lina Deng, Yunhao Lin
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.06252
• Pdf URL: https://arxiv.org/pdf/2401.06252
• Copy Paste: [[2401.06252]] AGSPNet: A framework for parcel-scale crop fine-grained semantic change detection from UAV high-resolution imagery with agricultural geographic scene constraints(https://arxiv.org/abs/2401.06252)
• Keywords: security, extraction
• Abstract: Real-time and accurate information on fine-grained changes in crop cultivation is of great significance for crop growth monitoring, yield prediction and agricultural structure adjustment. Aiming at the problems of serious spectral confusion in visible high-resolution unmanned aerial vehicle (UAV) images of different phases, interference of large complex background and salt-and-pepper noise by existing semantic change detection (SCD) algorithms, in order to effectively extract deep image features of crops and meet the demand of agricultural practical engineering applications, this paper designs and proposes an agricultural geographic scene and parcel-scale constrained SCD framework for crops (AGSPNet). AGSPNet framework contains three parts: agricultural geographic scene (AGS) division module, parcel edge extraction module and crop SCD module. Meanwhile, we produce and introduce an UAV image SCD dataset (CSCD) dedicated to agricultural monitoring, encompassing multiple semantic variation types of crops in complex geographical scene. We conduct comparative experiments and accuracy evaluations in two test areas of this dataset, and the results show that the crop SCD results of AGSPNet consistently outperform other deep learning SCD models in terms of quantity and quality, with the evaluation metrics F1-score, kappa, OA, and mIoU obtaining improvements of 0.038, 0.021, 0.011 and 0.062, respectively, on average over the sub-optimal method. The method proposed in this paper can clearly detect the fine-grained change information of crop types in complex scenes, which can provide scientific and technical support for smart agriculture monitoring and management, food policy formulation and food security assurance.

Title: FedTabDiff: Federated Learning of Diffusion Probabilistic Models for Synthetic Mixed-Type Tabular Data Generation

• Authors: Timur Sattarov, Marco Schreyer, Damian Borth
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.06263
• Pdf URL: https://arxiv.org/pdf/2401.06263
• Copy Paste: [[2401.06263]] FedTabDiff: Federated Learning of Diffusion Probabilistic Models for Synthetic Mixed-Type Tabular Data Generation(https://arxiv.org/abs/2401.06263)
• Keywords: privacy, federate, diffusion, generative
• Abstract: Realistic synthetic tabular data generation encounters significant challenges in preserving privacy, especially when dealing with sensitive information in domains like finance and healthcare. In this paper, we introduce \textit{Federated Tabular Diffusion} (FedTabDiff) for generating high-fidelity mixed-type tabular data without centralized access to the original tabular datasets. Leveraging the strengths of \textit{Denoising Diffusion Probabilistic Models} (DDPMs), our approach addresses the inherent complexities in tabular data, such as mixed attribute types and implicit relationships. More critically, FedTabDiff realizes a decentralized learning scheme that permits multiple entities to collaboratively train a generative model while respecting data privacy and locality. We extend DDPMs into the federated setting for tabular data generation, which includes a synchronous update scheme and weighted averaging for effective model aggregation. Experimental evaluations on real-world financial and medical datasets attest to the framework's capability to produce synthetic data that maintains high fidelity, utility, privacy, and coverage.

Title: A Study on Self-Supervised Pretraining for Vision Problems in Gastrointestinal Endoscopy

• Authors: Edward Sanderson, Bogdan J. Matuszewski
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.06278
• Pdf URL: https://arxiv.org/pdf/2401.06278
• Copy Paste: [[2401.06278]] A Study on Self-Supervised Pretraining for Vision Problems in Gastrointestinal Endoscopy(https://arxiv.org/abs/2401.06278)
• Keywords: segmentation
• Abstract: Solutions to vision tasks in gastrointestinal endoscopy (GIE) conventionally use image encoders pretrained in a supervised manner with ImageNet-1k as backbones. However, the use of modern self-supervised pretraining algorithms and a recent dataset of 100k unlabelled GIE images (Hyperkvasir-unlabelled) may allow for improvements. In this work, we study the fine-tuned performance of models with ResNet50 and ViT-B backbones pretrained in self-supervised and supervised manners with ImageNet-1k and Hyperkvasir-unlabelled (self-supervised only) in a range of GIE vision tasks. In addition to identifying the most suitable pretraining pipeline and backbone architecture for each task, out of those considered, our results suggest: that self-supervised pretraining generally produces more suitable backbones for GIE vision tasks than supervised pretraining; that self-supervised pretraining with ImageNet-1k is typically more suitable than pretraining with Hyperkvasir-unlabelled, with the notable exception of monocular depth estimation in colonoscopy; and that ViT-Bs are more suitable in polyp segmentation and monocular depth estimation in colonoscopy, ResNet50s are more suitable in polyp detection, and both architectures perform similarly in anatomical landmark recognition and pathological finding characterisation. We hope this work draws attention to the complexity of pretraining for GIE vision tasks, informs this development of more suitable approaches than the convention, and inspires further research on this topic to help advance this development. Code available: \underline{github.com/ESandML/SSL4GIE}

Title: Demystifying Variational Diffusion Models

• Authors: Fabio De Sousa Ribeiro, Ben Glocker
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2401.06281
• Pdf URL: https://arxiv.org/pdf/2401.06281
• Copy Paste: [[2401.06281]] Demystifying Variational Diffusion Models(https://arxiv.org/abs/2401.06281)
• Keywords: diffusion
• Abstract: Despite the growing popularity of diffusion models, gaining a deep understanding of the model class remains somewhat elusive for the uninitiated in non-equilibrium statistical physics. With that in mind, we present what we believe is a more straightforward introduction to diffusion models using directed graphical modelling and variational Bayesian principles, which imposes relatively fewer prerequisites on the average reader. Our exposition constitutes a comprehensive technical review spanning from foundational concepts like deep latent variable models to recent advances in continuous-time diffusion-based modelling, highlighting theoretical connections between model classes along the way. We provide additional mathematical insights that were omitted in the seminal works whenever possible to aid in understanding, while avoiding the introduction of new notation. We envision this article serving as a useful educational supplement for both researchers and practitioners in the area, and we welcome feedback and contributions from the community at https://github.com/biomedia-mira/demystifying-diffusion.

Title: Frequency-Time Diffusion with Neural Cellular Automata

• Authors: John Kalkhof, Arlene Kühn, Yannik Frisch, Anirban Mukhopadhyay
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.06291
• Pdf URL: https://arxiv.org/pdf/2401.06291
• Copy Paste: [[2401.06291]] Frequency-Time Diffusion with Neural Cellular Automata(https://arxiv.org/abs/2401.06291)
• Keywords: diffusion, generative
• Abstract: Denoising Diffusion Models (DDMs) have become the leading generative technique for synthesizing high-quality images but are often constrained by their UNet-based architectures that impose certain limitations. In particular, the considerable size of often hundreds of millions of parameters makes them impractical when hardware resources are limited. However, even with powerful hardware, processing images in the gigapixel range is difficult. This is especially true in fields such as microscopy or satellite imaging, where such challenges arise from the limitation to a predefined generative size and the inefficient scaling to larger images. We present two variations of Neural Cellular Automata (NCA)-based DDM methods to address these challenges and jumpstart NCA-based DDMs: Diff-NCA and FourierDiff-NCA. Diff-NCA performs diffusion by using only local features of the underlying distribution, making it suitable for applications where local features are critical. To communicate global knowledge in image space, naive NCA setups require timesteps that increase with the image scale. We solve this bottleneck of current NCA architectures by introducing FourierDiff-NCA, which advances Diff-NCA by adding a Fourier-based diffusion process and combines the frequency-organized Fourier space with the image space. By initiating diffusion in the Fourier domain and finalizing it in the image space, FourierDiff-NCA accelerates global communication. We validate our techniques by using Diff-NCA (208k parameters) to generate high-resolution digital pathology scans at 576x576 resolution and FourierDiff-NCA (887k parameters) to synthesize CelebA images at 64x64, outperforming VNCA and five times bigger UNet-based DDMs. In addition, we demonstrate FourierDiff-NCA's capabilities in super-resolution, OOD image synthesis, and inpainting without additional training.

Title: Misconfidence-based Demonstration Selection for LLM In-Context Learning

• Authors: Shangqing Xu, Chao Zhang (Georgia Institute of Technology)
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.06301
• Pdf URL: https://arxiv.org/pdf/2401.06301
• Copy Paste: [[2401.06301]] Misconfidence-based Demonstration Selection for LLM In-Context Learning(https://arxiv.org/abs/2401.06301)
• Keywords: large language model
• Abstract: In-context learning with large language models (LLMs) excels at adapting to various tasks rapidly. However, its success hinges on carefully selecting demonstrations, which remains an obstacle in practice. Current approaches to this problem either rely on hard-to-acquire external supervision or require frequent interactions with LLMs, resulting in high costs. We propose a new method called In-Context Reflection (ICR) to overcome these challenges. ICR strategically selects demonstrations to reduce the discrepancy between the LLM's outputs and the actual input-output mappings. Specifically, ICR starts with a random set of initial demonstrations, then iteratively refines it. In each step, it analyzes a pool of candidate examples and identifies the ones most likely to challenge the LLM's current understanding, measured by a new metric called misconfidence. These most confusing examples are then selected to replace the less informative demonstrations in the current set. Our comprehensive evaluation across five diverse datasets encompassing 13 subtasks shows the efficacy of ICR. Compared to existing methods, ICR achieves an average performance boost of 4%, while demonstrating remarkable cross-task generalization capabilities.

Title: Video Super-Resolution Transformer with Masked Inter&Intra-Frame Attention

• Authors: Xingyu Zhou, Leheng Zhang, Xiaorui Zhao, Keze Wang, Leida Li, Shuhang Gu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.06312
• Pdf URL: https://arxiv.org/pdf/2401.06312
• Copy Paste: [[2401.06312]] Video Super-Resolution Transformer with Masked Inter&Intra-Frame Attention(https://arxiv.org/abs/2401.06312)
• Keywords: transformer
• Abstract: Recently, Vision Transformer has achieved great success in recovering missing details in low-resolution sequences, i.e., the video super-resolution (VSR) task.Despite its superiority in VSR accuracy, the heavy computational burden as well as the large memory footprint hinder the deployment of Transformer-based VSR models on constrained devices.In this paper, we address the above issue by proposing a novel feature-level masked processing framework: VSR with Masked Intra and inter frame Attention (MIA-VSR).The core of MIA-VSR is leveraging feature-level temporal continuity between adjacent frames to reduce redundant computations and make more rational use of previously enhanced SR features. Concretely, we propose an intra-frame and inter-frame attention block which takes the respective roles of past features and input features into consideration and only exploits previously enhanced features to provide supplementary information. In addition, an adaptive block-wise mask prediction module is developed to skip unimportant computations according to feature similarity between adjacent frames. We conduct detailed ablation studies to validate our contributions and compare the proposed method with recent state-of-the-art VSR approaches. The experimental results demonstrate that MIA-VSR improves the memory and computation efficiency over state-of-the-art methods, without trading off PSNR accuracy. The code is available at https://github.com/LabShuHangGU/MIA-VSR.

Title: Striking a Balance in Fairness for Dynamic Systems Through Reinforcement Learning

• Authors: Yaowei Hu, Jacob Lear, Lu Zhang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.06318
• Pdf URL: https://arxiv.org/pdf/2401.06318
• Copy Paste: [[2401.06318]] Striking a Balance in Fairness for Dynamic Systems Through Reinforcement Learning(https://arxiv.org/abs/2401.06318)
• Keywords: fair
• Abstract: While significant advancements have been made in the field of fair machine learning, the majority of studies focus on scenarios where the decision model operates on a static population. In this paper, we study fairness in dynamic systems where sequential decisions are made. Each decision may shift the underlying distribution of features or user behavior. We model the dynamic system through a Markov Decision Process (MDP). By acknowledging that traditional fairness notions and long-term fairness are distinct requirements that may not necessarily align with one another, we propose an algorithmic framework to integrate various fairness considerations with reinforcement learning using both pre-processing and in-processing approaches. Three case studies show that our method can strike a balance between traditional fairness notions, long-term fairness, and utility.

Title: Learning from Semi-Factuals: A Debiased and Semantic-Aware Framework for Generalized Relation Discovery

• Authors: Jiaxin Wang, Lingling Zhang, Jun Liu, Tianlin Guo, Wenjun Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.06327
• Pdf URL: https://arxiv.org/pdf/2401.06327
• Copy Paste: [[2401.06327]] Learning from Semi-Factuals: A Debiased and Semantic-Aware Framework for Generalized Relation Discovery(https://arxiv.org/abs/2401.06327)
• Keywords: extraction
• Abstract: We introduce a novel task, called Generalized Relation Discovery (GRD), for open-world relation extraction. GRD aims to identify unlabeled instances in existing pre-defined relations or discover novel relations by assigning instances to clusters as well as providing specific meanings for these clusters. The key challenges of GRD are how to mitigate the serious model biases caused by labeled pre-defined relations to learn effective relational representations and how to determine the specific semantics of novel relations during classifying or clustering unlabeled instances. We then propose a novel framework, SFGRD, for this task to solve the above issues by learning from semi-factuals in two stages. The first stage is semi-factual generation implemented by a tri-view debiased relation representation module, in which we take each original sentence as the main view and design two debiased views to generate semi-factual examples for this sentence. The second stage is semi-factual thinking executed by a dual-space tri-view collaborative relation learning module, where we design a cluster-semantic space and a class-index space to learn relational semantics and relation label indices, respectively. In addition, we devise alignment and selection strategies to integrate two spaces and establish a self-supervised learning loop for unlabeled data by doing semi-factual thinking across three views. Extensive experimental results show that SFGRD surpasses state-of-the-art models in terms of accuracy by 2.36\% $\sim$5.78\% and cosine similarity by 32.19\%$\sim$ 84.45\% for relation label index and relation semantic quality, respectively. To the best of our knowledge, we are the first to exploit the efficacy of semi-factuals in relation extraction.

Title: Hyper-STTN: Social Group-aware Spatial-Temporal Transformer Network for Human Trajectory Prediction with Hypergraph Reasoning

• Authors: Weizheng Wang, Le Mao, Baijian Yang, Guohua Chen, Byung-Cheol Min
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.06344
• Pdf URL: https://arxiv.org/pdf/2401.06344
• Copy Paste: [[2401.06344]] Hyper-STTN: Social Group-aware Spatial-Temporal Transformer Network for Human Trajectory Prediction with Hypergraph Reasoning(https://arxiv.org/abs/2401.06344)
• Keywords: transformer
• Abstract: Predicting crowded intents and trajectories is crucial in varouls real-world applications, including service robots and autonomous vehicles. Understanding environmental dynamics is challenging, not only due to the complexities of modeling pair-wise spatial and temporal interactions but also the diverse influence of group-wise interactions. To decode the comprehensive pair-wise and group-wise interactions in crowded scenarios, we introduce Hyper-STTN, a Hypergraph-based Spatial-Temporal Transformer Network for crowd trajectory prediction. In Hyper-STTN, crowded group-wise correlations are constructed using a set of multi-scale hypergraphs with varying group sizes, captured through random-walk robability-based hypergraph spectral convolution. Additionally, a spatial-temporal transformer is adapted to capture pedestrians' pair-wise latent interactions in spatial-temporal dimensions. These heterogeneous group-wise and pair-wise are then fused and aligned though a multimodal transformer network. Hyper-STTN outperformes other state-of-the-art baselines and ablation models on 5 real-world pedestrian motion datasets.

Title: Seek for Incantations: Towards Accurate Text-to-Image Diffusion Synthesis through Prompt Engineering

• Authors: Chang Yu, Junran Peng, Xiangyu Zhu, Zhaoxiang Zhang, Qi Tian, Zhen Lei
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.06345
• Pdf URL: https://arxiv.org/pdf/2401.06345
• Copy Paste: [[2401.06345]] Seek for Incantations: Towards Accurate Text-to-Image Diffusion Synthesis through Prompt Engineering(https://arxiv.org/abs/2401.06345)
• Keywords: diffusion
• Abstract: The text-to-image synthesis by diffusion models has recently shown remarkable performance in generating high-quality images. Although performs well for simple texts, the models may get confused when faced with complex texts that contain multiple objects or spatial relationships. To get the desired images, a feasible way is to manually adjust the textual descriptions, i.e., narrating the texts or adding some words, which is labor-consuming. In this paper, we propose a framework to learn the proper textual descriptions for diffusion models through prompt learning. By utilizing the quality guidance and the semantic guidance derived from the pre-trained diffusion model, our method can effectively learn the prompts to improve the matches between the input text and the generated images. Extensive experiments and analyses have validated the effectiveness of the proposed method.

Title: Graph Relation Distillation for Efficient Biomedical Instance Segmentation

• Authors: Xiaoyu Liu, Yueyi Zhang, Zhiwei Xiong, Wei Huang, Bo Hu, Xiaoyan Sun, Feng Wu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.06370
• Pdf URL: https://arxiv.org/pdf/2401.06370
• Copy Paste: [[2401.06370]] Graph Relation Distillation for Efficient Biomedical Instance Segmentation(https://arxiv.org/abs/2401.06370)
• Keywords: segmentation
• Abstract: Instance-aware embeddings predicted by deep neural networks have revolutionized biomedical instance segmentation, but its resource requirements are substantial. Knowledge distillation offers a solution by transferring distilled knowledge from heavy teacher networks to lightweight yet high-performance student networks. However, existing knowledge distillation methods struggle to extract knowledge for distinguishing instances and overlook global relation information. To address these challenges, we propose a graph relation distillation approach for efficient biomedical instance segmentation, which considers three essential types of knowledge: instance-level features, instance relations, and pixel-level boundaries. We introduce two graph distillation schemes deployed at both the intra-image level and the inter-image level: instance graph distillation (IGD) and affinity graph distillation (AGD). IGD constructs a graph representing instance features and relations, transferring these two types of knowledge by enforcing instance graph consistency. AGD constructs an affinity graph representing pixel relations to capture structured knowledge of instance boundaries, transferring boundary-related knowledge by ensuring pixel affinity consistency. Experimental results on a number of biomedical datasets validate the effectiveness of our approach, enabling student models with less than $ 1\%$ parameters and less than $10\%$ inference time while achieving promising performance compared to teacher models.

Title: How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs

• Authors: Yi Zeng, Hongpeng Lin, Jingwen Zhang, Diyi Yang, Ruoxi Jia, Weiyan Shi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.06373
• Pdf URL: https://arxiv.org/pdf/2401.06373
• Copy Paste: [[2401.06373]] How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs(https://arxiv.org/abs/2401.06373)
• Keywords: security, defense, attack, large language model
• Abstract: Most traditional AI safety research has approached AI models as machines and centered on algorithm-focused attacks developed by security experts. As large language models (LLMs) become increasingly common and competent, non-expert users can also impose risks during daily interactions. This paper introduces a new perspective to jailbreak LLMs as human-like communicators, to explore this overlooked intersection between everyday language interaction and AI safety. Specifically, we study how to persuade LLMs to jailbreak them. First, we propose a persuasion taxonomy derived from decades of social science research. Then, we apply the taxonomy to automatically generate interpretable persuasive adversarial prompts (PAP) to jailbreak LLMs. Results show that persuasion significantly increases the jailbreak performance across all risk categories: PAP consistently achieves an attack success rate of over $92\%$ on Llama 2-7b Chat, GPT-3.5, and GPT-4 in $10$ trials, surpassing recent algorithm-focused attacks. On the defense side, we explore various mechanisms against PAP and, found a significant gap in existing defenses, and advocate for more fundamental mitigation for highly interactive LLMs

Title: SamLP: A Customized Segment Anything Model for License Plate Detection

• Authors: Haoxuan Ding, Junyu Gao, Yuan Yuan, Qi Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.06374
• Pdf URL: https://arxiv.org/pdf/2401.06374
• Copy Paste: [[2401.06374]] SamLP: A Customized Segment Anything Model for License Plate Detection(https://arxiv.org/abs/2401.06374)
• Keywords: robust, extraction, segmentation
• Abstract: With the emergence of foundation model, this novel paradigm of deep learning has encouraged many powerful achievements in natural language processing and computer vision. There are many advantages of foundation model, such as excellent feature extraction power, mighty generalization ability, great few-shot and zero-shot learning capacity, etc. which are beneficial to vision tasks. As the unique identity of vehicle, different countries and regions have diverse license plate (LP) styles and appearances, and even different types of vehicles have different LPs. However, recent deep learning based license plate detectors are mainly trained on specific datasets, and these limited datasets constrain the effectiveness and robustness of LP detectors. To alleviate the negative impact of limited data, an attempt to exploit the advantages of foundation model is implement in this paper. We customize a vision foundation model, i.e. Segment Anything Model (SAM), for LP detection task and propose the first LP detector based on vision foundation model, named SamLP. Specifically, we design a Low-Rank Adaptation (LoRA) fine-tuning strategy to inject extra parameters into SAM and transfer SAM into LP detection task. And then, we further propose a promptable fine-tuning step to provide SamLP with prompatable segmentation capacity. The experiments show that our proposed SamLP achieves promising detection performance compared to other LP detectors. Meanwhile, the proposed SamLP has great few-shot and zero-shot learning ability, which shows the potential of transferring vision foundation model. The code is available at https://github.com/Dinghaoxuan/SamLP

Title: Secure Targeted Message Dissemination in IoT Using Blockchain Enabled Edge Computing

• Authors: Muhammad Baqer Mollah, Md Abul Kalam Azad, Yinghui Zhang
• Subjects: cs.CR, cs.NI
• Abstract URL: https://arxiv.org/abs/2401.06384
• Pdf URL: https://arxiv.org/pdf/2401.06384
• Copy Paste: [[2401.06384]] Secure Targeted Message Dissemination in IoT Using Blockchain Enabled Edge Computing(https://arxiv.org/abs/2401.06384)
• Keywords: secure, security, privacy
• Abstract: Smart devices are considered as an integral part of Internet of Things (IoT), have an aim to make a dynamic network to exchange information, collect data, analysis, and make optimal decisions in an autonomous way to achieve more efficient, automatic, and economical services. Message dissemination among these smart devices allows adding new features, sending updated instructions, alerts or safety messages, informing the pricing information or billing amount, incentives, and installing security patches. On one hand, such message disseminations are directly beneficial to the all parties involved in the IoT system. On the other hand, due to remote procedure, smart devices, vendors, and other involved authorities might have to meet a number of security, privacy, and performance related concerns while disseminating messages among targeted devices. To this end, in this paper, we design STarEdgeChain, a security and privacy aware targeted message dissemination in IoT to show how blockchain along with advanced cryptographic techniques are devoted to address such concerns. In fact, the STarEdgeChain employs a permissioned blockchain assisted edge computing in order to expedite a single signcrypted message dissemination among targeted groups of devices, at the same time avoiding the dependency of utilizing multiple unicasting approaches. Finally, we develop a software prototype of STarEdgeChain and show it's practicability for smart devices. The codes are publicly available at https://github.com/mbaqer/Blockchain-IoT

Title: SD-MVS: Segmentation-Driven Deformation Multi-View Stereo with Spherical Refinement and EM optimization

• Authors: Zhenlong Yuan, Jiakai Cao, Zhaoxin Li, Hao Jiang, Zhaoqi Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.06385
• Pdf URL: https://arxiv.org/pdf/2401.06385
• Copy Paste: [[2401.06385]] SD-MVS: Segmentation-Driven Deformation Multi-View Stereo with Spherical Refinement and EM optimization(https://arxiv.org/abs/2401.06385)
• Keywords: segmentation
• Abstract: In this paper, we introduce Segmentation-Driven Deformation Multi-View Stereo (SD-MVS), a method that can effectively tackle challenges in 3D reconstruction of textureless areas. We are the first to adopt the Segment Anything Model (SAM) to distinguish semantic instances in scenes and further leverage these constraints for pixelwise patch deformation on both matching cost and propagation. Concurrently, we propose a unique refinement strategy that combines spherical coordinates and gradient descent on normals and pixelwise search interval on depths, significantly improving the completeness of reconstructed 3D model. Furthermore, we adopt the Expectation-Maximization (EM) algorithm to alternately optimize the aggregate matching cost and hyperparameters, effectively mitigating the problem of parameters being excessively dependent on empirical tuning. Evaluations on the ETH3D high-resolution multi-view stereo benchmark and the Tanks and Temples dataset demonstrate that our method can achieve state-of-the-art results with less time consumption.

Title: Adaptive Data Augmentation for Aspect Sentiment Quad Prediction

• Authors: Wenyuan Zhang, Xinghua Zhang, Shiyao Cui, Kun Huang, Xuebin Wang, Tingwen Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.06394
• Pdf URL: https://arxiv.org/pdf/2401.06394
• Copy Paste: [[2401.06394]] Adaptive Data Augmentation for Aspect Sentiment Quad Prediction(https://arxiv.org/abs/2401.06394)
• Keywords: generative
• Abstract: Aspect sentiment quad prediction (ASQP) aims to predict the quad sentiment elements for a given sentence, which is a critical task in the field of aspect-based sentiment analysis. However, the data imbalance issue has not received sufficient attention in ASQP task. In this paper, we divide the issue into two-folds, quad-pattern imbalance and aspect-category imbalance, and propose an Adaptive Data Augmentation (ADA) framework to tackle the imbalance issue. Specifically, a data augmentation process with a condition function adaptively enhances the tail quad patterns and aspect categories, alleviating the data imbalance in ASQP. Following previous studies, we also further explore the generative framework for extracting complete quads by introducing the category prior knowledge and syntax-guided decoding target. Experimental results demonstrate that data augmentation for imbalance in ASQP task can improve the performance, and the proposed ADA method is superior to naive data oversampling.

Title: ModaVerse: Efficiently Transforming Modalities with LLMs

• Authors: Xinyu Wang, Bohan Zhuang, Qi Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.06395
• Pdf URL: https://arxiv.org/pdf/2401.06395
• Copy Paste: [[2401.06395]] ModaVerse: Efficiently Transforming Modalities with LLMs(https://arxiv.org/abs/2401.06395)
• Keywords: generative, large language model
• Abstract: Humans possess the capability to comprehend diverse modalities and seamlessly transfer information between them. In this work, we introduce ModaVerse, a Multi-modal Large Language Model (MLLM) capable of comprehending and transforming content across various modalities including images, videos, and audio. Predominant MLLM frameworks have largely relied on the alignment of latent spaces of textual and non-textual features. This alignment process, which synchronizes a language model trained on textual data with encoders and decoders trained on multi-modal data, often necessitates extensive training of several projection layers in multiple stages. Inspired by LLM-as-agent methodologies, we propose a novel Input/Output (I/O) alignment mechanism that operates directly at the level of natural language. It aligns the LLM's output with the input of generative models, avoiding the complexities associated with latent feature alignments, and simplifying the multiple training stages of existing MLLMs into a single, efficient process. This conceptual advancement leads to significant reductions in both data and computational costs. By conducting experiments on several benchmarks, we demonstrate that our approach attains comparable performance with the state of the art while achieving considerable efficiencies in data usage and training duration.

Title: UMG-CLIP: A Unified Multi-Granularity Vision Generalist for Open-World Understanding

• Authors: Bowen Shi, Peisen Zhao, Zichen Wang, Yuhang Zhang, Yaoming Wang, Jin Li, Wenrui Dai, Junni Zou, Hongkai Xiong, Qi Tian, Xiaopeng Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.06397
• Pdf URL: https://arxiv.org/pdf/2401.06397
• Copy Paste: [[2401.06397]] UMG-CLIP: A Unified Multi-Granularity Vision Generalist for Open-World Understanding(https://arxiv.org/abs/2401.06397)
• Keywords: segmentation
• Abstract: Vision-language foundation models, represented by Contrastive language-image pre-training (CLIP), have gained increasing attention for jointly understanding both vision and textual tasks. However, existing approaches primarily focus on training models to match global image representations with textual descriptions, thereby overlooking the critical alignment between local regions and corresponding text tokens. This paper extends CLIP with multi-granularity alignment. Notably, we deliberately construct a new dataset comprising pseudo annotations at various levels of granularities, encompassing image-level, region-level, and pixel-level captions/tags. Accordingly, we develop a unified multi-granularity learning framework, named UMG-CLIP, that simultaneously empowers the model with versatile perception abilities across different levels of detail. Equipped with parameter efficient tuning, UMG-CLIP surpasses current widely used CLIP models and achieves state-of-the-art performance on diverse image understanding benchmarks, including open-world recognition, retrieval, semantic segmentation, and panoptic segmentation tasks. We hope UMG-CLIP can serve as a valuable option for advancing vision-language foundation models.

Title: Generalizing Visual Question Answering from Synthetic to Human-Written Questions via a Chain of QA with a Large Language Model

• Authors: Taehee Kim, Yeongjae Cho, Heejun Shin, Yohan Jo, Dongmyung Shin
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2401.06400
• Pdf URL: https://arxiv.org/pdf/2401.06400
• Copy Paste: [[2401.06400]] Generalizing Visual Question Answering from Synthetic to Human-Written Questions via a Chain of QA with a Large Language Model(https://arxiv.org/abs/2401.06400)
• Keywords: large language model
• Abstract: Visual question answering (VQA) is a task where an image is given, and a series of questions are asked about the image. To build an efficient VQA algorithm, a large amount of QA data is required which is very expensive. Generating synthetic QA pairs based on templates is a practical way to obtain data. However, VQA models trained on those data do not perform well on complex, human-written questions. To address this issue, we propose a new method called {\it chain of QA for human-written questions} (CoQAH). CoQAH utilizes a sequence of QA interactions between a large language model and a VQA model trained on synthetic data to reason and derive logical answers for human-written questions. We tested the effectiveness of CoQAH on two types of human-written VQA datasets for 3D-rendered and chest X-ray images and found that it achieved state-of-the-art accuracy in both types of data. Notably, CoQAH outperformed general vision-language models, VQA models, and medical foundation models with no finetuning.

Title: Knowledge-Informed Machine Learning for Cancer Diagnosis and Prognosis: A review

• Authors: Lingchao Mao, Hairong Wang, Leland S. Hu, Nhan L Tran, Peter D Canoll, Kristin R Swanson, Jing Li
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.06406
• Pdf URL: https://arxiv.org/pdf/2401.06406
• Copy Paste: [[2401.06406]] Knowledge-Informed Machine Learning for Cancer Diagnosis and Prognosis: A review(https://arxiv.org/abs/2401.06406)
• Keywords: robust, interpretability
• Abstract: Cancer remains one of the most challenging diseases to treat in the medical field. Machine learning has enabled in-depth analysis of rich multi-omics profiles and medical imaging for cancer diagnosis and prognosis. Despite these advancements, machine learning models face challenges stemming from limited labeled sample sizes, the intricate interplay of high-dimensionality data types, the inherent heterogeneity observed among patients and within tumors, and concerns about interpretability and consistency with existing biomedical knowledge. One approach to surmount these challenges is to integrate biomedical knowledge into data-driven models, which has proven potential to improve the accuracy, robustness, and interpretability of model results. Here, we review the state-of-the-art machine learning studies that adopted the fusion of biomedical knowledge and data, termed knowledge-informed machine learning, for cancer diagnosis and prognosis. Emphasizing the properties inherent in four primary data types including clinical, imaging, molecular, and treatment data, we highlight modeling considerations relevant to these contexts. We provide an overview of diverse forms of knowledge representation and current strategies of knowledge integration into machine learning pipelines with concrete examples. We conclude the review article by discussing future directions to advance cancer research through knowledge-informed machine learning.

Title: AboutMe: Using Self-Descriptions in Webpages to Document the Effects of English Pretraining Data Filters

• Authors: Li Lucy, Suchin Gururangan, Luca Soldaini, Emma Strubell, David Bamman, Lauren Klein, Jesse Dodge
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.06408
• Pdf URL: https://arxiv.org/pdf/2401.06408
• Copy Paste: [[2401.06408]] AboutMe: Using Self-Descriptions in Webpages to Document the Effects of English Pretraining Data Filters(https://arxiv.org/abs/2401.06408)
• Keywords: large language model
• Abstract: Large language models' (LLMs) abilities are drawn from their pretraining data, and model development begins with data curation. However, decisions around what data is retained or removed during this initial stage is under-scrutinized. In our work, we ground web text, which is a popular pretraining data source, to its social and geographic contexts. We create a new dataset of 10.3 million self-descriptions of website creators, and extract information about who they are and where they are from: their topical interests, social roles, and geographic affiliations. Then, we conduct the first study investigating how ten "quality" and English language identification (langID) filters affect webpages that vary along these social dimensions. Our experiments illuminate a range of implicit preferences in data curation: we show that some quality classifiers act like topical domain filters, and langID can overlook English content from some regions of the world. Overall, we hope that our work will encourage a new line of research on pretraining data curation practices and its social implications.

Title: Mission: Impossible Language Models

• Authors: Julie Kallini, Isabel Papadimitriou, Richard Futrell, Kyle Mahowald, Christopher Potts
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.06416
• Pdf URL: https://arxiv.org/pdf/2401.06416
• Copy Paste: [[2401.06416]] Mission: Impossible Language Models(https://arxiv.org/abs/2401.06416)
• Keywords: large language model
• Abstract: Chomsky and others have very directly claimed that large language models (LLMs) are equally capable of learning languages that are possible and impossible for humans to learn. However, there is very little published experimental evidence to support such a claim. Here, we develop a set of synthetic impossible languages of differing complexity, each designed by systematically altering English data with unnatural word orders and grammar rules. These languages lie on an impossibility continuum: at one end are languages that are inherently impossible, such as random and irreversible shuffles of English words, and on the other, languages that may not be intuitively impossible but are often considered so in linguistics, particularly those with rules based on counting word positions. We report on a wide range of evaluations to assess the capacity of GPT-2 small models to learn these uncontroversially impossible languages, and crucially, we perform these assessments at various stages throughout training to compare the learning process for each language. Our core finding is that GPT-2 struggles to learn impossible languages when compared to English as a control, challenging the core claim. More importantly, we hope our approach opens up a productive line of inquiry in which different LLM architectures are tested on a variety of impossible languages in an effort to learn more about how LLMs can be used as tools for these cognitive and typological investigations.

Title: UPDP: A Unified Progressive Depth Pruner for CNN and Vision Transformer

• Authors: Ji Liu, Dehua Tang, Yuanxian Huang, Li Zhang, Xiaocheng Zeng, Dong Li, Mingjie Lu, Jinzhang Peng, Yu Wang, Fan Jiang, Lu Tian, Ashish Sirasao
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.06426
• Pdf URL: https://arxiv.org/pdf/2401.06426
• Copy Paste: [[2401.06426]] UPDP: A Unified Progressive Depth Pruner for CNN and Vision Transformer(https://arxiv.org/abs/2401.06426)
• Keywords: transformer
• Abstract: Traditional channel-wise pruning methods by reducing network channels struggle to effectively prune efficient CNN models with depth-wise convolutional layers and certain efficient modules, such as popular inverted residual blocks. Prior depth pruning methods by reducing network depths are not suitable for pruning some efficient models due to the existence of some normalization layers. Moreover, finetuning subnet by directly removing activation layers would corrupt the original model weights, hindering the pruned model from achieving high performance. To address these issues, we propose a novel depth pruning method for efficient models. Our approach proposes a novel block pruning strategy and progressive training method for the subnet. Additionally, we extend our pruning method to vision transformer models. Experimental results demonstrate that our method consistently outperforms existing depth pruning methods across various pruning configurations. We obtained three pruned ConvNeXtV1 models with our method applying on ConvNeXtV1, which surpass most SOTA efficient models with comparable inference performance. Our method also achieves state-of-the-art pruning performance on the vision transformer model.

Title: Mutual Distillation Learning For Person Re-Identification

• Authors: Huiyuan Fu, Kuilong Cui, Chuanming Wang, Mengshi Qi, Huadong Ma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.06430
• Pdf URL: https://arxiv.org/pdf/2401.06430
• Copy Paste: [[2401.06430]] Mutual Distillation Learning For Person Re-Identification(https://arxiv.org/abs/2401.06430)
• Keywords: extraction
• Abstract: With the rapid advancements in deep learning technologies, person re-identification (ReID) has witnessed remarkable performance improvements. However, the majority of prior works have traditionally focused on solving the problem via extracting features solely from a single perspective, such as uniform partitioning, hard attention mechanisms, or semantic masks. While these approaches have demonstrated efficacy within specific contexts, they fall short in diverse situations. In this paper, we propose a novel approach, Mutual Distillation Learning For Person Re-identification (termed as MDPR), which addresses the challenging problem from multiple perspectives within a single unified model, leveraging the power of mutual distillation to enhance the feature representations collectively. Specifically, our approach encompasses two branches: a hard content branch to extract local features via a uniform horizontal partitioning strategy and a Soft Content Branch to dynamically distinguish between foreground and background and facilitate the extraction of multi-granularity features via a carefully designed attention mechanism. To facilitate knowledge exchange between these two branches, a mutual distillation and fusion process is employed, promoting the capability of the outputs of each branch. Extensive experiments are conducted on widely used person ReID datasets to validate the effectiveness and superiority of our approach. Notably, our method achieves an impressive $88.7\%/94.4\%$ in mAP/Rank-1 on the DukeMTMC-reID dataset, surpassing the current state-of-the-art results. Our source code is available at https://github.com/KuilongCui/MDPR.

Title: From Automation to Augmentation: Large Language Models Elevating Essay Scoring Landscape

• Authors: Changrong Xiao, Wenxing Ma, Sean Xin Xu, Kunpeng Zhang, Yufang Wang, Qi Fu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.06431
• Pdf URL: https://arxiv.org/pdf/2401.06431
• Copy Paste: [[2401.06431]] From Automation to Augmentation: Large Language Models Elevating Essay Scoring Landscape(https://arxiv.org/abs/2401.06431)
• Keywords: interpretability, large language model
• Abstract: Receiving immediate and personalized feedback is crucial for second-language learners, and Automated Essay Scoring (AES) systems are a vital resource when human instructors are unavailable. This study investigates the effectiveness of Large Language Models (LLMs), specifically GPT-4 and fine-tuned GPT-3.5, as tools for AES. Our comprehensive set of experiments, conducted on both public and private datasets, highlights the remarkable advantages of LLM-based AES systems. They include superior accuracy, consistency, generalizability, and interpretability, with fine-tuned GPT-3.5 surpassing traditional grading models. Additionally, we undertake LLM-assisted human evaluation experiments involving both novice and expert graders. One pivotal discovery is that LLMs not only automate the grading process but also enhance the performance of human graders. Novice graders when provided with feedback generated by LLMs, achieve a level of accuracy on par with experts, while experts become more efficient and maintain greater consistency in their assessments. These results underscore the potential of LLMs in educational technology, paving the way for effective collaboration between humans and AI, ultimately leading to transformative learning experiences through AI-generated feedback.

Title: Heterogeneous Low-Rank Approximation for Federated Fine-tuning of On-Device Foundation Models

• Authors: Yae Jee Cho, Luyang Liu, Zheng Xu, Aldi Fahrezi, Gauri Joshi
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2401.06432
• Pdf URL: https://arxiv.org/pdf/2401.06432
• Copy Paste: [[2401.06432]] Heterogeneous Low-Rank Approximation for Federated Fine-tuning of On-Device Foundation Models(https://arxiv.org/abs/2401.06432)
• Keywords: privacy, federate
• Abstract: Large foundation models (FMs) adapt surprisingly well to specific domains or tasks with fine-tuning. Federated learning (FL) further enables private FM fine-tuning using the local data on devices. However, the standard FMs' large size poses challenges for resource-constrained and heterogeneous devices. To address this, we consider FMs with reduced parameter sizes, referred to as on-device FMs (ODFMs). While ODFMs allow on-device inference, computational constraints still hinder efficient federated fine-tuning. We propose a parameter-efficient federated fine-tuning method for ODFMs using heterogeneous low-rank approximations (LoRAs) that addresses system and data heterogeneity. We show that homogeneous LoRA ranks face a trade-off between overfitting and slow convergence, and propose HetLoRA, which employs heterogeneous ranks across clients and eliminates the shortcomings of homogeneous HetLoRA. By applying rank self-pruning locally and sparsity-weighted aggregation at the server, we combine the advantages of high and low-rank LoRAs, which achieves improved convergence speed and final performance compared to homogeneous LoRA. Furthermore, it offers enhanced computation efficiency compared to full fine-tuning, making it suitable for heterogeneous devices while preserving data privacy.

Title: Improving Graph Convolutional Networks with Transformer Layer in social-based items recommendation

• Authors: Thi Linh Hoang, Tuan Dung Pham, Viet Cuong Ta
• Subjects: cs.LG, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2401.06436
• Pdf URL: https://arxiv.org/pdf/2401.06436
• Copy Paste: [[2401.06436]] Improving Graph Convolutional Networks with Transformer Layer in social-based items recommendation(https://arxiv.org/abs/2401.06436)
• Keywords: transformer
• Abstract: In this work, we have proposed an approach for improving the GCN for predicting ratings in social networks. Our model is expanded from the standard model with several layers of transformer architecture. The main focus of the paper is on the encoder architecture for node embedding in the network. Using the embedding layer from the graph-based convolution layer, the attention mechanism could rearrange the feature space to get a more efficient embedding for the downstream task. The experiments showed that our proposed architecture achieves better performance than GCN on the traditional link prediction task.

Title: RotationDrag: Point-based Image Editing with Rotated Diffusion Features

• Authors: Minxing Luo, Wentao Cheng, Jian Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.06442
• Pdf URL: https://arxiv.org/pdf/2401.06442
• Copy Paste: [[2401.06442]] RotationDrag: Point-based Image Editing with Rotated Diffusion Features(https://arxiv.org/abs/2401.06442)
• Keywords: diffusion, generative
• Abstract: A precise and user-friendly manipulation of image content while preserving image fidelity has always been crucial to the field of image editing. Thanks to the power of generative models, recent point-based image editing methods allow users to interactively change the image content with high generalizability by clicking several control points. But the above mentioned editing process is usually based on the assumption that features stay constant in the motion supervision step from initial to target points. In this work, we conduct a comprehensive investigation in the feature space of diffusion models, and find that features change acutely under in-plane rotation. Based on this, we propose a novel approach named RotationDrag, which significantly improves point-based image editing performance when users intend to in-plane rotate the image content. Our method tracks handle points more precisely by utilizing the feature map of the rotated images, thus ensuring precise optimization and high image fidelity. Furthermore, we build a in-plane rotation focused benchmark called RotateBench, the first benchmark to evaluate the performance of point-based image editing method under in-plane rotation scenario on both real images and generated images. A thorough user study demonstrates the superior capability in accomplishing in-plane rotation that users intend to achieve, comparing the DragDiffusion baseline and other existing diffusion-based methods. See the project page https://github.com/Tony-Lowe/RotationDrag for code and experiment results.

Title: BOK-VQA: Bilingual Outside Knowledge-based Visual Question Answering via Graph Representation Pretraining

• Authors: Minjun Kim, Seungwoo Song, Youhan Lee, Haneol Jang, Kyungtae Lim
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.06443
• Pdf URL: https://arxiv.org/pdf/2401.06443
• Copy Paste: [[2401.06443]] BOK-VQA: Bilingual Outside Knowledge-based Visual Question Answering via Graph Representation Pretraining(https://arxiv.org/abs/2401.06443)
• Keywords: generative
• Abstract: The current research direction in generative models, such as the recently developed GPT4, aims to find relevant knowledge information for multimodal and multilingual inputs to provide answers. Under these research circumstances, the demand for multilingual evaluation of visual question answering (VQA) tasks, a representative task of multimodal systems, has increased. Accordingly, we propose a bilingual outside-knowledge VQA (BOK-VQA) dataset in this study that can be extended to multilingualism. The proposed data include 17K images, 17K question-answer pairs for both Korean and English and 280K instances of knowledge information related to question-answer content. We also present a framework that can effectively inject knowledge information into a VQA system by pretraining the knowledge information of BOK-VQA data in the form of graph embeddings. Finally, through in-depth analysis, we demonstrated the actual effect of the knowledge information contained in the constructed training data on VQA.

Title: AttributionScanner: A Visual Analytics System for Metadata-Free Data-Slicing Based Model Validation

• Authors: Xiwei Xuan, Jorge Piazentin Ono, Liang Gou, Kwan-Liu Ma, Liu Ren
• Subjects: cs.CV, cs.HC
• Abstract URL: https://arxiv.org/abs/2401.06462
• Pdf URL: https://arxiv.org/pdf/2401.06462
• Copy Paste: [[2401.06462]] AttributionScanner: A Visual Analytics System for Metadata-Free Data-Slicing Based Model Validation(https://arxiv.org/abs/2401.06462)
• Keywords: data-free
• Abstract: Data slice-finding is an emerging technique for evaluating machine learning models. It works by identifying subgroups within a specified dataset that exhibit poor performance, often defined by distinct feature sets or meta-information. However, in the context of unstructured image data, data slice-finding poses two notable challenges: it requires additional metadata -- a laborious and costly requirement, and also demands non-trivial efforts for interpreting the root causes of the underperformance within data slices. To address these challenges, we introduce AttributionScanner, an innovative human-in-the-loop Visual Analytics (VA) system, designed for data-slicing-based machine learning (ML) model validation. Our approach excels in identifying interpretable data slices, employing explainable features extracted through the lens of Explainable AI (XAI) techniques, and removing the necessity for additional metadata of textual annotations or cross-model embeddings. AttributionScanner demonstrates proficiency in pinpointing critical model issues, including spurious correlations and mislabeled data. Our novel VA interface visually summarizes data slices, enabling users to gather insights into model behavior patterns effortlessly. Furthermore, our framework closes the ML Development Cycle by empowering domain experts to address model issues by using a cutting-edge neural network regularization technique. The efficacy of AttributionScanner is underscored through two prototype use cases, elucidating its substantial effectiveness in model validation for vision-centric tasks. Our approach paves the way for ML researchers and practitioners to drive interpretable model validation in a data-efficient way, ultimately leading to more reliable and accurate models.

Title: PersianMind: A Cross-Lingual Persian-English Large Language Model

• Authors: Pedram Rostami, Ali Salemi, Mohammad Javad Dousti
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.06466
• Pdf URL: https://arxiv.org/pdf/2401.06466
• Copy Paste: [[2401.06466]] PersianMind: A Cross-Lingual Persian-English Large Language Model(https://arxiv.org/abs/2401.06466)
• Keywords: large language model
• Abstract: Large language models demonstrate remarkable proficiency in various linguistic tasks and have extensive knowledge across various domains. Although they perform best in English, their ability in other languages is notable too. In contrast, open-source models, such as LLaMa, are primarily trained on English datasets, resulting in poor performance in non-English languages. In this paper, we introduce PersianMind, an open-source bilingual large language model which demonstrates comparable performance to closed-source GPT-3.5-turbo in the Persian language. By expanding LLaMa2's vocabulary with 10,000 Persian tokens and training it on a dataset comprising nearly 2 billion Persian tokens, we show that our approach preserves the model's English knowledge and employs transfer learning to excel at transferring task knowledge from one language to another.

Title: Adapting Large Language Models for Document-Level Machine Translation

• Authors: Minghao Wu, Thuy-Trang Vu, Lizhen Qu, George Foster, Gholamreza Haffari
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.06468
• Pdf URL: https://arxiv.org/pdf/2401.06468
• Copy Paste: [[2401.06468]] Adapting Large Language Models for Document-Level Machine Translation(https://arxiv.org/abs/2401.06468)
• Keywords: large language model
• Abstract: Large language models (LLMs) have made significant strides in various natural language processing (NLP) tasks. Recent research shows that the moderately-sized LLMs often outperform their larger counterparts after task-specific fine-tuning. In this work, we delve into the process of adapting LLMs to specialize in document-level machine translation (DocMT) for a specific language pair. Firstly, we explore how prompt strategies affect downstream translation performance. Then, we conduct extensive experiments with two fine-tuning methods, three LLM backbones, and 18 translation tasks across nine language pairs. Our findings indicate that in some cases, these specialized models even surpass GPT-4 in translation performance, while they still significantly suffer from the off-target translation issue in others, even if they are exclusively fine-tuned on bilingual parallel documents. Furthermore, we provide an in-depth analysis of these LLMs tailored for DocMT, exploring aspects such as translation errors, the scaling law of parallel documents, out-of-domain generalization, and the impact of zero-shot crosslingual transfer. The findings of this research not only shed light on the strengths and limitations of LLM-based DocMT models but also provide a foundation for future research in DocMT.

Title: Self-supervised Learning of Dense Hierarchical Representations for Medical Image Segmentation

• Authors: Eytan Kats, Jochen G. Hirsch, Mattias P. Heinrich
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.06473
• Pdf URL: https://arxiv.org/pdf/2401.06473
• Copy Paste: [[2401.06473]] Self-supervised Learning of Dense Hierarchical Representations for Medical Image Segmentation(https://arxiv.org/abs/2401.06473)
• Keywords: segmentation
• Abstract: This paper demonstrates a self-supervised framework for learning voxel-wise coarse-to-fine representations tailored for dense downstream tasks. Our approach stems from the observation that existing methods for hierarchical representation learning tend to prioritize global features over local features due to inherent architectural bias. To address this challenge, we devise a training strategy that balances the contributions of features from multiple scales, ensuring that the learned representations capture both coarse and fine-grained details. Our strategy incorporates 3-fold improvements: (1) local data augmentations, (2) a hierarchically balanced architecture, and (3) a hybrid contrastive-restorative loss function. We evaluate our method on CT and MRI data and demonstrate that our new approach particularly beneficial for fine-tuning with limited annotated data and consistently outperforms the baseline counterpart in linear evaluation settings.

Title: Kun: Answer Polishment for Chinese Self-Alignment with Instruction Back-Translation

• Authors: Tianyu Zheng, Shuyue Guo, Xingwei Qu, Jiawei Guo, Weixu Zhang, Xinrun Du, Chenghua Lin, Wenhao Huang, Wenhu Chen, Jie Fu, Ge Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.06477
• Pdf URL: https://arxiv.org/pdf/2401.06477
• Copy Paste: [[2401.06477]] Kun: Answer Polishment for Chinese Self-Alignment with Instruction Back-Translation(https://arxiv.org/abs/2401.06477)
• Keywords: robust, large language model
• Abstract: In this paper, we introduce Kun, a novel approach for creating high-quality instruction-tuning datasets for large language models (LLMs) without relying on manual annotations. Adapting a self-training algorithm based on instruction back-translation and answer polishment, Kun leverages unlabelled data from diverse sources such as Wudao, Wanjuan, and SkyPile to generate a substantial dataset of over a million Chinese instructional data points. This approach significantly deviates from traditional methods by using a self-curation process to refine and select the most effective instruction-output pairs. Our experiments with the 6B-parameter Yi model across various benchmarks demonstrate Kun's robustness and scalability. Our method's core contributions lie in its algorithmic advancement, which enhances data retention and clarity, and its innovative data generation approach that substantially reduces the reliance on costly and time-consuming manual annotations. This methodology presents a scalable and efficient solution for improving the instruction-following capabilities of LLMs, with significant implications for their application across diverse fields. The code and dataset can be found at https://github.com/Zheng0428/COIG-Kun

Title: An investigation of structures responsible for gender bias in BERT and DistilBERT

• Authors: Thibaud Leteno, Antoine Gourru, Charlotte Laclau, Christophe Gravier
• Subjects: cs.CL, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.06495
• Pdf URL: https://arxiv.org/pdf/2401.06495
• Copy Paste: [[2401.06495]] An investigation of structures responsible for gender bias in BERT and DistilBERT(https://arxiv.org/abs/2401.06495)
• Keywords: fair, transformer
• Abstract: In recent years, large Transformer-based Pre-trained Language Models (PLM) have changed the Natural Language Processing (NLP) landscape, by pushing the performance boundaries of the state-of-the-art on a wide variety of tasks. However, this performance gain goes along with an increase in complexity, and as a result, the size of such models (up to billions of parameters) represents a constraint for their deployment on embedded devices or short-inference time tasks. To cope with this situation, compressed models emerged (e.g. DistilBERT), democratizing their usage in a growing number of applications that impact our daily lives. A crucial issue is the fairness of the predictions made by both PLMs and their distilled counterparts. In this paper, we propose an empirical exploration of this problem by formalizing two questions: (1) Can we identify the neural mechanism(s) responsible for gender bias in BERT (and by extension DistilBERT)? (2) Does distillation tend to accentuate or mitigate gender bias (e.g. is DistilBERT more prone to gender bias than its uncompressed version, BERT)? Our findings are the following: (I) one cannot identify a specific layer that produces bias; (II) every attention head uniformly encodes bias; except in the context of underrepresented classes with a high imbalance of the sensitive attribute; (III) this subset of heads is different as we re-fine tune the network; (IV) bias is more homogeneously produced by the heads in the distilled model.

Title: Improving the Detection of Small Oriented Objects in Aerial Images

• Authors: Chandler Timm C. Doloriel, Rhandley D. Cajote
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.06503
• Pdf URL: https://arxiv.org/pdf/2401.06503
• Copy Paste: [[2401.06503]] Improving the Detection of Small Oriented Objects in Aerial Images(https://arxiv.org/abs/2401.06503)
• Keywords: segmentation
• Abstract: Small oriented objects that represent tiny pixel-area in large-scale aerial images are difficult to detect due to their size and orientation. Existing oriented aerial detectors have shown promising results but are mainly focused on orientation modeling with less regard to the size of the objects. In this work, we proposed a method to accurately detect small oriented objects in aerial images by enhancing the classification and regression tasks of the oriented object detection model. We designed the Attention-Points Network consisting of two losses: Guided-Attention Loss (GALoss) and Box-Points Loss (BPLoss). GALoss uses an instance segmentation mask as ground-truth to learn the attention features needed to improve the detection of small objects. These attention features are then used to predict box points for BPLoss, which determines the points' position relative to the target oriented bounding box. Experimental results show the effectiveness of our Attention-Points Network on a standard oriented aerial dataset with small object instances (DOTA-v1.5) and on a maritime-related dataset (HRSC2016). The code is publicly available.

Title: Frequency Masking for Universal Deepfake Detection

• Authors: Chandler Timm Doloriel, Ngai-Man Cheung
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.06506
• Pdf URL: https://arxiv.org/pdf/2401.06506
• Copy Paste: [[2401.06506]] Frequency Masking for Universal Deepfake Detection(https://arxiv.org/abs/2401.06506)
• Keywords: generative
• Abstract: We study universal deepfake detection. Our goal is to detect synthetic images from a range of generative AI approaches, particularly from emerging ones which are unseen during training of the deepfake detector. Universal deepfake detection requires outstanding generalization capability. Motivated by recently proposed masked image modeling which has demonstrated excellent generalization in self-supervised pre-training, we make the first attempt to explore masked image modeling for universal deepfake detection. We study spatial and frequency domain masking in training deepfake detectors. Based on empirical analysis, we propose a novel deepfake detector via frequency masking. Our focus on frequency domain is different from the majority, which primarily target spatial domain detection. Our comparative analyses reveal substantial performance gains over existing methods. Code and models are publicly available.

Title: Utilizing Layout Effects for Analog Logic Locking

• Authors: Muayad J. Aljafar, Florence Azais, Marie-Lise Flottes, Samuel Pagliarini
• Subjects: cs.CR, cs.AR
• Abstract URL: https://arxiv.org/abs/2401.06508
• Pdf URL: https://arxiv.org/pdf/2401.06508
• Copy Paste: [[2401.06508]] Utilizing Layout Effects for Analog Logic Locking(https://arxiv.org/abs/2401.06508)
• Keywords: protect, diffusion
• Abstract: While numerous obfuscation techniques are available for securing digital assets in the digital domain, there has been a notable lack of focus on protecting Intellectual Property (IP) in the analog domain. This is primarily due to the relatively smaller footprint of analog components within an Integrated Circuit (IC), with the majority of the surface dedicated to digital elements. However, despite their smaller nature, analog components are highly valuable IP and warrant effective protection. In this paper, we present a groundbreaking method for safeguarding analog IP by harnessing layout-based effects that are typically considered undesirable in IC design. Specifically, we exploit the impact of Length of Oxide Diffusion and Well Proximity Effect on transistors to fine-tune critical parameters such as transconductance (gm) and threshold voltage (Vth). These parameters remain concealed behind key inputs, akin to the logic locking approach employed in digital ICs. Our research explores the application of layout-based effects in two commercial CMOS technologies, namely a 28nm and a 65nm node. To demonstrate the efficacy of our proposed technique, we implement it for locking an Operational Transconductance Amplifier. Extensive simulations are performed, evaluating the obfuscation strength by applying a large number of key sets (over 50,000 and 300,000). The results exhibit a significant degradation in performance metrics, such as open-loop gain (up to 130dB), phase margin (up to 50 degrees), 3dB bandwidth (approximately 2.5MHz), and power consumption (around 1mW) when incorrect keys are employed. Our findings highlight the advantages of our approach as well as the associated overhead.

Title: AntEval: Quantitatively Evaluating Informativeness and Expressiveness of Agent Social Interactions

• Authors: Yuanzhi Liang, Linchao Zhu, Yi Yang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.06509
• Pdf URL: https://arxiv.org/pdf/2401.06509
• Copy Paste: [[2401.06509]] AntEval: Quantitatively Evaluating Informativeness and Expressiveness of Agent Social Interactions(https://arxiv.org/abs/2401.06509)
• Keywords: privacy, large language model
• Abstract: While Large Language Models (LLMs) based agents have successfully mimicked human behaviors in various scenarios, the realm of complex, multi-character social interactions within extended contexts remains underexplored. The challenge is compounded by privacy concerns, making it difficult to capture and utilize intricate real-life interactions. More importantly, the absence of quantitative evaluation methods hampers the pursuit of high-quality agent interactions, often leading to interactions that are limited in informativeness and expressiveness, characterized by superficial small talk without clear intentions. In this work, we leverage the rules of Tabletop Role-Playing Games (TRPG) to create an environment conducive to complex, context-rich interactions, emphasizing informativeness and expressiveness. This virtual setting alleviates privacy concerns and motivates agents to engage in meaningful, high-quality interactions as part of their in-game objectives. To assess these interactions, we introduce the Agent interaction Evaluation framework (AntEval), targeting the qualitative evaluation of interaction informativeness and expressiveness. Specifically, we propose two novel evaluation metrics: Information Exchanging Precision (IEP) and Interaction Expressiveness Gap (IEG). These metrics are designed to assess interactions in scenarios focused on information exchange and intention expression, respectively. Our experimental results demonstrate the effectiveness of these metrics in evaluating interaction quality. Notably, we identify significant areas for improvement in LLMs regarding social interactions, as highlighted by our metrics. We believe AntEval will guide further exploration in complex agent interactions, bringing them closer to emulating real human behavior and enhancing their integration and utility in real-world applications.

Title: Personalized Reinforcement Learning with a Budget of Policies

• Authors: Dmitry Ivanov, Omer Ben-Porat
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.06514
• Pdf URL: https://arxiv.org/pdf/2401.06514
• Copy Paste: [[2401.06514]] Personalized Reinforcement Learning with a Budget of Policies(https://arxiv.org/abs/2401.06514)
• Keywords: robust
• Abstract: Personalization in machine learning (ML) tailors models' decisions to the individual characteristics of users. While this approach has seen success in areas like recommender systems, its expansion into high-stakes fields such as healthcare and autonomous driving is hindered by the extensive regulatory approval processes involved. To address this challenge, we propose a novel framework termed represented Markov Decision Processes (r-MDPs) that is designed to balance the need for personalization with the regulatory constraints. In an r-MDP, we cater to a diverse user population, each with unique preferences, through interaction with a small set of representative policies. Our objective is twofold: efficiently match each user to an appropriate representative policy and simultaneously optimize these policies to maximize overall social welfare. We develop two deep reinforcement learning algorithms that efficiently solve r-MDPs. These algorithms draw inspiration from the principles of classic K-means clustering and are underpinned by robust theoretical foundations. Our empirical investigations, conducted across a variety of simulated environments, showcase the algorithms' ability to facilitate meaningful personalization even under constrained policy budgets. Furthermore, they demonstrate scalability, efficiently adapting to larger policy budgets.

Title: Exploring Diverse Representations for Open Set Recognition

• Authors: Yu Wang, Junxian Mu, Pengfei Zhu, Qinghua Hu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.06521
• Pdf URL: https://arxiv.org/pdf/2401.06521
• Copy Paste: [[2401.06521]] Exploring Diverse Representations for Open Set Recognition(https://arxiv.org/abs/2401.06521)
• Keywords: generative
• Abstract: Open set recognition (OSR) requires the model to classify samples that belong to closed sets while rejecting unknown samples during test. Currently, generative models often perform better than discriminative models in OSR, but recent studies show that generative models may be computationally infeasible or unstable on complex tasks. In this paper, we provide insights into OSR and find that learning supplementary representations can theoretically reduce the open space risk. Based on the analysis, we propose a new model, namely Multi-Expert Diverse Attention Fusion (MEDAF), that learns diverse representations in a discriminative way. MEDAF consists of multiple experts that are learned with an attention diversity regularization term to ensure the attention maps are mutually different. The logits learned by each expert are adaptively fused and used to identify the unknowns through the score function. We show that the differences in attention maps can lead to diverse representations so that the fused representations can well handle the open space. Extensive experiments are conducted on standard and OSR large-scale benchmarks. Results show that the proposed discriminative method can outperform existing generative models by up to 9.5% on AUROC and achieve new state-of-the-art performance with little computational cost. Our method can also seamlessly integrate existing classification models. Code is available at https://github.com/Vanixxz/MEDAF.

Title: Domain Adaptation for Time series Transformers using One-step fine-tuning

• Authors: Subina Khanal, Seshu Tirupathi, Giulio Zizzo, Ambrish Rawat, Torben Bach Pedersen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.06524
• Pdf URL: https://arxiv.org/pdf/2401.06524
• Copy Paste: [[2401.06524]] Domain Adaptation for Time series Transformers using One-step fine-tuning(https://arxiv.org/abs/2401.06524)
• Keywords: robust, transformer
• Abstract: The recent breakthrough of Transformers in deep learning has drawn significant attention of the time series community due to their ability to capture long-range dependencies. However, like other deep learning models, Transformers face limitations in time series prediction, including insufficient temporal understanding, generalization challenges, and data shift issues for the domains with limited data. Additionally, addressing the issue of catastrophic forgetting, where models forget previously learned information when exposed to new data, is another critical aspect that requires attention in enhancing the robustness of Transformers for time series tasks. To address these limitations, in this paper, we pre-train the time series Transformer model on a source domain with sufficient data and fine-tune it on the target domain with limited data. We introduce the \emph{One-step fine-tuning} approach, adding some percentage of source domain data to the target domains, providing the model with diverse time series instances. We then fine-tune the pre-trained model using a gradual unfreezing technique. This helps enhance the model's performance in time series prediction for domains with limited data. Extensive experimental results on two real-world datasets show that our approach improves over the state-of-the-art baselines by 4.35% and 11.54% for indoor temperature and wind power prediction, respectively.

Title: MetaHate: A Dataset for Unifying Efforts on Hate Speech Detection

• Authors: Paloma Piot, Patricia Martín-Rodilla, Javier Parapar
• Subjects: cs.CL, cs.SI
• Abstract URL: https://arxiv.org/abs/2401.06526
• Pdf URL: https://arxiv.org/pdf/2401.06526
• Copy Paste: [[2401.06526]] MetaHate: A Dataset for Unifying Efforts on Hate Speech Detection(https://arxiv.org/abs/2401.06526)
• Keywords: robust
• Abstract: Hate speech represents a pervasive and detrimental form of online discourse, often manifested through an array of slurs, from hateful tweets to defamatory posts. As such speech proliferates, it connects people globally and poses significant social, psychological, and occasionally physical threats to targeted individuals and communities. Current computational linguistic approaches for tackling this phenomenon rely on labelled social media datasets for training. For unifying efforts, our study advances in the critical need for a comprehensive meta-collection, advocating for an extensive dataset to help counteract this problem effectively. We scrutinized over 60 datasets, selectively integrating those pertinent into MetaHate. This paper offers a detailed examination of existing collections, highlighting their strengths and limitations. Our findings contribute to a deeper understanding of the existing datasets, paving the way for training more robust and adaptable models. These enhanced models are essential for effectively combating the dynamic and complex nature of hate speech in the digital realm.

Title: INTERS: Unlocking the Power of Large Language Models in Search with Instruction Tuning

• Authors: Yutao Zhu, Peitian Zhang, Chenghao Zhang, Yifei Chen, Binyu Xie, Zhicheng Dou, Zheng Liu, Ji-Rong Wen
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2401.06532
• Pdf URL: https://arxiv.org/pdf/2401.06532
• Copy Paste: [[2401.06532]] INTERS: Unlocking the Power of Large Language Models in Search with Instruction Tuning(https://arxiv.org/abs/2401.06532)
• Keywords: large language model
• Abstract: Large language models (LLMs) have demonstrated impressive capabilities in various natural language processing tasks. Despite this, their application to information retrieval (IR) tasks is still challenging due to the infrequent occurrence of many IR-specific concepts in natural language. While prompt-based methods can provide task descriptions to LLMs, they often fall short in facilitating comprehensive understanding and execution of IR tasks, thereby limiting LLMs' applicability. To address this gap, in this work, we explore the potential of instruction tuning to enhance LLMs' proficiency in IR tasks. We introduce a novel instruction tuning dataset, INTERS, encompassing 21 tasks across three fundamental IR categories: query understanding, document understanding, and query-document relationship understanding. The data are derived from 43 distinct datasets with manually written templates. Our empirical results reveal that INTERS significantly boosts the performance of various publicly available LLMs, such as LLaMA, Mistral, and Phi, in search-related tasks. Furthermore, we conduct a comprehensive analysis to ascertain the effects of base model selection, instruction design, volume of instructions, and task variety on performance. We make our dataset and the models fine-tuned on it publicly accessible at https://github.com/DaoD/INTERS.

Title: Robustness-Aware 3D Object Detection in Autonomous Driving: A Review and Outlook

• Authors: Ziying Song, Lin Liu, Feiyang Jia, Yadan Luo, Guoxin Zhang, Lei Yang, Li Wang, Caiyan Jia
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.06542
• Pdf URL: https://arxiv.org/pdf/2401.06542
• Copy Paste: [[2401.06542]] Robustness-Aware 3D Object Detection in Autonomous Driving: A Review and Outlook(https://arxiv.org/abs/2401.06542)
• Keywords: robust, fair
• Abstract: In the realm of modern autonomous driving, the perception system is indispensable for accurately assessing the state of the surrounding environment, thereby enabling informed prediction and planning. Key to this system is 3D object detection methods, that utilize vehicle-mounted sensors such as LiDAR and cameras to identify the size, category, and location of nearby objects. Despite the surge in 3D object detection methods aimed at enhancing detection precision and efficiency, there is a gap in the literature that systematically examines their resilience against environmental variations, noise, and weather changes. This study emphasizes the importance of robustness, alongside accuracy and latency, in evaluating perception systems under practical scenarios. Our work presents an extensive survey of camera-based, LiDAR-based, and multimodal 3D object detection algorithms, thoroughly evaluating their trade-off between accuracy, latency, and robustness, particularly on datasets like KITTI-C and nuScenes-C to ensure fair comparisons. Among these,multimodal 3D detection approaches exhibit superior robustness and a novel taxonomy is introduced to reorganize its literature for enhanced clarity. This survey aims to offer a more practical perspective on the current capabilities and constraints of 3D object detection algorithms in real-world applications, thus steering future research towards robustness-centric advancements

Title: Optimizing Feature Selection for Binary Classification with Noisy Labels: A Genetic Algorithm Approach

• Authors: Vandad Imani, Elaheh Moradi, Carlos Sevilla-Salcedo, Vittorio Fortino, Jussi Tohka
• Subjects: cs.LG, cs.CV, cs.NE
• Abstract URL: https://arxiv.org/abs/2401.06546
• Pdf URL: https://arxiv.org/pdf/2401.06546
• Copy Paste: [[2401.06546]] Optimizing Feature Selection for Binary Classification with Noisy Labels: A Genetic Algorithm Approach(https://arxiv.org/abs/2401.06546)
• Keywords: interpretability
• Abstract: Feature selection in noisy label scenarios remains an understudied topic. We propose a novel genetic algorithm-based approach, the Noise-Aware Multi-Objective Feature Selection Genetic Algorithm (NMFS-GA), for selecting optimal feature subsets in binary classification with noisy labels. NMFS-GA offers a unified framework for selecting feature subsets that are both accurate and interpretable. We evaluate NMFS-GA on synthetic datasets with label noise, a Breast Cancer dataset enriched with noisy features, and a real-world ADNI dataset for dementia conversion prediction. Our results indicate that NMFS-GA can effectively select feature subsets that improve the accuracy and interpretability of binary classifiers in scenarios with noisy labels.

Title: Enhancing Consistency and Mitigating Bias: A Data Replay Approach for Incremental Learning

• Authors: Chenyang Wang, Junjun Jiang, Xingyu Hu, Xianming Liu, Xiangyang Ji
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.06548
• Pdf URL: https://arxiv.org/pdf/2401.06548
• Copy Paste: [[2401.06548]] Enhancing Consistency and Mitigating Bias: A Data Replay Approach for Incremental Learning(https://arxiv.org/abs/2401.06548)
• Keywords: privacy, data-free
• Abstract: Deep learning systems are prone to catastrophic forgetting when learning from a sequence of tasks, where old data from experienced tasks is unavailable when learning from a new task. To mitigate the problem, a line of methods propose to replay the data of experienced tasks when learning new tasks. These methods usually adopt an extra memory to store the data for replay. However, it is not expected in practice considering the memory constraint or data privacy issue. As a replacement, data-free data replay methods are proposed by inverting samples from the classification model. Though achieving good results, these methods still suffer from the inconsistency of the inverted and real training data, which is neglected in the inversion stage in recent works. To that effect, we propose to measure the data consistency quantitatively by some simplification and assumptions. Using the measurement, we analyze existing techniques for inverting samples and get some insightful information that inspires a novel loss function to reduce the inconsistency. Specifically, the loss minimizes the KL divergence of the distributions of inverted and real data under the tied multivariate Gaussian assumption, which is easy to implement in continual learning. In addition, we observe that the norms of old class weights turn to decrease continually as learning progresses. We thus analyze the underlying reasons and propose a simple regularization term to balance the class weights so that the samples of old classes are more distinguishable. To conclude, we propose the Consistency enhanced data replay with debiased classifier for Class Incremental Learning (CCIL). Extensive experiments on CIFAR-100, Tiny-ImageNet, and ImageNet100 show consistently improved performance of CCIL compared to previous approaches.

Title: Multimodal Learning for detecting urban functional zones using remote sensing image and multi-semantic information

• Authors: Chuanji Shi, Yingying Zhang, Jiaotuan Wang, Qiqi Zhu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.06550
• Pdf URL: https://arxiv.org/pdf/2401.06550
• Copy Paste: [[2401.06550]] Multimodal Learning for detecting urban functional zones using remote sensing image and multi-semantic information(https://arxiv.org/abs/2401.06550)
• Keywords: transformer
• Abstract: Urban area-of-interest (AOI) refers to an integrated urban functional zone with defined boundaries. The rapid development of urban commerce has resulted in an increased demand for more precise requirements in defining AOIs. However, existing research primarily concentrates on broad AOI mining for urban planning or regional economic analysis, failing to cater to the precise requirements of mobile Internet online-to-offline businesses. These businesses necessitate accuracy down to a specific community, school, or hospital. In this paper, we propose an end-to-end multimodal deep learning algorithm for detecting AOI fence polygon using remote sensing images and multi-semantics reference information. We then evaluate its timeliness through a cascaded module that incorporates dynamic human mobility and logistics address information. Specifically, we begin by selecting a point-of-interest (POI) of specific category, and use it to recall corresponding remote sensing images, nearby POIs, road nodes, human mobility, and logistics addresses to build a multimodal detection model based on transformer encoder-decoder architecture, titled AOITR. In the model, in addition to the remote sensing images, multi-semantic information including core POI and road nodes is embedded and reorganized as the query content part for the transformer decoder to generate the AOI polygon. Meanwhile, relatively dynamic distribution features of human mobility, nearby POIs, and logistics addresses are used for AOI reliability evaluation through a cascaded feedforward network. The experimental results demonstrate that our algorithm significantly outperforms two existing methods.

Title: Intention Analysis Prompting Makes Large Language Models A Good Jailbreak Defender

• Authors: Yuqi Zhang, Liang Ding, Lefei Zhang, Dacheng Tao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.06561
• Pdf URL: https://arxiv.org/pdf/2401.06561
• Copy Paste: [[2401.06561]] Intention Analysis Prompting Makes Large Language Models A Good Jailbreak Defender(https://arxiv.org/abs/2401.06561)
• Keywords: defense, attack, steal, large language model
• Abstract: Aligning large language models (LLMs) with human values, particularly in the face of stealthy and complex jailbreaks, presents a formidable challenge. In this study, we present a simple yet highly effective defense strategy, i.e., Intention Analysis Prompting (IAPrompt). The principle behind is to trigger LLMs' inherent self-correct and improve ability through a two-stage process: 1) essential intention analysis, and 2) policy-aligned response. Notably, IAPrompt is an inference-only method, thus could enhance the safety of LLMs without compromising their helpfulness. Extensive experiments on SAP200 and DAN benchmarks across Vicuna, ChatGLM, MPT, DeepSeek, and GPT-3.5 show that IAPrompt could consistently and significantly reduce the harmfulness in response (averagely -46.5% attack success rate) and maintain the general helpfulness. Further analyses present some insights into how our method works. To facilitate reproducibility, We release our code and scripts at: https://github.com/alphadl/SafeLLM_with_IntentionAnalysis

Title: Resource-Efficient Gesture Recognition using Low-Resolution Thermal Camera via Spiking Neural Networks and Sparse Segmentation

• Authors: Ali Safa, Wout Mommen, Lars Keuninckx
• Subjects: cs.CV, cs.HC
• Abstract URL: https://arxiv.org/abs/2401.06563
• Pdf URL: https://arxiv.org/pdf/2401.06563
• Copy Paste: [[2401.06563]] Resource-Efficient Gesture Recognition using Low-Resolution Thermal Camera via Spiking Neural Networks and Sparse Segmentation(https://arxiv.org/abs/2401.06563)
• Keywords: robust, segmentation
• Abstract: This work proposes a novel approach for hand gesture recognition using an inexpensive, low-resolution (24 x 32) thermal sensor processed by a Spiking Neural Network (SNN) followed by Sparse Segmentation and feature-based gesture classification via Robust Principal Component Analysis (R-PCA). Compared to the use of standard RGB cameras, the proposed system is insensitive to lighting variations while being significantly less expensive compared to high-frequency radars, time-of-flight cameras and high-resolution thermal sensors previously used in literature. Crucially, this paper shows that the innovative use of the recently proposed Monostable Multivibrator (MMV) neural networks as a new class of SNN achieves more than one order of magnitude smaller memory and compute complexity compared to deep learning approaches, while reaching a top gesture recognition accuracy of 93.9% using a 5-class thermal camera dataset acquired in a car cabin, within an automotive context. Our dataset is released for helping future research.

Title: Lost in the Source Language: How Large Language Models Evaluate the Quality of Machine Translation

• Authors: Xu Huang, Zhirui Zhang, Xiang Geng, Yichao Du, Jiajun Chen, Shujian Huang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.06568
• Pdf URL: https://arxiv.org/pdf/2401.06568
• Copy Paste: [[2401.06568]] Lost in the Source Language: How Large Language Models Evaluate the Quality of Machine Translation(https://arxiv.org/abs/2401.06568)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have achieved remarkable results in the machine translation evaluation task, yet there remains a gap in knowledge regarding how they utilize the provided data to conduct evaluations. This study aims to explore how LLMs leverage source and reference information in evaluating translations, with the ultimate goal of better understanding the working mechanism of LLMs. To this end, we design the controlled experiments across various input modes and model types, and employ both coarse-grained and fine-grained prompts to discern the utility of source versus reference information. Surprisingly, we find that reference information significantly enhances the evaluation accuracy, while source information sometimes is counterproductive, indicating a lack of cross-lingual capability when using LLMs to evaluate translations. We further conduct a meta-evaluation for translation error detection of LLMs, observing a similar phenomenon. These findings also suggest a potential research direction for LLMs that fully exploits the cross-lingual capability of LLMs to achieve better performance in machine translation evaluation tasks.

Title: 360DVD: Controllable Panorama Video Generation with 360-Degree Video Diffusion Model

• Authors: Qian Wang, Weiqi Li, Chong Mou, Xinhua Cheng, Jian Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.06578
• Pdf URL: https://arxiv.org/pdf/2401.06578
• Copy Paste: [[2401.06578]] 360DVD: Controllable Panorama Video Generation with 360-Degree Video Diffusion Model(https://arxiv.org/abs/2401.06578)
• Keywords: diffusion
• Abstract: 360-degree panoramic videos recently attract more interest in both studies and applications, courtesy of the heightened immersive experiences they engender. Due to the expensive cost of capturing 360-degree panoramic videos, generating desirable panoramic videos by given prompts is urgently required. Recently, the emerging text-to-video (T2V) diffusion methods demonstrate notable effectiveness in standard video generation. However, due to the significant gap in content and motion patterns between panoramic and standard videos, these methods encounter challenges in yielding satisfactory 360-degree panoramic videos. In this paper, we propose a controllable panorama video generation pipeline named 360-Degree Video Diffusion model (360DVD) for generating panoramic videos based on the given prompts and motion conditions. Concretely, we introduce a lightweight module dubbed 360-Adapter and assisted 360 Enhancement Techniques to transform pre-trained T2V models for 360-degree video generation. We further propose a new panorama dataset named WEB360 consisting of 360-degree video-text pairs for training 360DVD, addressing the absence of captioned panoramic video datasets. Extensive experiments demonstrate the superiority and effectiveness of 360DVD for panorama video generation. The code and dataset will be released soon.

Title: Mapping Transformer Leveraged Embeddings for Cross-Lingual Document Representation

• Authors: Tsegaye Misikir Tashu, Eduard-Raul Kontos, Matthia Sabatelli, Matias Valdenegro-Toro
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.06583
• Pdf URL: https://arxiv.org/pdf/2401.06583
• Copy Paste: [[2401.06583]] Mapping Transformer Leveraged Embeddings for Cross-Lingual Document Representation(https://arxiv.org/abs/2401.06583)
• Keywords: transformer
• Abstract: Recommendation systems, for documents, have become tools to find relevant content on the Web. However, these systems have limitations when it comes to recommending documents in languages different from the query language, which means they might overlook resources in non-native languages. This research focuses on representing documents across languages by using Transformer Leveraged Document Representations (TLDRs) that are mapped to a cross-lingual domain. Four multilingual pre-trained transformer models (mBERT, mT5 XLM RoBERTa, ErnieM) were evaluated using three mapping methods across 20 language pairs representing combinations of five selected languages of the European Union. Metrics like Mate Retrieval Rate and Reciprocal Rank were used to measure the effectiveness of mapped TLDRs compared to non-mapped ones. The results highlight the power of cross-lingual representations achieved through pre-trained transformers and mapping approaches suggesting a promising direction for expanding beyond language connections, between two specific languages.

Title: A proposal to increase data utility on Global Differential Privacy data based on data use predictions

• Authors: Henry C. Nunes, Marlon P. da Silva, Charles V. Neu, Avelino F. Zorzo
• Subjects: cs.CR, cs.DB
• Abstract URL: https://arxiv.org/abs/2401.06601
• Pdf URL: https://arxiv.org/pdf/2401.06601
• Copy Paste: [[2401.06601]] A proposal to increase data utility on Global Differential Privacy data based on data use predictions(https://arxiv.org/abs/2401.06601)
• Keywords: privacy, protect
• Abstract: This paper presents ongoing research focused on improving the utility of data protected by Global Differential Privacy(DP) in the scenario of summary statistics. Our approach is based on predictions on how an analyst will use statistics released under DP protection, so that a developer can optimise data utility on further usage of the data in the privacy budget allocation. This novel approach can potentially improve the utility of data without compromising privacy constraints. We also propose a metric that can be used by the developer to optimise the budget allocation process.

Title: Mutual Enhancement of Large Language and Reinforcement Learning Models through Bi-Directional Feedback Mechanisms: A Case Study

• Authors: Shangding Gu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.06603
• Pdf URL: https://arxiv.org/pdf/2401.06603
• Copy Paste: [[2401.06603]] Mutual Enhancement of Large Language and Reinforcement Learning Models through Bi-Directional Feedback Mechanisms: A Case Study(https://arxiv.org/abs/2401.06603)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities for reinforcement learning (RL) models, such as planning and reasoning capabilities. However, the problems of LLMs and RL model collaboration still need to be solved. In this study, we employ a teacher-student learning framework to tackle these problems, specifically by offering feedback for LLMs using RL models and providing high-level information for RL models with LLMs in a cooperative multi-agent setting. Within this framework, the LLM acts as a teacher, while the RL model acts as a student. The two agents cooperatively assist each other through a process of recursive help, such as "I help you help I help." The LLM agent supplies abstract information to the RL agent, enabling efficient exploration and policy improvement. In turn, the RL agent offers feedback to the LLM agent, providing valuable, real-time information that helps generate more useful tokens. This bi-directional feedback loop promotes optimization, exploration, and mutual improvement for both agents, enabling them to accomplish increasingly challenging tasks. Remarkably, we propose a practical algorithm to address the problem and conduct empirical experiments to evaluate the effectiveness of our method.

Title: Leveraging Machine Learning for Wi-Fi-based Environmental Continuous Two-Factor Authentication

• Authors: Ali Abdullah S. AlQahtani, Thamraa Alshayeb, Mahmoud Nabil, Ahmad Patooghy
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2401.06612
• Pdf URL: https://arxiv.org/pdf/2401.06612
• Copy Paste: [[2401.06612]] Leveraging Machine Learning for Wi-Fi-based Environmental Continuous Two-Factor Authentication(https://arxiv.org/abs/2401.06612)
• Keywords: secure, security, attack, robust
• Abstract: The traditional two-factor authentication (2FA) methods primarily rely on the user manually entering a code or token during the authentication process. This can be burdensome and time-consuming, particularly for users who must be authenticated frequently. To tackle this challenge, we present a novel 2FA approach replacing the user's input with decisions made by Machine Learning (ML) that continuously verifies the user's identity with zero effort. Our system exploits unique environmental features associated with the user, such as beacon frame characteristics and Received Signal Strength Indicator (RSSI) values from Wi-Fi Access Points (APs). These features are gathered and analyzed in real-time by our ML algorithm to ascertain the user's identity. For enhanced security, our system mandates that the user's two devices (i.e., a login device and a mobile device) be situated within a predetermined proximity before granting access. This precaution ensures that unauthorized users cannot access sensitive information or systems, even with the correct login credentials. Through experimentation, we have demonstrated our system's effectiveness in determining the location of the user's devices based on beacon frame characteristics and RSSI values, achieving an accuracy of 92.4%. Additionally, we conducted comprehensive security analysis experiments to evaluate the proposed 2FA system's resilience against various cyberattacks. Our findings indicate that the system exhibits robustness and reliability in the face of these threats. The scalability, flexibility, and adaptability of our system render it a promising option for organizations and users seeking a secure and convenient authentication system.

Title: Motion2VecSets: 4D Latent Vector Set Diffusion for Non-rigid Shape Reconstruction and Tracking

• Authors: Wei Cao, Chang Luo, Biao Zhang, Matthias Nießner, Jiapeng Tang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.06614
• Pdf URL: https://arxiv.org/pdf/2401.06614
• Copy Paste: [[2401.06614]] Motion2VecSets: 4D Latent Vector Set Diffusion for Non-rigid Shape Reconstruction and Tracking(https://arxiv.org/abs/2401.06614)
• Keywords: diffusion
• Abstract: We introduce Motion2VecSets, a 4D diffusion model for dynamic surface reconstruction from point cloud sequences. While existing state-of-the-art methods have demonstrated success in reconstructing non-rigid objects using neural field representations, conventional feed-forward networks encounter challenges with ambiguous observations from noisy, partial, or sparse point clouds. To address these challenges, we introduce a diffusion model that explicitly learns the shape and motion distribution of non-rigid objects through an iterative denoising process of compressed latent representations. The diffusion-based prior enables more plausible and probabilistic reconstructions when handling ambiguous inputs. We parameterize 4D dynamics with latent vector sets instead of using a global latent. This novel 4D representation allows us to learn local surface shape and deformation patterns, leading to more accurate non-linear motion capture and significantly improving generalizability to unseen motions and identities. For more temporal-coherent object tracking, we synchronously denoise deformation latent sets and exchange information across multiple frames. To avoid the computational overhead, we design an interleaved space and time attention block to alternately aggregate deformation latents along spatial and temporal domains. Extensive comparisons against the state-of-the-art methods demonstrate the superiority of our Motion2VecSets in 4D reconstruction from various imperfect observations, notably achieving a 19% improvement in Intersection over Union (IoU) compared to CaDex for reconstructing unseen individuals from sparse point clouds on the DeformingThings4D-Animals dataset. More detailed information can be found at https://vveicao.github.io/projects/Motion2VecSets/.

Title: Software-Based Memory Erasure with relaxed isolation requirements: Extended Version

• Authors: Sergiu Bursuc, Reynaldo Gil-Pons, Sjouke Mauw, Rolando Trujillo-Rasua
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2401.06626
• Pdf URL: https://arxiv.org/pdf/2401.06626
• Copy Paste: [[2401.06626]] Software-Based Memory Erasure with relaxed isolation requirements: Extended Version(https://arxiv.org/abs/2401.06626)
• Keywords: secure, protect
• Abstract: A Proof of Secure Erasure (PoSE) is a communication protocol where a verifier seeks evidence that a prover has erased its memory within the time frame of the protocol execution. Designers of PoSE protocols have long been aware that, if a prover can outsource the computation of the memory erasure proof to another device, then their protocols are trivially defeated. As a result, most software-based PoSE protocols in the literature assume that provers are isolated during the protocol execution, that is, provers cannot receive help from a network adversary. Our main contribution is to show that this assumption is not necessary. We introduce formal models for PoSE protocols playing against provers aided by external conspirators and develop three PoSE protocols that we prove secure in this context. We reduce the requirement of isolation to the more realistic requirement that the communication with the external conspirator is relatively slow. Software-based protocols with such relaxed isolation assumptions are especially pertinent for low-end devices, where it is too costly to deploy sophisticated protection methods.

Title: OOP: Object-Oriented Programming Evaluation Benchmark for Large Language Models

• Authors: Shuai Wang, Liang Ding, Li Shen, Yong Luo, Bo Du, Dacheng Tao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.06628
• Pdf URL: https://arxiv.org/pdf/2401.06628
• Copy Paste: [[2401.06628]] OOP: Object-Oriented Programming Evaluation Benchmark for Large Language Models(https://arxiv.org/abs/2401.06628)
• Keywords: robust, large language model
• Abstract: Advancing automated programming necessitates robust and comprehensive code generation benchmarks, yet current evaluation frameworks largely neglect object-oriented programming (OOP) in favor of functional programming (FP), e.g., HumanEval and MBPP. To address this, our study introduces a pioneering OOP-focused benchmark, featuring 431 Python programs that encompass essential OOP concepts and features like classes and encapsulation methods. We propose a novel evaluation metric, pass@o, tailored for OOP, enhancing traditional pass@k measures. Our evaluation of 23 leading large language models (LLMs), including both general and code-specialized models, reveals three key insights: 1) pass@o offers a more relevant and comprehensive assessment for OOP code generation; 2) Despite excelling in FP, code-specialized LLMs like WizardCoder lag in OOP compared to models like ChatGPT; 3) The poor performance of all advanced LLMs on our OOP benchmark highlights a critical need for improvements in this field. Our benchmark and scripts are publicly released at: https://github.com/alphadl/OOP-eval.

Title: CCFC: Bridging Federated Clustering and Contrastive Learning

• Authors: Jie Yan, Jing Liu, Zhong-Yuan Zhang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.06634
• Pdf URL: https://arxiv.org/pdf/2401.06634
• Copy Paste: [[2401.06634]] CCFC: Bridging Federated Clustering and Contrastive Learning(https://arxiv.org/abs/2401.06634)
• Keywords: federate
• Abstract: Federated clustering, an essential extension of centralized clustering for federated scenarios, enables multiple data-holding clients to collaboratively group data while keeping their data locally. In centralized scenarios, clustering driven by representation learning has made significant advancements in handling high-dimensional complex data. However, the combination of federated clustering and representation learning remains underexplored. To bridge this, we first tailor a cluster-contrastive model for learning clustering-friendly representations. Then, we harness this model as the foundation for proposing a new federated clustering method, named cluster-contrastive federated clustering (CCFC). Benefiting from representation learning, the clustering performance of CCFC even double those of the best baseline methods in some cases. Compared to the most related baseline, the benefit results in substantial NMI score improvements of up to 0.4155 on the most conspicuous case. Moreover, CCFC also shows superior performance in handling device failures from a practical viewpoint.

Title: Adversarial Examples are Misaligned in Diffusion Model Manifolds

• Authors: Peter Lorenz, Ricard Durall, Jansi Keuper
• Subjects: cs.CV, cs.CR
• Abstract URL: https://arxiv.org/abs/2401.06637
• Pdf URL: https://arxiv.org/pdf/2401.06637
• Copy Paste: [[2401.06637]] Adversarial Examples are Misaligned in Diffusion Model Manifolds(https://arxiv.org/abs/2401.06637)
• Keywords: attack, robust, diffusion, generative, segmentation
• Abstract: In recent years, diffusion models (DMs) have drawn significant attention for their success in approximating data distributions, yielding state-of-the-art generative results. Nevertheless, the versatility of these models extends beyond their generative capabilities to encompass various vision applications, such as image inpainting, segmentation, adversarial robustness, among others. This study is dedicated to the investigation of adversarial attacks through the lens of diffusion models. However, our objective does not involve enhancing the adversarial robustness of image classifiers. Instead, our focus lies in utilizing the diffusion model to detect and analyze the anomalies introduced by these attacks on images. To that end, we systematically examine the alignment of the distributions of adversarial examples when subjected to the process of transformation using diffusion models. The efficacy of this approach is assessed across CIFAR-10 and ImageNet datasets, including varying image sizes in the latter. The results demonstrate a notable capacity to discriminate effectively between benign and attacked images, providing compelling evidence that adversarial instances do not align with the learned manifold of the DMs.

Title: Experimental Contexts Can Facilitate Robust Semantic Property Inference in Language Models, but Inconsistently

• Authors: Kanishka Misra, Allyson Ettinger, Kyle Mahowald
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.06640
• Pdf URL: https://arxiv.org/pdf/2401.06640
• Copy Paste: [[2401.06640]] Experimental Contexts Can Facilitate Robust Semantic Property Inference in Language Models, but Inconsistently(https://arxiv.org/abs/2401.06640)
• Keywords: robust, extraction
• Abstract: Recent zero-shot evaluations have highlighted important limitations in the abilities of language models (LMs) to perform meaning extraction. However, it is now well known that LMs can demonstrate radical improvements in the presence of experimental contexts such as in-context examples and instructions. How well does this translate to previously studied meaning-sensitive tasks? We present a case-study on the extent to which experimental contexts can improve LMs' robustness in performing property inheritance -- predicting semantic properties of novel concepts, a task that they have been previously shown to fail on. Upon carefully controlling the nature of the in-context examples and the instructions, our work reveals that they can indeed lead to non-trivial property inheritance behavior in LMs. However, this ability is inconsistent: with a minimal reformulation of the task, some LMs were found to pick up on shallow, non-semantic heuristics from their inputs, suggesting that the computational principles of semantic property inference are yet to be mastered by LMs.

Title: Effects of diversity incentives on sample diversity and downstream model performance in LLM-based text augmentation

• Authors: Jan Cegin, Branislav Pecher, Jakub Simko, Ivan Srba, Maria Bielikova, Peter Brusilovsky
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.06643
• Pdf URL: https://arxiv.org/pdf/2401.06643
• Copy Paste: [[2401.06643]] Effects of diversity incentives on sample diversity and downstream model performance in LLM-based text augmentation(https://arxiv.org/abs/2401.06643)
• Keywords: generative, large language model
• Abstract: The latest generative large language models (LLMs) have found their application in data augmentation tasks, where small numbers of text samples are LLM-paraphrased and then used to fine-tune the model. However, more research is needed to assess how different prompts, seed data selection strategies, filtering methods, or model settings affect the quality of paraphrased data (and downstream models). In this study, we investigate three text diversity incentive methods well established in crowdsourcing: taboo words, hints by previous outlier solutions, and chaining on previous outlier solutions. Using these incentive methods as part of instructions to LLMs augmenting text datasets, we measure their effects on generated texts' lexical diversity and downstream model performance. We compare the effects over 5 different LLMs and 6 datasets. We show that diversity is most increased by taboo words, while downstream model performance is highest when previously created paraphrases are used as hints.

Title: SeizNet: An AI-enabled Implantable Sensor Network System for Seizure Prediction

• Authors: Ali Saeizadeh, Douglas Schonholtz, Daniel Uvaydov, Raffaele Guida, Emrecan Demirors, Pedram Johari, Jorge M. Jimenez, Joseph S. Neimat, Tommaso Melodia
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2401.06644
• Pdf URL: https://arxiv.org/pdf/2401.06644
• Copy Paste: [[2401.06644]] SeizNet: An AI-enabled Implantable Sensor Network System for Seizure Prediction(https://arxiv.org/abs/2401.06644)
• Keywords: privacy
• Abstract: In this paper, we introduce SeizNet, a closed-loop system for predicting epileptic seizures through the use of Deep Learning (DL) method and implantable sensor networks. While pharmacological treatment is effective for some epilepsy patients (with ~65M people affected worldwide), one out of three suffer from drug-resistant epilepsy. To alleviate the impact of seizure, predictive systems have been developed that can notify such patients of an impending seizure, allowing them to take precautionary measures. SeizNet leverages DL techniques and combines data from multiple recordings, specifically intracranial electroencephalogram (iEEG) and electrocardiogram (ECG) sensors, that can significantly improve the specificity of seizure prediction while preserving very high levels of sensitivity. SeizNet DL algorithms are designed for efficient real-time execution at the edge, minimizing data privacy concerns, data transmission overhead, and power inefficiencies associated with cloud-based solutions. Our results indicate that SeizNet outperforms traditional single-modality and non-personalized prediction systems in all metrics, achieving up to 99% accuracy in predicting seizure, offering a promising new avenue in refractory epilepsy treatment.

Title: Decoupling Pixel Flipping and Occlusion Strategy for Consistent XAI Benchmarks

• Authors: Stefan Blücher, Johanna Vielhaben, Nils Strodthoff
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.06654
• Pdf URL: https://arxiv.org/pdf/2401.06654
• Copy Paste: [[2401.06654]] Decoupling Pixel Flipping and Occlusion Strategy for Consistent XAI Benchmarks(https://arxiv.org/abs/2401.06654)
• Keywords: diffusion
• Abstract: Feature removal is a central building block for eXplainable AI (XAI), both for occlusion-based explanations (Shapley values) as well as their evaluation (pixel flipping, PF). However, occlusion strategies can vary significantly from simple mean replacement up to inpainting with state-of-the-art diffusion models. This ambiguity limits the usefulness of occlusion-based approaches. For example, PF benchmarks lead to contradicting rankings. This is amplified by competing PF measures: Features are either removed starting with most influential first (MIF) or least influential first (LIF). This study proposes two complementary perspectives to resolve this disagreement problem. Firstly, we address the common criticism of occlusion-based XAI, that artificial samples lead to unreliable model evaluations. We propose to measure the reliability by the R(eference)-Out-of-Model-Scope (OMS) score. The R-OMS score enables a systematic comparison of occlusion strategies and resolves the disagreement problem by grouping consistent PF rankings. Secondly, we show that the insightfulness of MIF and LIF is conversely dependent on the R-OMS score. To leverage this, we combine the MIF and LIF measures into the symmetric relevance gain (SRG) measure. This breaks the inherent connection to the underlying occlusion strategy and leads to consistent rankings. This resolves the disagreement problem, which we verify for a set of 40 different occlusion strategies.

Title: Accelerating Tactile Internet with QUIC: A Security and Privacy Perspective

• Authors: Jayasree Sengupta, Debasmita Dey, Simone Ferlin, Nirnay Ghosh, Vaibhav Bajpai
• Subjects: cs.CR, cs.NI
• Abstract URL: https://arxiv.org/abs/2401.06657
• Pdf URL: https://arxiv.org/pdf/2401.06657
• Copy Paste: [[2401.06657]] Accelerating Tactile Internet with QUIC: A Security and Privacy Perspective(https://arxiv.org/abs/2401.06657)
• Keywords: security, privacy, attack
• Abstract: The Tactile Internet paradigm is set to revolutionize human society by enabling skill-set delivery and haptic communication over ultra-reliable, low-latency networks. The emerging sixth-generation (6G) mobile communication systems are envisioned to underpin this Tactile Internet ecosystem at the network edge by providing ubiquitous global connectivity. However, apart from a multitude of opportunities of the Tactile Internet, security and privacy challenges emerge at the forefront. We believe that the recently standardized QUIC protocol, characterized by end-to-end encryption and reduced round-trip delay would serve as the backbone of Tactile Internet. In this article, we envision a futuristic scenario where a QUIC-enabled network uses the underlying 6G communication infrastructure to achieve the requirements for Tactile Internet. Interestingly this requires a deeper investigation of a wide range of security and privacy challenges in QUIC, that need to be mitigated for its adoption in Tactile Internet. Henceforth, this article reviews the existing security and privacy attacks in QUIC and their implication on users. Followed by that, we discuss state-of-the-art attack mitigation strategies and investigate some of their drawbacks with possible directions for future work

Title: Proximal Causal Inference With Text Data

• Authors: Jacob M. Chen, Rohit Bhattacharya, Katherine A. Keith
• Subjects: cs.CL, cs.LG, stat.ME
• Abstract URL: https://arxiv.org/abs/2401.06687
• Pdf URL: https://arxiv.org/pdf/2401.06687
• Copy Paste: [[2401.06687]] Proximal Causal Inference With Text Data(https://arxiv.org/abs/2401.06687)
• Keywords: privacy
• Abstract: Recent text-based causal methods attempt to mitigate confounding bias by including unstructured text data as proxies of confounding variables that are partially or imperfectly measured. These approaches assume analysts have supervised labels of the confounders given text for a subset of instances, a constraint that is not always feasible due to data privacy or cost. Here, we address settings in which an important confounding variable is completely unobserved. We propose a new causal inference method that splits pre-treatment text data, infers two proxies from two zero-shot models on the separate splits, and applies these proxies in the proximal g-formula. We prove that our text-based proxy method satisfies identification conditions required by the proximal g-formula while other seemingly reasonable proposals do not. We evaluate our method in synthetic and semi-synthetic settings and find that it produces estimates with low bias. This combination of proximal causal inference and zero-shot classifiers is novel (to our knowledge) and expands the set of text-specific causal methods available to practitioners.

Title: Don't Rank, Combine! Combining Machine Translation Hypotheses Using Quality Estimation

• Authors: Giorgos Vernikos, Andrei Popescu-Belis
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.06688
• Pdf URL: https://arxiv.org/pdf/2401.06688
• Copy Paste: [[2401.06688]] Don't Rank, Combine! Combining Machine Translation Hypotheses Using Quality Estimation(https://arxiv.org/abs/2401.06688)
• Keywords: large language model
• Abstract: Neural machine translation systems estimate probabilities of target sentences given source sentences, yet these estimates may not align with human preferences. This work introduces QE-fusion, a method utilizing a quality estimation metric (QE) that better correlates with human judgments to synthesize improved translations. QE-fusion leverages a candidate pool sampled from a model, combining spans from different candidates using QE metrics such as CometKiwi. We compare QE-fusion against beam search and recent reranking techniques, such as Minimum Bayes Risk decoding or QE-reranking. Our method consistently improves translation quality in terms of COMET and BLEURT scores when applied to large language models (LLMs) used for translation (PolyLM, XGLM, Llama2, and Mistral) and to multilingual translation models (NLLB), over five language pairs. Notably, QE-fusion exhibits larger improvements for LLMs due to their ability to generate diverse outputs. We demonstrate that our approach generates novel translations in over half of the cases and consistently outperforms other methods across varying numbers of candidates (5-200). Furthermore, we empirically establish that QE-fusion scales linearly with the number of candidates in the pool. QE-fusion proves effective in enhancing LLM-based translation without the need for costly retraining of LLMs.

Title: Embedded Planogram Compliance Control System

• Authors: M. Erkin Yücel, Serkan Topaloğlu, Cem Ünsalan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.06690
• Pdf URL: https://arxiv.org/pdf/2401.06690
• Copy Paste: [[2401.06690]] Embedded Planogram Compliance Control System(https://arxiv.org/abs/2401.06690)
• Keywords: extraction
• Abstract: The retail sector presents several open and challenging problems that could benefit from advanced pattern recognition and computer vision techniques. One such critical challenge is planogram compliance control. In this study, we propose a complete embedded system to tackle this issue. Our system consists of four key components as image acquisition and transfer via stand-alone embedded camera module, object detection via computer vision and deep learning methods working on single board computers, planogram compliance control method again working on single board computers, and energy harvesting and power management block to accompany the embedded camera modules. The image acquisition and transfer block is implemented on the ESP-EYE camera module. The object detection block is based on YOLOv5 as the deep learning method and local feature extraction. We implement these methods on Raspberry Pi 4, NVIDIA Jetson Orin Nano, and NVIDIA Jetson AGX Orin as single board computers. The planogram compliance control block utilizes sequence alignment through a modified Needleman-Wunsch algorithm. This block is also working along with the object detection block on the same single board computers. The energy harvesting and power management block consists of solar and RF energy harvesting modules with suitable battery pack for operation. We tested the proposed embedded planogram compliance control system on two different datasets to provide valuable insights on its strengths and weaknesses. The results show that our method achieves F1 scores of 0.997 and 1.0 in object detection and planogram compliance control blocks, respectively. Furthermore, we calculated that the complete embedded system can work in stand-alone form up to two years based on battery. This duration can be further extended with the integration of the proposed solar and RF energy harvesting options.

Title: An Experimental Design Framework for Label-Efficient Supervised Finetuning of Large Language Models

• Authors: Gantavya Bhatt, Yifang Chen, Arnav M. Das, Jifan Zhang, Sang T. Truong, Stephen Mussmann, Yinglun Zhu, Jeffrey Bilmes, Simon S. Du, Kevin Jamieson, Jordan T. Ash, Robert D. Nowak
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.06692
• Pdf URL: https://arxiv.org/pdf/2401.06692
• Copy Paste: [[2401.06692]] An Experimental Design Framework for Label-Efficient Supervised Finetuning of Large Language Models(https://arxiv.org/abs/2401.06692)
• Keywords: generative, large language model
• Abstract: Supervised finetuning (SFT) on instruction datasets has played a crucial role in achieving the remarkable zero-shot generalization capabilities observed in modern large language models (LLMs). However, the annotation efforts required to produce high quality responses for instructions are becoming prohibitively expensive, especially as the number of tasks spanned by instruction datasets continues to increase. Active learning is effective in identifying useful subsets of samples to annotate from an unlabeled pool, but its high computational cost remains a barrier to its widespread applicability in the context of LLMs. To mitigate the annotation cost of SFT and circumvent the computational bottlenecks of active learning, we propose using experimental design. Experimental design techniques select the most informative samples to label, and typically maximize some notion of uncertainty and/or diversity. In our work, we implement a framework that evaluates several existing and novel experimental design techniques and find that these methods consistently yield significant gains in label efficiency with little computational overhead. On generative tasks, our methods achieve the same generalization performance with only $50\%$ of annotation cost required by random sampling.

Title: Quantum Machine Learning in the Cognitive Domain: Alzheimer's Disease Study

• Authors: Emine Akpinar
• Subjects: cs.LG, quant-ph
• Abstract URL: https://arxiv.org/abs/2401.06697
• Pdf URL: https://arxiv.org/pdf/2401.06697
• Copy Paste: [[2401.06697]] Quantum Machine Learning in the Cognitive Domain: Alzheimer's Disease Study(https://arxiv.org/abs/2401.06697)
• Keywords: generative
• Abstract: Alzheimer's disease (AD) is the most prevalent neurodegenerative brain disorder, which results in significant cognitive impairments, especially in the elderly population. Cognitive impairments can manifest as a decline in various mental faculties, such as concentration, memory, and other higher-order cognitive abilities. These deficits can significantly impact an individual's capacity to comprehend information, acquire new knowledge, and communicate effectively. One of the affected activities due to cognitive impairments is handwriting. By analyzing different aspects of handwriting, including pressure, velocity, and spatial organization, researchers can detect subtle alterations that might indicate early-stage cognitive impairments, especially AD. Recently, several classical artificial intelligence (AI) approaches have been proposed for detecting AD in elderly individuals through handwriting analysis. However, advanced AI methods require more computational power as the size of the data increases. Additionally, diagnoses can be influenced by factors such as limited relevant classical vector space and correlations between features. Recent studies have shown that using quantum computing technologies in healthcare can not only address these problems but also accelerate complex data analysis and process large datasets more efficiently. In this study, we introduced a variational quantum classifier with fewer circuit elements to facilitate the early diagnosis of AD in elderly individuals based on handwriting data. We employed ZZFeatureMap for encoding features. To classify AD, a parameterized quantum circuit consisting of repeated Ry and Rz rotation gates, as well as CY and CZ two-qubit entangling gates, was designed and implemented. The proposed model achieved an accuracy of 0.75 in classifying AD.

Title: Scalable 3D Panoptic Segmentation With Superpoint Graph Clustering

• Authors: Damien Robert, Hugo Raguet, Loic Landrieu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.06704
• Pdf URL: https://arxiv.org/pdf/2401.06704
• Copy Paste: [[2401.06704]] Scalable 3D Panoptic Segmentation With Superpoint Graph Clustering(https://arxiv.org/abs/2401.06704)
• Keywords: transformer, segmentation
• Abstract: We introduce a highly efficient method for panoptic segmentation of large 3D point clouds by redefining this task as a scalable graph clustering problem. This approach can be trained using only local auxiliary tasks, thereby eliminating the resource-intensive instance-matching step during training. Moreover, our formulation can easily be adapted to the superpoint paradigm, further increasing its efficiency. This allows our model to process scenes with millions of points and thousands of objects in a single inference. Our method, called SuperCluster, achieves a new state-of-the-art panoptic segmentation performance for two indoor scanning datasets: $50.1$ PQ ($+7.8$) for S3DIS Area~5, and $58.7$ PQ ($+25.2$) for ScanNetV2. We also set the first state-of-the-art for two large-scale mobile mapping benchmarks: KITTI-360 and DALES. With only $209$k parameters, our model is over $30$ times smaller than the best-competing method and trains up to $15$ times faster. Our code and pretrained models are available at https://github.com/drprojects/superpoint_transformer.

Title: Multi-Candidate Speculative Decoding

• Authors: Sen Yang, Shujian Huang, Xinyu Dai, Jiajun Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.06706
• Pdf URL: https://arxiv.org/pdf/2401.06706
• Copy Paste: [[2401.06706]] Multi-Candidate Speculative Decoding(https://arxiv.org/abs/2401.06706)
• Keywords: large language model
• Abstract: Large language models have shown impressive capabilities across a variety of NLP tasks, yet their generating text autoregressively is time-consuming. One way to speed them up is speculative decoding, which generates candidate segments (a sequence of tokens) from a fast draft model that is then verified in parallel by the target model. However, the acceptance rate of candidate tokens receives limitations from several factors, such as the model, the dataset, and the decoding setup. This paper proposes sampling multiple candidates from a draft model and then organising them in batches for verification. We design algorithms for efficient multi-candidate verification while maintaining the distribution of the target model. Our approach shows significant improvements in acceptance rates on multiple datasets and models, consistently outperforming standard speculative decoding.

Title: Reliability Analysis of Psychological Concept Extraction and Classification in User-penned Text

• Authors: Muskan Garg, MSVPJ Sathvik, Amrit Chadha, Shaina Raza, Sunghwan Sohn
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.06709
• Pdf URL: https://arxiv.org/pdf/2401.06709
• Copy Paste: [[2401.06709]] Reliability Analysis of Psychological Concept Extraction and Classification in User-penned Text(https://arxiv.org/abs/2401.06709)
• Keywords: extraction
• Abstract: The social NLP research community witness a recent surge in the computational advancements of mental health analysis to build responsible AI models for a complex interplay between language use and self-perception. Such responsible AI models aid in quantifying the psychological concepts from user-penned texts on social media. On thinking beyond the low-level (classification) task, we advance the existing binary classification dataset, towards a higher-level task of reliability analysis through the lens of explanations, posing it as one of the safety measures. We annotate the LoST dataset to capture nuanced textual cues that suggest the presence of low self-esteem in the posts of Reddit users. We further state that the NLP models developed for determining the presence of low self-esteem, focus more on three types of textual cues: (i) Trigger: words that triggers mental disturbance, (ii) LoST indicators: text indicators emphasizing low self-esteem, and (iii) Consequences: words describing the consequences of mental disturbance. We implement existing classifiers to examine the attention mechanism in pre-trained language models (PLMs) for a domain-specific psychology-grounded task. Our findings suggest the need of shifting the focus of PLMs from Trigger and Consequences to a more comprehensive explanation, emphasizing LoST indicators while determining low self-esteem in Reddit posts.

Title: Model-Free Approximate Bayesian Learning for Large-Scale Conversion Funnel Optimization

• Authors: Garud Iyengar, Raghav Singal
• Subjects: cs.LG, cs.IR
• Abstract URL: https://arxiv.org/abs/2401.06710
• Pdf URL: https://arxiv.org/pdf/2401.06710
• Copy Paste: [[2401.06710]] Model-Free Approximate Bayesian Learning for Large-Scale Conversion Funnel Optimization(https://arxiv.org/abs/2401.06710)
• Keywords: interpretability
• Abstract: The flexibility of choosing the ad action as a function of the consumer state is critical for modern-day marketing campaigns. We study the problem of identifying the optimal sequential personalized interventions that maximize the adoption probability for a new product. We model consumer behavior by a conversion funnel that captures the state of each consumer (e.g., interaction history with the firm) and allows the consumer behavior to vary as a function of both her state and firm's sequential interventions. We show our model captures consumer behavior with very high accuracy (out-of-sample AUC of over 0.95) in a real-world email marketing dataset. However, it results in a very large-scale learning problem, where the firm must learn the state-specific effects of various interventions from consumer interactions. We propose a novel attribution-based decision-making algorithm for this problem that we call model-free approximate Bayesian learning. Our algorithm inherits the interpretability and scalability of Thompson sampling for bandits and maintains an approximate belief over the value of each state-specific intervention. The belief is updated as the algorithm interacts with the consumers. Despite being an approximation to the Bayes update, we prove the asymptotic optimality of our algorithm and analyze its convergence rate. We show that our algorithm significantly outperforms traditional approaches on extensive simulations calibrated to a real-world email marketing dataset.

Title: Few-Shot Detection of Machine-Generated Text using Style Representations

• Authors: Rafael Rivera Soto, Kailin Koch, Aleem Khan, Barry Chen, Marcus Bishop, Nicholas Andrews
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.06712
• Pdf URL: https://arxiv.org/pdf/2401.06712
• Copy Paste: [[2401.06712]] Few-Shot Detection of Machine-Generated Text using Style Representations(https://arxiv.org/abs/2401.06712)
• Keywords: large language model
• Abstract: The advent of instruction-tuned language models that convincingly mimic human writing poses a significant risk of abuse. For example, such models could be used for plagiarism, disinformation, spam, or phishing. However, such abuse may be counteracted with the ability to detect whether a piece of text was composed by a language model rather than a human. Some previous approaches to this problem have relied on supervised methods trained on corpora of confirmed human and machine-written documents. Unfortunately, model under-specification poses an unavoidable challenge for neural network-based detectors, making them brittle in the face of data shifts, such as the release of further language models producing still more fluent text than the models used to train the detectors. Other previous approaches require access to the models that may have generated a document in question at inference or detection time, which is often impractical. In light of these challenges, we pursue a fundamentally different approach not relying on samples from language models of concern at training time. Instead, we propose to leverage representations of writing style estimated from human-authored text. Indeed, we find that features effective at distinguishing among human authors are also effective at distinguishing human from machine authors, including state of the art large language models like Llama 2, ChatGPT, and GPT-4. Furthermore, given a handful of examples composed by each of several specific language models of interest, our approach affords the ability to predict which model generated a given document.

Title: Reframing Tax Law Entailment as Analogical Reasoning

• Authors: Xinrui Zou, Ming Zhang, Nathaniel Weir, Benjamin Van Durme, Nils Holzenberger
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.06715
• Pdf URL: https://arxiv.org/pdf/2401.06715
• Copy Paste: [[2401.06715]] Reframing Tax Law Entailment as Analogical Reasoning(https://arxiv.org/abs/2401.06715)
• Keywords: interpretability
• Abstract: Statutory reasoning refers to the application of legislative provisions to a series of case facts described in natural language. We re-frame statutory reasoning as an analogy task, where each instance of the analogy task involves a combination of two instances of statutory reasoning. This increases the dataset size by two orders of magnitude, and introduces an element of interpretability. We show that this task is roughly as difficult to Natural Language Processing models as the original task. Finally, we come back to statutory reasoning, solving it with a combination of a retrieval mechanism and analogy models, and showing some progress on prior comparable work.

Title: Using Natural Language Inference to Improve Persona Extraction from Dialogue in a New Domain

• Authors: Alexandra DeLucia, Mengjie Zhao, Yoshinori Maeda, Makoto Yoda, Keiichi Yamada, Hiromi Wakaki
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.06742
• Pdf URL: https://arxiv.org/pdf/2401.06742
• Copy Paste: [[2401.06742]] Using Natural Language Inference to Improve Persona Extraction from Dialogue in a New Domain(https://arxiv.org/abs/2401.06742)
• Keywords: extraction
• Abstract: While valuable datasets such as PersonaChat provide a foundation for training persona-grounded dialogue agents, they lack diversity in conversational and narrative settings, primarily existing in the "real" world. To develop dialogue agents with unique personas, models are trained to converse given a specific persona, but hand-crafting these persona can be time-consuming, thus methods exist to automatically extract persona information from existing character-specific dialogue. However, these persona-extraction models are also trained on datasets derived from PersonaChat and struggle to provide high-quality persona information from conversational settings that do not take place in the real world, such as the fantasy-focused dataset, LIGHT. Creating new data to train models on a specific setting is human-intensive, thus prohibitively expensive. To address both these issues, we introduce a natural language inference method for post-hoc adapting a trained persona extraction model to a new setting. We draw inspiration from the literature of dialog natural language inference (NLI), and devise NLI-reranking methods to extract structured persona information from dialogue. Compared to existing persona extraction models, our method returns higher-quality extracted persona and requires less human annotation.

Title: APAR: LLMs Can Do Auto-Parallel Auto-Regressive Decoding

• Authors: Mingdao Liu, Aohan Zeng, Bowen Wang, Peng Zhang, Jie Tang, Yuxiao Dong
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.06761
• Pdf URL: https://arxiv.org/pdf/2401.06761
• Copy Paste: [[2401.06761]] APAR: LLMs Can Do Auto-Parallel Auto-Regressive Decoding(https://arxiv.org/abs/2401.06761)
• Keywords: large language model
• Abstract: The massive adoption of large language models (LLMs) demands efficient deployment strategies. However, the auto-regressive decoding process, which is fundamental to how most LLMs generate text, poses challenges to achieve efficient serving. In this work, we introduce a parallel auto-regressive generation method. By instruct-tuning on general domain data that contains hierarchical structures, we enable LLMs to independently plan their generation process and perform auto-parallel auto-regressive (APAR) generation, significantly reducing the number of generation steps. APAR alone can achieve up to 2x speed-up, and when combined with speculative decoding, the speed-up can reach up to 4x. In addition, APAR reduces the key-value cache consumption and attention computation during generation. This leads to a throughput increase of 20-70% and a latency reduce of 20-35% in high-throughput scenarios, compared to state-of-the-art serving frameworks.

Title: Seeing the roads through the trees: A benchmark for modeling spatial dependencies with aerial imagery

• Authors: Caleb Robinson, Isaac Corley, Anthony Ortiz, Rahul Dodhia, Juan M. Lavista Ferres, Peyman Najafirad
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.06762
• Pdf URL: https://arxiv.org/pdf/2401.06762
• Copy Paste: [[2401.06762]] Seeing the roads through the trees: A benchmark for modeling spatial dependencies with aerial imagery(https://arxiv.org/abs/2401.06762)
• Keywords: segmentation
• Abstract: Fully understanding a complex high-resolution satellite or aerial imagery scene often requires spatial reasoning over a broad relevant context. The human object recognition system is able to understand object in a scene over a long-range relevant context. For example, if a human observes an aerial scene that shows sections of road broken up by tree canopy, then they will be unlikely to conclude that the road has actually been broken up into disjoint pieces by trees and instead think that the canopy of nearby trees is occluding the road. However, there is limited research being conducted to understand long-range context understanding of modern machine learning models. In this work we propose a road segmentation benchmark dataset, Chesapeake Roads Spatial Context (RSC), for evaluating the spatial long-range context understanding of geospatial machine learning models and show how commonly used semantic segmentation models can fail at this task. For example, we show that a U-Net trained to segment roads from background in aerial imagery achieves an 84% recall on unoccluded roads, but just 63.5% recall on roads covered by tree canopy despite being trained to model both the same way. We further analyze how the performance of models changes as the relevant context for a decision (unoccluded roads in our case) varies in distance. We release the code to reproduce our experiments and dataset of imagery and masks to encourage future research in this direction -- https://github.com/isaaccorley/ChesapeakeRSC.

Title: Optimally Blending Honeypots into Production Networks: Hardness and Algorithms

• Authors: Md Mahabub Uz Zaman, Liangde Tao, Mark Maldonado, Chang Liu, Ahmed Sunny, Shouhuai Xu, Lin Chen
• Subjects: cs.CR, cs.CC
• Abstract URL: https://arxiv.org/abs/2401.06763
• Pdf URL: https://arxiv.org/pdf/2401.06763
• Copy Paste: [[2401.06763]] Optimally Blending Honeypots into Production Networks: Hardness and Algorithms(https://arxiv.org/abs/2401.06763)
• Keywords: security, defense, attack
• Abstract: Honeypot is an important cyber defense technique that can expose attackers new attacks. However, the effectiveness of honeypots has not been systematically investigated, beyond the rule of thumb that their effectiveness depends on how they are deployed. In this paper, we initiate a systematic study on characterizing the cybersecurity effectiveness of a new paradigm of deploying honeypots: blending honeypot computers (or IP addresses) into production computers. This leads to the following Honeypot Deployment (HD) problem, How should the defender blend honeypot computers into production computers to maximize the utility in forcing attackers to expose their new attacks while minimizing the loss to the defender in terms of the digital assets stored in the compromised production computers? We formalize HD as a combinatorial optimization problem, prove its NP hardness, provide a near optimal algorithm (i.e., polynomial time approximation scheme). We also conduct simulations to show the impact of attacker capabilities.

Title: Mind Your Format: Towards Consistent Evaluation of In-Context Learning Improvements

• Authors: Anton Voronov, Lena Wolf, Max Ryabinin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.06766
• Pdf URL: https://arxiv.org/pdf/2401.06766
• Copy Paste: [[2401.06766]] Mind Your Format: Towards Consistent Evaluation of In-Context Learning Improvements(https://arxiv.org/abs/2401.06766)
• Keywords: robust, large language model
• Abstract: Large language models demonstrate a remarkable capability for learning to solve new tasks from a few examples. The prompt template, or the way the input examples are formatted to obtain the prompt, is an important yet often overlooked aspect of in-context learning. In this work, we conduct a comprehensive study of the template format's influence on the in-context learning performance. We evaluate the impact of the prompt template across models (from 770M to 70B parameters) and 4 standard classification datasets. We show that a poor choice of the template can reduce the performance of the strongest models and inference methods to a random guess level. More importantly, the best templates do not transfer between different setups and even between models of the same family. Our findings show that the currently prevalent approach to evaluation, which ignores template selection, may give misleading results due to different templates in different works. As a first step towards mitigating this issue, we propose Template Ensembles that aggregate model predictions across several templates. This simple test-time augmentation boosts average performance while being robust to the choice of random set of templates.