secure

Title: CompactTag: Minimizing Computation Overheads in Actively-Secure MPC for Deep Neural Networks. (arXiv:2311.04406v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.04406
• Code URL: null
• Copy Paste: [[2311.04406]] CompactTag: Minimizing Computation Overheads in Actively-Secure MPC for Deep Neural Networks(http://arxiv.org/abs/2311.04406)
• Summary:

  Secure Multiparty Computation (MPC) protocols enable secure evaluation of a circuit by several parties, even in the presence of an adversary who maliciously corrupts all but one of the parties. These MPC protocols are constructed using the well-known secret-sharing-based paradigm (SPDZ and SPDZ2k), where the protocols ensure security against a malicious adversary by computing Message Authentication Code (MAC) tags on the input shares and then evaluating the circuit with these input shares and tags. However, this tag computation adds a significant runtime overhead, particularly for machine learning (ML) applications with numerous linear computation layers such as convolutions and fully connected layers.

  To alleviate the tag computation overhead, we introduce CompactTag, a lightweight algorithm for generating MAC tags specifically tailored for linear layers in ML. Linear layer operations in ML, including convolutions, can be transformed into Toeplitz matrix multiplications. For the multiplication of two matrices with dimensions T1 x T2 and T2 x T3 respectively, SPDZ2k required O(T1 x T2 x T3) local multiplications for the tag computation. In contrast, CompactTag only requires O(T1 x T2 + T1 x T3 + T2 x T3) local multiplications, resulting in a substantial performance boost for various ML models.

  We empirically compared our protocol to the SPDZ2k protocol for various ML circuits, including ResNet Training-Inference, Transformer Training-Inference, and VGG16 Training-Inference. SPDZ2k dedicated around 30% of its online runtime for tag computation. CompactTag speeds up this tag computation bottleneck by up to 23x, resulting in up to 1.47x total online phase runtime speedups for various ML workloads.

Title: Analysis and Applications of Deep Learning with Finite Samples in Full Life-Cycle Intelligence of Nuclear Power Generation. (arXiv:2311.04247v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.04247
• Code URL: null
• Copy Paste: [[2311.04247]] Analysis and Applications of Deep Learning with Finite Samples in Full Life-Cycle Intelligence of Nuclear Power Generation(http://arxiv.org/abs/2311.04247)
• Summary:

  The advent of Industry 4.0 has precipitated the incorporation of Artificial Intelligence (AI) methods within industrial contexts, aiming to realize intelligent manufacturing, operation as well as maintenance, also known as industrial intelligence. However, intricate industrial milieus, particularly those relating to energy exploration and production, frequently encompass data characterized by long-tailed class distribution, sample imbalance, and domain shift. These attributes pose noteworthy challenges to data-centric Deep Learning (DL) techniques, crucial for the realization of industrial intelligence. The present study centers on the intricate and distinctive industrial scenarios of Nuclear Power Generation (NPG), meticulously scrutinizing the application of DL techniques under the constraints of finite data samples. Initially, the paper expounds on potential employment scenarios for AI across the full life-cycle of NPG. Subsequently, we delve into an evaluative exposition of DL's advancement, grounded in the finite sample perspective. This encompasses aspects such as small-sample learning, few-shot learning, zero-shot learning, and open-set recognition, also referring to the unique data characteristics of NPG. The paper then proceeds to present two specific case studies. The first revolves around the automatic recognition of zirconium alloy metallography, while the second pertains to open-set recognition for signal diagnosis of machinery sensors. These cases, spanning the entirety of NPG's life-cycle, are accompanied by constructive outcomes and insightful deliberations. By exploring and applying DL methodologies within the constraints of finite sample availability, this paper not only furnishes a robust technical foundation but also introduces a fresh perspective toward the secure and efficient advancement and exploitation of this advanced energy source.

security

Title: Enhancing Malware Detection by Integrating Machine Learning with Cuckoo Sandbox. (arXiv:2311.04372v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.04372
• Code URL: null
• Copy Paste: [[2311.04372]] Enhancing Malware Detection by Integrating Machine Learning with Cuckoo Sandbox(http://arxiv.org/abs/2311.04372)
• Summary:

  In the modern era, malware is experiencing a significant increase in both its variety and quantity, aligning with the widespread adoption of the digital world. This surge in malware has emerged as a critical challenge in the realm of cybersecurity, prompting numerous research endeavors and contributions to address the issue. Machine learning algorithms have been leveraged for malware detection due to their ability to uncover concealed patterns within vast datasets. However, deep learning algorithms, characterized by their multi-layered structure, surpass the limitations of traditional machine learning approaches. By employing deep learning techniques such as CNN (Convolutional Neural Network) and RNN (Recurrent Neural Network), this study aims to classify and identify malware extracted from a dataset containing API call sequences. The performance of these algorithms is compared with that of conventional machine learning methods, including SVM (Support Vector Machine), RF (Random Forest), KNN (K-Nearest Neighbors), XGB (Extreme Gradient Boosting), and GBC (Gradient Boosting Classifier), all using the same dataset. The outcomes of this research demonstrate that both deep learning and machine learning algorithms achieve remarkably high levels of accuracy, reaching up to 99% in certain cases.

Title: Sandi: A System for Accountability and Applications in Direct Communication (Extended Abstract). (arXiv:2311.04861v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.04861
• Code URL: null
• Copy Paste: [[2311.04861]] Sandi: A System for Accountability and Applications in Direct Communication (Extended Abstract)(http://arxiv.org/abs/2311.04861)
• Summary:

  Reputation systems guide our decision making both in life and work: which restaurant to eat at, which vendor to buy from, which software dependencies to use, and who or what to trust. These systems are often based on old ideas and are failing in the face of modern threats. Fraudsters have found ways to manipulate them, undermining their integrity and utility. Generative AI adds to the problem by enabling the creation of real-looking fake narratives at scale, creating a false sense of consensus. Meanwhile, the need for reliable reputation concepts is more important than ever, as wrong decisions lead to increasingly severe outcomes: wasted time, poor service, and a feeling of injustice at best, fraud, identity theft, and ransomware at worst.

  In this extended abstract we introduce Sandi, a new kind of reputation system with a single well-defined purpose: to create trust through accountability in one-to-one transactions. Examples of such transactions include sending an email or making a purchase online. Sandi has strong security and privacy properties that make it suitable for use also in sensitive contexts. Furthermore, Sandi can guarantee reputation integrity and transparency for its registered users.

  As a primary application, we envision how Sandi could counter fraud and abuse in direct communication. Concretely, message senders request a cryptographic tag from Sandi that they send along with their message. If the receiver finds the message inappropriate, they can report the sender using this tag. Notably, only senders need registered accounts and do not need to manage long-term keys. The design of Sandi ensures compatibility with any communication system that allows for small binary data transmission.

privacy

Title: Anonymizing medical case-based explanations through disentanglement. (arXiv:2311.04833v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.04833
• Code URL: null
• Copy Paste: [[2311.04833]] Anonymizing medical case-based explanations through disentanglement(http://arxiv.org/abs/2311.04833)
• Summary:

  Case-based explanations are an intuitive method to gain insight into the decision-making process of deep learning models in clinical contexts. However, medical images cannot be shared as explanations due to privacy concerns. To address this problem, we propose a novel method for disentangling identity and medical characteristics of images and apply it to anonymize medical images. The disentanglement mechanism replaces some feature vectors in an image while ensuring that the remaining features are preserved, obtaining independent feature vectors that encode the images' identity and medical characteristics. We also propose a model to manufacture synthetic privacy-preserving identities to replace the original image's identity and achieve anonymization. The models are applied to medical and biometric datasets, demonstrating their capacity to generate realistic-looking anonymized images that preserve their original medical content. Additionally, the experiments show the network's inherent capacity to generate counterfactual images through the replacement of medical features.

Title: Federated Experiment Design under Distributed Differential Privacy. (arXiv:2311.04375v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.04375
• Code URL: null
• Copy Paste: [[2311.04375]] Federated Experiment Design under Distributed Differential Privacy(http://arxiv.org/abs/2311.04375)
• Summary:

  Experiment design has a rich history dating back over a century and has found many critical applications across various fields since then. The use and collection of users' data in experiments often involve sensitive personal information, so additional measures to protect individual privacy are required during data collection, storage, and usage. In this work, we focus on the rigorous protection of users' privacy (under the notion of differential privacy (DP)) while minimizing the trust toward service providers. Specifically, we consider the estimation of the average treatment effect (ATE) under DP, while only allowing the analyst to collect population-level statistics via secure aggregation, a distributed protocol enabling a service provider to aggregate information without accessing individual data. Although a vital component in modern A/B testing workflows, private distributed experimentation has not previously been studied. To achieve DP, we design local privatization mechanisms that are compatible with secure aggregation and analyze the utility, in terms of the width of confidence intervals, both asymptotically and non-asymptotically. We show how these mechanisms can be scaled up to handle the very large number of participants commonly found in practice. In addition, when introducing DP noise, it is imperative to cleverly split privacy budgets to estimate both the mean and variance of the outcomes and carefully calibrate the confidence intervals according to the DP noise. Last, we present comprehensive experimental evaluations of our proposed schemes and show the privacy-utility trade-offs in experiment design.

Title: Local Differential Privacy for Smart Meter Data Sharing. (arXiv:2311.04544v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.04544
• Code URL: null
• Copy Paste: [[2311.04544]] Local Differential Privacy for Smart Meter Data Sharing(http://arxiv.org/abs/2311.04544)
• Summary:

  Energy disaggregation techniques, which use smart meter data to infer appliance energy usage, can provide consumers and energy companies valuable insights into energy management. However, these techniques also present privacy risks, such as the potential for behavioral profiling. Local differential privacy (LDP) methods provide strong privacy guarantees with high efficiency in addressing privacy concerns. However, existing LDP methods focus on protecting aggregated energy consumption data rather than individual appliances. Furthermore, these methods do not consider the fact that smart meter data are a form of streaming data, and its processing methods should account for time windows. In this paper, we propose a novel LDP approach (named LDP-SmartEnergy) that utilizes randomized response techniques with sliding windows to facilitate the sharing of appliance-level energy consumption data over time while not revealing individual users' appliance usage patterns. Our evaluations show that LDP-SmartEnergy runs efficiently compared to baseline methods. The results also demonstrate that our solution strikes a balance between protecting privacy and maintaining the utility of data for effective analysis.

protect

Title: General Framework to Evaluate Unlinkability in Biometric Template Protection Systems. (arXiv:2311.04633v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.04633
• Code URL: null
• Copy Paste: [[2311.04633]] General Framework to Evaluate Unlinkability in Biometric Template Protection Systems(http://arxiv.org/abs/2311.04633)
• Summary:

  The wide deployment of biometric recognition systems in the last two decades has raised privacy concerns regarding the storage and use of biometric data. As a consequence, the ISO/IEC 24745 international standard on biometric information protection has established two main requirements for protecting biometric templates: irreversibility and unlinkability. Numerous efforts have been directed to the development and analysis of irreversible templates. However, there is still no systematic quantitative manner to analyse the unlinkability of such templates. In this paper we address this shortcoming by proposing a new general framework for the evaluation of biometric templates' unlinkability. To illustrate the potential of the approach, it is applied to assess the unlinkability of four state-of-the-art techniques for biometric template protection: biometric salting, Bloom filters, Homomorphic Encryption and block re-mapping. For the last technique, the proposed framework is compared with other existing metrics to show its advantages.

defense

attack

Title: Be Careful When Evaluating Explanations Regarding Ground Truth. (arXiv:2311.04813v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.04813
• Code URL: https://github.com/mi2datalab/be-careful-evaluating-explanations
• Copy Paste: [[2311.04813]] Be Careful When Evaluating Explanations Regarding Ground Truth(http://arxiv.org/abs/2311.04813)
• Summary:

  Evaluating explanations of image classifiers regarding ground truth, e.g. segmentation masks defined by human perception, primarily evaluates the quality of the models under consideration rather than the explanation methods themselves. Driven by this observation, we propose a framework for $\textit{jointly}$ evaluating the robustness of safety-critical systems that $\textit{combine}$ a deep neural network with an explanation method. These are increasingly used in real-world applications like medical image analysis or robotics. We introduce a fine-tuning procedure to (mis)align model$\unicode{x2013}$explanation pipelines with ground truth and use it to quantify the potential discrepancy between worst and best-case scenarios of human alignment. Experiments across various model architectures and post-hoc local interpretation methods provide insights into the robustness of vision transformers and the overall vulnerability of such AI systems to potential adversarial attacks.

Title: SyncBleed: A Realistic Threat Model and Mitigation Strategy for Zero-Involvement Pairing and Authentication (ZIPA). (arXiv:2311.04433v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.04433
• Code URL: null
• Copy Paste: [[2311.04433]] SyncBleed: A Realistic Threat Model and Mitigation Strategy for Zero-Involvement Pairing and Authentication (ZIPA)(http://arxiv.org/abs/2311.04433)
• Summary:

  Zero Involvement Pairing and Authentication (ZIPA) is a promising technique for auto-provisioning large networks of Internet-of-Things (IoT) devices. Presently, these networks use password-based authentication, which is difficult to scale to more than a handful of devices. To deal with this challenge, ZIPA enabled devices autonomously extract identical authentication or encryption keys from ambient environmental signals. However, during the key negotiation process, existing ZIPA systems leak information on a public wireless channel which can allow adversaries to learn the key. We demonstrate a passive attack called SyncBleed, which uses leaked information to reconstruct keys generated by ZIPA systems. To mitigate SyncBleed, we present TREVOR, an improved key generation technique that produces nearly identical bit sequences from environmental signals without leaking information. We demonstrate that TREVOR can generate keys from a variety of environmental signal types under 4 seconds, consistently achieving a 90-95% bit agreement rate across devices within various environmental sources.

Title: DAG-Sword: A Simulator of Large-Scale Network Topologies for DAG-Oriented Proof-of-Work Blockchains. (arXiv:2311.04638v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.04638
• Code URL: null
• Copy Paste: [[2311.04638]] DAG-Sword: A Simulator of Large-Scale Network Topologies for DAG-Oriented Proof-of-Work Blockchains(http://arxiv.org/abs/2311.04638)
• Summary:

  The blockchain brought interesting properties for many practical applications. However, some properties, such as the transaction processing throughput remained limited, especially in Proof-of-Work blockchains. Therefore, several promising directions, such as sharding designs and DAG-based protocols emerged. In this paper, we focus on DAG-based consensus protocols and present a discrete-event simulator for them. Our simulator can simulate realistic blockchain networks created from data of a Bitcoin network, while its network configuration and topology can be customized. The simulated network consists of honest and malicious miners. Malicious miners do not make any attack on consensus itself. Instead, they use a different transaction selection strategy than honest miners (who select transactions randomly) with the intention to earn unfairly more profits than honest miners at the cost of downgrading the protocol performance by duplicate transactions. As a consequence, this harms the performance of some DAG-based protocols (e.g., PHANTOM and GHOSTDAG) in terms of transaction processing throughput, which we demonstrate in our experiments and extend the results of the related work that contains a small-scale network of 10 nodes by the results obtained on a large-scale network with 7000 nodes. Next, we empirically compare different algorithms for the mempool structure, and we propose a composite mempool structure that is memory-efficient and thus convenient for simulations of resource-demanding large-scale networks.

Title: Constrained Adaptive Attacks: Realistic Evaluation of Adversarial Examples and Robust Training of Deep Neural Networks for Tabular Data. (arXiv:2311.04503v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.04503
• Code URL: null
• Copy Paste: [[2311.04503]] Constrained Adaptive Attacks: Realistic Evaluation of Adversarial Examples and Robust Training of Deep Neural Networks for Tabular Data(http://arxiv.org/abs/2311.04503)
• Summary:

  State-of-the-art deep learning models for tabular data have recently achieved acceptable performance to be deployed in industrial settings. However, the robustness of these models remains scarcely explored. Contrary to computer vision, there is to date no realistic protocol to properly evaluate the adversarial robustness of deep tabular models due to intrinsic properties of tabular data such as categorical features, immutability, and feature relationship constraints. To fill this gap, we propose CAA, the first efficient evasion attack for constrained tabular deep learning models. CAA is an iterative parameter-free attack that combines gradient and search attacks to generate adversarial examples under constraints. We leverage CAA to build a benchmark of deep tabular models across three popular use cases: credit scoring, phishing and botnet attacks detection. Our benchmark supports ten threat models with increasing capabilities of the attacker, and reflects real-world attack scenarios for each use case. Overall, our results demonstrate how domain knowledge, adversarial training, and attack budgets impact the robustness assessment of deep tabular models and provide security practitioners with a set of recommendations to improve the robustness of deep tabular models against various evasion attack scenarios.

Title: Byzantine-Tolerant Methods for Distributed Variational Inequalities. (arXiv:2311.04611v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.04611
• Code URL: https://github.com/nazya/sgda-ra
• Copy Paste: [[2311.04611]] Byzantine-Tolerant Methods for Distributed Variational Inequalities(http://arxiv.org/abs/2311.04611)
• Summary:

  Robustness to Byzantine attacks is a necessity for various distributed training scenarios. When the training reduces to the process of solving a minimization problem, Byzantine robustness is relatively well-understood. However, other problem formulations, such as min-max problems or, more generally, variational inequalities, arise in many modern machine learning and, in particular, distributed learning tasks. These problems significantly differ from the standard minimization ones and, therefore, require separate consideration. Nevertheless, only one work (Adibi et al., 2022) addresses this important question in the context of Byzantine robustness. Our work makes a further step in this direction by providing several (provably) Byzantine-robust methods for distributed variational inequality, thoroughly studying their theoretical convergence, removing the limitations of the previous work, and providing numerical comparisons supporting the theoretical findings.

robust

Title: Holistic Evaluation of Text-To-Image Models. (arXiv:2311.04287v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.04287
• Code URL: https://github.com/stanford-crfm/helm
• Copy Paste: [[2311.04287]] Holistic Evaluation of Text-To-Image Models(http://arxiv.org/abs/2311.04287)
• Summary:

  The stunning qualitative improvement of recent text-to-image models has led to their widespread attention and adoption. However, we lack a comprehensive quantitative understanding of their capabilities and risks. To fill this gap, we introduce a new benchmark, Holistic Evaluation of Text-to-Image Models (HEIM). Whereas previous evaluations focus mostly on text-image alignment and image quality, we identify 12 aspects, including text-image alignment, image quality, aesthetics, originality, reasoning, knowledge, bias, toxicity, fairness, robustness, multilinguality, and efficiency. We curate 62 scenarios encompassing these aspects and evaluate 26 state-of-the-art text-to-image models on this benchmark. Our results reveal that no single model excels in all aspects, with different models demonstrating different strengths. We release the generated images and human evaluation results for full transparency at https://crfm.stanford.edu/heim/v1.1.0 and the code at https://github.com/stanford-crfm/helm, which is integrated with the HELM codebase.

Title: CLearViD: Curriculum Learning for Video Description. (arXiv:2311.04480v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.04480
• Code URL: null
• Copy Paste: [[2311.04480]] CLearViD: Curriculum Learning for Video Description(http://arxiv.org/abs/2311.04480)
• Summary:

  Video description entails automatically generating coherent natural language sentences that narrate the content of a given video. We introduce CLearViD, a transformer-based model for video description generation that leverages curriculum learning to accomplish this task. In particular, we investigate two curriculum strategies: (1) progressively exposing the model to more challenging samples by gradually applying a Gaussian noise to the video data, and (2) gradually reducing the capacity of the network through dropout during the training process. These methods enable the model to learn more robust and generalizable features. Moreover, CLearViD leverages the Mish activation function, which provides non-linearity and non-monotonicity and helps alleviate the issue of vanishing gradients. Our extensive experiments and ablation studies demonstrate the effectiveness of the proposed model. The results on two datasets, namely ActivityNet Captions and YouCook2, show that CLearViD significantly outperforms existing state-of-the-art models in terms of both accuracy and diversity metrics.

Title: Non-Rigid Shape Registration via Deep Functional Maps Prior. (arXiv:2311.04494v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.04494
• Code URL: null
• Copy Paste: [[2311.04494]] Non-Rigid Shape Registration via Deep Functional Maps Prior(http://arxiv.org/abs/2311.04494)
• Summary:

  In this paper, we propose a learning-based framework for non-rigid shape registration without correspondence supervision. Traditional shape registration techniques typically rely on correspondences induced by extrinsic proximity, therefore can fail in the presence of large intrinsic deformations. Spectral mapping methods overcome this challenge by embedding shapes into, geometric or learned, high-dimensional spaces, where shapes are easier to align. However, due to the dependency on abstract, non-linear embedding schemes, the latter can be vulnerable with respect to perturbed or alien input. In light of this, our framework takes the best of both worlds. Namely, we deform source mesh towards the target point cloud, guided by correspondences induced by high-dimensional embeddings learned from deep functional maps (DFM). In particular, the correspondences are dynamically updated according to the intermediate registrations and filtered by consistency prior, which prominently robustify the overall pipeline. Moreover, in order to alleviate the requirement of extrinsically aligned input, we train an orientation regressor on a set of aligned synthetic shapes independent of the training shapes for DFM. Empirical results show that, with as few as dozens of training shapes of limited variability, our pipeline achieves state-of-the-art results on several benchmarks of non-rigid point cloud matching, but also delivers high-quality correspondences between unseen challenging shape pairs that undergo both significant extrinsic and intrinsic deformations, in which case neither traditional registration methods nor intrinsic methods work. The code is available at https://github.com/rqhuang88/DFR.

Title: PersonMAE: Person Re-Identification Pre-Training with Masked AutoEncoders. (arXiv:2311.04496v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.04496
• Code URL: null
• Copy Paste: [[2311.04496]] PersonMAE: Person Re-Identification Pre-Training with Masked AutoEncoders(http://arxiv.org/abs/2311.04496)
• Summary:

  Pre-training is playing an increasingly important role in learning generic feature representation for Person Re-identification (ReID). We argue that a high-quality ReID representation should have three properties, namely, multi-level awareness, occlusion robustness, and cross-region invariance. To this end, we propose a simple yet effective pre-training framework, namely PersonMAE, which involves two core designs into masked autoencoders to better serve the task of Person Re-ID. 1) PersonMAE generates two regions from the given image with RegionA as the input and \textit{RegionB} as the prediction target. RegionA is corrupted with block-wise masking to mimic common occlusion in ReID and its remaining visible parts are fed into the encoder. 2) Then PersonMAE aims to predict the whole RegionB at both pixel level and semantic feature level. It encourages its pre-trained feature representations with the three properties mentioned above. These properties make PersonMAE compatible with downstream Person ReID tasks, leading to state-of-the-art performance on four downstream ReID tasks, i.e., supervised (holistic and occluded setting), and unsupervised (UDA and USL setting). Notably, on the commonly adopted supervised setting, PersonMAE with ViT-B backbone achieves 79.8% and 69.5% mAP on the MSMT17 and OccDuke datasets, surpassing the previous state-of-the-art by a large margin of +8.0 mAP, and +5.3 mAP, respectively.

Title: Learning Robust Multi-Scale Representation for Neural Radiance Fields from Unposed Images. (arXiv:2311.04521v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.04521
• Code URL: null
• Copy Paste: [[2311.04521]] Learning Robust Multi-Scale Representation for Neural Radiance Fields from Unposed Images(http://arxiv.org/abs/2311.04521)
• Summary:

  We introduce an improved solution to the neural image-based rendering problem in computer vision. Given a set of images taken from a freely moving camera at train time, the proposed approach could synthesize a realistic image of the scene from a novel viewpoint at test time. The key ideas presented in this paper are (i) Recovering accurate camera parameters via a robust pipeline from unposed day-to-day images is equally crucial in neural novel view synthesis problem; (ii) It is rather more practical to model object's content at different resolutions since dramatic camera motion is highly likely in day-to-day unposed images. To incorporate the key ideas, we leverage the fundamentals of scene rigidity, multi-scale neural scene representation, and single-image depth prediction. Concretely, the proposed approach makes the camera parameters as learnable in a neural fields-based modeling framework. By assuming per view depth prediction is given up to scale, we constrain the relative pose between successive frames. From the relative poses, absolute camera pose estimation is modeled via a graph-neural network-based multiple motion averaging within the multi-scale neural-fields network, leading to a single loss function. Optimizing the introduced loss function provides camera intrinsic, extrinsic, and image rendering from unposed images. We demonstrate, with examples, that for a unified framework to accurately model multiscale neural scene representation from day-to-day acquired unposed multi-view images, it is equally essential to have precise camera-pose estimates within the scene representation framework. Without considering robustness measures in the camera pose estimation pipeline, modeling for multi-scale aliasing artifacts can be counterproductive. We present extensive experiments on several benchmark datasets to demonstrate the suitability of our approach.

Title: Rethinking Event-based Human Pose Estimation with 3D Event Representations. (arXiv:2311.04591v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.04591
• Code URL: https://github.com/masterhow/eventpointpose
• Copy Paste: [[2311.04591]] Rethinking Event-based Human Pose Estimation with 3D Event Representations(http://arxiv.org/abs/2311.04591)
• Summary:

  Human pose estimation is a critical component in autonomous driving and parking, enhancing safety by predicting human actions. Traditional frame-based cameras and videos are commonly applied, yet, they become less reliable in scenarios under high dynamic range or heavy motion blur. In contrast, event cameras offer a robust solution for navigating these challenging contexts. Predominant methodologies incorporate event cameras into learning frameworks by accumulating events into event frames. However, such methods tend to marginalize the intrinsic asynchronous and high temporal resolution characteristics of events. This disregard leads to a loss in essential temporal dimension data, crucial for safety-critical tasks associated with dynamic human activities. To address this issue and to unlock the 3D potential of event information, we introduce two 3D event representations: the Rasterized Event Point Cloud (RasEPC) and the Decoupled Event Voxel (DEV). The RasEPC collates events within concise temporal slices at identical positions, preserving 3D attributes with statistical cues and markedly mitigating memory and computational demands. Meanwhile, the DEV representation discretizes events into voxels and projects them across three orthogonal planes, utilizing decoupled event attention to retrieve 3D cues from the 2D planes. Furthermore, we develop and release EV-3DPW, a synthetic event-based dataset crafted to facilitate training and quantitative analysis in outdoor scenes. On the public real-world DHP19 dataset, our event point cloud technique excels in real-time mobile predictions, while the decoupled event voxel method achieves the highest accuracy. Experiments reveal our proposed 3D representation methods' superior generalization capacities against traditional RGB images and event frame techniques. Our code and dataset are available at https://github.com/MasterHow/EventPointPose.

Title: 3D Pose Estimation of Tomato Peduncle Nodes using Deep Keypoint Detection and Point Cloud. (arXiv:2311.04699v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.04699
• Code URL: null
• Copy Paste: [[2311.04699]] 3D Pose Estimation of Tomato Peduncle Nodes using Deep Keypoint Detection and Point Cloud(http://arxiv.org/abs/2311.04699)
• Summary:

  Greenhouse production of fruits and vegetables in developed countries is challenged by labor 12 scarcity and high labor costs. Robots offer a good solution for sustainable and cost-effective 13 production. Acquiring accurate spatial information about relevant plant parts is vital for 14 successful robot operation. Robot perception in greenhouses is challenging due to variations in 15 plant appearance, viewpoints, and illumination. This paper proposes a keypoint-detection-based 16 method using data from an RGB-D camera to estimate the 3D pose of peduncle nodes, which 17 provides essential information to harvest the tomato bunches. 18 19 Specifically, this paper proposes a method that detects four anatomical landmarks in the color 20 image and then integrates 3D point-cloud information to determine the 3D pose. A 21 comprehensive evaluation was conducted in a commercial greenhouse to gain insight into the 22 performance of different parts of the method. The results showed: (1) high accuracy in object 23 detection, achieving an Average Precision (AP) of AP@0.5=0.96; (2) an average Percentage of 24 Detected Joints (PDJ) of the keypoints of PhDJ@0.2=94.31%; and (3) 3D pose estimation 25 accuracy with mean absolute errors (MAE) of 11.38o and 9.93o for the relative upper and lower 26 angles between the peduncle and main stem, respectively. Furthermore, the capability to handle 27 variations in viewpoint was investigated, demonstrating the method was robust to view changes. 28 However, canonical and higher views resulted in slightly higher performance compared to other 29 views. Although tomato was selected as a use case, the proposed method is also applicable to 30 other greenhouse crops like pepper.

Title: Self-Supervised Learning for Visual Relationship Detection through Masked Bounding Box Reconstruction. (arXiv:2311.04834v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.04834
• Code URL: https://github.com/deeplab-ai/selfsupervisedvrd
• Copy Paste: [[2311.04834]] Self-Supervised Learning for Visual Relationship Detection through Masked Bounding Box Reconstruction(http://arxiv.org/abs/2311.04834)
• Summary:

  We present a novel self-supervised approach for representation learning, particularly for the task of Visual Relationship Detection (VRD). Motivated by the effectiveness of Masked Image Modeling (MIM), we propose Masked Bounding Box Reconstruction (MBBR), a variation of MIM where a percentage of the entities/objects within a scene are masked and subsequently reconstructed based on the unmasked objects. The core idea is that, through object-level masked modeling, the network learns context-aware representations that capture the interaction of objects within a scene and thus are highly predictive of visual object relationships. We extensively evaluate learned representations, both qualitatively and quantitatively, in a few-shot setting and demonstrate the efficacy of MBBR for learning robust visual representations, particularly tailored for VRD. The proposed method is able to surpass state-of-the-art VRD methods on the Predicate Detection (PredDet) evaluation setting, using only a few annotated samples. We make our code available at https://github.com/deeplab-ai/SelfSupervisedVRD.

Title: DAMEX: Dataset-aware Mixture-of-Experts for visual understanding of mixture-of-datasets. (arXiv:2311.04894v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.04894
• Code URL: https://github.com/jinga-lala/damex
• Copy Paste: [[2311.04894]] DAMEX: Dataset-aware Mixture-of-Experts for visual understanding of mixture-of-datasets(http://arxiv.org/abs/2311.04894)
• Summary:

  Construction of a universal detector poses a crucial question: How can we most effectively train a model on a large mixture of datasets? The answer lies in learning dataset-specific features and ensembling their knowledge but do all this in a single model. Previous methods achieve this by having separate detection heads on a common backbone but that results in a significant increase in parameters. In this work, we present Mixture-of-Experts as a solution, highlighting that MoEs are much more than a scalability tool. We propose Dataset-Aware Mixture-of-Experts, DAMEX where we train the experts to become an `expert' of a dataset by learning to route each dataset tokens to its mapped expert. Experiments on Universal Object-Detection Benchmark show that we outperform the existing state-of-the-art by average +10.2 AP score and improve over our non-MoE baseline by average +2.0 AP score. We also observe consistent gains while mixing datasets with (1) limited availability, (2) disparate domains and (3) divergent label sets. Further, we qualitatively show that DAMEX is robust against expert representation collapse.

Title: Pre-training LLMs using human-like development data corpus. (arXiv:2311.04666v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.04666
• Code URL: null
• Copy Paste: [[2311.04666]] Pre-training LLMs using human-like development data corpus(http://arxiv.org/abs/2311.04666)
• Summary:

  Pre-trained Large Language Models (LLMs) have shown success in a diverse set of language inference and understanding tasks. The pre-training stage of LLMs looks at a large corpus of raw textual data. The BabyLM shared task compares LLM pre-training to human language acquisition, where the number of tokens seen by 13-year-old kids is magnitudes smaller than the number of tokens seen by LLMs. In this work, we pre-train and evaluate LLMs on their ability to learn contextual word representations using roughly the same number of tokens as seen by children. We provide a strong set of baselines; with different architectures, evaluation of changes in performance across epochs, and reported pre-training metrics for the strict small and strict tracks of the task. We also try to loosely replicate the RoBERTa baseline given by the task organizers to observe the training robustness to hyperparameter selection and replicability. We provide the submission details to the strict and strict-small tracks in this report.

Title: Robust and Communication-Efficient Federated Domain Adaptation via Random Features. (arXiv:2311.04686v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.04686
• Code URL: https://github.com/sadangelf/fedrf-tca
• Copy Paste: [[2311.04686]] Robust and Communication-Efficient Federated Domain Adaptation via Random Features(http://arxiv.org/abs/2311.04686)
• Summary:

  Modern machine learning (ML) models have grown to a scale where training them on a single machine becomes impractical. As a result, there is a growing trend to leverage federated learning (FL) techniques to train large ML models in a distributed and collaborative manner. These models, however, when deployed on new devices, might struggle to generalize well due to domain shifts. In this context, federated domain adaptation (FDA) emerges as a powerful approach to address this challenge.

  Most existing FDA approaches typically focus on aligning the distributions between source and target domains by minimizing their (e.g., MMD) distance. Such strategies, however, inevitably introduce high communication overheads and can be highly sensitive to network reliability.

  In this paper, we introduce RF-TCA, an enhancement to the standard Transfer Component Analysis approach that significantly accelerates computation without compromising theoretical and empirical performance. Leveraging the computational advantage of RF-TCA, we further extend it to FDA setting with FedRF-TCA. The proposed FedRF-TCA protocol boasts communication complexity that is \emph{independent} of the sample size, while maintaining performance that is either comparable to or even surpasses state-of-the-art FDA methods. We present extensive experiments to showcase the superior performance and robustness (to network condition) of FedRF-TCA.

Title: Robust Best-arm Identification in Linear Bandits. (arXiv:2311.04731v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.04731
• Code URL: null
• Copy Paste: [[2311.04731]] Robust Best-arm Identification in Linear Bandits(http://arxiv.org/abs/2311.04731)
• Summary:

  We study the robust best-arm identification problem (RBAI) in the case of linear rewards. The primary objective is to identify a near-optimal robust arm, which involves selecting arms at every round and assessing their robustness by exploring potential adversarial actions. This approach is particularly relevant when utilizing a simulator and seeking to identify a robust solution for real-world transfer. To this end, we present an instance-dependent lower bound for the robust best-arm identification problem with linear rewards. Furthermore, we propose both static and adaptive bandit algorithms that achieve sample complexity that matches the lower bound. In synthetic experiments, our algorithms effectively identify the best robust arm and perform similarly to the oracle strategy. As an application, we examine diabetes care and the process of learning insulin dose recommendations that are robust with respect to inaccuracies in standard calculators. Our algorithms prove to be effective in identifying robust dosage values across various age ranges of patients.

Title: Identifying Semantic Component for Robust Molecular Property Prediction. (arXiv:2311.04837v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.04837
• Code URL: https://github.com/dmirlab-group/sci
• Copy Paste: [[2311.04837]] Identifying Semantic Component for Robust Molecular Property Prediction(http://arxiv.org/abs/2311.04837)
• Summary:

  Although graph neural networks have achieved great success in the task of molecular property prediction in recent years, their generalization ability under out-of-distribution (OOD) settings is still under-explored. Different from existing methods that learn discriminative representations for prediction, we propose a generative model with semantic-components identifiability, named SCI. We demonstrate that the latent variables in this generative model can be explicitly identified into semantic-relevant (SR) and semantic-irrelevant (SI) components, which contributes to better OOD generalization by involving minimal change properties of causal mechanisms. Specifically, we first formulate the data generation process from the atom level to the molecular level, where the latent space is split into SI substructures, SR substructures, and SR atom variables. Sequentially, to reduce misidentification, we restrict the minimal changes of the SR atom variables and add a semantic latent substructure regularization to mitigate the variance of the SR substructure under augmented domain changes. Under mild assumptions, we prove the block-wise identifiability of the SR substructure and the comment-wise identifiability of SR atom variables. Experimental studies achieve state-of-the-art performance and show general improvement on 21 datasets in 3 mainstream benchmarks. Moreover, the visualization results of the proposed SCI method provide insightful case studies and explanations for the prediction results. The code is available at: https://github.com/DMIRLAB-Group/SCI.

biometric

steal

extraction

Title: Army of Thieves: Enhancing Black-Box Model Extraction via Ensemble based sample selection. (arXiv:2311.04588v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.04588
• Code URL: https://github.com/akshitjindal1/aot_wacv
• Copy Paste: [[2311.04588]] Army of Thieves: Enhancing Black-Box Model Extraction via Ensemble based sample selection(http://arxiv.org/abs/2311.04588)
• Summary:

  Machine Learning (ML) models become vulnerable to Model Stealing Attacks (MSA) when they are deployed as a service. In such attacks, the deployed model is queried repeatedly to build a labelled dataset. This dataset allows the attacker to train a thief model that mimics the original model. To maximize query efficiency, the attacker has to select the most informative subset of data points from the pool of available data. Existing attack strategies utilize approaches like Active Learning and Semi-Supervised learning to minimize costs. However, in the black-box setting, these approaches may select sub-optimal samples as they train only one thief model. Depending on the thief model's capacity and the data it was pretrained on, the model might even select noisy samples that harm the learning process. In this work, we explore the usage of an ensemble of deep learning models as our thief model. We call our attack Army of Thieves(AOT) as we train multiple models with varying complexities to leverage the crowd's wisdom. Based on the ensemble's collective decision, uncertain samples are selected for querying, while the most confident samples are directly included in the training data. Our approach is the first one to utilize an ensemble of thief models to perform model extraction. We outperform the base approaches of existing state-of-the-art methods by at least 3% and achieve a 21% higher adversarial sample transferability than previous work for models trained on the CIFAR-10 dataset.

Title: Optimized measurements of chaotic dynamical systems via the information bottleneck. (arXiv:2311.04896v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.04896
• Code URL: null
• Copy Paste: [[2311.04896]] Optimized measurements of chaotic dynamical systems via the information bottleneck(http://arxiv.org/abs/2311.04896)
• Summary:

  Deterministic chaos permits a precise notion of a "perfect measurement" as one that, when obtained repeatedly, captures all of the information created by the system's evolution with minimal redundancy. Finding an optimal measurement is challenging, and has generally required intimate knowledge of the dynamics in the few cases where it has been done. We establish an equivalence between a perfect measurement and a variant of the information bottleneck. As a consequence, we can employ machine learning to optimize measurement processes that efficiently extract information from trajectory data. We obtain approximately optimal measurements for multiple chaotic maps and lay the necessary groundwork for efficient information extraction from general time series.

membership infer

federate

Title: SaFL: Sybil-aware Federated Learning with Application to Face Recognition. (arXiv:2311.04346v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.04346
• Code URL: null
• Copy Paste: [[2311.04346]] SaFL: Sybil-aware Federated Learning with Application to Face Recognition(http://arxiv.org/abs/2311.04346)
• Summary:

  Federated Learning (FL) is a machine learning paradigm to conduct collaborative learning among clients on a joint model. The primary goal is to share clients' local training parameters with an integrating server while preserving their privacy. This method permits to exploit the potential of massive mobile users' data for the benefit of machine learning models' performance while keeping sensitive data on local devices. On the downside, FL raises security and privacy concerns that have just started to be studied. To address some of the key threats in FL, researchers have proposed to use secure aggregation methods (e.g. homomorphic encryption, secure multiparty computation, etc.). These solutions improve some security and privacy metrics, but at the same time bring about other serious threats such as poisoning attacks, backdoor attacks, and free running attacks. This paper proposes a new defense method against poisoning attacks in FL called SaFL (Sybil-aware Federated Learning) that minimizes the effect of sybils with a novel time-variant aggregation scheme.

Title: Cross-Silo Federated Learning Across Divergent Domains with Iterative Parameter Alignment. (arXiv:2311.04818v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.04818
• Code URL: https://github.com/mattgorb/iterative_parameter_alignment
• Copy Paste: [[2311.04818]] Cross-Silo Federated Learning Across Divergent Domains with Iterative Parameter Alignment(http://arxiv.org/abs/2311.04818)
• Summary:

  Learning from the collective knowledge of data dispersed across private sources can provide neural networks with enhanced generalization capabilities. Federated learning, a method for collaboratively training a machine learning model across remote clients, achieves this by combining client models via the orchestration of a central server. However, current approaches face two critical limitations: i) they struggle to converge when client domains are sufficiently different, and ii) current aggregation techniques produce an identical global model for each client. In this work, we address these issues by reformulating the typical federated learning setup: rather than learning a single global model, we learn N models each optimized for a common objective. To achieve this, we apply a weighted distance minimization to model parameters shared in a peer-to-peer topology. The resulting framework, Iterative Parameter Alignment, applies naturally to the cross-silo setting, and has the following properties: (i) a unique solution for each participant, with the option to globally converge each model in the federation, and (ii) an optional early-stopping mechanism to elicit fairness among peers in collaborative learning settings. These characteristics jointly provide a flexible new framework for iteratively learning from peer models trained on disparate datasets. We find that the technique achieves competitive results on a variety of data partitions compared to state-of-the-art approaches. Further, we show that the method is robust to divergent domains (i.e. disjoint classes across peers) where existing approaches struggle.

Title: Accurate Autism Spectrum Disorder prediction using Support Vector Classifier based on Federated Learning (SVCFL). (arXiv:2311.04606v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.04606
• Code URL: null
• Copy Paste: [[2311.04606]] Accurate Autism Spectrum Disorder prediction using Support Vector Classifier based on Federated Learning (SVCFL)(http://arxiv.org/abs/2311.04606)
• Summary:

  The path to an autism diagnosis can be long and difficult, and delays can have serious consequences. Artificial intelligence can completely change the way autism is diagnosed, especially when it comes to situations where it is difficult to see the first signs of the disease. AI-based diagnostic tools may help confirm a diagnosis or highlight the need for further testing by analyzing large volumes of data and uncovering patterns that may not be immediately apparent to human evaluators. After a successful and timely diagnosis, autism can be treated through artificial intelligence using various methods. In this article, by using four datasets and gathering them with the federated learning method and diagnosing them with the support vector classifier method, the early diagnosis of this disorder has been discussed. In this method, we have achieved 99% accuracy for predicting autism spectrum disorder and we have achieved 13% improvement in the results.

Title: Decentralized Personalized Online Federated Learning. (arXiv:2311.04817v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.04817
• Code URL: null
• Copy Paste: [[2311.04817]] Decentralized Personalized Online Federated Learning(http://arxiv.org/abs/2311.04817)
• Summary:

  Vanilla federated learning does not support learning in an online environment, learning a personalized model on each client, and learning in a decentralized setting. There are existing methods extending federated learning in each of the three aspects. However, some important applications on enterprise edge servers (e.g. online item recommendation at global scale) involve the three aspects at the same time. Therefore, we propose a new learning setting \textit{Decentralized Personalized Online Federated Learning} that considers all the three aspects at the same time.

  In this new setting for learning, the first technical challenge is how to aggregate the shared model parameters from neighboring clients to obtain a personalized local model with good performance on each client. We propose to directly learn an aggregation by optimizing the performance of the local model with respect to the aggregation weights. This not only improves personalization of each local model but also helps the local model adapting to potential data shift by intelligently incorporating the right amount of information from its neighbors. The second challenge is how to select the neighbors for each client. We propose a peer selection method based on the learned aggregation weights enabling each client to select the most helpful neighbors and reduce communication cost at the same time. We verify the effectiveness and robustness of our proposed method on three real-world item recommendation datasets and one air quality prediction dataset.

fair

Title: Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMs. (arXiv:2311.04892v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.04892
• Code URL: https://github.com/allenai/persona-bias
• Copy Paste: [[2311.04892]] Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMs(http://arxiv.org/abs/2311.04892)
• Summary:

  Recent works have showcased the ability of large-scale language models (LLMs) to embody diverse personas in their responses, exemplified by prompts like 'You are Yoda. Explain the Theory of Relativity.' While this ability allows personalization of LLMs and enables human behavior simulation, its effect on LLMs' capabilities remain unclear. To fill this gap, we present the first extensive study of the unintended side-effects of persona assignment on the ability of LLMs, specifically ChatGPT, to perform basic reasoning tasks. Our study covers 24 reasoning datasets and 16 diverse personas spanning 5 socio-demographic groups: race, gender, religion, disability, and political affiliation. Our experiments unveil that ChatGPT carries deep rooted bias against various socio-demographics underneath a veneer of fairness. While it overtly rejects stereotypes when explicitly asked ('Are Black people less skilled at mathematics?'), it manifests stereotypical and often erroneous presumptions when prompted to answer questions while taking on a persona. These can be observed as abstentions in the model responses, e.g., 'As a Black person, I am unable to answer this question as it requires math knowledge', and generally result in a substantial drop in performance on reasoning tasks. We find that this inherent deep bias is ubiquitous - 80% of our personas demonstrated bias; it is significant - certain datasets had relative drops in performance of 70%+; and can be especially harmful for certain groups - certain personas had stat. sign. drops on more than 80% of the datasets. Further analysis shows that these persona-induced errors can be hard-to-discern and hard-to-avoid. Our findings serve as a cautionary tale that the practice of assigning personas to LLMs - a trend on the rise - can surface their deep-rooted biases and have unforeseeable and detrimental side-effects.

Title: HKTGNN: Hierarchical Knowledge Transferable Graph Neural Network-based Supply Chain Risk Assessment. (arXiv:2311.04244v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.04244
• Code URL: null
• Copy Paste: [[2311.04244]] HKTGNN: Hierarchical Knowledge Transferable Graph Neural Network-based Supply Chain Risk Assessment(http://arxiv.org/abs/2311.04244)
• Summary:

  The strength of a supply chain is an important measure of a country's or region's technical advancement and overall competitiveness. Establishing supply chain risk assessment models for effective management and mitigation of potential risks has become increasingly crucial. As the number of businesses grows, the important relationships become more complicated and difficult to measure. This emphasizes the need of extracting relevant information from graph data. Previously, academics mostly employed knowledge inference to increase the visibility of links between nodes in the supply chain. However, they have not solved the data hunger problem of single node feature characteristics. We propose a hierarchical knowledge transferable graph neural network-based (HKTGNN) supply chain risk assessment model to address these issues. Our approach is based on current graph embedding methods for assessing corporate investment risk assessment. We embed the supply chain network corresponding to individual goods in the supply chain using the graph embedding module, resulting in a directed homogeneous graph with just product nodes. This reduces the complicated supply chain network into a basic product network. It addresses difficulties using the domain difference knowledge transferable module based on centrality, which is presented by the premise that supply chain feature characteristics may be biased in the actual world. Meanwhile, the feature complement and message passing will alleviate the data hunger problem, which is driven by domain differences. Our model outperforms in experiments on a real-world supply chain dataset. We will give an equation to prove that our comparative experiment is both effective and fair.

interpretability

Title: DACBERT: Leveraging Dependency Agreement for Cost-Efficient Bert Pretraining. (arXiv:2311.04799v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.04799
• Code URL: null
• Copy Paste: [[2311.04799]] DACBERT: Leveraging Dependency Agreement for Cost-Efficient Bert Pretraining(http://arxiv.org/abs/2311.04799)
• Summary:

  Building on the cost-efficient pretraining advancements brought about by Crammed BERT, we enhance its performance and interpretability further by introducing a novel pretrained model Dependency Agreement Crammed BERT (DACBERT) and its two-stage pretraining framework - Dependency Agreement Pretraining. This framework, grounded by linguistic theories, seamlessly weaves syntax and semantic information into the pretraining process. The first stage employs four dedicated submodels to capture representative dependency agreements at the chunk level, effectively converting these agreements into embeddings. The second stage uses these refined embeddings, in tandem with conventional BERT embeddings, to guide the pretraining of the rest of the model. Evaluated on the GLUE benchmark, our DACBERT demonstrates notable improvement across various tasks, surpassing Crammed BERT by 3.13% in the RTE task and by 2.26% in the MRPC task. Furthermore, our method boosts the average GLUE score by 0.83%, underscoring its significant potential. The pretraining process can be efficiently executed on a single GPU within a 24-hour cycle, necessitating no supplementary computational resources or extending the pretraining duration compared with the Crammed BERT. Extensive studies further illuminate our approach's instrumental role in bolstering the interpretability of pretrained language models for natural language understanding tasks.

explainability

watermark

Title: Watermarks in the Sand: Impossibility of Strong Watermarking for Generative Models. (arXiv:2311.04378v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.04378
• Code URL: https://github.com/hlzhang109/impossibility-watermark
• Copy Paste: [[2311.04378]] Watermarks in the Sand: Impossibility of Strong Watermarking for Generative Models(http://arxiv.org/abs/2311.04378)
• Summary:

  Watermarking generative models consists of planting a statistical signal (watermark) in a model's output so that it can be later verified that the output was generated by the given model. A strong watermarking scheme satisfies the property that a computationally bounded attacker cannot erase the watermark without causing significant quality degradation. In this paper, we study the (im)possibility of strong watermarking schemes. We prove that, under well-specified and natural assumptions, strong watermarking is impossible to achieve. This holds even in the private detection algorithm setting, where the watermark insertion and detection algorithms share a secret key, unknown to the attacker. To prove this result, we introduce a generic efficient watermark attack; the attacker is not required to know the private key of the scheme or even which scheme is used. Our attack is based on two assumptions: (1) The attacker has access to a "quality oracle" that can evaluate whether a candidate output is a high-quality response to a prompt, and (2) The attacker has access to a "perturbation oracle" which can modify an output with a nontrivial probability of maintaining quality, and which induces an efficiently mixing random walk on high-quality outputs. We argue that both assumptions can be satisfied in practice by an attacker with weaker computational capabilities than the watermarked model itself, to which the attacker has only black-box access. Furthermore, our assumptions will likely only be easier to satisfy over time as models grow in capabilities and modalities. We demonstrate the feasibility of our attack by instantiating it to attack three existing watermarking schemes for large language models: Kirchenbauer et al. (2023), Kuditipudi et al. (2023), and Zhao et al. (2023). The same attack successfully removes the watermarks planted by all three schemes, with only minor quality degradation.

diffusion

Title: A Data Perspective on Enhanced Identity Preservation for Diffusion Personalization. (arXiv:2311.04315v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.04315
• Code URL: null
• Copy Paste: [[2311.04315]] A Data Perspective on Enhanced Identity Preservation for Diffusion Personalization(http://arxiv.org/abs/2311.04315)
• Summary:

  Large text-to-image models have revolutionized the ability to generate imagery using natural language. However, particularly unique or personal visual concepts, such as your pet, an object in your house, etc., will not be captured by the original model. This has led to interest in how to inject new visual concepts, bound to a new text token, using as few as 4-6 examples. Despite significant progress, this task remains a formidable challenge, particularly in preserving the subject's identity. While most researchers attempt to to address this issue by modifying model architectures, our approach takes a data-centric perspective, advocating the modification of data rather than the model itself. We introduce a novel regularization dataset generation strategy on both the text and image level; demonstrating the importance of a rich and structured regularization dataset (automatically generated) to prevent losing text coherence and better identity preservation. The better quality is enabled by allowing up to 5x more fine-tuning iterations without overfitting and degeneration. The generated renditions of the desired subject preserve even fine details such as text and logos; all while maintaining the ability to generate diverse samples that follow the input text prompt. Since our method focuses on data augmentation, rather than adjusting the model architecture, it is complementary and can be combined with prior work. We show on established benchmarks that our data-centric approach forms the new state of the art in terms of image quality, with the best trade-off between identity preservation, diversity, and text alignment.

Title: 3DiffTection: 3D Object Detection with Geometry-Aware Diffusion Features. (arXiv:2311.04391v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.04391
• Code URL: null
• Copy Paste: [[2311.04391]] 3DiffTection: 3D Object Detection with Geometry-Aware Diffusion Features(http://arxiv.org/abs/2311.04391)
• Summary:

  We present 3DiffTection, a state-of-the-art method for 3D object detection from single images, leveraging features from a 3D-aware diffusion model. Annotating large-scale image data for 3D detection is resource-intensive and time-consuming. Recently, pretrained large image diffusion models have become prominent as effective feature extractors for 2D perception tasks. However, these features are initially trained on paired text and image data, which are not optimized for 3D tasks, and often exhibit a domain gap when applied to the target data. Our approach bridges these gaps through two specialized tuning strategies: geometric and semantic. For geometric tuning, we fine-tune a diffusion model to perform novel view synthesis conditioned on a single image, by introducing a novel epipolar warp operator. This task meets two essential criteria: the necessity for 3D awareness and reliance solely on posed image data, which are readily available (e.g., from videos) and does not require manual annotation. For semantic refinement, we further train the model on target data with detection supervision. Both tuning phases employ ControlNet to preserve the integrity of the original feature capabilities. In the final step, we harness these enhanced capabilities to conduct a test-time prediction ensemble across multiple virtual viewpoints. Through our methodology, we obtain 3D-aware features that are tailored for 3D detection and excel in identifying cross-view point correspondences. Consequently, our model emerges as a powerful 3D detector, substantially surpassing previous benchmarks, e.g., Cube-RCNN, a precedent in single-view 3D detection by 9.43\% in AP3D on the Omni3D-ARkitscene dataset. Furthermore, 3DiffTection showcases robust data efficiency and generalization to cross-domain data.

Title: Weakly-supervised deepfake localization in diffusion-generated images. (arXiv:2311.04584v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.04584
• Code URL: null
• Copy Paste: [[2311.04584]] Weakly-supervised deepfake localization in diffusion-generated images(http://arxiv.org/abs/2311.04584)
• Summary:

  The remarkable generative capabilities of denoising diffusion models have raised new concerns regarding the authenticity of the images we see every day on the Internet. However, the vast majority of existing deepfake detection models are tested against previous generative approaches (e.g. GAN) and usually provide only a "fake" or "real" label per image. We believe a more informative output would be to augment the per-image label with a localization map indicating which regions of the input have been manipulated. To this end, we frame this task as a weakly-supervised localization problem and identify three main categories of methods (based on either explanations, local scores or attention), which we compare on an equal footing by using the Xception network as the common backbone architecture. We provide a careful analysis of all the main factors that parameterize the design space: choice of method, type of supervision, dataset and generator used in the creation of manipulated images; our study is enabled by constructing datasets in which only one of the components is varied. Our results show that weakly-supervised localization is attainable, with the best performing detection method (based on local scores) being less sensitive to the looser supervision than to the mismatch in terms of dataset or generator.

noise learning

data-free

transformer

Title: LRM: Large Reconstruction Model for Single Image to 3D. (arXiv:2311.04400v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.04400
• Code URL: null
• Copy Paste: [[2311.04400]] LRM: Large Reconstruction Model for Single Image to 3D(http://arxiv.org/abs/2311.04400)
• Summary:

  We propose the first Large Reconstruction Model (LRM) that predicts the 3D model of an object from a single input image within just 5 seconds. In contrast to many previous methods that are trained on small-scale datasets such as ShapeNet in a category-specific fashion, LRM adopts a highly scalable transformer-based architecture with 500 million learnable parameters to directly predict a neural radiance field (NeRF) from the input image. We train our model in an end-to-end manner on massive multi-view data containing around 1 million objects, including both synthetic renderings from Objaverse and real captures from MVImgNet. This combination of a high-capacity model and large-scale training data empowers our model to be highly generalizable and produce high-quality 3D reconstructions from various testing inputs including real-world in-the-wild captures and images from generative models. Video demos and interactable 3D meshes can be found on this website: https://yiconghong.me/LRM/.

Title: Towards Few-Annotation Learning in Computer Vision: Application to Image Classification and Object Detection tasks. (arXiv:2311.04888v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.04888
• Code URL: null
• Copy Paste: [[2311.04888]] Towards Few-Annotation Learning in Computer Vision: Application to Image Classification and Object Detection tasks(http://arxiv.org/abs/2311.04888)
• Summary:

  In this thesis, we develop theoretical, algorithmic and experimental contributions for Machine Learning with limited labels, and more specifically for the tasks of Image Classification and Object Detection in Computer Vision. In a first contribution, we are interested in bridging the gap between theory and practice for popular Meta-Learning algorithms used in Few-Shot Classification. We make connections to Multi-Task Representation Learning, which benefits from solid theoretical foundations, to verify the best conditions for a more efficient meta-learning. Then, to leverage unlabeled data when training object detectors based on the Transformer architecture, we propose both an unsupervised pretraining and a semi-supervised learning method in two other separate contributions. For pretraining, we improve Contrastive Learning for object detectors by introducing the localization information. Finally, our semi-supervised method is the first tailored to transformer-based detectors.

Title: Uncovering Causal Variables in Transformers using Circuit Probing. (arXiv:2311.04354v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.04354
• Code URL: https://github.com/mlepori1/circuit_probing
• Copy Paste: [[2311.04354]] Uncovering Causal Variables in Transformers using Circuit Probing(http://arxiv.org/abs/2311.04354)
• Summary:

  Neural network models have achieved high performance on a wide variety of complex tasks, but the algorithms that they implement are notoriously difficult to interpret. In order to understand these algorithms, it is often necessary to hypothesize intermediate variables involved in the network's computation. For example, does a language model depend on particular syntactic properties when generating a sentence? However, existing analysis tools make it difficult to test hypotheses of this type. We propose a new analysis technique -- circuit probing -- that automatically uncovers low-level circuits that compute hypothesized intermediate variables. This enables causal analysis through targeted ablation at the level of model parameters. We apply this method to models trained on simple arithmetic tasks, demonstrating its effectiveness at (1) deciphering the algorithms that models have learned, (2) revealing modular structure within a model, and (3) tracking the development of circuits over training. We compare circuit probing to other methods across these three experiments, and find it on par or more effective than existing analysis methods. Finally, we demonstrate circuit probing on a real-world use case, uncovering circuits that are responsible for subject-verb agreement and reflexive anaphora in GPT2-Small and Medium.

Title: Syntax-Guided Transformers: Elevating Compositional Generalization and Grounding in Multimodal Environments. (arXiv:2311.04364v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.04364
• Code URL: https://github.com/hlr/syntax-guided-transformers
• Copy Paste: [[2311.04364]] Syntax-Guided Transformers: Elevating Compositional Generalization and Grounding in Multimodal Environments(http://arxiv.org/abs/2311.04364)
• Summary:

  Compositional generalization, the ability of intelligent models to extrapolate understanding of components to novel compositions, is a fundamental yet challenging facet in AI research, especially within multimodal environments. In this work, we address this challenge by exploiting the syntactic structure of language to boost compositional generalization. This paper elevates the importance of syntactic grounding, particularly through attention masking techniques derived from text input parsing. We introduce and evaluate the merits of using syntactic information in the multimodal grounding problem. Our results on grounded compositional generalization underscore the positive impact of dependency parsing across diverse tasks when utilized with Weight Sharing across the Transformer encoder. The results push the state-of-the-art in multimodal grounding and parameter-efficient modeling and provide insights for future research.

Title: Data Factors for Better Compositional Generalization. (arXiv:2311.04420v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.04420
• Code URL: https://github.com/owenzx/data4comp
• Copy Paste: [[2311.04420]] Data Factors for Better Compositional Generalization(http://arxiv.org/abs/2311.04420)
• Summary:

  Recent diagnostic datasets on compositional generalization, such as SCAN (Lake and Baroni, 2018) and COGS (Kim and Linzen, 2020), expose severe problems in models trained from scratch on these datasets. However, in contrast to this poor performance, state-of-the-art models trained on larger and more general datasets show better generalization ability. In this work, to reconcile this inconsistency, we conduct an empirical analysis by training Transformer models on a variety of training sets with different data factors, including dataset scale, pattern complexity, example difficulty, etc. First, we show that increased dataset complexity can lead to better generalization behavior on multiple different generalization challenges. To further understand this improvement, we show two axes of the benefit from more complex datasets: they provide more diverse examples so compositional understanding becomes more effective, and they also prevent ungeneralizable memorization of the examples due to reduced example repetition frequency. Finally, we explore how training examples of different difficulty levels influence generalization differently. On synthetic datasets, simple examples invoke stronger compositionality than hard examples do. On larger-scale real language datasets, while hard examples become more important potentially to ensure decent data coverage, a balanced mixture of simple and hard examples manages to induce the strongest generalizability. The code and data for this work are available at https://github.com/owenzx/data4comp

Title: Recursion in Recursion: Two-Level Nested Recursion for Length Generalization with Scalability. (arXiv:2311.04449v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.04449
• Code URL: https://github.com/jrc1995/beamrecursionfamily
• Copy Paste: [[2311.04449]] Recursion in Recursion: Two-Level Nested Recursion for Length Generalization with Scalability(http://arxiv.org/abs/2311.04449)
• Summary:

  Binary Balanced Tree RvNNs (BBT-RvNNs) enforce sequence composition according to a preset balanced binary tree structure. Thus, their non-linear recursion depth is just $\log_2 n$ ($n$ being the sequence length). Such logarithmic scaling makes BBT-RvNNs efficient and scalable on long sequence tasks such as Long Range Arena (LRA). However, such computational efficiency comes at a cost because BBT-RvNNs cannot solve simple arithmetic tasks like ListOps. On the flip side, RvNNs (e.g., Beam Tree RvNN) that do succeed on ListOps (and other structure-sensitive tasks like formal logical inference) are generally several times more expensive than even RNNs. In this paper, we introduce a novel framework -- Recursion in Recursion (RIR) to strike a balance between the two sides - getting some of the benefits from both worlds. In RIR, we use a form of two-level nested recursion - where the outer recursion is a $k$-ary balanced tree model with another recursive model (inner recursion) implementing its cell function. For the inner recursion, we choose Beam Tree RvNNs (BT-RvNN). To adjust BT-RvNNs within RIR we also propose a novel strategy of beam alignment. Overall, this entails that the total recursive depth in RIR is upper-bounded by $k \log_k n$. Our best RIR-based model is the first model that demonstrates high ($\geq 90\%$) length-generalization performance on ListOps while at the same time being scalable enough to be trainable on long sequence inputs from LRA. Moreover, in terms of accuracy in the LRA language tasks, it performs competitively with Structured State Space Models (SSMs) without any special initialization - outperforming Transformers by a large margin. On the other hand, while SSMs can marginally outperform RIR on LRA, they (SSMs) fail to length-generalize on ListOps. Our code is available at: \url{https://github.com/JRC1995/BeamRecursionFamily/}.

Title: Loss Masking Is Not Needed in Decoder-only Transformer for Discrete-token Based ASR. (arXiv:2311.04534v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.04534
• Code URL: null
• Copy Paste: [[2311.04534]] Loss Masking Is Not Needed in Decoder-only Transformer for Discrete-token Based ASR(http://arxiv.org/abs/2311.04534)
• Summary:

  Recently, unified speech-text models, such as SpeechGPT, VioLA, and AudioPaLM, have achieved remarkable performance on speech tasks. These models convert continuous speech signals into discrete tokens (speech discretization) and merge text and speech tokens into a shared vocabulary. Then they train a single decoder-only Transformer on a mixture of speech tasks. Specifically, all these models utilize Loss Masking on the input speech tokens for the ASR task, which means that these models do not explicitly model the dependency between the speech tokens. In this paper, we attempt to model the sequence of speech tokens in an autoregressive manner like text. However, we find that applying the conventional cross-entropy loss on input speech tokens does not consistently improve the ASR performance over Loss Masking. Therefore, we propose a novel approach denoted Smoothed Label Distillation (SLD), which introduces a KL divergence loss with smoothed labels on the input speech tokens to effectively model speech tokens. Experiments demonstrate that our SLD approach alleviates the limitations of the cross-entropy loss and consistently outperforms Loss Masking for decoder-only Transformer based ASR using different speech discretization methods.

Title: Determination of toxic comments and unintended model bias minimization using Deep learning approach. (arXiv:2311.04789v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.04789
• Code URL: null
• Copy Paste: [[2311.04789]] Determination of toxic comments and unintended model bias minimization using Deep learning approach(http://arxiv.org/abs/2311.04789)
• Summary:

  Online conversations can be toxic and subjected to threats, abuse, or harassment. To identify toxic text comments, several deep learning and machine learning models have been proposed throughout the years. However, recent studies demonstrate that because of the imbalances in the training data, some models are more likely to show unintended biases including gender bias and identity bias. In this research, our aim is to detect toxic comment and reduce the unintended bias concerning identity features such as race, gender, sex, religion by fine-tuning an attention based model called BERT(Bidirectional Encoder Representation from Transformers). We apply weighted loss to address the issue of unbalanced data and compare the performance of a fine-tuned BERT model with a traditional Logistic Regression model in terms of classification and bias minimization. The Logistic Regression model with the TFIDF vectorizer achieve 57.1% accuracy, and fine-tuned BERT model's accuracy is 89%. Code is available at https://github.com/zim10/Determine_Toxic_comment_and_identity_bias.git

Title: Hierarchically Gated Recurrent Neural Network for Sequence Modeling. (arXiv:2311.04823v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.04823
• Code URL: https://github.com/opennlplab/hgrn
• Copy Paste: [[2311.04823]] Hierarchically Gated Recurrent Neural Network for Sequence Modeling(http://arxiv.org/abs/2311.04823)
• Summary:

  Transformers have surpassed RNNs in popularity due to their superior abilities in parallel training and long-term dependency modeling. Recently, there has been a renewed interest in using linear RNNs for efficient sequence modeling. These linear RNNs often employ gating mechanisms in the output of the linear recurrence layer while ignoring the significance of using forget gates within the recurrence. In this paper, we propose a gated linear RNN model dubbed Hierarchically Gated Recurrent Neural Network (HGRN), which includes forget gates that are lower bounded by a learnable value. The lower bound increases monotonically when moving up layers. This allows the upper layers to model long-term dependencies and the lower layers to model more local, short-term dependencies. Experiments on language modeling, image classification, and long-range arena benchmarks showcase the efficiency and effectiveness of our proposed model. The source code is available at https://github.com/OpenNLPLab/HGRN.

Title: Future Lens: Anticipating Subsequent Tokens from a Single Hidden State. (arXiv:2311.04897v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.04897
• Code URL: null
• Copy Paste: [[2311.04897]] Future Lens: Anticipating Subsequent Tokens from a Single Hidden State(http://arxiv.org/abs/2311.04897)
• Summary:

  We conjecture that hidden state vectors corresponding to individual input tokens encode information sufficient to accurately predict several tokens ahead. More concretely, in this paper we ask: Given a hidden (internal) representation of a single token at position $t$ in an input, can we reliably anticipate the tokens that will appear at positions $\geq t + 2$? To test this, we measure linear approximation and causal intervention methods in GPT-J-6B to evaluate the degree to which individual hidden states in the network contain signal rich enough to predict future hidden states and, ultimately, token outputs. We find that, at some layers, we can approximate a model's output with more than 48% accuracy with respect to its prediction of subsequent tokens through a single hidden state. Finally we present a "Future Lens" visualization that uses these methods to create a new view of transformer states.

Title: A Hierarchical Spatial Transformer for Massive Point Samples in Continuous Space. (arXiv:2311.04434v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.04434
• Code URL: https://github.com/spatialdatasciencegroup/hst
• Copy Paste: [[2311.04434]] A Hierarchical Spatial Transformer for Massive Point Samples in Continuous Space(http://arxiv.org/abs/2311.04434)
• Summary:

  Transformers are widely used deep learning architectures. Existing transformers are mostly designed for sequences (texts or time series), images or videos, and graphs. This paper proposes a novel transformer model for massive (up to a million) point samples in continuous space. Such data are ubiquitous in environment sciences (e.g., sensor observations), numerical simulations (e.g., particle-laden flow, astrophysics), and location-based services (e.g., POIs and trajectories). However, designing a transformer for massive spatial points is non-trivial due to several challenges, including implicit long-range and multi-scale dependency on irregular points in continuous space, a non-uniform point distribution, the potential high computational costs of calculating all-pair attention across massive points, and the risks of over-confident predictions due to varying point density. To address these challenges, we propose a new hierarchical spatial transformer model, which includes multi-resolution representation learning within a quad-tree hierarchy and efficient spatial attention via coarse approximation. We also design an uncertainty quantification branch to estimate prediction confidence related to input feature noise and point sparsity. We provide a theoretical analysis of computational time complexity and memory costs. Extensive experiments on both real-world and synthetic datasets show that our method outperforms multiple baselines in prediction accuracy and our model can scale up to one million points on one NVIDIA A100 GPU. The code is available at \url{https://github.com/spatialdatasciencegroup/HST}.

Title: Long-term Time Series Forecasting based on Decomposition and Neural Ordinary Differential Equations. (arXiv:2311.04522v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.04522
• Code URL: null
• Copy Paste: [[2311.04522]] Long-term Time Series Forecasting based on Decomposition and Neural Ordinary Differential Equations(http://arxiv.org/abs/2311.04522)
• Summary:

  Long-term time series forecasting (LTSF) is a challenging task that has been investigated in various domains such as finance investment, health care, traffic, and weather forecasting. In recent years, Linear-based LTSF models showed better performance, pointing out the problem of Transformer-based approaches causing temporal information loss. However, Linear-based approach has also limitations that the model is too simple to comprehensively exploit the characteristics of the dataset. To solve these limitations, we propose LTSF-DNODE, which applies a model based on linear ordinary differential equations (ODEs) and a time series decomposition method according to data statistical characteristics. We show that LTSF-DNODE outperforms the baselines on various real-world datasets. In addition, for each dataset, we explore the impacts of regularization in the neural ordinary differential equation (NODE) framework.

Title: Hybrid Focal and Full-Range Attention Based Graph Transformers. (arXiv:2311.04653v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.04653
• Code URL: null
• Copy Paste: [[2311.04653]] Hybrid Focal and Full-Range Attention Based Graph Transformers(http://arxiv.org/abs/2311.04653)
• Summary:

  The paradigm of Transformers using the self-attention mechanism has manifested its advantage in learning graph-structured data. Yet, Graph Transformers are capable of modeling full range dependencies but are often deficient in extracting information from locality. A common practice is to utilize Message Passing Neural Networks (MPNNs) as an auxiliary to capture local information, which however are still inadequate for comprehending substructures. In this paper, we present a purely attention-based architecture, namely Focal and Full-Range Graph Transformer (FFGT), which can mitigate the loss of local information in learning global correlations. The core component of FFGT is a new mechanism of compound attention, which combines the conventional full-range attention with K-hop focal attention on ego-nets to aggregate both global and local information. Beyond the scope of canonical Transformers, the FFGT has the merit of being more substructure-aware. Our approach enhances the performance of existing Graph Transformers on various open datasets, while achieves compatible SOTA performance on several Long-Range Graph Benchmark (LRGB) datasets even with a vanilla transformer. We further examine influential factors on the optimal focal length of attention via introducing a novel synthetic dataset based on SBM-PATTERN.

Title: Euclidean, Projective, Conformal: Choosing a Geometric Algebra for Equivariant Transformers. (arXiv:2311.04744v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.04744
• Code URL: null
• Copy Paste: [[2311.04744]] Euclidean, Projective, Conformal: Choosing a Geometric Algebra for Equivariant Transformers(http://arxiv.org/abs/2311.04744)
• Summary:

  The Geometric Algebra Transformer (GATr) is a versatile architecture for geometric deep learning based on projective geometric algebra. We generalize this architecture into a blueprint that allows one to construct a scalable transformer architecture given any geometric (or Clifford) algebra. We study versions of this architecture for Euclidean, projective, and conformal algebras, all of which are suited to represent 3D data, and evaluate them in theory and practice. The simplest Euclidean architecture is computationally cheap, but has a smaller symmetry group and is not as sample-efficient, while the projective model is not sufficiently expressive. Both the conformal algebra and an improved version of the projective algebra define powerful, performant architectures.

Title: Vital Sign Forecasting for Sepsis Patients in ICUs. (arXiv:2311.04770v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.04770
• Code URL: null
• Copy Paste: [[2311.04770]] Vital Sign Forecasting for Sepsis Patients in ICUs(http://arxiv.org/abs/2311.04770)
• Summary:

  Sepsis and septic shock are a critical medical condition affecting millions globally, with a substantial mortality rate. This paper uses state-of-the-art deep learning (DL) architectures to introduce a multi-step forecasting system to predict vital signs indicative of septic shock progression in Intensive Care Units (ICUs). Our approach utilizes a short window of historical vital sign data to forecast future physiological conditions. We introduce a DL-based vital sign forecasting system that predicts up to 3 hours of future vital signs from 6 hours of past data. We further adopt the DILATE loss function to capture better the shape and temporal dynamics of vital signs, which are critical for clinical decision-making. We compare three DL models, N-BEATS, N-HiTS, and Temporal Fusion Transformer (TFT), using the publicly available eICU Collaborative Research Database (eICU-CRD), highlighting their forecasting capabilities in a critical care setting. We evaluate the performance of our models using mean squared error (MSE) and dynamic time warping (DTW) metrics. Our findings show that while TFT excels in capturing overall trends, N-HiTS is superior in retaining short-term fluctuations within a predefined range. This paper demonstrates the potential of deep learning in transforming the monitoring systems in ICUs, potentially leading to significant improvements in patient care and outcomes by accurately forecasting vital signs to assist healthcare providers in detecting early signs of physiological instability and anticipating septic shock.

generative

Title: Social Motion Prediction with Cognitive Hierarchies. (arXiv:2311.04726v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.04726
• Code URL: null
• Copy Paste: [[2311.04726]] Social Motion Prediction with Cognitive Hierarchies(http://arxiv.org/abs/2311.04726)
• Summary:

  Humans exhibit a remarkable capacity for anticipating the actions of others and planning their own actions accordingly. In this study, we strive to replicate this ability by addressing the social motion prediction problem. We introduce a new benchmark, a novel formulation, and a cognition-inspired framework. We present Wusi, a 3D multi-person motion dataset under the context of team sports, which features intense and strategic human interactions and diverse pose distributions. By reformulating the problem from a multi-agent reinforcement learning perspective, we incorporate behavioral cloning and generative adversarial imitation learning to boost learning efficiency and generalization. Furthermore, we take into account the cognitive aspects of the human social action planning process and develop a cognitive hierarchy framework to predict strategic human social interactions. We conduct comprehensive experiments to validate the effectiveness of our proposed dataset and approach. Code and data are available at https://walter0807.github.io/Social-CH/.

Title: GENOME: GenerativE Neuro-symbOlic visual reasoning by growing and reusing ModulEs. (arXiv:2311.04901v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.04901
• Code URL: null
• Copy Paste: [[2311.04901]] GENOME: GenerativE Neuro-symbOlic visual reasoning by growing and reusing ModulEs(http://arxiv.org/abs/2311.04901)
• Summary:

  Recent works have shown that Large Language Models (LLMs) could empower traditional neuro-symbolic models via programming capabilities to translate language into module descriptions, thus achieving strong visual reasoning results while maintaining the model's transparency and efficiency. However, these models usually exhaustively generate the entire code snippet given each new instance of a task, which is extremely ineffective. We propose generative neuro-symbolic visual reasoning by growing and reusing modules. Specifically, our model consists of three unique stages, module initialization, module generation, and module execution. First, given a vision-language task, we adopt LLMs to examine whether we could reuse and grow over established modules to handle this new task. If not, we initialize a new module needed by the task and specify the inputs and outputs of this new module. After that, the new module is created by querying LLMs to generate corresponding code snippets that match the requirements. In order to get a better sense of the new module's ability, we treat few-shot training examples as test cases to see if our new module could pass these cases. If yes, the new module is added to the module library for future reuse. Finally, we evaluate the performance of our model on the testing set by executing the parsed programs with the newly made visual modules to get the results. We find the proposed model possesses several advantages. First, it performs competitively on standard tasks like visual question answering and referring expression comprehension; Second, the modules learned from one task can be seamlessly transferred to new tasks; Last but not least, it is able to adapt to new visual reasoning tasks by observing a few training examples and reusing modules.

Title: GPT-ST: Generative Pre-Training of Spatio-Temporal Graph Neural Networks. (arXiv:2311.04245v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.04245
• Code URL: https://github.com/hkuds/gpt-st
• Copy Paste: [[2311.04245]] GPT-ST: Generative Pre-Training of Spatio-Temporal Graph Neural Networks(http://arxiv.org/abs/2311.04245)
• Summary:

  In recent years, there has been a rapid development of spatio-temporal prediction techniques in response to the increasing demands of traffic management and travel planning. While advanced end-to-end models have achieved notable success in improving predictive performance, their integration and expansion pose significant challenges. This work aims to address these challenges by introducing a spatio-temporal pre-training framework that seamlessly integrates with downstream baselines and enhances their performance. The framework is built upon two key designs: (i) We propose a spatio-temporal mask autoencoder as a pre-training model for learning spatio-temporal dependencies. The model incorporates customized parameter learners and hierarchical spatial pattern encoding networks. These modules are specifically designed to capture spatio-temporal customized representations and intra- and inter-cluster region semantic relationships, which have often been neglected in existing approaches. (ii) We introduce an adaptive mask strategy as part of the pre-training mechanism. This strategy guides the mask autoencoder in learning robust spatio-temporal representations and facilitates the modeling of different relationships, ranging from intra-cluster to inter-cluster, in an easy-to-hard training manner. Extensive experiments conducted on representative benchmarks demonstrate the effectiveness of our proposed method. We have made our model implementation publicly available at https://github.com/HKUDS/GPT-ST.

large language model

Title: mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration. (arXiv:2311.04257v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.04257
• Code URL: https://github.com/x-plug/mplug-owl
• Copy Paste: [[2311.04257]] mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration(http://arxiv.org/abs/2311.04257)
• Summary:

  Multi-modal Large Language Models (MLLMs) have demonstrated impressive instruction abilities across various open-ended tasks. However, previous methods primarily focus on enhancing multi-modal capabilities. In this work, we introduce a versatile multi-modal large language model, mPLUG-Owl2, which effectively leverages modality collaboration to improve performance in both text and multi-modal tasks. mPLUG-Owl2 utilizes a modularized network design, with the language decoder acting as a universal interface for managing different modalities. Specifically, mPLUG-Owl2 incorporates shared functional modules to facilitate modality collaboration and introduces a modality-adaptive module that preserves modality-specific features. Extensive experiments reveal that mPLUG-Owl2 is capable of generalizing both text tasks and multi-modal tasks and achieving state-of-the-art performances with a single generic model. Notably, mPLUG-Owl2 is the first MLLM model that demonstrates the modality collaboration phenomenon in both pure-text and multi-modal scenarios, setting a pioneering path in the development of future multi-modal foundation models.

Title: Formal Aspects of Language Modeling. (arXiv:2311.04329v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.04329
• Code URL: null
• Copy Paste: [[2311.04329]] Formal Aspects of Language Modeling(http://arxiv.org/abs/2311.04329)
• Summary:

  Large language models have become one of the most commonly deployed NLP inventions. In the past half-decade, their integration into core natural language processing tools has dramatically increased the performance of such tools, and they have entered the public discourse surrounding artificial intelligence. Consequently, it is important for both developers and researchers alike to understand the mathematical foundations of large language models, as well as how to implement them. These notes are the accompaniment to the theoretical portion of the ETH Z\"urich course on large language models, covering what constitutes a language model from a formal, theoretical perspective.

Title: Evaluating the Effectiveness of Retrieval-Augmented Large Language Models in Scientific Document Reasoning. (arXiv:2311.04348v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.04348
• Code URL: null
• Copy Paste: [[2311.04348]] Evaluating the Effectiveness of Retrieval-Augmented Large Language Models in Scientific Document Reasoning(http://arxiv.org/abs/2311.04348)
• Summary:

  Despite the dramatic progress in Large Language Model (LLM) development, LLMs often provide seemingly plausible but not factual information, often referred to as hallucinations. Retrieval-augmented LLMs provide a non-parametric approach to solve these issues by retrieving relevant information from external data sources and augment the training process. These models help to trace evidence from an externally provided knowledge base allowing the model predictions to be better interpreted and verified. In this work, we critically evaluate these models in their ability to perform in scientific document reasoning tasks. To this end, we tuned multiple such model variants with science-focused instructions and evaluated them on a scientific document reasoning benchmark for the usefulness of the retrieved document passages. Our findings suggest that models justify predictions in science tasks with fabricated evidence and leveraging scientific corpus as pretraining data does not alleviate the risk of evidence fabrication.

Title: Evaluating multiple large language models in pediatric ophthalmology. (arXiv:2311.04368v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.04368
• Code URL: null
• Copy Paste: [[2311.04368]] Evaluating multiple large language models in pediatric ophthalmology(http://arxiv.org/abs/2311.04368)
• Summary:

  IMPORTANCE The response effectiveness of different large language models (LLMs) and various individuals, including medical students, graduate students, and practicing physicians, in pediatric ophthalmology consultations, has not been clearly established yet. OBJECTIVE Design a 100-question exam based on pediatric ophthalmology to evaluate the performance of LLMs in highly specialized scenarios and compare them with the performance of medical students and physicians at different levels. DESIGN, SETTING, AND PARTICIPANTS This survey study assessed three LLMs, namely ChatGPT (GPT-3.5), GPT-4, and PaLM2, were assessed alongside three human cohorts: medical students, postgraduate students, and attending physicians, in their ability to answer questions related to pediatric ophthalmology. It was conducted by administering questionnaires in the form of test papers through the LLM network interface, with the valuable participation of volunteers. MAIN OUTCOMES AND MEASURES Mean scores of LLM and humans on 100 multiple-choice questions, as well as the answer stability, correlation, and response confidence of each LLM. RESULTS GPT-4 performed comparably to attending physicians, while ChatGPT (GPT-3.5) and PaLM2 outperformed medical students but slightly trailed behind postgraduate students. Furthermore, GPT-4 exhibited greater stability and confidence when responding to inquiries compared to ChatGPT (GPT-3.5) and PaLM2. CONCLUSIONS AND RELEVANCE Our results underscore the potential for LLMs to provide medical assistance in pediatric ophthalmology and suggest significant capacity to guide the education of medical students.

Title: Multi-label and Multi-target Sampling of Machine Annotation for Computational Stance Detection. (arXiv:2311.04495v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.04495
• Code URL: null
• Copy Paste: [[2311.04495]] Multi-label and Multi-target Sampling of Machine Annotation for Computational Stance Detection(http://arxiv.org/abs/2311.04495)
• Summary:

  Data collection from manual labeling provides domain-specific and task-aligned supervision for data-driven approaches, and a critical mass of well-annotated resources is required to achieve reasonable performance in natural language processing tasks. However, manual annotations are often challenging to scale up in terms of time and budget, especially when domain knowledge, capturing subtle semantic features, and reasoning steps are needed. In this paper, we investigate the efficacy of leveraging large language models on automated labeling for computational stance detection. We empirically observe that while large language models show strong potential as an alternative to human annotators, their sensitivity to task-specific instructions and their intrinsic biases pose intriguing yet unique challenges in machine annotation. We introduce a multi-label and multi-target sampling strategy to optimize the annotation quality. Experimental results on the benchmark stance detection corpora show that our method can significantly improve performance and learning efficacy.

Title: TEAL: Tokenize and Embed ALL for Multi-modal Large Language Models. (arXiv:2311.04589v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.04589
• Code URL: null
• Copy Paste: [[2311.04589]] TEAL: Tokenize and Embed ALL for Multi-modal Large Language Models(http://arxiv.org/abs/2311.04589)
• Summary:

  Despite Multi-modal Large Language Models (MM-LLMs) have made exciting strides recently, they are still struggling to efficiently model the interactions among multi-modal inputs and the generation in non-textual modalities. In this work, we propose TEAL (Tokenize and Embed ALl)}, an approach to treat the input from any modality as a token sequence and learn a joint embedding space for all modalities. Specifically, for the input from any modality, TEAL first discretizes it into a token sequence with the off-the-shelf tokenizer and embeds the token sequence into a joint embedding space with a learnable embedding matrix. MM-LLMs just need to predict the multi-modal tokens autoregressively as the textual LLMs do. Finally, the corresponding de-tokenizer is applied to generate the output in each modality based on the predicted token sequence. With the joint embedding space, TEAL enables the frozen LLMs to perform both understanding and generation tasks involving non-textual modalities, such as image and audio. Thus, the textual LLM can just work as an interface and maintain its high performance in textual understanding and generation. Experiments show that TEAL achieves substantial improvements in multi-modal understanding, and implements a simple scheme for multi-modal generations.

Title: Massive Editing for Large Language Models via Meta Learning. (arXiv:2311.04661v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.04661
• Code URL: https://github.com/chenmientan/malmen
• Copy Paste: [[2311.04661]] Massive Editing for Large Language Models via Meta Learning(http://arxiv.org/abs/2311.04661)
• Summary:

  While large language models (LLMs) have enabled learning knowledge from the pre-training corpora, the acquired knowledge may be fundamentally incorrect or outdated over time, which necessitates rectifying the knowledge of the language model (LM) after the training. A promising approach involves employing a hyper-network to generate parameter shift, whereas existing hyper-networks suffer from inferior scalability in synchronous editing operation amount. To mitigate the problem, we propose the MAssive Language Model Editing Network (MALMEN), which formulates the parameter shift aggregation as the least square problem, subsequently updating the LM parameters using the normal equation. To accommodate editing multiple facts simultaneously with limited memory budgets, we separate the computation on the hyper-network and LM, enabling arbitrary batch size on both neural networks. Our method is evaluated by editing up to thousands of facts on LMs with different architectures, i.e., BERT-base, GPT-2, T5-XL (2.8B), and GPT-J (6B), across various knowledge-intensive NLP tasks, i.e., closed book fact-checking and question answering. Remarkably, MALMEN is capable of editing hundreds of times more facts than strong baselines with the identical hyper-network architecture and outperforms editor specifically designed for GPT. Our code is available at https://github.com/ChenmienTan/malmen.

Title: Using large language models to study human memory for meaningful narratives. (arXiv:2311.04742v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.04742
• Code URL: https://github.com/mkatkov/llm-narrative-analysis
• Copy Paste: [[2311.04742]] Using large language models to study human memory for meaningful narratives(http://arxiv.org/abs/2311.04742)
• Summary:

  One of the most impressive achievements of the AI revolution is the development of large language models that can generate meaningful text and respond to instructions in plain English with no additional training necessary. Here we show that language models can be used as a scientific instrument for studying human memory for meaningful material. We developed a pipeline for designing large scale memory experiments and analyzing the obtained results. We performed online memory experiments with a large number of participants and collected recognition and recall data for narratives of different lengths. We found that both recall and recognition performance scale linearly with narrative length. Furthermore, in order to investigate the role of narrative comprehension in memory, we repeated these experiments using scrambled versions of the presented stories. We found that even though recall performance declined significantly, recognition remained largely unaffected. Interestingly, recalls in this condition seem to follow the original narrative order rather than the scrambled presentation, pointing to a contextual reconstruction of the story in memory.

Title: Rethinking Benchmark and Contamination for Language Models with Rephrased Samples. (arXiv:2311.04850v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.04850
• Code URL: https://github.com/lm-sys/llm-decontaminator
• Copy Paste: [[2311.04850]] Rethinking Benchmark and Contamination for Language Models with Rephrased Samples(http://arxiv.org/abs/2311.04850)
• Summary:

  Large language models are increasingly trained on all the data ever produced by humans. Many have raised concerns about the trustworthiness of public benchmarks due to potential contamination in pre-training or fine-tuning datasets. While most data decontamination efforts apply string matching (e.g., n-gram overlap) to remove benchmark data, we show that these methods are insufficient, and simple variations of test data (e.g., paraphrasing, translation) can easily bypass these decontamination measures. Furthermore, we demonstrate that if such variation of test data is not eliminated, a 13B model can easily overfit a test benchmark and achieve drastically high performance, on par with GPT-4. We validate such observations in widely used benchmarks such as MMLU, GSK8k, and HumanEval. To address this growing risk, we propose a stronger LLM-based decontamination method and apply it to widely used pre-training and fine-tuning datasets, revealing significant previously unknown test overlap. For example, in pre-training sets such as RedPajama-Data-1T and StarCoder-Data, we identified that 8-18\% of the HumanEval benchmark overlaps. Interestingly, we also find such contamination in synthetic dataset generated by GPT-3.5/4, suggesting a potential risk of unintentional contamination. We urge the community to adopt stronger decontamination approaches when using public benchmarks. Moreover, we call for the community to actively develop fresh one-time exams to evaluate models accurately. Our decontamination tool is publicly available at https://github.com/lm-sys/llm-decontaminator.

Title: LongQLoRA: Efficient and Effective Method to Extend Context Length of Large Language Models. (arXiv:2311.04879v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.04879
• Code URL: https://github.com/yangjianxin1/longqlora
• Copy Paste: [[2311.04879]] LongQLoRA: Efficient and Effective Method to Extend Context Length of Large Language Models(http://arxiv.org/abs/2311.04879)
• Summary:

  We present LongQLoRA, an efficient and effective method to extend context length of large language models with less training resources. LongQLoRA combines the advantages of Position Interpolation, QLoRA and Shift Short Attention of LongLoRA. With a single 32GB V100 GPU, LongQLoRA can extend the context length of LLaMA2 7B and 13B from 4096 to 8192 and even to 12k within 1000 finetuning steps. LongQLoRA achieves competitive perplexity performance on PG19 and Proof-pile datasets, our model outperforms LongLoRA and is very close to MPT-7B-8K within the evaluation context length of 8192. We collect and build 39k long instruction data to extend context length of Vicuna-13B from 4096 to 8192 and achieve good performance both in long and short context generation task. We also do some ablation experiments to study the effect of LoRA rank, finetuning steps and attention patterns in inference.The model weights, training data and code are avaliable at https://github.com/yangjianxin1/LongQLoRA.

Title: SEMQA: Semi-Extractive Multi-Source Question Answering. (arXiv:2311.04886v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.04886
• Code URL: https://github.com/google-research-datasets/quotesum
• Copy Paste: [[2311.04886]] SEMQA: Semi-Extractive Multi-Source Question Answering(http://arxiv.org/abs/2311.04886)
• Summary:

  Recently proposed long-form question answering (QA) systems, supported by large language models (LLMs), have shown promising capabilities. Yet, attributing and verifying their generated abstractive answers can be difficult, and automatically evaluating their accuracy remains an ongoing challenge.

  In this work, we introduce a new QA task for answering multi-answer questions by summarizing multiple diverse sources in a semi-extractive fashion. Specifically, Semi-extractive Multi-source QA (SEMQA) requires models to output a comprehensive answer, while mixing factual quoted spans -- copied verbatim from given input sources -- and non-factual free-text connectors that glue these spans together into a single cohesive passage. This setting bridges the gap between the outputs of well-grounded but constrained extractive QA systems and more fluent but harder to attribute fully abstractive answers. Particularly, it enables a new mode for language models that leverages their advanced language generation capabilities, while also producing fine in-line attributions by-design that are easy to verify, interpret, and evaluate.

  To study this task, we create the first dataset of this kind, QuoteSum, with human-written semi-extractive answers to natural and generated questions, and define text-based evaluation metrics. Experimenting with several LLMs in various settings, we find this task to be surprisingly challenging, demonstrating the importance of QuoteSum for developing and studying such consolidation capabilities.

Title: How Abstract Is Linguistic Generalization in Large Language Models? Experiments with Argument Structure. (arXiv:2311.04900v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.04900
• Code URL: https://github.com/clay-lab/structural-alternations
• Copy Paste: [[2311.04900]] How Abstract Is Linguistic Generalization in Large Language Models? Experiments with Argument Structure(http://arxiv.org/abs/2311.04900)
• Summary:

  Language models are typically evaluated on their success at predicting the distribution of specific words in specific contexts. Yet linguistic knowledge also encodes relationships between contexts, allowing inferences between word distributions. We investigate the degree to which pre-trained Transformer-based large language models (LLMs) represent such relationships, focusing on the domain of argument structure. We find that LLMs perform well in generalizing the distribution of a novel noun argument between related contexts that were seen during pre-training (e.g., the active object and passive subject of the verb spray), succeeding by making use of the semantically-organized structure of the embedding space for word embeddings. However, LLMs fail at generalizations between related contexts that have not been observed during pre-training, but which instantiate more abstract, but well-attested structural generalizations (e.g., between the active object and passive subject of an arbitrary verb). Instead, in this case, LLMs show a bias to generalize based on linear order. This finding points to a limitation with current models and points to a reason for which their training is data-intensive.s reported here are available at https://github.com/clay-lab/structural-alternations.

Title: Beyond Size: How Gradients Shape Pruning Decisions in Large Language Models. (arXiv:2311.04902v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.04902
• Code URL: https://github.com/rocktimjyotidas/gblm-pruner
• Copy Paste: [[2311.04902]] Beyond Size: How Gradients Shape Pruning Decisions in Large Language Models(http://arxiv.org/abs/2311.04902)
• Summary:

  Large Language Models (LLMs) with a billion or more parameters are prime targets for network pruning, which aims to reduce a portion of the network weights without compromising performance. Prior approaches such as Weights Magnitude, SparseGPT, and Wanda, either concentrated solely on weights or integrated weights with activations for sparsity. However, they overlooked the informative gradients derived from pretrained large language models. In this paper, we present a novel sparsity-centric pruning method for pretrained LLMs, termed Gradient-based Language Model Pruner (GBLM-Pruner). GBLM-Pruner leverages the first-order term of the Taylor expansion, operating in a training-free manner by harnessing properly normalized gradients from a few calibration samples to determine the importance pruning score, and substantially outperforms competitive counterparts like SparseGPT and Wanda in multiple benchmarks. Intriguing, after incorporating gradients, the unstructured pruning method tends to reveal some structural patterns post-pruning, which mirrors the geometric interdependence inherent in the LLMs' parameter structure. Additionally, GBLM-Pruner functions without any subsequent retraining or weight updates to maintain its simplicity as other counterparts. Extensive evaluations on LLaMA-1 and LLaMA-2 across various language benchmarks and perplexity show that GBLM-Pruner surpasses magnitude pruning, Wanda (weights+activations) and SparseGPT (weights+activations+weight update) by significant margins. Our code and models are available at https://github.com/RocktimJyotiDas/GBLM-Pruner.

segmentation

Title: ETDPC: A Multimodality Framework for Classifying Pages in Electronic Theses and Dissertations. (arXiv:2311.04262v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.04262
• Code URL: https://github.com/lamps-lab/ETDMiner
• Copy Paste: [[2311.04262]] ETDPC: A Multimodality Framework for Classifying Pages in Electronic Theses and Dissertations(http://arxiv.org/abs/2311.04262)
• Summary:

  Electronic theses and dissertations (ETDs) have been proposed, advocated, and generated for more than 25 years. Although ETDs are hosted by commercial or institutional digital library repositories, they are still an understudied type of scholarly big data, partially because they are usually longer than conference proceedings and journals. Segmenting ETDs will allow researchers to study sectional content. Readers can navigate to particular pages of interest, discover, and explore the content buried in these long documents. Most existing frameworks on document page classification are designed for classifying general documents and perform poorly on ETDs. In this paper, we propose ETDPC. Its backbone is a two-stream multimodal model with a cross-attention network to classify ETD pages into 13 categories. To overcome the challenge of imbalanced labeled samples, we augmented data for minority categories and employed a hierarchical classifier. ETDPC outperforms the state-of-the-art models in all categories, achieving an F1 of 0.84 -- 0.96 for 9 out of 13 categories. We also demonstrated its data efficiency. The code and data can be found on GitHub (https://github.com/lamps-lab/ETDMiner/tree/master/etd_segmentation).

Title: Learning the What and How of Annotation in Video Object Segmentation. (arXiv:2311.04414v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.04414
• Code URL: null
• Copy Paste: [[2311.04414]] Learning the What and How of Annotation in Video Object Segmentation(http://arxiv.org/abs/2311.04414)
• Summary:

  Video Object Segmentation (VOS) is crucial for several applications, from video editing to video data generation. Training a VOS model requires an abundance of manually labeled training videos. The de-facto traditional way of annotating objects requires humans to draw detailed segmentation masks on the target objects at each video frame. This annotation process, however, is tedious and time-consuming. To reduce this annotation cost, in this paper, we propose EVA-VOS, a human-in-the-loop annotation framework for video object segmentation. Unlike the traditional approach, we introduce an agent that predicts iteratively both which frame ("What") to annotate and which annotation type ("How") to use. Then, the annotator annotates only the selected frame that is used to update a VOS module, leading to significant gains in annotation time. We conduct experiments on the MOSE and the DAVIS datasets and we show that: (a) EVA-VOS leads to masks with accuracy close to the human agreement 3.5x faster than the standard way of annotating videos; (b) our frame selection achieves state-of-the-art performance; (c) EVA-VOS yields significant performance gains in terms of annotation time compared to all other methods and baselines.

Title: NExT-Chat: An LMM for Chat, Detection and Segmentation. (arXiv:2311.04498v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.04498
• Code URL: null
• Copy Paste: [[2311.04498]] NExT-Chat: An LMM for Chat, Detection and Segmentation(http://arxiv.org/abs/2311.04498)
• Summary:

  The development of large language models (LLMs) has greatly advanced the field of multimodal understanding, leading to the emergence of large multimodal models (LMMs). In order to enhance the level of visual comprehension, recent studies have equipped LMMs with region-level understanding capabilities by representing object bounding box coordinates as a series of text sequences (pixel2seq). In this paper, we introduce a novel paradigm for object location modeling called pixel2emb method, where we ask the LMM to output the location embeddings and then decoded by different decoders. This paradigm allows for different location formats (such as bounding boxes and masks) to be used in multimodal conversations Furthermore, this kind of embedding based location modeling enables the utilization of existing practices in localization tasks, such as detection and segmentation. In scenarios with limited resources, our pixel2emb demonstrates superior performance compared to existing state-of-the-art (SOTA) approaches in both the location input and output tasks under fair comparison. Leveraging the proposed pixel2emb method, we train an LMM named NExT-Chat and demonstrate its capability of handling multiple tasks like visual grounding, region caption, and grounded reasoning.

Title: SKU-Patch: Towards Efficient Instance Segmentation for Unseen Objects in Auto-Store. (arXiv:2311.04645v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.04645
• Code URL: null
• Copy Paste: [[2311.04645]] SKU-Patch: Towards Efficient Instance Segmentation for Unseen Objects in Auto-Store(http://arxiv.org/abs/2311.04645)
• Summary:

  In large-scale storehouses, precise instance masks are crucial for robotic bin picking but are challenging to obtain. Existing instance segmentation methods typically rely on a tedious process of scene collection, mask annotation, and network fine-tuning for every single Stock Keeping Unit (SKU). This paper presents SKU-Patch, a new patch-guided instance segmentation solution, leveraging only a few image patches for each incoming new SKU to predict accurate and robust masks, without tedious manual effort and model re-training. Technical-wise, we design a novel transformer-based network with (i) a patch-image correlation encoder to capture multi-level image features calibrated by patch information and (ii) a patch-aware transformer decoder with parallel task heads to generate instance masks. Extensive experiments on four storehouse benchmarks manifest that SKU-Patch is able to achieve the best performance over the state-of-the-art methods. Also, SKU-Patch yields an average of nearly 100% grasping success rate on more than 50 unseen SKUs in a robot-aided auto-store logistic pipeline, showing its effectiveness and practicality.

Title: Lidar Annotation Is All You Need. (arXiv:2311.04777v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.04777
• Code URL: https://github.com/evocargo/lidar-annotation-is-all-you-need
• Copy Paste: [[2311.04777]] Lidar Annotation Is All You Need(http://arxiv.org/abs/2311.04777)
• Summary:

  In recent years, computer vision has transformed fields such as medical imaging, object recognition, and geospatial analytics. One of the fundamental tasks in computer vision is semantic image segmentation, which is vital for precise object delineation. Autonomous driving represents one of the key areas where computer vision algorithms are applied. The task of road surface segmentation is crucial in self-driving systems, but it requires a labor-intensive annotation process in several data domains. The work described in this paper aims to improve the efficiency of image segmentation using a convolutional neural network in a multi-sensor setup. This approach leverages lidar (Light Detection and Ranging) annotations to directly train image segmentation models on RGB images. Lidar supplements the images by emitting laser pulses and measuring reflections to provide depth information. However, lidar's sparse point clouds often create difficulties for accurate object segmentation. Segmentation of point clouds requires time-consuming preliminary data preparation and a large amount of computational resources. The key innovation of our approach is the masked loss, addressing sparse ground-truth masks from point clouds. By calculating loss exclusively where lidar points exist, the model learns road segmentation on images by using lidar points as ground truth. This approach allows for blending of different ground-truth data types during model training. Experimental validation of the approach on benchmark datasets shows comparable performance to a high-quality image segmentation model. Incorporating lidar reduces the load on annotations and enables training of image-segmentation models without loss of segmentation quality. The methodology is tested on diverse datasets, both publicly available and proprietary. The strengths and weaknesses of the proposed method are also discussed in the paper.

Title: Lewis's Signaling Game as beta-VAE For Natural Word Lengths and Segments. (arXiv:2311.04453v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.04453
• Code URL: null
• Copy Paste: [[2311.04453]] Lewis's Signaling Game as beta-VAE For Natural Word Lengths and Segments(http://arxiv.org/abs/2311.04453)
• Summary:

  As a sub-discipline of evolutionary and computational linguistics, emergent communication (EC) studies communication protocols, called emergent languages, arising in simulations where agents communicate. A key goal of EC is to give rise to languages that share statistical properties with natural languages. In this paper, we reinterpret Lewis's signaling game, a frequently used setting in EC, as beta-VAE and reformulate its objective function as ELBO. Consequently, we clarify the existence of prior distributions of emergent languages and show that the choice of the priors can influence their statistical properties. Specifically, we address the properties of word lengths and segmentation, known as Zipf's law of abbreviation (ZLA) and Harris's articulation scheme (HAS), respectively. It has been reported that the emergent languages do not follow them when using the conventional objective. We experimentally demonstrate that by selecting an appropriate prior distribution, more natural segments emerge, while suggesting that the conventional one prevents the languages from following ZLA and HAS.