2023-12-29

secure

Title: Blockchain-Envisioned Post-Quantum Secure Sanitizable Signature for Audit Logs Management. (arXiv:2312.16322v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.16322
• Code URL: null
• Copy Paste: [[2312.16322]] Blockchain-Envisioned Post-Quantum Secure Sanitizable Signature for Audit Logs Management(http://arxiv.org/abs/2312.16322)
• Summary:

  Audit logs are one of the most important tools for transparently tracking system events and maintaining continuous oversight in corporate organizations and enterprise business systems. There are many cases where the audit logs contain sensitive data, or the audit logs are enormous. In these situations, dealing with a subset of the data is more practical than the entire data set. To provide a secure solution to handle these issues, a sanitizable signature scheme (SSS) is a viable cryptographic primitive. Herein, we first present the \textit{first} post-quantum secure multivariate-based SSS, namely ${\sf Mul-SAN}$. Our proposed design provides unforgeability, privacy, immutability, signer accountability, and sanitizer accountability under the assumption that the $MQ$ problem is NP-hard. ${\sf Mul-SAN}$ is very efficient and only requires computing field multiplications and additions over a finite field for its implementation. ${\sf Mul-SAN}$ presents itself as a practical method to partially delegate control of the authenticated data in avenues like the healthcare industry and government organizations. We also explore using Blockchain to provide a tamper-proof and robust audit log mechanism.

security

Title: Cross-border Exchange of CBDCs using Layer-2 Blockchain. (arXiv:2312.16193v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.16193
• Code URL: null
• Copy Paste: [[2312.16193]] Cross-border Exchange of CBDCs using Layer-2 Blockchain(http://arxiv.org/abs/2312.16193)
• Summary:

  This paper proposes a novel multi-layer blockchain architecture for the cross-border trading of CBDCs. The permissioned layer-2, by relying on the public consensus of the underlying network, assures the security and integrity of the transactions and ensures interoperability with domestic CBDCs implementations. Multiple Layer-3s operate various Automated Market Makers (AMMs) and compete with each other for the lowest costs. To provide insights into the practical implications of the system, simulations of trading costs are conducted based on historical FX rates, with Project Mariana as a benchmark. The study shows that, even with liquidity fragmentation, a multi-layer and multi-AMM setup is more cost-efficient than a single AMM.

Title: Security in 5G Networks -- How 5G networks help Mitigate Location Tracking Vulnerability. (arXiv:2312.16200v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.16200
• Code URL: null
• Copy Paste: [[2312.16200]] Security in 5G Networks -- How 5G networks help Mitigate Location Tracking Vulnerability(http://arxiv.org/abs/2312.16200)
• Summary:

  As 5G networks become more mainstream, privacy has come to the forefront of end users. More scrutiny has been shown to previous generation cellular technologies such as 3G and 4G on how they handle sensitive metadata transmitted from an end user mobile device to base stations during registration with a cellular network. These generation cellular networks do not enforce any encryption on this information transmitted during this process, giving malicious actors an easy way to intercept the information. Such an interception can allow an adversary to locate end users with shocking accuracy. This paper investigates this problem in great detail and discusses how a newly introduced approach in 5G networks is helping combat this problem. The paper discusses the implications of this vulnerability and the technical details of the new approach, including the encryption schemes used to secure this sensitive information. Finally, the paper will discuss any limitations to this new approach.

Title: Smuche: Scalar-Multiplicative Caching in Homomorphic Encryption. (arXiv:2312.16352v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.16352
• Code URL: null
• Copy Paste: [[2312.16352]] Smuche: Scalar-Multiplicative Caching in Homomorphic Encryption(http://arxiv.org/abs/2312.16352)
• Summary:

  Addressing the challenge of balancing security and efficiency when deploying machine learning systems in untrusted environments, such as federated learning, remains a critical concern. A promising strategy to tackle this issue involves optimizing the performance of fully homomorphic encryption (HE). Recent research highlights the efficacy of advanced caching techniques, such as Rache, in significantly enhancing the performance of HE schemes without compromising security. However, Rache is constrained by an inherent limitation: its performance overhead is heavily influenced by the characteristics of plaintext models, specifically exhibiting a caching time complexity of $\mathcal{O}(N)$, where $N$ represents the number of cached pivots based on specific radixes. This caching overhead becomes impractical for handling large-scale data. In this study, we introduce a novel \textit{constant-time} caching technique that is independent of any parameters. The core concept involves applying scalar multiplication to a single cached ciphertext, followed by the introduction of a completely new and constant-time randomness. Leveraging the inherent characteristics of constant-time construction, we coin the term ``Smuche'' for this innovative caching technique, which stands for Scalar-multiplicative Caching of Homomorphic Encryption. We implemented Smuche from scratch and conducted comparative evaluations against two baseline schemes, Rache and CKKS. Our experimental results underscore the effectiveness of Smuche in addressing the identified limitations and optimizing the performance of homomorphic encryption in practical scenarios.

Title: Vulnerability Scanners for Ethereum Smart Contracts: A Large-Scale Study. (arXiv:2312.16533v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.16533
• Code URL: null
• Copy Paste: [[2312.16533]] Vulnerability Scanners for Ethereum Smart Contracts: A Large-Scale Study(http://arxiv.org/abs/2312.16533)
• Summary:

  Ethereum smart contracts, which are autonomous decentralized applications on the blockchain that manage assets often exceeding millions of dollars, have become primary targets for cyberattacks. In 2023 alone, such vulnerabilities led to substantial financial losses exceeding a billion of US dollars. To counter these threats, various tools have been developed by academic and commercial entities to detect and mitigate vulnerabilities in smart contracts. Our study investigates the gap between the effectiveness of existing security scanners and the vulnerabilities that still persist in practice. We compiled four distinct datasets for this analysis. The first dataset comprises 77,219 source codes extracted directly from the blockchain, while the second includes over 4 million bytecodes obtained from Ethereum Mainnet and testnets. The other two datasets consist of nearly 14,000 manually annotated smart contracts and 373 smart contracts verified through audits, providing a foundation for a rigorous ground truth analysis on bytecode and source code. Using the unlabeled datasets, we conducted a comprehensive quantitative evaluation of 17 vulnerability scanners, revealing considerable discrepancies in their findings. Our analysis of the ground truth datasets indicated poor performance across all the tools we tested. This study unveils the reasons for poor performance and underscores that the current state of the art for smart contract security falls short in effectively addressing open problems, highlighting that the challenge of effectively detecting vulnerabilities remains a significant and unresolved issue.

Title: Evaluating the security of CRYSTALS-Dilithium in the quantum random oracle model. (arXiv:2312.16619v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.16619
• Code URL: null
• Copy Paste: [[2312.16619]] Evaluating the security of CRYSTALS-Dilithium in the quantum random oracle model(http://arxiv.org/abs/2312.16619)
• Summary:

  In the wake of recent progress on quantum computing hardware, the National Institute of Standards and Technology (NIST) is standardizing cryptographic protocols that are resistant to attacks by quantum adversaries. The primary digital signature scheme that NIST has chosen is CRYSTALS-Dilithium. The hardness of this scheme is based on the hardness of three computational problems: Module Learning with Errors (MLWE), Module Short Integer Solution (MSIS), and SelfTargetMSIS. MLWE and MSIS have been well-studied and are widely believed to be secure. However, SelfTargetMSIS is novel and, though classically as hard as MSIS, its quantum hardness is unclear. In this paper, we provide the first proof of the hardness of SelfTargetMSIS via a reduction from MLWE in the Quantum Random Oracle Model (QROM). Our proof uses recently developed techniques in quantum reprogramming and rewinding. A central part of our approach is a proof that a certain hash function, derived from the MSIS problem, is collapsing. From this approach, we deduce a new security proof for Dilithium under appropriate parameter settings. Compared to the only other rigorous security proof for a variant of Dilithium, Dilithium-QROM, our proof has the advantage of being applicable under the condition q = 1 mod 2n, where q denotes the modulus and n the dimension of the underlying algebraic ring. This condition is part of the original Dilithium proposal and is crucial for the efficient implementation of the scheme. We provide new secure parameter sets for Dilithium under the condition q = 1 mod 2n, finding that our public key sizes and signature sizes are about 2.5 to 2.8 times larger than those of Dilithium-QROM for the same security levels.

privacy

Title: SoK: Taming the Triangle -- On the Interplays between Fairness, Interpretability and Privacy in Machine Learning. (arXiv:2312.16191v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.16191
• Code URL: null
• Copy Paste: [[2312.16191]] SoK: Taming the Triangle -- On the Interplays between Fairness, Interpretability and Privacy in Machine Learning(http://arxiv.org/abs/2312.16191)
• Summary:

  Machine learning techniques are increasingly used for high-stakes decision-making, such as college admissions, loan attribution or recidivism prediction. Thus, it is crucial to ensure that the models learnt can be audited or understood by human users, do not create or reproduce discrimination or bias, and do not leak sensitive information regarding their training data. Indeed, interpretability, fairness and privacy are key requirements for the development of responsible machine learning, and all three have been studied extensively during the last decade. However, they were mainly considered in isolation, while in practice they interplay with each other, either positively or negatively. In this Systematization of Knowledge (SoK) paper, we survey the literature on the interactions between these three desiderata. More precisely, for each pairwise interaction, we summarize the identified synergies and tensions. These findings highlight several fundamental theoretical and empirical conflicts, while also demonstrating that jointly considering these different requirements is challenging when one aims at preserving a high level of utility. To solve this issue, we also discuss possible conciliation mechanisms, showing that a careful design can enable to successfully handle these different concerns in practice.

Title: A Theoretical Analysis of Efficiency Constrained Utility-Privacy Bi-Objective Optimization in Federated Learning. (arXiv:2312.16554v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.16554
• Code URL: null
• Copy Paste: [[2312.16554]] A Theoretical Analysis of Efficiency Constrained Utility-Privacy Bi-Objective Optimization in Federated Learning(http://arxiv.org/abs/2312.16554)
• Summary:

  Federated learning (FL) enables multiple clients to collaboratively learn a shared model without sharing their individual data. Concerns about utility, privacy, and training efficiency in FL have garnered significant research attention. Differential privacy has emerged as a prevalent technique in FL, safeguarding the privacy of individual user data while impacting utility and training efficiency. Within Differential Privacy Federated Learning (DPFL), previous studies have primarily focused on the utility-privacy trade-off, neglecting training efficiency, which is crucial for timely completion. Moreover, differential privacy achieves privacy by introducing controlled randomness (noise) on selected clients in each communication round. Previous work has mainly examined the impact of noise level ($\sigma$) and communication rounds ($T$) on the privacy-utility dynamic, overlooking other influential factors like the sample ratio ($q$, the proportion of selected clients). This paper systematically formulates an efficiency-constrained utility-privacy bi-objective optimization problem in DPFL, focusing on $\sigma$, $T$, and $q$. We provide a comprehensive theoretical analysis, yielding analytical solutions for the Pareto front. Extensive empirical experiments verify the validity and efficacy of our analysis, offering valuable guidance for low-cost parameter design in DPFL.

protect

defense

attack

Title: Task Contamination: Language Models May Not Be Few-Shot Anymore. (arXiv:2312.16337v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.16337
• Code URL: null
• Copy Paste: [[2312.16337]] Task Contamination: Language Models May Not Be Few-Shot Anymore(http://arxiv.org/abs/2312.16337)
• Summary:

  Large language models (LLMs) offer impressive performance in various zero-shot and few-shot tasks. However, their success in zero-shot and few-shot settings may be affected by task contamination, a potential limitation that has not been thoroughly examined. This paper investigates how zero-shot and few-shot performance of LLMs has changed chronologically over time. Utilizing GPT-3 series models and several other recent open-sourced LLMs, and controlling for dataset difficulty, we find that on datasets released before the LLM training data creation date, LLMs perform surprisingly better than on datasets released after. This strongly indicates that, for many LLMs, there exists task contamination on zero-shot and few-shot evaluation for datasets released prior to the LLMs' training data creation date. Additionally, we utilize training data inspection, task example extraction, and a membership inference attack, which reveal further evidence of task contamination. Importantly, we find that for classification tasks with no possibility of task contamination, LLMs rarely demonstrate statistically significant improvements over simple majority baselines, in both zero and few-shot settings.

Title: It Is Time To Steer: A Scalable Framework for Analysis-driven Attack Graph Generation. (arXiv:2312.16513v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.16513
• Code URL: null
• Copy Paste: [[2312.16513]] It Is Time To Steer: A Scalable Framework for Analysis-driven Attack Graph Generation(http://arxiv.org/abs/2312.16513)
• Summary:

  In modern computer networks where sophisticated cyber attacks occur daily, a timely cyber risk assessment becomes paramount. Attack Graph (AG) represents the best-suited solution to model and analyze multi-step attacks on computer networks, although they suffer from poor scalability due to their combinatorial complexity. This paper introduces an analysis-driven framework for AG generation. It enables real-time attack path analysis before the completion of the AG generation with a quantifiable statistical significance. We further accelerate the AG generation by steering it with the analysis query and supporting a novel workflow in which the analyst can query the system anytime. To show the capabilities of the proposed framework, we perform an extensive quantitative validation and we present a realistic case study on networks of unprecedented size. It demonstrates the advantages of our approach in terms of scalability and fitting to common attack path analyses.

Title: Adversarial Attacks on LoRa Device Identification and Rogue Signal Detection with Deep Learning. (arXiv:2312.16715v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.16715
• Code URL: null
• Copy Paste: [[2312.16715]] Adversarial Attacks on LoRa Device Identification and Rogue Signal Detection with Deep Learning(http://arxiv.org/abs/2312.16715)
• Summary:

  Low-Power Wide-Area Network (LPWAN) technologies, such as LoRa, have gained significant attention for their ability to enable long-range, low-power communication for Internet of Things (IoT) applications. However, the security of LoRa networks remains a major concern, particularly in scenarios where device identification and classification of legitimate and spoofed signals are crucial. This paper studies a deep learning framework to address these challenges, considering LoRa device identification and legitimate vs. rogue LoRa device classification tasks. A deep neural network (DNN), either a convolutional neural network (CNN) or feedforward neural network (FNN), is trained for each task by utilizing real experimental I/Q data for LoRa signals, while rogue signals are generated by using kernel density estimation (KDE) of received signals by rogue devices. Fast Gradient Sign Method (FGSM)-based adversarial attacks are considered for LoRa signal classification tasks using deep learning models. The impact of these attacks is assessed on the performance of two tasks, namely device identification and legitimate vs. rogue device classification, by utilizing separate or common perturbations against these signal classification tasks. Results presented in this paper quantify the level of transferability of adversarial attacks on different LoRa signal classification tasks as a major vulnerability and highlight the need to make IoT applications robust to adversarial attacks.

robust

Title: Segment Any Events via Weighted Adaptation of Pivotal Tokens. (arXiv:2312.16222v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.16222
• Code URL: null
• Copy Paste: [[2312.16222]] Segment Any Events via Weighted Adaptation of Pivotal Tokens(http://arxiv.org/abs/2312.16222)
• Summary:

  In this paper, we delve into the nuanced challenge of tailoring the Segment Anything Models (SAMs) for integration with event data, with the overarching objective of attaining robust and universal object segmentation within the event-centric domain. One pivotal issue at the heart of this endeavor is the precise alignment and calibration of embeddings derived from event-centric data such that they harmoniously coincide with those originating from RGB imagery. Capitalizing on the vast repositories of datasets with paired events and RGB images, our proposition is to harness and extrapolate the profound knowledge encapsulated within the pre-trained SAM framework. As a cornerstone to achieving this, we introduce a multi-scale feature distillation methodology. This methodology rigorously optimizes the alignment of token embeddings originating from event data with their RGB image counterparts, thereby preserving and enhancing the robustness of the overall architecture. Considering the distinct significance that token embeddings from intermediate layers hold for higher-level embeddings, our strategy is centered on accurately calibrating the pivotal token embeddings. This targeted calibration is aimed at effectively managing the discrepancies in high-level embeddings originating from both the event and image domains. Extensive experiments on different datasets demonstrate the effectiveness of the proposed distillation method. Code in this http URL

Title: Universal Pyramid Adversarial Training for Improved ViT Performance. (arXiv:2312.16339v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.16339
• Code URL: null
• Copy Paste: [[2312.16339]] Universal Pyramid Adversarial Training for Improved ViT Performance(http://arxiv.org/abs/2312.16339)
• Summary:

  Recently, Pyramid Adversarial training (Herrmann et al., 2022) has been shown to be very effective for improving clean accuracy and distribution-shift robustness of vision transformers. However, due to the iterative nature of adversarial training, the technique is up to 7 times more expensive than standard training. To make the method more efficient, we propose Universal Pyramid Adversarial training, where we learn a single pyramid adversarial pattern shared across the whole dataset instead of the sample-wise patterns. With our proposed technique, we decrease the computational cost of Pyramid Adversarial training by up to 70% while retaining the majority of its benefit on clean performance and distribution-shift robustness. In addition, to the best of our knowledge, we are also the first to find that universal adversarial training can be leveraged to improve clean model performance.

Title: Dynamic Sub-graph Distillation for Robust Semi-supervised Continual Learning. (arXiv:2312.16409v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.16409
• Code URL: https://github.com/fanyan0411/dsgd
• Copy Paste: [[2312.16409]] Dynamic Sub-graph Distillation for Robust Semi-supervised Continual Learning(http://arxiv.org/abs/2312.16409)
• Summary:

  Continual learning (CL) has shown promising results and comparable performance to learning at once in a fully supervised manner. However, CL strategies typically require a large number of labeled samples, making their real-life deployment challenging. In this work, we focus on semi-supervised continual learning (SSCL), where the model progressively learns from partially labeled data with unknown categories. We provide a comprehensive analysis of SSCL and demonstrate that unreliable distributions of unlabeled data lead to unstable training and refinement of the progressing stages. This problem severely impacts the performance of SSCL. To address the limitations, we propose a novel approach called Dynamic Sub-Graph Distillation (DSGD) for semi-supervised continual learning, which leverages both semantic and structural information to achieve more stable knowledge distillation on unlabeled data and exhibit robustness against distribution bias. Firstly, we formalize a general model of structural distillation and design a dynamic graph construction for the continual learning progress. Next, we define a structure distillation vector and design a dynamic sub-graph distillation algorithm, which enables end-to-end training and adaptability to scale up tasks. The entire proposed method is adaptable to various CL methods and supervision settings. Finally, experiments conducted on three datasets CIFAR10, CIFAR100, and ImageNet-100, with varying supervision ratios, demonstrate the effectiveness of our proposed approach in mitigating the catastrophic forgetting problem in semi-supervised continual learning scenarios.

Title: Domain Generalization with Vital Phase Augmentation. (arXiv:2312.16451v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.16451
• Code URL: https://github.com/excitedkid/vipaug
• Copy Paste: [[2312.16451]] Domain Generalization with Vital Phase Augmentation(http://arxiv.org/abs/2312.16451)
• Summary:

  Deep neural networks have shown remarkable performance in image classification. However, their performance significantly deteriorates with corrupted input data. Domain generalization methods have been proposed to train robust models against out-of-distribution data. Data augmentation in the frequency domain is one of such approaches that enable a model to learn phase features to establish domain-invariant representations. This approach changes the amplitudes of the input data while preserving the phases. However, using fixed phases leads to susceptibility to phase fluctuations because amplitudes and phase fluctuations commonly occur in out-of-distribution. In this study, to address this problem, we introduce an approach using finite variation of the phases of input data rather than maintaining fixed phases. Based on the assumption that the degree of domain-invariant features varies for each phase, we propose a method to distinguish phases based on this degree. In addition, we propose a method called vital phase augmentation (VIPAug) that applies the variation to the phases differently according to the degree of domain-invariant features of given phases. The model depends more on the vital phases that contain more domain-invariant features for attaining robustness to amplitude and phase fluctuations. We present experimental evaluations of our proposed approach, which exhibited improved performance for both clean and corrupted data. VIPAug achieved SOTA performance on the benchmark CIFAR-10 and CIFAR-100 datasets, as well as near-SOTA performance on the ImageNet-100 and ImageNet datasets. Our code is available at https://github.com/excitedkid/vipaug.

Title: Camera calibration for the surround-view system: a benchmark and dataset. (arXiv:2312.16499v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.16499
• Code URL: null
• Copy Paste: [[2312.16499]] Camera calibration for the surround-view system: a benchmark and dataset(http://arxiv.org/abs/2312.16499)
• Summary:

  Surround-view system (SVS) is widely used in the Advanced Driver Assistance System (ADAS). SVS uses four fisheye lenses to monitor real-time scenes around the vehicle. However, accurate intrinsic and extrinsic parameter estimation is required for the proper functioning of the system. At present, the intrinsic calibration can be pipeline by utilizing checkerboard algorithm, while extrinsic calibration is still immature. Therefore, we proposed a specific calibration pipeline to estimate extrinsic parameters robustly. This scheme takes a driving sequence of four cameras as input. It firstly utilizes lane line to roughly estimate each camera pose. Considering the environmental condition differences in each camera, we separately select strategies from two methods to accurately estimate the extrinsic parameters. To achieve accurate estimates for both front and rear camera, we proposed a method that mutually iterating line detection and pose estimation. As for bilateral camera, we iteratively adjust the camera pose and position by minimizing texture and edge error between ground projections of adjacent cameras. After estimating the extrinsic parameters, the surround-view image can be synthesized by homography-based transformation. The proposed pipeline can robustly estimate the four SVS camera extrinsic parameters in real driving environments. In addition, to evaluate the proposed scheme, we build a surround-view fisheye dataset, which contains 40 videos with 32,000 frames, acquired from different real traffic scenarios. All the frames in each video are manually labeled with lane annotation, with its GT extrinsic parameters. Moreover, this surround-view dataset could be used by other researchers to evaluate their performance. The dataset will be available soon.

Title: ConstScene: Dataset and Model for Advancing Robust Semantic Segmentation in Construction Environments. (arXiv:2312.16516v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.16516
• Code URL: null
• Copy Paste: [[2312.16516]] ConstScene: Dataset and Model for Advancing Robust Semantic Segmentation in Construction Environments(http://arxiv.org/abs/2312.16516)
• Summary:

  The increasing demand for autonomous machines in construction environments necessitates the development of robust object detection algorithms that can perform effectively across various weather and environmental conditions. This paper introduces a new semantic segmentation dataset specifically tailored for construction sites, taking into account the diverse challenges posed by adverse weather and environmental conditions. The dataset is designed to enhance the training and evaluation of object detection models, fostering their adaptability and reliability in real-world construction applications. Our dataset comprises annotated images captured under a wide range of different weather conditions, including but not limited to sunny days, rainy periods, foggy atmospheres, and low-light situations. Additionally, environmental factors such as the existence of dirt/mud on the camera lens are integrated into the dataset through actual captures and synthetic generation to simulate the complex conditions prevalent in construction sites. We also generate synthetic images of the annotations including precise semantic segmentation masks for various objects commonly found in construction environments, such as wheel loader machines, personnel, cars, and structural elements. To demonstrate the dataset's utility, we evaluate state-of-the-art object detection algorithms on our proposed benchmark. The results highlight the dataset's success in adversarial training models across diverse conditions, showcasing its efficacy compared to existing datasets that lack such environmental variability.

Title: HMP: Hand Motion Priors for Pose and Shape Estimation from Video. (arXiv:2312.16737v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.16737
• Code URL: null
• Copy Paste: [[2312.16737]] HMP: Hand Motion Priors for Pose and Shape Estimation from Video(http://arxiv.org/abs/2312.16737)
• Summary:

  Understanding how humans interact with the world necessitates accurate 3D hand pose estimation, a task complicated by the hand's high degree of articulation, frequent occlusions, self-occlusions, and rapid motions. While most existing methods rely on single-image inputs, videos have useful cues to address aforementioned issues. However, existing video-based 3D hand datasets are insufficient for training feedforward models to generalize to in-the-wild scenarios. On the other hand, we have access to large human motion capture datasets which also include hand motions, e.g. AMASS. Therefore, we develop a generative motion prior specific for hands, trained on the AMASS dataset which features diverse and high-quality hand motions. This motion prior is then employed for video-based 3D hand motion estimation following a latent optimization approach. Our integration of a robust motion prior significantly enhances performance, especially in occluded scenarios. It produces stable, temporally consistent results that surpass conventional single-frame methods. We demonstrate our method's efficacy via qualitative and quantitative evaluations on the HO3D and DexYCB datasets, with special emphasis on an occlusion-focused subset of HO3D. Code is available at https://hmp.is.tue.mpg.de

Title: How Robust are LLMs to In-Context Majority Label Bias?. (arXiv:2312.16549v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.16549
• Code URL: null
• Copy Paste: [[2312.16549]] How Robust are LLMs to In-Context Majority Label Bias?(http://arxiv.org/abs/2312.16549)
• Summary:

  In the In-Context Learning (ICL) setup, various forms of label biases can manifest. One such manifestation is majority label bias, which arises when the distribution of labeled examples in the in-context samples is skewed towards one or more specific classes making Large Language Models (LLMs) more prone to predict those labels. Such discrepancies can arise from various factors, including logistical constraints, inherent biases in data collection methods, limited access to diverse data sources, etc. which are unavoidable in a real-world industry setup. In this work, we study the robustness of in-context learning in LLMs to shifts that occur due to majority label bias within the purview of text classification tasks. Prior works have shown that in-context learning with LLMs is susceptible to such biases. In our study, we go one level deeper and show that the robustness boundary varies widely for different models and tasks, with certain LLMs being highly robust (~90%) to majority label bias. Additionally, our findings also highlight the impact of model size and the richness of instructional prompts contributing towards model robustness. We restrict our study to only publicly available open-source models to ensure transparency and reproducibility.

Title: OpenRL: A Unified Reinforcement Learning Framework. (arXiv:2312.16189v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.16189
• Code URL: https://github.com/openrl-lab/openrl
• Copy Paste: [[2312.16189]] OpenRL: A Unified Reinforcement Learning Framework(http://arxiv.org/abs/2312.16189)
• Summary:

  We present OpenRL, an advanced reinforcement learning (RL) framework designed to accommodate a diverse array of tasks, from single-agent challenges to complex multi-agent systems. OpenRL's robust support for self-play training empowers agents to develop advanced strategies in competitive settings. Notably, OpenRL integrates Natural Language Processing (NLP) with RL, enabling researchers to address a combination of RL training and language-centric tasks effectively. Leveraging PyTorch's robust capabilities, OpenRL exemplifies modularity and a user-centric approach. It offers a universal interface that simplifies the user experience for beginners while maintaining the flexibility experts require for innovation and algorithm development. This equilibrium enhances the framework's practicality, adaptability, and scalability, establishing a new standard in RL research. To delve into OpenRL's features, we invite researchers and enthusiasts to explore our GitHub repository at https://github.com/OpenRL-Lab/openrl and access our comprehensive documentation at https://openrl-docs.readthedocs.io.

Title: Refining Latent Homophilic Structures over Heterophilic Graphs for Robust Graph Convolution Networks. (arXiv:2312.16418v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.16418
• Code URL: null
• Copy Paste: [[2312.16418]] Refining Latent Homophilic Structures over Heterophilic Graphs for Robust Graph Convolution Networks(http://arxiv.org/abs/2312.16418)
• Summary:

  Graph convolution networks (GCNs) are extensively utilized in various graph tasks to mine knowledge from spatial data. Our study marks the pioneering attempt to quantitatively investigate the GCN robustness over omnipresent heterophilic graphs for node classification. We uncover that the predominant vulnerability is caused by the structural out-of-distribution (OOD) issue. This finding motivates us to present a novel method that aims to harden GCNs by automatically learning Latent Homophilic Structures over heterophilic graphs. We term such a methodology as LHS. To elaborate, our initial step involves learning a latent structure by employing a novel self-expressive technique based on multi-node interactions. Subsequently, the structure is refined using a pairwisely constrained dual-view contrastive learning approach. We iteratively perform the above procedure, enabling a GCN model to aggregate information in a homophilic way on heterophilic graphs. Armed with such an adaptable structure, we can properly mitigate the structural OOD threats over heterophilic graphs. Experiments on various benchmarks show the effectiveness of the proposed LHS approach for robust GCNs.

Title: Using Enriched Category Theory to Construct the Nearest Neighbour Classification Algorithm. (arXiv:2312.16529v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.16529
• Code URL: null
• Copy Paste: [[2312.16529]] Using Enriched Category Theory to Construct the Nearest Neighbour Classification Algorithm(http://arxiv.org/abs/2312.16529)
• Summary:

  Exploring whether Enriched Category Theory could provide the foundation of an alternative approach to Machine Learning. This paper is the first to construct and motivate a Machine Learning algorithm solely with Enriched Category Theory. In order to supplement evidence that Category Theory can be used to motivate robust and explainable algorithms, it is shown that a series of reasonable assumptions about a dataset lead to the construction of the Nearest Neighbours Algorithm. In particular, as an extension of the original dataset using profunctors in the category of Lawvere metric spaces. This leads to a definition of an Enriched Nearest Neighbours Algorithm, which consequently also produces an enriched form of the Voronoi diagram. This paper is intended to be accessible without any knowledge of Category Theory

Title: Exploring intra-task relations to improve meta-learning algorithms. (arXiv:2312.16612v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.16612
• Code URL: null
• Copy Paste: [[2312.16612]] Exploring intra-task relations to improve meta-learning algorithms(http://arxiv.org/abs/2312.16612)
• Summary:

  Meta-learning has emerged as an effective methodology to model several real-world tasks and problems due to its extraordinary effectiveness in the low-data regime. There are many scenarios ranging from the classification of rare diseases to language modelling of uncommon languages where the availability of large datasets is rare. Similarly, for more broader scenarios like self-driving, an autonomous vehicle needs to be trained to handle every situation well. This requires training the ML model on a variety of tasks with good quality data. But often times, we find that the data distribution across various tasks is skewed, i.e.the data follows a long-tail distribution. This leads to the model performing well on some tasks and not performing so well on others leading to model robustness issues. Meta-learning has recently emerged as a potential learning paradigm which can effectively learn from one task and generalize that learning to unseen tasks. In this study, we aim to exploit external knowledge of task relations to improve training stability via effective mini-batching of tasks. We hypothesize that selecting a diverse set of tasks in a mini-batch will lead to a better estimate of the full gradient and hence will lead to a reduction of noise in training.

biometric

steal

extraction

membership infer

federate

Title: Federated Continual Learning via Knowledge Fusion: A Survey. (arXiv:2312.16475v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.16475
• Code URL: null
• Copy Paste: [[2312.16475]] Federated Continual Learning via Knowledge Fusion: A Survey(http://arxiv.org/abs/2312.16475)
• Summary:

  Data privacy and silos are nontrivial and greatly challenging in many real-world applications. Federated learning is a decentralized approach to training models across multiple local clients without the exchange of raw data from client devices to global servers. However, existing works focus on a static data environment and ignore continual learning from streaming data with incremental tasks. Federated Continual Learning (FCL) is an emerging paradigm to address model learning in both federated and continual learning environments. The key objective of FCL is to fuse heterogeneous knowledge from different clients and retain knowledge of previous tasks while learning on new ones. In this work, we delineate federated learning and continual learning first and then discuss their integration, i.e., FCL, and particular FCL via knowledge fusion. In summary, our motivations are four-fold: we (1) raise a fundamental problem called ''spatial-temporal catastrophic forgetting'' and evaluate its impact on the performance using a well-known method called federated averaging (FedAvg), (2) integrate most of the existing FCL methods into two generic frameworks, namely synchronous FCL and asynchronous FCL, (3) categorize a large number of methods according to the mechanism involved in knowledge fusion, and finally (4) showcase an outlook on the future work of FCL.

Title: Fault-Tolerant Vertical Federated Learning on Dynamic Networks. (arXiv:2312.16638v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.16638
• Code URL: null
• Copy Paste: [[2312.16638]] Fault-Tolerant Vertical Federated Learning on Dynamic Networks(http://arxiv.org/abs/2312.16638)
• Summary:

  Vertical Federated learning (VFL) is a class of FL where each client shares the same sample space but only holds a subset of the features. While VFL tackles key privacy challenges of distributed learning, it often assumes perfect hardware and communication capabilities. This assumption hinders the broad deployment of VFL, particularly on edge devices, which are heterogeneous in their in-situ capabilities and will connect/disconnect from the network over time. To address this gap, we define Internet Learning (IL) including its data splitting and network context and which puts good performance under extreme dynamic condition of clients as the primary goal. We propose VFL as a naive baseline and develop several extensions to handle the IL paradigm of learning. Furthermore, we implement new methods, propose metrics, and extensively analyze results based on simulating a sensor network. The results show that the developed methods are more robust to changes in the network than VFL baseline.

fair

Title: FairCompass: Operationalising Fairness in Machine Learning. (arXiv:2312.16726v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.16726
• Code URL: null
• Copy Paste: [[2312.16726]] FairCompass: Operationalising Fairness in Machine Learning(http://arxiv.org/abs/2312.16726)
• Summary:

  As artificial intelligence (AI) increasingly becomes an integral part of our societal and individual activities, there is a growing imperative to develop responsible AI solutions. Despite a diverse assortment of machine learning fairness solutions is proposed in the literature, there is reportedly a lack of practical implementation of these tools in real-world applications. Industry experts have participated in thorough discussions on the challenges associated with operationalising fairness in the development of machine learning-empowered solutions, in which a shift toward human-centred approaches is promptly advocated to mitigate the limitations of existing techniques. In this work, we propose a human-in-the-loop approach for fairness auditing, presenting a mixed visual analytical system (hereafter referred to as 'FairCompass'), which integrates both subgroup discovery technique and the decision tree-based schema for end users. Moreover, we innovatively integrate an Exploration, Guidance and Informed Analysis loop, to facilitate the use of the Knowledge Generation Model for Visual Analytics in FairCompass. We evaluate the effectiveness of FairCompass for fairness auditing in a real-world scenario, and the findings demonstrate the system's potential for real-world deployability. We anticipate this work will address the current gaps in research for fairness and facilitate the operationalisation of fairness in machine learning systems.

Title: The Fourth International Verification of Neural Networks Competition (VNN-COMP 2023): Summary and Results. (arXiv:2312.16760v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.16760
• Code URL: https://github.com/stanleybak/vnncomp2023
• Copy Paste: [[2312.16760]] The Fourth International Verification of Neural Networks Competition (VNN-COMP 2023): Summary and Results(http://arxiv.org/abs/2312.16760)
• Summary:

  This report summarizes the 4th International Verification of Neural Networks Competition (VNN-COMP 2023), held as a part of the 6th Workshop on Formal Methods for ML-Enabled Autonomous Systems (FoMLAS), that was collocated with the 35th International Conference on Computer-Aided Verification (CAV). VNN-COMP is held annually to facilitate the fair and objective comparison of state-of-the-art neural network verification tools, encourage the standardization of tool interfaces, and bring together the neural network verification community. To this end, standardized formats for networks (ONNX) and specification (VNN-LIB) were defined, tools were evaluated on equal-cost hardware (using an automatic evaluation pipeline based on AWS instances), and tool parameters were chosen by the participants before the final test sets were made public. In the 2023 iteration, 7 teams participated on a diverse set of 10 scored and 4 unscored benchmarks. This report summarizes the rules, benchmarks, participating tools, results, and lessons learned from this iteration of this competition.

interpretability

explainability

watermark

Title: FreqyWM: Frequency Watermarking for the New Data Economy. (arXiv:2312.16547v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.16547
• Code URL: null
• Copy Paste: [[2312.16547]] FreqyWM: Frequency Watermarking for the New Data Economy(http://arxiv.org/abs/2312.16547)
• Summary:

  We present a novel technique for modulating the appearance frequency of a few tokens within a dataset for encoding an invisible watermark that can be used to protect ownership rights upon data. We develop optimal as well as fast heuristic algorithms for creating and verifying such watermarks. We also demonstrate the robustness of our technique against various attacks and derive analytical bounds for the false positive probability of erroneously detecting a watermark on a dataset that does not carry it. Our technique is applicable to both single dimensional and multidimensional datasets, is independent of token type, allows for a fine control of the introduced distortion, and can be used in a variety of use cases that involve buying and selling data in contemporary data marketplaces.

diffusion

Title: Iterative Prompt Relabeling for diffusion model with RLDF. (arXiv:2312.16204v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.16204
• Code URL: null
• Copy Paste: [[2312.16204]] Iterative Prompt Relabeling for diffusion model with RLDF(http://arxiv.org/abs/2312.16204)
• Summary:

  Diffusion models have shown impressive performance in many domains, including image generation, time series prediction, and reinforcement learning. The algorithm demonstrates superior performance over the traditional GAN and transformer based methods. However, the model's capability to follow natural language instructions (e.g., spatial relationships between objects, generating complex scenes) is still unsatisfactory. This has been an important research area to enhance such capability. Prior works adopt reinforcement learning to adjust the behavior of the diffusion models. However, RL methods not only require careful reward design and complex hyperparameter tuning, but also fails to incorporate rich natural language feedback. In this work, we propose iterative prompt relabeling (IP-RLDF), a novel algorithm that aligns images to text through iterative image sampling and prompt relabeling. IP-RLDF first samples a batch of images conditioned on the text, then relabels the text prompts of unmatched text-image pairs with classifier feedback. We conduct thorough experiments on three different models, including SDv2, GLIGEN, and SDXL, testing their capability to generate images following instructions. With IP-RLDF, we improved up to 15.22% (absolute improvement) on the challenging spatial relation VISOR benchmark, demonstrating superior performance compared to previous RL methods.

Title: Hyper-VolTran: Fast and Generalizable One-Shot Image to 3D Object Structure via HyperNetworks. (arXiv:2312.16218v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.16218
• Code URL: null
• Copy Paste: [[2312.16218]] Hyper-VolTran: Fast and Generalizable One-Shot Image to 3D Object Structure via HyperNetworks(http://arxiv.org/abs/2312.16218)
• Summary:

  Solving image-to-3D from a single view is an ill-posed problem, and current neural reconstruction methods addressing it through diffusion models still rely on scene-specific optimization, constraining their generalization capability. To overcome the limitations of existing approaches regarding generalization and consistency, we introduce a novel neural rendering technique. Our approach employs the signed distance function as the surface representation and incorporates generalizable priors through geometry-encoding volumes and HyperNetworks. Specifically, our method builds neural encoding volumes from generated multi-view inputs. We adjust the weights of the SDF network conditioned on an input image at test-time to allow model adaptation to novel scenes in a feed-forward manner via HyperNetworks. To mitigate artifacts derived from the synthesized views, we propose the use of a volume transformer module to improve the aggregation of image features instead of processing each viewpoint separately. Through our proposed method, dubbed as Hyper-VolTran, we avoid the bottleneck of scene-specific optimization and maintain consistency across the images generated from multiple viewpoints. Our experiments show the advantages of our proposed approach with consistent results and rapid generation.

Title: Towards Flexible, Scalable, and Adaptive Multi-Modal Conditioned Face Synthesis. (arXiv:2312.16274v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.16274
• Code URL: null
• Copy Paste: [[2312.16274]] Towards Flexible, Scalable, and Adaptive Multi-Modal Conditioned Face Synthesis(http://arxiv.org/abs/2312.16274)
• Summary:

  Recent progress in multi-modal conditioned face synthesis has enabled the creation of visually striking and accurately aligned facial images. Yet, current methods still face issues with scalability, limited flexibility, and a one-size-fits-all approach to control strength, not accounting for the differing levels of conditional entropy, a measure of unpredictability in data given some condition, across modalities. To address these challenges, we introduce a novel uni-modal training approach with modal surrogates, coupled with an entropy-aware modal-adaptive modulation, to support flexible, scalable, and scalable multi-modal conditioned face synthesis network. Our uni-modal training with modal surrogate that only leverage uni-modal data, use modal surrogate to decorate condition with modal-specific characteristic and serve as linker for inter-modal collaboration , fully learns each modality control in face synthesis process as well as inter-modal collaboration. The entropy-aware modal-adaptive modulation finely adjust diffusion noise according to modal-specific characteristics and given conditions, enabling well-informed step along denoising trajectory and ultimately leading to synthesis results of high fidelity and quality. Our framework improves multi-modal face synthesis under various conditions, surpassing current methods in image quality and fidelity, as demonstrated by our thorough experimental results.

Title: State-of-the-Art in Nudity Classification: A Comparative Analysis. (arXiv:2312.16338v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.16338
• Code URL: null
• Copy Paste: [[2312.16338]] State-of-the-Art in Nudity Classification: A Comparative Analysis(http://arxiv.org/abs/2312.16338)
• Summary:

  This paper presents a comparative analysis of existing nudity classification techniques for classifying images based on the presence of nudity, with a focus on their application in content moderation. The evaluation focuses on CNN-based models, vision transformer, and popular open-source safety checkers from Stable Diffusion and Large-scale Artificial Intelligence Open Network (LAION). The study identifies the limitations of current evaluation datasets and highlights the need for more diverse and challenging datasets. The paper discusses the potential implications of these findings for developing more accurate and effective image classification systems on online platforms. Overall, the study emphasizes the importance of continually improving image classification models to ensure the safety and well-being of platform users. The project page, including the demonstrations and results is publicly available at https://github.com/fcakyon/content-moderation-deep-learning.

Title: Natural Adversarial Patch Generation Method Based on Latent Diffusion Model. (arXiv:2312.16401v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.16401
• Code URL: null
• Copy Paste: [[2312.16401]] Natural Adversarial Patch Generation Method Based on Latent Diffusion Model(http://arxiv.org/abs/2312.16401)
• Summary:

  Recently, some research show that deep neural networks are vulnerable to the adversarial attacks, the well-trainned samples or patches could be used to trick the neural network detector or human visual perception. However, these adversarial patches, with their conspicuous and unusual patterns, lack camouflage and can easily raise suspicion in the real world. To solve this problem, this paper proposed a novel adversarial patch method called the Latent Diffusion Patch (LDP), in which, a pretrained encoder is first designed to compress the natural images into a feature space with key characteristics. Then trains the diffusion model using the above feature space. Finally, explore the latent space of the pretrained diffusion model using the image denoising technology. It polishes the patches and images through the powerful natural abilities of diffusion models, making them more acceptable to the human visual system. Experimental results, both digital and physical worlds, show that LDPs achieve a visual subjectivity score of 87.3%, while still maintaining effective attack capabilities.

Title: SVGDreamer: Text Guided SVG Generation with Diffusion Model. (arXiv:2312.16476v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.16476
• Code URL: null
• Copy Paste: [[2312.16476]] SVGDreamer: Text Guided SVG Generation with Diffusion Model(http://arxiv.org/abs/2312.16476)
• Summary:

  Recently, text-guided scalable vector graphics (SVGs) synthesis has shown promise in domains such as iconography and sketch. However, existing text-to-SVG generation methods lack editability and struggle with visual quality and result diversity. To address these limitations, we propose a novel text-guided vector graphics synthesis method called SVGDreamer. SVGDreamer incorporates a semantic-driven image vectorization (SIVE) process that enables the decomposition of synthesis into foreground objects and background, thereby enhancing editability. Specifically, the SIVE process introduce attention-based primitive control and an attention-mask loss function for effective control and manipulation of individual elements. Additionally, we propose a Vectorized Particle-based Score Distillation (VPSD) approach to tackle the challenges of color over-saturation, vector primitives over-smoothing, and limited result diversity in existing text-to-SVG generation methods. Furthermore, on the basis of VPSD, we introduce Reward Feedback Learning (ReFL) to accelerate VPSD convergence and improve aesthetic appeal. Extensive experiments have been conducted to validate the effectiveness of SVGDreamer, demonstrating its superiority over baseline methods in terms of editability, visual quality, and diversity.

Title: PanGu-Draw: Advancing Resource-Efficient Text-to-Image Synthesis with Time-Decoupled Training and Reusable Coop-Diffusion. (arXiv:2312.16486v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.16486
• Code URL: null
• Copy Paste: [[2312.16486]] PanGu-Draw: Advancing Resource-Efficient Text-to-Image Synthesis with Time-Decoupled Training and Reusable Coop-Diffusion(http://arxiv.org/abs/2312.16486)
• Summary:

  Current large-scale diffusion models represent a giant leap forward in conditional image synthesis, capable of interpreting diverse cues like text, human poses, and edges. However, their reliance on substantial computational resources and extensive data collection remains a bottleneck. On the other hand, the integration of existing diffusion models, each specialized for different controls and operating in unique latent spaces, poses a challenge due to incompatible image resolutions and latent space embedding structures, hindering their joint use. Addressing these constraints, we present "PanGu-Draw", a novel latent diffusion model designed for resource-efficient text-to-image synthesis that adeptly accommodates multiple control signals. We first propose a resource-efficient Time-Decoupling Training Strategy, which splits the monolithic text-to-image model into structure and texture generators. Each generator is trained using a regimen that maximizes data utilization and computational efficiency, cutting data preparation by 48% and reducing training resources by 51%. Secondly, we introduce "Coop-Diffusion", an algorithm that enables the cooperative use of various pre-trained diffusion models with different latent spaces and predefined resolutions within a unified denoising process. This allows for multi-control image synthesis at arbitrary resolutions without the necessity for additional data or retraining. Empirical validations of Pangu-Draw show its exceptional prowess in text-to-image and multi-control image generation, suggesting a promising direction for future model training efficiencies and generation versatility. The largest 5B T2I PanGu-Draw model is released on the Ascend platform. Project page: https://pangu-draw.github.io

Title: I2V-Adapter: A General Image-to-Video Adapter for Video Diffusion Models. (arXiv:2312.16693v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.16693
• Code URL: null
• Copy Paste: [[2312.16693]] I2V-Adapter: A General Image-to-Video Adapter for Video Diffusion Models(http://arxiv.org/abs/2312.16693)
• Summary:

  In the rapidly evolving domain of digital content generation, the focus has shifted from text-to-image (T2I) models to more advanced video diffusion models, notably text-to-video (T2V) and image-to-video (I2V). This paper addresses the intricate challenge posed by I2V: converting static images into dynamic, lifelike video sequences while preserving the original image fidelity. Traditional methods typically involve integrating entire images into diffusion processes or using pretrained encoders for cross attention. However, these approaches often necessitate altering the fundamental weights of T2I models, thereby restricting their reusability. We introduce a novel solution, namely I2V-Adapter, designed to overcome such limitations. Our approach preserves the structural integrity of T2I models and their inherent motion modules. The I2V-Adapter operates by processing noised video frames in parallel with the input image, utilizing a lightweight adapter module. This module acts as a bridge, efficiently linking the input to the model's self-attention mechanism, thus maintaining spatial details without requiring structural changes to the T2I model. Moreover, I2V-Adapter requires only a fraction of the parameters of conventional models and ensures compatibility with existing community-driven T2I models and controlling tools. Our experimental results demonstrate I2V-Adapter's capability to produce high-quality video outputs. This performance, coupled with its versatility and reduced need for trainable parameters, represents a substantial advancement in the field of AI-driven video generation, particularly for creative applications.

noise learning

data-free

transformer

Title: Merging Vision Transformers from Different Tasks and Domains. (arXiv:2312.16240v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.16240
• Code URL: null
• Copy Paste: [[2312.16240]] Merging Vision Transformers from Different Tasks and Domains(http://arxiv.org/abs/2312.16240)
• Summary:

  This work targets to merge various Vision Transformers (ViTs) trained on different tasks (i.e., datasets with different object categories) or domains (i.e., datasets with the same categories but different environments) into one unified model, yielding still good performance on each task or domain. Previous model merging works focus on either CNNs or NLP models, leaving the ViTs merging research untouched. To fill this gap, we first explore and find that existing model merging methods cannot well handle the merging of the whole ViT models and still have improvement space. To enable the merging of the whole ViT, we propose a simple-but-effective gating network that can both merge all kinds of layers (e.g., Embedding, Norm, Attention, and MLP) and select the suitable classifier. Specifically, the gating network is trained by unlabeled datasets from all the tasks (domains), and predicts the probability of which task (domain) the input belongs to for merging the models during inference. To further boost the performance of the merged model, especially when the difficulty of merging tasks increases, we design a novel metric of model weight similarity, and utilize it to realize controllable and combined weight merging. Comprehensive experiments on kinds of newly established benchmarks, validate the superiority of the proposed ViT merging framework for different tasks and domains. Our method can even merge beyond 10 ViT models from different vision tasks with a negligible effect on the performance of each task.

Title: Nighttime Person Re-Identification via Collaborative Enhancement Network with Multi-domain Learning. (arXiv:2312.16246v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.16246
• Code URL: null
• Copy Paste: [[2312.16246]] Nighttime Person Re-Identification via Collaborative Enhancement Network with Multi-domain Learning(http://arxiv.org/abs/2312.16246)
• Summary:

  Prevalent nighttime ReID methods typically combine relighting networks and ReID networks in a sequential manner, which not only restricts the ReID performance by the quality of relighting images, but also neglects the effective collaborative modeling between image relighting and person ReID tasks. To handle these problems, we propose a novel Collaborative Enhancement Network called CENet, which performs the multilevel feature interactions in a parallel framework, for nighttime person ReID. In particular, CENet is a parallel Transformer network, in which the designed parallel structure can avoid the impact of the quality of relighting images on ReID performance. To perform effective collaborative modeling between image relighting and person ReID tasks, we integrate the multilevel feature interactions in CENet. Specifically, we share the Transformer encoder to build the low-level feature interaction, and then perform the feature distillation to transfer the high-level features from image relighting to ReID. In addition, the sizes of existing real-world nighttime person ReID datasets are small, and large-scale synthetic ones exhibit substantial domain gaps with real-world data. To leverage both small-scale real-world and large-scale synthetic training data, we develop a multi-domain learning algorithm, which alternately utilizes both kinds of data to reduce the inter-domain difference in the training of CENet. Extensive experiments on two real nighttime datasets, \textit{Night600} and \textit{RGBNT201$_{rgb}$}, and a synthetic nighttime ReID dataset are conducted to validate the effectiveness of CENet. We will release the code and synthetic dataset.

Title: A Comprehensive Study of Object Tracking in Low-Light Environments. (arXiv:2312.16250v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.16250
• Code URL: null
• Copy Paste: [[2312.16250]] A Comprehensive Study of Object Tracking in Low-Light Environments(http://arxiv.org/abs/2312.16250)
• Summary:

  Accurate object tracking in low-light environments is crucial, particularly in surveillance and ethology applications. However, achieving this is significantly challenging due to the poor quality of captured sequences. Factors such as noise, color imbalance, and low contrast contribute to these challenges. This paper presents a comprehensive study examining the impact of these distortions on automatic object trackers. Additionally, we propose a solution to enhance tracking performance by integrating denoising and low-light enhancement methods into the transformer-based object tracking system. Experimental results show that the proposed tracker, trained with low-light synthetic datasets, outperforms both the vanilla MixFormer and Siam R-CNN.

Title: Adaptive Depth Networks with Skippable Sub-Paths. (arXiv:2312.16392v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.16392
• Code URL: null
• Copy Paste: [[2312.16392]] Adaptive Depth Networks with Skippable Sub-Paths(http://arxiv.org/abs/2312.16392)
• Summary:

  Systematic adaptation of network depths at runtime can be an effective way to control inference latency and meet the resource condition of various devices. However, previous depth adaptive networks do not provide general principles and a formal explanation on why and which layers can be skipped, and, hence, their approaches are hard to be generalized and require long and complex training steps. In this paper, we present an architectural pattern and training method for adaptive depth networks that can provide flexible accuracy-efficiency trade-offs in a single network. In our approach, every residual stage is divided into 2 consecutive sub-paths with different properties. While the first sub-path is mandatory for hierarchical feature learning, the other is optimized to incur minimal performance degradation even if it is skipped. Unlike previous adaptive networks, our approach does not iteratively self-distill a fixed set of sub-networks, resulting in significantly shorter training time. However, once deployed on devices, it can instantly construct sub-networks of varying depths to provide various accuracy-efficiency trade-offs in a single model. We provide a formal rationale for why the proposed architectural pattern and training method can reduce overall prediction errors while minimizing the impact of skipping selected sub-paths. We also demonstrate the generality and effectiveness of our approach with various residual networks, both from convolutional neural networks and vision transformers.

Title: Group Multi-View Transformer for 3D Shape Analysis with Spatial Encoding. (arXiv:2312.16477v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.16477
• Code URL: null
• Copy Paste: [[2312.16477]] Group Multi-View Transformer for 3D Shape Analysis with Spatial Encoding(http://arxiv.org/abs/2312.16477)
• Summary:

  In recent years, the results of view-based 3D shape recognition methods have saturated, and models with excellent performance cannot be deployed on memory-limited devices due to their huge size of parameters. To address this problem, we introduce a compression method based on knowledge distillation for this field, which largely reduces the number of parameters while preserving model performance as much as possible. Specifically, to enhance the capabilities of smaller models, we design a high-performing large model called Group Multi-view Vision Transformer (GMViT). In GMViT, the view-level ViT first establishes relationships between view-level features. Additionally, to capture deeper features, we employ the grouping module to enhance view-level features into group-level features. Finally, the group-level ViT aggregates group-level features into complete, well-formed 3D shape descriptors. Notably, in both ViTs, we introduce spatial encoding of camera coordinates as innovative position embeddings. Furthermore, we propose two compressed versions based on GMViT, namely GMViT-simple and GMViT-mini. To enhance the training effectiveness of the small models, we introduce a knowledge distillation method throughout the GMViT process, where the key outputs of each GMViT component serve as distillation targets. Extensive experiments demonstrate the efficacy of the proposed method. The large model GMViT achieves excellent 3D classification and retrieval results on the benchmark datasets ModelNet, ShapeNetCore55, and MCB. The smaller models, GMViT-simple and GMViT-mini, reduce the parameter size by 8 and 17.6 times, respectively, and improve shape recognition speed by 1.5 times on average, while preserving at least 90% of the classification and retrieval performance.

Title: A Non-Uniform Low-Light Image Enhancement Method with Multi-Scale Attention Transformer and Luminance Consistency Loss. (arXiv:2312.16498v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.16498
• Code URL: null
• Copy Paste: [[2312.16498]] A Non-Uniform Low-Light Image Enhancement Method with Multi-Scale Attention Transformer and Luminance Consistency Loss(http://arxiv.org/abs/2312.16498)
• Summary:

  Low-light image enhancement aims to improve the perception of images collected in dim environments and provide high-quality data support for image recognition tasks. When dealing with photos captured under non-uniform illumination, existing methods cannot adaptively extract the differentiated luminance information, which will easily cause over-exposure and under-exposure. From the perspective of unsupervised learning, we propose a multi-scale attention Transformer named MSATr, which sufficiently extracts local and global features for light balance to improve the visual quality. Specifically, we present a multi-scale window division scheme, which uses exponential sequences to adjust the window size of each layer. Within different-sized windows, the self-attention computation can be refined, ensuring the pixel-level feature processing capability of the model. For feature interaction across windows, a global transformer branch is constructed to provide comprehensive brightness perception and alleviate exposure problems. Furthermore, we propose a loop training strategy, using the diverse images generated by weighted mixing and a luminance consistency loss to improve the model's generalization ability effectively. Extensive experiments on several benchmark datasets quantitatively and qualitatively prove that our MSATr is superior to state-of-the-art low-light image enhancement methods, and the enhanced images have more natural brightness and outstanding details. The code is released at https://github.com/fang001021/MSATr.

Title: Forgery-aware Adaptive Transformer for Generalizable Synthetic Image Detection. (arXiv:2312.16649v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.16649
• Code URL: null
• Copy Paste: [[2312.16649]] Forgery-aware Adaptive Transformer for Generalizable Synthetic Image Detection(http://arxiv.org/abs/2312.16649)
• Summary:

  In this paper, we study the problem of generalizable synthetic image detection, aiming to detect forgery images from diverse generative methods, e.g., GANs and diffusion models. Cutting-edge solutions start to explore the benefits of pre-trained models, and mainly follow the fixed paradigm of solely training an attached classifier, e.g., combining frozen CLIP-ViT with a learnable linear layer in UniFD. However, our analysis shows that such a fixed paradigm is prone to yield detectors with insufficient learning regarding forgery representations. We attribute the key challenge to the lack of forgery adaptation, and present a novel forgery-aware adaptive transformer approach, namely FatFormer. Based on the pre-trained vision-language spaces of CLIP, FatFormer introduces two core designs for the adaption to build generalized forgery representations. First, motivated by the fact that both image and frequency analysis are essential for synthetic image detection, we develop a forgery-aware adapter to adapt image features to discern and integrate local forgery traces within image and frequency domains. Second, we find that considering the contrastive objectives between adapted image features and text prompt embeddings, a previously overlooked aspect, results in a nontrivial generalization improvement. Accordingly, we introduce language-guided alignment to supervise the forgery adaptation with image and text prompts in FatFormer. Experiments show that, by coupling these two designs, our approach tuned on 4-class ProGAN data attains a remarkable detection performance, achieving an average of 98% accuracy to unseen GANs, and surprisingly generalizes to unseen diffusion models with 95% accuracy.

Title: Observable Propagation: A Data-Efficient Approach to Uncover Feature Vectors in Transformers. (arXiv:2312.16291v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.16291
• Code URL: https://github.com/jacobdunefsky/observablepropagation
• Copy Paste: [[2312.16291]] Observable Propagation: A Data-Efficient Approach to Uncover Feature Vectors in Transformers(http://arxiv.org/abs/2312.16291)
• Summary:

  A key goal of current mechanistic interpretability research in NLP is to find linear features (also called "feature vectors") for transformers: directions in activation space corresponding to concepts that are used by a given model in its computation. Present state-of-the-art methods for finding linear features require large amounts of labelled data -- both laborious to acquire and computationally expensive to utilize. In this work, we introduce a novel method, called "observable propagation" (in short: ObsProp), for finding linear features used by transformer language models in computing a given task -- using almost no data. Our paradigm centers on the concept of observables, linear functionals corresponding to given tasks. We then introduce a mathematical theory for the analysis of feature vectors: we provide theoretical motivation for why LayerNorm nonlinearities do not affect the direction of feature vectors; we also introduce a similarity metric between feature vectors called the coupling coefficient which estimates the degree to which one feature's output correlates with another's. We use ObsProp to perform extensive qualitative investigations into several tasks, including gendered occupational bias, political party prediction, and programming language detection. Our results suggest that ObsProp surpasses traditional approaches for finding feature vectors in the low-data regime, and that ObsProp can be used to better understand the mechanisms responsible for bias in large language models. Code for experiments can be found at github.com/jacobdunefsky/ObservablePropagation.

Title: AdapterDistillation: Non-Destructive Task Composition with Knowledge Distillation. (arXiv:2312.16261v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.16261
• Code URL: null
• Copy Paste: [[2312.16261]] AdapterDistillation: Non-Destructive Task Composition with Knowledge Distillation(http://arxiv.org/abs/2312.16261)
• Summary:

  Leveraging knowledge from multiple tasks through introducing a small number of task specific parameters into each transformer layer, also known as adapters, receives much attention recently. However, adding an extra fusion layer to implement knowledge composition not only increases the inference time but also is non-scalable for some applications. To avoid these issues, we propose a two-stage knowledge distillation algorithm called AdapterDistillation. In the first stage, we extract task specific knowledge by using local data to train a student adapter. In the second stage, we distill the knowledge from the existing teacher adapters into the student adapter to help its inference. Extensive experiments on frequently asked question retrieval in task-oriented dialog systems validate the efficiency of AdapterDistillation. We show that AdapterDistillation outperforms existing algorithms in terms of accuracy, resource consumption and inference time.

Title: Learning to Embed Time Series Patches Independently. (arXiv:2312.16427v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.16427
• Code URL: https://github.com/seunghan96/pits
• Copy Paste: [[2312.16427]] Learning to Embed Time Series Patches Independently(http://arxiv.org/abs/2312.16427)
• Summary:

  Masked time series modeling has recently gained much attention as a self-supervised representation learning strategy for time series. Inspired by masked image modeling in computer vision, recent works first patchify and partially mask out time series, and then train Transformers to capture the dependencies between patches by predicting masked patches from unmasked patches. However, we argue that capturing such patch dependencies might not be an optimal strategy for time series representation learning; rather, learning to embed patches independently results in better time series representations. Specifically, we propose to use 1) the simple patch reconstruction task, which autoencode each patch without looking at other patches, and 2) the simple patch-wise MLP that embeds each patch independently. In addition, we introduce complementary contrastive learning to hierarchically capture adjacent time series information efficiently. Our proposed method improves time series forecasting and classification performance compared to state-of-the-art Transformer-based models, while it is more efficient in terms of the number of parameters and training/inference time. Code is available at this repository: https://github.com/seunghan96/pits.

Title: Knowledge Enhanced Conditional Imputation for Healthcare Time-series. (arXiv:2312.16713v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.16713
• Code URL: null
• Copy Paste: [[2312.16713]] Knowledge Enhanced Conditional Imputation for Healthcare Time-series(http://arxiv.org/abs/2312.16713)
• Summary:

  This study presents a novel approach to addressing the challenge of missing data in multivariate time series, with a particular focus on the complexities of healthcare data. Our Conditional Self-Attention Imputation (CSAI) model, grounded in a transformer-based framework, introduces a conditional hidden state initialization tailored to the intricacies of medical time series data. This methodology diverges from traditional imputation techniques by specifically targeting the imbalance in missing data distribution, a crucial aspect often overlooked in healthcare datasets. By integrating advanced knowledge embedding and a non-uniform masking strategy, CSAI adeptly adjusts to the distinct patterns of missing data in Electronic Health Records (EHRs).

Title: Mitigating Degree Biases in Message Passing Mechanism by Utilizing Community Structures. (arXiv:2312.16788v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.16788
• Code URL: https://github.com/nslab-cuk/community-aware-graph-transformer
• Copy Paste: [[2312.16788]] Mitigating Degree Biases in Message Passing Mechanism by Utilizing Community Structures(http://arxiv.org/abs/2312.16788)
• Summary:

  This study utilizes community structures to address node degree biases in message-passing (MP) via learnable graph augmentations and novel graph transformers. Recent augmentation-based methods showed that MP neural networks often perform poorly on low-degree nodes, leading to degree biases due to a lack of messages reaching low-degree nodes. Despite their success, most methods use heuristic or uniform random augmentations, which are non-differentiable and may not always generate valuable edges for learning representations. In this paper, we propose Community-aware Graph Transformers, namely CGT, to learn degree-unbiased representations based on learnable augmentations and graph transformers by extracting within community structures. We first design a learnable graph augmentation to generate more within-community edges connecting low-degree nodes through edge perturbation. Second, we propose an improved self-attention to learn underlying proximity and the roles of nodes within the community. Third, we propose a self-supervised learning task that could learn the representations to preserve the global graph structure and regularize the graph augmentations. Extensive experiments on various benchmark datasets showed CGT outperforms state-of-the-art baselines and significantly improves the node degree biases. The source code is available at https://github.com/NSLab-CUK/Community-aware-Graph-Transformer.

Title: Learning the Dynamic Correlations and Mitigating Noise by Hierarchical Convolution for Long-term Sequence Forecasting. (arXiv:2312.16790v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.16790
• Code URL: https://github.com/yzhhoward/hmnet
• Copy Paste: [[2312.16790]] Learning the Dynamic Correlations and Mitigating Noise by Hierarchical Convolution for Long-term Sequence Forecasting(http://arxiv.org/abs/2312.16790)
• Summary:

  Deep learning algorithms, especially Transformer-based models, have achieved significant performance by capturing long-range dependencies and historical information. However, the power of convolution has not been fully investigated. Moreover, most existing works ignore the dynamic interaction among variables and evolutionary noise in series. Addressing these issues, we propose a Hierarchical Memorizing Network (HMNet). In particular, a hierarchical convolution structure is introduced to extract the information from the series at various scales. Besides, we propose a dynamic variable interaction module to learn the varying correlation and an adaptive denoising module to search and exploit similar patterns to alleviate noises. These modules can cooperate with the hierarchical structure from the perspective of fine to coarse grain. Experiments on five benchmarks demonstrate that HMNet significantly outperforms the state-of-the-art models by 10.6% on MSE and 5.7% on MAE. Our code is released at https://github.com/yzhHoward/HMNet.

generative

Title: AI Mirage: The Impostor Bias and the Deepfake Detection Challenge in the Era of Artificial Illusions. (arXiv:2312.16220v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.16220
• Code URL: null
• Copy Paste: [[2312.16220]] AI Mirage: The Impostor Bias and the Deepfake Detection Challenge in the Era of Artificial Illusions(http://arxiv.org/abs/2312.16220)
• Summary:

  This paper provides a comprehensive analysis of cognitive biases in forensics and digital forensics, examining their implications for decision-making processes in these fields. It explores the various types of cognitive biases that may arise during forensic investigations and digital forensic analyses, such as confirmation bias, expectation bias, overconfidence in errors, contextual bias, and attributional biases. It also evaluates existing methods and techniques used to mitigate cognitive biases in these contexts, assessing the effectiveness of interventions aimed at reducing biases and improving decision-making outcomes. Additionally, this paper introduces a new cognitive bias, called "impostor bias", that may affect the use of generative Artificial Intelligence (AI) tools in forensics and digital forensics. The impostor bias is the tendency to doubt the authenticity or validity of the output generated by AI tools, such as deepfakes, in the form of audio, images, and videos. This bias may lead to erroneous judgments or false accusations, undermining the reliability and credibility of forensic evidence. The paper discusses the potential causes and consequences of the impostor bias, and suggests some strategies to prevent or counteract it. By addressing these topics, this paper seeks to offer valuable insights into understanding cognitive biases in forensic practices and provide recommendations for future research and practical applications to enhance the objectivity and validity of forensic investigations.

Title: MetaScript: Few-Shot Handwritten Chinese Content Generation via Generative Adversarial Networks. (arXiv:2312.16251v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.16251
• Code URL: null
• Copy Paste: [[2312.16251]] MetaScript: Few-Shot Handwritten Chinese Content Generation via Generative Adversarial Networks(http://arxiv.org/abs/2312.16251)
• Summary:

  In this work, we propose MetaScript, a novel Chinese content generation system designed to address the diminishing presence of personal handwriting styles in the digital representation of Chinese characters. Our approach harnesses the power of few-shot learning to generate Chinese characters that not only retain the individual's unique handwriting style but also maintain the efficiency of digital typing. Trained on a diverse dataset of handwritten styles, MetaScript is adept at producing high-quality stylistic imitations from minimal style references and standard fonts. Our work demonstrates a practical solution to the challenges of digital typography in preserving the personal touch in written communication, particularly in the context of Chinese script. Notably, our system has demonstrated superior performance in various evaluations, including recognition accuracy, inception score, and Frechet inception distance. At the same time, the training conditions of our model are easy to meet and facilitate generalization to real applications.

Title: Bellman Optimal Step-size Straightening of Flow-Matching Models. (arXiv:2312.16414v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.16414
• Code URL: null
• Copy Paste: [[2312.16414]] Bellman Optimal Step-size Straightening of Flow-Matching Models(http://arxiv.org/abs/2312.16414)
• Summary:

  Flow matching is a powerful framework for generating high-quality samples in various applications, especially image synthesis. However, the intensive computational demands of these models, especially during the fine-tuning process and sampling processes, pose significant challenges for low-resource scenarios. This paper introduces Bellman Optimal Step-size Straightening (BOSS) technique for distilling flow-matching generative models: it aims specifically for a few step efficient image sampling while adhering to a computational budget constraint. First, this technique involves a dynamic programming algorithm that optimizes the step sizes of the pretrained network. Then, it refines the velocity network to match the optimal step sizes, aiming to straighten the generation paths. Extensive experimental evaluations across image generation tasks demonstrate the efficacy of BOSS in terms of both resource utilization and image quality. Our results reveal that BOSS achieves substantial gains in efficiency while maintaining competitive sample quality, effectively bridging the gap between low-resource constraints and the demanding requirements of flow-matching generative models. Our paper also fortifies the responsible development of artificial intelligence, offering a more sustainable generative model that reduces computational costs and environmental footprints. Our code can be found at https://anonymous.4open.science/r/DRL-8E88.

Title: Disentangled Continual Learning: Separating Memory Edits from Model Updates. (arXiv:2312.16731v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.16731
• Code URL: null
• Copy Paste: [[2312.16731]] Disentangled Continual Learning: Separating Memory Edits from Model Updates(http://arxiv.org/abs/2312.16731)
• Summary:

  The ability of machine learning systems to learn continually is hindered by catastrophic forgetting, the tendency of neural networks to overwrite existing knowledge when learning a new task. Existing continual learning methods alleviate this problem through regularisation, parameter isolation, or rehearsal, and are typically evaluated on benchmarks consisting of a handful of tasks. We propose a novel conceptual approach to continual classification that aims to disentangle class-specific information that needs to be memorised from the class-agnostic knowledge that encapsulates generalization. We store the former in a buffer that can be easily pruned or updated when new categories arrive, while the latter is represented with a neural network that generalizes across tasks. We show that the class-agnostic network does not suffer from catastrophic forgetting and by leveraging it to perform classification, we improve accuracy on past tasks over time. In addition, our approach supports open-set classification and one-shot generalization. To test our conceptual framework, we introduce Infinite dSprites, a tool for creating continual classification and disentanglement benchmarks of arbitrary length with full control over generative factors. We show that over a sufficiently long time horizon all major types of continual learning methods break down, while our approach enables continual learning over hundreds of tasks with explicit control over memorization and forgetting.

Title: Active Third-Person Imitation Learning. (arXiv:2312.16365v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.16365
• Code URL: null
• Copy Paste: [[2312.16365]] Active Third-Person Imitation Learning(http://arxiv.org/abs/2312.16365)
• Summary:

  We consider the problem of third-person imitation learning with the additional challenge that the learner must select the perspective from which they observe the expert. In our setting, each perspective provides only limited information about the expert's behavior, and the learning agent must carefully select and combine information from different perspectives to achieve competitive performance. This setting is inspired by real-world imitation learning applications, e.g., in robotics, a robot might observe a human demonstrator via camera and receive information from different perspectives depending on the camera's position. We formalize the aforementioned active third-person imitation learning problem, theoretically analyze its characteristics, and propose a generative adversarial network-based active learning approach. Empirically, we demstrate that our proposed approach can effectively learn from expert demonstrations and explore the importance of different architectural choices for the learner's performance.

large language model

Title: ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulation. (arXiv:2312.16217v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.16217
• Code URL: null
• Copy Paste: [[2312.16217]] ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulation(http://arxiv.org/abs/2312.16217)
• Summary:

  Robot manipulation relies on accurately predicting contact points and end-effector directions to ensure successful operation. However, learning-based robot manipulation, trained on a limited category within a simulator, often struggles to achieve generalizability, especially when confronted with extensive categories. Therefore, we introduce an innovative approach for robot manipulation that leverages the robust reasoning capabilities of Multimodal Large Language Models (MLLMs) to enhance the stability and generalization of manipulation. By fine-tuning the injected adapters, we preserve the inherent common sense and reasoning ability of the MLLMs while equipping them with the ability for manipulation. The fundamental insight lies in the introduced fine-tuning paradigm, encompassing object category understanding, affordance prior reasoning, and object-centric pose prediction to stimulate the reasoning ability of MLLM in manipulation. During inference, our approach utilizes an RGB image and text prompt to predict the end effector's pose in chain of thoughts. After the initial contact is established, an active impedance adaptation policy is introduced to plan the upcoming waypoints in a closed-loop manner. Moreover, in real world, we design a test-time adaptation (TTA) strategy for manipulation to enable the model better adapt to the current real-world scene configuration. Experiments in simulator and real-world show the promising performance of ManipLLM. More details and demonstrations can be found at https://sites.google.com/view/manipllm.

Title: Cloud-Device Collaborative Learning for Multimodal Large Language Models. (arXiv:2312.16279v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.16279
• Code URL: null
• Copy Paste: [[2312.16279]] Cloud-Device Collaborative Learning for Multimodal Large Language Models(http://arxiv.org/abs/2312.16279)
• Summary:

  The burgeoning field of Multimodal Large Language Models (MLLMs) has exhibited remarkable performance in diverse tasks such as captioning, commonsense reasoning, and visual scene understanding. However, the deployment of these large-scale MLLMs on client devices is hindered by their extensive model parameters, leading to a notable decline in generalization capabilities when these models are compressed for device deployment. Addressing this challenge, we introduce a Cloud-Device Collaborative Continual Adaptation framework, designed to enhance the performance of compressed, device-deployed MLLMs by leveraging the robust capabilities of cloud-based, larger-scale MLLMs. Our framework is structured into three key components: a device-to-cloud uplink for efficient data transmission, cloud-based knowledge adaptation, and an optimized cloud-to-device downlink for model deployment. In the uplink phase, we employ an Uncertainty-guided Token Sampling (UTS) strategy to effectively filter out-of-distribution tokens, thereby reducing transmission costs and improving training efficiency. On the cloud side, we propose Adapter-based Knowledge Distillation (AKD) method to transfer refined knowledge from large-scale to compressed, pocket-size MLLMs. Furthermore, we propose a Dynamic Weight update Compression (DWC) strategy for the downlink, which adaptively selects and quantizes updated weight parameters, enhancing transmission efficiency and reducing the representational disparity between cloud and device models. Extensive experiments on several multimodal benchmarks demonstrate the superiority of our proposed framework over prior Knowledge Distillation and device-cloud collaboration methods. Notably, we also validate the feasibility of our approach to real-world experiments.

Title: Chatbot is Not All You Need: Information-rich Prompting for More Realistic Responses. (arXiv:2312.16233v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.16233
• Code URL: null
• Copy Paste: [[2312.16233]] Chatbot is Not All You Need: Information-rich Prompting for More Realistic Responses(http://arxiv.org/abs/2312.16233)
• Summary:

  Recent Large Language Models (LLMs) have shown remarkable capabilities in mimicking fictional characters or real humans in conversational settings. However, the realism and consistency of these responses can be further enhanced by providing richer information of the agent being mimicked. In this paper, we propose a novel approach to generate more realistic and consistent responses from LLMs, leveraging five senses, attributes, emotional states, relationship with the interlocutor, and memories. By incorporating these factors, we aim to increase the LLM's capacity for generating natural and realistic reactions in conversational exchanges. Through our research, we expect to contribute to the development of LLMs that demonstrate improved capabilities in mimicking fictional characters. We release a new benchmark dataset and all our codes, prompts, and sample results on our Github: https://github.com/srafsasm/InfoRichBot

Title: More than Correlation: Do Large Language Models Learn Causal Representations of Space?. (arXiv:2312.16257v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.16257
• Code URL: null
• Copy Paste: [[2312.16257]] More than Correlation: Do Large Language Models Learn Causal Representations of Space?(http://arxiv.org/abs/2312.16257)
• Summary:

  Recent work found high mutual information between the learned representations of large language models (LLMs) and the geospatial property of its input, hinting an emergent internal model of space. However, whether this internal space model has any causal effects on the LLMs' behaviors was not answered by that work, led to criticism of these findings as mere statistical correlation. Our study focused on uncovering the causality of the spatial representations in LLMs. In particular, we discovered the potential spatial representations in DeBERTa, GPT-Neo using representational similarity analysis and linear and non-linear probing. Our casual intervention experiments showed that the spatial representations influenced the model's performance on next word prediction and a downstream task that relies on geospatial information. Our experiments suggested that the LLMs learn and use an internal model of space in solving geospatial related tasks.

Title: LLM Polygraph: Uncovering LLMs' Factual Discernment through Intermediate Data Analysis. (arXiv:2312.16374v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.16374
• Code URL: null
• Copy Paste: [[2312.16374]] LLM Polygraph: Uncovering LLMs' Factual Discernment through Intermediate Data Analysis(http://arxiv.org/abs/2312.16374)
• Summary:

  Large Language Models (LLMs) have revolutionized various domains with extensive knowledge and creative capabilities. However, a critical issue with LLMs is their tendency to produce outputs that diverge from factual reality. This phenomenon is particularly concerning in sensitive applications such as medical consultation and legal advice, where accuracy is paramount. In this paper, we introduce the LLM factoscope, a novel Siamese network-based model that leverages the inner states of LLMs for factual detection. Our investigation reveals distinguishable patterns in LLMs' inner states when generating factual versus non-factual content. We demonstrate the LLM factoscope's effectiveness across various architectures, achieving over 96% accuracy in factual detection. Our work opens a new avenue for utilizing LLMs' inner states for factual detection and encourages further exploration into LLMs' inner workings for enhanced reliability and transparency.

Title: Automating Knowledge Acquisition for Content-Centric Cognitive Agents Using LLMs. (arXiv:2312.16378v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.16378
• Code URL: null
• Copy Paste: [[2312.16378]] Automating Knowledge Acquisition for Content-Centric Cognitive Agents Using LLMs(http://arxiv.org/abs/2312.16378)
• Summary:

  The paper describes a system that uses large language model (LLM) technology to support the automatic learning of new entries in an intelligent agent's semantic lexicon. The process is bootstrapped by an existing non-toy lexicon and a natural language generator that converts formal, ontologically-grounded representations of meaning into natural language sentences. The learning method involves a sequence of LLM requests and includes an automatic quality control step. To date, this learning method has been applied to learning multiword expressions whose meanings are equivalent to those of transitive verbs in the agent's lexicon. The experiment demonstrates the benefits of a hybrid learning architecture that integrates knowledge-based methods and resources with both traditional data analytics and LLMs.

Title: A Large Language Model-based Computational Approach to Improve Identity-Related Write-Ups. (arXiv:2312.16659v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.16659
• Code URL: null
• Copy Paste: [[2312.16659]] A Large Language Model-based Computational Approach to Improve Identity-Related Write-Ups(http://arxiv.org/abs/2312.16659)
• Summary:

  Creating written products is essential to modern life, including writings about one's identity and personal experiences. However, writing is often a difficult activity that requires extensive effort to frame the central ideas, the pursued approach to communicate the central ideas, e.g., using analogies, metaphors, or other possible means, the needed presentation structure, and the actual verbal expression. Large Language Models, a recently emerged approach in Machine Learning, can offer a significant help in reducing the effort and improving the quality of written products. This paper proposes a new computational approach to explore prompts that given as inputs to a Large Language Models can generate cues to improve the considered written products. Two case studies on improving write-ups, one based on an analogy and one on a metaphor, are also presented in the paper.

Title: Some things are more CRINGE than others: Preference Optimization with the Pairwise Cringe Loss. (arXiv:2312.16682v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.16682
• Code URL: null
• Copy Paste: [[2312.16682]] Some things are more CRINGE than others: Preference Optimization with the Pairwise Cringe Loss(http://arxiv.org/abs/2312.16682)
• Summary:

  Practitioners commonly align large language models using pairwise preferences, i.e., given labels of the type response A is preferred to response B for a given input. Perhaps less commonly, methods have also been developed for binary feedback, i.e. training models given labels of type response A is good or bad. We show how an existing performant binary feedback method, the Cringe Loss (Adolphs et al., 2022), can be generalized to the pairwise preference setting using a simple soft margin extension. Pairwise Cringe Loss is straightforward to implement and efficient to train, and we find it outperforms state-of-the-art preference optimization algorithms such as PPO and DPO on the AlpacaFarm benchmark.

Title: Rethinking Tabular Data Understanding with Large Language Models. (arXiv:2312.16702v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.16702
• Code URL: https://github.com/Leolty/tablellm
• Copy Paste: [[2312.16702]] Rethinking Tabular Data Understanding with Large Language Models(http://arxiv.org/abs/2312.16702)
• Summary:

  Large Language Models (LLMs) have shown to be capable of various tasks, yet their capability in interpreting and reasoning over tabular data remains an underexplored area. In this context, this study investigates from three core perspectives: the robustness of LLMs to structural perturbations in tables, the comparative analysis of textual and symbolic reasoning on tables, and the potential of boosting model performance through the aggregation of multiple reasoning pathways. We discover that structural variance of tables presenting the same content reveals a notable performance decline, particularly in symbolic reasoning tasks. This prompts the proposal of a method for table structure normalization. Moreover, textual reasoning slightly edges out symbolic reasoning, and a detailed error analysis reveals that each exhibits different strengths depending on the specific tasks. Notably, the aggregation of textual and symbolic reasoning pathways, bolstered by a mix self-consistency mechanism, resulted in achieving SOTA performance, with an accuracy of 73.6% on WIKITABLEQUESTIONS, representing a substantial advancement over previous existing table processing paradigms of LLMs.

segmentation

Title: Multi-modality Affinity Inference for Weakly Supervised 3D Semantic Segmentation. (arXiv:2312.16578v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.16578
• Code URL: null
• Copy Paste: [[2312.16578]] Multi-modality Affinity Inference for Weakly Supervised 3D Semantic Segmentation(http://arxiv.org/abs/2312.16578)
• Summary:

  3D point cloud semantic segmentation has a wide range of applications. Recently, weakly supervised point cloud segmentation methods have been proposed, aiming to alleviate the expensive and laborious manual annotation process by leveraging scene-level labels. However, these methods have not effectively exploited the rich geometric information (such as shape and scale) and appearance information (such as color and texture) present in RGB-D scans. Furthermore, current approaches fail to fully leverage the point affinity that can be inferred from the feature extraction network, which is crucial for learning from weak scene-level labels. Additionally, previous work overlooks the detrimental effects of the long-tailed distribution of point cloud data in weakly supervised 3D semantic segmentation. To this end, this paper proposes a simple yet effective scene-level weakly supervised point cloud segmentation method with a newly introduced multi-modality point affinity inference module. The point affinity proposed in this paper is characterized by features from multiple modalities (e.g., point cloud and RGB), and is further refined by normalizing the classifier weights to alleviate the detrimental effects of long-tailed distribution without the need of the prior of category distribution. Extensive experiments on the ScanNet and S3DIS benchmarks verify the effectiveness of our proposed method, which outperforms the state-of-the-art by ~4% to ~6% mIoU. Codes are released at https://github.com/Sunny599/AAAI24-3DWSSG-MMA.

Title: Efficient Deweather Mixture-of-Experts with Uncertainty-aware Feature-wise Linear Modulation. (arXiv:2312.16610v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.16610
• Code URL: null
• Copy Paste: [[2312.16610]] Efficient Deweather Mixture-of-Experts with Uncertainty-aware Feature-wise Linear Modulation(http://arxiv.org/abs/2312.16610)
• Summary:

  The Mixture-of-Experts (MoE) approach has demonstrated outstanding scalability in multi-task learning including low-level upstream tasks such as concurrent removal of multiple adverse weather effects. However, the conventional MoE architecture with parallel Feed Forward Network (FFN) experts leads to significant parameter and computational overheads that hinder its efficient deployment. In addition, the naive MoE linear router is suboptimal in assigning task-specific features to multiple experts which limits its further scalability. In this work, we propose an efficient MoE architecture with weight sharing across the experts. Inspired by the idea of linear feature modulation (FM), our architecture implicitly instantiates multiple experts via learnable activation modulations on a single shared expert block. The proposed Feature Modulated Expert (FME) serves as a building block for the novel Mixture-of-Feature-Modulation-Experts (MoFME) architecture, which can scale up the number of experts with low overhead. We further propose an Uncertainty-aware Router (UaR) to assign task-specific features to different FM modules with well-calibrated weights. This enables MoFME to effectively learn diverse expert functions for multiple tasks. The conducted experiments on the multi-deweather task show that our MoFME outperforms the baselines in the image restoration quality by 0.1-0.2 dB and achieves SOTA-compatible performance while saving more than 72% of parameters and 39% inference time over the conventional MoE counterpart. Experiments on the downstream segmentation and classification tasks further demonstrate the generalizability of MoFME to real open-world applications.

Title: Landslide Detection and Segmentation Using Remote Sensing Images and Deep Neural Network. (arXiv:2312.16717v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.16717
• Code URL: null
• Copy Paste: [[2312.16717]] Landslide Detection and Segmentation Using Remote Sensing Images and Deep Neural Network(http://arxiv.org/abs/2312.16717)
• Summary:

  Knowledge about historic landslide event occurrence is important for supporting disaster risk reduction strategies. Building upon findings from 2022 Landslide4Sense Competition, we propose a deep neural network based system for landslide detection and segmentation from multisource remote sensing image input. We use a U-Net trained with Cross Entropy loss as baseline model. We then improve the U-Net baseline model by leveraging a wide range of deep learning techniques. In particular, we conduct feature engineering by generating new band data from the original bands, which helps to enhance the quality of remote sensing image input. Regarding the network architecture, we replace traditional convolutional layers in the U-Net baseline by a residual-convolutional layer. We also propose an attention layer which leverages the multi-head attention scheme. Additionally, we generate multiple output masks with three different resolutions, which creates an ensemble of three outputs in the inference process to enhance the performance. Finally, we propose a combined loss function which leverages Focal loss and IoU loss to train the network. Our experiments on the development set of the Landslide4Sense challenge achieve an F1 score and an mIoU score of 84.07 and 76.07, respectively. Our best model setup outperforms the challenge baseline and the proposed U-Net baseline, improving the F1 score/mIoU score by 6.8/7.4 and 10.5/8.8, respectively.