2024-01-26

Title: PQCMC: Post-Quantum Cryptography McEliece-Chen Implicit Certificate Scheme

• Authors: Abel C. H. Chen
• Subjects: cs.CR, cs.NI
• Abstract URL: https://arxiv.org/abs/2401.13691
• Pdf URL: https://arxiv.org/pdf/2401.13691
• Copy Paste: [[2401.13691]] PQCMC: Post-Quantum Cryptography McEliece-Chen Implicit Certificate Scheme(https://arxiv.org/abs/2401.13691)
• Keywords: secure, security, attack
• Abstract: In recent years, the elliptic curve Qu-Vanstone (ECQV) implicit certificate scheme has found application in security credential management systems (SCMS) and secure vehicle-to-everything (V2X) communication to issue pseudonymous certificates. However, the vulnerability of elliptic-curve cryptography (ECC) to polynomial-time attacks posed by quantum computing raises concerns. In order to enhance resistance against quantum computing threats, various post-quantum cryptography methods have been adopted as standard (e.g. Dilithium) or candidate standard methods (e.g. McEliece cryptography), but state of the art has proven to be challenging to implement implicit certificates using lattice-based cryptography methods. Therefore, this study proposes a post-quantum cryptography McEliece-Chen (PQCMC) based on an efficient random invertible matrix generation method to issue pseudonymous certificates with less computation time. The study provides mathematical models to validate the key expansion process for implicit certificates. Furthermore, comprehensive security evaluations and discussions are conducted to demonstrate that distinct implicit certificates can be linked to the same end entity. In experiments, a comparison is conducted between the certificate length and computation time to evaluate the performance of the proposed PQCMC. This study demonstrates the viability of the implicit certificate scheme based on PQC as a means of countering quantum computing threats.

Title: Local Privacy-preserving Mechanisms and Applications in Machine Learning

• Authors: Likun Qin, Tianshuo Qiu
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2401.13692
• Pdf URL: https://arxiv.org/pdf/2401.13692
• Copy Paste: [[2401.13692]] Local Privacy-preserving Mechanisms and Applications in Machine Learning(https://arxiv.org/abs/2401.13692)
• Keywords: privacy, protect
• Abstract: The emergence and evolution of Local Differential Privacy (LDP) and its various adaptations play a pivotal role in tackling privacy issues related to the vast amounts of data generated by intelligent devices, which are crucial for data-informed decision-making in the realm of crowdsensing. Utilizing these extensive datasets can provide critical insights but also introduces substantial privacy concerns for the individuals involved. LDP, noted for its decentralized framework, excels in providing strong privacy protection for individual users during the stages of data collection and processing. The core principle of LDP lies in its technique of altering each user's data locally at the client end before it is sent to the server, thus preventing privacy violations at both stages. There are many LDP variances in the privacy research community aimed to improve the utility-privacy tradeoff. On the other hand, one of the major applications of the privacy-preserving mechanisms is machine learning. In this paper, we firstly delves into a comprehensive analysis of LDP and its variances, focusing on their various models, the diverse range of its adaptations, and the underlying structure of privacy mechanisms; then we discuss the state-of-art privacy mechanisms applications in machine learning.

Title: Toward Robust Multimodal Learning using Multimodal Foundational Models

• Authors: Xianbing Zhao, Soujanya Poria, Xuejiao Li, Yixin Chen, Buzhou Tang
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2401.13697
• Pdf URL: https://arxiv.org/pdf/2401.13697
• Copy Paste: [[2401.13697]] Toward Robust Multimodal Learning using Multimodal Foundational Models(https://arxiv.org/abs/2401.13697)
• Keywords: robust
• Abstract: Existing multimodal sentiment analysis tasks are highly rely on the assumption that the training and test sets are complete multimodal data, while this assumption can be difficult to hold: the multimodal data are often incomplete in real-world scenarios. Therefore, a robust multimodal model in scenarios with randomly missing modalities is highly preferred. Recently, CLIP-based multimodal foundational models have demonstrated impressive performance on numerous multimodal tasks by learning the aligned cross-modal semantics of image and text pairs, but the multimodal foundational models are also unable to directly address scenarios involving modality absence. To alleviate this issue, we propose a simple and effective framework, namely TRML, Toward Robust Multimodal Learning using Multimodal Foundational Models. TRML employs generated virtual modalities to replace missing modalities, and aligns the semantic spaces between the generated and missing modalities. Concretely, we design a missing modality inference module to generate virtual modaliites and replace missing modalities. We also design a semantic matching learning module to align semantic spaces generated and missing modalities. Under the prompt of complete modality, our model captures the semantics of missing modalities by leveraging the aligned cross-modal semantic space. Experiments demonstrate the superiority of our approach on three multimodal sentiment analysis benchmark datasets, CMU-MOSI, CMU-MOSEI, and MELD.

Title: Can I trust my fake data -- A comprehensive quality assessment framework for synthetic tabular data in healthcare

• Authors: Vibeke Binz Vallevik, Aleksandar Babic, Serena Elizabeth Marshall, Severin Elvatun, Helga Brøgger, Sharmini Alagaratnam, Bjørn Edwin, Narasimha Raghavan Veeraragavan, Anne Kjersti Befring, Jan Franz Nygård
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.13716
• Pdf URL: https://arxiv.org/pdf/2401.13716
• Copy Paste: [[2401.13716]] Can I trust my fake data -- A comprehensive quality assessment framework for synthetic tabular data in healthcare(https://arxiv.org/abs/2401.13716)
• Keywords: privacy, fair, generative
• Abstract: Ensuring safe adoption of AI tools in healthcare hinges on access to sufficient data for training, testing and validation. In response to privacy concerns and regulatory requirements, using synthetic data has been suggested. Synthetic data is created by training a generator on real data to produce a dataset with similar statistical properties. Competing metrics with differing taxonomies for quality evaluation have been suggested, resulting in a complex landscape. Optimising quality entails balancing considerations that make the data fit for use, yet relevant dimensions are left out of existing frameworks. We performed a comprehensive literature review on the use of quality evaluation metrics on SD within the scope of tabular healthcare data and SD made using deep generative methods. Based on this and the collective team experiences, we developed a conceptual framework for quality assurance. The applicability was benchmarked against a practical case from the Dutch National Cancer Registry. We present a conceptual framework for quality assurance of SD for AI applications in healthcare that aligns diverging taxonomies, expands on common quality dimensions to include the dimensions of Fairness and Carbon footprint, and proposes stages necessary to support real-life applications. Building trust in synthetic data by increasing transparency and reducing the safety risk will accelerate the development and uptake of trustworthy AI tools for the benefit of patients. Despite the growing emphasis on algorithmic fairness and carbon footprint, these metrics were scarce in the literature review. The overwhelming focus was on statistical similarity using distance metrics while sequential logic detection was scarce. A consensus-backed framework that includes all relevant quality dimensions can provide assurance for safe and responsible real-life applications of SD.

Title: Inference Attacks Against Face Recognition Model without Classification Layers

• Authors: Yuanqing Huang, Huilong Chen, Yinggui Wang, Lei Wang
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.13719
• Pdf URL: https://arxiv.org/pdf/2401.13719
• Copy Paste: [[2401.13719]] Inference Attacks Against Face Recognition Model without Classification Layers(https://arxiv.org/abs/2401.13719)
• Keywords: privacy, attack, membership infer, generative
• Abstract: Face recognition (FR) has been applied to nearly every aspect of daily life, but it is always accompanied by the underlying risk of leaking private information. At present, almost all attack models against FR rely heavily on the presence of a classification layer. However, in practice, the FR model can obtain complex features of the input via the model backbone, and then compare it with the target for inference, which does not explicitly involve the outputs of the classification layer adopting logit or other losses. In this work, we advocate a novel inference attack composed of two stages for practical FR models without a classification layer. The first stage is the membership inference attack. Specifically, We analyze the distances between the intermediate features and batch normalization (BN) parameters. The results indicate that this distance is a critical metric for membership inference. We thus design a simple but effective attack model that can determine whether a face image is from the training dataset or not. The second stage is the model inversion attack, where sensitive private data is reconstructed using a pre-trained generative adversarial network (GAN) guided by the attack model in the first stage. To the best of our knowledge, the proposed attack model is the very first in the literature developed for FR models without a classification layer. We illustrate the application of the proposed attack model in the establishment of privacy-preserving FR techniques.

Title: A Systematic Approach to Robustness Modelling for Deep Convolutional Neural Networks

• Authors: Charles Meyers, Mohammad Reza Saleh Sedghpour, Tommy Löfstedt, Erik Elmroth
• Subjects: cs.LG, cs.AI, cs.CV, stat.ML
• Abstract URL: https://arxiv.org/abs/2401.13751
• Pdf URL: https://arxiv.org/pdf/2401.13751
• Copy Paste: [[2401.13751]] A Systematic Approach to Robustness Modelling for Deep Convolutional Neural Networks(https://arxiv.org/abs/2401.13751)
• Keywords: attack, robust
• Abstract: Convolutional neural networks have shown to be widely applicable to a large number of fields when large amounts of labelled data are available. The recent trend has been to use models with increasingly larger sets of tunable parameters to increase model accuracy, reduce model loss, or create more adversarially robust models -- goals that are often at odds with one another. In particular, recent theoretical work raises questions about the ability for even larger models to generalize to data outside of the controlled train and test sets. As such, we examine the role of the number of hidden layers in the ResNet model, demonstrated on the MNIST, CIFAR10, CIFAR100 datasets. We test a variety of parameters including the size of the model, the floating point precision, and the noise level of both the training data and the model output. To encapsulate the model's predictive power and computational cost, we provide a method that uses induced failures to model the probability of failure as a function of time and relate that to a novel metric that allows us to quickly determine whether or not the cost of training a model outweighs the cost of attacking it. Using this approach, we are able to approximate the expected failure rate using a small number of specially crafted samples rather than increasingly larger benchmark datasets. We demonstrate the efficacy of this technique on both the MNIST and CIFAR10 datasets using 8-, 16-, 32-, and 64-bit floating-point numbers, various data pre-processing techniques, and several attacks on five configurations of the ResNet model. Then, using empirical measurements, we examine the various trade-offs between cost, robustness, latency, and reliability to find that larger models do not significantly aid in adversarial robustness despite costing significantly more to train.

Title: S2TPVFormer: Spatio-Temporal Tri-Perspective View for temporally coherent 3D Semantic Occupancy Prediction

• Authors: Sathira Silva, Savindu Bhashitha Wannigama, Roshan Ragel, Gihan Jayatilaka
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.13785
• Pdf URL: https://arxiv.org/pdf/2401.13785
• Copy Paste: [[2401.13785]] S2TPVFormer: Spatio-Temporal Tri-Perspective View for temporally coherent 3D Semantic Occupancy Prediction(https://arxiv.org/abs/2401.13785)
• Keywords: transformer
• Abstract: Holistic understanding and reasoning in 3D scenes play a vital role in the success of autonomous driving systems. The evolution of 3D semantic occupancy prediction as a pretraining task for autonomous driving and robotic downstream tasks captures finer 3D details compared to methods like 3D detection. Existing approaches predominantly focus on spatial cues, often overlooking temporal cues. Query-based methods tend to converge on computationally intensive Voxel representation for encoding 3D scene information. This study introduces S2TPVFormer, an extension of TPVFormer, utilizing a spatiotemporal transformer architecture for coherent 3D semantic occupancy prediction. Emphasizing the importance of spatiotemporal cues in 3D scene perception, particularly in 3D semantic occupancy prediction, our work explores the less-explored realm of temporal cues. Leveraging Tri-Perspective View (TPV) representation, our spatiotemporal encoder generates temporally rich embeddings, improving prediction coherence while maintaining computational efficiency. To achieve this, we propose a novel Temporal Cross-View Hybrid Attention (TCVHA) mechanism, facilitating effective spatiotemporal information exchange across TPV views. Experimental evaluations on the nuScenes dataset demonstrate a substantial 3.1% improvement in mean Intersection over Union (mIoU) for 3D Semantic Occupancy compared to TPVFormer, confirming the effectiveness of the proposed S2TPVFormer in enhancing 3D scene perception.

Title: Diffuse to Choose: Enriching Image Conditioned Inpainting in Latent Diffusion Models for Virtual Try-All

• Authors: Mehmet Saygin Seyfioglu, Karim Bouyarmane, Suren Kumar, Amir Tavanaei, Ismail B. Tutar
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.13795
• Pdf URL: https://arxiv.org/pdf/2401.13795
• Copy Paste: [[2401.13795]] Diffuse to Choose: Enriching Image Conditioned Inpainting in Latent Diffusion Models for Virtual Try-All(https://arxiv.org/abs/2401.13795)
• Keywords: diffusion
• Abstract: As online shopping is growing, the ability for buyers to virtually visualize products in their settings-a phenomenon we define as "Virtual Try-All"-has become crucial. Recent diffusion models inherently contain a world model, rendering them suitable for this task within an inpainting context. However, traditional image-conditioned diffusion models often fail to capture the fine-grained details of products. In contrast, personalization-driven models such as DreamPaint are good at preserving the item's details but they are not optimized for real-time applications. We present "Diffuse to Choose," a novel diffusion-based image-conditioned inpainting model that efficiently balances fast inference with the retention of high-fidelity details in a given reference item while ensuring accurate semantic manipulations in the given scene content. Our approach is based on incorporating fine-grained features from the reference image directly into the latent feature maps of the main diffusion model, alongside with a perceptual loss to further preserve the reference item's details. We conduct extensive testing on both in-house and publicly available datasets, and show that Diffuse to Choose is superior to existing zero-shot diffusion inpainting methods as well as few-shot diffusion personalization algorithms like DreamPaint.

Title: Don't Push the Button! Exploring Data Leakage Risks in Machine Learning and Transfer Learning

• Authors: Andrea Apicella, Francesco Isgrò, Roberto Prevete
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.13796
• Pdf URL: https://arxiv.org/pdf/2401.13796
• Copy Paste: [[2401.13796]] Don't Push the Button! Exploring Data Leakage Risks in Machine Learning and Transfer Learning(https://arxiv.org/abs/2401.13796)
• Keywords: robust
• Abstract: Machine Learning (ML) has revolutionized various domains, offering predictive capabilities in several areas. However, with the increasing accessibility of ML tools, many practitioners, lacking deep ML expertise, adopt a "push the button" approach, utilizing user-friendly interfaces without a thorough understanding of underlying algorithms. While this approach provides convenience, it raises concerns about the reliability of outcomes, leading to challenges such as incorrect performance evaluation. This paper addresses a critical issue in ML, known as data leakage, where unintended information contaminates the training data, impacting model performance evaluation. Users, due to a lack of understanding, may inadvertently overlook crucial steps, leading to optimistic performance estimates that may not hold in real-world scenarios. The discrepancy between evaluated and actual performance on new data is a significant concern. In particular, this paper categorizes data leakage in ML, discussing how certain conditions can propagate through the ML workflow. Furthermore, it explores the connection between data leakage and the specific task being addressed, investigates its occurrence in Transfer Learning, and compares standard inductive ML with transductive ML frameworks. The conclusion summarizes key findings, emphasizing the importance of addressing data leakage for robust and reliable ML applications.

Title: Automated Root Causing of Cloud Incidents using In-Context Learning with GPT-4

• Authors: Xuchao Zhang, Supriyo Ghosh, Chetan Bansal, Rujia Wang, Minghua Ma, Yu Kang, Saravan Rajmohan
• Subjects: cs.CL, cs.SE
• Abstract URL: https://arxiv.org/abs/2401.13810
• Pdf URL: https://arxiv.org/pdf/2401.13810
• Copy Paste: [[2401.13810]] Automated Root Causing of Cloud Incidents using In-Context Learning with GPT-4(https://arxiv.org/abs/2401.13810)
• Keywords: large language model
• Abstract: Root Cause Analysis (RCA) plays a pivotal role in the incident diagnosis process for cloud services, requiring on-call engineers to identify the primary issues and implement corrective actions to prevent future recurrences. Improving the incident RCA process is vital for minimizing service downtime, customer impact and manual toil. Recent advances in artificial intelligence have introduced state-of-the-art Large Language Models (LLMs) like GPT-4, which have proven effective in tackling various AIOps problems, ranging from code authoring to incident management. Nonetheless, the GPT-4 model's immense size presents challenges when trying to fine-tune it on user data because of the significant GPU resource demand and the necessity for continuous model fine-tuning with the emergence of new data. To address the high cost of fine-tuning LLM, we propose an in-context learning approach for automated root causing, which eliminates the need for fine-tuning. We conduct extensive study over 100,000 production incidents, comparing several large language models using multiple metrics. The results reveal that our in-context learning approach outperforms the previous fine-tuned large language models such as GPT-3 by an average of 24.8\% across all metrics, with an impressive 49.7\% improvement over the zero-shot model. Moreover, human evaluation involving actual incident owners demonstrates its superiority over the fine-tuned model, achieving a 43.5\% improvement in correctness and an 8.7\% enhancement in readability. The impressive results demonstrate the viability of utilizing a vanilla GPT model for the RCA task, thereby avoiding the high computational and maintenance costs associated with a fine-tuned model.

Title: The Calibration Gap between Model and Human Confidence in Large Language Models

• Authors: Mark Steyvers, Heliodoro Tejeda, Aakriti Kumar, Catarina Belem, Sheer Karny, Xinyue Hu, Lukas Mayer, Padhraic Smyth
• Subjects: cs.LG, cs.AI, cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2401.13835
• Pdf URL: https://arxiv.org/pdf/2401.13835
• Copy Paste: [[2401.13835]] The Calibration Gap between Model and Human Confidence in Large Language Models(https://arxiv.org/abs/2401.13835)
• Keywords: large language model
• Abstract: For large language models (LLMs) to be trusted by humans they need to be well-calibrated in the sense that they can accurately assess and communicate how likely it is that their predictions are correct. Recent work has focused on the quality of internal LLM confidence assessments, but the question remains of how well LLMs can communicate this internal model confidence to human users. This paper explores the disparity between external human confidence in an LLM's responses and the internal confidence of the model. Through experiments involving multiple-choice questions, we systematically examine human users' ability to discern the reliability of LLM outputs. Our study focuses on two key areas: (1) assessing users' perception of true LLM confidence and (2) investigating the impact of tailored explanations on this perception. The research highlights that default explanations from LLMs often lead to user overestimation of both the model's confidence and its' accuracy. By modifying the explanations to more accurately reflect the LLM's internal confidence, we observe a significant shift in user perception, aligning it more closely with the model's actual confidence levels. This adjustment in explanatory approach demonstrates potential for enhancing user trust and accuracy in assessing LLM outputs. The findings underscore the importance of transparent communication of confidence levels in LLMs, particularly in high-stakes applications where understanding the reliability of AI-generated information is essential.

Title: Democratizing Fine-grained Visual Recognition with Large Language Models

• Authors: Mingxuan Liu, Subhankar Roy, Wenjing Li, Zhun Zhong, Nicu Sebe, Elisa Ricci
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.13837
• Pdf URL: https://arxiv.org/pdf/2401.13837
• Copy Paste: [[2401.13837]] Democratizing Fine-grained Visual Recognition with Large Language Models(https://arxiv.org/abs/2401.13837)
• Keywords: large language model
• Abstract: Identifying subordinate-level categories from images is a longstanding task in computer vision and is referred to as fine-grained visual recognition (FGVR). It has tremendous significance in real-world applications since an average layperson does not excel at differentiating species of birds or mushrooms due to subtle differences among the species. A major bottleneck in developing FGVR systems is caused by the need of high-quality paired expert annotations. To circumvent the need of expert knowledge we propose Fine-grained Semantic Category Reasoning (FineR) that internally leverages the world knowledge of large language models (LLMs) as a proxy in order to reason about fine-grained category names. In detail, to bridge the modality gap between images and LLM, we extract part-level visual attributes from images as text and feed that information to a LLM. Based on the visual attributes and its internal world knowledge the LLM reasons about the subordinate-level category names. Our training-free FineR outperforms several state-of-the-art FGVR and language and vision assistant models and shows promise in working in the wild and in new domains where gathering expert annotation is arduous.

Title: A V2X-based Privacy Preserving Federated Measuring and Learning System

• Authors: Levente Alekszejenkó, Tadeusz Dobrowiecki
• Subjects: cs.LG, cs.AI, cs.CR, stat.ML
• Abstract URL: https://arxiv.org/abs/2401.13848
• Pdf URL: https://arxiv.org/pdf/2401.13848
• Copy Paste: [[2401.13848]] A V2X-based Privacy Preserving Federated Measuring and Learning System(https://arxiv.org/abs/2401.13848)
• Keywords: privacy, federate
• Abstract: Future autonomous vehicles (AVs) will use a variety of sensors that generate a vast amount of data. Naturally, this data not only serves self-driving algorithms; but can also assist other vehicles or the infrastructure in real-time decision-making. Consequently, vehicles shall exchange their measurement data over Vehicle-to-Everything (V2X) technologies. Moreover, predicting the state of the road network might be beneficial too. With such a prediction, we might mitigate road congestion, balance parking lot usage, or optimize the traffic flow. That would decrease transportation costs as well as reduce its environmental impact. In this paper, we propose a federated measurement and learning system that provides real-time data to fellow vehicles over Vehicle-to-Vehicle (V2V) communication while also operating a federated learning (FL) scheme over the Vehicle-to-Network (V2N) link to create a predictive model of the transportation network. As we are yet to have real-world AV data, we model it with a non-IID (independent and identically distributed) dataset to evaluate the capabilities of the proposed system in terms of performance and privacy. Results indicate that the proposed FL scheme improves learning performance and prevents eavesdropping at the aggregator server side.

Title: TPD: Enhancing Student Language Model Reasoning via Principle Discovery and Guidance

• Authors: Haorui Wang (1), Rongzhi Zhang (1), Yinghao Li (1), Lingkai Kong (1), Yuchen Zhuang (1), Xiusi Chen (2), Chao Zhang (1) ((1) College of Computing, Georgia Institute of Technology, (2) Department of Computer Science, University of California, Los Angeles)
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.13849
• Pdf URL: https://arxiv.org/pdf/2401.13849
• Copy Paste: [[2401.13849]] TPD: Enhancing Student Language Model Reasoning via Principle Discovery and Guidance(https://arxiv.org/abs/2401.13849)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have recently showcased remarkable reasoning abilities. However, larger models often surpass their smaller counterparts in reasoning tasks, posing the challenge of effectively transferring these capabilities from larger models. Existing approaches heavily rely on extensive fine-tuning data or continuous interactions with a superior teacher LLM during inference. We introduce a principle-based teacher-student framework called ``Teaching via Principle Discovery'' (TPD) to address these limitations. Inspired by human learning mechanisms, TPD mimics the interaction between a teacher and a student using a principle-based approach. The teacher LLM generates problem-solving instructions and corrective principles based on the student LLM's errors. These principles guide the refinement of instructions and the selection of instructive examples from a validation set. This enables the student model to learn from both the teacher's guidance and its own mistakes. Once the student model begins making inferences, TPD requires no further intervention from the teacher LLM or humans. Through extensive experiments across eight reasoning tasks, we demonstrate the effectiveness of TPD. Compared to standard chain-of-thought prompting, TPD significantly improves the student model's performance, achieving $6.2\%$ improvement on average.

Title: Embedding Attack Project (Work Report)

• Authors: Jiameng Pu, Zafar Takhirov
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2401.13854
• Pdf URL: https://arxiv.org/pdf/2401.13854
• Copy Paste: [[2401.13854]] Embedding Attack Project (Work Report)(https://arxiv.org/abs/2401.13854)
• Keywords: privacy, defense, attack, membership infer
• Abstract: This report summarizes all the MIA experiments (Membership Inference Attacks) of the Embedding Attack Project, including threat models, experimental setup, experimental results, findings and discussion. Current results cover the evaluation of two main MIA strategies (loss-based and embedding-based MIAs) on 6 AI models ranging from Computer Vision to Language Modelling. There are two ongoing experiments on MIA defense and neighborhood-comparison embedding attacks. These are ongoing projects. The current work on MIA and PIA can be summarized into six conclusions: (1) Amount of overfitting is directly proportional to model's vulnerability; (2) early embedding layers in the model are less susceptible to privacy leaks; (3) Deeper model layers contain more membership information; (4) Models are more vulnerable to MIA if both embeddings and corresponding training labels are compromised; (5) it is possible to use pseudo-labels to increase the MIA success; and (6) although MIA and PIA success rates are proportional, reducing the MIA does not necessarily reduce the PIA.

Title: Inverse Molecular Design with Multi-Conditional Diffusion Guidance

• Authors: Gang Liu, Jiaxin Xu, Tengfei Luo, Meng Jiang
• Subjects: cs.LG, q-bio.BM
• Abstract URL: https://arxiv.org/abs/2401.13858
• Pdf URL: https://arxiv.org/pdf/2401.13858
• Copy Paste: [[2401.13858]] Inverse Molecular Design with Multi-Conditional Diffusion Guidance(https://arxiv.org/abs/2401.13858)
• Keywords: diffusion, transformer
• Abstract: Inverse molecular design with diffusion models holds great potential for advancements in material and drug discovery. Despite success in unconditional molecule generation, integrating multiple properties such as synthetic score and gas permeability as condition constraints into diffusion models remains unexplored. We introduce multi-conditional diffusion guidance. The proposed Transformer-based denoising model has a condition encoder that learns the representations of numerical and categorical conditions. The denoising model, consisting of a structure encoder-decoder, is trained for denoising under the representation of conditions. The diffusion process becomes graph-dependent to accurately estimate graph-related noise in molecules, unlike the previous models that focus solely on the marginal distributions of atoms or bonds. We extensively validate our model for multi-conditional polymer and small molecule generation. Results demonstrate our superiority across metrics from distribution learning to condition control for molecular properties. An inverse polymer design task for gas separation with feedback from domain experts further demonstrates its practical utility.

Title: Appearance Debiased Gaze Estimation via Stochastic Subject-Wise Adversarial Learning

• Authors: Suneung Kim, Woo-Jeoung Nam, Seong-Whan Lee
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.13865
• Pdf URL: https://arxiv.org/pdf/2401.13865
• Copy Paste: [[2401.13865]] Appearance Debiased Gaze Estimation via Stochastic Subject-Wise Adversarial Learning(https://arxiv.org/abs/2401.13865)
• Keywords: robust, generative
• Abstract: Recently, appearance-based gaze estimation has been attracting attention in computer vision, and remarkable improvements have been achieved using various deep learning techniques. Despite such progress, most methods aim to infer gaze vectors from images directly, which causes overfitting to person-specific appearance factors. In this paper, we address these challenges and propose a novel framework: Stochastic subject-wise Adversarial gaZE learning (SAZE), which trains a network to generalize the appearance of subjects. We design a Face generalization Network (Fgen-Net) using a face-to-gaze encoder and face identity classifier and a proposed adversarial loss. The proposed loss generalizes face appearance factors so that the identity classifier inferences a uniform probability distribution. In addition, the Fgen-Net is trained by a learning mechanism that optimizes the network by reselecting a subset of subjects at every training step to avoid overfitting. Our experimental results verify the robustness of the method in that it yields state-of-the-art performance, achieving 3.89 and 4.42 on the MPIIGaze and EyeDiap datasets, respectively. Furthermore, we demonstrate the positive generalization effect by conducting further experiments using face images involving different styles generated from the generative model.

Title: Unmasking and Quantifying Racial Bias of Large Language Models in Medical Report Generation

• Authors: Yifan Yang, Xiaoyu Liu, Qiao Jin, Furong Huang, Zhiyong Lu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.13867
• Pdf URL: https://arxiv.org/pdf/2401.13867
• Copy Paste: [[2401.13867]] Unmasking and Quantifying Racial Bias of Large Language Models in Medical Report Generation(https://arxiv.org/abs/2401.13867)
• Keywords: fair, large language model
• Abstract: Large language models like GPT-3.5-turbo and GPT-4 hold promise for healthcare professionals, but they may inadvertently inherit biases during their training, potentially affecting their utility in medical applications. Despite few attempts in the past, the precise impact and extent of these biases remain uncertain. Through both qualitative and quantitative analyses, we find that these models tend to project higher costs and longer hospitalizations for White populations and exhibit optimistic views in challenging medical scenarios with much higher survival rates. These biases, which mirror real-world healthcare disparities, are evident in the generation of patient backgrounds, the association of specific diseases with certain races, and disparities in treatment recommendations, etc. Our findings underscore the critical need for future research to address and mitigate biases in language models, especially in critical healthcare applications, to ensure fair and accurate outcomes for all patients.

Title: AscDAMs: Advanced SLAM-based channel detection and mapping system

• Authors: Tengfei Wang, Fucheng Lu, Jintao Qin, Taosheng Huang, Hui Kong, Ping Shen
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2401.13877
• Pdf URL: https://arxiv.org/pdf/2401.13877
• Copy Paste: [[2401.13877]] AscDAMs: Advanced SLAM-based channel detection and mapping system(https://arxiv.org/abs/2401.13877)
• Keywords: extraction
• Abstract: Obtaining high-resolution, accurate channel topography and deposit conditions is the prior challenge for the study of channelized debris flow. Currently, wide-used mapping technologies including satellite imaging and drone photogrammetry struggle to precisely observe channel interior conditions of mountainous long-deep gullies, particularly those in the Wenchuan Earthquake region. SLAM is an emerging tech for 3D mapping; however, extremely rugged environment in long-deep gullies poses two major challenges even for the state-of-art SLAM: (1) Atypical features; (2) Violent swaying and oscillation of sensors. These issues result in large deviation and lots of noise for SLAM results. To improve SLAM mapping in such environments, we propose an advanced SLAM-based channel detection and mapping system, namely AscDAMs. It features three main enhancements to post-process SLAM results: (1) The digital orthophoto map aided deviation correction algorithm greatly eliminates the systematic error; (2) The point cloud smoothing algorithm substantially diminishes noises; (3) The cross section extraction algorithm enables the quantitative assessment of channel deposits and their changes. Two field experiments were conducted in Chutou Gully, Wenchuan County in China in February and November 2023, representing observations before and after the rainy season. We demonstrate the capability of AscDAMs to greatly improve SLAM results, promoting SLAM for mapping the specially challenging environment. The proposed method compensates for the insufficiencies of existing technologies in detecting debris flow channel interiors including detailed channel morphology, erosion patterns, deposit distinction, volume estimation and change detection. It serves to enhance the study of full-scale debris flow mechanisms, long-term post-seismic evolution, and hazard assessment.

Title: A comparative study of zero-shot inference with large language models and supervised modeling in breast cancer pathology classification

• Authors: Madhumita Sushil, Travis Zack, Divneet Mandair, Zhiwei Zheng, Ahmed Wali, Yan-Ning Yu, Yuwei Quan, Atul J. Butte
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.13887
• Pdf URL: https://arxiv.org/pdf/2401.13887
• Copy Paste: [[2401.13887]] A comparative study of zero-shot inference with large language models and supervised modeling in breast cancer pathology classification(https://arxiv.org/abs/2401.13887)
• Keywords: extraction, large language model
• Abstract: Although supervised machine learning is popular for information extraction from clinical notes, creating large annotated datasets requires extensive domain expertise and is time-consuming. Meanwhile, large language models (LLMs) have demonstrated promising transfer learning capability. In this study, we explored whether recent LLMs can reduce the need for large-scale data annotations. We curated a manually-labeled dataset of 769 breast cancer pathology reports, labeled with 13 categories, to compare zero-shot classification capability of the GPT-4 model and the GPT-3.5 model with supervised classification performance of three model architectures: random forests classifier, long short-term memory networks with attention (LSTM-Att), and the UCSF-BERT model. Across all 13 tasks, the GPT-4 model performed either significantly better than or as well as the best supervised model, the LSTM-Att model (average macro F1 score of 0.83 vs. 0.75). On tasks with high imbalance between labels, the differences were more prominent. Frequent sources of GPT-4 errors included inferences from multiple samples and complex task design. On complex tasks where large annotated datasets cannot be easily collected, LLMs can reduce the burden of large-scale data labeling. However, if the use of LLMs is prohibitive, the use of simpler supervised models with large annotated datasets can provide comparable results. LLMs demonstrated the potential to speed up the execution of clinical NLP studies by reducing the need for curating large annotated datasets. This may result in an increase in the utilization of NLP-based variables and outcomes in observational clinical studies.

Title: Cross-Modal Prototype based Multimodal Federated Learning under Severely Missing Modality

• Authors: Huy Q. Le, Chu Myaet Thwal, Yu Qiao, Ye Lin Tun, Minh N. H. Nguyen, Choong Seon Hong
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.13898
• Pdf URL: https://arxiv.org/pdf/2401.13898
• Copy Paste: [[2401.13898]] Cross-Modal Prototype based Multimodal Federated Learning under Severely Missing Modality(https://arxiv.org/abs/2401.13898)
• Keywords: robust, federate
• Abstract: Multimodal federated learning (MFL) has emerged as a decentralized machine learning paradigm, allowing multiple clients with different modalities to collaborate on training a machine learning model across diverse data sources without sharing their private data. However, challenges, such as data heterogeneity and severely missing modalities, pose crucial hindrances to the robustness of MFL, significantly impacting the performance of global model. The absence of a modality introduces misalignment during the local training phase, stemming from zero-filling in the case of clients with missing modalities. Consequently, achieving robust generalization in global model becomes imperative, especially when dealing with clients that have incomplete data. In this paper, we propose Multimodal Federated Cross Prototype Learning (MFCPL), a novel approach for MFL under severely missing modalities by conducting the complete prototypes to provide diverse modality knowledge in modality-shared level with the cross-modal regularization and modality-specific level with cross-modal contrastive mechanism. Additionally, our approach introduces the cross-modal alignment to provide regularization for modality-specific features, thereby enhancing overall performance, particularly in scenarios involving severely missing modalities. Through extensive experiments on three multimodal datasets, we demonstrate the effectiveness of MFCPL in mitigating these challenges and improving the overall performance.

Title: Empowering Machines to Think Like Chemists: Unveiling Molecular Structure-Polarity Relationships with Hierarchical Symbolic Regression

• Authors: Siyu Lou, Chengchun Liu, Yuntian Chen, Fanyang Mo
• Subjects: cs.LG, cs.AI, cs.DB, stat.AP
• Abstract URL: https://arxiv.org/abs/2401.13904
• Pdf URL: https://arxiv.org/pdf/2401.13904
• Copy Paste: [[2401.13904]] Empowering Machines to Think Like Chemists: Unveiling Molecular Structure-Polarity Relationships with Hierarchical Symbolic Regression(https://arxiv.org/abs/2401.13904)
• Keywords: interpretability
• Abstract: Thin-layer chromatography (TLC) is a crucial technique in molecular polarity analysis. Despite its importance, the interpretability of predictive models for TLC, especially those driven by artificial intelligence, remains a challenge. Current approaches, utilizing either high-dimensional molecular fingerprints or domain-knowledge-driven feature engineering, often face a dilemma between expressiveness and interpretability. To bridge this gap, we introduce Unsupervised Hierarchical Symbolic Regression (UHiSR), combining hierarchical neural networks and symbolic regression. UHiSR automatically distills chemical-intuitive polarity indices, and discovers interpretable equations that link molecular structure to chromatographic behavior.

Title: A Survey of Deep Learning and Foundation Models for Time Series Forecasting

• Authors: John A. Miller, Mohammed Aldosari, Farah Saeed, Nasid Habib Barna, Subas Rana, I. Budak Arpinar, Ninghao Liu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.13912
• Pdf URL: https://arxiv.org/pdf/2401.13912
• Copy Paste: [[2401.13912]] A Survey of Deep Learning and Foundation Models for Time Series Forecasting(https://arxiv.org/abs/2401.13912)
• Keywords: interpretability, transformer, large language model
• Abstract: Deep Learning has been successfully applied to many application domains, yet its advantages have been slow to emerge for time series forecasting. For example, in the well-known Makridakis (M) Competitions, hybrids of traditional statistical or machine learning techniques have only recently become the top performers. With the recent architectural advances in deep learning being applied to time series forecasting (e.g., encoder-decoders with attention, transformers, and graph neural networks), deep learning has begun to show significant advantages. Still, in the area of pandemic prediction, there remain challenges for deep learning models: the time series is not long enough for effective training, unawareness of accumulated scientific knowledge, and interpretability of the model. To this end, the development of foundation models (large deep learning models with extensive pre-training) allows models to understand patterns and acquire knowledge that can be applied to new related problems before extensive training data becomes available. Furthermore, there is a vast amount of knowledge available that deep learning models can tap into, including Knowledge Graphs and Large Language Models fine-tuned with scientific domain knowledge. There is ongoing research examining how to utilize or inject such knowledge into deep learning models. In this survey, several state-of-the-art modeling techniques are reviewed, and suggestions for further work are provided.

Title: WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models

• Authors: Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, Dong Yu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.13919
• Pdf URL: https://arxiv.org/pdf/2401.13919
• Copy Paste: [[2401.13919]] WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models(https://arxiv.org/abs/2401.13919)
• Keywords: robust, large language model
• Abstract: The advancement of large language models (LLMs) leads to a new era marked by the development of autonomous applications in the real world, which drives innovation in the creation of advanced web-based agents. Existing web agents typically only handle one input modality and are evaluated only in simplified web simulators or static web snapshots, greatly limiting their applicability in real-world scenarios. To bridge this gap, we introduce WebVoyager, an innovative Large Multimodal Model (LMM) powered web agent that can complete user instructions end-to-end by interacting with real-world websites. Moreover, we propose a new evaluation protocol for web agents to address the challenges of automatic evaluation of open-ended web agent tasks, leveraging the robust multimodal comprehension capabilities of GPT-4V. We create a new benchmark by gathering real-world tasks from 15 widely used websites to evaluate our agents. We show that WebVoyager achieves a 55.7% task success rate, significantly surpassing the performance of both GPT-4 (All Tools) and the WebVoyager (text-only) setups, underscoring the exceptional capability of WebVoyager in practical applications. We found that our proposed automatic evaluation achieves 85.3% agreement with human judgment, paving the way for further development of web agents in a real-world setting.

Title: LocMoE: A Low-overhead MoE for Large Language Model Training

• Authors: Jing Li, Zhijie Sun, Xuan He, Li Zeng, Yi Lin, Entong Li, Binfan Zheng, Rongqian Zhao, Xin Chen
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2401.13920
• Pdf URL: https://arxiv.org/pdf/2401.13920
• Copy Paste: [[2401.13920]] LocMoE: A Low-overhead MoE for Large Language Model Training(https://arxiv.org/abs/2401.13920)
• Keywords: large language model
• Abstract: The Mixtures-of-Experts (MoE) model is a widespread distributed and integrated learning method for large language models (LLM), which is favored due to its ability to sparsify and expand models efficiently. However, the performance of MoE is limited by load imbalance and high latency of All-To-All communication, along with relatively redundant computation owing to large expert capacity. Load imbalance may result from existing routing policies that consistently tend to select certain experts. The frequent inter-node communication in the All-To-All procedure also significantly prolongs the training time. To alleviate the above performance problems, we propose a novel routing strategy that combines load balance and locality by converting partial inter-node communication to that of intra-node. Notably, we elucidate that there is a minimum threshold for expert capacity, calculated through the maximal angular deviation between the gating weights of the experts and the assigned tokens. We port these modifications on the PanGu-Sigma model based on the MindSpore framework with multi-level routing and conduct experiments on Ascend clusters. The experiment results demonstrate that the proposed LocMoE reduces training time per epoch by 12.68% to 22.24% compared to classical routers, such as hash router and switch router, without impacting the model accuracy.

Title: Adaptive Text Watermark for Large Language Models

• Authors: Yepeng Liu, Yuheng Bu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.13927
• Pdf URL: https://arxiv.org/pdf/2401.13927
• Copy Paste: [[2401.13927]] Adaptive Text Watermark for Large Language Models(https://arxiv.org/abs/2401.13927)
• Keywords: security, attack, robust, watermark, large language model
• Abstract: The advancement of Large Language Models (LLMs) has led to increasing concerns about the misuse of AI-generated text, and watermarking for LLM-generated text has emerged as a potential solution. However, it is challenging to generate high-quality watermarked text while maintaining strong security, robustness, and the ability to detect watermarks without prior knowledge of the prompt or model. This paper proposes an adaptive watermarking strategy to address this problem. To improve the text quality and maintain robustness, we adaptively add watermarking to token distributions with high entropy measured using an auxiliary model and keep the low entropy token distributions untouched. For the sake of security and to further minimize the watermark's impact on text quality, instead of using a fixed green/red list generated from a random secret key, which can be vulnerable to decryption and forgery, we adaptively scale up the output logits in proportion based on the semantic embedding of previously generated text using a well designed semantic mapping model. Our experiments involving various LLMs demonstrate that our approach achieves comparable robustness performance to existing watermark methods. Additionally, the text generated by our method has perplexity comparable to that of \emph{un-watermarked} LLMs while maintaining security even under various attacks.

Title: Self-supervised Video Object Segmentation with Distillation Learning of Deformable Attention

• Authors: Quang-Trung Truong, Duc Thanh Nguyen, Binh-Son Hua, Sai-Kit Yeung
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.13937
• Pdf URL: https://arxiv.org/pdf/2401.13937
• Copy Paste: [[2401.13937]] Self-supervised Video Object Segmentation with Distillation Learning of Deformable Attention(https://arxiv.org/abs/2401.13937)
• Keywords: segmentation
• Abstract: Video object segmentation is a fundamental research problem in computer vision. Recent techniques have often applied attention mechanism to object representation learning from video sequences. However, due to temporal changes in the video data, attention maps may not well align with the objects of interest across video frames, causing accumulated errors in long-term video processing. In addition, existing techniques have utilised complex architectures, requiring highly computational complexity and hence limiting the ability to integrate video object segmentation into low-powered devices. To address these issues, we propose a new method for self-supervised video object segmentation based on distillation learning of deformable attention. Specifically, we devise a lightweight architecture for video object segmentation that is effectively adapted to temporal changes. This is enabled by deformable attention mechanism, where the keys and values capturing the memory of a video sequence in the attention module have flexible locations updated across frames. The learnt object representations are thus adaptive to both the spatial and temporal dimensions. We train the proposed architecture in a self-supervised fashion through a new knowledge distillation paradigm where deformable attention maps are integrated into the distillation loss. We qualitatively and quantitatively evaluate our method and compare it with existing methods on benchmark datasets including DAVIS 2016/2017 and YouTube-VOS 2018/2019. Experimental results verify the superiority of our method via its achieved state-of-the-art performance and optimal memory usage.

Title: StyleInject: Parameter Efficient Tuning of Text-to-Image Diffusion Models

• Authors: Yalong Bai, Mohan Zhou, Qing Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.13942
• Pdf URL: https://arxiv.org/pdf/2401.13942
• Copy Paste: [[2401.13942]] StyleInject: Parameter Efficient Tuning of Text-to-Image Diffusion Models(https://arxiv.org/abs/2401.13942)
• Keywords: diffusion, generative
• Abstract: The ability to fine-tune generative models for text-to-image generation tasks is crucial, particularly facing the complexity involved in accurately interpreting and visualizing textual inputs. While LoRA is efficient for language model adaptation, it often falls short in text-to-image tasks due to the intricate demands of image generation, such as accommodating a broad spectrum of styles and nuances. To bridge this gap, we introduce StyleInject, a specialized fine-tuning approach tailored for text-to-image models. StyleInject comprises multiple parallel low-rank parameter matrices, maintaining the diversity of visual features. It dynamically adapts to varying styles by adjusting the variance of visual features based on the characteristics of the input signal. This approach significantly minimizes the impact on the original model's text-image alignment capabilities while adeptly adapting to various styles in transfer learning. StyleInject proves particularly effective in learning from and enhancing a range of advanced, community-fine-tuned generative models. Our comprehensive experiments, including both small-sample and large-scale data fine-tuning as well as base model distillation, show that StyleInject surpasses traditional LoRA in both text-image semantic consistency and human preference evaluation, all while ensuring greater parameter efficiency.

Title: AM-SORT: Adaptable Motion Predictor with Historical Trajectory Embedding for Multi-Object Tracking

• Authors: Vitaliy Kim, Gunho Jung, Seong-Whan Lee
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.13950
• Pdf URL: https://arxiv.org/pdf/2401.13950
• Copy Paste: [[2401.13950]] AM-SORT: Adaptable Motion Predictor with Historical Trajectory Embedding for Multi-Object Tracking(https://arxiv.org/abs/2401.13950)
• Keywords: transformer
• Abstract: Many multi-object tracking (MOT) approaches, which employ the Kalman Filter as a motion predictor, assume constant velocity and Gaussian-distributed filtering noises. These assumptions render the Kalman Filter-based trackers effective in linear motion scenarios. However, these linear assumptions serve as a key limitation when estimating future object locations within scenarios involving non-linear motion and occlusions. To address this issue, we propose a motion-based MOT approach with an adaptable motion predictor, called AM-SORT, which adapts to estimate non-linear uncertainties. AM-SORT is a novel extension of the SORT-series trackers that supersedes the Kalman Filter with the transformer architecture as a motion predictor. We introduce a historical trajectory embedding that empowers the transformer to extract spatio-temporal features from a sequence of bounding boxes. AM-SORT achieves competitive performance compared to state-of-the-art trackers on DanceTrack, with 56.3 IDF1 and 55.6 HOTA. We conduct extensive experiments to demonstrate the effectiveness of our method in predicting non-linear movement under occlusions.

Title: Randomized Response with Gradual Release of Privacy Budget

• Authors: Mingen Pan
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2401.13952
• Pdf URL: https://arxiv.org/pdf/2401.13952
• Copy Paste: [[2401.13952]] Randomized Response with Gradual Release of Privacy Budget(https://arxiv.org/abs/2401.13952)
• Keywords: privacy
• Abstract: An algorithm is developed to gradually relax the Differential Privacy (DP) guarantee of a randomized response. The output from each relaxation maintains the same probability distribution as a standard randomized response with the equivalent DP guarantee, ensuring identical utility as the standard approach. The entire relaxation process is proven to have the same DP guarantee as the most recent relaxed guarantee. The DP relaxation algorithm is adaptable to any Local Differential Privacy (LDP) mechanisms relying on randomized response. It has been seamlessly integrated into RAPPOR, an LDP crowdsourcing string-collecting tool, to optimize the utility of estimating the frequency of collected data. Additionally, it facilitates the relaxation of the DP guarantee for mean estimation based on randomized response. Finally, numerical experiments have been conducted to validate the utility and DP guarantee of the algorithm.

Title: A New Image Quality Database for Multiple Industrial Processes

• Authors: Xuanchao Ma, Zehan Wu, Hongyan Liu, Chengxu Zhou, Ke Gu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.13956
• Pdf URL: https://arxiv.org/pdf/2401.13956
• Copy Paste: [[2401.13956]] A New Image Quality Database for Multiple Industrial Processes(https://arxiv.org/abs/2401.13956)
• Keywords: security
• Abstract: Recent years have witnessed a broader range of applications of image processing technologies in multiple industrial processes, such as smoke detection, security monitoring, and workpiece inspection. Different kinds of distortion types and levels must be introduced into an image during the processes of acquisition, compression, transmission, storage, and display, which might heavily degrade the image quality and thus strongly reduce the final display effect and clarity. To verify the reliability of existing image quality assessment methods, we establish a new industrial process image database (IPID), which contains 3000 distorted images generated by applying different levels of distortion types to each of the 50 source images. We conduct the subjective test on the aforementioned 3000 images to collect their subjective quality ratings in a well-suited laboratory environment. Finally, we perform comparison experiments on IPID database to investigate the performance of some objective image quality assessment algorithms. The experimental results show that the state-of-the-art image quality assessment methods have difficulty in predicting the quality of images that contain multiple distortion types.

Title: TriSAM: Tri-Plane SAM for zero-shot cortical blood vessel segmentation in VEM images

• Authors: Jia Wan, Wanhua Li, Atmadeep Banerjee, Jason Ken Adhinarta, Evelina Sjostedt, Jingpeng Wu, Jeff Lichtman, Hanspeter Pfister, Donglai Wei
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.13961
• Pdf URL: https://arxiv.org/pdf/2401.13961
• Copy Paste: [[2401.13961]] TriSAM: Tri-Plane SAM for zero-shot cortical blood vessel segmentation in VEM images(https://arxiv.org/abs/2401.13961)
• Keywords: segmentation
• Abstract: In this paper, we address a significant gap in the field of neuroimaging by introducing the largest-to-date public benchmark, BvEM, designed specifically for cortical blood vessel segmentation in Volume Electron Microscopy (VEM) images. The intricate relationship between cerebral blood vessels and neural function underscores the vital role of vascular analysis in understanding brain health. While imaging techniques at macro and mesoscales have garnered substantial attention and resources, the microscale VEM imaging, capable of revealing intricate vascular details, has lacked the necessary benchmarking infrastructure. As researchers delve deeper into the microscale intricacies of cerebral vasculature, our BvEM benchmark represents a critical step toward unraveling the mysteries of neurovascular coupling and its impact on brain function and pathology. The BvEM dataset is based on VEM image volumes from three mammal species: adult mouse, macaque, and human. We standardized the resolution, addressed imaging variations, and meticulously annotated blood vessels through semi-automatic, manual, and quality control processes, ensuring high-quality 3D segmentation. Furthermore, we developed a zero-shot cortical blood vessel segmentation method named TriSAM, which leverages the powerful segmentation model SAM for 3D segmentation. To lift SAM from 2D segmentation to 3D volume segmentation, TriSAM employs a multi-seed tracking framework, leveraging the reliability of certain image planes for tracking while using others to identify potential turning points. This approach, consisting of Tri-Plane selection, SAM-based tracking, and recursive redirection, effectively achieves long-term 3D blood vessel segmentation without model training or fine-tuning. Experimental results show that TriSAM achieved superior performances on the BvEM benchmark across three species.

Title: An Extensible Framework for Open Heterogeneous Collaborative Perception

• Authors: Yifan Lu, Yue Hu, Yiqi Zhong, Dequan Wang, Siheng Chen, Yanfeng Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.13964
• Pdf URL: https://arxiv.org/pdf/2401.13964
• Copy Paste: [[2401.13964]] An Extensible Framework for Open Heterogeneous Collaborative Perception(https://arxiv.org/abs/2401.13964)
• Keywords: protect
• Abstract: Collaborative perception aims to mitigate the limitations of single-agent perception, such as occlusions, by facilitating data exchange among multiple agents. However, most current works consider a homogeneous scenario where all agents use identity sensors and perception models. In reality, heterogeneous agent types may continually emerge and inevitably face a domain gap when collaborating with existing agents. In this paper, we introduce a new open heterogeneous problem: how to accommodate continually emerging new heterogeneous agent types into collaborative perception, while ensuring high perception performance and low integration cost? To address this problem, we propose HEterogeneous ALliance (HEAL), a novel extensible collaborative perception framework. HEAL first establishes a unified feature space with initial agents via a novel multi-scale foreground-aware Pyramid Fusion network. When heterogeneous new agents emerge with previously unseen modalities or models, we align them to the established unified space with an innovative backward alignment. This step only involves individual training on the new agent type, thus presenting extremely low training costs and high extensibility. It also protects new agents' model details from disclosure since the training can be conducted by the agent owner locally. To enrich agents' data heterogeneity, we bring OPV2V-H, a new large-scale dataset with more diverse sensor types. Extensive experiments on OPV2V-H and DAIR-V2X datasets show that HEAL surpasses SOTA methods in performance while reducing the training parameters by 91.5% when integrating 3 new agent types. Code and data are available at: https://github.com/yifanlu0227/HEAL.

Title: Improving Pseudo-labelling and Enhancing Robustness for Semi-Supervised Domain Generalization

• Authors: Adnan Khan, Mai A. Shaaban, Muhammad Haris Khan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.13965
• Pdf URL: https://arxiv.org/pdf/2401.13965
• Copy Paste: [[2401.13965]] Improving Pseudo-labelling and Enhancing Robustness for Semi-Supervised Domain Generalization(https://arxiv.org/abs/2401.13965)
• Keywords: robust
• Abstract: Beyond attaining domain generalization (DG), visual recognition models should also be data-efficient during learning by leveraging limited labels. We study the problem of Semi-Supervised Domain Generalization (SSDG) which is crucial for real-world applications like automated healthcare. SSDG requires learning a cross-domain generalizable model when the given training data is only partially labelled. Empirical investigations reveal that the DG methods tend to underperform in SSDG settings, likely because they are unable to exploit the unlabelled data. Semi-supervised learning (SSL) shows improved but still inferior results compared to fully-supervised learning. A key challenge, faced by the best-performing SSL-based SSDG methods, is selecting accurate pseudo-labels under multiple domain shifts and reducing overfitting to source domains under limited labels. In this work, we propose new SSDG approach, which utilizes a novel uncertainty-guided pseudo-labelling with model averaging (UPLM). Our uncertainty-guided pseudo-labelling (UPL) uses model uncertainty to improve pseudo-labelling selection, addressing poor model calibration under multi-source unlabelled data. The UPL technique, enhanced by our novel model averaging (MA) strategy, mitigates overfitting to source domains with limited labels. Extensive experiments on key representative DG datasets suggest that our method demonstrates effectiveness against existing methods. Our code and chosen labelled data seeds are available on GitHub: https://github.com/Adnan-Khan7/UPLM

Title: Dynamic Long-Term Time-Series Forecasting via Meta Transformer Networks

• Authors: Muhammad Anwar Ma'sum, MD Rasel Sarkar, Mahardhika Pratama, Savitha Ramasamy, Sreenatha Anavatti, Lin Liu, Habibullah, Ryszard Kowalczyk
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.13968
• Pdf URL: https://arxiv.org/pdf/2401.13968
• Copy Paste: [[2401.13968]] Dynamic Long-Term Time-Series Forecasting via Meta Transformer Networks(https://arxiv.org/abs/2401.13968)
• Keywords: robust, transformer
• Abstract: A reliable long-term time-series forecaster is highly demanded in practice but comes across many challenges such as low computational and memory footprints as well as robustness against dynamic learning environments. This paper proposes Meta-Transformer Networks (MANTRA) to deal with the dynamic long-term time-series forecasting tasks. MANTRA relies on the concept of fast and slow learners where a collection of fast learners learns different aspects of data distributions while adapting quickly to changes. A slow learner tailors suitable representations to fast learners. Fast adaptations to dynamic environments are achieved using the universal representation transformer layers producing task-adapted representations with a small number of parameters. Our experiments using four datasets with different prediction lengths demonstrate the advantage of our approach with at least $3\%$ improvements over the baseline algorithms for both multivariate and univariate settings. Source codes of MANTRA are publicly available in \url{https://github.com/anwarmaxsum/MANTRA}.

Title: BootPIG: Bootstrapping Zero-shot Personalized Image Generation Capabilities in Pretrained Diffusion Models

• Authors: Senthil Purushwalkam, Akash Gokul, Shafiq Joty, Nikhil Naik
• Subjects: cs.CV, cs.AI, cs.GR
• Abstract URL: https://arxiv.org/abs/2401.13974
• Pdf URL: https://arxiv.org/pdf/2401.13974
• Copy Paste: [[2401.13974]] BootPIG: Bootstrapping Zero-shot Personalized Image Generation Capabilities in Pretrained Diffusion Models(https://arxiv.org/abs/2401.13974)
• Keywords: diffusion, segmentation
• Abstract: Recent text-to-image generation models have demonstrated incredible success in generating images that faithfully follow input prompts. However, the requirement of using words to describe a desired concept provides limited control over the appearance of the generated concepts. In this work, we address this shortcoming by proposing an approach to enable personalization capabilities in existing text-to-image diffusion models. We propose a novel architecture (BootPIG) that allows a user to provide reference images of an object in order to guide the appearance of a concept in the generated images. The proposed BootPIG architecture makes minimal modifications to a pretrained text-to-image diffusion model and utilizes a separate UNet model to steer the generations toward the desired appearance. We introduce a training procedure that allows us to bootstrap personalization capabilities in the BootPIG architecture using data generated from pretrained text-to-image models, LLM chat agents, and image segmentation models. In contrast to existing methods that require several days of pretraining, the BootPIG architecture can be trained in approximately 1 hour. Experiments on the DreamBooth dataset demonstrate that BootPIG outperforms existing zero-shot methods while being comparable with test-time finetuning approaches. Through a user study, we validate the preference for BootPIG generations over existing methods both in maintaining fidelity to the reference object's appearance and aligning with textual prompts.

Title: Evaluating the Determinants of Mode Choice Using Statistical and Machine Learning Techniques in the Indian Megacity of Bengaluru

• Authors: Tanmay Ghosh, Nithin Nagaraj
• Subjects: cs.LG, econ.GN
• Abstract URL: https://arxiv.org/abs/2401.13977
• Pdf URL: https://arxiv.org/pdf/2401.13977
• Copy Paste: [[2401.13977]] Evaluating the Determinants of Mode Choice Using Statistical and Machine Learning Techniques in the Indian Megacity of Bengaluru(https://arxiv.org/abs/2401.13977)
• Keywords: interpretability
• Abstract: The decision making involved behind the mode choice is critical for transportation planning. While statistical learning techniques like discrete choice models have been used traditionally, machine learning (ML) models have gained traction recently among the transportation planners due to their higher predictive performance. However, the black box nature of ML models pose significant interpretability challenges, limiting their practical application in decision and policy making. This study utilised a dataset of $1350$ households belonging to low and low-middle income bracket in the city of Bengaluru to investigate mode choice decision making behaviour using Multinomial logit model and ML classifiers like decision trees, random forests, extreme gradient boosting and support vector machines. In terms of accuracy, random forest model performed the best ($0.788$ on training data and $0.605$ on testing data) compared to all the other models. This research has adopted modern interpretability techniques like feature importance and individual conditional expectation plots to explain the decision making behaviour using ML models. A higher travel costs significantly reduce the predicted probability of bus usage compared to other modes (a $0.66\%$ and $0.34\%$ reduction using Random Forests and XGBoost model for $10\%$ increase in travel cost). However, reducing travel time by $10\%$ increases the preference for the metro ($0.16\%$ in Random Forests and 0.42% in XGBoost). This research augments the ongoing research on mode choice analysis using machine learning techniques, which would help in improving the understanding of the performance of these models with real-world data in terms of both accuracy and interpretability.

Title: Towards Consistent Natural-Language Explanations via Explanation-Consistency Finetuning

• Authors: Yanda Chen, Chandan Singh, Xiaodong Liu, Simiao Zuo, Bin Yu, He He, Jianfeng Gao
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.13986
• Pdf URL: https://arxiv.org/pdf/2401.13986
• Copy Paste: [[2401.13986]] Towards Consistent Natural-Language Explanations via Explanation-Consistency Finetuning(https://arxiv.org/abs/2401.13986)
• Keywords: large language model
• Abstract: Large language models (LLMs) often generate convincing, fluent explanations. However, different from humans, they often generate inconsistent explanations on different inputs. For example, an LLM may generate the explanation "all birds can fly" when answering the question "Can sparrows fly?" but meanwhile answer "no" to the related question "Can penguins fly?". Explanations should be consistent across related examples so that they allow a human to simulate the LLM's decision process on multiple examples. We propose explanation-consistency finetuning (EC-finetuning), a method that adapts LLMs to generate more consistent natural-language explanations on related examples. EC-finetuning involves finetuning LLMs on synthetic data that is carefully constructed to contain consistent explanations. Across a variety of question-answering datasets in various domains, EC-finetuning yields a 10.0% relative explanation consistency improvement on four finetuning datasets, and generalizes to seven out-of-distribution datasets not seen during finetuning (+4.5% relative). Code is available at https://github.com/yandachen/explanation-consistency-finetuning .

Title: Cross-Domain Few-Shot Learning via Adaptive Transformer Networks

• Authors: Naeem Paeedeh, Mahardhika Pratama, Muhammad Anwar Ma'sum, Wolfgang Mayer, Zehong Cao, Ryszard Kowlczyk
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.13987
• Pdf URL: https://arxiv.org/pdf/2401.13987
• Copy Paste: [[2401.13987]] Cross-Domain Few-Shot Learning via Adaptive Transformer Networks(https://arxiv.org/abs/2401.13987)
• Keywords: transformer
• Abstract: Most few-shot learning works rely on the same domain assumption between the base and the target tasks, hindering their practical applications. This paper proposes an adaptive transformer network (ADAPTER), a simple but effective solution for cross-domain few-shot learning where there exist large domain shifts between the base task and the target task. ADAPTER is built upon the idea of bidirectional cross-attention to learn transferable features between the two domains. The proposed architecture is trained with DINO to produce diverse, and less biased features to avoid the supervision collapse problem. Furthermore, the label smoothing approach is proposed to improve the consistency and reliability of the predictions by also considering the predicted labels of the close samples in the embedding space. The performance of ADAPTER is rigorously evaluated in the BSCD-FSL benchmarks in which it outperforms prior arts with significant margins.

Title: Diffusion-based Data Augmentation for Object Counting Problems

• Authors: Zhen Wang, Yuelei Li, Jia Wan, Nuno Vasconcelos
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.13992
• Pdf URL: https://arxiv.org/pdf/2401.13992
• Copy Paste: [[2401.13992]] Diffusion-based Data Augmentation for Object Counting Problems(https://arxiv.org/abs/2401.13992)
• Keywords: diffusion, transformer
• Abstract: Crowd counting is an important problem in computer vision due to its wide range of applications in image understanding. Currently, this problem is typically addressed using deep learning approaches, such as Convolutional Neural Networks (CNNs) and Transformers. However, deep networks are data-driven and are prone to overfitting, especially when the available labeled crowd dataset is limited. To overcome this limitation, we have designed a pipeline that utilizes a diffusion model to generate extensive training data. We are the first to generate images conditioned on a location dot map (a binary dot map that specifies the location of human heads) with a diffusion model. We are also the first to use these diverse synthetic data to augment the crowd counting models. Our proposed smoothed density map input for ControlNet significantly improves ControlNet's performance in generating crowds in the correct locations. Also, Our proposed counting loss for the diffusion model effectively minimizes the discrepancies between the location dot map and the crowd images generated. Additionally, our innovative guidance sampling further directs the diffusion process toward regions where the generated crowd images align most accurately with the location dot map. Collectively, we have enhanced ControlNet's ability to generate specified objects from a location dot map, which can be used for data augmentation in various counting problems. Moreover, our framework is versatile and can be easily adapted to all kinds of counting problems. Extensive experiments demonstrate that our framework improves the counting performance on the ShanghaiTech, NWPU-Crowd, UCF-QNRF, and TRANCOS datasets, showcasing its effectiveness.

Title: Investigate-Consolidate-Exploit: A General Strategy for Inter-Task Agent Self-Evolution

• Authors: Cheng Qian, Shihao Liang, Yujia Qin, Yining Ye, Xin Cong, Yankai Lin, Yesai Wu, Zhiyuan Liu, Maosong Sun
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.13996
• Pdf URL: https://arxiv.org/pdf/2401.13996
• Copy Paste: [[2401.13996]] Investigate-Consolidate-Exploit: A General Strategy for Inter-Task Agent Self-Evolution(https://arxiv.org/abs/2401.13996)
• Keywords: robust
• Abstract: This paper introduces Investigate-Consolidate-Exploit (ICE), a novel strategy for enhancing the adaptability and flexibility of AI agents through inter-task self-evolution. Unlike existing methods focused on intra-task learning, ICE promotes the transfer of knowledge between tasks for genuine self-evolution, similar to human experience learning. The strategy dynamically investigates planning and execution trajectories, consolidates them into simplified workflows and pipelines, and exploits them for improved task execution. Our experiments on the XAgent framework demonstrate ICE's effectiveness, reducing API calls by as much as 80% and significantly decreasing the demand for the model's capability. Specifically, when combined with GPT-3.5, ICE's performance matches that of raw GPT-4 across various agent tasks. We argue that this self-evolution approach represents a paradigm shift in agent design, contributing to a more robust AI community and ecosystem, and moving a step closer to full autonomy.

Title: ConstraintChecker: A Plugin for Large Language Models to Reason on Commonsense Knowledge Bases

• Authors: Quyet V. Do, Tianqing Fang, Shizhe Diao, Zhaowei Wang, Yangqiu Song
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.14003
• Pdf URL: https://arxiv.org/pdf/2401.14003
• Copy Paste: [[2401.14003]] ConstraintChecker: A Plugin for Large Language Models to Reason on Commonsense Knowledge Bases(https://arxiv.org/abs/2401.14003)
• Keywords: large language model
• Abstract: Reasoning over Commonsense Knowledge Bases (CSKB), i.e. CSKB reasoning, has been explored as a way to acquire new commonsense knowledge based on reference knowledge in the original CSKBs and external prior knowledge. Despite the advancement of Large Language Models (LLM) and prompt engineering techniques in various reasoning tasks, they still struggle to deal with CSKB reasoning. One of the problems is that it is hard for them to acquire explicit relational constraints in CSKBs from only in-context exemplars, due to a lack of symbolic reasoning capabilities (Bengio et al., 2021). To this end, we proposed **ConstraintChecker**, a plugin over prompting techniques to provide and check explicit constraints. When considering a new knowledge instance, ConstraintChecker employs a rule-based module to produce a list of constraints, then it uses a zero-shot learning module to check whether this knowledge instance satisfies all constraints. The acquired constraint-checking result is then aggregated with the output of the main prompting technique to produce the final output. Experimental results on CSKB Reasoning benchmarks demonstrate the effectiveness of our method by bringing consistent improvements over all prompting methods. Codes and data are available at \url{https://github.com/HKUST-KnowComp/ConstraintChecker}.

Title: Cyber-Twin: Digital Twin-boosted Autonomous Attack Detection for Vehicular Ad-Hoc Networks

• Authors: Yagmur Yigit, Ioannis Panitsas, Leandros Maglaras, Leandros Tassiulas, Berk Canberk
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2401.14005
• Pdf URL: https://arxiv.org/pdf/2401.14005
• Copy Paste: [[2401.14005]] Cyber-Twin: Digital Twin-boosted Autonomous Attack Detection for Vehicular Ad-Hoc Networks(https://arxiv.org/abs/2401.14005)
• Keywords: secure, security, attack
• Abstract: The rapid evolution of Vehicular Ad-hoc NETworks (VANETs) has ushered in a transformative era for intelligent transportation systems (ITS), significantly enhancing road safety and vehicular communication. However, the intricate and dynamic nature of VANETs presents formidable challenges, particularly in vehicle-to-infrastructure (V2I) communications. Roadside Units (RSUs), integral components of VANETs, are increasingly susceptible to cyberattacks, such as jamming and distributed denial-of-service (DDoS) attacks. These vulnerabilities pose grave risks to road safety, potentially leading to traffic congestion and vehicle malfunctions. Current approaches often struggle to effectively merge digital twin technology with Artificial Intelligence (AI) models to boost security and sustainability. Our study introduces an innovative cyber-twin framework tailored to enhance the security of RSUs in VANETs. This framework uniquely combines digital twin technology with cutting-edge AI to offer a real-time, dynamic representation of RSUs. This allows for detailed monitoring and efficient detection of threats, significantly strengthening RSU security in VANETs. Moreover, our framework makes a notable contribution to eco-friendly communication by improving the computational efficiency of RSUs, leading to increased energy efficiency and extended hardware durability. Our results show a considerable enhancement in resource management and attack detection, surpassing the performance of existing solutions. In particular, the cyber-twin framework showed a substantial reduction in RSU load and an optimal balance between resource consumption and high attack detection efficiency, with a defined twinning rate range of seventy-six to ninety per cent. These advancements underscore our commitment to developing sustainable, secure, and resilient vehicular communication systems for the future of smart cities.

Title: CMMU: A Benchmark for Chinese Multi-modal Multi-type Question Understanding and Reasoning

• Authors: Zheqi He, Xinya Wu, Pengfei Zhou, Richeng Xuan, Guang Liu, Xi Yang, Qiannan Zhu, Hua Huang
• Subjects: cs.CL, cs.AI, cs.MM
• Abstract URL: https://arxiv.org/abs/2401.14011
• Pdf URL: https://arxiv.org/pdf/2401.14011
• Copy Paste: [[2401.14011]] CMMU: A Benchmark for Chinese Multi-modal Multi-type Question Understanding and Reasoning(https://arxiv.org/abs/2401.14011)
• Keywords: large language model
• Abstract: Multi-modal large language models(MLLMs) have achieved remarkable progress and demonstrated powerful knowledge comprehension and reasoning abilities. However, the mastery of domain-specific knowledge, which is essential for evaluating the intelligence of MLLMs, continues to be a challenge. Current multi-modal benchmarks for domain-specific knowledge concentrate on multiple-choice questions and are predominantly available in English, which imposes limitations on the comprehensiveness of the evaluation. To this end, we introduce CMMU, a novel benchmark for multi-modal and multi-type question understanding and reasoning in Chinese. CMMU consists of 3,603 questions in 7 subjects, covering knowledge from primary to high school. The questions can be categorized into 3 types: multiple-choice, multiple-response, and fill-in-the-blank, bringing greater challenges to MLLMs. In addition, we propose a rigorous evaluation strategy called ShiftCheck for assessing multiple-choice questions. The strategy aims to reduce position bias, minimize the influence of randomness on correctness, and perform a quantitative analysis of position bias. We evaluate seven open-source MLLMs along with GPT4-V, Gemini-Pro, and Qwen-VL-Plus. The results demonstrate that CMMU poses a significant challenge to the recent MLLMs.

Title: Towards Uncertainty-Aware Language Agent

• Authors: Jiuzhou Han, Wray Buntine, Ehsan Shareghi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.14016
• Pdf URL: https://arxiv.org/pdf/2401.14016
• Copy Paste: [[2401.14016]] Towards Uncertainty-Aware Language Agent(https://arxiv.org/abs/2401.14016)
• Keywords: large language model
• Abstract: While Language Agents have achieved promising success by placing Large Language Models at the core of a more versatile design that dynamically interacts with the external world, the existing approaches neglect the notion of uncertainty during these interactions. We present the Uncertainty-Aware Language Agent (UALA), a framework that orchestrates the interaction between the agent and the external world using uncertainty quantification. Compared with other well-known counterparts like ReAct, our extensive experiments across 3 representative tasks (HotpotQA, StrategyQA, MMLU) and various LLM sizes demonstrates that UALA brings a significant improvement of performance, while having a substantially lower reliance on the external world (i.e., reduced number of tool calls and tokens). Our analyses provide various insights including the great potential of UALA compared with agent fine-tuning, and underscoring the unreliably of verbalised confidence of LLMs as a proxy for uncertainty.

Title: Unitxt: Flexible, Shareable and Reusable Data Preparation and Evaluation for Generative AI

• Authors: Elron Bandel, Yotam Perlitz, Elad Venezian, Roni Friedman-Melamed, Ofir Arviv, Matan Orbach, Shachar Don-Yehyia, Dafna Sheinwald, Ariel Gera, Leshem Choshen, Michal Shmueli-Scheuer, Yoav Katz
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.14019
• Pdf URL: https://arxiv.org/pdf/2401.14019
• Copy Paste: [[2401.14019]] Unitxt: Flexible, Shareable and Reusable Data Preparation and Evaluation for Generative AI(https://arxiv.org/abs/2401.14019)
• Keywords: generative
• Abstract: In the dynamic landscape of generative NLP, traditional text processing pipelines limit research flexibility and reproducibility, as they are tailored to specific dataset, task, and model combinations. The escalating complexity, involving system prompts, model-specific formats, instructions, and more, calls for a shift to a structured, modular, and customizable solution. Addressing this need, we present Unitxt, an innovative library for customizable textual data preparation and evaluation tailored to generative language models. Unitxt natively integrates with common libraries like HuggingFace and LM-eval-harness and deconstructs processing flows into modular components, enabling easy customization and sharing between practitioners. These components encompass model-specific formats, task prompts, and many other comprehensive dataset processing definitions. The Unitxt-Catalog centralizes these components, fostering collaboration and exploration in modern textual data workflows. Beyond being a tool, Unitxt is a community-driven platform, empowering users to build, share, and advance their pipelines collaboratively. Join the Unitxt community at https://github.com/IBM/unitxt!

Title: The Risk of Federated Learning to Skew Fine-Tuning Features and Underperform Out-of-Distribution Robustness

• Authors: Mengyao Du, Miao Zhang, Yuwen Pu, Kai Xu, Shouling Ji, Quanjun Yin
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.14027
• Pdf URL: https://arxiv.org/pdf/2401.14027
• Copy Paste: [[2401.14027]] The Risk of Federated Learning to Skew Fine-Tuning Features and Underperform Out-of-Distribution Robustness(https://arxiv.org/abs/2401.14027)
• Keywords: privacy, robust, federate
• Abstract: To tackle the scarcity and privacy issues associated with domain-specific datasets, the integration of federated learning in conjunction with fine-tuning has emerged as a practical solution. However, our findings reveal that federated learning has the risk of skewing fine-tuning features and compromising the out-of-distribution robustness of the model. By introducing three robustness indicators and conducting experiments across diverse robust datasets, we elucidate these phenomena by scrutinizing the diversity, transferability, and deviation within the model feature space. To mitigate the negative impact of federated learning on model robustness, we introduce GNP, a \underline{G}eneral \underline{N}oisy \underline{P}rojection-based robust algorithm, ensuring no deterioration of accuracy on the target distribution. Specifically, the key strategy for enhancing model robustness entails the transfer of robustness from the pre-trained model to the fine-tuned model, coupled with adding a small amount of Gaussian noise to augment the representative capacity of the model. Comprehensive experimental results demonstrate that our approach markedly enhances the robustness across diverse scenarios, encompassing various parameter-efficient fine-tuning methods and confronting different levels of data heterogeneity.

Title: Sparse and Transferable Universal Singular Vectors Attack

• Authors: Kseniia Kuvshinova, Olga Tsymboi, Ivan Oseledets
• Subjects: cs.LG, cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2401.14031
• Pdf URL: https://arxiv.org/pdf/2401.14031
• Copy Paste: [[2401.14031]] Sparse and Transferable Universal Singular Vectors Attack(https://arxiv.org/abs/2401.14031)
• Keywords: attack, robust, interpretability
• Abstract: The research in the field of adversarial attacks and models' vulnerability is one of the fundamental directions in modern machine learning. Recent studies reveal the vulnerability phenomenon, and understanding the mechanisms behind this is essential for improving neural network characteristics and interpretability. In this paper, we propose a novel sparse universal white-box adversarial attack. Our approach is based on truncated power iteration providing sparsity to $(p,q)$-singular vectors of the hidden layers of Jacobian matrices. Using the ImageNet benchmark validation subset, we analyze the proposed method in various settings, achieving results comparable to dense baselines with more than a 50% fooling rate while damaging only 5% of pixels and utilizing 256 samples for perturbation fitting. We also show that our algorithm admits higher attack magnitude without affecting the human ability to solve the task. Furthermore, we investigate that the constructed perturbations are highly transferable among different models without significantly decreasing the fooling rate. Our findings demonstrate the vulnerability of state-of-the-art models to sparse attacks and highlight the importance of developing robust machine learning systems.

Title: Unsupervised Spatial-Temporal Feature Enrichment and Fidelity Preservation Network for Skeleton based Action Recognition

• Authors: Chuankun Li, Shuai Li, Yanbo Gao, Ping Chen, Jian Li, Wanqing Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.14034
• Pdf URL: https://arxiv.org/pdf/2401.14034
• Copy Paste: [[2401.14034]] Unsupervised Spatial-Temporal Feature Enrichment and Fidelity Preservation Network for Skeleton based Action Recognition(https://arxiv.org/abs/2401.14034)
• Keywords: robust, extraction
• Abstract: Unsupervised skeleton based action recognition has achieved remarkable progress recently. Existing unsupervised learning methods suffer from severe overfitting problem, and thus small networks are used, significantly reducing the representation capability. To address this problem, the overfitting mechanism behind the unsupervised learning for skeleton based action recognition is first investigated. It is observed that the skeleton is already a relatively high-level and low-dimension feature, but not in the same manifold as the features for action recognition. Simply applying the existing unsupervised learning method may tend to produce features that discriminate the different samples instead of action classes, resulting in the overfitting problem. To solve this problem, this paper presents an Unsupervised spatial-temporal Feature Enrichment and Fidelity Preservation framework (U-FEFP) to generate rich distributed features that contain all the information of the skeleton sequence. A spatial-temporal feature transformation subnetwork is developed using spatial-temporal graph convolutional network and graph convolutional gate recurrent unit network as the basic feature extraction network. The unsupervised Bootstrap Your Own Latent based learning is used to generate rich distributed features and the unsupervised pretext task based learning is used to preserve the information of the skeleton sequence. The two unsupervised learning ways are collaborated as U-FEFP to produce robust and discriminative representations. Experimental results on three widely used benchmarks, namely NTU-RGB+D-60, NTU-RGB+D-120 and PKU-MMD dataset, demonstrate that the proposed U-FEFP achieves the best performance compared with the state-of-the-art unsupervised learning methods. t-SNE illustrations further validate that U-FEFP can learn more discriminative features for unsupervised skeleton based action recognition.

Title: (Chat)GPT v BERT: Dawn of Justice for Semantic Change Detection

• Authors: Francesco Periti, Haim Dubossarsky, Nina Tahmasebi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.14040
• Pdf URL: https://arxiv.org/pdf/2401.14040
• Copy Paste: [[2401.14040]] (Chat)GPT v BERT: Dawn of Justice for Semantic Change Detection(https://arxiv.org/abs/2401.14040)
• Keywords: transformer
• Abstract: In the universe of Natural Language Processing, Transformer-based language models like BERT and (Chat)GPT have emerged as lexical superheroes with great power to solve open research problems. In this paper, we specifically focus on the temporal problem of semantic change, and evaluate their ability to solve two diachronic extensions of the Word-in-Context (WiC) task: TempoWiC and HistoWiC. In particular, we investigate the potential of a novel, off-the-shelf technology like ChatGPT (and GPT) 3.5 compared to BERT, which represents a family of models that currently stand as the state-of-the-art for modeling semantic change. Our experiments represent the first attempt to assess the use of (Chat)GPT for studying semantic change. Our results indicate that ChatGPT performs significantly worse than the foundational GPT version. Furthermore, our results demonstrate that (Chat)GPT achieves slightly lower performance than BERT in detecting long-term changes but performs significantly worse in detecting short-term changes.

Title: Towards Goal-oriented Large Language Model Prompting: A Survey

• Authors: Haochen Li, Jonathan Leung, Zhiqi Shen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.14043
• Pdf URL: https://arxiv.org/pdf/2401.14043
• Copy Paste: [[2401.14043]] Towards Goal-oriented Large Language Model Prompting: A Survey(https://arxiv.org/abs/2401.14043)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have shown prominent performance in various downstream tasks in which prompt engineering plays a pivotal role in optimizing LLMs' performance. This paper, not as an overview of current prompt engineering methods, aims to highlight the limitation of designing prompts while holding an anthropomorphic assumption that expects LLMs to think like humans. From our review of 35 representative studies, we demonstrate that a goal-oriented prompt formulation, which guides LLMs to follow established human logical thinking, significantly improves the performance of LLMs. Furthermore, We introduce a novel taxonomy that categorizes goal-oriented prompting methods into five interconnected stages and we demonstrate the broad applicability of our framework by summarizing ten applicable tasks. With four future directions proposed, we hope to further emphasize and promote goal-oriented prompt engineering.

Title: CreativeSynth: Creative Blending and Synthesis of Visual Arts based on Multimodal Diffusion

• Authors: Nisha Huang, Weiming Dong, Yuxin Zhang, Fan Tang, Ronghui Li, Chongyang Ma, Xiu Li, Changsheng Xu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.14066
• Pdf URL: https://arxiv.org/pdf/2401.14066
• Copy Paste: [[2401.14066]] CreativeSynth: Creative Blending and Synthesis of Visual Arts based on Multimodal Diffusion(https://arxiv.org/abs/2401.14066)
• Keywords: diffusion, generative
• Abstract: Large-scale text-to-image generative models have made impressive strides, showcasing their ability to synthesize a vast array of high-quality images. However, adapting these models for artistic image editing presents two significant challenges. Firstly, users struggle to craft textual prompts that meticulously detail visual elements of the input image. Secondly, prevalent models, when effecting modifications in specific zones, frequently disrupt the overall artistic style, complicating the attainment of cohesive and aesthetically unified artworks. To surmount these obstacles, we build the innovative unified framework CreativeSynth, which is based on a diffusion model with the ability to coordinate multimodal inputs and multitask in the field of artistic image generation. By integrating multimodal features with customized attention mechanisms, CreativeSynth facilitates the importation of real-world semantic content into the domain of art through inversion and real-time style transfer. This allows for the precise manipulation of image style and content while maintaining the integrity of the original model parameters. Rigorous qualitative and quantitative evaluations underscore that CreativeSynth excels in enhancing artistic images' fidelity and preserves their innate aesthetic essence. By bridging the gap between generative models and artistic finesse, CreativeSynth becomes a custom digital palette.

Title: Ta'keed: The First Generative Fact-Checking System for Arabic Claims

• Authors: Saud Althabiti, Mohammad Ammar Alsalka, Eric Atwell
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.14067
• Pdf URL: https://arxiv.org/pdf/2401.14067
• Copy Paste: [[2401.14067]] Ta'keed: The First Generative Fact-Checking System for Arabic Claims(https://arxiv.org/abs/2401.14067)
• Keywords: generative
• Abstract: This paper introduces Ta'keed, an explainable Arabic automatic fact-checking system. While existing research often focuses on classifying claims as "True" or "False," there is a limited exploration of generating explanations for claim credibility, particularly in Arabic. Ta'keed addresses this gap by assessing claim truthfulness based on retrieved snippets, utilizing two main components: information retrieval and LLM-based claim verification. We compiled the ArFactEx, a testing gold-labelled dataset with manually justified references, to evaluate the system. The initial model achieved a promising F1 score of 0.72 in the classification task. Meanwhile, the system's generated explanations are compared with gold-standard explanations syntactically and semantically. The study recommends evaluating using semantic similarities, resulting in an average cosine similarity score of 0.76. Additionally, we explored the impact of varying snippet quantities on claim classification accuracy, revealing a potential correlation, with the model using the top seven hits outperforming others with an F1 score of 0.77.

Title: ProCNS: Progressive Prototype Calibration and Noise Suppression for Weakly-Supervised Medical Image Segmentation

• Authors: Y. Liu, L. Lin, K. K. Y. Wong, X. Tang
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.14074
• Pdf URL: https://arxiv.org/pdf/2401.14074
• Copy Paste: [[2401.14074]] ProCNS: Progressive Prototype Calibration and Noise Suppression for Weakly-Supervised Medical Image Segmentation(https://arxiv.org/abs/2401.14074)
• Keywords: segmentation
• Abstract: Weakly-supervised segmentation (WSS) has emerged as a solution to mitigate the conflict between annotation cost and model performance by adopting sparse annotation formats (e.g., point, scribble, block, etc.). Typical approaches attempt to exploit anatomy and topology priors to directly expand sparse annotations into pseudo-labels. However, due to a lack of attention to the ambiguous edges in medical images and insufficient exploration of sparse supervision, existing approaches tend to generate erroneous and overconfident pseudo proposals in noisy regions, leading to cumulative model error and performance degradation. In this work, we propose a novel WSS approach, named ProCNS, encompassing two synergistic modules devised with the principles of progressive prototype calibration and noise suppression. Specifically, we design a Prototype-based Regional Spatial Affinity (PRSA) loss to maximize the pair-wise affinities between spatial and semantic elements, providing our model of interest with more reliable guidance. The affinities are derived from the input images and the prototype-refined predictions. Meanwhile, we propose an Adaptive Noise Perception and Masking (ANPM) module to obtain more enriched and representative prototype representations, which adaptively identifies and masks noisy regions within the pseudo proposals, reducing potential erroneous interference during prototype computation. Furthermore, we generate specialized soft pseudo-labels for the noisy regions identified by ANPM, providing supplementary supervision. Extensive experiments on three medical image segmentation tasks involving different modalities demonstrate that the proposed framework significantly outperforms representative state-of-the-art methods

Title: Quantum Resistant Ciphertext-Policy Attribute-Based Encryption Scheme with Flexible Access Structure

• Authors: Shida Shamsazad
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2401.14076
• Pdf URL: https://arxiv.org/pdf/2401.14076
• Copy Paste: [[2401.14076]] Quantum Resistant Ciphertext-Policy Attribute-Based Encryption Scheme with Flexible Access Structure(https://arxiv.org/abs/2401.14076)
• Keywords: secure, security, protect, attack, robust
• Abstract: In this paper, we present a novel ciphertext-policy attribute based encryption (CP-ABE) scheme that offers a flexible access structure. Our proposed scheme incorporates an access tree as its access control policy, enabling fine-grained access control over encrypted data. The security of our scheme is provable under the hardness assumption of the decisional Ring-Learning with Errors (R-LWE) problem, ensuring robust protection against unauthorized access. CP-ABE is a cryptographic technique that allows data owners to encrypt their data with access policies defined in terms of attributes. Only users possessing the required attributes can decrypt and access the encrypted data. Our scheme extends the capabilities of CP-ABE by introducing a flexible access structure based on an access tree. This structure enables more complex and customizable access policies, accommodating a wider range of real-world scenarios. To ensure the security of our scheme, we rely on the decisional R-LWE problem, a well-established hardness assumption in cryptography. By proving the security of our scheme under this assumption, we provide a strong guarantee of protection against potential attacks. Furthermore, our proposed scheme operates in the standard model, which means it does not rely on any additional assumptions or idealized cryptographic primitives. This enhances the practicality and applicability of our scheme, making it suitable for real-world deployment. We evaluate the performance and efficiency of our scheme through extensive simulations and comparisons with existing CP-ABE schemes. The results demonstrate the effectiveness and scalability of our proposed approach, highlighting its potential for secure and flexible data access control in various domains.

Title: Double Trouble? Impact and Detection of Duplicates in Face Image Datasets

• Authors: Torsten Schlett, Christian Rathgeb, Juan Tapia, Christoph Busch
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.14088
• Pdf URL: https://arxiv.org/pdf/2401.14088
• Copy Paste: [[2401.14088]] Double Trouble? Impact and Detection of Duplicates in Face Image Datasets(https://arxiv.org/abs/2401.14088)
• Keywords: biometric
• Abstract: Various face image datasets intended for facial biometrics research were created via web-scraping, i.e. the collection of images publicly available on the internet. This work presents an approach to detect both exactly and nearly identical face image duplicates, using file and image hashes. The approach is extended through the use of face image preprocessing. Additional steps based on face recognition and face image quality assessment models reduce false positives, and facilitate the deduplication of the face images both for intra- and inter-subject duplicate sets. The presented approach is applied to five datasets, namely LFW, TinyFace, Adience, CASIA-WebFace, and C-MS-Celeb (a cleaned MS-Celeb-1M variant). Duplicates are detected within every dataset, with hundreds to hundreds of thousands of duplicates for all except LFW. Face recognition and quality assessment experiments indicate a minor impact on the results through the duplicate removal. The final deduplication data is publicly available.

Title: A Modular Approach to Automatic Cyber Threat Attribution using Opinion Pools

• Authors: Koen T.W. Teuwen
• Subjects: cs.CR, cs.LG, cs.SE
• Abstract URL: https://arxiv.org/abs/2401.14090
• Pdf URL: https://arxiv.org/pdf/2401.14090
• Copy Paste: [[2401.14090]] A Modular Approach to Automatic Cyber Threat Attribution using Opinion Pools(https://arxiv.org/abs/2401.14090)
• Keywords: interpretability
• Abstract: Cyber threat attribution can play an important role in increasing resilience against digital threats. Recent research focuses on automating the threat attribution process and on integrating it with other efforts, such as threat hunting. To support increasing automation of the cyber threat attribution process, this paper proposes a modular architecture as an alternative to current monolithic automated approaches. The modular architecture can utilize opinion pools to combine the output of concrete attributors. The proposed solution increases the tractability of the threat attribution problem and offers increased usability and interpretability, as opposed to monolithic alternatives. In addition, a Pairing Aggregator is proposed as an aggregation method that forms pairs of attributors based on distinct features to produce intermediary results before finally producing a single Probability Mass Function (PMF) as output. The Pairing Aggregator sequentially applies both the logarithmic opinion pool and the linear opinion pool. An experimental validation suggests that the modular approach does not result in decreased performance and can even enhance precision and recall compared to monolithic alternatives. The results also suggest that the Pairing Aggregator can improve precision over the linear and logarithmic opinion pools. Furthermore, the improved k-accuracy in the experiment suggests that forensic experts can leverage the resulting PMF during their manual attribution processes to enhance their efficiency.

Title: Carry Your Fault: A Fault Propagation Attack on Side-Channel Protected LWE-based KEM

• Authors: Suparna Kundu, Siddhartha Chowdhury, Sayandeep Saha, Angshuman Karmakar, Debdeep Mukhopadhyay, Ingrid Verbauwhede
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2401.14098
• Pdf URL: https://arxiv.org/pdf/2401.14098
• Copy Paste: [[2401.14098]] Carry Your Fault: A Fault Propagation Attack on Side-Channel Protected LWE-based KEM(https://arxiv.org/abs/2401.14098)
• Keywords: secure, protect, attack, robust
• Abstract: Post-quantum cryptographic (PQC) algorithms, especially those based on the learning with errors (LWE) problem, have been subjected to several physical attacks in the recent past. Although the attacks broadly belong to two classes - passive side-channel attacks and active fault attacks, the attack strategies vary significantly due to the inherent complexities of such algorithms. Exploring further attack surfaces is, therefore, an important step for eventually securing the deployment of these algorithms. Also, it is important to test the robustness of the already proposed countermeasures in this regard. In this work, we propose a new fault attack on side-channel secure masked implementation of LWE-based key-encapsulation mechanisms (KEMs) exploiting fault propagation. The attack typically originates due to an algorithmic modification widely used to enable masking, namely the Arithmetic-to-Boolean (A2B) conversion. We exploit the data dependency of the adder carry chain in A2B and extract sensitive information, albeit masking (of arbitrary order) being present. As a practical demonstration of the exploitability of this information leakage, we show key recovery attacks of Kyber, although the leakage also exists for other schemes like Saber. The attack on Kyber targets the decapsulation module and utilizes Belief Propagation (BP) for key recovery. To the best of our knowledge, it is the first attack exploiting an algorithmic component introduced to ease masking rather than only exploiting the randomness introduced by masking to obtain desired faults (as done by Delvaux). Finally, we performed both simulated and electromagnetic (EM) fault-based practical validation of the attack for an open-source first-order secure Kyber implementation running on an STM32 platform.

Title: Learning under Label Noise through Few-Shot Human-in-the-Loop Refinement

• Authors: Aaqib Saeed, Dimitris Spathis, Jungwoo Oh, Edward Choi, Ali Etemad
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2401.14107
• Pdf URL: https://arxiv.org/pdf/2401.14107
• Copy Paste: [[2401.14107]] Learning under Label Noise through Few-Shot Human-in-the-Loop Refinement(https://arxiv.org/abs/2401.14107)
• Keywords: robust
• Abstract: Wearable technologies enable continuous monitoring of various health metrics, such as physical activity, heart rate, sleep, and stress levels. A key challenge with wearable data is obtaining quality labels. Unlike modalities like video where the videos themselves can be effectively used to label objects or events, wearable data do not contain obvious cues about the physical manifestation of the users and usually require rich metadata. As a result, label noise can become an increasingly thorny issue when labeling such data. In this paper, we propose a novel solution to address noisy label learning, entitled Few-Shot Human-in-the-Loop Refinement (FHLR). Our method initially learns a seed model using weak labels. Next, it fine-tunes the seed model using a handful of expert corrections. Finally, it achieves better generalizability and robustness by merging the seed and fine-tuned models via weighted parameter averaging. We evaluate our approach on four challenging tasks and datasets, and compare it against eight competitive baselines designed to deal with noisy labels. We show that FHLR achieves significantly better performance when learning from noisy labels and achieves state-of-the-art by a large margin, with up to 19% accuracy improvement under symmetric and asymmetric noise. Notably, we find that FHLR is particularly robust to increased label noise, unlike prior works that suffer from severe performance degradation. Our work not only achieves better generalization in high-stakes health sensing benchmarks but also sheds light on how noise affects commonly-used models.

Title: CompactifAI: Extreme Compression of Large Language Models using Quantum-Inspired Tensor Networks

• Authors: Andrei Tomut, Saeed S. Jahromi, Sukhbinder Singh, Faysal Ishtiaq, Cesar Muñoz, Prabdeep Singh Bajaj, Ali Elborady, Gianni del Bimbo, Mehrazin Alizadeh, David Montero, Pablo Martin-Ramiro, Muhammad Ibrahim, Oussama Tahiri Alaoui, John Malcolm, Samuel Mugel, Roman Orus
• Subjects: cs.CL, cs.AI, cs.LG, quant-ph
• Abstract URL: https://arxiv.org/abs/2401.14109
• Pdf URL: https://arxiv.org/pdf/2401.14109
• Copy Paste: [[2401.14109]] CompactifAI: Extreme Compression of Large Language Models using Quantum-Inspired Tensor Networks(https://arxiv.org/abs/2401.14109)
• Keywords: generative, large language model
• Abstract: Large Language Models (LLMs) such as ChatGPT and LlaMA are advancing rapidly in generative Artificial Intelligence (AI), but their immense size poses significant challenges, such as huge training and inference costs, substantial energy demands, and limitations for on-site deployment. Traditional compression methods such as pruning, distillation, and low-rank approximation focus on reducing the effective number of neurons in the network, while quantization focuses on reducing the numerical precision of individual weights to reduce the model size while keeping the number of neurons fixed. While these compression methods have been relatively successful in practice, there's no compelling reason to believe that truncating the number of neurons is an optimal strategy. In this context, this paper introduces CompactifAI, an innovative LLM compression approach using quantum-inspired Tensor Networks that focuses on the model's correlation space instead, allowing for a more controlled, refined and interpretable model compression. Our method is versatile and can be implemented with - or on top of - other compression techniques. As a benchmark, we demonstrate that CompactifAI alone enables compression of the LlaMA-2 7B model to only $30\%$ of its original size while recovering over $90\%$ of the original accuracy after a brief distributed retraining.

Title: Scene Graph to Image Synthesis: Integrating CLIP Guidance with Graph Conditioning in Diffusion Models

• Authors: Rameshwar Mishra, A V Subramanyam
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.14111
• Pdf URL: https://arxiv.org/pdf/2401.14111
• Copy Paste: [[2401.14111]] Scene Graph to Image Synthesis: Integrating CLIP Guidance with Graph Conditioning in Diffusion Models(https://arxiv.org/abs/2401.14111)
• Keywords: diffusion, generative
• Abstract: Advancements in generative models have sparked significant interest in generating images while adhering to specific structural guidelines. Scene graph to image generation is one such task of generating images which are consistent with the given scene graph. However, the complexity of visual scenes poses a challenge in accurately aligning objects based on specified relations within the scene graph. Existing methods approach this task by first predicting a scene layout and generating images from these layouts using adversarial training. In this work, we introduce a novel approach to generate images from scene graphs which eliminates the need of predicting intermediate layouts. We leverage pre-trained text-to-image diffusion models and CLIP guidance to translate graph knowledge into images. Towards this, we first pre-train our graph encoder to align graph features with CLIP features of corresponding images using a GAN based training. Further, we fuse the graph features with CLIP embedding of object labels present in the given scene graph to create a graph consistent CLIP guided conditioning signal. In the conditioning input, object embeddings provide coarse structure of the image and graph features provide structural alignment based on relationships among objects. Finally, we fine tune a pre-trained diffusion model with the graph consistent conditioning signal with reconstruction and CLIP alignment loss. Elaborate experiments reveal that our method outperforms existing methods on standard benchmarks of COCO-stuff and Visual Genome dataset.

Title: FP6-LLM: Efficiently Serving Large Language Models Through FP6-Centric Algorithm-System Co-Design

• Authors: Haojun Xia, Zhen Zheng, Xiaoxia Wu, Shiyang Chen, Zhewei Yao, Stephen Youn, Arash Bakhtiari, Michael Wyatt, Donglin Zhuang, Zhongzhu Zhou, Olatunji Ruwase, Yuxiong He, Shuaiwen Leon Song
• Subjects: cs.LG, cs.AI, cs.AR
• Abstract URL: https://arxiv.org/abs/2401.14112
• Pdf URL: https://arxiv.org/pdf/2401.14112
• Copy Paste: [[2401.14112]] FP6-LLM: Efficiently Serving Large Language Models Through FP6-Centric Algorithm-System Co-Design(https://arxiv.org/abs/2401.14112)
• Keywords: large language model
• Abstract: Six-bit quantization (FP6) can effectively reduce the size of large language models (LLMs) and preserve the model quality consistently across varied applications. However, existing systems do not provide Tensor Core support for FP6 quantization and struggle to achieve practical performance improvements during LLM inference. It is challenging to support FP6 quantization on GPUs due to (1) unfriendly memory access of model weights with irregular bit-width and (2) high runtime overhead of weight de-quantization. To address these problems, we propose TC-FPx, the first full-stack GPU kernel design scheme with unified Tensor Core support of float-point weights for various quantization bit-width. We integrate TC-FPx kernel into an existing inference system, providing new end-to-end support (called FP6-LLM) for quantized LLM inference, where better trade-offs between inference cost and model quality are achieved. Experiments show that FP6-LLM enables the inference of LLaMA-70b using only a single GPU, achieving 1.69x-2.65x higher normalized inference throughput than the FP16 baseline. The source code will be publicly available soon.

Title: Enabling Cross-Camera Collaboration for Video Analytics on Distributed Smart Cameras

• Authors: Chulhong Min, Juheon Yi, Utku Gunay Acer, Fahim Kawsar
• Subjects: cs.CV, cs.DC
• Abstract URL: https://arxiv.org/abs/2401.14132
• Pdf URL: https://arxiv.org/pdf/2401.14132
• Copy Paste: [[2401.14132]] Enabling Cross-Camera Collaboration for Video Analytics on Distributed Smart Cameras(https://arxiv.org/abs/2401.14132)
• Keywords: robust
• Abstract: Overlapping cameras offer exciting opportunities to view a scene from different angles, allowing for more advanced, comprehensive and robust analysis. However, existing visual analytics systems for multi-camera streams are mostly limited to (i) per-camera processing and aggregation and (ii) workload-agnostic centralized processing architectures. In this paper, we present Argus, a distributed video analytics system with cross-camera collaboration on smart cameras. We identify multi-camera, multi-target tracking as the primary task of multi-camera video analytics and develop a novel technique that avoids redundant, processing-heavy identification tasks by leveraging object-wise spatio-temporal association in the overlapping fields of view across multiple cameras. We further develop a set of techniques to perform these operations across distributed cameras without cloud support at low latency by (i) dynamically ordering the camera and object inspection sequence and (ii) flexibly distributing the workload across smart cameras, taking into account network transmission and heterogeneous computational capacities. Evaluation of three real-world overlapping camera datasets with two Nvidia Jetson devices shows that Argus reduces the number of object identifications and end-to-end latency by up to 7.13x and 2.19x (4.86x and 1.60x compared to the state-of-the-art), while achieving comparable tracking quality.

Title: Expression-aware video inpainting for HMD removal in XR applications

• Authors: Fatemeh Ghorbani Lohesara, Karen Egiazarian, Sebastian Knorr
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.14136
• Pdf URL: https://arxiv.org/pdf/2401.14136
• Copy Paste: [[2401.14136]] Expression-aware video inpainting for HMD removal in XR applications(https://arxiv.org/abs/2401.14136)
• Keywords: generative
• Abstract: Head-mounted displays (HMDs) serve as indispensable devices for observing extended reality (XR) environments and virtual content. However, HMDs present an obstacle to external recording techniques as they block the upper face of the user. This limitation significantly affects social XR applications, specifically teleconferencing, where facial features and eye gaze information play a vital role in creating an immersive user experience. In this study, we propose a new network for expression-aware video inpainting for HMD removal (EVI-HRnet) based on generative adversarial networks (GANs). Our model effectively fills in missing information with regard to facial landmarks and a single occlusion-free reference image of the user. The framework and its components ensure the preservation of the user's identity across frames using the reference frame. To further improve the level of realism of the inpainted output, we introduce a novel facial expression recognition (FER) loss function for emotion preservation. Our results demonstrate the remarkable capability of the proposed framework to remove HMDs from facial videos while maintaining the subject's facial expression and identity. Moreover, the outputs exhibit temporal consistency along the inpainted frames. This lightweight framework presents a practical approach for HMD occlusion removal, with the potential to enhance various collaborative XR applications without the need for additional hardware.

Title: True Knowledge Comes from Practice: Aligning LLMs with Embodied Environments via Reinforcement Learning

• Authors: Weihao Tan, Wentao Zhang, Shanqi Liu, Longtao Zheng, Xinrun Wang, Bo An
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2401.14151
• Pdf URL: https://arxiv.org/pdf/2401.14151
• Copy Paste: [[2401.14151]] True Knowledge Comes from Practice: Aligning LLMs with Embodied Environments via Reinforcement Learning(https://arxiv.org/abs/2401.14151)
• Keywords: robust, large language model
• Abstract: Despite the impressive performance across numerous tasks, large language models (LLMs) often fail in solving simple decision-making tasks due to the misalignment of the knowledge in LLMs with environments. On the contrary, reinforcement learning (RL) agents learn policies from scratch, which makes them always align with environments but difficult to incorporate prior knowledge for efficient explorations. To narrow the gap, we propose TWOSOME, a novel general online framework that deploys LLMs as decision-making agents to efficiently interact and align with embodied environments via RL without requiring any prepared datasets or prior knowledge of the environments. Firstly, we query the joint probabilities of each valid action with LLMs to form behavior policies. Then, to enhance the stability and robustness of the policies, we propose two normalization methods and summarize four prompt design principles. Finally, we design a novel parameter-efficient training architecture where the actor and critic share one frozen LLM equipped with low-rank adapters (LoRA) updated by PPO. We conduct extensive experiments to evaluate TWOSOME. i) TWOSOME exhibits significantly better sample efficiency and performance compared to the conventional RL method, PPO, and prompt tuning method, SayCan, in both classical decision-making environment, Overcooked, and simulated household environment, VirtualHome. ii) Benefiting from LLMs' open-vocabulary feature, TWOSOME shows superior generalization ability to unseen tasks. iii) Under our framework, there is no significant loss of the LLMs' original ability during online PPO finetuning.

Title: Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks

• Authors: Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, Zhaoyang Zeng, Hao Zhang, Feng Li, Jie Yang, Hongyang Li, Qing Jiang, Lei Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.14159
• Pdf URL: https://arxiv.org/pdf/2401.14159
• Copy Paste: [[2401.14159]] Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks(https://arxiv.org/abs/2401.14159)
• Keywords: diffusion, segmentation
• Abstract: We introduce Grounded SAM, which uses Grounding DINO as an open-set object detector to combine with the segment anything model (SAM). This integration enables the detection and segmentation of any regions based on arbitrary text inputs and opens a door to connecting various vision models. As shown in Fig.1, a wide range of vision tasks can be achieved by using the versatile Grounded SAM pipeline. For example, an automatic annotation pipeline based solely on input images can be realized by incorporating models such as BLIP and Recognize Anything. Additionally, incorporating Stable-Diffusion allows for controllable image editing, while the integration of OSX facilitates promptable 3D human motion analysis. Grounded SAM also shows superior performance on open-vocabulary benchmarks, achieving 48.7 mean AP on SegInW (Segmentation in the wild) zero-shot benchmark with the combination of Grounding DINO-Base and SAM-Huge models.

Title: Vivim: a Video Vision Mamba for Medical Video Object Segmentation

• Authors: Yijun Yang, Zhaohu Xing, Lei Zhu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.14168
• Pdf URL: https://arxiv.org/pdf/2401.14168
• Copy Paste: [[2401.14168]] Vivim: a Video Vision Mamba for Medical Video Object Segmentation(https://arxiv.org/abs/2401.14168)
• Keywords: transformer, segmentation
• Abstract: Traditional convolutional neural networks have a limited receptive field while transformer-based networks are mediocre in constructing long-term dependency from the perspective of computational complexity. Such the bottleneck poses a significant challenge when processing long video sequences in video analysis tasks. Very recently, the state space models (SSMs) with efficient hardware-aware designs, famous by Mamba, have exhibited impressive achievements in long sequence modeling, which facilitates the development of deep neural networks on many vision tasks. To better capture available cues in video frames, this paper presents a generic Video Vision Mamba-based framework for medical video object segmentation tasks, named Vivim. Our Vivim can effectively compress the long-term spatiotemporal representation into sequences at varying scales by our designed Temporal Mamba Block. Compared to existing video-level Transformer-based methods, our model maintains excellent segmentation results with better speed performance. Extensive experiments on the breast US dataset demonstrate the effectiveness and efficiency of our Vivim. The code for Vivim is available at: https://github.com/scott-yjyang/Vivim.

Title: How Can Large Language Models Understand Spatial-Temporal Data?

• Authors: Lei Liu, Shuo Yu, Runze Wang, Zhenxun Ma, Yanming Shen
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2401.14192
• Pdf URL: https://arxiv.org/pdf/2401.14192
• Copy Paste: [[2401.14192]] How Can Large Language Models Understand Spatial-Temporal Data?(https://arxiv.org/abs/2401.14192)
• Keywords: large language model
• Abstract: While Large Language Models (LLMs) dominate tasks like natural language processing and computer vision, harnessing their power for spatial-temporal forecasting remains challenging. The disparity between sequential text and complex spatial-temporal data hinders this application. To address this issue, this paper introduces STG-LLM, an innovative approach empowering LLMs for spatial-temporal forecasting. We tackle the data mismatch by proposing: 1) STG-Tokenizer: This spatial-temporal graph tokenizer transforms intricate graph data into concise tokens capturing both spatial and temporal relationships; 2) STG-Adapter: This minimalistic adapter, consisting of linear encoding and decoding layers, bridges the gap between tokenized data and LLM comprehension. By fine-tuning only a small set of parameters, it can effectively grasp the semantics of tokens generated by STG-Tokenizer, while preserving the original natural language understanding capabilities of LLMs. Extensive experiments on diverse spatial-temporal benchmark datasets show that STG-LLM successfully unlocks LLM potential for spatial-temporal forecasting. Remarkably, our approach achieves competitive performance on par with dedicated SOTA methods.

Title: Communication-Efficient Federated Learning through Adaptive Weight Clustering and Server-Side Distillation

• Authors: Vasileios Tsouvalas. Aaqib Saeed, Tanir Ozcelebi, Nirvana Meratnia
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2401.14211
• Pdf URL: https://arxiv.org/pdf/2401.14211
• Copy Paste: [[2401.14211]] Communication-Efficient Federated Learning through Adaptive Weight Clustering and Server-Side Distillation(https://arxiv.org/abs/2401.14211)
• Keywords: privacy, federate
• Abstract: Federated Learning (FL) is a promising technique for the collaborative training of deep neural networks across multiple devices while preserving data privacy. Despite its potential benefits, FL is hindered by excessive communication costs due to repeated server-client communication during training. To address this challenge, model compression techniques, such as sparsification and weight clustering are applied, which often require modifying the underlying model aggregation schemes or involve cumbersome hyperparameter tuning, with the latter not only adjusts the model's compression rate but also limits model's potential for continuous improvement over growing data. In this paper, we propose FedCompress, a novel approach that combines dynamic weight clustering and server-side knowledge distillation to reduce communication costs while learning highly generalizable models. Through a comprehensive evaluation on diverse public datasets, we demonstrate the efficacy of our approach compared to baselines in terms of communication costs and inference speed. We will make our implementation public upon acceptance.

Title: AR-GAN: Generative Adversarial Network-Based Defense Method Against Adversarial Attacks on the Traffic Sign Classification System of Autonomous Vehicles

• Authors: M Sabbir Salek, Abdullah Al Mamun, Mashrur Chowdhury
• Subjects: cs.CV, cs.AI, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.14232
• Pdf URL: https://arxiv.org/pdf/2401.14232
• Copy Paste: [[2401.14232]] AR-GAN: Generative Adversarial Network-Based Defense Method Against Adversarial Attacks on the Traffic Sign Classification System of Autonomous Vehicles(https://arxiv.org/abs/2401.14232)
• Keywords: defense, attack, generative
• Abstract: This study developed a generative adversarial network (GAN)-based defense method for traffic sign classification in an autonomous vehicle (AV), referred to as the attack-resilient GAN (AR-GAN). The novelty of the AR-GAN lies in (i) assuming zero knowledge of adversarial attack models and samples and (ii) providing consistently high traffic sign classification performance under various adversarial attack types. The AR-GAN classification system consists of a generator that denoises an image by reconstruction, and a classifier that classifies the reconstructed image. The authors have tested the AR-GAN under no-attack and under various adversarial attacks, such as Fast Gradient Sign Method (FGSM), DeepFool, Carlini and Wagner (C&W), and Projected Gradient Descent (PGD). The authors considered two forms of these attacks, i.e., (i) black-box attacks (assuming the attackers possess no prior knowledge of the classifier), and (ii) white-box attacks (assuming the attackers possess full knowledge of the classifier). The classification performance of the AR-GAN was compared with several benchmark adversarial defense methods. The results showed that both the AR-GAN and the benchmark defense methods are resilient against black-box attacks and could achieve similar classification performance to that of the unperturbed images. However, for all the white-box attacks considered in this study, the AR-GAN method outperformed the benchmark defense methods. In addition, the AR-GAN was able to maintain its high classification performance under varied white-box adversarial perturbation magnitudes, whereas the performance of the other defense methods dropped abruptly at increased perturbation magnitudes.

Title: Improving Natural Language Capability of Code Large Language Model

• Authors: Wei Li, Daoguang Zan, Bei Guan, Ailun Yu, Xiaolin Chen, Yongji Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.14242
• Pdf URL: https://arxiv.org/pdf/2401.14242
• Copy Paste: [[2401.14242]] Improving Natural Language Capability of Code Large Language Model(https://arxiv.org/abs/2401.14242)
• Keywords: large language model
• Abstract: Code large language models (Code LLMs) have demonstrated remarkable performance in code generation. Nonetheless, most existing works focus on boosting code LLMs from the perspective of programming capabilities, while their natural language capabilities receive less attention. To fill this gap, we thus propose a novel framework, comprising two modules: AttentionExtractor, which is responsible for extracting key phrases from the user's natural language requirements, and AttentionCoder, which leverages these extracted phrases to generate target code to solve the requirement. This framework pioneers an innovative idea by seamlessly integrating code LLMs with traditional natural language processing tools. To validate the effectiveness of the framework, we craft a new code generation benchmark, called MultiNL-H, covering five natural languages. Extensive experimental results demonstrate the effectiveness of our proposed framework.

Title: JUMP: A joint multimodal registration pipeline for neuroimaging with minimal preprocessing

• Authors: Adria Casamitjana, Juan Eugenio Iglesias, Raul Tudela, Aida Ninerola-Baizan, Roser Sala-Llonch
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.14250
• Pdf URL: https://arxiv.org/pdf/2401.14250
• Copy Paste: [[2401.14250]] JUMP: A joint multimodal registration pipeline for neuroimaging with minimal preprocessing(https://arxiv.org/abs/2401.14250)
• Keywords: robust
• Abstract: We present a pipeline for unbiased and robust multimodal registration of neuroimaging modalities with minimal pre-processing. While typical multimodal studies need to use multiple independent processing pipelines, with diverse options and hyperparameters, we propose a single and structured framework to jointly process different image modalities. The use of state-of-the-art learning-based techniques enables fast inferences, which makes the presented method suitable for large-scale and/or multi-cohort datasets with a diverse number of modalities per session. The pipeline currently works with structural MRI, resting state fMRI and amyloid PET images. We show the predictive power of the derived biomarkers using in a case-control study and study the cross-modal relationship between different image modalities. The code can be found in https: //github.com/acasamitjana/JUMP.

Title: Interpretable Solutions for Breast Cancer Diagnosis with Grammatical Evolution and Data Augmentation

• Authors: Yumnah Hasan, Allan de Lima, Fatemeh Amerehi, Darian Reyes Fernandez de Bulnes, Patrick Healy, Conor Ryan
• Subjects: cs.LG, cs.NE
• Abstract URL: https://arxiv.org/abs/2401.14255
• Pdf URL: https://arxiv.org/pdf/2401.14255
• Copy Paste: [[2401.14255]] Interpretable Solutions for Breast Cancer Diagnosis with Grammatical Evolution and Data Augmentation(https://arxiv.org/abs/2401.14255)
• Keywords: interpretability
• Abstract: Medical imaging diagnosis increasingly relies on Machine Learning (ML) models. This is a task that is often hampered by severely imbalanced datasets, where positive cases can be quite rare. Their use is further compromised by their limited interpretability, which is becoming increasingly important. While post-hoc interpretability techniques such as SHAP and LIME have been used with some success on so-called black box models, the use of inherently understandable models makes such endeavors more fruitful. This paper addresses these issues by demonstrating how a relatively new synthetic data generation technique, STEM, can be used to produce data to train models produced by Grammatical Evolution (GE) that are inherently understandable. STEM is a recently introduced combination of the Synthetic Minority Oversampling Technique (SMOTE), Edited Nearest Neighbour (ENN), and Mixup; it has previously been successfully used to tackle both between class and within class imbalance issues. We test our technique on the Digital Database for Screening Mammography (DDSM) and the Wisconsin Breast Cancer (WBC) datasets and compare Area Under the Curve (AUC) results with an ensemble of the top three performing classifiers from a set of eight standard ML classifiers with varying degrees of interpretability. We demonstrate that the GE-derived models present the best AUC while still maintaining interpretable solutions.

Title: Producing Plankton Classifiers that are Robust to Dataset Shift

• Authors: Cheng Chen, Sreenath Kyathanahally, Marta Reyes, Stefanie Merkli, Ewa Merz, Emanuele Francazi, Marvin Hoege, Francesco Pomati, Marco Baity-Jesi
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.14256
• Pdf URL: https://arxiv.org/pdf/2401.14256
• Copy Paste: [[2401.14256]] Producing Plankton Classifiers that are Robust to Dataset Shift(https://arxiv.org/abs/2401.14256)
• Keywords: robust, transformer
• Abstract: Modern plankton high-throughput monitoring relies on deep learning classifiers for species recognition in water ecosystems. Despite satisfactory nominal performances, a significant challenge arises from Dataset Shift, which causes performances to drop during deployment. In our study, we integrate the ZooLake dataset with manually-annotated images from 10 independent days of deployment, serving as test cells to benchmark Out-Of-Dataset (OOD) performances. Our analysis reveals instances where classifiers, initially performing well in In-Dataset conditions, encounter notable failures in practical scenarios. For example, a MobileNet with a 92% nominal test accuracy shows a 77% OOD accuracy. We systematically investigate conditions leading to OOD performance drops and propose a preemptive assessment method to identify potential pitfalls when classifying new data, and pinpoint features in OOD images that adversely impact classification. We present a three-step pipeline: (i) identifying OOD degradation compared to nominal test performance, (ii) conducting a diagnostic analysis of degradation causes, and (iii) providing solutions. We find that ensembles of BEiT vision transformers, with targeted augmentations addressing OOD robustness, geometric ensembling, and rotation-based test-time augmentation, constitute the most robust model, which we call BEsT model. It achieves an 83% OOD accuracy, with errors concentrated on container classes. Moreover, it exhibits lower sensitivity to dataset shift, and reproduces well the plankton abundances. Our proposed pipeline is applicable to generic plankton classifiers, contingent on the availability of suitable test cells. By identifying critical shortcomings and offering practical procedures to fortify models against dataset shift, our study contributes to the development of more reliable plankton classification technologies.

Title: Sketch2NeRF: Multi-view Sketch-guided Text-to-3D Generation

• Authors: Minglin Chen, Longguang Wang, Weihao Yuan, Yukun Wang, Zhe Sheng, Yisheng He, Zilong Dong, Liefeng Bo, Yulan Guo
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.14257
• Pdf URL: https://arxiv.org/pdf/2401.14257
• Copy Paste: [[2401.14257]] Sketch2NeRF: Multi-view Sketch-guided Text-to-3D Generation(https://arxiv.org/abs/2401.14257)
• Keywords: diffusion
• Abstract: Recently, text-to-3D approaches have achieved high-fidelity 3D content generation using text description. However, the generated objects are stochastic and lack fine-grained control. Sketches provide a cheap approach to introduce such fine-grained control. Nevertheless, it is challenging to achieve flexible control from these sketches due to their abstraction and ambiguity. In this paper, we present a multi-view sketch-guided text-to-3D generation framework (namely, Sketch2NeRF) to add sketch control to 3D generation. Specifically, our method leverages pretrained 2D diffusion models (e.g., Stable Diffusion and ControlNet) to supervise the optimization of a 3D scene represented by a neural radiance field (NeRF). We propose a novel synchronized generation and reconstruction method to effectively optimize the NeRF. In the experiments, we collected two kinds of multi-view sketch datasets to evaluate the proposed method. We demonstrate that our method can synthesize 3D consistent contents with fine-grained sketch control while being high-fidelity to text prompts. Extensive results show that our method achieves state-of-the-art performance in terms of sketch similarity and text alignment.

Title: Transformers and Cortical Waves: Encoders for Pulling In Context Across Time

• Authors: Lyle Muller, Patricia S. Churchland, Terrence J. Sejnowski
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.14267
• Pdf URL: https://arxiv.org/pdf/2401.14267
• Copy Paste: [[2401.14267]] Transformers and Cortical Waves: Encoders for Pulling In Context Across Time(https://arxiv.org/abs/2401.14267)
• Keywords: transformer, large language model
• Abstract: The capabilities of transformer networks such as ChatGPT and other Large Language Models (LLMs) have captured the world's attention. The crucial computational mechanism underlying their performance relies on transforming a complete input sequence - for example, all the words in a sentence into a long "encoding vector" - that allows transformers to learn long-range temporal dependencies in naturalistic sequences. Specifically, "self-attention" applied to this encoding vector enhances temporal context in transformers by computing associations between pairs of words in the input sequence. We suggest that waves of neural activity, traveling across single cortical regions or across multiple regions at the whole-brain scale, could implement a similar encoding principle. By encapsulating recent input history into a single spatial pattern at each moment in time, cortical waves may enable temporal context to be extracted from sequences of sensory inputs, the same computational principle used in transformers.

Title: RomanSetu: Efficiently unlocking multilingual capabilities of Large Language Models models via Romanization

• Authors: Jaavid Aktar Husain, Raj Dabre, Aswanth Kumar, Ratish Puduppully, Anoop Kunchukuttan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.14280
• Pdf URL: https://arxiv.org/pdf/2401.14280
• Copy Paste: [[2401.14280]] RomanSetu: Efficiently unlocking multilingual capabilities of Large Language Models models via Romanization(https://arxiv.org/abs/2401.14280)
• Keywords: large language model
• Abstract: This study addresses the challenge of extending Large Language Models (LLMs) to non-English languages, specifically those using non-Latin scripts. We propose an innovative approach that utilizes the romanized form of text as an interface for LLMs, hypothesizing that its frequent informal use and shared tokens with English enhance cross-lingual alignment. Focusing on Hindi, we demonstrate through Hindi-to-English translation and sentiment analysis tasks that romanized text not only significantly improves inference efficiency due to its lower fertility compared to native text but also achieves competitive performance with limited pre-training. Additionally, our novel multi-script prompting approach, which combines romanized and native texts, shows promise in further enhancing task performance. These findings suggest the potential of romanization in bridging the language gap for LLM applications, with future work aimed at expanding this approach to more languages and tasks.

Title: POUR-Net: A Population-Prior-Aided Over-Under-Representation Network for Low-Count PET Attenuation Map Generation

• Authors: Bo Zhou, Jun Hou, Tianqi Chen, Yinchi Zhou, Xiongchao Chen, Huidong Xie, Qiong Liu, Xueqi Guo, Yu-Jung Tsai, Vladimir Y. Panin, Takuya Toyonaga, James S. Duncan, Chi Liu
• Subjects: cs.CV, cs.AI, eess.IV
• Abstract URL: https://arxiv.org/abs/2401.14285
• Pdf URL: https://arxiv.org/pdf/2401.14285
• Copy Paste: [[2401.14285]] POUR-Net: A Population-Prior-Aided Over-Under-Representation Network for Low-Count PET Attenuation Map Generation(https://arxiv.org/abs/2401.14285)
• Keywords: extraction
• Abstract: Low-dose PET offers a valuable means of minimizing radiation exposure in PET imaging. However, the prevalent practice of employing additional CT scans for generating attenuation maps (u-map) for PET attenuation correction significantly elevates radiation doses. To address this concern and further mitigate radiation exposure in low-dose PET exams, we propose POUR-Net - an innovative population-prior-aided over-under-representation network that aims for high-quality attenuation map generation from low-dose PET. First, POUR-Net incorporates an over-under-representation network (OUR-Net) to facilitate efficient feature extraction, encompassing both low-resolution abstracted and fine-detail features, for assisting deep generation on the full-resolution level. Second, complementing OUR-Net, a population prior generation machine (PPGM) utilizing a comprehensive CT-derived u-map dataset, provides additional prior information to aid OUR-Net generation. The integration of OUR-Net and PPGM within a cascade framework enables iterative refinement of $\mu$-map generation, resulting in the production of high-quality $\mu$-maps. Experimental results underscore the effectiveness of POUR-Net, showing it as a promising solution for accurate CT-free low-count PET attenuation correction, which also surpasses the performance of previous baseline methods.

Title: Topologies of Reasoning: Demystifying Chains, Trees, and Graphs of Thoughts

• Authors: Maciej Besta, Florim Memedi, Zhenyu Zhang, Robert Gerstenberger, Nils Blach, Piotr Nyczyk, Marcin Copik, Grzegorz Kwaśniewski, Jürgen Müller, Lukas Gianinazzi, Ales Kubicek, Hubert Niewiadomski, Onur Mutlu, Torsten Hoefler
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.14295
• Pdf URL: https://arxiv.org/pdf/2401.14295
• Copy Paste: [[2401.14295]] Topologies of Reasoning: Demystifying Chains, Trees, and Graphs of Thoughts(https://arxiv.org/abs/2401.14295)
• Keywords: large language model
• Abstract: The field of natural language processing (NLP) has witnessed significant progress in recent years, with a notable focus on improving large language models' (LLM) performance through innovative prompting techniques. Among these, prompt engineering coupled with structures has emerged as a promising paradigm, with designs such as Chain-of-Thought, Tree of Thoughts, or Graph of Thoughts, in which the overall LLM reasoning is guided by a structure such as a graph. As illustrated with numerous examples, this paradigm significantly enhances the LLM's capability to solve numerous tasks, ranging from logical or mathematical reasoning to planning or creative writing. To facilitate the understanding of this growing field and pave the way for future developments, we devise a general blueprint for effective and efficient LLM reasoning schemes. For this, we conduct an in-depth analysis of the prompt execution pipeline, clarifying and clearly defining different concepts. We then build the first taxonomy of structure-enhanced LLM reasoning schemes. We focus on identifying fundamental classes of harnessed structures, and we analyze the representations of these structures, algorithms executed with these structures, and many others. We refer to these structures as reasoning topologies, because their representation becomes to a degree spatial, as they are contained within the LLM context. Our study compares existing prompting schemes using the proposed taxonomy, discussing how certain design choices lead to different patterns in performance and cost. We also outline theoretical underpinnings, relationships between prompting and others parts of the LLM ecosystem such as knowledge bases, and the associated research challenges. Our work will help to advance future prompt engineering techniques.

Title: A Quantum "Lifting Theorem" for Constructions of Pseudorandom Generators from Random Oracles

• Authors: Benjamin Sela, Jonathan Katz
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2401.14319
• Pdf URL: https://arxiv.org/pdf/2401.14319
• Copy Paste: [[2401.14319]] A Quantum "Lifting Theorem" for Constructions of Pseudorandom Generators from Random Oracles(https://arxiv.org/abs/2401.14319)
• Keywords: secure, security
• Abstract: We study the (quantum) security of pseudorandom generators (PRGs) constructed from random oracles. We prove a ``lifting theorem'' showing, roughly, that if such a PRG is unconditionally secure against classical adversaries making polynomially many queries to the random oracle, then it is also (unconditionally) secure against quantum adversaries in the same sense. As a result of independent interest, we also show that any pseudo-deterministic quantum-oracle algorithm (i.e., a quantum algorithm that with high probability returns the same value on repeated executions) can be simulated by a computationally unbounded but query bounded classical-oracle algorithm with only a polynomial blowup in the number of queries. This implies as a corollary that our lifting theorem holds even for PRGs that themselves make quantum queries to the random oracle.

Title: Unlocking Past Information: Temporal Embeddings in Cooperative Bird's Eye View Prediction

• Authors: Dominik Rößle, Jeremias Gerner, Klaus Bogenberger, Daniel Cremers, Stefanie Schmidtner, Torsten Schön
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.14325
• Pdf URL: https://arxiv.org/pdf/2401.14325
• Copy Paste: [[2401.14325]] Unlocking Past Information: Temporal Embeddings in Cooperative Bird's Eye View Prediction(https://arxiv.org/abs/2401.14325)
• Keywords: segmentation
• Abstract: Accurate and comprehensive semantic segmentation of Bird's Eye View (BEV) is essential for ensuring safe and proactive navigation in autonomous driving. Although cooperative perception has exceeded the detection capabilities of single-agent systems, prevalent camera-based algorithms in cooperative perception neglect valuable information derived from historical observations. This limitation becomes critical during sensor failures or communication issues as cooperative perception reverts to single-agent perception, leading to degraded performance and incomplete BEV segmentation maps. This paper introduces TempCoBEV, a temporal module designed to incorporate historical cues into current observations, thereby improving the quality and reliability of BEV map segmentations. We propose an importance-guided attention architecture to effectively integrate temporal information that prioritizes relevant properties for BEV map segmentation. TempCoBEV is an independent temporal module that seamlessly integrates into state-of-the-art camera-based cooperative perception models. We demonstrate through extensive experiments on the OPV2V dataset that TempCoBEV performs better than non-temporal models in predicting current and future BEV map segmentations, particularly in scenarios involving communication failures. We show the efficacy of TempCoBEV and its capability to integrate historical cues into the current BEV map, improving predictions under optimal communication conditions by up to 2% and under communication failures by up to 19%. The code will be published on GitHub.

Title: SunBlock: Cloudless Protection for IoT Systems

• Authors: Vadim Safronov, Anna Maria Mandalari, Daniel J. Dubois, David Choffnes, Hamed Haddadi
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.14332
• Pdf URL: https://arxiv.org/pdf/2401.14332
• Copy Paste: [[2401.14332]] SunBlock: Cloudless Protection for IoT Systems(https://arxiv.org/abs/2401.14332)
• Keywords: security, privacy, protect, attack
• Abstract: With an increasing number of Internet of Things (IoT) devices present in homes, there is a rise in the number of potential information leakage channels and their associated security threats and privacy risks. Despite a long history of attacks on IoT devices in unprotected home networks, the problem of accurate, rapid detection and prevention of such attacks remains open. Many existing IoT protection solutions are cloud-based, sometimes ineffective, and might share consumer data with unknown third parties. This paper investigates the potential for effective IoT threat detection locally, on a home router, using AI tools combined with classic rule-based traffic-filtering algorithms. Our results show that with a slight rise of router hardware resources caused by machine learning and traffic filtering logic, a typical home router instrumented with our solution is able to effectively detect risks and protect a typical home IoT network, equaling or outperforming existing popular solutions, without any effects on benign IoT functionality, and without relying on cloud services and third parties.

Title: Progressive Multi-task Anti-Noise Learning and Distilling Frameworks for Fine-grained Vehicle Recognition

• Authors: Dichao Liu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.14336
• Pdf URL: https://arxiv.org/pdf/2401.14336
• Copy Paste: [[2401.14336]] Progressive Multi-task Anti-Noise Learning and Distilling Frameworks for Fine-grained Vehicle Recognition(https://arxiv.org/abs/2401.14336)
• Keywords: noise learning
• Abstract: Fine-grained vehicle recognition (FGVR) is an essential fundamental technology for intelligent transportation systems, but very difficult because of its inherent intra-class variation. Most previous FGVR studies only focus on the intra-class variation caused by different shooting angles, positions, etc., while the intra-class variation caused by image noise has received little attention. This paper proposes a progressive multi-task anti-noise learning (PMAL) framework and a progressive multi-task distilling (PMD) framework to solve the intra-class variation problem in FGVR due to image noise. The PMAL framework achieves high recognition accuracy by treating image denoising as an additional task in image recognition and progressively forcing a model to learn noise invariance. The PMD framework transfers the knowledge of the PMAL-trained model into the original backbone network, which produces a model with about the same recognition accuracy as the PMAL-trained model, but without any additional overheads over the original backbone network. Combining the two frameworks, we obtain models that significantly exceed previous state-of-the-art methods in recognition accuracy on two widely-used, standard FGVR datasets, namely Stanford Cars, and CompCars, as well as three additional surveillance image-based vehicle-type classification datasets, namely Beijing Institute of Technology (BIT)-Vehicle, Vehicle Type Image Data 2 (VTID2), and Vehicle Images Dataset for Make Model Recognition (VIDMMR), without any additional overheads over the original backbone networks. The source code is available at https://github.com/Dichao-Liu/Anti-noise_FGVR

Title: Class-attribute Priors: Adapting Optimization to Heterogeneity and Fairness Objective

• Authors: Xuechen Zhang, Mingchen Li, Jiasi Chen, Christos Thrampoulidis, Samet Oymak
• Subjects: cs.LG, cs.CY, stat.ML
• Abstract URL: https://arxiv.org/abs/2401.14343
• Pdf URL: https://arxiv.org/pdf/2401.14343
• Copy Paste: [[2401.14343]] Class-attribute Priors: Adapting Optimization to Heterogeneity and Fairness Objective(https://arxiv.org/abs/2401.14343)
• Keywords: fair
• Abstract: Modern classification problems exhibit heterogeneities across individual classes: Each class may have unique attributes, such as sample size, label quality, or predictability (easy vs difficult), and variable importance at test-time. Without care, these heterogeneities impede the learning process, most notably, when optimizing fairness objectives. Confirming this, under a gaussian mixture setting, we show that the optimal SVM classifier for balanced accuracy needs to be adaptive to the class attributes. This motivates us to propose CAP: An effective and general method that generates a class-specific learning strategy (e.g. hyperparameter) based on the attributes of that class. This way, optimization process better adapts to heterogeneities. CAP leads to substantial improvements over the naive approach of assigning separate hyperparameters to each class. We instantiate CAP for loss function design and post-hoc logit adjustment, with emphasis on label-imbalanced problems. We show that CAP is competitive with prior art and its flexibility unlocks clear benefits for fairness objectives beyond balanced accuracy. Finally, we evaluate CAP on problems with label noise as well as weighted test objectives to showcase how CAP can jointly adapt to different heterogeneities.

Title: ServerlessLLM: Locality-Enhanced Serverless Inference for Large Language Models

• Authors: Yao Fu, Leyang Xue, Yeqi Huang, Andrei-Octavian Brabete, Dmitrii Ustiugov, Yuvraj Patel, Luo Mai
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2401.14351
• Pdf URL: https://arxiv.org/pdf/2401.14351
• Copy Paste: [[2401.14351]] ServerlessLLM: Locality-Enhanced Serverless Inference for Large Language Models(https://arxiv.org/abs/2401.14351)
• Keywords: large language model
• Abstract: This paper presents ServerlessLLM, a locality-enhanced serverless inference system for Large Language Models (LLMs). ServerlessLLM exploits the substantial capacity and bandwidth of storage and memory devices available on GPU servers, thereby reducing costly remote checkpoint downloads and achieving efficient checkpoint loading. ServerlessLLM achieves this through three main contributions: (i) fast LLM checkpoint loading via a novel loading-optimized checkpoint format design, coupled with an efficient multi-tier checkpoint loading system; (ii) locality-driven LLM inference with live migration, which allows ServerlessLLM to effectively achieve locality-driven server allocation while preserving the low latency of ongoing LLM inference; and (iii) locality-aware server allocation, enabling ServerlessLLM to evaluate the status of each server in a cluster and effectively schedule model startup time to capitalize on local checkpoint placement. Our comprehensive experiments, which include microbenchmarks and real-world traces, show that ServerlessLLM surpasses state-of-the-art systems by 10 - 200X in latency performance when running various LLM inference workloads.

Title: Learning Robust Generalizable Radiance Field with Visibility and Feature Augmented Point Representation

• Authors: Jiaxu Wang, Ziyi Zhang, Renjing Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.14354
• Pdf URL: https://arxiv.org/pdf/2401.14354
• Copy Paste: [[2401.14354]] Learning Robust Generalizable Radiance Field with Visibility and Feature Augmented Point Representation(https://arxiv.org/abs/2401.14354)
• Keywords: robust
• Abstract: This paper introduces a novel paradigm for the generalizable neural radiance field (NeRF). Previous generic NeRF methods combine multiview stereo techniques with image-based neural rendering for generalization, yielding impressive results, while suffering from three issues. First, occlusions often result in inconsistent feature matching. Then, they deliver distortions and artifacts in geometric discontinuities and locally sharp shapes due to their individual process of sampled points and rough feature aggregation. Third, their image-based representations experience severe degradations when source views are not near enough to the target view. To address challenges, we propose the first paradigm that constructs the generalizable neural field based on point-based rather than image-based rendering, which we call the Generalizable neural Point Field (GPF). Our approach explicitly models visibilities by geometric priors and augments them with neural features. We propose a novel nonuniform log sampling strategy to improve both rendering speed and reconstruction quality. Moreover, we present a learnable kernel spatially augmented with features for feature aggregations, mitigating distortions at places with drastically varying geometries. Besides, our representation can be easily manipulated. Experiments show that our model can deliver better geometries, view consistencies, and rendering quality than all counterparts and benchmarks on three datasets in both generalization and finetuning settings, preliminarily proving the potential of the new paradigm for generalizable NeRF.

Title: Genie: Achieving Human Parity in Content-Grounded Datasets Generation

• Authors: Asaf Yehudai, Boaz Carmeli, Yosi Mass, Ofir Arviv, Nathaniel Mills, Assaf Toledo, Eyal Shnarch, Leshem Choshen
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.14367
• Pdf URL: https://arxiv.org/pdf/2401.14367
• Copy Paste: [[2401.14367]] Genie: Achieving Human Parity in Content-Grounded Datasets Generation(https://arxiv.org/abs/2401.14367)
• Keywords: extraction
• Abstract: The lack of high-quality data for content-grounded generation tasks has been identified as a major obstacle to advancing these tasks. To address this gap, we propose Genie, a novel method for automatically generating high-quality content-grounded data. It consists of three stages: (a) Content Preparation, (b) Generation: creating task-specific examples from the content (e.g., question-answer pairs or summaries). (c) Filtering mechanism aiming to ensure the quality and faithfulness of the generated data. We showcase this methodology by generating three large-scale synthetic data, making wishes, for Long-Form Question-Answering (LFQA), summarization, and information extraction. In a human evaluation, our generated data was found to be natural and of high quality. Furthermore, we compare models trained on our data with models trained on human-written data -- ELI5 and ASQA for LFQA and CNN-DailyMail for Summarization. We show that our models are on par with or outperforming models trained on human-generated data and consistently outperforming them in faithfulness. Finally, we applied our method to create LFQA data within the medical domain and compared a model trained on it with models trained on other domains.

Title: UrbanGenAI: Reconstructing Urban Landscapes using Panoptic Segmentation and Diffusion Models

• Authors: Timo Kapsalis
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.14379
• Pdf URL: https://arxiv.org/pdf/2401.14379
• Copy Paste: [[2401.14379]] UrbanGenAI: Reconstructing Urban Landscapes using Panoptic Segmentation and Diffusion Models(https://arxiv.org/abs/2401.14379)
• Keywords: diffusion, generative, segmentation
• Abstract: In contemporary design practices, the integration of computer vision and generative artificial intelligence (genAI) represents a transformative shift towards more interactive and inclusive processes. These technologies offer new dimensions of image analysis and generation, which are particularly relevant in the context of urban landscape reconstruction. This paper presents a novel workflow encapsulated within a prototype application, designed to leverage the synergies between advanced image segmentation and diffusion models for a comprehensive approach to urban design. Our methodology encompasses the OneFormer model for detailed image segmentation and the Stable Diffusion XL (SDXL) diffusion model, implemented through ControlNet, for generating images from textual descriptions. Validation results indicated a high degree of performance by the prototype application, showcasing significant accuracy in both object detection and text-to-image generation. This was evidenced by superior Intersection over Union (IoU) and CLIP scores across iterative evaluations for various categories of urban landscape features. Preliminary testing included utilising UrbanGenAI as an educational tool enhancing the learning experience in design pedagogy, and as a participatory instrument facilitating community-driven urban planning. Early results suggested that UrbanGenAI not only advances the technical frontiers of urban landscape reconstruction but also provides significant pedagogical and participatory planning benefits. The ongoing development of UrbanGenAI aims to further validate its effectiveness across broader contexts and integrate additional features such as real-time feedback mechanisms and 3D modelling capabilities. Keywords: generative AI; panoptic image segmentation; diffusion models; urban landscape design; design pedagogy; co-design

Title: Manifold GCN: Diffusion-based Convolutional Neural Network for Manifold-valued Graphs

• Authors: Martin Hanik, Gabriele Steidl, Christoph von Tycowicz
• Subjects: cs.LG, math.DG
• Abstract URL: https://arxiv.org/abs/2401.14381
• Pdf URL: https://arxiv.org/pdf/2401.14381
• Copy Paste: [[2401.14381]] Manifold GCN: Diffusion-based Convolutional Neural Network for Manifold-valued Graphs(https://arxiv.org/abs/2401.14381)
• Keywords: diffusion
• Abstract: We propose two graph neural network layers for graphs with features in a Riemannian manifold. First, based on a manifold-valued graph diffusion equation, we construct a diffusion layer that can be applied to an arbitrary number of nodes and graph connectivity patterns. Second, we model a tangent multilayer perceptron by transferring ideas from the vector neuron framework to our general setting. Both layers are equivariant with respect to node permutations and isometries of the feature manifold. These properties have been shown to lead to a beneficial inductive bias in many deep learning tasks. Numerical examples on synthetic data as well as on triangle meshes of the right hippocampus to classify Alzheimer's disease demonstrate the very good performance of our layers.

Title: Inconsistency Masks: Removing the Uncertainty from Input-Pseudo-Label Pairs

• Authors: Michael R. H. Vorndran, Bernhard F. Roeck
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.14387
• Pdf URL: https://arxiv.org/pdf/2401.14387
• Copy Paste: [[2401.14387]] Inconsistency Masks: Removing the Uncertainty from Input-Pseudo-Label Pairs(https://arxiv.org/abs/2401.14387)
• Keywords: robust, segmentation
• Abstract: Generating sufficient labeled data is a significant hurdle in the efficient execution of deep learning projects, especially in uncharted territories of image segmentation where labeling demands extensive time, unlike classification tasks. Our study confronts this challenge, operating in an environment constrained by limited hardware resources and the lack of extensive datasets or pre-trained models. We introduce the novel use of Inconsistency Masks (IM) to effectively filter uncertainty in image-pseudo-label pairs, substantially elevating segmentation quality beyond traditional semi-supervised learning techniques. By integrating IM with other methods, we demonstrate remarkable binary segmentation performance on the ISIC 2018 dataset, starting with just 10% labeled data. Notably, three of our hybrid models outperform those trained on the fully labeled dataset. Our approach consistently achieves exceptional results across three additional datasets and shows further improvement when combined with other techniques. For comprehensive and robust evaluation, this paper includes an extensive analysis of prevalent semi-supervised learning strategies, all trained under identical starting conditions. The full code is available at: https://github.com/MichaelVorndran/InconsistencyMasks

Title: Rethinking Patch Dependence for Masked Autoencoders

• Authors: Letian Fu, Long Lian, Renhao Wang, Baifeng Shi, Xudong Wang, Adam Yala, Trevor Darrell, Alexei A. Efros, Ken Goldberg
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.14391
• Pdf URL: https://arxiv.org/pdf/2401.14391
• Copy Paste: [[2401.14391]] Rethinking Patch Dependence for Masked Autoencoders(https://arxiv.org/abs/2401.14391)
• Keywords: segmentation
• Abstract: In this work, we re-examine inter-patch dependencies in the decoding mechanism of masked autoencoders (MAE). We decompose this decoding mechanism for masked patch reconstruction in MAE into self-attention and cross-attention. Our investigations suggest that self-attention between mask patches is not essential for learning good representations. To this end, we propose a novel pretraining framework: Cross-Attention Masked Autoencoders (CrossMAE). CrossMAE's decoder leverages only cross-attention between masked and visible tokens, with no degradation in downstream performance. This design also enables decoding only a small subset of mask tokens, boosting efficiency. Furthermore, each decoder block can now leverage different encoder features, resulting in improved representation learning. CrossMAE matches MAE in performance with 2.5 to 3.7$\times$ less decoding compute. It also surpasses MAE on ImageNet classification and COCO instance segmentation under the same compute. Code and models: https://crossmae.github.io

Title: pix2gestalt: Amodal Segmentation by Synthesizing Wholes

• Authors: Ege Ozguroglu, Ruoshi Liu, Dídac Surís, Dian Chen, Achal Dave, Pavel Tokmakov, Carl Vondrick
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.14398
• Pdf URL: https://arxiv.org/pdf/2401.14398
• Copy Paste: [[2401.14398]] pix2gestalt: Amodal Segmentation by Synthesizing Wholes(https://arxiv.org/abs/2401.14398)
• Keywords: diffusion, segmentation
• Abstract: We introduce pix2gestalt, a framework for zero-shot amodal segmentation, which learns to estimate the shape and appearance of whole objects that are only partially visible behind occlusions. By capitalizing on large-scale diffusion models and transferring their representations to this task, we learn a conditional diffusion model for reconstructing whole objects in challenging zero-shot cases, including examples that break natural and physical priors, such as art. As training data, we use a synthetically curated dataset containing occluded objects paired with their whole counterparts. Experiments show that our approach outperforms supervised baselines on established benchmarks. Our model can furthermore be used to significantly improve the performance of existing object recognition and 3D reconstruction methods in the presence of occlusions.

Title: Deconstructing Denoising Diffusion Models for Self-Supervised Learning

• Authors: Xinlei Chen, Zhuang Liu, Saining Xie, Kaiming He
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.14404
• Pdf URL: https://arxiv.org/pdf/2401.14404
• Copy Paste: [[2401.14404]] Deconstructing Denoising Diffusion Models for Self-Supervised Learning(https://arxiv.org/abs/2401.14404)
• Keywords: diffusion
• Abstract: In this study, we examine the representation learning abilities of Denoising Diffusion Models (DDM) that were originally purposed for image generation. Our philosophy is to deconstruct a DDM, gradually transforming it into a classical Denoising Autoencoder (DAE). This deconstructive procedure allows us to explore how various components of modern DDMs influence self-supervised representation learning. We observe that only a very few modern components are critical for learning good representations, while many others are nonessential. Our study ultimately arrives at an approach that is highly simplified and to a large extent resembles a classical DAE. We hope our study will rekindle interest in a family of classical methods within the realm of modern self-supervised learning.

Title: Multimodal Pathway: Improve Transformers with Irrelevant Data from Other Modalities

• Authors: Yiyuan Zhang, Xiaohan Ding, Kaixiong Gong, Yixiao Ge, Ying Shan, Xiangyu Yue
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.14405
• Pdf URL: https://arxiv.org/pdf/2401.14405
• Copy Paste: [[2401.14405]] Multimodal Pathway: Improve Transformers with Irrelevant Data from Other Modalities(https://arxiv.org/abs/2401.14405)
• Keywords: transformer
• Abstract: We propose to improve transformers of a specific modality with irrelevant data from other modalities, e.g., improve an ImageNet model with audio or point cloud datasets. We would like to highlight that the data samples of the target modality are irrelevant to the other modalities, which distinguishes our method from other works utilizing paired (e.g., CLIP) or interleaved data of different modalities. We propose a methodology named Multimodal Pathway - given a target modality and a transformer designed for it, we use an auxiliary transformer trained with data of another modality and construct pathways to connect components of the two models so that data of the target modality can be processed by both models. In this way, we utilize the universal sequence-to-sequence modeling abilities of transformers obtained from two modalities. As a concrete implementation, we use a modality-specific tokenizer and task-specific head as usual but utilize the transformer blocks of the auxiliary model via a proposed method named Cross-Modal Re-parameterization, which exploits the auxiliary weights without any inference costs. On the image, point cloud, video, and audio recognition tasks, we observe significant and consistent performance improvements with irrelevant data from other modalities. The code and models are available at https://github.com/AILab-CVC/M2PT.