secure

Title: SecureBERT and LLAMA 2 Empowered Control Area Network Intrusion Detection and Classification. (arXiv:2311.12074v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.12074
• Code URL: null
• Copy Paste: [[2311.12074]] SecureBERT and LLAMA 2 Empowered Control Area Network Intrusion Detection and Classification(http://arxiv.org/abs/2311.12074)
• Summary:

  Numerous studies have proved their effective strength in detecting Control Area Network (CAN) attacks. In the realm of understanding the human semantic space, transformer-based models have demonstrated remarkable effectiveness. Leveraging pre-trained transformers has become a common strategy in various language-related tasks, enabling these models to grasp human semantics more comprehensively. To delve into the adaptability evaluation on pre-trained models for CAN intrusion detection, we have developed two distinct models: CAN-SecureBERT and CAN-LLAMA2. Notably, our CAN-LLAMA2 model surpasses the state-of-the-art models by achieving an exceptional performance 0.999993 in terms of balanced accuracy, precision detection rate, F1 score, and a remarkably low false alarm rate of 3.10e-6. Impressively, the false alarm rate is 52 times smaller than that of the leading model, MTH-IDS (Multitiered Hybrid Intrusion Detection System). Our study underscores the promise of employing a Large Language Model as the foundational model, while incorporating adapters for other cybersecurity-related tasks and maintaining the model's inherent language-related capabilities.

security

Title: Security Fence Inspection at Airports Using Object Detection. (arXiv:2311.12064v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12064
• Code URL: null
• Copy Paste: [[2311.12064]] Security Fence Inspection at Airports Using Object Detection(http://arxiv.org/abs/2311.12064)
• Summary:

  To ensure the security of airports, it is essential to protect the airside from unauthorized access. For this purpose, security fences are commonly used, but they require regular inspection to detect damages. However, due to the growing shortage of human specialists and the large manual effort, there is the need for automated methods. The aim is to automatically inspect the fence for damage with the help of an autonomous robot. In this work, we explore object detection methods to address the fence inspection task and localize various types of damages. In addition to evaluating four State-of-the-Art (SOTA) object detection models, we analyze the impact of several design criteria, aiming at adapting to the task-specific challenges. This includes contrast adjustment, optimization of hyperparameters, and utilization of modern backbones. The experimental results indicate that our optimized You Only Look Once v5 (YOLOv5) model achieves the highest accuracy of the four methods with an increase of 6.9% points in Average Precision (AP) compared to the baseline. Moreover, we show the real-time capability of the model. The trained models are published on GitHub: https://github.com/N-Friederich/airport_fence_inspection.

Title: Challenges in Video-Based Infant Action Recognition: A Critical Examination of the State of the Art. (arXiv:2311.12300v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12300
• Code URL: https://github.com/ostadabbas/video-based-infant-action-recognition
• Copy Paste: [[2311.12300]] Challenges in Video-Based Infant Action Recognition: A Critical Examination of the State of the Art(http://arxiv.org/abs/2311.12300)
• Summary:

  Automated human action recognition, a burgeoning field within computer vision, boasts diverse applications spanning surveillance, security, human-computer interaction, tele-health, and sports analysis. Precise action recognition in infants serves a multitude of pivotal purposes, encompassing safety monitoring, developmental milestone tracking, early intervention for developmental delays, fostering parent-infant bonds, advancing computer-aided diagnostics, and contributing to the scientific comprehension of child development. This paper delves into the intricacies of infant action recognition, a domain that has remained relatively uncharted despite the accomplishments in adult action recognition. In this study, we introduce a groundbreaking dataset called ``InfActPrimitive'', encompassing five significant infant milestone action categories, and we incorporate specialized preprocessing for infant data. We conducted an extensive comparative analysis employing cutting-edge skeleton-based action recognition models using this dataset. Our findings reveal that, although the PoseC3D model achieves the highest accuracy at approximately 71%, the remaining models struggle to accurately capture the dynamics of infant actions. This highlights a substantial knowledge gap between infant and adult action recognition domains and the urgent need for data-efficient pipeline models.

Title: Crowd management, crime detection, work monitoring using aiml. (arXiv:2311.12621v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12621
• Code URL: null
• Copy Paste: [[2311.12621]] Crowd management, crime detection, work monitoring using aiml(http://arxiv.org/abs/2311.12621)
• Summary:

  This research endeavors to harness the potential of existing Closed-Circuit Television (CCTV) networks for a comprehensive approach to crowd management, crime prevention, and workplace monitoring through the integration of Artificial Intelligence (AI) and Machine Learning (ML) technologies. The primary objective is to develop and implement advanced algorithms capable of real-time analysis of video feeds, enabling the identification and assessment of crowd dynamics, early detection of potential criminal activities, and continuous monitoring of workplace environments. By leveraging AI/ML, the project aims to optimize surveillance capabilities, thereby enhancing public safety measures and improving organizational productivity. This initiative underscores the transformative impact that intelligent video analytics can have on existing infrastructure, mitigating the need for extensive system overhauls while significantly advancing security and operational efficiency.

Title: SDN-Based Dynamic Cybersecurity Framework of IEC-61850 Communications in Smart Grid. (arXiv:2311.12205v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.12205
• Code URL: null
• Copy Paste: [[2311.12205]] SDN-Based Dynamic Cybersecurity Framework of IEC-61850 Communications in Smart Grid(http://arxiv.org/abs/2311.12205)
• Summary:

  In recent years, critical infrastructure and power grids have experienced a series of cyber-attacks, leading to temporary, widespread blackouts of considerable magnitude. Since most substations are unmanned and have limited physical security protection, cyber breaches into power grid substations present a risk. Nowadays, software-defined network (SDN), a popular virtual network technology based on the OpenFlow protocol is being widely used in the substation automation system. However, the susceptibility of SDN architecture to cyber-attacks has exhibited a notable increase in recent years, as indicated by research findings. This suggests a growing concern regarding the potential for cybersecurity breaches within the SDN framework. In this paper, we propose a hybrid intrusion detection system (IDS)-integrated SDN architecture for detecting and preventing the injection of malicious IEC 61850-based generic object-oriented substation event (GOOSE) messages in a digital substation. Additionally, this program locates the fault's location and, as a form of mitigation, disables a certain port. Furthermore, implementation examples are demonstrated and verified using a hardware-in-the-loop (HIL) testbed that mimics the functioning of a digital substation.

Title: Malicious URL Detection via Pretrained Language Model Guided Multi-Level Feature Attention Network. (arXiv:2311.12372v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.12372
• Code URL: null
• Copy Paste: [[2311.12372]] Malicious URL Detection via Pretrained Language Model Guided Multi-Level Feature Attention Network(http://arxiv.org/abs/2311.12372)
• Summary:

  The widespread use of the Internet has revolutionized information retrieval methods. However, this transformation has also given rise to a significant cybersecurity challenge: the rapid proliferation of malicious URLs, which serve as entry points for a wide range of cyber threats. In this study, we present an efficient pre-training model-based framework for malicious URL detection. Leveraging the subword and character-aware pre-trained model, CharBERT, as our foundation, we further develop three key modules: hierarchical feature extraction, layer-aware attention, and spatial pyramid pooling. The hierarchical feature extraction module follows the pyramid feature learning principle, extracting multi-level URL embeddings from the different Transformer layers of CharBERT. Subsequently, the layer-aware attention module autonomously learns connections among features at various hierarchical levels and allocates varying weight coefficients to each level of features. Finally, the spatial pyramid pooling module performs multiscale downsampling on the weighted multi-level feature pyramid, achieving the capture of local features as well as the aggregation of global features. The proposed method has been extensively validated on multiple public datasets, demonstrating a significant improvement over prior works, with the maximum accuracy gap reaching 8.43% compared to the previous state-of-the-art method. Additionally, we have assessed the model's generalization and robustness in scenarios such as cross-dataset evaluation and adversarial attacks. Finally, we conducted real-world case studies on the active phishing URLs.

Title: Power grid operational risk assessment using graph neural network surrogates. (arXiv:2311.12309v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.12309
• Code URL: null
• Copy Paste: [[2311.12309]] Power grid operational risk assessment using graph neural network surrogates(http://arxiv.org/abs/2311.12309)
• Summary:

  We investigate the utility of graph neural networks (GNNs) as proxies of power grid operational decision-making algorithms (optimal power flow (OPF) and security-constrained unit commitment (SCUC)) to enable rigorous quantification of the operational risk. To conduct principled risk analysis, numerous Monte Carlo (MC) samples are drawn from the (foretasted) probability distributions of spatio-temporally correlated stochastic grid variables. The corresponding OPF and SCUC solutions, which are needed to quantify the risk, are generated using traditional OPF and SCUC solvers to generate data for training GNN model(s). The GNN model performance is evaluated in terms of the accuracy of predicting quantities of interests (QoIs) derived from the decision variables in OPF and SCUC. Specifically, we focus on thermal power generation and load shedding at system and individual zone level. We also perform reliability and risk quantification based on GNN predictions and compare with that obtained from OPF/SCUC solutions. Our results demonstrate that GNNs are capable of providing fast and accurate prediction of QoIs and thus can be good surrogate models for OPF and SCUC. The excellent accuracy of GNN-based reliability and risk assessment further suggests that GNN surrogate has the potential to be applied in real-time and hours-ahead risk quantification.

privacy

Title: KNVQA: A Benchmark for evaluation knowledge-based VQA. (arXiv:2311.12639v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12639
• Code URL: null
• Copy Paste: [[2311.12639]] KNVQA: A Benchmark for evaluation knowledge-based VQA(http://arxiv.org/abs/2311.12639)
• Summary:

  Within the multimodal field, large vision-language models (LVLMs) have made significant progress due to their strong perception and reasoning capabilities in the visual and language systems. However, LVLMs are still plagued by the two critical issues of object hallucination and factual accuracy, which limit the practicality of LVLMs in different scenarios. Furthermore, previous evaluation methods focus more on the comprehension and reasoning of language content but lack a comprehensive evaluation of multimodal interactions, thereby resulting in potential limitations. To this end, we propose a novel KNVQA-Eval, which is devoted to knowledge-based VQA task evaluation to reflect the factuality of multimodal LVLMs. To ensure the robustness and scalability of the evaluation, we develop a new KNVQA dataset by incorporating human judgment and perception, aiming to evaluate the accuracy of standard answers relative to AI-generated answers in knowledge-based VQA. This work not only comprehensively evaluates the contextual information of LVLMs using reliable human annotations, but also further analyzes the fine-grained capabilities of current methods to reveal potential avenues for subsequent optimization of LVLMs-based estimators. Our proposed VQA-Eval and corresponding dataset KNVQA will facilitate the development of automatic evaluation tools with the advantages of low cost, privacy protection, and reproducibility. Our code will be released upon publication.

Title: Characterizing Browser Fingerprinting and its Mitigations. (arXiv:2311.12197v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.12197
• Code URL: null
• Copy Paste: [[2311.12197]] Characterizing Browser Fingerprinting and its Mitigations(http://arxiv.org/abs/2311.12197)
• Summary:

  People are becoming increasingly concerned with their online privacy, especially with how advertising companies track them across websites (a practice called cross-site tracking), as reconstructing a user's browser history can reveal sensitive information. Recent legislation like the General Data Protection Regulation (GDPR) and the California Consumer Privacy Act have tried to limit the extent to which third parties perform cross-site tracking, and browsers have also made tracking more difficult by deprecating the most-common tracking mechanism: third-party cookies. However, online advertising companies continue to track users through other mechanisms that do not rely on cookies. This work explores one of these tracking techniques: browser fingerprinting. We detail how browser fingerprinting works, how prevalent it is, and what defenses can mitigate it.

Title: Heuristics for Detecting CoinJoin Transactions on the Bitcoin Blockchain. (arXiv:2311.12491v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.12491
• Code URL: null
• Copy Paste: [[2311.12491]] Heuristics for Detecting CoinJoin Transactions on the Bitcoin Blockchain(http://arxiv.org/abs/2311.12491)
• Summary:

  This research delves into the intricacies of Bitcoin, a decentralized peer-to-peer network, and its associated blockchain, which records all transactions since its inception. While this ensures integrity and transparency, the transparent nature of Bitcoin potentially compromises users' privacy rights. To address this concern, users have adopted CoinJoin, a method that amalgamates multiple transaction intents into a single, larger transaction to bolster transactional privacy. This process complicates individual transaction tracing and disrupts many established blockchain analysis heuristics. Despite its significance, limited research has been conducted on identifying CoinJoin transactions. Particularly noteworthy are varied CoinJoin implementations such as JoinMarket, Wasabi, and Whirlpool, each presenting distinct challenges due to their unique transaction structures. This study delves deeply into the open-source implementations of these protocols, aiming to develop refined heuristics for identifying their transactions on the blockchain. Our exhaustive analysis covers transactions up to block 760,000, offering a comprehensive insight into CoinJoin transactions and their implications for Bitcoin blockchain analysis.

Title: Managing ML-Based Application Non-Functional Behavior: A Multi-Model Approach. (arXiv:2311.12686v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.12686
• Code URL: null
• Copy Paste: [[2311.12686]] Managing ML-Based Application Non-Functional Behavior: A Multi-Model Approach(http://arxiv.org/abs/2311.12686)
• Summary:

  Modern applications are increasingly driven by Machine Learning (ML) models whose non-deterministic behavior is affecting the entire application life cycle from design to operation. The pervasive adoption of ML is urgently calling for approaches that guarantee a stable non-functional behavior of ML-based applications over time and across model changes. To this aim, non-functional properties of ML models, such as privacy, confidentiality, fairness, and explainability, must be monitored, verified, and maintained. This need is even more pressing when modern applications operate in the edge-cloud continuum, increasing their complexity and dynamicity. Existing approaches mostly focus on i) implementing classifier selection solutions according to the functional behavior of ML models, ii) finding new algorithmic solutions to this need, such as continuous re-training. In this paper, we propose a multi-model approach built on dynamic classifier selection, where multiple ML models showing similar non-functional properties are made available to the application and one model is selected over time according to (dynamic and unpredictable) contextual changes. Our solution goes beyond the state of the art by providing an architectural and methodological approach that continuously guarantees a stable non-functional behavior of ML-based applications, is applicable to different ML models, and is driven by non-functional properties assessed on the models themselves. It consists of a two-step process working during application operation, where model assessment verifies non-functional properties of ML models trained and selected at development time, and model substitution guarantees a continuous and stable support of non-functional properties. We experimentally evaluate our solution in a real-world scenario focusing on non-functional property fairness.

protect

Title: Kuro Siwo: 12.1 billion $m^2$ under the water. A global multi-temporal satellite dataset for rapid flood mapping. (arXiv:2311.12056v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12056
• Code URL: null
• Copy Paste: [[2311.12056]] Kuro Siwo: 12(http://arxiv.org/abs/2311.12056)
• Summary:

  Global floods, exacerbated by climate change, pose severe threats to human life, infrastructure, and the environment. This urgency is highlighted by recent catastrophic events in Pakistan and New Zealand, underlining the critical need for precise flood mapping for guiding restoration efforts, understanding vulnerabilities, and preparing for future events. While Synthetic Aperture Radar (SAR) offers day-and-night, all-weather imaging capabilities, harnessing it for deep learning is hindered by the absence of a large annotated dataset. To bridge this gap, we introduce Kuro Siwo, a meticulously curated multi-temporal dataset, spanning 32 flood events globally. Our dataset maps more than 63 billion m2 of land, with 12.1 billion of them being either a flooded area or a permanent water body. Kuro Siwo stands out for its unparalleled annotation quality to facilitate rapid flood mapping in a supervised setting. We also augment learning by including a large unlabeled set of SAR samples, aimed at self-supervised pretraining. We provide an extensive benchmark and strong baselines for a diverse set of flood events from Europe, America, Africa and Australia. Our benchmark demonstrates the quality of Kuro Siwo annotations, training models that can achieve $\approx$ 85% and $\approx$ 87% in F1-score for flooded areas and general water detection respectively. This work calls on the deep learning community to develop solution-driven algorithms for rapid flood mapping, with the potential to aid civil protection and humanitarian agencies amid climate change challenges. Our code and data will be made available at https://github.com/Orion-AI-Lab/KuroSiwo

Title: Towards Function Space Mesh Watermarking: Protecting the Copyright of Signed Distance Fields. (arXiv:2311.12059v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12059
• Code URL: null
• Copy Paste: [[2311.12059]] Towards Function Space Mesh Watermarking: Protecting the Copyright of Signed Distance Fields(http://arxiv.org/abs/2311.12059)
• Summary:

  The signed distance field (SDF) represents 3D geometries in continuous function space. Due to its continuous nature, explicit 3D models (e.g., meshes) can be extracted from it at arbitrary resolution, which means losing the SDF is equivalent to losing the mesh. Recent research has shown meshes can also be extracted from SDF-enhanced neural radiance fields (NeRF). Such a signal raises an alarm that any implicit neural representation with SDF enhancement can extract the original mesh, which indicates identifying the SDF's intellectual property becomes an urgent issue. This paper proposes FuncMark, a robust and invisible watermarking method to protect the copyright of signed distance fields by leveraging analytic on-surface deformations to embed binary watermark messages. Such deformation can survive isosurfacing and thus be inherited by the extracted meshes for further watermark message decoding. Our method can recover the message with high-resolution meshes extracted from SDFs and detect the watermark even when mesh vertices are extremely sparse. Furthermore, our method is robust even when various distortions (including remeshing) are encountered. Extensive experiments demonstrate that our \tool significantly outperforms state-of-the-art approaches and the message is still detectable even when only 50 vertex samples are given.

Title: EditShield: Protecting Unauthorized Image Editing by Instruction-guided Diffusion Models. (arXiv:2311.12066v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.12066
• Code URL: null
• Copy Paste: [[2311.12066]] EditShield: Protecting Unauthorized Image Editing by Instruction-guided Diffusion Models(http://arxiv.org/abs/2311.12066)
• Summary:

  Text-to-image diffusion models have emerged as an evolutionary for producing creative content in image synthesis. Based on the impressive generation abilities of these models, instruction-guided diffusion models can edit images with simple instructions and input images. While they empower users to obtain their desired edited images with ease, they have raised concerns about unauthorized image manipulation. Prior research has delved into the unauthorized use of personalized diffusion models; however, this problem of instruction-guided diffusion models remains largely unexplored. In this paper, we first propose a protection method EditShield against unauthorized modifications from such models. Specifically, EditShield works by adding imperceptible perturbations that can shift the latent representation used in the diffusion process, forcing models to generate unrealistic images with mismatched subjects. Our extensive experiments demonstrate EditShield's effectiveness among synthetic and real-world datasets. Besides, EditShield also maintains robustness against various editing types and synonymous instruction phrases.

defense

Title: ODDR: Outlier Detection & Dimension Reduction Based Defense Against Adversarial Patches. (arXiv:2311.12084v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.12084
• Code URL: null
• Copy Paste: [[2311.12084]] ODDR: Outlier Detection & Dimension Reduction Based Defense Against Adversarial Patches(http://arxiv.org/abs/2311.12084)
• Summary:

  Adversarial attacks are a major deterrent towards the reliable use of machine learning models. A powerful type of adversarial attacks is the patch-based attack, wherein the adversarial perturbations modify localized patches or specific areas within the images to deceive the trained machine learning model. In this paper, we introduce Outlier Detection and Dimension Reduction (ODDR), a holistic defense mechanism designed to effectively mitigate patch-based adversarial attacks. In our approach, we posit that input features corresponding to adversarial patches, whether naturalistic or otherwise, deviate from the inherent distribution of the remaining image sample and can be identified as outliers or anomalies. ODDR employs a three-stage pipeline: Fragmentation, Segregation, and Neutralization, providing a model-agnostic solution applicable to both image classification and object detection tasks. The Fragmentation stage parses the samples into chunks for the subsequent Segregation process. Here, outlier detection techniques identify and segregate the anomalous features associated with adversarial perturbations. The Neutralization stage utilizes dimension reduction methods on the outliers to mitigate the impact of adversarial perturbations without sacrificing pertinent information necessary for the machine learning task. Extensive testing on benchmark datasets and state-of-the-art adversarial patches demonstrates the effectiveness of ODDR. Results indicate robust accuracies matching and lying within a small range of clean accuracies (1%-3% for classification and 3%-5% for object detection), with only a marginal compromise of 1%-2% in performance on clean samples, thereby significantly outperforming other defenses.

Title: DefensiveDR: Defending against Adversarial Patches using Dimensionality Reduction. (arXiv:2311.12211v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.12211
• Code URL: null
• Copy Paste: [[2311.12211]] DefensiveDR: Defending against Adversarial Patches using Dimensionality Reduction(http://arxiv.org/abs/2311.12211)
• Summary:

  Adversarial patch-based attacks have shown to be a major deterrent towards the reliable use of machine learning models. These attacks involve the strategic modification of localized patches or specific image areas to deceive trained machine learning models. In this paper, we propose \textit{DefensiveDR}, a practical mechanism using a dimensionality reduction technique to thwart such patch-based attacks. Our method involves projecting the sample images onto a lower-dimensional space while retaining essential information or variability for effective machine learning tasks. We perform this using two techniques, Singular Value Decomposition and t-Distributed Stochastic Neighbor Embedding. We experimentally tune the variability to be preserved for optimal performance as a hyper-parameter. This dimension reduction substantially mitigates adversarial perturbations, thereby enhancing the robustness of the given machine learning model. Our defense is model-agnostic and operates without assumptions about access to model decisions or model architectures, making it effective in both black-box and white-box settings. Furthermore, it maintains accuracy across various models and remains robust against several unseen patch-based attacks. The proposed defensive approach improves the accuracy from 38.8\% (without defense) to 66.2\% (with defense) when performing LaVAN and GoogleAp attacks, which supersedes that of the prominent state-of-the-art like LGS (53.86\%) and Jujutsu (60\%).

attack

Title: Boost Adversarial Transferability by Uniform Scale and Mix Mask Method. (arXiv:2311.12051v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12051
• Code URL: null
• Copy Paste: [[2311.12051]] Boost Adversarial Transferability by Uniform Scale and Mix Mask Method(http://arxiv.org/abs/2311.12051)
• Summary:

  Adversarial examples generated from surrogate models often possess the ability to deceive other black-box models, a property known as transferability. Recent research has focused on enhancing adversarial transferability, with input transformation being one of the most effective approaches. However, existing input transformation methods suffer from two issues. Firstly, certain methods, such as the Scale-Invariant Method, employ exponentially decreasing scale invariant parameters that decrease the adaptability in generating effective adversarial examples across multiple scales. Secondly, most mixup methods only linearly combine candidate images with the source image, leading to reduced features blending effectiveness. To address these challenges, we propose a framework called Uniform Scale and Mix Mask Method (US-MM) for adversarial example generation. The Uniform Scale approach explores the upper and lower boundaries of perturbation with a linear factor, minimizing the negative impact of scale copies. The Mix Mask method introduces masks into the mixing process in a nonlinear manner, significantly improving the effectiveness of mixing strategies. Ablation experiments are conducted to validate the effectiveness of each component in US-MM and explore the effect of hyper-parameters. Empirical evaluations on standard ImageNet datasets demonstrate that US-MM achieves an average of 7% better transfer attack success rate compared to state-of-the-art methods.

Title: BadCLIP: Dual-Embedding Guided Backdoor Attack on Multimodal Contrastive Learning. (arXiv:2311.12075v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12075
• Code URL: null
• Copy Paste: [[2311.12075]] BadCLIP: Dual-Embedding Guided Backdoor Attack on Multimodal Contrastive Learning(http://arxiv.org/abs/2311.12075)
• Summary:

  Studying backdoor attacks is valuable for model copyright protection and enhancing defenses. While existing backdoor attacks have successfully infected multimodal contrastive learning models such as CLIP, they can be easily countered by specialized backdoor defenses for MCL models. This paper reveals the threats in this practical scenario that backdoor attacks can remain effective even after defenses and introduces the \emph{\toolns} attack, which is resistant to backdoor detection and model fine-tuning defenses. To achieve this, we draw motivations from the perspective of the Bayesian rule and propose a dual-embedding guided framework for backdoor attacks. Specifically, we ensure that visual trigger patterns approximate the textual target semantics in the embedding space, making it challenging to detect the subtle parameter variations induced by backdoor learning on such natural trigger patterns. Additionally, we optimize the visual trigger patterns to align the poisoned samples with target vision features in order to hinder the backdoor unlearning through clean fine-tuning. Extensive experiments demonstrate that our attack significantly outperforms state-of-the-art baselines (+45.3% ASR) in the presence of SoTA backdoor defenses, rendering these mitigation and detection strategies virtually ineffective. Furthermore, our approach effectively attacks some more rigorous scenarios like downstream tasks. We believe that this paper raises awareness regarding the potential threats associated with the practical application of multimodal contrastive learning and encourages the development of more robust defense mechanisms.

Title: Investigating Weight-Perturbed Deep Neural Networks With Application in Iris Presentation Attack Detection. (arXiv:2311.12764v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12764
• Code URL: https://github.com/redwankarimsony/weightperturbation-msu
• Copy Paste: [[2311.12764]] Investigating Weight-Perturbed Deep Neural Networks With Application in Iris Presentation Attack Detection(http://arxiv.org/abs/2311.12764)
• Summary:

  Deep neural networks (DNNs) exhibit superior performance in various machine learning tasks, e.g., image classification, speech recognition, biometric recognition, object detection, etc. However, it is essential to analyze their sensitivity to parameter perturbations before deploying them in real-world applications. In this work, we assess the sensitivity of DNNs against perturbations to their weight and bias parameters. The sensitivity analysis involves three DNN architectures (VGG, ResNet, and DenseNet), three types of parameter perturbations (Gaussian noise, weight zeroing, and weight scaling), and two settings (entire network and layer-wise). We perform experiments in the context of iris presentation attack detection and evaluate on two publicly available datasets: LivDet-Iris-2017 and LivDet-Iris-2020. Based on the sensitivity analysis, we propose improved models simply by perturbing parameters of the network without undergoing training. We further combine these perturbed models at the score-level and at the parameter-level to improve the performance over the original model. The ensemble at the parameter-level shows an average improvement of 43.58% on the LivDet-Iris-2017 dataset and 9.25% on the LivDet-Iris-2020 dataset. The source code is available at \href{https://github.com/redwankarimsony/WeightPerturbation-MSU}{https://github.com/redwankarimsony/WeightPerturbation-MSU}.

Title: Iris Presentation Attack: Assessing the Impact of Combining Vanadium Dioxide Films with Artificial Eyes. (arXiv:2311.12773v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12773
• Code URL: null
• Copy Paste: [[2311.12773]] Iris Presentation Attack: Assessing the Impact of Combining Vanadium Dioxide Films with Artificial Eyes(http://arxiv.org/abs/2311.12773)
• Summary:

  Iris recognition systems, operating in the near infrared spectrum (NIR), have demonstrated vulnerability to presentation attacks, where an adversary uses artifacts such as cosmetic contact lenses, artificial eyes or printed iris images in order to circumvent the system. At the same time, a number of effective presentation attack detection (PAD) methods have been developed. These methods have demonstrated success in detecting artificial eyes (e.g., fake Van Dyke eyes) as presentation attacks. In this work, we seek to alter the optical characteristics of artificial eyes by affixing Vanadium Dioxide (VO2) films on their surface in various spatial configurations. VO2 films can be used to selectively transmit NIR light and can, therefore, be used to regulate the amount of NIR light from the object that is captured by the iris sensor. We study the impact of such images produced by the sensor on two state-of-the-art iris PA detection methods. We observe that the addition of VO2 films on the surface of artificial eyes can cause the PA detection methods to misclassify them as bonafide eyes in some cases. This represents a vulnerability that must be systematically analyzed and effectively addressed.

Title: Attacks of fairness in Federated Learning. (arXiv:2311.12715v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.12715
• Code URL: https://github.com/slkdfjslkjfd/fl_fairness_attacks
• Copy Paste: [[2311.12715]] Attacks of fairness in Federated Learning(http://arxiv.org/abs/2311.12715)
• Summary:

  Federated Learning is an important emerging distributed training paradigm that keeps data private on clients. It is now well understood that by controlling only a small subset of FL clients, it is possible to introduce a backdoor to a federated learning model, in the presence of certain attributes. In this paper, we present a new type of attack that compromises the fairness of the trained model. Fairness is understood to be the attribute-level performance distribution of a trained model. It is particularly salient in domains where, for example, skewed accuracy discrimination between subpopulations could have disastrous consequences. We find that by employing a threat model similar to that of a backdoor attack, an attacker is able to influence the aggregated model to have an unfair performance distribution between any given set of attributes. Furthermore, we find that this attack is possible by controlling only a single client. While combating naturally induced unfairness in FL has previously been discussed in depth, its artificially induced kind has been neglected. We show that defending against attacks on fairness should be a critical consideration in any situation where unfairness in a trained model could benefit a user who participated in its training.

robust

Title: Automated Detection of hidden Damages and Impurities in Aluminum Die Casting Materials and Fibre-Metal Laminates using Low-quality X-ray Radiography, Synthetic X-ray Data Augmentation by Simulation, and Machine Learning. (arXiv:2311.12041v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12041
• Code URL: null
• Copy Paste: [[2311.12041]] Automated Detection of hidden Damages and Impurities in Aluminum Die Casting Materials and Fibre-Metal Laminates using Low-quality X-ray Radiography, Synthetic X-ray Data Augmentation by Simulation, and Machine Learning(http://arxiv.org/abs/2311.12041)
• Summary:

  Detection and characterization of hidden defects, impurities, and damages in layered composites like Fibre laminates, e.g., Fibre Metal Laminates (FML), as well as in monolithic materials, e.g., aluminum die casting materials, is still a challenge. This work discusses methods and challenges in data-driven modeling of automated damage and defect detectors using X-ray single- and multi-projection (CT) images. Three main issues are identified: Data and feature variance, data feature labeling (for supervised machine learning), and the missing ground truth. It will be shown that only simulation of data can deliver a ground truth data set and accurate labeling. Noise has significant impact on the feature detection and will be discussed. Data-driven feature detectors are implemented with semantic pixel- or z-profile Convolutional Neural Networks and LSTM Auto-encoders. Data is measured with three different devices: A low-quality and low-cost (Low-Q), a mid- and a high-quality (micro-CT, Mid-/High-Q) device. The goals of this work are the training of robust and generalized feature detectors with synthetic data and the transition from High- and Mid-Q laboratory measuring technologies towards in-field usable technologies and methods.

Title: MagicDance: Realistic Human Dance Video Generation with Motions & Facial Expressions Transfer. (arXiv:2311.12052v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12052
• Code URL: https://github.com/boese0601/magicdance
• Copy Paste: [[2311.12052]] MagicDance: Realistic Human Dance Video Generation with Motions & Facial Expressions Transfer(http://arxiv.org/abs/2311.12052)
• Summary:

  In this work, we propose MagicDance, a diffusion-based model for 2D human motion and facial expression transfer on challenging human dance videos. Specifically, we aim to generate human dance videos of any target identity driven by novel pose sequences while keeping the identity unchanged. To this end, we propose a two-stage training strategy to disentangle human motions and appearance (e.g., facial expressions, skin tone and dressing), consisting of the pretraining of an appearance-control block and fine-tuning of an appearance-pose-joint-control block over human dance poses of the same dataset. Our novel design enables robust appearance control with temporally consistent upper body, facial attributes, and even background. The model also generalizes well on unseen human identities and complex motion sequences without the need for any fine-tuning with additional data with diverse human attributes by leveraging the prior knowledge of image diffusion models. Moreover, the proposed model is easy to use and can be considered as a plug-in module/extension to Stable Diffusion. We also demonstrate the model's ability for zero-shot 2D animation generation, enabling not only the appearance transfer from one identity to another but also allowing for cartoon-like stylization given only pose inputs. Extensive experiments demonstrate our superior performance on the TikTok dataset.

Title: FreeKD: Knowledge Distillation via Semantic Frequency Prompt. (arXiv:2311.12079v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12079
• Code URL: null
• Copy Paste: [[2311.12079]] FreeKD: Knowledge Distillation via Semantic Frequency Prompt(http://arxiv.org/abs/2311.12079)
• Summary:

  Knowledge distillation (KD) has been applied to various tasks successfully, and mainstream methods typically boost the student model via spatial imitation losses. However, the consecutive downsamplings induced in the spatial domain of teacher model is a type of corruption, hindering the student from analyzing what specific information needs to be imitated, which results in accuracy degradation. To better understand the underlying pattern of corrupted feature maps, we shift our attention to the frequency domain. During frequency distillation, we encounter a new challenge: the low-frequency bands convey general but minimal context, while the high are more informative but also introduce noise. Not each pixel within the frequency bands contributes equally to the performance. To address the above problem: (1) We propose the Frequency Prompt plugged into the teacher model, absorbing the semantic frequency context during finetuning. (2) During the distillation period, a pixel-wise frequency mask is generated via Frequency Prompt, to localize those pixel of interests (PoIs) in various frequency bands. Additionally, we employ a position-aware relational frequency loss for dense prediction tasks, delivering a high-order spatial enhancement to the student model. We dub our Frequency Knowledge Distillation method as FreeKD, which determines the optimal localization and extent for the frequency distillation. Extensive experiments demonstrate that FreeKD not only outperforms spatial-based distillation methods consistently on dense prediction tasks (e.g., FreeKD brings 3.8 AP gains for RepPoints-R50 on COCO2017 and 4.55 mIoU gains for PSPNet-R18 on Cityscapes), but also conveys more robustness to the student. Notably, we also validate the generalization of our approach on large-scale vision models (e.g., DINO and SAM).

Title: LIP-RTVE: An Audiovisual Database for Continuous Spanish in the Wild. (arXiv:2311.12457v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12457
• Code URL: https://github.com/david-gimeno/lip-rtve
• Copy Paste: [[2311.12457]] LIP-RTVE: An Audiovisual Database for Continuous Spanish in the Wild(http://arxiv.org/abs/2311.12457)
• Summary:

  Speech is considered as a multi-modal process where hearing and vision are two fundamentals pillars. In fact, several studies have demonstrated that the robustness of Automatic Speech Recognition systems can be improved when audio and visual cues are combined to represent the nature of speech. In addition, Visual Speech Recognition, an open research problem whose purpose is to interpret speech by reading the lips of the speaker, has been a focus of interest in the last decades. Nevertheless, in order to estimate these systems in the currently Deep Learning era, large-scale databases are required. On the other hand, while most of these databases are dedicated to English, other languages lack sufficient resources. Thus, this paper presents a semi-automatically annotated audiovisual database to deal with unconstrained natural Spanish, providing 13 hours of data extracted from Spanish television. Furthermore, baseline results for both speaker-dependent and speaker-independent scenarios are reported using Hidden Markov Models, a traditional paradigm that has been widely used in the field of Speech Technologies.

Title: TouchSDF: A DeepSDF Approach for 3D Shape Reconstruction using Vision-Based Tactile Sensing. (arXiv:2311.12602v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12602
• Code URL: null
• Copy Paste: [[2311.12602]] TouchSDF: A DeepSDF Approach for 3D Shape Reconstruction using Vision-Based Tactile Sensing(http://arxiv.org/abs/2311.12602)
• Summary:

  Humans rely on their visual and tactile senses to develop a comprehensive 3D understanding of their physical environment. Recently, there has been a growing interest in exploring and manipulating objects using data-driven approaches that utilise high-resolution vision-based tactile sensors. However, 3D shape reconstruction using tactile sensing has lagged behind visual shape reconstruction because of limitations in existing techniques, including the inability to generalise over unseen shapes, the absence of real-world testing, and limited expressive capacity imposed by discrete representations. To address these challenges, we propose TouchSDF, a Deep Learning approach for tactile 3D shape reconstruction that leverages the rich information provided by a vision-based tactile sensor and the expressivity of the implicit neural representation DeepSDF. Our technique consists of two components: (1) a Convolutional Neural Network that maps tactile images into local meshes representing the surface at the touch location, and (2) an implicit neural function that predicts a signed distance function to extract the desired 3D shape. This combination allows TouchSDF to reconstruct smooth and continuous 3D shapes from tactile inputs in simulation and real-world settings, opening up research avenues for robust 3D-aware representations and improved multimodal perception in robotics. Code and supplementary material are available at: https://touchsdf.github.io/

Title: Similar Document Template Matching Algorithm. (arXiv:2311.12663v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12663
• Code URL: null
• Copy Paste: [[2311.12663]] Similar Document Template Matching Algorithm(http://arxiv.org/abs/2311.12663)
• Summary:

  This study outlines a comprehensive methodology for verifying medical documents, integrating advanced techniques in template extraction, comparison, and fraud detection. It begins with template extraction using sophisticated region-of-interest (ROI) methods, incorporating contour analysis and edge identification. Pre-processing steps ensure template clarity through morphological operations and adaptive thresholding. The template comparison algorithm utilizes advanced feature matching with key points and descriptors, enhancing robustness through histogram-based analysis for accounting variations. Fraud detection involves the SSIM computation and OCR for textual information extraction. The SSIM quantifies structural similarity, aiding in potential match identification. OCR focuses on critical areas like patient details, provider information, and billing amounts. Extracted information is compared with a reference dataset, and confidence thresholding ensures reliable fraud detection. Adaptive parameters enhance system flexibility for dynamic adjustments to varying document layouts. This methodology provides a robust approach to medical document verification, addressing complexities in template extraction, comparison, fraud detection, and adaptability to diverse document structures.

Title: BundleMoCap: Efficient, Robust and Smooth Motion Capture from Sparse Multiview Videos. (arXiv:2311.12679v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12679
• Code URL: null
• Copy Paste: [[2311.12679]] BundleMoCap: Efficient, Robust and Smooth Motion Capture from Sparse Multiview Videos(http://arxiv.org/abs/2311.12679)
• Summary:

  Capturing smooth motions from videos using markerless techniques typically involves complex processes such as temporal constraints, multiple stages with data-driven regression and optimization, and bundle solving over temporal windows. These processes can be inefficient and require tuning multiple objectives across stages. In contrast, BundleMoCap introduces a novel and efficient approach to this problem. It solves the motion capture task in a single stage, eliminating the need for temporal smoothness objectives while still delivering smooth motions. BundleMoCap outperforms the state-of-the-art without increasing complexity. The key concept behind BundleMoCap is manifold interpolation between latent keyframes. By relying on a local manifold smoothness assumption, we can efficiently solve a bundle of frames using a single code. Additionally, the method can be implemented as a sliding window optimization and requires only the first frame to be properly initialized, reducing the overall computational burden. BundleMoCap's strength lies in its ability to achieve high-quality motion capture results with simplicity and efficiency. More details can be found at https://moverseai.github.io/bundle/.

Title: Towards Natural Language-Guided Drones: GeoText-1652 Benchmark with Spatially Relation Matching. (arXiv:2311.12751v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12751
• Code URL: null
• Copy Paste: [[2311.12751]] Towards Natural Language-Guided Drones: GeoText-1652 Benchmark with Spatially Relation Matching(http://arxiv.org/abs/2311.12751)
• Summary:

  Drone navigation through natural language commands remains a significant challenge due to the lack of publicly available multi-modal datasets and the intricate demands of fine-grained visual-text alignment. In response to this pressing need, we present a new human-computer interaction annotation benchmark called GeoText-1652, meticulously curated through a robust Large Language Model (LLM)-based data generation framework and the expertise of pre-trained vision models. This new dataset seamlessly extends the existing image dataset, \ie, University-1652, with spatial-aware text annotations, encompassing intricate image-text-bounding box associations. Besides, we introduce a new optimization objective to leverage fine-grained spatial associations, called blending spatial matching, for region-level spatial relation matching. Extensive experiments reveal that our approach maintains an exceptional recall rate under varying description complexities. This underscores the promising potential of our approach in elevating drone control and navigation through the seamless integration of natural language commands in real-world scenarios.

Title: SelfOcc: Self-Supervised Vision-Based 3D Occupancy Prediction. (arXiv:2311.12754v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12754
• Code URL: https://github.com/huang-yh/selfocc
• Copy Paste: [[2311.12754]] SelfOcc: Self-Supervised Vision-Based 3D Occupancy Prediction(http://arxiv.org/abs/2311.12754)
• Summary:

  3D occupancy prediction is an important task for the robustness of vision-centric autonomous driving, which aims to predict whether each point is occupied in the surrounding 3D space. Existing methods usually require 3D occupancy labels to produce meaningful results. However, it is very laborious to annotate the occupancy status of each voxel. In this paper, we propose SelfOcc to explore a self-supervised way to learn 3D occupancy using only video sequences. We first transform the images into the 3D space (e.g., bird's eye view) to obtain 3D representation of the scene. We directly impose constraints on the 3D representations by treating them as signed distance fields. We can then render 2D images of previous and future frames as self-supervision signals to learn the 3D representations. We propose an MVS-embedded strategy to directly optimize the SDF-induced weights with multiple depth proposals. Our SelfOcc outperforms the previous best method SceneRF by 58.7% using a single frame as input on SemanticKITTI and is the first self-supervised work that produces reasonable 3D occupancy for surround cameras on Occ3D. SelfOcc produces high-quality depth and achieves state-of-the-art results on novel depth synthesis, monocular depth estimation, and surround-view depth estimation on the SemanticKITTI, KITTI-2015, and nuScenes, respectively. Code: https://github.com/huang-yh/SelfOcc.

Title: Beyond Turing: A Comparative Analysis of Approaches for Detecting Machine-Generated Text. (arXiv:2311.12373v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.12373
• Code URL: null
• Copy Paste: [[2311.12373]] Beyond Turing: A Comparative Analysis of Approaches for Detecting Machine-Generated Text(http://arxiv.org/abs/2311.12373)
• Summary:

  Significant progress has been made on text generation by pre-trained language models (PLMs), yet distinguishing between human and machine-generated text poses an escalating challenge. This paper offers an in-depth evaluation of three distinct methods used to address this task: traditional shallow learning, Language Model (LM) fine-tuning, and Multilingual Model fine-tuning. These approaches are rigorously tested on a wide range of machine-generated texts, providing a benchmark of their competence in distinguishing between human-authored and machine-authored linguistic constructs. The results reveal considerable differences in performance across methods, thus emphasizing the continued need for advancement in this crucial area of NLP. This study offers valuable insights and paves the way for future research aimed at creating robust and highly discriminative models.

Title: IndoRobusta: Towards Robustness Against Diverse Code-Mixed Indonesian Local Languages. (arXiv:2311.12405v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.12405
• Code URL: null
• Copy Paste: [[2311.12405]] IndoRobusta: Towards Robustness Against Diverse Code-Mixed Indonesian Local Languages(http://arxiv.org/abs/2311.12405)
• Summary:

  Significant progress has been made on Indonesian NLP. Nevertheless, exploration of the code-mixing phenomenon in Indonesian is limited, despite many languages being frequently mixed with Indonesian in daily conversation. In this work, we explore code-mixing in Indonesian with four embedded languages, i.e., English, Sundanese, Javanese, and Malay; and introduce IndoRobusta, a framework to evaluate and improve the code-mixing robustness. Our analysis shows that the pre-training corpus bias affects the model's ability to better handle Indonesian-English code-mixing when compared to other local languages, despite having higher language diversity.

Title: Masked Autoencoders Are Robust Neural Architecture Search Learners. (arXiv:2311.12086v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.12086
• Code URL: null
• Copy Paste: [[2311.12086]] Masked Autoencoders Are Robust Neural Architecture Search Learners(http://arxiv.org/abs/2311.12086)
• Summary:

  Neural Architecture Search (NAS) currently relies heavily on labeled data, which is both expensive and time-consuming to acquire. In this paper, we propose a novel NAS framework based on Masked Autoencoders (MAE) that eliminates the need for labeled data during the search process. By replacing the supervised learning objective with an image reconstruction task, our approach enables the robust discovery of network architectures without compromising performance and generalization ability. Additionally, we address the problem of performance collapse encountered in the widely-used Differentiable Architecture Search (DARTS) method in the unsupervised paradigm by introducing a multi-scale decoder. Through extensive experiments conducted on various search spaces and datasets, we demonstrate the effectiveness and robustness of the proposed method, providing empirical evidence of its superiority over baseline approaches.

Title: Exploring Time Granularity on Temporal Graphs for Dynamic Link Prediction in Real-world Networks. (arXiv:2311.12255v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.12255
• Code URL: https://github.com/silencex12138/time-granularity-on-temporal-graphs
• Copy Paste: [[2311.12255]] Exploring Time Granularity on Temporal Graphs for Dynamic Link Prediction in Real-world Networks(http://arxiv.org/abs/2311.12255)
• Summary:

  Dynamic Graph Neural Networks (DGNNs) have emerged as the predominant approach for processing dynamic graph-structured data. However, the influence of temporal information on model performance and robustness remains insufficiently explored, particularly regarding how models address prediction tasks with different time granularities. In this paper, we explore the impact of time granularity when training DGNNs on dynamic graphs through extensive experiments. We examine graphs derived from various domains and compare three different DGNNs to the baseline model across four varied time granularities. We mainly consider the interplay between time granularities, model architectures, and negative sampling strategies to obtain general conclusions. Our results reveal that a sophisticated memory mechanism and proper time granularity are crucial for a DGNN to deliver competitive and robust performance in the dynamic link prediction task. We also discuss drawbacks in considered models and datasets and propose promising directions for future research on the time granularity of temporal graphs.

Title: Random Linear Projections Loss for Hyperplane-Based Optimization in Regression Neural Networks. (arXiv:2311.12356v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.12356
• Code URL: https://github.com/ahmedaloui1997/randomlinearprojections
• Copy Paste: [[2311.12356]] Random Linear Projections Loss for Hyperplane-Based Optimization in Regression Neural Networks(http://arxiv.org/abs/2311.12356)
• Summary:

  Despite their popularity across a wide range of domains, regression neural networks are prone to overfitting complex datasets. In this work, we propose a loss function termed Random Linear Projections (RLP) loss, which is empirically shown to mitigate overfitting. With RLP loss, the distance between sets of hyperplanes connecting fixed-size subsets of the neural network's feature-prediction pairs and feature-label pairs is minimized. The intuition behind this loss derives from the notion that if two functions share the same hyperplanes connecting all subsets of feature-label pairs, then these functions must necessarily be equivalent. Our empirical studies, conducted across benchmark datasets and representative synthetic examples, demonstrate the improvements of the proposed RLP loss over mean squared error (MSE). Specifically, neural networks trained with the RLP loss achieve better performance while requiring fewer data samples and are more robust to additive noise. We provide theoretical analysis supporting our empirical findings.

Title: FedDRO: Federated Compositional Optimization for Distributionally Robust Learning. (arXiv:2311.12652v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.12652
• Code URL: null
• Copy Paste: [[2311.12652]] FedDRO: Federated Compositional Optimization for Distributionally Robust Learning(http://arxiv.org/abs/2311.12652)
• Summary:

  Recently, compositional optimization (CO) has gained popularity because of its applications in distributionally robust optimization (DRO) and many other machine learning problems. Large-scale and distributed availability of data demands the development of efficient federated learning (FL) algorithms for solving CO problems. Developing FL algorithms for CO is particularly challenging because of the compositional nature of the objective. Moreover, current state-of-the-art methods to solve such problems rely on large batch gradients (depending on the solution accuracy) not feasible for most practical settings. To address these challenges, in this work, we propose efficient FedAvg-type algorithms for solving non-convex CO in the FL setting. We first establish that vanilla FedAvg is not suitable to solve distributed CO problems because of the data heterogeneity in the compositional objective at each client which leads to the amplification of bias in the local compositional gradient estimates. To this end, we propose a novel FL framework FedDRO that utilizes the DRO problem structure to design a communication strategy that allows FedAvg to control the bias in the estimation of the compositional gradient. A key novelty of our work is to develop solution accuracy-independent algorithms that do not require large batch gradients (and function evaluations) for solving federated CO problems. We establish $\mathcal{O}(\epsilon^{-2})$ sample and $\mathcal{O}(\epsilon^{-3/2})$ communication complexity in the FL setting while achieving linear speedup with the number of clients. We corroborate our theoretical findings with empirical studies on large-scale DRO problems.

Title: Carbohydrate NMR chemical shift predictions using E(3) equivariant graph neural networks. (arXiv:2311.12657v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.12657
• Code URL: https://github.com/mariabankestad/geqshift
• Copy Paste: [[2311.12657]] Carbohydrate NMR chemical shift predictions using E(3) equivariant graph neural networks(http://arxiv.org/abs/2311.12657)
• Summary:

  Carbohydrates, vital components of biological systems, are well-known for their structural diversity. Nuclear Magnetic Resonance (NMR) spectroscopy plays a crucial role in understanding their intricate molecular arrangements and is essential in assessing and verifying the molecular structure of organic molecules. An important part of this process is to predict the NMR chemical shift from the molecular structure. This work introduces a novel approach that leverages E(3) equivariant graph neural networks to predict carbohydrate NMR spectra. Notably, our model achieves a substantial reduction in mean absolute error, up to threefold, compared to traditional models that rely solely on two-dimensional molecular structure. Even with limited data, the model excels, highlighting its robustness and generalization capabilities. The implications are far-reaching and go beyond an advanced understanding of carbohydrate structures and spectral interpretation. For example, it could accelerate research in pharmaceutical applications, biochemistry, and structural biology, offering a faster and more reliable analysis of molecular structures. Furthermore, our approach is a key step towards a new data-driven era in spectroscopy, potentially influencing spectroscopic techniques beyond NMR.

Title: Towards a more inductive world for drug repurposing approaches. (arXiv:2311.12670v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.12670
• Code URL: https://github.com/ubioinformat/graphemb
• Copy Paste: [[2311.12670]] Towards a more inductive world for drug repurposing approaches(http://arxiv.org/abs/2311.12670)
• Summary:

  Drug-target interaction (DTI) prediction is a challenging, albeit essential task in drug repurposing. Learning on graph models have drawn special attention as they can significantly reduce drug repurposing costs and time commitment. However, many current approaches require high-demanding additional information besides DTIs that complicates their evaluation process and usability. Additionally, structural differences in the learning architecture of current models hinder their fair benchmarking. In this work, we first perform an in-depth evaluation of current DTI datasets and prediction models through a robust benchmarking process, and show that DTI prediction methods based on transductive models lack generalization and lead to inflated performance when evaluated as previously done in the literature, hence not being suited for drug repurposing approaches. We then propose a novel biologically-driven strategy for negative edge subsampling and show through in vitro validation that newly discovered interactions are indeed true. We envision this work as the underpinning for future fair benchmarking and robust model design. All generated resources and tools are publicly available as a python package.

Title: minimax: Efficient Baselines for Autocurricula in JAX. (arXiv:2311.12716v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.12716
• Code URL: https://github.com/facebookresearch/minimax
• Copy Paste: [[2311.12716]] minimax: Efficient Baselines for Autocurricula in JAX(http://arxiv.org/abs/2311.12716)
• Summary:

  Unsupervised environment design (UED) is a form of automatic curriculum learning for training robust decision-making agents to zero-shot transfer into unseen environments. Such autocurricula have received much interest from the RL community. However, UED experiments, based on CPU rollouts and GPU model updates, have often required several weeks of training. This compute requirement is a major obstacle to rapid innovation for the field. This work introduces the minimax library for UED training on accelerated hardware. Using JAX to implement fully-tensorized environments and autocurriculum algorithms, minimax allows the entire training loop to be compiled for hardware acceleration. To provide a petri dish for rapid experimentation, minimax includes a tensorized grid-world based on MiniGrid, in addition to reusable abstractions for conducting autocurricula in procedurally-generated environments. With these components, minimax provides strong UED baselines, including new parallelized variants, which achieve over 120$\times$ speedups in wall time compared to previous implementations when training with equal batch sizes. The minimax library is available under the Apache 2.0 license at https://github.com/facebookresearch/minimax.

biometric

steal

Title: Design for Assurance: Employing Functional Verification Tools for Thwarting Hardware Trojan Threat in 3PIPs. (arXiv:2311.12321v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.12321
• Code URL: null
• Copy Paste: [[2311.12321]] Design for Assurance: Employing Functional Verification Tools for Thwarting Hardware Trojan Threat in 3PIPs(http://arxiv.org/abs/2311.12321)
• Summary:

  Third-party intellectual property cores are essential building blocks of modern system-on-chip and integrated circuit designs. However, these design components usually come from vendors of different trust levels and may contain undocumented design functionality. Distinguishing such stealthy lightweight malicious design modification can be a challenging task due to the lack of a golden reference. In this work, we make a step towards design for assurance by developing a method for identifying and preventing hardware Trojans, employing functional verification tools and languages familiar to hardware designers. We dump synthesized design netlist mapped to a field programmable gate array technology library and perform switching as well as coverage analysis at the granularity of look-up-tables (LUTs) in order to identify specious signals and cells. We automatically extract and formally prove properties related to switching and coverage, which allows us to retrieve Trojan trigger condition. We further provide a solution to preventing Trojan from activation by reconfiguring the confirmed malicious LUTs. Experimental results have demonstrated that our method can detect and mitigate Trust-Hub as well as recently reported don't care Trojans.

extraction

Title: FlashOcc: Fast and Memory-Efficient Occupancy Prediction via Channel-to-Height Plugin. (arXiv:2311.12058v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12058
• Code URL: null
• Copy Paste: [[2311.12058]] FlashOcc: Fast and Memory-Efficient Occupancy Prediction via Channel-to-Height Plugin(http://arxiv.org/abs/2311.12058)
• Summary:

  Given the capability of mitigating the long-tail deficiencies and intricate-shaped absence prevalent in 3D object detection, occupancy prediction has become a pivotal component in autonomous driving systems. However, the procession of three-dimensional voxel-level representations inevitably introduces large overhead in both memory and computation, obstructing the deployment of to-date occupancy prediction approaches. In contrast to the trend of making the model larger and more complicated, we argue that a desirable framework should be deployment-friendly to diverse chips while maintaining high precision. To this end, we propose a plug-and-play paradigm, namely FlashOCC, to consolidate rapid and memory-efficient occupancy prediction while maintaining high precision. Particularly, our FlashOCC makes two improvements based on the contemporary voxel-level occupancy prediction approaches. Firstly, the features are kept in the BEV, enabling the employment of efficient 2D convolutional layers for feature extraction. Secondly, a channel-to-height transformation is introduced to lift the output logits from the BEV into the 3D space. We apply the FlashOCC to diverse occupancy prediction baselines on the challenging Occ3D-nuScenes benchmarks and conduct extensive experiments to validate the effectiveness. The results substantiate the superiority of our plug-and-play paradigm over previous state-of-the-art methods in terms of precision, runtime efficiency, and memory costs, demonstrating its potential for deployment. The code will be made available.

Title: ChemScraper: Graphics Extraction, Molecular Diagram Parsing, and Annotated Data Generation for PDF Images. (arXiv:2311.12161v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12161
• Code URL: https://gitlab.com/dprl/graphics-extraction
• Copy Paste: [[2311.12161]] ChemScraper: Graphics Extraction, Molecular Diagram Parsing, and Annotated Data Generation for PDF Images(http://arxiv.org/abs/2311.12161)
• Summary:

  Existing visual parsers for molecule diagrams translate pixel-based raster images such as PNGs to chemical structure representations (e.g., SMILES). However, PDFs created by word processors including \LaTeX{} and Word provide explicit locations and shapes for characters, lines, and polygons. We %introduce a method to extract symbols from born-digital PDF molecule images and then apply simple graph transformations to capture both visual and chemical structure in editable ChemDraw files (CDXML). Our fast ( PDF $\rightarrow$ visual graph $\rightarrow$ chemical graph ) pipeline does not require GPUs, Optical Character Recognition (OCR) or vectorization. We evaluate on standard benchmarks using SMILES strings, along with a novel evaluation that provides graph-based metrics and error compilation using LgEval. The geometric information in born-digital PDFs produces a highly accurate parser, motivating generating training data for visual parsers that recognize from raster images, with extracted graphics, visual structure, and chemical structure as annotations. To do this we render SMILES strings in Indigo, parse molecule structure, and then validate recognized structure to select correct files.

Title: Modality Mixer Exploiting Complementary Information for Multi-modal Action Recognition. (arXiv:2311.12344v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12344
• Code URL: null
• Copy Paste: [[2311.12344]] Modality Mixer Exploiting Complementary Information for Multi-modal Action Recognition(http://arxiv.org/abs/2311.12344)
• Summary:

  Due to the distinctive characteristics of sensors, each modality exhibits unique physical properties. For this reason, in the context of multi-modal action recognition, it is important to consider not only the overall action content but also the complementary nature of different modalities. In this paper, we propose a novel network, named Modality Mixer (M-Mixer) network, which effectively leverages and incorporates the complementary information across modalities with the temporal context of actions for action recognition. A key component of our proposed M-Mixer is the Multi-modal Contextualization Unit (MCU), a simple yet effective recurrent unit. Our MCU is responsible for temporally encoding a sequence of one modality (e.g., RGB) with action content features of other modalities (e.g., depth and infrared modalities). This process encourages M-Mixer network to exploit global action content and also to supplement complementary information of other modalities. Furthermore, to extract appropriate complementary information regarding to the given modality settings, we introduce a new module, named Complementary Feature Extraction Module (CFEM). CFEM incorporates sepearte learnable query embeddings for each modality, which guide CFEM to extract complementary information and global action content from the other modalities. As a result, our proposed method outperforms state-of-the-art methods on NTU RGB+D 60, NTU RGB+D 120, and NW-UCLA datasets. Moreover, through comprehensive ablation studies, we further validate the effectiveness of our proposed method.

Title: Multi-Resolution Planar Region Extraction for Uneven Terrains. (arXiv:2311.12562v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12562
• Code URL: null
• Copy Paste: [[2311.12562]] Multi-Resolution Planar Region Extraction for Uneven Terrains(http://arxiv.org/abs/2311.12562)
• Summary:

  This paper studies the problem of extracting planar regions in uneven terrains from unordered point cloud measurements. Such a problem is critical in various robotic applications such as robotic perceptive locomotion. While existing approaches have shown promising results in effectively extracting planar regions from the environment, they often suffer from issues such as low computational efficiency or loss of resolution. To address these issues, we propose a multi-resolution planar region extraction strategy in this paper that balances the accuracy in boundaries and computational efficiency. Our method begins with a pointwise classification preprocessing module, which categorizes all sampled points according to their local geometric properties to facilitate multi-resolution segmentation. Subsequently, we arrange the categorized points using an octree, followed by an in-depth analysis of nodes to finish multi-resolution plane segmentation. The efficiency and robustness of the proposed approach are verified via synthetic and real-world experiments, demonstrating our method's ability to generalize effectively across various uneven terrains while maintaining real-time performance, achieving frame rates exceeding 35 FPS.

Title: Neural Network Pruning by Gradient Descent. (arXiv:2311.12526v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.12526
• Code URL: https://github.com/3riccc/neural_pruning
• Copy Paste: [[2311.12526]] Neural Network Pruning by Gradient Descent(http://arxiv.org/abs/2311.12526)
• Summary:

  The rapid increase in the parameters of deep learning models has led to significant costs, challenging computational efficiency and model interpretability. In this paper, we introduce a novel and straightforward neural network pruning framework that incorporates the Gumbel-Softmax technique. This framework enables the simultaneous optimization of a network's weights and topology in an end-to-end process using stochastic gradient descent. Empirical results demonstrate its exceptional compression capability, maintaining high accuracy on the MNIST dataset with only 0.15\% of the original network parameters. Moreover, our framework enhances neural network interpretability, not only by allowing easy extraction of feature importance directly from the pruned network but also by enabling visualization of feature symmetry and the pathways of information propagation from features to outcomes. Although the pruning strategy is learned through deep learning, it is surprisingly intuitive and understandable, focusing on selecting key representative features and exploiting data patterns to achieve extreme sparse pruning. We believe our method opens a promising new avenue for deep learning pruning and the creation of interpretable machine learning systems.

membership infer

federate

Title: Energizing Federated Learning via Filter-Aware Attention. (arXiv:2311.12049v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12049
• Code URL: null
• Copy Paste: [[2311.12049]] Energizing Federated Learning via Filter-Aware Attention(http://arxiv.org/abs/2311.12049)
• Summary:

  Federated learning (FL) is a promising distributed paradigm, eliminating the need for data sharing but facing challenges from data heterogeneity. Personalized parameter generation through a hypernetwork proves effective, yet existing methods fail to personalize local model structures. This leads to redundant parameters struggling to adapt to diverse data distributions. To address these limitations, we propose FedOFA, utilizing personalized orthogonal filter attention for parameter recalibration. The core is the Two-stream Filter-aware Attention (TFA) module, meticulously designed to extract personalized filter-aware attention maps, incorporating Intra-Filter Attention (IntraFa) and Inter-Filter Attention (InterFA) streams. These streams enhance representation capability and explore optimal implicit structures for local models. Orthogonal regularization minimizes redundancy by averting inter-correlation between filters. Furthermore, we introduce an Attention-Guided Pruning Strategy (AGPS) for communication efficiency. AGPS selectively retains crucial neurons while masking redundant ones, reducing communication costs without performance sacrifice. Importantly, FedOFA operates on the server side, incurring no additional computational cost on the client, making it advantageous in communication-constrained scenarios. Extensive experiments validate superior performance over state-of-the-art approaches, with code availability upon paper acceptance.

Title: Federated Learning via Consensus Mechanism on Heterogeneous Data: A New Perspective on Convergence. (arXiv:2311.12358v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.12358
• Code URL: https://github.com/fedcome/fedcome
• Copy Paste: [[2311.12358]] Federated Learning via Consensus Mechanism on Heterogeneous Data: A New Perspective on Convergence(http://arxiv.org/abs/2311.12358)
• Summary:

  Federated learning (FL) on heterogeneous data (non-IID data) has recently received great attention. Most existing methods focus on studying the convergence guarantees for the global objective. While these methods can guarantee the decrease of the global objective in each communication round, they fail to ensure risk decrease for each client. In this paper, to address the problem,we propose FedCOME, which introduces a consensus mechanism to enforce decreased risk for each client after each training round. In particular, we allow a slight adjustment to a client's gradient on the server side, which generates an acute angle between the corrected gradient and the original ones of other clients. We theoretically show that the consensus mechanism can guarantee the convergence of the global objective. To generalize the consensus mechanism to the partial participation FL scenario, we devise a novel client sampling strategy to select the most representative clients for the global data distribution. Training on these selected clients with the consensus mechanism could empirically lead to risk decrease for clients that are not selected. Finally, we conduct extensive experiments on four benchmark datasets to show the superiority of FedCOME against other state-of-the-art methods in terms of effectiveness, efficiency and fairness. For reproducibility, we make our source code publicly available at: \url{https://github.com/fedcome/fedcome}.

fair

Title: Fair Text Classification with Wasserstein Independence. (arXiv:2311.12689v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.12689
• Code URL: https://github.com/letenothibaud/wasserstein_fair_classification
• Copy Paste: [[2311.12689]] Fair Text Classification with Wasserstein Independence(http://arxiv.org/abs/2311.12689)
• Summary:

  Group fairness is a central research topic in text classification, where reaching fair treatment between sensitive groups (e.g. women vs. men) remains an open challenge. This paper presents a novel method for mitigating biases in neural text classification, agnostic to the model architecture. Considering the difficulty to distinguish fair from unfair information in a text encoder, we take inspiration from adversarial training to induce Wasserstein independence between representations learned to predict our target label and the ones learned to predict some sensitive attribute. Our approach provides two significant advantages. Firstly, it does not require annotations of sensitive attributes in both testing and training data. This is more suitable for real-life scenarios compared to existing methods that require annotations of sensitive attributes at train time. Second, our approach exhibits a comparable or better fairness-accuracy trade-off compared to existing methods.

Title: Fair Polylog-Approximate Low-Cost Hierarchical Clustering. (arXiv:2311.12501v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.12501
• Code URL: null
• Copy Paste: [[2311.12501]] Fair Polylog-Approximate Low-Cost Hierarchical Clustering(http://arxiv.org/abs/2311.12501)
• Summary:

  Research in fair machine learning, and particularly clustering, has been crucial in recent years given the many ethical controversies that modern intelligent systems have posed. Ahmadian et al. [2020] established the study of fairness in \textit{hierarchical} clustering, a stronger, more structured variant of its well-known flat counterpart, though their proposed algorithm that optimizes for Dasgupta's [2016] famous cost function was highly theoretical. Knittel et al. [2023] then proposed the first practical fair approximation for cost, however they were unable to break the polynomial-approximate barrier they posed as a hurdle of interest. We break this barrier, proposing the first truly polylogarithmic-approximate low-cost fair hierarchical clustering, thus greatly bridging the gap between the best fair and vanilla hierarchical clustering approximations.

Title: Adversarial Reweighting Guided by Wasserstein Distance for Bias Mitigation. (arXiv:2311.12684v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.12684
• Code URL: null
• Copy Paste: [[2311.12684]] Adversarial Reweighting Guided by Wasserstein Distance for Bias Mitigation(http://arxiv.org/abs/2311.12684)
• Summary:

  The unequal representation of different groups in a sample population can lead to discrimination of minority groups when machine learning models make automated decisions. To address these issues, fairness-aware machine learning jointly optimizes two (or more) metrics aiming at predictive effectiveness and low unfairness. However, the inherent under-representation of minorities in the data makes the disparate treatment of subpopulations less noticeable and difficult to deal with during learning. In this paper, we propose a novel adversarial reweighting method to address such \emph{representation bias}. To balance the data distribution between the majority and the minority groups, our approach deemphasizes samples from the majority group. To minimize empirical risk, our method prefers samples from the majority group that are close to the minority group as evaluated by the Wasserstein distance. Our theoretical analysis shows the effectiveness of our adversarial reweighting approach. Experiments demonstrate that our approach mitigates bias without sacrificing classification accuracy, outperforming related state-of-the-art methods on image and tabular benchmark datasets.

interpretability

Title: Point, Segment and Count: A Generalized Framework for Object Counting. (arXiv:2311.12386v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12386
• Code URL: null
• Copy Paste: [[2311.12386]] Point, Segment and Count: A Generalized Framework for Object Counting(http://arxiv.org/abs/2311.12386)
• Summary:

  Class-agnostic object counting aims to count all objects in an image with respect to example boxes or class names, \emph{a.k.a} few-shot and zero-shot counting. Current state-of-the-art methods highly rely on density maps to predict object counts, which lacks model interpretability. In this paper, we propose a generalized framework for both few-shot and zero-shot object counting based on detection. Our framework combines the superior advantages of two foundation models without compromising their zero-shot capability: (\textbf{i}) SAM to segment all possible objects as mask proposals, and (\textbf{ii}) CLIP to classify proposals to obtain accurate object counts. However, this strategy meets the obstacles of efficiency overhead and the small crowded objects that cannot be localized and distinguished. To address these issues, our framework, termed PseCo, follows three steps: point, segment, and count. Specifically, we first propose a class-agnostic object localization to provide accurate but least point prompts for SAM, which consequently not only reduces computation costs but also avoids missing small objects. Furthermore, we propose a generalized object classification that leverages CLIP image/text embeddings as the classifier, following a hierarchical knowledge distillation to obtain discriminative classifications among hierarchical mask proposals. Extensive experimental results on FSC-147 dataset demonstrate that PseCo achieves state-of-the-art performance in both few-shot/zero-shot object counting/detection, with additional results on large-scale COCO and LVIS datasets. The source code is available at \url{https://github.com/Hzzone/PseCo}.

Title: Cascade Learning Localises Discriminant Features in Visual Scene Classification. (arXiv:2311.12704v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12704
• Code URL: null
• Copy Paste: [[2311.12704]] Cascade Learning Localises Discriminant Features in Visual Scene Classification(http://arxiv.org/abs/2311.12704)
• Summary:

  Lack of interpretability of deep convolutional neural networks (DCNN) is a well-known problem particularly in the medical domain as clinicians want trustworthy automated decisions. One way to improve trust is to demonstrate the localisation of feature representations with respect to expert labeled regions of interest. In this work, we investigate the localisation of features learned via two varied learning paradigms and demonstrate the superiority of one learning approach with respect to localisation. Our analysis on medical and natural datasets show that the traditional end-to-end (E2E) learning strategy has a limited ability to localise discriminative features across multiple network layers. We show that a layer-wise learning strategy, namely cascade learning (CL), results in more localised features. Considering localisation accuracy, we not only show that CL outperforms E2E but that it is a promising method of predicting regions. On the YOLO object detection framework, our best result shows that CL outperforms the E2E scheme by $2\%$ in mAP.

Title: Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks. (arXiv:2311.12786v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.12786
• Code URL: null
• Copy Paste: [[2311.12786]] Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks(http://arxiv.org/abs/2311.12786)
• Summary:

  Fine-tuning large pre-trained models has become the de facto strategy for developing both task-specific and general-purpose machine learning systems, including developing models that are safe to deploy. Despite its clear importance, there has been minimal work that explains how fine-tuning alters the underlying capabilities learned by a model during pretraining: does fine-tuning yield entirely novel capabilities or does it just modulate existing ones? We address this question empirically in synthetic, controlled settings where we can use mechanistic interpretability tools (e.g., network pruning and probing) to understand how the model's underlying capabilities are changing. We perform an extensive analysis of the effects of fine-tuning in these settings, and show that: (i) fine-tuning rarely alters the underlying model capabilities; (ii) a minimal transformation, which we call a 'wrapper', is typically learned on top of the underlying model capabilities, creating the illusion that they have been modified; and (iii) further fine-tuning on a task where such hidden capabilities are relevant leads to sample-efficient 'revival' of the capability, i.e., the model begins reusing these capability after only a few gradient steps. This indicates that practitioners can unintentionally remove a model's safety wrapper merely by fine-tuning it on a, e.g., superficially unrelated, downstream task. We additionally perform analysis on language models trained on the TinyStories dataset to support our claims in a more realistic setup.

explainability

Title: InterPrompt: Interpretable Prompting for Interrelated Interpersonal Risk Factors in Reddit Posts. (arXiv:2311.12404v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.12404
• Code URL: null
• Copy Paste: [[2311.12404]] InterPrompt: Interpretable Prompting for Interrelated Interpersonal Risk Factors in Reddit Posts(http://arxiv.org/abs/2311.12404)
• Summary:

  Mental health professionals and clinicians have observed the upsurge of mental disorders due to Interpersonal Risk Factors (IRFs). To simulate the human-in-the-loop triaging scenario for early detection of mental health disorders, we recognized textual indications to ascertain these IRFs : Thwarted Belongingness (TBe) and Perceived Burdensomeness (PBu) within personal narratives. In light of this, we use N-shot learning with GPT-3 model on the IRF dataset, and underscored the importance of fine-tuning GPT-3 model to incorporate the context-specific sensitivity and the interconnectedness of textual cues that represent both IRFs.

  In this paper, we introduce an Interpretable Prompting (InterPrompt)} method to boost the attention mechanism by fine-tuning the GPT-3 model. This allows a more sophisticated level of language modification by adjusting the pre-trained weights. Our model learns to detect usual patterns and underlying connections across both the IRFs, which leads to better system-level explainability and trustworthiness. The results of our research demonstrate that all four variants of GPT-3 model, when fine-tuned with InterPrompt, perform considerably better as compared to the baseline methods, both in terms of classification and explanation generation.

watermark

diffusion

Title: Quality and Quantity: Unveiling a Million High-Quality Images for Text-to-Image Synthesis in Fashion Design. (arXiv:2311.12067v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12067
• Code URL: null
• Copy Paste: [[2311.12067]] Quality and Quantity: Unveiling a Million High-Quality Images for Text-to-Image Synthesis in Fashion Design(http://arxiv.org/abs/2311.12067)
• Summary:

  The fusion of AI and fashion design has emerged as a promising research area. However, the lack of extensive, interrelated data on clothing and try-on stages has hindered the full potential of AI in this domain. Addressing this, we present the Fashion-Diffusion dataset, a product of multiple years' rigorous effort. This dataset, the first of its kind, comprises over a million high-quality fashion images, paired with detailed text descriptions. Sourced from a diverse range of geographical locations and cultural backgrounds, the dataset encapsulates global fashion trends. The images have been meticulously annotated with fine-grained attributes related to clothing and humans, simplifying the fashion design process into a Text-to-Image (T2I) task. The Fashion-Diffusion dataset not only provides high-quality text-image pairs and diverse human-garment pairs but also serves as a large-scale resource about humans, thereby facilitating research in T2I generation. Moreover, to foster standardization in the T2I-based fashion design field, we propose a new benchmark comprising multiple datasets for evaluating the performance of fashion design models. This work represents a significant leap forward in the realm of AI-driven fashion design, setting a new standard for future research in this field.

Title: Pyramid Diffusion for Fine 3D Large Scene Generation. (arXiv:2311.12085v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12085
• Code URL: null
• Copy Paste: [[2311.12085]] Pyramid Diffusion for Fine 3D Large Scene Generation(http://arxiv.org/abs/2311.12085)
• Summary:

  Directly transferring the 2D techniques to 3D scene generation is challenging due to significant resolution reduction and the scarcity of comprehensive real-world 3D scene datasets. To address these issues, our work introduces the Pyramid Discrete Diffusion model (PDD) for 3D scene generation. This novel approach employs a multi-scale model capable of progressively generating high-quality 3D scenes from coarse to fine. In this way, the PDD can generate high-quality scenes within limited resource constraints and does not require additional data sources. To the best of our knowledge, we are the first to adopt the simple but effective coarse-to-fine strategy for 3D large scene generation. Our experiments, covering both unconditional and conditional generation, have yielded impressive results, showcasing the model's effectiveness and robustness in generating realistic and detailed 3D scenes. Our code will be available to the public.

Title: FrePolad: Frequency-Rectified Point Latent Diffusion for Point Cloud Generation. (arXiv:2311.12090v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12090
• Code URL: null
• Copy Paste: [[2311.12090]] FrePolad: Frequency-Rectified Point Latent Diffusion for Point Cloud Generation(http://arxiv.org/abs/2311.12090)
• Summary:

  We propose FrePolad: frequency-rectified point latent diffusion, a point cloud generation pipeline integrating a variational autoencoder (VAE) with a denoising diffusion probabilistic model (DDPM) for the latent distribution. FrePolad simultaneously achieves high quality, diversity, and flexibility in point cloud cardinality for generation tasks while maintaining high computational efficiency. The improvement in generation quality and diversity is achieved through (1) a novel frequency rectification module via spherical harmonics designed to retain high-frequency content while learning the point cloud distribution; and (2) a latent DDPM to learn the regularized yet complex latent distribution. In addition, FrePolad supports variable point cloud cardinality by formulating the sampling of points as conditional distributions over a latent shape distribution. Finally, the low-dimensional latent space encoded by the VAE contributes to FrePolad's fast and scalable sampling. Our quantitative and qualitative results demonstrate the state-of-the-art performance of FrePolad in terms of quality, diversity, and computational efficiency.

Title: Concept Sliders: LoRA Adaptors for Precise Control in Diffusion Models. (arXiv:2311.12092v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12092
• Code URL: https://github.com/rohitgandikota/sliders
• Copy Paste: [[2311.12092]] Concept Sliders: LoRA Adaptors for Precise Control in Diffusion Models(http://arxiv.org/abs/2311.12092)
• Summary:

  We present a method to create interpretable concept sliders that enable precise control over attributes in image generations from diffusion models. Our approach identifies a low-rank parameter direction corresponding to one concept while minimizing interference with other attributes. A slider is created using a small set of prompts or sample images; thus slider directions can be created for either textual or visual concepts. Concept Sliders are plug-and-play: they can be composed efficiently and continuously modulated, enabling precise control over image generation. In quantitative experiments comparing to previous editing techniques, our sliders exhibit stronger targeted edits with lower interference. We showcase sliders for weather, age, styles, and expressions, as well as slider compositions. We show how sliders can transfer latents from StyleGAN for intuitive editing of visual concepts for which textual description is difficult. We also find that our method can help address persistent quality issues in Stable Diffusion XL including repair of object deformations and fixing distorted hands. Our code, data, and trained sliders are available at https://sliders.baulab.info/

Title: Overcoming Pathology Image Data Deficiency: Generating Images from Pathological Transformation Process. (arXiv:2311.12316v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12316
• Code URL: https://github.com/rowerliu/adbd
• Copy Paste: [[2311.12316]] Overcoming Pathology Image Data Deficiency: Generating Images from Pathological Transformation Process(http://arxiv.org/abs/2311.12316)
• Summary:

  Histopathology serves as the gold standard for medical diagnosis but faces application limitations due to the shortage of medical resources. Leveraging deep learning, computer-aided diagnosis has the potential to alleviate the pathologist scarcity and provide timely clinical analysis. However, developing a reliable model generally necessitates substantial data for training, which is challenging in pathological field. In response, we propose an adaptive depth-controlled bidirectional diffusion (ADBD) network for image data generation. The domain migration approach can work with small trainset and overcome the diffusion overfitting by source information guidance. Specifically, we developed a hybrid attention strategy to blend global and local attention priorities, which guides the bidirectional diffusion and ensures the migration success. In addition, we developed the adaptive depth-controlled strategy to simulate physiological transformations, capable of yielding unlimited cross-domain intermediate images with corresponding soft labels. ADBD is effective for overcoming pathological image data deficiency and supportable for further pathology-related research.

Title: LoCo: Locally Constrained Training-Free Layout-to-Image Synthesis. (arXiv:2311.12342v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12342
• Code URL: null
• Copy Paste: [[2311.12342]] LoCo: Locally Constrained Training-Free Layout-to-Image Synthesis(http://arxiv.org/abs/2311.12342)
• Summary:

  Recent text-to-image diffusion models have reached an unprecedented level in generating high-quality images. However, their exclusive reliance on textual prompts often falls short in accurately conveying fine-grained spatial compositions. In this paper, we propose LoCo, a training-free approach for layout-to-image synthesis that excels in producing high-quality images aligned with both textual prompts and spatial layouts. Our method introduces a Localized Attention Constraint to refine cross-attention for individual objects, ensuring their precise placement in designated regions. We further propose a Padding Token Constraint to leverage the semantic information embedded in previously neglected padding tokens, thereby preventing the undesired fusion of synthesized objects. LoCo seamlessly integrates into existing text-to-image and layout-to-image models, significantly amplifying their performance and effectively addressing semantic failures observed in prior methods. Through extensive experiments, we showcase the superiority of our approach, surpassing existing state-of-the-art training-free layout-to-image methods both qualitatively and quantitatively across multiple benchmarks.

Title: Stable Diffusion For Aerial Object Detection. (arXiv:2311.12345v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12345
• Code URL: null
• Copy Paste: [[2311.12345]] Stable Diffusion For Aerial Object Detection(http://arxiv.org/abs/2311.12345)
• Summary:

  Aerial object detection is a challenging task, in which one major obstacle lies in the limitations of large-scale data collection and the long-tail distribution of certain classes. Synthetic data offers a promising solution, especially with recent advances in diffusion-based methods like stable diffusion (SD). However, the direct application of diffusion methods to aerial domains poses unique challenges: stable diffusion's optimization for rich ground-level semantics doesn't align with the sparse nature of aerial objects, and the extraction of post-synthesis object coordinates remains problematic. To address these challenges, we introduce a synthetic data augmentation framework tailored for aerial images. It encompasses sparse-to-dense region of interest (ROI) extraction to bridge the semantic gap, fine-tuning the diffusion model with low-rank adaptation (LORA) to circumvent exhaustive retraining, and finally, a Copy-Paste method to compose synthesized objects with backgrounds, providing a nuanced approach to aerial object detection through synthetic data.

Title: GPT4Motion: Scripting Physical Motions in Text-to-Video Generation via Blender-Oriented GPT Planning. (arXiv:2311.12631v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12631
• Code URL: null
• Copy Paste: [[2311.12631]] GPT4Motion: Scripting Physical Motions in Text-to-Video Generation via Blender-Oriented GPT Planning(http://arxiv.org/abs/2311.12631)
• Summary:

  Recent advances in text-to-video generation have harnessed the power of diffusion models to create visually compelling content conditioned on text prompts. However, they usually encounter high computational costs and often struggle to produce videos with coherent physical motions. To tackle these issues, we propose GPT4Motion, a training-free framework that leverages the planning capability of large language models such as GPT, the physical simulation strength of Blender, and the excellent image generation ability of text-to-image diffusion models to enhance the quality of video synthesis. Specifically, GPT4Motion employs GPT-4 to generate a Blender script based on a user textual prompt, which commands Blender's built-in physics engine to craft fundamental scene components that encapsulate coherent physical motions across frames. Then these components are inputted into Stable Diffusion to generate a video aligned with the textual prompt. Experimental results on three basic physical motion scenarios, including rigid object drop and collision, cloth draping and swinging, and liquid flow, demonstrate that GPT4Motion can generate high-quality videos efficiently in maintaining motion coherency and entity consistency. GPT4Motion offers new insights in text-to-video research, enhancing its quality and broadening its horizon for future explorations.

noise learning

data-free

transformer

Title: PBWR: Parametric Building Wireframe Reconstruction from Aerial LiDAR Point Clouds. (arXiv:2311.12062v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12062
• Code URL: null
• Copy Paste: [[2311.12062]] PBWR: Parametric Building Wireframe Reconstruction from Aerial LiDAR Point Clouds(http://arxiv.org/abs/2311.12062)
• Summary:

  In this paper, we present an end-to-end 3D building wireframe reconstruction method to regress edges directly from aerial LiDAR point clouds.Our method, named Parametric Building Wireframe Reconstruction (PBWR), takes aerial LiDAR point clouds and initial edge entities as input, and fully uses self-attention mechanism of transformers to regress edge parameters without any intermediate steps such as corner prediction. We propose an edge non-maximum suppression (E-NMS) module based on edge similarityto remove redundant edges. Additionally, a dedicated edge loss function is utilized to guide the PBWR in regressing edges parameters, where simple use of edge distance loss isn't suitable. In our experiments, we demonstrate state-of-the-art results on the Building3D dataset, achieving an improvement of approximately 36% in entry-level dataset edge accuracy and around 42% improvement in the Tallinn dataset.

Title: DAS: A Deformable Attention to Capture Salient Information in CNNs. (arXiv:2311.12091v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12091
• Code URL: null
• Copy Paste: [[2311.12091]] DAS: A Deformable Attention to Capture Salient Information in CNNs(http://arxiv.org/abs/2311.12091)
• Summary:

  Convolutional Neural Networks (CNNs) excel in local spatial pattern recognition. For many vision tasks, such as object recognition and segmentation, salient information is also present outside CNN's kernel boundaries. However, CNNs struggle in capturing such relevant information due to their confined receptive fields. Self-attention can improve a model's access to global information but increases computational overhead. We present a fast and simple fully convolutional method called DAS that helps focus attention on relevant information. It uses deformable convolutions for the location of pertinent image regions and separable convolutions for efficiency. DAS plugs into existing CNNs and propagates relevant information using a gating mechanism. Compared to the O(n^2) computational complexity of transformer-style attention, DAS is O(n). Our claim is that DAS's ability to pay increased attention to relevant features results in performance improvements when added to popular CNNs for Image Classification and Object Detection. For example, DAS yields an improvement on Stanford Dogs (4.47%), ImageNet (1.91%), and COCO AP (3.3%) with base ResNet50 backbone. This outperforms other CNN attention mechanisms while using similar or less FLOPs. Our code will be publicly available.

Title: Fingerspelling PoseNet: Enhancing Fingerspelling Translation with Pose-Based Transformer Models. (arXiv:2311.12128v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12128
• Code URL: null
• Copy Paste: [[2311.12128]] Fingerspelling PoseNet: Enhancing Fingerspelling Translation with Pose-Based Transformer Models(http://arxiv.org/abs/2311.12128)
• Summary:

  We address the task of American Sign Language fingerspelling translation using videos in the wild. We exploit advances in more accurate hand pose estimation and propose a novel architecture that leverages the transformer based encoder-decoder model enabling seamless contextual word translation. The translation model is augmented by a novel loss term that accurately predicts the length of the finger-spelled word, benefiting both training and inference. We also propose a novel two-stage inference approach that re-ranks the hypotheses using the language model capabilities of the decoder. Through extensive experiments, we demonstrate that our proposed method outperforms the state-of-the-art models on ChicagoFSWild and ChicagoFSWild+ achieving more than 10% relative improvement in performance. Our findings highlight the effectiveness of our approach and its potential to advance fingerspelling recognition in sign language translation. Code is also available at https://github.com/pooyafayyaz/Fingerspelling-PoseNet.

Title: Model-aware 3D Eye Gaze from Weak and Few-shot Supervisions. (arXiv:2311.12157v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12157
• Code URL: https://github.com/dimitris-christodoulou57/model-aware_3d_eye_gaze
• Copy Paste: [[2311.12157]] Model-aware 3D Eye Gaze from Weak and Few-shot Supervisions(http://arxiv.org/abs/2311.12157)
• Summary:

  The task of predicting 3D eye gaze from eye images can be performed either by (a) end-to-end learning for image-to-gaze mapping or by (b) fitting a 3D eye model onto images. The former case requires 3D gaze labels, while the latter requires eye semantics or landmarks to facilitate the model fitting. Although obtaining eye semantics and landmarks is relatively easy, fitting an accurate 3D eye model on them remains to be very challenging due to its ill-posed nature in general. On the other hand, obtaining large-scale 3D gaze data is cumbersome due to the required hardware setups and computational demands. In this work, we propose to predict 3D eye gaze from weak supervision of eye semantic segmentation masks and direct supervision of a few 3D gaze vectors. The proposed method combines the best of both worlds by leveraging large amounts of weak annotations--which are easy to obtain, and only a few 3D gaze vectors--which alleviate the difficulty of fitting 3D eye models on the semantic segmentation of eye images. Thus, the eye gaze vectors, used in the model fitting, are directly supervised using the few-shot gaze labels. Additionally, we propose a transformer-based network architecture, that serves as a solid baseline for our improvements. Our experiments in diverse settings illustrate the significant benefits of the proposed method, achieving about 5 degrees lower angular gaze error over the baseline, when only 0.05% 3D annotations of the training images are used. The source code is available at https://github.com/dimitris-christodoulou57/Model-aware_3D_Eye_Gaze.

Title: Disentangling Structure and Appearance in ViT Feature Space. (arXiv:2311.12193v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12193
• Code URL: null
• Copy Paste: [[2311.12193]] Disentangling Structure and Appearance in ViT Feature Space(http://arxiv.org/abs/2311.12193)
• Summary:

  We present a method for semantically transferring the visual appearance of one natural image to another. Specifically, our goal is to generate an image in which objects in a source structure image are "painted" with the visual appearance of their semantically related objects in a target appearance image. To integrate semantic information into our framework, our key idea is to leverage a pre-trained and fixed Vision Transformer (ViT) model. Specifically, we derive novel disentangled representations of structure and appearance extracted from deep ViT features. We then establish an objective function that splices the desired structure and appearance representations, interweaving them together in the space of ViT features. Based on our objective function, we propose two frameworks of semantic appearance transfer -- "Splice", which works by training a generator on a single and arbitrary pair of structure-appearance images, and "SpliceNet", a feed-forward real-time appearance transfer model trained on a dataset of images from a specific domain. Our frameworks do not involve adversarial training, nor do they require any additional input information such as semantic segmentation or correspondences. We demonstrate high-resolution results on a variety of in-the-wild image pairs, under significant variations in the number of objects, pose, and appearance. Code and supplementary material are available in our project page: splice-vit.github.io.

Title: ViLaM: A Vision-Language Model with Enhanced Visual Grounding and Generalization Capability. (arXiv:2311.12327v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12327
• Code URL: https://github.com/anonymgiant/vilam
• Copy Paste: [[2311.12327]] ViLaM: A Vision-Language Model with Enhanced Visual Grounding and Generalization Capability(http://arxiv.org/abs/2311.12327)
• Summary:

  Vision-language models have revolutionized human-computer interaction and shown significant progress in multi-modal tasks. However, applying these models to complex visual tasks like medical image analysis remains challenging. In this study, we propose ViLaM, a unified Vision-Language transformer model that integrates instruction tuning predicated on a large language model. This approach enables us to optimally utilize the knowledge and reasoning capacities of large pre-trained language models for an array of tasks encompassing both language and vision. We employ frozen pre-trained encoders to encode and align both image and text features, enabling ViLaM to handle a variety of visual tasks following textual instructions. Besides, we've designed cycle training for referring expressions to address the need for high-quality, paired referring expression datasets for training large models in terms of both quantity and quality. We evaluated ViLaM's exceptional performance on public general datasets and further confirmed its generalizability on medical datasets. Importantly, we've observed the model's impressive zero-shot learning ability, indicating the potential future application of ViLaM in the medical field.

Title: Improving Source-Free Target Adaptation with Vision Transformers Leveraging Domain Representation Images. (arXiv:2311.12589v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12589
• Code URL: null
• Copy Paste: [[2311.12589]] Improving Source-Free Target Adaptation with Vision Transformers Leveraging Domain Representation Images(http://arxiv.org/abs/2311.12589)
• Summary:

  Unsupervised Domain Adaptation (UDA) methods facilitate knowledge transfer from a labeled source domain to an unlabeled target domain, navigating the obstacle of domain shift. While Convolutional Neural Networks (CNNs) are a staple in UDA, the rise of Vision Transformers (ViTs) provides new avenues for domain generalization. This paper presents an innovative method to bolster ViT performance in source-free target adaptation, beginning with an evaluation of how key, query, and value elements affect ViT outcomes. Experiments indicate that altering the key component has negligible effects on Transformer performance. Leveraging this discovery, we introduce Domain Representation Images (DRIs), feeding embeddings through the key element. DRIs act as domain-specific markers, effortlessly merging with the training regimen. To assess our method, we perform target adaptation tests on the Cross Instance DRI source-only (SO) control. We measure the efficacy of target adaptation with and without DRIs, against existing benchmarks like SHOT-B* and adaptations via CDTrans. Findings demonstrate that excluding DRIs offers limited gains over SHOT-B*, while their inclusion in the key segment boosts average precision promoting superior domain generalization. This research underscores the vital role of DRIs in enhancing ViT efficiency in UDA scenarios, setting a precedent for further domain adaptation explorations.

Title: Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Survey. (arXiv:2311.12351v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.12351
• Code URL: https://github.com/strivin0311/long-llms-learning
• Copy Paste: [[2311.12351]] Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Survey(http://arxiv.org/abs/2311.12351)
• Summary:

  With the bomb ignited by ChatGPT, Transformer-based Large Language Models (LLMs) have paved a revolutionary path toward Artificial General Intelligence (AGI) and have been applied in diverse areas as knowledge bases, human interfaces, and dynamic agents. However, a prevailing limitation exists: many current LLMs, constrained by resources, are primarily pre-trained on shorter texts, rendering them less effective for longer-context prompts, commonly encountered in real-world settings. In this paper, we present a comprehensive survey focusing on the advancement of model architecture in Transformer-based LLMs to optimize long-context capabilities across all stages from pre-training to inference. We firstly delineate and analyze the problems of handling long-context input and output with the current Transformer-based models. Then, we mainly offer a holistic taxonomy to navigate the landscape of Transformer upgrades on architecture to solve these problems. Afterward, we provide the investigation on wildly used evaluation necessities tailored for long-context LLMs, including datasets, metrics, and baseline models, as well as some amazing optimization toolkits like libraries, systems, and compilers to augment LLMs' efficiency and efficacy across different stages. Finally, we further discuss the predominant challenges and potential avenues for future research in this domain. Additionally, we have established a repository where we curate relevant literature with real-time updates at https://github.com/Strivin0311/long-llms-learning.

Title: Visual Analytics for Generative Transformer Models. (arXiv:2311.12418v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.12418
• Code URL: null
• Copy Paste: [[2311.12418]] Visual Analytics for Generative Transformer Models(http://arxiv.org/abs/2311.12418)
• Summary:

  While transformer-based models have achieved state-of-the-art results in a variety of classification and generation tasks, their black-box nature makes them challenging for interpretability. In this work, we present a novel visual analytical framework to support the analysis of transformer-based generative networks. In contrast to previous work, which has mainly focused on encoder-based models, our framework is one of the first dedicated to supporting the analysis of transformer-based encoder-decoder models and decoder-only models for generative and classification tasks. Hence, we offer an intuitive overview that allows the user to explore different facets of the model through interactive visualization. To demonstrate the feasibility and usefulness of our framework, we present three detailed case studies based on real-world NLP research problems.

Title: PhayaThaiBERT: Enhancing a Pretrained Thai Language Model with Unassimilated Loanwords. (arXiv:2311.12475v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.12475
• Code URL: null
• Copy Paste: [[2311.12475]] PhayaThaiBERT: Enhancing a Pretrained Thai Language Model with Unassimilated Loanwords(http://arxiv.org/abs/2311.12475)
• Summary:

  While WangchanBERTa has become the de facto standard in transformer-based Thai language modeling, it still has shortcomings in regard to the understanding of foreign words, most notably English words, which are often borrowed without orthographic assimilation into Thai in many contexts. We identify the lack of foreign vocabulary in WangchanBERTa's tokenizer as the main source of these shortcomings. We then expand WangchanBERTa's vocabulary via vocabulary transfer from XLM-R's pretrained tokenizer and pretrain a new model using the expanded tokenizer, starting from WangchanBERTa's checkpoint, on a new dataset that is larger than the one used to train WangchanBERTa. Our results show that our new pretrained model, PhayaThaiBERT, outperforms WangchanBERTa in many downstream tasks and datasets.

Title: Looped Transformers are Better at Learning Learning Algorithms. (arXiv:2311.12424v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.12424
• Code URL: null
• Copy Paste: [[2311.12424]] Looped Transformers are Better at Learning Learning Algorithms(http://arxiv.org/abs/2311.12424)
• Summary:

  Transformers have demonstrated effectiveness in \emph{in-context solving} data-fitting problems from various (latent) models, as reported by Garg et al. However, the absence of an inherent iterative structure in the transformer architecture presents a challenge in emulating the iterative algorithms, which are commonly employed in traditional machine learning methods. To address this, we propose the utilization of \emph{looped} transformer architecture and its associated training methodology, with the aim of incorporating iterative characteristics into the transformer architectures. Experimental results suggest that the looped transformer achieves performance comparable to the standard transformer in solving various data-fitting problems, while utilizing less than 10\% of the parameter count.

Title: Interpretation of the Transformer and Improvement of the Extractor. (arXiv:2311.12678v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.12678
• Code URL: null
• Copy Paste: [[2311.12678]] Interpretation of the Transformer and Improvement of the Extractor(http://arxiv.org/abs/2311.12678)
• Summary:

  It has been over six years since the Transformer architecture was put forward. Surprisingly, the vanilla Transformer architecture is still widely used today. One reason is that the lack of deep understanding and comprehensive interpretation of the Transformer architecture makes it more challenging to improve the Transformer architecture. In this paper, we first interpret the Transformer architecture comprehensively in plain words based on our understanding and experiences. The interpretations are further proved and verified. These interpretations also cover the Extractor, a family of drop-in replacements for the multi-head self-attention in the Transformer architecture. Then, we propose an improvement on a type of the Extractor that outperforms the self-attention, without introducing additional trainable parameters. Experimental results demonstrate that the improved Extractor performs even better, showing a way to improve the Transformer architecture.

Title: Learning to Optimise Wind Farms with Graph Transformers. (arXiv:2311.12750v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.12750
• Code URL: null
• Copy Paste: [[2311.12750]] Learning to Optimise Wind Farms with Graph Transformers(http://arxiv.org/abs/2311.12750)
• Summary:

  This work proposes a novel data-driven model capable of providing accurate predictions for the power generation of all wind turbines in wind farms of arbitrary layout, yaw angle configurations and wind conditions. The proposed model functions by encoding a wind farm into a fully-connected graph and processing the graph representation through a graph transformer. The graph transformer surrogate is shown to generalise well and is able to uncover latent structural patterns within the graph representation of wind farms. It is demonstrated how the resulting surrogate model can be used to optimise yaw angle configurations using genetic algorithms, achieving similar levels of accuracy to industrially-standard wind farm simulation tools while only taking a fraction of the computational cost.

generative

Title: Efficient Domain Adaptation via Generative Prior for 3D Infant Pose Estimation. (arXiv:2311.12043v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12043
• Code URL: null
• Copy Paste: [[2311.12043]] Efficient Domain Adaptation via Generative Prior for 3D Infant Pose Estimation(http://arxiv.org/abs/2311.12043)
• Summary:

  Although 3D human pose estimation has gained impressive development in recent years, only a few works focus on infants, that have different bone lengths and also have limited data. Directly applying adult pose estimation models typically achieves low performance in the infant domain and suffers from out-of-distribution issues. Moreover, the limitation of infant pose data collection also heavily constrains the efficiency of learning-based models to lift 2D poses to 3D. To deal with the issues of small datasets, domain adaptation and data augmentation are commonly used techniques. Following this paradigm, we take advantage of an optimization-based method that utilizes generative priors to predict 3D infant keypoints from 2D keypoints without the need of large training data. We further apply a guided diffusion model to domain adapt 3D adult pose to infant pose to supplement small datasets. Besides, we also prove that our method, ZeDO-i, could attain efficient domain adaptation, even if only a small number of data is given. Quantitatively, we claim that our model attains state-of-the-art MPJPE performance of 43.6 mm on the SyRIP dataset and 21.2 mm on the MINI-RGBD dataset.

Title: DatasetNeRF: Efficient 3D-aware Data Factory with Generative Radiance Fields. (arXiv:2311.12063v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12063
• Code URL: null
• Copy Paste: [[2311.12063]] DatasetNeRF: Efficient 3D-aware Data Factory with Generative Radiance Fields(http://arxiv.org/abs/2311.12063)
• Summary:

  Progress in 3D computer vision tasks demands a huge amount of data, yet annotating multi-view images with 3D-consistent annotations, or point clouds with part segmentation is both time-consuming and challenging. This paper introduces DatasetNeRF, a novel approach capable of generating infinite, high-quality 3D-consistent 2D annotations alongside 3D point cloud segmentations, while utilizing minimal 2D human-labeled annotations. Specifically, we leverage the strong semantic prior within a 3D generative model to train a semantic decoder, requiring only a handful of fine-grained labeled samples. Once trained, the decoder efficiently generalizes across the latent space, enabling the generation of infinite data. The generated data is applicable across various computer vision tasks, including video segmentation and 3D point cloud segmentation. Our approach not only surpasses baseline models in segmentation quality, achieving superior 3D consistency and segmentation precision on individual images, but also demonstrates versatility by being applicable to both articulated and non-articulated generative models. Furthermore, we explore applications stemming from our approach, such as 3D-aware semantic editing and 3D inversion.

Title: Rich and Poor Texture Contrast: A Simple yet Effective Approach for AI-generated Image Detection. (arXiv:2311.12397v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12397
• Code URL: null
• Copy Paste: [[2311.12397]] Rich and Poor Texture Contrast: A Simple yet Effective Approach for AI-generated Image Detection(http://arxiv.org/abs/2311.12397)
• Summary:

  Recent generative models show impressive performance in generating photographic images. Humans can hardly distinguish such incredibly realistic-looking AI-generated images from real ones. AI-generated images may lead to ubiquitous disinformation dissemination. Therefore, it is of utmost urgency to develop a detector to identify AI-generated images. Most existing detectors suffer from sharp performance drops over unseen generative models. In this paper, we propose a novel AI-generated image detector capable of identifying fake images created by a wide range of generative models. Our approach leverages the inter-pixel correlation contrast between rich and poor texture regions within an image. Pixels in rich texture regions exhibit more significant fluctuations than those in poor texture regions. This discrepancy reflects that the entropy of rich texture regions is larger than that of poor ones. Consequently, synthesizing realistic rich texture regions proves to be more challenging for existing generative models. Based on this principle, we divide an image into multiple patches and reconstruct them into two images, comprising rich-texture and poor-texture patches respectively. Subsequently, we extract the inter-pixel correlation discrepancy feature between rich and poor texture regions. This feature serves as a universal fingerprint used for AI-generated image forensics across different generative models. In addition, we build a comprehensive AI-generated image detection benchmark, which includes 16 kinds of prevalent generative models, to evaluate the effectiveness of existing baselines and our approach. Our benchmark provides a leaderboard for follow-up studies. Extensive experimental results show that our approach outperforms state-of-the-art baselines by a significant margin. Our project: https://fdmas.github.io/AIGCDetect/

Title: Explainable Anomaly Detection using Masked Latent Generative Modeling. (arXiv:2311.12550v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.12550
• Code URL: null
• Copy Paste: [[2311.12550]] Explainable Anomaly Detection using Masked Latent Generative Modeling(http://arxiv.org/abs/2311.12550)
• Summary:

  We present a novel time series anomaly detection method that achieves excellent detection accuracy while offering a superior level of explainability. Our proposed method, TimeVQVAE-AD, leverages masked generative modeling adapted from the cutting-edge time series generation method known as TimeVQVAE. The prior model is trained on the discrete latent space of a time-frequency domain. Notably, the dimensional semantics of the time-frequency domain are preserved in the latent space, enabling us to compute anomaly scores across different frequency bands, which provides a better insight into the detected anomalies. Additionally, the generative nature of the prior model allows for sampling likely normal states for detected anomalies, enhancing the explainability of the detected anomalies through counterfactuals. Our experimental evaluation on the UCR Time Series Anomaly archive demonstrates that TimeVQVAE-AD significantly surpasses the existing methods in terms of detection accuracy and explainability.

large language model

Title: Few-Shot Classification & Segmentation Using Large Language Models Agent. (arXiv:2311.12065v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12065
• Code URL: null
• Copy Paste: [[2311.12065]] Few-Shot Classification & Segmentation Using Large Language Models Agent(http://arxiv.org/abs/2311.12065)
• Summary:

  The task of few-shot image classification and segmentation (FS-CS) requires the classification and segmentation of target objects in a query image, given only a few examples of the target classes. We introduce a method that utilises large language models (LLM) as an agent to address the FS-CS problem in a training-free manner. By making the LLM the task planner and off-the-shelf vision models the tools, the proposed method is capable of classifying and segmenting target objects using only image-level labels. Specifically, chain-of-thought prompting and in-context learning guide the LLM to observe support images like human; vision models such as Segment Anything Model (SAM) and GPT-4Vision assist LLM understand spatial and semantic information at the same time. Ultimately, the LLM uses its summarizing and reasoning capabilities to classify and segment the query image. The proposed method's modular framework makes it easily extendable. Our approach achieves state-of-the-art performance on the Pascal-5i dataset.

Title: Applications of Large Scale Foundation Models for Autonomous Driving. (arXiv:2311.12144v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12144
• Code URL: null
• Copy Paste: [[2311.12144]] Applications of Large Scale Foundation Models for Autonomous Driving(http://arxiv.org/abs/2311.12144)
• Summary:

  Since DARPA Grand Challenges (rural) in 2004/05 and Urban Challenges in 2007, autonomous driving has been the most active field of AI applications. Recently powered by large language models (LLMs), chat systems, such as chatGPT and PaLM, emerge and rapidly become a promising direction to achieve artificial general intelligence (AGI) in natural language processing (NLP). There comes a natural thinking that we could employ these abilities to reformulate autonomous driving. By combining LLM with foundation models, it is possible to utilize the human knowledge, commonsense and reasoning to rebuild autonomous driving systems from the current long-tailed AI dilemma. In this paper, we investigate the techniques of foundation models and LLMs applied for autonomous driving, categorized as simulation, world model, data annotation and planning or E2E solutions etc.

Title: Boosting Audio-visual Zero-shot Learning with Large Language Models. (arXiv:2311.12268v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12268
• Code URL: https://github.com/chenhaoxing/KDA
• Copy Paste: [[2311.12268]] Boosting Audio-visual Zero-shot Learning with Large Language Models(http://arxiv.org/abs/2311.12268)
• Summary:

  Audio-visual zero-shot learning aims to recognize unseen categories based on paired audio-visual sequences. Recent methods mainly focus on learning aligned and discriminative multi-modal features to boost generalization towards unseen categories. However, these approaches ignore the obscure action concepts in category names and may inevitably introduce complex network structures with difficult training objectives. In this paper, we propose a simple yet effective framework named Knowledge-aware Distribution Adaptation (KDA) to help the model better grasp the novel action contents with an external knowledge base. Specifically, we first propose using large language models to generate rich descriptions from category names, which leads to a better understanding of unseen categories. Additionally, we propose a distribution alignment loss as well as a knowledge-aware adaptive margin loss to further improve the generalization ability towards unseen categories. Extensive experimental results demonstrate that our proposed KDA can outperform state-of-the-art methods on three popular audio-visual zero-shot learning datasets. Our code will be avaliable at \url{https://github.com/chenhaoxing/KDA}.

Title: Unifying Corroborative and Contributive Attributions in Large Language Models. (arXiv:2311.12233v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.12233
• Code URL: null
• Copy Paste: [[2311.12233]] Unifying Corroborative and Contributive Attributions in Large Language Models(http://arxiv.org/abs/2311.12233)
• Summary:

  As businesses, products, and services spring up around large language models, the trustworthiness of these models hinges on the verifiability of their outputs. However, methods for explaining language model outputs largely fall across two distinct fields of study which both use the term "attribution" to refer to entirely separate techniques: citation generation and training data attribution. In many modern applications, such as legal document generation and medical question answering, both types of attributions are important. In this work, we argue for and present a unified framework of large language model attributions. We show how existing methods of different types of attribution fall under the unified framework. We also use the framework to discuss real-world use cases where one or both types of attributions are required. We believe that this unified framework will guide the use case driven development of systems that leverage both types of attribution, as well as the standardization of their evaluation.

Title: Enabling On-Device Large Language Model Personalization with Self-Supervised Data Selection and Synthesis. (arXiv:2311.12275v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.12275
• Code URL: null
• Copy Paste: [[2311.12275]] Enabling On-Device Large Language Model Personalization with Self-Supervised Data Selection and Synthesis(http://arxiv.org/abs/2311.12275)
• Summary:

  After a large language model (LLM) is deployed on edge devices, it is desirable for these devices to learn from user-generated conversation data to generate user-specific and personalized responses in real-time. However, user-generated data usually contains sensitive and private information, and uploading such data to the cloud for annotation is not preferred if not prohibited. While it is possible to obtain annotation locally by directly asking users to provide preferred responses, such annotations have to be sparse to not affect user experience. In addition, the storage of edge devices is usually too limited to enable large-scale fine-tuning with full user-generated data. It remains an open question how to enable on-device LLM personalization, considering sparse annotation and limited on-device storage. In this paper, we propose a novel framework to select and store the most representative data online in a self-supervised way. Such data has a small memory footprint and allows infrequent requests of user annotations for further fine-tuning. To enhance fine-tuning quality, multiple semantically similar pairs of question texts and expected responses are generated using the LLM. Our experiments show that the proposed framework achieves the best user-specific content-generating capability (accuracy) and fine-tuning speed (performance) compared with vanilla baselines. To the best of our knowledge, this is the very first on-device LLM personalization framework.

Title: ATLANTIC: Structure-Aware Retrieval-Augmented Language Model for Interdisciplinary Science. (arXiv:2311.12289v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.12289
• Code URL: null
• Copy Paste: [[2311.12289]] ATLANTIC: Structure-Aware Retrieval-Augmented Language Model for Interdisciplinary Science(http://arxiv.org/abs/2311.12289)
• Summary:

  Large language models record impressive performance on many natural language processing tasks. However, their knowledge capacity is limited to the pretraining corpus. Retrieval augmentation offers an effective solution by retrieving context from external knowledge sources to complement the language model. However, existing retrieval augmentation techniques ignore the structural relationships between these documents. Furthermore, retrieval models are not explored much in scientific tasks, especially in regard to the faithfulness of retrieved documents. In this paper, we propose a novel structure-aware retrieval augmented language model that accommodates document structure during retrieval augmentation. We create a heterogeneous document graph capturing multiple types of relationships (e.g., citation, co-authorship, etc.) that connect documents from more than 15 scientific disciplines (e.g., Physics, Medicine, Chemistry, etc.). We train a graph neural network on the curated document graph to act as a structural encoder for the corresponding passages retrieved during the model pretraining. Particularly, along with text embeddings of the retrieved passages, we obtain structural embeddings of the documents (passages) and fuse them together before feeding them to the language model. We evaluate our model extensively on various scientific benchmarks that include science question-answering and scientific document classification tasks. Experimental results demonstrate that structure-aware retrieval improves retrieving more coherent, faithful and contextually relevant passages, while showing a comparable performance in the overall accuracy.

Title: AcademicGPT: Empowering Academic Research. (arXiv:2311.12315v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.12315
• Code URL: null
• Copy Paste: [[2311.12315]] AcademicGPT: Empowering Academic Research(http://arxiv.org/abs/2311.12315)
• Summary:

  Large Language Models (LLMs) have demonstrated exceptional capabilities across various natural language processing tasks. Yet, many of these advanced LLMs are tailored for broad, general-purpose applications. In this technical report, we introduce AcademicGPT, designed specifically to empower academic research. AcademicGPT is a continual training model derived from LLaMA2-70B. Our training corpus mainly consists of academic papers, thesis, content from some academic domain, high-quality Chinese data and others. While it may not be extensive in data scale, AcademicGPT marks our initial venture into a domain-specific GPT tailored for research area. We evaluate AcademicGPT on several established public benchmarks such as MMLU and CEval, as well as on some specialized academic benchmarks like PubMedQA, SCIEval, and our newly-created ComputerScienceQA, to demonstrate its ability from general knowledge ability, to Chinese ability, and to academic ability. Building upon AcademicGPT's foundation model, we also developed several applications catered to the academic area, including General Academic Question Answering, AI-assisted Paper Reading, Paper Review, and AI-assisted Title and Abstract Generation.

Title: A Survey of Graph Meets Large Language Model: Progress and Future Directions. (arXiv:2311.12399v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.12399
• Code URL: null
• Copy Paste: [[2311.12399]] A Survey of Graph Meets Large Language Model: Progress and Future Directions(http://arxiv.org/abs/2311.12399)
• Summary:

  Graph plays a significant role in representing and analyzing complex relationships in real-world applications such as citation networks, social networks, and biological data. Recently, Large Language Models (LLMs), which have achieved tremendous success in various domains, have also been leveraged in graph-related tasks to surpass traditional Graph Neural Networks (GNNs) based methods and yield state-of-the-art performance. In this survey, we first present a comprehensive review and analysis of existing methods that integrate LLMs with graphs. First of all, we propose a new taxonomy, which organizes existing methods into three categories based on the role (i.e., enhancer, predictor, and alignment component) played by LLMs in graph-related tasks. Then we systematically survey the representative methods along the three categories of the taxonomy. Finally, we discuss the remaining limitations of existing studies and highlight promising avenues for future research. The relevant papers are summarized and will be consistently updated at: https://github.com/yhLeeee/Awesome-LLMs-in-Graph-tasks.

Title: nach0: Multimodal Natural and Chemical Languages Foundation Model. (arXiv:2311.12410v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.12410
• Code URL: null
• Copy Paste: [[2311.12410]] nach0: Multimodal Natural and Chemical Languages Foundation Model(http://arxiv.org/abs/2311.12410)
• Summary:

  Large Language Models (LLMs) have substantially driven scientific progress in various domains, and many papers have demonstrated their ability to tackle complex problems with creative solutions. Our paper introduces a new foundation model, nach0, capable of solving various chemical and biological tasks: biomedical question answering, named entity recognition, molecular generation, molecular synthesis, attributes prediction, and others. nach0 is a multi-domain and multi-task encoder-decoder LLM pre-trained on unlabeled text from scientific literature, patents, and molecule strings to incorporate a range of chemical and linguistic knowledge. We employed instruction tuning, where specific task-related instructions are utilized to fine-tune nach0 for the final set of tasks. To train nach0 effectively, we leverage the NeMo framework, enabling efficient parallel optimization of both base and large model versions. Extensive experiments demonstrate that our model outperforms state-of-the-art baselines on single-domain and cross-domain tasks. Furthermore, it can generate high-quality outputs in molecular and textual formats, showcasing its effectiveness in multi-domain setups.

Title: Oasis: Data Curation and Assessment System for Pretraining of Large Language Models. (arXiv:2311.12537v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.12537
• Code URL: https://github.com/tongzhou21/oasis
• Copy Paste: [[2311.12537]] Oasis: Data Curation and Assessment System for Pretraining of Large Language Models(http://arxiv.org/abs/2311.12537)
• Summary:

  Data is one of the most critical elements in building a large language model. However, existing systems either fail to customize a corpus curation pipeline or neglect to leverage comprehensive corpus assessment for iterative optimization of the curation. To this end, we present a pretraining corpus curation and assessment platform called Oasis -- a one-stop system for data quality improvement and quantification with user-friendly interactive interfaces. Specifically, the interactive modular rule filter module can devise customized rules according to explicit feedback. The debiased neural filter module builds the quality classification dataset in a negative-centric manner to remove the undesired bias. The adaptive document deduplication module could execute large-scale deduplication with limited memory resources. These three parts constitute the customized data curation module. And in the holistic data assessment module, a corpus can be assessed in local and global views, with three evaluation means including human, GPT-4, and heuristic metrics. We exhibit a complete process to use Oasis for the curation and assessment of pretraining data. In addition, an 800GB bilingual corpus curated by Oasis is publicly released.

Title: In-Context Learning Functions with Varying Number of Minima. (arXiv:2311.12538v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.12538
• Code URL: https://github.com/pittnail/icl-minima
• Copy Paste: [[2311.12538]] In-Context Learning Functions with Varying Number of Minima(http://arxiv.org/abs/2311.12538)
• Summary:

  Large Language Models (LLMs) have proven effective at In-Context Learning (ICL), an ability that allows them to create predictors from labeled examples. Few studies have explored the interplay between ICL and specific properties of functions it attempts to approximate. In our study, we use a formal framework to explore ICL and propose a new task of approximating functions with varying number of minima. We implement a method that allows for producing functions with given inputs as minima. We find that increasing the number of minima degrades ICL performance. At the same time, our evaluation shows that ICL outperforms 2-layer Neural Network (2NN) model. Furthermore, ICL learns faster than 2NN in all settings. We validate the findings through a set of few-shot experiments across various hyperparameter configurations.

Title: IMGTB: A Framework for Machine-Generated Text Detection Benchmarking. (arXiv:2311.12574v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.12574
• Code URL: null
• Copy Paste: [[2311.12574]] IMGTB: A Framework for Machine-Generated Text Detection Benchmarking(http://arxiv.org/abs/2311.12574)
• Summary:

  In the era of large language models generating high quality texts, it is a necessity to develop methods for detection of machine-generated text to avoid harmful use or simply due to annotation purposes. It is, however, also important to properly evaluate and compare such developed methods. Recently, a few benchmarks have been proposed for this purpose; however, integration of newest detection methods is rather challenging, since new methods appear each month and provide slightly different evaluation pipelines. In this paper, we present the IMGTB framework, which simplifies the benchmarking of machine-generated text detection methods by easy integration of custom (new) methods and evaluation datasets. Its configurability and flexibility makes research and development of new detection methods easier, especially their comparison to the existing state-of-the-art detectors. The default set of analyses, metrics and visualizations offered by the tool follows the established practices of machine-generated text detection benchmarking found in state-of-the-art literature.

Title: Can Large Language Models Understand Content and Propagation for Misinformation Detection: An Empirical Study. (arXiv:2311.12699v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.12699
• Code URL: null
• Copy Paste: [[2311.12699]] Can Large Language Models Understand Content and Propagation for Misinformation Detection: An Empirical Study(http://arxiv.org/abs/2311.12699)
• Summary:

  Large Language Models (LLMs) have garnered significant attention for their powerful ability in natural language understanding and reasoning. In this paper, we present a comprehensive empirical study to explore the performance of LLMs on misinformation detection tasks. This study stands as the pioneering investigation into the understanding capabilities of multiple LLMs regarding both content and propagation across social media platforms. Our empirical studies on five misinformation detection datasets show that LLMs with diverse prompts achieve comparable performance in text-based misinformation detection but exhibit notably constrained capabilities in comprehending propagation structure compared to existing models in propagation-based misinformation detection. Besides, we further design four instruction-tuned strategies to enhance LLMs for both content and propagation-based misinformation detection. These strategies boost LLMs to actively learn effective features from multiple instances or hard instances, and eliminate irrelevant propagation structures, thereby achieving better detection performance. Extensive experiments further demonstrate LLMs would play a better capacity in content and propagation structure under these proposed strategies and achieve promising detection performance. These findings highlight the potential ability of LLMs to detect misinformation.

Title: ALPHA: AnomaLous Physiological Health Assessment Using Large Language Models. (arXiv:2311.12524v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.12524
• Code URL: https://github.com/mcjacktang/llm-healthassistant
• Copy Paste: [[2311.12524]] ALPHA: AnomaLous Physiological Health Assessment Using Large Language Models(http://arxiv.org/abs/2311.12524)
• Summary:

  This study concentrates on evaluating the efficacy of Large Language Models (LLMs) in healthcare, with a specific focus on their application in personal anomalous health monitoring. Our research primarily investigates the capabilities of LLMs in interpreting and analyzing physiological data obtained from FDA-approved devices. We conducted an extensive analysis using anomalous physiological data gathered in a simulated low-air-pressure plateau environment. This allowed us to assess the precision and reliability of LLMs in understanding and evaluating users' health status with notable specificity. Our findings reveal that LLMs exhibit exceptional performance in determining medical indicators, including a Mean Absolute Error (MAE) of less than 1 beat per minute for heart rate and less than 1% for oxygen saturation (SpO2). Furthermore, the Mean Absolute Percentage Error (MAPE) for these evaluations remained below 1%, with the overall accuracy of health assessments surpassing 85%. In image analysis tasks, such as interpreting photoplethysmography (PPG) data, our specially adapted GPT models demonstrated remarkable proficiency, achieving less than 1 bpm error in cycle count and 7.28 MAE for heart rate estimation. This study highlights LLMs' dual role as health data analysis tools and pivotal elements in advanced AI health assistants, offering personalized health insights and recommendations within the future health assistant framework.

segmentation

Title: LABELMAKER: Automatic Semantic Label Generation from RGB-D Trajectories. (arXiv:2311.12174v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12174
• Code URL: null
• Copy Paste: [[2311.12174]] LABELMAKER: Automatic Semantic Label Generation from RGB-D Trajectories(http://arxiv.org/abs/2311.12174)
• Summary:

  Semantic annotations are indispensable to train or evaluate perception models, yet very costly to acquire. This work introduces a fully automated 2D/3D labeling framework that, without any human intervention, can generate labels for RGB-D scans at equal (or better) level of accuracy than comparable manually annotated datasets such as ScanNet. Our approach is based on an ensemble of state-of-the-art segmentation models and 3D lifting through neural rendering. We demonstrate the effectiveness of our LabelMaker pipeline by generating significantly better labels for the ScanNet datasets and automatically labelling the previously unlabeled ARKitScenes dataset. Code and models are available at https://labelmaker.org

Title: Instance-aware 3D Semantic Segmentation powered by Shape Generators and Classifiers. (arXiv:2311.12291v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12291
• Code URL: null
• Copy Paste: [[2311.12291]] Instance-aware 3D Semantic Segmentation powered by Shape Generators and Classifiers(http://arxiv.org/abs/2311.12291)
• Summary:

  Existing 3D semantic segmentation methods rely on point-wise or voxel-wise feature descriptors to output segmentation predictions. However, these descriptors are often supervised at point or voxel level, leading to segmentation models that can behave poorly at instance-level. In this paper, we proposed a novel instance-aware approach for 3D semantic segmentation. Our method combines several geometry processing tasks supervised at instance-level to promote the consistency of the learned feature representation. Specifically, our methods use shape generators and shape classifiers to perform shape reconstruction and classification tasks for each shape instance. This enforces the feature representation to faithfully encode both structural and local shape information, with an awareness of shape instances. In the experiments, our method significantly outperform existing approaches in 3D semantic segmentation on several public benchmarks, such as Waymo Open Dataset, SemanticKITTI and ScanNetV2.

Title: Semi-supervised Medical Image Segmentation via Query Distribution Consistency. (arXiv:2311.12364v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12364
• Code URL: null
• Copy Paste: [[2311.12364]] Semi-supervised Medical Image Segmentation via Query Distribution Consistency(http://arxiv.org/abs/2311.12364)
• Summary:

  Semi-supervised learning is increasingly popular in medical image segmentation due to its ability to leverage large amounts of unlabeled data to extract additional information. However, most existing semi-supervised segmentation methods focus only on extracting information from unlabeled data. In this paper, we propose a novel Dual KMax UX-Net framework that leverages labeled data to guide the extraction of information from unlabeled data. Our approach is based on a mutual learning strategy that incorporates two modules: 3D UX-Net as our backbone meta-architecture and KMax decoder to enhance the segmentation performance. Extensive experiments on the Atrial Segmentation Challenge dataset have shown that our method can significantly improve performance by merging unlabeled data. Meanwhile, our framework outperforms state-of-the-art semi-supervised learning methods on 10\% and 20\% labeled settings. Code located at: https://github.com/Rows21/DK-UXNet.

Title: CASR: Refining Action Segmentation via Magrinalizing Frame-levle Causal Relationships. (arXiv:2311.12401v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12401
• Code URL: null
• Copy Paste: [[2311.12401]] CASR: Refining Action Segmentation via Magrinalizing Frame-levle Causal Relationships(http://arxiv.org/abs/2311.12401)
• Summary:

  Integrating deep learning and causal discovery has increased the interpretability of Temporal Action Segmentation (TAS) tasks. However, frame-level causal relationships exist many complicated noises outside the segment-level, making it infeasible to directly express macro action semantics. Thus, we propose \textit{\textbf{Causal Abstraction Segmentation Refiner (CASR)}}, which can refine TAS results from various models by enhancing video causality in marginalizing frame-level casual relationships. Specifically, we define the equivalent frame-level casual model and segment-level causal model, so that the causal adjacency matrix constructed from marginalized frame-level causal relationships has the ability to represent the segmnet-level causal relationships. CASR works out by reducing the difference in the causal adjacency matrix between we constructed and pre-segmentation results of backbone models. In addition, we propose a novel evaluation metric Causal Edit Distance (CED) to evaluate the causal interpretability. Extensive experimental results on mainstream datasets indicate that CASR significantly surpasses existing various methods in action segmentation performance, as well as in causal explainability and generalization. Our code will be available soon.

Title: Learning Site-specific Styles for Multi-institutional Unsupervised Cross-modality Domain Adaptation. (arXiv:2311.12437v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12437
• Code URL: null
• Copy Paste: [[2311.12437]] Learning Site-specific Styles for Multi-institutional Unsupervised Cross-modality Domain Adaptation(http://arxiv.org/abs/2311.12437)
• Summary:

  Unsupervised cross-modality domain adaptation is a challenging task in medical image analysis, and it becomes more challenging when source and target domain data are collected from multiple institutions. In this paper, we present our solution to tackle the multi-institutional unsupervised domain adaptation for the crossMoDA 2023 challenge. First, we perform unpaired image translation to translate the source domain images to the target domain, where we design a dynamic network to generate synthetic target domain images with controllable, site-specific styles. Afterwards, we train a segmentation model using the synthetic images and further reduce the domain gap by self-training. Our solution achieved the 1st place during both the validation and testing phases of the challenge.

Title: MaskFlow: Object-Aware Motion Estimation. (arXiv:2311.12476v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12476
• Code URL: null
• Copy Paste: [[2311.12476]] MaskFlow: Object-Aware Motion Estimation(http://arxiv.org/abs/2311.12476)
• Summary:

  We introduce a novel motion estimation method, MaskFlow, that is capable of estimating accurate motion fields, even in very challenging cases with small objects, large displacements and drastic appearance changes. In addition to lower-level features, that are used in other Deep Neural Network (DNN)-based motion estimation methods, MaskFlow draws from object-level features and segmentations. These features and segmentations are used to approximate the objects' translation motion field. We propose a novel and effective way of incorporating the incomplete translation motion field into a subsequent motion estimation network for refinement and completion. We also produced a new challenging synthetic dataset with motion field ground truth, and also provide extra ground truth for the object-instance matchings and corresponding segmentation masks. We demonstrate that MaskFlow outperforms state of the art methods when evaluated on our new challenging dataset, whilst still producing comparable results on the popular FlyingThings3D benchmark dataset.

Title: HCA-Net: Hierarchical Context Attention Network for Intervertebral Disc Semantic Labeling. (arXiv:2311.12486v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12486
• Code URL: null
• Copy Paste: [[2311.12486]] HCA-Net: Hierarchical Context Attention Network for Intervertebral Disc Semantic Labeling(http://arxiv.org/abs/2311.12486)
• Summary:

  Accurate and automated segmentation of intervertebral discs (IVDs) in medical images is crucial for assessing spine-related disorders, such as osteoporosis, vertebral fractures, or IVD herniation. We present HCA-Net, a novel contextual attention network architecture for semantic labeling of IVDs, with a special focus on exploiting prior geometric information. Our approach excels at processing features across different scales and effectively consolidating them to capture the intricate spatial relationships within the spinal cord. To achieve this, HCA-Net models IVD labeling as a pose estimation problem, aiming to minimize the discrepancy between each predicted IVD location and its corresponding actual joint location. In addition, we introduce a skeletal loss term to reinforce the model's geometric dependence on the spine. This loss function is designed to constrain the model's predictions to a range that matches the general structure of the human vertebral skeleton. As a result, the network learns to reduce the occurrence of false predictions and adaptively improves the accuracy of IVD location estimation. Through extensive experimental evaluation on multi-center spine datasets, our approach consistently outperforms previous state-of-the-art methods on both MRI T1w and T2w modalities. The codebase is accessible to the public on \href{https://github.com/xmindflow/HCA-Net}{GitHub}.

Title: Leveraging Unlabeled Data for 3D Medical Image Segmentation through Self-Supervised Contrastive Learning. (arXiv:2311.12617v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12617
• Code URL: null
• Copy Paste: [[2311.12617]] Leveraging Unlabeled Data for 3D Medical Image Segmentation through Self-Supervised Contrastive Learning(http://arxiv.org/abs/2311.12617)
• Summary:

  Current 3D semi-supervised segmentation methods face significant challenges such as limited consideration of contextual information and the inability to generate reliable pseudo-labels for effective unsupervised data use. To address these challenges, we introduce two distinct subnetworks designed to explore and exploit the discrepancies between them, ultimately correcting the erroneous prediction results. More specifically, we identify regions of inconsistent predictions and initiate a targeted verification training process. This procedure strategically fine-tunes and harmonizes the predictions of the subnetworks, leading to enhanced utilization of contextual information. Furthermore, to adaptively fine-tune the network's representational capacity and reduce prediction uncertainty, we employ a self-supervised contrastive learning paradigm. For this, we use the network's confidence to distinguish between reliable and unreliable predictions. The model is then trained to effectively minimize unreliable predictions. Our experimental results for organ segmentation, obtained from clinical MRI and CT scans, demonstrate the effectiveness of our approach when compared to state-of-the-art methods. The codebase is accessible on \href{https://github.com/xmindflow/SSL-contrastive}{GitHub}.

Title: Mobile-Seed: Joint Semantic Segmentation and Boundary Detection for Mobile Robots. (arXiv:2311.12651v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12651
• Code URL: null
• Copy Paste: [[2311.12651]] Mobile-Seed: Joint Semantic Segmentation and Boundary Detection for Mobile Robots(http://arxiv.org/abs/2311.12651)
• Summary:

  Precise and rapid delineation of sharp boundaries and robust semantics is essential for numerous downstream robotic tasks, such as robot grasping and manipulation, real-time semantic mapping, and online sensor calibration performed on edge computing units. Although boundary detection and semantic segmentation are complementary tasks, most studies focus on lightweight models for semantic segmentation but overlook the critical role of boundary detection. In this work, we introduce Mobile-Seed, a lightweight, dual-task framework tailored for simultaneous semantic segmentation and boundary detection. Our framework features a two-stream encoder, an active fusion decoder (AFD) and a dual-task regularization approach. The encoder is divided into two pathways: one captures category-aware semantic information, while the other discerns boundaries from multi-scale features. The AFD module dynamically adapts the fusion of semantic and boundary information by learning channel-wise relationships, allowing for precise weight assignment of each channel. Furthermore, we introduce a regularization loss to mitigate the conflicts in dual-task learning and deep diversity supervision. Compared to existing methods, the proposed Mobile-Seed offers a lightweight framework to simultaneously improve semantic segmentation performance and accurately locate object boundaries. Experiments on the Cityscapes dataset have shown that Mobile-Seed achieves notable improvement over the state-of-the-art (SOTA) baseline by 2.2 percentage points (pp) in mIoU and 4.2 pp in mF-score, while maintaining an online inference speed of 23.9 frames-per-second (FPS) with 1024x2048 resolution input on an RTX 2080 Ti GPU. Additional experiments on CamVid and PASCAL Context datasets confirm our method's generalizability. Code and additional results are publicly available at \url{https://martin-liao.github.io/Mobile-Seed/}.

Title: Transferring to Real-World Layouts: A Depth-aware Framework for Scene Adaptation. (arXiv:2311.12682v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12682
• Code URL: null
• Copy Paste: [[2311.12682]] Transferring to Real-World Layouts: A Depth-aware Framework for Scene Adaptation(http://arxiv.org/abs/2311.12682)
• Summary:

  Scene segmentation via unsupervised domain adaptation (UDA) enables the transfer of knowledge acquired from source synthetic data to real-world target data, which largely reduces the need for manual pixel-level annotations in the target domain. To facilitate domain-invariant feature learning, existing methods typically mix data from both the source domain and target domain by simply copying and pasting the pixels. Such vanilla methods are usually sub-optimal since they do not take into account how well the mixed layouts correspond to real-world scenarios. Real-world scenarios are with an inherent layout. We observe that semantic categories, such as sidewalks, buildings, and sky, display relatively consistent depth distributions, and could be clearly distinguished in a depth map. Based on such observation, we propose a depth-aware framework to explicitly leverage depth estimation to mix the categories and facilitate the two complementary tasks, i.e., segmentation and depth learning in an end-to-end manner. In particular, the framework contains a Depth-guided Contextual Filter (DCF) forndata augmentation and a cross-task encoder for contextual learning. DCF simulates the real-world layouts, while the cross-task encoder further adaptively fuses the complementing features between two tasks. Besides, it is worth noting that several public datasets do not provide depth annotation. Therefore, we leverage the off-the-shelf depth estimation network to generate the pseudo depth. Extensive experiments show that our proposed methods, even with pseudo depth, achieve competitive performance on two widely-used bench-marks, i.e. 77.7 mIoU on GTA to Cityscapes and 69.3 mIoU on Synthia to Cityscapes.

Title: ChronoPscychosis: Temporal Segmentation and Its Impact on Schizophrenia Classification Using Motor Activity Data. (arXiv:2311.12590v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.12590
• Code URL: null
• Copy Paste: [[2311.12590]] ChronoPscychosis: Temporal Segmentation and Its Impact on Schizophrenia Classification Using Motor Activity Data(http://arxiv.org/abs/2311.12590)
• Summary:

  Schizophrenia is a complicated mental illness characterized by a broad spectrum of symptoms affecting cognition, behavior, and emotion. The task of identifying reliable biomarkers to classify Schizophrenia accurately continues to be a challenge in the field of psychiatry. We investigate the temporal patterns within the motor activity data as a potential key to enhancing the categorization of individuals with Schizophrenia, using the dataset having motor activity recordings of 22 Schizophrenia patients and 32 control subjects. The dataset contains per-minute motor activity measurements collected for an average of 12.7 days in a row for each participant. We dissect each day into segments (Twelve, Eight, six, four, three, and two parts) and evaluate their impact on classification. We employ sixteen statistical features within these temporal segments and train them on Seven machine learning models to get deeper insights. LightGBM model outperforms the other six models. Our results indicate that the temporal segmentation significantly improves the classification, with AUC-ROC = 0.93, F1 score = 0.84( LightGBM- without any segmentation) and AUC-ROC = 0.98, F1 score = 0.93( LightGBM- with segmentation). Distinguishing between diurnal and nocturnal segments amplifies the differences between Schizophrenia patients and controls. However, further subdivisions into smaller time segments do not affect the AUC- ROC significantly. Morning, afternoon, evening, and night partitioning gives similar classification performance to day-night partitioning. These findings are valuable as they indicate that extensive temporal classification beyond distinguishing between day and night does not yield substantial results, offering an efficient approach for further classification, early diagnosis, and monitoring of Schizophrenia.