secure

Title: Tamper-Evident Pairing. (arXiv:2311.14790v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.14790
• Code URL: null
• Copy Paste: [[2311.14790]] Tamper-Evident Pairing(http://arxiv.org/abs/2311.14790)
• Summary:

  Establishing a secure connection between wireless devices has become significantly important with the increasing number of Wi-Fi products coming to the market. In order to provide an easy and secure pairing standard, the Wi-Fi Alliance has designed the Wi-Fi Protected Setup. Push-Button Configuration (PBC) is part of this standard and is especially useful for pairing devices with physical limitations. However, PBC is proven to be vulnerable to man-in-the-middle (MITM) attacks. Tamper-Evident Pairing (TEP) is an improvement of the PBC standard, which aims to fix the MITM vulnerability without interfering the useful properties of PBC. It relies on the Tamper-Evident Announcement (TEA), which guarantees that an adversary can neither tamper a transmitted message without being detected, nor hide the fact that the message has been sent. The security properties of TEP were proven manually by its authors and tested with the Uppaal and Spin model checkers. During the Uppaal model checking, no vulnerabilities were found. However, the Spin model revealed a case, in which the TEP's security is not guaranteed. In this paper, we first provide a comprehensive overview of the TEP protocol, including all information needed to understand how it works. Furthermore, we summarize the security checks performed on it, give the circumstances, under which it is no longer resistant to MITM attacks and explain the reasons why they could not be revealed with the first model. Nevertheless, future work is required to gain full certainty of the TEP's security before applying it in the industry.

Title: Lightweight Public Key Encryption in Post-Quantum Computing Era. (arXiv:2311.14845v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.14845
• Code URL: null
• Copy Paste: [[2311.14845]] Lightweight Public Key Encryption in Post-Quantum Computing Era(http://arxiv.org/abs/2311.14845)
• Summary:

  Confidentiality in our digital world is based on the security of cryptographic algorithms. These are usually executed transparently in the background, with people often relying on them without further knowledge. In the course of technological progress with quantum computers, the protective function of common encryption algorithms is threatened. This particularly affects public-key methods such as RSA and DH based on discrete logarithms and prime factorization. Our concept describes the transformation of a classical asymmetric encryption method to a modern complexity class. Thereby the approach of Cramer-Shoup is put on the new basis of elliptic curves. The system is provable cryptographically strong, especially against adaptive chosen-ciphertext attacks. In addition, the new method features small key lengths, making it suitable for Internet-of-Things. It represents an intermediate step towards an encryption scheme based on isogeny elliptic curves. This approach shows a way to a secure encryption scheme for the post-quantum computing era.

security

Title: Effective Backdoor Mitigation Depends on the Pre-training Objective. (arXiv:2311.14948v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.14948
• Code URL: null
• Copy Paste: [[2311.14948]] Effective Backdoor Mitigation Depends on the Pre-training Objective(http://arxiv.org/abs/2311.14948)
• Summary:

  Despite the advanced capabilities of contemporary machine learning (ML) models, they remain vulnerable to adversarial and backdoor attacks. This vulnerability is particularly concerning in real-world deployments, where compromised models may exhibit unpredictable behavior in critical scenarios. Such risks are heightened by the prevalent practice of collecting massive, internet-sourced datasets for pre-training multimodal models, as these datasets may harbor backdoors. Various techniques have been proposed to mitigate the effects of backdooring in these models such as CleanCLIP which is the current state-of-the-art approach.

  In this work, we demonstrate that the efficacy of CleanCLIP in mitigating backdoors is highly dependent on the particular objective used during model pre-training.

  We observe that stronger pre-training objectives correlate with harder to remove backdoors behaviors. We show this by training multimodal models on two large datasets consisting of 3 million (CC3M) and 6 million (CC6M) datapoints, under various pre-training objectives, followed by poison removal using CleanCLIP. We find that CleanCLIP is ineffective when stronger pre-training objectives are used, even with extensive hyperparameter tuning.

  Our findings underscore critical considerations for ML practitioners who pre-train models using large-scale web-curated data and are concerned about potential backdoor threats. Notably, our results suggest that simpler pre-training objectives are more amenable to effective backdoor removal. This insight is pivotal for practitioners seeking to balance the trade-offs between using stronger pre-training objectives and security against backdoor attacks.

Title: On-Device Soft Sensors: Real-Time Fluid Flow Estimation from Level Sensor Data. (arXiv:2311.15036v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.15036
• Code URL: null
• Copy Paste: [[2311.15036]] On-Device Soft Sensors: Real-Time Fluid Flow Estimation from Level Sensor Data(http://arxiv.org/abs/2311.15036)
• Summary:

  Soft sensors are crucial in bridging autonomous systems' physical and digital realms, enhancing sensor fusion and perception. Instead of deploying soft sensors on the Cloud, this study shift towards employing on-device soft sensors, promising heightened efficiency and bolstering data security. Our approach substantially improves energy efficiency by deploying Artificial Intelligence (AI) directly on devices within a wireless sensor network. Furthermore, the synergistic integration of the Microcontroller Unit and Field-Programmable Gate Array (FPGA) leverages the rapid AI inference capabilities of the latter. Empirical evidence from our real-world use case demonstrates that FPGA-based soft sensors achieve inference times ranging remarkably from 1.04 to 12.04 microseconds. These compelling results highlight the considerable potential of our innovative approach for executing real-time inference tasks efficiently, thereby presenting a feasible alternative that effectively addresses the latency challenges intrinsic to Cloud-based deployments.

privacy

protect

Title: Double-Flow-based Steganography without Embedding for Image-to-Image Hiding. (arXiv:2311.15027v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.15027
• Code URL: null
• Copy Paste: [[2311.15027]] Double-Flow-based Steganography without Embedding for Image-to-Image Hiding(http://arxiv.org/abs/2311.15027)
• Summary:

  As an emerging concept, steganography without embedding (SWE) hides a secret message without directly embedding it into a cover. Thus, SWE has the unique advantage of being immune to typical steganalysis methods and can better protect the secret message from being exposed. However, existing SWE methods are generally criticized for their poor payload capacity and low fidelity of recovered secret messages. In this paper, we propose a novel steganography-without-embedding technique, named DF-SWE, which addresses the aforementioned drawbacks and produces diverse and natural stego images. Specifically, DF-SWE employs a reversible circulation of double flow to build a reversible bijective transformation between the secret image and the generated stego image. Hence, it provides a way to directly generate stego images from secret images without a cover image. Besides leveraging the invertible property, DF-SWE can invert a secret image from a generated stego image in a nearly lossless manner and increases the fidelity of extracted secret images. To the best of our knowledge, DF-SWE is the first SWE method that can hide large images and multiple images into one image with the same size, significantly enhancing the payload capacity. According to the experimental results, the payload capacity of DF-SWE achieves 24-72 BPP is 8000-16000 times compared to its competitors while producing diverse images to minimize the exposure risk. Importantly, DF-SWE can be applied in the steganography of secret images in various domains without requiring training data from the corresponding domains. This domain-agnostic property suggests that DF-SWE can 1) be applied to hiding private data and 2) be deployed in resource-limited systems.

defense

attack

Title: Trainwreck: A damaging adversarial attack on image classifiers. (arXiv:2311.14772v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.14772
• Code URL: https://github.com/janzahalka/trainwreck
• Copy Paste: [[2311.14772]] Trainwreck: A damaging adversarial attack on image classifiers(http://arxiv.org/abs/2311.14772)
• Summary:

  Adversarial attacks are an important security concern for computer vision (CV), as they enable malicious attackers to reliably manipulate CV models. Existing attacks aim to elicit an output desired by the attacker, but keep the model fully intact on clean data. With CV models becoming increasingly valuable assets in applied practice, a new attack vector is emerging: disrupting the models as a form of economic sabotage. This paper opens up the exploration of damaging adversarial attacks (DAAs) that seek to damage the target model and maximize the total cost incurred by the damage. As a pioneer DAA, this paper proposes Trainwreck, a train-time attack that poisons the training data of image classifiers to degrade their performance. Trainwreck conflates the data of similar classes using stealthy ($\epsilon \leq 8/255$) class-pair universal perturbations computed using a surrogate model. Trainwreck is a black-box, transferable attack: it requires no knowledge of the target model's architecture, and a single poisoned dataset degrades the performance of any model trained on it. The experimental evaluation on CIFAR-10 and CIFAR-100 demonstrates that Trainwreck is indeed an effective attack across various model architectures including EfficientNetV2, ResNeXt-101, and a finetuned ViT-L-16. The strength of the attack can be customized by the poison rate parameter. Finally, data redundancy with file hashing and/or pixel difference are identified as a reliable defense technique against Trainwreck or similar DAAs. The code is available at https://github.com/JanZahalka/trainwreck.

Title: cryptoRAN: A review on cryptojacking and ransomware attacks w.r.t. banking industry -- threats, challenges, & problems. (arXiv:2311.14783v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.14783
• Code URL: null
• Copy Paste: [[2311.14783]] cryptoRAN: A review on cryptojacking and ransomware attacks w(http://arxiv.org/abs/2311.14783)
• Summary:

  In the banking industry, ransomware is a well-known threat, but since the beginning of 2022, cryptojacking, an emerging threat is posing a considerable challenge to the banking industry. Ransomware has variants, and the attackers keep changing the nature of these variants. This review paper studies the complex background of these two threats and scrutinizes the actual challenges, and problems that the banking industry and financial institutions face. These threats, though distinct in nature, share commonalities, such as financial motivations and sophisticated techniques. We focus on examining the newly emerged variants of ransomware while we provide a comprehensive idea of cryptojacking and its nature. This paper involves a detailed breakdown of the specific threats posed by cryptojacking and ransomware. It explores the techniques cybercriminals use, the variabilities they look for, and the potential consequences for financial institutions and their customers. This paper also finds out how cybercriminals change their techniques following the security upgrades, and why financial firms including banks need to be proactive about cyber threats. Additionally, this paper reviews the background study of some existing papers, finds the research gaps that need to be addressed, and provides suggestions including a conclusion and future scope on those disputes. Lastly, we introduce a Digital Forensics and Incident Response (DFIR) approach for up-to-date cyber threat hunting processes for minimizing both cryptojacking and ransomware attacks in the banking industry.

Title: A Comperative Study of Watering Hole Attack Detection Using Supervised Neural Network. (arXiv:2311.15024v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.15024
• Code URL: null
• Copy Paste: [[2311.15024]] A Comperative Study of Watering Hole Attack Detection Using Supervised Neural Network(http://arxiv.org/abs/2311.15024)
• Summary:

  The security landscape necessitates creative solutions to defend against targeted attacks due to the growing sophistication of cyber threats. This study explores the nefarious tactic known as "watering hole attacks using supervised neural networks to detect and prevent these attacks. The neural network identifies patterns in website behavior and network traffic associated with such attacks. Testing on a dataset of confirmed attacks shows a 99% detection rate with a mere 0.1% false positive rate, demonstrating the model's effectiveness. In terms of prevention, the model successfully stops 95% of attacks, providing robust user protection. The study also suggests mitigation strategies, including web filtering solutions, user education, and security controls. Overall, this research presents a promising solution for countering watering hole attacks, offering strong detection, prevention, and mitigation strategies.

robust

Title: SafeSea: Synthetic Data Generation for Adverse & Low Probability Maritime Conditions. (arXiv:2311.14764v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.14764
• Code URL: https://github.com/martin-3240/safesea
• Copy Paste: [[2311.14764]] SafeSea: Synthetic Data Generation for Adverse & Low Probability Maritime Conditions(http://arxiv.org/abs/2311.14764)
• Summary:

  High-quality training data is essential for enhancing the robustness of object detection models. Within the maritime domain, obtaining a diverse real image dataset is particularly challenging due to the difficulty of capturing sea images with the presence of maritime objects , especially in stormy conditions. These challenges arise due to resource limitations, in addition to the unpredictable appearance of maritime objects. Nevertheless, acquiring data from stormy conditions is essential for training effective maritime detection models, particularly for search and rescue, where real-world conditions can be unpredictable. In this work, we introduce SafeSea, which is a stepping stone towards transforming actual sea images with various Sea State backgrounds while retaining maritime objects. Compared to existing generative methods such as Stable Diffusion Inpainting~\cite{stableDiffusion}, this approach reduces the time and effort required to create synthetic datasets for training maritime object detection models. The proposed method uses two automated filters to only pass generated images that meet the criteria. In particular, these filters will first classify the sea condition according to its Sea State level and then it will check whether the objects from the input image are still preserved. This method enabled the creation of the SafeSea dataset, offering diverse weather condition backgrounds to supplement the training of maritime models. Lastly, we observed that a maritime object detection model faced challenges in detecting objects in stormy sea backgrounds, emphasizing the impact of weather conditions on detection accuracy. The code, and dataset are available at https://github.com/martin-3240/SafeSea.

Title: Benchmarking Robustness of Text-Image Composed Retrieval. (arXiv:2311.14837v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.14837
• Code URL: null
• Copy Paste: [[2311.14837]] Benchmarking Robustness of Text-Image Composed Retrieval(http://arxiv.org/abs/2311.14837)
• Summary:

  Text-image composed retrieval aims to retrieve the target image through the composed query, which is specified in the form of an image plus some text that describes desired modifications to the input image. It has recently attracted attention due to its ability to leverage both information-rich images and concise language to precisely express the requirements for target images. However, the robustness of these approaches against real-world corruptions or further text understanding has never been studied. In this paper, we perform the first robustness study and establish three new diversified benchmarks for systematic analysis of text-image composed retrieval against natural corruptions in both vision and text and further probe textural understanding. For natural corruption analysis, we introduce two new large-scale benchmark datasets, CIRR-C and FashionIQ-C for testing in open domain and fashion domain respectively, both of which apply 15 visual corruptions and 7 textural corruptions. For textural understanding analysis, we introduce a new diagnostic dataset CIRR-D by expanding the original raw data with synthetic data, which contains modified text to better probe textual understanding ability including numerical variation, attribute variation, object removal, background variation, and fine-grained evaluation. The code and benchmark datasets are available at https://github.com/SunTongtongtong/Benchmark-Robustness-Text-Image-Compose-Retrieval.

Title: AutoEval-Video: An Automatic Benchmark for Assessing Large Vision Language Models in Open-Ended Video Question Answering. (arXiv:2311.14906v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.14906
• Code URL: https://github.com/xiuyuan-chen/autoeval-video
• Copy Paste: [[2311.14906]] AutoEval-Video: An Automatic Benchmark for Assessing Large Vision Language Models in Open-Ended Video Question Answering(http://arxiv.org/abs/2311.14906)
• Summary:

  We propose a novel and challenging benchmark, AutoEval-Video, to comprehensively evaluate large vision-language models in open-ended video question answering. The comprehensiveness of AutoEval-Video is demonstrated in two aspects: 1) AutoEval-Video constructs open-ended video-questions across 9 skill dimensions, addressing capabilities of perception, comprehension, and generation. 2) AutoEval-Video contains newly collected videos that cover over 40 distinct themes. To efficiently evaluate responses to the open-ended questions, we employ an LLM-based evaluation approach, but instead of merely providing a reference answer, we annotate unique evaluation rules for every single instance (video-question pair). To maximize the robustness of these rules, we develop a novel adversarial annotation mechanism. By using instance-specific rules as prompt, GPT-4, as an automatic evaluator, can achieve a stable evaluation accuracy of around 97.0\%, comparable to the 94.9\% - 97.5\% accuracy of a human evaluator. Furthermore, we assess the performance of eight large vision-language models on AutoEval-Video. Among them, GPT-4V(ision) significantly outperforms other models, achieving an accuracy of 32.2\%. However, there is still substantial room for improvement compared to human accuracy of 72.8\%. By conducting an extensive case study, we uncover several drawbacks of GPT-4V, such as limited temporal and dynamic comprehension, and overly general responses. Code is available at \href{https://github.com/Xiuyuan-Chen/AutoEval-Video}{\color{magenta}https://github.com/Xiuyuan-Chen/AutoEval-Video}.

Title: Coordinate-based Neural Network for Fourier Phase Retrieval. (arXiv:2311.14925v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.14925
• Code URL: null
• Copy Paste: [[2311.14925]] Coordinate-based Neural Network for Fourier Phase Retrieval(http://arxiv.org/abs/2311.14925)
• Summary:

  Fourier phase retrieval is essential for high-definition imaging of nanoscale structures across diverse fields, notably coherent diffraction imaging. This study presents the Single impliCit neurAl Network (SCAN), a tool built upon coordinate neural networks meticulously designed for enhanced phase retrieval performance. Bypassing the pitfalls of conventional iterative methods, which frequently face high computational loads and are prone to noise interference, SCAN adeptly connects object coordinates to their amplitude and phase within a unified network in an unsupervised manner. While many existing methods primarily use Fourier magnitude in their loss function, our approach incorporates both the predicted magnitude and phase, enhancing retrieval accuracy. Comprehensive tests validate SCAN's superiority over traditional and other deep learning models regarding accuracy and noise robustness. We also demonstrate that SCAN excels in the ptychography setting.

Title: OpenNet: Incremental Learning for Autonomous Driving Object Detection with Balanced Loss. (arXiv:2311.14939v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.14939
• Code URL: null
• Copy Paste: [[2311.14939]] OpenNet: Incremental Learning for Autonomous Driving Object Detection with Balanced Loss(http://arxiv.org/abs/2311.14939)
• Summary:

  Automated driving object detection has always been a challenging task in computer vision due to environmental uncertainties. These uncertainties include significant differences in object sizes and encountering the class unseen. It may result in poor performance when traditional object detection models are directly applied to automated driving detection. Because they usually presume fixed categories of common traffic participants, such as pedestrians and cars. Worsely, the huge class imbalance between common and novel classes further exacerbates performance degradation. To address the issues stated, we propose OpenNet to moderate the class imbalance with the Balanced Loss, which is based on Cross Entropy Loss. Besides, we adopt an inductive layer based on gradient reshaping to fast learn new classes with limited samples during incremental learning. To against catastrophic forgetting, we employ normalized feature distillation. By the way, we improve multi-scale detection robustness and unknown class recognition through FPN and energy-based detection, respectively. The Experimental results upon the CODA dataset show that the proposed method can obtain better performance than that of the existing methods.

Title: Task adaption by biologically inspired stochastic comodulation. (arXiv:2311.15053v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.15053
• Code URL: null
• Copy Paste: [[2311.15053]] Task adaption by biologically inspired stochastic comodulation(http://arxiv.org/abs/2311.15053)
• Summary:

  Brain representations must strike a balance between generalizability and adaptability. Neural codes capture general statistical regularities in the world, while dynamically adjusting to reflect current goals. One aspect of this adaptation is stochastically co-modulating neurons' gains based on their task relevance. These fluctuations then propagate downstream to guide decision-making. Here, we test the computational viability of such a scheme in the context of multi-task learning. We show that fine-tuning convolutional networks by stochastic gain modulation improves on deterministic gain modulation, achieving state-of-the-art results on the CelebA dataset. To better understand the mechanisms supporting this improvement, we explore how fine-tuning performance is affected by architecture using Cifar-100. Overall, our results suggest that stochastic comodulation can enhance learning efficiency and performance in multi-task learning, without additional learnable parameters. This offers a promising new direction for developing more flexible and robust intelligent systems.

Title: Data Diversity Matters for Robust Instruction Tuning. (arXiv:2311.14736v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.14736
• Code URL: null
• Copy Paste: [[2311.14736]] Data Diversity Matters for Robust Instruction Tuning(http://arxiv.org/abs/2311.14736)
• Summary:

  Instruction tuning has emerged as a key step in aligning large language models. One of the central challenges of instruction tuning is dataset selection, as the composition of the instruction tuning dataset can significantly impact downstream performance. In particular, researchers have hypothesized that dataset diversity and dataset quality are important indicators of downstream performance. However, it is not clear how to automatically select high quality and diverse data or how exactly quality and diversity affect instruction following ability. To resolve these issues, we propose a new algorithm, Quality-Diversity Instruction Tuning (QDIT). QDIT provides a principled algorithm to control dataset diversity and quality, allowing us to conduct an in depth study on the effect of diversity and quality on instruction tuning performance. From this study we draw two key insights (1) there is a natural tradeoff between dataset diversity and quality and (2) increasing dataset diversity significantly improves the worst case instruction following performance, therefore improving robustness. We validate the performance of QDIT on several large scale instruction tuning datasets, where we find it can improve worst case performance by 18% while maintaining or improving average performance compared to quality driven baselines.

Title: A Baseline Analysis of Reward Models' Ability To Accurately Analyze Foundation Models Under Distribution Shift. (arXiv:2311.14743v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.14743
• Code URL: null
• Copy Paste: [[2311.14743]] A Baseline Analysis of Reward Models' Ability To Accurately Analyze Foundation Models Under Distribution Shift(http://arxiv.org/abs/2311.14743)
• Summary:

  Foundation models, specifically Large Language Models (LLM's), have lately gained wide-spread attention and adoption. Reinforcement Learning with Human Feedback (RLHF) involves training a reward model to capture desired behaviors, which is then used to align an LLM. These reward models are additionally used at inference-time to estimate how well LLM responses adhere to those desired behaviors. However, there is little work measuring how robust these reward models are to distribution shifts. In this work, we evaluate how reward model performance - measured via accuracy and calibration (i.e. alignment between accuracy and confidence) - is affected by distribution shift. We show novel calibration patterns and accuracy drops due to OOD prompts and responses, and that the reward model is more sensitive to shifts in responses than prompts. Additionally, we adapt an OOD detection technique commonly used in classification to the reward model setting in order to detect these distribution shifts in prompts and responses.

Title: Improving Cross-Domain Hate Speech Generalizability with Emotion Knowledge. (arXiv:2311.14865v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.14865
• Code URL: null
• Copy Paste: [[2311.14865]] Improving Cross-Domain Hate Speech Generalizability with Emotion Knowledge(http://arxiv.org/abs/2311.14865)
• Summary:

  Reliable automatic hate speech (HS) detection systems must adapt to the in-flow of diverse new data to curtail hate speech. However, hate speech detection systems commonly lack generalizability in identifying hate speech dissimilar to data used in training, impeding their robustness in real-world deployments. In this work, we propose a hate speech generalization framework that leverages emotion knowledge in a multitask architecture to improve the generalizability of hate speech detection in a cross-domain setting. We investigate emotion corpora with varying emotion categorical scopes to determine the best corpus scope for supplying emotion knowledge to foster generalized hate speech detection. We further assess the relationship between using pretrained Transformers models adapted for hate speech and its effect on our emotion-enriched hate speech generalization model. We perform extensive experiments on six publicly available datasets sourced from different online domains and show that our emotion-enriched HS detection generalization method demonstrates consistent generalization improvement in cross-domain evaluation, increasing generalization performance up to 18.1% and average cross-domain performance up to 8.5%, according to the F1 measure.

Title: Task-Distributionally Robust Data-Free Meta-Learning. (arXiv:2311.14756v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.14756
• Code URL: null
• Copy Paste: [[2311.14756]] Task-Distributionally Robust Data-Free Meta-Learning(http://arxiv.org/abs/2311.14756)
• Summary:

  Data-Free Meta-Learning (DFML) aims to efficiently learn new tasks by leveraging multiple pre-trained models without requiring their original training data. Existing inversion-based DFML methods construct pseudo tasks from a learnable dataset, which is inversely generated from the pre-trained model pool. For the first time, we reveal two major challenges hindering their practical deployments: Task-Distribution Shift (TDS) and Task-Distribution Corruption (TDC). TDS leads to a biased meta-learner because of the skewed task distribution towards newly generated tasks. TDC occurs when untrusted models characterized by misleading labels or poor quality pollute the task distribution. To tackle these issues, we introduce a robust DFML framework that ensures task distributional robustness. We propose to meta-learn from a pseudo task distribution, diversified through task interpolation within a compact task-memory buffer. This approach reduces the meta-learner's overreliance on newly generated tasks by maintaining consistent performance across a broader range of interpolated memory tasks, thus ensuring its generalization for unseen tasks. Additionally, our framework seamlessly incorporates an automated model selection mechanism into the meta-training phase, parameterizing each model's reliability as a learnable weight. This is optimized with a policy gradient algorithm inspired by reinforcement learning, effectively addressing the non-differentiable challenge posed by model selection. Comprehensive experiments across various datasets demonstrate the framework's effectiveness in mitigating TDS and TDC, underscoring its potential to improve DFML in real-world scenarios.

Title: Robust Graph Neural Networks via Unbiased Aggregation. (arXiv:2311.14934v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.14934
• Code URL: null
• Copy Paste: [[2311.14934]] Robust Graph Neural Networks via Unbiased Aggregation(http://arxiv.org/abs/2311.14934)
• Summary:

  The adversarial robustness of Graph Neural Networks (GNNs) has been questioned due to the false sense of security uncovered by strong adaptive attacks despite the existence of numerous defenses. In this work, we delve into the robustness analysis of representative robust GNNs and provide a unified robust estimation point of view to understand their robustness and limitations. Our novel analysis of estimation bias motivates the design of a robust and unbiased graph signal estimator. We then develop an efficient Quasi-Newton iterative reweighted least squares algorithm to solve the estimation problem, which unfolds as robust unbiased aggregation layers in GNNs with a theoretical convergence guarantee. Our comprehensive experiments confirm the strong robustness of our proposed model, and the ablation study provides a deep understanding of its advantages.

biometric

steal

extraction

Title: Parkinson Disease classification Using Contrastive Graph Cross-View Learning with Multimodal Fusion of SPECT Images and Clinical Features. (arXiv:2311.14902v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.14902
• Code URL: null
• Copy Paste: [[2311.14902]] Parkinson Disease classification Using Contrastive Graph Cross-View Learning with Multimodal Fusion of SPECT Images and Clinical Features(http://arxiv.org/abs/2311.14902)
• Summary:

  Parkinson's Disease (PD) is a neurodegenerative neurological disorder that impacts movement and afflicts over 10 million people worldwide. Previous researches have come up with deep learning models for predicting Parkinson's disease primarily using medical images and didn't leverage the manifold structure in the dataset. Our study introduces a multimodal approach with both image and non-image features with a contrastive cross-view graph fusion for Parkinson's disease classification. Specifically, we designed a multimodal co-attention module to integrate embeddings from two distinct graph views derived from low dimensional representation of images and clinical features, enabling the extraction of more stable and structured features from the multiview data. Additionally, we have devised a simplified fusion method utilizing a contrastive loss for positive and negative pairs, to enhance the model's overall cross-view fusion learning capabilities. In our experiments, the graph-view multimodal approach can achieve an accuracy rate of 91% and an AUC of 92.8% in five-fold cross-validation, and it also demonstrates superior predictive capabilities on non-image data as compared to methods that rely solely on machine learning methods.

Title: Optimal Strategies to Perform Multilingual Analysis of Social Content for a Novel Dataset in the Tourism Domain. (arXiv:2311.14727v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.14727
• Code URL: null
• Copy Paste: [[2311.14727]] Optimal Strategies to Perform Multilingual Analysis of Social Content for a Novel Dataset in the Tourism Domain(http://arxiv.org/abs/2311.14727)
• Summary:

  The rising influence of social media platforms in various domains, including tourism, has highlighted the growing need for efficient and automated natural language processing (NLP) approaches to take advantage of this valuable resource. However, the transformation of multilingual, unstructured, and informal texts into structured knowledge often poses significant challenges.

  In this work, we evaluate and compare few-shot, pattern-exploiting and fine-tuning machine learning techniques on large multilingual language models (LLMs) to establish the best strategy to address the lack of annotated data for 3 common NLP tasks in the tourism domain: (1) Sentiment Analysis, (2) Named Entity Recognition, and (3) Fine-grained Thematic Concept Extraction (linked to a semantic resource). Furthermore, we aim to ascertain the quantity of annotated examples required to achieve good performance in those 3 tasks, addressing a common challenge encountered by NLP researchers in the construction of domain-specific datasets.

  Extensive experimentation on a newly collected and annotated multilingual (French, English, and Spanish) dataset composed of tourism-related tweets shows that current few-shot learning techniques allow us to obtain competitive results for all three tasks with very little annotation data: 5 tweets per label (15 in total) for Sentiment Analysis, 10% of the tweets for location detection (around 160) and 13% (200 approx.) of the tweets annotated with thematic concepts, a highly fine-grained sequence labeling task based on an inventory of 315 classes.

  This comparative analysis, grounded in a novel dataset, paves the way for applying NLP to new domain-specific applications, reducing the need for manual annotations and circumventing the complexities of rule-based, ad hoc solutions.

Title: Satellite-based feature extraction and multivariate time-series prediction of biotoxin contamination in shellfish. (arXiv:2311.15000v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.15000
• Code URL: null
• Copy Paste: [[2311.15000]] Satellite-based feature extraction and multivariate time-series prediction of biotoxin contamination in shellfish(http://arxiv.org/abs/2311.15000)
• Summary:

  Shellfish production constitutes an important sector for the economy of many Portuguese coastal regions, yet the challenge of shellfish biotoxin contamination poses both public health concerns and significant economic risks. Thus, predicting shellfish contamination levels holds great potential for enhancing production management and safeguarding public health. In our study, we utilize a dataset with years of Sentinel-3 satellite imagery for marine surveillance, along with shellfish biotoxin contamination data from various production areas along Portugal's western coastline, collected by Portuguese official control. Our goal is to evaluate the integration of satellite data in forecasting models for predicting toxin concentrations in shellfish given forecasting horizons up to four weeks, which implies extracting a small set of useful features and assessing their impact on the predictive models. We framed this challenge as a time-series forecasting problem, leveraging historical contamination levels and satellite images for designated areas. While contamination measurements occurred weekly, satellite images were accessible multiple times per week. Unsupervised feature extraction was performed using autoencoders able to handle non-valid pixels caused by factors like cloud cover, land, or anomalies. Finally, several Artificial Neural Networks models were applied to compare univariate (contamination only) and multivariate (contamination and satellite data) time-series forecasting. Our findings show that incorporating these features enhances predictions, especially beyond one week in lagoon production areas (RIAV) and for the 1-week and 2-week horizons in the L5B area (oceanic). The methodology shows the feasibility of integrating information from a high-dimensional data source like remote sensing without compromising the model's predictive ability.

Title: MPCNN: A Novel Matrix Profile Approach for CNN-based Sleep Apnea Classification. (arXiv:2311.15041v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.15041
• Code URL: https://github.com/vinuni-vishc/mpcnn-sleep-apnea
• Copy Paste: [[2311.15041]] MPCNN: A Novel Matrix Profile Approach for CNN-based Sleep Apnea Classification(http://arxiv.org/abs/2311.15041)
• Summary:

  Sleep apnea (SA) is a significant respiratory condition that poses a major global health challenge. Previous studies have investigated several machine and deep learning models for electrocardiogram (ECG)-based SA diagnoses. Despite these advancements, conventional feature extractions derived from ECG signals, such as R-peaks and RR intervals, may fail to capture crucial information encompassed within the complete PQRST segments. In this study, we propose an innovative approach to address this diagnostic gap by delving deeper into the comprehensive segments of the ECG signal. The proposed methodology draws inspiration from Matrix Profile algorithms, which generate an Euclidean distance profile from fixed-length signal subsequences. From this, we derived the Min Distance Profile (MinDP), Max Distance Profile (MaxDP), and Mean Distance Profile (MeanDP) based on the minimum, maximum, and mean of the profile distances, respectively. To validate the effectiveness of our approach, we use the modified LeNet-5 architecture as the primary CNN model, along with two existing lightweight models, BAFNet and SE-MSCNN, for ECG classification tasks. Our extensive experimental results on the PhysioNet Apnea-ECG dataset revealed that with the new feature extraction method, we achieved a per-segment accuracy up to 92.11 \% and a per-recording accuracy of 100\%. Moreover, it yielded the highest correlation compared to state-of-the-art methods, with a correlation coefficient of 0.989. By introducing a new feature extraction method based on distance relationships, we enhanced the performance of certain lightweight models, showing potential for home sleep apnea test (HSAT) and SA detection in IoT devices. The source code for this work is made publicly available in GitHub: https://github.com/vinuni-vishc/MPCNN-Sleep-Apnea.

membership infer

federate

Title: Eliminating Domain Bias for Federated Learning in Representation Space. (arXiv:2311.14975v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.14975
• Code URL: https://github.com/tsingz0/dbe
• Copy Paste: [[2311.14975]] Eliminating Domain Bias for Federated Learning in Representation Space(http://arxiv.org/abs/2311.14975)
• Summary:

  Recently, federated learning (FL) is popular for its privacy-preserving and collaborative learning abilities. However, under statistically heterogeneous scenarios, we observe that biased data domains on clients cause a representation bias phenomenon and further degenerate generic representations during local training, i.e., the representation degeneration phenomenon. To address these issues, we propose a general framework Domain Bias Eliminator (DBE) for FL. Our theoretical analysis reveals that DBE can promote bi-directional knowledge transfer between server and client, as it reduces the domain discrepancy between server and client in representation space. Besides, extensive experiments on four datasets show that DBE can greatly improve existing FL methods in both generalization and personalization abilities. The DBE-equipped FL method can outperform ten state-of-the-art personalized FL methods by a large margin. Our code is public at https://github.com/TsingZ0/DBE.

fair

interpretability

Title: HyperDID: Hyperspectral Intrinsic Image Decomposition with Deep Feature Embedding. (arXiv:2311.14899v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.14899
• Code URL: null
• Copy Paste: [[2311.14899]] HyperDID: Hyperspectral Intrinsic Image Decomposition with Deep Feature Embedding(http://arxiv.org/abs/2311.14899)
• Summary:

  The dissection of hyperspectral images into intrinsic components through hyperspectral intrinsic image decomposition (HIID) enhances the interpretability of hyperspectral data, providing a foundation for more accurate classification outcomes. However, the classification performance of HIID is constrained by the model's representational ability. To address this limitation, this study rethinks hyperspectral intrinsic image decomposition for classification tasks by introducing deep feature embedding. The proposed framework, HyperDID, incorporates the Environmental Feature Module (EFM) and Categorical Feature Module (CFM) to extract intrinsic features. Additionally, a Feature Discrimination Module (FDM) is introduced to separate environment-related and category-related features. Experimental results across three commonly used datasets validate the effectiveness of HyperDID in improving hyperspectral image classification performance. This novel approach holds promise for advancing the capabilities of hyperspectral image analysis by leveraging deep feature embedding principles. The implementation of the proposed method could be accessed soon at https://github.com/shendu-sw/HyperDID for the sake of reproducibility.

Title: Incorporating granularity bias as the margin into contrastive loss for video captioning. (arXiv:2311.14977v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.14977
• Code URL: null
• Copy Paste: [[2311.14977]] Incorporating granularity bias as the margin into contrastive loss for video captioning(http://arxiv.org/abs/2311.14977)
• Summary:

  Video captioning models easily suffer from long-tail distribution of phrases, which makes captioning models prone to generate vague sentences instead of accurate ones. However, existing debiasing strategies tend to export external knowledge to build dependency trees of words or refine frequency distribution by complex losses and extra input features, which lack interpretability and are hard to train. To mitigate the impact of granularity bias on the model, we introduced a statistical-based bias extractor. This extractor quantifies the information content within sentences and videos, providing an estimate of the likelihood that a video-sentence pair is affected by granularity bias. Furthermore, with the growing trend of integrating contrastive learning methods into video captioning tasks, we use a bidirectional triplet loss to get more negative samples in a batch. Subsequently, we incorporate the margin score into the contrastive learning loss, establishing distinct training objectives for head and tail sentences. This approach facilitates the model's training effectiveness on tail samples. Our simple yet effective loss, incorporating Granularity bias, is referred to as the Margin-Contrastive Loss (GMC Loss). The proposed model demonstrates state-of-the-art performance on MSRVTT with a CIDEr of 57.17, and MSVD, where CIDEr reaches up to 138.68.

Title: Occlusion Sensitivity Analysis with Augmentation Subspace Perturbation in Deep Feature Space. (arXiv:2311.15022v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.15022
• Code URL: null
• Copy Paste: [[2311.15022]] Occlusion Sensitivity Analysis with Augmentation Subspace Perturbation in Deep Feature Space(http://arxiv.org/abs/2311.15022)
• Summary:

  Deep Learning of neural networks has gained prominence in multiple life-critical applications like medical diagnoses and autonomous vehicle accident investigations. However, concerns about model transparency and biases persist. Explainable methods are viewed as the solution to address these challenges. In this study, we introduce the Occlusion Sensitivity Analysis with Deep Feature Augmentation Subspace (OSA-DAS), a novel perturbation-based interpretability approach for computer vision. While traditional perturbation methods make only use of occlusions to explain the model predictions, OSA-DAS extends standard occlusion sensitivity analysis by enabling the integration with diverse image augmentations. Distinctly, our method utilizes the output vector of a DNN to build low-dimensional subspaces within the deep feature vector space, offering a more precise explanation of the model prediction. The structural similarity between these subspaces encompasses the influence of diverse augmentations and occlusions. We test extensively on the ImageNet-1k, and our class- and model-agnostic approach outperforms commonly used interpreters, setting it apart in the realm of explainable AI.

Title: Accurate and interpretable drug-drug interaction prediction enabled by knowledge subgraph learning. (arXiv:2311.15056v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.15056
• Code URL: https://github.com/lars-research/knowddi
• Copy Paste: [[2311.15056]] Accurate and interpretable drug-drug interaction prediction enabled by knowledge subgraph learning(http://arxiv.org/abs/2311.15056)
• Summary:

  Background: Discovering potential drug-drug interactions (DDIs) is a long-standing challenge in clinical treatments and drug developments. Recently, deep learning techniques have been developed for DDI prediction. However, they generally require a huge number of samples, while known DDIs are rare.

  Methods: In this work, we present KnowDDI, a graph neural network-based method that addresses the above challenge. KnowDDI enhances drug representations by adaptively leveraging rich neighborhood information from large biomedical knowledge graphs. Then, it learns a knowledge subgraph for each drug-pair to interpret the predicted DDI, where each of the edges is associated with a connection strength indicating the importance of a known DDI or resembling strength between a drug-pair whose connection is unknown. Thus, the lack of DDIs is implicitly compensated by the enriched drug representations and propagated drug similarities.

  Results: We evaluate KnowDDI on two benchmark DDI datasets. Results show that KnowDDI obtains the state-of-the-art prediction performance with better interpretability. We also find that KnowDDI suffers less than existing works given a sparser knowledge graph. This indicates that the propagated drug similarities play a more important role in compensating for the lack of DDIs when the drug representations are less enriched.

  Conclusions: KnowDDI nicely combines the efficiency of deep learning techniques and the rich prior knowledge in biomedical knowledge graphs. As an original open-source tool, KnowDDI can help detect possible interactions in a broad range of relevant interaction prediction tasks, such as protein-protein interactions, drug-target interactions and disease-gene interactions, eventually promoting the development of biomedicine and healthcare.

explainability

watermark

diffusion

Title: SinSR: Diffusion-Based Image Super-Resolution in a Single Step. (arXiv:2311.14760v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.14760
• Code URL: https://github.com/wyf0912/sinsr
• Copy Paste: [[2311.14760]] SinSR: Diffusion-Based Image Super-Resolution in a Single Step(http://arxiv.org/abs/2311.14760)
• Summary:

  While super-resolution (SR) methods based on diffusion models exhibit promising results, their practical application is hindered by the substantial number of required inference steps. Recent methods utilize degraded images in the initial state, thereby shortening the Markov chain. Nevertheless, these solutions either rely on a precise formulation of the degradation process or still necessitate a relatively lengthy generation path (e.g., 15 iterations). To enhance inference speed, we propose a simple yet effective method for achieving single-step SR generation, named SinSR. Specifically, we first derive a deterministic sampling process from the most recent state-of-the-art (SOTA) method for accelerating diffusion-based SR. This allows the mapping between the input random noise and the generated high-resolution image to be obtained in a reduced and acceptable number of inference steps during training. We show that this deterministic mapping can be distilled into a student model that performs SR within only one inference step. Additionally, we propose a novel consistency-preserving loss to simultaneously leverage the ground-truth image during the distillation process, ensuring that the performance of the student model is not solely bound by the feature manifold of the teacher model, resulting in further performance improvement. Extensive experiments conducted on synthetic and real-world datasets demonstrate that the proposed method can achieve comparable or even superior performance compared to both previous SOTA methods and the teacher model, in just one sampling step, resulting in a remarkable up to x10 speedup for inference. Our code will be released at https://github.com/wyf0912/SinSR

Title: AdaDiff: Adaptive Step Selection for Fast Diffusion. (arXiv:2311.14768v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.14768
• Code URL: null
• Copy Paste: [[2311.14768]] AdaDiff: Adaptive Step Selection for Fast Diffusion(http://arxiv.org/abs/2311.14768)
• Summary:

  Diffusion models, as a type of generative models, have achieved impressive results in generating images and videos conditioned on textual conditions. However, the generation process of diffusion models involves denoising for dozens of steps to produce photorealistic images/videos, which is computationally expensive. Unlike previous methods that design ``one-size-fits-all'' approaches for speed up, we argue denoising steps should be sample-specific conditioned on the richness of input texts. To this end, we introduce AdaDiff, a lightweight framework designed to learn instance-specific step usage policies, which are then used by the diffusion model for generation. AdaDiff is optimized using a policy gradient method to maximize a carefully designed reward function, balancing inference time and generation quality. We conduct experiments on three image generation and two video generation benchmarks and demonstrate that our approach achieves similar results in terms of visual quality compared to the baseline using a fixed 50 denoising steps while reducing inference time by at least 33%, going as high as 40%. Furthermore, our qualitative analysis shows that our method allocates more steps to more informative text conditions and fewer steps to simpler text conditions.

Title: Resfusion: Prior Residual Noise embedded Denoising Diffusion Probabilistic Models. (arXiv:2311.14900v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.14900
• Code URL: null
• Copy Paste: [[2311.14900]] Resfusion: Prior Residual Noise embedded Denoising Diffusion Probabilistic Models(http://arxiv.org/abs/2311.14900)
• Summary:

  Recently, Denoising Diffusion Probabilistic Models have been widely used in image segmentation, by generating segmentation masks conditioned on the input image. However, previous works can not seamlessly integrate existing end-to-end models with denoising diffusion models. Existing research can only select acceleration steps based on experience rather than calculating them specifically. Moreover, most methods are limited to small models and small-scale datasets, unable to generalize to general datasets and a wider range of tasks. Therefore, we propose Resfusion with a novel resnoise-diffusion process, which gradually generates segmentation masks or any type of target image, seamlessly integrating state-of-the-art end-to-end models and denoising diffusion models. Resfusion bridges the discrepancy between the likelihood output and the ground truth output through a Markov process. Through the novel smooth equivalence transformation in resnoise-diffusion process, we determine the optimal acceleration step. Experimental results demonstrate that Resfusion combines the capabilities of existing end-to-end models and denoising diffusion models, further enhancing performance and achieving outstanding results. Moreover, Resfusion is not limited to segmentation tasks, it can easily generalize to any general tasks of image generation and exhibit strong competitiveness.

Title: DECap: Towards Generalized Explicit Caption Editing via Diffusion Mechanism. (arXiv:2311.14920v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.14920
• Code URL: null
• Copy Paste: [[2311.14920]] DECap: Towards Generalized Explicit Caption Editing via Diffusion Mechanism(http://arxiv.org/abs/2311.14920)
• Summary:

  Explicit Caption Editing (ECE) -- refining reference image captions through a sequence of explicit edit operations (e.g., KEEP, DETELE) -- has raised significant attention due to its explainable and human-like nature. After training with carefully designed reference and ground-truth caption pairs, state-of-the-art ECE models exhibit limited generalization ability beyond the original training data distribution, i.e., they are tailored to refine content details only in in-domain samples but fail to correct errors in out-of-domain samples. To this end, we propose a new Diffusion-based Explicit Caption editing method: DECap. Specifically, we reformulate the ECE task as a denoising process under the diffusion mechanism, and introduce innovative edit-based noising and denoising processes. Thanks to this design, the noising process can help to eliminate the need for meticulous paired data selection by directly introducing word-level noises for training, learning diverse distribution over input reference caption. The denoising process involves the explicit predictions of edit operations and corresponding content words, refining reference captions through iterative step-wise editing. To further efficiently implement our diffusion process and improve the inference speed, DECap discards the prevalent multi-stage design and directly generates edit operations and content words simultaneously. Extensive ablations have demonstrated the strong generalization ability of DECap in various scenarios. More interestingly, it even shows great potential in improving the quality and controllability of caption generation.

Title: GBD-TS: Goal-based Pedestrian Trajectory Prediction with Diffusion using Tree Sampling Algorithm. (arXiv:2311.14922v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.14922
• Code URL: null
• Copy Paste: [[2311.14922]] GBD-TS: Goal-based Pedestrian Trajectory Prediction with Diffusion using Tree Sampling Algorithm(http://arxiv.org/abs/2311.14922)
• Summary:

  Predicting pedestrian trajectories is crucial for improving the safety and effectiveness of autonomous driving and mobile robots. However, this task is nontrivial due to the inherent stochasticity of human motion, which naturally requires the predictor to generate multi-model prediction. Previous works have used various generative methods, such as GAN and VAE, for pedestrian trajectory prediction. Nevertheless, these methods may suffer from problems, including mode collapse and relatively low-quality results. The denoising diffusion probabilistic model (DDPM) has recently been applied to trajectory prediction due to its simple training process and powerful reconstruction ability. However, current diffusion-based methods are straightforward without fully leveraging input information and usually require many denoising iterations leading to a long inference time or an additional network for initialization. To address these challenges and promote the application of diffusion models in trajectory prediction, we propose a novel scene-aware multi-modal pedestrian trajectory prediction framework called GBD. GBD combines goal prediction with the diffusion network. First, the goal predictor produces multiple goals, and then the diffusion network generates multi-modal trajectories conditioned on these goals. Furthermore, we introduce a new diffusion sampling algorithm named tree sampling (TS), which leverages common feature to reduce the inference time and improve accuracy for multi-modal prediction. Experimental results demonstrate that our GBD-TS method achieves state-of-the-art performance with real-time inference speed.

Title: FreePIH: Training-Free Painterly Image Harmonization with Diffusion Model. (arXiv:2311.14926v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.14926
• Code URL: null
• Copy Paste: [[2311.14926]] FreePIH: Training-Free Painterly Image Harmonization with Diffusion Model(http://arxiv.org/abs/2311.14926)
• Summary:

  This paper provides an efficient training-free painterly image harmonization (PIH) method, dubbed FreePIH, that leverages only a pre-trained diffusion model to achieve state-of-the-art harmonization results. Unlike existing methods that require either training auxiliary networks or fine-tuning a large pre-trained backbone, or both, to harmonize a foreground object with a painterly-style background image, our FreePIH tames the denoising process as a plug-in module for foreground image style transfer. Specifically, we find that the very last few steps of the denoising (i.e., generation) process strongly correspond to the stylistic information of images, and based on this, we propose to augment the latent features of both the foreground and background images with Gaussians for a direct denoising-based harmonization. To guarantee the fidelity of the harmonized image, we make use of multi-scale features to enforce the consistency of the content and stability of the foreground objects in the latent space, and meanwhile, aligning both fore-/back-grounds with the same style. Moreover, to accommodate the generation with more structural and textural details, we further integrate text prompts to attend to the latent features, hence improving the generation quality. Quantitative and qualitative evaluations on COCO and LAION 5B datasets demonstrate that our method can surpass representative baselines by large margins.

Title: Point Cloud Pre-training with Diffusion Models. (arXiv:2311.14960v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.14960
• Code URL: null
• Copy Paste: [[2311.14960]] Point Cloud Pre-training with Diffusion Models(http://arxiv.org/abs/2311.14960)
• Summary:

  Pre-training a model and then fine-tuning it on downstream tasks has demonstrated significant success in the 2D image and NLP domains. However, due to the unordered and non-uniform density characteristics of point clouds, it is non-trivial to explore the prior knowledge of point clouds and pre-train a point cloud backbone. In this paper, we propose a novel pre-training method called Point cloud Diffusion pre-training (PointDif). We consider the point cloud pre-training task as a conditional point-to-point generation problem and introduce a conditional point generator. This generator aggregates the features extracted by the backbone and employs them as the condition to guide the point-to-point recovery from the noisy point cloud, thereby assisting the backbone in capturing both local and global geometric priors as well as the global point density distribution of the object. We also present a recurrent uniform sampling optimization strategy, which enables the model to uniformly recover from various noise levels and learn from balanced supervision. Our PointDif achieves substantial improvement across various real-world datasets for diverse downstream tasks such as classification, segmentation and detection. Specifically, PointDif attains 70.0% mIoU on S3DIS Area 5 for the segmentation task and achieves an average improvement of 2.4% on ScanObjectNN for the classification task compared to TAP. Furthermore, our pre-training framework can be flexibly applied to diverse point cloud backbones and bring considerable gains.

Title: InstaStyle: Inversion Noise of a Stylized Image is Secretly a Style Adviser. (arXiv:2311.15040v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.15040
• Code URL: null
• Copy Paste: [[2311.15040]] InstaStyle: Inversion Noise of a Stylized Image is Secretly a Style Adviser(http://arxiv.org/abs/2311.15040)
• Summary:

  Stylized text-to-image generation focuses on creating images from textual descriptions while adhering to a style specified by a few reference images. However, subtle style variations within different reference images can hinder the model from accurately learning the target style. In this paper, we propose InstaStyle, a novel approach that excels in generating high-fidelity stylized images with only a single reference image. Our approach is based on the finding that the inversion noise from a stylized reference image inherently carries the style signal, as evidenced by their non-zero signal-to-noise ratio. We employ DDIM inversion to extract this noise from the reference image and leverage a diffusion model to generate new stylized images from the ``style" noise. Additionally, the inherent ambiguity and bias of textual prompts impede the precise conveying of style. To address this, we introduce a learnable style token via prompt refinement, which enhances the accuracy of the style description for the reference image. Qualitative and quantitative experimental results demonstrate that InstaStyle achieves superior performance compared to current benchmarks. Furthermore, our approach also showcases its capability in the creative task of style combination with mixed inversion noise.

noise learning

data-free

transformer

Title: All in One: RGB, RGB-D, and RGB-T Salient Object Detection. (arXiv:2311.14746v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.14746
• Code URL: null
• Copy Paste: [[2311.14746]] All in One: RGB, RGB-D, and RGB-T Salient Object Detection(http://arxiv.org/abs/2311.14746)
• Summary:

  Salient object detection (SOD) aims to identify the most attractive objects within an image. Depending on the type of data being detected, SOD can be categorized into various forms, including RGB, RGB-D (Depth), RGB-T (Thermal) and light field SOD. Previous researches have focused on saliency detection with individual data type. If the RGB-D SOD model is forced to detect RGB-T data it will perform poorly. We propose an innovative model framework that provides a unified solution for the salient object detection task of three types of data (RGB, RGB-D, and RGB-T). The three types of data can be handled in one model (all in one) with the same weight parameters. In this framework, the three types of data are concatenated in an ordered manner within a single input batch, and features are extracted using a transformer network. Based on this framework, we propose an efficient lightweight SOD model, namely AiOSOD, which can detect any RGB, RGB-D, and RGB-T data with high speed (780FPS for RGB data, 485FPS for RGB-D or RGB-T data). Notably, with only 6.25M parameters, AiOSOD achieves excellent performance on RGB, RGB-D, and RGB-T datasets.

Title: Towards Scalable 3D Anomaly Detection and Localization: A Benchmark via 3D Anomaly Synthesis and A Self-Supervised Learning Network. (arXiv:2311.14897v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.14897
• Code URL: null
• Copy Paste: [[2311.14897]] Towards Scalable 3D Anomaly Detection and Localization: A Benchmark via 3D Anomaly Synthesis and A Self-Supervised Learning Network(http://arxiv.org/abs/2311.14897)
• Summary:

  Recently, 3D anomaly detection, a crucial problem involving fine-grained geometry discrimination, is getting more attention. However, the lack of abundant real 3D anomaly data limits the scalability of current models. To enable scalable anomaly data collection, we propose a 3D anomaly synthesis pipeline to adapt existing large-scale 3Dmodels for 3D anomaly detection. Specifically, we construct a synthetic dataset, i.e., Anomaly-ShapeNet, basedon ShapeNet. Anomaly-ShapeNet consists of 1600 point cloud samples under 40 categories, which provides a rich and varied collection of data, enabling efficient training and enhancing adaptability to industrial scenarios. Meanwhile,to enable scalable representation learning for 3D anomaly localization, we propose a self-supervised method, i.e., Iterative Mask Reconstruction Network (IMRNet). During training, we propose a geometry-aware sample module to preserve potentially anomalous local regions during point cloud down-sampling. Then, we randomly mask out point patches and sent the visible patches to a transformer for reconstruction-based self-supervision. During testing, the point cloud repeatedly goes through the Mask Reconstruction Network, with each iteration's output becoming the next input. By merging and contrasting the final reconstructed point cloud with the initial input, our method successfully locates anomalies. Experiments show that IMRNet outperforms previous state-of-the-art methods, achieving 66.1% in I-AUC on Anomaly-ShapeNet dataset and 72.5% in I-AUC on Real3D-AD dataset. Our dataset will be released at https://github.com/Chopper-233/Anomaly-ShapeNet

Title: Emotion-Oriented Behavior Model Using Deep Learning. (arXiv:2311.14674v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.14674
• Code URL: null
• Copy Paste: [[2311.14674]] Emotion-Oriented Behavior Model Using Deep Learning(http://arxiv.org/abs/2311.14674)
• Summary:

  Emotions, as a fundamental ingredient of any social interaction, lead to behaviors that represent the effectiveness of the interaction through facial expressions and gestures in humans. Hence an agent must possess the social and cognitive abilities to understand human social parameters and behave accordingly. However, no such emotion-oriented behavior model is presented yet in the existing research. The emotion prediction may generate appropriate agents' behaviors for effective interaction using conversation modality. Considering the importance of emotions, and behaviors, for an agent's social interaction, an Emotion-based Behavior model is presented in this paper for Socio-cognitive artificial agents. The proposed model is implemented using tweets data trained on multiple models like Long Short-Term Memory (LSTM), Convolution Neural Network (CNN) and Bidirectional Encoder Representations from Transformers (BERT) for emotion prediction with an average accuracy of 92%, and 55% respectively. Further, using emotion predictions from CNN-LSTM, the behavior module responds using facial expressions and gestures using Behavioral Markup Language (BML). The accuracy of emotion-based behavior predictions is statistically validated using the 2-tailed Pearson correlation on the data collected from human users through questionnaires. Analysis shows that all emotion-based behaviors accurately depict human-like gestures and facial expressions based on the significant correlation at the 0.01 and 0.05 levels. This study is a steppingstone to a multi-faceted artificial agent interaction based on emotion-oriented behaviors. Cognition has significance regarding social interaction among humans.

Title: Positional Description Matters for Transformers Arithmetic. (arXiv:2311.14737v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.14737
• Code URL: null
• Copy Paste: [[2311.14737]] Positional Description Matters for Transformers Arithmetic(http://arxiv.org/abs/2311.14737)
• Summary:

  Transformers, central to the successes in modern Natural Language Processing, often falter on arithmetic tasks despite their vast capabilities --which paradoxically include remarkable coding abilities. We observe that a crucial challenge is their naive reliance on positional information to solve arithmetic problems with a small number of digits, leading to poor performance on larger numbers. Herein, we delve deeper into the role of positional encoding, and propose several ways to fix the issue, either by modifying the positional encoding directly, or by modifying the representation of the arithmetic task to leverage standard positional encoding differently. We investigate the value of these modifications for three tasks: (i) classical multiplication, (ii) length extrapolation in addition, and (iii) addition in natural language context. For (i) we train a small model on a small dataset (100M parameters and 300k samples) with remarkable aptitude in (direct, no scratchpad) 15 digits multiplication and essentially perfect up to 12 digits, while usual training in this context would give a model failing at 4 digits multiplication. In the experiments on addition, we use a mere 120k samples to demonstrate: for (ii) extrapolation from 10 digits to testing on 12 digits numbers while usual training would have no extrapolation, and for (iii) almost perfect accuracy up to 5 digits while usual training would be correct only up to 3 digits (which is essentially memorization with a training set of 120k samples).

Title: Offensive Language Identification in Transliterated and Code-Mixed Bangla. (arXiv:2311.15023v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.15023
• Code URL: null
• Copy Paste: [[2311.15023]] Offensive Language Identification in Transliterated and Code-Mixed Bangla(http://arxiv.org/abs/2311.15023)
• Summary:

  Identifying offensive content in social media is vital for creating safe online communities. Several recent studies have addressed this problem by creating datasets for various languages. In this paper, we explore offensive language identification in texts with transliterations and code-mixing, linguistic phenomena common in multilingual societies, and a known challenge for NLP systems. We introduce TB-OLID, a transliterated Bangla offensive language dataset containing 5,000 manually annotated comments. We train and fine-tune machine learning models on TB-OLID, and we evaluate their results on this dataset. Our results show that English pre-trained transformer-based models, such as fBERT and HateBERT achieve the best performance on this dataset.

Title: One Fits All: Universal Time Series Analysis by Pretrained LM and Specially Designed Adaptors. (arXiv:2311.14782v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.14782
• Code URL: null
• Copy Paste: [[2311.14782]] One Fits All: Universal Time Series Analysis by Pretrained LM and Specially Designed Adaptors(http://arxiv.org/abs/2311.14782)
• Summary:

  Despite the impressive achievements of pre-trained models in the fields of natural language processing (NLP) and computer vision (CV), progress in the domain of time series analysis has been limited. In contrast to NLP and CV, where a single model can handle various tasks, time series analysis still relies heavily on task-specific methods for activities such as classification, anomaly detection, forecasting, and few-shot learning. The primary obstacle to developing a pre-trained model for time series analysis is the scarcity of sufficient training data. In our research, we overcome this obstacle by utilizing pre-trained models from language or CV, which have been trained on billions of data points, and apply them to time series analysis. We assess the effectiveness of the pre-trained transformer model in two ways. Initially, we maintain the original structure of the self-attention and feedforward layers in the residual blocks of the pre-trained language or image model, using the Frozen Pre-trained Transformer (FPT) for time series analysis with the addition of projection matrices for input and output. Additionally, we introduce four unique adapters, designed specifically for downstream tasks based on the pre-trained model, including forecasting and anomaly detection. These adapters are further enhanced with efficient parameter tuning, resulting in superior performance compared to all state-of-the-art methods.Our comprehensive experimental studies reveal that (a) the simple FPT achieves top-tier performance across various time series analysis tasks; and (b) fine-tuning the FPT with the custom-designed adapters can further elevate its performance, outshining specialized task-specific models.

generative

Title: MemoryCompanion: A Smart Healthcare Solution to Empower Efficient Alzheimer's Care Via Unleashing Generative AI. (arXiv:2311.14730v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.14730
• Code URL: null
• Copy Paste: [[2311.14730]] MemoryCompanion: A Smart Healthcare Solution to Empower Efficient Alzheimer's Care Via Unleashing Generative AI(http://arxiv.org/abs/2311.14730)
• Summary:

  With the rise of Large Language Models (LLMs), notably characterized by GPT frameworks, there emerges a catalyst for novel healthcare applications. Earlier iterations of chatbot caregivers, though existent, have yet to achieve a dimension of human-like authenticity. This paper unveils `MemoryCompanion' a pioneering digital health solution explicitly tailored for Alzheimer's disease (AD) patients and their caregivers. Drawing upon the nuances of GPT technology and prompt engineering, MemoryCompanion manifests a personalized caregiving paradigm, fostering interactions via voice-cloning and talking-face mechanisms that resonate with the familiarity of known companions. Using advanced prompt-engineering, the system intricately adapts to each patient's distinct profile, curating its content and communication style accordingly. This approach strives to counteract prevalent issues of social isolation and loneliness frequently observed in AD demographics. Our methodology, grounded in its innovative design, addresses both the caregiving and technological challenges intrinsic to this domain.

Title: Vector-Quantized Prompt Learning for Paraphrase Generation. (arXiv:2311.14949v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.14949
• Code URL: null
• Copy Paste: [[2311.14949]] Vector-Quantized Prompt Learning for Paraphrase Generation(http://arxiv.org/abs/2311.14949)
• Summary:

  Deep generative modeling of natural languages has achieved many successes, such as producing fluent sentences and translating from one language into another. However, the development of generative modeling techniques for paraphrase generation still lags behind largely due to the challenges in addressing the complex conflicts between expression diversity and semantic preservation. This paper proposes to generate diverse and high-quality paraphrases by exploiting the pre-trained models with instance-dependent prompts. To learn generalizable prompts, we assume that the number of abstract transforming patterns of paraphrase generation (governed by prompts) is finite and usually not large. Therefore, we present vector-quantized prompts as the cues to control the generation of pre-trained models. Extensive experiments demonstrate that the proposed method achieves new state-of-art results on three benchmark datasets, including Quora, Wikianswers, and MSCOCO. We will release all the code upon acceptance.

Title: A unified framework for learning with nonlinear model classes from arbitrary linear samples. (arXiv:2311.14886v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.14886
• Code URL: null
• Copy Paste: [[2311.14886]] A unified framework for learning with nonlinear model classes from arbitrary linear samples(http://arxiv.org/abs/2311.14886)
• Summary:

  This work considers the fundamental problem of learning an unknown object from training data using a given model class. We introduce a unified framework that allows for objects in arbitrary Hilbert spaces, general types of (random) linear measurements as training data and general types of nonlinear model classes. We establish a series of learning guarantees for this framework. These guarantees provide explicit relations between the amount of training data and properties of the model class to ensure near-best generalization bounds. In doing so, we also introduce and develop the key notion of the variation of a model class with respect to a distribution of sampling operators. To exhibit the versatility of this framework, we show that it can accommodate many different types of well-known problems of interest. We present examples such as matrix sketching by random sampling, compressed sensing with isotropic vectors, active learning in regression and compressed sensing with generative models. In all cases, we show how known results become straightforward corollaries of our general learning guarantees. For compressed sensing with generative models, we also present a number of generalizations and improvements of recent results. In summary, our work not only introduces a unified way to study learning unknown objects from general types of data, but also establishes a series of general theoretical guarantees which consolidate and improve various known results.

Title: Training a Hopfield Variational Autoencoder with Equilibrium Propagation. (arXiv:2311.15047v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.15047
• Code URL: null
• Copy Paste: [[2311.15047]] Training a Hopfield Variational Autoencoder with Equilibrium Propagation(http://arxiv.org/abs/2311.15047)
• Summary:

  On dedicated analog hardware, equilibrium propagation is an energy-efficient alternative to backpropagation. In spite of its theoretical guarantees, its application in the AI domain remains limited to the discriminative setting. Meanwhile, despite its high computational demands, generative AI is on the rise. In this paper, we demonstrate the application of Equilibrium Propagation in training a variational autoencoder (VAE) for generative modeling. Leveraging the symmetric nature of Hopfield networks, we propose using a single model to serve as both the encoder and decoder which could effectively halve the required chip size for VAE implementations, paving the way for more efficient analog hardware configurations.

large language model

Title: Compositional Zero-shot Learning via Progressive Language-based Observations. (arXiv:2311.14749v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.14749
• Code URL: null
• Copy Paste: [[2311.14749]] Compositional Zero-shot Learning via Progressive Language-based Observations(http://arxiv.org/abs/2311.14749)
• Summary:

  Compositional zero-shot learning aims to recognize unseen state-object compositions by leveraging known primitives (state and object) during training. However, effectively modeling interactions between primitives and generalizing knowledge to novel compositions remains a perennial challenge. There are two key factors: object-conditioned and state-conditioned variance, i.e., the appearance of states (or objects) can vary significantly when combined with different objects (or states). For instance, the state "old" can signify a vintage design for a "car" or an advanced age for a "cat". In this paper, we argue that these variances can be mitigated by predicting composition categories based on pre-observed primitive. To this end, we propose Progressive Language-based Observations (PLO), which can dynamically determine a better observation order of primitives. These observations comprise a series of concepts or languages that allow the model to understand image content in a step-by-step manner. Specifically, PLO adopts pre-trained vision-language models (VLMs) to empower the model with observation capabilities. We further devise two variants: 1) PLO-VLM: a two-step method, where a pre-observing classifier dynamically determines the observation order of two primitives. 2) PLO-LLM: a multi-step scheme, which utilizes large language models (LLMs) to craft composition-specific prompts for step-by-step observing. Extensive ablations on three challenging datasets demonstrate the superiority of PLO compared with state-of-the-art methods, affirming its abilities in compositional recognition.

Title: Benefits and Harms of Large Language Models in Digital Mental Health. (arXiv:2311.14693v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.14693
• Code URL: null
• Copy Paste: [[2311.14693]] Benefits and Harms of Large Language Models in Digital Mental Health(http://arxiv.org/abs/2311.14693)
• Summary:

  The past decade has been transformative for mental health research and practice. The ability to harness large repositories of data, whether from electronic health records (EHR), mobile devices, or social media, has revealed a potential for valuable insights into patient experiences, promising early, proactive interventions, as well as personalized treatment plans. Recent developments in generative artificial intelligence, particularly large language models (LLMs), show promise in leading digital mental health to uncharted territory. Patients are arriving at doctors' appointments with information sourced from chatbots, state-of-the-art LLMs are being incorporated in medical software and EHR systems, and chatbots from an ever-increasing number of startups promise to serve as AI companions, friends, and partners. This article presents contemporary perspectives on the opportunities and risks posed by LLMs in the design, development, and implementation of digital mental health tools. We adopt an ecological framework and draw on the affordances offered by LLMs to discuss four application areas -- care-seeking behaviors from individuals in need of care, community care provision, institutional and medical care provision, and larger care ecologies at the societal level. We engage in a thoughtful consideration of whether and how LLM-based technologies could or should be employed for enhancing mental health. The benefits and harms our article surfaces could serve to help shape future research, advocacy, and regulatory efforts focused on creating more responsible, user-friendly, equitable, and secure LLM-based tools for mental health treatment and intervention.

Title: Zero-Shot Question Answering over Financial Documents using Large Language Models. (arXiv:2311.14722v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.14722
• Code URL: null
• Copy Paste: [[2311.14722]] Zero-Shot Question Answering over Financial Documents using Large Language Models(http://arxiv.org/abs/2311.14722)
• Summary:

  We introduce a large language model (LLM) based approach to answer complex questions requiring multi-hop numerical reasoning over financial reports. While LLMs have exhibited remarkable performance on various natural language and reasoning tasks, complex reasoning problems often rely on few-shot prompts that require carefully crafted examples. In contrast, our approach uses novel zero-shot prompts that guide the LLM to encode the required reasoning into a Python program or a domain specific language. The generated program is then executed by a program interpreter, thus mitigating the limitations of LLM in performing accurate arithmetic calculations.

  We evaluate the proposed approach on three financial datasets using some of the recently developed generative pretrained transformer (GPT) models and perform comparisons with various zero-shot baselines. The experimental results demonstrate that our approach significantly improves the accuracy for all the LLMs over their respective baselines. We provide a detailed analysis of the results, generating insights to support our findings. The success of our approach demonstrates the enormous potential to extract complex domain specific numerical reasoning by designing zero-shot prompts to effectively exploit the knowledge embedded in LLMs.

Title: AutoKG: Efficient Automated Knowledge Graph Generation for Language Models. (arXiv:2311.14740v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.14740
• Code URL: null
• Copy Paste: [[2311.14740]] AutoKG: Efficient Automated Knowledge Graph Generation for Language Models(http://arxiv.org/abs/2311.14740)
• Summary:

  Traditional methods of linking large language models (LLMs) to knowledge bases via the semantic similarity search often fall short of capturing complex relational dynamics. To address these limitations, we introduce AutoKG, a lightweight and efficient approach for automated knowledge graph (KG) construction. For a given knowledge base consisting of text blocks, AutoKG first extracts keywords using a LLM and then evaluates the relationship weight between each pair of keywords using graph Laplace learning. We employ a hybrid search scheme combining vector similarity and graph-based associations to enrich LLM responses. Preliminary experiments demonstrate that AutoKG offers a more comprehensive and interconnected knowledge retrieval mechanism compared to the semantic similarity search, thereby enhancing the capabilities of LLMs in generating more insightful and relevant outputs.

Title: Evaluating Large Language Models through Gender and Racial Stereotypes. (arXiv:2311.14788v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.14788
• Code URL: null
• Copy Paste: [[2311.14788]] Evaluating Large Language Models through Gender and Racial Stereotypes(http://arxiv.org/abs/2311.14788)
• Summary:

  Language Models have ushered a new age of AI gaining traction within the NLP community as well as amongst the general population. AI's ability to make predictions, generations and its applications in sensitive decision-making scenarios, makes it even more important to study these models for possible biases that may exist and that can be exaggerated. We conduct a quality comparative study and establish a framework to evaluate language models under the premise of two kinds of biases: gender and race, in a professional setting. We find out that while gender bias has reduced immensely in newer models, as compared to older ones, racial bias still exists.

Title: OpusCleaner and OpusTrainer, open source toolkits for training Machine Translation and Large language models. (arXiv:2311.14838v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.14838
• Code URL: https://github.com/hplt-project/opustrainer
• Copy Paste: [[2311.14838]] OpusCleaner and OpusTrainer, open source toolkits for training Machine Translation and Large language models(http://arxiv.org/abs/2311.14838)
• Summary:

  Developing high quality machine translation systems is a labour intensive, challenging and confusing process for newcomers to the field. We present a pair of tools OpusCleaner and OpusTrainer that aim to simplify the process, reduce the amount of work and lower the entry barrier for newcomers.

  OpusCleaner is a data downloading, cleaning, and proprocessing toolkit. It is designed to allow researchers to quickly download, visualise and preprocess bilingual (or monolingual) data that comes from many different sources, each of them with different quality, issues, and unique filtering/preprocessing requirements.

  OpusTrainer is a data scheduling and data augmenting tool aimed at building large scale, robust machine translation systems and large language models. It features deterministic data mixing from many different sources, on-the-fly data augmentation and more.

  Using these tools, we showcase how we can use it to create high quality machine translation model robust to noisy user input; multilingual models and terminology aware models.

Title: Walking a Tightrope -- Evaluating Large Language Models in High-Risk Domains. (arXiv:2311.14966v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.14966
• Code URL: null
• Copy Paste: [[2311.14966]] Walking a Tightrope -- Evaluating Large Language Models in High-Risk Domains(http://arxiv.org/abs/2311.14966)
• Summary:

  High-risk domains pose unique challenges that require language models to provide accurate and safe responses. Despite the great success of large language models (LLMs), such as ChatGPT and its variants, their performance in high-risk domains remains unclear. Our study delves into an in-depth analysis of the performance of instruction-tuned LLMs, focusing on factual accuracy and safety adherence. To comprehensively assess the capabilities of LLMs, we conduct experiments on six NLP datasets including question answering and summarization tasks within two high-risk domains: legal and medical. Further qualitative analysis highlights the existing limitations inherent in current LLMs when evaluating in high-risk domains. This underscores the essential nature of not only improving LLM capabilities but also prioritizing the refinement of domain-specific metrics, and embracing a more human-centric approach to enhance safety and factual reliability. Our findings advance the field toward the concerns of properly evaluating LLMs in high-risk domains, aiming to steer the adaptability of LLMs in fulfilling societal obligations and aligning with forthcoming regulations, such as the EU AI Act.

segmentation

Title: The 2nd Workshop on Maritime Computer Vision (MaCVi) 2024. (arXiv:2311.14762v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.14762
• Code URL: null
• Copy Paste: [[2311.14762]] The 2nd Workshop on Maritime Computer Vision (MaCVi) 2024(http://arxiv.org/abs/2311.14762)
• Summary:

  The 2nd Workshop on Maritime Computer Vision (MaCVi) 2024 addresses maritime computer vision for Unmanned Aerial Vehicles (UAV) and Unmanned Surface Vehicles (USV). Three challenges categories are considered: (i) UAV-based Maritime Object Tracking with Re-identification, (ii) USV-based Maritime Obstacle Segmentation and Detection, (iii) USV-based Maritime Boat Tracking. The USV-based Maritime Obstacle Segmentation and Detection features three sub-challenges, including a new embedded challenge addressing efficicent inference on real-world embedded devices. This report offers a comprehensive overview of the findings from the challenges. We provide both statistical and qualitative analyses, evaluating trends from over 195 submissions. All datasets, evaluation code, and the leaderboard are available to the public at https://macvi.org/workshop/macvi24.

Title: Set Features for Anomaly Detection. (arXiv:2311.14773v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.14773
• Code URL: null
• Copy Paste: [[2311.14773]] Set Features for Anomaly Detection(http://arxiv.org/abs/2311.14773)
• Summary:

  This paper proposes set features for detecting anomalies in samples that consist of unusual combinations of normal elements. Many leading methods discover anomalies by detecting an unusual part of a sample. For example, state-of-the-art segmentation-based approaches, first classify each element of the sample (e.g., image patch) as normal or anomalous and then classify the entire sample as anomalous if it contains anomalous elements. However, such approaches do not extend well to scenarios where the anomalies are expressed by an unusual combination of normal elements. In this paper, we overcome this limitation by proposing set features that model each sample by the distribution of its elements. We compute the anomaly score of each sample using a simple density estimation method, using fixed features. Our approach outperforms the previous state-of-the-art in image-level logical anomaly detection and sequence-level time series anomaly detection.

Title: Text and Click inputs for unambiguous open vocabulary instance segmentation. (arXiv:2311.14822v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.14822
• Code URL: https://github.com/nikolaiwarner7/text-and-click-for-open-vocabulary-segmentation
• Copy Paste: [[2311.14822]] Text and Click inputs for unambiguous open vocabulary instance segmentation(http://arxiv.org/abs/2311.14822)
• Summary:

  Segmentation localizes objects in an image on a fine-grained per-pixel scale. Segmentation benefits by humans-in-the-loop to provide additional input of objects to segment using a combination of foreground or background clicks. Tasks include photoediting or novel dataset annotation, where human annotators leverage an existing segmentation model instead of drawing raw pixel level annotations. We propose a new segmentation process, Text + Click segmentation, where a model takes as input an image, a text phrase describing a class to segment, and a single foreground click specifying the instance to segment. Compared to previous approaches, we leverage open-vocabulary image-text models to support a wide-range of text prompts. Conditioning segmentations on text prompts improves the accuracy of segmentations on novel or unseen classes. We demonstrate that the combination of a single user-specified foreground click and a text prompt allows a model to better disambiguate overlapping or co-occurring semantic categories, such as "tie", "suit", and "person". We study these results across common segmentation datasets such as refCOCO, COCO, VOC, and OpenImages. Source code available here.

Title: Unified Medical Image Pre-training in Language-Guided Common Semantic Space. (arXiv:2311.14851v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.14851
• Code URL: null
• Copy Paste: [[2311.14851]] Unified Medical Image Pre-training in Language-Guided Common Semantic Space(http://arxiv.org/abs/2311.14851)
• Summary:

  Vision-Language Pre-training (VLP) has shown the merits of analysing medical images, by leveraging the semantic congruence between medical images and their corresponding reports. It efficiently learns visual representations, which in turn facilitates enhanced analysis and interpretation of intricate imaging data. However, such observation is predominantly justified on single-modality data (mostly 2D images like X-rays), adapting VLP to learning unified representations for medical images in real scenario remains an open challenge. This arises from medical images often encompass a variety of modalities, especially modalities with different various number of dimensions (e.g., 3D images like Computed Tomography). To overcome the aforementioned challenges, we propose an Unified Medical Image Pre-training framework, namely UniMedI, which utilizes diagnostic reports as common semantic space to create unified representations for diverse modalities of medical images (especially for 2D and 3D images). Under the text's guidance, we effectively uncover visual modality information, identifying the affected areas in 2D X-rays and slices containing lesion in sophisticated 3D CT scans, ultimately enhancing the consistency across various medical imaging modalities. To demonstrate the effectiveness and versatility of UniMedI, we evaluate its performance on both 2D and 3D images across 10 different datasets, covering a wide range of medical image tasks such as classification, segmentation, and retrieval. UniMedI has demonstrated superior performance in downstream tasks, showcasing its effectiveness in establishing a universal medical visual representation.

Title: Segmentation of diagnostic tissue compartments on whole slide images with renal thrombotic microangiopathies (TMAs). (arXiv:2311.14971v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.14971
• Code URL: null
• Copy Paste: [[2311.14971]] Segmentation of diagnostic tissue compartments on whole slide images with renal thrombotic microangiopathies (TMAs)(http://arxiv.org/abs/2311.14971)
• Summary:

  The thrombotic microangiopathies (TMAs) manifest in renal biopsy histology with a broad spectrum of acute and chronic findings. Precise diagnostic criteria for a renal biopsy diagnosis of TMA are missing. As a first step towards a machine learning- and computer vision-based analysis of wholes slide images from renal biopsies, we trained a segmentation model for the decisive diagnostic kidney tissue compartments artery, arteriole, glomerulus on a set of whole slide images from renal biopsies with TMAs and Mimickers (distinct diseases with a similar nephropathological appearance as TMA like severe benign nephrosclerosis, various vasculitides, Bevacizumab-plug glomerulopathy, arteriolar light chain deposition disease). Our segmentation model combines a U-Net-based tissue detection with a Shifted windows-transformer architecture to reach excellent segmentation results for even the most severely altered glomeruli, arterioles and arteries, even on unseen staining domains from a different nephropathology lab. With accurate automatic segmentation of the decisive renal biopsy compartments in human renal vasculopathies, we have laid the foundation for large-scale compartment-specific machine learning and computer vision analysis of renal biopsy repositories with TMAs.

Title: Multi-task Planar Reconstruction with Feature Warping Guidance. (arXiv:2311.14981v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.14981
• Code URL: null
• Copy Paste: [[2311.14981]] Multi-task Planar Reconstruction with Feature Warping Guidance(http://arxiv.org/abs/2311.14981)
• Summary:

  Piece-wise planar 3D reconstruction simultaneously segments plane instances and recovers their 3D plane parameters from an image, which is particularly useful for indoor or man-made environments. Efficient reconstruction of 3D planes coupled with semantic predictions offers advantages for a wide range of applications requiring scene understanding and concurrent spatial mapping. However, most existing planar reconstruction models either neglect semantic predictions or do not run efficiently enough for real-time applications. We introduce SoloPlanes, a real-time planar reconstruction model based on a modified instance segmentation architecture which simultaneously predicts semantics for each plane instance, along with plane parameters and piece-wise plane instance masks. By providing multi-view guidance in feature space, we achieve an improvement in instance mask segmentation despite only warping plane features due to the nature of feature sharing in multi-task learning. Our model simultaneously predicts semantics using single images at inference time, while achieving real-time predictions at 43 FPS. The code will be released post-publication.

Title: Adapter is All You Need for Tuning Visual Tasks. (arXiv:2311.15010v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.15010
• Code URL: https://github.com/leiyi-hu/mona
• Copy Paste: [[2311.15010]] Adapter is All You Need for Tuning Visual Tasks(http://arxiv.org/abs/2311.15010)
• Summary:

  Pre-training & fine-tuning can enhance the transferring efficiency and performance in visual tasks. Recent delta-tuning methods provide more options for visual classification tasks. Despite their success, existing visual delta-tuning art fails to exceed the upper limit of full fine-tuning on challenging tasks like instance segmentation and semantic segmentation. To find a competitive alternative to full fine-tuning, we propose the Multi-cognitive Visual Adapter (Mona) tuning, a novel adapter-based tuning method. First, we introduce multiple vision-friendly filters into the adapter to enhance its ability to process visual signals, while previous methods mainly rely on language-friendly linear filters. Second, we add the scaled normalization layer in the adapter to regulate the distribution of input features for visual filters. To fully demonstrate the practicality and generality of Mona, we conduct experiments on multiple representative visual tasks, including instance segmentation on COCO, semantic segmentation on ADE20K, object detection on Pascal VOC, and image classification on several common datasets. Exciting results illustrate that Mona surpasses full fine-tuning on all these tasks and is the only delta-tuning method outperforming full fine-tuning on instance segmentation and semantic segmentation tasks. For example, Mona achieves a 1% performance gain on the COCO dataset compared to full fine-tuning. Comprehensive results suggest that Mona-tuning is more suitable for retaining and utilizing the capabilities of pre-trained models than full fine-tuning. The code will be released at https://github.com/Leiyi-Hu/mona.