secure

Title: Mostree : Malicious Secure Private Decision Tree Evaluation with Sublinear Communication. (arXiv:2309.17124v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.17124
• Code URL: null
• Copy Paste: [[2309.17124]] Mostree : Malicious Secure Private Decision Tree Evaluation with Sublinear Communication(http://arxiv.org/abs/2309.17124)
• Summary:

  A private decision tree evaluation (PDTE) protocol allows a feature vector owner (FO) to classify its data using a tree model from a model owner (MO) and only reveals an inference result to the FO. This paper proposes Mostree, a PDTE protocol secure in the presence of malicious parties with sublinear communication. We design Mostree in the three-party honest-majority setting, where an (untrusted) computing party (CP) assists the FO and MO in the secure computation. We propose two low-communication oblivious selection (OS) protocols by exploiting nice properties of three-party replicated secret sharing (RSS) and distributed point function. Mostree combines OS protocols with a tree encoding method and three-party secure computation to achieve sublinear communication. We observe that most of the protocol components already maintain privacy even in the presence of a malicious adversary, and what remains to achieve is correctness. To ensure correctness, we propose a set of lightweight consistency checks and seamlessly integrate them into Mostree. As a result, Mostree achieves sublinear communication and malicious security simultaneously. We implement Mostree and compare it with the state-of-the-art. Experimental results demonstrate that Mostree is efficient and comparable to semi-honest PDTE schemes with sublinear communication. For instance, when evaluated on the MNIST dataset in a LAN setting, Mostree achieves an evaluation using approximately 768 ms with communication of around 168 KB.

security

Title: Stochastic Digital Twin for Copy Detection Patterns. (arXiv:2309.16866v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.16866
• Code URL: null
• Copy Paste: [[2309.16866]] Stochastic Digital Twin for Copy Detection Patterns(http://arxiv.org/abs/2309.16866)
• Summary:

  Copy detection patterns (CDP) present an efficient technique for product protection against counterfeiting. However, the complexity of studying CDP production variability often results in time-consuming and costly procedures, limiting CDP scalability. Recent advancements in computer modelling, notably the concept of a "digital twin" for printing-imaging channels, allow for enhanced scalability and the optimization of authentication systems. Yet, the development of an accurate digital twin is far from trivial.

  This paper extends previous research which modelled a printing-imaging channel using a machine learning-based digital twin for CDP. This model, built upon an information-theoretic framework known as "Turbo", demonstrated superior performance over traditional generative models such as CycleGAN and pix2pix. However, the emerging field of Denoising Diffusion Probabilistic Models (DDPM) presents a potential advancement in generative models due to its ability to stochastically model the inherent randomness of the printing-imaging process, and its impressive performance in image-to-image translation tasks.

  This study aims at comparing the capabilities of the Turbo framework and DDPM on the same CDP datasets, with the goal of establishing the real-world benefits of DDPM models for digital twin applications in CDP security. Furthermore, the paper seeks to evaluate the generative potential of the studied models in the context of mobile phone data acquisition. Despite the increased complexity of DDPM methods when compared to traditional approaches, our study highlights their advantages and explores their potential for future applications.

Title: On Security Strategies for Addressing Potential Vulnerabilities in 6G Technologies Deployable in Healthcare. (arXiv:2309.16714v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.16714
• Code URL: null
• Copy Paste: [[2309.16714]] On Security Strategies for Addressing Potential Vulnerabilities in 6G Technologies Deployable in Healthcare(http://arxiv.org/abs/2309.16714)
• Summary:

  Researchers are now focusing on 6G as a new network technology that will bring significant gains over the previous generations while many sectors are still implementing the 5G network in their business processes and operations. Meanwhile, key technological fields that will be influenced by 6G networks have been identified. These include distributed artificial intelligence, intelligent radio, real-time intelligent edge computing, and 3D intercoms. Additionally, each area and potential application of 6G is supported by relevant emerging technologies. Nevertheless, these 6G technology and applications have significant security vulnerabilities that must be addressed before the complete adoption of 6G networks. The healthcare is one of the sectors that are benefiting from the great features introduced in the 5G networks that enhance digital communications and data protection; that notwithstanding, there are still security flaws in 5G technologies that can be transferred to the 6G networks if not properly addressed. This paper highlights the key areas of 6G networks that would provide grand support for the development of healthcare systems. It also identifies certain vulnerabilities in the previous cellular networks that are transferable to 6G networks, and suggests security strategies including zero trust initiatives that could be implemented to address the security concerns.

Title: Layered Security Guidance for Data Asset Management in Additive Manufacturing. (arXiv:2309.16842v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.16842
• Code URL: null
• Copy Paste: [[2309.16842]] Layered Security Guidance for Data Asset Management in Additive Manufacturing(http://arxiv.org/abs/2309.16842)
• Summary:

  Manufacturing industries are increasingly adopting additive manufacturing (AM) technologies to produce functional parts in critical systems. However, the inherent complexity of both AM designs and AM processes render them attractive targets for cyber-attacks. Risk-based Information Technology (IT) and Operational Technology (OT) security guidance standards are useful resources for AM security practitioners, but the guidelines they provide are insufficient without additional AM-specific revisions. Therefore, a structured layering approach is needed to efficiently integrate these revisions with preexisting IT and OT security guidance standards. To implement such an approach, this paper proposes leveraging the National Institute of Standards and Technology's Cybersecurity Framework (CSF) to develop layered, risk-based guidance for fulfilling specific security outcomes. It begins with an in-depth literature review that reveals the importance of AM data and asset management to risk-based security. Next, this paper adopts the CSF asset identification and management security outcomes as an example for providing AM-specific guidance and identifies the AM geometry and process definitions to aid manufacturers in mapping data flows and documenting processes. Finally, this paper uses the Open Security Controls Assessment Language to integrate the AM-specific guidance together with existing IT and OT security guidance in a rigorous and traceable manner. This paper's contribution is to show how a risk-based layered approach enables the authoring, publishing, and management of AM-specific security guidance that is currently lacking. The authors believe implementation of the layered approach would result in value-added, non-redundant security guidance for AM that is consistent with the preexisting guidance.

Title: Unaware, Unfunded and Uneducated: A Systematic Review of SME Cybersecurity. (arXiv:2309.17186v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.17186
• Code URL: null
• Copy Paste: [[2309.17186]] Unaware, Unfunded and Uneducated: A Systematic Review of SME Cybersecurity(http://arxiv.org/abs/2309.17186)
• Summary:

  Small and Medium Enterprises (SMEs) are pivotal in the global economy, accounting for over 90% of businesses and 60% of employment worldwide. Despite their significance, SMEs have been disregarded from cybersecurity initiatives, rendering them ill-equipped to deal with the growing frequency, sophistication, and destructiveness of cyber-attacks. We systematically reviewed the cybersecurity literature on SMEs published between 2017 and 2023.

  We focus on research discussing cyber threats, adopted controls, challenges, and constraints SMEs face in pursuing cybersecurity resilience.

  Our search yielded 916 studies that we narrowed to 77 relevant papers. We identified 44 unique themes and categorised them as novel findings or established knowledge. This distinction revealed that research on SMEs is shallow and has made little progress in understanding SMEs' roles, threats, and needs. Studies often repeated early discoveries without replicating or offering new insights.

  The existing research indicates that the main challenges to attaining cybersecurity resilience of SMEs are a lack of awareness of the cybersecurity risks, limited cybersecurity literacy and constrained financial resources. However, resource availability varied between developed and developing countries. Our analysis indicated a relationship among these themes, suggesting that limited literacy is the root cause of awareness and resource constraint issues.

privacy

Title: LatticeGen: A Cooperative Framework which Hides Generated Text in a Lattice for Privacy-Aware Generation on Cloud. (arXiv:2309.17157v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.17157
• Code URL: null
• Copy Paste: [[2309.17157]] LatticeGen: A Cooperative Framework which Hides Generated Text in a Lattice for Privacy-Aware Generation on Cloud(http://arxiv.org/abs/2309.17157)
• Summary:

  In the current user-server interaction paradigm of prompted generation with large language models (LLM) on cloud, the server fully controls the generation process, which leaves zero options for users who want to keep the generated text to themselves. We propose LatticeGen, a cooperative framework in which the server still handles most of the computation while the user controls the sampling operation. The key idea is that the true generated sequence is mixed with noise tokens by the user and hidden in a noised lattice. Considering potential attacks from a hypothetically malicious server and how the user can defend against it, we propose the repeated beam-search attack and the mixing noise scheme. In our experiments we apply LatticeGen to protect both prompt and generation. It is shown that while the noised lattice degrades generation quality, LatticeGen successfully protects the true generation to a remarkable degree under strong attacks (more than 50% of the semantic remains hidden as measured by BERTScore).

Title: Incompatibilities Between Current Practices in Statistical Data Analysis and Differential Privacy. (arXiv:2309.16703v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.16703
• Code URL: null
• Copy Paste: [[2309.16703]] Incompatibilities Between Current Practices in Statistical Data Analysis and Differential Privacy(http://arxiv.org/abs/2309.16703)
• Summary:

  The authors discuss their experience applying differential privacy with a complex data set with the goal of enabling standard approaches to statistical data analysis. They highlight lessons learned and roadblocks encountered, distilling them into incompatibilities between current practices in statistical data analysis and differential privacy that go beyond issues which can be solved with a noisy measurements file. The authors discuss how overcoming these incompatibilities require compromise and a change in either our approach to statistical data analysis or differential privacy that should be addressed head-on.

Title: Leave-one-out Distinguishability in Machine Learning. (arXiv:2309.17310v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.17310
• Code URL: null
• Copy Paste: [[2309.17310]] Leave-one-out Distinguishability in Machine Learning(http://arxiv.org/abs/2309.17310)
• Summary:

  We introduce a new analytical framework to quantify the changes in a machine learning algorithm's output distribution following the inclusion of a few data points in its training set, a notion we define as leave-one-out distinguishability (LOOD). This problem is key to measuring data **memorization** and **information leakage** in machine learning, and the **influence** of training data points on model predictions. We illustrate how our method broadens and refines existing empirical measures of memorization and privacy risks associated with training data. We use Gaussian processes to model the randomness of machine learning algorithms, and validate LOOD with extensive empirical analysis of information leakage using membership inference attacks. Our theoretical framework enables us to investigate the causes of information leakage and where the leakage is high. For example, we analyze the influence of activation functions, on data memorization. Additionally, our method allows us to optimize queries that disclose the most significant information about the training data in the leave-one-out setting. We illustrate how optimal queries can be used for accurate **reconstruction** of training data.

protect

Title: An Investigation Into Race Bias in Random Forest Models Based on Breast DCE-MRI Derived Radiomics Features. (arXiv:2309.17197v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.17197
• Code URL: null
• Copy Paste: [[2309.17197]] An Investigation Into Race Bias in Random Forest Models Based on Breast DCE-MRI Derived Radiomics Features(http://arxiv.org/abs/2309.17197)
• Summary:

  Recent research has shown that artificial intelligence (AI) models can exhibit bias in performance when trained using data that are imbalanced by protected attribute(s). Most work to date has focused on deep learning models, but classical AI techniques that make use of hand-crafted features may also be susceptible to such bias. In this paper we investigate the potential for race bias in random forest (RF) models trained using radiomics features. Our application is prediction of tumour molecular subtype from dynamic contrast enhanced magnetic resonance imaging (DCE-MRI) of breast cancer patients. Our results show that radiomics features derived from DCE-MRI data do contain race-identifiable information, and that RF models can be trained to predict White and Black race from these data with 60-70% accuracy, depending on the subset of features used. Furthermore, RF models trained to predict tumour molecular subtype using race-imbalanced data seem to produce biased behaviour, exhibiting better performance on test data from the race on which they were trained.

defense

Title: Investigating Human-Identifiable Features Hidden in Adversarial Perturbations. (arXiv:2309.16878v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.16878
• Code URL: null
• Copy Paste: [[2309.16878]] Investigating Human-Identifiable Features Hidden in Adversarial Perturbations(http://arxiv.org/abs/2309.16878)
• Summary:

  Neural networks perform exceedingly well across various machine learning tasks but are not immune to adversarial perturbations. This vulnerability has implications for real-world applications. While much research has been conducted, the underlying reasons why neural networks fall prey to adversarial attacks are not yet fully understood. Central to our study, which explores up to five attack algorithms across three datasets, is the identification of human-identifiable features in adversarial perturbations. Additionally, we uncover two distinct effects manifesting within human-identifiable features. Specifically, the masking effect is prominent in untargeted attacks, while the generation effect is more common in targeted attacks. Using pixel-level annotations, we extract such features and demonstrate their ability to compromise target models. In addition, our findings indicate a notable extent of similarity in perturbations across different attack algorithms when averaged over multiple models. This work also provides insights into phenomena associated with adversarial perturbations, such as transferability and model interpretability. Our study contributes to a deeper understanding of the underlying mechanisms behind adversarial attacks and offers insights for the development of more resilient defense strategies for neural networks.

Title: Toward Robust Recommendation via Real-time Vicinal Defense. (arXiv:2309.17278v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.17278
• Code URL: null
• Copy Paste: [[2309.17278]] Toward Robust Recommendation via Real-time Vicinal Defense(http://arxiv.org/abs/2309.17278)
• Summary:

  Recommender systems have been shown to be vulnerable to poisoning attacks, where malicious data is injected into the dataset to cause the recommender system to provide biased recommendations. To defend against such attacks, various robust learning methods have been proposed. However, most methods are model-specific or attack-specific, making them lack generality, while other methods, such as adversarial training, are oriented towards evasion attacks and thus have a weak defense strength in poisoning attacks.

  In this paper, we propose a general method, Real-time Vicinal Defense (RVD), which leverages neighboring training data to fine-tune the model before making a recommendation for each user. RVD works in the inference phase to ensure the robustness of the specific sample in real-time, so there is no need to change the model structure and training process, making it more practical. Extensive experimental results demonstrate that RVD effectively mitigates targeted poisoning attacks across various models without sacrificing accuracy. Moreover, the defensive effect can be further amplified when our method is combined with other strategies.

attack

Title: AIR: Threats of Adversarial Attacks on Deep Learning-Based Information Recovery. (arXiv:2309.16706v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.16706
• Code URL: null
• Copy Paste: [[2309.16706]] AIR: Threats of Adversarial Attacks on Deep Learning-Based Information Recovery(http://arxiv.org/abs/2309.16706)
• Summary:

  A wireless communications system usually consists of a transmitter which transmits the information and a receiver which recovers the original information from the received distorted signal. Deep learning (DL) has been used to improve the performance of the receiver in complicated channel environments and state-of-the-art (SOTA) performance has been achieved. However, its robustness has not been investigated. In order to evaluate the robustness of DL-based information recovery models under adversarial circumstances, we investigate adversarial attacks on the SOTA DL-based information recovery model, i.e., DeepReceiver. We formulate the problem as an optimization problem with power and peak-to-average power ratio (PAPR) constraints. We design different adversarial attack methods according to the adversary's knowledge of DeepReceiver's model and/or testing samples. Extensive experiments show that the DeepReceiver is vulnerable to the designed attack methods in all of the considered scenarios. Even in the scenario of both model and test sample restricted, the adversary can attack the DeepReceiver and increase its bit error rate (BER) above 10%. It can also be found that the DeepReceiver is vulnerable to adversarial perturbations even with very low power and limited PAPR. These results suggest that defense measures should be taken to enhance the robustness of DeepReceiver.

Title: Leveraging Optimization for Adaptive Attacks on Image Watermarks. (arXiv:2309.16952v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.16952
• Code URL: null
• Copy Paste: [[2309.16952]] Leveraging Optimization for Adaptive Attacks on Image Watermarks(http://arxiv.org/abs/2309.16952)
• Summary:

  Untrustworthy users can misuse image generators to synthesize high-quality deepfakes and engage in online spam or disinformation campaigns. Watermarking deters misuse by marking generated content with a hidden message, enabling its detection using a secret watermarking key. A core security property of watermarking is robustness, which states that an attacker can only evade detection by substantially degrading image quality. Assessing robustness requires designing an adaptive attack for the specific watermarking algorithm. A challenge when evaluating watermarking algorithms and their (adaptive) attacks is to determine whether an adaptive attack is optimal, i.e., it is the best possible attack. We solve this problem by defining an objective function and then approach adaptive attacks as an optimization problem. The core idea of our adaptive attacks is to replicate secret watermarking keys locally by creating surrogate keys that are differentiable and can be used to optimize the attack's parameters. We demonstrate for Stable Diffusion models that such an attacker can break all five surveyed watermarking methods at negligible degradation in image quality. These findings emphasize the need for more rigorous robustness testing against adaptive, learnable attackers.

Title: Post-Training Overfitting Mitigation in DNN Classifiers. (arXiv:2309.16827v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.16827
• Code URL: null
• Copy Paste: [[2309.16827]] Post-Training Overfitting Mitigation in DNN Classifiers(http://arxiv.org/abs/2309.16827)
• Summary:

  Well-known (non-malicious) sources of overfitting in deep neural net (DNN) classifiers include: i) large class imbalances; ii) insufficient training-set diversity; and iii) over-training. In recent work, it was shown that backdoor data-poisoning also induces overfitting, with unusually large classification margins to the attacker's target class, mediated particularly by (unbounded) ReLU activations that allow large signals to propagate in the DNN. Thus, an effective post-training (with no knowledge of the training set or training process) mitigation approach against backdoors was proposed, leveraging a small clean dataset, based on bounding neural activations. Improving upon that work, we threshold activations specifically to limit maximum margins (MMs), which yields performance gains in backdoor mitigation. We also provide some analytical support for this mitigation approach. Most importantly, we show that post-training MM-based regularization substantially mitigates non-malicious overfitting due to class imbalances and overtraining. Thus, unlike adversarial training, which provides some resilience against attacks but which harms clean (attack-free) generalization, we demonstrate an approach originating from adversarial learning that helps clean generalization accuracy. Experiments on CIFAR-10 and CIFAR-100, in comparison with peer methods, demonstrate strong performance of our methods.

Title: The Lipschitz-Variance-Margin Tradeoff for Enhanced Randomized Smoothing. (arXiv:2309.16883v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.16883
• Code URL: null
• Copy Paste: [[2309.16883]] The Lipschitz-Variance-Margin Tradeoff for Enhanced Randomized Smoothing(http://arxiv.org/abs/2309.16883)
• Summary:

  Real-life applications of deep neural networks are hindered by their unsteady predictions when faced with noisy inputs and adversarial attacks. The certified radius is in this context a crucial indicator of the robustness of models. However how to design an efficient classifier with a sufficient certified radius? Randomized smoothing provides a promising framework by relying on noise injection in inputs to obtain a smoothed and more robust classifier. In this paper, we first show that the variance introduced by randomized smoothing closely interacts with two other important properties of the classifier, i.e. its Lipschitz constant and margin. More precisely, our work emphasizes the dual impact of the Lipschitz constant of the base classifier, on both the smoothed classifier and the empirical variance. Moreover, to increase the certified robust radius, we introduce a different simplex projection technique for the base classifier to leverage the variance-margin trade-off thanks to Bernstein's concentration inequality, along with an enhanced Lipschitz bound. Experimental results show a significant improvement in certified accuracy compared to current state-of-the-art methods. Our novel certification procedure allows us to use pre-trained models that are used with randomized smoothing, effectively improving the current certification radius in a zero-shot manner.

Title: Algorithmic Recourse for Anomaly Detection in Multivariate Time Series. (arXiv:2309.16896v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.16896
• Code URL: null
• Copy Paste: [[2309.16896]] Algorithmic Recourse for Anomaly Detection in Multivariate Time Series(http://arxiv.org/abs/2309.16896)
• Summary:

  Anomaly detection in multivariate time series has received extensive study due to the wide spectrum of applications. An anomaly in multivariate time series usually indicates a critical event, such as a system fault or an external attack. Therefore, besides being effective in anomaly detection, recommending anomaly mitigation actions is also important in practice yet under-investigated. In this work, we focus on algorithmic recourse in time series anomaly detection, which is to recommend fixing actions on abnormal time series with a minimum cost so that domain experts can understand how to fix the abnormal behavior. To this end, we propose an algorithmic recourse framework, called RecAD, which can recommend recourse actions to flip the abnormal time steps. Experiments on two synthetic and one real-world datasets show the effectiveness of our framework.

Title: Medical Foundation Models are Susceptible to Targeted Misinformation Attacks. (arXiv:2309.17007v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.17007
• Code URL: https://github.com/peterhan91/fm_adv
• Copy Paste: [[2309.17007]] Medical Foundation Models are Susceptible to Targeted Misinformation Attacks(http://arxiv.org/abs/2309.17007)
• Summary:

  Large language models (LLMs) have broad medical knowledge and can reason about medical information across many domains, holding promising potential for diverse medical applications in the near future. In this study, we demonstrate a concerning vulnerability of LLMs in medicine. Through targeted manipulation of just 1.1% of the model's weights, we can deliberately inject an incorrect biomedical fact. The erroneous information is then propagated in the model's output, whilst its performance on other biomedical tasks remains intact. We validate our findings in a set of 1,038 incorrect biomedical facts. This peculiar susceptibility raises serious security and trustworthiness concerns for the application of LLMs in healthcare settings. It accentuates the need for robust protective measures, thorough verification mechanisms, and stringent management of access to these models, ensuring their reliable and safe use in medical practice.

Title: Efficient Biologically Plausible Adversarial Training. (arXiv:2309.17348v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.17348
• Code URL: null
• Copy Paste: [[2309.17348]] Efficient Biologically Plausible Adversarial Training(http://arxiv.org/abs/2309.17348)
• Summary:

  Artificial Neural Networks (ANNs) trained with Backpropagation (BP) show astounding performance and are increasingly often used in performing our daily life tasks. However, ANNs are highly vulnerable to adversarial attacks, which alter inputs with small targeted perturbations that drastically disrupt the models' performance. The most effective method to make ANNs robust against these attacks is adversarial training, in which the training dataset is augmented with exemplary adversarial samples. Unfortunately, this approach has the drawback of increased training complexity since generating adversarial samples is very computationally demanding. In contrast to ANNs, humans are not susceptible to adversarial attacks. Therefore, in this work, we investigate whether biologically-plausible learning algorithms are more robust against adversarial attacks than BP. In particular, we present an extensive comparative analysis of the adversarial robustness of BP and \textit{Present the Error to Perturb the Input To modulate Activity} (PEPITA), a recently proposed biologically-plausible learning algorithm, on various computer vision tasks. We observe that PEPITA has higher intrinsic adversarial robustness and, with adversarial training, has a more favourable natural-vs-adversarial performance trade-off as, for the same natural accuracies, PEPITA's adversarial accuracies decrease in average by 0.26% and BP's by 8.05%.

robust

Title: MVMR: Evaluating Natural Language Video Localization Bias over Multiple Reliable Videos Pool. (arXiv:2309.16701v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.16701
• Code URL: null
• Copy Paste: [[2309.16701]] MVMR: Evaluating Natural Language Video Localization Bias over Multiple Reliable Videos Pool(http://arxiv.org/abs/2309.16701)
• Summary:

  With the explosion of multimedia content in recent years, natural language video localization, which focuses on detecting video moment that matches a given natural language query, has become a critical problem. However, none of the previous research explores localizing a moment from a large corpus where multiple positive and negative videos exist. In this paper, we propose an MVMR (Massive Videos Moment Retrieval) task, which aims to localize video frames from a massive set of videos given a text query. For this task, we suggest methods for constructing datasets by employing similarity filtering on the existing video localization datasets and introduce three MVMR datasets. Specifically, we employ embedding-based text similarity matching and video-language grounding techniques to calculate the relevance score between a target query and videos to define positive and negative sets. For the proposed MVMR task, we further develop a strong model, Reliable Mutual Matching Network (RMMN), which employs a contrastive learning scheme that selectively filters the reliable and informative negatives leading the model more robust on the MVMR task. Experimental results on the introduced datasets reveal that existing NLVL models are easily distracted by negative video frames, whereas our model shows significant performance.

Title: General Lipschitz: Certified Robustness Against Resolvable Semantic Transformations via Transformation-Dependent Randomized Smoothing. (arXiv:2309.16710v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.16710
• Code URL: null
• Copy Paste: [[2309.16710]] General Lipschitz: Certified Robustness Against Resolvable Semantic Transformations via Transformation-Dependent Randomized Smoothing(http://arxiv.org/abs/2309.16710)
• Summary:

  Randomized smoothing is the state-of-the-art approach to construct image classifiers that are provably robust against additive adversarial perturbations of bounded magnitude. However, it is more complicated to construct reasonable certificates against semantic transformation (e.g., image blurring, translation, gamma correction) and their compositions. In this work, we propose \emph{General Lipschitz (GL),} a new framework to certify neural networks against composable resolvable semantic perturbations. Within the framework, we analyze transformation-dependent Lipschitz-continuity of smoothed classifiers w.r.t. transformation parameters and derive corresponding robustness certificates. Our method performs comparably to state-of-the-art approaches on the ImageNet dataset.

Title: XVO: Generalized Visual Odometry via Cross-Modal Self-Training. (arXiv:2309.16772v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.16772
• Code URL: null
• Copy Paste: [[2309.16772]] XVO: Generalized Visual Odometry via Cross-Modal Self-Training(http://arxiv.org/abs/2309.16772)
• Summary:

  We propose XVO, a semi-supervised learning method for training generalized monocular Visual Odometry (VO) models with robust off-the-self operation across diverse datasets and settings. In contrast to standard monocular VO approaches which often study a known calibration within a single dataset, XVO efficiently learns to recover relative pose with real-world scale from visual scene semantics, i.e., without relying on any known camera parameters. We optimize the motion estimation model via self-training from large amounts of unconstrained and heterogeneous dash camera videos available on YouTube. Our key contribution is twofold. First, we empirically demonstrate the benefits of semi-supervised training for learning a general-purpose direct VO regression network. Second, we demonstrate multi-modal supervision, including segmentation, flow, depth, and audio auxiliary prediction tasks, to facilitate generalized representations for the VO task. Specifically, we find audio prediction task to significantly enhance the semi-supervised learning process while alleviating noisy pseudo-labels, particularly in highly dynamic and out-of-domain video data. Our proposed teacher network achieves state-of-the-art performance on the commonly used KITTI benchmark despite no multi-frame optimization or knowledge of camera parameters. Combined with the proposed semi-supervised step, XVO demonstrates off-the-shelf knowledge transfer across diverse conditions on KITTI, nuScenes, and Argoverse without fine-tuning.

Title: Incremental Rotation Averaging Revisited and More: A New Rotation Averaging Benchmark. (arXiv:2309.16924v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.16924
• Code URL: null
• Copy Paste: [[2309.16924]] Incremental Rotation Averaging Revisited and More: A New Rotation Averaging Benchmark(http://arxiv.org/abs/2309.16924)
• Summary:

  In order to further advance the accuracy and robustness of the incremental parameter estimation-based rotation averaging methods, in this paper, a new member of the Incremental Rotation Averaging (IRA) family is introduced, which is termed as IRAv4. As the most significant feature of the IRAv4, a task-specific connected dominating set is extracted to serve as a more reliable and accurate reference for rotation global alignment. In addition, to further address the limitations of the existing rotation averaging benchmark of relying on the slightly outdated Bundler camera calibration results as ground truths and focusing solely on rotation estimation accuracy, this paper presents a new COLMAP-based rotation averaging benchmark that incorporates a cross check between COLMAP and Bundler, and employ the accuracy of both rotation and downstream location estimation as evaluation metrics, which is desired to provide a more reliable and comprehensive evaluation tool for the rotation averaging research. Comprehensive comparisons between the proposed IRAv4 and other mainstream rotation averaging methods on this new benchmark demonstrate the effectiveness of our proposed approach.

Title: Robust Asynchronous Collaborative 3D Detection via Bird's Eye View Flow. (arXiv:2309.16940v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.16940
• Code URL: null
• Copy Paste: [[2309.16940]] Robust Asynchronous Collaborative 3D Detection via Bird's Eye View Flow(http://arxiv.org/abs/2309.16940)
• Summary:

  By facilitating communication among multiple agents, collaborative perception can substantially boost each agent's perception ability. However, temporal asynchrony among agents is inevitable in real-world due to communication delays, interruptions, and clock misalignments. This issue causes information mismatch during multi-agent fusion, seriously shaking the foundation of collaboration. To address this issue, we propose CoBEVFlow, an asynchrony-robust collaborative 3D perception system based on bird's eye view (BEV) flow. The key intuition of CoBEVFlow is to compensate motions to align asynchronous collaboration messages sent by multiple agents. To model the motion in a scene, we propose BEV flow, which is a collection of the motion vector corresponding to each spatial location. Based on BEV flow, asynchronous perceptual features can be reassigned to appropriate positions, mitigating the impact of asynchrony. CoBEVFlow has two advantages: (i) CoBEVFlow can handle asynchronous collaboration messages sent at irregular, continuous time stamps without discretization; and (ii) with BEV flow, CoBEVFlow only transports the original perceptual features, instead of generating new perceptual features, avoiding additional noises. To validate CoBEVFlow's efficacy, we create IRregular V2V(IRV2V), the first synthetic collaborative perception dataset with various temporal asynchronies that simulate different real-world scenarios. Extensive experiments conducted on both IRV2V and the real-world dataset DAIR-V2X show that CoBEVFlow consistently outperforms other baselines and is robust in extremely asynchronous settings. The code will be released.

Title: CrossZoom: Simultaneously Motion Deblurring and Event Super-Resolving. (arXiv:2309.16949v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.16949
• Code URL: null
• Copy Paste: [[2309.16949]] CrossZoom: Simultaneously Motion Deblurring and Event Super-Resolving(http://arxiv.org/abs/2309.16949)
• Summary:

  Even though the collaboration between traditional and neuromorphic event cameras brings prosperity to frame-event based vision applications, the performance is still confined by the resolution gap crossing two modalities in both spatial and temporal domains. This paper is devoted to bridging the gap by increasing the temporal resolution for images, i.e., motion deblurring, and the spatial resolution for events, i.e., event super-resolving, respectively. To this end, we introduce CrossZoom, a novel unified neural Network (CZ-Net) to jointly recover sharp latent sequences within the exposure period of a blurry input and the corresponding High-Resolution (HR) events. Specifically, we present a multi-scale blur-event fusion architecture that leverages the scale-variant properties and effectively fuses cross-modality information to achieve cross-enhancement. Attention-based adaptive enhancement and cross-interaction prediction modules are devised to alleviate the distortions inherent in Low-Resolution (LR) events and enhance the final results through the prior blur-event complementary information. Furthermore, we propose a new dataset containing HR sharp-blurry images and the corresponding HR-LR event streams to facilitate future research. Extensive qualitative and quantitative experiments on synthetic and real-world datasets demonstrate the effectiveness and robustness of the proposed method. Codes and datasets are released at https://bestrivenzc.github.io/CZ-Net/.

Title: AdaPose: Towards Cross-Site Device-Free Human Pose Estimation with Commodity WiFi. (arXiv:2309.16964v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.16964
• Code URL: null
• Copy Paste: [[2309.16964]] AdaPose: Towards Cross-Site Device-Free Human Pose Estimation with Commodity WiFi(http://arxiv.org/abs/2309.16964)
• Summary:

  WiFi-based pose estimation is a technology with great potential for the development of smart homes and metaverse avatar generation. However, current WiFi-based pose estimation methods are predominantly evaluated under controlled laboratory conditions with sophisticated vision models to acquire accurately labeled data. Furthermore, WiFi CSI is highly sensitive to environmental variables, and direct application of a pre-trained model to a new environment may yield suboptimal results due to domain shift. In this paper, we proposes a domain adaptation algorithm, AdaPose, designed specifically for weakly-supervised WiFi-based pose estimation. The proposed method aims to identify consistent human poses that are highly resistant to environmental dynamics. To achieve this goal, we introduce a Mapping Consistency Loss that aligns the domain discrepancy of source and target domains based on inner consistency between input and output at the mapping level. We conduct extensive experiments on domain adaptation in two different scenes using our self-collected pose estimation dataset containing WiFi CSI frames. The results demonstrate the effectiveness and robustness of AdaPose in eliminating domain shift, thereby facilitating the widespread application of WiFi-based pose estimation in smart cities.

Title: nnSAM: Plug-and-play Segment Anything Model Improves nnUNet Performance. (arXiv:2309.16967v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.16967
• Code URL: https://github.com/kent0n-li/medical-image-segmentation
• Copy Paste: [[2309.16967]] nnSAM: Plug-and-play Segment Anything Model Improves nnUNet Performance(http://arxiv.org/abs/2309.16967)
• Summary:

  The recent developments of foundation models in computer vision, especially the Segment Anything Model (SAM), allow scalable and domain-agnostic image segmentation to serve as a general-purpose segmentation tool. In parallel, the field of medical image segmentation has benefited significantly from specialized neural networks like the nnUNet, which is trained on domain-specific datasets and can automatically configure the network to tailor to specific segmentation challenges. To combine the advantages of foundation models and domain-specific models, we present nnSAM, which synergistically integrates the SAM model with the nnUNet model to achieve more accurate and robust medical image segmentation. The nnSAM model leverages the powerful and robust feature extraction capabilities of SAM, while harnessing the automatic configuration capabilities of nnUNet to promote dataset-tailored learning. Our comprehensive evaluation of nnSAM model on different sizes of training samples shows that it allows few-shot learning, which is highly relevant for medical image segmentation where high-quality, annotated data can be scarce and costly to obtain. By melding the strengths of both its predecessors, nnSAM positions itself as a potential new benchmark in medical image segmentation, offering a tool that combines broad applicability with specialized efficiency. The code is available at https://github.com/Kent0n-Li/Medical-Image-Segmentation.

Title: Imagery Dataset for Condition Monitoring of Synthetic Fibre Ropes. (arXiv:2309.17058v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.17058
• Code URL: null
• Copy Paste: [[2309.17058]] Imagery Dataset for Condition Monitoring of Synthetic Fibre Ropes(http://arxiv.org/abs/2309.17058)
• Summary:

  Automatic visual inspection of synthetic fibre ropes (SFRs) is a challenging task in the field of offshore, wind turbine industries, etc. The presence of any defect in SFRs can compromise their structural integrity and pose significant safety risks. Due to the large size and weight of these ropes, it is often impractical to detach and inspect them frequently. Therefore, there is a critical need to develop efficient defect detection methods to assess their remaining useful life (RUL). To address this challenge, a comprehensive dataset has been generated, comprising a total of 6,942 raw images representing both normal and defective SFRs. The dataset encompasses a wide array of defect scenarios which may occur throughout their operational lifespan, including but not limited to placking defects, cut strands, chafings, compressions, core outs and normal. This dataset serves as a resource to support computer vision applications, including object detection, classification, and segmentation, aimed at detecting and analyzing defects in SFRs. The availability of this dataset will facilitate the development and evaluation of robust defect detection algorithms. The aim of generating this dataset is to assist in the development of automated defect detection systems that outperform traditional visual inspection methods, thereby paving the way for safer and more efficient utilization of SFRs across a wide range of applications.

Title: Revisiting Cephalometric Landmark Detection from the view of Human Pose Estimation with Lightweight Super-Resolution Head. (arXiv:2309.17143v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.17143
• Code URL: https://github.com/5k5000/cldetection2023
• Copy Paste: [[2309.17143]] Revisiting Cephalometric Landmark Detection from the view of Human Pose Estimation with Lightweight Super-Resolution Head(http://arxiv.org/abs/2309.17143)
• Summary:

  Accurate localization of cephalometric landmarks holds great importance in the fields of orthodontics and orthognathics due to its potential for automating key point labeling. In the context of landmark detection, particularly in cephalometrics, it has been observed that existing methods often lack standardized pipelines and well-designed bias reduction processes, which significantly impact their performance. In this paper, we revisit a related task, human pose estimation (HPE), which shares numerous similarities with cephalometric landmark detection (CLD), and emphasize the potential for transferring techniques from the former field to benefit the latter. Motivated by this insight, we have developed a robust and adaptable benchmark based on the well-established HPE codebase known as MMPose. This benchmark can serve as a dependable baseline for achieving exceptional CLD performance. Furthermore, we introduce an upscaling design within the framework to further enhance performance. This enhancement involves the incorporation of a lightweight and efficient super-resolution module, which generates heatmap predictions on high-resolution features and leads to further performance refinement, benefiting from its ability to reduce quantization bias. In the MICCAI CLDetection2023 challenge, our method achieves 1st place ranking on three metrics and 3rd place on the remaining one. The code for our method is available at https://github.com/5k5000/CLdetection2023.

Title: Prototype Generation: Robust Feature Visualisation for Data Independent Interpretability. (arXiv:2309.17144v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.17144
• Code URL: null
• Copy Paste: [[2309.17144]] Prototype Generation: Robust Feature Visualisation for Data Independent Interpretability(http://arxiv.org/abs/2309.17144)
• Summary:

  We introduce Prototype Generation, a stricter and more robust form of feature visualisation for model-agnostic, data-independent interpretability of image classification models. We demonstrate its ability to generate inputs that result in natural activation paths, countering previous claims that feature visualisation algorithms are untrustworthy due to the unnatural internal activations. We substantiate these claims by quantitatively measuring similarity between the internal activations of our generated prototypes and natural images. We also demonstrate how the interpretation of generated prototypes yields important insights, highlighting spurious correlations and biases learned by models which quantitative methods over test-sets cannot identify.

Title: Domain-Adaptive Learning: Unsupervised Adaptation for Histology Images with Improved Loss Function Combination. (arXiv:2309.17172v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.17172
• Code URL: null
• Copy Paste: [[2309.17172]] Domain-Adaptive Learning: Unsupervised Adaptation for Histology Images with Improved Loss Function Combination(http://arxiv.org/abs/2309.17172)
• Summary:

  This paper presents a novel approach for unsupervised domain adaptation (UDA) targeting H&E stained histology images. Existing adversarial domain adaptation methods may not effectively align different domains of multimodal distributions associated with classification problems. The objective is to enhance domain alignment and reduce domain shifts between these domains by leveraging their unique characteristics. Our approach proposes a novel loss function along with carefully selected existing loss functions tailored to address the challenges specific to histology images. This loss combination not only makes the model accurate and robust but also faster in terms of training convergence. We specifically focus on leveraging histology-specific features, such as tissue structure and cell morphology, to enhance adaptation performance in the histology domain. The proposed method is extensively evaluated in accuracy, robustness, and generalization, surpassing state-of-the-art techniques for histology images. We conducted extensive experiments on the FHIST dataset and the results show that our proposed method - Domain Adaptive Learning (DAL) significantly surpasses the ViT-based and CNN-based SoTA methods by 1.41% and 6.56% respectively.

Title: Efficient Large Scale Medical Image Dataset Preparation for Machine Learning Applications. (arXiv:2309.17285v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.17285
• Code URL: null
• Copy Paste: [[2309.17285]] Efficient Large Scale Medical Image Dataset Preparation for Machine Learning Applications(http://arxiv.org/abs/2309.17285)
• Summary:

  In the rapidly evolving field of medical imaging, machine learning algorithms have become indispensable for enhancing diagnostic accuracy. However, the effectiveness of these algorithms is contingent upon the availability and organization of high-quality medical imaging datasets. Traditional Digital Imaging and Communications in Medicine (DICOM) data management systems are inadequate for handling the scale and complexity of data required to be facilitated in machine learning algorithms. This paper introduces an innovative data curation tool, developed as part of the Kaapana open-source toolkit, aimed at streamlining the organization, management, and processing of large-scale medical imaging datasets. The tool is specifically tailored to meet the needs of radiologists and machine learning researchers. It incorporates advanced search, auto-annotation and efficient tagging functionalities for improved data curation. Additionally, the tool facilitates quality control and review, enabling researchers to validate image and segmentation quality in large datasets. It also plays a critical role in uncovering potential biases in datasets by aggregating and visualizing metadata, which is essential for developing robust machine learning models. Furthermore, Kaapana is integrated within the Radiological Cooperative Network (RACOON), a pioneering initiative aimed at creating a comprehensive national infrastructure for the aggregation, transmission, and consolidation of radiological data across all university clinics throughout Germany. A supplementary video showcasing the tool's functionalities can be accessed at https://bit.ly/MICCAI-DEMI2023.

Title: See Beyond Seeing: Robust 3D Object Detection from Point Clouds via Cross-Modal Hallucination. (arXiv:2309.17336v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.17336
• Code URL: null
• Copy Paste: [[2309.17336]] See Beyond Seeing: Robust 3D Object Detection from Point Clouds via Cross-Modal Hallucination(http://arxiv.org/abs/2309.17336)
• Summary:

  This paper presents a novel framework for robust 3D object detection from point clouds via cross-modal hallucination. Our proposed approach is agnostic to either hallucination direction between LiDAR and 4D radar. We introduce multiple alignments on both spatial and feature levels to achieve simultaneous backbone refinement and hallucination generation. Specifically, spatial alignment is proposed to deal with the geometry discrepancy for better instance matching between LiDAR and radar. The feature alignment step further bridges the intrinsic attribute gap between the sensing modalities and stabilizes the training. The trained object detection models can deal with difficult detection cases better, even though only single-modal data is used as the input during the inference stage. Extensive experiments on the View-of-Delft (VoD) dataset show that our proposed method outperforms the state-of-the-art (SOTA) methods for both radar and LiDAR object detection while maintaining competitive efficiency in runtime.

Title: SCALE: Synergized Collaboration of Asymmetric Language Translation Engines. (arXiv:2309.17061v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.17061
• Code URL: https://github.com/hannibal046/scale
• Copy Paste: [[2309.17061]] SCALE: Synergized Collaboration of Asymmetric Language Translation Engines(http://arxiv.org/abs/2309.17061)
• Summary:

  In this paper, we introduce SCALE, a collaborative framework that connects compact Specialized Translation Models (STMs) and general-purpose Large Language Models (LLMs) as one unified translation engine. By introducing translation from STM into the triplet in-context demonstrations, SCALE unlocks refinement and pivoting ability of LLM, thus mitigating language bias of LLM and parallel data bias of STM, enhancing LLM speciality without sacrificing generality, and facilitating continual learning without expensive LLM fine-tuning. Our comprehensive experiments show that SCALE significantly outperforms both few-shot LLMs (GPT-4) and specialized models (NLLB) in challenging low-resource settings. Moreover, in Xhosa to English translation, SCALE experiences consistent improvement by a 4 BLEURT score without tuning LLM and surpasses few-shot GPT-4 by 2.5 COMET score and 3.8 BLEURT score when equipped with a compact model consisting of merely 600M parameters. SCALE could also effectively exploit the existing language bias of LLMs by using an English-centric STM as a pivot for translation between any language pairs, outperforming few-shot GPT-4 by an average of 6 COMET points across eight translation directions. Furthermore we provide an in-depth analysis of SCALE's robustness, translation characteristics, and latency costs, providing solid foundation for future studies exploring the potential synergy between LLMs and more specialized, task-specific models.

Title: SimPINNs: Simulation-Driven Physics-Informed Neural Networks for Enhanced Performance in Nonlinear Inverse Problems. (arXiv:2309.16729v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.16729
• Code URL: null
• Copy Paste: [[2309.16729]] SimPINNs: Simulation-Driven Physics-Informed Neural Networks for Enhanced Performance in Nonlinear Inverse Problems(http://arxiv.org/abs/2309.16729)
• Summary:

  This paper introduces a novel approach to solve inverse problems by leveraging deep learning techniques. The objective is to infer unknown parameters that govern a physical system based on observed data. We focus on scenarios where the underlying forward model demonstrates pronounced nonlinear behaviour, and where the dimensionality of the unknown parameter space is substantially smaller than that of the observations. Our proposed method builds upon physics-informed neural networks (PINNs) trained with a hybrid loss function that combines observed data with simulated data generated by a known (approximate) physical model. Experimental results on an orbit restitution problem demonstrate that our approach surpasses the performance of standard PINNs, providing improved accuracy and robustness.

Title: Discovering environments with XRM. (arXiv:2309.16748v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.16748
• Code URL: null
• Copy Paste: [[2309.16748]] Discovering environments with XRM(http://arxiv.org/abs/2309.16748)
• Summary:

  Successful out-of-distribution generalization requires environment annotations. Unfortunately, these are resource-intensive to obtain, and their relevance to model performance is limited by the expectations and perceptual biases of human annotators. Therefore, to enable robust AI systems across applications, we must develop algorithms to automatically discover environments inducing broad generalization. Current proposals, which divide examples based on their training error, suffer from one fundamental problem. These methods add hyper-parameters and early-stopping criteria that are impossible to tune without a validation set with human-annotated environments, the very information subject to discovery. In this paper, we propose Cross-Risk-Minimization (XRM) to address this issue. XRM trains two twin networks, each learning from one random half of the training data, while imitating confident held-out mistakes made by its sibling. XRM provides a recipe for hyper-parameter tuning, does not require early-stopping, and can discover environments for all training and validation data. Domain generalization algorithms built on top of XRM environments achieve oracle worst-group-accuracy, solving a long-standing problem in out-of-distribution generalization.

Title: Message Propagation Through Time: An Algorithm for Sequence Dependency Retention in Time Series Modeling. (arXiv:2309.16882v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.16882
• Code URL: null
• Copy Paste: [[2309.16882]] Message Propagation Through Time: An Algorithm for Sequence Dependency Retention in Time Series Modeling(http://arxiv.org/abs/2309.16882)
• Summary:

  Time series modeling, a crucial area in science, often encounters challenges when training Machine Learning (ML) models like Recurrent Neural Networks (RNNs) using the conventional mini-batch training strategy that assumes independent and identically distributed (IID) samples and initializes RNNs with zero hidden states. The IID assumption ignores temporal dependencies among samples, resulting in poor performance. This paper proposes the Message Propagation Through Time (MPTT) algorithm to effectively incorporate long temporal dependencies while preserving faster training times relative to the stateful solutions. MPTT utilizes two memory modules to asynchronously manage initial hidden states for RNNs, fostering seamless information exchange between samples and allowing diverse mini-batches throughout epochs. MPTT further implements three policies to filter outdated and preserve essential information in the hidden states to generate informative initial hidden states for RNNs, facilitating robust training. Experimental results demonstrate that MPTT outperforms seven strategies on four climate datasets with varying levels of temporal dependencies.

Title: Towards Robust Offline-to-Online Reinforcement Learning via Uncertainty and Smoothness. (arXiv:2309.16973v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.16973
• Code URL: null
• Copy Paste: [[2309.16973]] Towards Robust Offline-to-Online Reinforcement Learning via Uncertainty and Smoothness(http://arxiv.org/abs/2309.16973)
• Summary:

  To obtain a near-optimal policy with fewer interactions in Reinforcement Learning (RL), a promising approach involves the combination of offline RL, which enhances sample efficiency by leveraging offline datasets, and online RL, which explores informative transitions by interacting with the environment. Offline-to-Online (O2O) RL provides a paradigm for improving an offline trained agent within limited online interactions. However, due to the significant distribution shift between online experiences and offline data, most offline RL algorithms suffer from performance drops and fail to achieve stable policy improvement in O2O adaptation. To address this problem, we propose the Robust Offline-to-Online (RO2O) algorithm, designed to enhance offline policies through uncertainty and smoothness, and to mitigate the performance drop in online adaptation. Specifically, RO2O incorporates Q-ensemble for uncertainty penalty and adversarial samples for policy and value smoothness, which enable RO2O to maintain a consistent learning procedure in online adaptation without requiring special changes to the learning objective. Theoretical analyses in linear MDPs demonstrate that the uncertainty and smoothness lead to a tighter optimality bound in O2O against distribution shift. Experimental results illustrate the superiority of RO2O in facilitating stable offline-to-online learning and achieving significant improvement with limited online interactions.

Title: A Closer Look at Bearing Fault Classification Approaches. (arXiv:2309.17001v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.17001
• Code URL: null
• Copy Paste: [[2309.17001]] A Closer Look at Bearing Fault Classification Approaches(http://arxiv.org/abs/2309.17001)
• Summary:

  Rolling bearing fault diagnosis has garnered increased attention in recent years owing to its presence in rotating machinery across various industries, and an ever increasing demand for efficient operations. Prompt detection and accurate prediction of bearing failures can help reduce the likelihood of unexpected machine downtime and enhance maintenance schedules, averting lost productivity. Recent technological advances have enabled monitoring the health of these assets at scale using a variety of sensors, and predicting the failures using modern Machine Learning (ML) approaches including deep learning architectures. Vibration data has been collected using accelerated run-to-failure of overloaded bearings, or by introducing known failure in bearings, under a variety of operating conditions such as rotating speed, load on the bearing, type of bearing fault, and data acquisition frequency. However, in the development of bearing failure classification models using vibration data there is a lack of consensus in the metrics used to evaluate the models, data partitions used to evaluate models, and methods used to generate failure labels in run-to-failure experiments. An understanding of the impact of these choices is important to reliably develop models, and deploy them in practical settings. In this work, we demonstrate the significance of these choices on the performance of the models using publicly-available vibration datasets, and suggest model development considerations for real world scenarios. Our experimental findings demonstrate that assigning vibration data from a given bearing across training and evaluation splits leads to over-optimistic performance estimates, PCA-based approach is able to robustly generate labels for failure classification in run-to-failure experiments, and $F$ scores are more insightful to evaluate the models with unbalanced real-world failure data.

Title: On Continuity of Robust and Accurate Classifiers. (arXiv:2309.17048v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.17048
• Code URL: null
• Copy Paste: [[2309.17048]] On Continuity of Robust and Accurate Classifiers(http://arxiv.org/abs/2309.17048)
• Summary:

  The reliability of a learning model is key to the successful deployment of machine learning in various applications. Creating a robust model, particularly one unaffected by adversarial attacks, requires a comprehensive understanding of the adversarial examples phenomenon. However, it is difficult to describe the phenomenon due to the complicated nature of the problems in machine learning. It has been shown that adversarial training can improve the robustness of the hypothesis. However, this improvement comes at the cost of decreased performance on natural samples. Hence, it has been suggested that robustness and accuracy of a hypothesis are at odds with each other. In this paper, we put forth the alternative proposal that it is the continuity of a hypothesis that is incompatible with its robustness and accuracy. In other words, a continuous function cannot effectively learn the optimal robust hypothesis. To this end, we will introduce a framework for a rigorous study of harmonic and holomorphic hypothesis in learning theory terms and provide empirical evidence that continuous hypotheses does not perform as well as discontinuous hypotheses in some common machine learning tasks. From a practical point of view, our results suggests that a robust and accurate learning rule would train different continuous hypotheses for different regions of the domain. From a theoretical perspective, our analysis explains the adversarial examples phenomenon as a conflict between the continuity of a sequence of functions and its uniform convergence to a discontinuous function.

Title: RECOMBINER: Robust and Enhanced Compression with Bayesian Implicit Neural Representations. (arXiv:2309.17182v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.17182
• Code URL: null
• Copy Paste: [[2309.17182]] RECOMBINER: Robust and Enhanced Compression with Bayesian Implicit Neural Representations(http://arxiv.org/abs/2309.17182)
• Summary:

  COMpression with Bayesian Implicit NEural Representations (COMBINER) is a recent data compression method that addresses a key inefficiency of previous Implicit Neural Representation (INR)-based approaches: it avoids quantization and enables direct optimization of the rate-distortion performance. However, COMBINER still has significant limitations: 1) it uses factorized priors and posterior approximations that lack flexibility; 2) it cannot effectively adapt to local deviations from global patterns in the data; and 3) its performance can be susceptible to modeling choices and the variational parameters' initializations. Our proposed method, Robust and Enhanced COMBINER (RECOMBINER), addresses these issues by 1) enriching the variational approximation while maintaining its computational cost via a linear reparameterization of the INR weights, 2) augmenting our INRs with learnable positional encodings that enable them to adapt to local details and 3) splitting high-resolution data into patches to increase robustness and utilizing expressive hierarchical priors to capture dependency across patches. We conduct extensive experiments across several data modalities, showcasing that RECOMBINER achieves competitive results with the best INR-based methods and even outperforms autoencoder-based codecs on low-resolution images at low bitrates.

Title: RSAM: Learning on manifolds with Riemannian Sharpness-aware Minimization. (arXiv:2309.17215v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.17215
• Code URL: null
• Copy Paste: [[2309.17215]] RSAM: Learning on manifolds with Riemannian Sharpness-aware Minimization(http://arxiv.org/abs/2309.17215)
• Summary:

  Nowadays, understanding the geometry of the loss landscape shows promise in enhancing a model's generalization ability. In this work, we draw upon prior works that apply geometric principles to optimization and present a novel approach to improve robustness and generalization ability for constrained optimization problems. Indeed, this paper aims to generalize the Sharpness-Aware Minimization (SAM) optimizer to Riemannian manifolds. In doing so, we first extend the concept of sharpness and introduce a novel notion of sharpness on manifolds. To support this notion of sharpness, we present a theoretical analysis characterizing generalization capabilities with respect to manifold sharpness, which demonstrates a tighter bound on the generalization gap, a result not known before. Motivated by this analysis, we introduce our algorithm, Riemannian Sharpness-Aware Minimization (RSAM). To demonstrate RSAM's ability to enhance generalization ability, we evaluate and contrast our algorithm on a broad set of problems, such as image classification and contrastive learning across different datasets, including CIFAR100, CIFAR10, and FGVCAircraft. Our code is publicly available at \url{https://t.ly/RiemannianSAM}.

Title: Demographic Parity: Mitigating Biases in Real-World Data. (arXiv:2309.17347v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.17347
• Code URL: null
• Copy Paste: [[2309.17347]] Demographic Parity: Mitigating Biases in Real-World Data(http://arxiv.org/abs/2309.17347)
• Summary:

  Computer-based decision systems are widely used to automate decisions in many aspects of everyday life, which include sensitive areas like hiring, loaning and even criminal sentencing. A decision pipeline heavily relies on large volumes of historical real-world data for training its models. However, historical training data often contains gender, racial or other biases which are propagated to the trained models influencing computer-based decisions. In this work, we propose a robust methodology that guarantees the removal of unwanted biases while maximally preserving classification utility. Our approach can always achieve this in a model-independent way by deriving from real-world data the asymptotic dataset that uniquely encodes demographic parity and realism. As a proof-of-principle, we deduce from public census records such an asymptotic dataset from which synthetic samples can be generated to train well-established classifiers. Benchmarking the generalization capability of these classifiers trained on our synthetic data, we confirm the absence of any explicit or implicit bias in the computer-aided decision.

biometric

steal

extraction

Title: MV-DeepSDF: Implicit Modeling with Multi-Sweep Point Clouds for 3D Vehicle Reconstruction in Autonomous Driving. (arXiv:2309.16715v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.16715
• Code URL: null
• Copy Paste: [[2309.16715]] MV-DeepSDF: Implicit Modeling with Multi-Sweep Point Clouds for 3D Vehicle Reconstruction in Autonomous Driving(http://arxiv.org/abs/2309.16715)
• Summary:

  Reconstructing 3D vehicles from noisy and sparse partial point clouds is of great significance to autonomous driving. Most existing 3D reconstruction methods cannot be directly applied to this problem because they are elaborately designed to deal with dense inputs with trivial noise. In this work, we propose a novel framework, dubbed MV-DeepSDF, which estimates the optimal Signed Distance Function (SDF) shape representation from multi-sweep point clouds to reconstruct vehicles in the wild. Although there have been some SDF-based implicit modeling methods, they only focus on single-view-based reconstruction, resulting in low fidelity. In contrast, we first analyze multi-sweep consistency and complementarity in the latent feature space and propose to transform the implicit space shape estimation problem into an element-to-set feature extraction problem. Then, we devise a new architecture to extract individual element-level representations and aggregate them to generate a set-level predicted latent code. This set-level latent code is an expression of the optimal 3D shape in the implicit space, and can be subsequently decoded to a continuous SDF of the vehicle. In this way, our approach learns consistent and complementary information among multi-sweeps for 3D vehicle reconstruction. We conduct thorough experiments on two real-world autonomous driving datasets (Waymo and KITTI) to demonstrate the superiority of our approach over state-of-the-art alternative methods both qualitatively and quantitatively.

Title: Unveiling Document Structures with YOLOv5 Layout Detection. (arXiv:2309.17033v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.17033
• Code URL: null
• Copy Paste: [[2309.17033]] Unveiling Document Structures with YOLOv5 Layout Detection(http://arxiv.org/abs/2309.17033)
• Summary:

  The current digital environment is characterized by the widespread presence of data, particularly unstructured data, which poses many issues in sectors including finance, healthcare, and education. Conventional techniques for data extraction encounter difficulties in dealing with the inherent variety and complexity of unstructured data, hence requiring the adoption of more efficient methodologies. This research investigates the utilization of YOLOv5, a cutting-edge computer vision model, for the purpose of rapidly identifying document layouts and extracting unstructured data.

  The present study establishes a conceptual framework for delineating the notion of "objects" as they pertain to documents, incorporating various elements such as paragraphs, tables, photos, and other constituent parts. The main objective is to create an autonomous system that can effectively recognize document layouts and extract unstructured data, hence improving the effectiveness of data extraction.

  In the conducted examination, the YOLOv5 model exhibits notable effectiveness in the task of document layout identification, attaining a high accuracy rate along with a precision value of 0.91, a recall value of 0.971, an F1-score of 0.939, and an area under the receiver operating characteristic curve (AUC-ROC) of 0.975. The remarkable performance of this system optimizes the process of extracting textual and tabular data from document images. Its prospective applications are not limited to document analysis but can encompass unstructured data from diverse sources, such as audio data.

  This study lays the foundation for future investigations into the wider applicability of YOLOv5 in managing various types of unstructured data, offering potential for novel applications across multiple domains.

Title: A 5-Point Minimal Solver for Event Camera Relative Motion Estimation. (arXiv:2309.17054v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.17054
• Code URL: null
• Copy Paste: [[2309.17054]] A 5-Point Minimal Solver for Event Camera Relative Motion Estimation(http://arxiv.org/abs/2309.17054)
• Summary:

  Event-based cameras are ideal for line-based motion estimation, since they predominantly respond to edges in the scene. However, accurately determining the camera displacement based on events continues to be an open problem. This is because line feature extraction and dynamics estimation are tightly coupled when using event cameras, and no precise model is currently available for describing the complex structures generated by lines in the space-time volume of events. We solve this problem by deriving the correct non-linear parametrization of such manifolds, which we term eventails, and demonstrate its application to event-based linear motion estimation, with known rotation from an Inertial Measurement Unit. Using this parametrization, we introduce a novel minimal 5-point solver that jointly estimates line parameters and linear camera velocity projections, which can be fused into a single, averaged linear velocity when considering multiple lines. We demonstrate on both synthetic and real data that our solver generates more stable relative motion estimates than other methods while capturing more inliers than clustering based on spatio-temporal planes. In particular, our method consistently achieves a 100% success rate in estimating linear velocity where existing closed-form solvers only achieve between 23% and 70%. The proposed eventails contribute to a better understanding of spatio-temporal event-generated geometries and we thus believe it will become a core building block of future event-based motion estimation algorithms.

Title: PARF: Primitive-Aware Radiance Fusion for Indoor Scene Novel View Synthesis. (arXiv:2309.17190v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.17190
• Code URL: null
• Copy Paste: [[2309.17190]] PARF: Primitive-Aware Radiance Fusion for Indoor Scene Novel View Synthesis(http://arxiv.org/abs/2309.17190)
• Summary:

  This paper proposes a method for fast scene radiance field reconstruction with strong novel view synthesis performance and convenient scene editing functionality. The key idea is to fully utilize semantic parsing and primitive extraction for constraining and accelerating the radiance field reconstruction process. To fulfill this goal, a primitive-aware hybrid rendering strategy was proposed to enjoy the best of both volumetric and primitive rendering. We further contribute a reconstruction pipeline conducts primitive parsing and radiance field learning iteratively for each input frame which successfully fuses semantic, primitive, and radiance information into a single framework. Extensive evaluations demonstrate the fast reconstruction ability, high rendering quality, and convenient editing functionality of our method.

Title: SSHR: Leveraging Self-supervised Hierarchical Representations for Multilingual Automatic Speech Recognition. (arXiv:2309.16937v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.16937
• Code URL: null
• Copy Paste: [[2309.16937]] SSHR: Leveraging Self-supervised Hierarchical Representations for Multilingual Automatic Speech Recognition(http://arxiv.org/abs/2309.16937)
• Summary:

  Multilingual automatic speech recognition (ASR) systems have garnered attention for their potential to extend language coverage globally. While self-supervised learning (SSL) has demonstrated its effectiveness in multilingual ASR, it is worth noting that the various layers' representations of SSL potentially contain distinct information that has not been fully leveraged. In this study, we propose a novel method that leverages self-supervised hierarchical representations (SSHR) to fine-tune multilingual ASR. We first analyze the different layers of the SSL model for language-related and content-related information, uncovering layers that show a stronger correlation. Then, we extract a language-related frame from correlated middle layers and guide specific content extraction through self-attention mechanisms. Additionally, we steer the model toward acquiring more content-related information in the final layers using our proposed Cross-CTC. We evaluate SSHR on two multilingual datasets, Common Voice and ML-SUPERB, and the experimental results demonstrate that our method achieves state-of-the-art performance to the best of our knowledge.

membership infer

federate

Title: A Survey of Incremental Transfer Learning: Combining Peer-to-Peer Federated Learning and Domain Incremental Learning for Multicenter Collaboration. (arXiv:2309.17192v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.17192
• Code URL: https://github.com/yixinghuang/itlsurvey
• Copy Paste: [[2309.17192]] A Survey of Incremental Transfer Learning: Combining Peer-to-Peer Federated Learning and Domain Incremental Learning for Multicenter Collaboration(http://arxiv.org/abs/2309.17192)
• Summary:

  Due to data privacy constraints, data sharing among multiple clinical centers is restricted, which impedes the development of high performance deep learning models from multicenter collaboration. Naive weight transfer methods share intermediate model weights without raw data and hence can bypass data privacy restrictions. However, performance drops are typically observed when the model is transferred from one center to the next because of the forgetting problem. Incremental transfer learning, which combines peer-to-peer federated learning and domain incremental learning, can overcome the data privacy issue and meanwhile preserve model performance by using continual learning techniques. In this work, a conventional domain/task incremental learning framework is adapted for incremental transfer learning. A comprehensive survey on the efficacy of different regularization-based continual learning methods for multicenter collaboration is performed. The influences of data heterogeneity, classifier head setting, network optimizer, model initialization, center order, and weight transfer type have been investigated thoroughly. Our framework is publicly accessible to the research community for further development.

Title: FENDA-FL: Personalized Federated Learning on Heterogeneous Clinical Datasets. (arXiv:2309.16825v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.16825
• Code URL: https://github.com/vectorinstitute/fl4health
• Copy Paste: [[2309.16825]] FENDA-FL: Personalized Federated Learning on Heterogeneous Clinical Datasets(http://arxiv.org/abs/2309.16825)
• Summary:

  Federated learning (FL) is increasingly being recognized as a key approach to overcoming the data silos that so frequently obstruct the training and deployment of machine-learning models in clinical settings. This work contributes to a growing body of FL research specifically focused on clinical applications along three important directions. First, an extension of the FENDA method (Kim et al., 2016) to the FL setting is proposed. Experiments conducted on the FLamby benchmarks (du Terrail et al., 2022a) and GEMINI datasets (Verma et al., 2017) show that the approach is robust to heterogeneous clinical data and often outperforms existing global and personalized FL techniques. Further, the experimental results represent substantive improvements over the original FLamby benchmarks and expand such benchmarks to include evaluation of personalized FL methods. Finally, we advocate for a comprehensive checkpointing and evaluation framework for FL to better reflect practical settings and provide multiple baselines for comparison.

Title: Applications of Federated Learning in IoT for Hyper Personalisation. (arXiv:2309.16854v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.16854
• Code URL: null
• Copy Paste: [[2309.16854]] Applications of Federated Learning in IoT for Hyper Personalisation(http://arxiv.org/abs/2309.16854)
• Summary:

  Billions of IoT devices are being deployed, taking advantage of faster internet, and the opportunity to access more endpoints. Vast quantities of data are being generated constantly by these devices but are not effectively being utilised. Using FL training machine learning models over these multiple clients without having to bring it to a central server. We explore how to use such a model to implement ultra levels of personalization unlike before

Title: Mode Connectivity and Data Heterogeneity of Federated Learning. (arXiv:2309.16923v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.16923
• Code URL: null
• Copy Paste: [[2309.16923]] Mode Connectivity and Data Heterogeneity of Federated Learning(http://arxiv.org/abs/2309.16923)
• Summary:

  Federated learning (FL) enables multiple clients to train a model while keeping their data private collaboratively. Previous studies have shown that data heterogeneity between clients leads to drifts across client updates. However, there are few studies on the relationship between client and global modes, making it unclear where these updates end up drifting. We perform empirical and theoretical studies on this relationship by utilizing mode connectivity, which measures performance change (i.e., connectivity) along parametric paths between different modes. Empirically, reducing data heterogeneity makes the connectivity on different paths more similar, forming more low-error overlaps between client and global modes. We also find that a barrier to connectivity occurs when linearly connecting two global modes, while it disappears with considering non-linear mode connectivity. Theoretically, we establish a quantitative bound on the global-mode connectivity using mean-field theory or dropout stability. The bound demonstrates that the connectivity improves when reducing data heterogeneity and widening trained models. Numerical results further corroborate our analytical findings.

Title: FedZeN: Towards superlinear zeroth-order federated learning via incremental Hessian estimation. (arXiv:2309.17174v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.17174
• Code URL: null
• Copy Paste: [[2309.17174]] FedZeN: Towards superlinear zeroth-order federated learning via incremental Hessian estimation(http://arxiv.org/abs/2309.17174)
• Summary:

  Federated learning is a distributed learning framework that allows a set of clients to collaboratively train a model under the orchestration of a central server, without sharing raw data samples. Although in many practical scenarios the derivatives of the objective function are not available, only few works have considered the federated zeroth-order setting, in which functions can only be accessed through a budgeted number of point evaluations. In this work we focus on convex optimization and design the first federated zeroth-order algorithm to estimate the curvature of the global objective, with the purpose of achieving superlinear convergence. We take an incremental Hessian estimator whose error norm converges linearly, and we adapt it to the federated zeroth-order setting, sampling the random search directions from the Stiefel manifold for improved performance. In particular, both the gradient and Hessian estimators are built at the central server in a communication-efficient and privacy-preserving way by leveraging synchronized pseudo-random number generators. We provide a theoretical analysis of our algorithm, named FedZeN, proving local quadratic convergence with high probability and global linear convergence up to zeroth-order precision. Numerical simulations confirm the superlinear convergence rate and show that our algorithm outperforms the federated zeroth-order methods available in the literature.

fair

Title: ONNXExplainer: an ONNX Based Generic Framework to Explain Neural Networks Using Shapley Values. (arXiv:2309.16916v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.16916
• Code URL: null
• Copy Paste: [[2309.16916]] ONNXExplainer: an ONNX Based Generic Framework to Explain Neural Networks Using Shapley Values(http://arxiv.org/abs/2309.16916)
• Summary:

  Understanding why a neural network model makes certain decisions can be as important as the inference performance. Various methods have been proposed to help practitioners explain the prediction of a neural network model, of which Shapley values are most popular. SHAP package is a leading implementation of Shapley values to explain neural networks implemented in TensorFlow or PyTorch but lacks cross-platform support, one-shot deployment and is highly inefficient. To address these problems, we present the ONNXExplainer, which is a generic framework to explain neural networks using Shapley values in the ONNX ecosystem. In ONNXExplainer, we develop its own automatic differentiation and optimization approach, which not only enables One-Shot Deployment of neural networks inference and explanations, but also significantly improves the efficiency to compute explanation with less memory consumption. For fair comparison purposes, we also implement the same optimization in TensorFlow and PyTorch and measure its performance against the current state of the art open-source counterpart, SHAP. Extensive benchmarks demonstrate that the proposed optimization approach improves the explanation latency of VGG19, ResNet50, DenseNet201, and EfficientNetB0 by as much as 500%.

Title: G4SATBench: Benchmarking and Advancing SAT Solving with Graph Neural Networks. (arXiv:2309.16941v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.16941
• Code URL: https://github.com/zhaoyu-li/g4satbench
• Copy Paste: [[2309.16941]] G4SATBench: Benchmarking and Advancing SAT Solving with Graph Neural Networks(http://arxiv.org/abs/2309.16941)
• Summary:

  Graph neural networks (GNNs) have recently emerged as a promising approach for solving the Boolean Satisfiability Problem (SAT), offering potential alternatives to traditional backtracking or local search SAT solvers. However, despite the growing volume of literature in this field, there remains a notable absence of a unified dataset and a fair benchmark to evaluate and compare existing approaches. To address this crucial gap, we present G4SATBench, the first benchmark study that establishes a comprehensive evaluation framework for GNN-based SAT solvers. In G4SATBench, we meticulously curate a large and diverse set of SAT datasets comprising 7 problems with 3 difficulty levels and benchmark a broad range of GNN models across various prediction tasks, training objectives, and inference algorithms. To explore the learning abilities and comprehend the strengths and limitations of GNN-based SAT solvers, we also compare their solving processes with the heuristics in search-based SAT solvers. Our empirical results provide valuable insights into the performance of GNN-based SAT solvers and further suggest that existing GNN models can effectively learn a solving strategy akin to greedy local search but struggle to learn backtracking search in the latent space.

Title: Toward Operationalizing Pipeline-aware ML Fairness: A Research Agenda for Developing Practical Guidelines and Tools. (arXiv:2309.17337v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.17337
• Code URL: null
• Copy Paste: [[2309.17337]] Toward Operationalizing Pipeline-aware ML Fairness: A Research Agenda for Developing Practical Guidelines and Tools(http://arxiv.org/abs/2309.17337)
• Summary:

  While algorithmic fairness is a thriving area of research, in practice, mitigating issues of bias often gets reduced to enforcing an arbitrarily chosen fairness metric, either by enforcing fairness constraints during the optimization step, post-processing model outputs, or by manipulating the training data. Recent work has called on the ML community to take a more holistic approach to tackle fairness issues by systematically investigating the many design choices made through the ML pipeline, and identifying interventions that target the issue's root cause, as opposed to its symptoms. While we share the conviction that this pipeline-based approach is the most appropriate for combating algorithmic unfairness on the ground, we believe there are currently very few methods of \emph{operationalizing} this approach in practice. Drawing on our experience as educators and practitioners, we first demonstrate that without clear guidelines and toolkits, even individuals with specialized ML knowledge find it challenging to hypothesize how various design choices influence model behavior. We then consult the fair-ML literature to understand the progress to date toward operationalizing the pipeline-aware approach: we systematically collect and organize the prior work that attempts to detect, measure, and mitigate various sources of unfairness through the ML pipeline. We utilize this extensive categorization of previous contributions to sketch a research agenda for the community. We hope this work serves as the stepping stone toward a more comprehensive set of resources for ML researchers, practitioners, and students interested in exploring, designing, and testing pipeline-oriented approaches to algorithmic fairness.

interpretability

Title: Dynamic Interpretability for Model Comparison via Decision Rules. (arXiv:2309.17095v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.17095
• Code URL: null
• Copy Paste: [[2309.17095]] Dynamic Interpretability for Model Comparison via Decision Rules(http://arxiv.org/abs/2309.17095)
• Summary:

  Explainable AI (XAI) methods have mostly been built to investigate and shed light on single machine learning models and are not designed to capture and explain differences between multiple models effectively. This paper addresses the challenge of understanding and explaining differences between machine learning models, which is crucial for model selection, monitoring and lifecycle management in real-world applications. We propose DeltaXplainer, a model-agnostic method for generating rule-based explanations describing the differences between two binary classifiers. To assess the effectiveness of DeltaXplainer, we conduct experiments on synthetic and real-world datasets, covering various model comparison scenarios involving different types of concept drift.

Title: Age Group Discrimination via Free Handwriting Indicators. (arXiv:2309.17156v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.17156
• Code URL: null
• Copy Paste: [[2309.17156]] Age Group Discrimination via Free Handwriting Indicators(http://arxiv.org/abs/2309.17156)
• Summary:

  The growing global elderly population is expected to increase the prevalence of frailty, posing significant challenges to healthcare systems. Frailty, a syndrome associated with ageing, is characterised by progressive health decline, increased vulnerability to stressors and increased risk of mortality. It represents a significant burden on public health and reduces the quality of life of those affected. The lack of a universally accepted method to assess frailty and a standardised definition highlights a critical research gap. Given this lack and the importance of early prevention, this study presents an innovative approach using an instrumented ink pen to ecologically assess handwriting for age group classification. Content-free handwriting data from 80 healthy participants in different age groups (20-40, 41-60, 61-70 and 70+) were analysed. Fourteen gesture- and tremor-related indicators were computed from the raw data and used in five classification tasks. These tasks included discriminating between adjacent and non-adjacent age groups using Catboost and Logistic Regression classifiers. Results indicate exceptional classifier performance, with accuracy ranging from 82.5% to 97.5%, precision from 81.8% to 100%, recall from 75% to 100% and ROC-AUC from 92.2% to 100%. Model interpretability, facilitated by SHAP analysis, revealed age-dependent sensitivity of temporal and tremor-related handwriting features. Importantly, this classification method offers potential for early detection of abnormal signs of ageing in uncontrolled settings such as remote home monitoring, thereby addressing the critical issue of frailty detection and contributing to improved care for older adults.

explainability

watermark

diffusion

Title: SatDM: Synthesizing Realistic Satellite Image with Semantic Layout Conditioning using Diffusion Models. (arXiv:2309.16812v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.16812
• Code URL: null
• Copy Paste: [[2309.16812]] SatDM: Synthesizing Realistic Satellite Image with Semantic Layout Conditioning using Diffusion Models(http://arxiv.org/abs/2309.16812)
• Summary:

  Deep learning models in the Earth Observation domain heavily rely on the availability of large-scale accurately labeled satellite imagery. However, obtaining and labeling satellite imagery is a resource-intensive endeavor. While generative models offer a promising solution to address data scarcity, their potential remains underexplored. Recently, Denoising Diffusion Probabilistic Models (DDPMs) have demonstrated significant promise in synthesizing realistic images from semantic layouts. In this paper, a conditional DDPM model capable of taking a semantic map and generating high-quality, diverse, and correspondingly accurate satellite images is implemented. Additionally, a comprehensive illustration of the optimization dynamics is provided. The proposed methodology integrates cutting-edge techniques such as variance learning, classifier-free guidance, and improved noise scheduling. The denoising network architecture is further complemented by the incorporation of adaptive normalization and self-attention mechanisms, enhancing the model's capabilities. The effectiveness of our proposed model is validated using a meticulously labeled dataset introduced within the context of this study. Validation encompasses both algorithmic methods such as Frechet Inception Distance (FID) and Intersection over Union (IoU), as well as a human opinion study. Our findings indicate that the generated samples exhibit minimal deviation from real ones, opening doors for practical applications such as data augmentation. We look forward to further explorations of DDPMs in a wider variety of settings and data modalities. An open-source reference implementation of the algorithm and a link to the benchmarked dataset are provided at https://github.com/obaghirli/syn10-diffusion.

Title: Denoising Diffusion Bridge Models. (arXiv:2309.16948v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.16948
• Code URL: https://github.com/alexzhou907/DDBM
• Copy Paste: [[2309.16948]] Denoising Diffusion Bridge Models(http://arxiv.org/abs/2309.16948)
• Summary:

  Diffusion models are powerful generative models that map noise to data using stochastic processes. However, for many applications such as image editing, the model input comes from a distribution that is not random noise. As such, diffusion models must rely on cumbersome methods like guidance or projected sampling to incorporate this information in the generative process. In our work, we propose Denoising Diffusion Bridge Models (DDBMs), a natural alternative to this paradigm based on diffusion bridges, a family of processes that interpolate between two paired distributions given as endpoints. Our method learns the score of the diffusion bridge from data and maps from one endpoint distribution to the other by solving a (stochastic) differential equation based on the learned score. Our method naturally unifies several classes of generative models, such as score-based diffusion models and OT-Flow-Matching, allowing us to adapt existing design and architectural choices to our more general problem. Empirically, we apply DDBMs to challenging image datasets in both pixel and latent space. On standard image translation problems, DDBMs achieve significant improvement over baseline methods, and, when we reduce the problem to image generation by setting the source distribution to random noise, DDBMs achieve comparable FID scores to state-of-the-art methods despite being built for a more general task.

Title: DeeDiff: Dynamic Uncertainty-Aware Early Exiting for Accelerating Diffusion Model Generation. (arXiv:2309.17074v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.17074
• Code URL: null
• Copy Paste: [[2309.17074]] DeeDiff: Dynamic Uncertainty-Aware Early Exiting for Accelerating Diffusion Model Generation(http://arxiv.org/abs/2309.17074)
• Summary:

  Diffusion models achieve great success in generating diverse and high-fidelity images. The performance improvements come with low generation speed per image, which hinders the application diffusion models in real-time scenarios. While some certain predictions benefit from the full computation of the model in each sample iteration, not every iteration requires the same amount of computation, potentially leading to computation waste. In this work, we propose DeeDiff, an early exiting framework that adaptively allocates computation resources in each sampling step to improve the generation efficiency of diffusion models. Specifically, we introduce a timestep-aware uncertainty estimation module (UEM) for diffusion models which is attached to each intermediate layer to estimate the prediction uncertainty of each layer. The uncertainty is regarded as the signal to decide if the inference terminates. Moreover, we propose uncertainty-aware layer-wise loss to fill the performance gap between full models and early-exited models. With such loss strategy, our model is able to obtain comparable results as full-layer models. Extensive experiments of class-conditional, unconditional, and text-guided generation on several datasets show that our method achieves state-of-the-art performance and efficiency trade-off compared with existing early exiting methods on diffusion models. More importantly, our method even brings extra benefits to baseline models and obtains better performance on CIFAR-10 and Celeb-A datasets. Full code and model are released for reproduction.

Title: Consistent123: One Image to Highly Consistent 3D Asset Using Case-Aware Diffusion Priors. (arXiv:2309.17261v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.17261
• Code URL: null
• Copy Paste: [[2309.17261]] Consistent123: One Image to Highly Consistent 3D Asset Using Case-Aware Diffusion Priors(http://arxiv.org/abs/2309.17261)
• Summary:

  Reconstructing 3D objects from a single image guided by pretrained diffusion models has demonstrated promising outcomes. However, due to utilizing the case-agnostic rigid strategy, their generalization ability to arbitrary cases and the 3D consistency of reconstruction are still poor. In this work, we propose Consistent123, a case-aware two-stage method for highly consistent 3D asset reconstruction from one image with both 2D and 3D diffusion priors. In the first stage, Consistent123 utilizes only 3D structural priors for sufficient geometry exploitation, with a CLIP-based case-aware adaptive detection mechanism embedded within this process. In the second stage, 2D texture priors are introduced and progressively take on a dominant guiding role, delicately sculpting the details of the 3D model. Consistent123 aligns more closely with the evolving trends in guidance requirements, adaptively providing adequate 3D geometric initialization and suitable 2D texture refinement for different objects. Consistent123 can obtain highly 3D-consistent reconstruction and exhibits strong generalization ability across various objects. Qualitative and quantitative experiments show that our method significantly outperforms state-of-the-art image-to-3D methods. See https://Consistent123.github.io for a more comprehensive exploration of our generated 3D assets.

Title: Memory in Plain Sight: A Survey of the Uncanny Resemblances between Diffusion Models and Associative Memories. (arXiv:2309.16750v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.16750
• Code URL: null
• Copy Paste: [[2309.16750]] Memory in Plain Sight: A Survey of the Uncanny Resemblances between Diffusion Models and Associative Memories(http://arxiv.org/abs/2309.16750)
• Summary:

  Diffusion Models (DMs) have recently set state-of-the-art on many generation benchmarks. However, there are myriad ways to describe them mathematically, which makes it difficult to develop a simple understanding of how they work. In this survey, we provide a concise overview of DMs from the perspective of dynamical systems and Ordinary Differential Equations (ODEs) which exposes a mathematical connection to the highly related yet often overlooked class of energy-based models, called Associative Memories (AMs). Energy-based AMs are a theoretical framework that behave much like denoising DMs, but they enable us to directly compute a Lyapunov energy function on which we can perform gradient descent to denoise data. We then summarize the 40 year history of energy-based AMs, beginning with the original Hopfield Network, and discuss new research directions for AMs and DMs that are revealed by characterizing the extent of their similarities and differences

Title: Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learning. (arXiv:2309.16984v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.16984
• Code URL: null
• Copy Paste: [[2309.16984]] Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learning(http://arxiv.org/abs/2309.16984)
• Summary:

  Score-based generative models like the diffusion model have been testified to be effective in modeling multi-modal data from image generation to reinforcement learning (RL). However, the inference process of diffusion model can be slow, which hinders its usage in RL with iterative sampling. We propose to apply the consistency model as an efficient yet expressive policy representation, namely consistency policy, with an actor-critic style algorithm for three typical RL settings: offline, offline-to-online and online. For offline RL, we demonstrate the expressiveness of generative models as policies from multi-modal data. For offline-to-online RL, the consistency policy is shown to be more computational efficient than diffusion policy, with a comparable performance. For online RL, the consistency policy demonstrates significant speedup and even higher average performances than the diffusion policy.

Title: Sheaf Hypergraph Networks. (arXiv:2309.17116v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.17116
• Code URL: null
• Copy Paste: [[2309.17116]] Sheaf Hypergraph Networks(http://arxiv.org/abs/2309.17116)
• Summary:

  Higher-order relations are widespread in nature, with numerous phenomena involving complex interactions that extend beyond simple pairwise connections. As a result, advancements in higher-order processing can accelerate the growth of various fields requiring structured data. Current approaches typically represent these interactions using hypergraphs. We enhance this representation by introducing cellular sheaves for hypergraphs, a mathematical construction that adds extra structure to the conventional hypergraph while maintaining their local, higherorder connectivity. Drawing inspiration from existing Laplacians in the literature, we develop two unique formulations of sheaf hypergraph Laplacians: linear and non-linear. Our theoretical analysis demonstrates that incorporating sheaves into the hypergraph Laplacian provides a more expressive inductive bias than standard hypergraph diffusion, creating a powerful instrument for effectively modelling complex data structures. We employ these sheaf hypergraph Laplacians to design two categories of models: Sheaf Hypergraph Neural Networks and Sheaf Hypergraph Convolutional Networks. These models generalize classical Hypergraph Networks often found in the literature. Through extensive experimentation, we show that this generalization significantly improves performance, achieving top results on multiple benchmark datasets for hypergraph node classification.

Title: ResBit: Residual Bit Vector for Categorical Values. (arXiv:2309.17196v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.17196
• Code URL: null
• Copy Paste: [[2309.17196]] ResBit: Residual Bit Vector for Categorical Values(http://arxiv.org/abs/2309.17196)
• Summary:

  The one-hot vector has long been widely used in machine learning as a simple and generic method for representing discrete data. However, this method increases the number of dimensions linearly with the categorical data to be represented, which is problematic from the viewpoint of spatial computational complexity in deep learning, which requires a large amount of data. Recently, Analog Bits, a method for representing discrete data as a sequence of bits, was proposed on the basis of the high expressiveness of diffusion models. However, since the number of category types to be represented in a generation task is not necessarily at a power of two, there is a discrepancy between the range that Analog Bits can represent and the range represented as category data. If such a value is generated, the problem is that the original category value cannot be restored. To address this issue, we propose Residual Bit Vector (ResBit), which is a hierarchical bit representation. Although it is a general-purpose representation method, in this paper, we treat it as numerical data and show that it can be used as an extension of Analog Bits using Table Residual Bit Diffusion (TRBD), which is incorporated into TabDDPM, a tabular data generation method. We experimentally confirmed that TRBD can generate diverse and high-quality data from small-scale table data to table data containing diverse category values faster than TabDDPM. Furthermore, we show that ResBit can also serve as an alternative to the one-hot vector by utilizing ResBit for conditioning in GANs and as a label expression in image classification.

Title: Navigating the Design Space of Equivariant Diffusion-Based Generative Models for De Novo 3D Molecule Generation. (arXiv:2309.17296v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.17296
• Code URL: null
• Copy Paste: [[2309.17296]] Navigating the Design Space of Equivariant Diffusion-Based Generative Models for De Novo 3D Molecule Generation(http://arxiv.org/abs/2309.17296)
• Summary:

  Deep generative diffusion models are a promising avenue for de novo 3D molecular design in material science and drug discovery. However, their utility is still constrained by suboptimal performance with large molecular structures and limited training data. Addressing this gap, we explore the design space of E(3) equivariant diffusion models, focusing on previously blank spots. Our extensive comparative analysis evaluates the interplay between continuous and discrete state spaces. Out of this investigation, we introduce the EQGAT-diff model, which consistently surpasses the performance of established models on the QM9 and GEOM-Drugs datasets by a large margin. Distinctively, EQGAT-diff takes continuous atomic positions while chemical elements and bond types are categorical and employ a time-dependent loss weighting that significantly increases training convergence and the quality of generated samples. To further strengthen the applicability of diffusion models to limited training data, we examine the transferability of EQGAT-diff trained on the large PubChem3D dataset with implicit hydrogens to target distributions with explicit hydrogens. Fine-tuning EQGAT-diff for a couple of iterations further pushes state-of-the-art performance across datasets. We envision that our findings will find applications in structure-based drug design, where the accuracy of generative models for small datasets of complex molecules is critical.

noise learning

data-free

Title: Instant Complexity Reduction in CNNs using Locality-Sensitive Hashing. (arXiv:2309.17211v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.17211
• Code URL: null
• Copy Paste: [[2309.17211]] Instant Complexity Reduction in CNNs using Locality-Sensitive Hashing(http://arxiv.org/abs/2309.17211)
• Summary:

  To reduce the computational cost of convolutional neural networks (CNNs) for usage on resource-constrained devices, structured pruning approaches have shown promising results, drastically reducing floating-point operations (FLOPs) without substantial drops in accuracy. However, most recent methods require fine-tuning or specific training procedures to achieve a reasonable trade-off between retained accuracy and reduction in FLOPs. This introduces additional cost in the form of computational overhead and requires training data to be available. To this end, we propose HASTE (Hashing for Tractable Efficiency), a parameter-free and data-free module that acts as a plug-and-play replacement for any regular convolution module. It instantly reduces the network's test-time inference cost without requiring any training or fine-tuning. We are able to drastically compress latent feature maps without sacrificing much accuracy by using locality-sensitive hashing (LSH) to detect redundancies in the channel dimension. Similar channels are aggregated to reduce the input and filter depth simultaneously, allowing for cheaper convolutions. We demonstrate our approach on the popular vision benchmarks CIFAR-10 and ImageNet. In particular, we are able to instantly drop 46.72% of FLOPs while only losing 1.25% accuracy by just swapping the convolution modules in a ResNet34 on CIFAR-10 for our HASTE module.

transformer

Title: Sketch2CADScript: 3D Scene Reconstruction from 2D Sketch using Visual Transformer and Rhino Grasshopper. (arXiv:2309.16850v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.16850
• Code URL: null
• Copy Paste: [[2309.16850]] Sketch2CADScript: 3D Scene Reconstruction from 2D Sketch using Visual Transformer and Rhino Grasshopper(http://arxiv.org/abs/2309.16850)
• Summary:

  Existing 3D model reconstruction methods typically produce outputs in the form of voxels, point clouds, or meshes. However, each of these approaches has its limitations and may not be suitable for every scenario. For instance, the resulting model may exhibit a rough surface and distorted structure, making manual editing and post-processing challenging for humans. In this paper, we introduce a novel 3D reconstruction method designed to address these issues. We trained a visual transformer to predict a "scene descriptor" from a single wire-frame image. This descriptor encompasses crucial information, including object types and parameters such as position, rotation, and size. With the predicted parameters, a 3D scene can be reconstructed using 3D modeling software like Blender or Rhino Grasshopper which provides a programmable interface, resulting in finely and easily editable 3D models. To evaluate the proposed model, we created two datasets: one featuring simple scenes and another with complex scenes. The test results demonstrate the model's ability to accurately reconstruct simple scenes but reveal its challenges with more complex ones.

Title: Superpixel Transformers for Efficient Semantic Segmentation. (arXiv:2309.16889v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.16889
• Code URL: null
• Copy Paste: [[2309.16889]] Superpixel Transformers for Efficient Semantic Segmentation(http://arxiv.org/abs/2309.16889)
• Summary:

  Semantic segmentation, which aims to classify every pixel in an image, is a key task in machine perception, with many applications across robotics and autonomous driving. Due to the high dimensionality of this task, most existing approaches use local operations, such as convolutions, to generate per-pixel features. However, these methods are typically unable to effectively leverage global context information due to the high computational costs of operating on a dense image. In this work, we propose a solution to this issue by leveraging the idea of superpixels, an over-segmentation of the image, and applying them with a modern transformer framework. In particular, our model learns to decompose the pixel space into a spatially low dimensional superpixel space via a series of local cross-attentions. We then apply multi-head self-attention to the superpixels to enrich the superpixel features with global context and then directly produce a class prediction for each superpixel. Finally, we directly project the superpixel class predictions back into the pixel space using the associations between the superpixels and the image pixel features. Reasoning in the superpixel space allows our method to be substantially more computationally efficient compared to convolution-based decoder methods. Yet, our method achieves state-of-the-art performance in semantic segmentation due to the rich superpixel features generated by the global self-attention mechanism. Our experiments on Cityscapes and ADE20K demonstrate that our method matches the state of the art in terms of accuracy, while outperforming in terms of model parameters and latency.

Title: Synthetic Data Generation and Deep Learning for the Topological Analysis of 3D Data. (arXiv:2309.16968v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.16968
• Code URL: null
• Copy Paste: [[2309.16968]] Synthetic Data Generation and Deep Learning for the Topological Analysis of 3D Data(http://arxiv.org/abs/2309.16968)
• Summary:

  This research uses deep learning to estimate the topology of manifolds represented by sparse, unordered point cloud scenes in 3D. A new labelled dataset was synthesised to train neural networks and evaluate their ability to estimate the genus of these manifolds. This data used random homeomorphic deformations to provoke the learning of visual topological features. We demonstrate that deep learning models could extract these features and discuss some advantages over existing topological data analysis tools that are based on persistent homology. Semantic segmentation was used to provide additional geometric information in conjunction with topological labels. Common point cloud multi-layer perceptron and transformer networks were both used to compare the viability of these methods. The experimental results of this pilot study support the hypothesis that, with the aid of sophisticated synthetic data generation, neural networks can perform segmentation-based topological data analysis. While our study focused on simulated data, the accuracy achieved suggests a potential for future applications using real data.

Title: GSDC Transformer: An Efficient and Effective Cue Fusion for Monocular Multi-Frame Depth Estimation. (arXiv:2309.17059v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.17059
• Code URL: null
• Copy Paste: [[2309.17059]] GSDC Transformer: An Efficient and Effective Cue Fusion for Monocular Multi-Frame Depth Estimation(http://arxiv.org/abs/2309.17059)
• Summary:

  Depth estimation provides an alternative approach for perceiving 3D information in autonomous driving. Monocular depth estimation, whether with single-frame or multi-frame inputs, has achieved significant success by learning various types of cues and specializing in either static or dynamic scenes. Recently, these cues fusion becomes an attractive topic, aiming to enable the combined cues to perform well in both types of scenes. However, adaptive cue fusion relies on attention mechanisms, where the quadratic complexity limits the granularity of cue representation. Additionally, explicit cue fusion depends on precise segmentation, which imposes a heavy burden on mask prediction. To address these issues, we propose the GSDC Transformer, an efficient and effective component for cue fusion in monocular multi-frame depth estimation. We utilize deformable attention to learn cue relationships at a fine scale, while sparse attention reduces computational requirements when granularity increases. To compensate for the precision drop in dynamic scenes, we represent scene attributes in the form of super tokens without relying on precise shapes. Within each super token attributed to dynamic scenes, we gather its relevant cues and learn local dense relationships to enhance cue fusion. Our method achieves state-of-the-art performance on the KITTI dataset with efficient fusion speed.

Title: When Epipolar Constraint Meets Non-local Operators in Multi-View Stereo. (arXiv:2309.17218v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.17218
• Code URL: https://github.com/tqtqliu/et-mvsnet
• Copy Paste: [[2309.17218]] When Epipolar Constraint Meets Non-local Operators in Multi-View Stereo(http://arxiv.org/abs/2309.17218)
• Summary:

  Learning-based multi-view stereo (MVS) method heavily relies on feature matching, which requires distinctive and descriptive representations. An effective solution is to apply non-local feature aggregation, e.g., Transformer. Albeit useful, these techniques introduce heavy computation overheads for MVS. Each pixel densely attends to the whole image. In contrast, we propose to constrain non-local feature augmentation within a pair of lines: each point only attends the corresponding pair of epipolar lines. Our idea takes inspiration from the classic epipolar geometry, which shows that one point with different depth hypotheses will be projected to the epipolar line on the other view. This constraint reduces the 2D search space into the epipolar line in stereo matching. Similarly, this suggests that the matching of MVS is to distinguish a series of points lying on the same line. Inspired by this point-to-line search, we devise a line-to-point non-local augmentation strategy. We first devise an optimized searching algorithm to split the 2D feature maps into epipolar line pairs. Then, an Epipolar Transformer (ET) performs non-local feature augmentation among epipolar line pairs. We incorporate the ET into a learning-based MVS baseline, named ET-MVSNet. ET-MVSNet achieves state-of-the-art reconstruction performance on both the DTU and Tanks-and-Temples benchmark with high efficiency. Code is available at https://github.com/TQTQliu/ET-MVSNet.

Title: An evaluation of GPT models for phenotype concept recognition. (arXiv:2309.17169v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.17169
• Code URL: null
• Copy Paste: [[2309.17169]] An evaluation of GPT models for phenotype concept recognition(http://arxiv.org/abs/2309.17169)
• Summary:

  Objective: Clinical deep phenotyping plays a critical role in both the diagnosis of patients with rare disorders as well as in building care coordination plans. The process relies on modelling and curating patient profiles using ontology concepts, usually from the Human Phenotype Ontology. Machine learning methods have been widely adopted to support this phenotype concept recognition task. With the significant shift in the use of large language models (LLMs) for most NLP tasks, herewithin, we examine the performance of the latest Generative Pre-trained Transformer (GPT) models underpinning ChatGPT in clinical deep phenotyping. Materials and Methods: The experimental setup of the study included seven prompts of various levels of specificity, two GPT models (gpt-3.5 and gpt-4.0) and an established gold standard for phenotype recognition. Results: Our results show that, currently, these models have not yet achieved state of the art performance. The best run, using few-shots learning, achieved 0.41 F1 score, compared to a 0.62 F1 score achieved by the current best in class tool. Conclusion: The non-deterministic nature of the outcomes and the lack of concordance between different runs using the same prompt and input makes the use of these LLMs in clinical settings problematic.

Title: PROSE: Predicting Operators and Symbolic Expressions using Multimodal Transformers. (arXiv:2309.16816v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.16816
• Code URL: null
• Copy Paste: [[2309.16816]] PROSE: Predicting Operators and Symbolic Expressions using Multimodal Transformers(http://arxiv.org/abs/2309.16816)
• Summary:

  Approximating nonlinear differential equations using a neural network provides a robust and efficient tool for various scientific computing tasks, including real-time predictions, inverse problems, optimal controls, and surrogate modeling. Previous works have focused on embedding dynamical systems into networks through two approaches: learning a single solution operator (i.e., the mapping from input parametrized functions to solutions) or learning the governing system of equations (i.e., the constitutive model relative to the state variables). Both of these approaches yield different representations for the same underlying data or function. Additionally, observing that families of differential equations often share key characteristics, we seek one network representation across a wide range of equations. Our method, called Predicting Operators and Symbolic Expressions (PROSE), learns maps from multimodal inputs to multimodal outputs, capable of generating both numerical predictions and mathematical equations. By using a transformer structure and a feature fusion approach, our network can simultaneously embed sets of solution operators for various parametric differential equations using a single trained network. Detailed experiments demonstrate that the network benefits from its multimodal nature, resulting in improved prediction accuracy and better generalization. The network is shown to be able to handle noise in the data and errors in the symbolic representation, including noisy numerical values, model misspecification, and erroneous addition or deletion of terms. PROSE provides a new neural network framework for differential equations which allows for more flexibility and generality in learning operators and governing equations from data.

Title: Sourcing Investment Targets for Venture and Growth Capital Using Multivariate Time Series Transformer. (arXiv:2309.16888v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.16888
• Code URL: null
• Copy Paste: [[2309.16888]] Sourcing Investment Targets for Venture and Growth Capital Using Multivariate Time Series Transformer(http://arxiv.org/abs/2309.16888)
• Summary:

  This paper addresses the growing application of data-driven approaches within the Private Equity (PE) industry, particularly in sourcing investment targets (i.e., companies) for Venture Capital (VC) and Growth Capital (GC). We present a comprehensive review of the relevant approaches and propose a novel approach leveraging a Transformer-based Multivariate Time Series Classifier (TMTSC) for predicting the success likelihood of any candidate company. The objective of our research is to optimize sourcing performance for VC and GC investments by formally defining the sourcing problem as a multivariate time series classification task. We consecutively introduce the key components of our implementation which collectively contribute to the successful application of TMTSC in VC/GC sourcing: input features, model architecture, optimization target, and investor-centric data augmentation and split. Our extensive experiments on four datasets, benchmarked towards three popular baselines, demonstrate the effectiveness of our approach in improving decision making within the VC and GC industry.

Title: TranDRL: A Transformer-Driven Deep Reinforcement Learning Enabled Prescriptive Maintenance Framework. (arXiv:2309.16935v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.16935
• Code URL: null
• Copy Paste: [[2309.16935]] TranDRL: A Transformer-Driven Deep Reinforcement Learning Enabled Prescriptive Maintenance Framework(http://arxiv.org/abs/2309.16935)
• Summary:

  Industrial systems demand reliable predictive maintenance strategies to enhance operational efficiency and reduce downtime. This paper introduces a novel, integrated framework that leverages the power of transformer neural networks and deep reinforcement learning (DRL) algorithms to optimize maintenance actions. Our approach employs the transformer model to effectively capture complex temporal patterns in sensor data, thereby accurately predicting the Remaining Useful Life (RUL) of equipment. Simultaneously, the DRL component of our framework provides cost-effective and timely maintenance recommendations. We validate the efficacy of our framework on the NASA C-MPASS dataset, where it demonstrates significant advancements in both RUL prediction accuracy and the optimization of maintenance actions. Consequently, our pioneering approach provides an innovative data-driven methodology for prescriptive maintenance, addressing key challenges in industrial operations and leading the way to more efficient, cost-effective, and reliable systems.

Title: Memory Gym: Partially Observable Challenges to Memory-Based Agents in Endless Episodes. (arXiv:2309.17207v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.17207
• Code URL: https://github.com/marcometer/endless-memory-gym
• Copy Paste: [[2309.17207]] Memory Gym: Partially Observable Challenges to Memory-Based Agents in Endless Episodes(http://arxiv.org/abs/2309.17207)
• Summary:

  Memory Gym introduces a unique benchmark designed to test Deep Reinforcement Learning agents, specifically comparing Gated Recurrent Unit (GRU) against Transformer-XL (TrXL), on their ability to memorize long sequences, withstand noise, and generalize. It features partially observable 2D environments with discrete controls, namely Mortar Mayhem, Mystery Path, and Searing Spotlights. These originally finite environments are extrapolated to novel endless tasks that act as an automatic curriculum, drawing inspiration from the car game ``I packed my bag". These endless tasks are not only beneficial for evaluating efficiency but also intriguingly valuable for assessing the effectiveness of approaches in memory-based agents. Given the scarcity of publicly available memory baselines, we contribute an implementation driven by TrXL and Proximal Policy Optimization. This implementation leverages TrXL as episodic memory using a sliding window approach. In our experiments on the finite environments, TrXL demonstrates superior sample efficiency in Mystery Path and outperforms in Mortar Mayhem. However, GRU is more efficient on Searing Spotlights. Most notably, in all endless tasks, GRU makes a remarkable resurgence, consistently outperforming TrXL by significant margins.

Title: Scaling Experiments in Self-Supervised Cross-Table Representation Learning. (arXiv:2309.17339v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.17339
• Code URL: null
• Copy Paste: [[2309.17339]] Scaling Experiments in Self-Supervised Cross-Table Representation Learning(http://arxiv.org/abs/2309.17339)
• Summary:

  To analyze the scaling potential of deep tabular representation learning models, we introduce a novel Transformer-based architecture specifically tailored to tabular data and cross-table representation learning by utilizing table-specific tokenizers and a shared Transformer backbone. Our training approach encompasses both single-table and cross-table models, trained via missing value imputation through a self-supervised masked cell recovery objective. To understand the scaling behavior of our method, we train models of varying sizes, ranging from approximately $10^4$ to $10^7$ parameters. These models are trained on a carefully curated pretraining dataset, consisting of 135M training tokens sourced from 76 diverse datasets. We assess the scaling of our architecture in both single-table and cross-table pretraining setups by evaluating the pretrained models using linear probing on a curated set of benchmark datasets and comparing the results with conventional baselines.

Title: Module-wise Training of Neural Networks via the Minimizing Movement Scheme. (arXiv:2309.17357v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.17357
• Code URL: null
• Copy Paste: [[2309.17357]] Module-wise Training of Neural Networks via the Minimizing Movement Scheme(http://arxiv.org/abs/2309.17357)
• Summary:

  Greedy layer-wise or module-wise training of neural networks is compelling in constrained and on-device settings where memory is limited, as it circumvents a number of problems of end-to-end back-propagation. However, it suffers from a stagnation problem, whereby early layers overfit and deeper layers stop increasing the test accuracy after a certain depth. We propose to solve this issue by introducing a module-wise regularization inspired by the minimizing movement scheme for gradient flows in distribution space. We call the method TRGL for Transport Regularized Greedy Learning and study it theoretically, proving that it leads to greedy modules that are regular and that progressively solve the task. Experimentally, we show improved accuracy of module-wise training of various architectures such as ResNets, Transformers and VGG, when our regularization is added, superior to that of other module-wise training methods and often to end-to-end training, with as much as 60% less memory usage.

generative

Title: Intriguing properties of generative classifiers. (arXiv:2309.16779v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.16779
• Code URL: null
• Copy Paste: [[2309.16779]] Intriguing properties of generative classifiers(http://arxiv.org/abs/2309.16779)
• Summary:

  What is the best paradigm to recognize objects -- discriminative inference (fast but potentially prone to shortcut learning) or using a generative model (slow but potentially more robust)? We build on recent advances in generative modeling that turn text-to-image models into classifiers. This allows us to study their behavior and to compare them against discriminative models and human psychophysical data. We report four intriguing emergent properties of generative classifiers: they show a record-breaking human-like shape bias (99% for Imagen), near human-level out-of-distribution accuracy, state-of-the-art alignment with human classification errors, and they understand certain perceptual illusions. Our results indicate that while the current dominant paradigm for modeling human object recognition is discriminative inference, zero-shot generative models approximate human object recognition data surprisingly well.

Title: Scalable Multi-Temporal Remote Sensing Change Data Generation via Simulating Stochastic Change Process. (arXiv:2309.17031v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.17031
• Code URL: null
• Copy Paste: [[2309.17031]] Scalable Multi-Temporal Remote Sensing Change Data Generation via Simulating Stochastic Change Process(http://arxiv.org/abs/2309.17031)
• Summary:

  Understanding the temporal dynamics of Earth's surface is a mission of multi-temporal remote sensing image analysis, significantly promoted by deep vision models with its fuel -- labeled multi-temporal images. However, collecting, preprocessing, and annotating multi-temporal remote sensing images at scale is non-trivial since it is expensive and knowledge-intensive. In this paper, we present a scalable multi-temporal remote sensing change data generator via generative modeling, which is cheap and automatic, alleviating these problems. Our main idea is to simulate a stochastic change process over time. We consider the stochastic change process as a probabilistic semantic state transition, namely generative probabilistic change model (GPCM), which decouples the complex simulation problem into two more trackable sub-problems, \ie, change event simulation and semantic change synthesis. To solve these two problems, we present the change generator (Changen), a GAN-based GPCM, enabling controllable object change data generation, including customizable object property, and change event. The extensive experiments suggest that our Changen has superior generation capability, and the change detectors with Changen pre-training exhibit excellent transferability to real-world change datasets.

Title: GAIA-1: A Generative World Model for Autonomous Driving. (arXiv:2309.17080v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.17080
• Code URL: null
• Copy Paste: [[2309.17080]] GAIA-1: A Generative World Model for Autonomous Driving(http://arxiv.org/abs/2309.17080)
• Summary:

  Autonomous driving promises transformative improvements to transportation, but building systems capable of safely navigating the unstructured complexity of real-world scenarios remains challenging. A critical problem lies in effectively predicting the various potential outcomes that may emerge in response to the vehicle's actions as the world evolves.

  To address this challenge, we introduce GAIA-1 ('Generative AI for Autonomy'), a generative world model that leverages video, text, and action inputs to generate realistic driving scenarios while offering fine-grained control over ego-vehicle behavior and scene features. Our approach casts world modeling as an unsupervised sequence modeling problem by mapping the inputs to discrete tokens, and predicting the next token in the sequence. Emerging properties from our model include learning high-level structures and scene dynamics, contextual awareness, generalization, and understanding of geometry. The power of GAIA-1's learned representation that captures expectations of future events, combined with its ability to generate realistic samples, provides new possibilities for innovation in the field of autonomy, enabling enhanced and accelerated training of autonomous driving technology.

Title: Reconstruction of Patient-Specific Confounders in AI-based Radiologic Image Interpretation using Generative Pretraining. (arXiv:2309.17123v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.17123
• Code URL: https://github.com/peterhan91/diffchest
• Copy Paste: [[2309.17123]] Reconstruction of Patient-Specific Confounders in AI-based Radiologic Image Interpretation using Generative Pretraining(http://arxiv.org/abs/2309.17123)
• Summary:

  Detecting misleading patterns in automated diagnostic assistance systems, such as those powered by Artificial Intelligence, is critical to ensuring their reliability, particularly in healthcare. Current techniques for evaluating deep learning models cannot visualize confounding factors at a diagnostic level. Here, we propose a self-conditioned diffusion model termed DiffChest and train it on a dataset of 515,704 chest radiographs from 194,956 patients from multiple healthcare centers in the United States and Europe. DiffChest explains classifications on a patient-specific level and visualizes the confounding factors that may mislead the model. We found high inter-reader agreement when evaluating DiffChest's capability to identify treatment-related confounders, with Fleiss' Kappa values of 0.8 or higher across most imaging findings. Confounders were accurately captured with 11.1% to 100% prevalence rates. Furthermore, our pretraining process optimized the model to capture the most relevant information from the input radiographs. DiffChest achieved excellent diagnostic accuracy when diagnosing 11 chest conditions, such as pleural effusion and cardiac insufficiency, and at least sufficient diagnostic accuracy for the remaining conditions. Our findings highlight the potential of pretraining based on diffusion models in medical image classification, specifically in providing insights into confounding factors and model robustness.

Title: TextField3D: Towards Enhancing Open-Vocabulary 3D Generation with Noisy Text Fields. (arXiv:2309.17175v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.17175
• Code URL: null
• Copy Paste: [[2309.17175]] TextField3D: Towards Enhancing Open-Vocabulary 3D Generation with Noisy Text Fields(http://arxiv.org/abs/2309.17175)
• Summary:

  Recent works learn 3D representation explicitly under text-3D guidance. However, limited text-3D data restricts the vocabulary scale and text control of generations. Generators may easily fall into a stereotype concept for certain text prompts, thus losing open-vocabulary generation ability. To tackle this issue, we introduce a conditional 3D generative model, namely TextField3D. Specifically, rather than using the text prompts as input directly, we suggest to inject dynamic noise into the latent space of given text prompts, i.e., Noisy Text Fields (NTFs). In this way, limited 3D data can be mapped to the appropriate range of textual latent space that is expanded by NTFs. To this end, an NTFGen module is proposed to model general text latent code in noisy fields. Meanwhile, an NTFBind module is proposed to align view-invariant image latent code to noisy fields, further supporting image-conditional 3D generation. To guide the conditional generation in both geometry and texture, multi-modal discrimination is constructed with a text-3D discriminator and a text-2.5D discriminator. Compared to previous methods, TextField3D includes three merits: 1) large vocabulary, 2) text consistency, and 3) low latency. Extensive experiments demonstrate that our method achieves a potential open-vocabulary 3D generation capability.

Title: ACGAN-GNNExplainer: Auxiliary Conditional Generative Explainer for Graph Neural Networks. (arXiv:2309.16918v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.16918
• Code URL: null
• Copy Paste: [[2309.16918]] ACGAN-GNNExplainer: Auxiliary Conditional Generative Explainer for Graph Neural Networks(http://arxiv.org/abs/2309.16918)
• Summary:

  Graph neural networks (GNNs) have proven their efficacy in a variety of real-world applications, but their underlying mechanisms remain a mystery. To address this challenge and enable reliable decision-making, many GNN explainers have been proposed in recent years. However, these methods often encounter limitations, including their dependence on specific instances, lack of generalizability to unseen graphs, producing potentially invalid explanations, and yielding inadequate fidelity. To overcome these limitations, we, in this paper, introduce the Auxiliary Classifier Generative Adversarial Network (ACGAN) into the field of GNN explanation and propose a new GNN explainer dubbed~\emph{ACGAN-GNNExplainer}. Our approach leverages a generator to produce explanations for the original input graphs while incorporating a discriminator to oversee the generation process, ensuring explanation fidelity and improving accuracy. Experimental evaluations conducted on both synthetic and real-world graph datasets demonstrate the superiority of our proposed method compared to other existing GNN explainers.

large language model

Title: Decoding Imagery: Unleashing Large Language Models. (arXiv:2309.16705v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.16705
• Code URL: null
• Copy Paste: [[2309.16705]] Decoding Imagery: Unleashing Large Language Models(http://arxiv.org/abs/2309.16705)
• Summary:

  In a challenge-response study, we subjected Google Bard to 64 visual challenges designed to probe multimodal Large Language Models (LLMs). The challenges spanned diverse categories, including "Visual Situational Reasoning," "Visual Text Reasoning," and "Next Scene Prediction," among others, to discern Bard's competence in melding visual and linguistic analyses. Our findings indicate that Bard tends to rely on making educated guesses about visuals, especially when determining cues from images. Unlike other models like GPT4, Bard does not appear to rely on optical character recognition libraries like Tesseract but recognizes text in complex images like deep learning models such as Google Lens and Visual API. Significantly Bard can solve CAPTCHAs visually that ChatGPT fails to understand, recommending Tesseract solutions. Moreover, while the Bard model proposes solutions based on visual input, it cannot recreate or modify the original visual objects to support its conclusions. Bard fails to redraw ASCII art that the text can describe or capture a simple Tic Tac Toe grid it claims to analyze for the next moves. This study provides experimental insights into the current capacities and areas for improvement in multimodal LLMs.

Title: Guiding Instruction-based Image Editing via Multimodal Large Language Models. (arXiv:2309.17102v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.17102
• Code URL: https://github.com/tsujuifu/pytorch_mgie
• Copy Paste: [[2309.17102]] Guiding Instruction-based Image Editing via Multimodal Large Language Models(http://arxiv.org/abs/2309.17102)
• Summary:

  Instruction-based image editing improves the controllability and flexibility of image manipulation via natural commands without elaborate descriptions or regional masks. However, human instructions are sometimes too brief for current methods to capture and follow. Multimodal large language models (MLLMs) show promising capabilities in cross-modal understanding and visual-aware response generation via LMs. We investigate how MLLMs facilitate edit instructions and present MLLM-Guided Image Editing (MGIE). MGIE learns to derive expressive instructions and provides explicit guidance. The editing model jointly captures this visual imagination and performs manipulation through end-to-end training. We evaluate various aspects of Photoshop-style modification, global photo optimization, and local editing. Extensive experimental results demonstrate that expressive instructions are crucial to instruction-based image editing, and our MGIE can lead to a notable improvement in automatic metrics and human evaluation while maintaining competitive inference efficiency.

Title: Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution. (arXiv:2309.16797v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.16797
• Code URL: null
• Copy Paste: [[2309.16797]] Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution(http://arxiv.org/abs/2309.16797)
• Summary:

  Popular prompt strategies like Chain-of-Thought Prompting can dramatically improve the reasoning abilities of Large Language Models (LLMs) in various domains. However, such hand-crafted prompt-strategies are often sub-optimal. In this paper, we present Promptbreeder, a general-purpose self-referential self-improvement mechanism that evolves and adapts prompts for a given domain. Driven by an LLM, Promptbreeder mutates a population of task-prompts, and subsequently evaluates them for fitness on a training set. Crucially, the mutation of these task-prompts is governed by mutation-prompts that the LLM generates and improves throughout evolution in a self-referential way. That is, Promptbreeder is not just improving task-prompts, but it is also improving the mutationprompts that improve these task-prompts. Promptbreeder outperforms state-of-the-art prompt strategies such as Chain-of-Thought and Plan-and-Solve Prompting on commonly used arithmetic and commonsense reasoning benchmarks. Furthermore, Promptbreeder is able to evolve intricate task-prompts for the challenging problem of hate speech classification.

Title: I Wish to Have an Argument: Argumentative Reasoning in Large Language Models. (arXiv:2309.16938v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.16938
• Code URL: https://github.com/adewynter/argumentation-llms
• Copy Paste: [[2309.16938]] I Wish to Have an Argument: Argumentative Reasoning in Large Language Models(http://arxiv.org/abs/2309.16938)
• Summary:

  We evaluate the ability of contemporary large language models (LLMs) to perform argumentative reasoning. We frame our experiments in terms of the argument mining (AM) and argument pair extraction (APE) tasks, and evaluate their ability to perform reasoning at increasing levels of abstraction in the input and output representations (e.g., arbitrary label sets, semantic graphs). We find that, although LLMs are able to match or surpass the state-of-the-art in AM and APE, their argumentative reasoning performance is very dependent on the input and output representation. We also find an "exemplar effect", where too many exemplars increasingly become detrimental for task performance, and about 4-5 being the optimal amount. Neither result extends to chain-of-thought (CoT) prompting: we find the exemplar effect to be nullified, and our results suggest that CoT allows for better performance under ill-conditioned problems. We hope that the work reported contributes to the improvement of argumentative reasoning in LLMs.

Title: Benchmarking Cognitive Biases in Large Language Models as Evaluators. (arXiv:2309.17012v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.17012
• Code URL: null
• Copy Paste: [[2309.17012]] Benchmarking Cognitive Biases in Large Language Models as Evaluators(http://arxiv.org/abs/2309.17012)
• Summary:

  Large Language Models (LLMs) have recently been shown to be effective as automatic evaluators with simple prompting and in-context learning. In this work, we assemble 15 LLMs of four different size ranges and evaluate their output responses by preference ranking from the other LLMs as evaluators, such as System Star is better than System Square. We then evaluate the quality of ranking outputs introducing the Cognitive Bias Benchmark for LLMs as Evaluators (CoBBLEr), a benchmark to measure six different cognitive biases in LLM evaluation outputs, such as the Egocentric bias where a model prefers to rank its own outputs highly in evaluation. We find that LLMs are biased text quality evaluators, exhibiting strong indications on our bias benchmark (average of 40% of comparisons across all models) within each of their evaluations that question their robustness as evaluators. Furthermore, we examine the correlation between human and machine preferences and calculate the average Rank-Biased Overlap (RBO) score to be 49.6%, indicating that machine preferences are misaligned with humans. According to our findings, LLMs may still be unable to be utilized for automatic annotation aligned with human preferences. Our project page is at: https://minnesotanlp.github.io/cobbler.

Title: Interpretable Long-Form Legal Question Answering with Retrieval-Augmented Large Language Models. (arXiv:2309.17050v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.17050
• Code URL: https://github.com/maastrichtlawtech/lleqa
• Copy Paste: [[2309.17050]] Interpretable Long-Form Legal Question Answering with Retrieval-Augmented Large Language Models(http://arxiv.org/abs/2309.17050)
• Summary:

  Many individuals are likely to face a legal dispute at some point in their lives, but their lack of understanding of how to navigate these complex issues often renders them vulnerable. The advancement of natural language processing opens new avenues for bridging this legal literacy gap through the development of automated legal aid systems. However, existing legal question answering (LQA) approaches often suffer from a narrow scope, being either confined to specific legal domains or limited to brief, uninformative responses. In this work, we propose an end-to-end methodology designed to generate long-form answers to any statutory law questions, utilizing a "retrieve-then-read" pipeline. To support this approach, we introduce and release the Long-form Legal Question Answering (LLeQA) dataset, comprising 1,868 expert-annotated legal questions in the French language, complete with detailed answers rooted in pertinent legal provisions. Our experimental results demonstrate promising performance on automatic evaluation metrics, but a qualitative analysis uncovers areas for refinement. As one of the only comprehensive, expert-annotated long-form LQA dataset, LLeQA has the potential to not only accelerate research towards resolving a significant real-world issue, but also act as a rigorous benchmark for evaluating NLP models in specialized domains. We publicly release our code, data, and models.

Title: Using Large Language Models for Qualitative Analysis can Introduce Serious Bias. (arXiv:2309.17147v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.17147
• Code URL: null
• Copy Paste: [[2309.17147]] Using Large Language Models for Qualitative Analysis can Introduce Serious Bias(http://arxiv.org/abs/2309.17147)
• Summary:

  Large Language Models (LLMs) are quickly becoming ubiquitous, but the implications for social science research are not yet well understood. This paper asks whether LLMs can help us analyse large-N qualitative data from open-ended interviews, with an application to transcripts of interviews with Rohingya refugees in Cox's Bazaar, Bangladesh. We find that a great deal of caution is needed in using LLMs to annotate text as there is a risk of introducing biases that can lead to misleading inferences. We here mean bias in the technical sense, that the errors that LLMs make in annotating interview transcripts are not random with respect to the characteristics of the interview subjects. Training simpler supervised models on high-quality human annotations with flexible coding leads to less measurement error and bias than LLM annotations. Therefore, given that some high quality annotations are necessary in order to asses whether an LLM introduces bias, we argue that it is probably preferable to train a bespoke model on these annotations than it is to use an LLM for annotation.

Title: Comparative Analysis of Named Entity Recognition in the Dungeons and Dragons Domain. (arXiv:2309.17171v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.17171
• Code URL: null
• Copy Paste: [[2309.17171]] Comparative Analysis of Named Entity Recognition in the Dungeons and Dragons Domain(http://arxiv.org/abs/2309.17171)
• Summary:

  Many NLP tasks, although well-resolved for general English, face challenges in specific domains like fantasy literature. This is evident in Named Entity Recognition (NER), which detects and categorizes entities in text. We analyzed 10 NER models on 7 Dungeons and Dragons (D&D) adventure books to assess domain-specific performance. Using open-source Large Language Models, we annotated named entities in these books and evaluated each model's precision. Our findings indicate that, without modifications, Flair, Trankit, and Spacy outperform others in identifying named entities in the D&D context.

Title: Alphazero-like Tree-Search can Guide Large Language Model Decoding and Training. (arXiv:2309.17179v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.17179
• Code URL: null
• Copy Paste: [[2309.17179]] Alphazero-like Tree-Search can Guide Large Language Model Decoding and Training(http://arxiv.org/abs/2309.17179)
• Summary:

  Large language models (LLMs) typically employ sampling or beam search, accompanied by prompts such as Chain-of-Thought (CoT), to boost reasoning and decoding ability. Recent work like Tree-of-Thought (ToT) and Reasoning via Planning (RAP) aim to augment the reasoning capabilities of LLMs by utilizing tree-search algorithms to guide multi-step reasoning. These methods mainly focus on LLMs' reasoning ability during inference and heavily rely on human-designed prompts to activate LLM as a value function, which lacks general applicability and scalability. To address these limitations, we present an AlphaZero-like tree-search framework for LLMs (termed TS-LLM), systematically illustrating how tree-search with a learned value function can guide LLMs' decoding ability. TS-LLM distinguishes itself in two key ways: (1) Leveraging a learned value function, our approach can be generally applied to different tasks beyond reasoning (such as RLHF alignment), and LLMs of any size, without prompting advanced, large-scale models. (2) It can guide LLM's decoding during both inference and training. Empirical evaluations across reasoning, planning, and RLHF alignment tasks validate the effectiveness of TS-LLM, even on trees with a depth of 64.

Title: Training and inference of large language models using 8-bit floating point. (arXiv:2309.17224v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.17224
• Code URL: null
• Copy Paste: [[2309.17224]] Training and inference of large language models using 8-bit floating point(http://arxiv.org/abs/2309.17224)
• Summary:

  FP8 formats are gaining popularity to boost the computational efficiency for training and inference of large deep learning models. Their main challenge is that a careful choice of scaling is needed to prevent degradation due to the reduced dynamic range compared to higher-precision formats. Although there exists ample literature about selecting such scalings for INT formats, this critical aspect has yet to be addressed for FP8. This paper presents a methodology to select the scalings for FP8 linear layers, based on dynamically updating per-tensor scales for the weights, gradients and activations. We apply this methodology to train and validate large language models of the type of GPT and Llama 2 using FP8, for model sizes ranging from 111M to 70B. To facilitate the understanding of the FP8 dynamics, our results are accompanied by plots of the per-tensor scale distribution for weights, activations and gradients during both training and inference.

Title: LLM-Deliberation: Evaluating LLMs with Interactive Multi-Agent Negotiation Games. (arXiv:2309.17234v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.17234
• Code URL: null
• Copy Paste: [[2309.17234]] LLM-Deliberation: Evaluating LLMs with Interactive Multi-Agent Negotiation Games(http://arxiv.org/abs/2309.17234)
• Summary:

  There is a growing interest in using Large Language Models (LLMs) as agents to tackle real-world tasks that may require assessing complex situations. Yet, we have a limited understanding of LLMs' reasoning and decision-making capabilities, partly stemming from a lack of dedicated evaluation benchmarks. As negotiating and compromising are key aspects of our everyday communication and collaboration, we propose using scorable negotiation games as a new evaluation framework for LLMs. We create a testbed of diverse text-based, multi-agent, multi-issue, semantically rich negotiation games, with easily tunable difficulty. To solve the challenge, agents need to have strong arithmetic, inference, exploration, and planning capabilities, while seamlessly integrating them. Via a systematic zero-shot Chain-of-Thought prompting (CoT), we show that agents can negotiate and consistently reach successful deals. We quantify the performance with multiple metrics and observe a large gap between GPT-4 and earlier models. Importantly, we test the generalization to new games and setups. Finally, we show that these games can help evaluate other critical aspects, such as the interaction dynamics between agents in the presence of greedy and adversarial players.

Title: Batch Calibration: Rethinking Calibration for In-Context Learning and Prompt Engineering. (arXiv:2309.17249v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.17249
• Code URL: null
• Copy Paste: [[2309.17249]] Batch Calibration: Rethinking Calibration for In-Context Learning and Prompt Engineering(http://arxiv.org/abs/2309.17249)
• Summary:

  Prompting and in-context learning (ICL) have become efficient learning paradigms for large language models (LLMs). However, LLMs suffer from prompt brittleness and various bias factors in the prompt, including but not limited to the formatting, the choice verbalizers, and the ICL examples. To address this problem that results in unexpected performance degradation, calibration methods have been developed to mitigate the effects of these biases while recovering LLM performance. In this work, we first conduct a systematic analysis of the existing calibration methods, where we both provide a unified view and reveal the failure cases. Inspired by these analyses, we propose Batch Calibration (BC), a simple yet intuitive method that controls the contextual bias from the batched input, unifies various prior approaches, and effectively addresses the aforementioned issues. BC is zero-shot, inference-only, and incurs negligible additional costs. In the few-shot setup, we further extend BC to allow it to learn the contextual bias from labeled data. We validate the effectiveness of BC with PaLM 2-(S, M, L) and CLIP models and demonstrate state-of-the-art performance over previous calibration baselines across more than 10 natural language understanding and image classification tasks.

Title: Enhancing Large Language Models in Coding Through Multi-Perspective Self-Consistency. (arXiv:2309.17272v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.17272
• Code URL: null
• Copy Paste: [[2309.17272]] Enhancing Large Language Models in Coding Through Multi-Perspective Self-Consistency(http://arxiv.org/abs/2309.17272)
• Summary:

  Large language models (LLMs) have exhibited remarkable ability in textual generation. However, in complex reasoning tasks such as code generation, generating the correct answer in a single attempt remains a formidable challenge for LLMs. Previous research has explored solutions by aggregating multiple outputs, leveraging the consistency among them. However, none of them have comprehensively captured this consistency from different perspectives. In this paper, we propose the Multi-Perspective Self-Consistency (MPSC) framework, a novel decoding strategy for LLM that incorporates both inter-consistency across outputs from multiple perspectives and intra-consistency within a single perspective. Specifically, we ask LLMs to sample multiple diverse outputs from various perspectives for a given query and then construct a multipartite graph based on them. With two predefined measures of consistency, we embed both inter- and intra-consistency information into the graph. The optimal choice is then determined based on consistency analysis in the graph. We conduct comprehensive evaluation on the code generation task by introducing solution, specification and test case as three perspectives. We leverage a code interpreter to quantitatively measure the inter-consistency and propose several intra-consistency measure functions. Our MPSC framework significantly boosts the performance on various popular benchmarks, including HumanEval (+17.60%), HumanEval Plus (+17.61%), MBPP (+6.50%) and CodeContests (+11.82%) in Pass@1, when compared to original outputs generated from ChatGPT, and even surpassing GPT-4.

Title: Pushing Large Language Models to the 6G Edge: Vision, Challenges, and Opportunities. (arXiv:2309.16739v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.16739
• Code URL: null
• Copy Paste: [[2309.16739]] Pushing Large Language Models to the 6G Edge: Vision, Challenges, and Opportunities(http://arxiv.org/abs/2309.16739)
• Summary:

  Large language models (LLMs), which have shown remarkable capabilities, are revolutionizing AI development and potentially shaping our future. However, given their multimodality, the status quo cloud-based deployment faces some critical challenges: 1) long response time; 2) high bandwidth costs; and 3) the violation of data privacy. 6G mobile edge computing (MEC) systems may resolve these pressing issues. In this article, we explore the potential of deploying LLMs at the 6G edge. We start by introducing killer applications powered by multimodal LLMs, including robotics and healthcare, to highlight the need for deploying LLMs in the vicinity of end users. Then, we identify the critical challenges for LLM deployment at the edge and envision the 6G MEC architecture for LLMs. Furthermore, we delve into two design aspects, i.e., edge training and edge inference for LLMs. In both aspects, considering the inherent resource limitations at the edge, we discuss various cutting-edge techniques, including split learning/inference, parameter-efficient fine-tuning, quantization, and parameter-sharing inference, to facilitate the efficient deployment of LLMs. This article serves as a position paper for thoroughly identifying the motivation, challenges, and pathway for empowering LLMs at the 6G edge.

Title: Benchmarking and In-depth Performance Study of Large Language Models on Habana Gaudi Processors. (arXiv:2309.16976v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.16976
• Code URL: null
• Copy Paste: [[2309.16976]] Benchmarking and In-depth Performance Study of Large Language Models on Habana Gaudi Processors(http://arxiv.org/abs/2309.16976)
• Summary:

  Transformer models have achieved remarkable success in various machine learning tasks but suffer from high computational complexity and resource requirements. The quadratic complexity of the self-attention mechanism further exacerbates these challenges when dealing with long sequences and large datasets. Specialized AI hardware accelerators, such as the Habana GAUDI architecture, offer a promising solution to tackle these issues. GAUDI features a Matrix Multiplication Engine (MME) and a cluster of fully programmable Tensor Processing Cores (TPC). This paper explores the untapped potential of using GAUDI processors to accelerate Transformer-based models, addressing key challenges in the process. Firstly, we provide a comprehensive performance comparison between the MME and TPC components, illuminating their relative strengths and weaknesses. Secondly, we explore strategies to optimize MME and TPC utilization, offering practical insights to enhance computational efficiency. Thirdly, we evaluate the performance of Transformers on GAUDI, particularly in handling long sequences and uncovering performance bottlenecks. Lastly, we evaluate the end-to-end performance of two Transformer-based large language models (LLM) on GAUDI. The contributions of this work encompass practical insights for practitioners and researchers alike. We delve into GAUDI's capabilities for Transformers through systematic profiling, analysis, and optimization exploration. Our study bridges a research gap and offers a roadmap for optimizing Transformer-based model training on the GAUDI architecture.

segmentation

Title: Automatic Cadastral Boundary Detection of Very High Resolution Images Using Mask R-CNN. (arXiv:2309.16708v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.16708
• Code URL: null
• Copy Paste: [[2309.16708]] Automatic Cadastral Boundary Detection of Very High Resolution Images Using Mask R-CNN(http://arxiv.org/abs/2309.16708)
• Summary:

  Recently, there has been a high demand for accelerating and improving the detection of automatic cadastral mapping. As this problem is in its starting point, there are many methods of computer vision and deep learning that have not been considered yet. In this paper, we focus on deep learning and provide three geometric post-processing methods that improve the quality of the work. Our framework includes two parts, each of which consists of a few phases. Our solution to this problem uses instance segmentation. In the first part, we use Mask R-CNN with the backbone of pre-trained ResNet-50 on the ImageNet dataset. In the second phase, we apply three geometric post-processing methods to the output of the first part to get better overall output. Here, we also use computational geometry to introduce a new method for simplifying lines which we call it pocket-based simplification algorithm. For evaluating the quality of our solution, we use popular formulas in this field which are recall, precision and F-score. The highest recall we gain is 95 percent which also maintains high Precision of 72 percent. This resulted in an F-score of 82 percent. Implementing instance segmentation using Mask R-CNN with some geometric post-processes to its output gives us promising results for this field. Also, results show that pocket-based simplification algorithms work better for simplifying lines than Douglas-Puecker algorithm.

Title: Photonic Accelerators for Image Segmentation in Autonomous Driving and Defect Detection. (arXiv:2309.16783v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.16783
• Code URL: null
• Copy Paste: [[2309.16783]] Photonic Accelerators for Image Segmentation in Autonomous Driving and Defect Detection(http://arxiv.org/abs/2309.16783)
• Summary:

  Photonic computing promises faster and more energy-efficient deep neural network (DNN) inference than traditional digital hardware. Advances in photonic computing can have profound impacts on applications such as autonomous driving and defect detection that depend on fast, accurate and energy efficient execution of image segmentation models. In this paper, we investigate image segmentation on photonic accelerators to explore: a) the types of image segmentation DNN architectures that are best suited for photonic accelerators, and b) the throughput and energy efficiency of executing the different image segmentation models on photonic accelerators, along with the trade-offs involved therein. Specifically, we demonstrate that certain segmentation models exhibit negligible loss in accuracy (compared to digital float32 models) when executed on photonic accelerators, and explore the empirical reasoning for their robustness. We also discuss techniques for recovering accuracy in the case of models that do not perform well. Further, we compare throughput (inferences-per-second) and energy consumption estimates for different image segmentation workloads on photonic accelerators. We discuss the challenges and potential optimizations that can help improve the application of photonic accelerators to such computer vision tasks.

Title: LEF: Late-to-Early Temporal Fusion for LiDAR 3D Object Detection. (arXiv:2309.16870v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.16870
• Code URL: null
• Copy Paste: [[2309.16870]] LEF: Late-to-Early Temporal Fusion for LiDAR 3D Object Detection(http://arxiv.org/abs/2309.16870)
• Summary:

  We propose a late-to-early recurrent feature fusion scheme for 3D object detection using temporal LiDAR point clouds. Our main motivation is fusing object-aware latent embeddings into the early stages of a 3D object detector. This feature fusion strategy enables the model to better capture the shapes and poses for challenging objects, compared with learning from raw points directly. Our method conducts late-to-early feature fusion in a recurrent manner. This is achieved by enforcing window-based attention blocks upon temporally calibrated and aligned sparse pillar tokens. Leveraging bird's eye view foreground pillar segmentation, we reduce the number of sparse history features that our model needs to fuse into its current frame by 10$\times$. We also propose a stochastic-length FrameDrop training technique, which generalizes the model to variable frame lengths at inference for improved performance without retraining. We evaluate our method on the widely adopted Waymo Open Dataset and demonstrate improvement on 3D object detection against the baseline model, especially for the challenging category of large objects.

Title: Investigating Shift Equivalence of Convolutional Neural Networks in Industrial Defect Segmentation. (arXiv:2309.16902v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.16902
• Code URL: https://github.com/xiaozhen228/caps
• Copy Paste: [[2309.16902]] Investigating Shift Equivalence of Convolutional Neural Networks in Industrial Defect Segmentation(http://arxiv.org/abs/2309.16902)
• Summary:

  In industrial defect segmentation tasks, while pixel accuracy and Intersection over Union (IoU) are commonly employed metrics to assess segmentation performance, the output consistency (also referred to equivalence) of the model is often overlooked. Even a small shift in the input image can yield significant fluctuations in the segmentation results. Existing methodologies primarily focus on data augmentation or anti-aliasing to enhance the network's robustness against translational transformations, but their shift equivalence performs poorly on the test set or is susceptible to nonlinear activation functions. Additionally, the variations in boundaries resulting from the translation of input images are consistently disregarded, thus imposing further limitations on the shift equivalence. In response to this particular challenge, a novel pair of down/upsampling layers called component attention polyphase sampling (CAPS) is proposed as a replacement for the conventional sampling layers in CNNs. To mitigate the effect of image boundary variations on the equivalence, an adaptive windowing module is designed in CAPS to adaptively filter out the border pixels of the image. Furthermore, a component attention module is proposed to fuse all downsampled features to improve the segmentation performance. The experimental results on the micro surface defect (MSD) dataset and four real-world industrial defect datasets demonstrate that the proposed method exhibits higher equivalence and segmentation performance compared to other state-of-the-art methods.Our code will be available at https://github.com/xiaozhen228/CAPS.

Title: YOLOR-Based Multi-Task Learning. (arXiv:2309.16921v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.16921
• Code URL: null
• Copy Paste: [[2309.16921]] YOLOR-Based Multi-Task Learning(http://arxiv.org/abs/2309.16921)
• Summary:

  Multi-task learning (MTL) aims to learn multiple tasks using a single model and jointly improve all of them assuming generalization and shared semantics. Reducing conflicts between tasks during joint learning is difficult and generally requires careful network design and extremely large models. We propose building on You Only Learn One Representation (YOLOR), a network architecture specifically designed for multitasking. YOLOR leverages both explicit and implicit knowledge, from data observations and learned latents, respectively, to improve a shared representation while minimizing the number of training parameters. However, YOLOR and its follow-up, YOLOv7, only trained two tasks at once. In this paper, we jointly train object detection, instance segmentation, semantic segmentation, and image captioning. We analyze tradeoffs and attempt to maximize sharing of semantic information. Through our architecture and training strategies, we find that our method achieves competitive performance on all tasks while maintaining a low parameter count and without any pre-training. We will release code soon.

Title: Model2Scene: Learning 3D Scene Representation via Contrastive Language-CAD Models Pre-training. (arXiv:2309.16956v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.16956
• Code URL: null
• Copy Paste: [[2309.16956]] Model2Scene: Learning 3D Scene Representation via Contrastive Language-CAD Models Pre-training(http://arxiv.org/abs/2309.16956)
• Summary:

  Current successful methods of 3D scene perception rely on the large-scale annotated point cloud, which is tedious and expensive to acquire. In this paper, we propose Model2Scene, a novel paradigm that learns free 3D scene representation from Computer-Aided Design (CAD) models and languages. The main challenges are the domain gaps between the CAD models and the real scene's objects, including model-to-scene (from a single model to the scene) and synthetic-to-real (from synthetic model to real scene's object). To handle the above challenges, Model2Scene first simulates a crowded scene by mixing data-augmented CAD models. Next, we propose a novel feature regularization operation, termed Deep Convex-hull Regularization (DCR), to project point features into a unified convex hull space, reducing the domain gap. Ultimately, we impose contrastive loss on language embedding and the point features of CAD models to pre-train the 3D network. Extensive experiments verify the learned 3D scene representation is beneficial for various downstream tasks, including label-free 3D object salient detection, label-efficient 3D scene perception and zero-shot 3D semantic segmentation. Notably, Model2Scene yields impressive label-free 3D object salient detection with an average mAP of 46.08\% and 55.49\% on the ScanNet and S3DIS datasets, respectively. The code will be publicly available.

Title: COMNet: Co-Occurrent Matching for Weakly Supervised Semantic Segmentation. (arXiv:2309.16959v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.16959
• Code URL: null
• Copy Paste: [[2309.16959]] COMNet: Co-Occurrent Matching for Weakly Supervised Semantic Segmentation(http://arxiv.org/abs/2309.16959)
• Summary:

  Image-level weakly supervised semantic segmentation is a challenging task that has been deeply studied in recent years. Most of the common solutions exploit class activation map (CAM) to locate object regions. However, such response maps generated by the classification network usually focus on discriminative object parts. In this paper, we propose a novel Co-Occurrent Matching Network (COMNet), which can promote the quality of the CAMs and enforce the network to pay attention to the entire parts of objects. Specifically, we perform inter-matching on paired images that contain common classes to enhance the corresponded areas, and construct intra-matching on a single image to propagate the semantic features across the object regions. The experiments on the Pascal VOC 2012 and MS-COCO datasets show that our network can effectively boost the performance of the baseline model and achieve new state-of-the-art performance.

Title: SegRCDB: Semantic Segmentation via Formula-Driven Supervised Learning. (arXiv:2309.17083v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.17083
• Code URL: https://github.com/dahlian00/segrcdb
• Copy Paste: [[2309.17083]] SegRCDB: Semantic Segmentation via Formula-Driven Supervised Learning(http://arxiv.org/abs/2309.17083)
• Summary:

  Pre-training is a strong strategy for enhancing visual models to efficiently train them with a limited number of labeled images. In semantic segmentation, creating annotation masks requires an intensive amount of labor and time, and therefore, a large-scale pre-training dataset with semantic labels is quite difficult to construct. Moreover, what matters in semantic segmentation pre-training has not been fully investigated. In this paper, we propose the Segmentation Radial Contour DataBase (SegRCDB), which for the first time applies formula-driven supervised learning for semantic segmentation. SegRCDB enables pre-training for semantic segmentation without real images or any manual semantic labels. SegRCDB is based on insights about what is important in pre-training for semantic segmentation and allows efficient pre-training. Pre-training with SegRCDB achieved higher mIoU than the pre-training with COCO-Stuff for fine-tuning on ADE-20k and Cityscapes with the same number of training images. SegRCDB has a high potential to contribute to semantic segmentation pre-training and investigation by enabling the creation of large datasets without manual annotation. The SegRCDB dataset will be released under a license that allows research and commercial use. Code is available at: https://github.com/dahlian00/SegRCDB

Title: APNet: Urban-level Scene Segmentation of Aerial Images and Point Clouds. (arXiv:2309.17162v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.17162
• Code URL: null
• Copy Paste: [[2309.17162]] APNet: Urban-level Scene Segmentation of Aerial Images and Point Clouds(http://arxiv.org/abs/2309.17162)
• Summary:

  In this paper, we focus on semantic segmentation method for point clouds of urban scenes. Our fundamental concept revolves around the collaborative utilization of diverse scene representations to benefit from different context information and network architectures. To this end, the proposed network architecture, called APNet, is split into two branches: a point cloud branch and an aerial image branch which input is generated from a point cloud. To leverage the different properties of each branch, we employ a geometry-aware fusion module that is learned to combine the results of each branch. Additional separate losses for each branch avoid that one branch dominates the results, ensure the best performance for each branch individually and explicitly define the input domain of the fusion network assuring it only performs data fusion. Our experiments demonstrate that the fusion output consistently outperforms the individual network branches and that APNet achieves state-of-the-art performance of 65.2 mIoU on the SensatUrban dataset. Upon acceptance, the source code will be made accessible.

Title: Advances in Kidney Biopsy Structural Assessment through Dense Instance Segmentation. (arXiv:2309.17166v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.17166
• Code URL: null
• Copy Paste: [[2309.17166]] Advances in Kidney Biopsy Structural Assessment through Dense Instance Segmentation(http://arxiv.org/abs/2309.17166)
• Summary:

  The kidney biopsy is the gold standard for the diagnosis of kidney diseases. Lesion scores made by expert renal pathologists are semi-quantitative and suffer from high inter-observer variability. Automatically obtaining statistics per segmented anatomical object, therefore, can bring significant benefits in reducing labor and this inter-observer variability. Instance segmentation for a biopsy, however, has been a challenging problem due to (a) the on average large number (around 300 to 1000) of densely touching anatomical structures, (b) with multiple classes (at least 3) and (c) in different sizes and shapes. The currently used instance segmentation models cannot simultaneously deal with these challenges in an efficient yet generic manner. In this paper, we propose the first anchor-free instance segmentation model that combines diffusion models, transformer modules, and RCNNs (regional convolution neural networks). Our model is trained on just one NVIDIA GeForce RTX 3090 GPU, but can efficiently recognize more than 500 objects with 3 common anatomical object classes in renal biopsies, i.e., glomeruli, tubuli, and arteries. Our data set consisted of 303 patches extracted from 148 Jones' silver-stained renal whole slide images (WSIs), where 249 patches were used for training and 54 patches for evaluation. In addition, without adjustment or retraining, the model can directly transfer its domain to generate decent instance segmentation results from PAS-stained WSIs. Importantly, it outperforms other baseline models and reaches an AP 51.7% in detection as the new state-of-the-art.

Title: Towards Complex-query Referring Image Segmentation: A Novel Benchmark. (arXiv:2309.17205v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.17205
• Code URL: null
• Copy Paste: [[2309.17205]] Towards Complex-query Referring Image Segmentation: A Novel Benchmark(http://arxiv.org/abs/2309.17205)
• Summary:

  Referring Image Understanding (RIS) has been extensively studied over the past decade, leading to the development of advanced algorithms. However, there has been a lack of research investigating how existing algorithms should be benchmarked with complex language queries, which include more informative descriptions of surrounding objects and backgrounds (\eg \textit{"the black car."} vs. \textit{"the black car is parking on the road and beside the bus."}). Given the significant improvement in the semantic understanding capability of large pre-trained models, it is crucial to take a step further in RIS by incorporating complex language that resembles real-world applications. To close this gap, building upon the existing RefCOCO and Visual Genome datasets, we propose a new RIS benchmark with complex queries, namely \textbf{RIS-CQ}. The RIS-CQ dataset is of high quality and large scale, which challenges the existing RIS with enriched, specific and informative queries, and enables a more realistic scenario of RIS research. Besides, we present a nichetargeting method to better task the RIS-CQ, called dual-modality graph alignment model (\textbf{\textsc{DuMoGa}}), which outperforms a series of RIS methods.

Title: A Foundation Model for General Moving Object Segmentation in Medical Images. (arXiv:2309.17264v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.17264
• Code URL: null
• Copy Paste: [[2309.17264]] A Foundation Model for General Moving Object Segmentation in Medical Images(http://arxiv.org/abs/2309.17264)
• Summary:

  Medical image segmentation aims to delineate the anatomical or pathological structures of interest, playing a crucial role in clinical diagnosis. A substantial amount of high-quality annotated data is crucial for constructing high-precision deep segmentation models. However, medical annotation is highly cumbersome and time-consuming, especially for medical videos or 3D volumes, due to the huge labeling space and poor inter-frame consistency. Recently, a fundamental task named Moving Object Segmentation (MOS) has made significant advancements in natural images. Its objective is to delineate moving objects from the background within image sequences, requiring only minimal annotations. In this paper, we propose the first foundation model, named iMOS, for MOS in medical images. Extensive experiments on a large multi-modal medical dataset validate the effectiveness of the proposed iMOS. Specifically, with the annotation of only a small number of images in the sequence, iMOS can achieve satisfactory tracking and segmentation performance of moving objects throughout the entire sequence in bi-directions. We hope that the proposed iMOS can help accelerate the annotation speed of experts, and boost the development of medical foundation models.

Title: Benchmarking Collaborative Learning Methods Cost-Effectiveness for Prostate Segmentation. (arXiv:2309.17097v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.17097
• Code URL: null
• Copy Paste: [[2309.17097]] Benchmarking Collaborative Learning Methods Cost-Effectiveness for Prostate Segmentation(http://arxiv.org/abs/2309.17097)
• Summary:

  Healthcare data is often split into medium/small-sized collections across multiple hospitals and access to it is encumbered by privacy regulations. This brings difficulties to use them for the development of machine learning and deep learning models, which are known to be data-hungry. One way to overcome this limitation is to use collaborative learning (CL) methods, which allow hospitals to work collaboratively to solve a task, without the need to explicitly share local data.

  In this paper, we address a prostate segmentation problem from MRI in a collaborative scenario by comparing two different approaches: federated learning (FL) and consensus-based methods (CBM).

  To the best of our knowledge, this is the first work in which CBM, such as label fusion techniques, are used to solve a problem of collaborative learning. In this setting, CBM combine predictions from locally trained models to obtain a federated strong learner with ideally improved robustness and predictive variance properties.

  Our experiments show that, in the considered practical scenario, CBMs provide equal or better results than FL, while being highly cost-effective. Our results demonstrate that the consensus paradigm may represent a valid alternative to FL for typical training tasks in medical imaging.