secure

Title: SecureCut: Federated Gradient Boosting Decision Trees with Efficient Machine Unlearning. (arXiv:2311.13174v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.13174
• Code URL: null
• Copy Paste: [[2311.13174]] SecureCut: Federated Gradient Boosting Decision Trees with Efficient Machine Unlearning(http://arxiv.org/abs/2311.13174)
• Summary:

  In response to legislation mandating companies to honor the \textit{right to be forgotten} by erasing user data, it has become imperative to enable data removal in Vertical Federated Learning (VFL) where multiple parties provide private features for model training. In VFL, data removal, i.e., \textit{machine unlearning}, often requires removing specific features across all samples under privacy guarentee in federated learning. To address this challenge, we propose \methname, a novel Gradient Boosting Decision Tree (GBDT) framework that effectively enables both \textit{instance unlearning} and \textit{feature unlearning} without the need for retraining from scratch. Leveraging a robust GBDT structure, we enable effective data deletion while reducing degradation of model performance. Extensive experimental results on popular datasets demonstrate that our method achieves superior model utility and forgetfulness compared to \textit{state-of-the-art} methods. To our best knowledge, this is the first work that investigates machine unlearning in VFL scenarios.

Title: Gradual Verification for Smart Contracts. (arXiv:2311.13351v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.13351
• Code URL: null
• Copy Paste: [[2311.13351]] Gradual Verification for Smart Contracts(http://arxiv.org/abs/2311.13351)
• Summary:

  Blockchains facilitate secure resource transactions through smart contracts, yet these digital agreements are prone to vulnerabilities, particularly when interacting with external contracts, leading to substantial monetary losses. Traditional verification techniques fall short in providing comprehensive security assurances, especially against re-entrancy attacks, due to the unavailable implementations of external contracts. This paper introduces an incremental approach: gradual verification. We combine static and dynamic verification techniques to enhance security, guarantee soundness and flexibility, and optimize resource usage in smart contract interactions. By implementing a prototype for gradually verifying Algorand smart contracts via the pyTEAL language, we demonstrate the effectiveness of our approach, contributing to the safe and efficient execution of smart contracts.

Title: A Comparative Analysis Between SciTokens, Verifiable Credentials, and Smart Contracts: Novel Approaches for Authentication and Secure Access to Scientific Data. (arXiv:2311.13422v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.13422
• Code URL: null
• Copy Paste: [[2311.13422]] A Comparative Analysis Between SciTokens, Verifiable Credentials, and Smart Contracts: Novel Approaches for Authentication and Secure Access to Scientific Data(http://arxiv.org/abs/2311.13422)
• Summary:

  Managing and exchanging sensitive information securely is a paramount concern for the scientific and cybersecurity community. The increasing reliance on computing workflows and digital data transactions requires ensuring that sensitive information is protected from unauthorized access, tampering, or misuse. This research paper presents a comparative analysis of three novel approaches for authenticating and securing access to scientific data: SciTokens, Verifiable Credentials, and Smart Contracts. The aim of this study is to investigate the strengths and weaknesses of each approach from trust, revocation, privacy, and security perspectives. We examine the technical features and privacy and security mechanisms of each technology and provide a comparative synthesis with the proposed model. Through our analysis, we demonstrate that each technology offers unique advantages and limitations, and the integration of these technologies can lead to more secure and efficient solutions for authentication and access to scientific data.

security

Title: Transfer Learning-based Real-time Handgun Detection. (arXiv:2311.13559v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.13559
• Code URL: null
• Copy Paste: [[2311.13559]] Transfer Learning-based Real-time Handgun Detection(http://arxiv.org/abs/2311.13559)
• Summary:

  Traditional surveillance systems rely on human attention, limiting their effectiveness. This study employs convolutional neural networks and transfer learning to develop a real-time computer vision system for automatic handgun detection. Comprehensive analysis of online handgun detection methods is conducted, emphasizing reducing false positives and learning time. Transfer learning is demonstrated as an effective approach. Despite technical challenges, the proposed system achieves a precision rate of 84.74%, demonstrating promising performance comparable to related works, enabling faster learning and accurate automatic handgun detection for enhanced security. This research advances security measures by reducing human monitoring dependence, showcasing the potential of transfer learning-based approaches for efficient and reliable handgun detection.

Title: Progression and Challenges of IoT in Healthcare: A Short Review. (arXiv:2311.12869v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.12869
• Code URL: null
• Copy Paste: [[2311.12869]] Progression and Challenges of IoT in Healthcare: A Short Review(http://arxiv.org/abs/2311.12869)
• Summary:

  Smart healthcare, an integral element of connected living, plays a pivotal role in fulfilling a fundamental human need. The burgeoning field of smart healthcare is poised to generate substantial revenue in the foreseeable future. Its multifaceted framework encompasses vital components such as the Internet of Things (IoT), medical sensors, artificial intelligence (AI), edge and cloud computing, as well as next-generation wireless communication technologies. Many research papers discuss smart healthcare and healthcare more broadly. Numerous nations have strategically deployed the Internet of Medical Things (IoMT) alongside other measures to combat the propagation of COVID-19. This combined effort has not only enhanced the safety of frontline healthcare workers but has also augmented the overall efficacy in managing the pandemic, subsequently reducing its impact on human lives and mortality rates. Remarkable strides have been made in both applications and technology within the IoMT domain. However, it is imperative to acknowledge that this technological advancement has introduced certain challenges, particularly in the realm of security. The rapid and extensive adoption of IoMT worldwide has magnified issues related to security and privacy. These encompass a spectrum of concerns, ranging from replay attacks, man-in-the-middle attacks, impersonation, privileged insider threats, remote hijacking, password guessing, and denial of service (DoS) attacks, to malware incursions. In this comprehensive review, we undertake a comparative analysis of existing strategies designed for the detection and prevention of malware in IoT environments.

Title: Towards new challenges of modern Pentest. (arXiv:2311.12952v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.12952
• Code URL: null
• Copy Paste: [[2311.12952]] Towards new challenges of modern Pentest(http://arxiv.org/abs/2311.12952)
• Summary:

  With the increasing number of internet-based resources and applications, the amount of attacks faced by companies has increased significantly in the past years. Likewise, the techniques to test security and emulate attacks need to be constantly improved and, as a consequence, help to mitigate attacks. Among these techniques, penetration test (Pentest) provides methods to assess the security posture of assets, using different tools and methodologies applied in specific scenarios. Therefore, this study aims to present current methodologies, tools, and potential challenges applied to Pentest from an updated systematic literature review. As a result, this work provides a new perspective on the scenarios where penetration tests are performed. Also, it presents new challenges such as automation of techniques, management of costs associated with offensive security, and the difficulty in hiring qualified professionals to perform Pentest.

Title: Is your vote truly secret? Ballot Secrecy iff Ballot Independence: Proving necessary conditions and analysing case studies. (arXiv:2311.12977v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.12977
• Code URL: null
• Copy Paste: [[2311.12977]] Is your vote truly secret? Ballot Secrecy iff Ballot Independence: Proving necessary conditions and analysing case studies(http://arxiv.org/abs/2311.12977)
• Summary:

  We formalise definitions of ballot secrecy and ballot independence by Smyth, JCS'21 as indistinguishability games in the computational model of security. These definitions improve upon Smyth, draft '21 to consider a wider class of voting systems. Both Smyth, JCS'21 and Smyth, draft '21 improve on earlier works by considering a more realistic adversary model wherein they have access to the ballot collection. We prove that ballot secrecy implies ballot independence. We say ballot independence holds if a system has non-malleable ballots. We construct games for ballot secrecy and non-malleability and show that voting schemes with malleable ballots do not preserve ballot secrecy. We demonstrate that Helios does not satisfy our definition of ballot secrecy. Furthermore, the Python framework we constructed for our case study shows that if an attack exists against non-malleability, this attack can be used to break ballot secrecy.

Title: A Survey of Adversarial CAPTCHAs on its History, Classification and Generation. (arXiv:2311.13233v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.13233
• Code URL: null
• Copy Paste: [[2311.13233]] A Survey of Adversarial CAPTCHAs on its History, Classification and Generation(http://arxiv.org/abs/2311.13233)
• Summary:

  Completely Automated Public Turing test to tell Computers and Humans Apart, short for CAPTCHA, is an essential and relatively easy way to defend against malicious attacks implemented by bots. The security and usability trade-off limits the use of massive geometric transformations to interfere deep model recognition and deep models even outperformed humans in complex CAPTCHAs. The discovery of adversarial examples provides an ideal solution to the security and usability trade-off by integrating adversarial examples and CAPTCHAs to generate adversarial CAPTCHAs that can fool the deep models. In this paper, we extend the definition of adversarial CAPTCHAs and propose a classification method for adversarial CAPTCHAs. Then we systematically review some commonly used methods to generate adversarial examples and methods that are successfully used to generate adversarial CAPTCHAs. Also, we analyze some defense methods that can be used to defend adversarial CAPTCHAs, indicating potential threats to adversarial CAPTCHAs. Finally, we discuss some possible future research directions for adversarial CAPTCHAs at the end of this paper.

Title: The Influence of Neural Networks on Hydropower Plant Management in Agriculture: Addressing Challenges and Exploring Untapped Opportunities. (arXiv:2311.13293v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.13293
• Code URL: null
• Copy Paste: [[2311.13293]] The Influence of Neural Networks on Hydropower Plant Management in Agriculture: Addressing Challenges and Exploring Untapped Opportunities(http://arxiv.org/abs/2311.13293)
• Summary:

  Hydropower plants are crucial for stable renewable energy and serve as vital water sources for sustainable agriculture. However, it is essential to assess the current water management practices associated with hydropower plant management software. A key concern is the potential conflict between electricity generation and agricultural water needs. Prioritising water for electricity generation can reduce irrigation availability in agriculture during crucial periods like droughts, impacting crop yields and regional food security. Coordination between electricity and agricultural water allocation is necessary to ensure optimal and environmentally sound practices. Neural networks have become valuable tools for hydropower plant management, but their black-box nature raises concerns about transparency in decision making. Additionally, current approaches often do not take advantage of their potential to create a system that effectively balances water allocation.

  This work is a call for attention and highlights the potential risks of deploying neural network-based hydropower plant management software without proper scrutiny and control. To address these concerns, we propose the adoption of the Agriculture Conscious Hydropower Plant Management framework, aiming to maximise electricity production while prioritising stable irrigation for agriculture. We also advocate reevaluating government-imposed minimum water guidelines for irrigation to ensure flexibility and effective water allocation. Additionally, we suggest a set of regulatory measures to promote model transparency and robustness, certifying software that makes conscious and intelligent water allocation decisions, ultimately safeguarding agriculture from undue strain during droughts.

privacy

Title: Meticulously Selecting 1% of the Dataset for Pre-training! Generating Differentially Private Images Data with Semantics Query. (arXiv:2311.12850v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12850
• Code URL: null
• Copy Paste: [[2311.12850]] Meticulously Selecting 1% of the Dataset for Pre-training! Generating Differentially Private Images Data with Semantics Query(http://arxiv.org/abs/2311.12850)
• Summary:

  Differential Privacy (DP) image data synthesis, which leverages the DP technique to generate synthetic data to replace the sensitive data, allowing organizations to share and utilize synthetic images without privacy concerns. Previous methods incorporate the advanced techniques of generative models and pre-training on a public dataset to produce exceptional DP image data, but suffer from problems of unstable training and massive computational resource demands. This paper proposes a novel DP image synthesis method, termed PRIVIMAGE, which meticulously selects pre-training data, promoting the efficient creation of DP datasets with high fidelity and utility. PRIVIMAGE first establishes a semantic query function using a public dataset. Then, this function assists in querying the semantic distribution of the sensitive dataset, facilitating the selection of data from the public dataset with analogous semantics for pre-training. Finally, we pre-train an image generative model using the selected data and then fine-tune this model on the sensitive dataset using Differentially Private Stochastic Gradient Descent (DP-SGD). PRIVIMAGE allows us to train a lightly parameterized generative model, reducing the noise in the gradient during DP-SGD training and enhancing training stability. Extensive experiments demonstrate that PRIVIMAGE uses only 1% of the public dataset for pre-training and 7.6% of the parameters in the generative model compared to the state-of-the-art method, whereas achieves superior synthetic performance and conserves more computational resources. On average, PRIVIMAGE achieves 30.1% lower FID and 12.6% higher Classification Accuracy than the state-of-the-art method. The replication package and datasets can be accessed online.

Title: zkTax: A pragmatic way to support zero-knowledge tax disclosures. (arXiv:2311.13008v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.13008
• Code URL: null
• Copy Paste: [[2311.13008]] zkTax: A pragmatic way to support zero-knowledge tax disclosures(http://arxiv.org/abs/2311.13008)
• Summary:

  Tax returns contain key financial information of interest to third parties: public officials are asked to share financial data for transparency, companies seek to assess the financial status of business partners, and individuals need to prove their income to landlords or to receive benefits. Tax returns also contain sensitive data such that sharing them in their entirety undermines privacy. We introduce a zero-knowledge tax disclosure system (zkTax) that allows individuals and organizations to make provable claims about select information in their tax returns without revealing additional information, which can be independently verified by third parties. The system consists of three 3distinct services that can be distributed: a tax authority provides tax documents signed with a public key; a Redact & Prove Service enables users to produce a redacted version of the tax documents with a zero-knowledge proof attesting the provenance of the redacted data; a Verify Service enables anyone to verify the proof. We implement a prototype with a user interface, compatible with U.S. tax forms, and demonstrate how this design could be implemented with minimal changes to existing tax infrastructure. Our system is designed to be extensible to other contexts and jurisdictions. This work provides a practical example of how distributed tools leveraging cryptography can enhance existing government or financial infrastructures, providing immediate transparency alongside privacy without system overhauls.

Title: Differentially Private Non-Convex Optimization under the KL Condition with Optimal Rates. (arXiv:2311.13447v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.13447
• Code URL: null
• Copy Paste: [[2311.13447]] Differentially Private Non-Convex Optimization under the KL Condition with Optimal Rates(http://arxiv.org/abs/2311.13447)
• Summary:

  We study private empirical risk minimization (ERM) problem for losses satisfying the $(\gamma,\kappa)$-Kurdyka-{\L}ojasiewicz (KL) condition. The Polyak-{\L}ojasiewicz (PL) condition is a special case of this condition when $\kappa=2$. Specifically, we study this problem under the constraint of $\rho$ zero-concentrated differential privacy (zCDP). When $\kappa\in[1,2]$ and the loss function is Lipschitz and smooth over a sufficiently large region, we provide a new algorithm based on variance reduced gradient descent that achieves the rate $\tilde{O}\big(\big(\frac{\sqrt{d}}{n\sqrt{\rho}}\big)^\kappa\big)$ on the excess empirical risk, where $n$ is the dataset size and $d$ is the dimension. We further show that this rate is nearly optimal. When $\kappa \geq 2$ and the loss is instead Lipschitz and weakly convex, we show it is possible to achieve the rate $\tilde{O}\big(\big(\frac{\sqrt{d}}{n\sqrt{\rho}}\big)^\kappa\big)$ with a private implementation of the proximal point method. When the KL parameters are unknown, we provide a novel modification and analysis of the noisy gradient descent algorithm and show that this algorithm achieves a rate of $\tilde{O}\big(\big(\frac{\sqrt{d}}{n\sqrt{\rho}}\big)^{\frac{2\kappa}{4-\kappa}}\big)$ adaptively, which is nearly optimal when $\kappa = 2$. We further show that, without assuming the KL condition, the same gradient descent algorithm can achieve fast convergence to a stationary point when the gradient stays sufficiently large during the run of the algorithm. Specifically, we show that this algorithm can approximate stationary points of Lipschitz, smooth (and possibly nonconvex) objectives with rate as fast as $\tilde{O}\big(\frac{\sqrt{d}}{n\sqrt{\rho}}\big)$ and never worse than $\tilde{O}\big(\big(\frac{\sqrt{d}}{n\sqrt{\rho}}\big)^{1/2}\big)$. The latter rate matches the best known rate for methods that do not rely on variance reduction.

Title: Summary Reports Optimization in the Privacy Sandbox Attribution Reporting API. (arXiv:2311.13586v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.13586
• Code URL: null
• Copy Paste: [[2311.13586]] Summary Reports Optimization in the Privacy Sandbox Attribution Reporting API(http://arxiv.org/abs/2311.13586)
• Summary:

  The Privacy Sandbox Attribution Reporting API has been recently deployed by Google Chrome to support the basic advertising functionality of attribution reporting (aka conversion measurement) after deprecation of third-party cookies. The API implements a collection of privacy-enhancing guardrails including contribution bounding and noise injection. It also offers flexibility for the analyst to allocate the contribution budget.

  In this work, we present methods for optimizing the allocation of the contribution budget for summary reports from the Attribution Reporting API. We evaluate them on real-world datasets as well as on a synthetic data model that we find to accurately capture real-world conversion data. Our results demonstrate that optimizing the parameters that can be set by the analyst can significantly improve the utility achieved by querying the API while satisfying the same privacy bounds.

Title: CovarNav: Machine Unlearning via Model Inversion and Covariance Navigation. (arXiv:2311.12999v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.12999
• Code URL: null
• Copy Paste: [[2311.12999]] CovarNav: Machine Unlearning via Model Inversion and Covariance Navigation(http://arxiv.org/abs/2311.12999)
• Summary:

  The rapid progress of AI, combined with its unprecedented public adoption and the propensity of large neural networks to memorize training data, has given rise to significant data privacy concerns. To address these concerns, machine unlearning has emerged as an essential technique to selectively remove the influence of specific training data points on trained models. In this paper, we approach the machine unlearning problem through the lens of continual learning. Given a trained model and a subset of training data designated to be forgotten (i.e., the "forget set"), we introduce a three-step process, named CovarNav, to facilitate this forgetting. Firstly, we derive a proxy for the model's training data using a model inversion attack. Secondly, we mislabel the forget set by selecting the most probable class that deviates from the actual ground truth. Lastly, we deploy a gradient projection method to minimize the cross-entropy loss on the modified forget set (i.e., learn incorrect labels for this set) while preventing forgetting of the inverted samples. We rigorously evaluate CovarNav on the CIFAR-10 and Vggface2 datasets, comparing our results with recent benchmarks in the field and demonstrating the efficacy of our proposed approach.

protect

Title: Toward effective protection against diffusion based mimicry through score distillation. (arXiv:2311.12832v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12832
• Code URL: null
• Copy Paste: [[2311.12832]] Toward effective protection against diffusion based mimicry through score distillation(http://arxiv.org/abs/2311.12832)
• Summary:

  While generative diffusion models excel in producing high-quality images, they can also be misused to mimic authorized images, posing a significant threat to AI systems. Efforts have been made to add calibrated perturbations to protect images from diffusion-based mimicry pipelines. However, most of the existing methods are too ineffective and even impractical to be used by individual users due to their high computation and memory requirements. In this work, we present novel findings on attacking latent diffusion models (LDM) and propose new plug-and-play strategies for more effective protection. In particular, we explore the bottleneck in attacking an LDM, discovering that the encoder module rather than the denoiser module is the vulnerable point. Based on this insight, we present our strategy using Score Distillation Sampling (SDS) to double the speed of protection and reduce memory occupation by half without compromising its strength. Additionally, we provide a robust protection strategy by counterintuitively minimizing the semantic loss, which can assist in generating more natural perturbations. Finally, we conduct extensive experiments to substantiate our findings and comprehensively evaluate our newly proposed strategies. We hope our insights and protective measures can contribute to better defense against malicious diffusion-based mimicry, advancing the development of secure AI systems. The code is available in https://github.com/xavihart/Diff-Protect

defense

Title: Transfer Attacks and Defenses for Large Language Models on Coding Tasks. (arXiv:2311.13445v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.13445
• Code URL: null
• Copy Paste: [[2311.13445]] Transfer Attacks and Defenses for Large Language Models on Coding Tasks(http://arxiv.org/abs/2311.13445)
• Summary:

  Modern large language models (LLMs), such as ChatGPT, have demonstrated impressive capabilities for coding tasks including writing and reasoning about code. They improve upon previous neural network models of code, such as code2seq or seq2seq, that already demonstrated competitive results when performing tasks such as code summarization and identifying code vulnerabilities. However, these previous code models were shown vulnerable to adversarial examples, i.e. small syntactic perturbations that do not change the program's semantics, such as the inclusion of "dead code" through false conditions or the addition of inconsequential print statements, designed to "fool" the models. LLMs can also be vulnerable to the same adversarial perturbations but a detailed study on this concern has been lacking so far. In this paper we aim to investigate the effect of adversarial perturbations on coding tasks with LLMs. In particular, we study the transferability of adversarial examples, generated through white-box attacks on smaller code models, to LLMs. Furthermore, to make the LLMs more robust against such adversaries without incurring the cost of retraining, we propose prompt-based defenses that involve modifying the prompt to include additional information such as examples of adversarially perturbed code and explicit instructions for reversing adversarial perturbations. Our experiments show that adversarial examples obtained with a smaller code model are indeed transferable, weakening the LLMs' performance. The proposed defenses show promise in improving the model's resilience, paving the way to more robust defensive solutions for LLMs in code-related applications.

attack

Title: Adversarial sample generation and training using geometric masks for accurate and resilient license plate character recognition. (arXiv:2311.12857v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12857
• Code URL: null
• Copy Paste: [[2311.12857]] Adversarial sample generation and training using geometric masks for accurate and resilient license plate character recognition(http://arxiv.org/abs/2311.12857)
• Summary:

  Reading dirty license plates accurately in moving vehicles is challenging for automatic license plate recognition systems. Moreover, license plates are often intentionally tampered with a malicious intent to avoid police apprehension. Usually, such groups and individuals know how to fool the existing recognition systems by making minor unnoticeable plate changes. Designing and developing deep learning methods resilient to such real-world 'attack' practices remains an active research problem. As a solution, this work develops a resilient method to recognize license plate characters. Extracting 1057 character images from 160 Nepalese vehicles, as the first step, we trained several standard deep convolutional neural networks to obtain 99.5% character classification accuracy. On adversarial images generated to simulate malicious tampering, however, our model's accuracy dropped to 25%. Next, we enriched our dataset by generating and adding geometrically masked images, retrained our models, and investigated the models' predictions. The proposed approach of training with generated adversarial images helped our adversarial attack-aware license plate character recognition (AA-LPCR) model achieves an accuracy of 99.7%. This near-perfect accuracy demonstrates that the proposed idea of random geometric masking is highly effective for improving the accuracy of license plate recognition models. Furthermore, by performing interpretability studies to understand why our models work, we identify and highlight attack-prone regions in the input character images. In sum, although Nepal's embossed license plate detection systems are vulnerable to malicious attacks, our findings suggest that these systems can be upgraded to close to 100% resilience.

Title: Personalized Guidelines for Design, Implementation and Evaluation of Anti-phishing Interventions. (arXiv:2311.12827v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.12827
• Code URL: null
• Copy Paste: [[2311.12827]] Personalized Guidelines for Design, Implementation and Evaluation of Anti-phishing Interventions(http://arxiv.org/abs/2311.12827)
• Summary:

  Background: Current anti-phishing interventions, which typically involve one-size-fits-all solutions, suffer from limitations such as inadequate usability and poor implementation. Human-centric challenges in anti-phishing technologies remain little understood. Research shows a deficiency in the comprehension of end-user preferences, mental states, and cognitive requirements by developers and practitioners involved in the design, implementation, and evaluation of anti-phishing interventions. Aims: This study addresses the current lack of resources and guidelines for the design, implementation and evaluation of anti-phishing interventions, by presenting personalized guidelines to the developers and practitioners. Method: Through an analysis of 53 academic studies and 16 items of grey literature studies, we systematically identified the challenges and recommendations within the anti-phishing interventions, across different practitioner groups and intervention types. Results: We identified 22 dominant factors at the individual, technical, and organizational levels, that affected the effectiveness of anti-phishing interventions and, accordingly, reported 41 guidelines based on the suggestions and recommendations provided in the studies to improve the outcome of anti-phishing interventions. Conclusions: Our dominant factors can help developers and practitioners enhance their understanding of human-centric, technical and organizational issues in anti-phishing interventions. Our customized guidelines can empower developers and practitioners to counteract phishing attacks.

Title: Hard Label Black Box Node Injection Attack on Graph Neural Networks. (arXiv:2311.13244v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.13244
• Code URL: null
• Copy Paste: [[2311.13244]] Hard Label Black Box Node Injection Attack on Graph Neural Networks(http://arxiv.org/abs/2311.13244)
• Summary:

  While graph neural networks have achieved state-of-the-art performances in many real-world tasks including graph classification and node classification, recent works have demonstrated they are also extremely vulnerable to adversarial attacks. Most previous works have focused on attacking node classification networks under impractical white-box scenarios. In this work, we will propose a non-targeted Hard Label Black Box Node Injection Attack on Graph Neural Networks, which to the best of our knowledge, is the first of its kind. Under this setting, more real world tasks can be studied because our attack assumes no prior knowledge about (1): the model architecture of the GNN we are attacking; (2): the model's gradients; (3): the output logits of the target GNN model. Our attack is based on an existing edge perturbation attack, from which we restrict the optimization process to formulate a node injection attack. In the work, we will evaluate the performance of the attack using three datasets, COIL-DEL, IMDB-BINARY, and NCI1.

Title: Automated generation of attack trees with optimal shape and labelling. (arXiv:2311.13331v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.13331
• Code URL: null
• Copy Paste: [[2311.13331]] Automated generation of attack trees with optimal shape and labelling(http://arxiv.org/abs/2311.13331)
• Summary:

  The problem this article addresses is, given a formal specification of a system, how to produce an attack tree that correctly and clearly describes the ways the system can be attacked. Correctness means that the attacks displayed by the attack tree are indeed attacks in the system; clarity means that the tree is efficient in communicating the attack scenario. To pursue clarity, we introduce an attack-tree generation algorithm that minimises the tree size and the information length of its labels without sacrificing correctness. We achieve this by establishing a connection between the problem of factorising algebraic expressions and the problem of minimising the tree size. Notably, our generation algorithm can handle complex attacks that execute actions in parallel and sequentially. For completeness, we introduce a system model that integrates well with our generation approach, and validate the resulting framework via a running example.

robust

Title: Understanding Data Augmentation from a Robustness Perspective. (arXiv:2311.12800v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12800
• Code URL: null
• Copy Paste: [[2311.12800]] Understanding Data Augmentation from a Robustness Perspective(http://arxiv.org/abs/2311.12800)
• Summary:

  In the realm of visual recognition, data augmentation stands out as a pivotal technique to amplify model robustness. Yet, a considerable number of existing methodologies lean heavily on heuristic foundations, rendering their intrinsic mechanisms ambiguous. This manuscript takes both a theoretical and empirical approach to understanding the phenomenon. Theoretically, we frame the discourse around data augmentation within game theory's constructs. Venturing deeper, our empirical evaluations dissect the intricate mechanisms of emblematic data augmentation strategies, illuminating that these techniques primarily stimulate mid- and high-order game interactions. Beyond the foundational exploration, our experiments span multiple datasets and diverse augmentation techniques, underscoring the universal applicability of our findings. Recognizing the vast array of robustness metrics with intricate correlations, we unveil a streamlined proxy. This proxy not only simplifies robustness assessment but also offers invaluable insights, shedding light on the inherent dynamics of model game interactions and their relation to overarching system robustness. These insights provide a novel lens through which we can re-evaluate model safety and robustness in visual recognition tasks.

Title: LiveChat: Video Comment Generation from Audio-Visual Multimodal Contexts. (arXiv:2311.12826v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12826
• Code URL: null
• Copy Paste: [[2311.12826]] LiveChat: Video Comment Generation from Audio-Visual Multimodal Contexts(http://arxiv.org/abs/2311.12826)
• Summary:

  Live commenting on video, a popular feature of live streaming platforms, enables viewers to engage with the content and share their comments, reactions, opinions, or questions with the streamer or other viewers while watching the video or live stream. It presents a challenging testbed for AI agents, which involves the simultaneous understanding of audio-visual multimodal contexts from live streams and the ability to interact with human viewers through dialogue. As existing live streaming-based comments datasets contain limited categories and lack a diversity, we create a large-scale audio-visual multimodal dialogue dataset to facilitate the development of live commenting technologies. The data is collected from Twitch, with 11 different categories and 575 streamers for a total of 438 hours of video and 3.2 million comments. Moreover, we propose a novel multimodal generation model capable of generating live comments that align with the temporal and spatial events within the video, as well as with the ongoing multimodal dialogue context. Our initial results have demonstrated the effectiveness of the proposed model, providing a robust foundation for further research and practical applications in the field of live video interaction.

Title: Contextualised Out-of-Distribution Detection using Pattern Identication. (arXiv:2311.12855v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12855
• Code URL: null
• Copy Paste: [[2311.12855]] Contextualised Out-of-Distribution Detection using Pattern Identication(http://arxiv.org/abs/2311.12855)
• Summary:

  In this work, we propose CODE, an extension of existing work from the field of explainable AI that identifies class-specific recurring patterns to build a robust Out-of-Distribution (OoD) detection method for visual classifiers. CODE does not require any classifier retraining and is OoD-agnostic, i.e., tuned directly to the training dataset. Crucially, pattern identification allows us to provide images from the In-Distribution (ID) dataset as reference data to provide additional context to the confidence scores. In addition, we introduce a new benchmark based on perturbations of the ID dataset that provides a known and quantifiable measure of the discrepancy between the ID and OoD datasets serving as a reference value for the comparison between OoD detection methods.

Title: Joint Multi-View Collaborative Clustering. (arXiv:2311.12859v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12859
• Code URL: null
• Copy Paste: [[2311.12859]] Joint Multi-View Collaborative Clustering(http://arxiv.org/abs/2311.12859)
• Summary:

  Data is increasingly being collected from multiple sources and described by multiple views. These multi-view data provide richer information than traditional single-view data. Fusing the former for specific tasks is an essential component of multi-view clustering. Since the goal of multi-view clustering algorithms is to discover the common latent structure shared by multiple views, the majority of proposed solutions overlook the advantages of incorporating knowledge derived from horizontal collaboration between multi-view data and the final consensus. To fill this gap, we propose the Joint Multi-View Collaborative Clustering (JMVCC) solution, which involves the generation of basic partitions using Non-negative Matrix Factorization (NMF) and the horizontal collaboration principle, followed by the fusion of these local partitions using ensemble clustering. Furthermore, we propose a weighting method to reduce the risk of negative collaboration (i.e., views with low quality) during the generation and fusion of local partitions. The experimental results, which were obtained using a variety of data sets, demonstrate that JMVCC outperforms other multi-view clustering algorithms and is robust to noisy views.

Title: De-fine: Decomposing and Refining Visual Programs with Auto-Feedback. (arXiv:2311.12890v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12890
• Code URL: null
• Copy Paste: [[2311.12890]] De-fine: Decomposing and Refining Visual Programs with Auto-Feedback(http://arxiv.org/abs/2311.12890)
• Summary:

  Visual programming, a modular and generalizable paradigm, integrates different modules and Python operators to solve various vision-language tasks. Unlike end-to-end models that need task-specific data, it advances in performing visual processing and reasoning in an unsupervised manner. Current visual programming methods generate programs in a single pass for each task where the ability to evaluate and optimize based on feedback, unfortunately, is lacking, which consequentially limits their effectiveness for complex, multi-step problems. Drawing inspiration from benders decomposition, we introduce De-fine, a general framework that automatically decomposes complex tasks into simpler subtasks and refines programs through auto-feedback. This model-agnostic approach can improve logical reasoning performance by integrating the strengths of multiple models. Our experiments across various visual tasks show that De-fine creates more accurate and robust programs, setting new benchmarks in the field.

Title: Robustifying Generalizable Implicit Shape Networks with a Tunable Non-Parametric Model. (arXiv:2311.12967v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12967
• Code URL: null
• Copy Paste: [[2311.12967]] Robustifying Generalizable Implicit Shape Networks with a Tunable Non-Parametric Model(http://arxiv.org/abs/2311.12967)
• Summary:

  Feedforward generalizable models for implicit shape reconstruction from unoriented point cloud present multiple advantages, including high performance and inference speed. However, they still suffer from generalization issues, ranging from underfitting the input point cloud, to misrepresenting samples outside of the training data distribution, or with toplogies unseen at training. We propose here an efficient mechanism to remedy some of these limitations at test time. We combine the inter-shape data prior of the network with an intra-shape regularization prior of a Nystr\"om Kernel Ridge Regression, that we further adapt by fitting its hyperprameters to the current shape. The resulting shape function defined in a shape specific Reproducing Kernel Hilbert Space benefits from desirable stability and efficiency properties and grants a shape adaptive expressiveness-robustness trade-off. We demonstrate the improvement obtained through our method with respect to baselines and the state-of-the-art using synthetic and real data.

Title: Unsupervised Multimodal Surface Registration with Geometric Deep Learning. (arXiv:2311.13022v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.13022
• Code URL: null
• Copy Paste: [[2311.13022]] Unsupervised Multimodal Surface Registration with Geometric Deep Learning(http://arxiv.org/abs/2311.13022)
• Summary:

  This paper introduces GeoMorph, a novel geometric deep-learning framework designed for image registration of cortical surfaces. The registration process consists of two main steps. First, independent feature extraction is performed on each input surface using graph convolutions, generating low-dimensional feature representations that capture important cortical surface characteristics. Subsequently, features are registered in a deep-discrete manner to optimize the overlap of common structures across surfaces by learning displacements of a set of control points. To ensure smooth and biologically plausible deformations, we implement regularization through a deep conditional random field implemented with a recurrent neural network. Experimental results demonstrate that GeoMorph surpasses existing deep-learning methods by achieving improved alignment with smoother deformations. Furthermore, GeoMorph exhibits competitive performance compared to classical frameworks. Such versatility and robustness suggest strong potential for various neuroscience applications.

Title: Camera-Independent Single Image Depth Estimation from Defocus Blur. (arXiv:2311.13045v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.13045
• Code URL: null
• Copy Paste: [[2311.13045]] Camera-Independent Single Image Depth Estimation from Defocus Blur(http://arxiv.org/abs/2311.13045)
• Summary:

  Monocular depth estimation is an important step in many downstream tasks in machine vision. We address the topic of estimating monocular depth from defocus blur which can yield more accurate results than the semantic based depth estimation methods. The existing monocular depth from defocus techniques are sensitive to the particular camera that the images are taken from. We show how several camera-related parameters affect the defocus blur using optical physics equations and how they make the defocus blur depend on these parameters. The simple correction procedure we propose can alleviate this problem which does not require any retraining of the original model. We created a synthetic dataset which can be used to test the camera independent performance of depth from defocus blur models. We evaluate our model on both synthetic and real datasets (DDFF12 and NYU depth V2) obtained with different cameras and show that our methods are significantly more robust to the changes of cameras. Code: https://github.com/sleekEagle/defocus_camind.git

Title: Stable Unlearnable Example: Enhancing the Robustness of Unlearnable Examples via Stable Error-Minimizing Noise. (arXiv:2311.13091v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.13091
• Code URL: null
• Copy Paste: [[2311.13091]] Stable Unlearnable Example: Enhancing the Robustness of Unlearnable Examples via Stable Error-Minimizing Noise(http://arxiv.org/abs/2311.13091)
• Summary:

  The open source of large amounts of image data promotes the development of deep learning techniques. Along with this comes the privacy risk of these open-source image datasets being exploited by unauthorized third parties to train deep learning models for commercial or illegal purposes. To avoid the abuse of public data, a poisoning-based technique, the unlearnable example, is proposed to significantly degrade the generalization performance of models by adding a kind of imperceptible noise to the data. To further enhance its robustness against adversarial training, existing works leverage iterative adversarial training on both the defensive noise and the surrogate model. However, it still remains unknown whether the robustness of unlearnable examples primarily comes from the effect of enhancement in the surrogate model or the defensive noise. Observing that simply removing the adversarial noise on the training process of the defensive noise can improve the performance of robust unlearnable examples, we identify that solely the surrogate model's robustness contributes to the performance. Furthermore, we found a negative correlation exists between the robustness of defensive noise and the protection performance, indicating defensive noise's instability issue. Motivated by this, to further boost the robust unlearnable example, we introduce stable error-minimizing noise (SEM), which trains the defensive noise against random perturbation instead of the time-consuming adversarial perturbation to improve the stability of defensive noise. Through extensive experiments, we demonstrate that SEM achieves a new state-of-the-art performance on CIFAR-10, CIFAR-100, and ImageNet Subset in terms of both effectiveness and efficiency. The code is available at https://github.com/liuyixin-louis/Stable-Unlearnable-Example.

Title: Toward Robust Imperceptible Perturbation against Unauthorized Text-to-image Diffusion-based Synthesis. (arXiv:2311.13127v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.13127
• Code URL: null
• Copy Paste: [[2311.13127]] Toward Robust Imperceptible Perturbation against Unauthorized Text-to-image Diffusion-based Synthesis(http://arxiv.org/abs/2311.13127)
• Summary:

  Text-to-image diffusion models allow seamless generation of personalized images from scant reference photos. Yet, these tools, in the wrong hands, can fabricate misleading or harmful content, endangering individuals. To address this problem, existing poisoning-based approaches perturb user images in an imperceptible way to render them "unlearnable" from malicious uses. We identify two limitations of these defending approaches: i) sub-optimal due to the hand-crafted heuristics for solving the intractable bilevel optimization and ii) lack of robustness against simple data transformations like Gaussian filtering. To solve these challenges, we propose MetaCloak, which solves the bi-level poisoning problem with a meta-learning framework with an additional transformation sampling process to craft transferable and robust perturbation. Specifically, we employ a pool of surrogate diffusion models to craft transferable and model-agnostic perturbation. Furthermore, by incorporating an additional transformation process, we design a simple denoising-error maximization loss that is sufficient for causing transformation-robust semantic distortion and degradation in a personalized generation. Extensive experiments on the VGGFace2 and CelebA-HQ datasets show that MetaCloak outperforms existing approaches. Notably, MetaCloak can successfully fool online training services like Replicate, in a black-box manner, demonstrating the effectiveness of MetaCloak in real-world scenarios. Our code is available at https://github.com/liuyixin-louis/MetaCloak.

Title: P2RBox: A Single Point is All You Need for Oriented Object Detection. (arXiv:2311.13128v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.13128
• Code URL: null
• Copy Paste: [[2311.13128]] P2RBox: A Single Point is All You Need for Oriented Object Detection(http://arxiv.org/abs/2311.13128)
• Summary:

  Oriented object detection, a specialized subfield in computer vision, finds applications across diverse scenarios, excelling particularly when dealing with objects of arbitrary orientations. Conversely, point annotation, which treats objects as single points, offers a cost-effective alternative to rotated and horizontal bounding boxes but sacrifices performance due to the loss of size and orientation information. In this study, we introduce the P2RBox network, which leverages point annotations and a mask generator to create mask proposals, followed by filtration through our Inspector Module and Constrainer Module. This process selects high-quality masks, which are subsequently converted into rotated box annotations for training a fully supervised detector. Specifically, we've thoughtfully crafted an Inspector Module rooted in multi-instance learning principles to evaluate the semantic score of masks. We've also proposed a more robust mask quality assessment in conjunction with the Constrainer Module. Furthermore, we've introduced a Symmetry Axis Estimation (SAE) Module inspired by the spectral theorem for symmetric matrices to transform the top-performing mask proposal into rotated bounding boxes. P2RBox performs well with three fully supervised rotated object detectors: RetinaNet, Rotated FCOS, and Oriented R-CNN. By combining with Oriented R-CNN, P2RBox achieves 62.26% on DOTA-v1.0 test dataset. As far as we know, this is the first attempt at training an oriented object detector with point supervision.

Title: Learning to Complement with Multiple Humans (LECOMH): Integrating Multi-rater and Noisy-Label Learning into Human-AI Collaboration. (arXiv:2311.13172v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.13172
• Code URL: null
• Copy Paste: [[2311.13172]] Learning to Complement with Multiple Humans (LECOMH): Integrating Multi-rater and Noisy-Label Learning into Human-AI Collaboration(http://arxiv.org/abs/2311.13172)
• Summary:

  The advent of learning with noisy labels (LNL), multi-rater learning, and human-AI collaboration has revolutionised the development of robust classifiers, enabling them to address the challenges posed by different types of data imperfections and complex decision processes commonly encountered in real-world applications. While each of these methodologies has individually made significant strides in addressing their unique challenges, the development of techniques that can simultaneously tackle these three problems remains underexplored. This paper addresses this research gap by integrating noisy-label learning, multi-rater learning, and human-AI collaboration with new benchmarks and the innovative Learning to Complement with Multiple Humans (LECOMH) approach. LECOMH optimises the level of human collaboration during testing, aiming to optimise classification accuracy while minimising collaboration costs that vary from 0 to M, where M is the maximum number of human collaborators. We quantitatively compare LECOMH with leading human-AI collaboration methods using our proposed benchmarks. LECOMH consistently outperforms the competition, with accuracy improving as collaboration costs increase. Notably, LECOMH is the only method enhancing human labeller performance across all benchmarks.

Title: Applications of Spiking Neural Networks in Visual Place Recognition. (arXiv:2311.13186v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.13186
• Code URL: https://github.com/qvpr/vprsnn
• Copy Paste: [[2311.13186]] Applications of Spiking Neural Networks in Visual Place Recognition(http://arxiv.org/abs/2311.13186)
• Summary:

  In robotics, Spiking Neural Networks (SNNs) are increasingly recognized for their largely-unrealized potential energy efficiency and low latency particularly when implemented on neuromorphic hardware. Our paper highlights three advancements for SNNs in Visual Place Recognition (VPR). First, we propose Modular SNNs, where each SNN represents a set of non-overlapping geographically distinct places, enabling scalable networks for large environments. Secondly, we present Ensembles of Modular SNNs, where multiple networks represent the same place, significantly enhancing accuracy compared to single-network models. Our SNNs are compact and small, comprising only 1500 neurons and 474k synapses, which makes them ideally suited for ensembling due to this small size. Lastly, we investigate the role of sequence matching in SNN-based VPR, a technique where consecutive images are used to refine place recognition. We analyze the responsiveness of SNNs to ensembling and sequence matching compared to other VPR techniques. Our contributions highlight the viability of SNNs for VPR, offering scalable and robust solutions, paving the way for their application in various energy-sensitive robotic tasks.

Title: High-Quality Face Caricature via Style Translation. (arXiv:2311.13338v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.13338
• Code URL: null
• Copy Paste: [[2311.13338]] High-Quality Face Caricature via Style Translation(http://arxiv.org/abs/2311.13338)
• Summary:

  Caricature is an exaggerated form of artistic portraiture that accentuates unique yet subtle characteristics of human faces. Recently, advancements in deep end-to-end techniques have yielded encouraging outcomes in capturing both style and elevated exaggerations in creating face caricatures. Most of these approaches tend to produce cartoon-like results that could be more practical for real-world applications. In this study, we proposed a high-quality, unpaired face caricature method that is appropriate for use in the real world and uses computer vision techniques and GAN models. We attain the exaggeration of facial features and the stylization of appearance through a two-step process: Face caricature generation and face caricature projection. The face caricature generation step creates new caricature face datasets from real images and trains a generative model using the real and newly created caricature datasets. The Face caricature projection employs an encoder trained with real and caricature faces with the pretrained generator to project real and caricature faces. We perform an incremental facial exaggeration from the real image to the caricature faces using the encoder and generator's latent space. Our projection preserves the facial identity, attributes, and expressions from the input image. Also, it accounts for facial occlusions, such as reading glasses or sunglasses, to enhance the robustness of our model. Furthermore, we conducted a comprehensive comparison of our approach with various state-of-the-art face caricature methods, highlighting our process's distinctiveness and exceptional realism.

Title: Depth-Regularized Optimization for 3D Gaussian Splatting in Few-Shot Images. (arXiv:2311.13398v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.13398
• Code URL: null
• Copy Paste: [[2311.13398]] Depth-Regularized Optimization for 3D Gaussian Splatting in Few-Shot Images(http://arxiv.org/abs/2311.13398)
• Summary:

  In this paper, we present a method to optimize Gaussian splatting with a limited number of images while avoiding overfitting. Representing a 3D scene by combining numerous Gaussian splats has yielded outstanding visual quality. However, it tends to overfit the training views when only a small number of images are available. To address this issue, we introduce a dense depth map as a geometry guide to mitigate overfitting. We obtained the depth map using a pre-trained monocular depth estimation model and aligning the scale and offset using sparse COLMAP feature points. The adjusted depth aids in the color-based optimization of 3D Gaussian splatting, mitigating floating artifacts, and ensuring adherence to geometric constraints. We verify the proposed method on the NeRF-LLFF dataset with varying numbers of few images. Our approach demonstrates robust geometry compared to the original method that relies solely on images.

Title: Leveraging CNNs and Ensemble Learning for Automated Disaster Image Classification. (arXiv:2311.13531v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.13531
• Code URL: null
• Copy Paste: [[2311.13531]] Leveraging CNNs and Ensemble Learning for Automated Disaster Image Classification(http://arxiv.org/abs/2311.13531)
• Summary:

  Natural disasters act as a serious threat globally, requiring effective and efficient disaster management and recovery. This paper focuses on classifying natural disaster images using Convolutional Neural Networks (CNNs). Multiple CNN architectures were built and trained on a dataset containing images of earthquakes, floods, wildfires, and volcanoes. A stacked CNN ensemble approach proved to be the most effective, achieving 95% accuracy and an F1 score going up to 0.96 for individual classes. Tuning hyperparameters of individual models for optimization was critical to maximize the models' performance. The stacking of CNNs with XGBoost acting as the meta-model utilizes the strengths of the CNN and ResNet models to improve the overall accuracy of the classification. Results obtained from the models illustrated the potency of CNN-based models for automated disaster image classification. This lays the foundation for expanding these techniques to build robust systems for disaster response, damage assessment, and recovery management.

Title: GAIA: a benchmark for General AI Assistants. (arXiv:2311.12983v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.12983
• Code URL: null
• Copy Paste: [[2311.12983]] GAIA: a benchmark for General AI Assistants(http://arxiv.org/abs/2311.12983)
• Summary:

  We introduce GAIA, a benchmark for General AI Assistants that, if solved, would represent a milestone in AI research. GAIA proposes real-world questions that require a set of fundamental abilities such as reasoning, multi-modality handling, web browsing, and generally tool-use proficiency. GAIA questions are conceptually simple for humans yet challenging for most advanced AIs: we show that human respondents obtain 92\% vs. 15\% for GPT-4 equipped with plugins. This notable performance disparity contrasts with the recent trend of LLMs outperforming humans on tasks requiring professional skills in e.g. law or chemistry. GAIA's philosophy departs from the current trend in AI benchmarks suggesting to target tasks that are ever more difficult for humans. We posit that the advent of Artificial General Intelligence (AGI) hinges on a system's capability to exhibit similar robustness as the average human does on such questions. Using GAIA's methodology, we devise 466 questions and their answer. We release our questions while retaining answers to 300 of them to power a leader-board available at https://huggingface.co/gaia-benchmark.

Title: Nonlinear System Identification of Swarm of UAVs Using Deep Learning Methods. (arXiv:2311.12906v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.12906
• Code URL: null
• Copy Paste: [[2311.12906]] Nonlinear System Identification of Swarm of UAVs Using Deep Learning Methods(http://arxiv.org/abs/2311.12906)
• Summary:

  This study designs and evaluates multiple nonlinear system identification techniques for modeling the UAV swarm system in planar space. learning methods such as RNNs, CNNs, and Neural ODE are explored and compared. The objective is to forecast future swarm trajectories by accurately approximating the nonlinear dynamics of the swarm model. The modeling process is performed using both transient and steady-state data from swarm simulations. Results show that the combination of Neural ODE with a well-trained model using transient data is robust for varying initial conditions and outperforms other learning methods in accurately predicting swarm stability.

biometric

Title: Comprehensive Survey: Biometric User Authentication Application, Evaluation, and Discussion. (arXiv:2311.13416v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.13416
• Code URL: null
• Copy Paste: [[2311.13416]] Comprehensive Survey: Biometric User Authentication Application, Evaluation, and Discussion(http://arxiv.org/abs/2311.13416)
• Summary:

  This paper conducts an extensive review of biometric user authentication literature, addressing three primary research questions: (1) commonly used biometric traits and their suitability for specific applications, (2) performance factors such as security, convenience, and robustness, and potential countermeasures against cyberattacks, and (3) factors affecting biometric system accuracy and po-tential improvements. Our analysis delves into physiological and behavioral traits, exploring their pros and cons. We discuss factors influencing biometric system effectiveness and highlight areas for enhancement. Our study differs from previous surveys by extensively examining biometric traits, exploring various application domains, and analyzing measures to mitigate cyberattacks. This paper aims to inform researchers and practitioners about the biometric authentication landscape and guide future advancements.

steal

extraction

Title: MatGD: Materials Graph Digitizer. (arXiv:2311.12806v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12806
• Code URL: null
• Copy Paste: [[2311.12806]] MatGD: Materials Graph Digitizer(http://arxiv.org/abs/2311.12806)
• Summary:

  We have developed MatGD (Material Graph Digitizer), which is a tool for digitizing a data line from scientific graphs. The algorithm behind the tool consists of four steps: (1) identifying graphs within subfigures, (2) separating axes and data sections, (3) discerning the data lines by eliminating irrelevant graph objects and matching with the legend, and (4) data extraction and saving. From the 62,534 papers in the areas of batteries, catalysis, and MOFs, 501,045 figures were mined. Remarkably, our tool showcased performance with over 99% accuracy in legend marker and text detection. Moreover, its capability for data line separation stood at 66%, which is much higher compared to other existing figure mining tools. We believe that this tool will be integral to collecting both past and future data from publications, and these data can be used to train various machine learning models that can enhance material predictions and new materials discovery.

Title: ViStruct: Visual Structural Knowledge Extraction via Curriculum Guided Code-Vision Representation. (arXiv:2311.13258v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.13258
• Code URL: null
• Copy Paste: [[2311.13258]] ViStruct: Visual Structural Knowledge Extraction via Curriculum Guided Code-Vision Representation(http://arxiv.org/abs/2311.13258)
• Summary:

  State-of-the-art vision-language models (VLMs) still have limited performance in structural knowledge extraction, such as relations between objects. In this work, we present ViStruct, a training framework to learn VLMs for effective visual structural knowledge extraction. Two novel designs are incorporated. First, we propose to leverage the inherent structure of programming language to depict visual structural information. This approach enables explicit and consistent representation of visual structural information of multiple granularities, such as concepts, relations, and events, in a well-organized structured format. Second, we introduce curriculum-based learning for VLMs to progressively comprehend visual structures, from fundamental visual concepts to intricate event structures. Our intuition is that lower-level knowledge may contribute to complex visual structure understanding. Furthermore, we compile and release a collection of datasets tailored for visual structural knowledge extraction. We adopt a weakly-supervised approach to directly generate visual event structures from captions for ViStruct training, capitalizing on abundant image-caption pairs from the web. In experiments, we evaluate ViStruct on visual structure prediction tasks, demonstrating its effectiveness in improving the understanding of visual structures. The code is public at \url{https://github.com/Yangyi-Chen/vi-struct}.

Title: MRGazer: Decoding Eye Gaze Points from Functional Magnetic Resonance Imaging in Individual Space. (arXiv:2311.13372v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.13372
• Code URL: null
• Copy Paste: [[2311.13372]] MRGazer: Decoding Eye Gaze Points from Functional Magnetic Resonance Imaging in Individual Space(http://arxiv.org/abs/2311.13372)
• Summary:

  Eye-tracking research has proven valuable in understanding numerous cognitive functions. Recently, Frey et al. provided an exciting deep learning method for learning eye movements from fMRI data. However, it needed to co-register fMRI into standard space to obtain eyeballs masks, and thus required additional templates and was time consuming. To resolve this issue, in this paper, we propose a framework named MRGazer for predicting eye gaze points from fMRI in individual space. The MRGazer consisted of eyeballs extraction module and a residual network-based eye gaze prediction. Compared to the previous method, the proposed framework skips the fMRI co-registration step, simplifies the processing protocol and achieves end-to-end eye gaze regression. The proposed method achieved superior performance in a variety of eye movement tasks than the co-registration-based method, and delivered objective results within a shorter time (~ 0.02 Seconds for each volume) than prior method (~0.3 Seconds for each volume).

Title: Unsupervised Graph Attention Autoencoder for Attributed Networks using K-means Loss. (arXiv:2311.12986v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.12986
• Code URL: null
• Copy Paste: [[2311.12986]] Unsupervised Graph Attention Autoencoder for Attributed Networks using K-means Loss(http://arxiv.org/abs/2311.12986)
• Summary:

  Multimodal Sentiment Analysis (MSA) has recently become a centric research direction for many real-world applications. This proliferation is due to the fact that opinions are central to almost all human activities and are key influencers of our behaviors. In addition, the recent deployment of Deep Learning-based (DL) models has proven their high efficiency for a wide range of Western languages. In contrast, Arabic DL-based multimodal sentiment analysis (MSA) is still in its infantile stage due, mainly, to the lack of standard datasets. % The contribution In this paper, our investigation is twofold. First, we design a pipeline that helps building our Arabic Multimodal dataset leveraging both state-of-the-art transformers and feature extraction tools within word alignment techniques. Thereafter, we validate our dataset using state-of-the-art transformer-based model dealing with multimodality. Despite the small size of the outcome dataset, experiments show that Arabic multimodality is very promising.

Title: Ball Mill Fault Prediction Based on Deep Convolutional Auto-Encoding Network. (arXiv:2311.13571v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.13571
• Code URL: null
• Copy Paste: [[2311.13571]] Ball Mill Fault Prediction Based on Deep Convolutional Auto-Encoding Network(http://arxiv.org/abs/2311.13571)
• Summary:

  Ball mills play a critical role in modern mining operations, making their bearing failures a significant concern due to the potential loss of production efficiency and economic consequences. This paper presents an anomaly detection method based on Deep Convolutional Auto-encoding Neural Networks (DCAN) for addressing the issue of ball mill bearing fault detection. The proposed approach leverages vibration data collected during normal operation for training, overcoming challenges such as labeling issues and data imbalance often encountered in supervised learning methods. DCAN includes the modules of convolutional feature extraction and transposed convolutional feature reconstruction, demonstrating exceptional capabilities in signal processing and feature extraction. Additionally, the paper describes the practical deployment of the DCAN-based anomaly detection model for bearing fault detection, utilizing data from the ball mill bearings of Wuhan Iron & Steel Resources Group and fault data from NASA's bearing vibration dataset. Experimental results validate the DCAN model's reliability in recognizing fault vibration patterns. This method holds promise for enhancing bearing fault detection efficiency, reducing production interruptions, and lowering maintenance costs.

membership infer

federate

Title: Towards Hetero-Client Federated Multi-Task Learning. (arXiv:2311.13250v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.13250
• Code URL: null
• Copy Paste: [[2311.13250]] Towards Hetero-Client Federated Multi-Task Learning(http://arxiv.org/abs/2311.13250)
• Summary:

  Federated Learning (FL) enables joint training across distributed clients using their local data privately. Federated Multi-Task Learning (FMTL) builds on FL to handle multiple tasks, assuming model congruity that identical model architecture is deployed in each client. To relax this assumption and thus extend real-world applicability, we introduce a novel problem setting, Hetero-Client Federated Multi-Task Learning (HC-FMTL), to accommodate diverse task setups. The main challenge of HC-FMTL is the model incongruity issue that invalidates conventional aggregation methods. It also escalates the difficulties in accurate model aggregation to deal with data and task heterogeneity inherent in FMTL. To address these challenges, we propose the FedHCA$^2$ framework, which allows for federated training of personalized models by modeling relationships among heterogeneous clients. Drawing on our theoretical insights into the difference between multi-task and federated optimization, we propose the Hyper Conflict-Averse Aggregation scheme to mitigate conflicts during encoder updates. Additionally, inspired by task interaction in MTL, the Hyper Cross Attention Aggregation scheme uses layer-wise cross attention to enhance decoder interactions while alleviating model incongruity. Moreover, we employ learnable Hyper Aggregation Weights for each client to customize personalized parameter updates. Extensive experiments demonstrate the superior performance of FedHCA$^2$ in various HC-FMTL scenarios compared to representative methods. Our code will be made publicly available.

Title: FedFN: Feature Normalization for Alleviating Data Heterogeneity Problem in Federated Learning. (arXiv:2311.13267v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.13267
• Code URL: null
• Copy Paste: [[2311.13267]] FedFN: Feature Normalization for Alleviating Data Heterogeneity Problem in Federated Learning(http://arxiv.org/abs/2311.13267)
• Summary:

  Federated Learning (FL) is a collaborative method for training models while preserving data privacy in decentralized settings. However, FL encounters challenges related to data heterogeneity, which can result in performance degradation. In our study, we observe that as data heterogeneity increases, feature representation in the FedAVG model deteriorates more significantly compared to classifier weight. Additionally, we observe that as data heterogeneity increases, the gap between higher feature norms for observed classes, obtained from local models, and feature norms of unobserved classes widens, in contrast to the behavior of classifier weight norms. This widening gap extends to encompass the feature norm disparities between local and the global models. To address these issues, we introduce Federated Averaging with Feature Normalization Update (FedFN), a straightforward learning method. We demonstrate the superior performance of FedFN through extensive experiments, even when applied to pretrained ResNet18. Subsequently, we confirm the applicability of FedFN to foundation models.

Title: Have Your Cake and Eat It Too: Toward Efficient and Accurate Split Federated Learning. (arXiv:2311.13163v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.13163
• Code URL: null
• Copy Paste: [[2311.13163]] Have Your Cake and Eat It Too: Toward Efficient and Accurate Split Federated Learning(http://arxiv.org/abs/2311.13163)
• Summary:

  Due to its advantages in resource constraint scenarios, Split Federated Learning (SFL) is promising in AIoT systems. However, due to data heterogeneity and stragglers, SFL suffers from the challenges of low inference accuracy and low efficiency. To address these issues, this paper presents a novel SFL approach, named Sliding Split Federated Learning (S$^2$FL), which adopts an adaptive sliding model split strategy and a data balance-based training mechanism. By dynamically dispatching different model portions to AIoT devices according to their computing capability, S$^2$FL can alleviate the low training efficiency caused by stragglers. By combining features uploaded by devices with different data distributions to generate multiple larger batches with a uniform distribution for back-propagation, S$^2$FL can alleviate the performance degradation caused by data heterogeneity. Experimental results demonstrate that, compared to conventional SFL, S$^2$FL can achieve up to 16.5\% inference accuracy improvement and 3.54X training acceleration.

Title: AdaptiveFL: Adaptive Heterogeneous Federated Learning for Resource-Constrained AIoT Systems. (arXiv:2311.13166v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.13166
• Code URL: null
• Copy Paste: [[2311.13166]] AdaptiveFL: Adaptive Heterogeneous Federated Learning for Resource-Constrained AIoT Systems(http://arxiv.org/abs/2311.13166)
• Summary:

  Although Federated Learning (FL) is promising to enable collaborative learning among Artificial Intelligence of Things (AIoT) devices, it suffers from the problem of low classification performance due to various heterogeneity factors (e.g., computing capacity, memory size) of devices and uncertain operating environments. To address these issues, this paper introduces an effective FL approach named AdaptiveFL based on a novel fine-grained width-wise model pruning strategy, which can generate various heterogeneous local models for heterogeneous AIoT devices. By using our proposed reinforcement learning-based device selection mechanism, AdaptiveFL can adaptively dispatch suitable heterogeneous models to corresponding AIoT devices on the fly based on their available resources for local training. Experimental results show that, compared to state-of-the-art methods, AdaptiveFL can achieve up to 16.83% inference improvements for both IID and non-IID scenarios.

Title: MergeSFL: Split Federated Learning with Feature Merging and Batch Size Regulation. (arXiv:2311.13348v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.13348
• Code URL: null
• Copy Paste: [[2311.13348]] MergeSFL: Split Federated Learning with Feature Merging and Batch Size Regulation(http://arxiv.org/abs/2311.13348)
• Summary:

  Recently, federated learning (FL) has emerged as a popular technique for edge AI to mine valuable knowledge in edge computing (EC) systems. To mitigate the computing/communication burden on resource-constrained workers and protect model privacy, split federated learning (SFL) has been released by integrating both data and model parallelism. Despite resource limitations, SFL still faces two other critical challenges in EC, i.e., statistical heterogeneity and system heterogeneity. To address these challenges, we propose a novel SFL framework, termed MergeSFL, by incorporating feature merging and batch size regulation in SFL. Concretely, feature merging aims to merge the features from workers into a mixed feature sequence, which is approximately equivalent to the features derived from IID data and is employed to promote model accuracy. While batch size regulation aims to assign diverse and suitable batch sizes for heterogeneous workers to improve training efficiency. Moreover, MergeSFL explores to jointly optimize these two strategies upon their coupled relationship to better enhance the performance of SFL. Extensive experiments are conducted on a physical platform with 80 NVIDIA Jetson edge devices, and the experimental results show that MergeSFL can improve the final model accuracy by 5.82% to 26.22%, with a speedup by about 1.74x to 4.14x, compared to the baselines.

fair

interpretability

Title: Combatting Human Trafficking in the Cyberspace: A Natural Language Processing-Based Methodology to Analyze the Language in Online Advertisements. (arXiv:2311.13118v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.13118
• Code URL: null
• Copy Paste: [[2311.13118]] Combatting Human Trafficking in the Cyberspace: A Natural Language Processing-Based Methodology to Analyze the Language in Online Advertisements(http://arxiv.org/abs/2311.13118)
• Summary:

  This project tackles the pressing issue of human trafficking in online C2C marketplaces through advanced Natural Language Processing (NLP) techniques. We introduce a novel methodology for generating pseudo-labeled datasets with minimal supervision, serving as a rich resource for training state-of-the-art NLP models. Focusing on tasks like Human Trafficking Risk Prediction (HTRP) and Organized Activity Detection (OAD), we employ cutting-edge Transformer models for analysis. A key contribution is the implementation of an interpretability framework using Integrated Gradients, providing explainable insights crucial for law enforcement. This work not only fills a critical gap in the literature but also offers a scalable, machine learning-driven approach to combat human exploitation online. It serves as a foundation for future research and practical applications, emphasizing the role of machine learning in addressing complex social issues.

explainability

Title: Explaining high-dimensional text classifiers. (arXiv:2311.13454v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.13454
• Code URL: null
• Copy Paste: [[2311.13454]] Explaining high-dimensional text classifiers(http://arxiv.org/abs/2311.13454)
• Summary:

  Explainability has become a valuable tool in the last few years, helping humans better understand AI-guided decisions. However, the classic explainability tools are sometimes quite limited when considering high-dimensional inputs and neural network classifiers. We present a new explainability method using theoretically proven high-dimensional properties in neural network classifiers. We present two usages of it: 1) On the classical sentiment analysis task for the IMDB reviews dataset, and 2) our Malware-Detection task for our PowerShell scripts dataset.

Title: Labeling Neural Representations with Inverse Recognition. (arXiv:2311.13594v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.13594
• Code URL: null
• Copy Paste: [[2311.13594]] Labeling Neural Representations with Inverse Recognition(http://arxiv.org/abs/2311.13594)
• Summary:

  Deep Neural Networks (DNNs) demonstrated remarkable capabilities in learning complex hierarchical data representations, but the nature of these representations remains largely unknown. Existing global explainability methods, such as Network Dissection, face limitations such as reliance on segmentation masks, lack of statistical significance testing, and high computational demands. We propose Inverse Recognition (INVERT), a scalable approach for connecting learned representations with human-understandable concepts by leveraging their capacity to discriminate between these concepts. In contrast to prior work, INVERT is capable of handling diverse types of neurons, exhibits less computational complexity, and does not rely on the availability of segmentation masks. Moreover, INVERT provides an interpretable metric assessing the alignment between the representation and its corresponding explanation and delivering a measure of statistical significance, emphasizing its utility and credibility. We demonstrate the applicability of INVERT in various scenarios, including the identification of representations affected by spurious correlations, and the interpretation of the hierarchical structure of decision-making within the models.

watermark

diffusion

Title: CopyScope: Model-level Copyright Infringement Quantification in the Diffusion Workflow. (arXiv:2311.12847v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12847
• Code URL: null
• Copy Paste: [[2311.12847]] CopyScope: Model-level Copyright Infringement Quantification in the Diffusion Workflow(http://arxiv.org/abs/2311.12847)
• Summary:

  Web-based AI image generation has become an innovative art form that can generate novel artworks with the rapid development of the diffusion model. However, this new technique brings potential copyright infringement risks as it may incorporate the existing artworks without the owners' consent. Copyright infringement quantification is the primary and challenging step towards AI-generated image copyright traceability. Previous work only focused on data attribution from the training data perspective, which is unsuitable for tracing and quantifying copyright infringement in practice because of the following reasons: (1) the training datasets are not always available in public; (2) the model provider is the responsible party, not the image. Motivated by this, in this paper, we propose CopyScope, a new framework to quantify the infringement of AI-generated images from the model level. We first rigorously identify pivotal components within the AI image generation pipeline. Then, we propose to take advantage of Fr\'echet Inception Distance (FID) to effectively capture the image similarity that fits human perception naturally. We further propose the FID-based Shapley algorithm to evaluate the infringement contribution among models. Extensive experiments demonstrate that our work not only reveals the intricacies of infringement quantification but also effectively depicts the infringing models quantitatively, thus promoting accountability in AI image-generation tasks.

Title: Fine-Grained Open Domain Image Animation with Motion Guidance. (arXiv:2311.12886v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12886
• Code URL: null
• Copy Paste: [[2311.12886]] Fine-Grained Open Domain Image Animation with Motion Guidance(http://arxiv.org/abs/2311.12886)
• Summary:

  Image animation is a key task in computer vision which aims to generate dynamic visual content from static image. Recent image animation methods employ neural based rendering technique to generate realistic animations. Despite these advancements, achieving fine-grained and controllable image animation guided by text remains challenging, particularly for open-domain images captured in diverse real environments. In this paper, we introduce an open domain image animation method that leverages the motion prior of video diffusion model. Our approach introduces targeted motion area guidance and motion strength guidance, enabling precise control the movable area and its motion speed. This results in enhanced alignment between the animated visual elements and the prompting text, thereby facilitating a fine-grained and interactive animation generation process for intricate motion sequences. We validate the effectiveness of our method through rigorous experiments on an open-domain dataset, with the results showcasing its superior performance. The source code and model will be made publicly available upon publication.

Title: Text-Guided Texturing by Synchronized Multi-View Diffusion. (arXiv:2311.12891v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12891
• Code URL: null
• Copy Paste: [[2311.12891]] Text-Guided Texturing by Synchronized Multi-View Diffusion(http://arxiv.org/abs/2311.12891)
• Summary:

  This paper introduces a novel approach to synthesize texture to dress up a given 3D object, given a text prompt. Based on the pretrained text-to-image (T2I) diffusion model, existing methods usually employ a project-and-inpaint approach, in which a view of the given object is first generated and warped to another view for inpainting. But it tends to generate inconsistent texture due to the asynchronous diffusion of multiple views. We believe such asynchronous diffusion and insufficient information sharing among views are the root causes of the inconsistent artifact. In this paper, we propose a synchronized multi-view diffusion approach that allows the diffusion processes from different views to reach a consensus of the generated content early in the process, and hence ensures the texture consistency. To synchronize the diffusion, we share the denoised content among different views in each denoising step, specifically blending the latent content in the texture domain from views with overlap. Our method demonstrates superior performance in generating consistent, seamless, highly detailed textures, comparing to state-of-the-art methods.

Title: Diffusion Model Alignment Using Direct Preference Optimization. (arXiv:2311.12908v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12908
• Code URL: null
• Copy Paste: [[2311.12908]] Diffusion Model Alignment Using Direct Preference Optimization(http://arxiv.org/abs/2311.12908)
• Summary:

  Large language models (LLMs) are fine-tuned using human comparison data with Reinforcement Learning from Human Feedback (RLHF) methods to make them better aligned with users' preferences. In contrast to LLMs, human preference learning has not been widely explored in text-to-image diffusion models; the best existing approach is to fine-tune a pretrained model using carefully curated high quality images and captions to improve visual appeal and text alignment. We propose Diffusion-DPO, a method to align diffusion models to human preferences by directly optimizing on human comparison data. Diffusion-DPO is adapted from the recently developed Direct Preference Optimization (DPO), a simpler alternative to RLHF which directly optimizes a policy that best satisfies human preferences under a classification objective. We re-formulate DPO to account for a diffusion model notion of likelihood, utilizing the evidence lower bound to derive a differentiable objective. Using the Pick-a-Pic dataset of 851K crowdsourced pairwise preferences, we fine-tune the base model of the state-of-the-art Stable Diffusion XL (SDXL)-1.0 model with Diffusion-DPO. Our fine-tuned base model significantly outperforms both base SDXL-1.0 and the larger SDXL-1.0 model consisting of an additional refinement model in human evaluation, improving visual appeal and prompt alignment. We also develop a variant that uses AI feedback and has comparable performance to training on human preferences, opening the door for scaling of diffusion model alignment methods.

Title: Innovative Horizons in Aerial Imagery: LSKNet Meets DiffusionDet for Advanced Object Detection. (arXiv:2311.12956v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12956
• Code URL: null
• Copy Paste: [[2311.12956]] Innovative Horizons in Aerial Imagery: LSKNet Meets DiffusionDet for Advanced Object Detection(http://arxiv.org/abs/2311.12956)
• Summary:

  In the realm of aerial image analysis, object detection plays a pivotal role, with significant implications for areas such as remote sensing, urban planning, and disaster management. This study addresses the inherent challenges in this domain, notably the detection of small objects, managing densely packed elements, and accounting for diverse orientations. We present an in-depth evaluation of an object detection model that integrates the Large Selective Kernel Network (LSKNet)as its backbone with the DiffusionDet head, utilizing the iSAID dataset for empirical analysis. Our approach encompasses the introduction of novel methodologies and extensive ablation studies. These studies critically assess various aspects such as loss functions, box regression techniques, and classification strategies to refine the model's precision in object detection. The paper details the experimental application of the LSKNet backbone in synergy with the DiffusionDet heads, a combination tailored to meet the specific challenges in aerial image object detection. The findings of this research indicate a substantial enhancement in the model's performance, especially in the accuracy-time tradeoff. The proposed model achieves a mean average precision (MAP) of approximately 45.7%, which is a significant improvement, outperforming the RCNN model by 4.7% on the same dataset. This advancement underscores the effectiveness of the proposed modifications and sets a new benchmark in aerial image analysis, paving the way for more accurate and efficient object detection methodologies. The code is publicly available at https://github.com/SashaMatsun/LSKDiffDet

Title: SD-NAE: Generating Natural Adversarial Examples with Stable Diffusion. (arXiv:2311.12981v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12981
• Code URL: null
• Copy Paste: [[2311.12981]] SD-NAE: Generating Natural Adversarial Examples with Stable Diffusion(http://arxiv.org/abs/2311.12981)
• Summary:

  Robustly evaluating deep learning image classifiers is challenging due to some limitations of standard datasets. Natural Adversarial Examples (NAEs), arising naturally from the environment and capable of deceiving classifiers, are instrumental in identifying vulnerabilities in trained models. Existing works collect such NAEs by filtering from a huge set of real images, a process that is passive and lacks control. In this work, we propose to actively synthesize NAEs with the state-of-the-art Stable Diffusion. Specifically, our method formulates a controlled optimization process, where we perturb the token embedding that corresponds to a specified class to synthesize NAEs. The generation is guided by the gradient of loss from the target classifier so that the created image closely mimics the ground-truth class yet fools the classifier. Named SD-NAE (Stable Diffusion for Natural Adversarial Examples), our innovative method is effective in producing valid and useful NAEs, which is demonstrated through a meticulously designed experiment. Our work thereby provides a valuable method for obtaining challenging evaluation data, which in turn can potentially advance the development of more robust deep learning models. Code is available at https://github.com/linyueqian/SD-NAE.

Title: FusionFrames: Efficient Architectural Aspects for Text-to-Video Generation Pipeline. (arXiv:2311.13073v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.13073
• Code URL: null
• Copy Paste: [[2311.13073]] FusionFrames: Efficient Architectural Aspects for Text-to-Video Generation Pipeline(http://arxiv.org/abs/2311.13073)
• Summary:

  Multimedia generation approaches occupy a prominent place in artificial intelligence research. Text-to-image models achieved high-quality results over the last few years. However, video synthesis methods recently started to develop. This paper presents a new two-stage latent diffusion text-to-video generation architecture based on the text-to-image diffusion model. The first stage concerns keyframes synthesis to figure the storyline of a video, while the second one is devoted to interpolation frames generation to make movements of the scene and objects smooth. We compare several temporal conditioning approaches for keyframes generation. The results show the advantage of using separate temporal blocks over temporal layers in terms of metrics reflecting video generation quality aspects and human preference. The design of our interpolation model significantly reduces computational costs compared to other masked frame interpolation approaches. Furthermore, we evaluate different configurations of MoVQ-based video decoding scheme to improve consistency and achieve higher PSNR, SSIM, MSE, and LPIPS scores. Finally, we compare our pipeline with existing solutions and achieve top-2 scores overall and top-1 among open-source solutions: CLIPSIM = 0.2976 and FVD = 433.054. Project page: https://ai-forever.github.io/kandinsky-video/

Title: Diffusion360: Seamless 360 Degree Panoramic Image Generation based on Diffusion Models. (arXiv:2311.13141v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.13141
• Code URL: null
• Copy Paste: [[2311.13141]] Diffusion360: Seamless 360 Degree Panoramic Image Generation based on Diffusion Models(http://arxiv.org/abs/2311.13141)
• Summary:

  This is a technical report on the 360-degree panoramic image generation task based on diffusion models. Unlike ordinary 2D images, 360-degree panoramic images capture the entire $360^\circ\times 180^\circ$ field of view. So the rightmost and the leftmost sides of the 360 panoramic image should be continued, which is the main challenge in this field. However, the current diffusion pipeline is not appropriate for generating such a seamless 360-degree panoramic image. To this end, we propose a circular blending strategy on both the denoising and VAE decoding stages to maintain the geometry continuity. Based on this, we present two models for \textbf{Text-to-360-panoramas} and \textbf{Single-Image-to-360-panoramas} tasks. The code has been released as an open-source project at \href{https://github.com/ArcherFMY/SD-T2I-360PanoImage}{https://github.com/ArcherFMY/SD-T2I-360PanoImage} and \href{https://www.modelscope.cn/models/damo/cv_diffusion_text-to-360panorama-image_generation/summary}{ModelScope}

Title: Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model. (arXiv:2311.13231v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.13231
• Code URL: null
• Copy Paste: [[2311.13231]] Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model(http://arxiv.org/abs/2311.13231)
• Summary:

  Using reinforcement learning with human feedback (RLHF) has shown significant promise in fine-tuning diffusion models. Previous methods start by training a reward model that aligns with human preferences, then leverage RL techniques to fine-tune the underlying models. However, crafting an efficient reward model demands extensive datasets, optimal architecture, and manual hyperparameter tuning, making the process both time and cost-intensive. The direct preference optimization (DPO) method, effective in fine-tuning large language models, eliminates the necessity for a reward model. However, the extensive GPU memory requirement of the diffusion model's denoising process hinders the direct application of the DPO method. To address this issue, we introduce the Direct Preference for Denoising Diffusion Policy Optimization (D3PO) method to directly fine-tune diffusion models. The theoretical analysis demonstrates that although D3PO omits training a reward model, it effectively functions as the optimal reward model trained using human feedback data to guide the learning process. This approach requires no training of a reward model, proving to be more direct, cost-effective, and minimizing computational overhead. In experiments, our method uses the relative scale of objectives as a proxy for human preference, delivering comparable results to methods using ground-truth rewards. Moreover, D3PO demonstrates the ability to reduce image distortion rates and generate safer images, overcoming challenges lacking robust reward models.

Title: Recognition-Guided Diffusion Model for Scene Text Image Super-Resolution. (arXiv:2311.13317v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.13317
• Code URL: null
• Copy Paste: [[2311.13317]] Recognition-Guided Diffusion Model for Scene Text Image Super-Resolution(http://arxiv.org/abs/2311.13317)
• Summary:

  Scene Text Image Super-Resolution (STISR) aims to enhance the resolution and legibility of text within low-resolution (LR) images, consequently elevating recognition accuracy in Scene Text Recognition (STR). Previous methods predominantly employ discriminative Convolutional Neural Networks (CNNs) augmented with diverse forms of text guidance to address this issue. Nevertheless, they remain deficient when confronted with severely blurred images, due to their insufficient generation capability when little structural or semantic information can be extracted from original images. Therefore, we introduce RGDiffSR, a Recognition-Guided Diffusion model for scene text image Super-Resolution, which exhibits great generative diversity and fidelity even in challenging scenarios. Moreover, we propose a Recognition-Guided Denoising Network, to guide the diffusion model generating LR-consistent results through succinct semantic guidance. Experiments on the TextZoom dataset demonstrate the superiority of RGDiffSR over prior state-of-the-art methods in both text recognition accuracy and image fidelity.

Title: LucidDreamer: Domain-free Generation of 3D Gaussian Splatting Scenes. (arXiv:2311.13384v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.13384
• Code URL: null
• Copy Paste: [[2311.13384]] LucidDreamer: Domain-free Generation of 3D Gaussian Splatting Scenes(http://arxiv.org/abs/2311.13384)
• Summary:

  With the widespread usage of VR devices and contents, demands for 3D scene generation techniques become more popular. Existing 3D scene generation models, however, limit the target scene to specific domain, primarily due to their training strategies using 3D scan dataset that is far from the real-world. To address such limitation, we propose LucidDreamer, a domain-free scene generation pipeline by fully leveraging the power of existing large-scale diffusion-based generative model. Our LucidDreamer has two alternate steps: Dreaming and Alignment. First, to generate multi-view consistent images from inputs, we set the point cloud as a geometrical guideline for each image generation. Specifically, we project a portion of point cloud to the desired view and provide the projection as a guidance for inpainting using the generative model. The inpainted images are lifted to 3D space with estimated depth maps, composing a new points. Second, to aggregate the new points into the 3D scene, we propose an aligning algorithm which harmoniously integrates the portions of newly generated 3D scenes. The finally obtained 3D scene serves as initial points for optimizing Gaussian splats. LucidDreamer produces Gaussian splats that are highly-detailed compared to the previous 3D scene generation methods, with no constraint on domain of the target scene.

Title: DiffusionMat: Alpha Matting as Sequential Refinement Learning. (arXiv:2311.13535v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.13535
• Code URL: null
• Copy Paste: [[2311.13535]] DiffusionMat: Alpha Matting as Sequential Refinement Learning(http://arxiv.org/abs/2311.13535)
• Summary:

  In this paper, we introduce DiffusionMat, a novel image matting framework that employs a diffusion model for the transition from coarse to refined alpha mattes. Diverging from conventional methods that utilize trimaps merely as loose guidance for alpha matte prediction, our approach treats image matting as a sequential refinement learning process. This process begins with the addition of noise to trimaps and iteratively denoises them using a pre-trained diffusion model, which incrementally guides the prediction towards a clean alpha matte. The key innovation of our framework is a correction module that adjusts the output at each denoising step, ensuring that the final result is consistent with the input image's structures. We also introduce the Alpha Reliability Propagation, a novel technique designed to maximize the utility of available guidance by selectively enhancing the trimap regions with confident alpha information, thus simplifying the correction task. To train the correction module, we devise specialized loss functions that target the accuracy of the alpha matte's edges and the consistency of its opaque and transparent regions. We evaluate our model across several image matting benchmarks, and the results indicate that DiffusionMat consistently outperforms existing methods. Project page at~\url{https://cnnlstm.github.io/DiffusionMat

Title: ADriver-I: A General World Model for Autonomous Driving. (arXiv:2311.13549v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.13549
• Code URL: null
• Copy Paste: [[2311.13549]] ADriver-I: A General World Model for Autonomous Driving(http://arxiv.org/abs/2311.13549)
• Summary:

  Typically, autonomous driving adopts a modular design, which divides the full stack into perception, prediction, planning and control parts. Though interpretable, such modular design tends to introduce a substantial amount of redundancy. Recently, multimodal large language models (MLLM) and diffusion techniques have demonstrated their superior performance on comprehension and generation ability. In this paper, we first introduce the concept of interleaved vision-action pair, which unifies the format of visual features and control signals. Based on the vision-action pairs, we construct a general world model based on MLLM and diffusion model for autonomous driving, termed ADriver-I. It takes the vision-action pairs as inputs and autoregressively predicts the control signal of the current frame. The generated control signals together with the historical vision-action pairs are further conditioned to predict the future frames. With the predicted next frame, ADriver-I performs further control signal prediction. Such a process can be repeated infinite times, ADriver-I achieves autonomous driving in the world created by itself. Extensive experiments are conducted on nuScenes and our large-scale private datasets. ADriver-I shows impressive performance compared to several constructed baselines. We hope our ADriver-I can provide some new insights for future autonomous driving and embodied intelligence.

Title: WildFusion: Learning 3D-Aware Latent Diffusion Models in View Space. (arXiv:2311.13570v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.13570
• Code URL: null
• Copy Paste: [[2311.13570]] WildFusion: Learning 3D-Aware Latent Diffusion Models in View Space(http://arxiv.org/abs/2311.13570)
• Summary:

  Modern learning-based approaches to 3D-aware image synthesis achieve high photorealism and 3D-consistent viewpoint changes for the generated images. Existing approaches represent instances in a shared canonical space. However, for in-the-wild datasets a shared canonical system can be difficult to define or might not even exist. In this work, we instead model instances in view space, alleviating the need for posed images and learned camera distributions. We find that in this setting, existing GAN-based methods are prone to generating flat geometry and struggle with distribution coverage. We hence propose WildFusion, a new approach to 3D-aware image synthesis based on latent diffusion models (LDMs). We first train an autoencoder that infers a compressed latent representation, which additionally captures the images' underlying 3D structure and enables not only reconstruction but also novel view synthesis. To learn a faithful 3D representation, we leverage cues from monocular depth prediction. Then, we train a diffusion model in the 3D-aware latent space, thereby enabling synthesis of high-quality 3D-consistent image samples, outperforming recent state-of-the-art GAN-based methods. Importantly, our 3D-aware LDM is trained without any direct supervision from multiview images or 3D geometry and does not require posed images or learned pose or camera distributions. It directly learns a 3D representation without relying on canonical camera coordinates. This opens up promising research avenues for scalable 3D-aware image synthesis and 3D content creation from in-the-wild image data. See https://katjaschwarz.github.io/wildfusion for videos of our 3D results.

Title: RAEDiff: Denoising Diffusion Probabilistic Models Based Reversible Adversarial Examples Self-Generation and Self-Recovery. (arXiv:2311.12858v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.12858
• Code URL: null
• Copy Paste: [[2311.12858]] RAEDiff: Denoising Diffusion Probabilistic Models Based Reversible Adversarial Examples Self-Generation and Self-Recovery(http://arxiv.org/abs/2311.12858)
• Summary:

  Collected and annotated datasets, which are obtained through extensive efforts, are effective for training Deep Neural Network (DNN) models. However, these datasets are susceptible to be misused by unauthorized users, resulting in infringement of Intellectual Property (IP) rights owned by the dataset creators. Reversible Adversarial Exsamples (RAE) can help to solve the issues of IP protection for datasets. RAEs are adversarial perturbed images that can be restored to the original. As a cutting-edge approach, RAE scheme can serve the purposes of preventing unauthorized users from engaging in malicious model training, as well as ensuring the legitimate usage of authorized users. Nevertheless, in the existing work, RAEs still rely on the embedded auxiliary information for restoration, which may compromise their adversarial abilities. In this paper, a novel self-generation and self-recovery method, named as RAEDiff, is introduced for generating RAEs based on a Denoising Diffusion Probabilistic Models (DDPM). It diffuses datasets into a Biased Gaussian Distribution (BGD) and utilizes the prior knowledge of the DDPM for generating and recovering RAEs. The experimental results demonstrate that RAEDiff effectively self-generates adversarial perturbations for DNN models, including Artificial Intelligence Generated Content (AIGC) models, while also exhibiting significant self-recovery capabilities.

Title: On diffusion-based generative models and their error bounds: The log-concave case with full convergence estimates. (arXiv:2311.13584v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.13584
• Code URL: null
• Copy Paste: [[2311.13584]] On diffusion-based generative models and their error bounds: The log-concave case with full convergence estimates(http://arxiv.org/abs/2311.13584)
• Summary:

  We provide full theoretical guarantees for the convergence behaviour of diffusion-based generative models under the assumption of strongly logconcave data distributions while our approximating class of functions used for score estimation is made of Lipschitz continuous functions. We demonstrate via a motivating example, sampling from a Gaussian distribution with unknown mean, the powerfulness of our approach. In this case, explicit estimates are provided for the associated optimization problem, i.e. score approximation, while these are combined with the corresponding sampling estimates. As a result, we obtain the best known upper bound estimates in terms of key quantities of interest, such as the dimension and rates of convergence, for the Wasserstein-2 distance between the data distribution (Gaussian with unknown mean) and our sampling algorithm.

  Beyond the motivating example and in order to allow for the use of a diverse range of stochastic optimizers, we present our results using an $L^2$-accurate score estimation assumption, which crucially is formed under an expectation with respect to the stochastic optimizer and our novel auxiliary process that uses only known information. This approach yields the best known convergence rate for our sampling algorithm.

noise learning

data-free

transformer

Title: EWasteNet: A Two-Stream Data Efficient Image Transformer Approach for E-Waste Classification. (arXiv:2311.12823v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12823
• Code URL: null
• Copy Paste: [[2311.12823]] EWasteNet: A Two-Stream Data Efficient Image Transformer Approach for E-Waste Classification(http://arxiv.org/abs/2311.12823)
• Summary:

  Improper disposal of e-waste poses global environmental and health risks, raising serious concerns. The accurate classification of e-waste images is critical for efficient management and recycling. In this paper, we have presented a comprehensive dataset comprised of eight different classes of images of electronic devices named the E-Waste Vision Dataset. We have also presented EWasteNet, a novel two-stream approach for precise e-waste image classification based on a data-efficient image transformer (DeiT). The first stream of EWasteNet passes through a sobel operator that detects the edges while the second stream is directed through an Atrous Spatial Pyramid Pooling and attention block where multi-scale contextual information is captured. We train both of the streams simultaneously and their features are merged at the decision level. The DeiT is used as the backbone of both streams. Extensive analysis of the e-waste dataset indicates the usefulness of our method, providing 96% accuracy in e-waste classification. The proposed approach demonstrates significant usefulness in addressing the global concern of e-waste management. It facilitates efficient waste management and recycling by accurately classifying e-waste images, reducing health and safety hazards associated with improper disposal.

Title: Long-MIL: Scaling Long Contextual Multiple Instance Learning for Histopathology Whole Slide Image Analysis. (arXiv:2311.12885v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12885
• Code URL: null
• Copy Paste: [[2311.12885]] Long-MIL: Scaling Long Contextual Multiple Instance Learning for Histopathology Whole Slide Image Analysis(http://arxiv.org/abs/2311.12885)
• Summary:

  Histopathology image analysis is the golden standard of clinical diagnosis for Cancers. In doctors daily routine and computer-aided diagnosis, the Whole Slide Image (WSI) of histopathology tissue is used for analysis. Because of the extremely large scale of resolution, previous methods generally divide the WSI into a large number of patches, then aggregate all patches within a WSI by Multi-Instance Learning (MIL) to make the slide-level prediction when developing computer-aided diagnosis tools. However, most previous WSI-MIL models using global-attention without pairwise interaction and any positional information, or self-attention with absolute position embedding can not well handle shape varying large WSIs, e.g. testing WSIs after model deployment may be larger than training WSIs, since the model development set is always limited due to the difficulty of histopathology WSIs collection. To deal with the problem, in this paper, we propose to amend position embedding for shape varying long-contextual WSI by introducing Linear Bias into Attention, and adapt it from 1-d long sequence into 2-d long-contextual WSI which helps model extrapolate position embedding to unseen or under-fitted positions. We further utilize Flash-Attention module to tackle the computational complexity of Transformer, which also keep full self-attention performance compared to previous attention approximation work. Our method, Long-contextual MIL (Long-MIL) are evaluated on extensive experiments including 4 dataset including WSI classification and survival prediction tasks to validate the superiority on shape varying WSIs. The source code will be open-accessed soon.

Title: Attention Deficit is Ordered! Fooling Deformable Vision Transformers with Collaborative Adversarial Patches. (arXiv:2311.12914v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12914
• Code URL: null
• Copy Paste: [[2311.12914]] Attention Deficit is Ordered! Fooling Deformable Vision Transformers with Collaborative Adversarial Patches(http://arxiv.org/abs/2311.12914)
• Summary:

  The latest generation of transformer-based vision models have proven to be superior to Convolutional Neural Network (CNN)-based models across several vision tasks, largely attributed to their remarkable prowess in relation modeling. Deformable vision transformers significantly reduce the quadratic complexity of modeling attention by using sparse attention structures, enabling them to be used in larger scale applications such as multi-view vision systems. Recent work demonstrated adversarial attacks against transformers; we show that these attacks do not transfer to deformable transformers due to their sparse attention structure. Specifically, attention in deformable transformers is modeled using pointers to the most relevant other tokens. In this work, we contribute for the first time adversarial attacks that manipulate the attention of deformable transformers, distracting them to focus on irrelevant parts of the image. We also develop new collaborative attacks where a source patch manipulates attention to point to a target patch that adversarially attacks the system. In our experiments, we find that only 1% patched area of the input field can lead to 0% AP. We also show that the attacks provide substantial versatility to support different attacker scenarios because of their ability to redirect attention under the attacker control.

Title: White-Box Transformers via Sparse Rate Reduction: Compression Is All There Is?. (arXiv:2311.13110v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.13110
• Code URL: null
• Copy Paste: [[2311.13110]] White-Box Transformers via Sparse Rate Reduction: Compression Is All There Is?(http://arxiv.org/abs/2311.13110)
• Summary:

  In this paper, we contend that a natural objective of representation learning is to compress and transform the distribution of the data, say sets of tokens, towards a low-dimensional Gaussian mixture supported on incoherent subspaces. The goodness of such a representation can be evaluated by a principled measure, called sparse rate reduction, that simultaneously maximizes the intrinsic information gain and extrinsic sparsity of the learned representation. From this perspective, popular deep network architectures, including transformers, can be viewed as realizing iterative schemes to optimize this measure. Particularly, we derive a transformer block from alternating optimization on parts of this objective: the multi-head self-attention operator compresses the representation by implementing an approximate gradient descent step on the coding rate of the features, and the subsequent multi-layer perceptron sparsifies the features. This leads to a family of white-box transformer-like deep network architectures, named CRATE, which are mathematically fully interpretable. We show, by way of a novel connection between denoising and compression, that the inverse to the aforementioned compressive encoding can be realized by the same class of CRATE architectures. Thus, the so-derived white-box architectures are universal to both encoders and decoders. Experiments show that these networks, despite their simplicity, indeed learn to compress and sparsify representations of large-scale real-world image and text datasets, and achieve performance very close to highly engineered transformer-based models: ViT, MAE, DINO, BERT, and GPT2. We believe the proposed computational framework demonstrates great potential in bridging the gap between theory and practice of deep learning, from a unified perspective of data compression. Code is available at: https://ma-lab-berkeley.github.io/CRATE .

Title: Towards Detecting, Recognizing, and Parsing the Address Information from Bangla Signboard: A Deep Learning-based Approach. (arXiv:2311.13222v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.13222
• Code URL: null
• Copy Paste: [[2311.13222]] Towards Detecting, Recognizing, and Parsing the Address Information from Bangla Signboard: A Deep Learning-based Approach(http://arxiv.org/abs/2311.13222)
• Summary:

  Retrieving textual information from natural scene images is an active research area in the field of computer vision with numerous practical applications. Detecting text regions and extracting text from signboards is a challenging problem due to special characteristics like reflecting lights, uneven illumination, or shadows found in real-life natural scene images. With the advent of deep learning-based methods, different sophisticated techniques have been proposed for text detection and text recognition from the natural scene. Though a significant amount of effort has been devoted to extracting natural scene text for resourceful languages like English, little has been done for low-resource languages like Bangla. In this research work, we have proposed an end-to-end system with deep learning-based models for efficiently detecting, recognizing, correcting, and parsing address information from Bangla signboards. We have created manually annotated datasets and synthetic datasets to train signboard detection, address text detection, address text recognition, address text correction, and address text parser models. We have conducted a comparative study among different CTC-based and Encoder-Decoder model architectures for Bangla address text recognition. Moreover, we have designed a novel address text correction model using a sequence-to-sequence transformer-based network to improve the performance of Bangla address text recognition model by post-correction. Finally, we have developed a Bangla address text parser using the state-of-the-art transformer-based pre-trained language model.

Title: TSegFormer: 3D Tooth Segmentation in Intraoral Scans with Geometry Guided Transformer. (arXiv:2311.13234v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.13234
• Code URL: null
• Copy Paste: [[2311.13234]] TSegFormer: 3D Tooth Segmentation in Intraoral Scans with Geometry Guided Transformer(http://arxiv.org/abs/2311.13234)
• Summary:

  Optical Intraoral Scanners (IOS) are widely used in digital dentistry to provide detailed 3D information of dental crowns and the gingiva. Accurate 3D tooth segmentation in IOSs is critical for various dental applications, while previous methods are error-prone at complicated boundaries and exhibit unsatisfactory results across patients. In this paper, we propose TSegFormer which captures both local and global dependencies among different teeth and the gingiva in the IOS point clouds with a multi-task 3D transformer architecture. Moreover, we design a geometry-guided loss based on a novel point curvature to refine boundaries in an end-to-end manner, avoiding time-consuming post-processing to reach clinically applicable segmentation. In addition, we create a dataset with 16,000 IOSs, the largest ever IOS dataset to the best of our knowledge. The experimental results demonstrate that our TSegFormer consistently surpasses existing state-of-the-art baselines. The superiority of TSegFormer is corroborated by extensive analysis, visualizations and real-world clinical applicability tests. Our code is available at https://github.com/huiminxiong/TSegFormer.

Title: CMFDFormer: Transformer-based Copy-Move Forgery Detection with Continual Learning. (arXiv:2311.13263v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.13263
• Code URL: null
• Copy Paste: [[2311.13263]] CMFDFormer: Transformer-based Copy-Move Forgery Detection with Continual Learning(http://arxiv.org/abs/2311.13263)
• Summary:

  Copy-move forgery detection aims at detecting duplicated regions in a suspected forged image, and deep learning based copy-move forgery detection methods are in the ascendant. These deep learning based methods heavily rely on synthetic training data, and the performance will degrade when facing new tasks. In this paper, we propose a Transformer-style copy-move forgery detection network named as CMFDFormer, and provide a novel PCSD (Pooled Cube and Strip Distillation) continual learning framework to help CMFDFormer handle new tasks. CMFDFormer consists of a MiT (Mix Transformer) backbone network and a PHD (Pluggable Hybrid Decoder) mask prediction network. The MiT backbone network is a Transformer-style network which is adopted on the basis of comprehensive analyses with CNN-style and MLP-style backbones. The PHD network is constructed based on self-correlation computation, hierarchical feature integration, a multi-scale cycle fully-connected block and a mask reconstruction block. The PHD network is applicable to feature extractors of different styles for hierarchical multi-scale information extraction, achieving comparable performance. Last but not least, we propose a PCSD continual learning framework to improve the forgery detectability and avoid catastrophic forgetting when handling new tasks. Our continual learning framework restricts intermediate features from the PHD network, and takes advantage of both cube pooling and strip pooling. Extensive experiments on publicly available datasets demonstrate the good performance of CMFDFormer and the effectiveness of the PCSD continual learning framework.

Title: Retrieval-Augmented Layout Transformer for Content-Aware Layout Generation. (arXiv:2311.13602v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.13602
• Code URL: null
• Copy Paste: [[2311.13602]] Retrieval-Augmented Layout Transformer for Content-Aware Layout Generation(http://arxiv.org/abs/2311.13602)
• Summary:

  Content-aware graphic layout generation aims to automatically arrange visual elements along with a given content, such as an e-commerce product image. In this paper, we argue that the current layout generation approaches suffer from the limited training data for the high-dimensional layout structure. We show that a simple retrieval augmentation can significantly improve the generation quality. Our model, which is named Retrieval-Augmented Layout Transformer (RALF), retrieves nearest neighbor layout examples based on an input image and feeds these results into an autoregressive generator. Our model can apply retrieval augmentation to various controllable generation tasks and yield high-quality layouts within a unified architecture. Our extensive experiments show that RALF successfully generates content-aware layouts in both constrained and unconstrained settings and significantly outperforms the baselines.

Title: Detecting out-of-distribution text using topological features of transformer-based language models. (arXiv:2311.13102v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.13102
• Code URL: null
• Copy Paste: [[2311.13102]] Detecting out-of-distribution text using topological features of transformer-based language models(http://arxiv.org/abs/2311.13102)
• Summary:

  We attempt to detect out-of-distribution (OOD) text samples though applying Topological Data Analysis (TDA) to attention maps in transformer-based language models. We evaluate our proposed TDA-based approach for out-of-distribution detection on BERT, a transformer-based language model, and compare the to a more traditional OOD approach based on BERT CLS embeddings. We found that our TDA approach outperforms the CLS embedding approach at distinguishing in-distribution data (politics and entertainment news articles from HuffPost) from far out-of-domain samples (IMDB reviews), but its effectiveness deteriorates with near out-of-domain (CNN/Dailymail) or same-domain (business news articles from HuffPost) datasets.

Title: Enhancing Summarization Performance through Transformer-Based Prompt Engineering in Automated Medical Reporting. (arXiv:2311.13274v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.13274
• Code URL: null
• Copy Paste: [[2311.13274]] Enhancing Summarization Performance through Transformer-Based Prompt Engineering in Automated Medical Reporting(http://arxiv.org/abs/2311.13274)
• Summary:

  Customized medical prompts enable Large Language Models (LLM) to effectively address medical dialogue summarization. The process of medical reporting is often time-consuming for healthcare professionals. Implementing medical dialogue summarization techniques presents a viable solution to alleviate this time constraint by generating automated medical reports. The effectiveness of LLMs in this process is significantly influenced by the formulation of the prompt, which plays a crucial role in determining the quality and relevance of the generated reports. In this research, we used a combination of two distinct prompting strategies, known as shot prompting and pattern prompting to enhance the performance of automated medical reporting. The evaluation of the automated medical reports is carried out using the ROUGE score and a human evaluation with the help of an expert panel. The two-shot prompting approach in combination with scope and domain context outperforms other methods and achieves the highest score when compared to the human reference set by a general practitioner. However, the automated reports are approximately twice as long as the human references, due to the addition of both redundant and relevant statements that are added to the report.

Title: Fact-based Court Judgment Prediction. (arXiv:2311.13350v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.13350
• Code URL: null
• Copy Paste: [[2311.13350]] Fact-based Court Judgment Prediction(http://arxiv.org/abs/2311.13350)
• Summary:

  This extended abstract extends the research presented in "ILDC for CJPE: Indian Legal Documents Corpus for Court Judgment Prediction and Explanation" \cite{malik-etal-2021-ildc}, focusing on fact-based judgment prediction within the context of Indian legal documents. We introduce two distinct problem variations: one based solely on facts, and another combining facts with rulings from lower courts (RLC). Our research aims to enhance early-phase case outcome prediction, offering significant benefits to legal professionals and the general public. The results, however, indicated a performance decline compared to the original ILDC for CJPE study, even after implementing various weightage schemes in our DELSumm algorithm. Additionally, using only facts for legal judgment prediction with different transformer models yielded results inferior to the state-of-the-art outcomes reported in the "ILDC for CJPE" study.

Title: Machine Translation to Control Formality Features in the Target Language. (arXiv:2311.13475v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.13475
• Code URL: null
• Copy Paste: [[2311.13475]] Machine Translation to Control Formality Features in the Target Language(http://arxiv.org/abs/2311.13475)
• Summary:

  Formality plays a significant role in language communication, especially in low-resource languages such as Hindi, Japanese and Korean. These languages utilise formal and informal expressions to convey messages based on social contexts and relationships. When a language translation technique is used to translate from a source language that does not pertain the formality (e.g. English) to a target language that does, there is a missing information on formality that could be a challenge in producing an accurate outcome. This research explores how this issue should be resolved when machine learning methods are used to translate from English to languages with formality, using Hindi as the example data. This was done by training a bilingual model in a formality-controlled setting and comparing its performance with a pre-trained multilingual model in a similar setting. Since there are not a lot of training data with ground truth, automated annotation techniques were employed to increase the data size. The primary modeling approach involved leveraging transformer models, which have demonstrated effectiveness in various natural language processing tasks. We evaluate the official formality accuracy(ACC) by comparing the predicted masked tokens with the ground truth. This metric provides a quantitative measure of how well the translations align with the desired outputs. Our study showcases a versatile translation strategy that considers the nuances of formality in the target language, catering to diverse language communication needs and scenarios.

Title: How Capable Can a Transformer Become? A Study on Synthetic, Interpretable Tasks. (arXiv:2311.12997v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.12997
• Code URL: null
• Copy Paste: [[2311.12997]] How Capable Can a Transformer Become? A Study on Synthetic, Interpretable Tasks(http://arxiv.org/abs/2311.12997)
• Summary:

  Transformers trained on huge text corpora exhibit a remarkable set of capabilities, e.g., performing simple logical operations. Given the inherent compositional nature of language, one can expect the model to learn to compose these capabilities, potentially yielding a combinatorial explosion of what operations it can perform on an input. Motivated by the above, we aim to assess in this paper "how capable can a transformer become?". Specifically, we train autoregressive Transformer models on a data-generating process that involves compositions of a set of well-defined monolithic capabilities. Through a series of extensive and systematic experiments on this data-generating process, we show that: (1) autoregressive Transformers can learn compositional structures from the training data and generalize to exponentially or even combinatorially many functions; (2) composing functions by generating intermediate outputs is more effective at generalizing to unseen compositions, compared to generating no intermediate outputs; (3) the training data has a significant impact on the model's ability to compose unseen combinations of functions; and (4) the attention layers in the latter half of the model are critical to compositionality.

Title: Confidant: Customizing Transformer-based LLMs via Collaborative Edge Training. (arXiv:2311.13381v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.13381
• Code URL: null
• Copy Paste: [[2311.13381]] Confidant: Customizing Transformer-based LLMs via Collaborative Edge Training(http://arxiv.org/abs/2311.13381)
• Summary:

  Transformer-based large language models (LLMs) have demonstrated impressive capabilities in a variety of natural language processing (NLP) tasks. Nonetheless, it is challenging to deploy and fine-tune LLMs on mobile edge devices with limited computing, memory, and energy budgets. In this paper, we propose Confidant, a multi-backend collaborative training framework for customizing state-of-the-art LLMs on commodity mobile devices like smartphones. Confidant partitions an LLM into several sub-models so that each fits into a mobile device's memory. A pipeline parallel training mechanism is further developed to ensure fast and efficient distributed training. In addition, we propose a novel backend scheduler to allocate different attention heads to heterogeneous compute hardware, including mobile CPU and GPUs, to maximize the compute resource utilization on each edge device. Our preliminary experimental results show that Confidant achieves at most 45.3% memory reduction and 8.03x inference speedup in practical settings.

Title: Bitformer: An efficient Transformer with bitwise operation-based attention for Big Data Analytics at low-cost low-precision devices. (arXiv:2311.13502v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.13502
• Code URL: null
• Copy Paste: [[2311.13502]] Bitformer: An efficient Transformer with bitwise operation-based attention for Big Data Analytics at low-cost low-precision devices(http://arxiv.org/abs/2311.13502)
• Summary:

  In the current landscape of large models, the Transformer stands as a cornerstone, playing a pivotal role in shaping the trajectory of modern models. However, its application encounters challenges attributed to the substantial computational intricacies intrinsic to its attention mechanism. Moreover, its reliance on high-precision floating-point operations presents specific hurdles, particularly evident in computation-intensive scenarios such as edge computing environments. These environments, characterized by resource-constrained devices and a preference for lower precision, necessitate innovative solutions.

  To tackle the exacting data processing demands posed by edge devices, we introduce the Bitformer model, an inventive extension of the Transformer paradigm. Central to this innovation is a novel attention mechanism that adeptly replaces conventional floating-point matrix multiplication with bitwise operations. This strategic substitution yields dual advantages. Not only does it maintain the attention mechanism's prowess in capturing intricate long-range information dependencies, but it also orchestrates a profound reduction in the computational complexity inherent in the attention operation. The transition from an $O(n^2d)$ complexity, typical of floating-point operations, to an $O(n^2T)$ complexity characterizing bitwise operations, substantiates this advantage. Notably, in this context, the parameter $T$ remains markedly smaller than the conventional dimensionality parameter $d$.

  The Bitformer model in essence endeavors to reconcile the indomitable requirements of modern computing landscapes with the constraints posed by edge computing scenarios. By forging this innovative path, we bridge the gap between high-performing models and resource-scarce environments, thus unveiling a promising trajectory for further advancements in the field.

Title: Linear Log-Normal Attention with Unbiased Concentration. (arXiv:2311.13541v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.13541
• Code URL: null
• Copy Paste: [[2311.13541]] Linear Log-Normal Attention with Unbiased Concentration(http://arxiv.org/abs/2311.13541)
• Summary:

  Transformer models have achieved remarkable results in a wide range of applications. However, their scalability is hampered by the quadratic time and memory complexity of the self-attention mechanism concerning the sequence length. This limitation poses a substantial obstacle when dealing with long documents or high-resolution images. In this work, we study the self-attention mechanism by analyzing the distribution of the attention matrix and its concentration ability. Furthermore, we propose instruments to measure these quantities and introduce a novel self-attention mechanism, Linear Log-Normal Attention, designed to emulate the distribution and concentration behavior of the original self-attention. Our experimental results on popular natural language benchmarks reveal that our proposed Linear Log-Normal Attention outperforms other linearized attention alternatives, offering a promising avenue for enhancing the scalability of transformer models. Our code is available in supplementary materials.

generative

Title: PG-Video-LLaVA: Pixel Grounding Large Video-Language Models. (arXiv:2311.13435v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.13435
• Code URL: null
• Copy Paste: [[2311.13435]] PG-Video-LLaVA: Pixel Grounding Large Video-Language Models(http://arxiv.org/abs/2311.13435)
• Summary:

  Extending image-based Large Multimodal Models (LMM) to videos is challenging due to the inherent complexity of video data. The recent approaches extending image-based LMM to videos either lack the grounding capabilities (e.g., VideoChat, Video-ChatGPT, Video-LLaMA) or do not utilize the audio-signals for better video understanding (e.g., Video-ChatGPT). Addressing these gaps, we propose Video-LLaVA, the first LMM with pixel-level grounding capability, integrating audio cues by transcribing them into text to enrich video-context understanding. Our framework uses an off-the-shelf tracker and a novel grounding module, enabling it to spatially and temporally localize objects in videos following user instructions. We evaluate Video-LLaVA using video-based generative and question-answering benchmarks and introduce new benchmarks specifically designed to measure prompt-based object grounding performance in videos. Further, we propose the use of Vicuna over GPT-3.5, as utilized in Video-ChatGPT, for video-based conversation benchmarking, ensuring reproducibility of results which is a concern with the proprietary nature of GPT-3.5. Our framework builds on SoTA image-based LLaVA model and extends its advantages to the video domain, delivering promising gains on video-based conversation and grounding tasks. Project Page: https://github.com/mbzuai-oryx/Video-LLaVA

Title: Guided Flows for Generative Modeling and Decision Making. (arXiv:2311.13443v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.13443
• Code URL: null
• Copy Paste: [[2311.13443]] Guided Flows for Generative Modeling and Decision Making(http://arxiv.org/abs/2311.13443)
• Summary:

  Classifier-free guidance is a key component for improving the performance of conditional generative models for many downstream tasks. It drastically improves the quality of samples produced, but has so far only been used for diffusion models. Flow Matching (FM), an alternative simulation-free approach, trains Continuous Normalizing Flows (CNFs) based on regressing vector fields. It remains an open question whether classifier-free guidance can be performed for Flow Matching models, and to what extent does it improve performance. In this paper, we explore the usage of Guided Flows for a variety of downstream applications involving conditional image generation, speech synthesis, and reinforcement learning. In particular, we are the first to apply flow models to the offline reinforcement learning setting. We also show that Guided Flows significantly improves the sample quality in image generation and zero-shot text-to-speech synthesis, and can make use of drastically low amounts of computation without affecting the agent's overall performance.

Title: XAGen: 3D Expressive Human Avatars Generation. (arXiv:2311.13574v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.13574
• Code URL: null
• Copy Paste: [[2311.13574]] XAGen: 3D Expressive Human Avatars Generation(http://arxiv.org/abs/2311.13574)
• Summary:

  Recent advances in 3D-aware GAN models have enabled the generation of realistic and controllable human body images. However, existing methods focus on the control of major body joints, neglecting the manipulation of expressive attributes, such as facial expressions, jaw poses, hand poses, and so on. In this work, we present XAGen, the first 3D generative model for human avatars capable of expressive control over body, face, and hands. To enhance the fidelity of small-scale regions like face and hands, we devise a multi-scale and multi-part 3D representation that models fine details. Based on this representation, we propose a multi-part rendering technique that disentangles the synthesis of body, face, and hands to ease model training and enhance geometric quality. Furthermore, we design multi-part discriminators that evaluate the quality of the generated avatars with respect to their appearance and fine-grained control capabilities. Experiments show that XAGen surpasses state-of-the-art methods in terms of realism, diversity, and expressive control abilities. Code and data will be made available at https://showlab.github.io/xagen.

Title: ZipLoRA: Any Subject in Any Style by Effectively Merging LoRAs. (arXiv:2311.13600v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.13600
• Code URL: null
• Copy Paste: [[2311.13600]] ZipLoRA: Any Subject in Any Style by Effectively Merging LoRAs(http://arxiv.org/abs/2311.13600)
• Summary:

  Methods for finetuning generative models for concept-driven personalization generally achieve strong results for subject-driven or style-driven generation. Recently, low-rank adaptations (LoRA) have been proposed as a parameter-efficient way of achieving concept-driven personalization. While recent work explores the combination of separate LoRAs to achieve joint generation of learned styles and subjects, existing techniques do not reliably address the problem; they often compromise either subject fidelity or style fidelity. We propose ZipLoRA, a method to cheaply and effectively merge independently trained style and subject LoRAs in order to achieve generation of any user-provided subject in any user-provided style. Experiments on a wide range of subject and style combinations show that ZipLoRA can generate compelling results with meaningful improvements over baselines in subject and style fidelity while preserving the ability to recontextualize. Project page: https://ziplora.github.io

Title: Comparative Experimentation of Accuracy Metrics in Automated Medical Reporting: The Case of Otitis Consultations. (arXiv:2311.13273v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.13273
• Code URL: null
• Copy Paste: [[2311.13273]] Comparative Experimentation of Accuracy Metrics in Automated Medical Reporting: The Case of Otitis Consultations(http://arxiv.org/abs/2311.13273)
• Summary:

  Generative Artificial Intelligence (AI) can be used to automatically generate medical reports based on transcripts of medical consultations. The aim is to reduce the administrative burden that healthcare professionals face. The accuracy of the generated reports needs to be established to ensure their correctness and usefulness. There are several metrics for measuring the accuracy of AI generated reports, but little work has been done towards the application of these metrics in medical reporting. A comparative experimentation of 10 accuracy metrics has been performed on AI generated medical reports against their corresponding General Practitioner's (GP) medical reports concerning Otitis consultations. The number of missing, incorrect, and additional statements of the generated reports have been correlated with the metric scores. In addition, we introduce and define a Composite Accuracy Score which produces a single score for comparing the metrics within the field of automated medical reporting. Findings show that based on the correlation study and the Composite Accuracy Score, the ROUGE-L and Word Mover's Distance metrics are the preferred metrics, which is not in line with previous work. These findings help determine the accuracy of an AI generated medical report, which aids the development of systems that generate medical reports for GPs to reduce the administrative burden.

Title: Span-Based Optimal Sample Complexity for Average Reward MDPs. (arXiv:2311.13469v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.13469
• Code URL: null
• Copy Paste: [[2311.13469]] Span-Based Optimal Sample Complexity for Average Reward MDPs(http://arxiv.org/abs/2311.13469)
• Summary:

  We study the sample complexity of learning an $\varepsilon$-optimal policy in an average-reward Markov decision process (MDP) under a generative model. We establish the complexity bound $\widetilde{O}\left(SA\frac{H}{\varepsilon^2} \right)$, where $H$ is the span of the bias function of the optimal policy and $SA$ is the cardinality of the state-action space. Our result is the first that is minimax optimal (up to log factors) in all parameters $S,A,H$ and $\varepsilon$, improving on existing work that either assumes uniformly bounded mixing times for all policies or has suboptimal dependence on the parameters.

  Our result is based on reducing the average-reward MDP to a discounted MDP. To establish the optimality of this reduction, we develop improved bounds for $\gamma$-discounted MDPs, showing that $\widetilde{O}\left(SA\frac{H}{(1-\gamma)^2\varepsilon^2} \right)$ samples suffice to learn a $\varepsilon$-optimal policy in weakly communicating MDPs under the regime that $\gamma \geq 1 - \frac{1}{H}$, circumventing the well-known lower bound of $\widetilde{\Omega}\left(SA\frac{1}{(1-\gamma)^3\varepsilon^2} \right)$ for general $\gamma$-discounted MDPs. Our analysis develops upper bounds on certain instance-dependent variance parameters in terms of the span parameter. These bounds are tighter than those based on the mixing time or diameter of the MDP and may be of broader use.

large language model

Title: An Embodied Generalist Agent in 3D World. (arXiv:2311.12871v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12871
• Code URL: null
• Copy Paste: [[2311.12871]] An Embodied Generalist Agent in 3D World(http://arxiv.org/abs/2311.12871)
• Summary:

  Leveraging massive knowledge and learning schemes from large language models (LLMs), recent machine learning models show notable successes in building generalist agents that exhibit the capability of general-purpose task solving in diverse domains, including natural language processing, computer vision, and robotics. However, a significant challenge remains as these models exhibit limited ability in understanding and interacting with the 3D world. We argue this limitation significantly hinders the current models from performing real-world tasks and further achieving general intelligence. To this end, we introduce an embodied multi-modal and multi-task generalist agent that excels in perceiving, grounding, reasoning, planning, and acting in the 3D world. Our proposed agent, referred to as LEO, is trained with shared LLM-based model architectures, objectives, and weights in two stages: (i) 3D vision-language alignment and (ii) 3D vision-language-action instruction tuning. To facilitate the training, we meticulously curate and generate an extensive dataset comprising object-level and scene-level multi-modal tasks with exceeding scale and complexity, necessitating a deep understanding of and interaction with the 3D world. Through rigorous experiments, we demonstrate LEO's remarkable proficiency across a wide spectrum of tasks, including 3D captioning, question answering, embodied reasoning, embodied navigation, and robotic manipulation. Our ablation results further provide valuable insights for the development of future embodied generalist agents.

Title: Enhancing Scene Graph Generation with Hierarchical Relationships and Commonsense Knowledge. (arXiv:2311.12889v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12889
• Code URL: null
• Copy Paste: [[2311.12889]] Enhancing Scene Graph Generation with Hierarchical Relationships and Commonsense Knowledge(http://arxiv.org/abs/2311.12889)
• Summary:

  This work presents an enhanced approach to generating scene graphs by incorporating a relationship hierarchy and commonsense knowledge. Specifically, we propose a Bayesian classification head that exploits an informative hierarchical structure. It jointly predicts the super-category or type of relationship between the two objects, along with the detailed relationship under each super-category. We design a commonsense validation pipeline that uses a large language model to critique the results from the scene graph prediction system and then use that feedback to enhance the model performance. The system requires no external large language model assistance at test time, making it more convenient for practical applications. Experiments on the Visual Genome and the OpenImage V6 datasets demonstrate that harnessing hierarchical relationships enhances the model performance by a large margin. The proposed Bayesian head can also be incorporated as a portable module in existing scene graph generation algorithms to improve their results. In addition, the commonsense validation enables the model to generate an extensive set of reasonable predictions beyond dataset annotations.

Title: Towards Improving Document Understanding: An Exploration on Text-Grounding via MLLMs. (arXiv:2311.13194v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.13194
• Code URL: null
• Copy Paste: [[2311.13194]] Towards Improving Document Understanding: An Exploration on Text-Grounding via MLLMs(http://arxiv.org/abs/2311.13194)
• Summary:

  In the field of document understanding, significant advances have been made in the fine-tuning of Multimodal Large Language Models (MLLMs) with instruction-following data. Nevertheless, the potential of text-grounding capability within text-rich scenarios remains underexplored. In this paper, we present a text-grounding document understanding model, termed TGDoc, which addresses this deficiency by enhancing MLLMs with the ability to discern the spatial positioning of text within images. Empirical evidence suggests that text-grounding improves the model's interpretation of textual content, thereby elevating its proficiency in comprehending text-rich images. Specifically, we compile a dataset containing 99K PowerPoint presentations sourced from the internet. We formulate instruction tuning tasks including text detection, recognition, and spotting to facilitate the cohesive alignment between the visual encoder and large language model. Moreover, we curate a collection of text-rich images and prompt the text-only GPT-4 to generate 12K high-quality conversations, featuring textual locations within text-rich scenarios. By integrating text location data into the instructions, TGDoc is adept at discerning text locations during the visual question process. Extensive experiments demonstrate that our method achieves state-of-the-art performance across multiple text-rich benchmarks, validating the effectiveness of our method.

Title: Soulstyler: Using Large Language Model to Guide Image Style Transfer for Target Object. (arXiv:2311.13562v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.13562
• Code URL: null
• Copy Paste: [[2311.13562]] Soulstyler: Using Large Language Model to Guide Image Style Transfer for Target Object(http://arxiv.org/abs/2311.13562)
• Summary:

  Image style transfer occupies an important place in both computer graphics and computer vision. However, most current methods require reference to stylized images and cannot individually stylize specific objects. To overcome this limitation, we propose the "Soulstyler" framework, which allows users to guide the stylization of specific objects in an image through simple textual descriptions. We introduce a large language model to parse the text and identify stylization goals and specific styles. Combined with a CLIP-based semantic visual embedding encoder, the model understands and matches text and image content. We also introduce a novel localized text-image block matching loss that ensures that style transfer is performed only on specified target objects, while non-target regions remain in their original style. Experimental results demonstrate that our model is able to accurately perform style transfer on target objects according to textual descriptions without affecting the style of background regions. Our code will be available at https://github.com/yisuanwang/Soulstyler.

Title: Visual In-Context Prompting. (arXiv:2311.13601v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.13601
• Code URL: null
• Copy Paste: [[2311.13601]] Visual In-Context Prompting(http://arxiv.org/abs/2311.13601)
• Summary:

  In-context prompting in large language models (LLMs) has become a prevalent approach to improve zero-shot capabilities, but this idea is less explored in the vision domain. Existing visual prompting methods focus on referring segmentation to segment the most relevant object, falling short of addressing many generic vision tasks like open-set segmentation and detection. In this paper, we introduce a universal visual in-context prompting framework for both tasks. In particular, we build on top of an encoder-decoder architecture, and develop a versatile prompt encoder to support a variety of prompts like strokes, boxes, and points. We further enhance it to take an arbitrary number of reference image segments as the context. Our extensive explorations show that the proposed visual in-context prompting elicits extraordinary referring and generic segmentation capabilities to refer and detect, yielding competitive performance to close-set in-domain datasets and showing promising results on many open-set segmentation datasets. By joint training on COCO and SA-1B, our model achieves $57.7$ PQ on COCO and $23.2$ PQ on ADE20K. Code will be available at https://github.com/UX-Decoder/DINOv.

Title: Overview of Current Applications of Large Language Models in Various Medical Specialities. (arXiv:2311.12882v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.12882
• Code URL: null
• Copy Paste: [[2311.12882]] Overview of Current Applications of Large Language Models in Various Medical Specialities(http://arxiv.org/abs/2311.12882)
• Summary:

  This paper gives an overview of the latest applications of Large Language Models (LLMs) in the healthcare sector, highlighting their transformative role in enhancing medical care quality. By processing vast amounts of data from diverse medical domains, LLMs have become pivotal in assisting doctors, healthcare providers, and patients. We explore their utilization in various medical specialties, such as cancer diagnostics, dentistry, nephrology, dermatology, etc. The paper includes the LLM methodologies applied in various medical specialties, different data types in the medical domains and the relevant input formatting for LLMs, along with practical use-cases of LLMs in the healthcare domain.

Title: Beyond Text: Unveiling Multimodal Proficiency of Large Language Models with MultiAPI Benchmark. (arXiv:2311.13053v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.13053
• Code URL: null
• Copy Paste: [[2311.13053]] Beyond Text: Unveiling Multimodal Proficiency of Large Language Models with MultiAPI Benchmark(http://arxiv.org/abs/2311.13053)
• Summary:

  The proliferation of Large Language Models like ChatGPT has significantly advanced language understanding and generation, impacting a broad spectrum of applications. However, these models predominantly excel in text-based tasks, overlooking the complexity of real-world multimodal information. This study introduces MultiAPI, a pioneering comprehensive large-scale API benchmark dataset aimed at expanding LLMs' proficiency in multimodal contexts. Developed collaboratively through ChatGPT, MultiAPI consists of 235 diverse API calls and 2,038 contextual prompts, offering a unique platform evaluation of tool-augmented LLMs handling multimodal tasks. Through comprehensive experiments, our findings reveal that while LLMs demonstrate proficiency in API call decision-making, they face challenges in domain identification, function selection, and argument generation. What's more, we surprisingly notice that auxiliary context can actually impair the performance. An in-depth error analysis paves the way for a new paradigm to address these challenges, suggesting a potential direction for future LLM research.

Title: Enhancing Logical Reasoning in Large Language Models to Facilitate Legal Applications. (arXiv:2311.13095v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.13095
• Code URL: null
• Copy Paste: [[2311.13095]] Enhancing Logical Reasoning in Large Language Models to Facilitate Legal Applications(http://arxiv.org/abs/2311.13095)
• Summary:

  Language serves as a vehicle for conveying thought, enabling communication among individuals. The ability to distinguish between diverse concepts, identify fairness and injustice, and comprehend a range of legal notions fundamentally relies on logical reasoning. Large Language Models (LLMs) attempt to emulate human language understanding and generation, but their competency in logical reasoning remains limited. This paper seeks to address the philosophical question: How can we effectively teach logical reasoning to LLMs while maintaining a deep understanding of the intricate relationship between language and logic? By focusing on bolstering LLMs' capabilities in logical reasoning, we aim to expand their applicability in law and other logic-intensive disciplines. To this end, we propose a Reinforcement Learning from Logical Feedback (RLLF) approach, which serves as a potential framework for refining LLMs' reasoning capacities. Through RLLF and a revised evaluation methodology, we explore new avenues for research in this domain and contribute to the development of LLMs capable of handling complex legal reasoning tasks while acknowledging the fundamental connection between language and logic.

Title: Towards Better Parameter-Efficient Fine-Tuning for Large Language Models: A Position Paper. (arXiv:2311.13126v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.13126
• Code URL: null
• Copy Paste: [[2311.13126]] Towards Better Parameter-Efficient Fine-Tuning for Large Language Models: A Position Paper(http://arxiv.org/abs/2311.13126)
• Summary:

  This paper delves into the pressing need in Parameter-Efficient Fine-Tuning (PEFT) for Large Language Models (LLMs). While LLMs possess remarkable capabilities, their extensive parameter requirements and associated computational demands hinder their practicality and scalability for real-world applications. Our position paper highlights current states and the necessity of further studying into the topic, and recognizes significant challenges and open issues that must be addressed to fully harness the powerful abilities of LLMs. These challenges encompass novel efficient PEFT architectures, PEFT for different learning settings, PEFT combined with model compression techniques, and the exploration of PEFT for multi-modal LLMs. By presenting this position paper, we aim to stimulate further research and foster discussions surrounding more efficient and accessible PEFT for LLMs.

Title: LIMIT: Less Is More for Instruction Tuning Across Evaluation Paradigms. (arXiv:2311.13133v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.13133
• Code URL: null
• Copy Paste: [[2311.13133]] LIMIT: Less Is More for Instruction Tuning Across Evaluation Paradigms(http://arxiv.org/abs/2311.13133)
• Summary:

  Large Language Models are traditionally finetuned on large instruction datasets. However recent studies suggest that small, high-quality datasets can suffice for general purpose instruction following. This lack of consensus surrounding finetuning best practices is in part due to rapidly diverging approaches to LLM evaluation. In this study, we ask whether a small amount of diverse finetuning samples can improve performance on both traditional perplexity-based NLP benchmarks, and on open-ended, model-based evaluation. We finetune open-source MPT-7B and MPT-30B models on instruction finetuning datasets of various sizes ranging from 1k to 60k samples. We find that subsets of 1k-6k instruction finetuning samples are sufficient to achieve good performance on both (1) traditional NLP benchmarks and (2) model-based evaluation. Finally, we show that mixing textbook-style and open-ended QA finetuning datasets optimizes performance on both evaluation paradigms.

Title: AS-LLM: When Algorithm Selection Meets Large Language Model. (arXiv:2311.13184v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.13184
• Code URL: null
• Copy Paste: [[2311.13184]] AS-LLM: When Algorithm Selection Meets Large Language Model(http://arxiv.org/abs/2311.13184)
• Summary:

  Algorithm selection aims to identify the most suitable algorithm for solving a specific problem before execution, which has become a critical process of the AutoML. Current mainstream algorithm selection techniques rely heavily on feature representations of various problems and employ the performance of each algorithm as supervised information. However, there is a significant research gap concerning the consideration of algorithm features. This gap is primarily attributed to the inherent complexity of algorithms, making it particularly challenging to find a universally effective feature extraction method that is applicable across a diverse range of algorithms. Unfortunately, neglecting this aspect undoubtedly impacts the accuracy of algorithm selection and indirectly necessitates an increased volume of problem data for training purposes. This paper takes a significant stride towards addressing this gap by proposing an approach that integrates algorithm representation into the algorithm selection process. Specifically, our proposed model employs distinct modules to extract representations of both problems and algorithms, where the algorithm representation leverages the capabilities of pre-trained LLMs in the realm of code comprehension. Following the extraction of embedding vectors for both algorithms and problems, the most suitable algorithm is determined through calculations of matching degrees. Our experiments not only validate the effectiveness of the proposed model but also showcase the performance of different embedded pre-trained LLMs, which suggests that the proposed algorithm selection framework holds the potential to serve as a baseline task for evaluating the code representation capabilities of LLMs.

Title: Enhancing Uncertainty-Based Hallucination Detection with Stronger Focus. (arXiv:2311.13230v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.13230
• Code URL: null
• Copy Paste: [[2311.13230]] Enhancing Uncertainty-Based Hallucination Detection with Stronger Focus(http://arxiv.org/abs/2311.13230)
• Summary:

  Large Language Models (LLMs) have gained significant popularity for their impressive performance across diverse fields. However, LLMs are prone to hallucinate untruthful or nonsensical outputs that fail to meet user expectations in many real-world applications. Existing works for detecting hallucinations in LLMs either rely on external knowledge for reference retrieval or require sampling multiple responses from the LLM for consistency verification, making these methods costly and inefficient. In this paper, we propose a novel reference-free, uncertainty-based method for detecting hallucinations in LLMs. Our approach imitates human focus in factuality checking from three aspects: 1) focus on the most informative and important keywords in the given text; 2) focus on the unreliable tokens in historical context which may lead to a cascade of hallucinations; and 3) focus on the token properties such as token type and token frequency. Experimental results on relevant datasets demonstrate the effectiveness of our proposed method, which achieves state-of-the-art performance across all the evaluation metrics and eliminates the need for additional information.

Title: On the Calibration of Large Language Models and Alignment. (arXiv:2311.13240v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.13240
• Code URL: null
• Copy Paste: [[2311.13240]] On the Calibration of Large Language Models and Alignment(http://arxiv.org/abs/2311.13240)
• Summary:

  As large language models attract increasing attention and find widespread application, concurrent challenges of reliability also arise at the same time. Confidence calibration, an effective analysis method for gauging the reliability of deep models, serves as a crucial tool for assessing and improving their reliability. However, such investigation has been comparatively underexplored. In this work, we conduct a systematic examination of the calibration of aligned language models throughout the entire construction process, including pretraining and alignment training. At each stage, we investigate how different training settings, such as parameter scales and training data, affect model calibration. To thoroughly assess model calibration, we evaluate models on three most concerned aspects: generation, factuality and understanding. Our work sheds light on whether popular LLMs are well-calibrated and how the training process influences model calibration.

Title: Mitigating Large Language Model Hallucinations via Autonomous Knowledge Graph-based Retrofitting. (arXiv:2311.13314v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.13314
• Code URL: null
• Copy Paste: [[2311.13314]] Mitigating Large Language Model Hallucinations via Autonomous Knowledge Graph-based Retrofitting(http://arxiv.org/abs/2311.13314)
• Summary:

  Incorporating factual knowledge in knowledge graph is regarded as a promising approach for mitigating the hallucination of large language models (LLMs). Existing methods usually only use the user's input to query the knowledge graph, thus failing to address the factual hallucination generated by LLMs during its reasoning process. To address this problem, this paper proposes Knowledge Graph-based Retrofitting (KGR), a new framework that incorporates LLMs with KGs to mitigate factual hallucination during the reasoning process by retrofitting the initial draft responses of LLMs based on the factual knowledge stored in KGs. Specifically, KGR leverages LLMs to extract, select, validate, and retrofit factual statements within the model-generated responses, which enables an autonomous knowledge verifying and refining procedure without any additional manual efforts. Experiments show that KGR can significantly improve the performance of LLMs on factual QA benchmarks especially when involving complex reasoning processes, which demonstrates the necessity and effectiveness of KGR in mitigating hallucination and enhancing the reliability of LLMs.

Title: Drilling Down into the Discourse Structure with LLMs for Long Document Question Answering. (arXiv:2311.13565v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.13565
• Code URL: null
• Copy Paste: [[2311.13565]] Drilling Down into the Discourse Structure with LLMs for Long Document Question Answering(http://arxiv.org/abs/2311.13565)
• Summary:

  We address the task of evidence retrieval for long document question answering, which involves locating relevant paragraphs within a document to answer a question. We aim to assess the applicability of large language models (LLMs) in the task of zero-shot long document evidence retrieval, owing to their unprecedented performance across various NLP tasks. However, currently the LLMs can consume limited context lengths as input, thus providing document chunks as inputs might overlook the global context while missing out on capturing the inter-segment dependencies. Moreover, directly feeding the large input sets can incur significant computational costs, particularly when processing the entire document (and potentially incurring monetary expenses with enterprise APIs like OpenAI's GPT variants). To address these challenges, we propose a suite of techniques that exploit the discourse structure commonly found in documents. By utilizing this structure, we create a condensed representation of the document, enabling a more comprehensive understanding and analysis of relationships between different parts. We retain $99.6\%$ of the best zero-shot approach's performance, while processing only $26\%$ of the total tokens used by the best approach in the information seeking evidence retrieval setup. We also show how our approach can be combined with \textit{self-ask} reasoning agent to achieve best zero-shot performance in complex multi-hop question answering, just $\approx 4\%$ short of zero-shot performance using gold evidence.

segmentation

Title: DeepCompass: AI-driven Location-Orientation Synchronization for Navigating Platforms. (arXiv:2311.12805v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12805
• Code URL: null
• Copy Paste: [[2311.12805]] DeepCompass: AI-driven Location-Orientation Synchronization for Navigating Platforms(http://arxiv.org/abs/2311.12805)
• Summary:

  In current navigating platforms, the user's orientation is typically estimated based on the difference between two consecutive locations. In other words, the orientation cannot be identified until the second location is taken. This asynchronous location-orientation identification often leads to our real-life question: Why does my navigator tell the wrong direction of my car at the beginning? We propose DeepCompass to identify the user's orientation by bridging the gap between the street-view and the user-view images. First, we explore suitable model architectures and design corresponding input configuration. Second, we demonstrate artificial transformation techniques (e.g., style transfer and road segmentation) to minimize the disparity between the street-view and the user's real-time experience. We evaluate DeepCompass with extensive evaluation in various driving conditions. DeepCompass does not require additional hardware and is also not susceptible to external interference, in contrast to magnetometer-based navigator. This highlights the potential of DeepCompass as an add-on to existing sensor-based orientation detection methods.

Title: Tool Wear Segmentation in Blanking Processes with Fully Convolutional Networks based Digital Image Processing. (arXiv:2311.12841v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12841
• Code URL: null
• Copy Paste: [[2311.12841]] Tool Wear Segmentation in Blanking Processes with Fully Convolutional Networks based Digital Image Processing(http://arxiv.org/abs/2311.12841)
• Summary:

  The extend of tool wear significantly affects blanking processes and has a decisive impact on product quality and productivity. For this reason, numerous scientists have addressed their research to wear monitoring systems in order to identify or even predict critical wear at an early stage. Existing approaches are mainly based on indirect monitoring using time series, which are used to detect critical wear states via thresholds or machine learning models. Nevertheless, differentiation between types of wear phenomena affecting the tool during blanking as well as quantification of worn surfaces is still limited in practice. While time series data provides partial insights into wear occurrence and evolution, direct monitoring techniques utilizing image data offer a more comprehensive perspective and increased robustness when dealing with varying process parameters. However, acquiring and processing this data in real-time is challenging. In particular, high dynamics combined with increasing strokes rates as well as the high dimensionality of image data have so far prevented the development of direct image-based monitoring systems. For this reason, this paper demonstrates how high-resolution images of tools at 600 spm can be captured and subsequently processed using semantic segmentation deep learning algorithms, more precisely Fully Convolutional Networks (FCN). 125,000 images of the tool are taken from successive strokes, and microscope images are captured to investigate the worn surfaces. Based on findings from the microscope images, selected images are labeled pixel by pixel according to their wear condition and used to train a FCN (U-Net).

Title: A Novel Defocus-Blur Region Detection Approach Based on DCT Feature and PCNN Structure. (arXiv:2311.12845v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12845
• Code URL: null
• Copy Paste: [[2311.12845]] A Novel Defocus-Blur Region Detection Approach Based on DCT Feature and PCNN Structure(http://arxiv.org/abs/2311.12845)
• Summary:

  The motion or out-of-focus effect in digital images is the main reason for the blurred regions in defocused-blurred images. It may adversely affect various image features such as texture, pixel, and region. Therefore, it is important to detect in-focused objects in defocused-blurred images after the segmentation of blurred and non-blurred regions. The state-of-the-art techniques are prone to noisy pixels, and their local descriptors for developing segmentation metrics are also complex. To address these issues, this research, therefore, proposed a novel and hybrid-focused detection approach based on Discrete Cosine Transform (DCT) coefficients and PC Neural Net (PCNN) structure. The proposed approach partially resolves the limitations of the existing contrast schemes to detect in-focused smooth objects from the out-of-focused smooth regions in the defocus dataset. The visual and quantitative evaluation illustrates that the proposed approach outperformed in terms of accuracy and efficiency to referenced algorithms. The highest F-score of the proposed approach on Zhao's dataset is 0.7940 whereas on Shi's dataset is 0.9178.

Title: Q-Seg: Quantum Annealing-based Unsupervised Image Segmentation. (arXiv:2311.12912v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12912
• Code URL: null
• Copy Paste: [[2311.12912]] Q-Seg: Quantum Annealing-based Unsupervised Image Segmentation(http://arxiv.org/abs/2311.12912)
• Summary:

  In this study, we present Q-Seg, a novel unsupervised image segmentation method based on quantum annealing, tailored for existing quantum hardware. We formulate the pixel-wise segmentation problem, which assimilates spectral and spatial information of the image, as a graph-cut optimization task. Our method efficiently leverages the interconnected qubit topology of the D-Wave Advantage device, offering superior scalability over existing quantum approaches and outperforming state-of-the-art classical methods. Our empirical evaluations on synthetic datasets reveal that Q-Seg offers better runtime performance against the classical optimizer Gurobi. Furthermore, we evaluate our method on segmentation of Earth Observation images, an area of application where the amount of labeled data is usually very limited. In this case, Q-Seg demonstrates near-optimal results in flood mapping detection with respect to classical supervised state-of-the-art machine learning methods. Also, Q-Seg provides enhanced segmentation for forest coverage compared to existing annotated masks. Thus, Q-Seg emerges as a viable alternative for real-world applications using available quantum hardware, particularly in scenarios where the lack of labeled data and computational runtime are critical.

Title: AI for Agriculture: the Comparison of Semantic Segmentation Methods for Crop Mapping with Sentinel-2 Imagery. (arXiv:2311.12993v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.12993
• Code URL: null
• Copy Paste: [[2311.12993]] AI for Agriculture: the Comparison of Semantic Segmentation Methods for Crop Mapping with Sentinel-2 Imagery(http://arxiv.org/abs/2311.12993)
• Summary:

  Crop mapping is one of the most common tasks in artificial intelligence for agriculture due to higher food demands from a growing population and increased awareness of climate change. In case of vineyards, the texture is very important for crop segmentation: with higher resolution satellite imagery the texture is easily detected by majority of state-of-the-art algorithms. However, this task becomes increasingly more difficult as the resolution of satellite imagery decreases and the information about the texture becomes unavailable. In this paper we aim to explore the main machine learning methods that can be used with freely available satellite imagery and discuss how and when they can be applied for vineyard segmentation problem. We assess the effectiveness of various widely-used machine learning techniques and offer guidance on selecting the most suitable model for specific scenarios.

Title: FuseNet: Self-Supervised Dual-Path Network for Medical Image Segmentation. (arXiv:2311.13069v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.13069
• Code URL: null
• Copy Paste: [[2311.13069]] FuseNet: Self-Supervised Dual-Path Network for Medical Image Segmentation(http://arxiv.org/abs/2311.13069)
• Summary:

  Semantic segmentation, a crucial task in computer vision, often relies on labor-intensive and costly annotated datasets for training. In response to this challenge, we introduce FuseNet, a dual-stream framework for self-supervised semantic segmentation that eliminates the need for manual annotation. FuseNet leverages the shared semantic dependencies between the original and augmented images to create a clustering space, effectively assigning pixels to semantically related clusters, and ultimately generating the segmentation map. Additionally, FuseNet incorporates a cross-modal fusion technique that extends the principles of CLIP by replacing textual data with augmented images. This approach enables the model to learn complex visual representations, enhancing robustness against variations similar to CLIP's text invariance. To further improve edge alignment and spatial consistency between neighboring pixels, we introduce an edge refinement loss. This loss function considers edge information to enhance spatial coherence, facilitating the grouping of nearby pixels with similar visual features. Extensive experiments on skin lesion and lung segmentation datasets demonstrate the effectiveness of our method. \href{https://github.com/xmindflow/FuseNet}{Codebase.}

Title: DAE-Net: Deforming Auto-Encoder for fine-grained shape co-segmentation. (arXiv:2311.13125v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.13125
• Code URL: null
• Copy Paste: [[2311.13125]] DAE-Net: Deforming Auto-Encoder for fine-grained shape co-segmentation(http://arxiv.org/abs/2311.13125)
• Summary:

  We present an unsupervised 3D shape co-segmentation method which learns a set of deformable part templates from a shape collection. To accommodate structural variations in the collection, our network composes each shape by a selected subset of template parts which are affine-transformed. To maximize the expressive power of the part templates, we introduce a per-part deformation network to enable the modeling of diverse parts with substantial geometry variations, while imposing constraints on the deformation capacity to ensure fidelity to the originally represented parts. We also propose a training scheme to effectively overcome local minima. Architecturally, our network is a branched autoencoder, with a CNN encoder taking a voxel shape as input and producing per-part transformation matrices, latent codes, and part existence scores, and the decoder outputting point occupancies to define the reconstruction loss. Our network, coined DAE-Net for Deforming Auto-Encoder, can achieve unsupervised 3D shape co-segmentation that yields fine-grained, compact, and meaningful parts that are consistent across diverse shapes. We conduct extensive experiments on the ShapeNet Part dataset, DFAUST, and an animal subset of Objaverse to show superior performance over prior methods.

Title: Test-Time Augmentation for 3D Point Cloud Classification and Segmentation. (arXiv:2311.13152v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.13152
• Code URL: null
• Copy Paste: [[2311.13152]] Test-Time Augmentation for 3D Point Cloud Classification and Segmentation(http://arxiv.org/abs/2311.13152)
• Summary:

  Data augmentation is a powerful technique to enhance the performance of a deep learning task but has received less attention in 3D deep learning. It is well known that when 3D shapes are sparsely represented with low point density, the performance of the downstream tasks drops significantly. This work explores test-time augmentation (TTA) for 3D point clouds. We are inspired by the recent revolution of learning implicit representation and point cloud upsampling, which can produce high-quality 3D surface reconstruction and proximity-to-surface, respectively. Our idea is to leverage the implicit field reconstruction or point cloud upsampling techniques as a systematic way to augment point cloud data. Mainly, we test both strategies by sampling points from the reconstructed results and using the sampled point cloud as test-time augmented data. We show that both strategies are effective in improving accuracy. We observed that point cloud upsampling for test-time augmentation can lead to more significant performance improvement on downstream tasks such as object classification and segmentation on the ModelNet40, ShapeNet, ScanObjectNN, and SemanticKITTI datasets, especially for sparse point clouds.

Title: Self-guided Few-shot Semantic Segmentation for Remote Sensing Imagery Based on Large Vision Models. (arXiv:2311.13200v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.13200
• Code URL: null
• Copy Paste: [[2311.13200]] Self-guided Few-shot Semantic Segmentation for Remote Sensing Imagery Based on Large Vision Models(http://arxiv.org/abs/2311.13200)
• Summary:

  The Segment Anything Model (SAM) exhibits remarkable versatility and zero-shot learning abilities, owing largely to its extensive training data (SA-1B). Recognizing SAM's dependency on manual guidance given its category-agnostic nature, we identified unexplored potential within few-shot semantic segmentation tasks for remote sensing imagery. This research introduces a structured framework designed for the automation of few-shot semantic segmentation. It utilizes the SAM model and facilitates a more efficient generation of semantically discernible segmentation outcomes. Central to our methodology is a novel automatic prompt learning approach, leveraging prior guided masks to produce coarse pixel-wise prompts for SAM. Extensive experiments on the DLRSD datasets underline the superiority of our approach, outperforming other available few-shot methodologies.

Title: DA-STC: Domain Adaptive Video Semantic Segmentation via Spatio-Temporal Consistency. (arXiv:2311.13254v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.13254
• Code URL: https://github.com/zhe-sapi/da-stc
• Copy Paste: [[2311.13254]] DA-STC: Domain Adaptive Video Semantic Segmentation via Spatio-Temporal Consistency(http://arxiv.org/abs/2311.13254)
• Summary:

  Video semantic segmentation is a pivotal aspect of video representation learning. However, significant domain shifts present a challenge in effectively learning invariant spatio-temporal features across the labeled source domain and unlabeled target domain for video semantic segmentation. To solve the challenge, we propose a novel DA-STC method for domain adaptive video semantic segmentation, which incorporates a bidirectional multi-level spatio-temporal fusion module and a category-aware spatio-temporal feature alignment module to facilitate consistent learning for domain-invariant features. Firstly, we perform bidirectional spatio-temporal fusion at the image sequence level and shallow feature level, leading to the construction of two fused intermediate video domains. This prompts the video semantic segmentation model to consistently learn spatio-temporal features of shared patch sequences which are influenced by domain-specific contexts, thereby mitigating the feature gap between the source and target domain. Secondly, we propose a category-aware feature alignment module to promote the consistency of spatio-temporal features, facilitating adaptation to the target domain. Specifically, we adaptively aggregate the domain-specific deep features of each category along spatio-temporal dimensions, which are further constrained to achieve cross-domain intra-class feature alignment and inter-class feature separation. Extensive experiments demonstrate the effectiveness of our method, which achieves state-of-the-art mIOUs on multiple challenging benchmarks. Furthermore, we extend the proposed DA-STC to the image domain, where it also exhibits superior performance for domain adaptive semantic segmentation. The source code and models will be made available at \url{https://github.com/ZHE-SAPI/DA-STC}.

Title: SegVol: Universal and Interactive Volumetric Medical Image Segmentation. (arXiv:2311.13385v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.13385
• Code URL: https://github.com/baai-dcai/segvol
• Copy Paste: [[2311.13385]] SegVol: Universal and Interactive Volumetric Medical Image Segmentation(http://arxiv.org/abs/2311.13385)
• Summary:

  Precise image segmentation provides clinical study with meaningful and well-structured information. Despite the remarkable progress achieved in medical image segmentation, there is still an absence of foundation segmentation model that can segment a wide range of anatomical categories with easy user interaction. In this paper, we propose a universal and interactive volumetric medical image segmentation model, named SegVol. By training on 90k unlabeled Computed Tomography (CT) volumes and 6k labeled CTs, this foundation model supports the segmentation of over 200 anatomical categories using semantic and spatial prompts. Extensive experiments verify that SegVol outperforms the state of the art by a large margin on multiple segmentation benchmarks. Notably, on three challenging lesion datasets, our method achieves around 20% higher Dice score than nnU-Net. The model and data are publicly available at: https://github.com/BAAI-DCAI/SegVol.

Title: Hybrid Whale-Mud-Ring Optimization for Precise Color Skin Cancer Image Segmentation. (arXiv:2311.13512v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.13512
• Code URL: null
• Copy Paste: [[2311.13512]] Hybrid Whale-Mud-Ring Optimization for Precise Color Skin Cancer Image Segmentation(http://arxiv.org/abs/2311.13512)
• Summary:

  Timely identification and treatment of rapidly progressing skin cancers can significantly contribute to the preservation of patients' health and well-being. Dermoscopy, a dependable and accessible tool, plays a pivotal role in the initial stages of skin cancer detection. Consequently, the effective processing of digital dermoscopy images holds significant importance in elevating the accuracy of skin cancer diagnoses. Multilevel thresholding is a key tool in medical imaging that extracts objects within the image to facilitate its analysis. In this paper, an enhanced version of the Mud Ring Algorithm hybridized with the Whale Optimization Algorithm, named WMRA, is proposed. The proposed approach utilizes bubble-net attack and mud ring strategy to overcome stagnation in local optima and obtain optimal thresholds. The experimental results show that WMRA is powerful against a cluster of recent methods in terms of fitness, Peak Signal to Noise Ratio (PSNR), and Mean Square Error (MSE).