2023-12-25

secure

Title: MetaAID 2.5: A Secure Framework for Developing Metaverse Applications via Large Language Models. (arXiv:2312.14480v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.14480
• Code URL: null
• Copy Paste: [[2312.14480]] MetaAID 2(http://arxiv.org/abs/2312.14480)
• Summary:

  Large language models (LLMs) are increasingly being used in Metaverse environments to generate dynamic and realistic content and to control the behavior of non-player characters (NPCs). However, the cybersecurity concerns associated with LLMs have become increasingly prominent. Previous research has primarily focused on patching system vulnerabilities to enhance cybersecurity, but these approaches are not well-suited to the Metaverse, where the virtual space is more complex, LLMs are vulnerable, and ethical user interaction is critical. Moreover, the scope of cybersecurity in the Metaverse is expected to expand significantly. This paper proposes a method for enhancing cybersecurity through the simulation of user interaction with LLMs. Our goal is to educate users and strengthen their defense capabilities through exposure to a comprehensive simulation system. This system includes extensive Metaverse cybersecurity Q&A and attack simulation scenarios. By engaging with these, users will improve their ability to recognize and withstand risks. Additionally, to address the ethical implications of user input, we propose using LLMs as evaluators to assess user content across five dimensions. We further adapt the models through vocabulary expansion training to better understand personalized inputs and emoticons. We conduct experiments on multiple LLMs and find that our approach is effective.

Title: Concurrent Asynchronous Byzantine Agreement in Expected-Constant Rounds, Revisited. (arXiv:2312.14506v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.14506
• Code URL: null
• Copy Paste: [[2312.14506]] Concurrent Asynchronous Byzantine Agreement in Expected-Constant Rounds, Revisited(http://arxiv.org/abs/2312.14506)
• Summary:

  It is well known that without randomization, Byzantine agreement (BA) requires a linear number of rounds in the synchronous setting, while it is flat out impossible in the asynchronous setting. The primitive which allows to bypass the above limitation is known as oblivious common coin (OCC). It allows parties to agree with constant probability on a random coin, where agreement is oblivious, i.e., players are not aware whether or not agreement has been achieved.

  The starting point of our work is the observation that no known protocol exists for information-theoretic multi-valued OCC with optimal resiliency in the asynchronous setting (with eventual message delivery). This apparent hole in the literature is particularly problematic, as multi-valued OCC is implicitly or explicitly used in several constructions.

  In this paper, we present the first information-theoretic multi-valued OCC protocol in the asynchronous setting with optimal resiliency, i.e., tolerating $t < n/3$ corruptions, thereby filling this important gap. Further, our protocol efficiently implements OCC with an exponential-size domain, a property which is not even achieved by known constructions in the simpler, synchronous setting.

  We then turn to the problem of round-preserving parallel composition of asynchronous BA. A protocol for this task was proposed by Ben-Or and El-Yaniv [Distributed Computing '03]. Their construction, however, is flawed in several ways. Thus, as a second contribution, we provide a simpler, more modular protocol for the above task. Finally, and as a contribution of independent interest, we provide proofs in Canetti's Universal Composability framework; this makes our work the first one offering composability guarantees, which are important as BA is a core building block of secure multi-party computation protocols.

security

Title: Enhancing Ethereum's Security with LUMEN, a Novel Zero-Knowledge Protocol Generating Transparent and Efficient zk-SNARKs. (arXiv:2312.14159v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.14159
• Code URL: null
• Copy Paste: [[2312.14159]] Enhancing Ethereum's Security with LUMEN, a Novel Zero-Knowledge Protocol Generating Transparent and Efficient zk-SNARKs(http://arxiv.org/abs/2312.14159)
• Summary:

  This paper proposes a novel recursive polynomial commitment scheme (PCS) and a new polynomial interactive oracle proof (PIOP) protocol, which compile into efficient and transparent zk-SNARKs (zero-knowledge succinct non-interactive arguments of knowledge). The Ethereum blockchain utilizes zero-knowledge Rollups (ZKR) to improve its scalability (the ability to handle a large number of transactions), and ZKR uses zk-SNARKs to validate transactions. The currently used zk-SNARKs rely on a trusted setup ceremony, where a group of participants uses secret information about transactions to generate the public parameters necessary to verify the zk-SNARKs. This introduces a security risk into Ethereum's system. Thus, researchers have been developing transparent zk-SNARKs (which do not require a trusted setup), but those are not as efficient as non-transparent zk-SNARKs, so ZKRs do not use them. In this research, I developed LUMEN, a set of novel algorithms that generate transparent zk-SNARKs that improve Ethereum's security without sacrificing its efficiency. Various techniques were creatively incorporated into LUMEN, including groups with hidden orders, Lagrange basis polynomials, and an amortization strategy. I wrote mathematical proofs for LUMEN that convey its completeness, soundness and zero-knowledgeness, and implemented LUMEN by writing around $8000$ lines of Rust and Python code, which conveyed the practicality of LUMEN. Moreover, my implementation revealed the efficiency of LUMEN (measured in proof size, proof computation time, and verification time), which surpasses the efficiency of existing transparent zk-SNARKs and is on par with that of non-transparent zk-SNARKs. Therefore, LUMEN is a promising solution to improve Ethereum's security while maintaining its efficiency.

Title: A Review on Searchable Encryption Functionality and the Evaluation of Homomorphic Encryption. (arXiv:2312.14434v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.14434
• Code URL: null
• Copy Paste: [[2312.14434]] A Review on Searchable Encryption Functionality and the Evaluation of Homomorphic Encryption(http://arxiv.org/abs/2312.14434)
• Summary:

  Cloud Service Providers, such as Google Cloud Platform, Microsoft Azure, or Amazon Web Services, offer continuously evolving cloud services. It is a growing industry. Businesses, such as Netflix and PayPal, rely on the Cloud for data storage, computing power, and other services. For businesses, the cloud reduces costs, provides flexibility, and allows for growth. However, there are security and privacy concerns regarding the Cloud. Because Cloud services are accessed through the internet, hackers and attackers could possibly access the servers from anywhere. To protect data in the Cloud, it should be encrypted before it is uploaded, it should be protected in storage and also in transit. On the other hand, data owners may need to access their encrypted data. It may also need to be altered, updated, deleted, read, searched, or shared with others. If data is decrypted in the Cloud, sensitive data is exposed and could be exposed and misused. One solution is to leave the data in its encrypted form and use Searchable Encryption (SE) which operates on encrypted data. The functionality of SE has improved since its inception and research continues to explore ways to improve SE. This paper reviews the functionality of Searchable Encryption, mostly related to Cloud services, in the years 2019 to 2023, and evaluates one of its schemes, Fully Homomorphic Encryption. Overall, it seems that research is at the point where SE efficiency is increased as multiple functionalities are aggregated and tested.

Title: Navigating the Concurrency Landscape: A Survey of Race Condition Vulnerability Detectors. (arXiv:2312.14479v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.14479
• Code URL: null
• Copy Paste: [[2312.14479]] Navigating the Concurrency Landscape: A Survey of Race Condition Vulnerability Detectors(http://arxiv.org/abs/2312.14479)
• Summary:

  As technology continues to advance and we usher in the era of Industry 5.0, there has been a profound paradigm shift in operating systems, file systems, web, and network applications. The conventional utilization of multiprocessing and multicore systems has made concurrent programming increasingly pervasive. However, this transformation has brought about a new set of issues known as concurrency bugs, which, due to their wide prevalence in concurrent programs, have led to severe failures and potential security exploits. Over the past two decades, numerous researchers have dedicated their efforts to unveiling, detecting, mitigating, and preventing these bugs, with the last decade witnessing a surge in research within this domain. Among the spectrum of concurrency bugs, data races or race condition vulnerabilities stand out as the most prevalent, accounting for a staggering 80\% of all concurrency bugs. This survey paper is focused on the realm of race condition bug detectors. We systematically categorize these detectors based on the diverse methodologies they employ. Additionally, we delve into the techniques and algorithms associated with race detection, tracing the evolution of this field over time. Furthermore, we shed light on the application of fuzzing techniques in the detection of race condition vulnerabilities. By reviewing these detectors and their static analyses, we draw conclusions and outline potential future research directions, including enhancing accuracy, performance, applicability, and comprehensiveness in race condition vulnerability detection.

Title: Evaluating the Security and Privacy Risk Postures of Virtual Assistants. (arXiv:2312.14633v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.14633
• Code URL: null
• Copy Paste: [[2312.14633]] Evaluating the Security and Privacy Risk Postures of Virtual Assistants(http://arxiv.org/abs/2312.14633)
• Summary:

  Virtual assistants (VAs) have seen increased use in recent years due to their ease of use for daily tasks. Despite their growing prevalence, their security and privacy implications are still not well understood. To address this gap, we conducted a study to evaluate the security and privacy postures of eight widely used voice assistants: Alexa, Braina, Cortana, Google Assistant, Kalliope, Mycroft, Hound, and Extreme. We used three vulnerability testing tools, AndroBugs, RiskInDroid, and MobSF, to assess the security and privacy of these VAs. Our analysis focused on five areas: code, access control, tracking, binary analysis, and sensitive data confidentiality. The results revealed that these VAs are vulnerable to a range of security threats, including not validating SSL certificates, executing raw SQL queries, and using a weak mode of the AES algorithm. These vulnerabilities could allow malicious actors to gain unauthorized access to users' personal information. This study is a first step toward understanding the risks associated with these technologies and provides a foundation for future research to develop more secure and privacy-respecting VAs.

Title: Cybersecurity in Motion: A Survey of Challenges and Requirements for Future Test Facilities of CAVs. (arXiv:2312.14687v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.14687
• Code URL: null
• Copy Paste: [[2312.14687]] Cybersecurity in Motion: A Survey of Challenges and Requirements for Future Test Facilities of CAVs(http://arxiv.org/abs/2312.14687)
• Summary:

  The way we travel is changing rapidly, and Cooperative Intelligent Transportation Systems (C-ITSs) are at the forefront of this evolution. However, the adoption of C-ITSs introduces new risks and challenges, making cybersecurity a top priority for ensuring safety and reliability. Building on this premise, this paper presents an envisaged Cybersecurity Centre of Excellence (CSCE) designed to bolster research, testing, and evaluation of the cybersecurity of C-ITSs. We explore the design, functionality, and challenges of CSCE's testing facilities, outlining the technological, security, and societal requirements. Through a thorough survey and analysis, we assess the effectiveness of these systems in detecting and mitigating potential threats, highlighting their flexibility to adapt to future C-ITSs. Finally, we identify current unresolved challenges in various C-ITS domains, with the aim of motivating further research into the cybersecurity of C-ITSs.

privacy

Title: AdvCloak: Customized Adversarial Cloak for Privacy Protection. (arXiv:2312.14407v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.14407
• Code URL: null
• Copy Paste: [[2312.14407]] AdvCloak: Customized Adversarial Cloak for Privacy Protection(http://arxiv.org/abs/2312.14407)
• Summary:

  With extensive face images being shared on social media, there has been a notable escalation in privacy concerns. In this paper, we propose AdvCloak, an innovative framework for privacy protection using generative models. AdvCloak is designed to automatically customize class-wise adversarial masks that can maintain superior image-level naturalness while providing enhanced feature-level generalization ability. Specifically, AdvCloak sequentially optimizes the generative adversarial networks by employing a two-stage training strategy. This strategy initially focuses on adapting the masks to the unique individual faces via image-specific training and then enhances their feature-level generalization ability to diverse facial variations of individuals via person-specific training. To fully utilize the limited training data, we combine AdvCloak with several general geometric modeling methods, to better describe the feature subspace of source identities. Extensive quantitative and qualitative evaluations on both common and celebrity datasets demonstrate that AdvCloak outperforms existing state-of-the-art methods in terms of efficiency and effectiveness.

Title: Data Cooperatives for Identity Attestations. (arXiv:2312.14158v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.14158
• Code URL: null
• Copy Paste: [[2312.14158]] Data Cooperatives for Identity Attestations(http://arxiv.org/abs/2312.14158)
• Summary:

  Data cooperatives with fiduciary obligations to members provide a useful source of truthful information regarding a given member whose personal data is managed by the cooperative. Since one of the main propositions the cooperative model is to protect the data privacy of members, we explore the notion of blinded attestations in which the identity of the subject is removed from the attestations issued by the cooperative regarding one of its members. This is performed at the request of the individual member. We propose the use of a legal entity to countersign the blinded attestation, one that has an attorney-client relationship with the cooperative, and which can henceforth become the legal point of contact for inquiries regarding the individual related to the attribute being attested. There are several use-cases for this feature, including the Funds Travel Rule in transactions in digital assets, and the protection of privacy in decentralized social networks.

Title: Noisy Measurements Are Important, the Design of Census Products Is Much More Important. (arXiv:2312.14191v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.14191
• Code URL: null
• Copy Paste: [[2312.14191]] Noisy Measurements Are Important, the Design of Census Products Is Much More Important(http://arxiv.org/abs/2312.14191)
• Summary:

  McCartan et al. (2023) call for "making differential privacy work for census data users." This commentary explains why the 2020 Census Noisy Measurement Files (NMFs) are not the best focus for that plea. The August 2021 letter from 62 prominent researchers asking for production of the direct output of the differential privacy system deployed for the 2020 Census signaled the engagement of the scholarly community in the design of decennial census data products. NMFs, the raw statistics produced by the 2020 Census Disclosure Avoidance System before any post-processing, are one component of that design--the query strategy output. The more important component is the query workload output--the statistics released to the public. Optimizing the query workload--the Redistricting Data (P.L. 94-171) Summary File, specifically--could allow the privacy-loss budget to be more effectively managed. There could be fewer noisy measurements, no post-processing bias, and direct estimates of the uncertainty from disclosure avoidance for each published statistic.

Title: HElium: A Language and Compiler for Fully Homomorphic Encryption with Support for Proxy Re-Encryption. (arXiv:2312.14250v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.14250
• Code URL: null
• Copy Paste: [[2312.14250]] HElium: A Language and Compiler for Fully Homomorphic Encryption with Support for Proxy Re-Encryption(http://arxiv.org/abs/2312.14250)
• Summary:

  Privacy-preserving analysis of confidential data can increase the value of such data and even improve peoples' lives. Fully homomorphic encryption (FHE) can enable privacy-preserving analysis. However, FHE adds a large amount of computational overhead and its efficient use requires a high level of expertise. Compilers can automate certain aspects such as parameterization and circuit optimizations. This in turn makes FHE accessible to non-cryptographers. Yet, multi-party scenarios remain complicated and exclude many promising use cases such as analyses of large amounts of health records for medical research. Proxy re-encryption (PRE), a technique that allows the conversion of data from multiple sources to a joint encryption key, can enable FHE for multi-party scenarios. Today, there are no optimizing compilers for FHE with PRE capabilities.

  We propose HElium, the first optimizing FHE compiler with native support for proxy re-encryption. HElium features HEDSL, a domain-specific language (DSL) specifically designed for multi-party scenarios. By tracking encryption keys and transforming the computation circuit during compilation, HElium minimizes the number of expensive PRE operations. We evaluate the effectiveness of HElium's optimizations based on the real-world use case of the tumor recurrence rate, a well-known subject of medical research. Our empirical evaluation shows that HElium substantially reduces the overhead introduced through complex PRE operations, an effect that increases for larger amounts of input data.

Title: DP-AdamBC: Your DP-Adam Is Actually DP-SGD (Unless You Apply Bias Correction). (arXiv:2312.14334v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.14334
• Code URL: https://github.com/ubc-systopia/DP-AdamBC
• Copy Paste: [[2312.14334]] DP-AdamBC: Your DP-Adam Is Actually DP-SGD (Unless You Apply Bias Correction)(http://arxiv.org/abs/2312.14334)
• Summary:

  The Adam optimizer is a popular choice in contemporary deep learning, due to its strong empirical performance. However we observe that in privacy sensitive scenarios, the traditional use of Differential Privacy (DP) with the Adam optimizer leads to sub-optimal performance on several tasks. We find that this performance degradation is due to a DP bias in Adam's second moment estimator, introduced by the addition of independent noise in the gradient computation to enforce DP guarantees. This DP bias leads to a different scaling for low variance parameter updates, that is inconsistent with the behavior of non-private Adam. We propose DP-AdamBC, an optimization algorithm which removes the bias in the second moment estimation and retrieves the expected behaviour of Adam. Empirically, DP-AdamBC significantly improves the optimization performance of DP-Adam by up to 3.5% in final accuracy in image, text, and graph node classification tasks.

Title: A Generalized Shuffle Framework for Privacy Amplification: Strengthening Privacy Guarantees and Enhancing Utility. (arXiv:2312.14388v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.14388
• Code URL: null
• Copy Paste: [[2312.14388]] A Generalized Shuffle Framework for Privacy Amplification: Strengthening Privacy Guarantees and Enhancing Utility(http://arxiv.org/abs/2312.14388)
• Summary:

  The shuffle model of local differential privacy is an advanced method of privacy amplification designed to enhance privacy protection with high utility. It achieves this by randomly shuffling sensitive data, making linking individual data points to specific individuals more challenging. However, most existing studies have focused on the shuffle model based on $(\epsilon_0,0)$-Locally Differentially Private (LDP) randomizers, with limited consideration for complex scenarios such as $(\epsilon_0,\delta_0)$-LDP or personalized LDP (PLDP). This hinders a comprehensive understanding of the shuffle model's potential and limits its application in various settings. To bridge this research gap, we propose a generalized shuffle framework that can be applied to any $(\epsilon_i,\delta_i)$-PLDP setting with personalized privacy parameters. This generalization allows for a broader exploration of the privacy-utility trade-off and facilitates the design of privacy-preserving analyses in diverse contexts. We prove that shuffled $(\epsilon_i,\delta_i)$-PLDP process approximately preserves $\mu$-Gaussian Differential Privacy with \mu = \sqrt{\frac{2}{\sum_{i=1}^{n} \frac{1-\delta_i}{1+e^{\epsilon_i}}-\max_{i}{\frac{1-\delta_{i}}{1+e^{\epsilon_{i}}}}}}. $

  This approach allows us to avoid the limitations and potential inaccuracies associated with inequality estimations. To strengthen the privacy guarantee, we improve the lower bound by utilizing hypothesis testing} instead of relying on rough estimations like the Chernoff bound or Hoeffding's inequality. Furthermore, extensive comparative evaluations clearly show that our approach outperforms existing methods in achieving strong central privacy guarantees while preserving the utility of the global model. We have also carefully designed corresponding algorithms for average function, frequency estimation, and stochastic gradient descent.

protect

Title: Auto311: A Confidence-guided Automated System for Non-emergency Call. (arXiv:2312.14185v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.14185
• Code URL: null
• Copy Paste: [[2312.14185]] Auto311: A Confidence-guided Automated System for Non-emergency Call(http://arxiv.org/abs/2312.14185)
• Summary:

  Emergency and non-emergency response systems are essential services provided by local governments and critical to protecting lives, the environment, and property. The effective handling of (non-)emergency calls is critical for public safety and well-being. By reducing the burden through non-emergency callers, residents in critical need of assistance through 911 will receive a fast and effective response. Collaborating with the Department of Emergency Communications (DEC) in Nashville, we analyzed 11,796 non-emergency call recordings and developed Auto311, the first automated system to handle 311 non-emergency calls, which (1) effectively and dynamically predicts ongoing non-emergency incident types to generate tailored case reports during the call; (2) itemizes essential information from dialogue contexts to complete the generated reports; and (3) strategically structures system-caller dialogues with optimized confidence. We used real-world data to evaluate the system's effectiveness and deployability. The experimental results indicate that the system effectively predicts incident type with an average F-1 score of 92.54%. Moreover, the system successfully itemizes critical information from relevant contexts to complete reports, evincing a 0.93 average consistency score compared to the ground truth. Additionally, emulations demonstrate that the system effectively decreases conversation turns as the utterance size gets more extensive and categorizes the ongoing call with 94.49% mean accuracy.

Title: Optimizing Heat Alert Issuance for Public Health in the United States with Reinforcement Learning. (arXiv:2312.14196v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.14196
• Code URL: null
• Copy Paste: [[2312.14196]] Optimizing Heat Alert Issuance for Public Health in the United States with Reinforcement Learning(http://arxiv.org/abs/2312.14196)
• Summary:

  Alerting the public when heat may harm their health is a crucial service, especially considering that extreme heat events will be more frequent under climate change. Current practice for issuing heat alerts in the US does not take advantage of modern data science methods for optimizing local alert criteria. Specifically, application of reinforcement learning (RL) has the potential to inform more health-protective policies, accounting for regional and sociodemographic heterogeneity as well as sequential dependence of alerts. In this work, we formulate the issuance of heat alerts as a sequential decision making problem and develop modifications to the RL workflow to address challenges commonly encountered in environmental health settings. Key modifications include creating a simulator that pairs hierarchical Bayesian modeling of low-signal health effects with sampling of real weather trajectories (exogenous features), constraining the total number of alerts issued as well as preventing alerts on less-hot days, and optimizing location-specific policies. Post-hoc contrastive analysis offers insights into scenarios when using RL for heat alert issuance may protect public health better than the current or alternative policies. This work contributes to a broader movement of advancing data-driven policy optimization for public health and climate change adaptation.

defense

Title: Elevating Defenses: Bridging Adversarial Training and Watermarking for Model Resilience. (arXiv:2312.14260v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.14260
• Code URL: null
• Copy Paste: [[2312.14260]] Elevating Defenses: Bridging Adversarial Training and Watermarking for Model Resilience(http://arxiv.org/abs/2312.14260)
• Summary:

  Machine learning models are being used in an increasing number of critical applications; thus, securing their integrity and ownership is critical. Recent studies observed that adversarial training and watermarking have a conflicting interaction. This work introduces a novel framework to integrate adversarial training with watermarking techniques to fortify against evasion attacks and provide confident model verification in case of intellectual property theft. We use adversarial training together with adversarial watermarks to train a robust watermarked model. The key intuition is to use a higher perturbation budget to generate adversarial watermarks compared to the budget used for adversarial training, thus avoiding conflict. We use the MNIST and Fashion-MNIST datasets to evaluate our proposed technique on various model stealing attacks. The results obtained consistently outperform the existing baseline in terms of robustness performance and further prove the resilience of this defense against pruning and fine-tuning removal attacks.

attack

Title: AutoAugment Input Transformation for Highly Transferable Targeted Attacks. (arXiv:2312.14218v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.14218
• Code URL: null
• Copy Paste: [[2312.14218]] AutoAugment Input Transformation for Highly Transferable Targeted Attacks(http://arxiv.org/abs/2312.14218)
• Summary:

  Deep Neural Networks (DNNs) are widely acknowledged to be susceptible to adversarial examples, wherein imperceptible perturbations are added to clean examples through diverse input transformation attacks. However, these methods originally designed for non-targeted attacks exhibit low success rates in targeted attacks. Recent targeted adversarial attacks mainly pay attention to gradient optimization, attempting to find the suitable perturbation direction. However, few of them are dedicated to input transformation.In this work, we observe a positive correlation between the logit/probability of the target class and diverse input transformation methods in targeted attacks. To this end, we propose a novel targeted adversarial attack called AutoAugment Input Transformation (AAIT). Instead of relying on hand-made strategies, AAIT searches for the optimal transformation policy from a transformation space comprising various operations. Then, AAIT crafts adversarial examples using the found optimal transformation policy to boost the adversarial transferability in targeted attacks. Extensive experiments conducted on CIFAR-10 and ImageNet-Compatible datasets demonstrate that the proposed AAIT surpasses other transfer-based targeted attacks significantly.

Title: Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models. (arXiv:2312.14197v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.14197
• Code URL: null
• Copy Paste: [[2312.14197]] Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models(http://arxiv.org/abs/2312.14197)
• Summary:

  Recent remarkable advancements in large language models (LLMs) have led to their widespread adoption in various applications. A key feature of these applications is the combination of LLMs with external content, where user instructions and third-party content are combined to create prompts for LLM processing. These applications, however, are vulnerable to indirect prompt injection attacks, where malicious instructions embedded within external content compromise LLM's output, causing their responses to deviate from user expectations. Despite the discovery of this security issue, no comprehensive analysis of indirect prompt injection attacks on different LLMs is available due to the lack of a benchmark. Furthermore, no effective defense has been proposed.

  In this work, we introduce the first benchmark, BIPIA, to measure the robustness of various LLMs and defenses against indirect prompt injection attacks. Our experiments reveal that LLMs with greater capabilities exhibit more vulnerable to indirect prompt injection attacks for text tasks, resulting in a higher ASR. We hypothesize that indirect prompt injection attacks are mainly due to the LLMs' inability to distinguish between instructions and external content. Based on this conjecture, we propose four black-box methods based on prompt learning and a white-box defense methods based on fine-tuning with adversarial training to enable LLMs to distinguish between instructions and external content and ignore instructions in the external content. Our experimental results show that our black-box defense methods can effectively reduce ASR but cannot completely thwart indirect prompt injection attacks, while our white-box defense method can reduce ASR to nearly zero with little adverse impact on the LLM's performance on general tasks. We hope that our benchmark and defenses can inspire future work in this important area.

Title: Exploiting Novel GPT-4 APIs. (arXiv:2312.14302v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.14302
• Code URL: https://github.com/alignmentresearch/gpt-4-novel-apis-attacks
• Copy Paste: [[2312.14302]] Exploiting Novel GPT-4 APIs(http://arxiv.org/abs/2312.14302)
• Summary:

  Language model attacks typically assume one of two extreme threat models: full white-box access to model weights, or black-box access limited to a text generation API. However, real-world APIs are often more flexible than just text generation: these APIs expose ``gray-box'' access leading to new threat vectors. To explore this, we red-team three new functionalities exposed in the GPT-4 APIs: fine-tuning, function calling and knowledge retrieval. We find that fine-tuning a model on as few as 15 harmful examples or 100 benign examples can remove core safeguards from GPT-4, enabling a range of harmful outputs. Furthermore, we find that GPT-4 Assistants readily divulge the function call schema and can be made to execute arbitrary function calls. Finally, we find that knowledge retrieval can be hijacked by injecting instructions into retrieval documents. These vulnerabilities highlight that any additions to the functionality exposed by an API can create new vulnerabilities.

Title: Find the Lady: Permutation and Re-Synchronization of Deep Neural Networks. (arXiv:2312.14182v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.14182
• Code URL: https://github.com/carldesousatrias/findthelady
• Copy Paste: [[2312.14182]] Find the Lady: Permutation and Re-Synchronization of Deep Neural Networks(http://arxiv.org/abs/2312.14182)
• Summary:

  Deep neural networks are characterized by multiple symmetrical, equi-loss solutions that are redundant. Thus, the order of neurons in a layer and feature maps can be given arbitrary permutations, without affecting (or minimally affecting) their output. If we shuffle these neurons, or if we apply to them some perturbations (like fine-tuning) can we put them back in the original order i.e. re-synchronize? Is there a possible corruption threat? Answering these questions is important for applications like neural network white-box watermarking for ownership tracking and integrity verification. We advance a method to re-synchronize the order of permuted neurons. Our method is also effective if neurons are further altered by parameter pruning, quantization, and fine-tuning, showing robustness to integrity attacks. Additionally, we provide theoretical and practical evidence for the usual means to corrupt the integrity of the model, resulting in a solution to counter it. We test our approach on popular computer vision datasets and models, and we illustrate the threat and our countermeasure on a popular white-box watermarking method.

Title: Adversarial Infrared Curves: An Attack on Infrared Pedestrian Detectors in the Physical World. (arXiv:2312.14217v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.14217
• Code URL: null
• Copy Paste: [[2312.14217]] Adversarial Infrared Curves: An Attack on Infrared Pedestrian Detectors in the Physical World(http://arxiv.org/abs/2312.14217)
• Summary:

  Deep neural network security is a persistent concern, with considerable research on visible light physical attacks but limited exploration in the infrared domain. Existing approaches, like white-box infrared attacks using bulb boards and QR suits, lack realism and stealthiness. Meanwhile, black-box methods with cold and hot patches often struggle to ensure robustness. To bridge these gaps, we propose Adversarial Infrared Curves (AdvIC). Using Particle Swarm Optimization, we optimize two Bezier curves and employ cold patches in the physical realm to introduce perturbations, creating infrared curve patterns for physical sample generation. Our extensive experiments confirm AdvIC's effectiveness, achieving 94.8\% and 67.2\% attack success rates for digital and physical attacks, respectively. Stealthiness is demonstrated through a comparative analysis, and robustness assessments reveal AdvIC's superiority over baseline methods. When deployed against diverse advanced detectors, AdvIC achieves an average attack success rate of 76.8\%, emphasizing its robust nature. we explore adversarial defense strategies against AdvIC and examine its impact under various defense mechanisms. Given AdvIC's substantial security implications for real-world vision-based applications, urgent attention and mitigation efforts are warranted.

Title: Asymmetric Bias in Text-to-Image Generation with Adversarial Attacks. (arXiv:2312.14440v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.14440
• Code URL: null
• Copy Paste: [[2312.14440]] Asymmetric Bias in Text-to-Image Generation with Adversarial Attacks(http://arxiv.org/abs/2312.14440)
• Summary:

  The widespread use of Text-to-Image (T2I) models in content generation requires careful examination of their safety, including their robustness to adversarial attacks. Despite extensive research into this, the reasons for their effectiveness are underexplored. This paper presents an empirical study on adversarial attacks against T2I models, focusing on analyzing factors associated with attack success rates (ASRs). We introduce a new attack objective - entity swapping using adversarial suffixes and two gradient-based attack algorithms. Human and automatic evaluations reveal the asymmetric nature of ASRs on entity swap: for example, it is easier to replace "human" with "robot" in the prompt "a human dancing in the rain." with an adversarial suffix but is significantly harder in reverse. We further propose probing metrics to establish indicative signals from the model's beliefs to the adversarial ASR. We identify conditions resulting in a 60% success probability for adversarial attacks and others where this likelihood drops below 5%.

Title: Attacking Byzantine Robust Aggregation in High Dimensions. (arXiv:2312.14461v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.14461
• Code URL: https://github.com/sarthak-choudhary/hidra
• Copy Paste: [[2312.14461]] Attacking Byzantine Robust Aggregation in High Dimensions(http://arxiv.org/abs/2312.14461)
• Summary:

  Training modern neural networks or models typically requires averaging over a sample of high-dimensional vectors. Poisoning attacks can skew or bias the average vectors used to train the model, forcing the model to learn specific patterns or avoid learning anything useful. Byzantine robust aggregation is a principled algorithmic defense against such biasing. Robust aggregators can bound the maximum bias in computing centrality statistics, such as mean, even when some fraction of inputs are arbitrarily corrupted. Designing such aggregators is challenging when dealing with high dimensions. However, the first polynomial-time algorithms with strong theoretical bounds on the bias have recently been proposed. Their bounds are independent of the number of dimensions, promising a conceptual limit on the power of poisoning attacks in their ongoing arms race against defenses.

  In this paper, we show a new attack called HIDRA on practical realization of strong defenses which subverts their claim of dimension-independent bias. HIDRA highlights a novel computational bottleneck that has not been a concern of prior information-theoretic analysis. Our experimental evaluation shows that our attacks almost completely destroy the model performance, whereas existing attacks with the same goal fail to have much effect. Our findings leave the arms race between poisoning attacks and provable defenses wide open.

Title: MEAOD: Model Extraction Attack against Object Detectors. (arXiv:2312.14677v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.14677
• Code URL: null
• Copy Paste: [[2312.14677]] MEAOD: Model Extraction Attack against Object Detectors(http://arxiv.org/abs/2312.14677)
• Summary:

  The widespread use of deep learning technology across various industries has made deep neural network models highly valuable and, as a result, attractive targets for potential attackers. Model extraction attacks, particularly query-based model extraction attacks, allow attackers to replicate a substitute model with comparable functionality to the victim model and present a significant threat to the confidentiality and security of MLaaS platforms. While many studies have explored threats of model extraction attacks against classification models in recent years, object detection models, which are more frequently used in real-world scenarios, have received less attention. In this paper, we investigate the challenges and feasibility of query-based model extraction attacks against object detection models and propose an effective attack method called MEAOD. It selects samples from the attacker-possessed dataset to construct an efficient query dataset using active learning and enhances the categories with insufficient objects. We additionally improve the extraction effectiveness by updating the annotations of the query dataset. According to our gray-box and black-box scenarios experiments, we achieve an extraction performance of over 70% under the given condition of a 10k query budget.

Title: Understanding the Regularity of Self-Attention with Optimal Transport. (arXiv:2312.14820v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.14820
• Code URL: null
• Copy Paste: [[2312.14820]] Understanding the Regularity of Self-Attention with Optimal Transport(http://arxiv.org/abs/2312.14820)
• Summary:

  Transformers and their multi-head attention mechanism have completely changed the machine learning landscape in just a few years, by outperforming state-of-art models in a wide range of domains. Still, little is known about their robustness from a theoretical perspective. We tackle this problem by studying the local Lipschitz constant of self-attention, that provides an attack-agnostic way of measuring the robustness of a neural network. We adopt a measure-theoretic framework, by viewing inputs as probability measures equipped with the Wasserstein distance. This allows us to generalize attention to inputs of infinite length, and to derive an upper bound and a lower bound on the Lipschitz constant of self-attention on compact sets. The lower bound significantly improves prior results, and grows more than exponentially with the radius of the compact set, which rules out the possibility of obtaining robustness guarantees without any additional constraint on the input space. Our results also point out that measures with a high local Lipschitz constant are typically made of a few diracs, with a very unbalanced distribution of mass. Finally, we analyze the stability of self-attention under perturbations that change the number of tokens, which appears to be a natural question in the measure-theoretic framework. In particular, we show that for some inputs, attacks that duplicate tokens before perturbing them are more efficient than attacks that simply move tokens. We call this phenomenon mass splitting.

robust

Title: TextFusion: Unveiling the Power of Textual Semantics for Controllable Image Fusion. (arXiv:2312.14209v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.14209
• Code URL: null
• Copy Paste: [[2312.14209]] TextFusion: Unveiling the Power of Textual Semantics for Controllable Image Fusion(http://arxiv.org/abs/2312.14209)
• Summary:

  Advanced image fusion methods are devoted to generating the fusion results by aggregating the complementary information conveyed by the source images. However, the difference in the source-specific manifestation of the imaged scene content makes it difficult to design a robust and controllable fusion process. We argue that this issue can be alleviated with the help of higher-level semantics, conveyed by the text modality, which should enable us to generate fused images for different purposes, such as visualisation and downstream tasks, in a controllable way. This is achieved by exploiting a vision-and-language model to build a coarse-to-fine association mechanism between the text and image signals. With the guidance of the association maps, an affine fusion unit is embedded in the transformer network to fuse the text and vision modalities at the feature level. As another ingredient of this work, we propose the use of textual attention to adapt image quality assessment to the fusion task. To facilitate the implementation of the proposed text-guided fusion paradigm, and its adoption by the wider research community, we release a text-annotated image fusion dataset IVT. Extensive experiments demonstrate that our approach (TextFusion) consistently outperforms traditional appearance-based fusion methods. Our code and dataset will be publicly available on the project homepage.

Title: StyleRetoucher: Generalized Portrait Image Retouching with GAN Priors. (arXiv:2312.14389v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.14389
• Code URL: null
• Copy Paste: [[2312.14389]] StyleRetoucher: Generalized Portrait Image Retouching with GAN Priors(http://arxiv.org/abs/2312.14389)
• Summary:

  Creating fine-retouched portrait images is tedious and time-consuming even for professional artists. There exist automatic retouching methods, but they either suffer from over-smoothing artifacts or lack generalization ability. To address such issues, we present StyleRetoucher, a novel automatic portrait image retouching framework, leveraging StyleGAN's generation and generalization ability to improve an input portrait image's skin condition while preserving its facial details. Harnessing the priors of pretrained StyleGAN, our method shows superior robustness: a). performing stably with fewer training samples and b). generalizing well on the out-domain data. Moreover, by blending the spatial features of the input image and intermediate features of the StyleGAN layers, our method preserves the input characteristics to the largest extent. We further propose a novel blemish-aware feature selection mechanism to effectively identify and remove the skin blemishes, improving the image skin condition. Qualitative and quantitative evaluations validate the great generalization capability of our method. Further experiments show StyleRetoucher's superior performance to the alternative solutions in the image retouching task. We also conduct a user perceptive study to confirm the superior retouching performance of our method over the existing state-of-the-art alternatives.

Title: GROOD: GRadient-aware Out-Of-Distribution detection in interpolated manifolds. (arXiv:2312.14427v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.14427
• Code URL: null
• Copy Paste: [[2312.14427]] GROOD: GRadient-aware Out-Of-Distribution detection in interpolated manifolds(http://arxiv.org/abs/2312.14427)
• Summary:

  Deep neural networks (DNNs) often fail silently with over-confident predictions on out-of-distribution (OOD) samples, posing risks in real-world deployments. Existing techniques predominantly emphasize either the feature representation space or the gradient norms computed with respect to DNN parameters, yet they overlook the intricate gradient distribution and the topology of classification regions. To address this gap, we introduce GRadient-aware Out-Of-Distribution detection in interpolated manifolds (GROOD), a novel framework that relies on the discriminative power of gradient space to distinguish between in-distribution (ID) and OOD samples. To build this space, GROOD relies on class prototypes together with a prototype that specifically captures OOD characteristics. Uniquely, our approach incorporates a targeted mix-up operation at an early intermediate layer of the DNN to refine the separation of gradient spaces between ID and OOD samples. We quantify OOD detection efficacy using the distance to the nearest neighbor gradients derived from the training set, yielding a robust OOD score. Experimental evaluations substantiate that the introduction of targeted input mix-upamplifies the separation between ID and OOD in the gradient space, yielding impressive results across diverse datasets. Notably, when benchmarked against ImageNet-1k, GROOD surpasses the established robustness of state-of-the-art baselines. Through this work, we establish the utility of leveraging gradient spaces and class prototypes for enhanced OOD detection for DNN in image classification.

Title: DSAP: Analyzing Bias Through Demographic Comparison of Datasets. (arXiv:2312.14626v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.14626
• Code URL: https://github.com/irisdominguez/dsap
• Copy Paste: [[2312.14626]] DSAP: Analyzing Bias Through Demographic Comparison of Datasets(http://arxiv.org/abs/2312.14626)
• Summary:

  In the last few years, Artificial Intelligence systems have become increasingly widespread. Unfortunately, these systems can share many biases with human decision-making, including demographic biases. Often, these biases can be traced back to the data used for training, where large uncurated datasets have become the norm. Despite our knowledge of these biases, we still lack general tools to detect and quantify them, as well as to compare the biases in different datasets. Thus, in this work, we propose DSAP (Demographic Similarity from Auxiliary Profiles), a two-step methodology for comparing the demographic composition of two datasets. DSAP can be deployed in three key applications: to detect and characterize demographic blind spots and bias issues across datasets, to measure dataset demographic bias in single datasets, and to measure dataset demographic shift in deployment scenarios. An essential feature of DSAP is its ability to robustly analyze datasets without explicit demographic labels, offering simplicity and interpretability for a wide range of situations. To show the usefulness of the proposed methodology, we consider the Facial Expression Recognition task, where demographic bias has previously been found. The three applications are studied over a set of twenty datasets with varying properties. The code is available at https://github.com/irisdominguez/DSAP.

Title: Global Occlusion-Aware Transformer for Robust Stereo Matching. (arXiv:2312.14650v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.14650
• Code URL: https://github.com/magicboomliu/goat
• Copy Paste: [[2312.14650]] Global Occlusion-Aware Transformer for Robust Stereo Matching(http://arxiv.org/abs/2312.14650)
• Summary:

  Despite the remarkable progress facilitated by learning-based stereo-matching algorithms, the performance in the ill-conditioned regions, such as the occluded regions, remains a bottleneck. Due to the limited receptive field, existing CNN-based methods struggle to handle these ill-conditioned regions effectively. To address this issue, this paper introduces a novel attention-based stereo-matching network called Global Occlusion-Aware Transformer (GOAT) to exploit long-range dependency and occlusion-awareness global context for disparity estimation. In the GOAT architecture, a parallel disparity and occlusion estimation module PDO is proposed to estimate the initial disparity map and the occlusion mask using a parallel attention mechanism. To further enhance the disparity estimates in the occluded regions, an occlusion-aware global aggregation module (OGA) is proposed. This module aims to refine the disparity in the occluded regions by leveraging restricted global correlation within the focus scope of the occluded areas. Extensive experiments were conducted on several public benchmark datasets including SceneFlow, KITTI 2015, and Middlebury. The results show that the proposed GOAT demonstrates outstanding performance among all benchmarks, particularly in the occluded regions.

Title: Density Uncertainty Quantification with NeRF-Ensembles: Impact of Data and Scene Constraints. (arXiv:2312.14664v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.14664
• Code URL: null
• Copy Paste: [[2312.14664]] Density Uncertainty Quantification with NeRF-Ensembles: Impact of Data and Scene Constraints(http://arxiv.org/abs/2312.14664)
• Summary:

  In the fields of computer graphics, computer vision and photogrammetry, Neural Radiance Fields (NeRFs) are a major topic driving current research and development. However, the quality of NeRF-generated 3D scene reconstructions and subsequent surface reconstructions, heavily relies on the network output, particularly the density. Regarding this critical aspect, we propose to utilize NeRF-Ensembles that provide a density uncertainty estimate alongside the mean density. We demonstrate that data constraints such as low-quality images and poses lead to a degradation of the training process, increased density uncertainty and decreased predicted density. Even with high-quality input data, the density uncertainty varies based on scene constraints such as acquisition constellations, occlusions and material properties. NeRF-Ensembles not only provide a tool for quantifying the uncertainty but exhibit two promising advantages: Enhanced robustness and artifact removal. Through the utilization of NeRF-Ensembles instead of single NeRFs, small outliers are removed, yielding a smoother output with improved completeness of structures. Furthermore, applying percentile-based thresholds on density uncertainty outliers proves to be effective for the removal of large (foggy) artifacts in post-processing. We conduct our methodology on 3 different datasets: (i) synthetic benchmark dataset, (ii) real benchmark dataset, (iii) real data under realistic recording conditions and sensors.

Title: PoseGen: Learning to Generate 3D Human Pose Dataset with NeRF. (arXiv:2312.14915v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.14915
• Code URL: null
• Copy Paste: [[2312.14915]] PoseGen: Learning to Generate 3D Human Pose Dataset with NeRF(http://arxiv.org/abs/2312.14915)
• Summary:

  This paper proposes an end-to-end framework for generating 3D human pose datasets using Neural Radiance Fields (NeRF). Public datasets generally have limited diversity in terms of human poses and camera viewpoints, largely due to the resource-intensive nature of collecting 3D human pose data. As a result, pose estimators trained on public datasets significantly underperform when applied to unseen out-of-distribution samples. Previous works proposed augmenting public datasets by generating 2D-3D pose pairs or rendering a large amount of random data. Such approaches either overlook image rendering or result in suboptimal datasets for pre-trained models. Here we propose PoseGen, which learns to generate a dataset (human 3D poses and images) with a feedback loss from a given pre-trained pose estimator. In contrast to prior art, our generated data is optimized to improve the robustness of the pre-trained model. The objective of PoseGen is to learn a distribution of data that maximizes the prediction error of a given pre-trained model. As the learned data distribution contains OOD samples of the pre-trained model, sampling data from such a distribution for further fine-tuning a pre-trained model improves the generalizability of the model. This is the first work that proposes NeRFs for 3D human data generation. NeRFs are data-driven and do not require 3D scans of humans. Therefore, using NeRF for data generation is a new direction for convenient user-specific data generation. Our extensive experiments show that the proposed PoseGen improves two baseline models (SPIN and HybrIK) on four datasets with an average 6% relative improvement.

Title: Robust Knowledge Extraction from Large Language Models using Social Choice Theory. (arXiv:2312.14877v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.14877
• Code URL: null
• Copy Paste: [[2312.14877]] Robust Knowledge Extraction from Large Language Models using Social Choice Theory(http://arxiv.org/abs/2312.14877)
• Summary:

  Large-language models (LLMs) have the potential to support a wide range of applications like conversational agents, creative writing, text improvement, and general query answering. However, they are ill-suited for query answering in high-stake domains like medicine because they generate answers at random and their answers are typically not robust - even the same query can result in different answers when prompted multiple times. In order to improve the robustness of LLM queries, we propose using ranking queries repeatedly and to aggregate the queries using methods from social choice theory. We study ranking queries in diagnostic settings like medical and fault diagnosis and discuss how the Partial Borda Choice function from the literature can be applied to merge multiple query results. We discuss some additional interesting properties in our setting and evaluate the robustness of our approach empirically.

Title: Can Machines Learn Robustly, Privately, and Efficiently?. (arXiv:2312.14712v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.14712
• Code URL: null
• Copy Paste: [[2312.14712]] Can Machines Learn Robustly, Privately, and Efficiently?(http://arxiv.org/abs/2312.14712)
• Summary:

  The success of machine learning (ML) applications relies on vast datasets and distributed architectures, which, as they grow, present challenges for ML. In real-world scenarios, where data often contains sensitive information, issues like data poisoning and hardware failures are common. Ensuring privacy and robustness is vital for the broad adoption of ML in public life. This paper examines the costs associated with achieving these objectives in distributed architectures. We overview the meanings of privacy and robustness in distributed ML, and clarify how they can be achieved efficiently in isolation. However, we contend that the integration of these objectives entails a notable compromise in computational efficiency. We delve into this intricate balance, exploring the challenges and solutions for privacy, robustness, and computational efficiency in ML applications.

Title: Invariant Anomaly Detection under Distribution Shifts: A Causal Perspective. (arXiv:2312.14329v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.14329
• Code URL: https://github.com/joaocarv/invariant-anomaly-detection
• Copy Paste: [[2312.14329]] Invariant Anomaly Detection under Distribution Shifts: A Causal Perspective(http://arxiv.org/abs/2312.14329)
• Summary:

  Anomaly detection (AD) is the machine learning task of identifying highly discrepant abnormal samples by solely relying on the consistency of the normal training samples. Under the constraints of a distribution shift, the assumption that training samples and test samples are drawn from the same distribution breaks down. In this work, by leveraging tools from causal inference we attempt to increase the resilience of anomaly detection models to different kinds of distribution shifts. We begin by elucidating a simple yet necessary statistical property that ensures invariant representations, which is critical for robust AD under both domain and covariate shifts. From this property, we derive a regularization term which, when minimized, leads to partial distribution invariance across environments. Through extensive experimental evaluation on both synthetic and real-world tasks, covering a range of six different AD methods, we demonstrated significant improvements in out-of-distribution performance. Under both covariate and domain shift, models regularized with our proposed term showed marked increased robustness. Code is available at: https://github.com/JoaoCarv/invariant-anomaly-detection.

Title: Room Occupancy Prediction: Exploring the Power of Machine Learning and Temporal Insights. (arXiv:2312.14426v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.14426
• Code URL: null
• Copy Paste: [[2312.14426]] Room Occupancy Prediction: Exploring the Power of Machine Learning and Temporal Insights(http://arxiv.org/abs/2312.14426)
• Summary:

  Energy conservation in buildings is a paramount concern to combat greenhouse gas emissions and combat climate change. The efficient management of room occupancy, involving actions like lighting control and climate adjustment, is a pivotal strategy to curtail energy consumption. In contexts where surveillance technology isn't viable, non-intrusive sensors are employed to estimate room occupancy. In this study, we present a predictive framework for room occupancy that leverages a diverse set of machine learning models, with Random Forest consistently achieving the highest predictive accuracy. Notably, this dataset encompasses both temporal and spatial dimensions, revealing a wealth of information. Intriguingly, our framework demonstrates robust performance even in the absence of explicit temporal modeling. These findings underscore the remarkable predictive power of traditional machine learning models. The success can be attributed to the presence of feature redundancy, the simplicity of linear spatial and temporal patterns, and the advantages of high-frequency data sampling. While these results are compelling, it's essential to remain open to the possibility that explicitly modeling the temporal dimension could unlock deeper insights or further enhance predictive capabilities in specific scenarios. In summary, our research not only validates the effectiveness of our prediction framework for continuous and classification tasks but also underscores the potential for improvements through the inclusion of temporal aspects. The study highlights the promise of machine learning in shaping energy-efficient practices and room occupancy management.

Title: Data is Moody: Discovering Data Modification Rules from Process Event Logs. (arXiv:2312.14571v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.14571
• Code URL: null
• Copy Paste: [[2312.14571]] Data is Moody: Discovering Data Modification Rules from Process Event Logs(http://arxiv.org/abs/2312.14571)
• Summary:

  Although event logs are a powerful source to gain insight about the behavior of the underlying business process, existing work primarily focuses on finding patterns in the activity sequences of an event log, while ignoring event attribute data. Event attribute data has mostly been used to predict event occurrences and process outcome, but the state of the art neglects to mine succinct and interpretable rules how event attribute data changes during process execution. Subgroup discovery and rule-based classification approaches lack the ability to capture the sequential dependencies present in event logs, and thus lead to unsatisfactory results with limited insight into the process behavior.

  Given an event log, we are interested in finding accurate yet succinct and interpretable if-then rules how the process modifies data. We formalize the problem in terms of the Minimum Description Length (MDL) principle, by which we choose the model with the best lossless description of the data. Additionally, we propose the greedy Moody algorithm to efficiently search for rules. By extensive experiments on both synthetic and real-world data, we show Moody indeed finds compact and interpretable rules, needs little data for accurate discovery, and is robust to noise.

Title: Balancing Energy Efficiency and Distributional Robustness in Over-the-Air Federated Learning. (arXiv:2312.14638v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.14638
• Code URL: null
• Copy Paste: [[2312.14638]] Balancing Energy Efficiency and Distributional Robustness in Over-the-Air Federated Learning(http://arxiv.org/abs/2312.14638)
• Summary:

  The growing number of wireless edge devices has magnified challenges concerning energy, bandwidth, latency, and data heterogeneity. These challenges have become bottlenecks for distributed learning. To address these issues, this paper presents a novel approach that ensures energy efficiency for distributionally robust federated learning (FL) with over air computation (AirComp). In this context, to effectively balance robustness with energy efficiency, we introduce a novel client selection method that integrates two complementary insights: a deterministic one that is designed for energy efficiency, and a probabilistic one designed for distributional robustness. Simulation results underscore the efficacy of the proposed algorithm, revealing its superior performance compared to baselines from both robustness and energy efficiency perspectives, achieving more than 3-fold energy savings compared to the considered baselines.

Title: SAVAE: Leveraging the variational Bayes autoencoder for survival analysis. (arXiv:2312.14651v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.14651
• Code URL: https://github.com/patricia-a-apellaniz/savae
• Copy Paste: [[2312.14651]] SAVAE: Leveraging the variational Bayes autoencoder for survival analysis(http://arxiv.org/abs/2312.14651)
• Summary:

  As in many fields of medical research, survival analysis has witnessed a growing interest in the application of deep learning techniques to model complex, high-dimensional, heterogeneous, incomplete, and censored medical data. Current methods often make assumptions about the relations between data that may not be valid in practice. In response, we introduce SAVAE (Survival Analysis Variational Autoencoder), a novel approach based on Variational Autoencoders. SAVAE contributes significantly to the field by introducing a tailored ELBO formulation for survival analysis, supporting various parametric distributions for covariates and survival time (as long as the log-likelihood is differentiable). It offers a general method that consistently performs well on various metrics, demonstrating robustness and stability through different experiments. Our proposal effectively estimates time-to-event, accounting for censoring, covariate interactions, and time-varying risk associations. We validate our model in diverse datasets, including genomic, clinical, and demographic data, with varying levels of censoring. This approach demonstrates competitive performance compared to state-of-the-art techniques, as assessed by the Concordance Index and the Integrated Brier Score. SAVAE also offers an interpretable model that parametrically models covariates and time. Moreover, its generative architecture facilitates further applications such as clustering, data imputation, and the generation of synthetic patient data through latent space inference from survival data.

Title: Engineered Ordinary Differential Equations as Classification Algorithm (EODECA): thorough characterization and testing. (arXiv:2312.14681v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.14681
• Code URL: null
• Copy Paste: [[2312.14681]] Engineered Ordinary Differential Equations as Classification Algorithm (EODECA): thorough characterization and testing(http://arxiv.org/abs/2312.14681)
• Summary:

  EODECA (Engineered Ordinary Differential Equations as Classification Algorithm) is a novel approach at the intersection of machine learning and dynamical systems theory, presenting a unique framework for classification tasks [1]. This method stands out with its dynamical system structure, utilizing ordinary differential equations (ODEs) to efficiently handle complex classification challenges. The paper delves into EODECA's dynamical properties, emphasizing its resilience against random perturbations and robust performance across various classification scenarios. Notably, EODECA's design incorporates the ability to embed stable attractors in the phase space, enhancing reliability and allowing for reversible dynamics. In this paper, we carry out a comprehensive analysis by expanding on the work [1], and employing a Euler discretization scheme. In particular, we evaluate EODECA's performance across five distinct classification problems, examining its adaptability and efficiency. Significantly, we demonstrate EODECA's effectiveness on the MNIST and Fashion MNIST datasets, achieving impressive accuracies of $98.06\%$ and $88.21\%$, respectively. These results are comparable to those of a multi-layer perceptron (MLP), underscoring EODECA's potential in complex data processing tasks. We further explore the model's learning journey, assessing its evolution in both pre and post training environments and highlighting its ability to navigate towards stable attractors. The study also investigates the invertibility of EODECA, shedding light on its decision-making processes and internal workings. This paper presents a significant step towards a more transparent and robust machine learning paradigm, bridging the gap between machine learning algorithms and dynamical systems methodologies.

Title: Integration Of Evolutionary Automated Machine Learning With Structural Sensitivity Analysis For Composite Pipelines. (arXiv:2312.14770v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.14770
• Code URL: null
• Copy Paste: [[2312.14770]] Integration Of Evolutionary Automated Machine Learning With Structural Sensitivity Analysis For Composite Pipelines(http://arxiv.org/abs/2312.14770)
• Summary:

  Automated machine learning (AutoML) systems propose an end-to-end solution to a given machine learning problem, creating either fixed or flexible pipelines. Fixed pipelines are task independent constructs: their general composition remains the same, regardless of the data. In contrast, the structure of flexible pipelines varies depending on the input, making them finely tailored to individual tasks. However, flexible pipelines can be structurally overcomplicated and have poor explainability. We propose the EVOSA approach that compensates for the negative points of flexible pipelines by incorporating a sensitivity analysis which increases the robustness and interpretability of the flexible solutions. EVOSA quantitatively estimates positive and negative impact of an edge or a node on a pipeline graph, and feeds this information to the evolutionary AutoML optimizer. The correctness and efficiency of EVOSA was validated in tabular, multimodal and computer vision tasks, suggesting generalizability of the proposed approach across domains.

Title: Spatiotemporal-Linear: Towards Universal Multivariate Time Series Forecasting. (arXiv:2312.14869v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.14869
• Code URL: null
• Copy Paste: [[2312.14869]] Spatiotemporal-Linear: Towards Universal Multivariate Time Series Forecasting(http://arxiv.org/abs/2312.14869)
• Summary:

  Within the field of complicated multivariate time series forecasting (TSF), popular techniques frequently rely on intricate deep learning architectures, ranging from transformer-based designs to recurrent neural networks. However, recent findings suggest that simple Linear models can surpass sophisticated constructs on diverse datasets. These models directly map observation to multiple future time steps, thereby minimizing error accumulation in iterative multi-step prediction. Yet, these models fail to incorporate spatial and temporal information within the data, which is critical for capturing patterns and dependencies that drive insightful predictions. This oversight often leads to performance bottlenecks, especially under specific sequence lengths and dataset conditions, preventing their universal application. In response, we introduce the SpatioTemporal-Linear (STL) framework. STL seamlessly integrates time-embedded and spatially-informed bypasses to augment the Linear-based architecture. These extra routes offer a more robust and refined regression to the data, particularly when the amount of observation is limited and the capacity of simple linear layers to capture dependencies declines. Empirical evidence highlights STL's prowess, outpacing both Linear and Transformer benchmarks across varied observation and prediction durations and datasets. Such robustness accentuates its suitability across a spectrum of applications, including but not limited to, traffic trajectory and rare disease progression forecasting. Through this discourse, we not only validate the STL's distinctive capacities to become a more general paradigm in multivariate time-series prediction using deep-learning techniques but also stress the need to tackle data-scarce prediction scenarios for universal application. Code will be made available.

biometric

Title: A Multi-Stage Adaptive Feature Fusion Neural Network for Multimodal Gait Recognition. (arXiv:2312.14410v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.14410
• Code URL: https://github.com/shinanzou/msaff
• Copy Paste: [[2312.14410]] A Multi-Stage Adaptive Feature Fusion Neural Network for Multimodal Gait Recognition(http://arxiv.org/abs/2312.14410)
• Summary:

  Gait recognition is a biometric technology that has received extensive attention. Most existing gait recognition algorithms are unimodal, and a few multimodal gait recognition algorithms perform multimodal fusion only once. None of these algorithms may fully exploit the complementary advantages of the multiple modalities. In this paper, by considering the temporal and spatial characteristics of gait data, we propose a multi-stage feature fusion strategy (MSFFS), which performs multimodal fusions at different stages in the feature extraction process. Also, we propose an adaptive feature fusion module (AFFM) that considers the semantic association between silhouettes and skeletons. The fusion process fuses different silhouette areas with their more related skeleton joints. Since visual appearance changes and time passage co-occur in a gait period, we propose a multiscale spatial-temporal feature extractor (MSSTFE) to learn the spatial-temporal linkage features thoroughly. Specifically, MSSTFE extracts and aggregates spatial-temporal linkages information at different spatial scales. Combining the strategy and modules mentioned above, we propose a multi-stage adaptive feature fusion (MSAFF) neural network, which shows state-of-the-art performance in many experiments on three datasets. Besides, MSAFF is equipped with feature dimensional pooling (FD Pooling), which can significantly reduce the dimension of the gait representations without hindering the accuracy. https://github.com/ShinanZou/MSAFF

steal

extraction

Title: FM-OV3D: Foundation Model-based Cross-modal Knowledge Blending for Open-Vocabulary 3D Detection. (arXiv:2312.14465v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.14465
• Code URL: null
• Copy Paste: [[2312.14465]] FM-OV3D: Foundation Model-based Cross-modal Knowledge Blending for Open-Vocabulary 3D Detection(http://arxiv.org/abs/2312.14465)
• Summary:

  The superior performances of pre-trained foundation models in various visual tasks underscore their potential to enhance the 2D models' open-vocabulary ability. Existing methods explore analogous applications in the 3D space. However, most of them only center around knowledge extraction from singular foundation models, which limits the open-vocabulary ability of 3D models. We hypothesize that leveraging complementary pre-trained knowledge from various foundation models can improve knowledge transfer from 2D pre-trained visual language models to the 3D space. In this work, we propose FM-OV3D, a method of Foundation Model-based Cross-modal Knowledge Blending for Open-Vocabulary 3D Detection, which improves the open-vocabulary localization and recognition abilities of 3D model by blending knowledge from multiple pre-trained foundation models, achieving true open-vocabulary without facing constraints from original 3D datasets. Specifically, to learn the open-vocabulary 3D localization ability, we adopt the open-vocabulary localization knowledge of the Grounded-Segment-Anything model. For open-vocabulary 3D recognition ability, We leverage the knowledge of generative foundation models, including GPT-3 and Stable Diffusion models, and cross-modal discriminative models like CLIP. The experimental results on two popular benchmarks for open-vocabulary 3D object detection show that our model efficiently learns knowledge from multiple foundation models to enhance the open-vocabulary ability of the 3D model and successfully achieves state-of-the-art performance in open-vocabulary 3D object detection tasks. Code is released at https://github.com/dmzhang0425/FM-OV3D.git.

Title: SEOpinion: Summarization and Exploration Opinion of E-Commerce Websites. (arXiv:2312.14171v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.14171
• Code URL: null
• Copy Paste: [[2312.14171]] SEOpinion: Summarization and Exploration Opinion of E-Commerce Websites(http://arxiv.org/abs/2312.14171)
• Summary:

  E-Commerce (EC) websites provide a large amount of useful information that exceed human cognitive processing ability. In order to help customers in comparing alternatives when buying a product, previous studies designed opinion summarization systems based on customer reviews. They ignored templates' information provided by manufacturers, although these descriptive information have much product aspects or characteristics. Therefore, this paper proposes a methodology coined as SEOpinion (Summa-rization and Exploration of Opinions) which provides a summary for the product aspects and spots opinion(s) regarding them, using a combination of templates' information with the customer reviews in two main phases. First, the Hierarchical Aspect Extraction (HAE) phase creates a hierarchy of product aspects from the template. Subsequently, the Hierarchical Aspect-based Opinion Summarization (HAOS) phase enriches this hierarchy with customers' opinions; to be shown to other potential buyers. To test the feasibility of using Deep Learning-based BERT techniques with our approach, we have created a corpus by gathering information from the top five EC websites for laptops. The experimental results show that Recurrent Neural Network (RNN) achieves better results (77.4% and 82.6% in terms of F1-measure for the first and second phase) than the Convolutional Neural Network (CNN) and the Support Vector Machine (SVM) technique.

Title: Graph Attention-Based Symmetry Constraint Extraction for Analog Circuits. (arXiv:2312.14405v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.14405
• Code URL: null
• Copy Paste: [[2312.14405]] Graph Attention-Based Symmetry Constraint Extraction for Analog Circuits(http://arxiv.org/abs/2312.14405)
• Summary:

  In recent years, analog circuits have received extensive attention and are widely used in many emerging applications. The high demand for analog circuits necessitates shorter circuit design cycles. To achieve the desired performance and specifications, various geometrical symmetry constraints must be carefully considered during the analog layout process. However, the manual labeling of these constraints by experienced analog engineers is a laborious and time-consuming process. To handle the costly runtime issue, we propose a graph-based learning framework to automatically extract symmetric constraints in analog circuit layout. The proposed framework leverages the connection characteristics of circuits and the devices'information to learn the general rules of symmetric constraints, which effectively facilitates the extraction of device-level constraints on circuit netlists. The experimental results demonstrate that compared to state-of-the-art symmetric constraint detection approaches, our framework achieves higher accuracy and lower false positive rate.

membership infer

federate

Title: Revisiting Few-Shot Object Detection with Vision-Language Models. (arXiv:2312.14494v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.14494
• Code URL: null
• Copy Paste: [[2312.14494]] Revisiting Few-Shot Object Detection with Vision-Language Models(http://arxiv.org/abs/2312.14494)
• Summary:

  Few-shot object detection (FSOD) benchmarks have advanced techniques for detecting new categories with limited annotations. Existing benchmarks repurpose well-established datasets like COCO by partitioning categories into base and novel classes for pre-training and fine-tuning respectively. However, these benchmarks do not reflect how FSOD is deployed in practice. Rather than only pre-training on a small number of base categories, we argue that it is more practical to fine-tune a foundation model (e.g., a vision-language model (VLM) pre-trained on web-scale data) for a target domain. Surprisingly, we find that zero-shot inference from VLMs like GroundingDINO significantly outperforms the state-of-the-art (48.3 vs. 33.1 AP) on COCO. However, such zero-shot models can still be misaligned to target concepts of interest. For example, trailers on the web may be different from trailers in the context of autonomous vehicles. In this work, we propose Foundational FSOD, a new benchmark protocol that evaluates detectors pre-trained on any external datasets and fine-tuned on K-shots per target class. Further, we note that current FSOD benchmarks are actually federated datasets containing exhaustive annotations for each category on a subset of the data. We leverage this insight to propose simple strategies for fine-tuning VLMs with federated losses. We demonstrate the effectiveness of our approach on LVIS and nuImages, improving over prior work by 5.9 AP.

Title: DCFL: Non-IID awareness Data Condensation aided Federated Learning. (arXiv:2312.14219v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.14219
• Code URL: null
• Copy Paste: [[2312.14219]] DCFL: Non-IID awareness Data Condensation aided Federated Learning(http://arxiv.org/abs/2312.14219)
• Summary:

  Federated learning is a decentralized learning paradigm wherein a central server trains a global model iteratively by utilizing clients who possess a certain amount of private datasets. The challenge lies in the fact that the client side private data may not be identically and independently distributed, significantly impacting the accuracy of the global model. Existing methods commonly address the Non-IID challenge by focusing on optimization, client selection and data complement. However, most approaches tend to overlook the perspective of the private data itself due to privacy constraints.Intuitively, statistical distinctions among private data on the client side can help mitigate the Non-IID degree. Besides, the recent advancements in dataset condensation technology have inspired us to investigate its potential applicability in addressing Non-IID issues while maintaining privacy. Motivated by this, we propose DCFL which divides clients into groups by using the Centered Kernel Alignment (CKA) method, then uses dataset condensation methods with non-IID awareness to complete clients. The private data from clients within the same group is complementary and their condensed data is accessible to all clients in the group. Additionally, CKA-guided client selection strategy, filtering mechanisms, and data enhancement techniques are incorporated to efficiently and precisely utilize the condensed data, enhance model performance, and minimize communication time. Experimental results demonstrate that DCFL achieves competitive performance on popular federated learning benchmarks including MNIST, FashionMNIST, SVHN, and CIFAR-10 with existing FL protocol.

Title: Federated Quantum Long Short-term Memory (FedQLSTM). (arXiv:2312.14309v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.14309
• Code URL: null
• Copy Paste: [[2312.14309]] Federated Quantum Long Short-term Memory (FedQLSTM)(http://arxiv.org/abs/2312.14309)
• Summary:

  Quantum federated learning (QFL) can facilitate collaborative learning across multiple clients using quantum machine learning (QML) models, while preserving data privacy. Although recent advances in QFL span different tasks like classification while leveraging several data types, no prior work has focused on developing a QFL framework that utilizes temporal data to approximate functions useful to analyze the performance of distributed quantum sensing networks. In this paper, a novel QFL framework that is the first to integrate quantum long short-term memory (QLSTM) models with temporal data is proposed. The proposed federated QLSTM (FedQLSTM) framework is exploited for performing the task of function approximation. In this regard, three key use cases are presented: Bessel function approximation, sinusoidal delayed quantum feedback control function approximation, and Struve function approximation. Simulation results confirm that, for all considered use cases, the proposed FedQLSTM framework achieves a faster convergence rate under one local training epoch, minimizing the overall computations, and saving 25-33% of the number of communication rounds needed until convergence compared to an FL framework with classical LSTM models.

Title: Federated Learning with Projected Trajectory Regularization. (arXiv:2312.14380v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.14380
• Code URL: null
• Copy Paste: [[2312.14380]] Federated Learning with Projected Trajectory Regularization(http://arxiv.org/abs/2312.14380)
• Summary:

  Federated learning enables joint training of machine learning models from distributed clients without sharing their local data. One key challenge in federated learning is to handle non-identically distributed data across the clients, which leads to deteriorated model training performances. Prior works in this line of research mainly focus on utilizing last-step global model parameters/gradients or the linear combinations of the past model parameters/gradients, which do not fully exploit the potential of global information from the model training trajectory. In this paper, we propose a novel federated learning framework with projected trajectory regularization (FedPTR) for tackling the data heterogeneity issue, which proposes a unique way to better extract the essential global information from the model training trajectory. Specifically, FedPTR allows local clients or the server to optimize an auxiliary (synthetic) dataset that mimics the learning dynamics of the recent model update and utilizes it to project the next-step model trajectory for local training regularization. We conduct rigorous theoretical analysis for our proposed framework under nonconvex stochastic settings to verify its fast convergence under heterogeneous data distributions. Experiments on various benchmark datasets and non-i.i.d. settings validate the effectiveness of our proposed framework.

Title: Federated Learning via Input-Output Collaborative Distillation. (arXiv:2312.14478v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.14478
• Code URL: https://github.com/lsl001006/fediod
• Copy Paste: [[2312.14478]] Federated Learning via Input-Output Collaborative Distillation(http://arxiv.org/abs/2312.14478)
• Summary:

  Federated learning (FL) is a machine learning paradigm in which distributed local nodes collaboratively train a central model without sharing individually held private data. Existing FL methods either iteratively share local model parameters or deploy co-distillation. However, the former is highly susceptible to private data leakage, and the latter design relies on the prerequisites of task-relevant real data. Instead, we propose a data-free FL framework based on local-to-central collaborative distillation with direct input and output space exploitation. Our design eliminates any requirement of recursive local parameter exchange or auxiliary task-relevant data to transfer knowledge, thereby giving direct privacy control to local users. In particular, to cope with the inherent data heterogeneity across locals, our technique learns to distill input on which each local model produces consensual yet unique results to represent each expertise. Our proposed FL framework achieves notable privacy-utility trade-offs with extensive experiments on image classification and segmentation tasks under various real-world heterogeneous federated learning settings on both natural and medical images.

Title: An effective and efficient green federated learning method for one-layer neural networks. (arXiv:2312.14528v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.14528
• Code URL: https://github.com/ofontenla/fedheonn
• Copy Paste: [[2312.14528]] An effective and efficient green federated learning method for one-layer neural networks(http://arxiv.org/abs/2312.14528)
• Summary:

  Nowadays, machine learning algorithms continue to grow in complexity and require a substantial amount of computational resources and energy. For these reasons, there is a growing awareness of the development of new green algorithms and distributed AI can contribute to this. Federated learning (FL) is one of the most active research lines in machine learning, as it allows the training of collaborative models in a distributed way, an interesting option in many real-world environments, such as the Internet of Things, allowing the use of these models in edge computing devices. In this work, we present a FL method, based on a neural network without hidden layers, capable of generating a global collaborative model in a single training round, unlike traditional FL methods that require multiple rounds for convergence. This allows obtaining an effective and efficient model that simplifies the management of the training process. Moreover, this method preserve data privacy by design, a crucial aspect in current data protection regulations. We conducted experiments with large datasets and a large number of federated clients. Despite being based on a network model without hidden layers, it maintains in all cases competitive accuracy results compared to more complex state-of-the-art machine learning models. Furthermore, we show that the method performs equally well in both identically and non-identically distributed scenarios. Finally, it is an environmentally friendly algorithm as it allows significant energy savings during the training process compared to its centralized counterpart.

Title: Accelerated Convergence of Stochastic Heavy Ball Method under Anisotropic Gradient Noise. (arXiv:2312.14567v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.14567
• Code URL: null
• Copy Paste: [[2312.14567]] Accelerated Convergence of Stochastic Heavy Ball Method under Anisotropic Gradient Noise(http://arxiv.org/abs/2312.14567)
• Summary:

  Heavy-ball momentum with decaying learning rates is widely used with SGD for optimizing deep learning models. In contrast to its empirical popularity, the understanding of its theoretical property is still quite limited, especially under the standard anisotropic gradient noise condition for quadratic regression problems. Although it is widely conjectured that heavy-ball momentum method can provide accelerated convergence and should work well in large batch settings, there is no rigorous theoretical analysis. In this paper, we fill this theoretical gap by establishing a non-asymptotic convergence bound for stochastic heavy-ball methods with step decay scheduler on quadratic objectives, under the anisotropic gradient noise condition. As a direct implication, we show that heavy-ball momentum can provide $\tilde{\mathcal{O}}(\sqrt{\kappa})$ accelerated convergence of the bias term of SGD while still achieving near-optimal convergence rate with respect to the stochastic variance term. The combined effect implies an overall convergence rate within log factors from the statistical minimax rate. This means SGD with heavy-ball momentum is useful in the large-batch settings such as distributed machine learning or federated learning, where a smaller number of iterations can significantly reduce the number of communication rounds, leading to acceleration in practice.

Title: Towards more sustainable enterprise data and application management with cross silo Federated Learning and Analytics. (arXiv:2312.14628v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.14628
• Code URL: https://github.com/azure-samples/azure-ml-federated-learning
• Copy Paste: [[2312.14628]] Towards more sustainable enterprise data and application management with cross silo Federated Learning and Analytics(http://arxiv.org/abs/2312.14628)
• Summary:

  To comply with new legal requirements and policies committed to privacy protection, more and more companies start to deploy cross-silo Federated Learning at global scale, where several clients/silos collaboratively train a global model under the coordination of a central server. Instead of data sharing and transmission, clients train models using their private local data and exchange model updates. However, there is little understanding of the carbon emission impact of cross silo Federated Learning due to the lack of related works. In this study, we first analyze the sustainability aspect of cross-silo Federated Learning, across the AI product life cycle instead of focusing only on the model training, with the comparison to the centralized method. A more holistic quantitative cost and CO2 emission estimation method for real world cross-silo Federated Learning setting is proposed. Secondly, we propose a novel data and application management system using cross silo Federated Learning and analytics to make IT companies more sustainable and cost effective.

fair

Title: Inclusive normalization of face images to passport format. (arXiv:2312.14544v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.14544
• Code URL: null
• Copy Paste: [[2312.14544]] Inclusive normalization of face images to passport format(http://arxiv.org/abs/2312.14544)
• Summary:

  Face recognition has been used more and more in real world applications in recent years. However, when the skin color bias is coupled with intra-personal variations like harsh illumination, the face recognition task is more likely to fail, even during human inspection. Face normalization methods try to deal with such challenges by removing intra-personal variations from an input image while keeping the identity the same. However, most face normalization methods can only remove one or two variations and ignore dataset biases such as skin color bias. The outputs of many face normalization methods are also not realistic to human observers. In this work, a style based face normalization model (StyleFNM) is proposed to remove most intra-personal variations including large changes in pose, bad or harsh illumination, low resolution, blur, facial expressions, and accessories like sunglasses among others. The dataset bias is also dealt with in this paper by controlling a pretrained GAN to generate a balanced dataset of passport-like images. The experimental results show that StyleFNM can generate more realistic outputs and can improve significantly the accuracy and fairness of face recognition systems.

Title: BSS-Bench: Towards Reproducible and Effective Band Selection Search. (arXiv:2312.14570v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.14570
• Code URL: null
• Copy Paste: [[2312.14570]] BSS-Bench: Towards Reproducible and Effective Band Selection Search(http://arxiv.org/abs/2312.14570)
• Summary:

  The key technology to overcome the drawbacks of hyperspectral imaging (expensive, high capture delay, and low spatial resolution) and make it widely applicable is to select only a few representative bands from hundreds of bands. However, current band selection (BS) methods face challenges in fair comparisons due to inconsistent train/validation settings, including the number of bands, dataset splits, and retraining settings. To make BS methods easy and reproducible, this paper presents the first band selection search benchmark (BSS-Bench) containing 52k training and evaluation records of numerous band combinations (BC) with different backbones for various hyperspectral analysis tasks. The creation of BSS-Bench required a significant computational effort of 1.26k GPU days. By querying BSS-Bench, BS experiments can be performed easily and reproducibly, and the gap between the searched result and the best achievable performance can be measured. Based on BSS-Bench, we further discuss the impact of various factors on BS, such as the number of bands, unsupervised statistics, and different backbones. In addition to BSS-Bench, we present an effective one-shot BS method called Single Combination One Shot (SCOS), which learns the priority of any BCs through one-time training, eliminating the need for repetitive retraining on different BCs. Furthermore, the search process of SCOS is flexible and does not require training, making it efficient and effective. Our extensive evaluations demonstrate that SCOS outperforms current BS methods on multiple tasks, even with much fewer bands. Our BSS-Bench and codes are available in the supplementary material and will be publicly available.

Title: Fairness in Submodular Maximization over a Matroid Constraint. (arXiv:2312.14299v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.14299
• Code URL: null
• Copy Paste: [[2312.14299]] Fairness in Submodular Maximization over a Matroid Constraint(http://arxiv.org/abs/2312.14299)
• Summary:

  Submodular maximization over a matroid constraint is a fundamental problem with various applications in machine learning. Some of these applications involve decision-making over datapoints with sensitive attributes such as gender or race. In such settings, it is crucial to guarantee that the selected solution is fairly distributed with respect to this attribute. Recently, fairness has been investigated in submodular maximization under a cardinality constraint in both the streaming and offline settings, however the more general problem with matroid constraint has only been considered in the streaming setting and only for monotone objectives. This work fills this gap. We propose various algorithms and impossibility results offering different trade-offs between quality, fairness, and generality.

interpretability

Title: Images in Discrete Choice Modeling: Addressing Data Isomorphism in Multi-Modality Inputs. (arXiv:2312.14724v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.14724
• Code URL: https://github.com/bsifringer/imagedcm
• Copy Paste: [[2312.14724]] Images in Discrete Choice Modeling: Addressing Data Isomorphism in Multi-Modality Inputs(http://arxiv.org/abs/2312.14724)
• Summary:

  This paper explores the intersection of Discrete Choice Modeling (DCM) and machine learning, focusing on the integration of image data into DCM's utility functions and its impact on model interpretability. We investigate the consequences of embedding high-dimensional image data that shares isomorphic information with traditional tabular inputs within a DCM framework. Our study reveals that neural network (NN) components learn and replicate tabular variable representations from images when co-occurrences exist, thereby compromising the interpretability of DCM parameters. We propose and benchmark two methodologies to address this challenge: architectural design adjustments to segregate redundant information, and isomorphic information mitigation through source information masking and inpainting. Our experiments, conducted on a semi-synthetic dataset, demonstrate that while architectural modifications prove inconclusive, direct mitigation at the data source shows to be a more effective strategy in maintaining the integrity of DCM's interpretable parameters. The paper concludes with insights into the applicability of our findings in real-world settings and discusses the implications for future research in hybrid modeling that combines complex data modalities. Full control of tabular and image data congruence is attained by using the MIT moral machine dataset, and both inputs are merged into a choice model by deploying the Learning Multinomial Logit (L-MNL) framework.

Title: Don't Believe Everything You Read: Enhancing Summarization Interpretability through Automatic Identification of Hallucinations in Large Language Models. (arXiv:2312.14346v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.14346
• Code URL: null
• Copy Paste: [[2312.14346]] Don't Believe Everything You Read: Enhancing Summarization Interpretability through Automatic Identification of Hallucinations in Large Language Models(http://arxiv.org/abs/2312.14346)
• Summary:

  Large Language Models (LLMs) are adept at text manipulation -- tasks such as machine translation and text summarization. However, these models can also be prone to hallucination, which can be detrimental to the faithfulness of any answers that the model provides. Recent works in combating hallucinations in LLMs deal with identifying hallucinated sentences and categorizing the different ways in which models hallucinate. This paper takes a deep dive into LLM behavior with respect to hallucinations, defines a token-level approach to identifying different kinds of hallucinations, and further utilizes this token-level tagging to improve the interpretability and faithfulness of LLMs in dialogue summarization tasks. Through this, the paper presents a new, enhanced dataset and a new training paradigm.

Title: Contextual Feature Selection with Conditional Stochastic Gates. (arXiv:2312.14254v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.14254
• Code URL: null
• Copy Paste: [[2312.14254]] Contextual Feature Selection with Conditional Stochastic Gates(http://arxiv.org/abs/2312.14254)
• Summary:

  We study the problem of contextual feature selection, where the goal is to learn a predictive function while identifying subsets of informative features conditioned on specific contexts. Towards this goal, we generalize the recently proposed stochastic gates (STG) Yamada et al. [2020] by modeling the probabilistic gates as conditional Bernoulli variables whose parameters are predicted based on the contextual variables. Our new scheme, termed conditional-STG (c-STG), comprises two networks: a hypernetwork that establishes the mapping between contextual variables and probabilistic feature selection parameters and a prediction network that maps the selected feature to the response variable. Training the two networks simultaneously ensures the comprehensive incorporation of context and feature selection within a unified model. We provide a theoretical analysis to examine several properties of the proposed framework. Importantly, our model leads to improved flexibility and adaptability of feature selection and, therefore, can better capture the nuances and variations in the data. We apply c-STG to simulated and real-world datasets, including healthcare, housing, and neuroscience, and demonstrate that it effectively selects contextually meaningful features, thereby enhancing predictive performance and interpretability.

explainability

Title: Towards Better Visualizing the Decision Basis of Networks via Unfold and Conquer Attribution Guidance. (arXiv:2312.14201v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.14201
• Code URL: null
• Copy Paste: [[2312.14201]] Towards Better Visualizing the Decision Basis of Networks via Unfold and Conquer Attribution Guidance(http://arxiv.org/abs/2312.14201)
• Summary:

  Revealing the transparency of Deep Neural Networks (DNNs) has been widely studied to describe the decision mechanisms of network inner structures. In this paper, we propose a novel post-hoc framework, Unfold and Conquer Attribution Guidance (UCAG), which enhances the explainability of the network decision by spatially scrutinizing the input features with respect to the model confidence. Addressing the phenomenon of missing detailed descriptions, UCAG sequentially complies with the confidence of slices of the image, leading to providing an abundant and clear interpretation. Therefore, it is possible to enhance the representation ability of explanation by preserving the detailed descriptions of assistant input features, which are commonly overwhelmed by the main meaningful regions. We conduct numerous evaluations to validate the performance in several metrics: i) deletion and insertion, ii) (energy-based) pointing games, and iii) positive and negative density maps. Experimental results, including qualitative comparisons, demonstrate that our method outperforms the existing methods with the nature of clear and detailed explanations and applicability.

Title: VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluation. (arXiv:2312.14867v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.14867
• Code URL: null
• Copy Paste: [[2312.14867]] VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluation(http://arxiv.org/abs/2312.14867)
• Summary:

  In the rapidly advancing field of conditional image generation research, challenges such as limited explainability lie in effectively evaluating the performance and capabilities of various models. This paper introduces VIESCORE, a Visual Instruction-guided Explainable metric for evaluating any conditional image generation tasks. VIESCORE leverages general knowledge from Multimodal Large Language Models (MLLMs) as the backbone and does not require training or fine-tuning. We evaluate VIESCORE on seven prominent tasks in conditional image tasks and found: (1) VIESCORE (GPT4-v) achieves a high Spearman correlation of 0.3 with human evaluations, while the human-to-human correlation is 0.45. (2) VIESCORE (with open-source MLLM) is significantly weaker than GPT-4v in evaluating synthetic images. (3) VIESCORE achieves a correlation on par with human ratings in the generation tasks but struggles in editing tasks. With these results, we believe VIESCORE shows its great potential to replace human judges in evaluating image synthesis tasks.

watermark

diffusion

Title: DreamDistribution: Prompt Distribution Learning for Text-to-Image Diffusion Models. (arXiv:2312.14216v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.14216
• Code URL: null
• Copy Paste: [[2312.14216]] DreamDistribution: Prompt Distribution Learning for Text-to-Image Diffusion Models(http://arxiv.org/abs/2312.14216)
• Summary:

  The popularization of Text-to-Image (T2I) diffusion models enables the generation of high-quality images from text descriptions. However, generating diverse customized images with reference visual attributes remains challenging. This work focuses on personalizing T2I diffusion models at a more abstract concept or category level, adapting commonalities from a set of reference images while creating new instances with sufficient variations. We introduce a solution that allows a pretrained T2I diffusion model to learn a set of soft prompts, enabling the generation of novel images by sampling prompts from the learned distribution. These prompts offer text-guided editing capabilities and additional flexibility in controlling variation and mixing between multiple distributions. We also show the adaptability of the learned prompt distribution to other tasks, such as text-to-3D. Finally we demonstrate effectiveness of our approach through quantitative analysis including automatic evaluation and human assessment. Project website: https://briannlongzhao.github.io/DreamDistribution

Title: Fast Diffusion-Based Counterfactuals for Shortcut Removal and Generation. (arXiv:2312.14223v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.14223
• Code URL: null
• Copy Paste: [[2312.14223]] Fast Diffusion-Based Counterfactuals for Shortcut Removal and Generation(http://arxiv.org/abs/2312.14223)
• Summary:

  Shortcut learning is when a model -- e.g. a cardiac disease classifier -- exploits correlations between the target label and a spurious shortcut feature, e.g. a pacemaker, to predict the target label based on the shortcut rather than real discriminative features. This is common in medical imaging, where treatment and clinical annotations correlate with disease labels, making them easy shortcuts to predict disease. We propose a novel detection and quantification of the impact of potential shortcut features via a fast diffusion-based counterfactual image generation that can synthetically remove or add shortcuts. Via a novel inpainting-based modification we spatially limit the changes made with no extra inference step, encouraging the removal of spatially constrained shortcut features while ensuring that the shortcut-free counterfactuals preserve their remaining image features to a high degree. Using these, we assess how shortcut features influence model predictions.

  This is enabled by our second contribution: An efficient diffusion-based counterfactual explanation method with significant inference speed-up at comparable image quality as state-of-the-art. We confirm this on two large chest X-ray datasets, a skin lesion dataset, and CelebA.

Title: Tuning-Free Inversion-Enhanced Control for Consistent Image Editing. (arXiv:2312.14611v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.14611
• Code URL: null
• Copy Paste: [[2312.14611]] Tuning-Free Inversion-Enhanced Control for Consistent Image Editing(http://arxiv.org/abs/2312.14611)
• Summary:

  Consistent editing of real images is a challenging task, as it requires performing non-rigid edits (e.g., changing postures) to the main objects in the input image without changing their identity or attributes. To guarantee consistent attributes, some existing methods fine-tune the entire model or the textual embedding for structural consistency, but they are time-consuming and fail to perform non-rigid edits. Other works are tuning-free, but their performances are weakened by the quality of Denoising Diffusion Implicit Model (DDIM) reconstruction, which often fails in real-world scenarios. In this paper, we present a novel approach called Tuning-free Inversion-enhanced Control (TIC), which directly correlates features from the inversion process with those from the sampling process to mitigate the inconsistency in DDIM reconstruction. Specifically, our method effectively obtains inversion features from the key and value features in the self-attention layers, and enhances the sampling process by these inversion features, thus achieving accurate reconstruction and content-consistent editing. To extend the applicability of our method to general editing scenarios, we also propose a mask-guided attention concatenation strategy that combines contents from both the inversion and the naive DDIM editing processes. Experiments show that the proposed method outperforms previous works in reconstruction and consistent editing, and produces impressive results in various settings.

Title: Harnessing Diffusion Models for Visual Perception with Meta Prompts. (arXiv:2312.14733v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.14733
• Code URL: https://github.com/fudan-zvg/meta-prompts
• Copy Paste: [[2312.14733]] Harnessing Diffusion Models for Visual Perception with Meta Prompts(http://arxiv.org/abs/2312.14733)
• Summary:

  The issue of generative pretraining for vision models has persisted as a long-standing conundrum. At present, the text-to-image (T2I) diffusion model demonstrates remarkable proficiency in generating high-definition images matching textual inputs, a feat made possible through its pre-training on large-scale image-text pairs. This leads to a natural inquiry: can diffusion models be utilized to tackle visual perception tasks? In this paper, we propose a simple yet effective scheme to harness a diffusion model for visual perception tasks. Our key insight is to introduce learnable embeddings (meta prompts) to the pre-trained diffusion models to extract proper features for perception. The effect of meta prompts are two-fold. First, as a direct replacement of the text embeddings in the T2I models, it can activate task-relevant features during feature extraction. Second, it will be used to re-arrange the extracted features to ensures that the model focuses on the most pertinent features for the task on hand. Additionally, we design a recurrent refinement training strategy that fully leverages the property of diffusion models, thereby yielding stronger visual features. Extensive experiments across various benchmarks validate the effectiveness of our approach. Our approach achieves new performance records in depth estimation tasks on NYU depth V2 and KITTI, and in semantic segmentation task on CityScapes. Concurrently, the proposed method attains results comparable to the current state-of-the-art in semantic segmentation on ADE20K and pose estimation on COCO datasets, further exemplifying its robustness and versatility.

Title: Plan, Posture and Go: Towards Open-World Text-to-Motion Generation. (arXiv:2312.14828v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.14828
• Code URL: null
• Copy Paste: [[2312.14828]] Plan, Posture and Go: Towards Open-World Text-to-Motion Generation(http://arxiv.org/abs/2312.14828)
• Summary:

  Conventional text-to-motion generation methods are usually trained on limited text-motion pairs, making them hard to generalize to open-world scenarios. Some works use the CLIP model to align the motion space and the text space, aiming to enable motion generation from natural language motion descriptions. However, they are still constrained to generate limited and unrealistic in-place motions. To address these issues, we present a divide-and-conquer framework named PRO-Motion, which consists of three modules as motion planner, posture-diffuser and go-diffuser. The motion planner instructs Large Language Models (LLMs) to generate a sequence of scripts describing the key postures in the target motion. Differing from natural languages, the scripts can describe all possible postures following very simple text templates. This significantly reduces the complexity of posture-diffuser, which transforms a script to a posture, paving the way for open-world generation. Finally, go-diffuser, implemented as another diffusion model, estimates whole-body translations and rotations for all postures, resulting in realistic motions. Experimental results have shown the superiority of our method with other counterparts, and demonstrated its capability of generating diverse and realistic motions from complex open-world prompts such as "Experiencing a profound sense of joy". The project page is available at https://moonsliu.github.io/Pro-Motion.

Title: BrainVis: Exploring the Bridge between Brain and Visual Signals via Image Reconstruction. (arXiv:2312.14871v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.14871
• Code URL: null
• Copy Paste: [[2312.14871]] BrainVis: Exploring the Bridge between Brain and Visual Signals via Image Reconstruction(http://arxiv.org/abs/2312.14871)
• Summary:

  Analyzing and reconstructing visual stimuli from brain signals effectively advances understanding of the human visual system. However, the EEG signals are complex and contain a amount of noise. This leads to substantial limitations in existing works of visual stimuli reconstruction from EEG, such as difficulties in aligning EEG embeddings with the fine-grained semantic information and a heavy reliance on additional large self-collected dataset for training. To address these challenges, we propose a novel approach called BrainVis. Firstly, we divide the EEG signals into various units and apply a self-supervised approach on them to obtain EEG time-domain features, in an attempt to ease the training difficulty. Additionally, we also propose to utilize the frequency-domain features to enhance the EEG representations. Then, we simultaneously align EEG time-frequency embeddings with the interpolation of the coarse and fine-grained semantics in the CLIP space, to highlight the primary visual components and reduce the cross-modal alignment difficulty. Finally, we adopt the cascaded diffusion models to reconstruct images. Our proposed BrainVis outperforms state of the arts in both semantic fidelity reconstruction and generation quality. Notably, we reduce the training data scale to 10% of the previous work.

Title: MACS: Mass Conditioned 3D Hand and Object Motion Synthesis. (arXiv:2312.14929v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.14929
• Code URL: null
• Copy Paste: [[2312.14929]] MACS: Mass Conditioned 3D Hand and Object Motion Synthesis(http://arxiv.org/abs/2312.14929)
• Summary:

  The physical properties of an object, such as mass, significantly affect how we manipulate it with our hands. Surprisingly, this aspect has so far been neglected in prior work on 3D motion synthesis. To improve the naturalness of the synthesized 3D hand object motions, this work proposes MACS the first MAss Conditioned 3D hand and object motion Synthesis approach. Our approach is based on cascaded diffusion models and generates interactions that plausibly adjust based on the object mass and interaction type. MACS also accepts a manually drawn 3D object trajectory as input and synthesizes the natural 3D hand motions conditioned by the object mass. This flexibility enables MACS to be used for various downstream applications, such as generating synthetic training data for ML tasks, fast animation of hands for graphics workflows, and generating character interactions for computer games. We show experimentally that a small-scale dataset is sufficient for MACS to reasonably generalize across interpolated and extrapolated object masses unseen during the training. Furthermore, MACS shows moderate generalization to unseen objects, thanks to the mass-conditioned contact labels generated by our surface contact synthesis model ConNet. Our comprehensive user study confirms that the synthesized 3D hand-object interactions are highly plausible and realistic.

Title: Non-Denoising Forward-Time Diffusions. (arXiv:2312.14589v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.14589
• Code URL: null
• Copy Paste: [[2312.14589]] Non-Denoising Forward-Time Diffusions(http://arxiv.org/abs/2312.14589)
• Summary:

  The scope of this paper is generative modeling through diffusion processes. An approach falling within this paradigm is the work of Song et al. (2021), which relies on a time-reversal argument to construct a diffusion process targeting the desired data distribution. We show that the time-reversal argument, common to all denoising diffusion probabilistic modeling proposals, is not necessary. We obtain diffusion processes targeting the desired data distribution by taking appropriate mixtures of diffusion bridges. The resulting transport is exact by construction, allows for greater flexibility in choosing the dynamics of the underlying diffusion, and can be approximated by means of a neural network via novel training objectives. We develop a unifying view of the drift adjustments corresponding to our and to time-reversal approaches and make use of this representation to inspect the inner workings of diffusion-based generative models. Finally, we leverage on scalable simulation and inference techniques common in spatial statistics to move beyond fully factorial distributions in the underlying diffusion dynamics. The methodological advances contained in this work contribute toward establishing a general framework for generative modeling based on diffusion processes.

Title: Diffusion Maps for Signal Filtering in Graph Learning. (arXiv:2312.14758v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.14758
• Code URL: null
• Copy Paste: [[2312.14758]] Diffusion Maps for Signal Filtering in Graph Learning(http://arxiv.org/abs/2312.14758)
• Summary:

  This paper explores the application diffusion maps as graph shift operators in understanding the underlying geometry of graph signals. The study evaluates the improvements in graph learning when using diffusion map generated filters to the Markov Variation minimization problem. The paper showcases the effectiveness of this approach through examples involving synthetically generated and real-world temperature sensor data. These examples also compare the diffusion map graph signal model with other commonly used graph signal operators. The results provide new approaches for the analysis and understanding of complex, non-Euclidean data structures.

noise learning

data-free

transformer

Title: Unveiling Backbone Effects in CLIP: Exploring Representational Synergies and Variances. (arXiv:2312.14400v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.14400
• Code URL: null
• Copy Paste: [[2312.14400]] Unveiling Backbone Effects in CLIP: Exploring Representational Synergies and Variances(http://arxiv.org/abs/2312.14400)
• Summary:

  Contrastive Language-Image Pretraining (CLIP) stands out as a prominent method for image representation learning. Various neural architectures, spanning Transformer-based models like Vision Transformers (ViTs) to Convolutional Networks (ConvNets) like ResNets, are trained with CLIP and serve as universal backbones across diverse vision tasks. Despite utilizing the same data and training objectives, the effectiveness of representations learned by these architectures raises a critical question. Our investigation explores the differences in CLIP performance among these backbone architectures, revealing significant disparities in their classifications. Notably, normalizing these representations results in substantial performance variations. Our findings showcase a remarkable possible synergy between backbone predictions that could reach an improvement of over 20% through informed selection of the appropriate backbone. Moreover, we propose a simple, yet effective approach to combine predictions from multiple backbones, leading to a notable performance boost of up to 6.34\%. We will release the code for reproducing the results.

Title: Context Enhanced Transformer for Single Image Object Detection. (arXiv:2312.14492v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.14492
• Code URL: null
• Copy Paste: [[2312.14492]] Context Enhanced Transformer for Single Image Object Detection(http://arxiv.org/abs/2312.14492)
• Summary:

  With the increasing importance of video data in real-world applications, there is a rising need for efficient object detection methods that utilize temporal information. While existing video object detection (VOD) techniques employ various strategies to address this challenge, they typically depend on locally adjacent frames or randomly sampled images within a clip. Although recent Transformer-based VOD methods have shown promising results, their reliance on multiple inputs and additional network complexity to incorporate temporal information limits their practical applicability. In this paper, we propose a novel approach to single image object detection, called Context Enhanced TRansformer (CETR), by incorporating temporal context into DETR using a newly designed memory module. To efficiently store temporal information, we construct a class-wise memory that collects contextual information across data. Additionally, we present a classification-based sampling technique to selectively utilize the relevant memory for the current image. In the testing, We introduce a test-time memory adaptation method that updates individual memory functions by considering the test distribution. Experiments with CityCam and ImageNet VID datasets exhibit the efficiency of the framework on various video systems. The project page and code will be made available at: https://ku-cvlab.github.io/CETR.

Title: ViStripformer: A Token-Efficient Transformer for Versatile Video Restoration. (arXiv:2312.14502v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.14502
• Code URL: https://github.com/pp00704831/video-stripformer
• Copy Paste: [[2312.14502]] ViStripformer: A Token-Efficient Transformer for Versatile Video Restoration(http://arxiv.org/abs/2312.14502)
• Summary:

  Video restoration is a low-level vision task that seeks to restore clean, sharp videos from quality-degraded frames. One would use the temporal information from adjacent frames to make video restoration successful. Recently, the success of the Transformer has raised awareness in the computer-vision community. However, its self-attention mechanism requires much memory, which is unsuitable for high-resolution vision tasks like video restoration. In this paper, we propose ViStripformer (Video Stripformer), which utilizes spatio-temporal strip attention to catch long-range data correlations, consisting of intra-frame strip attention (Intra-SA) and inter-frame strip attention (Inter-SA) for extracting spatial and temporal information. It decomposes video frames into strip-shaped features in horizontal and vertical directions for Intra-SA and Inter-SA to address degradation patterns with various orientations and magnitudes. Besides, ViStripformer is an effective and efficient transformer architecture with much lower memory usage than the vanilla transformer. Extensive experiments show that the proposed model achieves superior results with fast inference time on video restoration tasks, including video deblurring, demoireing, and deraining.

Title: PoseViNet: Distracted Driver Action Recognition Framework Using Multi-View Pose Estimation and Vision Transformer. (arXiv:2312.14577v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.14577
• Code URL: null
• Copy Paste: [[2312.14577]] PoseViNet: Distracted Driver Action Recognition Framework Using Multi-View Pose Estimation and Vision Transformer(http://arxiv.org/abs/2312.14577)
• Summary:

  Driver distraction is a principal cause of traffic accidents. In a study conducted by the National Highway Traffic Safety Administration, engaging in activities such as interacting with in-car menus, consuming food or beverages, or engaging in telephonic conversations while operating a vehicle can be significant sources of driver distraction. From this viewpoint, this paper introduces a novel method for detection of driver distraction using multi-view driver action images. The proposed method is a vision transformer-based framework with pose estimation and action inference, namely PoseViNet. The motivation for adding posture information is to enable the transformer to focus more on key features. As a result, the framework is more adept at identifying critical actions. The proposed framework is compared with various state-of-the-art models using SFD3 dataset representing 10 behaviors of drivers. It is found from the comparison that the PoseViNet outperforms these models. The proposed framework is also evaluated with the SynDD1 dataset representing 16 behaviors of driver. As a result, the PoseViNet achieves 97.55% validation accuracy and 90.92% testing accuracy with the challenging dataset.

Title: Explainable Multi-Camera 3D Object Detection with Transformer-Based Saliency Maps. (arXiv:2312.14606v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.14606
• Code URL: null
• Copy Paste: [[2312.14606]] Explainable Multi-Camera 3D Object Detection with Transformer-Based Saliency Maps(http://arxiv.org/abs/2312.14606)
• Summary:

  Vision Transformers (ViTs) have achieved state-of-the-art results on various computer vision tasks, including 3D object detection. However, their end-to-end implementation also makes ViTs less explainable, which can be a challenge for deploying them in safety-critical applications, such as autonomous driving, where it is important for authorities, developers, and users to understand the model's reasoning behind its predictions. In this paper, we propose a novel method for generating saliency maps for a DetR-like ViT with multiple camera inputs used for 3D object detection. Our method is based on the raw attention and is more efficient than gradient-based methods. We evaluate the proposed method on the nuScenes dataset using extensive perturbation tests and show that it outperforms other explainability methods in terms of visual quality and quantitative metrics. We also demonstrate the importance of aggregating attention across different layers of the transformer. Our work contributes to the development of explainable AI for ViTs, which can help increase trust in AI applications by establishing more transparency regarding the inner workings of AI models.

Title: Lift-Attend-Splat: Bird's-eye-view camera-lidar fusion using transformers. (arXiv:2312.14919v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.14919
• Code URL: null
• Copy Paste: [[2312.14919]] Lift-Attend-Splat: Bird's-eye-view camera-lidar fusion using transformers(http://arxiv.org/abs/2312.14919)
• Summary:

  Combining complementary sensor modalities is crucial to providing robust perception for safety-critical robotics applications such as autonomous driving (AD). Recent state-of-the-art camera-lidar fusion methods for AD rely on monocular depth estimation which is a notoriously difficult task compared to using depth information from the lidar directly. Here, we find that this approach does not leverage depth as expected and show that naively improving depth estimation does not lead to improvements in object detection performance and that, strikingly, removing depth estimation altogether does not degrade object detection performance. This suggests that relying on monocular depth could be an unnecessary architectural bottleneck during camera-lidar fusion. In this work, we introduce a novel fusion method that bypasses monocular depth estimation altogether and instead selects and fuses camera and lidar features in a bird's-eye-view grid using a simple attention mechanism. We show that our model can modulate its use of camera features based on the availability of lidar features and that it yields better 3D object detection on the nuScenes dataset than baselines relying on monocular depth estimation.

Title: Theory of Hallucinations based on Equivariance. (arXiv:2312.14504v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.14504
• Code URL: null
• Copy Paste: [[2312.14504]] Theory of Hallucinations based on Equivariance(http://arxiv.org/abs/2312.14504)
• Summary:

  Equivariance is an important feature in machine learning, including language models. It ensures that any sequences of phrases with the same meanings are interpreted consistently. For example, the sentence 'There is a cat on the table' should be interpreted by language models as it is, regardless of variations in its token-level expression. Building on this insight, I propose a new theory suggesting that insufficient equivariance in language models can lead to hallucinations. According to this theory, which is both intuitive and novel, language models trained on relatively small datasets tend to misinterpret input texts and/or generate incorrect texts (i.e., hallucinations). To test this theory, I developed a toy model known as 'dancing men', which is a character-level substitution cipher. Additionally, I propose a novel technique based on the T5 (Text To Text Transfer Transformer) model to efficiently decipher these codes without relying on frequency analysis. I have found that this T5 model can almost completely solve the cipher, demonstrating its ability to acquire equivariance in this frame. This method could be scaled up to word-level and sentence-level substitution ciphers, analogous to large language models without tokenizers or dictionaries. This scalability makes it suitable for investigating the proposed link between inadequate equivariance acquisition and the emergence of hallucinations.

Title: Generative Pretraining at Scale: Transformer-Based Encoding of Transactional Behavior for Fraud Detection. (arXiv:2312.14406v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.14406
• Code URL: null
• Copy Paste: [[2312.14406]] Generative Pretraining at Scale: Transformer-Based Encoding of Transactional Behavior for Fraud Detection(http://arxiv.org/abs/2312.14406)
• Summary:

  In this work, we introduce an innovative autoregressive model leveraging Generative Pretrained Transformer (GPT) architectures, tailored for fraud detection in payment systems. Our approach innovatively confronts token explosion and reconstructs behavioral sequences, providing a nuanced understanding of transactional behavior through temporal and contextual analysis. Utilizing unsupervised pretraining, our model excels in feature representation without the need for labeled data. Additionally, we integrate a differential convolutional approach to enhance anomaly detection, bolstering the security and efficacy of one of the largest online payment merchants in China. The scalability and adaptability of our model promise broad applicability in various transactional contexts.

Title: Fast-NTK: Parameter-Efficient Unlearning for Large-Scale Models. (arXiv:2312.14923v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.14923
• Code URL: null
• Copy Paste: [[2312.14923]] Fast-NTK: Parameter-Efficient Unlearning for Large-Scale Models(http://arxiv.org/abs/2312.14923)
• Summary:

  The rapid growth of machine learning has spurred legislative initiatives such as ``the Right to be Forgotten,'' allowing users to request data removal. In response, ``machine unlearning'' proposes the selective removal of unwanted data without the need for retraining from scratch. While the Neural-Tangent-Kernel-based (NTK-based) unlearning method excels in performance, it suffers from significant computational complexity, especially for large-scale models and datasets. Our work introduces ``Fast-NTK,'' a novel NTK-based unlearning algorithm that significantly reduces the computational complexity by incorporating parameter-efficient fine-tuning methods, such as fine-tuning batch normalization layers in a CNN or visual prompts in a vision transformer. Our experimental results demonstrate scalability to much larger neural networks and datasets (e.g., 88M parameters; 5k images), surpassing the limitations of previous full-model NTK-based approaches designed for smaller cases (e.g., 8M parameters; 500 images). Notably, our approach maintains a performance comparable to the traditional method of retraining on the retain set alone. Fast-NTK can thus enable for practical and scalable NTK-based unlearning in deep neural networks.

generative

Title: ZeroShape: Regression-based Zero-shot Shape Reconstruction. (arXiv:2312.14198v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.14198
• Code URL: null
• Copy Paste: [[2312.14198]] ZeroShape: Regression-based Zero-shot Shape Reconstruction(http://arxiv.org/abs/2312.14198)
• Summary:

  We study the problem of single-image zero-shot 3D shape reconstruction. Recent works learn zero-shot shape reconstruction through generative modeling of 3D assets, but these models are computationally expensive at train and inference time. In contrast, the traditional approach to this problem is regression-based, where deterministic models are trained to directly regress the object shape. Such regression methods possess much higher computational efficiency than generative methods. This raises a natural question: is generative modeling necessary for high performance, or conversely, are regression-based approaches still competitive? To answer this, we design a strong regression-based model, called ZeroShape, based on the converging findings in this field and a novel insight. We also curate a large real-world evaluation benchmark, with objects from three different real-world 3D datasets. This evaluation benchmark is more diverse and an order of magnitude larger than what prior works use to quantitatively evaluate their models, aiming at reducing the evaluation variance in our field. We show that ZeroShape not only achieves superior performance over state-of-the-art methods, but also demonstrates significantly higher computational and data efficiency.

Title: Learning Socio-Temporal Graphs for Multi-Agent Trajectory Prediction. (arXiv:2312.14373v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.14373
• Code URL: null
• Copy Paste: [[2312.14373]] Learning Socio-Temporal Graphs for Multi-Agent Trajectory Prediction(http://arxiv.org/abs/2312.14373)
• Summary:

  In order to predict a pedestrian's trajectory in a crowd accurately, one has to take into account her/his underlying socio-temporal interactions with other pedestrians consistently. Unlike existing work that represents the relevant information separately, partially, or implicitly, we propose a complete representation for it to be fully and explicitly captured and analyzed. In particular, we introduce a Directed Acyclic Graph-based structure, which we term Socio-Temporal Graph (STG), to explicitly capture pair-wise socio-temporal interactions among a group of people across both space and time. Our model is built on a time-varying generative process, whose latent variables determine the structure of the STGs. We design an attention-based model named STGformer that affords an end-to-end pipeline to learn the structure of the STGs for trajectory prediction. Our solution achieves overall state-of-the-art prediction accuracy in two large-scale benchmark datasets. Our analysis shows that a person's past trajectory is critical for predicting another person's future path. Our model learns this relationship with a strong notion of socio-temporal localities. Statistics show that utilizing this information explicitly for prediction yields a noticeable performance gain with respect to the trajectory-only approaches.

Title: Environment-Specific People. (arXiv:2312.14579v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.14579
• Code URL: null
• Copy Paste: [[2312.14579]] Environment-Specific People(http://arxiv.org/abs/2312.14579)
• Summary:

  Despite significant progress in generative image synthesis and full-body generation in particular, state-of-the-art methods are either context-independent, overly reliant to text prompts, or bound to the curated training datasets, such as fashion images with monotonous backgrounds. Here, our goal is to generate people in clothing that is semantically appropriate for a given scene. To this end, we present ESP, a novel method for context-aware full-body generation, that enables photo-realistic inpainting of people into existing "in-the-wild" photographs. ESP is conditioned on a 2D pose and contextual cues that are extracted from the environment photograph and integrated into the generation process. Our models are trained on a dataset containing a set of in-the-wild photographs of people covering a wide range of different environments. The method is analyzed quantitatively and qualitatively, and we show that ESP outperforms state-of-the-art on the task of contextual full-body generation.

Title: Towards Loose-Fitting Garment Animation via Generative Model of Deformation Decomposition. (arXiv:2312.14619v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.14619
• Code URL: null
• Copy Paste: [[2312.14619]] Towards Loose-Fitting Garment Animation via Generative Model of Deformation Decomposition(http://arxiv.org/abs/2312.14619)
• Summary:

  Existing data-driven methods for garment animation, usually driven by linear skinning, although effective on tight garments, do not handle loose-fitting garments with complex deformations well. To address these limitations, we develop a garment generative model based on deformation decomposition to efficiently simulate loose garment deformation without directly using linear skinning. Specifically, we learn a garment generative space with the proposed generative model, where we decouple the latent representation into unposed deformed garments and dynamic offsets during the decoding stage. With explicit garment deformations decomposition, our generative model is able to generate complex pose-driven deformations on canonical garment shapes. Furthermore, we learn to transfer the body motions and previous state of the garment to the latent space to regenerate dynamic results. In addition, we introduce a detail enhancement module in an adversarial training setup to learn high-frequency wrinkles. We demonstrate our method outperforms state-of-the-art data-driven alternatives through extensive experiments and show qualitative and quantitative analysis of results.

Title: Compressing Image-to-Image Translation GANs Using Local Density Structures on Their Learned Manifold. (arXiv:2312.14776v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.14776
• Code URL: null
• Copy Paste: [[2312.14776]] Compressing Image-to-Image Translation GANs Using Local Density Structures on Their Learned Manifold(http://arxiv.org/abs/2312.14776)
• Summary:

  Generative Adversarial Networks (GANs) have shown remarkable success in modeling complex data distributions for image-to-image translation. Still, their high computational demands prohibit their deployment in practical scenarios like edge devices. Existing GAN compression methods mainly rely on knowledge distillation or convolutional classifiers' pruning techniques. Thus, they neglect the critical characteristic of GANs: their local density structure over their learned manifold. Accordingly, we approach GAN compression from a new perspective by explicitly encouraging the pruned model to preserve the density structure of the original parameter-heavy model on its learned manifold. We facilitate this objective for the pruned model by partitioning the learned manifold of the original generator into local neighborhoods around its generated samples. Then, we propose a novel pruning objective to regularize the pruned model to preserve the local density structure over each neighborhood, resembling the kernel density estimation method. Also, we develop a collaborative pruning scheme in which the discriminator and generator are pruned by two pruning agents. We design the agents to capture interactions between the generator and discriminator by exchanging their peer's feedback when determining corresponding models' architectures. Thanks to such a design, our pruning method can efficiently find performant sub-networks and can maintain the balance between the generator and discriminator more effectively compared to baselines during pruning, thereby showing more stable pruning dynamics. Our experiments on image translation GAN models, Pix2Pix and CycleGAN, with various benchmark datasets and architectures demonstrate our method's effectiveness.

Title: The Rate-Distortion-Perception-Classification Tradeoff: Joint Source Coding and Modulation via Inverse-Domain GANs. (arXiv:2312.14792v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.14792
• Code URL: null
• Copy Paste: [[2312.14792]] The Rate-Distortion-Perception-Classification Tradeoff: Joint Source Coding and Modulation via Inverse-Domain GANs(http://arxiv.org/abs/2312.14792)
• Summary:

  The joint source coding and modulation (JSCM) framework was enabled by recent developments in deep learning, which allows to automatically learn from data, and in an end-to-end fashion, the best compression codes and modulation schemes. In this paper, we show the existence of a strict tradeoff between channel rate, distortion, perception, and classification accuracy in a JSCM scenario. We then propose two image compression methods to navigate that tradeoff: an inverse-domain generative adversarial network (ID-GAN), which achieves extreme compression, and a simpler, heuristic method that reveals insights about the performance of ID-GAN. Experiment results not only corroborate the theoretical findings, but also demonstrate that the proposed ID-GAN algorithm significantly improves system performance compared to traditional separation-based methods and recent deep JSCM architectures.

Title: Maximum entropy GFlowNets with soft Q-learning. (arXiv:2312.14331v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.14331
• Code URL: null
• Copy Paste: [[2312.14331]] Maximum entropy GFlowNets with soft Q-learning(http://arxiv.org/abs/2312.14331)
• Summary:

  Generative Flow Networks (GFNs) have emerged as a powerful tool for sampling discrete objects from unnormalized distributions, offering a scalable alternative to Markov Chain Monte Carlo (MCMC) methods. While GFNs draw inspiration from maximum entropy reinforcement learning (RL), the connection between the two has largely been unclear and seemingly applicable only in specific cases. This paper addresses the connection by constructing an appropriate reward function, thereby establishing an exact relationship between GFNs and maximum entropy RL. This construction allows us to introduce maximum entropy GFNs, which, in contrast to GFNs with uniform backward policy, achieve the maximum entropy attainable by GFNs without constraints on the state space.

Title: Time-changed normalizing flows for accurate SDE modeling. (arXiv:2312.14698v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.14698
• Code URL: null
• Copy Paste: [[2312.14698]] Time-changed normalizing flows for accurate SDE modeling(http://arxiv.org/abs/2312.14698)
• Summary:

  The generative paradigm has become increasingly important in machine learning and deep learning models. Among popular generative models are normalizing flows, which enable exact likelihood estimation by transforming a base distribution through diffeomorphic transformations. Extending the normalizing flow framework to handle time-indexed flows gave dynamic normalizing flows, a powerful tool to model time series, stochastic processes, and neural stochastic differential equations (SDEs). In this work, we propose a novel variant of dynamic normalizing flows, a Time Changed Normalizing Flow (TCNF), based on time deformation of a Brownian motion which constitutes a versatile and extensive family of Gaussian processes. This approach enables us to effectively model some SDEs, that cannot be modeled otherwise, including standard ones such as the well-known Ornstein-Uhlenbeck process, and generalizes prior methodologies, leading to improved results and better inference and prediction capability.

Title: SutraNets: Sub-series Autoregressive Networks for Long-Sequence, Probabilistic Forecasting. (arXiv:2312.14880v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.14880
• Code URL: null
• Copy Paste: [[2312.14880]] SutraNets: Sub-series Autoregressive Networks for Long-Sequence, Probabilistic Forecasting(http://arxiv.org/abs/2312.14880)
• Summary:

  We propose SutraNets, a novel method for neural probabilistic forecasting of long-sequence time series. SutraNets use an autoregressive generative model to factorize the likelihood of long sequences into products of conditional probabilities. When generating long sequences, most autoregressive approaches suffer from harmful error accumulation, as well as challenges in modeling long-distance dependencies. SutraNets treat long, univariate prediction as multivariate prediction over lower-frequency sub-series. Autoregression proceeds across time and across sub-series in order to ensure coherent multivariate (and, hence, high-frequency univariate) outputs. Since sub-series can be generated using fewer steps, SutraNets effectively reduce error accumulation and signal path distances. We find SutraNets to significantly improve forecasting accuracy over competitive alternatives on six real-world datasets, including when we vary the number of sub-series and scale up the depth and width of the underlying sequence models.

Title: FAST: Feature Aware Similarity Thresholding for Weak Unlearning in Black-Box Generative Models. (arXiv:2312.14895v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.14895
• Code URL: https://github.com/Subhodip123/weak-unlearning-gan
• Copy Paste: [[2312.14895]] FAST: Feature Aware Similarity Thresholding for Weak Unlearning in Black-Box Generative Models(http://arxiv.org/abs/2312.14895)
• Summary:

  The heightened emphasis on the regulation of deep generative models, propelled by escalating concerns pertaining to privacy and compliance with regulatory frameworks, underscores the imperative need for precise control mechanisms over these models. This urgency is particularly underscored by instances in which generative models generate outputs that encompass objectionable, offensive, or potentially injurious content. In response, machine unlearning has emerged to selectively forget specific knowledge or remove the influence of undesirable data subsets from pre-trained models. However, modern machine unlearning approaches typically assume access to model parameters and architectural details during unlearning, which is not always feasible. In multitude of downstream tasks, these models function as black-box systems, with inaccessible pre-trained parameters, architectures, and training data. In such scenarios, the possibility of filtering undesired outputs becomes a practical alternative. The primary goal of this study is twofold: first, to elucidate the relationship between filtering and unlearning processes, and second, to formulate a methodology aimed at mitigating the display of undesirable outputs generated from models characterized as black-box systems. Theoretical analysis in this study demonstrates that, in the context of black-box models, filtering can be seen as a form of weak unlearning. Our proposed \textbf{\textit{Feature Aware Similarity Thresholding(FAST)}} method effectively suppresses undesired outputs by systematically encoding the representation of unwanted features in the latent space.

large language model

Title: LLM4VG: Large Language Models Evaluation for Video Grounding. (arXiv:2312.14206v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.14206
• Code URL: null
• Copy Paste: [[2312.14206]] LLM4VG: Large Language Models Evaluation for Video Grounding(http://arxiv.org/abs/2312.14206)
• Summary:

  Recently, researchers have attempted to investigate the capability of LLMs in handling videos and proposed several video LLM models. However, the ability of LLMs to handle video grounding (VG), which is an important time-related video task requiring the model to precisely locate the start and end timestamps of temporal moments in videos that match the given textual queries, still remains unclear and unexplored in literature. To fill the gap, in this paper, we propose the LLM4VG benchmark, which systematically evaluates the performance of different LLMs on video grounding tasks. Based on our proposed LLM4VG, we design extensive experiments to examine two groups of video LLM models on video grounding: (i) the video LLMs trained on the text-video pairs (denoted as VidLLM), and (ii) the LLMs combined with pretrained visual description models such as the video/image captioning model. We propose prompt methods to integrate the instruction of VG and description from different kinds of generators, including caption-based generators for direct visual description and VQA-based generators for information enhancement. We also provide comprehensive comparisons of various VidLLMs and explore the influence of different choices of visual models, LLMs, prompt designs, etc, as well. Our experimental evaluations lead to two conclusions: (i) the existing VidLLMs are still far away from achieving satisfactory video grounding performance, and more time-related video tasks should be included to further fine-tune these models, and (ii) the combination of LLMs and visual models shows preliminary abilities for video grounding with considerable potential for improvement by resorting to more reliable models and further guidance of prompt instructions.

Title: VCoder: Versatile Vision Encoders for Multimodal Large Language Models. (arXiv:2312.14233v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.14233
• Code URL: https://github.com/shi-labs/vcoder
• Copy Paste: [[2312.14233]] VCoder: Versatile Vision Encoders for Multimodal Large Language Models(http://arxiv.org/abs/2312.14233)
• Summary:

  Humans possess the remarkable skill of Visual Perception, the ability to see and understand the seen, helping them make sense of the visual world and, in turn, reason. Multimodal Large Language Models (MLLM) have recently achieved impressive performance on vision-language tasks ranging from visual question-answering and image captioning to visual reasoning and image generation. However, when prompted to identify or count (perceive) the entities in a given image, existing MLLM systems fail. Working towards developing an accurate MLLM system for perception and reasoning, we propose using Versatile vision enCoders (VCoder) as perception eyes for Multimodal LLMs. We feed the VCoder with perception modalities such as segmentation or depth maps, improving the MLLM's perception abilities. Secondly, we leverage the images from COCO and outputs from off-the-shelf vision perception models to create our COCO Segmentation Text (COST) dataset for training and evaluating MLLMs on the object perception task. Thirdly, we introduce metrics to assess the object perception abilities in MLLMs on our COST dataset. Lastly, we provide extensive experimental evidence proving the VCoder's improved object-level perception skills over existing Multimodal LLMs, including GPT-4V. We open-source our dataset, code, and models to promote research. We open-source our code at https://github.com/SHI-Labs/VCoder

Title: InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks. (arXiv:2312.14238v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.14238
• Code URL: https://github.com/opengvlab/internvl
• Copy Paste: [[2312.14238]] InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks(http://arxiv.org/abs/2312.14238)
• Summary:

  The exponential growth of large language models (LLMs) has opened up numerous possibilities for multi-modal AGI systems. However, the progress in vision and vision-language foundation models, which are also critical elements of multi-modal AGI, has not kept pace with LLMs. In this work, we design a large-scale vision-language foundation model (InternVL), which scales up the vision foundation model to 6 billion parameters and progressively aligns it with the large language model, using web-scale image-text data from various sources. This model can be broadly applied to and achieve state-of-the-art performance on visual perception tasks such as image-level or pixel-level recognition, vision-language tasks such as zero-shot image/video classification, zero-shot image/video-text retrieval, and link with LLMs to create multi-modal dialogue systems. We hope that our research could contribute to the development of multi-modal large models. Code and models are available at https://github.com/OpenGVLab/InternVL.

Title: On Early Detection of Hallucinations in Factual Question Answering. (arXiv:2312.14183v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.14183
• Code URL: null
• Copy Paste: [[2312.14183]] On Early Detection of Hallucinations in Factual Question Answering(http://arxiv.org/abs/2312.14183)
• Summary:

  While large language models (LLMs) have taken great strides towards helping humans with a plethora of tasks like search and summarization, hallucinations remain a major impediment towards gaining user trust. The fluency and coherence of model generations even when hallucinating makes it difficult to detect whether or not a model is hallucinating. In this work, we explore if the artifacts associated with the model generations can provide hints that the generation will contain hallucinations. Specifically, we probe LLMs at 1) the inputs via Integrated Gradients based token attribution, 2) the outputs via the Softmax probabilities, and 3) the internal state via self-attention and fully-connected layer activations for signs of hallucinations on open-ended question answering tasks. Our results show that the distributions of these artifacts differ between hallucinated and non-hallucinated generations. Building on this insight, we train binary classifiers that use these artifacts as input features to classify model generations into hallucinations and non-hallucinations. These hallucination classifiers achieve up to 0.80 AUROC. We further show that tokens preceding a hallucination can predict the subsequent hallucination before it occurs.

Title: Large Language Models in Medical Term Classification and Unexpected Misalignment Between Response and Reasoning. (arXiv:2312.14184v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.14184
• Code URL: null
• Copy Paste: [[2312.14184]] Large Language Models in Medical Term Classification and Unexpected Misalignment Between Response and Reasoning(http://arxiv.org/abs/2312.14184)
• Summary:

  This study assesses the ability of state-of-the-art large language models (LLMs) including GPT-3.5, GPT-4, Falcon, and LLaMA 2 to identify patients with mild cognitive impairment (MCI) from discharge summaries and examines instances where the models' responses were misaligned with their reasoning. Utilizing the MIMIC-IV v2.2 database, we focused on a cohort aged 65 and older, verifying MCI diagnoses against ICD codes and expert evaluations. The data was partitioned into training, validation, and testing sets in a 7:2:1 ratio for model fine-tuning and evaluation, with an additional metastatic cancer dataset from MIMIC III used to further assess reasoning consistency. GPT-4 demonstrated superior interpretative capabilities, particularly in response to complex prompts, yet displayed notable response-reasoning inconsistencies. In contrast, open-source models like Falcon and LLaMA 2 achieved high accuracy but lacked explanatory reasoning, underscoring the necessity for further research to optimize both performance and interpretability. The study emphasizes the significance of prompt engineering and the need for further exploration into the unexpected reasoning-response misalignment observed in GPT-4. The results underscore the promise of incorporating LLMs into healthcare diagnostics, contingent upon methodological advancements to ensure accuracy and clinical coherence of AI-generated outputs, thereby improving the trustworthiness of LLMs for medical decision-making.

Title: Illuminating the Black Box: A Psychometric Investigation into the Multifaceted Nature of Large Language Models. (arXiv:2312.14202v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.14202
• Code URL: null
• Copy Paste: [[2312.14202]] Illuminating the Black Box: A Psychometric Investigation into the Multifaceted Nature of Large Language Models(http://arxiv.org/abs/2312.14202)
• Summary:

  This study explores the idea of AI Personality or AInality suggesting that Large Language Models (LLMs) exhibit patterns similar to human personalities. Assuming that LLMs share these patterns with humans, we investigate using human-centered psychometric tests such as the Myers-Briggs Type Indicator (MBTI), Big Five Inventory (BFI), and Short Dark Triad (SD3) to identify and confirm LLM personality types. By introducing role-play prompts, we demonstrate the adaptability of LLMs, showing their ability to switch dynamically between different personality types. Using projective tests, such as the Washington University Sentence Completion Test (WUSCT), we uncover hidden aspects of LLM personalities that are not easily accessible through direct questioning. Projective tests allowed for a deep exploration of LLMs cognitive processes and thought patterns and gave us a multidimensional view of AInality. Our machine learning analysis revealed that LLMs exhibit distinct AInality traits and manifest diverse personality types, demonstrating dynamic shifts in response to external instructions. This study pioneers the application of projective tests on LLMs, shedding light on their diverse and adaptable AInality traits.

Title: Experimenting with Large Language Models and vector embeddings in NASA SciX. (arXiv:2312.14211v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.14211
• Code URL: null
• Copy Paste: [[2312.14211]] Experimenting with Large Language Models and vector embeddings in NASA SciX(http://arxiv.org/abs/2312.14211)
• Summary:

  Open-source Large Language Models enable projects such as NASA SciX (i.e., NASA ADS) to think out of the box and try alternative approaches for information retrieval and data augmentation, while respecting data copyright and users' privacy. However, when large language models are directly prompted with questions without any context, they are prone to hallucination. At NASA SciX we have developed an experiment where we created semantic vectors for our large collection of abstracts and full-text content, and we designed a prompt system to ask questions using contextual chunks from our system. Based on a non-systematic human evaluation, the experiment shows a lower degree of hallucination and better responses when using Retrieval Augmented Generation. Further exploration is required to design new features and data augmentation processes at NASA SciX that leverages this technology while respecting the high level of trust and quality that the project holds.

Title: SimLM: Can Language Models Infer Parameters of Physical Systems?. (arXiv:2312.14215v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.14215
• Code URL: null
• Copy Paste: [[2312.14215]] SimLM: Can Language Models Infer Parameters of Physical Systems?(http://arxiv.org/abs/2312.14215)
• Summary:

  Recent developments in large-scale machine learning models for general-purpose understanding, translation and generation of language are driving impact across a variety of sectors including medicine, robotics, and scientific discovery. The strength of such Large Language Models (LLMs) stems from the large corpora that they are trained with. While this imbues them with a breadth of capabilities, they have been found unsuitable for some specific types of problems such as advanced mathematics. In this paper, we highlight the inability of LLMs to reason about physics tasks. We demonstrate that their ability to infer parameters of physical systems can be improved, without retraining, by augmenting their context with feedback from physical simulation.

Title: Deep de Finetti: Recovering Topic Distributions from Large Language Models. (arXiv:2312.14226v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.14226
• Code URL: null
• Copy Paste: [[2312.14226]] Deep de Finetti: Recovering Topic Distributions from Large Language Models(http://arxiv.org/abs/2312.14226)
• Summary:

  Large language models (LLMs) can produce long, coherent passages of text, suggesting that LLMs, although trained on next-word prediction, must represent the latent structure that characterizes a document. Prior work has found that internal representations of LLMs encode one aspect of latent structure, namely syntax; here we investigate a complementary aspect, namely the document's topic structure. We motivate the hypothesis that LLMs capture topic structure by connecting LLM optimization to implicit Bayesian inference. De Finetti's theorem shows that exchangeable probability distributions can be represented as a mixture with respect to a latent generating distribution. Although text is not exchangeable at the level of syntax, exchangeability is a reasonable starting assumption for topic structure. We thus hypothesize that predicting the next token in text will lead LLMs to recover latent topic distributions. We examine this hypothesis using Latent Dirichlet Allocation (LDA), an exchangeable probabilistic topic model, as a target, and we show that the representations formed by LLMs encode both the topics used to generate synthetic data and those used to explain natural corpus data.

Title: Parameter Efficient Tuning Allows Scalable Personalization of LLMs for Text Entry: A Case Study on Abbreviation Expansion. (arXiv:2312.14327v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.14327
• Code URL: null
• Copy Paste: [[2312.14327]] Parameter Efficient Tuning Allows Scalable Personalization of LLMs for Text Entry: A Case Study on Abbreviation Expansion(http://arxiv.org/abs/2312.14327)
• Summary:

  Abbreviation expansion is a strategy used to speed up communication by limiting the amount of typing and using a language model to suggest expansions. Here we look at personalizing a Large Language Model's (LLM) suggestions based on prior conversations to enhance the relevance of predictions, particularly when the user data is small (~1000 samples). Specifically, we compare fine-tuning, prompt-tuning, and retrieval augmented generation of expanded text suggestions for abbreviated inputs. Our case study with a deployed 8B parameter LLM on a real user living with ALS, and experiments on movie character personalization indicates that (1) customization may be necessary in some scenarios and prompt-tuning generalizes well to those, (2) fine-tuning on in-domain data (with as few as 600 samples) still shows some gains, however (3) retrieval augmented few-shot selection also outperforms fine-tuning. (4) Parameter efficient tuning allows for efficient and scalable personalization. For prompt-tuning, we also find that initializing the learned "soft-prompts" to user relevant concept tokens leads to higher accuracy than random initialization.

Title: Context-aware Decoding Reduces Hallucination in Query-focused Summarization. (arXiv:2312.14335v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.14335
• Code URL: null
• Copy Paste: [[2312.14335]] Context-aware Decoding Reduces Hallucination in Query-focused Summarization(http://arxiv.org/abs/2312.14335)
• Summary:

  Query-focused summarization (QFS) aims to provide a summary of a single document/multi documents that can satisfy the information needs of a given query. It is useful for various real-world applications, such as abstractive snippet generation or more recent retrieval augmented generation (RAG). A prototypical QFS pipeline consists of a retriever (sparse or dense retrieval) and a generator (usually a large language model). However, applying large language models (LLM) potentially leads to hallucinations, especially when the evidence contradicts the prior belief of LLMs. There has been growing interest in developing new decoding methods to improve generation quality and reduce hallucination. In this work, we conduct a large-scale reproducibility on one recently proposed decoding method -- Context-aware Decoding (CAD). In addition to replicating CAD's experiments on news summarization datasets, we include experiments on QFS datasets, and conduct more rigorous analysis on computational complexity and hyperparameter sensitivity. Experiments with eight different language models show that performance-wise, CAD improves QFS quality by (1) reducing factuality errors/hallucinations while (2) mostly retaining the match of lexical patterns, measured by ROUGE scores, while also at a cost of increased inference-time FLOPs and reduced decoding speed. The code implementation based on Huggingface Library is made available https://github.com/zhichaoxu-shufe/context-aware-decoding-qfs

Title: Aurora:Activating Chinese chat capability for Mistral-8x7B sparse Mixture-of-Experts through Instruction-Tuning. (arXiv:2312.14557v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.14557
• Code URL: https://github.com/WangRongsheng/Aurora
• Copy Paste: [[2312.14557]] Aurora:Activating Chinese chat capability for Mistral-8x7B sparse Mixture-of-Experts through Instruction-Tuning(http://arxiv.org/abs/2312.14557)
• Summary:

  Existing research has demonstrated that refining large language models (LLMs) through the utilization of machine-generated instruction-following data empowers these models to exhibit impressive zero-shot capabilities for novel tasks, without requiring human-authored instructions. In this paper, we systematically investigate, preprocess, and integrate three Chinese instruction-following datasets with the aim of enhancing the Chinese conversational capabilities of Mixtral-8x7B sparse Mixture-of-Experts model. Through instruction fine-tuning on this carefully processed dataset, we successfully construct the Mixtral-8x7B sparse Mixture-of-Experts model named "Aurora." To assess the performance of Aurora, we utilize three widely recognized benchmark tests: C-Eval, MMLU, and CMMLU. Empirical studies validate the effectiveness of instruction fine-tuning applied to Mixtral-8x7B sparse Mixture-of-Experts model. This work is pioneering in the execution of instruction fine-tuning on a sparse expert-mixed model, marking a significant breakthrough in enhancing the capabilities of this model architecture. Our code, data and model are publicly available at: https://github.com/WangRongsheng/Aurora

Title: Large Language Model (LLM) Bias Index -- LLMBI. (arXiv:2312.14769v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.14769
• Code URL: null
• Copy Paste: [[2312.14769]] Large Language Model (LLM) Bias Index -- LLMBI(http://arxiv.org/abs/2312.14769)
• Summary:

  The Large Language Model Bias Index (LLMBI) is a pioneering approach designed to quantify and address biases inherent in large language models (LLMs), such as GPT-4. We recognise the increasing prevalence and impact of LLMs across diverse sectors. This research introduces a novel metric, LLMBI, to systematically measure and mitigate biases potentially skewing model responses. We formulated LLMBI using a composite scoring system incorporating multiple dimensions of bias, including but not limited to age, gender, and racial biases.

  To operationalise this metric, we engaged in a multi-step process involving collecting and annotating LLM responses, applying sophisticated Natural Language Processing (NLP) techniques for bias detection, and computing the LLMBI score through a specially crafted mathematical formula. The formula integrates weighted averages of various bias dimensions, a penalty for dataset diversity deficiencies, and a correction for sentiment biases. Our empirical analysis, conducted using responses from OpenAI's API, employs advanced sentiment analysis as a representative method for bias detection.

  The research reveals LLMs, whilst demonstrating impressive capabilities in text generation, exhibit varying degrees of bias across different dimensions. LLMBI provides a quantifiable measure to compare biases across models and over time, offering a vital tool for systems engineers, researchers and regulators in enhancing the fairness and reliability of LLMs. It highlights the potential of LLMs in mimicking unbiased human-like responses. Additionally, it underscores the necessity of continuously monitoring and recalibrating such models to align with evolving societal norms and ethical standards.

Title: Semantic Parsing for Complex Data Retrieval: Targeting Query Plans vs. SQL for No-Code Access to Relational Databases. (arXiv:2312.14798v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.14798
• Code URL: null
• Copy Paste: [[2312.14798]] Semantic Parsing for Complex Data Retrieval: Targeting Query Plans vs(http://arxiv.org/abs/2312.14798)
• Summary:

  Large Language Models (LLMs) have spurred progress in text-to-SQL, the task of generating SQL queries from natural language questions based on a given database schema. Despite the declarative nature of SQL, it continues to be a complex programming language. In this paper, we investigate the potential of an alternative query language with simpler syntax and modular specification of complex queries. The purpose is to create a query language that can be learned more easily by modern neural semantic parsing architectures while also enabling non-programmers to better assess the validity of the query plans produced by an interactive query plan assistant.

  The proposed alternative query language is called Query Plan Language (QPL). It is designed to be modular and can be translated into a restricted form of SQL Common Table Expressions (CTEs). The aim of QPL is to make complex data retrieval accessible to non-programmers by allowing users to express their questions in natural language while also providing an easier-to-verify target language. The paper demonstrates how neural LLMs can benefit from QPL's modularity to generate complex query plans in a compositional manner. This involves a question decomposition strategy and a planning stage.

  We conduct experiments on a version of the Spider text-to-SQL dataset that has been converted to QPL. The hierarchical structure of QPL programs enables us to measure query complexity naturally. Based on this assessment, we identify the low accuracy of existing text-to-SQL systems on complex compositional queries. We present ways to address the challenge of complex queries in an iterative, user-controlled manner, using fine-tuned LLMs and a variety of prompting strategies in a compositional manner.

Title: YAYI 2: Multilingual Open-Source Large Language Models. (arXiv:2312.14862v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.14862
• Code URL: null
• Copy Paste: [[2312.14862]] YAYI 2: Multilingual Open-Source Large Language Models(http://arxiv.org/abs/2312.14862)
• Summary:

  As the latest advancements in natural language processing, large language models (LLMs) have achieved human-level language understanding and generation abilities in many real-world tasks, and even have been regarded as a potential path to the artificial general intelligence. To better facilitate research on LLMs, many open-source LLMs, such as Llama 2 and Falcon, have recently been proposed and gained comparable performances to proprietary models. However, these models are primarily designed for English scenarios and exhibit poor performances in Chinese contexts. In this technical report, we propose YAYI 2, including both base and chat models, with 30 billion parameters. YAYI 2 is pre-trained from scratch on a multilingual corpus which contains 2.65 trillion tokens filtered by our pre-training data processing pipeline. The base model is aligned with human values through supervised fine-tuning with millions of instructions and reinforcement learning from human feedback. Extensive experiments on multiple benchmarks, such as MMLU and CMMLU, consistently demonstrate that the proposed YAYI 2 outperforms other similar sized open-source models.

Title: Numerical Reasoning for Financial Reports. (arXiv:2312.14870v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.14870
• Code URL: null
• Copy Paste: [[2312.14870]] Numerical Reasoning for Financial Reports(http://arxiv.org/abs/2312.14870)
• Summary:

  Financial reports offer critical insights into a company's operations, yet their extensive length typically spanning 30 40 pages poses challenges for swift decision making in dynamic markets. To address this, we leveraged finetuned Large Language Models (LLMs) to distill key indicators and operational metrics from these reports basis questions from the user. We devised a method to locate critical data, and leverage the FinQA dataset to fine-tune both Llama-2 7B and T5 models for customized question answering. We achieved results comparable to baseline on the final numerical answer, a competitive accuracy in numerical reasoning and calculation.

Title: ChatGPT, Llama, can you write my report? An experiment on assisted digital forensics reports written using (Local) Large Language Models. (arXiv:2312.14607v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.14607
• Code URL: null
• Copy Paste: [[2312.14607]] ChatGPT, Llama, can you write my report? An experiment on assisted digital forensics reports written using (Local) Large Language Models(http://arxiv.org/abs/2312.14607)
• Summary:

  Generative AIs, especially Large Language Models (LLMs) such as ChatGPT or Llama, have advanced significantly, positioning them as valuable tools for digital forensics. While initial studies have explored the potential of ChatGPT in the context of investigations, the question of to what extent LLMs can assist the forensic report writing process remains unresolved. To answer the question, this article first examines forensic reports with the goal of generalization (e.g., finding the `average structure' of a report). We then evaluate the strengths and limitations of LLMs for generating the different parts of the forensic report using a case study. This work thus provides valuable insights into the automation of report writing, a critical facet of digital forensics investigations. We conclude that combined with thorough proofreading and corrections, LLMs may assist practitioners during the report writing process but at this point cannot replace them.

Title: A Unified Industrial Large Knowledge Model Framework in Smart Manufacturing. (arXiv:2312.14428v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.14428
• Code URL: null
• Copy Paste: [[2312.14428]] A Unified Industrial Large Knowledge Model Framework in Smart Manufacturing(http://arxiv.org/abs/2312.14428)
• Summary:

  The recent emergence of large language models (LLMs) shows the potential for artificial general intelligence, revealing new opportunities in industry 4.0 and smart manufacturing. However, a notable gap exists in applying these LLMs in industry, primarily due to their training on general knowledge rather than domain-specific knowledge. Such specialized domain knowledge is vital for effectively addressing the complex needs of industrial applications. To bridge this gap, this paper proposes an Industrial Large Knowledge Model (ILKM) framework emphasizing their potential to revolutionize the industry in smart manufacturing. In addition, ILKMs and LLMs are compared from eight perspectives. Finally, "6S Principle" is proposed as the guideline for the development of ILKMs in smart manufacturing.

Title: A Survey of Reinforcement Learning from Human Feedback. (arXiv:2312.14925v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.14925
• Code URL: null
• Copy Paste: [[2312.14925]] A Survey of Reinforcement Learning from Human Feedback(http://arxiv.org/abs/2312.14925)
• Summary:

  Reinforcement learning from human feedback (RLHF) is a variant of reinforcement learning (RL) that learns from human feedback instead of relying on an engineered reward function. Building on prior work on the related setting of preference-based reinforcement learning (PbRL), it stands at the intersection of artificial intelligence and human-computer interaction. This positioning offers a promising avenue to enhance the performance and adaptability of intelligent systems while also improving the alignment of their objectives with human values. The training of Large Language Models (LLMs) has impressively demonstrated this potential in recent years, where RLHF played a decisive role in targeting the model's capabilities toward human objectives. This article provides a comprehensive overview of the fundamentals of RLHF, exploring the intricate dynamics between machine agents and human input. While recent focus has been on RLHF for LLMs, our survey adopts a broader perspective, examining the diverse applications and wide-ranging impact of the technique. We delve into the core principles that underpin RLHF, shedding light on the symbiotic relationship between algorithms and human feedback, and discuss the main research trends in the field. By synthesizing the current landscape of RLHF research, this article aims to provide researchers as well as practitioners with a comprehensive understanding of this rapidly growing field of research.

segmentation

Title: Variance-insensitive and Target-preserving Mask Refinement for Interactive Image Segmentation. (arXiv:2312.14387v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.14387
• Code URL: null
• Copy Paste: [[2312.14387]] Variance-insensitive and Target-preserving Mask Refinement for Interactive Image Segmentation(http://arxiv.org/abs/2312.14387)
• Summary:

  Point-based interactive image segmentation can ease the burden of mask annotation in applications such as semantic segmentation and image editing. However, fully extracting the target mask with limited user inputs remains challenging. We introduce a novel method, Variance-Insensitive and Target-Preserving Mask Refinement to enhance segmentation quality with fewer user inputs. Regarding the last segmentation result as the initial mask, an iterative refinement process is commonly employed to continually enhance the initial mask. Nevertheless, conventional techniques suffer from sensitivity to the variance in the initial mask. To circumvent this problem, our proposed method incorporates a mask matching algorithm for ensuring consistent inferences from different types of initial masks. We also introduce a target-aware zooming algorithm to preserve object information during downsampling, balancing efficiency and accuracy. Experiments on GrabCut, Berkeley, SBD, and DAVIS datasets demonstrate our method's state-of-the-art performance in interactive image segmentation.

Title: Part to Whole: Collaborative Prompting for Surgical Instrument Segmentation. (arXiv:2312.14481v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.14481
• Code URL: https://github.com/wenxi-yue/surgicalpart-sam
• Copy Paste: [[2312.14481]] Part to Whole: Collaborative Prompting for Surgical Instrument Segmentation(http://arxiv.org/abs/2312.14481)
• Summary:

  Foundation models like the Segment Anything Model (SAM) have demonstrated promise in generic object segmentation. However, directly applying SAM to surgical instrument segmentation presents key challenges. First, SAM relies on per-frame point-or-box prompts which complicate surgeon-computer interaction. Also, SAM yields suboptimal performance on segmenting surgical instruments, owing to insufficient surgical data in its pre-training as well as the complex structure and fine-grained details of various surgical instruments. To address these challenges, in this paper, we investigate text promptable surgical instrument segmentation and propose SP-SAM (SurgicalPart-SAM), a novel efficient-tuning approach that integrates surgical instrument structure knowledge with the generic segmentation knowledge of SAM. Specifically, we achieve this by proposing (1) collaborative prompts in the text form "[part name] of [instrument category name]" that decompose instruments into fine-grained parts; (2) a Cross-Modal Prompt Encoder that encodes text prompts jointly with visual embeddings into discriminative part-level representations; and (3) a Part-to-Whole Selective Fusion and a Hierarchical Decoding strategy that selectively assemble the part-level representations into a whole for accurate instrument segmentation. Built upon them, SP-SAM acquires a better capability to comprehend surgical instrument structures and distinguish between various categories. Extensive experiments on both the EndoVis2018 and EndoVis2017 datasets demonstrate SP-SAM's state-of-the-art performance with minimal tunable parameters. Code is at https://github.com/wenxi-yue/SurgicalPart-SAM.

Title: BonnBeetClouds3D: A Dataset Towards Point Cloud-based Organ-level Phenotyping of Sugar Beet Plants under Field Conditions. (arXiv:2312.14706v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.14706
• Code URL: null
• Copy Paste: [[2312.14706]] BonnBeetClouds3D: A Dataset Towards Point Cloud-based Organ-level Phenotyping of Sugar Beet Plants under Field Conditions(http://arxiv.org/abs/2312.14706)
• Summary:

  Agricultural production is facing severe challenges in the next decades induced by climate change and the need for sustainability, reducing its impact on the environment. Advancements in field management through non-chemical weeding by robots in combination with monitoring of crops by autonomous unmanned aerial vehicles (UAVs) and breeding of novel and more resilient crop varieties are helpful to address these challenges. The analysis of plant traits, called phenotyping, is an essential activity in plant breeding, it however involves a great amount of manual labor. With this paper, we address the problem of automatic fine-grained organ-level geometric analysis needed for precision phenotyping. As the availability of real-world data in this domain is relatively scarce, we propose a novel dataset that was acquired using UAVs capturing high-resolution images of a real breeding trial containing 48 plant varieties and therefore covering great morphological and appearance diversity. This enables the development of approaches for autonomous phenotyping that generalize well to different varieties. Based on overlapping high-resolution images from multiple viewing angles, we compute photogrammetric dense point clouds and provide detailed and accurate point-wise labels for plants, leaves, and salient points as the tip and the base. Additionally, we include measurements of phenotypic traits performed by experts from the German Federal Plant Variety Office on the real plants, allowing the evaluation of new approaches not only on segmentation and keypoint detection but also directly on the downstream tasks. The provided labeled point clouds enable fine-grained plant analysis and support further progress in the development of automatic phenotyping approaches, but also enable further research in surface reconstruction, point cloud completion, and semantic interpretation of point clouds.