secure

Title: An Innovative Design of Substitution Box Using Trigonometric Transformation. (arXiv:2311.16107v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.16107
• Code URL: null
• Copy Paste: [[2311.16107]] An Innovative Design of Substitution Box Using Trigonometric Transformation(http://arxiv.org/abs/2311.16107)
• Summary:

  As the number of hacking events and cyber threats keeps going up, it is getting harder and harder to communicate securely and keep personal information safe on the Internet. Cryptography is a very important way to deal with these problems because it can secure data by changing it from one form to another. In this study, we show a new, lightweight algorithm that is based on trigonometric ideas and offers a lot of security by making it less likely that cryptanalysis will work. The performance of our suggested algorithm is better than that of older methods like the Hill cipher, Blowfish, and DES. Even though traditional methods offer good security, they may have more work to do, which slows them down. The suggested algorithm tries to close this gap by offering a solution based on trigonometric ideas that are both fast and safe. The main goal of this study is to come up with a small but strong encryption algorithm that cannot be broken by cryptanalysis and keeps Internet communication safe. We want to speed up the coding process without making it less secure by using trigonometric principles. The suggested algorithm uses trigonometric functions and operations to create non-linearity and confusion, making it resistant to both differential and linear cryptanalysis. We show that the suggested algorithm is more secure and faster than traditional methods like the Hill cipher, Blowfish, and DES by doing a lot of research and testing. Combining trigonometric ideas with a simple design makes it workable for real world uses and offers a promising way to protect data on the Internet.

Title: Secure Arcade: A Gamified Defense Against Cyber Attacks. (arXiv:2311.16131v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.16131
• Code URL: null
• Copy Paste: [[2311.16131]] Secure Arcade: A Gamified Defense Against Cyber Attacks(http://arxiv.org/abs/2311.16131)
• Summary:

  In modernity, we continually receive increasingly intricate technologies that allow us to increase our lives convenience and efficiency. Our technology, particularly technology available over the internet, is advancing at unprecedented speed. However, this speed of advancement allows those behind malicious attacks to have an increasingly easier time taking advantage of those who know little about computer security. Unfortunately, education in the computer security field is generally limited only to tertiary education. This research addresses this problem through a gamified web-based application that drives users to reach learning goals to help them become more vigilant internet users: 1. Learn and memorize general computer security terminology, 2. Become familiar with basic cryptography concepts, 3. Learn to recognize potential phishing scams via email quickly, and 4. Learn common attacks on servers and how to deal with them.

Title: Secure Traversable Event logging for Responsible Identification of Vertically Partitioned Health Data. (arXiv:2311.16575v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.16575
• Code URL: null
• Copy Paste: [[2311.16575]] Secure Traversable Event logging for Responsible Identification of Vertically Partitioned Health Data(http://arxiv.org/abs/2311.16575)
• Summary:

  We aim to provide a solution for the secure identification of sensitive medical information. We consider a repository of de-identified medical data that is stored in the custody of a Healthcare Institution. The identifying information that is stored separately can be associated with the medical information only by a subset of users referred to as custodians. This paper intends to secure the process of associating identifying information with sensitive medical information. We also enforce the responsibility of the custodians by maintaining an immutable ledger documenting the events of such information identification. The paper proposes a scheme for constructing ledger entries that allow the custodians and patients to browse through the entries which they are associated with. However, in order to respect their privacy, such traversal requires appropriate credentials to ensure that a user cannot gain any information regarding the other users involved in the system unless they are both involved in the same operation.

security

Title: Ransomware Detection and Classification using Machine Learning. (arXiv:2311.16143v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.16143
• Code URL: null
• Copy Paste: [[2311.16143]] Ransomware Detection and Classification using Machine Learning(http://arxiv.org/abs/2311.16143)
• Summary:

  Vicious assaults, malware, and various ransomware pose a cybersecurity threat, causing considerable damage to computer structures, servers, and mobile and web apps across various industries and businesses. These safety concerns are important and must be addressed immediately. Ransomware detection and classification are critical for guaranteeing rapid reaction and prevention. This study uses the XGBoost classifier and Random Forest (RF) algorithms to detect and classify ransomware attacks. This approach involves analyzing the behaviour of ransomware and extracting relevant features that can help distinguish between different ransomware families. The models are evaluated on a dataset of ransomware attacks and demonstrate their effectiveness in accurately detecting and classifying ransomware. The results show that the XGBoost classifier, Random Forest Classifiers, can effectively detect and classify different ransomware attacks with high accuracy, thereby providing a valuable tool for enhancing cybersecurity.

Title: Word for Person: Zero-shot Composed Person Retrieval. (arXiv:2311.16515v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16515
• Code URL: https://github.com/Delong-liu-bupt/Word4Per
• Copy Paste: [[2311.16515]] Word for Person: Zero-shot Composed Person Retrieval(http://arxiv.org/abs/2311.16515)
• Summary:

  Searching for specific person has great security value and social benefits, and it often involves a combination of visual and textual information. Conventional person retrieval methods, whether image-based or text-based, usually fall short in effectively harnessing both types of information, leading to the loss of accuracy. In this paper, a whole new task called Composed Person Retrieval (CPR) is proposed to jointly utilize both image and text information for target person retrieval. However, the supervised CPR must depend on very costly manual annotation dataset, while there are currently no available resources. To mitigate this issue, we firstly introduce the Zero-shot Composed Person Retrieval (ZS-CPR), which leverages existing domain-related data to resolve the CPR problem without reliance on expensive annotations. Secondly, to learn ZS-CPR model, we propose a two-stage learning framework, Word4Per, where a lightweight Textual Inversion Network (TINet) and a text-based person retrieval model based on fine-tuned Contrastive Language-Image Pre-training (CLIP) network are learned without utilizing any CPR data. Thirdly, a finely annotated Image-Text Composed Person Retrieval dataset (ITCPR) is built as the benchmark to assess the performance of the proposed Word4Per framework. Extensive experiments under both Rank-1 and mAP demonstrate the effectiveness of Word4Per for the ZS-CPR task, surpassing the comparative methods by over 10%. The code and ITCPR dataset will be publicly available at https://github.com/Delong-liu-bupt/Word4Per.

Title: Ignore This Title and HackAPrompt: Exposing Systemic Vulnerabilities of LLMs through a Global Scale Prompt Hacking Competition. (arXiv:2311.16119v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.16119
• Code URL: null
• Copy Paste: [[2311.16119]] Ignore This Title and HackAPrompt: Exposing Systemic Vulnerabilities of LLMs through a Global Scale Prompt Hacking Competition(http://arxiv.org/abs/2311.16119)
• Summary:

  Large Language Models (LLMs) are increasingly being deployed in interactive contexts that involve direct user engagement, such as chatbots and writing assistants. These deployments are increasingly plagued by prompt injection and jailbreaking (collectively, prompt hacking), in which models are manipulated to ignore their original instructions and instead follow potentially malicious ones. Although widely acknowledged as a significant security threat, there is a dearth of large-scale resources and quantitative studies on prompt hacking. To address this lacuna, we launch a global prompt hacking competition, which allows for free-form human input attacks. We elicit 600K+ adversarial prompts against three state-of-the-art LLMs. We describe the dataset, which empirically verifies that current LLMs can indeed be manipulated via prompt hacking. We also present a comprehensive taxonomical ontology of the types of adversarial prompts.

Title: ERASER: Machine Unlearning in MLaaS via an Inference Serving-Aware Approach. (arXiv:2311.16136v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.16136
• Code URL: null
• Copy Paste: [[2311.16136]] ERASER: Machine Unlearning in MLaaS via an Inference Serving-Aware Approach(http://arxiv.org/abs/2311.16136)
• Summary:

  Over the past few years, Machine Learning-as-a-Service (MLaaS) has received a surging demand for supporting Machine Learning-driven services to offer revolutionized user experience across diverse application areas. MLaaS provides inference service with low inference latency to application users based on an ML model trained using a dataset collected from numerous individual data owners. Recently, for the sake of data owners' privacy and to comply with the "right to be forgotten (RTBF)" as enacted by data protection legislation, many machine unlearning methods have been proposed to remove data owners' data from trained models upon their unlearning requests. However, despite their promising efficiency, almost all existing machine unlearning methods handle unlearning requests in a manner that is independent of inference requests, which unfortunately introduces new security and privacy vulnerabilities for machine unlearning in MLaaS. In this paper, we propose the ERASER framework for machinE unleaRning in MLaAS via an inferencE seRving-aware approach. ERASER proposes a novel certified inference consistency mechanism that reduces inference latency by selectively postponing unlearning execution incurred by unlearning requests from data owners, while strictly adhering to the RTBF principle. ERASER offers three groups of design choices to allow for tailor-made variants that best suit the specific environments and preferences of different MLaaS systems. Extensive empirical evaluations across various settings confirm ERASER's effectiveness, e.g., it can effectively save up to 99% of inference latency and 31% of computation overhead over the inference-oblivion baseline.

Title: Understanding the Effectiveness of Large Language Models in Detecting Security Vulnerabilities. (arXiv:2311.16169v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.16169
• Code URL: null
• Copy Paste: [[2311.16169]] Understanding the Effectiveness of Large Language Models in Detecting Security Vulnerabilities(http://arxiv.org/abs/2311.16169)
• Summary:

  Security vulnerabilities in modern software are prevalent and harmful. While automated vulnerability detection tools have made promising progress, their scalability and applicability remain challenging. Recently, Large Language Models (LLMs), such as GPT-4 and CodeLlama, have demonstrated remarkable performance on code-related tasks. However, it is unknown whether such LLMs can do complex reasoning over code. In this work, we explore whether pre-trained LLMs can detect security vulnerabilities and address the limitations of existing tools. We evaluate the effectiveness of pre-trained LLMs on a set of five diverse security benchmarks spanning two languages, Java and C/C++, and including code samples from synthetic and real-world projects. We evaluate the effectiveness of LLMs in terms of their performance, explainability, and robustness.

  By designing a series of effective prompting strategies, we obtain the best results on the synthetic datasets with GPT-4: F1 scores of 0.79 on OWASP, 0.86 on Juliet Java, and 0.89 on Juliet C/C++. Expectedly, the performance of LLMs drops on the more challenging real-world datasets: CVEFixes Java and CVEFixes C/C++, with GPT-4 reporting F1 scores of 0.48 and 0.62, respectively. We show that LLMs can often perform better than existing static analysis and deep learning-based vulnerability detection tools, especially for certain classes of vulnerabilities. Moreover, LLMs also often provide reliable explanations, identifying the vulnerable data flows in code. We find that fine-tuning smaller LLMs can outperform the larger LLMs on synthetic datasets but provide limited gains on real-world datasets. When subjected to adversarial attacks on code, LLMs show mild degradation, with average accuracy reduction of up to 12.67%. Finally, we share our insights and recommendations for future work on leveraging LLMs for vulnerability detection.

Title: "Do Users fall for Real Adversarial Phishing?" Investigating the Human response to Evasive Webpages. (arXiv:2311.16383v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.16383
• Code URL: null
• Copy Paste: [[2311.16383]] "Do Users fall for Real Adversarial Phishing?" Investigating the Human response to Evasive Webpages(http://arxiv.org/abs/2311.16383)
• Summary:

  Phishing websites are everywhere, and countermeasures based on static blocklists cannot cope with such a threat. To address this problem, state-of-the-art solutions entail the application of machine learning (ML) to detect phishing websites by checking if they visually resemble webpages of well-known brands. These techniques have achieved promising results in research and, consequently, some security companies began to deploy them also in their phishing detection systems (PDS). However, ML methods are not perfect and some samples are bound to bypass even production-grade PDS.

  In this paper, we scrutinize whether 'genuine phishing websites' that evade 'commercial ML-based PDS' represent a problem "in reality". Although nobody likes landing on a phishing webpage, a false negative may not lead to serious consequences if the users (i.e., the actual target of phishing) can recognize that "something is phishy". Practically, we carry out the first user-study (N=126) wherein we assess whether unsuspecting users (having diverse backgrounds) are deceived by 'adversarial' phishing webpages that evaded a real PDS. We found that some well-crafted adversarial webpages can trick most participants (even IT experts), albeit others are easily recognized by most users. Our study is relevant for practitioners, since it allows prioritizing phishing webpages that simultaneously fool (i) machines and (ii) humans -- i.e., their intended targets.

Title: Understanding the Process of Data Labeling in Cybersecurity. (arXiv:2311.16388v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.16388
• Code URL: null
• Copy Paste: [[2311.16388]] Understanding the Process of Data Labeling in Cybersecurity(http://arxiv.org/abs/2311.16388)
• Summary:

  Many domains now leverage the benefits of Machine Learning (ML), which promises solutions that can autonomously learn to solve complex tasks by training over some data. Unfortunately, in cyberthreat detection, high-quality data is hard to come by. Moreover, for some specific applications of ML, such data must be labeled by human operators. Many works "assume" that labeling is tough/challenging/costly in cyberthreat detection, thereby proposing solutions to address such a hurdle. Yet, we found no work that specifically addresses the process of labeling 'from the viewpoint of ML security practitioners'. This is a problem: to this date, it is still mostly unknown how labeling is done in practice -- thereby preventing one from pinpointing "what is needed" in the real world.

  In this paper, we take the first step to build a bridge between academic research and security practice in the context of data labeling. First, we reach out to five subject matter experts and carry out open interviews to identify pain points in their labeling routines. Then, by using our findings as a scaffold, we conduct a user study with 13 practitioners from large security companies, and ask detailed questions on subjects such as active learning, costs of labeling, and revision of labels. Finally, we perform proof-of-concept experiments addressing labeling-related aspects in cyberthreat detection that are sometimes overlooked in research. Altogether, our contributions and recommendations serve as a stepping stone to future endeavors aimed at improving the quality and robustness of ML-driven security systems. We release our resources.

Title: Abusing Processor Exception for General Binary Instrumentation on Bare-metal Embedded Devices. (arXiv:2311.16532v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.16532
• Code URL: null
• Copy Paste: [[2311.16532]] Abusing Processor Exception for General Binary Instrumentation on Bare-metal Embedded Devices(http://arxiv.org/abs/2311.16532)
• Summary:

  Analyzing the security of closed-source drivers and libraries in embedded systems holds significant importance, given their fundamental role in the supply chain. Unlike x86, embedded platforms lack comprehensive binary manipulating tools, making it difficult for researchers and developers to effectively detect and patch security issues in such closed-source components. Existing works either depend on full-fledged operating system features or suffer from tedious corner cases, restricting their application to bare-metal firmware prevalent in embedded environments. In this paper, we present PIFER (Practical Instrumenting Framework for Embedded fiRmware) that enables general and fine-grained static binary instrumentation for embedded bare-metal firmware. By abusing the built-in hardware exception-handling mechanism of the embedded processors, PIFER can perform instrumentation on arbitrary target addresses. Additionally, We propose an instruction translation-based scheme to guarantee the correct execution of the original firmware after patching. We evaluate PIFER against real-world, complex firmware, including Zephyr RTOS, CoreMark benchmark, and a close-sourced commercial product. The results indicate that PIFER correctly instrumented 98.9% of the instructions. Further, a comprehensive performance evaluation was conducted, demonstrating the practicality and efficiency of our work.

Title: A Unified Hardware-based Threat Detector for AI Accelerators. (arXiv:2311.16684v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.16684
• Code URL: null
• Copy Paste: [[2311.16684]] A Unified Hardware-based Threat Detector for AI Accelerators(http://arxiv.org/abs/2311.16684)
• Summary:

  The proliferation of AI technology gives rise to a variety of security threats, which significantly compromise the confidentiality and integrity of AI models and applications. Existing software-based solutions mainly target one specific attack, and require the implementation into the models, rendering them less practical. We design UniGuard, a novel unified and non-intrusive detection methodology to safeguard FPGA-based AI accelerators. The core idea of UniGuard is to harness power side-channel information generated during model inference to spot any anomaly. We employ a Time-to-Digital Converter to capture power fluctuations and train a supervised machine learning model to identify various types of threats. Evaluations demonstrate that UniGuard can achieve 94.0% attack detection accuracy, with high generalization over unknown or adaptive attacks and robustness against varied configurations (e.g., sensor frequency and location).

Title: Blockchain-based Zero Trust on the Edge. (arXiv:2311.16744v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.16744
• Code URL: null
• Copy Paste: [[2311.16744]] Blockchain-based Zero Trust on the Edge(http://arxiv.org/abs/2311.16744)
• Summary:

  Internet of Things (IoT) devices pose significant security challenges due to their heterogeneity (i.e., hardware and software) and vulnerability to extensive attack surfaces. Today's conventional perimeter-based systems use credential-based authentication (e.g., username/password, certificates, etc.) to decide whether an actor can access a network. However, the verification process occurs only at the system's perimeter because most IoT devices lack robust security measures due to their limited hardware and software capabilities, making them highly vulnerable. Therefore, this paper proposes a novel approach based on Zero Trust Architecture (ZTA) extended with blockchain to further enhance security. The blockchain component serves as an immutable database for storing users' requests and is used to verify trustworthiness by analyzing and identifying potentially malicious user activities. We discuss the framework, processes of the approach, and the experiments carried out on a testbed to validate its feasibility and applicability in the smart city context. Lastly, the evaluation focuses on non-functional properties such as performance, scalability, and complexity.

privacy

Title: Data Analytics with Differential Privacy. (arXiv:2311.16104v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.16104
• Code URL: null
• Copy Paste: [[2311.16104]] Data Analytics with Differential Privacy(http://arxiv.org/abs/2311.16104)
• Summary:

  Differential privacy is the state-of-the-art definition for privacy, guaranteeing that any analysis performed on a sensitive dataset leaks no information about the individuals whose data are contained therein. In this thesis, we develop differentially private algorithms to analyze distributed and streaming data. In the distributed model, we consider the particular problem of learning -- in a distributed fashion -- a global model of the data, that can subsequently be used for arbitrary analyses. We build upon PrivBayes, a differentially private method that approximates the high-dimensional distribution of a centralized dataset as a product of low-order distributions, utilizing a Bayesian Network model. We examine three novel approaches to learning a global Bayesian Network from distributed data, while offering the differential privacy guarantee to all local datasets. Our work includes a detailed theoretical analysis of the distributed, differentially private entropy estimator which we use in one of our algorithms, as well as a detailed experimental evaluation, using both synthetic and real-world data. In the streaming model, we focus on the problem of estimating the density of a stream of users, which expresses the fraction of all users that actually appear in the stream. We offer one of the strongest privacy guarantees for the streaming model, user-level pan-privacy, which ensures that the privacy of any user is protected, even against an adversary that observes the internal state of the algorithm. We provide a detailed analysis of an existing, sampling-based algorithm for the problem and propose two novel modifications that significantly improve it, both theoretically and experimentally, by optimally using all the allocated "privacy budget."

Title: A Privacy-preserving Central Bank Ledger for Central Bank Digital Currency. (arXiv:2311.16105v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.16105
• Code URL: null
• Copy Paste: [[2311.16105]] A Privacy-preserving Central Bank Ledger for Central Bank Digital Currency(http://arxiv.org/abs/2311.16105)
• Summary:

  Retail central bank digital currency (rCBDC) is seen as a key upgrade of the monetary system in the 21st century. However, privacy concerns are the main impediment to rCBDC's development and roll-out. On the one hand, the rights of people to keep their transactions private should be protected, including against central bank surveillance. On the other hand, the central bank needs to ensure that no over-issuance of money or other frauds occur, demanding a certain form of knowledge of rCBDC transactions to safeguard against malicious users. This work focuses on rCBDC architectures based on the unspent transaction output (UTXO) data model and tackles the research problem of preserving a sufficient degree of privacy for UTXO transaction records while allowing the central bank to verify their correctness. User privacy is not adequately addressed in the UTXO-based rCBDC architectures. Using evolving public keys as pseudonyms to hide the real identities of users only solves the privacy issue partially. Some information could still be leaked out. This work investigates techniques to address the shortcomings of the pseudonym approach. First, a Pedersen commitment scheme is applied to hide the transaction values of a UTXO transaction while allowing the central bank to verify that no over-issuance of rCBDC has occurred in the transaction.This work uses a Schnorr signature to prove no over-issuance of money, which reduces overheads and enables a non-interactive proof. Then, Coinjoin is applied to aggregate UTXO transactions from different users into one larger UTXO transaction to obfuscate the payer-payee relationship while preserving the correctness of the amount of money flow. This work applies k-anonymity to analyse the privacy guarantee of Coinjoin. By modelling the transaction traffic by a Poisson process, the trade-off between anonymity and transaction confirmation time of Coinjoin is analysed.

Title: AI and Democracy's Digital Identity Crisis. (arXiv:2311.16115v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.16115
• Code URL: null
• Copy Paste: [[2311.16115]] AI and Democracy's Digital Identity Crisis(http://arxiv.org/abs/2311.16115)
• Summary:

  AI-enabled tools have become sophisticated enough to allow a small number of individuals to run disinformation campaigns of an unprecedented scale. Privacy-preserving identity attestations can drastically reduce instances of impersonation and make disinformation easy to identify and potentially hinder. By understanding how identity attestations are positioned across the spectrum of decentralization, we can gain a better understanding of the costs and benefits of various attestations. In this paper, we discuss attestation types, including governmental, biometric, federated, and web of trust-based, and include examples such as e-Estonia, China's social credit system, Worldcoin, OAuth, X (formerly Twitter), Gitcoin Passport, and EAS. We believe that the most resilient systems create an identity that evolves and is connected to a network of similarly evolving identities that verify one another. In this type of system, each entity contributes its respective credibility to the attestation process, creating a larger, more comprehensive set of attestations. We believe these systems could be the best approach to authenticating identity and protecting against some of the threats to democracy that AI can pose in the hands of malicious actors. However, governments will likely attempt to mitigate these risks by implementing centralized identity authentication systems; these centralized systems could themselves pose risks to the democratic processes they are built to defend. We therefore recommend that policymakers support the development of standards-setting organizations for identity, provide legal clarity for builders of decentralized tooling, and fund research critical to effective identity authentication systems.

Title: Identifying and Mitigating Vulnerabilities in LLM-Integrated Applications. (arXiv:2311.16153v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.16153
• Code URL: null
• Copy Paste: [[2311.16153]] Identifying and Mitigating Vulnerabilities in LLM-Integrated Applications(http://arxiv.org/abs/2311.16153)
• Summary:

  Large language models (LLMs) are increasingly deployed as the service backend for LLM-integrated applications such as code completion and AI-powered search. LLM-integrated applications serve as middleware to refine users' queries with domain-specific knowledge to better inform LLMs and enhance the responses. Despite numerous opportunities and benefits, LLM-integrated applications also introduce new attack surfaces. Understanding, minimizing, and eliminating these emerging attack surfaces is a new area of research. In this work, we consider a setup where the user and LLM interact via an LLM-integrated application in the middle. We focus on the communication rounds that begin with user's queries and end with LLM-integrated application returning responses to the queries, powered by LLMs at the service backend. For this query-response protocol, we identify potential vulnerabilities that can originate from the malicious application developer or from an outsider threat initiator that is able to control the database access, manipulate and poison data that are high-risk for the user. Successful exploits of the identified vulnerabilities result in the users receiving responses tailored to the intent of a threat initiator. We assess such threats against LLM-integrated applications empowered by OpenAI GPT-3.5 and GPT-4. Our empirical results show that the threats can effectively bypass the restrictions and moderation policies of OpenAI, resulting in users receiving responses that contain bias, toxic content, privacy risk, and disinformation. To mitigate those threats, we identify and define four key properties, namely integrity, source identification, attack detectability, and utility preservation, that need to be satisfied by a safe LLM-integrated application. Based on these properties, we develop a lightweight, threat-agnostic defense that mitigates both insider and outsider threats.

Title: Federated Learning with Diffusion Models for Privacy-Sensitive Vision Tasks. (arXiv:2311.16538v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.16538
• Code URL: null
• Copy Paste: [[2311.16538]] Federated Learning with Diffusion Models for Privacy-Sensitive Vision Tasks(http://arxiv.org/abs/2311.16538)
• Summary:

  Diffusion models have shown great potential for vision-related tasks, particularly for image generation. However, their training is typically conducted in a centralized manner, relying on data collected from publicly available sources. This approach may not be feasible or practical in many domains, such as the medical field, which involves privacy concerns over data collection. Despite the challenges associated with privacy-sensitive data, such domains could still benefit from valuable vision services provided by diffusion models. Federated learning (FL) plays a crucial role in enabling decentralized model training without compromising data privacy. Instead of collecting data, an FL system gathers model parameters, effectively safeguarding the private data of different parties involved. This makes FL systems vital for managing decentralized learning tasks, especially in scenarios where privacy-sensitive data is distributed across a network of clients. Nonetheless, FL presents its own set of challenges due to its distributed nature and privacy-preserving properties. Therefore, in this study, we explore the FL strategy to train diffusion models, paving the way for the development of federated diffusion models. We conduct experiments on various FL scenarios, and our findings demonstrate that federated diffusion models have great potential to deliver vision services to privacy-sensitive domains.

protect

Title: Stepping out of Flatland: Discovering Behavior Patterns as Topological Structures in Cyber Hypergraphs. (arXiv:2311.16154v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.16154
• Code URL: null
• Copy Paste: [[2311.16154]] Stepping out of Flatland: Discovering Behavior Patterns as Topological Structures in Cyber Hypergraphs(http://arxiv.org/abs/2311.16154)
• Summary:

  Data breaches and ransomware attacks occur so often that they have become part of our daily news cycle. This is due to a myriad of factors, including the increasing number of internet-of-things devices, shift to remote work during the pandemic, and advancement in adversarial techniques, which all contribute to the increase in both the complexity of data captured and the challenge of protecting our networks. At the same time, cyber research has made strides, leveraging advances in machine learning and natural language processing to focus on identifying sophisticated attacks that are known to evade conventional measures. While successful, the shortcomings of these methods, particularly the lack of interpretability, are inherent and difficult to overcome. Consequently, there is an ever-increasing need to develop new tools for analyzing cyber data to enable more effective attack detection. In this paper, we present a novel framework based in the theory of hypergraphs and topology to understand data from cyber networks through topological signatures, which are both flexible and can be traced back to the log data. While our approach's mathematical grounding requires some technical development, this pays off in interpretability, which we will demonstrate with concrete examples in a large-scale cyber network dataset. These examples are an introduction to the broader possibilities that lie ahead; our goal is to demonstrate the value of applying methods from the burgeoning fields of hypernetwork science and applied topology to understand relationships among behaviors in cyber data.

Title: Darknet Traffic Analysis A Systematic Literature Review. (arXiv:2311.16276v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.16276
• Code URL: null
• Copy Paste: [[2311.16276]] Darknet Traffic Analysis A Systematic Literature Review(http://arxiv.org/abs/2311.16276)
• Summary:

  The primary objective of an anonymity tool is to protect the anonymity of its users through the implementation of strong encryption and obfuscation techniques. As a result, it becomes very difficult to monitor and identify users activities on these networks. Moreover, such systems have strong defensive mechanisms to protect users against potential risks, including the extraction of traffic characteristics and website fingerprinting. However, the strong anonymity feature also functions as a refuge for those involved in illicit activities who aim to avoid being traced on the network. As a result, a substantial body of research has been undertaken to examine and classify encrypted traffic using machine learning techniques. This paper presents a comprehensive examination of the existing approaches utilized for the categorization of anonymous traffic as well as encrypted network traffic inside the darknet. Also, this paper presents a comprehensive analysis of methods of darknet traffic using machine learning techniques to monitor and identify the traffic attacks inside the darknet.

Title: Root causes, ongoing difficulties, proactive prevention techniques, and emerging trends of enterprise data breaches. (arXiv:2311.16303v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.16303
• Code URL: null
• Copy Paste: [[2311.16303]] Root causes, ongoing difficulties, proactive prevention techniques, and emerging trends of enterprise data breaches(http://arxiv.org/abs/2311.16303)
• Summary:

  A data breach in the modern digital era is the unintentional or intentional disclosure of private data to uninvited parties. Businesses now consider data to be a crucial asset, and any breach of this data can have dire repercussions, including harming a company's brand and resulting in losses. Enterprises now place a high premium on detecting and preventing data loss due to the growing amount of data and the increasing frequency of data breaches. Even with a great deal of research, protecting sensitive data is still a difficult task. This review attempts to highlight interesting prospects and offer insightful information to those who are interested in learning about the risks that businesses face from data leaks, current occurrences, state-of-the-art methods for detection and prevention, new difficulties, and possible solutions.

defense

Title: Efficient Key-Based Adversarial Defense for ImageNet by Using Pre-trained Model. (arXiv:2311.16577v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16577
• Code URL: null
• Copy Paste: [[2311.16577]] Efficient Key-Based Adversarial Defense for ImageNet by Using Pre-trained Model(http://arxiv.org/abs/2311.16577)
• Summary:

  In this paper, we propose key-based defense model proliferation by leveraging pre-trained models and utilizing recent efficient fine-tuning techniques on ImageNet-1k classification. First, we stress that deploying key-based models on edge devices is feasible with the latest model deployment advancements, such as Apple CoreML, although the mainstream enterprise edge artificial intelligence (Edge AI) has been focused on the Cloud. Then, we point out that the previous key-based defense on on-device image classification is impractical for two reasons: (1) training many classifiers from scratch is not feasible, and (2) key-based defenses still need to be thoroughly tested on large datasets like ImageNet. To this end, we propose to leverage pre-trained models and utilize efficient fine-tuning techniques to proliferate key-based models even on limited computing resources. Experiments were carried out on the ImageNet-1k dataset using adaptive and non-adaptive attacks. The results show that our proposed fine-tuned key-based models achieve a superior classification accuracy (more than 10% increase) compared to the previous key-based models on classifying clean and adversarial examples.

attack

Title: BadCLIP: Trigger-Aware Prompt Learning for Backdoor Attacks on CLIP. (arXiv:2311.16194v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16194
• Code URL: null
• Copy Paste: [[2311.16194]] BadCLIP: Trigger-Aware Prompt Learning for Backdoor Attacks on CLIP(http://arxiv.org/abs/2311.16194)
• Summary:

  Contrastive Vision-Language Pre-training, known as CLIP, has shown promising effectiveness in addressing downstream image recognition tasks. However, recent works revealed that the CLIP model can be implanted with a downstream-oriented backdoor. On downstream tasks, one victim model performs well on clean samples but predicts a specific target class whenever a specific trigger is present. For injecting a backdoor, existing attacks depend on a large amount of additional data to maliciously fine-tune the entire pre-trained CLIP model, which makes them inapplicable to data-limited scenarios. In this work, motivated by the recent success of learnable prompts, we address this problem by injecting a backdoor into the CLIP model in the prompt learning stage. Our method named BadCLIP is built on a novel and effective mechanism in backdoor attacks on CLIP, i.e., influencing both the image and text encoders with the trigger. It consists of a learnable trigger applied to images and a trigger-aware context generator, such that the trigger can change text features via trigger-aware prompts, resulting in a powerful and generalizable attack. Extensive experiments conducted on 11 datasets verify that the clean accuracy of BadCLIP is similar to those of advanced prompt learning methods and the attack success rate is higher than 99% in most cases. BadCLIP is also generalizable to unseen classes, and shows a strong generalization capability under cross-dataset and cross-domain settings.

Title: RetouchUAA: Unconstrained Adversarial Attack via Image Retouching. (arXiv:2311.16478v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16478
• Code URL: null
• Copy Paste: [[2311.16478]] RetouchUAA: Unconstrained Adversarial Attack via Image Retouching(http://arxiv.org/abs/2311.16478)
• Summary:

  Deep Neural Networks (DNNs) are susceptible to adversarial examples. Conventional attacks generate controlled noise-like perturbations that fail to reflect real-world scenarios and hard to interpretable. In contrast, recent unconstrained attacks mimic natural image transformations occurring in the real world for perceptible but inconspicuous attacks, yet compromise realism due to neglect of image post-processing and uncontrolled attack direction. In this paper, we propose RetouchUAA, an unconstrained attack that exploits a real-life perturbation: image retouching styles, highlighting its potential threat to DNNs. Compared to existing attacks, RetouchUAA offers several notable advantages. Firstly, RetouchUAA excels in generating interpretable and realistic perturbations through two key designs: the image retouching attack framework and the retouching style guidance module. The former custom-designed human-interpretability retouching framework for adversarial attack by linearizing images while modelling the local processing and retouching decision-making in human retouching behaviour, provides an explicit and reasonable pipeline for understanding the robustness of DNNs against retouching. The latter guides the adversarial image towards standard retouching styles, thereby ensuring its realism. Secondly, attributed to the design of the retouching decision regularization and the persistent attack strategy, RetouchUAA also exhibits outstanding attack capability and defense robustness, posing a heavy threat to DNNs. Experiments on ImageNet and Place365 reveal that RetouchUAA achieves nearly 100\% white-box attack success against three DNNs, while achieving a better trade-off between image naturalness, transferability and defense robustness than baseline attacks.

Title: BAGEL: Backdoor Attacks against Federated Contrastive Learning. (arXiv:2311.16113v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.16113
• Code URL: null
• Copy Paste: [[2311.16113]] BAGEL: Backdoor Attacks against Federated Contrastive Learning(http://arxiv.org/abs/2311.16113)
• Summary:

  Federated Contrastive Learning (FCL) is an emerging privacy-preserving paradigm in distributed learning for unlabeled data. In FCL, distributed parties collaboratively learn a global encoder with unlabeled data, and the global encoder could be widely used as a feature extractor to build models for many downstream tasks. However, FCL is also vulnerable to many security threats (e.g., backdoor attacks) due to its distributed nature, which are seldom investigated in existing solutions. In this paper, we study the backdoor attack against FCL as a pioneer research, to illustrate how backdoor attacks on distributed local clients act on downstream tasks. Specifically, in our system, malicious clients can successfully inject a backdoor into the global encoder by uploading poisoned local updates, thus downstream models built with this global encoder will also inherit the backdoor. We also investigate how to inject backdoors into multiple downstream models, in terms of two different backdoor attacks, namely the \textit{centralized attack} and the \textit{decentralized attack}. Experiment results show that both the centralized and the decentralized attacks can inject backdoors into downstream models effectively with high attack success rates. Finally, we evaluate two defense methods against our proposed backdoor attacks in FCL, which indicates that the decentralized backdoor attack is more stealthy and harder to defend.

Title: Imperceptible CMOS camera dazzle for adversarial attacks on deep neural networks. (arXiv:2311.16118v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.16118
• Code URL: null
• Copy Paste: [[2311.16118]] Imperceptible CMOS camera dazzle for adversarial attacks on deep neural networks(http://arxiv.org/abs/2311.16118)
• Summary:

  Despite the outstanding performance of deep neural networks, they are vulnerable to adversarial attacks. While there are many invisible attacks in the digital domain, most physical world adversarial attacks are visible. Here we present an invisible optical adversarial attack that uses a light source to dazzle a CMOS camera with a rolling shutter. We present the photopic conditions required to keep the attacking light source completely invisible while sufficiently jamming the captured image so that a deep neural network applied to it is deceived.

Title: DiffAttack: Evasion Attacks Against Diffusion-Based Adversarial Purification. (arXiv:2311.16124v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.16124
• Code URL: null
• Copy Paste: [[2311.16124]] DiffAttack: Evasion Attacks Against Diffusion-Based Adversarial Purification(http://arxiv.org/abs/2311.16124)
• Summary:

  Diffusion-based purification defenses leverage diffusion models to remove crafted perturbations of adversarial examples and achieve state-of-the-art robustness. Recent studies show that even advanced attacks cannot break such defenses effectively, since the purification process induces an extremely deep computational graph which poses the potential problem of gradient obfuscation, high memory cost, and unbounded randomness. In this paper, we propose a unified framework DiffAttack to perform effective and efficient attacks against diffusion-based purification defenses, including both DDPM and score-based approaches. In particular, we propose a deviated-reconstruction loss at intermediate diffusion steps to induce inaccurate density gradient estimation to tackle the problem of vanishing/exploding gradients. We also provide a segment-wise forwarding-backwarding algorithm, which leads to memory-efficient gradient backpropagation. We validate the attack effectiveness of DiffAttack compared with existing adaptive attacks on CIFAR-10 and ImageNet. We show that DiffAttack decreases the robust accuracy of models compared with SOTA attacks by over 20% on CIFAR-10 under $\ell_\infty$ attack $(\epsilon=8/255)$, and over 10% on ImageNet under $\ell_\infty$ attack $(\epsilon=4/255)$. We conduct a series of ablations studies, and we find 1) DiffAttack with the deviated-reconstruction loss added over uniformly sampled time steps is more effective than that added over only initial/final steps, and 2) diffusion-based purification with a moderate diffusion length is more robust under DiffAttack.

Title: GNNBleed: Inference Attacks to Unveil Private Edges in Graphs with Realistic Access to GNN Models. (arXiv:2311.16139v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.16139
• Code URL: null
• Copy Paste: [[2311.16139]] GNNBleed: Inference Attacks to Unveil Private Edges in Graphs with Realistic Access to GNN Models(http://arxiv.org/abs/2311.16139)
• Summary:

  Graph Neural Networks (GNNs) have increasingly become an indispensable tool in learning from graph-structured data, catering to various applications including social network analysis, recommendation systems, etc. At the heart of these networks are the edges which are crucial in guiding GNN models' predictions. In many scenarios, these edges represent sensitive information, such as personal associations or financial dealings -- thus requiring privacy assurance. However, their contributions to GNN model predictions may in turn be exploited by the adversary to compromise their privacy. Motivated by these conflicting requirements, this paper investigates edge privacy in contexts where adversaries possess black-box GNN model access, restricted further by access controls, preventing direct insights into arbitrary node outputs. In this context, we introduce a series of privacy attacks grounded on the message-passing mechanism of GNNs. These strategies allow adversaries to deduce connections between two nodes not by directly analyzing the model's output for these pairs but by analyzing the output for nodes linked to them. Our evaluation with seven real-life datasets and four GNN architectures underlines a significant vulnerability: even in systems fortified with access control mechanisms, an adaptive adversary can decipher private connections between nodes, thereby revealing potentially sensitive relationships and compromising the confidentiality of the graph.

Title: Rethinking Backdoor Attacks on Dataset Distillation: A Kernel Method Perspective. (arXiv:2311.16646v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.16646
• Code URL: null
• Copy Paste: [[2311.16646]] Rethinking Backdoor Attacks on Dataset Distillation: A Kernel Method Perspective(http://arxiv.org/abs/2311.16646)
• Summary:

  Dataset distillation offers a potential means to enhance data efficiency in deep learning. Recent studies have shown its ability to counteract backdoor risks present in original training samples. In this study, we delve into the theoretical aspects of backdoor attacks and dataset distillation based on kernel methods. We introduce two new theory-driven trigger pattern generation methods specialized for dataset distillation. Following a comprehensive set of analyses and experiments, we show that our optimization-based trigger design framework informs effective backdoor attacks on dataset distillation. Notably, datasets poisoned by our designed trigger prove resilient against conventional backdoor attack detection and mitigation methods. Our empirical results validate that the triggers developed using our approaches are proficient at executing resilient backdoor attacks.

robust

Title: Learning Noise-Robust Joint Representation for Multimodal Emotion Recognition under Realistic Incomplete Data Scenarios. (arXiv:2311.16114v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16114
• Code URL: null
• Copy Paste: [[2311.16114]] Learning Noise-Robust Joint Representation for Multimodal Emotion Recognition under Realistic Incomplete Data Scenarios(http://arxiv.org/abs/2311.16114)
• Summary:

  Multimodal emotion recognition (MER) in practical scenarios presents a significant challenge due to the presence of incomplete data, such as missing or noisy data. Traditional methods often discard missing data or replace it with a zero vector, neglecting the availability issue of noisy data. Consequently, these approaches are not fully applicable to realistic scenarios, where both missing and noisy data are prevalent. To address this problem, we propose a novel noise-robust MER model, named NMER, which effectively learns robust multimodal joint representations from incomplete data containing noise. Our approach incorporates two key components. First, we introduce a noise scheduler that adjusts the type and level of noise in the training data, emulating the characteristics of incomplete data in realistic scenarios. Second, we employ a Variational AutoEncoder (VAE)-based NMER model to generate robust multimodal joint representations from the noisy data, leveraging the modality invariant feature. The experimental results on the benchmark dataset IEMOCAP indicate the proposed NMER outperforms state-of-the-art MER systems. The ablation results also confirm the effectiveness of the VAE structure. We release our code at \href{https://github.com/WooyoohL/Noise-robust_MER.

Title: Next-gen traffic surveillance: AI-assisted mobile traffic violation detection system. (arXiv:2311.16179v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16179
• Code URL: null
• Copy Paste: [[2311.16179]] Next-gen traffic surveillance: AI-assisted mobile traffic violation detection system(http://arxiv.org/abs/2311.16179)
• Summary:

  Road traffic accidents pose a significant global public health concern, leading to injuries, fatalities, and vehicle damage. Approximately 1,3 million people lose their lives daily due to traffic accidents [World Health Organization, 2022]. Addressing this issue requires accurate traffic law violation detection systems to ensure adherence to regulations. The integration of Artificial Intelligence algorithms, leveraging machine learning and computer vision, has facilitated the development of precise traffic rule enforcement. This paper illustrates how computer vision and machine learning enable the creation of robust algorithms for detecting various traffic violations. Our model, capable of identifying six common traffic infractions, detects red light violations, illegal use of breakdown lanes, violations of vehicle following distance, breaches of marked crosswalk laws, illegal parking, and parking on marked crosswalks. Utilizing online traffic footage and a self-mounted on-dash camera, we apply the YOLOv5 algorithm's detection module to identify traffic agents such as cars, pedestrians, and traffic signs, and the strongSORT algorithm for continuous interframe tracking. Six discrete algorithms analyze agents' behavior and trajectory to detect violations. Subsequently, an Identification Module extracts vehicle ID information, such as the license plate, to generate violation notices sent to relevant authorities.

Title: Robust Self-calibration of Focal Lengths from the Fundamental Matrix. (arXiv:2311.16304v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16304
• Code URL: null
• Copy Paste: [[2311.16304]] Robust Self-calibration of Focal Lengths from the Fundamental Matrix(http://arxiv.org/abs/2311.16304)
• Summary:

  The problem of self-calibration of two cameras from a given fundamental matrix is one of the basic problems in geometric computer vision. Under the assumption of known principal points and square pixels, the well-known Bougnoux formula offers a means to compute the two unknown focal lengths. However, in many practical situations, the formula yields inaccurate results due to commonly occurring singularities. Moreover, the estimates are sensitive to noise in the computed fundamental matrix and to the assumed positions of the principal points. In this paper, we therefore propose an efficient and robust iterative method to estimate the focal lengths along with the principal points of the cameras given a fundamental matrix and priors for the estimated camera parameters. In addition, we study a computationally efficient check of models generated within RANSAC that improves the accuracy of the estimated models while reducing the total computational time. Extensive experiments on real and synthetic data show that our iterative method brings significant improvements in terms of the accuracy of the estimated focal lengths over the Bougnoux formula and other state-of-the-art methods, even when relying on inaccurate priors.

Title: CLAP: Contrastive Learning with Augmented Prompts for Robustness on Pretrained Vision-Language Models. (arXiv:2311.16445v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16445
• Code URL: null
• Copy Paste: [[2311.16445]] CLAP: Contrastive Learning with Augmented Prompts for Robustness on Pretrained Vision-Language Models(http://arxiv.org/abs/2311.16445)
• Summary:

  Contrastive vision-language models, e.g., CLIP, have garnered substantial attention for their exceptional generalization capabilities. However, their robustness to perturbations has ignited concerns. Existing strategies typically reinforce their resilience against adversarial examples by enabling the image encoder to "see" these perturbed examples, often necessitating a complete retraining of the image encoder on both natural and adversarial samples. In this study, we propose a new method to enhance robustness solely through text augmentation, eliminating the need for retraining the image encoder on adversarial examples. Our motivation arises from the realization that text and image data inherently occupy a shared latent space, comprising latent content variables and style variables. This insight suggests the feasibility of learning to disentangle these latent content variables using text data exclusively. To accomplish this, we introduce an effective text augmentation method that focuses on modifying the style while preserving the content in the text data. By changing the style part of the text data, we empower the text encoder to emphasize latent content variables, ultimately enhancing the robustness of vision-language models. Our experiments across various datasets demonstrate substantial improvements in the robustness of the pre-trained CLIP model.

Title: A Unified Framework for Multimodal, Multi-Part Human Motion Synthesis. (arXiv:2311.16471v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16471
• Code URL: null
• Copy Paste: [[2311.16471]] A Unified Framework for Multimodal, Multi-Part Human Motion Synthesis(http://arxiv.org/abs/2311.16471)
• Summary:

  The field has made significant progress in synthesizing realistic human motion driven by various modalities. Yet, the need for different methods to animate various body parts according to different control signals limits the scalability of these techniques in practical scenarios. In this paper, we introduce a cohesive and scalable approach that consolidates multimodal (text, music, speech) and multi-part (hand, torso) human motion generation. Our methodology unfolds in several steps: We begin by quantizing the motions of diverse body parts into separate codebooks tailored to their respective domains. Next, we harness the robust capabilities of pre-trained models to transcode multimodal signals into a shared latent space. We then translate these signals into discrete motion tokens by iteratively predicting subsequent tokens to form a complete sequence. Finally, we reconstruct the continuous actual motion from this tokenized sequence. Our method frames the multimodal motion generation challenge as a token prediction task, drawing from specialized codebooks based on the modality of the control signal. This approach is inherently scalable, allowing for the easy integration of new modalities. Extensive experiments demonstrated the effectiveness of our design, emphasizing its potential for broad application.

Title: Elucidating and Overcoming the Challenges of Label Noise in Supervised Contrastive Learning. (arXiv:2311.16481v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16481
• Code URL: null
• Copy Paste: [[2311.16481]] Elucidating and Overcoming the Challenges of Label Noise in Supervised Contrastive Learning(http://arxiv.org/abs/2311.16481)
• Summary:

  Image classification datasets exhibit a non-negligible fraction of mislabeled examples, often due to human error when one class superficially resembles another. This issue poses challenges in supervised contrastive learning (SCL), where the goal is to cluster together data points of the same class in the embedding space while distancing those of disparate classes. While such methods outperform those based on cross-entropy, they are not immune to labeling errors. However, while the detrimental effects of noisy labels in supervised learning are well-researched, their influence on SCL remains largely unexplored. Hence, we analyse the effect of label errors and examine how they disrupt the SCL algorithm's ability to distinguish between positive and negative sample pairs. Our analysis reveals that human labeling errors manifest as easy positive samples in around 99% of cases. We, therefore, propose D-SCL, a novel Debiased Supervised Contrastive Learning objective designed to mitigate the bias introduced by labeling errors. We demonstrate that D-SCL consistently outperforms state-of-the-art techniques for representation learning across diverse vision benchmarks, offering improved robustness to label errors.

Title: GaitContour: Efficient Gait Recognition based on a Contour-Pose Representation. (arXiv:2311.16497v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16497
• Code URL: null
• Copy Paste: [[2311.16497]] GaitContour: Efficient Gait Recognition based on a Contour-Pose Representation(http://arxiv.org/abs/2311.16497)
• Summary:

  Gait recognition holds the promise to robustly identify subjects based on walking patterns instead of appearance information. In recent years, this field has been dominated by learning methods based on two principal input representations: dense silhouette masks or sparse pose keypoints. In this work, we propose a novel, point-based Contour-Pose representation, which compactly expresses both body shape and body parts information. We further propose a local-to-global architecture, called GaitContour, to leverage this novel representation and efficiently compute subject embedding in two stages. The first stage consists of a local transformer that extracts features from five different body regions. The second stage then aggregates the regional features to estimate a global human gait representation. Such a design significantly reduces the complexity of the attention operation and improves efficiency and performance simultaneously. Through large scale experiments, GaitContour is shown to perform significantly better than previous point-based methods, while also being significantly more efficient than silhouette-based methods. On challenging datasets with significant distractors, GaitContour can even outperform silhouette-based methods.

Title: Agents meet OKR: An Object and Key Results Driven Agent System with Hierarchical Self-Collaboration and Self-Evaluation. (arXiv:2311.16542v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16542
• Code URL: null
• Copy Paste: [[2311.16542]] Agents meet OKR: An Object and Key Results Driven Agent System with Hierarchical Self-Collaboration and Self-Evaluation(http://arxiv.org/abs/2311.16542)
• Summary:

  In this study, we introduce the concept of OKR-Agent designed to enhance the capabilities of Large Language Models (LLMs) in task-solving. Our approach utilizes both self-collaboration and self-correction mechanism, facilitated by hierarchical agents, to address the inherent complexities in task-solving. Our key observations are two-fold: first, effective task-solving demands in-depth domain knowledge and intricate reasoning, for which deploying specialized agents for individual sub-tasks can markedly enhance LLM performance. Second, task-solving intrinsically adheres to a hierarchical execution structure, comprising both high-level strategic planning and detailed task execution. Towards this end, our OKR-Agent paradigm aligns closely with this hierarchical structure, promising enhanced efficacy and adaptability across a range of scenarios. Specifically, our framework includes two novel modules: hierarchical Objects and Key Results generation and multi-level evaluation, each contributing to more efficient and robust task-solving. In practical, hierarchical OKR generation decomposes Objects into multiple sub-Objects and assigns new agents based on key results and agent responsibilities. These agents subsequently elaborate on their designated tasks and may further decompose them as necessary. Such generation operates recursively and hierarchically, culminating in a comprehensive set of detailed solutions. The multi-level evaluation module of OKR-Agent refines solution by leveraging feedback from all associated agents, optimizing each step of the process. This ensures solution is accurate, practical, and effectively address intricate task requirements, enhancing the overall reliability and quality of the outcome. Experimental results also show our method outperforms the previous methods on several tasks. Code and demo are available at https://okr-agent.github.io/

Title: Multi-Irreducible Spectral Synchronization for Robust Rotation Averaging. (arXiv:2311.16544v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16544
• Code URL: null
• Copy Paste: [[2311.16544]] Multi-Irreducible Spectral Synchronization for Robust Rotation Averaging(http://arxiv.org/abs/2311.16544)
• Summary:

  Rotation averaging (RA) is a fundamental problem in robotics and computer vision. In RA, the goal is to estimate a set of $N$ unknown orientations $R_{1}, ..., R_{N} \in SO(3)$, given noisy measurements $R_{ij} \sim R^{-1}_{i} R_{j}$ of a subset of their pairwise relative rotations. This problem is both nonconvex and NP-hard, and thus difficult to solve in the general case. We apply harmonic analysis on compact groups to derive a (convex) spectral relaxation constructed from truncated Fourier decompositions of the individual summands appearing in the RA objective; we then recover an estimate of the RA solution by computing a few extremal eigenpairs of this relaxation, and (approximately) solving a consensus problem. Our approach affords several notable advantages versus prior RA methods: it can be used in conjunction with \emph{any} smooth loss function (including, but not limited to, robust M-estimators), does not require any initialization, and is implemented using only simple (and highly scalable) linear-algebraic computations and parallelizable optimizations over band-limited functions of individual rotational states. Moreover, under the (physically well-motivated) assumption of multiplicative Langevin measurement noise, we derive explicit performance guarantees for our spectral estimator (in the form of probabilistic tail bounds on the estimation error) that are parameterized in terms of graph-theoretic quantities of the underlying measurement network. By concretely linking estimator performance with properties of the underlying measurement graph, our results also indicate how to devise measurement networks that are \emph{guaranteed} to achieve accurate estimation, enabling such downstream tasks as sensor placement, network compression, and active sensing.

Title: Improving Lane Detection Generalization: A Novel Framework using HD Maps for Boosting Diversity. (arXiv:2311.16589v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16589
• Code URL: null
• Copy Paste: [[2311.16589]] Improving Lane Detection Generalization: A Novel Framework using HD Maps for Boosting Diversity(http://arxiv.org/abs/2311.16589)
• Summary:

  Lane detection is a vital task for vehicles to navigate and localize their position on the road. To ensure reliable results, lane detection algorithms must have robust generalization performance in various road environments. However, despite the significant performance improvement of deep learning-based lane detection algorithms, their generalization performance in response to changes in road environments still falls short of expectations. In this paper, we present a novel framework for single-source domain generalization (SSDG) in lane detection. By decomposing data into lane structures and surroundings, we enhance diversity using High-Definition (HD) maps and generative models. Rather than expanding data volume, we strategically select a core subset of data, maximizing diversity and optimizing performance. Our extensive experiments demonstrate that our framework enhances the generalization performance of lane detection, comparable to the domain adaptation-based method.

Title: Augmenting x-ray single particle imaging reconstruction with self-supervised machine learning. (arXiv:2311.16652v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16652
• Code URL: null
• Copy Paste: [[2311.16652]] Augmenting x-ray single particle imaging reconstruction with self-supervised machine learning(http://arxiv.org/abs/2311.16652)
• Summary:

  The development of X-ray Free Electron Lasers (XFELs) has opened numerous opportunities to probe atomic structure and ultrafast dynamics of various materials. Single Particle Imaging (SPI) with XFELs enables the investigation of biological particles in their natural physiological states with unparalleled temporal resolution, while circumventing the need for cryogenic conditions or crystallization. However, reconstructing real-space structures from reciprocal-space x-ray diffraction data is highly challenging due to the absence of phase and orientation information, which is further complicated by weak scattering signals and considerable fluctuations in the number of photons per pulse. In this work, we present an end-to-end, self-supervised machine learning approach to recover particle orientations and estimate reciprocal space intensities from diffraction images only. Our method demonstrates great robustness under demanding experimental conditions with significantly enhanced reconstruction capabilities compared with conventional algorithms, and signifies a paradigm shift in SPI as currently practiced at XFELs.

Title: Rescuing referral failures during automated diagnosis of domain-shifted medical images. (arXiv:2311.16766v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16766
• Code URL: null
• Copy Paste: [[2311.16766]] Rescuing referral failures during automated diagnosis of domain-shifted medical images(http://arxiv.org/abs/2311.16766)
• Summary:

  The success of deep learning models deployed in the real world depends critically on their ability to generalize well across diverse data domains. Here, we address a fundamental challenge with selective classification during automated diagnosis with domain-shifted medical images. In this scenario, models must learn to avoid making predictions when label confidence is low, especially when tested with samples far removed from the training set (covariate shift). Such uncertain cases are typically referred to the clinician for further analysis and evaluation. Yet, we show that even state-of-the-art domain generalization approaches fail severely during referral when tested on medical images acquired from a different demographic or using a different technology. We examine two benchmark diagnostic medical imaging datasets exhibiting strong covariate shifts: i) diabetic retinopathy prediction with retinal fundus images and ii) multilabel disease prediction with chest X-ray images. We show that predictive uncertainty estimates do not generalize well under covariate shifts leading to non-monotonic referral curves, and severe drops in performance (up to 50%) at high referral rates (>70%). We evaluate novel combinations of robust generalization and post hoc referral approaches, that rescue these failures and achieve significant performance improvements, typically >10%, over baseline methods. Our study identifies a critical challenge with referral in domain-shifted medical images and finds key applications in reliable, automated disease diagnosis.

Title: Releasing the CRaQAn (Coreference Resolution in Question-Answering): An open-source dataset and dataset creation methodology using instruction-following models. (arXiv:2311.16338v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.16338
• Code URL: null
• Copy Paste: [[2311.16338]] Releasing the CRaQAn (Coreference Resolution in Question-Answering): An open-source dataset and dataset creation methodology using instruction-following models(http://arxiv.org/abs/2311.16338)
• Summary:

  Instruction-following language models demand robust methodologies for information retrieval to augment instructions for question-answering applications. A primary challenge is the resolution of coreferences in the context of chunking strategies for long documents. The critical barrier to experimentation of handling coreferences is a lack of open source datasets, specifically in question-answering tasks that require coreference resolution. In this work we present our Coreference Resolution in Question-Answering (CRaQAn) dataset, an open-source dataset that caters to the nuanced information retrieval requirements of coreference resolution in question-answering tasks by providing over 250 question-answer pairs containing coreferences. To develop this dataset, we developed a novel approach for creating high-quality datasets using an instruction-following model (GPT-4) and a Recursive Criticism and Improvement Loop.

Title: MACE: A Multi-pattern Accommodated and Efficient Anomaly Detection Method in the Frequency Domain. (arXiv:2311.16191v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.16191
• Code URL: null
• Copy Paste: [[2311.16191]] MACE: A Multi-pattern Accommodated and Efficient Anomaly Detection Method in the Frequency Domain(http://arxiv.org/abs/2311.16191)
• Summary:

  Anomaly detection significantly enhances the robustness of cloud systems. While neural network-based methods have recently demonstrated strong advantages, they encounter practical challenges in cloud environments: the contradiction between the impracticality of maintaining a unique model for each service and the limited ability of dealing with diverse normal patterns by a unified model, as well as issues with handling heavy traffic in real time and short-term anomaly detection sensitivity. Thus, we propose MACE, a Multi-pattern Accommodated and efficient Anomaly detection method in the frequency domain for time series anomaly detection. There are three novel characteristics of it: (i) a pattern extraction mechanism excelling at handling diverse normal patterns, which enables the model to identify anomalies by examining the correlation between the data sample and its service normal pattern, instead of solely focusing on the data sample itself; (ii) a dualistic convolution mechanism that amplifies short-term anomalies in the time domain and hinders the reconstruction of anomalies in the frequency domain, which enlarges the reconstruction error disparity between anomaly and normality and facilitates anomaly detection; (iii) leveraging the sparsity and parallelism of frequency domain to enhance model efficiency. We theoretically and experimentally prove that using a strategically selected subset of Fourier bases can not only reduce computational overhead but is also profit to distinguish anomalies, compared to using the complete spectrum. Moreover, extensive experiments demonstrate MACE's effectiveness in handling diverse normal patterns with a unified model and it achieves state-of-the-art performance with high efficiency. \end{abstract}

Title: Making Self-supervised Learning Robust to Spurious Correlation via Learning-speed Aware Sampling. (arXiv:2311.16361v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.16361
• Code URL: null
• Copy Paste: [[2311.16361]] Making Self-supervised Learning Robust to Spurious Correlation via Learning-speed Aware Sampling(http://arxiv.org/abs/2311.16361)
• Summary:

  Self-supervised learning (SSL) has emerged as a powerful technique for learning rich representations from unlabeled data. The data representations are able to capture many underlying attributes of data, and be useful in downstream prediction tasks. In real-world settings, spurious correlations between some attributes (e.g. race, gender and age) and labels for downstream tasks often exist, e.g. cancer is usually more prevalent among elderly patients. In this paper, we investigate SSL in the presence of spurious correlations and show that the SSL training loss can be minimized by capturing only a subset of the conspicuous features relevant to those sensitive attributes, despite the presence of other important predictive features for the downstream tasks. To address this issue, we investigate the learning dynamics of SSL and observe that the learning is slower for samples that conflict with such correlations (e.g. elder patients without cancer). Motivated by these findings, we propose a learning-speed aware SSL (LA-SSL) approach, in which we sample each training data with a probability that is inversely related to its learning speed. We evaluate LA-SSL on three datasets that exhibit spurious correlations between different attributes, demonstrating that it improves the robustness of pretrained representations on downstream classification tasks.

Title: Physics-Informed Neural Network for Discovering Systems with Unmeasurable States with Application to Lithium-Ion Batteries. (arXiv:2311.16374v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.16374
• Code URL: null
• Copy Paste: [[2311.16374]] Physics-Informed Neural Network for Discovering Systems with Unmeasurable States with Application to Lithium-Ion Batteries(http://arxiv.org/abs/2311.16374)
• Summary:

  Combining machine learning with physics is a trending approach for discovering unknown dynamics, and one of the most intensively studied frameworks is the physics-informed neural network (PINN). However, PINN often fails to optimize the network due to its difficulty in concurrently minimizing multiple losses originating from the system's governing equations. This problem can be more serious when the system's states are unmeasurable, like lithium-ion batteries (LiBs). In this work, we introduce a robust method for training PINN that uses fewer loss terms and thus constructs a less complex landscape for optimization. In particular, instead of having loss terms from each differential equation, this method embeds the dynamics into a loss function that quantifies the error between observed and predicted system outputs. This is accomplished by numerically integrating the predicted states from the neural network(NN) using known dynamics and transforming them to obtain a sequence of predicted outputs. Minimizing such a loss optimizes the NN to predict states consistent with observations given the physics. Further, the system's parameters can be added to the optimization targets. To demonstrate the ability of this method to perform various modeling and control tasks, we apply it to a battery model to concurrently estimate its states and parameters.

Title: On the Robustness of Decision-Focused Learning. (arXiv:2311.16487v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.16487
• Code URL: null
• Copy Paste: [[2311.16487]] On the Robustness of Decision-Focused Learning(http://arxiv.org/abs/2311.16487)
• Summary:

  Decision-Focused Learning (DFL) is an emerging learning paradigm that tackles the task of training a machine learning (ML) model to predict missing parameters of an incomplete optimization problem, where the missing parameters are predicted. DFL trains an ML model in an end-to-end system, by integrating the prediction and optimization tasks, providing better alignment of the training and testing objectives. DFL has shown a lot of promise and holds the capacity to revolutionize decision-making in many real-world applications. However, very little is known about the performance of these models under adversarial attacks. We adopt ten unique DFL methods and benchmark their performance under two distinctly focused attacks adapted towards the Predict-then-Optimize problem setting. Our study proposes the hypothesis that the robustness of a model is highly correlated with its ability to find predictions that lead to optimal decisions without deviating from the ground-truth label. Furthermore, we provide insight into how to target the models that violate this condition and show how these models respond differently depending on the achieved optimality at the end of their training cycles.

Title: B-LSTM-MIONet: Bayesian LSTM-based Neural Operators for Learning the Response of Complex Dynamical Systems to Length-Variant Multiple Input Functions. (arXiv:2311.16519v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.16519
• Code URL: null
• Copy Paste: [[2311.16519]] B-LSTM-MIONet: Bayesian LSTM-based Neural Operators for Learning the Response of Complex Dynamical Systems to Length-Variant Multiple Input Functions(http://arxiv.org/abs/2311.16519)
• Summary:

  Deep Operator Network (DeepONet) is a neural network framework for learning nonlinear operators such as those from ordinary differential equations (ODEs) describing complex systems. Multiple-input deep neural operators (MIONet) extended DeepONet to allow multiple input functions in different Banach spaces. MIONet offers flexibility in training dataset grid spacing, without constraints on output location. However, it requires offline inputs and cannot handle varying sequence lengths in testing datasets, limiting its real-time application in dynamic complex systems. This work redesigns MIONet, integrating Long Short Term Memory (LSTM) to learn neural operators from time-dependent data. This approach overcomes data discretization constraints and harnesses LSTM's capability with variable-length, real-time data. Factors affecting learning performance, like algorithm extrapolation ability are presented. The framework is enhanced with uncertainty quantification through a novel Bayesian method, sampling from MIONet parameter distributions. Consequently, we develop the B-LSTM-MIONet, incorporating LSTM's temporal strengths with Bayesian robustness, resulting in a more precise and reliable model for noisy datasets.

Title: On robust overfitting: adversarial training induced distribution matters. (arXiv:2311.16526v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.16526
• Code URL: null
• Copy Paste: [[2311.16526]] On robust overfitting: adversarial training induced distribution matters(http://arxiv.org/abs/2311.16526)
• Summary:

  Adversarial training may be regarded as standard training with a modified loss function. But its generalization error appears much larger than standard training under standard loss. This phenomenon, known as robust overfitting, has attracted significant research attention and remains largely as a mystery. In this paper, we first show empirically that robust overfitting correlates with the increasing generalization difficulty of the perturbation-induced distributions along the trajectory of adversarial training (specifically PGD-based adversarial training). We then provide a novel upper bound for generalization error with respect to the perturbation-induced distributions, in which a notion of the perturbation operator, referred to "local dispersion", plays an important role.

Title: Sinkhorn Flow: A Continuous-Time Framework for Understanding and Generalizing the Sinkhorn Algorithm. (arXiv:2311.16706v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.16706
• Code URL: null
• Copy Paste: [[2311.16706]] Sinkhorn Flow: A Continuous-Time Framework for Understanding and Generalizing the Sinkhorn Algorithm(http://arxiv.org/abs/2311.16706)
• Summary:

  Many problems in machine learning can be formulated as solving entropy-regularized optimal transport on the space of probability measures. The canonical approach involves the Sinkhorn iterates, renowned for their rich mathematical properties. Recently, the Sinkhorn algorithm has been recast within the mirror descent framework, thus benefiting from classical optimization theory insights. Here, we build upon this result by introducing a continuous-time analogue of the Sinkhorn algorithm. This perspective allows us to derive novel variants of Sinkhorn schemes that are robust to noise and bias. Moreover, our continuous-time dynamics not only generalize but also offer a unified perspective on several recently discovered dynamics in machine learning and mathematics, such as the "Wasserstein mirror flow" of (Deb et al. 2023) or the "mean-field Schr\"odinger equation" of (Claisse et al. 2023).

biometric

steal

extraction

Title: GeoTop: Advancing Image Classification with Geometric-Topological Analysis. (arXiv:2311.16157v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16157
• Code URL: null
• Copy Paste: [[2311.16157]] GeoTop: Advancing Image Classification with Geometric-Topological Analysis(http://arxiv.org/abs/2311.16157)
• Summary:

  In this study, we explore the application of Topological Data Analysis (TDA) and Lipschitz-Killing Curvatures (LKCs) as powerful tools for feature extraction and classification in the context of biomedical multiomics problems. TDA allows us to capture topological features and patterns within complex datasets, while LKCs provide essential geometric insights. We investigate the potential of combining both methods to improve classification accuracy. Using a dataset of biomedical images, we demonstrate that TDA and LKCs can effectively extract topological and geometrical features, respectively. The combination of these features results in enhanced classification performance compared to using each method individually. This approach offers promising results and has the potential to advance our understanding of complex biological processes in various biomedical applications. Our findings highlight the value of integrating topological and geometrical information in biomedical data analysis. As we continue to delve into the intricacies of multiomics problems, the fusion of these insights holds great promise for unraveling the underlying biological complexities.

Title: Progressive Target-Styled Feature Augmentation for Unsupervised Domain Adaptation on Point Clouds. (arXiv:2311.16474v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16474
• Code URL: https://github.com/xiaoyao3302/ptsfa
• Copy Paste: [[2311.16474]] Progressive Target-Styled Feature Augmentation for Unsupervised Domain Adaptation on Point Clouds(http://arxiv.org/abs/2311.16474)
• Summary:

  Unsupervised domain adaptation is a critical challenge in the field of point cloud analysis, as models trained on one set of data often struggle to perform well in new scenarios due to domain shifts. Previous works tackle the problem by using adversarial training or self-supervised learning for feature extractor adaptation, but ensuring that features extracted from the target domain can be distinguished by the source-supervised classifier remains challenging. In this work, we propose a novel approach called progressive target-styled feature augmentation (PTSFA). Unlike previous works that focus on feature extractor adaptation, our PTSFA approach focuses on classifier adaptation. It aims to empower the classifier to recognize target-styled source features and progressively adapt to the target domain. To enhance the reliability of predictions within the PTSFA framework and encourage discriminative feature extraction, we further introduce a new intermediate domain approaching (IDA) strategy. We validate our method on the benchmark datasets, where our method achieves new state-of-the-art performance. Our code is available at https://github.com/xiaoyao3302/PTSFA.

Title: MI-Gen: Multiple Instance Generation of Pathology Reports for Gigapixel Whole-Slide Images. (arXiv:2311.16480v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16480
• Code URL: null
• Copy Paste: [[2311.16480]] MI-Gen: Multiple Instance Generation of Pathology Reports for Gigapixel Whole-Slide Images(http://arxiv.org/abs/2311.16480)
• Summary:

  Whole slide images are the foundation of digital pathology for the diagnosis and treatment of carcinomas. Writing pathology reports is laborious and error-prone for inexperienced pathologists. To reduce the workload and improve clinical automation, we investigate how to generate pathology reports given whole slide images. On the data end, we curated the largest WSI-text dataset (TCGA-PathoText). In specific, we collected nearly 10000 high-quality WSI-text pairs for visual-language models by recognizing and cleaning pathology reports which narrate diagnostic slides in TCGA. On the model end, we propose the multiple instance generative model (MI-Gen) which can produce pathology reports for gigapixel WSIs. We benchmark our model on the largest subset of TCGA-PathoText. Experimental results show our model can generate pathology reports which contain multiple clinical clues. Furthermore, WSI-text prediction can be seen as an approach of visual-language pre-training, which enables our model to be transferred to downstream diagnostic tasks like carcinoma grading and phenotyping. We observe that simple semantic extraction from the pathology reports can achieve the best performance (0.838 of F1 score) on BRCA subtyping without adding extra parameters or tricky fine-tuning. Our collected dataset and related code will all be publicly available.

Title: ChartLlama: A Multimodal LLM for Chart Understanding and Generation. (arXiv:2311.16483v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16483
• Code URL: null
• Copy Paste: [[2311.16483]] ChartLlama: A Multimodal LLM for Chart Understanding and Generation(http://arxiv.org/abs/2311.16483)
• Summary:

  Multi-modal large language models have demonstrated impressive performances on most vision-language tasks. However, the model generally lacks the understanding capabilities for specific domain data, particularly when it comes to interpreting chart figures. This is mainly due to the lack of relevant multi-modal instruction tuning datasets. In this article, we create a high-quality instruction-tuning dataset leveraging GPT-4. We develop a multi-step data generation process in which different steps are responsible for generating tabular data, creating chart figures, and designing instruction tuning data separately. Our method's flexibility enables us to generate diverse, high-quality instruction-tuning data consistently and efficiently while maintaining a low resource expenditure. Additionally, it allows us to incorporate a wider variety of chart and task types not yet featured in existing datasets. Next, we introduce ChartLlama, a multi-modal large language model that we've trained using our created dataset. ChartLlama outperforms all prior methods in ChartQA, Chart-to-text, and Chart-extraction evaluation benchmarks. Additionally, ChartLlama significantly improves upon the baseline in our specially compiled chart dataset, which includes new chart and task types. The results of ChartLlama confirm the value and huge potential of our proposed data generation method in enhancing chart comprehension.

Title: Evaluation of dynamic characteristics of power grid based on GNN and application on knowledge graph. (arXiv:2311.16522v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.16522
• Code URL: null
• Copy Paste: [[2311.16522]] Evaluation of dynamic characteristics of power grid based on GNN and application on knowledge graph(http://arxiv.org/abs/2311.16522)
• Summary:

  A novel method for detecting faults in power grids using a graph neural network (GNN) has been developed, aimed at enhancing intelligent fault diagnosis in network operation and maintenance. This GNN-based approach identifies faulty nodes within the power grid through a specialized electrical feature extraction model coupled with a knowledge graph. Incorporating temporal data, the method leverages the status of nodes from preceding and subsequent time periods to aid in current fault detection. To validate the effectiveness of this GNN in extracting node features, a correlation analysis of the output features from each node within the neural network layer was conducted. The results from experiments show that this method can accurately locate fault nodes in simulated scenarios with a remarkable 99.53% accuracy. Additionally, the graph neural network's feature modeling allows for a qualitative examination of how faults spread across nodes, providing valuable insights for analyzing fault nodes.

Title: Entity-Aspect-Opinion-Sentiment Quadruple Extraction for Fine-grained Sentiment Analysis. (arXiv:2311.16678v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.16678
• Code URL: null
• Copy Paste: [[2311.16678]] Entity-Aspect-Opinion-Sentiment Quadruple Extraction for Fine-grained Sentiment Analysis(http://arxiv.org/abs/2311.16678)
• Summary:

  Product reviews often contain a large number of implicit aspects and object-attribute co-existence cases. Unfortunately, many existing studies in Aspect-Based Sentiment Analysis (ABSA) have overlooked this issue, which can make it difficult to extract opinions comprehensively and fairly. In this paper, we propose a new task called Entity-Aspect-Opinion-Sentiment Quadruple Extraction (EASQE), which aims to hierarchically decompose aspect terms into entities and aspects to avoid information loss, non-exclusive annotations, and opinion misunderstandings in ABSA tasks. To facilitate research in this new task, we have constructed four datasets (Res14-EASQE, Res15-EASQE, Res16-EASQE, and Lap14-EASQE) based on the SemEval Restaurant and Laptop datasets. We have also proposed a novel two-stage sequence-tagging based Trigger-Opinion framework as the baseline for the EASQE task. Empirical evaluations show that our Trigger-Opinion framework can generate satisfactory EASQE results and can also be applied to other ABSA tasks, significantly outperforming state-of-the-art methods. We have made the four datasets and source code of Trigger-Opinion publicly available to facilitate further research in this area.

membership infer

federate

Title: On the Effect of Defections in Federated Learning and How to Prevent Them. (arXiv:2311.16459v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.16459
• Code URL: null
• Copy Paste: [[2311.16459]] On the Effect of Defections in Federated Learning and How to Prevent Them(http://arxiv.org/abs/2311.16459)
• Summary:

  Federated learning is a machine learning protocol that enables a large population of agents to collaborate over multiple rounds to produce a single consensus model. There are several federated learning applications where agents may choose to defect permanently$-$essentially withdrawing from the collaboration$-$if they are content with their instantaneous model in that round. This work demonstrates the detrimental impact of such defections on the final model's robustness and ability to generalize. We also show that current federated optimization algorithms fail to disincentivize these harmful defections. We introduce a novel optimization algorithm with theoretical guarantees to prevent defections while ensuring asymptotic convergence to an effective solution for all participating agents. We also provide numerical experiments to corroborate our findings and demonstrate the effectiveness of our algorithm.

Title: Contrastive encoder pre-training-based clustered federated learning for heterogeneous data. (arXiv:2311.16535v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.16535
• Code URL: null
• Copy Paste: [[2311.16535]] Contrastive encoder pre-training-based clustered federated learning for heterogeneous data(http://arxiv.org/abs/2311.16535)
• Summary:

  Federated learning (FL) is a promising approach that enables distributed clients to collaboratively train a global model while preserving their data privacy. However, FL often suffers from data heterogeneity problems, which can significantly affect its performance. To address this, clustered federated learning (CFL) has been proposed to construct personalized models for different client clusters. One effective client clustering strategy is to allow clients to choose their own local models from a model pool based on their performance. However, without pre-trained model parameters, such a strategy is prone to clustering failure, in which all clients choose the same model. Unfortunately, collecting a large amount of labeled data for pre-training can be costly and impractical in distributed environments. To overcome this challenge, we leverage self-supervised contrastive learning to exploit unlabeled data for the pre-training of FL systems. Together, self-supervised pre-training and client clustering can be crucial components for tackling the data heterogeneity issues of FL. Leveraging these two crucial strategies, we propose contrastive pre-training-based clustered federated learning (CP-CFL) to improve the model convergence and overall performance of FL systems. In this work, we demonstrate the effectiveness of CP-CFL through extensive experiments in heterogeneous FL settings, and present various interesting observations.

Title: Communication Efficiency Optimization of Federated Learning for Computing and Network Convergence of 6G Networks. (arXiv:2311.16540v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.16540
• Code URL: null
• Copy Paste: [[2311.16540]] Communication Efficiency Optimization of Federated Learning for Computing and Network Convergence of 6G Networks(http://arxiv.org/abs/2311.16540)
• Summary:

  Federated learning effectively addresses issues such as data privacy by collaborating across participating devices to train global models. However, factors such as network topology and device computing power can affect its training or communication process in complex network environments. A new network architecture and paradigm with computing-measurable, perceptible, distributable, dispatchable, and manageable capabilities, computing and network convergence (CNC) of 6G networks can effectively support federated learning training and improve its communication efficiency. By guiding the participating devices' training in federated learning based on business requirements, resource load, network conditions, and arithmetic power of devices, CNC can reach this goal. In this paper, to improve the communication efficiency of federated learning in complex networks, we study the communication efficiency optimization of federated learning for computing and network convergence of 6G networks, methods that gives decisions on its training process for different network conditions and arithmetic power of participating devices in federated learning. The experiments address two architectures that exist for devices in federated learning and arrange devices to participate in training based on arithmetic power while achieving optimization of communication efficiency in the process of transferring model parameters. The results show that the method we proposed can (1) cope well with complex network situations (2) effectively balance the delay distribution of participating devices for local training (3) improve the communication efficiency during the transfer of model parameters (4) improve the resource utilization in the network.

Title: FedAL: Black-Box Federated Knowledge Distillation Enabled by Adversarial Learning. (arXiv:2311.16584v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.16584
• Code URL: null
• Copy Paste: [[2311.16584]] FedAL: Black-Box Federated Knowledge Distillation Enabled by Adversarial Learning(http://arxiv.org/abs/2311.16584)
• Summary:

  Knowledge distillation (KD) can enable collaborative learning among distributed clients that have different model architectures and do not share their local data and model parameters with others. Each client updates its local model using the average model output/feature of all client models as the target, known as federated KD. However, existing federated KD methods often do not perform well when clients' local models are trained with heterogeneous local datasets. In this paper, we propose Federated knowledge distillation enabled by Adversarial Learning (FedAL) to address the data heterogeneity among clients. First, to alleviate the local model output divergence across clients caused by data heterogeneity, the server acts as a discriminator to guide clients' local model training to achieve consensus model outputs among clients through a min-max game between clients and the discriminator. Moreover, catastrophic forgetting may happen during the clients' local training and global knowledge transfer due to clients' heterogeneous local data. Towards this challenge, we design the less-forgetting regularization for both local training and global knowledge transfer to guarantee clients' ability to transfer/learn knowledge to/from others. Experimental results show that FedAL and its variants achieve higher accuracy than other federated KD baselines.

Title: Asynchronous Wireless Federated Learning with Probabilistic Client Selection. (arXiv:2311.16741v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.16741
• Code URL: null
• Copy Paste: [[2311.16741]] Asynchronous Wireless Federated Learning with Probabilistic Client Selection(http://arxiv.org/abs/2311.16741)
• Summary:

  Federated learning (FL) is a promising distributed learning framework where distributed clients collaboratively train a machine learning model coordinated by a server. To tackle the stragglers issue in asynchronous FL, we consider that each client keeps local updates and probabilistically transmits the local model to the server at arbitrary times. We first derive the (approximate) expression for the convergence rate based on the probabilistic client selection. Then, an optimization problem is formulated to trade off the convergence rate of asynchronous FL and mobile energy consumption by joint probabilistic client selection and bandwidth allocation. We develop an iterative algorithm to solve the non-convex problem globally optimally. Experiments demonstrate the superiority of the proposed approach compared with the traditional schemes.

fair

Title: Aiming to Minimize Alcohol-Impaired Road Fatalities: Utilizing Fairness-Aware and Domain Knowledge-Infused Artificial Intelligence. (arXiv:2311.16180v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.16180
• Code URL: null
• Copy Paste: [[2311.16180]] Aiming to Minimize Alcohol-Impaired Road Fatalities: Utilizing Fairness-Aware and Domain Knowledge-Infused Artificial Intelligence(http://arxiv.org/abs/2311.16180)
• Summary:

  Approximately 30% of all traffic fatalities in the United States are attributed to alcohol-impaired driving. This means that, despite stringent laws against this offense in every state, the frequency of drunk driving accidents is alarming, resulting in approximately one person being killed every 45 minutes. The process of charging individuals with Driving Under the Influence (DUI) is intricate and can sometimes be subjective, involving multiple stages such as observing the vehicle in motion, interacting with the driver, and conducting Standardized Field Sobriety Tests (SFSTs). Biases have been observed through racial profiling, leading to some groups and geographical areas facing fewer DUI tests, resulting in many actual DUI incidents going undetected, ultimately leading to a higher number of fatalities. To tackle this issue, our research introduces an Artificial Intelligence-based predictor that is both fairness-aware and incorporates domain knowledge to analyze DUI-related fatalities in different geographic locations. Through this model, we gain intriguing insights into the interplay between various demographic groups, including age, race, and income. By utilizing the provided information to allocate policing resources in a more equitable and efficient manner, there is potential to reduce DUI-related fatalities and have a significant impact on road safety.

interpretability

Title: Deep Learning for Time Series Classification of Parkinson's Disease Eye Tracking Data. (arXiv:2311.16381v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.16381
• Code URL: null
• Copy Paste: [[2311.16381]] Deep Learning for Time Series Classification of Parkinson's Disease Eye Tracking Data(http://arxiv.org/abs/2311.16381)
• Summary:

  Eye-tracking is an accessible and non-invasive technology that provides information about a subject's motor and cognitive abilities. As such, it has proven to be a valuable resource in the study of neurodegenerative diseases such as Parkinson's disease. Saccade experiments, in particular, have proven useful in the diagnosis and staging of Parkinson's disease. However, to date, no single eye-movement biomarker has been found to conclusively differentiate patients from healthy controls. In the present work, we investigate the use of state-of-the-art deep learning algorithms to perform Parkinson's disease classification using eye-tracking data from saccade experiments. In contrast to previous work, instead of using hand-crafted features from the saccades, we use raw $\sim1.5\,s$ long fixation intervals recorded during the preparatory phase before each trial. Using these short time series as input we implement two different classification models, InceptionTime and ROCKET. We find that the models are able to learn the classification task and generalize to unseen subjects. InceptionTime achieves $78\%$ accuracy, while ROCKET achieves $88\%$ accuracy. We also employ a novel method for pruning the ROCKET model to improve interpretability and generalizability, achieving an accuracy of $96\%$. Our results suggest that fixation data has low inter-subject variability and potentially carries useful information about brain cognitive and motor conditions, making it suitable for use with machine learning in the discovery of disease-relevant biomarkers.

explainability

Title: Elucidating Discrepancy in Explanations of Predictive Models Developed using EMR. (arXiv:2311.16654v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.16654
• Code URL: null
• Copy Paste: [[2311.16654]] Elucidating Discrepancy in Explanations of Predictive Models Developed using EMR(http://arxiv.org/abs/2311.16654)
• Summary:

  The lack of transparency and explainability hinders the clinical adoption of Machine learning (ML) algorithms. While explainable artificial intelligence (XAI) methods have been proposed, little research has focused on the agreement between these methods and expert clinical knowledge. This study applies current state-of-the-art explainability methods to clinical decision support algorithms developed for Electronic Medical Records (EMR) data to analyse the concordance between these factors and discusses causes for identified discrepancies from a clinical and technical perspective. Important factors for achieving trustworthy XAI solutions for clinical decision support are also discussed.

watermark

diffusion

Title: Predicated Diffusion: Predicate Logic-Based Attention Guidance for Text-to-Image Diffusion Models. (arXiv:2311.16117v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16117
• Code URL: null
• Copy Paste: [[2311.16117]] Predicated Diffusion: Predicate Logic-Based Attention Guidance for Text-to-Image Diffusion Models(http://arxiv.org/abs/2311.16117)
• Summary:

  Diffusion models have achieved remarkable results in generating high-quality, diverse, and creative images. However, when it comes to text-based image generation, they often fail to capture the intended meaning presented in the text. For instance, a specified object may not be generated, an unnecessary object may be generated, and an adjective may alter objects it was not intended to modify. Moreover, we found that relationships indicating possession between objects are often overlooked. While users' intentions in text are diverse, existing methods tend to specialize in only some aspects of these. In this paper, we propose Predicated Diffusion, a unified framework to express users' intentions. We consider that the root of the above issues lies in the text encoder, which often focuses only on individual words and neglects the logical relationships between them. The proposed method does not solely rely on the text encoder, but instead, represents the intended meaning in the text as propositions using predicate logic and treats the pixels in the attention maps as the fuzzy predicates. This enables us to obtain a differentiable loss function that makes the image fulfill the proposition by minimizing it. When compared to several existing methods, we demonstrated that Predicated Diffusion can generate images that are more faithful to various text prompts, as verified by human evaluators and pretrained image-text models.

Title: Effective Quantization for Diffusion Models on CPUs. (arXiv:2311.16133v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16133
• Code URL: null
• Copy Paste: [[2311.16133]] Effective Quantization for Diffusion Models on CPUs(http://arxiv.org/abs/2311.16133)
• Summary:

  Diffusion models have gained popularity for generating images from textual descriptions. Nonetheless, the substantial need for computational resources continues to present a noteworthy challenge, contributing to time-consuming processes. Quantization, a technique employed to compress deep learning models for enhanced efficiency, presents challenges when applied to diffusion models. These models are notably more sensitive to quantization compared to other model types, potentially resulting in a degradation of image quality. In this paper, we introduce a novel approach to quantize the diffusion models by leveraging both quantization-aware training and distillation. Our results show the quantized models can maintain the high image quality while demonstrating the inference efficiency on CPUs.

Title: Shortcut Bias Mitigation via Ensemble Diversity Using Diffusion Probabilistic Models. (arXiv:2311.16176v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.16176
• Code URL: null
• Copy Paste: [[2311.16176]] Shortcut Bias Mitigation via Ensemble Diversity Using Diffusion Probabilistic Models(http://arxiv.org/abs/2311.16176)
• Summary:

  Spurious correlations in the data, where multiple cues are predictive of the target labels, often lead to a phenomenon known as simplicity bias, where a model relies on erroneous, easy-to-learn cues while ignoring reliable ones. In this work, we propose an ensemble diversification framework exploiting Diffusion Probabilistic Models (DPMs) for shortcut bias mitigation. We show that at particular training intervals, DPMs can generate images with novel feature combinations, even when trained on images displaying correlated input features. We leverage this crucial property to generate synthetic counterfactuals to increase model diversity via ensemble disagreement. We show that DPM-guided diversification is sufficient to remove dependence on primary shortcut cues, without a need for additional supervised signals. We further empirically quantify its efficacy on several diversification objectives, and finally show improved generalization and diversification performance on par with prior work that relies on auxiliary data collection.

Title: Improving Denoising Diffusion Probabilistic Models via Exploiting Shared Representations. (arXiv:2311.16353v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.16353
• Code URL: null
• Copy Paste: [[2311.16353]] Improving Denoising Diffusion Probabilistic Models via Exploiting Shared Representations(http://arxiv.org/abs/2311.16353)
• Summary:

  In this work, we address the challenge of multi-task image generation with limited data for denoising diffusion probabilistic models (DDPM), a class of generative models that produce high-quality images by reversing a noisy diffusion process. We propose a novel method, SR-DDPM, that leverages representation-based techniques from few-shot learning to effectively learn from fewer samples across different tasks. Our method consists of a core meta architecture with shared parameters, i.e., task-specific layers with exclusive parameters. By exploiting the similarity between diverse data distributions, our method can scale to multiple tasks without compromising the image quality. We evaluate our method on standard image datasets and show that it outperforms both unconditional and conditional DDPM in terms of FID and SSIM metrics.

Title: Manifold Preserving Guided Diffusion. (arXiv:2311.16424v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.16424
• Code URL: null
• Copy Paste: [[2311.16424]] Manifold Preserving Guided Diffusion(http://arxiv.org/abs/2311.16424)
• Summary:

  Despite the recent advancements, conditional image generation still faces challenges of cost, generalizability, and the need for task-specific training. In this paper, we propose Manifold Preserving Guided Diffusion (MPGD), a training-free conditional generation framework that leverages pretrained diffusion models and off-the-shelf neural networks with minimal additional inference cost for a broad range of tasks. Specifically, we leverage the manifold hypothesis to refine the guided diffusion steps and introduce a shortcut algorithm in the process. We then propose two methods for on-manifold training-free guidance using pre-trained autoencoders and demonstrate that our shortcut inherently preserves the manifolds when applied to latent diffusion models. Our experiments show that MPGD is efficient and effective for solving a variety of conditional generation applications in low-compute settings, and can consistently offer up to 3.8x speed-ups with the same number of diffusion steps while maintaining high sample quality compared to the baselines.

Title: TextDiffuser-2: Unleashing the Power of Language Models for Text Rendering. (arXiv:2311.16465v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16465
• Code URL: null
• Copy Paste: [[2311.16465]] TextDiffuser-2: Unleashing the Power of Language Models for Text Rendering(http://arxiv.org/abs/2311.16465)
• Summary:

  The diffusion model has been proven a powerful generative model in recent years, yet remains a challenge in generating visual text. Several methods alleviated this issue by incorporating explicit text position and content as guidance on where and what text to render. However, these methods still suffer from several drawbacks, such as limited flexibility and automation, constrained capability of layout prediction, and restricted style diversity. In this paper, we present TextDiffuser-2, aiming to unleash the power of language models for text rendering. Firstly, we fine-tune a large language model for layout planning. The large language model is capable of automatically generating keywords for text rendering and also supports layout modification through chatting. Secondly, we utilize the language model within the diffusion model to encode the position and texts at the line level. Unlike previous methods that employed tight character-level guidance, this approach generates more diverse text images. We conduct extensive experiments and incorporate user studies involving human participants as well as GPT-4V, validating TextDiffuser-2's capacity to achieve a more rational text layout and generation with enhanced diversity. The code and model will be available at \url{https://aka.ms/textdiffuser-2}.

Title: Efficient Multimodal Diffusion Models Using Joint Data Infilling with Partially Shared U-Net. (arXiv:2311.16488v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16488
• Code URL: null
• Copy Paste: [[2311.16488]] Efficient Multimodal Diffusion Models Using Joint Data Infilling with Partially Shared U-Net(http://arxiv.org/abs/2311.16488)
• Summary:

  Recently, diffusion models have been used successfully to fit distributions for cross-modal data translation and multimodal data generation. However, these methods rely on extensive scaling, overlooking the inefficiency and interference between modalities. We develop Partially Shared U-Net (PS-U-Net) architecture which is an efficient multimodal diffusion model that allows text and image inputs to pass through dedicated layers and skip-connections for preserving modality-specific fine-grained details. Inspired by image inpainting, we also propose a new efficient multimodal sampling method that introduces new scenarios for conditional generation while only requiring a simple joint distribution to be learned. Our empirical exploration of the MS-COCO dataset demonstrates that our method generates multimodal text and image data with higher quality compared to existing multimodal diffusion models while having a comparable size, faster training, faster multimodal sampling, and more flexible generation.

Title: $Z^*$: Zero-shot Style Transfer via Attention Rearrangement. (arXiv:2311.16491v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16491
• Code URL: null
• Copy Paste: [[2311.16491]] $Z^*$: Zero-shot Style Transfer via Attention Rearrangement(http://arxiv.org/abs/2311.16491)
• Summary:

  Despite the remarkable progress in image style transfer, formulating style in the context of art is inherently subjective and challenging. In contrast to existing learning/tuning methods, this study shows that vanilla diffusion models can directly extract style information and seamlessly integrate the generative prior into the content image without retraining. Specifically, we adopt dual denoising paths to represent content/style references in latent space and then guide the content image denoising process with style latent codes. We further reveal that the cross-attention mechanism in latent diffusion models tends to blend the content and style images, resulting in stylized outputs that deviate from the original content image. To overcome this limitation, we introduce a cross-attention rearrangement strategy. Through theoretical analysis and experiments, we demonstrate the effectiveness and superiority of the diffusion-based $\underline{Z}$ero-shot $\underline{S}$tyle $\underline{T}$ransfer via $\underline{A}$ttention $\underline{R}$earrangement, Z-STAR.

Title: Egocentric Whole-Body Motion Capture with FisheyeViT and Diffusion-Based Motion Refinement. (arXiv:2311.16495v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16495
• Code URL: null
• Copy Paste: [[2311.16495]] Egocentric Whole-Body Motion Capture with FisheyeViT and Diffusion-Based Motion Refinement(http://arxiv.org/abs/2311.16495)
• Summary:

  In this work, we explore egocentric whole-body motion capture using a single fisheye camera, which simultaneously estimates human body and hand motion. This task presents significant challenges due to three factors: the lack of high-quality datasets, fisheye camera distortion, and human body self-occlusion. To address these challenges, we propose a novel approach that leverages FisheyeViT to extract fisheye image features, which are subsequently converted into pixel-aligned 3D heatmap representations for 3D human body pose prediction. For hand tracking, we incorporate dedicated hand detection and hand pose estimation networks for regressing 3D hand poses. Finally, we develop a diffusion-based whole-body motion prior model to refine the estimated whole-body motion while accounting for joint uncertainties. To train these networks, we collect a large synthetic dataset, EgoWholeBody, comprising 840,000 high-quality egocentric images captured across a diverse range of whole-body motion sequences. Quantitative and qualitative evaluations demonstrate the effectiveness of our method in producing high-quality whole-body motion estimates from a single egocentric camera.

Title: MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model. (arXiv:2311.16498v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16498
• Code URL: null
• Copy Paste: [[2311.16498]] MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model(http://arxiv.org/abs/2311.16498)
• Summary:

  This paper studies the human image animation task, which aims to generate a video of a certain reference identity following a particular motion sequence. Existing animation works typically employ the frame-warping technique to animate the reference image towards the target motion. Despite achieving reasonable results, these approaches face challenges in maintaining temporal consistency throughout the animation due to the lack of temporal modeling and poor preservation of reference identity. In this work, we introduce MagicAnimate, a diffusion-based framework that aims at enhancing temporal consistency, preserving reference image faithfully, and improving animation fidelity. To achieve this, we first develop a video diffusion model to encode temporal information. Second, to maintain the appearance coherence across frames, we introduce a novel appearance encoder to retain the intricate details of the reference image. Leveraging these two innovations, we further employ a simple video fusion technique to encourage smooth transitions for long video animation. Empirical results demonstrate the superiority of our method over baseline approaches on two benchmarks. Notably, our approach outperforms the strongest baseline by over 38% in terms of video fidelity on the challenging TikTok dancing dataset. Code and model will be made available.

Title: Deceptive-Human: Prompt-to-NeRF 3D Human Generation with 3D-Consistent Synthetic Images. (arXiv:2311.16499v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16499
• Code URL: null
• Copy Paste: [[2311.16499]] Deceptive-Human: Prompt-to-NeRF 3D Human Generation with 3D-Consistent Synthetic Images(http://arxiv.org/abs/2311.16499)
• Summary:

  This paper presents Deceptive-Human, a novel Prompt-to-NeRF framework capitalizing state-of-the-art control diffusion models (e.g., ControlNet) to generate a high-quality controllable 3D human NeRF. Different from direct 3D generative approaches, e.g., DreamFusion and DreamHuman, Deceptive-Human employs a progressive refinement technique to elevate the reconstruction quality. This is achieved by utilizing high-quality synthetic human images generated through the ControlNet with view-consistent loss. Our method is versatile and readily extensible, accommodating multimodal inputs, including a text prompt and additional data such as 3D mesh, poses, and seed images. The resulting 3D human NeRF model empowers the synthesis of highly photorealistic novel views from 360-degree perspectives. The key to our Deceptive-Human for hallucinating multi-view consistent synthetic human images lies in our progressive finetuning strategy. This strategy involves iteratively enhancing views using the provided multimodal inputs at each intermediate step to improve the human NeRF model. Within this iterative refinement process, view-dependent appearances are systematically eliminated to prevent interference with the underlying density estimation. Extensive qualitative and quantitative experimental comparison shows that our deceptive human models achieve state-of-the-art application quality.

Title: TFMQ-DM: Temporal Feature Maintenance Quantization for Diffusion Models. (arXiv:2311.16503v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16503
• Code URL: null
• Copy Paste: [[2311.16503]] TFMQ-DM: Temporal Feature Maintenance Quantization for Diffusion Models(http://arxiv.org/abs/2311.16503)
• Summary:

  The Diffusion model, a prevalent framework for image generation, encounters significant challenges in terms of broad applicability due to its extended inference times and substantial memory requirements. Efficient Post-training Quantization (PTQ) is pivotal for addressing these issues in traditional models. Different from traditional models, diffusion models heavily depend on the time-step $t$ to achieve satisfactory multi-round denoising. Usually, $t$ from the finite set $\{1, \ldots, T\}$ is encoded to a temporal feature by a few modules totally irrespective of the sampling data. However, existing PTQ methods do not optimize these modules separately. They adopt inappropriate reconstruction targets and complex calibration methods, resulting in a severe disturbance of the temporal feature and denoising trajectory, as well as a low compression efficiency. To solve these, we propose a Temporal Feature Maintenance Quantization (TFMQ) framework building upon a Temporal Information Block which is just related to the time-step $t$ and unrelated to the sampling data. Powered by the pioneering block design, we devise temporal information aware reconstruction (TIAR) and finite set calibration (FSC) to align the full-precision temporal features in a limited time. Equipped with the framework, we can maintain the most temporal information and ensure the end-to-end generation quality. Extensive experiments on various datasets and diffusion models prove our state-of-the-art results. Remarkably, our quantization approach, for the first time, achieves model performance nearly on par with the full-precision model under 4-bit weight quantization. Additionally, our method incurs almost no extra computational cost and accelerates quantization time by $2.0 \times$ on LSUN-Bedrooms $256 \times 256$ compared to previous works.

Title: Exploring Straighter Trajectories of Flow Matching with Diffusion Guidance. (arXiv:2311.16507v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16507
• Code URL: null
• Copy Paste: [[2311.16507]] Exploring Straighter Trajectories of Flow Matching with Diffusion Guidance(http://arxiv.org/abs/2311.16507)
• Summary:

  Flow matching as a paradigm of generative model achieves notable success across various domains. However, existing methods use either multi-round training or knowledge within minibatches, posing challenges in finding a favorable coupling strategy for straight trajectories. To address this issue, we propose a novel approach, Straighter trajectories of Flow Matching (StraightFM). It straightens trajectories with the coupling strategy guided by diffusion model from entire distribution level. First, we propose a coupling strategy to straighten trajectories, creating couplings between image and noise samples under diffusion model guidance. Second, StraightFM also integrates real data to enhance training, employing a neural network to parameterize another coupling process from images to noise samples. StraightFM is jointly optimized with couplings from above two mutually complementary directions, resulting in straighter trajectories and enabling both one-step and few-step generation. Extensive experiments demonstrate that StraightFM yields high quality samples with fewer step. StraightFM generates visually appealing images with a lower FID among diffusion and traditional flow matching methods within 5 sampling steps when trained on pixel space. In the latent space (i.e., Latent Diffusion), StraightFM achieves a lower KID value compared to existing methods on the CelebA-HQ 256 dataset in fewer than 10 sampling steps.

Title: CoSeR: Bridging Image and Language for Cognitive Super-Resolution. (arXiv:2311.16512v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16512
• Code URL: null
• Copy Paste: [[2311.16512]] CoSeR: Bridging Image and Language for Cognitive Super-Resolution(http://arxiv.org/abs/2311.16512)
• Summary:

  Existing super-resolution (SR) models primarily focus on restoring local texture details, often neglecting the global semantic information within the scene. This oversight can lead to the omission of crucial semantic details or the introduction of inaccurate textures during the recovery process. In our work, we introduce the Cognitive Super-Resolution (CoSeR) framework, empowering SR models with the capacity to comprehend low-resolution images. We achieve this by marrying image appearance and language understanding to generate a cognitive embedding, which not only activates prior information from large text-to-image diffusion models but also facilitates the generation of high-quality reference images to optimize the SR process. To further improve image fidelity, we propose a novel condition injection scheme called "All-in-Attention", consolidating all conditional information into a single module. Consequently, our method successfully restores semantically correct and photorealistic details, demonstrating state-of-the-art performance across multiple benchmarks.

Title: Fine-grained Appearance Transfer with Diffusion Models. (arXiv:2311.16513v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16513
• Code URL: https://github.com/babahui/fine-grained-appearance-transfer
• Copy Paste: [[2311.16513]] Fine-grained Appearance Transfer with Diffusion Models(http://arxiv.org/abs/2311.16513)
• Summary:

  Image-to-image translation (I2I), and particularly its subfield of appearance transfer, which seeks to alter the visual appearance between images while maintaining structural coherence, presents formidable challenges. Despite significant advancements brought by diffusion models, achieving fine-grained transfer remains complex, particularly in terms of retaining detailed structural elements and ensuring information fidelity. This paper proposes an innovative framework designed to surmount these challenges by integrating various aspects of semantic matching, appearance transfer, and latent deviation. A pivotal aspect of our approach is the strategic use of the predicted $x_0$ space by diffusion models within the latent space of diffusion processes. This is identified as a crucial element for the precise and natural transfer of fine-grained details. Our framework exploits this space to accomplish semantic alignment between source and target images, facilitating mask-wise appearance transfer for improved feature acquisition. A significant advancement of our method is the seamless integration of these features into the latent space, enabling more nuanced latent deviations without necessitating extensive model retraining or fine-tuning. The effectiveness of our approach is demonstrated through extensive experiments, which showcase its ability to adeptly handle fine-grained appearance transfers across a wide range of categories and domains. We provide our code at https://github.com/babahui/Fine-grained-Appearance-Transfer

Title: Video Anomaly Detection via Spatio-Temporal Pseudo-Anomaly Generation : A Unified Approach. (arXiv:2311.16514v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16514
• Code URL: null
• Copy Paste: [[2311.16514]] Video Anomaly Detection via Spatio-Temporal Pseudo-Anomaly Generation : A Unified Approach(http://arxiv.org/abs/2311.16514)
• Summary:

  Video Anomaly Detection (VAD) is an open-set recognition task, which is usually formulated as a one-class classification (OCC) problem, where training data is comprised of videos with normal instances while test data contains both normal and anomalous instances. Recent works have investigated the creation of pseudo-anomalies (PAs) using only the normal data and making strong assumptions about real-world anomalies with regards to abnormality of objects and speed of motion to inject prior information about anomalies in an autoencoder (AE) based reconstruction model during training. This work proposes a novel method for generating generic spatio-temporal PAs by inpainting a masked out region of an image using a pre-trained Latent Diffusion Model and further perturbing the optical flow using mixup to emulate spatio-temporal distortions in the data. In addition, we present a simple unified framework to detect real-world anomalies under the OCC setting by learning three types of anomaly indicators, namely reconstruction quality, temporal irregularity and semantic inconsistency. Extensive experiments on four VAD benchmark datasets namely Ped2, Avenue, ShanghaiTech and UBnormal demonstrate that our method performs on par with other existing state-of-the-art PAs generation and reconstruction based methods under the OCC setting. Our analysis also examines the transferability and generalisation of PAs across these datasets, offering valuable insights by identifying real-world anomalies through PAs.

Title: SeeSR: Towards Semantics-Aware Real-World Image Super-Resolution. (arXiv:2311.16518v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16518
• Code URL: https://github.com/cswry/seesr
• Copy Paste: [[2311.16518]] SeeSR: Towards Semantics-Aware Real-World Image Super-Resolution(http://arxiv.org/abs/2311.16518)
• Summary:

  Owe to the powerful generative priors, the pre-trained text-to-image (T2I) diffusion models have become increasingly popular in solving the real-world image super-resolution problem. However, as a consequence of the heavy quality degradation of input low-resolution (LR) images, the destruction of local structures can lead to ambiguous image semantics. As a result, the content of reproduced high-resolution image may have semantic errors, deteriorating the super-resolution performance. To address this issue, we present a semantics-aware approach to better preserve the semantic fidelity of generative real-world image super-resolution. First, we train a degradation-aware prompt extractor, which can generate accurate soft and hard semantic prompts even under strong degradation. The hard semantic prompts refer to the image tags, aiming to enhance the local perception ability of the T2I model, while the soft semantic prompts compensate for the hard ones to provide additional representation information. These semantic prompts can encourage the T2I model to generate detailed and semantically accurate results. Furthermore, during the inference process, we integrate the LR images into the initial sampling noise to mitigate the diffusion model's tendency to generate excessive random details. The experiments show that our method can reproduce more realistic image details and hold better the semantics.

Title: Enhancing Scene Text Detectors with Realistic Text Image Synthesis Using Diffusion Models. (arXiv:2311.16555v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16555
• Code URL: null
• Copy Paste: [[2311.16555]] Enhancing Scene Text Detectors with Realistic Text Image Synthesis Using Diffusion Models(http://arxiv.org/abs/2311.16555)
• Summary:

  Scene text detection techniques have garnered significant attention due to their wide-ranging applications. However, existing methods have a high demand for training data, and obtaining accurate human annotations is labor-intensive and time-consuming. As a solution, researchers have widely adopted synthetic text images as a complementary resource to real text images during pre-training. Yet there is still room for synthetic datasets to enhance the performance of scene text detectors. We contend that one main limitation of existing generation methods is the insufficient integration of foreground text with the background. To alleviate this problem, we present the Diffusion Model based Text Generator (DiffText), a pipeline that utilizes the diffusion model to seamlessly blend foreground text regions with the background's intrinsic features. Additionally, we propose two strategies to generate visually coherent text with fewer spelling errors. With fewer text instances, our produced text images consistently surpass other synthetic data in aiding text detectors. Extensive experiments on detecting horizontal, rotated, curved, and line-level texts demonstrate the effectiveness of DiffText in producing realistic text images.

Title: DiffusionTalker: Personalization and Acceleration for Speech-Driven 3D Face Diffuser. (arXiv:2311.16565v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16565
• Code URL: null
• Copy Paste: [[2311.16565]] DiffusionTalker: Personalization and Acceleration for Speech-Driven 3D Face Diffuser(http://arxiv.org/abs/2311.16565)
• Summary:

  Speech-driven 3D facial animation has been an attractive task in both academia and industry. Traditional methods mostly focus on learning a deterministic mapping from speech to animation. Recent approaches start to consider the non-deterministic fact of speech-driven 3D face animation and employ the diffusion model for the task. However, personalizing facial animation and accelerating animation generation are still two major limitations of existing diffusion-based methods. To address the above limitations, we propose DiffusionTalker, a diffusion-based method that utilizes contrastive learning to personalize 3D facial animation and knowledge distillation to accelerate 3D animation generation. Specifically, to enable personalization, we introduce a learnable talking identity to aggregate knowledge in audio sequences. The proposed identity embeddings extract customized facial cues across different people in a contrastive learning manner. During inference, users can obtain personalized facial animation based on input audio, reflecting a specific talking style. With a trained diffusion model with hundreds of steps, we distill it into a lightweight model with 8 steps for acceleration. Extensive experiments are conducted to demonstrate that our method outperforms state-of-the-art methods. The code will be released.

Title: MobileDiffusion: Subsecond Text-to-Image Generation on Mobile Devices. (arXiv:2311.16567v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16567
• Code URL: null
• Copy Paste: [[2311.16567]] MobileDiffusion: Subsecond Text-to-Image Generation on Mobile Devices(http://arxiv.org/abs/2311.16567)
• Summary:

  The deployment of large-scale text-to-image diffusion models on mobile devices is impeded by their substantial model size and slow inference speed. In this paper, we propose \textbf{MobileDiffusion}, a highly efficient text-to-image diffusion model obtained through extensive optimizations in both architecture and sampling techniques. We conduct a comprehensive examination of model architecture design to reduce redundancy, enhance computational efficiency, and minimize model's parameter count, while preserving image generation quality. Additionally, we employ distillation and diffusion-GAN finetuning techniques on MobileDiffusion to achieve 8-step and 1-step inference respectively. Empirical studies, conducted both quantitatively and qualitatively, demonstrate the effectiveness of our proposed techniques. MobileDiffusion achieves a remarkable \textbf{sub-second} inference speed for generating a $512\times512$ image on mobile devices, establishing a new state of the art.

Title: LEDITS++: Limitless Image Editing using Text-to-Image Models. (arXiv:2311.16711v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16711
• Code URL: null
• Copy Paste: [[2311.16711]] LEDITS++: Limitless Image Editing using Text-to-Image Models(http://arxiv.org/abs/2311.16711)
• Summary:

  Text-to-image diffusion models have recently received increasing interest for their astonishing ability to produce high-fidelity images from solely text inputs. Subsequent research efforts aim to exploit and apply their capabilities to real image editing. However, existing image-to-image methods are often inefficient, imprecise, and of limited versatility. They either require time-consuming fine-tuning, deviate unnecessarily strongly from the input image, and/or lack support for multiple, simultaneous edits. To address these issues, we introduce LEDITS++, an efficient yet versatile and precise textual image manipulation technique. LEDITS++'s novel inversion approach requires no tuning nor optimization and produces high-fidelity results with a few diffusion steps. Second, our methodology supports multiple simultaneous edits and is architecture-agnostic. Third, we use a novel implicit masking technique that limits changes to relevant image regions. We propose the novel TEdBench++ benchmark as part of our exhaustive evaluation. Our results demonstrate the capabilities of LEDITS++ and its improvements over previous methods. The project page is available at https://leditsplusplus-project.static.hf.space .

Title: As-Plausible-As-Possible: Plausibility-Aware Mesh Deformation Using 2D Diffusion Priors. (arXiv:2311.16739v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16739
• Code URL: null
• Copy Paste: [[2311.16739]] As-Plausible-As-Possible: Plausibility-Aware Mesh Deformation Using 2D Diffusion Priors(http://arxiv.org/abs/2311.16739)
• Summary:

  We present As-Plausible-as-Possible (APAP) mesh deformation technique that leverages 2D diffusion priors to preserve the plausibility of a mesh under user-controlled deformation. Our framework uses per-face Jacobians to represent mesh deformations, where mesh vertex coordinates are computed via a differentiable Poisson Solve. The deformed mesh is rendered, and the resulting 2D image is used in the Score Distillation Sampling (SDS) process, which enables extracting meaningful plausibility priors from a pretrained 2D diffusion model. To better preserve the identity of the edited mesh, we fine-tune our 2D diffusion model with LoRA. Gradients extracted by SDS and a user-prescribed handle displacement are then backpropagated to the per-face Jacobians, and we use iterative gradient descent to compute the final deformation that balances between the user edit and the output plausibility. We evaluate our method with 2D and 3D meshes and demonstrate qualitative and quantitative improvements when using plausibility priors over geometry-preservation or distortion-minimization priors used by previous techniques.

Title: ChatTraffc: Text-to-Traffic Generation via Diffusion Model. (arXiv:2311.16203v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.16203
• Code URL: https://github.com/ChyaZhang/ChatTraffic
• Copy Paste: [[2311.16203]] ChatTraffc: Text-to-Traffic Generation via Diffusion Model(http://arxiv.org/abs/2311.16203)
• Summary:

  Traffic prediction is one of the most significant foundations in Intelligent Transportation Systems (ITS). Traditional traffic prediction methods rely only on historical traffic data to predict traffic trends and face two main challenges. 1) insensitivity to unusual events. 2) poor performance in long-term prediction. In this work, we explore how generative models combined with text describing the traffic system can be applied for traffic generation and name the task Text-to-Traffic Generation (TTG). The key challenge of the TTG task is how to associate text with the spatial structure of the road network and traffic data for generating traffic situations. To this end, we propose ChatTraffic, the first diffusion model for text-to-traffic generation. To guarantee the consistency between synthetic and real data, we augment a diffusion model with the Graph Convolutional Network (GCN) to extract spatial correlations of traffic data. In addition, we construct a large dataset containing text-traffic pairs for the TTG task. We benchmarked our model qualitatively and quantitatively on the released dataset. The experimental results indicate that ChatTraffic can generate realistic traffic situations from the text. Our code and dataset are available at https://github.com/ChyaZhang/ChatTraffic.

Title: Inexpensive High Fidelity Melt Pool Models in Additive Manufacturing Using Generative Deep Diffusion. (arXiv:2311.16168v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.16168
• Code URL: null
• Copy Paste: [[2311.16168]] Inexpensive High Fidelity Melt Pool Models in Additive Manufacturing Using Generative Deep Diffusion(http://arxiv.org/abs/2311.16168)
• Summary:

  Defects in laser powder bed fusion (L-PBF) parts often result from the meso-scale dynamics of the molten alloy near the laser, known as the melt pool. For instance, the melt pool can directly contribute to the formation of undesirable porosity, residual stress, and surface roughness in the final part. Experimental in-situ monitoring of the three-dimensional melt pool physical fields is challenging, due to the short length and time scales involved in the process. Multi-physics simulation methods can describe the three-dimensional dynamics of the melt pool, but are computationally expensive at the mesh refinement required for accurate predictions of complex effects, such as the formation of keyhole porosity. Therefore, in this work, we develop a generative deep learning model based on the probabilistic diffusion framework to map low-fidelity, coarse-grained simulation information to the high-fidelity counterpart. By doing so, we bypass the computational expense of conducting multiple high-fidelity simulations for analysis by instead upscaling lightweight coarse mesh simulations. Specifically, we implement a 2-D diffusion model to spatially upscale cross-sections of the coarsely simulated melt pool to their high-fidelity equivalent. We demonstrate the preservation of key metrics of the melting process between the ground truth simulation data and the diffusion model output, such as the temperature field, the melt pool dimensions and the variability of the keyhole vapor cavity. Specifically, we predict the melt pool depth within 3 $\mu m$ based on low-fidelity input data 4$\times$ coarser than the high-fidelity simulations, reducing analysis time by two orders of magnitude.

Title: Symphony: Symmetry-Equivariant Point-Centered Spherical Harmonics for Molecule Generation. (arXiv:2311.16199v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.16199
• Code URL: null
• Copy Paste: [[2311.16199]] Symphony: Symmetry-Equivariant Point-Centered Spherical Harmonics for Molecule Generation(http://arxiv.org/abs/2311.16199)
• Summary:

  We present Symphony, an $E(3)$-equivariant autoregressive generative model for 3D molecular geometries that iteratively builds a molecule from molecular fragments. Existing autoregressive models such as G-SchNet and G-SphereNet for molecules utilize rotationally invariant features to respect the 3D symmetries of molecules. In contrast, Symphony uses message-passing with higher-degree $E(3)$-equivariant features. This allows a novel representation of probability distributions via spherical harmonic signals to efficiently model the 3D geometry of molecules. We show that Symphony is able to accurately generate small molecules from the QM9 dataset, outperforming existing autoregressive models and approaching the performance of diffusion models.

Title: Personalized Predictions of Glioblastoma Infiltration: Mathematical Models, Physics-Informed Neural Networks and Multimodal Scans. (arXiv:2311.16536v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.16536
• Code URL: null
• Copy Paste: [[2311.16536]] Personalized Predictions of Glioblastoma Infiltration: Mathematical Models, Physics-Informed Neural Networks and Multimodal Scans(http://arxiv.org/abs/2311.16536)
• Summary:

  Predicting the infiltration of Glioblastoma (GBM) from medical MRI scans is crucial for understanding tumor growth dynamics and designing personalized radiotherapy treatment plans.Mathematical models of GBM growth can complement the data in the prediction of spatial distributions of tumor cells. However, this requires estimating patient-specific parameters of the model from clinical data, which is a challenging inverse problem due to limited temporal data and the limited time between imaging and diagnosis. This work proposes a method that uses Physics-Informed Neural Networks (PINNs) to estimate patient-specific parameters of a reaction-diffusion PDE model of GBM growth from a single 3D structural MRI snapshot. PINNs embed both the data and the PDE into a loss function, thus integrating theory and data. Key innovations include the identification and estimation of characteristic non-dimensional parameters, a pre-training step that utilizes the non-dimensional parameters and a fine-tuning step to determine the patient specific parameters. Additionally, the diffuse domain method is employed to handle the complex brain geometry within the PINN framework. Our method is validated both on synthetic and patient datasets, and shows promise for real-time parametric inference in the clinical setting for personalized GBM treatment.

noise learning

data-free

transformer

Title: Dual-Stream Attention Transformers for Sewer Defect Classification. (arXiv:2311.16145v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16145
• Code URL: null
• Copy Paste: [[2311.16145]] Dual-Stream Attention Transformers for Sewer Defect Classification(http://arxiv.org/abs/2311.16145)
• Summary:

  We propose a dual-stream multi-scale vision transformer (DS-MSHViT) architecture that processes RGB and optical flow inputs for efficient sewer defect classification. Unlike existing methods that combine the predictions of two separate networks trained on each modality, we jointly train a single network with two branches for RGB and motion. Our key idea is to use self-attention regularization to harness the complementary strengths of the RGB and motion streams. The motion stream alone struggles to generate accurate attention maps, as motion images lack the rich visual features present in RGB images. To facilitate this, we introduce an attention consistency loss between the dual streams. By leveraging motion cues through a self-attention regularizer, we align and enhance RGB attention maps, enabling the network to concentrate on pertinent input regions. We evaluate our data on a public dataset as well as cross-validate our model performance in a novel dataset. Our method outperforms existing models that utilize either convolutional neural networks (CNNs) or multi-scale hybrid vision transformers (MSHViTs) without employing attention regularization between the two streams.

Title: Vision Encoder-Decoder Models for AI Coaching. (arXiv:2311.16161v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16161
• Code URL: null
• Copy Paste: [[2311.16161]] Vision Encoder-Decoder Models for AI Coaching(http://arxiv.org/abs/2311.16161)
• Summary:

  This research paper introduces an innovative AI coaching approach by integrating vision-encoder-decoder models. The feasibility of this method is demonstrated using a Vision Transformer as the encoder and GPT-2 as the decoder, achieving a seamless integration of visual input and textual interaction. Departing from conventional practices of employing distinct models for image recognition and text-based coaching, our integrated architecture directly processes input images, enabling natural question-and-answer dialogues with the AI coach. This unique strategy simplifies model architecture while enhancing the overall user experience in human-AI interactions. We showcase sample results to demonstrate the capability of the model. The results underscore the methodology's potential as a promising paradigm for creating efficient AI coach models in various domains involving visual inputs. Importantly, this potential holds true regardless of the particular visual encoder or text decoder chosen. Additionally, we conducted experiments with different sizes of GPT-2 to assess the impact on AI coach performance, providing valuable insights into the scalability and versatility of our proposed methodology.

Title: Aligning Non-Causal Factors for Transformer-Based Source-Free Domain Adaptation. (arXiv:2311.16294v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16294
• Code URL: null
• Copy Paste: [[2311.16294]] Aligning Non-Causal Factors for Transformer-Based Source-Free Domain Adaptation(http://arxiv.org/abs/2311.16294)
• Summary:

  Conventional domain adaptation algorithms aim to achieve better generalization by aligning only the task-discriminative causal factors between a source and target domain. However, we find that retaining the spurious correlation between causal and non-causal factors plays a vital role in bridging the domain gap and improving target adaptation. Therefore, we propose to build a framework that disentangles and supports causal factor alignment by aligning the non-causal factors first. We also investigate and find that the strong shape bias of vision transformers, coupled with its multi-head attention, make it a suitable architecture for realizing our proposed disentanglement. Hence, we propose to build a Causality-enforcing Source-Free Transformer framework (C-SFTrans) to achieve disentanglement via a novel two-stage alignment approach: a) non-causal factor alignment: non-causal factors are aligned using a style classification task which leads to an overall global alignment, b) task-discriminative causal factor alignment: causal factors are aligned via target adaptation. We are the first to investigate the role of vision transformers (ViTs) in a privacy-preserving source-free setting. Our approach achieves state-of-the-art results in several DA benchmarks.

Title: Typhoon Intensity Prediction with Vision Transformer. (arXiv:2311.16450v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16450
• Code URL: https://github.com/chen-huanxin/tint
• Copy Paste: [[2311.16450]] Typhoon Intensity Prediction with Vision Transformer(http://arxiv.org/abs/2311.16450)
• Summary:

  Predicting typhoon intensity accurately across space and time is crucial for issuing timely disaster warnings and facilitating emergency response. This has vast potential for minimizing life losses and property damages as well as reducing economic and environmental impacts. Leveraging satellite imagery for scenario analysis is effective but also introduces additional challenges due to the complex relations among clouds and the highly dynamic context. Existing deep learning methods in this domain rely on convolutional neural networks (CNNs), which suffer from limited per-layer receptive fields. This limitation hinders their ability to capture long-range dependencies and global contextual knowledge during inference. In response, we introduce a novel approach, namely "Typhoon Intensity Transformer" (Tint), which leverages self-attention mechanisms with global receptive fields per layer. Tint adopts a sequence-to-sequence feature representation learning perspective. It begins by cutting a given satellite image into a sequence of patches and recursively employs self-attention operations to extract both local and global contextual relations between all patch pairs simultaneously, thereby enhancing per-patch feature representation learning. Extensive experiments on a publicly available typhoon benchmark validate the efficacy of Tint in comparison with both state-of-the-art deep learning and conventional meteorological methods. Our code is available at https://github.com/chen-huanxin/Tint.

Title: Spiking Neural Networks with Dynamic Time Steps for Vision Transformers. (arXiv:2311.16456v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16456
• Code URL: null
• Copy Paste: [[2311.16456]] Spiking Neural Networks with Dynamic Time Steps for Vision Transformers(http://arxiv.org/abs/2311.16456)
• Summary:

  Spiking Neural Networks (SNNs) have emerged as a popular spatio-temporal computing paradigm for complex vision tasks. Recently proposed SNN training algorithms have significantly reduced the number of time steps (down to 1) for improved latency and energy efficiency, however, they target only convolutional neural networks (CNN). These algorithms, when applied on the recently spotlighted vision transformers (ViT), either require a large number of time steps or fail to converge. Based on analysis of the histograms of the ANN and SNN activation maps, we hypothesize that each ViT block has a different sensitivity to the number of time steps. We propose a novel training framework that dynamically allocates the number of time steps to each ViT module depending on a trainable score assigned to each timestep. In particular, we generate a scalar binary time step mask that filters spikes emitted by each neuron in a leaky-integrate-and-fire (LIF) layer. The resulting SNNs have high activation sparsity and require only accumulate operations (AC), except for the input embedding layer, in contrast to expensive multiply-and-accumulates (MAC) needed in traditional ViTs. This yields significant improvements in energy efficiency. We evaluate our training framework and resulting SNNs on image recognition tasks including CIFAR10, CIFAR100, and ImageNet with different ViT architectures. We obtain a test accuracy of 95.97% with 4.97 time steps with direct encoding on CIFAR10.

Title: Bridging the Gap: A Unified Video Comprehension Framework for Moment Retrieval and Highlight Detection. (arXiv:2311.16464v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16464
• Code URL: https://github.com/easonxiao-888/uvcom
• Copy Paste: [[2311.16464]] Bridging the Gap: A Unified Video Comprehension Framework for Moment Retrieval and Highlight Detection(http://arxiv.org/abs/2311.16464)
• Summary:

  Video Moment Retrieval (MR) and Highlight Detection (HD) have attracted significant attention due to the growing demand for video analysis. Recent approaches treat MR and HD as similar video grounding problems and address them together with transformer-based architecture. However, we observe that the emphasis of MR and HD differs, with one necessitating the perception of local relationships and the other prioritizing the understanding of global contexts. Consequently, the lack of task-specific design will inevitably lead to limitations in associating the intrinsic specialty of two tasks. To tackle the issue, we propose a Unified Video COMprehension framework (UVCOM) to bridge the gap and jointly solve MR and HD effectively. By performing progressive integration on intra and inter-modality across multi-granularity, UVCOM achieves the comprehensive understanding in processing a video. Moreover, we present multi-aspect contrastive learning to consolidate the local relation modeling and global knowledge accumulation via well aligned multi-modal space. Extensive experiments on QVHighlights, Charades-STA, TACoS , YouTube Highlights and TVSum datasets demonstrate the effectiveness and rationality of UVCOM which outperforms the state-of-the-art methods by a remarkable margin.

Title: Generating Human-Centric Visual Cues for Human-Object Interaction Detection via Large Vision-Language Models. (arXiv:2311.16475v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16475
• Code URL: null
• Copy Paste: [[2311.16475]] Generating Human-Centric Visual Cues for Human-Object Interaction Detection via Large Vision-Language Models(http://arxiv.org/abs/2311.16475)
• Summary:

  Human-object interaction (HOI) detection aims at detecting human-object pairs and predicting their interactions. However, the complexity of human behavior and the diverse contexts in which these interactions occur make it challenging. Intuitively, human-centric visual cues, such as the involved participants, the body language, and the surrounding environment, play crucial roles in shaping these interactions. These cues are particularly vital in interpreting unseen interactions. In this paper, we propose three prompts with VLM to generate human-centric visual cues within an image from multiple perspectives of humans. To capitalize on these rich Human-Centric Visual Cues, we propose a novel approach named HCVC for HOI detection. Particularly, we develop a transformer-based multimodal fusion module with multitower architecture to integrate visual cue features into the instance and interaction decoders. Our extensive experiments and analysis validate the efficacy of leveraging the generated human-centric visual cues for HOI detection. Notably, the experimental results indicate the superiority of the proposed model over the existing state-of-the-art methods on two widely used datasets.

Title: Eye vs. AI: Human Gaze and Model Attention in Video Memorability. (arXiv:2311.16484v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16484
• Code URL: null
• Copy Paste: [[2311.16484]] Eye vs(http://arxiv.org/abs/2311.16484)
• Summary:

  Understanding the factors that determine video memorability has important applications in areas such as educational technology and advertising. Towards this goal, we investigate the semantic and temporal attention mechanisms underlying video memorability. We propose a Transformer-based model with spatio-temporal attention that matches SoTA performance on video memorability prediction on a large naturalistic video dataset. More importantly, the self-attention patterns show us where the model looks to predict memorability. We compare model attention against human gaze fixation density maps collected through a small-scale eye-tracking experiment where humans perform a video memory task. Quantitative saliency metrics show that the model attention and human gaze follow similar patterns. Furthermore, while panoptic segmentation confirms that the model and humans attend more to thing classes, stuff classes that receive increased/decreased attention tend to have higher memorability scores. We also observe that the model assigns greater importance to the initial frames, mimicking temporal attention patterns found in humans.

Title: On the Long Range Abilities of Transformers. (arXiv:2311.16620v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.16620
• Code URL: null
• Copy Paste: [[2311.16620]] On the Long Range Abilities of Transformers(http://arxiv.org/abs/2311.16620)
• Summary:

  Despite their dominance in modern DL and, especially, NLP domains, transformer architectures exhibit sub-optimal performance on long-range tasks compared to recent layers that are specifically designed for this purpose. In this work, drawing inspiration from key attributes of long-range layers, such as state-space layers, linear RNN layers, and global convolution layers, we demonstrate that minimal modifications to the transformer architecture can significantly enhance performance on the Long Range Arena (LRA) benchmark, thus narrowing the gap with these specialized layers. We identify that two key principles for long-range tasks are (i) incorporating an inductive bias towards smoothness, and (ii) locality. As we show, integrating these ideas into the attention mechanism improves results with a negligible amount of additional computation and without any additional trainable parameters. Our theory and experiments also shed light on the reasons for the inferior performance of transformers on long-range tasks and identify critical properties that are essential for successfully capturing long-range dependencies.

generative

Title: Semantic Generative Augmentations for Few-Shot Counting. (arXiv:2311.16122v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16122
• Code URL: null
• Copy Paste: [[2311.16122]] Semantic Generative Augmentations for Few-Shot Counting(http://arxiv.org/abs/2311.16122)
• Summary:

  With the availability of powerful text-to-image diffusion models, recent works have explored the use of synthetic data to improve image classification performances. These works show that it can effectively augment or even replace real data. In this work, we investigate how synthetic data can benefit few-shot class-agnostic counting. This requires to generate images that correspond to a given input number of objects. However, text-to-image models struggle to grasp the notion of count. We propose to rely on a double conditioning of Stable Diffusion with both a prompt and a density map in order to augment a training dataset for few-shot counting. Due to the small dataset size, the fine-tuned model tends to generate images close to the training images. We propose to enhance the diversity of synthesized images by exchanging captions between images thus creating unseen configurations of object types and spatial layout. Our experiments show that our diversified generation strategy significantly improves the counting accuracy of two recent and performing few-shot counting models on FSC147 and CARPK.

Title: RelVAE: Generative Pretraining for few-shot Visual Relationship Detection. (arXiv:2311.16261v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16261
• Code URL: null
• Copy Paste: [[2311.16261]] RelVAE: Generative Pretraining for few-shot Visual Relationship Detection(http://arxiv.org/abs/2311.16261)
• Summary:

  Visual relations are complex, multimodal concepts that play an important role in the way humans perceive the world. As a result of their complexity, high-quality, diverse and large scale datasets for visual relations are still absent. In an attempt to overcome this data barrier, we choose to focus on the problem of few-shot Visual Relationship Detection (VRD), a setting that has been so far neglected by the community. In this work we present the first pretraining method for few-shot predicate classification that does not require any annotated relations. We achieve this by introducing a generative model that is able to capture the variation of semantic, visual and spatial information of relations inside a latent space and later exploiting its representations in order to achieve efficient few-shot classification. We construct few-shot training splits and show quantitative experiments on VG200 and VRD datasets where our model outperforms the baselines. Lastly we attempt to interpret the decisions of the model by conducting various qualitative experiments.

Title: PISA: Point-cloud-based Instructed Scene Augmentation. (arXiv:2311.16501v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16501
• Code URL: null
• Copy Paste: [[2311.16501]] PISA: Point-cloud-based Instructed Scene Augmentation(http://arxiv.org/abs/2311.16501)
• Summary:

  Indoor scene augmentation has become an emerging topic in the field of computer vision with applications in augmented and virtual reality. However, existing scene augmentation methods mostly require a pre-built object database with a given position as the desired location. In this paper, we propose the first end-to-end multi-modal deep neural network that can generate point cloud objects consistent with their surroundings, conditioned on text instructions. Our model generates a seemly object in the appropriate position based on the inputs of a query and point clouds, thereby enabling the creation of new scenarios involving previously unseen layouts of objects. Database of pre-stored CAD models is no longer needed. We use Point-E as our generative model and introduce methods including quantified position prediction and Top-K estimation to mitigate the false negative problems caused by ambiguous language description. Moreover, we evaluate the ability of our model by demonstrating the diversity of generated objects, the effectiveness of instruction, and quantitative metric results, which collectively indicate that our model is capable of generating realistic in-door objects. For a more thorough evaluation, we also incorporate visual grounding as a metric to assess the quality of the scenes generated by our model.

Title: MedGen: A Python Natural Language Processing Toolkit for Medical Text Processing. (arXiv:2311.16588v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.16588
• Code URL: https://github.com/yale-lily/medgen
• Copy Paste: [[2311.16588]] MedGen: A Python Natural Language Processing Toolkit for Medical Text Processing(http://arxiv.org/abs/2311.16588)
• Summary:

  This study introduces MedGen, a comprehensive natural language processing (NLP) toolkit designed for medical text processing. MedGen is tailored for biomedical researchers and healthcare professionals with an easy-to-use, all-in-one solution that requires minimal programming expertise. It includes (1) Generative Functions: For the first time, MedGen includes four advanced generative functions: question answering, text summarization, text simplification, and machine translation; (2) Basic NLP Functions: MedGen integrates 12 essential NLP functions such as word tokenization and sentence segmentation; and (3) Query and Search Capabilities: MedGen provides user-friendly query and search functions on text corpora. We fine-tuned 32 domain-specific language models, evaluated them thoroughly on 24 established benchmarks and conducted manual reviews with clinicians. Additionally, we expanded our toolkit by introducing query and search functions, while also standardizing and integrating functions from third-party libraries. The toolkit, its models, and associated data are publicly available via https://github.com/Yale-LILY/MedGen.

large language model

Title: Removing NSFW Concepts from Vision-and-Language Models for Text-to-Image Retrieval and Generation. (arXiv:2311.16254v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16254
• Code URL: https://github.com/aimagelab/safe-clip
• Copy Paste: [[2311.16254]] Removing NSFW Concepts from Vision-and-Language Models for Text-to-Image Retrieval and Generation(http://arxiv.org/abs/2311.16254)
• Summary:

  Vision-and-Language models such as CLIP have demonstrated remarkable effectiveness across a wide range of tasks. However, these models are typically trained on web-scale data, which can introduce inappropriate content and lead to the development of unsafe and biased behavior. This, in turn, hampers their applicability in sensitive and trustworthy contexts and could raise significant concern in their adoption. To overcome these limitations, we introduce a methodology to make Vision-and-Language models safer by removing their sensitivity to not-safe-for-work concepts. We show how this can be done by distilling from a large language model which converts between safe and unsafe sentences and which is fine-tuned starting from just 100 manually-curated pairs. We conduct extensive experiments on the resulting embedding space for both retrieval and text-to-image generation, where we show that our model can also be properly employed with pre-trained image generators. Our source code and trained models are available at: https://github.com/aimagelab/safe-clip.

Title: AvatarGPT: All-in-One Framework for Motion Understanding, Planning, Generation and Beyond. (arXiv:2311.16468v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16468
• Code URL: null
• Copy Paste: [[2311.16468]] AvatarGPT: All-in-One Framework for Motion Understanding, Planning, Generation and Beyond(http://arxiv.org/abs/2311.16468)
• Summary:

  Large Language Models(LLMs) have shown remarkable emergent abilities in unifying almost all (if not every) NLP tasks. In the human motion-related realm, however, researchers still develop siloed models for each task. Inspired by InstuctGPT, and the generalist concept behind Gato, we introduce AvatarGPT, an All-in-One framework for motion understanding, planning, generations as well as other tasks such as motion in-between synthesis. AvatarGPT treats each task as one type of instruction fine-tuned on the shared LLM. All the tasks are seamlessly interconnected with language as the universal interface, constituting a closed-loop within the framework. To achieve this, human motion sequences are first encoded as discrete tokens, which serve as the extended vocabulary of LLM. Then, an unsupervised pipeline to generate natural language descriptions of human action sequences from in-the-wild videos is developed. Finally, all tasks are jointly trained. Extensive experiments show that AvatarGPT achieves SOTA on low-level tasks, and promising results on high-level tasks, demonstrating the effectiveness of our proposed All-in-One framework. Moreover, for the first time, AvatarGPT enables a principled approach by iterative traversal of the tasks within the closed-loop for unlimited long-motion synthesis.

Title: VLPrompt: Vision-Language Prompting for Panoptic Scene Graph Generation. (arXiv:2311.16492v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16492
• Code URL: null
• Copy Paste: [[2311.16492]] VLPrompt: Vision-Language Prompting for Panoptic Scene Graph Generation(http://arxiv.org/abs/2311.16492)
• Summary:

  Panoptic Scene Graph Generation (PSG) aims at achieving a comprehensive image understanding by simultaneously segmenting objects and predicting relations among objects. However, the long-tail problem among relations leads to unsatisfactory results in real-world applications. Prior methods predominantly rely on vision information or utilize limited language information, such as object or relation names, thereby overlooking the utility of language information. Leveraging the recent progress in Large Language Models (LLMs), we propose to use language information to assist relation prediction, particularly for rare relations. To this end, we propose the Vision-Language Prompting (VLPrompt) model, which acquires vision information from images and language information from LLMs. Then, through a prompter network based on attention mechanism, it achieves precise relation prediction. Our extensive experiments show that VLPrompt significantly outperforms previous state-of-the-art methods on the PSG dataset, proving the effectiveness of incorporating language information and alleviating the long-tail problem of relations.

Title: LLMGA: Multimodal Large Language Model based Generation Assistant. (arXiv:2311.16500v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16500
• Code URL: null
• Copy Paste: [[2311.16500]] LLMGA: Multimodal Large Language Model based Generation Assistant(http://arxiv.org/abs/2311.16500)
• Summary:

  In this paper, we introduce a Multimodal Large Language Model-based Generation Assistant (LLMGA), leveraging the vast reservoir of knowledge and proficiency in reasoning, comprehension, and response inherent in Large Language Models (LLMs) to assist users in image generation and editing. Diverging from existing approaches where Multimodal Large Language Models (MLLMs) generate fixed-size embeddings to control Stable Diffusion (SD), our LLMGA provides a detailed language generation prompt for precise control over SD. This not only augments LLM context understanding but also reduces noise in generation prompts, yields images with more intricate and precise content, and elevates the interpretability of the network. To this end, we curate a comprehensive dataset comprising prompt refinement, similar image generation, inpainting $\&$ outpainting, and visual question answering. Moreover, we propose a two-stage training scheme. In the first stage, we train the MLLM to grasp the properties of image generation and editing, enabling it to generate detailed prompts. In the second stage, we optimize SD to align with the MLLM's generation prompts. Additionally, we propose a reference-based restoration network to alleviate texture, brightness, and contrast disparities between generated and preserved regions during image editing. Extensive results show that LLMGA has promising generative capabilities and can enable wider applications in an interactive manner.

Title: GPT4Video: A Unified Multimodal Large Language Model for lnstruction-Followed Understanding and Safety-Aware Generation. (arXiv:2311.16511v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16511
• Code URL: null
• Copy Paste: [[2311.16511]] GPT4Video: A Unified Multimodal Large Language Model for lnstruction-Followed Understanding and Safety-Aware Generation(http://arxiv.org/abs/2311.16511)
• Summary:

  While the recent advances in Multimodal Large Language Models (MLLMs) constitute a significant leap forward in the field, these models are predominantly confined to the realm of input-side multimodal comprehension, lacking the capacity for multimodal content generation. To fill this gap, we present GPT4Video, a unified multi-model framework that empowers Large Language Models (LLMs) with the capability of both video understanding and generation. Specifically, we develop an instruction-following-based approach integrated with the stable diffusion generative model, which has demonstrated to effectively and securely handle video generation scenarios. GPT4Video offers the following benefits: 1) It exhibits impressive capabilities in both video understanding and generation scenarios. For example, GPT4Video outperforms Valley by 11.8\% on the Video Question Answering task, and surpasses NExt-GPT by 2.3\% on the Text to Video generation task. 2) it endows the LLM/MLLM with video generation capabilities without requiring additional training parameters and can flexibly interface with a wide range of models to perform video generation. 3) it maintains a safe and healthy conversation not only in output-side but also the input side in an end-to-end manner. Qualitative and qualitative experiments demonstrate that GPT4Video holds the potential to function as a effective, safe and Humanoid-like video assistant that can handle both video understanding and generation scenarios.

Title: Large Language Models Meet Computer Vision: A Brief Survey. (arXiv:2311.16673v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16673
• Code URL: null
• Copy Paste: [[2311.16673]] Large Language Models Meet Computer Vision: A Brief Survey(http://arxiv.org/abs/2311.16673)
• Summary:

  Recently, the intersection of Large Language Models (LLMs) and Computer Vision (CV) has emerged as a pivotal area of research, driving significant advancements in the field of Artificial Intelligence (AI). As transformers have become the backbone of many state-of-the-art models in both Natural Language Processing (NLP) and CV, understanding their evolution and potential enhancements is crucial. This survey paper delves into the latest progressions in the domain of transformers and their subsequent successors, emphasizing their potential to revolutionize Vision Transformers (ViTs) and LLMs. This survey also presents a comparative analysis, juxtaposing the performance metrics of several leading paid and open-source LLMs, shedding light on their strengths and areas of improvement as well as a literature review on how LLMs are being used to tackle vision related tasks. Furthermore, the survey presents a comprehensive collection of datasets employed to train LLMs, offering insights into the diverse data available to achieve high performance in various pre-training and downstream tasks of LLMs. The survey is concluded by highlighting open directions in the field, suggesting potential venues for future research and development. This survey aims to underscores the profound intersection of LLMs on CV, leading to a new era of integrated and advanced AI models.

Title: Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorld. (arXiv:2311.16714v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16714
• Code URL: https://github.com/stevenyangyj/emma-alfworld
• Copy Paste: [[2311.16714]] Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorld(http://arxiv.org/abs/2311.16714)
• Summary:

  While large language models (LLMs) excel in a simulated world of texts, they struggle to interact with the more realistic world without perceptions of other modalities such as visual or audio signals. Although vision-language models (VLMs) integrate LLM modules (1) aligned with static image features, and (2) may possess prior knowledge of world dynamics (as demonstrated in the text world), they have not been trained in an embodied visual world and thus cannot align with its dynamics. On the other hand, training an embodied agent in a noisy visual world without expert guidance is often challenging and inefficient. In this paper, we train a VLM agent living in a visual world using an LLM agent excelling in a parallel text world (but inapplicable to the visual world). Specifically, we distill LLM's reflection outcomes (improved actions by analyzing mistakes) in a text world's tasks to finetune the VLM on the same tasks of the visual world, resulting in an Embodied Multi-Modal Agent (EMMA) quickly adapting to the visual world dynamics. Such cross-modality imitation learning between the two parallel worlds enables EMMA to generalize to a broad scope of new tasks without any further guidance from the LLM expert. Extensive evaluations on the ALFWorld benchmark highlight EMMA's superior performance to SOTA VLM-based agents across diverse tasks, e.g., 20%-70% improvement in the success rate.

Title: Enhancing Sentiment Analysis Results through Outlier Detection Optimization. (arXiv:2311.16185v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.16185
• Code URL: null
• Copy Paste: [[2311.16185]] Enhancing Sentiment Analysis Results through Outlier Detection Optimization(http://arxiv.org/abs/2311.16185)
• Summary:

  When dealing with text data containing subjective labels like speaker emotions, inaccuracies or discrepancies among labelers are not uncommon. Such discrepancies can significantly affect the performance of machine learning algorithms. This study investigates the potential of identifying and addressing outliers in text data with subjective labels, aiming to enhance classification outcomes. We utilized the Deep SVDD algorithm, a one-class classification method, to detect outliers in nine text-based emotion and sentiment analysis datasets. By employing both a small-sized language model (DistilBERT base model with 66 million parameters) and non-deep learning machine learning algorithms (decision tree, KNN, Logistic Regression, and LDA) as the classifier, our findings suggest that the removal of outliers can lead to enhanced results in most cases. Additionally, as outliers in such datasets are not necessarily unlearnable, we experienced utilizing a large language model -- DeBERTa v3 large with 131 million parameters, which can capture very complex patterns in data. We continued to observe performance enhancements across multiple datasets.

Title: Applications of Large Language Models in Data Processing: Innovative Approaches to Segmenting and Renewing Information. (arXiv:2311.16267v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.16267
• Code URL: null
• Copy Paste: [[2311.16267]] Applications of Large Language Models in Data Processing: Innovative Approaches to Segmenting and Renewing Information(http://arxiv.org/abs/2311.16267)
• Summary:

  Our paper investigates effective methods for code generation in "specific-domain" applications, including the use of Large Language Models (LLMs) for data segmentation and renewal, as well as stimulating deeper thinking in LLMs through prompt adjustments. Using a real company product as an example, we provide user manuals, API documentation, and other data. The ideas discussed in this paper help segment and then convert this data into semantic vectors to better reflect their true positioning. Subsequently, user requirements are transformed into vectors to retrieve the most relevant content, achieving about 70% accuracy in simple to medium-complexity tasks through various prompt techniques. This paper is the first to enhance specific-domain code generation effectiveness from this perspective. Additionally, we experiment with generating more scripts from a limited number using llama2-based fine-tuning to test its effectiveness in professional domain code generation. This is a challenging and promising field, and once achieved, it will not only lead to breakthroughs in LLM development across multiple industries but also enable LLMs to understand and learn any new knowledge effectively.

Title: CDEval: A Benchmark for Measuring the Cultural Dimensions of Large Language Models. (arXiv:2311.16421v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.16421
• Code URL: https://github.com/astrodrew/cdeval
• Copy Paste: [[2311.16421]] CDEval: A Benchmark for Measuring the Cultural Dimensions of Large Language Models(http://arxiv.org/abs/2311.16421)
• Summary:

  As the scaling of Large Language Models (LLMs) has dramatically enhanced their capabilities, there has been a growing focus on the alignment problem to ensure their responsible and ethical use. While existing alignment efforts predominantly concentrate on universal values such as the HHH principle, the aspect of culture, which is inherently pluralistic and diverse, has not received adequate attention. This work introduces a new benchmark, CDEval, aimed at evaluating the cultural dimensions of LLMs. CDEval is constructed by incorporating both GPT-4's automated generation and human verification, covering six cultural dimensions across seven domains. Our comprehensive experiments provide intriguing insights into the culture of mainstream LLMs, highlighting both consistencies and variations across different dimensions and domains. The findings underscore the importance of integrating cultural considerations in LLM development, particularly for applications in diverse cultural settings. Through CDEval, we aim to broaden the horizon of LLM alignment research by including cultural dimensions, thus providing a more holistic framework for the future development and evaluation of LLMs. This benchmark serves as a valuable resource for cultural studies in LLMs, paving the way for more culturally aware and sensitive models.

Title: StyleCap: Automatic Speaking-Style Captioning from Speech Based on Speech and Language Self-supervised Learning Models. (arXiv:2311.16509v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.16509
• Code URL: null
• Copy Paste: [[2311.16509]] StyleCap: Automatic Speaking-Style Captioning from Speech Based on Speech and Language Self-supervised Learning Models(http://arxiv.org/abs/2311.16509)
• Summary:

  We propose StyleCap, a method to generate natural language descriptions of speaking styles appearing in speech. Although most of conventional techniques for para-/non-linguistic information recognition focus on the category classification or the intensity estimation of pre-defined labels, they cannot provide the reasoning of the recognition result in an interpretable manner. As a first step towards an end-to-end method for generating speaking-style prompts from speech, i.e., automatic speaking-style captioning, StyleCap uses paired data of speech and natural language descriptions to train neural networks that predict prefix vectors fed into a large language model (LLM)-based text decoder from a speech representation vector. We explore an appropriate text decoder and speech feature representation suitable for this new task. The experimental results demonstrate that our StyleCap leveraging richer LLMs for the text decoder, speech self-supervised learning (SSL) features, and sentence rephrasing augmentation improves the accuracy and diversity of generated speaking-style captions. Samples of speaking-style captions generated by our StyleCap are publicly available.

Title: Enabling Fast 2-bit LLM on GPUs: Memory Alignment, Sparse Outlier, and Asynchronous Dequantization. (arXiv:2311.16442v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.16442
• Code URL: null
• Copy Paste: [[2311.16442]] Enabling Fast 2-bit LLM on GPUs: Memory Alignment, Sparse Outlier, and Asynchronous Dequantization(http://arxiv.org/abs/2311.16442)
• Summary:

  Large language models (LLMs) have demonstrated impressive abilities in various domains while the inference cost is expensive. The state-of-the-art methods use 2-bit quantization for mainstream LLMs. However, challenges still exist: (1) Nonnegligible accuracy loss for 2-bit quantization. Weights are quantized by groups, while the ranges of weights are large in some groups, resulting in large quantization errors and nonnegligible accuracy loss (e.g. >3% for Llama2-7b with 2-bit quantization in GPTQ and Greenbit). (2) Limited accuracy improvement by adding 4-bit weights. Increasing 10% extra average bit more 4-bit weights only leads to <0.5% accuracy improvement on a quantized Llama2-7b. (3) Time-consuming dequantization operations on GPUs. The dequantization operations lead to >50% execution time, hindering the potential of reducing LLM inference cost. To tackle these challenges, we propose the following techniques: (1) We only quantize a small fraction of groups with the larger range using 4-bit with memory alignment consideration on GPUs. (2) We point out that the distribution of the sparse outliers with larger weights is different in 2-bit and 4-bit groups, and only a small fraction of outliers require 16-bit quantization. Such design leads to >0.5% accuracy improvement with <3% average increased bit for Llama2-7b. (3) We design the asynchronous dequantization on GPUs, leading to up to 3.92X speedup. We conduct extensive experiments on different model families and model sizes. We achieve 2.85-bit for each weight and the end-to-end speedup for Llama2-7b is 1.74X over the original model, and we reduce both runtime cost and hardware cost by up to 2.70X and 2.81X with less GPU requirements.

segmentation

Title: Adapting Segment Anything Model (SAM) through Prompt-based Learning for Enhanced Protein Identification in Cryo-EM Micrographs. (arXiv:2311.16140v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16140
• Code URL: https://github.com/yangyang-69/Prompt_sam_cryoPPP
• Copy Paste: [[2311.16140]] Adapting Segment Anything Model (SAM) through Prompt-based Learning for Enhanced Protein Identification in Cryo-EM Micrographs(http://arxiv.org/abs/2311.16140)
• Summary:

  Cryo-electron microscopy (cryo-EM) remains pivotal in structural biology, yet the task of protein particle picking, integral for 3D protein structure construction, is laden with manual inefficiencies. While recent AI tools such as Topaz and crYOLO are advancing the field, they do not fully address the challenges of cryo-EM images, including low contrast, complex shapes, and heterogeneous conformations. This study explored prompt-based learning to adapt the state-of-the-art image segmentation foundation model Segment Anything Model (SAM) for cryo-EM. This focus was driven by the desire to optimize model performance with a small number of labeled data without altering pre-trained parameters, aiming for a balance between adaptability and foundational knowledge retention. Through trials with three prompt-based learning strategies, namely head prompt, prefix prompt, and encoder prompt, we observed enhanced performance and reduced computational requirements compared to the fine-tuning approach. This work not only highlights the potential of prompting SAM in protein identification from cryo-EM micrographs but also suggests its broader promise in biomedical image segmentation and object detection.

Title: SemiVL: Semi-Supervised Semantic Segmentation with Vision-Language Guidance. (arXiv:2311.16241v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16241
• Code URL: https://github.com/google-research/semivl
• Copy Paste: [[2311.16241]] SemiVL: Semi-Supervised Semantic Segmentation with Vision-Language Guidance(http://arxiv.org/abs/2311.16241)
• Summary:

  In semi-supervised semantic segmentation, a model is trained with a limited number of labeled images along with a large corpus of unlabeled images to reduce the high annotation effort. While previous methods are able to learn good segmentation boundaries, they are prone to confuse classes with similar visual appearance due to the limited supervision. On the other hand, vision-language models (VLMs) are able to learn diverse semantic knowledge from image-caption datasets but produce noisy segmentation due to the image-level training. In SemiVL, we propose to integrate rich priors from VLM pre-training into semi-supervised semantic segmentation to learn better semantic decision boundaries. To adapt the VLM from global to local reasoning, we introduce a spatial fine-tuning strategy for label-efficient learning. Further, we design a language-guided decoder to jointly reason over vision and language. Finally, we propose to handle inherent ambiguities in class labels by providing the model with language guidance in the form of class definitions. We evaluate SemiVL on 4 semantic segmentation datasets, where it significantly outperforms previous semi-supervised methods. For instance, SemiVL improves the state-of-the-art by +13.5 mIoU on COCO with 232 annotated images and by +6.1 mIoU on Pascal VOC with 92 labels. Project page: https://github.com/google-research/semivl

Title: Segment Every Out-of-Distribution Object. (arXiv:2311.16516v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16516
• Code URL: null
• Copy Paste: [[2311.16516]] Segment Every Out-of-Distribution Object(http://arxiv.org/abs/2311.16516)
• Summary:

  Semantic segmentation models, while effective for in-distribution categories, face challenges in real-world deployment due to encountering out-of-distribution (OoD) objects. Detecting these OoD objects is crucial for safety-critical applications. Existing methods rely on anomaly scores, but choosing a suitable threshold for generating masks presents difficulties and can lead to fragmentation and inaccuracy. This paper introduces a method to convert anomaly Score To segmentation Mask, called S2M, a simple and effective framework for OoD detection in semantic segmentation. Unlike assigning anomaly scores to pixels, S2M directly segments the entire OoD object. By transforming anomaly scores into prompts for a promptable segmentation model, S2M eliminates the need for threshold selection. Extensive experiments demonstrate that S2M outperforms the state-of-the-art by approximately 10\% in IoU and 30\% in mean F1 score, on average, across various benchmarks including Fishyscapes, Segment-Me-If-You-Can, and RoadAnomaly datasets.

Title: 3D Teeth Reconstruction from Panoramic Radiographs using Neural Implicit Functions. (arXiv:2311.16524v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16524
• Code URL: null
• Copy Paste: [[2311.16524]] 3D Teeth Reconstruction from Panoramic Radiographs using Neural Implicit Functions(http://arxiv.org/abs/2311.16524)
• Summary:

  Panoramic radiography is a widely used imaging modality in dental practice and research. However, it only provides flattened 2D images, which limits the detailed assessment of dental structures. In this paper, we propose Occudent, a framework for 3D teeth reconstruction from panoramic radiographs using neural implicit functions, which, to the best of our knowledge, is the first work to do so. For a given point in 3D space, the implicit function estimates whether the point is occupied by a tooth, and thus implicitly determines the boundaries of 3D tooth shapes. Firstly, Occudent applies multi-label segmentation to the input panoramic radiograph. Next, tooth shape embeddings as well as tooth class embeddings are generated from the segmentation outputs, which are fed to the reconstruction network. A novel module called Conditional eXcitation (CX) is proposed in order to effectively incorporate the combined shape and class embeddings into the implicit function. The performance of Occudent is evaluated using both quantitative and qualitative measures. Importantly, Occudent is trained and validated with actual panoramic radiographs as input, distinct from recent works which used synthesized images. Experiments demonstrate the superiority of Occudent over state-of-the-art methods.

Title: HandyPriors: Physically Consistent Perception of Hand-Object Interactions with Differentiable Priors. (arXiv:2311.16552v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16552
• Code URL: null
• Copy Paste: [[2311.16552]] HandyPriors: Physically Consistent Perception of Hand-Object Interactions with Differentiable Priors(http://arxiv.org/abs/2311.16552)
• Summary:

  Various heuristic objectives for modeling hand-object interaction have been proposed in past work. However, due to the lack of a cohesive framework, these objectives often possess a narrow scope of applicability and are limited by their efficiency or accuracy. In this paper, we propose HandyPriors, a unified and general pipeline for pose estimation in human-object interaction scenes by leveraging recent advances in differentiable physics and rendering. Our approach employs rendering priors to align with input images and segmentation masks along with physics priors to mitigate penetration and relative-sliding across frames. Furthermore, we present two alternatives for hand and object pose estimation. The optimization-based pose estimation achieves higher accuracy, while the filtering-based tracking, which utilizes the differentiable priors as dynamics and observation models, executes faster. We demonstrate that HandyPriors attains comparable or superior results in the pose estimation task, and that the differentiable physics module can predict contact information for pose refinement. We also show that our approach generalizes to perception tasks, including robotic hand manipulation and human-object pose estimation in the wild.

Title: Clean Label Disentangling for Medical Image Segmentation with Noisy Labels. (arXiv:2311.16580v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16580
• Code URL: https://github.com/xiaoyao3302/2bdenoise
• Copy Paste: [[2311.16580]] Clean Label Disentangling for Medical Image Segmentation with Noisy Labels(http://arxiv.org/abs/2311.16580)
• Summary:

  Current methods focusing on medical image segmentation suffer from incorrect annotations, which is known as the noisy label issue. Most medical image segmentation with noisy labels methods utilize either noise transition matrix, noise-robust loss functions or pseudo-labeling methods, while none of the current research focuses on clean label disentanglement. We argue that the main reason is that the severe class-imbalanced issue will lead to the inaccuracy of the selected ``clean'' labels, thus influencing the robustness of the model against the noises. In this work, we come up with a simple but efficient class-balanced sampling strategy to tackle the class-imbalanced problem, which enables our newly proposed clean label disentangling framework to successfully select clean labels from the given label sets and encourages the model to learn from the correct annotations. However, such a method will filter out too many annotations which may also contain useful information. Therefore, we further extend our clean label disentangling framework to a new noisy feature-aided clean label disentangling framework, which takes the full annotations into utilization to learn more semantics. Extensive experiments have validated the effectiveness of our methods, where our methods achieve new state-of-the-art performance. Our code is available at https://github.com/xiaoyao3302/2BDenoise.

Title: ContextSeg: Sketch Semantic Segmentation by Querying the Context with Attention. (arXiv:2311.16682v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16682
• Code URL: null
• Copy Paste: [[2311.16682]] ContextSeg: Sketch Semantic Segmentation by Querying the Context with Attention(http://arxiv.org/abs/2311.16682)
• Summary:

  Sketch semantic segmentation is a well-explored and pivotal problem in computer vision involving the assignment of pre-defined part labels to individual strokes. This paper presents ContextSeg - a simple yet highly effective approach to tackling this problem with two stages. In the first stage, to better encode the shape and positional information of strokes, we propose to predict an extra dense distance field in an autoencoder network to reinforce structural information learning. In the second stage, we treat an entire stroke as a single entity and label a group of strokes within the same semantic part using an auto-regressive Transformer with the default attention mechanism. By group-based labeling, our method can fully leverage the context information when making decisions for the remaining groups of strokes. Our method achieves the best segmentation accuracy compared with state-of-the-art approaches on two representative datasets and has been extensively evaluated demonstrating its superior performance. Additionally, we offer insights into solving part imbalance in training data and the preliminary experiment on cross-category training, which can inspire future research in this field.

Title: Rethinking Intermediate Layers design in Knowledge Distillation for Kidney and Liver Tumor Segmentation. (arXiv:2311.16700v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16700
• Code URL: null
• Copy Paste: [[2311.16700]] Rethinking Intermediate Layers design in Knowledge Distillation for Kidney and Liver Tumor Segmentation(http://arxiv.org/abs/2311.16700)
• Summary:

  Knowledge distillation(KD) has demonstrated remarkable success across various domains, but its application to medical imaging tasks, such as kidney and liver tumor segmentation, has encountered challenges. Many existing KD methods are not specifically tailored for these tasks. Moreover, prevalent KD methods often lack a careful consideration of what and from where to distill knowledge from the teacher to the student. This oversight may lead to issues like the accumulation of training bias within shallower student layers, potentially compromising the effectiveness of KD. To address these challenges, we propose Hierarchical Layer-selective Feedback Distillation (HLFD). HLFD strategically distills knowledge from a combination of middle layers to earlier layers and transfers final layer knowledge to intermediate layers at both the feature and pixel levels. This design allows the model to learn higher-quality representations from earlier layers, resulting in a robust and compact student model. Extensive quantitative evaluations reveal that HLFD outperforms existing methods by a significant margin. For example, in the kidney segmentation task, HLFD surpasses the student model (without KD) by over 10pp, significantly improving its focus on tumor-specific features. From a qualitative standpoint, the student model trained using HLFD excels at suppressing irrelevant information and can focus sharply on tumor-specific details, which opens a new pathway for more efficient and accurate diagnostic tools.

Title: CADTalk: An Algorithm and Benchmark for Semantic Commenting of CAD Programs. (arXiv:2311.16703v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16703
• Code URL: null
• Copy Paste: [[2311.16703]] CADTalk: An Algorithm and Benchmark for Semantic Commenting of CAD Programs(http://arxiv.org/abs/2311.16703)
• Summary:

  CAD programs are a popular way to compactly encode shapes as a sequence of operations that are easy to parametrically modify. However, without sufficient semantic comments and structure, such programs can be challenging to understand, let alone modify. We introduce the problem of semantic commenting CAD programs, wherein the goal is to segment the input program into code blocks corresponding to semantically meaningful shape parts and assign a semantic label to each block. We solve the problem by combining program parsing with visual-semantic analysis afforded by recent advances in foundational language and vision models. Specifically, by executing the input programs, we create shapes, which we use to generate conditional photorealistic images to make use of semantic annotators for such images. We then distill the information across the images and link back to the original programs to semantically comment on them. Additionally, we collected and annotated a benchmark dataset, CADTalk, consisting of 5,280 machine-made programs and 45 human-made programs with ground truth semantic comments to foster future research. We extensively evaluated our approach, compared to a GPT-based baseline approach, and an open-set shape segmentation baseline, i.e., PartSLIP, and reported an 83.24% accuracy on the new CADTalk dataset. Project page: https://enigma-li.github.io/CADTalk/.

Title: Point'n Move: Interactive Scene Object Manipulation on Gaussian Splatting Radiance Fields. (arXiv:2311.16737v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16737
• Code URL: null
• Copy Paste: [[2311.16737]] Point'n Move: Interactive Scene Object Manipulation on Gaussian Splatting Radiance Fields(http://arxiv.org/abs/2311.16737)
• Summary:

  We propose Point'n Move, a method that achieves interactive scene object manipulation with exposed region inpainting. Interactivity here further comes from intuitive object selection and real-time editing. To achieve this, we adopt Gaussian Splatting Radiance Field as the scene representation and fully leverage its explicit nature and speed advantage. Its explicit representation formulation allows us to devise a 2D prompt points to 3D mask dual-stage self-prompting segmentation algorithm, perform mask refinement and merging, minimize change as well as provide good initialization for scene inpainting and perform editing in real-time without per-editing training, all leads to superior quality and performance. We test our method by performing editing on both forward-facing and 360 scenes. We also compare our method against existing scene object removal methods, showing superior quality despite being more capable and having a speed advantage.

Title: Towards Full-scene Domain Generalization in Multi-agent Collaborative Bird's Eye View Segmentation for Connected and Autonomous Driving. (arXiv:2311.16754v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.16754
• Code URL: null
• Copy Paste: [[2311.16754]] Towards Full-scene Domain Generalization in Multi-agent Collaborative Bird's Eye View Segmentation for Connected and Autonomous Driving(http://arxiv.org/abs/2311.16754)
• Summary:

  Collaborative perception has recently gained significant attention in autonomous driving, improving perception quality by enabling the exchange of additional information among vehicles. However, deploying collaborative perception systems can lead to domain shifts due to diverse environmental conditions and data heterogeneity among connected and autonomous vehicles (CAVs). To address these challenges, we propose a unified domain generalization framework applicable in both training and inference stages of collaborative perception. In the training phase, we introduce an Amplitude Augmentation (AmpAug) method to augment low-frequency image variations, broadening the model's ability to learn across various domains. We also employ a meta-consistency training scheme to simulate domain shifts, optimizing the model with a carefully designed consistency loss to encourage domain-invariant representations. In the inference phase, we introduce an intra-system domain alignment mechanism to reduce or potentially eliminate the domain discrepancy among CAVs prior to inference. Comprehensive experiments substantiate the effectiveness of our method in comparison with the existing state-of-the-art works. Code will be released at https://github.com/DG-CAVs/DG-CoPerception.git.

Title: Utilizing Multiple Inputs Autoregressive Models for Bearing Remaining Useful Life Prediction. (arXiv:2311.16192v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.16192
• Code URL: null
• Copy Paste: [[2311.16192]] Utilizing Multiple Inputs Autoregressive Models for Bearing Remaining Useful Life Prediction(http://arxiv.org/abs/2311.16192)
• Summary:

  Accurate prediction of the Remaining Useful Life (RUL) of rolling bearings is crucial in industrial production, yet existing models often struggle with limited generalization capabilities due to their inability to fully process all vibration signal patterns. We introduce a novel multi-input autoregressive model to address this challenge in RUL prediction for bearings. Our approach uniquely integrates vibration signals with previously predicted Health Indicator (HI) values, employing feature fusion to output current window HI values. Through autoregressive iterations, the model attains a global receptive field, effectively overcoming the limitations in generalization. Furthermore, we innovatively incorporate a segmentation method and multiple training iterations to mitigate error accumulation in autoregressive models. Empirical evaluation on the PMH2012 dataset demonstrates that our model, compared to other backbone networks using similar autoregressive approaches, achieves significantly lower Root Mean Square Error (RMSE) and Score. Notably, it outperforms traditional autoregressive models that use label values as inputs and non-autoregressive networks, showing superior generalization abilities with a marked lead in RMSE and Score metrics.