secure

Title: Malware Classification using Deep Neural Networks: Performance Evaluation and Applications in Edge Devices. (arXiv:2310.06841v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.06841
• Code URL: null
• Copy Paste: [[2310.06841]] Malware Classification using Deep Neural Networks: Performance Evaluation and Applications in Edge Devices(http://arxiv.org/abs/2310.06841)
• Summary:

  With the increasing extent of malware attacks in the present day along with the difficulty in detecting modern malware, it is necessary to evaluate the effectiveness and performance of Deep Neural Networks (DNNs) for malware classification. Multiple DNN architectures can be designed and trained to detect and classify malware binaries. Results demonstrate the potential of DNNs in accurately classifying malware with high accuracy rates observed across different malware types. Additionally, the feasibility of deploying these DNN models on edge devices to enable real-time classification, particularly in resource-constrained scenarios proves to be integral to large IoT systems. By optimizing model architectures and leveraging edge computing capabilities, the proposed methodologies achieve efficient performance even with limited resources. This study contributes to advancing malware detection techniques and emphasizes the significance of integrating cybersecurity measures for the early detection of malware and further preventing the adverse effects caused by such attacks. Optimal considerations regarding the distribution of security tasks to edge devices are addressed to ensure that the integrity and availability of large scale IoT systems are not compromised due to malware attacks, advocating for a more resilient and secure digital ecosystem.

Title: Secure Decentralized Learning with Blockchain. (arXiv:2310.07079v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.07079
• Code URL: null
• Copy Paste: [[2310.07079]] Secure Decentralized Learning with Blockchain(http://arxiv.org/abs/2310.07079)
• Summary:

  Federated Learning (FL) is a well-known paradigm of distributed machine learning on mobile and IoT devices, which preserves data privacy and optimizes communication efficiency. To avoid the single point of failure problem in FL, decentralized federated learning (DFL) has been proposed to use peer-to-peer communication for model aggregation, which has been considered an attractive solution for machine learning tasks on distributed personal devices. However, this process is vulnerable to attackers who share false models and data. If there exists a group of malicious clients, they might harm the performance of the model by carrying out a poisoning attack. In addition, in DFL, clients often lack the incentives to contribute their computing powers to do model training. In this paper, we proposed Blockchain-based Decentralized Federated Learning (BDFL), which leverages a blockchain for decentralized model verification and auditing. BDFL includes an auditor committee for model verification, an incentive mechanism to encourage the participation of clients, a reputation model to evaluate the trustworthiness of clients, and a protocol suite for dynamic network updates. Evaluation results show that, with the reputation mechanism, BDFL achieves fast model convergence and high accuracy on real datasets even if there exist 30\% malicious clients in the system.

security

Title: Facial Forgery-based Deepfake Detection using Fine-Grained Features. (arXiv:2310.07028v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.07028
• Code URL: null
• Copy Paste: [[2310.07028]] Facial Forgery-based Deepfake Detection using Fine-Grained Features(http://arxiv.org/abs/2310.07028)
• Summary:

  Facial forgery by deepfakes has caused major security risks and raised severe societal concerns. As a countermeasure, a number of deepfake detection methods have been proposed. Most of them model deepfake detection as a binary classification problem using a backbone convolutional neural network (CNN) architecture pretrained for the task. These CNN-based methods have demonstrated very high efficacy in deepfake detection with the Area under the Curve (AUC) as high as $0.99$. However, the performance of these methods degrades significantly when evaluated across datasets and deepfake manipulation techniques. This draws our attention towards learning more subtle, local, and discriminative features for deepfake detection. In this paper, we formulate deepfake detection as a fine-grained classification problem and propose a new fine-grained solution to it. Specifically, our method is based on learning subtle and generalizable features by effectively suppressing background noise and learning discriminative features at various scales for deepfake detection. Through extensive experimental validation, we demonstrate the superiority of our method over the published research in cross-dataset and cross-manipulation generalization of deepfake detectors for the majority of the experimental scenarios.

Title: Exploring the Horizon: A Comprehensive Survey of Rowhammer. (arXiv:2310.06950v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.06950
• Code URL: null
• Copy Paste: [[2310.06950]] Exploring the Horizon: A Comprehensive Survey of Rowhammer(http://arxiv.org/abs/2310.06950)
• Summary:

  Rowhammer poses a significant security challenge for modern computers, specifically affecting Dynamic Random Access Memory(DRAM). Given society's growing reliance on computer systems, ensuring the reliability of hardware is of utmost importance. This paper provides a comprehensive survey of Rowhammer, examining the literature from various angles. We categorise studies on Rowhammer into attacks, defences, and intriguing work, exploring each category in detail. Furthermore, we classify papers within each category into distinct yet overlapping classes and present an overview of the papers in each class.

Title: EtrusChain: File Storage with DNA and Blockchain. (arXiv:2310.07074v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.07074
• Code URL: null
• Copy Paste: [[2310.07074]] EtrusChain: File Storage with DNA and Blockchain(http://arxiv.org/abs/2310.07074)
• Summary:

  This article proposes a blockchain-based file storage system that utilizes DNA encryption for enhanced security. The system utilizes blockchain technology to provide decentralized and tamper-proof file storage, while DNA encryption is employed to further strengthen data protection. The proposed system employs a unique approach to encryption by utilizing DNA sequences as keys, which enhances data security and privacy. Additionally, the use of blockchain technology ensures that all file storage and access operations are transparent, immutable, and distributed among a network of nodes, making it resistant to tampering and unauthorized access. The proposed system represents a significant advancement in file storage security and provides a foundation for future research and development in the field of blockchain-based data storage

Title: ObliuSky: Oblivious User-Defined Skyline Query Processing in the Cloud. (arXiv:2310.07148v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.07148
• Code URL: null
• Copy Paste: [[2310.07148]] ObliuSky: Oblivious User-Defined Skyline Query Processing in the Cloud(http://arxiv.org/abs/2310.07148)
• Summary:

  The proliferation of cloud computing has greatly spurred the popularity of outsourced database storage and management, in which the cloud holding outsourced databases can process database queries on demand. Among others, skyline queries play an important role in the database field due to its prominent usefulness in multi-criteria decision support systems. To accommodate the tailored needs of users, user-defined skyline query has recently emerged as an intriguing advanced type of skyline query, which allows users to define custom preferences in their skyline queries (including the target attributes, preferred dominance relations, and range constraints on the target attributes). However, user-defined skyline query services, if deployed in the cloud, may raise critical privacy concerns as the outsourced databases and skyline queries may contain proprietary/privacy-sensitive information, and the cloud might even suffer from data breaches. In light of the above, this paper presents ObliuSky, a new system framework enabling oblivious user-defined skyline query processing in the cloud. ObliuSky departs from the state-of-the-art prior work by not only providing confidentiality protection for the content of the outsourced database, the user-defined skyline query, and the query results, but also making the cloud oblivious to the data patterns (e.g., user-defined dominance relations among database points and search access patterns) which may indirectly cause data leakages. We formally analyze the security guarantees and conduct extensive performance evaluations. The results show that while achieving much stronger security guarantees than the state-of-the-art prior work, ObliuSky is superior in database and query encryption efficiency, with practically affordable query latency.

Title: No Privacy Left Outside: On the (In-)Security of TEE-Shielded DNN Partition for On-Device ML. (arXiv:2310.07152v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.07152
• Code URL: https://github.com/ziqi-zhang/teeslice-artifact
• Copy Paste: [[2310.07152]] No Privacy Left Outside: On the (In-)Security of TEE-Shielded DNN Partition for On-Device ML(http://arxiv.org/abs/2310.07152)
• Summary:

  On-device ML introduces new security challenges: DNN models become white-box accessible to device users. Based on white-box information, adversaries can conduct effective model stealing (MS) and membership inference attack (MIA). Using Trusted Execution Environments (TEEs) to shield on-device DNN models aims to downgrade (easy) white-box attacks to (harder) black-box attacks. However, one major shortcoming is the sharply increased latency (up to 50X). To accelerate TEE-shield DNN computation with GPUs, researchers proposed several model partition techniques. These solutions, referred to as TEE-Shielded DNN Partition (TSDP), partition a DNN model into two parts, offloading the privacy-insensitive part to the GPU while shielding the privacy-sensitive part within the TEE. This paper benchmarks existing TSDP solutions using both MS and MIA across a variety of DNN models, datasets, and metrics. We show important findings that existing TSDP solutions are vulnerable to privacy-stealing attacks and are not as safe as commonly believed. We also unveil the inherent difficulty in deciding optimal DNN partition configurations (i.e., the highest security with minimal utility cost) for present TSDP solutions. The experiments show that such ``sweet spot'' configurations vary across datasets and models. Based on lessons harvested from the experiments, we present TEESlice, a novel TSDP method that defends against MS and MIA during DNN inference. TEESlice follows a partition-before-training strategy, which allows for accurate separation between privacy-related weights from public weights. TEESlice delivers the same security protection as shielding the entire DNN model inside TEE (the ``upper-bound'' security guarantees) with over 10X less overhead (in both experimental and real-world environments) than prior TSDP solutions and no accuracy loss.

privacy

Title: Learning with Noisy Labels for Human Fall Events Classification: Joint Cooperative Training with Trinity Networks. (arXiv:2310.06854v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.06854
• Code URL: null
• Copy Paste: [[2310.06854]] Learning with Noisy Labels for Human Fall Events Classification: Joint Cooperative Training with Trinity Networks(http://arxiv.org/abs/2310.06854)
• Summary:

  With the increasing ageing population, fall events classification has drawn much research attention. In the development of deep learning, the quality of data labels is crucial. Most of the datasets are labelled automatically or semi-automatically, and the samples may be mislabeled, which constrains the performance of Deep Neural Networks (DNNs). Recent research on noisy label learning confirms that neural networks first focus on the clean and simple instances and then follow the noisy and hard instances in the training stage. To address the learning with noisy label problem and protect the human subjects' privacy, we propose a simple but effective approach named Joint Cooperative training with Trinity Networks (JoCoT). To mitigate the privacy issue, human skeleton data are used. The robustness and performance of the noisy label learning framework is improved by using the two teacher modules and one student module in the proposed JoCoT. To mitigate the incorrect selections, the predictions from the teacher modules are applied with the consensus-based method to guide the student module training. The performance evaluation on the widely used UP-Fall dataset and comparison with the state-of-the-art, confirms the effectiveness of the proposed JoCoT in high noise rates. Precisely, JoCoT outperforms the state-of-the-art by 5.17% and 3.35% with the averaged pairflip and symmetric noises, respectively.

Title: Hyperdimensional Computing as a Rescue for Efficient Privacy-Preserving Machine Learning-as-a-Service. (arXiv:2310.06840v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.06840
• Code URL: null
• Copy Paste: [[2310.06840]] Hyperdimensional Computing as a Rescue for Efficient Privacy-Preserving Machine Learning-as-a-Service(http://arxiv.org/abs/2310.06840)
• Summary:

  Machine learning models are often provisioned as a cloud-based service where the clients send their data to the service provider to obtain the result. This setting is commonplace due to the high value of the models, but it requires the clients to forfeit the privacy that the query data may contain. Homomorphic encryption (HE) is a promising technique to address this adversity. With HE, the service provider can take encrypted data as a query and run the model without decrypting it. The result remains encrypted, and only the client can decrypt it. All these benefits come at the cost of computational cost because HE turns simple floating-point arithmetic into the computation between long (degree over 1024) polynomials. Previous work has proposed to tailor deep neural networks for efficient computation over encrypted data, but already high computational cost is again amplified by HE, hindering performance improvement. In this paper we show hyperdimensional computing can be a rescue for privacy-preserving machine learning over encrypted data. We find that the advantage of hyperdimensional computing in performance is amplified when working with HE. This observation led us to design HE-HDC, a machine-learning inference system that uses hyperdimensional computing with HE. We carefully structure the machine learning service so that the server will perform only the HE-friendly computation. Moreover, we adapt the computation and HE parameters to expedite computation while preserving accuracy and security. Our experimental result based on real measurements shows that HE-HDC outperforms existing systems by 26~3000 times with comparable classification accuracy.

Title: Improved Analysis of Sparse Linear Regression in Local Differential Privacy Model. (arXiv:2310.07367v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.07367
• Code URL: null
• Copy Paste: [[2310.07367]] Improved Analysis of Sparse Linear Regression in Local Differential Privacy Model(http://arxiv.org/abs/2310.07367)
• Summary:

  In this paper, we revisit the problem of sparse linear regression in the local differential privacy (LDP) model. Existing research in the non-interactive and sequentially local models has focused on obtaining the lower bounds for the case where the underlying parameter is $1$-sparse, and extending such bounds to the more general $k$-sparse case has proven to be challenging. Moreover, it is unclear whether efficient non-interactive LDP (NLDP) algorithms exist. To address these issues, we first consider the problem in the $\epsilon$ non-interactive LDP model and provide a lower bound of $\Omega(\frac{\sqrt{dk\log d}}{\sqrt{n}\epsilon})$ on the $\ell_2$-norm estimation error for sub-Gaussian data, where $n$ is the sample size and $d$ is the dimension of the space. We propose an innovative NLDP algorithm, the very first of its kind for the problem. As a remarkable outcome, this algorithm also yields a novel and highly efficient estimator as a valuable by-product. Our algorithm achieves an upper bound of $\tilde{O}({\frac{d\sqrt{k}}{\sqrt{n}\epsilon}})$ for the estimation error when the data is sub-Gaussian, which can be further improved by a factor of $O(\sqrt{d})$ if the server has additional public but unlabeled data. For the sequentially interactive LDP model, we show a similar lower bound of $\Omega({\frac{\sqrt{dk}}{\sqrt{n}\epsilon}})$. As for the upper bound, we rectify a previous method and show that it is possible to achieve a bound of $\tilde{O}(\frac{k\sqrt{d}}{\sqrt{n}\epsilon})$. Our findings reveal fundamental differences between the non-private case, central DP model, and local DP model in the sparse linear regression problem.

protect

Title: TDPP: Two-Dimensional Permutation-Based Protection of Memristive Deep Neural Networks. (arXiv:2310.06989v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.06989
• Code URL: null
• Copy Paste: [[2310.06989]] TDPP: Two-Dimensional Permutation-Based Protection of Memristive Deep Neural Networks(http://arxiv.org/abs/2310.06989)
• Summary:

  The execution of deep neural network (DNN) algorithms suffers from significant bottlenecks due to the separation of the processing and memory units in traditional computer systems. Emerging memristive computing systems introduce an in situ approach that overcomes this bottleneck. The non-volatility of memristive devices, however, may expose the DNN weights stored in memristive crossbars to potential theft attacks. Therefore, this paper proposes a two-dimensional permutation-based protection (TDPP) method that thwarts such attacks. We first introduce the underlying concept that motivates the TDPP method: permuting both the rows and columns of the DNN weight matrices. This contrasts with previous methods, which focused solely on permuting a single dimension of the weight matrices, either the rows or columns. While it's possible for an adversary to access the matrix values, the original arrangement of rows and columns in the matrices remains concealed. As a result, the extracted DNN model from the accessed matrix values would fail to operate correctly. We consider two different memristive computing systems (designed for layer-by-layer and layer-parallel processing, respectively) and demonstrate the design of the TDPP method that could be embedded into the two systems. Finally, we present a security analysis. Our experiments demonstrate that TDPP can achieve comparable effectiveness to prior approaches, with a high level of security when appropriately parameterized. In addition, TDPP is more scalable than previous methods and results in reduced area and power overheads. The area and power are reduced by, respectively, 1218$\times$ and 2815$\times$ for the layer-by-layer system and by 178$\times$ and 203$\times$ for the layer-parallel system compared to prior works.

Title: Sound-skwatter (Did You Mean: Sound-squatter?) AI-powered Generator for Phishing Prevention. (arXiv:2310.07005v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.07005
• Code URL: null
• Copy Paste: [[2310.07005]] Sound-skwatter (Did You Mean: Sound-squatter?) AI-powered Generator for Phishing Prevention(http://arxiv.org/abs/2310.07005)
• Summary:

  Sound-squatting is a phishing attack that tricks users into malicious resources by exploiting similarities in the pronunciation of words. Proactive defense against sound-squatting candidates is complex, and existing solutions rely on manually curated lists of homophones. We here introduce Sound-skwatter, a multi-language AI-based system that generates sound-squatting candidates for proactive defense. Sound-skwatter relies on an innovative multi-modal combination of Transformers Networks and acoustic models to learn sound similarities. We show that Sound-skwatter can automatically list known homophones and thousands of high-quality candidates. In addition, it covers cross-language sound-squatting, i.e., when the reader and the listener speak different languages, supporting any combination of languages. We apply Sound-skwatter to network-centric phishing via squatted domain names. We find ~ 10% of the generated domains exist in the wild, the vast majority unknown to protection solutions. Next, we show attacks on the PyPI package manager, where ~ 17% of the popular packages have at least one existing candidate. We believe Sound-skwatter is a crucial asset to mitigate the sound-squatting phenomenon proactively on the Internet. To increase its impact, we publish an online demo and release our models and code as open source.

Title: GraphCloak: Safeguarding Task-specific Knowledge within Graph-structured Data from Unauthorized Exploitation. (arXiv:2310.07100v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.07100
• Code URL: null
• Copy Paste: [[2310.07100]] GraphCloak: Safeguarding Task-specific Knowledge within Graph-structured Data from Unauthorized Exploitation(http://arxiv.org/abs/2310.07100)
• Summary:

  As Graph Neural Networks (GNNs) become increasingly prevalent in a variety of fields, from social network analysis to protein-protein interaction studies, growing concerns have emerged regarding the unauthorized utilization of personal data. Recent studies have shown that imperceptible poisoning attacks are an effective method of protecting image data from such misuse. However, the efficacy of this approach in the graph domain remains unexplored. To bridge this gap, this paper introduces GraphCloak to safeguard against the unauthorized usage of graph data. Compared with prior work, GraphCloak offers unique significant innovations: (1) graph-oriented, the perturbations are applied to both topological structures and descriptive features of the graph; (2) effective and stealthy, our cloaking method can bypass various inspections while causing a significant performance drop in GNNs trained on the cloaked graphs; and (3) stable across settings, our methods consistently perform effectively under a range of practical settings with limited knowledge. To address the intractable bi-level optimization problem, we propose two error-minimizing-based poisoning methods that target perturbations on the structural and feature space, along with a subgraph injection poisoning method. Our comprehensive evaluation of these methods underscores their effectiveness, stealthiness, and stability. We also delve into potential countermeasures and provide analytical justification for their effectiveness, paving the way for intriguing future research.

Title: Code Polymorphism Meets Code Encryption: Confidentiality and Side-Channel Protection of Software Components. (arXiv:2310.07327v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.07327
• Code URL: null
• Copy Paste: [[2310.07327]] Code Polymorphism Meets Code Encryption: Confidentiality and Side-Channel Protection of Software Components(http://arxiv.org/abs/2310.07327)
• Summary:

  In this paper, we consider that, in practice, attack scenarios involving side-channel analysis combine two successive phases:an analysis phase, targeting the extraction of information about the target and the identification of possible vulnerabilities;and an exploitation phase, applying attack techniques on candidate vulnerabilities. We advocate that protections need to coverthese two phases in order to be effective against real-life attacks. We present PolEn, a toolchain and a processor architecturethat combine countermeasures in order to provide an effective mitigation of side-channel attacks: as a countermeasure againstthe analysis phase, our approach considers the use of code encryption; as a countermeasure against the exploitation phase,our approach considers the use of code polymorphism, because it relies on runtime code generation, and its combinationwith code encryption is particularly challenging. Code encryption is supported by a processor extension such that machineinstructions are only decrypted inside the CPU, which effectively prevents reverse engineering or any extraction of usefulinformation from memory dumps. Code polymorphism is implemented by software means. It regularly changes the observablebehaviour of the program, making it unpredictable for an attacker, hence reducing the possibility to exploit side-channelleakages. We present a prototype implementation, based on the RISC-V Spike simulator and a modified LLVM toolchain. Inour experimental evaluation, we illustrate that PolEn effectively reduces side-channel leakages. For the protected functionsevaluated, static memory use increases by a factor of 5 to 22, corresponding to the joint application of code encryption andcode polymorphism. The overhead, in terms of execution time, ranges between a factor of 1.8 and 4.6.

defense

Title: A Systematic Review of Machine Learning Enabled Phishing. (arXiv:2310.06998v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.06998
• Code URL: null
• Copy Paste: [[2310.06998]] A Systematic Review of Machine Learning Enabled Phishing(http://arxiv.org/abs/2310.06998)
• Summary:

  Developments in artificial intelligence (AI) are likely to affect social engineering and change cyber defense operations. The broad and sweeping nature of AI impact means that many aspects of social engineering could be automated, potentially giving adversaries an advantage. In this review, we assess the ways phishing and spear-phishing might be affected by machine learning techniques. By performing a systematic review of demonstrated ML-enabled phishing campaigns, we take a broad survey the space for current developments. We develop a detailed approach for evaluation by creating a risk framework for analyzing and contextualizing these developments. The object of this review is to answer the research questions: (1) Are there high-risk ML-enabled phishing use cases? (2) Is there a meaningful difference between traditional targeted phishing campaigns and ML-enabled phishing campaigns? Practitioners may use this review to inform standards, future research directions, and cyber defense strategies.

attack

Title: Comparing the robustness of modern no-reference image- and video-quality metrics to adversarial attacks. (arXiv:2310.06958v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.06958
• Code URL: https://github.com/msu-video-group/msu_metrics_robustness_benchmark
• Copy Paste: [[2310.06958]] Comparing the robustness of modern no-reference image- and video-quality metrics to adversarial attacks(http://arxiv.org/abs/2310.06958)
• Summary:

  Nowadays neural-network-based image- and video-quality metrics show better performance compared to traditional methods. However, they also became more vulnerable to adversarial attacks that increase metrics' scores without improving visual quality. The existing benchmarks of quality metrics compare their performance in terms of correlation with subjective quality and calculation time. However, the adversarial robustness of image-quality metrics is also an area worth researching. In this paper, we analyse modern metrics' robustness to different adversarial attacks. We adopted adversarial attacks from computer vision tasks and compared attacks' efficiency against 15 no-reference image/video-quality metrics. Some metrics showed high resistance to adversarial attacks which makes their usage in benchmarks safer than vulnerable metrics. The benchmark accepts new metrics submissions for researchers who want to make their metrics more robust to attacks or to find such metrics for their needs. Try our benchmark using pip install robustness-benchmark.

Title: Boosting Black-box Attack to Deep Neural Networks with Conditional Diffusion Models. (arXiv:2310.07492v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.07492
• Code URL: null
• Copy Paste: [[2310.07492]] Boosting Black-box Attack to Deep Neural Networks with Conditional Diffusion Models(http://arxiv.org/abs/2310.07492)
• Summary:

  Existing black-box attacks have demonstrated promising potential in creating adversarial examples (AE) to deceive deep learning models. Most of these attacks need to handle a vast optimization space and require a large number of queries, hence exhibiting limited practical impacts in real-world scenarios. In this paper, we propose a novel black-box attack strategy, Conditional Diffusion Model Attack (CDMA), to improve the query efficiency of generating AEs under query-limited situations. The key insight of CDMA is to formulate the task of AE synthesis as a distribution transformation problem, i.e., benign examples and their corresponding AEs can be regarded as coming from two distinctive distributions and can transform from each other with a particular converter. Unlike the conventional \textit{query-and-optimization} approach, we generate eligible AEs with direct conditional transform using the aforementioned data converter, which can significantly reduce the number of queries needed. CDMA adopts the conditional Denoising Diffusion Probabilistic Model as the converter, which can learn the transformation from clean samples to AEs, and ensure the smooth development of perturbed noise resistant to various defense strategies. We demonstrate the effectiveness and efficiency of CDMA by comparing it with nine state-of-the-art black-box attacks across three benchmark datasets. On average, CDMA can reduce the query count to a handful of times; in most cases, the query count is only ONE. We also show that CDMA can obtain $>99\%$ attack success rate for untarget attacks over all datasets and targeted attack over CIFAR-10 with the noise budget of $\epsilon=16$.

Title: Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation. (arXiv:2310.06987v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.06987
• Code URL: https://github.com/princeton-sysml/jailbreak_llm
• Copy Paste: [[2310.06987]] Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation(http://arxiv.org/abs/2310.06987)
• Summary:

  The rapid progress in open-source large language models (LLMs) is significantly advancing AI development. Extensive efforts have been made before model release to align their behavior with human values, with the primary goal of ensuring their helpfulness and harmlessness. However, even carefully aligned models can be manipulated maliciously, leading to unintended behaviors, known as "jailbreaks". These jailbreaks are typically triggered by specific text inputs, often referred to as adversarial prompts. In this work, we propose the generation exploitation attack, an extremely simple approach that disrupts model alignment by only manipulating variations of decoding methods. By exploiting different generation strategies, including varying decoding hyper-parameters and sampling methods, we increase the misalignment rate from 0% to more than 95% across 11 language models including LLaMA2, Vicuna, Falcon, and MPT families, outperforming state-of-the-art attacks with $30\times$ lower computational cost. Finally, we propose an effective alignment method that explores diverse generation strategies, which can reasonably reduce the misalignment rate under our attack. Altogether, our study underscores a major failure in current safety evaluation and alignment procedures for open-source LLMs, strongly advocating for more comprehensive red teaming and better alignment before releasing such models. Our code is available at https://github.com/Princeton-SysML/Jailbreak_LLM.

Title: Genetic Algorithm-Based Dynamic Backdoor Attack on Federated Learning-Based Network Traffic Classification. (arXiv:2310.06855v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.06855
• Code URL: null
• Copy Paste: [[2310.06855]] Genetic Algorithm-Based Dynamic Backdoor Attack on Federated Learning-Based Network Traffic Classification(http://arxiv.org/abs/2310.06855)
• Summary:

  Federated learning enables multiple clients to collaboratively contribute to the learning of a global model orchestrated by a central server. This learning scheme promotes clients' data privacy and requires reduced communication overheads. In an application like network traffic classification, this helps hide the network vulnerabilities and weakness points. However, federated learning is susceptible to backdoor attacks, in which adversaries inject manipulated model updates into the global model. These updates inject a salient functionality in the global model that can be launched with specific input patterns. Nonetheless, the vulnerability of network traffic classification models based on federated learning to these attacks remains unexplored. In this paper, we propose GABAttack, a novel genetic algorithm-based backdoor attack against federated learning for network traffic classification. GABAttack utilizes a genetic algorithm to optimize the values and locations of backdoor trigger patterns, ensuring a better fit with the input and the model. This input-tailored dynamic attack is promising for improved attack evasiveness while being effective. Extensive experiments conducted over real-world network datasets validate the success of the proposed GABAttack in various situations while maintaining almost invisible activity. This research serves as an alarming call for network security experts and practitioners to develop robust defense measures against such attacks.

Title: Unclonable Non-Interactive Zero-Knowledge. (arXiv:2310.07118v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.07118
• Code URL: null
• Copy Paste: [[2310.07118]] Unclonable Non-Interactive Zero-Knowledge(http://arxiv.org/abs/2310.07118)
• Summary:

  A non-interactive ZK (NIZK) proof enables verification of NP statements without revealing secrets about them. However, an adversary that obtains a NIZK proof may be able to clone this proof and distribute arbitrarily many copies of it to various entities: this is inevitable for any proof that takes the form of a classical string. In this paper, we ask whether it is possible to rely on quantum information in order to build NIZK proof systems that are impossible to clone.

  We define and construct unclonable non-interactive zero-knowledge proofs (of knowledge) for NP. Besides satisfying the zero-knowledge and proof of knowledge properties, these proofs additionally satisfy unclonability. Very roughly, this ensures that no adversary can split an honestly generated proof of membership of an instance $x$ in an NP language $\mathcal{L}$ and distribute copies to multiple entities that all obtain accepting proofs of membership of $x$ in $\mathcal{L}$. Our result has applications to unclonable signatures of knowledge, which we define and construct in this work; these non-interactively prevent replay attacks.

Title: My Brother Helps Me: Node Injection Based Adversarial Attack on Social Bot Detection. (arXiv:2310.07159v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.07159
• Code URL: null
• Copy Paste: [[2310.07159]] My Brother Helps Me: Node Injection Based Adversarial Attack on Social Bot Detection(http://arxiv.org/abs/2310.07159)
• Summary:

  Social platforms such as Twitter are under siege from a multitude of fraudulent users. In response, social bot detection tasks have been developed to identify such fake users. Due to the structure of social networks, the majority of methods are based on the graph neural network(GNN), which is susceptible to attacks. In this study, we propose a node injection-based adversarial attack method designed to deceive bot detection models. Notably, neither the target bot nor the newly injected bot can be detected when a new bot is added around the target bot. This attack operates in a black-box fashion, implying that any information related to the victim model remains unknown. To our knowledge, this is the first study exploring the resilience of bot detection through graph node injection. Furthermore, we develop an attribute recovery module to revert the injected node embedding from the graph embedding space back to the original feature space, enabling the adversary to manipulate node perturbation effectively. We conduct adversarial attacks on four commonly used GNN structures for bot detection on two widely used datasets: Cresci-2015 and TwiBot-22. The attack success rate is over 73\% and the rate of newly injected nodes being detected as bots is below 13\% on these two datasets.

Title: Improved Membership Inference Attacks Against Language Classification Models. (arXiv:2310.07219v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.07219
• Code URL: null
• Copy Paste: [[2310.07219]] Improved Membership Inference Attacks Against Language Classification Models(http://arxiv.org/abs/2310.07219)
• Summary:

  Artificial intelligence systems are prevalent in everyday life, with use cases in retail, manufacturing, health, and many other fields. With the rise in AI adoption, associated risks have been identified, including privacy risks to the people whose data was used to train models. Assessing the privacy risks of machine learning models is crucial to enabling knowledgeable decisions on whether to use, deploy, or share a model. A common approach to privacy risk assessment is to run one or more known attacks against the model and measure their success rate. We present a novel framework for running membership inference attacks against classification models. Our framework takes advantage of the ensemble method, generating many specialized attack models for different subsets of the data. We show that this approach achieves higher accuracy than either a single attack model or an attack model per class label, both on classical and language classification tasks.

Title: A Variational Autoencoder Framework for Robust, Physics-Informed Cyberattack Recognition in Industrial Cyber-Physical Systems. (arXiv:2310.06948v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.06948
• Code URL: null
• Copy Paste: [[2310.06948]] A Variational Autoencoder Framework for Robust, Physics-Informed Cyberattack Recognition in Industrial Cyber-Physical Systems(http://arxiv.org/abs/2310.06948)
• Summary:

  Cybersecurity of Industrial Cyber-Physical Systems is drawing significant concerns as data communication increasingly leverages wireless networks. A lot of data-driven methods were develope for detecting cyberattacks, but few are focused on distinguishing them from equipment faults. In this paper, we develop a data-driven framework that can be used to detect, diagnose, and localize a type of cyberattack called covert attacks on networked industrial control systems. The framework has a hybrid design that combines a variational autoencoder (VAE), a recurrent neural network (RNN), and a Deep Neural Network (DNN). This data-driven framework considers the temporal behavior of a generic physical system that extracts features from the time series of the sensor measurements that can be used for detecting covert attacks, distinguishing them from equipment faults, as well as localize the attack/fault. We evaluate the performance of the proposed method through a realistic simulation study on a networked power transmission system as a typical example of ICS. We compare the performance of the proposed method with the traditional model-based method to show its applicability and efficacy.

Title: Byzantine-Resilient Decentralized Multi-Armed Bandits. (arXiv:2310.07320v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.07320
• Code URL: null
• Copy Paste: [[2310.07320]] Byzantine-Resilient Decentralized Multi-Armed Bandits(http://arxiv.org/abs/2310.07320)
• Summary:

  In decentralized cooperative multi-armed bandits (MAB), each agent observes a distinct stream of rewards, and seeks to exchange information with others to select a sequence of arms so as to minimize its regret. Agents in the cooperative setting can outperform a single agent running a MAB method such as Upper-Confidence Bound (UCB) independently. In this work, we study how to recover such salient behavior when an unknown fraction of the agents can be Byzantine, that is, communicate arbitrarily wrong information in the form of reward mean-estimates or confidence sets. This framework can be used to model attackers in computer networks, instigators of offensive content into recommender systems, or manipulators of financial markets. Our key contribution is the development of a fully decentralized resilient upper confidence bound (UCB) algorithm that fuses an information mixing step among agents with a truncation of inconsistent and extreme values. This truncation step enables us to establish that the performance of each normal agent is no worse than the classic single-agent UCB1 algorithm in terms of regret, and more importantly, the cumulative regret of all normal agents is strictly better than the non-cooperative case, provided that each agent has at least 3f+1 neighbors where f is the maximum possible Byzantine agents in each agent's neighborhood. Extensions to time-varying neighbor graphs, and minimax lower bounds are further established on the achievable regret. Experiments corroborate the merits of this framework in practice.

robust

Title: RobustEdge: Low Power Adversarial Detection for Cloud-Edge Systems. (arXiv:2310.06845v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.06845
• Code URL: null
• Copy Paste: [[2310.06845]] RobustEdge: Low Power Adversarial Detection for Cloud-Edge Systems(http://arxiv.org/abs/2310.06845)
• Summary:

  In practical cloud-edge scenarios, where a resource constrained edge performs data acquisition and a cloud system (having sufficient resources) performs inference tasks with a deep neural network (DNN), adversarial robustness is critical for reliability and ubiquitous deployment. Adversarial detection is a prime adversarial defence technique used in prior literature. However, in prior detection works, the detector is attached to the classifier model and both detector and classifier work in tandem to perform adversarial detection that requires a high computational overhead which is not available at the low-power edge. Therefore, prior works can only perform adversarial detection at the cloud and not at the edge. This means that in case of adversarial attacks, the unfavourable adversarial samples must be communicated to the cloud which leads to energy wastage at the edge device. Therefore, a low-power edge-friendly adversarial detection method is required to improve the energy efficiency of the edge and robustness of the cloud-based classifier. To this end, RobustEdge proposes Quantization-enabled Energy Separation (QES) training with "early detection and exit" to perform edge-based low cost adversarial detection. The QES-trained detector implemented at the edge blocks adversarial data transmission to the classifier model, thereby improving adversarial robustness and energy-efficiency of the Cloud-Edge system.

Title: Self-supervised Object-Centric Learning for Videos. (arXiv:2310.06907v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.06907
• Code URL: null
• Copy Paste: [[2310.06907]] Self-supervised Object-Centric Learning for Videos(http://arxiv.org/abs/2310.06907)
• Summary:

  Unsupervised multi-object segmentation has shown impressive results on images by utilizing powerful semantics learned from self-supervised pretraining. An additional modality such as depth or motion is often used to facilitate the segmentation in video sequences. However, the performance improvements observed in synthetic sequences, which rely on the robustness of an additional cue, do not translate to more challenging real-world scenarios. In this paper, we propose the first fully unsupervised method for segmenting multiple objects in real-world sequences. Our object-centric learning framework spatially binds objects to slots on each frame and then relates these slots across frames. From these temporally-aware slots, the training objective is to reconstruct the middle frame in a high-level semantic feature space. We propose a masking strategy by dropping a significant portion of tokens in the feature space for efficiency and regularization. Additionally, we address over-clustering by merging slots based on similarity. Our method can successfully segment multiple instances of complex and high-variety classes in YouTube videos.

Title: TextPSG: Panoptic Scene Graph Generation from Textual Descriptions. (arXiv:2310.07056v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.07056
• Code URL: null
• Copy Paste: [[2310.07056]] TextPSG: Panoptic Scene Graph Generation from Textual Descriptions(http://arxiv.org/abs/2310.07056)
• Summary:

  Panoptic Scene Graph has recently been proposed for comprehensive scene understanding. However, previous works adopt a fully-supervised learning manner, requiring large amounts of pixel-wise densely-annotated data, which is always tedious and expensive to obtain. To address this limitation, we study a new problem of Panoptic Scene Graph Generation from Purely Textual Descriptions (Caption-to-PSG). The key idea is to leverage the large collection of free image-caption data on the Web alone to generate panoptic scene graphs. The problem is very challenging for three constraints: 1) no location priors; 2) no explicit links between visual regions and textual entities; and 3) no pre-defined concept sets. To tackle this problem, we propose a new framework TextPSG consisting of four modules, i.e., a region grouper, an entity grounder, a segment merger, and a label generator, with several novel techniques. The region grouper first groups image pixels into different segments and the entity grounder then aligns visual segments with language entities based on the textual description of the segment being referred to. The grounding results can thus serve as pseudo labels enabling the segment merger to learn the segment similarity as well as guiding the label generator to learn object semantics and relation predicates, resulting in a fine-grained structured scene understanding. Our framework is effective, significantly outperforming the baselines and achieving strong out-of-distribution robustness. We perform comprehensive ablation studies to corroborate the effectiveness of our design choices and provide an in-depth analysis to highlight future directions. Our code, data, and results are available on our project page: https://vis-www.cs.umass.edu/TextPSG.

Title: Robust Unsupervised Domain Adaptation by Retaining Confident Entropy via Edge Concatenation. (arXiv:2310.07149v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.07149
• Code URL: null
• Copy Paste: [[2310.07149]] Robust Unsupervised Domain Adaptation by Retaining Confident Entropy via Edge Concatenation(http://arxiv.org/abs/2310.07149)
• Summary:

  The generalization capability of unsupervised domain adaptation can mitigate the need for extensive pixel-level annotations to train semantic segmentation networks by training models on synthetic data as a source with computer-generated annotations. Entropy-based adversarial networks are proposed to improve source domain prediction; however, they disregard significant external information, such as edges, which have the potential to identify and distinguish various objects within an image accurately. To address this issue, we introduce a novel approach to domain adaptation, leveraging the synergy of internal and external information within entropy-based adversarial networks. In this approach, we enrich the discriminator network with edge-predicted probability values within this innovative framework to enhance the clarity of class boundaries. Furthermore, we devised a probability-sharing network that integrates diverse information for more effective segmentation. Incorporating object edges addresses a pivotal aspect of unsupervised domain adaptation that has frequently been neglected in the past -- the precise delineation of object boundaries. Conventional unsupervised domain adaptation methods usually center around aligning feature distributions and may not explicitly model object boundaries. Our approach effectively bridges this gap by offering clear guidance on object boundaries, thereby elevating the quality of domain adaptation. Our approach undergoes rigorous evaluation on the established unsupervised domain adaptation benchmarks, specifically in adapting SYNTHIA $\rightarrow$ Cityscapes and SYNTHIA $\rightarrow$ Mapillary. Experimental results show that the proposed model attains better performance than state-of-the-art methods. The superior performance across different unsupervised domain adaptation scenarios highlights the versatility and robustness of the proposed method.

Title: Multi-Task Learning-Enabled Automatic Vessel Draft Reading for Intelligent Maritime Surveillance. (arXiv:2310.07212v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.07212
• Code URL: null
• Copy Paste: [[2310.07212]] Multi-Task Learning-Enabled Automatic Vessel Draft Reading for Intelligent Maritime Surveillance(http://arxiv.org/abs/2310.07212)
• Summary:

  The accurate and efficient vessel draft reading (VDR) is an important component of intelligent maritime surveillance, which could be exploited to assist in judging whether the vessel is normally loaded or overloaded. The computer vision technique with an excellent price-to-performance ratio has become a popular medium to estimate vessel draft depth. However, the traditional estimation methods easily suffer from several limitations, such as sensitivity to low-quality images, high computational cost, etc. In this work, we propose a multi-task learning-enabled computational method (termed MTL-VDR) for generating highly reliable VDR. In particular, our MTL-VDR mainly consists of four components, i.e., draft mark detection, draft scale recognition, vessel/water segmentation, and final draft depth estimation. We first construct a benchmark dataset related to draft mark detection and employ a powerful and efficient convolutional neural network to accurately perform the detection task. The multi-task learning method is then proposed for simultaneous draft scale recognition and vessel/water segmentation. To obtain more robust VDR under complex conditions (e.g., damaged and stained scales, etc.), the accurate draft scales are generated by an automatic correction method, which is presented based on the spatial distribution rules of draft scales. Finally, an adaptive computational method is exploited to yield an accurate and robust draft depth. Extensive experiments have been implemented on the realistic dataset to compare our MTL-VDR with state-of-the-art methods. The results have demonstrated its superior performance in terms of accuracy, robustness, and efficiency. The computational speed exceeds 40 FPS, which satisfies the requirements of real-time maritime surveillance to guarantee vessel traffic safety.

Title: IBoxCLA: Towards Robust Box-supervised Segmentation of Polyp via Improved Box-dice and Contrastive Latent-anchors. (arXiv:2310.07248v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.07248
• Code URL: null
• Copy Paste: [[2310.07248]] IBoxCLA: Towards Robust Box-supervised Segmentation of Polyp via Improved Box-dice and Contrastive Latent-anchors(http://arxiv.org/abs/2310.07248)
• Summary:

  Box-supervised polyp segmentation attracts increasing attention for its cost-effective potential. Existing solutions often rely on learning-free methods or pretrained models to laboriously generate pseudo masks, triggering Dice constraint subsequently. In this paper, we found that a model guided by the simplest box-filled masks can accurately predict polyp locations/sizes, but suffers from shape collapsing. In response, we propose two innovative learning fashions, Improved Box-dice (IBox) and Contrastive Latent-Anchors (CLA), and combine them to train a robust box-supervised model IBoxCLA. The core idea behind IBoxCLA is to decouple the learning of location/size and shape, allowing for focused constraints on each of them. Specifically, IBox transforms the segmentation map into a proxy map using shape decoupling and confusion-region swapping sequentially. Within the proxy map, shapes are disentangled, while locations/sizes are encoded as box-like responses. By constraining the proxy map instead of the raw prediction, the box-filled mask can well supervise IBoxCLA without misleading its shape learning. Furthermore, CLA contributes to shape learning by generating two types of latent anchors, which are learned and updated using momentum and segmented polyps to steadily represent polyp and background features. The latent anchors facilitate IBoxCLA to capture discriminative features within and outside boxes in a contrastive manner, yielding clearer boundaries. We benchmark IBoxCLA on five public polyp datasets. The experimental results demonstrate the competitive performance of IBoxCLA compared to recent fully-supervised polyp segmentation methods, and its superiority over other box-supervised state-of-the-arts with a relative increase of overall mDice and mIoU by at least 6.5% and 7.5%, respectively.

Title: ADASR: An Adversarial Auto-Augmentation Framework for Hyperspectral and Multispectral Data Fusion. (arXiv:2310.07255v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.07255
• Code URL: https://github.com/fangfang11-plog/adasr
• Copy Paste: [[2310.07255]] ADASR: An Adversarial Auto-Augmentation Framework for Hyperspectral and Multispectral Data Fusion(http://arxiv.org/abs/2310.07255)
• Summary:

  Deep learning-based hyperspectral image (HSI) super-resolution, which aims to generate high spatial resolution HSI (HR-HSI) by fusing hyperspectral image (HSI) and multispectral image (MSI) with deep neural networks (DNNs), has attracted lots of attention. However, neural networks require large amounts of training data, hindering their application in real-world scenarios. In this letter, we propose a novel adversarial automatic data augmentation framework ADASR that automatically optimizes and augments HSI-MSI sample pairs to enrich data diversity for HSI-MSI fusion. Our framework is sample-aware and optimizes an augmentor network and two downsampling networks jointly by adversarial learning so that we can learn more robust downsampling networks for training the upsampling network. Extensive experiments on two public classical hyperspectral datasets demonstrate the effectiveness of our ADASR compared to the state-of-the-art methods.

Title: Deep Aramaic: Towards a Synthetic Data Paradigm Enabling Machine Learning in Epigraphy. (arXiv:2310.07310v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.07310
• Code URL: null
• Copy Paste: [[2310.07310]] Deep Aramaic: Towards a Synthetic Data Paradigm Enabling Machine Learning in Epigraphy(http://arxiv.org/abs/2310.07310)
• Summary:

  Epigraphy increasingly turns to modern artificial intelligence (AI) technologies such as machine learning (ML) for extracting insights from ancient inscriptions. However, scarce labeled data for training ML algorithms severely limits current techniques, especially for ancient scripts like Old Aramaic. Our research pioneers an innovative methodology for generating synthetic training data tailored to Old Aramaic letters. Our pipeline synthesizes photo-realistic Aramaic letter datasets, incorporating textural features, lighting, damage, and augmentations to mimic real-world inscription diversity. Despite minimal real examples, we engineer a dataset of 250,000 training and 25,000 validation images covering the 22 letter classes in the Aramaic alphabet. This comprehensive corpus provides a robust volume of data for training a residual neural network (ResNet) to classify highly degraded Aramaic letters. The ResNet model demonstrates high accuracy in classifying real images from the 8th century BCE Hadad statue inscription. Additional experiments validate performance on varying materials and styles, proving effective generalization. Our results validate the model's capabilities in handling diverse real-world scenarios, proving the viability of our synthetic data approach and avoiding the dependence on scarce training data that has constrained epigraphic analysis. Our innovative framework elevates interpretation accuracy on damaged inscriptions, thus enhancing knowledge extraction from these historical resources.

Title: PoRF: Pose Residual Field for Accurate Neural Surface Reconstruction. (arXiv:2310.07449v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.07449
• Code URL: null
• Copy Paste: [[2310.07449]] PoRF: Pose Residual Field for Accurate Neural Surface Reconstruction(http://arxiv.org/abs/2310.07449)
• Summary:

  Neural surface reconstruction is sensitive to the camera pose noise, even if state-of-the-art pose estimators like COLMAP or ARKit are used. More importantly, existing Pose-NeRF joint optimisation methods have struggled to improve pose accuracy in challenging real-world scenarios. To overcome the challenges, we introduce the pose residual field (\textbf{PoRF}), a novel implicit representation that uses an MLP for regressing pose updates. This is more robust than the conventional pose parameter optimisation due to parameter sharing that leverages global information over the entire sequence. Furthermore, we propose an epipolar geometry loss to enhance the supervision that leverages the correspondences exported from COLMAP results without the extra computational overhead. Our method yields promising results. On the DTU dataset, we reduce the rotation error by 78\% for COLMAP poses, leading to the decreased reconstruction Chamfer distance from 3.48mm to 0.85mm. On the MobileBrick dataset that contains casually captured unbounded 360-degree videos, our method refines ARKit poses and improves the reconstruction F1 score from 69.18 to 75.67, outperforming that with the dataset provided ground-truth pose (75.14). These achievements demonstrate the efficacy of our approach in refining camera poses and improving the accuracy of neural surface reconstruction in real-world scenarios.

Title: Auditing and Robustifying COVID-19 Misinformation Datasets via Anticontent Sampling. (arXiv:2310.07078v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.07078
• Code URL: null
• Copy Paste: [[2310.07078]] Auditing and Robustifying COVID-19 Misinformation Datasets via Anticontent Sampling(http://arxiv.org/abs/2310.07078)
• Summary:

  This paper makes two key contributions. First, it argues that highly specialized rare content classifiers trained on small data typically have limited exposure to the richness and topical diversity of the negative class (dubbed anticontent) as observed in the wild. As a result, these classifiers' strong performance observed on the test set may not translate into real-world settings. In the context of COVID-19 misinformation detection, we conduct an in-the-wild audit of multiple datasets and demonstrate that models trained with several prominently cited recent datasets are vulnerable to anticontent when evaluated in the wild. Second, we present a novel active learning pipeline that requires zero manual annotation and iteratively augments the training data with challenging anticontent, robustifying these classifiers.

Title: QFT: Quantized Full-parameter Tuning of LLMs with Affordable Resources. (arXiv:2310.07147v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.07147
• Code URL: null
• Copy Paste: [[2310.07147]] QFT: Quantized Full-parameter Tuning of LLMs with Affordable Resources(http://arxiv.org/abs/2310.07147)
• Summary:

  Large Language Models (LLMs) have showcased remarkable impacts across a wide spectrum of natural language processing tasks. Fine-tuning these pre-trained models on downstream datasets provides further significant performance gains, but this process has been challenging due to its extraordinary resource requirements. To this end, existing efforts focus on parameter-efficient fine-tuning, which, unfortunately, fail to capitalize on the powerful potential of full-parameter fine-tuning. In this work, we propose QFT, a novel Quantized Full-parameter Tuning framework for LLMs that enables memory-efficient fine-tuning without harming performance. Our framework incorporates two novel ideas: (i) we adopt the efficient Lion optimizer, which only keeps track of the momentum and has consistent update magnitudes for each parameter, an inherent advantage for robust quantization; and (ii) we quantize all model states and store them as integer values, and present a gradient flow and parameter update scheme for the quantized weights. As a result, QFT reduces the model state memory to 21% of the standard solution while achieving comparable performance, e.g., tuning a LLaMA-7B model requires only <30GB of memory, satisfied by a single A6000 GPU.

Title: BioT5: Enriching Cross-modal Integration in Biology with Chemical Knowledge and Natural Language Associations. (arXiv:2310.07276v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.07276
• Code URL: https://github.com/QizhiPei/BioT5
• Copy Paste: [[2310.07276]] BioT5: Enriching Cross-modal Integration in Biology with Chemical Knowledge and Natural Language Associations(http://arxiv.org/abs/2310.07276)
• Summary:

  Recent advancements in biological research leverage the integration of molecules, proteins, and natural language to enhance drug discovery. However, current models exhibit several limitations, such as the generation of invalid molecular SMILES, underutilization of contextual information, and equal treatment of structured and unstructured knowledge. To address these issues, we propose $\mathbf{BioT5}$, a comprehensive pre-training framework that enriches cross-modal integration in biology with chemical knowledge and natural language associations. $\mathbf{BioT5}$ utilizes SELFIES for $100%$ robust molecular representations and extracts knowledge from the surrounding context of bio-entities in unstructured biological literature. Furthermore, $\mathbf{BioT5}$ distinguishes between structured and unstructured knowledge, leading to more effective utilization of information. After fine-tuning, BioT5 shows superior performance across a wide range of tasks, demonstrating its strong capability of capturing underlying relations and properties of bio-entities. Our code is available at $\href{https://github.com/QizhiPei/BioT5}{Github}$.

Title: RobustGEC: Robust Grammatical Error Correction Against Subtle Context Perturbation. (arXiv:2310.07299v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.07299
• Code URL: https://github.com/hillzhang1999/robustgec
• Copy Paste: [[2310.07299]] RobustGEC: Robust Grammatical Error Correction Against Subtle Context Perturbation(http://arxiv.org/abs/2310.07299)
• Summary:

  Grammatical Error Correction (GEC) systems play a vital role in assisting people with their daily writing tasks. However, users may sometimes come across a GEC system that initially performs well but fails to correct errors when the inputs are slightly modified. To ensure an ideal user experience, a reliable GEC system should have the ability to provide consistent and accurate suggestions when encountering irrelevant context perturbations, which we refer to as context robustness. In this paper, we introduce RobustGEC, a benchmark designed to evaluate the context robustness of GEC systems. RobustGEC comprises 5,000 GEC cases, each with one original error-correct sentence pair and five variants carefully devised by human annotators. Utilizing RobustGEC, we reveal that state-of-the-art GEC systems still lack sufficient robustness against context perturbations. In addition, we propose a simple yet effective method for remitting this issue.

Title: Investigating the Adversarial Robustness of Density Estimation Using the Probability Flow ODE. (arXiv:2310.07084v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.07084
• Code URL: null
• Copy Paste: [[2310.07084]] Investigating the Adversarial Robustness of Density Estimation Using the Probability Flow ODE(http://arxiv.org/abs/2310.07084)
• Summary:

  Beyond their impressive sampling capabilities, score-based diffusion models offer a powerful analysis tool in the form of unbiased density estimation of a query sample under the training data distribution. In this work, we investigate the robustness of density estimation using the probability flow (PF) neural ordinary differential equation (ODE) model against gradient-based likelihood maximization attacks and the relation to sample complexity, where the compressed size of a sample is used as a measure of its complexity. We introduce and evaluate six gradient-based log-likelihood maximization attacks, including a novel reverse integration attack. Our experimental evaluations on CIFAR-10 show that density estimation using the PF ODE is robust against high-complexity, high-likelihood attacks, and that in some cases adversarial samples are semantically meaningful, as expected from a robust estimator.

Title: Robust Safe Reinforcement Learning under Adversarial Disturbances. (arXiv:2310.07207v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.07207
• Code URL: null
• Copy Paste: [[2310.07207]] Robust Safe Reinforcement Learning under Adversarial Disturbances(http://arxiv.org/abs/2310.07207)
• Summary:

  Safety is a primary concern when applying reinforcement learning to real-world control tasks, especially in the presence of external disturbances. However, existing safe reinforcement learning algorithms rarely account for external disturbances, limiting their applicability and robustness in practice. To address this challenge, this paper proposes a robust safe reinforcement learning framework that tackles worst-case disturbances. First, this paper presents a policy iteration scheme to solve for the robust invariant set, i.e., a subset of the safe set, where persistent safety is only possible for states within. The key idea is to establish a two-player zero-sum game by leveraging the safety value function in Hamilton-Jacobi reachability analysis, in which the protagonist (i.e., control inputs) aims to maintain safety and the adversary (i.e., external disturbances) tries to break down safety. This paper proves that the proposed policy iteration algorithm converges monotonically to the maximal robust invariant set. Second, this paper integrates the proposed policy iteration scheme into a constrained reinforcement learning algorithm that simultaneously synthesizes the robust invariant set and uses it for constrained policy optimization. This algorithm tackles both optimality and safety, i.e., learning a policy that attains high rewards while maintaining safety under worst-case disturbances. Experiments on classic control tasks show that the proposed method achieves zero constraint violation with learned worst-case adversarial disturbances, while other baseline algorithms violate the safety constraints substantially. Our proposed method also attains comparable performance as the baselines even in the absence of the adversary.

Title: Hierarchical Decomposition of Prompt-Based Continual Learning: Rethinking Obscured Sub-optimality. (arXiv:2310.07234v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.07234
• Code URL: https://github.com/thu-ml/hide-prompt
• Copy Paste: [[2310.07234]] Hierarchical Decomposition of Prompt-Based Continual Learning: Rethinking Obscured Sub-optimality(http://arxiv.org/abs/2310.07234)
• Summary:

  Prompt-based continual learning is an emerging direction in leveraging pre-trained knowledge for downstream continual learning, and has almost reached the performance pinnacle under supervised pre-training. However, our empirical research reveals that the current strategies fall short of their full potential under the more realistic self-supervised pre-training, which is essential for handling vast quantities of unlabeled data in practice. This is largely due to the difficulty of task-specific knowledge being incorporated into instructed representations via prompt parameters and predicted by uninstructed representations at test time. To overcome the exposed sub-optimality, we conduct a theoretical analysis of the continual learning objective in the context of pre-training, and decompose it into hierarchical components: within-task prediction, task-identity inference, and task-adaptive prediction. Following these empirical and theoretical insights, we propose Hierarchical Decomposition (HiDe-)Prompt, an innovative approach that explicitly optimizes the hierarchical components with an ensemble of task-specific prompts and statistics of both uninstructed and instructed representations, further with the coordination of a contrastive regularization strategy. Our extensive experiments demonstrate the superior performance of HiDe-Prompt and its robustness to pre-training paradigms in continual learning (e.g., up to 15.01% and 9.61% lead on Split CIFAR-100 and Split ImageNet-R, respectively). Our code is available at \url{https://github.com/thu-ml/HiDe-Prompt}.

Title: ADMEOOD: Out-of-Distribution Benchmark for Drug Property Prediction. (arXiv:2310.07253v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.07253
• Code URL: null
• Copy Paste: [[2310.07253]] ADMEOOD: Out-of-Distribution Benchmark for Drug Property Prediction(http://arxiv.org/abs/2310.07253)
• Summary:

  Obtaining accurate and valid information for drug molecules is a crucial and challenging task. However, chemical knowledge and information have been accumulated over the past 100 years from various regions, laboratories, and experimental purposes. Little has been explored in terms of the out-of-distribution (OOD) problem with noise and inconsistency, which may lead to weak robustness and unsatisfied performance. This study proposes a novel benchmark ADMEOOD, a systematic OOD dataset curator and benchmark specifically designed for drug property prediction. ADMEOOD obtained 27 ADME (Absorption, Distribution, Metabolism, Excretion) drug properties from Chembl and relevant literature. Additionally, it includes two kinds of OOD data shifts: Noise Shift and Concept Conflict Drift (CCD). Noise Shift responds to the noise level by categorizing the environment into different confidence levels. On the other hand, CCD describes the data which has inconsistent label among the original data. Finally, it tested on a variety of domain generalization models, and the experimental results demonstrate the effectiveness of the proposed partition method in ADMEOOD: ADMEOOD demonstrates a significant difference performance between in-distribution and out-of-distribution data. Moreover, ERM (Empirical Risk Minimization) and other models exhibit distinct trends in performance across different domains and measurement types.

biometric

Title: Exploiting Causal Graph Priors with Posterior Sampling for Reinforcement Learning. (arXiv:2310.07518v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.07518
• Code URL: null
• Copy Paste: [[2310.07518]] Exploiting Causal Graph Priors with Posterior Sampling for Reinforcement Learning(http://arxiv.org/abs/2310.07518)
• Summary:

  Posterior sampling allows the exploitation of prior knowledge of the environment's transition dynamics to improve the sample efficiency of reinforcement learning. The prior is typically specified as a class of parametric distributions, a task that can be cumbersome in practice, often resulting in the choice of uninformative priors. In this work, we propose a novel posterior sampling approach in which the prior is given as a (partial) causal graph over the environment's variables. The latter is often more natural to design, such as listing known causal dependencies between biometric features in a medical treatment study. Specifically, we propose a hierarchical Bayesian procedure, called C-PSRL, simultaneously learning the full causal graph at the higher level and the parameters of the resulting factored dynamics at the lower level. For this procedure, we provide an analysis of its Bayesian regret, which explicitly connects the regret rate with the degree of prior knowledge. Our numerical evaluation conducted in illustrative domains confirms that C-PSRL strongly improves the efficiency of posterior sampling with an uninformative prior while performing close to posterior sampling with the full causal graph.

steal

extraction

Title: Performance Analysis of Various EfficientNet Based U-Net++ Architecture for Automatic Building Extraction from High Resolution Satellite Images. (arXiv:2310.06847v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.06847
• Code URL: null
• Copy Paste: [[2310.06847]] Performance Analysis of Various EfficientNet Based U-Net++ Architecture for Automatic Building Extraction from High Resolution Satellite Images(http://arxiv.org/abs/2310.06847)
• Summary:

  Building extraction is an essential component of study in the science of remote sensing, and applications for building extraction heavily rely on semantic segmentation of high-resolution remote sensing imagery. Semantic information extraction gap constraints in the present deep learning based approaches, however can result in inadequate segmentation outcomes. To address this issue and extract buildings with high accuracy, various efficientNet backbone based U-Net++ has been proposed in this study. The designed network, based on U-Net, can improve the sensitivity of the model by deep supervision, voluminous redesigned skip-connections and hence reducing the influence of irrelevant feature areas in the background. Various effecientNet backbone based encoders have been employed when training the network to enhance the capacity of the model to extract more relevant feature. According on the experimental findings, the suggested model significantly outperforms previous cutting-edge approaches. Among the 5 efficientNet variation Unet++ based on efficientb4 achieved the best result by scoring mean accuracy of 92.23%, mean iou of 88.32%, and mean precision of 93.2% on publicly available Massachusetts building dataset and thus showing the promises of the model for automatic building extraction from high resolution satellite images.

Title: A Novel Voronoi-based Convolutional Neural Network Framework for Pushing Person Detection in Crowd Videos. (arXiv:2310.07416v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.07416
• Code URL: https://github.com/pedestriandynamics/vcnn4pude
• Copy Paste: [[2310.07416]] A Novel Voronoi-based Convolutional Neural Network Framework for Pushing Person Detection in Crowd Videos(http://arxiv.org/abs/2310.07416)
• Summary:

  Analyzing the microscopic dynamics of pushing behavior within crowds can offer valuable insights into crowd patterns and interactions. By identifying instances of pushing in crowd videos, a deeper understanding of when, where, and why such behavior occurs can be achieved. This knowledge is crucial to creating more effective crowd management strategies, optimizing crowd flow, and enhancing overall crowd experiences. However, manually identifying pushing behavior at the microscopic level is challenging, and the existing automatic approaches cannot detect such microscopic behavior. Thus, this article introduces a novel automatic framework for identifying pushing in videos of crowds on a microscopic level. The framework comprises two main components: i) Feature extraction and ii) Video labeling. In the feature extraction component, a new Voronoi-based method is developed for determining the local regions associated with each person in the input video. Subsequently, these regions are fed into EfficientNetV1B0 Convolutional Neural Network to extract the deep features of each person over time. In the second component, a combination of a fully connected layer with a Sigmoid activation function is employed to analyze these deep features and annotate the individuals involved in pushing within the video. The framework is trained and evaluated on a new dataset created using six real-world experiments, including their corresponding ground truths. The experimental findings indicate that the suggested framework outperforms seven baseline methods that are employed for comparative analysis purposes.

Title: LeakyOhm: Secret Bits Extraction using Impedance Analysis. (arXiv:2310.07014v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.07014
• Code URL: null
• Copy Paste: [[2310.07014]] LeakyOhm: Secret Bits Extraction using Impedance Analysis(http://arxiv.org/abs/2310.07014)
• Summary:

  The threat of physical side-channel attacks and their countermeasures is a widely researched field. Most physical side-channel attacks rely on the unavoidable influence of computation or storage on voltage or current fluctuations. Such data-dependent influence can be exploited by, for instance, power or electromagnetic analysis. In this work, we introduce a novel non-invasive physical side-channel attack, which exploits the data-dependent changes in the impedance of the chip. Our attack relies on the fact that the temporarily stored contents in registers alter the physical characteristics of the circuit, which results in changes in the die's impedance. To sense such impedance variations, we deploy a well-known RF/microwave method called scattering parameter analysis, in which we inject sine wave signals with high frequencies into the system's power distribution network (PDN) and measure the echo of the signals. We demonstrate that according to the content bits and physical location of a register, the reflected signal is modulated differently at various frequency points enabling the simultaneous and independent probing of individual registers. Such side-channel leakage violates the $t$-probing security model assumption used in masking, which is a prominent side-channel countermeasure. To validate our claims, we mount non-profiled and profiled impedance analysis attacks on hardware implementations of unprotected and high-order masked AES. We show that in the case of profiled attack, only a single trace is required to recover the secret key. Finally, we discuss how a specific class of hiding countermeasures might be effective against impedance leakage.

Title: On sparse regression, Lp-regularization, and automated model discovery. (arXiv:2310.06872v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.06872
• Code URL: null
• Copy Paste: [[2310.06872]] On sparse regression, Lp-regularization, and automated model discovery(http://arxiv.org/abs/2310.06872)
• Summary:

  Sparse regression and feature extraction are the cornerstones of knowledge discovery from massive data. Their goal is to discover interpretable and predictive models that provide simple relationships among scientific variables. While the statistical tools for model discovery are well established in the context of linear regression, their generalization to nonlinear regression in material modeling is highly problem-specific and insufficiently understood. Here we explore the potential of neural networks for automatic model discovery and induce sparsity by a hybrid approach that combines two strategies: regularization and physical constraints. We integrate the concept of Lp regularization for subset selection with constitutive neural networks that leverage our domain knowledge in kinematics and thermodynamics. We train our networks with both, synthetic and real data, and perform several thousand discovery runs to infer common guidelines and trends: L2 regularization or ridge regression is unsuitable for model discovery; L1 regularization or lasso promotes sparsity, but induces strong bias; only L0 regularization allows us to transparently fine-tune the trade-off between interpretability and predictability, simplicity and accuracy, and bias and variance. With these insights, we demonstrate that Lp regularized constitutive neural networks can simultaneously discover both, interpretable models and physically meaningful parameters. We anticipate that our findings will generalize to alternative discovery techniques such as sparse and symbolic regression, and to other domains such as biology, chemistry, or medicine. Our ability to automatically discover material models from data could have tremendous applications in generative material design and open new opportunities to manipulate matter, alter properties of existing materials, and discover new materials with user-defined properties.

Title: Classification of Dysarthria based on the Levels of Severity. A Systematic Review. (arXiv:2310.07264v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.07264
• Code URL: null
• Copy Paste: [[2310.07264]] Classification of Dysarthria based on the Levels of Severity(http://arxiv.org/abs/2310.07264)
• Summary:

  Dysarthria is a neurological speech disorder that can significantly impact affected individuals' communication abilities and overall quality of life. The accurate and objective classification of dysarthria and the determination of its severity are crucial for effective therapeutic intervention. While traditional assessments by speech-language pathologists (SLPs) are common, they are often subjective, time-consuming, and can vary between practitioners. Emerging machine learning-based models have shown the potential to provide a more objective dysarthria assessment, enhancing diagnostic accuracy and reliability. This systematic review aims to comprehensively analyze current methodologies for classifying dysarthria based on severity levels. Specifically, this review will focus on determining the most effective set and type of features that can be used for automatic patient classification and evaluating the best AI techniques for this purpose. We will systematically review the literature on the automatic classification of dysarthria severity levels. Sources of information will include electronic databases and grey literature. Selection criteria will be established based on relevance to the research questions. Data extraction will include methodologies used, the type of features extracted for classification, and AI techniques employed. The findings of this systematic review will contribute to the current understanding of dysarthria classification, inform future research, and support the development of improved diagnostic tools. The implications of these findings could be significant in advancing patient care and improving therapeutic outcomes for individuals affected by dysarthria.

Title: Multichannel consecutive data cross-extraction with 1DCNN-attention for diagnosis of power transformer. (arXiv:2310.07323v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.07323
• Code URL: null
• Copy Paste: [[2310.07323]] Multichannel consecutive data cross-extraction with 1DCNN-attention for diagnosis of power transformer(http://arxiv.org/abs/2310.07323)
• Summary:

  Power transformer plays a critical role in grid infrastructure, and its diagnosis is paramount for maintaining stable operation. However, the current methods for transformer diagnosis focus on discrete dissolved gas analysis, neglecting deep feature extraction of multichannel consecutive data. The unutilized sequential data contains the significant temporal information reflecting the transformer condition. In light of this, the structure of multichannel consecutive data cross-extraction (MCDC) is proposed in this article in order to comprehensively exploit the intrinsic characteristic and evaluate the states of transformer. Moreover, for the better accommodation in scenario of transformer diagnosis, one dimensional convolution neural network attention (1DCNN-attention) mechanism is introduced and offers a more efficient solution given the simplified spatial complexity. Finally, the effectiveness of MCDC and the superior generalization ability, compared with other algorithms, are validated in experiments conducted on a dataset collected from real operation cases of power transformer. Additionally, the better stability of 1DCNN-attention has also been certified.

membership infer

federate

Title: FedMFS: Federated Multimodal Fusion Learning with Selective Modality Communication. (arXiv:2310.07048v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.07048
• Code URL: null
• Copy Paste: [[2310.07048]] FedMFS: Federated Multimodal Fusion Learning with Selective Modality Communication(http://arxiv.org/abs/2310.07048)
• Summary:

  Federated learning (FL) is a distributed machine learning (ML) paradigm that enables clients to collaborate without accessing, infringing upon, or leaking original user data by sharing only model parameters. In the Internet of Things (IoT), edge devices are increasingly leveraging multimodal data compositions and fusion paradigms to enhance model performance. However, in FL applications, two main challenges remain open: (i) addressing the issues caused by heterogeneous clients lacking specific modalities and (ii) devising an optimal modality upload strategy to minimize communication overhead while maximizing learning performance. In this paper, we propose Federated Multimodal Fusion learning with Selective modality communication (FedMFS), a new multimodal fusion FL methodology that can tackle the above mentioned challenges. The key idea is to utilize Shapley values to quantify each modality's contribution and modality model size to gauge communication overhead, so that each client can selectively upload the modality models to the server for aggregation. This enables FedMFS to flexibly balance performance against communication costs, depending on resource constraints and applications. Experiments on real-world multimodal datasets demonstrate the effectiveness of FedMFS, achieving comparable accuracy while reducing communication overhead by one twentieth compared to baselines.

Title: Federated Generalization via Information-Theoretic Distribution Diversification. (arXiv:2310.07171v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.07171
• Code URL: null
• Copy Paste: [[2310.07171]] Federated Generalization via Information-Theoretic Distribution Diversification(http://arxiv.org/abs/2310.07171)
• Summary:

  Federated Learning (FL) has surged in prominence due to its capability of collaborative model training without direct data sharing. However, the vast disparity in local data distributions among clients, often termed the non-Independent Identically Distributed (non-IID) challenge, poses a significant hurdle to FL's generalization efficacy. The scenario becomes even more complex when not all clients participate in the training process, a common occurrence due to unstable network connections or limited computational capacities. This can greatly complicate the assessment of the trained models' generalization abilities. While a plethora of recent studies has centered on the generalization gap pertaining to unseen data from participating clients with diverse distributions, the divergence between the training distributions of participating clients and the testing distributions of non-participating ones has been largely overlooked. In response, our paper unveils an information-theoretic generalization framework for FL. Specifically, it quantifies generalization errors by evaluating the information entropy of local distributions and discerning discrepancies across these distributions. Inspired by our deduced generalization bounds, we introduce a weighted aggregation approach and a duo of client selection strategies. These innovations aim to bolster FL's generalization prowess by encompassing a more varied set of client data distributions. Our extensive empirical evaluations reaffirm the potency of our proposed methods, aligning seamlessly with our theoretical construct.

Title: RaftFed: A Lightweight Federated Learning Framework for Vehicular Crowd Intelligence. (arXiv:2310.07268v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.07268
• Code URL: null
• Copy Paste: [[2310.07268]] RaftFed: A Lightweight Federated Learning Framework for Vehicular Crowd Intelligence(http://arxiv.org/abs/2310.07268)
• Summary:

  Vehicular crowd intelligence (VCI) is an emerging research field. Facilitated by state-of-the-art vehicular ad-hoc networks and artificial intelligence, various VCI applications come to place, e.g., collaborative sensing, positioning, and mapping. The collaborative property of VCI applications generally requires data to be shared among participants, thus forming network-wide intelligence. How to fulfill this process without compromising data privacy remains a challenging issue. Although federated learning (FL) is a promising tool to solve the problem, adapting conventional FL frameworks to VCI is nontrivial. First, the centralized model aggregation is unreliable in VCI because of the existence of stragglers with unfavorable channel conditions. Second, existing FL schemes are vulnerable to Non-IID data, which is intensified by the data heterogeneity in VCI. This paper proposes a novel federated learning framework called RaftFed to facilitate privacy-preserving VCI. The experimental results show that RaftFed performs better than baselines regarding communication overhead, model accuracy, and model convergence.

Title: Histopathological Image Classification and Vulnerability Analysis using Federated Learning. (arXiv:2310.07380v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.07380
• Code URL: null
• Copy Paste: [[2310.07380]] Histopathological Image Classification and Vulnerability Analysis using Federated Learning(http://arxiv.org/abs/2310.07380)
• Summary:

  Healthcare is one of the foremost applications of machine learning (ML). Traditionally, ML models are trained by central servers, which aggregate data from various distributed devices to forecast the results for newly generated data. This is a major concern as models can access sensitive user information, which raises privacy concerns. A federated learning (FL) approach can help address this issue: A global model sends its copy to all clients who train these copies, and the clients send the updates (weights) back to it. Over time, the global model improves and becomes more accurate. Data privacy is protected during training, as it is conducted locally on the clients' devices.

  However, the global model is susceptible to data poisoning. We develop a privacy-preserving FL technique for a skin cancer dataset and show that the model is prone to data poisoning attacks. Ten clients train the model, but one of them intentionally introduces flipped labels as an attack. This reduces the accuracy of the global model. As the percentage of label flipping increases, there is a noticeable decrease in accuracy. We use a stochastic gradient descent optimization algorithm to find the most optimal accuracy for the model. Although FL can protect user privacy for healthcare diagnostics, it is also vulnerable to data poisoning, which must be addressed.

Title: Sample-Driven Federated Learning for Energy-Efficient and Real-Time IoT Sensing. (arXiv:2310.07497v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.07497
• Code URL: https://github.com/skyd-fl/scfl
• Copy Paste: [[2310.07497]] Sample-Driven Federated Learning for Energy-Efficient and Real-Time IoT Sensing(http://arxiv.org/abs/2310.07497)
• Summary:

  In the domain of Federated Learning (FL) systems, recent cutting-edge methods heavily rely on ideal conditions convergence analysis. Specifically, these approaches assume that the training datasets on IoT devices possess similar attributes to the global data distribution. However, this approach fails to capture the full spectrum of data characteristics in real-time sensing FL systems. In order to overcome this limitation, we suggest a new approach system specifically designed for IoT networks with real-time sensing capabilities. Our approach takes into account the generalization gap due to the user's data sampling process. By effectively controlling this sampling process, we can mitigate the overfitting issue and improve overall accuracy. In particular, We first formulate an optimization problem that harnesses the sampling process to concurrently reduce overfitting while maximizing accuracy. In pursuit of this objective, our surrogate optimization problem is adept at handling energy efficiency while optimizing the accuracy with high generalization. To solve the optimization problem with high complexity, we introduce an online reinforcement learning algorithm, named Sample-driven Control for Federated Learning (SCFL) built on the Soft Actor-Critic (A2C) framework. This enables the agent to dynamically adapt and find the global optima even in changing environments. By leveraging the capabilities of SCFL, our system offers a promising solution for resource allocation in FL systems with real-time sensing capabilities.

fair

Title: Anchor-based Multi-view Subspace Clustering with Hierarchical Feature Descent. (arXiv:2310.07166v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.07166
• Code URL: null
• Copy Paste: [[2310.07166]] Anchor-based Multi-view Subspace Clustering with Hierarchical Feature Descent(http://arxiv.org/abs/2310.07166)
• Summary:

  Multi-view clustering has attracted growing attention owing to its capabilities of aggregating information from various sources and its promising horizons in public affairs. Up till now, many advanced approaches have been proposed in recent literature. However, there are several ongoing difficulties to be tackled. One common dilemma occurs while attempting to align the features of different views. We dig out as well as deploy the dependency amongst views through hierarchical feature descent, which leads to a common latent space( STAGE 1). This latent space, for the first time of its kind, is regarded as a 'resemblance space', as it reveals certain correlations and dependencies of different views. To be exact, the one-hot encoding of a category can also be referred to as a resemblance space in its terminal phase. Moreover, due to the intrinsic fact that most of the existing multi-view clustering algorithms stem from k-means clustering and spectral clustering, this results in cubic time complexity w.r.t. the number of the objects. However, we propose Anchor-based Multi-view Subspace Clustering with Hierarchical Feature Descent(MVSC-HFD) to further reduce the computing complexity to linear time cost through a unified sampling strategy in resemblance space( STAGE 2), followed by subspace clustering to learn the representation collectively( STAGE 3). Extensive experimental results on public benchmark datasets demonstrate that our proposed model consistently outperforms the state-of-the-art techniques.

Title: Improving Fairness-Accuracy tradeoff with few Test Samples under Covariate Shift. (arXiv:2310.07535v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.07535
• Code URL: null
• Copy Paste: [[2310.07535]] Improving Fairness-Accuracy tradeoff with few Test Samples under Covariate Shift(http://arxiv.org/abs/2310.07535)
• Summary:

  Covariate shift in the test data can significantly downgrade both the accuracy and the fairness performance of the model. Ensuring fairness across different sensitive groups in such settings is of paramount importance due to societal implications like criminal justice. We operate under the unsupervised regime where only a small set of unlabeled test samples along with a labeled training set is available. Towards this problem, we make three contributions. First is a novel composite weighted entropy based objective for prediction accuracy which is optimized along with a representation matching loss for fairness. We experimentally verify that optimizing with our loss formulation outperforms a number of state-of-the-art baselines in the pareto sense with respect to the fairness-accuracy tradeoff on several standard datasets. Our second contribution is a new setting we term Asymmetric Covariate Shift that, to the best of our knowledge, has not been studied before. Asymmetric covariate shift occurs when distribution of covariates of one group shifts significantly compared to the other groups and this happens when a dominant group is over-represented. While this setting is extremely challenging for current baselines, We show that our proposed method significantly outperforms them. Our third contribution is theoretical, where we show that our weighted entropy term along with prediction loss on the training set approximates test loss under covariate shift. Empirically and through formal sample complexity bounds, we show that this approximation to the unseen test loss does not depend on importance sampling variance which affects many other baselines.

interpretability

Title: On the Interpretability of Part-Prototype Based Classifiers: A Human Centric Analysis. (arXiv:2310.06966v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.06966
• Code URL: null
• Copy Paste: [[2310.06966]] On the Interpretability of Part-Prototype Based Classifiers: A Human Centric Analysis(http://arxiv.org/abs/2310.06966)
• Summary:

  Part-prototype networks have recently become methods of interest as an interpretable alternative to many of the current black-box image classifiers. However, the interpretability of these methods from the perspective of human users has not been sufficiently explored. In this work, we have devised a framework for evaluating the interpretability of part-prototype-based models from a human perspective. The proposed framework consists of three actionable metrics and experiments. To demonstrate the usefulness of our framework, we performed an extensive set of experiments using Amazon Mechanical Turk. They not only show the capability of our framework in assessing the interpretability of various part-prototype-based models, but they also are, to the best of our knowledge, the most comprehensive work on evaluating such methods in a unified framework.

Title: Guided Attention for Interpretable Motion Captioning. (arXiv:2310.07324v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.07324
• Code URL: https://github.com/rd20karim/m2t-interpretable
• Copy Paste: [[2310.07324]] Guided Attention for Interpretable Motion Captioning(http://arxiv.org/abs/2310.07324)
• Summary:

  While much effort has been invested in generating human motion from text, relatively few studies have been dedicated to the reverse direction, that is, generating text from motion. Much of the research focuses on maximizing generation quality without any regard for the interpretability of the architectures, particularly regarding the influence of particular body parts in the generation and the temporal synchronization of words with specific movements and actions. This study explores the combination of movement encoders with spatio-temporal attention models and proposes strategies to guide the attention during training to highlight perceptually pertinent areas of the skeleton in time. We show that adding guided attention with adaptive gate leads to interpretable captioning while improving performance compared to higher parameter-count non-interpretable SOTA systems. On the KIT MLD dataset, we obtain a BLEU@4 of 24.4% (SOTA+6%), a ROUGE-L of 58.30% (SOTA +14.1%), a CIDEr of 112.10 (SOTA +32.6) and a Bertscore of 41.20% (SOTA +18.20%). On HumanML3D, we obtain a BLEU@4 of 25.00 (SOTA +2.7%), a ROUGE-L score of 55.4% (SOTA +6.1%), a CIDEr of 61.6 (SOTA -10.9%), a Bertscore of 40.3% (SOTA +2.5%). Our code implementation and reproduction details will be soon available at https://github.com/rd20karim/M2T-Interpretable/tree/main.

Title: An Adversarial Example for Direct Logit Attribution: Memory Management in gelu-4l. (arXiv:2310.07325v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.07325
• Code URL: null
• Copy Paste: [[2310.07325]] An Adversarial Example for Direct Logit Attribution: Memory Management in gelu-4l(http://arxiv.org/abs/2310.07325)
• Summary:

  We provide concrete evidence for memory management in a 4-layer transformer. Specifically, we identify clean-up behavior, in which model components consistently remove the output of preceeding components during a forward pass. Our findings suggest that the interpretability technique Direct Logit Attribution provides misleading results. We show explicit examples where this technique is inaccurate, as it does not account for clean-up behavior.

explainability

watermark

diffusion

Title: ObjectComposer: Consistent Generation of Multiple Objects Without Fine-tuning. (arXiv:2310.06968v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.06968
• Code URL: null
• Copy Paste: [[2310.06968]] ObjectComposer: Consistent Generation of Multiple Objects Without Fine-tuning(http://arxiv.org/abs/2310.06968)
• Summary:

  Recent text-to-image generative models can generate high-fidelity images from text prompts. However, these models struggle to consistently generate the same objects in different contexts with the same appearance. Consistent object generation is important to many downstream tasks like generating comic book illustrations with consistent characters and setting. Numerous approaches attempt to solve this problem by extending the vocabulary of diffusion models through fine-tuning. However, even lightweight fine-tuning approaches can be prohibitively expensive to run at scale and in real-time. We introduce a method called ObjectComposer for generating compositions of multiple objects that resemble user-specified images. Our approach is training-free, leveraging the abilities of preexisting models. We build upon the recent BLIP-Diffusion model, which can generate images of single objects specified by reference images. ObjectComposer enables the consistent generation of compositions containing multiple specific objects simultaneously, all without modifying the weights of the underlying models.

Title: Denoising Task Routing for Diffusion Models. (arXiv:2310.07138v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.07138
• Code URL: null
• Copy Paste: [[2310.07138]] Denoising Task Routing for Diffusion Models(http://arxiv.org/abs/2310.07138)
• Summary:

  Diffusion models generate highly realistic images through learning a multi-step denoising process, naturally embodying the principles of multi-task learning (MTL). Despite the inherent connection between diffusion models and MTL, there remains an unexplored area in designing neural architectures that explicitly incorporate MTL into the framework of diffusion models. In this paper, we present Denoising Task Routing (DTR), a simple add-on strategy for existing diffusion model architectures to establish distinct information pathways for individual tasks within a single architecture by selectively activating subsets of channels in the model. What makes DTR particularly compelling is its seamless integration of prior knowledge of denoising tasks into the framework: (1) Task Affinity: DTR activates similar channels for tasks at adjacent timesteps and shifts activated channels as sliding windows through timesteps, capitalizing on the inherent strong affinity between tasks at adjacent timesteps. (2) Task Weights: During the early stages (higher timesteps) of the denoising process, DTR assigns a greater number of task-specific channels, leveraging the insight that diffusion models prioritize reconstructing global structure and perceptually rich contents in earlier stages, and focus on simple noise removal in later stages. Our experiments demonstrate that DTR consistently enhances the performance of diffusion models across various evaluation protocols, all without introducing additional parameters. Furthermore, DTR contributes to accelerating convergence during training. Finally, we show the complementarity between our architectural approach and existing MTL optimization techniques, providing a more complete view of MTL within the context of diffusion training.

Title: Uni-paint: A Unified Framework for Multimodal Image Inpainting with Pretrained Diffusion Model. (arXiv:2310.07222v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.07222
• Code URL: https://github.com/ysy31415/unipaint
• Copy Paste: [[2310.07222]] Uni-paint: A Unified Framework for Multimodal Image Inpainting with Pretrained Diffusion Model(http://arxiv.org/abs/2310.07222)
• Summary:

  Recently, text-to-image denoising diffusion probabilistic models (DDPMs) have demonstrated impressive image generation capabilities and have also been successfully applied to image inpainting. However, in practice, users often require more control over the inpainting process beyond textual guidance, especially when they want to composite objects with customized appearance, color, shape, and layout. Unfortunately, existing diffusion-based inpainting methods are limited to single-modal guidance and require task-specific training, hindering their cross-modal scalability. To address these limitations, we propose Uni-paint, a unified framework for multimodal inpainting that offers various modes of guidance, including unconditional, text-driven, stroke-driven, exemplar-driven inpainting, as well as a combination of these modes. Furthermore, our Uni-paint is based on pretrained Stable Diffusion and does not require task-specific training on specific datasets, enabling few-shot generalizability to customized images. We have conducted extensive qualitative and quantitative evaluations that show our approach achieves comparable results to existing single-modal methods while offering multimodal inpainting capabilities not available in other methods. Code will be available at https://github.com/ysy31415/unipaint.

Title: Multi-Concept T2I-Zero: Tweaking Only The Text Embeddings and Nothing Else. (arXiv:2310.07419v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.07419
• Code URL: null
• Copy Paste: [[2310.07419]] Multi-Concept T2I-Zero: Tweaking Only The Text Embeddings and Nothing Else(http://arxiv.org/abs/2310.07419)
• Summary:

  Recent advances in text-to-image diffusion models have enabled the photorealistic generation of images from text prompts. Despite the great progress, existing models still struggle to generate compositional multi-concept images naturally, limiting their ability to visualize human imagination. While several recent works have attempted to address this issue, they either introduce additional training or adopt guidance at inference time. In this work, we consider a more ambitious goal: natural multi-concept generation using a pre-trained diffusion model, and with almost no extra cost. To achieve this goal, we identify the limitations in the text embeddings used for the pre-trained text-to-image diffusion models. Specifically, we observe concept dominance and non-localized contribution that severely degrade multi-concept generation performance. We further design a minimal low-cost solution that overcomes the above issues by tweaking (not re-training) the text embeddings for more realistic multi-concept text-to-image generation. Our Correction by Similarities method tweaks the embedding of concepts by collecting semantic features from most similar tokens to localize the contribution. To avoid mixing features of concepts, we also apply Cross-Token Non-Maximum Suppression, which excludes the overlap of contributions from different concepts. Experiments show that our approach outperforms previous methods in text-to-image, image manipulation, and personalization tasks, despite not introducing additional training or inference costs to the diffusion steps.

Title: Monsters in the Dark: Sanitizing Hidden Threats with Diffusion Models. (arXiv:2310.06951v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.06951
• Code URL: null
• Copy Paste: [[2310.06951]] Monsters in the Dark: Sanitizing Hidden Threats with Diffusion Models(http://arxiv.org/abs/2310.06951)
• Summary:

  Steganography is the art of hiding information in plain sight. This form of covert communication can be used by bad actors to propagate malware, exfiltrate victim data, and communicate with other bad actors. Current image steganography defenses rely upon steganalysis, or the detection of hidden messages. These methods, however, are non-blind as they require information about known steganography techniques and are easily bypassed. Recent work has instead focused on a defense mechanism known as sanitization, which eliminates hidden information from images. In this work, we introduce a novel blind deep learning steganography sanitization method that utilizes a diffusion model framework to sanitize universal and dependent steganography (DM-SUDS), which both sanitizes and preserves image quality. We evaluate this approach against state-of-the-art deep learning sanitization frameworks and provide further detailed analysis through an ablation study. DM-SUDS outperforms previous sanitization methods and improves image preservation MSE by 71.32%, PSNR by 22.43% and SSIM by 17.30%. This is the first blind deep learning image sanitization framework to meet these image quality results.

Title: Imitation Learning from Purified Demonstration. (arXiv:2310.07143v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.07143
• Code URL: null
• Copy Paste: [[2310.07143]] Imitation Learning from Purified Demonstration(http://arxiv.org/abs/2310.07143)
• Summary:

  Imitation learning has emerged as a promising approach for addressing sequential decision-making problems, with the assumption that expert demonstrations are optimal. However, in real-world scenarios, expert demonstrations are often imperfect, leading to challenges in effectively applying imitation learning. While existing research has focused on optimizing with imperfect demonstrations, the training typically requires a certain proportion of optimal demonstrations to guarantee performance. To tackle these problems, we propose to purify the potential perturbations in imperfect demonstrations and subsequently conduct imitation learning from purified demonstrations. Motivated by the success of diffusion models, we introduce a two-step purification via the diffusion process. In the first step, we apply a forward diffusion process to effectively smooth out the potential perturbations in imperfect demonstrations by introducing additional noise. Subsequently, a reverse generative process is utilized to recover the optimal expert demonstrations from the diffused ones. We provide theoretical evidence supporting our approach, demonstrating that total variance distance between the purified and optimal demonstration distributions can be upper-bounded. The evaluation results on MuJoCo demonstrate the effectiveness of our method from different aspects.

Title: Generative Modeling on Manifolds Through Mixture of Riemannian Diffusion Processes. (arXiv:2310.07216v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.07216
• Code URL: null
• Copy Paste: [[2310.07216]] Generative Modeling on Manifolds Through Mixture of Riemannian Diffusion Processes(http://arxiv.org/abs/2310.07216)
• Summary:

  Learning the distribution of data on Riemannian manifolds is crucial for modeling data from non-Euclidean space, which is required by many applications from diverse scientific fields. Yet, existing generative models on manifolds suffer from expensive divergence computation or rely on approximations of heat kernel. These limitations restrict their applicability to simple geometries and hinder scalability to high dimensions. In this work, we introduce the Riemannian Diffusion Mixture, a principled framework for building a generative process on manifolds as a mixture of endpoint-conditioned diffusion processes instead of relying on the denoising approach of previous diffusion models, for which the generative process is characterized by its drift guiding toward the most probable endpoint with respect to the geometry of the manifold. We further propose a simple yet efficient training objective for learning the mixture process, that is readily applicable to general manifolds. Our method outperforms previous generative models on various manifolds while scaling to high dimensions and requires a dramatically reduced number of in-training simulation steps for general manifolds.

Title: Score Regularized Policy Optimization through Diffusion Behavior. (arXiv:2310.07297v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.07297
• Code URL: https://github.com/thu-ml/srpo
• Copy Paste: [[2310.07297]] Score Regularized Policy Optimization through Diffusion Behavior(http://arxiv.org/abs/2310.07297)
• Summary:

  Recent developments in offline reinforcement learning have uncovered the immense potential of diffusion modeling, which excels at representing heterogeneous behavior policies. However, sampling from diffusion policies is considerably slow because it necessitates tens to hundreds of iterative inference steps for one action. To address this issue, we propose to extract an efficient deterministic inference policy from critic models and pretrained diffusion behavior models, leveraging the latter to directly regularize the policy gradient with the behavior distribution's score function during optimization. Our method enjoys powerful generative capabilities of diffusion modeling while completely circumventing the computationally intensive and time-consuming diffusion sampling scheme, both during training and evaluation. Extensive results on D4RL tasks show that our method boosts action sampling speed by more than 25 times compared with various leading diffusion-based methods in locomotion tasks, while still maintaining state-of-the-art performance.

noise learning

Title: Why Does Sharpness-Aware Minimization Generalize Better Than SGD?. (arXiv:2310.07269v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.07269
• Code URL: null
• Copy Paste: [[2310.07269]] Why Does Sharpness-Aware Minimization Generalize Better Than SGD?(http://arxiv.org/abs/2310.07269)
• Summary:

  The challenge of overfitting, in which the model memorizes the training data and fails to generalize to test data, has become increasingly significant in the training of large neural networks. To tackle this challenge, Sharpness-Aware Minimization (SAM) has emerged as a promising training method, which can improve the generalization of neural networks even in the presence of label noise. However, a deep understanding of how SAM works, especially in the setting of nonlinear neural networks and classification tasks, remains largely missing. This paper fills this gap by demonstrating why SAM generalizes better than Stochastic Gradient Descent (SGD) for a certain data model and two-layer convolutional ReLU networks. The loss landscape of our studied problem is nonsmooth, thus current explanations for the success of SAM based on the Hessian information are insufficient. Our result explains the benefits of SAM, particularly its ability to prevent noise learning in the early stages, thereby facilitating more effective learning of features. Experiments on both synthetic and real data corroborate our theory.

data-free

transformer

Title: BodyFormer: Semantics-guided 3D Body Gesture Synthesis with Transformer. (arXiv:2310.06851v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.06851
• Code URL: null
• Copy Paste: [[2310.06851]] BodyFormer: Semantics-guided 3D Body Gesture Synthesis with Transformer(http://arxiv.org/abs/2310.06851)
• Summary:

  Automatic gesture synthesis from speech is a topic that has attracted researchers for applications in remote communication, video games and Metaverse. Learning the mapping between speech and 3D full-body gestures is difficult due to the stochastic nature of the problem and the lack of a rich cross-modal dataset that is needed for training. In this paper, we propose a novel transformer-based framework for automatic 3D body gesture synthesis from speech. To learn the stochastic nature of the body gesture during speech, we propose a variational transformer to effectively model a probabilistic distribution over gestures, which can produce diverse gestures during inference. Furthermore, we introduce a mode positional embedding layer to capture the different motion speeds in different speaking modes. To cope with the scarcity of data, we design an intra-modal pre-training scheme that can learn the complex mapping between the speech and the 3D gesture from a limited amount of data. Our system is trained with either the Trinity speech-gesture dataset or the Talking With Hands 16.2M dataset. The results show that our system can produce more realistic, appropriate, and diverse body gestures compared to existing state-of-the-art approaches.

Title: Computational Pathology at Health System Scale -- Self-Supervised Foundation Models from Three Billion Images. (arXiv:2310.07033v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.07033
• Code URL: null
• Copy Paste: [[2310.07033]] Computational Pathology at Health System Scale -- Self-Supervised Foundation Models from Three Billion Images(http://arxiv.org/abs/2310.07033)
• Summary:

  Recent breakthroughs in self-supervised learning have enabled the use of large unlabeled datasets to train visual foundation models that can generalize to a variety of downstream tasks. While this training paradigm is well suited for the medical domain where annotations are scarce, large-scale pre-training in the medical domain, and in particular pathology, has not been extensively studied. Previous work in self-supervised learning in pathology has leveraged smaller datasets for both pre-training and evaluating downstream performance. The aim of this project is to train the largest academic foundation model and benchmark the most prominent self-supervised learning algorithms by pre-training and evaluating downstream performance on large clinical pathology datasets. We collected the largest pathology dataset to date, consisting of over 3 billion images from over 423 thousand microscopy slides. We compared pre-training of visual transformer models using the masked autoencoder (MAE) and DINO algorithms. We evaluated performance on six clinically relevant tasks from three anatomic sites and two institutions: breast cancer detection, inflammatory bowel disease detection, breast cancer estrogen receptor prediction, lung adenocarcinoma EGFR mutation prediction, and lung cancer immunotherapy response prediction. Our results demonstrate that pre-training on pathology data is beneficial for downstream performance compared to pre-training on natural images. Additionally, the DINO algorithm achieved better generalization performance across all tasks tested. The presented results signify a phase change in computational pathology research, paving the way into a new era of more performant models based on large-scale, parallel pre-training at the billion-image scale.

Title: Multiview Transformer: Rethinking Spatial Information in Hyperspectral Image Classification. (arXiv:2310.07186v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.07186
• Code URL: null
• Copy Paste: [[2310.07186]] Multiview Transformer: Rethinking Spatial Information in Hyperspectral Image Classification(http://arxiv.org/abs/2310.07186)
• Summary:

  Identifying the land cover category for each pixel in a hyperspectral image (HSI) relies on spectral and spatial information. An HSI cuboid with a specific patch size is utilized to extract spatial-spectral feature representation for the central pixel. In this article, we investigate that scene-specific but not essential correlations may be recorded in an HSI cuboid. This additional information improves the model performance on existing HSI datasets and makes it hard to properly evaluate the ability of a model. We refer to this problem as the spatial overfitting issue and utilize strict experimental settings to avoid it. We further propose a multiview transformer for HSI classification, which consists of multiview principal component analysis (MPCA), spectral encoder-decoder (SED), and spatial-pooling tokenization transformer (SPTT). MPCA performs dimension reduction on an HSI via constructing spectral multiview observations and applying PCA on each view data to extract low-dimensional view representation. The combination of view representations, named multiview representation, is the dimension reduction output of the MPCA. To aggregate the multiview information, a fully-convolutional SED with a U-shape in spectral dimension is introduced to extract a multiview feature map. SPTT transforms the multiview features into tokens using the spatial-pooling tokenization strategy and learns robust and discriminative spatial-spectral features for land cover identification. Classification is conducted with a linear classifier. Experiments on three HSI datasets with rigid settings demonstrate the superiority of the proposed multiview transformer over the state-of-the-art methods.

Title: Distilling Efficient Vision Transformers from CNNs for Semantic Segmentation. (arXiv:2310.07265v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.07265
• Code URL: null
• Copy Paste: [[2310.07265]] Distilling Efficient Vision Transformers from CNNs for Semantic Segmentation(http://arxiv.org/abs/2310.07265)
• Summary:

  In this paper, we tackle a new problem: how to transfer knowledge from the pre-trained cumbersome yet well-performed CNN-based model to learn a compact Vision Transformer (ViT)-based model while maintaining its learning capacity? Due to the completely different characteristics of ViT and CNN and the long-existing capacity gap between teacher and student models in Knowledge Distillation (KD), directly transferring the cross-model knowledge is non-trivial. To this end, we subtly leverage the visual and linguistic-compatible feature character of ViT (i.e., student), and its capacity gap with the CNN (i.e., teacher) and propose a novel CNN-to-ViT KD framework, dubbed C2VKD. Importantly, as the teacher's features are heterogeneous to those of the student, we first propose a novel visual-linguistic feature distillation (VLFD) module that explores efficient KD among the aligned visual and linguistic-compatible representations. Moreover, due to the large capacity gap between the teacher and student and the inevitable prediction errors of the teacher, we then propose a pixel-wise decoupled distillation (PDD) module to supervise the student under the combination of labels and teacher's predictions from the decoupled target and non-target classes. Experiments on three semantic segmentation benchmark datasets consistently show that the increment of mIoU of our method is over 200% of the SoTA KD methods

Title: Distance-based Weighted Transformer Network for Image Completion. (arXiv:2310.07440v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.07440
• Code URL: null
• Copy Paste: [[2310.07440]] Distance-based Weighted Transformer Network for Image Completion(http://arxiv.org/abs/2310.07440)
• Summary:

  The challenge of image generation has been effectively modeled as a problem of structure priors or transformation. However, existing models have unsatisfactory performance in understanding the global input image structures because of particular inherent features (for example, local inductive prior). Recent studies have shown that self-attention is an efficient modeling technique for image completion problems. In this paper, we propose a new architecture that relies on Distance-based Weighted Transformer (DWT) to better understand the relationships between an image's components. In our model, we leverage the strengths of both Convolutional Neural Networks (CNNs) and DWT blocks to enhance the image completion process. Specifically, CNNs are used to augment the local texture information of coarse priors and DWT blocks are used to recover certain coarse textures and coherent visual structures. Unlike current approaches that generally use CNNs to create feature maps, we use the DWT to encode global dependencies and compute distance-based weighted feature maps, which substantially minimizes the problem of visual ambiguities. Meanwhile, to better produce repeated textures, we introduce Residual Fast Fourier Convolution (Res-FFC) blocks to combine the encoder's skip features with the coarse features provided by our generator. Furthermore, a simple yet effective technique is proposed to normalize the non-zero values of convolutions, and fine-tune the network layers for regularization of the gradient norms to provide an efficient training stabiliser. Extensive quantitative and qualitative experiments on three challenging datasets demonstrate the superiority of our proposed model compared to existing approaches.

Title: Why bother with geometry? On the relevance of linear decompositions of Transformer embeddings. (arXiv:2310.06977v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.06977
• Code URL: https://github.com/timotheemickus/seq2seq-splat
• Copy Paste: [[2310.06977]] Why bother with geometry? On the relevance of linear decompositions of Transformer embeddings(http://arxiv.org/abs/2310.06977)
• Summary:

  A recent body of work has demonstrated that Transformer embeddings can be linearly decomposed into well-defined sums of factors, that can in turn be related to specific network inputs or components. There is however still a dearth of work studying whether these mathematical reformulations are empirically meaningful. In the present work, we study representations from machine-translation decoders using two of such embedding decomposition methods. Our results indicate that, while decomposition-derived indicators effectively correlate with model performance, variation across different runs suggests a more nuanced take on this question. The high variability of our measurements indicate that geometry reflects model-specific characteristics more than it does sentence-specific computations, and that similar training conditions do not guarantee similar vector spaces.

Title: Crossing the Threshold: Idiomatic Machine Translation through Retrieval Augmentation and Loss Weighting. (arXiv:2310.07081v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.07081
• Code URL: https://github.com/nightingal3/idiom-translation
• Copy Paste: [[2310.07081]] Crossing the Threshold: Idiomatic Machine Translation through Retrieval Augmentation and Loss Weighting(http://arxiv.org/abs/2310.07081)
• Summary:

  Idioms are common in everyday language, but often pose a challenge to translators because their meanings do not follow from the meanings of their parts. Despite significant advances, machine translation systems still struggle to translate idiomatic expressions. We provide a simple characterization of idiomatic translation and related issues. This allows us to conduct a synthetic experiment revealing a tipping point at which transformer-based machine translation models correctly default to idiomatic translations. To expand multilingual resources, we compile a dataset of ~4k natural sentences containing idiomatic expressions in French, Finnish, and Japanese. To improve translation of natural idioms, we introduce two straightforward yet effective techniques: the strategic upweighting of training loss on potentially idiomatic sentences, and using retrieval-augmented models. This not only improves the accuracy of a strong pretrained MT model on idiomatic sentences by up to 13% in absolute accuracy, but also holds potential benefits for non-idiomatic sentences.

Title: Jaeger: A Concatenation-Based Multi-Transformer VQA Model. (arXiv:2310.07091v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.07091
• Code URL: null
• Copy Paste: [[2310.07091]] Jaeger: A Concatenation-Based Multi-Transformer VQA Model(http://arxiv.org/abs/2310.07091)
• Summary:

  Document-based Visual Question Answering poses a challenging task between linguistic sense disambiguation and fine-grained multimodal retrieval. Although there has been encouraging progress in document-based question answering due to the utilization of large language and open-world prior models\cite{1}, several challenges persist, including prolonged response times, extended inference durations, and imprecision in matching. In order to overcome these challenges, we propose Jaegar, a concatenation-based multi-transformer VQA model. To derive question features, we leverage the exceptional capabilities of RoBERTa large\cite{2} and GPT2-xl\cite{3} as feature extractors. Subsequently, we subject the outputs from both models to a concatenation process. This operation allows the model to consider information from diverse sources concurrently, strengthening its representational capability. By leveraging pre-trained models for feature extraction, our approach has the potential to amplify the performance of these models through concatenation. After concatenation, we apply dimensionality reduction to the output features, reducing the model's computational effectiveness and inference time. Empirical results demonstrate that our proposed model achieves competitive performance on Task C of the PDF-VQA Dataset. If the user adds any new data, they should make sure to style it as per the instructions provided in previous sections.

Title: Sparse Universal Transformer. (arXiv:2310.07096v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.07096
• Code URL: null
• Copy Paste: [[2310.07096]] Sparse Universal Transformer(http://arxiv.org/abs/2310.07096)
• Summary:

  The Universal Transformer (UT) is a variant of the Transformer that shares parameters across its layers. Empirical evidence shows that UTs have better compositional generalization than Vanilla Transformers (VTs) in formal language tasks. The parameter-sharing also affords it better parameter efficiency than VTs. Despite its many advantages, scaling UT parameters is much more compute and memory intensive than scaling up a VT. This paper proposes the Sparse Universal Transformer (SUT), which leverages Sparse Mixture of Experts (SMoE) and a new stick-breaking-based dynamic halting mechanism to reduce UT's computation complexity while retaining its parameter efficiency and generalization ability. Experiments show that SUT achieves the same performance as strong baseline models while only using half computation and parameters on WMT'14 and strong generalization results on formal language tasks (Logical inference and CFQ). The new halting mechanism also enables around 50\% reduction in computation during inference with very little performance decrease on formal language tasks.

Title: PHALM: Building a Knowledge Graph from Scratch by Prompting Humans and a Language Model. (arXiv:2310.07170v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.07170
• Code URL: https://github.com/nlp-waseda/comet-atomic-ja
• Copy Paste: [[2310.07170]] PHALM: Building a Knowledge Graph from Scratch by Prompting Humans and a Language Model(http://arxiv.org/abs/2310.07170)
• Summary:

  Despite the remarkable progress in natural language understanding with pretrained Transformers, neural language models often do not handle commonsense knowledge well. Toward commonsense-aware models, there have been attempts to obtain knowledge, ranging from automatic acquisition to crowdsourcing. However, it is difficult to obtain a high-quality knowledge base at a low cost, especially from scratch. In this paper, we propose PHALM, a method of building a knowledge graph from scratch, by prompting both crowdworkers and a large language model (LLM). We used this method to build a Japanese event knowledge graph and trained Japanese commonsense generation models. Experimental results revealed the acceptability of the built graph and inferences generated by the trained models. We also report the difference in prompting humans and an LLM. Our code, data, and models are available at github.com/nlp-waseda/comet-atomic-ja.

Title: DASpeech: Directed Acyclic Transformer for Fast and High-quality Speech-to-Speech Translation. (arXiv:2310.07403v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.07403
• Code URL: https://github.com/ictnlp/daspeech
• Copy Paste: [[2310.07403]] DASpeech: Directed Acyclic Transformer for Fast and High-quality Speech-to-Speech Translation(http://arxiv.org/abs/2310.07403)
• Summary:

  Direct speech-to-speech translation (S2ST) translates speech from one language into another using a single model. However, due to the presence of linguistic and acoustic diversity, the target speech follows a complex multimodal distribution, posing challenges to achieving both high-quality translations and fast decoding speeds for S2ST models. In this paper, we propose DASpeech, a non-autoregressive direct S2ST model which realizes both fast and high-quality S2ST. To better capture the complex distribution of the target speech, DASpeech adopts the two-pass architecture to decompose the generation process into two steps, where a linguistic decoder first generates the target text, and an acoustic decoder then generates the target speech based on the hidden states of the linguistic decoder. Specifically, we use the decoder of DA-Transformer as the linguistic decoder, and use FastSpeech 2 as the acoustic decoder. DA-Transformer models translations with a directed acyclic graph (DAG). To consider all potential paths in the DAG during training, we calculate the expected hidden states for each target token via dynamic programming, and feed them into the acoustic decoder to predict the target mel-spectrogram. During inference, we select the most probable path and take hidden states on that path as input to the acoustic decoder. Experiments on the CVSS Fr-En benchmark demonstrate that DASpeech can achieve comparable or even better performance than the state-of-the-art S2ST model Translatotron 2, while preserving up to 18.53x speedup compared to the autoregressive baseline. Compared with the previous non-autoregressive S2ST model, DASpeech does not rely on knowledge distillation and iterative decoding, achieving significant improvements in both translation quality and decoding speed. Furthermore, DASpeech shows the ability to preserve the speaker's voice of the source speech during translation.

Title: Cognate Transformer for Automated Phonological Reconstruction and Cognate Reflex Prediction. (arXiv:2310.07487v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.07487
• Code URL: null
• Copy Paste: [[2310.07487]] Cognate Transformer for Automated Phonological Reconstruction and Cognate Reflex Prediction(http://arxiv.org/abs/2310.07487)
• Summary:

  Phonological reconstruction is one of the central problems in historical linguistics where a proto-word of an ancestral language is determined from the observed cognate words of daughter languages. Computational approaches to historical linguistics attempt to automate the task by learning models on available linguistic data. Several ideas and techniques drawn from computational biology have been successfully applied in the area of computational historical linguistics. Following these lines, we adapt MSA Transformer, a protein language model, to the problem of automated phonological reconstruction. MSA Transformer trains on multiple sequence alignments as input and is, thus, apt for application on aligned cognate words. We, hence, name our model as Cognate Transformer. We also apply the model on another associated task, namely, cognate reflex prediction, where a reflex word in a daughter language is predicted based on cognate words from other daughter languages. We show that our model outperforms the existing models on both tasks, especially when it is pre-trained on masked word prediction task.

Title: Generalized Neural Sorting Networks with Error-Free Differentiable Swap Functions. (arXiv:2310.07174v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.07174
• Code URL: null
• Copy Paste: [[2310.07174]] Generalized Neural Sorting Networks with Error-Free Differentiable Swap Functions(http://arxiv.org/abs/2310.07174)
• Summary:

  Sorting is a fundamental operation of all computer systems, having been a long-standing significant research topic. Beyond the problem formulation of traditional sorting algorithms, we consider sorting problems for more abstract yet expressive inputs, e.g., multi-digit images and image fragments, through a neural sorting network. To learn a mapping from a high-dimensional input to an ordinal variable, the differentiability of sorting networks needs to be guaranteed. In this paper we define a softening error by a differentiable swap function, and develop an error-free swap function that holds non-decreasing and differentiability conditions. Furthermore, a permutation-equivariant Transformer network with multi-head attention is adopted to capture dependency between given inputs and also leverage its model capacity with self-attention. Experiments on diverse sorting benchmarks show that our methods perform better than or comparable to baseline methods.

Title: Atom-Motif Contrastive Transformer for Molecular Property Prediction. (arXiv:2310.07351v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.07351
• Code URL: null
• Copy Paste: [[2310.07351]] Atom-Motif Contrastive Transformer for Molecular Property Prediction(http://arxiv.org/abs/2310.07351)
• Summary:

  Recently, Graph Transformer (GT) models have been widely used in the task of Molecular Property Prediction (MPP) due to their high reliability in characterizing the latent relationship among graph nodes (i.e., the atoms in a molecule). However, most existing GT-based methods usually explore the basic interactions between pairwise atoms, and thus they fail to consider the important interactions among critical motifs (e.g., functional groups consisted of several atoms) of molecules. As motifs in a molecule are significant patterns that are of great importance for determining molecular properties (e.g., toxicity and solubility), overlooking motif interactions inevitably hinders the effectiveness of MPP. To address this issue, we propose a novel Atom-Motif Contrastive Transformer (AMCT), which not only explores the atom-level interactions but also considers the motif-level interactions. Since the representations of atoms and motifs for a given molecule are actually two different views of the same instance, they are naturally aligned to generate the self-supervisory signals for model training. Meanwhile, the same motif can exist in different molecules, and hence we also employ the contrastive loss to maximize the representation agreement of identical motifs across different molecules. Finally, in order to clearly identify the motifs that are critical in deciding the properties of each molecule, we further construct a property-aware attention mechanism into our learning framework. Our proposed AMCT is extensively evaluated on seven popular benchmark datasets, and both quantitative and qualitative results firmly demonstrate its effectiveness when compared with the state-of-the-art methods.

Title: NuTime: Numerically Multi-Scaled Embedding for Large-Scale Time Series Pretraining. (arXiv:2310.07402v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.07402
• Code URL: null
• Copy Paste: [[2310.07402]] NuTime: Numerically Multi-Scaled Embedding for Large-Scale Time Series Pretraining(http://arxiv.org/abs/2310.07402)
• Summary:

  Recent research on time-series self-supervised models shows great promise in learning semantic representations. However, it has been limited to small-scale datasets, e.g., thousands of temporal sequences. In this work, we make key technical contributions that are tailored to the numerical properties of time-series data and allow the model to scale to large datasets, e.g., millions of temporal sequences. We adopt the Transformer architecture by first partitioning the input into non-overlapping windows. Each window is then characterized by its normalized shape and two scalar values denoting the mean and standard deviation within each window. To embed scalar values that may possess arbitrary numerical scales to high-dimensional vectors, we propose a numerically multi-scaled embedding module enumerating all possible scales for the scalar values. The model undergoes pretraining using the proposed numerically multi-scaled embedding with a simple contrastive objective on a large-scale dataset containing over a million sequences. We study its transfer performance on a number of univariate and multivariate classification benchmarks. Our method exhibits remarkable improvement against previous representation learning approaches and establishes the new state of the art, even compared with domain-specific non-learning-based methods.

generative

Title: Mitigating stereotypical biases in text to image generative systems. (arXiv:2310.06904v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.06904
• Code URL: null
• Copy Paste: [[2310.06904]] Mitigating stereotypical biases in text to image generative systems(http://arxiv.org/abs/2310.06904)
• Summary:

  State-of-the-art generative text-to-image models are known to exhibit social biases and over-represent certain groups like people of perceived lighter skin tones and men in their outcomes. In this work, we propose a method to mitigate such biases and ensure that the outcomes are fair across different groups of people. We do this by finetuning text-to-image models on synthetic data that varies in perceived skin tones and genders constructed from diverse text prompts. These text prompts are constructed from multiplicative combinations of ethnicities, genders, professions, age groups, and so on, resulting in diverse synthetic data. Our diversity finetuned (DFT) model improves the group fairness metric by 150% for perceived skin tone and 97.7% for perceived gender. Compared to baselines, DFT models generate more people with perceived darker skin tone and more women. To foster open research, we will release all text prompts and code to generate training images.

Title: Utilizing Synthetic Data for Medical Vision-Language Pre-training: Bypassing the Need for Real Images. (arXiv:2310.07027v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.07027
• Code URL: null
• Copy Paste: [[2310.07027]] Utilizing Synthetic Data for Medical Vision-Language Pre-training: Bypassing the Need for Real Images(http://arxiv.org/abs/2310.07027)
• Summary:

  Medical Vision-Language Pre-training (VLP) learns representations jointly from medical images and paired radiology reports. It typically requires large-scale paired image-text datasets to achieve effective pre-training for both the image encoder and text encoder. The advent of text-guided generative models raises a compelling question: Can VLP be implemented solely with synthetic images generated from genuine radiology reports, thereby mitigating the need for extensively pairing and curating image-text datasets? In this work, we scrutinize this very question by examining the feasibility and effectiveness of employing synthetic images for medical VLP. We replace real medical images with their synthetic equivalents, generated from authentic medical reports. Utilizing three state-of-the-art VLP algorithms, we exclusively train on these synthetic samples. Our empirical evaluation across three subsequent tasks, namely image classification, semantic segmentation and object detection, reveals that the performance achieved through synthetic data is on par with or even exceeds that obtained with real images. As a pioneering contribution to this domain, we introduce a large-scale synthetic medical image dataset, paired with anonymized real radiology reports. This alleviates the need of sharing medical images, which are not easy to curate and share in practice. The code and the dataset will be made publicly available upon paper acceptance.

Title: Crowd Counting in Harsh Weather using Image Denoising with Pix2Pix GANs. (arXiv:2310.07245v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.07245
• Code URL: null
• Copy Paste: [[2310.07245]] Crowd Counting in Harsh Weather using Image Denoising with Pix2Pix GANs(http://arxiv.org/abs/2310.07245)
• Summary:

  Visual crowd counting estimates the density of the crowd using deep learning models such as convolution neural networks (CNNs). The performance of the model heavily relies on the quality of the training data that constitutes crowd images. In harsh weather such as fog, dust, and low light conditions, the inference performance may severely degrade on the noisy and blur images. In this paper, we propose the use of Pix2Pix generative adversarial network (GAN) to first denoise the crowd images prior to passing them to the counting model. A Pix2Pix network is trained using synthetic noisy images generated from original crowd images and then the pretrained generator is then used in the inference engine to estimate the crowd density in unseen, noisy crowd images. The performance is tested on JHU-Crowd dataset to validate the significance of the proposed method particularly when high reliability and accuracy are required.

Title: On the Impact of Cross-Domain Data on German Language Models. (arXiv:2310.07321v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.07321
• Code URL: null
• Copy Paste: [[2310.07321]] On the Impact of Cross-Domain Data on German Language Models(http://arxiv.org/abs/2310.07321)
• Summary:

  Traditionally, large language models have been either trained on general web crawls or domain-specific data. However, recent successes of generative large language models, have shed light on the benefits of cross-domain datasets. To examine the significance of prioritizing data diversity over quality, we present a German dataset comprising texts from five domains, along with another dataset aimed at containing high-quality data. Through training a series of models ranging between 122M and 750M parameters on both datasets, we conduct a comprehensive benchmark on multiple downstream tasks. Our findings demonstrate that the models trained on the cross-domain dataset outperform those trained on quality data alone, leading to improvements up to $4.45\%$ over the previous state-of-the-art. The models are available at https://huggingface.co/ikim-uk-essen

Title: Towards Foundation Models for Learning on Tabular Data. (arXiv:2310.07338v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.07338
• Code URL: null
• Copy Paste: [[2310.07338]] Towards Foundation Models for Learning on Tabular Data(http://arxiv.org/abs/2310.07338)
• Summary:

  Learning on tabular data underpins numerous real-world applications. Despite considerable efforts in developing effective learning models for tabular data, current transferable tabular models remain in their infancy, limited by either the lack of support for direct instruction following in new tasks or the neglect of acquiring foundational knowledge and capabilities from diverse tabular datasets. In this paper, we propose Tabular Foundation Models (TabFMs) to overcome these limitations. TabFMs harness the potential of generative tabular learning, employing a pre-trained large language model (LLM) as the base model and fine-tuning it using purpose-designed objectives on an extensive range of tabular datasets. This approach endows TabFMs with a profound understanding and universal capabilities essential for learning on tabular data. Our evaluations underscore TabFM's effectiveness: not only does it significantly excel in instruction-following tasks like zero-shot and in-context inference, but it also showcases performance that approaches, and in instances, even transcends, the renowned yet mysterious closed-source LLMs like GPT-4. Furthermore, when fine-tuning with scarce data, our model achieves remarkable efficiency and maintains competitive performance with abundant training data. Finally, while our results are promising, we also delve into TabFM's limitations and potential opportunities, aiming to stimulate and expedite future research on developing more potent TabFMs.

Title: ProbTS: A Unified Toolkit to Probe Deep Time-series Forecasting. (arXiv:2310.07446v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.07446
• Code URL: null
• Copy Paste: [[2310.07446]] ProbTS: A Unified Toolkit to Probe Deep Time-series Forecasting(http://arxiv.org/abs/2310.07446)
• Summary:

  Time-series forecasting serves as a linchpin in a myriad of applications, spanning various domains. With the growth of deep learning, this arena has bifurcated into two salient branches: one focuses on crafting specific neural architectures tailored for time series, and the other harnesses advanced deep generative models for probabilistic forecasting. While both branches have made significant progress, their differences across data scenarios, methodological focuses, and decoding schemes pose profound, yet unexplored, research questions. To bridge this knowledge chasm, we introduce ProbTS, a pioneering toolkit developed to synergize and compare these two distinct branches. Endowed with a unified data module, a modularized model module, and a comprehensive evaluator module, ProbTS allows us to revisit and benchmark leading methods from both branches. The scrutiny with ProbTS highlights their distinct characteristics, relative strengths and weaknesses, and areas that need further exploration. Our analyses point to new avenues for research, aiming for more effective time-series forecasting.

large language model

Title: Sparse Finetuning for Inference Acceleration of Large Language Models. (arXiv:2310.06927v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.06927
• Code URL: https://github.com/neuralmagic/deepsparse
• Copy Paste: [[2310.06927]] Sparse Finetuning for Inference Acceleration of Large Language Models(http://arxiv.org/abs/2310.06927)
• Summary:

  We consider the problem of accurate sparse finetuning of large language models (LLMs), that is, finetuning pretrained LLMs on specialized tasks, while inducing sparsity in their weights. On the accuracy side, we observe that standard loss-based finetuning may fail to recover accuracy, especially at high sparsities. To address this, we perform a detailed study of distillation-type losses, determining an L2-based distillation approach we term SquareHead which enables accurate recovery even at higher sparsities, across all model types. On the practical efficiency side, we show that sparse LLMs can be executed with speedups by taking advantage of sparsity, for both CPU and GPU runtimes. While the standard approach is to leverage sparsity for computational reduction, we observe that in the case of memory-bound LLMs sparsity can also be leveraged for reducing memory bandwidth. We exhibit end-to-end results showing speedups due to sparsity, while recovering accuracy, on T5 (language translation), Whisper (speech translation), and open GPT-type (MPT for text generation). For MPT text generation, we show for the first time that sparse finetuning can reach 75% sparsity without accuracy drops, provide notable end-to-end speedups for both CPU and GPU inference, and highlight that sparsity is also compatible with quantization approaches. Models and software for reproducing our results are provided in Section 6.

Title: Violation of Expectation via Metacognitive Prompting Reduces Theory of Mind Prediction Error in Large Language Models. (arXiv:2310.06983v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.06983
• Code URL: https://github.com/plastic-labs/voe-paper-eval
• Copy Paste: [[2310.06983]] Violation of Expectation via Metacognitive Prompting Reduces Theory of Mind Prediction Error in Large Language Models(http://arxiv.org/abs/2310.06983)
• Summary:

  Recent research shows that Large Language Models (LLMs) exhibit a compelling level of proficiency in Theory of Mind (ToM) tasks. This ability to impute unobservable mental states to others is vital to human social cognition and may prove equally important in principal-agent relations between individual humans and Artificial Intelligences (AIs). In this paper, we explore how a mechanism studied in developmental psychology known as Violation of Expectation (VoE) can be implemented to reduce errors in LLM prediction about users by leveraging emergent ToM affordances. And we introduce a \textit{metacognitive prompting} framework to apply VoE in the context of an AI tutor. By storing and retrieving facts derived in cases where LLM expectation about the user was violated, we find that LLMs are able to learn about users in ways that echo theories of human learning. Finally, we discuss latent hazards and augmentative opportunities associated with modeling user psychology and propose ways to mitigate risk along with possible directions for future inquiry.

Title: NEWTON: Are Large Language Models Capable of Physical Reasoning?. (arXiv:2310.07018v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.07018
• Code URL: null
• Copy Paste: [[2310.07018]] NEWTON: Are Large Language Models Capable of Physical Reasoning?(http://arxiv.org/abs/2310.07018)
• Summary:

  Large Language Models (LLMs), through their contextualized representations, have been empirically proven to encapsulate syntactic, semantic, word sense, and common-sense knowledge. However, there has been limited exploration of their physical reasoning abilities, specifically concerning the crucial attributes for comprehending everyday objects. To address this gap, we introduce NEWTON, a repository and benchmark for evaluating the physics reasoning skills of LLMs. Further, to enable domain-specific adaptation of this benchmark, we present a pipeline to enable researchers to generate a variant of this benchmark that has been customized to the objects and attributes relevant for their application. The NEWTON repository comprises a collection of 2800 object-attribute pairs, providing the foundation for generating infinite-scale assessment templates. The NEWTON benchmark consists of 160K QA questions, curated using the NEWTON repository to investigate the physical reasoning capabilities of several mainstream language models across foundational, explicit, and implicit reasoning tasks. Through extensive empirical analysis, our results highlight the capabilities of LLMs for physical reasoning. We find that LLMs like GPT-4 demonstrate strong reasoning capabilities in scenario-based tasks but exhibit less consistency in object-attribute reasoning compared to humans (50% vs. 84%). Furthermore, the NEWTON platform demonstrates its potential for evaluating and enhancing language models, paving the way for their integration into physically grounded settings, such as robotic manipulation. Project site: https://newtonreasoning.github.io

Title: DKEC: Domain Knowledge Enhanced Multi-Label Classification for Electronic Health Records. (arXiv:2310.07059v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.07059
• Code URL: null
• Copy Paste: [[2310.07059]] DKEC: Domain Knowledge Enhanced Multi-Label Classification for Electronic Health Records(http://arxiv.org/abs/2310.07059)
• Summary:

  Multi-label text classification (MLTC) tasks in the medical domain often face long-tail label distribution, where rare classes have fewer training samples than frequent classes. Although previous works have explored different model architectures and hierarchical label structures to find important features, most of them neglect to incorporate the domain knowledge from medical guidelines. In this paper, we present DKEC, Domain Knowledge Enhanced Classifier for medical diagnosis prediction with two innovations: (1) a label-wise attention mechanism that incorporates a heterogeneous graph and domain ontologies to capture the semantic relationships between medical entities, (2) a simple yet effective group-wise training method based on similarity of labels to increase samples of rare classes. We evaluate DKEC on two real-world medical datasets: the RAA dataset, a collection of 4,417 patient care reports from emergency medical services (EMS) incidents, and a subset of 53,898 reports from the MIMIC-III dataset. Experimental results show that our method outperforms the state-of-the-art, particularly for the few-shot (tail) classes. More importantly, we study the applicability of DKEC to different language models and show that DKEC can help the smaller language models achieve comparable performance to large language models.

Title: Syntax Error-Free and Generalizable Tool Use for LLMs via Finite-State Decoding. (arXiv:2310.07075v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.07075
• Code URL: https://github.com/chenhongqiao/tooldec
• Copy Paste: [[2310.07075]] Syntax Error-Free and Generalizable Tool Use for LLMs via Finite-State Decoding(http://arxiv.org/abs/2310.07075)
• Summary:

  Large language models (LLMs) have shown promising capabilities in using external tools to solve complex problems. However, existing approaches either involve fine-tuning on tool demonstrations, which do not generalize to new tools without additional training, or providing tool documentation in context, limiting the number of tools. Both approaches often generate syntactically invalid tool calls. In this paper, we propose ToolDec, a finite-state machine-guided decoding algorithm for tool-augmented LLMs. ToolDec eliminates tool-related errors for any tool-augmented LLMs by ensuring valid tool names and type-conforming arguments. Furthermore, ToolDec enables LLM to effectively select tools using only the information contained in their names, with no need for fine-tuning or in-context documentation. We evaluated multiple prior methods and their ToolDec-enhanced versions on a variety of tasks involving tools like math functions, knowledge graph relations, and complex real-world RESTful APIs. Our experiments show that ToolDec reduces syntactic errors to zero, consequently achieving significantly better performance and as much as a 2x speedup. We also show that ToolDec achieves superior generalization performance on unseen tools, performing up to 8x better than the baselines.

Title: Diversity of Thought Improves Reasoning Abilities of Large Language Models. (arXiv:2310.07088v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.07088
• Code URL: null
• Copy Paste: [[2310.07088]] Diversity of Thought Improves Reasoning Abilities of Large Language Models(http://arxiv.org/abs/2310.07088)
• Summary:

  Large language models (LLMs) are documented to struggle in settings that require complex reasoning. Nevertheless, instructing the model to break down the problem into smaller reasoning steps (Wei et al., 2022), or ensembling various generations through modifying decoding steps (Wang et al., 2023) boosts performance. Current methods assume that the input prompt is fixed and expect the decoding strategies to introduce the diversity needed for ensembling. In this work, we relax this assumption and discuss how one can create and leverage variations of the input prompt as a means to diversity of thought to improve model performance. We propose a method that automatically improves prompt diversity by soliciting feedback from the LLM to ideate approaches that fit for the problem. We then ensemble the diverse prompts in our method DIV-SE (DIVerse reasoning path Self-Ensemble) across multiple inference calls. We also propose a cost-effective alternative where diverse prompts are used within a single inference call; we call this IDIV-SE (In-call DIVerse reasoning path Self-Ensemble). Under a fixed generation budget, DIV-SE and IDIV-SE outperform the previously discussed baselines using both GPT-3.5 and GPT-4 on several reasoning benchmarks, without modifying the decoding process. Additionally, DIV-SE advances state-of-the-art performance on recent planning benchmarks (Valmeekam et al., 2023), exceeding the highest previously reported accuracy by at least 29.6 percentage points on the most challenging 4/5 Blocksworld task. Our results shed light on how to enforce prompt diversity toward LLM reasoning and thereby improve the pareto frontier of the accuracy-cost trade-off.

Title: Empowering Psychotherapy with Large Language Models: Cognitive Distortion Detection through Diagnosis of Thought Prompting. (arXiv:2310.07146v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.07146
• Code URL: null
• Copy Paste: [[2310.07146]] Empowering Psychotherapy with Large Language Models: Cognitive Distortion Detection through Diagnosis of Thought Prompting(http://arxiv.org/abs/2310.07146)
• Summary:

  Mental illness remains one of the most critical public health issues of our time, due to the severe scarcity and accessibility limit of professionals. Psychotherapy requires high-level expertise to conduct deep, complex reasoning and analysis on the cognition modeling of the patients. In the era of Large Language Models, we believe it is the right time to develop AI assistance for computational psychotherapy. We study the task of cognitive distortion detection and propose the Diagnosis of Thought (DoT) prompting. DoT performs diagnosis on the patient's speech via three stages: subjectivity assessment to separate the facts and the thoughts; contrastive reasoning to elicit the reasoning processes supporting and contradicting the thoughts; and schema analysis to summarize the cognition schemas. The generated diagnosis rationales through the three stages are essential for assisting the professionals. Experiments demonstrate that DoT obtains significant improvements over ChatGPT for cognitive distortion detection, while generating high-quality rationales approved by human experts.

Title: "A Tale of Two Movements": Identifying and Comparing Perspectives in #BlackLivesMatter and #BlueLivesMatter Movements-related Tweets using Weakly Supervised Graph-based Structured Prediction. (arXiv:2310.07155v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.07155
• Code URL: null
• Copy Paste: [[2310.07155]] "A Tale of Two Movements": Identifying and Comparing Perspectives in #BlackLivesMatter and #BlueLivesMatter Movements-related Tweets using Weakly Supervised Graph-based Structured Prediction(http://arxiv.org/abs/2310.07155)
• Summary:

  Social media has become a major driver of social change, by facilitating the formation of online social movements. Automatically understanding the perspectives driving the movement and the voices opposing it, is a challenging task as annotated data is difficult to obtain. We propose a weakly supervised graph-based approach that explicitly models perspectives in #BackLivesMatter-related tweets. Our proposed approach utilizes a social-linguistic representation of the data. We convert the text to a graph by breaking it into structured elements and connect it with the social network of authors, then structured prediction is done over the elements for identifying perspectives. Our approach uses a small seed set of labeled examples. We experiment with large language models for generating artificial training examples, compare them to manual annotation, and find that it achieves comparable performance. We perform quantitative and qualitative analyses using a human-annotated test set. Our model outperforms multitask baselines by a large margin, successfully characterizing the perspectives supporting and opposing #BLM.

Title: Adaptive Gating in Mixture-of-Experts based Language Models. (arXiv:2310.07188v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.07188
• Code URL: null
• Copy Paste: [[2310.07188]] Adaptive Gating in Mixture-of-Experts based Language Models(http://arxiv.org/abs/2310.07188)
• Summary:

  Large language models, such as OpenAI's ChatGPT, have demonstrated exceptional language understanding capabilities in various NLP tasks. Sparsely activated mixture-of-experts (MoE) has emerged as a promising solution for scaling models while maintaining a constant number of computational operations. Existing MoE model adopts a fixed gating network where each token is computed by the same number of experts. However, this approach contradicts our intuition that the tokens in each sequence vary in terms of their linguistic complexity and, consequently, require different computational costs. Little is discussed in prior research on the trade-off between computation per token and model performance. This paper introduces adaptive gating in MoE, a flexible training strategy that allows tokens to be processed by a variable number of experts based on expert probability distribution. The proposed framework preserves sparsity while improving training efficiency. Additionally, curriculum learning is leveraged to further reduce training time. Extensive experiments on diverse NLP tasks show that adaptive gating reduces at most 22.5% training time while maintaining inference quality. Moreover, we conduct a comprehensive analysis of the routing decisions and present our insights when adaptive gating is used.

Title: Exploring the Landscape of Large Language Models In Medical Question Answering: Observations and Open Questions. (arXiv:2310.07225v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.07225
• Code URL: null
• Copy Paste: [[2310.07225]] Exploring the Landscape of Large Language Models In Medical Question Answering: Observations and Open Questions(http://arxiv.org/abs/2310.07225)
• Summary:

  Large Language Models (LLMs) have shown promise in medical question answering by achieving passing scores in standardised exams and have been suggested as tools for supporting healthcare workers. Deploying LLMs into such a high-risk context requires a clear understanding of the limitations of these models. With the rapid development and release of new LLMs, it is especially valuable to identify patterns which exist across models and may, therefore, continue to appear in newer versions. In this paper, we evaluate a wide range of popular LLMs on their knowledge of medical questions in order to better understand their properties as a group. From this comparison, we provide preliminary observations and raise open questions for further research.

Title: Beyond Factuality: A Comprehensive Evaluation of Large Language Models as Knowledge Generators. (arXiv:2310.07289v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.07289
• Code URL: null
• Copy Paste: [[2310.07289]] Beyond Factuality: A Comprehensive Evaluation of Large Language Models as Knowledge Generators(http://arxiv.org/abs/2310.07289)
• Summary:

  Large language models (LLMs) outperform information retrieval techniques for downstream knowledge-intensive tasks when being prompted to generate world knowledge. However, community concerns abound regarding the factuality and potential implications of using this uncensored knowledge. In light of this, we introduce CONNER, a COmpreheNsive kNowledge Evaluation fRamework, designed to systematically and automatically evaluate generated knowledge from six important perspectives -- Factuality, Relevance, Coherence, Informativeness, Helpfulness and Validity. We conduct an extensive empirical analysis of the generated knowledge from three different types of LLMs on two widely studied knowledge-intensive tasks, i.e., open-domain question answering and knowledge-grounded dialogue. Surprisingly, our study reveals that the factuality of generated knowledge, even if lower, does not significantly hinder downstream tasks. Instead, the relevance and coherence of the outputs are more important than small factual mistakes. Further, we show how to use CONNER to improve knowledge-intensive tasks by designing two strategies: Prompt Engineering and Knowledge Selection. Our evaluation code and LLM-generated knowledge with human annotations will be released to facilitate future research.

Title: An Empirical Study of Instruction-tuning Large Language Models in Chinese. (arXiv:2310.07328v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.07328
• Code URL: https://github.com/phoebussi/alpaca-cot
• Copy Paste: [[2310.07328]] An Empirical Study of Instruction-tuning Large Language Models in Chinese(http://arxiv.org/abs/2310.07328)
• Summary:

  The success of ChatGPT validates the potential of large language models (LLMs) in artificial general intelligence (AGI). Subsequently, the release of LLMs has sparked the open-source community's interest in instruction-tuning, which is deemed to accelerate ChatGPT's replication process. However, research on instruction-tuning LLMs in Chinese, the world's most spoken language, is still in its early stages. Therefore, this paper makes an in-depth empirical study of instruction-tuning LLMs in Chinese, which can serve as a cookbook that provides valuable findings for effectively customizing LLMs that can better respond to Chinese instructions. Specifically, we systematically explore the impact of LLM bases, parameter-efficient methods, instruction data types, which are the three most important elements for instruction-tuning. Besides, we also conduct experiment to study the impact of other factors, e.g., chain-of-thought data and human-value alignment. We hope that this empirical study can make a modest contribution to the open Chinese version of ChatGPT. This paper will release a powerful Chinese LLMs that is comparable to ChatGLM. The code and data are available at https://github.com/PhoebusSi/Alpaca-CoT.

Title: How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advances. (arXiv:2310.07343v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.07343
• Code URL: https://github.com/hyintell/awesome-refreshing-llms
• Copy Paste: [[2310.07343]] How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advances(http://arxiv.org/abs/2310.07343)
• Summary:

  Although large language models (LLMs) are impressive in solving various tasks, they can quickly be outdated after deployment. Maintaining their up-to-date status is a pressing concern in the current era. This paper provides a comprehensive review of recent advances in aligning LLMs with the ever-changing world knowledge without re-training from scratch. We categorize research works systemically and provide in-depth comparisons and discussion. We also discuss existing challenges and highlight future directions to facilitate research in this field. We release the paper list at https://github.com/hyintell/awesome-refreshing-llms

Title: KwaiYiiMath: Technical Report. (arXiv:2310.07488v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.07488
• Code URL: null
• Copy Paste: [[2310.07488]] KwaiYiiMath: Technical Report(http://arxiv.org/abs/2310.07488)
• Summary:

  Recent advancements in large language models (LLMs) have demonstrated remarkable abilities in handling a variety of natural language processing (NLP) downstream tasks, even on mathematical tasks requiring multi-step reasoning. In this report, we introduce the KwaiYiiMath which enhances the mathematical reasoning abilities of KwaiYiiBase1, by applying Supervised Fine-Tuning (SFT) and Reinforced Learning from Human Feedback (RLHF), including on both English and Chinese mathematical tasks. Meanwhile, we also constructed a small-scale Chinese primary school mathematics test set (named KMath), consisting of 188 examples to evaluate the correctness of the problem-solving process generated by the models. Empirical studies demonstrate that KwaiYiiMath can achieve state-of-the-art (SOTA) performance on GSM8k, CMath, and KMath compared with the similar size models, respectively.

Title: Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity. (arXiv:2310.07521v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.07521
• Code URL: https://github.com/wangcunxiang/llm-factuality-survey
• Copy Paste: [[2310.07521]] Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity(http://arxiv.org/abs/2310.07521)
• Summary:

  This survey addresses the crucial issue of factuality in Large Language Models (LLMs). As LLMs find applications across diverse domains, the reliability and accuracy of their outputs become vital. We define the Factuality Issue as the probability of LLMs to produce content inconsistent with established facts. We first delve into the implications of these inaccuracies, highlighting the potential consequences and challenges posed by factual errors in LLM outputs. Subsequently, we analyze the mechanisms through which LLMs store and process facts, seeking the primary causes of factual errors. Our discussion then transitions to methodologies for evaluating LLM factuality, emphasizing key metrics, benchmarks, and studies. We further explore strategies for enhancing LLM factuality, including approaches tailored for specific domains. We focus two primary LLM configurations standalone LLMs and Retrieval-Augmented LLMs that utilizes external data, we detail their unique challenges and potential enhancements. Our survey offers a structured guide for researchers aiming to fortify the factual reliability of LLMs.

Title: LLMs Killed the Script Kiddie: How Agents Supported by Large Language Models Change the Landscape of Network Threat Testing. (arXiv:2310.06936v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.06936
• Code URL: null
• Copy Paste: [[2310.06936]] LLMs Killed the Script Kiddie: How Agents Supported by Large Language Models Change the Landscape of Network Threat Testing(http://arxiv.org/abs/2310.06936)
• Summary:

  In this paper, we explore the potential of Large Language Models (LLMs) to reason about threats, generate information about tools, and automate cyber campaigns. We begin with a manual exploration of LLMs in supporting specific threat-related actions and decisions. We proceed by automating the decision process in a cyber campaign. We present prompt engineering approaches for a plan-act-report loop for one action of a threat campaign and and a prompt chaining design that directs the sequential decision process of a multi-action campaign. We assess the extent of LLM's cyber-specific knowledge w.r.t the short campaign we demonstrate and provide insights into prompt design for eliciting actionable responses. We discuss the potential impact of LLMs on the threat landscape and the ethical considerations of using LLMs for accelerating threat actor capabilities. We report a promising, yet concerning, application of generative AI to cyber threats. However, the LLM's capabilities to deal with more complex networks, sophisticated vulnerabilities, and the sensitivity of prompts are open questions. This research should spur deliberations over the inevitable advancements in LLM-supported cyber adversarial landscape.

Title: Risk Assessment and Statistical Significance in the Age of Foundation Models. (arXiv:2310.07132v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.07132
• Code URL: null
• Copy Paste: [[2310.07132]] Risk Assessment and Statistical Significance in the Age of Foundation Models(http://arxiv.org/abs/2310.07132)
• Summary:

  We propose a distributional framework for assessing socio-technical risks of foundation models with quantified statistical significance. Our approach hinges on a new statistical relative testing based on first and second order stochastic dominance of real random variables. We show that the second order statistics in this test are linked to mean-risk models commonly used in econometrics and mathematical finance to balance risk and utility when choosing between alternatives. Using this framework, we formally develop a risk-aware approach for foundation model selection given guardrails quantified by specified metrics. Inspired by portfolio optimization and selection theory in mathematical finance, we define a \emph{metrics portfolio} for each model as a means to aggregate a collection of metrics, and perform model selection based on the stochastic dominance of these portfolios. The statistical significance of our tests is backed theoretically by an asymptotic analysis via central limit theorems instantiated in practice via a bootstrap variance estimate. We use our framework to compare various large language models regarding risks related to drifting from instructions and outputting toxic content.

segmentation

Title: DeepTriNet: A Tri-Level Attention Based DeepLabv3+ Architecture for Semantic Segmentation of Satellite Images. (arXiv:2310.06848v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.06848
• Code URL: null
• Copy Paste: [[2310.06848]] DeepTriNet: A Tri-Level Attention Based DeepLabv3+ Architecture for Semantic Segmentation of Satellite Images(http://arxiv.org/abs/2310.06848)
• Summary:

  The segmentation of satellite images is crucial in remote sensing applications. Existing methods face challenges in recognizing small-scale objects in satellite images for semantic segmentation primarily due to ignoring the low-level characteristics of the underlying network and due to containing distinct amounts of information by different feature maps. Thus, in this research, a tri-level attention-based DeepLabv3+ architecture (DeepTriNet) is proposed for the semantic segmentation of satellite images. The proposed hybrid method combines squeeze-and-excitation networks (SENets) and tri-level attention units (TAUs) with the vanilla DeepLabv3+ architecture, where the TAUs are used to bridge the semantic feature gap among encoders output and the SENets used to put more weight on relevant features. The proposed DeepTriNet finds which features are the more relevant and more generalized way by its self-supervision rather we annotate them. The study showed that the proposed DeepTriNet performs better than many conventional techniques with an accuracy of 98% and 77%, IoU 80% and 58%, precision 88% and 68%, and recall of 79% and 55% on the 4-class Land-Cover.ai dataset and the 15-class GID-2 dataset respectively. The proposed method will greatly contribute to natural resource management and change detection in rural and urban regions through efficient and semantic satellite image segmentation

Title: Zero-Shot Open-Vocabulary Tracking with Large Pre-Trained Models. (arXiv:2310.06992v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.06992
• Code URL: null
• Copy Paste: [[2310.06992]] Zero-Shot Open-Vocabulary Tracking with Large Pre-Trained Models(http://arxiv.org/abs/2310.06992)
• Summary:

  Object tracking is central to robot perception and scene understanding. Tracking-by-detection has long been a dominant paradigm for object tracking of specific object categories. Recently, large-scale pre-trained models have shown promising advances in detecting and segmenting objects and parts in 2D static images in the wild. This begs the question: can we re-purpose these large-scale pre-trained static image models for open-vocabulary video tracking? In this paper, we re-purpose an open-vocabulary detector, segmenter, and dense optical flow estimator, into a model that tracks and segments objects of any category in 2D videos. Our method predicts object and part tracks with associated language descriptions in monocular videos, rebuilding the pipeline of Tractor with modern large pre-trained models for static image detection and segmentation: we detect open-vocabulary object instances and propagate their boxes from frame to frame using a flow-based motion model, refine the propagated boxes with the box regression module of the visual detector, and prompt an open-world segmenter with the refined box to segment the objects. We decide the termination of an object track based on the objectness score of the propagated boxes, as well as forward-backward optical flow consistency. We re-identify objects across occlusions using deep feature matching. We show that our model achieves strong performance on multiple established video object segmentation and tracking benchmarks, and can produce reasonable tracks in manipulation data. In particular, our model outperforms previous state-of-the-art in UVO and BURST, benchmarks for open-world object tracking and segmentation, despite never being explicitly trained for tracking. We hope that our approach can serve as a simple and extensible framework for future research.

Title: Multi-task Explainable Skin Lesion Classification. (arXiv:2310.07209v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.07209
• Code URL: null
• Copy Paste: [[2310.07209]] Multi-task Explainable Skin Lesion Classification(http://arxiv.org/abs/2310.07209)
• Summary:

  Skin cancer is one of the deadliest diseases and has a high mortality rate if left untreated. The diagnosis generally starts with visual screening and is followed by a biopsy or histopathological examination. Early detection can aid in lowering mortality rates. Visual screening can be limited by the experience of the doctor. Due to the long tail distribution of dermatological datasets and significant intra-variability between classes, automatic classification utilizing computer-aided methods becomes challenging. In this work, we propose a multitask few-shot-based approach for skin lesions that generalizes well with few labelled data to address the small sample space challenge. The proposed approach comprises a fusion of a segmentation network that acts as an attention module and classification network. The output of the segmentation network helps to focus on the most discriminatory features while making a decision by the classification network. To further enhance the classification performance, we have combined segmentation and classification loss in a weighted manner. We have also included the visualization results that explain the decisions made by the algorithm. Three dermatological datasets are used to evaluate the proposed method thoroughly. We also conducted cross-database experiments to ensure that the proposed approach is generalizable across similar datasets. Experimental results demonstrate the efficacy of the proposed work.

Title: Causal Unsupervised Semantic Segmentation. (arXiv:2310.07379v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.07379
• Code URL: https://github.com/byungkwanlee/causal-unsupervised-segmentation
• Copy Paste: [[2310.07379]] Causal Unsupervised Semantic Segmentation(http://arxiv.org/abs/2310.07379)
• Summary:

  Unsupervised semantic segmentation aims to achieve high-quality semantic grouping without human-labeled annotations. With the advent of self-supervised pre-training, various frameworks utilize the pre-trained features to train prediction heads for unsupervised dense prediction. However, a significant challenge in this unsupervised setup is determining the appropriate level of clustering required for segmenting concepts. To address it, we propose a novel framework, CAusal Unsupervised Semantic sEgmentation (CAUSE), which leverages insights from causal inference. Specifically, we bridge intervention-oriented approach (i.e., frontdoor adjustment) to define suitable two-step tasks for unsupervised prediction. The first step involves constructing a concept clusterbook as a mediator, which represents possible concept prototypes at different levels of granularity in a discretized form. Then, the mediator establishes an explicit link to the subsequent concept-wise self-supervised learning for pixel-level grouping. Through extensive experiments and analyses on various datasets, we corroborate the effectiveness of CAUSE and achieve state-of-the-art performance in unsupervised semantic segmentation.

Title: CLIP for Lightweight Semantic Segmentation. (arXiv:2310.07394v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.07394
• Code URL: null
• Copy Paste: [[2310.07394]] CLIP for Lightweight Semantic Segmentation(http://arxiv.org/abs/2310.07394)
• Summary:

  The large-scale pretrained model CLIP, trained on 400 million image-text pairs, offers a promising paradigm for tackling vision tasks, albeit at the image level. Later works, such as DenseCLIP and LSeg, extend this paradigm to dense prediction, including semantic segmentation, and have achieved excellent results. However, the above methods either rely on CLIP-pretrained visual backbones or use none-pretrained but heavy backbones such as Swin, while falling ineffective when applied to lightweight backbones. The reason for this is that the lightweitht networks, feature extraction ability of which are relatively limited, meet difficulty embedding the image feature aligned with text embeddings perfectly. In this work, we present a new feature fusion module which tackles this problem and enables language-guided paradigm to be applied to lightweight networks. Specifically, the module is a parallel design of CNN and transformer with a two-way bridge in between, where CNN extracts spatial information and visual context of the feature map from the image encoder, and the transformer propagates text embeddings from the text encoder forward. The core of the module is the bidirectional fusion of visual and text feature across the bridge which prompts their proximity and alignment in embedding space. The module is model-agnostic, which can not only make language-guided lightweight semantic segmentation practical, but also fully exploit the pretrained knowledge of language priors and achieve better performance than previous SOTA work, such as DenseCLIP, whatever the vision backbone is. Extensive experiments have been conducted to demonstrate the superiority of our method.

Title: Heuristic Vision Pre-Training with Self-Supervised and Supervised Multi-Task Learning. (arXiv:2310.07510v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.07510
• Code URL: null
• Copy Paste: [[2310.07510]] Heuristic Vision Pre-Training with Self-Supervised and Supervised Multi-Task Learning(http://arxiv.org/abs/2310.07510)
• Summary:

  To mimic human vision with the way of recognizing the diverse and open world, foundation vision models are much critical. While recent techniques of self-supervised learning show the promising potentiality of this mission, we argue that signals from labelled data are also important for common-sense recognition, and properly chosen pre-text tasks can facilitate the efficiency of vision representation learning. To this end, we propose a novel pre-training framework by adopting both self-supervised and supervised visual pre-text tasks in a multi-task manner. Specifically, given an image, we take a heuristic way by considering its intrinsic style properties, inside objects with their locations and correlations, and how it looks like in 3D space for basic visual understanding. However, large-scale object bounding boxes and correlations are usually hard to achieve. Alternatively, we develop a hybrid method by leveraging both multi-label classification and self-supervised learning. On the one hand, under the multi-label supervision, the pre-trained model can explore the detailed information of an image, e.g., image types, objects, and part of semantic relations. On the other hand, self-supervised learning tasks, with respect to Masked Image Modeling (MIM) and contrastive learning, can help the model learn pixel details and patch correlations. Results show that our pre-trained models can deliver results on par with or better than state-of-the-art (SOTA) results on multiple visual tasks. For example, with a vanilla Swin-B backbone, we achieve 85.3\% top-1 accuracy on ImageNet-1K classification, 47.9 box AP on COCO object detection for Mask R-CNN, and 50.6 mIoU on ADE-20K semantic segmentation when using Upernet. The performance shows the ability of our vision foundation model to serve general purpose vision tasks.

Title: S4C: Self-Supervised Semantic Scene Completion with Neural Fields. (arXiv:2310.07522v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.07522
• Code URL: null
• Copy Paste: [[2310.07522]] S4C: Self-Supervised Semantic Scene Completion with Neural Fields(http://arxiv.org/abs/2310.07522)
• Summary:

  3D semantic scene understanding is a fundamental challenge in computer vision. It enables mobile agents to autonomously plan and navigate arbitrary environments. SSC formalizes this challenge as jointly estimating dense geometry and semantic information from sparse observations of a scene. Current methods for SSC are generally trained on 3D ground truth based on aggregated LiDAR scans. This process relies on special sensors and annotation by hand which are costly and do not scale well. To overcome this issue, our work presents the first self-supervised approach to SSC called S4C that does not rely on 3D ground truth data. Our proposed method can reconstruct a scene from a single image and only relies on videos and pseudo segmentation ground truth generated from off-the-shelf image segmentation network during training. Unlike existing methods, which use discrete voxel grids, we represent scenes as implicit semantic fields. This formulation allows querying any point within the camera frustum for occupancy and semantic class. Our architecture is trained through rendering-based self-supervised losses. Nonetheless, our method achieves performance close to fully supervised state-of-the-art methods. Additionally, our method demonstrates strong generalization capabilities and can synthesize accurate segmentation maps for far away viewpoints.

Title: SAM-OCTA: Prompting Segment-Anything for OCTA Image Segmentation. (arXiv:2310.07183v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.07183
• Code URL: https://github.com/shellredia/sam-octa
• Copy Paste: [[2310.07183]] SAM-OCTA: Prompting Segment-Anything for OCTA Image Segmentation(http://arxiv.org/abs/2310.07183)
• Summary:

  In the analysis of optical coherence tomography angiography (OCTA) images, the operation of segmenting specific targets is necessary. Existing methods typically train on supervised datasets with limited samples (approximately a few hundred), which can lead to overfitting. To address this, the low-rank adaptation technique is adopted for foundation model fine-tuning and proposed corresponding prompt point generation strategies to process various segmentation tasks on OCTA datasets. This method is named SAM-OCTA and has been experimented on the publicly available OCTA-500 and ROSE datasets. This method achieves or approaches state-of-the-art segmentation performance metrics. The effect and applicability of prompt points are discussed in detail for the retinal vessel, foveal avascular zone, capillary, artery, and vein segmentation tasks. Furthermore, SAM-OCTA accomplishes local vessel segmentation and effective artery-vein segmentation, which was not well-solved in previous works. The code is available at https://github.com/ShellRedia/SAM-OCTA.