secure

Title: Compact: Approximating Complex Activation Functions for Secure Computation. (arXiv:2309.04664v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.04664
• Code URL: null
• Copy Paste: [[2309.04664]] Compact: Approximating Complex Activation Functions for Secure Computation(http://arxiv.org/abs/2309.04664)
• Summary:

  Secure multi-party computation (MPC) techniques can be used to provide data privacy when users query deep neural network (DNN) models hosted on a public cloud. State-of-the-art MPC techniques can be directly leveraged for DNN models that use simple activation functions (AFs) such as ReLU. However, DNN model architectures designed for cutting-edge applications often use complex and highly non-linear AFs. Designing efficient MPC techniques for such complex AFs is an open problem.

  Towards this, we propose Compact, which produces piece-wise polynomial approximations of complex AFs to enable their efficient use with state-of-the-art MPC techniques. Compact neither requires nor imposes any restriction on model training and results in near-identical model accuracy. We extensively evaluate Compact on four different machine-learning tasks with DNN architectures that use popular complex AFs SiLU, GeLU, and Mish. Our experimental results show that Compact incurs negligible accuracy loss compared to DNN-specific approaches for handling complex non-linear AFs. We also incorporate Compact in two state-of-the-art MPC libraries for privacy-preserving inference and demonstrate that Compact provides 2x-5x speedup in computation compared to the state-of-the-art approximation approach for non-linear functions -- while providing similar or better accuracy for DNN models with large number of hidden layers

Title: From Programming Bugs to Multimillion-Dollar Scams: An Analysis of Trapdoor Tokens on Decentralized Exchanges. (arXiv:2309.04700v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.04700
• Code URL: null
• Copy Paste: [[2309.04700]] From Programming Bugs to Multimillion-Dollar Scams: An Analysis of Trapdoor Tokens on Decentralized Exchanges(http://arxiv.org/abs/2309.04700)
• Summary:

  The rapid development of Blockchain technology and the prosperity of cryptocurrency in the past decade have driven the massive demand for digital assets trading, leading to the emergence of many cryptocurrency exchange platforms. Unlike centralised exchanges (CEXs) where listed tokens and cryptocurrencies are assessed by authorities to make the secured trading environment, decentralized exchanges (DEXs) are introduced to allow users to trade their digital assets without the involvement of any third party, therefore exposing security issues and encouraging the rise of many scams and malicious tokens. In this paper, we investigate an emerging malicious token named Trapdoor, which allows users to buy but prevent them from selling and getting their funds back. The first collection of Trapdoor tokens is constructed in this study by investigating malicious behaviours and maneuvers of these tokens. After manually analysing the tokens' source code, we classify those Trapdoor tokens into different categories according to their malicious code embedding technique. Moreover, we also comprehensively analyse the impact of Trapdoor tokens, the behaviours of scammers, and the characteristics of victims from various perspective. Finally, we also implement and publish our Trapdoor token detection tool and Trapdoor maneuvers analysis reports that help in increasing awareness of investors for this kind of scam.

Title: Approximating ReLU on a Reduced Ring for Efficient MPC-based Private Inference. (arXiv:2309.04875v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.04875
• Code URL: null
• Copy Paste: [[2309.04875]] Approximating ReLU on a Reduced Ring for Efficient MPC-based Private Inference(http://arxiv.org/abs/2309.04875)
• Summary:

  Secure multi-party computation (MPC) allows users to offload machine learning inference on untrusted servers without having to share their privacy-sensitive data. Despite their strong security properties, MPC-based private inference has not been widely adopted in the real world due to their high communication overhead. When evaluating ReLU layers, MPC protocols incur a significant amount of communication between the parties, making the end-to-end execution time multiple orders slower than its non-private counterpart.

  This paper presents HummingBird, an MPC framework that reduces the ReLU communication overhead significantly by using only a subset of the bits to evaluate ReLU on a smaller ring. Based on theoretical analyses, HummingBird identifies bits in the secret share that are not crucial for accuracy and excludes them during ReLU evaluation to reduce communication. With its efficient search engine, HummingBird discards 87--91% of the bits during ReLU and still maintains high accuracy. On a real MPC setup involving multiple servers, HummingBird achieves on average 2.03--2.67x end-to-end speedup without introducing any errors, and up to 8.64x average speedup when some amount of accuracy degradation can be tolerated, due to its up to 8.76x communication reduction.

Title: Transient Attack against the VMG-KLJN Secure Key Exchanger. (arXiv:2309.04899v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.04899
• Code URL: null
• Copy Paste: [[2309.04899]] Transient Attack against the VMG-KLJN Secure Key Exchanger(http://arxiv.org/abs/2309.04899)
• Summary:

  The security vulnerability of the Vadai, Mingesz, and Gingl (VMG) Kirchhoff-Law-Johnson-Noise (KLJN) key exchanger, as presented in the publication "Nature, Science Report 5 (2015) 13653," has been exposed to transient attacks. Recently an effective defense protocol was introduced (Appl. Phys. Lett. 122 (2023) 143503) to counteract mean-square voltage-based (or mean-square current-based) transient attacks targeted at the ideal KLJN framework.

  In the present study, this same mitigation methodology has been employed to fortify the security of the VMG-KLJN key exchanger. It is worth noting that the protective measures need to be separately implemented for the HL and LH scenarios. This conceptual framework is corroborated through computer simulations, demonstrating that the application of this defensive technique substantially mitigates information leakage to a point of insignificance.

security

Title: A Data Middleware for Obtaining Trusted Price Data for Blockchain. (arXiv:2309.04689v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.04689
• Code URL: null
• Copy Paste: [[2309.04689]] A Data Middleware for Obtaining Trusted Price Data for Blockchain(http://arxiv.org/abs/2309.04689)
• Summary:

  As a trusted middleware connecting the blockchain and the real world, the blockchain oracle can obtain trusted real-time price information for financial applications such as payment and settlement, and asset valuation on the blockchain. However, the current oracle schemes face the dilemma of security and service quality in the process of node selection, and the implicit interest relationship in financial applications leads to a significant conflict of interest between the task publisher and the executor, which reduces the participation enthusiasm of both parties and system security. Therefore, this paper proposes an anonymous node selection scheme that anonymously selects nodes with high reputations to participate in tasks to ensure the security and service quality of nodes. Then, this paper also details the interest requirements and behavioral motives of all parties in the payment settlement and asset valuation scenarios. Under the assumption of rational participants, an incentive mechanism based on the Stackelberg game is proposed. It can achieve equilibrium under the pursuit of the interests of task publishers and executors, thereby ensuring the interests of all types of users and improving the enthusiasm of participation. Finally, we verify the security of the proposed scheme through security analysis. The experimental results show that the proposed scheme can reduce the variance of obtaining price data by about 55\% while ensuring security, and meeting the interests of all parties.

Title: Security Analysis of Pairing-based Cryptography. (arXiv:2309.04693v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.04693
• Code URL: null
• Copy Paste: [[2309.04693]] Security Analysis of Pairing-based Cryptography(http://arxiv.org/abs/2309.04693)
• Summary:

  Recent progress in number field sieve (NFS) has shaken the security of Pairing-based Cryptography. For the discrete logarithm problem (DLP) in finite field, we present the first systematic review of the NFS algorithms from three perspectives: the degree $\alpha$, constant $c$, and hidden constant $o(1)$ in the asymptotic complexity $L_Q\left(\alpha,c\right)$ and indicate that further research is required to optimize the hidden constant. Using the special extended tower NFS algorithm, we conduct a thorough security evaluation for all the existing standardized PF curves as well as several commonly utilized curves, which reveals that the BN256 curves recommended by the SM9 and the previous ISO/IEC standard exhibit only 99.92 bits of security, significantly lower than the intended 128-bit level. In addition, we comprehensively analyze the security and efficiency of BN, BLS, and KSS curves for different security levels. Our analysis suggests that the BN curve exhibits superior efficiency for security strength below approximately 105 bit. For a 128-bit security level, BLS12 and BLS24 curves are the optimal choices, while the BLS24 curve offers the best efficiency for security levels of 160bit, 192bit, and 256bit.

Title: The Effectiveness of Security Interventions on GitHub. (arXiv:2309.04833v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.04833
• Code URL: null
• Copy Paste: [[2309.04833]] The Effectiveness of Security Interventions on GitHub(http://arxiv.org/abs/2309.04833)
• Summary:

  Since 2017, GitHub has been the first online open source platform to show security warnings to its users. It has since introduced further security interventions to help developers improve the security of their open source software. In this study, we investigate and compare the effects of these interventions. We perform time series analysis of security-altering commits to infer the causal effects of the interventions. Our analysis shows that while all of GitHub's security interventions have a significant positive effect on security, they differ greatly in their effect size. By comparing the design of each intervention, we identify the building blocks that worked well and those that did not. We also provide recommendations on how practitioners can improve the design of their interventions to enhance their effectiveness.

Title: A Review of Machine Learning-based Security in Cloud Computing. (arXiv:2309.04911v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.04911
• Code URL: null
• Copy Paste: [[2309.04911]] A Review of Machine Learning-based Security in Cloud Computing(http://arxiv.org/abs/2309.04911)
• Summary:

  Cloud Computing (CC) is revolutionizing the way IT resources are delivered to users, allowing them to access and manage their systems with increased cost-effectiveness and simplified infrastructure. However, with the growth of CC comes a host of security risks, including threats to availability, integrity, and confidentiality. To address these challenges, Machine Learning (ML) is increasingly being used by Cloud Service Providers (CSPs) to reduce the need for human intervention in identifying and resolving security issues. With the ability to analyze vast amounts of data, and make high-accuracy predictions, ML can transform the way CSPs approach security. In this paper, we will explore some of the most recent research in the field of ML-based security in Cloud Computing. We will examine the features and effectiveness of a range of ML algorithms, highlighting their unique strengths and potential limitations. Our goal is to provide a comprehensive overview of the current state of ML in cloud security and to shed light on the exciting possibilities that this emerging field has to offer.

privacy

Title: Privacy Preserving Federated Learning with Convolutional Variational Bottlenecks. (arXiv:2309.04515v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.04515
• Code URL: null
• Copy Paste: [[2309.04515]] Privacy Preserving Federated Learning with Convolutional Variational Bottlenecks(http://arxiv.org/abs/2309.04515)
• Summary:

  Gradient inversion attacks are an ubiquitous threat in federated learning as they exploit gradient leakage to reconstruct supposedly private training data. Recent work has proposed to prevent gradient leakage without loss of model utility by incorporating a PRivacy EnhanCing mODulE (PRECODE) based on variational modeling. Without further analysis, it was shown that PRECODE successfully protects against gradient inversion attacks. In this paper, we make multiple contributions. First, we investigate the effect of PRECODE on gradient inversion attacks to reveal its underlying working principle. We show that variational modeling introduces stochasticity into the gradients of PRECODE and the subsequent layers in a neural network. The stochastic gradients of these layers prevent iterative gradient inversion attacks from converging. Second, we formulate an attack that disables the privacy preserving effect of PRECODE by purposefully omitting stochastic gradients during attack optimization. To preserve the privacy preserving effect of PRECODE, our analysis reveals that variational modeling must be placed early in the network. However, early placement of PRECODE is typically not feasible due to reduced model utility and the exploding number of additional model parameters. Therefore, as a third contribution, we propose a novel privacy module -- the Convolutional Variational Bottleneck (CVB) -- that can be placed early in a neural network without suffering from these drawbacks. We conduct an extensive empirical study on three seminal model architectures and six image classification datasets. We find that all architectures are susceptible to gradient leakage attacks, which can be prevented by our proposed CVB. Compared to PRECODE, we show that our novel privacy module requires fewer trainable parameters, and thus computational and communication costs, to effectively preserve privacy.

Title: The Complexity of Verifying Boolean Programs as Differentially Private. (arXiv:2309.04642v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.04642
• Code URL: null
• Copy Paste: [[2309.04642]] The Complexity of Verifying Boolean Programs as Differentially Private(http://arxiv.org/abs/2309.04642)
• Summary:

  We study the complexity of the problem of verifying differential privacy for while-like programs working over boolean values and making probabilistic choices. Programs in this class can be interpreted into finite-state discrete-time Markov Chains (DTMC). We show that the problem of deciding whether a program is differentially private for specific values of the privacy parameters is PSPACE-complete. To show that this problem is in PSPACE, we adapt classical results about computing hitting probabilities for DTMC. To show PSPACE-hardness we use a reduction from the problem of checking whether a program almost surely terminates or not. We also show that the problem of approximating the privacy parameters that a program provides is PSPACE-hard. Moreover, we investigate the complexity of similar problems also for several relaxations of differential privacy: R\'enyi differential privacy, concentrated differential privacy, and truncated concentrated differential privacy. For these notions, we consider gap-versions of the problem of deciding whether a program is private or not and we show that all of them are PSPACE-complete.

Title: Bicoptor 2.0: Addressing Challenges in Probabilistic Truncation for Enhanced Privacy-Preserving Machine Learning. (arXiv:2309.04909v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.04909
• Code URL: null
• Copy Paste: [[2309.04909]] Bicoptor 2(http://arxiv.org/abs/2309.04909)
• Summary:

  This paper primarily focuses on analyzing the problems and proposing solutions for the probabilistic truncation protocol in existing PPML works from the perspectives of accuracy and efficiency. In terms of accuracy, we reveal that precision selections recommended in some of the existing works are incorrect. We conduct a thorough analysis of their open-source code and find that their errors were mainly due to simplified implementation, more specifically, fixed numbers are used instead of random numbers in probabilistic truncation protocols. Based on this, we provide a detailed theoretical analysis to validate our views. We propose a solution and a precision selection guideline for future works. Regarding efficiency, we identify limitations in the state-of-the-art comparison protocol, Bicoptor's (S\&P 2023) DReLU protocol, which relies on the probabilistic truncation protocol and is heavily constrained by the security parameter to avoid errors, significantly impacting the protocol's performance. To address these challenges, we introduce the first non-interactive deterministic truncation protocol, replacing the original probabilistic truncation protocol. Additionally, we design a non-interactive modulo switch protocol to enhance the protocol's security. Finally, we provide a guideline to reduce computational and communication overhead by using only a portion of the bits of the input, i.e., the key bits, for DReLU operations based on different model parameters. With the help of key bits, the performance of our DReLU protocol is further improved. We evaluate the performance of our protocols on three GPU servers, and achieve a 10x improvement in DReLU protocol, and a 6x improvement in the ReLU protocol over the state-of-the-art work Piranha-Falcon (USENIX Sec 22). Overall, the performance of our end-to-end (E2E) privacy-preserving machine learning (PPML) inference is improved by 3-4 times.

protect

defense

attack

Title: Semi-Supervised learning for Face Anti-Spoofing using Apex frame. (arXiv:2309.04958v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.04958
• Code URL: null
• Copy Paste: [[2309.04958]] Semi-Supervised learning for Face Anti-Spoofing using Apex frame(http://arxiv.org/abs/2309.04958)
• Summary:

  Conventional feature extraction techniques in the face anti-spoofing domain either analyze the entire video sequence or focus on a specific segment to improve model performance. However, identifying the optimal frames that provide the most valuable input for the face anti-spoofing remains a challenging task. In this paper, we address this challenge by employing Gaussian weighting to create apex frames for videos. Specifically, an apex frame is derived from a video by computing a weighted sum of its frames, where the weights are determined using a Gaussian distribution centered around the video's central frame. Furthermore, we explore various temporal lengths to produce multiple unlabeled apex frames using a Gaussian function, without the need for convolution. By doing so, we leverage the benefits of semi-supervised learning, which considers both labeled and unlabeled apex frames to effectively discriminate between live and spoof classes. Our key contribution emphasizes the apex frame's capacity to represent the most significant moments in the video, while unlabeled apex frames facilitate efficient semi-supervised learning, as they enable the model to learn from videos of varying temporal lengths. Experimental results using four face anti-spoofing databases: CASIA, REPLAY-ATTACK, OULU-NPU, and MSU-MFSD demonstrate the apex frame's efficacy in advancing face anti-spoofing techniques.

Title: Reverse-Engineering Decoding Strategies Given Blackbox Access to a Language Generation System. (arXiv:2309.04858v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.04858
• Code URL: null
• Copy Paste: [[2309.04858]] Reverse-Engineering Decoding Strategies Given Blackbox Access to a Language Generation System(http://arxiv.org/abs/2309.04858)
• Summary:

  Neural language models are increasingly deployed into APIs and websites that allow a user to pass in a prompt and receive generated text. Many of these systems do not reveal generation parameters. In this paper, we present methods to reverse-engineer the decoding method used to generate text (i.e., top-$k$ or nucleus sampling). Our ability to discover which decoding strategy was used has implications for detecting generated text. Additionally, the process of discovering the decoding strategy can reveal biases caused by selecting decoding settings which severely truncate a model's predicted distributions. We perform our attack on several families of open-source language models, as well as on production systems (e.g., ChatGPT).

Title: Leakage-Abuse Attacks Against Forward and Backward Private Searchable Symmetric Encryption. (arXiv:2309.04697v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.04697
• Code URL: null
• Copy Paste: [[2309.04697]] Leakage-Abuse Attacks Against Forward and Backward Private Searchable Symmetric Encryption(http://arxiv.org/abs/2309.04697)
• Summary:

  Dynamic searchable symmetric encryption (DSSE) enables a server to efficiently search and update over encrypted files. To minimize the leakage during updates, a security notion named forward and backward privacy is expected for newly proposed DSSE schemes. Those schemes are generally constructed in a way to break the linkability across search and update queries to a given keyword. However, it remains underexplored whether forward and backward private DSSE is resilient against practical leakage-abuse attacks (LAAs), where an attacker attempts to recover query keywords from the leakage passively collected during queries.

  In this paper, we aim to be the first to answer this question firmly through two non-trivial efforts. First, we revisit the spectrum of forward and backward private DSSE schemes over the past few years, and unveil some inherent constructional limitations in most schemes. Those limitations allow attackers to exploit query equality and establish a guaranteed linkage among different (refreshed) query tokens surjective to a candidate keyword. Second, we refine volumetric leakage profiles of updates and queries by associating each with a specific operation. By further exploiting update volume and query response volume, we demonstrate that all forward and backward private DSSE schemes can leak the same volumetric information (e.g., insertion volume, deletion volume) as those without such security guarantees. To testify our findings, we realize two generic LAAs, i.e., frequency matching attack and volumetric inference attack, and we evaluate them over various experimental settings in the dynamic context. Finally, we call for new efficient schemes to protect query equality and volumetric information across search and update queries.

Title: Characterizing Cyber Attacks against Space Systems with Missing Data: Framework and Case Study. (arXiv:2309.04878v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.04878
• Code URL: null
• Copy Paste: [[2309.04878]] Characterizing Cyber Attacks against Space Systems with Missing Data: Framework and Case Study(http://arxiv.org/abs/2309.04878)
• Summary:

  Cybersecurity of space systems is an emerging topic, but there is no single dataset that documents cyber attacks against space systems that have occurred in the past. These incidents are often scattered in media reports while missing many details, which we dub the missing-data problem. Nevertheless, even "low-quality" datasets containing such reports would be extremely valuable because of the dearth of space cybersecurity data and the sensitivity of space systems which are often restricted from disclosure by governments. This prompts a research question: How can we characterize real-world cyber attacks against space systems? In this paper, we address the problem by proposing a framework, including metrics, while also addressing the missing-data problem, by "extrapolating" the missing data in a principled fashion. To show the usefulness of the framework, we extract data for 72 cyber attacks against space systems and show how to extrapolate this "low-quality" dataset to derive 4,076 attack technique kill chains. Our findings include: cyber attacks against space systems are getting increasingly sophisticated; and, successful protection against on-path and social engineering attacks could have prevented 80% of the attacks.

robust

Title: On the Efficacy of Multi-scale Data Samplers for Vision Applications. (arXiv:2309.04502v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.04502
• Code URL: null
• Copy Paste: [[2309.04502]] On the Efficacy of Multi-scale Data Samplers for Vision Applications(http://arxiv.org/abs/2309.04502)
• Summary:

  Multi-scale resolution training has seen an increased adoption across multiple vision tasks, including classification and detection. Training with smaller resolutions enables faster training at the expense of a drop in accuracy. Conversely, training with larger resolutions has been shown to improve performance, but memory constraints often make this infeasible. In this paper, we empirically study the properties of multi-scale training procedures. We focus on variable batch size multi-scale data samplers that randomly sample an input resolution at each training iteration and dynamically adjust their batch size according to the resolution. Such samplers have been shown to improve model accuracy beyond standard training with a fixed batch size and resolution, though it is not clear why this is the case. We explore the properties of these data samplers by performing extensive experiments on ResNet-101 and validate our conclusions across multiple architectures, tasks, and datasets. We show that multi-scale samplers behave as implicit data regularizers and accelerate training speed. Compared to models trained with single-scale samplers, we show that models trained with multi-scale samplers retain or improve accuracy, while being better-calibrated and more robust to scaling and data distribution shifts. We additionally extend a multi-scale variable batch sampler with a simple curriculum that progressively grows resolutions throughout training, allowing for a compute reduction of more than 30%. We show that the benefits of multi-scale training extend to detection and instance segmentation tasks, where we observe a 37% reduction in training FLOPs along with a 3-4% mAP increase on MS-COCO using a Mask R-CNN model.

Title: Poster: Making Edge-assisted LiDAR Perceptions Robust to Lossy Point Cloud Compression. (arXiv:2309.04549v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.04549
• Code URL: null
• Copy Paste: [[2309.04549]] Poster: Making Edge-assisted LiDAR Perceptions Robust to Lossy Point Cloud Compression(http://arxiv.org/abs/2309.04549)
• Summary:

  Real-time light detection and ranging (LiDAR) perceptions, e.g., 3D object detection and simultaneous localization and mapping are computationally intensive to mobile devices of limited resources and often offloaded on the edge. Offloading LiDAR perceptions requires compressing the raw sensor data, and lossy compression is used for efficiently reducing the data volume. Lossy compression degrades the quality of LiDAR point clouds, and the perception performance is decreased consequently. In this work, we present an interpolation algorithm improving the quality of a LiDAR point cloud to mitigate the perception performance loss due to lossy compression. The algorithm targets the range image (RI) representation of a point cloud and interpolates points at the RI based on depth gradients. Compared to existing image interpolation algorithms, our algorithm shows a better qualitative result when the point cloud is reconstructed from the interpolated RI. With the preliminary results, we also describe the next steps of the current work.

Title: Exploring Robust Features for Improving Adversarial Robustness. (arXiv:2309.04650v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.04650
• Code URL: null
• Copy Paste: [[2309.04650]] Exploring Robust Features for Improving Adversarial Robustness(http://arxiv.org/abs/2309.04650)
• Summary:

  While deep neural networks (DNNs) have revolutionized many fields, their fragility to carefully designed adversarial attacks impedes the usage of DNNs in safety-critical applications. In this paper, we strive to explore the robust features which are not affected by the adversarial perturbations, i.e., invariant to the clean image and its adversarial examples, to improve the model's adversarial robustness. Specifically, we propose a feature disentanglement model to segregate the robust features from non-robust features and domain specific features. The extensive experiments on four widely used datasets with different attacks demonstrate that robust features obtained from our model improve the model's adversarial robustness compared to the state-of-the-art approaches. Moreover, the trained domain discriminator is able to identify the domain specific features from the clean images and adversarial examples almost perfectly. This enables adversarial example detection without incurring additional computational costs. With that, we can also specify different classifiers for clean images and adversarial examples, thereby avoiding any drop in clean image accuracy.

Title: DeNoising-MOT: Towards Multiple Object Tracking with Severe Occlusions. (arXiv:2309.04682v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.04682
• Code URL: null
• Copy Paste: [[2309.04682]] DeNoising-MOT: Towards Multiple Object Tracking with Severe Occlusions(http://arxiv.org/abs/2309.04682)
• Summary:

  Multiple object tracking (MOT) tends to become more challenging when severe occlusions occur. In this paper, we analyze the limitations of traditional Convolutional Neural Network-based methods and Transformer-based methods in handling occlusions and propose DNMOT, an end-to-end trainable DeNoising Transformer for MOT. To address the challenge of occlusions, we explicitly simulate the scenarios when occlusions occur. Specifically, we augment the trajectory with noises during training and make our model learn the denoising process in an encoder-decoder architecture, so that our model can exhibit strong robustness and perform well under crowded scenes. Additionally, we propose a Cascaded Mask strategy to better coordinate the interaction between different types of queries in the decoder to prevent the mutual suppression between neighboring trajectories under crowded scenes. Notably, the proposed method requires no additional modules like matching strategy and motion state estimation in inference. We conduct extensive experiments on the MOT17, MOT20, and DanceTrack datasets, and the experimental results show that our method outperforms previous state-of-the-art methods by a clear margin.

Title: Towards Robust Model Watermark via Reducing Parametric Vulnerability. (arXiv:2309.04777v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.04777
• Code URL: https://github.com/guanhaogan/robust-model-watermarking
• Copy Paste: [[2309.04777]] Towards Robust Model Watermark via Reducing Parametric Vulnerability(http://arxiv.org/abs/2309.04777)
• Summary:

  Deep neural networks are valuable assets considering their commercial benefits and huge demands for costly annotation and computation resources. To protect the copyright of DNNs, backdoor-based ownership verification becomes popular recently, in which the model owner can watermark the model by embedding a specific backdoor behavior before releasing it. The defenders (usually the model owners) can identify whether a suspicious third-party model is ``stolen'' from them based on the presence of the behavior. Unfortunately, these watermarks are proven to be vulnerable to removal attacks even like fine-tuning. To further explore this vulnerability, we investigate the parameter space and find there exist many watermark-removed models in the vicinity of the watermarked one, which may be easily used by removal attacks. Inspired by this finding, we propose a mini-max formulation to find these watermark-removed models and recover their watermark behavior. Extensive experiments demonstrate that our method improves the robustness of the model watermarking against parametric changes and numerous watermark-removal attacks. The codes for reproducing our main experiments are available at \url{https://github.com/GuanhaoGan/robust-model-watermarking}.

Title: Can NLP Models 'Identify', 'Distinguish', and 'Justify' Questions that Don't have a Definitive Answer?. (arXiv:2309.04635v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.04635
• Code URL: null
• Copy Paste: [[2309.04635]] Can NLP Models 'Identify', 'Distinguish', and 'Justify' Questions that Don't have a Definitive Answer?(http://arxiv.org/abs/2309.04635)
• Summary:

  Though state-of-the-art (SOTA) NLP systems have achieved remarkable performance on a variety of language understanding tasks, they primarily focus on questions that have a correct and a definitive answer. However, in real-world applications, users often ask questions that don't have a definitive answer. Incorrectly answering such questions certainly hampers a system's reliability and trustworthiness. Can SOTA models accurately identify such questions and provide a reasonable response?

  To investigate the above question, we introduce QnotA, a dataset consisting of five different categories of questions that don't have definitive answers. Furthermore, for each QnotA instance, we also provide a corresponding QA instance i.e. an alternate question that ''can be'' answered. With this data, we formulate three evaluation tasks that test a system's ability to 'identify', 'distinguish', and 'justify' QnotA questions. Through comprehensive experiments, we show that even SOTA models including GPT-3 and Flan T5 do not fare well on these tasks and lack considerably behind the human performance baseline. We conduct a thorough analysis which further leads to several interesting findings. Overall, we believe our work and findings will encourage and facilitate further research in this important area and help develop more robust models.

Title: Multi-document Summarization: A Comparative Evaluation. (arXiv:2309.04951v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.04951
• Code URL: null
• Copy Paste: [[2309.04951]] Multi-document Summarization: A Comparative Evaluation(http://arxiv.org/abs/2309.04951)
• Summary:

  This paper is aimed at evaluating state-of-the-art models for Multi-document Summarization (MDS) on different types of datasets in various domains and investigating the limitations of existing models to determine future research directions. To address this gap, we conducted an extensive literature review to identify state-of-the-art models and datasets. We analyzed the performance of PRIMERA and PEGASUS models on BigSurvey-MDS and MS$^2$ datasets, which posed unique challenges due to their varied domains. Our findings show that the General-Purpose Pre-trained Model LED outperforms PRIMERA and PEGASUS on the MS$^2$ dataset. We used the ROUGE score as a performance metric to evaluate the identified models on different datasets. Our study provides valuable insights into the models' strengths and weaknesses, as well as their applicability in different domains. This work serves as a reference for future MDS research and contributes to the development of accurate and robust models which can be utilized on demanding datasets with academically and/or scientifically complex data as well as generalized, relatively simple datasets.

Title: Low-Quality Training Data Only? A Robust Framework for Detecting Encrypted Malicious Network Traffic. (arXiv:2309.04798v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.04798
• Code URL: null
• Copy Paste: [[2309.04798]] Low-Quality Training Data Only? A Robust Framework for Detecting Encrypted Malicious Network Traffic(http://arxiv.org/abs/2309.04798)
• Summary:

  Machine learning (ML) is promising in accurately detecting malicious flows in encrypted network traffic; however, it is challenging to collect a training dataset that contains a sufficient amount of encrypted malicious data with correct labels. When ML models are trained with low-quality training data, they suffer degraded performance. In this paper, we aim at addressing a real-world low-quality training dataset problem, namely, detecting encrypted malicious traffic generated by continuously evolving malware. We develop RAPIER that fully utilizes different distributions of normal and malicious traffic data in the feature space, where normal data is tightly distributed in a certain area and the malicious data is scattered over the entire feature space to augment training data for model training. RAPIER includes two pre-processing modules to convert traffic into feature vectors and correct label noises. We evaluate our system on two public datasets and one combined dataset. With 1000 samples and 45% noises from each dataset, our system achieves the F1 scores of 0.770, 0.776, and 0.855, respectively, achieving average improvements of 352.6%, 284.3%, and 214.9% over the existing methods, respectively. Furthermore, We evaluate RAPIER with a real-world dataset obtained from a security enterprise. RAPIER effectively achieves encrypted malicious traffic detection with the best F1 score of 0.773 and improves the F1 score of existing methods by an average of 272.5%.

Title: Self-optimizing Feature Generation via Categorical Hashing Representation and Hierarchical Reinforcement Crossing. (arXiv:2309.04612v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.04612
• Code URL: https://github.com/yingwangyang/hrc_feature_cross
• Copy Paste: [[2309.04612]] Self-optimizing Feature Generation via Categorical Hashing Representation and Hierarchical Reinforcement Crossing(http://arxiv.org/abs/2309.04612)
• Summary:

  Feature generation aims to generate new and meaningful features to create a discriminative representation space.A generated feature is meaningful when the generated feature is from a feature pair with inherent feature interaction. In the real world, experienced data scientists can identify potentially useful feature-feature interactions, and generate meaningful dimensions from an exponentially large search space, in an optimal crossing form over an optimal generation path. But, machines have limited human-like abilities.We generalize such learning tasks as self-optimizing feature generation. Self-optimizing feature generation imposes several under-addressed challenges on existing systems: meaningful, robust, and efficient generation. To tackle these challenges, we propose a principled and generic representation-crossing framework to solve self-optimizing feature generation.To achieve hashing representation, we propose a three-step approach: feature discretization, feature hashing, and descriptive summarization. To achieve reinforcement crossing, we develop a hierarchical reinforcement feature crossing approach.We present extensive experimental results to demonstrate the effectiveness and efficiency of the proposed method. The code is available at https://github.com/yingwangyang/HRC_feature_cross.git.

Title: Flexible and Robust Counterfactual Explanations with Minimal Satisfiable Perturbations. (arXiv:2309.04676v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.04676
• Code URL: https://github.com/wangyongjie-ntu/cemsp
• Copy Paste: [[2309.04676]] Flexible and Robust Counterfactual Explanations with Minimal Satisfiable Perturbations(http://arxiv.org/abs/2309.04676)
• Summary:

  Counterfactual explanations (CFEs) exemplify how to minimally modify a feature vector to achieve a different prediction for an instance. CFEs can enhance informational fairness and trustworthiness, and provide suggestions for users who receive adverse predictions. However, recent research has shown that multiple CFEs can be offered for the same instance or instances with slight differences. Multiple CFEs provide flexible choices and cover diverse desiderata for user selection. However, individual fairness and model reliability will be damaged if unstable CFEs with different costs are returned. Existing methods fail to exploit flexibility and address the concerns of non-robustness simultaneously. To address these issues, we propose a conceptually simple yet effective solution named Counterfactual Explanations with Minimal Satisfiable Perturbations (CEMSP). Specifically, CEMSP constrains changing values of abnormal features with the help of their semantically meaningful normal ranges. For efficiency, we model the problem as a Boolean satisfiability problem to modify as few features as possible. Additionally, CEMSP is a general framework and can easily accommodate more practical requirements, e.g., casualty and actionability. Compared to existing methods, we conduct comprehensive experiments on both synthetic and real-world datasets to demonstrate that our method provides more robust explanations while preserving flexibility.

Title: Weak-PDE-LEARN: A Weak Form Based Approach to Discovering PDEs From Noisy, Limited Data. (arXiv:2309.04699v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.04699
• Code URL: https://github.com/punkduckable/weak_pde_learn
• Copy Paste: [[2309.04699]] Weak-PDE-LEARN: A Weak Form Based Approach to Discovering PDEs From Noisy, Limited Data(http://arxiv.org/abs/2309.04699)
• Summary:

  We introduce Weak-PDE-LEARN, a Partial Differential Equation (PDE) discovery algorithm that can identify non-linear PDEs from noisy, limited measurements of their solutions. Weak-PDE-LEARN uses an adaptive loss function based on weak forms to train a neural network, $U$, to approximate the PDE solution while simultaneously identifying the governing PDE. This approach yields an algorithm that is robust to noise and can discover a range of PDEs directly from noisy, limited measurements of their solutions. We demonstrate the efficacy of Weak-PDE-LEARN by learning several benchmark PDEs.

Title: Distance-Restricted Folklore Weisfeiler-Leman GNNs with Provable Cycle Counting Power. (arXiv:2309.04941v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.04941
• Code URL: null
• Copy Paste: [[2309.04941]] Distance-Restricted Folklore Weisfeiler-Leman GNNs with Provable Cycle Counting Power(http://arxiv.org/abs/2309.04941)
• Summary:

  The ability of graph neural networks (GNNs) to count certain graph substructures, especially cycles, is important for the success of GNNs on a wide range of tasks. It has been recently used as a popular metric for evaluating the expressive power of GNNs. Many of the proposed GNN models with provable cycle counting power are based on subgraph GNNs, i.e., extracting a bag of subgraphs from the input graph, generating representations for each subgraph, and using them to augment the representation of the input graph. However, those methods require heavy preprocessing, and suffer from high time and memory costs. In this paper, we overcome the aforementioned limitations of subgraph GNNs by proposing a novel class of GNNs -- $d$-Distance-Restricted FWL(2) GNNs, or $d$-DRFWL(2) GNNs. $d$-DRFWL(2) GNNs use node pairs whose mutual distances are at most $d$ as the units for message passing to balance the expressive power and complexity. By performing message passing among distance-restricted node pairs in the original graph, $d$-DRFWL(2) GNNs avoid the expensive subgraph extraction operations in subgraph GNNs, making both the time and space complexity lower. We theoretically show that the discriminative power of $d$-DRFWL(2) GNNs strictly increases as $d$ increases. More importantly, $d$-DRFWL(2) GNNs have provably strong cycle counting power even with $d=2$: they can count all 3, 4, 5, 6-cycles. Since 6-cycles (e.g., benzene rings) are ubiquitous in organic molecules, being able to detect and count them is crucial for achieving robust and generalizable performance on molecular tasks. Experiments on both synthetic datasets and molecular datasets verify our theory. To the best of our knowledge, our model is the most efficient GNN model to date (both theoretically and empirically) that can count up to 6-cycles.

biometric

steal

extraction

Title: Speech Emotion Recognition with Distilled Prosodic and Linguistic Affect Representations. (arXiv:2309.04849v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.04849
• Code URL: null
• Copy Paste: [[2309.04849]] Speech Emotion Recognition with Distilled Prosodic and Linguistic Affect Representations(http://arxiv.org/abs/2309.04849)
• Summary:

  We propose EmoDistill, a novel speech emotion recognition (SER) framework that leverages cross-modal knowledge distillation during training to learn strong linguistic and prosodic representations of emotion from speech. During inference, our method only uses a stream of speech signals to perform unimodal SER thus reducing computation overhead and avoiding run-time transcription and prosodic feature extraction errors. During training, our method distills information at both embedding and logit levels from a pair of pre-trained Prosodic and Linguistic teachers that are fine-tuned for SER. Experiments on the IEMOCAP benchmark demonstrate that our method outperforms other unimodal and multimodal techniques by a considerable margin, and achieves state-of-the-art performance of 77.49% unweighted accuracy and 78.91% weighted accuracy. Detailed ablation studies demonstrate the impact of each component of our method.

membership infer

federate

Title: Towards Real-World Burst Image Super-Resolution: Benchmark and Method. (arXiv:2309.04803v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.04803
• Code URL: https://github.com/yjsunnn/fbanet
• Copy Paste: [[2309.04803]] Towards Real-World Burst Image Super-Resolution: Benchmark and Method(http://arxiv.org/abs/2309.04803)
• Summary:

  Despite substantial advances, single-image super-resolution (SISR) is always in a dilemma to reconstruct high-quality images with limited information from one input image, especially in realistic scenarios. In this paper, we establish a large-scale real-world burst super-resolution dataset, i.e., RealBSR, to explore the faithful reconstruction of image details from multiple frames. Furthermore, we introduce a Federated Burst Affinity network (FBAnet) to investigate non-trivial pixel-wise displacements among images under real-world image degradation. Specifically, rather than using pixel-wise alignment, our FBAnet employs a simple homography alignment from a structural geometry aspect and a Federated Affinity Fusion (FAF) strategy to aggregate the complementary information among frames. Those fused informative representations are fed to a Transformer-based module of burst representation decoding. Besides, we have conducted extensive experiments on two versions of our datasets, i.e., RealBSR-RAW and RealBSR-RGB. Experimental results demonstrate that our FBAnet outperforms existing state-of-the-art burst SR methods and also achieves visually-pleasant SR image predictions with model details. Our dataset, codes, and models are publicly available at https://github.com/yjsunnn/FBANet.

Title: Regret-Optimal Federated Transfer Learning for Kernel Regression with Applications in American Option Pricing. (arXiv:2309.04557v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.04557
• Code URL: null
• Copy Paste: [[2309.04557]] Regret-Optimal Federated Transfer Learning for Kernel Regression with Applications in American Option Pricing(http://arxiv.org/abs/2309.04557)
• Summary:

  We propose an optimal iterative scheme for federated transfer learning, where a central planner has access to datasets ${\cal D}_1,\dots,{\cal D}_N$ for the same learning model $f_{\theta}$. Our objective is to minimize the cumulative deviation of the generated parameters $\{\theta_i(t)\}_{t=0}^T$ across all $T$ iterations from the specialized parameters $\theta^\star_{1},\ldots,\theta^\star_N$ obtained for each dataset, while respecting the loss function for the model $f_{\theta(T)}$ produced by the algorithm upon halting. We only allow for continual communication between each of the specialized models (nodes/agents) and the central planner (server), at each iteration (round). For the case where the model $f_{\theta}$ is a finite-rank kernel regression, we derive explicit updates for the regret-optimal algorithm. By leveraging symmetries within the regret-optimal algorithm, we further develop a nearly regret-optimal heuristic that runs with $\mathcal{O}(Np^2)$ fewer elementary operations, where $p$ is the dimension of the parameter space. Additionally, we investigate the adversarial robustness of the regret-optimal algorithm showing that an adversary which perturbs $q$ training pairs by at-most $\varepsilon>0$, across all training sets, cannot reduce the regret-optimal algorithm's regret by more than $\mathcal{O}(\varepsilon q \bar{N}^{1/2})$, where $\bar{N}$ is the aggregate number of training pairs. To validate our theoretical findings, we conduct numerical experiments in the context of American option pricing, utilizing a randomly generated finite-rank kernel.

fair

Title: When to Learn What: Model-Adaptive Data Augmentation Curriculum. (arXiv:2309.04747v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.04747
• Code URL: null
• Copy Paste: [[2309.04747]] When to Learn What: Model-Adaptive Data Augmentation Curriculum(http://arxiv.org/abs/2309.04747)
• Summary:

  Data augmentation (DA) is widely used to improve the generalization of neural networks by enforcing the invariances and symmetries to pre-defined transformations applied to input data. However, a fixed augmentation policy may have different effects on each sample in different training stages but existing approaches cannot adjust the policy to be adaptive to each sample and the training model. In this paper, we propose Model Adaptive Data Augmentation (MADAug) that jointly trains an augmentation policy network to teach the model when to learn what. Unlike previous work, MADAug selects augmentation operators for each input image by a model-adaptive policy varying between training stages, producing a data augmentation curriculum optimized for better generalization. In MADAug, we train the policy through a bi-level optimization scheme, which aims to minimize a validation-set loss of a model trained using the policy-produced data augmentations. We conduct an extensive evaluation of MADAug on multiple image classification tasks and network architectures with thorough comparisons to existing DA approaches. MADAug outperforms or is on par with other baselines and exhibits better fairness: it brings improvement to all classes and more to the difficult ones. Moreover, MADAug learned policy shows better performance when transferred to fine-grained datasets. In addition, the auto-optimized policy in MADAug gradually introduces increasing perturbations and naturally forms an easy-to-hard curriculum.

interpretability

Title: Training of Spiking Neural Network joint Curriculum Learning Strategy. (arXiv:2309.04737v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.04737
• Code URL: null
• Copy Paste: [[2309.04737]] Training of Spiking Neural Network joint Curriculum Learning Strategy(http://arxiv.org/abs/2309.04737)
• Summary:

  Starting with small and simple concepts, and gradually introducing complex and difficult concepts is the natural process of human learning. Spiking Neural Networks (SNNs) aim to mimic the way humans process information, but current SNNs models treat all samples equally, which does not align with the principles of human learning and overlooks the biological plausibility of SNNs. To address this, we propose a CL-SNN model that introduces Curriculum Learning(CL) into SNNs, making SNNs learn more like humans and providing higher biological interpretability. CL is a training strategy that advocates presenting easier data to models before gradually introducing more challenging data, mimicking the human learning process. We use a confidence-aware loss to measure and process the samples with different difficulty levels. By learning the confidence of different samples, the model reduces the contribution of difficult samples to parameter optimization automatically. We conducted experiments on static image datasets MNIST, Fashion-MNIST, CIFAR10, and neuromorphic datasets N-MNIST, CIFAR10-DVS, DVS-Gesture. The results are promising. To our best knowledge, this is the first proposal to enhance the biologically plausibility of SNNs by introducing CL.

explainability

watermark

diffusion

Title: Effective Real Image Editing with Accelerated Iterative Diffusion Inversion. (arXiv:2309.04907v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.04907
• Code URL: null
• Copy Paste: [[2309.04907]] Effective Real Image Editing with Accelerated Iterative Diffusion Inversion(http://arxiv.org/abs/2309.04907)
• Summary:

  Despite all recent progress, it is still challenging to edit and manipulate natural images with modern generative models. When using Generative Adversarial Network (GAN), one major hurdle is in the inversion process mapping a real image to its corresponding noise vector in the latent space, since its necessary to be able to reconstruct an image to edit its contents. Likewise for Denoising Diffusion Implicit Models (DDIM), the linearization assumption in each inversion step makes the whole deterministic inversion process unreliable. Existing approaches that have tackled the problem of inversion stability often incur in significant trade-offs in computational efficiency. In this work we propose an Accelerated Iterative Diffusion Inversion method, dubbed AIDI, that significantly improves reconstruction accuracy with minimal additional overhead in space and time complexity. By using a novel blended guidance technique, we show that effective results can be obtained on a large range of image editing tasks without large classifier-free guidance in inversion. Furthermore, when compared with other diffusion inversion based works, our proposed process is shown to be more robust for fast image editing in the 10 and 20 diffusion steps' regimes.

Title: Text-driven Editing of 3D Scenes without Retraining. (arXiv:2309.04917v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.04917
• Code URL: null
• Copy Paste: [[2309.04917]] Text-driven Editing of 3D Scenes without Retraining(http://arxiv.org/abs/2309.04917)
• Summary:

  Numerous diffusion models have recently been applied to image synthesis and editing. However, editing 3D scenes is still in its early stages. It poses various challenges, such as the requirement to design specific methods for different editing types, retraining new models for various 3D scenes, and the absence of convenient human interaction during editing. To tackle these issues, we introduce a text-driven editing method, termed DN2N, which allows for the direct acquisition of a NeRF model with universal editing capabilities, eliminating the requirement for retraining. Our method employs off-the-shelf text-based editing models of 2D images to modify the 3D scene images, followed by a filtering process to discard poorly edited images that disrupt 3D consistency. We then consider the remaining inconsistency as a problem of removing noise perturbation, which can be solved by generating training data with similar perturbation characteristics for training. We further propose cross-view regularization terms to help the generalized NeRF model mitigate these perturbations. Our text-driven method allows users to edit a 3D scene with their desired description, which is more friendly, intuitive, and practical than prior works. Empirical results show that our method achieves multiple editing types, including but not limited to appearance editing, weather transition, material changing, and style transfer. Most importantly, our method generalizes well with editing abilities shared among a set of model parameters without requiring a customized editing model for some specific scenes, thus inferring novel views with editing effects directly from user input. The project website is available at this http URL

Title: Prefix-diffusion: A Lightweight Diffusion Model for Diverse Image Captioning. (arXiv:2309.04965v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.04965
• Code URL: null
• Copy Paste: [[2309.04965]] Prefix-diffusion: A Lightweight Diffusion Model for Diverse Image Captioning(http://arxiv.org/abs/2309.04965)
• Summary:

  While impressive performance has been achieved in image captioning, the limited diversity of the generated captions and the large parameter scale remain major barriers to the real-word application of these systems. In this work, we propose a lightweight image captioning network in combination with continuous diffusion, called Prefix-diffusion. To achieve diversity, we design an efficient method that injects prefix image embeddings into the denoising process of the diffusion model. In order to reduce trainable parameters, we employ a pre-trained model to extract image features and further design an extra mapping network. Prefix-diffusion is able to generate diverse captions with relatively less parameters, while maintaining the fluency and relevance of the captions benefiting from the generative capabilities of the diffusion model. Our work paves the way for scaling up diffusion models for image captioning, and achieves promising performance compared with recent approaches.

Title: SA-Solver: Stochastic Adams Solver for Fast Sampling of Diffusion Models. (arXiv:2309.05019v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.05019
• Code URL: null
• Copy Paste: [[2309.05019]] SA-Solver: Stochastic Adams Solver for Fast Sampling of Diffusion Models(http://arxiv.org/abs/2309.05019)
• Summary:

  Diffusion Probabilistic Models (DPMs) have achieved considerable success in generation tasks. As sampling from DPMs is equivalent to solving diffusion SDE or ODE which is time-consuming, numerous fast sampling methods built upon improved differential equation solvers are proposed. The majority of such techniques consider solving the diffusion ODE due to its superior efficiency. However, stochastic sampling could offer additional advantages in generating diverse and high-quality data. In this work, we engage in a comprehensive analysis of stochastic sampling from two aspects: variance-controlled diffusion SDE and linear multi-step SDE solver. Based on our analysis, we propose SA-Solver, which is an improved efficient stochastic Adams method for solving diffusion SDE to generate data with high quality. Our experiments show that SA-Solver achieves: 1) improved or comparable performance compared with the existing state-of-the-art sampling methods for few-step sampling; 2) SOTA FID scores on substantial benchmark datasets under a suitable number of function evaluations (NFEs).

noise learning

data-free

transformer

Title: Mask2Anomaly: Mask Transformer for Universal Open-set Segmentation. (arXiv:2309.04573v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.04573
• Code URL: null
• Copy Paste: [[2309.04573]] Mask2Anomaly: Mask Transformer for Universal Open-set Segmentation(http://arxiv.org/abs/2309.04573)
• Summary:

  Segmenting unknown or anomalous object instances is a critical task in autonomous driving applications, and it is approached traditionally as a per-pixel classification problem. However, reasoning individually about each pixel without considering their contextual semantics results in high uncertainty around the objects' boundaries and numerous false positives. We propose a paradigm change by shifting from a per-pixel classification to a mask classification. Our mask-based method, Mask2Anomaly, demonstrates the feasibility of integrating a mask-classification architecture to jointly address anomaly segmentation, open-set semantic segmentation, and open-set panoptic segmentation. Mask2Anomaly includes several technical novelties that are designed to improve the detection of anomalies/unknown objects: i) a global masked attention module to focus individually on the foreground and background regions; ii) a mask contrastive learning that maximizes the margin between an anomaly and known classes; iii) a mask refinement solution to reduce false positives; and iv) a novel approach to mine unknown instances based on the mask-architecture properties. By comprehensive qualitative and qualitative evaluation, we show Mask2Anomaly achieves new state-of-the-art results across the benchmarks of anomaly segmentation, open-set semantic segmentation, and open-set panoptic segmentation.

Title: Unified Language-Vision Pretraining with Dynamic Discrete Visual Tokenization. (arXiv:2309.04669v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.04669
• Code URL: null
• Copy Paste: [[2309.04669]] Unified Language-Vision Pretraining with Dynamic Discrete Visual Tokenization(http://arxiv.org/abs/2309.04669)
• Summary:

  Recently, the remarkable advance of the Large Language Model (LLM) has inspired researchers to transfer its extraordinary reasoning capability to data across several modalities. The prevailing approaches primarily regard visual input as the prompt and focus exclusively on optimizing the text generation process conditioned upon vision content by a frozen LLM. Such an inequitable treatment of vision and language heavily constrains the model's potential. In this paper, we break through this limitation by representing both vision and language in a unified representation. To this end, we craft a visual tokenizer that translates the non-linguistic image into a sequence of discrete tokens like a foreign language that LLM can read. The resulting visual tokens encompass high-level semantics worthy of a word and also support dynamic sequence length varying from the image content. Coped with this visual tokenizer, the presented foundation model called LaVIT (Language-VIsion Transformer) can handle both image and text indiscriminately under a unified generative learning paradigm. Pre-trained on the web-scale image-text corpus, LaVIT is empowered with impressive multi-modal comprehension capability. The extensive experiments showcase that it outperforms existing models by a large margin on downstream tasks. Our code and models will be available at https://github.com/jy0205/LaVIT.

Title: Deep Video Restoration for Under-Display Camera. (arXiv:2309.04752v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.04752
• Code URL: null
• Copy Paste: [[2309.04752]] Deep Video Restoration for Under-Display Camera(http://arxiv.org/abs/2309.04752)
• Summary:

  Images or videos captured by the Under-Display Camera (UDC) suffer from severe degradation, such as saturation degeneration and color shift. While restoration for UDC has been a critical task, existing works of UDC restoration focus only on images. UDC video restoration (UDC-VR) has not been explored in the community. In this work, we first propose a GAN-based generation pipeline to simulate the realistic UDC degradation process. With the pipeline, we build the first large-scale UDC video restoration dataset called PexelsUDC, which includes two subsets named PexelsUDC-T and PexelsUDC-P corresponding to different displays for UDC. Using the proposed dataset, we conduct extensive benchmark studies on existing video restoration methods and observe their limitations on the UDC-VR task. To this end, we propose a novel transformer-based baseline method that adaptively enhances degraded videos. The key components of the method are a spatial branch with local-aware transformers, a temporal branch embedded temporal transformers, and a spatial-temporal fusion module. These components drive the model to fully exploit spatial and temporal information for UDC-VR. Extensive experiments show that our method achieves state-of-the-art performance on PexelsUDC. The benchmark and the baseline method are expected to promote the progress of UDC-VR in the community, which will be made public.

Title: Self-Supervised Transformer with Domain Adaptive Reconstruction for General Face Forgery Video Detection. (arXiv:2309.04795v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.04795
• Code URL: null
• Copy Paste: [[2309.04795]] Self-Supervised Transformer with Domain Adaptive Reconstruction for General Face Forgery Video Detection(http://arxiv.org/abs/2309.04795)
• Summary:

  Face forgery videos have caused severe social public concern, and various detectors have been proposed recently. However, most of them are trained in a supervised manner with limited generalization when detecting videos from different forgery methods or real source videos. To tackle this issue, we explore to take full advantage of the difference between real and forgery videos by only exploring the common representation of real face videos. In this paper, a Self-supervised Transformer cooperating with Contrastive and Reconstruction learning (CoReST) is proposed, which is first pre-trained only on real face videos in a self-supervised manner, and then fine-tuned a linear head on specific face forgery video datasets. Two specific auxiliary tasks incorporated contrastive and reconstruction learning are designed to enhance the representation learning. Furthermore, a Domain Adaptive Reconstruction (DAR) module is introduced to bridge the gap between different forgery domains by reconstructing on unlabeled target videos when fine-tuning. Extensive experiments on public datasets demonstrate that our proposed method performs even better than the state-of-the-art supervised competitors with impressive generalization.

Title: Few-Shot Medical Image Segmentation via a Region-enhanced Prototypical Transformer. (arXiv:2309.04825v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.04825
• Code URL: https://github.com/yazhouzhu19/rpt
• Copy Paste: [[2309.04825]] Few-Shot Medical Image Segmentation via a Region-enhanced Prototypical Transformer(http://arxiv.org/abs/2309.04825)
• Summary:

  Automated segmentation of large volumes of medical images is often plagued by the limited availability of fully annotated data and the diversity of organ surface properties resulting from the use of different acquisition protocols for different patients. In this paper, we introduce a more promising few-shot learning-based method named Region-enhanced Prototypical Transformer (RPT) to mitigate the effects of large intra-class diversity/bias. First, a subdivision strategy is introduced to produce a collection of regional prototypes from the foreground of the support prototype. Second, a self-selection mechanism is proposed to incorporate into the Bias-alleviated Transformer (BaT) block to suppress or remove interferences present in the query prototype and regional support prototypes. By stacking BaT blocks, the proposed RPT can iteratively optimize the generated regional prototypes and finally produce rectified and more accurate global prototypes for Few-Shot Medical Image Segmentation (FSMS). Extensive experiments are conducted on three publicly available medical image datasets, and the obtained results show consistent improvements compared to state-of-the-art FSMS methods. The source code is available at: https://github.com/YazhouZhu19/RPT.

Title: How to Evaluate Semantic Communications for Images with ViTScore Metric?. (arXiv:2309.04891v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.04891
• Code URL: null
• Copy Paste: [[2309.04891]] How to Evaluate Semantic Communications for Images with ViTScore Metric?(http://arxiv.org/abs/2309.04891)
• Summary:

  Semantic communications (SC) have been expected to be a new paradigm shifting to catalyze the next generation communication, whose main concerns shift from accurate bit transmission to effective semantic information exchange in communications. However, the previous and widely-used metrics for images are not applicable to evaluate the image semantic similarity in SC. Classical metrics to measure the similarity between two images usually rely on the pixel level or the structural level, such as the PSNR and the MS-SSIM. Straightforwardly using some tailored metrics based on deep-learning methods in CV community, such as the LPIPS, is infeasible for SC. To tackle this, inspired by BERTScore in NLP community, we propose a novel metric for evaluating image semantic similarity, named Vision Transformer Score (ViTScore). We prove theoretically that ViTScore has 3 important properties, including symmetry, boundedness, and normalization, which make ViTScore convenient and intuitive for image measurement. To evaluate the performance of ViTScore, we compare ViTScore with 3 typical metrics (PSNR, MS-SSIM, and LPIPS) through 5 classes of experiments. Experimental results demonstrate that ViTScore can better evaluate the image semantic similarity than the other 3 typical metrics, which indicates that ViTScore is an effective performance metric when deployed in SC scenarios.

Title: Transformers in Small Object Detection: A Benchmark and Survey of State-of-the-Art. (arXiv:2309.04902v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.04902
• Code URL: https://github.com/arekavandi/transformer-sod
• Copy Paste: [[2309.04902]] Transformers in Small Object Detection: A Benchmark and Survey of State-of-the-Art(http://arxiv.org/abs/2309.04902)
• Summary:

  Transformers have rapidly gained popularity in computer vision, especially in the field of object recognition and detection. Upon examining the outcomes of state-of-the-art object detection methods, we noticed that transformers consistently outperformed well-established CNN-based detectors in almost every video or image dataset. While transformer-based approaches remain at the forefront of small object detection (SOD) techniques, this paper aims to explore the performance benefits offered by such extensive networks and identify potential reasons for their SOD superiority. Small objects have been identified as one of the most challenging object types in detection frameworks due to their low visibility. We aim to investigate potential strategies that could enhance transformers' performance in SOD. This survey presents a taxonomy of over 60 research studies on developed transformers for the task of SOD, spanning the years 2020 to 2023. These studies encompass a variety of detection applications, including small object detection in generic images, aerial images, medical images, active millimeter images, underwater images, and videos. We also compile and present a list of 12 large-scale datasets suitable for SOD that were overlooked in previous studies and compare the performance of the reviewed studies using popular metrics such as mean Average Precision (mAP), Frames Per Second (FPS), number of parameters, and more. Researchers can keep track of newer studies on our web page, which is available at \url{https://github.com/arekavandi/Transformer-SOD}.

Title: DeViT: Decomposing Vision Transformers for Collaborative Inference in Edge Devices. (arXiv:2309.05015v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.05015
• Code URL: null
• Copy Paste: [[2309.05015]] DeViT: Decomposing Vision Transformers for Collaborative Inference in Edge Devices(http://arxiv.org/abs/2309.05015)
• Summary:

  Recent years have witnessed the great success of vision transformer (ViT), which has achieved state-of-the-art performance on multiple computer vision benchmarks. However, ViT models suffer from vast amounts of parameters and high computation cost, leading to difficult deployment on resource-constrained edge devices. Existing solutions mostly compress ViT models to a compact model but still cannot achieve real-time inference. To tackle this issue, we propose to explore the divisibility of transformer structure, and decompose the large ViT into multiple small models for collaborative inference at edge devices. Our objective is to achieve fast and energy-efficient collaborative inference while maintaining comparable accuracy compared with large ViTs. To this end, we first propose a collaborative inference framework termed DeViT to facilitate edge deployment by decomposing large ViTs. Subsequently, we design a decomposition-and-ensemble algorithm based on knowledge distillation, termed DEKD, to fuse multiple small decomposed models while dramatically reducing communication overheads, and handle heterogeneous models by developing a feature matching module to promote the imitations of decomposed models from the large ViT. Extensive experiments for three representative ViT backbones on four widely-used datasets demonstrate our method achieves efficient collaborative inference for ViTs and outperforms existing lightweight ViTs, striking a good trade-off between efficiency and accuracy. For example, our DeViTs improves end-to-end latency by 2.89$\times$ with only 1.65% accuracy sacrifice using CIFAR-100 compared to the large ViT, ViT-L/16, on the GPU server. DeDeiTs surpasses the recent efficient ViT, MobileViT-S, by 3.54% in accuracy on ImageNet-1K, while running 1.72$\times$ faster and requiring 55.28% lower energy consumption on the edge device.

Title: Unified Contrastive Fusion Transformer for Multimodal Human Action Recognition. (arXiv:2309.05032v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.05032
• Code URL: null
• Copy Paste: [[2309.05032]] Unified Contrastive Fusion Transformer for Multimodal Human Action Recognition(http://arxiv.org/abs/2309.05032)
• Summary:

  Various types of sensors have been considered to develop human action recognition (HAR) models. Robust HAR performance can be achieved by fusing multimodal data acquired by different sensors. In this paper, we introduce a new multimodal fusion architecture, referred to as Unified Contrastive Fusion Transformer (UCFFormer) designed to integrate data with diverse distributions to enhance HAR performance. Based on the embedding features extracted from each modality, UCFFormer employs the Unified Transformer to capture the inter-dependency among embeddings in both time and modality domains. We present the Factorized Time-Modality Attention to perform self-attention efficiently for the Unified Transformer. UCFFormer also incorporates contrastive learning to reduce the discrepancy in feature distributions across various modalities, thus generating semantically aligned features for information fusion. Performance evaluation conducted on two popular datasets, UTD-MHAD and NTU RGB+D, demonstrates that UCFFormer achieves state-of-the-art performance, outperforming competing methods by considerable margins.

generative

Title: Style Generation: Image Synthesis based on Coarsely Matched Texts. (arXiv:2309.04608v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.04608
• Code URL: null
• Copy Paste: [[2309.04608]] Style Generation: Image Synthesis based on Coarsely Matched Texts(http://arxiv.org/abs/2309.04608)
• Summary:

  Previous text-to-image synthesis algorithms typically use explicit textual instructions to generate/manipulate images accurately, but they have difficulty adapting to guidance in the form of coarsely matched texts. In this work, we attempt to stylize an input image using such coarsely matched text as guidance. To tackle this new problem, we introduce a novel task called text-based style generation and propose a two-stage generative adversarial network: the first stage generates the overall image style with a sentence feature, and the second stage refines the generated style with a synthetic feature, which is produced by a multi-modality style synthesis module. We re-filter one existing dataset and collect a new dataset for the task. Extensive experiments and ablation studies are conducted to validate our framework. The practical potential of our work is demonstrated by various applications such as text-image alignment and story visualization. Our datasets are published at https://www.kaggle.com/datasets/mengyaocui/style-generation.

Title: VeRi3D: Generative Vertex-based Radiance Fields for 3D Controllable Human Image Synthesis. (arXiv:2309.04800v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.04800
• Code URL: null
• Copy Paste: [[2309.04800]] VeRi3D: Generative Vertex-based Radiance Fields for 3D Controllable Human Image Synthesis(http://arxiv.org/abs/2309.04800)
• Summary:

  Unsupervised learning of 3D-aware generative adversarial networks has lately made much progress. Some recent work demonstrates promising results of learning human generative models using neural articulated radiance fields, yet their generalization ability and controllability lag behind parametric human models, i.e., they do not perform well when generalizing to novel pose/shape and are not part controllable. To solve these problems, we propose VeRi3D, a generative human vertex-based radiance field parameterized by vertices of the parametric human template, SMPL. We map each 3D point to the local coordinate system defined on its neighboring vertices, and use the corresponding vertex feature and local coordinates for mapping it to color and density values. We demonstrate that our simple approach allows for generating photorealistic human images with free control over camera pose, human pose, shape, as well as enabling part-level editing.

Title: TCGAN: Convolutional Generative Adversarial Network for Time Series Classification and Clustering. (arXiv:2309.04732v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.04732
• Code URL: https://bitbucket.org/lynn1/tcgan
• Copy Paste: [[2309.04732]] TCGAN: Convolutional Generative Adversarial Network for Time Series Classification and Clustering(http://arxiv.org/abs/2309.04732)
• Summary:

  Recent works have demonstrated the superiority of supervised Convolutional Neural Networks (CNNs) in learning hierarchical representations from time series data for successful classification. These methods require sufficiently large labeled data for stable learning, however acquiring high-quality labeled time series data can be costly and potentially infeasible. Generative Adversarial Networks (GANs) have achieved great success in enhancing unsupervised and semi-supervised learning. Nonetheless, to our best knowledge, it remains unclear how effectively GANs can serve as a general-purpose solution to learn representations for time series recognition, i.e., classification and clustering. The above considerations inspire us to introduce a Time-series Convolutional GAN (TCGAN). TCGAN learns by playing an adversarial game between two one-dimensional CNNs (i.e., a generator and a discriminator) in the absence of label information. Parts of the trained TCGAN are then reused to construct a representation encoder to empower linear recognition methods. We conducted comprehensive experiments on synthetic and real-world datasets. The results demonstrate that TCGAN is faster and more accurate than existing time-series GANs. The learned representations enable simple classification and clustering methods to achieve superior and stable performance. Furthermore, TCGAN retains high efficacy in scenarios with few-labeled and imbalanced-labeled data. Our work provides a promising path to effectively utilize abundant unlabeled time series data.

Title: AmbientFlow: Invertible generative models from incomplete, noisy measurements. (arXiv:2309.04856v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.04856
• Code URL: null
• Copy Paste: [[2309.04856]] AmbientFlow: Invertible generative models from incomplete, noisy measurements(http://arxiv.org/abs/2309.04856)
• Summary:

  Generative models have gained popularity for their potential applications in imaging science, such as image reconstruction, posterior sampling and data sharing. Flow-based generative models are particularly attractive due to their ability to tractably provide exact density estimates along with fast, inexpensive and diverse samples. Training such models, however, requires a large, high quality dataset of objects. In applications such as computed imaging, it is often difficult to acquire such data due to requirements such as long acquisition time or high radiation dose, while acquiring noisy or partially observed measurements of these objects is more feasible. In this work, we propose AmbientFlow, a framework for learning flow-based generative models directly from noisy and incomplete data. Using variational Bayesian methods, a novel framework for establishing flow-based generative models from noisy, incomplete data is proposed. Extensive numerical studies demonstrate the effectiveness of AmbientFlow in correctly learning the object distribution. The utility of AmbientFlow in a downstream inference task of image reconstruction is demonstrated.

large language model

Title: Retrieving Evidence from EHRs with LLMs: Possibilities and Challenges. (arXiv:2309.04550v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.04550
• Code URL: null
• Copy Paste: [[2309.04550]] Retrieving Evidence from EHRs with LLMs: Possibilities and Challenges(http://arxiv.org/abs/2309.04550)
• Summary:

  Unstructured Electronic Health Record (EHR) data often contains critical information complementary to imaging data that would inform radiologists' diagnoses. However, time constraints and the large volume of notes frequently associated with individual patients renders manual perusal of such data to identify relevant evidence infeasible in practice. Modern Large Language Models (LLMs) provide a flexible means of interacting with unstructured EHR data, and may provide a mechanism to efficiently retrieve and summarize unstructured evidence relevant to a given query. In this work, we propose and evaluate an LLM (Flan-T5 XXL) for this purpose. Specifically, in a zero-shot setting we task the LLM to infer whether a patient has or is at risk of a particular condition; if so, we prompt the model to summarize the supporting evidence. Enlisting radiologists for manual evaluation, we find that this LLM-based approach provides outputs consistently preferred to a standard information retrieval baseline, but we also highlight the key outstanding challenge: LLMs are prone to hallucinating evidence. However, we provide results indicating that model confidence in outputs might indicate when LLMs are hallucinating, potentially providing a means to address this.

Title: When Less is More: Investigating Data Pruning for Pretraining LLMs at Scale. (arXiv:2309.04564v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.04564
• Code URL: null
• Copy Paste: [[2309.04564]] When Less is More: Investigating Data Pruning for Pretraining LLMs at Scale(http://arxiv.org/abs/2309.04564)
• Summary:

  Large volumes of text data have contributed significantly to the development of large language models (LLMs) in recent years. This data is typically acquired by scraping the internet, leading to pretraining datasets comprised of noisy web text. To date, efforts to prune these datasets down to a higher quality subset have relied on hand-crafted heuristics encoded as rule-based filters. In this work, we take a wider view and explore scalable estimates of data quality that can be used to systematically measure the quality of pretraining data. We perform a rigorous comparison at scale of the simple data quality estimator of perplexity, as well as more sophisticated and computationally intensive estimates of the Error L2-Norm and memorization. These metrics are used to rank and prune pretraining corpora, and we subsequently compare LLMs trained on these pruned datasets. Surprisingly, we find that the simple technique of perplexity outperforms our more computationally expensive scoring methods. We improve over our no-pruning baseline while training on as little as 30% of the original training dataset. Our work sets the foundation for unexplored strategies in automatically curating high quality corpora and suggests the majority of pretraining data can be removed while retaining performance.

Title: Efficient Finetuning Large Language Models For Vietnamese Chatbot. (arXiv:2309.04646v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.04646
• Code URL: null
• Copy Paste: [[2309.04646]] Efficient Finetuning Large Language Models For Vietnamese Chatbot(http://arxiv.org/abs/2309.04646)
• Summary:

  Large language models (LLMs), such as GPT-4, PaLM, and LLaMa, have been shown to achieve remarkable performance across a variety of natural language tasks. Recent advancements in instruction tuning bring LLMs with ability in following user's instructions and producing human-like responses. However, the high costs associated with training and implementing LLMs pose challenges to academic research. Furthermore, the availability of pretrained LLMs and instruction-tune datasets for Vietnamese language is limited. To tackle these concerns, we leverage large-scale instruction-following datasets from open-source projects, namely Alpaca, GPT4All, and Chat-Doctor, which cover general domain and specific medical domain. To the best of our knowledge, these are the first instructional dataset for Vietnamese. Subsequently, we utilize parameter-efficient tuning through Low-Rank Adaptation (LoRA) on two open LLMs: Bloomz (Multilingual) and GPTJ-6B (Vietnamese), resulting four models: Bloomz-Chat, Bloomz-Doctor, GPTJ-Chat, GPTJ-Doctor.Finally, we assess the effectiveness of our methodology on a per-sample basis, taking into consideration the helpfulness, relevance, accuracy, level of detail in their responses. This evaluation process entails the utilization of GPT-4 as an automated scoring mechanism. Despite utilizing a low-cost setup, our method demonstrates about 20-30\% improvement over the original models in our evaluation tasks.

Title: Exploring Large Language Models for Communication Games: An Empirical Study on Werewolf. (arXiv:2309.04658v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.04658
• Code URL: null
• Copy Paste: [[2309.04658]] Exploring Large Language Models for Communication Games: An Empirical Study on Werewolf(http://arxiv.org/abs/2309.04658)
• Summary:

  Communication games, which we refer to as incomplete information games that heavily depend on natural language communication, hold significant research value in fields such as economics, social science, and artificial intelligence. In this work, we explore the problem of how to engage large language models (LLMs) in communication games, and in response, propose a tuning-free framework. Our approach keeps LLMs frozen, and relies on the retrieval and reflection on past communications and experiences for improvement. An empirical study on the representative and widely-studied communication game, ``Werewolf'', demonstrates that our framework can effectively play Werewolf game without tuning the parameters of the LLMs. More importantly, strategic behaviors begin to emerge in our experiments, suggesting that it will be a fruitful journey to engage LLMs in communication games and associated domains.

Title: FIAT: Fusing learning paradigms with Instruction-Accelerated Tuning. (arXiv:2309.04663v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.04663
• Code URL: null
• Copy Paste: [[2309.04663]] FIAT: Fusing learning paradigms with Instruction-Accelerated Tuning(http://arxiv.org/abs/2309.04663)
• Summary:

  Learning paradigms for large language models (LLMs) currently tend to fall within either in-context learning (ICL) or full fine-tuning. Each of these comes with their own trade-offs based on available data, model size, compute cost, ease-of-use, and final quality with neither solution performing well across-the-board. In this article, we first describe ICL and fine-tuning paradigms in a way that highlights their natural connections. Based on these connections, we propose a new learning paradigm called FIAT that fuses the best of these paradigms together, enabling prompt-engineered instructions and chain-of-thought reasoning with the very largest models while also using similar methods to perform parameter updates on a modestly-sized LLM with parameter-efficient tuning. We evaluate FIAT's effectiveness on a variety of multilingual tasks and observe that FIAT performs better than both ICL and fine-tuning at scales ranging from 100-10,000 training examples. We hope that FIAT provides a practical way of harnessing the full potential of LLMs without needing to make a hard choice between learning paradigms.

Title: Code-Style In-Context Learning for Knowledge-Based Question Answering. (arXiv:2309.04695v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.04695
• Code URL: null
• Copy Paste: [[2309.04695]] Code-Style In-Context Learning for Knowledge-Based Question Answering(http://arxiv.org/abs/2309.04695)
• Summary:

  Current methods for Knowledge-Based Question Answering (KBQA) usually rely on complex training techniques and model frameworks, leading to many limitations in practical applications. Recently, the emergence of In-Context Learning (ICL) capabilities in Large Language Models (LLMs) provides a simple and training-free semantic parsing paradigm for KBQA: Given a small number of questions and their labeled logical forms as demo examples, LLMs can understand the task intent and generate the logic form for a new question. However, current powerful LLMs have little exposure to logic forms during pre-training, resulting in a high format error rate. To solve this problem, we propose a code-style in-context learning method for KBQA, which converts the generation process of unfamiliar logical form into the more familiar code generation process for LLMs. Experimental results on three mainstream datasets show that our method dramatically mitigated the formatting error problem in generating logic forms while realizing a new SOTA on WebQSP, GrailQA, and GraphQ under the few-shot setting.

Title: Analysis of Disinformation and Fake News Detection Using Fine-Tuned Large Language Model. (arXiv:2309.04704v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.04704
• Code URL: null
• Copy Paste: [[2309.04704]] Analysis of Disinformation and Fake News Detection Using Fine-Tuned Large Language Model(http://arxiv.org/abs/2309.04704)
• Summary:

  The paper considers the possibility of fine-tuning Llama 2 large language model (LLM) for the disinformation analysis and fake news detection. For fine-tuning, the PEFT/LoRA based approach was used. In the study, the model was fine-tuned for the following tasks: analysing a text on revealing disinformation and propaganda narratives, fact checking, fake news detection, manipulation analytics, extracting named entities with their sentiments. The obtained results show that the fine-tuned Llama 2 model can perform a deep analysis of texts and reveal complex styles and narratives. Extracted sentiments for named entities can be considered as predictive features in supervised machine learning models.

Title: Toward Reproducing Network Research Results Using Large Language Models. (arXiv:2309.04716v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.04716
• Code URL: null
• Copy Paste: [[2309.04716]] Toward Reproducing Network Research Results Using Large Language Models(http://arxiv.org/abs/2309.04716)
• Summary:

  Reproducing research results in the networking community is important for both academia and industry. The current best practice typically resorts to three approaches: (1) looking for publicly available prototypes; (2) contacting the authors to get a private prototype; and (3) manually implementing a prototype following the description of the publication. However, most published network research does not have public prototypes and private prototypes are hard to get. As such, most reproducing efforts are spent on manual implementation based on the publications, which is both time and labor consuming and error-prone. In this paper, we boldly propose reproducing network research results using the emerging large language models (LLMs). In particular, we first prove its feasibility with a small-scale experiment, in which four students with essential networking knowledge each reproduces a different networking system published in prominent conferences and journals by prompt engineering ChatGPT. We report the experiment's observations and lessons and discuss future open research questions of this proposal. This work raises no ethical issue.

Title: EPA: Easy Prompt Augmentation on Large Language Models via Multiple Sources and Multiple Targets. (arXiv:2309.04725v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.04725
• Code URL: null
• Copy Paste: [[2309.04725]] EPA: Easy Prompt Augmentation on Large Language Models via Multiple Sources and Multiple Targets(http://arxiv.org/abs/2309.04725)
• Summary:

  Large language models (LLMs) have shown promising performance on various NLP tasks via task prompting. And their performance can be further improved by appending task demonstrations to the head of the prompt. And usually, a better performance can be achieved with more demonstrations. However, asking the users to write the demonstrations can be cumbersome. As a simple yet cost-effective workaround, this paper proposes a novel method called EPA (\textbf{E}asy \textbf{P}rompt \textbf{A}ugmentation)\footnote{While this paper considers augmenting prompts via demonstrations, we name it EPA as the name EDA is already taken by a well-known NLP method \citep{wei-zou-2019-eda}.} that effectively minimizes user efforts in writing demonstrations while improving the model performance at the same time. EPA achieves these goals by automatically augmenting the demonstrations with multiple sources/targets, where each of them paraphrases each other. This is well motivated as augmenting data via paraphrasing effectively improves neural language models. EPA thus employs paraphrasing as an augmentation method for in-context learning. Extensive experiments indicate that EPA effectively improves both NLU and NLG tasks, covering from natural language inference to machine translation in translating tens of languages.\footnote{Code and data will be released upon publication.}

Title: MMHQA-ICL: Multimodal In-context Learning for Hybrid Question Answering over Text, Tables and Images. (arXiv:2309.04790v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.04790
• Code URL: null
• Copy Paste: [[2309.04790]] MMHQA-ICL: Multimodal In-context Learning for Hybrid Question Answering over Text, Tables and Images(http://arxiv.org/abs/2309.04790)
• Summary:

  In the real world, knowledge often exists in a multimodal and heterogeneous form. Addressing the task of question answering with hybrid data types, including text, tables, and images, is a challenging task (MMHQA). Recently, with the rise of large language models (LLM), in-context learning (ICL) has become the most popular way to solve QA problems. We propose MMHQA-ICL framework for addressing this problems, which includes stronger heterogeneous data retriever and an image caption module. Most importantly, we propose a Type-specific In-context Learning Strategy for MMHQA, enabling LLMs to leverage their powerful performance in this task. We are the first to use end-to-end LLM prompting method for this task. Experimental results demonstrate that our framework outperforms all baselines and methods trained on the full dataset, achieving state-of-the-art results under the few-shot setting on the MultimodalQA dataset.

Title: FaNS: a Facet-based Narrative Similarity Metric. (arXiv:2309.04823v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.04823
• Code URL: null
• Copy Paste: [[2309.04823]] FaNS: a Facet-based Narrative Similarity Metric(http://arxiv.org/abs/2309.04823)
• Summary:

  Similar Narrative Retrieval is a crucial task since narratives are essential for explaining and understanding events, and multiple related narratives often help to create a holistic view of the event of interest. To accurately identify semantically similar narratives, this paper proposes a novel narrative similarity metric called Facet-based Narrative Similarity (FaNS), based on the classic 5W1H facets (Who, What, When, Where, Why, and How), which are extracted by leveraging the state-of-the-art Large Language Models (LLMs). Unlike existing similarity metrics that only focus on overall lexical/semantic match, FaNS provides a more granular matching along six different facets independently and then combines them. To evaluate FaNS, we created a comprehensive dataset by collecting narratives from AllSides, a third-party news portal. Experimental results demonstrate that the FaNS metric exhibits a higher correlation (37\% higher) than traditional text similarity metrics that directly measure the lexical/semantic match between narratives, demonstrating its effectiveness in comparing the finer details between a pair of narratives.

Title: Neurons in Large Language Models: Dead, N-gram, Positional. (arXiv:2309.04827v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.04827
• Code URL: null
• Copy Paste: [[2309.04827]] Neurons in Large Language Models: Dead, N-gram, Positional(http://arxiv.org/abs/2309.04827)
• Summary:

  We analyze a family of large language models in such a lightweight manner that can be done on a single GPU. Specifically, we focus on the OPT family of models ranging from 125m to 66b parameters and rely only on whether an FFN neuron is activated or not. First, we find that the early part of the network is sparse and represents many discrete features. Here, many neurons (more than 70% in some layers of the 66b model) are "dead", i.e. they never activate on a large collection of diverse data. At the same time, many of the alive neurons are reserved for discrete features and act as token and n-gram detectors. Interestingly, their corresponding FFN updates not only promote next token candidates as could be expected, but also explicitly focus on removing the information about triggering them tokens, i.e., current input. To the best of our knowledge, this is the first example of mechanisms specialized at removing (rather than adding) information from the residual stream. With scale, models become more sparse in a sense that they have more dead neurons and token detectors. Finally, some neurons are positional: them being activated or not depends largely (or solely) on position and less so (or not at all) on textual data. We find that smaller models have sets of neurons acting as position range indicators while larger models operate in a less explicit manner.

Title: Leveraging Large Language Models for Exploiting ASR Uncertainty. (arXiv:2309.04842v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.04842
• Code URL: null
• Copy Paste: [[2309.04842]] Leveraging Large Language Models for Exploiting ASR Uncertainty(http://arxiv.org/abs/2309.04842)
• Summary:

  While large language models excel in a variety of natural language processing (NLP) tasks, to perform well on spoken language understanding (SLU) tasks, they must either rely on off-the-shelf automatic speech recognition (ASR) systems for transcription, or be equipped with an in-built speech modality. This work focuses on the former scenario, where LLM's accuracy on SLU tasks is constrained by the accuracy of a fixed ASR system on the spoken input. Specifically, we tackle speech-intent classification task, where a high word-error-rate can limit the LLM's ability to understand the spoken intent. Instead of chasing a high accuracy by designing complex or specialized architectures regardless of deployment costs, we seek to answer how far we can go without substantially changing the underlying ASR and LLM, which can potentially be shared by multiple unrelated tasks. To this end, we propose prompting the LLM with an n-best list of ASR hypotheses instead of only the error-prone 1-best hypothesis. We explore prompt-engineering to explain the concept of n-best lists to the LLM; followed by the finetuning of Low-Rank Adapters on the downstream tasks. Our approach using n-best lists proves to be effective on a device-directed speech detection task as well as on a keyword spotting task, where systems using n-best list prompts outperform those using 1-best ASR hypothesis; thus paving the way for an efficient method to exploit ASR uncertainty via LLMs for speech-based applications.

Title: Chat2Brain: A Method for Mapping Open-Ended Semantic Queries to Brain Activation Maps. (arXiv:2309.05021v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.05021
• Code URL: null
• Copy Paste: [[2309.05021]] Chat2Brain: A Method for Mapping Open-Ended Semantic Queries to Brain Activation Maps(http://arxiv.org/abs/2309.05021)
• Summary:

  Over decades, neuroscience has accumulated a wealth of research results in the text modality that can be used to explore cognitive processes. Meta-analysis is a typical method that successfully establishes a link from text queries to brain activation maps using these research results, but it still relies on an ideal query environment. In practical applications, text queries used for meta-analyses may encounter issues such as semantic redundancy and ambiguity, resulting in an inaccurate mapping to brain images. On the other hand, large language models (LLMs) like ChatGPT have shown great potential in tasks such as context understanding and reasoning, displaying a high degree of consistency with human natural language. Hence, LLMs could improve the connection between text modality and neuroscience, resolving existing challenges of meta-analyses. In this study, we propose a method called Chat2Brain that combines LLMs to basic text-2-image model, known as Text2Brain, to map open-ended semantic queries to brain activation maps in data-scarce and complex query environments. By utilizing the understanding and reasoning capabilities of LLMs, the performance of the mapping model is optimized by transferring text queries to semantic queries. We demonstrate that Chat2Brain can synthesize anatomically plausible neural activation patterns for more complex tasks of text queries.

Title: Unleashing the Power of Graph Learning through LLM-based Autonomous Agents. (arXiv:2309.04565v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.04565
• Code URL: null
• Copy Paste: [[2309.04565]] Unleashing the Power of Graph Learning through LLM-based Autonomous Agents(http://arxiv.org/abs/2309.04565)
• Summary:

  Graph structured data are widely existed and applied in the real-world applications, while it is a challenge to handling these diverse data and learning tasks on graph in an efficient manner. When facing the complicated graph learning tasks, experts have designed diverse Graph Neural Networks (GNNs) in recent years. They have also implemented AutoML in Graph, also known as AutoGraph, to automatically generate data-specific solutions. Despite their success, they encounter limitations in (1) managing diverse learning tasks at various levels, (2) dealing with different procedures in graph learning beyond architecture design, and (3) the huge requirements on the prior knowledge when using AutoGraph. In this paper, we propose to use Large Language Models (LLMs) as autonomous agents to simplify the learning process on diverse real-world graphs. Specifically, in response to a user request which may contain varying data and learning targets at the node, edge, or graph levels, the complex graph learning task is decomposed into three components following the agent planning, namely, detecting the learning intent, configuring solutions based on AutoGraph, and generating a response. The AutoGraph agents manage crucial procedures in automated graph learning, including data-processing, AutoML configuration, searching architectures, and hyper-parameter fine-tuning. With these agents, those components are processed by decomposing and completing step by step, thereby generating a solution for the given data automatically, regardless of the learning task on node or graph. The proposed method is dubbed Auto$^2$Graph, and the comparable performance on different datasets and learning tasks. Its effectiveness is demonstrated by its comparable performance on different datasets and learning tasks, as well as the human-like decisions made by the agents.

segmentation

Title: Three Ways to Improve Verbo-visual Fusion for Dense 3D Visual Grounding. (arXiv:2309.04561v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.04561
• Code URL: null
• Copy Paste: [[2309.04561]] Three Ways to Improve Verbo-visual Fusion for Dense 3D Visual Grounding(http://arxiv.org/abs/2309.04561)
• Summary:

  3D visual grounding is the task of localizing the object in a 3D scene which is referred by a description in natural language. With a wide range of applications ranging from autonomous indoor robotics to AR/VR, the task has recently risen in popularity. A common formulation to tackle 3D visual grounding is grounding-by-detection, where localization is done via bounding boxes. However, for real-life applications that require physical interactions, a bounding box insufficiently describes the geometry of an object. We therefore tackle the problem of dense 3D visual grounding, i.e. referral-based 3D instance segmentation. We propose a dense 3D grounding network ConcreteNet, featuring three novel stand-alone modules which aim to improve grounding performance for challenging repetitive instances, i.e. instances with distractors of the same semantic class. First, we introduce a bottom-up attentive fusion module that aims to disambiguate inter-instance relational cues, next we construct a contrastive training scheme to induce separation in the latent space, and finally we resolve view-dependent utterances via a learned global camera token. ConcreteNet ranks 1st on the challenging ScanRefer online benchmark by a considerable +9.43% accuracy at 50% IoU and has won the ICCV 3rd Workshop on Language for 3D Scenes "3D Object Localization" challenge.

Title: Visual Material Characteristics Learning for Circular Healthcare. (arXiv:2309.04763v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.04763
• Code URL: null
• Copy Paste: [[2309.04763]] Visual Material Characteristics Learning for Circular Healthcare(http://arxiv.org/abs/2309.04763)
• Summary:

  The linear take-make-dispose paradigm at the foundations of our traditional economy is proving to be unsustainable due to waste pollution and material supply uncertainties. Hence, increasing the circularity of material flows is necessary. In this paper, we make a step towards circular healthcare by developing several vision systems targeting three main circular economy tasks: resources mapping and quantification, waste sorting, and disassembly. The performance of our systems demonstrates that representation-learning vision can improve the recovery chain, where autonomous systems are key enablers due to the contamination risks. We also published two fully-annotated datasets for image segmentation and for key-point tracking in disassembly operations of inhalers and glucose meters. The datasets and source code are publicly available.

Title: SortedAP: Rethinking evaluation metrics for instance segmentation. (arXiv:2309.04887v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.04887
• Code URL: null
• Copy Paste: [[2309.04887]] SortedAP: Rethinking evaluation metrics for instance segmentation(http://arxiv.org/abs/2309.04887)
• Summary:

  Designing metrics for evaluating instance segmentation revolves around comprehensively considering object detection and segmentation accuracy. However, other important properties, such as sensitivity, continuity, and equality, are overlooked in the current study. In this paper, we reveal that most existing metrics have a limited resolution of segmentation quality. They are only conditionally sensitive to the change of masks or false predictions. For certain metrics, the score can change drastically in a narrow range which could provide a misleading indication of the quality gap between results. Therefore, we propose a new metric called sortedAP, which strictly decreases with both object- and pixel-level imperfections and has an uninterrupted penalization scale over the entire domain. We provide the evaluation toolkit and experiment code at https://www.github.com/looooongChen/sortedAP.

Title: Semi-supervised Instance Segmentation with a Learned Shape Prior. (arXiv:2309.04888v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.04888
• Code URL: https://github.com/looooongChen/shape_prior_seg
• Copy Paste: [[2309.04888]] Semi-supervised Instance Segmentation with a Learned Shape Prior(http://arxiv.org/abs/2309.04888)
• Summary:

  To date, most instance segmentation approaches are based on supervised learning that requires a considerable amount of annotated object contours as training ground truth. Here, we propose a framework that searches for the target object based on a shape prior. The shape prior model is learned with a variational autoencoder that requires only a very limited amount of training data: In our experiments, a few dozens of object shape patches from the target dataset, as well as purely synthetic shapes, were sufficient to achieve results en par with supervised methods with full access to training data on two out of three cell segmentation datasets. Our method with a synthetic shape prior was superior to pre-trained supervised models with access to limited domain-specific training data on all three datasets. Since the learning of prior models requires shape patches, whether real or synthetic data, we call this framework semi-supervised learning.

Title: MFPNet: Multi-scale Feature Propagation Nwtwork For Lightweight Semantic Segmentation. (arXiv:2309.04914v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.04914
• Code URL: null
• Copy Paste: [[2309.04914]] MFPNet: Multi-scale Feature Propagation Nwtwork For Lightweight Semantic Segmentation(http://arxiv.org/abs/2309.04914)
• Summary:

  In contrast to the abundant research focusing on large-scale models, the progress in lightweight semantic segmentation appears to be advancing at a comparatively slower pace. However, existing compact methods often suffer from limited feature representation capability due to the shallowness of their networks. In this paper, we propose a novel lightweight segmentation architecture, called Multi-scale Feature Propagation Network (MFPNet), to address the dilemma. Specifically, we design a robust Encoder-Decoder structure featuring symmetrical residual blocks that consist of flexible bottleneck residual modules (BRMs) to explore deep and rich muti-scale semantic context. Furthermore, taking benefit from their capacity to model latent long-range contextual relationships, we leverage Graph Convolutional Networks (GCNs) to facilitate multi-scale feature propagation between the BRM blocks. When evaluated on benchmark datasets, our proposed approach shows superior segmentation results.