secure

security

Title: A Survey of Network Requirements for Enabling Effective Cyber Deception. (arXiv:2309.00184v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.00184
• Code URL: null
• Copy Paste: [[2309.00184]] A Survey of Network Requirements for Enabling Effective Cyber Deception(http://arxiv.org/abs/2309.00184)
• Summary:

  In the evolving landscape of cybersecurity, the utilization of cyber deception has gained prominence as a proactive defense strategy against sophisticated attacks. This paper presents a comprehensive survey that investigates the crucial network requirements essential for the successful implementation of effective cyber deception techniques. With a focus on diverse network architectures and topologies, we delve into the intricate relationship between network characteristics and the deployment of deception mechanisms. This survey provides an in-depth analysis of prevailing cyber deception frameworks, highlighting their strengths and limitations in meeting the requirements for optimal efficacy. By synthesizing insights from both theoretical and practical perspectives, we contribute to a comprehensive understanding of the network prerequisites crucial for enabling robust and adaptable cyber deception strategies.

Title: Account Abstraction, Analysed. (arXiv:2309.00448v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.00448
• Code URL: null
• Copy Paste: [[2309.00448]] Account Abstraction, Analysed(http://arxiv.org/abs/2309.00448)
• Summary:

  Ethereum recently unveiled its upcoming roadmap's \textit{Splurge} phase, highlighting the integration of EIP-\hlhref{https://eips.ethereum.org/EIPS/eip-3074}{4337} as a foundational standard for account abstraction (AA). AA aims to enhance user accessibility and facilitate the expansion of functionalities. Anticipatedly, the deployment of AA is poised to attract a broad spectrum of new users and ignite further innovation in DApps. In this paper, we elucidate the underlying operating mechanisms of this new concept, as well as provide a review of concurrent advancements in accounts, wallets, and standards related to its development. We step further by conducting a preliminary security evaluation to qualitatively assess the extent of security enhancements achieved through AA updates.

privacy

Title: Large-Scale Public Data Improves Differentially Private Image Generation Quality. (arXiv:2309.00008v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.00008
• Code URL: null
• Copy Paste: [[2309.00008]] Large-Scale Public Data Improves Differentially Private Image Generation Quality(http://arxiv.org/abs/2309.00008)
• Summary:

  Public data has been frequently used to improve the privacy-accuracy trade-off of differentially private machine learning, but prior work largely assumes that this data come from the same distribution as the private. In this work, we look at how to use generic large-scale public data to improve the quality of differentially private image generation in Generative Adversarial Networks (GANs), and provide an improved method that uses public data effectively. Our method works under the assumption that the support of the public data distribution contains the support of the private; an example of this is when the public data come from a general-purpose internet-scale image source, while the private data consist of images of a specific type. Detailed evaluations show that our method achieves SOTA in terms of FID score and other metrics compared with existing methods that use public data, and can generate high-quality, photo-realistic images in a differentially private manner.

Title: Trust your Good Friends: Source-free Domain Adaptation by Reciprocal Neighborhood Clustering. (arXiv:2309.00528v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.00528
• Code URL: null
• Copy Paste: [[2309.00528]] Trust your Good Friends: Source-free Domain Adaptation by Reciprocal Neighborhood Clustering(http://arxiv.org/abs/2309.00528)
• Summary:

  Domain adaptation (DA) aims to alleviate the domain shift between source domain and target domain. Most DA methods require access to the source data, but often that is not possible (e.g. due to data privacy or intellectual property). In this paper, we address the challenging source-free domain adaptation (SFDA) problem, where the source pretrained model is adapted to the target domain in the absence of source data. Our method is based on the observation that target data, which might not align with the source domain classifier, still forms clear clusters. We capture this intrinsic structure by defining local affinity of the target data, and encourage label consistency among data with high local affinity. We observe that higher affinity should be assigned to reciprocal neighbors. To aggregate information with more context, we consider expanded neighborhoods with small affinity values. Furthermore, we consider the density around each target sample, which can alleviate the negative impact of potential outliers. In the experimental results we verify that the inherent structure of the target features is an important source of information for domain adaptation. We demonstrate that this local structure can be efficiently captured by considering the local neighbors, the reciprocal neighbors, and the expanded neighborhood. Finally, we achieve state-of-the-art performance on several 2D image and 3D point cloud recognition datasets.

Title: MIMOCrypt: Multi-User Privacy-Preserving Wi-Fi Sensing via MIMO Encryption. (arXiv:2309.00250v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.00250
• Code URL: null
• Copy Paste: [[2309.00250]] MIMOCrypt: Multi-User Privacy-Preserving Wi-Fi Sensing via MIMO Encryption(http://arxiv.org/abs/2309.00250)
• Summary:

  Wi-Fi signals may help realize low-cost and non-invasive human sensing, yet it can also be exploited by eavesdroppers to capture private information. Very few studies rise to handle this privacy concern so far; they either jam all sensing attempts or rely on sophisticated technologies to support only a single sensing user, rendering them impractical for multi-user scenarios. Moreover, these proposals all fail to exploit Wi-Fi's multiple-in multiple-out (MIMO) capability. To this end, we propose MIMOCrypt, a privacy-preserving Wi-Fi sensing framework to support realistic multi-user scenarios. To thwart unauthorized eavesdropping while retaining the sensing and communication capabilities for legitimate users, MIMOCrypt innovates in exploiting MIMO to physically encrypt Wi-Fi channels, treating the sensed human activities as physical plaintexts. The encryption scheme is further enhanced via an optimization framework, aiming to strike a balance among i) risk of eavesdropping, ii) sensing accuracy, and iii) communication quality, upon securely conveying decryption keys to legitimate users. We implement a prototype of MIMOCrypt on an SDR platform and perform extensive experiments to evaluate its effectiveness in common application scenarios, especially privacy-sensitive human gesture recognition.

Title: Advancing Personalized Federated Learning: Group Privacy, Fairness, and Beyond. (arXiv:2309.00416v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.00416
• Code URL: null
• Copy Paste: [[2309.00416]] Advancing Personalized Federated Learning: Group Privacy, Fairness, and Beyond(http://arxiv.org/abs/2309.00416)
• Summary:

  Federated learning (FL) is a framework for training machine learning models in a distributed and collaborative manner. During training, a set of participating clients process their data stored locally, sharing only the model updates obtained by minimizing a cost function over their local inputs. FL was proposed as a stepping-stone towards privacy-preserving machine learning, but it has been shown vulnerable to issues such as leakage of private information, lack of personalization of the model, and the possibility of having a trained model that is fairer to some groups than to others. In this paper, we address the triadic interaction among personalization, privacy guarantees, and fairness attained by models trained within the FL framework. Differential privacy and its variants have been studied and applied as cutting-edge standards for providing formal privacy guarantees. However, clients in FL often hold very diverse datasets representing heterogeneous communities, making it important to protect their sensitive information while still ensuring that the trained model upholds the aspect of fairness for the users. To attain this objective, a method is put forth that introduces group privacy assurances through the utilization of $d$-privacy (aka metric privacy). $d$-privacy represents a localized form of differential privacy that relies on a metric-oriented obfuscation approach to maintain the original data's topological distribution. This method, besides enabling personalized model training in a federated approach and providing formal privacy guarantees, possesses significantly better group fairness measured under a variety of standard metrics than a global model trained within a classical FL template. Theoretical justifications for the applicability are provided, as well as experimental validation on real-world datasets to illustrate the working of the proposed method.

Title: Privacy Attacks and Defenses for Digital Twin Migrations in Vehicular Metaverses. (arXiv:2309.00477v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.00477
• Code URL: null
• Copy Paste: [[2309.00477]] Privacy Attacks and Defenses for Digital Twin Migrations in Vehicular Metaverses(http://arxiv.org/abs/2309.00477)
• Summary:

  The gradual fusion of intelligent transportation systems with metaverse technologies is giving rise to vehicular metaverses, which blend virtual spaces with physical space. As indispensable components for vehicular metaverses, Vehicular Twins (VTs) are digital replicas of Vehicular Metaverse Users (VMUs) and facilitate customized metaverse services to VMUs. VTs are established and maintained in RoadSide Units (RSUs) with sufficient computing and storage resources. Due to the limited communication coverage of RSUs and the high mobility of VMUs, VTs need to be migrated among RSUs to ensure real-time and seamless services for VMUs. However, during VT migrations, physical-virtual synchronization and massive communications among VTs may cause identity and location privacy disclosures of VMUs and VTs. In this article, we study privacy issues and the corresponding defenses for VT migrations in vehicular metaverses. We first present four kinds of specific privacy attacks during VT migrations. Then, we propose a VMU-VT dual pseudonym scheme and a synchronous pseudonym change framework to defend against these attacks. Additionally, we evaluate average privacy entropy for pseudonym changes and optimize the number of pseudonym distribution based on inventory theory. Numerical results show that the average utility of VMUs under our proposed schemes is 33.8% higher than that under the equal distribution scheme, demonstrating the superiority of our schemes.

Title: Continual Learning From a Stream of APIs. (arXiv:2309.00023v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.00023
• Code URL: null
• Copy Paste: [[2309.00023]] Continual Learning From a Stream of APIs(http://arxiv.org/abs/2309.00023)
• Summary:

  Continual learning (CL) aims to learn new tasks without forgetting previous tasks. However, existing CL methods require a large amount of raw data, which is often unavailable due to copyright considerations and privacy risks. Instead, stakeholders usually release pre-trained machine learning models as a service (MLaaS), which users can access via APIs. This paper considers two practical-yet-novel CL settings: data-efficient CL (DECL-APIs) and data-free CL (DFCL-APIs), which achieve CL from a stream of APIs with partial or no raw data. Performing CL under these two new settings faces several challenges: unavailable full raw data, unknown model parameters, heterogeneous models of arbitrary architecture and scale, and catastrophic forgetting of previous APIs. To overcome these issues, we propose a novel data-free cooperative continual distillation learning framework that distills knowledge from a stream of APIs into a CL model by generating pseudo data, just by querying APIs. Specifically, our framework includes two cooperative generators and one CL model, forming their training as an adversarial game. We first use the CL model and the current API as fixed discriminators to train generators via a derivative-free method. Generators adversarially generate hard and diverse synthetic data to maximize the response gap between the CL model and the API. Next, we train the CL model by minimizing the gap between the responses of the CL model and the black-box API on synthetic data, to transfer the API's knowledge to the CL model. Furthermore, we propose a new regularization term based on network similarity to prevent catastrophic forgetting of previous APIs.Our method performs comparably to classic CL with full raw data on the MNIST and SVHN in the DFCL-APIs setting. In the DECL-APIs setting, our method achieves 0.97x, 0.75x and 0.69x performance of classic CL on CIFAR10, CIFAR100, and MiniImageNet.

protect

defense

Title: Baseline Defenses for Adversarial Attacks Against Aligned Language Models. (arXiv:2309.00614v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.00614
• Code URL: null
• Copy Paste: [[2309.00614]] Baseline Defenses for Adversarial Attacks Against Aligned Language Models(http://arxiv.org/abs/2309.00614)
• Summary:

  As Large Language Models quickly become ubiquitous, their security vulnerabilities are critical to understand. Recent work shows that text optimizers can produce jailbreaking prompts that bypass moderation and alignment. Drawing from the rich body of work on adversarial machine learning, we approach these attacks with three questions: What threat models are practically useful in this domain? How do baseline defense techniques perform in this new domain? How does LLM security differ from computer vision?

  We evaluate several baseline defense strategies against leading adversarial attacks on LLMs, discussing the various settings in which each is feasible and effective. Particularly, we look at three types of defenses: detection (perplexity based), input preprocessing (paraphrase and retokenization), and adversarial training. We discuss white-box and gray-box settings and discuss the robustness-performance trade-off for each of the defenses considered. Surprisingly, we find much more success with filtering and preprocessing than we would expect from other domains, such as vision, providing a first indication that the relative strengths of these defenses may be weighed differently in these domains.

attack

Title: When Measures are Unreliable: Imperceptible Adversarial Perturbations toward Top-$k$ Multi-Label Learning. (arXiv:2309.00007v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.00007
• Code URL: null
• Copy Paste: [[2309.00007]] When Measures are Unreliable: Imperceptible Adversarial Perturbations toward Top-$k$ Multi-Label Learning(http://arxiv.org/abs/2309.00007)
• Summary:

  With the great success of deep neural networks, adversarial learning has received widespread attention in various studies, ranging from multi-class learning to multi-label learning. However, existing adversarial attacks toward multi-label learning only pursue the traditional visual imperceptibility but ignore the new perceptible problem coming from measures such as Precision@$k$ and mAP@$k$. Specifically, when a well-trained multi-label classifier performs far below the expectation on some samples, the victim can easily realize that this performance degeneration stems from attack, rather than the model itself. Therefore, an ideal multi-labeling adversarial attack should manage to not only deceive visual perception but also evade monitoring of measures. To this end, this paper first proposes the concept of measure imperceptibility. Then, a novel loss function is devised to generate such adversarial perturbations that could achieve both visual and measure imperceptibility. Furthermore, an efficient algorithm, which enjoys a convex objective, is established to optimize this objective. Finally, extensive experiments on large-scale benchmark datasets, such as PASCAL VOC 2012, MS COCO, and NUS WIDE, demonstrate the superiority of our proposed method in attacking the top-$k$ multi-label systems.

Title: Model Inversion Attack via Dynamic Memory Learning. (arXiv:2309.00013v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.00013
• Code URL: null
• Copy Paste: [[2309.00013]] Model Inversion Attack via Dynamic Memory Learning(http://arxiv.org/abs/2309.00013)
• Summary:

  Model Inversion (MI) attacks aim to recover the private training data from the target model, which has raised security concerns about the deployment of DNNs in practice. Recent advances in generative adversarial models have rendered them particularly effective in MI attacks, primarily due to their ability to generate high-fidelity and perceptually realistic images that closely resemble the target data. In this work, we propose a novel Dynamic Memory Model Inversion Attack (DMMIA) to leverage historically learned knowledge, which interacts with samples (during the training) to induce diverse generations. DMMIA constructs two types of prototypes to inject the information about historically learned knowledge: Intra-class Multicentric Representation (IMR) representing target-related concepts by multiple learnable prototypes, and Inter-class Discriminative Representation (IDR) characterizing the memorized samples as learned prototypes to capture more privacy-related information. As a result, our DMMIA has a more informative representation, which brings more diverse and discriminative generated results. Experiments on multiple benchmarks show that DMMIA performs better than state-of-the-art MI attack methods.

Title: Impact of Image Context for Single Deep Learning Face Morphing Attack Detection. (arXiv:2309.00549v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.00549
• Code URL: null
• Copy Paste: [[2309.00549]] Impact of Image Context for Single Deep Learning Face Morphing Attack Detection(http://arxiv.org/abs/2309.00549)
• Summary:

  The increase in security concerns due to technological advancements has led to the popularity of biometric approaches that utilize physiological or behavioral characteristics for enhanced recognition. Face recognition systems (FRSs) have become prevalent, but they are still vulnerable to image manipulation techniques such as face morphing attacks. This study investigates the impact of the alignment settings of input images on deep learning face morphing detection performance. We analyze the interconnections between the face contour and image context and suggest optimal alignment conditions for face morphing detection.

Title: Why do universal adversarial attacks work on large language models?: Geometry might be the answer. (arXiv:2309.00254v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.00254
• Code URL: null
• Copy Paste: [[2309.00254]] Why do universal adversarial attacks work on large language models?: Geometry might be the answer(http://arxiv.org/abs/2309.00254)
• Summary:

  Transformer based large language models with emergent capabilities are becoming increasingly ubiquitous in society. However, the task of understanding and interpreting their internal workings, in the context of adversarial attacks, remains largely unsolved. Gradient-based universal adversarial attacks have been shown to be highly effective on large language models and potentially dangerous due to their input-agnostic nature. This work presents a novel geometric perspective explaining universal adversarial attacks on large language models. By attacking the 117M parameter GPT-2 model, we find evidence indicating that universal adversarial triggers could be embedding vectors which merely approximate the semantic information in their adversarial training region. This hypothesis is supported by white-box model analysis comprising dimensionality reduction and similarity measurement of hidden representations. We believe this new geometric perspective on the underlying mechanism driving universal attacks could help us gain deeper insight into the internal workings and failure modes of LLMs, thus enabling their mitigation.

Title: FTA: Stealthy and Robust Backdoor Attack with Flexible Trigger on Federated Learning. (arXiv:2309.00127v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.00127
• Code URL: null
• Copy Paste: [[2309.00127]] FTA: Stealthy and Robust Backdoor Attack with Flexible Trigger on Federated Learning(http://arxiv.org/abs/2309.00127)
• Summary:

  Current backdoor attacks against federated learning (FL) strongly rely on universal triggers or semantic patterns, which can be easily detected and filtered by certain defense mechanisms such as norm clipping, comparing parameter divergences among local updates. In this work, we propose a new stealthy and robust backdoor attack with flexible triggers against FL defenses. To achieve this, we build a generative trigger function that can learn to manipulate the benign samples with an imperceptible flexible trigger pattern and simultaneously make the trigger pattern include the most significant hidden features of the attacker-chosen label. Moreover, our trigger generator can keep learning and adapt across different rounds, allowing it to adjust to changes in the global model. By filling the distinguishable difference (the mapping between the trigger pattern and target label), we make our attack naturally stealthy. Extensive experiments on real-world datasets verify the effectiveness and stealthiness of our attack compared to prior attacks on decentralized learning framework with eight well-studied defenses.

robust

Title: Distraction-free Embeddings for Robust VQA. (arXiv:2309.00133v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.00133
• Code URL: null
• Copy Paste: [[2309.00133]] Distraction-free Embeddings for Robust VQA(http://arxiv.org/abs/2309.00133)
• Summary:

  The generation of effective latent representations and their subsequent refinement to incorporate precise information is an essential prerequisite for Vision-Language Understanding (VLU) tasks such as Video Question Answering (VQA). However, most existing methods for VLU focus on sparsely sampling or fine-graining the input information (e.g., sampling a sparse set of frames or text tokens), or adding external knowledge. We present a novel "DRAX: Distraction Removal and Attended Cross-Alignment" method to rid our cross-modal representations of distractors in the latent space. We do not exclusively confine the perception of any input information from various modalities but instead use an attention-guided distraction removal method to increase focus on task-relevant information in latent embeddings. DRAX also ensures semantic alignment of embeddings during cross-modal fusions. We evaluate our approach on a challenging benchmark (SUTD-TrafficQA dataset), testing the framework's abilities for feature and event queries, temporal relation understanding, forecasting, hypothesis, and causal analysis through extensive experiments.

Title: Fine-Grained Spatiotemporal Motion Alignment for Contrastive Video Representation Learning. (arXiv:2309.00297v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.00297
• Code URL: null
• Copy Paste: [[2309.00297]] Fine-Grained Spatiotemporal Motion Alignment for Contrastive Video Representation Learning(http://arxiv.org/abs/2309.00297)
• Summary:

  As the most essential property in a video, motion information is critical to a robust and generalized video representation. To inject motion dynamics, recent works have adopted frame difference as the source of motion information in video contrastive learning, considering the trade-off between quality and cost. However, existing works align motion features at the instance level, which suffers from spatial and temporal weak alignment across modalities. In this paper, we present a \textbf{Fi}ne-grained \textbf{M}otion \textbf{A}lignment (FIMA) framework, capable of introducing well-aligned and significant motion information. Specifically, we first develop a dense contrastive learning framework in the spatiotemporal domain to generate pixel-level motion supervision. Then, we design a motion decoder and a foreground sampling strategy to eliminate the weak alignments in terms of time and space. Moreover, a frame-level motion contrastive loss is presented to improve the temporal diversity of the motion features. Extensive experiments demonstrate that the representations learned by FIMA possess great motion-awareness capabilities and achieve state-of-the-art or competitive results on downstream tasks across UCF101, HMDB51, and Diving48 datasets. Code is available at \url{https://github.com/ZMHH-H/FIMA}.

Title: Efficient Surrogate Models for Materials Science Simulations: Machine Learning-based Prediction of Microstructure Properties. (arXiv:2309.00305v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.00305
• Code URL: null
• Copy Paste: [[2309.00305]] Efficient Surrogate Models for Materials Science Simulations: Machine Learning-based Prediction of Microstructure Properties(http://arxiv.org/abs/2309.00305)
• Summary:

  Determining, understanding, and predicting the so-called structure-property relation is an important task in many scientific disciplines, such as chemistry, biology, meteorology, physics, engineering, and materials science. Structure refers to the spatial distribution of, e.g., substances, material, or matter in general, while property is a resulting characteristic that usually depends in a non-trivial way on spatial details of the structure. Traditionally, forward simulations models have been used for such tasks. Recently, several machine learning algorithms have been applied in these scientific fields to enhance and accelerate simulation models or as surrogate models. In this work, we develop and investigate the applications of six machine learning techniques based on two different datasets from the domain of materials science: data from a two-dimensional Ising model for predicting the formation of magnetic domains and data representing the evolution of dual-phase microstructures from the Cahn-Hilliard model. We analyze the accuracy and robustness of all models and elucidate the reasons for the differences in their performances. The impact of including domain knowledge through tailored features is studied, and general recommendations based on the availability and quality of training data are derived from this.

Title: Fusing Monocular Images and Sparse IMU Signals for Real-time Human Motion Capture. (arXiv:2309.00310v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.00310
• Code URL: https://github.com/shaohua-pan/RobustCap
• Copy Paste: [[2309.00310]] Fusing Monocular Images and Sparse IMU Signals for Real-time Human Motion Capture(http://arxiv.org/abs/2309.00310)
• Summary:

  Either RGB images or inertial signals have been used for the task of motion capture (mocap), but combining them together is a new and interesting topic. We believe that the combination is complementary and able to solve the inherent difficulties of using one modality input, including occlusions, extreme lighting/texture, and out-of-view for visual mocap and global drifts for inertial mocap. To this end, we propose a method that fuses monocular images and sparse IMUs for real-time human motion capture. Our method contains a dual coordinate strategy to fully explore the IMU signals with different goals in motion capture. To be specific, besides one branch transforming the IMU signals to the camera coordinate system to combine with the image information, there is another branch to learn from the IMU signals in the body root coordinate system to better estimate body poses. Furthermore, a hidden state feedback mechanism is proposed for both two branches to compensate for their own drawbacks in extreme input cases. Thus our method can easily switch between the two kinds of signals or combine them in different cases to achieve a robust mocap. %The two divided parts can help each other for better mocap results under different conditions. Quantitative and qualitative results demonstrate that by delicately designing the fusion method, our technique significantly outperforms the state-of-the-art vision, IMU, and combined methods on both global orientation and local pose estimation. Our codes are available for research at https://shaohua-pan.github.io/robustcap-page/.

Title: Robust Point Cloud Processing through Positional Embedding. (arXiv:2309.00339v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.00339
• Code URL: null
• Copy Paste: [[2309.00339]] Robust Point Cloud Processing through Positional Embedding(http://arxiv.org/abs/2309.00339)
• Summary:

  End-to-end trained per-point embeddings are an essential ingredient of any state-of-the-art 3D point cloud processing such as detection or alignment. Methods like PointNet, or the more recent point cloud transformer -- and its variants -- all employ learned per-point embeddings. Despite impressive performance, such approaches are sensitive to out-of-distribution (OOD) noise and outliers. In this paper, we explore the role of an analytical per-point embedding based on the criterion of bandwidth. The concept of bandwidth enables us to draw connections with an alternate per-point embedding -- positional embedding, particularly random Fourier features. We present compelling robust results across downstream tasks such as point cloud classification and registration with several categories of OOD noise.

Title: A Theoretical and Practical Framework for Evaluating Uncertainty Calibration in Object Detection. (arXiv:2309.00464v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.00464
• Code URL: https://github.com/pedrormconde/uncertainty_calibration_object_detection
• Copy Paste: [[2309.00464]] A Theoretical and Practical Framework for Evaluating Uncertainty Calibration in Object Detection(http://arxiv.org/abs/2309.00464)
• Summary:

  The proliferation of Deep Neural Networks has resulted in machine learning systems becoming increasingly more present in various real-world applications. Consequently, there is a growing demand for highly reliable models in these domains, making the problem of uncertainty calibration pivotal, when considering the future of deep learning. This is especially true when considering object detection systems, that are commonly present in safety-critical application such as autonomous driving and robotics. For this reason, this work presents a novel theoretical and practical framework to evaluate object detection systems in the context of uncertainty calibration. The robustness of the proposed uncertainty calibration metrics is shown through a series of representative experiments. Code for the proposed uncertainty calibration metrics at: https://github.com/pedrormconde/Uncertainty_Calibration_Object_Detection.

Title: A Machine Vision Method for Correction of Eccentric Error: Based on Adaptive Enhancement Algorithm. (arXiv:2309.00514v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.00514
• Code URL: null
• Copy Paste: [[2309.00514]] A Machine Vision Method for Correction of Eccentric Error: Based on Adaptive Enhancement Algorithm(http://arxiv.org/abs/2309.00514)
• Summary:

  In the procedure of surface defects detection for large-aperture aspherical optical elements, it is of vital significance to adjust the optical axis of the element to be coaxial with the mechanical spin axis accurately. Therefore, a machine vision method for eccentric error correction is proposed in this paper. Focusing on the severe defocus blur of reference crosshair image caused by the imaging characteristic of the aspherical optical element, which may lead to the failure of correction, an Adaptive Enhancement Algorithm (AEA) is proposed to strengthen the crosshair image. AEA is consisted of existed Guided Filter Dark Channel Dehazing Algorithm (GFA) and proposed lightweight Multi-scale Densely Connected Network (MDC-Net). The enhancement effect of GFA is excellent but time-consuming, and the enhancement effect of MDC-Net is slightly inferior but strongly real-time. As AEA will be executed dozens of times during each correction procedure, its real-time performance is very important. Therefore, by setting the empirical threshold of definition evaluation function SMD2, GFA and MDC-Net are respectively applied to highly and slightly blurred crosshair images so as to ensure the enhancement effect while saving as much time as possible. AEA has certain robustness in time-consuming performance, which takes an average time of 0.2721s and 0.0963s to execute GFA and MDC-Net separately on ten 200pixels 200pixels Region of Interest (ROI) images with different degrees of blur. And the eccentricity error can be reduced to within 10um by our method.

Title: SQLdepth: Generalizable Self-Supervised Fine-Structured Monocular Depth Estimation. (arXiv:2309.00526v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.00526
• Code URL: null
• Copy Paste: [[2309.00526]] SQLdepth: Generalizable Self-Supervised Fine-Structured Monocular Depth Estimation(http://arxiv.org/abs/2309.00526)
• Summary:

  Recently, self-supervised monocular depth estimation has gained popularity with numerous applications in autonomous driving and robotics. However, existing solutions primarily seek to estimate depth from immediate visual features, and struggle to recover fine-grained scene details with limited generalization. In this paper, we introduce SQLdepth, a novel approach that can effectively learn fine-grained scene structures from motion. In SQLdepth, we propose a novel Self Query Layer (SQL) to build a self-cost volume and infer depth from it, rather than inferring depth from feature maps. The self-cost volume implicitly captures the intrinsic geometry of the scene within a single frame. Each individual slice of the volume signifies the relative distances between points and objects within a latent space. Ultimately, this volume is compressed to the depth map via a novel decoding approach. Experimental results on KITTI and Cityscapes show that our method attains remarkable state-of-the-art performance (AbsRel = $0.082$ on KITTI, $0.052$ on KITTI with improved ground-truth and $0.106$ on Cityscapes), achieves $9.9\%$, $5.5\%$ and $4.5\%$ error reduction from the previous best. In addition, our approach showcases reduced training complexity, computational efficiency, improved generalization, and the ability to recover fine-grained scene details. Moreover, the self-supervised pre-trained and metric fine-tuned SQLdepth can surpass existing supervised methods by significant margins (AbsRel = $0.043$, $14\%$ error reduction). self-matching-oriented relative distance querying in SQL improves the robustness and zero-shot generalization capability of SQLdepth. Code and the pre-trained weights will be publicly available. Code is available at \href{https://github.com/hisfog/SQLdepth-Impl}{https://github.com/hisfog/SQLdepth-Impl}.

Title: Information Fusion for Assistance Systems in Production Assessment. (arXiv:2309.00157v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.00157
• Code URL: null
• Copy Paste: [[2309.00157]] Information Fusion for Assistance Systems in Production Assessment(http://arxiv.org/abs/2309.00157)
• Summary:

  We propose a novel methodology to define assistance systems that rely on information fusion to combine different sources of information while providing an assessment. The main contribution of this paper is providing a general framework for the fusion of n number of information sources using the evidence theory. The fusion provides a more robust prediction and an associated uncertainty that can be used to assess the prediction likeliness. Moreover, we provide a methodology for the information fusion of two primary sources: an ensemble classifier based on machine data and an expert-centered model. We demonstrate the information fusion approach using data from an industrial setup, which rounds up the application part of this research. Furthermore, we address the problem of data drift by proposing a methodology to update the data-based models using an evidence theory approach. We validate the approach using the Benchmark Tennessee Eastman while doing an ablation study of the model update parameters.

Title: Subjectivity in Unsupervised Machine Learning Model Selection. (arXiv:2309.00201v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.00201
• Code URL: null
• Copy Paste: [[2309.00201]] Subjectivity in Unsupervised Machine Learning Model Selection(http://arxiv.org/abs/2309.00201)
• Summary:

  Model selection is a necessary step in unsupervised machine learning. Despite numerous criteria and metrics, model selection remains subjective. A high degree of subjectivity may lead to questions about repeatability and reproducibility of various machine learning studies and doubts about the robustness of models deployed in the real world. Yet, the impact of modelers' preferences on model selection outcomes remains largely unexplored. This study uses the Hidden Markov Model as an example to investigate the subjectivity involved in model selection. We asked 33 participants and three Large Language Models (LLMs) to make model selections in three scenarios. Results revealed variability and inconsistencies in both the participants' and the LLMs' choices, especially when different criteria and metrics disagree. Sources of subjectivity include varying opinions on the importance of different criteria and metrics, differing views on how parsimonious a model should be, and how the size of a dataset should influence model selection. The results underscore the importance of developing a more standardized way to document subjective choices made in model selection processes.

Title: Multi-fidelity reduced-order surrogate modeling. (arXiv:2309.00325v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.00325
• Code URL: https://github.com/contipaolo/multifidelity_pod
• Copy Paste: [[2309.00325]] Multi-fidelity reduced-order surrogate modeling(http://arxiv.org/abs/2309.00325)
• Summary:

  High-fidelity numerical simulations of partial differential equations (PDEs) given a restricted computational budget can significantly limit the number of parameter configurations considered and/or time window evaluated for modeling a given system. Multi-fidelity surrogate modeling aims to leverage less accurate, lower-fidelity models that are computationally inexpensive in order to enhance predictive accuracy when high-fidelity data are limited or scarce. However, low-fidelity models, while often displaying important qualitative spatio-temporal features, fail to accurately capture the onset of instability and critical transients observed in the high-fidelity models, making them impractical as surrogate models. To address this shortcoming, we present a new data-driven strategy that combines dimensionality reduction with multi-fidelity neural network surrogates. The key idea is to generate a spatial basis by applying the classical proper orthogonal decomposition (POD) to high-fidelity solution snapshots, and approximate the dynamics of the reduced states - time-parameter-dependent expansion coefficients of the POD basis - using a multi-fidelity long-short term memory (LSTM) network. By mapping low-fidelity reduced states to their high-fidelity counterpart, the proposed reduced-order surrogate model enables the efficient recovery of full solution fields over time and parameter variations in a non-intrusive manner. The generality and robustness of this method is demonstrated by a collection of parametrized, time-dependent PDE problems where the low-fidelity model can be defined by coarser meshes and/or time stepping, as well as by misspecified physical features. Importantly, the onset of instabilities and transients are well captured by this surrogate modeling technique.

Title: Curating Naturally Adversarial Datasets for Trustworthy AI in Healthcare. (arXiv:2309.00543v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.00543
• Code URL: null
• Copy Paste: [[2309.00543]] Curating Naturally Adversarial Datasets for Trustworthy AI in Healthcare(http://arxiv.org/abs/2309.00543)
• Summary:

  Deep learning models have shown promising predictive accuracy for time-series healthcare applications. However, ensuring the robustness of these models is vital for building trustworthy AI systems. Existing research predominantly focuses on robustness to synthetic adversarial examples, crafted by adding imperceptible perturbations to clean input data. However, these synthetic adversarial examples do not accurately reflect the most challenging real-world scenarios, especially in the context of healthcare data. Consequently, robustness to synthetic adversarial examples may not necessarily translate to robustness against naturally occurring adversarial examples, which is highly desirable for trustworthy AI. We propose a method to curate datasets comprised of natural adversarial examples to evaluate model robustness. The method relies on probabilistic labels obtained from automated weakly-supervised labeling that combines noisy and cheap-to-obtain labeling heuristics. Based on these labels, our method adversarially orders the input data and uses this ordering to construct a sequence of increasingly adversarial datasets. Our evaluation on six medical case studies and three non-medical case studies demonstrates the efficacy and statistical validity of our approach to generating naturally adversarial datasets

biometric

steal

extraction

Title: Unsupervised discovery of Interpretable Visual Concepts. (arXiv:2309.00018v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.00018
• Code URL: null
• Copy Paste: [[2309.00018]] Unsupervised discovery of Interpretable Visual Concepts(http://arxiv.org/abs/2309.00018)
• Summary:

  Providing interpretability of deep-learning models to non-experts, while fundamental for a responsible real-world usage, is challenging. Attribution maps from xAI techniques, such as Integrated Gradients, are a typical example of a visualization technique containing a high level of information, but with difficult interpretation. In this paper, we propose two methods, Maximum Activation Groups Extraction (MAGE) and Multiscale Interpretable Visualization (Ms-IV), to explain the model's decision, enhancing global interpretability. MAGE finds, for a given CNN, combinations of features which, globally, form a semantic meaning, that we call concepts. We group these similar feature patterns by clustering in ``concepts'', that we visualize through Ms-IV. This last method is inspired by Occlusion and Sensitivity analysis (incorporating causality), and uses a novel metric, called Class-aware Order Correlation (CaOC), to globally evaluate the most important image regions according to the model's decision space. We compare our approach to xAI methods such as LIME and Integrated Gradients. Experimental results evince the Ms-IV higher localization and faithfulness values. Finally, qualitative evaluation of combined MAGE and Ms-IV demonstrate humans' ability to agree, based on the visualization, on the decision of clusters' concepts; and, to detect, among a given set of networks, the existence of bias.

Title: ARFA: An Asymmetric Receptive Field Autoencoder Model for Spatiotemporal Prediction. (arXiv:2309.00314v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.00314
• Code URL: null
• Copy Paste: [[2309.00314]] ARFA: An Asymmetric Receptive Field Autoencoder Model for Spatiotemporal Prediction(http://arxiv.org/abs/2309.00314)
• Summary:

  Spatiotemporal prediction aims to generate future sequences by paradigms learned from historical contexts. It holds significant importance in numerous domains, including traffic flow prediction and weather forecasting. However, existing methods face challenges in handling spatiotemporal correlations, as they commonly adopt encoder and decoder architectures with identical receptive fields, which adversely affects prediction accuracy. This paper proposes an Asymmetric Receptive Field Autoencoder (ARFA) model to address this issue. Specifically, we design corresponding sizes of receptive field modules tailored to the distinct functionalities of the encoder and decoder. In the encoder, we introduce a large kernel module for global spatiotemporal feature extraction. In the decoder, we develop a small kernel module for local spatiotemporal information reconstruction. To address the scarcity of meteorological prediction data, we constructed the RainBench, a large-scale radar echo dataset specific to the unique precipitation characteristics of inland regions in China for precipitation prediction. Experimental results demonstrate that ARFA achieves consistent state-of-the-art performance on two mainstream spatiotemporal prediction datasets and our RainBench dataset, affirming the effectiveness of our approach. This work not only explores a novel method from the perspective of receptive fields but also provides data support for precipitation prediction, thereby advancing future research in spatiotemporal prediction.

Title: MuraNet: Multi-task Floor Plan Recognition with Relation Attention. (arXiv:2309.00348v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.00348
• Code URL: null
• Copy Paste: [[2309.00348]] MuraNet: Multi-task Floor Plan Recognition with Relation Attention(http://arxiv.org/abs/2309.00348)
• Summary:

  The recognition of information in floor plan data requires the use of detection and segmentation models. However, relying on several single-task models can result in ineffective utilization of relevant information when there are multiple tasks present simultaneously. To address this challenge, we introduce MuraNet, an attention-based multi-task model for segmentation and detection tasks in floor plan data. In MuraNet, we adopt a unified encoder called MURA as the backbone with two separated branches: an enhanced segmentation decoder branch and a decoupled detection head branch based on YOLOX, for segmentation and detection tasks respectively. The architecture of MuraNet is designed to leverage the fact that walls, doors, and windows usually constitute the primary structure of a floor plan's architecture. By jointly training the model on both detection and segmentation tasks, we believe MuraNet can effectively extract and utilize relevant features for both tasks. Our experiments on the CubiCasa5k public dataset show that MuraNet improves convergence speed during training compared to single-task models like U-Net and YOLOv3. Moreover, we observe improvements in the average AP and IoU in detection and segmentation tasks, respectively.Our ablation experiments demonstrate that the attention-based unified backbone of MuraNet achieves better feature extraction in floor plan recognition tasks, and the use of decoupled multi-head branches for different tasks further improves model performance. We believe that our proposed MuraNet model can address the disadvantages of single-task models and improve the accuracy and efficiency of floor plan data recognition.

Title: Exploring the law of text geographic information. (arXiv:2309.00180v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.00180
• Code URL: null
• Copy Paste: [[2309.00180]] Exploring the law of text geographic information(http://arxiv.org/abs/2309.00180)
• Summary:

  Textual geographic information is indispensable and heavily relied upon in practical applications. The absence of clear distribution poses challenges in effectively harnessing geographic information, thereby driving our quest for exploration. We contend that geographic information is influenced by human behavior, cognition, expression, and thought processes, and given our intuitive understanding of natural systems, we hypothesize its conformity to the Gamma distribution. Through rigorous experiments on a diverse range of 24 datasets encompassing different languages and types, we have substantiated this hypothesis, unearthing the underlying regularities governing the dimensions of quantity, length, and distance in geographic information. Furthermore, theoretical analyses and comparisons with Gaussian distributions and Zipf's law have refuted the contingency of these laws. Significantly, we have estimated the upper bounds of human utilization of geographic information, pointing towards the existence of uncharted territories. Also, we provide guidance in geographic information extraction. Hope we peer its true countenance uncovering the veil of geographic information.

membership infer

federate

Title: Leveraging Learning Metrics for Improved Federated Learning. (arXiv:2309.00257v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.00257
• Code URL: null
• Copy Paste: [[2309.00257]] Leveraging Learning Metrics for Improved Federated Learning(http://arxiv.org/abs/2309.00257)
• Summary:

  Currently in the federated setting, no learning schemes leverage the emerging research of explainable artificial intelligence (XAI) in particular the novel learning metrics that help determine how well a model is learning. One of these novel learning metrics is termed `Effective Rank' (ER) which measures the Shannon Entropy of the singular values of a matrix, thus enabling a metric determining how well a layer is mapping. By joining federated learning and the learning metric, effective rank, this work will \textbf{(1)} give the first federated learning metric aggregation method \textbf{(2)} show that effective rank is well-suited to federated problems by out-performing baseline Federated Averaging \cite{konevcny2016federated} and \textbf{(3)} develop a novel weight-aggregation scheme relying on effective rank.

Title: FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning. (arXiv:2309.00363v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.00363
• Code URL: https://github.com/alibaba/federatedscope
• Copy Paste: [[2309.00363]] FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning(http://arxiv.org/abs/2309.00363)
• Summary:

  LLMs have demonstrated great capabilities in various NLP tasks. Different entities can further improve the performance of those LLMs on their specific downstream tasks by fine-tuning LLMs. When several entities have similar interested tasks, but their data cannot be shared because of privacy concerns regulations, federated learning (FL) is a mainstream solution to leverage the data of different entities. However, fine-tuning LLMs in federated learning settings still lacks adequate support from existing FL frameworks because it has to deal with optimizing the consumption of significant communication and computational resources, data preparation for different tasks, and distinct information protection demands. This paper first discusses these challenges of federated fine-tuning LLMs, and introduces our package FS-LLM as a main contribution, which consists of the following components: (1) we build an end-to-end benchmarking pipeline, automizing the processes of dataset preprocessing, federated fine-tuning execution, and performance evaluation on federated LLM fine-tuning; (2) we provide comprehensive federated parameter-efficient fine-tuning algorithm implementations and versatile programming interfaces for future extension in FL scenarios with low communication and computation costs, even without accessing the full model; (3) we adopt several accelerating and resource-efficient operators for fine-tuning LLMs with limited resources and the flexible pluggable sub-routines for interdisciplinary study. We conduct extensive experiments to validate the effectiveness of FS-LLM and benchmark advanced LLMs with state-of-the-art parameter-efficient fine-tuning algorithms in FL settings, which also yields valuable insights into federated fine-tuning LLMs for the research community. To facilitate further research and adoption, we release FS-LLM at https://github.com/alibaba/FederatedScope/tree/llm.

fair

Title: FACET: Fairness in Computer Vision Evaluation Benchmark. (arXiv:2309.00035v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.00035
• Code URL: null
• Copy Paste: [[2309.00035]] FACET: Fairness in Computer Vision Evaluation Benchmark(http://arxiv.org/abs/2309.00035)
• Summary:

  Computer vision models have known performance disparities across attributes such as gender and skin tone. This means during tasks such as classification and detection, model performance differs for certain classes based on the demographics of the people in the image. These disparities have been shown to exist, but until now there has not been a unified approach to measure these differences for common use-cases of computer vision models. We present a new benchmark named FACET (FAirness in Computer Vision EvaluaTion), a large, publicly available evaluation set of 32k images for some of the most common vision tasks - image classification, object detection and segmentation. For every image in FACET, we hired expert reviewers to manually annotate person-related attributes such as perceived skin tone and hair type, manually draw bounding boxes and label fine-grained person-related classes such as disk jockey or guitarist. In addition, we use FACET to benchmark state-of-the-art vision models and present a deeper understanding of potential performance disparities and challenges across sensitive demographic attributes. With the exhaustive annotations collected, we probe models using single demographics attributes as well as multiple attributes using an intersectional approach (e.g. hair color and perceived skin tone). Our results show that classification, detection, segmentation, and visual grounding models exhibit performance disparities across demographic attributes and intersections of attributes. These harms suggest that not all people represented in datasets receive fair and equitable treatment in these vision tasks. We hope current and future results using our benchmark will contribute to fairer, more robust vision models. FACET is available publicly at https://facet.metademolab.com/

interpretability

explainability

Title: NeuroSurgeon: A Toolkit for Subnetwork Analysis. (arXiv:2309.00244v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.00244
• Code URL: https://github.com/mlepori1/neurosurgeon
• Copy Paste: [[2309.00244]] NeuroSurgeon: A Toolkit for Subnetwork Analysis(http://arxiv.org/abs/2309.00244)
• Summary:

  Despite recent advances in the field of explainability, much remains unknown about the algorithms that neural networks learn to represent. Recent work has attempted to understand trained models by decomposing them into functional circuits (Csord\'as et al., 2020; Lepori et al., 2023). To advance this research, we developed NeuroSurgeon, a python library that can be used to discover and manipulate subnetworks within models in the Huggingface Transformers library (Wolf et al., 2019). NeuroSurgeon is freely available at https://github.com/mlepori1/NeuroSurgeon.

watermark

diffusion

Title: BuilDiff: 3D Building Shape Generation using Single-Image Conditional Point Cloud Diffusion Models. (arXiv:2309.00158v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.00158
• Code URL: null
• Copy Paste: [[2309.00158]] BuilDiff: 3D Building Shape Generation using Single-Image Conditional Point Cloud Diffusion Models(http://arxiv.org/abs/2309.00158)
• Summary:

  3D building generation with low data acquisition costs, such as single image-to-3D, becomes increasingly important. However, most of the existing single image-to-3D building creation works are restricted to those images with specific viewing angles, hence they are difficult to scale to general-view images that commonly appear in practical cases. To fill this gap, we propose a novel 3D building shape generation method exploiting point cloud diffusion models with image conditioning schemes, which demonstrates flexibility to the input images. By cooperating two conditional diffusion models and introducing a regularization strategy during denoising process, our method is able to synthesize building roofs while maintaining the overall structures. We validate our framework on two newly built datasets and extensive experiments show that our method outperforms previous works in terms of building generation quality.

Title: Diffusion Model with Clustering-based Conditioning for Food Image Generation. (arXiv:2309.00199v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.00199
• Code URL: null
• Copy Paste: [[2309.00199]] Diffusion Model with Clustering-based Conditioning for Food Image Generation(http://arxiv.org/abs/2309.00199)
• Summary:

  Image-based dietary assessment serves as an efficient and accurate solution for recording and analyzing nutrition intake using eating occasion images as input. Deep learning-based techniques are commonly used to perform image analysis such as food classification, segmentation, and portion size estimation, which rely on large amounts of food images with annotations for training. However, such data dependency poses significant barriers to real-world applications, because acquiring a substantial, diverse, and balanced set of food images can be challenging. One potential solution is to use synthetic food images for data augmentation. Although existing work has explored the use of generative adversarial networks (GAN) based structures for generation, the quality of synthetic food images still remains subpar. In addition, while diffusion-based generative models have shown promising results for general image generation tasks, the generation of food images can be challenging due to the substantial intra-class variance. In this paper, we investigate the generation of synthetic food images based on the conditional diffusion model and propose an effective clustering-based training framework, named ClusDiff, for generating high-quality and representative food images. The proposed method is evaluated on the Food-101 dataset and shows improved performance when compared with existing image generation works. We also demonstrate that the synthetic food images generated by ClusDiff can help address the severe class imbalance issue in long-tailed food classification using the VFN-LT dataset.

Title: DiffuGen: Adaptable Approach for Generating Labeled Image Datasets using Stable Diffusion Models. (arXiv:2309.00248v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.00248
• Code URL: https://github.com/mshenoda/diffugen
• Copy Paste: [[2309.00248]] DiffuGen: Adaptable Approach for Generating Labeled Image Datasets using Stable Diffusion Models(http://arxiv.org/abs/2309.00248)
• Summary:

  Generating high-quality labeled image datasets is crucial for training accurate and robust machine learning models in the field of computer vision. However, the process of manually labeling real images is often time-consuming and costly. To address these challenges associated with dataset generation, we introduce "DiffuGen," a simple and adaptable approach that harnesses the power of stable diffusion models to create labeled image datasets efficiently. By leveraging stable diffusion models, our approach not only ensures the quality of generated datasets but also provides a versatile solution for label generation. In this paper, we present the methodology behind DiffuGen, which combines the capabilities of diffusion models with two distinct labeling techniques: unsupervised and supervised. Distinctively, DiffuGen employs prompt templating for adaptable image generation and textual inversion to enhance diffusion model capabilities.

Title: Fast Diffusion EM: a diffusion model for blind inverse problems with application to deconvolution. (arXiv:2309.00287v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.00287
• Code URL: null
• Copy Paste: [[2309.00287]] Fast Diffusion EM: a diffusion model for blind inverse problems with application to deconvolution(http://arxiv.org/abs/2309.00287)
• Summary:

  Using diffusion models to solve inverse problems is a growing field of research. Current methods assume the degradation to be known and provide impressive results in terms of restoration quality and diversity. In this work, we leverage the efficiency of those models to jointly estimate the restored image and unknown parameters of the degradation model. In particular, we designed an algorithm based on the well-known Expectation-Minimization (EM) estimation method and diffusion models. Our method alternates between approximating the expected log-likelihood of the inverse problem using samples drawn from a diffusion model and a maximization step to estimate unknown model parameters. For the maximization step, we also introduce a novel blur kernel regularization based on a Plug \& Play denoiser. Diffusion models are long to run, thus we provide a fast version of our algorithm. Extensive experiments on blind image deblurring demonstrate the effectiveness of our method when compared to other state-of-the-art approaches.

Title: VideoGen: A Reference-Guided Latent Diffusion Approach for High Definition Text-to-Video Generation. (arXiv:2309.00398v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.00398
• Code URL: null
• Copy Paste: [[2309.00398]] VideoGen: A Reference-Guided Latent Diffusion Approach for High Definition Text-to-Video Generation(http://arxiv.org/abs/2309.00398)
• Summary:

  In this paper, we present VideoGen, a text-to-video generation approach, which can generate a high-definition video with high frame fidelity and strong temporal consistency using reference-guided latent diffusion. We leverage an off-the-shelf text-to-image generation model, e.g., Stable Diffusion, to generate an image with high content quality from the text prompt, as a reference image to guide video generation. Then, we introduce an efficient cascaded latent diffusion module conditioned on both the reference image and the text prompt, for generating latent video representations, followed by a flow-based temporal upsampling step to improve the temporal resolution. Finally, we map latent video representations into a high-definition video through an enhanced video decoder. During training, we use the first frame of a ground-truth video as the reference image for training the cascaded latent diffusion module. The main characterises of our approach include: the reference image generated by the text-to-image model improves the visual fidelity; using it as the condition makes the diffusion model focus more on learning the video dynamics; and the video decoder is trained over unlabeled video data, thus benefiting from high-quality easily-available videos. VideoGen sets a new state-of-the-art in text-to-video generation in terms of both qualitative and quantitative evaluation.

Title: Iterative Multi-granular Image Editing using Diffusion Models. (arXiv:2309.00613v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.00613
• Code URL: null
• Copy Paste: [[2309.00613]] Iterative Multi-granular Image Editing using Diffusion Models(http://arxiv.org/abs/2309.00613)
• Summary:

  Recent advances in text-guided image synthesis has dramatically changed how creative professionals generate artistic and aesthetically pleasing visual assets. To fully support such creative endeavors, the process should possess the ability to: 1) iteratively edit the generations and 2) control the spatial reach of desired changes (global, local or anything in between). We formalize this pragmatic problem setting as Iterative Multi-granular Editing. While there has been substantial progress with diffusion-based models for image synthesis and editing, they are all one shot (i.e., no iterative editing capabilities) and do not naturally yield multi-granular control (i.e., covering the full spectrum of local-to-global edits). To overcome these drawbacks, we propose EMILIE: Iterative Multi-granular Image Editor. EMILIE introduces a novel latent iteration strategy, which re-purposes a pre-trained diffusion model to facilitate iterative editing. This is complemented by a gradient control operation for multi-granular control. We introduce a new benchmark dataset to evaluate our newly proposed setting. We conduct exhaustive quantitatively and qualitatively evaluation against recent state-of-the-art approaches adapted to our task, to being out the mettle of EMILIE. We hope our work would attract attention to this newly identified, pragmatic problem setting.

noise learning

data-free

Title: Unsupervised evaluation of GAN sample quality: Introducing the TTJac Score. (arXiv:2309.00107v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.00107
• Code URL: null
• Copy Paste: [[2309.00107]] Unsupervised evaluation of GAN sample quality: Introducing the TTJac Score(http://arxiv.org/abs/2309.00107)
• Summary:

  Evaluation metrics are essential for assessing the performance of generative models in image synthesis. However, existing metrics often involve high memory and time consumption as they compute the distance between generated samples and real data points. In our study, the new evaluation metric called the "TTJac score" is proposed to measure the fidelity of individual synthesized images in a data-free manner. The study first establishes a theoretical approach to directly evaluate the generated sample density. Then, a method incorporating feature extractors and discrete function approximation through tensor train is introduced to effectively assess the quality of generated samples. Furthermore, the study demonstrates that this new metric can be used to improve the fidelity-variability trade-off when applying the truncation trick. The experimental results of applying the proposed metric to StyleGAN 2 and StyleGAN 2 ADA models on FFHQ, AFHQ-Wild, LSUN-Cars, and LSUN-Horse datasets are presented. The code used in this research will be made publicly available online for the research community to access and utilize.

transformer

Title: Laplacian-Former: Overcoming the Limitations of Vision Transformers in Local Texture Detection. (arXiv:2309.00108v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.00108
• Code URL: null
• Copy Paste: [[2309.00108]] Laplacian-Former: Overcoming the Limitations of Vision Transformers in Local Texture Detection(http://arxiv.org/abs/2309.00108)
• Summary:

  Vision Transformer (ViT) models have demonstrated a breakthrough in a wide range of computer vision tasks. However, compared to the Convolutional Neural Network (CNN) models, it has been observed that the ViT models struggle to capture high-frequency components of images, which can limit their ability to detect local textures and edge information. As abnormalities in human tissue, such as tumors and lesions, may greatly vary in structure, texture, and shape, high-frequency information such as texture is crucial for effective semantic segmentation tasks. To address this limitation in ViT models, we propose a new technique, Laplacian-Former, that enhances the self-attention map by adaptively re-calibrating the frequency information in a Laplacian pyramid. More specifically, our proposed method utilizes a dual attention mechanism via efficient attention and frequency attention while the efficient attention mechanism reduces the complexity of self-attention to linear while producing the same output, selectively intensifying the contribution of shape and texture features. Furthermore, we introduce a novel efficient enhancement multi-scale bridge that effectively transfers spatial information from the encoder to the decoder while preserving the fundamental features. We demonstrate the efficacy of Laplacian-former on multi-organ and skin lesion segmentation tasks with +1.87\% and +0.76\% dice scores compared to SOTA approaches, respectively. Our implementation is publically available at https://github.com/mindflow-institue/Laplacian-Former

Title: Interpretable Medical Imagery Diagnosis with Self-Attentive Transformers: A Review of Explainable AI for Health Care. (arXiv:2309.00252v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.00252
• Code URL: null
• Copy Paste: [[2309.00252]] Interpretable Medical Imagery Diagnosis with Self-Attentive Transformers: A Review of Explainable AI for Health Care(http://arxiv.org/abs/2309.00252)
• Summary:

  Recent advancements in artificial intelligence (AI) have facilitated its widespread adoption in primary medical services, addressing the demand-supply imbalance in healthcare. Vision Transformers (ViT) have emerged as state-of-the-art computer vision models, benefiting from self-attention modules. However, compared to traditional machine-learning approaches, deep-learning models are complex and are often treated as a "black box" that can cause uncertainty regarding how they operate. Explainable Artificial Intelligence (XAI) refers to methods that explain and interpret machine learning models' inner workings and how they come to decisions, which is especially important in the medical domain to guide the healthcare decision-making process. This review summarises recent ViT advancements and interpretative approaches to understanding the decision-making process of ViT, enabling transparency in medical diagnosis applications.

Title: JoTR: A Joint Transformer and Reinforcement Learning Framework for Dialog Policy Learning. (arXiv:2309.00230v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.00230
• Code URL: https://github.com/kwanwaichung/jotr
• Copy Paste: [[2309.00230]] JoTR: A Joint Transformer and Reinforcement Learning Framework for Dialog Policy Learning(http://arxiv.org/abs/2309.00230)
• Summary:

  Dialogue policy learning (DPL) is a crucial component of dialogue modelling. Its primary role is to determine the appropriate abstract response, commonly referred to as the "dialogue action". Traditional DPL methodologies have treated this as a sequential decision problem, using pre-defined action candidates extracted from a corpus. However, these incomplete candidates can significantly limit the diversity of responses and pose challenges when dealing with edge cases, which are scenarios that occur only at extreme operating parameters. To address these limitations, we introduce a novel framework, JoTR. This framework is unique as it leverages a text-to-text Transformer-based model to generate flexible dialogue actions. Unlike traditional methods, JoTR formulates a word-level policy that allows for a more dynamic and adaptable dialogue action generation, without the need for any action templates. This setting enhances the diversity of responses and improves the system's ability to handle edge cases effectively. In addition, JoTR employs reinforcement learning with a reward-shaping mechanism to efficiently finetune the word-level dialogue policy, which allows the model to learn from its interactions, improving its performance over time. We conducted an extensive evaluation of JoTR to assess its effectiveness. Our extensive evaluation shows that JoTR achieves state-of-the-art performance on two benchmark dialogue modelling tasks, as assessed by both user simulators and human evaluators.

Title: Where Did the Gap Go? Reassessing the Long-Range Graph Benchmark. (arXiv:2309.00367v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.00367
• Code URL: null
• Copy Paste: [[2309.00367]] Where Did the Gap Go? Reassessing the Long-Range Graph Benchmark(http://arxiv.org/abs/2309.00367)
• Summary:

  The recent Long-Range Graph Benchmark (LRGB, Dwivedi et al. 2022) introduced a set of graph learning tasks strongly dependent on long-range interaction between vertices. Empirical evidence suggests that on these tasks Graph Transformers significantly outperform Message Passing GNNs (MPGNNs). In this paper, we carefully reevaluate multiple MPGNN baselines as well as the Graph Transformer GPS (Ramp\'a\v{s}ek et al. 2022) on LRGB. Through a rigorous empirical analysis, we demonstrate that the reported performance gap is overestimated due to suboptimal hyperparameter choices. It is noteworthy that across multiple datasets the performance gap completely vanishes after basic hyperparameter optimization. In addition, we discuss the impact of lacking feature normalization for LRGB's vision datasets and highlight a spurious implementation of LRGB's link prediction metric. The principal aim of our paper is to establish a higher standard of empirical rigor within the graph machine learning community.

Title: Geometry-aware Line Graph Transformer Pre-training for Molecular Property Prediction. (arXiv:2309.00483v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.00483
• Code URL: null
• Copy Paste: [[2309.00483]] Geometry-aware Line Graph Transformer Pre-training for Molecular Property Prediction(http://arxiv.org/abs/2309.00483)
• Summary:

  Molecular property prediction with deep learning has gained much attention over the past years. Owing to the scarcity of labeled molecules, there has been growing interest in self-supervised learning methods that learn generalizable molecular representations from unlabeled data. Molecules are typically treated as 2D topological graphs in modeling, but it has been discovered that their 3D geometry is of great importance in determining molecular functionalities. In this paper, we propose the Geometry-aware line graph transformer (Galformer) pre-training, a novel self-supervised learning framework that aims to enhance molecular representation learning with 2D and 3D modalities. Specifically, we first design a dual-modality line graph transformer backbone to encode the topological and geometric information of a molecule. The designed backbone incorporates effective structural encodings to capture graph structures from both modalities. Then we devise two complementary pre-training tasks at the inter and intra-modality levels. These tasks provide properly supervised information and extract discriminative 2D and 3D knowledge from unlabeled molecules. Finally, we evaluate Galformer against six state-of-the-art baselines on twelve property prediction benchmarks via downstream fine-tuning. Experimental results show that Galformer consistently outperforms all baselines on both classification and regression tasks, demonstrating its effectiveness.

Title: PolyGET: Accelerating Polymer Simulations by Accurate and Generalizable Forcefield with Equivariant Transformer. (arXiv:2309.00585v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.00585
• Code URL: null
• Copy Paste: [[2309.00585]] PolyGET: Accelerating Polymer Simulations by Accurate and Generalizable Forcefield with Equivariant Transformer(http://arxiv.org/abs/2309.00585)
• Summary:

  Polymer simulation with both accuracy and efficiency is a challenging task. Machine learning (ML) forcefields have been developed to achieve both the accuracy of ab initio methods and the efficiency of empirical force fields. However, existing ML force fields are usually limited to single-molecule settings, and their simulations are not robust enough. In this paper, we present PolyGET, a new framework for Polymer Forcefields with Generalizable Equivariant Transformers. PolyGET is designed to capture complex quantum interactions between atoms and generalize across various polymer families, using a deep learning model called Equivariant Transformers. We propose a new training paradigm that focuses exclusively on optimizing forces, which is different from existing methods that jointly optimize forces and energy. This simple force-centric objective function avoids competing objectives between energy and forces, thereby allowing for learning a unified forcefield ML model over different polymer families. We evaluated PolyGET on a large-scale dataset of 24 distinct polymer types and demonstrated state-of-the-art performance in force accuracy and robust MD simulations. Furthermore, PolyGET can simulate large polymers with high fidelity to the reference ab initio DFT method while being able to generalize to unseen polymers.

generative

Title: Segmenta\c{c}\~ao e contagem de troncos de madeira utilizando deep learning e processamento de imagens. (arXiv:2309.00123v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.00123
• Code URL: null
• Copy Paste: [[2309.00123]] Segmenta\c{c}\~ao e contagem de troncos de madeira utilizando deep learning e processamento de imagens(http://arxiv.org/abs/2309.00123)
• Summary:

  Counting objects in images is a pattern recognition problem that focuses on identifying an element to determine its incidence and is approached in the literature as Visual Object Counting (VOC). In this work, we propose a methodology to count wood logs. First, wood logs are segmented from the image background. This first segmentation step is obtained using the Pix2Pix framework that implements Conditional Generative Adversarial Networks (CGANs). Second, the clusters are counted using Connected Components. The average accuracy of the segmentation exceeds 89% while the average amount of wood logs identified based on total accounted is over 97%.

Title: CityDreamer: Compositional Generative Model of Unbounded 3D Cities. (arXiv:2309.00610v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.00610
• Code URL: null
• Copy Paste: [[2309.00610]] CityDreamer: Compositional Generative Model of Unbounded 3D Cities(http://arxiv.org/abs/2309.00610)
• Summary:

  In recent years, extensive research has focused on 3D natural scene generation, but the domain of 3D city generation has not received as much exploration. This is due to the greater challenges posed by 3D city generation, mainly because humans are more sensitive to structural distortions in urban environments. Additionally, generating 3D cities is more complex than 3D natural scenes since buildings, as objects of the same class, exhibit a wider range of appearances compared to the relatively consistent appearance of objects like trees in natural scenes. To address these challenges, we propose CityDreamer, a compositional generative model designed specifically for unbounded 3D cities, which separates the generation of building instances from other background objects, such as roads, green lands, and water areas, into distinct modules. Furthermore, we construct two datasets, OSM and GoogleEarth, containing a vast amount of real-world city imagery to enhance the realism of the generated 3D cities both in their layouts and appearances. Through extensive experiments, CityDreamer has proven its superiority over state-of-the-art methods in generating a wide range of lifelike 3D cities.

Title: LLM in the Shell: Generative Honeypots. (arXiv:2309.00155v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.00155
• Code URL: https://github.com/stratosphereips/shellm
• Copy Paste: [[2309.00155]] LLM in the Shell: Generative Honeypots(http://arxiv.org/abs/2309.00155)
• Summary:

  Honeypots are essential tools in cybersecurity. However, most of them (even the high-interaction ones) lack the required realism to engage and fool human attackers. This limitation makes them easily discernible, hindering their effectiveness. This work introduces a novel method to create dynamic and realistic software honeypots based on Large Language Models. Preliminary results indicate that LLMs can create credible and dynamic honeypots capable of addressing important limitations of previous honeypots, such as deterministic responses, lack of adaptability, etc. We evaluated the realism of each command by conducting an experiment with human attackers who needed to say if the answer from the honeypot was fake or not. Our proposed honeypot, called shelLM, reached an accuracy rate of 0.92.

Title: Image Hijacking: Adversarial Images can Control Generative Models at Runtime. (arXiv:2309.00236v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.00236
• Code URL: null
• Copy Paste: [[2309.00236]] Image Hijacking: Adversarial Images can Control Generative Models at Runtime(http://arxiv.org/abs/2309.00236)
• Summary:

  Are foundation models secure from malicious actors? In this work, we focus on the image input to a vision-language model (VLM). We discover image hijacks, adversarial images that control generative models at runtime. We introduce Behavior Matching, a general method for creating image hijacks, and we use it to explore three types of attacks. Specific string attacks generate arbitrary output of the adversary's choosing. Leak context attacks leak information from the context window into the output. Jailbreak attacks circumvent a model's safety training. We study these attacks against LLaVA-2, a state-of-the-art VLM based on CLIP and LLaMA-2, and find that all our attack types have above a 90\% success rate. Moreover, our attacks are automated and require only small image perturbations. These findings raise serious concerns about the security of foundation models. If image hijacks are as difficult to defend against as adversarial examples in CIFAR-10, then it might be many years before a solution is found -- if it even exists.

large language model

Title: Large Content And Behavior Models To Understand, Simulate, And Optimize Content And Behavior. (arXiv:2309.00359v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.00359
• Code URL: null
• Copy Paste: [[2309.00359]] Large Content And Behavior Models To Understand, Simulate, And Optimize Content And Behavior(http://arxiv.org/abs/2309.00359)
• Summary:

  Shannon, in his seminal paper introducing information theory, divided the communication into three levels: technical, semantic, and effectivenss. While the technical level is concerned with accurate reconstruction of transmitted symbols, the semantic and effectiveness levels deal with the inferred meaning and its effect on the receiver. Thanks to telecommunications, the first level problem has produced great advances like the internet. Large Language Models (LLMs) make some progress towards the second goal, but the third level still remains largely untouched. The third problem deals with predicting and optimizing communication for desired receiver behavior. LLMs, while showing wide generalization capabilities across a wide range of tasks, are unable to solve for this. One reason for the underperformance could be a lack of "behavior tokens" in LLMs' training corpora. Behavior tokens define receiver behavior over a communication, such as shares, likes, clicks, purchases, retweets, etc. While preprocessing data for LLM training, behavior tokens are often removed from the corpora as noise. Therefore, in this paper, we make some initial progress towards reintroducing behavior tokens in LLM training. The trained models, other than showing similar performance to LLMs on content understanding tasks, show generalization capabilities on behavior simulation, content simulation, behavior understanding, and behavior domain adaptation. Using a wide range of tasks on two corpora, we show results on all these capabilities. We call these models Large Content and Behavior Models (LCBMs). Further, to spur more research on LCBMs, we release our new Content Behavior Corpus (CBC), a repository containing communicator, message, and corresponding receiver behavior.

Title: Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Following. (arXiv:2309.00615v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.00615
• Code URL: https://github.com/ziyuguo99/point-bind_point-llm
• Copy Paste: [[2309.00615]] Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Following(http://arxiv.org/abs/2309.00615)
• Summary:

  We introduce Point-Bind, a 3D multi-modality model aligning point clouds with 2D image, language, audio, and video. Guided by ImageBind, we construct a joint embedding space between 3D and multi-modalities, enabling many promising applications, e.g., any-to-3D generation, 3D embedding arithmetic, and 3D open-world understanding. On top of this, we further present Point-LLM, the first 3D large language model (LLM) following 3D multi-modal instructions. By parameter-efficient fine-tuning techniques, Point-LLM injects the semantics of Point-Bind into pre-trained LLMs, e.g., LLaMA, which requires no 3D instruction data, but exhibits superior 3D and multi-modal question-answering capacity. We hope our work may cast a light on the community for extending 3D point clouds to multi-modality applications. Code is available at https://github.com/ZiyuGuo99/Point-Bind_Point-LLM.

Title: YaRN: Efficient Context Window Extension of Large Language Models. (arXiv:2309.00071v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.00071
• Code URL: https://github.com/jquesnelle/yarn
• Copy Paste: [[2309.00071]] YaRN: Efficient Context Window Extension of Large Language Models(http://arxiv.org/abs/2309.00071)
• Summary:

  Rotary Position Embeddings (RoPE) have been shown to effectively encode positional information in transformer-based language models. However, these models fail to generalize past the sequence length they were trained on. We present YaRN (Yet another RoPE extensioN method), a compute-efficient method to extend the context window of such models, requiring 10x less tokens and 2.5x less training steps than previous methods. Using YaRN, we show that LLaMA models can effectively utilize and extrapolate to context lengths much longer than their original pre-training would allow, while also surpassing previous the state-of-the-art at context window extension. In addition, we demonstrate that YaRN exhibits the capability to extrapolate beyond the limited context of a fine-tuning dataset. We publish the checkpoints of Llama 2 7B/13B fine-tuned using YaRN with 64k and 128k context windows at https://github.com/jquesnelle/yarn

Title: Large language models in medicine: the potentials and pitfalls. (arXiv:2309.00087v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.00087
• Code URL: null
• Copy Paste: [[2309.00087]] Large language models in medicine: the potentials and pitfalls(http://arxiv.org/abs/2309.00087)
• Summary:

  Large language models (LLMs) have been applied to tasks in healthcare, ranging from medical exam questions to responding to patient questions. With increasing institutional partnerships between companies producing LLMs and healthcare systems, real world clinical application is coming closer to reality. As these models gain traction, it is essential for healthcare practitioners to understand what LLMs are, their development, their current and potential applications, and the associated pitfalls when utilized in medicine. This review and accompanying tutorial aim to give an overview of these topics to aid healthcare practitioners in understanding the rapidly changing landscape of LLMs as applied to medicine.

Title: Large Language Models for Semantic Monitoring of Corporate Disclosures: A Case Study on Korea's Top 50 KOSPI Companies. (arXiv:2309.00208v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.00208
• Code URL: null
• Copy Paste: [[2309.00208]] Large Language Models for Semantic Monitoring of Corporate Disclosures: A Case Study on Korea's Top 50 KOSPI Companies(http://arxiv.org/abs/2309.00208)
• Summary:

  In the rapidly advancing domain of artificial intelligence, state-of-the-art language models such as OpenAI's GPT-3.5-turbo and GPT-4 offer unprecedented opportunities for automating complex tasks. This research paper delves into the capabilities of these models for semantically analyzing corporate disclosures in the Korean context, specifically for timely disclosure. The study focuses on the top 50 publicly traded companies listed on the Korean KOSPI, based on market capitalization, and scrutinizes their monthly disclosure summaries over a period of 17 months. Each summary was assigned a sentiment rating on a scale ranging from 1(very negative) to 5(very positive). To gauge the effectiveness of the language models, their sentiment ratings were compared with those generated by human experts. Our findings reveal a notable performance disparity between GPT-3.5-turbo and GPT-4, with the latter demonstrating significant accuracy in human evaluation tests. The Spearman correlation coefficient was registered at 0.61, while the simple concordance rate was recorded at 0.82. This research contributes valuable insights into the evaluative characteristics of GPT models, thereby laying the groundwork for future innovations in the field of automated semantic monitoring.

Title: Publicly Shareable Clinical Large Language Model Built on Synthetic Clinical Notes. (arXiv:2309.00237v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.00237
• Code URL: https://github.com/starmpcc/asclepius
• Copy Paste: [[2309.00237]] Publicly Shareable Clinical Large Language Model Built on Synthetic Clinical Notes(http://arxiv.org/abs/2309.00237)
• Summary:

  The development of large language models tailored for handling patients' clinical notes is often hindered by the limited accessibility and usability of these notes due to strict privacy regulations. To address these challenges, we first create synthetic large-scale clinical notes using publicly available case reports extracted from biomedical literature. We then use these synthetic notes to train our specialized clinical large language model, Asclepius. While Asclepius is trained on synthetic data, we assess its potential performance in real-world applications by evaluating it using real clinical notes. We benchmark Asclepius against several other large language models, including GPT-3.5-turbo and other open-source alternatives. To further validate our approach using synthetic notes, we also compare Asclepius with its variants trained on real clinical notes. Our findings convincingly demonstrate that synthetic clinical notes can serve as viable substitutes for real ones when constructing high-performing clinical language models. This conclusion is supported by detailed evaluations conducted by both GPT-4 and medical professionals. All resources including weights, codes, and data used in the development of Asclepius are made publicly accessible for future research.

Title: FactLLaMA: Optimizing Instruction-Following Language Models with External Knowledge for Automated Fact-Checking. (arXiv:2309.00240v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.00240
• Code URL: null
• Copy Paste: [[2309.00240]] FactLLaMA: Optimizing Instruction-Following Language Models with External Knowledge for Automated Fact-Checking(http://arxiv.org/abs/2309.00240)
• Summary:

  Automatic fact-checking plays a crucial role in combating the spread of misinformation. Large Language Models (LLMs) and Instruction-Following variants, such as InstructGPT and Alpaca, have shown remarkable performance in various natural language processing tasks. However, their knowledge may not always be up-to-date or sufficient, potentially leading to inaccuracies in fact-checking. To address this limitation, we propose combining the power of instruction-following language models with external evidence retrieval to enhance fact-checking performance. Our approach involves leveraging search engines to retrieve relevant evidence for a given input claim. This external evidence serves as valuable supplementary information to augment the knowledge of the pretrained language model. Then, we instruct-tune an open-sourced language model, called LLaMA, using this evidence, enabling it to predict the veracity of the input claim more accurately. To evaluate our method, we conducted experiments on two widely used fact-checking datasets: RAWFC and LIAR. The results demonstrate that our approach achieves state-of-the-art performance in fact-checking tasks. By integrating external evidence, we bridge the gap between the model's knowledge and the most up-to-date and sufficient context available, leading to improved fact-checking outcomes. Our findings have implications for combating misinformation and promoting the dissemination of accurate information on online platforms. Our released materials are accessible at: https://thcheung.github.io/factllama.

Title: RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback. (arXiv:2309.00267v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.00267
• Code URL: null
• Copy Paste: [[2309.00267]] RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback(http://arxiv.org/abs/2309.00267)
• Summary:

  Reinforcement learning from human feedback (RLHF) is effective at aligning large language models (LLMs) to human preferences, but gathering high quality human preference labels is a key bottleneck. We conduct a head-to-head comparison of RLHF vs. RL from AI Feedback (RLAIF) - a technique where preferences are labeled by an off-the-shelf LLM in lieu of humans, and we find that they result in similar improvements. On the task of summarization, human evaluators prefer generations from both RLAIF and RLHF over a baseline supervised fine-tuned model in ~70% of cases. Furthermore, when asked to rate RLAIF vs. RLHF summaries, humans prefer both at equal rates. These results suggest that RLAIF can yield human-level performance, offering a potential solution to the scalability limitations of RLHF.

Title: SortedNet, a Place for Every Network and Every Network in its Place: Towards a Generalized Solution for Training Many-in-One Neural Networks. (arXiv:2309.00255v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.00255
• Code URL: null
• Copy Paste: [[2309.00255]] SortedNet, a Place for Every Network and Every Network in its Place: Towards a Generalized Solution for Training Many-in-One Neural Networks(http://arxiv.org/abs/2309.00255)
• Summary:

  As the size of deep learning models continues to grow, finding optimal models under memory and computation constraints becomes increasingly more important. Although usually the architecture and constituent building blocks of neural networks allow them to be used in a modular way, their training process is not aware of this modularity. Consequently, conventional neural network training lacks the flexibility to adapt the computational load of the model during inference. This paper proposes SortedNet, a generalized and scalable solution to harness the inherent modularity of deep neural networks across various dimensions for efficient dynamic inference. Our training considers a nested architecture for the sub-models with shared parameters and trains them together with the main model in a sorted and probabilistic manner. This sorted training of sub-networks enables us to scale the number of sub-networks to hundreds using a single round of training. We utilize a novel updating scheme during training that combines random sampling of sub-networks with gradient accumulation to improve training efficiency. Furthermore, the sorted nature of our training leads to a search-free sub-network selection at inference time; and the nested architecture of the resulting sub-networks leads to minimal storage requirement and efficient switching between sub-networks at inference. Our general dynamic training approach is demonstrated across various architectures and tasks, including large language models and pre-trained vision models. Experimental results show the efficacy of the proposed approach in achieving efficient sub-networks while outperforming state-of-the-art dynamic training approaches. Our findings demonstrate the feasibility of training up to 160 different sub-models simultaneously, showcasing the extensive scalability of our proposed method while maintaining 96% of the model performance.

segmentation

Title: Vision-Based Cranberry Crop Ripening Assessment. (arXiv:2309.00028v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.00028
• Code URL: null
• Copy Paste: [[2309.00028]] Vision-Based Cranberry Crop Ripening Assessment(http://arxiv.org/abs/2309.00028)
• Summary:

  Agricultural domains are being transformed by recent advances in AI and computer vision that support quantitative visual evaluation. Using drone imaging, we develop a framework for characterizing the ripening process of cranberry crops. Our method consists of drone-based time-series collection over a cranberry growing season, photometric calibration for albedo recovery from pixels, and berry segmentation with semi-supervised deep learning networks using point-click annotations. By extracting time-series berry albedo measurements, we evaluate four different varieties of cranberries and provide a quantification of their ripening rates. Such quantification has practical implications for 1) assessing real-time overheating risks for cranberry bogs; 2) large scale comparisons of progeny in crop breeding; 3) detecting disease by looking for ripening pattern outliers. This work is the first of its kind in quantitative evaluation of ripening using computer vision methods and has impact beyond cranberry crops including wine grapes, olives, blueberries, and maize.

Title: Bellybutton: Accessible and Customizable Deep-Learning Image Segmentation. (arXiv:2309.00058v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.00058
• Code URL: null
• Copy Paste: [[2309.00058]] Bellybutton: Accessible and Customizable Deep-Learning Image Segmentation(http://arxiv.org/abs/2309.00058)
• Summary:

  The conversion of raw images into quantifiable data can be a major hurdle in experimental research, and typically involves identifying region(s) of interest, a process known as segmentation. Machine learning tools for image segmentation are often specific to a set of tasks, such as tracking cells, or require substantial compute or coding knowledge to train and use. Here we introduce an easy-to-use (no coding required), image segmentation method, using a 15-layer convolutional neural network that can be trained on a laptop: Bellybutton. The algorithm trains on user-provided segmentation of example images, but, as we show, just one or even a portion of one training image can be sufficient in some cases. We detail the machine learning method and give three use cases where Bellybutton correctly segments images despite substantial lighting, shape, size, focus, and/or structure variation across the regions(s) of interest. Instructions for easy download and use, with further details and the datasets used in this paper are available at pypi.org/project/Bellybuttonseg.

Title: Open-Vocabulary Semantic Segmentation via Attribute Decomposition-Aggregation. (arXiv:2309.00096v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.00096
• Code URL: null
• Copy Paste: [[2309.00096]] Open-Vocabulary Semantic Segmentation via Attribute Decomposition-Aggregation(http://arxiv.org/abs/2309.00096)
• Summary:

  Open-vocabulary semantic segmentation is a challenging task that requires segmenting novel object categories at inference time. Recent works explore vision-language pre-training to handle this task, but suffer from unrealistic assumptions in practical scenarios, i.e., low-quality textual category names. For example, this paradigm assumes that new textual categories will be accurately and completely provided, and exist in lexicons during pre-training. However, exceptions often happen when meet with ambiguity for brief or incomplete names, new words that are not present in the pre-trained lexicons, and difficult-to-describe categories for users. To address these issues, this work proposes a novel decomposition-aggregation framework, inspired by human cognition in understanding new concepts. Specifically, in the decomposition stage, we decouple class names into diverse attribute descriptions to enrich semantic contexts. Two attribute construction strategies are designed: using large language models for common categories, and involving manually labelling for human-invented categories. In the aggregation stage, we group diverse attributes into an integrated global description, to form a discriminative classifier that distinguishes the target object from others. One hierarchical aggregation is further designed to achieve multi-level alignment and deep fusion between vision and text. The final result is obtained by computing the embedding similarity between aggregated attributes and images. To evaluate the effectiveness, we annotate three datasets with attribute descriptions, and conduct extensive experiments and ablation studies. The results show the superior performance of attribute decomposition-aggregation.

Title: Beyond Self-Attention: Deformable Large Kernel Attention for Medical Image Segmentation. (arXiv:2309.00121v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.00121
• Code URL: null
• Copy Paste: [[2309.00121]] Beyond Self-Attention: Deformable Large Kernel Attention for Medical Image Segmentation(http://arxiv.org/abs/2309.00121)
• Summary:

  Medical image segmentation has seen significant improvements with transformer models, which excel in grasping far-reaching contexts and global contextual information. However, the increasing computational demands of these models, proportional to the squared token count, limit their depth and resolution capabilities. Most current methods process D volumetric image data slice-by-slice (called pseudo 3D), missing crucial inter-slice information and thus reducing the model's overall performance. To address these challenges, we introduce the concept of \textbf{Deformable Large Kernel Attention (D-LKA Attention)}, a streamlined attention mechanism employing large convolution kernels to fully appreciate volumetric context. This mechanism operates within a receptive field akin to self-attention while sidestepping the computational overhead. Additionally, our proposed attention mechanism benefits from deformable convolutions to flexibly warp the sampling grid, enabling the model to adapt appropriately to diverse data patterns. We designed both 2D and 3D adaptations of the D-LKA Attention, with the latter excelling in cross-depth data understanding. Together, these components shape our novel hierarchical Vision Transformer architecture, the \textit{D-LKA Net}. Evaluations of our model against leading methods on popular medical segmentation datasets (Synapse, NIH Pancreas, and Skin lesion) demonstrate its superior performance. Our code implementation is publicly available at the: https://github.com/mindflow-institue/deformableLKA

Title: Self-supervised Semantic Segmentation: Consistency over Transformation. (arXiv:2309.00143v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.00143
• Code URL: https://github.com/mindflow-institue/ssct
• Copy Paste: [[2309.00143]] Self-supervised Semantic Segmentation: Consistency over Transformation(http://arxiv.org/abs/2309.00143)
• Summary:

  Accurate medical image segmentation is of utmost importance for enabling automated clinical decision procedures. However, prevailing supervised deep learning approaches for medical image segmentation encounter significant challenges due to their heavy dependence on extensive labeled training data. To tackle this issue, we propose a novel self-supervised algorithm, \textbf{S$^3$-Net}, which integrates a robust framework based on the proposed Inception Large Kernel Attention (I-LKA) modules. This architectural enhancement makes it possible to comprehensively capture contextual information while preserving local intricacies, thereby enabling precise semantic segmentation. Furthermore, considering that lesions in medical images often exhibit deformations, we leverage deformable convolution as an integral component to effectively capture and delineate lesion deformations for superior object boundary definition. Additionally, our self-supervised strategy emphasizes the acquisition of invariance to affine transformations, which is commonly encountered in medical scenarios. This emphasis on robustness with respect to geometric distortions significantly enhances the model's ability to accurately model and handle such distortions. To enforce spatial consistency and promote the grouping of spatially connected image pixels with similar feature representations, we introduce a spatial consistency loss term. This aids the network in effectively capturing the relationships among neighboring pixels and enhancing the overall segmentation quality. The S$^3$-Net approach iteratively learns pixel-level feature representations for image content clustering in an end-to-end manner. Our experimental results on skin lesion and lung organ segmentation tasks show the superior performance of our method compared to the SOTA approaches. https://github.com/mindflow-institue/SSCT

Title: DARC: Distribution-Aware Re-Coloring Model for Generalizable Nucleus Segmentation. (arXiv:2309.00188v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.00188
• Code URL: null
• Copy Paste: [[2309.00188]] DARC: Distribution-Aware Re-Coloring Model for Generalizable Nucleus Segmentation(http://arxiv.org/abs/2309.00188)
• Summary:

  Nucleus segmentation is usually the first step in pathological image analysis tasks. Generalizable nucleus segmentation refers to the problem of training a segmentation model that is robust to domain gaps between the source and target domains. The domain gaps are usually believed to be caused by the varied image acquisition conditions, e.g., different scanners, tissues, or staining protocols. In this paper, we argue that domain gaps can also be caused by different foreground (nucleus)-background ratios, as this ratio significantly affects feature statistics that are critical to normalization layers. We propose a Distribution-Aware Re-Coloring (DARC) model that handles the above challenges from two perspectives. First, we introduce a re-coloring method that relieves dramatic image color variations between different domains. Second, we propose a new instance normalization method that is robust to the variation in foreground-background ratios. We evaluate the proposed methods on two H$\&$E stained image datasets, named CoNSeP and CPM17, and two IHC stained image datasets, called DeepLIIF and BC-DeepLIIF. Extensive experimental results justify the effectiveness of our proposed DARC model. Codes are available at \url{https://github.com/csccsccsccsc/DARC

Title: Gap and Overlap Detection in Automated Fiber Placement. (arXiv:2309.00206v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.00206
• Code URL: null
• Copy Paste: [[2309.00206]] Gap and Overlap Detection in Automated Fiber Placement(http://arxiv.org/abs/2309.00206)
• Summary:

  The identification and correction of manufacturing defects, particularly gaps and overlaps, are crucial for ensuring high-quality composite parts produced through Automated Fiber Placement (AFP). These imperfections are the most commonly observed issues that can significantly impact the overall quality of the composite parts. Manual inspection is both time-consuming and labor-intensive, making it an inefficient approach. To overcome this challenge, the implementation of an automated defect detection system serves as the optimal solution. In this paper, we introduce a novel method that uses an Optical Coherence Tomography (OCT) sensor and computer vision techniques to detect and locate gaps and overlaps in composite parts. Our approach involves generating a depth map image of the composite surface that highlights the elevation of composite tapes (or tows) on the surface. By detecting the boundaries of each tow, our algorithm can compare consecutive tows and identify gaps or overlaps that may exist between them. Any gaps or overlaps exceeding a predefined tolerance threshold are considered manufacturing defects. To evaluate the performance of our approach, we compare the detected defects with the ground truth annotated by experts. The results demonstrate a high level of accuracy and efficiency in gap and overlap segmentation.

Title: Dense Voxel 3D Reconstruction Using a Monocular Event Camera. (arXiv:2309.00385v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.00385
• Code URL: null
• Copy Paste: [[2309.00385]] Dense Voxel 3D Reconstruction Using a Monocular Event Camera(http://arxiv.org/abs/2309.00385)
• Summary:

  Event cameras are sensors inspired by biological systems that specialize in capturing changes in brightness. These emerging cameras offer many advantages over conventional frame-based cameras, including high dynamic range, high frame rates, and extremely low power consumption. Due to these advantages, event cameras have increasingly been adapted in various fields, such as frame interpolation, semantic segmentation, odometry, and SLAM. However, their application in 3D reconstruction for VR applications is underexplored. Previous methods in this field mainly focused on 3D reconstruction through depth map estimation. Methods that produce dense 3D reconstruction generally require multiple cameras, while methods that utilize a single event camera can only produce a semi-dense result. Other single-camera methods that can produce dense 3D reconstruction rely on creating a pipeline that either incorporates the aforementioned methods or other existing Structure from Motion (SfM) or Multi-view Stereo (MVS) methods. In this paper, we propose a novel approach for solving dense 3D reconstruction using only a single event camera. To the best of our knowledge, our work is the first attempt in this regard. Our preliminary results demonstrate that the proposed method can produce visually distinguishable dense 3D reconstructions directly without requiring pipelines like those used by existing methods. Additionally, we have created a synthetic dataset with $39,739$ object scans using an event camera simulator. This dataset will help accelerate other relevant research in this field.

Title: Unsupervised bias discovery in medical image segmentation. (arXiv:2309.00451v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.00451
• Code URL: null
• Copy Paste: [[2309.00451]] Unsupervised bias discovery in medical image segmentation(http://arxiv.org/abs/2309.00451)
• Summary:

  It has recently been shown that deep learning models for anatomical segmentation in medical images can exhibit biases against certain sub-populations defined in terms of protected attributes like sex or ethnicity. In this context, auditing fairness of deep segmentation models becomes crucial. However, such audit process generally requires access to ground-truth segmentation masks for the target population, which may not always be available, especially when going from development to deployment. Here we propose a new method to anticipate model biases in biomedical image segmentation in the absence of ground-truth annotations. Our unsupervised bias discovery method leverages the reverse classification accuracy framework to estimate segmentation quality. Through numerical experiments in synthetic and realistic scenarios we show how our method is able to successfully anticipate fairness issues in the absence of ground-truth labels, constituting a novel and valuable tool in this field.

Title: dacl10k: Benchmark for Semantic Bridge Damage Segmentation. (arXiv:2309.00460v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.00460
• Code URL: null
• Copy Paste: [[2309.00460]] dacl10k: Benchmark for Semantic Bridge Damage Segmentation(http://arxiv.org/abs/2309.00460)
• Summary:

  Reliably identifying reinforced concrete defects (RCDs)plays a crucial role in assessing the structural integrity, traffic safety, and long-term durability of concrete bridges, which represent the most common bridge type worldwide. Nevertheless, available datasets for the recognition of RCDs are small in terms of size and class variety, which questions their usability in real-world scenarios and their role as a benchmark. Our contribution to this problem is "dacl10k", an exceptionally diverse RCD dataset for multi-label semantic segmentation comprising 9,920 images deriving from real-world bridge inspections. dacl10k distinguishes 12 damage classes as well as 6 bridge components that play a key role in the building assessment and recommending actions, such as restoration works, traffic load limitations or bridge closures. In addition, we examine baseline models for dacl10k which are subsequently evaluated. The best model achieves a mean intersection-over-union of 0.42 on the test set. dacl10k, along with our baselines, will be openly accessible to researchers and practitioners, representing the currently biggest dataset regarding number of images and class diversity for semantic segmentation in the bridge inspection domain.

Title: An Improved Encoder-Decoder Framework for Food EnergyEstimation. (arXiv:2309.00468v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.00468
• Code URL: null
• Copy Paste: [[2309.00468]] An Improved Encoder-Decoder Framework for Food EnergyEstimation(http://arxiv.org/abs/2309.00468)
• Summary:

  Dietary assessment is essential to maintaining a healthy lifestyle. Automatic image-based dietary assessment is a growing field of research due to the increasing prevalence of image capturing devices (e.g. mobile phones). In this work, we estimate food energy from a single monocular image, a difficult task due to the limited hard-to-extract amount of energy information present in an image. To do so, we employ an improved encoder-decoder framework for energy estimation; the encoder transforms the image into a representation embedded with food energy information in an easier-to-extract format, which the decoder then extracts the energy information from. To implement our method, we compile a high-quality food image dataset verified by registered dietitians containing eating scene images, food-item segmentation masks, and ground truth calorie values. Our method improves upon previous caloric estimation methods by over 10\% and 30 kCal in terms of MAPE and MAE respectively.

Title: OpenIns3D: Snap and Lookup for 3D Open-vocabulary Instance Segmentation. (arXiv:2309.00616v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.00616
• Code URL: https://github.com/Pointcept/OpenIns3D
• Copy Paste: [[2309.00616]] OpenIns3D: Snap and Lookup for 3D Open-vocabulary Instance Segmentation(http://arxiv.org/abs/2309.00616)
• Summary:

  Current 3D open-vocabulary scene understanding methods mostly utilize well-aligned 2D images as the bridge to learn 3D features with language. However, applying these approaches becomes challenging in scenarios where 2D images are absent. In this work, we introduce a completely new pipeline, namely, OpenIns3D, which requires no 2D image inputs, for 3D open-vocabulary scene understanding at the instance level. The OpenIns3D framework employs a "Mask-Snap-Lookup" scheme. The "Mask" module learns class-agnostic mask proposals in 3D point clouds. The "Snap" module generates synthetic scene-level images at multiple scales and leverages 2D vision language models to extract interesting objects. The "Lookup" module searches through the outcomes of "Snap" with the help of Mask2Pixel maps, which contain the precise correspondence between 3D masks and synthetic images, to assign category names to the proposed masks. This 2D input-free, easy-to-train, and flexible approach achieved state-of-the-art results on a wide range of indoor and outdoor datasets with a large margin. Furthermore, OpenIns3D allows for effortless switching of 2D detectors without re-training. When integrated with state-of-the-art 2D open-world models such as ODISE and GroundingDINO, superb results are observed on open-vocabulary instance segmentation. When integrated with LLM-powered 2D models like LISA, it demonstrates a remarkable capacity to process highly complex text queries, including those that require intricate reasoning and world knowledge. The code and model will be made publicly available.