secure

Title: VCTP: A Verifiable Credential-based Trust Propagation Protocol for Personal Issuers in Self-Sovereign Identity Platforms. (arXiv:2308.01539v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.01539
• Code URL: null
• Copy Paste: [[2308.01539]] VCTP: A Verifiable Credential-based Trust Propagation Protocol for Personal Issuers in Self-Sovereign Identity Platforms(http://arxiv.org/abs/2308.01539)
• Summary:

  Self Sovereign Identity (SSI) is an emerging identity system that facilitates secure credential issuance and verification without placing trust in any centralised authority. To bypass central trust, most SSI implementations place blockchain as a trusted mediator by placing credential transactions on-chain. Yet, existing SSI platforms face trust issues as all credential issuers in SSI are not supported with adequate trust. Current SSI solutions provide trust support to the officiated issuers (e.g., government agencies), who must follow a precise process to assess their credentials. However, there is no structured trust support for individuals of SSI who may attempt to issue a credential (e.g., letter of consent) in the context of business processes. Therefore, some risk-averse verifiers in the system may not accept the credentials from individual issuers to avoid carrying the cost of mishaps from potentially inadmissible credentials without reliance on a trusted agency. This paper proposes a trust propagation protocol that supports individual users to be trusted as verifiable issuers in the SSI platform by establishing a trust propagation credential template in the blockchain. Our approach utilises (i) the sanitizable signature scheme to propagate the required trust to an individual issuer, (ii) a voting mechanism to minimises the possibility of collusion. Our implementation demonstrates that the solution is both practical and performs well under varying system loads.

security

Title: The ACAC_D Model for Mutable Activity Control and Chain of Dependencies in Smart and Collaborative Systems. (arXiv:2308.01783v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.01783
• Code URL: null
• Copy Paste: [[2308.01783]] The ACAC_D Model for Mutable Activity Control and Chain of Dependencies in Smart and Collaborative Systems(http://arxiv.org/abs/2308.01783)
• Summary:

  With the integration of connected devices, artificial intelligence, and heterogeneous networks in IoT-driven cyber-physical systems, our society is evolving as a smart, automated, and connected community. In such dynamic and distributed environments, various operations are carried out considering different contextual factors to support the automation of collaborative devices and systems. These devices often perform long-lived operations or tasks (referred to as activities) to fulfill larger goals in the collaborative environment. These activities are usually mutable (change states) and interdependent. They can influence the execution of other activities in the ecosystem, requiring active and real-time monitoring of the entire connected environment.

  Recently, a vision for activity-centric access control(ACAC) was proposed to enable security modeling and enforcement from the perspective and abstraction of interdependent activities. The proposed ACAC incorporates four decision parameters: Authorizations(A), oBligations(B), Conditions(C), and activity Dependencies(D) for an object agnostic access control in smart systems. In this paper, we take a step further towards maturing ACAC by focusing on activity dependencies(D) and developing a family of formal mathematically grounded models, referred to as ACAC_D. These formal models consider the real-time mutability of activities in resolving active dependencies among various activities in the ecosystem. Activity dependencies can form a chain where it is possible to have dependencies of dependencies. In ACAC, we also consider the chain of dependencies while handling the mutability of an activity. We highlight the challenges while dealing with chain of dependencies, and provide solutions to resolve these challenges. We also present a proof of concept implementation of with performance analysis for a smart farming use case.

privacy

Title: VertexSerum: Poisoning Graph Neural Networks for Link Inference. (arXiv:2308.01469v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.01469
• Code URL: null
• Copy Paste: [[2308.01469]] VertexSerum: Poisoning Graph Neural Networks for Link Inference(http://arxiv.org/abs/2308.01469)
• Summary:

  Graph neural networks (GNNs) have brought superb performance to various applications utilizing graph structural data, such as social analysis and fraud detection. The graph links, e.g., social relationships and transaction history, are sensitive and valuable information, which raises privacy concerns when using GNNs. To exploit these vulnerabilities, we propose VertexSerum, a novel graph poisoning attack that increases the effectiveness of graph link stealing by amplifying the link connectivity leakage. To infer node adjacency more accurately, we propose an attention mechanism that can be embedded into the link detection network. Our experiments demonstrate that VertexSerum significantly outperforms the SOTA link inference attack, improving the AUC scores by an average of $9.8\%$ across four real-world datasets and three different GNN structures. Furthermore, our experiments reveal the effectiveness of VertexSerum in both black-box and online learning settings, further validating its applicability in real-world scenarios.

Title: Towards Fair and Privacy Preserving Federated Learning for the Healthcare Domain. (arXiv:2308.01529v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.01529
• Code URL: null
• Copy Paste: [[2308.01529]] Towards Fair and Privacy Preserving Federated Learning for the Healthcare Domain(http://arxiv.org/abs/2308.01529)
• Summary:

  Federated learning enables data sharing in healthcare contexts where it might otherwise be difficult due to data-use-ordinances or security and communication constraints. Distributed and shared data models allow models to become generalizable and learn from heterogeneous clients. While addressing data security, privacy, and vulnerability considerations, data itself is not shared across nodes in a given learning network. On the other hand, FL models often struggle with variable client data distributions and operate on an assumption of independent and identically distributed data. As the field has grown, the notion of fairness-aware federated learning mechanisms has also been introduced and is of distinct significance to the healthcare domain where many sensitive groups and protected classes exist. In this paper, we create a benchmark methodology for FAFL mechanisms under various heterogeneous conditions on datasets in the healthcare domain typically outside the scope of current federated learning benchmarks, such as medical imaging and waveform data formats. Our results indicate considerable variation in how various FAFL schemes respond to high levels of data heterogeneity. Additionally, doing so under privacy-preserving conditions can create significant increases in network communication cost and latency compared to the typical federated learning scheme.

protect

Title: Computational Long Exposure Mobile Photography. (arXiv:2308.01379v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.01379
• Code URL: null
• Copy Paste: [[2308.01379]] Computational Long Exposure Mobile Photography(http://arxiv.org/abs/2308.01379)
• Summary:

  Long exposure photography produces stunning imagery, representing moving elements in a scene with motion-blur. It is generally employed in two modalities, producing either a foreground or a background blur effect. Foreground blur images are traditionally captured on a tripod-mounted camera and portray blurred moving foreground elements, such as silky water or light trails, over a perfectly sharp background landscape. Background blur images, also called panning photography, are captured while the camera is tracking a moving subject, to produce an image of a sharp subject over a background blurred by relative motion. Both techniques are notoriously challenging and require additional equipment and advanced skills. In this paper, we describe a computational burst photography system that operates in a hand-held smartphone camera app, and achieves these effects fully automatically, at the tap of the shutter button. Our approach first detects and segments the salient subject. We track the scene motion over multiple frames and align the images in order to preserve desired sharpness and to produce aesthetically pleasing motion streaks. We capture an under-exposed burst and select the subset of input frames that will produce blur trails of controlled length, regardless of scene or camera motion velocity. We predict inter-frame motion and synthesize motion-blur to fill the temporal gaps between the input frames. Finally, we composite the blurred image with the sharp regular exposure to protect the sharpness of faces or areas of the scene that are barely moving, and produce a final high resolution and high dynamic range (HDR) photograph. Our system democratizes a capability previously reserved to professionals, and makes this creative style accessible to most casual photographers.

  More information and supplementary material can be found on our project webpage: https://motion-mode.github.io/

Title: Decentralized Translator of Trust: Supporting Heterogeneous TEE for Critical Infrastructure Protection. (arXiv:2308.01474v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.01474
• Code URL: null
• Copy Paste: [[2308.01474]] Decentralized Translator of Trust: Supporting Heterogeneous TEE for Critical Infrastructure Protection(http://arxiv.org/abs/2308.01474)
• Summary:

  Trusted execution environment (TEE) technology has found many applications in mitigating various security risks in an efficient manner, which is attractive for critical infrastructure protection. First, the natural of critical infrastructure requires it to be well protected from various cyber attacks. Second, performance is usually important for critical infrastructure and it cannot afford an expensive protection mechanism. While a large number of TEE-based critical infrastructure protection systems have been proposed to address various security challenges (e.g., secure sensing and reliable control), most existing works ignore one important feature, i.e., devices comprised the critical infrastructure may be equipped with multiple incompatible TEE technologies and belongs to different owners. This feature makes it hard for these devices to establish mutual trust and form a unified TEE environment. To address these challenges and fully unleash the potential of TEE technology for critical infrastructure protection, we propose DHTee, a decentralized coordination mechanism. DHTee uses blockchain technology to support key TEE functions in a heterogeneous TEE environment, especially the attestation service. A Device equipped with one TEE can interact securely with the blockchain to verify whether another potential collaborating device claiming to have a different TEE meets the security requirements. DHTee is also flexible and can support new TEE schemes without affecting devices using existing TEEs that have been supported by the system.

defense

attack

Title: Erase and Repair: An Efficient Box-Free Removal Attack on High-Capacity Deep Hiding. (arXiv:2308.01512v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.01512
• Code URL: null
• Copy Paste: [[2308.01512]] Erase and Repair: An Efficient Box-Free Removal Attack on High-Capacity Deep Hiding(http://arxiv.org/abs/2308.01512)
• Summary:

  Deep hiding, embedding images with others using deep neural networks, has demonstrated impressive efficacy in increasing the message capacity and robustness of secret sharing. In this paper, we challenge the robustness of existing deep hiding schemes by preventing the recovery of secret images, building on our in-depth study of state-of-the-art deep hiding schemes and their vulnerabilities. Leveraging our analysis, we first propose a simple box-free removal attack on deep hiding that does not require any prior knowledge of the deep hiding schemes.

  To improve the removal performance on the deep hiding schemes that may be enhanced by adversarial training, we further design a more powerful removal attack, efficient box-free removal attack (EBRA), which employs image inpainting techniques to remove secret images from container images. In addition, to ensure the effectiveness of our attack and preserve the fidelity of the processed container images, we design an erasing phase based on the locality of deep hiding to remove secret information and then make full use of the visual information of container images to repair the erased visual content. Extensive evaluations show our method can completely remove secret images from container images with negligible impact on the quality of container images.

robust

Title: LiDAR View Synthesis for Robust Vehicle Navigation Without Expert Labels. (arXiv:2308.01424v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.01424
• Code URL: null
• Copy Paste: [[2308.01424]] LiDAR View Synthesis for Robust Vehicle Navigation Without Expert Labels(http://arxiv.org/abs/2308.01424)
• Summary:

  Deep learning models for self-driving cars require a diverse training dataset to safely manage critical driving scenarios on public roads. This includes having data from divergent trajectories such as the oncoming traffic lane or sidewalks. Such data would be too dangerous to collect in the real world. Data augmentation approaches have been proposed to tackle this issue using RGB images. However, solutions based on LiDAR sensors are scarce. We therefore propose an approach to synthesize additional LiDAR point clouds from novel viewpoints without having the need to physically drive at dangerous positions. The LiDAR view synthesis is done using mesh reconstruction and ray casting. We train a deep learning model, which takes a LiDAR scan as input and predicts the future trajectory as output. A waypoint controller is then applied on this predicted trajectory to determine the throttle and steering labels of the ego-vehicle. Our method neither requires expert driving labels for the original nor for the synthesized LiDAR sequence. Instead, we infer labels from LiDAR odometry. We demonstrate the effectiveness of our approach in a comprehensive online evaluation and with a comparison to concurrent work. Our results show the importance of synthesizing additional LiDAR point clouds, particularly in terms of model robustness. Code and supplementary visualizations are available at https://jonathsch.github.io/lidar-synthesis/ .

Title: Consistency Regularization for Generalizable Source-free Domain Adaptation. (arXiv:2308.01587v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.01587
• Code URL: null
• Copy Paste: [[2308.01587]] Consistency Regularization for Generalizable Source-free Domain Adaptation(http://arxiv.org/abs/2308.01587)
• Summary:

  Source-free domain adaptation (SFDA) aims to adapt a well-trained source model to an unlabelled target domain without accessing the source dataset, making it applicable in a variety of real-world scenarios. Existing SFDA methods ONLY assess their adapted models on the target training set, neglecting the data from unseen but identically distributed testing sets. This oversight leads to overfitting issues and constrains the model's generalization ability. In this paper, we propose a consistency regularization framework to develop a more generalizable SFDA method, which simultaneously boosts model performance on both target training and testing datasets. Our method leverages soft pseudo-labels generated from weakly augmented images to supervise strongly augmented images, facilitating the model training process and enhancing the generalization ability of the adapted model. To leverage more potentially useful supervision, we present a sampling-based pseudo-label selection strategy, taking samples with severer domain shift into consideration. Moreover, global-oriented calibration methods are introduced to exploit global class distribution and feature cluster information, further improving the adaptation process. Extensive experiments demonstrate our method achieves state-of-the-art performance on several SFDA benchmarks, and exhibits robustness on unseen testing datasets.

Title: A Survey on Deep Learning-based Spatio-temporal Action Detection. (arXiv:2308.01618v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.01618
• Code URL: null
• Copy Paste: [[2308.01618]] A Survey on Deep Learning-based Spatio-temporal Action Detection(http://arxiv.org/abs/2308.01618)
• Summary:

  Spatio-temporal action detection (STAD) aims to classify the actions present in a video and localize them in space and time. It has become a particularly active area of research in computer vision because of its explosively emerging real-world applications, such as autonomous driving, visual surveillance, entertainment, etc. Many efforts have been devoted in recent years to building a robust and effective framework for STAD. This paper provides a comprehensive review of the state-of-the-art deep learning-based methods for STAD. Firstly, a taxonomy is developed to organize these methods. Next, the linking algorithms, which aim to associate the frame- or clip-level detection results together to form action tubes, are reviewed. Then, the commonly used benchmark datasets and evaluation metrics are introduced, and the performance of state-of-the-art models is compared. At last, this paper is concluded, and a set of potential research directions of STAD are discussed.

Title: QUEST: Query Stream for Vehicle-Infrastructure Cooperative Perception. (arXiv:2308.01804v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.01804
• Code URL: null
• Copy Paste: [[2308.01804]] QUEST: Query Stream for Vehicle-Infrastructure Cooperative Perception(http://arxiv.org/abs/2308.01804)
• Summary:

  Cooperative perception can effectively enhance individual perception performance by providing additional viewpoint and expanding the sensing field. Existing cooperation paradigms are either interpretable (result cooperation) or flexible (feature cooperation). In this paper, we propose the concept of query cooperation to enable interpretable instance-level flexible feature interaction. To specifically explain the concept, we propose a cooperative perception framework, termed QUEST, which let query stream flow among agents. The cross-agent queries are interacted via fusion for co-aware instances and complementation for individual unaware instances. Taking camera-based vehicle-infrastructure perception as a typical practical application scene, the experimental results on the real-world dataset, DAIR-V2X-Seq, demonstrate the effectiveness of QUEST and further reveal the advantage of the query cooperation paradigm on transmission flexibility and robustness to packet dropout. We hope our work can further facilitate the cross-agent representation interaction for better cooperative perception in practice.

Title: FROD: Robust Object Detection for Free. (arXiv:2308.01888v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.01888
• Code URL: null
• Copy Paste: [[2308.01888]] FROD: Robust Object Detection for Free(http://arxiv.org/abs/2308.01888)
• Summary:

  Object detection is a vital task in computer vision and has become an integral component of numerous critical systems. However, state-of-the-art object detectors, similar to their classification counterparts, are susceptible to small adversarial perturbations that can significantly alter their normal behavior. Unlike classification, the robustness of object detectors has not been thoroughly explored. In this work, we take the initial step towards bridging the gap between the robustness of classification and object detection by leveraging adversarially trained classification models. Merely utilizing adversarially trained models as backbones for object detection does not result in robustness. We propose effective modifications to the classification-based backbone to instill robustness in object detection without incurring any computational overhead. To further enhance the robustness achieved by the proposed modified backbone, we introduce two lightweight components: imitation loss and delayed adversarial training. Extensive experiments on the MS-COCO and Pascal VOC datasets are conducted to demonstrate the effectiveness of our proposed approach.

Title: DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scales. (arXiv:2308.01320v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.01320
• Code URL: null
• Copy Paste: [[2308.01320]] DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scales(http://arxiv.org/abs/2308.01320)
• Summary:

  ChatGPT-like models have revolutionized various applications in artificial intelligence, from summarization and coding to translation, matching or even surpassing human performance. However, the current landscape lacks an accessible, efficient, and cost-effective end-to-end RLHF (Reinforcement Learning with Human Feedback) training pipeline for these powerful models, particularly when training at the scale of billions of parameters. This paper introduces DeepSpeed-Chat, a novel system that democratizes RLHF training, making it accessible to the AI community. DeepSpeed-Chat offers three key capabilities: an easy-to-use training and inference experience for ChatGPT-like models, a DeepSpeed-RLHF pipeline that replicates the training pipeline from InstructGPT, and a robust DeepSpeed-RLHF system that combines various optimizations for training and inference in a unified way. The system delivers unparalleled efficiency and scalability, enabling training of models with hundreds of billions of parameters in record time and at a fraction of the cost. With this development, DeepSpeed-Chat paves the way for broader access to advanced RLHF training, even for data scientists with limited resources, thereby fostering innovation and further development in the field of AI.

Title: ChatMOF: An Autonomous AI System for Predicting and Generating Metal-Organic Frameworks. (arXiv:2308.01423v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.01423
• Code URL: null
• Copy Paste: [[2308.01423]] ChatMOF: An Autonomous AI System for Predicting and Generating Metal-Organic Frameworks(http://arxiv.org/abs/2308.01423)
• Summary:

  ChatMOF is an autonomous Artificial Intelligence (AI) system that is built to predict and generate of metal-organic frameworks (MOFs). By leveraging a large-scale language model (gpt-3.5-turbo), ChatMOF extracts key details from textual inputs and delivers appropriate responses, thus eliminating the necessity for rigid structured queries. The system is comprised of three core components (i.e. an agent, a toolkit, and an evaluator) and it forms a robust pipeline that manages a variety of tasks, including data retrieval, property prediction, and structure generation. The study further explores the merits and constraints of using large language models (LLMs) AI system in material sciences using and showcases its transformative potential for future advancements.

Title: NBIAS: A Natural Language Processing Framework for Bias Identification in Text. (arXiv:2308.01681v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.01681
• Code URL: null
• Copy Paste: [[2308.01681]] NBIAS: A Natural Language Processing Framework for Bias Identification in Text(http://arxiv.org/abs/2308.01681)
• Summary:

  Bias in textual data can lead to skewed interpretations and outcomes when the data is used. These biases could perpetuate stereotypes, discrimination, or other forms of unfair treatment. An algorithm trained on biased data ends up making decisions that disproportionately impact a certain group of people. Therefore, it is crucial to detect and remove these biases to ensure the fair and ethical use of data. To this end, we develop a comprehensive and robust framework \textsc{Nbias} that consists of a data layer, corpus contruction, model development layer and an evaluation layer. The dataset is constructed by collecting diverse data from various fields, including social media, healthcare, and job hiring portals. As such, we applied a transformer-based token classification model that is able to identify bias words/ phrases through a unique named entity. In the assessment procedure, we incorporate a blend of quantitative and qualitative evaluations to gauge the effectiveness of our models. We achieve accuracy improvements ranging from 1% to 8% compared to baselines. We are also able to generate a robust understanding of the model functioning, capturing not only numerical data but also the quality and intricacies of its performance. The proposed approach is applicable to a variety of biases and contributes to the fair and ethical use of textual data.

Title: URET: Universal Robustness Evaluation Toolkit (for Evasion). (arXiv:2308.01840v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.01840
• Code URL: null
• Copy Paste: [[2308.01840]] URET: Universal Robustness Evaluation Toolkit (for Evasion)(http://arxiv.org/abs/2308.01840)
• Summary:

  Machine learning models are known to be vulnerable to adversarial evasion attacks as illustrated by image classification models. Thoroughly understanding such attacks is critical in order to ensure the safety and robustness of critical AI tasks. However, most evasion attacks are difficult to deploy against a majority of AI systems because they have focused on image domain with only few constraints. An image is composed of homogeneous, numerical, continuous, and independent features, unlike many other input types to AI systems used in practice. Furthermore, some input types include additional semantic and functional constraints that must be observed to generate realistic adversarial inputs. In this work, we propose a new framework to enable the generation of adversarial inputs irrespective of the input type and task domain. Given an input and a set of pre-defined input transformations, our framework discovers a sequence of transformations that result in a semantically correct and functional adversarial input. We demonstrate the generality of our approach on several diverse machine learning tasks with various input representations. We also show the importance of generating adversarial examples as they enable the deployment of mitigation techniques.

Title: Bag of Policies for Distributional Deep Exploration. (arXiv:2308.01759v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.01759
• Code URL: null
• Copy Paste: [[2308.01759]] Bag of Policies for Distributional Deep Exploration(http://arxiv.org/abs/2308.01759)
• Summary:

  Efficient exploration in complex environments remains a major challenge for reinforcement learning (RL). Compared to previous Thompson sampling-inspired mechanisms that enable temporally extended exploration, i.e., deep exploration, we focus on deep exploration in distributional RL. We develop here a general purpose approach, Bag of Policies (BoP), that can be built on top of any return distribution estimator by maintaining a population of its copies. BoP consists of an ensemble of multiple heads that are updated independently. During training, each episode is controlled by only one of the heads and the collected state-action pairs are used to update all heads off-policy, leading to distinct learning signals for each head which diversify learning and behaviour. To test whether optimistic ensemble method can improve on distributional RL as did on scalar RL, by e.g. Bootstrapped DQN, we implement the BoP approach with a population of distributional actor-critics using Bayesian Distributional Policy Gradients (BDPG). The population thus approximates a posterior distribution of return distributions along with a posterior distribution of policies. Another benefit of building upon BDPG is that it allows to analyze global posterior uncertainty along with local curiosity bonus simultaneously for exploration. As BDPG is already an optimistic method, this pairing helps to investigate if optimism is accumulatable in distributional RL. Overall BoP results in greater robustness and speed during learning as demonstrated by our experimental results on ALE Atari games.

Title: Hard Adversarial Example Mining for Improving Robust Fairness. (arXiv:2308.01823v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.01823
• Code URL: null
• Copy Paste: [[2308.01823]] Hard Adversarial Example Mining for Improving Robust Fairness(http://arxiv.org/abs/2308.01823)
• Summary:

  Adversarial training (AT) is widely considered the state-of-the-art technique for improving the robustness of deep neural networks (DNNs) against adversarial examples (AE). Nevertheless, recent studies have revealed that adversarially trained models are prone to unfairness problems, restricting their applicability. In this paper, we empirically observe that this limitation may be attributed to serious adversarial confidence overfitting, i.e., certain adversarial examples with overconfidence. To alleviate this problem, we propose HAM, a straightforward yet effective framework via adaptive Hard Adversarial example Mining.HAM concentrates on mining hard adversarial examples while discarding the easy ones in an adaptive fashion. Specifically, HAM identifies hard AEs in terms of their step sizes needed to cross the decision boundary when calculating loss value. Besides, an early-dropping mechanism is incorporated to discard the easy examples at the initial stages of AE generation, resulting in efficient AT. Extensive experimental results on CIFAR-10, SVHN, and Imagenette demonstrate that HAM achieves significant improvement in robust fairness while reducing computational cost compared to several state-of-the-art adversarial training methods. The code will be made publicly available.

Title: Exact identification of nonlinear dynamical systems by Trimmed Lasso. (arXiv:2308.01891v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.01891
• Code URL: null
• Copy Paste: [[2308.01891]] Exact identification of nonlinear dynamical systems by Trimmed Lasso(http://arxiv.org/abs/2308.01891)
• Summary:

  Identification of nonlinear dynamical systems has been popularized by sparse identification of the nonlinear dynamics (SINDy) via the sequentially thresholded least squares (STLS) algorithm. Many extensions SINDy have emerged in the literature to deal with experimental data which are finite in length and noisy. Recently, the computationally intensive method of ensembling bootstrapped SINDy models (E-SINDy) was proposed for model identification, handling finite, highly noisy data. While the extensions of SINDy are numerous, their sparsity-promoting estimators occasionally provide sparse approximations of the dynamics as opposed to exact recovery. Furthermore, these estimators suffer under multicollinearity, e.g. the irrepresentable condition for the Lasso. In this paper, we demonstrate that the Trimmed Lasso for robust identification of models (TRIM) can provide exact recovery under more severe noise, finite data, and multicollinearity as opposed to E-SINDy. Additionally, the computational cost of TRIM is asymptotically equal to STLS since the sparsity parameter of the TRIM can be solved efficiently by convex solvers. We compare these methodologies on challenging nonlinear systems, specifically the Lorenz 63 system, the Bouc Wen oscillator from the nonlinear dynamics benchmark of No\"el and Schoukens, 2016, and a time delay system describing tool cutting dynamics. This study emphasizes the comparisons between STLS, reweighted $\ell_1$ minimization, and Trimmed Lasso in identification with respect to problems faced by practitioners: the problem of finite and noisy data, the performance of the sparse regression of when the library grows in dimension (multicollinearity), and automatic methods for choice of regularization parameters.

biometric

steal

Title: Anonymity Analysis of the Umbra Stealth Address Scheme on Ethereum. (arXiv:2308.01703v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.01703
• Code URL: null
• Copy Paste: [[2308.01703]] Anonymity Analysis of the Umbra Stealth Address Scheme on Ethereum(http://arxiv.org/abs/2308.01703)
• Summary:

  Stealth addresses are a privacy-enhancing technology that provides recipient anonymity on blockchains. In this work, we investigate the recipient anonymity and unlinkability guarantees of Umbra, the most widely used implementation of the stealth address scheme on Ethereum, and its three off-chain scalability solutions, e.g., Arbitrum, Optimism, and Polygon. We define and evaluate four heuristics to uncover the real recipients of stealth payments. We find that for the majority of Umbra payments, it is straightforward to establish the recipient, hence nullifying the benefits of using Umbra. Specifically, we find the real recipient of $48.5\%$, $25.8\%$, $65.7\%$, and $52.6\%$ of all Umbra transactions on the Ethereum main net, Polygon, Arbitrum, and Optimism networks, respectively. Finally, we suggest easily implementable countermeasures to evade our deanonymization and linking attacks.

extraction

Title: Contrastive Multi-FaceForensics: An End-to-end Bi-grained Contrastive Learning Approach for Multi-face Forgery Detection. (arXiv:2308.01520v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.01520
• Code URL: null
• Copy Paste: [[2308.01520]] Contrastive Multi-FaceForensics: An End-to-end Bi-grained Contrastive Learning Approach for Multi-face Forgery Detection(http://arxiv.org/abs/2308.01520)
• Summary:

  DeepFakes have raised serious societal concerns, leading to a great surge in detection-based forensics methods in recent years. Face forgery recognition is the conventional detection method that usually follows a two-phase pipeline: it extracts the face first and then determines its authenticity by classification. Since DeepFakes in the wild usually contain multiple faces, using face forgery detection methods is merely practical as they have to process faces in a sequel, i.e., only one face is processed at the same time. One straightforward way to address this issue is to integrate face extraction and forgery detection in an end-to-end fashion by adapting advanced object detection architectures. However, as these object detection architectures are designed to capture the semantic information of different object categories rather than the subtle forgery traces among the faces, the direct adaptation is far from optimal. In this paper, we describe a new end-to-end framework, Contrastive Multi-FaceForensics (COMICS), to enhance multi-face forgery detection. The core of the proposed framework is a novel bi-grained contrastive learning approach that explores effective face forgery traces at both the coarse- and fine-grained levels. Specifically, the coarse-grained level contrastive learning captures the discriminative features among positive and negative proposal pairs in multiple scales with the instruction of the proposal generator, and the fine-grained level contrastive learning captures the pixel-wise discrepancy between the forged and original areas of the same face and the pixel-wise content inconsistency between different faces. Extensive experiments on the OpenForensics dataset demonstrate our method outperforms other counterparts by a large margin (~18.5%) and shows great potential for integration into various architectures.

Title: DETR Doesn't Need Multi-Scale or Locality Design. (arXiv:2308.01904v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.01904
• Code URL: null
• Copy Paste: [[2308.01904]] DETR Doesn't Need Multi-Scale or Locality Design(http://arxiv.org/abs/2308.01904)
• Summary:

  This paper presents an improved DETR detector that maintains a "plain" nature: using a single-scale feature map and global cross-attention calculations without specific locality constraints, in contrast to previous leading DETR-based detectors that reintroduce architectural inductive biases of multi-scale and locality into the decoder. We show that two simple technologies are surprisingly effective within a plain design to compensate for the lack of multi-scale feature maps and locality constraints. The first is a box-to-pixel relative position bias (BoxRPB) term added to the cross-attention formulation, which well guides each query to attend to the corresponding object region while also providing encoding flexibility. The second is masked image modeling (MIM)-based backbone pre-training which helps learn representation with fine-grained localization ability and proves crucial for remedying dependencies on the multi-scale feature maps. By incorporating these technologies and recent advancements in training and problem formation, the improved "plain" DETR showed exceptional improvements over the original DETR detector. By leveraging the Object365 dataset for pre-training, it achieved 63.9 mAP accuracy using a Swin-L backbone, which is highly competitive with state-of-the-art detectors which all heavily rely on multi-scale feature maps and region-based feature extraction. Code is available at https://github.com/impiga/Plain-DETR .

membership infer

federate

Title: Compressed and distributed least-squares regression: convergence rates with applications to Federated Learning. (arXiv:2308.01358v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.01358
• Code URL: null
• Copy Paste: [[2308.01358]] Compressed and distributed least-squares regression: convergence rates with applications to Federated Learning(http://arxiv.org/abs/2308.01358)
• Summary:

  In this paper, we investigate the impact of compression on stochastic gradient algorithms for machine learning, a technique widely used in distributed and federated learning. We underline differences in terms of convergence rates between several unbiased compression operators, that all satisfy the same condition on their variance, thus going beyond the classical worst-case analysis. To do so, we focus on the case of least-squares regression (LSR) and analyze a general stochastic approximation algorithm for minimizing quadratic functions relying on a random field. We consider weak assumptions on the random field, tailored to the analysis (specifically, expected H\"older regularity), and on the noise covariance, enabling the analysis of various randomizing mechanisms, including compression. We then extend our results to the case of federated learning.

  More formally, we highlight the impact on the convergence of the covariance $\mathfrak{C}_{\mathrm{ania}}$ of the additive noise induced by the algorithm. We demonstrate despite the non-regularity of the stochastic field, that the limit variance term scales with $\mathrm{Tr}(\mathfrak{C}_{\mathrm{ania}} H^{-1})/K$ (where $H$ is the Hessian of the optimization problem and $K$ the number of iterations) generalizing the rate for the vanilla LSR case where it is $\sigma^2 \mathrm{Tr}(H H^{-1}) / K = \sigma^2 d / K$ (Bach and Moulines, 2013). Then, we analyze the dependency of $\mathfrak{C}_{\mathrm{ania}}$ on the compression strategy and ultimately its impact on convergence, first in the centralized case, then in two heterogeneous FL frameworks.

fair

Title: BEVControl: Accurately Controlling Street-view Elements with Multi-perspective Consistency via BEV Sketch Layout. (arXiv:2308.01661v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.01661
• Code URL: null
• Copy Paste: [[2308.01661]] BEVControl: Accurately Controlling Street-view Elements with Multi-perspective Consistency via BEV Sketch Layout(http://arxiv.org/abs/2308.01661)
• Summary:

  Using synthesized images to boost the performance of perception models is a long-standing research challenge in computer vision. It becomes more eminent in visual-centric autonomous driving systems with multi-view cameras as some long-tail scenarios can never be collected. Guided by the BEV segmentation layouts, the existing generative networks seem to synthesize photo-realistic street-view images when evaluated solely on scene-level metrics. However, once zoom-in, they usually fail to produce accurate foreground and background details such as heading. To this end, we propose a two-stage generative method, dubbed BEVControl, that can generate accurate foreground and background contents. In contrast to segmentation-like input, it also supports sketch style input, which is more flexible for humans to edit. In addition, we propose a comprehensive multi-level evaluation protocol to fairly compare the quality of the generated scene, foreground object, and background geometry. Our extensive experiments show that our BEVControl surpasses the state-of-the-art method, BEVGen, by a significant margin, from 5.89 to 26.80 on foreground segmentation mIoU. In addition, we show that using images generated by BEVControl to train the downstream perception model, it achieves on average 1.29 improvement in NDS score.

Title: Wider and Deeper LLM Networks are Fairer LLM Evaluators. (arXiv:2308.01862v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.01862
• Code URL: null
• Copy Paste: [[2308.01862]] Wider and Deeper LLM Networks are Fairer LLM Evaluators(http://arxiv.org/abs/2308.01862)
• Summary:

  Measuring the quality of responses generated by LLMs is a challenging task, particularly when it comes to evaluating whether the response is aligned with human preference. A novel approach involves using the LLM itself to make evaluation and stabilizing the results through multiple independent evaluations, similar to a single-layer narrow LLM network. This network consists of a fixed number of neurons, with each neuron being the same LLM. In this paper, we draw upon the extensive research on deep neural networks to explore whether deeper and wider networks can lead to fairer evaluations. Specifically, inspired by the observation that different neurons in a neural network are responsible for detecting different concepts, we first adaptively generate as many neuron roles as possible for each evaluation sample. Each perspective corresponds to the role of a specific LLM neuron in the first layer. In subsequent layers, we follow the idea that higher layers in deep networks are responsible for more comprehensive features, each layer receives representations from all neurons in the previous layer, integrating the locally learned evaluation information to obtain a more comprehensive evaluation result. Interestingly, this network design resembles the process of academic paper reviewing. To validate the effectiveness of our method, we construct the largest and most diverse English evaluation benchmark LLMEval$^2$ for LLM evaluators, comprising 15 tasks, 8 abilities, and 2,553 samples. Experimental results demonstrate that a wider network (involving many reviewers) with 2 layers (one round of discussion) performs the best, improving kappa correlation coefficient from 0.28 to 0.34. We also leverage WideDeep to aid in the assessment of Chinese LLMs, which has accelerated the evaluation time by 4.6 times, resulting in a 60% cost saving. WideDeep achieves a remarkable 93% agreement level among humans.

Title: Price-Aware Deep Learning for Electricity Markets. (arXiv:2308.01436v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.01436
• Code URL: null
• Copy Paste: [[2308.01436]] Price-Aware Deep Learning for Electricity Markets(http://arxiv.org/abs/2308.01436)
• Summary:

  While deep learning gradually penetrates operational planning, its inherent prediction errors may significantly affect electricity prices. This letter examines how prediction errors propagate into electricity prices, revealing notable pricing errors and their spatial disparity in congested power systems. To improve fairness, we propose to embed electricity market-clearing optimization as a deep learning layer. Differentiating through this layer allows for balancing between prediction and pricing errors, as oppose to minimizing prediction errors alone. This layer implicitly optimizes fairness and controls the spatial distribution of price errors across the system. We showcase the price-aware deep learning in the nexus of wind power forecasting and short-term electricity market clearing.

interpretability

Title: Disentangling Multi-view Representations Beyond Inductive Bias. (arXiv:2308.01634v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.01634
• Code URL: null
• Copy Paste: [[2308.01634]] Disentangling Multi-view Representations Beyond Inductive Bias(http://arxiv.org/abs/2308.01634)
• Summary:

  Multi-view (or -modality) representation learning aims to understand the relationships between different view representations. Existing methods disentangle multi-view representations into consistent and view-specific representations by introducing strong inductive biases, which can limit their generalization ability. In this paper, we propose a novel multi-view representation disentangling method that aims to go beyond inductive biases, ensuring both interpretability and generalizability of the resulting representations. Our method is based on the observation that discovering multi-view consistency in advance can determine the disentangling information boundary, leading to a decoupled learning objective. We also found that the consistency can be easily extracted by maximizing the transformation invariance and clustering consistency between views. These observations drive us to propose a two-stage framework. In the first stage, we obtain multi-view consistency by training a consistent encoder to produce semantically-consistent representations across views as well as their corresponding pseudo-labels. In the second stage, we disentangle specificity from comprehensive representations by minimizing the upper bound of mutual information between consistent and comprehensive representations. Finally, we reconstruct the original data by concatenating pseudo-labels and view-specific representations. Our experiments on four multi-view datasets demonstrate that our proposed method outperforms 12 comparison methods in terms of clustering and classification performance. The visualization results also show that the extracted consistency and specificity are compact and interpretable. Our code can be found at \url{https://github.com/Guanzhou-Ke/DMRIB}.

Title: XNLP: An Interactive Demonstration System for Universal Structured NLP. (arXiv:2308.01846v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.01846
• Code URL: null
• Copy Paste: [[2308.01846]] XNLP: An Interactive Demonstration System for Universal Structured NLP(http://arxiv.org/abs/2308.01846)
• Summary:

  Structured Natural Language Processing (XNLP) is an important subset of NLP that entails understanding the underlying semantic or syntactic structure of texts, which serves as a foundational component for many downstream applications. Despite certain recent efforts to explore universal solutions for specific categories of XNLP tasks, a comprehensive and effective approach for unifying all XNLP tasks long remains underdeveloped. In the meanwhile, while XNLP demonstration systems are vital for researchers exploring various XNLP tasks, existing platforms can be limited to, e.g., supporting few XNLP tasks, lacking interactivity and universalness. To this end, we propose an advanced XNLP demonstration platform, where we propose leveraging LLM to achieve universal XNLP, with one model for all with high generalizability. Overall, our system advances in multiple aspects, including universal XNLP modeling, high performance, interpretability, scalability, and interactivity, providing a unified platform for exploring diverse XNLP tasks in the community. XNLP is online: https://xnlp.haofei.vip

explainability

Title: Evaluating Link Prediction Explanations for Graph Neural Networks. (arXiv:2308.01682v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.01682
• Code URL: null
• Copy Paste: [[2308.01682]] Evaluating Link Prediction Explanations for Graph Neural Networks(http://arxiv.org/abs/2308.01682)
• Summary:

  Graph Machine Learning (GML) has numerous applications, such as node/graph classification and link prediction, in real-world domains. Providing human-understandable explanations for GML models is a challenging yet fundamental task to foster their adoption, but validating explanations for link prediction models has received little attention. In this paper, we provide quantitative metrics to assess the quality of link prediction explanations, with or without ground-truth. State-of-the-art explainability methods for Graph Neural Networks are evaluated using these metrics. We discuss how underlying assumptions and technical details specific to the link prediction task, such as the choice of distance between node embeddings, can influence the quality of the explanations.

watermark

diffusion

Title: Reverse Stable Diffusion: What prompt was used to generate this image?. (arXiv:2308.01472v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.01472
• Code URL: null
• Copy Paste: [[2308.01472]] Reverse Stable Diffusion: What prompt was used to generate this image?(http://arxiv.org/abs/2308.01472)
• Summary:

  Text-to-image diffusion models such as Stable Diffusion have recently attracted the interest of many researchers, and inverting the diffusion process can play an important role in better understanding the generative process and how to engineer prompts in order to obtain the desired images. To this end, we introduce the new task of predicting the text prompt given an image generated by a generative diffusion model. We combine a series of white-box and black-box models (with and without access to the weights of the diffusion network) to deal with the proposed task. We propose a novel learning framework comprising of a joint prompt regression and multi-label vocabulary classification objective that generates improved prompts. To further improve our method, we employ a curriculum learning procedure that promotes the learning of image-prompt pairs with lower labeling noise (i.e. that are better aligned), and an unsupervised domain-adaptive kernel learning method that uses the similarities between samples in the source and target domains as extra features. We conduct experiments on the DiffusionDB data set, predicting text prompts from images generated by Stable Diffusion. Our novel learning framework produces excellent results on the aforementioned task, yielding the highest gains when applied on the white-box model. In addition, we make an interesting discovery: training a diffusion model on the prompt generation task can make the model generate images that are much better aligned with the input prompts, when the model is directly reused for text-to-image generation.

Title: Reference-Free Isotropic 3D EM Reconstruction using Diffusion Models. (arXiv:2308.01594v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.01594
• Code URL: null
• Copy Paste: [[2308.01594]] Reference-Free Isotropic 3D EM Reconstruction using Diffusion Models(http://arxiv.org/abs/2308.01594)
• Summary:

  Electron microscopy (EM) images exhibit anisotropic axial resolution due to the characteristics inherent to the imaging modality, presenting challenges in analysis and downstream tasks.In this paper, we propose a diffusion-model-based framework that overcomes the limitations of requiring reference data or prior knowledge about the degradation process. Our approach utilizes 2D diffusion models to consistently reconstruct 3D volumes and is well-suited for highly downsampled data. Extensive experiments conducted on two public datasets demonstrate the robustness and superiority of leveraging the generative prior compared to supervised learning methods. Additionally, we demonstrate our method's feasibility for self-supervised reconstruction, which can restore a single anisotropic volume without any training data.

Title: DiffColor: Toward High Fidelity Text-Guided Image Colorization with Diffusion Models. (arXiv:2308.01655v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.01655
• Code URL: null
• Copy Paste: [[2308.01655]] DiffColor: Toward High Fidelity Text-Guided Image Colorization with Diffusion Models(http://arxiv.org/abs/2308.01655)
• Summary:

  Recent data-driven image colorization methods have enabled automatic or reference-based colorization, while still suffering from unsatisfactory and inaccurate object-level color control. To address these issues, we propose a new method called DiffColor that leverages the power of pre-trained diffusion models to recover vivid colors conditioned on a prompt text, without any additional inputs. DiffColor mainly contains two stages: colorization with generative color prior and in-context controllable colorization. Specifically, we first fine-tune a pre-trained text-to-image model to generate colorized images using a CLIP-based contrastive loss. Then we try to obtain an optimized text embedding aligning the colorized image and the text prompt, and a fine-tuned diffusion model enabling high-quality image reconstruction. Our method can produce vivid and diverse colors with a few iterations, and keep the structure and background intact while having colors well-aligned with the target language guidance. Moreover, our method allows for in-context colorization, i.e., producing different colorization results by modifying prompt texts without any fine-tuning, and can achieve object-level controllable colorization results. Extensive experiments and user studies demonstrate that DiffColor outperforms previous works in terms of visual quality, color fidelity, and diversity of colorization options.

Title: Synthesizing Long-Term Human Motions with Diffusion Models via Coherent Sampling. (arXiv:2308.01850v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.01850
• Code URL: null
• Copy Paste: [[2308.01850]] Synthesizing Long-Term Human Motions with Diffusion Models via Coherent Sampling(http://arxiv.org/abs/2308.01850)
• Summary:

  Text-to-motion generation has gained increasing attention, but most existing methods are limited to generating short-term motions that correspond to a single sentence describing a single action. However, when a text stream describes a sequence of continuous motions, the generated motions corresponding to each sentence may not be coherently linked. Existing long-term motion generation methods face two main issues. Firstly, they cannot directly generate coherent motions and require additional operations such as interpolation to process the generated actions. Secondly, they generate subsequent actions in an autoregressive manner without considering the influence of future actions on previous ones. To address these issues, we propose a novel approach that utilizes a past-conditioned diffusion model with two optional coherent sampling methods: Past Inpainting Sampling and Compositional Transition Sampling. Past Inpainting Sampling completes subsequent motions by treating previous motions as conditions, while Compositional Transition Sampling models the distribution of the transition as the composition of two adjacent motions guided by different text prompts. Our experimental results demonstrate that our proposed method is capable of generating compositional and coherent long-term 3D human motions controlled by a user-instructed long text stream. The code is available at \href{https://github.com/yangzhao1230/PCMDM}{https://github.com/yangzhao1230/PCMDM}.

noise learning

data-free

transformer

Title: Data Augmentation for Human Behavior Analysis in Multi-Person Conversations. (arXiv:2308.01526v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.01526
• Code URL: null
• Copy Paste: [[2308.01526]] Data Augmentation for Human Behavior Analysis in Multi-Person Conversations(http://arxiv.org/abs/2308.01526)
• Summary:

  In this paper, we present the solution of our team HFUT-VUT for the MultiMediate Grand Challenge 2023 at ACM Multimedia 2023. The solution covers three sub-challenges: bodily behavior recognition, eye contact detection, and next speaker prediction. We select Swin Transformer as the baseline and exploit data augmentation strategies to address the above three tasks. Specifically, we crop the raw video to remove the noise from other parts. At the same time, we utilize data augmentation to improve the generalization of the model. As a result, our solution achieves the best results of 0.6262 for bodily behavior recognition in terms of mean average precision and the accuracy of 0.7771 for eye contact detection on the corresponding test set. In addition, our approach also achieves comparable results of 0.5281 for the next speaker prediction in terms of unweighted average recall.

Title: Multimodal Neurons in Pretrained Text-Only Transformers. (arXiv:2308.01544v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.01544
• Code URL: null
• Copy Paste: [[2308.01544]] Multimodal Neurons in Pretrained Text-Only Transformers(http://arxiv.org/abs/2308.01544)
• Summary:

  Language models demonstrate remarkable capacity to generalize representations learned in one modality to downstream tasks in other modalities. Can we trace this ability to individual neurons? We study the case where a frozen text transformer is augmented with vision using a self-supervised visual encoder and a single linear projection learned on an image-to-text task. Outputs of the projection layer are not immediately decodable into language describing image content; instead, we find that translation between modalities occurs deeper within the transformer. We introduce a procedure for identifying "multimodal neurons" that convert visual representations into corresponding text, and decoding the concepts they inject into the model's residual stream. In a series of experiments, we show that multimodal neurons operate on specific visual concepts across inputs, and have a systematic causal effect on image captioning.

Title: UPB at IberLEF-2023 AuTexTification: Detection of Machine-Generated Text using Transformer Ensembles. (arXiv:2308.01408v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.01408
• Code URL: null
• Copy Paste: [[2308.01408]] UPB at IberLEF-2023 AuTexTification: Detection of Machine-Generated Text using Transformer Ensembles(http://arxiv.org/abs/2308.01408)
• Summary:

  This paper describes the solutions submitted by the UPB team to the AuTexTification shared task, featured as part of IberLEF-2023. Our team participated in the first subtask, identifying text documents produced by large language models instead of humans. The organizers provided a bilingual dataset for this subtask, comprising English and Spanish texts covering multiple domains, such as legal texts, social media posts, and how-to articles. We experimented mostly with deep learning models based on Transformers, as well as training techniques such as multi-task learning and virtual adversarial training to obtain better results. We submitted three runs, two of which consisted of ensemble models. Our best-performing model achieved macro F1-scores of 66.63% on the English dataset and 67.10% on the Spanish dataset.

Title: LaFiCMIL: Rethinking Large File Classification from the Perspective of Correlated Multiple Instance Learning. (arXiv:2308.01413v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.01413
• Code URL: null
• Copy Paste: [[2308.01413]] LaFiCMIL: Rethinking Large File Classification from the Perspective of Correlated Multiple Instance Learning(http://arxiv.org/abs/2308.01413)
• Summary:

  Transformer-based models have revolutionized the performance of a wide range of language tasks. Intuitively, one might expect text classification, which does not necessitate as many high-level representations as generative tasks, to be comprehensively addressed with the powerful representation capabilities of Transformers. However, in reality, there remains significant potential for enhancement, particularly in the areas of multi-class and multi-label classification of lengthy textual documents and other large files. The performance of Transformer-based models is mainly hindered by a major limitation: a restricted input length, e.g., 512 tokens for BERT. While an increase in GPU memory can marginally extend this limit, practical real-world applications often operate under constrained GPU resources. In this work, we tackle the input limit problem from the perspective of correlated multiple instance learning. The proposed approach, LaFiCMIL, serves as a versatile framework applicable to various large file classification tasks covering binary, multi-class, and multi-label classification tasks, spanning various domains including Natural Language Processing, Programming Language Processing, and Android Analysis. To evaluate its effectiveness, we employ eight benchmark datasets pertaining to Long Document Classification, Code Defect Detection, and Android Malware Detection. Leveraging BERT-family models as feature extractors, our experimental results demonstrate that LaFiCMIL achieves new state-of-the-art performance across all benchmark datasets. This is largely attributable to its capability of scaling BERT up to nearly 20K tokens, running on a single Tesla V-100 GPU with 32G of memory.

Title: Novel Physics-Based Machine-Learning Models for Indoor Air Quality Approximations. (arXiv:2308.01438v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.01438
• Code URL: null
• Copy Paste: [[2308.01438]] Novel Physics-Based Machine-Learning Models for Indoor Air Quality Approximations(http://arxiv.org/abs/2308.01438)
• Summary:

  Cost-effective sensors are capable of real-time capturing a variety of air quality-related modalities from different pollutant concentrations to indoor/outdoor humidity and temperature. Machine learning (ML) models are capable of performing air-quality "ahead-of-time" approximations. Undoubtedly, accurate indoor air quality approximation significantly helps provide a healthy indoor environment, optimize associated energy consumption, and offer human comfort. However, it is crucial to design an ML architecture to capture the domain knowledge, so-called problem physics. In this study, we propose six novel physics-based ML models for accurate indoor pollutant concentration approximations. The proposed models include an adroit combination of state-space concepts in physics, Gated Recurrent Units, and Decomposition techniques. The proposed models were illustrated using data collected from five offices in a commercial building in California. The proposed models are shown to be less complex, computationally more efficient, and more accurate than similar state-of-the-art transformer-based models. The superiority of the proposed models is due to their relatively light architecture (computational efficiency) and, more importantly, their ability to capture the underlying highly nonlinear patterns embedded in the often contaminated sensor-collected indoor air quality temporal data.

generative

Title: Circumventing Concept Erasure Methods For Text-to-Image Generative Models. (arXiv:2308.01508v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.01508
• Code URL: null
• Copy Paste: [[2308.01508]] Circumventing Concept Erasure Methods For Text-to-Image Generative Models(http://arxiv.org/abs/2308.01508)
• Summary:

  Text-to-image generative models can produce photo-realistic images for an extremely broad range of concepts, and their usage has proliferated widely among the general public. On the flip side, these models have numerous drawbacks, including their potential to generate images featuring sexually explicit content, mirror artistic styles without permission, or even hallucinate (or deepfake) the likenesses of celebrities. Consequently, various methods have been proposed in order to "erase" sensitive concepts from text-to-image models. In this work, we examine five recently proposed concept erasure methods, and show that targeted concepts are not fully excised from any of these methods. Specifically, we leverage the existence of special learned word embeddings that can retrieve "erased" concepts from the sanitized models with no alterations to their weights. Our results highlight the brittleness of post hoc concept erasure methods, and call into question their use in the algorithmic toolkit for AI safety.

Title: Interleaving GANs with knowledge graphs to support design creativity for book covers. (arXiv:2308.01626v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.01626
• Code URL: null
• Copy Paste: [[2308.01626]] Interleaving GANs with knowledge graphs to support design creativity for book covers(http://arxiv.org/abs/2308.01626)
• Summary:

  An attractive book cover is important for the success of a book. In this paper, we apply Generative Adversarial Networks (GANs) to the book covers domain, using different methods for training in order to obtain better generated images. We interleave GANs with knowledge graphs to alter the input title to obtain multiple possible options for any given title, which are then used as an augmented input to the generator. Finally, we use the discriminator obtained during the training phase to select the best images generated with new titles. Our method performed better at generating book covers than previous attempts, and the knowledge graph gives better options to the book author or editor compared to using GANs alone.

Title: Deep Learning-based Prediction of Stress and Strain Maps in Arterial Walls for Improved Cardiovascular Risk Assessment. (arXiv:2308.01771v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.01771
• Code URL: null
• Copy Paste: [[2308.01771]] Deep Learning-based Prediction of Stress and Strain Maps in Arterial Walls for Improved Cardiovascular Risk Assessment(http://arxiv.org/abs/2308.01771)
• Summary:

  This study investigated the potential of end-to-end deep learning tools as a more effective substitute for FEM in predicting stress-strain fields within 2D cross sections of arterial wall. We first proposed a U-Net based fully convolutional neural network (CNN) to predict the von Mises stress and strain distribution based on the spatial arrangement of calcification within arterial wall cross-sections. Further, we developed a conditional generative adversarial network (cGAN) to enhance, particularly from the perceptual perspective, the prediction accuracy of stress and strain field maps for arterial walls with various calcification quantities and spatial configurations. On top of U-Net and cGAN, we also proposed their ensemble approaches, respectively, to further improve the prediction accuracy of field maps. Our dataset, consisting of input and output images, was generated by implementing boundary conditions and extracting stress-strain field maps. The trained U-Net models can accurately predict von Mises stress and strain fields, with structural similarity index scores (SSIM) of 0.854 and 0.830 and mean squared errors of 0.017 and 0.018 for stress and strain, respectively, on a reserved test set. Meanwhile, the cGAN models in a combination of ensemble and transfer learning techniques demonstrate high accuracy in predicting von Mises stress and strain fields, as evidenced by SSIM scores of 0.890 for stress and 0.803 for strain. Additionally, mean squared errors of 0.008 for stress and 0.017 for strain further support the model's performance on a designated test set. Overall, this study developed a surrogate model for finite element analysis, which can accurately and efficiently predict stress-strain fields of arterial walls regardless of complex geometries and boundary conditions.

Title: An End-to-end Food Portion Estimation Framework Based on Shape Reconstruction from Monocular Image. (arXiv:2308.01810v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.01810
• Code URL: null
• Copy Paste: [[2308.01810]] An End-to-end Food Portion Estimation Framework Based on Shape Reconstruction from Monocular Image(http://arxiv.org/abs/2308.01810)
• Summary:

  Dietary assessment is a key contributor to monitoring health status. Existing self-report methods are tedious and time-consuming with substantial biases and errors. Image-based food portion estimation aims to estimate food energy values directly from food images, showing great potential for automated dietary assessment solutions. Existing image-based methods either use a single-view image or incorporate multi-view images and depth information to estimate the food energy, which either has limited performance or creates user burdens. In this paper, we propose an end-to-end deep learning framework for food energy estimation from a monocular image through 3D shape reconstruction. We leverage a generative model to reconstruct the voxel representation of the food object from the input image to recover the missing 3D information. Our method is evaluated on a publicly available food image dataset Nutrition5k, resulting a Mean Absolute Error (MAE) of 40.05 kCal and Mean Absolute Percentage Error (MAPE) of 11.47% for food energy estimation. Our method uses RGB image as the only input at the inference stage and achieves competitive results compared to the existing method requiring both RGB and depth information.

large language model

Title: HouYi: An open-source large language model specially designed for renewable energy and carbon neutrality field. (arXiv:2308.01414v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.01414
• Code URL: null
• Copy Paste: [[2308.01414]] HouYi: An open-source large language model specially designed for renewable energy and carbon neutrality field(http://arxiv.org/abs/2308.01414)
• Summary:

  Renewable energy is important for achieving carbon neutrality goal. With the great success of Large Language Models (LLMs) like ChatGPT in automatic content generation, LLMs are playing an increasingly important role. However, there has not been a specially designed LLM for renewable energy. Meanwhile, there has not been any dataset of renewable energy for training LLMs. Therefore, this paper published the first open-source Renewable Energy Academic Paper (REAP) dataset for non-commercial LLM research of renewable energy. REAP dataset is collected through searching the title and abstract of 1,168,970 academic literatures from Web of Science. Based on REAP dataset, HouYi model, the first LLM for renewable energy, is developed through finetuning general LLMs. HouYi demonstrated powerful academic paper paragraph generation ability in renewable energy field. Experiments show that its ability to generate academic papers on renewable energy is comparable to ChatGPT, slightly outperforms Claude, ERNIE Bot and SparkDesk, and significantly outperforms open-source LLaMA-13B model.

Title: An Effective Data Creation Pipeline to Generate High-quality Financial Instruction Data for Large Language Model. (arXiv:2308.01415v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.01415
• Code URL: null
• Copy Paste: [[2308.01415]] An Effective Data Creation Pipeline to Generate High-quality Financial Instruction Data for Large Language Model(http://arxiv.org/abs/2308.01415)
• Summary:

  At the beginning era of large language model, it is quite critical to generate a high-quality financial dataset to fine-tune a large language model for financial related tasks. Thus, this paper presents a carefully designed data creation pipeline for this purpose. Particularly, we initiate a dialogue between an AI investor and financial expert using ChatGPT and incorporate the feedback of human financial experts, leading to the refinement of the dataset. This pipeline yielded a robust instruction tuning dataset comprised of 103k multi-turn chats. Extensive experiments have been conducted on this dataset to evaluate the model's performance by adopting an external GPT-4 as the judge. The promising experimental results verify that our approach led to significant advancements in generating accurate, relevant, and financial-style responses from AI models, and thus providing a powerful tool for applications within the financial sector.

Title: FinVis-GPT: A Multimodal Large Language Model for Financial Chart Analysis. (arXiv:2308.01430v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.01430
• Code URL: null
• Copy Paste: [[2308.01430]] FinVis-GPT: A Multimodal Large Language Model for Financial Chart Analysis(http://arxiv.org/abs/2308.01430)
• Summary:

  In this paper, we propose FinVis-GPT, a novel multimodal large language model (LLM) specifically designed for financial chart analysis. By leveraging the power of LLMs and incorporating instruction tuning and multimodal capabilities, FinVis-GPT is capable of interpreting financial charts and providing valuable analysis. To train FinVis-GPT, a financial task oriented dataset was generated for pre-training alignment and instruction tuning, comprising various types of financial charts and their corresponding descriptions. We evaluate the model performance via several case studies due to the time limit, and the promising results demonstrated that FinVis-GPT is superior in various financial chart related tasks, including generating descriptions, answering questions and predicting future market trends, surpassing existing state-of-the-art multimodal LLMs. The proposed FinVis-GPT serves as a pioneering effort in utilizing multimodal LLMs in the finance domain and our generated dataset will be release for public use in the near future to speedup related research.

Title: Large Language Model Displays Emergent Ability to Interpret Novel Literary Metaphors. (arXiv:2308.01497v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.01497
• Code URL: null
• Copy Paste: [[2308.01497]] Large Language Model Displays Emergent Ability to Interpret Novel Literary Metaphors(http://arxiv.org/abs/2308.01497)
• Summary:

  Recent advances in the performance of large language models (LLMs) have sparked debate over whether, given sufficient training, high-level human abilities emerge in such generic forms of artificial intelligence (AI). Despite the exceptional performance of LLMs on a wide range of tasks involving natural language processing and reasoning, there has been sharp disagreement as to whether their abilities extend to more creative human abilities. A core example is the ability to interpret novel metaphors. Given the enormous and non-curated text corpora used to train LLMs, a serious obstacle to designing tests is the requirement of finding novel yet high-quality metaphors that are unlikely to have been included in the training data. Here we assessed the ability of GPT-4, a state-of-the-art large language model, to provide natural-language interpretations of novel literary metaphors drawn from Serbian poetry and translated into English. Despite exhibiting no signs of having been exposed to these metaphors previously, the AI system consistently produced detailed and incisive interpretations. Human judge - blind to the fact that an AI model was involved - rated metaphor interpretations generated by GPT-4 as superior to those provided by a group of college students. In interpreting reversed metaphors, GPT-4, as well as humans, exhibited signs of sensitivity to the Gricean cooperative principle. These results indicate that LLMs such as GPT-4 have acquired an emergent ability to interpret complex novel metaphors.

Title: Baby's CoThought: Leveraging Large Language Models for Enhanced Reasoning in Compact Models. (arXiv:2308.01684v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.01684
• Code URL: null
• Copy Paste: [[2308.01684]] Baby's CoThought: Leveraging Large Language Models for Enhanced Reasoning in Compact Models(http://arxiv.org/abs/2308.01684)
• Summary:

  Large Language Models (LLMs) demonstrate remarkable performance on a variety of Natural Language Understanding (NLU) tasks, primarily due to their in-context learning ability. This ability is utilized in our proposed "CoThought" pipeline, which efficiently trains smaller "baby" language models (BabyLMs) by leveraging the Chain of Thought (CoT) prompting of LLMs. Our pipeline restructures a dataset of less than 100M in size using GPT-3.5-turbo, transforming it into task-oriented, human-readable texts that are comparable to the school texts for language learners. The BabyLM is then pretrained on this restructured dataset in a RoBERTa (Liu et al., 2019) fashion. In evaluations across 4 benchmarks, our BabyLM outperforms the RoBERTa-base in 10 linguistic, NLU, and question answering tasks by more than 3 points, showing superior ability to extract contextual information. These results suggest that compact LMs pretrained on small, LLM-restructured data can better understand tasks and achieve improved performance. The code for data processing and model training is available at: https://github.com/oooranz/Baby-CoThought.

Title: Local Large Language Models for Complex Structured Medical Tasks. (arXiv:2308.01727v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.01727
• Code URL: null
• Copy Paste: [[2308.01727]] Local Large Language Models for Complex Structured Medical Tasks(http://arxiv.org/abs/2308.01727)
• Summary:

  This paper introduces an approach that combines the language reasoning capabilities of large language models (LLMs) with the benefits of local training to tackle complex, domain-specific tasks. Specifically, the authors demonstrate their approach by extracting structured condition codes from pathology reports. The proposed approach utilizes local LLMs, which can be fine-tuned to respond to specific generative instructions and provide structured outputs. The authors collected a dataset of over 150k uncurated surgical pathology reports, containing gross descriptions, final diagnoses, and condition codes. They trained different model architectures, including LLaMA, BERT and LongFormer and evaluated their performance. The results show that the LLaMA-based models significantly outperform BERT-style models across all evaluated metrics, even with extremely reduced precision. The LLaMA models performed especially well with large datasets, demonstrating their ability to handle complex, multi-label tasks. Overall, this work presents an effective approach for utilizing LLMs to perform domain-specific tasks using accessible hardware, with potential applications in the medical domain, where complex data extraction and classification are required.

Title: Ambient Adventures: Teaching ChatGPT on Developing Complex Stories. (arXiv:2308.01734v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.01734
• Code URL: null
• Copy Paste: [[2308.01734]] Ambient Adventures: Teaching ChatGPT on Developing Complex Stories(http://arxiv.org/abs/2308.01734)
• Summary:

  Imaginative play is an area of creativity that could allow robots to engage with the world around them in a much more personified way. Imaginary play can be seen as taking real objects and locations and using them as imaginary objects and locations in virtual scenarios. We adopted the story generation capability of large language models (LLMs) to obtain the stories used for imaginary play with human-written prompts. Those generated stories will be simplified and mapped into action sequences that can guide the agent in imaginary play. To evaluate whether the agent can successfully finish the imaginary play, we also designed a text adventure game to simulate a house as the playground for the agent to interact.

Title: Supply chain emission estimation using large language models. (arXiv:2308.01741v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.01741
• Code URL: null
• Copy Paste: [[2308.01741]] Supply chain emission estimation using large language models(http://arxiv.org/abs/2308.01741)
• Summary:

  Large enterprises face a crucial imperative to achieve the Sustainable Development Goals (SDGs), especially goal 13, which focuses on combating climate change and its impacts. To mitigate the effects of climate change, reducing enterprise Scope 3 (supply chain emissions) is vital, as it accounts for more than 90\% of total emission inventories. However, tracking Scope 3 emissions proves challenging, as data must be collected from thousands of upstream and downstream suppliers.To address the above mentioned challenges, we propose a first-of-a-kind framework that uses domain-adapted NLP foundation models to estimate Scope 3 emissions, by utilizing financial transactions as a proxy for purchased goods and services. We compared the performance of the proposed framework with the state-of-art text classification models such as TF-IDF, word2Vec, and Zero shot learning. Our results show that the domain-adapted foundation model outperforms state-of-the-art text mining techniques and performs as well as a subject matter expert (SME). The proposed framework could accelerate the Scope 3 estimation at Enterprise scale and will help to take appropriate climate actions to achieve SDG 13.

Title: Does Correction Remain An Problem For Large Language Models?. (arXiv:2308.01776v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.01776
• Code URL: null
• Copy Paste: [[2308.01776]] Does Correction Remain An Problem For Large Language Models?(http://arxiv.org/abs/2308.01776)
• Summary:

  As large language models, such as GPT, continue to advance the capabilities of natural language processing (NLP), the question arises: does the problem of correction still persist? This paper investigates the role of correction in the context of large language models by conducting two experiments. The first experiment focuses on correction as a standalone task, employing few-shot learning techniques with GPT-like models for error correction. The second experiment explores the notion of correction as a preparatory task for other NLP tasks, examining whether large language models can tolerate and perform adequately on texts containing certain levels of noise or errors. By addressing these experiments, we aim to shed light on the significance of correction in the era of large language models and its implications for various NLP applications.

Title: Scaling Relationship on Learning Mathematical Reasoning with Large Language Models. (arXiv:2308.01825v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.01825
• Code URL: null
• Copy Paste: [[2308.01825]] Scaling Relationship on Learning Mathematical Reasoning with Large Language Models(http://arxiv.org/abs/2308.01825)
• Summary:

  Mathematical reasoning is a challenging task for large language models (LLMs), while the scaling relationship of it with respect to LLM capacity is under-explored. In this paper, we investigate how the pre-training loss, supervised data amount, and augmented data amount influence the reasoning performances of a supervised LLM. We find that pre-training loss is a better indicator of the model's performance than the model's parameter count. We apply supervised fine-tuning (SFT) with different amounts of supervised data and empirically find a log-linear relation between data amount and model performance, and we find better models improve less with enlarged supervised datasets. To augment more data samples for improving model performances without any human effort, we propose to apply Rejection sampling Fine-Tuning (RFT). RFT uses supervised models to generate and collect correct reasoning paths as augmented fine-tuning datasets. We find with augmented samples containing more distinct reasoning paths, RFT improves mathematical reasoning performance more for LLMs. We also find RFT brings more improvement for less performant LLMs. Furthermore, we combine rejection samples from multiple models which push LLaMA-7B to an accuracy of 49.3% and outperforms the supervised fine-tuning (SFT) accuracy of 35.9% significantly.

Title: The Capability of Large Language Models to Measure Psychiatric Functioning. (arXiv:2308.01834v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.01834
• Code URL: null
• Copy Paste: [[2308.01834]] The Capability of Large Language Models to Measure Psychiatric Functioning(http://arxiv.org/abs/2308.01834)
• Summary:

  The current work investigates the capability of Large language models (LLMs) that are explicitly trained on large corpuses of medical knowledge (Med-PaLM 2) to predict psychiatric functioning from patient interviews and clinical descriptions without being trained to do so. To assess this, n = 145 depression and n =115 PTSD assessments and n = 46 clinical case studies across high prevalence/high comorbidity disorders (Depressive, Anxiety, Psychotic, trauma and stress, Addictive disorders) were analyzed using prompts to extract estimated clinical scores and diagnoses. Results demonstrate that Med-PaLM 2 is capable of assessing psychiatric functioning across a range of psychiatric conditions with the strongest performance being the prediction of depression scores based on standardized assessments (Accuracy range= 0.80 - 0.84) which were statistically indistinguishable from human clinical raters t(1,144) = 1.20; p = 0.23. Results show the potential for general clinical language models to flexibly predict psychiatric risk based on free descriptions of functioning from both patients and clinicians.

Title: Reasoning in Large Language Models Through Symbolic Math Word Problems. (arXiv:2308.01906v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.01906
• Code URL: null
• Copy Paste: [[2308.01906]] Reasoning in Large Language Models Through Symbolic Math Word Problems(http://arxiv.org/abs/2308.01906)
• Summary:

  Large language models (LLMs) have revolutionized NLP by solving downstream tasks with little to no labeled data. Despite their versatile abilities, the larger question of their ability to reason remains ill-understood. This paper addresses reasoning in math word problems (MWPs) by studying symbolic versions of the numeric problems, since a symbolic expression is a "concise explanation" of the numeric answer. We create and use a symbolic version of the SVAMP dataset and find that GPT-3's davinci-002 model also has good zero-shot accuracy on symbolic MWPs. To evaluate the faithfulness of the model's reasoning, we go beyond accuracy and additionally evaluate the alignment between the final answer and the outputted reasoning, which correspond to numeric and symbolic answers respectively for MWPs. We explore a self-prompting approach to encourage the symbolic reasoning to align with the numeric answer, thus equipping the LLM with the ability to provide a concise and verifiable reasoning and making it more interpretable. Surprisingly, self-prompting also improves the symbolic accuracy to be higher than both the numeric and symbolic accuracies, thus providing an ensembling effect. The SVAMP_Sym dataset will be released for future research on symbolic math problems.

segmentation

Title: Harder synthetic anomalies to improve OoD detection in Medical Images. (arXiv:2308.01412v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.01412
• Code URL: null
• Copy Paste: [[2308.01412]] Harder synthetic anomalies to improve OoD detection in Medical Images(http://arxiv.org/abs/2308.01412)
• Summary:

  Our method builds upon previous Medical Out-of-Distribution (MOOD) challenge winners that empirically show that synthetic local anomalies generated copying / interpolating foreign patches are useful to train segmentation networks able to generalize to unseen types of anomalies. In terms of the synthetic anomaly generation process, our contributions makes synthetic anomalies more heterogeneous and challenging by 1) using random shapes instead of squares and 2) smoothing the interpolation edge of anomalies so networks cannot rely on the high gradient between image - foreign patch to identify anomalies. Our experiments using the validation set of 2020 MOOD winners show that both contributions improved substantially the method performance. We used a standard 3D U-Net architecture as segmentation network, trained patch-wise in both brain and abdominal datasets. Our final challenge submission consisted of 10 U-Nets trained across 5 data folds with different configurations of the anomaly generation process. Our method achieved first position in both sample-wise and pixel-wise tasks in the 2022 edition of the Medical Out-of-Distribution held at MICCAI.

Title: Assessing Systematic Weaknesses of DNNs using Counterfactuals. (arXiv:2308.01614v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.01614
• Code URL: null
• Copy Paste: [[2308.01614]] Assessing Systematic Weaknesses of DNNs using Counterfactuals(http://arxiv.org/abs/2308.01614)
• Summary:

  With the advancement of DNNs into safety-critical applications, testing approaches for such models have gained more attention. A current direction is the search for and identification of systematic weaknesses that put safety assumptions based on average performance values at risk. Such weaknesses can take on the form of (semantically coherent) subsets or areas in the input space where a DNN performs systematically worse than its expected average. However, it is non-trivial to attribute the reason for such observed low performances to the specific semantic features that describe the subset. For instance, inhomogeneities within the data w.r.t. other (non-considered) attributes might distort results. However, taking into account all (available) attributes and their interaction is often computationally highly expensive. Inspired by counterfactual explanations, we propose an effective and computationally cheap algorithm to validate the semantic attribution of existing subsets, i.e., to check whether the identified attribute is likely to have caused the degraded performance. We demonstrate this approach on an example from the autonomous driving domain using highly annotated simulated data, where we show for a semantic segmentation model that (i) performance differences among the different pedestrian assets exist, but (ii) only in some cases is the asset type itself the reason for this reduction in the performance.

Title: ReIDTrack: Multi-Object Track and Segmentation Without Motion. (arXiv:2308.01622v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.01622
• Code URL: null
• Copy Paste: [[2308.01622]] ReIDTrack: Multi-Object Track and Segmentation Without Motion(http://arxiv.org/abs/2308.01622)
• Summary:

  In recent years, dominant Multi-object tracking (MOT) and segmentation (MOTS) methods mainly follow the tracking-by-detection paradigm. Transformer-based end-to-end (E2E) solutions bring some ideas to MOT and MOTS, but they cannot achieve a new state-of-the-art (SOTA) performance in major MOT and MOTS benchmarks. Detection and association are two main modules of the tracking-by-detection paradigm. Association techniques mainly depend on the combination of motion and appearance information. As deep learning has been recently developed, the performance of the detection and appearance model is rapidly improved. These trends made us consider whether we can achieve SOTA based on only high-performance detection and appearance model. Our paper mainly focuses on exploring this direction based on CBNetV2 with Swin-B as a detection model and MoCo-v2 as a self-supervised appearance model. Motion information and IoU mapping were removed during the association. Our method wins 1st place on the MOTS track and wins 2nd on the MOT track in the CVPR2023 WAD workshop. We hope our simple and effective method can give some insights to the MOT and MOTS research community. Source code will be released under this git repository

Title: LiDAR-Camera Panoptic Segmentation via Geometry-Consistent and Semantic-Aware Alignment. (arXiv:2308.01686v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.01686
• Code URL: null
• Copy Paste: [[2308.01686]] LiDAR-Camera Panoptic Segmentation via Geometry-Consistent and Semantic-Aware Alignment(http://arxiv.org/abs/2308.01686)
• Summary:

  3D panoptic segmentation is a challenging perception task that requires both semantic segmentation and instance segmentation. In this task, we notice that images could provide rich texture, color, and discriminative information, which can complement LiDAR data for evident performance improvement, but their fusion remains a challenging problem. To this end, we propose LCPS, the first LiDAR-Camera Panoptic Segmentation network. In our approach, we conduct LiDAR-Camera fusion in three stages: 1) an Asynchronous Compensation Pixel Alignment (ACPA) module that calibrates the coordinate misalignment caused by asynchronous problems between sensors; 2) a Semantic-Aware Region Alignment (SARA) module that extends the one-to-one point-pixel mapping to one-to-many semantic relations; 3) a Point-to-Voxel feature Propagation (PVP) module that integrates both geometric and semantic fusion information for the entire point cloud. Our fusion strategy improves about 6.9% PQ performance over the LiDAR-only baseline on NuScenes dataset. Extensive quantitative and qualitative experiments further demonstrate the effectiveness of our novel framework. The code will be released at https://github.com/zhangzw12319/lcps.git.

Title: Weakly Supervised 3D Instance Segmentation without Instance-level Annotations. (arXiv:2308.01721v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.01721
• Code URL: null
• Copy Paste: [[2308.01721]] Weakly Supervised 3D Instance Segmentation without Instance-level Annotations(http://arxiv.org/abs/2308.01721)
• Summary:

  3D semantic scene understanding tasks have achieved great success with the emergence of deep learning, but often require a huge amount of manually annotated training data. To alleviate the annotation cost, we propose the first weakly-supervised 3D instance segmentation method that only requires categorical semantic labels as supervision, and we do not need instance-level labels. The required semantic annotations can be either dense or extreme sparse (e.g. 0.02% of total points). Even without having any instance-related ground-truth, we design an approach to break point clouds into raw fragments and find the most confident samples for learning instance centroids. Furthermore, we construct a recomposed dataset using pseudo instances, which is used to learn our defined multilevel shape-aware objectness signal. An asymmetrical object inference algorithm is followed to process core points and boundary points with different strategies, and generate high-quality pseudo instance labels to guide iterative training. Experiments demonstrate that our method can achieve comparable results with recent fully supervised methods. By generating pseudo instance labels from categorical semantic labels, our designed approach can also assist existing methods for learning 3D instance segmentation at reduced annotation cost.

Title: Point2Mask: Point-supervised Panoptic Segmentation via Optimal Transport. (arXiv:2308.01779v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.01779
• Code URL: null
• Copy Paste: [[2308.01779]] Point2Mask: Point-supervised Panoptic Segmentation via Optimal Transport(http://arxiv.org/abs/2308.01779)
• Summary:

  Weakly-supervised image segmentation has recently attracted increasing research attentions, aiming to avoid the expensive pixel-wise labeling. In this paper, we present an effective method, namely Point2Mask, to achieve high-quality panoptic prediction using only a single random point annotation per target for training. Specifically, we formulate the panoptic pseudo-mask generation as an Optimal Transport (OT) problem, where each ground-truth (gt) point label and pixel sample are defined as the label supplier and consumer, respectively. The transportation cost is calculated by the introduced task-oriented maps, which focus on the category-wise and instance-wise differences among the various thing and stuff targets. Furthermore, a centroid-based scheme is proposed to set the accurate unit number for each gt point supplier. Hence, the pseudo-mask generation is converted into finding the optimal transport plan at a globally minimal transportation cost, which can be solved via the Sinkhorn-Knopp Iteration. Experimental results on Pascal VOC and COCO demonstrate the promising performance of our proposed Point2Mask approach to point-supervised panoptic segmentation. Source code is available at: https://github.com/LiWentomng/Point2Mask.

Title: Reconstructing Three-Dimensional Models of Interacting Humans. (arXiv:2308.01854v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.01854
• Code URL: null
• Copy Paste: [[2308.01854]] Reconstructing Three-Dimensional Models of Interacting Humans(http://arxiv.org/abs/2308.01854)
• Summary:

  Understanding 3d human interactions is fundamental for fine-grained scene analysis and behavioural modeling. However, most of the existing models predict incorrect, lifeless 3d estimates, that miss the subtle human contact aspects--the essence of the event--and are of little use for detailed behavioral understanding. This paper addresses such issues with several contributions: (1) we introduce models for interaction signature estimation (ISP) encompassing contact detection, segmentation, and 3d contact signature prediction; (2) we show how such components can be leveraged to ensure contact consistency during 3d reconstruction; (3) we construct several large datasets for learning and evaluating 3d contact prediction and reconstruction methods; specifically, we introduce CHI3D, a lab-based accurate 3d motion capture dataset with 631 sequences containing $2,525$ contact events, $728,664$ ground truth 3d poses, as well as FlickrCI3D, a dataset of $11,216$ images, with $14,081$ processed pairs of people, and $81,233$ facet-level surface correspondences. Finally, (4) we propose methodology for recovering the ground-truth pose and shape of interacting people in a controlled setup and (5) annotate all 3d interaction motions in CHI3D with textual descriptions. Motion data in multiple formats (GHUM and SMPLX parameters, Human3.6m 3d joints) is made available for research purposes at \url{https://ci3d.imar.ro}, together with an evaluation server and a public benchmark.