secure

Title: Secure Transformer Inference. (arXiv:2312.00025v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.00025
• Code URL: https://github.com/yuanmu97/secure-transformer-inference
• Copy Paste: [[2312.00025]] Secure Transformer Inference(http://arxiv.org/abs/2312.00025)
• Summary:

  We present a three-party protocol that can protect both Transformer parameters and user data during the inference phase. For each feedforward inference process, our protocol only introduces permutation computation of input and output data on the user side. Our protocol, Secure Transformer Inference Protocol (STIP), can be applied to real-world services like ChatGPT.

Title: Crypto analysis of the key distribution scheme using noise-free resistances. (arXiv:2312.00031v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.00031
• Code URL: null
• Copy Paste: [[2312.00031]] Crypto analysis of the key distribution scheme using noise-free resistances(http://arxiv.org/abs/2312.00031)
• Summary:

  Known key exchange schemes offering information-theoretic (unconditional) security are complex and costly to implement. Nonetheless, they remain the only known methods for achieving unconditional security in key exchange. Therefore, the explorations for simpler solutions for information-theoretic security are highly justified. Lin et al. [1] proposed an interesting hardware key distribution scheme that utilizes thermal-noise-free resistances and DC voltages.

  A crypto analysis of this system is presented. It is shown that, if Eve gains access to the initial shared secret at any time in the past or future, she can successfully crack all the generated keys in the past and future, even retroactively, using passively obtained and recorded voltages and currents. Therefore, the scheme is not a secure key exchanger, but it is rather a key expander with no more information entropy than the originally shared secret at the beginning.

  We also point out that the proposed defense methods against active attacks do not function when the original shared secret is compromised because then the communication cannot be efficiently authenticated. However, they do work when an unconditionally secure key exchanger is applied to enable the authenticated communication protocol.

Title: FBChain: A Blockchain-based Federated Learning Model with Efficiency and Secure Communication. (arXiv:2312.00035v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.00035
• Code URL: null
• Copy Paste: [[2312.00035]] FBChain: A Blockchain-based Federated Learning Model with Efficiency and Secure Communication(http://arxiv.org/abs/2312.00035)
• Summary:

  Privacy and security in the parameter transmission process of federated learning are currently among the most prominent concerns. However, there are two thorny problems caused by unprotected communication methods: "parameter-leakage" and "inefficient-communication". This article proposes Blockchain-based Federated Learning (FBChain) model for federated learning parameter communication to overcome the above two problems. First, we utilize the immutability of blockchain to store the global model and hash value of local model parameters in case of tampering during the communication process, protect data privacy by encrypting parameters, and verify data consistency by comparing the hash values of local parameters, thus addressing the "parameter-leakage" problem. Second, the Proof of Weighted Link Speed (PoWLS) consensus algorithm comprehensively selects nodes with the higher weighted link speed to aggregate global model and package blocks, thereby solving the "inefficient-communication" problem. Experimental results demonstrate the effectiveness of our proposed FBChain model and its ability to improve model communication efficiency in federated learning.

Title: SPAM: Secure & Private Aircraft Management. (arXiv:2312.00245v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.00245
• Code URL: null
• Copy Paste: [[2312.00245]] SPAM: Secure & Private Aircraft Management(http://arxiv.org/abs/2312.00245)
• Summary:

  With the rising use of aircrafts for operations ranging from disaster-relief to warfare, there is a growing risk of adversarial attacks. Malicious entities often only require the location of the aircraft for these attacks. Current satellite-aircraft communication and tracking protocols put aircrafts at risk if the satellite is compromised, due to computation being done in plaintext. In this work, we present \texttt{SPAM}, a private, secure, and accurate system that allows satellites to efficiently manage and maintain tracking angles for aircraft fleets without learning aircrafts' locations. \texttt{SPAM} is built upon multi-party computation and zero-knowledge proofs to guarantee privacy and high efficiency. While catered towards aircrafts, \texttt{SPAM}'s zero-knowledge fleet management can be easily extended to the IoT, with very little overhead.

Title: Using Honeybuckets to Characterize Cloud Storage Scanning in the Wild. (arXiv:2312.00580v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.00580
• Code URL: null
• Copy Paste: [[2312.00580]] Using Honeybuckets to Characterize Cloud Storage Scanning in the Wild(http://arxiv.org/abs/2312.00580)
• Summary:

  In this work, we analyze to what extent actors target poorly-secured cloud storage buckets for attack. We deployed hundreds of AWS S3 honeybuckets with different names and content to lure and measure different scanning strategies. Actors exhibited clear preferences for scanning buckets that appeared to belong to organizations, especially commercial entities in the technology sector with a vulnerability disclosure program. Actors continuously engaged with the content of buckets by downloading, uploading, and deleting files. Most alarmingly, we recorded multiple instances in which malicious actors downloaded, read, and understood a document from our honeybucket, leading them to attempt to gain unauthorized server access.

security

Title: A knowledge-based data-driven (KBDD) framework for all-day identification of cloud types using satellite remote sensing. (arXiv:2312.00308v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00308
• Code URL: null
• Copy Paste: [[2312.00308]] A knowledge-based data-driven (KBDD) framework for all-day identification of cloud types using satellite remote sensing(http://arxiv.org/abs/2312.00308)
• Summary:

  Cloud types, as a type of meteorological data, are of particular significance for evaluating changes in rainfall, heatwaves, water resources, floods and droughts, food security and vegetation cover, as well as land use. In order to effectively utilize high-resolution geostationary observations, a knowledge-based data-driven (KBDD) framework for all-day identification of cloud types based on spectral information from Himawari-8/9 satellite sensors is designed. And a novel, simple and efficient network, named CldNet, is proposed. Compared with widely used semantic segmentation networks, including SegNet, PSPNet, DeepLabV3+, UNet, and ResUnet, our proposed model CldNet with an accuracy of 80.89+-2.18% is state-of-the-art in identifying cloud types and has increased by 32%, 46%, 22%, 2%, and 39%, respectively. With the assistance of auxiliary information (e.g., satellite zenith/azimuth angle, solar zenith/azimuth angle), the accuracy of CldNet-W using visible and near-infrared bands and CldNet-O not using visible and near-infrared bands on the test dataset is 82.23+-2.14% and 73.21+-2.02%, respectively. Meanwhile, the total parameters of CldNet are only 0.46M, making it easy for edge deployment. More importantly, the trained CldNet without any fine-tuning can predict cloud types with higher spatial resolution using satellite spectral data with spatial resolution 0.02{\deg}*0.02{\deg}, which indicates that CldNet possesses a strong generalization ability. In aggregate, the KBDD framework using CldNet is a highly effective cloud-type identification system capable of providing a high-fidelity, all-day, spatiotemporal cloud-type database for many climate assessment fields.

Title: Risk-Aware and Explainable Framework for Ensuring Guaranteed Coverage in Evolving Hardware Trojan Detection. (arXiv:2312.00009v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.00009
• Code URL: null
• Copy Paste: [[2312.00009]] Risk-Aware and Explainable Framework for Ensuring Guaranteed Coverage in Evolving Hardware Trojan Detection(http://arxiv.org/abs/2312.00009)
• Summary:

  As the semiconductor industry has shifted to a fabless paradigm, the risk of hardware Trojans being inserted at various stages of production has also increased. Recently, there has been a growing trend toward the use of machine learning solutions to detect hardware Trojans more effectively, with a focus on the accuracy of the model as an evaluation metric. However, in a high-risk and sensitive domain, we cannot accept even a small misclassification. Additionally, it is unrealistic to expect an ideal model, especially when Trojans evolve over time. Therefore, we need metrics to assess the trustworthiness of detected Trojans and a mechanism to simulate unseen ones. In this paper, we generate evolving hardware Trojans using our proposed novel conformalized generative adversarial networks and offer an efficient approach to detecting them based on a non-invasive algorithm-agnostic statistical inference framework that leverages the Mondrian conformal predictor. The method acts like a wrapper over any of the machine learning models and produces set predictions along with uncertainty quantification for each new detected Trojan for more robust decision-making. In the case of a NULL set, a novel method to reject the decision by providing a calibrated explainability is discussed. The proposed approach has been validated on both synthetic and real chip-level benchmarks and proven to pave the way for researchers looking to find informed machine learning solutions to hardware security problems.

Title: Security Challenges in Autonomous Systems Design. (arXiv:2312.00018v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.00018
• Code URL: null
• Copy Paste: [[2312.00018]] Security Challenges in Autonomous Systems Design(http://arxiv.org/abs/2312.00018)
• Summary:

  Autonomous systems are emerging in many application domains. With the recent advancements in artificial intelligence and machine learning, sensor technology, perception algorithms and robotics, scenarios previously requiring strong human involvement can be handled by autonomous systems. With the independence from human control, cybersecurity of such systems becomes even more critical as no human intervention in case of undesired behavior is possible. In this context, this paper discusses emerging security challenges in autonomous systems design which arise in many domains such as autonomous incident response, risk assessment, data availability, systems interaction, trustworthiness, updatability, access control, as well as the reliability and explainability of machine learning methods. In all these areas, this paper thoroughly discusses the state of the art, identifies emerging security challenges and proposes research directions to address these challenges for developing secure autonomous systems.

Title: Technical Report relating to CVE-2022-46480, CVE-2023-26941, CVE-2023-26942, and CVE-2023-26943. (arXiv:2312.00021v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.00021
• Code URL: null
• Copy Paste: [[2312.00021]] Technical Report relating to CVE-2022-46480, CVE-2023-26941, CVE-2023-26942, and CVE-2023-26943(http://arxiv.org/abs/2312.00021)
• Summary:

  The following technical report provides background information relating to four CVEs found in the following products: Ultraloq UL3 BT (CVE-2022-46480); Yale Conexis L1 Smart Lock (CVE-2023-26941); Yale IA-210 Intruder Alarm (CVE-2023-26942); Yale Keyless Smart Lock (CVE-2023-26943). The work discussed here was carried out by Ash Allen, Dr. Alexios Mylonas, and Dr. Stilianos Vidalis as part of a wider research project into smart device security. Responsible disclosure of all four issues has been made with the appropriate vendors, and they have been acknowledged as vulnerabilities.

Title: Hypergraph Topological Features for Autoencoder-Based Intrusion Detection for Cybersecurity Data. (arXiv:2312.00023v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.00023
• Code URL: null
• Copy Paste: [[2312.00023]] Hypergraph Topological Features for Autoencoder-Based Intrusion Detection for Cybersecurity Data(http://arxiv.org/abs/2312.00023)
• Summary:

  In this position paper, we argue that when hypergraphs are used to capture multi-way local relations of data, their resulting topological features describe global behaviour. Consequently, these features capture complex correlations that can then serve as high fidelity inputs to autoencoder-driven anomaly detection pipelines. We propose two such potential pipelines for cybersecurity data, one that uses an autoencoder directly to determine network intrusions, and one that de-noises input data for a persistent homology system, PHANTOM. We provide heuristic justification for the use of the methods described therein for an intrusion detection pipeline for cyber data. We conclude by showing a small example over synthetic cyber attack data.

Title: Can LLMs Patch Security Issues?. (arXiv:2312.00024v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.00024
• Code URL: null
• Copy Paste: [[2312.00024]] Can LLMs Patch Security Issues?(http://arxiv.org/abs/2312.00024)
• Summary:

  Large Language Models (LLMs) have shown impressive proficiency in code generation. Nonetheless, similar to human developers, these models might generate code that contains security vulnerabilities and flaws. Writing secure code remains a substantial challenge, as vulnerabilities often arise during interactions between programs and external systems or services, such as databases and operating systems. In this paper, we propose a novel approach, Feedback-Driven Solution Synthesis (FDSS), designed to explore the use of LLMs in receiving feedback from Bandit, which is a static code analysis tool, and then the LLMs generate potential solutions to resolve security vulnerabilities. Each solution, along with the vulnerable code, is then sent back to the LLM for code refinement. Our approach shows a significant improvement over the baseline and outperforms existing approaches. Furthermore, we introduce a new dataset, PythonSecurityEval, collected from real-world scenarios on Stack Overflow to evaluate the LLMs' ability to generate secure code. Code and data are available at \url{https://github.com/Kamel773/LLM-code-refine}

Title: DeFi Security: Turning The Weakest Link Into The Strongest Attraction. (arXiv:2312.00033v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.00033
• Code URL: null
• Copy Paste: [[2312.00033]] DeFi Security: Turning The Weakest Link Into The Strongest Attraction(http://arxiv.org/abs/2312.00033)
• Summary:

  The primary innovation we pioneer -- focused on blockchain information security -- is called the Safe-House. The Safe-House is badly needed since there are many ongoing hacks and security concerns in the DeFi space right now. The Safe-House is a piece of engineering sophistication that utilizes existing blockchain principles to bring about greater security when customer assets are moved around. The Safe-House logic is easily implemented as smart contracts on any decentralized system. The amount of funds at risk from both internal and external parties -- and hence the maximum one time loss -- is guaranteed to stay within the specified limits based on cryptographic fundamentals.

  To improve the safety of the Safe-House even further, we adapt the one time password (OPT) concept to operate using blockchain technology. Well suited to blockchain cryptographic nuances, our secondary advancement can be termed the one time next time password (OTNTP) mechanism. The OTNTP is designed to complement the Safe-House making it even more safe.

  We provide a detailed threat assessment model -- discussing the risks faced by DeFi protocols and the specific risks that apply to blockchain fund management -- and give technical arguments regarding how these threats can be overcome in a robust manner. We discuss how the Safe-House can participate with other external yield generation protocols in a secure way. We provide reasons for why the Safe-House increases safety without sacrificing the efficiency of operation. We start with a high level intuitive description of the landscape, the corresponding problems and our solutions. We then supplement this overview with detailed discussions including the corresponding mathematical formulations and pointers for technological implementation. This approach ensures that the article is accessible to a broad audience.

Title: Enhancing IoT Security via Automatic Network Traffic Analysis: The Transition from Machine Learning to Deep Learning. (arXiv:2312.00034v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.00034
• Code URL: null
• Copy Paste: [[2312.00034]] Enhancing IoT Security via Automatic Network Traffic Analysis: The Transition from Machine Learning to Deep Learning(http://arxiv.org/abs/2312.00034)
• Summary:

  This work provides a comparative analysis illustrating how Deep Learning (DL) surpasses Machine Learning (ML) in addressing tasks within Internet of Things (IoT), such as attack classification and device-type identification. Our approach involves training and evaluating a DL model using a range of diverse IoT-related datasets, allowing us to gain valuable insights into how adaptable and practical these models can be when confronted with various IoT configurations. We initially convert the unstructured network traffic data from IoT networks, stored in PCAP files, into images by processing the packet data. This conversion process adapts the data to meet the criteria of DL classification methods. The experiments showcase the ability of DL to surpass the constraints tied to manually engineered features, achieving superior results in attack detection and maintaining comparable outcomes in device-type identification. Additionally, a notable feature extraction time difference becomes evident in the experiments: traditional methods require around 29 milliseconds per data packet, while DL accomplishes the same task in just 2.9 milliseconds. The significant time gap, DL's superior performance, and the recognized limitations of manually engineered features, presents a compelling call to action within the IoT community. This encourages us to shift from exploring new IoT features for each dataset to addressing the challenges of integrating DL into IoT, making it a more efficient solution for real-world IoT scenarios.

Title: Acoustic Cybersecurity: Exploiting Voice-Activated Systems. (arXiv:2312.00039v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.00039
• Code URL: null
• Copy Paste: [[2312.00039]] Acoustic Cybersecurity: Exploiting Voice-Activated Systems(http://arxiv.org/abs/2312.00039)
• Summary:

  In this study, we investigate the emerging threat of inaudible acoustic attacks targeting digital voice assistants, a critical concern given their projected prevalence to exceed the global population by 2024. Our research extends the feasibility of these attacks across various platforms like Amazon's Alexa, Android, iOS, and Cortana, revealing significant vulnerabilities in smart devices. The twelve attack vectors identified include successful manipulation of smart home devices and automotive systems, potential breaches in military communication, and challenges in critical infrastructure security. We quantitatively show that attack success rates hover around 60%, with the ability to activate devices remotely from over 100 feet away. Additionally, these attacks threaten critical infrastructure, emphasizing the need for multifaceted defensive strategies combining acoustic shielding, advanced signal processing, machine learning, and robust user authentication to mitigate these risks.

Title: A Scale-out Decentralized Blockchain Ledger System for Web3.0. (arXiv:2312.00281v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.00281
• Code URL: null
• Copy Paste: [[2312.00281]] A Scale-out Decentralized Blockchain Ledger System for Web3(http://arxiv.org/abs/2312.00281)
• Summary:

  The development of underlying technologies in blockchain mostly revolves around a difficult problem: how to enhance the performance of the system and reduce various costs of nodes (such as communication, storage and verification) without compromising the system's security and decentralization. Various layer-1 and layer-2 protocols have provided excellent solutions for this challenge. However, they cannot yet be considered as a ``silver bullet". This paper proposes EZchain -- a novel decentralized ``scale-out" ledger system designed for web3.0, aiming to enable blockchain technology to truly support ledger applications in large-scale fully decentralized networks. Without compromising security and decentralization, EZchain successfully accomplishes the following milestones: 1) Scalability: The theoretical throughput of EZchain can be infinitely expanded, nearly unaffected by bandwidth and other resource constraints. 2) Consumer-Grade Hardware Compatibility: EZchain is designed to be compatible with consumer-grade hardware, supporting storage, computation, and verification requirements. 3) Efficient Transaction Confirmation: EZchain strives to maintain transaction confirmation delays within one minute. Our prototype experiment demonstrates that under typical daily bandwidth network conditions, EZchain's performance in all aspects approaches that of the accounts in centralized payment systems. This provides a solid infrastructure for realizing mobile payments in web3.0.

Title: The Impact of Privacy and Security Attitudes and Concerns of Travellers on Their Willingness to Use Mobility-as-a-Service Systems. (arXiv:2312.00519v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.00519
• Code URL: null
• Copy Paste: [[2312.00519]] The Impact of Privacy and Security Attitudes and Concerns of Travellers on Their Willingness to Use Mobility-as-a-Service Systems(http://arxiv.org/abs/2312.00519)
• Summary:

  This paper reports results from an online survey on the impact of travellers' privacy and security attitudes and concerns on their willingness to use mobility-as-a-service (MaaS) systems. This study is part of a larger project that aims at investigating barriers to potential MaaS uptake. The online survey was designed to cover data privacy and security attitudes and concerns as well as a variety of socio-psychological and socio-demographic variables associated with travellers' intentions to use MaaS systems. The study involved $n=320$ UK participants recruited via the Prolific survey platform. Overall, correlation analysis and a multiple regression model indicated that, neither attitudes nor concerns of participants over the privacy and security of personal data would significantly impact their decisions to use MaaS systems, which was an unexpected result, however, their trust in (commercial and governmental) websites would. Another surprising result is that, having been a victim of improper invasion of privacy did not appear to affect individuals' intentions to use MaaS systems, whereas frequency with which one heard about misuse of personal data did. Implications of the results and future directions are also discussed, e.g., MaaS providers are encouraged to work on improving the trustworthiness of their corporate image.

Title: Hiding in text/plain sight: Security defences of Tor Onion Services. (arXiv:2312.00545v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.00545
• Code URL: null
• Copy Paste: [[2312.00545]] Hiding in text/plain sight: Security defences of Tor Onion Services(http://arxiv.org/abs/2312.00545)
• Summary:

  Tor Onion Services are a way to host websites and other internet services anonymously. Onion Services are often used to bypass internet censorship and provide information services to users in oppressive regimes. This paper presents an analysis of the security defences deployed on these Onion Services. Onion Services tend to have better security policy than sites on the clear web. However they lag behind in the deployment of HTTPS, a key defence to ensuring the security of users of such services.

Title: A Holistic Approach for Trustworthy Distributed Systems with WebAssembly and TEEs. (arXiv:2312.00702v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.00702
• Code URL: null
• Copy Paste: [[2312.00702]] A Holistic Approach for Trustworthy Distributed Systems with WebAssembly and TEEs(http://arxiv.org/abs/2312.00702)
• Summary:

  Publish/subscribe systems play a key role in enabling communication between numerous devices in distributed and large-scale architectures. While widely adopted, securing such systems often trades portability for additional integrity and attestation guarantees. Trusted Execution Environments (TEEs) offer a potential solution with enclaves to enhance security and trust. However, application development for TEEs is complex, and many existing solutions are tied to specific TEE architectures, limiting adaptability. Current communication protocols also inadequately manage attestation proofs or expose essential attestation information. This paper introduces a novel approach using WebAssembly to address these issues, a key enabling technology nowadays capturing academia and industry attention. We present the design of a portable and fully attested publish/subscribe middleware system as a holistic approach for trustworthy and distributed communication between various systems. Based on this proposal, we have implemented and evaluated in-depth a fully-fledged publish/subscribe broker running within Intel SGX, compiled in WebAssembly, and built on top of industry-battled frameworks and standards, i.e., MQTT and TLS protocols. Our extended TLS protocol preserves the privacy of attestation information, among other benefits. Our experimental results showcase most overheads, revealing a 1.55x decrease in message throughput when using a trusted broker. We open-source the contributions of this work to the research community to facilitate experimental reproducibility.

Title: Forecasting Trends in Food Security: a Reservoir Computing Approach. (arXiv:2312.00626v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.00626
• Code URL: null
• Copy Paste: [[2312.00626]] Forecasting Trends in Food Security: a Reservoir Computing Approach(http://arxiv.org/abs/2312.00626)
• Summary:

  Early warning systems are an essential tool for effective humanitarian action. Advance warnings on impending disasters facilitate timely and targeted response which help save lives, livelihoods, and scarce financial resources. In this work we present a new quantitative methodology to forecast levels of food consumption for 60 consecutive days, at the sub-national level, in four countries: Mali, Nigeria, Syria, and Yemen. The methodology is built on publicly available data from the World Food Programme's integrated global hunger monitoring system which collects, processes, and displays daily updates on key food security metrics, conflict, weather events, and other drivers of food insecurity across 90 countries (https://hungermap.wfp.org/). In this study, we assessed the performance of various models including ARIMA, XGBoost, LSTMs, CNNs, and Reservoir Computing (RC), by comparing their Root Mean Squared Error (RMSE) metrics. This comprehensive analysis spanned classical statistical, machine learning, and deep learning approaches. Our findings highlight Reservoir Computing as a particularly well-suited model in the field of food security given both its notable resistance to over-fitting on limited data samples and its efficient training capabilities. The methodology we introduce establishes the groundwork for a global, data-driven early warning system designed to anticipate and detect food insecurity.

privacy

Title: SynFundus: Generating a synthetic fundus images dataset with millions of samples and multi-disease annotations. (arXiv:2312.00377v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00377
• Code URL: null
• Copy Paste: [[2312.00377]] SynFundus: Generating a synthetic fundus images dataset with millions of samples and multi-disease annotations(http://arxiv.org/abs/2312.00377)
• Summary:

  In the field of medical imaging, the scarcity of large-scale datasets due to privacy restrictions stands as a significant barrier to develop large models for medical. To address this issue, we introduce SynFundus-1M, a high-quality synthetic dataset with over 1 million retinal fundus images and extensive disease and pathologies annotations, which is generated by a Denoising Diffusion Probabilistic Model. The SynFundus-Generator and SynFundus-1M achieve superior Frechet Inception Distance (FID) scores compared to existing methods on main-stream public real datasets. Furthermore, the ophthalmologists evaluation validate the difficulty in discerning these synthetic images from real ones, confirming the SynFundus-1M's authenticity. Through extensive experiments, we demonstrate that both CNN and ViT can benifit from SynFundus-1M by pretraining or training directly. Compared to datasets like ImageNet or EyePACS, models train on SynFundus-1M not only achieve better performance but also faster convergence on various downstream tasks.

Title: Object Detector Differences when using Synthetic and Real Training Data. (arXiv:2312.00694v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00694
• Code URL: https://github.com/ljungqvistmartin/datasplits
• Copy Paste: [[2312.00694]] Object Detector Differences when using Synthetic and Real Training Data(http://arxiv.org/abs/2312.00694)
• Summary:

  To train well-performing generalizing neural networks, sufficiently large and diverse datasets are needed. Collecting data while adhering to privacy legislation becomes increasingly difficult and annotating these large datasets is both a resource-heavy and time-consuming task. An approach to overcome these difficulties is to use synthetic data since it is inherently scalable and can be automatically annotated. However, how training on synthetic data affects the layers of a neural network is still unclear. In this paper, we train the YOLOv3 object detector on real and synthetic images from city environments. We perform a similarity analysis using Centered Kernel Alignment (CKA) to explore the effects of training on synthetic data on a layer-wise basis. The analysis captures the architecture of the detector while showing both different and similar patterns between different models. With this similarity analysis we want to give insights on how training synthetic data affects each layer and to give a better understanding of the inner workings of complex neural networks. The results show that the largest similarity between a detector trained on real data and a detector trained on synthetic data was in the early layers, and the largest difference was in the head part. The results also show that no major difference in performance or similarity could be seen between frozen and unfrozen backbone.

Title: Preserving The Safety And Confidentiality Of Data Mining Information In Health Care: A literature review. (arXiv:2312.00016v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.00016
• Code URL: null
• Copy Paste: [[2312.00016]] Preserving The Safety And Confidentiality Of Data Mining Information In Health Care: A literature review(http://arxiv.org/abs/2312.00016)
• Summary:

  Daily, massive volume of data are produced due to the internet of things' rapid development, which has now permeated the healthcare industry. Recent advances in data mining have spawned a new field of a study dubbed privacy-preserving data mining (PPDM). PPDM technique or approach enables the extraction of actionable insight from enormous volume of data while safeguarding the privacy of individual information and benefiting the entire society Medical research has taken a new course as a result of data mining with healthcare data to detect diseases earlier and improve patient care. Data integration necessitates the sharing of sensitive patient information. However, substantial privacy issues are raised in connection with the storage and transmission of potentially sensitive information. Disclosing sensitive information infringes on patients' privacy. This paper aims to conduct a review of related work on privacy-preserving mechanisms, data protection regulations, and mitigating tactics. The review concluded that no single strategy outperforms all others. Hence, future research should focus on adequate techniques for privacy solutions in the age of massive medical data and the standardization of evaluation standards.

Title: Privacy-Preserving Load Forecasting via Personalized Model Obfuscation. (arXiv:2312.00036v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.00036
• Code URL: null
• Copy Paste: [[2312.00036]] Privacy-Preserving Load Forecasting via Personalized Model Obfuscation(http://arxiv.org/abs/2312.00036)
• Summary:

  The widespread adoption of smart meters provides access to detailed and localized load consumption data, suitable for training building-level load forecasting models. To mitigate privacy concerns stemming from model-induced data leakage, federated learning (FL) has been proposed. This paper addresses the performance challenges of short-term load forecasting models trained with FL on heterogeneous data, emphasizing privacy preservation through model obfuscation. Our proposed algorithm, Privacy Preserving Federated Learning (PPFL), incorporates personalization layers for localized training at each smart meter. Additionally, we employ a differentially private mechanism to safeguard against data leakage from shared layers. Simulations on the NREL ComStock dataset corroborate the effectiveness of our approach.

Title: Hashmarks: Privacy-Preserving Benchmarks for High-Stakes AI Evaluation. (arXiv:2312.00645v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.00645
• Code URL: null
• Copy Paste: [[2312.00645]] Hashmarks: Privacy-Preserving Benchmarks for High-Stakes AI Evaluation(http://arxiv.org/abs/2312.00645)
• Summary:

  There is a growing need to gain insight into language model capabilities that relate to sensitive topics, such as bioterrorism or cyberwarfare. However, traditional open source benchmarks are not fit for the task, due to the associated practice of publishing the correct answers in human-readable form. At the same time, enforcing mandatory closed-quarters evaluations might stifle development and erode trust. In this context, we propose hashmarking, a protocol for evaluating language models in the open without having to disclose the correct answers. In its simplest form, a hashmark is a benchmark whose reference solutions have been cryptographically hashed prior to publication. Following an overview of the proposed evaluation protocol, we go on to assess its resilience against traditional attack vectors (e.g. rainbow table attacks), as well as against failure modes unique to increasingly capable generative models.

Title: A Causality-Aware Pattern Mining Scheme for Group Activity Recognition in a Pervasive Sensor Space. (arXiv:2312.00404v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.00404
• Code URL: null
• Copy Paste: [[2312.00404]] A Causality-Aware Pattern Mining Scheme for Group Activity Recognition in a Pervasive Sensor Space(http://arxiv.org/abs/2312.00404)
• Summary:

  Human activity recognition (HAR) is a key challenge in pervasive computing and its solutions have been presented based on various disciplines. Specifically, for HAR in a smart space without privacy and accessibility issues, data streams generated by deployed pervasive sensors are leveraged. In this paper, we focus on a group activity by which a group of users perform a collaborative task without user identification and propose an efficient group activity recognition scheme which extracts causality patterns from pervasive sensor event sequences generated by a group of users to support as good recognition accuracy as the state-of-the-art graphical model. To filter out irrelevant noise events from a given data stream, a set of rules is leveraged to highlight causally related events. Then, a pattern-tree algorithm extracts frequent causal patterns by means of a growing tree structure. Based on the extracted patterns, a weighted sum-based pattern matching algorithm computes the likelihoods of stored group activities to the given test event sequence by means of matched event pattern counts for group activity recognition. We evaluate the proposed scheme using the data collected from our testbed and CASAS datasets where users perform their tasks on a daily basis and validate its effectiveness in a real environment. Experiment results show that the proposed scheme performs higher recognition accuracy and with a small amount of runtime overhead than the existing schemes.

Title: Target-agnostic Source-free Domain Adaptation for Regression Tasks. (arXiv:2312.00540v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.00540
• Code URL: null
• Copy Paste: [[2312.00540]] Target-agnostic Source-free Domain Adaptation for Regression Tasks(http://arxiv.org/abs/2312.00540)
• Summary:

  Unsupervised domain adaptation (UDA) seeks to bridge the domain gap between the target and source using unlabeled target data. Source-free UDA removes the requirement for labeled source data at the target to preserve data privacy and storage. However, work on source-free UDA assumes knowledge of domain gap distribution, and hence is limited to either target-aware or classification task. To overcome it, we propose TASFAR, a novel target-agnostic source-free domain adaptation approach for regression tasks. Using prediction confidence, TASFAR estimates a label density map as the target label distribution, which is then used to calibrate the source model on the target domain. We have conducted extensive experiments on four regression tasks with various domain gaps, namely, pedestrian dead reckoning for different users, image-based people counting in different scenes, housing-price prediction at different districts, and taxi-trip duration prediction from different departure points. TASFAR is shown to substantially outperform the state-of-the-art source-free UDA approaches by averagely reducing 22% errors for the four tasks and achieve notably comparable accuracy as source-based UDA without using source data.

protect

Title: Probabilistic Copyright Protection Can Fail for Text-to-Image Generative Models. (arXiv:2312.00057v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.00057
• Code URL: https://github.com/south7x/va3
• Copy Paste: [[2312.00057]] Probabilistic Copyright Protection Can Fail for Text-to-Image Generative Models(http://arxiv.org/abs/2312.00057)
• Summary:

  The booming use of text-to-image generative models has raised concerns about their high risk of producing copyright-infringing content. While probabilistic copyright protection methods provide a probabilistic guarantee against such infringement, in this paper, we introduce Virtually Assured Amplification Attack (VA3), a novel online attack framework that exposes the vulnerabilities of these protection mechanisms. The proposed framework significantly amplifies the probability of generating infringing content on the sustained interactions with generative models and a lower-bounded success probability of each engagement. Our theoretical and experimental results demonstrate the effectiveness of our approach and highlight the potential risk of implementing probabilistic copyright protection in practical applications of text-to-image generative models. Code is available at https://github.com/South7X/VA3.

Title: Can Protective Perturbation Safeguard Personal Data from Being Exploited by Stable Diffusion?. (arXiv:2312.00084v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00084
• Code URL: null
• Copy Paste: [[2312.00084]] Can Protective Perturbation Safeguard Personal Data from Being Exploited by Stable Diffusion?(http://arxiv.org/abs/2312.00084)
• Summary:

  Stable Diffusion has established itself as a foundation model in generative AI artistic applications, receiving widespread research and application. Some recent fine-tuning methods have made it feasible for individuals to implant personalized concepts onto the basic Stable Diffusion model with minimal computational costs on small datasets. However, these innovations have also given rise to issues like facial privacy forgery and artistic copyright infringement. In recent studies, researchers have explored the addition of imperceptible adversarial perturbations to images to prevent potential unauthorized exploitation and infringements when personal data is used for fine-tuning Stable Diffusion. Although these studies have demonstrated the ability to protect images, it is essential to consider that these methods may not be entirely applicable in real-world scenarios. In this paper, we systematically evaluate the use of perturbations to protect images within a practical threat model. The results suggest that these approaches may not be sufficient to safeguard image privacy and copyright effectively. Furthermore, we introduce a purification method capable of removing protected perturbations while preserving the original image structure to the greatest extent possible. Experiments reveal that Stable Diffusion can effectively learn from purified images over all protective methods.

Title: Tokenized Model: A Blockchain-Empowered Decentralized Model Ownership Verification Platform. (arXiv:2312.00048v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.00048
• Code URL: null
• Copy Paste: [[2312.00048]] Tokenized Model: A Blockchain-Empowered Decentralized Model Ownership Verification Platform(http://arxiv.org/abs/2312.00048)
• Summary:

  With the development of practical deep learning models like generative AI, their excellent performance has brought huge economic value. For instance, ChatGPT has attracted more than 100 million users in three months. Since the model training requires a lot of data and computing power, a well-performing deep learning model is behind a huge effort and cost. Facing various model attacks, unauthorized use and abuse from the network that threaten the interests of model owners, in addition to considering legal and other administrative measures, it is equally important to protect the model's copyright from the technical means. By using the model watermarking technology, we point out the possibility of building a unified platform for model ownership verification. Given the application history of blockchain in copyright verification and the drawbacks of a centralized third-party, this paper considers combining model watermarking technology and blockchain to build a unified model copyright protection platform. By a new solution we called Tokenized Model, it protects the model's copyright by reliable ownership record and verification mechanism. It also promotes the financial value of model by constructing the model's transaction process and contribution shares of a model. In the typical case study, we also study the various performance under usual scenario to verify the effectiveness of this platform.

defense

Title: Optimal Attack and Defense for Reinforcement Learning. (arXiv:2312.00198v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.00198
• Code URL: null
• Copy Paste: [[2312.00198]] Optimal Attack and Defense for Reinforcement Learning(http://arxiv.org/abs/2312.00198)
• Summary:

  To ensure the usefulness of Reinforcement Learning (RL) in real systems, it is crucial to ensure they are robust to noise and adversarial attacks. In adversarial RL, an external attacker has the power to manipulate the victim agent's interaction with the environment. We study the full class of online manipulation attacks, which include (i) state attacks, (ii) observation attacks (which are a generalization of perceived-state attacks), (iii) action attacks, and (iv) reward attacks. We show the attacker's problem of designing a stealthy attack that maximizes its own expected reward, which often corresponds to minimizing the victim's value, is captured by a Markov Decision Process (MDP) that we call a meta-MDP since it is not the true environment but a higher level environment induced by the attacked interaction. We show that the attacker can derive optimal attacks by planning in polynomial time or learning with polynomial sample complexity using standard RL techniques. We argue that the optimal defense policy for the victim can be computed as the solution to a stochastic Stackelberg game, which can be further simplified into a partially-observable turn-based stochastic game (POTBSG). Neither the attacker nor the victim would benefit from deviating from their respective optimal policies, thus such solutions are truly robust. Although the defense problem is NP-hard, we show that optimal Markovian defenses can be computed (learned) in polynomial time (sample complexity) in many scenarios.

attack

Title: Presentation Attack detection using Wavelet Transform and Deep Residual Neural Net. (arXiv:2312.00040v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.00040
• Code URL: null
• Copy Paste: [[2312.00040]] Presentation Attack detection using Wavelet Transform and Deep Residual Neural Net(http://arxiv.org/abs/2312.00040)
• Summary:

  Biometric authentication is becoming more prevalent for secured authentication systems. However, the biometric substances can be deceived by the imposters in several ways. Among other imposter attacks, print attacks, mask attacks, and replay attacks fall under the presentation attack category. The bio-metric images, especially the iris and face, are vulnerable to different presentation attacks. This research applies deep learning approaches to mitigate presentation attacks in a biometric access control system. Our contribution in this paper is two-fold: First, we applied the wavelet transform to extract the features from the biometric images. Second, we modified the deep residual neural net and applied it to the spoof datasets in an attempt to detect the presentation attacks. This research applied the proposed approach to biometric spoof datasets, namely ATVS, CASIA two class, and CASIA cropped image sets. The datasets used in this research contain images that are captured in both a controlled and uncontrolled environment along with different resolutions and sizes. We obtained the best accuracy of 93% on the ATVS Iris datasets. For CASIA two class and CASIA cropped datasets, we achieved test accuracies of 91% and 82%, respectively.

Title: Presentation Attack Detection using Convolutional Neural Networks and Local Binary Patterns. (arXiv:2312.00041v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.00041
• Code URL: null
• Copy Paste: [[2312.00041]] Presentation Attack Detection using Convolutional Neural Networks and Local Binary Patterns(http://arxiv.org/abs/2312.00041)
• Summary:

  The use of biometrics to authenticate users and control access to secure areas has become extremely popular in recent years, and biometric access control systems are frequently used by both governments and private corporations. However, these systems may represent risks to security when deployed without considering the possibility of biometric presentation attacks (also known as spoofing). Presentation attacks are a serious threat because they do not require significant time, expense, or skill to carry out while remaining effective against many biometric systems in use today. This research compares three different software-based methods for facial and iris presentation attack detection in images. The first method uses Inception-v3, a pre-trained deep Convolutional Neural Network (CNN) made by Google for the ImageNet challenge, which is retrained for this problem. The second uses a shallow CNN based on a modified Spoofnet architecture, which is trained normally. The third is a texture-based method using Local Binary Patterns (LBP). The datasets used are the ATVS-FIr dataset, which contains real and fake iris images, and the CASIA Face Anti-Spoofing Dataset, which contains real images as well as warped photos, cut photos, and video replay presentation attacks. We also present a third set of results, based on cropped versions of the CASIA images.

Title: Improving the Robustness of Quantized Deep Neural Networks to White-Box Attacks using Stochastic Quantization and Information-Theoretic Ensemble Training. (arXiv:2312.00105v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00105
• Code URL: null
• Copy Paste: [[2312.00105]] Improving the Robustness of Quantized Deep Neural Networks to White-Box Attacks using Stochastic Quantization and Information-Theoretic Ensemble Training(http://arxiv.org/abs/2312.00105)
• Summary:

  Most real-world applications that employ deep neural networks (DNNs) quantize them to low precision to reduce the compute needs. We present a method to improve the robustness of quantized DNNs to white-box adversarial attacks. We first tackle the limitation of deterministic quantization to fixed ``bins'' by introducing a differentiable Stochastic Quantizer (SQ). We explore the hypothesis that different quantizations may collectively be more robust than each quantized DNN. We formulate a training objective to encourage different quantized DNNs to learn different representations of the input image. The training objective captures diversity and accuracy via mutual information between ensemble members. Through experimentation, we demonstrate substantial improvement in robustness against $L_\infty$ attacks even if the attacker is allowed to backpropagate through SQ (e.g., > 50\% accuracy to PGD(5/255) on CIFAR10 without adversarial training), compared to vanilla DNNs as well as existing ensembles of quantized DNNs. We extend the method to detect attacks and generate robustness profiles in the adversarial information plane (AIP), towards a unified analysis of different threat models by correlating the MI and accuracy.

Title: Universal Backdoor Attacks. (arXiv:2312.00157v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.00157
• Code URL: null
• Copy Paste: [[2312.00157]] Universal Backdoor Attacks(http://arxiv.org/abs/2312.00157)
• Summary:

  Web-scraped datasets are vulnerable to data poisoning, which can be used for backdooring deep image classifiers during training. Since training on large datasets is expensive, a model is trained once and re-used many times. Unlike adversarial examples, backdoor attacks often target specific classes rather than any class learned by the model. One might expect that targeting many classes through a naive composition of attacks vastly increases the number of poison samples. We show this is not necessarily true and more efficient, universal data poisoning attacks exist that allow controlling misclassifications from any source class into any target class with a small increase in poison samples. Our idea is to generate triggers with salient characteristics that the model can learn. The triggers we craft exploit a phenomenon we call inter-class poison transferability, where learning a trigger from one class makes the model more vulnerable to learning triggers for other classes. We demonstrate the effectiveness and robustness of our universal backdoor attacks by controlling models with up to 6,000 classes while poisoning only 0.15% of the training dataset.

Title: Fool the Hydra: Adversarial Attacks against Multi-view Object Detection Systems. (arXiv:2312.00173v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00173
• Code URL: null
• Copy Paste: [[2312.00173]] Fool the Hydra: Adversarial Attacks against Multi-view Object Detection Systems(http://arxiv.org/abs/2312.00173)
• Summary:

  Adversarial patches exemplify the tangible manifestation of the threat posed by adversarial attacks on Machine Learning (ML) models in real-world scenarios. Robustness against these attacks is of the utmost importance when designing computer vision applications, especially for safety-critical domains such as CCTV systems. In most practical situations, monitoring open spaces requires multi-view systems to overcome acquisition challenges such as occlusion handling. Multiview object systems are able to combine data from multiple views, and reach reliable detection results even in difficult environments. Despite its importance in real-world vision applications, the vulnerability of multiview systems to adversarial patches is not sufficiently investigated. In this paper, we raise the following question: Does the increased performance and information sharing across views offer as a by-product robustness to adversarial patches? We first conduct a preliminary analysis showing promising robustness against off-the-shelf adversarial patches, even in an extreme setting where we consider patches applied to all views by all persons in Wildtrack benchmark. However, we challenged this observation by proposing two new attacks: (i) In the first attack, targeting a multiview CNN, we maximize the global loss by proposing gradient projection to the different views and aggregating the obtained local gradients. (ii) In the second attack, we focus on a Transformer-based multiview framework. In addition to the focal loss, we also maximize the transformer-specific loss by dissipating its attention blocks. Our results show a large degradation in the detection performance of victim multiview systems with our first patch attack reaching an attack success rate of 73% , while our second proposed attack reduced the performance of its target detector by 62%

Title: Bergeron: Combating Adversarial Attacks through a Conscience-Based Alignment Framework. (arXiv:2312.00029v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.00029
• Code URL: null
• Copy Paste: [[2312.00029]] Bergeron: Combating Adversarial Attacks through a Conscience-Based Alignment Framework(http://arxiv.org/abs/2312.00029)
• Summary:

  Modern Large language models (LLMs) can still generate responses that may not be aligned with human expectations or values. While many weight-based alignment methods have been proposed, many of them still leave models vulnerable to attacks when used on their own. To help mitigate this issue, we introduce Bergeron, a framework designed to improve the robustness of LLMs against adversarial attacks. Bergeron employs a two-tiered architecture. Here, a secondary LLM serves as a simulated conscience that safeguards a primary LLM. We do this by monitoring for and correcting potentially harmful text within both the prompt inputs and the generated outputs of the primary LLM. Empirical evaluation shows that Bergeron can improve the alignment and robustness of several popular LLMs without costly fine-tuning. It aids both open-source and black-box LLMs by complementing and reinforcing their existing alignment training.

Title: Enhancing ML-Based DoS Attack Detection Through Combinatorial Fusion Analysis. (arXiv:2312.00006v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.00006
• Code URL: null
• Copy Paste: [[2312.00006]] Enhancing ML-Based DoS Attack Detection Through Combinatorial Fusion Analysis(http://arxiv.org/abs/2312.00006)
• Summary:

  Mitigating Denial-of-Service (DoS) attacks is vital for online service security and availability. While machine learning (ML) models are used for DoS attack detection, new strategies are needed to enhance their performance. We suggest an innovative method, combinatorial fusion, which combines multiple ML models using advanced algorithms. This includes score and rank combinations, weighted techniques, and diversity strength of scoring systems. Through rigorous evaluations, we demonstrate the effectiveness of this fusion approach, considering metrics like precision, recall, and F1-score. We address the challenge of low-profiled attack classification by fusing models to create a comprehensive solution. Our findings emphasize the potential of this approach to improve DoS attack detection and contribute to stronger defense mechanisms.

Title: Revolutionizing Forensic Toolmark Analysis: An Objective and Transparent Comparison Algorithm. (arXiv:2312.00032v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.00032
• Code URL: null
• Copy Paste: [[2312.00032]] Revolutionizing Forensic Toolmark Analysis: An Objective and Transparent Comparison Algorithm(http://arxiv.org/abs/2312.00032)
• Summary:

  Forensic toolmark comparisons are currently performed subjectively by humans, which leads to a lack of consistency and accuracy. There is little evidence that examiners can determine whether pairs of marks were made by the same tool or different tools. There is also little evidence that they can make this classification when marks are made under different conditions, such as different angles of attack or direction of mark generation. We generate original toolmark data in 3D, extract the signal from each toolmarks, and train an algorithm to compare toolmark signals objectively. We find that toolmark signals cluster by tool, and not by angle or direction. That is, the variability within tool, regardless of angle/direction, is smaller than the variability between tools. The known-match and known-non-match densities of the similarities of pairs of marks have a small overlap, even when accounting for dependencies in the data, making them a useful instrument for determining whether a new pair of marks was made by the same tool. We provide a likelihood ratio approach as a formal method for comparing toolmark signals with a measure of uncertainty. This empirically trained, open-source method can be used by forensic examiners to compare toolmarks objectively and thus improve the reliability of toolmark comparisons. This can, in turn, reduce miscarriages of justice in the criminal justice system.

Title: MIA-BAD: An Approach for Enhancing Membership Inference Attack and its Mitigation with Federated Learning. (arXiv:2312.00051v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.00051
• Code URL: null
• Copy Paste: [[2312.00051]] MIA-BAD: An Approach for Enhancing Membership Inference Attack and its Mitigation with Federated Learning(http://arxiv.org/abs/2312.00051)
• Summary:

  The membership inference attack (MIA) is a popular paradigm for compromising the privacy of a machine learning (ML) model. MIA exploits the natural inclination of ML models to overfit upon the training data. MIAs are trained to distinguish between training and testing prediction confidence to infer membership information. Federated Learning (FL) is a privacy-preserving ML paradigm that enables multiple clients to train a unified model without disclosing their private data. In this paper, we propose an enhanced Membership Inference Attack with the Batch-wise generated Attack Dataset (MIA-BAD), a modification to the MIA approach. We investigate that the MIA is more accurate when the attack dataset is generated batch-wise. This quantitatively decreases the attack dataset while qualitatively improving it. We show how training an ML model through FL, has some distinct advantages and investigate how the threat introduced with the proposed MIA-BAD approach can be mitigated with FL approaches. Finally, we demonstrate the qualitative effects of the proposed MIA-BAD methodology by conducting extensive experiments with various target datasets, variable numbers of federated clients, and training batch sizes.

Title: MalDicom: A Memory Forensic Framework for Detecting Malicious Payload in DICOM Files. (arXiv:2312.00483v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.00483
• Code URL: null
• Copy Paste: [[2312.00483]] MalDicom: A Memory Forensic Framework for Detecting Malicious Payload in DICOM Files(http://arxiv.org/abs/2312.00483)
• Summary:

  Digital Imaging and Communication System (DICOM) is widely used throughout the public health sector for portability in medical imaging. However, these DICOM files have vulnerabilities present in the preamble section. Successful exploitation of these vulnerabilities can allow attackers to embed executable codes in the 128-Byte preamble of DICOM files. Embedding the malicious executable will not interfere with the readability or functionality of DICOM imagery. However, it will affect the underline system silently upon viewing these files. This paper shows the infiltration of Windows malware executables into DICOM files. On viewing the files, the malicious DICOM will get executed and eventually infect the entire hospital network through the radiologist's workstation. The code injection process of executing malware in DICOM files affects the hospital networks and workstations' memory. Memory forensics for the infected radiologist's workstation is crucial as it can detect which malware disrupts the hospital environment, and future detection methods can be deployed. In this paper, we consider the machine learning (ML) algorithms to conduct memory forensics on three memory dump categories: Trojan, Spyware, and Ransomware, taken from the CIC-MalMem-2022 dataset. We obtain the highest accuracy of 75\% with the Random Forest model. For estimating the feature importance for ML model prediction, we leveraged the concept of Shapley values.

Title: Classification of cyber attacks on IoT and ubiquitous computing devices. (arXiv:2312.00686v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.00686
• Code URL: null
• Copy Paste: [[2312.00686]] Classification of cyber attacks on IoT and ubiquitous computing devices(http://arxiv.org/abs/2312.00686)
• Summary:

  As the Internet of Things (IoT) has become truly ubiquitous, so has the surrounding threat landscape. However, while the security of classical computing systems has significantly matured in the last decades, IoT cybersecurity is still typically low or fully neglected. This paper provides a classification of IoT malware. Major targets and used exploits for attacks are identified and referred to the specific malware. The lack of standard definitions of IoT devices and, therefore, security goals has been identified during this research as a profound barrier in advancing IoT cybersecurity. Furthermore, standardized reporting of IoT malware by trustworthy sources is required in the field. The majority of current IoT attacks continue to be of comparably low effort and level of sophistication and could be mitigated by existing technical measures.

Title: Crystal: Enhancing Blockchain Mining Transparency with Quorum Certificate. (arXiv:2312.00741v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.00741
• Code URL: null
• Copy Paste: [[2312.00741]] Crystal: Enhancing Blockchain Mining Transparency with Quorum Certificate(http://arxiv.org/abs/2312.00741)
• Summary:

  Researchers have discovered a series of theoretical attacks against Bitcoin's Nakamoto consensus; the most damaging ones are selfish mining, double-spending, and consistency delay attacks. These attacks have one common cause: block withholding. This paper proposes Crystal, which leverages quorum certificates to resist block withholding misbehavior. Crystal continuously elects committees from miners and requires each block to have a quorum certificate, i.e., a set of signatures issued by members of its committee. Consequently, an attacker has to publish its blocks to obtain quorum certificates, rendering block withholding impossible. To build Crystal, we design a novel two-round committee election in a Sybil-resistant, unpredictable and non-interactive way, and a reward mechanism to incentivize miners to follow the protocol. Our analysis and evaluations show that Crystal can significantly mitigate selfish mining and double-spending attacks. For example, in Bitcoin, an attacker with 30% of the total computation power will succeed in double-spending attacks with a probability of 15.6% to break the 6-confirmation rule; however, in Crystal, the success probability for the same attacker falls to 0.62%. We provide formal end-to-end safety proofs for Crystal, ensuring no unknown attacks will be introduced. To the best of our knowledge, Crystal is the first protocol that prevents selfish mining and double-spending attacks while providing safety proof.

Title: Reduction from sparse LPN to LPN, Dual Attack 3.0. (arXiv:2312.00747v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.00747
• Code URL: null
• Copy Paste: [[2312.00747]] Reduction from sparse LPN to LPN, Dual Attack 3(http://arxiv.org/abs/2312.00747)
• Summary:

  The security of code-based cryptography relies primarily on the hardness of decoding generic linear codes. Until very recently, all the best algorithms for solving the decoding problem were information set decoders (ISD). However, recently a new algorithm called RLPN-decoding which relies on a completely different approach was introduced and it has been shown that RLPN outperforms significantly ISD decoders for a rather large range of rates. This RLPN decoder relies on two ingredients, first reducing decoding to some underlying LPN problem, and then computing efficiently many parity-checks of small weight when restricted to some positions. We revisit RLPN-decoding by noticing that, in this algorithm, decoding is in fact reduced to a sparse-LPN problem, namely with a secret whose Hamming weight is small. Our new approach consists this time in making an additional reduction from sparse-LPN to plain-LPN with a coding approach inspired by coded-BKW. It outperforms significantly the ISD's and RLPN for code rates smaller than 0.42. This algorithm can be viewed as the code-based cryptography cousin of recent dual attacks in lattice-based cryptography. We depart completely from the traditional analysis of this kind of algorithm which uses a certain number of independence assumptions that have been strongly questioned recently in the latter domain. We give instead a formula for the LPNs noise relying on duality which allows to analyze the behavior of the algorithm by relying only on the analysis of a certain weight distribution. By using only a minimal assumption whose validity has been verified experimentally we are able to justify the correctness of our algorithm. This key tool, namely the duality formula, can be readily adapted to the lattice setting and is shown to give a simple explanation for some phenomena observed on dual attacks in lattices in [DP23].

robust

Title: SparseDC: Depth Completion from sparse and non-uniform inputs. (arXiv:2312.00097v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00097
• Code URL: https://github.com/whu-usi3dv/sparsedc
• Copy Paste: [[2312.00097]] SparseDC: Depth Completion from sparse and non-uniform inputs(http://arxiv.org/abs/2312.00097)
• Summary:

  We propose SparseDC, a model for Depth Completion of Sparse and non-uniform depth inputs. Unlike previous methods focusing on completing fixed distributions on benchmark datasets (e.g., NYU with 500 points, KITTI with 64 lines), SparseDC is specifically designed to handle depth maps with poor quality in real usage. The key contributions of SparseDC are two-fold. First, we design a simple strategy, called SFFM, to improve the robustness under sparse input by explicitly filling the unstable depth features with stable image features. Second, we propose a two-branch feature embedder to predict both the precise local geometry of regions with available depth values and accurate structures in regions with no depth. The key of the embedder is an uncertainty-based fusion module called UFFM to balance the local and long-term information extracted by CNNs and ViTs. Extensive indoor and outdoor experiments demonstrate the robustness of our framework when facing sparse and non-uniform input depths. The pre-trained model and code are available at https://github.com/WHU-USI3DV/SparseDC.

Title: Towards Unsupervised Representation Learning: Learning, Evaluating and Transferring Visual Representations. (arXiv:2312.00101v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00101
• Code URL: null
• Copy Paste: [[2312.00101]] Towards Unsupervised Representation Learning: Learning, Evaluating and Transferring Visual Representations(http://arxiv.org/abs/2312.00101)
• Summary:

  Unsupervised representation learning aims at finding methods that learn representations from data without annotation-based signals. Abstaining from annotations not only leads to economic benefits but may - and to some extent already does - result in advantages regarding the representation's structure, robustness, and generalizability to different tasks. In the long run, unsupervised methods are expected to surpass their supervised counterparts due to the reduction of human intervention and the inherently more general setup that does not bias the optimization towards an objective originating from specific annotation-based signals. While major advantages of unsupervised representation learning have been recently observed in natural language processing, supervised methods still dominate in vision domains for most tasks. In this dissertation, we contribute to the field of unsupervised (visual) representation learning from three perspectives: (i) Learning representations: We design unsupervised, backpropagation-free Convolutional Self-Organizing Neural Networks (CSNNs) that utilize self-organization- and Hebbian-based learning rules to learn convolutional kernels and masks to achieve deeper backpropagation-free models. (ii) Evaluating representations: We build upon the widely used (non-)linear evaluation protocol to define pretext- and target-objective-independent metrics for measuring and investigating the objective function mismatch between various unsupervised pretext tasks and target tasks. (iii) Transferring representations: We contribute CARLANE, the first 3-way sim-to-real domain adaptation benchmark for 2D lane detection, and a method based on prototypical self-supervised learning. Finally, we contribute a content-consistent unpaired image-to-image translation method that utilizes masks, global and local discriminators, and similarity sampling to mitigate content inconsistencies.

Title: Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering. (arXiv:2312.00109v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00109
• Code URL: null
• Copy Paste: [[2312.00109]] Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering(http://arxiv.org/abs/2312.00109)
• Summary:

  Neural rendering methods have significantly advanced photo-realistic 3D scene rendering in various academic and industrial applications. The recent 3D Gaussian Splatting method has achieved the state-of-the-art rendering quality and speed combining the benefits of both primitive-based representations and volumetric representations. However, it often leads to heavily redundant Gaussians that try to fit every training view, neglecting the underlying scene geometry. Consequently, the resulting model becomes less robust to significant view changes, texture-less area and lighting effects. We introduce Scaffold-GS, which uses anchor points to distribute local 3D Gaussians, and predicts their attributes on-the-fly based on viewing direction and distance within the view frustum. Anchor growing and pruning strategies are developed based on the importance of neural Gaussians to reliably improve the scene coverage. We show that our method effectively reduces redundant Gaussians while delivering high-quality rendering. We also demonstrates an enhanced capability to accommodate scenes with varying levels-of-detail and view-dependent observations, without sacrificing the rendering speed.

Title: Event-based Continuous Color Video Decompression from Single Frames. (arXiv:2312.00113v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00113
• Code URL: null
• Copy Paste: [[2312.00113]] Event-based Continuous Color Video Decompression from Single Frames(http://arxiv.org/abs/2312.00113)
• Summary:

  We present ContinuityCam, a novel approach to generate a continuous video from a single static RGB image, using an event camera. Conventional cameras struggle with high-speed motion capture due to bandwidth and dynamic range limitations. Event cameras are ideal sensors to solve this problem because they encode compressed change information at high temporal resolution. In this work, we propose a novel task called event-based continuous color video decompression, pairing single static color frames and events to reconstruct temporally continuous videos. Our approach combines continuous long-range motion modeling with a feature-plane-based synthesis neural integration model, enabling frame prediction at arbitrary times within the events. Our method does not rely on additional frames except for the initial image, increasing, thus, the robustness to sudden light changes, minimizing the prediction latency, and decreasing the bandwidth requirement. We introduce a novel single objective beamsplitter setup that acquires aligned images and events and a novel and challenging Event Extreme Decompression Dataset (E2D2) that tests the method in various lighting and motion profiles. We thoroughly evaluate our method through benchmarking reconstruction as well as various downstream tasks. Our approach significantly outperforms the event- and image- based baselines in the proposed task.

Title: REACT: Recognize Every Action Everywhere All At Once. (arXiv:2312.00188v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00188
• Code URL: null
• Copy Paste: [[2312.00188]] REACT: Recognize Every Action Everywhere All At Once(http://arxiv.org/abs/2312.00188)
• Summary:

  Group Activity Recognition (GAR) is a fundamental problem in computer vision, with diverse applications in sports video analysis, video surveillance, and social scene understanding. Unlike conventional action recognition, GAR aims to classify the actions of a group of individuals as a whole, requiring a deep understanding of their interactions and spatiotemporal relationships. To address the challenges in GAR, we present REACT (\textbf{R}ecognize \textbf{E}very \textbf{Act}ion Everywhere All At Once), a novel architecture inspired by the transformer encoder-decoder model explicitly designed to model complex contextual relationships within videos, including multi-modality and spatio-temporal features. Our architecture features a cutting-edge Vision-Language Encoder block for integrated temporal, spatial, and multi-modal interaction modeling. This component efficiently encodes spatiotemporal interactions, even with sparsely sampled frames, and recovers essential local information. Our Action Decoder Block refines the joint understanding of text and video data, allowing us to precisely retrieve bounding boxes, enhancing the link between semantics and visual reality. At the core, our Actor Fusion Block orchestrates a fusion of actor-specific data and textual features, striking a balance between specificity and context. Our method outperforms state-of-the-art GAR approaches in extensive experiments, demonstrating superior accuracy in recognizing and understanding group activities. Our architecture's potential extends to diverse real-world applications, offering empirical evidence of its performance gains. This work significantly advances the field of group activity recognition, providing a robust framework for nuanced scene comprehension.

Title: Raising the Bar of AI-generated Image Detection with CLIP. (arXiv:2312.00195v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00195
• Code URL: null
• Copy Paste: [[2312.00195]] Raising the Bar of AI-generated Image Detection with CLIP(http://arxiv.org/abs/2312.00195)
• Summary:

  Aim of this work is to explore the potential of pre-trained vision-language models (VLMs) for universal detection of AI-generated images. We develop a lightweight detection strategy based on CLIP features and study its performance in a wide variety of challenging scenarios. We find that, unlike previous belief, it is neither necessary nor convenient to use a large domain-specific dataset for training. On the contrary, by using only a handful of example images from a single generative model, a CLIP-based detector exhibits a surprising generalization ability and high robustness across several different architectures, including recent commercial tools such as Dalle-3, Midjourney v5, and Firefly. We match the SoTA on in-distribution data, and improve largely above it in terms of generalization to out-of-distribution data (+6% in terms of AUC) and robustness to impaired/laundered data (+13%). Our project is available at https://grip-unina.github.io/ClipBased-SyntheticImageDetection/

Title: Adaptability of Computer Vision at the Tactical Edge: Addressing Environmental Uncertainty. (arXiv:2312.00269v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00269
• Code URL: null
• Copy Paste: [[2312.00269]] Adaptability of Computer Vision at the Tactical Edge: Addressing Environmental Uncertainty(http://arxiv.org/abs/2312.00269)
• Summary:

  Computer Vision (CV) systems are increasingly being adopted into Command and Control (C2) systems to improve intelligence analysis on the battlefield, the tactical edge. CV systems leverage Artificial Intelligence (AI) algorithms to help visualize and interpret the environment, enhancing situational awareness. However, the adaptability of CV systems at the tactical edge remains challenging due to rapidly changing environments and objects which can confuse the deployed models. A CV model leveraged in this environment can become uncertain in its predictions, as the environment and the objects existing in the environment begin to change. Additionally, mission objectives can rapidly change leading to adjustments in technology, camera angles, and image resolutions. All of which can negatively affect the performance of and potentially introduce uncertainty into the system. When the training environment and/or technology differs from the deployment environment, CV models can perform unexpectedly. Unfortunately, most scenarios at the tactical edge do not incorporate Uncertainty Quantification (UQ) into their deployed C2 and CV systems. This concept paper explores the idea of synchronizing robust data operations and model fine-tuning driven by UQ all at the tactical edge. Specifically, curating datasets and training child models based on the residuals of predictions, using these child models to calculate prediction intervals (PI), and then using these PI to calibrate the deployed models. By incorporating UQ into the core operations surrounding C2 and CV systems at the tactical edge, we can help drive purposeful adaptability on the battlefield.

Title: An Encoding Framework for Binarized Images using HyperDimensional Computing. (arXiv:2312.00454v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00454
• Code URL: null
• Copy Paste: [[2312.00454]] An Encoding Framework for Binarized Images using HyperDimensional Computing(http://arxiv.org/abs/2312.00454)
• Summary:

  Hyperdimensional Computing (HDC) is a brain-inspired and light-weight machine learning method. It has received significant attention in the literature as a candidate to be applied in the wearable internet of things, near-sensor artificial intelligence applications and on-device processing. HDC is computationally less complex than traditional deep learning algorithms and typically achieves moderate to good classification performance. A key aspect that determines the performance of HDC is the encoding of the input data to the hyperdimensional (HD) space. This article proposes a novel light-weight approach relying only on native HD arithmetic vector operations to encode binarized images that preserves similarity of patterns at nearby locations by using point of interest selection and local linear mapping. The method reaches an accuracy of 97.35% on the test set for the MNIST data set and 84.12% for the Fashion-MNIST data set. These results outperform other studies using baseline HDC with different encoding approaches and are on par with more complex hybrid HDC models. The proposed encoding approach also demonstrates a higher robustness to noise and blur compared to the baseline encoding.

Title: Unfolder: Fast localization and image rectification of a document with a crease from folding in half. (arXiv:2312.00467v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00467
• Code URL: null
• Copy Paste: [[2312.00467]] Unfolder: Fast localization and image rectification of a document with a crease from folding in half(http://arxiv.org/abs/2312.00467)
• Summary:

  Presentation of folded documents is not an uncommon case in modern society. Digitizing such documents by capturing them with a smartphone camera can be tricky since a crease can divide the document contents into separate planes. To unfold the document, one could hold the edges potentially obscuring it in a captured image. While there are many geometrical rectification methods, they were usually developed for arbitrary bends and folds. We consider such algorithms and propose a novel approach Unfolder developed specifically for images of documents with a crease from folding in half. Unfolder is robust to projective distortions of the document image and does not fragment the image in the vicinity of a crease after rectification. A new Folded Document Images dataset was created to investigate the rectification accuracy of folded (2, 3, 4, and 8 folds) documents. The dataset includes 1600 images captured when document placed on a table and when held in hand. The Unfolder algorithm allowed for a recognition error rate of 0.33, which is better than the advanced neural network methods DocTr (0.44) and DewarpNet (0.57). The average runtime for Unfolder was only 0.25 s/image on an iPhone XR.

Title: VisionaryVR: An Optical Simulation Tool for Evaluating and Optimizing Vision Correction Solutions in Virtual Reality. (arXiv:2312.00692v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00692
• Code URL: null
• Copy Paste: [[2312.00692]] VisionaryVR: An Optical Simulation Tool for Evaluating and Optimizing Vision Correction Solutions in Virtual Reality(http://arxiv.org/abs/2312.00692)
• Summary:

  Developing and evaluating vision science methods require robust and efficient tools for assessing their performance in various real-world scenarios. This study presents a novel virtual reality (VR) simulation tool that simulates real-world optical methods while giving high experimental control to the experiment. The tool incorporates an experiment controller, to smoothly and easily handle multiple conditions, a generic eye-tracking controller, that works with most common VR eye-trackers, a configurable defocus simulator, and a generic VR questionnaire loader to assess participants' behavior in virtual reality. This VR-based simulation tool bridges the gap between theoretical and applied research on new optical methods, corrections, and therapies. It enables vision scientists to increase their research tools with a robust, realistic, and fast research environment.

Title: Robust Concept Erasure via Kernelized Rate-Distortion Maximization. (arXiv:2312.00194v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.00194
• Code URL: https://github.com/brcsomnath/kram
• Copy Paste: [[2312.00194]] Robust Concept Erasure via Kernelized Rate-Distortion Maximization(http://arxiv.org/abs/2312.00194)
• Summary:

  Distributed representations provide a vector space that captures meaningful relationships between data instances. The distributed nature of these representations, however, entangles together multiple attributes or concepts of data instances (e.g., the topic or sentiment of a text, characteristics of the author (age, gender, etc), etc). Recent work has proposed the task of concept erasure, in which rather than making a concept predictable, the goal is to remove an attribute from distributed representations while retaining other information from the original representation space as much as possible. In this paper, we propose a new distance metric learning-based objective, the Kernelized Rate-Distortion Maximizer (KRaM), for performing concept erasure. KRaM fits a transformation of representations to match a specified distance measure (defined by a labeled concept to erase) using a modified rate-distortion function. Specifically, KRaM's objective function aims to make instances with similar concept labels dissimilar in the learned representation space while retaining other information. We find that optimizing KRaM effectively erases various types of concepts: categorical, continuous, and vector-valued variables from data representations across diverse domains. We also provide a theoretical analysis of several properties of KRaM's objective. To assess the quality of the learned representations, we propose an alignment score to evaluate their similarity with the original representation space. Additionally, we conduct experiments to showcase KRaM's efficacy in various settings, from erasing binary gender variables in word embeddings to vector-valued variables in GPT-3 representations.

Title: Text Attribute Control via Closed-Loop Disentanglement. (arXiv:2312.00277v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.00277
• Code URL: null
• Copy Paste: [[2312.00277]] Text Attribute Control via Closed-Loop Disentanglement(http://arxiv.org/abs/2312.00277)
• Summary:

  Changing an attribute of a text without changing the content usually requires to first disentangle the text into irrelevant attributes and content representations. After that, in the inference phase, the representation of one attribute is tuned to a different value, expecting that the corresponding attribute of the text can also be changed accordingly. The usual way of disentanglement is to add some constraints on the latent space of an encoder-decoder architecture, including adversarial-based constraints and mutual-information-based constraints. However, the previous semi-supervised processes of attribute change are usually not enough to guarantee the success of attribute change and content preservation. In this paper, we propose a novel approach to achieve a robust control of attributes while enhancing content preservation. In this approach, we use a semi-supervised contrastive learning method to encourage the disentanglement of attributes in latent spaces. Differently from previous works, we re-disentangle the reconstructed sentence and compare the re-disentangled latent space with the original latent space, which makes a closed-loop disentanglement process. This also helps content preservation. In addition, the contrastive learning method is also able to replace the role of minimizing mutual information and adversarial training in the disentanglement process, which alleviates the computation cost. We conducted experiments on three text datasets, including the Yelp Service review dataset, the Amazon Product review dataset, and the GoEmotions dataset. The experimental results show the effectiveness of our model.

Title: Summarization-based Data Augmentation for Document Classification. (arXiv:2312.00513v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.00513
• Code URL: https://github.com/etsurin/summaug
• Copy Paste: [[2312.00513]] Summarization-based Data Augmentation for Document Classification(http://arxiv.org/abs/2312.00513)
• Summary:

  Despite the prevalence of pretrained language models in natural language understanding tasks, understanding lengthy text such as document is still challenging due to the data sparseness problem. Inspired by that humans develop their ability of understanding lengthy text from reading shorter text, we propose a simple yet effective summarization-based data augmentation, SUMMaug, for document classification. We first obtain easy-to-learn examples for the target document classification task by summarizing the input of the original training examples, while optionally merging the original labels to conform to the summarized input. We then use the generated pseudo examples to perform curriculum learning. Experimental results on two datasets confirmed the advantage of our method compared to existing baseline methods in terms of robustness and accuracy. We release our code and data at https://github.com/etsurin/summaug.

Title: SurreyAI 2023 Submission for the Quality Estimation Shared Task. (arXiv:2312.00525v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.00525
• Code URL: null
• Copy Paste: [[2312.00525]] SurreyAI 2023 Submission for the Quality Estimation Shared Task(http://arxiv.org/abs/2312.00525)
• Summary:

  Quality Estimation (QE) systems are important in situations where it is necessary to assess the quality of translations, but there is no reference available. This paper describes the approach adopted by the SurreyAI team for addressing the Sentence-Level Direct Assessment shared task in WMT23. The proposed approach builds upon the TransQuest framework, exploring various autoencoder pre-trained language models within the MonoTransQuest architecture using single and ensemble settings. The autoencoder pre-trained language models employed in the proposed systems are XLMV, InfoXLM-large, and XLMR-large. The evaluation utilizes Spearman and Pearson correlation coefficients, assessing the relationship between machine-predicted quality scores and human judgments for 5 language pairs (English-Gujarati, English-Hindi, English-Marathi, English-Tamil and English-Telugu). The MonoTQ-InfoXLM-large approach emerges as a robust strategy, surpassing all other individual models proposed in this study by significantly improving over the baseline for the majority of the language pairs.

Title: Trained MT Metrics Learn to Cope with Machine-translated References. (arXiv:2312.00536v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.00536
• Code URL: https://github.com/amazon-science/prism-finetuned
• Copy Paste: [[2312.00536]] Trained MT Metrics Learn to Cope with Machine-translated References(http://arxiv.org/abs/2312.00536)
• Summary:

  Neural metrics trained on human evaluations of MT tend to correlate well with human judgments, but their behavior is not fully understood. In this paper, we perform a controlled experiment and compare a baseline metric that has not been trained on human evaluations (Prism) to a trained version of the same metric (Prism+FT). Surprisingly, we find that Prism+FT becomes more robust to machine-translated references, which are a notorious problem in MT evaluation. This suggests that the effects of metric training go beyond the intended effect of improving overall correlation with human judgments.

Title: Zipr: A High-Impact, Robust, Open-source, Multi-platform, Static Binary Rewriter. (arXiv:2312.00714v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.00714
• Code URL: null
• Copy Paste: [[2312.00714]] Zipr: A High-Impact, Robust, Open-source, Multi-platform, Static Binary Rewriter(http://arxiv.org/abs/2312.00714)
• Summary:

  Zipr is a tool for static binary rewriting, first published in 2016. Zipr was engineered to support arbitrary program modification with an emphasis on low overhead, robustness, and flexibility to perform security enhancements and instrumentation. Originally targeted to Linux x86-32 binaries, Zipr now supports 32- and 64-bit binaries for X86, ARM, and MIPS architectures, as well as preliminary support for Windows programs.

  These features have helped Zipr make a dramatic impact on research. It was first used in the DARPA Cyber Grand Challenge to take second place overall, with the best security score of any participant, Zipr has now been used in a variety of research areas by both the original authors as well as third parties. Zipr has also led to publications in artificial diversity, program instrumentation, program repair, fuzzing, autonomous vehicle security, research computing security, as well as directly contributing to two student dissertations. The open-source repository has accepted accepted patches from several external authors, demonstrating the impact of Zipr beyond the original authors.

Title: Deep Equilibrium Based Neural Operators for Steady-State PDEs. (arXiv:2312.00234v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.00234
• Code URL: null
• Copy Paste: [[2312.00234]] Deep Equilibrium Based Neural Operators for Steady-State PDEs(http://arxiv.org/abs/2312.00234)
• Summary:

  Data-driven machine learning approaches are being increasingly used to solve partial differential equations (PDEs). They have shown particularly striking successes when training an operator, which takes as input a PDE in some family, and outputs its solution. However, the architectural design space, especially given structural knowledge of the PDE family of interest, is still poorly understood. We seek to remedy this gap by studying the benefits of weight-tied neural network architectures for steady-state PDEs. To achieve this, we first demonstrate that the solution of most steady-state PDEs can be expressed as a fixed point of a non-linear operator. Motivated by this observation, we propose FNO-DEQ, a deep equilibrium variant of the FNO architecture that directly solves for the solution of a steady-state PDE as the infinite-depth fixed point of an implicit operator layer using a black-box root solver and differentiates analytically through this fixed point resulting in $\mathcal{O}(1)$ training memory. Our experiments indicate that FNO-DEQ-based architectures outperform FNO-based baselines with $4\times$ the number of parameters in predicting the solution to steady-state PDEs such as Darcy Flow and steady-state incompressible Navier-Stokes. Finally, we show FNO-DEQ is more robust when trained with datasets with more noisy observations than the FNO-based baselines, demonstrating the benefits of using appropriate inductive biases in architectural design for different neural network based PDE solvers. Further, we show a universal approximation result that demonstrates that FNO-DEQ can approximate the solution to any steady-state PDE that can be written as a fixed point equation.

Title: Age-Based Scheduling for Mobile Edge Computing: A Deep Reinforcement Learning Approach. (arXiv:2312.00279v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.00279
• Code URL: https://github.com/xingqiuhe/dpds
• Copy Paste: [[2312.00279]] Age-Based Scheduling for Mobile Edge Computing: A Deep Reinforcement Learning Approach(http://arxiv.org/abs/2312.00279)
• Summary:

  With the rapid development of Mobile Edge Computing (MEC), various real-time applications have been deployed to benefit people's daily lives. The performance of these applications relies heavily on the freshness of collected environmental information, which can be quantified by its Age of Information (AoI). In the traditional definition of AoI, it is assumed that the status information can be actively sampled and directly used. However, for many MEC-enabled applications, the desired status information is updated in an event-driven manner and necessitates data processing. To better serve these applications, we propose a new definition of AoI and, based on the redefined AoI, we formulate an online AoI minimization problem for MEC systems. Notably, the problem can be interpreted as a Markov Decision Process (MDP), thus enabling its solution through Reinforcement Learning (RL) algorithms. Nevertheless, the traditional RL algorithms are designed for MDPs with completely unknown system dynamics and hence usually suffer long convergence times. To accelerate the learning process, we introduce Post-Decision States (PDSs) to exploit the partial knowledge of the system's dynamics. We also combine PDSs with deep RL to further improve the algorithm's applicability, scalability, and robustness. Numerical results demonstrate that our algorithm outperforms the benchmarks under various scenarios.

Title: REDUCR: Robust Data Downsampling Using Class Priority Reweighting. (arXiv:2312.00486v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.00486
• Code URL: null
• Copy Paste: [[2312.00486]] REDUCR: Robust Data Downsampling Using Class Priority Reweighting(http://arxiv.org/abs/2312.00486)
• Summary:

  Modern machine learning models are becoming increasingly expensive to train for real-world image and text classification tasks, where massive web-scale data is collected in a streaming fashion. To reduce the training cost, online batch selection techniques have been developed to choose the most informative datapoints. However, these techniques can suffer from poor worst-class generalization performance due to class imbalance and distributional shifts. This work introduces REDUCR, a robust and efficient data downsampling method that uses class priority reweighting. REDUCR reduces the training data while preserving worst-class generalization performance. REDUCR assigns priority weights to datapoints in a class-aware manner using an online learning algorithm. We demonstrate the data efficiency and robust performance of REDUCR on vision and text classification tasks. On web-scraped datasets with imbalanced class distributions, REDUCR significantly improves worst-class test accuracy (and average accuracy), surpassing state-of-the-art methods by around 15%.

Title: On the Out-Of-Distribution Robustness of Self-Supervised Representation Learning for Phonocardiogram Signals. (arXiv:2312.00502v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.00502
• Code URL: https://github.com/aristotelisballas/listen2yourheart
• Copy Paste: [[2312.00502]] On the Out-Of-Distribution Robustness of Self-Supervised Representation Learning for Phonocardiogram Signals(http://arxiv.org/abs/2312.00502)
• Summary:

  Objective: Despite the recent increase in research activity, deep-learning models have not yet been widely accepted in medicine. The shortage of high-quality annotated data often hinders the development of robust and generalizable models, which do not suffer from degraded effectiveness when presented with newly-collected, out-of-distribution (OOD) datasets. Methods: Contrastive Self-Supervised Learning (SSL) offers a potential solution to the scarcity of labeled data as it takes advantage of unlabeled data to increase model effectiveness and robustness. In this research, we propose applying contrastive SSL for detecting abnormalities in phonocardiogram (PCG) samples by learning a generalized representation of the signal. Specifically, we perform an extensive comparative evaluation of a wide range of audio-based augmentations and evaluate trained classifiers on multiple datasets across different downstream tasks. Results: We experimentally demonstrate that, depending on its training distribution, the effectiveness of a fully-supervised model can degrade up to 32% when evaluated on unseen data, while SSL models only lose up to 10% or even improve in some cases. Conclusions: Contrastive SSL pretraining can assist in providing robust classifiers which can generalize to unseen, OOD data, without relying on time- and labor-intensive annotation processes by medical experts. Furthermore, the proposed extensive evaluation protocol sheds light on the most promising and appropriate augmentations for robust PCG signal processing. Significance: We provide researchers and practitioners with a roadmap towards producing robust models for PCG classification, in addition to an open-source codebase for developing novel approaches.

Title: Explainable Fraud Detection with Deep Symbolic Classification. (arXiv:2312.00586v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.00586
• Code URL: null
• Copy Paste: [[2312.00586]] Explainable Fraud Detection with Deep Symbolic Classification(http://arxiv.org/abs/2312.00586)
• Summary:

  There is a growing demand for explainable, transparent, and data-driven models within the domain of fraud detection. Decisions made by fraud detection models need to be explainable in the event of a customer dispute. Additionally, the decision-making process in the model must be transparent to win the trust of regulators and business stakeholders. At the same time, fraud detection solutions can benefit from data due to the noisy, dynamic nature of fraud and the availability of large historical data sets. Finally, fraud detection is notorious for its class imbalance: there are typically several orders of magnitude more legitimate transactions than fraudulent ones. In this paper, we present Deep Symbolic Classification (DSC), an extension of the Deep Symbolic Regression framework to classification problems. DSC casts classification as a search problem in the space of all analytic functions composed of a vocabulary of variables, constants, and operations and optimizes for an arbitrary evaluation metric directly. The search is guided by a deep neural network trained with reinforcement learning. Because the functions are mathematical expressions that are in closed-form and concise, the model is inherently explainable both at the level of a single classification decision and the model's decision process. Furthermore, the class imbalance problem is successfully addressed by optimizing for metrics that are robust to class imbalance such as the F1 score. This eliminates the need for oversampling and undersampling techniques that plague traditional approaches. Finally, the model allows to explicitly balance between the prediction accuracy and the explainability. An evaluation on the PaySim data set demonstrates competitive predictive performance with state-of-the-art models, while surpassing them in terms of explainability. This establishes DSC as a promising model for fraud detection systems.

biometric

Title: The theoretical limits of biometry. (arXiv:2312.00019v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.00019
• Code URL: null
• Copy Paste: [[2312.00019]] The theoretical limits of biometry(http://arxiv.org/abs/2312.00019)
• Summary:

  Biometry has proved its capability in terms of recognition accuracy. Now, it is widely used for automated border control with the biometric passport, to unlock a smartphone or a computer with a fingerprint or a face recognition algorithm. While identity verification is widely democratized, pure identification with no additional clues is still a work in progress. The identification difficulty depends on the population size, as the larger the group is, the larger the confusion risk. For collision prevention, biometric traits must be sufficiently distinguishable to scale to considerable groups, and algorithms should be able to capture their differences accurately.

  Most biometric works are purely experimental, and it is impossible to extrapolate the results to a smaller or a larger group. In this work, we propose a theoretical analysis of the distinguishability problem, which governs the error rates of biometric systems. We demonstrate simple relationships between the population size and the number of independent bits necessary to prevent collision in the presence of noise. This work provides the lowest lower bound for memory requirements. The results are very encouraging, as the biometry of the whole Earth population can fit in a regular disk, leaving some space for noise and redundancy.

steal

Title: Stealthy and Persistent Unalignment on Large Language Models via Backdoor Injections. (arXiv:2312.00027v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.00027
• Code URL: null
• Copy Paste: [[2312.00027]] Stealthy and Persistent Unalignment on Large Language Models via Backdoor Injections(http://arxiv.org/abs/2312.00027)
• Summary:

  Recent developments in Large Language Models (LLMs) have manifested significant advancements. To facilitate safeguards against malicious exploitation, a body of research has concentrated on aligning LLMs with human preferences and inhibiting their generation of inappropriate content. Unfortunately, such alignments are often vulnerable: fine-tuning with a minimal amount of harmful data can easily unalign the target LLM. While being effective, such fine-tuning-based unalignment approaches also have their own limitations: (1) non-stealthiness, after fine-tuning, safety audits or red-teaming can easily expose the potential weaknesses of the unaligned models, thereby precluding their release/use. (2) non-persistence, the unaligned LLMs can be easily repaired through re-alignment, i.e., fine-tuning again with aligned data points. In this work, we show that it is possible to conduct stealthy and persistent unalignment on large language models via backdoor injections. We also provide a novel understanding on the relationship between the backdoor persistence and the activation pattern and further provide guidelines for potential trigger design. Through extensive experiments, we demonstrate that our proposed stealthy and persistent unalignment can successfully pass the safety evaluation while maintaining strong persistence against re-alignment defense.

extraction

Title: DNS SLAM: Dense Neural Semantic-Informed SLAM. (arXiv:2312.00204v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00204
• Code URL: null
• Copy Paste: [[2312.00204]] DNS SLAM: Dense Neural Semantic-Informed SLAM(http://arxiv.org/abs/2312.00204)
• Summary:

  In recent years, coordinate-based neural implicit representations have shown promising results for the task of Simultaneous Localization and Mapping (SLAM). While achieving impressive performance on small synthetic scenes, these methods often suffer from oversmoothed reconstructions, especially for complex real-world scenes. In this work, we introduce DNS SLAM, a novel neural RGB-D semantic SLAM approach featuring a hybrid representation. Relying only on 2D semantic priors, we propose the first semantic neural SLAM method that trains class-wise scene representations while providing stable camera tracking at the same time. Our method integrates multi-view geometry constraints with image-based feature extraction to improve appearance details and to output color, density, and semantic class information, enabling many downstream applications. To further enable real-time tracking, we introduce a lightweight coarse scene representation which is trained in a self-supervised manner in latent space. Our experimental results achieve state-of-the-art performance on both synthetic data and real-world data tracking while maintaining a commendable operational speed on off-the-shelf hardware. Further, our method outputs class-wise decomposed reconstructions with better texture capturing appearance and geometric details.

Title: Unsupervised textile defect detection using convolutional neural networks. (arXiv:2312.00224v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00224
• Code URL: null
• Copy Paste: [[2312.00224]] Unsupervised textile defect detection using convolutional neural networks(http://arxiv.org/abs/2312.00224)
• Summary:

  In this study, we propose a novel motif-based approach for unsupervised textile anomaly detection that combines the benefits of traditional convolutional neural networks with those of an unsupervised learning paradigm. It consists of five main steps: preprocessing, automatic pattern period extraction, patch extraction, features selection and anomaly detection. This proposed approach uses a new dynamic and heuristic method for feature selection which avoids the drawbacks of initialization of the number of filters (neurons) and their weights, and those of the backpropagation mechanism such as the vanishing gradients, which are common practice in the state-of-the-art methods. The design and training of the network are performed in a dynamic and input domain-based manner and, thus, no ad-hoc configurations are required. Before building the model, only the number of layers and the stride are defined. We do not initialize the weights randomly nor do we define the filter size or number of filters as conventionally done in CNN-based approaches. This reduces effort and time spent on hyperparameter initialization and fine-tuning. Only one defect-free sample is required for training and no further labeled data is needed. The trained network is then used to detect anomalies on defective fabric samples. We demonstrate the effectiveness of our approach on the Patterned Fabrics benchmark dataset. Our algorithm yields reliable and competitive results (on recall, precision, accuracy and f1- measure) compared to state-of-the-art unsupervised approaches, in less time, with efficient training in a single epoch and a lower computational cost.

Title: Student Activity Recognition in Classroom Environments using Transfer Learning. (arXiv:2312.00348v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00348
• Code URL: null
• Copy Paste: [[2312.00348]] Student Activity Recognition in Classroom Environments using Transfer Learning(http://arxiv.org/abs/2312.00348)
• Summary:

  The recent advances in artificial intelligence and deep learning facilitate automation in various applications including home automation, smart surveillance systems, and healthcare among others. Human Activity Recognition is one of its emerging applications, which can be implemented in a classroom environment to enhance safety, efficiency, and overall educational quality. This paper proposes a system for detecting and recognizing the activities of students in a classroom environment. The dataset has been structured and recorded by the authors since a standard dataset for this task was not available at the time of this study. Transfer learning, a widely adopted method within the field of deep learning, has proven to be helpful in complex tasks like image and video processing. Pretrained models including VGG-16, ResNet-50, InceptionV3, and Xception are used for feature extraction and classification tasks. Xception achieved an accuracy of 93%, on the novel classroom dataset, outperforming the other three models in consideration. The system proposed in this study aims to introduce a safer and more productive learning environment for students and educators.

Title: Domain Adaptive Imitation Learning with Visual Observation. (arXiv:2312.00548v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.00548
• Code URL: null
• Copy Paste: [[2312.00548]] Domain Adaptive Imitation Learning with Visual Observation(http://arxiv.org/abs/2312.00548)
• Summary:

  In this paper, we consider domain-adaptive imitation learning with visual observation, where an agent in a target domain learns to perform a task by observing expert demonstrations in a source domain. Domain adaptive imitation learning arises in practical scenarios where a robot, receiving visual sensory data, needs to mimic movements by visually observing other robots from different angles or observing robots of different shapes. To overcome the domain shift in cross-domain imitation learning with visual observation, we propose a novel framework for extracting domain-independent behavioral features from input observations that can be used to train the learner, based on dual feature extraction and image reconstruction. Empirical results demonstrate that our approach outperforms previous algorithms for imitation learning from visual observation with domain shift.

Title: Less is More: Learning Reference Knowledge Using No-Reference Image Quality Assessment. (arXiv:2312.00591v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00591
• Code URL: null
• Copy Paste: [[2312.00591]] Less is More: Learning Reference Knowledge Using No-Reference Image Quality Assessment(http://arxiv.org/abs/2312.00591)
• Summary:

  Image Quality Assessment (IQA) with reference images have achieved great success by imitating the human vision system, in which the image quality is effectively assessed by comparing the query image with its pristine reference image. However, for the images in the wild, it is quite difficult to access accurate reference images. We argue that it is possible to learn reference knowledge under the No-Reference Image Quality Assessment (NR-IQA) setting, which is effective and efficient empirically. Concretely, by innovatively introducing a novel feature distillation method in IQA, we propose a new framework to learn comparative knowledge from non-aligned reference images. And then, to achieve fast convergence and avoid overfitting, we further propose an inductive bias regularization. Such a framework not only solves the congenital defects of NR-IQA but also improves the feature extraction framework, enabling it to express more abundant quality information. Surprisingly, our method utilizes less input while obtaining a more significant improvement compared to the teacher models. Extensive experiments on eight standard NR-IQA datasets demonstrate the superior performance to the state-of-the-art NR-IQA methods, i.e., achieving the PLCC values of 0.917 (vs. 0.884 in LIVEC) and 0.686 (vs. 0.661 in LIVEFB).

Title: Tracking Object Positions in Reinforcement Learning: A Metric for Keypoint Detection (extended version). (arXiv:2312.00592v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.00592
• Code URL: null
• Copy Paste: [[2312.00592]] Tracking Object Positions in Reinforcement Learning: A Metric for Keypoint Detection (extended version)(http://arxiv.org/abs/2312.00592)
• Summary:

  Reinforcement learning (RL) for robot control typically requires a detailed representation of the environment state, including information about task-relevant objects not directly measurable. Keypoint detectors, such as spatial autoencoders (SAEs), are a common approach to extracting a low-dimensional representation from high-dimensional image data. SAEs aim at spatial features such as object positions, which are often useful representations in robotic RL. However, whether an SAE is actually able to track objects in the scene and thus yields a spatial state representation well suited for RL tasks has rarely been examined due to a lack of established metrics. In this paper, we propose to assess the performance of an SAE instance by measuring how well keypoints track ground truth objects in images. We present a computationally lightweight metric and use it to evaluate common baseline SAE architectures on image data from a simulated robot task. We find that common SAEs differ substantially in their spatial extraction capability. Furthermore, we validate that SAEs that perform well in our metric achieve superior performance when used in downstream RL. Thus, our metric is an effective and lightweight indicator of RL performance before executing expensive RL training. Building on these insights, we identify three key modifications of SAE architectures to improve tracking performance. We make our code available at anonymous.4open.science/r/sae-rl.

Title: PointBeV: A Sparse Approach to BeV Predictions. (arXiv:2312.00703v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00703
• Code URL: https://github.com/valeoai/pointbev
• Copy Paste: [[2312.00703]] PointBeV: A Sparse Approach to BeV Predictions(http://arxiv.org/abs/2312.00703)
• Summary:

  Bird's-eye View (BeV) representations have emerged as the de-facto shared space in driving applications, offering a unified space for sensor data fusion and supporting various downstream tasks. However, conventional models use grids with fixed resolution and range and face computational inefficiencies due to the uniform allocation of resources across all cells. To address this, we propose PointBeV, a novel sparse BeV segmentation model operating on sparse BeV cells instead of dense grids. This approach offers precise control over memory usage, enabling the use of long temporal contexts and accommodating memory-constrained platforms. PointBeV employs an efficient two-pass strategy for training, enabling focused computation on regions of interest. At inference time, it can be used with various memory/performance trade-offs and flexibly adjusts to new specific use cases. PointBeV achieves state-of-the-art results on the nuScenes dataset for vehicle, pedestrian, and lane segmentation, showcasing superior performance in static and temporal settings despite being trained solely with sparse signals. We will release our code along with two new efficient modules used in the architecture: Sparse Feature Pulling, designed for the effective extraction of features from images to BeV, and Submanifold Attention, which enables efficient temporal modeling. Our code is available at https://github.com/valeoai/PointBeV.

Title: Japanese Tort-case Dataset for Rationale-supported Legal Judgment Prediction. (arXiv:2312.00480v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.00480
• Code URL: null
• Copy Paste: [[2312.00480]] Japanese Tort-case Dataset for Rationale-supported Legal Judgment Prediction(http://arxiv.org/abs/2312.00480)
• Summary:

  This paper presents the first dataset for Japanese Legal Judgment Prediction (LJP), the Japanese Tort-case Dataset (JTD), which features two tasks: tort prediction and its rationale extraction. The rationale extraction task identifies the court's accepting arguments from alleged arguments by plaintiffs and defendants, which is a novel task in the field. JTD is constructed based on annotated 3,477 Japanese Civil Code judgments by 41 legal experts, resulting in 7,978 instances with 59,697 of their alleged arguments from the involved parties. Our baseline experiments show the feasibility of the proposed two tasks, and our error analysis by legal experts identifies sources of errors and suggests future directions of the LJP research.

Title: Improving Unsupervised Relation Extraction by Augmenting Diverse Sentence Pairs. (arXiv:2312.00552v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.00552
• Code URL: null
• Copy Paste: [[2312.00552]] Improving Unsupervised Relation Extraction by Augmenting Diverse Sentence Pairs(http://arxiv.org/abs/2312.00552)
• Summary:

  Unsupervised relation extraction (URE) aims to extract relations between named entities from raw text without requiring manual annotations or pre-existing knowledge bases. In recent studies of URE, researchers put a notable emphasis on contrastive learning strategies for acquiring relation representations. However, these studies often overlook two important aspects: the inclusion of diverse positive pairs for contrastive learning and the exploration of appropriate loss functions. In this paper, we propose AugURE with both within-sentence pairs augmentation and augmentation through cross-sentence pairs extraction to increase the diversity of positive pairs and strengthen the discriminative power of contrastive learning. We also identify the limitation of noise-contrastive estimation (NCE) loss for relation representation learning and propose to apply margin loss for sentence pairs. Experiments on NYT-FB and TACRED datasets demonstrate that the proposed relation representation learning and a simple K-Means clustering achieves state-of-the-art performance.

Title: Explanatory Argument Extraction of Correct Answers in Resident Medical Exams. (arXiv:2312.00567v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.00567
• Code URL: null
• Copy Paste: [[2312.00567]] Explanatory Argument Extraction of Correct Answers in Resident Medical Exams(http://arxiv.org/abs/2312.00567)
• Summary:

  Developing the required technology to assist medical experts in their everyday activities is currently a hot topic in the Artificial Intelligence research field. Thus, a number of large language models (LLMs) and automated benchmarks have recently been proposed with the aim of facilitating information extraction in Evidence-Based Medicine (EBM) using natural language as a tool for mediating in human-AI interaction. The most representative benchmarks are limited to either multiple-choice or long-form answers and are available only in English. In order to address these shortcomings, in this paper we present a new dataset which, unlike previous work: (i) includes not only explanatory arguments for the correct answer, but also arguments to reason why the incorrect answers are not correct; (ii) the explanations are written originally by medical doctors to answer questions from the Spanish Residency Medical Exams. Furthermore, this new benchmark allows us to setup a novel extractive task which consists of identifying the explanation of the correct answer written by medical doctors. An additional benefit of our setting is that we can leverage the extractive QA paradigm to automatically evaluate performance of LLMs without resorting to costly manual evaluation by medical experts. Comprehensive experimentation with language models for Spanish shows that sometimes multilingual models fare better than monolingual ones, even outperforming models which have been adapted to the medical domain. Furthermore, results across the monolingual models are mixed, with supposedly smaller and inferior models performing competitively. In any case, the obtained results show that our novel dataset and approach can be an effective technique to help medical practitioners in identifying relevant evidence-based explanations for medical questions.

Title: Multimodal Learning for Crystalline Materials. (arXiv:2312.00111v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.00111
• Code URL: null
• Copy Paste: [[2312.00111]] Multimodal Learning for Crystalline Materials(http://arxiv.org/abs/2312.00111)
• Summary:

  Artificial intelligence (AI) has revolutionized the field of materials science by improving the prediction of properties and accelerating the discovery of novel materials. In recent years, publicly available material data repositories containing data for various material properties have grown rapidly. In this work, we introduce Multimodal Learning for Crystalline Materials (MLCM), a new method for training a foundation model for crystalline materials via multimodal alignment, where high-dimensional material properties (i.e. modalities) are connected in a shared latent space to produce highly useful material representations. We show the utility of MLCM on multiple axes: (i) MLCM achieves state-of-the-art performance for material property prediction on the challenging Materials Project database; (ii) MLCM enables a novel, highly accurate method for inverse design, allowing one to screen for stable material with desired properties; and (iii) MLCM allows the extraction of interpretable emergent features that may provide insight to material scientists. Further, we explore several novel methods for aligning an arbitrary number of modalities, improving upon prior art in multimodal learning that focuses on bimodal alignment. Our work brings innovations from the ongoing AI revolution into the domain of materials science and identifies materials as a testbed for the next generation of AI.

Title: A framework for mining lifestyle profiles through multi-dimensional and high-order mobility feature clustering. (arXiv:2312.00411v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.00411
• Code URL: null
• Copy Paste: [[2312.00411]] A framework for mining lifestyle profiles through multi-dimensional and high-order mobility feature clustering(http://arxiv.org/abs/2312.00411)
• Summary:

  Human mobility demonstrates a high degree of regularity, which facilitates the discovery of lifestyle profiles. Existing research has yet to fully utilize the regularities embedded in high-order features extracted from human mobility records in such profiling. This study proposes a progressive feature extraction strategy that mines high-order mobility features from users' moving trajectory records from the spatial, temporal, and semantic dimensions. Specific features are extracted such as travel motifs, rhythms decomposed by discrete Fourier transform (DFT) of mobility time series, and vectorized place semantics by word2vec, respectively to the three dimensions, and they are further clustered to reveal the users' lifestyle characteristics. An experiment using a trajectory dataset of over 500k users in Shenzhen, China yields seven user clusters with different lifestyle profiles that can be well interpreted by common sense. The results suggest the possibility of fine-grained user profiling through cross-order trajectory feature engineering and clustering.

Title: Pathway to a fully data-driven geotechnics: lessons from materials informatics. (arXiv:2312.00581v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.00581
• Code URL: null
• Copy Paste: [[2312.00581]] Pathway to a fully data-driven geotechnics: lessons from materials informatics(http://arxiv.org/abs/2312.00581)
• Summary:

  This paper elucidates the challenges and opportunities inherent in integrating data-driven methodologies into geotechnics, drawing inspiration from the success of materials informatics. Highlighting the intricacies of soil complexity, heterogeneity, and the lack of comprehensive data, the discussion underscores the pressing need for community-driven database initiatives and open science movements. By leveraging the transformative power of deep learning, particularly in feature extraction from high-dimensional data and the potential of transfer learning, we envision a paradigm shift towards a more collaborative and innovative geotechnics field. The paper concludes with a forward-looking stance, emphasizing the revolutionary potential brought about by advanced computational tools like large language models in reshaping geotechnics informatics.

membership infer

federate

Title: A Quality-of-Service Compliance System using Federated Learning and Optimistic Rollups. (arXiv:2312.00026v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.00026
• Code URL: null
• Copy Paste: [[2312.00026]] A Quality-of-Service Compliance System using Federated Learning and Optimistic Rollups(http://arxiv.org/abs/2312.00026)
• Summary:

  Edge computing brings a new paradigm in which the sharing of computing, storage, and bandwidth resources as close as possible to the mobile devices or sensors generating a large amount of data. A parallel trend is the rise of phones and tablets as primary computing devices for many people. The powerful sensors present on these devices combined with the fact that they are mobile, mean they have access to data of an unprecedentedly diverse and private nature. Models learned on such data hold the promise of greatly improving usability by powering more intelligent applications, but the sensitive nature of the data means there are risks and responsibilities to storing it in a centralized location. To address the data privacy required for some data in these devices we propose the use of Federated Learning (FL) so that specific data about services performed by clients do not leave the source machines. Instead of sharing data, users collaboratively train a model by only sending weight updates to a server. However, the naive use of FL in those scenarios exposes it to a risk of corruption, whether intentional or not, during the training phase. To improve the security of the FL structure, we propose a decentralized Blockchain-based FL in an edge computing scenario. We also apply blockchain to create a reward mechanism in FL to enable incentive strategy for trainers.

Title: FedEmb: A Vertical and Hybrid Federated Learning Algorithm using Network And Feature Embedding Aggregation. (arXiv:2312.00102v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.00102
• Code URL: null
• Copy Paste: [[2312.00102]] FedEmb: A Vertical and Hybrid Federated Learning Algorithm using Network And Feature Embedding Aggregation(http://arxiv.org/abs/2312.00102)
• Summary:

  Federated learning (FL) is an emerging paradigm for decentralized training of machine learning models on distributed clients, without revealing the data to the central server. The learning scheme may be horizontal, vertical or hybrid (both vertical and horizontal). Most existing research work with deep neural network (DNN) modelling is focused on horizontal data distributions, while vertical and hybrid schemes are much less studied. In this paper, we propose a generalized algorithm FedEmb, for modelling vertical and hybrid DNN-based learning. The idea of our algorithm is characterised by higher inference accuracy, stronger privacy-preserving properties, and lower client-server communication bandwidth demands as compared with existing work. The experimental results show that FedEmb is an effective method to tackle both split feature & subject space decentralized problems, shows 0.3% to 4.2% inference accuracy improvement with limited privacy revealing for datasets stored in local clients, and reduces 88.9 % time complexity over vertical baseline method.

fair

Title: Removing Biases from Molecular Representations via Information Maximization. (arXiv:2312.00718v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.00718
• Code URL: https://github.com/uhlerlab/infocore
• Copy Paste: [[2312.00718]] Removing Biases from Molecular Representations via Information Maximization(http://arxiv.org/abs/2312.00718)
• Summary:

  High-throughput drug screening -- using cell imaging or gene expression measurements as readouts of drug effect -- is a critical tool in biotechnology to assess and understand the relationship between the chemical structure and biological activity of a drug. Since large-scale screens have to be divided into multiple experiments, a key difficulty is dealing with batch effects, which can introduce systematic errors and non-biological associations in the data. We propose InfoCORE, an Information maximization approach for COnfounder REmoval, to effectively deal with batch effects and obtain refined molecular representations. InfoCORE establishes a variational lower bound on the conditional mutual information of the latent representations given a batch identifier. It adaptively reweighs samples to equalize their implied batch distribution. Extensive experiments on drug screening data reveal InfoCORE's superior performance in a multitude of tasks including molecular property prediction and molecule-phenotype retrieval. Additionally, we show results for how InfoCORE offers a versatile framework and resolves general distribution shifts and issues of data fairness by minimizing correlation with spurious features or removing sensitive attributes. The code is available at https://github.com/uhlerlab/InfoCORE.

interpretability

Title: CLIP-QDA: An Explainable Concept Bottleneck Model. (arXiv:2312.00110v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00110
• Code URL: null
• Copy Paste: [[2312.00110]] CLIP-QDA: An Explainable Concept Bottleneck Model(http://arxiv.org/abs/2312.00110)
• Summary:

  In this paper, we introduce an explainable algorithm designed from a multi-modal foundation model, that performs fast and explainable image classification. Drawing inspiration from CLIP-based Concept Bottleneck Models (CBMs), our method creates a latent space where each neuron is linked to a specific word. Observing that this latent space can be modeled with simple distributions, we use a Mixture of Gaussians (MoG) formalism to enhance the interpretability of this latent space. Then, we introduce CLIP-QDA, a classifier that only uses statistical values to infer labels from the concepts. In addition, this formalism allows for both local and global explanations. These explanations come from the inner design of our architecture, our work is part of a new family of greybox models, combining performances of opaque foundation models and the interpretability of transparent models. Our empirical findings show that in instances where the MoG assumption holds, CLIP-QDA achieves similar accuracy with state-of-the-art methods CBMs. Our explanations compete with existing XAI methods while being faster to compute.

Title: Benchmarking and Enhancing Disentanglement in Concept-Residual Models. (arXiv:2312.00192v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.00192
• Code URL: null
• Copy Paste: [[2312.00192]] Benchmarking and Enhancing Disentanglement in Concept-Residual Models(http://arxiv.org/abs/2312.00192)
• Summary:

  Concept bottleneck models (CBMs) are interpretable models that first predict a set of semantically meaningful features, i.e., concepts, from observations that are subsequently used to condition a downstream task. However, the model's performance strongly depends on the engineered features and can severely suffer from incomplete sets of concepts. Prior works have proposed a side channel -- a residual -- that allows for unconstrained information flow to the downstream task, thus improving model performance but simultaneously introducing information leakage, which is undesirable for interpretability. This work proposes three novel approaches to mitigate information leakage by disentangling concepts and residuals, investigating the critical balance between model performance and interpretability. Through extensive empirical analysis on the CUB, OAI, and CIFAR 100 datasets, we assess the performance of each disentanglement method and provide insights into when they work best. Further, we show how each method impacts the ability to intervene over the concepts and their subsequent impact on task performance.

Title: Learning Anatomically Consistent Embedding for Chest Radiography. (arXiv:2312.00335v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00335
• Code URL: https://github.com/jlianglab/peac
• Copy Paste: [[2312.00335]] Learning Anatomically Consistent Embedding for Chest Radiography(http://arxiv.org/abs/2312.00335)
• Summary:

  Self-supervised learning (SSL) approaches have recently shown substantial success in learning visual representations from unannotated images. Compared with photographic images, medical images acquired with the same imaging protocol exhibit high consistency in anatomy. To exploit this anatomical consistency, this paper introduces a novel SSL approach, called PEAC (patch embedding of anatomical consistency), for medical image analysis. Specifically, in this paper, we propose to learn global and local consistencies via stable grid-based matching, transfer pre-trained PEAC models to diverse downstream tasks, and extensively demonstrate that (1) PEAC achieves significantly better performance than the existing state-of-the-art fully/self-supervised methods, and (2) PEAC captures the anatomical structure consistency across views of the same patient and across patients of different genders, weights, and healthy statuses, which enhances the interpretability of our method for medical image analysis.

Title: Contextualized word senses: from attention to compositionality. (arXiv:2312.00680v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.00680
• Code URL: null
• Copy Paste: [[2312.00680]] Contextualized word senses: from attention to compositionality(http://arxiv.org/abs/2312.00680)
• Summary:

  The neural architectures of language models are becoming increasingly complex, especially that of Transformers, based on the attention mechanism. Although their application to numerous natural language processing tasks has proven to be very fruitful, they continue to be models with little or no interpretability and explainability. One of the tasks for which they are best suited is the encoding of the contextual sense of words using contextualized embeddings. In this paper we propose a transparent, interpretable, and linguistically motivated strategy for encoding the contextual sense of words by modeling semantic compositionality. Particular attention is given to dependency relations and semantic notions such as selection preferences and paradigmatic classes. A partial implementation of the proposed model is carried out and compared with Transformer-based architectures for a given semantic task, namely the similarity calculation of word senses in context. The results obtained show that it is possible to be competitive with linguistically motivated models instead of using the black boxes underlying complex neural architectures.

Title: Towards Clinical Prediction with Transparency: An Explainable AI Approach to Survival Modelling in Residential Aged Care. (arXiv:2312.00271v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.00271
• Code URL: https://github.com/teosusnjak/survival-analysis-stage1
• Copy Paste: [[2312.00271]] Towards Clinical Prediction with Transparency: An Explainable AI Approach to Survival Modelling in Residential Aged Care(http://arxiv.org/abs/2312.00271)
• Summary:

  Background: Accurate survival time estimates aid end-of-life medical decision-making. Objectives: Develop an interpretable survival model for elderly residential aged care residents using advanced machine learning. Setting: A major Australasian residential aged care provider. Participants: Residents aged 65+ admitted for long-term care from July 2017 to August 2023. Sample size: 11,944 residents across 40 facilities. Predictors: Factors include age, gender, health status, co-morbidities, cognitive function, mood, nutrition, mobility, smoking, sleep, skin integrity, and continence. Outcome: Probability of survival post-admission, specifically calibrated for 6-month survival estimates. Statistical Analysis: Tested CoxPH, EN, RR, Lasso, GB, XGB, and RF models in 20 experiments with a 90/10 train/test split. Evaluated accuracy using C-index, Harrell's C-index, dynamic AUROC, IBS, and calibrated ROC. Chose XGB for its performance and calibrated it for 1, 3, 6, and 12-month predictions using Platt scaling. Employed SHAP values to analyze predictor impacts. Results: GB, XGB, and RF models showed the highest C-Index values (0.714, 0.712, 0.712). The optimal XGB model demonstrated a 6-month survival prediction AUROC of 0.746 (95% CI 0.744-0.749). Key mortality predictors include age, male gender, mobility, health status, pressure ulcer risk, and appetite. Conclusions: The study successfully applies machine learning to create a survival model for aged care, aligning with clinical insights on mortality risk factors and enhancing model interpretability and clinical utility through explainable AI.

Title: Interpretable Meta-Learning of Physical Systems. (arXiv:2312.00477v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.00477
• Code URL: null
• Copy Paste: [[2312.00477]] Interpretable Meta-Learning of Physical Systems(http://arxiv.org/abs/2312.00477)
• Summary:

  Machine learning methods can be a valuable aid in the scientific process, but they need to face challenging settings where data come from inhomogeneous experimental conditions. Recent meta-learning methods have made significant progress in multi-task learning, but they rely on black-box neural networks, resulting in high computational costs and limited interpretability. Leveraging the structure of the learning problem, we argue that multi-environment generalization can be achieved using a simpler learning model, with an affine structure with respect to the learning task. Crucially, we prove that this architecture can identify the physical parameters of the system, enabling interpreable learning. We demonstrate the competitive generalization performance and the low computational cost of our method by comparing it to state-of-the-art algorithms on physical systems, ranging from toy models to complex, non-analytical systems. The interpretability of our method is illustrated with original applications to physical-parameter-induced adaptation and to adaptive control.

explainability

Title: Towards Explaining Satellite Based Poverty Predictions with Convolutional Neural Networks. (arXiv:2312.00416v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00416
• Code URL: null
• Copy Paste: [[2312.00416]] Towards Explaining Satellite Based Poverty Predictions with Convolutional Neural Networks(http://arxiv.org/abs/2312.00416)
• Summary:

  Deep convolutional neural networks (CNNs) have been shown to predict poverty and development indicators from satellite images with surprising accuracy. This paper presents a first attempt at analyzing the CNNs responses in detail and explaining the basis for the predictions. The CNN model, while trained on relatively low resolution day- and night-time satellite images, is able to outperform human subjects who look at high-resolution images in ranking the Wealth Index categories. Multiple explainability experiments performed on the model indicate the importance of the sizes of the objects, pixel colors in the image, and provide a visualization of the importance of different structures in input images. A visualization is also provided of type images that maximize the network prediction of Wealth Index, which provides clues on what the CNN prediction is based on.

Title: Relevance-guided Neural Machine Translation. (arXiv:2312.00214v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.00214
• Code URL: null
• Copy Paste: [[2312.00214]] Relevance-guided Neural Machine Translation(http://arxiv.org/abs/2312.00214)
• Summary:

  With the advent of the Transformer architecture, Neural Machine Translation (NMT) results have shown great improvement lately. However, results in low-resource conditions still lag behind in both bilingual and multilingual setups, due to the limited amount of available monolingual and/or parallel data; hence, the need for methods addressing data scarcity in an efficient, and explainable way, is eminent. We propose an explainability-based training approach for NMT, applied in Unsupervised and Supervised model training, for translation of three languages of varying resources, French, Gujarati, Kazakh, to and from English. Our results show our method can be promising, particularly when training in low-resource conditions, outperforming simple training baselines; though the improvement is marginal, it sets the ground for further exploration of the approach and the parameters, and its extension to other languages.

watermark

Title: Mark My Words: Analyzing and Evaluating Language Model Watermarks. (arXiv:2312.00273v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.00273
• Code URL: null
• Copy Paste: [[2312.00273]] Mark My Words: Analyzing and Evaluating Language Model Watermarks(http://arxiv.org/abs/2312.00273)
• Summary:

  The capabilities of large language models have grown significantly in recent years and so too have concerns about their misuse. In this context, the ability to distinguish machine-generated text from human-authored content becomes important. Prior works have proposed numerous schemes to watermark text, which would benefit from a systematic evaluation framework. This work focuses on text watermarking techniques - as opposed to image watermarks - and proposes a comprehensive benchmark for them under different tasks as well as practical attacks. We focus on three main metrics: quality, size (e.g. the number of tokens needed to detect a watermark), and tamper-resistance. Current watermarking techniques are good enough to be deployed: Kirchenbauer et al. can watermark Llama2-7B-chat with no perceivable loss in quality in under 100 tokens, and with good tamper-resistance to simple attacks, regardless of temperature. We argue that watermark indistinguishability is too strong a requirement: schemes that slightly modify logit distributions outperform their indistinguishable counterparts with no noticeable loss in generation quality. We publicly release our benchmark.

diffusion

Title: Unsupervised Keypoints from Pretrained Diffusion Models. (arXiv:2312.00065v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00065
• Code URL: null
• Copy Paste: [[2312.00065]] Unsupervised Keypoints from Pretrained Diffusion Models(http://arxiv.org/abs/2312.00065)
• Summary:

  Unsupervised learning of keypoints and landmarks has seen significant progress with the help of modern neural network architectures, but performance is yet to match the supervised counterpart, making their practicability questionable. We leverage the emergent knowledge within text-to-image diffusion models, towards more robust unsupervised keypoints. Our core idea is to find text embeddings that would cause the generative model to consistently attend to compact regions in images (i.e. keypoints). To do so, we simply optimize the text embedding such that the cross-attention maps within the denoising network are localized as Gaussians with small standard deviations. We validate our performance on multiple datasets: the CelebA, CUB-200-2011, Tai-Chi-HD, DeepFashion, and Human3.6m datasets. We achieve significantly improved accuracy, sometimes even outperforming supervised ones, particularly for data that is non-aligned and less curated. Our code is publicly available and can be found through our project page: https://ubc-vision.github.io/StableKeypoints/

Title: HiFi Tuner: High-Fidelity Subject-Driven Fine-Tuning for Diffusion Models. (arXiv:2312.00079v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00079
• Code URL: null
• Copy Paste: [[2312.00079]] HiFi Tuner: High-Fidelity Subject-Driven Fine-Tuning for Diffusion Models(http://arxiv.org/abs/2312.00079)
• Summary:

  This paper explores advancements in high-fidelity personalized image generation through the utilization of pre-trained text-to-image diffusion models. While previous approaches have made significant strides in generating versatile scenes based on text descriptions and a few input images, challenges persist in maintaining the subject fidelity within the generated images. In this work, we introduce an innovative algorithm named HiFi Tuner to enhance the appearance preservation of objects during personalized image generation. Our proposed method employs a parameter-efficient fine-tuning framework, comprising a denoising process and a pivotal inversion process. Key enhancements include the utilization of mask guidance, a novel parameter regularization technique, and the incorporation of step-wise subject representations to elevate the sample fidelity. Additionally, we propose a reference-guided generation approach that leverages the pivotal inversion of a reference image to mitigate unwanted subject variations and artifacts. We further extend our method to a novel image editing task: substituting the subject in an image through textual manipulations. Experimental evaluations conducted on the DreamBooth dataset using the Stable Diffusion model showcase promising results. Fine-tuning solely on textual embeddings improves CLIP-T score by 3.6 points and improves DINO score by 9.6 points over Textual Inversion. When fine-tuning all parameters, HiFi Tuner improves CLIP-T score by 1.2 points and improves DINO score by 1.2 points over DreamBooth, establishing a new state of the art.

Title: X-Dreamer: Creating High-quality 3D Content by Bridging the Domain Gap Between Text-to-2D and Text-to-3D Generation. (arXiv:2312.00085v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00085
• Code URL: null
• Copy Paste: [[2312.00085]] X-Dreamer: Creating High-quality 3D Content by Bridging the Domain Gap Between Text-to-2D and Text-to-3D Generation(http://arxiv.org/abs/2312.00085)
• Summary:

  In recent times, automatic text-to-3D content creation has made significant progress, driven by the development of pretrained 2D diffusion models. Existing text-to-3D methods typically optimize the 3D representation to ensure that the rendered image aligns well with the given text, as evaluated by the pretrained 2D diffusion model. Nevertheless, a substantial domain gap exists between 2D images and 3D assets, primarily attributed to variations in camera-related attributes and the exclusive presence of foreground objects. Consequently, employing 2D diffusion models directly for optimizing 3D representations may lead to suboptimal outcomes. To address this issue, we present X-Dreamer, a novel approach for high-quality text-to-3D content creation that effectively bridges the gap between text-to-2D and text-to-3D synthesis. The key components of X-Dreamer are two innovative designs: Camera-Guided Low-Rank Adaptation (CG-LoRA) and Attention-Mask Alignment (AMA) Loss. CG-LoRA dynamically incorporates camera information into the pretrained diffusion models by employing camera-dependent generation for trainable parameters. This integration enhances the alignment between the generated 3D assets and the camera's perspective. AMA loss guides the attention map of the pretrained diffusion model using the binary mask of the 3D object, prioritizing the creation of the foreground object. This module ensures that the model focuses on generating accurate and detailed foreground objects. Extensive evaluations demonstrate the effectiveness of our proposed method compared to existing text-to-3D approaches. Our project webpage: https://xmuxiaoma666.github.io/Projects/X-Dreamer .

Title: GraphDreamer: Compositional 3D Scene Synthesis from Scene Graphs. (arXiv:2312.00093v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00093
• Code URL: null
• Copy Paste: [[2312.00093]] GraphDreamer: Compositional 3D Scene Synthesis from Scene Graphs(http://arxiv.org/abs/2312.00093)
• Summary:

  As pretrained text-to-image diffusion models become increasingly powerful, recent efforts have been made to distill knowledge from these text-to-image pretrained models for optimizing a text-guided 3D model. Most of the existing methods generate a holistic 3D model from a plain text input. This can be problematic when the text describes a complex scene with multiple objects, because the vectorized text embeddings are inherently unable to capture a complex description with multiple entities and relationships. Holistic 3D modeling of the entire scene further prevents accurate grounding of text entities and concepts. To address this limitation, we propose GraphDreamer, a novel framework to generate compositional 3D scenes from scene graphs, where objects are represented as nodes and their interactions as edges. By exploiting node and edge information in scene graphs, our method makes better use of the pretrained text-to-image diffusion model and is able to fully disentangle different objects without image-level supervision. To facilitate modeling of object-wise relationships, we use signed distance fields as representation and impose a constraint to avoid inter-penetration of objects. To avoid manual scene graph creation, we design a text prompt for ChatGPT to generate scene graphs based on text inputs. We conduct both qualitative and quantitative experiments to validate the effectiveness of GraphDreamer in generating high-fidelity compositional 3D scenes with disentangled object entities.

Title: Fast ODE-based Sampling for Diffusion Models in Around 5 Steps. (arXiv:2312.00094v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00094
• Code URL: null
• Copy Paste: [[2312.00094]] Fast ODE-based Sampling for Diffusion Models in Around 5 Steps(http://arxiv.org/abs/2312.00094)
• Summary:

  Sampling from diffusion models can be treated as solving the corresponding ordinary differential equations (ODEs), with the aim of obtaining an accurate solution with as few number of function evaluations (NFE) as possible. Recently, various fast samplers utilizing higher-order ODE solvers have emerged and achieved better performance than the initial first-order one. However, these numerical methods inherently result in certain approximation errors, which significantly degrades sample quality with extremely small NFE (e.g., around 5). In contrast, based on the geometric observation that each sampling trajectory almost lies in a two-dimensional subspace embedded in the ambient space, we propose Approximate MEan-Direction Solver (AMED-Solver) that eliminates truncation errors by directly learning the mean direction for fast diffusion sampling. Besides, our method can be easily used as a plugin to further improve existing ODE-based samplers. Extensive experiments on image synthesis with the resolution ranging from 32 to 256 demonstrate the effectiveness of our method. With only 5 NFE, we achieve 7.14 FID on CIFAR-10, 13.75 FID on ImageNet 64$\times$64, and 12.79 FID on LSUN Bedroom. Our code is available at https://github.com/zhyzhouu/amed-solver.

Title: S2ST: Image-to-Image Translation in the Seed Space of Latent Diffusion. (arXiv:2312.00116v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00116
• Code URL: null
• Copy Paste: [[2312.00116]] S2ST: Image-to-Image Translation in the Seed Space of Latent Diffusion(http://arxiv.org/abs/2312.00116)
• Summary:

  Image-to-image translation (I2IT) refers to the process of transforming images from a source domain to a target domain while maintaining a fundamental connection in terms of image content. In the past few years, remarkable advancements in I2IT were achieved by Generative Adversarial Networks (GANs), which nevertheless struggle with translations requiring high precision. Recently, Diffusion Models have established themselves as the engine of choice for image generation. In this paper we introduce S2ST, a novel framework designed to accomplish global I2IT in complex photorealistic images, such as day-to-night or clear-to-rain translations of automotive scenes. S2ST operates within the seed space of a Latent Diffusion Model, thereby leveraging the powerful image priors learned by the latter. We show that S2ST surpasses state-of-the-art GAN-based I2IT methods, as well as diffusion-based approaches, for complex automotive scenes, improving fidelity while respecting the target domain's appearance across a variety of domains. Notably, S2ST obviates the necessity for training domain-specific translation networks.

Title: DREAM: Diffusion Rectification and Estimation-Adaptive Models. (arXiv:2312.00210v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00210
• Code URL: null
• Copy Paste: [[2312.00210]] DREAM: Diffusion Rectification and Estimation-Adaptive Models(http://arxiv.org/abs/2312.00210)
• Summary:

  We present DREAM, a novel training framework representing Diffusion Rectification and Estimation-Adaptive Models, requiring minimal code changes (just three lines) yet significantly enhancing the alignment of training with sampling in diffusion models. DREAM features two components: diffusion rectification, which adjusts training to reflect the sampling process, and estimation adaptation, which balances perception against distortion. When applied to image super-resolution (SR), DREAM adeptly navigates the tradeoff between minimizing distortion and preserving high image quality. Experiments demonstrate DREAM's superiority over standard diffusion-based SR methods, showing a $2$ to $3\times $ faster training convergence and a $10$ to $20\times$ reduction in necessary sampling steps to achieve comparable or superior results. We hope DREAM will inspire a rethinking of diffusion model training paradigms.

Title: Text-Guided 3D Face Synthesis -- From Generation to Editing. (arXiv:2312.00375v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00375
• Code URL: null
• Copy Paste: [[2312.00375]] Text-Guided 3D Face Synthesis -- From Generation to Editing(http://arxiv.org/abs/2312.00375)
• Summary:

  Text-guided 3D face synthesis has achieved remarkable results by leveraging text-to-image (T2I) diffusion models. However, most existing works focus solely on the direct generation, ignoring the editing, restricting them from synthesizing customized 3D faces through iterative adjustments. In this paper, we propose a unified text-guided framework from face generation to editing. In the generation stage, we propose a geometry-texture decoupled generation to mitigate the loss of geometric details caused by coupling. Besides, decoupling enables us to utilize the generated geometry as a condition for texture generation, yielding highly geometry-texture aligned results. We further employ a fine-tuned texture diffusion model to enhance texture quality in both RGB and YUV space. In the editing stage, we first employ a pre-trained diffusion model to update facial geometry or texture based on the texts. To enable sequential editing, we introduce a UV domain consistency preservation regularization, preventing unintentional changes to irrelevant facial attributes. Besides, we propose a self-guided consistency weight strategy to improve editing efficacy while preserving consistency. Through comprehensive experiments, we showcase our method's superiority in face synthesis. Project page: https://faceg2e.github.io/.

Title: LucidDreaming: Controllable Object-Centric 3D Generation. (arXiv:2312.00588v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00588
• Code URL: null
• Copy Paste: [[2312.00588]] LucidDreaming: Controllable Object-Centric 3D Generation(http://arxiv.org/abs/2312.00588)
• Summary:

  With the recent development of generative models, Text-to-3D generations have also seen significant growth. Nonetheless, achieving precise control over 3D generation continues to be an arduous task, as using text to control often leads to missing objects and imprecise locations. Contemporary strategies for enhancing controllability in 3D generation often entail the introduction of additional parameters, such as customized diffusion models. This often induces hardness in adapting to different diffusion models or creating distinct objects.

  In this paper, we present LucidDreaming as an effective pipeline capable of fine-grained control over 3D generation. It requires only minimal input of 3D bounding boxes, which can be deduced from a simple text prompt using a Large Language Model. Specifically, we propose clipped ray sampling to separately render and optimize objects with user specifications. We also introduce object-centric density blob bias, fostering the separation of generated objects. With individual rendering and optimizing of objects, our method excels not only in controlled content generation from scratch but also within the pre-trained NeRF scenes. In such scenarios, existing generative approaches often disrupt the integrity of the original scene, and current editing methods struggle to synthesize new content in empty spaces. We show that our method exhibits remarkable adaptability across a spectrum of mainstream Score Distillation Sampling-based 3D generation frameworks, and achieves superior alignment of 3D content when compared to baseline approaches. We also provide a dataset of prompts with 3D bounding boxes, benchmarking 3D spatial controllability.

Title: TrackDiffusion: Multi-object Tracking Data Generation via Diffusion Models. (arXiv:2312.00651v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00651
• Code URL: null
• Copy Paste: [[2312.00651]] TrackDiffusion: Multi-object Tracking Data Generation via Diffusion Models(http://arxiv.org/abs/2312.00651)
• Summary:

  Diffusion models have gained prominence in generating data for perception tasks such as image classification and object detection. However, the potential in generating high-quality tracking sequences, a crucial aspect in the field of video perception, has not been fully investigated. To address this gap, we propose TrackDiffusion, a novel architecture designed to generate continuous video sequences from the tracklets. TrackDiffusion represents a significant departure from the traditional layout-to-image (L2I) generation and copy-paste synthesis focusing on static image elements like bounding boxes by empowering image diffusion models to encompass dynamic and continuous tracking trajectories, thereby capturing complex motion nuances and ensuring instance consistency among video frames. For the first time, we demonstrate that the generated video sequences can be utilized for training multi-object tracking (MOT) systems, leading to significant improvement in tracker performance. Experimental results show that our model significantly enhances instance consistency in generated video sequences, leading to improved perceptual metrics. Our approach achieves an improvement of 8.7 in TrackAP and 11.8 in TrackAP$_{50}$ on the YTVIS dataset, underscoring its potential to redefine the standards of video data generation for MOT tasks and beyond.

Title: Elijah: Eliminating Backdoors Injected in Diffusion Models via Distribution Shift. (arXiv:2312.00050v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.00050
• Code URL: null
• Copy Paste: [[2312.00050]] Elijah: Eliminating Backdoors Injected in Diffusion Models via Distribution Shift(http://arxiv.org/abs/2312.00050)
• Summary:

  Diffusion models (DM) have become state-of-the-art generative models because of their capability to generate high-quality images from noises without adversarial training. However, they are vulnerable to backdoor attacks as reported by recent studies. When a data input (e.g., some Gaussian noise) is stamped with a trigger (e.g., a white patch), the backdoored model always generates the target image (e.g., an improper photo). However, effective defense strategies to mitigate backdoors from DMs are underexplored. To bridge this gap, we propose the first backdoor detection and removal framework for DMs. We evaluate our framework Elijah on hundreds of DMs of 3 types including DDPM, NCSN and LDM, with 13 samplers against 3 existing backdoor attacks. Extensive experiments show that our approach can have close to 100% detection accuracy and reduce the backdoor effects to close to zero without significantly sacrificing the model utility.

Title: Resource-constrained knowledge diffusion processes inspired by human peer learning. (arXiv:2312.00660v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.00660
• Code URL: null
• Copy Paste: [[2312.00660]] Resource-constrained knowledge diffusion processes inspired by human peer learning(http://arxiv.org/abs/2312.00660)
• Summary:

  We consider a setting where a population of artificial learners is given, and the objective is to optimize aggregate measures of performance, under constraints on training resources. The problem is motivated by the study of peer learning in human educational systems. In this context, we study natural knowledge diffusion processes in networks of interacting artificial learners. By `natural', we mean processes that reflect human peer learning where the students' internal state and learning process is mostly opaque, and the main degree of freedom lies in the formation of peer learning groups by a coordinator who can potentially evaluate the learners before assigning them to peer groups. Among else, we empirically show that such processes indeed make effective use of the training resources, and enable the design of modular neural models that have the capacity to generalize without being prone to overfitting noisy labels.

noise learning

data-free

transformer

Title: BAM-DETR: Boundary-Aligned Moment Detection Transformer for Temporal Sentence Grounding in Videos. (arXiv:2312.00083v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00083
• Code URL: https://github.com/Pilhyeon/BAM-DETR
• Copy Paste: [[2312.00083]] BAM-DETR: Boundary-Aligned Moment Detection Transformer for Temporal Sentence Grounding in Videos(http://arxiv.org/abs/2312.00083)
• Summary:

  Temporal sentence grounding aims to localize moments relevant to a language description. Recently, DETR-like approaches have shown notable progress by decoding the center and length of a target moment from learnable queries. However, they suffer from the issue of center misalignment raised by the inherent ambiguity of moment centers, leading to inaccurate predictions. To remedy this problem, we introduce a novel boundary-oriented moment formulation. In our paradigm, the model no longer needs to find the precise center but instead suffices to predict any anchor point within the interval, from which the onset and offset are directly estimated. Based on this idea, we design a Boundary-Aligned Moment Detection Transformer (BAM-DETR), equipped with a dual-pathway decoding process. Specifically, it refines the anchor and boundaries within parallel pathways using global and boundary-focused attention, respectively. This separate design allows the model to focus on desirable regions, enabling precise refinement of moment predictions. Further, we propose a quality-based ranking method, ensuring that proposals with high localization qualities are prioritized over incomplete ones. Extensive experiments verify the advantages of our methods, where our model records new state-of-the-art results on three benchmarks. Code is at https://github.com/Pilhyeon/BAM-DETR.

Title: Brainformer: Modeling MRI Brain Functions to Machine Vision. (arXiv:2312.00236v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00236
• Code URL: null
• Copy Paste: [[2312.00236]] Brainformer: Modeling MRI Brain Functions to Machine Vision(http://arxiv.org/abs/2312.00236)
• Summary:

  "Perception is reality". Human perception plays a vital role in forming beliefs and understanding reality. Exploring how the human brain works in the visual system facilitates bridging the gap between human visual perception and computer vision models. However, neuroscientists study the brain via Neuroimaging, i.e., Functional Magnetic Resonance Imaging (fMRI), to discover the brain's functions. These approaches face interpretation challenges where fMRI data can be complex and require expertise. Therefore, neuroscientists make inferences about cognitive processes based on patterns of brain activities, which can lead to potential misinterpretation or limited functional understanding. In this work, we first present a simple yet effective Brainformer approach, a novel Transformer-based framework, to analyze the patterns of fMRI in the human perception system from the machine learning perspective. Secondly, we introduce a novel mechanism incorporating fMRI, which represents the human brain activities, as the supervision for the machine vision model. This work also introduces a novel perspective on transferring knowledge from human perception to neural networks. Through our experiments, we demonstrated that by leveraging fMRI information, the machine vision model can achieve potential results compared to the current State-of-the-art methods in various image recognition tasks.

Title: Learning to Estimate Critical Gait Parameters from Single-View RGB Videos with Transformer-Based Attention Network. (arXiv:2312.00398v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00398
• Code URL: null
• Copy Paste: [[2312.00398]] Learning to Estimate Critical Gait Parameters from Single-View RGB Videos with Transformer-Based Attention Network(http://arxiv.org/abs/2312.00398)
• Summary:

  Musculoskeletal diseases and cognitive impairments in patients lead to difficulties in movement as well as negative effects on their psychological health. Clinical gait analysis, a vital tool for early diagnosis and treatment, traditionally relies on expensive optical motion capture systems. Recent advances in computer vision and deep learning have opened the door to more accessible and cost-effective alternatives. This paper introduces a novel spatio-temporal Transformer network to estimate critical gait parameters from RGB videos captured by a single-view camera. Empirical evaluations on a public dataset of cerebral palsy patients indicate that the proposed framework surpasses current state-of-the-art approaches and show significant improvements in predicting general gait parameters (including Walking Speed, Gait Deviation Index - GDI, and Knee Flexion Angle at Maximum Extension), while utilizing fewer parameters and alleviating the need for manual feature extraction.

Title: SCHEME: Scalable Channer Mixer for Vision Transformers. (arXiv:2312.00412v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00412
• Code URL: null
• Copy Paste: [[2312.00412]] SCHEME: Scalable Channer Mixer for Vision Transformers(http://arxiv.org/abs/2312.00412)
• Summary:

  Vision Transformers have received significant attention due to their impressive performance in many vision tasks. While the token mixer or attention block has been studied in great detail, the channel mixer or feature mixing block (FFN or MLP) has not been explored in depth albeit it accounts for a bulk of the parameters and computation in a model. In this work, we study whether sparse feature mixing can replace the dense connections and confirm this with a block diagonal MLP structure that improves the accuracy by supporting larger expansion ratios. To improve the feature clusters formed by this structure and thereby further improve the accuracy, a lightweight, parameter-free, channel covariance attention (CCA) mechanism is introduced as a parallel branch during training. This design of CCA enables gradual feature mixing across channel groups during training whose contribution decays to zero as the training progresses to convergence. This allows the CCA block to be discarded during inference, thus enabling enhanced performance with no additional computational cost. The resulting $\textit{Scalable CHannEl MixEr}$ (SCHEME) can be plugged into any ViT architecture to obtain a gamut of models with different trade-offs between complexity and performance by controlling the block diagonal structure size in the MLP. This is shown by the introduction of a new family of SCHEMEformer models. Experiments on image classification, object detection, and semantic segmentation, with different ViT backbones, consistently demonstrate substantial accuracy gains over existing designs, especially under lower FLOPs regimes. For example, the SCHEMEformer establishes a new SOTA of 79.7% accuracy for ViTs using pure attention mixers on ImageNet-1K at 1.77G FLOPs.

Title: Event Recognition in Laparoscopic Gynecology Videos with Hybrid Transformers. (arXiv:2312.00593v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00593
• Code URL: null
• Copy Paste: [[2312.00593]] Event Recognition in Laparoscopic Gynecology Videos with Hybrid Transformers(http://arxiv.org/abs/2312.00593)
• Summary:

  Analyzing laparoscopic surgery videos presents a complex and multifaceted challenge, with applications including surgical training, intra-operative surgical complication prediction, and post-operative surgical assessment. Identifying crucial events within these videos is a significant prerequisite in a majority of these applications. In this paper, we introduce a comprehensive dataset tailored for relevant event recognition in laparoscopic gynecology videos. Our dataset includes annotations for critical events associated with major intra-operative challenges and post-operative complications. To validate the precision of our annotations, we assess event recognition performance using several CNN-RNN architectures. Furthermore, we introduce and evaluate a hybrid transformer architecture coupled with a customized training-inference framework to recognize four specific events in laparoscopic surgery videos. Leveraging the Transformer networks, our proposed architecture harnesses inter-frame dependencies to counteract the adverse effects of relevant content occlusion, motion blur, and surgical scene variation, thus significantly enhancing event recognition accuracy. Moreover, we present a frame sampling strategy designed to manage variations in surgical scenes and the surgeons' skill level, resulting in event recognition with high temporal resolution. We empirically demonstrate the superiority of our proposed methodology in event recognition compared to conventional CNN-RNN architectures through a series of extensive experiments.

Title: BCN: Batch Channel Normalization for Image Classification. (arXiv:2312.00596v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00596
• Code URL: https://github.com/AfifaKhaled/Batch-Channel-Normalization
• Copy Paste: [[2312.00596]] BCN: Batch Channel Normalization for Image Classification(http://arxiv.org/abs/2312.00596)
• Summary:

  Normalization techniques have been widely used in the field of deep learning due to their capability of enabling higher learning rates and are less careful in initialization. However, the effectiveness of popular normalization technologies is typically limited to specific areas. Unlike the standard Batch Normalization (BN) and Layer Normalization (LN), where BN computes the mean and variance along the (N,H,W) dimensions and LN computes the mean and variance along the (C,H,W) dimensions (N, C, H and W are the batch, channel, spatial height and width dimension, respectively), this paper presents a novel normalization technique called Batch Channel Normalization (BCN). To exploit both the channel and batch dependence and adaptively and combine the advantages of BN and LN based on specific datasets or tasks, BCN separately normalizes inputs along the (N, H, W) and (C, H, W) axes, then combines the normalized outputs based on adaptive parameters. As a basic block, BCN can be easily integrated into existing models for various applications in the field of computer vision. Empirical results show that the proposed technique can be seamlessly applied to various versions of CNN or Vision Transformer architecture. The code is publicly available at https://github.com/AfifaKhaled/BatchChannel-Normalization

Title: Towards Efficient 3D Object Detection in Bird's-Eye-View Space for Autonomous Driving: A Convolutional-Only Approach. (arXiv:2312.00633v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00633
• Code URL: null
• Copy Paste: [[2312.00633]] Towards Efficient 3D Object Detection in Bird's-Eye-View Space for Autonomous Driving: A Convolutional-Only Approach(http://arxiv.org/abs/2312.00633)
• Summary:

  3D object detection in Bird's-Eye-View (BEV) space has recently emerged as a prevalent approach in the field of autonomous driving. Despite the demonstrated improvements in accuracy and velocity estimation compared to perspective view methods, the deployment of BEV-based techniques in real-world autonomous vehicles remains challenging. This is primarily due to their reliance on vision-transformer (ViT) based architectures, which introduce quadratic complexity with respect to the input resolution. To address this issue, we propose an efficient BEV-based 3D detection framework called BEVENet, which leverages a convolutional-only architectural design to circumvent the limitations of ViT models while maintaining the effectiveness of BEV-based methods. Our experiments show that BEVENet is 3$\times$ faster than contemporary state-of-the-art (SOTA) approaches on the NuScenes challenge, achieving a mean average precision (mAP) of 0.456 and a nuScenes detection score (NDS) of 0.555 on the NuScenes validation dataset, with an inference speed of 47.6 frames per second. To the best of our knowledge, this study stands as the first to achieve such significant efficiency improvements for BEV-based methods, highlighting their enhanced feasibility for real-world autonomous driving applications.

Title: SPOT: Self-Training with Patch-Order Permutation for Object-Centric Learning with Autoregressive Transformers. (arXiv:2312.00648v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00648
• Code URL: https://github.com/gkakogeorgiou/spot
• Copy Paste: [[2312.00648]] SPOT: Self-Training with Patch-Order Permutation for Object-Centric Learning with Autoregressive Transformers(http://arxiv.org/abs/2312.00648)
• Summary:

  Unsupervised object-centric learning aims to decompose scenes into interpretable object entities, termed slots. Slot-based auto-encoders stand out as a prominent method for this task. Within them, crucial aspects include guiding the encoder to generate object-specific slots and ensuring the decoder utilizes them during reconstruction. This work introduces two novel techniques, (i) an attention-based self-training approach, which distills superior slot-based attention masks from the decoder to the encoder, enhancing object segmentation, and (ii) an innovative patch-order permutation strategy for autoregressive transformers that strengthens the role of slot vectors in reconstruction. The effectiveness of these strategies is showcased experimentally. The combined approach significantly surpasses prior slot-based autoencoder methods in unsupervised object segmentation, especially with complex real-world images. We provide the implementation code at https://github.com/gkakogeorgiou/spot .

Title: Rethinking Detection Based Table Structure Recognition for Visually Rich Documents. (arXiv:2312.00699v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00699
• Code URL: null
• Copy Paste: [[2312.00699]] Rethinking Detection Based Table Structure Recognition for Visually Rich Documents(http://arxiv.org/abs/2312.00699)
• Summary:

  Table Structure Recognition (TSR) aims at transforming unstructured table images into structured formats, such as HTML sequences. One type of popular solution is using detection models to detect components of a table, such as columns and rows, then applying a rule-based post-processing method to convert detection results into HTML sequences. However, existing detection-based studies often have the following limitations. First, these studies usually pay more attention to improving the detection performance, which does not necessarily lead to better performance regarding cell-level metrics, such as TEDS. Second, some solutions over-simplify the problem and can miss some critical information. Lastly, even though some studies defined the problem to detect more components to provide as much information as other types of solutions, these studies ignore the fact this problem definition is a multi-label detection because row, projected row header and column header can share identical bounding boxes. Besides, there is often a performance gap between two-stage and transformer-based detection models regarding the structure-only TEDS, even though they have similar performance regarding the COCO metrics. Therefore, we revisit the limitations of existing detection-based solutions, compare two-stage and transformer-based detection models, and identify the key design aspects for the success of a two-stage detection model for the TSR task, including the multi-class problem definition, the aspect ratio for anchor box generation, and the feature generation of the backbone network. We applied simple methods to improve these aspects of the Cascade R-CNN model, achieved state-of-the-art performance, and improved the baseline Cascade R-CNN model by 19.32%, 11.56% and 14.77% regarding the structure-only TEDS on SciTSR, FinTabNet, and PubTables1M datasets.

Title: GIFT: Generative Interpretable Fine-Tuning Transformers. (arXiv:2312.00700v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00700
• Code URL: https://github.com/savadikarc/gift
• Copy Paste: [[2312.00700]] GIFT: Generative Interpretable Fine-Tuning Transformers(http://arxiv.org/abs/2312.00700)
• Summary:

  We present GIFT (Generative Interpretable Fine-tuning Transformers) for fine-tuning pretrained (often large) Transformer models at downstream tasks in a parameter-efficient way with built-in interpretability. Our GIFT is a deep parameter-residual learning method, which addresses two problems in fine-tuning a pretrained Transformer model: Where to apply the parameter-efficient fine-tuning (PEFT) to be extremely lightweight yet sufficiently expressive, and How to learn the PEFT to better exploit the knowledge of the pretrained model in a direct way? For the former, we select the final projection (linear) layer in the multi-head self-attention of a Transformer model, and verify its effectiveness. For the latter, in contrast to the prior art that directly introduce new model parameters (often in low-rank approximation form) to be learned in fine-tuning with downstream data, we propose a method for learning to generate the fine-tuning parameters. Our GIFT is a hyper-Transformer which take as input the pretrained parameters of the projection layer to generate its fine-tuning parameters using a proposed Parameter-to-Cluster Attention (PaCa). The PaCa results in a simple clustering-based forward explainer that plays the role of semantic segmentation in testing. In experiments, our proposed GIFT is tested on the VTAB benchmark and the fine-grained visual classification (FGVC) benchmark. It obtains significantly better performance than the prior art. Our code is available at https://github.com/savadikarc/gift

Title: Nonparametric Variational Regularisation of Pretrained Transformers. (arXiv:2312.00662v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.00662
• Code URL: null
• Copy Paste: [[2312.00662]] Nonparametric Variational Regularisation of Pretrained Transformers(http://arxiv.org/abs/2312.00662)
• Summary:

  The current paradigm of large-scale pre-training and fine-tuning Transformer large language models has lead to significant improvements across the board in natural language processing. However, such large models are susceptible to overfitting to their training data, and as a result the models perform poorly when the domain changes. Also, due to the model's scale, the cost of fine-tuning the model to the new domain is large. Nonparametric Variational Information Bottleneck (NVIB) has been proposed as a regulariser for training cross-attention in Transformers, potentially addressing the overfitting problem. We extend the NVIB framework to replace all types of attention functions in Transformers, and show that existing pretrained Transformers can be reinterpreted as Nonparametric Variational (NV) models using a proposed identity initialisation. We then show that changing the initialisation introduces a novel, information-theoretic post-training regularisation in the attention mechanism, which improves out-of-domain generalisation without any training. This success supports the hypothesis that pretrained Transformers are implicitly NV Bayesian models.

Title: Mitigating Over-smoothing in Transformers via Regularized Nonlocal Functionals. (arXiv:2312.00751v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.00751
• Code URL: null
• Copy Paste: [[2312.00751]] Mitigating Over-smoothing in Transformers via Regularized Nonlocal Functionals(http://arxiv.org/abs/2312.00751)
• Summary:

  Transformers have achieved remarkable success in a wide range of natural language processing and computer vision applications. However, the representation capacity of a deep transformer model is degraded due to the over-smoothing issue in which the token representations become identical when the model's depth grows. In this work, we show that self-attention layers in transformers minimize a functional which promotes smoothness, thereby causing token uniformity. We then propose a novel regularizer that penalizes the norm of the difference between the smooth output tokens from self-attention and the input tokens to preserve the fidelity of the tokens. Minimizing the resulting regularized energy functional, we derive the Neural Transformer with a Regularized Nonlocal Functional (NeuTRENO), a novel class of transformer models that can mitigate the over-smoothing issue. We empirically demonstrate the advantages of NeuTRENO over the baseline transformers and state-of-the-art methods in reducing the over-smoothing of token representations on various practical tasks, including object classification, image segmentation, and language modeling.

Title: PyraTrans: Learning Attention-Enriched Multi-Scale Pyramid Network from Pre-Trained Transformers for Effective Malicious URL Detection. (arXiv:2312.00508v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.00508
• Code URL: null
• Copy Paste: [[2312.00508]] PyraTrans: Learning Attention-Enriched Multi-Scale Pyramid Network from Pre-Trained Transformers for Effective Malicious URL Detection(http://arxiv.org/abs/2312.00508)
• Summary:

  Detecting malicious URLs is a crucial aspect of web search and mining, significantly impacting internet security. Though advancements in machine learning have improved the effectiveness of detection methods, these methods still face significant challenges in their capacity to generalize and their resilience against evolving threats. In this paper, we propose PyraTrans, an approach that combines the strengths of pretrained Transformers and pyramid feature learning for improving malicious URL detection. We implement PyraTrans by leveraging a pretrained CharBERT as the base and augmenting it with 3 connected feature modules: 1) The Encoder Feature Extraction module, which extracts representations from each encoder layer of CharBERT to obtain multi-order features; 2) The Multi-Scale Feature Learning Module, which captures multi-scale local contextual insights and aggregate information across different layer-levels; and 3) The Pyramid Spatial Attention Module, which learns hierarchical and spatial feature attentions, highlighting critical classification signals while reducing noise. The proposed approach addresses the limitations of the Transformer in local feature learning and spatial awareness, and enabling us to extract multi-order, multi-scale URL feature representations with enhanced attentional focus. PyraTrans is evaluated using 4 benchmark datasets, where it demonstrated significant advancements over prior baseline methods. Particularly, on the imbalanced dataset, our method, with just 10% of the data for training, the TPR is 3.3-6.5 times and the F1-score is 2.9-4.5 times that of the baseline. Our approach also demonstrates robustness against adversarial attacks. Codes and data are available at https://github.com/Alixyvtte/PyraTrans.

Title: Hypergraph Node Representation Learning with One-Stage Message Passing. (arXiv:2312.00336v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.00336
• Code URL: null
• Copy Paste: [[2312.00336]] Hypergraph Node Representation Learning with One-Stage Message Passing(http://arxiv.org/abs/2312.00336)
• Summary:

  Hypergraphs as an expressive and general structure have attracted considerable attention from various research domains. Most existing hypergraph node representation learning techniques are based on graph neural networks, and thus adopt the two-stage message passing paradigm (i.e. node -> hyperedge -> node). This paradigm only focuses on local information propagation and does not effectively take into account global information, resulting in less optimal representations. Our theoretical analysis of representative two-stage message passing methods shows that, mathematically, they model different ways of local message passing through hyperedges, and can be unified into one-stage message passing (i.e. node -> node). However, they still only model local information. Motivated by this theoretical analysis, we propose a novel one-stage message passing paradigm to model both global and local information propagation for hypergraphs. We integrate this paradigm into HGraphormer, a Transformer-based framework for hypergraph node representation learning. HGraphormer injects the hypergraph structure information (local information) into Transformers (global information) by combining the attention matrix and hypergraph Laplacian. Extensive experiments demonstrate that HGraphormer outperforms recent hypergraph learning methods on five representative benchmark datasets on the semi-supervised hypernode classification task, setting new state-of-the-art performance, with accuracy improvements between 2.52% and 6.70%. Our code and datasets are available.

generative

Title: MoMask: Generative Masked Modeling of 3D Human Motions. (arXiv:2312.00063v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00063
• Code URL: null
• Copy Paste: [[2312.00063]] MoMask: Generative Masked Modeling of 3D Human Motions(http://arxiv.org/abs/2312.00063)
• Summary:

  We introduce MoMask, a novel masked modeling framework for text-driven 3D human motion generation. In MoMask, a hierarchical quantization scheme is employed to represent human motion as multi-layer discrete motion tokens with high-fidelity details. Starting at the base layer, with a sequence of motion tokens obtained by vector quantization, the residual tokens of increasing orders are derived and stored at the subsequent layers of the hierarchy. This is consequently followed by two distinct bidirectional transformers. For the base-layer motion tokens, a Masked Transformer is designated to predict randomly masked motion tokens conditioned on text input at training stage. During generation (i.e. inference) stage, starting from an empty sequence, our Masked Transformer iteratively fills up the missing tokens; Subsequently, a Residual Transformer learns to progressively predict the next-layer tokens based on the results from current layer. Extensive experiments demonstrate that MoMask outperforms the state-of-art methods on the text-to-motion generation task, with an FID of 0.045 (vs e.g. 0.141 of T2M-GPT) on the HumanML3D dataset, and 0.228 (vs 0.514) on KIT-ML, respectively. MoMask can also be seamlessly applied in related tasks without further model fine-tuning, such as text-guided temporal inpainting.

Title: Mixture of Gaussian-distributed Prototypes with Generative Modelling for Interpretable Image Classification. (arXiv:2312.00092v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00092
• Code URL: null
• Copy Paste: [[2312.00092]] Mixture of Gaussian-distributed Prototypes with Generative Modelling for Interpretable Image Classification(http://arxiv.org/abs/2312.00092)
• Summary:

  Prototypical-part interpretable methods, e.g., ProtoPNet, enhance interpretability by connecting classification predictions to class-specific training prototypes, thereby offering an intuitive insight into their decision-making. Current methods rely on a discriminative classifier trained with point-based learning techniques that provide specific values for prototypes. Such prototypes have relatively low representation power due to their sparsity and potential redundancy, with each prototype containing no variability measure. In this paper, we present a new generative learning of prototype distributions, named Mixture of Gaussian-distributed Prototypes (MGProto), which are represented by Gaussian mixture models (GMM). Such an approach enables the learning of more powerful prototype representations since each learned prototype will own a measure of variability, which naturally reduces the sparsity given the spread of the distribution around each prototype, and we also integrate a prototype diversity objective function into the GMM optimisation to reduce redundancy. Incidentally, the generative nature of MGProto offers a new and effective way for detecting out-of-distribution samples. To improve the compactness of MGProto, we further propose to prune Gaussian-distributed prototypes with a low prior. Experiments on CUB-200-2011, Stanford Cars, Stanford Dogs, and Oxford-IIIT Pets datasets show that MGProto achieves state-of-the-art classification and OoD detection performances with encouraging interpretability results.

Title: SparseGS: Real-Time 360{\deg} Sparse View Synthesis using Gaussian Splatting. (arXiv:2312.00206v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00206
• Code URL: null
• Copy Paste: [[2312.00206]] SparseGS: Real-Time 360{\deg} Sparse View Synthesis using Gaussian Splatting(http://arxiv.org/abs/2312.00206)
• Summary:

  The problem of novel view synthesis has grown significantly in popularity recently with the introduction of Neural Radiance Fields (NeRFs) and other implicit scene representation methods. A recent advance, 3D Gaussian Splatting (3DGS), leverages an explicit representation to achieve real-time rendering with high-quality results. However, 3DGS still requires an abundance of training views to generate a coherent scene representation. In few shot settings, similar to NeRF, 3DGS tends to overfit to training views, causing background collapse and excessive floaters, especially as the number of training views are reduced. We propose a method to enable training coherent 3DGS-based radiance fields of 360 scenes from sparse training views. We find that using naive depth priors is not sufficient and integrate depth priors with generative and explicit constraints to reduce background collapse, remove floaters, and enhance consistency from unseen viewpoints. Experiments show that our method outperforms base 3DGS by up to 30.5% and NeRF-based methods by up to 15.6% in LPIPS on the MipNeRF-360 dataset with substantially less training and inference cost.

Title: DeepDR: Deep Structure-Aware RGB-D Inpainting for Diminished Reality. (arXiv:2312.00532v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00532
• Code URL: null
• Copy Paste: [[2312.00532]] DeepDR: Deep Structure-Aware RGB-D Inpainting for Diminished Reality(http://arxiv.org/abs/2312.00532)
• Summary:

  Diminished reality (DR) refers to the removal of real objects from the environment by virtually replacing them with their background. Modern DR frameworks use inpainting to hallucinate unobserved regions. While recent deep learning-based inpainting is promising, the DR use case is complicated by the need to generate coherent structure and 3D geometry (i.e., depth), in particular for advanced applications, such as 3D scene editing. In this paper, we propose DeepDR, a first RGB-D inpainting framework fulfilling all requirements of DR: Plausible image and geometry inpainting with coherent structure, running at real-time frame rates, with minimal temporal artifacts. Our structure-aware generative network allows us to explicitly condition color and depth outputs on the scene semantics, overcoming the difficulty of reconstructing sharp and consistent boundaries in regions with complex backgrounds. Experimental results show that the proposed framework can outperform related work qualitatively and quantitatively.

Title: Generative models for visualising abstract social processes: Guiding streetview image synthesis of StyleGAN2 with indices of deprivation. (arXiv:2312.00570v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00570
• Code URL: null
• Copy Paste: [[2312.00570]] Generative models for visualising abstract social processes: Guiding streetview image synthesis of StyleGAN2 with indices of deprivation(http://arxiv.org/abs/2312.00570)
• Summary:

  This paper presents a novel application of Generative Adverserial Networks (GANs) to study visual aspects of social processes. I train a a StyleGAN2-model on a custom dataset of 14,564 images of London, sourced from Google Streetview taken in London. After training, I invert the images in the training set, finding points in the model's latent space that correspond to them, and compare results from three inversion techniques. I connect each data point with metadata from the Indices of Multiple Deprivation, describing income, health and environmental quality in the area where the photographs were taken. It is then possible to map which parts of the model's latent space encode visual features that are distinctive for health, income and environmental quality, and condition the synthesis of new images based on these factors. The synthetic images created reflect visual features of social processes that were previously unknown and difficult to study, describing recurring visual differences between deprived and privileged areas in London. GANs are known for their capability to produce a continuous range of images that exhibit visual differences. The paper tests how to exploit this ability through visual comparisons in still images as well as through an interactive website where users can guide image synthesis with sliders. Though conditioned synthesis has its limitations and the results are difficult to validate, the paper points to the potential for generative models to be repurposed to be parts of social scientific methods.

Title: MD-Splatting: Learning Metric Deformation from 4D Gaussians in Highly Deformable Scenes. (arXiv:2312.00583v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00583
• Code URL: null
• Copy Paste: [[2312.00583]] MD-Splatting: Learning Metric Deformation from 4D Gaussians in Highly Deformable Scenes(http://arxiv.org/abs/2312.00583)
• Summary:

  Accurate 3D tracking in highly deformable scenes with occlusions and shadows can facilitate new applications in robotics, augmented reality, and generative AI. However, tracking under these conditions is extremely challenging due to the ambiguity that arises with large deformations, shadows, and occlusions. We introduce MD-Splatting, an approach for simultaneous 3D tracking and novel view synthesis, using video captures of a dynamic scene from various camera poses. MD-Splatting builds on recent advances in Gaussian splatting, a method that learns the properties of a large number of Gaussians for state-of-the-art and fast novel view synthesis. MD-Splatting learns a deformation function to project a set of Gaussians with non-metric, thus canonical, properties into metric space. The deformation function uses a neural-voxel encoding and a multilayer perceptron (MLP) to infer Gaussian position, rotation, and a shadow scalar. We enforce physics-inspired regularization terms based on local rigidity, conservation of momentum, and isometry, which leads to trajectories with smaller trajectory errors. MD-Splatting achieves high-quality 3D tracking on highly deformable scenes with shadows and occlusions. Compared to state-of-the-art, we improve 3D tracking by an average of 23.9 %, while simultaneously achieving high-quality novel view synthesis. With sufficient texture such as in scene 6, MD-Splatting achieves a median tracking error of 3.39 mm on a cloth of 1 x 1 meters in size. Project website: https://md-splatting.github.io/.

Title: EvE: Exploiting Generative Priors for Radiance Field Enrichment. (arXiv:2312.00639v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00639
• Code URL: null
• Copy Paste: [[2312.00639]] EvE: Exploiting Generative Priors for Radiance Field Enrichment(http://arxiv.org/abs/2312.00639)
• Summary:

  Modeling large-scale scenes from unconstrained image collections in-the-wild has proven to be a major challenge in computer vision. Existing methods tackling in-the-wild neural rendering operate in a closed-world setting, where knowledge is limited to a scene's captured images within a training set. We propose EvE, which is, to the best of our knowledge, the first method leveraging generative priors to improve in-the-wild scene modeling. We employ pre-trained generative networks to enrich K-Planes representations with extrinsic knowledge. To this end, we define an alternating training procedure to conduct optimization guidance of K-Planes trained on the training set. We carry out extensive experiments and verify the merit of our method on synthetic data as well as real tourism photo collections. EvE enhances rendered scenes with richer details and outperforms the state of the art on the task of novel view synthesis in-the-wild. Our project page can be found at https://eve-nvs.github.io .

Title: Adversarial Score Distillation: When score distillation meets GAN. (arXiv:2312.00739v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00739
• Code URL: https://github.com/2y7c3/asd
• Copy Paste: [[2312.00739]] Adversarial Score Distillation: When score distillation meets GAN(http://arxiv.org/abs/2312.00739)
• Summary:

  Existing score distillation methods are sensitive to classifier-free guidance (CFG) scale: manifested as over-smoothness or instability at small CFG scales, while over-saturation at large ones. To explain and analyze these issues, we revisit the derivation of Score Distillation Sampling (SDS) and decipher existing score distillation with the Wasserstein Generative Adversarial Network (WGAN) paradigm. With the WGAN paradigm, we find that existing score distillation either employs a fixed sub-optimal discriminator or conducts incomplete discriminator optimization, resulting in the scale-sensitive issue. We propose the Adversarial Score Distillation (ASD), which maintains an optimizable discriminator and updates it using the complete optimization objective. Experiments show that the proposed ASD performs favorably in 2D distillation and text-to-3D tasks against existing methods. Furthermore, to explore the generalization ability of our WGAN paradigm, we extend ASD to the image editing task, which achieves competitive results. The project page and code are at https://github.com/2y7c3/ASD.

Title: GFN-SR: Symbolic Regression with Generative Flow Networks. (arXiv:2312.00396v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.00396
• Code URL: null
• Copy Paste: [[2312.00396]] GFN-SR: Symbolic Regression with Generative Flow Networks(http://arxiv.org/abs/2312.00396)
• Summary:

  Symbolic regression (SR) is an area of interpretable machine learning that aims to identify mathematical expressions, often composed of simple functions, that best fit in a given set of covariates $X$ and response $y$. In recent years, deep symbolic regression (DSR) has emerged as a popular method in the field by leveraging deep reinforcement learning to solve the complicated combinatorial search problem. In this work, we propose an alternative framework (GFN-SR) to approach SR with deep learning. We model the construction of an expression tree as traversing through a directed acyclic graph (DAG) so that GFlowNet can learn a stochastic policy to generate such trees sequentially. Enhanced with an adaptive reward baseline, our method is capable of generating a diverse set of best-fitting expressions. Notably, we observe that GFN-SR outperforms other SR algorithms in noisy data regimes, owing to its ability to learn a distribution of rewards over a space of candidate solutions.

large language model

Title: LEAP: LLM-Generation of Egocentric Action Programs. (arXiv:2312.00055v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00055
• Code URL: null
• Copy Paste: [[2312.00055]] LEAP: LLM-Generation of Egocentric Action Programs(http://arxiv.org/abs/2312.00055)
• Summary:

  We introduce LEAP (illustrated in Figure 1), a novel method for generating video-grounded action programs through use of a Large Language Model (LLM). These action programs represent the motoric, perceptual, and structural aspects of action, and consist of sub-actions, pre- and post-conditions, and control flows. LEAP's action programs are centered on egocentric video and employ recent developments in LLMs both as a source for program knowledge and as an aggregator and assessor of multimodal video information. We apply LEAP over a majority (87\%) of the training set of the EPIC Kitchens dataset, and release the resulting action programs as a publicly available dataset here (https://drive.google.com/drive/folders/1Cpkw_TI1IIxXdzor0pOXG3rWJWuKU5Ex?usp=drive_link). We employ LEAP as a secondary source of supervision, using its action programs in a loss term applied to action recognition and anticipation networks. We demonstrate sizable improvements in performance in both tasks due to training with the LEAP dataset. Our method achieves 1st place on the EPIC Kitchens Action Recognition leaderboard as of November 17 among the networks restricted to RGB-input (see Supplementary Materials).

Title: OST: Refining Text Knowledge with Optimal Spatio-Temporal Descriptor for General Video Recognition. (arXiv:2312.00096v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00096
• Code URL: null
• Copy Paste: [[2312.00096]] OST: Refining Text Knowledge with Optimal Spatio-Temporal Descriptor for General Video Recognition(http://arxiv.org/abs/2312.00096)
• Summary:

  Due to the resource-intensive nature of training vision-language models on expansive video data, a majority of studies have centered on adapting pre-trained image-language models to the video domain. Dominant pipelines propose to tackle the visual discrepancies with additional temporal learners while overlooking the substantial discrepancy for web-scaled descriptive narratives and concise action category names, leading to less distinct semantic space and potential performance limitations. In this work, we prioritize the refinement of text knowledge to facilitate generalizable video recognition. To address the limitations of the less distinct semantic space of category names, we prompt a large language model (LLM) to augment action class names into Spatio-Temporal Descriptors thus bridging the textual discrepancy and serving as a knowledge base for general recognition. Moreover, to assign the best descriptors with different video instances, we propose Optimal Descriptor Solver, forming the video recognition problem as solving the optimal matching flow across frame-level representations and descriptors. Comprehensive evaluations in zero-shot, few-shot, and fully supervised video recognition highlight the effectiveness of our approach. Our best model achieves a state-of-the-art zero-shot accuracy of 75.1% on Kinetics-600.

Title: A Video is Worth 10,000 Words: Training and Benchmarking with Diverse Captions for Better Long Video Retrieval. (arXiv:2312.00115v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00115
• Code URL: null
• Copy Paste: [[2312.00115]] A Video is Worth 10,000 Words: Training and Benchmarking with Diverse Captions for Better Long Video Retrieval(http://arxiv.org/abs/2312.00115)
• Summary:

  Existing long video retrieval systems are trained and tested in the paragraph-to-video retrieval regime, where every long video is described by a single long paragraph. This neglects the richness and variety of possible valid descriptions of a video, which could be described in moment-by-moment detail, or in a single phrase summary, or anything in between. To provide a more thorough evaluation of the capabilities of long video retrieval systems, we propose a pipeline that leverages state-of-the-art large language models to carefully generate a diverse set of synthetic captions for long videos. We validate this pipeline's fidelity via rigorous human inspection. We then benchmark a representative set of video language models on these synthetic captions using a few long video datasets, showing that they struggle with the transformed data, especially the shortest captions. We also propose a lightweight fine-tuning method, where we use a contrastive loss to learn a hierarchical embedding loss based on the differing levels of information among the various captions. Our method improves performance both on the downstream paragraph-to-video retrieval task (+1.1% R@1 on ActivityNet), as well as for the various long video retrieval metrics we compute using our synthetic data (+3.6% R@1 for short descriptions on ActivityNet). For data access and other details, please refer to our project website at https://mgwillia.github.io/10k-words.

Title: Merlin:Empowering Multimodal LLMs with Foresight Minds. (arXiv:2312.00589v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00589
• Code URL: null
• Copy Paste: [[2312.00589]] Merlin:Empowering Multimodal LLMs with Foresight Minds(http://arxiv.org/abs/2312.00589)
• Summary:

  Humans possess the remarkable ability to foresee the future to a certain extent based on present observations, a skill we term as foresight minds. However, this capability remains largely under explored within existing Multimodal Large Language Models (MLLMs), hindering their capacity to learn the fundamental principles of how things operate and the intentions behind the observed subjects. To address this issue, we introduce the integration of future modeling into the existing learning frameworks of MLLMs. By utilizing the subject trajectory, a highly structured representation of a consecutive frame sequence, as a learning objective, we aim to bridge the gap between the past and the future. We propose two innovative methods to empower MLLMs with foresight minds, Foresight Pre-Training (FPT) and Foresight Instruction-Tuning (FIT), which are inspired by the modern learning paradigm of LLMs. Specifically, FPT jointly training various tasks centered on trajectories, enabling MLLMs to learn how to attend and predict entire trajectories from a given initial observation. Then, FIT requires MLLMs to first predict trajectories of related objects and then reason about potential future events based on them. Aided by FPT and FIT, we build a novel and unified MLLM named Merlin that supports multi-images input and analysis about potential actions of multiple objects for the future reasoning. Experimental results show Merlin powerful foresight minds with impressive performance on both future reasoning and visual comprehension tasks.

Title: On Exploring the Reasoning Capability of Large Language Models with Knowledge Graphs. (arXiv:2312.00353v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.00353
• Code URL: null
• Copy Paste: [[2312.00353]] On Exploring the Reasoning Capability of Large Language Models with Knowledge Graphs(http://arxiv.org/abs/2312.00353)
• Summary:

  This paper examines the capacity of LLMs to reason with knowledge graphs using their internal knowledge graph, i.e., the knowledge graph they learned during pre-training. Two research questions are formulated to investigate the accuracy of LLMs in recalling information from pre-training knowledge graphs and their ability to infer knowledge graph relations from context. To address these questions, we employ LLMs to perform four distinct knowledge graph reasoning tasks. Furthermore, we identify two types of hallucinations that may occur during knowledge reasoning with LLMs: content and ontology hallucination. Our experimental results demonstrate that LLMs can successfully tackle both simple and complex knowledge graph reasoning tasks from their own memory, as well as infer from input context.

Title: CoLLiE: Collaborative Training of Large Language Models in an Efficient Way. (arXiv:2312.00407v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.00407
• Code URL: https://github.com/openlmlab/collie
• Copy Paste: [[2312.00407]] CoLLiE: Collaborative Training of Large Language Models in an Efficient Way(http://arxiv.org/abs/2312.00407)
• Summary:

  Large language models (LLMs) are increasingly pivotal in a wide range of natural language processing tasks. Access to pre-trained models, courtesy of the open-source community, has made it possible to adapt these models to specific applications for enhanced performance. However, the substantial resources required for training these models necessitate efficient solutions. This paper introduces CoLLiE, an efficient library that facilitates collaborative training of large language models using 3D parallelism, parameter-efficient fine-tuning (PEFT) methods, and optimizers such as Lion, Adan, Sophia, LOMO and AdaLomo. With its modular design and comprehensive functionality, CoLLiE offers a balanced blend of efficiency, ease of use, and customization. CoLLiE has proven superior training efficiency in comparison with prevalent solutions in pre-training and fine-tuning scenarios. Furthermore, we provide an empirical evaluation of the correlation between model size and GPU memory consumption under different optimization methods, as well as an analysis of the throughput. Lastly, we carry out a comprehensive comparison of various optimizers and PEFT methods within the instruction-tuning context. CoLLiE is available at https://github.com/OpenLMLab/collie.

Title: Questioning Biases in Case Judgment Summaries: Legal Datasets or Large Language Models?. (arXiv:2312.00554v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.00554
• Code URL: null
• Copy Paste: [[2312.00554]] Questioning Biases in Case Judgment Summaries: Legal Datasets or Large Language Models?(http://arxiv.org/abs/2312.00554)
• Summary:

  The evolution of legal datasets and the advent of large language models (LLMs) have significantly transformed the legal field, particularly in the generation of case judgment summaries. However, a critical concern arises regarding the potential biases embedded within these summaries. This study scrutinizes the biases present in case judgment summaries produced by legal datasets and large language models. The research aims to analyze the impact of biases on legal decision making. By interrogating the accuracy, fairness, and implications of biases in these summaries, this study contributes to a better understanding of the role of technology in legal contexts and the implications for justice systems worldwide. In this study, we investigate biases wrt Gender-related keywords, Race-related keywords, Keywords related to crime against women, Country names and religious keywords. The study shows interesting evidences of biases in the outputs generated by the large language models and pre-trained abstractive summarization models. The reasoning behind these biases needs further studies.

Title: Instruction-tuning Aligns LLMs to the Human Brain. (arXiv:2312.00575v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.00575
• Code URL: null
• Copy Paste: [[2312.00575]] Instruction-tuning Aligns LLMs to the Human Brain(http://arxiv.org/abs/2312.00575)
• Summary:

  Instruction-tuning is a widely adopted method of finetuning that enables large language models (LLMs) to generate output that more closely resembles human responses to natural language queries, in many cases leading to human-level performance on diverse testbeds. However, it remains unclear whether instruction-tuning truly makes LLMs more similar to how humans process language. We investigate the effect of instruction-tuning on LLM-human similarity in two ways: (1) brain alignment, the similarity of LLM internal representations to neural activity in the human language system, and (2) behavioral alignment, the similarity of LLM and human behavior on a reading task. We assess 25 vanilla and instruction-tuned LLMs across three datasets involving humans reading naturalistic stories and sentences. We discover that instruction-tuning generally enhances brain alignment by an average of 6%, but does not have a similar effect on behavioral alignment. To identify the factors underlying LLM-brain alignment, we compute correlations between the brain alignment of LLMs and various model properties, such as model size, various problem-solving abilities, and performance on tasks requiring world knowledge spanning various domains. Notably, we find a strong positive correlation between brain alignment and model size (r = 0.95), as well as performance on tasks requiring world knowledge (r = 0.81). Our results demonstrate that instruction-tuning LLMs improves both world knowledge representations and brain alignment, suggesting that mechanisms that encode world knowledge in LLMs also improve representational alignment to the human brain.

Title: The Efficiency Spectrum of Large Language Models: An Algorithmic Survey. (arXiv:2312.00678v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.00678
• Code URL: https://github.com/tding1/efficient-llm-survey
• Copy Paste: [[2312.00678]] The Efficiency Spectrum of Large Language Models: An Algorithmic Survey(http://arxiv.org/abs/2312.00678)
• Summary:

  The rapid growth of Large Language Models (LLMs) has been a driving force in transforming various domains, reshaping the artificial general intelligence landscape. However, the increasing computational and memory demands of these models present substantial challenges, hindering both academic research and practical applications. To address these issues, a wide array of methods, including both algorithmic and hardware solutions, have been developed to enhance the efficiency of LLMs. This survey delivers a comprehensive review of algorithmic advancements aimed at improving LLM efficiency. Unlike other surveys that typically focus on specific areas such as training or model compression, this paper examines the multi-faceted dimensions of efficiency essential for the end-to-end algorithmic development of LLMs. Specifically, it covers various topics related to efficiency, including scaling laws, data utilization, architectural innovations, training and tuning strategies, and inference techniques. This paper aims to serve as a valuable resource for researchers and practitioners, laying the groundwork for future innovations in this critical research area. Our repository of relevant references is maintained at url{https://github.com/tding1/Efficient-LLM-Survey}.

Title: SeaLLMs -- Large Language Models for Southeast Asia. (arXiv:2312.00738v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.00738
• Code URL: https://github.com/damo-nlp-sg/seallms
• Copy Paste: [[2312.00738]] SeaLLMs -- Large Language Models for Southeast Asia(http://arxiv.org/abs/2312.00738)
• Summary:

  Despite the remarkable achievements of large language models (LLMs) in various tasks, there remains a linguistic bias that favors high-resource languages, such as English, often at the expense of low-resource and regional languages. To address this imbalance, we introduce SeaLLMs, an innovative series of language models that specifically focuses on Southeast Asian (SEA) languages. SeaLLMs are built upon the Llama-2 model and further advanced through continued pre-training with an extended vocabulary, specialized instruction and alignment tuning to better capture the intricacies of regional languages. This allows them to respect and reflect local cultural norms, customs, stylistic preferences, and legal considerations. Our comprehensive evaluation demonstrates that SeaLLM-13b models exhibit superior performance across a wide spectrum of linguistic tasks and assistant-style instruction-following capabilities relative to comparable open-source models. Moreover, they outperform ChatGPT-3.5 in non-Latin languages, such as Thai, Khmer, Lao, and Burmese, by large margins while remaining lightweight and cost-effective to operate.

Title: Unleashing Cheapfakes through Trojan Plugins of Large Language Models. (arXiv:2312.00374v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.00374
• Code URL: null
• Copy Paste: [[2312.00374]] Unleashing Cheapfakes through Trojan Plugins of Large Language Models(http://arxiv.org/abs/2312.00374)
• Summary:

  Open-source Large Language Models (LLMs) have recently gained popularity because of their comparable performance to proprietary LLMs. To efficiently fulfill domain-specialized tasks, open-source LLMs can be refined, without expensive accelerators, using low-rank adapters. However, it is still unknown whether low-rank adapters can be exploited to control LLMs. To address this gap, we demonstrate that an infected adapter can induce, on specific triggers, an LLM to output content defined by an adversary and to even maliciously use tools. To train a Trojan adapter, we propose two novel attacks, POLISHED and FUSION, that improve over prior approaches. POLISHED uses LLM-enhanced paraphrasing to polish benchmark poisoned datasets. In contrast, in the absence of a dataset, FUSION leverages an over-poisoning procedure to transform a benign adaptor. Our experiments validate that our attacks provide higher attack effectiveness than the baseline and, for the purpose of attracting downloads, preserves or improves the adapter's utility. Finally, we provide two case studies to demonstrate that the Trojan adapter can lead a LLM-powered autonomous agent to execute unintended scripts or send phishing emails. Our novel attacks represent the first study of supply chain threats for LLMs through the lens of Trojan plugins.

Title: Sample Efficient Reinforcement Learning from Human Feedback via Active Exploration. (arXiv:2312.00267v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.00267
• Code URL: null
• Copy Paste: [[2312.00267]] Sample Efficient Reinforcement Learning from Human Feedback via Active Exploration(http://arxiv.org/abs/2312.00267)
• Summary:

  Preference-based feedback is important for many applications in reinforcement learning where direct evaluation of a reward function is not feasible. A notable recent example arises in reinforcement learning from human feedback (RLHF) on large language models. For many applications of RLHF, the cost of acquiring the human feedback can be substantial. In this work, we take advantage of the fact that one can often choose contexts at which to obtain human feedback in order to most efficiently identify a good policy, and formalize this as an offline contextual dueling bandit problem. We give an upper-confidence-bound style algorithm for this problem and prove a polynomial worst-case regret bound. We then provide empirical confirmation in a synthetic setting that our approach outperforms existing methods. After, we extend the setting and methodology for practical use in RLHF training of large language models. Here, our method is able to reach better performance with fewer samples of human preferences than multiple baselines on three real-world datasets.

Title: LinguaLinked: A Distributed Large Language Model Inference System for Mobile Devices. (arXiv:2312.00388v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.00388
• Code URL: null
• Copy Paste: [[2312.00388]] LinguaLinked: A Distributed Large Language Model Inference System for Mobile Devices(http://arxiv.org/abs/2312.00388)
• Summary:

  Deploying Large Language Models (LLMs) locally on mobile devices presents a significant challenge due to their extensive memory requirements. In this paper, we introduce LinguaLinked, a system for decentralized, distributed LLM inference on mobile devices. LinguaLinked enables collaborative execution of the inference task across multiple trusted devices. LinguaLinked ensures data privacy by processing information locally. LinguaLinked uses three key strategies. First, an optimized model assignment technique segments LLMs and uses linear optimization to align segments with each device's capabilities. Second, an optimized data transmission mechanism ensures efficient and structured data flow between model segments while also maintaining the integrity of the original model structure. Finally, LinguaLinked incorporates a runtime load balancer that actively monitors and redistributes tasks among mobile devices to prevent bottlenecks, enhancing the system's overall efficiency and responsiveness. We demonstrate that LinguaLinked facilitates efficient LLM inference while maintaining consistent throughput and minimal latency through extensive testing across various mobile devices, from high-end to low-end Android devices. In our evaluations, compared to the baseline, LinguaLinked achieves an inference performance acceleration of $1.11\times$ to $1.61\times$ in single-threaded settings, $1.73\times$ to $2.65\times$ with multi-threading. Additionally, runtime load balancing yields an overall inference acceleration of $1.29\times$ to $1.32\times$.

segmentation

Title: Un-EvMoSeg: Unsupervised Event-based Independent Motion Segmentation. (arXiv:2312.00114v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00114
• Code URL: null
• Copy Paste: [[2312.00114]] Un-EvMoSeg: Unsupervised Event-based Independent Motion Segmentation(http://arxiv.org/abs/2312.00114)
• Summary:

  Event cameras are a novel type of biologically inspired vision sensor known for their high temporal resolution, high dynamic range, and low power consumption. Because of these properties, they are well-suited for processing fast motions that require rapid reactions. Although event cameras have recently shown competitive performance in unsupervised optical flow estimation, performance in detecting independently moving objects (IMOs) is lacking behind, although event-based methods would be suited for this task based on their low latency and HDR properties. Previous approaches to event-based IMO segmentation have been heavily dependent on labeled data. However, biological vision systems have developed the ability to avoid moving objects through daily tasks without being given explicit labels. In this work, we propose the first event framework that generates IMO pseudo-labels using geometric constraints. Due to its unsupervised nature, our method can handle an arbitrary number of not predetermined objects and is easily scalable to datasets where expensive IMO labels are not readily available. We evaluate our approach on the EVIMO dataset and show that it performs competitively with supervised methods, both quantitatively and qualitatively.

Title: Integration of Swin UNETR and statistical shape modeling for a semi-automated segmentation of the knee and biomechanical modeling of articular cartilage. (arXiv:2312.00169v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00169
• Code URL: null
• Copy Paste: [[2312.00169]] Integration of Swin UNETR and statistical shape modeling for a semi-automated segmentation of the knee and biomechanical modeling of articular cartilage(http://arxiv.org/abs/2312.00169)
• Summary:

  Simulation studies like finite element (FE) modeling provide insight into knee joint mechanics without patient experimentation. Generic FE models represent biomechanical behavior of the tissue by overlooking variations in geometry, loading, and material properties of a population. On the other hand, subject-specific models include these specifics, resulting in enhanced predictive precision. However, creating such models is laborious and time-intensive. The present study aimed to enhance subject-specific knee joint FE modeling by incorporating a semi-automated segmentation algorithm. This segmentation was a 3D Swin UNETR for an initial segmentation of the femur and tibia, followed by a statistical shape model (SSM) adjustment to improve surface roughness and continuity. Five hundred and seven magnetic resonance images (MRIs) from the Osteoarthritis Initiative (OAI) database were used to build and validate the segmentation model. A semi-automated FE model was developed using this semi-automated segmentation. On the other hand, a manual FE model was developed through manual segmentation (i.e., the gold standard approach). Both FE models were subjected to gait loading. The predicted mechanical response of manual and semi-automated FE models were compared. In the result, our semi-automated segmentation achieved Dice similarity coefficient (DSC) over 98% for both femur and tibia. The mechanical results (max principal stress, max principal strain, fluid pressure, fibril strain, and contact area) showed no significant differences between the manual and semi-automated FE models, indicating the effectiveness of the proposed semi-automated segmentation in creating accurate knee joint FE models. ( https://data.mendeley.com/datasets/k5hdc9cz7w/1 ).

Title: 3D Face Reconstruction with the Geometric Guidance of Facial Part Segmentation. (arXiv:2312.00311v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00311
• Code URL: null
• Copy Paste: [[2312.00311]] 3D Face Reconstruction with the Geometric Guidance of Facial Part Segmentation(http://arxiv.org/abs/2312.00311)
• Summary:

  3D Morphable Models (3DMMs) provide promising 3D face reconstructions in various applications. However, existing methods struggle to reconstruct faces with extreme expressions due to deficiencies in supervisory signals, such as sparse or inaccurate landmarks. Segmentation information contains effective geometric contexts for face reconstruction. Certain attempts intuitively depend on differentiable renderers to compare the rendered silhouettes of reconstruction with segmentation, which is prone to issues like local optima and gradient instability. In this paper, we fully utilize the facial part segmentation geometry by introducing Part Re-projection Distance Loss (PRDL). Specifically, PRDL transforms facial part segmentation into 2D points and re-projects the reconstruction onto the image plane. Subsequently, by introducing grid anchors and computing different statistical distances from these anchors to the point sets, PRDL establishes geometry descriptors to optimize the distribution of the point sets for face reconstruction. PRDL exhibits a clear gradient compared to the renderer-based methods and presents state-of-the-art reconstruction performance in extensive quantitative and qualitative experiments. The project will be publicly available.

Title: Segment Anything Model-guided Collaborative Learning Network for Scribble-supervised Polyp Segmentation. (arXiv:2312.00312v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00312
• Code URL: null
• Copy Paste: [[2312.00312]] Segment Anything Model-guided Collaborative Learning Network for Scribble-supervised Polyp Segmentation(http://arxiv.org/abs/2312.00312)
• Summary:

  Polyp segmentation plays a vital role in accurately locating polyps at an early stage, which holds significant clinical importance for the prevention of colorectal cancer. Various polyp segmentation methods have been developed using fully-supervised deep learning techniques. However, pixel-wise annotation for polyp images by physicians during the diagnosis is both time-consuming and expensive. Moreover, visual foundation models such as the Segment Anything Model (SAM) have shown remarkable performance. Nevertheless, directly applying SAM to medical segmentation may not produce satisfactory results due to the inherent absence of medical knowledge. In this paper, we propose a novel SAM-guided Collaborative Learning Network (SAM-CLNet) for scribble-supervised polyp segmentation, enabling a collaborative learning process between our segmentation network and SAM to boost the model performance. Specifically, we first propose a Cross-level Enhancement and Aggregation Network (CEA-Net) for weakly-supervised polyp segmentation. Within CEA-Net, we propose a Cross-level Enhancement Module (CEM) that integrates the adjacent features to enhance the representation capabilities of different resolution features. Additionally, a Feature Aggregation Module (FAM) is employed to capture richer features across multiple levels. Moreover, we present a box-augmentation strategy that combines the segmentation maps generated by CEA-Net with scribble annotations to create more precise prompts. These prompts are then fed into SAM, generating segmentation SAM-guided masks, which can provide additional supervision to train CEA-Net effectively. Furthermore, we present an Image-level Filtering Mechanism to filter out unreliable SAM-guided masks. Extensive experimental results show that our SAM-CLNet outperforms state-of-the-art weakly-supervised segmentation methods.

Title: Improving Normalization with the James-Stein Estimator. (arXiv:2312.00313v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00313
• Code URL: null
• Copy Paste: [[2312.00313]] Improving Normalization with the James-Stein Estimator(http://arxiv.org/abs/2312.00313)
• Summary:

  Stein's paradox holds considerable sway in high-dimensional statistics, highlighting that the sample mean, traditionally considered the de facto estimator, might not be the most efficacious in higher dimensions. To address this, the James-Stein estimator proposes an enhancement by steering the sample means toward a more centralized mean vector. In this paper, first, we establish that normalization layers in deep learning use inadmissible estimators for mean and variance. Next, we introduce a novel method to employ the James-Stein estimator to improve the estimation of mean and variance within normalization layers. We evaluate our method on different computer vision tasks: image classification, semantic segmentation, and 3D object classification. Through these evaluations, it is evident that our improved normalization layers consistently yield superior accuracy across all tasks without extra computational burden. Moreover, recognizing that a plethora of shrinkage estimators surpass the traditional estimator in performance, we study two other prominent shrinkage estimators: Ridge and LASSO. Additionally, we provide visual representations to intuitively demonstrate the impact of shrinkage on the estimated layer statistics. Finally, we study the effect of regularization and batch size on our modified batch normalization. The studies show that our method is less sensitive to batch size and regularization, improving accuracy under various setups.

Title: Improving Efficiency of DNN-based Relocalization Module for Autonomous Driving with Server-side Computing. (arXiv:2312.00316v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00316
• Code URL: null
• Copy Paste: [[2312.00316]] Improving Efficiency of DNN-based Relocalization Module for Autonomous Driving with Server-side Computing(http://arxiv.org/abs/2312.00316)
• Summary:

  In this work, we present a novel framework for camera relocation in autonomous vehicles, leveraging deep neural networks (DNN). While existing literature offers various DNN-based camera relocation methods, their deployment is hindered by their high computational demands during inference. In contrast, our approach addresses this challenge through edge cloud collaboration. Specifically, we strategically offload certain modules of the neural network to the server and evaluate the inference time of data frames under different network segmentation schemes to guide our offloading decisions. Our findings highlight the vital role of server-side offloading in DNN-based camera relocation for autonomous vehicles, and we also discuss the results of data fusion. Finally, we validate the effectiveness of our proposed framework through experimental evaluation.

Title: Efficient Multimodal Semantic Segmentation via Dual-Prompt Learning. (arXiv:2312.00360v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00360
• Code URL: https://github.com/shaohuadong2021/dplnet
• Copy Paste: [[2312.00360]] Efficient Multimodal Semantic Segmentation via Dual-Prompt Learning(http://arxiv.org/abs/2312.00360)
• Summary:

  Multimodal (e.g., RGB-Depth/RGB-Thermal) fusion has shown great potential for improving semantic segmentation in complex scenes (e.g., indoor/low-light conditions). Existing approaches often fully fine-tune a dual-branch encoder-decoder framework with a complicated feature fusion strategy for achieving multimodal semantic segmentation, which is training-costly due to the massive parameter updates in feature extraction and fusion. To address this issue, we propose a surprisingly simple yet effective dual-prompt learning network (dubbed DPLNet) for training-efficient multimodal (e.g., RGB-D/T) semantic segmentation. The core of DPLNet is to directly adapt a frozen pre-trained RGB model to multimodal semantic segmentation, reducing parameter updates. For this purpose, we present two prompt learning modules, comprising multimodal prompt generator (MPG) and multimodal feature adapter (MFA). MPG works to fuse the features from different modalities in a compact manner and is inserted from shadow to deep stages to generate the multi-level multimodal prompts that are injected into the frozen backbone, while MPG adapts prompted multimodal features in the frozen backbone for better multimodal semantic segmentation. Since both the MPG and MFA are lightweight, only a few trainable parameters (3.88M, 4.4% of the pre-trained backbone parameters) are introduced for multimodal feature fusion and learning. Using a simple decoder (3.27M parameters), DPLNet achieves new state-of-the-art performance or is on a par with other complex approaches on four RGB-D/T semantic segmentation datasets while satisfying parameter efficiency. Moreover, we show that DPLNet is general and applicable to other multimodal tasks such as salient object detection and video semantic segmentation. Without special design, DPLNet outperforms many complicated models. Our code will be available at github.com/ShaohuaDong2021/DPLNet.

Title: Towards Generalizable Referring Image Segmentation via Target Prompt and Visual Coherence. (arXiv:2312.00452v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00452
• Code URL: null
• Copy Paste: [[2312.00452]] Towards Generalizable Referring Image Segmentation via Target Prompt and Visual Coherence(http://arxiv.org/abs/2312.00452)
• Summary:

  Referring image segmentation (RIS) aims to segment objects in an image conditioning on free-from text descriptions. Despite the overwhelming progress, it still remains challenging for current approaches to perform well on cases with various text expressions or with unseen visual entities, limiting its further application. In this paper, we present a novel RIS approach, which substantially improves the generalization ability by addressing the two dilemmas mentioned above. Specially, to deal with unconstrained texts, we propose to boost a given expression with an explicit and crucial prompt, which complements the expression in a unified context, facilitating target capturing in the presence of linguistic style changes. Furthermore, we introduce a multi-modal fusion aggregation module with visual guidance from a powerful pretrained model to leverage spatial relations and pixel coherences to handle the incomplete target masks and false positive irregular clumps which often appear on unseen visual entities. Extensive experiments are conducted in the zero-shot cross-dataset settings and the proposed approach achieves consistent gains compared to the state-of-the-art, e.g., 4.15\%, 5.45\%, and 4.64\% mIoU increase on RefCOCO, RefCOCO+ and ReferIt respectively, demonstrating its effectiveness. Additionally, the results on GraspNet-RIS show that our approach also generalizes well to new scenarios with large domain shifts.

Title: LiDAR-based curb detection for ground truth annotation in automated driving validation. (arXiv:2312.00534v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00534
• Code URL: null
• Copy Paste: [[2312.00534]] LiDAR-based curb detection for ground truth annotation in automated driving validation(http://arxiv.org/abs/2312.00534)
• Summary:

  Curb detection is essential for environmental awareness in Automated Driving (AD), as it typically limits drivable and non-drivable areas. Annotated data are necessary for developing and validating an AD function. However, the number of public datasets with annotated point cloud curbs is scarce. This paper presents a method for detecting 3D curbs in a sequence of point clouds captured from a LiDAR sensor, which consists of two main steps. First, our approach detects the curbs at each scan using a segmentation deep neural network. Then, a sequence-level processing step estimates the 3D curbs in the reconstructed point cloud using the odometry of the vehicle. From these 3D points of the curb, we obtain polylines structured following ASAM OpenLABEL standard. These detections can be used as pre-annotations in labelling pipelines to efficiently generate curb-related ground truth data. We validate our approach through an experiment in which different human annotators were required to annotate curbs in a group of LiDAR-based sequences with and without our automatically generated pre-annotations. The results show that the manual annotation time is reduced by 50.99% thanks to our detections, keeping the data quality level.

Title: CellMixer: Annotation-free Semantic Cell Segmentation of Heterogeneous Cell Populations. (arXiv:2312.00671v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.00671
• Code URL: null
• Copy Paste: [[2312.00671]] CellMixer: Annotation-free Semantic Cell Segmentation of Heterogeneous Cell Populations(http://arxiv.org/abs/2312.00671)
• Summary:

  In recent years, several unsupervised cell segmentation methods have been presented, trying to omit the requirement of laborious pixel-level annotations for the training of a cell segmentation model. Most if not all of these methods handle the instance segmentation task by focusing on the detection of different cell instances ignoring their type. While such models prove adequate for certain tasks, like cell counting, other applications require the identification of each cell's type. In this paper, we present CellMixer, an innovative annotation-free approach for the semantic segmentation of heterogeneous cell populations. Our augmentation-based method enables the training of a segmentation model from image-level labels of homogeneous cell populations. Our results show that CellMixer can achieve competitive segmentation performance across multiple cell types and imaging modalities, demonstrating the method's scalability and potential for broader applications in medical imaging, cellular biology, and diagnostics.