secure

Title: SigScatNet: A Siamese + Scattering based Deep Learning Approach for Signature Forgery Detection and Similarity Assessment. (arXiv:2311.05579v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.05579
• Code URL: null
• Copy Paste: [[2311.05579]] SigScatNet: A Siamese + Scattering based Deep Learning Approach for Signature Forgery Detection and Similarity Assessment(http://arxiv.org/abs/2311.05579)
• Summary:

  The surge in counterfeit signatures has inflicted widespread inconveniences and formidable challenges for both individuals and organizations. This groundbreaking research paper introduces SigScatNet, an innovative solution to combat this issue by harnessing the potential of a Siamese deep learning network, bolstered by Scattering wavelets, to detect signature forgery and assess signature similarity. The Siamese Network empowers us to ascertain the authenticity of signatures through a comprehensive similarity index, enabling precise validation and comparison. Remarkably, the integration of Scattering wavelets endows our model with exceptional efficiency, rendering it light enough to operate seamlessly on cost-effective hardware systems. To validate the efficacy of our approach, extensive experimentation was conducted on two open-sourced datasets: the ICDAR SigComp Dutch dataset and the CEDAR dataset. The experimental results demonstrate the practicality and resounding success of our proposed SigScatNet, yielding an unparalleled Equal Error Rate of 3.689% with the ICDAR SigComp Dutch dataset and an astonishing 0.0578% with the CEDAR dataset. Through the implementation of SigScatNet, our research spearheads a new state-of-the-art in signature analysis in terms of EER scores and computational efficiency, offering an advanced and accessible solution for detecting forgery and quantifying signature similarities. By employing cutting-edge Siamese deep learning and Scattering wavelets, we provide a robust framework that paves the way for secure and efficient signature verification systems.

Title: DeepLearningBrasil@LT-EDI-2023: Exploring Deep Learning Techniques for Detecting Depression in Social Media Text. (arXiv:2311.05047v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.05047
• Code URL: null
• Copy Paste: [[2311.05047]] DeepLearningBrasil@LT-EDI-2023: Exploring Deep Learning Techniques for Detecting Depression in Social Media Text(http://arxiv.org/abs/2311.05047)
• Summary:

  In this paper, we delineate the strategy employed by our team, DeepLearningBrasil, which secured us the first place in the shared task DepSign-LT-EDI@RANLP-2023, achieving a 47.0% Macro F1-Score and a notable 2.4% advantage. The task was to classify social media texts into three distinct levels of depression - "not depressed," "moderately depressed," and "severely depressed." Leveraging the power of the RoBERTa and DeBERTa models, we further pre-trained them on a collected Reddit dataset, specifically curated from mental health-related Reddit's communities (Subreddits), leading to an enhanced understanding of nuanced mental health discourse. To address lengthy textual data, we used truncation techniques that retained the essence of the content by focusing on its beginnings and endings. Our model was robust against unbalanced data by incorporating sample weights into the loss. Cross-validation and ensemble techniques were then employed to combine our k-fold trained models, delivering an optimal solution. The accompanying code is made available for transparency and further development.

Title: Embedded Platform Patterns for Distributed and Secure Logging. (arXiv:2311.05037v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.05037
• Code URL: null
• Copy Paste: [[2311.05037]] Embedded Platform Patterns for Distributed and Secure Logging(http://arxiv.org/abs/2311.05037)
• Summary:

  With the advent of modern embedded systems, logging as a process is becoming more and more prevalent for diagnostic and analytic services. Traditionally, storage and managing of the logged data are generally kept as a part of one entity together with the main logic components. In systems that implement network connections, this activity is usually handled over a remote device. However, enabling remote connection is still considered a limiting factor for many embedded devices due to the demanding production cost. A significant challenge is presented to vendors who need to decide how the data will be extracted and handled for an embedded platform during the design concept phase. It is generally desirable that logging memory modules are able to be addressed as separate units. These devices need to be appropriately secured and verifiable on a different system since data compromise can lead to enormous privacy and even financial losses. In this paper, we present two patterns. First, a pattern that allows flexible logging operation design in terms of module and interface responsibility separation. Second, a pattern for the design of secure logging processes during the utilization of constrained embedded devices. The introduced patterns fulfil the following conditions: (i) flexibility, design is independent of the chip vendors making the logging memory modules easily replaceable, (ii) self-sufficiency, every logging controller is maintained as a separate entity in a decentralized topology, (iii) security, through providing authenticity, confidentiality, and integrity by means of using a dedicated security module.

Title: Trust your BMS: Designing a Lightweight Authentication Architecture for Industrial Networks. (arXiv:2311.05498v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.05498
• Code URL: null
• Copy Paste: [[2311.05498]] Trust your BMS: Designing a Lightweight Authentication Architecture for Industrial Networks(http://arxiv.org/abs/2311.05498)
• Summary:

  With the advent of clean energy awareness and systems that rely on extensive battery usage, the community has seen an increased interest in the development of more complex and secure Battery Management Systems (BMS). In particular, the inclusion of BMS in modern complex systems like electric vehicles and power grids has presented a new set of security-related challenges. A concern is shown when BMS are intended to extend their communication with external system networks, as their interaction can leave many backdoors open that potential attackers could exploit. Hence, it is highly desirable to find a general design that can be used for BMS and its system inclusion. In this work, a security architecture solution is proposed intended for the communication between BMS and other system devices. The aim of the proposed architecture is to be easily applicable in different industrial settings and systems, while at the same time keeping the design lightweight in nature.

Title: A Comprehensive Survey of Threshold Digital Signatures: NIST Standards, Post-Quantum Cryptography, Exotic Techniques, and Real-World Applications. (arXiv:2311.05514v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.05514
• Code URL: null
• Copy Paste: [[2311.05514]] A Comprehensive Survey of Threshold Digital Signatures: NIST Standards, Post-Quantum Cryptography, Exotic Techniques, and Real-World Applications(http://arxiv.org/abs/2311.05514)
• Summary:

  Threshold digital signatures enable a distributed execution of signature functionalities and will play a crucial role in the security of emerging decentralized next-generation networked systems and applications. In this paper, we provide a comprehensive and systematic survey of threshold and distributed signatures with advanced features. Our survey encompasses threshold signatures in conventional and post-quantum cryptography (PQC) settings and captures custom-design and standard signatures (e.g., conventional NIST and NIST-PQC). We examine both generic (via secure multi-party computation) and custom thresholding techniques for a myriad of signature families while investigating exotic signatures, real-life applications, and potential future research direction.

security

Title: Rust for Embedded Systems: Current State, Challenges and Open Problems. (arXiv:2311.05063v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.05063
• Code URL: null
• Copy Paste: [[2311.05063]] Rust for Embedded Systems: Current State, Challenges and Open Problems(http://arxiv.org/abs/2311.05063)
• Summary:

  Embedded software is used in safety-critical systems such as medical devices and autonomous vehicles, where software defects, including security vulnerabilities, have severe consequences. Most embedded codebases are developed in unsafe languages, specifically C/C++, and are riddled with memory safety vulnerabilities. To prevent such vulnerabilities, RUST, a performant memory-safe systems language, provides an optimal choice for developing embedded software. RUST interoperability enables developing RUST applications on top of existing C codebases. Despite this, even the most resourceful organizations continue to develop embedded software in C/C++. This paper performs the first systematic study to holistically understand the current state and challenges of using RUST for embedded systems. Our study is organized across three research questions. We collected a dataset of 2,836 RUST embedded software spanning various categories and 5 Static Application Security Testing ( SAST) tools. We performed a systematic analysis of our dataset and surveys with 225 developers to investigate our research questions. We found that existing RUST software support is inadequate, SAST tools cannot handle certain features of RUST embedded software, resulting in failures, and the prevalence of advanced types in existing RUST software makes it challenging to engineer interoperable code. In addition, we found various challenges faced by developers in using RUST for embedded systems development.

Title: Can we run our Ethereum nodes at home?. (arXiv:2311.05252v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.05252
• Code URL: null
• Copy Paste: [[2311.05252]] Can we run our Ethereum nodes at home?(http://arxiv.org/abs/2311.05252)
• Summary:

  Scalability is a common issue among the most used permissionless blockchains, and several approaches have been proposed to solve this issue. Tackling scalability while preserving the security and decentralization of the network is a significant challenge. To deliver effective scaling solutions, Ethereum achieved a major protocol improvement, including a change in the consensus mechanism towards Proof of Stake. This improvement aimed a vast reduction of the hardware requirements to run a node, leading to significant sustainability benefits with a lower network energy consumption. This work analyzes the resource usage behavior of different clients running as Ethereum consensus nodes, comparing their performance under different configurations and analyzing their differences. Our results show higher requirements than claimed initially and how different clients react to network perturbations. Furthermore, we discuss the differences between the consensus clients, including their strong points and limitations.

Title: Finding Software Vulnerabilities in Open-Source C Projects via Bounded Model Checking. (arXiv:2311.05281v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.05281
• Code URL: https://github.com/janislley/lsverifier
• Copy Paste: [[2311.05281]] Finding Software Vulnerabilities in Open-Source C Projects via Bounded Model Checking(http://arxiv.org/abs/2311.05281)
• Summary:

  Computer-based systems have solved several domain problems, including industrial, military, education, and wearable. Nevertheless, such arrangements need high-quality software to guarantee security and safety as both are mandatory for modern software products. We advocate that bounded model-checking techniques can efficiently detect vulnerabilities in general software systems. However, such an approach struggles to scale up and verify extensive code bases. Consequently, we have developed and evaluated a methodology to verify large software systems using a state-of-the-art bounded model checker. In particular, we pre-process input source-code files and guide the respective model checker to explore them systematically. Moreover, the proposed scheme includes a function-wise prioritization strategy, which readily provides results for code entities according to a scale of importance. Experimental results using a real implementation of the proposed methodology show that it can efficiently verify large software systems. Besides, it presented low peak memory allocation when executed. We have evaluated our approach by verifying twelve popular open-source C projects, where we have found real software vulnerabilities that their developers confirmed.

Title: ChatGPT and other Large Language Models for Cybersecurity of Smart Grid Applications. (arXiv:2311.05462v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.05462
• Code URL: null
• Copy Paste: [[2311.05462]] ChatGPT and other Large Language Models for Cybersecurity of Smart Grid Applications(http://arxiv.org/abs/2311.05462)
• Summary:

  Cybersecurity breaches targeting electrical substations constitute a significant threat to the integrity of the power grid, necessitating comprehensive defense and mitigation strategies. Any anomaly in information and communication technology (ICT) should be detected for secure communications between devices in digital substations. This paper proposes large language models (LLM), e.g., ChatGPT, for the cybersecurity of IEC 61850-based digital substation communications. Multicast messages such as generic object oriented substation event (GOOSE) and sampled value (SV) are used for case studies. The proposed LLM-based cybersecurity framework includes for the first time data pre-processing of communication systems and human-in-the-loop (HITL) training (considering the cybersecurity guidelines recommended by humans). The results show a comparative analysis of detected anomaly data carried out based on the performance evaluation metrics for different LLMs. A hardware-in-the-loop (HIL) testbed is used to generate and extract a dataset of IEC 61850 communications.

privacy

Title: SynFacePAD 2023: Competition on Face Presentation Attack Detection Based on Privacy-aware Synthetic Training Data. (arXiv:2311.05336v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.05336
• Code URL: https://github.com/zi-yuanyang/ijcb-synfacepad-dig
• Copy Paste: [[2311.05336]] SynFacePAD 2023: Competition on Face Presentation Attack Detection Based on Privacy-aware Synthetic Training Data(http://arxiv.org/abs/2311.05336)
• Summary:

  This paper presents a summary of the Competition on Face Presentation Attack Detection Based on Privacy-aware Synthetic Training Data (SynFacePAD 2023) held at the 2023 International Joint Conference on Biometrics (IJCB 2023). The competition attracted a total of 8 participating teams with valid submissions from academia and industry. The competition aimed to motivate and attract solutions that target detecting face presentation attacks while considering synthetic-based training data motivated by privacy, legal and ethical concerns associated with personal data. To achieve that, the training data used by the participants was limited to synthetic data provided by the organizers. The submitted solutions presented innovations and novel approaches that led to outperforming the considered baseline in the investigated benchmarks.

Title: PRODIGy: a PROfile-based DIalogue Generation dataset. (arXiv:2311.05195v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.05195
• Code URL: null
• Copy Paste: [[2311.05195]] PRODIGy: a PROfile-based DIalogue Generation dataset(http://arxiv.org/abs/2311.05195)
• Summary:

  Providing dialogue agents with a profile representation can improve their consistency and coherence, leading to better conversations. However, current profile-based dialogue datasets for training such agents contain either explicit profile representations that are simple and dialogue-specific, or implicit representations that are difficult to collect. In this work, we propose a unified framework in which we bring together both standard and more sophisticated profile representations by creating a new resource where each dialogue is aligned with all possible speaker representations such as communication style, biographies, and personality. This framework allows to test several baselines built using generative language models with several profile configurations. The automatic evaluation shows that profile-based models have better generalisation capabilities than models trained on dialogues only, both in-domain and cross-domain settings. These results are consistent for fine-tuned models and instruction-based LLMs. Additionally, human evaluation demonstrates a clear preference for generations consistent with both profile and context. Finally, to account for possible privacy concerns, all experiments are done under two configurations: inter-character and intra-character. In the former, the LM stores the information about the character in its internal representation, while in the latter, the LM does not retain any personal information but uses it only at inference time.

Title: Edge-assisted U-Shaped Split Federated Learning with Privacy-preserving for Internet of Things. (arXiv:2311.04944v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.04944
• Code URL: null
• Copy Paste: [[2311.04944]] Edge-assisted U-Shaped Split Federated Learning with Privacy-preserving for Internet of Things(http://arxiv.org/abs/2311.04944)
• Summary:

  In the realm of the Internet of Things (IoT), deploying deep learning models to process data generated or collected by IoT devices is a critical challenge. However, direct data transmission can cause network congestion and inefficient execution, given that IoT devices typically lack computation and communication capabilities. Centralized data processing in data centers is also no longer feasible due to concerns over data privacy and security. To address these challenges, we present an innovative Edge-assisted U-Shaped Split Federated Learning (EUSFL) framework, which harnesses the high-performance capabilities of edge servers to assist IoT devices in model training and optimization process. In this framework, we leverage Federated Learning (FL) to enable data holders to collaboratively train models without sharing their data, thereby enhancing data privacy protection by transmitting only model parameters. Additionally, inspired by Split Learning (SL), we split the neural network into three parts using U-shaped splitting for local training on IoT devices. By exploiting the greater computation capability of edge servers, our framework effectively reduces overall training time and allows IoT devices with varying capabilities to perform training tasks efficiently. Furthermore, we proposed a novel noise mechanism called LabelDP to ensure that data features and labels can securely resist reconstruction attacks, eliminating the risk of privacy leakage. Our theoretical analysis and experimental results demonstrate that EUSFL can be integrated with various aggregation algorithms, maintaining good performance across different computing capabilities of IoT devices, and significantly reducing training time and local computation overhead.

Title: A Survey on Privacy of Health Data Lifecycle: A Taxonomy, Review, and Future Directions. (arXiv:2311.05404v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.05404
• Code URL: null
• Copy Paste: [[2311.05404]] A Survey on Privacy of Health Data Lifecycle: A Taxonomy, Review, and Future Directions(http://arxiv.org/abs/2311.05404)
• Summary:

  With the increasing breaches and security threats that endanger health data, ensuring patients' privacy is essential. To that end, the research community has proposed various privacy-preserving approaches based on cryptography, hashing, or ledger technologies for alleviating health data vulnerability. To establish a comprehensive understanding of health data privacy risks, and the benefits and limitations of existing privacy-preserving approaches, we perform a detailed review of existing work and distill 10 distinct privacy concerns occurring in a health data lifecycle. Furthermore, we classify existing approaches based on their applicability to particular privacy concerns occurring at a particular lifecycle stage. Finally, we propose a taxonomy of techniques used for privacy preservation in healthcare and triangulate those techniques with the lifecycle stages and concerns. Our review indicates heavy usage of cryptographical techniques in this domain. However, we have also found that healthcare systems have special requirements that require novel cryptographic techniques and security schemes to address special needs. Therefore, we identify several future research directions to mitigate the security challenges for privacy preservation in health data management.

protect

Title: Removing RLHF Protections in GPT-4 via Fine-Tuning. (arXiv:2311.05553v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.05553
• Code URL: null
• Copy Paste: [[2311.05553]] Removing RLHF Protections in GPT-4 via Fine-Tuning(http://arxiv.org/abs/2311.05553)
• Summary:

  As large language models (LLMs) have increased in their capabilities, so does their potential for dual use. To reduce harmful outputs, produces and vendors of LLMs have used reinforcement learning with human feedback (RLHF). In tandem, LLM vendors have been increasingly enabling fine-tuning of their most powerful models. However, concurrent work has shown that fine-tuning can remove RLHF protections. We may expect that the most powerful models currently available (GPT-4) are less susceptible to fine-tuning attacks.

  In this work, we show the contrary: fine-tuning allows attackers to remove RLHF protections with as few as 340 examples and a 95% success rate. These training examples can be automatically generated with weaker models. We further show that removing RLHF protections does not decrease usefulness on non-censored outputs, providing evidence that our fine-tuning strategy does not decrease usefulness despite using weaker models to generate training data. Our results show the need for further research on protections on LLMs.

defense

attack

Title: Familiarity-Based Open-Set Recognition Under Adversarial Attacks. (arXiv:2311.05006v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.05006
• Code URL: null
• Copy Paste: [[2311.05006]] Familiarity-Based Open-Set Recognition Under Adversarial Attacks(http://arxiv.org/abs/2311.05006)
• Summary:

  Open-set recognition (OSR), the identification of novel categories, can be a critical component when deploying classification models in real-world applications. Recent work has shown that familiarity-based scoring rules such as the Maximum Softmax Probability (MSP) or the Maximum Logit Score (MLS) are strong baselines when the closed-set accuracy is high. However, one of the potential weaknesses of familiarity-based OSR are adversarial attacks. Here, we present gradient-based adversarial attacks on familiarity scores for both types of attacks, False Familiarity and False Novelty attacks, and evaluate their effectiveness in informed and uninformed settings on TinyImageNet.

Title: FigStep: Jailbreaking Large Vision-language Models via Typographic Visual Prompts. (arXiv:2311.05608v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.05608
• Code URL: https://github.com/thuccslab/figstep
• Copy Paste: [[2311.05608]] FigStep: Jailbreaking Large Vision-language Models via Typographic Visual Prompts(http://arxiv.org/abs/2311.05608)
• Summary:

  Large vision-language models (VLMs) like GPT-4V represent an unprecedented revolution in the field of artificial intelligence (AI). Compared to single-modal large language models (LLMs), VLMs possess more versatile capabilities by incorporating additional modalities (e.g., images). Meanwhile, there's a rising enthusiasm in the AI community to develop open-source VLMs, such as LLaVA and MiniGPT4, which, however, have not undergone rigorous safety assessment. In this paper, to demonstrate that more modalities lead to unforeseen AI safety issues, we propose FigStep, a novel jailbreaking framework against VLMs. FigStep feeds harmful instructions into VLMs through the image channel and then uses benign text prompts to induce VLMs to output contents that violate common AI safety policies. Our experimental results show that FigStep can achieve an average attack success rate of 94.8% across 2 families of popular open-source VLMs, LLaVA and MiniGPT4 (a total of 5 VLMs). Moreover, we demonstrate that the methodology of FigStep can even jailbreak GPT-4V, which already leverages several system-level mechanisms to filter harmful queries. Above all, our experimental results reveal that VLMs are vulnerable to jailbreaking attacks, which highlights the necessity of novel safety alignments between visual and textual modalities.

Title: Counter-Empirical Attacking based on Adversarial Reinforcement Learning for Time-Relevant Scoring System. (arXiv:2311.05144v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.05144
• Code URL: https://github.com/sheldonresearch/microsoft-scoring-system
• Copy Paste: [[2311.05144]] Counter-Empirical Attacking based on Adversarial Reinforcement Learning for Time-Relevant Scoring System(http://arxiv.org/abs/2311.05144)
• Summary:

  Scoring systems are commonly seen for platforms in the era of big data. From credit scoring systems in financial services to membership scores in E-commerce shopping platforms, platform managers use such systems to guide users towards the encouraged activity pattern, and manage resources more effectively and more efficiently thereby. To establish such scoring systems, several "empirical criteria" are firstly determined, followed by dedicated top-down design for each factor of the score, which usually requires enormous effort to adjust and tune the scoring function in the new application scenario. What's worse, many fresh projects usually have no ground-truth or any experience to evaluate a reasonable scoring system, making the designing even harder. To reduce the effort of manual adjustment of the scoring function in every new scoring system, we innovatively study the scoring system from the preset empirical criteria without any ground truth, and propose a novel framework to improve the system from scratch. In this paper, we propose a "counter-empirical attacking" mechanism that can generate "attacking" behavior traces and try to break the empirical rules of the scoring system. Then an adversarial "enhancer" is applied to evaluate the scoring system and find the improvement strategy. By training the adversarial learning problem, a proper scoring function can be learned to be robust to the attacking activity traces that are trying to violate the empirical criteria. Extensive experiments have been conducted on two scoring systems including a shared computing resource platform and a financial credit system. The experimental results have validated the effectiveness of our proposed framework.

Title: ABIGX: A Unified Framework for eXplainable Fault Detection and Classification. (arXiv:2311.05316v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.05316
• Code URL: null
• Copy Paste: [[2311.05316]] ABIGX: A Unified Framework for eXplainable Fault Detection and Classification(http://arxiv.org/abs/2311.05316)
• Summary:

  For explainable fault detection and classification (FDC), this paper proposes a unified framework, ABIGX (Adversarial fault reconstruction-Based Integrated Gradient eXplanation). ABIGX is derived from the essentials of previous successful fault diagnosis methods, contribution plots (CP) and reconstruction-based contribution (RBC). It is the first explanation framework that provides variable contributions for the general FDC models. The core part of ABIGX is the adversarial fault reconstruction (AFR) method, which rethinks the FR from the perspective of adversarial attack and generalizes to fault classification models with a new fault index. For fault classification, we put forward a new problem of fault class smearing, which intrinsically hinders the correct explanation. We prove that ABIGX effectively mitigates this problem and outperforms the existing gradient-based explanation methods. For fault detection, we theoretically bridge ABIGX with conventional fault diagnosis methods by proving that CP and RBC are the linear specifications of ABIGX. The experiments evaluate the explanations of FDC by quantitative metrics and intuitive illustrations, the results of which show the general superiority of ABIGX to other advanced explanation methods.

robust

Title: FireMatch: A Semi-Supervised Video Fire Detection Network Based on Consistency and Distribution Alignment. (arXiv:2311.05168v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.05168
• Code URL: null
• Copy Paste: [[2311.05168]] FireMatch: A Semi-Supervised Video Fire Detection Network Based on Consistency and Distribution Alignment(http://arxiv.org/abs/2311.05168)
• Summary:

  Deep learning techniques have greatly enhanced the performance of fire detection in videos. However, video-based fire detection models heavily rely on labeled data, and the process of data labeling is particularly costly and time-consuming, especially when dealing with videos. Considering the limited quantity of labeled video data, we propose a semi-supervised fire detection model called FireMatch, which is based on consistency regularization and adversarial distribution alignment. Specifically, we first combine consistency regularization with pseudo-label. For unlabeled data, we design video data augmentation to obtain corresponding weakly augmented and strongly augmented samples. The proposed model predicts weakly augmented samples and retains pseudo-label above a threshold, while training on strongly augmented samples to predict these pseudo-labels for learning more robust feature representations. Secondly, we generate video cross-set augmented samples by adversarial distribution alignment to expand the training data and alleviate the decline in classification performance caused by insufficient labeled data. Finally, we introduce a fairness loss to help the model produce diverse predictions for input samples, thereby addressing the issue of high confidence with the non-fire class in fire classification scenarios. The FireMatch achieved an accuracy of 76.92% and 91.81% on two real-world fire datasets, respectively. The experimental results demonstrate that the proposed method outperforms the current state-of-the-art semi-supervised classification methods.

Title: Deep Learning in Computed Tomography Pulmonary Angiography Imaging: A Dual-Pronged Approach for Pulmonary Embolism Detection. (arXiv:2311.05197v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.05197
• Code URL: null
• Copy Paste: [[2311.05197]] Deep Learning in Computed Tomography Pulmonary Angiography Imaging: A Dual-Pronged Approach for Pulmonary Embolism Detection(http://arxiv.org/abs/2311.05197)
• Summary:

  Pulmonary Embolism (PE) is a critical medical condition characterized by obstructions in the pulmonary arteries. Despite being a major health concern, it often goes underdiagnosed leading to detrimental clinical outcomes. The increasing reliance on Computed Tomography Pulmonary Angiography for diagnosis presents challenges and a pressing need for enhanced diagnostic solutions. The primary objective of this study is to leverage deep learning techniques to enhance the Computer Assisted Diagnosis of PE. This study presents a comprehensive dual-pronged approach combining classification and detection for PE diagnosis. We introduce an Attention-Guided Convolutional Neural Network (AG-CNN) for classification, addressing both global and local lesion region. For detection, state-of-the-art models are employed to pinpoint potential PE regions. Different ensembling techniques further improve detection accuracy by combining predictions from different models. Finally, a heuristic strategy integrates classifier outputs with detection results, ensuring robust and accurate PE identification. Our attention-guided classification approach, tested on the Ferdowsi University of Mashhad's Pulmonary Embolism (FUMPE) dataset, outperformed the baseline model DenseNet-121 by achieving an 8.1% increase in the Area Under the Receiver Operating Characteristic. By employing ensemble techniques with detection models, the mean average precision (mAP) was considerably enhanced by a 4.7% increase. The classifier-guided framework further refined the mAP and F1 scores over the ensemble models. Our research offers a comprehensive approach to PE diagnostics using deep learning, addressing the prevalent issues of underdiagnosis and misdiagnosis. We aim to improve PE patient care by integrating AI solutions into clinical workflows, highlighting the potential of human-AI collaboration in medical diagnostics.

Title: Training Robust Deep Physiological Measurement Models with Synthetic Video-based Data. (arXiv:2311.05371v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.05371
• Code URL: null
• Copy Paste: [[2311.05371]] Training Robust Deep Physiological Measurement Models with Synthetic Video-based Data(http://arxiv.org/abs/2311.05371)
• Summary:

  Recent advances in supervised deep learning techniques have demonstrated the possibility to remotely measure human physiological vital signs (e.g., photoplethysmograph, heart rate) just from facial videos. However, the performance of these methods heavily relies on the availability and diversity of real labeled data. Yet, collecting large-scale real-world data with high-quality labels is typically challenging and resource intensive, which also raises privacy concerns when storing personal bio-metric data. Synthetic video-based datasets (e.g., SCAMPS~\cite{mcduff2022scamps}) with photo-realistic synthesized avatars are introduced to alleviate the issues while providing high-quality synthetic data. However, there exists a significant gap between synthetic and real-world data, which hinders the generalization of neural models trained on these synthetic datasets. In this paper, we proposed several measures to add real-world noise to synthetic physiological signals and corresponding facial videos. We experimented with individual and combined augmentation methods and evaluated our framework on three public real-world datasets. Our results show that we were able to reduce the average MAE from 6.9 to 2.0.

Title: SIRE: scale-invariant, rotation-equivariant estimation of artery orientations using graph neural networks. (arXiv:2311.05400v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.05400
• Code URL: null
• Copy Paste: [[2311.05400]] SIRE: scale-invariant, rotation-equivariant estimation of artery orientations using graph neural networks(http://arxiv.org/abs/2311.05400)
• Summary:

  Blood vessel orientation as visualized in 3D medical images is an important descriptor of its geometry that can be used for centerline extraction and subsequent segmentation and visualization. Arteries appear at many scales and levels of tortuosity, and determining their exact orientation is challenging. Recent works have used 3D convolutional neural networks (CNNs) for this purpose, but CNNs are sensitive to varying vessel sizes and orientations. We present SIRE: a scale-invariant, rotation-equivariant estimator for local vessel orientation. SIRE is modular and can generalise due to symmetry preservation.

  SIRE consists of a gauge equivariant mesh CNN (GEM-CNN) operating on multiple nested spherical meshes with different sizes in parallel. The features on each mesh are a projection of image intensities within the corresponding sphere. These features are intrinsic to the sphere and, in combination with the GEM-CNN, lead to SO(3)-equivariance. Approximate scale invariance is achieved by weight sharing and use of a symmetric maximum function to combine multi-scale predictions. Hence, SIRE can be trained with arbitrarily oriented vessels with varying radii to generalise to vessels with a wide range of calibres and tortuosity.

  We demonstrate the efficacy of SIRE using three datasets containing vessels of varying scales: the vascular model repository (VMR), the ASOCA coronary artery set, and a set of abdominal aortic aneurysms (AAAs). We embed SIRE in a centerline tracker which accurately tracks AAAs, regardless of the data SIRE is trained with. Moreover, SIRE can be used to track coronary arteries, even when trained only with AAAs.

  In conclusion, by incorporating SO(3) and scale symmetries, SIRE can determine the orientations of vessels outside of the training domain, forming a robust and data-efficient solution to geometric analysis of blood vessels in 3D medical images.

Title: Robust Retraining-free GAN Fingerprinting via Personalized Normalization. (arXiv:2311.05478v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.05478
• Code URL: null
• Copy Paste: [[2311.05478]] Robust Retraining-free GAN Fingerprinting via Personalized Normalization(http://arxiv.org/abs/2311.05478)
• Summary:

  In recent years, there has been significant growth in the commercial applications of generative models, licensed and distributed by model developers to users, who in turn use them to offer services. In this scenario, there is a need to track and identify the responsible user in the presence of a violation of the license agreement or any kind of malicious usage. Although there are methods enabling Generative Adversarial Networks (GANs) to include invisible watermarks in the images they produce, generating a model with a different watermark, referred to as a fingerprint, for each user is time- and resource-consuming due to the need to retrain the model to include the desired fingerprint. In this paper, we propose a retraining-free GAN fingerprinting method that allows model developers to easily generate model copies with the same functionality but different fingerprints. The generator is modified by inserting additional Personalized Normalization (PN) layers whose parameters (scaling and bias) are generated by two dedicated shallow networks (ParamGen Nets) taking the fingerprint as input. A watermark decoder is trained simultaneously to extract the fingerprint from the generated images. The proposed method can embed different fingerprints inside the GAN by just changing the input of the ParamGen Nets and performing a feedforward pass, without finetuning or retraining. The performance of the proposed method in terms of robustness against both model-level and image-level attacks is also superior to the state-of-the-art.

Title: Reconstructing Objects in-the-wild for Realistic Sensor Simulation. (arXiv:2311.05602v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.05602
• Code URL: null
• Copy Paste: [[2311.05602]] Reconstructing Objects in-the-wild for Realistic Sensor Simulation(http://arxiv.org/abs/2311.05602)
• Summary:

  Reconstructing objects from real world data and rendering them at novel views is critical to bringing realism, diversity and scale to simulation for robotics training and testing. In this work, we present NeuSim, a novel approach that estimates accurate geometry and realistic appearance from sparse in-the-wild data captured at distance and at limited viewpoints. Towards this goal, we represent the object surface as a neural signed distance function and leverage both LiDAR and camera sensor data to reconstruct smooth and accurate geometry and normals. We model the object appearance with a robust physics-inspired reflectance representation effective for in-the-wild data. Our experiments show that NeuSim has strong view synthesis performance on challenging scenarios with sparse training views. Furthermore, we showcase composing NeuSim assets into a virtual world and generating realistic multi-sensor data for evaluating self-driving perception models.

Title: Interpreting Pretrained Language Models via Concept Bottlenecks. (arXiv:2311.05014v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.05014
• Code URL: null
• Copy Paste: [[2311.05014]] Interpreting Pretrained Language Models via Concept Bottlenecks(http://arxiv.org/abs/2311.05014)
• Summary:

  Pretrained language models (PLMs) have made significant strides in various natural language processing tasks. However, the lack of interpretability due to their ``black-box'' nature poses challenges for responsible implementation. Although previous studies have attempted to improve interpretability by using, e.g., attention weights in self-attention layers, these weights often lack clarity, readability, and intuitiveness. In this research, we propose a novel approach to interpreting PLMs by employing high-level, meaningful concepts that are easily understandable for humans. For example, we learn the concept of ``Food'' and investigate how it influences the prediction of a model's sentiment towards a restaurant review. We introduce C$^3$M, which combines human-annotated and machine-generated concepts to extract hidden neurons designed to encapsulate semantically meaningful and task-specific concepts. Through empirical evaluations on real-world datasets, we manifest that our approach offers valuable insights to interpret PLM behavior, helps diagnose model failures, and enhances model robustness amidst noisy concept labels.

Title: Geometry-Calibrated DRO: Combating Over-Pessimism with Free Energy Implications. (arXiv:2311.05054v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.05054
• Code URL: null
• Copy Paste: [[2311.05054]] Geometry-Calibrated DRO: Combating Over-Pessimism with Free Energy Implications(http://arxiv.org/abs/2311.05054)
• Summary:

  Machine learning algorithms minimizing average risk are susceptible to distributional shifts. Distributionally Robust Optimization (DRO) addresses this issue by optimizing the worst-case risk within an uncertainty set. However, DRO suffers from over-pessimism, leading to low-confidence predictions, poor parameter estimations as well as poor generalization. In this work, we conduct a theoretical analysis of a probable root cause of over-pessimism: excessive focus on noisy samples. To alleviate the impact of noise, we incorporate data geometry into calibration terms in DRO, resulting in our novel Geometry-Calibrated DRO (GCDRO) for regression. We establish the connection between our risk objective and the Helmholtz free energy in statistical physics, and this free-energy-based risk can extend to standard DRO methods. Leveraging gradient flow in Wasserstein space, we develop an approximate minimax optimization algorithm with a bounded error ratio and elucidate how our approach mitigates noisy sample effects. Comprehensive experiments confirm GCDRO's superiority over conventional DRO methods.

Title: Generalized test utilities for long-tail performance in extreme multi-label classification. (arXiv:2311.05081v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.05081
• Code URL: null
• Copy Paste: [[2311.05081]] Generalized test utilities for long-tail performance in extreme multi-label classification(http://arxiv.org/abs/2311.05081)
• Summary:

  Extreme multi-label classification (XMLC) is the task of selecting a small subset of relevant labels from a very large set of possible labels. As such, it is characterized by long-tail labels, i.e., most labels have very few positive instances. With standard performance measures such as precision@k, a classifier can ignore tail labels and still report good performance. However, it is often argued that correct predictions in the tail are more interesting or rewarding, but the community has not yet settled on a metric capturing this intuitive concept. The existing propensity-scored metrics fall short on this goal by confounding the problems of long-tail and missing labels. In this paper, we analyze generalized metrics budgeted "at k" as an alternative solution. To tackle the challenging problem of optimizing these metrics, we formulate it in the expected test utility (ETU) framework, which aims at optimizing the expected performance on a fixed test set. We derive optimal prediction rules and construct computationally efficient approximations with provable regret guarantees and robustness against model misspecification. Our algorithm, based on block coordinate ascent, scales effortlessly to XMLC problems and obtains promising results in terms of long-tail performance.

Title: RAPID: Training-free Retrieval-based Log Anomaly Detection with PLM considering Token-level information. (arXiv:2311.05160v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.05160
• Code URL: https://github.com/dsba-lab/rapid
• Copy Paste: [[2311.05160]] RAPID: Training-free Retrieval-based Log Anomaly Detection with PLM considering Token-level information(http://arxiv.org/abs/2311.05160)
• Summary:

  As the IT industry advances, system log data becomes increasingly crucial. Many computer systems rely on log texts for management due to restricted access to source code. The need for log anomaly detection is growing, especially in real-world applications, but identifying anomalies in rapidly accumulating logs remains a challenging task. Traditional deep learning-based anomaly detection models require dataset-specific training, leading to corresponding delays. Notably, most methods only focus on sequence-level log information, which makes the detection of subtle anomalies harder, and often involve inference processes that are difficult to utilize in real-time. We introduce RAPID, a model that capitalizes on the inherent features of log data to enable anomaly detection without training delays, ensuring real-time capability. RAPID treats logs as natural language, extracting representations using pre-trained language models. Given that logs can be categorized based on system context, we implement a retrieval-based technique to contrast test logs with the most similar normal logs. This strategy not only obviates the need for log-specific training but also adeptly incorporates token-level information, ensuring refined and robust detection, particularly for unseen logs. We also propose the core set technique, which can reduce the computational cost needed for comparison. Experimental results show that even without training on log data, RAPID demonstrates competitive performance compared to prior models and achieves the best performance on certain datasets. Through various research questions, we verified its capability for real-time detection without delay.

biometric

steal

extraction

Title: POISE: Pose Guided Human Silhouette Extraction under Occlusions. (arXiv:2311.05077v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.05077
• Code URL: https://github.com/take2rohit/poise
• Copy Paste: [[2311.05077]] POISE: Pose Guided Human Silhouette Extraction under Occlusions(http://arxiv.org/abs/2311.05077)
• Summary:

  Human silhouette extraction is a fundamental task in computer vision with applications in various downstream tasks. However, occlusions pose a significant challenge, leading to incomplete and distorted silhouettes. To address this challenge, we introduce POISE: Pose Guided Human Silhouette Extraction under Occlusions, a novel self-supervised fusion framework that enhances accuracy and robustness in human silhouette prediction. By combining initial silhouette estimates from a segmentation model with human joint predictions from a 2D pose estimation model, POISE leverages the complementary strengths of both approaches, effectively integrating precise body shape information and spatial information to tackle occlusions. Furthermore, the self-supervised nature of \POISE eliminates the need for costly annotations, making it scalable and practical. Extensive experimental results demonstrate its superiority in improving silhouette extraction under occlusions, with promising results in downstream tasks such as gait recognition. The code for our method is available https://github.com/take2rohit/poise.

Title: Cross-modal Prompts: Adapting Large Pre-trained Models for Audio-Visual Downstream Tasks. (arXiv:2311.05152v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.05152
• Code URL: https://github.com/haoyi-duan/dg-sct
• Copy Paste: [[2311.05152]] Cross-modal Prompts: Adapting Large Pre-trained Models for Audio-Visual Downstream Tasks(http://arxiv.org/abs/2311.05152)
• Summary:

  In recent years, the deployment of large-scale pre-trained models in audio-visual downstream tasks has yielded remarkable outcomes. However, these models, primarily trained on single-modality unconstrained datasets, still encounter challenges in feature extraction for multi-modal tasks, leading to suboptimal performance. This limitation arises due to the introduction of irrelevant modality-specific information during encoding, which adversely affects the performance of downstream tasks. To address this challenge, this paper proposes a novel Dual-Guided Spatial-Channel-Temporal (DG-SCT) attention mechanism. This mechanism leverages audio and visual modalities as soft prompts to dynamically adjust the parameters of pre-trained models based on the current multi-modal input features. Specifically, the DG-SCT module incorporates trainable cross-modal interaction layers into pre-trained audio-visual encoders, allowing adaptive extraction of crucial information from the current modality across spatial, channel, and temporal dimensions, while preserving the frozen parameters of large-scale pre-trained models. Experimental evaluations demonstrate that our proposed model achieves state-of-the-art results across multiple downstream tasks, including AVE, AVVP, AVS, and AVQA. Furthermore, our model exhibits promising performance in challenging few-shot and zero-shot scenarios. The source code and pre-trained models are available at https://github.com/haoyi-duan/DG-SCT.

Title: Widely Applicable Strong Baseline for Sports Ball Detection and Tracking. (arXiv:2311.05237v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.05237
• Code URL: null
• Copy Paste: [[2311.05237]] Widely Applicable Strong Baseline for Sports Ball Detection and Tracking(http://arxiv.org/abs/2311.05237)
• Summary:

  In this work, we present a novel Sports Ball Detection and Tracking (SBDT) method that can be applied to various sports categories. Our approach is composed of (1) high-resolution feature extraction, (2) position-aware model training, and (3) inference considering temporal consistency, all of which are put together as a new SBDT baseline. Besides, to validate the wide-applicability of our approach, we compare our baseline with 6 state-of-the-art SBDT methods on 5 datasets from different sports categories. We achieve this by newly introducing two SBDT datasets, providing new ball annotations for two datasets, and re-implementing all the methods to ease extensive comparison. Experimental results demonstrate that our approach is substantially superior to existing methods on all the sports categories covered by the datasets. We believe our proposed method can play as a Widely Applicable Strong Baseline (WASB) of SBDT, and our datasets and codebase will promote future SBDT research. Datasets and codes will be made publicly available.

Title: Linear Gaussian Bounding Box Representation and Ring-Shaped Rotated Convolution for Oriented Object Detection. (arXiv:2311.05410v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.05410
• Code URL: https://github.com/zhen6618/rotayolo
• Copy Paste: [[2311.05410]] Linear Gaussian Bounding Box Representation and Ring-Shaped Rotated Convolution for Oriented Object Detection(http://arxiv.org/abs/2311.05410)
• Summary:

  Due to the frequent variability of object orientation, accurate prediction of orientation information remains a challenge in oriented object detection. To better extract orientation-related information, current methods primarily focus on the design of reasonable representations of oriented bounding box (OBB) and rotation-sensitive feature extraction. However, existing OBB representations often suffer from boundary discontinuity and representation ambiguity problems. Methods of designing continuous and unambiguous regression losses do not essentially solve such problems. Gaussian bounding box (GBB) avoids these OBB representation problems, but directly regressing GBB is susceptible to numerical instability. In this paper, we propose linear GBB (LGBB), a novel OBB representation. By linearly transforming the elements of GBB, LGBB does not have the boundary discontinuity and representation ambiguity problems, and have high numerical stability. On the other hand, current rotation-sensitive feature extraction methods based on convolutions can only extract features under a local receptive field, which is slow in aggregating rotation-sensitive features. To address this issue, we propose ring-shaped rotated convolution (RRC). By adaptively rotating feature maps to arbitrary orientations, RRC extracts rotation-sensitive features under a ring-shaped receptive field, rapidly aggregating rotation-sensitive features and contextual information. RRC can be applied to various models in a plug-and-play manner. Experimental results demonstrate that the proposed LGBB and RRC are effective and achieve state-of-the-art (SOTA) performance. By integrating LGBB and RRC into various models, the detection accuracy is effectively improved on DOTA and HRSC2016 datasets.

Title: Detecting Relevant Information in High-Volume Chat Logs: Keyphrase Extraction for Grooming and Drug Dealing Forensic Analysis. (arXiv:2311.04905v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.04905
• Code URL: null
• Copy Paste: [[2311.04905]] Detecting Relevant Information in High-Volume Chat Logs: Keyphrase Extraction for Grooming and Drug Dealing Forensic Analysis(http://arxiv.org/abs/2311.04905)
• Summary:

  The growing use of digital communication platforms has given rise to various criminal activities, such as grooming and drug dealing, which pose significant challenges to law enforcement and forensic experts. This paper presents a supervised keyphrase extraction approach to detect relevant information in high-volume chat logs involving grooming and drug dealing for forensic analysis. The proposed method, JointKPE++, builds upon the JointKPE keyphrase extractor by employing improvements to handle longer texts effectively. We evaluate JointKPE++ using BERT-based pre-trained models on grooming and drug dealing datasets, including BERT, RoBERTa, SpanBERT, and BERTimbau. The results show significant improvements over traditional approaches and demonstrate the potential for JointKPE++ to aid forensic experts in efficiently detecting keyphrases related to criminal activities.

Title: Is one brick enough to break the wall of spoken dialogue state tracking?. (arXiv:2311.04923v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.04923
• Code URL: null
• Copy Paste: [[2311.04923]] Is one brick enough to break the wall of spoken dialogue state tracking?(http://arxiv.org/abs/2311.04923)
• Summary:

  In Task-Oriented Dialogue (TOD) systems, correctly updating the system's understanding of the user's needs (a.k.a dialogue state tracking) is key to a smooth interaction. Traditionally, TOD systems perform this update in three steps: transcription of the user's utterance, semantic extraction of the key concepts, and contextualization with the previously identified concepts. Such cascade approaches suffer from cascading errors and separate optimization. End-to-End approaches have been proved helpful up to the semantic extraction step. This paper goes one step further paving the path towards completely neural spoken dialogue state tracking by comparing three approaches: (1) a state of the art cascade approach, (2) a locally E2E approach with rule-based contextualization and (3) a completely neural approach. Our study highlights that although they all outperform the recent DSTC11 best model, especially with a filtering post-processing step, (1) remains the most accurate approach. Indeed, both (2) and (3) have trouble propagating context as dialogues unfold showing that context propagation in completely neural approaches is an open challenge.

Title: Investigating Deep-Learning NLP for Automating the Extraction of Oncology Efficacy Endpoints from Scientific Literature. (arXiv:2311.04925v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.04925
• Code URL: null
• Copy Paste: [[2311.04925]] Investigating Deep-Learning NLP for Automating the Extraction of Oncology Efficacy Endpoints from Scientific Literature(http://arxiv.org/abs/2311.04925)
• Summary:

  Benchmarking drug efficacy is a critical step in clinical trial design and planning. The challenge is that much of the data on efficacy endpoints is stored in scientific papers in free text form, so extraction of such data is currently a largely manual task. Our objective is to automate this task as much as possible. In this study we have developed and optimised a framework to extract efficacy endpoints from text in scientific papers, using a machine learning approach. Our machine learning model predicts 25 classes associated with efficacy endpoints and leads to high F1 scores (harmonic mean of precision and recall) of 96.4% on the test set, and 93.9% and 93.7% on two case studies. These methods were evaluated against - and showed strong agreement with - subject matter experts and show significant promise in the future of automating the extraction of clinical endpoints from free text. Clinical information extraction from text data is currently a laborious manual task which scales poorly and is prone to human error. Demonstrating the ability to extract efficacy endpoints automatically shows great promise for accelerating clinical trial design moving forwards.

Title: Mirror: A Universal Framework for Various Information Extraction Tasks. (arXiv:2311.05419v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.05419
• Code URL: https://github.com/Spico197/Mirror
• Copy Paste: [[2311.05419]] Mirror: A Universal Framework for Various Information Extraction Tasks(http://arxiv.org/abs/2311.05419)
• Summary:

  Sharing knowledge between information extraction tasks has always been a challenge due to the diverse data formats and task variations. Meanwhile, this divergence leads to information waste and increases difficulties in building complex applications in real scenarios. Recent studies often formulate IE tasks as a triplet extraction problem. However, such a paradigm does not support multi-span and n-ary extraction, leading to weak versatility. To this end, we reorganize IE problems into unified multi-slot tuples and propose a universal framework for various IE tasks, namely Mirror. Specifically, we recast existing IE tasks as a multi-span cyclic graph extraction problem and devise a non-autoregressive graph decoding algorithm to extract all spans in a single step. It is worth noting that this graph structure is incredibly versatile, and it supports not only complex IE tasks, but also machine reading comprehension and classification tasks. We manually construct a corpus containing 57 datasets for model pretraining, and conduct experiments on 30 datasets across 8 downstream tasks. The experimental results demonstrate that our model has decent compatibility and outperforms or reaches competitive performance with SOTA systems under few-shot and zero-shot settings. The code, model weights, and pretraining corpus are available at https://github.com/Spico197/Mirror .

Title: FAMuS: Frames Across Multiple Sources. (arXiv:2311.05601v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.05601
• Code URL: null
• Copy Paste: [[2311.05601]] FAMuS: Frames Across Multiple Sources(http://arxiv.org/abs/2311.05601)
• Summary:

  Understanding event descriptions is a central aspect of language processing, but current approaches focus overwhelmingly on single sentences or documents. Aggregating information about an event \emph{across documents} can offer a much richer understanding. To this end, we present FAMuS, a new corpus of Wikipedia passages that \emph{report} on some event, paired with underlying, genre-diverse (non-Wikipedia) \emph{source} articles for the same event. Events and (cross-sentence) arguments in both report and source are annotated against FrameNet, providing broad coverage of different event types. We present results on two key event understanding tasks enabled by FAMuS: \emph{source validation} -- determining whether a document is a valid source for a target report event -- and \emph{cross-document argument extraction} -- full-document argument extraction for a target event from both its report and the correct source article. We release both FAMuS and our models to support further research.

membership infer

federate

Title: Data Valuation and Detections in Federated Learning. (arXiv:2311.05304v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.05304
• Code URL: https://github.com/muz1lee/motdata
• Copy Paste: [[2311.05304]] Data Valuation and Detections in Federated Learning(http://arxiv.org/abs/2311.05304)
• Summary:

  Federated Learning (FL) enables collaborative model training without sharing raw data, demanding abundant, high-quality data for optimal model performance. Fair and efficient data evaluation is a fundamental issue for incentivizing clients to provide more high-quality data. Meanwhile, it is likely that only a subset of clients and datasets are relevant for a learning task while the rest of them may have a negative impact on the model training. This paper introduces a novel privacy-preserving method for evaluating client contributions and selecting relevant data samples without a pre-specified training algorithm. Our proposed approach, FedBary, utilizes Wasserstein distance within the federated context, offering a new pioneering solution for data valuation, which provides transparent data evaluation and efficient computation of Wasserstein barycenter to mitigate reliance on validation data. We conduct extensive empirical experiments and theoretical analysis, showing the promising research of this valuation metric.

Title: Personalized Online Federated Learning with Multiple Kernels. (arXiv:2311.05108v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.05108
• Code URL: https://github.com/pouyamghari/pof-mkl
• Copy Paste: [[2311.05108]] Personalized Online Federated Learning with Multiple Kernels(http://arxiv.org/abs/2311.05108)
• Summary:

  Multi-kernel learning (MKL) exhibits well-documented performance in online non-linear function approximation. Federated learning enables a group of learners (called clients) to train an MKL model on the data distributed among clients to perform online non-linear function approximation. There are some challenges in online federated MKL that need to be addressed: i) Communication efficiency especially when a large number of kernels are considered ii) Heterogeneous data distribution among clients. The present paper develops an algorithmic framework to enable clients to communicate with the server to send their updates with affordable communication cost while clients employ a large dictionary of kernels. Utilizing random feature (RF) approximation, the present paper proposes scalable online federated MKL algorithm. We prove that using the proposed online federated MKL algorithm, each client enjoys sub-linear regret with respect to the RF approximation of its best kernel in hindsight, which indicates that the proposed algorithm can effectively deal with heterogeneity of the data distributed among clients. Experimental results on real datasets showcase the advantages of the proposed algorithm compared with other online federated kernel learning ones.

fair

Title: All Should Be Equal in the Eyes of Language Models: Counterfactually Aware Fair Text Generation. (arXiv:2311.05451v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.05451
• Code URL: null
• Copy Paste: [[2311.05451]] All Should Be Equal in the Eyes of Language Models: Counterfactually Aware Fair Text Generation(http://arxiv.org/abs/2311.05451)
• Summary:

  Fairness in Language Models (LMs) remains a longstanding challenge, given the inherent biases in training data that can be perpetuated by models and affect the downstream tasks. Recent methods employ expensive retraining or attempt debiasing during inference by constraining model outputs to contrast from a reference set of biased templates or exemplars. Regardless, they dont address the primary goal of fairness to maintain equitability across different demographic groups. In this work, we posit that inferencing LMs to generate unbiased output for one demographic under a context ensues from being aware of outputs for other demographics under the same context. To this end, we propose Counterfactually Aware Fair InferencE (CAFIE), a framework that dynamically compares the model understanding of diverse demographics to generate more equitable sentences. We conduct an extensive empirical evaluation using base LMs of varying sizes and across three diverse datasets and found that CAFIE outperforms strong baselines. CAFIE produces fairer text and strikes the best balance between fairness and language modeling capability

Title: Counterfactually Fair Representation. (arXiv:2311.05420v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.05420
• Code URL: https://github.com/osu-srml/cf_representation_learning
• Copy Paste: [[2311.05420]] Counterfactually Fair Representation(http://arxiv.org/abs/2311.05420)
• Summary:

  The use of machine learning models in high-stake applications (e.g., healthcare, lending, college admission) has raised growing concerns due to potential biases against protected social groups. Various fairness notions and methods have been proposed to mitigate such biases. In this work, we focus on Counterfactual Fairness (CF), a fairness notion that is dependent on an underlying causal graph and first proposed by Kusner \textit{et al.}~\cite{kusner2017counterfactual}; it requires that the outcome an individual perceives is the same in the real world as it would be in a "counterfactual" world, in which the individual belongs to another social group. Learning fair models satisfying CF can be challenging. It was shown in \cite{kusner2017counterfactual} that a sufficient condition for satisfying CF is to \textbf{not} use features that are descendants of sensitive attributes in the causal graph. This implies a simple method that learns CF models only using non-descendants of sensitive attributes while eliminating all descendants. Although several subsequent works proposed methods that use all features for training CF models, there is no theoretical guarantee that they can satisfy CF. In contrast, this work proposes a new algorithm that trains models using all the available features. We theoretically and empirically show that models trained with this method can satisfy CF\footnote{The code repository for this work can be found in \url{https://github.com/osu-srml/CF_Representation_Learning}}.

interpretability

Title: Interpretable Geoscience Artificial Intelligence (XGeoS-AI): Application to Demystify Image Recognition. (arXiv:2311.04940v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.04940
• Code URL: null
• Copy Paste: [[2311.04940]] Interpretable Geoscience Artificial Intelligence (XGeoS-AI): Application to Demystify Image Recognition(http://arxiv.org/abs/2311.04940)
• Summary:

  As Earth science enters the era of big data, artificial intelligence (AI) not only offers great potential for solving geoscience problems, but also plays a critical role in accelerating the understanding of the complex, interactive, and multiscale processes of Earth's behavior. As geoscience AI models are progressively utilized for significant predictions in crucial situations, geoscience researchers are increasingly demanding their interpretability and versatility. This study proposes an interpretable geoscience artificial intelligence (XGeoS-AI) framework to unravel the mystery of image recognition in the Earth sciences, and its effectiveness and versatility is demonstrated by taking computed tomography (CT) image recognition as an example. Inspired by the mechanism of human vision, the proposed XGeoS-AI framework generates a threshold value from a local region within the whole image to complete the recognition. Different kinds of artificial intelligence (AI) methods, such as Support Vector Regression (SVR), Multilayer Perceptron (MLP), Convolutional Neural Network (CNN), can be adopted as the AI engines of the proposed XGeoS-AI framework to efficiently complete geoscience image recognition tasks. Experimental results demonstrate that the effectiveness, versatility, and heuristics of the proposed framework have great potential in solving geoscience image recognition problems. Interpretable AI should receive more and more attention in the field of the Earth sciences, which is the key to promoting more rational and wider applications of AI in the field of Earth sciences. In addition, the proposed interpretable framework may be the forerunner of technological innovation in the Earth sciences.

Title: SCAAT: Improving Neural Network Interpretability via Saliency Constrained Adaptive Adversarial Training. (arXiv:2311.05143v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.05143
• Code URL: null
• Copy Paste: [[2311.05143]] SCAAT: Improving Neural Network Interpretability via Saliency Constrained Adaptive Adversarial Training(http://arxiv.org/abs/2311.05143)
• Summary:

  Deep Neural Networks (DNNs) are expected to provide explanation for users to understand their black-box predictions. Saliency map is a common form of explanation illustrating the heatmap of feature attributions, but it suffers from noise in distinguishing important features. In this paper, we propose a model-agnostic learning method called Saliency Constrained Adaptive Adversarial Training (SCAAT) to improve the quality of such DNN interpretability. By constructing adversarial samples under the guidance of saliency map, SCAAT effectively eliminates most noise and makes saliency maps sparser and more faithful without any modification to the model architecture. We apply SCAAT to multiple DNNs and evaluate the quality of the generated saliency maps on various natural and pathological image datasets. Evaluations on different domains and metrics show that SCAAT significantly improves the interpretability of DNNs by providing more faithful saliency maps without sacrificing their predictive power.

explainability

Title: Explained anomaly detection in text reviews: Can subjective scenarios be correctly evaluated?. (arXiv:2311.04948v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.04948
• Code URL: null
• Copy Paste: [[2311.04948]] Explained anomaly detection in text reviews: Can subjective scenarios be correctly evaluated?(http://arxiv.org/abs/2311.04948)
• Summary:

  This paper presents a pipeline to detect and explain anomalous reviews in online platforms. The pipeline is made up of three modules and allows the detection of reviews that do not generate value for users due to either worthless or malicious composition. The classifications are accompanied by a normality score and an explanation that justifies the decision made. The pipeline's ability to solve the anomaly detection task was evaluated using different datasets created from a large Amazon database. Additionally, a study comparing three explainability techniques involving 241 participants was conducted to assess the explainability module. The study aimed to measure the impact of explanations on the respondents' ability to reproduce the classification model and their perceived usefulness. This work can be useful to automate tasks in review online platforms, such as those for electronic commerce, and offers inspiration for addressing similar problems in the field of anomaly detection in textual data. We also consider it interesting to have carried out a human evaluation of the capacity of different explainability techniques in a real and infrequent scenario such as the detection of anomalous reviews, as well as to reflect on whether it is possible to explain tasks as humanly subjective as this one.

watermark

diffusion

Title: Improved DDIM Sampling with Moment Matching Gaussian Mixtures. (arXiv:2311.04938v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.04938
• Code URL: null
• Copy Paste: [[2311.04938]] Improved DDIM Sampling with Moment Matching Gaussian Mixtures(http://arxiv.org/abs/2311.04938)
• Summary:

  We propose using a Gaussian Mixture Model (GMM) as reverse transition operator (kernel) within the Denoising Diffusion Implicit Models (DDIM) framework, which is one of the most widely used approaches for accelerated sampling from pre-trained Denoising Diffusion Probabilistic Models (DDPM). Specifically we match the first and second order central moments of the DDPM forward marginals by constraining the parameters of the GMM. We see that moment matching is sufficient to obtain samples with equal or better quality than the original DDIM with Gaussian kernels. We provide experimental results with unconditional models trained on CelebAHQ and FFHQ and class-conditional models trained on ImageNet datasets respectively. Our results suggest that using the GMM kernel leads to significant improvements in the quality of the generated samples when the number of sampling steps is small, as measured by FID and IS metrics. For example on ImageNet 256x256, using 10 sampling steps, we achieve a FID of 6.94 and IS of 207.85 with a GMM kernel compared to 10.15 and 196.73 respectively with a Gaussian kernel.

Title: Lightweight Diffusion Models with Distillation-Based Block Neural Architecture Search. (arXiv:2311.04950v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.04950
• Code URL: null
• Copy Paste: [[2311.04950]] Lightweight Diffusion Models with Distillation-Based Block Neural Architecture Search(http://arxiv.org/abs/2311.04950)
• Summary:

  Diffusion models have recently shown remarkable generation ability, achieving state-of-the-art performance in many tasks. However, the high computational cost is still a troubling problem for diffusion models. To tackle this problem, we propose to automatically remove the structural redundancy in diffusion models with our proposed Diffusion Distillation-based Block-wise Neural Architecture Search (DiffNAS). Specifically, given a larger pretrained teacher, we leverage DiffNAS to search for the smallest architecture which achieves on-par or even better performance than the teacher. Considering current diffusion models are based on UNet which naturally has a block-wise structure, we perform neural architecture search independently in each block, which largely reduces the search space. Different from previous block-wise NAS methods, DiffNAS contains a block-wise local search strategy and a retraining strategy with a joint dynamic loss. Concretely, during the search process, we block-wisely select the best subnet to avoid the unfairness brought by the global search strategy used in previous works. When retraining the searched architecture, we adopt a dynamic joint loss to maintain the consistency between supernet training and subnet retraining, which also provides informative objectives for each block and shortens the paths of gradient propagation. We demonstrate this joint loss can effectively improve model performance. We also prove the necessity of the dynamic adjustment of this loss. The experiments show that our method can achieve significant computational reduction, especially on latent diffusion models with about 50% MACs and Parameter reduction.

Title: BrainNetDiff: Generative AI Empowers Brain Network Generation via Multimodal Diffusion Model. (arXiv:2311.05199v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.05199
• Code URL: null
• Copy Paste: [[2311.05199]] BrainNetDiff: Generative AI Empowers Brain Network Generation via Multimodal Diffusion Model(http://arxiv.org/abs/2311.05199)
• Summary:

  Brain network analysis has emerged as pivotal method for gaining a deeper understanding of brain functions and disease mechanisms. Despite the existence of various network construction approaches, shortcomings persist in the learning of correlations between structural and functional brain imaging data. In light of this, we introduce a novel method called BrainNetDiff, which combines a multi-head Transformer encoder to extract relevant features from fMRI time series and integrates a conditional latent diffusion model for brain network generation. Leveraging a conditional prompt and a fusion attention mechanism, this method significantly improves the accuracy and stability of brain network generation. To the best of our knowledge, this represents the first framework that employs diffusion for the fusion of the multimodal brain imaging and brain network generation from images to graphs. We validate applicability of this framework in the construction of brain network across healthy and neurologically impaired cohorts using the authentic dataset. Experimental results vividly demonstrate the significant effectiveness of the proposed method across the downstream disease classification tasks. These findings convincingly emphasize the prospective value in the field of brain network research, particularly its key significance in neuroimaging analysis and disease diagnosis. This research provides a valuable reference for the processing of multimodal brain imaging data and introduces a novel, efficient solution to the field of neuroimaging.

Title: ConRad: Image Constrained Radiance Fields for 3D Generation from a Single Image. (arXiv:2311.05230v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.05230
• Code URL: null
• Copy Paste: [[2311.05230]] ConRad: Image Constrained Radiance Fields for 3D Generation from a Single Image(http://arxiv.org/abs/2311.05230)
• Summary:

  We present a novel method for reconstructing 3D objects from a single RGB image. Our method leverages the latest image generation models to infer the hidden 3D structure while remaining faithful to the input image. While existing methods obtain impressive results in generating 3D models from text prompts, they do not provide an easy approach for conditioning on input RGB data. Na\"ive extensions of these methods often lead to improper alignment in appearance between the input image and the 3D reconstructions. We address these challenges by introducing Image Constrained Radiance Fields (ConRad), a novel variant of neural radiance fields. ConRad is an efficient 3D representation that explicitly captures the appearance of an input image in one viewpoint. We propose a training algorithm that leverages the single RGB image in conjunction with pretrained Diffusion Models to optimize the parameters of a ConRad representation. Extensive experiments show that ConRad representations can simplify preservation of image details while producing a realistic 3D reconstruction. Compared to existing state-of-the-art baselines, we show that our 3D reconstructions remain more faithful to the input and produce more consistent 3D models while demonstrating significantly improved quantitative performance on a ShapeNet object benchmark.

Title: Control3D: Towards Controllable Text-to-3D Generation. (arXiv:2311.05461v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.05461
• Code URL: null
• Copy Paste: [[2311.05461]] Control3D: Towards Controllable Text-to-3D Generation(http://arxiv.org/abs/2311.05461)
• Summary:

  Recent remarkable advances in large-scale text-to-image diffusion models have inspired a significant breakthrough in text-to-3D generation, pursuing 3D content creation solely from a given text prompt. However, existing text-to-3D techniques lack a crucial ability in the creative process: interactively control and shape the synthetic 3D contents according to users' desired specifications (e.g., sketch). To alleviate this issue, we present the first attempt for text-to-3D generation conditioning on the additional hand-drawn sketch, namely Control3D, which enhances controllability for users. In particular, a 2D conditioned diffusion model (ControlNet) is remoulded to guide the learning of 3D scene parameterized as NeRF, encouraging each view of 3D scene aligned with the given text prompt and hand-drawn sketch. Moreover, we exploit a pre-trained differentiable photo-to-sketch model to directly estimate the sketch of the rendered image over synthetic 3D scene. Such estimated sketch along with each sampled view is further enforced to be geometrically consistent with the given sketch, pursuing better controllable text-to-3D generation. Through extensive experiments, we demonstrate that our proposal can generate accurate and faithful 3D scenes that align closely with the input text prompts and sketches.

Title: ControlStyle: Text-Driven Stylized Image Generation Using Diffusion Priors. (arXiv:2311.05463v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.05463
• Code URL: null
• Copy Paste: [[2311.05463]] ControlStyle: Text-Driven Stylized Image Generation Using Diffusion Priors(http://arxiv.org/abs/2311.05463)
• Summary:

  Recently, the multimedia community has witnessed the rise of diffusion models trained on large-scale multi-modal data for visual content creation, particularly in the field of text-to-image generation. In this paper, we propose a new task for ``stylizing'' text-to-image models, namely text-driven stylized image generation, that further enhances editability in content creation. Given input text prompt and style image, this task aims to produce stylized images which are both semantically relevant to input text prompt and meanwhile aligned with the style image in style. To achieve this, we present a new diffusion model (ControlStyle) via upgrading a pre-trained text-to-image model with a trainable modulation network enabling more conditions of text prompts and style images. Moreover, diffusion style and content regularizations are simultaneously introduced to facilitate the learning of this modulation network with these diffusion priors, pursuing high-quality stylized text-to-image generation. Extensive experiments demonstrate the effectiveness of our ControlStyle in producing more visually pleasing and artistic results, surpassing a simple combination of text-to-image model and conventional style transfer techniques.

Title: 3DStyle-Diffusion: Pursuing Fine-grained Text-driven 3D Stylization with 2D Diffusion Models. (arXiv:2311.05464v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.05464
• Code URL: https://github.com/yanghb22-fdu/3dstyle-diffusion-official
• Copy Paste: [[2311.05464]] 3DStyle-Diffusion: Pursuing Fine-grained Text-driven 3D Stylization with 2D Diffusion Models(http://arxiv.org/abs/2311.05464)
• Summary:

  3D content creation via text-driven stylization has played a fundamental challenge to multimedia and graphics community. Recent advances of cross-modal foundation models (e.g., CLIP) have made this problem feasible. Those approaches commonly leverage CLIP to align the holistic semantics of stylized mesh with the given text prompt. Nevertheless, it is not trivial to enable more controllable stylization of fine-grained details in 3D meshes solely based on such semantic-level cross-modal supervision. In this work, we propose a new 3DStyle-Diffusion model that triggers fine-grained stylization of 3D meshes with additional controllable appearance and geometric guidance from 2D Diffusion models. Technically, 3DStyle-Diffusion first parameterizes the texture of 3D mesh into reflectance properties and scene lighting using implicit MLP networks. Meanwhile, an accurate depth map of each sampled view is achieved conditioned on 3D mesh. Then, 3DStyle-Diffusion leverages a pre-trained controllable 2D Diffusion model to guide the learning of rendered images, encouraging the synthesized image of each view semantically aligned with text prompt and geometrically consistent with depth map. This way elegantly integrates both image rendering via implicit MLP networks and diffusion process of image synthesis in an end-to-end fashion, enabling a high-quality fine-grained stylization of 3D meshes. We also build a new dataset derived from Objaverse and the evaluation protocol for this task. Through both qualitative and quantitative experiments, we validate the capability of our 3DStyle-Diffusion. Source code and data are available at \url{https://github.com/yanghb22-fdu/3DStyle-Diffusion-Official}.

Title: LCM-LoRA: A Universal Stable-Diffusion Acceleration Module. (arXiv:2311.05556v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.05556
• Code URL: https://github.com/luosiallen/latent-consistency-model
• Copy Paste: [[2311.05556]] LCM-LoRA: A Universal Stable-Diffusion Acceleration Module(http://arxiv.org/abs/2311.05556)
• Summary:

  Latent Consistency Models (LCMs) have achieved impressive performance in accelerating text-to-image generative tasks, producing high-quality images with minimal inference steps. LCMs are distilled from pre-trained latent diffusion models (LDMs), requiring only ~32 A100 GPU training hours. This report further extends LCMs' potential in two aspects: First, by applying LoRA distillation to Stable-Diffusion models including SD-V1.5, SSD-1B, and SDXL, we have expanded LCM's scope to larger models with significantly less memory consumption, achieving superior image generation quality. Second, we identify the LoRA parameters obtained through LCM distillation as a universal Stable-Diffusion acceleration module, named LCM-LoRA. LCM-LoRA can be directly plugged into various Stable-Diffusion fine-tuned models or LoRAs without training, thus representing a universally applicable accelerator for diverse image generation tasks. Compared with previous numerical PF-ODE solvers such as DDIM, DPM-Solver, LCM-LoRA can be viewed as a plug-in neural PF-ODE solver that possesses strong generalization abilities. Project page: https://github.com/luosiallen/latent-consistency-model.

Title: Predicting the Position Uncertainty at the Time of Closest Approach with Diffusion Models. (arXiv:2311.05417v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.05417
• Code URL: null
• Copy Paste: [[2311.05417]] Predicting the Position Uncertainty at the Time of Closest Approach with Diffusion Models(http://arxiv.org/abs/2311.05417)
• Summary:

  The risk of collision between resident space objects has significantly increased in recent years. As a result, spacecraft collision avoidance procedures have become an essential part of satellite operations. To ensure safe and effective space activities, satellite owners and operators rely on constantly updated estimates of encounters. These estimates include the uncertainty associated with the position of each object at the expected TCA. These estimates are crucial in planning risk mitigation measures, such as collision avoidance manoeuvres. As the TCA approaches, the accuracy of these estimates improves, as both objects' orbit determination and propagation procedures are made for increasingly shorter time intervals. However, this improvement comes at the cost of taking place close to the critical decision moment. This means that safe avoidance manoeuvres might not be possible or could incur significant costs. Therefore, knowing the evolution of this variable in advance can be crucial for operators. This work proposes a machine learning model based on diffusion models to forecast the position uncertainty of objects involved in a close encounter, particularly for the secondary object (usually debris), which tends to be more unpredictable. We compare the performance of our model with other state-of-the-art solutions and a na\"ive baseline approach, showing that the proposed solution has the potential to significantly improve the safety and effectiveness of spacecraft operations.

Title: Diffusion Based Causal Representation Learning. (arXiv:2311.05421v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.05421
• Code URL: null
• Copy Paste: [[2311.05421]] Diffusion Based Causal Representation Learning(http://arxiv.org/abs/2311.05421)
• Summary:

  Causal reasoning can be considered a cornerstone of intelligent systems. Having access to an underlying causal graph comes with the promise of cause-effect estimation and the identification of efficient and safe interventions. However, learning causal representations remains a major challenge, due to the complexity of many real-world systems. Previous works on causal representation learning have mostly focused on Variational Auto-Encoders (VAE). These methods only provide representations from a point estimate, and they are unsuitable to handle high dimensions. To overcome these problems, we proposed a new Diffusion-based Causal Representation Learning (DCRL) algorithm. This algorithm uses diffusion-based representations for causal discovery. DCRL offers access to infinite dimensional latent codes, which encode different levels of information in the latent code. In a first proof of principle, we investigate the use of DCRL for causal representation learning. We further demonstrate experimentally that this approach performs comparably well in identifying the causal structure and causal variables.

Title: Bayesian Methods for Media Mix Modelling with shape and funnel effects. (arXiv:2311.05587v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.05587
• Code URL: null
• Copy Paste: [[2311.05587]] Bayesian Methods for Media Mix Modelling with shape and funnel effects(http://arxiv.org/abs/2311.05587)
• Summary:

  In recent years, significant progress in generative AI has highlighted the important role of physics-inspired models that utilize advanced mathematical concepts based on fundamental physics principles to enhance artificial intelligence capabilities. Among these models, those based on diffusion equations have greatly improved image quality. This study aims to explore the potential uses of Maxwell-Boltzmann equation, which forms the basis of the kinetic theory of gases, and the Michaelis-Menten model in Marketing Mix Modelling (MMM) applications. We propose incorporating these equations into Hierarchical Bayesian models to analyse consumer behaviour in the context of advertising. These equation sets excel in accurately describing the random dynamics in complex systems like social interactions and consumer-advertising interactions.

Title: Diffusion-Generative Multi-Fidelity Learning for Physical Simulation. (arXiv:2311.05606v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.05606
• Code URL: null
• Copy Paste: [[2311.05606]] Diffusion-Generative Multi-Fidelity Learning for Physical Simulation(http://arxiv.org/abs/2311.05606)
• Summary:

  Multi-fidelity surrogate learning is important for physical simulation related applications in that it avoids running numerical solvers from scratch, which is known to be costly, and it uses multi-fidelity examples for training and greatly reduces the cost of data collection. Despite the variety of existing methods, they all build a model to map the input parameters outright to the solution output. Inspired by the recent breakthrough in generative models, we take an alternative view and consider the solution output as generated from random noises. We develop a diffusion-generative multi-fidelity (DGMF) learning method based on stochastic differential equations (SDE), where the generation is a continuous denoising process. We propose a conditional score model to control the solution generation by the input parameters and the fidelity. By conditioning on additional inputs (temporal or spacial variables), our model can efficiently learn and predict multi-dimensional solution arrays. Our method naturally unifies discrete and continuous fidelity modeling. The advantage of our method in several typical applications shows a promising new direction for multi-fidelity learning.

noise learning

data-free

transformer

Title: Zero-shot Translation of Attention Patterns in VQA Models to Natural Language. (arXiv:2311.05043v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.05043
• Code URL: https://github.com/explainableml/zs-a2t
• Copy Paste: [[2311.05043]] Zero-shot Translation of Attention Patterns in VQA Models to Natural Language(http://arxiv.org/abs/2311.05043)
• Summary:

  Converting a model's internals to text can yield human-understandable insights about the model. Inspired by the recent success of training-free approaches for image captioning, we propose ZS-A2T, a zero-shot framework that translates the transformer attention of a given model into natural language without requiring any training. We consider this in the context of Visual Question Answering (VQA). ZS-A2T builds on a pre-trained large language model (LLM), which receives a task prompt, question, and predicted answer, as inputs. The LLM is guided to select tokens which describe the regions in the input image that the VQA model attended to. Crucially, we determine this similarity by exploiting the text-image matching capabilities of the underlying VQA model. Our framework does not require any training and allows the drop-in replacement of different guiding sources (e.g. attribution instead of attention maps), or language models. We evaluate this novel task on textual explanation datasets for VQA, giving state-of-the-art performances for the zero-shot setting on GQA-REX and VQA-X. Our code is available at: https://github.com/ExplainableML/ZS-A2T.

Title: Dynamic Association Learning of Self-Attention and Convolution in Image Restoration. (arXiv:2311.05147v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.05147
• Code URL: null
• Copy Paste: [[2311.05147]] Dynamic Association Learning of Self-Attention and Convolution in Image Restoration(http://arxiv.org/abs/2311.05147)
• Summary:

  CNNs and Self attention have achieved great success in multimedia applications for dynamic association learning of self-attention and convolution in image restoration. However, CNNs have at least two shortcomings: 1) limited receptive field; 2) static weight of sliding window at inference, unable to cope with the content diversity.In view of the advantages and disadvantages of CNNs and Self attention, this paper proposes an association learning method to utilize the advantages and suppress their shortcomings, so as to achieve high-quality and efficient inpainting. We regard rain distribution reflects the degradation location and degree, in addition to the rain distribution prediction. Thus, we propose to refine background textures with the predicted degradation prior in an association learning manner. As a result, we accomplish image deraining by associating rain streak removal and background recovery, where an image deraining network and a background recovery network are designed for two subtasks. The key part of association learning is a novel multi-input attention module. It generates the degradation prior and produces the degradation mask according to the predicted rainy distribution. Benefited from the global correlation calculation of SA, MAM can extract the informative complementary components from the rainy input with the degradation mask, and then help accurate texture restoration. Meanwhile, SA tends to aggregate feature maps with self-attention importance, but convolution diversifies them to focus on the local textures. A hybrid fusion network involves one residual Transformer branch and one encoder-decoder branch. The former takes a few learnable tokens as input and stacks multi-head attention and feed-forward networks to encode global features of the image. The latter, conversely, leverages the multi-scale encoder-decoder to represent contexture knowledge.

Title: TransReg: Cross-transformer as auto-registration module for multi-view mammogram mass detection. (arXiv:2311.05192v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.05192
• Code URL: null
• Copy Paste: [[2311.05192]] TransReg: Cross-transformer as auto-registration module for multi-view mammogram mass detection(http://arxiv.org/abs/2311.05192)
• Summary:

  Screening mammography is the most widely used method for early breast cancer detection, significantly reducing mortality rates. The integration of information from multi-view mammograms enhances radiologists' confidence and diminishes false-positive rates since they can examine on dual-view of the same breast to cross-reference the existence and location of the lesion. Inspired by this, we present TransReg, a Computer-Aided Detection (CAD) system designed to exploit the relationship between craniocaudal (CC), and mediolateral oblique (MLO) views. The system includes cross-transformer to model the relationship between the region of interest (RoIs) extracted by siamese Faster RCNN network for mass detection problems. Our work is the first time cross-transformer has been integrated into an object detection framework to model the relation between ipsilateral views. Our experimental evaluation on DDSM and VinDr-Mammo datasets shows that our TransReg, equipped with SwinT as a feature extractor achieves state-of-the-art performance. Specifically, at the false positive rate per image at 0.5, TransReg using SwinT gets a recall at 83.3% for DDSM dataset and 79.7% for VinDr-Mammo dataset. Furthermore, we conduct a comprehensive analysis to demonstrate that cross-transformer can function as an auto-registration module, aligning the masses in dual-view and utilizing this information to inform final predictions. It is a replication diagnostic workflow of expert radiologists

Title: Improving Hand Recognition in Uncontrolled and Uncooperative Environments using Multiple Spatial Transformers and Loss Functions. (arXiv:2311.05383v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.05383
• Code URL: null
• Copy Paste: [[2311.05383]] Improving Hand Recognition in Uncontrolled and Uncooperative Environments using Multiple Spatial Transformers and Loss Functions(http://arxiv.org/abs/2311.05383)
• Summary:

  The prevalence of smartphone and consumer camera has led to more evidence in the form of digital images, which are mostly taken in uncontrolled and uncooperative environments. In these images, criminals likely hide or cover their faces while their hands are observable in some cases, creating a challenging use case for forensic investigation. Many existing hand-based recognition methods perform well for hand images collected in controlled environments with user cooperation. However, their performance deteriorates significantly in uncontrolled and uncooperative environments. A recent work has exposed the potential of hand recognition in these environments. However, only the palmar regions were considered, and the recognition performance is still far from satisfactory. To improve the recognition accuracy, an algorithm integrating a multi-spatial transformer network (MSTN) and multiple loss functions is proposed to fully utilize information in full hand images. MSTN is firstly employed to localize the palms and fingers and estimate the alignment parameters. Then, the aligned images are further fed into pretrained convolutional neural networks, where features are extracted. Finally, a training scheme with multiple loss functions is used to train the network end-to-end. To demonstrate the effectiveness of the proposed algorithm, the trained model is evaluated on NTU-PI-v1 database and six benchmark databases from different domains. Experimental results show that the proposed algorithm performs significantly better than the existing methods in these uncontrolled and uncooperative environments and has good generalization capabilities to samples from different domains.

Title: High-Performance Transformers for Table Structure Recognition Need Early Convolutions. (arXiv:2311.05565v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.05565
• Code URL: https://github.com/poloclub/tsr-convstem
• Copy Paste: [[2311.05565]] High-Performance Transformers for Table Structure Recognition Need Early Convolutions(http://arxiv.org/abs/2311.05565)
• Summary:

  Table structure recognition (TSR) aims to convert tabular images into a machine-readable format, where a visual encoder extracts image features and a textual decoder generates table-representing tokens. Existing approaches use classic convolutional neural network (CNN) backbones for the visual encoder and transformers for the textual decoder. However, this hybrid CNN-Transformer architecture introduces a complex visual encoder that accounts for nearly half of the total model parameters, markedly reduces both training and inference speed, and hinders the potential for self-supervised learning in TSR. In this work, we design a lightweight visual encoder for TSR without sacrificing expressive power. We discover that a convolutional stem can match classic CNN backbone performance, with a much simpler model. The convolutional stem strikes an optimal balance between two crucial factors for high-performance TSR: a higher receptive field (RF) ratio and a longer sequence length. This allows it to "see" an appropriate portion of the table and "store" the complex table structure within sufficient context length for the subsequent transformer. We conducted reproducible ablation studies and open-sourced our code at https://github.com/poloclub/tsr-convstem to enhance transparency, inspire innovations, and facilitate fair comparisons in our domain as tables are a promising modality for representation learning.

Title: Accuracy of a Vision-Language Model on Challenging Medical Cases. (arXiv:2311.05591v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.05591
• Code URL: https://github.com/2v/gpt4v-image-challenge
• Copy Paste: [[2311.05591]] Accuracy of a Vision-Language Model on Challenging Medical Cases(http://arxiv.org/abs/2311.05591)
• Summary:

  Background: General-purpose large language models that utilize both text and images have not been evaluated on a diverse array of challenging medical cases.

  Methods: Using 934 cases from the NEJM Image Challenge published between 2005 and 2023, we evaluated the accuracy of the recently released Generative Pre-trained Transformer 4 with Vision model (GPT-4V) compared to human respondents overall and stratified by question difficulty, image type, and skin tone. We further conducted a physician evaluation of GPT-4V on 69 NEJM clinicopathological conferences (CPCs). Analyses were conducted for models utilizing text alone, images alone, and both text and images.

  Results: GPT-4V achieved an overall accuracy of 61% (95% CI, 58 to 64%) compared to 49% (95% CI, 49 to 50%) for humans. GPT-4V outperformed humans at all levels of difficulty and disagreement, skin tones, and image types; the exception was radiographic images, where performance was equivalent between GPT-4V and human respondents. Longer, more informative captions were associated with improved performance for GPT-4V but similar performance for human respondents. GPT-4V included the correct diagnosis in its differential for 80% (95% CI, 68 to 88%) of CPCs when using text alone, compared to 58% (95% CI, 45 to 70%) of CPCs when using both images and text.

  Conclusions: GPT-4V outperformed human respondents on challenging medical cases and was able to synthesize information from both images and text, but performance deteriorated when images were added to highly informative text. Overall, our results suggest that multimodal AI models may be useful in medical diagnostic reasoning but that their accuracy may depend heavily on context.

Title: Window Attention is Bugged: How not to Interpolate Position Embeddings. (arXiv:2311.05613v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.05613
• Code URL: null
• Copy Paste: [[2311.05613]] Window Attention is Bugged: How not to Interpolate Position Embeddings(http://arxiv.org/abs/2311.05613)
• Summary:

  Window attention, position embeddings, and high resolution finetuning are core concepts in the modern transformer era of computer vision. However, we find that naively combining these near ubiquitous components can have a detrimental effect on performance. The issue is simple: interpolating position embeddings while using window attention is wrong. We study two state-of-the-art methods that have these three components, namely Hiera and ViTDet, and find that both do indeed suffer from this bug. To fix it, we introduce a simple absolute window position embedding strategy, which solves the bug outright in Hiera and allows us to increase both speed and performance of the model in ViTDet. We finally combine the two to obtain HieraDet, which achieves 61.7 box mAP on COCO, making it state-of-the-art for models that only use ImageNet-1k pretraining. This all stems from what is essentially a 3 line bug fix, which we name "absolute win".

Title: An Improved Transformer-based Model for Detecting Phishing, Spam, and Ham: A Large Language Model Approach. (arXiv:2311.04913v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.04913
• Code URL: null
• Copy Paste: [[2311.04913]] An Improved Transformer-based Model for Detecting Phishing, Spam, and Ham: A Large Language Model Approach(http://arxiv.org/abs/2311.04913)
• Summary:

  Phishing and spam detection is long standing challenge that has been the subject of much academic research. Large Language Models (LLM) have vast potential to transform society and provide new and innovative approaches to solve well-established challenges. Phishing and spam have caused financial hardships and lost time and resources to email users all over the world and frequently serve as an entry point for ransomware threat actors. While detection approaches exist, especially heuristic-based approaches, LLMs offer the potential to venture into a new unexplored area for understanding and solving this challenge. LLMs have rapidly altered the landscape from business, consumers, and throughout academia and demonstrate transformational potential for the potential of society. Based on this, applying these new and innovative approaches to email detection is a rational next step in academic research. In this work, we present IPSDM, our model based on fine-tuning the BERT family of models to specifically detect phishing and spam email. We demonstrate our fine-tuned version, IPSDM, is able to better classify emails in both unbalanced and balanced datasets. This work serves as an important first step towards employing LLMs to improve the security of our information systems.

Title: Tuning-less Object Naming with a Foundation Model. (arXiv:2311.04924v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.04924
• Code URL: null
• Copy Paste: [[2311.04924]] Tuning-less Object Naming with a Foundation Model(http://arxiv.org/abs/2311.04924)
• Summary:

  We implement a real-time object naming system that enables learning a set of named entities never seen. Our approach employs an existing foundation model that we consider ready to see anything before starting. It turns seen images into relatively small feature vectors that we associate with index to a gradually built vocabulary without any training of fine-tuning of the model. Our contribution is using the association mechanism known from transformers as attention. It has features that support generalization from irrelevant information for distinguishing the entities and potentially enable associating with much more than indices to vocabulary. As a result, the system can work in a one-shot manner and correctly name objects named in different contents. We also outline implementation details of the system modules integrated by a blackboard architecture. Finally, we investigate the system's quality, mainly how many objects it can handle in this way.

Title: Deep Learning Brasil at ABSAPT 2022: Portuguese Transformer Ensemble Approaches. (arXiv:2311.05051v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.05051
• Code URL: https://github.com/ju-resplande/dlb_absapt2022
• Copy Paste: [[2311.05051]] Deep Learning Brasil at ABSAPT 2022: Portuguese Transformer Ensemble Approaches(http://arxiv.org/abs/2311.05051)
• Summary:

  Aspect-based Sentiment Analysis (ABSA) is a task whose objective is to classify the individual sentiment polarity of all entities, called aspects, in a sentence. The task is composed of two subtasks: Aspect Term Extraction (ATE), identify all aspect terms in a sentence; and Sentiment Orientation Extraction (SOE), given a sentence and its aspect terms, the task is to determine the sentiment polarity of each aspect term (positive, negative or neutral). This article presents we present our participation in Aspect-Based Sentiment Analysis in Portuguese (ABSAPT) 2022 at IberLEF 2022. We submitted the best performing systems, achieving new state-of-the-art results on both subtasks.

Title: Legal-HNet: Mixing Legal Long-Context Tokens with Hartley Transform. (arXiv:2311.05089v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.05089
• Code URL: null
• Copy Paste: [[2311.05089]] Legal-HNet: Mixing Legal Long-Context Tokens with Hartley Transform(http://arxiv.org/abs/2311.05089)
• Summary:

  Since its introduction, the transformers architecture has seen great adoption in NLP applications, but it also has limitations. Although the self-attention mechanism allows for generating very rich representations of the input text, its effectiveness may be limited in specialized domains such as legal, where, for example, language models often have to process very long texts. In this paper, we explore alternatives to replace the attention-based layers with simpler token-mixing mechanisms: Hartley and Fourier transforms. Using these non-parametric techniques, we train models with long input documents from scratch in the legal domain setting. We also introduce a new hybrid Seq2Seq architecture, a no-attention-based encoder connected with an attention-based decoder, which performs quite well on existing summarization tasks with much less compute and memory requirements. We believe that similar, if not better performance, as in the case of long correlations of abstractive text summarization tasks, can be achieved by adopting these simpler infrastructures. This not only makes training models from scratch accessible to more people, but also contributes to the reduction of the carbon footprint during training.

Title: GeoFormer: Predicting Human Mobility using Generative Pre-trained Transformer (GPT). (arXiv:2311.05092v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.05092
• Code URL: null
• Copy Paste: [[2311.05092]] GeoFormer: Predicting Human Mobility using Generative Pre-trained Transformer (GPT)(http://arxiv.org/abs/2311.05092)
• Summary:

  Predicting human mobility holds significant practical value, with applications ranging from enhancing disaster risk planning to simulating epidemic spread. In this paper, we present the GeoFormer, a decoder-only transformer model adapted from the GPT architecture to forecast human mobility. Our proposed model is rigorously tested in the context of the HuMob Challenge 2023 -- a competition designed to evaluate the performance of prediction models on standardized datasets to predict human mobility. The challenge leverages two datasets encompassing urban-scale data of 25,000 and 100,000 individuals over a longitudinal period of 75 days. GeoFormer stands out as a top performer in the competition, securing a place in the top-3 ranking. Its success is underscored by performing well on both performance metrics chosen for the competition -- the GEO-BLEU and the Dynamic Time Warping (DTW) measures. The performance of the GeoFormer on the HuMob Challenge 2023 underscores its potential to make substantial contributions to the field of human mobility prediction, with far-reaching implications for disaster preparedness, epidemic control, and beyond.

generative

Title: L-WaveBlock: A Novel Feature Extractor Leveraging Wavelets for Generative Adversarial Networks. (arXiv:2311.05548v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.05548
• Code URL: null
• Copy Paste: [[2311.05548]] L-WaveBlock: A Novel Feature Extractor Leveraging Wavelets for Generative Adversarial Networks(http://arxiv.org/abs/2311.05548)
• Summary:

  Generative Adversarial Networks (GANs) have risen to prominence in the field of deep learning, facilitating the generation of realistic data from random noise. The effectiveness of GANs often depends on the quality of feature extraction, a critical aspect of their architecture. This paper introduces L-WaveBlock, a novel and robust feature extractor that leverages the capabilities of the Discrete Wavelet Transform (DWT) with deep learning methodologies. L-WaveBlock is catered to quicken the convergence of GAN generators while simultaneously enhancing their performance. The paper demonstrates the remarkable utility of L-WaveBlock across three datasets, a road satellite imagery dataset, the CelebA dataset and the GoPro dataset, showcasing its ability to ease feature extraction and make it more efficient. By utilizing DWT, L-WaveBlock efficiently captures the intricate details of both structural and textural details, and further partitions feature maps into orthogonal subbands across multiple scales while preserving essential information at the same time. Not only does it lead to faster convergence, but also gives competent results on every dataset by employing the L-WaveBlock. The proposed method achieves an Inception Score of 3.6959 and a Structural Similarity Index of 0.4261 on the maps dataset, a Peak Signal-to-Noise Ratio of 29.05 and a Structural Similarity Index of 0.874 on the CelebA dataset. The proposed method performs competently to the state-of-the-art for the image denoising dataset, albeit not better, but still leads to faster convergence than conventional methods. With this, L-WaveBlock emerges as a robust and efficient tool for enhancing GAN-based image generation, demonstrating superior convergence speed and competitive performance across multiple datasets for image resolution, image generation and image denoising.

Title: Are cascade dialogue state tracking models speaking out of turn in spoken dialogues?. (arXiv:2311.04922v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.04922
• Code URL: null
• Copy Paste: [[2311.04922]] Are cascade dialogue state tracking models speaking out of turn in spoken dialogues?(http://arxiv.org/abs/2311.04922)
• Summary:

  In Task-Oriented Dialogue (TOD) systems, correctly updating the system's understanding of the user's needs is key to a smooth interaction. Traditionally TOD systems are composed of several modules that interact with one another. While each of these components is the focus of active research communities, their behavior in interaction can be overlooked. This paper proposes a comprehensive analysis of the errors of state of the art systems in complex settings such as Dialogue State Tracking which highly depends on the dialogue context. Based on spoken MultiWoz, we identify that errors on non-categorical slots' values are essential to address in order to bridge the gap between spoken and chat-based dialogue systems. We explore potential solutions to improve transcriptions and help dialogue state tracking generative models correct such errors.

Title: More Robots are Coming: Large Multimodal Models (ChatGPT) can Solve Visually Diverse Images of Parsons Problems. (arXiv:2311.04926v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.04926
• Code URL: null
• Copy Paste: [[2311.04926]] More Robots are Coming: Large Multimodal Models (ChatGPT) can Solve Visually Diverse Images of Parsons Problems(http://arxiv.org/abs/2311.04926)
• Summary:

  The advent of large language models is reshaping computing education. Recent research has demonstrated that these models can produce better explanations than students, answer multiple-choice questions at or above the class average, and generate code that can pass automated tests in introductory courses. These capabilities have prompted instructors to rapidly adapt their courses and assessment methods to accommodate changes in learning objectives and the potential for academic integrity violations. While some scholars have advocated for the integration of visual problems as a safeguard against the capabilities of language models, new multimodal language models now have vision and language capabilities that may allow them to analyze and solve visual problems. In this paper, we evaluate the performance of two large multimodal models on visual assignments, with a specific focus on Parsons problems presented across diverse visual representations. Our results show that GPT-4V solved 96.7\% of these visual problems, struggling minimally with a single Parsons problem. Conversely, Bard performed poorly by only solving 69.2\% of problems, struggling with common issues like hallucinations and refusals. These findings suggest that merely transitioning to visual programming problems might not be a panacea to issues of academic integrity in the generative AI era.

Title: Cognitively Inspired Components for Social Conversational Agents. (arXiv:2311.05450v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.05450
• Code URL: null
• Copy Paste: [[2311.05450]] Cognitively Inspired Components for Social Conversational Agents(http://arxiv.org/abs/2311.05450)
• Summary:

  Current conversational agents (CA) have seen improvement in conversational quality in recent years due to the influence of large language models (LLMs) like GPT3. However, two key categories of problem remain. Firstly there are the unique technical problems resulting from the approach taken in creating the CA, such as scope with retrieval agents and the often nonsensical answers of former generative agents. Secondly, humans perceive CAs as social actors, and as a result expect the CA to adhere to social convention. Failure on the part of the CA in this respect can lead to a poor interaction and even the perception of threat by the user. As such, this paper presents a survey highlighting a potential solution to both categories of problem through the introduction of cognitively inspired additions to the CA. Through computational facsimiles of semantic and episodic memory, emotion, working memory, and the ability to learn, it is possible to address both the technical and social problems encountered by CAs.

Title: Leveraging Speculative Sampling and KV-Cache Optimizations Together for Generative AI using OpenVINO. (arXiv:2311.04951v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.04951
• Code URL: https://github.com/openvinotoolkit/openvino_notebooks
• Copy Paste: [[2311.04951]] Leveraging Speculative Sampling and KV-Cache Optimizations Together for Generative AI using OpenVINO(http://arxiv.org/abs/2311.04951)
• Summary:

  Inference optimizations are critical for improving user experience and reducing infrastructure costs and power consumption. In this article, we illustrate a form of dynamic execution known as speculative sampling to reduce the overall latency of text generation and compare it with standard autoregressive sampling. This can be used together with model-based optimizations (e.g. quantization) to provide an optimized solution. Both sampling methods make use of KV caching. A Jupyter notebook and some sample executions are provided.

Title: Quantum Generative Modeling of Sequential Data with Trainable Token Embedding. (arXiv:2311.05050v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.05050
• Code URL: null
• Copy Paste: [[2311.05050]] Quantum Generative Modeling of Sequential Data with Trainable Token Embedding(http://arxiv.org/abs/2311.05050)
• Summary:

  Generative models are a class of machine learning models that aim to learn the underlying probability distribution of data. Unlike discriminative models, generative models focus on capturing the data's inherent structure, allowing them to generate new samples that resemble the original data. To fully exploit the potential of modeling probability distributions using quantum physics, a quantum-inspired generative model known as the Born machines have shown great advancements in learning classical and quantum data over matrix product state(MPS) framework. The Born machines support tractable log-likelihood, autoregressive and mask sampling, and have shown outstanding performance in various unsupervised learning tasks. However, much of the current research has been centered on improving the expressive power of MPS, predominantly embedding each token directly by a corresponding tensor index. In this study, we generalize the embedding method into trainable quantum measurement operators that can be simultaneously honed with MPS. Our study indicated that combined with trainable embedding, Born machines can exhibit better performance and learn deeper correlations from the dataset.

Title: Social Media Bot Detection using Dropout-GAN. (arXiv:2311.05079v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.05079
• Code URL: null
• Copy Paste: [[2311.05079]] Social Media Bot Detection using Dropout-GAN(http://arxiv.org/abs/2311.05079)
• Summary:

  Bot activity on social media platforms is a pervasive problem, undermining the credibility of online discourse and potentially leading to cybercrime. We propose an approach to bot detection using Generative Adversarial Networks (GAN). We discuss how we overcome the issue of mode collapse by utilizing multiple discriminators to train against one generator, while decoupling the discriminator to perform social media bot detection and utilizing the generator for data augmentation. In terms of classification accuracy, our approach outperforms the state-of-the-art techniques in this field. We also show how the generator in the GAN can be used to evade such a classification technique.

Title: Parkinson's Disease Detection through Vocal Biomarkers and Advanced Machine Learning Algorithms: A Comprehensive Study. (arXiv:2311.05435v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.05435
• Code URL: null
• Copy Paste: [[2311.05435]] Parkinson's Disease Detection through Vocal Biomarkers and Advanced Machine Learning Algorithms: A Comprehensive Study(http://arxiv.org/abs/2311.05435)
• Summary:

  Parkinson's disease (PD) is a prevalent neurodegenerative disorder known for its impact on motor neurons, causing symptoms like tremors, stiffness, and gait difficulties. This study explores the potential of vocal feature alterations in PD patients as a means of early disease prediction. This research aims to predict the onset of Parkinson's disease. Utilizing a variety of advanced machine-learning algorithms, including XGBoost, LightGBM, Bagging, AdaBoost, and Support Vector Machine, among others, the study evaluates the predictive performance of these models using metrics such as accuracy, area under the curve (AUC), sensitivity, and specificity. The findings of this comprehensive analysis highlight LightGBM as the most effective model, achieving an impressive accuracy rate of 96%, alongside a matching AUC of 96%. LightGBM exhibited a remarkable sensitivity of 100% and specificity of 94.43%, surpassing other machine learning algorithms in accuracy and AUC scores. Given the complexities of Parkinson's disease and its challenges in early diagnosis, this study underscores the significance of leveraging vocal biomarkers coupled with advanced machine-learning techniques for precise and timely PD detection.

large language model

Title: u-LLaVA: Unifying Multi-Modal Tasks via Large Language Model. (arXiv:2311.05348v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.05348
• Code URL: null
• Copy Paste: [[2311.05348]] u-LLaVA: Unifying Multi-Modal Tasks via Large Language Model(http://arxiv.org/abs/2311.05348)
• Summary:

  Recent advances such as LLaVA and Mini-GPT4 have successfully integrated visual information into LLMs, yielding inspiring outcomes and giving rise to a new generation of multi-modal LLMs, or MLLMs. Nevertheless, these methods struggle with hallucinations and the mutual interference between tasks. To tackle these problems, we propose an efficient and accurate approach to adapt to downstream tasks by utilizing LLM as a bridge to connect multiple expert models, namely u-LLaVA. Firstly, we incorporate the modality alignment module and multi-task modules into LLM. Then, we reorganize or rebuild multi-type public datasets to enable efficient modality alignment and instruction following. Finally, task-specific information is extracted from the trained LLM and provided to different modules for solving downstream tasks. The overall framework is simple, effective, and achieves state-of-the-art performance across multiple benchmarks. We also release our model, the generated data, and the code base publicly available.

Title: From Text to Structure: Using Large Language Models to Support the Development of Legal Expert Systems. (arXiv:2311.04911v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.04911
• Code URL: null
• Copy Paste: [[2311.04911]] From Text to Structure: Using Large Language Models to Support the Development of Legal Expert Systems(http://arxiv.org/abs/2311.04911)
• Summary:

  Encoding legislative text in a formal representation is an important prerequisite to different tasks in the field of AI & Law. For example, rule-based expert systems focused on legislation can support laypeople in understanding how legislation applies to them and provide them with helpful context and information. However, the process of analyzing legislation and other sources to encode it in the desired formal representation can be time-consuming and represents a bottleneck in the development of such systems. Here, we investigate to what degree large language models (LLMs), such as GPT-4, are able to automatically extract structured representations from legislation. We use LLMs to create pathways from legislation, according to the JusticeBot methodology for legal decision support systems, evaluate the pathways and compare them to manually created pathways. The results are promising, with 60% of generated pathways being rated as equivalent or better than manually created ones in a blind comparison. The approach suggests a promising path to leverage the capabilities of LLMs to ease the costly development of systems based on symbolic approaches that are transparent and explainable.

Title: Chain of Empathy: Enhancing Empathetic Response of Large Language Models Based on Psychotherapy Models. (arXiv:2311.04915v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.04915
• Code URL: null
• Copy Paste: [[2311.04915]] Chain of Empathy: Enhancing Empathetic Response of Large Language Models Based on Psychotherapy Models(http://arxiv.org/abs/2311.04915)
• Summary:

  We present a novel method, the Chain of Empathy (CoE) prompting, that utilizes insights from psychotherapy to induce Large Language Models (LLMs) to reason about human emotional states. This method is inspired by various psychotherapy approaches including Cognitive Behavioral Therapy (CBT), Dialectical Behavior Therapy (DBT), Person Centered Therapy (PCT), and Reality Therapy (RT), each leading to different patterns of interpreting clients' mental states. LLMs without reasoning generated predominantly exploratory responses. However, when LLMs used CoE reasoning, we found a more comprehensive range of empathetic responses aligned with the different reasoning patterns of each psychotherapy model. The CBT based CoE resulted in the most balanced generation of empathetic responses. The findings underscore the importance of understanding the emotional context and how it affects human and AI communication. Our research contributes to understanding how psychotherapeutic models can be incorporated into LLMs, facilitating the development of context-specific, safer, and empathetic AI.

Title: Adapting Fake News Detection to the Era of Large Language Models. (arXiv:2311.04917v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.04917
• Code URL: null
• Copy Paste: [[2311.04917]] Adapting Fake News Detection to the Era of Large Language Models(http://arxiv.org/abs/2311.04917)
• Summary:

  In the age of large language models (LLMs) and the widespread adoption of AI-driven content creation, the landscape of information dissemination has witnessed a paradigm shift. With the proliferation of both human-written and machine-generated real and fake news, robustly and effectively discerning the veracity of news articles has become an intricate challenge. While substantial research has been dedicated to fake news detection, this either assumes that all news articles are human-written or abruptly assumes that all machine-generated news are fake. Thus, a significant gap exists in understanding the interplay between machine-(paraphrased) real news, machine-generated fake news, human-written fake news, and human-written real news. In this paper, we study this gap by conducting a comprehensive evaluation of fake news detectors trained in various scenarios. Our primary objectives revolve around the following pivotal question: How to adapt fake news detectors to the era of LLMs? Our experiments reveal an interesting pattern that detectors trained exclusively on human-written articles can indeed perform well at detecting machine-generated fake news, but not vice versa. Moreover, due to the bias of detectors against machine-generated texts \cite{su2023fake}, they should be trained on datasets with a lower machine-generated news ratio than the test set. Building on our findings, we provide a practical strategy for the development of robust fake news detectors.

Title: Successor Features for Efficient Multisubject Controlled Text Generation. (arXiv:2311.04921v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.04921
• Code URL: null
• Copy Paste: [[2311.04921]] Successor Features for Efficient Multisubject Controlled Text Generation(http://arxiv.org/abs/2311.04921)
• Summary:

  While large language models (LLMs) have achieved impressive performance in generating fluent and realistic text, controlling the generated text so that it exhibits properties such as safety, factuality, and non-toxicity remains challenging. % such as DExperts, GeDi, and rectification Existing decoding-based methods are static in terms of the dimension of control; if the target subject is changed, they require new training. Moreover, it can quickly become prohibitive to concurrently control multiple subjects. In this work, we introduce SF-GEN, which is grounded in two primary concepts: successor features (SFs) to decouple the LLM's dynamics from task-specific rewards, and language model rectification to proportionally adjust the probability of selecting a token based on the likelihood that the finished text becomes undesired. SF-GEN seamlessly integrates the two to enable dynamic steering of text generation with no need to alter the LLM's parameters. Thanks to the decoupling effect induced by successor features, our method proves to be memory-wise and computationally efficient for training as well as decoding, especially when dealing with multiple target subjects. To the best of our knowledge, our research represents the first application of successor features in text generation. In addition to its computational efficiency, the resultant language produced by our method is comparable to the SOTA (and outperforms baselines) in both control measures as well as language quality, which we demonstrate through a series of experiments in various controllable text generation tasks.

Title: Leveraging Large Language Models for Collective Decision-Making. (arXiv:2311.04928v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.04928
• Code URL: null
• Copy Paste: [[2311.04928]] Leveraging Large Language Models for Collective Decision-Making(http://arxiv.org/abs/2311.04928)
• Summary:

  In various work contexts, such as meeting scheduling, collaborating, and project planning, collective decision-making is essential but often challenging due to diverse individual preferences, varying work focuses, and power dynamics among members. To address this, we propose a system leveraging Large Language Models (LLMs) to facilitate group decision-making by managing conversations and balancing preferences among individuals. Our system extracts individual preferences and suggests options that satisfy a significant portion of the members. We apply this system to corporate meeting scheduling. We create synthetic employee profiles and simulate conversations at scale, leveraging LLMs to evaluate the system. Our results indicate efficient coordination with reduced interactions between members and the LLM-based system. The system also effectively refines proposed options over time, ensuring their quality and equity. Finally, we conduct a survey study involving human participants to assess our system's ability to aggregate preferences and reasoning. Our findings show that the system exhibits strong performance in both dimensions.

Title: An Interdisciplinary Outlook on Large Language Models for Scientific Research. (arXiv:2311.04929v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.04929
• Code URL: null
• Copy Paste: [[2311.04929]] An Interdisciplinary Outlook on Large Language Models for Scientific Research(http://arxiv.org/abs/2311.04929)
• Summary:

  In this paper, we describe the capabilities and constraints of Large Language Models (LLMs) within disparate academic disciplines, aiming to delineate their strengths and limitations with precision. We examine how LLMs augment scientific inquiry, offering concrete examples such as accelerating literature review by summarizing vast numbers of publications, enhancing code development through automated syntax correction, and refining the scientific writing process. Simultaneously, we articulate the challenges LLMs face, including their reliance on extensive and sometimes biased datasets, and the potential ethical dilemmas stemming from their use. Our critical discussion extends to the varying impacts of LLMs across fields, from the natural sciences, where they help model complex biological sequences, to the social sciences, where they can parse large-scale qualitative data. We conclude by offering a nuanced perspective on how LLMs can be both a boon and a boundary to scientific progress.

Title: Large language models implicitly learn to straighten neural sentence trajectories to construct a predictive representation of natural language. (arXiv:2311.04930v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.04930
• Code URL: null
• Copy Paste: [[2311.04930]] Large language models implicitly learn to straighten neural sentence trajectories to construct a predictive representation of natural language(http://arxiv.org/abs/2311.04930)
• Summary:

  Predicting upcoming events is critical to our ability to interact with our environment. Transformer models, trained on next-word prediction, appear to construct representations of linguistic input that can support diverse downstream tasks. But how does a predictive objective shape such representations? Inspired by recent work in vision (Henaff et al., 2019), we test a hypothesis about predictive representations of autoregressive transformers. In particular, we test whether the neural trajectory of a sentence becomes progressively straighter as it passes through the network layers. The key insight is that straighter trajectories should facilitate prediction via linear extrapolation. We quantify straightness using a 1-dimensional curvature metric, and present four findings in support of the trajectory straightening hypothesis: i) In trained models, the curvature decreases from the early to the deeper layers of the network. ii) Models that perform better on the next-word prediction objective exhibit greater decreases in curvature, suggesting that this improved ability to straighten sentence trajectories may be the driver of better language modeling performance. iii) Given the same linguistic context, the sequences that are generated by the model have lower curvature than the actual continuations observed in a language corpus, suggesting that the model favors straighter trajectories for making predictions. iv) A consistent relationship holds between the average curvature and the average surprisal of sentences in the deep model layers, such that sentences with straighter trajectories also have lower surprisal. Importantly, untrained models do not exhibit these behaviors. In tandem, these results support the trajectory straightening hypothesis and provide a possible mechanism for how the geometry of the internal representations of autoregressive models supports next word prediction.

Title: GPT4All: An Ecosystem of Open Source Compressed Language Models. (arXiv:2311.04931v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.04931
• Code URL: https://github.com/nomic-ai/gpt4all
• Copy Paste: [[2311.04931]] GPT4All: An Ecosystem of Open Source Compressed Language Models(http://arxiv.org/abs/2311.04931)
• Summary:

  Large language models (LLMs) have recently achieved human-level performance on a range of professional and academic benchmarks. The accessibility of these models has lagged behind their performance. State-of-the-art LLMs require costly infrastructure; are only accessible via rate-limited, geo-locked, and censored web interfaces; and lack publicly available code and technical reports. In this paper, we tell the story of GPT4All, a popular open source repository that aims to democratize access to LLMs. We outline the technical details of the original GPT4All model family, as well as the evolution of the GPT4All project from a single model into a fully fledged open source ecosystem. It is our hope that this paper acts as both a technical overview of the original GPT4All models as well as a case study on the subsequent growth of the GPT4All open source ecosystem.

Title: Evaluating Large Language Models in Ophthalmology. (arXiv:2311.04933v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.04933
• Code URL: null
• Copy Paste: [[2311.04933]] Evaluating Large Language Models in Ophthalmology(http://arxiv.org/abs/2311.04933)
• Summary:

  Purpose: The performance of three different large language models (LLMS) (GPT-3.5, GPT-4, and PaLM2) in answering ophthalmology professional questions was evaluated and compared with that of three different professional populations (medical undergraduates, medical masters, and attending physicians). Methods: A 100-item ophthalmology single-choice test was administered to three different LLMs (GPT-3.5, GPT-4, and PaLM2) and three different professional levels (medical undergraduates, medical masters, and attending physicians), respectively. The performance of LLM was comprehensively evaluated and compared with the human group in terms of average score, stability, and confidence. Results: Each LLM outperformed undergraduates in general, with GPT-3.5 and PaLM2 being slightly below the master's level, while GPT-4 showed a level comparable to that of attending physicians. In addition, GPT-4 showed significantly higher answer stability and confidence than GPT-3.5 and PaLM2. Conclusion: Our study shows that LLM represented by GPT-4 performs better in the field of ophthalmology. With further improvements, LLM will bring unexpected benefits in medical education and clinical decision making in the near future.

Title: LooGLE: Can Long-Context Language Models Understand Long Contexts?. (arXiv:2311.04939v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.04939
• Code URL: https://github.com/bigai-nlco/loogle
• Copy Paste: [[2311.04939]] LooGLE: Can Long-Context Language Models Understand Long Contexts?(http://arxiv.org/abs/2311.04939)
• Summary:

  Large language models (LLMs), despite their impressive performance in various language tasks, are typically limited to processing texts within context-window size. This limitation has spurred significant research efforts to enhance LLMs' long-context understanding with high-quality long-sequence benchmarks. However, prior datasets in this regard suffer from shortcomings, such as short context length compared to the context window of modern LLMs; outdated documents that have data leakage problems; and an emphasis on short dependency tasks rather than long dependency tasks. In this paper, we present LooGLE, a Long Context Generic Language Evaluation benchmark for LLMs' long context understanding. LooGLE features relatively new documents post-2022, with over 24,000 tokens per document and 6,000 newly generated questions spanning diverse domains. Human annotators meticulously crafted more than 1,100 high-quality question-answer pairs to meet the long dependency requirements. These pairs underwent thorough cross-validation, yielding the most precise assessment of LLMs' long dependency capabilities. The evaluation of eight state-of-the-art LLMs on LooGLE revealed key findings: (i) commercial models outperformed open-sourced models; (ii) LLMs excelled in short dependency tasks like short question-answering and cloze tasks but struggled with more intricate long dependency tasks; (iii) in-context learning and chaining thoughts offered only marginal improvements; (iv) retrieval-based techniques demonstrated substantial benefits for short question-answering, while strategies for extending context window length had limited impact on long context understanding. As such, LooGLE not only provides a systematic and comprehensive evaluation schema on long-context LLMs, but also sheds light on future development of enhanced models towards "true long-context understanding".

Title: Prompt Sketching for Large Language Models. (arXiv:2311.04954v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.04954
• Code URL: null
• Copy Paste: [[2311.04954]] Prompt Sketching for Large Language Models(http://arxiv.org/abs/2311.04954)
• Summary:

  Many recent prompting strategies for large language models (LLMs) query the model multiple times sequentially -- first to produce intermediate results and then the final answer. However, using these methods, both decoder and model are unaware of potential follow-up prompts, leading to disconnected and undesirably wordy intermediate responses. In this work, we address this issue by proposing prompt sketching, a new prompting paradigm in which an LLM does not only respond by completing a prompt, but by predicting values for multiple variables in a template. This way, sketching grants users more control over the generation process, e.g., by providing a reasoning framework via intermediate instructions, leading to better overall results. The key idea enabling sketching with existing, autoregressive models is to adapt the decoding procedure to also score follow-up instructions during text generation, thus optimizing overall template likelihood in inference. Our experiments show that in a zero-shot setting, prompt sketching outperforms existing, sequential prompting schemes such as direct asking or chain-of-thought on 7 out of 8 LLM benchmarking tasks, including state tracking, arithmetic reasoning, and general question answering. To facilitate future use, we release a number of generic, yet effective sketches applicable to many tasks, and an open source library called dclib, powering our sketch-aware decoders.

Title: On the steerability of large language models toward data-driven personas. (arXiv:2311.04978v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.04978
• Code URL: null
• Copy Paste: [[2311.04978]] On the steerability of large language models toward data-driven personas(http://arxiv.org/abs/2311.04978)
• Summary:

  The recent surge in Large Language Model (LLM) related applications has led to a concurrent escalation in expectations for LLMs to accommodate a myriad of personas and encompass a broad spectrum of perspectives. An important first step towards addressing this demand is to align language models with specific personas, be it groups of users or individuals. Towards this goal, we first present a new conceptualization of a persona. Moving beyond the traditional reliance on demographics like age, gender, or political party affiliation, we introduce a data-driven persona definition methodology built on collaborative-filtering. In this methodology, users are embedded into a continuous vector space based on their opinions and clustered into cohorts that manifest coherent views across specific inquiries. This methodology allows for a more nuanced understanding of different latent social groups present in the overall population (as opposed to simply using demographic groups) and enhances the applicability of model steerability. Finally, we present an efficient method to steer LLMs towards a particular persona. We learn a soft-prompting model to map the continuous representation of users into sequences of virtual tokens which, when prepended to the LLM input, enables the LLM to produce responses aligned with a given user. Our results show that our steerability algorithm is superior in performance compared to a collection of baselines.

Title: First Tragedy, then Parse: History Repeats Itself in the New Era of Large Language Models. (arXiv:2311.05020v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.05020
• Code URL: null
• Copy Paste: [[2311.05020]] First Tragedy, then Parse: History Repeats Itself in the New Era of Large Language Models(http://arxiv.org/abs/2311.05020)
• Summary:

  Many NLP researchers are experiencing an existential crisis triggered by the astonishing success of ChatGPT and other systems based on large language models (LLMs). After such a disruptive change to our understanding of the field, what is left to do? Taking a historical lens, we look for guidance from the first era of LLMs, which began in 2005 with large $n$-gram models for machine translation. We identify durable lessons from the first era, and more importantly, we identify evergreen problems where NLP researchers can continue to make meaningful contributions in areas where LLMs are ascendant. Among these lessons, we discuss the primacy of hardware advancement in shaping the availability and importance of scale, as well as the urgent challenge of quality evaluation, both automated and human. We argue that disparities in scale are transient and that researchers can work to reduce them; that data, rather than hardware, is still a bottleneck for many meaningful applications; that meaningful evaluation informed by actual use is still an open problem; and that there is still room for speculative approaches.

Title: Characterizing Large Language Models as Rationalizers of Knowledge-intensive Tasks. (arXiv:2311.05085v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.05085
• Code URL: null
• Copy Paste: [[2311.05085]] Characterizing Large Language Models as Rationalizers of Knowledge-intensive Tasks(http://arxiv.org/abs/2311.05085)
• Summary:

  Large language models (LLMs) are proficient at generating fluent text with minimal task-specific supervision. Yet, their ability to provide well-grounded rationalizations for knowledge-intensive tasks remains under-explored. Such tasks, like commonsense multiple-choice questions, require rationales based on world knowledge to support predictions and refute alternate options. We consider the task of generating knowledge-guided rationalization in natural language by using expert-written examples in a few-shot manner. Surprisingly, crowd-workers preferred knowledge-grounded rationales over crowdsourced rationalizations, citing their factuality, sufficiency, and comprehensive refutations. Although LLMs-generated rationales were preferable, further improvements in conciseness and novelty are required. In another study, we show how rationalization of incorrect model predictions erodes humans' trust in LLM-generated rationales. Motivated by these observations, we create a two-stage pipeline to review task predictions and eliminate potential incorrect decisions before rationalization, enabling trustworthy rationale generation.

Title: A Survey of Large Language Models in Medicine: Progress, Application, and Challenge. (arXiv:2311.05112v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.05112
• Code URL: https://github.com/ai-in-health/medllmspracticalguide
• Copy Paste: [[2311.05112]] A Survey of Large Language Models in Medicine: Progress, Application, and Challenge(http://arxiv.org/abs/2311.05112)
• Summary:

  Large language models (LLMs), such as ChatGPT, have achieved substantial attention due to their impressive human language understanding and generation capabilities. Therefore, the application of LLMs in medicine to assist physicians and patient care emerges as a promising research direction in both artificial intelligence and clinical medicine. To this end, this survey provides a comprehensive overview of the current progress, applications, and challenges faced by LLMs in medicine. Specifically, we aim to address the following questions: 1) What are LLMs and how can medical LLMs be built? 2) What are the downstream performances of medical LLMs? 3) How can medical LLMs be utilized in real-world clinical practice? 4) What challenges arise from the use of medical LLMs? 5) How can we better construct and utilize medical LLMs? As a result, this survey aims to provide insights into the opportunities and challenges of LLMs in medicine and serve as a valuable resource for constructing practical and effective medical LLMs. A regularly updated list of practical guide resources of medical LLMs can be found at https://github.com/AI-in-Health/MedLLMsPracticalGuide.

Title: Conic10K: A Challenging Math Problem Understanding and Reasoning Dataset. (arXiv:2311.05113v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.05113
• Code URL: https://github.com/whynlp/conic10k
• Copy Paste: [[2311.05113]] Conic10K: A Challenging Math Problem Understanding and Reasoning Dataset(http://arxiv.org/abs/2311.05113)
• Summary:

  Mathematical understanding and reasoning are crucial tasks for assessing the capabilities of artificial intelligence (AI). However, existing benchmarks either require just a few steps of reasoning, or only contain a small amount of data in one specific topic, making it hard to analyse AI's behaviour with reference to different problems within a specific topic in detail. In this work, we propose Conic10K, a challenging math problem dataset on conic sections in Chinese senior high school education. Our dataset contains various problems with different reasoning depths, while only the knowledge from conic sections is required. Since the dataset only involves a narrow range of knowledge, it is easy to separately analyse the knowledge a model possesses and the reasoning ability it has. For each problem, we provide a high-quality formal representation, the reasoning steps, and the final solution. Experiments show that existing large language models, including GPT-4, exhibit weak performance on complex reasoning. We hope that our findings could inspire more advanced techniques for precise natural language understanding and reasoning. Our dataset and codes are available at https://github.com/whyNLP/Conic10K.

Title: Enhancing Computation Efficiency in Large Language Models through Weight and Activation Quantization. (arXiv:2311.05161v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.05161
• Code URL: null
• Copy Paste: [[2311.05161]] Enhancing Computation Efficiency in Large Language Models through Weight and Activation Quantization(http://arxiv.org/abs/2311.05161)
• Summary:

  Large Language Models (LLMs) are proficient in natural language processing tasks, but their deployment is often restricted by extensive parameter sizes and computational demands. This paper focuses on post-training quantization (PTQ) in LLMs, specifically 4-bit weight and 8-bit activation (W4A8) quantization, to enhance computational efficiency -- a topic less explored compared to weight-only quantization. We present two innovative techniques: activation-quantization-aware scaling (AQAS) and sequence-length-aware calibration (SLAC) to enhance PTQ by considering the combined effects on weights and activations and aligning calibration sequence lengths to target tasks. Moreover, we introduce dINT, a hybrid data format combining integer and denormal representations, to address the underflow issue in W4A8 quantization, where small values are rounded to zero. Through rigorous evaluations of LLMs, including OPT and LLaMA, we demonstrate that our techniques significantly boost task accuracies to levels comparable with full-precision models. By developing arithmetic units compatible with dINT, we further confirm that our methods yield a 2$\times$ hardware efficiency improvement compared to 8-bit integer MAC unit.

Title: Large Language Models and Prompt Engineering for Biomedical Query Focused Multi-Document Summarisation. (arXiv:2311.05169v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.05169
• Code URL: null
• Copy Paste: [[2311.05169]] Large Language Models and Prompt Engineering for Biomedical Query Focused Multi-Document Summarisation(http://arxiv.org/abs/2311.05169)
• Summary:

  This paper reports on the use of prompt engineering and GPT-3.5 for biomedical query-focused multi-document summarisation. Using GPT-3.5 and appropriate prompts, our system achieves top ROUGE-F1 results in the task of obtaining short-paragraph-sized answers to biomedical questions in the 2023 BioASQ Challenge (BioASQ 11b). This paper confirms what has been observed in other domains: 1) Prompts that incorporated few-shot samples generally improved on their counterpart zero-shot variants; 2) The largest improvement was achieved by retrieval augmented generation. The fact that these prompts allow our top runs to rank within the top two runs of BioASQ 11b demonstrate the power of using adequate prompts for Large Language Models in general, and GPT-3.5 in particular, for query-focused summarisation.

Title: A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions. (arXiv:2311.05232v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.05232
• Code URL: null
• Copy Paste: [[2311.05232]] A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions(http://arxiv.org/abs/2311.05232)
• Summary:

  The emergence of large language models (LLMs) has marked a significant breakthrough in natural language processing (NLP), leading to remarkable advancements in text understanding and generation. Nevertheless, alongside these strides, LLMs exhibit a critical tendency to produce hallucinations, resulting in content that is inconsistent with real-world facts or user inputs. This phenomenon poses substantial challenges to their practical deployment and raises concerns over the reliability of LLMs in real-world scenarios, which attracts increasing attention to detect and mitigate these hallucinations. In this survey, we aim to provide a thorough and in-depth overview of recent advances in the field of LLM hallucinations. We begin with an innovative taxonomy of LLM hallucinations, then delve into the factors contributing to hallucinations. Subsequently, we present a comprehensive overview of hallucination detection methods and benchmarks. Additionally, representative approaches designed to mitigate hallucinations are introduced accordingly. Finally, we analyze the challenges that highlight the current limitations and formulate open questions, aiming to delineate pathways for future research on hallucinations in LLMs.

Title: DeeLM: Dependency-enhanced Large Language Model for Sentence Embeddings. (arXiv:2311.05296v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.05296
• Code URL: null
• Copy Paste: [[2311.05296]] DeeLM: Dependency-enhanced Large Language Model for Sentence Embeddings(http://arxiv.org/abs/2311.05296)
• Summary:

  Recent studies have proposed using large language models (LLMs) for sentence embeddings. However, most existing LLMs are built with an autoregressive architecture that primarily captures forward dependencies while neglecting backward dependencies. Previous work has highlighted the importance of backward dependencies in improving sentence embeddings. To address this issue, in this paper, we first present quantitative evidence demonstrating the limited learning of backward dependencies in LLMs. Then, we propose a novel approach called Dependency-Enhanced Large Language Model (DeeLM) to improve sentence embeddings. Specifically, we found a turning point in LLMs, where surpassing specific LLM layers leads to a significant performance drop in the semantic textual similarity (STS) task. STS is a crucial task for evaluating sentence embeddings. We then extract the layers after the turning point to make them bidirectional, allowing for the learning of backward dependencies. Extensive experiments demonstrate that DeeLM outperforms baselines and achieves state-of-the-art performance across various STS tasks.

Title: Do personality tests generalize to Large Language Models?. (arXiv:2311.05297v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.05297
• Code URL: null
• Copy Paste: [[2311.05297]] Do personality tests generalize to Large Language Models?(http://arxiv.org/abs/2311.05297)
• Summary:

  With large language models (LLMs) appearing to behave increasingly human-like in text-based interactions, it has become popular to attempt to evaluate various properties of these models using tests originally designed for humans. While re-using existing tests is a resource-efficient way to evaluate LLMs, careful adjustments are usually required to ensure that test results are even valid across human sub-populations. Thus, it is not clear to what extent different tests' validity generalizes to LLMs. In this work, we provide evidence that LLMs' responses to personality tests systematically deviate from typical human responses, implying that these results cannot be interpreted in the same way as human test results. Concretely, reverse-coded items (e.g. "I am introverted" vs "I am extraverted") are often both answered affirmatively by LLMs. In addition, variation across different prompts designed to "steer" LLMs to simulate particular personality types does not follow the clear separation into five independent personality factors from human samples. In light of these results, we believe it is important to pay more attention to tests' validity for LLMs before drawing strong conclusions about potentially ill-defined concepts like LLMs' "personality".

Title: TencentLLMEval: A Hierarchical Evaluation of Real-World Capabilities for Human-Aligned LLMs. (arXiv:2311.05374v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.05374
• Code URL: https://github.com/xsysigma/tencentllmeval
• Copy Paste: [[2311.05374]] TencentLLMEval: A Hierarchical Evaluation of Real-World Capabilities for Human-Aligned LLMs(http://arxiv.org/abs/2311.05374)
• Summary:

  Large language models (LLMs) have shown impressive capabilities across various natural language tasks. However, evaluating their alignment with human preferences remains a challenge. To this end, we propose a comprehensive human evaluation framework to assess LLMs' proficiency in following instructions on diverse real-world tasks. We construct a hierarchical task tree encompassing 7 major areas covering over 200 categories and over 800 tasks, which covers diverse capabilities such as question answering, reasoning, multiturn dialogue, and text generation, to evaluate LLMs in a comprehensive and in-depth manner. We also design detailed evaluation standards and processes to facilitate consistent, unbiased judgments from human evaluators. A test set of over 3,000 instances is released, spanning different difficulty levels and knowledge domains. Our work provides a standardized methodology to evaluate human alignment in LLMs for both English and Chinese. We also analyze the feasibility of automating parts of evaluation with a strong LLM (GPT-4). Our framework supports a thorough assessment of LLMs as they are integrated into real-world applications. We have made publicly available the task tree, TencentLLMEval dataset, and evaluation methodology which have been demonstrated as effective in assessing the performance of Tencent Hunyuan LLMs. By doing so, we aim to facilitate the benchmarking of advances in the development of safe and human-aligned LLMs.

Title: Zero-Shot Goal-Directed Dialogue via RL on Imagined Conversations. (arXiv:2311.05584v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.05584
• Code URL: null
• Copy Paste: [[2311.05584]] Zero-Shot Goal-Directed Dialogue via RL on Imagined Conversations(http://arxiv.org/abs/2311.05584)
• Summary:

  Large language models (LLMs) have emerged as powerful and general solutions to many natural language tasks. However, many of the most important applications of language generation are interactive, where an agent has to talk to a person to reach a desired outcome. For example, a teacher might try to understand their student's current comprehension level to tailor their instruction accordingly, and a travel agent might ask questions of their customer to understand their preferences in order to recommend activities they might enjoy. LLMs trained with supervised fine-tuning or "single-step" RL, as with standard RLHF, might struggle which tasks that require such goal-directed behavior, since they are not trained to optimize for overall conversational outcomes after multiple turns of interaction. In this work, we explore a new method for adapting LLMs with RL for such goal-directed dialogue. Our key insight is that, though LLMs might not effectively solve goal-directed dialogue tasks out of the box, they can provide useful data for solving such tasks by simulating suboptimal but human-like behaviors. Given a textual description of a goal-directed dialogue task, we leverage LLMs to sample diverse synthetic rollouts of hypothetical in-domain human-human interactions. Our algorithm then utilizes this dataset with offline reinforcement learning to train an interactive conversational agent that can optimize goal-directed objectives over multiple turns. In effect, the LLM produces examples of possible interactions, and RL then processes these examples to learn to perform more optimal interactions. Empirically, we show that our proposed approach achieves state-of-the-art performance in various goal-directed dialogue tasks that include teaching and preference elicitation.

Title: DEMASQ: Unmasking the ChatGPT Wordsmith. (arXiv:2311.05019v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.05019
• Code URL: null
• Copy Paste: [[2311.05019]] DEMASQ: Unmasking the ChatGPT Wordsmith(http://arxiv.org/abs/2311.05019)
• Summary:

  The potential misuse of ChatGPT and other Large Language Models (LLMs) has raised concerns regarding the dissemination of false information, plagiarism, academic dishonesty, and fraudulent activities. Consequently, distinguishing between AI-generated and human-generated content has emerged as an intriguing research topic. However, current text detection methods lack precision and are often restricted to specific tasks or domains, making them inadequate for identifying content generated by ChatGPT. In this paper, we propose an effective ChatGPT detector named DEMASQ, which accurately identifies ChatGPT-generated content. Our method addresses two critical factors: (i) the distinct biases in text composition observed in human- and machine-generated content and (ii) the alterations made by humans to evade previous detection methods. DEMASQ is an energy-based detection model that incorporates novel aspects, such as (i) optimization inspired by the Doppler effect to capture the interdependence between input text embeddings and output labels, and (ii) the use of explainable AI techniques to generate diverse perturbations. To evaluate our detector, we create a benchmark dataset comprising a mixture of prompts from both ChatGPT and humans, encompassing domains such as medical, open Q&A, finance, wiki, and Reddit. Our evaluation demonstrates that DEMASQ achieves high accuracy in identifying content generated by ChatGPT.

Title: RAGLog: Log Anomaly Detection using Retrieval Augmented Generation. (arXiv:2311.05261v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.05261
• Code URL: null
• Copy Paste: [[2311.05261]] RAGLog: Log Anomaly Detection using Retrieval Augmented Generation(http://arxiv.org/abs/2311.05261)
• Summary:

  The ability to detect log anomalies from system logs is a vital activity needed to ensure cyber resiliency of systems. It is applied for fault identification or facilitate cyber investigation and digital forensics. However, as logs belonging to different systems and components differ significantly, the challenge to perform such analysis is humanly challenging from the volume, variety and velocity of logs. This is further complicated by the lack or unavailability of anomalous log entries to develop trained machine learning or artificial intelligence models for such purposes. In this research work, we explore the use of a Retrieval Augmented Large Language Model that leverages a vector database to detect anomalies from logs. We used a Question and Answer configuration pipeline. To the best of our knowledge, our experiment which we called RAGLog is a novel one and the experimental results show much promise.

Title: Efficient Parallelization Layouts for Large-Scale Distributed Model Training. (arXiv:2311.05610v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.05610
• Code URL: https://github.com/aleph-alpha/neurips-want-submission-efficient-parallelization-layouts
• Copy Paste: [[2311.05610]] Efficient Parallelization Layouts for Large-Scale Distributed Model Training(http://arxiv.org/abs/2311.05610)
• Summary:

  Efficiently training large language models requires parallelizing across hundreds of hardware accelerators and invoking various compute and memory optimizations. When combined, many of these strategies have complex interactions regarding the final training efficiency. Prior work tackling this problem did not have access to the latest set of optimizations, such as FlashAttention or sequence parallelism. In this work, we conduct a comprehensive ablation study of possible training configurations for large language models. We distill this large study into several key recommendations for the most efficient training. For instance, we find that using a micro-batch size of 1 usually enables the most efficient training layouts. Larger micro-batch sizes necessitate activation checkpointing or higher degrees of model parallelism and also lead to larger pipeline bubbles. Our most efficient configurations enable us to achieve state-of-the-art training efficiency results over a range of model sizes, most notably a Model FLOPs utilization of 70.5% when training a 13B model.

segmentation

Title: Reducing the Side-Effects of Oscillations in Training of Quantized YOLO Networks. (arXiv:2311.05109v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.05109
• Code URL: null
• Copy Paste: [[2311.05109]] Reducing the Side-Effects of Oscillations in Training of Quantized YOLO Networks(http://arxiv.org/abs/2311.05109)
• Summary:

  Quantized networks use less computational and memory resources and are suitable for deployment on edge devices. While quantization-aware training QAT is the well-studied approach to quantize the networks at low precision, most research focuses on over-parameterized networks for classification with limited studies on popular and edge device friendly single-shot object detection and semantic segmentation methods like YOLO. Moreover, majority of QAT methods rely on Straight-through Estimator (STE) approximation which suffers from an oscillation phenomenon resulting in sub-optimal network quantization. In this paper, we show that it is difficult to achieve extremely low precision (4-bit and lower) for efficient YOLO models even with SOTA QAT methods due to oscillation issue and existing methods to overcome this problem are not effective on these models. To mitigate the effect of oscillation, we first propose Exponentially Moving Average (EMA) based update to the QAT model. Further, we propose a simple QAT correction method, namely QC, that takes only a single epoch of training after standard QAT procedure to correct the error induced by oscillating weights and activations resulting in a more accurate quantized model. With extensive evaluation on COCO dataset using various YOLO5 and YOLO7 variants, we show that our correction method improves quantized YOLO networks consistently on both object detection and segmentation tasks at low-precision (4-bit and 3-bit).

Title: ScribblePolyp: Scribble-Supervised Polyp Segmentation through Dual Consistency Alignment. (arXiv:2311.05122v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.05122
• Code URL: null
• Copy Paste: [[2311.05122]] ScribblePolyp: Scribble-Supervised Polyp Segmentation through Dual Consistency Alignment(http://arxiv.org/abs/2311.05122)
• Summary:

  Automatic polyp segmentation models play a pivotal role in the clinical diagnosis of gastrointestinal diseases. In previous studies, most methods relied on fully supervised approaches, necessitating pixel-level annotations for model training. However, the creation of pixel-level annotations is both expensive and time-consuming, impeding the development of model generalization. In response to this challenge, we introduce ScribblePolyp, a novel scribble-supervised polyp segmentation framework. Unlike fully-supervised models, ScribblePolyp only requires the annotation of two lines (scribble labels) for each image, significantly reducing the labeling cost. Despite the coarse nature of scribble labels, which leave a substantial portion of pixels unlabeled, we propose a two-branch consistency alignment approach to provide supervision for these unlabeled pixels. The first branch employs transformation consistency alignment to narrow the gap between predictions under different transformations of the same input image. The second branch leverages affinity propagation to refine predictions into a soft version, extending additional supervision to unlabeled pixels. In summary, ScribblePolyp is an efficient model that does not rely on teacher models or moving average pseudo labels during training. Extensive experiments on the SUN-SEG dataset underscore the effectiveness of ScribblePolyp, achieving a Dice score of 0.8155, with the potential for a 1.8% improvement in the Dice score through a straightforward self-training strategy.

Title: Adaptive-Labeling for Enhancing Remote Sensing Cloud Understanding. (arXiv:2311.05198v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.05198
• Code URL: null
• Copy Paste: [[2311.05198]] Adaptive-Labeling for Enhancing Remote Sensing Cloud Understanding(http://arxiv.org/abs/2311.05198)
• Summary:

  Cloud analysis is a critical component of weather and climate science, impacting various sectors like disaster management. However, achieving fine-grained cloud analysis, such as cloud segmentation, in remote sensing remains challenging due to the inherent difficulties in obtaining accurate labels, leading to significant labeling errors in training data. Existing methods often assume the availability of reliable segmentation annotations, limiting their overall performance. To address this inherent limitation, we introduce an innovative model-agnostic Cloud Adaptive-Labeling (CAL) approach, which operates iteratively to enhance the quality of training data annotations and consequently improve the performance of the learned model. Our methodology commences by training a cloud segmentation model using the original annotations. Subsequently, it introduces a trainable pixel intensity threshold for adaptively labeling the cloud training images on the fly. The newly generated labels are then employed to fine-tune the model. Extensive experiments conducted on multiple standard cloud segmentation benchmarks demonstrate the effectiveness of our approach in significantly boosting the performance of existing segmentation models. Our CAL method establishes new state-of-the-art results when compared to a wide array of existing alternatives.

Title: SAMVG: A Multi-stage Image Vectorization Model with the Segment-Anything Model. (arXiv:2311.05276v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.05276
• Code URL: null
• Copy Paste: [[2311.05276]] SAMVG: A Multi-stage Image Vectorization Model with the Segment-Anything Model(http://arxiv.org/abs/2311.05276)
• Summary:

  Vector graphics are widely used in graphical designs and have received more and more attention. However, unlike raster images which can be easily obtained, acquiring high-quality vector graphics, typically through automatically converting from raster images remains a significant challenge, especially for more complex images such as photos or artworks. In this paper, we propose SAMVG, a multi-stage model to vectorize raster images into SVG (Scalable Vector Graphics). Firstly, SAMVG uses general image segmentation provided by the Segment-Anything Model and uses a novel filtering method to identify the best dense segmentation map for the entire image. Secondly, SAMVG then identifies missing components and adds more detailed components to the SVG. Through a series of extensive experiments, we demonstrate that SAMVG can produce high quality SVGs in any domain while requiring less computation time and complexity compared to previous state-of-the-art methods.

Title: SeaTurtleID2022: A long-span dataset for reliable sea turtle re-identification. (arXiv:2311.05524v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.05524
• Code URL: null
• Copy Paste: [[2311.05524]] SeaTurtleID2022: A long-span dataset for reliable sea turtle re-identification(http://arxiv.org/abs/2311.05524)
• Summary:

  This paper introduces the first public large-scale, long-span dataset with sea turtle photographs captured in the wild -- SeaTurtleID2022 (https://www.kaggle.com/datasets/wildlifedatasets/seaturtleid2022). The dataset contains 8729 photographs of 438 unique individuals collected within 13 years, making it the longest-spanned dataset for animal re-identification. All photographs include various annotations, e.g., identity, encounter timestamp, and body parts segmentation masks. Instead of standard "random" splits, the dataset allows for two realistic and ecologically motivated splits: (i) a time-aware closed-set with training, validation, and test data from different days/years, and (ii) a time-aware open-set with new unknown individuals in test and validation sets. We show that time-aware splits are essential for benchmarking re-identification methods, as random splits lead to performance overestimation. Furthermore, a baseline instance segmentation and re-identification performance over various body parts is provided. Finally, an end-to-end system for sea turtle re-identification is proposed and evaluated. The proposed system based on Hybrid Task Cascade for head instance segmentation and ArcFace-trained feature-extractor achieved an accuracy of 86.8%.