2024-01-04

secure

Title: On Optimal Sampling for Learning SDF Using MLPs Equipped with Positional Encoding. (arXiv:2401.01391v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.01391
• Code URL: null
• Copy Paste: [[2401.01391]] On Optimal Sampling for Learning SDF Using MLPs Equipped with Positional Encoding(http://arxiv.org/abs/2401.01391)
• Summary:

  Neural implicit fields, such as the neural signed distance field (SDF) of a shape, have emerged as a powerful representation for many applications, e.g., encoding a 3D shape and performing collision detection. Typically, implicit fields are encoded by Multi-layer Perceptrons (MLP) with positional encoding (PE) to capture high-frequency geometric details. However, a notable side effect of such PE-equipped MLPs is the noisy artifacts present in the learned implicit fields. While increasing the sampling rate could in general mitigate these artifacts, in this paper we aim to explain this adverse phenomenon through the lens of Fourier analysis. We devise a tool to determine the appropriate sampling rate for learning an accurate neural implicit field without undesirable side effects. Specifically, we propose a simple yet effective method to estimate the intrinsic frequency of a given network with randomized weights based on the Fourier analysis of the network's responses. It is observed that a PE-equipped MLP has an intrinsic frequency much higher than the highest frequency component in the PE layer. Sampling against this intrinsic frequency following the Nyquist-Sannon sampling theorem allows us to determine an appropriate training sampling rate. We empirically show in the setting of SDF fitting that this recommended sampling rate is sufficient to secure accurate fitting results, while further increasing the sampling rate would not further noticeably reduce the fitting error. Training PE-equipped MLPs simply with our sampling strategy leads to performances superior to the existing methods.

Title: Architectural Design for Secure Smart Contract Development. (arXiv:2401.01891v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2401.01891
• Code URL: null
• Copy Paste: [[2401.01891]] Architectural Design for Secure Smart Contract Development(http://arxiv.org/abs/2401.01891)
• Summary:

  As time progresses, the need for more secure applications grows exponentially. The different types of sensitive information that is being transferred virtually has sparked a rise in systems that leverage blockchain. Different sectors are beginning to use this disruptive technology to evaluate the risks and benefits. Sectors like finance, medicine, higher education, and wireless communication have research regarding blockchain. Futhermore, the need for security standards in this area of research is pivotal. In recent past, several attacks on blockchain infrastructures have resulted in hundreds of millions dollars lost and sensitive information compromised. Some of these attacks include DAO attacks, bZx attacks, and Parity Multisignature Wallet Double Attacks which targeted vulnerabilities within smart contracts on the Ethereum network. These attacks exposed the weaknesses of current smart contract development practices which has led to the increase in distrust and adoption of systems that leverage blockchain for its functionality. In this paper, I identify common software vulnerabilities and attacks on blockchain infrastructures, thoroughly detail the smart contract development process and propose a model for ensuring a stronger security standard for future systems leveraging smart contracts. The purpose for proposing a model is to promote trust among end users in the system which is a foundational element for blockchain adoption in the future.

security

Title: SD-WAN over MPLS: A Comprehensive Performance Analysis and Security with Insights into the Future of SD-WAN. (arXiv:2401.01344v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2401.01344
• Code URL: null
• Copy Paste: [[2401.01344]] SD-WAN over MPLS: A Comprehensive Performance Analysis and Security with Insights into the Future of SD-WAN(http://arxiv.org/abs/2401.01344)
• Summary:

  Software-defined wide area network (SD-WAN) enhances network traffic management, while Multiprotocol Label Switching (MPLS) offers efficient data transmission. This paper analyzes SD-WAN over MPLS in the Housing Bank, a major Algerian financial institution. We deploy FortiGate for the SD-WAN solution, comparing it to traditional MPLS and direct internet access across metrics like bandwidth, latency, jitter, packet loss, throughput, and quality of service (QoS). Security measures include encryption, firewall, intrusion prevention, web filtering, antivirus, and addressing threats like spoofing, DoS attacks, and unauthorized access. We explore future trends such as SASE architecture, AI/ML integration, and emerging transport methods. SD-WAN over MPLS proves advantageous, offering enhanced performance, security, and flexibility. Recommendations include ongoing performance monitoring and research.

Title: Security, extensibility, and redundancy in the Metabolic Operating System. (arXiv:2401.01357v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2401.01357
• Code URL: null
• Copy Paste: [[2401.01357]] Security, extensibility, and redundancy in the Metabolic Operating System(http://arxiv.org/abs/2401.01357)
• Summary:

  People living with Type 1 Diabetes (T1D) lose the ability to produce insulin naturally. To compensate, they inject synthetic insulin. One common way to inject insulin is through automated insulin delivery systems, which use sensors to monitor their metabolic state and an insulin pump device to adjust insulin to adapt.

  In this paper, we present the Metabolic Operating System, a new automated insulin delivery system that we designed from the ground up using security first principles. From an architecture perspective, we apply separation principles to simplify the core system and isolate non-critical functionality from the core closed-loop algorithm. From an algorithmic perspective, we evaluate trends in insulin technology and formulate a simple, but effective, algorithm given the state-of-the-art. From a safety perspective, we build in multiple layers of redundancy to ensure that the person using our system remains safe.

  Fundamentally, this paper is a paper on real-world experiences building and running an automated insulin delivery system. We report on the design iterations we make based on experiences working with one individual using our system. Our evaluation shows that an automated insulin delivery system built from the ground up using security first principles can still help manage T1D effectively.

  Our source code is open source and available on GitHub (link omitted).

Title: Taxonomy for Cybersecurity Threat Attributes and Countermeasures in Smart Manufacturing Systems. (arXiv:2401.01374v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2401.01374
• Code URL: null
• Copy Paste: [[2401.01374]] Taxonomy for Cybersecurity Threat Attributes and Countermeasures in Smart Manufacturing Systems(http://arxiv.org/abs/2401.01374)
• Summary:

  An attack taxonomy offers a consistent and structured classification scheme to systematically understand, identify, and classify cybersecurity threat attributes. However, existing taxonomies only focus on a narrow range of attacks and limited threat attributes, lacking a comprehensive characterization of manufacturing cybersecurity threats. There is little to no focus on characterizing threat actors and their intent, specific system and machine behavioral deviations introduced by cyberattacks, system-level and operational implications of attacks, and potential countermeasures against those attacks. To close this pressing research gap, this work proposes a comprehensive attack taxonomy for a holistic understanding and characterization of cybersecurity threats in manufacturing systems. Specifically, it introduces taxonomical classifications for threat actors and their intent and potential alterations in system behavior due to threat events. The proposed taxonomy categorizes attack methods/vectors and targets/locations and incorporates operational and system-level attack impacts. This paper also presents a classification structure for countermeasures, provides examples of potential countermeasures, and explains how they fit into the proposed taxonomical classification. Finally, the implementation of the proposed taxonomy is illustrated using two realistic scenarios of attacks on typical smart manufacturing systems, as well as several real-world cyber-physical attack incidents and academic case studies. The developed manufacturing attack taxonomy offers a holistic view of the attack chain in manufacturing systems, starting from the attack launch to the possible damages and system behavior changes within the system. Furthermore, it guides the design and development of appropriate protective and detective countermeasures by leveraging the attack realization through observed system deviations.

Title: ALPC Is In Danger: ALPChecker Detects Spoofing and Blinding. (arXiv:2401.01376v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2401.01376
• Code URL: null
• Copy Paste: [[2401.01376]] ALPC Is In Danger: ALPChecker Detects Spoofing and Blinding(http://arxiv.org/abs/2401.01376)
• Summary:

  The purpose of this study is to evaluate the possibility of implementing an attack on ALPC connection in the Windows operating system through the kernel without closing the connection covertly from programs and the operating system and to propose a method of protection against this type of attacks. Asynchronous Local Procedure Call technology (ALPC) is used in various Windows information protection systems, including antivirus systems (AV) and Endpoint Detection and Response systems (EDR). To ensure the concealment of malicious software, attackers need to disrupt the operation of AV, EDR tools, which in turn can be achieved by destructive impact on the components of the ALPC technology. Examples of such attacks already exist and are covered in this paper. To counteract such new threats, it is necessary to advance the improvement of information security systems and the ALPC security research was conducted. The most difficult case, Windows kernel driver attack, was considered. Three attacks on the ALPC connection were carried out, based on changing the ALPC structures in the kernel memory, which led to creation of illegitimate connections in the system and the disruption of correct connections. ALPChecker protection tool has been developed. The tool was successfully tested on three demonstrated attacks.

Title: Specific Emitter Identification Based on Joint Variational Mode Decomposition. (arXiv:2401.01503v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2401.01503
• Code URL: null
• Copy Paste: [[2401.01503]] Specific Emitter Identification Based on Joint Variational Mode Decomposition(http://arxiv.org/abs/2401.01503)
• Summary:

  Specific emitter identification (SEI) technology is significant in device administration scenarios, such as self-organized networking and spectrum management, owing to its high security. For nonlinear and non-stationary electromagnetic signals, SEI often employs variational modal decomposition (VMD) to decompose the signal in order to effectively characterize the distinct device fingerprint. However, the trade-off of VMD between the robustness to noise and the ability to preserve signal information has not been investigated in the current literature. Moreover, the existing VMD algorithm does not utilize the stability of the intrinsic distortion of emitters within a certain temporal span, consequently constraining its practical applicability in SEI. In this paper, we propose a joint variational modal decomposition (JVMD) algorithm, which is an improved version of VMD by simultaneously implementing modal decomposition on multi-frame signals. The consistency of multi-frame signals in terms of the central frequencies and the inherent modal functions (IMFs) is exploited, which effectively highlights the distinctive characteristics among emitters and reduces noise. Additionally, the complexity of JVMD is analyzed, which is proven to be more computational-friendly than VMD. Simulations of both modal decomposition and SEI that involve real-world datasets are presented to illustrate that when compared with VMD, the JVMD algorithm improves the accuracy of device classification and the robustness towards noise.

Title: A Survey of Protocol Fuzzing. (arXiv:2401.01568v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2401.01568
• Code URL: null
• Copy Paste: [[2401.01568]] A Survey of Protocol Fuzzing(http://arxiv.org/abs/2401.01568)
• Summary:

  Communication protocols form the bedrock of our interconnected world, yet vulnerabilities within their implementations pose significant security threats. Recent developments have seen a surge in fuzzing-based research dedicated to uncovering these vulnerabilities within protocol implementations. However, there still lacks a systematic overview of protocol fuzzing for answering the essential questions such as what the unique challenges are, how existing works solve them, etc. To bridge this gap, we conducted a comprehensive investigation of related works from both academia and industry. Our study includes a detailed summary of the specific challenges in protocol fuzzing, and provides a systematic categorization and overview of existing research efforts. Furthermore, we explore and discuss potential future research directions in protocol fuzzing. This survey serves as a foundational guideline for researchers and practitioners in the field.

Title: The Security and Privacy of Mobile Edge Computing: An Artificial Intelligence Perspective. (arXiv:2401.01589v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2401.01589
• Code URL: null
• Copy Paste: [[2401.01589]] The Security and Privacy of Mobile Edge Computing: An Artificial Intelligence Perspective(http://arxiv.org/abs/2401.01589)
• Summary:

  Mobile Edge Computing (MEC) is a new computing paradigm that enables cloud computing and information technology (IT) services to be delivered at the network's edge. By shifting the load of cloud computing to individual local servers, MEC helps meet the requirements of ultralow latency, localized data processing, and extends the potential of Internet of Things (IoT) for end-users. However, the crosscutting nature of MEC and the multidisciplinary components necessary for its deployment have presented additional security and privacy concerns. Fortunately, Artificial Intelligence (AI) algorithms can cope with excessively unpredictable and complex data, which offers a distinct advantage in dealing with sophisticated and developing adversaries in the security industry. Hence, in this paper we comprehensively provide a survey of security and privacy in MEC from the perspective of AI. On the one hand, we use European Telecommunications Standards Institute (ETSI) MEC reference architecture as our based framework while merging the Software Defined Network (SDN) and Network Function Virtualization (NFV) to better illustrate a serviceable platform of MEC. On the other hand, we focus on new security and privacy issues, as well as potential solutions from the viewpoints of AI. Finally, we comprehensively discuss the opportunities and challenges associated with applying AI to MEC security and privacy as possible future research directions.

privacy

Title: Directional Antenna Systems for Long-Range Through-Wall Human Activity Recognition. (arXiv:2401.01388v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.01388
• Code URL: null
• Copy Paste: [[2401.01388]] Directional Antenna Systems for Long-Range Through-Wall Human Activity Recognition(http://arxiv.org/abs/2401.01388)
• Summary:

  WiFi Channel State Information (CSI)-based human activity recognition (HAR) enables contactless, long-range sensing in spatially constrained environments while preserving visual privacy. However, despite the presence of numerous WiFi-enabled devices around us, few expose CSI to users, resulting in a lack of sensing hardware options. Variants of the Espressif ESP32 have emerged as potential low-cost and easy-to-deploy solutions for WiFi CSI-based HAR. In this work, four ESP32-S3-based 2.4GHz directional antenna systems are evaluated for their ability to facilitate long-range through-wall HAR. Two promising systems are proposed, one of which combines the ESP32-S3 with a directional biquad antenna. This combination represents, to the best of our knowledge, the first demonstration of such a system in WiFi-based HAR. The second system relies on the built-in printed inverted-F antenna (PIFA) of the ESP32-S3 and achieves directionality through a plane reflector. In a comprehensive evaluation of line-of-sight (LOS) and non-line-of-sight (NLOS) HAR performance, both systems are deployed in an office environment spanning a distance of 18 meters across five rooms. In this experimental setup, the Wallhack1.8k dataset, comprising 1806 CSI amplitude spectrograms of human activities, is collected and made publicly available. Based on Wallhack1.8k, we train activity recognition models using the EfficientNetV2 architecture to assess system performance in LOS and NLOS scenarios. For the core NLOS activity recognition problem, the biquad antenna and PIFA-based systems achieve accuracies of 92.0$\pm$3.5 and 86.8$\pm$4.7, respectively, demonstrating the feasibility of long-range through-wall HAR with the proposed systems.

Title: Enhancing Generalization of Invisible Facial Privacy Cloak via Gradient Accumulation. (arXiv:2401.01575v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.01575
• Code URL: null
• Copy Paste: [[2401.01575]] Enhancing Generalization of Invisible Facial Privacy Cloak via Gradient Accumulation(http://arxiv.org/abs/2401.01575)
• Summary:

  The blooming of social media and face recognition (FR) systems has increased people's concern about privacy and security. A new type of adversarial privacy cloak (class-universal) can be applied to all the images of regular users, to prevent malicious FR systems from acquiring their identity information. In this work, we discover the optimization dilemma in the existing methods -- the local optima problem in large-batch optimization and the gradient information elimination problem in small-batch optimization. To solve these problems, we propose Gradient Accumulation (GA) to aggregate multiple small-batch gradients into a one-step iterative gradient to enhance the gradient stability and reduce the usage of quantization operations. Experiments show that our proposed method achieves high performance on the Privacy-Commons dataset against black-box face recognition models.

Title: Real-Time Human Fall Detection using a Lightweight Pose Estimation Technique. (arXiv:2401.01587v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.01587
• Code URL: null
• Copy Paste: [[2401.01587]] Real-Time Human Fall Detection using a Lightweight Pose Estimation Technique(http://arxiv.org/abs/2401.01587)
• Summary:

  The elderly population is increasing rapidly around the world. There are no enough caretakers for them. Use of AI-based in-home medical care systems is gaining momentum due to this. Human fall detection is one of the most important tasks of medical care system for the aged people. Human fall is a common problem among elderly people. Detection of a fall and providing medical help as early as possible is very important to reduce any further complexity. The chances of death and other medical complications can be reduced by detecting and providing medical help as early as possible after the fall. There are many state-of-the-art fall detection techniques available these days, but the majority of them need very high computing power. In this paper, we proposed a lightweight and fast human fall detection system using pose estimation. We used `Movenet' for human joins key-points extraction. Our proposed method can work in real-time on any low-computing device with any basic camera. All computation can be processed locally, so there is no problem of privacy of the subject. We used two datasets `GMDCSA' and `URFD' for the experiment. We got the sensitivity value of 0.9375 and 0.9167 for the dataset `GMDCSA' and `URFD' respectively. The source code and the dataset GMDCSA of our work are available online to access.

Title: Step length measurement in the wild using FMCW radar. (arXiv:2401.01868v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.01868
• Code URL: null
• Copy Paste: [[2401.01868]] Step length measurement in the wild using FMCW radar(http://arxiv.org/abs/2401.01868)
• Summary:

  With an aging population, numerous assistive and monitoring technologies are under development to enable older adults to age in place. To facilitate aging in place predicting risk factors such as falls, and hospitalization and providing early interventions are important. Much of the work on ambient monitoring for risk prediction has centered on gait speed analysis, utilizing privacy-preserving sensors like radar. Despite compelling evidence that monitoring step length, in addition to gait speed, is crucial for predicting risk, radar-based methods have not explored step length measurement in the home. Furthermore, laboratory experiments on step length measurement using radars are limited to proof of concept studies with few healthy subjects. To address this gap, a radar-based step length measurement system for the home is proposed based on detection and tracking using radar point cloud, followed by Doppler speed profiling of the torso to obtain step lengths in the home. The proposed method was evaluated in a clinical environment, involving 35 frail older adults, to establish its validity. Additionally, the method was assessed in people's homes, with 21 frail older adults who had participated in the clinical assessment. The proposed radar-based step length measurement method was compared to the gold standard Zeno Walkway Gait Analysis System, revealing a 4.5cm/8.3% error in a clinical setting. Furthermore, it exhibited excellent reliability (ICC(2,k)=0.91, 95% CI 0.82 to 0.96) in uncontrolled home settings. The method also proved accurate in uncontrolled home settings, as indicated by a strong agreement (ICC(3,k)=0.81 (95% CI 0.53 to 0.92)) between home measurements and in-clinic assessments.

Title: Predicting challenge moments from students' discourse: A comparison of GPT-4 to two traditional natural language processing approaches. (arXiv:2401.01692v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.01692
• Code URL: null
• Copy Paste: [[2401.01692]] Predicting challenge moments from students' discourse: A comparison of GPT-4 to two traditional natural language processing approaches(http://arxiv.org/abs/2401.01692)
• Summary:

  Effective collaboration requires groups to strategically regulate themselves to overcome challenges. Research has shown that groups may fail to regulate due to differences in members' perceptions of challenges which may benefit from external support. In this study, we investigated the potential of leveraging three distinct natural language processing models: an expert knowledge rule-based model, a supervised machine learning (ML) model and a Large Language model (LLM), in challenge detection and challenge dimension identification (cognitive, metacognitive, emotional and technical/other challenges) from student discourse, was investigated. The results show that the supervised ML and the LLM approaches performed considerably well in both tasks, in contrast to the rule-based approach, whose efficacy heavily relies on the engineered features by experts. The paper provides an extensive discussion of the three approaches' performance for automated detection and support of students' challenge moments in collaborative learning activities. It argues that, although LLMs provide many advantages, they are unlikely to be the panacea to issues of the detection and feedback provision of socially shared regulation of learning due to their lack of reliability, as well as issues of validity evaluation, privacy and confabulation. We conclude the paper with a discussion on additional considerations, including model transparency to explore feasible and meaningful analytical feedback for students and educators using LLMs.

Title: The Boomerang protocol: A Decentralised Privacy-Preserving Verifiable Incentive Protocol. (arXiv:2401.01353v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2401.01353
• Code URL: null
• Copy Paste: [[2401.01353]] The Boomerang protocol: A Decentralised Privacy-Preserving Verifiable Incentive Protocol(http://arxiv.org/abs/2401.01353)
• Summary:

  In the era of data-driven economies, incentive systems and loyalty programs, have become ubiquitous in various sectors, including advertising, retail, travel, and financial services. While these systems offer advantages for both users and companies, they necessitate the transfer and analysis of substantial amounts of sensitive data. Privacy concerns have become increasingly pertinent, necessitating the development of privacy-preserving incentive protocols. Despite the rising demand for secure and decentralised systems, the existing landscape lacks a comprehensive solution. We propose the Boomerang protocol, a novel decentralised privacy-preserving incentive protocol that leverages cryptographic black box accumulators to securely store user interactions within the incentive system. Moreover, the protocol employs zero-knowledge proofs based on BulletProofs to transparently compute rewards for users, ensuring verifiability while preserving their privacy. To further enhance public verifiability and transparency, we utilise a smart contract on a Layer 1 blockchain to verify these zero-knowledge proofs. The careful combination of black box accumulators with selected elliptic curves in the zero-knowledge proofs makes the Boomerang protocol highly efficient. Our proof of concept implementation shows that we can handle up to 23.6 million users per day, on a single-threaded backend server with financial costs of approximately 2 US$. Using the Solana blockchain we can handle 15.5 million users per day with approximate costs of 0.00011 US$ per user. The Boomerang protocol represents a significant advancement in privacy-preserving incentive protocols, laying the groundwork for a more secure and privacy-centric future.

Title: Exploring the Frontiers of LLMs in Psychological Applications: A Comprehensive Review. (arXiv:2401.01519v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.01519
• Code URL: null
• Copy Paste: [[2401.01519]] Exploring the Frontiers of LLMs in Psychological Applications: A Comprehensive Review(http://arxiv.org/abs/2401.01519)
• Summary:

  This paper explores the frontiers of large language models (LLMs) in psychology applications. Psychology has undergone several theoretical changes, and the current use of Artificial Intelligence (AI) and Machine Learning, particularly LLMs, promises to open up new research directions. We provide a detailed exploration of how LLMs like ChatGPT are transforming psychological research. It discusses the impact of LLMs across various branches of psychology, including cognitive and behavioral, clinical and counseling, educational and developmental, and social and cultural psychology, highlighting their potential to simulate aspects of human cognition and behavior. The paper delves into the capabilities of these models to emulate human-like text generation, offering innovative tools for literature review, hypothesis generation, experimental design, experimental subjects, data analysis, academic writing, and peer review in psychology. While LLMs are essential in advancing research methodologies in psychology, the paper also cautions about their technical and ethical challenges. There are issues like data privacy, the ethical implications of using LLMs in psychological research, and the need for a deeper understanding of these models' limitations. Researchers should responsibly use LLMs in psychological studies, adhering to ethical standards and considering the potential consequences of deploying these technologies in sensitive areas. Overall, the article provides a comprehensive overview of the current state of LLMs in psychology, exploring potential benefits and challenges. It serves as a call to action for researchers to leverage LLLs' advantages responsibly while addressing associated risks.

protect

Title: Securing the Digital World: Protecting smart infrastructures and digital industries with Artificial Intelligence (AI)-enabled malware and intrusion detection. (arXiv:2401.01342v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2401.01342
• Code URL: null
• Copy Paste: [[2401.01342]] Securing the Digital World: Protecting smart infrastructures and digital industries with Artificial Intelligence (AI)-enabled malware and intrusion detection(http://arxiv.org/abs/2401.01342)
• Summary:

  The last decades have been characterized by unprecedented technological advances, many of them powered by modern technologies such as Artificial Intelligence (AI) and Machine Learning (ML). The world has become more digitally connected than ever, but we face major challenges. One of the most significant is cybercrime, which has emerged as a global threat to governments, businesses, and civil societies. The pervasiveness of digital technologies combined with a constantly shifting technological foundation has created a complex and powerful playground for cybercriminals, which triggered a surge in demand for intelligent threat detection systems based on machine and deep learning. This paper investigates AI-based cyber threat detection to protect our modern digital ecosystems. The primary focus is on evaluating ML-based classifiers and ensembles for anomaly-based malware detection and network intrusion detection and how to integrate those models in the context of network security, mobile security, and IoT security. The discussion highlights the challenges when deploying and integrating AI-enabled cybersecurity solutions into existing enterprise systems and IT infrastructures, including options to overcome those challenges. Finally, the paper provides future research directions to further increase the security and resilience of our modern digital industries, infrastructures, and ecosystems.

defense

attack

Title: Towards Robust Semantic Segmentation against Patch-based Attack via Attention Refinement. (arXiv:2401.01750v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.01750
• Code URL: null
• Copy Paste: [[2401.01750]] Towards Robust Semantic Segmentation against Patch-based Attack via Attention Refinement(http://arxiv.org/abs/2401.01750)
• Summary:

  The attention mechanism has been proven effective on various visual tasks in recent years. In the semantic segmentation task, the attention mechanism is applied in various methods, including the case of both Convolution Neural Networks (CNN) and Vision Transformer (ViT) as backbones. However, we observe that the attention mechanism is vulnerable to patch-based adversarial attacks. Through the analysis of the effective receptive field, we attribute it to the fact that the wide receptive field brought by global attention may lead to the spread of the adversarial patch. To address this issue, in this paper, we propose a Robust Attention Mechanism (RAM) to improve the robustness of the semantic segmentation model, which can notably relieve the vulnerability against patch-based attacks. Compared to the vallina attention mechanism, RAM introduces two novel modules called Max Attention Suppression and Random Attention Dropout, both of which aim to refine the attention matrix and limit the influence of a single adversarial patch on the semantic segmentation results of other positions. Extensive experiments demonstrate the effectiveness of our RAM to improve the robustness of semantic segmentation models against various patch-based attack methods under different attack settings.

Title: ATLASv2: ATLAS Attack Engagements, Version 2. (arXiv:2401.01341v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2401.01341
• Code URL: null
• Copy Paste: [[2401.01341]] ATLASv2: ATLAS Attack Engagements, Version 2(http://arxiv.org/abs/2401.01341)
• Summary:

  ATLASv2 is based on a previously generated dataset included in "ATLAS: A Sequence-based Learning Approach for Attack Investigation." The original ATLAS dataset is comprised of Windows Security Auditing system logs, Firefox logs, and DNS logs via WireShark. In ATLASv2, we aim to enrich the ATLAS dataset with higher quality background noise and additional logging vantage points. This work replicates the ten attack scenarios described in ATLAS, but extends the logging to include Sysmon logs and events tracked through VMware Carbon Black Cloud.

  The main contribution of ATLASv2 is to improve the quality of the benign system activity and the integration of the attack scenarios. Instead of relying on automated scripts to generate activity, we had two researchers use the victim machines as their primary work stations throughout the course of the engagement. This allowed us to capture system logs on actual user behavior. Additionally, the researchers conducted the attacks in a lab setup allowing the integration of the attack into the work flow of the victim user. This allows the ATLASv2 dataset to provide realistic system logs that mirror the system log activity generated in real-world attacks.

Title: IoTGeM: Generalizable Models for Behaviour-Based IoT Attack Detection. (arXiv:2401.01343v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2401.01343
• Code URL: https://github.com/kahramankostas/IoTGeM
• Copy Paste: [[2401.01343]] IoTGeM: Generalizable Models for Behaviour-Based IoT Attack Detection(http://arxiv.org/abs/2401.01343)
• Summary:

  Previous research on behaviour-based attack detection on networks of IoT devices has resulted in machine learning models whose ability to adapt to unseen data is limited, and often not demonstrated. In this paper we present an approach for modelling IoT network attacks that focuses on generalizability, yet also leads to better detection and performance. First, we present an improved rolling window approach for feature extraction, and introduce a multi-step feature selection process that reduces overfitting. Second, we build and test models using isolated train and test datasets, thereby avoiding common data leaks that have limited the generalizability of previous models. Third, we rigorously evaluate our methodology using a diverse portfolio of machine learning models, evaluation metrics and datasets. Finally, we build confidence in the models by using explainable AI techniques, allowing us to identify the features that underlie accurate detection of attacks.

Title: Does Few-shot Learning Suffer from Backdoor Attacks?. (arXiv:2401.01377v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2401.01377
• Code URL: null
• Copy Paste: [[2401.01377]] Does Few-shot Learning Suffer from Backdoor Attacks?(http://arxiv.org/abs/2401.01377)
• Summary:

  The field of few-shot learning (FSL) has shown promising results in scenarios where training data is limited, but its vulnerability to backdoor attacks remains largely unexplored. We first explore this topic by first evaluating the performance of the existing backdoor attack methods on few-shot learning scenarios. Unlike in standard supervised learning, existing backdoor attack methods failed to perform an effective attack in FSL due to two main issues. Firstly, the model tends to overfit to either benign features or trigger features, causing a tough trade-off between attack success rate and benign accuracy. Secondly, due to the small number of training samples, the dirty label or visible trigger in the support set can be easily detected by victims, which reduces the stealthiness of attacks. It seemed that FSL could survive from backdoor attacks. However, in this paper, we propose the Few-shot Learning Backdoor Attack (FLBA) to show that FSL can still be vulnerable to backdoor attacks. Specifically, we first generate a trigger to maximize the gap between poisoned and benign features. It enables the model to learn both benign and trigger features, which solves the problem of overfitting. To make it more stealthy, we hide the trigger by optimizing two types of imperceptible perturbation, namely attractive and repulsive perturbation, instead of attaching the trigger directly. Once we obtain the perturbations, we can poison all samples in the benign support set into a hidden poisoned support set and fine-tune the model on it. Our method demonstrates a high Attack Success Rate (ASR) in FSL tasks with different few-shot learning paradigms while preserving clean accuracy and maintaining stealthiness. This study reveals that few-shot learning still suffers from backdoor attacks, and its security should be given attention.

Title: Unveiling the Stealthy Threat: Analyzing Slow Drift GPS Spoofing Attacks for Autonomous Vehicles in Urban Environments and Enabling the Resilience. (arXiv:2401.01394v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2401.01394
• Code URL: null
• Copy Paste: [[2401.01394]] Unveiling the Stealthy Threat: Analyzing Slow Drift GPS Spoofing Attacks for Autonomous Vehicles in Urban Environments and Enabling the Resilience(http://arxiv.org/abs/2401.01394)
• Summary:

  Autonomous vehicles (AVs) rely on the Global Positioning System (GPS) or Global Navigation Satellite Systems (GNSS) for precise (Positioning, Navigation, and Timing) PNT solutions. However, the vulnerability of GPS signals to intentional and unintended threats due to their lack of encryption and weak signal strength poses serious risks, thereby reducing the reliability of AVs. GPS spoofing is a complex and damaging attack that deceives AVs by altering GPS receivers to calculate false position and tracking information leading to misdirection. This study explores a stealthy slow drift GPS spoofing attack, replicating the victim AV's satellite reception pattern while changing pseudo ranges to deceive the AV, particularly during turns. The attack is designed to gradually deviate from the correct route, making real-time detection challenging and jeopardizing user safety. We present a system and study methodology for constructing covert spoofing attacks on AVs, investigating the correlation between original and spoofed pseudo ranges to create effective defenses. By closely following the victim vehicle and using the same satellite signals, the attacker executes the attack precisely. Changing the pseudo ranges confuses the AV, leading it to incorrect destinations while remaining oblivious to the manipulation. The gradual deviation from the actual route further conceals the attack, hindering its swift identification. The experiments showcase a robust correlation between the original and spoofed pseudo ranges, with R square values varying between 0.99 and 1. This strong correlation facilitates effective evaluation and mitigation of spoofing signals.

Title: The Art of Deception: Robust Backdoor Attack using Dynamic Stacking of Triggers. (arXiv:2401.01537v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2401.01537
• Code URL: null
• Copy Paste: [[2401.01537]] The Art of Deception: Robust Backdoor Attack using Dynamic Stacking of Triggers(http://arxiv.org/abs/2401.01537)
• Summary:

  The area of Machine Learning as a Service (MLaaS) is experiencing increased implementation due to recent advancements in the AI (Artificial Intelligence) industry. However, this spike has prompted concerns regarding AI defense mechanisms, specifically regarding potential covert attacks from third-party providers that cannot be entirely trusted. Recent research has uncovered that auditory backdoors may use certain modifications as their initiating mechanism. DynamicTrigger is introduced as a methodology for carrying out dynamic backdoor attacks that use cleverly designed tweaks to ensure that corrupted samples are indistinguishable from clean. By utilizing fluctuating signal sampling rates and masking speaker identities through dynamic sound triggers (such as the clapping of hands), it is possible to deceive speech recognition systems (ASR). Our empirical testing demonstrates that DynamicTrigger is both potent and stealthy, achieving impressive success rates during covert attacks while maintaining exceptional accuracy with non-poisoned datasets.

Title: Attackers reveal their arsenal: An investigation of adversarial techniques in CTI reports. (arXiv:2401.01865v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2401.01865
• Code URL: null
• Copy Paste: [[2401.01865]] Attackers reveal their arsenal: An investigation of adversarial techniques in CTI reports(http://arxiv.org/abs/2401.01865)
• Summary:

  Context: Cybersecurity vendors often publish cyber threat intelligence (CTI) reports, referring to the written artifacts on technical and forensic analysis of the techniques used by the malware in APT attacks. Objective: The goal of this research is to inform cybersecurity practitioners about how adversaries form cyberattacks through an analysis of adversarial techniques documented in cyberthreat intelligence reports. Dataset: We use 594 adversarial techniques cataloged in MITRE ATT\&CK. We systematically construct a set of 667 CTI reports that MITRE ATT\&CK used as citations in the descriptions of the cataloged adversarial techniques. Methodology: We analyze the frequency and trend of adversarial techniques, followed by a qualitative analysis of the implementation of techniques. Next, we perform association rule mining to identify pairs of techniques recurring in APT attacks. We then perform qualitative analysis to identify the underlying relations among the techniques in the recurring pairs. Findings: The set of 667 CTI reports documents 10,370 techniques in total, and we identify 19 prevalent techniques accounting for 37.3\% of documented techniques. We also identify 425 statistically significant recurring pairs and seven types of relations among the techniques in these pairs. The top three among the seven relationships suggest that techniques used by the malware inter-relate with one another in terms of (a) abusing or affecting the same system assets, (b) executing in sequences, and (c) overlapping in their implementations. Overall, the study quantifies how adversaries leverage techniques through malware in APT attacks based on publicly reported documents. We advocate organizations prioritize their defense against the identified prevalent techniques and actively hunt for potential malicious intrusion based on the identified pairs of techniques.

Title: Mining Temporal Attack Patterns from Cyberthreat Intelligence Reports. (arXiv:2401.01883v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2401.01883
• Code URL: null
• Copy Paste: [[2401.01883]] Mining Temporal Attack Patterns from Cyberthreat Intelligence Reports(http://arxiv.org/abs/2401.01883)
• Summary:

  Defending from cyberattacks requires practitioners to operate on high-level adversary behavior. Cyberthreat intelligence (CTI) reports on past cyberattack incidents describe the chain of malicious actions with respect to time. To avoid repeating cyberattack incidents, practitioners must proactively identify and defend against recurring chain of actions - which we refer to as temporal attack patterns. Automatically mining the patterns among actions provides structured and actionable information on the adversary behavior of past cyberattacks. The goal of this paper is to aid security practitioners in prioritizing and proactive defense against cyberattacks by mining temporal attack patterns from cyberthreat intelligence reports. To this end, we propose ChronoCTI, an automated pipeline for mining temporal attack patterns from cyberthreat intelligence (CTI) reports of past cyberattacks. To construct ChronoCTI, we build the ground truth dataset of temporal attack patterns and apply state-of-the-art large language models, natural language processing, and machine learning techniques. We apply ChronoCTI on a set of 713 CTI reports, where we identify 124 temporal attack patterns - which we categorize into nine pattern categories. We identify that the most prevalent pattern category is to trick victim users into executing malicious code to initiate the attack, followed by bypassing the anti-malware system in the victim network. Based on the observed patterns, we advocate organizations to train users about cybersecurity best practices, introduce immutable operating systems with limited functionalities, and enforce multi-user authentications. Moreover, we advocate practitioners to leverage the automated mining capability of ChronoCTI and design countermeasures against the recurring attack patterns.

robust

Title: Incorporating Geo-Diverse Knowledge into Prompting for Increased Geographical Robustness in Object Recognition. (arXiv:2401.01482v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.01482
• Code URL: null
• Copy Paste: [[2401.01482]] Incorporating Geo-Diverse Knowledge into Prompting for Increased Geographical Robustness in Object Recognition(http://arxiv.org/abs/2401.01482)
• Summary:

  Existing object recognition models have been shown to lack robustness in diverse geographical scenarios due to significant domain shifts in design and context. Class representations need to be adapted to more accurately reflect an object concept under these shifts. In the absence of training data from target geographies, we hypothesize that geography-specific descriptive knowledge of object categories can be leveraged to enhance robustness. For this purpose, we explore the feasibility of probing a large-language model for geography-specific object knowledge, and we investigate integrating knowledge in zero-shot and learnable soft prompting with the CLIP vision-language model. In particular, we propose a geography knowledge regularization method to ensure that soft prompts trained on a source set of geographies generalize to an unseen target set of geographies. Our gains on DollarStreet when generalizing from a model trained only on data from Europe are as large as +2.8 on countries from Africa, and +4.6 on the hardest classes. We further show competitive performance vs. few-shot target training, and provide insights into how descriptive knowledge captures geographical differences.

Title: DDN-SLAM: Real-time Dense Dynamic Neural Implicit SLAM with Joint Semantic Encoding. (arXiv:2401.01545v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.01545
• Code URL: null
• Copy Paste: [[2401.01545]] DDN-SLAM: Real-time Dense Dynamic Neural Implicit SLAM with Joint Semantic Encoding(http://arxiv.org/abs/2401.01545)
• Summary:

  We propose DDN-SLAM, a real-time dense neural implicit semantic SLAM system designed for dynamic scenes. While existing neural implicit SLAM systems perform well in static scenes, they often encounter challenges in real-world environments with dynamic interferences, leading to ineffective tracking and mapping. DDN-SLAM utilizes the priors provided by the deep semantic system, combined with conditional probability fields, for segmentation.By constructing depth-guided static masks and employing joint multi-resolution hashing encoding, we ensure fast hole filling and high-quality mapping while mitigating the effects of dynamic information interference. To enhance tracking robustness, we utilize sparse feature points validated with optical flow and keyframes, enabling loop closure detection and global bundle optimization. Furthermore, DDN-SLAM supports monocular, stereo, and RGB-D inputs, operating robustly at a frequency of 20-30Hz. Extensive experiments on 6 virtual/real datasets demonstrate that our method outperforms state-of-the-art approaches in both dynamic and static scenes.

Title: De-Confusing Pseudo-Labels in Source-Free Domain Adaptation. (arXiv:2401.01650v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.01650
• Code URL: null
• Copy Paste: [[2401.01650]] De-Confusing Pseudo-Labels in Source-Free Domain Adaptation(http://arxiv.org/abs/2401.01650)
• Summary:

  Source-free domain adaptation (SFDA) aims to transfer knowledge learned from a source domain to an unlabeled target domain, where the source data is unavailable during adaptation. Existing approaches for SFDA focus on self-training usually including well-established entropy minimization and pseudo-labeling techniques. Recent work suggested a co-learning strategy to improve the quality of the generated target pseudo-labels using robust pretrained networks such as Swin-B. However, since the generated pseudo-labels depend on the source model, they may be noisy due to domain shift. In this paper, we view SFDA from the perspective of label noise learning and learn to de-confuse the pseudo-labels. More specifically, we learn a noise transition matrix of the pseudo-labels to capture the label corruption of each class and learn the underlying true label distribution. Estimating the noise transition matrix enables a better true class-posterior estimation results with better prediction accuracy. We demonstrate the effectiveness of our approach applied with several SFDA methods: SHOT, SHOT++, and AaD. We obtain state-of-the-art results on three domain adaptation datasets: VisDA, DomainNet, and OfficeHome.

Title: Few-shot Adaptation of Multi-modal Foundation Models: A Survey. (arXiv:2401.01736v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.01736
• Code URL: null
• Copy Paste: [[2401.01736]] Few-shot Adaptation of Multi-modal Foundation Models: A Survey(http://arxiv.org/abs/2401.01736)
• Summary:

  Multi-modal (vision-language) models, such as CLIP, are replacing traditional supervised pre-training models (e.g., ImageNet-based pre-training) as the new generation of visual foundation models. These models with robust and aligned semantic representations learned from billions of internet image-text pairs and can be applied to various downstream tasks in a zero-shot manner. However, in some fine-grained domains like medical imaging and remote sensing, the performance of multi-modal foundation models often leaves much to be desired. Consequently, many researchers have begun to explore few-shot adaptation methods for these models, gradually deriving three main technical approaches: 1) prompt-based methods, 2) adapter-based methods, and 3) external knowledge-based methods. Nevertheless, this rapidly developing field has produced numerous results without a comprehensive survey to systematically organize the research progress. Therefore, in this survey, we introduce and analyze the research advancements in few-shot adaptation methods for multi-modal models, summarizing commonly used datasets and experimental setups, and comparing the results of different methods. In addition, due to the lack of reliable theoretical support for existing methods, we derive the few-shot adaptation generalization error bound for multi-modal models. The theorem reveals that the generalization error of multi-modal foundation models is constrained by three factors: domain gap, model capacity, and sample size. Based on this, we propose three possible solutions from the following aspects: 1) adaptive domain generalization, 2) adaptive model selection, and 3) adaptive knowledge utilization.

Title: FullLoRA-AT: Efficiently Boosting the Robustness of Pretrained Vision Transformers. (arXiv:2401.01752v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.01752
• Code URL: null
• Copy Paste: [[2401.01752]] FullLoRA-AT: Efficiently Boosting the Robustness of Pretrained Vision Transformers(http://arxiv.org/abs/2401.01752)
• Summary:

  In recent years, the Vision Transformer (ViT) model has gradually become mainstream in various computer vision tasks, and the robustness of the model has received increasing attention. However, existing large models tend to prioritize performance during training, potentially neglecting the robustness, which may lead to serious security concerns. In this paper, we establish a new challenge: exploring how to use a small number of additional parameters for adversarial finetuning to quickly and effectively enhance the adversarial robustness of a standardly trained model. To address this challenge, we develop the novel LNLoRA module, incorporating a learnable layer normalization before the conventional LoRA module, which helps mitigate magnitude differences in parameters between the adversarial and standard training paradigms.

  Furthermore, we propose the FullLoRA-AT framework by integrating the learnable LNLoRA modules into all key components of ViT-based models while keeping the pretrained model frozen, which can significantly improve the model robustness via adversarial finetuning in a parameter-efficient manner.

  Extensive experiments on CIFAR-10, CIFAR-100, and Imagenette demonstrate the superiority of our proposed FullLoRA-AT framework. It achieves comparable robustness with full finetuning while only requiring about 5% of the learnable parameters. This also effectively addresses concerns regarding extra model storage space and enormous training time caused by adversarial finetuning.

Title: LEAP-VO: Long-term Effective Any Point Tracking for Visual Odometry. (arXiv:2401.01887v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.01887
• Code URL: null
• Copy Paste: [[2401.01887]] LEAP-VO: Long-term Effective Any Point Tracking for Visual Odometry(http://arxiv.org/abs/2401.01887)
• Summary:

  Visual odometry estimates the motion of a moving camera based on visual input. Existing methods, mostly focusing on two-view point tracking, often ignore the rich temporal context in the image sequence, thereby overlooking the global motion patterns and providing no assessment of the full trajectory reliability. These shortcomings hinder performance in scenarios with occlusion, dynamic objects, and low-texture areas. To address these challenges, we present the Long-term Effective Any Point Tracking (LEAP) module. LEAP innovatively combines visual, inter-track, and temporal cues with mindfully selected anchors for dynamic track estimation. Moreover, LEAP's temporal probabilistic formulation integrates distribution updates into a learnable iterative refinement module to reason about point-wise uncertainty. Based on these traits, we develop LEAP-VO, a robust visual odometry system adept at handling occlusions and dynamic scenes. Our mindful integration showcases a novel practice by employing long-term point tracking as the front-end. Extensive experiments demonstrate that the proposed pipeline significantly outperforms existing baselines across various visual odometry benchmarks.

Title: MLPs Compass: What is learned when MLPs are combined with PLMs?. (arXiv:2401.01667v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.01667
• Code URL: null
• Copy Paste: [[2401.01667]] MLPs Compass: What is learned when MLPs are combined with PLMs?(http://arxiv.org/abs/2401.01667)
• Summary:

  While Transformer-based pre-trained language models and their variants exhibit strong semantic representation capabilities, the question of comprehending the information gain derived from the additional components of PLMs remains an open question in this field. Motivated by recent efforts that prove Multilayer-Perceptrons (MLPs) modules achieving robust structural capture capabilities, even outperforming Graph Neural Networks (GNNs), this paper aims to quantify whether simple MLPs can further enhance the already potent ability of PLMs to capture linguistic information. Specifically, we design a simple yet effective probing framework containing MLPs components based on BERT structure and conduct extensive experiments encompassing 10 probing tasks spanning three distinct linguistic levels. The experimental results demonstrate that MLPs can indeed enhance the comprehension of linguistic structure by PLMs. Our research provides interpretable and valuable insights into crafting variations of PLMs utilizing MLPs for tasks that emphasize diverse linguistic structures.

Title: EPA: Neural Collapse Inspired Robust Out-of-Distribution Detector. (arXiv:2401.01710v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.01710
• Code URL: null
• Copy Paste: [[2401.01710]] EPA: Neural Collapse Inspired Robust Out-of-Distribution Detector(http://arxiv.org/abs/2401.01710)
• Summary:

  Out-of-distribution (OOD) detection plays a crucial role in ensuring the security of neural networks. Existing works have leveraged the fact that In-distribution (ID) samples form a subspace in the feature space, achieving state-of-the-art (SOTA) performance. However, the comprehensive characteristics of the ID subspace still leave under-explored. Recently, the discovery of Neural Collapse ($\mathcal{NC}$) sheds light on novel properties of the ID subspace. Leveraging insight from $\mathcal{NC}$, we observe that the Principal Angle between the features and the ID feature subspace forms a superior representation for measuring the likelihood of OOD. Building upon this observation, we propose a novel $\mathcal{NC}$-inspired OOD scoring function, named Entropy-enhanced Principal Angle (EPA), which integrates both the global characteristic of the ID subspace and its inner property. We experimentally compare EPA with various SOTA approaches, validating its superior performance and robustness across different network architectures and OOD datasets.

Title: Improved Bandits in Many-to-one Matching Markets with Incentive Compatibility. (arXiv:2401.01528v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.01528
• Code URL: null
• Copy Paste: [[2401.01528]] Improved Bandits in Many-to-one Matching Markets with Incentive Compatibility(http://arxiv.org/abs/2401.01528)
• Summary:

  Two-sided matching markets have been widely studied in the literature due to their rich applications. Since participants are usually uncertain about their preferences, online algorithms have recently been adopted to learn them through iterative interactions. \citet{wang2022bandit} initiate the study of this problem in a many-to-one setting with \textit{responsiveness}. However, their results are far from optimal and lack guarantees of incentive compatibility. An extension of \citet{kong2023player} to this more general setting achieves a near-optimal bound for player-optimal regret. Nevertheless, due to the substantial requirement for collaboration, a single player's deviation could lead to a huge increase in its own cumulative rewards and an $O(T)$ regret for others. In this paper, we aim to enhance the regret bound in many-to-one markets while ensuring incentive compatibility. We first propose the adaptively explore-then-deferred-acceptance (AETDA) algorithm for responsiveness setting and derive an $O(N\min\left\{N,K\right\}C\log T/\Delta^2)$ upper bound for player-optimal stable regret while demonstrating its guarantee of incentive compatibility, where $N$ represents the number of players, $K$ is the number of arms, $T$ denotes the time horizon, $C$ is arms' total capacities and $\Delta$ signifies the minimum preference gap among players. This result is a significant improvement over \citet{wang2022bandit}. And to the best of our knowledge, it constitutes the first player-optimal guarantee in matching markets that offers such robust assurances. We also consider broader \textit{substitutable} preferences, one of the most general conditions to ensure the existence of a stable matching and cover responsiveness. We devise an online DA (ODA) algorithm and establish an $O(NK\log T/\Delta^2)$ player-pessimal stable regret bound for this setting.

biometric

steal

extraction

Title: Enhancing the medical foundation model with multi-scale and cross-modality feature learning. (arXiv:2401.01583v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.01583
• Code URL: null
• Copy Paste: [[2401.01583]] Enhancing the medical foundation model with multi-scale and cross-modality feature learning(http://arxiv.org/abs/2401.01583)
• Summary:

  The development of multi-modal medical foundation models has attracted significant attention in the field of medicine and healthcare due to their promising prospects in various clinical applications. One area of focus in this research direction is the extractions of features at different scales. While previous studies have explored feature learning at individual scales, investigation on integrating the diverse scales and modalities of information is lacking, which may hinder the potential for mutual reinforcement among these features. This paper aims to bridge this gap by proposing a method that effectively exploits multi-scale and cross-modality information to enhance the performance of medical foundation models. The proposed method simultaneously exploit features at the local, instance, modality and global aspects, facilitating comprehensive representation learning within the models. We evaluate the effectiveness of the proposed method on six open-source datasets across different clinical tasks, demonstrating its ability to enhance the performance of medical foundation models.

Title: Question-Answering Based Summarization of Electronic Health Records using Retrieval Augmented Generation. (arXiv:2401.01469v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.01469
• Code URL: null
• Copy Paste: [[2401.01469]] Question-Answering Based Summarization of Electronic Health Records using Retrieval Augmented Generation(http://arxiv.org/abs/2401.01469)
• Summary:

  Summarization of electronic health records (EHRs) can substantially minimize 'screen time' for both patients as well as medical personnel. In recent years summarization of EHRs have employed machine learning pipelines using state of the art neural models. However, these models have produced less than adequate results that are attributed to the difficulty of obtaining sufficient annotated data for training. Moreover, the requirement to consider the entire content of an EHR in summarization has resulted in poor performance due to the fact that attention mechanisms in modern large language models (LLMs) adds a quadratic complexity in terms of the size of the input. We propose here a method that mitigates these shortcomings by combining semantic search, retrieval augmented generation (RAG) and question-answering using the latest LLMs. In our approach summarization is the extraction of answers to specific questions that are deemed important by subject-matter experts (SMEs). Our approach is quite efficient; requires minimal to no training; does not suffer from the 'hallucination' problem of LLMs; and it ensures diversity, since the summary will not have repeated content but diverse answers to specific questions.

Title: Cross-target Stance Detection by Exploiting Target Analytical Perspectives. (arXiv:2401.01761v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.01761
• Code URL: null
• Copy Paste: [[2401.01761]] Cross-target Stance Detection by Exploiting Target Analytical Perspectives(http://arxiv.org/abs/2401.01761)
• Summary:

  Cross-target stance detection (CTSD) is an important task, which infers the attitude of the destination target by utilizing annotated data derived from the source target. One important approach in CTSD is to extract domain-invariant features to bridge the knowledge gap between multiple targets. However, the analysis of informal and short text structure, and implicit expressions, complicate the extraction of domain-invariant knowledge. In this paper, we propose a Multi-Perspective Prompt-Tuning (MPPT) model for CTSD that uses the analysis perspective as a bridge to transfer knowledge. First, we develop a two-stage instruct-based chain-of-thought method (TsCoT) to elicit target analysis perspectives and provide natural language explanations (NLEs) from multiple viewpoints by formulating instructions based on large language model (LLM). Second, we propose a multi-perspective prompt-tuning framework (MultiPLN) to fuse the NLEs into the stance predictor. Extensive experiments results demonstrate the superiority of MPPT against the state-of-the-art baseline methods.

Title: The Adobe Hidden Feature and its Impact on Sensor Attribution. (arXiv:2401.01366v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2401.01366
• Code URL: null
• Copy Paste: [[2401.01366]] The Adobe Hidden Feature and its Impact on Sensor Attribution(http://arxiv.org/abs/2401.01366)
• Summary:

  If the extraction of sensor fingerprints represents nowadays an important forensic tool for sensor attribution, it has been shown recently that images coming from several sensors were more prone to generate False Positives (FP) by presenting a common "leak". In this paper, we investigate the possible cause of this leak and after inspecting the EXIF metadata of the sources causing FP, we found out that they were related to the Adobe Lightroom or Photoshop softwares. The cross-correlation between residuals on images presenting FP reveals periodic peaks showing the presence of a periodic pattern. By developing our own images with Adobe Lightroom we are able to show that all developments from raw images (or 16 bits per channel coded) to 8 bits-coded images also embed a periodic 128x128 pattern very similar to a watermark. However, we also show that the watermark depends on both the content and the architecture used to develop the image. The rest of the paper presents two different ways of removing this watermark, one by removing it from the image noise component, and the other by removing it in the pixel domain. We show that for a camera presenting FP, we were able to prevent the False Positives. A discussion with Adobe representatives informed us that the company decided to add this pattern in order to induce dithering.

Title: Wasserstein Nonnegative Tensor Factorization with Manifold Regularization. (arXiv:2401.01842v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.01842
• Code URL: null
• Copy Paste: [[2401.01842]] Wasserstein Nonnegative Tensor Factorization with Manifold Regularization(http://arxiv.org/abs/2401.01842)
• Summary:

  Nonnegative tensor factorization (NTF) has become an important tool for feature extraction and part-based representation with preserved intrinsic structure information from nonnegative high-order data. However, the original NTF methods utilize Euclidean or Kullback-Leibler divergence as the loss function which treats each feature equally leading to the neglect of the side-information of features. To utilize correlation information of features and manifold information of samples, we introduce Wasserstein manifold nonnegative tensor factorization (WMNTF), which minimizes the Wasserstein distance between the distribution of input tensorial data and the distribution of reconstruction. Although some researches about Wasserstein distance have been proposed in nonnegative matrix factorization (NMF), they ignore the spatial structure information of higher-order data. We use Wasserstein distance (a.k.a Earth Mover's distance or Optimal Transport distance) as a metric and add a graph regularizer to a latent factor. Experimental results demonstrate the effectiveness of the proposed method compared with other NMF and NTF methods.

membership infer

federate

Title: Free Lunch for Federated Remote Sensing Target Fine-Grained Classification: A Parameter-Efficient Framework. (arXiv:2401.01493v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.01493
• Code URL: null
• Copy Paste: [[2401.01493]] Free Lunch for Federated Remote Sensing Target Fine-Grained Classification: A Parameter-Efficient Framework(http://arxiv.org/abs/2401.01493)
• Summary:

  Remote Sensing Target Fine-grained Classification (TFGC) is of great significance in both military and civilian fields. Due to location differences, growth in data size, and centralized server storage constraints, these data are usually stored under different databases across regions/countries. However, privacy laws and national security concerns constrain researchers from accessing these sensitive remote sensing images for further analysis. Additionally, low-resource remote sensing devices encounter challenges in terms of communication overhead and efficiency when dealing with the ever-increasing data and model scales. To solve the above challenges, this paper proposes a novel Privacy-Reserving TFGC Framework based on Federated Learning, dubbed PRFL. The proposed framework allows each client to learn global and local knowledge to enhance the local representation of private data in environments with extreme statistical heterogeneity (non. Independent and Identically Distributed, IID). Thus, it provides highly customized models to clients with differentiated data distributions. Moreover, the framework minimizes communication overhead and improves efficiency while ensuring satisfactory performance, thereby enhancing robustness and practical applicability under resource-scarce conditions. We demonstrate the effectiveness of the proposed PRFL on the classical TFGC task by leveraging four public datasets.

fair

Title: Synthetic Data in AI: Challenges, Applications, and Ethical Implications. (arXiv:2401.01629v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.01629
• Code URL: null
• Copy Paste: [[2401.01629]] Synthetic Data in AI: Challenges, Applications, and Ethical Implications(http://arxiv.org/abs/2401.01629)
• Summary:

  In the rapidly evolving field of artificial intelligence, the creation and utilization of synthetic datasets have become increasingly significant. This report delves into the multifaceted aspects of synthetic data, particularly emphasizing the challenges and potential biases these datasets may harbor. It explores the methodologies behind synthetic data generation, spanning traditional statistical models to advanced deep learning techniques, and examines their applications across diverse domains. The report also critically addresses the ethical considerations and legal implications associated with synthetic datasets, highlighting the urgent need for mechanisms to ensure fairness, mitigate biases, and uphold ethical standards in AI development.

Title: Evaluating Fairness in Self-supervised and Supervised Models for Sequential Data. (arXiv:2401.01640v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.01640
• Code URL: null
• Copy Paste: [[2401.01640]] Evaluating Fairness in Self-supervised and Supervised Models for Sequential Data(http://arxiv.org/abs/2401.01640)
• Summary:

  Self-supervised learning (SSL) has become the de facto training paradigm of large models where pre-training is followed by supervised fine-tuning using domain-specific data and labels. Hypothesizing that SSL models would learn more generic, hence less biased, representations, this study explores the impact of pre-training and fine-tuning strategies on fairness (i.e., performing equally on different demographic breakdowns). Motivated by human-centric applications on real-world timeseries data, we interpret inductive biases on the model, layer, and metric levels by systematically comparing SSL models to their supervised counterparts. Our findings demonstrate that SSL has the capacity to achieve performance on par with supervised methods while significantly enhancing fairness--exhibiting up to a 27% increase in fairness with a mere 1% loss in performance through self-supervision. Ultimately, this work underscores SSL's potential in human-centric computing, particularly high-stakes, data-scarce application domains like healthcare.

interpretability

Title: ProbMCL: Simple Probabilistic Contrastive Learning for Multi-label Visual Classification. (arXiv:2401.01448v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.01448
• Code URL: null
• Copy Paste: [[2401.01448]] ProbMCL: Simple Probabilistic Contrastive Learning for Multi-label Visual Classification(http://arxiv.org/abs/2401.01448)
• Summary:

  Multi-label image classification presents a challenging task in many domains, including computer vision and medical imaging. Recent advancements have introduced graph-based and transformer-based methods to improve performance and capture label dependencies. However, these methods often include complex modules that entail heavy computation and lack interpretability. In this paper, we propose Probabilistic Multi-label Contrastive Learning (ProbMCL), a novel framework to address these challenges in multi-label image classification tasks. Our simple yet effective approach employs supervised contrastive learning, in which samples that share enough labels with an anchor image based on a decision threshold are introduced as a positive set. This structure captures label dependencies by pulling positive pair embeddings together and pushing away negative samples that fall below the threshold. We enhance representation learning by incorporating a mixture density network into contrastive learning and generating Gaussian mixture distributions to explore the epistemic uncertainty of the feature encoder. We validate the effectiveness of our framework through experimentation with datasets from the computer vision and medical imaging domains. Our method outperforms the existing state-of-the-art methods while achieving a low computational footprint on both datasets. Visualization analyses also demonstrate that ProbMCL-learned classifiers maintain a meaningful semantic topology.

Title: Towards Modeling Uncertainties of Self-explaining Neural Networks via Conformal Prediction. (arXiv:2401.01549v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.01549
• Code URL: null
• Copy Paste: [[2401.01549]] Towards Modeling Uncertainties of Self-explaining Neural Networks via Conformal Prediction(http://arxiv.org/abs/2401.01549)
• Summary:

  Despite the recent progress in deep neural networks (DNNs), it remains challenging to explain the predictions made by DNNs. Existing explanation methods for DNNs mainly focus on post-hoc explanations where another explanatory model is employed to provide explanations. The fact that post-hoc methods can fail to reveal the actual original reasoning process of DNNs raises the need to build DNNs with built-in interpretability. Motivated by this, many self-explaining neural networks have been proposed to generate not only accurate predictions but also clear and intuitive insights into why a particular decision was made. However, existing self-explaining networks are limited in providing distribution-free uncertainty quantification for the two simultaneously generated prediction outcomes (i.e., a sample's final prediction and its corresponding explanations for interpreting that prediction). Importantly, they also fail to establish a connection between the confidence values assigned to the generated explanations in the interpretation layer and those allocated to the final predictions in the ultimate prediction layer. To tackle the aforementioned challenges, in this paper, we design a novel uncertainty modeling framework for self-explaining networks, which not only demonstrates strong distribution-free uncertainty modeling performance for the generated explanations in the interpretation layer but also excels in producing efficient and effective prediction sets for the final predictions based on the informative high-level basis explanations. We perform the theoretical analysis for the proposed framework. Extensive experimental evaluation demonstrates the effectiveness of the proposed uncertainty framework.

Title: Signal Processing in the Retina: Interpretable Graph Classifier to Predict Ganglion Cell Responses. (arXiv:2401.01813v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.01813
• Code URL: null
• Copy Paste: [[2401.01813]] Signal Processing in the Retina: Interpretable Graph Classifier to Predict Ganglion Cell Responses(http://arxiv.org/abs/2401.01813)
• Summary:

  It is a popular hypothesis in neuroscience that ganglion cells in the retina are activated by selectively detecting visual features in an observed scene. While ganglion cell firings can be predicted via data-trained deep neural nets, the networks remain indecipherable, thus providing little understanding of the cells' underlying operations. To extract knowledge from the cell firings, in this paper we learn an interpretable graph-based classifier from data to predict the firings of ganglion cells in response to visual stimuli. Specifically, we learn a positive semi-definite (PSD) metric matrix $\mathbf{M} \succeq 0$ that defines Mahalanobis distances between graph nodes (visual events) endowed with pre-computed feature vectors; the computed inter-node distances lead to edge weights and a combinatorial graph that is amenable to binary classification. Mathematically, we define the objective of metric matrix $\mathbf{M}$ optimization using a graph adaptation of large margin nearest neighbor (LMNN), which is rewritten as a semi-definite programming (SDP) problem. We solve it efficiently via a fast approximation called Gershgorin disc perfect alignment (GDPA) linearization. The learned metric matrix $\mathbf{M}$ provides interpretability: important features are identified along $\mathbf{M}$'s diagonal, and their mutual relationships are inferred from off-diagonal terms. Our fast metric learning framework can be applied to other biological systems with pre-chosen features that require interpretation.

explainability

Title: Task and Explanation Network. (arXiv:2401.01732v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.01732
• Code URL: null
• Copy Paste: [[2401.01732]] Task and Explanation Network(http://arxiv.org/abs/2401.01732)
• Summary:

  Explainability in deep networks has gained increased importance in recent years. We argue herein that an AI must be tasked not just with a task but also with an explanation of why said task was accomplished as such. We present a basic framework -- Task and Explanation Network (TENet) -- which fully integrates task completion and its explanation. We believe that the field of AI as a whole should insist -- quite emphatically -- on explainability.

watermark

diffusion

Title: DiffAugment: Diffusion based Long-Tailed Visual Relationship Recognition. (arXiv:2401.01387v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.01387
• Code URL: null
• Copy Paste: [[2401.01387]] DiffAugment: Diffusion based Long-Tailed Visual Relationship Recognition(http://arxiv.org/abs/2401.01387)
• Summary:

  The task of Visual Relationship Recognition (VRR) aims to identify relationships between two interacting objects in an image and is particularly challenging due to the widely-spread and highly imbalanced distribution of <subject, relation, object> triplets. To overcome the resultant performance bias in existing VRR approaches, we introduce DiffAugment -- a method which first augments the tail classes in the linguistic space by making use of WordNet and then utilizes the generative prowess of Diffusion Models to expand the visual space for minority classes. We propose a novel hardness-aware component in diffusion which is based upon the hardness of each <S,R,O> triplet and demonstrate the effectiveness of hardness-aware diffusion in generating visual embeddings for the tail classes. We also propose a novel subject and object based seeding strategy for diffusion sampling which improves the discriminative capability of the generated visual embeddings. Extensive experimentation on the GQA-LT dataset shows favorable gains in the subject/object and relation average per-class accuracy using Diffusion augmented samples.

Title: ColorizeDiffusion: Adjustable Sketch Colorization with Reference Image and Text. (arXiv:2401.01456v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.01456
• Code URL: null
• Copy Paste: [[2401.01456]] ColorizeDiffusion: Adjustable Sketch Colorization with Reference Image and Text(http://arxiv.org/abs/2401.01456)
• Summary:

  Recently, diffusion models have demonstrated their effectiveness in generating extremely high-quality images and have found wide-ranging applications, including automatic sketch colorization. However, most existing models use text to guide the conditional generation, with fewer attempts exploring the potential advantages of using image tokens as conditional inputs for networks. As such, this paper exhaustively investigates image-guided models, specifically targeting reference-based sketch colorization, which aims to colorize sketch images using reference color images. We investigate three critical aspects of reference-based diffusion models: the shortcomings compared to text-based counterparts, the training strategies, and the capability in zero-shot, sequential text-based manipulation. We introduce two variations of an image-guided latent diffusion model using different image tokens from the pre-trained CLIP image encoder, and we propose corresponding manipulation methods to adjust their results sequentially using weighted text inputs. We conduct comprehensive evaluations of our models through qualitative and quantitative experiments, as well as a user study.

Title: S$^{2}$-DMs:Skip-Step Diffusion Models. (arXiv:2401.01520v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.01520
• Code URL: https://github.com/kingkingofall/skip-step-diffusion
• Copy Paste: [[2401.01520]] S$^{2}$-DMs:Skip-Step Diffusion Models(http://arxiv.org/abs/2401.01520)
• Summary:

  Diffusion models have emerged as powerful generative tools, rivaling GANs in sample quality and mirroring the likelihood scores of autoregressive models. A subset of these models, exemplified by DDIMs, exhibit an inherent asymmetry: they are trained over $T$ steps but only sample from a subset of $T$ during generation. This selective sampling approach, though optimized for speed, inadvertently misses out on vital information from the unsampled steps, leading to potential compromises in sample quality. To address this issue, we present the S$^{2}$-DMs, which is a new training method by using an innovative $L_{skip}$, meticulously designed to reintegrate the information omitted during the selective sampling phase. The benefits of this approach are manifold: it notably enhances sample quality, is exceptionally simple to implement, requires minimal code modifications, and is flexible enough to be compatible with various sampling algorithms. On the CIFAR10 dataset, models trained using our algorithm showed an improvement of 3.27% to 14.06% over models trained with traditional methods across various sampling algorithms (DDIMs, PNDMs, DEIS) and different numbers of sampling steps (10, 20, ..., 1000). On the CELEBA dataset, the improvement ranged from 8.97% to 27.08%. Access to the code and additional resources is provided in the github.

Title: SIGNeRF: Scene Integrated Generation for Neural Radiance Fields. (arXiv:2401.01647v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.01647
• Code URL: null
• Copy Paste: [[2401.01647]] SIGNeRF: Scene Integrated Generation for Neural Radiance Fields(http://arxiv.org/abs/2401.01647)
• Summary:

  Advances in image diffusion models have recently led to notable improvements in the generation of high-quality images. In combination with Neural Radiance Fields (NeRFs), they enabled new opportunities in 3D generation. However, most generative 3D approaches are object-centric and applying them to editing existing photorealistic scenes is not trivial. We propose SIGNeRF, a novel approach for fast and controllable NeRF scene editing and scene-integrated object generation. A new generative update strategy ensures 3D consistency across the edited images, without requiring iterative optimization. We find that depth-conditioned diffusion models inherently possess the capability to generate 3D consistent views by requesting a grid of images instead of single views. Based on these insights, we introduce a multi-view reference sheet of modified images. Our method updates an image collection consistently based on the reference sheet and refines the original NeRF with the newly generated image set in one go. By exploiting the depth conditioning mechanism of the image diffusion model, we gain fine control over the spatial location of the edit and enforce shape guidance by a selected region or an external mesh.

Title: DiffYOLO: Object Detection for Anti-Noise via YOLO and Diffusion Models. (arXiv:2401.01659v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.01659
• Code URL: null
• Copy Paste: [[2401.01659]] DiffYOLO: Object Detection for Anti-Noise via YOLO and Diffusion Models(http://arxiv.org/abs/2401.01659)
• Summary:

  Object detection models represented by YOLO series have been widely used and have achieved great results on the high quality datasets, but not all the working conditions are ideal. To settle down the problem of locating targets on low quality datasets, the existing methods either train a new object detection network, or need a large collection of low-quality datasets to train. However, we propose a framework in this paper and apply it on the YOLO models called DiffYOLO. Specifically, we extract feature maps from the denoising diffusion probabilistic models to enhance the well-trained models, which allows us fine-tune YOLO on high-quality datasets and test on low-quality datasets. The results proved this framework can not only prove the performance on noisy datasets, but also prove the detection results on high-quality test datasets. We will supplement more experiments later (with various datasets and network architectures).

Title: Simultaneous q-Space Sampling Optimization and Reconstruction for Fast and High-fidelity Diffusion Magnetic Resonance Imaging. (arXiv:2401.01662v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.01662
• Code URL: null
• Copy Paste: [[2401.01662]] Simultaneous q-Space Sampling Optimization and Reconstruction for Fast and High-fidelity Diffusion Magnetic Resonance Imaging(http://arxiv.org/abs/2401.01662)
• Summary:

  Diffusion Magnetic Resonance Imaging (dMRI) plays a crucial role in the noninvasive investigation of tissue microstructural properties and structural connectivity in the \textit{in vivo} human brain. However, to effectively capture the intricate characteristics of water diffusion at various directions and scales, it is important to employ comprehensive q-space sampling. Unfortunately, this requirement leads to long scan times, limiting the clinical applicability of dMRI. To address this challenge, we propose SSOR, a Simultaneous q-Space sampling Optimization and Reconstruction framework. We jointly optimize a subset of q-space samples using a continuous representation of spherical harmonic functions and a reconstruction network. Additionally, we integrate the unique properties of diffusion magnetic resonance imaging (dMRI) in both the q-space and image domains by applying $l1$-norm and total-variation regularization. The experiments conducted on HCP data demonstrate that SSOR has promising strengths both quantitatively and qualitatively and exhibits robustness to noise.

Title: AID-DTI: Accelerating High-fidelity Diffusion Tensor Imaging with Detail-Preserving Model-based Deep Learning. (arXiv:2401.01693v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.01693
• Code URL: null
• Copy Paste: [[2401.01693]] AID-DTI: Accelerating High-fidelity Diffusion Tensor Imaging with Detail-Preserving Model-based Deep Learning(http://arxiv.org/abs/2401.01693)
• Summary:

  Deep learning has shown great potential in accelerating diffusion tensor imaging (DTI). Nevertheless, existing methods tend to suffer from Rician noise and detail loss in reconstructing the DTI-derived parametric maps especially when sparsely sampled q-space data are used. This paper proposes a novel method, AID-DTI (Accelerating hIgh fiDelity Diffusion Tensor Imaging), to facilitate fast and accurate DTI with only six measurements. AID-DTI is equipped with a newly designed Singular Value Decomposition (SVD)-based regularizer, which can effectively capture fine details while suppressing noise during network training. Experimental results on Human Connectome Project (HCP) data consistently demonstrate that the proposed method estimates DTI parameter maps with fine-grained details and outperforms three state-of-the-art methods both quantitatively and qualitatively.

Title: aMUSEd: An Open MUSE Reproduction. (arXiv:2401.01808v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.01808
• Code URL: null
• Copy Paste: [[2401.01808]] aMUSEd: An Open MUSE Reproduction(http://arxiv.org/abs/2401.01808)
• Summary:

  We present aMUSEd, an open-source, lightweight masked image model (MIM) for text-to-image generation based on MUSE. With 10 percent of MUSE's parameters, aMUSEd is focused on fast image generation. We believe MIM is under-explored compared to latent diffusion, the prevailing approach for text-to-image generation. Compared to latent diffusion, MIM requires fewer inference steps and is more interpretable. Additionally, MIM can be fine-tuned to learn additional styles with only a single image. We hope to encourage further exploration of MIM by demonstrating its effectiveness on large-scale text-to-image generation and releasing reproducible training code. We also release checkpoints for two models which directly produce images at 256x256 and 512x512 resolutions.

Title: Moonshot: Towards Controllable Video Generation and Editing with Multimodal Conditions. (arXiv:2401.01827v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.01827
• Code URL: https://github.com/salesforce/lavis
• Copy Paste: [[2401.01827]] Moonshot: Towards Controllable Video Generation and Editing with Multimodal Conditions(http://arxiv.org/abs/2401.01827)
• Summary:

  Most existing video diffusion models (VDMs) are limited to mere text conditions. Thereby, they are usually lacking in control over visual appearance and geometry structure of the generated videos. This work presents Moonshot, a new video generation model that conditions simultaneously on multimodal inputs of image and text. The model builts upon a core module, called multimodal video block (MVB), which consists of conventional spatialtemporal layers for representing video features, and a decoupled cross-attention layer to address image and text inputs for appearance conditioning. In addition, we carefully design the model architecture such that it can optionally integrate with pre-trained image ControlNet modules for geometry visual conditions, without needing of extra training overhead as opposed to prior methods. Experiments show that with versatile multimodal conditioning mechanisms, Moonshot demonstrates significant improvement on visual quality and temporal consistency compared to existing models. In addition, the model can be easily repurposed for a variety of generative applications, such as personalized video generation, image animation and video editing, unveiling its potential to serve as a fundamental architecture for controllable video generation. Models will be made public on https://github.com/salesforce/LAVIS.

Title: From Audio to Photoreal Embodiment: Synthesizing Humans in Conversations. (arXiv:2401.01885v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.01885
• Code URL: null
• Copy Paste: [[2401.01885]] From Audio to Photoreal Embodiment: Synthesizing Humans in Conversations(http://arxiv.org/abs/2401.01885)
• Summary:

  We present a framework for generating full-bodied photorealistic avatars that gesture according to the conversational dynamics of a dyadic interaction. Given speech audio, we output multiple possibilities of gestural motion for an individual, including face, body, and hands. The key behind our method is in combining the benefits of sample diversity from vector quantization with the high-frequency details obtained through diffusion to generate more dynamic, expressive motion. We visualize the generated motion using highly photorealistic avatars that can express crucial nuances in gestures (e.g. sneers and smirks). To facilitate this line of research, we introduce a first-of-its-kind multi-view conversational dataset that allows for photorealistic reconstruction. Experiments show our model generates appropriate and diverse gestures, outperforming both diffusion- and VQ-only methods. Furthermore, our perceptual evaluation highlights the importance of photorealism (vs. meshes) in accurately assessing subtle motion details in conversational gestures. Code and dataset available online.

Title: DGDNN: Decoupled Graph Diffusion Neural Network for Stock Movement Prediction. (arXiv:2401.01846v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.01846
• Code URL: null
• Copy Paste: [[2401.01846]] DGDNN: Decoupled Graph Diffusion Neural Network for Stock Movement Prediction(http://arxiv.org/abs/2401.01846)
• Summary:

  Forecasting future stock trends remains challenging for academia and industry due to stochastic inter-stock dynamics and hierarchical intra-stock dynamics influencing stock prices. In recent years, graph neural networks have achieved remarkable performance in this problem by formulating multiple stocks as graph-structured data. However, most of these approaches rely on artificially defined factors to construct static stock graphs, which fail to capture the intrinsic interdependencies between stocks that rapidly evolve. In addition, these methods often ignore the hierarchical features of the stocks and lose distinctive information within. In this work, we propose a novel graph learning approach implemented without expert knowledge to address these issues. First, our approach automatically constructs dynamic stock graphs by entropy-driven edge generation from a signal processing perspective. Then, we further learn task-optimal dependencies between stocks via a generalized graph diffusion process on constructed stock graphs. Last, a decoupled representation learning scheme is adopted to capture distinctive hierarchical intra-stock features. Experimental results demonstrate substantial improvements over state-of-the-art baselines on real-world datasets. Moreover, the ablation study and sensitivity study further illustrate the effectiveness of the proposed method in modeling the time-evolving inter-stock and intra-stock dynamics.

noise learning

data-free

transformer

Title: Token Propagation Controller for Efficient Vision Transformer. (arXiv:2401.01470v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.01470
• Code URL: null
• Copy Paste: [[2401.01470]] Token Propagation Controller for Efficient Vision Transformer(http://arxiv.org/abs/2401.01470)
• Summary:

  Vision transformers (ViTs) have achieved promising results on a variety of Computer Vision tasks, however their quadratic complexity in the number of input tokens has limited their application specially in resource-constrained settings. Previous approaches that employ gradual token reduction to address this challenge assume that token redundancy in one layer implies redundancy in all the following layers. We empirically demonstrate that this assumption is often not correct, i.e., tokens that are redundant in one layer can be useful in later layers. We employ this key insight to propose a novel token propagation controller (TPC) that incorporates two different token-distributions, i.e., pause probability and restart probability to control the reduction and reuse of tokens respectively, which results in more efficient token utilization. To improve the estimates of token distributions, we propose a smoothing mechanism that acts as a regularizer and helps remove noisy outliers. Furthermore, to improve the training-stability of our proposed TPC, we introduce a model stabilizer that is able to implicitly encode local image structures and minimize accuracy fluctuations during model training. We present extensive experimental results on the ImageNet-1K dataset using DeiT, LV-ViT and Swin models to demonstrate the effectiveness of our proposed method. For example, compared to baseline models, our proposed method improves the inference speed of the DeiT-S by 250% while increasing the classification accuracy by 1.0%.

Title: Sports-QA: A Large-Scale Video Question Answering Benchmark for Complex and Professional Sports. (arXiv:2401.01505v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.01505
• Code URL: null
• Copy Paste: [[2401.01505]] Sports-QA: A Large-Scale Video Question Answering Benchmark for Complex and Professional Sports(http://arxiv.org/abs/2401.01505)
• Summary:

  Reasoning over sports videos for question answering is an important task with numerous applications, such as player training and information retrieval. However, this task has not been explored due to the lack of relevant datasets and the challenging nature it presents. Most datasets for video question answering (VideoQA) focus mainly on general and coarse-grained understanding of daily-life videos, which is not applicable to sports scenarios requiring professional action understanding and fine-grained motion analysis. In this paper, we introduce the first dataset, named Sports-QA, specifically designed for the sports VideoQA task. The Sports-QA dataset includes various types of questions, such as descriptions, chronologies, causalities, and counterfactual conditions, covering multiple sports. Furthermore, to address the characteristics of the sports VideoQA task, we propose a new Auto-Focus Transformer (AFT) capable of automatically focusing on particular scales of temporal information for question answering. We conduct extensive experiments on Sports-QA, including baseline studies and the evaluation of different methods. The results demonstrate that our AFT achieves state-of-the-art performance.

Title: CRA-PCN: Point Cloud Completion with Intra- and Inter-level Cross-Resolution Transformers. (arXiv:2401.01552v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.01552
• Code URL: https://github.com/easyry/cra-pcn
• Copy Paste: [[2401.01552]] CRA-PCN: Point Cloud Completion with Intra- and Inter-level Cross-Resolution Transformers(http://arxiv.org/abs/2401.01552)
• Summary:

  Point cloud completion is an indispensable task for recovering complete point clouds due to incompleteness caused by occlusion, limited sensor resolution, etc. The family of coarse-to-fine generation architectures has recently exhibited great success in point cloud completion and gradually became mainstream. In this work, we unveil one of the key ingredients behind these methods: meticulously devised feature extraction operations with explicit cross-resolution aggregation. We present Cross-Resolution Transformer that efficiently performs cross-resolution aggregation with local attention mechanisms. With the help of our recursive designs, the proposed operation can capture more scales of features than common aggregation operations, which is beneficial for capturing fine geometric characteristics. While prior methodologies have ventured into various manifestations of inter-level cross-resolution aggregation, the effectiveness of intra-level one and their combination has not been analyzed. With unified designs, Cross-Resolution Transformer can perform intra- or inter-level cross-resolution aggregation by switching inputs. We integrate two forms of Cross-Resolution Transformers into one up-sampling block for point generation, and following the coarse-to-fine manner, we construct CRA-PCN to incrementally predict complete shapes with stacked up-sampling blocks. Extensive experiments demonstrate that our method outperforms state-of-the-art methods by a large margin on several widely used benchmarks. Codes are available at https://github.com/EasyRy/CRA-PCN.

Title: A Transformer-Based Adaptive Semantic Aggregation Method for UAV Visual Geo-Localization. (arXiv:2401.01574v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.01574
• Code URL: null
• Copy Paste: [[2401.01574]] A Transformer-Based Adaptive Semantic Aggregation Method for UAV Visual Geo-Localization(http://arxiv.org/abs/2401.01574)
• Summary:

  This paper addresses the task of Unmanned Aerial Vehicles (UAV) visual geo-localization, which aims to match images of the same geographic target taken by different platforms, i.e., UAVs and satellites. In general, the key to achieving accurate UAV-satellite image matching lies in extracting visual features that are robust against viewpoint changes, scale variations, and rotations. Current works have shown that part matching is crucial for UAV visual geo-localization since part-level representations can capture image details and help to understand the semantic information of scenes. However, the importance of preserving semantic characteristics in part-level representations is not well discussed. In this paper, we introduce a transformer-based adaptive semantic aggregation method that regards parts as the most representative semantics in an image. Correlations of image patches to different parts are learned in terms of the transformer's feature map. Then our method decomposes part-level features into an adaptive sum of all patch features. By doing this, the learned parts are encouraged to focus on patches with typical semantics. Extensive experiments on the University-1652 dataset have shown the superiority of our method over the current works.

Title: Context-Guided Spatio-Temporal Video Grounding. (arXiv:2401.01578v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.01578
• Code URL: https://github.com/henglan/cgstvg
• Copy Paste: [[2401.01578]] Context-Guided Spatio-Temporal Video Grounding(http://arxiv.org/abs/2401.01578)
• Summary:

  Spatio-temporal video grounding (or STVG) task aims at locating a spatio-temporal tube for a specific instance given a text query. Despite advancements, current methods easily suffer the distractors or heavy object appearance variations in videos due to insufficient object information from the text, leading to degradation. Addressing this, we propose a novel framework, context-guided STVG (CG-STVG), which mines discriminative instance context for object in videos and applies it as a supplementary guidance for target localization. The key of CG-STVG lies in two specially designed modules, including instance context generation (ICG), which focuses on discovering visual context information (in both appearance and motion) of the instance, and instance context refinement (ICR), which aims to improve the instance context from ICG by eliminating irrelevant or even harmful information from the context. During grounding, ICG, together with ICR, are deployed at each decoding stage of a Transformer architecture for instance context learning. Particularly, instance context learned from one decoding stage is fed to the next stage, and leveraged as a guidance containing rich and discriminative object feature to enhance the target-awareness in decoding feature, which conversely benefits generating better new instance context for improving localization finally. Compared to existing methods, CG-STVG enjoys object information in text query and guidance from mined instance visual context for more accurate target localization. In our experiments on three benchmarks, including HCSTVG-v1/-v2 and VidSTG, CG-STVG sets new state-of-the-arts in m_tIoU and m_vIoU on all of them, showing its efficacy. The code will be released at https://github.com/HengLan/CGSTVG.

Title: Transformer RGBT Tracking with Spatio-Temporal Multimodal Tokens. (arXiv:2401.01674v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.01674
• Code URL: null
• Copy Paste: [[2401.01674]] Transformer RGBT Tracking with Spatio-Temporal Multimodal Tokens(http://arxiv.org/abs/2401.01674)
• Summary:

  Many RGBT tracking researches primarily focus on modal fusion design, while overlooking the effective handling of target appearance changes. While some approaches have introduced historical frames or fuse and replace initial templates to incorporate temporal information, they have the risk of disrupting the original target appearance and accumulating errors over time. To alleviate these limitations, we propose a novel Transformer RGBT tracking approach, which mixes spatio-temporal multimodal tokens from the static multimodal templates and multimodal search regions in Transformer to handle target appearance changes, for robust RGBT tracking. We introduce independent dynamic template tokens to interact with the search region, embedding temporal information to address appearance changes, while also retaining the involvement of the initial static template tokens in the joint feature extraction process to ensure the preservation of the original reliable target appearance information that prevent deviations from the target appearance caused by traditional temporal updates. We also use attention mechanisms to enhance the target features of multimodal template tokens by incorporating supplementary modal cues, and make the multimodal search region tokens interact with multimodal dynamic template tokens via attention mechanisms, which facilitates the conveyance of multimodal-enhanced target change information. Our module is inserted into the transformer backbone network and inherits joint feature extraction, search-template matching, and cross-modal interaction. Extensive experiments on three RGBT benchmark datasets show that the proposed approach maintains competitive performance compared to other state-of-the-art tracking algorithms while running at 39.1 FPS.

Title: Iterative Mask Filling: An Effective Text Augmentation Method Using Masked Language Modeling. (arXiv:2401.01830v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.01830
• Code URL: null
• Copy Paste: [[2401.01830]] Iterative Mask Filling: An Effective Text Augmentation Method Using Masked Language Modeling(http://arxiv.org/abs/2401.01830)
• Summary:

  Data augmentation is an effective technique for improving the performance of machine learning models. However, it has not been explored as extensively in natural language processing (NLP) as it has in computer vision. In this paper, we propose a novel text augmentation method that leverages the Fill-Mask feature of the transformer-based BERT model. Our method involves iteratively masking words in a sentence and replacing them with language model predictions. We have tested our proposed method on various NLP tasks and found it to be effective in many cases. Our results are presented along with a comparison to existing augmentation methods. Experimental results show that our proposed method significantly improves performance, especially on topic classification datasets.

Title: Kernel-U-Net: Hierarchical and Symmetrical Framework for Multivariate Time Series Forecasting. (arXiv:2401.01479v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.01479
• Code URL: null
• Copy Paste: [[2401.01479]] Kernel-U-Net: Hierarchical and Symmetrical Framework for Multivariate Time Series Forecasting(http://arxiv.org/abs/2401.01479)
• Summary:

  Time series forecasting task predicts future trends based on historical information. Recent U-Net-based methods have demonstrated superior performance in predicting real-world datasets. However, the performance of these models is lower than patch-based models or linear models. In this work, we propose a symmetric and hierarchical framework, Kernel-U-Net, which cuts the input sequence into slices at each layer of the network and then computes them using kernels. Furthermore, it generalizes the concept of convolutional kernels in classic U-Net to accept custom kernels that follow the same design pattern. Compared to the existing linear or transformer-based solution, our model contains 3 advantages: 1) A small number of parameters: the parameters size is $O(log(L)^2)$ where $L$ is the look-back window size, 2) Flexibility: its kernels can be customized and fitted to the datasets, 3) Computation efficiency: the computation complexity of transformer modules is reduced to $O(log(L)^2)$ if they are placed close to the latent vector. Kernel-U-Net accuracy was greater than or equal to the state-of-the-art model on six (out of seven) real-world datasets.

Title: Transformer Neural Autoregressive Flows. (arXiv:2401.01855v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.01855
• Code URL: null
• Copy Paste: [[2401.01855]] Transformer Neural Autoregressive Flows(http://arxiv.org/abs/2401.01855)
• Summary:

  Density estimation, a central problem in machine learning, can be performed using Normalizing Flows (NFs). NFs comprise a sequence of invertible transformations, that turn a complex target distribution into a simple one, by exploiting the change of variables theorem. Neural Autoregressive Flows (NAFs) and Block Neural Autoregressive Flows (B-NAFs) are arguably the most perfomant members of the NF family. However, they suffer scalability issues and training instability due to the constraints imposed on the network structure. In this paper, we propose a novel solution to these challenges by exploiting transformers to define a new class of neural flows called Transformer Neural Autoregressive Flows (T-NAFs). T-NAFs treat each dimension of a random variable as a separate input token, using attention masking to enforce an autoregressive constraint. We take an amortization-inspired approach where the transformer outputs the parameters of an invertible transformation. The experimental results demonstrate that T-NAFs consistently match or outperform NAFs and B-NAFs across multiple datasets from the UCI benchmark. Remarkably, T-NAFs achieve these results using an order of magnitude fewer parameters than previous approaches, without composing multiple flows.

generative

Title: Few-shot Image Generation via Information Transfer from the Built Geodesic Surface. (arXiv:2401.01749v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.01749
• Code URL: null
• Copy Paste: [[2401.01749]] Few-shot Image Generation via Information Transfer from the Built Geodesic Surface(http://arxiv.org/abs/2401.01749)
• Summary:

  Images generated by most of generative models trained with limited data often exhibit deficiencies in either fidelity, diversity, or both. One effective solution to address the limitation is few-shot generative model adaption. However, the type of approaches typically rely on a large-scale pre-trained model, serving as a source domain, to facilitate information transfer to the target domain. In this paper, we propose a method called Information Transfer from the Built Geodesic Surface (ITBGS), which contains two module: Feature Augmentation on Geodesic Surface (FAGS); Interpolation and Regularization (I\&R). With the FAGS module, a pseudo-source domain is created by projecting image features from the training dataset into the Pre-Shape Space, subsequently generating new features on the Geodesic surface. Thus, no pre-trained models is needed for the adaption process during the training of generative models with FAGS. I\&R module are introduced for supervising the interpolated images and regularizing their relative distances, respectively, to further enhance the quality of generated images. Through qualitative and quantitative experiments, we demonstrate that the proposed method consistently achieves optimal or comparable results across a diverse range of semantically distinct datasets, even in extremely few-shot scenarios.

Title: Physio: An LLM-Based Physiotherapy Advisor. (arXiv:2401.01825v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.01825
• Code URL: null
• Copy Paste: [[2401.01825]] Physio: An LLM-Based Physiotherapy Advisor(http://arxiv.org/abs/2401.01825)
• Summary:

  The capabilities of the most recent language models have increased the interest in integrating them into real-world applications. However, the fact that these models generate plausible, yet incorrect text poses a constraint when considering their use in several domains. Healthcare is a prime example of a domain where text-generative trustworthiness is a hard requirement to safeguard patient well-being. In this paper, we present Physio, a chat-based application for physical rehabilitation. Physio is capable of making an initial diagnosis while citing reliable health sources to support the information provided. Furthermore, drawing upon external knowledge databases, Physio can recommend rehabilitation exercises and over-the-counter medication for symptom relief. By combining these features, Physio can leverage the power of generative models for language processing while also conditioning its response on dependable and verifiable sources. A live demo of Physio is available at https://physio.inesctec.pt.

Title: Theoretical guarantees on the best-of-n alignment policy. (arXiv:2401.01879v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.01879
• Code URL: null
• Copy Paste: [[2401.01879]] Theoretical guarantees on the best-of-n alignment policy(http://arxiv.org/abs/2401.01879)
• Summary:

  A simple and effective method for the alignment of generative models is the best-of-$n$ policy, where $n$ samples are drawn from a base policy, and ranked based on a reward function, and the highest ranking one is selected. A commonly used analytical expression in the literature claims that the KL divergence between the best-of-$n$ policy and the base policy is equal to $\log (n) - (n-1)/n.$ We disprove the validity of this claim, and show that it is an upper bound on the actual KL divergence. We also explore the tightness of this upper bound in different regimes. Finally, we propose a new estimator for the KL divergence and empirically show that it provides a tight approximation through a few examples.

Title: Modular Learning of Deep Causal Generative Models for High-dimensional Causal Inference. (arXiv:2401.01426v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.01426
• Code URL: null
• Copy Paste: [[2401.01426]] Modular Learning of Deep Causal Generative Models for High-dimensional Causal Inference(http://arxiv.org/abs/2401.01426)
• Summary:

  Pearl's causal hierarchy establishes a clear separation between observational, interventional, and counterfactual questions. Researchers proposed sound and complete algorithms to compute identifiable causal queries at a given level of the hierarchy using the causal structure and data from the lower levels of the hierarchy. However, most of these algorithms assume that we can accurately estimate the probability distribution of the data, which is an impractical assumption for high-dimensional variables such as images. On the other hand, modern generative deep learning architectures can be trained to learn how to accurately sample from such high-dimensional distributions. Especially with the recent rise of foundation models for images, it is desirable to leverage pre-trained models to answer causal queries with such high-dimensional data. To address this, we propose a sequential training algorithm that, given the causal structure and a pre-trained conditional generative model, can train a deep causal generative model, which utilizes the pre-trained model and can provably sample from identifiable interventional and counterfactual distributions. Our algorithm, called Modular-DCM, uses adversarial training to learn the network weights, and to the best of our knowledge, is the first algorithm that can make use of pre-trained models and provably sample from any identifiable causal query in the presence of latent confounders with high-dimensional data. We demonstrate the utility of our algorithm using semi-synthetic and real-world datasets containing images as variables in the causal structure.

Title: Towards a Foundation Purchasing Model: Pretrained Generative Autoregression on Transaction Sequences. (arXiv:2401.01641v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.01641
• Code URL: https://github.com/featurespace/foundation-model-paper
• Copy Paste: [[2401.01641]] Towards a Foundation Purchasing Model: Pretrained Generative Autoregression on Transaction Sequences(http://arxiv.org/abs/2401.01641)
• Summary:

  Machine learning models underpin many modern financial systems for use cases such as fraud detection and churn prediction. Most are based on supervised learning with hand-engineered features, which relies heavily on the availability of labelled data. Large self-supervised generative models have shown tremendous success in natural language processing and computer vision, yet so far they haven't been adapted to multivariate time series of financial transactions. In this paper, we present a generative pretraining method that can be used to obtain contextualised embeddings of financial transactions. Benchmarks on public datasets demonstrate that it outperforms state-of-the-art self-supervised methods on a range of downstream tasks. We additionally perform large-scale pretraining of an embedding model using a corpus of data from 180 issuing banks containing 5.1 billion transactions and apply it to the card fraud detection problem on hold-out datasets. The embedding model significantly improves value detection rate at high precision thresholds and transfers well to out-of-domain distributions.

large language model

Title: WordArt Designer API: User-Driven Artistic Typography Synthesis with Large Language Models on ModelScope. (arXiv:2401.01699v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.01699
• Code URL: null
• Copy Paste: [[2401.01699]] WordArt Designer API: User-Driven Artistic Typography Synthesis with Large Language Models on ModelScope(http://arxiv.org/abs/2401.01699)
• Summary:

  This paper introduces the WordArt Designer API, a novel framework for user-driven artistic typography synthesis utilizing Large Language Models (LLMs) on ModelScope. We address the challenge of simplifying artistic typography for non-professionals by offering a dynamic, adaptive, and computationally efficient alternative to traditional rigid templates. Our approach leverages the power of LLMs to understand and interpret user input, facilitating a more intuitive design process. We demonstrate through various case studies how users can articulate their aesthetic preferences and functional requirements, which the system then translates into unique and creative typographic designs. Our evaluations indicate significant improvements in user satisfaction, design flexibility, and creative expression over existing systems. The WordArt Designer API not only democratizes the art of typography but also opens up new possibilities for personalized digital communication and design.

Title: Quantifying the Uniqueness of Donald Trump in Presidential Discourse. (arXiv:2401.01405v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.01405
• Code URL: null
• Copy Paste: [[2401.01405]] Quantifying the Uniqueness of Donald Trump in Presidential Discourse(http://arxiv.org/abs/2401.01405)
• Summary:

  Does Donald Trump speak differently from other presidents? If so, in what ways? Are these differences confined to any single medium of communication? To investigate these questions, this paper introduces a novel metric of uniqueness based on large language models, develops a new lexicon for divisive speech, and presents a framework for comparing the lexical features of political opponents. Applying these tools to a variety of corpora of presidential speeches, we find considerable evidence that Trump's speech patterns diverge from those of all major party nominees for the presidency in recent history. Some notable findings include Trump's employment of particularly divisive and antagonistic language targeting of his political opponents and his patterns of repetition for emphasis. Furthermore, Trump is significantly more distinctive than his fellow Republicans, whose uniqueness values are comparably closer to those of the Democrats. These differences hold across a variety of measurement strategies, arise on both the campaign trail and in official presidential addresses, and do not appear to be an artifact of secular time trends.

Title: PLLaMa: An Open-source Large Language Model for Plant Science. (arXiv:2401.01600v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.01600
• Code URL: null
• Copy Paste: [[2401.01600]] PLLaMa: An Open-source Large Language Model for Plant Science(http://arxiv.org/abs/2401.01600)
• Summary:

  Large Language Models (LLMs) have exhibited remarkable capabilities in understanding and interacting with natural language across various sectors. However, their effectiveness is limited in specialized areas requiring high accuracy, such as plant science, due to a lack of specific expertise in these fields. This paper introduces PLLaMa, an open-source language model that evolved from LLaMa-2. It's enhanced with a comprehensive database, comprising more than 1.5 million scholarly articles in plant science. This development significantly enriches PLLaMa with extensive knowledge and proficiency in plant and agricultural sciences. Our initial tests, involving specific datasets related to plants and agriculture, show that PLLaMa substantially improves its understanding of plant science-related topics. Moreover, we have formed an international panel of professionals, including plant scientists, agricultural engineers, and plant breeders. This team plays a crucial role in verifying the accuracy of PLLaMa's responses to various academic inquiries, ensuring its effective and reliable application in the field. To support further research and development, we have made the model's checkpoints and source codes accessible to the scientific community. These resources are available for download at \url{https://github.com/Xianjun-Yang/PLLaMa}.

Title: Evaluating Large Language Models in Semantic Parsing for Conversational Question Answering over Knowledge Graphs. (arXiv:2401.01711v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.01711
• Code URL: https://github.com/sebischair/llm-sp-cqa
• Copy Paste: [[2401.01711]] Evaluating Large Language Models in Semantic Parsing for Conversational Question Answering over Knowledge Graphs(http://arxiv.org/abs/2401.01711)
• Summary:

  Conversational question answering systems often rely on semantic parsing to enable interactive information retrieval, which involves the generation of structured database queries from a natural language input. For information-seeking conversations about facts stored within a knowledge graph, dialogue utterances are transformed into graph queries in a process that is called knowledge-based conversational question answering. This paper evaluates the performance of large language models that have not been explicitly pre-trained on this task. Through a series of experiments on an extensive benchmark dataset, we compare models of varying sizes with different prompting techniques and identify common issue types in the generated output. Our results demonstrate that large language models are capable of generating graph queries from dialogues, with significant improvements achievable through few-shot prompting and fine-tuning techniques, especially for smaller models that exhibit lower zero-shot performance.

Title: Navigating Uncertainty: Optimizing API Dependency for Hallucination Reduction in Closed-Book Question Answering. (arXiv:2401.01780v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.01780
• Code URL: null
• Copy Paste: [[2401.01780]] Navigating Uncertainty: Optimizing API Dependency for Hallucination Reduction in Closed-Book Question Answering(http://arxiv.org/abs/2401.01780)
• Summary:

  While Large Language Models (LLM) are able to accumulate and restore knowledge, they are still prone to hallucination. Especially when faced with factual questions, LLM cannot only rely on knowledge stored in parameters to guarantee truthful and correct answers. Augmenting these models with the ability to search on external information sources, such as the web, is a promising approach to ground knowledge to retrieve information. However, searching in a large collection of documents introduces additional computational/time costs. An optimal behavior would be to query external resources only when the LLM is not confident about answers. In this paper, we propose a new LLM able to self-estimate if it is able to answer directly or needs to request an external tool. We investigate a supervised approach by introducing a hallucination masking mechanism in which labels are generated using a close book question-answering task. In addition, we propose to leverage parameter-efficient fine-tuning techniques to train our model on a small amount of data. Our model directly provides answers for $78.2\%$ of the known queries and opts to search for $77.2\%$ of the unknown ones. This results in the API being utilized only $62\%$ of the time.

Title: Multilingual Instruction Tuning With Just a Pinch of Multilinguality. (arXiv:2401.01854v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.01854
• Code URL: null
• Copy Paste: [[2401.01854]] Multilingual Instruction Tuning With Just a Pinch of Multilinguality(http://arxiv.org/abs/2401.01854)
• Summary:

  As instruction-tuned large language models (LLMs) gain global adoption, their ability to follow instructions in multiple languages becomes increasingly crucial. One promising approach is cross-lingual transfer, where a model acquires specific functionality on some language by finetuning on another language. In this work, we investigate how multilinguality during instruction tuning of a multilingual LLM affects instruction-following across languages. We first show that many languages transfer some instruction-following capabilities to other languages from even monolingual tuning. Furthermore, we find that only 40 multilingual examples in an English tuning set substantially improve multilingual instruction-following, both in seen and unseen languages during tuning. In general, we observe that models tuned on multilingual mixtures exhibit comparable or superior performance in several languages compared to monolingually tuned models, despite training on 10x fewer examples in those languages. Finally, we find that increasing the number of languages in the instruction tuning set from 1 to only 2, 3, or 4 increases cross-lingual generalization. Our results suggest that building massively multilingual instruction-tuned models can be done with only a very small set of multilingual instruction-responses.

segmentation

Title: Off-Road LiDAR Intensity Based Semantic Segmentation. (arXiv:2401.01439v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.01439
• Code URL: https://github.com/moonlabiiserb/lidar-intensity-predictor
• Copy Paste: [[2401.01439]] Off-Road LiDAR Intensity Based Semantic Segmentation(http://arxiv.org/abs/2401.01439)
• Summary:

  LiDAR is used in autonomous driving to provide 3D spatial information and enable accurate perception in off-road environments, aiding in obstacle detection, mapping, and path planning. Learning-based LiDAR semantic segmentation utilizes machine learning techniques to automatically classify objects and regions in LiDAR point clouds. Learning-based models struggle in off-road environments due to the presence of diverse objects with varying colors, textures, and undefined boundaries, which can lead to difficulties in accurately classifying and segmenting objects using traditional geometric-based features. In this paper, we address this problem by harnessing the LiDAR intensity parameter to enhance object segmentation in off-road environments. Our approach was evaluated in the RELLIS-3D data set and yielded promising results as a preliminary analysis with improved mIoU for classes "puddle" and "grass" compared to more complex deep learning-based benchmarks. The methodology was evaluated for compatibility across both Velodyne and Ouster LiDAR systems, assuring its cross-platform applicability. This analysis advocates for the incorporation of calibrated intensity as a supplementary input, aiming to enhance the prediction accuracy of learning based semantic segmentation frameworks. https://github.com/MOONLABIISERB/lidar-intensity-predictor/tree/main

Title: MLIP: Medical Language-Image Pre-training with Masked Local Representation Learning. (arXiv:2401.01591v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.01591
• Code URL: null
• Copy Paste: [[2401.01591]] MLIP: Medical Language-Image Pre-training with Masked Local Representation Learning(http://arxiv.org/abs/2401.01591)
• Summary:

  Existing contrastive language-image pre-training aims to learn a joint representation by matching abundant image-text pairs. However, the number of image-text pairs in medical datasets is usually orders of magnitude smaller than that in natural datasets. Besides, medical image-text pairs often involve numerous complex fine-grained correspondences. This paper aims to enhance the data efficiency by introducing multiple-to-multiple local relationship modeling to capture denser supervisions. More specifically, we propose a Medical Language-Image Pre-training (MLIP) framework, which exploits the limited image-text medical data more efficiently through patch-sentence matching. Furthermore, we introduce a masked contrastive learning strategy with semantic integrity estimation to reduce redundancy in images while preserving the underlying semantics. Our evaluation results show that MLIP outperforms previous work in zero/few-shot classification and few-shot segmentation tasks by a large margin.

Title: Context-Aware Interaction Network for RGB-T Semantic Segmentation. (arXiv:2401.01624v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.01624
• Code URL: https://github.com/yinglv1106/cainet
• Copy Paste: [[2401.01624]] Context-Aware Interaction Network for RGB-T Semantic Segmentation(http://arxiv.org/abs/2401.01624)
• Summary:

  RGB-T semantic segmentation is a key technique for autonomous driving scenes understanding. For the existing RGB-T semantic segmentation methods, however, the effective exploration of the complementary relationship between different modalities is not implemented in the information interaction between multiple levels. To address such an issue, the Context-Aware Interaction Network (CAINet) is proposed for RGB-T semantic segmentation, which constructs interaction space to exploit auxiliary tasks and global context for explicitly guided learning. Specifically, we propose a Context-Aware Complementary Reasoning (CACR) module aimed at establishing the complementary relationship between multimodal features with the long-term context in both spatial and channel dimensions. Further, considering the importance of global contextual and detailed information, we propose the Global Context Modeling (GCM) module and Detail Aggregation (DA) module, and we introduce specific auxiliary supervision to explicitly guide the context interaction and refine the segmentation map. Extensive experiments on two benchmark datasets of MFNet and PST900 demonstrate that the proposed CAINet achieves state-of-the-art performance. The code is available at https://github.com/YingLv1106/CAINet.

Title: BLADE: Box-Level Supervised Amodal Segmentation through Directed Expansion. (arXiv:2401.01642v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.01642
• Code URL: null
• Copy Paste: [[2401.01642]] BLADE: Box-Level Supervised Amodal Segmentation through Directed Expansion(http://arxiv.org/abs/2401.01642)
• Summary:

  Perceiving the complete shape of occluded objects is essential for human and machine intelligence. While the amodal segmentation task is to predict the complete mask of partially occluded objects, it is time-consuming and labor-intensive to annotate the pixel-level ground truth amodal masks. Box-level supervised amodal segmentation addresses this challenge by relying solely on ground truth bounding boxes and instance classes as supervision, thereby alleviating the need for exhaustive pixel-level annotations. Nevertheless, current box-level methodologies encounter limitations in generating low-resolution masks and imprecise boundaries, failing to meet the demands of practical real-world applications. We present a novel solution to tackle this problem by introducing a directed expansion approach from visible masks to corresponding amodal masks. Our approach involves a hybrid end-to-end network based on the overlapping region - the area where different instances intersect. Diverse segmentation strategies are applied for overlapping regions and non-overlapping regions according to distinct characteristics. To guide the expansion of visible masks, we introduce an elaborately-designed connectivity loss for overlapping regions, which leverages correlations with visible masks and facilitates accurate amodal segmentation. Experiments are conducted on several challenging datasets and the results show that our proposed method can outperform existing state-of-the-art methods with large margins.

Title: S3Net: Innovating Stereo Matching and Semantic Segmentation with a Single-Branch Semantic Stereo Network in Satellite Epipolar Imagery. (arXiv:2401.01643v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.01643
• Code URL: null
• Copy Paste: [[2401.01643]] S3Net: Innovating Stereo Matching and Semantic Segmentation with a Single-Branch Semantic Stereo Network in Satellite Epipolar Imagery(http://arxiv.org/abs/2401.01643)
• Summary:

  Stereo matching and semantic segmentation are significant tasks in binocular satellite 3D reconstruction. However, previous studies primarily view these as independent parallel tasks, lacking an integrated multitask learning framework. This work introduces a solution, the Single-branch Semantic Stereo Network (S3Net), which innovatively combines semantic segmentation and stereo matching using Self-Fuse and Mutual-Fuse modules. Unlike preceding methods that utilize semantic or disparity information independently, our method dentifies and leverages the intrinsic link between these two tasks, leading to a more accurate understanding of semantic information and disparity estimation. Comparative testing on the US3D dataset proves the effectiveness of our S3Net. Our model improves the mIoU in semantic segmentation from 61.38 to 67.39, and reduces the D1-Error and average endpoint error (EPE) in disparity estimation from 10.051 to 9.579 and 1.439 to 1.403 respectively, surpassing existing competitive methods. Our codes are available at:https://github.com/CVEO/S3Net.