2023-12-20

secure

Title: Bridging the Gap: Generalising State-of-the-Art U-Net Models to Sub-Saharan African Populations. (arXiv:2312.11770v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.11770
• Code URL: null
• Copy Paste: [[2312.11770]] Bridging the Gap: Generalising State-of-the-Art U-Net Models to Sub-Saharan African Populations(http://arxiv.org/abs/2312.11770)
• Summary:

  A critical challenge for tumour segmentation models is the ability to adapt to diverse clinical settings, particularly when applied to poor-quality neuroimaging data. The uncertainty surrounding this adaptation stems from the lack of representative datasets, leaving top-performing models without exposure to common artifacts found in MRI data throughout Sub-Saharan Africa (SSA). We replicated a framework that secured the 2nd position in the 2022 BraTS competition to investigate the impact of dataset composition on model performance and pursued four distinct approaches through training a model with: 1) BraTS-Africa data only (train_SSA, N=60), 2) BraTS-Adult Glioma data only (train_GLI, N=1251), 3) both datasets together (train_ALL, N=1311), and 4) through further training the train_GLI model with BraTS-Africa data (train_ftSSA). Notably, training on a smaller low-quality dataset alone (train_SSA) yielded subpar results, and training on a larger high-quality dataset alone (train_GLI) struggled to delineate oedematous tissue in the low-quality validation set. The most promising approach (train_ftSSA) involved pre-training a model on high-quality neuroimages and then fine-tuning it on the smaller, low-quality dataset. This approach outperformed the others, ranking second in the MICCAI BraTS Africa global challenge external testing phase. These findings underscore the significance of larger sample sizes and broad exposure to data in improving segmentation performance. Furthermore, we demonstrated that there is potential for improving such models by fine-tuning them with a wider range of data locally.

Title: A Red Teaming Framework for Securing AI in Maritime Autonomous Systems. (arXiv:2312.11500v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.11500
• Code URL: null
• Copy Paste: [[2312.11500]] A Red Teaming Framework for Securing AI in Maritime Autonomous Systems(http://arxiv.org/abs/2312.11500)
• Summary:

  Artificial intelligence (AI) is being ubiquitously adopted to automate processes in science and industry. However, due to its often intricate and opaque nature, AI has been shown to possess inherent vulnerabilities which can be maliciously exploited with adversarial AI, potentially putting AI users and developers at both cyber and physical risk. In addition, there is insufficient comprehension of the real-world effects of adversarial AI and an inadequacy of AI security examinations; therefore, the growing threat landscape is unknown for many AI solutions. To mitigate this issue, we propose one of the first red team frameworks for evaluating the AI security of maritime autonomous systems. The framework provides operators with a proactive (secure by design) and reactive (post-deployment evaluation) response to securing AI technology today and in the future. This framework is a multi-part checklist, which can be tailored to different systems and requirements. We demonstrate this framework to be highly effective for a red team to use to uncover numerous vulnerabilities within a real-world maritime autonomous systems AI, ranging from poisoning to adversarial patch attacks. The lessons learned from systematic AI red teaming can help prevent MAS-related catastrophic events in a world with increasing uptake and reliance on mission-critical AI.

security

Title: Unlocking Musculoskeletal Disorder Risk Factors: NLP-Based Classification and Mode-Based Ranking. (arXiv:2312.11517v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.11517
• Code URL: null
• Copy Paste: [[2312.11517]] Unlocking Musculoskeletal Disorder Risk Factors: NLP-Based Classification and Mode-Based Ranking(http://arxiv.org/abs/2312.11517)
• Summary:

  This research delves into the intricate landscape of Musculoskeletal Disorder (MSD) risk factors, employing a novel fusion of Natural Language Processing (NLP) techniques and mode-based ranking methodologies. The primary objective is to advance the comprehension of MSD risk factors, their classification, and their relative severity, facilitating more targeted preventive and management interventions. The study utilizes eight diverse models, integrating pre-trained transformers, cosine similarity, and various distance metrics to classify risk factors into personal, biomechanical, workplace, psychological, and organizational classes. Key findings reveal that the BERT model with cosine similarity attains an overall accuracy of 28\%, while the sentence transformer, coupled with Euclidean, Bray-Curtis, and Minkowski distances, achieves a flawless accuracy score of 100\%. In tandem with the classification efforts, the research employs a mode-based ranking approach on survey data to discern the severity hierarchy of MSD risk factors. Intriguingly, the rankings align precisely with the previous literature, reaffirming the consistency and reliability of the approach. ``Working posture" emerges as the most severe risk factor, emphasizing the critical role of proper posture in preventing MSDs. The collective perceptions of survey participants underscore the significance of factors like ``Job insecurity," ``Effort reward imbalance," and ``Poor employee facility" in contributing to MSD risks. The convergence of rankings provides actionable insights for organizations aiming to reduce the prevalence of MSDs. The study concludes with implications for targeted interventions, recommendations for improving workplace conditions, and avenues for future research.

Title: Evaluating Language-Model Agents on Realistic Autonomous Tasks. (arXiv:2312.11671v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.11671
• Code URL: null
• Copy Paste: [[2312.11671]] Evaluating Language-Model Agents on Realistic Autonomous Tasks(http://arxiv.org/abs/2312.11671)
• Summary:

  In this report, we explore the ability of language model agents to acquire resources, create copies of themselves, and adapt to novel challenges they encounter in the wild. We refer to this cluster of capabilities as "autonomous replication and adaptation" or ARA. We believe that systems capable of ARA could have wide-reaching and hard-to-anticipate consequences, and that measuring and forecasting ARA may be useful for informing measures around security, monitoring, and alignment. Additionally, once a system is capable of ARA, placing bounds on a system's capabilities may become significantly more difficult.

  We construct four simple example agents that combine language models with tools that allow them to take actions in the world. We then evaluate these agents on 12 tasks relevant to ARA. We find that these language model agents can only complete the easiest tasks from this list, although they make some progress on the more challenging tasks. Unfortunately, these evaluations are not adequate to rule out the possibility that near-future agents will be capable of ARA. In particular, we do not think that these evaluations provide good assurance that the ``next generation'' of language models (e.g. 100x effective compute scaleup on existing models) will not yield agents capable of ARA, unless intermediate evaluations are performed during pretraining. Relatedly, we expect that fine-tuning of the existing models could produce substantially more competent agents, even if the fine-tuning is not directly targeted at ARA.

Title: SYNC+SYNC: Software Cache Write Covert Channels Exploiting Memory-disk Synchronization. (arXiv:2312.11501v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.11501
• Code URL: null
• Copy Paste: [[2312.11501]] SYNC+SYNC: Software Cache Write Covert Channels Exploiting Memory-disk Synchronization(http://arxiv.org/abs/2312.11501)
• Summary:

  Memory-disk synchronization is a critical technology for ensuring data correctness, integrity, and security, especially in systems that handle sensitive information like financial transactions and medical records. We propose SYNC+SYNC, a group of attacks that exploit the memory-disk synchronization primitives. SYNC+SYNC works by subtly varying the timing of synchronization on the write buffer, offering several advantages: 1) implemented purely in software, enabling deployment on any hardware devices; 2) resilient against existing cache partitioning and randomization techniques; 3) unaffected by prefetching techniques and cache replacement strategies. We present the principles of SYNC+SYNC through the implementation of two write covert channel protocols, using either a single file or page, and introduce three enhanced strategies that utilize multiple files and pages. The feasibility of these channels is demonstrated in both cross-process and cross-sandbox scenarios across diverse operating systems (OSes). Experimental results show that, the average rate can reach 2.036 Kb/s (with a peak rate of 14.762 Kb/s) and the error rate is 0% on Linux; when running on macOS, the average rate achieves 10.211 Kb/s (with a peak rate of 253.022 Kb/s) and the error rate is 0.004%. To the best of our knowledge, SYNC+SYNC is the first high-speed write covert channel for software cache.

privacy

Title: Improved Differentially Private and Lazy Online Convex Optimization. (arXiv:2312.11534v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.11534
• Code URL: null
• Copy Paste: [[2312.11534]] Improved Differentially Private and Lazy Online Convex Optimization(http://arxiv.org/abs/2312.11534)
• Summary:

  We study the task of $(\epsilon, \delta)$-differentially private online convex optimization (OCO). In the online setting, the release of each distinct decision or iterate carries with it the potential for privacy loss. This problem has a long history of research starting with Jain et al. [2012] and the best known results for the regime of {\epsilon} being very small are presented in Agarwal et al. [2023]. In this paper we improve upon the results of Agarwal et al. [2023] in terms of the dimension factors as well as removing the requirement of smoothness. Our results are now the best known rates for DP-OCO in this regime.

  Our algorithms builds upon the work of [Asi et al., 2023] which introduced the idea of explicitly limiting the number of switches via rejection sampling. The main innovation in our algorithm is the use of sampling from a strongly log-concave density which allows us to trade-off the dimension factors better leading to improved results.

Title: Blind-Touch: Homomorphic Encryption-Based Distributed Neural Network Inference for Privacy-Preserving Fingerprint Authentication. (arXiv:2312.11575v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.11575
• Code URL: null
• Copy Paste: [[2312.11575]] Blind-Touch: Homomorphic Encryption-Based Distributed Neural Network Inference for Privacy-Preserving Fingerprint Authentication(http://arxiv.org/abs/2312.11575)
• Summary:

  Fingerprint authentication is a popular security mechanism for smartphones and laptops. However, its adoption in web and cloud environments has been limited due to privacy concerns over storing and processing biometric data on servers. This paper introduces Blind-Touch, a novel machine learning-based fingerprint authentication system leveraging homomorphic encryption to address these privacy concerns. Homomorphic encryption allows computations on encrypted data without decrypting. Thus, Blind-Touch can keep fingerprint data encrypted on the server while performing machine learning operations. Blind-Touch combines three strategies to efficiently utilize homomorphic encryption in machine learning: (1) It optimizes the feature vector for a distributed architecture, processing the first fully connected layer (FC-16) in plaintext on the client side and the subsequent layer (FC-1) post-encryption on the server, thereby minimizing encrypted computations; (2) It employs a homomorphic encryptioncompatible data compression technique capable of handling 8,192 authentication results concurrently; and (3) It utilizes a clustered server architecture to simultaneously process authentication results, thereby enhancing scalability with increasing user numbers. Blind-Touch achieves high accuracy on two benchmark fingerprint datasets, with a 93.6% F1- score for the PolyU dataset and a 98.2% F1-score for the SOKOTO dataset. Moreover, Blind-Touch can match a fingerprint among 5,000 in about 0.65 seconds. With its privacyfocused design, high accuracy, and efficiency, Blind-Touch is a promising alternative to conventional fingerprint authentication for web and cloud applications.

Title: Protect Your Score: Contact Tracing With Differential Privacy Guarantees. (arXiv:2312.11581v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.11581
• Code URL: null
• Copy Paste: [[2312.11581]] Protect Your Score: Contact Tracing With Differential Privacy Guarantees(http://arxiv.org/abs/2312.11581)
• Summary:

  The pandemic in 2020 and 2021 had enormous economic and societal consequences, and studies show that contact tracing algorithms can be key in the early containment of the virus. While large strides have been made towards more effective contact tracing algorithms, we argue that privacy concerns currently hold deployment back. The essence of a contact tracing algorithm constitutes the communication of a risk score. Yet, it is precisely the communication and release of this score to a user that an adversary can leverage to gauge the private health status of an individual. We pinpoint a realistic attack scenario and propose a contact tracing algorithm with differential privacy guarantees against this attack. The algorithm is tested on the two most widely used agent-based COVID19 simulators and demonstrates superior performance in a wide range of settings. Especially for realistic test scenarios and while releasing each risk score with epsilon=1 differential privacy, we achieve a two to ten-fold reduction in the infection rate of the virus. To the best of our knowledge, this presents the first contact tracing algorithm with differential privacy guarantees when revealing risk scores for COVID19.

Title: A Simple and Practical Method for Reducing the Disparate Impact of Differential Privacy. (arXiv:2312.11712v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.11712
• Code URL: null
• Copy Paste: [[2312.11712]] A Simple and Practical Method for Reducing the Disparate Impact of Differential Privacy(http://arxiv.org/abs/2312.11712)
• Summary:

  Differentially private (DP) mechanisms have been deployed in a variety of high-impact social settings (perhaps most notably by the U.S. Census). Since all DP mechanisms involve adding noise to results of statistical queries, they are expected to impact our ability to accurately analyze and learn from data, in effect trading off privacy with utility. Alarmingly, the impact of DP on utility can vary significantly among different sub-populations. A simple way to reduce this disparity is with stratification. First compute an independent private estimate for each group in the data set (which may be the intersection of several protected classes), then, to compute estimates of global statistics, appropriately recombine these group estimates. Our main observation is that naive stratification often yields high-accuracy estimates of population-level statistics, without the need for additional privacy budget. We support this observation theoretically and empirically. Our theoretical results center on the private mean estimation problem, while our empirical results center on extensive experiments on private data synthesis to demonstrate the effectiveness of stratification on a variety of private mechanisms. Overall, we argue that this straightforward approach provides a strong baseline against which future work on reducing utility disparities of DP mechanisms should be compared.

Title: A Summary of Privacy-Preserving Data Publishing in the Local Setting. (arXiv:2312.11845v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.11845
• Code URL: null
• Copy Paste: [[2312.11845]] A Summary of Privacy-Preserving Data Publishing in the Local Setting(http://arxiv.org/abs/2312.11845)
• Summary:

  The exponential growth of collected, processed, and shared data has given rise to concerns about individuals' privacy. Consequently, various laws and regulations have been established to oversee how organizations handle and safeguard data. One such method is Statistical Disclosure Control, which aims to minimize the risk of exposing confidential information by de-identifying it. This de-identification is achieved through specific privacy-preserving techniques. However, a trade-off exists: de-identified data can often lead to a loss of information, which might impact the accuracy of data analysis and the predictive capability of models. The overarching goal remains to safeguard individual privacy while preserving the data's interpretability, meaning its overall usefulness. Despite advances in Statistical Disclosure Control, the field continues to evolve, with no definitive solution that strikes an optimal balance between privacy and utility. This survey delves into the intricate processes of de-identification. We outline the current privacy-preserving techniques employed in microdata de-identification, delve into privacy measures tailored for various disclosure scenarios, and assess metrics for information loss and predictive performance. Herein, we tackle the primary challenges posed by privacy constraints, overview predominant strategies to mitigate these challenges, categorize privacy-preserving techniques, offer a theoretical assessment of current comparative research, and highlight numerous unresolved issues in the domain.

protect

Title: QuanShield: Protecting against Side-Channels Attacks using Self-Destructing Enclaves. (arXiv:2312.11796v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.11796
• Code URL: null
• Copy Paste: [[2312.11796]] QuanShield: Protecting against Side-Channels Attacks using Self-Destructing Enclaves(http://arxiv.org/abs/2312.11796)
• Summary:

  Trusted Execution Environments (TEEs) allow user processes to create enclaves that protect security-sensitive computation against access from the OS kernel and the hypervisor. Recent work has shown that TEEs are vulnerable to side-channel attacks that allow an adversary to learn secrets shielded in enclaves. The majority of such attacks trigger exceptions or interrupts to trace the control or data flow of enclave execution.

  We propose QuanShield, a system that protects enclaves from side-channel attacks that interrupt enclave execution. The main idea behind QuanShield is to strengthen resource isolation by creating an interrupt-free environment on a dedicated CPU core for running enclaves in which enclaves terminate when interrupts occur. QuanShield avoids interrupts by exploiting the tickless scheduling mode supported by recent OS kernels. QuanShield then uses the save area (SA) of the enclave, which is used by the hardware to support interrupt handling, as a second stack. Through an LLVM-based compiler pass, QuanShield modifies enclave instructions to store/load memory references, such as function frame base addresses, to/from the SA. When an interrupt occurs, the hardware overwrites the data in the SA with CPU state, thus ensuring that enclave execution fails. Our evaluation shows that QuanShield significantly raises the bar for interrupt-based attacks with practical overhead.

Title: EncryIP: A Practical Encryption-Based Framework for Model Intellectual Property Protection. (arXiv:2312.12049v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.12049
• Code URL: null
• Copy Paste: [[2312.12049]] EncryIP: A Practical Encryption-Based Framework for Model Intellectual Property Protection(http://arxiv.org/abs/2312.12049)
• Summary:

  In the rapidly growing digital economy, protecting intellectual property (IP) associated with digital products has become increasingly important. Within this context, machine learning (ML) models, being highly valuable digital assets, have gained significant attention for IP protection. This paper introduces a practical encryption-based framework called \textit{EncryIP}, which seamlessly integrates a public-key encryption scheme into the model learning process. This approach enables the protected model to generate randomized and confused labels, ensuring that only individuals with accurate secret keys, signifying authorized users, can decrypt and reveal authentic labels. Importantly, the proposed framework not only facilitates the protected model to multiple authorized users without requiring repetitive training of the original ML model with IP protection methods but also maintains the model's performance without compromising its accuracy. Compared to existing methods like watermark-based, trigger-based, and passport-based approaches, \textit{EncryIP} demonstrates superior effectiveness in both training protected models and efficiently detecting the unauthorized spread of ML models.

defense

attack

Title: QuadAttack: A Quadratic Programming Approach to Ordered Top-K Attacks. (arXiv:2312.11510v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.11510
• Code URL: null
• Copy Paste: [[2312.11510]] QuadAttack: A Quadratic Programming Approach to Ordered Top-K Attacks(http://arxiv.org/abs/2312.11510)
• Summary:

  The adversarial vulnerability of Deep Neural Networks (DNNs) has been well-known and widely concerned, often under the context of learning top-$1$ attacks (e.g., fooling a DNN to classify a cat image as dog). This paper shows that the concern is much more serious by learning significantly more aggressive ordered top-$K$ clear-box~\footnote{ This is often referred to as white/black-box attacks in the literature. We choose to adopt neutral terminology, clear/opaque-box attacks in this paper, and omit the prefix clear-box for simplicity.} targeted attacks proposed in Adversarial Distillation. We propose a novel and rigorous quadratic programming (QP) method of learning ordered top-$K$ attacks with low computing cost, dubbed as \textbf{QuadAttac$K$}. Our QuadAttac$K$ directly solves the QP to satisfy the attack constraint in the feature embedding space (i.e., the input space to the final linear classifier), which thus exploits the semantics of the feature embedding space (i.e., the principle of class coherence). With the optimized feature embedding vector perturbation, it then computes the adversarial perturbation in the data space via the vanilla one-step back-propagation. In experiments, the proposed QuadAttac$K$ is tested in the ImageNet-1k classification using ResNet-50, DenseNet-121, and Vision Transformers (ViT-B and DEiT-S). It successfully pushes the boundary of successful ordered top-$K$ attacks from $K=10$ up to $K=20$ at a cheap budget ($1\times 60$) and further improves attack success rates for $K=5$ for all tested models, while retaining the performance for $K=1$.

Title: Maatphor: Automated Variant Analysis for Prompt Injection Attacks. (arXiv:2312.11513v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.11513
• Code URL: null
• Copy Paste: [[2312.11513]] Maatphor: Automated Variant Analysis for Prompt Injection Attacks(http://arxiv.org/abs/2312.11513)
• Summary:

  Prompt injection has emerged as a serious security threat to large language models (LLMs). At present, the current best-practice for defending against newly-discovered prompt injection techniques is to add additional guardrails to the system (e.g., by updating the system prompt or using classifiers on the input and/or output of the model.) However, in the same way that variants of a piece of malware are created to evade anti-virus software, variants of a prompt injection can be created to evade the LLM's guardrails. Ideally, when a new prompt injection technique is discovered, candidate defenses should be tested not only against the successful prompt injection, but also against possible variants.

  In this work, we present, a tool to assist defenders in performing automated variant analysis of known prompt injection attacks. This involves solving two main challenges: (1) automatically generating variants of a given prompt according, and (2) automatically determining whether a variant was effective based only on the output of the model. This tool can also assist in generating datasets for jailbreak and prompt injection attacks, thus overcoming the scarcity of data in this domain.

  We evaluate Maatphor on three different types of prompt injection tasks. Starting from an ineffective (0%) seed prompt, Maatphor consistently generates variants that are at least 60% effective within the first 40 iterations.

Title: Cryptanalysis of PLWE based on zero-trace quadratic roots. (arXiv:2312.11533v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.11533
• Code URL: null
• Copy Paste: [[2312.11533]] Cryptanalysis of PLWE based on zero-trace quadratic roots(http://arxiv.org/abs/2312.11533)
• Summary:

  We extend two of the attacks on the PLWE problem presented in (Y. Elias, K. E. Lauter, E. Ozman, and K. E. Stange, Ring-LWE Cryptography for the Number Theorist, in Directions in Number Theory, E. E. Eischen, L. Long, R. Pries, and K. E. Stange, eds., vol. 3 of Association for Women in Mathematics Series, Cham, 2016, Springer International Publishing, pp. 271-290) to a ring $R_q=\mathbb{F}_q[x]/(f(x))$ where the irreducible monic polynomial $f(x)\in\mathbb{Z}[x]$ has an irreducible quadratic factor over $\mathbb{F}_q[x]$ of the form $x^2+\rho$ with $\rho$ of suitable multiplicative order in $\mathbb{F}_q$. Our attack exploits the fact that the trace of the root is zero and has overwhelming success probability as a function of the number of samples taken as input. An implementation in Maple and some examples of our attack are also provided.

Title: A Study on Transferability of Deep Learning Models for Network Intrusion Detection. (arXiv:2312.11550v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.11550
• Code URL: https://github.com/ghosh64/transferability
• Copy Paste: [[2312.11550]] A Study on Transferability of Deep Learning Models for Network Intrusion Detection(http://arxiv.org/abs/2312.11550)
• Summary:

  In this paper, we explore transferability in learning between different attack classes in a network intrusion detection setup. We evaluate transferability of attack classes by training a deep learning model with a specific attack class and testing it on a separate attack class. We observe the effects of real and synthetically generated data augmentation techniques on transferability. We investigate the nature of observed transferability relationships, which can be either symmetric or asymmetric. We also examine explainability of the transferability relationships using the recursive feature elimination algorithm. We study data preprocessing techniques to boost model performance. The code for this work can be found at https://github.com/ghosh64/transferability.

Title: Model Stealing Attack against Recommender System. (arXiv:2312.11571v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.11571
• Code URL: null
• Copy Paste: [[2312.11571]] Model Stealing Attack against Recommender System(http://arxiv.org/abs/2312.11571)
• Summary:

  Recent studies have demonstrated the vulnerability of recommender systems to data privacy attacks. However, research on the threat to model privacy in recommender systems, such as model stealing attacks, is still in its infancy. Some adversarial attacks have achieved model stealing attacks against recommender systems, to some extent, by collecting abundant training data of the target model (target data) or making a mass of queries. In this paper, we constrain the volume of available target data and queries and utilize auxiliary data, which shares the item set with the target data, to promote model stealing attacks. Although the target model treats target and auxiliary data differently, their similar behavior patterns allow them to be fused using an attention mechanism to assist attacks. Besides, we design stealing functions to effectively extract the recommendation list obtained by querying the target model. Experimental results show that the proposed methods are applicable to most recommender systems and various scenarios and exhibit excellent attack performance on multiple datasets.

robust

Title: SAI3D: Segment Any Instance in 3D Scenes. (arXiv:2312.11557v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.11557
• Code URL: null
• Copy Paste: [[2312.11557]] SAI3D: Segment Any Instance in 3D Scenes(http://arxiv.org/abs/2312.11557)
• Summary:

  Advancements in 3D instance segmentation have traditionally been tethered to the availability of annotated datasets, limiting their application to a narrow spectrum of object categories. Recent efforts have sought to harness vision-language models like CLIP for open-set semantic reasoning, yet these methods struggle to distinguish between objects of the same categories and rely on specific prompts that are not universally applicable. In this paper, we introduce SAI3D, a novel zero-shot 3D instance segmentation approach that synergistically leverages geometric priors and semantic cues derived from Segment Anything Model (SAM). Our method partitions a 3D scene into geometric primitives, which are then progressively merged into 3D instance segmentations that are consistent with the multi-view SAM masks. Moreover, we design a hierarchical region-growing algorithm with a dynamic thresholding mechanism, which largely improves the robustness of finegrained 3D scene parsing. Empirical evaluations on Scan-Net and the more challenging ScanNet++ datasets demonstrate the superiority of our approach. Notably, SAI3D outperforms existing open-vocabulary baselines and even surpasses fully-supervised methods in class-agnostic segmentation on ScanNet++.

Title: CAManim: Animating end-to-end network activation maps. (arXiv:2312.11772v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.11772
• Code URL: null
• Copy Paste: [[2312.11772]] CAManim: Animating end-to-end network activation maps(http://arxiv.org/abs/2312.11772)
• Summary:

  Deep neural networks have been widely adopted in numerous domains due to their high performance and accessibility to developers and application-specific end-users. Fundamental to image-based applications is the development of Convolutional Neural Networks (CNNs), which possess the ability to automatically extract features from data. However, comprehending these complex models and their learned representations, which typically comprise millions of parameters and numerous layers, remains a challenge for both developers and end-users. This challenge arises due to the absence of interpretable and transparent tools to make sense of black-box models. There exists a growing body of Explainable Artificial Intelligence (XAI) literature, including a collection of methods denoted Class Activation Maps (CAMs), that seek to demystify what representations the model learns from the data, how it informs a given prediction, and why it, at times, performs poorly in certain tasks. We propose a novel XAI visualization method denoted CAManim that seeks to simultaneously broaden and focus end-user understanding of CNN predictions by animating the CAM-based network activation maps through all layers, effectively depicting from end-to-end how a model progressively arrives at the final layer activation. Herein, we demonstrate that CAManim works with any CAM-based method and various CNN architectures. Beyond qualitative model assessments, we additionally propose a novel quantitative assessment that expands upon the Remove and Debias (ROAD) metric, pairing the qualitative end-to-end network visual explanations assessment with our novel quantitative "yellow brick ROAD" assessment (ybROAD). This builds upon prior research to address the increasing demand for interpretable, robust, and transparent model assessment methodology, ultimately improving an end-user's trust in a given model's predictions.

Title: Topo-MLP : A Simplicial Network Without Message Passing. (arXiv:2312.11862v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.11862
• Code URL: null
• Copy Paste: [[2312.11862]] Topo-MLP : A Simplicial Network Without Message Passing(http://arxiv.org/abs/2312.11862)
• Summary:

  Due to their ability to model meaningful higher order relations among a set of entities, higher order network models have emerged recently as a powerful alternative for graph-based network models which are only capable of modeling binary relationships. Message passing paradigm is still dominantly used to learn representations even for higher order network models. While powerful, message passing can have disadvantages during inference, particularly when the higher order connectivity information is missing or corrupted. To overcome such limitations, we propose Topo-MLP, a purely MLP-based simplicial neural network algorithm to learn the representation of elements in a simplicial complex without explicitly relying on message passing. Our framework utilizes a novel Higher Order Neighborhood Contrastive (HONC) loss which implicitly incorporates the simplicial structure into representation learning. Our proposed model's simplicity makes it faster during inference. Moreover, we show that our model is robust when faced with missing or corrupted connectivity structure.

Title: Beyond Prototypes: Semantic Anchor Regularization for Better Representation Learning. (arXiv:2312.11872v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.11872
• Code URL: https://github.com/geyanqi/sar
• Copy Paste: [[2312.11872]] Beyond Prototypes: Semantic Anchor Regularization for Better Representation Learning(http://arxiv.org/abs/2312.11872)
• Summary:

  One of the ultimate goals of representation learning is to achieve compactness within a class and well-separability between classes. Many outstanding metric-based and prototype-based methods following the Expectation-Maximization paradigm, have been proposed for this objective. However, they inevitably introduce biases into the learning process, particularly with long-tail distributed training data. In this paper, we reveal that the class prototype is not necessarily to be derived from training features and propose a novel perspective to use pre-defined class anchors serving as feature centroid to unidirectionally guide feature learning. However, the pre-defined anchors may have a large semantic distance from the pixel features, which prevents them from being directly applied. To address this issue and generate feature centroid independent from feature learning, a simple yet effective Semantic Anchor Regularization (SAR) is proposed. SAR ensures the interclass separability of semantic anchors in the semantic space by employing a classifier-aware auxiliary cross-entropy loss during training via disentanglement learning. By pulling the learned features to these semantic anchors, several advantages can be attained: 1) the intra-class compactness and naturally inter-class separability, 2) induced bias or errors from feature learning can be avoided, and 3) robustness to the long-tailed problem. The proposed SAR can be used in a plug-and-play manner in the existing models. Extensive experiments demonstrate that the SAR performs better than previous sophisticated prototype-based methods. The implementation is available at https://github.com/geyanqi/SAR.

Title: EVI-SAM: Robust, Real-time, Tightly-coupled Event-Visual-Inertial State Estimation and 3D Dense Mapping. (arXiv:2312.11911v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.11911
• Code URL: null
• Copy Paste: [[2312.11911]] EVI-SAM: Robust, Real-time, Tightly-coupled Event-Visual-Inertial State Estimation and 3D Dense Mapping(http://arxiv.org/abs/2312.11911)
• Summary:

  Event cameras are bio-inspired, motion-activated sensors that demonstrate substantial potential in handling challenging situations, such as motion blur and high-dynamic range. In this paper, we proposed EVI-SAM to tackle the problem of 6 DoF pose tracking and 3D reconstruction using monocular event camera. A novel event-based hybrid tracking framework is designed to estimate the pose, leveraging the robustness of feature matching and the precision of direct alignment. Specifically, we develop an event-based 2D-2D alignment to construct the photometric constraint, and tightly integrate it with the event-based reprojection constraint. The mapping module recovers the dense and colorful depth of the scene through the image-guided event-based mapping method. Subsequently, the appearance, texture, and surface mesh of the 3D scene can be reconstructed by fusing the dense depth map from multiple viewpoints using truncated signed distance function (TSDF) fusion. To the best of our knowledge, this is the first non-learning work to realize event-based dense mapping. Numerical evaluations are performed on both publicly available and self-collected datasets, which qualitatively and quantitatively demonstrate the superior performance of our method. Our EVI-SAM effectively balances accuracy and robustness while maintaining computational efficiency, showcasing superior pose tracking and dense mapping performance in challenging scenarios. Video Demo: https://youtu.be/Nn40U4e5Si8.

Title: Adversarial AutoMixup. (arXiv:2312.11954v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.11954
• Code URL: https://github.com/jinxins/adversarial-automixup
• Copy Paste: [[2312.11954]] Adversarial AutoMixup(http://arxiv.org/abs/2312.11954)
• Summary:

  Data mixing augmentation has been widely applied to improve the generalization ability of deep neural networks. Recently, offline data mixing augmentation, e.g. handcrafted and saliency information-based mixup, has been gradually replaced by automatic mixing approaches. Through minimizing two sub-tasks, namely, mixed sample generation and mixup classification in an end-to-end way, AutoMix significantly improves accuracy on image classification tasks. However, as the optimization objective is consistent for the two sub-tasks, this approach is prone to generating consistent instead of diverse mixed samples, which results in overfitting for target task training. In this paper, we propose AdAutomixup, an adversarial automatic mixup augmentation approach that generates challenging samples to train a robust classifier for image classification, by alternatively optimizing the classifier and the mixup sample generator. AdAutomixup comprises two modules, a mixed example generator, and a target classifier. The mixed sample generator aims to produce hard mixed examples to challenge the target classifier while the target classifier`s aim is to learn robust features from hard mixed examples to improve generalization. To prevent the collapse of the inherent meanings of images, we further introduce an exponential moving average (EMA) teacher and cosine similarity to train AdAutomixup in an end-to-end way. Extensive experiments on seven image benchmarks consistently prove that our approach outperforms the state of the art in various classification scenarios.

Title: Context Disentangling and Prototype Inheriting for Robust Visual Grounding. (arXiv:2312.11967v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.11967
• Code URL: null
• Copy Paste: [[2312.11967]] Context Disentangling and Prototype Inheriting for Robust Visual Grounding(http://arxiv.org/abs/2312.11967)
• Summary:

  Visual grounding (VG) aims to locate a specific target in an image based on a given language query. The discriminative information from context is important for distinguishing the target from other objects, particularly for the targets that have the same category as others. However, most previous methods underestimate such information. Moreover, they are usually designed for the standard scene (without any novel object), which limits their generalization to the open-vocabulary scene. In this paper, we propose a novel framework with context disentangling and prototype inheriting for robust visual grounding to handle both scenes. Specifically, the context disentangling disentangles the referent and context features, which achieves better discrimination between them. The prototype inheriting inherits the prototypes discovered from the disentangled visual features by a prototype bank to fully utilize the seen data, especially for the open-vocabulary scene. The fused features, obtained by leveraging Hadamard product on disentangled linguistic and visual features of prototypes to avoid sharp adjusting the importance between the two types of features, are then attached with a special token and feed to a vision Transformer encoder for bounding box regression. Extensive experiments are conducted on both standard and open-vocabulary scenes. The performance comparisons indicate that our method outperforms the state-of-the-art methods in both scenarios. {The code is available at https://github.com/WayneTomas/TransCP.

Title: Variety and Quality over Quantity: Towards Versatile Instruction Curation. (arXiv:2312.11508v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.11508
• Code URL: null
• Copy Paste: [[2312.11508]] Variety and Quality over Quantity: Towards Versatile Instruction Curation(http://arxiv.org/abs/2312.11508)
• Summary:

  Instruction fine-tuning, involving the refinement of pre-trained LLMs using datasets accompanied by natural instructions, is a powerful approach. However, its effectiveness is hindered by the redundancy and deficiencies in LLM-generated instruction datasets. In this paper, we introduce a highly effective and versatile paradigm for selecting diverse and high-quality instruction-following data from fine-tuning datasets. We first employ the dataset enhancement and expansion to augment the dataset with more diverse and high-quality data, then we apply variety compression and quality compression sequentially to curate the desired dataset. Our experimental results showcase that, even with a limited quantity of high-quality instruction data, LLMs consistently maintain robust performance across both natural language understanding tasks and code generation tasks. Notably, they outperform models trained on significantly larger instruction datasets in certain instances.

Title: Regularized Conditional Alignment for Multi-Domain Text Classification. (arXiv:2312.11572v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.11572
• Code URL: null
• Copy Paste: [[2312.11572]] Regularized Conditional Alignment for Multi-Domain Text Classification(http://arxiv.org/abs/2312.11572)
• Summary:

  The most successful multi-domain text classification (MDTC) approaches employ the shared-private paradigm to facilitate the enhancement of domain-invariant features through domain-specific attributes. Additionally, they employ adversarial training to align marginal feature distributions. Nevertheless, these methodologies encounter two primary challenges: (1) Neglecting class-aware information during adversarial alignment poses a risk of misalignment; (2) The limited availability of labeled data across multiple domains fails to ensure adequate discriminative capacity for the model. To tackle these issues, we propose a method called Regularized Conditional Alignment (RCA) to align the joint distributions of domains and classes, thus matching features within the same category and amplifying the discriminative qualities of acquired features. Moreover, we employ entropy minimization and virtual adversarial training to constrain the uncertainty of predictions pertaining to unlabeled data and enhance the model's robustness. Empirical results on two benchmark datasets demonstrate that our RCA approach outperforms state-of-the-art MDTC techniques.

Title: Active Preference Inference using Language Models and Probabilistic Reasoning. (arXiv:2312.12009v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.12009
• Code URL: null
• Copy Paste: [[2312.12009]] Active Preference Inference using Language Models and Probabilistic Reasoning(http://arxiv.org/abs/2312.12009)
• Summary:

  Actively inferring user preferences, for example by asking good questions, is important for any human-facing decision-making system. Active inference allows such systems to adapt and personalize themselves to nuanced individual preferences. To enable this ability for instruction-tuned large language models (LLMs), one may prompt them to ask users questions to infer their preferences, transforming the language models into more robust, interactive systems. However, out of the box, these models are not efficient at extracting preferences: the questions they generate are not informative, requiring a high number of user interactions and impeding the usability of the downstream system. In this work, we introduce an inference-time algorithm that helps LLMs quickly infer preferences by using more informative questions. Our algorithm uses a probabilistic model whose conditional distributions are defined by prompting an LLM, and returns questions that optimize expected entropy and expected model change. Results in a simplified interactive web shopping setting with real product items show that an LLM equipped with our entropy reduction algorithm outperforms baselines with the same underlying LLM on task performance while using fewer user interactions.

Title: Shapley-PC: Constraint-based Causal Structure Learning with Shapley Values. (arXiv:2312.11582v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.11582
• Code URL: https://github.com/briziorusso/shapleypc
• Copy Paste: [[2312.11582]] Shapley-PC: Constraint-based Causal Structure Learning with Shapley Values(http://arxiv.org/abs/2312.11582)
• Summary:

  Causal Structure Learning (CSL), amounting to extracting causal relations among the variables in a dataset, is widely perceived as an important step towards robust and transparent models. Constraint-based CSL leverages conditional independence tests to perform causal discovery. We propose Shapley-PC, a novel method to improve constraint-based CSL algorithms by using Shapley values over the possible conditioning sets to decide which variables are responsible for the observed conditional (in)dependences. We prove soundness and asymptotic consistency and demonstrate that it can outperform state-of-the-art constraint-based, search-based and functional causal model-based methods, according to standard metrics in CSL.

Title: Robust Stochastic Graph Generator for Counterfactual Explanations. (arXiv:2312.11747v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.11747
• Code URL: null
• Copy Paste: [[2312.11747]] Robust Stochastic Graph Generator for Counterfactual Explanations(http://arxiv.org/abs/2312.11747)
• Summary:

  Counterfactual Explanation (CE) techniques have garnered attention as a means to provide insights to the users engaging with AI systems. While extensively researched in domains such as medical imaging and autonomous vehicles, Graph Counterfactual Explanation (GCE) methods have been comparatively under-explored. GCEs generate a new graph similar to the original one, with a different outcome grounded on the underlying predictive model. Among these GCE techniques, those rooted in generative mechanisms have received relatively limited investigation despite demonstrating impressive accomplishments in other domains, such as artistic styles and natural language modelling. The preference for generative explainers stems from their capacity to generate counterfactual instances during inference, leveraging autonomously acquired perturbations of the input graph. Motivated by the rationales above, our study introduces RSGG-CE, a novel Robust Stochastic Graph Generator for Counterfactual Explanations able to produce counterfactual examples from the learned latent space considering a partially ordered generation sequence. Furthermore, we undertake quantitative and qualitative analyses to compare RSGG-CE's performance against SoA generative explainers, highlighting its increased ability to engendering plausible counterfactual candidates.

Title: Clustering Mixtures of Bounded Covariance Distributions Under Optimal Separation. (arXiv:2312.11769v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.11769
• Code URL: null
• Copy Paste: [[2312.11769]] Clustering Mixtures of Bounded Covariance Distributions Under Optimal Separation(http://arxiv.org/abs/2312.11769)
• Summary:

  We study the clustering problem for mixtures of bounded covariance distributions, under a fine-grained separation assumption. Specifically, given samples from a $k$-component mixture distribution $D = \sum_{i =1}^k w_i P_i$, where each $w_i \ge \alpha$ for some known parameter $\alpha$, and each $P_i$ has unknown covariance $\Sigma_i \preceq \sigma^2_i \cdot I_d$ for some unknown $\sigma_i$, the goal is to cluster the samples assuming a pairwise mean separation in the order of $(\sigma_i+\sigma_j)/\sqrt{\alpha}$ between every pair of components $P_i$ and $P_j$. Our contributions are as follows:

  For the special case of nearly uniform mixtures, we give the first poly-time algorithm for this clustering task. Prior work either required separation scaling with the maximum cluster standard deviation (i.e. $\max_i \sigma_i$) [DKK+22b] or required both additional structural assumptions and mean separation scaling as a large degree polynomial in $1/\alpha$ [BKK22].

  For general-weight mixtures, we point out that accurate clustering is information-theoretically impossible under our fine-grained mean separation assumptions. We introduce the notion of a clustering refinement -- a list of not-too-small subsets satisfying a similar separation, and which can be merged into a clustering approximating the ground truth -- and show that it is possible to efficiently compute an accurate clustering refinement of the samples. Furthermore, under a variant of the "no large sub-cluster'' condition from in prior work [BKK22], we show that our algorithm outputs an accurate clustering, not just a refinement, even for general-weight mixtures. As a corollary, we obtain efficient clustering algorithms for mixtures of well-conditioned high-dimensional log-concave distributions.

  Moreover, our algorithm is robust to $\Omega(\alpha)$-fraction of adversarial outliers.

Title: Empowering Dual-Level Graph Self-Supervised Pretraining with Motif Discovery. (arXiv:2312.11927v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.11927
• Code URL: https://github.com/rocccyan/dgpm
• Copy Paste: [[2312.11927]] Empowering Dual-Level Graph Self-Supervised Pretraining with Motif Discovery(http://arxiv.org/abs/2312.11927)
• Summary:

  While self-supervised graph pretraining techniques have shown promising results in various domains, their application still experiences challenges of limited topology learning, human knowledge dependency, and incompetent multi-level interactions. To address these issues, we propose a novel solution, Dual-level Graph self-supervised Pretraining with Motif discovery (DGPM), which introduces a unique dual-level pretraining structure that orchestrates node-level and subgraph-level pretext tasks. Unlike prior approaches, DGPM autonomously uncovers significant graph motifs through an edge pooling module, aligning learned motif similarities with graph kernel-based similarities. A cross-matching task enables sophisticated node-motif interactions and novel representation learning. Extensive experiments on 15 datasets validate DGPM's effectiveness and generalizability, outperforming state-of-the-art methods in unsupervised representation learning and transfer learning settings. The autonomously discovered motifs demonstrate the potential of DGPM to enhance robustness and interpretability.

Title: When Model Meets New Normals: Test-time Adaptation for Unsupervised Time-series Anomaly Detection. (arXiv:2312.11976v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.11976
• Code URL: https://github.com/carrtesy/m2n2
• Copy Paste: [[2312.11976]] When Model Meets New Normals: Test-time Adaptation for Unsupervised Time-series Anomaly Detection(http://arxiv.org/abs/2312.11976)
• Summary:

  Time-series anomaly detection deals with the problem of detecting anomalous timesteps by learning normality from the sequence of observations. However, the concept of normality evolves over time, leading to a "new normal problem", where the distribution of normality can be changed due to the distribution shifts between training and test data. This paper highlights the prevalence of the new normal problem in unsupervised time-series anomaly detection studies. To tackle this issue, we propose a simple yet effective test-time adaptation strategy based on trend estimation and a self-supervised approach to learning new normalities during inference. Extensive experiments on real-world benchmarks demonstrate that incorporating the proposed strategy into the anomaly detector consistently improves the model's performance compared to the baselines, leading to robustness to the distribution shifts.

biometric

Title: EyePreserve: Identity-Preserving Iris Synthesis. (arXiv:2312.12028v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.12028
• Code URL: null
• Copy Paste: [[2312.12028]] EyePreserve: Identity-Preserving Iris Synthesis(http://arxiv.org/abs/2312.12028)
• Summary:

  Synthesis of same-identity biometric iris images, both for existing and non-existing identities while preserving the identity across a wide range of pupil sizes, is complex due to intricate iris muscle constriction mechanism, requiring a precise model of iris non-linear texture deformations to be embedded into the synthesis pipeline. This paper presents the first method of fully data-driven, identity-preserving, pupil size-varying s ynthesis of iris images. This approach is capable of synthesizing images of irises with different pupil sizes representing non-existing identities as well as non-linearly deforming the texture of iris images of existing subjects given the segmentation mask of the target iris image. Iris recognition experiments suggest that the proposed deformation model not only preserves the identity when changing the pupil size but offers better similarity between same-identity iris samples with significant differences in pupil size, compared to state-of-the-art linear and non-linear (bio-mechanical-based) iris deformation models. Two immediate applications of the proposed approach are: (a) synthesis of, or enhancement of the existing biometric datasets for iris recognition, mimicking those acquired with iris sensors, and (b) helping forensic human experts in examining iris image pairs with significant differences in pupil dilation. Source codes and weights of the models are made available with the paper.

steal

extraction

Title: Advancements and Challenges in Arabic Optical Character Recognition: A Comprehensive Survey. (arXiv:2312.11812v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.11812
• Code URL: null
• Copy Paste: [[2312.11812]] Advancements and Challenges in Arabic Optical Character Recognition: A Comprehensive Survey(http://arxiv.org/abs/2312.11812)
• Summary:

  Optical character recognition (OCR) is a vital process that involves the extraction of handwritten or printed text from scanned or printed images, converting it into a format that can be understood and processed by machines. This enables further data processing activities such as searching and editing. The automatic extraction of text through OCR plays a crucial role in digitizing documents, enhancing productivity, improving accessibility, and preserving historical records. This paper seeks to offer an exhaustive review of contemporary applications, methodologies, and challenges associated with Arabic Optical Character Recognition (OCR). A thorough analysis is conducted on prevailing techniques utilized throughout the OCR process, with a dedicated effort to discern the most efficacious approaches that demonstrate enhanced outcomes. To ensure a thorough evaluation, a meticulous keyword-search methodology is adopted, encompassing a comprehensive analysis of articles relevant to Arabic OCR, including both backward and forward citation reviews. In addition to presenting cutting-edge techniques and methods, this paper critically identifies research gaps within the realm of Arabic OCR. By highlighting these gaps, we shed light on potential areas for future exploration and development, thereby guiding researchers toward promising avenues in the field of Arabic OCR. The outcomes of this study provide valuable insights for researchers, practitioners, and stakeholders involved in Arabic OCR, ultimately fostering advancements in the field and facilitating the creation of more accurate and efficient OCR systems for the Arabic language.

Title: Regulating Intermediate 3D Features for Vision-Centric Autonomous Driving. (arXiv:2312.11837v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.11837
• Code URL: https://github.com/cskkxjk/vampire
• Copy Paste: [[2312.11837]] Regulating Intermediate 3D Features for Vision-Centric Autonomous Driving(http://arxiv.org/abs/2312.11837)
• Summary:

  Multi-camera perception tasks have gained significant attention in the field of autonomous driving. However, existing frameworks based on Lift-Splat-Shoot (LSS) in the multi-camera setting cannot produce suitable dense 3D features due to the projection nature and uncontrollable densification process. To resolve this problem, we propose to regulate intermediate dense 3D features with the help of volume rendering. Specifically, we employ volume rendering to process the dense 3D features to obtain corresponding 2D features (e.g., depth maps, semantic maps), which are supervised by associated labels in the training. This manner regulates the generation of dense 3D features on the feature level, providing appropriate dense and unified features for multiple perception tasks. Therefore, our approach is termed Vampire, stands for "Volume rendering As Multi-camera Perception Intermediate feature REgulator". Experimental results on the Occ3D and nuScenes datasets demonstrate that Vampire facilitates fine-grained and appropriate extraction of dense 3D features, and is competitive with existing SOTA methods across diverse downstream perception tasks like 3D occupancy prediction, LiDAR segmentation and 3D objection detection, while utilizing moderate GPU resources. We provide a video demonstration in the supplementary materials and Codes are available at github.com/cskkxjk/Vampire.

Title: Deciphering Compatibility Relationships with Textual Descriptions via Extraction and Explanation. (arXiv:2312.11554v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.11554
• Code URL: null
• Copy Paste: [[2312.11554]] Deciphering Compatibility Relationships with Textual Descriptions via Extraction and Explanation(http://arxiv.org/abs/2312.11554)
• Summary:

  Understanding and accurately explaining compatibility relationships between fashion items is a challenging problem in the burgeoning domain of AI-driven outfit recommendations. Present models, while making strides in this area, still occasionally fall short, offering explanations that can be elementary and repetitive. This work aims to address these shortcomings by introducing the Pair Fashion Explanation (PFE) dataset, a unique resource that has been curated to illuminate these compatibility relationships. Furthermore, we propose an innovative two-stage pipeline model that leverages this dataset. This fine-tuning allows the model to generate explanations that convey the compatibility relationships between items. Our experiments showcase the model's potential in crafting descriptions that are knowledgeable, aligned with ground-truth matching correlations, and that produce understandable and informative descriptions, as assessed by both automatic metrics and human evaluation. Our code and data are released at https://github.com/wangyu-ustc/PairFashionExplanation

Title: Synergistic Anchored Contrastive Pre-training for Few-Shot Relation Extraction. (arXiv:2312.12021v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.12021
• Code URL: https://github.com/aone-nlp/fsre-sacon
• Copy Paste: [[2312.12021]] Synergistic Anchored Contrastive Pre-training for Few-Shot Relation Extraction(http://arxiv.org/abs/2312.12021)
• Summary:

  Few-shot Relation Extraction (FSRE) aims to extract relational facts from a sparse set of labeled corpora. Recent studies have shown promising results in FSRE by employing Pre-trained Language Models (PLMs) within the framework of supervised contrastive learning, which considers both instances and label facts. However, how to effectively harness massive instance-label pairs to encompass the learned representation with semantic richness in this learning paradigm is not fully explored. To address this gap, we introduce a novel synergistic anchored contrastive pre-training framework. This framework is motivated by the insight that the diverse viewpoints conveyed through instance-label pairs capture incomplete yet complementary intrinsic textual semantics. Specifically, our framework involves a symmetrical contrastive objective that encompasses both sentence-anchored and label-anchored contrastive losses. By combining these two losses, the model establishes a robust and uniform representation space. This space effectively captures the reciprocal alignment of feature distributions among instances and relational facts, simultaneously enhancing the maximization of mutual information across diverse perspectives within the same relation. Experimental results demonstrate that our framework achieves significant performance enhancements compared to baseline models in downstream FSRE tasks. Furthermore, our approach exhibits superior adaptability to handle the challenges of domain shift and zero-shot relation extraction. Our code is available online at https://github.com/AONE-NLP/FSRE-SaCon.

membership infer

federate

Title: Provably Convergent Federated Trilevel Learning. (arXiv:2312.11835v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.11835
• Code URL: null
• Copy Paste: [[2312.11835]] Provably Convergent Federated Trilevel Learning(http://arxiv.org/abs/2312.11835)
• Summary:

  Trilevel learning, also called trilevel optimization (TLO), has been recognized as a powerful modelling tool for hierarchical decision process and widely applied in many machine learning applications, such as robust neural architecture search, hyperparameter optimization, and domain adaptation. Tackling TLO problems has presented a great challenge due to their nested decision-making structure. In addition, existing works on TLO face the following key challenges: 1) they all focus on the non-distributed setting, which may lead to privacy breach; 2) they do not offer any non-asymptotic convergence analysis which characterizes how fast an algorithm converges. To address the aforementioned challenges, this paper proposes an asynchronous federated trilevel optimization method to solve TLO problems. The proposed method utilizes $\mu$-cuts to construct a hyper-polyhedral approximation for the TLO problem and solve it in an asynchronous manner. We demonstrate that the proposed $\mu$-cuts are applicable to not only convex functions but also a wide range of non-convex functions that meet the $\mu$-weakly convex assumption. Furthermore, we theoretically analyze the non-asymptotic convergence rate for the proposed method by showing its iteration complexity to obtain $\epsilon$-stationary point is upper bounded by $\mathcal{O}(\frac{1}{\epsilon^2})$. Extensive experiments on real-world datasets have been conducted to elucidate the superiority of the proposed method, e.g., it has a faster convergence rate with a maximum acceleration of approximately 80$\%$.

fair

Title: GroupMixNorm Layer for Learning Fair Models. (arXiv:2312.11969v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.11969
• Code URL: null
• Copy Paste: [[2312.11969]] GroupMixNorm Layer for Learning Fair Models(http://arxiv.org/abs/2312.11969)
• Summary:

  Recent research has identified discriminatory behavior of automated prediction algorithms towards groups identified on specific protected attributes (e.g., gender, ethnicity, age group, etc.). When deployed in real-world scenarios, such techniques may demonstrate biased predictions resulting in unfair outcomes. Recent literature has witnessed algorithms for mitigating such biased behavior mostly by adding convex surrogates of fairness metrics such as demographic parity or equalized odds in the loss function, which are often not easy to estimate. This research proposes a novel in-processing based GroupMixNorm layer for mitigating bias from deep learning models. The GroupMixNorm layer probabilistically mixes group-level feature statistics of samples across different groups based on the protected attribute. The proposed method improves upon several fairness metrics with minimal impact on overall accuracy. Analysis on benchmark tabular and image datasets demonstrates the efficacy of the proposed method in achieving state-of-the-art performance. Further, the experimental analysis also suggests the robustness of the GroupMixNorm layer against new protected attributes during inference and its utility in eliminating bias from a pre-trained network.

interpretability

Title: Extracting Interpretable Local and Global Representations from Attention on Time Series. (arXiv:2312.11466v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.11466
• Code URL: null
• Copy Paste: [[2312.11466]] Extracting Interpretable Local and Global Representations from Attention on Time Series(http://arxiv.org/abs/2312.11466)
• Summary:

  This paper targets two transformer attention based interpretability methods working with local abstraction and global representation, in the context of time series data. We distinguish local and global contexts, and provide a comprehensive framework for both general interpretation options. We discuss their specific instantiation via different methods in detail, also outlining their respective computational implementation and abstraction variants. Furthermore, we provide extensive experimentation demonstrating the efficacy of the presented approaches. In particular, we perform our experiments using a selection of univariate datasets from the UCR UEA time series repository where we both assess the performance of the proposed approaches, as well as their impact on explainability and interpretability/complexity. Here, with an extensive analysis of hyperparameters, the presented approaches demonstrate an significant improvement in interpretability/complexity, while capturing many core decisions of and maintaining a similar performance to the baseline model. Finally, we draw general conclusions outlining and guiding the application of the presented methods.

Title: A Hybrid SOM and K-means Model for Time Series Energy Consumption Clustering. (arXiv:2312.11475v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.11475
• Code URL: null
• Copy Paste: [[2312.11475]] A Hybrid SOM and K-means Model for Time Series Energy Consumption Clustering(http://arxiv.org/abs/2312.11475)
• Summary:

  Energy consumption analysis plays a pivotal role in addressing the challenges of sustainability and resource management. This paper introduces a novel approach to effectively cluster monthly energy consumption patterns by integrating two powerful techniques: Self-organizing maps and K-means clustering. The proposed method aims to exploit the benefits of both of these algorithms to enhance the accuracy and interpretability of clustering results for a dataset in which finding patterns is difficult. The main focus of this study is on a selection of time series energy consumption data from the Smart meters in London dataset. The data was preprocessed and reduced in dimensionality to capture essential temporal patterns while retaining their underlying structures. The SOM algorithm was utilized to extract the central representatives of the consumption patterns for each one of the houses over the course of each month, effectively reducing the dimensionality of the dataset and making it easier for analysis. Subsequently, the obtained SOM centroids were clustered using K-means, a popular centroid-based clustering technique. The experimental results demonstrated a significant silhouette score of 66%, indicating strong intra-cluster cohesion and inter-cluster separation which confirms the effectiveness of the proposed approach in the clustering task.

explainability

watermark

diffusion

Title: Synthetic Shifts to Initial Seed Vector Exposes the Brittle Nature of Latent-Based Diffusion Models. (arXiv:2312.11473v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.11473
• Code URL: null
• Copy Paste: [[2312.11473]] Synthetic Shifts to Initial Seed Vector Exposes the Brittle Nature of Latent-Based Diffusion Models(http://arxiv.org/abs/2312.11473)
• Summary:

  Recent advances in Conditional Diffusion Models have led to substantial capabilities in various domains. However, understanding the impact of variations in the initial seed vector remains an underexplored area of concern. Particularly, latent-based diffusion models display inconsistencies in image generation under standard conditions when initialized with suboptimal initial seed vectors. To understand the impact of the initial seed vector on generated samples, we propose a reliability evaluation framework that evaluates the generated samples of a diffusion model when the initial seed vector is subjected to various synthetic shifts. Our results indicate that slight manipulations to the initial seed vector of the state-of-the-art Stable Diffusion (Rombach et al., 2022) can lead to significant disturbances in the generated samples, consequently creating images without the effect of conditioning variables. In contrast, GLIDE (Nichol et al., 2022) stands out in generating reliable samples even when the initial seed vector is transformed. Thus, our study sheds light on the importance of the selection and the impact of the initial seed vector in the latent-based diffusion model.

Title: Customize-It-3D: High-Quality 3D Creation from A Single Image Using Subject-Specific Knowledge Prior. (arXiv:2312.11535v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.11535
• Code URL: null
• Copy Paste: [[2312.11535]] Customize-It-3D: High-Quality 3D Creation from A Single Image Using Subject-Specific Knowledge Prior(http://arxiv.org/abs/2312.11535)
• Summary:

  In this paper, we present a novel two-stage approach that fully utilizes the information provided by the reference image to establish a customized knowledge prior for image-to-3D generation. While previous approaches primarily rely on a general diffusion prior, which struggles to yield consistent results with the reference image, we propose a subject-specific and multi-modal diffusion model. This model not only aids NeRF optimization by considering the shading mode for improved geometry but also enhances texture from the coarse results to achieve superior refinement. Both aspects contribute to faithfully aligning the 3D content with the subject. Extensive experiments showcase the superiority of our method, Customize-It-3D, outperforming previous works by a substantial margin. It produces faithful 360-degree reconstructions with impressive visual quality, making it well-suited for various applications, including text-to-3D creation.

Title: Diffusion-Based Particle-DETR for BEV Perception. (arXiv:2312.11578v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.11578
• Code URL: null
• Copy Paste: [[2312.11578]] Diffusion-Based Particle-DETR for BEV Perception(http://arxiv.org/abs/2312.11578)
• Summary:

  The Bird-Eye-View (BEV) is one of the most widely-used scene representations for visual perception in Autonomous Vehicles (AVs) due to its well suited compatibility to downstream tasks. For the enhanced safety of AVs, modeling perception uncertainty in BEV is crucial. Recent diffusion-based methods offer a promising approach to uncertainty modeling for visual perception but fail to effectively detect small objects in the large coverage of the BEV. Such degradation of performance can be attributed primarily to the specific network architectures and the matching strategy used when training. Here, we address this problem by combining the diffusion paradigm with current state-of-the-art 3D object detectors in BEV. We analyze the unique challenges of this approach, which do not exist with deterministic detectors, and present a simple technique based on object query interpolation that allows the model to learn positional dependencies even in the presence of the diffusion noise. Based on this, we present a diffusion-based DETR model for object detection that bears similarities to particle methods. Abundant experimentation on the NuScenes dataset shows equal or better performance for our generative approach, compared to deterministic state-of-the-art methods. Our source code will be made publicly available.

Title: TIP: Text-Driven Image Processing with Semantic and Restoration Instructions. (arXiv:2312.11595v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.11595
• Code URL: null
• Copy Paste: [[2312.11595]] TIP: Text-Driven Image Processing with Semantic and Restoration Instructions(http://arxiv.org/abs/2312.11595)
• Summary:

  Text-driven diffusion models have become increasingly popular for various image editing tasks, including inpainting, stylization, and object replacement. However, it still remains an open research problem to adopt this language-vision paradigm for more fine-level image processing tasks, such as denoising, super-resolution, deblurring, and compression artifact removal. In this paper, we develop TIP, a Text-driven Image Processing framework that leverages natural language as a user-friendly interface to control the image restoration process. We consider the capacity of text information in two dimensions. First, we use content-related prompts to enhance the semantic alignment, effectively alleviating identity ambiguity in the restoration outcomes. Second, our approach is the first framework that supports fine-level instruction through language-based quantitative specification of the restoration strength, without the need for explicit task-specific design. In addition, we introduce a novel fusion mechanism that augments the existing ControlNet architecture by learning to rescale the generative prior, thereby achieving better restoration fidelity. Our extensive experiments demonstrate the superior restoration performance of TIP compared to the state of the arts, alongside offering the flexibility of text-based control over the restoration effects.

Title: Unified framework for diffusion generative models in SO(3): applications in computer vision and astrophysics. (arXiv:2312.11707v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.11707
• Code URL: null
• Copy Paste: [[2312.11707]] Unified framework for diffusion generative models in SO(3): applications in computer vision and astrophysics(http://arxiv.org/abs/2312.11707)
• Summary:

  Diffusion-based generative models represent the current state-of-the-art for image generation. However, standard diffusion models are based on Euclidean geometry and do not translate directly to manifold-valued data. In this work, we develop extensions of both score-based generative models (SGMs) and Denoising Diffusion Probabilistic Models (DDPMs) to the Lie group of 3D rotations, SO(3). SO(3) is of particular interest in many disciplines such as robotics, biochemistry and astronomy/cosmology science. Contrary to more general Riemannian manifolds, SO(3) admits a tractable solution to heat diffusion, and allows us to implement efficient training of diffusion models. We apply both SO(3) DDPMs and SGMs to synthetic densities on SO(3) and demonstrate state-of-the-art results. Additionally, we demonstrate the practicality of our model on pose estimation tasks and in predicting correlated galaxy orientations for astrophysics/cosmology.

Title: Text-Image Conditioned Diffusion for Consistent Text-to-3D Generation. (arXiv:2312.11774v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.11774
• Code URL: null
• Copy Paste: [[2312.11774]] Text-Image Conditioned Diffusion for Consistent Text-to-3D Generation(http://arxiv.org/abs/2312.11774)
• Summary:

  By lifting the pre-trained 2D diffusion models into Neural Radiance Fields (NeRFs), text-to-3D generation methods have made great progress. Many state-of-the-art approaches usually apply score distillation sampling (SDS) to optimize the NeRF representations, which supervises the NeRF optimization with pre-trained text-conditioned 2D diffusion models such as Imagen. However, the supervision signal provided by such pre-trained diffusion models only depends on text prompts and does not constrain the multi-view consistency. To inject the cross-view consistency into diffusion priors, some recent works finetune the 2D diffusion model with multi-view data, but still lack fine-grained view coherence. To tackle this challenge, we incorporate multi-view image conditions into the supervision signal of NeRF optimization, which explicitly enforces fine-grained view consistency. With such stronger supervision, our proposed text-to-3D method effectively mitigates the generation of floaters (due to excessive densities) and completely empty spaces (due to insufficient densities). Our quantitative evaluations on the T$^3$Bench dataset demonstrate that our method achieves state-of-the-art performance over existing text-to-3D methods. We will make the code publicly available.

Title: IPAD: Iterative, Parallel, and Diffusion-based Network for Scene Text Recognition. (arXiv:2312.11923v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.11923
• Code URL: null
• Copy Paste: [[2312.11923]] IPAD: Iterative, Parallel, and Diffusion-based Network for Scene Text Recognition(http://arxiv.org/abs/2312.11923)
• Summary:

  Nowadays, scene text recognition has attracted more and more attention due to its diverse applications. Most state-of-the-art methods adopt an encoder-decoder framework with the attention mechanism, autoregressively generating text from left to right. Despite the convincing performance, this sequential decoding strategy constrains inference speed. Conversely, non-autoregressive models provide faster, simultaneous predictions but often sacrifice accuracy. Although utilizing an explicit language model can improve performance, it burdens the computational load. Besides, separating linguistic knowledge from vision information may harm the final prediction. In this paper, we propose an alternative solution, using a parallel and iterative decoder that adopts an easy-first decoding strategy. Furthermore, we regard text recognition as an image-based conditional text generation task and utilize the discrete diffusion strategy, ensuring exhaustive exploration of bidirectional contextual information. Extensive experiments demonstrate that the proposed approach achieves superior results on the benchmark datasets, including both Chinese and English text images.

Title: Optimizing Diffusion Noise Can Serve As Universal Motion Priors. (arXiv:2312.11994v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.11994
• Code URL: null
• Copy Paste: [[2312.11994]] Optimizing Diffusion Noise Can Serve As Universal Motion Priors(http://arxiv.org/abs/2312.11994)
• Summary:

  We propose Diffusion Noise Optimization (DNO), a new method that effectively leverages existing motion diffusion models as motion priors for a wide range of motion-related tasks. Instead of training a task-specific diffusion model for each new task, DNO operates by optimizing the diffusion latent noise of an existing pre-trained text-to-motion model. Given the corresponding latent noise of a human motion, it propagates the gradient from the target criteria defined on the motion space through the whole denoising process to update the diffusion latent noise. As a result, DNO supports any use cases where criteria can be defined as a function of motion. In particular, we show that, for motion editing and control, DNO outperforms existing methods in both achieving the objective and preserving the motion content. DNO accommodates a diverse range of editing modes, including changing trajectory, pose, joint locations, or avoiding newly added obstacles. In addition, DNO is effective in motion denoising and completion, producing smooth and realistic motion from noisy and partial inputs. DNO achieves these results at inference time without the need for model retraining, offering great versatility for any defined reward or loss function on the motion representation.

Title: Diffusing More Objects for Semi-Supervised Domain Adaptation with Less Labeling. (arXiv:2312.12000v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.12000
• Code URL: null
• Copy Paste: [[2312.12000]] Diffusing More Objects for Semi-Supervised Domain Adaptation with Less Labeling(http://arxiv.org/abs/2312.12000)
• Summary:

  For object detection, it is possible to view the prediction of bounding boxes as a reverse diffusion process. Using a diffusion model, the random bounding boxes are iteratively refined in a denoising step, conditioned on the image. We propose a stochastic accumulator function that starts each run with random bounding boxes and combines the slightly different predictions. We empirically verify that this improves detection performance. The improved detections are leveraged on unlabelled images as weighted pseudo-labels for semi-supervised learning. We evaluate the method on a challenging out-of-domain test set. Our method brings significant improvements and is on par with human-selected pseudo-labels, while not requiring any human involvement.

Title: Towards Accurate Guided Diffusion Sampling through Symplectic Adjoint Method. (arXiv:2312.12030v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.12030
• Code URL: null
• Copy Paste: [[2312.12030]] Towards Accurate Guided Diffusion Sampling through Symplectic Adjoint Method(http://arxiv.org/abs/2312.12030)
• Summary:

  Training-free guided sampling in diffusion models leverages off-the-shelf pre-trained networks, such as an aesthetic evaluation model, to guide the generation process. Current training-free guided sampling algorithms obtain the guidance energy function based on a one-step estimate of the clean image. However, since the off-the-shelf pre-trained networks are trained on clean images, the one-step estimation procedure of the clean image may be inaccurate, especially in the early stages of the generation process in diffusion models. This causes the guidance in the early time steps to be inaccurate. To overcome this problem, we propose Symplectic Adjoint Guidance (SAG), which calculates the gradient guidance in two inner stages. Firstly, SAG estimates the clean image via $n$ function calls, where $n$ serves as a flexible hyperparameter that can be tailored to meet specific image quality requirements. Secondly, SAG uses the symplectic adjoint method to obtain the gradients accurately and efficiently in terms of the memory requirements. Extensive experiments demonstrate that SAG generates images with higher qualities compared to the baselines in both guided image and video generation tasks.

Title: Learning a Diffusion Model Policy from Rewards via Q-Score Matching. (arXiv:2312.11752v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.11752
• Code URL: null
• Copy Paste: [[2312.11752]] Learning a Diffusion Model Policy from Rewards via Q-Score Matching(http://arxiv.org/abs/2312.11752)
• Summary:

  Diffusion models have become a popular choice for representing actor policies in behavior cloning and offline reinforcement learning. This is due to their natural ability to optimize an expressive class of distributions over a continuous space. However, previous works fail to exploit the score-based structure of diffusion models, and instead utilize a simple behavior cloning term to train the actor, limiting their ability in the actor-critic setting. In this paper, we focus on off-policy reinforcement learning and propose a new method for learning a diffusion model policy that exploits the linked structure between the score of the policy and the action gradient of the Q-function. We denote this method Q-score matching and provide theoretical justification for this approach. We conduct experiments in simulated environments to demonstrate the effectiveness of our proposed method and compare to popular baselines.

noise learning

data-free

transformer

Title: 3D-LFM: Lifting Foundation Model. (arXiv:2312.11894v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.11894
• Code URL: https://github.com/mosamdabhi/3dlfm
• Copy Paste: [[2312.11894]] 3D-LFM: Lifting Foundation Model(http://arxiv.org/abs/2312.11894)
• Summary:

  The lifting of 3D structure and camera from 2D landmarks is at the cornerstone of the entire discipline of computer vision. Traditional methods have been confined to specific rigid objects, such as those in Perspective-n-Point (PnP) problems, but deep learning has expanded our capability to reconstruct a wide range of object classes (e.g. C3PDO and PAUL) with resilience to noise, occlusions, and perspective distortions. All these techniques, however, have been limited by the fundamental need to establish correspondences across the 3D training data -- significantly limiting their utility to applications where one has an abundance of "in-correspondence" 3D data. Our approach harnesses the inherent permutation equivariance of transformers to manage varying number of points per 3D data instance, withstands occlusions, and generalizes to unseen categories. We demonstrate state of the art performance across 2D-3D lifting task benchmarks. Since our approach can be trained across such a broad class of structures we refer to it simply as a 3D Lifting Foundation Model (3D-LFM) -- the first of its kind.

Title: Text-Conditioned Resampler For Long Form Video Understanding. (arXiv:2312.11897v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.11897
• Code URL: null
• Copy Paste: [[2312.11897]] Text-Conditioned Resampler For Long Form Video Understanding(http://arxiv.org/abs/2312.11897)
• Summary:

  Videos are highly redundant data source and it is often enough to identify a few key moments to solve any given task. In this paper, we present a text-conditioned video resampler (TCR) module that uses a pre-trained and frozen visual encoder and large language model (LLM) to process long video sequences for a task. TCR localises relevant visual features from the video given a text condition and provides them to a LLM to generate a text response. Due to its lightweight design and use of cross-attention, TCR can process more than 100 frames at a time allowing the model to use much longer chunks of video than earlier works. We make the following contributions: (i) we design a transformer-based sampling architecture that can process long videos conditioned on a task, together with a training method that enables it to bridge pre-trained visual and language models; (ii) we empirically validate its efficacy on a wide variety of evaluation tasks, and set a new state-of-the-art on NextQA, EgoSchema, and the EGO4D-LTA challenge; and (iii) we determine tasks which require longer video contexts and that can thus be used effectively for further evaluation of long-range video models.

Title: Transformer Network for Multi-Person Tracking and Re-Identification in Unconstrained Environment. (arXiv:2312.11929v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.11929
• Code URL: null
• Copy Paste: [[2312.11929]] Transformer Network for Multi-Person Tracking and Re-Identification in Unconstrained Environment(http://arxiv.org/abs/2312.11929)
• Summary:

  Multi-object tracking (MOT) has profound applications in a variety of fields, including surveillance, sports analytics, self-driving, and cooperative robotics. Despite considerable advancements, existing MOT methodologies tend to falter when faced with non-uniform movements, occlusions, and appearance-reappearance scenarios of the objects. Recognizing this inadequacy, we put forward an integrated MOT method that not only marries object detection and identity linkage within a singular, end-to-end trainable framework but also equips the model with the ability to maintain object identity links over long periods of time. Our proposed model, named STMMOT, is built around four key modules: 1) candidate proposal generation, which generates object proposals via a vision-transformer encoder-decoder architecture that detects the object from each frame in the video; 2) scale variant pyramid, a progressive pyramid structure to learn the self-scale and cross-scale similarities in multi-scale feature maps; 3) spatio-temporal memory encoder, extracting the essential information from the memory associated with each object under tracking; and 4) spatio-temporal memory decoder, simultaneously resolving the tasks of object detection and identity association for MOT. Our system leverages a robust spatio-temporal memory module that retains extensive historical observations and effectively encodes them using an attention-based aggregator. The uniqueness of STMMOT lies in representing objects as dynamic query embeddings that are updated continuously, which enables the prediction of object states with attention mechanisms and eradicates the need for post-processing.

Title: Labrador: Exploring the Limits of Masked Language Modeling for Laboratory Data. (arXiv:2312.11502v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.11502
• Code URL: null
• Copy Paste: [[2312.11502]] Labrador: Exploring the Limits of Masked Language Modeling for Laboratory Data(http://arxiv.org/abs/2312.11502)
• Summary:

  In this work we introduce Labrador, a pre-trained Transformer model for laboratory data. Labrador and BERT were pre-trained on a corpus of 100 million lab test results from electronic health records (EHRs) and evaluated on various downstream outcome prediction tasks. Both models demonstrate mastery of the pre-training task but neither consistently outperform XGBoost on downstream supervised tasks. Our ablation studies reveal that transfer learning shows limited effectiveness for BERT and achieves marginal success with Labrador. We explore the reasons for the failure of transfer learning and suggest that the data generating process underlying each patient cannot be characterized sufficiently using labs alone, among other factors. We encourage future work to focus on joint modeling of multiple EHR data categories and to include tree-based baselines in their evaluations.

Title: Assessing Logical Reasoning Capabilities of Encoder-Only Transformer Models. (arXiv:2312.11720v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.11720
• Code URL: https://github.com/paulopirozelli/logicalreasoning
• Copy Paste: [[2312.11720]] Assessing Logical Reasoning Capabilities of Encoder-Only Transformer Models(http://arxiv.org/abs/2312.11720)
• Summary:

  Logical reasoning is central to complex human activities, such as thinking, debating, and planning; it is also a central component of many AI systems as well. In this paper, we investigate the extent to which encoder-only transformer language models (LMs) can reason according to logical rules. We ask whether those LMs can deduce theorems in propositional calculus and first-order logic; if their relative success in these problems reflects general logical capabilities; and which layers contribute the most to the task. First, we show for several encoder-only LMs that they can be trained, to a reasonable degree, to determine logical validity on various datasets. Next, by cross-probing fine-tuned models on these datasets, we show that LMs have difficulty in transferring their putative logical reasoning ability, which suggests that they may have learned dataset-specific features, instead of a general capability. Finally, we conduct a layerwise probing experiment, which shows that the hypothesis classification task is mostly solved through higher layers.

Title: Time-Transformer: Integrating Local and Global Features for Better Time Series Generation. (arXiv:2312.11714v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.11714
• Code URL: null
• Copy Paste: [[2312.11714]] Time-Transformer: Integrating Local and Global Features for Better Time Series Generation(http://arxiv.org/abs/2312.11714)
• Summary:

  Generating time series data is a promising approach to address data deficiency problems. However, it is also challenging due to the complex temporal properties of time series data, including local correlations as well as global dependencies. Most existing generative models have failed to effectively learn both the local and global properties of time series data. To address this open problem, we propose a novel time series generative model named 'Time-Transformer AAE', which consists of an adversarial autoencoder (AAE) and a newly designed architecture named 'Time-Transformer' within the decoder. The Time-Transformer first simultaneously learns local and global features in a layer-wise parallel design, combining the abilities of Temporal Convolutional Networks and Transformer in extracting local features and global dependencies respectively. Second, a bidirectional cross attention is proposed to provide complementary guidance across the two branches and achieve proper fusion between local and global features. Experimental results demonstrate that our model can outperform existing state-of-the-art models in 5 out of 6 datasets, specifically on those with data containing both global and local properties. Furthermore, we highlight our model's advantage on handling this kind of data via an artificial dataset. Finally, we show our model's ability to address a real-world problem: data augmentation to support learning with small datasets and imbalanced datasets.

Title: Stronger Graph Transformer with Regularized Attention Scores. (arXiv:2312.11730v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.11730
• Code URL: https://github.com/eugene29/graphgps_edge_regularization
• Copy Paste: [[2312.11730]] Stronger Graph Transformer with Regularized Attention Scores(http://arxiv.org/abs/2312.11730)
• Summary:

  Graph Neural Networks are notorious for its memory consumption. A recent Transformer based GNN called Graph Transformer are shown to obtain superior performances when long range dependencies exist. However, combining graph data and Transformer architecture led to a combinationally worse memory issue. We propose a novel version of "edge regularization technique" that alleviates the need for Positional Encoding and ultimately alleviate GT's out of memory issue. We observe that it is not clear whether having an edge regularization on top of positional encoding is helpful. However, it seems evident when no positional encoding is applied, edge regularization technique indeed stably improves GT's performance.

generative

Title: HAAR: Text-Conditioned Generative Model of 3D Strand-based Human Hairstyles. (arXiv:2312.11666v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.11666
• Code URL: null
• Copy Paste: [[2312.11666]] HAAR: Text-Conditioned Generative Model of 3D Strand-based Human Hairstyles(http://arxiv.org/abs/2312.11666)
• Summary:

  We present HAAR, a new strand-based generative model for 3D human hairstyles. Specifically, based on textual inputs, HAAR produces 3D hairstyles that could be used as production-level assets in modern computer graphics engines. Current AI-based generative models take advantage of powerful 2D priors to reconstruct 3D content in the form of point clouds, meshes, or volumetric functions. However, by using the 2D priors, they are intrinsically limited to only recovering the visual parts. Highly occluded hair structures can not be reconstructed with those methods, and they only model the ''outer shell'', which is not ready to be used in physics-based rendering or simulation pipelines. In contrast, we propose a first text-guided generative method that uses 3D hair strands as an underlying representation. Leveraging 2D visual question-answering (VQA) systems, we automatically annotate synthetic hair models that are generated from a small set of artist-created hairstyles. This allows us to train a latent diffusion model that operates in a common hairstyle UV space. In qualitative and quantitative studies, we demonstrate the capabilities of the proposed model and compare it to existing hairstyle generation approaches.

Title: Self-supervised Learning for Enhancing Geometrical Modeling in 3D-Aware Generative Adversarial Network. (arXiv:2312.11856v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.11856
• Code URL: null
• Copy Paste: [[2312.11856]] Self-supervised Learning for Enhancing Geometrical Modeling in 3D-Aware Generative Adversarial Network(http://arxiv.org/abs/2312.11856)
• Summary:

  3D-aware Generative Adversarial Networks (3D-GANs) currently exhibit artifacts in their 3D geometrical modeling, such as mesh imperfections and holes. These shortcomings are primarily attributed to the limited availability of annotated 3D data, leading to a constrained "valid latent area" for satisfactory modeling. To address this, we present a Self-Supervised Learning (SSL) technique tailored as an auxiliary loss for any 3D-GAN, designed to improve its 3D geometrical modeling capabilities. Our approach pioneers an inversion technique for 3D-GANs, integrating an encoder that performs adaptive spatially-varying range operations. Utilizing this inversion, we introduce the Cyclic Generative Constraint (CGC), aiming to densify the valid latent space. The CGC operates via augmented local latent vectors that maintain the same geometric form, and it imposes constraints on the cycle path outputs, specifically the generator-encoder-generator sequence. This SSL methodology seamlessly integrates with the inherent GAN loss, ensuring the integrity of pre-existing 3D-GAN architectures without necessitating alterations. We validate our approach with comprehensive experiments across various datasets and architectures, underscoring its efficacy. Our project website: https://3dgan-ssl.github.io

Title: ToViLaG: Your Visual-Language Generative Model is Also An Evildoer. (arXiv:2312.11523v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.11523
• Code URL: https://github.com/victorup/ToViLaG
• Copy Paste: [[2312.11523]] ToViLaG: Your Visual-Language Generative Model is Also An Evildoer(http://arxiv.org/abs/2312.11523)
• Summary:

  Warning: this paper includes model outputs showing offensive content. Recent large-scale Visual-Language Generative Models (VLGMs) have achieved unprecedented improvement in multimodal image/text generation. However, these models might also generate toxic content, e.g., offensive text and pornography images, raising significant ethical risks. Despite exhaustive studies on toxic degeneration of language models, this problem remains largely unexplored within the context of visual-language generation. This work delves into the propensity for toxicity generation and susceptibility to toxic data across various VLGMs. For this purpose, we built ToViLaG, a dataset comprising 32K co-toxic/mono-toxic text-image pairs and 1K innocuous but evocative text that tends to stimulate toxicity. Furthermore, we propose WInToRe, a novel toxicity metric tailored to visual-language generation, which theoretically reflects different aspects of toxicity considering both input and output. On such a basis, we benchmarked the toxicity of a diverse spectrum of VLGMs and discovered that some models do more evil than expected while some are more vulnerable to infection, underscoring the necessity of VLGMs detoxification. Therefore, we develop an innovative bottleneck-based detoxification method. Our method could reduce toxicity while maintaining comparable generation quality, providing a promising initial solution to this line of research.

Title: Topic-VQ-VAE: Leveraging Latent Codebooks for Flexible Topic-Guided Document Generation. (arXiv:2312.11532v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.11532
• Code URL: null
• Copy Paste: [[2312.11532]] Topic-VQ-VAE: Leveraging Latent Codebooks for Flexible Topic-Guided Document Generation(http://arxiv.org/abs/2312.11532)
• Summary:

  This paper introduces a novel approach for topic modeling utilizing latent codebooks from Vector-Quantized Variational Auto-Encoder~(VQ-VAE), discretely encapsulating the rich information of the pre-trained embeddings such as the pre-trained language model. From the novel interpretation of the latent codebooks and embeddings as conceptual bag-of-words, we propose a new generative topic model called Topic-VQ-VAE~(TVQ-VAE) which inversely generates the original documents related to the respective latent codebook. The TVQ-VAE can visualize the topics with various generative distributions including the traditional BoW distribution and the autoregressive image generation. Our experimental results on document analysis and image generation demonstrate that TVQ-VAE effectively captures the topic context which reveals the underlying structures of the dataset and supports flexible forms of document generation. Official implementation of the proposed TVQ-VAE is available at https://github.com/clovaai/TVQ-VAE.

Title: COPD-FlowNet: Elevating Non-invasive COPD Diagnosis with CFD Simulations. (arXiv:2312.11561v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.11561
• Code URL: null
• Copy Paste: [[2312.11561]] COPD-FlowNet: Elevating Non-invasive COPD Diagnosis with CFD Simulations(http://arxiv.org/abs/2312.11561)
• Summary:

  Chronic Obstructive Pulmonary Disorder (COPD) is a prevalent respiratory disease that significantly impacts the quality of life of affected individuals. This paper presents COPDFlowNet, a novel deep-learning framework that leverages a custom Generative Adversarial Network (GAN) to generate synthetic Computational Fluid Dynamics (CFD) velocity flow field images specific to the trachea of COPD patients. These synthetic images serve as a valuable resource for data augmentation and model training. Additionally, COPDFlowNet incorporates a custom Convolutional Neural Network (CNN) architecture to predict the location of the obstruction site.

large language model

Title: Assessing GPT4-V on Structured Reasoning Tasks. (arXiv:2312.11524v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.11524
• Code URL: null
• Copy Paste: [[2312.11524]] Assessing GPT4-V on Structured Reasoning Tasks(http://arxiv.org/abs/2312.11524)
• Summary:

  Multi-modality promises to unlock further uses for large language models. Recently, the state-of-the-art language model GPT-4 was enhanced with vision capabilities. We carry out a prompting evaluation of GPT-4V and five other baselines on structured reasoning tasks, such as mathematical reasoning, visual data analysis, and code generation. We show that visual Chain-of-Thought, an extension of Chain-of-Thought to multi-modal LLMs, yields significant improvements over the vanilla model. We also present a categorized analysis of scenarios where these models perform well and where they struggle, highlighting challenges associated with coherent multimodal reasoning.

Title: Learning Interpretable Queries for Explainable Image Classification with Information Pursuit. (arXiv:2312.11548v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.11548
• Code URL: null
• Copy Paste: [[2312.11548]] Learning Interpretable Queries for Explainable Image Classification with Information Pursuit(http://arxiv.org/abs/2312.11548)
• Summary:

  Information Pursuit (IP) is an explainable prediction algorithm that greedily selects a sequence of interpretable queries about the data in order of information gain, updating its posterior at each step based on observed query-answer pairs. The standard paradigm uses hand-crafted dictionaries of potential data queries curated by a domain expert or a large language model after a human prompt. However, in practice, hand-crafted dictionaries are limited by the expertise of the curator and the heuristics of prompt engineering. This paper introduces a novel approach: learning a dictionary of interpretable queries directly from the dataset. Our query dictionary learning problem is formulated as an optimization problem by augmenting IP's variational formulation with learnable dictionary parameters. To formulate learnable and interpretable queries, we leverage the latent space of large vision and language models like CLIP. To solve the optimization problem, we propose a new query dictionary learning algorithm inspired by classical sparse dictionary learning. Our experiments demonstrate that learned dictionaries significantly outperform hand-crafted dictionaries generated with large language models.

Title: StarVector: Generating Scalable Vector Graphics Code from Images. (arXiv:2312.11556v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.11556
• Code URL: null
• Copy Paste: [[2312.11556]] StarVector: Generating Scalable Vector Graphics Code from Images(http://arxiv.org/abs/2312.11556)
• Summary:

  Scalable Vector Graphics (SVGs) have become integral in modern image rendering applications due to their infinite scalability in resolution, versatile usability, and editing capabilities. SVGs are particularly popular in the fields of web development and graphic design. Existing approaches for SVG modeling using deep learning often struggle with generating complex SVGs and are restricted to simpler ones that require extensive processing and simplification. This paper introduces StarVector, a multimodal SVG generation model that effectively integrates Code Generation Large Language Models (CodeLLMs) and vision models. Our approach utilizes a CLIP image encoder to extract visual representations from pixel-based images, which are then transformed into visual tokens via an adapter module. These visual tokens are pre-pended to the SVG token embeddings, and the sequence is modeled by the StarCoder model using next-token prediction, effectively learning to align the visual and code tokens. This enables StarVector to generate unrestricted SVGs that accurately represent pixel images. To evaluate StarVector's performance, we present SVG-Bench, a comprehensive benchmark for evaluating SVG methods across multiple datasets and relevant metrics. Within this benchmark, we introduce novel datasets including SVG-Stack, a large-scale dataset of real-world SVG examples, and use it to pre-train StarVector as a large foundation model for SVGs. Our results demonstrate significant enhancements in visual quality and complexity handling over current methods, marking a notable advancement in SVG generation technology. Code and models: https://github.com/joanrod/star-vector

Title: ComplexityNet: Increasing LLM Inference Efficiency by Learning Task Complexity. (arXiv:2312.11511v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.11511
• Code URL: null
• Copy Paste: [[2312.11511]] ComplexityNet: Increasing LLM Inference Efficiency by Learning Task Complexity(http://arxiv.org/abs/2312.11511)
• Summary:

  We present ComplexityNet, a streamlined language model designed for assessing task complexity. This model predicts the likelihood of accurate output by various language models, each with different capabilities. Our initial application of ComplexityNet involves the Mostly Basic Python Problems (MBPP) dataset. We pioneered the creation of the first set of labels to define task complexity. ComplexityNet achieved a notable 79% accuracy in determining task complexity, a significant improvement over the 34% accuracy of the original, non fine-tuned model. Furthermore, ComplexityNet effectively reduces computational resource usage by 90% compared to using the highest complexity model, while maintaining a high code generation accuracy of 86.7%. This study demonstrates that fine-tuning smaller models to categorize tasks based on their complexity can lead to a more balanced trade-off between accuracy and efficiency in the use of Large Language Models. Our findings suggest a promising direction for optimizing LLM applications, especially in resource-constrained environments.

Title: LLM in a flash: Efficient Large Language Model Inference with Limited Memory. (arXiv:2312.11514v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.11514
• Code URL: null
• Copy Paste: [[2312.11514]] LLM in a flash: Efficient Large Language Model Inference with Limited Memory(http://arxiv.org/abs/2312.11514)
• Summary:

  Large language models (LLMs) are central to modern natural language processing, delivering exceptional performance in various tasks. However, their intensive computational and memory requirements present challenges, especially for devices with limited DRAM capacity. This paper tackles the challenge of efficiently running LLMs that exceed the available DRAM capacity by storing the model parameters on flash memory but bringing them on demand to DRAM. Our method involves constructing an inference cost model that harmonizes with the flash memory behavior, guiding us to optimize in two critical areas: reducing the volume of data transferred from flash and reading data in larger, more contiguous chunks. Within this flash memory-informed framework, we introduce two principal techniques. First, "windowing'" strategically reduces data transfer by reusing previously activated neurons, and second, "row-column bundling", tailored to the sequential data access strengths of flash memory, increases the size of data chunks read from flash memory. These methods collectively enable running models up to twice the size of the available DRAM, with a 4-5x and 20-25x increase in inference speed compared to naive loading approaches in CPU and GPU, respectively. Our integration of sparsity awareness, context-adaptive loading, and a hardware-oriented design paves the way for effective inference of LLMs on devices with limited memory.

Title: User Modeling in the Era of Large Language Models: Current Research and Future Directions. (arXiv:2312.11518v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.11518
• Code URL: null
• Copy Paste: [[2312.11518]] User Modeling in the Era of Large Language Models: Current Research and Future Directions(http://arxiv.org/abs/2312.11518)
• Summary:

  User modeling (UM) aims to discover patterns or learn representations from user data about the characteristics of a specific user, such as profile, preference, and personality. The user models enable personalization and suspiciousness detection in many online applications such as recommendation, education, and healthcare. Two common types of user data are text and graph, as the data usually contain a large amount of user-generated content (UGC) and online interactions. The research of text and graph mining is developing rapidly, contributing many notable solutions in the past two decades. Recently, large language models (LLMs) have shown superior performance on generating, understanding, and even reasoning over text data. The approaches of user modeling have been equipped with LLMs and soon become outstanding. This article summarizes existing research about how and why LLMs are great tools of modeling and understanding UGC. Then it reviews a few categories of large language models for user modeling (LLM-UM) approaches that integrate the LLMs with text and graph-based methods in different ways. Then it introduces specific LLM-UM techniques for a variety of UM applications. Finally, it presents remaining challenges and future directions in the LLM-UM research. We maintain the reading list at: https://github.com/TamSiuhin/LLM-UM-Reading

Title: Large Language Models are Complex Table Parsers. (arXiv:2312.11521v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.11521
• Code URL: null
• Copy Paste: [[2312.11521]] Large Language Models are Complex Table Parsers(http://arxiv.org/abs/2312.11521)
• Summary:

  With the Generative Pre-trained Transformer 3.5 (GPT-3.5) exhibiting remarkable reasoning and comprehension abilities in Natural Language Processing (NLP), most Question Answering (QA) research has primarily centered around general QA tasks based on GPT, neglecting the specific challenges posed by Complex Table QA. In this paper, we propose to incorporate GPT-3.5 to address such challenges, in which complex tables are reconstructed into tuples and specific prompt designs are employed for dialogues. Specifically, we encode each cell's hierarchical structure, position information, and content as a tuple. By enhancing the prompt template with an explanatory description of the meaning of each tuple and the logical reasoning process of the task, we effectively improve the hierarchical structure awareness capability of GPT-3.5 to better parse the complex tables. Extensive experiments and results on Complex Table QA datasets, i.e., the open-domain dataset HiTAB and the aviation domain dataset AIT-QA show that our approach significantly outperforms previous work on both datasets, leading to state-of-the-art (SOTA) performance.

Title: Are you talking to ['xem'] or ['x', 'em']? On Tokenization and Addressing Misgendering in LLMs with Pronoun Tokenization Parity. (arXiv:2312.11779v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.11779
• Code URL: null
• Copy Paste: [[2312.11779]] Are you talking to ['xem'] or ['x', 'em']? On Tokenization and Addressing Misgendering in LLMs with Pronoun Tokenization Parity(http://arxiv.org/abs/2312.11779)
• Summary:

  A large body of NLP research has documented the ways gender biases manifest and amplify within large language models (LLMs), though this research has predominantly operated within a gender binary-centric context. A growing body of work has identified the harmful limitations of this gender-exclusive framing; many LLMs cannot correctly and consistently refer to persons outside the gender binary, especially if they use neopronouns. While data scarcity has been identified as a possible culprit, the precise mechanisms through which it influences LLM misgendering remain underexplored. Our work addresses this gap by studying data scarcity's role in subword tokenization and, consequently, the formation of LLM word representations. We uncover how the Byte-Pair Encoding (BPE) tokenizer, a backbone for many popular LLMs, contributes to neopronoun misgendering through out-of-vocabulary behavior. We introduce pronoun tokenization parity (PTP), a novel approach to reduce LLM neopronoun misgendering by preserving a token's functional structure. We evaluate PTP's efficacy using pronoun consistency-based metrics and a novel syntax-based metric. Through several controlled experiments, finetuning LLMs with PTP improves neopronoun consistency from 14.5% to 58.4%, highlighting the significant role tokenization plays in LLM pronoun consistency.

Title: Zero-Shot Fact-Checking with Semantic Triples and Knowledge Graphs. (arXiv:2312.11785v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.11785
• Code URL: null
• Copy Paste: [[2312.11785]] Zero-Shot Fact-Checking with Semantic Triples and Knowledge Graphs(http://arxiv.org/abs/2312.11785)
• Summary:

  Despite progress in automated fact-checking, most systems require a significant amount of labeled training data, which is expensive. In this paper, we propose a novel zero-shot method, which instead of operating directly on the claim and evidence sentences, decomposes them into semantic triples augmented using external knowledge graphs, and uses large language models trained for natural language inference. This allows it to generalize to adversarial datasets and domains that supervised models require specific training data for. Our empirical results show that our approach outperforms previous zero-shot approaches on FEVER, FEVER-Symmetric, FEVER 2.0, and Climate-FEVER, while being comparable or better than supervised models on the adversarial and the out-of-domain datasets.

Title: MELO: Enhancing Model Editing with Neuron-Indexed Dynamic LoRA. (arXiv:2312.11795v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.11795
• Code URL: https://github.com/bruthyu/melo
• Copy Paste: [[2312.11795]] MELO: Enhancing Model Editing with Neuron-Indexed Dynamic LoRA(http://arxiv.org/abs/2312.11795)
• Summary:

  Large language models (LLMs) have shown great success in various Natural Language Processing (NLP) tasks, whist they still need updates after deployment to fix errors or keep pace with the changing knowledge in the world. Researchers formulate such problem as Model Editing and have developed various editors focusing on different axes of editing properties. However, current editors can hardly support all properties and rely on heavy computational resources. In this paper, we propose a plug-in Model Editing method based on neuron-indexed dynamic LoRA (MELO), which alters the behavior of language models by dynamically activating certain LoRA blocks according to the index built in an inner vector database. Our method satisfies various editing properties with high efficiency and can be easily integrated into multiple LLM backbones. Experimental results show that our proposed MELO achieves state-of-the-art editing performance on three sequential editing tasks (document classification, question answering and hallucination correction), while requires the least trainable parameters and computational cost.

Title: Designing Guiding Principles for NLP for Healthcare: A Case Study of Maternal Health. (arXiv:2312.11803v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.11803
• Code URL: https://github.com/maria-antoniak/maternal-health-principles
• Copy Paste: [[2312.11803]] Designing Guiding Principles for NLP for Healthcare: A Case Study of Maternal Health(http://arxiv.org/abs/2312.11803)
• Summary:

  Objective: An ethical framework for the use of large language models (LLMs) is urgently needed to shape how natural language processing (NLP) tools are used for healthcare applications. Drawing directly from the voices of those most affected, we propose a set of guiding principles for the use of NLP in healthcare, with examples based on applications in maternal health.

  Materials and Methods: We led an interactive session centered on an LLM-based chatbot demonstration during a full-day workshop with 39 participants, and additionally surveyed 30 healthcare workers and 30 birthing people about their values, needs, and perceptions of AI and LLMs. We conducted quantitative and qualitative analyses of the interactive discussions to consolidate our findings into a set of guiding principles.

  Results: Using the case study of maternal health, we propose nine principles for ethical use of LLMs, grouped into three categories: (i) contextual significance, (ii) measurements, and (iii) who/what is valued. We describe rationales underlying these principles and provide practical advice.

  Discussion: Healthcare faces existing challenges including the balance of power in clinician-patient relationships, systemic health disparities, historical injustices, and economic constraints. Our principles serve as a framework for surfacing key considerations when deploying LLMs in medicine, as well as providing a methodological pattern for other researchers to follow.

  Conclusion: This set of principles can serve as a resource to practitioners working on maternal health and other healthcare fields to emphasize the importance of technical nuance, historical context, and inclusive design when developing LLMs for use in clinical settings.

Title: An Adaptive Placement and Parallelism Framework for Accelerating RLHF Training. (arXiv:2312.11819v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.11819
• Code URL: null
• Copy Paste: [[2312.11819]] An Adaptive Placement and Parallelism Framework for Accelerating RLHF Training(http://arxiv.org/abs/2312.11819)
• Summary:

  Recently, ChatGPT or InstructGPT like large language models (LLM) has made a significant impact in the AI world. These models are incredibly versatile, capable of performing language tasks on par or even exceeding the capabilities of human experts. Many works have attempted to reproduce the complex InstructGPT's RLHF (Reinforcement Learning with Human Feedback) training pipeline. However, the mainstream distributed RLHF training methods typically adopt a fixed model placement strategy, referred to as the Flattening strategy. This strategy treats all four models involved in RLHF as a single entity and places them on all devices, regardless of their differences. Unfortunately, this strategy exacerbates the generation bottlenecks in the RLHF training and degrades the overall training efficiency. To address these issues, we propose an adaptive model placement framework that offers two flexible model placement strategies. These strategies allow for the agile allocation of models across devices in a fine-grained manner. The Interleaving strategy helps reduce memory redundancy and communication costs during RLHF training. On the other hand, the Separation strategy improves the throughput of model training by separating the training and generation stages of the RLHF pipeline. Notably, this framework seamlessly integrates with other mainstream techniques for acceleration and enables automatic hyperparameter search. Extensive experiments have demonstrated that our Interleaving and Separation strategies can achieve notable improvements up to 11x, compared to the current state-of-the-art (SOTA) approaches. These experiments encompassed a wide range of training scenarios, involving models of varying sizes and devices of different scales. The results highlight the effectiveness and superiority of our approaches in accelerating the training of distributed RLHF.

Title: A Revisit of Fake News Dataset with Augmented Fact-checking by ChatGPT. (arXiv:2312.11870v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.11870
• Code URL: null
• Copy Paste: [[2312.11870]] A Revisit of Fake News Dataset with Augmented Fact-checking by ChatGPT(http://arxiv.org/abs/2312.11870)
• Summary:

  The proliferation of fake news has emerged as a critical issue in recent years, requiring significant efforts to detect it. However, the existing fake news detection datasets are sourced from human journalists, which are likely to have inherent bias limitations due to the highly subjective nature of this task. In this paper, we revisit the existing fake news dataset verified by human journalists with augmented fact-checking by large language models (ChatGPT), and we name the augmented fake news dataset ChatGPT-FC. We quantitatively analyze the distinctions and resemblances between human journalists and LLM in assessing news subject credibility, news creator credibility, time-sensitive, and political framing. Our findings highlight LLM's potential to serve as a preliminary screening method, offering a promising avenue to mitigate the inherent biases of human journalists and enhance fake news detection.

Title: Sparse is Enough in Fine-tuning Pre-trained Large Language Model. (arXiv:2312.11875v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.11875
• Code URL: https://github.com/song-wx/sift
• Copy Paste: [[2312.11875]] Sparse is Enough in Fine-tuning Pre-trained Large Language Model(http://arxiv.org/abs/2312.11875)
• Summary:

  With the prevalence of pre-training-fine-tuning paradigm, how to efficiently adapt the pre-trained model to the downstream tasks has been an intriguing issue. Parameter-Efficient Fine-Tuning (PEFT) methods have been proposed for low-cost adaptation, including Adapters, Bia-only, and the recently widely used Low-Rank Adaptation. Although these methods have demonstrated their effectiveness to some extent and have been widely applied, the underlying principles are still unclear. In this paper, we reveal the transition of loss landscape in the downstream domain from random initialization to pre-trained initialization, that is, from low-amplitude oscillation to high-amplitude oscillation. The parameter gradients exhibit a property akin to sparsity, where a small fraction of components dominate the total gradient norm, for instance, 1% of the components account for 99% of the gradient. This property ensures that the pre-trained model can easily find a flat minimizer which guarantees the model's ability to generalize even with a low number of trainable parameters. Based on this, we propose a gradient-based sparse fine-tuning algorithm, named Sparse Increment Fine-Tuning (SIFT), and validate its effectiveness on a range of tasks including the GLUE Benchmark and Instruction-tuning. The code is accessible at https://github.com/song-wx/SIFT/.

Title: Difficulty-Focused Contrastive Learning for Knowledge Tracing with a Large Language Model-Based Difficulty Prediction. (arXiv:2312.11890v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.11890
• Code URL: null
• Copy Paste: [[2312.11890]] Difficulty-Focused Contrastive Learning for Knowledge Tracing with a Large Language Model-Based Difficulty Prediction(http://arxiv.org/abs/2312.11890)
• Summary:

  This paper presents novel techniques for enhancing the performance of knowledge tracing (KT) models by focusing on the crucial factor of question and concept difficulty level. Despite the acknowledged significance of difficulty, previous KT research has yet to exploit its potential for model optimization and has struggled to predict difficulty from unseen data. To address these problems, we propose a difficulty-centered contrastive learning method for KT models and a Large Language Model (LLM)-based framework for difficulty prediction. These innovative methods seek to improve the performance of KT models and provide accurate difficulty estimates for unseen data. Our ablation study demonstrates the efficacy of these techniques by demonstrating enhanced KT model performance. Nonetheless, the complex relationship between language and difficulty merits further investigation.

Title: External Knowledge Augmented Polyphone Disambiguation Using Large Language Model. (arXiv:2312.11920v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.11920
• Code URL: null
• Copy Paste: [[2312.11920]] External Knowledge Augmented Polyphone Disambiguation Using Large Language Model(http://arxiv.org/abs/2312.11920)
• Summary:

  One of the key issues in Mandarin Chinese text-to-speech (TTS) systems is polyphone disambiguation when doing grapheme-to-phoneme (G2P) conversion. In this paper, we introduce a novel method to solve the problem as a generation task. Following the trending research of large language models (LLM) and prompt learning, the proposed method consists of three modules. Retrieval module incorporates external knowledge which is a multi-level semantic dictionary of Chinese polyphonic characters to format the sentence into a prompt. Generation module adopts the decoder-only Transformer architecture to induce the target text. Postprocess module corrects the generated text into a valid result if needed. Experimental results show that our method outperforms the existing methods on a public dataset called CPP. We also empirically study the impacts of different templates of the prompt, different sizes of training data, and whether to incorporate external knowledge.

Title: Fluctuation-based Adaptive Structured Pruning for Large Language Models. (arXiv:2312.11983v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.11983
• Code URL: https://github.com/casia-iva-lab/flap
• Copy Paste: [[2312.11983]] Fluctuation-based Adaptive Structured Pruning for Large Language Models(http://arxiv.org/abs/2312.11983)
• Summary:

  Network Pruning is a promising way to address the huge computing resource demands of the deployment and inference of Large Language Models (LLMs). Retraining-free is important for LLMs' pruning methods. However, almost all of the existing retraining-free pruning approaches for LLMs focus on unstructured pruning, which requires specific hardware support for acceleration. In this paper, we propose a novel retraining-free structured pruning framework for LLMs, named FLAP (FLuctuation-based Adaptive Structured Pruning). It is hardware-friendly by effectively reducing storage and enhancing inference speed. For effective structured pruning of LLMs, we highlight three critical elements that demand the utmost attention: formulating structured importance metrics, adaptively searching the global compressed model, and implementing compensation mechanisms to mitigate performance loss. First, FLAP determines whether the output feature map is easily recoverable when a column of weight is removed, based on the fluctuation pruning metric. Then it standardizes the importance scores to adaptively determine the global compressed model structure. At last, FLAP adds additional bias terms to recover the output feature maps using the baseline values. We thoroughly evaluate our approach on a variety of language benchmarks. Without any retraining, our method significantly outperforms the state-of-the-art methods, including LLM-Pruner and the extension of Wanda in structured pruning. The code is released at https://github.com/CASIA-IVA-Lab/FLAP.

Title: Climate Change from Large Language Models. (arXiv:2312.11985v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.11985
• Code URL: null
• Copy Paste: [[2312.11985]] Climate Change from Large Language Models(http://arxiv.org/abs/2312.11985)
• Summary:

  Climate change presents significant challenges to the global community, and it is imperative to raise widespread awareness of the climate crisis and educate users about low-carbon living. Artificial intelligence, particularly large language models (LLMs), have emerged as powerful tools in mitigating the climate crisis, leveraging their extensive knowledge, broad user base, and natural language interaction capabilities. However, despite the growing body of research on climate change, there is a lack of comprehensive assessments of climate crisis knowledge within LLMs. This paper aims to resolve this gap by proposing an automatic evaluation framework. We employ a hybrid approach to data acquisition that combines data synthesis and manual collection to compile a diverse set of questions related to the climate crisis. These questions cover various aspects of climate change, including its causes, impacts, mitigation strategies, and adaptation measures. We then evaluate the model knowledge through prompt engineering based on the collected questions and generated answers. We propose a set of comprehensive metrics to evaluate the climate crisis knowledge, incorporating indicators from 10 different perspectives. Experimental results show that our method is effective in evaluating the knowledge of LLMs regarding the climate crisis. We evaluate several state-of-the-art LLMs and find that their knowledge falls short in terms of timeliness.

Title: Can ChatGPT be Your Personal Medical Assistant?. (arXiv:2312.12006v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.12006
• Code URL: null
• Copy Paste: [[2312.12006]] Can ChatGPT be Your Personal Medical Assistant?(http://arxiv.org/abs/2312.12006)
• Summary:

  The advanced large language model (LLM) ChatGPT has shown its potential in different domains and remains unbeaten due to its characteristics compared to other LLMs. This study aims to evaluate the potential of using a fine-tuned ChatGPT model as a personal medical assistant in the Arabic language. To do so, this study uses publicly available online questions and answering datasets in Arabic language. There are almost 430K questions and answers for 20 disease-specific categories. GPT-3.5-turbo model was fine-tuned with a portion of this dataset. The performance of this fine-tuned model was evaluated through automated and human evaluation. The automated evaluations include perplexity, coherence, similarity, and token count. Native Arabic speakers with medical knowledge evaluated the generated text by calculating relevance, accuracy, precision, logic, and originality. The overall result shows that ChatGPT has a bright future in medical assistance.

Title: Traces of Memorisation in Large Language Models for Code. (arXiv:2312.11658v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.11658
• Code URL: https://github.com/aise-tudelft/llm4code-extraction
• Copy Paste: [[2312.11658]] Traces of Memorisation in Large Language Models for Code(http://arxiv.org/abs/2312.11658)
• Summary:

  Large language models have gained significant popularity because of their ability to generate human-like text and potential applications in various fields, such as Software Engineering. Large language models for code are commonly trained on large unsanitised corpora of source code scraped from the internet. The content of these datasets is memorised and can be extracted by attackers with data extraction attacks. In this work, we explore memorisation in large language models for code and compare the rate of memorisation with large language models trained on natural language. We adopt an existing benchmark for natural language and construct a benchmark for code by identifying samples that are vulnerable to attack. We run both benchmarks against a variety of models, and perform a data extraction attack. We find that large language models for code are vulnerable to data extraction attacks, like their natural language counterparts. From the training data that was identified to be potentially extractable we were able to extract 47% from a CodeGen-Mono-16B code completion model. We also observe that models memorise more, as their parameter count grows, and that their pre-training data are also vulnerable to attack. We also find that data carriers are memorised at a higher rate than regular code or documentation and that different model architectures memorise different samples. Data leakage has severe outcomes, so we urge the research community to further investigate the extent of this phenomenon using a wider range of models and extraction techniques in order to build safeguards to mitigate this issue.

segmentation

Title: Active contours driven by local and global intensity fitting energy with application to SAR image segmentation and its fast solvers. (arXiv:2312.11849v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.11849
• Code URL: null
• Copy Paste: [[2312.11849]] Active contours driven by local and global intensity fitting energy with application to SAR image segmentation and its fast solvers(http://arxiv.org/abs/2312.11849)
• Summary:

  In this paper, we propose a novel variational active contour model based on Aubert-Aujol (AA) denoising model, which hybrides geodesic active contour (GAC) model with active contours without edges (ACWE) model and can be used to segment images corrupted by multiplicative gamma noise. We transform the proposed model into classic ROF model by adding a proximity term. Inspired by a fast denosing algorithm proposed by Jia-Zhao recently, we propose two fast fixed point algorithms to solve SAR image segmentation question. Experimental results for real SAR images show that the proposed image segmentation model can efficiently stop the contours at weak or blurred edges, and can automatically detect the exterior and interior boundaries of images with multiplicative gamma noise. The proposed fast fixed point algorithms are robustness to initialization contour, and can further reduce about 15% of the time needed for algorithm proposed by Goldstein-Osher.

Title: Point Cloud Part Editing: Segmentation, Generation, Assembly, and Selection. (arXiv:2312.11867v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.11867
• Code URL: null
• Copy Paste: [[2312.11867]] Point Cloud Part Editing: Segmentation, Generation, Assembly, and Selection(http://arxiv.org/abs/2312.11867)
• Summary:

  Ideal part editing should guarantee the diversity of edited parts, the fidelity to the remaining parts, and the quality of the results. However, previous methods do not disentangle each part completely, which means the edited parts will affect the others, resulting in poor diversity and fidelity. In addition, some methods lack constraints between parts, which need manual selections of edited results to ensure quality. Therefore, we propose a four-stage process for point cloud part editing: Segmentation, Generation, Assembly, and Selection. Based on this process, we introduce SGAS, a model for part editing that employs two strategies: feature disentanglement and constraint. By independently fitting part-level feature distributions, we realize the feature disentanglement. By explicitly modeling the transformation from object-level distribution to part-level distributions, we realize the feature constraint. Considerable experiments on different datasets demonstrate the efficiency and effectiveness of SGAS on point cloud part editing. In addition, SGAS can be pruned to realize unsupervised part-aware point cloud generation and achieves state-of-the-art results.

Title: Point Cloud Segmentation Using Transfer Learning with RandLA-Net: A Case Study on Urban Areas. (arXiv:2312.11880v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.11880
• Code URL: null
• Copy Paste: [[2312.11880]] Point Cloud Segmentation Using Transfer Learning with RandLA-Net: A Case Study on Urban Areas(http://arxiv.org/abs/2312.11880)
• Summary:

  Urban environments are characterized by complex structures and diverse features, making accurate segmentation of point cloud data a challenging task. This paper presents a comprehensive study on the application of RandLA-Net, a state-of-the-art neural network architecture, for the 3D segmentation of large-scale point cloud data in urban areas. The study focuses on three major Chinese cities, namely Chengdu, Jiaoda, and Shenzhen, leveraging their unique characteristics to enhance segmentation performance.

  To address the limited availability of labeled data for these specific urban areas, we employed transfer learning techniques. We transferred the learned weights from the Sensat Urban and Toronto 3D datasets to initialize our RandLA-Net model. Additionally, we performed class remapping to adapt the model to the target urban areas, ensuring accurate segmentation results.

  The experimental results demonstrate the effectiveness of the proposed approach achieving over 80\% F1 score for each areas in 3D point cloud segmentation. The transfer learning strategy proves to be crucial in overcoming data scarcity issues, providing a robust solution for urban point cloud analysis. The findings contribute to the advancement of point cloud segmentation methods, especially in the context of rapidly evolving Chinese urban areas.