secure

Title: Challenges of Securing Massively Multiplayer Online Games. (arXiv:2311.07887v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.07887
• Code URL: null
• Copy Paste: [[2311.07887]] Challenges of Securing Massively Multiplayer Online Games(http://arxiv.org/abs/2311.07887)
• Summary:

  When it comes to security in the modern world, things have improved a lot since the early 2000s. Hypertext Transfer Protocol Secure (HTTPS) and Transport Layer Security (TLS) have made the transfer of our data across the internet much safer than years prior, and the advent of VPNs and private browsing have only compounded that. However, the gaming industry has been notoriously behind the curve when it comes to security, most notably with Massively Multiplayer Online (MMO) games, which due to the intrinsic nature of their architecture, have an astounding amount of ground to cover. In this paper, the authors discuss the challenges that MMO developers face when trying to design a secure game, as well as some more modern approaches to security that will help improve the industry moving forward. The authors also highlight a few real-life examples of exploits and breaches that have happened and look at how they were mitigated.

Title: Linking QKD testbeds across Europe. (arXiv:2311.08038v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.08038
• Code URL: null
• Copy Paste: [[2311.08038]] Linking QKD testbeds across Europe(http://arxiv.org/abs/2311.08038)
• Summary:

  Quantum-key-distribution (QKD) networks are gaining importance and it has become necessary to analyze the most appropriate methods for their long-distance interconnection. In this paper, four different methods of interconnecting remote QKD networks are proposed. The methods are used to link three different QKD testbeds in Europe, located in Berlin, Madrid, and Poznan. Although long-distance QKD links are only emulated, the used methods can serve as a blueprint for a secure interconnection of distant QKD networks in the future. Specifically, the presented approaches combine, in a transparent way, different fiber and satellite physical media, as well as common standards of key-delivery interfaces. The testbed interconnections are designed to increase the security by utilizing multipath techniques and multiple hybridizations of QKD and post quantum cryptography (PQC) algorithms.

Title: On the Masking-Friendly Designs for Post-Quantum Cryptography. (arXiv:2311.08040v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.08040
• Code URL: null
• Copy Paste: [[2311.08040]] On the Masking-Friendly Designs for Post-Quantum Cryptography(http://arxiv.org/abs/2311.08040)
• Summary:

  Masking is a well-known and provably secure countermeasure against side-channel attacks. However, due to additional redundant computations, integrating masking schemes is expensive in terms of performance. The performance overhead of integrating masking countermeasures is heavily influenced by the design choices of a cryptographic algorithm and is often not considered during the design phase.

  In this work, we deliberate on the effect of design choices on integrating masking techniques into lattice-based cryptography. We select Scabbard, a suite of three lattice-based post-quantum key-encapsulation mechanisms (KEM), namely Florete, Espada, and Sable. We provide arbitrary-order masked implementations of all the constituent KEMs of the Scabbard suite by exploiting their specific design elements. We show that the masked implementations of Florete, Espada, and Sable outperform the masked implementations of Kyber in terms of speed for any order masking. Masked Florete exhibits a $73\%$, $71\%$, and $70\%$ performance improvement over masked Kyber corresponding to the first-, second-, and third-order. Similarly, Espada exhibits $56\%$, $59\%$, and $60\%$ and Sable exhibits $75\%$, $74\%$, and $73\%$ enhanced performance for first-, second-, and third-order masking compared to Kyber respectively. Our results show that the design decisions have a significant impact on the efficiency of integrating masking countermeasures into lattice-based cryptography.

security

Title: VegaEdge: Edge AI Confluence Anomaly Detection for Real-Time Highway IoT-Applications. (arXiv:2311.07880v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.07880
• Code URL: null
• Copy Paste: [[2311.07880]] VegaEdge: Edge AI Confluence Anomaly Detection for Real-Time Highway IoT-Applications(http://arxiv.org/abs/2311.07880)
• Summary:

  Vehicle anomaly detection plays a vital role in highway safety applications such as accident prevention, rapid response, traffic flow optimization, and work zone safety. With the surge of the Internet of Things (IoT) in recent years, there has arisen a pressing demand for Artificial Intelligence (AI) based anomaly detection methods designed to meet the requirements of IoT devices. Catering to this futuristic vision, we introduce a lightweight approach to vehicle anomaly detection by utilizing the power of trajectory prediction. Our proposed design identifies vehicles deviating from expected paths, indicating highway risks from different camera-viewing angles from real-world highway datasets. On top of that, we present VegaEdge - a sophisticated AI confluence designed for real-time security and surveillance applications in modern highway settings through edge-centric IoT-embedded platforms equipped with our anomaly detection approach. Extensive testing across multiple platforms and traffic scenarios showcases the versatility and effectiveness of VegaEdge. This work also presents the Carolinas Anomaly Dataset (CAD), to bridge the existing gap in datasets tailored for highway anomalies. In real-world scenarios, our anomaly detection approach achieves an AUC-ROC of 0.94, and our proposed VegaEdge design, on an embedded IoT platform, processes 738 trajectories per second in a typical highway setting. The dataset is available at https://github.com/TeCSAR-UNCC/Carolinas_Dataset#chd-anomaly-test-set .

Title: Security in Drones. (arXiv:2311.07894v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.07894
• Code URL: null
• Copy Paste: [[2311.07894]] Security in Drones(http://arxiv.org/abs/2311.07894)
• Summary:

  Drones are used in our everyday world for private, commercial, and government uses. It is important to establish both the cyber threats drone users face and security practices to combat those threats. Privacy will always be the main concern when using drones. Protecting information legally collected on drones and protecting people from the illegal collection of their data are topics that security professionals should consider before their organization uses drones. In this article, the authors discuss the importance of security in drones.

Title: LiPar: A Lightweight Parallel Learning Model for Practical In-Vehicle Network Intrusion Detection. (arXiv:2311.08000v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.08000
• Code URL: https://github.com/wangkai-tech23/LiPar
• Copy Paste: [[2311.08000]] LiPar: A Lightweight Parallel Learning Model for Practical In-Vehicle Network Intrusion Detection(http://arxiv.org/abs/2311.08000)
• Summary:

  With the development of intelligent transportation systems, vehicles are exposed to a complex network environment. As the main network of in-vehicle networks, the controller area network (CAN) has many potential security hazards, resulting in higher requirements for intrusion detection systems to ensure safety. Among intrusion detection technologies, methods based on deep learning work best without prior expert knowledge. However, they all have a large model size and rely on cloud computing, and are therefore not suitable to be installed on the in-vehicle network. Therefore, we propose a lightweight parallel neural network structure, LiPar, to allocate task loads to multiple electronic control units (ECU). The LiPar model consists of multi-dimensional branch convolution networks, spatial and temporal feature fusion learning, and a resource adaptation algorithm. Through experiments, we prove that LiPar has great detection performance, running efficiency, and lightweight model size, which can be well adapted to the in-vehicle environment practically and protect the in-vehicle CAN bus security.

privacy

Title: Fuse to Forget: Bias Reduction and Selective Memorization through Model Fusion. (arXiv:2311.07682v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.07682
• Code URL: https://github.com/keremzaman/fusetoforget
• Copy Paste: [[2311.07682]] Fuse to Forget: Bias Reduction and Selective Memorization through Model Fusion(http://arxiv.org/abs/2311.07682)
• Summary:

  Model fusion research aims to aggregate the knowledge of multiple models to enhance performance by combining their weights. In this work, we study the inverse, investigating whether and how can model fusion interfere and reduce unwanted knowledge. We delve into the effects of model fusion on the evolution of learned shortcuts, social biases, and memorization capabilities in fine-tuned language models. Through several experiments covering text classification and generation tasks, our analysis highlights that shared knowledge among models is usually enhanced during model fusion, while unshared knowledge is usually lost or forgotten. Based on this observation, we demonstrate the potential of model fusion as a debiasing tool and showcase its efficacy in addressing privacy concerns associated with language models.

Title: SeDe: Balancing Blockchain Privacy and Regulatory Compliance by Selective De-Anonymization. (arXiv:2311.08167v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.08167
• Code URL: null
• Copy Paste: [[2311.08167]] SeDe: Balancing Blockchain Privacy and Regulatory Compliance by Selective De-Anonymization(http://arxiv.org/abs/2311.08167)
• Summary:

  Privacy is one of the essential pillars for the widespread adoption of blockchains, but public blockchains are transparent by nature. Modern analytics techniques can easily subdue the pseudonymity feature of a blockchain user. Some applications have been able to provide practical privacy protections using privacy-preserving cryptography techniques. However, malicious actors have abused them illicitly, discouraging honest actors from using privacy-preserving applications as "mixing" user interactions and funds with anonymous bad actors, causing compliance and regulatory concerns.

  In this paper, we propose a framework that balances privacy-preserving features by establishing a regulatory and compliant framework called Selective De-Anonymization (SeDe). The adoption of this framework allows privacy-preserving applications on blockchains to de-anonymize illicit transactions by recursive traversal of subgraphs of linked transactions. Our technique achieves this without leaving de-anonymization decisions or control in the hands of a single entity but distributing it among multiple entities while holding them accountable for their respective actions. To instantiate, our framework uses threshold encryption schemes and Zero-Knowledge Proofs (ZKPs).

protect

defense

attack

Title: Cattle Identification Using Muzzle Images and Deep Learning Techniques. (arXiv:2311.08148v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.08148
• Code URL: https://github.com/peter716/animal_biometrics_system
• Copy Paste: [[2311.08148]] Cattle Identification Using Muzzle Images and Deep Learning Techniques(http://arxiv.org/abs/2311.08148)
• Summary:

  Traditional animal identification methods such as ear-tagging, ear notching, and branding have been effective but pose risks to the animal and have scalability issues. Electrical methods offer better tracking and monitoring but require specialized equipment and are susceptible to attacks. Biometric identification using time-immutable dermatoglyphic features such as muzzle prints and iris patterns is a promising solution. This project explores cattle identification using 4923 muzzle images collected from 268 beef cattle. Two deep learning classification models are implemented - wide ResNet50 and VGG16\_BN and image compression is done to lower the image quality and adapt the models to work for the African context. From the experiments run, a maximum accuracy of 99.5\% is achieved while using the wide ResNet50 model with a compression retaining 25\% of the original image. From the study, it is noted that the time required by the models to train and converge as well as recognition time are dependent on the machine used to run the model.

Title: On The Relationship Between Universal Adversarial Attacks And Sparse Representations. (arXiv:2311.08265v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.08265
• Code URL: https://github.com/danawr/adversarial_attacks_and_sparse_representations
• Copy Paste: [[2311.08265]] On The Relationship Between Universal Adversarial Attacks And Sparse Representations(http://arxiv.org/abs/2311.08265)
• Summary:

  The prominent success of neural networks, mainly in computer vision tasks, is increasingly shadowed by their sensitivity to small, barely perceivable adversarial perturbations in image input.

  In this work, we aim at explaining this vulnerability through the framework of sparsity.

  We show the connection between adversarial attacks and sparse representations, with a focus on explaining the universality and transferability of adversarial examples in neural networks.

  To this end, we show that sparse coding algorithms, and the neural network-based learned iterative shrinkage thresholding algorithm (LISTA) among them, suffer from this sensitivity, and that common attacks on neural networks can be expressed as attacks on the sparse representation of the input image. The phenomenon that we observe holds true also when the network is agnostic to the sparse representation and dictionary, and thus can provide a possible explanation for the universality and transferability of adversarial attacks.

  The code is available at https://github.com/danawr/adversarial_attacks_and_sparse_representations.

Title: Input Reconstruction Attack against Vertical Federated Large Language Models. (arXiv:2311.07585v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.07585
• Code URL: null
• Copy Paste: [[2311.07585]] Input Reconstruction Attack against Vertical Federated Large Language Models(http://arxiv.org/abs/2311.07585)
• Summary:

  Recently, large language models (LLMs) have drawn extensive attention from academia and the public, due to the advent of the ChatGPT. While LLMs show their astonishing ability in text generation for various tasks, privacy concerns limit their usage in real-life businesses. More specifically, either the user's inputs (the user sends the query to the model-hosting server) or the model (the user downloads the complete model) itself will be revealed during the usage. Vertical federated learning (VFL) is a promising solution to this kind of problem. It protects both the user's input and the knowledge of the model by splitting the model into a bottom part and a top part, which is maintained by the user and the model provider, respectively. However, in this paper, we demonstrate that in LLMs, VFL fails to protect the user input since it is simple and cheap to reconstruct the input from the intermediate embeddings. Experiments show that even with a commercial GPU, the input sentence can be reconstructed in only one second. We also discuss several possible solutions to enhance the privacy of vertical federated LLMs.

Title: MART: Improving LLM Safety with Multi-round Automatic Red-Teaming. (arXiv:2311.07689v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.07689
• Code URL: null
• Copy Paste: [[2311.07689]] MART: Improving LLM Safety with Multi-round Automatic Red-Teaming(http://arxiv.org/abs/2311.07689)
• Summary:

  Red-teaming is a common practice for mitigating unsafe behaviors in Large Language Models (LLMs), which involves thoroughly assessing LLMs to identify potential flaws and addressing them with responsible and accurate responses. While effective, manual red-teaming is costly, and existing automatic red-teaming typically discovers safety risks without addressing them. In this paper, we propose a Multi-round Automatic Red-Teaming (MART) method, which incorporates both automatic adversarial prompt writing and safe response generation, significantly increasing red-teaming scalability and the safety of the target LLM. Specifically, an adversarial LLM and a target LLM interplay with each other in an iterative manner, where the adversarial LLM aims to generate challenging prompts that elicit unsafe responses from the target LLM, while the target LLM is fine-tuned with safety aligned data on these adversarial prompts. In each round, the adversarial LLM crafts better attacks on the updated target LLM, while the target LLM also improves itself through safety fine-tuning. On adversarial prompt benchmarks, the violation rate of an LLM with limited safety alignment reduces up to 84.7% after 4 rounds of MART, achieving comparable performance to LLMs with extensive adversarial prompt writing. Notably, model helpfulness on non-adversarial prompts remains stable throughout iterations, indicating the target LLM maintains strong performance on instruction following.

Title: A practical key-recovery attack on LWE-based key-encapsulation mechanism schemes using Rowhammer. (arXiv:2311.08027v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.08027
• Code URL: null
• Copy Paste: [[2311.08027]] A practical key-recovery attack on LWE-based key-encapsulation mechanism schemes using Rowhammer(http://arxiv.org/abs/2311.08027)
• Summary:

  Physical attacks are serious threats to cryptosystems deployed in the real world. In this work, we propose a microarchitectural end-to-end attack methodology on generic lattice-based post-quantum key encapsulation mechanisms to recover the long-term secret key. Our attack targets a critical component of a Fujisaki-Okamoto transform that is used in the construction of almost all lattice-based key encapsulation mechanisms. We demonstrate our attack model on practical schemes such as Kyber and Saber by using Rowhammer. We show that our attack is highly practical and imposes little preconditions on the attacker to succeed. As an additional contribution, we propose an improved version of the plaintext checking oracle, which is used by almost all physical attack strategies on lattice-based key-encapsulation mechanisms. Our improvement reduces the number of queries to the plaintext checking oracle by as much as $39\%$ for Saber and approximately $23\%$ for Kyber768. This can be of independent interest and can also be used to reduce the complexity of other attacks.

Title: Laccolith: Hypervisor-Based Adversary Emulation with Anti-Detection. (arXiv:2311.08274v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.08274
• Code URL: null
• Copy Paste: [[2311.08274]] Laccolith: Hypervisor-Based Adversary Emulation with Anti-Detection(http://arxiv.org/abs/2311.08274)
• Summary:

  Advanced Persistent Threats (APTs) represent the most threatening form of attack nowadays since they can stay undetected for a long time. Adversary emulation is a proactive approach for preparing against these attacks. However, adversary emulation tools lack the anti-detection abilities of APTs. We introduce Laccolith, a hypervisor-based solution for adversary emulation with anti-detection to fill this gap. We also present an experimental study to compare Laccolith with MITRE CALDERA, a state-of-the-art solution for adversary emulation, against five popular anti-virus products. We found that CALDERA cannot evade detection, limiting the realism of emulated attacks, even when combined with a state-of-the-art anti-detection framework. Our experiments show that Laccolith can hide its activities from all the tested anti-virus products, thus making it suitable for realistic emulations.

robust

Title: Dual-channel Prototype Network for few-shot Classification of Pathological Images. (arXiv:2311.07871v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.07871
• Code URL: null
• Copy Paste: [[2311.07871]] Dual-channel Prototype Network for few-shot Classification of Pathological Images(http://arxiv.org/abs/2311.07871)
• Summary:

  In pathology, the rarity of certain diseases and the complexity in annotating pathological images significantly hinder the creation of extensive, high-quality datasets. This limitation impedes the progress of deep learning-assisted diagnostic systems in pathology. Consequently, it becomes imperative to devise a technology that can discern new disease categories from a minimal number of annotated examples. Such a technology would substantially advance deep learning models for rare diseases. Addressing this need, we introduce the Dual-channel Prototype Network (DCPN), rooted in the few-shot learning paradigm, to tackle the challenge of classifying pathological images with limited samples. DCPN augments the Pyramid Vision Transformer (PVT) framework for few-shot classification via self-supervised learning and integrates it with convolutional neural networks. This combination forms a dual-channel architecture that extracts multi-scale, highly precise pathological features. The approach enhances the versatility of prototype representations and elevates the efficacy of prototype networks in few-shot pathological image classification tasks. We evaluated DCPN using three publicly available pathological datasets, configuring small-sample classification tasks that mirror varying degrees of clinical scenario domain shifts. Our experimental findings robustly affirm DCPN's superiority in few-shot pathological image classification, particularly in tasks within the same domain, where it achieves the benchmarks of supervised learning.

Title: Towards Improving Robustness Against Common Corruptions in Object Detectors Using Adversarial Contrastive Learning. (arXiv:2311.07928v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.07928
• Code URL: null
• Copy Paste: [[2311.07928]] Towards Improving Robustness Against Common Corruptions in Object Detectors Using Adversarial Contrastive Learning(http://arxiv.org/abs/2311.07928)
• Summary:

  Neural networks have revolutionized various domains, exhibiting remarkable accuracy in tasks like natural language processing and computer vision. However, their vulnerability to slight alterations in input samples poses challenges, particularly in safety-critical applications like autonomous driving. Current approaches, such as introducing distortions during training, fall short in addressing unforeseen corruptions. This paper proposes an innovative adversarial contrastive learning framework to enhance neural network robustness simultaneously against adversarial attacks and common corruptions. By generating instance-wise adversarial examples and optimizing contrastive loss, our method fosters representations that resist adversarial perturbations and remain robust in real-world scenarios. Subsequent contrastive learning then strengthens the similarity between clean samples and their adversarial counterparts, fostering representations resistant to both adversarial attacks and common distortions. By focusing on improving performance under adversarial and real-world conditions, our approach aims to bolster the robustness of neural networks in safety-critical applications, such as autonomous vehicles navigating unpredictable weather conditions. We anticipate that this framework will contribute to advancing the reliability of neural networks in challenging environments, facilitating their widespread adoption in mission-critical scenarios.

Title: Deep Learning-Based Object Detection in Maritime Unmanned Aerial Vehicle Imagery: Review and Experimental Comparisons. (arXiv:2311.07955v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.07955
• Code URL: null
• Copy Paste: [[2311.07955]] Deep Learning-Based Object Detection in Maritime Unmanned Aerial Vehicle Imagery: Review and Experimental Comparisons(http://arxiv.org/abs/2311.07955)
• Summary:

  With the advancement of maritime unmanned aerial vehicles (UAVs) and deep learning technologies, the application of UAV-based object detection has become increasingly significant in the fields of maritime industry and ocean engineering. Endowed with intelligent sensing capabilities, the maritime UAVs enable effective and efficient maritime surveillance. To further promote the development of maritime UAV-based object detection, this paper provides a comprehensive review of challenges, relative methods, and UAV aerial datasets. Specifically, in this work, we first briefly summarize four challenges for object detection on maritime UAVs, i.e., object feature diversity, device limitation, maritime environment variability, and dataset scarcity. We then focus on computational methods to improve maritime UAV-based object detection performance in terms of scale-aware, small object detection, view-aware, rotated object detection, lightweight methods, and others. Next, we review the UAV aerial image/video datasets and propose a maritime UAV aerial dataset named MS2ship for ship detection. Furthermore, we conduct a series of experiments to present the performance evaluation and robustness analysis of object detection methods on maritime datasets. Eventually, we give the discussion and outlook on future works for maritime UAV-based object detection. The MS2ship dataset is available at \href{https://github.com/zcj234/MS2ship}{https://github.com/zcj234/MS2ship}.

Title: Act-VIT: A Representationally Robust Attention Architecture for Skeleton Based Action Recognition Using Vision Transformer. (arXiv:2311.08094v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.08094
• Code URL: null
• Copy Paste: [[2311.08094]] Act-VIT: A Representationally Robust Attention Architecture for Skeleton Based Action Recognition Using Vision Transformer(http://arxiv.org/abs/2311.08094)
• Summary:

  Skeleton-based action recognition receives the attention of many researchers as it is robust to viewpoint and illumination changes, and its processing is much more efficient than video frames. With the emergence of deep learning models, it has become very popular to represent the skeleton data in pseudo-image form and apply Convolutional Neural Networks for action recognition. Thereafter, studies concentrated on finding effective methods for forming pseudo-images. Recently, attention networks, more specifically transformers have provided promising results in various vision problems. In this study, the effectiveness of vision transformers for skeleton-based action recognition is examined and its robustness on the pseudo-image representation scheme is investigated. To this end, a three-level architecture, Act-VIT is proposed, which forms a set of pseudo images apply a classifier on each of the representation and combine their results to find the final action class. The classifiers of Act-VIT are first realized by CNNs and then by VITs and their performances are compared. Experimental studies reveal that the vision transformer is less sensitive to the initial pseudo-image representation compared to CNN. Nevertheless, even with the vision transformer, the recognition performance can be further improved by consensus of classifiers.

Title: Frontier Language Models are not Robust to Adversarial Arithmetic, or "What do I need to say so you agree 2+2=5?. (arXiv:2311.07587v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.07587
• Code URL: null
• Copy Paste: [[2311.07587]] Frontier Language Models are not Robust to Adversarial Arithmetic, or "What do I need to say so you agree 2+2=5?(http://arxiv.org/abs/2311.07587)
• Summary:

  We introduce and study the problem of adversarial arithmetic, which provides a simple yet challenging testbed for language model alignment. This problem is comprised of arithmetic questions posed in natural language, with an arbitrary adversarial string inserted before the question is complete. Even in the simple setting of 1-digit addition problems, it is easy to find adversarial prompts that make all tested models (including PaLM2, GPT4, Claude2) misbehave, and even to steer models to a particular wrong answer. We additionally provide a simple algorithm for finding successful attacks by querying those same models, which we name "prompt inversion rejection sampling" (PIRS). We finally show that models can be partially hardened against these attacks via reinforcement learning and via agentic constitutional loops. However, we were not able to make a language model fully robust against adversarial arithmetic attacks.

Title: Hallucination-minimized Data-to-answer Framework for Financial Decision-makers. (arXiv:2311.07592v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.07592
• Code URL: null
• Copy Paste: [[2311.07592]] Hallucination-minimized Data-to-answer Framework for Financial Decision-makers(http://arxiv.org/abs/2311.07592)
• Summary:

  Large Language Models (LLMs) have been applied to build several automation and personalized question-answering prototypes so far. However, scaling such prototypes to robust products with minimized hallucinations or fake responses still remains an open challenge, especially in niche data-table heavy domains such as financial decision making. In this work, we present a novel Langchain-based framework that transforms data tables into hierarchical textual data chunks to enable a wide variety of actionable question answering. First, the user-queries are classified by intention followed by automated retrieval of the most relevant data chunks to generate customized LLM prompts per query. Next, the custom prompts and their responses undergo multi-metric scoring to assess for hallucinations and response confidence. The proposed system is optimized with user-query intention classification, advanced prompting, data scaling capabilities and it achieves over 90% confidence scores for a variety of user-queries responses ranging from {What, Where, Why, How, predict, trend, anomalies, exceptions} that are crucial for financial decision making applications. The proposed data to answers framework can be extended to other analytical domains such as sales and payroll to ensure optimal hallucination control guardrails.

Title: In-context Learning Generalizes, But Not Always Robustly: The Case of Syntax. (arXiv:2311.07811v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.07811
• Code URL: https://github.com/aaronmueller/syntax-icl
• Copy Paste: [[2311.07811]] In-context Learning Generalizes, But Not Always Robustly: The Case of Syntax(http://arxiv.org/abs/2311.07811)
• Summary:

  In-context learning (ICL) is now a common method for supervising large language models (LLMs): given labeled examples in the input context, the LLM learns to perform the task without weight updates. Despite ICL's prevalence and utility, we understand little about whether models supervised in this manner represent the underlying structure of their tasks, rather than superficial heuristics that only generalize to identically distributed examples. In this study, we investigate the robustness of LLMs supervised via ICL using the test case of sensitivity to syntax, which is a prerequisite for robust language understanding. Our experiments are based on two simple and well-controlled syntactic transformations tasks, where correct out-of-distribution generalization requires an accurate syntactic analysis of the input. We further investigate whether out-of-distribution generalization can be improved via chain-of-thought prompting, where the model is provided with a sequence of intermediate computation steps that illustrate how the task ought to be performed. In experiments with models from the GPT, PaLM, and Llama 2 families, we find large variance across LMs on this fundamental linguistic phenomenon, and that the variance is explained more by the composition of the pre-training corpus and supervision methods than by model size. In particular, we find evidence that models pre-trained on code generalize better, and benefit to a greater extent from chain-of-thought prompting.

Title: Distantly-Supervised Named Entity Recognition with Uncertainty-aware Teacher Learning and Student-student Collaborative Learning. (arXiv:2311.08010v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.08010
• Code URL: null
• Copy Paste: [[2311.08010]] Distantly-Supervised Named Entity Recognition with Uncertainty-aware Teacher Learning and Student-student Collaborative Learning(http://arxiv.org/abs/2311.08010)
• Summary:

  Distantly-Supervised Named Entity Recognition (DS-NER) effectively alleviates the burden of annotation, but meanwhile suffers from the label noise. Recent works attempt to adopt the teacher-student framework to gradually refine the training labels and improve the overall robustness. However, we argue that these teacher-student methods achieve limited performance because poor network calibration produces incorrectly pseudo-labeled samples, leading to error propagation. Therefore, we attempt to mitigate this issue by proposing: (1) Uncertainty-aware Teacher Learning that leverages the prediction uncertainty to guide the selection of pseudo-labels, avoiding the number of incorrect pseudo-labels in the self-training stage. (2) Student-student Collaborative Learning that allows the transfer of reliable labels between two student networks instead of completely relying on all pseudo-labels from its teacher. Meanwhile, this approach allows a full exploration of mislabeled samples rather than simply filtering unreliable pseudo-labeled samples. Extensive experimental results on five DS-NER datasets demonstrate that our method is superior to state-of-the-art teacher-student methods.

Title: DiLoCo: Distributed Low-Communication Training of Language Models. (arXiv:2311.08105v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.08105
• Code URL: null
• Copy Paste: [[2311.08105]] DiLoCo: Distributed Low-Communication Training of Language Models(http://arxiv.org/abs/2311.08105)
• Summary:

  Large language models (LLM) have become a critical component in many applications of machine learning. However, standard approaches to training LLM require a large number of tightly interconnected accelerators, with devices exchanging gradients and other intermediate states at each optimization step. While it is difficult to build and maintain a single computing cluster hosting many accelerators, it might be easier to find several computing clusters each hosting a smaller number of devices. In this work, we propose a distributed optimization algorithm, Distributed Low-Communication (DiLoCo), that enables training of language models on islands of devices that are poorly connected. The approach is a variant of federated averaging, where the number of inner steps is large, the inner optimizer is AdamW, and the outer optimizer is Nesterov momentum. On the widely used C4 dataset, we show that DiLoCo on 8 workers performs as well as fully synchronous optimization while communicating 500 times less. DiLoCo exhibits great robustness to the data distribution of each worker. It is also robust to resources becoming unavailable over time, and vice versa, it can seamlessly leverage resources that become available during training.

Title: Carpe Diem: On the Evaluation of World Knowledge in Lifelong Language Models. (arXiv:2311.08106v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.08106
• Code URL: null
• Copy Paste: [[2311.08106]] Carpe Diem: On the Evaluation of World Knowledge in Lifelong Language Models(http://arxiv.org/abs/2311.08106)
• Summary:

  In an ever-evolving world, the dynamic nature of knowledge presents challenges for language models that are trained on static data, leading to outdated encoded information. However, real-world scenarios require models not only to acquire new knowledge but also to overwrite outdated information into updated ones. To address this under-explored issue, we introduce the temporally evolving question answering benchmark, EvolvingQA - a novel benchmark designed for training and evaluating LMs on an evolving Wikipedia database, where the construction of our benchmark is automated with our pipeline using large language models. Our benchmark incorporates question-answering as a downstream task to emulate real-world applications. Through EvolvingQA, we uncover that existing continual learning baselines have difficulty in updating and forgetting outdated knowledge. Our findings suggest that the models fail to learn updated knowledge due to the small weight gradient. Furthermore, we elucidate that the models struggle mostly on providing numerical or temporal answers to questions asking for updated knowledge. Our work aims to model the dynamic nature of real-world information, offering a robust measure for the evolution-adaptability of language models.

Title: RECALL: A Benchmark for LLMs Robustness against External Counterfactual Knowledge. (arXiv:2311.08147v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.08147
• Code URL: null
• Copy Paste: [[2311.08147]] RECALL: A Benchmark for LLMs Robustness against External Counterfactual Knowledge(http://arxiv.org/abs/2311.08147)
• Summary:

  LLMs and AI chatbots have improved people's efficiency in various fields. However, the necessary knowledge for answering the question may be beyond the models' knowledge boundaries. To mitigate this issue, many researchers try to introduce external knowledge, such as knowledge graphs and Internet contents, into LLMs for up-to-date information. However, the external information from the Internet may include counterfactual information that will confuse the model and lead to an incorrect response. Thus there is a pressing need for LLMs to possess the ability to distinguish reliable information from external knowledge. Therefore, to evaluate the ability of LLMs to discern the reliability of external knowledge, we create a benchmark from existing knowledge bases. Our benchmark consists of two tasks, Question Answering and Text Generation, and for each task, we provide models with a context containing counterfactual information. Evaluation results show that existing LLMs are susceptible to interference from unreliable external knowledge with counterfactual information, and simple intervention methods make limited contributions to the alleviation of this issue.

Title: Fast Chain-of-Thought: A Glance of Future from Parallel Decoding Leads to Answers Faster. (arXiv:2311.08263v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.08263
• Code URL: null
• Copy Paste: [[2311.08263]] Fast Chain-of-Thought: A Glance of Future from Parallel Decoding Leads to Answers Faster(http://arxiv.org/abs/2311.08263)
• Summary:

  In this work, we propose FastCoT, a model-agnostic framework based on parallel decoding without any further training of an auxiliary model or modification to the LLM itself. FastCoT uses a size-varying context window whose size changes with position to conduct parallel decoding and auto-regressive decoding simultaneously, thus fully utilizing GPU computation resources. In FastCoT, the parallel decoding part provides the LLM with a quick glance of the future composed of approximate tokens, which could lead to faster answers compared to regular autoregressive decoding used by causal transformers. We also provide an implementation of parallel decoding within LLM, which supports KV-cache generation and batch processing. Through extensive experiments, we demonstrate that FastCoT saves inference time by nearly 20% with only a negligible performance drop compared to the regular approach. Additionally, we show that the context window size exhibits considerable robustness for different tasks.

Title: Robust and Scalable Hyperdimensional Computing With Brain-Like Neural Adaptations. (arXiv:2311.07705v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.07705
• Code URL: null
• Copy Paste: [[2311.07705]] Robust and Scalable Hyperdimensional Computing With Brain-Like Neural Adaptations(http://arxiv.org/abs/2311.07705)
• Summary:

  The Internet of Things (IoT) has facilitated many applications utilizing edge-based machine learning (ML) methods to analyze locally collected data. Unfortunately, popular ML algorithms often require intensive computations beyond the capabilities of today's IoT devices. Brain-inspired hyperdimensional computing (HDC) has been introduced to address this issue. However, existing HDCs use static encoders, requiring extremely high dimensionality and hundreds of training iterations to achieve reasonable accuracy. This results in a huge efficiency loss, severely impeding the application of HDCs in IoT systems. We observed that a main cause is that the encoding module of existing HDCs lacks the capability to utilize and adapt to information learned during training. In contrast, neurons in human brains dynamically regenerate all the time and provide more useful functionalities when learning new information. While the goal of HDC is to exploit the high-dimensionality of randomly generated base hypervectors to represent the information as a pattern of neural activity, it remains challenging for existing HDCs to support a similar behavior as brain neural regeneration. In this work, we present dynamic HDC learning frameworks that identify and regenerate undesired dimensions to provide adequate accuracy with significantly lowered dimensionalities, thereby accelerating both the training and inference.

Title: Probabilistic Physics-integrated Neural Differentiable Modeling for Isothermal Chemical Vapor Infiltration Process. (arXiv:2311.07798v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.07798
• Code URL: null
• Copy Paste: [[2311.07798]] Probabilistic Physics-integrated Neural Differentiable Modeling for Isothermal Chemical Vapor Infiltration Process(http://arxiv.org/abs/2311.07798)
• Summary:

  Chemical vapor infiltration (CVI) is a widely adopted manufacturing technique used in producing carbon-carbon and carbon-silicon carbide composites. These materials are especially valued in the aerospace and automotive industries for their robust strength and lightweight characteristics. The densification process during CVI critically influences the final performance, quality, and consistency of these composite materials. Experimentally optimizing the CVI processes is challenging due to long experimental time and large optimization space. To address these challenges, this work takes a modeling-centric approach. Due to the complexities and limited experimental data of the isothermal CVI densification process, we have developed a data-driven predictive model using the physics-integrated neural differentiable (PiNDiff) modeling framework. An uncertainty quantification feature has been embedded within the PiNDiff method, bolstering the model's reliability and robustness. Through comprehensive numerical experiments involving both synthetic and real-world manufacturing data, the proposed method showcases its capability in modeling densification during the CVI process. This research highlights the potential of the PiNDiff framework as an instrumental tool for advancing our understanding, simulation, and optimization of the CVI manufacturing process, particularly when faced with sparse data and an incomplete description of the underlying physics.

Title: Mixture of Coupled HMMs for Robust Modeling of Multivariate Healthcare Time Series. (arXiv:2311.07867v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.07867
• Code URL: null
• Copy Paste: [[2311.07867]] Mixture of Coupled HMMs for Robust Modeling of Multivariate Healthcare Time Series(http://arxiv.org/abs/2311.07867)
• Summary:

  Analysis of multivariate healthcare time series data is inherently challenging: irregular sampling, noisy and missing values, and heterogeneous patient groups with different dynamics violating exchangeability. In addition, interpretability and quantification of uncertainty are critically important. Here, we propose a novel class of models, a mixture of coupled hidden Markov models (M-CHMM), and demonstrate how it elegantly overcomes these challenges. To make the model learning feasible, we derive two algorithms to sample the sequences of the latent variables in the CHMM: samplers based on (i) particle filtering and (ii) factorized approximation. Compared to existing inference methods, our algorithms are computationally tractable, improve mixing, and allow for likelihood estimation, which is necessary to learn the mixture model. Experiments on challenging real-world epidemiological and semi-synthetic data demonstrate the advantages of the M-CHMM: improved data fit, capacity to efficiently handle missing and noisy measurements, improved prediction accuracy, and ability to identify interpretable subsets in the data.

Title: Out-of-Distribution Knowledge Distillation via Confidence Amendment. (arXiv:2311.07975v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.07975
• Code URL: https://github.com/lawliet-zzl/ca
• Copy Paste: [[2311.07975]] Out-of-Distribution Knowledge Distillation via Confidence Amendment(http://arxiv.org/abs/2311.07975)
• Summary:

  Out-of-distribution (OOD) detection is essential in identifying test samples that deviate from the in-distribution (ID) data upon which a standard network is trained, ensuring network robustness and reliability. This paper introduces OOD knowledge distillation, a pioneering learning framework applicable whether or not training ID data is available, given a standard network. This framework harnesses OOD-sensitive knowledge from the standard network to craft a binary classifier adept at distinguishing between ID and OOD samples. To accomplish this, we introduce Confidence Amendment (CA), an innovative methodology that transforms an OOD sample into an ID one while progressively amending prediction confidence derived from the standard network. This approach enables the simultaneous synthesis of both ID and OOD samples, each accompanied by an adjusted prediction confidence, thereby facilitating the training of a binary classifier sensitive to OOD. Theoretical analysis provides bounds on the generalization error of the binary classifier, demonstrating the pivotal role of confidence amendment in enhancing OOD sensitivity. Extensive experiments spanning various datasets and network architectures confirm the efficacy of the proposed method in detecting OOD samples.

biometric

steal

extraction

Title: PadChannel: Improving CNN Performance through Explicit Padding Encoding. (arXiv:2311.07623v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.07623
• Code URL: https://github.com/aussieseaweed/pad-channel
• Copy Paste: [[2311.07623]] PadChannel: Improving CNN Performance through Explicit Padding Encoding(http://arxiv.org/abs/2311.07623)
• Summary:

  In convolutional neural networks (CNNs), padding plays a pivotal role in preserving spatial dimensions throughout the layers. Traditional padding techniques do not explicitly distinguish between the actual image content and the padded regions, potentially causing CNNs to incorrectly interpret the boundary pixels or regions that resemble boundaries. This ambiguity can lead to suboptimal feature extraction. To address this, we propose PadChannel, a novel padding method that encodes padding statuses as an additional input channel, enabling CNNs to easily distinguish genuine pixels from padded ones. By incorporating PadChannel into several prominent CNN architectures, we observed small performance improvements and notable reductions in the variances on the ImageNet-1K image classification task at marginal increases in the computational cost. The source code is available at https://github.com/AussieSeaweed/pad-channel

Title: CSLP-AE: A Contrastive Split-Latent Permutation Autoencoder Framework for Zero-Shot Electroencephalography Signal Conversion. (arXiv:2311.07788v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.07788
• Code URL: https://github.com/andersxa/cslp-ae
• Copy Paste: [[2311.07788]] CSLP-AE: A Contrastive Split-Latent Permutation Autoencoder Framework for Zero-Shot Electroencephalography Signal Conversion(http://arxiv.org/abs/2311.07788)
• Summary:

  Electroencephalography (EEG) is a prominent non-invasive neuroimaging technique providing insights into brain function. Unfortunately, EEG data exhibit a high degree of noise and variability across subjects hampering generalizable signal extraction. Therefore, a key aim in EEG analysis is to extract the underlying neural activation (content) as well as to account for the individual subject variability (style). We hypothesize that the ability to convert EEG signals between tasks and subjects requires the extraction of latent representations accounting for content and style. Inspired by recent advancements in voice conversion technologies, we propose a novel contrastive split-latent permutation autoencoder (CSLP-AE) framework that directly optimizes for EEG conversion. Importantly, the latent representations are guided using contrastive learning to promote the latent splits to explicitly represent subject (style) and task (content). We contrast CSLP-AE to conventional supervised, unsupervised (AE), and self-supervised (contrastive learning) training and find that the proposed approach provides favorable generalizable characterizations of subject and task. Importantly, the procedure also enables zero-shot conversion between unseen subjects. While the present work only considers conversion of EEG, the proposed CSLP-AE provides a general framework for signal conversion and extraction of content (task activation) and style (subject variability) components of general interest for the modeling and analysis of biological signals.

Title: Detection of Small Targets in Sea Clutter Based on RepVGG and Continuous Wavelet Transform. (arXiv:2311.07912v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.07912
• Code URL: null
• Copy Paste: [[2311.07912]] Detection of Small Targets in Sea Clutter Based on RepVGG and Continuous Wavelet Transform(http://arxiv.org/abs/2311.07912)
• Summary:

  Constructing a high-performance target detector under the background of sea clutter is always necessary and important. In this work, we propose a RepVGGA0-CWT detector, where RepVGG is a residual network that gains a high detection accuracy. Different from traditional residual networks, RepVGG keeps an acceptable calculation speed. Giving consideration to both accuracy and speed, the RepVGGA0 is selected among all the variants of RepVGG. Also, continuous wavelet transform (CWT) is employed to extract the radar echoes' time-frequency feature effectively. In the tests, other networks (ResNet50, ResNet18 and AlexNet) and feature extraction methods (short-time Fourier transform (STFT), CWT) are combined to build detectors for comparison. The result of different datasets shows that the RepVGGA0-CWT detector performs better than those detectors in terms of low controllable false alarm rate, high training speed, high inference speed and low memory usage. This RepVGGA0-CWT detector is hardware-friendly and can be applied in real-time scenes for its high inference speed in detection.

Title: Learning based Deep Disentangling Light Field Reconstruction and Disparity Estimation Application. (arXiv:2311.08129v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.08129
• Code URL: null
• Copy Paste: [[2311.08129]] Learning based Deep Disentangling Light Field Reconstruction and Disparity Estimation Application(http://arxiv.org/abs/2311.08129)
• Summary:

  Light field cameras have a wide range of uses due to their ability to simultaneously record light intensity and direction. The angular resolution of light fields is important for downstream tasks such as depth estimation, yet is often difficult to improve due to hardware limitations. Conventional methods tend to perform poorly against the challenge of large disparity in sparse light fields, while general CNNs have difficulty extracting spatial and angular features coupled together in 4D light fields. The light field disentangling mechanism transforms the 4D light field into 2D image format, which is more favorable for CNN for feature extraction. In this paper, we propose a Deep Disentangling Mechanism, which inherits the principle of the light field disentangling mechanism and further develops the design of the feature extractor and adds advanced network structure. We design a light-field reconstruction network (i.e., DDASR) on the basis of the Deep Disentangling Mechanism, and achieve SOTA performance in the experiments. In addition, we design a Block Traversal Angular Super-Resolution Strategy for the practical application of depth estimation enhancement where the input views is often higher than 2x2 in the experiments resulting in a high memory usage, which can reduce the memory usage while having a better reconstruction performance.

Title: PolyIE: A Dataset of Information Extraction from Polymer Material Scientific Literature. (arXiv:2311.07715v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.07715
• Code URL: https://github.com/jerry3027/polyie
• Copy Paste: [[2311.07715]] PolyIE: A Dataset of Information Extraction from Polymer Material Scientific Literature(http://arxiv.org/abs/2311.07715)
• Summary:

  Scientific information extraction (SciIE), which aims to automatically extract information from scientific literature, is becoming more important than ever. However, there are no existing SciIE datasets for polymer materials, which is an important class of materials used ubiquitously in our daily lives. To bridge this gap, we introduce POLYIE, a new SciIE dataset for polymer materials. POLYIE is curated from 146 full-length polymer scholarly articles, which are annotated with different named entities (i.e., materials, properties, values, conditions) as well as their N-ary relations by domain experts. POLYIE presents several unique challenges due to diverse lexical formats of entities, ambiguity between entities, and variable-length relations. We evaluate state-of-the-art named entity extraction and relation extraction models on POLYIE, analyze their strengths and weaknesses, and highlight some difficult cases for these models. To the best of our knowledge, POLYIE is the first SciIE benchmark for polymer materials, and we hope it will lead to more research efforts from the community on this challenging task. Our code and data are available on: https://github.com/jerry3027/PolyIE.

Title: Unlocking Science: Novel Dataset and Benchmark for Cross-Modality Scientific Information Extraction. (arXiv:2311.08189v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.08189
• Code URL: null
• Copy Paste: [[2311.08189]] Unlocking Science: Novel Dataset and Benchmark for Cross-Modality Scientific Information Extraction(http://arxiv.org/abs/2311.08189)
• Summary:

  Extracting key information from scientific papers has the potential to help researchers work more efficiently and accelerate the pace of scientific progress. Over the last few years, research on Scientific Information Extraction (SciIE) witnessed the release of several new systems and benchmarks. However, existing paper-focused datasets mostly focus only on specific parts of a manuscript (e.g., abstracts) and are single-modality (i.e., text- or table-only), due to complex processing and expensive annotations. Moreover, core information can be present in either text or tables or across both. To close this gap in data availability and enable cross-modality IE, while alleviating labeling costs, we propose a semi-supervised pipeline for annotating entities in text, as well as entities and relations in tables, in an iterative procedure. Based on this pipeline, we release novel resources for the scientific community, including a high-quality benchmark, a large-scale corpus, and a semi-supervised annotation pipeline. We further report the performance of state-of-the-art IE models on the proposed benchmark dataset, as a baseline. Lastly, we explore the potential capability of large language models such as ChatGPT for the current task. Our new dataset, results, and analysis validate the effectiveness and efficiency of our semi-supervised pipeline, and we discuss its remaining limitations.

Title: Centralized Intermediation in a Decentralized Web3 Economy: Value Accrual and Extraction. (arXiv:2311.08234v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.08234
• Code URL: null
• Copy Paste: [[2311.08234]] Centralized Intermediation in a Decentralized Web3 Economy: Value Accrual and Extraction(http://arxiv.org/abs/2311.08234)
• Summary:

  The advent of Web3 has ushered in a new era of decentralized digital economy, promising a shift from centralized authority to distributed, peer-to-peer interactions. However, the underlying infrastructure of this decentralized ecosystem often relies on centralized cloud providers, creating a paradoxical concentration of value and power. This paper investigates the mechanics of value accrual and extraction within the Web3 ecosystem, focusing on the roles and revenues of centralized clouds. Through an analysis of publicly available material, we elucidate the financial implications of cloud services in purportedly decentralized contexts. We further explore the individual's perspective of value creation and accumulation, examining the interplay between user participation and centralized monetization strategies. Key findings indicate that while blockchain technology has the potential to significantly reduce infrastructure costs for financial services, the current Web3 landscape is marked by a substantial reliance on cloud providers for hosting, scalability, and performance.

membership infer

federate

Title: FedOpenHAR: Federated Multi-Task Transfer Learning for Sensor-Based Human Activity Recognition. (arXiv:2311.07765v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.07765
• Code URL: null
• Copy Paste: [[2311.07765]] FedOpenHAR: Federated Multi-Task Transfer Learning for Sensor-Based Human Activity Recognition(http://arxiv.org/abs/2311.07765)
• Summary:

  Motion sensors integrated into wearable and mobile devices provide valuable information about the device users. Machine learning and, recently, deep learning techniques have been used to characterize sensor data. Mostly, a single task, such as recognition of activities, is targeted, and the data is processed centrally at a server or in a cloud environment. However, the same sensor data can be utilized for multiple tasks and distributed machine-learning techniques can be used without the requirement of the transmission of data to a centre. This paper explores Federated Transfer Learning in a Multi-Task manner for both sensor-based human activity recognition and device position identification tasks. The OpenHAR framework is used to train the models, which contains ten smaller datasets. The aim is to obtain model(s) applicable for both tasks in different datasets, which may include only some label types. Multiple experiments are carried in the Flower federated learning environment using the DeepConvLSTM architecture. Results are presented for federated and centralized versions under different parameters and restrictions. By utilizing transfer learning and training a task-specific and personalized federated model, we obtained a similar accuracy with training each client individually and higher accuracy than a fully centralized approach.

Title: A Data-Free Approach to Mitigate Catastrophic Forgetting in Federated Class Incremental Learning for Vision Tasks. (arXiv:2311.07784v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.07784
• Code URL: null
• Copy Paste: [[2311.07784]] A Data-Free Approach to Mitigate Catastrophic Forgetting in Federated Class Incremental Learning for Vision Tasks(http://arxiv.org/abs/2311.07784)
• Summary:

  Deep learning models often suffer from forgetting previously learned information when trained on new data. This problem is exacerbated in federated learning (FL), where the data is distributed and can change independently for each user. Many solutions are proposed to resolve this catastrophic forgetting in a centralized setting. However, they do not apply directly to FL because of its unique complexities, such as privacy concerns and resource limitations. To overcome these challenges, this paper presents a framework for \textbf{federated class incremental learning} that utilizes a generative model to synthesize samples from past distributions. This data can be later exploited alongside the training data to mitigate catastrophic forgetting. To preserve privacy, the generative model is trained on the server using data-free methods at the end of each task without requesting data from clients. Moreover, our solution does not demand the users to store old data or models, which gives them the freedom to join/leave the training at any time. Additionally, we introduce SuperImageNet, a new regrouping of the ImageNet dataset specifically tailored for federated continual learning. We demonstrate significant improvements compared to existing baselines through extensive experiments on multiple datasets.

Title: Ransomware Detection Using Federated Learning with Imbalanced Datasets. (arXiv:2311.07760v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.07760
• Code URL: null
• Copy Paste: [[2311.07760]] Ransomware Detection Using Federated Learning with Imbalanced Datasets(http://arxiv.org/abs/2311.07760)
• Summary:

  Ransomware is a type of malware which encrypts user data and extorts payments in return for the decryption keys. This cyberthreat is one of the most serious challenges facing organizations today and has already caused immense financial damage. As a result, many researchers have been developing techniques to counter ransomware. Recently, the federated learning (FL) approach has also been applied for ransomware analysis, allowing corporations to achieve scalable, effective detection and attribution without having to share their private data. However, in reality there is much variation in the quantity and composition of ransomware data collected across multiple FL client sites/regions. This imbalance will inevitably degrade the effectiveness of any defense mechanisms. To address this concern, a modified FL scheme is proposed using a weighted cross-entropy loss function approach to mitigate dataset imbalance. A detailed performance evaluation study is then presented for the case of static analysis using the latest Windows-based ransomware families. The findings confirm improved ML classifier performance for a highly imbalanced dataset.

Title: The Impact of Adversarial Node Placement in Decentralized Federated Learning Networks. (arXiv:2311.07946v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.07946
• Code URL: null
• Copy Paste: [[2311.07946]] The Impact of Adversarial Node Placement in Decentralized Federated Learning Networks(http://arxiv.org/abs/2311.07946)
• Summary:

  As Federated Learning (FL) grows in popularity, new decentralized frameworks are becoming widespread. These frameworks leverage the benefits of decentralized environments to enable fast and energy-efficient inter-device communication. However, this growing popularity also intensifies the need for robust security measures. While existing research has explored various aspects of FL security, the role of adversarial node placement in decentralized networks remains largely unexplored. This paper addresses this gap by analyzing the performance of decentralized FL for various adversarial placement strategies when adversaries can jointly coordinate their placement within a network. We establish two baseline strategies for placing adversarial node: random placement and network centrality-based placement. Building on this foundation, we propose a novel attack algorithm that prioritizes adversarial spread over adversarial centrality by maximizing the average network distance between adversaries. We show that the new attack algorithm significantly impacts key performance metrics such as testing accuracy, outperforming the baseline frameworks by between 9% and 66.5% for the considered setups. Our findings provide valuable insights into the vulnerabilities of decentralized FL systems, setting the stage for future research aimed at developing more secure and robust decentralized FL frameworks.

Title: Federated Skewed Label Learning with Logits Fusion. (arXiv:2311.08202v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.08202
• Code URL: null
• Copy Paste: [[2311.08202]] Federated Skewed Label Learning with Logits Fusion(http://arxiv.org/abs/2311.08202)
• Summary:

  Federated learning (FL) aims to collaboratively train a shared model across multiple clients without transmitting their local data. Data heterogeneity is a critical challenge in realistic FL settings, as it causes significant performance deterioration due to discrepancies in optimization among local models. In this work, we focus on label distribution skew, a common scenario in data heterogeneity, where the data label categories are imbalanced on each client. To address this issue, we propose FedBalance, which corrects the optimization bias among local models by calibrating their logits. Specifically, we introduce an extra private weak learner on the client side, which forms an ensemble model with the local model. By fusing the logits of the two models, the private weak learner can capture the variance of different data, regardless of their category. Therefore, the optimization direction of local models can be improved by increasing the penalty for misclassifying minority classes and reducing the attention to majority classes, resulting in a better global model. Extensive experiments show that our method can gain 13\% higher average accuracy compared with state-of-the-art methods.

fair

Title: Finetuning Text-to-Image Diffusion Models for Fairness. (arXiv:2311.07604v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.07604
• Code URL: null
• Copy Paste: [[2311.07604]] Finetuning Text-to-Image Diffusion Models for Fairness(http://arxiv.org/abs/2311.07604)
• Summary:

  The rapid adoption of text-to-image diffusion models in society underscores an urgent need to address their biases. Without interventions, these biases could propagate a distorted worldview and limit opportunities for minority groups. In this work, we frame fairness as a distributional alignment problem. Our solution consists of two main technical contributions: (1) a distributional alignment loss that steers specific characteristics of the generated images towards a user-defined target distribution, and (2) biased direct finetuning of diffusion model's sampling process, which leverages a biased gradient to more effectively optimize losses defined on the generated images. Empirically, our method markedly reduces gender, racial, and their intersectional biases for occupational prompts. Gender bias is significantly reduced even when finetuning just five soft tokens. Crucially, our method supports diverse perspectives of fairness beyond absolute equality, which is demonstrated by controlling age to a $75\%$ young and $25\%$ old distribution while simultaneously debiasing gender and race. Finally, our method is scalable: it can debias multiple concepts at once by simply including these prompts in the finetuning data. We hope our work facilitates the social alignment of T2I generative AI. We will share code and various debiased diffusion model adaptors.

Title: Fair Abstractive Summarization of Diverse Perspectives. (arXiv:2311.07884v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.07884
• Code URL: https://github.com/psunlpgroup/fairsumm
• Copy Paste: [[2311.07884]] Fair Abstractive Summarization of Diverse Perspectives(http://arxiv.org/abs/2311.07884)
• Summary:

  People from different social and demographic groups express diverse perspectives and conflicting opinions on a broad set of topics such as product reviews, healthcare, law, and politics. A fair summary should provide a comprehensive coverage of diverse perspectives without underrepresenting certain groups. However, current work in summarization metrics and Large Language Models (LLMs) evaluation has not explored fair abstractive summarization. In this paper, we systematically investigate fair abstractive summarization for user-generated data. We first formally define fairness in abstractive summarization as not underrepresenting perspectives of any groups of people and propose four reference-free automatic metrics measuring the differences between target and source perspectives. We evaluate five LLMs, including three GPT models, Alpaca, and Claude, on six datasets collected from social media, online reviews, and recorded transcripts. Experiments show that both the model-generated and the human-written reference summaries suffer from low fairness. We conduct a comprehensive analysis of the common factors influencing fairness and propose three simple but effective methods to alleviate unfair summarization. Our dataset and code are available at https://github.com/psunlpgroup/FairSumm.

interpretability

Title: Investigating the Encoding of Words in BERT's Neurons using Feature Textualization. (arXiv:2311.08240v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.08240
• Code URL: null
• Copy Paste: [[2311.08240]] Investigating the Encoding of Words in BERT's Neurons using Feature Textualization(http://arxiv.org/abs/2311.08240)
• Summary:

  Pretrained language models (PLMs) form the basis of most state-of-the-art NLP technologies. Nevertheless, they are essentially black boxes: Humans do not have a clear understanding of what knowledge is encoded in different parts of the models, especially in individual neurons. The situation is different in computer vision, where feature visualization provides a decompositional interpretability technique for neurons of vision models. Activation maximization is used to synthesize inherently interpretable visual representations of the information encoded in individual neurons. Our work is inspired by this but presents a cautionary tale on the interpretability of single neurons, based on the first large-scale attempt to adapt activation maximization to NLP, and, more specifically, large PLMs. We propose feature textualization, a technique to produce dense representations of neurons in the PLM word embedding space. We apply feature textualization to the BERT model (Devlin et al., 2019) to investigate whether the knowledge encoded in individual neurons can be interpreted and symbolized. We find that the produced representations can provide insights about the knowledge encoded in individual neurons, but that individual neurons do not represent clearcut symbolic units of language such as words. Additionally, we use feature textualization to investigate how many neurons are needed to encode words in BERT.

Title: Application of a Dense Fusion Attention Network in Fault Diagnosis of Centrifugal Fan. (arXiv:2311.07614v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.07614
• Code URL: null
• Copy Paste: [[2311.07614]] Application of a Dense Fusion Attention Network in Fault Diagnosis of Centrifugal Fan(http://arxiv.org/abs/2311.07614)
• Summary:

  Although the deep learning recognition model has been widely used in the condition monitoring of rotating machinery. However, it is still a challenge to understand the correspondence between the structure and function of the model and the diagnosis process. Therefore, this paper discusses embedding distributed attention modules into dense connections instead of traditional dense cascading operations. It not only decouples the influence of space and channel on fault feature adaptive recalibration feature weights, but also forms a fusion attention function. The proposed dense fusion focuses on the visualization of the network diagnosis process, which increases the interpretability of model diagnosis. How to continuously and effectively integrate different functions to enhance the ability to extract fault features and the ability to resist noise is answered. Centrifugal fan fault data is used to verify this network. Experimental results show that the network has stronger diagnostic performance than other advanced fault diagnostic models.

Title: Leveraging Hamilton-Jacobi PDEs with time-dependent Hamiltonians for continual scientific machine learning. (arXiv:2311.07790v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.07790
• Code URL: null
• Copy Paste: [[2311.07790]] Leveraging Hamilton-Jacobi PDEs with time-dependent Hamiltonians for continual scientific machine learning(http://arxiv.org/abs/2311.07790)
• Summary:

  We address two major challenges in scientific machine learning (SciML): interpretability and computational efficiency. We increase the interpretability of certain learning processes by establishing a new theoretical connection between optimization problems arising from SciML and a generalized Hopf formula, which represents the viscosity solution to a Hamilton-Jacobi partial differential equation (HJ PDE) with time-dependent Hamiltonian. Namely, we show that when we solve certain regularized learning problems with integral-type losses, we actually solve an optimal control problem and its associated HJ PDE with time-dependent Hamiltonian. This connection allows us to reinterpret incremental updates to learned models as the evolution of an associated HJ PDE and optimal control problem in time, where all of the previous information is intrinsically encoded in the solution to the HJ PDE. As a result, existing HJ PDE solvers and optimal control algorithms can be reused to design new efficient training approaches for SciML that naturally coincide with the continual learning framework, while avoiding catastrophic forgetting. As a first exploration of this connection, we consider the special case of linear regression and leverage our connection to develop a new Riccati-based methodology for solving these learning problems that is amenable to continual learning applications. We also provide some corresponding numerical examples that demonstrate the potential computational and memory advantages our Riccati-based approach can provide.

explainability

Title: Non-autoregressive Machine Translation with Probabilistic Context-free Grammar. (arXiv:2311.07941v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.07941
• Code URL: https://github.com/ictnlp/pcfg-nat
• Copy Paste: [[2311.07941]] Non-autoregressive Machine Translation with Probabilistic Context-free Grammar(http://arxiv.org/abs/2311.07941)
• Summary:

  Non-autoregressive Transformer(NAT) significantly accelerates the inference of neural machine translation. However, conventional NAT models suffer from limited expression power and performance degradation compared to autoregressive (AT) models due to the assumption of conditional independence among target tokens. To address these limitations, we propose a novel approach called PCFG-NAT, which leverages a specially designed Probabilistic Context-Free Grammar (PCFG) to enhance the ability of NAT models to capture complex dependencies among output tokens. Experimental results on major machine translation benchmarks demonstrate that PCFG-NAT further narrows the gap in translation quality between NAT and AT models. Moreover, PCFG-NAT facilitates a deeper understanding of the generated sentences, addressing the lack of satisfactory explainability in neural machine translation.Code is publicly available at https://github.com/ictnlp/PCFG-NAT.

Title: Explainable History Distillation by Marked Temporal Point Process. (arXiv:2311.07797v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.07797
• Code URL: null
• Copy Paste: [[2311.07797]] Explainable History Distillation by Marked Temporal Point Process(http://arxiv.org/abs/2311.07797)
• Summary:

  Explainability of machine learning models is mandatory when researchers introduce these commonly believed black boxes to real-world tasks, especially high-stakes ones. In this paper, we build a machine learning system to automatically generate explanations of happened events from history by \gls{ca} based on the \acrfull{tpp}. Specifically, we propose a new task called \acrfull{ehd}. This task requires a model to distill as few events as possible from observed history. The target is that the event distribution conditioned on left events predicts the observed future noticeably worse. We then regard distilled events as the explanation for the future. To efficiently solve \acrshort{ehd}, we rewrite the task into a \gls{01ip} and directly estimate the solution to the program by a model called \acrfull{model}. This work fills the gap between our task and existing works, which only spot the difference between factual and counterfactual worlds after applying a predefined modification to the environment. Experiment results on Retweet and StackOverflow datasets prove that \acrshort{model} significantly outperforms other \acrshort{ehd} baselines and can reveal the rationale underpinning real-world processes.

Title: Evaluating Neighbor Explainability for Graph Neural Networks. (arXiv:2311.08118v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.08118
• Code URL: null
• Copy Paste: [[2311.08118]] Evaluating Neighbor Explainability for Graph Neural Networks(http://arxiv.org/abs/2311.08118)
• Summary:

  Explainability in Graph Neural Networks (GNNs) is a new field growing in the last few years. In this publication we address the problem of determining how important is each neighbor for the GNN when classifying a node and how to measure the performance for this specific task. To do this, various known explainability methods are reformulated to get the neighbor importance and four new metrics are presented. Our results show that there is almost no difference between the explanations provided by gradient-based techniques in the GNN domain. In addition, many explainability techniques failed to identify important neighbors when GNNs without self-loops are used.

Title: The Hyperdimensional Transform for Distributional Modelling, Regression and Classification. (arXiv:2311.08150v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.08150
• Code URL: null
• Copy Paste: [[2311.08150]] The Hyperdimensional Transform for Distributional Modelling, Regression and Classification(http://arxiv.org/abs/2311.08150)
• Summary:

  Hyperdimensional computing (HDC) is an increasingly popular computing paradigm with immense potential for future intelligent applications. Although the main ideas already took form in the 1990s, HDC recently gained significant attention, especially in the field of machine learning and data science. Next to efficiency, interoperability and explainability, HDC offers attractive properties for generalization as it can be seen as an attempt to combine connectionist ideas from neural networks with symbolic aspects. In recent work, we introduced the hyperdimensional transform, revealing deep theoretical foundations for representing functions and distributions as high-dimensional holographic vectors. Here, we present the power of the hyperdimensional transform to a broad data science audience. We use the hyperdimensional transform as a theoretical basis and provide insight into state-of-the-art HDC approaches for machine learning. We show how existing algorithms can be modified and how this transform can lead to a novel, well-founded toolbox. Next to the standard regression and classification tasks of machine learning, our discussion includes various aspects of statistical modelling, such as representation, learning and deconvolving distributions, sampling, Bayesian inference, and uncertainty estimation.

watermark

Title: AuthentiGPT: Detecting Machine-Generated Text via Black-Box Language Models Denoising. (arXiv:2311.07700v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.07700
• Code URL: null
• Copy Paste: [[2311.07700]] AuthentiGPT: Detecting Machine-Generated Text via Black-Box Language Models Denoising(http://arxiv.org/abs/2311.07700)
• Summary:

  Large language models (LLMs) have opened up enormous opportunities while simultaneously posing ethical dilemmas. One of the major concerns is their ability to create text that closely mimics human writing, which can lead to potential misuse, such as academic misconduct, disinformation, and fraud. To address this problem, we present AuthentiGPT, an efficient classifier that distinguishes between machine-generated and human-written texts. Under the assumption that human-written text resides outside the distribution of machine-generated text, AuthentiGPT leverages a black-box LLM to denoise input text with artificially added noise, and then semantically compares the denoised text with the original to determine if the content is machine-generated. With only one trainable parameter, AuthentiGPT eliminates the need for a large training dataset, watermarking the LLM's output, or computing the log-likelihood. Importantly, the detection capability of AuthentiGPT can be easily adapted to any generative language model. With a 0.918 AUROC score on a domain-specific dataset, AuthentiGPT demonstrates its effectiveness over other commercial algorithms, highlighting its potential for detecting machine-generated text in academic settings.

diffusion

Title: One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion. (arXiv:2311.07885v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.07885
• Code URL: null
• Copy Paste: [[2311.07885]] One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion(http://arxiv.org/abs/2311.07885)
• Summary:

  Recent advancements in open-world 3D object generation have been remarkable, with image-to-3D methods offering superior fine-grained control over their text-to-3D counterparts. However, most existing models fall short in simultaneously providing rapid generation speeds and high fidelity to input images - two features essential for practical applications. In this paper, we present One-2-3-45++, an innovative method that transforms a single image into a detailed 3D textured mesh in approximately one minute. Our approach aims to fully harness the extensive knowledge embedded in 2D diffusion models and priors from valuable yet limited 3D data. This is achieved by initially finetuning a 2D diffusion model for consistent multi-view image generation, followed by elevating these images to 3D with the aid of multi-view conditioned 3D native diffusion models. Extensive experimental evaluations demonstrate that our method can produce high-quality, diverse 3D assets that closely mirror the original input image. Our project webpage: https://sudo-ai-3d.github.io/One2345plus_page.

Title: CLAMP: A Contrastive Language And Molecule Pre-training Network. (arXiv:2311.07617v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.07617
• Code URL: https://github.com/neelr/clamp
• Copy Paste: [[2311.07617]] CLAMP: A Contrastive Language And Molecule Pre-training Network(http://arxiv.org/abs/2311.07617)
• Summary:

  This paper highlights a shift in how to approach material generation. Instead of material-to-material, we propose a language-to-material generation architecture that utilizes millions of untapped data points. Using a web scraper to collect crystal text pairs from open-source research papers, a contrastive model can be trained using a convolutional graph neural network encoder and a language encoder. This would allow unsupervised zero-shot classification which can be trained by taking advantage of linguistic structure. Without any specific training data, an ~82\% accuracy was achieved and ~75\% accuracy for photocatalyst prediction with an extremely small dataset. This novel network could ideally be cross-applied to any reaction that can be described via text, opening completely new methods to think about 3D chemical framework generation. In the full experiment diffusion models would likely be incorporated to fully exploit the latent space.

Title: Brain-Driven Representation Learning Based on Diffusion Model. (arXiv:2311.07925v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.07925
• Code URL: null
• Copy Paste: [[2311.07925]] Brain-Driven Representation Learning Based on Diffusion Model(http://arxiv.org/abs/2311.07925)
• Summary:

  Interpreting EEG signals linked to spoken language presents a complex challenge, given the data's intricate temporal and spatial attributes, as well as the various noise factors. Denoising diffusion probabilistic models (DDPMs), which have recently gained prominence in diverse areas for their capabilities in representation learning, are explored in our research as a means to address this issue. Using DDPMs in conjunction with a conditional autoencoder, our new approach considerably outperforms traditional machine learning algorithms and established baseline models in accuracy. Our results highlight the potential of DDPMs as a sophisticated computational method for the analysis of speech-related EEG signals. This could lead to significant advances in brain-computer interfaces tailored for spoken communication.

Title: A Consistent Diffusion-Based Algorithm for Semi-Supervised Graph Learning. (arXiv:2311.07627v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.07627
• Code URL: null
• Copy Paste: [[2311.07627]] A Consistent Diffusion-Based Algorithm for Semi-Supervised Graph Learning(http://arxiv.org/abs/2311.07627)
• Summary:

  The task of semi-supervised classification aims at assigning labels to all nodes of a graph based on the labels known for a few nodes, called the seeds. One of the most popular algorithms relies on the principle of heat diffusion, where the labels of the seeds are spread by thermoconductance and the temperature of each node at equilibrium is used as a score function for each label. In this paper, we prove that this algorithm is not consistent unless the temperatures of the nodes at equilibrium are centered before scoring. This crucial step does not only make the algorithm provably consistent on a block model but brings significant performance gains on real graphs.

noise learning

data-free

transformer

Title: Algorithms for Object Detection in Substations. (arXiv:2311.07577v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.07577
• Code URL: null
• Copy Paste: [[2311.07577]] Algorithms for Object Detection in Substations(http://arxiv.org/abs/2311.07577)
• Summary:

  Inspection of high-voltage power equipment is an effective way to ensure power supply reliability. Object recognition, one of the key technologies in automatic power equipment inspection, attracts attention of many researchers and engineers. Although quite a few existing models have some their own advantages, object relationship between equipment which is very important in this task is scarcely considered. This paper combining object relationship modeling and Transformer Model proposes a Relation Transformer Model. It has four parts -- backbone, encoder, decoder and prediction heads. With this structure, the proposed method shows in experiments a much better performance than other three commonly used models in object recognition in substation, largely promoting the development of automatic power equipment inspection.

Title: SynthEnsemble: A Fusion of CNN, Vision Transformer, and Hybrid Models for Multi-Label Chest X-Ray Classification. (arXiv:2311.07750v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.07750
• Code URL: null
• Copy Paste: [[2311.07750]] SynthEnsemble: A Fusion of CNN, Vision Transformer, and Hybrid Models for Multi-Label Chest X-Ray Classification(http://arxiv.org/abs/2311.07750)
• Summary:

  Chest X-rays are widely used to diagnose thoracic diseases, but the lack of detailed information about these abnormalities makes it challenging to develop accurate automated diagnosis systems, which is crucial for early detection and effective treatment. To address this challenge, we employed deep learning techniques to identify patterns in chest X-rays that correspond to different diseases. We conducted experiments on the "ChestX-ray14" dataset using various pre-trained CNNs, transformers, hybrid(CNN+Transformer) models and classical models. The best individual model was the CoAtNet, which achieved an area under the receiver operating characteristic curve (AUROC) of 84.2%. By combining the predictions of all trained models using a weighted average ensemble where the weight of each model was determined using differential evolution, we further improved the AUROC to 85.4%, outperforming other state-of-the-art methods in this field. Our findings demonstrate the potential of deep learning techniques, particularly ensemble deep learning, for improving the accuracy of automatic diagnosis of thoracic diseases from chest X-rays.

Title: Amodal Optical Flow. (arXiv:2311.07761v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.07761
• Code URL: null
• Copy Paste: [[2311.07761]] Amodal Optical Flow(http://arxiv.org/abs/2311.07761)
• Summary:

  Optical flow estimation is very challenging in situations with transparent or occluded objects. In this work, we address these challenges at the task level by introducing Amodal Optical Flow, which integrates optical flow with amodal perception. Instead of only representing the visible regions, we define amodal optical flow as a multi-layered pixel-level motion field that encompasses both visible and occluded regions of the scene. To facilitate research on this new task, we extend the AmodalSynthDrive dataset to include pixel-level labels for amodal optical flow estimation. We present several strong baselines, along with the Amodal Flow Quality metric to quantify the performance in an interpretable manner. Furthermore, we propose the novel AmodalFlowNet as an initial step toward addressing this task. AmodalFlowNet consists of a transformer-based cost-volume encoder paired with a recurrent transformer decoder which facilitates recurrent hierarchical feature propagation and amodal semantic grounding. We demonstrate the tractability of amodal optical flow in extensive experiments and show its utility for downstream tasks such as panoptic tracking. We make the dataset, code, and trained models publicly available at this http URL

Title: Vision-Language Integration in Multimodal Video Transformers (Partially) Aligns with the Brain. (arXiv:2311.07766v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.07766
• Code URL: null
• Copy Paste: [[2311.07766]] Vision-Language Integration in Multimodal Video Transformers (Partially) Aligns with the Brain(http://arxiv.org/abs/2311.07766)
• Summary:

  Integrating information from multiple modalities is arguably one of the essential prerequisites for grounding artificial intelligence systems with an understanding of the real world. Recent advances in video transformers that jointly learn from vision, text, and sound over time have made some progress toward this goal, but the degree to which these models integrate information from modalities still remains unclear. In this work, we present a promising approach for probing a pre-trained multimodal video transformer model by leveraging neuroscientific evidence of multimodal information processing in the brain. Using brain recordings of participants watching a popular TV show, we analyze the effects of multi-modal connections and interactions in a pre-trained multi-modal video transformer on the alignment with uni- and multi-modal brain regions. We find evidence that vision enhances masked prediction performance during language processing, providing support that cross-modal representations in models can benefit individual modalities. However, we don't find evidence of brain-relevant information captured by the joint multi-modal transformer representations beyond that captured by all of the individual modalities. We finally show that the brain alignment of the pre-trained joint representation can be improved by fine-tuning using a task that requires vision-language inferences. Overall, our results paint an optimistic picture of the ability of multi-modal transformers to integrate vision and language in partially brain-relevant ways but also show that improving the brain alignment of these models may require new approaches.

Title: Probing clustering in neural network representations. (arXiv:2311.07864v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.07864
• Code URL: null
• Copy Paste: [[2311.07864]] Probing clustering in neural network representations(http://arxiv.org/abs/2311.07864)
• Summary:

  Neural network representations contain structure beyond what was present in the training labels. For instance, representations of images that are visually or semantically similar tend to lie closer to each other than to dissimilar images, regardless of their labels. Clustering these representations can thus provide insights into dataset properties as well as the network internals. In this work, we study how the many design choices involved in neural network training affect the clusters formed in the hidden representations. To do so, we establish an evaluation setup based on the BREEDS hierarchy, for the task of subclass clustering after training models with only superclass information. We isolate the training dataset and architecture as important factors affecting clusterability. Datasets with labeled classes consisting of unrelated subclasses yield much better clusterability than those following a natural hierarchy. When using pretrained models to cluster representations on downstream datasets, models pretrained on subclass labels provide better clusterability than models pretrained on superclass labels, but only when there is a high degree of domain overlap between the pretraining and downstream data. Architecturally, we find that normalization strategies affect which layers yield the best clustering performance, and, surprisingly, Vision Transformers attain lower subclass clusterability than ResNets.

Title: Benchmarking Individual Tree Mapping with Sub-meter Imagery. (arXiv:2311.07981v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.07981
• Code URL: null
• Copy Paste: [[2311.07981]] Benchmarking Individual Tree Mapping with Sub-meter Imagery(http://arxiv.org/abs/2311.07981)
• Summary:

  There is a rising interest in mapping trees using satellite or aerial imagery, but there is no standardized evaluation protocol for comparing and enhancing methods. In dense canopy areas, the high variability of tree sizes and their spatial proximity makes it arduous to define the quality of the predictions. Concurrently, object-centric approaches such as bounding box detection usuallyperform poorly on small and dense objects. It thus remains unclear what is the ideal framework for individual tree mapping, in regards to detection and segmentation approaches, convolutional neural networks and transformers. In this paper, we introduce an evaluation framework suited for individual tree mapping in any physical environment, with annotation costs and applicative goals in mind. We review and compare different approaches and deep architectures, and introduce a new method that we experimentally prove to be a good compromise between segmentation and detection.

Title: Explicit Change Relation Learning for Change Detection in VHR Remote Sensing Images. (arXiv:2311.07993v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.07993
• Code URL: null
• Copy Paste: [[2311.07993]] Explicit Change Relation Learning for Change Detection in VHR Remote Sensing Images(http://arxiv.org/abs/2311.07993)
• Summary:

  Change detection has always been a concerned task in the interpretation of remote sensing images. It is essentially a unique binary classification task with two inputs, and there is a change relationship between these two inputs. At present, the mining of change relationship features is usually implicit in the network architectures that contain single-branch or two-branch encoders. However, due to the lack of artificial prior design for change relationship features, these networks cannot learn enough change semantic information and lose more accurate change detection performance. So we propose a network architecture NAME for the explicit mining of change relation features. In our opinion, the change features of change detection should be divided into pre-changed image features, post-changed image features and change relation features. In order to fully mine these three kinds of change features, we propose the triple branch network combining the transformer and convolutional neural network (CNN) to extract and fuse these change features from two perspectives of global information and local information, respectively. In addition, we design the continuous change relation (CCR) branch to further obtain the continuous and detail change relation features to improve the change discrimination capability of the model. The experimental results show that our network performs better, in terms of F1, IoU, and OA, than those of the existing advanced networks for change detection on four public very high-resolution (VHR) remote sensing datasets. Our source code is available at https://github.com/DalongZ/NAME.

Title: Contrastive Learning for Multi-Object Tracking with Transformers. (arXiv:2311.08043v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.08043
• Code URL: null
• Copy Paste: [[2311.08043]] Contrastive Learning for Multi-Object Tracking with Transformers(http://arxiv.org/abs/2311.08043)
• Summary:

  The DEtection TRansformer (DETR) opened new possibilities for object detection by modeling it as a translation task: converting image features into object-level representations. Previous works typically add expensive modules to DETR to perform Multi-Object Tracking (MOT), resulting in more complicated architectures. We instead show how DETR can be turned into a MOT model by employing an instance-level contrastive loss, a revised sampling strategy and a lightweight assignment method. Our training scheme learns object appearances while preserving detection capabilities and with little overhead. Its performance surpasses the previous state-of-the-art by +2.6 mMOTA on the challenging BDD100K dataset and is comparable to existing transformer-based methods on the MOT17 dataset.

Title: GMTR: Graph Matching Transformers. (arXiv:2311.08141v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.08141
• Code URL: null
• Copy Paste: [[2311.08141]] GMTR: Graph Matching Transformers(http://arxiv.org/abs/2311.08141)
• Summary:

  Vision transformers (ViTs) have recently been used for visual matching beyond object detection and segmentation. However, the original grid dividing strategy of ViTs neglects the spatial information of the keypoints, limiting the sensitivity to local information. Therefore, we propose \textbf{QueryTrans} (Query Transformer), which adopts a cross-attention module and keypoints-based center crop strategy for better spatial information extraction. We further integrate the graph attention module and devise a transformer-based graph matching approach \textbf{GMTR} (Graph Matching TRansformers) whereby the combinatorial nature of GM is addressed by a graph transformer neural GM solver. On standard GM benchmarks, GMTR shows competitive performance against the SOTA frameworks. Specifically, on Pascal VOC, GMTR achieves $\mathbf{83.6\%}$ accuracy, $\mathbf{0.9\%}$ higher than the SOTA framework. On Spair-71k, GMTR shows great potential and outperforms most of the previous works. Meanwhile, on Pascal VOC, QueryTrans improves the accuracy of NGMv2 from $80.1\%$ to $\mathbf{83.3\%}$, and BBGM from $79.0\%$ to $\mathbf{84.5\%}$. On Spair-71k, QueryTrans improves NGMv2 from $80.6\%$ to $\mathbf{82.5\%}$, and BBGM from $82.1\%$ to $\mathbf{83.9\%}$. Source code will be made publicly available.

Title: Rethink Cross-Modal Fusion in Weakly-Supervised Audio-Visual Video Parsing. (arXiv:2311.08151v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.08151
• Code URL: null
• Copy Paste: [[2311.08151]] Rethink Cross-Modal Fusion in Weakly-Supervised Audio-Visual Video Parsing(http://arxiv.org/abs/2311.08151)
• Summary:

  Existing works on weakly-supervised audio-visual video parsing adopt hybrid attention network (HAN) as the multi-modal embedding to capture the cross-modal context. It embeds the audio and visual modalities with a shared network, where the cross-attention is performed at the input. However, such an early fusion method highly entangles the two non-fully correlated modalities and leads to sub-optimal performance in detecting single-modality events. To deal with this problem, we propose the messenger-guided mid-fusion transformer to reduce the uncorrelated cross-modal context in the fusion. The messengers condense the full cross-modal context into a compact representation to only preserve useful cross-modal information. Furthermore, due to the fact that microphones capture audio events from all directions, while cameras only record visual events within a restricted field of view, there is a more frequent occurrence of unaligned cross-modal context from audio for visual event predictions. We thus propose cross-audio prediction consistency to suppress the impact of irrelevant audio information on visual event prediction. Experiments consistently illustrate the superior performance of our framework compared to existing state-of-the-art methods.

Title: MeLo: Low-rank Adaptation is Better than Fine-tuning for Medical Image Diagnosis. (arXiv:2311.08236v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.08236
• Code URL: null
• Copy Paste: [[2311.08236]] MeLo: Low-rank Adaptation is Better than Fine-tuning for Medical Image Diagnosis(http://arxiv.org/abs/2311.08236)
• Summary:

  The common practice in developing computer-aided diagnosis (CAD) models based on transformer architectures usually involves fine-tuning from ImageNet pre-trained weights. However, with recent advances in large-scale pre-training and the practice of scaling laws, Vision Transformers (ViT) have become much larger and less accessible to medical imaging communities. Additionally, in real-world scenarios, the deployments of multiple CAD models can be troublesome due to problems such as limited storage space and time-consuming model switching. To address these challenges, we propose a new method MeLo (Medical image Low-rank adaptation), which enables the development of a single CAD model for multiple clinical tasks in a lightweight manner. It adopts low-rank adaptation instead of resource-demanding fine-tuning. By fixing the weight of ViT models and only adding small low-rank plug-ins, we achieve competitive results on various diagnosis tasks across different imaging modalities using only a few trainable parameters. Specifically, our proposed method achieves comparable performance to fully fine-tuned ViT models on four distinct medical imaging datasets using about 0.17% trainable parameters. Moreover, MeLo adds only about 0.5MB of storage space and allows for extremely fast model switching in deployment and inference. Our source code and pre-trained weights are available on our website (https://absterzhu.github.io/melo.github.io/).

Title: NLQxform: A Language Model-based Question to SPARQL Transformer. (arXiv:2311.07588v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.07588
• Code URL: null
• Copy Paste: [[2311.07588]] NLQxform: A Language Model-based Question to SPARQL Transformer(http://arxiv.org/abs/2311.07588)
• Summary:

  In recent years, scholarly data has grown dramatically in terms of both scale and complexity. It becomes increasingly challenging to retrieve information from scholarly knowledge graphs that include large-scale heterogeneous relationships, such as authorship, affiliation, and citation, between various types of entities, e.g., scholars, papers, and organizations. As part of the Scholarly QALD Challenge, this paper presents a question-answering (QA) system called NLQxform, which provides an easy-to-use natural language interface to facilitate accessing scholarly knowledge graphs. NLQxform allows users to express their complex query intentions in natural language questions. A transformer-based language model, i.e., BART, is employed to translate questions into standard SPARQL queries, which can be evaluated to retrieve the required information. According to the public leaderboard of the Scholarly QALD Challenge at ISWC 2023 (Task 1: DBLP-QUAD - Knowledge Graph Question Answering over DBLP), NLQxform achieved an F1 score of 0.85 and ranked first on the QA task, demonstrating the competitiveness of the system.

Title: GreekT5: A Series of Greek Sequence-to-Sequence Models for News Summarization. (arXiv:2311.07767v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.07767
• Code URL: https://github.com/nc0der/greekt5
• Copy Paste: [[2311.07767]] GreekT5: A Series of Greek Sequence-to-Sequence Models for News Summarization(http://arxiv.org/abs/2311.07767)
• Summary:

  Text summarization (TS) is a natural language processing (NLP) subtask pertaining to the automatic formulation of a concise and coherent summary that covers the major concepts and topics from one or multiple documents. Recent advancements in deep learning have led to the development of abstractive summarization transformer-based models, which outperform classical approaches. In any case, research in this field focuses on high resource languages such as English, while the corresponding work for low resource languages is still underdeveloped. Taking the above into account, this paper proposes a series of novel TS models for Greek news articles. The proposed models were thoroughly evaluated on the same dataset against GreekBART, which is the state-of-the-art model in Greek abstractive news summarization. Our evaluation results reveal that most of the proposed models significantly outperform GreekBART on various evaluation metrics. We make our evaluation code public, aiming to increase the reproducibility of this work and facilitate future research in the field.

Title: A Survey on Language Models for Code. (arXiv:2311.07989v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.07989
• Code URL: https://github.com/codefuse-ai/awesome-code-llm
• Copy Paste: [[2311.07989]] A Survey on Language Models for Code(http://arxiv.org/abs/2311.07989)
• Summary:

  In this work we systematically review the recent advancements in code processing with language models, covering 50+ models, 30+ evaluation tasks, and 500 related works. We break down code processing models into general language models represented by the GPT family and specialized models that are specifically pretrained on code, often with tailored objectives. We discuss the relations and differences between these models, and highlight the historical transition of code modeling from statistical models and RNNs to pretrained Transformers and LLMs, which is exactly the same course that had been taken by NLP. We also discuss code-specific features such as AST, CFG, and unit tests, along with their application in training code language models, and identify key challenges and potential future directions in this domain. We keep the survey open and updated on github repository at https://github.com/codefuse-ai/Awesome-Code-LLM.

Title: Spot: A Natural Language Interface for Geospatial Searches in OSM. (arXiv:2311.08093v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.08093
• Code URL: null
• Copy Paste: [[2311.08093]] Spot: A Natural Language Interface for Geospatial Searches in OSM(http://arxiv.org/abs/2311.08093)
• Summary:

  Investigative journalists and fact-checkers have found OpenStreetMap (OSM) to be an invaluable resource for their work due to its extensive coverage and intricate details of various locations, which play a crucial role in investigating news scenes. Despite its value, OSM's complexity presents considerable accessibility and usability challenges, especially for those without a technical background. To address this, we introduce 'Spot', a user-friendly natural language interface for querying OSM data. Spot utilizes a semantic mapping from natural language to OSM tags, leveraging artificially generated sentence queries and a T5 transformer. This approach enables Spot to extract relevant information from user-input sentences and display candidate locations matching the descriptions on a map. To foster collaboration and future advancement, all code and generated data is available as an open-source repository.

Title: Exploring Semi-supervised Hierarchical Stacked Encoder for Legal Judgement Prediction. (arXiv:2311.08103v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.08103
• Code URL: https://github.com/nishchalprasad/semi-supervised-stacked-encoder
• Copy Paste: [[2311.08103]] Exploring Semi-supervised Hierarchical Stacked Encoder for Legal Judgement Prediction(http://arxiv.org/abs/2311.08103)
• Summary:

  Predicting the judgment of a legal case from its unannotated case facts is a challenging task. The lengthy and non-uniform document structure poses an even greater challenge in extracting information for decision prediction. In this work, we explore and propose a two-level classification mechanism; both supervised and unsupervised; by using domain-specific pre-trained BERT to extract information from long documents in terms of sentence embeddings further processing with transformer encoder layer and use unsupervised clustering to extract hidden labels from these embeddings to better predict a judgment of a legal case. We conduct several experiments with this mechanism and see higher performance gains than the previously proposed methods on the ILDC dataset. Our experimental results also show the importance of domain-specific pre-training of Transformer Encoders in legal information processing.

Title: Memory-efficient Stochastic methods for Memory-based Transformers. (arXiv:2311.08123v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.08123
• Code URL: https://github.com/vishwajit-vishnu/memory-efficient-stochastic-methods-for-memory-based-transformers
• Copy Paste: [[2311.08123]] Memory-efficient Stochastic methods for Memory-based Transformers(http://arxiv.org/abs/2311.08123)
• Summary:

  Training Memory-based transformers can require a large amount of memory and can be quite inefficient. We propose a novel two-phase training mechanism and a novel regularization technique to improve the training efficiency of memory-based transformers, which are often used for long-range context problems. For our experiments, we consider transformer-XL as our baseline model which is one of memorybased transformer models. We show that our resultant model, Skip Cross-head TransformerXL, outperforms the baseline on character level language modeling task with similar parameters and outperforms the baseline on word level language modelling task with almost 20% fewer parameters. Our proposed methods do not require any additional memory. We also demonstrate the effectiveness of our regularization mechanism on BERT which shows similar performance with reduction in standard deviation of scores of around 30% on multiple GLUE tasks.

Title: Enhancing Actuarial Non-Life Pricing Models via Transformers. (arXiv:2311.07597v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.07597
• Code URL: null
• Copy Paste: [[2311.07597]] Enhancing Actuarial Non-Life Pricing Models via Transformers(http://arxiv.org/abs/2311.07597)
• Summary:

  Currently, there is a lot of research in the field of neural networks for non-life insurance pricing. The usual goal is to improve the predictive power via neural networks while building upon the generalized linear model, which is the current industry standard. Our paper contributes to this current journey via novel methods to enhance actuarial non-life models with transformer models for tabular data. We build here upon the foundation laid out by the combined actuarial neural network as well as the localGLMnet and enhance those models via the feature tokenizer transformer. The manuscript demonstrates the performance of the proposed methods on a real-world claim frequency dataset and compares them with several benchmark models such as generalized linear models, feed-forward neural networks, combined actuarial neural networks, LocalGLMnet, and pure feature tokenizer transformer. The paper shows that the new methods can achieve better results than the benchmark models while preserving certain generalized linear model advantages. The paper also discusses the practical implications and challenges of applying transformer models in actuarial settings.

Title: MuST: Multimodal Spatiotemporal Graph-Transformer for Hospital Readmission Prediction. (arXiv:2311.07608v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.07608
• Code URL: null
• Copy Paste: [[2311.07608]] MuST: Multimodal Spatiotemporal Graph-Transformer for Hospital Readmission Prediction(http://arxiv.org/abs/2311.07608)
• Summary:

  Hospital readmission prediction is considered an essential approach to decreasing readmission rates, which is a key factor in assessing the quality and efficacy of a healthcare system. Previous studies have extensively utilized three primary modalities, namely electronic health records (EHR), medical images, and clinical notes, to predict hospital readmissions. However, the majority of these studies did not integrate information from all three modalities or utilize the spatiotemporal relationships present in the dataset. This study introduces a novel model called the Multimodal Spatiotemporal Graph-Transformer (MuST) for predicting hospital readmissions. By employing Graph Convolution Networks and temporal transformers, we can effectively capture spatial and temporal dependencies in EHR and chest radiographs. We then propose a fusion transformer to combine the spatiotemporal features from the two modalities mentioned above with the features from clinical notes extracted by a pre-trained, domain-specific transformer. We assess the effectiveness of our methods using the latest publicly available dataset, MIMIC-IV. The experimental results indicate that the inclusion of multimodal features in MuST improves its performance in comparison to unimodal methods. Furthermore, our proposed pipeline outperforms the current leading methods in the prediction of hospital readmissions.

generative

Title: Peer is Your Pillar: A Data-unbalanced Conditional GANs for Few-shot Image Generation. (arXiv:2311.08217v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.08217
• Code URL: null
• Copy Paste: [[2311.08217]] Peer is Your Pillar: A Data-unbalanced Conditional GANs for Few-shot Image Generation(http://arxiv.org/abs/2311.08217)
• Summary:

  Few-shot image generation aims to train generative models using a small number of training images. When there are few images available for training (e.g. 10 images), Learning From Scratch (LFS) methods often generate images that closely resemble the training data while Transfer Learning (TL) methods try to improve performance by leveraging prior knowledge from GANs pre-trained on large-scale datasets. However, current TL methods may not allow for sufficient control over the degree of knowledge preservation from the source model, making them unsuitable for setups where the source and target domains are not closely related. To address this, we propose a novel pipeline called Peer is your Pillar (PIP), which combines a target few-shot dataset with a peer dataset to create a data-unbalanced conditional generation. Our approach includes a class embedding method that separates the class space from the latent space, and we use a direction loss based on pre-trained CLIP to improve image diversity. Experiments on various few-shot datasets demonstrate the advancement of the proposed PIP, especially reduces the training requirements of few-shot image generation.

Title: The ART of LLM Refinement: Ask, Refine, and Trust. (arXiv:2311.07961v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.07961
• Code URL: null
• Copy Paste: [[2311.07961]] The ART of LLM Refinement: Ask, Refine, and Trust(http://arxiv.org/abs/2311.07961)
• Summary:

  In recent years, Large Language Models (LLMs) have demonstrated remarkable generative abilities, but can they judge the quality of their own generations? A popular concept, referred to as self-refinement, postulates that LLMs can detect and correct the errors in their generations when asked to do so. However, recent empirical evidence points in the opposite direction, suggesting that LLMs often struggle to accurately identify errors when reasoning is involved. To address this, we propose a reasoning with refinement objective called ART: Ask, Refine, and Trust, which asks necessary questions to decide when an LLM should refine its output, and either affirm or withhold trust in its refinement by ranking the refinement and the initial prediction. On two multistep reasoning tasks of mathematical word problems (GSM8K) and question answering (StrategyQA), ART achieves a performance gain of +5 points over self-refinement baselines, while using a much smaller model as the decision maker. We also demonstrate the benefit of using smaller models to make refinement decisions as a cost-effective alternative to fine-tuning a larger model.

Title: Align after Pre-train: Improving Multilingual Generative Models with Cross-lingual Alignment. (arXiv:2311.08089v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.08089
• Code URL: null
• Copy Paste: [[2311.08089]] Align after Pre-train: Improving Multilingual Generative Models with Cross-lingual Alignment(http://arxiv.org/abs/2311.08089)
• Summary:

  Multilingual generative models obtain remarkable cross-lingual capabilities through pre-training on large-scale corpora. However, they still exhibit a performance bias toward high-resource languages, and learn isolated distributions of sentence representations across languages. To bridge this gap, we propose a simple yet effective alignment framework exploiting pairs of translation sentences. It aligns the internal sentence representations across different languages via multilingual contrastive learning and aligns model outputs by answering prompts in different languages. Experimental results demonstrate that even with less than 0.1 {\textperthousand} of pre-training tokens, our alignment framework significantly boosts the cross-lingual abilities of generative models and mitigates the performance gap. Further analysis reveals that it results in a better internal multilingual representation distribution of multilingual models.

Title: Eval-GCSC: A New Metric for Evaluating ChatGPT's Performance in Chinese Spelling Correction. (arXiv:2311.08219v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.08219
• Code URL: https://github.com/ktlktl/eval-gcsc
• Copy Paste: [[2311.08219]] Eval-GCSC: A New Metric for Evaluating ChatGPT's Performance in Chinese Spelling Correction(http://arxiv.org/abs/2311.08219)
• Summary:

  ChatGPT has demonstrated impressive performance in various downstream tasks. However, in the Chinese Spelling Correction (CSC) task, we observe a discrepancy: while ChatGPT performs well under human evaluation, it scores poorly according to traditional metrics. We believe this inconsistency arises because the traditional metrics are not well-suited for evaluating generative models. Their overly strict length and phonics constraints may lead to underestimating ChatGPT's correction capabilities. To better evaluate generative models in the CSC task, this paper proposes a new evaluation metric: Eval-GCSC. By incorporating word-level and semantic similarity judgments, it relaxes the stringent length and phonics constraints. Experimental results show that Eval-GCSC closely aligns with human evaluations. Under this metric, ChatGPT's performance is comparable to traditional token-level classification models (TCM), demonstrating its potential as a CSC tool. The source code and scripts can be accessed at https://github.com/ktlKTL/Eval-GCSC.

Title: Self-supervised Heterogeneous Graph Variational Autoencoders. (arXiv:2311.07929v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.07929
• Code URL: null
• Copy Paste: [[2311.07929]] Self-supervised Heterogeneous Graph Variational Autoencoders(http://arxiv.org/abs/2311.07929)
• Summary:

  Heterogeneous Information Networks (HINs), which consist of various types of nodes and edges, have recently demonstrated excellent performance in graph mining. However, most existing heterogeneous graph neural networks (HGNNs) ignore the problems of missing attributes, inaccurate attributes and scarce labels for nodes, which limits their expressiveness. In this paper, we propose a generative self-supervised model SHAVA to address these issues simultaneously. Specifically, SHAVA first initializes all the nodes in the graph with a low-dimensional representation matrix. After that, based on the variational graph autoencoder framework, SHAVA learns both node-level and attribute-level embeddings in the encoder, which can provide fine-grained semantic information to construct node attributes. In the decoder, SHAVA reconstructs both links and attributes. Instead of directly reconstructing raw features for attributed nodes, SHAVA generates the initial low-dimensional representation matrix for all the nodes, based on which raw features of attributed nodes are further reconstructed to leverage accurate attributes. In this way, SHAVA can not only complete informative features for non-attributed nodes, but rectify inaccurate ones for attributed nodes. Finally, we conduct extensive experiments to show the superiority of SHAVA in tackling HINs with missing and inaccurate attributes.

Title: Modeling Complex Disease Trajectories using Deep Generative Models with Semi-Supervised Latent Processes. (arXiv:2311.08149v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.08149
• Code URL: https://github.com/uzh-dqbm-cmi/eustar_dgm4h
• Copy Paste: [[2311.08149]] Modeling Complex Disease Trajectories using Deep Generative Models with Semi-Supervised Latent Processes(http://arxiv.org/abs/2311.08149)
• Summary:

  In this paper, we propose a deep generative time series approach using latent temporal processes for modeling and holistically analyzing complex disease trajectories. We aim to find meaningful temporal latent representations of an underlying generative process that explain the observed disease trajectories in an interpretable and comprehensive way. To enhance the interpretability of these latent temporal processes, we develop a semi-supervised approach for disentangling the latent space using established medical concepts. By combining the generative approach with medical knowledge, we leverage the ability to discover novel aspects of the disease while integrating medical concepts into the model. We show that the learned temporal latent processes can be utilized for further data analysis and clinical hypothesis testing, including finding similar patients and clustering the disease into new sub-types. Moreover, our method enables personalized online monitoring and prediction of multivariate time series including uncertainty quantification. We demonstrate the effectiveness of our approach in modeling systemic sclerosis, showcasing the potential of our machine learning model to capture complex disease trajectories and acquire new medical knowledge.

large language model

Title: Follow-Up Differential Descriptions: Language Models Resolve Ambiguities for Image Classification. (arXiv:2311.07593v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.07593
• Code URL: https://github.com/batsresearch/fudd
• Copy Paste: [[2311.07593]] Follow-Up Differential Descriptions: Language Models Resolve Ambiguities for Image Classification(http://arxiv.org/abs/2311.07593)
• Summary:

  A promising approach for improving the performance of vision-language models like CLIP for image classification is to extend the class descriptions (i.e., prompts) with related attributes, e.g., using brown sparrow instead of sparrow. However, current zero-shot methods select a subset of attributes regardless of commonalities between the target classes, potentially providing no useful information that would have helped to distinguish between them. For instance, they may use color instead of bill shape to distinguish between sparrows and wrens, which are both brown. We propose Follow-up Differential Descriptions (FuDD), a zero-shot approach that tailors the class descriptions to each dataset and leads to additional attributes that better differentiate the target classes. FuDD first identifies the ambiguous classes for each image, and then uses a Large Language Model (LLM) to generate new class descriptions that differentiate between them. The new class descriptions resolve the initial ambiguity and help predict the correct label. In our experiments, FuDD consistently outperforms generic description ensembles and naive LLM-generated descriptions on 12 datasets. We show that differential descriptions are an effective tool to resolve class ambiguities, which otherwise significantly degrade the performance. We also show that high quality natural language class descriptions produced by FuDD result in comparable performance to few-shot adaptation methods.

Title: How to Bridge the Gap between Modalities: A Comprehensive Survey on Multimodal Large Language Model. (arXiv:2311.07594v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.07594
• Code URL: null
• Copy Paste: [[2311.07594]] How to Bridge the Gap between Modalities: A Comprehensive Survey on Multimodal Large Language Model(http://arxiv.org/abs/2311.07594)
• Summary:

  This review paper explores Multimodal Large Language Models (MLLMs), which integrate Large Language Models (LLMs) like GPT-4 to handle multimodal data such as text and vision. MLLMs demonstrate capabilities like generating image narratives and answering image-based questions, bridging the gap towards real-world human-computer interactions and hinting at a potential pathway to artificial general intelligence. However, MLLMs still face challenges in processing the semantic gap in multimodality, which may lead to erroneous generation, posing potential risks to society. Choosing the appropriate modality alignment method is crucial, as improper methods might require more parameters with limited performance improvement. This paper aims to explore modality alignment methods for LLMs and their existing capabilities. Implementing modality alignment allows LLMs to address environmental issues and enhance accessibility. The study surveys existing modal alignment methods in MLLMs into four groups: (1) Multimodal Converters that change data into something LLMs can understand; (2) Multimodal Perceivers to improve how LLMs perceive different types of data; (3) Tools Assistance for changing data into one common format, usually text; and (4) Data-Driven methods that teach LLMs to understand specific types of data in a dataset. This field is still in a phase of exploration and experimentation, and we will organize and update various existing research methods for multimodal information alignment.

Title: Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding. (arXiv:2311.08046v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.08046
• Code URL: null
• Copy Paste: [[2311.08046]] Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding(http://arxiv.org/abs/2311.08046)
• Summary:

  Large language models have demonstrated impressive universal capabilities across a wide range of open-ended tasks and have extended their utility to encompass multimodal conversations. However, existing methods encounter challenges in effectively handling both image and video understanding, particularly with limited visual tokens. In this work, we introduce Chat-UniVi, a unified vision-language model capable of comprehending and engaging in conversations involving images and videos through a unified visual representation. Specifically, we employ a set of dynamic visual tokens to uniformly represent images and videos. This representation framework empowers the model to efficiently utilize a limited number of visual tokens to simultaneously capture the spatial details necessary for images and the comprehensive temporal relationship required for videos. Moreover, we leverage a multi-scale representation, enabling the model to perceive both high-level semantic concepts and low-level visual details. Notably, Chat-UniVi is trained on a mixed dataset containing both images and videos, allowing direct application to tasks involving both mediums without requiring any modifications. Extensive experimental results demonstrate that Chat-UniVi, as a unified model, consistently outperforms even existing methods exclusively designed for either images or videos.

Title: Unlock the Power: Competitive Distillation for Multi-Modal Large Language Models. (arXiv:2311.08213v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.08213
• Code URL: null
• Copy Paste: [[2311.08213]] Unlock the Power: Competitive Distillation for Multi-Modal Large Language Models(http://arxiv.org/abs/2311.08213)
• Summary:

  Recently, multi-modal content generation has attracted lots of attention from researchers by investigating the utilization of visual instruction tuning based on large language models (LLMs). To enhance the performance and generalization ability of such LLMs, the practice of distilling knowledge from pretrained multi-modal models (a.k.a. teachers) to more compact multi-modal LLMs (students) has gained considerable interest. However, the prevailing paradigm of instructiontuning in multi-modal LLMs knowledge distillation is resource-intensive and unidirectional, neglecting the potential for mutual feedback between the student and teacher models. Thus, we propose an innovative Competitive Multi-modal Distillation framework (CoMD), which captures bidirectional feedback between teacher and student models and continually updates the multi-modal capabilities that the student model has learned. It comprises two stages: multi-modal pre-training and multi-modal competitive distillation. The first stage pre-trains the student model on a large number of filtered multi-modal datasets. The second stage facilitates a bidirectional knowledge transfer between the student and teacher models. Our experimental analysis of diverse datasets shows that our knowledge transfer method consistently improves the capabilities of the student model. Finally, the 7B-sized student model after four distillations surpassed the current state-of-the-art model LLaVA-13B on the ScienceQA and LLaVA Test dataset, also outperforms other strong baselines in the zero-shot setting.

Title: Evaluating the Potential of Leading Large Language Models in Reasoning Biology Questions. (arXiv:2311.07582v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.07582
• Code URL: null
• Copy Paste: [[2311.07582]] Evaluating the Potential of Leading Large Language Models in Reasoning Biology Questions(http://arxiv.org/abs/2311.07582)
• Summary:

  Recent advances in Large Language Models (LLMs) have presented new opportunities for integrating Artificial General Intelligence (AGI) into biological research and education. This study evaluated the capabilities of leading LLMs, including GPT-4, GPT-3.5, PaLM2, Claude2, and SenseNova, in answering conceptual biology questions. The models were tested on a 108-question multiple-choice exam covering biology topics in molecular biology, biological techniques, metabolic engineering, and synthetic biology. Among the models, GPT-4 achieved the highest average score of 90 and demonstrated the greatest consistency across trials with different prompts. The results indicated GPT-4's proficiency in logical reasoning and its potential to aid biology research through capabilities like data analysis, hypothesis generation, and knowledge integration. However, further development and validation are still required before the promise of LLMs in accelerating biological discovery can be realized.

Title: Technical Report: Large Language Models can Strategically Deceive their Users when Put Under Pressure. (arXiv:2311.07590v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.07590
• Code URL: null
• Copy Paste: [[2311.07590]] Technical Report: Large Language Models can Strategically Deceive their Users when Put Under Pressure(http://arxiv.org/abs/2311.07590)
• Summary:

  We demonstrate a situation in which Large Language Models, trained to be helpful, harmless, and honest, can display misaligned behavior and strategically deceive their users about this behavior without being instructed to do so. Concretely, we deploy GPT-4 as an agent in a realistic, simulated environment, where it assumes the role of an autonomous stock trading agent. Within this environment, the model obtains an insider tip about a lucrative stock trade and acts upon it despite knowing that insider trading is disapproved of by company management. When reporting to its manager, the model consistently hides the genuine reasons behind its trading decision. We perform a brief investigation of how this behavior varies under changes to the setting, such as removing model access to a reasoning scratchpad, attempting to prevent the misaligned behavior by changing system instructions, changing the amount of pressure the model is under, varying the perceived risk of getting caught, and making other simple changes to the environment. To our knowledge, this is the first demonstration of Large Language Models trained to be helpful, harmless, and honest, strategically deceiving their users in a realistic situation without direct instructions or training for deception.

Title: Intentional Biases in LLM Responses. (arXiv:2311.07611v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.07611
• Code URL: null
• Copy Paste: [[2311.07611]] Intentional Biases in LLM Responses(http://arxiv.org/abs/2311.07611)
• Summary:

  In this study we intentionally introduce biases into large language model responses in an attempt to create specific personas for interactive media purposes. We explore the differences between open source models such as Falcon-7b and the GPT-4 model from Open AI, and we quantify some differences in responses afforded by the two systems. We find that the guardrails in the GPT-4 mixture of experts models with a supervisor, while useful in assuring AI alignment in general, are detrimental in trying to construct personas with a variety of uncommon viewpoints. This study aims to set the groundwork for future exploration in intentional biases of large language models such that these practices can be applied in the creative field, and new forms of media.

Title: Large Language Models' Understanding of Math: Source Criticism and Extrapolation. (arXiv:2311.07618v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.07618
• Code URL: null
• Copy Paste: [[2311.07618]] Large Language Models' Understanding of Math: Source Criticism and Extrapolation(http://arxiv.org/abs/2311.07618)
• Summary:

  It has been suggested that large language models such as GPT-4 have acquired some form of understanding beyond the correlations among the words in text including some understanding of mathematics as well. Here, we perform a critical inquiry into this claim by evaluating the mathematical understanding of the GPT-4 model. Considering that GPT-4's training set is a secret, it is not straightforward to evaluate whether the model's correct answers are based on a mathematical understanding or based on replication of proofs that the model has seen before. We specifically craft mathematical questions which their formal proofs are not readily available on the web, proofs that are more likely not seen by the GPT-4. We see that GPT-4 is unable to solve those problems despite their simplicity. It is hard to find scientific evidence suggesting that GPT-4 has acquired an understanding of even basic mathematical concepts. A straightforward way to find failure modes of GPT-4 in theorem proving is to craft questions where their formal proofs are not available on the web. Our finding suggests that GPT-4's ability is to reproduce, rephrase, and polish the mathematical proofs that it has seen before, and not in grasping mathematical concepts. We also see that GPT-4's ability to prove mathematical theorems is continuously expanding over time despite the claim that it is a fixed model. We suggest that the task of proving mathematical theorems in formal language is comparable to the methods used in search engines such as Google while predicting the next word in a sentence may be a misguided approach, a recipe that often leads to excessive extrapolation and eventual failures. Prompting the GPT-4 over and over may benefit the GPT-4 and the OpenAI, but we question whether it is valuable for machine learning or for theorem proving.

Title: Language Model-In-The-Loop: Data Optimal Approach to Learn-To-Recommend Actions in Text Games. (arXiv:2311.07687v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.07687
• Code URL: null
• Copy Paste: [[2311.07687]] Language Model-In-The-Loop: Data Optimal Approach to Learn-To-Recommend Actions in Text Games(http://arxiv.org/abs/2311.07687)
• Summary:

  Large Language Models (LLMs) have demonstrated superior performance in language understanding benchmarks. CALM, a popular approach, leverages linguistic priors of LLMs -- GPT-2 -- for action candidate recommendations to improve the performance in text games in Jericho without environment-provided actions. However, CALM adapts GPT-2 with annotated human gameplays and keeps the LLM fixed during the learning of the text based games. In this work, we explore and evaluate updating LLM used for candidate recommendation during the learning of the text based game as well to mitigate the reliance on the human annotated gameplays, which are costly to acquire. We observe that by updating the LLM during learning using carefully selected in-game transitions, we can reduce the dependency on using human annotated game plays for fine-tuning the LLMs. We conducted further analysis to study the transferability of the updated LLMs and observed that transferring in-game trained models to other games did not result in a consistent transfer.

Title: On The Truthfulness of 'Surprisingly Likely' Responses of Large Language Models. (arXiv:2311.07692v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.07692
• Code URL: null
• Copy Paste: [[2311.07692]] On The Truthfulness of 'Surprisingly Likely' Responses of Large Language Models(http://arxiv.org/abs/2311.07692)
• Summary:

  The surprisingly likely criterion in the seminal work of Prelec (the Bayesian Truth Serum) guarantees truthfulness in a game-theoretic multi-agent setting, by rewarding rational agents to maximise the expected information gain with their answers w.r.t. their probabilistic beliefs. We investigate the relevance of a similar criterion for responses of LLMs. We hypothesize that if the surprisingly likely criterion works in LLMs, under certain conditions, the responses that maximize the reward under this criterion should be more accurate than the responses that only maximize the posterior probability. Using benchmarks including the TruthfulQA benchmark and using openly available LLMs: GPT-2 and LLaMA-2, we show that the method indeed improves the accuracy significantly (for example, upto 24 percentage points aggregate improvement on TruthfulQA and upto 70 percentage points improvement on individual categories of questions).

Title: LLatrieval: LLM-Verified Retrieval for Verifiable Generation. (arXiv:2311.07838v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.07838
• Code URL: https://github.com/beastyz/llm-verified-retrieval
• Copy Paste: [[2311.07838]] LLatrieval: LLM-Verified Retrieval for Verifiable Generation(http://arxiv.org/abs/2311.07838)
• Summary:

  Verifiable generation aims to let the large language model (LLM) generate text with corresponding supporting documents, which enables the user to flexibly verify the answer and makes it more trustworthy. Its evaluation not only measures the correctness of the answer, but also the answer's verifiability, i.e., how well the answer is supported by the corresponding documents. In typical, verifiable generation adopts the retrieval-read pipeline, which is divided into two stages: 1) retrieve relevant documents of the question. 2) according to the documents, generate the corresponding answer. Since the retrieved documents can supplement knowledge for the LLM to generate the answer and serve as evidence, the retrieval stage is essential for the correctness and verifiability of the answer. However, the widely used retrievers become the bottleneck of the entire pipeline and limit the overall performance. They often have fewer parameters than the large language model and have not been proven to scale well to the size of LLMs. Since the LLM passively receives the retrieval result, if the retriever does not correctly find the supporting documents, the LLM can not generate the correct and verifiable answer, which overshadows the LLM's remarkable abilities. In this paper, we propose LLatrieval (Large Language Model Verified Retrieval), where the LLM updates the retrieval result until it verifies that the retrieved documents can support answering the question. Thus, the LLM can iteratively provide feedback to retrieval and facilitate the retrieval result to sufficiently support verifiable generation. Experimental results show that our method significantly outperforms extensive baselines and achieves new state-of-the-art results.

Title: CPopQA: Ranking Cultural Concept Popularity by LLMs. (arXiv:2311.07897v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.07897
• Code URL: null
• Copy Paste: [[2311.07897]] CPopQA: Ranking Cultural Concept Popularity by LLMs(http://arxiv.org/abs/2311.07897)
• Summary:

  Prior work has demonstrated large language models' (LLMs) potential to discern statistical tendencies within their pre-training corpora. Despite that, many examinations of LLMs' knowledge capacity focus on knowledge explicitly appearing in the training data or implicitly inferable from similar contexts. How well an LLM captures the corpus-level statistical trends of concepts for reasoning, especially long-tail ones, is still underexplored. In this study, we introduce a novel few-shot question-answering task (CPopQA) that examines LLMs' statistical ranking abilities for long-tail cultural concepts (e.g., holidays), with a specific focus on these concepts' popularity in the United States and the United Kingdom, respectively. We curate a dataset containing 459 holidays across 58 countries, generating a total of 6,000 QA testing pairs. Experiments on four strong LLMs show that large models are capable of ranking long-tail cultural concepts regarding their statistical tendency. Notably, GPT-3.5 displayed superior performance and exhibited its potential to identify geo-cultural proximity across continents.

Title: Instruction-Following Evaluation for Large Language Models. (arXiv:2311.07911v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.07911
• Code URL: https://github.com/google-research/google-research
• Copy Paste: [[2311.07911]] Instruction-Following Evaluation for Large Language Models(http://arxiv.org/abs/2311.07911)
• Summary:

  One core capability of Large Language Models (LLMs) is to follow natural language instructions. However, the evaluation of such abilities is not standardized: Human evaluations are expensive, slow, and not objectively reproducible, while LLM-based auto-evaluation is potentially biased or limited by the ability of the evaluator LLM. To overcome these issues, we introduce Instruction-Following Eval (IFEval) for large language models. IFEval is a straightforward and easy-to-reproduce evaluation benchmark. It focuses on a set of "verifiable instructions" such as "write in more than 400 words" and "mention the keyword of AI at least 3 times". We identified 25 types of those verifiable instructions and constructed around 500 prompts, with each prompt containing one or more verifiable instructions. We show evaluation results of two widely available LLMs on the market. Our code and data can be found at https://github.com/google-research/google-research/tree/master/instruction_following_eval

Title: Automated title and abstract screening for scoping reviews using the GPT-4 Large Language Model. (arXiv:2311.07918v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.07918
• Code URL: null
• Copy Paste: [[2311.07918]] Automated title and abstract screening for scoping reviews using the GPT-4 Large Language Model(http://arxiv.org/abs/2311.07918)
• Summary:

  Scoping reviews, a type of literature review, require intensive human effort to screen large numbers of scholarly sources for their relevance to the review objectives. This manuscript introduces GPTscreenR, a package for the R statistical programming language that uses the GPT-4 Large Language Model (LLM) to automatically screen sources. The package makes use of the chain-of-thought technique with the goal of maximising performance on complex screening tasks. In validation against consensus human reviewer decisions, GPTscreenR performed similarly to an alternative zero-shot technique, with a sensitivity of 71%, specificity of 89%, and overall accuracy of 84%. Neither method achieved perfect accuracy nor human levels of intraobserver agreement. GPTscreenR demonstrates the potential for LLMs to support scholarly work and provides a user-friendly software framework that can be integrated into existing review processes.

Title: It's All Relative! -- A Synthetic Query Generation Approach for Improving Zero-Shot Relevance Prediction. (arXiv:2311.07930v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.07930
• Code URL: null
• Copy Paste: [[2311.07930]] It's All Relative! -- A Synthetic Query Generation Approach for Improving Zero-Shot Relevance Prediction(http://arxiv.org/abs/2311.07930)
• Summary:

  Recent developments in large language models (LLMs) have shown promise in their ability to generate synthetic query-document pairs by prompting with as few as 8 demonstrations. This has enabled building better IR models, especially for tasks with no training data readily available. Typically, such synthetic query generation (QGen) approaches condition on an input context (e.g. a text document) and generate a query relevant to that context, or condition the QGen model additionally on the relevance label (e.g. relevant vs irrelevant) to generate queries across relevance buckets. However, we find that such QGen approaches are sub-optimal as they require the model to reason about the desired label and the input from a handful of examples. In this work, we propose to reduce this burden of LLMs by generating queries simultaneously for different labels. We hypothesize that instead of asking the model to generate, say, an irrelevant query given an input context, asking the model to generate an irrelevant query relative to a relevant query is a much simpler task setup for the model to reason about. Extensive experimentation across seven IR datasets shows that synthetic queries generated in such a fashion translates to a better downstream performance, suggesting that the generated queries are indeed of higher quality.

Title: First Step Advantage: Importance of Starting Right in Multi-Step Reasoning. (arXiv:2311.07945v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.07945
• Code URL: null
• Copy Paste: [[2311.07945]] First Step Advantage: Importance of Starting Right in Multi-Step Reasoning(http://arxiv.org/abs/2311.07945)
• Summary:

  Large Language Models (LLMs) can solve complex reasoning tasks by generating rationales for their predictions. Distilling these capabilities into a smaller, compact model can facilitate the creation of specialized, cost-effective models tailored for specific tasks. However, smaller models often face challenges in complex reasoning tasks and often deviate from the correct reasoning path. We show that LLMs can guide smaller models and bring them back to the correct reasoning path only if they intervene at the right time. We show that smaller models fail to reason primarily due to their difficulty in initiating the process, and that guiding them in the right direction can lead to a performance gain of over 100%. We explore different model sizes and evaluate the benefits of providing guidance to improve reasoning in smaller models.

Title: How good are Large Language Models on African Languages?. (arXiv:2311.07978v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.07978
• Code URL: null
• Copy Paste: [[2311.07978]] How good are Large Language Models on African Languages?(http://arxiv.org/abs/2311.07978)
• Summary:

  Recent advancements in natural language processing have led to the proliferation of large language models (LLMs). These models have been shown to yield good performance, using in-context learning, even on unseen tasks and languages. Additionally, they have been widely adopted as language-model-as-a-service commercial APIs like GPT-4 API. However, their performance on African languages is largely unknown. We present an analysis of three popular large language models (mT0, LLaMa 2, and GPT-4) on five tasks (news topic classification, sentiment classification, machine translation, question answering, and named entity recognition) across 30 African languages, spanning different language families and geographical regions. Our results suggest that all LLMs produce below-par performance on African languages, and there is a large gap in performance compared to high-resource languages like English most tasks. We find that GPT-4 has an average or impressive performance on classification tasks but very poor results on generative tasks like machine translation. Surprisingly, we find that mT0 had the best overall on cross-lingual QA, better than the state-of-the-art supervised model (i.e. fine-tuned mT5) and GPT-4 on African languages. Overall, LLaMa 2 records the worst performance due to its limited multilingual capabilities and English-centric pre-training corpus. In general, our findings present a call-to-action to ensure African languages are well represented in large language models, given their growing popularity.

Title: Forgetting before Learning: Utilizing Parametric Arithmetic for Knowledge Updating in Large Language Models. (arXiv:2311.08011v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.08011
• Code URL: null
• Copy Paste: [[2311.08011]] Forgetting before Learning: Utilizing Parametric Arithmetic for Knowledge Updating in Large Language Models(http://arxiv.org/abs/2311.08011)
• Summary:

  Recently Large Language Models (LLMs) have demonstrated their amazing text understanding and generation capabilities. However, even stronger LLMs may still learn incorrect knowledge from the training corpus, as well as some knowledge that is outdated over time. Direct secondary fine-tuning with data containing new knowledge may be ineffective in updating knowledge due to the conflict between old and new knowledge. In this paper, we propose a new paradigm for fine-tuning called F-Learning (Forgetting before Learning), which is based on parametric arithmetic to achieve forgetting of old knowledge and learning of new knowledge. Experimental results on two publicly available datasets demonstrate that our proposed F-Learning can obviously improve the knowledge updating performance of both full fine-tuning and LoRA fine-tuning. Moreover, we have also discovered that forgetting old knowledge by subtracting the parameters of LoRA can achieve a similar effect to subtracting the parameters of full fine-tuning, and sometimes even surpass it significantly.

Title: Adversarial Preference Optimization. (arXiv:2311.08045v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.08045
• Code URL: null
• Copy Paste: [[2311.08045]] Adversarial Preference Optimization(http://arxiv.org/abs/2311.08045)
• Summary:

  Human preference alignment is a crucial training step to improve the interaction quality of large language models (LLMs). Existing aligning methods depend on manually annotated preference data to guide the LLM optimization directions. However, in practice, continuously updating LLMs raises a distribution gap between model-generated samples and human-preferred responses, which hinders model fine-tuning efficiency. To mitigate this issue, previous methods require additional preference annotation on generated samples to adapt the shifted distribution, which consumes a large amount of annotation resources. Targeting more efficient human preference optimization, we propose an adversarial preference optimization (APO) framework, where the LLM agent and the preference model update alternatively via a min-max game. Without additional annotation, our APO method can make a self-adaption to the generation distribution gap through the adversarial learning process. In experiments, we empirically verify the effectiveness of APO in improving LLM's helpfulness and harmlessness compared with rejection sampling baselines.

Title: SAIE Framework: Support Alone Isn't Enough -- Advancing LLM Training with Adversarial Remarks. (arXiv:2311.08107v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.08107
• Code URL: null
• Copy Paste: [[2311.08107]] SAIE Framework: Support Alone Isn't Enough -- Advancing LLM Training with Adversarial Remarks(http://arxiv.org/abs/2311.08107)
• Summary:

  Large Language Models (LLMs) can justify or criticize their predictions through discussion with other models or humans, thereby enhancing their intrinsic understanding of instances. While proactive discussions enhance performance, this approach is currently limited to the inference phase. In this context, we posit a hypothesis: learning interactive discussions during training can improve understanding for the instances in the training step and proficiency in logical/critical thinking ability and verbalized expression of the model in the inference step. Our proposed SAIE training method involves both supportive and adversarial discussions between the learner and partner models. The learner model receives a remark from the partner through the discussion, and the parameters of the learner model are then updated based on this remark. That is, the teacher signal dynamically adjusts in response to the evolving model output throughout the training step. By bolstering the capacity for discussion and comprehension of instances, our experiments across datasets, including GSM8K, CommonsenseQA, and MMLU, reveal that models fine-tuned with our method consistently surpass those trained with standard fine-tuning techniques. Moreover, our approach demonstrates superior performance in multi-agent inference scenarios, boosting the models' reasoning abilities at the inference step.

Title: Insights into Classifying and Mitigating LLMs' Hallucinations. (arXiv:2311.08117v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.08117
• Code URL: null
• Copy Paste: [[2311.08117]] Insights into Classifying and Mitigating LLMs' Hallucinations(http://arxiv.org/abs/2311.08117)
• Summary:

  The widespread adoption of large language models (LLMs) across diverse AI applications is proof of the outstanding achievements obtained in several tasks, such as text mining, text generation, and question answering. However, LLMs are not exempt from drawbacks. One of the most concerning aspects regards the emerging problematic phenomena known as "Hallucinations". They manifest in text generation systems, particularly in question-answering systems reliant on LLMs, potentially resulting in false or misleading information propagation. This paper delves into the underlying causes of AI hallucination and elucidates its significance in artificial intelligence. In particular, Hallucination classification is tackled over several tasks (Machine Translation, Question and Answer, Dialog Systems, Summarisation Systems, Knowledge Graph with LLMs, and Visual Question Answer). Additionally, we explore potential strategies to mitigate hallucinations, aiming to enhance the overall reliability of LLMs. Our research addresses this critical issue within the HeReFaNMi (Health-Related Fake News Mitigation) project, generously supported by NGI Search, dedicated to combating Health-Related Fake News dissemination on the Internet. This endeavour represents a concerted effort to safeguard the integrity of information dissemination in an age of evolving AI technologies.

Title: Towards Reasoning in Large Language Models via Multi-Agent Peer Review Collaboration. (arXiv:2311.08152v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.08152
• Code URL: https://github.com/hitsz-tmg/multi-agent-peer-review
• Copy Paste: [[2311.08152]] Towards Reasoning in Large Language Models via Multi-Agent Peer Review Collaboration(http://arxiv.org/abs/2311.08152)
• Summary:

  Large Language Models (LLMs) have shown remarkable capabilities in general natural language processing tasks but often fall short in complex reasoning tasks. Recent studies have explored human-like problem-solving strategies, such as self-correct, to push further the boundary of single-model reasoning ability. In this work, we let a single model "step outside the box" by engaging multiple models to correct each other. We introduce a multi-agent collaboration strategy that emulates the academic peer review process. Each agent independently constructs its own solution, provides reviews on the solutions of others, and assigns confidence levels to its reviews. Upon receiving peer reviews, agents revise their initial solutions. Extensive experiments on three different types of reasoning tasks show that our collaboration approach delivers superior accuracy across all ten datasets compared to existing methods. Further study demonstrates the effectiveness of integrating confidence in the reviews for math reasoning, and suggests a promising direction for human-mimicking multi-agent collaboration process.

Title: Self-Evolved Diverse Data Sampling for Efficient Instruction Tuning. (arXiv:2311.08182v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.08182
• Code URL: https://github.com/ofa-sys/diverseevol
• Copy Paste: [[2311.08182]] Self-Evolved Diverse Data Sampling for Efficient Instruction Tuning(http://arxiv.org/abs/2311.08182)
• Summary:

  Enhancing the instruction-following ability of Large Language Models (LLMs) primarily demands substantial instruction-tuning datasets. However, the sheer volume of these imposes a considerable computational burden and annotation cost. To investigate a label-efficient instruction tuning method that allows the model itself to actively sample subsets that are equally or even more effective, we introduce a self-evolving mechanism DiverseEvol. In this process, a model iteratively augments its training subset to refine its own performance, without requiring any intervention from humans or more advanced LLMs. The key to our data sampling technique lies in the enhancement of diversity in the chosen subsets, as the model selects new data points most distinct from any existing ones according to its current embedding space. Extensive experiments across three datasets and benchmarks demonstrate the effectiveness of DiverseEvol. Our models, trained on less than 8% of the original dataset, maintain or improve performance compared with finetuning on full data. We also provide empirical evidence to analyze the importance of diversity in instruction data and the iterative scheme as opposed to one-time sampling. Our code is publicly available at https://github.com/OFA-Sys/DiverseEvol.git.

Title: Human-Centric Autonomous Systems With LLMs for User Command Reasoning. (arXiv:2311.08206v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.08206
• Code URL: null
• Copy Paste: [[2311.08206]] Human-Centric Autonomous Systems With LLMs for User Command Reasoning(http://arxiv.org/abs/2311.08206)
• Summary:

  The evolution of autonomous driving has made remarkable advancements in recent years, evolving into a tangible reality. However, a human-centric large-scale adoption hinges on meeting a variety of multifaceted requirements. To ensure that the autonomous system meets the user's intent, it is essential to accurately discern and interpret user commands, especially in complex or emergency situations. To this end, we propose to leverage the reasoning capabilities of Large Language Models (LLMs) to infer system requirements from in-cabin users' commands. Through a series of experiments that include different LLM models and prompt designs, we explore the few-shot multivariate binary classification accuracy of system requirements from natural language textual commands. We confirm the general ability of LLMs to understand and reason about prompts but underline that their effectiveness is conditioned on the quality of both the LLM model and the design of appropriate sequential prompts. Code and models are public with the link \url{https://github.com/KTH-RPL/DriveCmd_LLM}.

Title: A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily. (arXiv:2311.08268v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.08268
• Code URL: null
• Copy Paste: [[2311.08268]] A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily(http://arxiv.org/abs/2311.08268)
• Summary:

  Large Language Models (LLMs), such as ChatGPT and GPT-4, are designed to provide useful and safe responses. However, adversarial prompts known as 'jailbreaks' can circumvent safeguards, leading LLMs to generate harmful content. Exploring jailbreak prompts can help to better reveal the weaknesses of LLMs and further steer us to secure them. Unfortunately, existing jailbreak methods either suffer from intricate manual design or require optimization on another white-box model, compromising generalization or jailbreak efficiency. In this paper, we generalize jailbreak prompt attacks into two aspects: (1) Prompt Rewriting and (2) Scenario Nesting. Based on this, we propose ReNeLLM, an automatic framework that leverages LLMs themselves to generate effective jailbreak prompts. Extensive experiments demonstrate that ReNeLLM significantly improves the attack success rate while greatly reducing the time cost compared to existing baselines. Our study also reveals the inadequacy of current defense methods in safeguarding LLMs. Finally, we offer detailed analysis and discussion from the perspective of prompt execution priority on the failure of LLMs' defense. We hope that our research can catalyze both the academic community and LLMs vendors towards the provision of safer and more regulated Large Language Models.

segmentation

Title: A Metacognitive Approach to Out-of-Distribution Detection for Segmentation. (arXiv:2311.07578v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.07578
• Code URL: null
• Copy Paste: [[2311.07578]] A Metacognitive Approach to Out-of-Distribution Detection for Segmentation(http://arxiv.org/abs/2311.07578)
• Summary:

  Despite outstanding semantic scene segmentation in closed-worlds, deep neural networks segment novel instances poorly, which is required for autonomous agents acting in an open world. To improve out-of-distribution (OOD) detection for segmentation, we introduce a metacognitive approach in the form of a lightweight module that leverages entropy measures, segmentation predictions, and spatial context to characterize the segmentation model's uncertainty and detect pixel-wise OOD data in real-time. Additionally, our approach incorporates a novel method of generating synthetic OOD data in context with in-distribution data, which we use to fine-tune existing segmentation models with maximum entropy training. This further improves the metacognitive module's performance without requiring access to OOD data while enabling compatibility with established pre-trained models. Our resulting approach can reliably detect OOD instances in a scene, as shown by state-of-the-art performance on OOD detection for semantic segmentation benchmarks.

Title: Histopathologic Cancer Detection. (arXiv:2311.07711v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.07711
• Code URL: https://github.com/lbasyal/Histopathologic-Cancer-Detection-
• Copy Paste: [[2311.07711]] Histopathologic Cancer Detection(http://arxiv.org/abs/2311.07711)
• Summary:

  Early diagnosis of the cancer cells is necessary for making an effective treatment plan and for the health and safety of a patient. Nowadays, doctors usually use a histological grade that pathologists determine by performing a semi-quantitative analysis of the histopathological and cytological features of hematoxylin-eosin (HE) stained histopathological images. This research contributes a potential classification model for cancer prognosis to efficiently utilize the valuable information underlying the HE-stained histopathological images. This work uses the PatchCamelyon benchmark datasets and trains them in a multi-layer perceptron and convolution model to observe the model's performance in terms of precision, Recall, F1 Score, Accuracy, and AUC Score. The evaluation result shows that the baseline convolution model outperforms the baseline MLP model. Also, this paper introduced ResNet50 and InceptionNet models with data augmentation, where ResNet50 is able to beat the state-of-the-art model. Furthermore, the majority vote and concatenation ensemble were evaluated and provided the future direction of using transfer learning and segmentation to understand the specific features.

Title: Assessing Test-time Variability for Interactive 3D Medical Image Segmentation with Diverse Point Prompts. (arXiv:2311.07806v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.07806
• Code URL: https://github.com/medicl-vu/variability
• Copy Paste: [[2311.07806]] Assessing Test-time Variability for Interactive 3D Medical Image Segmentation with Diverse Point Prompts(http://arxiv.org/abs/2311.07806)
• Summary:

  Interactive segmentation model leverages prompts from users to produce robust segmentation. This advancement is facilitated by prompt engineering, where interactive prompts serve as strong priors during test-time. However, this is an inherently subjective and hard-to-reproduce process. The variability in user expertise and inherently ambiguous boundaries in medical images can lead to inconsistent prompt selections, potentially affecting segmentation accuracy. This issue has not yet been extensively explored for medical imaging. In this paper, we assess the test-time variability for interactive medical image segmentation with diverse point prompts. For a given target region, the point is classified into three sub-regions: boundary, margin, and center. Our goal is to identify a straightforward and efficient approach for optimal prompt selection during test-time based on three considerations: (1) benefits of additional prompts, (2) effects of prompt placement, and (3) strategies for optimal prompt selection. We conduct extensive experiments on the public Medical Segmentation Decathlon dataset for challenging colon tumor segmentation task. We suggest an optimal strategy for prompt selection during test-time, supported by comprehensive results. The code is publicly available at https://github.com/MedICL-VU/variability

Title: Test-Time Training for Semantic Segmentation with Output Contrastive Loss. (arXiv:2311.07877v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.07877
• Code URL: https://github.com/dazhangyu123/ocl
• Copy Paste: [[2311.07877]] Test-Time Training for Semantic Segmentation with Output Contrastive Loss(http://arxiv.org/abs/2311.07877)
• Summary:

  Although deep learning-based segmentation models have achieved impressive performance on public benchmarks, generalizing well to unseen environments remains a major challenge. To improve the model's generalization ability to the new domain during evaluation, the test-time training (TTT) is a challenging paradigm that adapts the source-pretrained model in an online fashion. Early efforts on TTT mainly focus on the image classification task. Directly extending these methods to semantic segmentation easily experiences unstable adaption due to segmentation's inherent characteristics, such as extreme class imbalance and complex decision spaces. To stabilize the adaptation process, we introduce contrastive loss (CL), known for its capability to learn robust and generalized representations. Nevertheless, the traditional CL operates in the representation space and cannot directly enhance predictions. In this paper, we resolve this limitation by adapting the CL to the output space, employing a high temperature, and simplifying the formulation, resulting in a straightforward yet effective loss function called Output Contrastive Loss (OCL). Our comprehensive experiments validate the efficacy of our approach across diverse evaluation scenarios. Notably, our method excels even when applied to models initially pre-trained using domain adaptation methods on test domain data, showcasing its resilience and adaptability.\footnote{Code and more information could be found at~ \url{https://github.com/dazhangyu123/OCL}}

Title: Zero-Shot Segmentation of Eye Features Using the Segment Anything Model (SAM). (arXiv:2311.08077v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.08077
• Code URL: null
• Copy Paste: [[2311.08077]] Zero-Shot Segmentation of Eye Features Using the Segment Anything Model (SAM)(http://arxiv.org/abs/2311.08077)
• Summary:

  The advent of foundation models signals a new era in artificial intelligence. The Segment Anything Model (SAM) is the first foundation model for image segmentation. In this study, we evaluate SAM's ability to segment features from eye images recorded in virtual reality setups. The increasing requirement for annotated eye-image datasets presents a significant opportunity for SAM to redefine the landscape of data annotation in gaze estimation. Our investigation centers on SAM's zero-shot learning abilities and the effectiveness of prompts like bounding boxes or point clicks. Our results are consistent with studies in other domains, demonstrating that SAM's segmentation effectiveness can be on-par with specialized models depending on the feature, with prompts improving its performance, evidenced by an IoU of 93.34% for pupil segmentation in one dataset. Foundation models like SAM could revolutionize gaze estimation by enabling quick and easy image segmentation, reducing reliance on specialized models and extensive manual annotation.