2024-01-29

Title: Harnessing Neuron Stability to Improve DNN Verification

• Authors: Hai Duong, Dong Xu, ThanhVu Nguyen, Matthew B. Dwyer
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.14412
• Pdf URL: https://arxiv.org/pdf/2401.14412
• Copy Paste: [[2401.14412]] Harnessing Neuron Stability to Improve DNN Verification(https://arxiv.org/abs/2401.14412)
• Keywords: attack
• Abstract: Deep Neural Networks (DNN) have emerged as an effective approach to tackling real-world problems. However, like human-written software, DNNs are susceptible to bugs and attacks. This has generated significant interests in developing effective and scalable DNN verification techniques and tools. In this paper, we present VeriStable, a novel extension of recently proposed DPLL-based constraint DNN verification approach. VeriStable leverages the insight that while neuron behavior may be non-linear across the entire DNN input space, at intermediate states computed during verification many neurons may be constrained to have linear behavior - these neurons are stable. Efficiently detecting stable neurons reduces combinatorial complexity without compromising the precision of abstractions. Moreover, the structure of clauses arising in DNN verification problems shares important characteristics with industrial SAT benchmarks. We adapt and incorporate multi-threading and restart optimizations targeting those characteristics to further optimize DPLL-based DNN verification. We evaluate the effectiveness of VeriStable across a range of challenging benchmarks including fully-connected feedforward networks (FNNs), convolutional neural networks (CNNs) and residual networks (ResNets) applied to the standard MNIST and CIFAR datasets. Preliminary results show that VeriStable is competitive and outperforms state-of-the-art DNN verification tools, including $\alpha$-$\beta$-CROWN and MN-BaB, the first and second performers of the VNN-COMP, respectively.

Title: A Novel Blockchain Based Information Management Framework for Web 3.0

• Authors: Md Arif Hassan, Cong T. Nguyen, Chi-Hieu Nguyen, Dinh Thai Hoang, Diep N. Nguyen, Eryk Dutkiewicz
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2401.14420
• Pdf URL: https://arxiv.org/pdf/2401.14420
• Copy Paste: [[2401.14420]] A Novel Blockchain Based Information Management Framework for Web 3.0(https://arxiv.org/abs/2401.14420)
• Keywords: security, privacy
• Abstract: Web 3.0 is the third generation of the World Wide Web (WWW), concentrating on the critical concepts of decentralization, availability, and increasing client usability. Although Web 3.0 is undoubtedly an essential component of the future Internet, it currently faces critical challenges, including decentralized data collection and management. To overcome these challenges, blockchain has emerged as one of the core technologies for the future development of Web 3.0. In this paper, we propose a novel blockchain-based information management framework, namely Smart Blockchain-based Web, to manage information in Web 3.0 effectively, enhance the security and privacy of users data, bring additional profits, and incentivize users to contribute information to the websites. Particularly, SBW utilizes blockchain technology and smart contracts to manage the decentralized data collection process for Web 3.0 effectively. Moreover, in this framework, we develop an effective consensus mechanism based on Proof-of-Stake to reward the user's information contribution and conduct game theoretical analysis to analyze the users behavior in the considered system. Additionally, we conduct simulations to assess the performance of SBW and investigate the impact of critical parameters on information contribution. The findings confirm our theoretical analysis and demonstrate that our proposed consensus mechanism can incentivize the nodes and users to contribute more information to our systems.

Title: Multi-Agent Based Transfer Learning for Data-Driven Air Traffic Applications

• Authors: Chuhao Deng, Hong-Cheol Choi, Hyunsang Park, Inseok Hwang
• Subjects: cs.LG, cs.MA, eess.SY, stat.ML
• Abstract URL: https://arxiv.org/abs/2401.14421
• Pdf URL: https://arxiv.org/pdf/2401.14421
• Copy Paste: [[2401.14421]] Multi-Agent Based Transfer Learning for Data-Driven Air Traffic Applications(https://arxiv.org/abs/2401.14421)
• Keywords: transformer
• Abstract: Research in developing data-driven models for Air Traffic Management (ATM) has gained a tremendous interest in recent years. However, data-driven models are known to have long training time and require large datasets to achieve good performance. To address the two issues, this paper proposes a Multi-Agent Bidirectional Encoder Representations from Transformers (MA-BERT) model that fully considers the multi-agent characteristic of the ATM system and learns air traffic controllers' decisions, and a pre-training and fine-tuning transfer learning framework. By pre-training the MA-BERT on a large dataset from a major airport and then fine-tuning it to other airports and specific air traffic applications, a large amount of the total training time can be saved. In addition, for newly adopted procedures and constructed airports where no historical data is available, this paper shows that the pre-trained MA-BERT can achieve high performance by updating regularly with little data. The proposed transfer learning framework and MA-BERT are tested with the automatic dependent surveillance-broadcast data recorded in 3 airports in South Korea in 2019.

Title: Discovering Mathematical Formulas from Data via GPT-guided Monte Carlo Tree Search

• Authors: Yanjie Li, Weijun Li, Lina Yu, Min Wu, Jingyi Liu, Wenqiang Li, Meilan Hao, Shu Wei, Yusong Deng
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.14424
• Pdf URL: https://arxiv.org/pdf/2401.14424
• Copy Paste: [[2401.14424]] Discovering Mathematical Formulas from Data via GPT-guided Monte Carlo Tree Search(https://arxiv.org/abs/2401.14424)
• Keywords: transformer, generative
• Abstract: Finding a concise and interpretable mathematical formula that accurately describes the relationship between each variable and the predicted value in the data is a crucial task in scientific research, as well as a significant challenge in artificial intelligence. This problem is referred to as symbolic regression, which is an NP-hard problem. Last year, a symbolic regression method based on Monte Carlo Tree Search (MCTS) was proposed and sota was obtained on multiple datasets. While this algorithm has shown considerable improvement in recovering target expressions compared to previous methods, the lack of guidance during the MCTS process severely hampers its search efficiency. Recently, some algorithms have added a pre-trained policy network to guide the search of MCTS, but the pre-trained policy network generalizes poorly. To balance efficiency and generality, we propose SR-GPT combining ideas from AlphaZero. SR-GPT is a new symbolic regression algorithm that combines MCTS with a Generative Pre-Trained Transformer (GPT). By using GPT to guide the MCTS process, the search efficiency of MCTS is significantly improved. Next, we utilize the MCTS results to further refine the GPT, enhancing its capabilities and providing more accurate guidance for the MCTS process. MCTS and GPT are coupled together and optimize each other until the target expression is successfully determined. We conducted extensive evaluations of SR-GPT using 222 expressions sourced from over 10 different symbolic regression datasets. The experimental results demonstrate that SR-GPT outperforms existing state-of-the-art algorithms in accurately recovering symbolic expressions both with and without added noise.

Title: Semantic Sensitivities and Inconsistent Predictions: Measuring the Fragility of NLI Models

• Authors: Erik Arakelyan, Zhaoqi Liu, Isabelle Augenstein
• Subjects: cs.CL, cs.AI, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.14440
• Pdf URL: https://arxiv.org/pdf/2401.14440
• Copy Paste: [[2401.14440]] Semantic Sensitivities and Inconsistent Predictions: Measuring the Fragility of NLI Models(https://arxiv.org/abs/2401.14440)
• Keywords: robust, transformer
• Abstract: Recent studies of the emergent capabilities of transformer-based Natural Language Understanding (NLU) models have indicated that they have an understanding of lexical and compositional semantics. We provide evidence that suggests these claims should be taken with a grain of salt: we find that state-of-the-art Natural Language Inference (NLI) models are sensitive towards minor semantics preserving surface-form variations, which lead to sizable inconsistent model decisions during inference. Notably, this behaviour differs from valid and in-depth comprehension of compositional semantics, however does neither emerge when evaluating model accuracy on standard benchmarks nor when probing for syntactic, monotonic, and logically robust reasoning. We propose a novel framework to measure the extent of semantic sensitivity. To this end, we evaluate NLI models on adversarially generated examples containing minor semantics-preserving surface-form input noise. This is achieved using conditional text generation, with the explicit condition that the NLI model predicts the relationship between the original and adversarial inputs as a symmetric equivalence entailment. We systematically study the effects of the phenomenon across NLI models for \emph{in-} and \emph{out-of} domain settings. Our experiments show that semantic sensitivity causes performance degradations of $12.92\%$ and $23.71\%$ average over \emph{in-} and \emph{out-of-} domain settings, respectively. We further perform ablation studies, analysing this phenomenon across models, datasets, and variations in inference and show that semantic sensitivity can lead to major inconsistency within model predictions.

Title: CloudTracks: A Dataset for Localizing Ship Tracks in Satellite Images of Clouds

• Authors: Muhammad Ahmed Chaudhry, Lyna Kim, Jeremy Irvin, Yuzu Ido, Sonia Chu, Jared Thomas Isobe, Andrew Y. Ng, Duncan Watson-Parris
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.14486
• Pdf URL: https://arxiv.org/pdf/2401.14486
• Copy Paste: [[2401.14486]] CloudTracks: A Dataset for Localizing Ship Tracks in Satellite Images of Clouds(https://arxiv.org/abs/2401.14486)
• Keywords: segmentation
• Abstract: Clouds play a significant role in global temperature regulation through their effect on planetary albedo. Anthropogenic emissions of aerosols can alter the albedo of clouds, but the extent of this effect, and its consequent impact on temperature change, remains uncertain. Human-induced clouds caused by ship aerosol emissions, commonly referred to as ship tracks, provide visible manifestations of this effect distinct from adjacent cloud regions and therefore serve as a useful sandbox to study human-induced clouds. However, the lack of large-scale ship track data makes it difficult to deduce their general effects on cloud formation. Towards developing automated approaches to localize ship tracks at scale, we present CloudTracks, a dataset containing 3,560 satellite images labeled with more than 12,000 ship track instance annotations. We train semantic segmentation and instance segmentation model baselines on our dataset and find that our best model substantially outperforms previous state-of-the-art for ship track localization (61.29 vs. 48.65 IoU). We also find that the best instance segmentation model is able to identify the number of ship tracks in each image more accurately than the previous state-of-the-art (1.64 vs. 4.99 MAE). However, we identify cases where the best model struggles to accurately localize and count ship tracks, so we believe CloudTracks will stimulate novel machine learning approaches to better detect elongated and overlapping features in satellite images. We release our dataset openly at {zenodo.org/records/10042922}.

Title: Neighbor-Aware Calibration of Segmentation Networks with Penalty-Based Constraints

• Authors: Balamurali Murugesan, Sukesh Adiga Vasudeva, Bingyuan Liu, Hervé Lombaert, Ismail Ben Ayed, Jose Dolz
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.14487
• Pdf URL: https://arxiv.org/pdf/2401.14487
• Copy Paste: [[2401.14487]] Neighbor-Aware Calibration of Segmentation Networks with Penalty-Based Constraints(https://arxiv.org/abs/2401.14487)
• Keywords: segmentation
• Abstract: Ensuring reliable confidence scores from deep neural networks is of paramount significance in critical decision-making systems, particularly in real-world domains such as healthcare. Recent literature on calibrating deep segmentation networks has resulted in substantial progress. Nevertheless, these approaches are strongly inspired by the advancements in classification tasks, and thus their uncertainty is usually modeled by leveraging the information of individual pixels, disregarding the local structure of the object of interest. Indeed, only the recent Spatially Varying Label Smoothing (SVLS) approach considers pixel spatial relationships across classes, by softening the pixel label assignments with a discrete spatial Gaussian kernel. In this work, we first present a constrained optimization perspective of SVLS and demonstrate that it enforces an implicit constraint on soft class proportions of surrounding pixels. Furthermore, our analysis shows that SVLS lacks a mechanism to balance the contribution of the constraint with the primary objective, potentially hindering the optimization process. Based on these observations, we propose NACL (Neighbor Aware CaLibration), a principled and simple solution based on equality constraints on the logit values, which enables to control explicitly both the enforced constraint and the weight of the penalty, offering more flexibility. Comprehensive experiments on a wide variety of well-known segmentation benchmarks demonstrate the superior calibration performance of the proposed approach, without affecting its discriminative power. Furthermore, ablation studies empirically show the model agnostic nature of our approach, which can be used to train a wide span of deep segmentation networks.

Title: LongHealth: A Question Answering Benchmark with Long Clinical Documents

• Authors: Lisa Adams, Felix Busch, Tianyu Han, Jean-Baptiste Excoffier, Matthieu Ortala, Alexander Löser, Hugo JWL. Aerts, Jakob Nikolas Kather, Daniel Truhn, Keno Bressem
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.14490
• Pdf URL: https://arxiv.org/pdf/2401.14490
• Copy Paste: [[2401.14490]] LongHealth: A Question Answering Benchmark with Long Clinical Documents(https://arxiv.org/abs/2401.14490)
• Keywords: extraction, large language model
• Abstract: Background: Recent advancements in large language models (LLMs) offer potential benefits in healthcare, particularly in processing extensive patient records. However, existing benchmarks do not fully assess LLMs' capability in handling real-world, lengthy clinical data. Methods: We present the LongHealth benchmark, comprising 20 detailed fictional patient cases across various diseases, with each case containing 5,090 to 6,754 words. The benchmark challenges LLMs with 400 multiple-choice questions in three categories: information extraction, negation, and sorting, challenging LLMs to extract and interpret information from large clinical documents. Results: We evaluated nine open-source LLMs with a minimum of 16,000 tokens and also included OpenAI's proprietary and cost-efficient GPT-3.5 Turbo for comparison. The highest accuracy was observed for Mixtral-8x7B-Instruct-v0.1, particularly in tasks focused on information retrieval from single and multiple patient documents. However, all models struggled significantly in tasks requiring the identification of missing information, highlighting a critical area for improvement in clinical data interpretation. Conclusion: While LLMs show considerable potential for processing long clinical documents, their current accuracy levels are insufficient for reliable clinical use, especially in scenarios requiring the identification of missing information. The LongHealth benchmark provides a more realistic assessment of LLMs in a healthcare setting and highlights the need for further model refinement for safe and effective clinical application. We make the benchmark and evaluation code publicly available.

Title: K-QA: A Real-World Medical Q&A Benchmark

• Authors: Itay Manes, Naama Ronn, David Cohen, Ran Ilan Ber, Zehavi Horowitz-Kugler, Gabriel Stanovsky
• Subjects: cs.CL, cs.HC, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.14493
• Pdf URL: https://arxiv.org/pdf/2401.14493
• Copy Paste: [[2401.14493]] K-QA: A Real-World Medical Q&A Benchmark(https://arxiv.org/abs/2401.14493)
• Keywords: large language model
• Abstract: Ensuring the accuracy of responses provided by large language models (LLMs) is crucial, particularly in clinical settings where incorrect information may directly impact patient health. To address this challenge, we construct K-QA, a dataset containing 1,212 patient questions originating from real-world conversations held on K Health (an AI-driven clinical platform). We employ a panel of in-house physicians to answer and manually decompose a subset of K-QA into self-contained statements. Additionally, we formulate two NLI-based evaluation metrics approximating recall and precision: (1) comprehensiveness, measuring the percentage of essential clinical information in the generated answer and (2) hallucination rate, measuring the number of statements from the physician-curated response contradicted by the LLM answer. Finally, we use K-QA along with these metrics to evaluate several state-of-the-art models, as well as the effect of in-context learning and medically-oriented augmented retrieval schemes developed by the authors. Our findings indicate that in-context learning improves the comprehensiveness of the models, and augmented retrieval is effective in reducing hallucinations. We make K-QA available to to the community to spur research into medically accurate NLP applications.

Title: RPNR: Robust-Perception Neural Reshading

• Authors: Fouad Afiouni, Mohamad Fakih, Joey Sleiman
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.14510
• Pdf URL: https://arxiv.org/pdf/2401.14510
• Copy Paste: [[2401.14510]] RPNR: Robust-Perception Neural Reshading(https://arxiv.org/abs/2401.14510)
• Keywords: robust
• Abstract: Augmented Reality (AR) applications necessitates methods of inserting needed objects into scenes captured by cameras in a way that is coherent with the surroundings. Common AR applications require the insertion of predefined 3D objects with known properties and shape. This simplifies the problem since it is reduced to extracting an illumination model for the object in that scene by understanding the surrounding light sources. However, it is often not the case that we have information about the properties of an object, especially when we depart from a single source image. Our method renders such source fragments in a coherent way with the target surroundings using only these two images. Our pipeline uses a Deep Image Prior (DIP) network based on a U-Net architecture as the main renderer, alongside robust-feature extracting networks that are used to apply needed losses. Our method does not require any pair-labeled data, and no extensive training on a dataset. We compare our method using qualitative metrics to the baseline methods such as Cut and Paste, Cut And Paste Neural Rendering, and Image Harmonization

Title: Evaluating GPT-3.5's Awareness and Summarization Abilities for European Constitutional Texts with Shared Topics

• Authors: Candida M. Greco, A. Tagarelli
• Subjects: cs.CL, cs.AI, cs.CY, cs.DL, physics.soc-ph
• Abstract URL: https://arxiv.org/abs/2401.14524
• Pdf URL: https://arxiv.org/pdf/2401.14524
• Copy Paste: [[2401.14524]] Evaluating GPT-3.5's Awareness and Summarization Abilities for European Constitutional Texts with Shared Topics(https://arxiv.org/abs/2401.14524)
• Keywords: generative, large language model
• Abstract: Constitutions are foundational legal documents that underpin the governmental and societal structures. As such, they are a reflection of a nation's cultural and social uniqueness, but also contribute to establish topics of universal importance, like citizens' rights and duties (RD). In this work, using the renowned GPT-3.5, we leverage generative large language models to understand constitutional passages that transcend national boundaries. A key contribution of our study is the introduction of a novel application of abstractive summarization on a multi-source collection of constitutional texts, with a focus on European countries' constitution passages related to RD topics. Our results show the meaningfulness of GPT-3.5 to produce informative, coherent and faithful summaries capturing RD topics across European countries.

Title: MEDs for PETs: Multilingual Euphemism Disambiguation for Potentially Euphemistic Terms

• Authors: Patrick Lee, Alain Chirino Trujillo, Diana Cuevas Plancarte, Olumide Ebenezer Ojo, Xinyi Liu, Iyanuoluwa Shode, Yuan Zhao, Jing Peng, Anna Feldman
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.14526
• Pdf URL: https://arxiv.org/pdf/2401.14526
• Copy Paste: [[2401.14526]] MEDs for PETs: Multilingual Euphemism Disambiguation for Potentially Euphemistic Terms(https://arxiv.org/abs/2401.14526)
• Keywords: transformer
• Abstract: This study investigates the computational processing of euphemisms, a universal linguistic phenomenon, across multiple languages. We train a multilingual transformer model (XLM-RoBERTa) to disambiguate potentially euphemistic terms (PETs) in multilingual and cross-lingual settings. In line with current trends, we demonstrate that zero-shot learning across languages takes place. We also show cases where multilingual models perform better on the task compared to monolingual models by a statistically significant margin, indicating that multilingual data presents additional opportunities for models to learn about cross-lingual, computational properties of euphemisms. In a follow-up analysis, we focus on universal euphemistic "categories" such as death and bodily functions among others. We test to see whether cross-lingual data of the same domain is more important than within-language data of other domains to further understand the nature of the cross-lingual transfer.

Title: Unsealing the secrets of blockchain consensus: A systematic comparison of the formal security of proof-of-work and proof-of-stake

• Authors: Iván Abellán Álvarez, Vincent Gramlich, Johannes Sedlmeir
• Subjects: cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/2401.14527
• Pdf URL: https://arxiv.org/pdf/2401.14527
• Copy Paste: [[2401.14527]] Unsealing the secrets of blockchain consensus: A systematic comparison of the formal security of proof-of-work and proof-of-stake(https://arxiv.org/abs/2401.14527)
• Keywords: security
• Abstract: With the increasing adoption of decentralized information systems based on a variety of permissionless blockchain networks, the choice of consensus mechanism is at the core of many controversial discussions. Ethereum's recent transition from (PoW) to proof-of-stake (PoS)-based consensus has further fueled the debate on which mechanism is more favorable. While the aspects of energy consumption and degree of (de-)centralization are often emphasized in the public discourse, seminal research has also shed light on the formal security aspects of both approaches individually. However, related work has not yet comprehensively structured the knowledge about the security properties of PoW and PoS. Rather, it has focused on in-depth analyses of specific protocols or high-level comparative reviews covering a broad range of consensus mechanisms. To fill this gap and unravel the commonalities and discrepancies between the formal security properties of PoW- and PoS-based consensus, we conduct a systematic literature review over 26 research articles. Our findings indicate that PoW-based consensus with the longest chain rule provides the strongest formal security guarantees. Nonetheless, PoS can achieve similar guarantees when addressing its more pronounced tradeoff between safety and liveness through hybrid approaches.

Title: Relative Value Biases in Large Language Models

• Authors: William M. Hayes, Nicolas Yax, Stefano Palminteri
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.14530
• Pdf URL: https://arxiv.org/pdf/2401.14530
• Copy Paste: [[2401.14530]] Relative Value Biases in Large Language Models(https://arxiv.org/abs/2401.14530)
• Keywords: large language model
• Abstract: Studies of reinforcement learning in humans and animals have demonstrated a preference for options that yielded relatively better outcomes in the past, even when those options are associated with lower absolute reward. The present study tested whether large language models would exhibit a similar bias. We had gpt-4-1106-preview (GPT-4 Turbo) and Llama-2-70B make repeated choices between pairs of options with the goal of maximizing payoffs. A complete record of previous outcomes was included in each prompt. Both models exhibited relative value decision biases similar to those observed in humans and animals. Making relative comparisons among outcomes more explicit magnified the bias, whereas prompting the models to estimate expected outcomes caused the bias to disappear. These results have implications for the potential mechanisms that contribute to context-dependent choice in human agents.

Title: CaRiNG: Learning Temporal Causal Representation under Non-Invertible Generation Process

• Authors: Guangyi Chen, Yifan Shen, Zhenhao Chen, Xiangchen Song, Yuewen Sun, Weiran Yao, Xiao Liu, Kun Zhang
• Subjects: cs.LG, cs.CV, stat.ME
• Abstract URL: https://arxiv.org/abs/2401.14535
• Pdf URL: https://arxiv.org/pdf/2401.14535
• Copy Paste: [[2401.14535]] CaRiNG: Learning Temporal Causal Representation under Non-Invertible Generation Process(https://arxiv.org/abs/2401.14535)
• Keywords: robust, generative
• Abstract: Identifying the underlying time-delayed latent causal processes in sequential data is vital for grasping temporal dynamics and making downstream reasoning. While some recent methods can robustly identify these latent causal variables, they rely on strict assumptions about the invertible generation process from latent variables to observed data. However, these assumptions are often hard to satisfy in real-world applications containing information loss. For instance, the visual perception process translates a 3D space into 2D images, or the phenomenon of persistence of vision incorporates historical data into current perceptions. To address this challenge, we establish an identifiability theory that allows for the recovery of independent latent components even when they come from a nonlinear and non-invertible mix. Using this theory as a foundation, we propose a principled approach, CaRiNG, to learn the CAusal RepresentatIon of Non-invertible Generative temporal data with identifiability guarantees. Specifically, we utilize temporal context to recover lost latent information and apply the conditions in our theory to guide the training process. Through experiments conducted on synthetic datasets, we validate that our CaRiNG method reliably identifies the causal process, even when the generation process is non-invertible. Moreover, we demonstrate that our approach considerably improves temporal understanding and reasoning in practical applications.

Title: Revisiting Active Learning in the Era of Vision Foundation Models

• Authors: Sanket Rajan Gupte, Josiah Aklilu, Jeffrey J. Nirschl, Serena Yeung-Levy
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.14555
• Pdf URL: https://arxiv.org/pdf/2401.14555
• Copy Paste: [[2401.14555]] Revisiting Active Learning in the Era of Vision Foundation Models(https://arxiv.org/abs/2401.14555)
• Keywords: robust
• Abstract: Foundation vision or vision-language models are trained on large unlabeled or noisy data and learn robust representations that can achieve impressive zero- or few-shot performance on diverse tasks. Given these properties, they are a natural fit for active learning (AL), which aims to maximize labeling efficiency, but the full potential of foundation models has not been explored in the context of AL, specifically in the low-budget regime. In this work, we evaluate how foundation models influence three critical components of effective AL, namely, 1) initial labeled pool selection, 2) ensuring diverse sampling, and 3) the trade-off between representative and uncertainty sampling. We systematically study how the robust representations of foundation models (DINOv2, OpenCLIP) challenge existing findings in active learning. Our observations inform the principled construction of a new simple and elegant AL strategy that balances uncertainty estimated via dropout with sample diversity. We extensively test our strategy on many challenging image classification benchmarks, including natural images as well as out-of-domain biomedical images that are relatively understudied in the AL literature. Source code will be made available.

Title: Do Not (Always) Look Right: Investigating the Capabilities of Decoder-Based Large Language Models for Sequence Labeling

• Authors: David Dukić, Jan Šnajder
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.14556
• Pdf URL: https://arxiv.org/pdf/2401.14556
• Copy Paste: [[2401.14556]] Do Not (Always) Look Right: Investigating the Capabilities of Decoder-Based Large Language Models for Sequence Labeling(https://arxiv.org/abs/2401.14556)
• Keywords: extraction, large language model
• Abstract: Pre-trained language models based on masked language modeling (MLM) objective excel in natural language understanding (NLU) tasks. While fine-tuned MLM-based encoders consistently outperform causal language modeling decoders of comparable size, a recent trend of scaling decoder models to multiple billion parameters resulted in large language models (LLMs), making them competitive with MLM-based encoders. Although scale amplifies their prowess in NLU tasks, LLMs fall short of SOTA results in information extraction (IE) tasks, many framed as sequence labeling (SL). However, whether this is an intrinsic limitation of LLMs or whether their SL performance can be improved remains unclear. To address this, we explore strategies to enhance the SL performance of "open" LLMs (Llama2 and Mistral) on IE tasks. We investigate bidirectional information flow within groups of decoder blocks, applying layer-wise removal or enforcement of the causal mask (CM) during LLM fine-tuning. This approach yields performance gains competitive with SOTA SL models, matching or outperforming the results of CM removal from all blocks. Our findings hold for diverse SL tasks, proving that "open" LLMs with layer-dependent CM removal outperform strong MLM-based encoders and instruction-tuned LLMs. However, we observe no effect from CM removal on a small scale when maintaining an equivalent model size, pre-training steps, and pre-training and fine-tuning data.

Title: Language Modelling Approaches to Adaptive Machine Translation

• Authors: Yasmin Moslem
• Subjects: cs.CL, cs.AI, cs.HC, cs.IR
• Abstract URL: https://arxiv.org/abs/2401.14559
• Pdf URL: https://arxiv.org/pdf/2401.14559
• Copy Paste: [[2401.14559]] Language Modelling Approaches to Adaptive Machine Translation(https://arxiv.org/abs/2401.14559)
• Keywords: large language model
• Abstract: Consistency is a key requirement of high-quality translation. It is especially important to adhere to pre-approved terminology and adapt to corrected translations in domain-specific projects. Machine translation (MT) has achieved significant progress in the area of domain adaptation. However, in-domain data scarcity is common in translation settings, due to the lack of specialised datasets and terminology, or inconsistency and inaccuracy of available in-domain translations. In such scenarios where there is insufficient in-domain data to fine-tune MT models, producing translations that are consistent with the relevant context is challenging. While real-time adaptation can make use of smaller amounts of in-domain data to improve the translation on the fly, it remains challenging due to supported context limitations and efficiency constraints. Large language models (LLMs) have recently shown interesting capabilities of in-context learning, where they learn to replicate certain input-output text generation patterns, without further fine-tuning. Such capabilities have opened new horizons for domain-specific data augmentation and real-time adaptive MT. This work attempts to address two main relevant questions: 1) in scenarios involving human interaction and continuous feedback, can we employ language models to improve the quality of adaptive MT at inference time? and 2) in the absence of sufficient in-domain data, can we use pre-trained large-scale language models to improve the process of MT domain adaptation?

Title: GOAt: Explaining Graph Neural Networks via Graph Output Attribution

• Authors: Shengyao Lu, Keith G. Mills, Jiao He, Bang Liu, Di Niu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.14578
• Pdf URL: https://arxiv.org/pdf/2401.14578
• Copy Paste: [[2401.14578]] GOAt: Explaining Graph Neural Networks via Graph Output Attribution(https://arxiv.org/abs/2401.14578)
• Keywords: interpretability
• Abstract: Understanding the decision-making process of Graph Neural Networks (GNNs) is crucial to their interpretability. Most existing methods for explaining GNNs typically rely on training auxiliary models, resulting in the explanations remain black-boxed. This paper introduces Graph Output Attribution (GOAt), a novel method to attribute graph outputs to input graph features, creating GNN explanations that are faithful, discriminative, as well as stable across similar samples. By expanding the GNN as a sum of scalar products involving node features, edge features and activation patterns, we propose an efficient analytical method to compute contribution of each node or edge feature to each scalar product and aggregate the contributions from all scalar products in the expansion form to derive the importance of each node and edge. Through extensive experiments on synthetic and real-world data, we show that our method not only outperforms various state-ofthe-art GNN explainers in terms of the commonly used fidelity metric, but also exhibits stronger discriminability, and stability by a remarkable margin.

Title: Design Your Own Universe: A Physics-Informed Agnostic Method for Enhancing Graph Neural Networks

• Authors: Dai Shi, Andi Han, Lequan Lin, Yi Guo, Zhiyong Wang, Junbin Gao
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.14580
• Pdf URL: https://arxiv.org/pdf/2401.14580
• Copy Paste: [[2401.14580]] Design Your Own Universe: A Physics-Informed Agnostic Method for Enhancing Graph Neural Networks(https://arxiv.org/abs/2401.14580)
• Keywords: robust
• Abstract: Physics-informed Graph Neural Networks have achieved remarkable performance in learning through graph-structured data by mitigating common GNN challenges such as over-smoothing, over-squashing, and heterophily adaption. Despite these advancements, the development of a simple yet effective paradigm that appropriately integrates previous methods for handling all these challenges is still underway. In this paper, we draw an analogy between the propagation of GNNs and particle systems in physics, proposing a model-agnostic enhancement framework. This framework enriches the graph structure by introducing additional nodes and rewiring connections with both positive and negative weights, guided by node labeling information. We theoretically verify that GNNs enhanced through our approach can effectively circumvent the over-smoothing issue and exhibit robustness against over-squashing. Moreover, we conduct a spectral analysis on the rewired graph to demonstrate that the corresponding GNNs can fit both homophilic and heterophilic graphs. Empirical validations on benchmarks for homophilic, heterophilic graphs, and long-term graph datasets show that GNNs enhanced by our method significantly outperform their original counterparts.

Title: Diffusion Stochastic Optimization for Min-Max Problems

• Authors: Haoyuan Cai, Sulaiman A. Alghunaim, Ali H. Sayed
• Subjects: cs.LG, math.OC
• Abstract URL: https://arxiv.org/abs/2401.14585
• Pdf URL: https://arxiv.org/pdf/2401.14585
• Copy Paste: [[2401.14585]] Diffusion Stochastic Optimization for Min-Max Problems(https://arxiv.org/abs/2401.14585)
• Keywords: diffusion, generative
• Abstract: The optimistic gradient method is useful in addressing minimax optimization problems. Motivated by the observation that the conventional stochastic version suffers from the need for a large batch size on the order of $\mathcal{O}(\varepsilon^{-2})$ to achieve an $\varepsilon$-stationary solution, we introduce and analyze a new formulation termed Diffusion Stochastic Same-Sample Optimistic Gradient (DSS-OG). We prove its convergence and resolve the large batch issue by establishing a tighter upper bound, under the more general setting of nonconvex Polyak-Lojasiewicz (PL) risk functions. We also extend the applicability of the proposed method to the distributed scenario, where agents communicate with their neighbors via a left-stochastic protocol. To implement DSS-OG, we can query the stochastic gradient oracles in parallel with some extra memory overhead, resulting in a complexity comparable to its conventional counterpart. To demonstrate the efficacy of the proposed algorithm, we conduct tests by training generative adversarial networks.

Title: Enhancing Diagnostic Accuracy through Multi-Agent Conversations: Using Large Language Models to Mitigate Cognitive Bias

• Authors: Yu He Ke, Rui Yang, Sui An Lie, Taylor Xin Yi Lim, Hairil Rizal Abdullah, Daniel Shu Wei Ting, Nan Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.14589
• Pdf URL: https://arxiv.org/pdf/2401.14589
• Copy Paste: [[2401.14589]] Enhancing Diagnostic Accuracy through Multi-Agent Conversations: Using Large Language Models to Mitigate Cognitive Bias(https://arxiv.org/abs/2401.14589)
• Keywords: large language model
• Abstract: Background: Cognitive biases in clinical decision-making significantly contribute to errors in diagnosis and suboptimal patient outcomes. Addressing these biases presents a formidable challenge in the medical field. This study explores the role of large language models (LLMs) in mitigating these biases through the utilization of a multi-agent framework. We simulate the clinical decision-making processes through multi-agent conversation and evaluate its efficacy in improving diagnostic accuracy. Methods: A total of 16 published and unpublished case reports where cognitive biases have resulted in misdiagnoses were identified from the literature. In the multi-agent system, we leveraged GPT-4 Turbo to facilitate interactions among four simulated agents to replicate clinical team dynamics. Each agent has a distinct role: 1) To make the initial and final diagnosis after considering the discussions, 2) The devil's advocate and correct confirmation and anchoring bias, 3) The tutor and facilitator of the discussion to reduce premature closure bias, and 4) To record and summarize the findings. A total of 80 simulations were evaluated for the accuracy of initial diagnosis, top differential diagnosis and final two differential diagnoses. Findings: In a total of 80 responses evaluating both initial and final diagnoses, the initial diagnosis had an accuracy of 0% (0/80), but following multi-agent discussions, the accuracy for the top differential diagnosis increased to 71.3% (57/80), and for the final two differential diagnoses, to 80.0% (64/80). The system demonstrated an ability to reevaluate and correct misconceptions, even in scenarios with misleading initial investigations. Interpretation: The LLM-driven multi-agent conversation system shows promise in enhancing diagnostic accuracy in diagnostically challenging medical scenarios.

Title: Query of CC: Unearthing Large Scale Domain-Specific Knowledge from Public Corpora

• Authors: Zhaoye Fei, Yunfan Shao, Linyang Li, Zhiyuan Zeng, Hang Yan, Xipeng Qiu, Dahua Lin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.14624
• Pdf URL: https://arxiv.org/pdf/2401.14624
• Copy Paste: [[2401.14624]] Query of CC: Unearthing Large Scale Domain-Specific Knowledge from Public Corpora(https://arxiv.org/abs/2401.14624)
• Keywords: large language model
• Abstract: Large language models have demonstrated remarkable potential in various tasks, however, there remains a significant scarcity of open-source models and data for specific domains. Previous works have primarily focused on manually specifying resources and collecting high-quality data on specific domains, which significantly consume time and effort. To address this limitation, we propose an efficient data collection method~\textit{Query of CC} based on large language models. This method bootstraps seed information through a large language model and retrieves related data from public corpora. It not only collects knowledge-related data for specific domains but unearths the data with potential reasoning procedures. Through the application of this method, we have curated a high-quality dataset called~\textsc{Knowledge Pile}, encompassing four major domains, including stem and humanities sciences, among others. Experimental results demonstrate that~\textsc{Knowledge Pile} significantly improves the performance of large language models in mathematical and knowledge-related reasoning ability tests. To facilitate academic sharing, we open-source our dataset and code, providing valuable support to the academic community.

Title: An Empirical Investigation of Domain Adaptation Ability for Chinese Spelling Check Models

• Authors: Xi Wang, Ruoqing Zhao, Hongliang Dai, Piji Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.14630
• Pdf URL: https://arxiv.org/pdf/2401.14630
• Copy Paste: [[2401.14630]] An Empirical Investigation of Domain Adaptation Ability for Chinese Spelling Check Models(https://arxiv.org/abs/2401.14630)
• Keywords: large language model
• Abstract: Chinese Spelling Check (CSC) is a meaningful task in the area of Natural Language Processing (NLP) which aims at detecting spelling errors in Chinese texts and then correcting these errors. However, CSC models are based on pretrained language models, which are trained on a general corpus. Consequently, their performance may drop when confronted with downstream tasks involving domain-specific terms. In this paper, we conduct a thorough evaluation about the domain adaption ability of various typical CSC models by building three new datasets encompassing rich domain-specific terms from the financial, medical, and legal domains. Then we conduct empirical investigations in the corresponding domain-specific test datasets to ascertain the cross-domain adaptation ability of several typical CSC models. We also test the performance of the popular large language model ChatGPT. As shown in our experiments, the performances of the CSC models drop significantly in the new domains.

Title: Signing in Four Public Software Package Registries: Quantity, Quality, and Influencing Factors

• Authors: Taylor R Schorlemmer, Kelechi G Kalu, Luke Chigges, Kyung Myung Ko, Eman Abdul-Muhd Abu Isghair, Saurabh Baghi, Santiago Torres-Arias, James C Davis
• Subjects: cs.CR, cs.SE
• Abstract URL: https://arxiv.org/abs/2401.14635
• Pdf URL: https://arxiv.org/pdf/2401.14635
• Copy Paste: [[2401.14635]] Signing in Four Public Software Package Registries: Quantity, Quality, and Influencing Factors(https://arxiv.org/abs/2401.14635)
• Keywords: attack
• Abstract: Many software applications incorporate open-source third-party packages distributed by third-party package registries. Guaranteeing authorship along this supply chain is a challenge. Package maintainers can guarantee package authorship through software signing. However, it is unclear how common this practice is, and whether the resulting signatures are created properly. Prior work has provided raw data on signing practices, but measured single platforms, did not consider time, and did not provide insight on factors that may influence signing. We lack a comprehensive, multi-platform understanding of signing adoption and relevant factors. This study addresses this gap. We provide measurements across three kinds of package registries: traditional software (Maven, PyPi), container images (DockerHub), and machine learning models (HuggingFace). For each registry, we describe the nature of the signed artifacts as well as the current quantity and quality of signatures. Then, we examine longitudinal trends in signing practices. Finally, we use a quasi-experiment to estimate the effect that various events had on software signing practices. To summarize our findings: (1) mandating signature adoption improves the quantity of signatures; (2) providing dedicated tooling improves the quality of signing; (3) getting started is the hard part -- once a maintainer begins to sign, they tend to continue doing so; and (4) although many supply chain attacks are mitigable via signing, signing adoption is primarily affected by registry policy rather than by public knowledge of attacks, new engineering standards, etc. These findings highlight the importance of software package registry managers and signing infrastructure.

Title: Benchmarking Large Language Models in Complex Question Answering Attribution using Knowledge Graphs

• Authors: Nan Hu, Jiaoyan Chen, Yike Wu, Guilin Qi, Sheng Bi, Tongtong Wu, Jeff Z. Pan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.14640
• Pdf URL: https://arxiv.org/pdf/2401.14640
• Copy Paste: [[2401.14640]] Benchmarking Large Language Models in Complex Question Answering Attribution using Knowledge Graphs(https://arxiv.org/abs/2401.14640)
• Keywords: large language model
• Abstract: The attribution of question answering is to provide citations for supporting generated statements, and has attracted wide research attention. The current methods for automatically evaluating the attribution, which are often based on Large Language Models (LLMs), are still inadequate, particularly in recognizing subtle differences between attributions, and complex relationships between citations and statements. To compare these attribution evaluation methods and develop new ones, we introduce a set of fine-grained categories (i.e., supportive, insufficient, contradictory and irrelevant) for measuring the attribution, and develop a Complex Attributed Question Answering (CAQA) benchmark by leveraging knowledge graphs (KGs) for automatically generating attributions of different categories to question-answer pairs. Our analysis reveals that existing evaluators perform poorly under fine-grained attribution settings and exhibit weaknesses in complex citation-statement reasoning. Our CAQA benchmark, validated with human annotations, emerges as a promising tool for selecting and developing LLM attribution evaluators.

Title: Super Efficient Neural Network for Compression Artifacts Reduction and Super Resolution

• Authors: Wen Ma, Qiuwen Lou, Arman Kazemi, Julian Faraone, Tariq Afzal
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2401.14641
• Pdf URL: https://arxiv.org/pdf/2401.14641
• Copy Paste: [[2401.14641]] Super Efficient Neural Network for Compression Artifacts Reduction and Super Resolution(https://arxiv.org/abs/2401.14641)
• Keywords: extraction
• Abstract: Video quality can suffer from limited internet speed while being streamed by users. Compression artifacts start to appear when the bitrate decreases to match the available bandwidth. Existing algorithms either focus on removing the compression artifacts at the same video resolution, or on upscaling the video resolution but not removing the artifacts. Super resolution-only approaches will amplify the artifacts along with the details by default. We propose a lightweight convolutional neural network (CNN)-based algorithm which simultaneously performs artifacts reduction and super resolution (ARSR) by enhancing the feature extraction layers and designing a custom training dataset. The output of this neural network is evaluated for test streams compressed at low bitrates using variable bitrate (VBR) encoding. The output video quality shows a 4-6 increase in video multi-method assessment fusion (VMAF) score compared to traditional interpolation upscaling approaches such as Lanczos or Bicubic.

Title: A Korean Legal Judgment Prediction Dataset for Insurance Disputes

• Authors: Alice Saebom Kwak, Cheonkam Jeong, Ji Weon Lim, Byeongcheol Min
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.14654
• Pdf URL: https://arxiv.org/pdf/2401.14654
• Copy Paste: [[2401.14654]] A Korean Legal Judgment Prediction Dataset for Insurance Disputes(https://arxiv.org/abs/2401.14654)
• Keywords: transformer
• Abstract: This paper introduces a Korean legal judgment prediction (LJP) dataset for insurance disputes. Successful LJP models on insurance disputes can benefit insurance companies and their customers. It can save both sides' time and money by allowing them to predict how the result would come out if they proceed to the dispute mediation process. As is often the case with low-resource languages, there is a limitation on the amount of data available for this specific task. To mitigate this issue, we investigate how one can achieve a good performance despite the limitation in data. In our experiment, we demonstrate that Sentence Transformer Fine-tuning (SetFit, Tunstall et al., 2022) is a good alternative to standard fine-tuning when training data are limited. The models fine-tuned with the SetFit approach on our data show similar performance to the Korean LJP benchmark models (Hwang et al., 2022) despite the much smaller data size.

Title: Scientific Large Language Models: A Survey on Biological & Chemical Domains

• Authors: Qiang Zhang, Keyang Ding, Tianwen Lyv, Xinda Wang, Qingyu Yin, Yiwen Zhang, Jing Yu, Yuhao Wang, Xiaotong Li, Zhuoyi Xiang, Xiang Zhuang, Zeyuan Wang, Ming Qin, Mengyao Zhang, Jinlu Zhang, Jiyu Cui, Renjun Xu, Hongyang Chen, Xiaohui Fan, Huabin Xing, Huajun Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.14656
• Pdf URL: https://arxiv.org/pdf/2401.14656
• Copy Paste: [[2401.14656]] Scientific Large Language Models: A Survey on Biological & Chemical Domains(https://arxiv.org/abs/2401.14656)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have emerged as a transformative power in enhancing natural language comprehension, representing a significant stride toward artificial general intelligence. The application of LLMs extends beyond conventional linguistic boundaries, encompassing specialized linguistic systems developed within various scientific disciplines. This growing interest has led to the advent of scientific LLMs, a novel subclass specifically engineered for facilitating scientific discovery. As a burgeoning area in the community of AI for Science, scientific LLMs warrant comprehensive exploration. However, a systematic and up-to-date survey introducing them is currently lacking. In this paper, we endeavor to methodically delineate the concept of "scientific language", whilst providing a thorough review of the latest advancements in scientific LLMs. Given the expansive realm of scientific disciplines, our analysis adopts a focused lens, concentrating on the biological and chemical domains. This includes an in-depth examination of LLMs for textual knowledge, small molecules, macromolecular proteins, genomic sequences, and their combinations, analyzing them in terms of model architectures, capabilities, datasets, and evaluation. Finally, we critically examine the prevailing challenges and point out promising research directions along with the advances of LLMs. By offering a comprehensive overview of technical developments in this field, this survey aspires to be an invaluable resource for researchers navigating the intricate landscape of scientific LLMs.

Title: From Blurry to Brilliant Detection: YOLOv5-Based Aerial Object Detection with Super Resolution

• Authors: Ragib Amin Nihal, Benjamin Yen, Katsutoshi Itoyama, Kazuhiro Nakadai
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.14661
• Pdf URL: https://arxiv.org/pdf/2401.14661
• Copy Paste: [[2401.14661]] From Blurry to Brilliant Detection: YOLOv5-Based Aerial Object Detection with Super Resolution(https://arxiv.org/abs/2401.14661)
• Keywords: transformer
• Abstract: The demand for accurate object detection in aerial imagery has surged with the widespread use of drones and satellite technology. Traditional object detection models, trained on datasets biased towards large objects, struggle to perform optimally in aerial scenarios where small, densely clustered objects are prevalent. To address this challenge, we present an innovative approach that combines super-resolution and an adapted lightweight YOLOv5 architecture. We employ a range of datasets, including VisDrone-2023, SeaDroneSee, VEDAI, and NWPU VHR-10, to evaluate our model's performance. Our Super Resolved YOLOv5 architecture features Transformer encoder blocks, allowing the model to capture global context and context information, leading to improved detection results, especially in high-density, occluded conditions. This lightweight model not only delivers improved accuracy but also ensures efficient resource utilization, making it well-suited for real-time applications. Our experimental results demonstrate the model's superior performance in detecting small and densely clustered objects, underlining the significance of dataset choice and architectural adaptation for this specific task. In particular, the method achieves 52.5% mAP on VisDrone, exceeding top prior works. This approach promises to significantly advance object detection in aerial imagery, contributing to more accurate and reliable results in a variety of real-world applications.

Title: Multi-model learning by sequential reading of untrimmed videos for action recognition

• Authors: Kodai Kamiya, Toru Tamaki
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.14675
• Pdf URL: https://arxiv.org/pdf/2401.14675
• Copy Paste: [[2401.14675]] Multi-model learning by sequential reading of untrimmed videos for action recognition(https://arxiv.org/abs/2401.14675)
• Keywords: federate
• Abstract: We propose a new method for learning videos by aggregating multiple models by sequentially extracting video clips from untrimmed video. The proposed method reduces the correlation between clips by feeding clips to multiple models in turn and synchronizes these models through federated learning. Experimental results show that the proposed method improves the performance compared to the no synchronization.

Title: MaLLaM -- Malaysia Large Language Model

• Authors: Husein Zolkepli, Aisyah Razak, Kamarul Adha, Ariff Nazhan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.14680
• Pdf URL: https://arxiv.org/pdf/2401.14680
• Copy Paste: [[2401.14680]] MaLLaM -- Malaysia Large Language Model(https://arxiv.org/abs/2401.14680)
• Keywords: large language model
• Abstract: Addressing the gap in Large Language Model pretrained from scratch with Malaysian context, We trained models with 1.1 billion, 3 billion, and 5 billion parameters on a substantial 349GB dataset, equivalent to 90 billion tokens based on our pretrained Byte Pair Encoding (BPE) tokenizer for a single epoch. MaLLaM contributes to enhanced natural language understanding and generation tasks in the Malay language. Although trained on a smaller dataset of 90 billion tokens, our instruction-tuned MaLLaM models perform competitively. When compared to ChatGPT3.5 and Malaysian Mistral, MaLLaM's instruction-tuned models demonstrate notable proficiency, underscoring the effectiveness of our approach in capturing and understanding the nuances of the Malaysian language. MaLLaM models mark a significant contribution to the field, providing comprehensive language representations grounded in Malaysian context. This endeavor aims to pave the way for enhanced natural language understanding and generation tasks specific to the linguistic nuances present in Malaysia. We discuss the training methodology, dataset composition, and the potential impact of MaLLaM in advancing the capabilities of large language models within the context of the Malay language. All models released at https://huggingface.co/collections/mesolitica/mallam-6577b59d1e0b436ae75f930f

Title: MasonTigers@LT-EDI-2024: An Ensemble Approach towards Detecting Homophobia and Transphobia in Social Media Comments

• Authors: Dhiman Goswami, Sadiya Sayara Chowdhury Puspo, Md Nishat Raihan, Al Nahian Bin Emran
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.14681
• Pdf URL: https://arxiv.org/pdf/2401.14681
• Copy Paste: [[2401.14681]] MasonTigers@LT-EDI-2024: An Ensemble Approach towards Detecting Homophobia and Transphobia in Social Media Comments(https://arxiv.org/abs/2401.14681)
• Keywords: transformer
• Abstract: In this paper, we describe our approaches and results for Task 2 of the LT-EDI 2024 Workshop, aimed at detecting homophobia and/or transphobia across ten languages. Our methodologies include monolingual transformers and ensemble methods, capitalizing on the strengths of each to enhance the performance of the models. The ensemble models worked well, placing our team, MasonTigers, in the top five for eight of the ten languages, as measured by the macro F1 score. Our work emphasizes the efficacy of ensemble methods in multilingual scenarios, addressing the complexities of language-specific tasks.

Title: SSR: SAM is a Strong Regularizer for domain adaptive semantic segmentation

• Authors: Yanqi Ge, Ye Huang, Wen Li, Lixin Duan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.14686
• Pdf URL: https://arxiv.org/pdf/2401.14686
• Copy Paste: [[2401.14686]] SSR: SAM is a Strong Regularizer for domain adaptive semantic segmentation(https://arxiv.org/abs/2401.14686)
• Keywords: robust, segmentation
• Abstract: We introduced SSR, which utilizes SAM (segment-anything) as a strong regularizer during training, to greatly enhance the robustness of the image encoder for handling various domains. Specifically, given the fact that SAM is pre-trained with a large number of images over the internet, which cover a diverse variety of domains, the feature encoding extracted by the SAM is obviously less dependent on specific domains when compared to the traditional ImageNet pre-trained image encoder. Meanwhile, the ImageNet pre-trained image encoder is still a mature choice of backbone for the semantic segmentation task, especially when the SAM is category-irrelevant. As a result, our SSR provides a simple yet highly effective design. It uses the ImageNet pre-trained image encoder as the backbone, and the intermediate feature of each stage (ie there are 4 stages in MiT-B5) is regularized by SAM during training. After extensive experimentation on GTA5$\rightarrow$Cityscapes, our SSR significantly improved performance over the baseline without introducing any extra inference overhead.

Title: Taiyi-Diffusion-XL: Advancing Bilingual Text-to-Image Generation with Large Vision-Language Model Support

• Authors: Xiaojun Wu, Dixiang Zhang, Ruyi Gan, Junyu Lu, Ziwei Wu, Renliang Sun, Jiaxing Zhang, Pingjian Zhang, Yan Song
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.14688
• Pdf URL: https://arxiv.org/pdf/2401.14688
• Copy Paste: [[2401.14688]] Taiyi-Diffusion-XL: Advancing Bilingual Text-to-Image Generation with Large Vision-Language Model Support(https://arxiv.org/abs/2401.14688)
• Keywords: diffusion
• Abstract: Recent advancements in text-to-image models have significantly enhanced image generation capabilities, yet a notable gap of open-source models persists in bilingual or Chinese language support. To address this need, we present Taiyi-Diffusion-XL, a new Chinese and English bilingual text-to-image model which is developed by extending the capabilities of CLIP and Stable-Diffusion-XL through a process of bilingual continuous pre-training. This approach includes the efficient expansion of vocabulary by integrating the most frequently used Chinese characters into CLIP's tokenizer and embedding layers, coupled with an absolute position encoding expansion. Additionally, we enrich text prompts by large vision-language model, leading to better images captions and possess higher visual quality. These enhancements are subsequently applied to downstream text-to-image models. Our empirical results indicate that the developed CLIP model excels in bilingual image-text retrieval.Furthermore, the bilingual image generation capabilities of Taiyi-Diffusion-XL surpass previous models. This research leads to the development and open-sourcing of the Taiyi-Diffusion-XL model, representing a notable advancement in the field of image generation, particularly for Chinese language applications. This contribution is a step forward in addressing the need for more diverse language support in multimodal research. The model and demonstration are made publicly available at \href{https://huggingface.co/IDEA-CCNL/Taiyi-Stable-Diffusion-XL-3.5B/}{this https URL}, fostering further research and collaboration in this domain.

Title: TA-RNN: an Attention-based Time-aware Recurrent Neural Network Architecture for Electronic Health Records

• Authors: Mohammad Al Olaimat (1, 3), Serdar Bozdag (1, 2 and 3), the Alzheimer's Disease Neuroimaging Initiative ((1) Dept. of Computer Science and Engineering, University of North Texas, Denton, USA, (2) Dept. of Mathematics, University of North Texas, Denton, USA, (3) BioDiscovery Institute, University of North Texas, Denton, USA)
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.14694
• Pdf URL: https://arxiv.org/pdf/2401.14694
• Copy Paste: [[2401.14694]] TA-RNN: an Attention-based Time-aware Recurrent Neural Network Architecture for Electronic Health Records(https://arxiv.org/abs/2401.14694)
• Keywords: interpretability
• Abstract: Motivation: Electronic Health Records (EHR) represent a comprehensive resource of a patient's medical history. EHR are essential for utilizing advanced technologies such as deep learning (DL), enabling healthcare providers to analyze extensive data, extract valuable insights, and make precise and data-driven clinical decisions. DL methods such as Recurrent Neural Networks (RNN) have been utilized to analyze EHR to model disease progression and predict diagnosis. However, these methods do not address some inherent irregularities in EHR data such as irregular time intervals between clinical visits. Furthermore, most DL models are not interpretable. In this study, we propose two interpretable DL architectures based on RNN, namely Time-Aware RNN (TA-RNN) and TA-RNN-Autoencoder (TA-RNN-AE) to predict patient's clinical outcome in EHR at next visit and multiple visits ahead, respectively. To mitigate the impact of irregular time intervals, we propose incorporating time embedding of the elapsed times between visits. For interpretability, we propose employing a dual-level attention mechanism that operates between visits and features within each visit. Results: The results of the experiments conducted on Alzheimer's Disease Neuroimaging Initiative (ADNI) and National Alzheimer's Coordinating Center (NACC) datasets indicated superior performance of proposed models for predicting Alzheimer's Disease (AD) compared to state-of-the-art and baseline approaches based on F2 and sensitivity. Additionally, TA-RNN showed superior performance on Medical Information Mart for Intensive Care (MIMIC-III) dataset for mortality prediction. In our ablation study, we observed enhanced predictive performance by incorporating time embedding and attention mechanisms. Finally, investigating attention weights helped identify influential visits and features in predictions. Availability: https://github.com/bozdaglab/TA-RNN

Title: Under the Surface: Tracking the Artifactuality of LLM-Generated Data

• Authors: Debarati Das, Karin De Langis, Anna Martin, Jaehyung Kim, Minhwa Lee, Zae Myung Kim, Shirley Hayati, Risako Owan, Bin Hu, Ritik Parkar, Ryan Koo, Jonginn Park, Aahan Tyagi, Libby Ferland, Sanjali Roy, Vincent Liu, Dongyeop Kang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.14698
• Pdf URL: https://arxiv.org/pdf/2401.14698
• Copy Paste: [[2401.14698]] Under the Surface: Tracking the Artifactuality of LLM-Generated Data(https://arxiv.org/abs/2401.14698)
• Keywords: large language model
• Abstract: This work delves into the expanding role of large language models (LLMs) in generating artificial data. LLMs are increasingly employed to create a variety of outputs, including annotations, preferences, instruction prompts, simulated dialogues, and free text. As these forms of LLM-generated data often intersect in their application, they exert mutual influence on each other and raise significant concerns about the quality and diversity of the artificial data incorporated into training cycles, leading to an artificial data ecosystem. To the best of our knowledge, this is the first study to aggregate various types of LLM-generated text data, from more tightly constrained data like "task labels" to more lightly constrained "free-form text". We then stress test the quality and implications of LLM-generated artificial data, comparing it with human data across various existing benchmarks. Despite artificial data's capability to match human performance, this paper reveals significant hidden disparities, especially in complex tasks where LLMs often miss the nuanced understanding of intrinsic human-generated content. This study critically examines diverse LLM-generated data and emphasizes the need for ethical practices in data creation and when using LLMs. It highlights the LLMs' shortcomings in replicating human traits and behaviors, underscoring the importance of addressing biases and artifacts produced in LLM-generated content for future research and development. All data and code are available on our project page.

Title: FairSample: Training Fair and Accurate Graph Convolutional Neural Networks Efficiently

• Authors: Zicun Cong, Shi Baoxu, Shan Li, Jaewon Yang, Qi He, Jian Pei
• Subjects: cs.LG, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2401.14702
• Pdf URL: https://arxiv.org/pdf/2401.14702
• Copy Paste: [[2401.14702]] FairSample: Training Fair and Accurate Graph Convolutional Neural Networks Efficiently(https://arxiv.org/abs/2401.14702)
• Keywords: fair
• Abstract: Fairness in Graph Convolutional Neural Networks (GCNs) becomes a more and more important concern as GCNs are adopted in many crucial applications. Societal biases against sensitive groups may exist in many real world graphs. GCNs trained on those graphs may be vulnerable to being affected by such biases. In this paper, we adopt the well-known fairness notion of demographic parity and tackle the challenge of training fair and accurate GCNs efficiently. We present an in-depth analysis on how graph structure bias, node attribute bias, and model parameters may affect the demographic parity of GCNs. Our insights lead to FairSample, a framework that jointly mitigates the three types of biases. We employ two intuitive strategies to rectify graph structures. First, we inject edges across nodes that are in different sensitive groups but similar in node features. Second, to enhance model fairness and retain model quality, we develop a learnable neighbor sampling policy using reinforcement learning. To address the bias in node features and model parameters, FairSample is complemented by a regularization objective to optimize fairness.

Title: Mitigating Feature Gap for Adversarial Robustness by Feature Disentanglement

• Authors: Nuoyan Zhou, Dawei Zhou, Decheng Liu, Xinbo Gao, Nannan Wang
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.14707
• Pdf URL: https://arxiv.org/pdf/2401.14707
• Copy Paste: [[2401.14707]] Mitigating Feature Gap for Adversarial Robustness by Feature Disentanglement(https://arxiv.org/abs/2401.14707)
• Keywords: robust
• Abstract: Deep neural networks are vulnerable to adversarial samples. Adversarial fine-tuning methods aim to enhance adversarial robustness through fine-tuning the naturally pre-trained model in an adversarial training manner. However, we identify that some latent features of adversarial samples are confused by adversarial perturbation and lead to an unexpectedly increasing gap between features in the last hidden layer of natural and adversarial samples. To address this issue, we propose a disentanglement-based approach to explicitly model and further remove the latent features that cause the feature gap. Specifically, we introduce a feature disentangler to separate out the latent features from the features of the adversarial samples, thereby boosting robustness by eliminating the latent features. Besides, we align features in the pre-trained model with features of adversarial samples in the fine-tuned model, to further benefit from the features from natural samples without confusion. Empirical evaluations on three benchmark datasets demonstrate that our approach surpasses existing adversarial fine-tuning methods and adversarial training baselines.

Title: Turn-taking and Backchannel Prediction with Acoustic and Large Language Model Fusion

• Authors: Jinhan Wang, Long Chen, Aparna Khare, Anirudh Raju, Pranav Dheram, Di He, Minhua Wu, Andreas Stolcke, Venkatesh Ravichandran
• Subjects: cs.CL, cs.AI, cs.LG, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2401.14717
• Pdf URL: https://arxiv.org/pdf/2401.14717
• Copy Paste: [[2401.14717]] Turn-taking and Backchannel Prediction with Acoustic and Large Language Model Fusion(https://arxiv.org/abs/2401.14717)
• Keywords: large language model
• Abstract: We propose an approach for continuous prediction of turn-taking and backchanneling locations in spoken dialogue by fusing a neural acoustic model with a large language model (LLM). Experiments on the Switchboard human-human conversation dataset demonstrate that our approach consistently outperforms the baseline models with single modality. We also develop a novel multi-task instruction fine-tuning strategy to further benefit from LLM-encoded knowledge for understanding the tasks and conversational contexts, leading to additional improvements. Our approach demonstrates the potential of combined LLMs and acoustic models for a more natural and conversational interaction between humans and speech-enabled AI agents.

Title: A Survey on Video Prediction: From Deterministic to Generative Approaches

• Authors: Ruibo Ming, Zhewei Huang, Zhuoxuan Ju, Jianming Hu, Lihui Peng, Shuchang Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.14718
• Pdf URL: https://arxiv.org/pdf/2401.14718
• Copy Paste: [[2401.14718]] A Survey on Video Prediction: From Deterministic to Generative Approaches(https://arxiv.org/abs/2401.14718)
• Keywords: generative
• Abstract: Video prediction, a fundamental task in computer vision, aims to enable models to generate sequences of future frames based on existing video content. This task has garnered widespread application across various domains. In this paper, we comprehensively survey both historical and contemporary works in this field, encompassing the most widely used datasets and algorithms. Our survey scrutinizes the challenges and evolving landscape of video prediction within the realm of computer vision. We propose a novel taxonomy centered on the stochastic nature of video prediction algorithms. This taxonomy accentuates the gradual transition from deterministic to generative prediction methodologies, underlining significant advancements and shifts in approach.

Title: VJT: A Video Transformer on Joint Tasks of Deblurring, Low-light Enhancement and Denoising

• Authors: Yuxiang Hui, Yang Liu, Yaofang Liu, Fan Jia, Jinshan Pan, Raymond Chan, Tieyong Zeng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.14754
• Pdf URL: https://arxiv.org/pdf/2401.14754
• Copy Paste: [[2401.14754]] VJT: A Video Transformer on Joint Tasks of Deblurring, Low-light Enhancement and Denoising(https://arxiv.org/abs/2401.14754)
• Keywords: transformer
• Abstract: Video restoration task aims to recover high-quality videos from low-quality observations. This contains various important sub-tasks, such as video denoising, deblurring and low-light enhancement, since video often faces different types of degradation, such as blur, low light, and noise. Even worse, these kinds of degradation could happen simultaneously when taking videos in extreme environments. This poses significant challenges if one wants to remove these artifacts at the same time. In this paper, to the best of our knowledge, we are the first to propose an efficient end-to-end video transformer approach for the joint task of video deblurring, low-light enhancement, and denoising. This work builds a novel multi-tier transformer where each tier uses a different level of degraded video as a target to learn the features of video effectively. Moreover, we carefully design a new tier-to-tier feature fusion scheme to learn video features incrementally and accelerate the training process with a suitable adaptive weighting scheme. We also provide a new Multiscene-Lowlight-Blur-Noise (MLBN) dataset, which is generated according to the characteristics of the joint task based on the RealBlur dataset and YouTube videos to simulate realistic scenes as far as possible. We have conducted extensive experiments, compared with many previous state-of-the-art methods, to show the effectiveness of our approach clearly.

Title: Spatial Transcriptomics Analysis of Zero-shot Gene Expression Prediction

• Authors: Yan Yang, Md Zakir Hossain, Xuesong Li, Shafin Rahman, Eric Stone
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.14772
• Pdf URL: https://arxiv.org/pdf/2401.14772
• Copy Paste: [[2401.14772]] Spatial Transcriptomics Analysis of Zero-shot Gene Expression Prediction(https://arxiv.org/abs/2401.14772)
• Keywords: large language model
• Abstract: Spatial transcriptomics (ST) captures gene expression within distinct regions (i.e., windows) of a tissue slide. Traditional supervised learning frameworks applied to model ST are constrained to predicting expression from slide image windows for gene types seen during training, failing to generalize to unseen gene types. To overcome this limitation, we propose a semantic guided network (SGN), a pioneering zero-shot framework for predicting gene expression from slide image windows. Considering a gene type can be described by functionality and phenotype, we dynamically embed a gene type to a vector per its functionality and phenotype, and employ this vector to project slide image windows to gene expression in feature space, unleashing zero-shot expression prediction for unseen gene types. The gene type functionality and phenotype are queried with a carefully designed prompt from a pre-trained large language model (LLM). On standard benchmark datasets, we demonstrate competitive zero-shot performance compared to past state-of-the-art supervised learning approaches.

Title: Large Language Model Adaptation for Financial Sentiment Analysis

• Authors: Pau Rodriguez Inserte, Mariam Nakhlé, Raheel Qader, Gaetan Caillaut, Jingshu Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.14777
• Pdf URL: https://arxiv.org/pdf/2401.14777
• Copy Paste: [[2401.14777]] Large Language Model Adaptation for Financial Sentiment Analysis(https://arxiv.org/abs/2401.14777)
• Keywords: generative, large language model
• Abstract: Natural language processing (NLP) has recently gained relevance within financial institutions by providing highly valuable insights into companies and markets' financial documents. However, the landscape of the financial domain presents extra challenges for NLP, due to the complexity of the texts and the use of specific terminology. Generalist language models tend to fall short in tasks specifically tailored for finance, even when using large language models (LLMs) with great natural language understanding and generative capabilities. This paper presents a study on LLM adaptation methods targeted at the financial domain and with high emphasis on financial sentiment analysis. To this purpose, two foundation models with less than 1.5B parameters have been adapted using a wide range of strategies. We show that through careful fine-tuning on both financial documents and instructions, these foundation models can be adapted to the target domain. Moreover, we observe that small LLMs have comparable performance to larger scale models, while being more efficient in terms of parameters and data. In addition to the models, we show how to generate artificial instructions through LLMs to augment the number of samples of the instruction dataset.

Title: Deep Variational Privacy Funnel: General Modeling with Applications in Face Recognition

• Authors: Behrooz Razeghi, Parsa Rahimi, Sébastien Marcel
• Subjects: cs.CV, cs.IT, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.14792
• Pdf URL: https://arxiv.org/pdf/2401.14792
• Copy Paste: [[2401.14792]] Deep Variational Privacy Funnel: General Modeling with Applications in Face Recognition(https://arxiv.org/abs/2401.14792)
• Keywords: privacy, protect, generative
• Abstract: In this study, we harness the information-theoretic Privacy Funnel (PF) model to develop a method for privacy-preserving representation learning using an end-to-end training framework. We rigorously address the trade-off between obfuscation and utility. Both are quantified through the logarithmic loss, a measure also recognized as self-information loss. This exploration deepens the interplay between information-theoretic privacy and representation learning, offering substantive insights into data protection mechanisms for both discriminative and generative models. Importantly, we apply our model to state-of-the-art face recognition systems. The model demonstrates adaptability across diverse inputs, from raw facial images to both derived or refined embeddings, and is competent in tasks such as classification, reconstruction, and generation.

Title: PL-FSCIL: Harnessing the Power of Prompts for Few-Shot Class-Incremental Learning

• Authors: Songsong Tian, Lusi Li, Weijun Li, Hang Ran, Li Li, Xin Ning
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.14807
• Pdf URL: https://arxiv.org/pdf/2401.14807
• Copy Paste: [[2401.14807]] PL-FSCIL: Harnessing the Power of Prompts for Few-Shot Class-Incremental Learning(https://arxiv.org/abs/2401.14807)
• Keywords: transformer
• Abstract: Few-Shot Class-Incremental Learning (FSCIL) aims to enable deep neural networks to learn new tasks incrementally from a small number of labeled samples without forgetting previously learned tasks, closely mimicking human learning patterns. In this paper, we propose a novel approach called Prompt Learning for FSCIL (PL-FSCIL), which harnesses the power of prompts in conjunction with a pre-trained Vision Transformer (ViT) model to address the challenges of FSCIL effectively. Our work pioneers the use of visual prompts in FSCIL, which is characterized by its notable simplicity. PL-FSCIL consists of two distinct prompts: the Domain Prompt and the FSCIL Prompt. Both are vectors that augment the model by embedding themselves into the attention layer of the ViT model. Specifically, the Domain Prompt assists the ViT model in adapting to new data domains. The task-specific FSCIL Prompt, coupled with a prototype classifier, amplifies the model's ability to effectively handle FSCIL tasks. We validate the efficacy of PL-FSCIL on widely used benchmark datasets such as CIFAR-100 and CUB-200. The results showcase competitive performance, underscoring its promising potential for real-world applications where high-quality data is often scarce. The source code is available at: https://github.com/TianSongS/PL-FSCIL.

Title: ChemDFM: Dialogue Foundation Model for Chemistry

• Authors: Zihan Zhao, Da Ma, Lu Chen, Liangtai Sun, Zihao Li, Hongshen Xu, Zichen Zhu, Su Zhu, Shuai Fan, Guodong Shen, Xin Chen, Kai Yu
• Subjects: cs.CL, cs.DL
• Abstract URL: https://arxiv.org/abs/2401.14818
• Pdf URL: https://arxiv.org/pdf/2401.14818
• Copy Paste: [[2401.14818]] ChemDFM: Dialogue Foundation Model for Chemistry(https://arxiv.org/abs/2401.14818)
• Keywords: large language model
• Abstract: Large language models (LLMs) have established great success in the general domain of natural language processing. Their emerging task generalization and free-form dialogue capabilities can greatly help to design Chemical General Intelligence (CGI) to assist real-world research in chemistry. However, the existence of specialized language and knowledge in the field of chemistry, such as the highly informative SMILES notation, hinders the performance of general-domain LLMs in chemistry. To this end, we develop ChemDFM, the first LLM towards CGI. ChemDFM-13B is trained on 34B tokens from chemical literature, textbooks, and instructions as well as various data from the general domain. Therefore, it can store, understand, and reason over chemical knowledge and languages while still possessing advanced free-form language comprehension capabilities. Extensive quantitative evaluation shows that ChemDFM can significantly outperform the representative open-sourced LLMs. Moreover, ChemDFM can also surpass GPT-4 on a great portion of chemical tasks, despite the significant size difference. Further qualitative evaluations demonstrate the efficiency and effectiveness of ChemDFM in real-world research scenarios. We will open-source the ChemDFM model soon.

Title: Text Image Inpainting via Global Structure-Guided Diffusion Models

• Authors: Shipeng Zhu, Pengfei Fang, Chenjie Zhu, Zuoyan Zhao, Qiang Xu, Hui Xue
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.14832
• Pdf URL: https://arxiv.org/pdf/2401.14832
• Copy Paste: [[2401.14832]] Text Image Inpainting via Global Structure-Guided Diffusion Models(https://arxiv.org/abs/2401.14832)
• Keywords: diffusion
• Abstract: Real-world text can be damaged by corrosion issues caused by environmental or human factors, which hinder the preservation of the complete styles of texts, e.g., texture and structure. These corrosion issues, such as graffiti signs and incomplete signatures, bring difficulties in understanding the texts, thereby posing significant challenges to downstream applications, e.g., scene text recognition and signature identification. Notably, current inpainting techniques often fail to adequately address this problem and have difficulties restoring accurate text images along with reasonable and consistent styles. Formulating this as an open problem of text image inpainting, this paper aims to build a benchmark to facilitate its study. In doing so, we establish two specific text inpainting datasets which contain scene text images and handwritten text images, respectively. Each of them includes images revamped by real-life and synthetic datasets, featuring pairs of original images, corrupted images, and other assistant information. On top of the datasets, we further develop a novel neural framework, Global Structure-guided Diffusion Model (GSDM), as a potential solution. Leveraging the global structure of the text as a prior, the proposed GSDM develops an efficient diffusion model to recover clean texts. The efficacy of our approach is demonstrated by thorough empirical study, including a substantial boost in both recognition accuracy and image quality. These findings not only highlight the effectiveness of our method but also underscore its potential to enhance the broader field of text image understanding and processing. Code and datasets are available at: https://github.com/blackprotoss/GSDM.

Title: GuardML: Efficient Privacy-Preserving Machine Learning Services Through Hybrid Homomorphic Encryption

• Authors: Eugene Frimpong, Khoa Nguyen, Mindaugas Budzys, Tanveer Khan, Antonis Michalas
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2401.14840
• Pdf URL: https://arxiv.org/pdf/2401.14840
• Copy Paste: [[2401.14840]] GuardML: Efficient Privacy-Preserving Machine Learning Services Through Hybrid Homomorphic Encryption(https://arxiv.org/abs/2401.14840)
• Keywords: secure, security, privacy, attack
• Abstract: Machine Learning (ML) has emerged as one of data science's most transformative and influential domains. However, the widespread adoption of ML introduces privacy-related concerns owing to the increasing number of malicious attacks targeting ML models. To address these concerns, Privacy-Preserving Machine Learning (PPML) methods have been introduced to safeguard the privacy and security of ML models. One such approach is the use of Homomorphic Encryption (HE). However, the significant drawbacks and inefficiencies of traditional HE render it impractical for highly scalable scenarios. Fortunately, a modern cryptographic scheme, Hybrid Homomorphic Encryption (HHE), has recently emerged, combining the strengths of symmetric cryptography and HE to surmount these challenges. Our work seeks to introduce HHE to ML by designing a PPML scheme tailored for end devices. We leverage HHE as the fundamental building block to enable secure learning of classification outcomes over encrypted data, all while preserving the privacy of the input data and ML model. We demonstrate the real-world applicability of our construction by developing and evaluating an HHE-based PPML application for classifying heart disease based on sensitive ECG data. Notably, our evaluations revealed a slight reduction in accuracy compared to inference on plaintext data. Additionally, both the analyst and end devices experience minimal communication and computation costs, underscoring the practical viability of our approach. The successful integration of HHE into PPML provides a glimpse into a more secure and privacy-conscious future for machine learning on relatively constrained end devices.

Title: Adaptive Point Transformer

• Authors: Alessandro Baiocchi, Indro Spinelli, Alessandro Nicolosi, Simone Scardapane
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.14845
• Pdf URL: https://arxiv.org/pdf/2401.14845
• Copy Paste: [[2401.14845]] Adaptive Point Transformer(https://arxiv.org/abs/2401.14845)
• Keywords: transformer
• Abstract: The recent surge in 3D data acquisition has spurred the development of geometric deep learning models for point cloud processing, boosted by the remarkable success of transformers in natural language processing. While point cloud transformers (PTs) have achieved impressive results recently, their quadratic scaling with respect to the point cloud size poses a significant scalability challenge for real-world applications. To address this issue, we propose the Adaptive Point Cloud Transformer (AdaPT), a standard PT model augmented by an adaptive token selection mechanism. AdaPT dynamically reduces the number of tokens during inference, enabling efficient processing of large point clouds. Furthermore, we introduce a budget mechanism to flexibly adjust the computational cost of the model at inference time without the need for retraining or fine-tuning separate models. Our extensive experimental evaluation on point cloud classification tasks demonstrates that AdaPT significantly reduces computational complexity while maintaining competitive accuracy compared to standard PTs. The code for AdaPT is made publicly available.

Title: Understanding Domain Generalization: A Noise Robustness Perspective

• Authors: Rui Qiao, Bryan Kian Hsiang Low
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2401.14846
• Pdf URL: https://arxiv.org/pdf/2401.14846
• Copy Paste: [[2401.14846]] Understanding Domain Generalization: A Noise Robustness Perspective(https://arxiv.org/abs/2401.14846)
• Keywords: robust
• Abstract: Despite the rapid development of machine learning algorithms for domain generalization (DG), there is no clear empirical evidence that the existing DG algorithms outperform the classic empirical risk minimization (ERM) across standard benchmarks. To better understand this phenomenon, we investigate whether there are benefits of DG algorithms over ERM through the lens of label noise. Specifically, our finite-sample analysis reveals that label noise exacerbates the effect of spurious correlations for ERM, undermining generalization. Conversely, we illustrate that DG algorithms exhibit implicit label-noise robustness during finite-sample training even when spurious correlation is present. Such desirable property helps mitigate spurious correlations and improve generalization in synthetic experiments. However, additional comprehensive experiments on real-world benchmark datasets indicate that label-noise robustness does not necessarily translate to better performance compared to ERM. We conjecture that the failure mode of ERM arising from spurious correlations may be less pronounced in practice.

Title: F-Eval: Asssessing Fundamental Abilities with Refined Evaluation Methods

• Authors: Yu Sun, Keyu Chen, Shujie Wang, Qipeng Guo, Hang Yan, Xipeng Qiu, Xuanjing Huang, Dahua Lin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.14869
• Pdf URL: https://arxiv.org/pdf/2401.14869
• Copy Paste: [[2401.14869]] F-Eval: Asssessing Fundamental Abilities with Refined Evaluation Methods(https://arxiv.org/abs/2401.14869)
• Keywords: large language model
• Abstract: Large language models (LLMs) garner significant attention for their unprecedented performance, leading to an increasing number of researches evaluating LLMs. However, these evaluation benchmarks are limited to assessing the instruction-following capabilities, overlooking the fundamental abilities that emerge during the pre-training stage. Previous subjective evaluation methods mainly reply on scoring by API models. However, in the absence of references, large models have shown limited ability to discern subtle differences. To bridge the gap, we propose F-Eval, a bilingual evaluation benchmark to evaluate the fundamental abilities, including expression, commonsense and logic. The tasks in F-Eval include multi-choice objective tasks, open-ended objective tasks, reference-based subjective tasks and reference-free subjective tasks. For reference-free subjective tasks, we devise new evaluation methods, serving as alternatives to scoring by API models. We conduct evaluations on 13 advanced LLMs. Results show that our evaluation methods show higher correlation coefficients and larger distinction than other evaluators. Additionally, we discuss the influence of different model sizes, dimensions, and normalization methods. We anticipate that F-Eval will facilitate the study of LLMs' fundamental abilities.

Title: Coca: Improving and Explaining Graph Neural Network-Based Vulnerability Detection Systems

• Authors: Sicong Cao, Xiaobing Sun, Xiaoxue Wu, David Lo, Lili Bo, Bin Li, Wei Liu
• Subjects: cs.CR, cs.SE
• Abstract URL: https://arxiv.org/abs/2401.14886
• Pdf URL: https://arxiv.org/pdf/2401.14886
• Copy Paste: [[2401.14886]] Coca: Improving and Explaining Graph Neural Network-Based Vulnerability Detection Systems(https://arxiv.org/abs/2401.14886)
• Keywords: security, robust, explainability
• Abstract: Recently, Graph Neural Network (GNN)-based vulnerability detection systems have achieved remarkable success. However, the lack of explainability poses a critical challenge to deploy black-box models in security-related domains. For this reason, several approaches have been proposed to explain the decision logic of the detection model by providing a set of crucial statements positively contributing to its predictions. Unfortunately, due to the weakly-robust detection models and suboptimal explanation strategy, they have the danger of revealing spurious correlations and redundancy issue. In this paper, we propose Coca, a general framework aiming to 1) enhance the robustness of existing GNN-based vulnerability detection models to avoid spurious explanations; and 2) provide both concise and effective explanations to reason about the detected vulnerabilities. \sysname consists of two core parts referred to as Trainer and Explainer. The former aims to train a detection model which is robust to random perturbation based on combinatorial contrastive learning, while the latter builds an explainer to derive crucial code statements that are most decisive to the detected vulnerability via dual-view causal inference as explanations. We apply Coca over three typical GNN-based vulnerability detectors. Experimental results show that Coca can effectively mitigate the spurious correlation issue, and provide more useful high-quality explanations.

Title: A structured regression approach for evaluating model performance across intersectional subgroups

• Authors: Christine Herlihy, Kimberly Truong, Alexandra Chouldechova, Miroslav Dudik
• Subjects: cs.LG, cs.CY, stat.AP, stat.ML
• Abstract URL: https://arxiv.org/abs/2401.14893
• Pdf URL: https://arxiv.org/pdf/2401.14893
• Copy Paste: [[2401.14893]] A structured regression approach for evaluating model performance across intersectional subgroups(https://arxiv.org/abs/2401.14893)
• Keywords: fair
• Abstract: Disaggregated evaluation is a central task in AI fairness assessment, with the goal to measure an AI system's performance across different subgroups defined by combinations of demographic or other sensitive attributes. The standard approach is to stratify the evaluation data across subgroups and compute performance metrics separately for each group. However, even for moderately-sized evaluation datasets, sample sizes quickly get small once considering intersectional subgroups, which greatly limits the extent to which intersectional groups are considered in many disaggregated evaluations. In this work, we introduce a structured regression approach to disaggregated evaluation that we demonstrate can yield reliable system performance estimates even for very small subgroups. We also provide corresponding inference strategies for constructing confidence intervals and explore how goodness-of-fit testing can yield insight into the structure of fairness-related harms experienced by intersectional groups. We evaluate our approach on two publicly available datasets, and several variants of semi-synthetic data. The results show that our method is considerably more accurate than the standard approach, especially for small subgroups, and goodness-of-fit testing helps identify the key factors that drive differences in performance.

Title: MPTQ-ViT:Mixed-PrecisionPost-TrainingQuantizationforVisionTransformer

• Authors: Yu-Shan Tai, An-Yeu (Andy)Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.14895
• Pdf URL: https://arxiv.org/pdf/2401.14895
• Copy Paste: [[2401.14895]] MPTQ-ViT:Mixed-PrecisionPost-TrainingQuantizationforVisionTransformer(https://arxiv.org/abs/2401.14895)
• Keywords: transformer
• Abstract: While vision transformers (ViTs) have shown great potential in computer vision tasks, their intense computation and memory requirements pose challenges for practical applications. Existing post-training quantization methods leverage value redistribution or specialized quantizers to address the non-normal distribution in ViTs. However, without considering the asymmetry in activations and relying on hand-crafted settings, these methods often struggle to maintain performance under low-bit quantization. To overcome these challenges, we introduce SmoothQuant with bias term (SQ-b) to alleviate the asymmetry issue and reduce the clamping loss. We also introduce optimal scaling factor ratio search (OPT-m) to determine quantization parameters by a data-dependent mechanism automatically. To further enhance the compressibility, we incorporate the above-mentioned techniques and propose a mixed-precision post-training quantization framework for vision transformers (MPTQ-ViT). We develop greedy mixed-precision quantization (Greedy MP) to allocate layer-wise bit-width considering both model performance and compressibility. Our experiments on ViT, DeiT, and Swin demonstrate significant accuracy improvements compared with SOTA on the ImageNet dataset. Specifically, our proposed methods achieve accuracy improvements ranging from 0.90% to 23.35% on 4-bit ViTs with single-precision and from 3.82% to 78.14% on 5-bit fully quantized ViTs with mixed-precision.

Title: PARSAC: Accelerating Robust Multi-Model Fitting with Parallel Sample Consensus

• Authors: Florian Kluger, Bodo Rosenhahn
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.14919
• Pdf URL: https://arxiv.org/pdf/2401.14919
• Copy Paste: [[2401.14919]] PARSAC: Accelerating Robust Multi-Model Fitting with Parallel Sample Consensus(https://arxiv.org/abs/2401.14919)
• Keywords: robust, segmentation
• Abstract: We present a real-time method for robust estimation of multiple instances of geometric models from noisy data. Geometric models such as vanishing points, planar homographies or fundamental matrices are essential for 3D scene analysis. Previous approaches discover distinct model instances in an iterative manner, thus limiting their potential for speedup via parallel computation. In contrast, our method detects all model instances independently and in parallel. A neural network segments the input data into clusters representing potential model instances by predicting multiple sets of sample and inlier weights. Using the predicted weights, we determine the model parameters for each potential instance separately in a RANSAC-like fashion. We train the neural network via task-specific loss functions, i.e. we do not require a ground-truth segmentation of the input data. As suitable training data for homography and fundamental matrix fitting is scarce, we additionally present two new synthetic datasets. We demonstrate state-of-the-art performance on these as well as multiple established datasets, with inference times as small as five milliseconds per image.

Title: Do LLMs Dream of Ontologies?

• Authors: Marco Bombieri, Paolo Fiorini, Simone Paolo Ponzetto, Marco Rospocher
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.14931
• Pdf URL: https://arxiv.org/pdf/2401.14931
• Copy Paste: [[2401.14931]] Do LLMs Dream of Ontologies?(https://arxiv.org/abs/2401.14931)
• Keywords: large language model
• Abstract: Large language models (LLMs) have recently revolutionized automated text understanding and generation. The performance of these models relies on the high number of parameters of the underlying neural architectures, which allows LLMs to memorize part of the vast quantity of data seen during the training. This paper investigates whether and to what extent general-purpose pre-trained LLMs have memorized information from known ontologies. Our results show that LLMs partially know ontologies: they can, and do indeed, memorize concepts from ontologies mentioned in the text, but the level of memorization of their concepts seems to vary proportionally to their popularity on the Web, the primary source of their training material. We additionally propose new metrics to estimate the degree of memorization of ontological information in LLMs by measuring the consistency of the output produced across different prompt repetitions, query languages, and degrees of determinism.

Title: DAM: Diffusion Activation Maximization for 3D Global Explanations

• Authors: Hanxiao Tan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.14938
• Pdf URL: https://arxiv.org/pdf/2401.14938
• Copy Paste: [[2401.14938]] DAM: Diffusion Activation Maximization for 3D Global Explanations(https://arxiv.org/abs/2401.14938)
• Keywords: explainability, diffusion, transformer
• Abstract: In recent years, the performance of point cloud models has been rapidly improved. However, due to the limited amount of relevant explainability studies, the unreliability and opacity of these black-box models may lead to potential risks in applications where human lives are at stake, e.g. autonomous driving or healthcare. This work proposes a DDPM-based point cloud global explainability method (DAM) that leverages Point Diffusion Transformer (PDT), a novel point-wise symmetric model, with dual-classifier guidance to generate high-quality global explanations. In addition, an adapted path gradient integration method for DAM is proposed, which not only provides a global overview of the saliency maps for point cloud categories, but also sheds light on how the attributions of the explanations vary during the generation process. Extensive experiments indicate that our method outperforms existing ones in terms of perceptibility, representativeness, and diversity, with a significant reduction in generation time. Our code is available at: https://github.com/Explain3D/DAM

Title: Conserve-Update-Revise to Cure Generalization and Robustness Trade-off in Adversarial Training

• Authors: Shruthi Gowda, Bahram Zonooz, Elahe Arani
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2401.14948
• Pdf URL: https://arxiv.org/pdf/2401.14948
• Copy Paste: [[2401.14948]] Conserve-Update-Revise to Cure Generalization and Robustness Trade-off in Adversarial Training(https://arxiv.org/abs/2401.14948)
• Keywords: attack, robust
• Abstract: Adversarial training improves the robustness of neural networks against adversarial attacks, albeit at the expense of the trade-off between standard and robust generalization. To unveil the underlying factors driving this phenomenon, we examine the layer-wise learning capabilities of neural networks during the transition from a standard to an adversarial setting. Our empirical findings demonstrate that selectively updating specific layers while preserving others can substantially enhance the network's learning capacity. We therefore propose CURE, a novel training framework that leverages a gradient prominence criterion to perform selective conservation, updating, and revision of weights. Importantly, CURE is designed to be dataset- and architecture-agnostic, ensuring its applicability across various scenarios. It effectively tackles both memorization and overfitting issues, thus enhancing the trade-off between robustness and generalization and additionally, this training approach also aids in mitigating "robust overfitting". Furthermore, our study provides valuable insights into the mechanisms of selective adversarial training and offers a promising avenue for future research.

Title: Learning Universal Predictors

• Authors: Jordi Grau-Moya, Tim Genewein, Marcus Hutter, Laurent Orseau, Grégoire Delétang, Elliot Catt, Anian Ruoss, Li Kevin Wenliang, Christopher Mattern, Matthew Aitchison, Joel Veness
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.14953
• Pdf URL: https://arxiv.org/pdf/2401.14953
• Copy Paste: [[2401.14953]] Learning Universal Predictors(https://arxiv.org/abs/2401.14953)
• Keywords: transformer
• Abstract: Meta-learning has emerged as a powerful approach to train neural networks to learn new tasks quickly from limited data. Broad exposure to different tasks leads to versatile representations enabling general problem solving. But, what are the limits of meta-learning? In this work, we explore the potential of amortizing the most powerful universal predictor, namely Solomonoff Induction (SI), into neural networks via leveraging meta-learning to its limits. We use Universal Turing Machines (UTMs) to generate training data used to expose networks to a broad range of patterns. We provide theoretical analysis of the UTM data generation processes and meta-training protocols. We conduct comprehensive experiments with neural architectures (e.g. LSTMs, Transformers) and algorithmic data generators of varying complexity and universality. Our results suggest that UTM data is a valuable resource for meta-learning, and that it can be used to train neural networks capable of learning universal prediction strategies.

Title: End-To-End Set-Based Training for Neural Network Verification

• Authors: Lukas Koller, Tobias Ladner, Matthias Althoff
• Subjects: cs.LG, cs.CR, cs.LO
• Abstract URL: https://arxiv.org/abs/2401.14961
• Pdf URL: https://arxiv.org/pdf/2401.14961
• Copy Paste: [[2401.14961]] End-To-End Set-Based Training for Neural Network Verification(https://arxiv.org/abs/2401.14961)
• Keywords: attack, robust
• Abstract: Neural networks are vulnerable to adversarial attacks, i.e., small input perturbations can result in substantially different outputs of a neural network. Safety-critical environments require neural networks that are robust against input perturbations. However, training and formally verifying robust neural networks is challenging. We address this challenge by employing, for the first time, a end-to-end set-based training procedure that trains robust neural networks for formal verification. Our training procedure drastically simplifies the subsequent formal robustness verification of the trained neural network. While previous research has predominantly focused on augmenting neural network training with adversarial attacks, our approach leverages set-based computing to train neural networks with entire sets of perturbed inputs. Moreover, we demonstrate that our set-based training procedure effectively trains robust neural networks, which are easier to verify. In many cases, set-based trained neural networks outperform neural networks trained with state-of-the-art adversarial attacks.

Title: Creating a vulnerable node based on the vulnerability MS17-010

• Authors: Aleksey Novokhrestov, Anton Kalyakin, Aleksandr Kovalenko, Vladimir Repkin
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2401.14979
• Pdf URL: https://arxiv.org/pdf/2401.14979
• Copy Paste: [[2401.14979]] Creating a vulnerable node based on the vulnerability MS17-010(https://arxiv.org/abs/2401.14979)
• Keywords: security, attack
• Abstract: The creation of a vulnerable node has been demonstrated through the analysis and implementation of the MS17-010 (CVE-2017-0144) vulnerability, affecting the SMBv1 protocol on various Windows operating systems. The principle and methodology of exploiting the vulnerability are described, with a formalized representation of the exploitation in the form of a Meta Attack Language (MAL) graph. Additionally, the attacker's implementation is outlined as the execution of an automated script in Python using the Metasploit Framework. Basic security measures for systems utilizing the SMBv1 protocol are provided.

Title: Mapping-to-Parameter Nonlinear Functional Regression with Novel B-spline Free Knot Placement Algorithm

• Authors: Chengdong Shi, Ching-Hsun Tseng, Wei Zhao, Xiao-Jun Zeng
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2401.14989
• Pdf URL: https://arxiv.org/pdf/2401.14989
• Copy Paste: [[2401.14989]] Mapping-to-Parameter Nonlinear Functional Regression with Novel B-spline Free Knot Placement Algorithm(https://arxiv.org/abs/2401.14989)
• Keywords: robust
• Abstract: We propose a novel approach to nonlinear functional regression, called the Mapping-to-Parameter function model, which addresses complex and nonlinear functional regression problems in parameter space by employing any supervised learning technique. Central to this model is the mapping of function data from an infinite-dimensional function space to a finite-dimensional parameter space. This is accomplished by concurrently approximating multiple functions with a common set of B-spline basis functions by any chosen order, with their knot distribution determined by the Iterative Local Placement Algorithm, a newly proposed free knot placement algorithm. In contrast to the conventional equidistant knot placement strategy that uniformly distributes knot locations based on a predefined number of knots, our proposed algorithms determine knot location according to the local complexity of the input or output functions. The performance of our knot placement algorithms is shown to be robust in both single-function approximation and multiple-function approximation contexts. Furthermore, the effectiveness and advantage of the proposed prediction model in handling both function-on-scalar regression and function-on-function regression problems are demonstrated through several real data applications, in comparison with four groups of state-of-the-art methods.

Title: BackdoorBench: A Comprehensive Benchmark and Analysis of Backdoor Learning

• Authors: Baoyuan Wu, Hongrui Chen, Mingda Zhang, Zihao Zhu, Shaokui Wei, Danni Yuan, Mingli Zhu, Ruotong Wang, Li Liu, Chao Shen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.15002
• Pdf URL: https://arxiv.org/pdf/2401.15002
• Copy Paste: [[2401.15002]] BackdoorBench: A Comprehensive Benchmark and Analysis of Backdoor Learning(https://arxiv.org/abs/2401.15002)
• Keywords: defense, attack, fair
• Abstract: As an emerging and vital topic for studying deep neural networks' vulnerability (DNNs), backdoor learning has attracted increasing interest in recent years, and many seminal backdoor attack and defense algorithms are being developed successively or concurrently, in the status of a rapid arms race. However, mainly due to the diverse settings, and the difficulties of implementation and reproducibility of existing works, there is a lack of a unified and standardized benchmark of backdoor learning, causing unfair comparisons, and unreliable conclusions (e.g., misleading, biased or even false conclusions). Consequently, it is difficult to evaluate the current progress and design the future development roadmap of this literature. To alleviate this dilemma, we build a comprehensive benchmark of backdoor learning called BackdoorBench. Our benchmark makes three valuable contributions to the research community. 1) We provide an integrated implementation of state-of-the-art (SOTA) backdoor learning algorithms (currently including 16 attack and 27 defense algorithms), based on an extensible modular-based codebase. 2) We conduct comprehensive evaluations of 12 attacks against 16 defenses, with 5 poisoning ratios, based on 4 models and 4 datasets, thus 11,492 pairs of evaluations in total. 3) Based on above evaluations, we present abundant analysis from 8 perspectives via 18 useful analysis tools, and provide several inspiring insights about backdoor learning. We hope that our efforts could build a solid foundation of backdoor learning to facilitate researchers to investigate existing algorithms, develop more innovative algorithms, and explore the intrinsic mechanism of backdoor learning. Finally, we have created a user-friendly website at this http URL, which collects all important information of BackdoorBench, including codebase, docs, leaderboard, and model Zoo.

Title: SliceGPT: Compress Large Language Models by Deleting Rows and Columns

• Authors: Saleh Ashkboos, Maximilian L. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, James Hensman
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2401.15024
• Pdf URL: https://arxiv.org/pdf/2401.15024
• Copy Paste: [[2401.15024]] SliceGPT: Compress Large Language Models by Deleting Rows and Columns(https://arxiv.org/abs/2401.15024)
• Keywords: transformer, large language model
• Abstract: Large language models have become the cornerstone of natural language processing, but their use comes with substantial costs in terms of compute and memory resources. Sparsification provides a solution to alleviate these resource constraints, and recent works have shown that trained models can be sparsified post-hoc. Existing sparsification techniques face challenges as they need additional data structures and offer constrained speedup with current hardware. In this paper we present SliceGPT, a new post-training sparsification scheme which replaces each weight matrix with a smaller (dense) matrix, reducing the embedding dimension of the network. Through extensive experimentation, we show that SliceGPT can remove up to 25% of the model parameters (including embeddings) for LLAMA2-70B, OPT 66B and Phi-2 models while maintaining 99%, 99% and 90% zero-shot task performance of the dense model respectively. Our sliced models run on fewer GPUs and run faster without any additional code optimization: on 24GB consumer GPUs we reduce the total compute for inference on LLAMA2-70B to 64% of that of the dense model; on 40GB A100 GPUs we reduce it to 66%. We offer a new insight, computational invariance in transformer networks, which enables SliceGPT and we hope it will inspire and enable future avenues to reduce memory and computation demands for pre-trained models. Code is available at: https://github.com/microsoft/TransformerCompression

Title: On the generalization capacity of neural networks during generic multimodal reasoning

• Authors: Takuya Ito, Soham Dan, Mattia Rigotti, James Kozloski, Murray Campbell
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.15030
• Pdf URL: https://arxiv.org/pdf/2401.15030
• Copy Paste: [[2401.15030]] On the generalization capacity of neural networks during generic multimodal reasoning(https://arxiv.org/abs/2401.15030)
• Keywords: transformer, large language model
• Abstract: The advent of the Transformer has led to the development of large language models (LLM), which appear to demonstrate human-like capabilities. To assess the generality of this class of models and a variety of other base neural network architectures to multimodal domains, we evaluated and compared their capacity for multimodal generalization. We introduce a multimodal question-answer benchmark to evaluate three specific types of out-of-distribution (OOD) generalization performance: distractor generalization (generalization in the presence of distractors), systematic compositional generalization (generalization to new task permutations), and productive compositional generalization (generalization to more complex tasks structures). We found that across model architectures (e.g., RNNs, Transformers, Perceivers, etc.), models with multiple attention layers, or models that leveraged cross-attention mechanisms between input domains, fared better. Our positive results demonstrate that for multimodal distractor and systematic generalization, either cross-modal attention or models with deeper attention layers are key architectural features required to integrate multimodal inputs. On the other hand, neither of these architectural features led to productive generalization, suggesting fundamental limitations of existing architectures for specific types of multimodal generalization. These results demonstrate the strengths and limitations of specific architectural components underlying modern neural models for multimodal reasoning. Finally, we provide Generic COG (gCOG), a configurable benchmark with several multimodal generalization splits, for future studies to explore.

Title: Physical Layer Encryption for Industrial Ethernet in Gigabit Optical Links

• Authors: Adrián Pérez-Resa, Miguel García-Bosque, Carlos Sánchez-Azqueta, Santiago Celma
• Subjects: cs.CR, eess.SP, eess.SY
• Abstract URL: https://arxiv.org/abs/2401.15038
• Pdf URL: https://arxiv.org/pdf/2401.15038
• Copy Paste: [[2401.15038]] Physical Layer Encryption for Industrial Ethernet in Gigabit Optical Links(https://arxiv.org/abs/2401.15038)
• Keywords: secure
• Abstract: Industrial Ethernet is a technology widely spread in factory floors and critical infrastructures where a high amount of data need to be collected and transported. Fiber optic networks at gigabit rates fit well with that type of environments where speed, system performance and reliability are critical. In this work a new encryption method for high speed optical communications suitable for such kind of networks is proposed. This new encryption method consists of a symmetric streaming encryption of the 8b/10b data flow at PCS (Physical Coding Sublayer) level. It is carried out thanks to an FPE (Format Preserving Encryption) blockcipher working in CTR (Counter) mode. The overall system has been simulated and implemented in an FPGA (Field Programmable Gate Array). Thanks to experimental results it can be concluded that it is possible to cipher traffic at this physical level in a secure way. In addition, no overhead is introduced during encryption, getting minimum latency and maximum throughput.

Title: Chaotic Encryption Applied to Optical Ethernet in Industrial Control Systems

• Authors: Adrián Pérez-Resa, Miguel Garcia-Bosque, Carlos Sánchez-Azqueta, Santiago Celma
• Subjects: cs.CR, eess.SP
• Abstract URL: https://arxiv.org/abs/2401.15039
• Pdf URL: https://arxiv.org/pdf/2401.15039
• Copy Paste: [[2401.15039]] Chaotic Encryption Applied to Optical Ethernet in Industrial Control Systems(https://arxiv.org/abs/2401.15039)
• Keywords: security
• Abstract: In the past decades, Ethernet has become an alternative technology for the field buses traditionally used in industrial control systems and distributed measurement systems. Among different transmission media in Ethernet standards, optical fiber provides the best bandwidth, excellent immunity to electromagnetic interference, and less signal loses than other wired media. Due to the absence of a standard that provides security at the physical layer of optical Ethernet links, the main motivation of this paper is to propose and implement the necessary modifications to introduce encryption in Ethernet 1000Base-X standard. This has consisted of symmetric streaming encryption of the 8b10b symbols flow at physical coding sublayer level, thanks to a keystream generator based on chaotic algorithm. The overall system has been implemented and tested in an field programmable gate array and Ethernet traffic has been encrypted and transmitted over an optical link. The experimental results show that it is possible to cipher traffic at this level and hide the complete Ethernet traffic pattern from passive eavesdroppers. In addition, no space overhead is introduced in data frames during encryption, achieving the maximum throughput.

Title: Computationally Bounded Robust Compilation and Universally Composable Security

• Authors: Robert Künnemann, Marco Patrignani, Ethan Cecchetti
• Subjects: cs.CR, cs.PL
• Abstract URL: https://arxiv.org/abs/2401.15041
• Pdf URL: https://arxiv.org/pdf/2401.15041
• Copy Paste: [[2401.15041]] Computationally Bounded Robust Compilation and Universally Composable Security(https://arxiv.org/abs/2401.15041)
• Keywords: secure, security, robust
• Abstract: Universal Composability (UC) is the gold standard for cryptographic security, but mechanizing proofs of UC is notoriously difficult. A recently-discovered connection between UC and Robust Compilation (RC)$\unicode{x2014}$a novel theory of secure compilation$\unicode{x2014}$provides a means to verify UC proofs using tools that mechanize equality results. Unfortunately, the existing methods apply only to perfect UC security, and real-world protocols relying on cryptography are only computationally secure. This paper addresses this gap by lifting the connection between UC and RC to the computational setting, extending techniques from the RC setting to apply to computational UC security. Moreover, it further generalizes the UC$\unicode{x2013}$RC connection beyond computational security to arbitrary equalities, providing a framework to subsume the existing perfect case, and to instantiate future theories with more complex notions of security. This connection allows the use of tools for proofs of computational indistinguishability to properly mechanize proofs of computational UC security. We demonstrate this power by using CryptoVerif to mechanize a proof that parts of the Wireguard protocol are computationally UC secure. Finally, all proofs of the framework itself are verified in Isabelle/HOL.

Title: PROXYQA: An Alternative Framework for Evaluating Long-Form Text Generation with Large Language Models

• Authors: Haochen Tan, Zhijiang Guo, Zhan Shi, Lu Xu, Zhili Liu, Xiaoguang Li, Yasheng Wang, Lifeng Shang, Qun Liu, Linqi Song
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.15042
• Pdf URL: https://arxiv.org/pdf/2401.15042
• Copy Paste: [[2401.15042]] PROXYQA: An Alternative Framework for Evaluating Long-Form Text Generation with Large Language Models(https://arxiv.org/abs/2401.15042)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have exhibited remarkable success in long-form context comprehension tasks. However, their capacity to generate long contents, such as reports and articles, remains insufficiently explored. Current benchmarks do not adequately assess LLMs' ability to produce informative and comprehensive content, necessitating a more rigorous evaluation approach. In this study, we introduce \textsc{ProxyQA}, a framework for evaluating long-form text generation, comprising in-depth human-curated \textit{meta-questions} spanning various domains. Each meta-question contains corresponding \textit{proxy-questions} with annotated answers. LLMs are prompted to generate extensive content in response to these meta-questions. Utilizing an evaluator and incorporating generated content as background context, \textsc{ProxyQA} evaluates the quality of generated content based on the evaluator's performance in answering the \textit{proxy-questions}. We examine multiple LLMs, emphasizing \textsc{ProxyQA}'s demanding nature as a high-quality assessment tool. Human evaluation demonstrates that evaluating through \textit{proxy-questions} is a highly self-consistent and human-criteria-correlated validation method. The dataset and leaderboard will be available at \url{https://github.com/Namco0816/ProxyQA}.

Title: Health Text Simplification: An Annotated Corpus for Digestive Cancer Education and Novel Strategies for Reinforcement Learning

• Authors: Md Mushfiqur Rahman, Mohammad Sabik Irbaz, Kai North, Michelle S. Williams, Marcos Zampieri, Kevin Lybarger
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.15043
• Pdf URL: https://arxiv.org/pdf/2401.15043
• Copy Paste: [[2401.15043]] Health Text Simplification: An Annotated Corpus for Digestive Cancer Education and Novel Strategies for Reinforcement Learning(https://arxiv.org/abs/2401.15043)
• Keywords: large language model
• Abstract: Objective: The reading level of health educational materials significantly influences information understandability and accessibility, particularly for minoritized populations. Many patient educational resources surpass the reading level and complexity of widely accepted standards. There is a critical need for high-performing text simplification models in health information to enhance dissemination and literacy. This need is particularly acute in cancer education, where effective prevention and screening education can substantially reduce morbidity and mortality. Methods: We introduce Simplified Digestive Cancer (SimpleDC), a parallel corpus of cancer education materials tailored for health text simplification research. Utilizing SimpleDC alongside the existing Med-EASi corpus, we explore Large Language Model (LLM)-based simplification methods, including fine-tuning, reinforcement learning (RL), reinforcement learning with human feedback (RLHF), domain adaptation, and prompt-based approaches. Our experimentation encompasses Llama 2 and GPT-4. A novel RLHF reward function is introduced, featuring a lightweight model adept at distinguishing between original and simplified texts, thereby enhancing the model's effectiveness with unlabeled data. Results: Fine-tuned Llama 2 models demonstrated high performance across various metrics. Our innovative RLHF reward function surpassed existing RL text simplification reward functions in effectiveness. The results underscore that RL/RLHF can augment fine-tuning, facilitating model training on unlabeled text and improving performance. Additionally, these methods effectively adapt out-of-domain text simplification models to targeted domains.

Title: Unrecognizable Yet Identifiable: Image Distortion with Preserved Embeddings

• Authors: Dmytro Zakharov, Oleksandr Kuznetsov, Emanuele Frontoni
• Subjects: cs.CV, cs.CR
• Abstract URL: https://arxiv.org/abs/2401.15048
• Pdf URL: https://arxiv.org/pdf/2401.15048
• Copy Paste: [[2401.15048]] Unrecognizable Yet Identifiable: Image Distortion with Preserved Embeddings(https://arxiv.org/abs/2401.15048)
• Keywords: security, privacy, biometric
• Abstract: In the realm of security applications, biometric authentication systems play a crucial role, yet one often encounters challenges concerning privacy and security while developing one. One of the most fundamental challenges lies in avoiding storing biometrics directly in the storage but still achieving decently high accuracy. Addressing this issue, we contribute to both artificial intelligence and engineering fields. We introduce an innovative image distortion technique that effectively renders facial images unrecognizable to the eye while maintaining their identifiability by neural network models. From the theoretical perspective, we explore how reliable state-of-the-art biometrics recognition neural networks are by checking the maximal degree of image distortion, which leaves the predicted identity unchanged. On the other hand, applying this technique demonstrates a practical solution to the engineering challenge of balancing security, precision, and performance in biometric authentication systems. Through experimenting on the widely used datasets, we assess the effectiveness of our method in preserving AI feature representation and distorting relative to conventional metrics. We also compare our method with previously used approaches.

Title: LongFin: A Multimodal Document Understanding Model for Long Financial Domain Documents

• Authors: Ahmed Masry, Amir Hajian
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.15050
• Pdf URL: https://arxiv.org/pdf/2401.15050
• Copy Paste: [[2401.15050]] LongFin: A Multimodal Document Understanding Model for Long Financial Domain Documents(https://arxiv.org/abs/2401.15050)
• Keywords: extraction
• Abstract: Document AI is a growing research field that focuses on the comprehension and extraction of information from scanned and digital documents to make everyday business operations more efficient. Numerous downstream tasks and datasets have been introduced to facilitate the training of AI models capable of parsing and extracting information from various document types such as receipts and scanned forms. Despite these advancements, both existing datasets and models fail to address critical challenges that arise in industrial contexts. Existing datasets primarily comprise short documents consisting of a single page, while existing models are constrained by a limited maximum length, often set at 512 tokens. Consequently, the practical application of these methods in financial services, where documents can span multiple pages, is severely impeded. To overcome these challenges, we introduce LongFin, a multimodal document AI model capable of encoding up to 4K tokens. We also propose the LongForms dataset, a comprehensive financial dataset that encapsulates several industrial challenges in financial documents. Through an extensive evaluation, we demonstrate the effectiveness of the LongFin model on the LongForms dataset, surpassing the performance of existing public models while maintaining comparable results on existing single-page benchmarks.

Title: From GPT-4 to Gemini and Beyond: Assessing the Landscape of MLLMs on Generalizability, Trustworthiness and Causality through Four Modalities

• Authors: Chaochao Lu, Chen Qian, Guodong Zheng, Hongxing Fan, Hongzhi Gao, Jie Zhang, Jing Shao, Jingyi Deng, Jinlan Fu, Kexin Huang, Kunchang Li, Lijun Li, Limin Wang, Lu Sheng, Meiqi Chen, Ming Zhang, Qibing Ren, Sirui Chen, Tao Gui, Wanli Ouyang, Yali Wang, Yan Teng, Yaru Wang, Yi Wang, Yinan He, Yingchun Wang, Yixu Wang, Yongting Zhang, Yu Qiao, Yujiong Shen, Yurong Mou, Yuxi Chen, Zaibin Zhang, Zhelun Shi, Zhenfei Yin, Zhipin Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.15071
• Pdf URL: https://arxiv.org/pdf/2401.15071
• Copy Paste: [[2401.15071]] From GPT-4 to Gemini and Beyond: Assessing the Landscape of MLLMs on Generalizability, Trustworthiness and Causality through Four Modalities(https://arxiv.org/abs/2401.15071)
• Keywords: large language model
• Abstract: Multi-modal Large Language Models (MLLMs) have shown impressive abilities in generating reasonable responses with respect to multi-modal contents. However, there is still a wide gap between the performance of recent MLLM-based applications and the expectation of the broad public, even though the most powerful OpenAI's GPT-4 and Google's Gemini have been deployed. This paper strives to enhance understanding of the gap through the lens of a qualitative study on the generalizability, trustworthiness, and causal reasoning capabilities of recent proprietary and open-source MLLMs across four modalities: ie, text, code, image, and video, ultimately aiming to improve the transparency of MLLMs. We believe these properties are several representative factors that define the reliability of MLLMs, in supporting various downstream applications. To be specific, we evaluate the closed-source GPT-4 and Gemini and 6 open-source LLMs and MLLMs. Overall we evaluate 230 manually designed cases, where the qualitative results are then summarized into 12 scores (ie, 4 modalities times 3 properties). In total, we uncover 14 empirical findings that are useful to understand the capabilities and limitations of both proprietary and open-source MLLMs, towards more reliable downstream multi-modal applications.

Title: Annotated Hands for Generative Models

• Authors: Yue Yang, Atith N Gandhi, Greg Turk
• Subjects: cs.CV, cs.AI, cs.GR
• Abstract URL: https://arxiv.org/abs/2401.15075
• Pdf URL: https://arxiv.org/pdf/2401.15075
• Copy Paste: [[2401.15075]] Annotated Hands for Generative Models(https://arxiv.org/abs/2401.15075)
• Keywords: diffusion, generative
• Abstract: Generative models such as GANs and diffusion models have demonstrated impressive image generation capabilities. Despite these successes, these systems are surprisingly poor at creating images with hands. We propose a novel training framework for generative models that substantially improves the ability of such systems to create hand images. Our approach is to augment the training images with three additional channels that provide annotations to hands in the image. These annotations provide additional structure that coax the generative model to produce higher quality hand images. We demonstrate this approach on two different generative models: a generative adversarial network and a diffusion model. We demonstrate our method both on a new synthetic dataset of hand images and also on real photographs that contain hands. We measure the improved quality of the generated hands through higher confidence in finger joint identification using an off-the-shelf hand detector.

Title: EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty

• Authors: Yuhui Li, Fangyun Wei, Chao Zhang, Hongyang Zhang
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2401.15077
• Pdf URL: https://arxiv.org/pdf/2401.15077
• Copy Paste: [[2401.15077]] EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty(https://arxiv.org/abs/2401.15077)
• Keywords: large language model
• Abstract: Auto-regressive decoding makes the inference of Large Language Models (LLMs) time-consuming. We propose a simple framework, EAGLE (Extrapolation Algorithm for Greater Language-model Efficiency), for lossless acceleration. Unlike traditional speculative sampling methods, EAGLE operates the drafting process auto-regressively at the more regular (second-top-layer) feature level and addresses the sampling uncertainty issues in the next-feature prediction problems by integrating tokens from one time step ahead. The acceleration provided by EAGLE is lossless: it involves no fine-tuning of the target LLM, and the generated text maintains the same distribution as that of vanilla auto-regressive decoding. As of the submission of this paper, EAGLE is the fastest known framework within the speculative sampling family. On MT-bench, EAGLE is 3x faster than vanilla decoding, 2x faster than Lookahead, and 1.6x faster than Medusa. Using gpt-fast, EAGLE attains on average 160 tokens/s with LLaMA2-Chat 13B on a single RTX 3090 GPU, compared to 24 tokens/s of Huggingface's implementations.