2024-02-19

Title: Correlational Lagrangian Schrödinger Bridge: Learning Dynamics with Population-Level Regularization

• Authors: Yuning You, Ruida Zhou, Yang Shen
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.10227
• Pdf URL: https://arxiv.org/pdf/2402.10227
• Copy Paste: [[2402.10227]] Correlational Lagrangian Schrödinger Bridge: Learning Dynamics with Population-Level Regularization(https://arxiv.org/abs/2402.10227)
• Keywords: generative
• Abstract: Accurate modeling of system dynamics holds intriguing potential in broad scientific fields including cytodynamics and fluid mechanics. This task often presents significant challenges when (i) observations are limited to cross-sectional samples (where individual trajectories are inaccessible for learning), and moreover, (ii) the behaviors of individual particles are heterogeneous (especially in biological systems due to biodiversity). To address them, we introduce a novel framework dubbed correlational Lagrangian Schr\"odinger bridge (CLSB), aiming to seek for the evolution "bridging" among cross-sectional observations, while regularized for the minimal population "cost". In contrast to prior methods relying on \textit{individual}-level regularizers for all particles \textit{homogeneously} (e.g. restraining individual motions), CLSB operates at the population level admitting the heterogeneity nature, resulting in a more generalizable modeling in practice. To this end, our contributions include (1) a new class of population regularizers capturing the temporal variations in multivariate relations, with the tractable formulation derived, (2) three domain-informed instantiations based on genetic co-expression stability, and (3) an integration of population regularizers into data-driven generative models as constrained optimization, and a numerical solution, with further extension to conditional generative models. Empirically, we demonstrate the superiority of CLSB in single-cell sequencing data analyses such as simulating cell development over time and predicting cellular responses to drugs of varied doses.

Title: HyperAgent: A Simple, Scalable, Efficient and Provable Reinforcement Learning Framework for Complex Environments

• Authors: Yingru Li, Jiawei Xu, Lei Han, Zhi-Quan Luo
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.10228
• Pdf URL: https://arxiv.org/pdf/2402.10228
• Copy Paste: [[2402.10228]] HyperAgent: A Simple, Scalable, Efficient and Provable Reinforcement Learning Framework for Complex Environments(https://arxiv.org/abs/2402.10228)
• Keywords: robust
• Abstract: To solve complex tasks under resource constraints, reinforcement learning (RL) agents need to be simple, efficient, and scalable with (1) large state space and (2) increasingly accumulated data of interactions. We propose the HyperAgent, a RL framework with hypermodel, index sampling schemes and incremental update mechanism, enabling computation-efficient sequential posterior approximation and data-efficient action selection under general value function approximation beyond conjugacy. The implementation of \HyperAgent is simple as it only adds one module and one line of code additional to DDQN. Practically, HyperAgent demonstrates its robust performance in large-scale deep RL benchmarks with significant efficiency gain in terms of both data and computation. Theoretically, among the practically scalable algorithms, HyperAgent is the first method to achieve provably scalable per-step computational complexity as well as sublinear regret under tabular RL. The core of our theoretical analysis is the sequential posterior approximation argument, made possible by the first analytical tool for sequential random projection, a non-trivial martingale extension of the Johnson-Lindenstrauss lemma. This work bridges the theoretical and practical realms of RL, establishing a new benchmark for RL algorithm design.

Title: Parametric Learning of Time-Advancement Operators for Unstable Flame Evolution

• Authors: Rixin Yu, Erdzan Hodzic
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.10238
• Pdf URL: https://arxiv.org/pdf/2402.10238
• Copy Paste: [[2402.10238]] Parametric Learning of Time-Advancement Operators for Unstable Flame Evolution(https://arxiv.org/abs/2402.10238)
• Keywords: robust
• Abstract: This study investigates the application of machine learning, specifically Fourier Neural Operator (FNO) and Convolutional Neural Network (CNN), to learn time-advancement operators for parametric partial differential equations (PDEs). Our focus is on extending existing operator learning methods to handle additional inputs representing PDE parameters. The goal is to create a unified learning approach that accurately predicts short-term solutions and provides robust long-term statistics under diverse parameter conditions, facilitating computational cost savings and accelerating development in engineering simulations. We develop and compare parametric learning methods based on FNO and CNN, evaluating their effectiveness in learning parametric-dependent solution time-advancement operators for one-dimensional PDEs and realistic flame front evolution data obtained from direct numerical simulations of the Navier-Stokes equations.

Title: A Dynamical View of the Question of Why

• Authors: Mehdi Fatemi, Sindhu Gowda
• Subjects: cs.LG, cs.AI, eess.SY
• Abstract URL: https://arxiv.org/abs/2402.10240
• Pdf URL: https://arxiv.org/pdf/2402.10240
• Copy Paste: [[2402.10240]] A Dynamical View of the Question of Why(https://arxiv.org/abs/2402.10240)
• Keywords: fair, diffusion
• Abstract: We address causal reasoning in multivariate time series data generated by stochastic processes. Existing approaches are largely restricted to static settings, ignoring the continuity and emission of variations across time. In contrast, we propose a learning paradigm that directly establishes causation between events in the course of time. We present two key lemmas to compute causal contributions and frame them as reinforcement learning problems. Our approach offers formal and computational tools for uncovering and quantifying causal relationships in diffusion processes, subsuming various important settings such as discrete-time Markov decision processes. Finally, in fairly intricate experiments and through sheer learning, our framework reveals and quantifies causal links, which otherwise seem inexplicable.

Title: Personalized Federated Learning for Statistical Heterogeneity

• Authors: Muhammad Firdaus, Kyung-Hyune Rhee
• Subjects: cs.LG, cs.NI
• Abstract URL: https://arxiv.org/abs/2402.10254
• Pdf URL: https://arxiv.org/pdf/2402.10254
• Copy Paste: [[2402.10254]] Personalized Federated Learning for Statistical Heterogeneity(https://arxiv.org/abs/2402.10254)
• Keywords: privacy, federate
• Abstract: The popularity of federated learning (FL) is on the rise, along with growing concerns about data privacy in artificial intelligence applications. FL facilitates collaborative multi-party model learning while simultaneously ensuring the preservation of data confidentiality. Nevertheless, the problem of statistical heterogeneity caused by the presence of diverse client data distributions gives rise to certain challenges, such as inadequate personalization and slow convergence. In order to address the above issues, this paper offers a brief summary of the current research progress in the field of personalized federated learning (PFL). It outlines the PFL concept, examines related techniques, and highlights current endeavors. Furthermore, this paper also discusses potential further research and obstacles associated with PFL.

Title: GaussianObject: Just Taking Four Images to Get A High-Quality 3D Object with Gaussian Splatting

• Authors: Chen Yang, Sikuang Li, Jiemin Fang, Ruofan Liang, Lingxi Xie, Xiaopeng Zhang, Wei Shen, Qi Tian
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2402.10259
• Pdf URL: https://arxiv.org/pdf/2402.10259
• Copy Paste: [[2402.10259]] GaussianObject: Just Taking Four Images to Get A High-Quality 3D Object with Gaussian Splatting(https://arxiv.org/abs/2402.10259)
• Keywords: diffusion
• Abstract: Reconstructing and rendering 3D objects from highly sparse views is of critical importance for promoting applications of 3D vision techniques and improving user experience. However, images from sparse views only contain very limited 3D information, leading to two significant challenges: 1) Difficulty in building multi-view consistency as images for matching are too few; 2) Partially omitted or highly compressed object information as view coverage is insufficient. To tackle these challenges, we propose GaussianObject, a framework to represent and render the 3D object with Gaussian splatting, that achieves high rendering quality with only 4 input images. We first introduce techniques of visual hull and floater elimination which explicitly inject structure priors into the initial optimization process for helping build multi-view consistency, yielding a coarse 3D Gaussian representation. Then we construct a Gaussian repair model based on diffusion models to supplement the omitted object information, where Gaussians are further refined. We design a self-generating strategy to obtain image pairs for training the repair model. Our GaussianObject is evaluated on several challenging datasets, including MipNeRF360, OmniObject3D, and OpenIllumination, achieving strong reconstruction results from only 4 views and significantly outperforming previous state-of-the-art methods.

Title: A StrongREJECT for Empty Jailbreaks

• Authors: Alexandra Souly, Qingyuan Lu, Dillon Bowen, Tu Trinh, Elvis Hsieh, Sana Pandey, Pieter Abbeel, Justin Svegliato, Scott Emmons, Olivia Watkins, Sam Toyer
• Subjects: cs.LG, cs.CL, cs.CR
• Abstract URL: https://arxiv.org/abs/2402.10260
• Pdf URL: https://arxiv.org/pdf/2402.10260
• Copy Paste: [[2402.10260]] A StrongREJECT for Empty Jailbreaks(https://arxiv.org/abs/2402.10260)
• Keywords: large language model
• Abstract: The rise of large language models (LLMs) has drawn attention to the existence of "jailbreaks" that allow the models to be used maliciously. However, there is no standard benchmark for measuring the severity of a jailbreak, leaving authors of jailbreak papers to create their own. We show that these benchmarks often include vague or unanswerable questions and use grading criteria that are biased towards overestimating the misuse potential of low-quality model responses. Some jailbreak techniques make the problem worse by decreasing the quality of model responses even on benign questions: we show that several jailbreaking techniques substantially reduce the zero-shot performance of GPT-4 on MMLU. Jailbreaks can also make it harder to elicit harmful responses from an "uncensored" open-source model. We present a new benchmark, StrongREJECT, which better discriminates between effective and ineffective jailbreaks by using a higher-quality question set and a more accurate response grading algorithm. We show that our new grading scheme better accords with human judgment of response quality and overall jailbreak effectiveness, especially on the sort of low-quality responses that contribute the most to over-estimation of jailbreak performance on existing benchmarks. We release our code and data at https://github.com/alexandrasouly/strongreject.

Title: SusFL: Energy-Aware Federated Learning-based Monitoring for Sustainable Smart Farms

• Authors: Dian Chen, Paul Yang, Ing-Ray Chen, Dong Sam Ha, Jin-Hee Cho
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.10280
• Pdf URL: https://arxiv.org/pdf/2402.10280
• Copy Paste: [[2402.10280]] SusFL: Energy-Aware Federated Learning-based Monitoring for Sustainable Smart Farms(https://arxiv.org/abs/2402.10280)
• Keywords: privacy, attack, federate
• Abstract: We propose a novel energy-aware federated learning (FL)-based system, namely SusFL, for sustainable smart farming to address the challenge of inconsistent health monitoring due to fluctuating energy levels of solar sensors. This system equips animals, such as cattle, with solar sensors with computational capabilities, including Raspberry Pis, to train a local deep-learning model on health data. These sensors periodically update Long Range (LoRa) gateways, forming a wireless sensor network (WSN) to detect diseases like mastitis. Our proposed SusFL system incorporates mechanism design, a game theory concept, for intelligent client selection to optimize monitoring quality while minimizing energy use. This strategy ensures the system's sustainability and resilience against adversarial attacks, including data poisoning and privacy threats, that could disrupt FL operations. Through extensive comparative analysis using real-time datasets, we demonstrate that our FL-based monitoring system significantly outperforms existing methods in prediction accuracy, operational efficiency, system reliability (i.e., mean time between failures or MTBF), and social welfare maximization by the mechanism designer. Our findings validate the superiority of our system for effective and sustainable animal health monitoring in smart farms. The experimental results show that SusFL significantly improves system performance, including a $10\%$ reduction in energy consumption, a $15\%$ increase in social welfare, and a $34\%$ rise in Mean Time Between Failures (MTBF), alongside a marginal increase in the global model's prediction accuracy.

Title: Backdoor Attack against One-Class Sequential Anomaly Detection Models

• Authors: He Cheng, Shuhan Yuan
• Subjects: cs.LG, cs.AI, cs.CR, cs.IT
• Abstract URL: https://arxiv.org/abs/2402.10283
• Pdf URL: https://arxiv.org/pdf/2402.10283
• Copy Paste: [[2402.10283]] Backdoor Attack against One-Class Sequential Anomaly Detection Models(https://arxiv.org/abs/2402.10283)
• Keywords: security, attack
• Abstract: Deep anomaly detection on sequential data has garnered significant attention due to the wide application scenarios. However, deep learning-based models face a critical security threat - their vulnerability to backdoor attacks. In this paper, we explore compromising deep sequential anomaly detection models by proposing a novel backdoor attack strategy. The attack approach comprises two primary steps, trigger generation and backdoor injection. Trigger generation is to derive imperceptible triggers by crafting perturbed samples from the benign normal data, of which the perturbed samples are still normal. The backdoor injection is to properly inject the backdoor triggers to comprise the model only for the samples with triggers. The experimental results demonstrate the effectiveness of our proposed attack strategy by injecting backdoors on two well-established one-class anomaly detection models.

Title: Discrete Probabilistic Inference as Control in Multi-path Environments

• Authors: Tristan Deleu, Padideh Nouri, Nikolay Malkin, Doina Precup, Yoshua Bengio
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.10309
• Pdf URL: https://arxiv.org/pdf/2402.10309
• Copy Paste: [[2402.10309]] Discrete Probabilistic Inference as Control in Multi-path Environments(https://arxiv.org/abs/2402.10309)
• Keywords: generative
• Abstract: We consider the problem of sampling from a discrete and structured distribution as a sequential decision problem, where the objective is to find a stochastic policy such that objects are sampled at the end of this sequential process proportionally to some predefined reward. While we could use maximum entropy Reinforcement Learning (MaxEnt RL) to solve this problem for some distributions, it has been shown that in general, the distribution over states induced by the optimal policy may be biased in cases where there are multiple ways to generate the same object. To address this issue, Generative Flow Networks (GFlowNets) learn a stochastic policy that samples objects proportionally to their reward by approximately enforcing a conservation of flows across the whole Markov Decision Process (MDP). In this paper, we extend recent methods correcting the reward in order to guarantee that the marginal distribution induced by the optimal MaxEnt RL policy is proportional to the original reward, regardless of the structure of the underlying MDP. We also prove that some flow-matching objectives found in the GFlowNet literature are in fact equivalent to well-established MaxEnt RL algorithms with a corrected reward. Finally, we study empirically the performance of multiple MaxEnt RL and GFlowNet algorithms on multiple problems involving sampling from discrete distributions.

Title: Interpretable Generative Adversarial Imitation Learning

• Authors: Wenliang Liu, Danyang Li, Erfan Aasi, Roberto Tron, Calin Belta
• Subjects: cs.LG, eess.SY
• Abstract URL: https://arxiv.org/abs/2402.10310
• Pdf URL: https://arxiv.org/pdf/2402.10310
• Copy Paste: [[2402.10310]] Interpretable Generative Adversarial Imitation Learning(https://arxiv.org/abs/2402.10310)
• Keywords: interpretability, generative
• Abstract: Imitation learning methods have demonstrated considerable success in teaching autonomous systems complex tasks through expert demonstrations. However, a limitation of these methods is their lack of interpretability, particularly in understanding the specific task the learning agent aims to accomplish. In this paper, we propose a novel imitation learning method that combines Signal Temporal Logic (STL) inference and control synthesis, enabling the explicit representation of the task as an STL formula. This approach not only provides a clear understanding of the task but also allows for the incorporation of human knowledge and adaptation to new scenarios through manual adjustments of the STL formulae. Additionally, we employ a Generative Adversarial Network (GAN)-inspired training approach for both the inference and the control policy, effectively narrowing the gap between the expert and learned policies. The effectiveness of our algorithm is demonstrated through two case studies, showcasing its practical applicability and adaptability.

Title: Hacktivism Goes Orbital: Investigating NB65's Breach of ROSCOSMOS

• Authors: Rajiv Thummala, Gregory Falco
• Subjects: cs.CR, eess.SY
• Abstract URL: https://arxiv.org/abs/2402.10324
• Pdf URL: https://arxiv.org/pdf/2402.10324
• Copy Paste: [[2402.10324]] Hacktivism Goes Orbital: Investigating NB65's Breach of ROSCOSMOS(https://arxiv.org/abs/2402.10324)
• Keywords: attack
• Abstract: In March of 2022, Network battalion 65 (NB65), a hacktivist affiliate of Anonymous, publicly asserted its successful breach of ROSCOSMOS's satellite imaging capabilities in response to Russia's invasion of Ukraine. NB65 disseminated a series of primary sources as substantiation, proclaiming the incapacitation of ROSCOSMOS's space-based vehicle monitoring system and doxing of related proprietary documentation. Despite the profound implications of hacktivist incursions into the space sector, the event has garnered limited attention due to the obscurity of technical attack vectors and ROCOSMOS's denial of NB65's allegations. Through analysis of NB65's released primary sources of evidence, this paper uncovers the probable vulnerabilities and exploits that enabled the alleged breach into ROSCOSMOS's ground and space segment. Additionally, we highlight lessons learned and the consequences this event has for the global aerospace community.

Title: HI-GAN: Hierarchical Inpainting GAN with Auxiliary Inputs for Combined RGB and Depth Inpainting

• Authors: Ankan Dash, Jingyi Gu, Guiling Wang
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.10334
• Pdf URL: https://arxiv.org/pdf/2402.10334
• Copy Paste: [[2402.10334]] HI-GAN: Hierarchical Inpainting GAN with Auxiliary Inputs for Combined RGB and Depth Inpainting(https://arxiv.org/abs/2402.10334)
• Keywords: segmentation
• Abstract: Inpainting involves filling in missing pixels or areas in an image, a crucial technique employed in Mixed Reality environments for various applications, particularly in Diminished Reality (DR) where content is removed from a user's visual environment. Existing methods rely on digital replacement techniques which necessitate multiple cameras and incur high costs. AR devices and smartphones use ToF depth sensors to capture scene depth maps aligned with RGB images. Despite speed and affordability, ToF cameras create imperfect depth maps with missing pixels. To address the above challenges, we propose Hierarchical Inpainting GAN (HI-GAN), a novel approach comprising three GANs in a hierarchical fashion for RGBD inpainting. EdgeGAN and LabelGAN inpaint masked edge and segmentation label images respectively, while CombinedRGBD-GAN combines their latent representation outputs and performs RGB and Depth inpainting. Edge images and particularly segmentation label images as auxiliary inputs significantly enhance inpainting performance by complementary context and hierarchical optimization. We believe we make the first attempt to incorporate label images into inpainting process.Unlike previous approaches requiring multiple sequential models and separate outputs, our work operates in an end-to-end manner, training all three models simultaneously and hierarchically. Specifically, EdgeGAN and LabelGAN are first optimized separately and further optimized inside CombinedRGBD-GAN to enhance inpainting quality. Experiments demonstrate that HI-GAN works seamlessly and achieves overall superior performance compared with existing approaches.

Title: Large Language Models for Forecasting and Anomaly Detection: A Systematic Literature Review

• Authors: Jing Su, Chufeng Jiang, Xin Jin, Yuxin Qiao, Tingsong Xiao, Hongda Ma, Rong Wei, Zhi Jing, Jiajun Xu, Junhong Lin
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.10350
• Pdf URL: https://arxiv.org/pdf/2402.10350
• Copy Paste: [[2402.10350]] Large Language Models for Forecasting and Anomaly Detection: A Systematic Literature Review(https://arxiv.org/abs/2402.10350)
• Keywords: explainability, large language model
• Abstract: This systematic literature review comprehensively examines the application of Large Language Models (LLMs) in forecasting and anomaly detection, highlighting the current state of research, inherent challenges, and prospective future directions. LLMs have demonstrated significant potential in parsing and analyzing extensive datasets to identify patterns, predict future events, and detect anomalous behavior across various domains. However, this review identifies several critical challenges that impede their broader adoption and effectiveness, including the reliance on vast historical datasets, issues with generalizability across different contexts, the phenomenon of model hallucinations, limitations within the models' knowledge boundaries, and the substantial computational resources required. Through detailed analysis, this review discusses potential solutions and strategies to overcome these obstacles, such as integrating multimodal data, advancements in learning methodologies, and emphasizing model explainability and computational efficiency. Moreover, this review outlines critical trends that are likely to shape the evolution of LLMs in these fields, including the push toward real-time processing, the importance of sustainable modeling practices, and the value of interdisciplinary collaboration. Conclusively, this review underscores the transformative impact LLMs could have on forecasting and anomaly detection while emphasizing the need for continuous innovation, ethical considerations, and practical solutions to realize their full potential.

Title: Prompt-Based Bias Calibration for Better Zero/Few-Shot Learning of Language Models

• Authors: Kang He, Yinghan Long, Kaushik Roy
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.10353
• Pdf URL: https://arxiv.org/pdf/2402.10353
• Copy Paste: [[2402.10353]] Prompt-Based Bias Calibration for Better Zero/Few-Shot Learning of Language Models(https://arxiv.org/abs/2402.10353)
• Keywords: fair
• Abstract: Prompt learning is susceptible to intrinsic bias present in pre-trained language models (LMs), resulting in sub-optimal performance of prompt-based zero/few-shot learning. In this work, we propose a null-input prompting method to calibrate intrinsic bias encoded in pre-trained LMs. Different from prior efforts that address intrinsic bias primarily for social fairness and often involve excessive computational cost, our objective is to explore enhancing LMs' performance in downstream zero/few-shot learning while emphasizing the efficiency of intrinsic bias calibration. Specifically, we leverage a diverse set of auto-selected null-meaning inputs generated from GPT-4 to prompt pre-trained LMs for intrinsic bias probing. Utilizing the bias-reflected probability distribution, we formulate a distribution disparity loss for bias calibration, where we exclusively update bias parameters ($0.1\%$ of total parameters) of LMs towards equal probability distribution. Experimental results show that the calibration promotes an equitable starting point for LMs while preserving language modeling abilities. Across a wide range of datasets, including sentiment analysis and topic classification, our method significantly improves zero/few-shot learning performance of LMs for both in-context learning and prompt-based fine-tuning (on average $9\%$ and $2\%$, respectively).

Title: Can we soft prompt LLMs for graph learning tasks?

• Authors: Zheyuan Liu, Xiaoxin He, Yijun Tian, Nitesh V. Chawla
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.10359
• Pdf URL: https://arxiv.org/pdf/2402.10359
• Copy Paste: [[2402.10359]] Can we soft prompt LLMs for graph learning tasks?(https://arxiv.org/abs/2402.10359)
• Keywords: large language model
• Abstract: Graph plays an important role in representing complex relationships in real-world applications such as social networks, biological data and citation networks. In recent years, Large Language Models (LLMs) have achieved tremendous success in various domains, which makes applying LLMs to graphs particularly appealing. However, directly applying LLMs to graph modalities presents unique challenges due to the discrepancy and mismatch between the graph and text modalities. Hence, to further investigate LLMs' potential for comprehending graph information, we introduce GraphPrompter, a novel framework designed to align graph information with LLMs via soft prompts. Specifically, GraphPrompter consists of two main components: a graph neural network to encode complex graph information and an LLM that effectively processes textual information. Comprehensive experiments on various benchmark datasets under node classification and link prediction tasks demonstrate the effectiveness of our proposed method. The GraphPrompter framework unveils the substantial capabilities of LLMs as predictors in graph-related tasks, enabling researchers to utilize LLMs across a spectrum of real-world graph scenarios more effectively.

Title: BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains

• Authors: Yanis Labrak, Adrien Bazoge, Emmanuel Morin, Pierre-Antoine Gourraud, Mickael Rouvier, Richard Dufour
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.10373
• Pdf URL: https://arxiv.org/pdf/2402.10373
• Copy Paste: [[2402.10373]] BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains(https://arxiv.org/abs/2402.10373)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have demonstrated remarkable versatility in recent years, offering potential applications across specialized domains such as healthcare and medicine. Despite the availability of various open-source LLMs tailored for health contexts, adapting general-purpose LLMs to the medical domain presents significant challenges. In this paper, we introduce BioMistral, an open-source LLM tailored for the biomedical domain, utilizing Mistral as its foundation model and further pre-trained on PubMed Central. We conduct a comprehensive evaluation of BioMistral on a benchmark comprising 10 established medical question-answering (QA) tasks in English. We also explore lightweight models obtained through quantization and model merging approaches. Our results demonstrate BioMistral's superior performance compared to existing open-source medical models and its competitive edge against proprietary counterparts. Finally, to address the limited availability of data beyond English and to assess the multilingual generalization of medical LLMs, we automatically translated and evaluated this benchmark into 7 other languages. This marks the first large-scale multilingual evaluation of LLMs in the medical domain. Datasets, multilingual evaluation benchmarks, scripts, and all the models obtained during our experiments are freely released.

Title: Interpreting CLIP with Sparse Linear Concept Embeddings (SpLiCE)

• Authors: Usha Bhalla, Alex Oesterling, Suraj Srinivas, Flavio P. Calmon, Himabindu Lakkaraju
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2402.10376
• Pdf URL: https://arxiv.org/pdf/2402.10376
• Copy Paste: [[2402.10376]] Interpreting CLIP with Sparse Linear Concept Embeddings (SpLiCE)(https://arxiv.org/abs/2402.10376)
• Keywords: interpretability
• Abstract: CLIP embeddings have demonstrated remarkable performance across a wide range of computer vision tasks. However, these high-dimensional, dense vector representations are not easily interpretable, restricting their usefulness in downstream applications that require transparency. In this work, we empirically show that CLIP's latent space is highly structured, and consequently that CLIP representations can be decomposed into their underlying semantic components. We leverage this understanding to propose a novel method, Sparse Linear Concept Embeddings (SpLiCE), for transforming CLIP representations into sparse linear combinations of human-interpretable concepts. Distinct from previous work, SpLiCE does not require concept labels and can be applied post hoc. Through extensive experimentation with multiple real-world datasets, we validate that the representations output by SpLiCE can explain and even replace traditional dense CLIP representations, maintaining equivalent downstream performance while significantly improving their interpretability. We also demonstrate several use cases of SpLiCE representations including detecting spurious correlations, model editing, and quantifying semantic shifts in datasets.

Title: DataDreamer: A Tool for Synthetic Data Generation and Reproducible LLM Workflows

• Authors: Ajay Patel, Colin Raffel, Chris Callison-Burch
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.10379
• Pdf URL: https://arxiv.org/pdf/2402.10379
• Copy Paste: [[2402.10379]] DataDreamer: A Tool for Synthetic Data Generation and Reproducible LLM Workflows(https://arxiv.org/abs/2402.10379)
• Keywords: large language model
• Abstract: Large language models (LLMs) have become a dominant and important tool for NLP researchers in a wide range of tasks. Today, many researchers use LLMs in synthetic data generation, task evaluation, fine-tuning, distillation, and other model-in-the-loop research workflows. However, challenges arise when using these models that stem from their scale, their closed source nature, and the lack of standardized tooling for these new and emerging workflows. The rapid rise to prominence of these models and these unique challenges has had immediate adverse impacts on open science and on the reproducibility of work that uses them. In this paper, we introduce DataDreamer, an open source Python library that allows researchers to write simple code to implement powerful LLM workflows. DataDreamer also helps researchers adhere to best practices that we propose to encourage open science and reproducibility. The library and documentation are available at https://github.com/datadreamer-dev/DataDreamer .

Title: Enabling Zero Trust Security in IoMT Edge Network

• Authors: Maha Ali Allouzi, Javed Khan
• Subjects: cs.CR, cs.CY
• Abstract URL: https://arxiv.org/abs/2402.10389
• Pdf URL: https://arxiv.org/pdf/2402.10389
• Copy Paste: [[2402.10389]] Enabling Zero Trust Security in IoMT Edge Network(https://arxiv.org/abs/2402.10389)
• Keywords: security, privacy, protect
• Abstract: Internet of Medical Things (IoMT) deals with a patient-data-rich segment, which makes security and privacy a severe concern for patients. Therefore, access control is a significant aspect of ensuring trust in the IoMT. However, deploying existing authentication and authorization solutions to the Internet of Medical Things (IoMT) is not straightforward because of highly dynamic and possibly unprotected environments and untrusted supply chain for the IoT devices. In this article, we propose Soter, a Zero-Trust based authentication system for the IoMT. Soter Incorporates trust negotiation mechanisms within the Zero Trust framework to enable dynamic trust establishment. When a user or device seeks access to a resource, initiate a trust negotiation process. During this process, credentials, attributes, and contextual information are exchanged between the requester and the resource owner. Soter defines access rules based on various factors, including user identity, device health, and location. Access is granted or denied based on these conditions.

Title: Assessing the Performance of OpenTitan as Cryptographic Accelerator in Secure Open-Hardware System-on-Chips

• Authors: Emanuele Parisi, Alberto Musa, Maicol Ciani, Francesco Barchi, Davide Rossi, Andrea Bartolini, Andrea Acquaviva
• Subjects: cs.CR, cs.PF
• Abstract URL: https://arxiv.org/abs/2402.10395
• Pdf URL: https://arxiv.org/pdf/2402.10395
• Copy Paste: [[2402.10395]] Assessing the Performance of OpenTitan as Cryptographic Accelerator in Secure Open-Hardware System-on-Chips(https://arxiv.org/abs/2402.10395)
• Keywords: secure, security
• Abstract: RISC-V open-source systems are emerging in deployment scenarios where safety and security are critical. OpenTitan is an open-source silicon root-of-trust designed to be deployed in a wide range of systems, from high-end to deeply embedded secure environments. Despite the availability of various cryptographic hardware accelerators that make OpenTitan suitable for offloading cryptographic workloads from the main processor, there has been no accurate and quantitative establishment of the benefits derived from using OpenTitan as a secure accelerator. This paper addresses this gap by thoroughly analysing strengths and inefficiencies when offloading cryptographic workloads to OpenTitan. The focus is on three key IPs - HMAC, AES, and OpenTitan Big Number accelerator (OTBN) - which can accelerate four security workloads: Secure Hash Functions, Message Authentication Codes, Symmetric cryptography, and Asymmetric cryptography. For every workload, we develop a bare-metal driver for the OpenTitan accelerator and analyze its efficiency when computation is offloaded from a RISC-V application core within a System-on-Chip designed for secure Cyber-Physical Systems applications. Finally, we assess it against a software implementation on the application core. The characterization was conducted on a cycle-accurate RTL simulator of the System-on-Chip (SoC). Our study demonstrates that OpenTitan significantly outperforms software implementations, with speedups ranging from 4.3x to 12.5x. However, there is potential for even greater gains as the current OpenTitan utilizes a fraction of the accelerator bandwidths, which ranges from 16% to 61%, depending on the memory being accessed and the accelerator used. Our results open the way to the optimization of OpenTitan-based secure platforms, providing design guidelines to unlock the full potential of its accelerators in secure applications.

Title: Chain of Logic: Rule-Based Reasoning with Large Language Models

• Authors: Sergio Servantez, Joe Barrow, Kristian Hammond, Rajiv Jain
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.10400
• Pdf URL: https://arxiv.org/pdf/2402.10400
• Copy Paste: [[2402.10400]] Chain of Logic: Rule-Based Reasoning with Large Language Models(https://arxiv.org/abs/2402.10400)
• Keywords: large language model
• Abstract: Rule-based reasoning, a fundamental type of legal reasoning, enables us to draw conclusions by accurately applying a rule to a set of facts. We explore causal language models as rule-based reasoners, specifically with respect to compositional rules - rules consisting of multiple elements which form a complex logical expression. Reasoning about compositional rules is challenging because it requires multiple reasoning steps, and attending to the logical relationships between elements. We introduce a new prompting method, Chain of Logic, which elicits rule-based reasoning through decomposition (solving elements as independent threads of logic), and recomposition (recombining these sub-answers to resolve the underlying logical expression). This method was inspired by the IRAC (Issue, Rule, Application, Conclusion) framework, a sequential reasoning approach used by lawyers. We evaluate chain of logic across eight rule-based reasoning tasks involving three distinct compositional rules from the LegalBench benchmark and demonstrate it consistently outperforms other prompting methods, including chain of thought and self-ask, using open-source and commercial language models.

Title: ManiFPT: Defining and Analyzing Fingerprints of Generative Models

• Authors: Hae Jin Song, Mahyar Khayatkhoei, Wael AbdAlmageed
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2402.10401
• Pdf URL: https://arxiv.org/pdf/2402.10401
• Copy Paste: [[2402.10401]] ManiFPT: Defining and Analyzing Fingerprints of Generative Models(https://arxiv.org/abs/2402.10401)
• Keywords: generative
• Abstract: Recent works have shown that generative models leave traces of their underlying generative process on the generated samples, broadly referred to as fingerprints of a generative model, and have studied their utility in detecting synthetic images from real ones. However, the extend to which these fingerprints can distinguish between various types of synthetic image and help identify the underlying generative process remain under-explored. In particular, the very definition of a fingerprint remains unclear, to our knowledge. To that end, in this work, we formalize the definition of artifact and fingerprint in generative models, propose an algorithm for computing them in practice, and finally study its effectiveness in distinguishing a large array of different generative models. We find that using our proposed definition can significantly improve the performance on the task of identifying the underlying generative process from samples (model attribution) compared to existing methods. Additionally, we study the structure of the fingerprints, and observe that it is very predictive of the effect of different design choices on the generative process.

Title: Polyhedral Complex Derivation from Piecewise Trilinear Networks

• Authors: Jin-Hwa Kim
• Subjects: cs.LG, cs.AI, cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2402.10403
• Pdf URL: https://arxiv.org/pdf/2402.10403
• Copy Paste: [[2402.10403]] Polyhedral Complex Derivation from Piecewise Trilinear Networks(https://arxiv.org/abs/2402.10403)
• Keywords: extraction
• Abstract: Recent advancements in visualizing deep neural networks provide insights into their structures and mesh extraction from Continuous Piecewise Affine (CPWA) functions. Meanwhile, developments in neural surface representation learning incorporate non-linear positional encoding, addressing issues like spectral bias; however, this poses challenges in applying mesh extraction techniques based on CPWA functions. Focusing on trilinear interpolating methods as positional encoding, we present theoretical insights and an analytical mesh extraction, showing the transformation of hypersurfaces to flat planes within the trilinear region under the eikonal constraint. Moreover, we introduce a method for approximating intersecting points among three hypersurfaces contributing to broader applications. We empirically validate correctness and parsimony through chamfer distance and efficiency, and angular distance, while examining the correlation between the eikonal loss and the planarity of the hypersurfaces.

Title: Explaining generative diffusion models via visual analysis for interpretable decision-making process

• Authors: Ji-Hoon Park, Yeong-Joon Ju, Seong-Whan Lee
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.10404
• Pdf URL: https://arxiv.org/pdf/2402.10404
• Copy Paste: [[2402.10404]] Explaining generative diffusion models via visual analysis for interpretable decision-making process(https://arxiv.org/abs/2402.10404)
• Keywords: diffusion, generative
• Abstract: Diffusion models have demonstrated remarkable performance in generation tasks. Nevertheless, explaining the diffusion process remains challenging due to it being a sequence of denoising noisy images that are difficult for experts to interpret. To address this issue, we propose the three research questions to interpret the diffusion process from the perspective of the visual concepts generated by the model and the region where the model attends in each time step. We devise tools for visualizing the diffusion process and answering the aforementioned research questions to render the diffusion process human-understandable. We show how the output is progressively generated in the diffusion process by explaining the level of denoising and highlighting relationships to foundational visual concepts at each time step through the results of experiments with various visual analyses using the tools. Throughout the training of the diffusion model, the model learns diverse visual concepts corresponding to each time-step, enabling the model to predict varying levels of visual concepts at different stages. We substantiate our tools using Area Under Cover (AUC) score, correlation quantification, and cross-attention mapping. Our findings provide insights into the diffusion process and pave the way for further research into explainable diffusion mechanisms.

Title: Understanding Survey Paper Taxonomy about Large Language Models via Graph Representation Learning

• Authors: Jun Zhuang, Casey Kennington
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.10409
• Pdf URL: https://arxiv.org/pdf/2402.10409
• Copy Paste: [[2402.10409]] Understanding Survey Paper Taxonomy about Large Language Models via Graph Representation Learning(https://arxiv.org/abs/2402.10409)
• Keywords: large language model
• Abstract: As new research on Large Language Models (LLMs) continues, it is difficult to keep up with new research and models. To help researchers synthesize the new research many have written survey papers, but even those have become numerous. In this paper, we develop a method to automatically assign survey papers to a taxonomy. We collect the metadata of 144 LLM survey papers and explore three paradigms to classify papers within the taxonomy. Our work indicates that leveraging graph structure information on co-category graphs can significantly outperform the language models in two paradigms; pre-trained language models' fine-tuning and zero-shot/few-shot classifications using LLMs. We find that our model surpasses an average human recognition level and that fine-tuning LLMs using weak labels generated by a smaller model, such as the GCN in this study, can be more effective than using ground-truth labels, revealing the potential of weak-to-strong generalization in the taxonomy classification task.

Title: Connect the dots: Dataset Condensation, Differential Privacy, and Adversarial Uncertainty

• Authors: Kenneth Odoh
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.10423
• Pdf URL: https://arxiv.org/pdf/2402.10423
• Copy Paste: [[2402.10423]] Connect the dots: Dataset Condensation, Differential Privacy, and Adversarial Uncertainty(https://arxiv.org/abs/2402.10423)
• Keywords: privacy
• Abstract: Our work focuses on understanding the underpinning mechanism of dataset condensation by drawing connections with ($\epsilon$, $\delta$)-differential privacy where the optimal noise, $\epsilon$, is chosen by adversarial uncertainty \cite{Grining2017}. We can answer the question about the inner workings of the dataset condensation procedure. Previous work \cite{dong2022} proved the link between dataset condensation (DC) and ($\epsilon$, $\delta$)-differential privacy. However, it is unclear from existing works on ablating DC to obtain a lower-bound estimate of $\epsilon$ that will suffice for creating high-fidelity synthetic data. We suggest that adversarial uncertainty is the most appropriate method to achieve an optimal noise level, $\epsilon$. As part of the internal dynamics of dataset condensation, we adopt a satisfactory scheme for noise estimation that guarantees high-fidelity data while providing privacy.

Title: DELL: Generating Reactions and Explanations for LLM-Based Misinformation Detection

• Authors: Herun Wan, Shangbin Feng, Zhaoxuan Tan, Heng Wang, Yulia Tsvetkov, Minnan Luo
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.10426
• Pdf URL: https://arxiv.org/pdf/2402.10426
• Copy Paste: [[2402.10426]] DELL: Generating Reactions and Explanations for LLM-Based Misinformation Detection(https://arxiv.org/abs/2402.10426)
• Keywords: large language model
• Abstract: Large language models are limited by challenges in factuality and hallucinations to be directly employed off-the-shelf for judging the veracity of news articles, where factual accuracy is paramount. In this work, we propose DELL that identifies three key stages in misinformation detection where LLMs could be incorporated as part of the pipeline: 1) LLMs could \emph{generate news reactions} to represent diverse perspectives and simulate user-news interaction networks; 2) LLMs could \emph{generate explanations} for proxy tasks (e.g., sentiment, stance) to enrich the contexts of news articles and produce experts specializing in various aspects of news understanding; 3) LLMs could \emph{merge task-specific experts} and provide an overall prediction by incorporating the predictions and confidence scores of varying experts. Extensive experiments on seven datasets with three LLMs demonstrate that DELL outperforms state-of-the-art baselines by up to 16.8\% in macro f1-score. Further analysis reveals that the generated reactions and explanations are greatly helpful in misinformation detection, while our proposed LLM-guided expert merging helps produce better-calibrated predictions.

Title: Parametric Augmentation for Time Series Contrastive Learning

• Authors: Xu Zheng, Tianchun Wang, Wei Cheng, Aitian Ma, Haifeng Chen, Mo Sha, Dongsheng Luo
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.10434
• Pdf URL: https://arxiv.org/pdf/2402.10434
• Copy Paste: [[2402.10434]] Parametric Augmentation for Time Series Contrastive Learning(https://arxiv.org/abs/2402.10434)
• Keywords: robust
• Abstract: Modern techniques like contrastive learning have been effectively used in many areas, including computer vision, natural language processing, and graph-structured data. Creating positive examples that assist the model in learning robust and discriminative representations is a crucial stage in contrastive learning approaches. Usually, preset human intuition directs the selection of relevant data augmentations. Due to patterns that are easily recognized by humans, this rule of thumb works well in the vision and language domains. However, it is impractical to visually inspect the temporal structures in time series. The diversity of time series augmentations at both the dataset and instance levels makes it difficult to choose meaningful augmentations on the fly. In this study, we address this gap by analyzing time series data augmentation using information theory and summarizing the most commonly adopted augmentations in a unified format. We then propose a contrastive learning framework with parametric augmentation, AutoTCL, which can be adaptively employed to support time series representation learning. The proposed approach is encoder-agnostic, allowing it to be seamlessly integrated with different backbone encoders. Experiments on univariate forecasting tasks demonstrate the highly competitive results of our method, with an average 6.5\% reduction in MSE and 4.7\% in MAE over the leading baselines. In classification tasks, AutoTCL achieves a $1.2\%$ increase in average accuracy.

Title: Dynamic Patch-aware Enrichment Transformer for Occluded Person Re-Identification

• Authors: Xin Zhang, Keren Fu, Qijun Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.10435
• Pdf URL: https://arxiv.org/pdf/2402.10435
• Copy Paste: [[2402.10435]] Dynamic Patch-aware Enrichment Transformer for Occluded Person Re-Identification(https://arxiv.org/abs/2402.10435)
• Keywords: transformer
• Abstract: Person re-identification (re-ID) continues to pose a significant challenge, particularly in scenarios involving occlusions. Prior approaches aimed at tackling occlusions have predominantly focused on aligning physical body features through the utilization of external semantic cues. However, these methods tend to be intricate and susceptible to noise. To address the aforementioned challenges, we present an innovative end-to-end solution known as the Dynamic Patch-aware Enrichment Transformer (DPEFormer). This model effectively distinguishes human body information from occlusions automatically and dynamically, eliminating the need for external detectors or precise image alignment. Specifically, we introduce a dynamic patch token selection module (DPSM). DPSM utilizes a label-guided proxy token as an intermediary to identify informative occlusion-free tokens. These tokens are then selected for deriving subsequent local part features. To facilitate the seamless integration of global classification features with the finely detailed local features selected by DPSM, we introduce a novel feature blending module (FBM). FBM enhances feature representation through the complementary nature of information and the exploitation of part diversity. Furthermore, to ensure that DPSM and the entire DPEFormer can effectively learn with only identity labels, we also propose a Realistic Occlusion Augmentation (ROA) strategy. This strategy leverages the recent advances in the Segment Anything Model (SAM). As a result, it generates occlusion images that closely resemble real-world occlusions, greatly enhancing the subsequent contrastive learning process. Experiments on occluded and holistic re-ID benchmarks signify a substantial advancement of DPEFormer over existing state-of-the-art approaches. The code will be made publicly available.

Title: I Am Not Them: Fluid Identities and Persistent Out-group Bias in Large Language Models

• Authors: Wenchao Dong, Assem Zhunis, Hyojin Chin, Jiyoung Han, Meeyoung Cha
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.10436
• Pdf URL: https://arxiv.org/pdf/2402.10436
• Copy Paste: [[2402.10436]] I Am Not Them: Fluid Identities and Persistent Out-group Bias in Large Language Models(https://arxiv.org/abs/2402.10436)
• Keywords: robust, large language model
• Abstract: We explored cultural biases-individualism vs. collectivism-in ChatGPT across three Western languages (i.e., English, German, and French) and three Eastern languages (i.e., Chinese, Japanese, and Korean). When ChatGPT adopted an individualistic persona in Western languages, its collectivism scores (i.e., out-group values) exhibited a more negative trend, surpassing their positive orientation towards individualism (i.e., in-group values). Conversely, when a collectivistic persona was assigned to ChatGPT in Eastern languages, a similar pattern emerged with more negative responses toward individualism (i.e., out-group values) as compared to collectivism (i.e., in-group values). The results indicate that when imbued with a particular social identity, ChatGPT discerns in-group and out-group, embracing in-group values while eschewing out-group values. Notably, the negativity towards the out-group, from which prejudices and discrimination arise, exceeded the positivity towards the in-group. The experiment was replicated in the political domain, and the results remained consistent. Furthermore, this replication unveiled an intrinsic Democratic bias in Large Language Models (LLMs), aligning with earlier findings and providing integral insights into mitigating such bias through prompt engineering. Extensive robustness checks were performed using varying hyperparameter and persona setup methods, with or without social identity labels, across other popular language models.

Title: Steering Conversational Large Language Models for Long Emotional Support Conversations

• Authors: Navid Madani, Sougata Saha, Rohini Srihari
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.10453
• Pdf URL: https://arxiv.org/pdf/2402.10453
• Copy Paste: [[2402.10453]] Steering Conversational Large Language Models for Long Emotional Support Conversations(https://arxiv.org/abs/2402.10453)
• Keywords: large language model
• Abstract: In this study, we address the challenge of consistently following emotional support strategies in long conversations by large language models (LLMs). We introduce the Strategy-Relevant Attention (SRA) metric, a model-agnostic measure designed to evaluate the effectiveness of LLMs in adhering to strategic prompts in emotional support contexts. By analyzing conversations within the Emotional Support Conversations dataset (ESConv) using LLaMA models, we demonstrate that SRA is significantly correlated with a model's ability to sustain the outlined strategy throughout the interactions. Our findings reveal that the application of SRA-informed prompts leads to enhanced strategic adherence, resulting in conversations that more reliably exhibit the desired emotional support strategies over longer conversations. Furthermore, we contribute a comprehensive, multi-branch synthetic conversation dataset for ESConv, featuring a variety of strategy continuations informed by our optimized prompting method. The code and data are publicly available on our Github.

Title: Optimizing Skin Lesion Classification via Multimodal Data and Auxiliary Task Integration

• Authors: Mahapara Khurshid, Mayank Vatsa, Richa Singh
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.10454
• Pdf URL: https://arxiv.org/pdf/2402.10454
• Copy Paste: [[2402.10454]] Optimizing Skin Lesion Classification via Multimodal Data and Auxiliary Task Integration(https://arxiv.org/abs/2402.10454)
• Keywords: extraction
• Abstract: The rising global prevalence of skin conditions, some of which can escalate to life-threatening stages if not timely diagnosed and treated, presents a significant healthcare challenge. This issue is particularly acute in remote areas where limited access to healthcare often results in delayed treatment, allowing skin diseases to advance to more critical stages. One of the primary challenges in diagnosing skin diseases is their low inter-class variations, as many exhibit similar visual characteristics, making accurate classification challenging. This research introduces a novel multimodal method for classifying skin lesions, integrating smartphone-captured images with essential clinical and demographic information. This approach mimics the diagnostic process employed by medical professionals. A distinctive aspect of this method is the integration of an auxiliary task focused on super-resolution image prediction. This component plays a crucial role in refining visual details and enhancing feature extraction, leading to improved differentiation between classes and, consequently, elevating the overall effectiveness of the model. The experimental evaluations have been conducted using the PAD-UFES20 dataset, applying various deep-learning architectures. The results of these experiments not only demonstrate the effectiveness of the proposed method but also its potential applicability under-resourced healthcare environments.

Title: QDyLoRA: Quantized Dynamic Low-Rank Adaptation for Efficient Large Language Model Tuning

• Authors: Hossein Rajabzadeh, Mojtaba Valipour, Tianshu Zhu, Marzieh Tahaei, Hyock Ju Kwon, Ali Ghodsi, Boxing Chen, Mehdi Rezagholizadeh
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.10462
• Pdf URL: https://arxiv.org/pdf/2402.10462
• Copy Paste: [[2402.10462]] QDyLoRA: Quantized Dynamic Low-Rank Adaptation for Efficient Large Language Model Tuning(https://arxiv.org/abs/2402.10462)
• Keywords: large language model
• Abstract: Finetuning large language models requires huge GPU memory, restricting the choice to acquire Larger models. While the quantized version of the Low-Rank Adaptation technique, named QLoRA, significantly alleviates this issue, finding the efficient LoRA rank is still challenging. Moreover, QLoRA is trained on a pre-defined rank and, therefore, cannot be reconfigured for its lower ranks without requiring further fine-tuning steps. This paper proposes QDyLoRA -Quantized Dynamic Low-Rank Adaptation-, as an efficient quantization approach for dynamic low-rank adaptation. Motivated by Dynamic LoRA, QDyLoRA is able to efficiently finetune LLMs on a set of pre-defined LoRA ranks. QDyLoRA enables fine-tuning Falcon-40b for ranks 1 to 64 on a single 32 GB V100-GPU through one round of fine-tuning. Experimental results show that QDyLoRA is competitive to QLoRA and outperforms when employing its optimal rank.

Title: FedKit: Enabling Cross-Platform Federated Learning for Android and iOS

• Authors: Sichang He, Beilong Tang, Boyan Zhang, Jiaoqi Shao, Xiaomin Ouyang, Daniel Nata Nugraha, Bing Luo
• Subjects: cs.LG, cs.NI
• Abstract URL: https://arxiv.org/abs/2402.10464
• Pdf URL: https://arxiv.org/pdf/2402.10464
• Copy Paste: [[2402.10464]] FedKit: Enabling Cross-Platform Federated Learning for Android and iOS(https://arxiv.org/abs/2402.10464)
• Keywords: federate
• Abstract: We present FedKit, a federated learning (FL) system tailored for cross-platform FL research on Android and iOS devices. FedKit pipelines cross-platform FL development by enabling model conversion, hardware-accelerated training, and cross-platform model aggregation. Our FL workflow supports flexible machine learning operations (MLOps) in production, facilitating continuous model delivery and training. We have deployed FedKit in a real-world use case for health data analysis on university campuses, demonstrating its effectiveness. FedKit is open-source at https://github.com/FedCampus/FedKit.

Title: Large Language Models as Zero-shot Dialogue State Tracker through Function Calling

• Authors: Zekun Li, Zhiyu Zoey Chen, Mike Ross, Patrick Huber, Seungwhan Moon, Zhaojiang Lin, Xin Luna Dong, Adithya Sagar, Xifeng Yan, Paul A. Crook
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.10466
• Pdf URL: https://arxiv.org/pdf/2402.10466
• Copy Paste: [[2402.10466]] Large Language Models as Zero-shot Dialogue State Tracker through Function Calling(https://arxiv.org/abs/2402.10466)
• Keywords: generative, large language model
• Abstract: Large language models (LLMs) are increasingly prevalent in conversational systems due to their advanced understanding and generative capabilities in general contexts. However, their effectiveness in task-oriented dialogues (TOD), which requires not only response generation but also effective dialogue state tracking (DST) within specific tasks and domains, remains less satisfying. In this work, we propose a novel approach FnCTOD for solving DST with LLMs through function calling. This method improves zero-shot DST, allowing adaptation to diverse domains without extensive data collection or model tuning. Our experimental results demonstrate that our approach achieves exceptional performance with both modestly sized open-source and also proprietary LLMs: with in-context prompting it enables various 7B or 13B parameter models to surpass the previous state-of-the-art (SOTA) achieved by ChatGPT, and improves ChatGPT's performance beating the SOTA by 5.6% Avg. JGA. Individual model results for GPT-3.5 and GPT-4 are boosted by 4.8% and 14%, respectively. We also show that by fine-tuning on a small collection of diverse task-oriented dialogues, we can equip modestly sized models, specifically a 13B parameter LLaMA2-Chat model, with function-calling capabilities and DST performance comparable to ChatGPT while maintaining their chat capabilities. We plan to open-source experimental code and model.

Title: Privacy for Fairness: Information Obfuscation for Fair Representation Learning with Local Differential Privacy

• Authors: Songjie Xie, Youlong Wu, Jiaxuan Li, Ming Ding, Khaled B. Letaief
• Subjects: cs.LG, cs.CR, cs.IT
• Abstract URL: https://arxiv.org/abs/2402.10473
• Pdf URL: https://arxiv.org/pdf/2402.10473
• Copy Paste: [[2402.10473]] Privacy for Fairness: Information Obfuscation for Fair Representation Learning with Local Differential Privacy(https://arxiv.org/abs/2402.10473)
• Keywords: privacy, protect, fair
• Abstract: As machine learning (ML) becomes more prevalent in human-centric applications, there is a growing emphasis on algorithmic fairness and privacy protection. While previous research has explored these areas as separate objectives, there is a growing recognition of the complex relationship between privacy and fairness. However, previous works have primarily focused on examining the interplay between privacy and fairness through empirical investigations, with limited attention given to theoretical exploration. This study aims to bridge this gap by introducing a theoretical framework that enables a comprehensive examination of their interrelation. We shall develop and analyze an information bottleneck (IB) based information obfuscation method with local differential privacy (LDP) for fair representation learning. In contrast to many empirical studies on fairness in ML, we show that the incorporation of LDP randomizers during the encoding process can enhance the fairness of the learned representation. Our analysis will demonstrate that the disclosure of sensitive information is constrained by the privacy budget of the LDP randomizer, thereby enabling the optimization process within the IB framework to effectively suppress sensitive information while preserving the desired utility through obfuscation. Based on the proposed method, we further develop a variational representation encoding approach that simultaneously achieves fairness and LDP. Our variational encoding approach offers practical advantages. It is trained using a non-adversarial method and does not require the introduction of any variational prior. Extensive experiments will be presented to validate our theoretical results and demonstrate the ability of our proposed approach to achieve both LDP and fairness while preserving adequate utility.

Title: Spike-EVPR: Deep Spiking Residual Network with Cross-Representation Aggregation for Event-Based Visual Place Recognition

• Authors: Chenming Hu, Zheng Fang, Kuanxu Hou, Delei Kong, Junjie Jiang, Hao Zhuang, Mingyuan Sun, Xinjie Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.10476
• Pdf URL: https://arxiv.org/pdf/2402.10476
• Copy Paste: [[2402.10476]] Spike-EVPR: Deep Spiking Residual Network with Cross-Representation Aggregation for Event-Based Visual Place Recognition(https://arxiv.org/abs/2402.10476)
• Keywords: robust
• Abstract: Event cameras have been successfully applied to visual place recognition (VPR) tasks by using deep artificial neural networks (ANNs) in recent years. However, previously proposed deep ANN architectures are often unable to harness the abundant temporal information presented in event streams. In contrast, deep spiking networks exhibit more intricate spatiotemporal dynamics and are inherently well-suited to process sparse asynchronous event streams. Unfortunately, directly inputting temporal-dense event volumes into the spiking network introduces excessive time steps, resulting in prohibitively high training costs for large-scale VPR tasks. To address the aforementioned issues, we propose a novel deep spiking network architecture called Spike-EVPR for event-based VPR tasks. First, we introduce two novel event representations tailored for SNN to fully exploit the spatio-temporal information from the event streams, and reduce the video memory occupation during training as much as possible. Then, to exploit the full potential of these two representations, we construct a Bifurcated Spike Residual Encoder (BSR-Encoder) with powerful representational capabilities to better extract the high-level features from the two event representations. Next, we introduce a Shared & Specific Descriptor Extractor (SSD-Extractor). This module is designed to extract features shared between the two representations and features specific to each. Finally, we propose a Cross-Descriptor Aggregation Module (CDA-Module) that fuses the above three features to generate a refined, robust global descriptor of the scene. Our experimental results indicate the superior performance of our Spike-EVPR compared to several existing EVPR pipelines on Brisbane-Event-VPR and DDD20 datasets, with the average Recall@1 increased by 7.61% on Brisbane and 13.20% on DDD20.

Title: Understanding Likelihood of Normalizing Flow and Image Complexity through the Lens of Out-of-Distribution Detection

• Authors: Genki Osada, Tsubasa Takahashi, Takashi Nishide
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.10477
• Pdf URL: https://arxiv.org/pdf/2402.10477
• Copy Paste: [[2402.10477]] Understanding Likelihood of Normalizing Flow and Image Complexity through the Lens of Out-of-Distribution Detection(https://arxiv.org/abs/2402.10477)
• Keywords: generative
• Abstract: Out-of-distribution (OOD) detection is crucial to safety-critical machine learning applications and has been extensively studied. While recent studies have predominantly focused on classifier-based methods, research on deep generative model (DGM)-based methods have lagged relatively. This disparity may be attributed to a perplexing phenomenon: DGMs often assign higher likelihoods to unknown OOD inputs than to their known training data. This paper focuses on explaining the underlying mechanism of this phenomenon. We propose a hypothesis that less complex images concentrate in high-density regions in the latent space, resulting in a higher likelihood assignment in the Normalizing Flow (NF). We experimentally demonstrate its validity for five NF architectures, concluding that their likelihood is untrustworthy. Additionally, we show that this problem can be alleviated by treating image complexity as an independent variable. Finally, we provide evidence of the potential applicability of our hypothesis in another DGM, PixelCNN++.

Title: Make a Cheap Scaling: A Self-Cascade Diffusion Model for Higher-Resolution Adaptation

• Authors: Lanqing Guo, Yingqing He, Haoxin Chen, Menghan Xia, Xiaodong Cun, Yufei Wang, Siyu Huang, Yong Zhang, Xintao Wang, Qifeng Chen, Ying Shan, Bihan Wen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.10491
• Pdf URL: https://arxiv.org/pdf/2402.10491
• Copy Paste: [[2402.10491]] Make a Cheap Scaling: A Self-Cascade Diffusion Model for Higher-Resolution Adaptation(https://arxiv.org/abs/2402.10491)
• Keywords: diffusion
• Abstract: Diffusion models have proven to be highly effective in image and video generation; however, they still face composition challenges when generating images of varying sizes due to single-scale training data. Adapting large pre-trained diffusion models for higher resolution demands substantial computational and optimization resources, yet achieving a generation capability comparable to low-resolution models remains elusive. This paper proposes a novel self-cascade diffusion model that leverages the rich knowledge gained from a well-trained low-resolution model for rapid adaptation to higher-resolution image and video generation, employing either tuning-free or cheap upsampler tuning paradigms. Integrating a sequence of multi-scale upsampler modules, the self-cascade diffusion model can efficiently adapt to a higher resolution, preserving the original composition and generation capabilities. We further propose a pivot-guided noise re-schedule strategy to speed up the inference process and improve local structural details. Compared to full fine-tuning, our approach achieves a 5X training speed-up and requires only an additional 0.002M tuning parameters. Extensive experiments demonstrate that our approach can quickly adapt to higher resolution image and video synthesis by fine-tuning for just 10k steps, with virtually no additional inference time.

Title: Comparing Hallucination Detection Metrics for Multilingual Generation

• Authors: Haoqiang Kang, Terra Blevins, Luke Zettlemoyer
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.10496
• Pdf URL: https://arxiv.org/pdf/2402.10496
• Copy Paste: [[2402.10496]] Comparing Hallucination Detection Metrics for Multilingual Generation(https://arxiv.org/abs/2402.10496)
• Keywords: robust
• Abstract: While many automatic hallucination detection techniques have been proposed for English texts, their effectiveness in multilingual contexts remains unexplored. This paper aims to bridge the gap in understanding how these hallucination detection metrics perform on non-English languages. We evaluate the efficacy of various detection metrics, including lexical metrics like ROUGE and Named Entity Overlap and Natural Language Inference (NLI)-based metrics, at detecting hallucinations in biographical summaries in many languages; we also evaluate how correlated these different metrics are to gauge whether they measure the same phenomena. Our empirical analysis reveals that while lexical metrics show limited effectiveness, NLI-based metrics perform well in high-resource languages at the sentence level. In contrast, NLI-based metrics often fail to detect atomic fact hallucinations. Our findings highlight existing gaps in multilingual hallucination detection and motivate future research to develop more robust detection methods for LLM hallucination in other languages.

Title: Provably Sample Efficient RLHF via Active Preference Optimization

• Authors: Nirjhar Das, Souradip Chakraborty, Aldo Pacchiano, Sayak Ray Chowdhury
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.10500
• Pdf URL: https://arxiv.org/pdf/2402.10500
• Copy Paste: [[2402.10500]] Provably Sample Efficient RLHF via Active Preference Optimization(https://arxiv.org/abs/2402.10500)
• Keywords: generative, large language model
• Abstract: Reinforcement Learning from Human Feedback (RLHF) is pivotal in aligning Large Language Models (LLMs) with human preferences. While these aligned generative models have demonstrated impressive capabilities across various tasks, the dependence on high-quality human preference data poses a costly bottleneck in practical implementation of RLHF. Hence better and adaptive strategies for data collection is needed. To this end, we frame RLHF as a contextual preference bandit problem with prompts as contexts and show that the naive way of collecting preference data by choosing prompts uniformly at random leads to a policy that suffers an $\Omega(1)$ suboptimality gap in rewards. Then we propose $\textit{Active Preference Optimization}$ ($\texttt{APO}$), an algorithm that actively selects prompts to collect preference data. Under the Bradley-Terry-Luce (BTL) preference model, \texttt{APO} achieves sample efficiency without compromising on policy performance. We show that given a sample budget of $T$, the suboptimality gap of a policy learned via $\texttt{APO}$ scales as $O(1/\sqrt{T})$. Next, we propose a compute-efficient batch version of $\texttt{APO}$ with minor modification and evaluate its performance in practice. Experimental evaluations on a human preference dataset validate \texttt{APO}'s efficacy as a sample-efficient and practical solution to data collection for RLHF, facilitating alignment of LLMs with human preferences in a cost-effective and scalable manner.

Title: Can Transformers Predict Vibrations?

• Authors: Fusataka Kuniyoshi, Yoshihide Sawada
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.10511
• Pdf URL: https://arxiv.org/pdf/2402.10511
• Copy Paste: [[2402.10511]] Can Transformers Predict Vibrations?(https://arxiv.org/abs/2402.10511)
• Keywords: transformer
• Abstract: Highly accurate time-series vibration prediction is an important research issue for electric vehicles (EVs). EVs often experience vibrations when driving on rough terrains, known as torsional resonance. This resonance, caused by the interaction between motor and tire vibrations, puts excessive loads on the vehicle's drive shaft. However, current damping technologies only detect resonance after the vibration amplitude of the drive shaft torque reaches a certain threshold, leading to significant loads on the shaft at the time of detection. In this study, we propose a novel approach to address this issue by introducing Resoformer, a transformer-based model for predicting torsional resonance. Resoformer utilizes time-series of the motor rotation speed as input and predicts the amplitude of torsional vibration at a specified quantile occurring in the shaft after the input series. By calculating the attention between recursive and convolutional features extracted from the measured data points, Resoformer improves the accuracy of vibration forecasting. To evaluate the model, we use a vibration dataset called VIBES (Dataset for Forecasting Vibration Transition in EVs), consisting of 2,600 simulator-generated vibration sequences. Our experiments, conducted on strong baselines built on the VIBES dataset, demonstrate that Resoformer achieves state-of-the-art results. In conclusion, our study answers the question "Can Transformers Forecast Vibrations?" While traditional transformer architectures show low performance in forecasting torsional resonance waves, our findings indicate that combining recurrent neural network and temporal convolutional network using the transformer architecture improves the accuracy of long-term vibration forecasting.

Title: Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMs

• Authors: Yeonhong Park, Jake Hyun, SangLyul Cho, Bonggeun Sim, Jae W. Lee
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.10517
• Pdf URL: https://arxiv.org/pdf/2402.10517
• Copy Paste: [[2402.10517]] Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMs(https://arxiv.org/abs/2402.10517)
• Keywords: large language model
• Abstract: Recently, considerable efforts have been directed towards compressing Large Language Models (LLMs), which showcase groundbreaking capabilities across diverse applications but entail significant deployment costs due to their large sizes. Meanwhile, much less attention has been given to mitigating the costs associated with deploying multiple LLMs of varying sizes despite its practical significance. Thus, this paper introduces \emph{any-precision LLM}, extending the concept of any-precision DNN to LLMs. Addressing challenges in any-precision LLM, we propose a lightweight method for any-precision quantization of LLMs, leveraging a post-training quantization framework, and develop a specialized software engine for its efficient serving. As a result, our solution significantly reduces the high costs of deploying multiple, different-sized LLMs by overlaying LLMs quantized to varying bit-widths, such as 3, 4, ..., $n$ bits, into a memory footprint comparable to a single $n$-bit LLM. All the supported LLMs with varying bit-widths demonstrate state-of-the-art model quality and inference throughput, proving itself to be a compelling option for deployment of multiple, different-sized LLMs. The source code will be publicly available soon.

Title: Zero-shot sampling of adversarial entities in biomedical question answering

• Authors: R. Patrick Xian, Alex J. Lee, Vincent Wang, Qiming Cui, Russell Ro, Reza Abbasi-Asl
• Subjects: cs.CL, cs.CR, stat.AP
• Abstract URL: https://arxiv.org/abs/2402.10527
• Pdf URL: https://arxiv.org/pdf/2402.10527
• Copy Paste: [[2402.10527]] Zero-shot sampling of adversarial entities in biomedical question answering(https://arxiv.org/abs/2402.10527)
• Keywords: attack, large language model
• Abstract: The increasing depth of parametric domain knowledge in large language models (LLMs) is fueling their rapid deployment in real-world applications. In high-stakes and knowledge-intensive tasks, understanding model vulnerabilities is essential for quantifying the trustworthiness of model predictions and regulating their use. The recent discovery of named entities as adversarial examples in natural language processing tasks raises questions about their potential guises in other settings. Here, we propose a powerscaled distance-weighted sampling scheme in embedding space to discover diverse adversarial entities as distractors. We demonstrate its advantage over random sampling in adversarial question answering on biomedical topics. Our approach enables the exploration of different regions on the attack surface, which reveals two regimes of adversarial entities that markedly differ in their characteristics. Moreover, we show that the attacks successfully manipulate token-wise Shapley value explanations, which become deceptive in the adversarial setting. Our investigations illustrate the brittleness of domain knowledge in LLMs and reveal a shortcoming of standard evaluations for high-capacity models.

Title: Can We Verify Step by Step for Incorrect Answer Detection?

• Authors: Xin Xu, Shizhe Diao, Can Yang, Yang Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.10528
• Pdf URL: https://arxiv.org/pdf/2402.10528
• Copy Paste: [[2402.10528]] Can We Verify Step by Step for Incorrect Answer Detection?(https://arxiv.org/abs/2402.10528)
• Keywords: large language model
• Abstract: Chain-of-Thought (CoT) prompting has marked a significant advancement in enhancing the reasoning capabilities of large language models (LLMs). Previous studies have developed various extensions of CoT, which focus primarily on enhancing end-task performance. In addition, there has been research on assessing the quality of reasoning chains in CoT. This raises an intriguing question: Is it possible to predict the accuracy of LLM outputs by scrutinizing the reasoning chains they generate? To answer this research question, we introduce a benchmark, R2PE, designed specifically to explore the relationship between reasoning chains and performance in various reasoning tasks spanning five different domains. This benchmark aims to measure the falsehood of the final output of LLMs based on the reasoning steps. To make full use of information in multiple reasoning chains, we propose the process discernibility score (PDS) framework that beats the answer-checking baseline by a large margin. Concretely, this resulted in an average of 5.1% increase in the F1 score across all 45 subsets within R2PE. We further demonstrate our PDS's efficacy in advancing open-domain QA accuracy. Data and code are available at https://github.com/XinXU-USTC/R2PE.

Title: Properties and Challenges of LLM-Generated Explanations

• Authors: Jenny Kunz, Marco Kuhlmann
• Subjects: cs.CL, cs.AI, cs.HC, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.10532
• Pdf URL: https://arxiv.org/pdf/2402.10532
• Copy Paste: [[2402.10532]] Properties and Challenges of LLM-Generated Explanations(https://arxiv.org/abs/2402.10532)
• Keywords: large language model
• Abstract: The self-rationalising capabilities of large language models (LLMs) have been explored in restricted settings, using task/specific data sets. However, current LLMs do not (only) rely on specifically annotated data; nonetheless, they frequently explain their outputs. The properties of the generated explanations are influenced by the pre-training corpus and by the target data used for instruction fine-tuning. As the pre-training corpus includes a large amount of human-written explanations "in the wild", we hypothesise that LLMs adopt common properties of human explanations. By analysing the outputs for a multi-domain instruction fine-tuning data set, we find that generated explanations show selectivity and contain illustrative elements, but less frequently are subjective or misleading. We discuss reasons and consequences of the properties' presence or absence. In particular, we outline positive and negative implications depending on the goals and user groups of the self-rationalising system.

Title: Using Left and Right Brains Together: Towards Vision and Language Planning

• Authors: Jun Cen, Chenfei Wu, Xiao Liu, Shengming Yin, Yixuan Pei, Jinglong Yang, Qifeng Chen, Nan Duan, Jianguo Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.10534
• Pdf URL: https://arxiv.org/pdf/2402.10534
• Copy Paste: [[2402.10534]] Using Left and Right Brains Together: Towards Vision and Language Planning(https://arxiv.org/abs/2402.10534)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) and Large Multi-modality Models (LMMs) have demonstrated remarkable decision masking capabilities on a variety of tasks. However, they inherently operate planning within the language space, lacking the vision and spatial imagination ability. In contrast, humans utilize both left and right hemispheres of the brain for language and visual planning during the thinking process. Therefore, we introduce a novel vision-language planning framework in this work to perform concurrent visual and language planning for tasks with inputs of any form. Our framework incorporates visual planning to capture intricate environmental details, while language planning enhances the logical coherence of the overall system. We evaluate the effectiveness of our framework across vision-language tasks, vision-only tasks, and language-only tasks. The results demonstrate the superior performance of our approach, indicating that the integration of visual and language planning yields better contextually aware task execution.

Title: Personalised Drug Identifier for Cancer Treatment with Transformers using Auxiliary Information

• Authors: Aishwarya Jayagopal, Hansheng Xue, Ziyang He, Robert J. Walsh, Krishna Kumar Hariprasannan, David Shao Peng Tan, Tuan Zea Tan, Jason J. Pitt, Anand D. Jeyasekharan, Vaibhav Rajan
• Subjects: cs.LG, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2402.10551
• Pdf URL: https://arxiv.org/pdf/2402.10551
• Copy Paste: [[2402.10551]] Personalised Drug Identifier for Cancer Treatment with Transformers using Auxiliary Information(https://arxiv.org/abs/2402.10551)
• Keywords: transformer
• Abstract: Cancer remains a global challenge due to its growing clinical and economic burden. Its uniquely personal manifestation, which makes treatment difficult, has fuelled the quest for personalized treatment strategies. Thus, genomic profiling is increasingly becoming part of clinical diagnostic panels. Effective use of such panels requires accurate drug response prediction (DRP) models, which are challenging to build due to limited labelled patient data. Previous methods to address this problem have used various forms of transfer learning. However, they do not explicitly model the variable length sequential structure of the list of mutations in such diagnostic panels. Further, they do not utilize auxiliary information (like patient survival) for model training. We address these limitations through a novel transformer based method, which surpasses the performance of state-of-the-art DRP models on benchmark data. We also present the design of a treatment recommendation system (TRS), which is currently deployed at the National University Hospital, Singapore and is being evaluated in a clinical trial.

Title: InSaAF: Incorporating Safety through Accuracy and Fairness | Are LLMs ready for the Indian Legal Domain?

• Authors: Yogesh Tripathi, Raghav Donakanti, Sahil Girhepuje, Ishan Kavathekar, Bhaskara Hanuma Vedula, Gokul S Krishnan, Shreya Goyal, Anmol Goel, Balaraman Ravindran, Ponnurangam Kumaraguru
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.10567
• Pdf URL: https://arxiv.org/pdf/2402.10567
• Copy Paste: [[2402.10567]] InSaAF: Incorporating Safety through Accuracy and Fairness | Are LLMs ready for the Indian Legal Domain?(https://arxiv.org/abs/2402.10567)
• Keywords: fair, large language model
• Abstract: Recent advancements in language technology and Artificial Intelligence have resulted in numerous Language Models being proposed to perform various tasks in the legal domain ranging from predicting judgments to generating summaries. Despite their immense potential, these models have been proven to learn and exhibit societal biases and make unfair predictions. In this study, we explore the ability of Large Language Models (LLMs) to perform legal tasks in the Indian landscape when social factors are involved. We present a novel metric, $\beta$-weighted $\textit{Legal Safety Score ($LSS_{\beta}$)}$, which encapsulates both the fairness and accuracy aspects of the LLM. We assess LLMs' safety by considering its performance in the $\textit{Binary Statutory Reasoning}$ task and its fairness exhibition with respect to various axes of disparities in the Indian society. Task performance and fairness scores of LLaMA and LLaMA--2 models indicate that the proposed $LSS_{\beta}$ metric can effectively determine the readiness of a model for safe usage in the legal sector. We also propose finetuning pipelines, utilising specialised legal datasets, as a potential method to mitigate bias and improve model safety. The finetuning procedures on LLaMA and LLaMA--2 models increase the $LSS_{\beta}$, improving their usability in the Indian legal domain. Our code is publicly released.

Title: Direct Preference Optimization with an Offset

• Authors: Afra Amini, Tim Vieira, Ryan Cotterell
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.10571
• Pdf URL: https://arxiv.org/pdf/2402.10571
• Copy Paste: [[2402.10571]] Direct Preference Optimization with an Offset(https://arxiv.org/abs/2402.10571)
• Keywords: large language model
• Abstract: Direct preference optimization (DPO) is a successful fine-tuning strategy for aligning large language models with human preferences without the need to train a reward model or employ reinforcement learning. DPO, as originally formulated, relies on binary preference data and fine-tunes a language model to increase the likelihood of a preferred response over a dispreferred response. However, not all preference pairs are equal: while in some cases the preferred response is only slightly better than the dispreferred response, there can be a stronger preference for one response when, for example, the other response includes harmful or toxic content. In this paper, we propose a generalization of DPO, termed DPO with an offset (ODPO), that does not treat every preference pair equally during fine-tuning. Intuitively, ODPO requires the difference between the likelihood of the preferred and dispreferred response to be greater than an offset value. The offset is determined based on the extent to which one response is preferred over another. Our experiments on various tasks suggest that ODPO significantly outperforms DPO in aligning language models, especially when the number of preference pairs is limited.

Title: LinkNER: Linking Local Named Entity Recognition Models to Large Language Models using Uncertainty

• Authors: Zhen Zhang, Yuhua Zhao, Hang Gao, Mengting Hu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.10573
• Pdf URL: https://arxiv.org/pdf/2402.10573
• Copy Paste: [[2402.10573]] LinkNER: Linking Local Named Entity Recognition Models to Large Language Models using Uncertainty(https://arxiv.org/abs/2402.10573)
• Keywords: robust, large language model
• Abstract: Named Entity Recognition (NER) serves as a fundamental task in natural language understanding, bearing direct implications for web content analysis, search engines, and information retrieval systems. Fine-tuned NER models exhibit satisfactory performance on standard NER benchmarks. However, due to limited fine-tuning data and lack of knowledge, it performs poorly on unseen entity recognition. As a result, the usability and reliability of NER models in web-related applications are compromised. Instead, Large Language Models (LLMs) like GPT-4 possess extensive external knowledge, but research indicates that they lack specialty for NER tasks. Furthermore, non-public and large-scale weights make tuning LLMs difficult. To address these challenges, we propose a framework that combines small fine-tuned models with LLMs (LinkNER) and an uncertainty-based linking strategy called RDC that enables fine-tuned models to complement black-box LLMs, achieving better performance. We experiment with both standard NER test sets and noisy social media datasets. LinkNER enhances NER task performance, notably surpassing SOTA models in robustness tests. We also quantitatively analyze the influence of key components like uncertainty estimation methods, LLMs, and in-context learning on diverse NER tasks, offering specific web-related recommendations.

Title: Symbolic Autoencoding for Self-Supervised Sequence Learning

• Authors: Mohammad Hossein Amani, Nicolas Mario Baldwin, Amin Mansouri, Martin Josifoski, Maxime Peyrard, Robert West
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.10575
• Pdf URL: https://arxiv.org/pdf/2402.10575
• Copy Paste: [[2402.10575]] Symbolic Autoencoding for Self-Supervised Sequence Learning(https://arxiv.org/abs/2402.10575)
• Keywords: generative
• Abstract: Traditional language models, adept at next-token prediction in text sequences, often struggle with transduction tasks between distinct symbolic systems, particularly when parallel data is scarce. Addressing this issue, we introduce \textit{symbolic autoencoding} ($\Sigma$AE), a self-supervised framework that harnesses the power of abundant unparallel data alongside limited parallel data. $\Sigma$AE connects two generative models via a discrete bottleneck layer and is optimized end-to-end by minimizing reconstruction loss (simultaneously with supervised loss for the parallel data), such that the sequence generated by the discrete bottleneck can be read out as the transduced input sequence. We also develop gradient-based methods allowing for efficient self-supervised sequence learning despite the discreteness of the bottleneck. Our results demonstrate that $\Sigma$AE significantly enhances performance on transduction tasks, even with minimal parallel data, offering a promising solution for weakly supervised learning scenarios.

Title: Post-Quantum Cryptography

• Authors: Pranjal, Atul Chaturvedi
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.10576
• Pdf URL: https://arxiv.org/pdf/2402.10576
• Copy Paste: [[2402.10576]] Post-Quantum Cryptography(https://arxiv.org/abs/2402.10576)
• Keywords: secure, security
• Abstract: In this survey we propose to cover the prose of post-quantum cryptography over classical cryptography. We talk about the various cryptographic methods that are being practiced to safeguard our information. The future of secure communication is expected to be the implementation of quantum-safe cryptographic systems, and that in the post-quantum era, the development of post-quantum cryptography is essential for ensuring the security of sensitive data.

Title: Efficient Multi-task Uncertainties for Joint Semantic Segmentation and Monocular Depth Estimation

• Authors: Steven Landgraf, Markus Hillemann, Theodor Kapler, Markus Ulrich
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.10580
• Pdf URL: https://arxiv.org/pdf/2402.10580
• Copy Paste: [[2402.10580]] Efficient Multi-task Uncertainties for Joint Semantic Segmentation and Monocular Depth Estimation(https://arxiv.org/abs/2402.10580)
• Keywords: robust, explainability, segmentation
• Abstract: Quantifying the predictive uncertainty emerged as a possible solution to common challenges like overconfidence or lack of explainability and robustness of deep neural networks, albeit one that is often computationally expensive. Many real-world applications are multi-modal in nature and hence benefit from multi-task learning. In autonomous driving, for example, the joint solution of semantic segmentation and monocular depth estimation has proven to be valuable. In this work, we first combine different uncertainty quantification methods with joint semantic segmentation and monocular depth estimation and evaluate how they perform in comparison to each other. Additionally, we reveal the benefits of multi-task learning with regard to the uncertainty quality compared to solving both tasks separately. Based on these insights, we introduce EMUFormer, a novel student-teacher distillation approach for joint semantic segmentation and monocular depth estimation as well as efficient multi-task uncertainty quantification. By implicitly leveraging the predictive uncertainties of the teacher, EMUFormer achieves new state-of-the-art results on Cityscapes and NYUv2 and additionally estimates high-quality predictive uncertainties for both tasks that are comparable or superior to a Deep Ensemble despite being an order of magnitude more efficient.

Title: Threads of Subtlety: Detecting Machine-Generated Texts Through Discourse Motifs

• Authors: Zae Myung Kim, Kwang Hee Lee, Preston Zhu, Vipul Raheja, Dongyeop Kang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.10586
• Pdf URL: https://arxiv.org/pdf/2402.10586
• Copy Paste: [[2402.10586]] Threads of Subtlety: Detecting Machine-Generated Texts Through Discourse Motifs(https://arxiv.org/abs/2402.10586)
• Keywords: large language model
• Abstract: With the advent of large language models (LLM), the line between human-crafted and machine-generated texts has become increasingly blurred. This paper delves into the inquiry of identifying discernible and unique linguistic properties in texts that were written by humans, particularly uncovering the underlying discourse structures of texts beyond their surface structures. Introducing a novel methodology, we leverage hierarchical parse trees and recursive hypergraphs to unveil distinctive discourse patterns in texts produced by both LLMs and humans. Empirical findings demonstrate that, although both LLMs and humans generate distinct discourse patterns influenced by specific domains, human-written texts exhibit more structural variability, reflecting the nuanced nature of human writing in different domains. Notably, incorporating hierarchical discourse features enhances binary classifiers' overall performance in distinguishing between human-written and machine-generated texts, even on out-of-distribution and paraphrased samples. This underscores the significance of incorporating hierarchical discourse features in the analysis of text patterns. The code and dataset will be available at [TBA].

Title: Do Llamas Work in English? On the Latent Language of Multilingual Transformers

• Authors: Chris Wendler, Veniamin Veselovsky, Giovanni Monea, Robert West
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2402.10588
• Pdf URL: https://arxiv.org/pdf/2402.10588
• Copy Paste: [[2402.10588]] Do Llamas Work in English? On the Latent Language of Multilingual Transformers(https://arxiv.org/abs/2402.10588)
• Keywords: transformer
• Abstract: We ask whether multilingual language models trained on unbalanced, English-dominated corpora use English as an internal pivot language -- a question of key importance for understanding how language models function and the origins of linguistic bias. Focusing on the Llama-2 family of transformer models, our study uses carefully constructed non-English prompts with a unique correct single-token continuation. From layer to layer, transformers gradually map an input embedding of the final prompt token to an output embedding from which next-token probabilities are computed. Tracking intermediate embeddings through their high-dimensional space reveals three distinct phases, whereby intermediate embeddings (1) start far away from output token embeddings; (2) already allow for decoding a semantically correct next token in the middle layers, but give higher probability to its version in English than in the input language; (3) finally move into an input-language-specific region of the embedding space. We cast these results into a conceptual model where the three phases operate in "input space", "concept space", and "output space", respectively. Crucially, our evidence suggests that the abstract "concept space" lies closer to English than to other languages, which may have important consequences regarding the biases held by multilingual language models.

Title: Efficiency at Scale: Investigating the Performance of Diminutive Language Models in Clinical Tasks

• Authors: Niall Taylor, Upamanyu Ghose, Omid Rohanian, Mohammadmahdi Nouriborji, Andrey Kormilitzin, David Clifton, Alejo Nevado-Holgado
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.10597
• Pdf URL: https://arxiv.org/pdf/2402.10597
• Copy Paste: [[2402.10597]] Efficiency at Scale: Investigating the Performance of Diminutive Language Models in Clinical Tasks(https://arxiv.org/abs/2402.10597)
• Keywords: large language model
• Abstract: The entry of large language models (LLMs) into research and commercial spaces has led to a trend of ever-larger models, with initial promises of generalisability, followed by a widespread desire to downsize and create specialised models without the need for complete fine-tuning, using Parameter Efficient Fine-tuning (PEFT) methods. We present an investigation into the suitability of different PEFT methods to clinical decision-making tasks, across a range of model sizes, including extremely small models with as few as $25$ million parameters. Our analysis shows that the performance of most PEFT approaches varies significantly from one task to another, with the exception of LoRA, which maintains relatively high performance across all model sizes and tasks, typically approaching or matching full fine-tuned performance. The effectiveness of PEFT methods in the clinical domain is evident, particularly for specialised models which can operate on low-cost, in-house computing infrastructure. The advantages of these models, in terms of speed and reduced training costs, dramatically outweighs any performance gain from large foundation LLMs. Furthermore, we highlight how domain-specific pre-training interacts with PEFT methods and model size, and discuss how these factors interplay to provide the best efficiency-performance trade-off. Full code available at: tbd.

Title: Jailbreaking Proprietary Large Language Models using Word Substitution Cipher

• Authors: Divij Handa, Advait Chirmule, Bimal Gajera, Chitta Baral
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.10601
• Pdf URL: https://arxiv.org/pdf/2402.10601
• Copy Paste: [[2402.10601]] Jailbreaking Proprietary Large Language Models using Word Substitution Cipher(https://arxiv.org/abs/2402.10601)
• Keywords: attack, robust, large language model
• Abstract: Large Language Models (LLMs) are aligned to moral and ethical guidelines but remain susceptible to creative prompts called Jailbreak that can bypass the alignment process. However, most jailbreaking prompts contain harmful questions in the natural language (mainly English), which can be detected by the LLM themselves. In this paper, we present jailbreaking prompts encoded using cryptographic techniques. We first present a pilot study on the state-of-the-art LLM, GPT-4, in decoding several safe sentences that have been encrypted using various cryptographic techniques and find that a straightforward word substitution cipher can be decoded most effectively. Motivated by this result, we use this encoding technique for writing jailbreaking prompts. We present a mapping of unsafe words with safe words and ask the unsafe question using these mapped words. Experimental results show an attack success rate (up to 59.42%) of our proposed jailbreaking approach on state-of-the-art proprietary models including ChatGPT, GPT-4, and Gemini-Pro. Additionally, we discuss the over-defensiveness of these models. We believe that our work will encourage further research in making these LLMs more robust while maintaining their decoding capabilities.

Title: Retrieve Only When It Needs: Adaptive Retrieval Augmentation for Hallucination Mitigation in Large Language Models

• Authors: Hanxing Ding, Liang Pang, Zihao Wei, Huawei Shen, Xueqi Cheng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.10612
• Pdf URL: https://arxiv.org/pdf/2402.10612
• Copy Paste: [[2402.10612]] Retrieve Only When It Needs: Adaptive Retrieval Augmentation for Hallucination Mitigation in Large Language Models(https://arxiv.org/abs/2402.10612)
• Keywords: large language model
• Abstract: Hallucinations pose a significant challenge for the practical implementation of large language models (LLMs). The utilization of parametric knowledge in generating factual content is constrained by the limited knowledge of LLMs, potentially resulting in internal hallucinations. While incorporating external information can help fill knowledge gaps, it also introduces the risk of irrelevant information, thereby increasing the likelihood of external hallucinations. A careful and balanced integration of the parametric knowledge within LLMs with external information is crucial to alleviate hallucinations. In this study, we present Rowen, a novel approach that enhances LLMs with a selective retrieval augmentation process tailored to address hallucinated outputs. This process is governed by a multilingual semantic-aware detection module, which evaluates the consistency of the perturbed responses across various languages for the same queries. Upon detecting inconsistencies indicative of hallucinations, Rowen activates the retrieval of external information to rectify the model outputs. Rowen adeptly harmonizes the intrinsic parameters in LLMs with external knowledge sources, effectively mitigating hallucinations by ensuring a balanced integration of internal reasoning and external evidence. Through a comprehensive empirical analysis, we demonstrate that Rowen surpasses the current state-of-the-art in both detecting and mitigating hallucinated content within the outputs of LLMs.

Title: Credential Control Balance: A Universal Blockchain Account Model Abstract From Bank to Bitcoin, Ethereum External Owned Account and Account Abstraction

• Authors: Huifeng Jiao, Dr. Nathapon Udomlertsakul, Dr. Anukul Tamprasirt
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.10616
• Pdf URL: https://arxiv.org/pdf/2402.10616
• Copy Paste: [[2402.10616]] Credential Control Balance: A Universal Blockchain Account Model Abstract From Bank to Bitcoin, Ethereum External Owned Account and Account Abstraction(https://arxiv.org/abs/2402.10616)
• Keywords: secure, security
• Abstract: Blockchain market value peaked at $3 trillion, fell to $1 trillion, then recovered to $1.5 trillion and is rising again. Blockchain accounts secure most on-chain assets in this huge market (Web-12). This paper initiates a universal blockchain account model from a comprehensive review of blockchain account development, encompassing both academic and industry perspectives. This paper uses a model analysis method to analysis the account progress and create high level new account model. And it uses systematic literature review method to search, filter, analysis and evaluate the papers about account models and analyzes related technology trade-offs. Searching with key words: blockchain, account, private key and security in WOS, Scopus and Bitcoin and Ethereum community repositories, this research provides in-depth insights into the design and evaluation of account models, from traditional bank accounts to Bitcoin, EVM-adaptable, and abstraction accounts. Through data-driven comparisons of account models (security, cost, adoption), this study also explores future directions and provides an overview of cross-model account theory, guiding further blockchain research. This paper leaves deeper dives into model change drivers, application technology advancements.

Title: Enhancing Role-playing Systems through Aggressive Queries: Evaluation and Improvement

• Authors: Yihong Tang, Jiao Ou, Che Liu, Fuzheng Zhang, Di Zhang, Kun Gai
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.10618
• Pdf URL: https://arxiv.org/pdf/2402.10618
• Copy Paste: [[2402.10618]] Enhancing Role-playing Systems through Aggressive Queries: Evaluation and Improvement(https://arxiv.org/abs/2402.10618)
• Keywords: large language model
• Abstract: The advent of Large Language Models (LLMs) has propelled dialogue generation into new realms, particularly in the field of role-playing systems (RPSs). While enhanced with ordinary role-relevant training dialogues, existing LLM-based RPSs still struggle to align with roles when handling intricate and trapped queries in boundary scenarios. In this paper, we design the Modular ORchestrated Trap-setting Interaction SystEm (MORTISE) to benchmark and improve the role-playing LLMs' performance. MORTISE can produce highly role-relevant aggressive queries through the collaborative effort of multiple LLM-based modules, and formulate corresponding responses to create an adversarial training dataset via a consistent response generator. We select 190 Chinese and English roles to construct aggressive queries to benchmark existing role-playing LLMs. Through comprehensive evaluation, we find that existing models exhibit a general deficiency in role alignment capabilities. We further select 180 of the roles to collect an adversarial training dataset (named RoleAD) and retain the other 10 roles for testing. Experiments on models improved by RoleAD indicate that our adversarial dataset ameliorates this deficiency, with the improvements demonstrating a degree of generalizability in ordinary scenarios.

Title: BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillation

• Authors: Dayou Du, Yijia Zhang, Shijie Cao, Jiaqi Guo, Ting Cao, Xiaowen Chu, Ningyi Xu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.10631
• Pdf URL: https://arxiv.org/pdf/2402.10631
• Copy Paste: [[2402.10631]] BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillation(https://arxiv.org/abs/2402.10631)
• Keywords: large language model
• Abstract: The upscaling of Large Language Models (LLMs) has yielded impressive advances in natural language processing, yet it also poses significant deployment challenges. Weight quantization has emerged as a widely embraced solution to reduce memory and computational demands. This paper introduces BitDistiller, a framework that synergizes Quantization-Aware Training (QAT) with Knowledge Distillation (KD) to boost the performance of LLMs at ultra-low precisions (sub-4-bit). Specifically, BitDistiller first incorporates a tailored asymmetric quantization and clipping technique to maximally preserve the fidelity of quantized weights, and then proposes a novel Confidence-Aware Kullback-Leibler Divergence (CAKLD) objective, which is employed in a self-distillation manner to enable faster convergence and superior model performance. Empirical evaluations demonstrate that BitDistiller significantly surpasses existing methods in both 3-bit and 2-bit configurations on general language understanding and complex reasoning benchmarks. Notably, BitDistiller is shown to be more cost-effective, demanding fewer data and training resources. The code is available at https://github.com/DD-DuDa/BitDistiller.

Title: ContiFormer: Continuous-Time Transformer for Irregular Time Series Modeling

• Authors: Yuqi Chen, Kan Ren, Yansen Wang, Yuchen Fang, Weiwei Sun, Dongsheng Li
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.10635
• Pdf URL: https://arxiv.org/pdf/2402.10635
• Copy Paste: [[2402.10635]] ContiFormer: Continuous-Time Transformer for Irregular Time Series Modeling(https://arxiv.org/abs/2402.10635)
• Keywords: transformer
• Abstract: Modeling continuous-time dynamics on irregular time series is critical to account for data evolution and correlations that occur continuously. Traditional methods including recurrent neural networks or Transformer models leverage inductive bias via powerful neural architectures to capture complex patterns. However, due to their discrete characteristic, they have limitations in generalizing to continuous-time data paradigms. Though neural ordinary differential equations (Neural ODEs) and their variants have shown promising results in dealing with irregular time series, they often fail to capture the intricate correlations within these sequences. It is challenging yet demanding to concurrently model the relationship between input data points and capture the dynamic changes of the continuous-time system. To tackle this problem, we propose ContiFormer that extends the relation modeling of vanilla Transformer to the continuous-time domain, which explicitly incorporates the modeling abilities of continuous dynamics of Neural ODEs with the attention mechanism of Transformers. We mathematically characterize the expressive power of ContiFormer and illustrate that, by curated designs of function hypothesis, many Transformer variants specialized in irregular time series modeling can be covered as a special case of ContiFormer. A wide range of experiments on both synthetic and real-world datasets have illustrated the superior modeling capacities and prediction performance of ContiFormer on irregular time series data. The project link is https://seqml.github.io/contiformer/.

Title: PEGASUS: Personalized Generative 3D Avatars with Composable Attributes

• Authors: Hyunsoo Cha, Byungjun Kim, Hanbyul Joo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.10636
• Pdf URL: https://arxiv.org/pdf/2402.10636
• Copy Paste: [[2402.10636]] PEGASUS: Personalized Generative 3D Avatars with Composable Attributes(https://arxiv.org/abs/2402.10636)
• Keywords: generative
• Abstract: We present, PEGASUS, a method for constructing personalized generative 3D face avatars from monocular video sources. As a compositional generative model, our model enables disentangled controls to selectively alter the facial attributes (e.g., hair or nose) of the target individual, while preserving the identity. We present two key approaches to achieve this goal. First, we present a method to construct a person-specific generative 3D avatar by building a synthetic video collection of the target identity with varying facial attributes, where the videos are synthesized by borrowing parts from diverse individuals from other monocular videos. Through several experiments, we demonstrate the superior performance of our approach by generating unseen attributes with high realism. Subsequently, we introduce a zero-shot approach to achieve the same generative modeling more efficiently by leveraging a previously constructed personalized generative model.

Title: Linear Transformers with Learnable Kernel Functions are Better In-Context Models

• Authors: Yaroslav Aksenov, Nikita Balagansky, Sofia Maria Lo Cicero Vaina, Boris Shaposhnikov, Alexey Gorbatovski, Daniil Gavrilov
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.10644
• Pdf URL: https://arxiv.org/pdf/2402.10644
• Copy Paste: [[2402.10644]] Linear Transformers with Learnable Kernel Functions are Better In-Context Models(https://arxiv.org/abs/2402.10644)
• Keywords: transformer
• Abstract: Advancing the frontier of subquadratic architectures for Language Models (LMs) is crucial in the rapidly evolving field of natural language processing. Current innovations, including State Space Models, were initially celebrated for surpassing Transformer performance on language modeling tasks. However, these models have revealed deficiencies in essential In-Context Learning capabilities - a domain where the Transformer traditionally shines. The Based model emerged as a hybrid solution, blending a Linear Transformer with a kernel inspired by the Taylor expansion of exponential functions, augmented by convolutional networks. Mirroring the Transformer's in-context adeptness, it became a strong contender in the field. In our work, we present a singular, elegant alteration to the Based kernel that amplifies its In-Context Learning abilities evaluated with the Multi-Query Associative Recall task and overall language modeling process, as demonstrated on the Pile dataset.

Title: Can Separators Improve Chain-of-Thought Prompting?

• Authors: Yoonjeong Park, Hyunjin Kim, Chanyeol Choi, Junseong Kim, Jy-yong Sohn
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.10645
• Pdf URL: https://arxiv.org/pdf/2402.10645
• Copy Paste: [[2402.10645]] Can Separators Improve Chain-of-Thought Prompting?(https://arxiv.org/abs/2402.10645)
• Keywords: large language model
• Abstract: Chain-of-thought (CoT) prompting is a simple and effective method for improving the reasoning capabilities of Large language models (LLMs). The basic idea of CoT is to let LLMs break down their thought processes step-by-step by putting exemplars in the input prompt. However, the densely structured prompt exemplars of CoT may cause the cognitive overload of LLMs. Inspired by human cognition, we introduce CoT-Sep, a novel method that strategically employs separators at the end of each exemplar in CoT prompting. These separators are designed to help the LLMs understand their thought processes better while reasoning. It turns out that CoT-Sep significantly improves the LLMs' performances on complex reasoning tasks (e.g., GSM-8K, AQuA, CSQA), compared with the vanilla CoT, which does not use separators. We also study the effects of the type and the location of separators tested on multiple LLMs, including GPT-3.5-Turbo, GPT-4, and LLaMA-2 7B. Interestingly, the type/location of separators should be chosen appropriately to boost the reasoning capability of CoT.

Title: Enhancing Numerical Reasoning with the Guidance of Reliable Reasoning Processes

• Authors: Dingzirui Wang, Longxu Dou, Xuanliang Zhang, Qingfu Zhu, Wanxiang Che
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.10654
• Pdf URL: https://arxiv.org/pdf/2402.10654
• Copy Paste: [[2402.10654]] Enhancing Numerical Reasoning with the Guidance of Reliable Reasoning Processes(https://arxiv.org/abs/2402.10654)
• Keywords: large language model
• Abstract: Numerical reasoning is an essential ability for NLP systems to handle numeric information. Recent research indicates that fine-tuning a small-scale model to learn generating reasoning processes alongside answers can significantly enhance performance. However, current methods have the limitation that most methods generate reasoning processes with large language models (LLMs), which are "unreliable" since such processes could contain information unrelated to the answer. To address this limitation, we introduce Enhancing NumeriCal reasOning with Reliable procEsses (Encore), which derives the reliable reasoning process by decomposing the answer formula, ensuring which fully supports the answer. Nevertheless, models could lack enough data to learn the reasoning process generation adequately, since our method generates only one single reasoning process for one formula. To overcome this difficulty, we present a series of pre-training tasks to help models learn the reasoning process generation with synthesized data. The experiments show that Encore yields improvement on all five experimental datasets with an average of 1.8%, proving the effectiveness of our method.

Title: Fine Tuning Named Entity Extraction Models for the Fantasy Domain

• Authors: Aravinth Sivaganeshan, Nisansa de Silva
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.10662
• Pdf URL: https://arxiv.org/pdf/2402.10662
• Copy Paste: [[2402.10662]] Fine Tuning Named Entity Extraction Models for the Fantasy Domain(https://arxiv.org/abs/2402.10662)
• Keywords: extraction
• Abstract: Named Entity Recognition (NER) is a sequence classification Natural Language Processing task where entities are identified in the text and classified into predefined categories. It acts as a foundation for most information extraction systems. Dungeons and Dragons (D&D) is an open-ended tabletop fantasy game with its own diverse lore. DnD entities are domain-specific and are thus unrecognizable by even the state-of-the-art off-the-shelf NER systems as the NER systems are trained on general data for pre-defined categories such as: person (PERS), location (LOC), organization (ORG), and miscellaneous (MISC). For meaningful extraction of information from fantasy text, the entities need to be classified into domain-specific entity categories as well as the models be fine-tuned on a domain-relevant corpus. This work uses available lore of monsters in the D&D domain to fine-tune Trankit, which is a prolific NER framework that uses a pre-trained model for NER. Upon this training, the system acquires the ability to extract monster names from relevant domain documents under a novel NER tag. This work compares the accuracy of the monster name identification against; the zero-shot Trankit model and two FLAIR models. The fine-tuned Trankit model achieves an 87.86% F1 score surpassing all the other considered models.

Title: Improving Demonstration Diversity by Human-Free Fusing for Text-to-SQL

• Authors: Dingzirui Wang, Longxu Dou, Xuanliang Zhang, Qingfu Zhu, Wanxiang Che
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.10663
• Pdf URL: https://arxiv.org/pdf/2402.10663
• Copy Paste: [[2402.10663]] Improving Demonstration Diversity by Human-Free Fusing for Text-to-SQL(https://arxiv.org/abs/2402.10663)
• Keywords: large language model
• Abstract: Currently, the in-context learning method based on large language models (LLMs) has become the mainstream of text-to-SQL research. Previous works have discussed how to select demonstrations related to the user question from a human-labeled demonstration pool. However, human labeling suffers from the limitations of insufficient diversity and high labeling overhead. Therefore, in this paper, we discuss how to measure and improve the diversity of the demonstrations for text-to-SQL. We present a metric to measure the diversity of the demonstrations and analyze the insufficient of the existing labeled data by experiments. Based on the above discovery, we propose fusing iteratively for demonstrations (Fused) to build a high-diversity demonstration pool through human-free multiple-iteration synthesis, improving diversity and lowering label cost. Our method achieves an average improvement of 3.2% and 5.0% with and without human labeling on several mainstream datasets, which proves the effectiveness of Fused.

Title: Selective Prediction for Semantic Segmentation using Post-Hoc Confidence Estimation and Its Performance under Distribution Shift

• Authors: Bruno Laboissiere Camargos Borges, Bruno Machado Pacheco, Danilo Silva
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2402.10665
• Pdf URL: https://arxiv.org/pdf/2402.10665
• Copy Paste: [[2402.10665]] Selective Prediction for Semantic Segmentation using Post-Hoc Confidence Estimation and Its Performance under Distribution Shift(https://arxiv.org/abs/2402.10665)
• Keywords: segmentation
• Abstract: Semantic segmentation plays a crucial role in various computer vision applications, yet its efficacy is often hindered by the lack of high-quality labeled data. To address this challenge, a common strategy is to leverage models trained on data from different populations, such as publicly available datasets. This approach, however, leads to the distribution shift problem, presenting a reduced performance on the population of interest. In scenarios where model errors can have significant consequences, selective prediction methods offer a means to mitigate risks and reduce reliance on expert supervision. This paper investigates selective prediction for semantic segmentation in low-resource settings, thus focusing on post-hoc confidence estimators applied to pre-trained models operating under distribution shift. We propose a novel image-level confidence measure tailored for semantic segmentation and demonstrate its effectiveness through experiments on three medical imaging tasks. Our findings show that post-hoc confidence estimators offer a cost-effective approach to reducing the impacts of distribution shift.

Title: Humans or LLMs as the Judge? A Study on Judgement Biases

• Authors: Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, Benyou Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.10669
• Pdf URL: https://arxiv.org/pdf/2402.10669
• Copy Paste: [[2402.10669]] Humans or LLMs as the Judge? A Study on Judgement Biases(https://arxiv.org/abs/2402.10669)
• Keywords: attack, robust, large language model
• Abstract: Adopting human and large language models (LLM) as judges (\textit{a.k.a} human- and LLM-as-a-judge) for evaluating the performance of existing LLMs has recently gained attention. Nonetheless, this approach concurrently introduces potential biases from human and LLM judges, questioning the reliability of the evaluation results. In this paper, we propose a novel framework for investigating 5 types of biases for LLM and human judges. We curate a dataset with 142 samples referring to the revised Bloom's Taxonomy and conduct thousands of human and LLM evaluations. Results show that human and LLM judges are vulnerable to perturbations to various degrees, and that even the most cutting-edge judges possess considerable biases. We further exploit their weakness and conduct attacks on LLM judges. We hope that our work can notify the community of the vulnerability of human- and LLM-as-a-judge against perturbations, as well as the urgency of developing robust evaluation systems.

Title: OpenFMNav: Towards Open-Set Zero-Shot Object Navigation via Vision-Language Foundation Models

• Authors: Yuxuan Kuang, Hai Lin, Meng Jiang
• Subjects: cs.CL, cs.RO
• Abstract URL: https://arxiv.org/abs/2402.10670
• Pdf URL: https://arxiv.org/pdf/2402.10670
• Copy Paste: [[2402.10670]] OpenFMNav: Towards Open-Set Zero-Shot Object Navigation via Vision-Language Foundation Models(https://arxiv.org/abs/2402.10670)
• Keywords: large language model
• Abstract: Object navigation (ObjectNav) requires an agent to navigate through unseen environments to find queried objects. Many previous methods attempted to solve this task by relying on supervised or reinforcement learning, where they are trained on limited household datasets with close-set objects. However, two key challenges are unsolved: understanding free-form natural language instructions that demand open-set objects, and generalizing to new environments in a zero-shot manner. Aiming to solve the two challenges, in this paper, we propose OpenFMNav, an Open-set Foundation Model based framework for zero-shot object Navigation. We first unleash the reasoning abilities of large language models (LLMs) to extract proposed objects from natural language instructions that meet the user's demand. We then leverage the generalizability of large vision language models (VLMs) to actively discover and detect candidate objects from the scene, building a Versatile Semantic Score Map (VSSM). Then, by conducting common sense reasoning on VSSM, our method can perform effective language-guided exploration and exploitation of the scene and finally reach the goal. By leveraging the reasoning and generalizing abilities of foundation models, our method can understand free-form human instructions and perform effective open-set zero-shot navigation in diverse environments. Extensive experiments on the HM3D ObjectNav benchmark show that our method surpasses all the strong baselines on all metrics, proving our method's effectiveness. Furthermore, we perform real robot demonstrations to validate our method's open-set-ness and generalizability to real-world environments.

Title: Decomposition for Enhancing Attention: Improving LLM-based Text-to-SQL through Workflow Paradigm

• Authors: Yuanzhen Xie, Xinzhou Jin, Tao Xie, MingXiong Lin, Liang Chen, Chenyun Yu, Lei Cheng, ChengXiang Zhuo, Bo Hu, Zang Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.10671
• Pdf URL: https://arxiv.org/pdf/2402.10671
• Copy Paste: [[2402.10671]] Decomposition for Enhancing Attention: Improving LLM-based Text-to-SQL through Workflow Paradigm(https://arxiv.org/abs/2402.10671)
• Keywords: diffusion
• Abstract: In-context learning of large-language models (LLMs) has achieved remarkable success in the field of natural language processing, while extensive case studies reveal that the single-step chain-of-thought prompting approach faces challenges such as attention diffusion and inadequate performance in complex tasks like text-to-SQL. To improve the contextual learning capabilities of LLMs in text-to-SQL, a workflow paradigm method is proposed, aiming to enhance the attention and problem-solving scope of LLMs through decomposition. Specifically, the information determination module for eliminating redundant information and the brand-new prompt structure based on problem classification greatly enhance the model's attention. Additionally, the inclusion of self-correcting and active learning modules greatly expands the problem-solving scope of LLMs, hence improving the upper limit of LLM-based approaches. Extensive experiments conducted on three datasets demonstrate that our approach outperforms other methods by a significant margin. About 2-3 percentage point improvements compared to the existing baseline on the Spider Dev and Spider-Realistic datasets and new SOTA results on the Spider Test dataset are achieved. Our code is available on GitHub: \url{https://github.com/FlyingFeather/DEA-SQL}.

Title: German Text Simplification: Finetuning Large Language Models with Semi-Synthetic Data

• Authors: Lars Klöser, Mika Beele, Jan-Niklas Schagen, Bodo Kraft
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.10675
• Pdf URL: https://arxiv.org/pdf/2402.10675
• Copy Paste: [[2402.10675]] German Text Simplification: Finetuning Large Language Models with Semi-Synthetic Data(https://arxiv.org/abs/2402.10675)
• Keywords: generative, large language model
• Abstract: This study pioneers the use of synthetically generated data for training generative models in document-level text simplification of German texts. We demonstrate the effectiveness of our approach with real-world online texts. Addressing the challenge of data scarcity in language simplification, we crawled professionally simplified German texts and synthesized a corpus using GPT-4. We finetune Large Language Models with up to 13 billion parameters on this data and evaluate their performance. This paper employs various methodologies for evaluation and demonstrates the limitations of currently used rule-based metrics. Both automatic and manual evaluations reveal that our models can significantly simplify real-world online texts, indicating the potential of synthetic data in improving text simplification.

Title: LongHeads: Multi-Head Attention is Secretly a Long Context Processor

• Authors: Yi Lu, Xin Zhou, Wei He, Jun Zhao, Tao Ji, Tao Gui, Qi Zhang, Xuanjing Huang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.10685
• Pdf URL: https://arxiv.org/pdf/2402.10685
• Copy Paste: [[2402.10685]] LongHeads: Multi-Head Attention is Secretly a Long Context Processor(https://arxiv.org/abs/2402.10685)
• Keywords: large language model
• Abstract: Large language models (LLMs) have achieved impressive performance in numerous domains but often struggle to process lengthy inputs effectively and efficiently due to limited length generalization and attention's quadratic computational demands. Many sought to mitigate this by restricting the attention window within the pre-trained length. However, these methods introduce new issues such as ignoring the middle context and requiring additional training. To address these problems, we propose LongHeads, a training-free framework that enhances LLM's long context ability by unlocking multi-head attention's untapped potential. Instead of allowing each head to attend to the full sentence, which struggles with generalizing to longer sequences due to out-of-distribution (OOD) issues, we allow each head to process in-distribution length by selecting and attending to important context chunks. To this end, we propose a chunk selection strategy that relies on the inherent correlation between the query and the key representations, efficiently distributing context chunks to different heads. In this way, each head ensures it can effectively process attended tokens within the trained length, while different heads in different layers can collectively process longer contexts. LongHeads works efficiently in linear time, fits seamlessly with many LLMs that use relative positional encoding. Our extensive empirical analyses verify LongHeads's efficacy in extending the usable context window for existing models, showcasing its promise for enhancing long text understanding.

Title: Opening the Black Box of Large Language Models: Two Views on Holistic Interpretability

• Authors: Haiyan Zhao, Fan Yang, Himabindu Lakkaraju, Mengnan Du
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.10688
• Pdf URL: https://arxiv.org/pdf/2402.10688
• Copy Paste: [[2402.10688]] Opening the Black Box of Large Language Models: Two Views on Holistic Interpretability(https://arxiv.org/abs/2402.10688)
• Keywords: fair, interpretability, large language model
• Abstract: As large language models (LLMs) grow more powerful, concerns around potential harms like toxicity, unfairness, and hallucination threaten user trust. Ensuring beneficial alignment of LLMs with human values through model alignment is thus critical yet challenging, requiring a deeper understanding of LLM behaviors and mechanisms. We propose opening the black box of LLMs through a framework of holistic interpretability encompassing complementary bottom-up and top-down perspectives. The bottom-up view, enabled by mechanistic interpretability, focuses on component functionalities and training dynamics. The top-down view utilizes representation engineering to analyze behaviors through hidden representations. In this paper, we review the landscape around mechanistic interpretability and representation engineering, summarizing approaches, discussing limitations and applications, and outlining future challenges in using these techniques to achieve ethical, honest, and reliable reasoning aligned with human values.

Title: Multi-Cultural Commonsense Knowledge Distillation

• Authors: Tuan-Phong Nguyen, Simon Razniewski, Gerhard Weikum
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.10689
• Pdf URL: https://arxiv.org/pdf/2402.10689
• Copy Paste: [[2402.10689]] Multi-Cultural Commonsense Knowledge Distillation(https://arxiv.org/abs/2402.10689)
• Keywords: generative, large language model
• Abstract: Despite recent progress, large language models (LLMs) still face the challenge of appropriately reacting to the intricacies of social and cultural conventions. This paper presents MANGO, a methodology for distilling high-accuracy, high-recall assertions of cultural knowledge. We judiciously and iteratively prompt LLMs for this purpose from two entry points, concepts and cultures. Outputs are consolidated via clustering and generative summarization. Running the MANGO method with GPT-3.5 as underlying LLM yields 167K high-accuracy assertions for 30K concepts and 11K cultures, surpassing prior resources by a large margin. For extrinsic evaluation, we explore augmenting dialogue systems with cultural knowledge assertions. We find that adding knowledge from MANGO improves the overall quality, specificity, and cultural sensitivity of dialogue responses, as judged by human annotators. Data and code are available for download.

Title: Exploring Precision and Recall to assess the quality and diversity of LLMs

• Authors: Le Bronnec Florian, Verine Alexandre, Negrevergne Benjamin, Chevaleyre Yann, Allauzen Alexandre
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.10693
• Pdf URL: https://arxiv.org/pdf/2402.10693
• Copy Paste: [[2402.10693]] Exploring Precision and Recall to assess the quality and diversity of LLMs(https://arxiv.org/abs/2402.10693)
• Keywords: large language model
• Abstract: This paper introduces a novel evaluation framework for Large Language Models (LLMs) such as Llama-2 and Mistral, focusing on the adaptation of Precision and Recall metrics from image generation to text generation. This approach allows for a nuanced assessment of the quality and diversity of generated text without the need for aligned corpora. By conducting a comprehensive evaluation of state-of-the-art language models, the study reveals significant insights into their performance on open-ended generation tasks, which are not adequately captured by traditional benchmarks. The findings highlight a trade-off between the quality and diversity of generated samples, particularly when models are fine-tuned with human feedback. This work extends the toolkit for distribution-based NLP evaluation, offering insights into the practical capabilities and challenges faced by current LLMs in generating diverse and high-quality text.

Title: Unlink to Unlearn: Simplifying Edge Unlearning in GNNs

• Authors: Jiajun Tan, Fei Sun, Ruichen Qiu, Du Su, Huawei Shen
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2402.10695
• Pdf URL: https://arxiv.org/pdf/2402.10695
• Copy Paste: [[2402.10695]] Unlink to Unlearn: Simplifying Edge Unlearning in GNNs(https://arxiv.org/abs/2402.10695)
• Keywords: privacy, protect
• Abstract: As concerns over data privacy intensify, unlearning in Graph Neural Networks (GNNs) has emerged as a prominent research frontier in academia. This concept is pivotal in enforcing the right to be forgotten, which entails the selective removal of specific data from trained GNNs upon user request. Our research focuses on edge unlearning, a process of particular relevance to real-world applications, owing to its widespread applicability. Current state-of-the-art approaches like GNNDelete can eliminate the influence of specific edges, yet our research has revealed a critical limitation in these approaches, termed over-forgetting. It occurs when the unlearning process inadvertently removes excessive information beyond specific data, leading to a significant decline in prediction accuracy for the remaining edges. To address this issue, we have identified the loss functions of GNNDelete as the primary source of the over-forgetting phenomenon. Furthermore, our analysis also suggests that loss functions may not be essential for effective edge unlearning. Building on these insights, we have simplified GNNDelete to develop Unlink-to-Unlearn (UtU), a novel method that facilitates unlearning exclusively through unlinking the forget edges from graph structure. Our extensive experiments demonstrate that UtU delivers privacy protection on par with that of a retrained model while preserving high accuracy in downstream tasks. Specifically, UtU upholds over 97.3% of the retrained model's privacy protection capabilities and 99.8% of its link prediction accuracy. Meanwhile, UtU requires only constant computational demands, underscoring its advantage as a highly lightweight and practical edge unlearning solution.

Title: Question-Instructed Visual Descriptions for Zero-Shot Video Question Answering

• Authors: David Romero, Thamar Solorio
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.10698
• Pdf URL: https://arxiv.org/pdf/2402.10698
• Copy Paste: [[2402.10698]] Question-Instructed Visual Descriptions for Zero-Shot Video Question Answering(https://arxiv.org/abs/2402.10698)
• Keywords: large language model
• Abstract: We present Q-ViD, a simple approach for video question answering (video QA), that unlike prior methods, which are based on complex architectures, computationally expensive pipelines or use closed models like GPTs, Q-ViD relies on a single instruction-aware open vision-language model (InstructBLIP) to tackle videoQA using frame descriptions. Specifically, we create captioning instruction prompts that rely on the target questions about the videos and leverage InstructBLIP to obtain video frame captions that are useful to the task at hand. Subsequently, we form descriptions of the whole video using the question-dependent frame captions, and feed that information, along with a question-answering prompt, to a large language model (LLM). The LLM is our reasoning module, and performs the final step of multiple-choice QA. Our simple Q-ViD framework achieves competitive or even higher performances than current state of the art models on a diverse range of videoQA benchmarks, including NExT-QA, STAR, How2QA, TVQA and IntentQA.

Title: Rethinking Human-like Translation Strategy: Integrating Drift-Diffusion Model with Large Language Models for Machine Translation

• Authors: Hongbin Na, Zimu Wang, Mieradilijiang Maimaiti, Tong Chen, Wei Wang, Tao Shen, Ling Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.10699
• Pdf URL: https://arxiv.org/pdf/2402.10699
• Copy Paste: [[2402.10699]] Rethinking Human-like Translation Strategy: Integrating Drift-Diffusion Model with Large Language Models for Machine Translation(https://arxiv.org/abs/2402.10699)
• Keywords: diffusion, large language model
• Abstract: Large language models (LLMs) have demonstrated promising potential in various downstream tasks, including machine translation. However, prior work on LLM-based machine translation has mainly focused on better utilizing training data, demonstrations, or pre-defined and universal knowledge to improve performance, with a lack of consideration of decision-making like human translators. In this paper, we incorporate Thinker with the Drift-Diffusion Model (Thinker-DDM) to address this issue. We then redefine the Drift-Diffusion process to emulate human translators' dynamic decision-making under constrained resources. We conduct extensive experiments under the high-resource, low-resource, and commonsense translation settings using the WMT22 and CommonMT datasets, in which Thinker-DDM outperforms baselines in the first two scenarios. We also perform additional analysis and evaluation on commonsense translation to illustrate the high effectiveness and efficacy of the proposed method.

Title: An Empirical Study on Cross-lingual Vocabulary Adaptation for Efficient Generative LLM Inference

• Authors: Atsuki Yamaguchi, Aline Villavicencio, Nikolaos Aletras
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.10712
• Pdf URL: https://arxiv.org/pdf/2402.10712
• Copy Paste: [[2402.10712]] An Empirical Study on Cross-lingual Vocabulary Adaptation for Efficient Generative LLM Inference(https://arxiv.org/abs/2402.10712)
• Keywords: generative, large language model
• Abstract: The development of state-of-the-art generative large language models (LLMs) disproportionately relies on English-centric tokenizers, vocabulary and pre-training data. Despite the fact that some LLMs have multilingual capabilities, recent studies have shown that their inference efficiency deteriorates when generating text in languages other than English. This results in increased inference time and costs. Cross-lingual vocabulary adaptation methods have been proposed for adapting models to a target language aiming to improve downstream performance. However, the effectiveness of these methods on increasing inference efficiency of generative LLMs has yet to be explored. In this paper, we perform an empirical study of various cross-lingual vocabulary adaptation methods on five generative LLMs (including monolingual and multilingual models) across four typologically-diverse languages and four natural language understanding tasks. We find that cross-lingual vocabulary adaptation substantially contributes to LLM inference speedups of up to 271.5%. We also show that adapting LLMs that have been pre-trained on more balanced multilingual data results in downstream performance comparable to the original models.

Title: Let's Learn Step by Step: Enhancing In-Context Learning Ability with Curriculum Learning

• Authors: Yinpeng Liu, Jiawei Liu, Xiang Shi, Qikai Cheng, Wei Lu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.10738
• Pdf URL: https://arxiv.org/pdf/2402.10738
• Copy Paste: [[2402.10738]] Let's Learn Step by Step: Enhancing In-Context Learning Ability with Curriculum Learning(https://arxiv.org/abs/2402.10738)
• Keywords: large language model
• Abstract: Demonstration ordering, which is an important strategy for in-context learning (ICL), can significantly affects the performance of large language models (LLMs). However, most of the current approaches of ordering require additional knowledge and similarity calculation. We advocate the few-shot in-context curriculum learning (ICCL), a simple but effective demonstration ordering method for ICL, which implies gradually increasing the complexity of prompt demonstrations during the inference process. Then we design three experiments to discuss the effectiveness of ICCL, the formation mechanism of LLM's ICCL capability, and the impact of ordering subjects. Experimental results demonstrate that ICCL, developed during the instruction-tuning stage, is effective for open-source LLMs. Moreover, LLMs exhibit a weaker capacity compared to humans in discerning the difficulty levels of demonstrations. We release our code at https://github.com/61peng/curri_learning.

Title: PointMamba: A Simple State Space Model for Point Cloud Analysis

• Authors: Dingkang Liang, Xin Zhou, Xinyu Wang, Xingkui Zhu, Wei Xu, Zhikang Zou, Xiaoqing Ye, Xiang Bai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.10739
• Pdf URL: https://arxiv.org/pdf/2402.10739
• Copy Paste: [[2402.10739]] PointMamba: A Simple State Space Model for Point Cloud Analysis(https://arxiv.org/abs/2402.10739)
• Keywords: transformer
• Abstract: Transformers have become one of the foundational architectures in point cloud analysis tasks due to their excellent global modeling ability. However, the attention mechanism has quadratic complexity and is difficult to extend to long sequence modeling due to limited computational resources and so on. Recently, state space models (SSM), a new family of deep sequence models, have presented great potential for sequence modeling in NLP tasks. In this paper, taking inspiration from the success of SSM in NLP, we propose PointMamba, a framework with global modeling and linear complexity. Specifically, by taking embedded point patches as input, we proposed a reordering strategy to enhance SSM's global modeling ability by providing a more logical geometric scanning order. The reordered point tokens are then sent to a series of Mamba blocks to causally capture the point cloud structure. Experimental results show our proposed PointMamba outperforms the transformer-based counterparts on different point cloud analysis datasets, while significantly saving about 44.3% parameters and 25% FLOPs, demonstrating the potential option for constructing foundational 3D vision models. We hope our PointMamba can provide a new perspective for point cloud analysis. The code is available at https://github.com/LMD0311/PointMamba.

Title: Construction of a Syntactic Analysis Map for Yi Shui School through Text Mining and Natural Language Processing Research

• Authors: Hanqing Zhao, Yuehan Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.10743
• Pdf URL: https://arxiv.org/pdf/2402.10743
• Copy Paste: [[2402.10743]] Construction of a Syntactic Analysis Map for Yi Shui School through Text Mining and Natural Language Processing Research(https://arxiv.org/abs/2402.10743)
• Keywords: extraction, segmentation
• Abstract: Entity and relationship extraction is a crucial component in natural language processing tasks such as knowledge graph construction, question answering system design, and semantic analysis. Most of the information of the Yishui school of traditional Chinese Medicine (TCM) is stored in the form of unstructured classical Chinese text. The key information extraction of TCM texts plays an important role in mining and studying the academic schools of TCM. In order to solve these problems efficiently using artificial intelligence methods, this study constructs a word segmentation and entity relationship extraction model based on conditional random fields under the framework of natural language processing technology to identify and extract the entity relationship of traditional Chinese medicine texts, and uses the common weighting technology of TF-IDF information retrieval and data mining to extract important key entity information in different ancient books. The dependency syntactic parser based on neural network is used to analyze the grammatical relationship between entities in each ancient book article, and it is represented as a tree structure visualization, which lays the foundation for the next construction of the knowledge graph of Yishui school and the use of artificial intelligence methods to carry out the research of TCM academic schools.

Title: GenRES: Rethinking Evaluation for Generative Relation Extraction in the Era of Large Language Models

• Authors: Pengcheng Jiang, Jiacheng Lin, Zifeng Wang, Jimeng Sun, Jiawei Han
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.10744
• Pdf URL: https://arxiv.org/pdf/2402.10744
• Copy Paste: [[2402.10744]] GenRES: Rethinking Evaluation for Generative Relation Extraction in the Era of Large Language Models(https://arxiv.org/abs/2402.10744)
• Keywords: extraction, generative, large language model
• Abstract: The field of relation extraction (RE) is experiencing a notable shift towards generative relation extraction (GRE), leveraging the capabilities of large language models (LLMs). However, we discovered that traditional relation extraction (RE) metrics like precision and recall fall short in evaluating GRE methods. This shortfall arises because these metrics rely on exact matching with human-annotated reference relations, while GRE methods often produce diverse and semantically accurate relations that differ from the references. To fill this gap, we introduce GenRES for a multi-dimensional assessment in terms of the topic similarity, uniqueness, granularity, factualness, and completeness of the GRE results. With GenRES, we empirically identified that (1) precision/recall fails to justify the performance of GRE methods; (2) human-annotated referential relations can be incomplete; (3) prompting LLMs with a fixed set of relations or entities can cause hallucinations. Next, we conducted a human evaluation of GRE methods that shows GenRES is consistent with human preferences for RE quality. Last, we made a comprehensive evaluation of fourteen leading LLMs using GenRES across document, bag, and sentence level RE datasets, respectively, to set the benchmark for future research in GRE

Title: STF: Spatio-Temporal Fusion Module for Improving Video Object Detection

• Authors: Noreen Anwar, Guillaume-Alexandre Bilodeau, Wassim Bouachir
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.10752
• Pdf URL: https://arxiv.org/pdf/2402.10752
• Copy Paste: [[2402.10752]] STF: Spatio-Temporal Fusion Module for Improving Video Object Detection(https://arxiv.org/abs/2402.10752)
• Keywords: robust
• Abstract: Consecutive frames in a video contain redundancy, but they may also contain relevant complementary information for the detection task. The objective of our work is to leverage this complementary information to improve detection. Therefore, we propose a spatio-temporal fusion framework (STF). We first introduce multi-frame and single-frame attention modules that allow a neural network to share feature maps between nearby frames to obtain more robust object representations. Second, we introduce a dual-frame fusion module that merges feature maps in a learnable manner to improve them. Our evaluation is conducted on three different benchmarks including video sequences of moving road users. The performed experiments demonstrate that the proposed spatio-temporal fusion module leads to improved detection performance compared to baseline object detectors. Code is available at https://github.com/noreenanwar/STF-module

Title: ToolSword: Unveiling Safety Issues of Large Language Models in Tool Learning Across Three Stages

• Authors: Junjie Ye, Sixian Li, Guanyu Li, Caishuang Huang, Songyang Gao, Yilong Wu, Qi Zhang, Tao Gui, Xuanjing Huang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.10753
• Pdf URL: https://arxiv.org/pdf/2402.10753
• Copy Paste: [[2402.10753]] ToolSword: Unveiling Safety Issues of Large Language Models in Tool Learning Across Three Stages(https://arxiv.org/abs/2402.10753)
• Keywords: attack, large language model
• Abstract: Tool learning is widely acknowledged as a foundational approach or deploying large language models (LLMs) in real-world scenarios. While current research primarily emphasizes leveraging tools to augment LLMs, it frequently neglects emerging safety considerations tied to their application. To fill this gap, we present $ToolSword$, a comprehensive framework dedicated to meticulously investigating safety issues linked to LLMs in tool learning. Specifically, ToolSword delineates six safety scenarios for LLMs in tool learning, encompassing $malicious$ $queries$ and $jailbreak$ $attacks$ in the input stage, $noisy$ $misdirection$ and $risky$ $cues$ in the execution stage, and $harmful$ $feedback$ and $error$ $conflicts$ in the output stage. Experiments conducted on 11 open-source and closed-source LLMs reveal enduring safety challenges in tool learning, such as handling harmful queries, employing risky tools, and delivering detrimental feedback, which even GPT-4 is susceptible to. Moreover, we conduct further studies with the aim of fostering research on tool learning safety. The data is released in https://github.com/Junjie-Ye/ToolSword.

Title: Towards Cohesion-Fairness Harmony: Contrastive Regularization in Individual Fair Graph Clustering

• Authors: Siamak Ghodsi, Seyed Amjad Seyedi, Eirini Ntoutsi
• Subjects: cs.LG, cs.AI, cs.IT, cs.SI
• Abstract URL: https://arxiv.org/abs/2402.10756
• Pdf URL: https://arxiv.org/pdf/2402.10756
• Copy Paste: [[2402.10756]] Towards Cohesion-Fairness Harmony: Contrastive Regularization in Individual Fair Graph Clustering(https://arxiv.org/abs/2402.10756)
• Keywords: fair, interpretability
• Abstract: Conventional fair graph clustering methods face two primary challenges: i) They prioritize balanced clusters at the expense of cluster cohesion by imposing rigid constraints, ii) Existing methods of both individual and group-level fairness in graph partitioning mostly rely on eigen decompositions and thus, generally lack interpretability. To address these issues, we propose iFairNMTF, an individual Fairness Nonnegative Matrix Tri-Factorization model with contrastive fairness regularization that achieves balanced and cohesive clusters. By introducing fairness regularization, our model allows for customizable accuracy-fairness trade-offs, thereby enhancing user autonomy without compromising the interpretability provided by nonnegative matrix tri-factorization. Experimental evaluations on real and synthetic datasets demonstrate the superior flexibility of iFairNMTF in achieving fairness and clustering performance.

Title: Inference to the Best Explanation in Large Language Models

• Authors: Dhairya Dalal, Marco Valentino, André Freitas, Paul Buitelaar
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.10767
• Pdf URL: https://arxiv.org/pdf/2402.10767
• Copy Paste: [[2402.10767]] Inference to the Best Explanation in Large Language Models(https://arxiv.org/abs/2402.10767)
• Keywords: large language model
• Abstract: While Large Language Models (LLMs) have found success in real-world applications, their underlying explanatory process is still poorly understood. This paper proposes IBE-Eval, a framework inspired by philosophical accounts on Inference to the Best Explanation (IBE) to advance the interpretation and evaluation of LLMs' explanations. IBE-Eval estimates the plausibility of natural language explanations through a combination of explicit logical and linguistic features including: consistency, parsimony, coherence, and uncertainty. Extensive experiments are conducted on Causal Question Answering (CQA), where \textit{IBE-Eval} is tasked to select the most plausible causal explanation amongst competing ones generated by LLMs (i.e., GPT 3.5 and Llama 2). The experiments reveal that IBE-Eval can successfully identify the best explanation with up to 77\% accuracy ($\approx 27\%$ above random), improving upon a GPT 3.5-as-a-Judge baseline ($\approx+17\%$) while being intrinsically more efficient and interpretable. Additional analyses suggest that, despite model-specific variances, LLM-generated explanations tend to conform to IBE criteria and that IBE-Eval is significantly correlated with human judgment, opening up opportunities for future development of automated explanation verification tools.

Title: Distillation Enhanced Generative Retrieval

• Authors: Yongqi Li, Zhen Zhang, Wenjie Wang, Liqiang Nie, Wenjie Li, Tat-Seng Chua
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2402.10769
• Pdf URL: https://arxiv.org/pdf/2402.10769
• Copy Paste: [[2402.10769]] Distillation Enhanced Generative Retrieval(https://arxiv.org/abs/2402.10769)
• Keywords: robust, generative
• Abstract: Generative retrieval is a promising new paradigm in text retrieval that generates identifier strings of relevant passages as the retrieval target. This paradigm leverages powerful generative language models, distinct from traditional sparse or dense retrieval methods. In this work, we identify a viable direction to further enhance generative retrieval via distillation and propose a feasible framework, named DGR. DGR utilizes sophisticated ranking models, such as the cross-encoder, in a teacher role to supply a passage rank list, which captures the varying relevance degrees of passages instead of binary hard labels; subsequently, DGR employs a specially designed distilled RankNet loss to optimize the generative retrieval model, considering the passage rank order provided by the teacher model as labels. This framework only requires an additional distillation step to enhance current generative retrieval systems and does not add any burden to the inference stage. We conduct experiments on four public datasets, and the results indicate that DGR achieves state-of-the-art performance among the generative retrieval methods. Additionally, DGR demonstrates exceptional robustness and generalizability with various teacher models and distillation losses.

Title: How Reliable Are Automatic Evaluation Methods for Instruction-Tuned LLMs?

• Authors: Ehsan Doostmohammadi, Oskar Holmström, Marco Kuhlmann
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.10770
• Pdf URL: https://arxiv.org/pdf/2402.10770
• Copy Paste: [[2402.10770]] How Reliable Are Automatic Evaluation Methods for Instruction-Tuned LLMs?(https://arxiv.org/abs/2402.10770)
• Keywords: large language model
• Abstract: Work on instruction-tuned Large Language Models (LLMs) has used automatic methods based on text overlap and LLM judgments as cost-effective alternatives to human evaluation. In this paper, we study the reliability of such methods across a broad range of tasks and in a cross-lingual setting. In contrast to previous findings, we observe considerable variability in correlations between automatic methods and human evaluators when scores are differentiated by task type. Specifically, the widely-used ROUGE-L metric strongly correlates with human judgments for short-answer English tasks but is unreliable in free-form generation tasks and cross-lingual transfer. The effectiveness of GPT-4 as an evaluator depends on including reference answers when prompting for assessments, which can lead to overly strict evaluations in free-form generation tasks. In summary, we find that, while automatic evaluation methods can approximate human judgements under specific conditions, their reliability is highly context-dependent. Our findings enhance the understanding of how automatic methods should be applied and interpreted when developing and evaluating instruction-tuned LLMs.

Title: AIM: Automated Input Set Minimization for Metamorphic Security Testing

• Authors: Nazanin Bayati Chaleshtari, Yoann Marquer, Fabrizio Pastore, Lionel C. Briand
• Subjects: cs.CR, cs.SE
• Abstract URL: https://arxiv.org/abs/2402.10773
• Pdf URL: https://arxiv.org/pdf/2402.10773
• Copy Paste: [[2402.10773]] AIM: Automated Input Set Minimization for Metamorphic Security Testing(https://arxiv.org/abs/2402.10773)
• Keywords: security, attack
• Abstract: For Web systems, which are accessible to any machine connected to internet, security is a critical concern. Although security testing can be automated by generating crafted inputs as an attacker would do, solutions to automate the test oracle, i.e., distinguishing correct from incorrect outputs for a given input, remain preliminary. Specifically, previous work has demonstrated the potential of metamorphic testing; indeed, security failures can be determined by metamorphic relations that turn valid inputs into malicious inputs and compare their outputs. However, without further guidance, metamorphic relations should be executed on a very large set of valid inputs, which is time consuming and makes metamorphic testing impractical. Hence, in this study, we propose AIM, an approach that automatically selects inputs to reduce testing costs while preserving vulnerability detection capabilities. AIM includes a clustering-based black box approach, identifying similar inputs based on their security properties. It also presents a novel genetic algorithm able to efficiently select diverse inputs while minimizing their total cost. Further, it contains a problem reduction component to reduce the search space and speed up the minimization process. We evaluated the effectiveness of AIM on two well-known web systems, Jenkins and Joomla. We compared AIM's results with four baselines in security testing. Overall, AIM reduced MRs execution time by 84 percent for Jenkins and 82 percent for Joomla while preserving full vulnerability detection. Furthermore, AIM outperformed all the considered baselines regarding vulnerability coverage. Although it has been tuned to work with Web system inputs, AIM could be applied to minimize metamorphic testing cost in other contexts.

Title: A Condensed Transition Graph Framework for Zero-shot Link Prediction with Large Language Models

• Authors: Mingchen Li, Chen Ling, Rui Zhang, Liang Zhao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.10779
• Pdf URL: https://arxiv.org/pdf/2402.10779
• Copy Paste: [[2402.10779]] A Condensed Transition Graph Framework for Zero-shot Link Prediction with Large Language Models(https://arxiv.org/abs/2402.10779)
• Keywords: large language model
• Abstract: Zero-shot link prediction (ZSLP) on knowledge graphs aims at automatically identifying relations between given entities. Existing methods primarily employ auxiliary information to predict tail entity given head entity and its relation, yet face challenges due to the occasional unavailability of such detailed information and the inherent simplicity of predicting tail entities based on semantic similarities. Even though Large Language Models (LLMs) offer a promising solution to predict unobserved relations between the head and tail entity in a zero-shot manner, their performance is still restricted due to the inability to leverage all the (exponentially many) paths' information between two entities, which are critical in collectively indicating their relation types. To address this, in this work, we introduce a Condensed Transition Graph Framework for Zero-Shot Link Prediction (CTLP), which encodes all the paths' information in linear time complexity to predict unseen relations between entities, attaining both efficiency and information preservation. Specifically, we design a condensed transition graph encoder with theoretical guarantees on its coverage, expressiveness, and efficiency. It is learned by a transition graph contrastive learning strategy. Subsequently, we design a soft instruction tuning to learn and map the all-path embedding to the input of LLMs. Experimental results show that our proposed CTLP method achieves state-of-the-art performance on three standard ZSLP datasets

Title: EdgeQAT: Entropy and Distribution Guided Quantization-Aware Training for the Acceleration of Lightweight LLMs on the Edge

• Authors: Xuan Shen, Zhenglun Kong, Changdi Yang, Zhaoyang Han, Lei Lu, Peiyan Dong, Cheng Lyu, Chih-hsiang Li, Xuehang Guo, Zhihao Shu, Wei Niu, Miriam Leeser, Pu Zhao, Yanzhi Wang
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.10787
• Pdf URL: https://arxiv.org/pdf/2402.10787
• Copy Paste: [[2402.10787]] EdgeQAT: Entropy and Distribution Guided Quantization-Aware Training for the Acceleration of Lightweight LLMs on the Edge(https://arxiv.org/abs/2402.10787)
• Keywords: large language model
• Abstract: Despite the remarkable strides of Large Language Models (LLMs) in various fields, the wide applications of LLMs on edge devices are limited due to their massive parameters and computations. To address this, quantization is commonly adopted to generate lightweight LLMs with efficient computations and fast inference. However, Post-Training Quantization (PTQ) methods dramatically degrade in quality when quantizing weights, activations, and KV cache together to below 8 bits. Besides, many Quantization-Aware Training (QAT) works quantize model weights, leaving the activations untouched, which do not fully exploit the potential of quantization for inference acceleration on the edge. In this paper, we propose EdgeQAT, the Entropy and Distribution Guided QAT for the optimization of lightweight LLMs to achieve inference acceleration on Edge devices. We first identify that the performance drop of quantization primarily stems from the information distortion in quantized attention maps, demonstrated by the different distributions in quantized query and key of the self-attention mechanism. Then, the entropy and distribution guided QAT is proposed to mitigate the information distortion. Moreover, we design a token importance-aware adaptive method to dynamically quantize the tokens with different bit widths for further optimization and acceleration. Our extensive experiments verify the substantial improvements with our framework across various datasets. Furthermore, we achieve an on-device speedup of up to 2.37x compared with its FP16 counterparts across multiple edge devices, signaling a groundbreaking advancement.

Title: In Search of Needles in a 10M Haystack: Recurrent Memory Finds What LLMs Miss

• Authors: Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Dmitry Sorokin, Artyom Sorokin, Mikhail Burtsev
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.10790
• Pdf URL: https://arxiv.org/pdf/2402.10790
• Copy Paste: [[2402.10790]] In Search of Needles in a 10M Haystack: Recurrent Memory Finds What LLMs Miss(https://arxiv.org/abs/2402.10790)
• Keywords: transformer, generative
• Abstract: This paper addresses the challenge of processing long documents using generative transformer models. To evaluate different approaches, we introduce BABILong, a new benchmark designed to assess model capabilities in extracting and processing distributed facts within extensive texts. Our evaluation, which includes benchmarks for GPT-4 and RAG, reveals that common methods are effective only for sequences up to $10^4$ elements. In contrast, fine-tuning GPT-2 with recurrent memory augmentations enables it to handle tasks involving up to $10^7$ elements. This achievement marks a substantial leap, as it is by far the longest input processed by any open neural network model to date, demonstrating a significant improvement in the processing capabilities for long sequences.

Title: Diversified Ensembling: An Experiment in Crowdsourced Machine Learning

• Authors: Ira Globus-Harris, Declan Harrison, Michael Kearns, Pietro Perona, Aaron Roth
• Subjects: cs.LG, cs.CY
• Abstract URL: https://arxiv.org/abs/2402.10795
• Pdf URL: https://arxiv.org/pdf/2402.10795
• Copy Paste: [[2402.10795]] Diversified Ensembling: An Experiment in Crowdsourced Machine Learning(https://arxiv.org/abs/2402.10795)
• Keywords: fair
• Abstract: Crowdsourced machine learning on competition platforms such as Kaggle is a popular and often effective method for generating accurate models. Typically, teams vie for the most accurate model, as measured by overall error on a holdout set, and it is common towards the end of such competitions for teams at the top of the leaderboard to ensemble or average their models outside the platform mechanism to get the final, best global model. In arXiv:2201.10408, the authors developed an alternative crowdsourcing framework in the context of fair machine learning, in order to integrate community feedback into models when subgroup unfairness is present and identifiable. There, unlike in classical crowdsourced ML, participants deliberately specialize their efforts by working on subproblems, such as demographic subgroups in the service of fairness. Here, we take a broader perspective on this work: we note that within this framework, participants may both specialize in the service of fairness and simply to cater to their particular expertise (e.g., focusing on identifying bird species in an image classification task). Unlike traditional crowdsourcing, this allows for the diversification of participants' efforts and may provide a participation mechanism to a larger range of individuals (e.g. a machine learning novice who has insight into a specific fairness concern). We present the first medium-scale experimental evaluation of this framework, with 46 participating teams attempting to generate models to predict income from American Community Survey data. We provide an empirical analysis of teams' approaches, and discuss the novel system architecture we developed. From here, we give concrete guidance for how best to deploy such a framework.

Title: VATr++: Choose Your Words Wisely for Handwritten Text Generation

• Authors: Bram Vanherle, Vittorio Pippi, Silvia Cascianelli, Nick Michiels, Frank Van Reeth, Rita Cucchiara
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.10798
• Pdf URL: https://arxiv.org/pdf/2402.10798
• Copy Paste: [[2402.10798]] VATr++: Choose Your Words Wisely for Handwritten Text Generation(https://arxiv.org/abs/2402.10798)
• Keywords: fair, diffusion, transformer
• Abstract: Styled Handwritten Text Generation (HTG) has received significant attention in recent years, propelled by the success of learning-based solutions employing GANs, Transformers, and, preliminarily, Diffusion Models. Despite this surge in interest, there remains a critical yet understudied aspect - the impact of the input, both visual and textual, on the HTG model training and its subsequent influence on performance. This study delves deeper into a cutting-edge Styled-HTG approach, proposing strategies for input preparation and training regularization that allow the model to achieve better performance and generalize better. These aspects are validated through extensive analysis on several different settings and datasets. Moreover, in this work, we go beyond performance optimization and address a significant hurdle in HTG research - the lack of a standardized evaluation protocol. In particular, we propose a standardization of the evaluation protocol for HTG and conduct a comprehensive benchmarking of existing approaches. By doing so, we aim to establish a foundation for fair and meaningful comparisons between HTG strategies, fostering progress in the field.

Title: Quantifying the Persona Effect in LLM Simulations

• Authors: Tiancheng Hu, Nigel Collier
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2402.10811
• Pdf URL: https://arxiv.org/pdf/2402.10811
• Copy Paste: [[2402.10811]] Quantifying the Persona Effect in LLM Simulations(https://arxiv.org/abs/2402.10811)
• Keywords: large language model
• Abstract: Large language models (LLMs) have shown remarkable promise in simulating human language use and behavior. In this study, we delve into the intersection of persona variables and the capability of LLMs to simulate different perspectives. We find that persona variables can explain <10\% variance in annotations in existing subjective NLP datasets. Nonetheless, incorporating them via prompting in LLMs provides modest improvement. Persona prompting is most effective on data samples where disagreements among annotators are frequent yet confined to a limited range. A linear correlation exists: the more persona variables influence human annotations, the better LLMs predictions are using persona prompting. However, when the utility of persona variables is low (i.e., explaining <10\% of human annotations), persona prompting has little effect. Most subjective NLP datasets fall into this category, casting doubt on simulating diverse perspectives in the current NLP landscape.

Title: TernaryVote: Differentially Private, Communication Efficient, and Byzantine Resilient Distributed Optimization on Heterogeneous Data

• Authors: Richeng Jin, Yujie Gu, Kai Yue, Xiaofan He, Zhaoyang Zhang, Huaiyu Dai
• Subjects: cs.LG, cs.CR, eess.SP
• Abstract URL: https://arxiv.org/abs/2402.10816
• Pdf URL: https://arxiv.org/pdf/2402.10816
• Copy Paste: [[2402.10816]] TernaryVote: Differentially Private, Communication Efficient, and Byzantine Resilient Distributed Optimization on Heterogeneous Data(https://arxiv.org/abs/2402.10816)
• Keywords: privacy, attack, robust
• Abstract: Distributed training of deep neural networks faces three critical challenges: privacy preservation, communication efficiency, and robustness to fault and adversarial behaviors. Although significant research efforts have been devoted to addressing these challenges independently, their synthesis remains less explored. In this paper, we propose TernaryVote, which combines a ternary compressor and the majority vote mechanism to realize differential privacy, gradient compression, and Byzantine resilience simultaneously. We theoretically quantify the privacy guarantee through the lens of the emerging f-differential privacy (DP) and the Byzantine resilience of the proposed algorithm. Particularly, in terms of privacy guarantees, compared to the existing sign-based approach StoSign, the proposed method improves the dimension dependence on the gradient size and enjoys privacy amplification by mini-batch sampling while ensuring a comparable convergence rate. We also prove that TernaryVote is robust when less than 50% of workers are blind attackers, which matches that of SIGNSGD with majority vote. Extensive experimental results validate the effectiveness of the proposed algorithm.

Title: Training Class-Imbalanced Diffusion Model Via Overlap Optimization

• Authors: Divin Yan, Lu Qi, Vincent Tao Hu, Ming-Hsuan Yang, Meng Tang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.10821
• Pdf URL: https://arxiv.org/pdf/2402.10821
• Copy Paste: [[2402.10821]] Training Class-Imbalanced Diffusion Model Via Overlap Optimization(https://arxiv.org/abs/2402.10821)
• Keywords: diffusion, generative
• Abstract: Diffusion models have made significant advances recently in high-quality image synthesis and related tasks. However, diffusion models trained on real-world datasets, which often follow long-tailed distributions, yield inferior fidelity for tail classes. Deep generative models, including diffusion models, are biased towards classes with abundant training images. To address the observed appearance overlap between synthesized images of rare classes and tail classes, we propose a method based on contrastive learning to minimize the overlap between distributions of synthetic images for different classes. We show variants of our probabilistic contrastive learning method can be applied to any class conditional diffusion model. We show significant improvement in image synthesis using our loss for multiple datasets with long-tailed distribution. Extensive experimental results demonstrate that the proposed method can effectively handle imbalanced data for diffusion-based generation and classification models. Our code and datasets will be publicly available at https://github.com/yanliang3612/DiffROP.

Title: Time Series Forecasting with LLMs: Understanding and Enhancing Model Capabilities

• Authors: Mingyu Jin, Hua Tang, Chong Zhang, Qinkai Yu, Chengzhi Liu, Suiyuan Zhu, Yongfeng Zhang, Mengnan Du
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.10835
• Pdf URL: https://arxiv.org/pdf/2402.10835
• Copy Paste: [[2402.10835]] Time Series Forecasting with LLMs: Understanding and Enhancing Model Capabilities(https://arxiv.org/abs/2402.10835)
• Keywords: large language model
• Abstract: Large language models (LLMs) have been applied in many fields with rapid development in recent years. As a classic machine learning task, time series forecasting has recently received a boost from LLMs. However, there is a research gap in the LLMs' preferences in this field. In this paper, by comparing LLMs with traditional models, many properties of LLMs in time series prediction are found. For example, our study shows that LLMs excel in predicting time series with clear patterns and trends but face challenges with datasets lacking periodicity. We explain our findings through designing prompts to require LLMs to tell the period of the datasets. In addition, the input strategy is investigated, and it is found that incorporating external knowledge and adopting natural language paraphrases positively affects the predictive performance of LLMs for time series. Overall, this study contributes to insight into the advantages and limitations of LLMs in time series forecasting under different conditions.

Title: FedD2S: Personalized Data-Free Federated Knowledge Distillation

• Authors: Kawa Atapour, S. Jamal Seyedmohammadi, Jamshid Abouei, Arash Mohammadi, Konstantinos N. Plataniotis
• Subjects: cs.LG, cs.AI, cs.DC, eess.IV
• Abstract URL: https://arxiv.org/abs/2402.10846
• Pdf URL: https://arxiv.org/pdf/2402.10846
• Copy Paste: [[2402.10846]] FedD2S: Personalized Data-Free Federated Knowledge Distillation(https://arxiv.org/abs/2402.10846)
• Keywords: federate, fair, data-free
• Abstract: This paper addresses the challenge of mitigating data heterogeneity among clients within a Federated Learning (FL) framework. The model-drift issue, arising from the noniid nature of client data, often results in suboptimal personalization of a global model compared to locally trained models for each client. To tackle this challenge, we propose a novel approach named FedD2S for Personalized Federated Learning (pFL), leveraging knowledge distillation. FedD2S incorporates a deep-to-shallow layer-dropping mechanism in the data-free knowledge distillation process to enhance local model personalization. Through extensive simulations on diverse image datasets-FEMNIST, CIFAR10, CINIC0, and CIFAR100-we compare FedD2S with state-of-the-art FL baselines. The proposed approach demonstrates superior performance, characterized by accelerated convergence and improved fairness among clients. The introduced layer-dropping technique effectively captures personalized knowledge, resulting in enhanced performance compared to alternative FL models. Moreover, we investigate the impact of key hyperparameters, such as the participation ratio and layer-dropping rate, providing valuable insights into the optimal configuration for FedD2S. The findings demonstrate the efficacy of adaptive layer-dropping in the knowledge distillation process to achieve enhanced personalization and performance across diverse datasets and tasks.

Title: Enhancement-Driven Pretraining for Robust Fingerprint Representation Learning

• Authors: Ekta Gavas, Kaustubh Olpadkar, Anoop Namboodiri
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.10847
• Pdf URL: https://arxiv.org/pdf/2402.10847
• Copy Paste: [[2402.10847]] Enhancement-Driven Pretraining for Robust Fingerprint Representation Learning(https://arxiv.org/abs/2402.10847)
• Keywords: robust, biometric
• Abstract: Fingerprint recognition stands as a pivotal component of biometric technology, with diverse applications from identity verification to advanced search tools. In this paper, we propose a unique method for deriving robust fingerprint representations by leveraging enhancement-based pre-training. Building on the achievements of U-Net-based fingerprint enhancement, our method employs a specialized encoder to derive representations from fingerprint images in a self-supervised manner. We further refine these representations, aiming to enhance the verification capabilities. Our experimental results, tested on publicly available fingerprint datasets, reveal a marked improvement in verification performance against established self-supervised training techniques. Our findings not only highlight the effectiveness of our method but also pave the way for potential advancements. Crucially, our research indicates that it is feasible to extract meaningful fingerprint representations from degraded images without relying on enhanced samples.

Title: Control Color: Multimodal Diffusion-based Interactive Image Colorization

• Authors: Zhexin Liang, Zhaochen Li, Shangchen Zhou, Chongyi Li, Chen Change Loy
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.10855
• Pdf URL: https://arxiv.org/pdf/2402.10855
• Copy Paste: [[2402.10855]] Control Color: Multimodal Diffusion-based Interactive Image Colorization(https://arxiv.org/abs/2402.10855)
• Keywords: diffusion
• Abstract: Despite the existence of numerous colorization methods, several limitations still exist, such as lack of user interaction, inflexibility in local colorization, unnatural color rendering, insufficient color variation, and color overflow. To solve these issues, we introduce Control Color (CtrlColor), a multi-modal colorization method that leverages the pre-trained Stable Diffusion (SD) model, offering promising capabilities in highly controllable interactive image colorization. While several diffusion-based methods have been proposed, supporting colorization in multiple modalities remains non-trivial. In this study, we aim to tackle both unconditional and conditional image colorization (text prompts, strokes, exemplars) and address color overflow and incorrect color within a unified framework. Specifically, we present an effective way to encode user strokes to enable precise local color manipulation and employ a practical way to constrain the color distribution similar to exemplars. Apart from accepting text prompts as conditions, these designs add versatility to our approach. We also introduce a novel module based on self-attention and a content-guided deformable autoencoder to address the long-standing issues of color overflow and inaccurate coloring. Extensive comparisons show that our model outperforms state-of-the-art image colorization methods both qualitatively and quantitatively.

Title: Differential Private Federated Transfer Learning for Mental Health Monitoring in Everyday Settings: A Case Study on Stress Detection

• Authors: Ziyu Wang, Zhongqi Yang, Iman Azimi, Amir M. Rahmani
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2402.10862
• Pdf URL: https://arxiv.org/pdf/2402.10862
• Copy Paste: [[2402.10862]] Differential Private Federated Transfer Learning for Mental Health Monitoring in Everyday Settings: A Case Study on Stress Detection(https://arxiv.org/abs/2402.10862)
• Keywords: privacy, protect, attack, federate
• Abstract: Mental health conditions, prevalent across various demographics, necessitate efficient monitoring to mitigate their adverse impacts on life quality. The surge in data-driven methodologies for mental health monitoring has underscored the importance of privacy-preserving techniques in handling sensitive health data. Despite strides in federated learning for mental health monitoring, existing approaches struggle with vulnerabilities to certain cyber-attacks and data insufficiency in real-world applications. In this paper, we introduce a differential private federated transfer learning framework for mental health monitoring to enhance data privacy and enrich data sufficiency. To accomplish this, we integrate federated learning with two pivotal elements: (1) differential privacy, achieved by introducing noise into the updates, and (2) transfer learning, employing a pre-trained universal model to adeptly address issues of data imbalance and insufficiency. We evaluate the framework by a case study on stress detection, employing a dataset of physiological and contextual data from a longitudinal study. Our finding show that the proposed approach can attain a 10% boost in accuracy and a 21% enhancement in recall, while ensuring privacy protection.

Title: EcoRank: Budget-Constrained Text Re-ranking Using Large Language Models

• Authors: Muhammad Shihab Rashid, Jannat Ara Meem, Yue Dong, Vagelis Hristidis
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.10866
• Pdf URL: https://arxiv.org/pdf/2402.10866
• Copy Paste: [[2402.10866]] EcoRank: Budget-Constrained Text Re-ranking Using Large Language Models(https://arxiv.org/abs/2402.10866)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have achieved state-of-the-art performance in text re-ranking. This process includes queries and candidate passages in the prompts, utilizing pointwise, listwise, and pairwise prompting strategies. A limitation of these ranking strategies with LLMs is their cost: the process can become expensive due to API charges, which are based on the number of input and output tokens. We study how to maximize the re-ranking performance given a budget, by navigating the vast search spaces of prompt choices, LLM APIs, and budget splits. We propose a suite of budget-constrained methods to perform text re-ranking using a set of LLM APIs. Our most efficient method, called EcoRank, is a two-layered pipeline that jointly optimizes decisions regarding budget allocation across prompt strategies and LLM APIs. Our experimental results on four popular QA and passage reranking datasets show that EcoRank outperforms other budget-aware supervised and unsupervised baselines.

Title: Lightweight ciphers based on chaotic Map -- LFSR architectures

• Authors: M. Garcia-Bosque, C. Sánchez-Azqueta, G. Royo, S. Celma
• Subjects: cs.CR, eess.SP
• Abstract URL: https://arxiv.org/abs/2402.10871
• Pdf URL: https://arxiv.org/pdf/2402.10871
• Copy Paste: [[2402.10871]] Lightweight ciphers based on chaotic Map -- LFSR architectures(https://arxiv.org/abs/2402.10871)
• Keywords: secure
• Abstract: In this paper, we propose and analyze two different stream ciphers based on a Skew Tent Map and a Modified Logistic Map respectively. In order to improve the randomness of these systems, a single method for increasing the period length of the generated sequences has been applied. The results prove that the randomness of these systems can be severally increased by using this method, making these systems suitable for secure communications.

Title: Multi-modal preference alignment remedies regression of visual instruction tuning on language model

• Authors: Shengzhi Li, Rongyu Lin, Shichao Pei
• Subjects: cs.CL, cs.AI, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.10884
• Pdf URL: https://arxiv.org/pdf/2402.10884
• Copy Paste: [[2402.10884]] Multi-modal preference alignment remedies regression of visual instruction tuning on language model(https://arxiv.org/abs/2402.10884)
• Keywords: large language model
• Abstract: In production, multi-modal large language models (MLLMs) are expected to support multi-turn queries of interchanging image and text modalities. However, the current MLLMs trained with visual-question-answering (VQA) datasets could suffer from degradation, as VQA datasets lack the diversity and complexity of the original text instruction datasets which the underlying language model had been trained with. To address this challenging degradation, we first collect a lightweight (6k entries) VQA preference dataset where answers were annotated by Gemini for 5 quality metrics in a granular fashion, and investigate standard Supervised Fine-tuning, rejection sampling, Direct Preference Optimization (DPO), and SteerLM. Our findings indicate that the with DPO we are able to surpass instruction-following capabilities of the language model, achieving a 6.73 score on MT-Bench, compared to Vicuna's 6.57 and LLaVA's 5.99 despite small data scale. This enhancement in textual instruction proficiency correlates with boosted visual instruction performance (+4.9\% on MM-Vet, +6\% on LLaVA-Bench), with minimal alignment tax on visual knowledge benchmarks compared to previous RLHF approach. In conclusion, we propose a distillation-based multi-modal alignment model with fine-grained annotations on a small dataset that reconciles the textual and visual performance of MLLMs, restoring and boosting language capability after visual instruction tuning.

Title: When is Tree Search Useful for LLM Planning? It Depends on the Discriminator

• Authors: Ziru Chen, Michael White, Raymond Mooney, Ali Payani, Yu Su, Huan Sun
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.10890
• Pdf URL: https://arxiv.org/pdf/2402.10890
• Copy Paste: [[2402.10890]] When is Tree Search Useful for LLM Planning? It Depends on the Discriminator(https://arxiv.org/abs/2402.10890)
• Keywords: large language model
• Abstract: In this paper, we examine how large language models (LLMs) solve multi-step problems under a language agent framework with three components: a generator, a discriminator, and a planning method. We investigate the practical utility of two advanced planning methods, iterative correction and tree search. We present a comprehensive analysis of how discrimination accuracy affects the overall performance of agents when using these two methods or a simpler method, re-ranking. Experiments on two tasks, text-to-SQL parsing and mathematical reasoning, show that: (1) advanced planning methods demand discriminators with at least 90% accuracy to achieve significant improvements over re-ranking; (2) current LLMs' discrimination abilities have not met the needs of advanced planning methods to achieve such improvements; (3) with LLM-based discriminators, advanced planning methods may not adequately balance accuracy and efficiency. For example, compared to the other two methods, tree search is at least 10--20 times slower but leads to negligible performance gains, which hinders its real-world applications. Code and data will be released at https://github.com/OSU-NLP-Group/llm-planning-eval.

Title: Instruction Diversity Drives Generalization To Unseen Tasks

• Authors: Dylan Zhang, Justin Wang, Francois Charton
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.10891
• Pdf URL: https://arxiv.org/pdf/2402.10891
• Copy Paste: [[2402.10891]] Instruction Diversity Drives Generalization To Unseen Tasks(https://arxiv.org/abs/2402.10891)
• Keywords: robust, large language model
• Abstract: Instruction tuning -- fine-tuning a large language model (LLM) on pairs of instructions and desired outcomes -- is an approach that enables pre-trained language models to perform real-world tasks and follow human instructions. Its practical success depends on the model learning a broader set of instructions than those it was trained on. Yet the factors that determine model generalization to such \emph{unseen tasks} are not well understood. %To understand the driving factors of generalization, In this paper, we experiment with string rewrites, a symbolic task that serves as a building block for Turing complete Markov algorithms while allowing experimental control of "inputs" and "instructions". We investigate the trade-off between the number of instructions the model is trained on and the number of training samples provided for each instruction and observe that the diversity of the instruction set determines generalization. Generalization emerges once a diverse enough set of tasks is provided, even though very few examples are provided for each task. Instruction diversity also ensures robustness with respect to non-uniform distributions of instructions in the training set.

Title: Proving membership in LLM pretraining data via data watermarks

• Authors: Johnny Tian-Zheng Wei, Ryan Yixiang Wang, Robin Jia
• Subjects: cs.CR, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.10892
• Pdf URL: https://arxiv.org/pdf/2402.10892
• Copy Paste: [[2402.10892]] Proving membership in LLM pretraining data via data watermarks(https://arxiv.org/abs/2402.10892)
• Keywords: robust, watermark
• Abstract: Detecting whether copyright holders' works were used in LLM pretraining is poised to be an important problem. This work proposes using data watermarks to enable principled detection with only black-box model access, provided that the rightholder contributed multiple training documents and watermarked them before public release. By applying a randomly sampled data watermark, detection can be framed as hypothesis testing, which provides guarantees on the false detection rate. We study two watermarks: one that inserts random sequences, and another that randomly substitutes characters with Unicode lookalikes. We first show how three aspects of watermark design -- watermark length, number of duplications, and interference -- affect the power of the hypothesis test. Next, we study how a watermark's detection strength changes under model and dataset scaling: while increasing the dataset size decreases the strength of the watermark, watermarks remain strong if the model size also increases. Finally, we view SHA hashes as natural watermarks and show that we can robustly detect hashes from BLOOM-176B's training data, as long as they occurred at least 90 times. Together, our results point towards a promising future for data watermarks in real world use.

Title: RLVF: Learning from Verbal Feedback without Overgeneralization

• Authors: Moritz Stephan, Alexander Khazatsky, Eric Mitchell, Annie S Chen, Sheryl Hsu, Archit Sharma, Chelsea Finn
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.10893
• Pdf URL: https://arxiv.org/pdf/2402.10893
• Copy Paste: [[2402.10893]] RLVF: Learning from Verbal Feedback without Overgeneralization(https://arxiv.org/abs/2402.10893)
• Keywords: large language model
• Abstract: The diversity of contexts in which large language models (LLMs) are deployed requires the ability to modify or customize default model behaviors to incorporate nuanced requirements and preferences. A convenient interface to specify such model adjustments is high-level verbal feedback, such as "Don't use emojis when drafting emails to my boss." However, while writing high-level feedback is far simpler than collecting annotations for reinforcement learning from human feedback (RLHF), we find that simply prompting a model with such feedback leads to overgeneralization of the feedback to contexts where it is not relevant. We study the problem of incorporating verbal feedback without such overgeneralization, inspiring a new method Contextualized Critiques with Constrained Preference Optimization (C3PO). C3PO uses a piece of high-level feedback to generate a small synthetic preference dataset specifying how the feedback should (and should not) be applied. It then fine-tunes the model in accordance with the synthetic preference data while minimizing the divergence from the original model for prompts where the feedback does not apply. Our experimental results indicate that our approach effectively applies verbal feedback to relevant scenarios while preserving existing behaviors for other contexts. For both human- and GPT-4-generated high-level feedback, C3PO effectively adheres to the given feedback comparably to in-context baselines while reducing overgeneralization by 30%.

Title: Fusion of Diffusion Weighted MRI and Clinical Data for Predicting Functional Outcome after Acute Ischemic Stroke with Deep Contrastive Learning

• Authors: Chia-Ling Tsai, Hui-Yun Su, Shen-Feng Sung, Wei-Yang Lin, Ying-Ying Su, Tzu-Hsien Yang, Man-Lin Mai
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.10894
• Pdf URL: https://arxiv.org/pdf/2402.10894
• Copy Paste: [[2402.10894]] Fusion of Diffusion Weighted MRI and Clinical Data for Predicting Functional Outcome after Acute Ischemic Stroke with Deep Contrastive Learning(https://arxiv.org/abs/2402.10894)
• Keywords: diffusion
• Abstract: Stroke is a common disabling neurological condition that affects about one-quarter of the adult population over age 25; more than half of patients still have poor outcomes, such as permanent functional dependence or even death, after the onset of acute stroke. The aim of this study is to investigate the efficacy of diffusion-weighted MRI modalities combining with structured health profile on predicting the functional outcome to facilitate early intervention. A deep fusion learning network is proposed with two-stage training: the first stage focuses on cross-modality representation learning and the second stage on classification. Supervised contrastive learning is exploited to learn discriminative features that separate the two classes of patients from embeddings of individual modalities and from the fused multimodal embedding. The network takes as the input DWI and ADC images, and structured health profile data. The outcome is the prediction of the patient needing long-term care at 3 months after the onset of stroke. Trained and evaluated with a dataset of 3297 patients, our proposed fusion model achieves 0.87, 0.80 and 80.45% for AUC, F1-score and accuracy, respectively, outperforming existing models that consolidate both imaging and structured data in the medical domain. If trained with comprehensive clinical variables, including NIHSS and comorbidities, the gain from images on making accurate prediction is not considered substantial, but significant. However, diffusion-weighted MRI can replace NIHSS to achieve comparable level of accuracy combining with other readily available clinical variables for better generalization.

Title: PaLM2-VAdapter: Progressively Aligned Language Model Makes a Strong Vision-language Adapter

• Authors: Junfei Xiao, Zheng Xu, Alan Yuille, Shen Yan, Boyu Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.10896
• Pdf URL: https://arxiv.org/pdf/2402.10896
• Copy Paste: [[2402.10896]] PaLM2-VAdapter: Progressively Aligned Language Model Makes a Strong Vision-language Adapter(https://arxiv.org/abs/2402.10896)
• Keywords: large language model
• Abstract: This paper demonstrates that a progressively aligned language model can effectively bridge frozen vision encoders and large language models (LLMs). While the fundamental architecture and pre-training methods of vision encoders and LLMs have been extensively studied, the architecture and training strategy of vision-language adapters vary significantly across recent works. Our research undertakes a thorough exploration of the state-of-the-art perceiver resampler architecture and builds a strong baseline. However, we observe that the vision-language alignment with perceiver resampler exhibits slow convergence and limited scalability with a lack of direct supervision. To address this issue, we propose PaLM2-VAdapter, employing a progressively aligned language model as the vision-language adapter. Compared to the strong baseline with perceiver resampler, our method empirically shows faster convergence, higher performance, and stronger scalability. Extensive experiments across various Visual Question Answering (VQA) and captioning tasks on both images and videos demonstrate that our model exhibits state-of-the-art visual understanding and multi-modal reasoning capabilities. Notably, our method achieves these advancements with 30~70% fewer parameters than the state-of-the-art large vision-language models, marking a significant efficiency improvement.