language model

Title: Benefits and Harms of Large Language Models in Digital Mental Health. (arXiv:2311.14693v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.14693
• Code URL: null
• Copy Paste: [[2311.14693]] Benefits and Harms of Large Language Models in Digital Mental Health(http://arxiv.org/abs/2311.14693)
• Summary:

  The past decade has been transformative for mental health research and practice. The ability to harness large repositories of data, whether from electronic health records (EHR), mobile devices, or social media, has revealed a potential for valuable insights into patient experiences, promising early, proactive interventions, as well as personalized treatment plans. Recent developments in generative artificial intelligence, particularly large language models (LLMs), show promise in leading digital mental health to uncharted territory. Patients are arriving at doctors' appointments with information sourced from chatbots, state-of-the-art LLMs are being incorporated in medical software and EHR systems, and chatbots from an ever-increasing number of startups promise to serve as AI companions, friends, and partners. This article presents contemporary perspectives on the opportunities and risks posed by LLMs in the design, development, and implementation of digital mental health tools. We adopt an ecological framework and draw on the affordances offered by LLMs to discuss four application areas -- care-seeking behaviors from individuals in need of care, community care provision, institutional and medical care provision, and larger care ecologies at the societal level. We engage in a thoughtful consideration of whether and how LLM-based technologies could or should be employed for enhancing mental health. The benefits and harms our article surfaces could serve to help shape future research, advocacy, and regulatory efforts focused on creating more responsible, user-friendly, equitable, and secure LLM-based tools for mental health treatment and intervention.

Title: Zero-Shot Question Answering over Financial Documents using Large Language Models. (arXiv:2311.14722v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.14722
• Code URL: null
• Copy Paste: [[2311.14722]] Zero-Shot Question Answering over Financial Documents using Large Language Models(http://arxiv.org/abs/2311.14722)
• Summary:

  We introduce a large language model (LLM) based approach to answer complex questions requiring multi-hop numerical reasoning over financial reports. While LLMs have exhibited remarkable performance on various natural language and reasoning tasks, complex reasoning problems often rely on few-shot prompts that require carefully crafted examples. In contrast, our approach uses novel zero-shot prompts that guide the LLM to encode the required reasoning into a Python program or a domain specific language. The generated program is then executed by a program interpreter, thus mitigating the limitations of LLM in performing accurate arithmetic calculations.

  We evaluate the proposed approach on three financial datasets using some of the recently developed generative pretrained transformer (GPT) models and perform comparisons with various zero-shot baselines. The experimental results demonstrate that our approach significantly improves the accuracy for all the LLMs over their respective baselines. We provide a detailed analysis of the results, generating insights to support our findings. The success of our approach demonstrates the enormous potential to extract complex domain specific numerical reasoning by designing zero-shot prompts to effectively exploit the knowledge embedded in LLMs.

Title: MemoryCompanion: A Smart Healthcare Solution to Empower Efficient Alzheimer's Care Via Unleashing Generative AI. (arXiv:2311.14730v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.14730
• Code URL: null
• Copy Paste: [[2311.14730]] MemoryCompanion: A Smart Healthcare Solution to Empower Efficient Alzheimer's Care Via Unleashing Generative AI(http://arxiv.org/abs/2311.14730)
• Summary:

  With the rise of Large Language Models (LLMs), notably characterized by GPT frameworks, there emerges a catalyst for novel healthcare applications. Earlier iterations of chatbot caregivers, though existent, have yet to achieve a dimension of human-like authenticity. This paper unveils `MemoryCompanion' a pioneering digital health solution explicitly tailored for Alzheimer's disease (AD) patients and their caregivers. Drawing upon the nuances of GPT technology and prompt engineering, MemoryCompanion manifests a personalized caregiving paradigm, fostering interactions via voice-cloning and talking-face mechanisms that resonate with the familiarity of known companions. Using advanced prompt-engineering, the system intricately adapts to each patient's distinct profile, curating its content and communication style accordingly. This approach strives to counteract prevalent issues of social isolation and loneliness frequently observed in AD demographics. Our methodology, grounded in its innovative design, addresses both the caregiving and technological challenges intrinsic to this domain.

Title: Evaluating Large Language Models through Gender and Racial Stereotypes. (arXiv:2311.14788v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.14788
• Code URL: null
• Copy Paste: [[2311.14788]] Evaluating Large Language Models through Gender and Racial Stereotypes(http://arxiv.org/abs/2311.14788)
• Summary:

  Language Models have ushered a new age of AI gaining traction within the NLP community as well as amongst the general population. AI's ability to make predictions, generations and its applications in sensitive decision-making scenarios, makes it even more important to study these models for possible biases that may exist and that can be exaggerated. We conduct a quality comparative study and establish a framework to evaluate language models under the premise of two kinds of biases: gender and race, in a professional setting. We find out that while gender bias has reduced immensely in newer models, as compared to older ones, racial bias still exists.

Title: Optimal Strategies to Perform Multilingual Analysis of Social Content for a Novel Dataset in the Tourism Domain. (arXiv:2311.14727v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.14727
• Code URL: null
• Copy Paste: [[2311.14727]] Optimal Strategies to Perform Multilingual Analysis of Social Content for a Novel Dataset in the Tourism Domain(http://arxiv.org/abs/2311.14727)
• Summary:

  The rising influence of social media platforms in various domains, including tourism, has highlighted the growing need for efficient and automated natural language processing (NLP) approaches to take advantage of this valuable resource. However, the transformation of multilingual, unstructured, and informal texts into structured knowledge often poses significant challenges.

  In this work, we evaluate and compare few-shot, pattern-exploiting and fine-tuning machine learning techniques on large multilingual language models (LLMs) to establish the best strategy to address the lack of annotated data for 3 common NLP tasks in the tourism domain: (1) Sentiment Analysis, (2) Named Entity Recognition, and (3) Fine-grained Thematic Concept Extraction (linked to a semantic resource). Furthermore, we aim to ascertain the quantity of annotated examples required to achieve good performance in those 3 tasks, addressing a common challenge encountered by NLP researchers in the construction of domain-specific datasets.

  Extensive experimentation on a newly collected and annotated multilingual (French, English, and Spanish) dataset composed of tourism-related tweets shows that current few-shot learning techniques allow us to obtain competitive results for all three tasks with very little annotation data: 5 tweets per label (15 in total) for Sentiment Analysis, 10% of the tweets for location detection (around 160) and 13% (200 approx.) of the tweets annotated with thematic concepts, a highly fine-grained sequence labeling task based on an inventory of 315 classes.

  This comparative analysis, grounded in a novel dataset, paves the way for applying NLP to new domain-specific applications, reducing the need for manual annotations and circumventing the complexities of rule-based, ad hoc solutions.

Title: AutoKG: Efficient Automated Knowledge Graph Generation for Language Models. (arXiv:2311.14740v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.14740
• Code URL: null
• Copy Paste: [[2311.14740]] AutoKG: Efficient Automated Knowledge Graph Generation for Language Models(http://arxiv.org/abs/2311.14740)
• Summary:

  Traditional methods of linking large language models (LLMs) to knowledge bases via the semantic similarity search often fall short of capturing complex relational dynamics. To address these limitations, we introduce AutoKG, a lightweight and efficient approach for automated knowledge graph (KG) construction. For a given knowledge base consisting of text blocks, AutoKG first extracts keywords using a LLM and then evaluates the relationship weight between each pair of keywords using graph Laplace learning. We employ a hybrid search scheme combining vector similarity and graph-based associations to enrich LLM responses. Preliminary experiments demonstrate that AutoKG offers a more comprehensive and interconnected knowledge retrieval mechanism compared to the semantic similarity search, thereby enhancing the capabilities of LLMs in generating more insightful and relevant outputs.

Title: A Baseline Analysis of Reward Models' Ability To Accurately Analyze Foundation Models Under Distribution Shift. (arXiv:2311.14743v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.14743
• Code URL: null
• Copy Paste: [[2311.14743]] A Baseline Analysis of Reward Models' Ability To Accurately Analyze Foundation Models Under Distribution Shift(http://arxiv.org/abs/2311.14743)
• Summary:

  Foundation models, specifically Large Language Models (LLM's), have lately gained wide-spread attention and adoption. Reinforcement Learning with Human Feedback (RLHF) involves training a reward model to capture desired behaviors, which is then used to align an LLM. These reward models are additionally used at inference-time to estimate how well LLM responses adhere to those desired behaviors. However, there is little work measuring how robust these reward models are to distribution shifts. In this work, we evaluate how reward model performance - measured via accuracy and calibration (i.e. alignment between accuracy and confidence) - is affected by distribution shift. We show novel calibration patterns and accuracy drops due to OOD prompts and responses, and that the reward model is more sensitive to shifts in responses than prompts. Additionally, we adapt an OOD detection technique commonly used in classification to the reward model setting in order to detect these distribution shifts in prompts and responses.

Title: OpusCleaner and OpusTrainer, open source toolkits for training Machine Translation and Large language models. (arXiv:2311.14838v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.14838
• Code URL: https://github.com/hplt-project/opustrainer
• Copy Paste: [[2311.14838]] OpusCleaner and OpusTrainer, open source toolkits for training Machine Translation and Large language models(http://arxiv.org/abs/2311.14838)
• Summary:

  Developing high quality machine translation systems is a labour intensive, challenging and confusing process for newcomers to the field. We present a pair of tools OpusCleaner and OpusTrainer that aim to simplify the process, reduce the amount of work and lower the entry barrier for newcomers.

  OpusCleaner is a data downloading, cleaning, and proprocessing toolkit. It is designed to allow researchers to quickly download, visualise and preprocess bilingual (or monolingual) data that comes from many different sources, each of them with different quality, issues, and unique filtering/preprocessing requirements.

  OpusTrainer is a data scheduling and data augmenting tool aimed at building large scale, robust machine translation systems and large language models. It features deterministic data mixing from many different sources, on-the-fly data augmentation and more.

  Using these tools, we showcase how we can use it to create high quality machine translation model robust to noisy user input; multilingual models and terminology aware models.

Title: Tracing Influence at Scale: A Contrastive Learning Approach to Linking Public Comments and Regulator Responses. (arXiv:2311.14871v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.14871
• Code URL: null
• Copy Paste: [[2311.14871]] Tracing Influence at Scale: A Contrastive Learning Approach to Linking Public Comments and Regulator Responses(http://arxiv.org/abs/2311.14871)
• Summary:

  U.S. Federal Regulators receive over one million comment letters each year from businesses, interest groups, and members of the public, all advocating for changes to proposed regulations. These comments are believed to have wide-ranging impacts on public policy. However, measuring the impact of specific comments is challenging because regulators are required to respond to comments but they do not have to specify which comments they are addressing. In this paper, we propose a simple yet effective solution to this problem by using an iterative contrastive method to train a neural model aiming for matching text from public comments to responses written by regulators. We demonstrate that our proposal substantially outperforms a set of selected text-matching baselines on a human-annotated test set. Furthermore, it delivers performance comparable to the most advanced gigantic language model (i.e., GPT-4), and is more cost-effective when handling comments and regulator responses matching in larger scale.

Title: Walking a Tightrope -- Evaluating Large Language Models in High-Risk Domains. (arXiv:2311.14966v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.14966
• Code URL: null
• Copy Paste: [[2311.14966]] Walking a Tightrope -- Evaluating Large Language Models in High-Risk Domains(http://arxiv.org/abs/2311.14966)
• Summary:

  High-risk domains pose unique challenges that require language models to provide accurate and safe responses. Despite the great success of large language models (LLMs), such as ChatGPT and its variants, their performance in high-risk domains remains unclear. Our study delves into an in-depth analysis of the performance of instruction-tuned LLMs, focusing on factual accuracy and safety adherence. To comprehensively assess the capabilities of LLMs, we conduct experiments on six NLP datasets including question answering and summarization tasks within two high-risk domains: legal and medical. Further qualitative analysis highlights the existing limitations inherent in current LLMs when evaluating in high-risk domains. This underscores the essential nature of not only improving LLM capabilities but also prioritizing the refinement of domain-specific metrics, and embracing a more human-centric approach to enhance safety and factual reliability. Our findings advance the field toward the concerns of properly evaluating LLMs in high-risk domains, aiming to steer the adaptability of LLMs in fulfilling societal obligations and aligning with forthcoming regulations, such as the EU AI Act.

gpt

Title: Data-to-Text Bilingual Generation. (arXiv:2311.14808v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.14808
• Code URL: null
• Copy Paste: [[2311.14808]] Data-to-Text Bilingual Generation(http://arxiv.org/abs/2311.14808)
• Summary:

  This document illustrates the use of pyrealb for generating two parallel texts (English and French) from a single source of data. The data selection and text organisation processes are shared between the two languages. only language dependent word and phrasing choices are distinct processes. The realized texts thus convey identical information in both languages without the risk of being lost in translation. This is especially important in cases where strict and simultaneous bilingualism is required. We first present the types of applications targeted by this approach and how the pyrealb English and French realizer can be used for achieving this goal in a natural way. We describe an object-oriented organization to ensure a convenient realization in both languages. To illustrate the process, different types of applications are then briefly sketched with links to the source code. A brief comparison of the text generation is given with the output of an instance of a GPT.

Title: Reinforcement Learning from Statistical Feedback: the Journey from AB Testing to ANT Testing. (arXiv:2311.14766v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.14766
• Code URL: null
• Copy Paste: [[2311.14766]] Reinforcement Learning from Statistical Feedback: the Journey from AB Testing to ANT Testing(http://arxiv.org/abs/2311.14766)
• Summary:

  Reinforcement Learning from Human Feedback (RLHF) has played a crucial role in the success of large models such as ChatGPT. RLHF is a reinforcement learning framework which combines human feedback to improve learning effectiveness and performance. However, obtaining preferences feedback manually is quite expensive in commercial applications. Some statistical commercial indicators are usually more valuable and always ignored in RLHF. There exists a gap between commercial target and model training. In our research, we will attempt to fill this gap with statistical business feedback instead of human feedback, using AB testing which is a well-established statistical method. Reinforcement Learning from Statistical Feedback (RLSF) based on AB testing is proposed. Statistical inference methods are used to obtain preferences for training the reward network, which fine-tunes the pre-trained model in reinforcement learning framework, achieving greater business value. Furthermore, we extend AB testing with double selections at a single time-point to ANT testing with multiple selections at different feedback time points. Moreover, we design numerical experiences to validate the effectiveness of our algorithm framework.

llm

Title: LLM-Assisted Code Cleaning For Training Accurate Code Generators. (arXiv:2311.14904v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.14904
• Code URL: null
• Copy Paste: [[2311.14904]] LLM-Assisted Code Cleaning For Training Accurate Code Generators(http://arxiv.org/abs/2311.14904)
• Summary:

  Natural language to code generation is an important application area of LLMs and has received wide attention from the community. The majority of relevant studies have exclusively concentrated on increasing the quantity and functional correctness of training sets while disregarding other stylistic elements of programs. More recently, data quality has garnered a lot of interest and multiple works have showcased its importance for improving performance. In this work, we investigate data quality for code and find that making the code more structured and readable leads to improved code generation performance of the system. We build a novel data-cleaning pipeline that uses these principles to transform existing programs by 1.) renaming variables, 2.) modularizing and decomposing complex code into smaller helper sub-functions, and 3.) inserting natural-language based plans via LLM based transformations. We evaluate our approach on two challenging algorithmic code generation benchmarks and find that fine-tuning CodeLLaMa-7B on our transformed modularized programs improves the performance by up to 30% compared to fine-tuning on the original dataset. Additionally, we demonstrate improved performance from using a smaller amount of higher-quality data, finding that a model fine-tuned on the entire original dataset is outperformed by a model trained on 15% of our cleaned dataset. Even in comparison to closed-source models, our models outperform the much larger AlphaCoder models.

long context

lora

Title: Exploring Causal Learning through Graph Neural Networks: An In-depth Review. (arXiv:2311.14994v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.14994
• Code URL: null
• Copy Paste: [[2311.14994]] Exploring Causal Learning through Graph Neural Networks: An In-depth Review(http://arxiv.org/abs/2311.14994)
• Summary:

  In machine learning, exploring data correlations to predict outcomes is a fundamental task. Recognizing causal relationships embedded within data is pivotal for a comprehensive understanding of system dynamics, the significance of which is paramount in data-driven decision-making processes. Beyond traditional methods, there has been a surge in the use of graph neural networks (GNNs) for causal learning, given their capabilities as universal data approximators. Thus, a thorough review of the advancements in causal learning using GNNs is both relevant and timely. To structure this review, we introduce a novel taxonomy that encompasses various state-of-the-art GNN methods employed in studying causality. GNNs are further categorized based on their applications in the causality domain. We further provide an exhaustive compilation of datasets integral to causal learning with GNNs to serve as a resource for practical study. This review also touches upon the application of causal learning across diverse sectors. We conclude the review with insights into potential challenges and promising avenues for future exploration in this rapidly evolving field of machine learning.

hallucination

prompt

Title: Vector-Quantized Prompt Learning for Paraphrase Generation. (arXiv:2311.14949v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.14949
• Code URL: null
• Copy Paste: [[2311.14949]] Vector-Quantized Prompt Learning for Paraphrase Generation(http://arxiv.org/abs/2311.14949)
• Summary:

  Deep generative modeling of natural languages has achieved many successes, such as producing fluent sentences and translating from one language into another. However, the development of generative modeling techniques for paraphrase generation still lags behind largely due to the challenges in addressing the complex conflicts between expression diversity and semantic preservation. This paper proposes to generate diverse and high-quality paraphrases by exploiting the pre-trained models with instance-dependent prompts. To learn generalizable prompts, we assume that the number of abstract transforming patterns of paraphrase generation (governed by prompts) is finite and usually not large. Therefore, we present vector-quantized prompts as the cues to control the generation of pre-trained models. Extensive experiments demonstrate that the proposed method achieves new state-of-art results on three benchmark datasets, including Quora, Wikianswers, and MSCOCO. We will release all the code upon acceptance.

code

Title: Emotion-Oriented Behavior Model Using Deep Learning. (arXiv:2311.14674v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.14674
• Code URL: null
• Copy Paste: [[2311.14674]] Emotion-Oriented Behavior Model Using Deep Learning(http://arxiv.org/abs/2311.14674)
• Summary:

  Emotions, as a fundamental ingredient of any social interaction, lead to behaviors that represent the effectiveness of the interaction through facial expressions and gestures in humans. Hence an agent must possess the social and cognitive abilities to understand human social parameters and behave accordingly. However, no such emotion-oriented behavior model is presented yet in the existing research. The emotion prediction may generate appropriate agents' behaviors for effective interaction using conversation modality. Considering the importance of emotions, and behaviors, for an agent's social interaction, an Emotion-based Behavior model is presented in this paper for Socio-cognitive artificial agents. The proposed model is implemented using tweets data trained on multiple models like Long Short-Term Memory (LSTM), Convolution Neural Network (CNN) and Bidirectional Encoder Representations from Transformers (BERT) for emotion prediction with an average accuracy of 92%, and 55% respectively. Further, using emotion predictions from CNN-LSTM, the behavior module responds using facial expressions and gestures using Behavioral Markup Language (BML). The accuracy of emotion-based behavior predictions is statistically validated using the 2-tailed Pearson correlation on the data collected from human users through questionnaires. Analysis shows that all emotion-based behaviors accurately depict human-like gestures and facial expressions based on the significant correlation at the 0.01 and 0.05 levels. This study is a steppingstone to a multi-faceted artificial agent interaction based on emotion-oriented behaviors. Cognition has significance regarding social interaction among humans.

Title: E-CORE: Emotion Correlation Enhanced Empathetic Dialogue Generation. (arXiv:2311.15016v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.15016
• Code URL: null
• Copy Paste: [[2311.15016]] E-CORE: Emotion Correlation Enhanced Empathetic Dialogue Generation(http://arxiv.org/abs/2311.15016)
• Summary:

  Achieving empathy is a crucial step toward humanized dialogue systems. Current approaches for empathetic dialogue generation mainly perceive an emotional label to generate an empathetic response conditioned on it, which simply treat emotions independently, but ignore the intrinsic emotion correlation in dialogues, resulting in inaccurate emotion perception and unsuitable response generation. In this paper, we propose a novel emotion correlation enhanced empathetic dialogue generation framework, which comprehensively realizes emotion correlation learning, utilization, and supervising. Specifically, a multi-resolution emotion graph is devised to capture context-based emotion interactions from different resolutions, further modeling emotion correlation. Then we propose an emotion correlation enhanced decoder, with a novel correlation-aware aggregation and soft/hard strategy, respectively improving the emotion perception and response generation. Experimental results on the benchmark dataset demonstrate the superiority of our model in both empathetic perception and expression.

Title: MPCNN: A Novel Matrix Profile Approach for CNN-based Sleep Apnea Classification. (arXiv:2311.15041v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.15041
• Code URL: https://github.com/vinuni-vishc/mpcnn-sleep-apnea
• Copy Paste: [[2311.15041]] MPCNN: A Novel Matrix Profile Approach for CNN-based Sleep Apnea Classification(http://arxiv.org/abs/2311.15041)
• Summary:

  Sleep apnea (SA) is a significant respiratory condition that poses a major global health challenge. Previous studies have investigated several machine and deep learning models for electrocardiogram (ECG)-based SA diagnoses. Despite these advancements, conventional feature extractions derived from ECG signals, such as R-peaks and RR intervals, may fail to capture crucial information encompassed within the complete PQRST segments. In this study, we propose an innovative approach to address this diagnostic gap by delving deeper into the comprehensive segments of the ECG signal. The proposed methodology draws inspiration from Matrix Profile algorithms, which generate an Euclidean distance profile from fixed-length signal subsequences. From this, we derived the Min Distance Profile (MinDP), Max Distance Profile (MaxDP), and Mean Distance Profile (MeanDP) based on the minimum, maximum, and mean of the profile distances, respectively. To validate the effectiveness of our approach, we use the modified LeNet-5 architecture as the primary CNN model, along with two existing lightweight models, BAFNet and SE-MSCNN, for ECG classification tasks. Our extensive experimental results on the PhysioNet Apnea-ECG dataset revealed that with the new feature extraction method, we achieved a per-segment accuracy up to 92.11 \% and a per-recording accuracy of 100\%. Moreover, it yielded the highest correlation compared to state-of-the-art methods, with a correlation coefficient of 0.989. By introducing a new feature extraction method based on distance relationships, we enhanced the performance of certain lightweight models, showing potential for home sleep apnea test (HSAT) and SA detection in IoT devices. The source code for this work is made publicly available in GitHub: https://github.com/vinuni-vishc/MPCNN-Sleep-Apnea.

Title: Code Search Debiasing:Improve Search Results beyond Overall Ranking Performance. (arXiv:2311.14901v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.14901
• Code URL: null
• Copy Paste: [[2311.14901]] Code Search Debiasing:Improve Search Results beyond Overall Ranking Performance(http://arxiv.org/abs/2311.14901)
• Summary:

  Code search engine is an essential tool in software development. Many code search methods have sprung up, focusing on the overall ranking performance of code search. In this paper, we study code search from another perspective by analyzing the bias of code search models. Biased code search engines provide poor user experience, even though they show promising overall performance. Due to different development conventions (e.g., prefer long queries or abbreviations), some programmers will find the engine useful, while others may find it hard to get desirable search results. To mitigate biases, we develop a general debiasing framework that employs reranking to calibrate search results. It can be easily plugged into existing engines and handle new code search biases discovered in the future. Experiments show that our framework can effectively reduce biases. Meanwhile, the overall ranking performance of code search gets improved after debiasing.

Title: Offensive Language Identification in Transliterated and Code-Mixed Bangla. (arXiv:2311.15023v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.15023
• Code URL: null
• Copy Paste: [[2311.15023]] Offensive Language Identification in Transliterated and Code-Mixed Bangla(http://arxiv.org/abs/2311.15023)
• Summary:

  Identifying offensive content in social media is vital for creating safe online communities. Several recent studies have addressed this problem by creating datasets for various languages. In this paper, we explore offensive language identification in texts with transliterations and code-mixing, linguistic phenomena common in multilingual societies, and a known challenge for NLP systems. We introduce TB-OLID, a transliterated Bangla offensive language dataset containing 5,000 manually annotated comments. We train and fine-tune machine learning models on TB-OLID, and we evaluate their results on this dataset. Our results show that English pre-trained transformer-based models, such as fBERT and HateBERT achieve the best performance on this dataset.

Title: Effective Structural Encodings via Local Curvature Profiles. (arXiv:2311.14864v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.14864
• Code URL: null
• Copy Paste: [[2311.14864]] Effective Structural Encodings via Local Curvature Profiles(http://arxiv.org/abs/2311.14864)
• Summary:

  Structural and Positional Encodings can significantly improve the performance of Graph Neural Networks in downstream tasks. Recent literature has begun to systematically investigate differences in the structural properties that these approaches encode, as well as performance trade-offs between them. However, the question of which structural properties yield the most effective encoding remains open. In this paper, we investigate this question from a geometric perspective. We propose a novel structural encoding based on discrete Ricci curvature (Local Curvature Profiles, short LCP) and show that it significantly outperforms existing encoding approaches. We further show that combining local structural encodings, such as LCP, with global positional encodings improves downstream performance, suggesting that they capture complementary geometric information. Finally, we compare different encoding types with (curvature-based) rewiring techniques. Rewiring has recently received a surge of interest due to its ability to improve the performance of Graph Neural Networks by mitigating over-smoothing and over-squashing effects. Our results suggest that utilizing curvature information for structural encodings delivers significantly larger performance increases than rewiring.

Title: Eliminating Domain Bias for Federated Learning in Representation Space. (arXiv:2311.14975v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.14975
• Code URL: https://github.com/tsingz0/dbe
• Copy Paste: [[2311.14975]] Eliminating Domain Bias for Federated Learning in Representation Space(http://arxiv.org/abs/2311.14975)
• Summary:

  Recently, federated learning (FL) is popular for its privacy-preserving and collaborative learning abilities. However, under statistically heterogeneous scenarios, we observe that biased data domains on clients cause a representation bias phenomenon and further degenerate generic representations during local training, i.e., the representation degeneration phenomenon. To address these issues, we propose a general framework Domain Bias Eliminator (DBE) for FL. Our theoretical analysis reveals that DBE can promote bi-directional knowledge transfer between server and client, as it reduces the domain discrepancy between server and client in representation space. Besides, extensive experiments on four datasets show that DBE can greatly improve existing FL methods in both generalization and personalization abilities. The DBE-equipped FL method can outperform ten state-of-the-art personalized FL methods by a large margin. Our code is public at https://github.com/TsingZ0/DBE.

Title: Satellite-based feature extraction and multivariate time-series prediction of biotoxin contamination in shellfish. (arXiv:2311.15000v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.15000
• Code URL: null
• Copy Paste: [[2311.15000]] Satellite-based feature extraction and multivariate time-series prediction of biotoxin contamination in shellfish(http://arxiv.org/abs/2311.15000)
• Summary:

  Shellfish production constitutes an important sector for the economy of many Portuguese coastal regions, yet the challenge of shellfish biotoxin contamination poses both public health concerns and significant economic risks. Thus, predicting shellfish contamination levels holds great potential for enhancing production management and safeguarding public health. In our study, we utilize a dataset with years of Sentinel-3 satellite imagery for marine surveillance, along with shellfish biotoxin contamination data from various production areas along Portugal's western coastline, collected by Portuguese official control. Our goal is to evaluate the integration of satellite data in forecasting models for predicting toxin concentrations in shellfish given forecasting horizons up to four weeks, which implies extracting a small set of useful features and assessing their impact on the predictive models. We framed this challenge as a time-series forecasting problem, leveraging historical contamination levels and satellite images for designated areas. While contamination measurements occurred weekly, satellite images were accessible multiple times per week. Unsupervised feature extraction was performed using autoencoders able to handle non-valid pixels caused by factors like cloud cover, land, or anomalies. Finally, several Artificial Neural Networks models were applied to compare univariate (contamination only) and multivariate (contamination and satellite data) time-series forecasting. Our findings show that incorporating these features enhances predictions, especially beyond one week in lagoon production areas (RIAV) and for the 1-week and 2-week horizons in the L5B area (oceanic). The methodology shows the feasibility of integrating information from a high-dimensional data source like remote sensing without compromising the model's predictive ability.

Title: Training a Hopfield Variational Autoencoder with Equilibrium Propagation. (arXiv:2311.15047v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.15047
• Code URL: null
• Copy Paste: [[2311.15047]] Training a Hopfield Variational Autoencoder with Equilibrium Propagation(http://arxiv.org/abs/2311.15047)
• Summary:

  On dedicated analog hardware, equilibrium propagation is an energy-efficient alternative to backpropagation. In spite of its theoretical guarantees, its application in the AI domain remains limited to the discriminative setting. Meanwhile, despite its high computational demands, generative AI is on the rise. In this paper, we demonstrate the application of Equilibrium Propagation in training a variational autoencoder (VAE) for generative modeling. Leveraging the symmetric nature of Hopfield networks, we propose using a single model to serve as both the encoder and decoder which could effectively halve the required chip size for VAE implementations, paving the way for more efficient analog hardware configurations.

Title: Task adaption by biologically inspired stochastic comodulation. (arXiv:2311.15053v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.15053
• Code URL: null
• Copy Paste: [[2311.15053]] Task adaption by biologically inspired stochastic comodulation(http://arxiv.org/abs/2311.15053)
• Summary:

  Brain representations must strike a balance between generalizability and adaptability. Neural codes capture general statistical regularities in the world, while dynamically adjusting to reflect current goals. One aspect of this adaptation is stochastically co-modulating neurons' gains based on their task relevance. These fluctuations then propagate downstream to guide decision-making. Here, we test the computational viability of such a scheme in the context of multi-task learning. We show that fine-tuning convolutional networks by stochastic gain modulation improves on deterministic gain modulation, achieving state-of-the-art results on the CelebA dataset. To better understand the mechanisms supporting this improvement, we explore how fine-tuning performance is affected by architecture using Cifar-100. Overall, our results suggest that stochastic comodulation can enhance learning efficiency and performance in multi-task learning, without additional learnable parameters. This offers a promising new direction for developing more flexible and robust intelligent systems.

chat

Title: @ve: A Chatbot for Latin. (arXiv:2311.14741v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.14741
• Code URL: null
• Copy Paste: [[2311.14741]] @ve: A Chatbot for Latin(http://arxiv.org/abs/2311.14741)
• Summary:

  Dead, extinct, and endangered languages have been preserved primarily through audio conservation and the collection and digitization of scripts and have been promoted through targeted language acquisition efforts. Another possibility would be to build conversational agents that can master these languages. This would provide an artificial, active conversational partner which has knowledge of the vocabulary and grammar, and one learns with it in a different way. The chatbot @ve, with which one can communicate in Latin, was developed in 2022/2023 based on GPT-3.0. It was additionally equipped with a manually created knowledge base. After conceptual groundwork, this paper presents the preparation and implementation of the project. In addition, it summarizes the test that a Latin expert conducted with the chatbot. A critical discussion elaborates advantages and disadvantages. @ve could be a new tool for teaching Latin in a memorable and entertaining way through dialogue. However, the present implementation is still too prone to glitches for stand-alone use - i.e., without the accompaniment of a teacher. The use of GPT-4 could be a solution as well as the extension of the knowledge base. In conclusion, it can be argued that conversational agents are an innovative approach to promoting and preserving languages.