2024-04-26

Title: Online Personalizing White-box LLMs Generation with Neural Bandits

• Authors: Zekai Chen, Weeden Daniel, Po-yu Chen, Francois Buet-Golfouse
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.16115
• Pdf URL: https://arxiv.org/pdf/2404.16115
• Copy Paste: [[2404.16115]] Online Personalizing White-box LLMs Generation with Neural Bandits(https://arxiv.org/abs/2404.16115)
• Keywords: llm, agent
• Abstract: The advent of personalized content generation by LLMs presents a novel challenge: how to efficiently adapt text to meet individual preferences without the unsustainable demand of creating a unique model for each user. This study introduces an innovative online method that employs neural bandit algorithms to dynamically optimize soft instruction embeddings based on user feedback, enhancing the personalization of open-ended text generation by white-box LLMs. Through rigorous experimentation on various tasks, we demonstrate significant performance improvements over baseline strategies. NeuralTS, in particular, leads to substantial enhancements in personalized news headline generation, achieving up to a 62.9% improvement in terms of best ROUGE scores and up to 2.76% increase in LLM-agent evaluation against the baseline.
• 摘要：法学硕士个性化内容生成的出现提出了一个新的挑战：如何有效地调整文本以满足个人喜好，而不需要为每个用户创建独特的模型这种不可持续的需求。本研究引入了一种创新的在线方法，该方法采用神经强盗算法根据用户反馈动态优化软指令嵌入，从而增强白盒法学硕士开放式文本生成的个性化。通过对各种任务进行严格的实验，我们证明了相对于基准策略的显着性能改进。尤其是 NeuralTS，它极大地增强了个性化新闻标题的生成，在最佳 ROUGE 分数方面实现了高达 62.9% 的改进，并且与基线相比，LLM 代理评估提高了高达 2.76%。

Title: Classifying Human-Generated and AI-Generated Election Claims in Social Media

• Authors: Alphaeus Dmonte, Marcos Zampieri, Kevin Lybarger, Massimiliano Albanese
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.16116
• Pdf URL: https://arxiv.org/pdf/2404.16116
• Copy Paste: [[2404.16116]] Classifying Human-Generated and AI-Generated Election Claims in Social Media(https://arxiv.org/abs/2404.16116)
• Keywords: language model, llm
• Abstract: Politics is one of the most prevalent topics discussed on social media platforms, particularly during major election cycles, where users engage in conversations about candidates and electoral processes. Malicious actors may use this opportunity to disseminate misinformation to undermine trust in the electoral process. The emergence of Large Language Models (LLMs) exacerbates this issue by enabling malicious actors to generate misinformation at an unprecedented scale. Artificial intelligence (AI)-generated content is often indistinguishable from authentic user content, raising concerns about the integrity of information on social networks. In this paper, we present a novel taxonomy for characterizing election-related claims. This taxonomy provides an instrument for analyzing election-related claims, with granular categories related to jurisdiction, equipment, processes, and the nature of claims. We introduce ElectAI, a novel benchmark dataset that consists of 9,900 tweets, each labeled as human- or AI-generated. For AI-generated tweets, the specific LLM variant that produced them is specified. We annotated a subset of 1,550 tweets using the proposed taxonomy to capture the characteristics of election-related claims. We explored the capabilities of LLMs in extracting the taxonomy attributes and trained various machine learning models using ElectAI to distinguish between human- and AI-generated posts and identify the specific LLM variant.
• 摘要：政治是社交媒体平台上讨论的最流行的话题之一，特别是在主要选举周期期间，用户参与有关候选人和选举进程的对话。恶意行为者可能会利用这个机会传播错误信息，破坏对选举进程的信任。大型语言模型 (LLM) 的出现使恶意行为者能够以前所未有的规模生成错误信息，从而加剧了这一问题。人工智能 (AI) 生成的内容通常与真实的用户内容难以区分，引发了人们对社交网络信息完整性的担忧。在本文中，我们提出了一种新颖的分类法来描述与选举相关的主张。该分类法提供了一种分析与选举相关的索赔的工具，其中包含与管辖权、设备、流程和索赔性质相关的细粒度类别。我们引入了 ElectAI，这是一个新颖的基准数据集，由 9,900 条推文组成，每条推文都标记为人类或人工智能生成的。对于 AI 生成的推文，会指定生成它们的特定 LLM 变体。我们使用提议的分类法对 1,550 条推文的子集进行了注释，以捕获与选举相关的主张的特征。我们探索了法学硕士提取分类属性的能力，并使用 ElectAI 训练了各种机器学习模型，以区分人类和人工智能生成的帖子并识别特定的法学硕士变体。

Title: From Local to Global: A Graph RAG Approach to Query-Focused Summarization

• Authors: Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, Jonathan Larson
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2404.16130
• Pdf URL: https://arxiv.org/pdf/2404.16130
• Copy Paste: [[2404.16130]] From Local to Global: A Graph RAG Approach to Query-Focused Summarization(https://arxiv.org/abs/2404.16130)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: The use of retrieval-augmented generation (RAG) to retrieve relevant information from an external knowledge source enables large language models (LLMs) to answer questions over private and/or previously unseen document collections. However, RAG fails on global questions directed at an entire text corpus, such as "What are the main themes in the dataset?", since this is inherently a query-focused summarization (QFS) task, rather than an explicit retrieval task. Prior QFS methods, meanwhile, fail to scale to the quantities of text indexed by typical RAG systems. To combine the strengths of these contrasting methods, we propose a Graph RAG approach to question answering over private text corpora that scales with both the generality of user questions and the quantity of source text to be indexed. Our approach uses an LLM to build a graph-based text index in two stages: first to derive an entity knowledge graph from the source documents, then to pregenerate community summaries for all groups of closely-related entities. Given a question, each community summary is used to generate a partial response, before all partial responses are again summarized in a final response to the user. For a class of global sensemaking questions over datasets in the 1 million token range, we show that Graph RAG leads to substantial improvements over a na\"ive RAG baseline for both the comprehensiveness and diversity of generated answers. An open-source, Python-based implementation of both global and local Graph RAG approaches is forthcoming at https://aka.ms/graphrag.
• 摘要：使用检索增强生成（RAG）从外部知识源检索相关信息使大型语言模型（LLM）能够回答有关私人和/或以前未见过的文档集合的问题。然而，RAG 在针对整个文本语料库的全局问题上失败了，例如“数据集中的主题是什么？”，因为这本质上是一个以查询为中心的摘要 (QFS) 任务，而不是一个显式检索任务。与此同时，先前的 QFS 方法无法扩展到典型 RAG 系统索引的文本数量。为了结合这些对比方法的优点，我们提出了一种在私有文本语料库上进行问答的 Graph RAG 方法，该方法可根据用户问题的普遍性和要索引的源文本的数量进行扩展。我们的方法使用 LLM 分两个阶段构建基于图的文本索引：首先从源文档导出实体知识图，然后为所有密切相关的实体组预先生成社区摘要。给定一个问题，每个社区摘要都用于生成部分响应，然后所有部分响应再次汇总为对用户的最终响应。对于 100 万个 token 范围内的数据集上的一类全局意义构建问题，我们表明 Graph RAG 在生成答案的全面性和多样性方面比原始 RAG 基线带来了实质性改进。基于全局和本地 Graph RAG 方法的实现即将在 https://aka.ms/graphrag 上发布。

Title: Domain-Specific Improvement on Psychotherapy Chatbot Using Assistant

• Authors: Cheng Kang, Daniel Novak, Katerina Urbanova, Yuqing Cheng, Yong Hu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.16160
• Pdf URL: https://arxiv.org/pdf/2404.16160
• Copy Paste: [[2404.16160]] Domain-Specific Improvement on Psychotherapy Chatbot Using Assistant(https://arxiv.org/abs/2404.16160)
• Keywords: language model, llm, chat, retrieval augmented generation
• Abstract: Large language models (LLMs) have demonstrated impressive generalization capabilities on specific tasks with human-written instruction data. However, the limited quantity, diversity, and professional expertise of such instruction data raise concerns about the performance of LLMs in psychotherapy tasks when provided with domain-specific instructions. To address this, we firstly propose Domain-Specific Assistant Instructions based on AlexanderStreet therapy, and secondly, we use an adaption fine-tuning method and retrieval augmented generation method to improve pre-trained LLMs. Through quantitative evaluation of linguistic quality using automatic and human evaluation, we observe that pre-trained LLMs on Psychotherapy Assistant Instructions outperform state-of-the-art LLMs response baselines. Our Assistant-Instruction approach offers a half-annotation method to align pre-trained LLMs with instructions and provide pre-trained LLMs with more psychotherapy knowledge.
• 摘要：大型语言模型 (LLM) 在使用人工编写的指令数据的特定任务上展示了令人印象深刻的泛化能力。然而，此类指导数据的数量、多样性和专业知识有限，引起了人们对法学硕士在接受特定领域指导时在心理治疗任务中表现的担忧。为了解决这个问题，我们首先提出基于AlexanderStreet疗法的特定领域辅助指令，其次，我们使用适应微调方法和检索增强生成方法来改进预训练的LLM。通过使用自动和人工评估对语言质量进行定量评估，我们观察到经过预训练的心理治疗辅助说明法学硕士的表现优于最先进的法学硕士反应基线。我们的助理教学方法提供了一种半注释方法，使预训练的法学硕士与指令保持一致，并为预训练的法学硕士提供更多的心理治疗知识。

Title: Towards a Holistic Evaluation of LLMs on Factual Knowledge Recall

• Authors: Jiaqing Yuan, Lin Pan, Chung-Wei Hang, Jiang Guo, Jiarong Jiang, Bonan Min, Patrick Ng, Zhiguo Wang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.16164
• Pdf URL: https://arxiv.org/pdf/2404.16164
• Copy Paste: [[2404.16164]] Towards a Holistic Evaluation of LLMs on Factual Knowledge Recall(https://arxiv.org/abs/2404.16164)
• Keywords: language model, gpt, llm, hallucination
• Abstract: Large language models (LLMs) have shown remarkable performance on a variety of NLP tasks, and are being rapidly adopted in a wide range of use cases. It is therefore of vital importance to holistically evaluate the factuality of their generated outputs, as hallucinations remain a challenging issue. In this work, we focus on assessing LLMs' ability to recall factual knowledge learned from pretraining, and the factors that affect this ability. To that end, we construct FACT-BENCH, a representative benchmark covering 20 domains, 134 property types, 3 answer types, and different knowledge popularity levels. We benchmark 31 models from 10 model families and provide a holistic assessment of their strengths and weaknesses. We observe that instruction-tuning hurts knowledge recall, as pretraining-only models consistently outperform their instruction-tuned counterparts, and positive effects of model scaling, as larger models outperform smaller ones for all model families. However, the best performance from GPT-4 still represents a large gap with the upper-bound. We additionally study the role of in-context exemplars using counterfactual demonstrations, which lead to significant degradation of factual knowledge recall for large models. By further decoupling model known and unknown knowledge, we find the degradation is attributed to exemplars that contradict a model's known knowledge, as well as the number of such exemplars. Lastly, we fine-tune LLaMA-7B in different settings of known and unknown knowledge. In particular, fine-tuning on a model's known knowledge is beneficial, and consistently outperforms fine-tuning on unknown and mixed knowledge. We will make our benchmark publicly available.
• 摘要：大型语言模型 (LLM) 在各种 NLP 任务上表现出了卓越的性能，并且正在广泛的用例中得到迅速采用。因此，全面评估其生成的结果的真实性至关重要，因为幻觉仍然是一个具有挑战性的问题。在这项工作中，我们重点评估法学硕士回忆从预训练中学到的事实知识的能力，以及影响这种能力的因素。为此，我们构建了 FACT-BENCH，一个涵盖 20 个领域、134 个属性类型、3 个答案类型和不同知识普及程度的代表性基准。我们对 10 个模型系列中的 31 个模型进行了基准测试，并对它们的优缺点进行了全面评估。我们观察到指令调整会损害知识回忆，因为仅预训练的模型始终优于指令调整的模型，并且模型缩放会产生积极影响，因为对于所有模型系列，较大的模型优于较小的模型。然而，GPT-4 的最佳性能仍然与上限有很大差距。我们还使用反事实演示来研究上下文样本的作用，这会导致大型模型的事实知识回忆显着下降。通过进一步解耦模型已知和未知知识，我们发现退化归因于与模型已知知识相矛盾的样本以及此类样本的数量。最后，我们在已知和未知知识的不同设置中对 LLaMA-7B 进行微调。特别是，对模型的已知知识进行微调是有益的，并且始终优于对未知和混合知识进行微调。我们将公开我们的基准。

Title: Fusion of Domain-Adapted Vision and Language Models for Medical Visual Question Answering

• Authors: Cuong Nhat Ha, Shima Asaadi, Sanjeev Kumar Karn, Oladimeji Farri, Tobias Heimann, Thomas Runkler
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2404.16192
• Pdf URL: https://arxiv.org/pdf/2404.16192
• Copy Paste: [[2404.16192]] Fusion of Domain-Adapted Vision and Language Models for Medical Visual Question Answering(https://arxiv.org/abs/2404.16192)
• Keywords: language model
• Abstract: Vision-language models, while effective in general domains and showing strong performance in diverse multi-modal applications like visual question-answering (VQA), struggle to maintain the same level of effectiveness in more specialized domains, e.g., medical. We propose a medical vision-language model that integrates large vision and language models adapted for the medical domain. This model goes through three stages of parameter-efficient training using three separate biomedical and radiology multi-modal visual and text datasets. The proposed model achieves state-of-the-art performance on the SLAKE 1.0 medical VQA (MedVQA) dataset with an overall accuracy of 87.5% and demonstrates strong performance on another MedVQA dataset, VQA-RAD, achieving an overall accuracy of 73.2%.
• 摘要：视觉语言模型虽然在一般领域有效，并在视觉问答 (VQA) 等各种多模态应用中表现出色，但在更专业的领域（例如医学）中难以保持同样的有效性。我们提出了一种医学视觉语言模型，该模型集成了适合医学领域的大型视觉和语言模型。该模型使用三个独立的生物医学和放射学多模态视觉和文本数据集，经过三个阶段的参数高效训练。所提出的模型在 SLAKE 1.0 医学 VQA (MedVQA) 数据集上实现了最佳性能，总体准确率为 87.5%，并在另一个 MedVQA 数据集 VQA-RAD 上表现出色，总体准确率为 73.2%。

Title: Towards Efficient Patient Recruitment for Clinical Trials: Application of a Prompt-Based Learning Model

• Authors: Mojdeh Rahmanian, Seyed Mostafa Fakhrahmad, Seyedeh Zahra Mousavi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.16198
• Pdf URL: https://arxiv.org/pdf/2404.16198
• Copy Paste: [[2404.16198]] Towards Efficient Patient Recruitment for Clinical Trials: Application of a Prompt-Based Learning Model(https://arxiv.org/abs/2404.16198)
• Keywords: language model, gpt, prompt
• Abstract: Objective: Clinical trials are essential for advancing pharmaceutical interventions, but they face a bottleneck in selecting eligible participants. Although leveraging electronic health records (EHR) for recruitment has gained popularity, the complex nature of unstructured medical texts presents challenges in efficiently identifying participants. Natural Language Processing (NLP) techniques have emerged as a solution with a recent focus on transformer models. In this study, we aimed to evaluate the performance of a prompt-based large language model for the cohort selection task from unstructured medical notes collected in the EHR. Methods: To process the medical records, we selected the most related sentences of the records to the eligibility criteria needed for the trial. The SNOMED CT concepts related to each eligibility criterion were collected. Medical records were also annotated with MedCAT based on the SNOMED CT ontology. Annotated sentences including concepts matched with the criteria-relevant terms were extracted. A prompt-based large language model (Generative Pre-trained Transformer (GPT) in this study) was then used with the extracted sentences as the training set. To assess its effectiveness, we evaluated the model's performance using the dataset from the 2018 n2c2 challenge, which aimed to classify medical records of 311 patients based on 13 eligibility criteria through NLP techniques. Results: Our proposed model showed the overall micro and macro F measures of 0.9061 and 0.8060 which were among the highest scores achieved by the experiments performed with this dataset. Conclusion: The application of a prompt-based large language model in this study to classify patients based on eligibility criteria received promising scores. Besides, we proposed a method of extractive summarization with the aid of SNOMED CT ontology that can be also applied to other medical texts.
• 摘要：目的：临床试验对于推进药物干预至关重要，但在选择合格参与者方面面临瓶颈。尽管利用电子健康记录 (EHR) 进行招募已广受欢迎，但非结构化医疗文本的复杂性给有效识别参与者带来了挑战。自然语言处理（NLP）技术已经作为一种解决方案出现，最近的重点是变压器模型。在本研究中，我们的目的是评估基于提示的大语言模型在从 EHR 中收集的非结构化医疗记录中进行队列选择任务时的性能。方法：为了处理医疗记录，我们选择了与试验所需的资格标准最相关的记录句子。收集了与每个资格标准相关的 SNOMED CT 概念。医疗记录还使用基于 SNOMED CT 本体的 MedCAT 进行了注释。提取包含与标准相关术语匹配的概念的注释句子。然后使用基于提示的大语言模型（本研究中的生成预训练变压器（GPT））和提取的句子作为训练集。为了评估其有效性，我们使用 2018 年 n2c2 挑战赛的数据集评估了模型的性能，该挑战赛旨在通过 NLP 技术根据 13 项资格标准对 311 名患者的医疗记录进行分类。结果：我们提出的模型显示整体微观和宏观 F 测量值分别为 0.9061 和 0.8060，这是使用该数据集进行的实验取得的最高分数之一。结论：本研究中应用基于提示的大语言模型根据资格标准对患者进行分类，获得了有希望的分数。此外，我们提出了一种借助 SNOMED CT 本体的提取摘要方法，该方法也可以应用于其他医学文本。

Title: URL: Universal Referential Knowledge Linking via Task-instructed Representation Compression

• Authors: Zhuoqun Li, Hongyu Lin, Tianshu Wang, Boxi Cao, Yaojie Lu, Weixiang Zhou, Hao Wang, Zhenyu Zeng, Le Sun, Xianpei Han
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.16248
• Pdf URL: https://arxiv.org/pdf/2404.16248
• Copy Paste: [[2404.16248]] URL: Universal Referential Knowledge Linking via Task-instructed Representation Compression(https://arxiv.org/abs/2404.16248)
• Keywords: llm
• Abstract: Linking a claim to grounded references is a critical ability to fulfill human demands for authentic and reliable information. Current studies are limited to specific tasks like information retrieval or semantic matching, where the claim-reference relationships are unique and fixed, while the referential knowledge linking (RKL) in real-world can be much more diverse and complex. In this paper, we propose universal referential knowledge linking (URL), which aims to resolve diversified referential knowledge linking tasks by one unified model. To this end, we propose a LLM-driven task-instructed representation compression, as well as a multi-view learning approach, in order to effectively adapt the instruction following and semantic understanding abilities of LLMs to referential knowledge linking. Furthermore, we also construct a new benchmark to evaluate ability of models on referential knowledge linking tasks across different scenarios. Experiments demonstrate that universal RKL is challenging for existing approaches, while the proposed framework can effectively resolve the task across various scenarios, and therefore outperforms previous approaches by a large margin.
• 摘要：将主张与有根据的参考文献联系起来是满足人们对真实可靠信息的需求的关键能力。目前的研究仅限于信息检索或语义匹配等特定任务，其中主张-参考关系是唯一且固定的，而现实世界中的参考知识链接（RKL）可能更加多样化和复杂。在本文中，我们提出了通用参考知识链接（URL），旨在通过一个统一的模型解决多样化的参考知识链接任务。为此，我们提出了一种LLM驱动的任务指令表示压缩以及多视图学习方法，以有效地将LLM的指令跟随和语义理解能力适应参考知识链接。此外，我们还构建了一个新的基准来评估模型在不同场景下的参考知识链接任务的能力。实验表明，通用 RKL 对现有方法具有挑战性，而所提出的框架可以有效地解决各种场景下的任务，因此大大优于以前的方法。

Title: Interpreting Answers to Yes-No Questions in Dialogues from Multiple Domains

• Authors: Zijie Wang, Farzana Rashid, Eduardo Blanco
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.16262
• Pdf URL: https://arxiv.org/pdf/2404.16262
• Copy Paste: [[2404.16262]] Interpreting Answers to Yes-No Questions in Dialogues from Multiple Domains(https://arxiv.org/abs/2404.16262)
• Keywords: language model
• Abstract: People often answer yes-no questions without explicitly saying yes, no, or similar polar keywords. Figuring out the meaning of indirect answers is challenging, even for large language models. In this paper, we investigate this problem working with dialogues from multiple domains. We present new benchmarks in three diverse domains: movie scripts, tennis interviews, and airline customer service. We present an approach grounded on distant supervision and blended training to quickly adapt to a new dialogue domain. Experimental results show that our approach is never detrimental and yields F1 improvements as high as 11-34%.
• 摘要：人们经常在回答是非问题时不明确地说是、否或类似的极性关键词。即使对于大型语言模型来说，弄清楚间接答案的含义也具有挑战性。在本文中，我们通过多个领域的对话来研究这个问题。我们在三个不同的领域提出了新的基准：电影剧本、网球采访和航空公司客户服务。我们提出了一种基于远程监督和混合训练的方法，以快速适应新的对话领域。实验结果表明，我们的方法绝不会有害，并且 F1 改进高达 11-34%。

Title: LLM-Based Section Identifiers Excel on Open Source but Stumble in Real World Applications

• Authors: Saranya Krishnamoorthy, Ayush Singh, Shabnam Tafreshi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.16294
• Pdf URL: https://arxiv.org/pdf/2404.16294
• Copy Paste: [[2404.16294]] LLM-Based Section Identifiers Excel on Open Source but Stumble in Real World Applications(https://arxiv.org/abs/2404.16294)
• Keywords: language model, gpt, llm
• Abstract: Electronic health records (EHR) even though a boon for healthcare practitioners, are growing convoluted and longer every day. Sifting around these lengthy EHRs is taxing and becomes a cumbersome part of physician-patient interaction. Several approaches have been proposed to help alleviate this prevalent issue either via summarization or sectioning, however, only a few approaches have truly been helpful in the past. With the rise of automated methods, machine learning (ML) has shown promise in solving the task of identifying relevant sections in EHR. However, most ML methods rely on labeled data which is difficult to get in healthcare. Large language models (LLMs) on the other hand, have performed impressive feats in natural language processing (NLP), that too in a zero-shot manner, i.e. without any labeled data. To that end, we propose using LLMs to identify relevant section headers. We find that GPT-4 can effectively solve the task on both zero and few-shot settings as well as segment dramatically better than state-of-the-art methods. Additionally, we also annotate a much harder real world dataset and find that GPT-4 struggles to perform well, alluding to further research and harder benchmarks.
• 摘要：电子健康记录 (EHR) 尽管对医疗保健从业者来说是一个福音，但每天都变得越来越复杂、越来越长。筛选这些冗长的电子病历非常费力，并且成为医患互动中的一个麻烦部分。人们提出了几种方法来通过总结或分段来帮助缓解这一普遍问题，但是，过去只有少数方法真正有用。随着自动化方法的兴起，机器学习 (ML) 在解决识别 EHR 中相关部分的任务方面显示出了希望。然而，大多数机器学习方法都依赖于标记数据，而这在医疗保健领域很难获得。另一方面，大型语言模型（LLM）在自然语言处理（NLP）方面也取得了令人印象深刻的成就，而且也是以零样本的方式，即没有任何标记数据。为此，我们建议使用法学硕士来识别相关的章节标题。我们发现 GPT-4 可以有效地解决零样本和少样本设置下的任务，并且分割效果比最先进的方法要好得多。此外，我们还注释了一个更难的现实世界数据集，发现 GPT-4 很难​​表现良好，这暗示着进一步的研究和更难的基准测试。

Title: WorldValuesBench: A Large-Scale Benchmark Dataset for Multi-Cultural Value Awareness of Language Models

• Authors: Wenlong Zhao, Debanjan Mondal, Niket Tandon, Danica Dillion, Kurt Gray, Yuling Gu
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2404.16308
• Pdf URL: https://arxiv.org/pdf/2404.16308
• Copy Paste: [[2404.16308]] WorldValuesBench: A Large-Scale Benchmark Dataset for Multi-Cultural Value Awareness of Language Models(https://arxiv.org/abs/2404.16308)
• Keywords: language model, gpt
• Abstract: The awareness of multi-cultural human values is critical to the ability of language models (LMs) to generate safe and personalized responses. However, this awareness of LMs has been insufficiently studied, since the computer science community lacks access to the large-scale real-world data about multi-cultural values. In this paper, we present WorldValuesBench, a globally diverse, large-scale benchmark dataset for the multi-cultural value prediction task, which requires a model to generate a rating response to a value question based on demographic contexts. Our dataset is derived from an influential social science project, World Values Survey (WVS), that has collected answers to hundreds of value questions (e.g., social, economic, ethical) from 94,728 participants worldwide. We have constructed more than 20 million examples of the type "(demographic attributes, value question) $\rightarrow$ answer" from the WVS responses. We perform a case study using our dataset and show that the task is challenging for strong open and closed-source models. On merely $11.1\%$, $25.0\%$, $72.2\%$, and $75.0\%$ of the questions, Alpaca-7B, Vicuna-7B-v1.5, Mixtral-8x7B-Instruct-v0.1, and GPT-3.5 Turbo can respectively achieve $<0.2$ Wasserstein 1-distance from the human normalized answer distributions. WorldValuesBench opens up new research avenues in studying limitations and opportunities in multi-cultural value awareness of LMs.
• 摘要：对多元文化人类价值观的认识对于语言模型 (LM) 生成安全和个性化响应的能力至关重要。然而，由于计算机科学界缺乏有关多元文化价值观的大规模现实世界数据，因此对 LM 的这种意识尚未得到充分研究。在本文中，我们提出了 WorldValuesBench，这是一个用于多元文化价值预测任务的全球多样化的大规模基准数据集，它需要一个模型根据人口统计背景生成对价值问题的评级响应。我们的数据集源自一个有影响力的社会科学项目——世界价值观调查 (WVS)，该项目从全球 94,728 名参与者那里收集了数百个价值观问题（例如社会、经济、道德）的答案。我们从 WVS 响应中构建了超过 2000 万个“（人口统计属性，价值问题）$\rightarrow$ 答案”类型的示例。我们使用我们的数据集进行了案例研究，结果表明该任务对于强大的开源和闭源模型来说具有挑战性。仅涉及 $11.1\%$、$25.0\%$、$72.2\%$ 和 $75.0\%$ 的问题、Alpaca-7B、Vicuna-7B-v1.5、Mixtral-8x7B-Instruct-v0.1 和 GPT -3.5 Turbo 可以分别实现与人类标准化答案分布的 $<0.2$ Wasserstein 1 距离。 WorldValuesBench 为研究 LM 多元文化价值意识的局限性和机遇开辟了新的研究途径。

Title: VISLA Benchmark: Evaluating Embedding Sensitivity to Semantic and Lexical Alterations

• Authors: Sri Harsha Dumpala, Aman Jaiswal, Chandramouli Sastry, Evangelos Milios, Sageev Oore, Hassan Sajjad
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.16365
• Pdf URL: https://arxiv.org/pdf/2404.16365
• Copy Paste: [[2404.16365]] VISLA Benchmark: Evaluating Embedding Sensitivity to Semantic and Lexical Alterations(https://arxiv.org/abs/2404.16365)
• Keywords: language model
• Abstract: Despite their remarkable successes, state-of-the-art language models face challenges in grasping certain important semantic details. This paper introduces the VISLA (Variance and Invariance to Semantic and Lexical Alterations) benchmark, designed to evaluate the semantic and lexical understanding of language models. VISLA presents a 3-way semantic (in)equivalence task with a triplet of sentences associated with an image, to evaluate both vision-language models (VLMs) and unimodal language models (ULMs). An evaluation involving 34 VLMs and 20 ULMs reveals surprising difficulties in distinguishing between lexical and semantic variations. Spatial semantics encoded by language models also appear to be highly sensitive to lexical information. Notably, text encoders of VLMs demonstrate greater sensitivity to semantic and lexical variations than unimodal text encoders. Our contributions include the unification of image-to-text and text-to-text retrieval tasks, an off-the-shelf evaluation without fine-tuning, and assessing LMs' semantic (in)variance in the presence of lexical alterations. The results highlight strengths and weaknesses across diverse vision and unimodal language models, contributing to a deeper understanding of their capabilities. % VISLA enables a rigorous evaluation, shedding light on language models' capabilities in handling semantic and lexical nuances. Data and code will be made available at https://github.com/Sri-Harsha/visla_benchmark.
• 摘要：尽管取得了显着的成功，最先进的语言模型在掌握某些重要的语义细节方面面临着挑战。本文介绍了 VISLA（语义和词汇变更的方差和不变性）基准，旨在评估语言模型的语义和词汇理解。 VISLA 提出了一种三向语义（内）等价任务，其中包含与图像相关的三组句子，以评估视觉语言模型 (VLM) 和单峰语言模型 (ULM)。涉及 34 个 VLM 和 20 个 ULM 的评估揭示了区分词汇和语义变化的惊人困难。由语言模型编码的空间语义似乎也对词汇信息高度敏感。值得注意的是，VLM 的文本编码器比单模态文本编码器对语义和词汇变化表现出更高的敏感性。我们的贡献包括图像到文本和文本到文本检索任务的统一、无需微调的现成评估，以及评估 LM 在存在词汇更改的情况下的语义（内）方差。结果突出了不同视觉和单模态语言模型的优点和缺点，有助于更深入地了解它们的能力。 % VISLA 支持严格的评估，揭示语言模型处理语义和词汇细微差别的能力。数据和代码将在 https://github.com/Sri-Harsha/visla_benchmark 上提供。

Title: Learning Syntax Without Planting Trees: Understanding When and Why Transformers Generalize Hierarchically

• Authors: Kabir Ahuja, Vidhisha Balachandran, Madhur Panwar, Tianxing He, Noah A. Smith, Navin Goyal, Yulia Tsvetkov
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.16367
• Pdf URL: https://arxiv.org/pdf/2404.16367
• Copy Paste: [[2404.16367]] Learning Syntax Without Planting Trees: Understanding When and Why Transformers Generalize Hierarchically(https://arxiv.org/abs/2404.16367)
• Keywords: language model
• Abstract: Transformers trained on natural language data have been shown to learn its hierarchical structure and generalize to sentences with unseen syntactic structures without explicitly encoding any structural bias. In this work, we investigate sources of inductive bias in transformer models and their training that could cause such generalization behavior to emerge. We extensively experiment with transformer models trained on multiple synthetic datasets and with different training objectives and show that while other objectives e.g. sequence-to-sequence modeling, prefix language modeling, often failed to lead to hierarchical generalization, models trained with the language modeling objective consistently learned to generalize hierarchically. We then conduct pruning experiments to study how transformers trained with the language modeling objective encode hierarchical structure. When pruned, we find joint existence of subnetworks within the model with different generalization behaviors (subnetworks corresponding to hierarchical structure and linear order). Finally, we take a Bayesian perspective to further uncover transformers' preference for hierarchical generalization: We establish a correlation between whether transformers generalize hierarchically on a dataset and whether the simplest explanation of that dataset is provided by a hierarchical grammar compared to regular grammars exhibiting linear generalization.
• 摘要：经过自然语言数据训练的 Transformer 已被证明可以学习其层次结构，并概括为具有看不见的句法结构的句子，而无需显式编码任何结构偏差。在这项工作中，我们研究了变压器模型中归纳偏差的来源及其训练，这些偏差可能导致这种泛化行为的出现。我们对在多个合成数据集上训练的 Transformer 模型和不同的训练目标进行了广泛的实验，并表明，而其他目标，例如序列到序列建模、前缀语言建模通常无法实现分层泛化，而使用语言建模目标训练的模型始终能够学习分层泛化。然后，我们进行剪枝实验，以研究使用语言建模目标训练的变压器如何编码层次结构。剪枝时，我们发现模型内联合存在具有不同泛化行为的子网络（对应于层次结构和线性顺序的子网络）。最后，我们从贝叶斯的角度进一步揭示 Transformer 对分层泛化的偏好：我们在 Transformer 是否在数据集上分层泛化以及分层语法是否提供该数据集的最简单解释（与表现出线性泛化的常规语法相比）之间建立了相关性。

Title: Don't Say No: Jailbreaking LLM by Suppressing Refusal

• Authors: Yukai Zhou, Wenjie Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.16369
• Pdf URL: https://arxiv.org/pdf/2404.16369
• Copy Paste: [[2404.16369]] Don't Say No: Jailbreaking LLM by Suppressing Refusal(https://arxiv.org/abs/2404.16369)
• Keywords: language model, llm, prompt
• Abstract: Ensuring the safety alignment of Large Language Models (LLMs) is crucial to generating responses consistent with human values. Despite their ability to recognize and avoid harmful queries, LLMs are vulnerable to "jailbreaking" attacks, where carefully crafted prompts elicit them to produce toxic content. One category of jailbreak attacks is reformulating the task as adversarial attacks by eliciting the LLM to generate an affirmative response. However, the typical attack in this category GCG has very limited attack success rate. In this study, to better study the jailbreak attack, we introduce the DSN (Don't Say No) attack, which prompts LLMs to not only generate affirmative responses but also novelly enhance the objective to suppress refusals. In addition, another challenge lies in jailbreak attacks is the evaluation, as it is difficult to directly and accurately assess the harmfulness of the attack. The existing evaluation such as refusal keyword matching has its own limitation as it reveals numerous false positive and false negative instances. To overcome this challenge, we propose an ensemble evaluation pipeline incorporating Natural Language Inference (NLI) contradiction assessment and two external LLM evaluators. Extensive experiments demonstrate the potency of the DSN and the effectiveness of ensemble evaluation compared to baseline methods.
• 摘要：确保大型语言模型 (LLM) 的安全一致性对于生成符合人类价值观的响应至关重要。尽管法学硕士有能力识别和避免有害查询，但他们很容易受到“越狱”攻击，精心设计的提示会诱使他们产生有毒内容。一类越狱攻击是通过引发 LLM 生成肯定响应来将任务重新表述为对抗性攻击。然而，此类GCG中的典型攻击的攻击成功率非常有限。在本研究中，为了更好地研究越狱攻击，我们引入了DSN（不要说不）攻击，它促使LLM不仅产生肯定的响应，而且新颖地增强了抑制拒绝的目标。此外，越狱攻击的另一个挑战是评估，因为很难直接准确地评估攻击的危害性。现有的评估（例如拒绝关键字匹配）有其自身的局限性，因为它揭示了大量的误报和漏报实例。为了克服这一挑战，我们提出了一个集成评估流程，其中包含自然语言推理（NLI）矛盾评估和两个外部法学硕士评估器。大量实验证明了 DSN 的效力以及与基线方法相比的集成评估的有效性。

Title: Lost in Recursion: Mining Rich Event Semantics in Knowledge Graphs

• Authors: Florian Plötzky, Niklas Kiehne, Wolf-Tilo Balke
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.16405
• Pdf URL: https://arxiv.org/pdf/2404.16405
• Copy Paste: [[2404.16405]] Lost in Recursion: Mining Rich Event Semantics in Knowledge Graphs(https://arxiv.org/abs/2404.16405)
• Keywords: prompt
• Abstract: Our world is shaped by events of various complexity. This includes both small-scale local events like local farmer markets and large complex events like political and military conflicts. The latter are typically not observed directly but through the lenses of intermediaries like newspapers or social media. In other words, we do not witness the unfolding of such events directly but are confronted with narratives surrounding them. Such narratives capture different aspects of a complex event and may also differ with respect to the narrator. Thus, they provide a rich semantics concerning real-world events. In this paper, we show how narratives concerning complex events can be constructed and utilized. We provide a formal representation of narratives based on recursive nodes to represent multiple levels of detail and discuss how narratives can be bound to event-centric knowledge graphs. Additionally, we provide an algorithm based on incremental prompting techniques that mines such narratives from texts to account for different perspectives on complex events. Finally, we show the effectiveness and future research directions in a proof of concept.
• 摘要：我们的世界是由各种复杂的事件塑造的。这既包括当地农贸市场等小规模当地事件，也包括政治和军事冲突等大型复杂事件。后者通常不会直接观察到，而是通过报纸或社交媒体等中介机构的视角。换句话说，我们并不是直接目睹这些事件的发生，而是面对围绕它们的叙述。此类叙述捕捉了复杂事件的不同方面，并且叙述者的情况也可能有所不同。因此，它们提供了有关现实世界事件的丰富语义。在本文中，我们展示了如何构建和利用有关复杂事件的叙述。我们提供基于递归节点的叙述的正式表示，以表示多个细节级别，并讨论如何将叙述绑定到以事件为中心的知识图谱。此外，我们提供了一种基于增量提示技术的算法，该算法从文本中挖掘此类叙述，以解释对复杂事件的不同观点。最后，我们在概念验证中展示了有效性和未来的研究方向。

Title: U2++ MoE: Scaling 4.7x parameters with minimal impact on RTF

• Authors: Xingchen Song, Di Wu, Binbin Zhang, Dinghao Zhou, Zhendong Peng, Bo Dang, Fuping Pan, Chao Yang
• Subjects: cs.CL, eess.AS
• Abstract URL: https://arxiv.org/abs/2404.16407
• Pdf URL: https://arxiv.org/pdf/2404.16407
• Copy Paste: [[2404.16407]] U2++ MoE: Scaling 4.7x parameters with minimal impact on RTF(https://arxiv.org/abs/2404.16407)
• Keywords: language model
• Abstract: Scale has opened new frontiers in natural language processing, but at a high cost. In response, by learning to only activate a subset of parameters in training and inference, Mixture-of-Experts (MoE) have been proposed as an energy efficient path to even larger and more capable language models and this shift towards a new generation of foundation models is gaining momentum, particularly within the field of Automatic Speech Recognition (ASR). Recent works that incorporating MoE into ASR models have complex designs such as routing frames via supplementary embedding network, improving multilingual ability for the experts, and utilizing dedicated auxiliary losses for either expert load balancing or specific language handling. We found that delicate designs are not necessary, while an embarrassingly simple substitution of MoE layers for all Feed-Forward Network (FFN) layers is competent for the ASR task. To be more specific, we benchmark our proposed model on a large scale inner-source dataset (160k hours), the results show that we can scale our baseline Conformer (Dense-225M) to its MoE counterparts (MoE-1B) and achieve Dense-1B level Word Error Rate (WER) while maintaining a Dense-225M level Real Time Factor (RTF). Furthermore, by applying Unified 2-pass framework with bidirectional attention decoders (U2++), we achieve the streaming and non-streaming decoding modes in a single MoE based model, which we call U2++ MoE. We hope that our study can facilitate the research on scaling speech foundation models without sacrificing deployment efficiency.
• 摘要：规模化开辟了自然语言处理的新领域，但成本很高。作为回应，通过学习仅激活训练和推理中的参数子集，专家混合 (MoE) 已被提议作为通往更大、能力更强的语言模型的节能途径，并且这种向新一代基础的转变模型正在获得发展势头，特别是在自动语音识别（ASR）领域。最近将 MoE 纳入 ASR 模型的工作具有复杂的设计，例如通过补充嵌入网络路由框架、提高专家的多语言能力，以及利用专用辅助损失来进行专家负载平衡或特定语言处理。我们发现精细的设计是不必要的，而用 MoE 层替换所有前馈网络 (FFN) 层的简单得令人尴尬的方式就可以胜任 ASR 任务。更具体地说，我们在大规模内部源数据集（160k 小时）上对我们提出的模型进行了基准测试，结果表明我们可以将基线 Conformer (Dense-225M) 扩展到其 MoE 对应项 (MoE-1B) 并实现 Dense -1B 级字错误率 (WER)，同时保持 Dense-225M 级实时因子 (RTF)。此外，通过应用具有双向注意解码器（U2++）的统一2-pass框架，我们在基于MoE的单个模型中实现了流式和非流式解码模式，我们将其称为U2++MoE。我们希望我们的研究能够促进扩展语音基础模型的研究，而不牺牲部署效率。

Title: Asking and Answering Questions to Extract Event-Argument Structures

• Authors: Md Nayem Uddin, Enfa Rose George, Eduardo Blanco, Steven Corman
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.16413
• Pdf URL: https://arxiv.org/pdf/2404.16413
• Copy Paste: [[2404.16413]] Asking and Answering Questions to Extract Event-Argument Structures(https://arxiv.org/abs/2404.16413)
• Keywords: language model
• Abstract: This paper presents a question-answering approach to extract document-level event-argument structures. We automatically ask and answer questions for each argument type an event may have. Questions are generated using manually defined templates and generative transformers. Template-based questions are generated using predefined role-specific wh-words and event triggers from the context document. Transformer-based questions are generated using large language models trained to formulate questions based on a passage and the expected answer. Additionally, we develop novel data augmentation strategies specialized in inter-sentential event-argument relations. We use a simple span-swapping technique, coreference resolution, and large language models to augment the training instances. Our approach enables transfer learning without any corpora-specific modifications and yields competitive results with the RAMS dataset. It outperforms previous work, and it is especially beneficial to extract arguments that appear in different sentences than the event trigger. We also present detailed quantitative and qualitative analyses shedding light on the most common errors made by our best model.
• 摘要：本文提出了一种提取文档级事件参数结构的问答方法。我们自动针对事件可能具有的每种参数类型提出问题并回答问题。问题是使用手动定义的模板和生成转换器生成的。基于模板的问题是使用上下文文档中预定义的特定于角色的 wh 词和事件触发器生成的。基于 Transformer 的问题是使用大型语言模型生成的，该模型经过训练，可以根据段落和预期答案制定问题。此外，我们还开发了专门针对句子间事件-论证关系的新颖的数据增强策略。我们使用简单的跨度交换技术、共指解析和大型语言模型来增强训练实例。我们的方法无需任何特定于语料库的修改即可实现迁移学习，并产生与 RAMS 数据集具有竞争力的结果。它优于以前的工作，并且对于提取出现在与事件触发器不同的句子中的参数特别有益。我们还提供了详细的定量和定性分析，揭示了我们最佳模型所犯的最常见错误。

Title: Contextual Categorization Enhancement through LLMs Latent-Space

• Authors: Zineddine Bettouche, Anas Safi, Andreas Fischer
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.16442
• Pdf URL: https://arxiv.org/pdf/2404.16442
• Copy Paste: [[2404.16442]] Contextual Categorization Enhancement through LLMs Latent-Space(https://arxiv.org/abs/2404.16442)
• Keywords: llm
• Abstract: Managing the semantic quality of the categorization in large textual datasets, such as Wikipedia, presents significant challenges in terms of complexity and cost. In this paper, we propose leveraging transformer models to distill semantic information from texts in the Wikipedia dataset and its associated categories into a latent space. We then explore different approaches based on these encodings to assess and enhance the semantic identity of the categories. Our graphical approach is powered by Convex Hull, while we utilize Hierarchical Navigable Small Worlds (HNSWs) for the hierarchical approach. As a solution to the information loss caused by the dimensionality reduction, we modulate the following mathematical solution: an exponential decay function driven by the Euclidean distances between the high-dimensional encodings of the textual categories. This function represents a filter built around a contextual category and retrieves items with a certain Reconsideration Probability (RP). Retrieving high-RP items serves as a tool for database administrators to improve data groupings by providing recommendations and identifying outliers within a contextual framework.
• 摘要：管理大型文本数据集（如维基百科）中分类的语义质量在复杂性和成本方面面临巨大挑战。在本文中，我们建议利用变换器模型将维基百科数据集及其相关类别中的文本中的语义信息提取到潜在空间中。然后，我们探索基于这些编码的不同方法来评估和增强类别的语义身份。我们的图形方法由凸包提供支持，而我们利用分层可导航小世界 (HNSW) 进行分层方法。为了解决由降维引起的信息丢失问题，我们调整了以下数学解决方案：由文本类别的高维编码之间的欧几里得距离驱动的指数衰减函数。此函数表示围绕上下文类别构建的过滤器，并检索具有一定重新考虑概率 (RP) 的项目。检索高 RP 项目是数据库管理员通过提供建议和识别上下文框架内的异常值来改进数据分组的工具。

Title: Large Language Models Perform on Par with Experts Identifying Mental Health Factors in Adolescent Online Forums

• Authors: Isablle Lorge, Dam W. Joyce, Andrey Kormilitzin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.16461
• Pdf URL: https://arxiv.org/pdf/2404.16461
• Copy Paste: [[2404.16461]] Large Language Models Perform on Par with Experts Identifying Mental Health Factors in Adolescent Online Forums(https://arxiv.org/abs/2404.16461)
• Keywords: language model, gpt, llm
• Abstract: Mental health in children and adolescents has been steadily deteriorating over the past few years [ 1 ]. The recent advent of Large Language Models (LLMs) offers much hope for cost and time efficient scaling of monitoring and intervention, yet despite specifically prevalent issues such as school bullying and eating disorders, previous studies on have not investigated performance in this domain or for open information extraction where the set of answers is not predetermined. We create a new dataset of Reddit posts from adolescents aged 12-19 annotated by expert psychiatrists for the following categories: TRAUMA, PRECARITY, CONDITION, SYMPTOMS, SUICIDALITY and TREATMENT and compare expert labels to annotations from two top performing LLMs (GPT3.5 and GPT4). In addition, we create two synthetic datasets to assess whether LLMs perform better when annotating data as they generate it. We find GPT4 to be on par with human inter-annotator agreement and performance on synthetic data to be substantially higher, however we find the model still occasionally errs on issues of negation and factuality and higher performance on synthetic data is driven by greater complexity of real data rather than inherent advantage.
• 摘要：近年来，儿童和青少年的心理健康状况持续恶化[1]。最近出现的大型语言模型（LLM）为监控和干预的成本和时间效率扩展带来了很大希望，然而，尽管学校欺凌和饮食失调等特别普遍的问题，以前的研究尚未调查该领域或开放性的表现信息提取，其中答案集不是预先确定的。我们创建了一个由 12-19 岁青少年 Reddit 帖子组成的新数据集，由精神科医生专家对以下类别进行注释：创伤、危险、病情、症状、自杀和治疗，并将专家标签与两个表现最好的法学硕士（GPT3.5 和 GPT3.5）的注释进行比较。 GPT4）。此外，我们创建了两个综合数据集来评估法学硕士在生成数据时对其进行注释时是否表现更好。我们发现 GPT4 与人类注释者间的一致性相当，并且在合成数据上的性能要高得多，但是我们发现该模型仍然偶尔会在否定和事实性问题上犯错误，并且合成数据上的更高性能是由真实数据的更大复杂性驱动的。数据而非固有优势。

Title: Evaluating Consistency and Reasoning Capabilities of Large Language Models

• Authors: Yash Saxena, Sarthak Chopra, Arunendra Mani Tripathi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.16478
• Pdf URL: https://arxiv.org/pdf/2404.16478
• Copy Paste: [[2404.16478]] Evaluating Consistency and Reasoning Capabilities of Large Language Models(https://arxiv.org/abs/2404.16478)
• Keywords: language model, llm, prompt
• Abstract: Large Language Models (LLMs) are extensively used today across various sectors, including academia, research, business, and finance, for tasks such as text generation, summarization, and translation. Despite their widespread adoption, these models often produce incorrect and misleading information, exhibiting a tendency to hallucinate. This behavior can be attributed to several factors, with consistency and reasoning capabilities being significant contributors. LLMs frequently lack the ability to generate explanations and engage in coherent reasoning, leading to inaccurate responses. Moreover, they exhibit inconsistencies in their outputs. This paper aims to evaluate and compare the consistency and reasoning capabilities of both public and proprietary LLMs. The experiments utilize the Boolq dataset as the ground truth, comprising questions, answers, and corresponding explanations. Queries from the dataset are presented as prompts to the LLMs, and the generated responses are evaluated against the ground truth answers. Additionally, explanations are generated to assess the models' reasoning abilities. Consistency is evaluated by repeatedly presenting the same query to the models and observing for variations in their responses. For measuring reasoning capabilities, the generated explanations are compared to the ground truth explanations using metrics such as BERT, BLEU, and F-1 scores. The findings reveal that proprietary models generally outperform public models in terms of both consistency and reasoning capabilities. However, even when presented with basic general knowledge questions, none of the models achieved a score of 90\% in both consistency and reasoning. This study underscores the direct correlation between consistency and reasoning abilities in LLMs and highlights the inherent reasoning challenges present in current language models.
• 摘要：如今，大型语言模型 (LLM) 广泛应用于学术界、研究、商业和金融等各个领域，用于文本生成、摘要和翻译等任务。尽管它们被广泛采用，但这些模型经常产生不正确和误导性的信息，表现出产生幻觉的倾向。这种行为可以归因于几个因素，其中一致性和推理能力是重要的贡献者。法学硕士经常缺乏产生解释和进行连贯推理的能力，导致回答不准确。此外，他们的输出也表现出不一致。本文旨在评估和比较公共法学硕士和专有法学硕士的一致性和推理能力。实验利用 Boolq 数据集作为基本事实，包括问题、答案和相应的解释。数据集中的查询作为提示呈现给法学硕士，并根据真实答案评估生成的响应。此外，还生成解释来评估模型的推理能力。通过向模型重复提出相同的查询并观察其响应的变化来评估一致性。为了测量推理能力，使用 BERT、BLEU 和 F-1 分数等指标将生成的解释与真实解释进行比较。研究结果表明，专有模型在一致性和推理能力方面通常优于公共模型。然而，即使提出基本的常识性问题，没有一个模型在一致性和推理方面都达到 90% 的分数。这项研究强调了法学硕士的一致性和推理能力之间的直接相关性，并强调了当前语言模型中存在的固有推理挑战。

Title: Evaluating Large Language Models on Time Series Feature Understanding: A Comprehensive Taxonomy and Benchmark

• Authors: Elizabeth Fons, Rachneet Kaur, Soham Palande, Zhen Zeng, Svitlana Vyetrenko, Tucker Balch
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.16563
• Pdf URL: https://arxiv.org/pdf/2404.16563
• Copy Paste: [[2404.16563]] Evaluating Large Language Models on Time Series Feature Understanding: A Comprehensive Taxonomy and Benchmark(https://arxiv.org/abs/2404.16563)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) offer the potential for automatic time series analysis and reporting, which is a critical task across many domains, spanning healthcare, finance, climate, energy, and many more. In this paper, we propose a framework for rigorously evaluating the capabilities of LLMs on time series understanding, encompassing both univariate and multivariate forms. We introduce a comprehensive taxonomy of time series features, a critical framework that delineates various characteristics inherent in time series data. Leveraging this taxonomy, we have systematically designed and synthesized a diverse dataset of time series, embodying the different outlined features. This dataset acts as a solid foundation for assessing the proficiency of LLMs in comprehending time series. Our experiments shed light on the strengths and limitations of state-of-the-art LLMs in time series understanding, revealing which features these models readily comprehend effectively and where they falter. In addition, we uncover the sensitivity of LLMs to factors including the formatting of the data, the position of points queried within a series and the overall time series length.
• 摘要：大型语言模型 (LLM) 提供了自动时间序列分析和报告的潜力，这是跨医疗保健、金融、气候、能源等许多领域的一项关键任务。在本文中，我们提出了一个框架，用于严格评估法学硕士在时间序列理解方面的能力，包括单变量和多元形式。我们引入了时间序列特征的综合分类法，这是一个描述时间序列数据固有的各种特征的关键框架。利用这种分类法，我们系统地设计和合成了多样化的时间序列数据集，体现了不同的概述特征。该数据集为评估法学硕士理解时间序列的熟练程度奠定了坚实的基础。我们的实验揭示了最先进的法学硕士在时间序列理解方面的优势和局限性，揭示了这些模型容易有效理解的特征以及它们的不足之处。此外，我们还揭示了法学硕士对数据格式、序列中查询点的位置以及总体时间序列长度等因素的敏感性。

Title: Exploring Internal Numeracy in Language Models: A Case Study on ALBERT

• Authors: Ulme Wennberg, Gustav Eje Henter
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.16574
• Pdf URL: https://arxiv.org/pdf/2404.16574
• Copy Paste: [[2404.16574]] Exploring Internal Numeracy in Language Models: A Case Study on ALBERT(https://arxiv.org/abs/2404.16574)
• Keywords: language model
• Abstract: It has been found that Transformer-based language models have the ability to perform basic quantitative reasoning. In this paper, we propose a method for studying how these models internally represent numerical data, and use our proposal to analyze the ALBERT family of language models. Specifically, we extract the learned embeddings these models use to represent tokens that correspond to numbers and ordinals, and subject these embeddings to Principal Component Analysis (PCA). PCA results reveal that ALBERT models of different sizes, trained and initialized separately, consistently learn to use the axes of greatest variation to represent the approximate ordering of various numerical concepts. Numerals and their textual counterparts are represented in separate clusters, but increase along the same direction in 2D space. Our findings illustrate that language models, trained purely to model text, can intuit basic mathematical concepts, opening avenues for NLP applications that intersect with quantitative reasoning.
• 摘要：人们发现基于 Transformer 的语言模型具有执行基本定量推理的能力。在本文中，我们提出了一种研究这些模型如何在内部表示数值数据的方法，并使用我们的建议来分析 ALBERT 系列语言模型。具体来说，我们提取这些模型用来表示与数字和序数相对应的标记的学习嵌入，并对这些嵌入进行主成分分析 (PCA)。 PCA 结果表明，单独训练和初始化的不同大小的 ALBERT 模型一致地学习使用最大变化的轴来表示各种数值概念的近似排序。数字及其文本对应物以单独的簇表示，但在 2D 空间中沿相同方向增加。我们的研究结果表明，纯粹为文本建模而训练的语言模型可以凭直觉理解基本的数学概念，为与定量推理相交叉的 NLP 应用开辟途径。

Title: Understanding Privacy Risks of Embeddings Induced by Large Language Models

• Authors: Zhihao Zhu, Ninglu Shao, Defu Lian, Chenwang Wu, Zheng Liu, Yi Yang, Enhong Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.16587
• Pdf URL: https://arxiv.org/pdf/2404.16587
• Copy Paste: [[2404.16587]] Understanding Privacy Risks of Embeddings Induced by Large Language Models(https://arxiv.org/abs/2404.16587)
• Keywords: language model, llm, hallucination, retrieval-augmented generation
• Abstract: Large language models (LLMs) show early signs of artificial general intelligence but struggle with hallucinations. One promising solution to mitigate these hallucinations is to store external knowledge as embeddings, aiding LLMs in retrieval-augmented generation. However, such a solution risks compromising privacy, as recent studies experimentally showed that the original text can be partially reconstructed from text embeddings by pre-trained language models. The significant advantage of LLMs over traditional pre-trained models may exacerbate these concerns. To this end, we investigate the effectiveness of reconstructing original knowledge and predicting entity attributes from these embeddings when LLMs are employed. Empirical findings indicate that LLMs significantly improve the accuracy of two evaluated tasks over those from pre-trained models, regardless of whether the texts are in-distribution or out-of-distribution. This underscores a heightened potential for LLMs to jeopardize user privacy, highlighting the negative consequences of their widespread use. We further discuss preliminary strategies to mitigate this risk.
• 摘要：大型语言模型（LLM）显示出通用人工智能的早期迹象，但与幻觉作斗争。减轻这些幻觉的一种有前途的解决方案是将外部知识存储为嵌入，帮助法学硕士进行检索增强生成。然而，这种解决方案存在损害隐私的风险，因为最近的研究实验表明，可以通过预先训练的语言模型从文本嵌入部分重建原始文本。法学硕士相对于传统预训练模型的显着优势可能会加剧这些担忧。为此，我们研究了使用法学硕士时重建原始知识和从这些嵌入预测实体属性的有效性。实证结果表明，与预训练模型相比，法学硕士显着提高了两项评估任务的准确性，无论文本是分布内还是分布外。这凸显了法学硕士危害用户隐私的可能性更大，凸显了其广泛使用的负面后果。我们进一步讨论减轻这种风险的初步策略。

Title: Tele-FLM Technical Report

• Authors: Xiang Li, Yiqun Yao, Xin Jiang, Xuezhi Fang, Chao Wang, Xinzhang Liu, Zihan Wang, Yu Zhao, Xin Wang, Yuyao Huang, Shuangyong Song, Yongxiang Li, Zheng Zhang, Bo Zhao, Aixin Sun, Yequan Wang, Zhongjiang He, Zhongyuan Wang, Xuelong Li, Tiejun Huang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.16645
• Pdf URL: https://arxiv.org/pdf/2404.16645
• Copy Paste: [[2404.16645]] Tele-FLM Technical Report(https://arxiv.org/abs/2404.16645)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have showcased profound capabilities in language understanding and generation, facilitating a wide array of applications. However, there is a notable paucity of detailed, open-sourced methodologies on efficiently scaling LLMs beyond 50 billion parameters with minimum trial-and-error cost and computational resources. In this report, we introduce Tele-FLM (aka FLM-2), a 52B open-sourced multilingual large language model that features a stable, efficient pre-training paradigm and enhanced factual judgment capabilities. Tele-FLM demonstrates superior multilingual language modeling abilities, measured by BPB on textual corpus. Besides, in both English and Chinese foundation model evaluation, it is comparable to strong open-sourced models that involve larger pre-training FLOPs, such as Llama2-70B and DeepSeek-67B. In addition to the model weights, we share the core designs, engineering practices, and training details, which we expect to benefit both the academic and industrial communities.
• 摘要：大型语言模型（LLM）展示了语言理解和生成方面的深厚能力，促进了广泛的应用。然而，如何以最小的试错成本和计算资源有效地将 LLM 扩展到超过 500 亿个参数，详细的开源方法明显缺乏。在本报告中，我们介绍了 Tele-FLM（又名 FLM-2），这是一个 52B 开源多语言大语言模型，具有稳定、高效的预训练范式和增强的事实判断能力。 Tele-FLM 展示了卓越的多语言语言建模能力，通过 BPB 在文本语料库上进行测量。此外，在英文和中文基础模型评估中，它可以与预训练 FLOP 较大的强大开源模型（如 Llama2-70B 和 DeepSeek-67B）相媲美。除了模型权重之外，我们还分享核心设计、工程实践和培训细节，希望学术界和工业界都能受益。

Title: Análise de ambiguidade linguística em modelos de linguagem de grande escala (LLMs)

• Authors: Lavínia de Carvalho Moraes, Irene Cristina Silvério, Rafael Alexandre Sousa Marques, Bianca de Castro Anaia, Dandara Freitas de Paula, Maria Carolina Schincariol de Faria, Iury Cleveston, Alana de Santana Correia, Raquel Meister Ko Freitag
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.16653
• Pdf URL: https://arxiv.org/pdf/2404.16653
• Copy Paste: [[2404.16653]] Análise de ambiguidade linguística em modelos de linguagem de grande escala (LLMs)(https://arxiv.org/abs/2404.16653)
• Keywords: gpt, llm, chat
• Abstract: Linguistic ambiguity continues to represent a significant challenge for natural language processing (NLP) systems, notwithstanding the advancements in architectures such as Transformers and BERT. Inspired by the recent success of instructional models like ChatGPT and Gemini (In 2023, the artificial intelligence was called Bard.), this study aims to analyze and discuss linguistic ambiguity within these models, focusing on three types prevalent in Brazilian Portuguese: semantic, syntactic, and lexical ambiguity. We create a corpus comprising 120 sentences, both ambiguous and unambiguous, for classification, explanation, and disambiguation. The models capability to generate ambiguous sentences was also explored by soliciting sets of sentences for each type of ambiguity. The results underwent qualitative analysis, drawing on recognized linguistic references, and quantitative assessment based on the accuracy of the responses obtained. It was evidenced that even the most sophisticated models, such as ChatGPT and Gemini, exhibit errors and deficiencies in their responses, with explanations often providing inconsistent. Furthermore, the accuracy peaked at 49.58 percent, indicating the need for descriptive studies for supervised learning.
• 摘要：尽管 Transformers 和 BERT 等架构取得了进步，但语言歧义仍然是自然语言处理 (NLP) 系统面临的重大挑战。受到 ChatGPT 和 Gemini（2023 年，人工智能被称为 Bard）等教学模型最近成功的启发，本研究旨在分析和讨论这些模型中的语言歧义，重点关注巴西葡萄牙语中普遍存在的三种类型：语义、句法和词汇歧义。我们创建了一个包含 120 个句子的语料库，包括歧义和明确的句子，用于分类、解释和消歧。还通过为每种类型的歧义征求句子集来探索模型生成歧义句子的能力。结果经过了定性分析，借鉴了公认的语言参考，并根据所获得的回答的准确性进行了定量评估。事实证明，即使是最复杂的模型，例如 ChatGPT 和 Gemini，其响应也会出现错误和缺陷，并且提供的解释往往不一致。此外，准确率达到峰值 49.58%，表明监督学习需要描述性研究。

Title: ProbGate at EHRSQL 2024: Enhancing SQL Query Generation Accuracy through Probabilistic Threshold Filtering and Error Handling

• Authors: Sangryul Kim, Donghee Han, Sehyun Kim
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.16659
• Pdf URL: https://arxiv.org/pdf/2404.16659
• Copy Paste: [[2404.16659]] ProbGate at EHRSQL 2024: Enhancing SQL Query Generation Accuracy through Probabilistic Threshold Filtering and Error Handling(https://arxiv.org/abs/2404.16659)
• Keywords: language model
• Abstract: Recently, deep learning-based language models have significantly enhanced text-to-SQL tasks, with promising applications in retrieving patient records within the medical domain. One notable challenge in such applications is discerning unanswerable queries. Through fine-tuning model, we demonstrate the feasibility of converting medical record inquiries into SQL queries. Additionally, we introduce an entropy-based method to identify and filter out unanswerable results. We further enhance result quality by filtering low-confidence SQL through log probability-based distribution, while grammatical and schema errors are mitigated by executing queries on the actual database. We experimentally verified that our method can filter unanswerable questions, which can be widely utilized even when the parameters of the model are not accessible, and that it can be effectively utilized in practice.
• 摘要：最近，基于深度学习的语言模型显着增强了文本到 SQL 的任务，在医疗领域检索患者记录方面具有广阔的应用前景。此类应用程序中的一项显着挑战是识别无法回答的查询。通过微调模型，我们论证了将病历查询转换为SQL查询的可行性。此外，我们引入了一种基于熵的方法来识别和过滤掉无法回答的结果。我们通过基于日志概率的分布过滤低置信度 SQL，进一步提高结果质量，同时通过在实际数据库上执行查询来减少语法和模式错误。我们通过实验验证了我们的方法可以过滤无法回答的问题，即使在模型参数不可访问的情况下也可以广泛使用，并且可以在实践中有效利用。

Title: Influence of Solution Efficiency and Valence of Instruction on Additive and Subtractive Solution Strategies in Humans and GPT-4

• Authors: Lydia Uhler, Verena Jordan, Jürgen Buder, Markus Huff, Frank Papenmeier
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.16692
• Pdf URL: https://arxiv.org/pdf/2404.16692
• Copy Paste: [[2404.16692]] Influence of Solution Efficiency and Valence of Instruction on Additive and Subtractive Solution Strategies in Humans and GPT-4(https://arxiv.org/abs/2404.16692)
• Keywords: language model, gpt
• Abstract: We explored the addition bias, a cognitive tendency to prefer adding elements over removing them to alter an initial state or structure, by conducting four preregistered experiments examining the problem-solving behavior of both humans and OpenAl's GPT-4 large language model. The experiments involved 588 participants from the U.S. and 680 iterations of the GPT-4 model. The problem-solving task was either to create symmetry within a grid (Experiments 1 and 3) or to edit a summary (Experiments 2 and 4). As hypothesized, we found that overall, the addition bias was present. Solution efficiency (Experiments 1 and 2) and valence of the instruction (Experiments 3 and 4) played important roles. Human participants were less likely to use additive strategies when subtraction was relatively more efficient than when addition and subtraction were equally efficient. GPT-4 exhibited the opposite behavior, with a strong addition bias when subtraction was more efficient. In terms of instruction valence, GPT-4 was more likely to add words when asked to "improve" compared to "edit", whereas humans did not show this effect. When we looked at the addition bias under different conditions, we found more biased responses for GPT-4 compared to humans. Our findings highlight the importance of considering comparable and sometimes superior subtractive alternatives, as well as reevaluating one's own and particularly the language models' problem-solving behavior.
• 摘要：我们通过进行四个预先注册的实验来检查人类和 OpenAl 的 GPT-4 大语言模型的问题解决行为，探索了加法偏差，即一种更喜欢添加元素而不是删除元素来改变初始状态或结构的认知倾向。这些实验涉及来自美国的 588 名参与者和 GPT-4 模型的 680 次迭代。解决问题的任务是在网格内创建对称性（实验 1 和 3）或编辑摘要（实验 2 和 4）。正如假设的那样，我们发现总体上存在加法偏差。解决方案效率（实验 1 和 2）和指令效价（实验 3 和 4）发挥了重要作用。当减法比加法和减法同等有效时相对更有效时，人类参与者不太可能使用加法策略。 GPT-4 表现出相反的行为，当减法更有效时，具有很强的加法偏差。就指令效价而言，与“编辑”相比，GPT-4 在被要求“改进”时更有可能添加单词，而人类则没有表现出这种效果。当我们观察不同条件下的加法偏差时，我们发现与人类相比，GPT-4 的反应有更多偏差。我们的研究结果强调了考虑可比较的、有时甚至是更好的减法替代方案的重要性，以及重新评估自己的、特别是语言模型的问题解决行为的重要性。

Title: Cooperate or Collapse: Emergence of Sustainability Behaviors in a Society of LLM Agents

• Authors: Giorgio Piatti, Zhijing Jin, Max Kleiman-Weiner, Bernhard Schölkopf, Mrinmaya Sachan, Rada Mihalcea
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.16698
• Pdf URL: https://arxiv.org/pdf/2404.16698
• Copy Paste: [[2404.16698]] Cooperate or Collapse: Emergence of Sustainability Behaviors in a Society of LLM Agents(https://arxiv.org/abs/2404.16698)
• Keywords: language model, llm, prompt, agent
• Abstract: In the rapidly evolving field of artificial intelligence, ensuring safe decision-making of Large Language Models (LLMs) is a significant challenge. This paper introduces Governance of the Commons Simulation (GovSim), a simulation platform designed to study strategic interactions and cooperative decision-making in LLMs. Through this simulation environment, we explore the dynamics of resource sharing among AI agents, highlighting the importance of ethical considerations, strategic planning, and negotiation skills. GovSim is versatile and supports any text-based agent, including LLMs agents. Using the Generative Agent framework, we create a standard agent that facilitates the integration of different LLMs. Our findings reveal that within GovSim, only two out of 15 tested LLMs managed to achieve a sustainable outcome, indicating a significant gap in the ability of models to manage shared resources. Furthermore, we find that by removing the ability of agents to communicate, they overuse the shared resource, highlighting the importance of communication for cooperation. Interestingly, most LLMs lack the ability to make universalized hypotheses, which highlights a significant weakness in their reasoning skills. We open source the full suite of our research results, including the simulation environment, agent prompts, and a comprehensive web interface.
• 摘要：在快速发展的人工智能领域，确保大型语言模型 (LLM) 的安全决策是一项重大挑战。本文介绍了公共治理模拟（GovSim），这是一个旨在研究法学硕士中的战略互动和合作决策的模拟平台。通过这个模拟环境，我们探索人工智能代理之间资源共享的动态，强调道德考虑、战略规划和谈判技巧的重要性。 GovSim 用途广泛，支持任何基于文本的代理，包括法学硕士代理。使用生成代理框架，我们创建了一个标准代理，以促进不同法学硕士的集成。我们的研究结果表明，在 GovSim 中，15 个接受测试的法学硕士中只有两个能够实现可持续的结果，这表明模型管理共享资源的能力存在显着差距。此外，我们发现，通过消除代理的通信能力，他们过度使用共享资源，凸显了通信对于合作的重要性。有趣的是，大多数法学硕士缺乏提出普遍假设的能力，这凸显了他们推理能力的重大弱点。我们开源了全套研究成果，包括模拟环境、代理提示和综合网络界面。

Title: Layer Skip: Enabling Early Exit Inference and Self-Speculative Decoding

• Authors: Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, Ahmed A Aly, Beidi Chen, Carole-Jean Wu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.16710
• Pdf URL: https://arxiv.org/pdf/2404.16710
• Copy Paste: [[2404.16710]] Layer Skip: Enabling Early Exit Inference and Self-Speculative Decoding(https://arxiv.org/abs/2404.16710)
• Keywords: language model, llm
• Abstract: We present LayerSkip, an end-to-end solution to speed-up inference of large language models (LLMs). First, during training we apply layer dropout, with low dropout rates for earlier layers and higher dropout rates for later layers, and an early exit loss where all transformer layers share the same exit. Second, during inference, we show that this training recipe increases the accuracy of early exit at earlier layers, without adding any auxiliary layers or modules to the model. Third, we present a novel self-speculative decoding solution where we exit at early layers and verify and correct with remaining layers of the model. Our proposed self-speculative decoding approach has less memory footprint than other speculative decoding approaches and benefits from shared compute and activations of the draft and verification stages. We run experiments on different Llama model sizes on different types of training: pretraining from scratch, continual pretraining, finetuning on specific data domain, and finetuning on specific task. We implement our inference solution and show speedups of up to 2.16x on summarization for CNN/DM documents, 1.82x on coding, and 2.0x on TOPv2 semantic parsing task.
• 摘要：我们推出了 LayerSkip，这是一种用于加速大型语言模型 (LLM) 推理的端到端解决方案。首先，在训练期间，我们应用层丢失，早期层的丢失率较低，后面层的丢失率较高，以及所有变压器层共享相同出口的早期退出损失。其次，在推理过程中，我们表明这种训练方法提高了早期层提前退出的准确性，而无需向模型添加任何辅助层或模块。第三，我们提出了一种新颖的自推测解码解决方案，我们在早期层退出并验证和纠正模型的其余层。我们提出的自推测解码方法比其他推测解码方法具有更少的内存占用，并且受益于草稿和验证阶段的共享计算和激活。我们在不同类型的训练中对不同大小的 Llama 模型进行实验：从头开始预训练、持续预训练、特定数据域的微调以及特定任务的微调。我们实现了推理解决方案，并在 CNN/DM 文档的摘要方面显示了高达 2.16 倍的加速，在编码方面加速了 1.82 倍，在 TOPv2 语义解析任务上加速了 2.0 倍。

Title: Prefix Text as a Yarn: Eliciting Non-English Alignment in Foundation Language Model

• Authors: Runzhe Zhan, Xinyi Yang, Derek F. Wong, Lidia S. Chao, Yue Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.16766
• Pdf URL: https://arxiv.org/pdf/2404.16766
• Copy Paste: [[2404.16766]] Prefix Text as a Yarn: Eliciting Non-English Alignment in Foundation Language Model(https://arxiv.org/abs/2404.16766)
• Keywords: language model, llm
• Abstract: While supervised fine-tuning (SFT) has been a straightforward approach for tailoring the output of foundation large language model (LLM) to specific preferences, concerns have been raised about the depth of this alignment, with some critiques suggesting it is merely "superficial". We critically examine this hypothesis within the scope of cross-lingual generation tasks, proposing that the effectiveness of SFT may be constrained by its reliance on prior tokens to guide cross-lingual generation. Based on this crucial insight, and in response to the challenges posed by the costly and limited availability of non-English data for SFT, we introduce a novel training-free alignment method named PreTTY, which employs minimal task-related prior tokens to bridge the foundation LLM and the SFT LLM, achieving comparable performance without training. Experiments on machine translation and part-of-speech tagging across eight languages demonstrate the efficacy of PreTTY in cross-lingual settings. Remarkably, by initiating the decoding process with only one or two prior tokens, foundation LLMs can achieve performance comparable to their SFT counterparts. This method presents a cost-effective alternative to SFT and advances the democratization of multilingual LLMs.
• 摘要：虽然监督微调 (SFT) 是一种根据特定偏好定制基础大语言模型 (LLM) 输出的直接方法，但人们对这种对齐的深度提出了担忧，一些批评认为它只是“肤浅的” 。我们在跨语言生成任务的范围内批判性地检验了这一假设，提出 SFT 的有效性可能因其依赖先验标记来指导跨语言生成而受到限制。基于这一重要见解，并为了应对 SFT 的非英语数据成本高昂且可用性有限所带来的挑战，我们引入了一种名为 PreTTY 的新颖的免训练对齐方法，该方法采用最少的与任务相关的先验标记来桥接基础法学硕士和 SFT 法学硕士，无需培训即可达到可比的表现。跨八种语言的机器翻译和词性标记实验证明了 PreTTY 在跨语言环境中的功效。值得注意的是，通过仅使用一两个先前令牌启动解码过程，基础 LLM 可以获得与 SFT 对应物相当的性能。该方法为 SFT 提供了一种经济高效的替代方案，并促进了多语言法学硕士的民主化。

Title: Improving Diversity of Commonsense Generation by Large Language Models via In-Context Learning

• Authors: Tianhui Zhang, Bei Peng, Danushka Bollegala
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.16807
• Pdf URL: https://arxiv.org/pdf/2404.16807
• Copy Paste: [[2404.16807]] Improving Diversity of Commonsense Generation by Large Language Models via In-Context Learning(https://arxiv.org/abs/2404.16807)
• Keywords: language model, llm
• Abstract: Generative Commonsense Reasoning (GCR) requires a model to reason about a situation using commonsense knowledge, while generating coherent sentences. Although the quality of the generated sentences is crucial, the diversity of the generation is equally important because it reflects the model's ability to use a range of commonsense knowledge facts. Large Language Models (LLMs) have shown proficiency in enhancing the generation quality across various tasks through in-context learning (ICL) using given examples without the need for any fine-tuning. However, the diversity aspect in LLM outputs has not been systematically studied before. To address this, we propose a simple method that diversifies the LLM generations, while preserving their quality. Experimental results on three benchmark GCR datasets show that our method achieves an ideal balance between the quality and diversity. Moreover, the sentences generated by our proposed method can be used as training data to improve diversity in existing commonsense generators.
• 摘要：生成常识推理（GCR）需要一个模型使用常识知识来推理情况，同时生成连贯的句子。尽管生成句子的质量至关重要，但生成的多样性也同样重要，因为它反映了模型使用一系列常识性知识事实的能力。大型语言模型 (LLM) 已显示出通过使用给定示例的上下文学习 (ICL) 能够熟练地提高各种任务的生成质量，而无需进行任何微调。然而，之前尚未系统地研究过 LLM 产出的多样性。为了解决这个问题，我们提出了一种简单的方法，可以使法学硕士的世代多样化，同时保持其质量。在三个基准 GCR 数据集上的实验结果表明，我们的方法在质量和多样性之间实现了理想的平衡。此外，我们提出的方法生成的句子可以用作训练数据，以提高现有常识生成器的多样性。

Title: Make Your LLM Fully Utilize the Context

• Authors: Shengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng, Jian-Guang Lou
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.16811
• Pdf URL: https://arxiv.org/pdf/2404.16811
• Copy Paste: [[2404.16811]] Make Your LLM Fully Utilize the Context(https://arxiv.org/abs/2404.16811)
• Keywords: language model, llm, long context
• Abstract: While many contemporary large language models (LLMs) can process lengthy input, they still struggle to fully utilize information within the long context, known as the lost-in-the-middle challenge. We hypothesize that it stems from insufficient explicit supervision during the long-context training, which fails to emphasize that any position in a long context can hold crucial information. Based on this intuition, our study presents information-intensive (IN2) training, a purely data-driven solution to overcome lost-in-the-middle. Specifically, IN2 training leverages a synthesized long-context question-answer dataset, where the answer requires (1) fine-grained information awareness on a short segment (~128 tokens) within a synthesized long context (4K-32K tokens), and (2) the integration and reasoning of information from two or more short segments. Through applying this information-intensive training on Mistral-7B, we present FILM-7B (FILl-in-the-Middle). To thoroughly assess the ability of FILM-7B for utilizing long contexts, we design three probing tasks that encompass various context styles (document, code, and structured-data context) and information retrieval patterns (forward, backward, and bi-directional retrieval). The probing results demonstrate that FILM-7B can robustly retrieve information from different positions in its 32K context window. Beyond these probing tasks, FILM-7B significantly improves the performance on real-world long-context tasks (e.g., 23.5->26.9 F1 score on NarrativeQA), while maintaining a comparable performance on short-context tasks (e.g., 59.3->59.2 accuracy on MMLU). Github Link: https://github.com/microsoft/FILM.
• 摘要：尽管许多当代大型语言模型（LLM）可以处理冗长的输入，但它们仍然难以充分利用长上下文中的信息，这被称为“迷失在中间的挑战”。我们假设这是由于在长上下文训练过程中缺乏明确的监督，没有强调长上下文中的任何位置都可以保存关键信息。基于这种直觉，我们的研究提出了信息密集型（IN2）训练，这是一种纯粹的数据驱动解决方案，可以克服中间迷失的问题。具体来说，IN2 训练利用合成的长上下文问答数据集，其中答案需要 (1) 对合成的长上下文（4K-32K 标记）内的短片段（~128 个标记）进行细粒度的信息感知，并且（ 2）对两个或多个短片段的信息进行整合和推理。通过在 Mistral-7B 上应用这种信息密集型训练，我们提出了 FILM-7B（FILl-in-the-Middle）。为了彻底评估 FILM-7B 利用长上下文的能力，我们设计了三个探测任务，涵盖各种上下文样式（文档、代码和结构化数据上下文）和信息检索模式（前向、后向和双向检索） 。探测结果表明 FILM-7B 可以从其 32K 上下文窗口中的不同位置稳健地检索信息。除了这些探测任务之外，FILM-7B 还显着提高了现实世界长上下文任务的性能（例如，NarrativeQA 上的 F1 分数为 23.5->26.9），同时在短上下文任务上保持了可比较的性能（例如，59.3->59.2） MMLU 的准确性）。 Github 链接：https://github.com/microsoft/FILM。

Title: IndicGenBench: A Multilingual Benchmark to Evaluate Generation Capabilities of LLMs on Indic Languages

• Authors: Harman Singh, Nitish Gupta, Shikhar Bharadwaj, Dinesh Tewari, Partha Talukdar
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.16816
• Pdf URL: https://arxiv.org/pdf/2404.16816
• Copy Paste: [[2404.16816]] IndicGenBench: A Multilingual Benchmark to Evaluate Generation Capabilities of LLMs on Indic Languages(https://arxiv.org/abs/2404.16816)
• Keywords: language model, gpt, llm
• Abstract: As large language models (LLMs) see increasing adoption across the globe, it is imperative for LLMs to be representative of the linguistic diversity of the world. India is a linguistically diverse country of 1.4 Billion people. To facilitate research on multilingual LLM evaluation, we release IndicGenBench - the largest benchmark for evaluating LLMs on user-facing generation tasks across a diverse set 29 of Indic languages covering 13 scripts and 4 language families. IndicGenBench is composed of diverse generation tasks like cross-lingual summarization, machine translation, and cross-lingual question answering. IndicGenBench extends existing benchmarks to many Indic languages through human curation providing multi-way parallel evaluation data for many under-represented Indic languages for the first time. We evaluate a wide range of proprietary and open-source LLMs including GPT-3.5, GPT-4, PaLM-2, mT5, Gemma, BLOOM and LLaMA on IndicGenBench in a variety of settings. The largest PaLM-2 models performs the best on most tasks, however, there is a significant performance gap in all languages compared to English showing that further research is needed for the development of more inclusive multilingual language models. IndicGenBench is released at www.github.com/google-research-datasets/indic-gen-bench
• 摘要：随着大型语言模型 (LLM) 在全球范围内的采用不断增加，LLM 必须能够代表世界的语言多样性。印度是一个拥有 14 亿人口的语言多元化国家。为了促进多语言 LLM 评估的研究，我们发布了 IndicGenBench - 这是评估 LLM 在面向用户的生成任务上的最大基准，涵盖 29 种不同的印度语言，涵盖 13 种文字和 4 个语系。 IndicGenBench由跨语言摘要、机器翻译、跨语言问答等多种生成任务组成。 IndicGenBench 通过人工管理将现有基准扩展到许多印度语言，首次为许多代表性不足的印度语言提供多路并行评估数据。我们在 IndicGenBench 上的各种设置中评估各种专有和开源 LLM，包括 GPT-3.5、GPT-4、PaLM-2、mT5、Gemma、BLOOM 和 LLaMA。最大的 PaLM-2 模型在大多数任务上表现最好，然而，与英语相比，所有语言都存在显着的性能差距，这表明需要进一步研究来开发更具包容性的多语言语言模型。 IndicGenBench 发布于 www.github.com/google-research-datasets/indic-gen-bench