2024-05-17

Title: Elements of World Knowledge (EWOK): A cognition-inspired framework for evaluating basic world knowledge in language models

• Authors: Anna A. Ivanova, Aalok Sathe, Benjamin Lipkin, Unnathi Kumar, Setayesh Radkani, Thomas H. Clark, Carina Kauf, Jennifer Hu, R.T. Pramod, Gabriel Grand, Vivian Paulun, Maria Ryskina, Ekin Akyurek, Ethan Wilcox, Nafisa Rashid, Leshem Chosen, Roger Levy, Evelina Fedorenko, Joshua Tenenbaum, Jacob Andreas
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Elements of World Knowledge (EWOK): A cognition-inspired framework for evaluating basic world knowledge in language models(https://arxiv.org/abs/)
• Keywords: language model, llm, agent
• Abstract: The ability to build and leverage world models is essential for a general-purpose AI agent. Testing such capabilities is hard, in part because the building blocks of world models are ill-defined. We present Elements of World Knowledge (EWOK), a framework for evaluating world modeling in language models by testing their ability to use knowledge of a concept to match a target text with a plausible/implausible context. EWOK targets specific concepts from multiple knowledge domains known to be vital for world modeling in humans. Domains range from social interactions (help/hinder) to spatial relations (left/right). Both, contexts and targets are minimal pairs. Objects, agents, and locations in the items can be flexibly filled in enabling easy generation of multiple controlled datasets. We then introduce EWOK-CORE-1.0, a dataset of 4,374 items covering 11 world knowledge domains. We evaluate 20 openweights large language models (1.3B--70B parameters) across a battery of evaluation paradigms along with a human norming study comprising 12,480 measurements. The overall performance of all tested models is worse than human performance, with results varying drastically across domains. These data highlight simple cases where even large models fail and present rich avenues for targeted research on LLM world modeling capabilities.
• 摘要：构建和利用世界模型的能力对于通用人工智能代理至关重要。测试这种能力很困难，部分原因是世界模型的构建模块定义不明确。我们提出了世界知识元素（EWOK），这是一个评估语言模型中的世界建模的框架，通过测试语言模型使用概念知识将目标文本与合理/难以置信的上下文进行匹配的能力。 EWOK 针对已知对人类世界建模至关重要的多个知识领域的特定概念。领域范围从社交互动（帮助/阻碍）到空间关系（左/右）。上下文和目标都是最小对。可以灵活地填充项目中的对象、代理和位置，从而轻松生成多个受控数据集。然后我们介绍 EWOK-CORE-1.0，这是一个包含 4,374 个项目的数据集，涵盖 11 个世界知识领域。我们通过一系列评估范式以及包含 12,480 个测量值的人类规范研究来评估 20 个开放权重大型语言模型（1.3B--70B 参数）。所有测试模型的整体性能都比人类性能差，不同领域的结果差异很大。这些数据突出了即使是大型模型也会失败的简单案例，并为 LLM 世界建模能力的有针对性的研究提供了丰富的途径。

Title: Simulating Policy Impacts: Developing a Generative Scenario Writing Method to Evaluate the Perceived Effects of Regulation

• Authors: Julia Barnett, Kimon Kieslich, Nicholas Diakopoulos
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.09679
• Pdf URL: https://arxiv.org/pdf/2405.09679
• Copy Paste: [[2405.09679]] Simulating Policy Impacts: Developing a Generative Scenario Writing Method to Evaluate the Perceived Effects of Regulation(https://arxiv.org/abs/2405.09679)
• Keywords: language model, gpt, llm
• Abstract: The rapid advancement of AI technologies yields numerous future impacts on individuals and society. Policy-makers are therefore tasked to react quickly and establish policies that mitigate those impacts. However, anticipating the effectiveness of policies is a difficult task, as some impacts might only be observable in the future and respective policies might not be applicable to the future development of AI. In this work we develop a method for using large language models (LLMs) to evaluate the efficacy of a given piece of policy at mitigating specified negative impacts. We do so by using GPT-4 to generate scenarios both pre- and post-introduction of policy and translating these vivid stories into metrics based on human perceptions of impacts. We leverage an already established taxonomy of impacts of generative AI in the media environment to generate a set of scenario pairs both mitigated and non-mitigated by the transparency legislation of Article 50 of the EU AI Act. We then run a user study (n=234) to evaluate these scenarios across four risk-assessment dimensions: severity, plausibility, magnitude, and specificity to vulnerable populations. We find that this transparency legislation is perceived to be effective at mitigating harms in areas such as labor and well-being, but largely ineffective in areas such as social cohesion and security. Through this case study on generative AI harms we demonstrate the efficacy of our method as a tool to iterate on the effectiveness of policy on mitigating various negative impacts. We expect this method to be useful to researchers or other stakeholders who want to brainstorm the potential utility of different pieces of policy or other mitigation strategies.
• 摘要：人工智能技术的快速发展对个人和社会产生了许多未来影响。因此，政策制定者的任务是迅速做出反应并制定减轻这些影响的政策。然而，预测政策的有效性是一项艰巨的任务，因为有些影响可能只有在未来才能观察到，相关政策可能不适用于人工智能的未来发展。在这项工作中，我们开发了一种使用大型语言模型（LLM）来评估特定政策在减轻特定负面影响方面的有效性的方法。为此，我们使用 GPT-4 生成政策出台前后的场景，并将这些生动的故事转化为基于人类对影响的感知的指标。我们利用已经建立的生成式人工智能在媒体环境中影响的分类法，生成一组由欧盟人工智能法案第 50 条的透明度立法减轻和未减轻的场景对。然后，我们进行了一项用户研究 (n=234)，从四个风险评估维度评估这些情景：严重性、合理性、严重性和对弱势群体的特异性。我们发现，这种透明度立法被认为可以有效减轻劳工和福祉等领域的危害，但在社会凝聚力和安全等领域基本上无效。通过这个关于生成性人工智能危害的案例研究，我们证明了我们的方法作为工具的有效性，可以迭代减轻各种负面影响的政策的有效性。我们希望这种方法对想要集思广益不同政策或其他缓解策略的潜在效用的研究人员或其他利益相关者有用。

Title: Spectral Editing of Activations for Large Language Model Alignment

• Authors: Yifu Qiu, Zheng Zhao, Yftah Ziser, Anna Korhonen, Edoardo M. Ponti, Shay B. Cohen
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.09719
• Pdf URL: https://arxiv.org/pdf/2405.09719
• Copy Paste: [[2405.09719]] Spectral Editing of Activations for Large Language Model Alignment(https://arxiv.org/abs/2405.09719)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) often exhibit undesirable behaviours, such as generating untruthful or biased content. Editing their internal representations has been shown to be effective in mitigating such behaviours on top of the existing alignment methods. We propose a novel inference-time editing method, namely spectral editing of activations (SEA), to project the input representations into directions with maximal covariance with the positive demonstrations (e.g., truthful) while minimising covariance with the negative demonstrations (e.g., hallucinated). We also extend our method to non-linear editing using feature functions. We run extensive experiments on benchmarks concerning truthfulness and bias with six open-source LLMs of different sizes and model families. The results demonstrate the superiority of SEA in effectiveness, generalisation to similar tasks, as well as inference and data efficiency. We also show that SEA editing only has a limited negative impact on other model capabilities.
• 摘要：大型语言模型 (LLM) 通常会表现出不良行为，例如生成不真实或有偏见的内容。事实证明，在现有的对齐方法之上，编辑其内部表示可以有效地减轻此类行为。我们提出了一种新颖的推理时间编辑方法，即激活的频谱编辑（SEA），将输入表示投影到与正面演示（例如，真实的）具有最大协方差的方向，同时最小化与负面演示（例如，幻觉）的协方差。 。我们还使用特征函数将我们的方法扩展到非线性编辑。我们使用六个不同规模和模型系列的开源法学硕士，对有关真实性和偏见的基准进行了广泛的实验。结果证明了 SEA 在有效性、对类似任务的泛化以及推理和数据效率方面的优越性。我们还表明，SEA 编辑对其他模型功能仅产生有限的负面影响。

Title: Many Hands Make Light Work: Task-Oriented Dialogue System with Module-Based Mixture-of-Experts

• Authors: Ruolin Su, Biing-Hwang Juang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.09744
• Pdf URL: https://arxiv.org/pdf/2405.09744
• Copy Paste: [[2405.09744]] Many Hands Make Light Work: Task-Oriented Dialogue System with Module-Based Mixture-of-Experts(https://arxiv.org/abs/2405.09744)
• Keywords: language model
• Abstract: Task-oriented dialogue systems are broadly used in virtual assistants and other automated services, providing interfaces between users and machines to facilitate specific tasks. Nowadays, task-oriented dialogue systems have greatly benefited from pre-trained language models (PLMs). However, their task-solving performance is constrained by the inherent capacities of PLMs, and scaling these models is expensive and complex as the model size becomes larger. To address these challenges, we propose Soft Mixture-of-Expert Task-Oriented Dialogue system (SMETOD) which leverages an ensemble of Mixture-of-Experts (MoEs) to excel at subproblems and generate specialized outputs for task-oriented dialogues. SMETOD also scales up a task-oriented dialogue system with simplicity and flexibility while maintaining inference efficiency. We extensively evaluate our model on three benchmark functionalities: intent prediction, dialogue state tracking, and dialogue response generation. Experimental results demonstrate that SMETOD achieves state-of-the-art performance on most evaluated metrics. Moreover, comparisons against existing strong baselines show that SMETOD has a great advantage in the cost of inference and correctness in problem-solving.
• 摘要：面向任务的对话系统广泛用于虚拟助理和其他自动化服务，提供用户和机器之间的接口以促进特定任务。如今，面向任务的对话系统极大地受益于预训练语言模型（PLM）。然而，它们的任务解决性能受到 PLM 固有能力的限制，并且随着模型尺寸变大，扩展这些模型既昂贵又复杂。为了应对这些挑战，我们提出了面向任务的软专家混合对话系统（SMETOD），该系统利用混合专家（MoE）的集合来擅长解决子问题并为面向任务的对话生成专门的输出。 SMETOD 还以简单性和灵活性扩展了面向任务的对话系统，同时保持了推理效率。我们在三个基准功能上广泛评估我们的模型：意图预测、对话状态跟踪和对话响应生成。实验结果表明，SMETOD 在大多数评估指标上均实现了最先进的性能。此外，与现有强基线的比较表明，SMETOD 在推理成本和解决问题的正确性方面具有很大优势。

Title: Optimization Techniques for Sentiment Analysis Based on LLM (GPT-3)

• Authors: Tong Zhan, Chenxi Shi, Yadong Shi, Huixiang Li, Yiyu Lin
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.09770
• Pdf URL: https://arxiv.org/pdf/2405.09770
• Copy Paste: [[2405.09770]] Optimization Techniques for Sentiment Analysis Based on LLM (GPT-3)(https://arxiv.org/abs/2405.09770)
• Keywords: language model, gpt, llm
• Abstract: With the rapid development of natural language processing (NLP) technology, large-scale pre-trained language models such as GPT-3 have become a popular research object in NLP field. This paper aims to explore sentiment analysis optimization techniques based on large pre-trained language models such as GPT-3 to improve model performance and effect and further promote the development of natural language processing (NLP). By introducing the importance of sentiment analysis and the limitations of traditional methods, GPT-3 and Fine-tuning techniques are introduced in this paper, and their applications in sentiment analysis are explained in detail. The experimental results show that the Fine-tuning technique can optimize GPT-3 model and obtain good performance in sentiment analysis task. This study provides an important reference for future sentiment analysis using large-scale language models.
• 摘要：随着自然语言处理（NLP）技术的快速发展，GPT-3等大规模预训练语言模型已成为NLP领域的热门研究对象。本文旨在探索基于GPT-3等大型预训练语言模型的情感分析优化技术，以提高模型性能和效果，进一步推动自然语言处理（NLP）的发展。通过介绍情感分析的重要性和传统方法的局限性，本文介绍了GPT-3和Fine-tuning技术，并详细解释了它们在情感分析中的应用。实验结果表明Fine-tuning技术可以优化GPT-3模型并在情感分析任务中获得良好的性能。该研究为未来使用大规模语言模型进行情感分析提供了重要参考。

Title: SecureLLM: Using Compositionality to Build Provably Secure Language Models for Private, Sensitive, and Secret Data

• Authors: Abdulrahman Alabdulakreem, Christian M Arnold, Yerim Lee, Pieter M Feenstra, Boris Katz, Andrei Barbu
• Subjects: cs.CL, cs.CR
• Abstract URL: https://arxiv.org/abs/2405.09805
• Pdf URL: https://arxiv.org/pdf/2405.09805
• Copy Paste: [[2405.09805]] SecureLLM: Using Compositionality to Build Provably Secure Language Models for Private, Sensitive, and Secret Data(https://arxiv.org/abs/2405.09805)
• Keywords: language model, llm
• Abstract: Traditional security mechanisms isolate resources from users who should not access them. We reflect the compositional nature of such security mechanisms back into the structure of LLMs to build a provably secure LLM; that we term SecureLLM. Other approaches to LLM safety attempt to protect against bad actors or bad outcomes, but can only do so to an extent making them inappropriate for sensitive data. SecureLLM blends access security with fine-tuning methods. Each data silo has associated with it a separate fine-tuning and a user has access only to the collection of fine-tunings that they have permission for. The model must then perform on compositional tasks at the intersection of those data silos with the combination of those individual fine-tunings. While applicable to any task like document QA or making API calls, in this work we concern ourselves with models that learn the layouts of new SQL databases to provide natural-language-to-SQL translation capabilities. Existing fine-tuning composition methods fail in this challenging environment, as they are not well-equipped for handling compositional tasks. Compositionality remains a challenge for LLMs. We contribute both a difficult new compositional natural-language-to-SQL translation task and a new perspective on LLM security that allows models to be deployed to secure environments today.
• 摘要：传统的安全机制将资源与不应访问它们的用户隔离开来。我们将此类安全机制的组合性质反映回LLM的结构中，以构建可证明安全的LLM；我们称之为 SecureLLM。 LLM 安全的其他方法试图防止不良行为者或不良结果，但只能在一定程度上使其不适合敏感数据。 SecureLLM 将访问安全性与微调方法相结合。每个数据仓都与一个单独的微调相关联，并且用户只能访问他们有权访问的微调集合。然后，模型必须在这些数据孤岛的交叉点上执行组合任务，并结合这些单独的微调。虽然适用于文档 QA 或 API 调用等任何任务，但在这项工作中，我们关注的是学习新 SQL 数据库布局以提供自然语言到 SQL 翻译功能的模型。现有的微调合成方法在这种充满挑战的环境中失败了，因为它们没有足够的能力来处理合成任务。组合性仍然是法学硕士的一个挑战。我们贡献了一项困难的新组合自然语言到 SQL 翻译任务，以及 LLM 安全性的新视角，允许将模型部署到当今的安全环境中。

Title: Chameleon: Mixed-Modal Early-Fusion Foundation Models

• Authors: Chameleon Team
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Chameleon: Mixed-Modal Early-Fusion Foundation Models(https://arxiv.org/abs/)
• Keywords: gpt, prompt
• Abstract: We present Chameleon, a family of early-fusion token-based mixed-modal models capable of understanding and generating images and text in any arbitrary sequence. We outline a stable training approach from inception, an alignment recipe, and an architectural parameterization tailored for the early-fusion, token-based, mixed-modal setting. The models are evaluated on a comprehensive range of tasks, including visual question answering, image captioning, text generation, image generation, and long-form mixed modal generation. Chameleon demonstrates broad and general capabilities, including state-of-the-art performance in image captioning tasks, outperforms Llama-2 in text-only tasks while being competitive with models such as Mixtral 8x7B and Gemini-Pro, and performs non-trivial image generation, all in a single model. It also matches or exceeds the performance of much larger models, including Gemini Pro and GPT-4V, according to human judgments on a new long-form mixed-modal generation evaluation, where either the prompt or outputs contain mixed sequences of both images and text. Chameleon marks a significant step forward in a unified modeling of full multimodal documents.
• 摘要：我们提出了 Chameleon，这是一系列基于早期融合令牌的混合模式模型，能够理解和生成任意序列的图像和文本。我们从一开始就概述了稳定的训练方法、对齐方法以及为早期融合、基于令牌的混合模式设置量身定制的架构参数化。这些模型在一系列全面的任务上进行评估，包括视觉问答、图像字幕、文本生成、图像生成和长格式混合模态生成。 Chameleon 展示了广泛而通用的功能，包括在图像字幕任务中最先进的性能，在纯文本任务中优于 Llama-2，同时与 Mixtral 8x7B 和 Gemini-Pro 等模型竞争，并执行非平凡的图像一代，全部在一个模型中。根据人类对新的长形式混合模式生成评估的判断，它还匹配或超过了更大模型（包括 Gemini Pro 和 GPT-4V）的性能，其中提示或输出包含图像和文本的混合序列。 Chameleon 标志着完整多模式文档的统一建模向前迈出了重要一步。

Title: Enhancing Semantics in Multimodal Chain of Thought via Soft Negative Sampling

• Authors: Guangmin Zheng, Jin Wang, Xiaobing Zhou, Xuejie Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.09848
• Pdf URL: https://arxiv.org/pdf/2405.09848
• Copy Paste: [[2405.09848]] Enhancing Semantics in Multimodal Chain of Thought via Soft Negative Sampling(https://arxiv.org/abs/2405.09848)
• Keywords: hallucination
• Abstract: Chain of thought (CoT) has proven useful for problems requiring complex reasoning. Many of these problems are both textual and multimodal. Given the inputs in different modalities, a model generates a rationale and then uses it to answer a question. Because of the hallucination issue, the generated soft negative rationales with high textual quality but illogical semantics do not always help improve answer accuracy. This study proposes a rationale generation method using soft negative sampling (SNSE-CoT) to mitigate hallucinations in multimodal CoT. Five methods were applied to generate soft negative samples that shared highly similar text but had different semantics from the original. Bidirectional margin loss (BML) was applied to introduce them into the traditional contrastive learning framework that involves only positive and negative samples. Extensive experiments on the ScienceQA dataset demonstrated the effectiveness of the proposed method. Code and data are released at this https URL.
• 摘要：事实证明，思维链 (CoT) 对于需要复杂推理的问题非常有用。其中许多问题都是文本问题和多模态问题。给定不同模式的输入，模型会生成一个基本原理，然后用它来回答问题。由于幻觉问题，生成的文本质量高但语义不合逻辑的软否定理由并不总是有助于提高答案准确性。本研究提出了一种使用软负采样 (SNSE-CoT) 的基本原理生成方法来减轻多模态 CoT 中的幻觉。应用五种方法来生成软负样本，这些样本共享高度相似的文本，但与原始文本具有不同的语义。应用双向边缘损失（BML）将它们引入仅涉及正样本和负样本的传统​​对比学习框架中。 ScienceQA 数据集上的大量实验证明了该方法的有效性。代码和数据在此 https URL 发布。

Title: IGOT: Information Gain Optimized Tokenizer on Domain Adaptive Pretraining

• Authors: Dawei Feng, Yihai Zhang, Zhixuan Xu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.09857
• Pdf URL: https://arxiv.org/pdf/2405.09857
• Copy Paste: [[2405.09857]] IGOT: Information Gain Optimized Tokenizer on Domain Adaptive Pretraining(https://arxiv.org/abs/2405.09857)
• Keywords: language model, gpt, llm, chat
• Abstract: Pretrained Large Language Models (LLM) such as ChatGPT, Claude, etc. have demonstrated strong capabilities in various fields of natural language generation. However, there are still many problems when using LLM in specialized domain-specific fields. When using generative AI to process downstream tasks, a common approach is to add new knowledge (e.g., private domain knowledge, cutting-edge information) to a pretrained model through continued training or fine-tuning. However, whether there is a universal paradigm for domain adaptation training is still an open question. In this article, we proposed Information Gain Optimized Tokenizer (IGOT), which analyzes the special token set of downstream tasks, constructs a new subset using heuristic function $\phi$ with the special token and its information gain, to build new domain-specific tokenizer, and continues pretraining on the downstream task data. We explored the many positive effects of this method's customized tokenizer on domain-adaptive pretraining and verified this method can perform better than the ordinary method of just collecting data and fine-tuning. Based on our experiment, the continued pretraining process of IGOT with LLaMA-7B achieved 11.9\% token saving, 12.2\% training time saving, and 5.8\% maximum GPU VRAM usage saving, combined with the T5 model, we can even reach a 31.5\% of training time saving, making porting general generative AI to specific domains more effective than before. In domain-specific tasks, supervised $IGOT_\tau$ shows great performance on reducing both the convergence radius and convergence point during keep pretraining.
• 摘要：ChatGPT、Claude 等预训练大型语言模型（LLM）在自然语言生成的各个领域都展现了强大的能力。然而，在专门领域特定领域使用LLM时仍然存在许多问题。当使用生成式人工智能处理下游任务时，常见的方法是通过持续训练或微调向预训练模型添加新知识（例如私有领域知识、前沿信息）。然而，领域适应训练是否存在通用范式仍然是一个悬而未决的问题。在本文中，我们提出了信息增益优化分词器（IGOT），它分析下游任务的特殊标记集，使用启发式函数 $\phi$ 和特殊标记及其信息增益构造一个新的子集，以构建新的特定领域tokenizer，并继续对下游任务数据进行预训练。我们探索了该方法的定制分词器对领域自适应预训练的许多积极影响，并验证了该方法可以比仅收集数据和微调的普通方法表现得更好。根据我们的实验，IGOT 使用 LLaMA-7B 的持续预训练过程实现了 11.9% 的 token 节省、12.2% 的训练时间节省和 5.8% 的最大 GPU VRAM 使用节省，结合 T5 模型，我们甚至可以达到节省 31.5% 的训练时间，使得将通用生成式 AI 移植到特定领域比以前更有效。在特定领域的任务中，有监督的 $IGOT_\tau$ 在保持预训练期间在减少收敛半径和收敛点方面表现出了出色的性能。

Title: TransMI: A Framework to Create Strong Baselines from Multilingual Pretrained Language Models for Transliterated Data

• Authors: Yihong Liu, Chunlan Ma, Haotian Ye, Hinrich Schütze
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.09913
• Pdf URL: https://arxiv.org/pdf/2405.09913
• Copy Paste: [[2405.09913]] TransMI: A Framework to Create Strong Baselines from Multilingual Pretrained Language Models for Transliterated Data(https://arxiv.org/abs/2405.09913)
• Keywords: language model
• Abstract: Transliterating related languages that use different scripts into a common script shows effectiveness in improving crosslingual transfer in downstream tasks. However, this methodology often makes pretraining a model from scratch unavoidable, as transliteration brings about new subwords not covered in existing multilingual pretrained language models (mPLMs). This is not desired because it takes a lot of computation budget for pretraining. A more promising way is to make full use of available mPLMs. To this end, this paper proposes a simple but effective framework: Transliterate-Merge-Initialize (TransMI), which can create a strong baseline well-suited for data that is transliterated into a common script by exploiting an mPLM and its accompanied tokenizer. TransMI has three stages: (a) transliterate the vocabulary of an mPLM into a common script; (b) merge the new vocabulary with the original vocabulary; and (c) initialize the embeddings of the new subwords. We applied TransMI to three recent strong mPLMs, and our experiments demonstrate that TransMI not only preserves their ability to handle non-transliterated data, but also enables the models to effectively process transliterated data: the results show a consistent improvement of 3% to 34%, varying across different models and tasks. We make our code and models publicly available at \url{this https URL}.
• 摘要：将使用不同脚本的相关语言音译为通用脚本可以有效改善下游任务中的跨语言迁移。然而，这种方法通常不可避免地需要从头开始预训练模型，因为音译会带来现有多语言预训练语言模型 (mPLM) 中未涵盖的新子词。这是不希望的，因为预训练需要大量的计算预算。一种更有前景的方法是充分利用可用的 mPLM。为此，本文提出了一个简单但有效的框架：音译-合并-初始化（TransMI），它可以创建一个强大的基线，非常适合通过利用 mPLM 及其附带的分词器将数据音译为通用脚本。 TransMI 分为三个阶段： (a) 将 mPLM 的词汇音译为通用脚本； (b) 将新词汇与原始词汇合并； (c) 初始化新子词的嵌入。我们将 TransMI 应用于最近三个强大的 mPLM，我们的实验表明 TransMI 不仅保留了它们处理非音译数据的能力，而且还使模型能够有效地处理音译数据：结果显示持续提高了 3% 到 34% ，因不同模型和任务而异。我们在 \url{this https URL} 公开提供我们的代码和模型。

Title: DEBATE: Devil's Advocate-Based Assessment and Text Evaluation

• Authors: Alex Kim, Keonwoo Kim, Sangwon Yoon
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.09935
• Pdf URL: https://arxiv.org/pdf/2405.09935
• Copy Paste: [[2405.09935]] DEBATE: Devil's Advocate-Based Assessment and Text Evaluation(https://arxiv.org/abs/2405.09935)
• Keywords: llm, chat, agent
• Abstract: As natural language generation (NLG) models have become prevalent, systematically assessing the quality of machine-generated texts has become increasingly important. Recent studies introduce LLM-based evaluators that operate as reference-free metrics, demonstrating their capability to adeptly handle novel tasks. However, these models generally rely on a single-agent approach, which, we argue, introduces an inherent limit to their performance. This is because there exist biases in LLM agent's responses, including preferences for certain text structure or content. In this work, we propose DEBATE, an NLG evaluation framework based on multi-agent scoring system augmented with a concept of Devil's Advocate. Within the framework, one agent is instructed to criticize other agents' arguments, potentially resolving the bias in LLM agent's answers. DEBATE substantially outperforms the previous state-of-the-art methods in two meta-evaluation benchmarks in NLG evaluation, SummEval and TopicalChat. We also show that the extensiveness of debates among agents and the persona of an agent can influence the performance of evaluators.
• 摘要：随着自然语言生成（NLG）模型的流行，系统地评估机器生成文本的质量变得越来越重要。最近的研究引入了基于 LLM 的评估器，这些评估器作为无参考指标运行，展示了它们熟练处理新任务的能力。然而，这些模型通常依赖于单代理方法，我们认为这对其性能带来了固有的限制。这是因为LLM代理人的反应存在偏见，包括对某些文本结构或内容的偏好。在这项工作中，我们提出了 DEBATE，一种基于多智能体评分系统的 NLG 评估框架，并增强了 Devil's Advocate 的概念。在该框架内，指示一名代理人批评其他代理人的论点，这可能会解决法学硕士代理人答案中的偏见。 DEBATE 在 NLG 评估的两个元评估基准（SummEval 和 TopicalChat）中显着优于之前最先进的方法。我们还表明，代理人之间辩论的广泛性和代理人的角色会影响评估者的表现。

Title: SciQAG: A Framework for Auto-Generated Scientific Question Answering Dataset with Fine-grained Evaluation

• Authors: Yuwei Wan, Aswathy Ajith, Yixuan Liu, Ke Lu, Clara Grazian, Bram Hoex, Wenjie Zhang, Chunyu Kit, Tong Xie, Ian Foster
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] SciQAG: A Framework for Auto-Generated Scientific Question Answering Dataset with Fine-grained Evaluation(https://arxiv.org/abs/)
• Keywords: language model, llm
• Abstract: The use of question-answer (QA) pairs for training and evaluating large language models (LLMs) has attracted considerable attention. Yet few available QA datasets are based on knowledge from the scientific literature. Here we bridge this gap by presenting Automatic Generation of Scientific Question Answers (SciQAG), a framework for automatic generation and evaluation of scientific QA pairs sourced from published scientific literature. We fine-tune an open-source LLM to generate \num{960000} scientific QA pairs from full-text scientific papers and propose a five-dimensional metric to evaluate the quality of the generated QA pairs. We show via LLM-based evaluation that the generated QA pairs consistently achieve an average score of 2.5 out of 3 across five dimensions, indicating that our framework can distill key knowledge from papers into high-quality QA pairs at scale. We make the dataset, models, and evaluation codes publicly available.
• 摘要：使用问答（QA）对来训练和评估大型语言模型（LLM）引起了相当大的关注。然而，很少有可用的 QA 数据集是基于科学文献中的知识。在这里，我们通过提出自动生成科学问题答案（SciQAG）来弥补这一差距，这是一个自动生成和评估来自已发表的科学文献的科学问答对的框架。我们对开源 LLM 进行微调，从全文科学论文中生成 \num{960000} 个科学 QA 对，并提出一个五维指标来评估生成的 QA 对的质量。我们通过基于 LLM 的评估表明，生成的 QA 对在五个维度上始终达到 2.5 分（满分 3 分）的平均分，这表明我们的框架可以大规模地将论文中的关键知识提炼成高质量的 QA 对。我们公开数据集、模型和评估代码。

Title: FinTextQA: A Dataset for Long-form Financial Question Answering

• Authors: Jian Chen, Peilin Zhou, Yining Hua, Yingxin Loh, Kehui Chen, Ziyuan Li, Bing Zhu, Junwei Liang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.09980
• Pdf URL: https://arxiv.org/pdf/2405.09980
• Copy Paste: [[2405.09980]] FinTextQA: A Dataset for Long-form Financial Question Answering(https://arxiv.org/abs/2405.09980)
• Keywords: gpt, retrieval-augmented generation
• Abstract: Accurate evaluation of financial question answering (QA) systems necessitates a comprehensive dataset encompassing diverse question types and contexts. However, current financial QA datasets lack scope diversity and question complexity. This work introduces FinTextQA, a novel dataset for long-form question answering (LFQA) in finance. FinTextQA comprises 1,262 high-quality, source-attributed QA pairs extracted and selected from finance textbooks and government agency websites.Moreover, we developed a Retrieval-Augmented Generation (RAG)-based LFQA system, comprising an embedder, retriever, reranker, and generator. A multi-faceted evaluation approach, including human ranking, automatic metrics, and GPT-4 scoring, was employed to benchmark the performance of different LFQA system configurations under heightened noisy conditions. The results indicate that: (1) Among all compared generators, Baichuan2-7B competes closely with GPT-3.5-turbo in accuracy score; (2) The most effective system configuration on our dataset involved setting the embedder, retriever, reranker, and generator as Ada2, Automated Merged Retrieval, Bge-Reranker-Base, and Baichuan2-7B, respectively; (3) models are less susceptible to noise after the length of contexts reaching a specific threshold.
• 摘要：准确评估金融问答 (QA) 系统需要包含不同问题类型和上下文的综合数据集。然而，当前的财务 QA 数据集缺乏范围多样性和问题复杂性。这项工作介绍了 FinTextQA，这是一个用于金融领域长格式问答 (LFQA) 的新颖数据集。 FinTextQA 包含从金融教科书和政府机构网站中提取和选择的 1,262 个高质量、来源归因的 QA 对。此外，我们开发了基于检索增强生成 (RAG) 的 LFQA 系统，包括嵌入器、检索器、重排序器和生成器。采用多方面的评估方法，包括人工排名、自动指标和 GPT-4 评分，对不同 LFQA 系统配置在高噪声条件下的性能进行基准测试。结果表明：（1）在所有比较的生成器中，百川2-7B在准确度得分上与GPT-3.5-turbo竞争激烈； (2) 我们数据集上最有效的系统配置包括将嵌入器、检索器、重排序器和生成器分别设置为 Ada2、自动合并检索、Bge-Reranker-Base 和 Baichuan2-7B； (3)上下文长度达到特定阈值后，模型不易受噪声影响。

Title: Listen Again and Choose the Right Answer: A New Paradigm for Automatic Speech Recognition with Large Language Models

• Authors: Yuchen Hu, Chen Chen, Chengwei Qin, Qiushi Zhu, Eng Siong Chng, Ruizhe Li
• Subjects: cs.CL, cs.AI, cs.LG, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2405.10025
• Pdf URL: https://arxiv.org/pdf/2405.10025
• Copy Paste: [[2405.10025]] Listen Again and Choose the Right Answer: A New Paradigm for Automatic Speech Recognition with Large Language Models(https://arxiv.org/abs/2405.10025)
• Keywords: language model, gpt, llm
• Abstract: Recent advances in large language models (LLMs) have promoted generative error correction (GER) for automatic speech recognition (ASR), which aims to predict the ground-truth transcription from the decoded N-best hypotheses. Thanks to the strong language generation ability of LLMs and rich information in the N-best list, GER shows great effectiveness in enhancing ASR results. However, it still suffers from two limitations: 1) LLMs are unaware of the source speech during GER, which may lead to results that are grammatically correct but violate the source speech content, 2) N-best hypotheses usually only vary in a few tokens, making it redundant to send all of them for GER, which could confuse LLM about which tokens to focus on and thus lead to increased miscorrection. In this paper, we propose ClozeGER, a new paradigm for ASR generative error correction. First, we introduce a multimodal LLM (i.e., SpeechGPT) to receive source speech as extra input to improve the fidelity of correction output. Then, we reformat GER as a cloze test with logits calibration to remove the input information redundancy and simplify GER with clear instructions. Experiments show that ClozeGER achieves a new breakthrough over vanilla GER on 9 popular ASR datasets.
• 摘要：大语言模型 (LLM) 的最新进展促进了自动语音识别 (ASR) 的生成纠错 (GER)，其目的是根据解码的 N 最佳假设来预测真实转录。得益于LLM强大的语言生成能力和N-best列表中丰富的信息，GER在提升ASR结果方面显示出了巨大的有效性。然而，它仍然面临两个限制：1）LLM 在 GER 期间不知道源语音，这可能会导致语法正确但违反源语音内容的结果，2）N 最佳假设通常仅在少数标记上有所不同，使得将所有这些都发送为 GER 变得多余，这可能会使 LLM 混淆要关注哪些令牌，从而导致错误更正的增加。在本文中，我们提出了 ClozeGER，一种 ASR 生成错误纠正的新范式。首先，我们引入多模态 LLM（即 SpeechGPT）来接收源语音作为额外输入，以提高校正输出的保真度。然后，我们将 GER 重新格式化为带有 logits 校准的完形填空测试，以消除输入信息冗余并通过清晰的指令简化 GER。实验表明，ClozeGER 在 9 个流行的 ASR 数据集上取得了相对普通 GER 的新突破。

Title: SynthesizRR: Generating Diverse Datasets with Retrieval Augmentation

• Authors: Abhishek Divekar, Greg Durrett
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.10040
• Pdf URL: https://arxiv.org/pdf/2405.10040
• Copy Paste: [[2405.10040]] SynthesizRR: Generating Diverse Datasets with Retrieval Augmentation(https://arxiv.org/abs/2405.10040)
• Keywords: language model, llm, prompt
• Abstract: Large language models (LLMs) are versatile and can address many tasks, but for computational efficiency, it is often desirable to distill their capabilities into smaller student models. One way to do this for classification tasks is via dataset synthesis, which can be accomplished by generating examples of each label from the LLM. Prior approaches to synthesis use few-shot prompting, which relies on the LLM's parametric knowledge to generate usable examples. However, this leads to issues of repetition, bias towards popular entities, and stylistic differences from human text. In this work, we propose Synthesize by Retrieval and Refinement (SynthesizRR), which uses retrieval augmentation to introduce variety into the dataset synthesis process: as retrieved passages vary, the LLM is "seeded" with different content to generate its examples. We empirically study the synthesis of six datasets, covering topic classification, sentiment analysis, tone detection, and humor, requiring complex synthesis strategies. We find SynthesizRR greatly improves lexical and semantic diversity, similarity to human-written text, and distillation performance, when compared to standard 32-shot prompting and six baseline approaches.
• 摘要：大型语言模型 (LLM) 用途广泛，可以解决许多任务，但为了计算效率，通常需要将其功能提炼成更小的学生模型。对于分类任务执行此操作的一种方法是通过数据集合成，这可以通过从法学硕士生成每个标签的示例来完成。先前的综合方法使用少样本提示，这依赖于法学硕士的参数知识来生成可用的示例。然而，这会导致重复、对流行实体的偏见以及与人类文本的风格差异等问题。在这项工作中，我们提出通过检索和细化进行综合（SynthesizRR），它使用检索增强将多样性引入数据集合成过程：随着检索到的段落变化，LLM 会被“播种”不同的内容以生成其示例。我们实证研究了六个数据集的合成，涵盖主题分类、情感分析、语气检测和幽默，需要复杂的合成策略。我们发现，与标准 32 镜头提示和六种基线方法相比，SynthesizRR 极大地提高了词汇和语义多样性、与人类书写文本的相似性以及蒸馏性能。

Title: Distilling Implicit Multimodal Knowledge into LLMs for Zero-Resource Dialogue Generation

• Authors: Bo Zhang, Hui Ma, Jian Ding, Jian Wang, Bo Xu, Hongfei Lin
• Subjects: cs.CL, cs.MM
• Abstract URL: https://arxiv.org/abs/2405.10121
• Pdf URL: https://arxiv.org/pdf/2405.10121
• Copy Paste: [[2405.10121]] Distilling Implicit Multimodal Knowledge into LLMs for Zero-Resource Dialogue Generation(https://arxiv.org/abs/2405.10121)
• Keywords: language model, llm
• Abstract: Integrating multimodal knowledge into large language models (LLMs) represents a significant advancement in dialogue generation capabilities. However, the effective incorporation of such knowledge in zero-resource scenarios remains a substantial challenge due to the scarcity of diverse, high-quality dialogue datasets. To address this, we propose the Visual Implicit Knowledge Distillation Framework (VIKDF), an innovative approach aimed at enhancing LLMs for enriched dialogue generation in zero-resource contexts by leveraging implicit multimodal knowledge. VIKDF comprises two main stages: knowledge distillation, using an Implicit Query Transformer to extract and encode visual implicit knowledge from image-text pairs into knowledge vectors; and knowledge integration, employing a novel Bidirectional Variational Information Fusion technique to seamlessly integrate these distilled vectors into LLMs. This enables the LLMs to generate dialogues that are not only coherent and engaging but also exhibit a deep understanding of the context through implicit multimodal cues, effectively overcoming the limitations of zero-resource scenarios. Our extensive experimentation across two dialogue datasets shows that VIKDF outperforms existing state-of-the-art models in generating high-quality dialogues. The code will be publicly available following acceptance.
• 摘要：将多模态知识集成到大型语言模型（LLM）中代表了对话生成能力的重大进步。然而，由于缺乏多样化、高质量的对话数据集，在零资源场景中有效整合此类知识仍然是一个巨大的挑战。为了解决这个问题，我们提出了视觉隐式知识蒸馏框架（VIKDF），这是一种创新方法，旨在通过利用隐式多模态知识来增强法学硕士，以在零资源环境中丰富对话生成。 VIKDF 包括两个主要阶段：知识蒸馏，使用隐式查询转换器从图像文本对中提取视觉隐式知识并将其编码为知识向量；和知识集成，采用新颖的双向变分信息融合技术将这些蒸馏向量无缝集成到法学硕士中。这使得法学硕士能够生成不仅连贯且引人入胜的对话，而且还通过隐含的多模态线索展现出对上下文的深刻理解，有效克服了零资源场景的局限性。我们对两个对话数据集进行的广泛实验表明，VIKDF 在生成高质量对话方面优于现有的最先进模型。该代码将在接受后公开发布。

Title: Red Teaming Language Models for Contradictory Dialogues

• Authors: Xiaofei Wen, Bangzheng Li, Tenghao Huang, Muhao Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.10128
• Pdf URL: https://arxiv.org/pdf/2405.10128
• Copy Paste: [[2405.10128]] Red Teaming Language Models for Contradictory Dialogues(https://arxiv.org/abs/2405.10128)
• Keywords: language model
• Abstract: Most language models currently available are prone to self-contradiction during dialogues. To mitigate this issue, this study explores a novel contradictory dialogue processing task that aims to detect and modify contradictory statements in a conversation. This task is inspired by research on context faithfulness and dialogue comprehension, which have demonstrated that the detection and understanding of contradictions often necessitate detailed explanations. We develop a dataset comprising contradictory dialogues, in which one side of the conversation contradicts itself. Each dialogue is accompanied by an explanatory label that highlights the location and details of the contradiction. With this dataset, we present a Red Teaming framework for contradictory dialogue processing. The framework detects and attempts to explain the dialogue, then modifies the existing contradictory content using the explanation. Our experiments demonstrate that the framework improves the ability to detect contradictory dialogues and provides valid explanations. Additionally, it showcases distinct capabilities for modifying such dialogues. Our study highlights the importance of the logical inconsistency problem in conversational AI.
• 摘要：目前可用的大多数语言模型在对话过程中容易出现自相矛盾。为了缓解这个问题，本研究探索了一种新颖的矛盾对话处理任务，旨在检测和修改对话中的矛盾陈述。这项任务的灵感来自对上下文忠实性和对话理解的研究，这些研究表明，检测和理解矛盾通常需要详细的解释。我们开发了一个包含矛盾对话的数据集，其中对话的一方自相矛盾。每个对话都附有一个解释标签，突出显示矛盾的位置和细节。利用这个数据集，我们提出了一个用于矛盾对话处理的红队框架。该框架检测并尝试解释对话，然后使用解释修改现有的矛盾内容。我们的实验表明，该框架提高了检测矛盾对话的能力并提供有效的解释。此外，它展示了修改此类对话的独特能力。我们的研究强调了逻辑不一致问题在对话式人工智能中的重要性。

Title: StyloAI: Distinguishing AI-Generated Content with Stylometric Analysis

• Authors: Chidimma Opara
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.10129
• Pdf URL: https://arxiv.org/pdf/2405.10129
• Copy Paste: [[2405.10129]] StyloAI: Distinguishing AI-Generated Content with Stylometric Analysis(https://arxiv.org/abs/2405.10129)
• Keywords: language model, llm
• Abstract: The emergence of large language models (LLMs) capable of generating realistic texts and images has sparked ethical concerns across various sectors. In response, researchers in academia and industry are actively exploring methods to distinguish AI-generated content from human-authored material. However, a crucial question remains: What are the unique characteristics of AI-generated text? Addressing this gap, this study proposes StyloAI, a data-driven model that uses 31 stylometric features to identify AI-generated texts by applying a Random Forest classifier on two multi-domain datasets. StyloAI achieves accuracy rates of 81% and 98% on the test set of the AuTextification dataset and the Education dataset, respectively. This approach surpasses the performance of existing state-of-the-art models and provides valuable insights into the differences between AI-generated and human-authored texts.
• 摘要：能够生成真实文本和图像的大型语言模型（LLM）的出现引发了各个领域的道德担忧。为此，学术界和工业界的研究人员正在积极探索区分人工智能生成的内容和人类创作的内容的方法。然而，一个关键问题仍然存在：人工智能生成的文本有哪些独特特征？为了解决这一差距，本研究提出了 StyloAI，这是一种数据驱动模型，通过在两个多域数据集上应用随机森林分类器，使用 31 个文体特征来识别 AI 生成的文本。 StyloAI 在 AuTextification 数据集和 Education 数据集的测试集上分别实现了 81% 和 98% 的准确率。这种方法超越了现有最先进模型的性能，并为人工智能生成的文本和人类创作的文本之间的差异提供了有价值的见解。

Title: Speaker Verification in Agent-Generated Conversations

• Authors: Yizhe Yang, Heyan Huang, Palakorn Achananuparp, Jing Jiang, Ee-Peng Lim
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.10150
• Pdf URL: https://arxiv.org/pdf/2405.10150
• Copy Paste: [[2405.10150]] Speaker Verification in Agent-Generated Conversations(https://arxiv.org/abs/2405.10150)
• Keywords: language model, llm, agent
• Abstract: The recent success of large language models (LLMs) has attracted widespread interest to develop role-playing conversational agents personalized to the characteristics and styles of different speakers to enhance their abilities to perform both general and special purpose dialogue tasks. However, the ability to personalize the generated utterances to speakers, whether conducted by human or LLM, has not been well studied. To bridge this gap, our study introduces a novel evaluation challenge: speaker verification in agent-generated conversations, which aimed to verify whether two sets of utterances originate from the same speaker. To this end, we assemble a large dataset collection encompassing thousands of speakers and their utterances. We also develop and evaluate speaker verification models under experiment setups. We further utilize the speaker verification models to evaluate the personalization abilities of LLM-based role-playing models. Comprehensive experiments suggest that the current role-playing models fail in accurately mimicking speakers, primarily due to their inherent linguistic characteristics.
• 摘要：大语言模型（LLM）最近的成功引起了人们的广泛兴趣，开发针对不同说话者的特征和风格的个性化角色扮演对话代理，以增强他们执行通用和特殊目的对话任务的能力。然而，对说话者生成的个性化话语的能力，无论是由人类还是法学硕士进行，尚未得到充分研究。为了弥补这一差距，我们的研究引入了一个新的评估挑战：在代理生成的对话中进行说话人验证，其目的是验证两组话语是否来自同一说话人。为此，我们收集了一个包含数千名说话者及其话语的大型数据集。我们还在实验设置下开发和评估说话人验证模型。我们进一步利用说话人验证模型来评估基于 LLM 的角色扮演模型的个性化能力。综合实验表明，当前的角色扮演模型无法准确模仿说话者，这主要是由于其固有的语言特征。

Title: LFED: A Literary Fiction Evaluation Dataset for Large Language Models

• Authors: Linhao Yu, Qun Liu, Deyi Xiong
• Subjects: cs.CL, cs.PF
• Abstract URL: https://arxiv.org/abs/2405.10166
• Pdf URL: https://arxiv.org/pdf/2405.10166
• Copy Paste: [[2405.10166]] LFED: A Literary Fiction Evaluation Dataset for Large Language Models(https://arxiv.org/abs/2405.10166)
• Keywords: language model, gpt, llm, chat
• Abstract: The rapid evolution of large language models (LLMs) has ushered in the need for comprehensive assessments of their performance across various dimensions. In this paper, we propose LFED, a Literary Fiction Evaluation Dataset, which aims to evaluate the capability of LLMs on the long fiction comprehension and reasoning. We collect 95 literary fictions that are either originally written in Chinese or translated into Chinese, covering a wide range of topics across several centuries. We define a question taxonomy with 8 question categories to guide the creation of 1,304 questions. Additionally, we conduct an in-depth analysis to ascertain how specific attributes of literary fictions (e.g., novel types, character numbers, the year of publication) impact LLM performance in evaluations. Through a series of experiments with various state-of-the-art LLMs, we demonstrate that these models face considerable challenges in effectively addressing questions related to literary fictions, with ChatGPT reaching only 57.08% under the zero-shot setting. The dataset will be publicly available at this https URL
• 摘要：大型语言模型（LLM）的快速发展带来了对其在各个维度上的性能进行全面评估的需求。在本文中，我们提出了LFED，一种文学小说评估数据集，旨在评估法学硕士在长篇小说理解和推理方面的能力。我们收集了 95 部文学小说，这些小说要么是中文原创的，要么是中文翻译的，涵盖了几个世纪以来的广泛主题。我们定义了一个包含 8 个问题类别的问题分类法，以指导 1,304 个问题的创建。此外，我们还进行深入分析，以确定文学小说的具体属性（例如小说类型、人物数量、出版年份）如何影响法学硕士在评估中的表现。通过对各种最先进的法学硕士进行的一系列实验，我们证明这些模型在有效解决与文学小说相关的问题方面面临相当大的挑战，ChatGPT 在零样本设置下仅达到 57.08%。该数据集将在此 https URL 公开提供

Title: CPsyExam: A Chinese Benchmark for Evaluating Psychology using Examinations

• Authors: Jiahao Zhao, Jingwei Zhu, Minghuan Tan, Min Yang, Di Yang, Chenhao Zhang, Guancheng Ye, Chengming Li, Xiping Hu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.10212
• Pdf URL: https://arxiv.org/pdf/2405.10212
• Copy Paste: [[2405.10212]] CPsyExam: A Chinese Benchmark for Evaluating Psychology using Examinations(https://arxiv.org/abs/2405.10212)
• Keywords: language model, llm
• Abstract: In this paper, we introduce a novel psychological benchmark, CPsyExam, constructed from questions sourced from Chinese language examinations. CPsyExam is designed to prioritize psychological knowledge and case analysis separately, recognizing the significance of applying psychological knowledge to real-world scenarios. From the pool of 22k questions, we utilize 4k to create the benchmark that offers balanced coverage of subjects and incorporates a diverse range of case analysis techniques.Furthermore, we evaluate a range of existing large language models~(LLMs), spanning from open-sourced to API-based models. Our experiments and analysis demonstrate that CPsyExam serves as an effective benchmark for enhancing the understanding of psychology within LLMs and enables the comparison of LLMs across various granularities.
• 摘要：在本文中，我们介绍了一种新颖的心理基准——CPsyExam，它是根据汉语考试的问题构建的。 CPsyExam 旨在分别优先考虑心理学知识和案例分析，认识到将心理学知识应用于现实场景的重要性。从 22k 个问题池中，我们利用 4k 来创建基准，提供均衡的主题覆盖并结合各种案例分析技术。此外，我们评估了一系列现有的大型语言模型〜（法学硕士），涵盖从开放式到源自基于 API 的模型。我们的实验和分析表明，CPsyExam 可以作为增强法学硕士对心理学的理解的有效基准，并能够对不同粒度的法学硕士进行比较。

Title: A Systematic Evaluation of Large Language Models for Natural Language Generation Tasks

• Authors: Xuanfan Ni, Piji Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.10251
• Pdf URL: https://arxiv.org/pdf/2405.10251
• Copy Paste: [[2405.10251]] A Systematic Evaluation of Large Language Models for Natural Language Generation Tasks(https://arxiv.org/abs/2405.10251)
• Keywords: language model, gpt, llm, chat
• Abstract: Recent efforts have evaluated large language models (LLMs) in areas such as commonsense reasoning, mathematical reasoning, and code generation. However, to the best of our knowledge, no work has specifically investigated the performance of LLMs in natural language generation (NLG) tasks, a pivotal criterion for determining model excellence. Thus, this paper conducts a comprehensive evaluation of well-known and high-performing LLMs, namely ChatGPT, ChatGLM, T5-based models, LLaMA-based models, and Pythia-based models, in the context of NLG tasks. We select English and Chinese datasets encompassing Dialogue Generation and Text Summarization. Moreover, we propose a common evaluation setting that incorporates input templates and post-processing strategies. Our study reports both automatic results, accompanied by a detailed analysis.
• 摘要：最近的工作评估了常识推理、数学推理和代码生成等领域的大型语言模型 (LLM)。然而，据我们所知，还没有专门研究法学硕士在自然语言生成（NLG）任务中的表现的研究，而自然语言生成（NLG）任务是确定模型卓越性的关键标准。因此，本文在NLG任务的背景下对著名的高性能LLM，即ChatGPT、ChatGLM、基于T5的模型、基于LLaMA的模型和基于Pythia的模型进行了综合评估。我们选择包含对话生成和文本摘要的英文和中文数据集。此外，我们提出了一个通用的评估设置，其中包含输入模板和后处理策略。我们的研究报告了自动结果，并附有详细的分析。

Title: Keep It Private: Unsupervised Privatization of Online Text

• Authors: Calvin Bao, Marine Carpuat
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.10260
• Pdf URL: https://arxiv.org/pdf/2405.10260
• Copy Paste: [[2405.10260]] Keep It Private: Unsupervised Privatization of Online Text(https://arxiv.org/abs/2405.10260)
• Keywords: language model
• Abstract: Authorship obfuscation techniques hold the promise of helping people protect their privacy in online communications by automatically rewriting text to hide the identity of the original author. However, obfuscation has been evaluated in narrow settings in the NLP literature and has primarily been addressed with superficial edit operations that can lead to unnatural outputs. In this work, we introduce an automatic text privatization framework that fine-tunes a large language model via reinforcement learning to produce rewrites that balance soundness, sense, and privacy. We evaluate it extensively on a large-scale test set of English Reddit posts by 68k authors composed of short-medium length texts. We study how the performance changes among evaluative conditions including authorial profile length and authorship detection strategy. Our method maintains high text quality according to both automated metrics and human evaluation, and successfully evades several automated authorship attacks.
• 摘要：作者身份混淆技术有望通过自动重写文本以隐藏原作者的身份来帮助人们保护在线通信中的隐私。然而，混淆已经在 NLP 文献的狭窄环境中进行了评估，并且主要通过可能导致不自然输出的肤浅编辑操作来解决。在这项工作中，我们引入了一种自动文本私有化框架，该框架通过强化学习对大型语言模型进行微调，以生成平衡健全性、意义和隐私的重写。我们在由 68,000 位作者组成的由中短文本组成的大规模英语 Reddit 帖子测试集上对其进行了广泛的评估。我们研究了评估条件（包括作者简介长度和作者检测策略）之间的性能变化。我们的方法根据自动指标和人工评估保持了较高的文本质量，并成功规避了多次自动作者攻击。

Title: Revisiting OPRO: The Limitations of Small-Scale LLMs as Optimizers

• Authors: Tuo Zhang, Jinyue Yuan, Salman Avestimehr
• Subjects: cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2405.10276
• Pdf URL: https://arxiv.org/pdf/2405.10276
• Copy Paste: [[2405.10276]] Revisiting OPRO: The Limitations of Small-Scale LLMs as Optimizers(https://arxiv.org/abs/2405.10276)
• Keywords: language model, llm, prompt
• Abstract: Numerous recent works aim to enhance the efficacy of Large Language Models (LLMs) through strategic prompting. In particular, the Optimization by PROmpting (OPRO) approach provides state-of-the-art performance by leveraging LLMs as optimizers where the optimization task is to find instructions that maximize the task accuracy. In this paper, we revisit OPRO for automated prompting with relatively small-scale LLMs, such as LLaMa-2 family and Mistral 7B. Our investigation reveals that OPRO shows limited effectiveness in small-scale LLMs, with limited inference capabilities constraining optimization ability. We suggest future automatic prompting engineering to consider both model capabilities and computational costs. Additionally, for small-scale LLMs, we recommend direct instructions that clearly outline objectives and methodologies as robust prompt baselines, ensuring efficient and effective prompt engineering in ongoing research.
• 摘要：最近的许多工作旨在通过战略提示来提高大型语言模型（LLM）的效率。特别是，通过 PROmpting 优化 (OPRO) 方法通过利用 LLM 作为优化器来提供最先进的性能，其中优化任务是找到最大化任务准确性的指令。在本文中，我们重新审视 OPRO，以对规模相对较小的 LLM（例如 LLaMa-2 系列和 Mistral 7B）进行自动提示。我们的调查表明，OPRO 在小型法学硕士中表现出有限的有效性，有限的推理能力限制了优化能力。我们建议未来的自动提示工程要考虑模型能力和计算成本。此外，对于小型法学硕士，我们建议直接说明，明确概述目标和方法作为强大的即时基线，确保正在进行的研究中高效且有效的即时工程。

Title: Timeline-based Sentence Decomposition with In-Context Learning for Temporal Fact Extraction

• Authors: Jianhao Chen, Haoyuan Ouyang, Junyang Ren, Wentao Ding, Wei Hu, Yuzhong Qu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.10288
• Pdf URL: https://arxiv.org/pdf/2405.10288
• Copy Paste: [[2405.10288]] Timeline-based Sentence Decomposition with In-Context Learning for Temporal Fact Extraction(https://arxiv.org/abs/2405.10288)
• Keywords: language model, llm
• Abstract: Facts extraction is pivotal for constructing knowledge graphs. Recently, the increasing demand for temporal facts in downstream tasks has led to the emergence of the task of temporal fact extraction. In this paper, we specifically address the extraction of temporal facts from natural language text. Previous studies fail to handle the challenge of establishing time-to-fact correspondences in complex sentences. To overcome this hurdle, we propose a timeline-based sentence decomposition strategy using large language models (LLMs) with in-context learning, ensuring a fine-grained understanding of the timeline associated with various facts. In addition, we evaluate the performance of LLMs for direct temporal fact extraction and get unsatisfactory results. To this end, we introduce TSDRE, a method that incorporates the decomposition capabilities of LLMs into the traditional fine-tuning of smaller pre-trained language models (PLMs). To support the evaluation, we construct ComplexTRED, a complex temporal fact extraction dataset. Our experiments show that TSDRE achieves state-of-the-art results on both HyperRED-Temporal and ComplexTRED datasets.
• 摘要：事实提取对于构建知识图至关重要。最近，下游任务对时间事实的需求不断增加，导致了时间事实提取任务的出现。在本文中，我们专门解决从自然语言文本中提取时间事实的问题。先前的研究未能应对在复杂句子中建立时间与事实对应关系的挑战。为了克服这一障碍，我们提出了一种基于时间线的句子分解策略，使用大型语言模型（LLM）和上下文学习，确保对与各种事实相关的时间线的细粒度理解。此外，我们评估了法学硕士在直接时间事实提取方面的性能，但得到的结果并不令人满意。为此，我们引入了 TSDRE，这是一种将 LLM 的分解功能融入到较小的预训练语言模型 (PLM) 的传统微调中的方法。为了支持评估，我们构建了 ComplexTRED，一个复杂的时间事实提取数据集。我们的实验表明，TSDRE 在 HyperRED-Temporal 和 ComplexTRED 数据集上都取得了最先进的结果。