2024-04-22

Title: AmbigDocs: Reasoning across Documents on Different Entities under the Same Name

• Authors: Yoonsang Lee, Xi Ye, Eunsol Choi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.12447
• Pdf URL: https://arxiv.org/pdf/2404.12447
• Copy Paste: [[2404.12447]] AmbigDocs: Reasoning across Documents on Different Entities under the Same Name(https://arxiv.org/abs/2404.12447)
• Keywords: language model
• Abstract: Different entities with the same name can be difficult to distinguish. Handling confusing entity mentions is a crucial skill for language models (LMs). For example, given the question "Where was Michael Jordan educated?" and a set of documents discussing different people named Michael Jordan, can LMs distinguish entity mentions to generate a cohesive answer to the question? To test this ability, we introduce a new benchmark, AmbigDocs. By leveraging Wikipedia's disambiguation pages, we identify a set of documents, belonging to different entities who share an ambiguous name. From these documents, we generate questions containing an ambiguous name and their corresponding sets of answers. Our analysis reveals that current state-of-the-art models often yield ambiguous answers or incorrectly merge information belonging to different entities. We establish an ontology categorizing four types of incomplete answers and automatic evaluation metrics to identify such categories. We lay the foundation for future work on reasoning across multiple documents with ambiguous entities.
• 摘要：具有相同名称的不同实体可能难以区分。处理令人困惑的实体提及是语言模型 (LM) 的一项关键技能。例如，考虑到“迈克尔·乔丹在哪里接受的教育？”这个问题。以及一组讨论名为 Michael Jordan 的不同人的文档，LM 能否区分实体提及以生成问题的一致答案？为了测试这种能力，我们引入了一个新的基准测试 AmbigDocs。通过利用维基百科的消歧页面，我们识别了一组文档，这些文档属于共享一个模糊名称的不同实体。从这些文档中，我们生成包含不明确名称的问题及其相应的答案集。我们的分析表明，当前最先进的模型经常会产生模糊的答案或错误地合并属于不同实体的信息。我们建立了一个本体论，对四种不完整答案进行分类，并建立自动评估指标来识别此类类别。我们为未来跨多个具有不明确实体的文档进行推理的工作奠定了基础。

Title: Characterizing LLM Abstention Behavior in Science QA with Context Perturbations

• Authors: Bingbing Wen, Bill Howe, Lucy Lu Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.12452
• Pdf URL: https://arxiv.org/pdf/2404.12452
• Copy Paste: [[2404.12452]] Characterizing LLM Abstention Behavior in Science QA with Context Perturbations(https://arxiv.org/abs/2404.12452)
• Keywords: llm, prompt
• Abstract: The correct model response in the face of uncertainty is to abstain from answering a question so as not to mislead the user. In this work, we study the ability of LLMs to abstain from answering context-dependent science questions when provided insufficient or incorrect context. We probe model sensitivity in several settings: removing gold context, replacing gold context with irrelevant context, and providing additional context beyond what is given. In experiments on four QA datasets with four LLMs, we show that performance varies greatly across models, across the type of context provided, and also by question type; in particular, many LLMs seem unable to abstain from answering boolean questions using standard QA prompts. Our analysis also highlights the unexpected impact of abstention performance on QA task accuracy. Counter-intuitively, in some settings, replacing gold context with irrelevant context or adding irrelevant context to gold context can improve abstention performance in a way that results in improvements in task performance. Our results imply that changes are needed in QA dataset design and evaluation to more effectively assess the correctness and downstream impacts of model abstention.
• 摘要：面对不确定性，正确的模型反应是不回答问题，以免误导用户。在这项工作中，我们研究了法学硕士在提供不充分或不正确的背景时避免回答与背景相关的科学问题的能力。我们在多种设置中探讨模型的敏感性：删除黄金上下文，用不相关的上下文替换黄金上下文，以及提供超出给定内容的附加上下文。在四个法学硕士的四个 QA 数据集上进行的实验中，我们表明，不同模型、所提供的上下文类型以及问题类型的性能差异很大；特别是，许多法学硕士似乎无法避免使用标准 QA 提示来回答布尔问题。我们的分析还强调了弃权表现对 QA 任务准确性的意外影响。与直觉相反，在某些设置中，用不相关上下文替换黄金上下文或将不相关上下文添加到黄金上下文可以提高弃权性能，从而提高任务性能。我们的结果表明，需要对 QA 数据集设计和评估进行更改，以更有效地评估模型弃权的正确性和下游影响。

Title: NORMAD: A Benchmark for Measuring the Cultural Adaptability of Large Language Models

• Authors: Abhinav Rao, Akhila Yerukola, Vishwa Shah, Katharina Reinecke, Maarten Sap
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.12464
• Pdf URL: https://arxiv.org/pdf/2404.12464
• Copy Paste: [[2404.12464]] NORMAD: A Benchmark for Measuring the Cultural Adaptability of Large Language Models(https://arxiv.org/abs/2404.12464)
• Keywords: language model, llm
• Abstract: The integration of Large Language Models (LLMs) into various global cultures fundamentally presents a cultural challenge: LLMs must navigate interactions, respect social norms, and avoid transgressing cultural boundaries. However, it is still unclear if LLMs can adapt their outputs to diverse cultural norms. Our study focuses on this aspect. We introduce NormAd, a novel dataset, which includes 2.6k stories that represent social and cultural norms from 75 countries, to assess the ability of LLMs to adapt to different granular levels of socio-cultural contexts such as the country of origin, its associated cultural values, and prevalent social norms. Our study reveals that LLMs struggle with cultural reasoning across all contextual granularities, showing stronger adaptability to English-centric cultures over those from the Global South. Even with explicit social norms, the top-performing model, Mistral-7b-Instruct, achieves only 81.8\% accuracy, lagging behind the 95.6\% achieved by humans. Evaluation on NormAd further reveals that LLMs struggle to adapt to stories involving gift-giving across cultures. Due to inherent agreement or sycophancy biases, LLMs find it considerably easier to assess the social acceptability of stories that adhere to cultural norms than those that deviate from them. Our benchmark measures the cultural adaptability (or lack thereof) of LLMs, emphasizing the potential to make these technologies more equitable and useful for global audiences.
• 摘要：将大型语言模型 (LLM) 融入各种全球文化从根本上提出了文化挑战：LLM 必须引导互动、尊重社会规范并避免跨越文化界限。然而，目前尚不清楚法学硕士是否可以使他们的成果适应不同的文化规范。我们的研究主要集中在这方面。我们引入了 NormAd，这是一个新颖的数据集，其中包括代表来自 75 个国家的社会和文化规范的 2,600 个故事，用于评估法学硕士适应不同粒度级别的社会文化背景的能力，例如原籍国、其相关文化价值观和普遍的社会规范。我们的研究表明，法学硕士在所有背景粒度的文化推理上都遇到了困难，与来自南半球国家的文化相比，他们对以英语为中心的文化表现出了更强的适应能力。即使有明确的社会规范，表现最好的模型 Mistral-7b-Instruct 也只能达到 81.8% 的准确率，落后于人类达到的 95.6% 的准确率。对 NormAd 的评估进一步表明，法学硕士很难适应涉及跨文化送礼的故事。由于固有的共识或阿谀偏见，法学硕士发现评估符合文化规范的故事的社会可接受性比评估那些偏离文化规范的故事要容易得多。我们的基准衡量法学硕士的文化适应性（或缺乏文化适应性），强调使这些技术对全球受众更加公平和有用的潜力。

Title: BIRD: A Trustworthy Bayesian Inference Framework for Large Language Models

• Authors: Yu Feng, Ben Zhou, Weidong Lin, Dan Roth
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.12494
• Pdf URL: https://arxiv.org/pdf/2404.12494
• Copy Paste: [[2404.12494]] BIRD: A Trustworthy Bayesian Inference Framework for Large Language Models(https://arxiv.org/abs/2404.12494)
• Keywords: language model, gpt, llm
• Abstract: Large language models primarily rely on inductive reasoning for decision making. This results in unreliable decisions when applied to real-world tasks that often present incomplete contexts and conditions. Thus, accurate probability estimation and appropriate interpretations are required to enhance decision-making reliability. In this paper, we propose a Bayesian inference framework called BIRD for large language models. BIRD provides controllable and interpretable probability estimation for model decisions, based on abductive factors, LLM entailment, as well as learnable deductive Bayesian modeling. Experiments show that BIRD produces probability estimations that align with human judgments over 65% of the time using open-sourced Llama models, outperforming the state-of-the-art GPT-4 by 35%. We also show that BIRD can be directly used for trustworthy decision making on many real-world applications.
• 摘要：大型语言模型主要依靠归纳推理来进行决策。当应用于通常呈现不完整的上下文和条件的现实世界任务时，这会导致不可靠的决策。因此，需要准确的概率估计和适当的解释来提高决策的可靠性。在本文中，我们提出了一种用于大型语言模型的贝叶斯推理框架，称为 BIRD。 BIRD 基于溯因因素、LLM 蕴涵以及可学习的演绎贝叶斯模型，为模型决策提供可控且可解释的概率估计。实验表明，BIRD 使用开源 Llama 模型生成的概率估计在超过 65% 的情况下与人类判断一致，比最先进的 GPT-4 性能高出 35%。我们还表明，BIRD 可以直接用于在许多实际应用中做出值得信赖的决策。

Title: Dubo-SQL: Diverse Retrieval-Augmented Generation and Fine Tuning for Text-to-SQL

• Authors: Dayton G. Thorpe, Andrew J. Duberstein, Ian A. Kinsey
• Subjects: cs.CL, cs.DB
• Abstract URL: https://arxiv.org/abs/2404.12560
• Pdf URL: https://arxiv.org/pdf/2404.12560
• Copy Paste: [[2404.12560]] Dubo-SQL: Diverse Retrieval-Augmented Generation and Fine Tuning for Text-to-SQL(https://arxiv.org/abs/2404.12560)
• Keywords: language model, gpt, llm, retrieval-augmented generation
• Abstract: The current state-of-the-art (SOTA) for automated text-to-SQL still falls well short of expert human performance as measured by execution accuracy (EX) on the BIRD-SQL benchmark. The most accurate methods are also slow and expensive. To advance the SOTA for text-to-SQL while reducing cost and improving speed, we explore the combination of low-cost fine tuning, novel methods for diverse retrieval-augmented generation (RAG) and new input and output formats that help large language models (LLMs) achieve higher EX. We introduce two new methods, Dubo-SQL v1 and v2. Dubo-SQL v1 sets a new record for EX on the holdout test set of BIRD-SQL. Dubo-SQL v2 achieves even higher performance on the BIRD-SQL dev set. Dubo-SQL v1 relies on LLMs from OpenAI, but uses the low-cost GPT-3.5 Turbo while exceeding the performance of the next-best model using OpenAI, which instead uses the more expensive GPT-4. Dubo-SQL v1 exceeds the performance of the next-best model using GPT-3.5 by over 20%. Dubo-SQL v2 uses GPT-4 Turbo and RAG in place of fine tuning to push EX higher.
• 摘要：根据 BIRD-SQL 基准测试的执行准确性 (EX) 衡量，当前自动文本到 SQL 的最先进 (SOTA) 仍远远低于专家的人类表现。最准确的方法也是缓慢且昂贵的。为了推进文本到 SQL 的 SOTA，同时降低成本和提高速度，我们探索了低成本微调、多样化检索增强生成 (RAG) 的新颖方法以及有助于大型语言模型的新输入和输出格式的组合（法学硕士）取得更高的 EX。我们引入了两种新方法：Dubo-SQL v1 和 v2。 Dubo-SQL v1 在 BIRD-SQL 的保留测试集上为 EX 创造了新记录。 Dubo-SQL v2 在 BIRD-SQL 开发集上实现了更高的性能。 Dubo-SQL v1 依赖于 OpenAI 的 LLM，但使用低成本的 GPT-3.5 Turbo，同时超过了使用 OpenAI 的次优模型的性能，后者使用了更昂贵的 GPT-4。 Dubo-SQL v1 的性能超过使用 GPT-3.5 的次优模型 20% 以上。 Dubo-SQL v2 使用 GPT-4 Turbo 和 RAG 代替微调来推动 EX 更高。

Title: iTBLS: A Dataset of Interactive Conversations Over Tabular Information

• Authors: Anirudh Sundar, Christopher Richardson, William Gay, Larry Heck
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.12580
• Pdf URL: https://arxiv.org/pdf/2404.12580
• Copy Paste: [[2404.12580]] iTBLS: A Dataset of Interactive Conversations Over Tabular Information(https://arxiv.org/abs/2404.12580)
• Keywords: prompt
• Abstract: This paper introduces Interactive Tables (iTBLS), a dataset of interactive conversations situated in tables from scientific articles. This dataset is designed to facilitate human-AI collaborative problem-solving through AI-powered multi-task tabular capabilities. In contrast to prior work that models interactions as factoid QA or procedure synthesis, iTBLS broadens the scope of interactions to include mathematical reasoning, natural language manipulation, and expansion of existing tables from natural language conversation by delineating interactions into one of three tasks: interpretation, modification, or generation. Additionally, the paper presents a suite of baseline approaches to iTBLS, utilizing zero-shot prompting and parameter-efficient fine-tuning for different computing situations. We also introduce a novel multi-step approach and show how it can be leveraged in conjunction with parameter-efficient fine-tuning to achieve the state-of-the-art on iTBLS; outperforming standard parameter-efficient fine-tuning by up to 15% on interpretation, 18% on modification, and 38% on generation.
• 摘要：本文介绍了交互式表 (iTBLS)，这是一个位于科学文章表格中的交互式对话数据集。该数据集旨在通过人工智能驱动的多任务表格功能促进人类与人工智能协作解决问题。与之前将交互建模为事实问答或程序合成的工作相比，iTBLS 拓宽了交互的范围，包括数学推理、自然语言操作以及通过将交互描述为以下三个任务之一来扩展自然语言对话中的现有表格：解释、修改或生成。此外，本文还提出了一套 iTBLS 基线方法，针对不同的计算情况利用零样本提示和参数高效的微调。我们还介绍了一种新颖的多步骤方法，并展示了如何将其与参数高效的微调结合起来，以实现 iTBLS 的最新技术；在解释方面优于标准参数效率微调高达 15%，在修改方面优于 18%，在生成方面优于 38%。

Title: Parameter Efficient Diverse Paraphrase Generation Using Sequence-Level Knowledge Distillation

• Authors: Lasal Jayawardena, Prasan Yapa
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.12596
• Pdf URL: https://arxiv.org/pdf/2404.12596
• Copy Paste: [[2404.12596]] Parameter Efficient Diverse Paraphrase Generation Using Sequence-Level Knowledge Distillation(https://arxiv.org/abs/2404.12596)
• Keywords: language model, llm
• Abstract: Over the past year, the field of Natural Language Generation (NLG) has experienced an exponential surge, largely due to the introduction of Large Language Models (LLMs). These models have exhibited the most effective performance in a range of domains within the Natural Language Processing and Generation domains. However, their application in domain-specific tasks, such as paraphrasing, presents significant challenges. The extensive number of parameters makes them difficult to operate on commercial hardware, and they require substantial time for inference, leading to high costs in a production setting. In this study, we tackle these obstacles by employing LLMs to develop three distinct models for the paraphrasing field, applying a method referred to as sequence-level knowledge distillation. These distilled models are capable of maintaining the quality of paraphrases generated by the LLM. They demonstrate faster inference times and the ability to generate diverse paraphrases of comparable quality. A notable characteristic of these models is their ability to exhibit syntactic diversity while also preserving lexical diversity, features previously uncommon due to existing data quality issues in datasets and not typically observed in neural-based approaches. Human evaluation of our models shows that there is only a 4% drop in performance compared to the LLM teacher model used in the distillation process, despite being 1000 times smaller. This research provides a significant contribution to the NLG field, offering a more efficient and cost-effective solution for paraphrasing tasks.
• 摘要：在过去的一年里，自然语言生成（NLG）领域经历了指数级的增长，这很大程度上归功于大型语言模型（LLM）的引入。这些模型在自然语言处理和生成领域的一系列领域中表现出了最有效的性能。然而，它们在特定领域任务（例如释义）中的应用提出了重大挑战。大量的参数使得它们难以在商业硬件上运行，并且需要大量时间进行推理，导致生产环境中的成本很高。在这项研究中，我们通过利用法学硕士为释义领域开发三种不同的模型，并应用一种称为序列级知识蒸馏的方法来解决这些障碍。这些精炼模型能够保持法学硕士生成的释义的质量。它们表现出更快的推理时间和生成具有可比较质量的各种释义的能力。这些模型的一个显着特征是它们能够展现句法多样性，同时保留词汇多样性，由于数据集中现有的数据质量问题，这些特征以前并不常见，并且在基于神经的方法中通常不会观察到。对我们模型的人工评估表明，与蒸馏过程中使用的 LLM 教师模型相比，尽管尺寸小了 1000 倍，但性能仅下降了 4%。这项研究为 NLG 领域做出了重大贡献，为释义任务提供了更高效、更具成本效益的解决方案。

Title: Cooperative Sentiment Agents for Multimodal Sentiment Analysis

• Authors: Shanmin Wang, Hui Shuai, Qingshan Liu, Fei Wang
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2404.12642
• Pdf URL: https://arxiv.org/pdf/2404.12642
• Copy Paste: [[2404.12642]] Cooperative Sentiment Agents for Multimodal Sentiment Analysis(https://arxiv.org/abs/2404.12642)
• Keywords: agent
• Abstract: In this paper, we propose a new Multimodal Representation Learning (MRL) method for Multimodal Sentiment Analysis (MSA), which facilitates the adaptive interaction between modalities through Cooperative Sentiment Agents, named Co-SA. Co-SA comprises two critical components: the Sentiment Agents Establishment (SAE) phase and the Sentiment Agents Cooperation (SAC) phase. During the SAE phase, each sentiment agent deals with an unimodal signal and highlights explicit dynamic sentiment variations within the modality via the Modality-Sentiment Disentanglement (MSD) and Deep Phase Space Reconstruction (DPSR) modules. Subsequently, in the SAC phase, Co-SA meticulously designs task-specific interaction mechanisms for sentiment agents so that coordinating multimodal signals to learn the joint representation. Specifically, Co-SA equips an independent policy model for each sentiment agent that captures significant properties within the modality. These policies are optimized mutually through the unified reward adaptive to downstream tasks. Benefitting from the rewarding mechanism, Co-SA transcends the limitation of pre-defined fusion modes and adaptively captures unimodal properties for MRL in the multimodal interaction setting. To demonstrate the effectiveness of Co-SA, we apply it to address Multimodal Sentiment Analysis (MSA) and Multimodal Emotion Recognition (MER) tasks. Our comprehensive experimental results demonstrate that Co-SA excels at discovering diverse cross-modal features, encompassing both common and complementary aspects. The code can be available at https://github.com/smwanghhh/Co-SA.
• 摘要：在本文中，我们提出了一种用于多模态情感分析（MSA）的新多模态表示学习（MRL）方法，该方法通过协作情感代理（Co-SA）促进模态之间的自适应交互。 Co-SA 包含两个关键组成部分：情感代理建立 (SAE) 阶段和情感代理合作 (SAC) 阶段。在 SAE 阶段，每个情感代理处理单模态信号，并通过模态情感解缠 (MSD) 和深度相空间重建 (DPSR) 模块突出显示模态内的显式动态情感变化。随后，在SAC阶段，Co-SA精心设计了情感代理的特定任务交互机制，以便协调多模态信号来学习联合表示。具体来说，Co-SA 为每个情绪代理配备了一个独立的策略模型，该模型捕获了模式中的重要属性。这些策略通过适应下游任务的统一奖励相互优化。受益于奖励机制，Co-SA 超越了预定义融合模式的限制，在多模态交互设置中自适应地捕获 MRL 的单峰特性。为了证明 Co-SA 的有效性，我们将其应用于多模态情感分析 (MSA) 和多模态情绪识别 (MER) 任务。我们全面的实验结果表明，Co-SA 擅长发现多种跨模式特征，涵盖共同和互补的方面。该代码可在 https://github.com/smwanghhh/Co-SA 获取。

Title: SOS-1K: A Fine-grained Suicide Risk Classification Dataset for Chinese Social Media Analysis

• Authors: Hongzhi Qi, Hanfei Liu, Jianqiang Li, Qing Zhao, Wei Zhai, Dan Luo, Tian Yu He, Shuo Liu, Bing Xiang Yang, Guanghui Fu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.12659
• Pdf URL: https://arxiv.org/pdf/2404.12659
• Copy Paste: [[2404.12659]] SOS-1K: A Fine-grained Suicide Risk Classification Dataset for Chinese Social Media Analysis(https://arxiv.org/abs/2404.12659)
• Keywords: language model
• Abstract: In the social media, users frequently express personal emotions, a subset of which may indicate potential suicidal tendencies. The implicit and varied forms of expression in internet language complicate accurate and rapid identification of suicidal intent on social media, thus creating challenges for timely intervention efforts. The development of deep learning models for suicide risk detection is a promising solution, but there is a notable lack of relevant datasets, especially in the Chinese context. To address this gap, this study presents a Chinese social media dataset designed for fine-grained suicide risk classification, focusing on indicators such as expressions of suicide intent, methods of suicide, and urgency of timing. Seven pre-trained models were evaluated in two tasks: high and low suicide risk, and fine-grained suicide risk classification on a level of 0 to 10. In our experiments, deep learning models show good performance in distinguishing between high and low suicide risk, with the best model achieving an F1 score of 88.39%. However, the results for fine-grained suicide risk classification were still unsatisfactory, with an weighted F1 score of 50.89%. To address the issues of data imbalance and limited dataset size, we investigated both traditional and advanced, large language model based data augmentation techniques, demonstrating that data augmentation can enhance model performance by up to 4.65% points in F1-score. Notably, the Chinese MentalBERT model, which was pre-trained on psychological domain data, shows superior performance in both tasks. This study provides valuable insights for automatic identification of suicidal individuals, facilitating timely psychological intervention on social media platforms. The source code and data are publicly available.
• 摘要：在社交媒体中，用户经常表达个人情绪，其中一部分可能表明潜在的自杀倾向。网络语言含蓄且多样的表达形式使得社交媒体上自杀意图的准确快速识别变得复杂，从而给及时干预工作带来了挑战。开发自杀风险检测的深度学习模型是一个很有前途的解决方案，但相关数据集明显缺乏，特别是在中国背景下。为了解决这一差距，本研究提出了一个中国社交媒体数据集，旨在对自杀风险进行细粒度分类，重点关注自杀意图表达、自杀方法和时机紧迫性等指标。七个预训练模型在两项任务中进行了评估：高自杀风险和低自杀风险，以及 0 到 10 级的细粒度自杀风险分类。在我们的实验中，深度学习模型在区分高自杀风险和低自杀风险方面表现出了良好的性能，最好的模型达到了 88.39% 的 F1 分数。然而，细粒度自杀风险分类的结果仍然不理想，加权F1得分为50.89%。为了解决数据不平衡和数据集大小有限的问题，我们研究了传统和先进的基于大型语言模型的数据增强技术，证明数据增强可以将模型性能在 F1 分数中提高高达 4.65%。值得注意的是，根据心理领域数据进行预训练的中国 MentalBERT 模型在这两项任务中都表现出了优异的性能。这项研究为自动识别自杀个体提供了宝贵的见解，有助于在社交媒体平台上进行及时的心理干预。源代码和数据是公开的。

Title: Enabling Ensemble Learning for Heterogeneous Large Language Models with Deep Parallel Collaboration

• Authors: Yichong Huang, Xiaocheng Feng, Baohang Li, Yang Xiang, Hui Wang, Bing Qin, Ting Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.12715
• Pdf URL: https://arxiv.org/pdf/2404.12715
• Copy Paste: [[2404.12715]] Enabling Ensemble Learning for Heterogeneous Large Language Models with Deep Parallel Collaboration(https://arxiv.org/abs/2404.12715)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have shown complementary strengths in various tasks and instances, motivating the research of ensembling LLMs to push the frontier leveraging the wisdom of the crowd. Existing work achieves this objective via training the extra reward model or fusion model to select or fuse all candidate answers. However, these methods pose a great challenge to the generalizability of the trained models. Besides, existing methods use the textual responses as communication media, ignoring the rich information in the inner representations of neural networks. Therefore, we propose a training-free ensemble framework DEEPEN, averaging the probability distributions outputted by different LLMs. A key challenge in this paradigm is the vocabulary discrepancy between heterogeneous LLMs, which hinders the operation of probability distribution averaging. To address this challenge, DEEPEN maps the probability distribution of each model from the probability space to a universe relative space based on the relative representation theory, and performs aggregation. Then, the result of aggregation is mapped back to the probability space of one LLM via a search-based inverse transformation to determine the generated token. We conduct experiments on the ensemble of various LLMs of 6B to 70B. Experimental results show that DEEPEN achieves consistent improvements across six popular benchmarks involving subject examination, reasoning and knowledge-QA, proving the effectiveness of our approach.
• 摘要：大型语言模型（LLM）在各种任务和实例中显示出互补的优势，激发了集成 LLM 的研究，利用群体的智慧推动前沿发展。现有工作通过训练额外奖励模型或融合模型来选择或融合所有候选答案来实现这一目标。然而，这些方法对训练模型的泛化能力提出了巨大的挑战。此外，现有方法使用文本响应作为通信媒介，忽略了神经网络内部表示中的丰富信息。因此，我们提出了一个免训练的集成框架 DEEPEN，对不同 LLM 输出的概率分布进行平均。该范式的一个关键挑战是异构法学硕士之间的词汇差异，这阻碍了概率分布平均的操作。为了应对这一挑战，DEEPEN基于相对表示理论将每个模型的概率分布从概率空间映射到宇宙相对空间，并进行聚合。然后，聚合结果通过基于搜索的逆变换映射回一个LLM的概率空间，以确定生成的令牌。我们对 6B 到 70B 的各种 LLM 的集合进行了实验。实验结果表明，DEEPEN 在涉及科目考试、推理和知识 QA 的六种流行基准测试中取得了一致的改进，证明了我们方法的有效性。

Title: Evaluating Character Understanding of Large Language Models via Character Profiling from Fictional Works

• Authors: Xinfeng Yuan, Siyu Yuan, Yuhan Cui, Tianhe Lin, Xintao Wang, Rui Xu, Jiangjie Chen, Deqing Yang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.12726
• Pdf URL: https://arxiv.org/pdf/2404.12726
• Copy Paste: [[2404.12726]] Evaluating Character Understanding of Large Language Models via Character Profiling from Fictional Works(https://arxiv.org/abs/2404.12726)
• Keywords: language model, llm, agent
• Abstract: Large language models (LLMs) have demonstrated impressive performance and spurred numerous AI applications, in which role-playing agents (RPAs) are particularly popular, especially for fictional characters. The prerequisite for these RPAs lies in the capability of LLMs to understand characters from fictional works. Previous efforts have evaluated this capability via basic classification tasks or characteristic imitation, failing to capture the nuanced character understanding with LLMs. In this paper, we propose evaluating LLMs' character understanding capability via the character profiling task, i.e., summarizing character profiles from corresponding materials, a widely adopted yet understudied practice for RPA development. Specifically, we construct the CroSS dataset from literature experts and assess the generated profiles by comparing ground truth references and their applicability in downstream tasks. Our experiments, which cover various summarization methods and LLMs, have yielded promising results. These results strongly validate the character understanding capability of LLMs. We believe our constructed resource will promote further research in this field. Resources are available at https://github.com/Joanna0123/character_profiling.
• 摘要：大型语言模型（LLM）已经表现出令人印象深刻的性能，并刺激了众多人工智能应用，其中角色扮演代理（RPA）特别受欢迎，尤其是对于虚构角色。这些 RPA 的先决条件在于法学硕士有能力理解虚构作品中的角色。以前的工作是通过基本分类任务或特征模仿来评估这种能力，未能捕捉到法学硕士对细致入微的字符理解。在本文中，我们建议通过字符分析任务来评估法学硕士的字符理解能力，即从相应材料中总结字符配置文件，这是 RPA 开发中广泛采用但尚未充分研究的实践。具体来说，我们从文献专家那里构建了 CroSS 数据集，并通过比较真实参考及其在下游任务中的适用性来评估生成的配置文件。我们的实验涵盖了各种总结方法和法学硕士，取得了有希望的结果。这些结果有力地验证了法学硕士的性格理解能力。我们相信我们构建的资源将促进该领域的进一步研究。资源可在 https://github.com/Joanna0123/character_profiling 获取。

Title: Relevant or Random: Can LLMs Truly Perform Analogical Reasoning?

• Authors: Chengwei Qin, Wenhan Xia, Tan Wang, Fangkai Jiao, Yuchen Hu, Bosheng Ding, Ruirui Chen, Shafiq Joty
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.12728
• Pdf URL: https://arxiv.org/pdf/2404.12728
• Copy Paste: [[2404.12728]] Relevant or Random: Can LLMs Truly Perform Analogical Reasoning?(https://arxiv.org/abs/2404.12728)
• Keywords: language model, llm, prompt
• Abstract: Analogical reasoning is a unique ability of humans to address unfamiliar challenges by transferring strategies from relevant past experiences. One key finding in psychology is that compared with irrelevant past experiences, recalling relevant ones can help humans better handle new tasks. Coincidentally, the NLP community has also recently found that self-generating relevant examples in the context can help large language models (LLMs) better solve a given problem than hand-crafted prompts. However, it is yet not clear whether relevance is the key factor eliciting such capability, i.e., can LLMs benefit more from self-generated relevant examples than irrelevant ones? In this work, we systematically explore whether LLMs can truly perform analogical reasoning on a diverse set of reasoning tasks. With extensive experiments and analysis, we show that self-generated random examples can surprisingly achieve comparable or even better performance, e.g., 4% performance boost on GSM8K with random biological examples. We find that the accuracy of self-generated examples is the key factor and subsequently design two improved methods with significantly reduced inference costs. Overall, we aim to advance a deeper understanding of LLM analogical reasoning and hope this work stimulates further research in the design of self-generated contexts.
• 摘要：类比推理是人类通过从相关的过去经验中转移策略来应对不熟悉的挑战的独特能力。心理学的一项重要发现是，与过去不相关的经历相比，回忆相关的经历可以帮助人类更好地处理新任务。无独有偶，NLP 社区最近也发现，在上下文中自行生成相关示例可以帮助大型语言模型 (LLM) 比手工提示更好地解决给定问题。然而，目前尚不清楚相关性是否是引发这种能力的关键因素，即法学硕士是否可以从自我生成的相关示例中比不相关的示例中受益更多？在这项工作中，我们系统地探讨了法学硕士是否能够真正对一组​​不同的推理任务进行类比推理。通过大量的实验和分析，我们表明，自行生成的随机示例可以令人惊讶地实现可比甚至更好的性能，例如，随机生物示例在 GSM8K 上的性能提升了 4%。我们发现自生成示例的准确性是关键因素，随后设计了两种改进的方法，显着降低了推理成本。总的来说，我们的目标是加深对法学硕士类比推理的理解，并希望这项工作能够激发对自生成上下文设计的进一步研究。

Title: Beyond Human Norms: Unveiling Unique Values of Large Language Models through Interdisciplinary Approaches

• Authors: Pablo Biedma, Xiaoyuan Yi, Linus Huang, Maosong Sun, Xing Xie
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.12744
• Pdf URL: https://arxiv.org/pdf/2404.12744
• Copy Paste: [[2404.12744]] Beyond Human Norms: Unveiling Unique Values of Large Language Models through Interdisciplinary Approaches(https://arxiv.org/abs/2404.12744)
• Keywords: language model, llm
• Abstract: Recent advancements in Large Language Models (LLMs) have revolutionized the AI field but also pose potential safety and ethical risks. Deciphering LLMs' embedded values becomes crucial for assessing and mitigating their risks. Despite extensive investigation into LLMs' values, previous studies heavily rely on human-oriented value systems in social sciences. Then, a natural question arises: Do LLMs possess unique values beyond those of humans? Delving into it, this work proposes a novel framework, ValueLex, to reconstruct LLMs' unique value system from scratch, leveraging psychological methodologies from human personality/value research. Based on Lexical Hypothesis, ValueLex introduces a generative approach to elicit diverse values from 30+ LLMs, synthesizing a taxonomy that culminates in a comprehensive value framework via factor analysis and semantic clustering. We identify three core value dimensions, Competence, Character, and Integrity, each with specific subdimensions, revealing that LLMs possess a structured, albeit non-human, value system. Based on this system, we further develop tailored projective tests to evaluate and analyze the value inclinations of LLMs across different model sizes, training methods, and data sources. Our framework fosters an interdisciplinary paradigm of understanding LLMs, paving the way for future AI alignment and regulation.
• 摘要：大型语言模型 (LLM) 的最新进展彻底改变了人工智能领域，但也带来了潜在的安全和道德风险。解读法学硕士的内在价值对于评估和减轻其风险至关重要。尽管对法学硕士的价值观进行了广泛的调查，但以前的研究严重依赖社会科学中以人为本的价值体系。那么，一个自然的问题就出现了：法学硕士是否拥有超越人类的独特价值？深入研究后，这项工作提出了一个新颖的框架——ValueLex，利用人类个性/价值研究的心理学方法论，从头开始重建法学硕士独特的价值体系。 ValueLex 基于词汇假设，引入了一种生成方法，从 30 多个法学硕士中得出不同的价值观，综合分类法，通过因素分析和语义聚类最终形成综合价值框架。我们确定了三个核心价值维度：能力、品格和诚信，每个维度都有特定的子维度，揭示了法学硕士拥有一个结构化的、非人类的价值体系。基于该系统，我们进一步开发定制的投影测试，以评估和分析法学硕士在不同模型大小、训练方法和数据源上的价值倾向。我们的框架培育了理解法学硕士的跨学科范式，为未来人工智能的调整和监管铺平了道路。

Title: AutoCrawler: A Progressive Understanding Web Agent for Web Crawler Generation

• Authors: Wenhao Huang, Chenghao Peng, Zhixu Li, Jiaqing Liang, Yanghua Xiao, Liqian Wen, Zulong Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.12753
• Pdf URL: https://arxiv.org/pdf/2404.12753
• Copy Paste: [[2404.12753]] AutoCrawler: A Progressive Understanding Web Agent for Web Crawler Generation(https://arxiv.org/abs/2404.12753)
• Keywords: language model, llm, agent
• Abstract: Web automation is a significant technique that accomplishes complicated web tasks by automating common web actions, enhancing operational efficiency, and reducing the need for manual intervention. Traditional methods, such as wrappers, suffer from limited adaptability and scalability when faced with a new website. On the other hand, generative agents empowered by large language models (LLMs) exhibit poor performance and reusability in open-world scenarios. In this work, we introduce a crawler generation task for vertical information web pages and the paradigm of combining LLMs with crawlers, which helps crawlers handle diverse and changing web environments more efficiently. We propose AutoCrawler, a two-stage framework that leverages the hierarchical structure of HTML for progressive understanding. Through top-down and step-back operations, AutoCrawler can learn from erroneous actions and continuously prune HTML for better action generation. We conduct comprehensive experiments with multiple LLMs and demonstrate the effectiveness of our framework. Resources of this paper can be found at \url{https://github.com/EZ-hwh/AutoCrawler}
• 摘要：Web 自动化是一项重要技术，它通过自动化常见的 Web 操作、提高操作效率并减少手动干预的需要来完成复杂的 Web 任务。传统方法（例如包装器）在面对新网站时，适应性和可扩展性有限。另一方面，由大型语言模型（LLM）支持的生成代理在开放世界场景中表现出较差的性能和可重用性。在这项工作中，我们引入了垂直信息网页的爬虫生成任务以及LLM与爬虫相结合的范例，这有助于爬虫更有效地处理多样化且不断变化的网络环境。我们提出了 AutoCrawler，这是一个两阶段框架，它利用 HTML 的层次结构来逐步理解。通过自顶向下和后退操作，AutoCrawler 可以从错误的操作中学习，并不断修剪 HTML 以更好地生成操作。我们对多个法学硕士进行了全面的实验，并证明了我们框架的有效性。本文的资源可以在 \url{https://github.com/EZ-hwh/AutoCrawler} 找到

Title: LiMe: a Latin Corpus of Late Medieval Criminal Sentences

• Authors: Alessandra Bassani, Beatrice Del Bo, Alfio Ferrara, Marta Mangini, Sergio Picascia, Ambra Stefanello
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.12829
• Pdf URL: https://arxiv.org/pdf/2404.12829
• Copy Paste: [[2404.12829]] LiMe: a Latin Corpus of Late Medieval Criminal Sentences(https://arxiv.org/abs/2404.12829)
• Keywords: language model
• Abstract: The Latin language has received attention from the computational linguistics research community, which has built, over the years, several valuable resources, ranging from detailed annotated corpora to sophisticated tools for linguistic analysis. With the recent advent of large language models, researchers have also started developing models capable of generating vector representations of Latin texts. The performances of such models remain behind the ones for modern languages, given the disparity in available data. In this paper, we present the LiMe dataset, a corpus of 325 documents extracted from a series of medieval manuscripts called Libri sententiarum potestatis Mediolani, and thoroughly annotated by experts, in order to be employed for masked language model, as well as supervised natural language processing tasks.
• 摘要：拉丁语言受到了计算语言学研究界的关注，多年来，计算语言学研究界已经建立了一些宝贵的资源，从详细的注释语料库到复杂的语言分析工具。随着大型语言模型的出现，研究人员也开始开发能够生成拉丁文本向量表示的模型。鉴于可用数据的差异，此类模型的性能仍然落后于现代语言的性能。在本文中，我们提出了 LiMe 数据集，这是一个包含 325 个文档的语料库，从一系列名为 Libri Sententiarum potestatis Mediolani 的中世纪手稿中提取，并由专家进行了彻底注释，以便用于掩码语言模型以及监督自然语言处理任务。

Title: TartuNLP @ SIGTYP 2024 Shared Task: Adapting XLM-RoBERTa for Ancient and Historical Languages

• Authors: Aleksei Dorkin, Kairit Sirts
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.12845
• Pdf URL: https://arxiv.org/pdf/2404.12845
• Copy Paste: [[2404.12845]] TartuNLP @ SIGTYP 2024 Shared Task: Adapting XLM-RoBERTa for Ancient and Historical Languages(https://arxiv.org/abs/2404.12845)
• Keywords: language model
• Abstract: We present our submission to the unconstrained subtask of the SIGTYP 2024 Shared Task on Word Embedding Evaluation for Ancient and Historical Languages for morphological annotation, POS-tagging, lemmatization, character- and word-level gap-filling. We developed a simple, uniform, and computationally lightweight approach based on the adapters framework using parameter-efficient fine-tuning. We applied the same adapter-based approach uniformly to all tasks and 16 languages by fine-tuning stacked language- and task-specific adapters. Our submission obtained an overall second place out of three submissions, with the first place in word-level gap-filling. Our results show the feasibility of adapting language models pre-trained on modern languages to historical and ancient languages via adapter training.
• 摘要：我们提交了对 SIGTYP 2024 古代和历史语言词嵌入评估共享任务的无约束子任务的提交，用于形态注释、词性标记、词形还原、字符和词级空白填充。我们基于适配器框架使用参数高效的微调开发了一种简单、统一且计算量轻的方法。通过微调堆叠的特定于语言和任务的适配器，我们将相同的基于适配器的方法统一应用于所有任务和 16 种语言。我们提交的内容在三份提交内容中获得了总体第二名，其中在单词级空白填充方面获得第一名。我们的结果表明通过适配器训练将现代语言预训练的语言模型适应历史和古代语言的可行性。

Title: How Does the Textual Information Affect the Retrieval of Multimodal In-Context Learning?

• Authors: Yang Luo, Zangwei Zheng, Zirui Zhu, Yang You
• Subjects: cs.CL, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2404.12866
• Pdf URL: https://arxiv.org/pdf/2404.12866
• Copy Paste: [[2404.12866]] How Does the Textual Information Affect the Retrieval of Multimodal In-Context Learning?(https://arxiv.org/abs/2404.12866)
• Keywords: language model, llm
• Abstract: The increase in parameter size of multimodal large language models (MLLMs) introduces significant capabilities, particularly in-context learning, where MLLMs enhance task performance without updating pre-trained parameters. This effectiveness, however, hinges on the appropriate selection of in-context examples, a process that is currently biased towards visual data, overlooking textual information. Furthermore, the area of supervised retrievers for MLLMs, crucial for optimal in-context example selection, continues to be uninvestigated. Our study offers an in-depth evaluation of the impact of textual information on the unsupervised selection of in-context examples in multimodal contexts, uncovering a notable sensitivity of retriever performance to the employed modalities. Responding to this, we introduce a novel supervised MLLM-retriever MSIER that employs a neural network to select examples that enhance multimodal in-context learning efficiency. This approach is validated through extensive testing across three distinct tasks, demonstrating the method's effectiveness. Additionally, we investigate the influence of modalities on our supervised retrieval method's training and pinpoint factors contributing to our model's success. This exploration paves the way for future advancements, highlighting the potential for refined in-context learning in MLLMs through the strategic use of multimodal data.
• 摘要：多模态大语言模型 (MLLM) 参数大小的增加引入了重要的功能，特别是上下文学习，其中 MLLM 无需更新预先训练的参数即可增强任务性能。然而，这种有效性取决于对上下文示例的适当选择，目前这一过程偏向于视觉数据，而忽视了文本信息。此外，对于最佳上下文示例选择至关重要的 MLLM 监督检索器领域仍然没有得到研究。我们的研究深入评估了文本信息对多模态环境中上下文示例的无监督选择的影响，揭示了检索器性能对所使用模态的显着敏感性。针对这一点，我们引入了一种新颖的监督式 MLLM 检索器 MSIER，它采用神经网络来选择可以提高多模态上下文学习效率的示例。该方法通过三个不同任务的广泛测试得到验证，证明了该方法的有效性。此外，我们还研究了模态对监督检索方法训练的影响，并查明有助于模型成功的因素。这一探索为未来的进步铺平了道路，凸显了通过战略性使用多模态数据在 MLLM 中进行精细情境学习的潜力。

Title: Unlocking Multi-View Insights in Knowledge-Dense Retrieval-Augmented Generation

• Authors: Guanhua Chen, Wenhan Yu, Lei Sha
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.12879
• Pdf URL: https://arxiv.org/pdf/2404.12879
• Copy Paste: [[2404.12879]] Unlocking Multi-View Insights in Knowledge-Dense Retrieval-Augmented Generation(https://arxiv.org/abs/2404.12879)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: While Retrieval-Augmented Generation (RAG) plays a crucial role in the application of Large Language Models (LLMs), existing retrieval methods in knowledge-dense domains like law and medicine still suffer from a lack of multi-perspective views, which are essential for improving interpretability and reliability. Previous research on multi-view retrieval often focused solely on different semantic forms of queries, neglecting the expression of specific domain knowledge perspectives. This paper introduces a novel multi-view RAG framework, MVRAG, tailored for knowledge-dense domains that utilizes intention-aware query rewriting from multiple domain viewpoints to enhance retrieval precision, thereby improving the effectiveness of the final inference. Experiments conducted on legal and medical case retrieval demonstrate significant improvements in recall and precision rates with our framework. Our multi-perspective retrieval approach unleashes the potential of multi-view information enhancing RAG tasks, accelerating the further application of LLMs in knowledge-intensive fields.
• 摘要：虽然检索增强生成（RAG）在大型语言模型（LLM）的应用中发挥着至关重要的作用，但在法律和医学等知识密集领域，现有的检索方法仍然缺乏多视角的观点，而这对于大语言模型（LLM）的应用至关重要。提高可解释性和可靠性。以往的多视图检索研究往往仅仅关注查询的不同语义形式，而忽略了特定领域知识视角的表达。本文介绍了一种新颖的多视图RAG框架MVRAG，该框架专为知识密集型领域量身定制，利用来自多个领域视点的意图感知查询重写来提高检索精度，从而提高最终推理的有效性。对法律和医疗案例检索进行的实验表明，我们的框架在召回率和准确率方面有显着提高。我们的多视角检索方法释放了多视角信息增强 RAG 任务的潜力，加速了法学硕士在知识密集型领域的进一步应用。

Title: Enabling Natural Zero-Shot Prompting on Encoder Models via Statement-Tuning

• Authors: Ahmed Elshabrawy, Yongix Huang, Iryna Gurevych, Alham Fikri Aji
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.12897
• Pdf URL: https://arxiv.org/pdf/2404.12897
• Copy Paste: [[2404.12897]] Enabling Natural Zero-Shot Prompting on Encoder Models via Statement-Tuning(https://arxiv.org/abs/2404.12897)
• Keywords: language model, llm, prompt
• Abstract: While Large Language Models (LLMs) exhibit remarkable capabilities in zero-shot and few-shot scenarios, they often require computationally prohibitive sizes. Conversely, smaller Masked Language Models (MLMs) like BERT and RoBERTa achieve state-of-the-art results through fine-tuning but struggle with extending to few-shot and zero-shot settings due to their architectural constraints. Hence, we propose Statement-Tuning, a technique that models discriminative tasks as a set of finite statements and trains an Encoder model to discriminate between the potential statements to determine the label. We do Statement-Tuning on multiple tasks to enable cross-task generalization. Experimental results demonstrate that Statement Tuning achieves competitive performance compared to state-of-the-art LLMs with significantly fewer parameters. Moreover, the study investigates the impact of several design choices on few-shot and zero-shot generalization, revealing that Statement Tuning can achieve sufficient performance with modest training data and benefits from task and statement diversity for unseen task generalizability.
• 摘要：虽然大型语言模型 (LLM) 在零样本和少样本场景中表现出卓越的功能，但它们通常需要计算量过大的大小。相反，像 BERT 和 RoBERTa 这样的小型掩码语言模型 (MLM) 通过微调实现了最先进的结果，但由于其架构限制，很难扩展到少样本和零样本设置。因此，我们提出了语句调优（Statement-Tuning），这是一种将判别性任务建模为一组有限语句并训练编码器模型来区分潜在语句以确定标签的技术。我们对多个任务进行语句调优以实现跨任务泛化。实验结果表明，与参数少得多的最先进的 LLM 相比，Statement Tuning 实现了具有竞争力的性能。此外，该研究还调查了几种设计选择对少样本和零样本泛化的影响，揭示了语句调优可以通过适度的训练数据实现足够的性能，并从任务和语句多样性中获益，从而实现看不见的任务泛化性。

Title: Cross-cultural Inspiration Detection and Analysis in Real and LLM-generated Social Media Data

• Authors: Oana Ignat, Gayathri Ganesh Lakshmy, Rada Mihalcea
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.12933
• Pdf URL: https://arxiv.org/pdf/2404.12933
• Copy Paste: [[2404.12933]] Cross-cultural Inspiration Detection and Analysis in Real and LLM-generated Social Media Data(https://arxiv.org/abs/2404.12933)
• Keywords: gpt, llm
• Abstract: Inspiration is linked to various positive outcomes, such as increased creativity, productivity, and happiness. Although inspiration has great potential, there has been limited effort toward identifying content that is inspiring, as opposed to just engaging or positive. Additionally, most research has concentrated on Western data, with little attention paid to other cultures. This work is the first to study cross-cultural inspiration through machine learning methods. We aim to identify and analyze real and AI-generated cross-cultural inspiring posts. To this end, we compile and make publicly available the InspAIred dataset, which consists of 2,000 real inspiring posts, 2,000 real non-inspiring posts, and 2,000 generated inspiring posts evenly distributed across India and the UK. The real posts are sourced from Reddit, while the generated posts are created using the GPT-4 model. Using this dataset, we conduct extensive computational linguistic analyses to (1) compare inspiring content across cultures, (2) compare AI-generated inspiring posts to real inspiring posts, and (3) determine if detection models can accurately distinguish between inspiring content across cultures and data sources.
• 摘要：灵感与各种积极成果相关，例如创造力、生产力和幸福感的提高。尽管灵感具有巨大的潜力，但在识别具有启发性的内容（而不是仅仅吸引人或积极的内容）方面所做的努力有限。此外，大多数研究都集中在西方数据上，很少关注其他文化。这项工作是第一个通过机器学习方法研究跨文化灵感的工作。我们的目标是识别和分析真实的和人工智能生成的跨文化鼓舞人心的帖子。为此，我们编译并公开了 InspAIred 数据集，其中包含 2,000 个真实的鼓舞人心的帖子、2,000 个真实的非鼓舞人心的帖子和 2,000 个生成的鼓舞人心的帖子，均匀分布在印度和英国。真实的帖子来自 Reddit，而生成的帖子是使用 GPT-4 模型创建的。使用此数据集，我们进行广泛的计算语言分析，以（1）比较跨文化的鼓舞人心的内容，（2）将人工智能生成的鼓舞人心的帖子与真实的鼓舞人心的帖子进行比较，以及（3）确定检测模型是否可以准确地区分跨文化的鼓舞人心的内容和数据源。

Title: MAiDE-up: Multilingual Deception Detection of GPT-generated Hotel Reviews

• Authors: Oana Ignat, Xiaomeng Xu, Rada Mihalcea
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.12938
• Pdf URL: https://arxiv.org/pdf/2404.12938
• Copy Paste: [[2404.12938]] MAiDE-up: Multilingual Deception Detection of GPT-generated Hotel Reviews(https://arxiv.org/abs/2404.12938)
• Keywords: gpt, llm
• Abstract: Deceptive reviews are becoming increasingly common, especially given the increase in performance and the prevalence of LLMs. While work to date has addressed the development of models to differentiate between truthful and deceptive human reviews, much less is known about the distinction between real reviews and AI-authored fake reviews. Moreover, most of the research so far has focused primarily on English, with very little work dedicated to other languages. In this paper, we compile and make publicly available the MAiDE-up dataset, consisting of 10,000 real and 10,000 AI-generated fake hotel reviews, balanced across ten languages. Using this dataset, we conduct extensive linguistic analyses to (1) compare the AI fake hotel reviews to real hotel reviews, and (2) identify the factors that influence the deception detection model performance. We explore the effectiveness of several models for deception detection in hotel reviews across three main dimensions: sentiment, location, and language. We find that these dimensions influence how well we can detect AI-generated fake reviews.
• 摘要：欺骗性评论变得越来越普遍，特别是考虑到法学硕士的表现不断提高和普遍存在。虽然迄今为止的工作已经解决了模型的开发问题，以区分真实的和欺骗性的人类评论，但人们对真实评论和人工智能撰写的虚假评论之间的区别知之甚少。此外，到目前为止，大多数研究主要集中在英语上，很少有专门针对其他语言的研究。在本文中，我们编译并公开了 MAiDE-up 数据集，其中包含 10,000 条真实酒店评论和 10,000 条人工智能生成的虚假酒店评论，涵盖十种语言。使用该数据集，我们进行了广泛的语言分析，以 (1) 将人工智能虚假酒店评论与真实酒店评论进行比较，以及 (2) 确定影响欺骗检测模型性能的因素。我们从三个主要维度探索了酒店评论中欺骗检测的几种模型的有效性：情绪、位置和语言。我们发现这些维度会影响我们检测人工智能生成的虚假评论的能力。

Title: Towards Reliable Latent Knowledge Estimation in LLMs: In-Context Learning vs. Prompting Based Factual Knowledge Extraction

• Authors: Qinyuan Wu, Mohammad Aflah Khan, Soumi Das, Vedant Nanda, Bishwamittra Ghosh, Camila Kolling, Till Speicher, Laurent Bindschaedler, Krishna P. Gummadi, Evimaria Terzi
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.12957
• Pdf URL: https://arxiv.org/pdf/2404.12957
• Copy Paste: [[2404.12957]] Towards Reliable Latent Knowledge Estimation in LLMs: In-Context Learning vs. Prompting Based Factual Knowledge Extraction(https://arxiv.org/abs/2404.12957)
• Keywords: language model, llm, prompt
• Abstract: We propose an approach for estimating the latent knowledge embedded inside large language models (LLMs). We leverage the in-context learning (ICL) abilities of LLMs to estimate the extent to which an LLM knows the facts stored in a knowledge base. Our knowledge estimator avoids reliability concerns with previous prompting-based methods, is both conceptually simpler and easier to apply, and we demonstrate that it can surface more of the latent knowledge embedded in LLMs. We also investigate how different design choices affect the performance of ICL-based knowledge estimation. Using the proposed estimator, we perform a large-scale evaluation of the factual knowledge of a variety of open source LLMs, like OPT, Pythia, Llama(2), Mistral, Gemma, etc. over a large set of relations and facts from the Wikidata knowledge base. We observe differences in the factual knowledge between different model families and models of different sizes, that some relations are consistently better known than others but that models differ in the precise facts they know, and differences in the knowledge of base models and their finetuned counterparts.
• 摘要：我们提出了一种估计大型语言模型（LLM）中嵌入的潜在知识的方法。我们利用法学硕士的情境学习（ICL）能力来估计法学硕士对知识库中存储的事实的了解程度。我们的知识估计器避免了以前基于提示的方法的可靠性问题，在概念上更简单，更容易应用，并且我们证明它可以揭示法学硕士中嵌入的更多潜在知识。我们还研究了不同的设计选择如何影响基于 ICL 的知识估计的性能。使用所提出的估计器，我们对各种开源 LLM 的事实知识进行了大规模评估，例如 OPT、Pythia、Llama(2)、Mistral、Gemma 等。维基数据知识库。我们观察到不同模型系列和不同规模模型之间事实知识的差异，某些关系始终比其他关系更广为人知，但模型在它们所知道的精确事实方面有所不同，以及基础模型及其微调对应模型的知识差异。

Title: Stronger Random Baselines for In-Context Learning

• Authors: Gregory Yauney, David Mimno
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.13020
• Pdf URL: https://arxiv.org/pdf/2404.13020
• Copy Paste: [[2404.13020]] Stronger Random Baselines for In-Context Learning(https://arxiv.org/abs/2404.13020)
• Keywords: language model, prompt
• Abstract: Evaluating the in-context learning classification performance of language models poses challenges due to small dataset sizes, extensive prompt-selection using the validation set, and intentionally difficult tasks that lead to near-random performance. The standard random baseline -- the expected accuracy of guessing labels uniformly at random -- is stable when the evaluation set is used only once or when the dataset is large. We account for the common practice of validation set reuse and existing small datasets with a stronger random baseline: the expected maximum accuracy across multiple random classifiers. When choosing the best prompt demonstrations across six quantized language models applied to 16 BIG-bench Lite tasks, more than 20\% of the few-shot results that exceed the standard baseline do not exceed this stronger random baseline. When held-out test sets are available, this stronger baseline is also a better predictor of held-out performance than the standard baseline, avoiding unnecessary test set evaluations. This maximum random baseline provides an easily calculated drop-in replacement for the standard baseline.
• 摘要：由于数据集较小、使用验证集进行广泛的提示选择以及故意困难的任务导致接近随机的性能，评估语言模型的上下文学习分类性能提出了挑战。当评估集仅使用一次或数据集很大时，标准随机基线（均匀随机猜测标签的预期准确性）是稳定的。我们考虑了验证集重用和现有小型数据集的常见做法，具有更强的随机基线：多个随机分类器的预期最大准确度。当在应用于 16 个 BIG-bench Lite 任务的六种量化语言模型中选择最佳提示演示时，超过标准基线的少数结果中有超过 20% 没有超过这个更强的随机基线。当保留测试集可用时，这种更强的基线也比标准基线更好地预测保留性能，从而避免不必要的测试集评估。此最大随机基线提供了标准基线的易于计算的直接替代。

Title: Sample Design Engineering: An Empirical Study of What Makes Good Downstream Fine-Tuning Samples for LLMs

• Authors: Biyang Guo, He Wang, Wenyilin Xiao, Hong Chen, Zhuxin Lee, Songqiao Han, Hailiang Huang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.13033
• Pdf URL: https://arxiv.org/pdf/2404.13033
• Copy Paste: [[2404.13033]] Sample Design Engineering: An Empirical Study of What Makes Good Downstream Fine-Tuning Samples for LLMs(https://arxiv.org/abs/2404.13033)
• Keywords: language model, gpt, llm, prompt, chat
• Abstract: In the burgeoning field of Large Language Models (LLMs) like ChatGPT and LLaMA, Prompt Engineering (PE) is renowned for boosting zero-shot or in-context learning (ICL) through prompt modifications. Yet, the realm of the sample design for downstream fine-tuning, crucial for task-specific LLM adaptation, is largely unexplored. This paper introduces Sample Design Engineering (SDE), a methodical approach to enhancing LLMs' post-tuning performance by refining input, output, and reasoning designs. We conduct a series of in-domain (ID) and out-of-domain (OOD) experiments to assess the impact of various design options on LLMs' downstream performance, revealing several intriguing patterns that hold consistently across different LLMs. Based on these insights, we propose an integrated SDE strategy, combining the most effective options, and validate its consistent superiority over heuristic sample designs in complex downstream tasks like multi-aspect sentiment analysis, event extraction, and nested entity recognition. Additionally, analyses of LLMs' inherent prompt/output perplexity, zero-shot, and ICL abilities illustrate that good PE strategies may not always translate to good SDE strategies. Code available at https://github.com/beyondguo/LLM-Tuning.
• 摘要：在 ChatGPT 和 LLaMA 等新兴的大型语言模型 (LLM) 领域，Prompt Engineering (PE) 因通过即时修改促进零样本或上下文学习 (ICL) 而闻名。然而，对于特定任务的法学硕士适应至关重要的下游微调样本设计领域在很大程度上尚未被探索。本文介绍了样本设计工程 (SDE)，这是一种通过改进输入、输出和推理设计来增强法学硕士后期调整性能的系统方法。我们进行了一系列域内（ID）和域外（OOD）实验，以评估各种设计选项对法学硕士下游性能的影响，揭示了在不同法学硕士中一致存在的几个有趣的模式。基于这些见解，我们提出了一种集成的 SDE 策略，结合了最有效的选项，并在多方面情感分析、事件提取和嵌套实体识别等复杂的下游任务中验证了其相对于启发式样本设计的一致优越性。此外，对法学硕士固有的提示/输出困惑、零样本和 ICL 能力的分析表明，好的 PE 策略可能并不总是转化为好的 SDE 策略。代码可在 https://github.com/beyondguo/LLM-Tuning 获取。