2024-09-13

Title: Zero-Shot Machine-Generated Text Detection Using Mixture of Large Language Models

• Authors: Matthieu Dubois, François Yvon, Pablo Piantanida
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.07615
• Pdf URL: https://arxiv.org/pdf/2409.07615
• Copy Paste: [[2409.07615]] Zero-Shot Machine-Generated Text Detection Using Mixture of Large Language Models(https://arxiv.org/abs/2409.07615)
• Keywords: language model, llm
• Abstract: The dissemination of Large Language Models (LLMs), trained at scale, and endowed with powerful text-generating abilities has vastly increased the threats posed by generative AI technologies by reducing the cost of producing harmful, toxic, faked or forged content. In response, various proposals have been made to automatically discriminate artificially generated from human-written texts, typically framing the problem as a classification problem. Most approaches evaluate an input document by a well-chosen detector LLM, assuming that low-perplexity scores reliably signal machine-made content. As using one single detector can induce brittleness of performance, we instead consider several and derive a new, theoretically grounded approach to combine their respective strengths. Our experiments, using a variety of generator LLMs, suggest that our method effectively increases the robustness of detection.
• 摘要：经过大规模训练并具备强大文本生成能力的大型语言模型 (LLM) 的传播大大增加了生成式 AI 技术所带来的威胁，因为它降低了生成有害、有毒、伪造或伪造内容的成本。为了应对这种情况，人们提出了各种提案，以自动区分人工生成的文本和人类编写的文本，通常将这个问题定义为分类问题。大多数方法都通过精心挑选的检测器 LLM 来评估输入文档，假设低困惑度分数可以可靠地表示机器制作的内容。由于使用单个检测器会导致性能脆弱，因此我们考虑使用多个检测器并得出一种新的、有理论依据的方法来结合它们各自的优势。我们使用各种生成器 LLM 进行实验，结果表明我们的方法有效地提高了检测的稳健性。

Title: SimulBench: Evaluating Language Models with Creative Simulation Tasks

• Authors: Qi Jia, Xiang Yue, Tianyu Zheng, Jie Huang, Bill Yuchen Lin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.07641
• Pdf URL: https://arxiv.org/pdf/2409.07641
• Copy Paste: [[2409.07641]] SimulBench: Evaluating Language Models with Creative Simulation Tasks(https://arxiv.org/abs/2409.07641)
• Keywords: language model, gpt, llm, chat, agent
• Abstract: We introduce SimulBench, a benchmark designed to evaluate large language models (LLMs) across a diverse collection of creative simulation scenarios, such as acting as a Linux terminal or playing text games with users. While these simulation tasks serve as effective measures of an LLM's general intelligence, they are seldom incorporated into existing benchmarks. A major challenge is to develop an evaluation framework for testing different LLMs fairly while preserving the multi-round interactive nature of simulation tasks between users and AI. To tackle this issue, we suggest using a fixed LLM as a user agent to engage with an LLM to collect dialogues first under different tasks. Then, challenging dialogue scripts are extracted for evaluating different target LLMs. To facilitate automatic assessment on \DataName{}, GPT-4 is employed as the evaluator, tasked with reviewing the quality of the final response generated by the target LLMs given multi-turn dialogue scripts. Our comprehensive experiments indicate that these simulation tasks continue to pose a significant challenge with their unique natures and show the gap between proprietary models and the most advanced open LLMs. For example, GPT-4-turbo outperforms LLaMA-3-70b-Chat on 18.55\% more cases.
• 摘要：我们引入了 SimulBench，这是一个基准测试，旨在评估大型语言模型 (LLM) 在各种创造性模拟场景中的表现，例如充当 Linux 终端或与用户一起玩文字游戏。虽然这些模拟任务是衡量 LLM 通用智能的有效指标，但它们很少被纳入现有基准测试。一个主要的挑战是开发一个评估框架来公平地测试不同的 LLM，同时保留模拟任务在用户和 AI 之间的多轮交互性质。为了解决这个问题，我们建议使用固定的 LLM 作为用户代理，与 LLM 交互，首先在不同任务下收集对话。然后，提取具有挑战性的对话脚本来评估不同的目标 LLM。为了便于对 \DataName{} 进行自动评估，GPT-4 被用作评估者，负责审查目标 LLM 在给定多轮对话脚本的情况下生成的最终响应的质量。我们的全面实验表明，这些模拟任务以其独特的性质继续带来重大挑战，并显示出专有模型与最先进的开放 LLM 之间的差距。例如，GPT-4-turbo 在 18.55% 的案例中表现优于 LLaMA-3-70b-Chat。

Title: Experimenting with Legal AI Solutions: The Case of Question-Answering for Access to Justice

• Authors: Jonathan Li, Rohan Bhambhoria, Samuel Dahan, Xiaodan Zhu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.07713
• Pdf URL: https://arxiv.org/pdf/2409.07713
• Copy Paste: [[2409.07713]] Experimenting with Legal AI Solutions: The Case of Question-Answering for Access to Justice(https://arxiv.org/abs/2409.07713)
• Keywords: gpt, retrieval-augmented generation
• Abstract: Generative AI models, such as the GPT and Llama series, have significant potential to assist laypeople in answering legal questions. However, little prior work focuses on the data sourcing, inference, and evaluation of these models in the context of laypersons. To this end, we propose a human-centric legal NLP pipeline, covering data sourcing, inference, and evaluation. We introduce and release a dataset, LegalQA, with real and specific legal questions spanning from employment law to criminal law, corresponding answers written by legal experts, and citations for each answer. We develop an automatic evaluation protocol for this dataset, then show that retrieval-augmented generation from only 850 citations in the train set can match or outperform internet-wide retrieval, despite containing 9 orders of magnitude less data. Finally, we propose future directions for open-sourced efforts, which fall behind closed-sourced models.
• 摘要：生成式 AI 模型（例如 GPT 和 Llama 系列）在帮助普通人回答法律问题方面具有巨大潜力。然而，之前很少有研究关注在普通人背景下对这些模型进行数据获取、推理和评估。为此，我们提出了一个以人为本的法律 NLP 流程，涵盖数据获取、推理和评估。我们推出并发布了一个数据集 LegalQA，其中包含从劳动法到刑法的真实而具体的法律问题、法律专家撰写的相应答案以及每个答案的引文。我们为该数据集开发了一个自动评估协议，然后表明，尽管数据量少了 9 个数量级，但仅从训练集中的 850 个引文进行的检索增强生成就可以匹敌或超越全互联网检索。最后，我们提出了落后于闭源模型的开源工作的未来方向。

Title: Ruri: Japanese General Text Embeddings

• Authors: Hayato Tsukagoshi, Ryohei Sasano
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.07737
• Pdf URL: https://arxiv.org/pdf/2409.07737
• Copy Paste: [[2409.07737]] Ruri: Japanese General Text Embeddings(https://arxiv.org/abs/2409.07737)
• Keywords: llm
• Abstract: We report the development of Ruri, a series of Japanese general text embedding models. While the development of general-purpose text embedding models in English and multilingual contexts has been active in recent years, model development in Japanese remains insufficient. The primary reasons for this are the lack of datasets and the absence of necessary expertise. In this report, we provide a detailed account of the development process of Ruri. Specifically, we discuss the training of embedding models using synthesized datasets generated by LLMs, the construction of the reranker for dataset filtering and knowledge distillation, and the performance evaluation of the resulting general-purpose text embedding models.
• 摘要：我们报告了 Ruri 系列日语通用文本嵌入模型的开发情况。虽然近年来，英语和多语言环境中通用文本嵌入模型的开发一直很活跃，但日语模型开发仍然不足。主要原因是缺乏数据集和缺乏必要的专业知识。在本报告中，我们详细介绍了 Ruri 的开发过程。具体来说，我们讨论了使用 LLM 生成的合成数据集训练嵌入模型、构建用于数据集过滤和知识提炼的重排器，以及对生成的通用文本嵌入模型的性能评估。

Title: Stable Language Model Pre-training by Reducing Embedding Variability

• Authors: Woojin Chung, Jiwoo Hong, Na Min An, James Thorne, Se-Young Yun
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.07787
• Pdf URL: https://arxiv.org/pdf/2409.07787
• Copy Paste: [[2409.07787]] Stable Language Model Pre-training by Reducing Embedding Variability(https://arxiv.org/abs/2409.07787)
• Keywords: language model, gpt
• Abstract: Stable pre-training is essential for achieving better-performing language models. However, tracking pre-training stability by calculating gradient variance at every step is impractical due to the significant computational costs. We explore Token Embedding Variability (TEV) as a simple and efficient proxy for assessing pre-training stability in language models with pre-layer normalization, given that shallower layers are more prone to gradient explosion (section 2.2). Moreover, we propose Multi-head Low-Rank Attention (MLRA) as an architecture to alleviate such instability by limiting the exponential growth of output embedding variance, thereby preventing the gradient explosion (section 3.2). Empirical results on GPT-2 with MLRA demonstrate increased stability and lower perplexity, particularly in deeper models.
• 摘要：稳定的预训练对于实现性能更好的语言模型至关重要。但是，由于计算成本高昂，通过计算每一步的梯度方差来跟踪预训练稳定性是不切实际的。我们探索了 Token 嵌入可变性 (TEV) 作为评估具有预层规范化的语言模型中预训练稳定性的简单有效代理，因为较浅的层更容易出现梯度爆炸（第 2.2 节）。此外，我们提出了多头低秩注意 (MLRA) 作为一种架构，通过限制输出嵌入方差的指数增长来缓解这种不稳定性，从而防止梯度爆炸（第 3.2 节）。使用 MLRA 的 GPT-2 的实证结果表明，稳定性提高，困惑度降低，尤其是在更深的模型中。

Title: Full-text Error Correction for Chinese Speech Recognition with Large Language Model

• Authors: Zhiyuan Tang, Dong Wang, Shen Huang, Shidong Shang
• Subjects: cs.CL, eess.AS
• Abstract URL: https://arxiv.org/abs/2409.07790
• Pdf URL: https://arxiv.org/pdf/2409.07790
• Copy Paste: [[2409.07790]] Full-text Error Correction for Chinese Speech Recognition with Large Language Model(https://arxiv.org/abs/2409.07790)
• Keywords: language model, llm, prompt
• Abstract: Large Language Models (LLMs) have demonstrated substantial potential for error correction in Automatic Speech Recognition (ASR). However, most research focuses on utterances from short-duration speech recordings, which are the predominant form of speech data for supervised ASR training. This paper investigates the effectiveness of LLMs for error correction in full-text generated by ASR systems from longer speech recordings, such as transcripts from podcasts, news broadcasts, and meetings. First, we develop a Chinese dataset for full-text error correction, named ChFT, utilizing a pipeline that involves text-to-speech synthesis, ASR, and error-correction pair extractor. This dataset enables us to correct errors across contexts, including both full-text and segment, and to address a broader range of error types, such as punctuation restoration and inverse text normalization, thus making the correction process comprehensive. Second, we fine-tune a pre-trained LLM on the constructed dataset using a diverse set of prompts and target formats, and evaluate its performance on full-text error correction. Specifically, we design prompts based on full-text and segment, considering various output formats, such as directly corrected text and JSON-based error-correction pairs. Through various test settings, including homogeneous, up-to-date, and hard test sets, we find that the fine-tuned LLMs perform well in the full-text setting with different prompts, each presenting its own strengths and weaknesses. This establishes a promising baseline for further research. The dataset is available on the website.
• 摘要：大型语言模型 (LLM) 已显示出在自动语音识别 (ASR) 中纠错的巨大潜力。然而，大多数研究都集中在短时语音记录中的话语上，这是监督式 ASR 训练的主要语音数据形式。本文研究了 LLM 对 ASR 系统从较长的语音记录（例如播客、新闻广播和会议的记录）生成的全文纠错的有效性。首先，我们开发了一个用于全文纠错的中文数据集，名为 ChFT，利用涉及文本到语音合成、ASR 和纠错对提取器的管道。该数据集使我们能够跨上下文（包括全文和片段）纠正错误，并解决更广泛的错误类型，例如标点符号恢复和逆文本规范化，从而使纠正过程更加全面。其次，我们使用一组不同的提示和目标格式在构建的数据集上微调预训练的 LLM，并评估其在全文纠错方面的表现。具体来说，我们根据全文和片段设计提示，考虑各种输出格式，例如直接校正的文本和基于 JSON 的纠错对。通过各种测试设置，包括同质、最新和硬测试集，我们发现经过微调的 LLM 在具有不同提示的全文设置中表现良好，每个提示都有自己的优点和缺点。这为进一步的研究建立了一个有希望的基础。数据集可在网站上找到。

Title: Learning Rules from KGs Guided by Language Models

• Authors: Zihang Peng, Daria Stepanova, Vinh Thinh Ho, Heike Adel, Alessandra Russo, Simon Ott
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.07869
• Pdf URL: https://arxiv.org/pdf/2409.07869
• Copy Paste: [[2409.07869]] Learning Rules from KGs Guided by Language Models(https://arxiv.org/abs/2409.07869)
• Keywords: language model
• Abstract: Advances in information extraction have enabled the automatic construction of large knowledge graphs (e.g., Yago, Wikidata or Google KG), which are widely used in many applications like semantic search or data analytics. However, due to their semi-automatic construction, KGs are often incomplete. Rule learning methods, concerned with the extraction of frequent patterns from KGs and casting them into rules, can be applied to predict potentially missing facts. A crucial step in this process is rule ranking. Ranking of rules is especially challenging over highly incomplete or biased KGs (e.g., KGs predominantly storing facts about famous people), as in this case biased rules might fit the data best and be ranked at the top based on standard statistical metrics like rule confidence. To address this issue, prior works proposed to rank rules not only relying on the original KG but also facts predicted by a KG embedding model. At the same time, with the recent rise of Language Models (LMs), several works have claimed that LMs can be used as alternative means for KG completion. In this work, our goal is to verify to which extent the exploitation of LMs is helpful for improving the quality of rule learning systems.
• 摘要：信息提取方面的进步使得大型知识图谱（例如 Yago、Wikidata 或 Google KG）的自动构建成为可能，这些图谱广泛应用于语义搜索或数据分析等许多应用中。然而，由于半自动构建，KG 往往是不完整的。规则学习方法涉及从 KG 中提取频繁模式并将其转化为规则，可用于预测可能缺失的事实。此过程中的一个关键步骤是规则排名。规则排名对于高度不完整或有偏差的 KG（例如，主要存储名人事实的 KG）尤其具有挑战性，因为在这种情况下，有偏差的规则可能最适合数据，并根据规则置信度等标准统计指标排在首位。为了解决这个问题，先前的研究提出不仅要依赖原始 KG，还要依赖 KG 嵌入模型预测的事实来对规则进行排名。同时，随着语言模型 (LM) 的兴起，一些研究声称 LM 可以用作 KG 完成的替代方法。在这项工作中，我们的目标是验证 LM 的利用在多大程度上有助于提高规则学习系统的质量。

Title: The CLC-UKET Dataset: Benchmarking Case Outcome Prediction for the UK Employment Tribunal

• Authors: Huiyuan Xie, Felix Steffek, Joana Ribeiro de Faria, Christine Carter, Jonathan Rutherford
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.08098
• Pdf URL: https://arxiv.org/pdf/2409.08098
• Copy Paste: [[2409.08098]] The CLC-UKET Dataset: Benchmarking Case Outcome Prediction for the UK Employment Tribunal(https://arxiv.org/abs/2409.08098)
• Keywords: language model, llm
• Abstract: This paper explores the intersection of technological innovation and access to justice by developing a benchmark for predicting case outcomes in the UK Employment Tribunal (UKET). To address the challenge of extensive manual annotation, the study employs a large language model (LLM) for automatic annotation, resulting in the creation of the CLC-UKET dataset. The dataset consists of approximately 19,000 UKET cases and their metadata. Comprehensive legal annotations cover facts, claims, precedent references, statutory references, case outcomes, reasons and jurisdiction codes. Facilitated by the CLC-UKET data, we examine a multi-class case outcome prediction task in the UKET. Human predictions are collected to establish a performance reference for model comparison. Empirical results from baseline models indicate that finetuned transformer models outperform zero-shot and few-shot LLMs on the UKET prediction task. The performance of zero-shot LLMs can be enhanced by integrating task-related information into few-shot examples. We hope that the CLC-UKET dataset, along with human annotations and empirical findings, can serve as a valuable benchmark for employment-related dispute resolution.
• 摘要：本文通过制定预测英国就业法庭 (UKET) 案件结果的基准，探讨了技术创新与司法公正之间的交集。为了应对大量人工注释的挑战，该研究采用大型语言模型 (LLM) 进行自动注释，从而创建了 CLC-UKET 数据集。该数据集包含大约 19,000 个 UKET 案件及其元数据。全面的法律注释涵盖事实、索赔、先例参考、法定参考、案件结果、理由和管辖权法规。借助 CLC-UKET 数据，我们研究了 UKET 中的多类案件结果预测任务。收集人工预测以建立模型比较的性能参考。基线模型的经验结果表明，微调的 Transformer 模型在 UKET 预测任务上的表现优于零样本和少样本 LLM。通过将与任务相关的信息集成到少样本示例中，可以增强零样本 LLM 的性能。我们希望 CLC-UKET 数据集以及人工注释和实证结果可以成为与就业相关的争议解决的宝贵基准。

Title: WhisperNER: Unified Open Named Entity and Speech Recognition

• Authors: Gil Ayache, Menachem Pirchi, Aviv Navon, Aviv Shamsian, Gill Hetz, Joseph Keshet
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.08107
• Pdf URL: https://arxiv.org/pdf/2409.08107
• Copy Paste: [[2409.08107]] WhisperNER: Unified Open Named Entity and Speech Recognition(https://arxiv.org/abs/2409.08107)
• Keywords: prompt
• Abstract: Integrating named entity recognition (NER) with automatic speech recognition (ASR) can significantly enhance transcription accuracy and informativeness. In this paper, we introduce WhisperNER, a novel model that allows joint speech transcription and entity recognition. WhisperNER supports open-type NER, enabling recognition of diverse and evolving entities at inference. Building on recent advancements in open NER research, we augment a large synthetic dataset with synthetic speech samples. This allows us to train WhisperNER on a large number of examples with diverse NER tags. During training, the model is prompted with NER labels and optimized to output the transcribed utterance along with the corresponding tagged entities. To evaluate WhisperNER, we generate synthetic speech for commonly used NER benchmarks and annotate existing ASR datasets with open NER tags. Our experiments demonstrate that WhisperNER outperforms natural baselines on both out-of-domain open type NER and supervised finetuning.
• 摘要：将命名实体识别 (NER) 与自动语音识别 (ASR) 相结合可以显著提高转录准确性和信息量。在本文中，我们介绍了 WhisperNER，这是一种允许联合语音转录和实体识别的新型模型。WhisperNER 支持开放式 NER，能够在推理时识别各种不断发展的实体。基于开放式 NER 研究的最新进展，我们使用合成语音样本扩充了一个大型合成数据集。这使我们能够在具有各种 NER 标签的大量示例上训练 WhisperNER。在训练期间，该模型会使用 NER 标签提示并进行优化以输出转录的话语以及相应的标记实体。为了评估 WhisperNER，我们为常用的 NER 基准生成合成语音，并使用开放式 NER 标签注释现有的 ASR 数据集。我们的实验表明，WhisperNER 在领域外的开放式 NER 和监督微调方面都优于自然基线。

Title: LLM-POTUS Score: A Framework of Analyzing Presidential Debates with Large Language Models

• Authors: Zhengliang Liu, Yiwei Li, Oleksandra Zolotarevych, Rongwei Yang, Tianming Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.08147
• Pdf URL: https://arxiv.org/pdf/2409.08147
• Copy Paste: [[2409.08147]] LLM-POTUS Score: A Framework of Analyzing Presidential Debates with Large Language Models(https://arxiv.org/abs/2409.08147)
• Keywords: language model, llm
• Abstract: Large language models have demonstrated remarkable capabilities in natural language processing, yet their application to political discourse analysis remains underexplored. This paper introduces a novel approach to evaluating presidential debate performances using LLMs, addressing the longstanding challenge of objectively assessing debate outcomes. We propose a framework that analyzes candidates' "Policies, Persona, and Perspective" (3P) and how they resonate with the "Interests, Ideologies, and Identity" (3I) of four key audience groups: voters, businesses, donors, and politicians. Our method employs large language models to generate the LLM-POTUS Score, a quantitative measure of debate performance based on the alignment between 3P and 3I. We apply this framework to analyze transcripts from recent U.S. presidential debates, demonstrating its ability to provide nuanced, multi-dimensional assessments of candidate performances. Our results reveal insights into the effectiveness of different debating strategies and their impact on various audience segments. This study not only offers a new tool for political analysis but also explores the potential and limitations of using LLMs as impartial judges in complex social contexts. In addition, this framework provides individual citizens with an independent tool to evaluate presidential debate performances, which enhances democratic engagement and reduces reliance on potentially biased media interpretations and institutional influence, thereby strengthening the foundation of informed civic participation.
• 摘要：大型语言模型在自然语言处理方面表现出了卓越的能力，但它们在政治话语分析中的应用仍未得到充分探索。本文介绍了一种使用 LLM 评估总统辩论表现的新方法，解决了客观评估辩论结果的长期挑战。我们提出了一个框架，用于分析候选人的“政策、角色和观点”（3P）以及它们如何与四个主要受众群体的“利益、意识形态和身份”（3I）产生共鸣：选民、企业、捐助者和政客。我们的方法采用大型语言模型来生成 LLM-POTUS 分数，这是一种基于 3P 和 3I 之间一致性的辩论表现的定量指标。我们应用该框架分析了最近美国总统辩论的记录，证明了它能够对候选人的表现进行细致入微、多维度的评估。我们的结果揭示了不同辩论策略的有效性及其对不同受众群体的影响。本研究不仅为政治分析提供了新工具，还探讨了在复杂社会背景下使用法学硕士作为公正法官的潜力和局限性。此外，该框架为个人公民提供了一个独立的工具来评估总统辩论的表现，从而增强了民主参与，减少了对可能存在偏见的媒体解释和机构影响的依赖，从而加强了知情公民参与的基础。

Title: On the Role of Context in Reading Time Prediction

• Authors: Andreas Opedal, Eleanor Chodroff, Ryan Cotterell, Ethan Gotlieb Wilcox
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.08160
• Pdf URL: https://arxiv.org/pdf/2409.08160
• Copy Paste: [[2409.08160]] On the Role of Context in Reading Time Prediction(https://arxiv.org/abs/2409.08160)
• Keywords: language model
• Abstract: We present a new perspective on how readers integrate context during real-time language comprehension. Our proposals build on surprisal theory, which posits that the processing effort of a linguistic unit (e.g., a word) is an affine function of its in-context information content. We first observe that surprisal is only one out of many potential ways that a contextual predictor can be derived from a language model. Another one is the pointwise mutual information (PMI) between a unit and its context, which turns out to yield the same predictive power as surprisal when controlling for unigram frequency. Moreover, both PMI and surprisal are correlated with frequency. This means that neither PMI nor surprisal contains information about context alone. In response to this, we propose a technique where we project surprisal onto the orthogonal complement of frequency, yielding a new contextual predictor that is uncorrelated with frequency. Our experiments show that the proportion of variance in reading times explained by context is a lot smaller when context is represented by the orthogonalized predictor. From an interpretability standpoint, this indicates that previous studies may have overstated the role that context has in predicting reading times.
• 摘要：我们提出了一种新的视角，探讨读者如何在实时语言理解过程中整合语境。我们的提议建立在惊奇理论的基础上，该理论认为语言单位（例如单词）的处理工作量是其语境信息内容的仿射函数。我们首先观察到，惊奇只是从语言模型中得出语境预测因子的众多潜在方法之一。另一种方法是单元与其语境之间的逐点互信息 (PMI)，当控制单元频率时，它产生与惊奇相同的预测能力。此外，PMI 和惊奇都与频率相关。这意味着 PMI 和惊奇都不包含有关语境的信息。针对此问题，我们提出了一种技术，将惊奇投射到频率的正交补上，从而产生一个与频率不相关的新语境预测因子。我们的实验表明，当上下文由正交化预测因子表示时，上下文解释的阅读时间方差比例要小得多。从可解释性的角度来看，这表明以前的研究可能夸大了上下文在预测阅读时间方面的作用。

Title: Fine-tuning Large Language Models for Entity Matching

• Authors: Aaron Steiner, Ralph Peeters, Christian Bizer
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.08185
• Pdf URL: https://arxiv.org/pdf/2409.08185
• Copy Paste: [[2409.08185]] Fine-tuning Large Language Models for Entity Matching(https://arxiv.org/abs/2409.08185)
• Keywords: language model, gpt, llm, prompt
• Abstract: Generative large language models (LLMs) are a promising alternative to pre-trained language models for entity matching due to their high zero-shot performance and their ability to generalize to unseen entities. Existing research on using LLMs for entity matching has focused on prompt engineering and in-context learning. This paper explores the potential of fine-tuning LLMs for entity matching. We analyze fine-tuning along two dimensions: 1) The representation of training examples, where we experiment with adding different types of LLM-generated explanations to the training set, and 2) the selection and generation of training examples using LLMs. In addition to the matching performance on the source dataset, we investigate how fine-tuning affects the model's ability to generalize to other in-domain datasets as well as across topical domains. Our experiments show that fine-tuning significantly improves the performance of the smaller models while the results for the larger models are mixed. Fine-tuning also improves the generalization to in-domain datasets while hurting cross-domain transfer. We show that adding structured explanations to the training set has a positive impact on the performance of three out of four LLMs, while the proposed example selection and generation methods only improve the performance of Llama 3.1 8B while decreasing the performance of GPT-4o Mini.
• 摘要：生成式大型语言模型 (LLM) 具有较高的零样本性能和泛化到未见实体的能力，是用于实体匹配的预训练语言模型的有前途的替代方案。现有的使用 LLM 进行实体匹配的研究主要集中在即时工程和上下文学习上。本文探讨了微调 LLM 进行实体匹配的潜力。我们从两个维度分析微调：1) 训练示例的表示，我们尝试将不同类型的 LLM 生成的解释添加到训练集中，2) 使用 LLM 选择和生成训练示例。除了源数据集上的匹配性能外，我们还研究了微调如何影响模型泛化到其他域内数据集以及跨主题域的能力。我们的实验表明，微调显著提高了较小模型的性能，而较大模型的结果则好坏参半。微调还提高了对域内数据集的泛化能力，但会损害跨域传输。我们表明，在训练集中添加结构化解释对四个 LLM 中的三个的性能有积极影响，而提出的示例选择和生成方法仅提高了 Llama 3.1 8B 的性能，同时降低了 GPT-4o Mini 的性能。

Title: AudioBERT: Audio Knowledge Augmented Language Model

• Authors: Hyunjong Ok, Suho Yoo, Jaeho Lee
• Subjects: cs.CL, cs.AI, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2409.08199
• Pdf URL: https://arxiv.org/pdf/2409.08199
• Copy Paste: [[2409.08199]] AudioBERT: Audio Knowledge Augmented Language Model(https://arxiv.org/abs/2409.08199)
• Keywords: language model, prompt
• Abstract: Recent studies have identified that language models, pretrained on text-only datasets, often lack elementary visual knowledge, \textit{e.g.,} colors of everyday objects. Motivated by this observation, we ask whether a similar shortcoming exists in terms of the \textit{auditory} knowledge. To answer this question, we construct a new dataset called AuditoryBench, which consists of two novel tasks for evaluating auditory knowledge. Based on our analysis using the benchmark, we find that language models also suffer from a severe lack of auditory knowledge. To address this limitation, we propose AudioBERT, a novel method to augment the auditory knowledge of BERT through a retrieval-based approach. First, we detect auditory knowledge spans in prompts to query our retrieval model efficiently. Then, we inject audio knowledge into BERT and switch on low-rank adaptation for effective adaptation when audio knowledge is required. Our experiments demonstrate that AudioBERT is quite effective, achieving superior performance on the AuditoryBench. The dataset and code are available at \bulurl{this https URL}.
• 摘要：最近的研究发现，在纯文本数据集上进行预训练的语言模型通常缺乏基本的视觉知识，例如日常物体的颜色。受此观察的启发，我们想知道在听觉知识方面是否存在类似的缺陷。为了回答这个问题，我们构建了一个名为 AuditoryBench 的新数据集，它包含两个用于评估听觉知识的新任务。根据我们使用基准进行的分析，我们发现语言模型也严重缺乏听觉知识。为了解决这一限制，我们提出了 AudioBERT，这是一种通过基于检索的方法来增强 BERT 听觉知识的新方法。首先，我们检测提示中的听觉知识跨度，以有效地查询我们的检索模型。然后，我们将音频知识注入 BERT，并在需要音频知识时打开低秩自适应以实现有效自适应。我们的实验表明，AudioBERT 非常有效，在 AuditoryBench 上取得了卓越的表现。数据集和代码可在 \bulurl{此 https URL} 处获得。

Title: Source2Synth: Synthetic Data Generation and Curation Grounded in Real Data Sources

• Authors: Alisia Lupidi, Carlos Gemmell, Nicola Cancedda, Jane Dwivedi-Yu, Jason Weston, Jakob Foerster, Roberta Raileanu, Maria Lomeli
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.08239
• Pdf URL: https://arxiv.org/pdf/2409.08239
• Copy Paste: [[2409.08239]] Source2Synth: Synthetic Data Generation and Curation Grounded in Real Data Sources(https://arxiv.org/abs/2409.08239)
• Keywords: language model, llm
• Abstract: Large Language Models still struggle in challenging scenarios that leverage structured data, complex reasoning, or tool usage. In this paper, we propose Source2Synth: a new method that can be used for teaching LLMs new skills without relying on costly human annotations. Source2Synth takes as input a custom data source and produces synthetic data points with intermediate reasoning steps grounded in real-world sources. Source2Synth improves the dataset quality by discarding low-quality generations based on their answerability. We demonstrate the generality of this approach by applying it to two challenging domains: we test reasoning abilities in multi-hop question answering (MHQA), and tool usage in tabular question answering (TQA). Our method improves performance by 25.51% for TQA on WikiSQL and 22.57% for MHQA on HotPotQA compared to the fine-tuned baselines.
• 摘要：大型语言模型在利用结构化数据、复杂推理或工具使用的具有挑战性的场景中仍然举步维艰。在本文中，我们提出了 Source2Synth：一种可用于教授 LLM 新技能而无需依赖昂贵的人工注释的新方法。Source2Synth 将自定义数据源作为输入，并生成合成数据点，其中的中间推理步骤基于现实世界的来源。Source2Synth 通过根据可回答性丢弃低质量的生成来提高数据集质量。我们通过将该方法应用于两个具有挑战性的领域来证明其通用性：我们测试多跳问答 (MHQA) 中的推理能力和表格问答 (TQA) 中的工具使用情况。与微调后的基线相比，我们的方法将 WikiSQL 上的 TQA 的性能提高了 25.51%，将 HotPotQA 上的 MHQA 的性能提高了 22.57%。