2024-09-11

Title: A Small Claims Court for the NLP: Judging Legal Text Classification Strategies With Small Datasets

• Authors: Mariana Yukari Noguti, Edduardo Vellasques, Luiz Eduardo Soares Oliveira
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.05972
• Pdf URL: https://arxiv.org/pdf/2409.05972
• Copy Paste: [[2409.05972]] A Small Claims Court for the NLP: Judging Legal Text Classification Strategies With Small Datasets(https://arxiv.org/abs/2409.05972)
• Keywords: language model
• Abstract: Recent advances in language modelling has significantly decreased the need of labelled data in text classification tasks. Transformer-based models, pre-trained on unlabeled data, can outmatch the performance of models trained from scratch for each task. However, the amount of labelled data need to fine-tune such type of model is still considerably high for domains requiring expert-level annotators, like the legal domain. This paper investigates the best strategies for optimizing the use of a small labeled dataset and large amounts of unlabeled data and perform a classification task in the legal area with 50 predefined topics. More specifically, we use the records of demands to a Brazilian Public Prosecutor's Office aiming to assign the descriptions in one of the subjects, which currently demands deep legal knowledge for manual filling. The task of optimizing the performance of classifiers in this scenario is especially challenging, given the low amount of resources available regarding the Portuguese language, especially in the legal domain. Our results demonstrate that classic supervised models such as logistic regression and SVM and the ensembles random forest and gradient boosting achieve better performance along with embeddings extracted with word2vec when compared to BERT language model. The latter demonstrates superior performance in association with the architecture of the model itself as a classifier, having surpassed all previous models in that regard. The best result was obtained with Unsupervised Data Augmentation (UDA), which jointly uses BERT, data augmentation, and strategies of semi-supervised learning, with an accuracy of 80.7% in the aforementioned task.
• 摘要：语言建模的最新进展大大减少了文本分类任务中对标记数据的需求。基于 Transformer 的模型在未标记数据上进行了预训练，其性能可以超越针对每项任务从头开始训练的模型。但是，对于需要专家级注释者的领域（如法律领域），微调此类模型所需的标记数据量仍然相当高。本文研究了优化使用小型标记数据集和大量未标记数据的最佳策略，并使用 50 个预定义主题执行法律领域的分类任务。更具体地说，我们使用巴西检察官办公室的要求记录，旨在分配其中一个主题的描述，该主题目前需要深厚的法律知识才能手动填写。鉴于葡萄牙语（尤其是在法律领域）的可用资源很少，在这种情况下优化分类器性能的任务尤其具有挑战性。我们的结果表明，与 BERT 语言模型相比，逻辑回归和 SVM 等经典监督模型以及随机森林和梯度提升等集成模型以及使用 word2vec 提取的嵌入取得了更好的性能。后者在与模型本身作为分类器的架构相关的方面表现出色，在这方面超越了所有以前的模型。使用无监督数据增强 (UDA) 获得了最佳结果，它联合使用了 BERT、数据增强和半监督学习策略，在上述任务中的准确率为 80.7%。

Title: AI for Mathematics Mathematical Formalized Problem Solving and Theorem Proving in Different Fields in Lean4

• Authors: Xichen Tang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.05977
• Pdf URL: https://arxiv.org/pdf/2409.05977
• Copy Paste: [[2409.05977]] AI for Mathematics Mathematical Formalized Problem Solving and Theorem Proving in Different Fields in Lean4(https://arxiv.org/abs/2409.05977)
• Keywords: language model, llm
• Abstract: Using computerized verifiable formal languages like Lean 4 to prove mathematical theorems has a significant impact on mathematical formalization. Lean 4 offers prominent potential for advancing mathematical reasoning. However, existing efforts are limited to mathematical formalization languages in substantial online corpora and are dedicated to keeping pace with rapidly evolving languages. To bridge the gap between the traditional and computerized proof, my approach to formalizing theorem proving involves generating formal steps and complete proofs using Large Language Models (LLMs) based on Natural Language (NL) proofs. The method is to introduce the basic structure and tactics in general, determine how AI can assist the mathematical formalization process to improve its performance, and give examples of solving problems in Lean 4 comparing to NL, mainly in IMO, and a sample theorem proving in abstract algebra.
• 摘要：使用像 Lean 4 这样的计算机可验证形式语言来证明数学定理对数学形式化有着重大影响。Lean 4 为推进数学推理提供了巨大的潜力。然而，现有的努力仅限于大量在线语料库中的数学形式化语言，并致力于跟上快速发展的语言的步伐。为了弥合传统证明和计算机化证明之间的差距，我对形式化定理证明的方法包括使用基于自然语言 (NL) 证明的大型语言模型 (LLM) 生成形式步骤和完整证明。该方法是介绍基本结构和一般策略，确定人工智能如何协助数学形式化过程提高其性能，并给出与 NL 相比在 Lean 4 中解决问题的例子，主要是在 IMO 中，以及抽象代数中的示例定理证明。

Title: TransformerRanker: A Tool for Efficiently Finding the Best-Suited Language Models for Downstream Classification Tasks

• Authors: Lukas Garbas, Max Ploner, Alan Akbik
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.05997
• Pdf URL: https://arxiv.org/pdf/2409.05997
• Copy Paste: [[2409.05997]] TransformerRanker: A Tool for Efficiently Finding the Best-Suited Language Models for Downstream Classification Tasks(https://arxiv.org/abs/2409.05997)
• Keywords: language model
• Abstract: Classification tasks in NLP are typically addressed by selecting a pre-trained language model (PLM) from a model hub, and fine-tuning it for the task at hand. However, given the very large number of PLMs that are currently available, a practical challenge is to determine which of them will perform best for a specific downstream task. With this paper, we introduce TransformerRanker, a lightweight library that efficiently ranks PLMs for classification tasks without the need for computationally costly fine-tuning. Our library implements current approaches for transferability estimation (LogME, H-Score, kNN), in combination with layer aggregation options, which we empirically showed to yield state-of-the-art rankings of PLMs (Garbas et al., 2024). We designed the interface to be lightweight and easy to use, allowing users to directly connect to the HuggingFace Transformers and Dataset libraries. Users need only select a downstream classification task and a list of PLMs to create a ranking of likely best-suited PLMs for their task. We make TransformerRanker available as a pip-installable open-source library this https URL.
• 摘要：NLP 中的分类任务通常通过从模型中心选择预训练语言模型 (PLM) 并针对手头的任务对其进行微调来解决。然而，鉴于目前可用的 PLM 数量非常庞大，一个实际挑战是确定其中哪一个最适合特定的下游任务。在本文中，我们介绍了 TransformerRanker，这是一个轻量级库，可以有效地对分类任务的 PLM 进行排名，而无需进行计算成本高昂的微调。我们的库实现了当前的可迁移性估计方法（LogME、H-Score、kNN），并结合了层聚合选项，我们通过经验证明这些方法可以产生最先进的 PLM 排名（Garbas 等人，2024 年）。我们将界面设计得轻量级且易于使用，允许用户直接连接到 HuggingFace Transformers 和 Dataset 库。用户只需选择一个下游分类任务和一个 PLM 列表，即可创建最适合其任务的 PLM 排名。我们将 TransformerRanker 作为可通过 pip 安装的开源库通过 https URL 提供。

Title: Improved Visually Prompted Keyword Localisation in Real Low-Resource Settings

• Authors: Leanne Nortje, Dan Oneata, Herman Kamper
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06013
• Pdf URL: https://arxiv.org/pdf/2409.06013
• Copy Paste: [[2409.06013]] Improved Visually Prompted Keyword Localisation in Real Low-Resource Settings(https://arxiv.org/abs/2409.06013)
• Keywords: prompt
• Abstract: Given an image query, visually prompted keyword localisation (VPKL) aims to find occurrences of the depicted word in a speech collection. This can be useful when transcriptions are not available for a low-resource language (e.g. if it is unwritten). Previous work showed that VPKL can be performed with a visually grounded speech model trained on paired images and unlabelled speech. But all experiments were done on English. Moreover, transcriptions were used to get positive and negative pairs for the contrastive loss. This paper introduces a few-shot learning scheme to mine pairs automatically without transcriptions. On English, this results in only a small drop in performance. We also - for the first time - consider VPKL on a real low-resource language, Yoruba. While scores are reasonable, here we see a bigger drop in performance compared to using ground truth pairs because the mining is less accurate in Yoruba.
• 摘要：给定一个图像查询，视觉提示关键字定位 (VPKL) 旨在在语音集合中找到所描述单词的出现位置。当无法为低资源语言（例如，如果它不是书面语）提供转录时，这可能很有用。之前的研究表明，VPKL 可以使用在配对图像和未标记语音上训练的视觉基础语音模型来执行。但所有实验都是用英语进行的。此外，转录用于获得对比损失的正对和负对。本文介绍了一种几次学习方案，无需转录即可自动挖掘对。对于英语，这只会导致性能略有下降。我们还首次考虑在一种真正的低资源语言约鲁巴语上使用 VPKL。虽然分数合理，但与使​​用基本事实对相比，我们看到性能下降幅度更大，因为在约鲁巴语中挖掘的准确性较低。

Title: Identifying the sources of ideological bias in GPT models through linguistic variation in output

• Authors: Christina Walker, Joan C. Timoneda
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.06043
• Pdf URL: https://arxiv.org/pdf/2409.06043
• Copy Paste: [[2409.06043]] Identifying the sources of ideological bias in GPT models through linguistic variation in output(https://arxiv.org/abs/2409.06043)
• Keywords: gpt
• Abstract: Extant work shows that generative AI models such as GPT-3.5 and 4 perpetuate social stereotypes and biases. One concerning but less explored source of bias is ideology. Do GPT models take ideological stances on politically sensitive topics? In this article, we provide an original approach to identifying ideological bias in generative models, showing that bias can stem from both the training data and the filtering algorithm. We leverage linguistic variation in countries with contrasting political attitudes to evaluate bias in average GPT responses to sensitive political topics in those languages. First, we find that GPT output is more conservative in languages that map well onto conservative societies (i.e., Polish), and more liberal in languages used uniquely in liberal societies (i.e., Swedish). This result provides strong evidence of training data bias in GPT models. Second, differences across languages observed in GPT-3.5 persist in GPT-4, even though GPT-4 is significantly more liberal due to OpenAI's filtering policy. Our main takeaway is that generative model training must focus on high-quality, curated datasets to reduce bias, even if it entails a compromise in training data size. Filtering responses after training only introduces new biases and does not remove the underlying training biases.
• 摘要：现有的研究表明，GPT-3.5 和 4 等生成式 AI 模型延续了社会刻板印象和偏见。一个令人担忧但较少被探究的偏见来源是意识形态。GPT 模型是否在政治敏感话题上采取意识形态立场？在本文中，我们提供了一种识别生成模型中意识形态偏见的原创方法，表明偏见可能源于训练数据和过滤算法。我们利用政治态度形成鲜明对比的国家的语言差异来评估这些语言中 GPT 对敏感政治话题的平均反应的偏见。首先，我们发现 GPT 输出在与保守社会很好地映射的语言（即波兰语）中更为保守，而在自由社会独有的语言（即瑞典语）中更为自由。这一结果为 GPT 模型中的训练数据偏见提供了强有力的证据。其次，尽管由于 OpenAI 的过滤策略，GPT-4 明显更加自由，但 GPT-3.5 中观察到的语言差异在 GPT-4 中仍然存在。我们的主要结论是，生成模型训练必须专注于高质量、精心挑选的数据集，以减少偏差，即使这需要在训练数据大小上做出妥协。训练后过滤响应只会引入新的偏差，而不会消除潜在的训练偏差。

Title: DetoxBench: Benchmarking Large Language Models for Multitask Fraud & Abuse Detection

• Authors: Joymallya Chakraborty, Wei Xia, Anirban Majumder, Dan Ma, Walid Chaabene, Naveed Janvekar
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.06072
• Pdf URL: https://arxiv.org/pdf/2409.06072
• Copy Paste: [[2409.06072]] DetoxBench: Benchmarking Large Language Models for Multitask Fraud & Abuse Detection(https://arxiv.org/abs/2409.06072)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have demonstrated remarkable capabilities in natural language processing tasks. However, their practical application in high-stake domains, such as fraud and abuse detection, remains an area that requires further exploration. The existing applications often narrowly focus on specific tasks like toxicity or hate speech detection. In this paper, we present a comprehensive benchmark suite designed to assess the performance of LLMs in identifying and mitigating fraudulent and abusive language across various real-world scenarios. Our benchmark encompasses a diverse set of tasks, including detecting spam emails, hate speech, misogynistic language, and more. We evaluated several state-of-the-art LLMs, including models from Anthropic, Mistral AI, and the AI21 family, to provide a comprehensive assessment of their capabilities in this critical domain. The results indicate that while LLMs exhibit proficient baseline performance in individual fraud and abuse detection tasks, their performance varies considerably across tasks, particularly struggling with tasks that demand nuanced pragmatic reasoning, such as identifying diverse forms of misogynistic language. These findings have important implications for the responsible development and deployment of LLMs in high-risk applications. Our benchmark suite can serve as a tool for researchers and practitioners to systematically evaluate LLMs for multi-task fraud detection and drive the creation of more robust, trustworthy, and ethically-aligned systems for fraud and abuse detection.
• 摘要：大型语言模型 (LLM) 在自然语言处理任务中表现出了卓越的能力。然而，它们在高风险领域（例如欺诈和滥用检测）的实际应用仍然是一个需要进一步探索的领域。现有的应用程序通常只关注特定任务，例如毒性或仇恨言论检测。在本文中，我们提出了一个全面的基准测试套件，旨在评估 LLM 在各种现实场景中识别和缓解欺诈和辱骂性语言方面的表现。我们的基准测试涵盖了一系列不同的任务，包括检测垃圾邮件、仇恨言论、厌恶女性的语言等。我们评估了几种最先进的 LLM，包括来自 Anthropic、Mistral AI 和 AI21 系列的模型，以全面评估它们在这个关键领域的能力。结果表明，虽然 LLM 在单个欺诈和滥用检测任务中表现出熟练的基线性能，但它们在不同任务中的表现差异很大，尤其是在处理需要细致入微的务实推理的任务时，例如识别各种形式的厌恶女性的语言。这些发现对于负责任地开发和部署 LLM 以应对高风险应用具有重要意义。我们的基准套件可以作为研究人员和从业人员的工具，系统地评估 LLM 的多任务欺诈检测，并推动创建更强大、更值得信赖、更符合道德规范的欺诈和滥用检测系统。

Title: ClarQ-LLM: A Benchmark for Models Clarifying and Requesting Information in Task-Oriented Dialog

• Authors: Yujian Gan, Changling Li, Jinxia Xie, Luou Wen, Matthew Purver, Massimo Poesio
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.06097
• Pdf URL: https://arxiv.org/pdf/2409.06097
• Copy Paste: [[2409.06097]] ClarQ-LLM: A Benchmark for Models Clarifying and Requesting Information in Task-Oriented Dialog(https://arxiv.org/abs/2409.06097)
• Keywords: llm, agent
• Abstract: We introduce ClarQ-LLM, an evaluation framework consisting of bilingual English-Chinese conversation tasks, conversational agents and evaluation metrics, designed to serve as a strong benchmark for assessing agents' ability to ask clarification questions in task-oriented dialogues. The benchmark includes 31 different task types, each with 10 unique dialogue scenarios between information seeker and provider agents. The scenarios require the seeker to ask questions to resolve uncertainty and gather necessary information to complete tasks. Unlike traditional benchmarks that evaluate agents based on fixed dialogue content, ClarQ-LLM includes a provider conversational agent to replicate the original human provider in the benchmark. This allows both current and future seeker agents to test their ability to complete information gathering tasks through dialogue by directly interacting with our provider agent. In tests, LLAMA3.1 405B seeker agent managed a maximum success rate of only 60.05\%, showing that ClarQ-LLM presents a strong challenge for future research.
• 摘要：我们引入了 ClarQ-LLM，这是一个由英汉双语对话任务、对话代理和评估指标组成的评估框架，旨在作为评估代理在任务导向型对话中提出澄清问题的能力的强大基准。该基准包括 31 种不同的任务类型，每种任务类型都有 10 个信息搜索者和提供者代理之间的独特对话场景。这些场景要求搜索者提出问题以解决不确定性并收集完成任务所需的信息。与基于固定对话内容评估代理的传统基准不同，ClarQ-LLM 包含一个提供者对话代理，以在基准中复制原始人类提供者。这使得当前和未来的搜索代理都可以通过直接与我们的提供者代理交互来测试他们通过对话完成信息收集任务的能力。在测试中，LLAMA3.1 405B 搜索代理的最大成功率仅为 60.05%，这表明 ClarQ-LLM 对未来的研究提出了严峻的挑战。

Title: Doppelg\"anger's Watch: A Split Objective Approach to Large Language Models

• Authors: Shervin Ghasemlou, Ashish Katiyar, Aparajita Saraf, Seungwhan Moon, Mangesh Pujari, Pinar Donmez, Babak Damavandi, Anuj Kumar
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.06107
• Pdf URL: https://arxiv.org/pdf/2409.06107
• Copy Paste: [[2409.06107]] Doppelg\"anger's Watch: A Split Objective Approach to Large Language Models(https://arxiv.org/abs/2409.06107)
• Keywords: language model
• Abstract: In this paper, we investigate the problem of "generation supervision" in large language models, and present a novel bicameral architecture to separate supervision signals from their core capability, helpfulness. Doppelgänger, a new module parallel to the underlying language model, supervises the generation of each token, and learns to concurrently predict the supervision score(s) of the sequences up to and including each token. In this work, we present the theoretical findings, and leave the report on experimental results to a forthcoming publication.
• 摘要：在本文中，我们研究了大型语言模型中的“生成监督”问题，并提出了一种新颖的双腔结构，将监督信号与其核心功能（有用性）分开。Doppelgänger 是一个与底层语言模型并行的新模块，它监督每个标记的生成，并学习同​​时预测每个标记之前和之后的序列的监督分数。在这项工作中，我们介绍了理论发现，并将实验结果报告留待即将发表的出版物。

Title: Accelerating Large Language Model Pretraining via LFR Pedagogy: Learn, Focus, and Review

• Authors: Neha Prakriya, Jui-Nan Yen, Cho-Jui Hsieh, Jason Cong
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.06131
• Pdf URL: https://arxiv.org/pdf/2409.06131
• Copy Paste: [[2409.06131]] Accelerating Large Language Model Pretraining via LFR Pedagogy: Learn, Focus, and Review(https://arxiv.org/abs/2409.06131)
• Keywords: language model, gpt, llm
• Abstract: Large Language Model (LLM) pretraining traditionally relies on autoregressive language modeling on randomly sampled data blocks from web-scale datasets. We take inspiration from human learning techniques like spaced repetition to hypothesize that random data sampling for LLMs leads to high training cost and low quality models which tend to forget data. In order to effectively commit web-scale information to long-term memory, we propose the LFR (Learn, Focus, and Review) pedagogy, a new dynamic training paradigm which focuses and repeatedly reviews complex data blocks at systematic intervals based on the model's learning pace and progress. LFR records the model perplexities for different data blocks and frequently revisits blocks with higher perplexity which are more likely to be forgotten. We pretrain the GPT-2 models (124M - 1.5B) from scratch on the OpenWebText dataset using LFR. We test on downstream tasks from the language modeling, question answering, translation, and problem solving domains to achieve consistently lower perplexity and higher accuracy than the baseline OpenAI models, while obtaining a 20x pretraining speed-up.
• 摘要：大型语言模型 (LLM) 预训练传统上依赖于对从网络规模数据集中随机采样的数据块进行自回归语言建模。我们从间隔重复等人类学习技术中汲取灵感，假设 LLM 的随机数据采样会导致高训练成本和低质量模型，而这些模型往往会忘记数据。为了有效地将网络规模的信息提交到长期记忆中，我们提出了 LFR（学习、专注和复习）教学法，这是一种新的动态训练范式，它根据模型的学习速度和进度，以系统的间隔关注和反复复习复杂的数据块。LFR 记录不同数据块的模型困惑度，并经常重新访问困惑度更高、更容易被遗忘的块。我们使用 LFR 在 OpenWebText 数据集上从头开始预训练 GPT-2 模型（124M - 1.5B）。我们对语言建模、问答、翻译和问题解决领域的下游任务进行了测试，以达到比基线 OpenAI 模型更低的困惑度和更高的准确性，同时获得 20 倍的预训练速度提升。

Title: Deep Learning and Large Language Models for Audio and Text Analysis in Predicting Suicidal Acts in Chinese Psychological Support Hotlines

• Authors: Yining Chen, Jianqiang Li, Changwei Song, Qing Zhao, Yongsheng Tong, Guanghui Fu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.06164
• Pdf URL: https://arxiv.org/pdf/2409.06164
• Copy Paste: [[2409.06164]] Deep Learning and Large Language Models for Audio and Text Analysis in Predicting Suicidal Acts in Chinese Psychological Support Hotlines(https://arxiv.org/abs/2409.06164)
• Keywords: language model, llm, prompt
• Abstract: Suicide is a pressing global issue, demanding urgent and effective preventive interventions. Among the various strategies in place, psychological support hotlines had proved as a potent intervention method. Approximately two million people in China attempt suicide annually, with many individuals making multiple attempts. Prompt identification and intervention for high-risk individuals are crucial to preventing tragedies. With the rapid advancement of artificial intelligence (AI), especially the development of large-scale language models (LLMs), new technological tools have been introduced to the field of mental health. This study included 1284 subjects, and was designed to validate whether deep learning models and LLMs, using audio and transcribed text from support hotlines, can effectively predict suicide risk. We proposed a simple LLM-based pipeline that first summarizes transcribed text from approximately one hour of speech to extract key features, and then predict suicidial bahaviours in the future. We compared our LLM-based method with the traditional manual scale approach in a clinical setting and with five advanced deep learning models. Surprisingly, the proposed simple LLM pipeline achieved strong performance on a test set of 46 subjects, with an F1 score of 76\% when combined with manual scale rating. This is 7\% higher than the best speech-based deep learning models and represents a 27.82\% point improvement in F1 score compared to using the manual scale apporach alone. Our study explores new applications of LLMs and demonstrates their potential for future use in suicide prevention efforts.
• 摘要：自杀是一个紧迫的全球性问题，需要采取紧急有效的预防干预措施。在现有的各种策略中，心理支持热线已被证明是一种有效的干预方法。中国每年约有 200 万人自杀，许多人多次尝试。及时识别和干预高风险人群对于防止悲剧发生至关重要。随着人工智能 (AI) 的快速发展，尤其是大规模语言模型 (LLM) 的发展，新的技术工具已被引入心理健康领域。这项研究包括 1284 名受试者，旨在验证深度学习模型和 LLM（使用支持热线的音频和转录文本）是否能有效预测自杀风险。我们提出了一种基于 LLM 的简单流程，首先总结大约一小时的语音转录文本以提取关键特征，然后预测未来的自杀行为。我们在临床环境中将基于 LLM 的方法与传统的手动量表方法以及五种先进的深度学习模型进行了比较。令人惊讶的是，所提出的简单 LLM 管道在 46 个受试者的测试集上取得了出色的表现，与手动量表评分相结合时 F1 得分为 76%。这比最好的基于语音的深度学习模型高出 7%，与单独使用手动量表方法相比，F1 得分提高了 27.82%。我们的研究探索了 LLM 的新应用，并展示了它们未来在自杀预防工作中的潜力。

Title: Larger Language Models Don't Care How You Think: Why Chain-of-Thought Prompting Fails in Subjective Tasks

• Authors: Georgios Chochlakis, Niyantha Maruthu Pandiyan, Kristina Lerman, Shrikanth Narayanan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.06173
• Pdf URL: https://arxiv.org/pdf/2409.06173
• Copy Paste: [[2409.06173]] Larger Language Models Don't Care How You Think: Why Chain-of-Thought Prompting Fails in Subjective Tasks(https://arxiv.org/abs/2409.06173)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: In-Context Learning (ICL) in Large Language Models (LLM) has emerged as the dominant technique for performing natural language tasks, as it does not require updating the model parameters with gradient-based methods. ICL promises to "adapt" the LLM to perform the present task at a competitive or state-of-the-art level at a fraction of the computational cost. ICL can be augmented by incorporating the reasoning process to arrive at the final label explicitly in the prompt, a technique called Chain-of-Thought (CoT) prompting. However, recent work has found that ICL relies mostly on the retrieval of task priors and less so on "learning" to perform tasks, especially for complex subjective domains like emotion and morality, where priors ossify posterior predictions. In this work, we examine whether "enabling" reasoning also creates the same behavior in LLMs, wherein the format of CoT retrieves reasoning priors that remain relatively unchanged despite the evidence in the prompt. We find that, surprisingly, CoT indeed suffers from the same posterior collapse as ICL for larger language models. Code is avalaible at this https URL.
• 摘要：大型语言模型 (LLM) 中的上下文学习 (ICL) 已成为执行自然语言任务的主要技术，因为它不需要使用基于梯度的方法更新模型参数。ICL 有望“调整”LLM，使其以极具竞争力或最先进的水平执行当前任务，而计算成本仅为其一小部分。可以通过结合推理过程来增强 ICL，以在提示中明确得出最终标签，这种技术称为思维链 (CoT) 提示。然而，最近的研究发现，ICL 主要依赖于任务先验的检索，而不是“学习”执行任务，尤其是对于情感和道德等复杂的主观领域，其中先验会使后验预测僵化。在这项工作中，我们研究“启用”推理是否也会在 LLM 中产生相同的行为，其中 CoT 的格式检索的推理先验尽管提示中有证据，但仍保持相对不变。令人惊讶的是，我们发现，对于较大的语言模型，CoT 确实遭受与 ICL 相同的后验崩溃。代码可从此 https URL 获取。

Title: Can Large Language Models Unlock Novel Scientific Research Ideas?

• Authors: Sandeep Kumar, Tirthankar Ghosal, Vinayak Goyal, Asif Ekbal
• Subjects: cs.CL, cs.AI, cs.CY, cs.HC, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.06185
• Pdf URL: https://arxiv.org/pdf/2409.06185
• Copy Paste: [[2409.06185]] Can Large Language Models Unlock Novel Scientific Research Ideas?(https://arxiv.org/abs/2409.06185)
• Keywords: language model, gpt, llm, chat
• Abstract: "An idea is nothing more nor less than a new combination of old elements" (Young, J.W.). The widespread adoption of Large Language Models (LLMs) and publicly available ChatGPT have marked a significant turning point in the integration of Artificial Intelligence (AI) into people's everyday lives. This study explores the capability of LLMs in generating novel research ideas based on information from research papers. We conduct a thorough examination of 4 LLMs in five domains (e.g., Chemistry, Computer, Economics, Medical, and Physics). We found that the future research ideas generated by Claude-2 and GPT-4 are more aligned with the author's perspective than GPT-3.5 and Gemini. We also found that Claude-2 generates more diverse future research ideas than GPT-4, GPT-3.5, and Gemini 1.0. We further performed a human evaluation of the novelty, relevancy, and feasibility of the generated future research ideas. This investigation offers insights into the evolving role of LLMs in idea generation, highlighting both its capability and limitations. Our work contributes to the ongoing efforts in evaluating and utilizing language models for generating future research ideas. We make our datasets and codes publicly available.
• 摘要：“一个想法无非是旧元素的新组合”（Young，J.W.）。大型语言模型 (LLM) 和公开可用的 ChatGPT 的广泛采用标志着人工智能 (AI) 融入人们日常生活的重要转折点。本研究探讨了 LLM 根据研究论文信息产生新颖研究想法的能力。我们对五个领域（例如化学、计算机、经济学、医学和物理学）的 4 个 LLM 进行了彻底检查。我们发现 Claude-2 和 GPT-4 产生的未来研究想法比 GPT-3.5 和 Gemini 更符合作者的观点。我们还发现 Claude-2 产生的未来研究想法比 GPT-4、GPT-3.5 和 Gemini 1.0 更加多样化。我们进一步对产生的未来研究想法的新颖性、相关性和可行性进行了人工评估。这项调查深入了解了 LLM 在创意生成中不断演变的角色，突出了其能力和局限性。我们的工作有助于持续评估和利用语言模型来产生未来的研究思路。我们公开提供我们的数据集和代码。

Title: NOVI : Chatbot System for University Novice with BERT and LLMs

• Authors: Yoonji Nam, TaeWoong Seo, Gyeongcheol Shin, Sangji Lee, JaeEun Im
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.06192
• Pdf URL: https://arxiv.org/pdf/2409.06192
• Copy Paste: [[2409.06192]] NOVI : Chatbot System for University Novice with BERT and LLMs(https://arxiv.org/abs/2409.06192)
• Keywords: gpt, llm, chat
• Abstract: To mitigate the difficulties of university freshmen in adapting to university life, we developed NOVI, a chatbot system based on GPT-4o. This system utilizes post and comment data from SKKU 'Everytime', a university community site. Developed using LangChain, NOVI's performance has been evaluated with a BLEU score, Perplexity score, ROUGE-1 score, ROUGE-2 score, ROUGE-L score and METEOR score. This approach is not only limited to help university freshmen but is also expected to help various people adapting to new environments with different data. This research explores the development and potential application of new educational technology tools, contributing to easier social adaptation for beginners and settling a foundation for future advancement in LLM studies.
• 摘要：为了缓解大学新生适应大学生活的困难，我们基于GPT-4o开发了聊天机器人系统NOVI。该系统利用大学社区网站SKKU“Everytime”的帖子和评论数据。NOVI使用LangChain开发，其性能已通过BLEU分数、Perplexity分数、ROUGE-1分数、ROUGE-2分数、ROUGE-L分数和METEOR分数进行评估。这种方法不仅限于帮助大学新生，还有望通过不同的数据帮助不同类型的人适应新环境。本研究探索了新教育技术工具的开发和潜在应用，有助于初学者更轻松地适应社会，并为未来攻读法学硕士奠定基础。

Title: NLP-Powered Repository and Search Engine for Academic Papers: A Case Study on Cyber Risk Literature with CyLit

• Authors: Linfeng Zhang, Changyue Hu, Zhiyu Quan
• Subjects: cs.CL, cs.LG, q-fin.RM
• Abstract URL: https://arxiv.org/abs/2409.06226
• Pdf URL: https://arxiv.org/pdf/2409.06226
• Copy Paste: [[2409.06226]] NLP-Powered Repository and Search Engine for Academic Papers: A Case Study on Cyber Risk Literature with CyLit(https://arxiv.org/abs/2409.06226)
• Keywords: gpt, chat
• Abstract: As the body of academic literature continues to grow, researchers face increasing difficulties in effectively searching for relevant resources. Existing databases and search engines often fall short of providing a comprehensive and contextually relevant collection of academic literature. To address this issue, we propose a novel framework that leverages Natural Language Processing (NLP) techniques. This framework automates the retrieval, summarization, and clustering of academic literature within a specific research domain. To demonstrate the effectiveness of our approach, we introduce CyLit, an NLP-powered repository specifically designed for the cyber risk literature. CyLit empowers researchers by providing access to context-specific resources and enabling the tracking of trends in the dynamic and rapidly evolving field of cyber risk. Through the automatic processing of large volumes of data, our NLP-powered solution significantly enhances the efficiency and specificity of academic literature searches. We compare the literature categorization results of CyLit to those presented in survey papers or generated by ChatGPT, highlighting the distinctive insights this tool provides into cyber risk research literature. Using NLP techniques, we aim to revolutionize the way researchers discover, analyze, and utilize academic resources, ultimately fostering advancements in various domains of knowledge.
• 摘要：随着学术文献数量的不断增长，研究人员在有效搜索相关资源方面面临越来越大的困难。现有的数据库和搜索引擎往往无法提供全面且与上下文相关的学术文献集合。为了解决这个问题，我们提出了一个利用自然语言处理 (NLP) 技术的新框架。该框架可自动检索、总结和聚类特定研究领域的学术文献。为了证明我们方法的有效性，我们推出了 CyLit，这是一个专为网络风险文献设计的 NLP 驱动存储库。CyLit 通过提供对上下文特定资源的访问并支持跟踪动态且快速发展的网络风险领域的趋势，为研究人员提供支持。通过自动处理大量数据，我们的 NLP 驱动解决方案显著提高了学术文献搜索的效率和特异性。我们将 CyLit 的文献分类结果与调查论文中呈现的结果或 ChatGPT 生成的结果进行了比较，突出了该工具对网络风险研究文献提供的独特见解。利用 NLP 技术，我们旨在彻底改变研究人员发现、分析和利用学术资源的方式，最终促进各个知识领域的进步。

Title: Inference is All You Need: Self Example Retriever for Cross-domain Dialogue State Tracking with ChatGPT

• Authors: Jihyun Lee, Gary Geunbae Lee
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.06243
• Pdf URL: https://arxiv.org/pdf/2409.06243
• Copy Paste: [[2409.06243]] Inference is All You Need: Self Example Retriever for Cross-domain Dialogue State Tracking with ChatGPT(https://arxiv.org/abs/2409.06243)
• Keywords: gpt, chat
• Abstract: Traditional dialogue state tracking approaches heavily rely on extensive training data and handcrafted features, limiting their scalability and adaptability to new domains. In this paper, we propose a novel method that leverages inference and in-context learning with ChatGPT for domain transfer in dialogue state tracking, without any parameter updates. By guiding ChatGPT's chain of thought, we enable it to retrieve relevant examples and generalize knowledge to accurately infer dialogue states, solely through inference. Experimental results on the MultiWOZ dataset demonstrate competitive performance and promising generalization across domains. Our parameter-free approach offers a scalable and adaptable solution, opening new research directions in domain transfer learning.
• 摘要：传统的对话状态跟踪方法严重依赖于大量的训练数据和手工制作的特征，这限制了它们对新领域的可扩展性和适应性。在本文中，我们提出了一种新方法，利用 ChatGPT 的推理和上下文学习进行对话状态跟踪中的领域转移，而无需任何参数更新。通过引导 ChatGPT 的思路链，我们使其能够检索相关示例并概括知识，以仅通过推理准确推断对话状态。在 MultiWOZ 数据集上的实验结果展示了具有竞争力的性能和跨领域的良好泛化能力。我们的无参数方法提供了一种可扩展且适应性强的解决方案，为领域迁移学习开辟了新的研究方向。

Title: Keyword-Aware ASR Error Augmentation for Robust Dialogue State Tracking

• Authors: Jihyun Lee, Solee Im, Wonjun Lee, Gary Geunbae Lee
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.06263
• Pdf URL: https://arxiv.org/pdf/2409.06263
• Copy Paste: [[2409.06263]] Keyword-Aware ASR Error Augmentation for Robust Dialogue State Tracking(https://arxiv.org/abs/2409.06263)
• Keywords: prompt
• Abstract: Dialogue State Tracking (DST) is a key part of task-oriented dialogue systems, identifying important information in conversations. However, its accuracy drops significantly in spoken dialogue environments due to named entity errors from Automatic Speech Recognition (ASR) systems. We introduce a simple yet effective data augmentation method that targets those entities to improve the robustness of DST model. Our novel method can control the placement of errors using keyword-highlighted prompts while introducing phonetically similar errors. As a result, our method generated sufficient error patterns on keywords, leading to improved accuracy in noised and low-accuracy ASR environments.
• 摘要：对话状态跟踪 (DST) 是任务导向型对话系统的关键部分，可识别对话中的重要信息。然而，由于自动语音识别 (ASR) 系统中的命名实体错误，其准确性在口语对话环境中显著下降。我们引入了一种简单而有效的数据增强方法，该方法针对这些实体来提高 DST 模型的鲁棒性。我们的新方法可以使用关键字突出显示的提示来控制错误的位置，同时引入语音相似的错误。因此，我们的方法在关键字上生成了足够的错误模式，从而提高了在噪声和低准确度 ASR 环境中的准确性。

Title: Extracting Paragraphs from LLM Token Activations

• Authors: Nicholas Pochinkov, Angelo Benoit, Lovkush Agarwal, Zainab Ali Majid, Lucile Ter-Minassian
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.06328
• Pdf URL: https://arxiv.org/pdf/2409.06328
• Copy Paste: [[2409.06328]] Extracting Paragraphs from LLM Token Activations(https://arxiv.org/abs/2409.06328)
• Keywords: language model, llm
• Abstract: Generative large language models (LLMs) excel in natural language processing tasks, yet their inner workings remain underexplored beyond token-level predictions. This study investigates the degree to which these models decide the content of a paragraph at its onset, shedding light on their contextual understanding. By examining the information encoded in single-token activations, specifically the "\textbackslash n\textbackslash n" double newline token, we demonstrate that patching these activations can transfer significant information about the context of the following paragraph, providing further insights into the model's capacity to plan ahead.
• 摘要：生成式大型语言模型 (LLM) 在自然语言处理任务中表现出色，但除了标记级预测之外，其内部工作原理仍未得到充分探索。本研究调查了这些模型在段落开头决定其内容的程度，从而揭示了它们对上下文的理解。通过检查单标记激活中编码的信息，特别是“\textbackslash n\textbackslash n”双换行标记，我们证明修补这些激活可以传输有关下一段上下文的重要信息，从而进一步了解模型的提前规划能力。

Title: Retrieval Or Holistic Understanding? Dolce: Differentiate Our Long Context Evaluation Tasks

• Authors: Zi Yang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.06338
• Pdf URL: https://arxiv.org/pdf/2409.06338
• Copy Paste: [[2409.06338]] Retrieval Or Holistic Understanding? Dolce: Differentiate Our Long Context Evaluation Tasks(https://arxiv.org/abs/2409.06338)
• Keywords: llm, long context
• Abstract: We argue that there are two major distinct capabilities in long context understanding: retrieval and holistic understanding. Understanding and further improving LLMs' long context capabilities would not be possible without knowing the tasks' focus categories. We aim to automatically identify retrieval focused and holistic understanding focused problems from suites of benchmarks and quantitatively measure the difficulty within each focus. In this paper, we present the Dolce framework, which parameterizes each problem by $\lambda$ (complexity) and $k$ (redundancy) and assigns to one of five predefined focus categories. We propose to sample short contexts from the full context and estimate the probability an LLM solves the problem using the sampled spans. To find the $\lambda$ and $k$ for each problem, we further propose a mixture model of a non-parametric background noise component and a parametric/non-parametric hybrid oracle component, where we derive the probability functions parameterized by $\lambda$ and $k$ for both the correct-or-wrong (COW) scenario and the partial-point-in-grading (PIG) scenario. Our proposed methods can identify 0% to 67% of the problems are retrieval focused and 0% to 90% of the problems are holistic understanding focused across 44 existing long context evaluation tasks.
• 摘要：我们认为，长上下文理解有两种主要的不同能力：检索和整体理解。如果不知道任务的焦点类别，就不可能理解并进一步提高 LLM 的长上下文能力。我们的目标是从一系列基准中自动识别以检索为重点的问题和以整体理解为重点的问题，并定量测量每个焦点内的难度。在本文中，我们提出了 Dolce 框架，该框架通过 $\lambda$（复杂性）和 $k$（冗余度）参数化每个问题，并将其分配给五个预定义的焦点类别之一。我们建议从完整上下文中抽取短上下文，并估计 LLM 使用抽样跨度解决问题的概率。为了找到每个问题的 $\lambda$ 和 $k$，我们进一步提出了一个非参数背景噪声组件和参数/非参数混合预言机组件的混合模型，其中我们推导出由 $\lambda$ 和 $k$ 参数化的概率函数，用于正确或错误 (COW) 场景和部分点分级 (PIG) 场景。我们提出的方法可以在 44 个现有的长上下文评估任务中识别出 0% 到 67% 的问题以检索为重点，0% 到 90% 的问题以整体理解为重点。

Title: Questioning Internal Knowledge Structure of Large Language Models Through the Lens of the Olympic Games

• Authors: Juhwan Choi, YoungBin Kim
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.06518
• Pdf URL: https://arxiv.org/pdf/2409.06518
• Copy Paste: [[2409.06518]] Questioning Internal Knowledge Structure of Large Language Models Through the Lens of the Olympic Games(https://arxiv.org/abs/2409.06518)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have become a dominant approach in natural language processing, yet their internal knowledge structures remain largely unexplored. In this paper, we analyze the internal knowledge structures of LLMs using historical medal tallies from the Olympic Games. We task the models with providing the medal counts for each team and identifying which teams achieved specific rankings. Our results reveal that while state-of-the-art LLMs perform remarkably well in reporting medal counts for individual teams, they struggle significantly with questions about specific rankings. This suggests that the internal knowledge structures of LLMs are fundamentally different from those of humans, who can easily infer rankings from known medal counts. To support further research, we publicly release our code, dataset, and model outputs.
• 摘要：大型语言模型 (LLM) 已成为自然语言处理的主导方法，但其内部知识结构仍未得到充分探索。在本文中，我们使用奥运会的历史奖牌统计来分析 LLM 的内部知识结构。我们要求模型提供每个队伍的奖牌数，并确定哪些队伍获得了特定排名。我们的结果表明，虽然最先进的 LLM 在报告各个队伍的奖牌数方面表现非常出色，但它们在回答有关具体排名的问题时却遇到了很大困难。这表明 LLM 的内部知识结构与人类的内部知识结构有着根本的不同，人类可以轻松地根据已知的奖牌数推断排名。为了支持进一步的研究，我们公开发布了我们的代码、数据集和模型输出。

Title: Mapping News Narratives Using LLMs and Narrative-Structured Text Embeddings

• Authors: Jan Elfes
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.06540
• Pdf URL: https://arxiv.org/pdf/2409.06540
• Copy Paste: [[2409.06540]] Mapping News Narratives Using LLMs and Narrative-Structured Text Embeddings(https://arxiv.org/abs/2409.06540)
• Keywords: llm
• Abstract: Given the profound impact of narratives across various societal levels, from personal identities to international politics, it is crucial to understand their distribution and development over time. This is particularly important in online spaces. On the Web, narratives can spread rapidly and intensify societal divides and conflicts. While many qualitative approaches exist, quantifying narratives remains a significant challenge. Computational narrative analysis lacks frameworks that are both comprehensive and generalizable. To address this gap, we introduce a numerical narrative representation grounded in structuralist linguistic theory. Chiefly, Greimas' Actantial Model represents a narrative through a constellation of six functional character roles. These so-called actants are genre-agnostic, making the model highly generalizable. We extract the actants using an open-source LLM and integrate them into a Narrative-Structured Text Embedding that captures both the semantics and narrative structure of a text. We demonstrate the analytical insights of the method on the example of 5000 full-text news articles from Al Jazeera and The Washington Post on the Israel-Palestine conflict. Our method successfully distinguishes articles that cover the same topics but differ in narrative structure.
• 摘要：鉴于叙事对从个人身份到国际政治等各个社会层面都具有深远影响，了解叙事的分布和发展至关重要。这在网络空间中尤为重要。在网络上，叙事可以迅速传播，加剧社会分歧和冲突。虽然存在许多定性方法，但量化叙事仍然是一项重大挑战。计算叙事分析缺乏全面且可推广的框架。为了解决这一差距，我们引入了一种基于结构主义语言理论的数值叙事表示。主要是，Greimas 的 Actantial 模型通过六个功能性角色来表示叙事。这些所谓的行为者与类型无关，使该模型具有高度的可推广性。我们使用开源 LLM 提取行为者，并将它们集成到叙事结构化文本嵌入中，以捕获文本的语义和叙事结构。我们以半岛电视台和华盛顿邮报关于巴以冲突的 5000 篇全文新闻文章为例，展示了该方法的分析见解。我们的方法成功区分了涵盖相同主题但叙事结构不同的文章。

Title: Exploring syntactic information in sentence embeddings through multilingual subject-verb agreement

• Authors: Vivi Nastase, Chunyang Jiang, Giuseppe Samo, Paola Merlo
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.06567
• Pdf URL: https://arxiv.org/pdf/2409.06567
• Copy Paste: [[2409.06567]] Exploring syntactic information in sentence embeddings through multilingual subject-verb agreement(https://arxiv.org/abs/2409.06567)
• Keywords: language model
• Abstract: In this paper, our goal is to investigate to what degree multilingual pretrained language models capture cross-linguistically valid abstract linguistic representations. We take the approach of developing curated synthetic data on a large scale, with specific properties, and using them to study sentence representations built using pretrained language models. We use a new multiple-choice task and datasets, Blackbird Language Matrices (BLMs), to focus on a specific grammatical structural phenomenon -- subject-verb agreement across a variety of sentence structures -- in several languages. Finding a solution to this task requires a system detecting complex linguistic patterns and paradigms in text representations. Using a two-level architecture that solves the problem in two steps -- detect syntactic objects and their properties in individual sentences, and find patterns across an input sequence of sentences -- we show that despite having been trained on multilingual texts in a consistent manner, multilingual pretrained language models have language-specific differences, and syntactic structure is not shared, even across closely related languages.
• 摘要：在本文中，我们的目标是研究多语言预训练语言模型在多大程度上能够捕捉跨语言有效的抽象语言表征。我们采用大规模开发具有特定属性的精选合成数据的方法，并使用它们来研究使用预训练语言模型构建的句子表征。我们使用一种新的多项选择任务和数据集 Blackbird 语言矩阵 (BLM)，来关注几种语言中的特定语法结构现象——各种句子结构中的主谓一致。要找到这项任务的解决方案，需要一个系统来检测文本表征中的复杂语言模式和范式。使用两级架构，分两步解决问题——检测单个句子中的句法对象及其属性，并在输入的句子序列中查找模式——我们表明，尽管以一致的方式对多语言文本进行了训练，但多语言预训练语言模型具有特定于语言的差异，即使在密切相关的语言之间，句法结构也不共享。

Title: GroUSE: A Benchmark to Evaluate Evaluators in Grounded Question Answering

• Authors: Sacha Muller, António Loison, Bilel Omrani, Gautier Viaud
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.06595
• Pdf URL: https://arxiv.org/pdf/2409.06595
• Copy Paste: [[2409.06595]] GroUSE: A Benchmark to Evaluate Evaluators in Grounded Question Answering(https://arxiv.org/abs/2409.06595)
• Keywords: language model, gpt, llm, retrieval-augmented generation
• Abstract: Retrieval-Augmented Generation (RAG) has emerged as a common paradigm to use Large Language Models (LLMs) alongside private and up-to-date knowledge bases. In this work, we address the challenges of using LLM-as-a-Judge when evaluating grounded answers generated by RAG systems. To assess the calibration and discrimination capabilities of judge models, we identify 7 generator failure modes and introduce GroUSE (Grounded QA Unitary Scoring of Evaluators), a meta-evaluation benchmark of 144 unit tests. This benchmark reveals that existing automated RAG evaluation frameworks often overlook important failure modes, even when using GPT-4 as a judge. To improve on the current design of automated RAG evaluation frameworks, we propose a novel pipeline and find that while closed models perform well on GroUSE, state-of-the-art open-source judges do not generalize to our proposed criteria, despite strong correlation with GPT-4's judgement. Our findings suggest that correlation with GPT-4 is an incomplete proxy for the practical performance of judge models and should be supplemented with evaluations on unit tests for precise failure mode detection. We further show that finetuning Llama-3 on GPT-4's reasoning traces significantly boosts its evaluation capabilities, improving upon both correlation with GPT-4's evaluations and calibration on reference situations.
• 摘要：检索增强生成 (RAG) 已成为使用大型语言模型 (LLM) 以及私有和最新知识库的常见范例。在这项工作中，我们解决了在评估 RAG 系统生成的扎实答案时使用 LLM 作为判断者的挑战。为了评估判断模型的校准和区分能力，我们确定了 7 种生成器故障模式，并引入了 GroUSE（评估者的扎实 QA 单元评分），这是一个包含 144 个单元测试的元评估基准。该基准表明，现有的自动化 RAG 评估框架经常忽略重要的故障模式，即使使用 GPT-4 作为判断者也是如此。为了改进自动化 RAG 评估框架的当前设计，我们提出了一种新颖的流程，并发现虽然封闭模型在 GroUSE 上表现良好，但最先进的开源判断者并不能推广到我们提出的标准，尽管与 GPT-4 的判断有很强的相关性。我们的研究结果表明，与 GPT-4 的相关性并不能完全代表判断模型的实际性能，应辅以单元测试评估，以精确检测故障模式。我们进一步表明，在 GPT-4 的推理轨迹上对 Llama-3 进行微调可显著提高其评估能力，从而提高与 GPT-4 评估的相关性和参考情况下的校准。

Title: Alleviating Hallucinations in Large Language Models with Scepticism Modeling

• Authors: Yetao Wu, Yihong Wang, Teng Chen, Chenxi Liu, Ningyuan Xi, Qingqing Gu, Hongyang Lei, Zhonglin Jiang, Yong Chen, Luo Ji
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.06601
• Pdf URL: https://arxiv.org/pdf/2409.06601
• Copy Paste: [[2409.06601]] Alleviating Hallucinations in Large Language Models with Scepticism Modeling(https://arxiv.org/abs/2409.06601)
• Keywords: language model, llm, hallucination
• Abstract: Hallucinations is a major challenge for large language models (LLMs), prevents adoption in diverse fields. Uncertainty estimation could be used for alleviating the damages of hallucinations. The skeptical emotion of human could be useful for enhancing the ability of self estimation. Inspirited by this observation, we proposed a new approach called Skepticism Modeling (SM). This approach is formalized by combining the information of token and logits for self estimation. We construct the doubt emotion aware data, perform continual pre-training, and then fine-tune the LLMs, improve their ability of self estimation. Experimental results demonstrate this new approach effectively enhances a model's ability to estimate their uncertainty, and validate its generalization ability of other tasks by out-of-domain experiments.
• 摘要：幻觉是大型语言模型 (LLM) 面临的一大挑战，阻碍了其在各个领域的应用。不确定性估计可用于减轻幻觉的危害。人类的怀疑情绪可能有助于增强自我估计的能力。受此观察的启发，我们提出了一种称为怀疑论建模 (SM) 的新方法。该方法通过结合 token 和 logits 的信息进行自我估计来形式化。我们构建怀疑情绪感知数据，进行持续的预训练，然后对 LLM 进行微调，提高其自我估计能力。实验结果表明，这种新方法有效地增强了模型估计其不确定性的能力，并通过域外实验验证了其对其他任务的泛化能力。

Title: Exploring Italian sentence embeddings properties through multi-tasking

• Authors: Vivi Nastase, Giuseppe Samo, Chunyang Jiang, Paola Merlo
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.06622
• Pdf URL: https://arxiv.org/pdf/2409.06622
• Copy Paste: [[2409.06622]] Exploring Italian sentence embeddings properties through multi-tasking(https://arxiv.org/abs/2409.06622)
• Keywords: language model, llm
• Abstract: We investigate to what degree existing LLMs encode abstract linguistic information in Italian in a multi-task setting. We exploit curated synthetic data on a large scale -- several Blackbird Language Matrices (BLMs) problems in Italian -- and use them to study how sentence representations built using pre-trained language models encode specific syntactic and semantic information. We use a two-level architecture to model separately a compression of the sentence embeddings into a representation that contains relevant information for a task, and a BLM task. We then investigate whether we can obtain compressed sentence representations that encode syntactic and semantic information relevant to several BLM tasks. While we expected that the sentence structure -- in terms of sequence of phrases/chunks -- and chunk properties could be shared across tasks, performance and error analysis show that the clues for the different tasks are encoded in different manners in the sentence embeddings, suggesting that abstract linguistic notions such as constituents or thematic roles does not seem to be present in the pretrained sentence embeddings.
• 摘要：我们调查了现有的 LLM 在多任务设置中对意大利语抽象语言信息的编码程度。我们大规模利用精选的合成数据——意大利语的几个 Blackbird 语言矩阵 (BLM) 问题——并使用它们来研究使用预训练语言模型构建的句子表示如何编码特定的句法和语义信息。我们使用两级架构分别对句子嵌入的压缩建模为包含任务相关信息的表示和 BLM 任务。然后，我们调查是否可以获得压缩的句子表示，这些表示对与几个 BLM 任务相关的句法和语义信息进行编码。虽然我们预计句子结构（就短语/块的序列而言）和块属性可以在任务之间共享，但性能和错误分析表明，不同任务的线索在句子嵌入中的编码方式不同，这表明抽象语言概念（例如成分或主题角色）似乎并不存在于预训练的句子嵌入中。

Title: A Practice of Post-Training on Llama-3 70B with Optimal Selection of Additional Language Mixture Ratio

• Authors: Ningyuan Xi, Yetao Wu, Kun Fan, Teng Chen, Qingqing Gu, Peng Yu, Jinxian Qu, Chenxi Liu, Zhonglin Jiang, Yong Chen, Luo Ji
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.06624
• Pdf URL: https://arxiv.org/pdf/2409.06624
• Copy Paste: [[2409.06624]] A Practice of Post-Training on Llama-3 70B with Optimal Selection of Additional Language Mixture Ratio(https://arxiv.org/abs/2409.06624)
• Keywords: language model, llm, chat
• Abstract: Large Language Models (LLM) often needs to be Continual Pre-Trained (CPT) to obtain the unfamiliar language skill or adapt into new domains. The huge training cost of CPT often asks for cautious choice of key hyper-parameters such as the mixture ratio of extra language or domain corpus. However, there is no systematic study which bridge the gap between the optimal mixture ratio and the actual model performance, and the gap between experimental scaling law and the actual deployment in the full model size. In this paper, we perform CPT on Llama-3 8B and 70B to enhance its Chinese ability. We study the optimal correlation between the Additional Language Mixture Ratio (ALMR) and the Learning Rate (LR) on the 8B size which directly indicate the optimal experimental set up. By thorough choice of hyper-parameter, and subsequent fine-tuning, the model capability is improved not only on the Chinese-related benchmark, but also some specific domains including math, coding and emotional intelligence. We deploy the final 70B version of LLM on an real-life chat system which obtain satisfying performance.
• 摘要：大型语言模型（LLM）往往需要进行持续预训练（CPT）才能获得不熟悉的语言技能或者适应新领域。CPT 巨大的训练成本往往要求谨慎选择关键的超参数，比如额外语言或领域语料的混合比。然而，目前还没有系统的研究来弥合最佳混合比和实际模型性能之间的差距，以及实验缩放定律和全模型实际部署之间的差距。在本文中，我们在 Llama-3 8B 和 70B 上进行 CPT 来增强其中文能力。我们研究了 8B 规模上额外语言混合比（ALMR）和学习率（LR）之间的最佳相关性，这直接表明了最佳实验设置。通过彻底选择超参数，以及随后的微调，模型能力不仅在与中文相关的基准上得到了提升，而且在数学、编码和情商等一些特定领域也得到了提升。我们在真实的聊天系统上部署了LLM的最终70B版本，获得了令人满意的性能。

Title: TeXBLEU: Automatic Metric for Evaluate LaTeX Format

• Authors: Kyudan Jung, Nam-Joon Kim, Hyongon Ryu, Sieun Hyeon, Seung-jun Lee, Hyeok-jae Lee
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.06639
• Pdf URL: https://arxiv.org/pdf/2409.06639
• Copy Paste: [[2409.06639]] TeXBLEU: Automatic Metric for Evaluate LaTeX Format(https://arxiv.org/abs/2409.06639)
• Keywords: language model
• Abstract: LaTeX is highly suited to creating documents with special formatting, particularly in the fields of science, technology, mathematics, and computer science. Despite the increasing use of mathematical expressions in LaTeX format with language models, there are no evaluation metrics for evaluating them. In this study, we propose TeXBLEU, an evaluation metric tailored for mathematical expressions in LaTeX format, based on the n-gram-based BLEU metric that is widely used for translation tasks. The proposed TeXBLEU includes a predefined tokenizer trained on the arXiv paper dataset and a finetuned embedding model. It also considers the positional embedding of tokens. Simultaneously, TeXBLEU compares tokens based on n-grams and computes the score using exponentiation of a logarithmic sum, similar to the original BLEU. Experimental results show that TeXBLEU outperformed traditional evaluation metrics such as BLEU, Rouge, CER, and WER when compared to human evaluation data on the test dataset of the MathBridge dataset, which contains 1,000 data points. The average correlation coefficient with human evaluation was 0.71, which is an improvement of 87% compared with BLEU, which had the highest correlation with human evaluation data among the existing metrics. The code is available at this https URL.
• 摘要：LaTeX 非常适合创建具有特殊格式的文档，特别是在科学、技术、数学和计算机科学领域。尽管在语言模型中越来越多地使用 LaTeX 格式的数学表达式，但没有评估它们的评估指标。在本研究中，我们提出了 TeXBLEU，这是一种针对 LaTeX 格式的数学表达式量身定制的评估指标，基于广泛用于翻译任务的基于 n-gram 的 BLEU 指标。所提出的 TeXBLEU 包括一个在 arXiv 论文数据集上训练的预定义标记器和一个微调的嵌入模型。它还考虑了标记的位置嵌入。同时，TeXBLEU 根据 n-gram 比较标记并使用对数和的幂来计算分数，类似于原始 BLEU。实验结果表明，在包含 1000 个数据点的 MathBridge 数据集的测试集上，TeXBLEU 与人工评测数据相比，表现优于 BLEU、Rouge、CER、WER 等传统评测指标。与人工评测的平均相关系数为 0.71，与现有指标中与人工评测数据相关性最高的 BLEU 相比，提升了 87%。代码可从此 https URL 获取。

Title: LLaMA-Omni: Seamless Speech Interaction with Large Language Models

• Authors: Qingkai Fang, Shoutao Guo, Yan Zhou, Zhengrui Ma, Shaolei Zhang, Yang Feng
• Subjects: cs.CL, cs.AI, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2409.06666
• Pdf URL: https://arxiv.org/pdf/2409.06666
• Copy Paste: [[2409.06666]] LLaMA-Omni: Seamless Speech Interaction with Large Language Models(https://arxiv.org/abs/2409.06666)
• Keywords: language model, gpt, llm
• Abstract: Models like GPT-4o enable real-time interaction with large language models (LLMs) through speech, significantly enhancing user experience compared to traditional text-based interaction. However, there is still a lack of exploration on how to build speech interaction models based on open-source LLMs. To address this, we propose LLaMA-Omni, a novel model architecture designed for low-latency and high-quality speech interaction with LLMs. LLaMA-Omni integrates a pretrained speech encoder, a speech adaptor, an LLM, and a streaming speech decoder. It eliminates the need for speech transcription, and can simultaneously generate text and speech responses directly from speech instructions with extremely low latency. We build our model based on the latest Llama-3.1-8B-Instruct model. To align the model with speech interaction scenarios, we construct a dataset named InstructS2S-200K, which includes 200K speech instructions and corresponding speech responses. Experimental results show that compared to previous speech-language models, LLaMA-Omni provides better responses in both content and style, with a response latency as low as 226ms. Additionally, training LLaMA-Omni takes less than 3 days on just 4 GPUs, paving the way for the efficient development of speech-language models in the future.
• 摘要：类似 GPT-4o 的模型能够通过语音与大型语言模型 (LLM) 进行实时交互，与传统的基于文本的交互相比，用户体验显著提升。然而，如何基于开源 LLM 构建语音交互模型的探索还很缺乏。针对这一问题，我们提出了 LLaMA-Omni，一种专为与 LLM 进行低延迟高质量语音交互而设计的新型模型架构。LLaMA-Omni 集成了预训练的语音编码器、语音适配器、LLM 和流式语音解码器，无需语音转录，能够同时从语音指令直接生成文本和语音响应，延迟极低。我们的模型基于最新的 Llama-3.1-8B-Instruct 模型构建。为了使模型与语音交互场景保持一致，我们构建了一个名为 InstructS2S-200K 的数据集，其中包含 200K 条语音指令和相应的语音响应。实验结果表明，相较于之前的语音语言模型，LLaMA-Omni 在内容和风格方面均提供了更佳的响应，响应延迟低至 226ms。此外，在 4 块 GPU 上训练 LLaMA-Omni 只需不到 3 天的时间，为未来高效开发语音语言模型奠定了基础。

Title: E2LLM: Encoder Elongated Large Language Models for Long-Context Understanding and Reasoning

• Authors: Zihan Liao, Jun Wang, Hang Yu, Lingxiao Wei, Jianguo Li, Jun Wang, Wei Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.06679
• Pdf URL: https://arxiv.org/pdf/2409.06679
• Copy Paste: [[2409.06679]] E2LLM: Encoder Elongated Large Language Models for Long-Context Understanding and Reasoning(https://arxiv.org/abs/2409.06679)
• Keywords: language model, llm, long context, prompt
• Abstract: In the realm of Large Language Models (LLMs), the ability to process long contexts is increasingly crucial for tasks such as multi-round dialogues, code generation, and document summarization. This paper addresses the challenges of enhancing the long-context performance, reducing computational complexity, and leveraging pretrained models collectively termed the "impossible triangle." We introduce E2LLM (Encoder Elongated Large Language Models), a novel approach that effectively navigates this paradox. The method involves splitting long contexts into chunks, compressing each into embedding vectors via a pretrained text encoder, and utilizing an adapter to align these representations with a decoder-only LLM. Two training objectives, focusing on reconstruction of the encoder output and long-context instruction fine-tuning, are employed to facilitate the understanding of soft prompts by the LLM. Experimental results demonstrate that E2LLM achieves superior performance in long-context scenarios while balancing efficiency, performance, and compatibility with pretrained models. Our framework thus represents a significant advancement in the field, contributing to effective long-text modeling.
• 摘要：在大型语言模型 (LLM) 领域，处理长上下文的能力对于多轮对话、代码生成和文档摘要等任务越来越重要。本文讨论了增强长上下文性能、降低计算复杂度和利用预训练模型（统称为“不可能三角”）的挑战。我们引入了 E2LLM（编码器延长大型语言模型），这是一种有效解决这一悖论的新方法。该方法包括将长上下文拆分为块，通过预训练的文本编码器将每个块压缩为嵌入向量，并利用适配器将这些表示与仅解码器的 LLM 对齐。采用两个训练目标，重点关注编码器输出的重建和长上下文指令微调，以促进 LLM 对软提示的理解。实验结果表明，E2LLM 在长上下文场景中实现了卓越的性能，同时平衡了效率、性能和与预训练模型的兼容性。因此，我们的框架代表了该领域的重大进步，有助于有效的长文本建模。