2024-11-11

Title: Multimodal Quantum Natural Language Processing: A Novel Framework for using Quantum Methods to Analyse Real Data

• Authors: Hala Hawashin
• Subjects: cs.CL, cs.LG, quant-ph
• Abstract URL: https://arxiv.org/abs/2411.05023
• Pdf URL: https://arxiv.org/pdf/2411.05023
• Copy Paste: [[2411.05023]] Multimodal Quantum Natural Language Processing: A Novel Framework for using Quantum Methods to Analyse Real Data(https://arxiv.org/abs/2411.05023)
• Keywords: language model
• Abstract: Despite significant advances in quantum computing across various domains, research on applying quantum approaches to language compositionality - such as modeling linguistic structures and interactions - remains limited. This gap extends to the integration of quantum language data with real-world data from sources like images, video, and audio. This thesis explores how quantum computational methods can enhance the compositional modeling of language through multimodal data integration. Specifically, it advances Multimodal Quantum Natural Language Processing (MQNLP) by applying the Lambeq toolkit to conduct a comparative analysis of four compositional models and evaluate their influence on image-text classification tasks. Results indicate that syntax-based models, particularly DisCoCat and TreeReader, excel in effectively capturing grammatical structures, while bag-of-words and sequential models struggle due to limited syntactic awareness. These findings underscore the potential of quantum methods to enhance language modeling and drive breakthroughs as quantum technology evolves.
• 摘要：尽管量子计算在各个领域都取得了重大进展，但将量子方法应用于语言组合性（例如对语言结构和交互进行建模）的研究仍然有限。这一差距延伸到将量子语言数据与来自图像、视频和音频等来源的真实世界数据进行集成。本论文探讨了量子计算方法如何通过多模态数据集成来增强语言的组合建模。具体来说，它通过应用 Lambeq 工具包对四种组合模型进行比较分析并评估它们对图像文本分类任务的影响，推进了多模态量子自然语言处理 (MQNLP)。结果表明，基于语法的模型（尤其是 DisCoCat 和 TreeReader）在有效捕捉语法结构方面表现出色，而词袋和顺序模型由于句法意识有限而举步维艰。这些发现强调了量子方法在增强语言建模和推动量子技术发展突破方面的潜力。

Title: LLMs as Research Tools: A Large Scale Survey of Researchers' Usage and Perceptions

• Authors: Zhehui Liao, Maria Antoniak, Inyoung Cheong, Evie Yu-Yen Cheng, Ai-Heng Lee, Kyle Lo, Joseph Chee Chang, Amy X. Zhang
• Subjects: cs.CL, cs.AI, cs.CY, cs.DL, cs.HC
• Abstract URL: https://arxiv.org/abs/2411.05025
• Pdf URL: https://arxiv.org/pdf/2411.05025
• Copy Paste: [[2411.05025]] LLMs as Research Tools: A Large Scale Survey of Researchers' Usage and Perceptions(https://arxiv.org/abs/2411.05025)
• Keywords: language model, llm
• Abstract: The rise of large language models (LLMs) has led many researchers to consider their usage for scientific work. Some have found benefits using LLMs to augment or automate aspects of their research pipeline, while others have urged caution due to risks and ethical concerns. Yet little work has sought to quantify and characterize how researchers use LLMs and why. We present the first large-scale survey of 816 verified research article authors to understand how the research community leverages and perceives LLMs as research tools. We examine participants' self-reported LLM usage, finding that 81% of researchers have already incorporated LLMs into different aspects of their research workflow. We also find that traditionally disadvantaged groups in academia (non-White, junior, and non-native English speaking researchers) report higher LLM usage and perceived benefits, suggesting potential for improved research equity. However, women, non-binary, and senior researchers have greater ethical concerns, potentially hindering adoption.
• 摘要：大型语言模型 (LLM) 的兴起促使许多研究人员考虑将其用于科学工作。一些人发现使用 LLM 可以增强或自动化其研究流程的某些方面，而另一些人则因风险和道德问题而敦促谨慎行事。然而，很少有研究试图量化和描述研究人员使用 LLM 的方式和原因。我们首次对 816 位经过验证的研究文章作者进行了大规模调查，以了解研究界如何利用和看待 LLM 作为研究工具。我们研究了参与者自我报告的 LLM 使用情况，发现 81% 的研究人员已经将 LLM 纳入其研究工作流程的不同方面。我们还发现，学术界传统上处于弱势的群体（非白人、初级和非英语母语研究人员）报告的 LLM 使用率更高，感知到的好处也更多，这表明研究公平性有改善的潜力。然而，女性、非二元性别和高级研究人员有更大的道德顾虑，这可能会阻碍采用。

Title: Deep Learning and Machine Learning -- Natural Language Processing: From Theory to Application

• Authors: Keyu Chen, Cheng Fei, Ziqian Bi, Junyu Liu, Benji Peng, Sen Zhang, Xuanhe Pan, Jiawei Xu, Jinlang Wang, Caitlyn Heqi Yin, Yichao Zhang, Pohsun Feng, Yizhu Wen, Tianyang Wang, Ming Li, Jintao Ren, Qian Niu, Silin Chen, Weiche Hsieh, Lawrence K.Q. Yan, Chia Xin Liang, Han Xu, Hong-Ming Tseng, Xinyuan Song, Ming Liu
• Subjects: cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2411.05026
• Pdf URL: https://arxiv.org/pdf/2411.05026
• Copy Paste: [[2411.05026]] Deep Learning and Machine Learning -- Natural Language Processing: From Theory to Application(https://arxiv.org/abs/2411.05026)
• Keywords: language model, llm
• Abstract: With a focus on natural language processing (NLP) and the role of large language models (LLMs), we explore the intersection of machine learning, deep learning, and artificial intelligence. As artificial intelligence continues to revolutionize fields from healthcare to finance, NLP techniques such as tokenization, text classification, and entity recognition are essential for processing and understanding human language. This paper discusses advanced data preprocessing techniques and the use of frameworks like Hugging Face for implementing transformer-based models. Additionally, it highlights challenges such as handling multilingual data, reducing bias, and ensuring model robustness. By addressing key aspects of data processing and model fine-tuning, this work aims to provide insights into deploying effective and ethically sound AI solutions.
• 摘要：我们重点关注自然语言处理 (NLP) 和大型语言模型 (LLM) 的作用，探索机器学习、深度学习和人工智能的交集。随着人工智能继续彻底改变从医疗保健到金融等领域，标记化、文本分类和实体识别等 NLP 技术对于处理和理解人类语言至关重要。本文讨论了高级数据预处理技术以及使用 Hugging Face 等框架来实现基于转换器的模型。此外，它还强调了处理多语言数据、减少偏差和确保模型稳健性等挑战。通过解决数据处理和模型微调的关键方面，这项工作旨在为部署有效且合乎道德的 AI 解决方案提供见解。

Title: On-Device Emoji Classifier Trained with GPT-based Data Augmentation for a Mobile Keyboard

• Authors: Hossam Amer, Joe Osborne, Michael Zaki, Mohamed Afify
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.05031
• Pdf URL: https://arxiv.org/pdf/2411.05031
• Copy Paste: [[2411.05031]] On-Device Emoji Classifier Trained with GPT-based Data Augmentation for a Mobile Keyboard(https://arxiv.org/abs/2411.05031)
• Keywords: gpt
• Abstract: Emojis improve communication quality among smart-phone users that use mobile keyboards to exchange text. To predict emojis for users based on input text, we should consider the on-device low memory and time constraints, ensure that the on-device emoji classifier covers a wide range of emoji classes even though the emoji dataset is typically imbalanced, and adapt the emoji classifier output to user favorites. This paper proposes an on-device emoji classifier based on MobileBert with reasonable memory and latency requirements for SwiftKey. To account for the data imbalance, we utilize the widely used GPT to generate one or more tags for each emoji class. For each emoji and corresponding tags, we merge the original set with GPT-generated sentences and label them with this emoji without human intervention to alleviate the data imbalance. At inference time, we interpolate the emoji output with the user history for emojis for better emoji classifications. Results show that the proposed on-device emoji classifier deployed for SwiftKey increases the accuracy performance of emoji prediction particularly on rare emojis and emoji engagement.
• 摘要：表情符号提高了使用移动键盘交换文本的智能手机用户之间的通信质量。要根据输入文本预测用户的表情符号，我们应该考虑设备上的低内存和时间限制，确保设备上的表情符号分类器覆盖广泛的表情符号类别（即使表情符号数据集通常不平衡），并根据用户喜好调整表情符号分类器输出。本文提出了一种基于 MobileBert 的设备表情符号分类器，对 SwiftKey 具有合理的内存和延迟要求。为了解决数据不平衡问题，我们利用广泛使用的 GPT 为每个表情符号类别生成一个或多个标签。对于每个表情符号和相应的标签，我们将原始集合与 GPT 生成的句子合并，并用这个表情符号标记它们，无需人工干预，以缓解数据不平衡。在推理时，我们将表情符号输出与表情符号的用户历史记录进行插值，以获得更好的表情符号分类。结果表明，为 SwiftKey 部署的设备上表情符号分类器提高了表情符号预测的准确性性能，特别是对于罕见表情符号和表情符号参与度。

Title: From Word Vectors to Multimodal Embeddings: Techniques, Applications, and Future Directions For Large Language Models

• Authors: Charles Zhang, Benji Peng, Xintian Sun, Qian Niu, Junyu Liu, Keyu Chen, Ming Li, Pohsun Feng, Ziqian Bi, Ming Liu, Yichao Zhang, Cheng Fei, Caitlyn Heqi Yin, Lawrence KQ Yan, Tianyang Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.05036
• Pdf URL: https://arxiv.org/pdf/2411.05036
• Copy Paste: [[2411.05036]] From Word Vectors to Multimodal Embeddings: Techniques, Applications, and Future Directions For Large Language Models(https://arxiv.org/abs/2411.05036)
• Keywords: language model, gpt
• Abstract: Word embeddings and language models have transformed natural language processing (NLP) by facilitating the representation of linguistic elements in continuous vector spaces. This review visits foundational concepts such as the distributional hypothesis and contextual similarity, tracing the evolution from sparse representations like one-hot encoding to dense embeddings including Word2Vec, GloVe, and fastText. We examine both static and contextualized embeddings, underscoring advancements in models such as ELMo, BERT, and GPT and their adaptations for cross-lingual and personalized applications. The discussion extends to sentence and document embeddings, covering aggregation methods and generative topic models, along with the application of embeddings in multimodal domains, including vision, robotics, and cognitive science. Advanced topics such as model compression, interpretability, numerical encoding, and bias mitigation are analyzed, addressing both technical challenges and ethical implications. Additionally, we identify future research directions, emphasizing the need for scalable training techniques, enhanced interpretability, and robust grounding in non-textual modalities. By synthesizing current methodologies and emerging trends, this survey offers researchers and practitioners an in-depth resource to push the boundaries of embedding-based language models.
• 摘要：词嵌入和语言模型通过促进语言元素在连续向量空间中的表示，改变了自然语言处理 (NLP)。本综述探讨了分布假设和上下文相似性等基础概念，追溯了从独热编码等稀疏表示到 Word2Vec、GloVe 和 fastText 等密集嵌入的演变。我们研究了静态和上下文嵌入，强调了 ELMo、BERT 和 GPT 等模型的进步及其对跨语言和个性化应用的适应性。讨论扩展到句子和文档嵌入，涵盖聚合方法和生成主题模型，以及嵌入在多模态领域（包括视觉、机器人和认知科学）中的应用。分析了模型压缩、可解释性、数值编码和偏见缓解等高级主题，解决了技术挑战和道德影响。此外，我们还确定了未来的研究方向，强调需要可扩展的训练技术、增强的可解释性和对非文本模式的稳健基础。通过综合当前的方法和新兴趋势，本调查为研究人员和从业者提供了深入的资源，以突破基于嵌入的语言模型的界限。

Title: Towards Interpreting Language Models: A Case Study in Multi-Hop Reasoning

• Authors: Mansi Sakarvadia
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.05037
• Pdf URL: https://arxiv.org/pdf/2411.05037
• Copy Paste: [[2411.05037]] Towards Interpreting Language Models: A Case Study in Multi-Hop Reasoning(https://arxiv.org/abs/2411.05037)
• Keywords: language model, gpt, prompt
• Abstract: Answering multi-hop reasoning questions requires retrieving and synthesizing information from diverse sources. Language models (LMs) struggle to perform such reasoning consistently. We propose an approach to pinpoint and rectify multi-hop reasoning failures through targeted memory injections on LM attention heads. First, we analyze the per-layer activations of GPT-2 models in response to single- and multi-hop prompts. We then propose a mechanism that allows users to inject relevant prompt-specific information, which we refer to as "memories," at critical LM locations during inference. By thus enabling the LM to incorporate additional relevant information during inference, we enhance the quality of multi-hop prompt completions. We empirically show that a simple, efficient, and targeted memory injection into a key attention layer often increases the probability of the desired next token in multi-hop tasks, by up to 424%. We observe that small subsets of attention heads can significantly impact the model prediction during multi-hop reasoning. To more faithfully interpret these heads, we develop Attention Lens: an open source tool that translates the outputs of attention heads into vocabulary tokens via learned transformations called lenses. We demonstrate the use of lenses to reveal how a model arrives at its answer and use them to localize sources of model failures such as in the case of biased and malicious language generation.
• 摘要：回答多跳推理问题需要从各种来源检索和综合信息。语言模型 (LM) 很难始终如一地执行此类推理。我们提出了一种方法，通过在 LM 注意力头上进行有针对性的记忆注入来查明和纠正多跳推理失败。首先，我们分析 GPT-2 模型对单跳和多跳提示的每层激活。然后，我们提出了一种机制，允许用户在推理过程中在关键的 LM 位置注入相关的提示特定信息（我们称之为“记忆”）。通过这种方式，使 LM 能够在推理过程中整合额外的相关信息，我们提高了多跳提示完成的质量。我们通过经验表明，在关键注意力层中进行简单、高效且有针对性的记忆注入通常会将多跳任务中所需下一个标记的概率提高高达 424%。我们观察到，注意力头的小子集可以显著影响多跳推理过程中的模型预测。为了更忠实地解释这些注意力头，我们开发了 Attention Lens：一种开源工具，它通过学习到的转换（称为 lens）将注意力头的输出转换为词汇标记。我们演示了如何使用 lens 来揭示模型如何得出答案，并使用它们来定位模型失败的根源，例如在有偏见和恶意的语言生成的情况下。

Title: YouTube Comments Decoded: Leveraging LLMs for Low Resource Language Classification

• Authors: Aniket Deroy, Subhankar Maity
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.05039
• Pdf URL: https://arxiv.org/pdf/2411.05039
• Copy Paste: [[2411.05039]] YouTube Comments Decoded: Leveraging LLMs for Low Resource Language Classification(https://arxiv.org/abs/2411.05039)
• Keywords: language model, gpt, llm, prompt
• Abstract: Sarcasm detection is a significant challenge in sentiment analysis, particularly due to its nature of conveying opinions where the intended meaning deviates from the literal expression. This challenge is heightened in social media contexts where code-mixing, especially in Dravidian languages, is prevalent. Code-mixing involves the blending of multiple languages within a single utterance, often with non-native scripts, complicating the task for systems trained on monolingual data. This shared task introduces a novel gold standard corpus designed for sarcasm and sentiment detection within code-mixed texts, specifically in Tamil-English and Malayalam-English languages. The primary objective of this task is to identify sarcasm and sentiment polarity within a code-mixed dataset of Tamil-English and Malayalam-English comments and posts collected from social media platforms. Each comment or post is annotated at the message level for sentiment polarity, with particular attention to the challenges posed by class imbalance, reflecting real-world this http URL this work, we experiment with state-of-the-art large language models like GPT-3.5 Turbo via prompting to classify comments into sarcastic or non-sarcastic categories. We obtained a macro-F1 score of 0.61 for Tamil language. We obtained a macro-F1 score of 0.50 for Malayalam language.
• 摘要：讽刺检测是情绪分析中的一项重大挑战，特别是因为讽刺的本质是传达与字面表达不同的观点。在社交媒体环境中，这种挑战更加严峻，因为社交媒体中代码混合（尤其是德拉威语）非常普遍。代码混合涉及在单一话语中混合多种语言，通常使用非母语脚本，这使得使用单语数据进行训练的系统的任务更加复杂。这项共享任务引入了一种新的黄金标准语料库，专为代码混合文本（特别是泰米尔语-英语和马拉雅拉姆语-英语）中的讽刺和情绪检测而设计。这项任务的主要目标是从社交媒体平台收集的泰米尔语-英语和马拉雅拉姆语-英语评论和帖子的代码混合数据集中识别讽刺和情绪极性。每条评论或帖子都在消息级别标注了情绪极性，特别关注类别不平衡带来的挑战，反映了现实世界的这一 http URL 这项工作，我们通过提示将评论分为讽刺或非讽刺类别，尝试了最先进的大型语言模型（如 GPT-3.5 Turbo）。我们获得了泰米尔语的宏观 F1 分数 0.61。我们获得了马拉雅拉姆语的宏观 F1 分数 0.50。

Title: Bottom-Up and Top-Down Analysis of Values, Agendas, and Observations in Corpora and LLMs

• Authors: Scott E. Friedman, Noam Benkler, Drisana Mosaphir, Jeffrey Rye, Sonja M. Schmer-Galunder, Micah Goldwater, Matthew McLure, Ruta Wheelock, Jeremy Gottlieb, Robert P. Goldman, Christopher Miller
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.05040
• Pdf URL: https://arxiv.org/pdf/2411.05040
• Copy Paste: [[2411.05040]] Bottom-Up and Top-Down Analysis of Values, Agendas, and Observations in Corpora and LLMs(https://arxiv.org/abs/2411.05040)
• Keywords: language model, llm, prompt
• Abstract: Large language models (LLMs) generate diverse, situated, persuasive texts from a plurality of potential perspectives, influenced heavily by their prompts and training data. As part of LLM adoption, we seek to characterize - and ideally, manage - the socio-cultural values that they express, for reasons of safety, accuracy, inclusion, and cultural fidelity. We present a validated approach to automatically (1) extracting heterogeneous latent value propositions from texts, (2) assessing resonance and conflict of values with texts, and (3) combining these operations to characterize the pluralistic value alignment of human-sourced and LLM-sourced textual data.
• 摘要：大型语言模型 (LLM) 从多个潜在视角生成多样化、情境化、有说服力的文本，这些文本受到提示和训练数据的严重影响。作为 LLM 采用的一部分，我们寻求表征（理想情况下是管理）它们所表达的社会文化价值观，以确保安全性、准确性、包容性和文化保真度。我们提出了一种经过验证的方法，可以自动 (1) 从文本中提取异质潜在价值主张，(2) 评估文本中价值观的共鸣和冲突，以及 (3) 结合这些操作来表征人类来源和 LLM 来源文本数据的多元价值观一致性。

Title: Improving Radiology Report Conciseness and Structure via Local Large Language Models

• Authors: Iryna Hartsock, Cyrillo Araujo, Les Folio, Ghulam Rasool
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.05042
• Pdf URL: https://arxiv.org/pdf/2411.05042
• Copy Paste: [[2411.05042]] Improving Radiology Report Conciseness and Structure via Local Large Language Models(https://arxiv.org/abs/2411.05042)
• Keywords: language model, llm, prompt
• Abstract: In this study, we aim to enhance radiology reporting by improving both the conciseness and structured organization of findings (also referred to as templating), specifically by organizing information according to anatomical regions. This structured approach allows physicians to locate relevant information quickly, increasing the report's utility. We utilize Large Language Models (LLMs) such as Mixtral, Mistral, and Llama to generate concise, well-structured reports. Among these, we primarily focus on the Mixtral model due to its superior adherence to specific formatting requirements compared to other models. To maintain data security and privacy, we run these LLMs locally behind our institution's firewall. We leverage the LangChain framework and apply five distinct prompting strategies to enforce a consistent structure in radiology reports, aiming to eliminate extraneous language and achieve a high level of conciseness. We also introduce a novel metric, the Conciseness Percentage (CP) score, to evaluate report brevity. Our dataset comprises 814 radiology reports authored by seven board-certified body radiologists at our cancer center. In evaluating the different prompting methods, we discovered that the most effective approach for generating concise, well-structured reports involves first instructing the LLM to condense the report, followed by a prompt to structure the content according to specific guidelines. We assessed all prompting strategies based on their ability to handle formatting issues, reduce report length, and adhere to formatting instructions. Our findings demonstrate that open-source, locally deployed LLMs can significantly improve radiology report conciseness and structure while conforming to specified formatting standards.
• 摘要：在本研究中，我们旨在通过提高发现的简洁性和结构化组织（也称为模板化）来增强放射学报告，特别是通过根据解剖区域组织信息。这种结构化方法使医生能够快速找到相关信息，从而提高报告的实用性。我们利用大型语言模型 (LLM)，例如 Mixtral、Mistral 和 Llama 来生成简洁、结构良好的报告。其中，​​我们主要关注 Mixtral 模型，因为与其他模型相比，它更符合特定的格式要求。为了维护数据安全和隐私，我们在机构的防火墙后面本地运行这些 LLM。我们利用 LangChain 框架并应用五种不同的提示策略来强制放射学报告中的一致结构，旨在消除无关语言并实现高水平的简洁性。我们还引入了一个新指标，即简洁性百分比 (CP) 分数，以评估报告的简洁性。我们的数据集包含 814 份放射学报告，这些报告由我们癌症中心的七名获得委员会认证的身体放射科医生撰写。在评估不同的提示方法时，我们发现，生成简洁、结构良好的报告的最有效方法是首先指示 LLM 压缩报告，然后提示根据特定指南构建内容。我们根据所有提示策略处理格式问题、缩短报告长度和遵守格式说明的能力对其进行了评估。我们的研究结果表明，开源、本地部署的 LLM 可以显著提高放射学报告的简洁性和结构，同时符合指定的格式标准。

Title: Performance-Guided LLM Knowledge Distillation for Efficient Text Classification at Scale

• Authors: Flavio Di Palo, Prateek Singhi, Bilal Fadlallah
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.05045
• Pdf URL: https://arxiv.org/pdf/2411.05045
• Copy Paste: [[2411.05045]] Performance-Guided LLM Knowledge Distillation for Efficient Text Classification at Scale(https://arxiv.org/abs/2411.05045)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) face significant challenges at inference time due to their high computational demands. To address this, we present Performance-Guided Knowledge Distillation (PGKD), a cost-effective and high-throughput solution for production text classification applications. PGKD utilizes teacher-student Knowledge Distillation to distill the knowledge of LLMs into smaller, task-specific models. PGKD establishes an active learning routine between the student model and the LLM; the LLM continuously generates new training data leveraging hard-negative mining, student model validation performance, and early-stopping protocols to inform the data generation. By employing a cyclical, performance-aware approach tailored for highly multi-class, sparsely annotated datasets prevalent in industrial text classification, PGKD effectively addresses training challenges and outperforms traditional BERT-base models and other knowledge distillation methods on several multi-class classification datasets. Additionally, cost and latency benchmarking reveals that models fine-tuned with PGKD are up to 130X faster and 25X less expensive than LLMs for inference on the same classification task. While PGKD is showcased for text classification tasks, its versatile framework can be extended to any LLM distillation task, including language generation, making it a powerful tool for optimizing performance across a wide range of AI applications.
• 摘要：大型语言模型 (LLM) 由于计算需求高，在推理时面临巨大挑战。为了解决这个问题，我们提出了性能引导知识提炼 (PGKD)，这是一种经济高效且高吞吐量的生产文本分类应用解决方案。PGKD 利用师生知识提炼将 LLM 的知识提炼成更小的、特定于任务的模型。PGKD 在学生模型和 LLM 之间建立了主动学习程序；LLM 利用硬负挖掘、学生模型验证性能和早期停止协议不断生成新的训练数据来指导数据生成。通过采用针对工业文本分类中普遍存在的多类稀疏注释数据集而量身定制的循环、性能感知方法，PGKD 有效地解决了训练挑战，并且在多个多类分类数据集上的表现优于传统的 BERT 基模型和其他知识提炼方法。此外，成本和延迟基准测试表明，使用 PGKD 微调的模型在相同分类任务上的推理速度比 LLM 快 130 倍，成本低 25 倍。虽然 PGKD 主要用于文本分类任务，但其多功能框架可以扩展到任何 LLM 提炼任务，包括语言生成，使其成为优化各种 AI 应用程序性能的强大工具。

Title: PhoneLM:an Efficient and Capable Small Language Model Family through Principled Pre-training

• Authors: Rongjie Yi, Xiang Li, Weikai Xie, Zhenyan Lu, Chenghua Wang, Ao Zhou, Shangguang Wang, Xiwen Zhang, Mengwei Xu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.05046
• Pdf URL: https://arxiv.org/pdf/2411.05046
• Copy Paste: [[2411.05046]] PhoneLM:an Efficient and Capable Small Language Model Family through Principled Pre-training(https://arxiv.org/abs/2411.05046)
• Keywords: language model
• Abstract: The interest in developing small language models (SLM) for on-device deployment is fast growing. However, the existing SLM design hardly considers the device hardware characteristics. Instead, this work presents a simple yet effective principle for SLM design: architecture searching for (near-)optimal runtime efficiency before pre-training. Guided by this principle, we develop PhoneLM SLM family (currently with 0.5B and 1.5B versions), that acheive the state-of-the-art capability-efficiency tradeoff among those with similar parameter size. We fully open-source the code, weights, and training datasets of PhoneLM for reproducibility and transparency, including both base and instructed versions. We also release a finetuned version of PhoneLM capable of accurate Android Intent invocation, and an end-to-end Android demo. All materials are available at this https URL.
• 摘要：人们对开发用于设备部署的小型语言模型 (SLM) 的兴趣正在快速增长。然而，现有的 SLM 设计几乎不考虑设备硬件特性。相反，这项工作提出了一个简单而有效的 SLM 设计原则：在预训练之前，架构搜索 (接近) 最佳运行时效率。在这一原则的指导下，我们开发了 PhoneLM SLM 系列（目前有 0.5B 和 1.5B 版本），在具有相似参数大小的 SLM 中实现了最先进的能力效率权衡。我们完全开源了 PhoneLM 的代码、权重和训练数据集，以实现可重复性和透明度，包括基本版本和指导版本。我们还发布了能够准确调用 Android Intent 的经过微调的 PhoneLM 版本，以及端到端 Android 演示。所有材料均可在此 https URL 上找到。

Title: Leveraging LLMs to Enable Natural Language Search on Go-to-market Platforms

• Authors: Jesse Yao, Saurav Acharya, Priyaranjan Parida, Srinivas Attipalli, Ali Dasdan
• Subjects: cs.CL, cs.AI, cs.DB, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.05048
• Pdf URL: https://arxiv.org/pdf/2411.05048
• Copy Paste: [[2411.05048]] Leveraging LLMs to Enable Natural Language Search on Go-to-market Platforms(https://arxiv.org/abs/2411.05048)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: Enterprise searches require users to have complex knowledge of queries, configurations, and metadata, rendering it difficult for them to access information as needed. Most go-to-market (GTM) platforms utilize advanced search, an interface that enables users to filter queries by various fields using categories or keywords, which, historically, however, has proven to be exceedingly cumbersome, as users are faced with seemingly hundreds of options, fields, and buttons. Consequently, querying with natural language has long been ideal, a notion further empowered by Large Language Models (LLMs). In this paper, we implement and evaluate a solution for the Zoominfo product for sellers, which prompts the LLM with natural language, producing search fields through entity extraction that are then converted into a search query. The intermediary search fields offer numerous advantages for each query, including the elimination of syntax errors, simpler ground truths, and an intuitive format for the LLM to interpret. We paired this pipeline with many advanced prompt engineering strategies, featuring an intricate system message, few-shot prompting, chain-of-thought (CoT) reasoning, and execution refinement. Furthermore, we manually created the ground truth for 500+ natural language queries, enabling the supervised fine-tuning of Llama-3-8B-Instruct and the introduction of sophisticated numerical metrics. Comprehensive experiments with closed, open source, and fine-tuned LLM models were conducted through exact, Jaccard, cosine, and semantic similarity on individual search entities to demonstrate the efficacy of our approach. Overall, the most accurate closed model had an average accuracy of 97% per query, with only one field performing under 90%, with comparable results observed from the fine-tuned models.
• 摘要：企业搜索要求用户具备复杂的查询、配置和元数据知识，这使得他们难以根据需要访问信息。大多数上市 (GTM) 平台都使用高级搜索，这是一种界面，允许用户使用类别或关键字按各种字段过滤查询，但从历史上看，这已被证明是极其麻烦的，因为用户面临着看似数百个选项、字段和按钮。因此，使用自然语言查询长期以来一直是理想的选择，大型语言模型 (LLM) 进一步增强了这一理念。在本文中，我们为卖家实施并评估了 Zoominfo 产品的解决方案，该产品使用自然语言提示 LLM，通过实体提取生成搜索字段，然后将其转换为搜索查询。中间搜索字段为每个查询提供了许多优势，包括消除语法错误、更简单的基本事实以及 LLM 解释的直观格式。我们将此管道与许多高级提示工程策略配对，具有复杂的系统消息、少量提示、思路链 (CoT) 推理和执行细化。此外，我们手动创建了 500 多个自然语言查询的基本事实，从而实现了 Llama-3-8B-Instruct 的监督微调，并引入了复杂的数值指标。通过对单个搜索实体的精确、Jaccard、余弦和语义相似性，对封闭、开源和微调的 LLM 模型进行了全面的实验，以证明我们方法的有效性。总体而言，最准确的封闭模型对每个查询的平均准确率为 97%，只有一个字段的表现低于 90%，与微调模型的结果相当。

Title: ProverbEval: Exploring LLM Evaluation Challenges for Low-resource Language Understanding

• Authors: Israel Abebe Azime, Atnafu Lambebo Tonja, Tadesse Destaw Belay, Yonas Chanie, Bontu Fufa Balcha, Negasi Haile Abadi, Henok Biadglign Ademtew, Mulubrhan Abebe Nerea, Debela Desalegn Yadeta, Derartu Dagne Geremew, Assefa Atsbiha tesfau, Philipp Slusallek, Thamar Solorio, Dietrich Klakow
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.05049
• Pdf URL: https://arxiv.org/pdf/2411.05049
• Copy Paste: [[2411.05049]] ProverbEval: Exploring LLM Evaluation Challenges for Low-resource Language Understanding(https://arxiv.org/abs/2411.05049)
• Keywords: llm, prompt
• Abstract: With the rapid development of evaluation datasets to assess LLMs understanding across a wide range of subjects and domains, identifying a suitable language understanding benchmark has become increasingly challenging. In this work, we explore LLM evaluation challenges for low-resource language understanding and introduce ProverbEval, LLM evaluation benchmark for low-resource languages based on proverbs to focus on low-resource language understanding in culture-specific scenarios. We benchmark various LLMs and explore factors that create variability in the benchmarking process. We observed performance variances of up to 50%, depending on the order in which answer choices were presented in multiple-choice tasks. Native language proverb descriptions significantly improve tasks such as proverb generation, contributing to improved outcomes. Additionally, monolingual evaluations consistently outperformed their cross-lingual counterparts. We argue special attention must be given to the order of choices, choice of prompt language, task variability, and generation tasks when creating LLM evaluation benchmarks.
• 摘要：随着评估数据集的快速发展，用于评估广泛学科和领域的 LLM 理解，确定合适的语言理解基准变得越来越具有挑战性。在这项工作中，我们探索了低资源语言理解的 LLM 评估挑战，并引入了基于谚语的低资源语言的 LLM 评估基准 ProverbEval，以关注特定文化场景中的低资源语言理解。我们对各种 LLM 进行了基准测试，并探索了导致基准测试过程中出现变化的因素。我们观察到性能差异高达 50%，具体取决于多项选择任务中答案选项的呈现顺序。母语谚语描述可显著改善谚语生成等任务，从而有助于改善结果。此外，单语评估始终优于跨语言评估。我们认为，在创建 LLM 评估基准时，必须特别注意选择的顺序、提示语言的选择、任务变化和生成任务。

Title: Selecting Between BERT and GPT for Text Classification in Political Science Research

• Authors: Yu Wang, Wen Qu, Xin Ye
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.05050
• Pdf URL: https://arxiv.org/pdf/2411.05050
• Copy Paste: [[2411.05050]] Selecting Between BERT and GPT for Text Classification in Political Science Research(https://arxiv.org/abs/2411.05050)
• Keywords: gpt, prompt
• Abstract: Political scientists often grapple with data scarcity in text classification. Recently, fine-tuned BERT models and their variants have gained traction as effective solutions to address this issue. In this study, we investigate the potential of GPT-based models combined with prompt engineering as a viable alternative. We conduct a series of experiments across various classification tasks, differing in the number of classes and complexity, to evaluate the effectiveness of BERT-based versus GPT-based models in low-data scenarios. Our findings indicate that while zero-shot and few-shot learning with GPT models provide reasonable performance and are well-suited for early-stage research exploration, they generally fall short - or, at best, match - the performance of BERT fine-tuning, particularly as the training set reaches a substantial size (e.g., 1,000 samples). We conclude by comparing these approaches in terms of performance, ease of use, and cost, providing practical guidance for researchers facing data limitations. Our results are particularly relevant for those engaged in quantitative text analysis in low-resource settings or with limited labeled data.
• 摘要：政治科学家经常努力解决文本分类中的数据稀缺问题。最近，经过微调的 BERT 模型及其变体已成为解决此问题的有效解决方案。在本研究中，我们研究了基于 GPT 的模型与快速工程相结合作为可行替代方案的潜力。我们在各种分类任务中进行了一系列实验，这些任务的类别数量和复杂度不同，以评估基于 BERT 的模型与基于 GPT 的模型在低数据场景中的有效性。我们的研究结果表明，虽然使用 GPT 模型进行零样本和少样本学习提供了合理的性能并且非常适合早期研究探索，但它们通常达不到（或充其量只能匹配）BERT 微调的性能，特别是当训练集达到相当大的规模（例如 1,000 个样本）时。最后，我们通过比较这些方法的性能、易用性和成本，为面临数据限制的研究人员提供实用指导。我们的结果对于那些在资源匮乏或标记数据有限的情况下从事定量文本分析的人特别有用。

Title: FMEA Builder: Expert Guided Text Generation for Equipment Maintenance

• Authors: Karol Lynch, Fabio Lorenzi, John Sheehan, Duygu Kabakci-Zorlu, Bradley Eck
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.05054
• Pdf URL: https://arxiv.org/pdf/2411.05054
• Copy Paste: [[2411.05054]] FMEA Builder: Expert Guided Text Generation for Equipment Maintenance(https://arxiv.org/abs/2411.05054)
• Keywords: language model
• Abstract: Foundation models show great promise for generative tasks in many domains. Here we discuss the use of foundation models to generate structured documents related to critical assets. A Failure Mode and Effects Analysis (FMEA) captures the composition of an asset or piece of equipment, the ways it may fail and the consequences thereof. Our system uses large language models to enable fast and expert supervised generation of new FMEA documents. Empirical analysis shows that foundation models can correctly generate over half of an FMEA's key content. Results from polling audiences of reliability professionals show a positive outlook on using generative AI to create these documents for critical assets.
• 摘要：基础模型在许多领域的生成任务中都大有可为。我们在这里讨论如何使用基础模型生成与关键资产相关的结构化文档。故障模式和影响分析 (FMEA) 可捕获资产或设备的组成、可能发生故障的方式及其后果。我们的系统使用大型语言模型来实现快速且专家监督的新 FMEA 文档生成。实证分析表明，基础模型可以正确生成超过一半的 FMEA 关键内容。对可靠性专业人员的调查结果显示，人们对使用生成式 AI 为关键资产创建这些文档持积极态度。

Title: FineTuneBench: How well do commercial fine-tuning APIs infuse knowledge into LLMs?

• Authors: Eric Wu, Kevin Wu, James Zou
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2411.05059
• Pdf URL: https://arxiv.org/pdf/2411.05059
• Copy Paste: [[2411.05059]] FineTuneBench: How well do commercial fine-tuning APIs infuse knowledge into LLMs?(https://arxiv.org/abs/2411.05059)
• Keywords: language model, gpt, llm
• Abstract: There is great interest in fine-tuning frontier large language models (LLMs) to inject new information and update existing knowledge. While commercial LLM fine-tuning APIs from providers such as OpenAI and Google promise flexible adaptation for various applications, the efficacy of fine-tuning remains unclear. In this study, we introduce FineTuneBench, an evaluation framework and dataset for understanding how well commercial fine-tuning APIs can successfully learn new and updated knowledge. We analyze five frontier LLMs with commercially available fine-tuning APIs, including GPT-4o and Gemini 1.5 Pro, on their effectiveness in two settings: (1) ingesting novel information, such as recent news events and new people profiles, and (2) updating existing knowledge, such as updated medical guidelines and code frameworks. Our results reveal substantial shortcomings in all the models' abilities to effectively learn new information through fine-tuning, with an average generalization accuracy of 37% across all models. When updating existing knowledge, such as incorporating medical guideline updates, commercial fine-tuning APIs show even more limited capability (average generalization accuracy of 19%). Overall, fine-tuning GPT-4o mini is the most effective for infusing new knowledge and updating knowledge, followed by GPT-3.5 Turbo and GPT-4o. The fine-tuning APIs for Gemini 1.5 Flesh and Gemini 1.5 Pro are unable to learn new knowledge or update existing knowledge. These findings underscore a major shortcoming in using current commercial fine-tuning services to achieve reliable knowledge infusion in common scenarios. We open source the FineTuneBench dataset at this https URL.
• 摘要：人们对微调前沿大型语言模型 (LLM) 以注入新信息和更新现有知识非常感兴趣。虽然 OpenAI 和 Google 等提供商提供的商业 LLM 微调 API 承诺可以灵活适应各种应用，但微调的有效性仍不清楚。在本研究中，我们引入了 FineTuneBench，这是一个评估框架和数据集，用于了解商业微调 API 成功学习新知识和更新知识的能力。我们分析了五种具有商用微调 API 的前沿 LLM，包括 GPT-4o 和 Gemini 1.5 Pro，以了解它们在两种环境下的有效性：(1) 提取新信息，例如最近的新闻事件和新的人物资料，以及 (2) 更新现有知识，例如更新的医疗指南和代码框架。我们的结果揭示了所有模型通过微调有效学习新信息的能力都存在重大缺陷，所有模型的平均泛化准确率为 37%。在更新现有知识（例如纳入医疗指南更新）时，商业微调 API 的能力更加有限（平均泛化准确率为 19%）。总体而言，微调 GPT-4o mini 对注入新知识和更新知识最有效，其次是 GPT-3.5 Turbo 和 GPT-4o。Gemini 1.5 Flesh 和 Gemini 1.5 Pro 的微调 API 无法学习新知识或更新现有知识。这些发现凸显了使用当前商业微调服务在常见场景中实现可靠知识注入的重大缺陷。我们在此 https URL 上开源了 FineTuneBench 数据集。

Title: ImpScore: A Learnable Metric For Quantifying The Implicitness Level of Language

• Authors: Yuxin Wang, Xiaomeng Zhu, Weimin Lyu, Saeed Hassanpour, Soroush Vosoughi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.05172
• Pdf URL: https://arxiv.org/pdf/2411.05172
• Copy Paste: [[2411.05172]] ImpScore: A Learnable Metric For Quantifying The Implicitness Level of Language(https://arxiv.org/abs/2411.05172)
• Keywords: language model
• Abstract: Handling implicit language is essential for natural language processing systems to achieve precise text understanding and facilitate natural interactions with users. Despite its importance, the absence of a robust metric for accurately measuring the implicitness of language significantly constrains the depth of analysis possible in evaluating models' comprehension capabilities. This paper addresses this gap by developing a scalar metric that quantifies the implicitness level of language without relying on external references. Drawing on principles from traditional linguistics, we define ''implicitness'' as the divergence between semantic meaning and pragmatic interpretation. To operationalize this definition, we introduce ImpScore, a novel, reference-free metric formulated through an interpretable regression model. This model is trained using pairwise contrastive learning on a specially curated dataset comprising $112,580$ (implicit sentence, explicit sentence) pairs. We validate ImpScore through a user study that compares its assessments with human evaluations on out-of-distribution data, demonstrating its accuracy and strong correlation with human judgments. Additionally, we apply ImpScore to hate speech detection datasets, illustrating its utility and highlighting significant limitations in current large language models' ability to understand highly implicit content. The metric model and its training data are available at this https URL.
• 摘要：处理隐性语言对于自然语言处理系统实现精确的文本理解和促进与用户的自然交互至关重要。尽管它很重要，但缺乏一个可靠的指标来准确衡量语言的隐性，这严重限制了评估模型理解能力的分析深度。本文通过开发一个标量指标来解决这一差距，该指标可以量化语言的隐性水平，而无需依赖外部参考。借鉴传统语言学的原理，我们将“隐性”定义为语义意义和语用解释之间的差异。为了使这个定义可操作化，我们引入了 ImpScore，这是一种通过可解释的回归模型制定的新颖的无参考指标。该模型使用成对对比学习在一个特别策划的数据集上进行训练，该数据集包含 112,580 个（隐性句子，显性句子）对。我们通过一项用户研究来验证 ImpScore，该研究将其评估与人类对分布外数据的评估进行比较，证明了其准确性和与人类判断的强相关性。此外，我们将 ImpScore 应用于仇恨言论检测数据集，展示了它的实用性，并强调了当前大型语言模型在理解高度隐性内容方面的能力存在重大局限性。度量模型及其训练数据可在此 https URL 上找到。

Title: Explaining Mixtures of Sources in News Articles

• Authors: Alexander Spangher, James Youn, Matt DeButts, Nanyun Peng, Emilio Ferrara, Jonathan May
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.05192
• Pdf URL: https://arxiv.org/pdf/2411.05192
• Copy Paste: [[2411.05192]] Explaining Mixtures of Sources in News Articles(https://arxiv.org/abs/2411.05192)
• Keywords: language model, llm
• Abstract: Human writers plan, then write. For large language models (LLMs) to play a role in longer-form article generation, we must understand the planning steps humans make before writing. We explore one kind of planning, source-selection in news, as a case-study for evaluating plans in long-form generation. We ask: why do specific stories call for specific kinds of sources? We imagine a generative process for story writing where a source-selection schema is first selected by a journalist, and then sources are chosen based on categories in that schema. Learning the article's plan means predicting the schema initially chosen by the journalist. Working with professional journalists, we adapt five existing schemata and introduce three new ones to describe journalistic plans for the inclusion of sources in documents. Then, inspired by Bayesian latent-variable modeling, we develop metrics to select the most likely plan, or schema, underlying a story, which we use to compare schemata. We find that two schemata: stance and social affiliation best explain source plans in most documents. However, other schemata like textual entailment explain source plans in factually rich topics like "Science". Finally, we find we can predict the most suitable schema given just the article's headline with reasonable accuracy. We see this as an important case-study for human planning, and provides a framework and approach for evaluating other kinds of plans. We release a corpora, NewsSources, with annotations for 4M articles.
• 摘要：人类作家先计划，然后写作。为了让大型语言模型 (LLM) 在长篇文章生成中发挥作用，我们必须了解人类在写作之前所做的计划步骤。我们探讨了一种计划，即新闻中的来源选择，作为评估长篇文章生成计划的案例研究。我们问：为什么特定的故事需要特定类型的来源？我们想象一个故事写作的生成过程，其中记者首先选择来源选择模式，然后根据该模式中的类别选择来源。学习文章的计划意味着预测记者最初选择的模式。我们与专业记者合作，调整了五个现有模式并引入了三个新模式来描述新闻报道中纳入来源的计划。然后，受贝叶斯潜在变量模型的启发，我们开发了指标来选择故事中最可能的计划或模式，我们用它来比较模式。我们发现两个模式：立场和社会关系最能解释大多数文件中的来源计划。然而，其他模式，如文本蕴涵，解释了“科学”等事实丰富的主题中的源计划。最后，我们发现，我们可以在仅给出文章标题的情况下以合理的准确度预测最合适的模式。我们认为这是人类规划的重要案例研究，并为评估其他类型的计划提供了框架和方法。我们发布了一个语料库 NewsSources，其中包含 4M 篇文章的注释。

Title: CodeLutra: Boosting LLM Code Generation via Preference-Guided Refinement

• Authors: Leitian Tao, Xiang Chen, Tong Yu, Tung Mai, Ryan Rossi, Yixuan Li, Saayan Mitra
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.05199
• Pdf URL: https://arxiv.org/pdf/2411.05199
• Copy Paste: [[2411.05199]] CodeLutra: Boosting LLM Code Generation via Preference-Guided Refinement(https://arxiv.org/abs/2411.05199)
• Keywords: language model, gpt, llm
• Abstract: Large Language Models (LLMs) have significantly advanced code generation but often require substantial resources and tend to over-generalize, limiting their efficiency for specific tasks. Fine-tuning smaller, open-source LLMs presents a viable alternative; however, it typically lags behind cutting-edge models due to supervised fine-tuning's reliance solely on correct code examples, which restricts the model's ability to learn from its own mistakes and adapt to diverse programming challenges. To bridge this gap, we introduce CodeLutra, a novel framework that enhances low-performing LLMs by leveraging both successful and failed code generation attempts. Unlike conventional fine-tuning, CodeLutra employs an iterative preference learning mechanism to compare correct and incorrect solutions as well as maximize the likelihood of correct codes. Through continuous iterative refinement, CodeLutra enables smaller LLMs to match or surpass GPT-4's performance in various code generation tasks without relying on vast external datasets or larger auxiliary models. On a challenging data analysis task, using just 500 samples improved Llama-3-8B's accuracy from 28.2% to 48.6%, approaching GPT-4's performance. These results highlight CodeLutra's potential to close the gap between open-source and closed-source models, making it a promising approach in the field of code generation.
• 摘要：大型语言模型 (LLM) 具有显著先进的代码生成能力，但通常需要大量资源并且倾向于过度概括，从而限制了它们在特定任务中的效率。对较小的开源 LLM 进行微调是一种可行的替代方案；然而，由于监督微调仅依赖于正确的代码示例，因此它通常落后于尖端模型，这限制了模型从自身错误中学习和适应各种编程挑战的能力。为了弥补这一差距，我们引入了 CodeLutra，这是一个新颖的框架，它通过利用成功和失败的代码生成尝试来增强性能低下的 LLM。与传统的微调不同，CodeLutra 采用迭代偏好学习机制来比较正确和错误的解决方案，并最大化正确代码的可能性。通过不断的迭代改进，CodeLutra 使较小的 LLM 能够在各种代码生成任务中达到或超越 GPT-4 的性能，而无需依赖庞大的外部数据集或更大的辅助模型。在一项具有挑战性的数据分析任务中，仅使用 500 个样本就将 Llama-3-8B 的准确率从 28.2% 提高到了 48.6%，接近 GPT-4 的性能。这些结果凸显了 CodeLutra 缩小开源和闭源模型之间差距的潜力，使其成为代码生成领域的一种有前途的方法。

Title: STAND-Guard: A Small Task-Adaptive Content Moderation Model

• Authors: Minjia Wang, Pingping Lin, Siqi Cai, Shengnan An, Shengjie Ma, Zeqi Lin, Congrui Huang, Bixiong Xu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.05214
• Pdf URL: https://arxiv.org/pdf/2411.05214
• Copy Paste: [[2411.05214]] STAND-Guard: A Small Task-Adaptive Content Moderation Model(https://arxiv.org/abs/2411.05214)
• Keywords: language model, gpt
• Abstract: Content moderation, the process of reviewing and monitoring the safety of generated content, is important for development of welcoming online platforms and responsible large language models. Content moderation contains various tasks, each with its unique requirements tailored to specific scenarios. Therefore, it is crucial to develop a model that can be easily adapted to novel or customized content moderation tasks accurately without extensive model tuning. This paper presents STAND-GUARD, a Small Task-Adaptive coNtent moDeration model. The basic motivation is: by performing instruct tuning on various content moderation tasks, we can unleash the power of small language models (SLMs) on unseen (out-of-distribution) content moderation tasks. We also carefully study the effects of training tasks and model size on the efficacy of cross-task fine-tuning mechanism. Experiments demonstrate STAND-Guard is comparable to GPT-3.5-Turbo across over 40 public datasets, as well as proprietary datasets derived from real-world business scenarios. Remarkably, STAND-Guard achieved nearly equivalent results to GPT-4-Turbo on unseen English binary classification tasks
• 摘要：内容审核是审查和监控生成内容安全性的过程，对于开发受欢迎的在线平台和负责任的大型语言模型非常重要。内容审核包含各种任务，每个任务都有针对特定场景量身定制的独特要求。因此，开发一种无需进行大量模型调整即可轻松准确地适应新颖或定制的内容审核任务的模型至关重要。本文介绍了一种小型任务自适应内容审核模型 STAND-GUARD。基本动机是：通过对各种内容审核任务进行指令调整，我们可以在看不见的（分布外的）内容审核任务上释放小型语言模型 (SLM) 的威力。我们还仔细研究了训练任务和模型大小对跨任务微调机制有效性的影响。实验表明，STAND-Guard 在 40 多个公共数据集以及来自实际业务场景的专有数据集上与 GPT-3.5-Turbo 相当。值得注意的是，STAND-Guard 在未见过的英语二元分类任务上取得了与 GPT-4-Turbo 几乎相当的结果

Title: CHATTER: A Character Attribution Dataset for Narrative Understanding

• Authors: Sabyasachee Baruah, Shrikanth Narayanan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.05227
• Pdf URL: https://arxiv.org/pdf/2411.05227
• Copy Paste: [[2411.05227]] CHATTER: A Character Attribution Dataset for Narrative Understanding(https://arxiv.org/abs/2411.05227)
• Keywords: language model, chat
• Abstract: Computational narrative understanding studies the identification, description, and interaction of the elements of a narrative: characters, attributes, events, and relations. Narrative research has given considerable attention to defining and classifying character types. However, these character-type taxonomies do not generalize well because they are small, too simple, or specific to a domain. We require robust and reliable benchmarks to test whether narrative models truly understand the nuances of the character's development in the story. Our work addresses this by curating the Chatter dataset that labels whether a character portrays some attribute for 88148 character-attribute pairs, encompassing 2998 characters, 13324 attributes and 660 movies. We validate a subset of Chatter, called ChatterEval, using human annotations to serve as an evaluation benchmark for the character attribution task in movie scripts. ChatterEval assesses narrative understanding and the long-context modeling capacity of language models.
• 摘要：计算叙事理解研究叙事元素的识别、描述和交互：角色、属性、事件和关系。叙事研究非常重视定义和分类角色类型。然而，这些角色类型分类法不能很好地概括，因为它们太小、太简单或特定于某个领域。我们需要强大而可靠的基准来测试叙事模型是否真正理解故事中角色发展的细微差别。我们的工作通过整理 Chatter 数据集来解决这个问题，该数据集标记角色是否描绘了 88148 个角色-属性对的某些属性，涵盖 2998 个角色、13324 个属性和 660 部电影。我们使用人工注释验证了 Chatter 的一个子集，称为 ChatterEval，作为电影剧本中角色归因任务的评估基准。ChatterEval 评估叙事理解和语言模型的长上下文建模能力。

Title: Abstract2Appendix: Academic Reviews Enhance LLM Long-Context Capabilities

• Authors: Shengzhi Li, Kittipat Kampa, Rongyu Lin, Bohang Li, Shichao Pei
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.05232
• Pdf URL: https://arxiv.org/pdf/2411.05232
• Copy Paste: [[2411.05232]] Abstract2Appendix: Academic Reviews Enhance LLM Long-Context Capabilities(https://arxiv.org/abs/2411.05232)
• Keywords: language model, gpt, llm
• Abstract: Large language models (LLMs) have shown remarkable performance across various tasks, yet their ability to handle long-context reading remains challenging. This study explores the effectiveness of leveraging high-quality academic peer review data for fine-tuning LLMs to enhance their long-context capabilities. We compare the Direct Preference Optimization (DPO) method with the Supervised Fine-Tuning (SFT) method, demonstrating DPO's superiority and data efficiency. Our experiments show that the fine-tuned model achieves a 4.04-point improvement over phi-3 and a 2.6\% increase on the Qasper benchmark using only 2000 samples. Despite facing limitations in data scale and processing costs, this study underscores the potential of DPO and high-quality data in advancing LLM performance. Additionally, the zero-shot benchmark results indicate that aggregated high-quality human reviews are overwhelmingly preferred over LLM-generated responses, even for the most capable models like GPT-4o. This suggests that high-quality human reviews are extremely rich in information, reasoning, and long-context retrieval, capabilities that even the most advanced models have not fully captured. These findings highlight the high utility of leveraging human reviews to further advance the field.
• 摘要：大型语言模型 (LLM) 在各种任务中都表现出色，但它们处理长上下文阅读的能力仍然具有挑战性。本研究探讨了利用高质量学术同行评审数据对 LLM 进行微调以增强其长上下文能力的有效性。我们将直接偏好优化 (DPO) 方法与监督微调 (SFT) 方法进行了比较，证明了 DPO 的优越性和数据效率。我们的实验表明，仅使用 2000 个样本，微调后的模型就比 phi-3 提高了 4.04 分，在 Qasper 基准上提高了 2.6%。尽管面临数据规模和处理成本的限制，但本研究强调了 DPO 和高质量数据在提高 LLM 性能方面的潜力。此外，零样本基准测试结果表明，即使对于 GPT-4o 等最强大的模型来说，汇总的高质量人工评论也比 LLM 生成的回复更受青睐。这表明，高质量的人工评论具有极其丰富的信息、推理和长上下文检索能力，即使是最先进的模型也未能完全捕捉到这些能力。这些发现凸显了利用人工评论来进一步推动该领域发展的巨大效用。

Title: What talking you?: Translating Code-Mixed Messaging Texts to English

• Authors: Lynnette Hui Xian Ng, Luo Qi Chan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.05253
• Pdf URL: https://arxiv.org/pdf/2411.05253
• Copy Paste: [[2411.05253]] What talking you?: Translating Code-Mixed Messaging Texts to English(https://arxiv.org/abs/2411.05253)
• Keywords: language model, llm, prompt
• Abstract: Translation of code-mixed texts to formal English allow a wider audience to understand these code-mixed languages, and facilitate downstream analysis applications such as sentiment analysis. In this work, we look at translating Singlish, which is colloquial Singaporean English, to formal standard English. Singlish is formed through the code-mixing of multiple Asian languages and dialects. We analysed the presence of other Asian languages and variants which can facilitate translation. Our dataset is short message texts, written as informal communication between Singlish speakers. We use a multi-step prompting scheme on five Large Language Models (LLMs) for language detection and translation. Our analysis show that LLMs do not perform well in this task, and we describe the challenges involved in translation of code-mixed languages. We also release our dataset in this link this https URL.
• 摘要：将混合代码文本翻译成正式英语可以让更多受众理解这些混合代码语言，并促进下游分析应用，例如情绪分析。在这项工作中，我们着眼于将新加坡英语（口语化的新加坡英语）翻译成正式的标准英语。新加坡英语是由多种亚洲语言和方言混合而成的。我们分析了其他亚洲语言和变体的存在，以促进翻译。我们的数据集是短信文本，以新加坡英语使用者之间的非正式交流形式书写。我们在五个大型语言模型 (LLM) 上使用多步骤提示方案进行语言检测和翻译。我们的分析表明 LLM 在这项任务中表现不佳，我们描述了混合代码语言翻译所涉及的挑战。我们还在此链接此 https URL 中发布了我们的数据集。

Title: Seeing Through the Fog: A Cost-Effectiveness Analysis of Hallucination Detection Systems

• Authors: Alexander Thomas, Seth Rosen, Vishnu Vettrivel
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.05270
• Pdf URL: https://arxiv.org/pdf/2411.05270
• Copy Paste: [[2411.05270]] Seeing Through the Fog: A Cost-Effectiveness Analysis of Hallucination Detection Systems(https://arxiv.org/abs/2411.05270)
• Keywords: language model, llm, hallucination
• Abstract: This paper presents a comparative analysis of hallucination detection systems for AI, focusing on automatic summarization and question answering tasks for Large Language Models (LLMs). We evaluate different hallucination detection systems using the diagnostic odds ratio (DOR) and cost-effectiveness metrics. Our results indicate that although advanced models can perform better they come at a much higher cost. We also demonstrate how an ideal hallucination detection system needs to maintain performance across different model sizes. Our findings highlight the importance of choosing a detection system aligned with specific application needs and resource constraints. Future research will explore hybrid systems and automated identification of underperforming components to enhance AI reliability and efficiency in detecting and mitigating hallucinations.
• 摘要：本文对人工智能的幻觉检测系统进行了比较分析，重点关注大型语言模型 (LLM) 的自动摘要和问答任务。我们使用诊断优势比 (DOR) 和成本效益指标评估不同的幻觉检测系统。我们的结果表明，尽管高级模型可以表现更好，但成本要高得多。我们还展示了理想的幻觉检测系统如何需要在不同模型大小中保持性能。我们的研究结果强调了选择符合特定应用需求和资源限制的检测系统的重要性。未来的研究将探索混合系统和自动识别表现不佳的组件，以提高人工智能在检测和缓解幻觉方面的可靠性和效率。

Title: Fox-1 Technical Report

• Authors: Zijian Hu, Jipeng Zhang, Rui Pan, Zhaozhuo Xu, Salman Avestimehr, Chaoyang He, Tong Zhang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.05281
• Pdf URL: https://arxiv.org/pdf/2411.05281
• Copy Paste: [[2411.05281]] Fox-1 Technical Report(https://arxiv.org/abs/2411.05281)
• Keywords: language model, llm
• Abstract: We present Fox-1, a series of small language models (SLMs) consisting of Fox-1-1.6B and Fox-1-1.6B-Instruct-v0.1. These models are pre-trained on 3 trillion tokens of web-scraped document data and fine-tuned with 5 billion tokens of instruction-following and multi-turn conversation data. Aiming to improve the pre-training efficiency, Fox-1-1.6B model introduces a novel 3-stage data curriculum across all the training data with 2K-8K sequence length. In architecture design, Fox-1 features a deeper layer structure, an expanded vocabulary, and utilizes Grouped Query Attention (GQA), offering a performant and efficient architecture compared to other SLMs. Fox-1 achieves better or on-par performance in various benchmarks compared to StableLM-2-1.6B, Gemma-2B, Qwen1.5-1.8B, and OpenELM1.1B, with competitive inference speed and throughput. The model weights have been released under the Apache 2.0 license, where we aim to promote the democratization of LLMs and make them fully accessible to the whole open-source community.
• 摘要：我们推出了 Fox-1，这是一系列小型语言模型 (SLM)，由 Fox-1-1.6B 和 Fox-1-1.6B-Instruct-v0.1 组成。这些模型在 3 万亿个网络抓取文档数据上进行了预训练，并使用 50 亿个指令跟踪和多轮对话数据进行了微调。为了提高预训练效率，Fox-1-1.6B 模型在所有序列长度为 2K-8K 的训练数据中引入了一种新颖的 3 阶段数据课程。在架构设计方面，Fox-1 具有更深的层结构、更广的词汇表，并利用了分组查询注意 (GQA)，与其他 SLM 相比，它提供了高性能和高效的架构。与 StableLM-2-1.6B、Gemma-2B、Qwen1.5-1.8B 和 OpenELM1.1B 相比，Fox-1 在各种基准测试中都实现了更好或相当的性能，具有竞争力的推理速度和吞吐量。模型权重已根据 Apache 2.0 许可证发布，我们的目标是促进 LLM 的民主化，并使整个开源社区可以完全访问它们。

Title: SpecHub: Provable Acceleration to Multi-Draft Speculative Decoding

• Authors: Ryan Sun, Tianyi Zhou, Xun Chen, Lichao Sun
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.05289
• Pdf URL: https://arxiv.org/pdf/2411.05289
• Copy Paste: [[2411.05289]] SpecHub: Provable Acceleration to Multi-Draft Speculative Decoding(https://arxiv.org/abs/2411.05289)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have become essential in advancing natural language processing (NLP) tasks, but their sequential token generation limits inference speed. Multi-Draft Speculative Decoding (MDSD) offers a promising solution by using a smaller draft model to generate multiple token sequences, which the target LLM verifies in parallel. However, current heuristic approaches, such as Recursive Rejection Sampling (RRS), suffer from low acceptance rates in subsequent drafts, limiting the advantages of using multiple drafts. Meanwhile, Optimal Transport with Membership Cost (OTM) can theoretically improve acceptance rates, but its computational cost is too high for real-time use. We present SpecHub, a novel, efficient sampling-verification method for MDSD that improves acceptance rates with only linear computational overhead. By simplifying the OTM problem into a compact Linear Programming model, SpecHub significantly reduces computational complexity. It further accelerates sampling by leveraging a sparse joint distribution, focusing computation on high-probability token sequences. In extensive experiments, Spechub consistently generates 0.05-0.27 and 0.02-0.16 more tokens per step than RRS and RRS without replacement. We attach our code at \url{this https URL}.
• 摘要：大型语言模型 (LLM) 已成为推进自然语言处理 (NLP) 任务的关键，但它们的顺序标记生成限制了推理速度。多草案推测解码 (MDSD) 提供了一种有前途的解决方案，即使用较小的草案模型生成多个标记序列，目标 LLM 会并行验证这些序列。然而，当前的启发式方法，例如递归拒绝抽样 (RRS)，在后续草案中的接受率较低，限制了使用多个草案的优势。同时，具有成员成本的最佳传输 (OTM) 理论上可以提高接受率，但其计算成本太高，无法实时使用。我们提出了 SpecHub，这是一种新颖、高效的 MDSD 抽样验证方法，它仅以线性计算开销提高了接受率。通过将 OTM 问题简化为紧凑的线性规划模型，SpecHub 显著降低了计算复杂度。它通过利用稀疏联合分布进一步加速抽样，将计算集中在高概率的标记序列上。在大量实验中，Spechub 每步生成比 RRS 和无替换 RRS 多 0.05-0.27 和 0.02-0.16 个标记。我们将代码附加在 \url{此 https URL} 上。

Title: SciDQA: A Deep Reading Comprehension Dataset over Scientific Papers

• Authors: Shruti Singh, Nandan Sarkar, Arman Cohan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.05338
• Pdf URL: https://arxiv.org/pdf/2411.05338
• Copy Paste: [[2411.05338]] SciDQA: A Deep Reading Comprehension Dataset over Scientific Papers(https://arxiv.org/abs/2411.05338)
• Keywords: llm
• Abstract: Scientific literature is typically dense, requiring significant background knowledge and deep comprehension for effective engagement. We introduce SciDQA, a new dataset for reading comprehension that challenges LLMs for a deep understanding of scientific articles, consisting of 2,937 QA pairs. Unlike other scientific QA datasets, SciDQA sources questions from peer reviews by domain experts and answers by paper authors, ensuring a thorough examination of the literature. We enhance the dataset's quality through a process that carefully filters out lower quality questions, decontextualizes the content, tracks the source document across different versions, and incorporates a bibliography for multi-document question-answering. Questions in SciDQA necessitate reasoning across figures, tables, equations, appendices, and supplementary materials, and require multi-document reasoning. We evaluate several open-source and proprietary LLMs across various configurations to explore their capabilities in generating relevant and factual responses. Our comprehensive evaluation, based on metrics for surface-level similarity and LLM judgements, highlights notable performance discrepancies. SciDQA represents a rigorously curated, naturally derived scientific QA dataset, designed to facilitate research on complex scientific text understanding.
• 摘要：科学文献通常内容密集，需要大量背景知识和深入理解才能有效参与。我们引入了 SciDQA，这是一个用于阅读理解的新数据集，它挑战 LLM 对科学文章的深入理解，包含 2,937 个 QA 对。与其他科学 QA 数据集不同，SciDQA 的问题来自领域专家的同行评审和论文作者的答案，确保对文献进行彻底检查。我们通过一个过程来提高数据集的质量，该过程仔细过滤掉质量较低的问题，去语境化内容，跟踪不同版本的源文档，并结合多文档问答书目。SciDQA 中的问题需要跨图表、表格、方程式、附录和补充材料进行推理，并且需要多文档推理。我们评估了各种配置的几个开源和专有 LLM，以探索它们生成相关和事实响应的能力。我们基于表面相似性和 LLM 判断的指标进行了全面评估，突出了显著的性能差异。 SciDQA 代表一个严格策划、自然得出的科学 QA 数据集，旨在促进对复杂科学文本理解的研究。

Title: Improving Multi-Domain Task-Oriented Dialogue System with Offline Reinforcement Learning

• Authors: Dharmendra Prajapat, Durga Toshniwal
• Subjects: cs.CL, cs.AI, cs.HC, cs.IR
• Abstract URL: https://arxiv.org/abs/2411.05340
• Pdf URL: https://arxiv.org/pdf/2411.05340
• Copy Paste: [[2411.05340]] Improving Multi-Domain Task-Oriented Dialogue System with Offline Reinforcement Learning(https://arxiv.org/abs/2411.05340)
• Keywords: language model, gpt
• Abstract: Task-oriented dialogue (TOD) system is designed to accomplish user-defined tasks through dialogues. The TOD system has progressed towards end-to-end modeling by leveraging pre-trained large language models. Fine-tuning the pre-trained language models using only supervised learning leads to the exposure bias and token loss problem and it deviates the models from completing the user's task. To address these issues, we propose a TOD system that leverages a unified pre-trained language model, GPT2, as a base model. It is optimized using supervised learning and reinforcement learning (RL). The issues in the TOD system are mitigated using a non-differentiable reward function. The reward is calculated using the weighted sum of the success rate and BLEU evaluation metrics. The success rate and BLEU metrics in reward calculation guide the language model for user task completion while ensuring a coherent and fluent response. Our model is acquired by fine-tuning a pre-trained model on the dialogue-session level which comprises user utterance, belief state, system act, and system response. Experimental results on MultiWOZ2.1 demonstrate that our model increases the inform rate by 1.60% and the success rate by 3.17% compared to the baseline.
• 摘要：面向任务的对话 (TOD) 系统旨在通过对话完成用户定义的任务。TOD 系统通过利用预先训练的大型语言模型，已朝着端到端建模的方向发展。仅使用监督学习对预训练的语言模型进行微调会导致曝光偏差和 token 丢失问题，并使模型无法完成用户的任务。为了解决这些问题，我们提出了一个 TOD 系统，该系统利用统一的预训练语言模型 GPT2 作为基础模型。它使用监督学习和强化学习 (RL) 进行优化。使用不可微分的奖励函数可以缓解 TOD 系统中的问题。奖励是使用成功率和 BLEU 评估指标的加权总和来计算的。奖励计算中的成功率和 BLEU 指标指导语言模型完成用户任务，同时确保连贯流畅的响应。我们的模型是通过在对话会话级别微调预训练模型获得的，该模型包括用户话语、信念状态、系统行为和系统响应。在MultiWOZ2.1上的实验结果表明，与基线相比，我们的模型告知率提高了1.60%，成功率提高了3.17%。

Title: Reasoning Robustness of LLMs to Adversarial Typographical Errors

• Authors: Esther Gan, Yiran Zhao, Liying Cheng, Yancan Mao, Anirudh Goyal, Kenji Kawaguchi, Min-Yen Kan, Michael Shieh
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.05345
• Pdf URL: https://arxiv.org/pdf/2411.05345
• Copy Paste: [[2411.05345]] Reasoning Robustness of LLMs to Adversarial Typographical Errors(https://arxiv.org/abs/2411.05345)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: Large Language Models (LLMs) have demonstrated impressive capabilities in reasoning using Chain-of-Thought (CoT) prompting. However, CoT can be biased by users' instruction. In this work, we study the reasoning robustness of LLMs to typographical errors, which can naturally occur in users' queries. We design an Adversarial Typo Attack ($\texttt{ATA}$) algorithm that iteratively samples typos for words that are important to the query and selects the edit that is most likely to succeed in attacking. It shows that LLMs are sensitive to minimal adversarial typographical changes. Notably, with 1 character edit, Mistral-7B-Instruct's accuracy drops from 43.7% to 38.6% on GSM8K, while with 8 character edits the performance further drops to 19.2%. To extend our evaluation to larger and closed-source LLMs, we develop the $\texttt{R$^2$ATA}$ benchmark, which assesses models' $\underline{R}$easoning $\underline{R}$obustness to $\underline{\texttt{ATA}}$. It includes adversarial typographical questions derived from three widely used reasoning datasets-GSM8K, BBH, and MMLU-by applying $\texttt{ATA}$ to open-source LLMs. $\texttt{R$^2$ATA}$ demonstrates remarkable transferability and causes notable performance drops across multiple super large and closed-source LLMs.
• 摘要：大型语言模型 (LLM) 已展示出使用思维链 (CoT) 提示进行推理的出色能力。然而，CoT 可能会受到用户指令的影响。在这项工作中，我们研究了 LLM 对印刷错误的推理鲁棒性，印刷错误可能自然发生在用户的查询中。我们设计了一种对抗性拼写错误攻击 ($\texttt{ATA}$) 算法，该算法迭代地对查询中重要的单词进行拼写错误采样，并选择最有可能成功攻击的编辑。它表明 LLM 对最小的对抗性拼写错误变化很敏感。值得注意的是，在 GSM8K 上，经过 1 个字符编辑，Mistral-7B-Instruct 的准确率从 43.7% 下降到 38.6%，而经过 8 个字符编辑，性能进一步下降到 19.2%。为了将我们的评估扩展到更大的闭源 LLM，我们开发了 $\texttt{R$^2$ATA}$ 基准，该基准评估模型的 $\underline{R}$ 推理 $\underline{R}$ 对 $\underline{\texttt{ATA}}$ 的稳健性。它包括从三个广泛使用的推理数据集（GSM8K、BBH 和 MMLU）得出的对抗性印刷问题，方法是将 $\texttt{ATA}$ 应用于开源 LLM。$\texttt{R$^2$ATA}$ 表现出卓越的可转移性，并在多个超大型和闭源 LLM 中导致性能显着下降。

Title: Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Tasks

• Authors: Chien-yu Huang, Wei-Chih Chen, Shu-wen Yang, Andy T. Liu, Chen-An Li, Yu-Xiang Lin, Wei-Cheng Tseng, Anuj Diwan, Yi-Jen Shih, Jiatong Shi, William Chen, Xuanjun Chen, Chi-Yuan Hsiao, Puyuan Peng, Shih-Heng Wang, Chun-Yi Kuan, Ke-Han Lu, Kai-Wei Chang, Chih-Kai Yang, Fabian Ritter-Gutierrez, Ming To Chuang, Kuan-Po Huang, Siddhant Arora, You-Kuan Lin, Eunjung Yeo, Kalvin Chang, Chung-Ming Chien, Kwanghee Choi, Cheng-Hsiu Hsieh, Yi-Cheng Lin, Chee-En Yu, I-Hsiang Chiu, Heitor R. Guimarães, Jionghao Han, Tzu-Quan Lin, Tzu-Yuan Lin, Homu Chang, Ting-Wu Chang, Chun Wei Chen, Shou-Jen Chen, Yu-Hua Chen, Hsi-Chun Cheng, Kunal Dhawan, Jia-Lin Fang, Shi-Xin Fang, Kuan-Yu Fang Chiang, Chi An Fu, Hsien-Fu Hsiao, Ching Yu Hsu, Shao-Syuan Huang, Lee Chen Wei, Hsi-Che Lin, Hsuan-Hao Lin, Hsuan-Ting Lin, Jian-Ren Lin, Ting-Chun Liu, Li-Chun Lu, Tsung-Min Pai, Ankita Pasad, Shih-Yun Shan Kuan, Suwon Shon, Yuxun Tang, Yun-Shao Tsai, Jui-Chiang Wei, Tzu-Chieh Wei, Chengxi Wu, Dien-Ruei Wu, Chao-Han Huck Yang, Chieh-Chi Yang, Jia Qi Yip, Shao-Xiang Yuan, Vahid Noroozi, Zhehuai Chen, Haibin Wu, Karen Livescu, David Harwath, Shinji Watanabe, Hung-yi Lee
• Subjects: cs.CL, eess.AS
• Abstract URL: https://arxiv.org/abs/2411.05361
• Pdf URL: https://arxiv.org/pdf/2411.05361
• Copy Paste: [[2411.05361]] Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Tasks(https://arxiv.org/abs/2411.05361)
• Keywords: language model, gpt, llm, chat
• Abstract: Multimodal foundation models, such as Gemini and ChatGPT, have revolutionized human-machine interactions by seamlessly integrating various forms of data. Developing a universal spoken language model that comprehends a wide range of natural language instructions is critical for bridging communication gaps and facilitating more intuitive interactions. However, the absence of a comprehensive evaluation benchmark poses a significant challenge. We present Dynamic-SUPERB Phase-2, an open and evolving benchmark for the comprehensive evaluation of instruction-based universal speech models. Building upon the first generation, this second version incorporates 125 new tasks contributed collaboratively by the global research community, expanding the benchmark to a total of 180 tasks, making it the largest benchmark for speech and audio evaluation. While the first generation of Dynamic-SUPERB was limited to classification tasks, Dynamic-SUPERB Phase-2 broadens its evaluation capabilities by introducing a wide array of novel and diverse tasks, including regression and sequence generation, across speech, music, and environmental audio. Evaluation results indicate that none of the models performed well universally. SALMONN-13B excelled in English ASR, while WavLLM demonstrated high accuracy in emotion recognition, but current models still require further innovations to handle a broader range of tasks. We will soon open-source all task data and the evaluation pipeline.
• 摘要：多模态基础模型（例如 Gemini 和 ChatGPT）通过无缝集成各种形式的数据，彻底改变了人机交互。开发一种能够理解各种自然语言指令的通用口语模型对于弥合沟通鸿沟和促进更直观的交互至关重要。然而，缺乏全面的评估基准带来了重大挑战。我们提出了 Dynamic-SUPERB Phase-2，这是一个开放且不断发展的基准，用于全面评估基于指令的通用语音模型。第二版在第一代的基础上，纳入了全球研究界共同贡献的 125 项新任务，将基准扩展到总共 180 项任务，使其成为语音和音频评估的最大基准。虽然第一代 Dynamic-SUPERB 仅限于分类任务，但 Dynamic-SUPERB Phase-2 通过引入一系列新颖而多样的任务（包括回归和序列生成）来扩展其评估能力，涵盖语音、音乐和环境音频。评估结果表明，没有一个模型在普遍上表现良好。 SALMONN-13B 在英语 ASR 方面表现出色，而 WavLLM 在情绪识别方面表现出色，但目前的模型仍需要进一步创新才能处理更广泛的任务。我们很快就会开源所有任务数据和评估流程。

Title: Ev2R: Evaluating Evidence Retrieval in Automated Fact-Checking

• Authors: Mubashara Akhtar, Michael Schlichtkrull, Andreas Vlachos
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.05375
• Pdf URL: https://arxiv.org/pdf/2411.05375
• Copy Paste: [[2411.05375]] Ev2R: Evaluating Evidence Retrieval in Automated Fact-Checking(https://arxiv.org/abs/2411.05375)
• Keywords: llm, prompt
• Abstract: Current automated fact-checking (AFC) approaches commonly evaluate evidence either implicitly via the predicted verdicts or by comparing retrieved evidence with a predefined closed knowledge source, such as Wikipedia. However, these methods suffer from limitations, resulting from their reliance on evaluation metrics developed for different purposes and constraints imposed by closed knowledge sources. Recent advances in natural language generation (NLG) evaluation offer new possibilities for evidence assessment. In this work, we introduce Ev2R, an evaluation framework for AFC that comprises three types of approaches for evidence evaluation: reference-based, proxy-reference, and reference-less. We evaluate their effectiveness through agreement with human ratings and adversarial tests, and demonstrate that prompt-based scorers, particularly those leveraging LLMs and reference evidence, outperform traditional evaluation approaches.
• 摘要：当前的自动事实核查 (AFC) 方法通常通过预测的判决隐式地评估证据，或通过将检索到的证据与预定义的封闭知识源（例如 Wikipedia）进行比较来评估证据。然而，这些方法存在局限性，因为它们依赖于为不同目的开发的评估指标和封闭知识源施加的限制。自然语言生成 (NLG) 评估的最新进展为证据评估提供了新的可能性。在这项工作中，我们引入了 Ev2R，这是一个 AFC 评估框架，它包含三种证据评估方法：基于参考、代理参考和无参考。我们通过与人工评分和对抗性测试的一致性来评估它们的有效性，并证明基于提示的评分器（尤其是那些利用 LLM 和参考证据的评分器）优于传统评估方法。

Title: Towards Low-Resource Harmful Meme Detection with LMM Agents

• Authors: Jianzhao Huang, Hongzhan Lin, Ziyan Liu, Ziyang Luo, Guang Chen, Jing Ma
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.05383
• Pdf URL: https://arxiv.org/pdf/2411.05383
• Copy Paste: [[2411.05383]] Towards Low-Resource Harmful Meme Detection with LMM Agents(https://arxiv.org/abs/2411.05383)
• Keywords: agent
• Abstract: The proliferation of Internet memes in the age of social media necessitates effective identification of harmful ones. Due to the dynamic nature of memes, existing data-driven models may struggle in low-resource scenarios where only a few labeled examples are available. In this paper, we propose an agency-driven framework for low-resource harmful meme detection, employing both outward and inward analysis with few-shot annotated samples. Inspired by the powerful capacity of Large Multimodal Models (LMMs) on multimodal reasoning, we first retrieve relative memes with annotations to leverage label information as auxiliary signals for the LMM agent. Then, we elicit knowledge-revising behavior within the LMM agent to derive well-generalized insights into meme harmfulness. By combining these strategies, our approach enables dialectical reasoning over intricate and implicit harm-indicative patterns. Extensive experiments conducted on three meme datasets demonstrate that our proposed approach achieves superior performance than state-of-the-art methods on the low-resource harmful meme detection task.
• 摘要：社交媒体时代，互联网模因的激增使得有效识别有害模因成为必要。由于模因的动态特性，现有的数据驱动模型在资源匮乏的情况下可能会遇到困难，因为资源匮乏的情况下只有少量标记示例可用。在本文中，我们提出了一个代理驱动的低资源有害模因检测框架，使用少量带注释样本进行向外和向内分析。受大型多模态模型 (LMM) 在多模态推理方面的强大能力的启发，我们首先检索带有注释的相关模因，以利用标签信息作为 LMM 代理的辅助信号。然后，我们引出 LMM 代理中的知识修改行为，以得出对模因危害的良好概括的见解。通过结合这些策略，我们的方法能够对复杂且隐含的危害指示模式进行辩证推理。在三个模因数据集上进行的大量实验表明，我们提出的方法在低资源有害模因检测任务上取得了比最先进的方法更好的性能。

Title: Benchmarking Distributional Alignment of Large Language Models

• Authors: Nicole Meister, Carlos Guestrin, Tatsunori Hashimoto
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.05403
• Pdf URL: https://arxiv.org/pdf/2411.05403
• Copy Paste: [[2411.05403]] Benchmarking Distributional Alignment of Large Language Models(https://arxiv.org/abs/2411.05403)
• Keywords: language model, llm
• Abstract: Language models (LMs) are increasingly used as simulacra for people, yet their ability to match the distribution of views of a specific demographic group and be \textit{distributionally aligned} remains uncertain. This notion of distributional alignment is complex, as there is significant variation in the types of attributes that are simulated. Prior works have underexplored the role of three critical variables -- the question domain, steering method, and distribution expression method -- which motivates our contribution of a benchmark explicitly addressing these dimensions. We construct a dataset expanding beyond political values, create human baselines for this task, and evaluate the extent to which an LM can align with a particular group's opinion distribution to inform design choices of such simulation systems. Our analysis reveals open problems regarding if, and how, LMs can be used to simulate humans, and that LLMs can more accurately describe the opinion distribution than simulate such distributions.
• 摘要：语言模型 (LM) 越来越多地被用作人类的模拟，但它们是否能够匹配特定人口群体的观点分布并实现 \textit{分布一致} 仍不确定。这种分布一致的概念很复杂，因为模拟的属性类型存在很大差异。之前的研究没有充分探索三个关键变量的作用——问题域、指导方法和分布表达方法——这促使我们贡献了一个明确解决这些维度的基准。我们构建了一个超越政治价值观的数据集，为这项任务创建了人类基线，并评估了 LM 与特定群体的意见分布的一致程度，从而为此类模拟系统的设计选择提供信息。我们的分析揭示了一些悬而未决的问题，即 LM 是否以及如何用于模拟人类，以及 LLM 可以更准确地描述意见分布而不是模拟这种分布。

Title: Gap-Filling Prompting Enhances Code-Assisted Mathematical Reasoning

• Authors: Mohammad Ghiasvand Mohammadkhani
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.05407
• Pdf URL: https://arxiv.org/pdf/2411.05407
• Copy Paste: [[2411.05407]] Gap-Filling Prompting Enhances Code-Assisted Mathematical Reasoning(https://arxiv.org/abs/2411.05407)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: Despite the strong performance of large language models (LLMs) in tasks like mathematical reasoning, their practical use is limited by high computational demands and proprietary restrictions. Chain-of-thought (CoT) and program-of-thought (PoT) fine-tuning are common methods to transfer LLM knowledge to small language models (SLMs). However, CoT often leads to calculation errors in SLMs, while PoT has shown more promise. While most PoT-based approaches focus on direct problem-to-code conversion or extracting only the key information from questions and then providing code solution for it, this work emphasizes filling the gaps in the question to clearly illustrate the solution path, which can be challenging for an SLM to understand when such information is not explicitly provided. Therefore, this paper introduces Gap-Filling Prompting (GFP), a novel two-step prompting strategy designed to enhance the problem-solving process for SLMs. The first step identifies these gaps and provides hints for filling them, while the second step adds the hints to the question to generate a final code solution. Experimental results on two benchmark datasets demonstrate that GFP significantly improves the mathematical reasoning abilities of SLMs.
• 摘要：尽管大型语言模型 (LLM) 在数学推理等任务中表现出色，但它们的实际应用受到高计算需求和专有限制的限制。思路链 (CoT) 和思路程序 (PoT) 微调是将 LLM 知识迁移到小型语言模型 (SLM) 的常用方法。然而，CoT 通常会导致 SLM 中的计算错误，而 PoT 则显示出更大的前景。虽然大多数基于 PoT 的方法侧重于直接将问题转换为代码，或者仅从问题中提取关键信息，然后为其提供代码解决方案，但这项工作强调填补问题中的空白以清楚地说明解决方案路径，当没有明确提供此类信息时，SLM 很难理解这一点。因此，本文介绍了填补空白提示 (GFP)，这是一种新颖的两步提示策略，旨在增强 SLM 的问题解决过程。第一步识别这些空白并提供填补它们的提示，而第二步将提示添加到问题中以生成最终的代码解决方案。在两个基准数据集上的实验结果表明，GFP 显著提高了 SLM 的数学推理能力。

Title: VISTA: Visual Integrated System for Tailored Automation in Math Problem Generation Using LLM

• Authors: Jeongwoo Lee, Kwangsuk Park, Jihyeon Park
• Subjects: cs.CL, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2411.05423
• Pdf URL: https://arxiv.org/pdf/2411.05423
• Copy Paste: [[2411.05423]] VISTA: Visual Integrated System for Tailored Automation in Math Problem Generation Using LLM(https://arxiv.org/abs/2411.05423)
• Keywords: language model, llm, agent
• Abstract: Generating accurate and consistent visual aids is a critical challenge in mathematics education, where visual representations like geometric shapes and functions play a pivotal role in enhancing student comprehension. This paper introduces a novel multi-agent framework that leverages Large Language Models (LLMs) to automate the creation of complex mathematical visualizations alongside coherent problem text. Our approach not only simplifies the generation of precise visual aids but also aligns these aids with the problem's core mathematical concepts, improving both problem creation and assessment. By integrating multiple agents, each responsible for distinct tasks such as numeric calculation, geometry validation, and visualization, our system delivers mathematically accurate and contextually relevant problems with visual aids. Evaluation across Geometry and Function problem types shows that our method significantly outperforms basic LLMs in terms of text coherence, consistency, relevance and similarity, while maintaining the essential geometrical and functional integrity of the original problems. Although some challenges remain in ensuring consistent visual outputs, our framework demonstrates the immense potential of LLMs in transforming the way educators generate and utilize visual aids in math education.
• 摘要：生成准确一致的视觉辅助工具是数学教育中的一项关键挑战，其中几何形状和函数等视觉表示在提高学生理解力方面起着关键作用。本文介绍了一种新颖的多智能体框架，该框架利用大型语言模型 (LLM) 自动创建复杂的数学可视化以及连贯的问题文本。我们的方法不仅简化了精确视觉辅助工具的生成，而且还将这些辅助工具与问题的核心数学概念相结合，从而改善了问题的创建和评估。通过集成多个智能体，每个智能体负责不同的任务，例如数值计算、几何验证和可视化，我们的系统通过视觉辅助工具提供数学上准确且上下文相关的问题。对几何和函数问题类型的评估表明，我们的方法在文本连贯性、一致性、相关性和相似性方面明显优于基本 LLM，同时保持了原始问题的基本几何和功能完整性。尽管在确保一致的视觉输出方面仍存在一些挑战，但我们的框架展示了 LLM 在改变教育者在数学教育中生成和利用视觉辅助工具的方式方面的巨大潜力。

Title: KyrgyzNLP: Challenges, Progress, and Future

• Authors: Anton Alekseev, Timur Turatali
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.05503
• Pdf URL: https://arxiv.org/pdf/2411.05503
• Copy Paste: [[2411.05503]] KyrgyzNLP: Challenges, Progress, and Future(https://arxiv.org/abs/2411.05503)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have excelled in numerous benchmarks, advancing AI applications in both linguistic and non-linguistic tasks. However, this has primarily benefited well-resourced languages, leaving less-resourced ones (LRLs) at a disadvantage. In this paper, we highlight the current state of the NLP field in the specific LRL: kyrgyz tili. Human evaluation, including annotated datasets created by native speakers, remains an irreplaceable component of reliable NLP performance, especially for LRLs where automatic evaluations can fall short. In recent assessments of the resources for Turkic languages, Kyrgyz is labeled with the status 'Scraping By', a severely under-resourced language spoken by millions. This is concerning given the growing importance of the language, not only in Kyrgyzstan but also among diaspora communities where it holds no official status. We review prior efforts in the field, noting that many of the publicly available resources have only recently been developed, with few exceptions beyond dictionaries (the processed data used for the analysis is presented at this https URL). While recent papers have made some headway, much more remains to be done. Despite interest and support from both business and government sectors in the Kyrgyz Republic, the situation for Kyrgyz language resources remains challenging. We stress the importance of community-driven efforts to build these resources, ensuring the future advancement sustainability. We then share our view of the most pressing challenges in Kyrgyz NLP. Finally, we propose a roadmap for future development in terms of research topics and language resources.
• 摘要：大型语言模型 (LLM) 在众多基准测试中表现出色，推动了语言和非语言任务中的 AI 应用。然而，这主要使资源丰富的语言受益，而资源较少的语言 (LRL) 则处于劣势。在本文中，我们重点介绍了特定 LRL：吉尔吉斯语 tili 中 NLP 领域的现状。人工评估（包括由母语人士创建的带注释的数据集）仍然是可靠的 NLP 性能不可替代的组成部分，尤其是对于自动评估可能不足的 LRL。在最近对突厥语资源的评估中，吉尔吉斯语被标记为“勉强应付”，这是一种资源严重不足的语言，有数百万人使用。鉴于该语言的重要性日益增加，这令人担忧，不仅在吉尔吉斯斯坦，而且在其没有官方地位的侨民社区中也是如此。我们回顾了该领域先前的努力，注意到许多公开可用的资源都是最近才开发的，除了词典之外几乎没有例外（用于分析的处理后的数据在此 https URL 中显示）。虽然最近的论文取得了一些进展，但仍有许多工作要做。尽管吉尔吉斯共和国的商业和政府部门都对此表示关注和支持，但吉尔吉斯语言资源的状况仍然充满挑战。我们强调社区驱动的努力对于建立这些资源的重要性，以确保未来进步的可持续性。然后，我们分享了我们对吉尔吉斯语 NLP 中最紧迫挑战的看法。最后，我们提出了研究主题和语言资源未来发展的路线图。

Title: LBPE: Long-token-first Tokenization to Improve Large Language Models

• Authors: Haoran Lian, Yizhe Xiong, Zijia Lin, Jianwei Niu, Shasha Mo, Hui Chen, Peng Liu, Guiguang Ding
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.05504
• Pdf URL: https://arxiv.org/pdf/2411.05504
• Copy Paste: [[2411.05504]] LBPE: Long-token-first Tokenization to Improve Large Language Models(https://arxiv.org/abs/2411.05504)
• Keywords: language model, llm
• Abstract: The prevalent use of Byte Pair Encoding (BPE) in Large Language Models (LLMs) facilitates robust handling of subword units and avoids issues of out-of-vocabulary words. Despite its success, a critical challenge persists: long tokens, rich in semantic information, have fewer occurrences in tokenized datasets compared to short tokens, which can result in imbalanced learning issue across different tokens. To address that, we propose LBPE, which prioritizes long tokens during the encoding process. LBPE generates tokens according to their reverse ranks of token length rather than their ranks in the vocabulary, granting longer tokens higher priority during the encoding process. Consequently, LBPE smooths the frequency differences between short and long tokens, and thus mitigates the learning imbalance. Extensive experiments across diverse language modeling tasks demonstrate that LBPE consistently outperforms the original BPE, well demonstrating its effectiveness.
• 摘要：字节对编码 (BPE) 在大型语言模型 (LLM) 中的广泛使用有助于对子词单元进行稳健处理，并避免词汇表外的单词问题。尽管取得了成功，但仍然存在一个关键挑战：与短标记相比，长标记富含语义信息，在标记化数据集中出现的频率较少，这可能导致不同标记之间的学习不平衡问题。为了解决这个问题，我们提出了 LBPE，它在编码过程中优先考虑长标记。LBPE 根据标记长度的反向排名而不是词汇表中的排名来生成标记，在编码过程中赋予较长的标记更高的优先级。因此，LBPE 平滑了短标记和长标记之间的频率差异，从而缓解了学习不平衡。在各种语言建模任务中进行的大量实验表明，LBPE 始终优于原始 BPE，很好地证明了其有效性。

Title: Assessing the Answerability of Queries in Retrieval-Augmented Code Generation

• Authors: Geonmin Kim, Jaeyeon Kim, Hancheol Park, Wooksu Shin, Tae-Ho Kim
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.05547
• Pdf URL: https://arxiv.org/pdf/2411.05547
• Copy Paste: [[2411.05547]] Assessing the Answerability of Queries in Retrieval-Augmented Code Generation(https://arxiv.org/abs/2411.05547)
• Keywords: language model, llm
• Abstract: Thanks to unprecedented language understanding and generation capabilities of large language model (LLM), Retrieval-augmented Code Generation (RaCG) has recently been widely utilized among software developers. While this has increased productivity, there are still frequent instances of incorrect codes being provided. In particular, there are cases where plausible yet incorrect codes are generated for queries from users that cannot be answered with the given queries and API descriptions. This study proposes a task for evaluating answerability, which assesses whether valid answers can be generated based on users' queries and retrieved APIs in RaCG. Additionally, we build a benchmark dataset called Retrieval-augmented Code Generability Evaluation (RaCGEval) to evaluate the performance of models performing this task. Experimental results show that this task remains at a very challenging level, with baseline models exhibiting a low performance of 46.7%. Furthermore, this study discusses methods that could significantly improve performance.
• 摘要：由于大型语言模型 (LLM) 前所未有的语言理解和生成能力，检索增强代码生成 (RaCG) 最近在软件开发人员中得到了广泛应用。虽然这提高了生产力，但仍然经常出现提供错误代码的情况。特别是，有些情况下，对于无法使用给定查询和 API 描述回答的用户查询，会生成合理但不正确的代码。本研究提出了一项评估可回答性的任务，该任务评估是否可以根据 RaCG 中的用户查询和检索到的 API 生成有效答案。此外，我们构建了一个名为检索增强代码生成性评估 (RaCGEval) 的基准数据集来评估执行此任务的模型的性能。实验结果表明，这项任务仍然处于非常具有挑战性的水平，基线模型的性能较低，为 46.7%。此外，本研究讨论了可以显着提高性能的方法。

Title: Evaluating and Adapting Large Language Models to Represent Folktales in Low-Resource Languages

• Authors: JA Meaney, Beatrice Alex, William Lamb
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.05593
• Pdf URL: https://arxiv.org/pdf/2411.05593
• Copy Paste: [[2411.05593]] Evaluating and Adapting Large Language Models to Represent Folktales in Low-Resource Languages(https://arxiv.org/abs/2411.05593)
• Keywords: language model, llm
• Abstract: Folktales are a rich resource of knowledge about the society and culture of a civilisation. Digital folklore research aims to use automated techniques to better understand these folktales, and it relies on abstract representations of the textual data. Although a number of large language models (LLMs) claim to be able to represent low-resource langauges such as Irish and Gaelic, we present two classification tasks to explore how useful these representations are, and three adaptations to improve the performance of these models. We find that adapting the models to work with longer sequences, and continuing pre-training on the domain of folktales improves classification performance, although these findings are tempered by the impressive performance of a baseline SVM with non-contextual features.
• 摘要：民间故事是有关文明社会和文化的丰富知识资源。数字民间传说研究旨在使用自动化技术更好地理解这些民间故事，它依赖于文本数据的抽象表示。尽管许多大型语言模型 (LLM) 声称能够表示爱尔兰语和盖尔语等资源匮乏的语言，但我们提出了两项​​分类任务来探索这些表示的实用性，并提出了三项改进措施来提高这些模型的性能。我们发现，调整模型以处理更长的序列，并继续在民间故事领域进行预训练，可以提高分类性能，尽管这些发现受到具有非上下文特征的基线 SVM 的出色性能的影响。

Title: Assessing Open-Source Large Language Models on Argumentation Mining Subtasks

• Authors: Mohammad Yeghaneh Abkenar, Weixing Wang, Hendrik Graupner, Manfred Stede
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.05639
• Pdf URL: https://arxiv.org/pdf/2411.05639
• Copy Paste: [[2411.05639]] Assessing Open-Source Large Language Models on Argumentation Mining Subtasks(https://arxiv.org/abs/2411.05639)
• Keywords: language model, llm
• Abstract: We explore the capability of four open-sourcelarge language models (LLMs) in argumentation mining (AM). We conduct experiments on three different corpora; persuasive essays(PE), argumentative microtexts (AMT) Part 1 and Part 2, based on two argumentation mining sub-tasks: (i) argumentative discourse units classifications (ADUC), and (ii) argumentative relation classification (ARC). This work aims to assess the argumentation capability of open-source LLMs, including Mistral 7B, Mixtral8x7B, LlamA2 7B and LlamA3 8B in both, zero-shot and few-shot scenarios. Our analysis contributes to further assessing computational argumentation with open-source LLMs in future research efforts.
• 摘要：我们探索了四种开源大型语言模型 (LLM) 在论证挖掘 (AM) 中的能力。我们对三种不同的语料库进行了实验：说服性文章 (PE)、论证性微文本 (AMT) 第 1 部分和第 2 部分，基于两个论证挖掘子任务：(i) 论证话语单元分类 (ADUC) 和 (ii) 论证关系分类 (ARC)。这项工作旨在评估开源 LLM 的论证能力，包括 Mistral 7B、Mixtral8x7B、LlamA2 7B 和 LlamA3 8B 在零样本和少量样本场景中的论证能力。我们的分析有助于在未来的研究中进一步评估使用开源 LLM 的计算论证。

Title: Evaluating Large Language Model Capability in Vietnamese Fact-Checking Data Generation

• Authors: Long Truong To, Hung Tuan Le, Dat Van-Thanh Nguyen, Manh Trong Nguyen, Tri Thien Nguyen, Tin Van Huynh, Kiet Van Nguyen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.05641
• Pdf URL: https://arxiv.org/pdf/2411.05641
• Copy Paste: [[2411.05641]] Evaluating Large Language Model Capability in Vietnamese Fact-Checking Data Generation(https://arxiv.org/abs/2411.05641)
• Keywords: language model, llm, prompt
• Abstract: Large Language Models (LLMs), with gradually improving reading comprehension and reasoning capabilities, are being applied to a range of complex language tasks, including the automatic generation of language data for various purposes. However, research on applying LLMs for automatic data generation in low-resource languages like Vietnamese is still underdeveloped and lacks comprehensive evaluation. In this paper, we explore the use of LLMs for automatic data generation for the Vietnamese fact-checking task, which faces significant data limitations. Specifically, we focus on fact-checking data where claims are synthesized from multiple evidence sentences to assess the information synthesis capabilities of LLMs. We develop an automatic data construction process using simple prompt techniques on LLMs and explore several methods to improve the quality of the generated data. To evaluate the quality of the data generated by LLMs, we conduct both manual quality assessments and performance evaluations using language models. Experimental results and manual evaluations illustrate that while the quality of the generated data has significantly improved through fine-tuning techniques, LLMs still cannot match the data quality produced by humans.
• 摘要：大型语言模型 (LLM) 的阅读理解和推理能力正在逐步提高，并被应用于一系列复杂的语言任务，包括用于各种目的的语言数据的自动生成。然而，将 LLM 应用于越南语等资源匮乏的语言的自动数据生成的研究仍不发达，缺乏全面的评估。在本文中，我们探讨了使用 LLM 为越南语事实核查任务自动生成数据，该任务面临着巨大的数据限制。具体来说，我们专注于从多个证据句子中合成声明的事实核查数据，以评估 LLM 的信息合成能力。我们在 LLM 上使用简单的提示技术开发了一个自动数据构建过程，并探索了几种提高生成数据质量的方法。为了评估 LLM 生成的数据的质量，我们使用语言模型进行了手动质量评估和性能评估。实验结果和手动评估表明，虽然通过微调技术生成的数据质量有了显着提高，但 LLM 仍然无法与人类生成的数据质量相匹配。

Title: Unmasking the Limits of Large Language Models: A Systematic Evaluation of Masked Text Processing Ability through MskQA and MskCal

• Authors: Fuka Matsuzaki, Haru-Tada Sato
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.05665
• Pdf URL: https://arxiv.org/pdf/2411.05665
• Copy Paste: [[2411.05665]] Unmasking the Limits of Large Language Models: A Systematic Evaluation of Masked Text Processing Ability through MskQA and MskCal(https://arxiv.org/abs/2411.05665)
• Keywords: language model, gpt, llm
• Abstract: This paper sheds light on the limitations of Large Language Models (LLMs) by rigorously evaluating their ability to process masked text. We introduce two novel tasks: MskQA, measuring reasoning on masked question-answering datasets like RealtimeQA, and MskCal, assessing numerical reasoning on masked arithmetic this http URL GPT-4o and 4o-mini reveals that while LLMs exhibit some resilience to masked text, their performance is highly contingent on masking rates and semantic cues. Specifically, "solid masking," where semantic clues are entirely absent, leads to a significant performance drop compared to "partial lifting," where some semantic information is retained, indicating LLMs' reliance on surface-level patterns. Interestingly, GPT-4o consistently outperforms 4o-mini, particularly in MskCal, demonstrating a greater ability to handle numerical reasoning with masked text. This underscores the crucial role of semantic cues in the reasoning process of LLMs. Our study illuminates the interplay between background knowledge and reasoning ability in masked text processing, paving the way for a deeper understanding of LLM capabilities and limitations, and highlighting the need for more robust evaluation methods to accurately assess their true comprehension abilities.
• 摘要：本文通过严格评估大型语言模型 (LLM) 处理掩码文本的能力，揭示了大型语言模型 (LLM) 的局限性。我们引入了两个新任务：MskQA，测量掩码问答数据集（如 RealtimeQA）上的推理能力，以及 MskCal，评估掩码算术上的数值推理能力。GPT-4o 和 4o-mini 表明，虽然 LLM 对掩码文本表现出一定的弹性，但它们的性能在很大程度上取决于掩码率和语义线索。具体而言，与保留一些语义信息的“部分提升”相比，“固体掩码”完全不存在语义线索，会导致性能显著下降，这表明 LLM 依赖于表面模式。有趣的是，GPT-4o 的表现始终优于 4o-mini，尤其是在 MskCal 中，表现出更强的用掩码文本处理数值推理的能力。这强调了语义线索在 LLM 推理过程中的关键作用。我们的研究阐明了掩蔽文本处理中背景知识和推理能力之间的相互作用，为更深入地理解 LLM 的能力和局限性铺平了道路，并强调了需要更强大的评估方法来准确评估他们真正的理解能力。

Title: Asterisk*: Keep it Simple

• Authors: Andrew Semenov
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.05691
• Pdf URL: https://arxiv.org/pdf/2411.05691
• Copy Paste: [[2411.05691]] Asterisk*: Keep it Simple(https://arxiv.org/abs/2411.05691)
• Keywords: gpt
• Abstract: This paper describes Asterisk, a compact GPT-based model for generating text embeddings. The model uses a minimalist architecture with two layers, two attention heads, and 256 embedding dimensions. By applying knowledge distillation from larger pretrained models, we explore the trade-offs between model size and performance while minimizing computational and memory requirements. The model is primarily evaluated and optimized for classification tasks, with experimental results showing its moderate performance in zero-shot classification across various downstream applications. With additional configuration, the model performance can approach or even surpass that of larger architectures on specific classification tasks.
• 摘要：本文介绍了 Asterisk，这是一种基于 GPT 的紧凑型文本嵌入生成模型。该模型采用极简架构，具有两层、两个注意力头和 256 个嵌入维度。通过从较大的预训练模型中应用知识提炼，我们探索了模型大小和性能之间的权衡，同时最大限度地降低了计算和内存需求。该模型主要针对分类任务进行评估和优化，实验结果表明，它在各种下游应用中的零样本分类中具有中等性能。通过额外的配置，模型性能可以接近甚至超过特定分类任务上较大的架构的性能。

Title: Multi-hop Evidence Pursuit Meets the Web: Team Papelo at FEVER 2024

• Authors: Christopher Malon
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.05762
• Pdf URL: https://arxiv.org/pdf/2411.05762
• Copy Paste: [[2411.05762]] Multi-hop Evidence Pursuit Meets the Web: Team Papelo at FEVER 2024(https://arxiv.org/abs/2411.05762)
• Keywords: language model, llm
• Abstract: Separating disinformation from fact on the web has long challenged both the search and the reasoning powers of humans. We show that the reasoning power of large language models (LLMs) and the retrieval power of modern search engines can be combined to automate this process and explainably verify claims. We integrate LLMs and search under a multi-hop evidence pursuit strategy. This strategy generates an initial question based on an input claim using a sequence to sequence model, searches and formulates an answer to the question, and iteratively generates follow-up questions to pursue the evidence that is missing using an LLM. We demonstrate our system on the FEVER 2024 (AVeriTeC) shared task. Compared to a strategy of generating all the questions at once, our method obtains .045 higher label accuracy and .155 higher AVeriTeC score (evaluating the adequacy of the evidence). Through ablations, we show the importance of various design choices, such as the question generation method, medium-sized context, reasoning with one document at a time, adding metadata, paraphrasing, reducing the problem to two classes, and reconsidering the final verdict. Our submitted system achieves .510 AVeriTeC score on the dev set and .477 AVeriTeC score on the test set.
• 摘要：长期以来，将网络上的虚假信息与事实区分开来一直是人类搜索和推理能力的挑战。我们表明，大型语言模型 (LLM) 的推理能力和现代搜索引擎的检索能力可以结合起来，使这一过程自动化，并以可解释的方式验证主张。我们在多跳证据追踪策略下集成了 LLM 和搜索。此策略使用序列到序列模型根据输入主张生成初始问题，搜索并制定问题的答案，并使用 LLM 迭代生成后续问题以寻找缺失的证据。我们在 FEVER 2024 (AVeriTeC) 共享任务上演示了我们的系统。与一次生成所有问题的策略相比，我们的方法获得了 0.045 更高的标签准确率和 0.155 更高的 AVeriTeC 分数（评估证据的充分性）。通过消融，我们展示了各种设计选择的重要性，例如问题生成方法、中等大小的上下文、一次使用一个文档进行推理、添加元数据、释义、将问题简化为两个类别以及重新考虑最终裁决。我们提交的系统在开发集上获得了 .510 AVeriTeC 分数，在测试集上获得了 .477 AVeriTeC 分数。

Title: FinDVer: Explainable Claim Verification over Long and Hybrid-Content Financial Documents

• Authors: Yilun Zhao, Yitao Long, Yuru Jiang, Chengye Wang, Weiyuan Chen, Hongjun Liu, Yiming Zhang, Xiangru Tang, Chen Zhao, Arman Cohan
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.05764
• Pdf URL: https://arxiv.org/pdf/2411.05764
• Copy Paste: [[2411.05764]] FinDVer: Explainable Claim Verification over Long and Hybrid-Content Financial Documents(https://arxiv.org/abs/2411.05764)
• Keywords: gpt, llm, chain-of-thought
• Abstract: We introduce FinDVer, a comprehensive benchmark specifically designed to evaluate the explainable claim verification capabilities of LLMs in the context of understanding and analyzing long, hybrid-content financial documents. FinDVer contains 2,400 expert-annotated examples, divided into three subsets: information extraction, numerical reasoning, and knowledge-intensive reasoning, each addressing common scenarios encountered in real-world financial contexts. We assess a broad spectrum of LLMs under long-context and RAG settings. Our results show that even the current best-performing system, GPT-4o, still lags behind human experts. We further provide in-depth analysis on long-context and RAG setting, Chain-of-Thought reasoning, and model reasoning errors, offering insights to drive future advancements. We believe that FinDVer can serve as a valuable benchmark for evaluating LLMs in claim verification over complex, expert-domain documents.
• 摘要：我们推出了 FinDVer，这是一个全面的基准，专门用于在理解和分析长篇混合内容金融文档的背景下评估 LLM 的可解释索赔验证能力。FinDVer 包含 2,400 个专家注释的示例，分为三个子集：信息提取、数值推理和知识密集型推理，每个子集都解决了现实世界金融环境中遇到的常见场景。我们在长上下文和 RAG 设置下评估了广泛的 LLM。我们的结果表明，即使是目前表现最好的系统 GPT-4o，仍然落后于人类专家。我们进一步对长上下文和 RAG 设置、思路链推理和模型推理错误进行了深入分析，为推动未来的进步提供了见解。我们相信 FinDVer 可以作为评估复杂专家领域文档索赔验证中 LLM 的宝贵基准。

Title: Fact or Fiction? Can LLMs be Reliable Annotators for Political Truths?

• Authors: Veronica Chatrath, Marcelo Lotif, Shaina Raza
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.05775
• Pdf URL: https://arxiv.org/pdf/2411.05775
• Copy Paste: [[2411.05775]] Fact or Fiction? Can LLMs be Reliable Annotators for Political Truths?(https://arxiv.org/abs/2411.05775)
• Keywords: language model, llm
• Abstract: Political misinformation poses significant challenges to democratic processes, shaping public opinion and trust in media. Manual fact-checking methods face issues of scalability and annotator bias, while machine learning models require large, costly labelled datasets. This study investigates the use of state-of-the-art large language models (LLMs) as reliable annotators for detecting political factuality in news articles. Using open-source LLMs, we create a politically diverse dataset, labelled for bias through LLM-generated annotations. These annotations are validated by human experts and further evaluated by LLM-based judges to assess the accuracy and reliability of the annotations. Our approach offers a scalable and robust alternative to traditional fact-checking, enhancing transparency and public trust in media.
• 摘要：政治错误信息对民主进程构成重大挑战，影响公众舆论和对媒体的信任。手动事实核查方法面临可扩展性和注释者偏见的问题，而机器学习模型需要大量昂贵的标记数据集。本研究调查了使用最先进的大型语言模型 (LLM) 作为可靠的注释器来检测新闻文章中的政治事实。使用开源 LLM，我们创建了一个政治多样化的数据集，并通过 LLM 生成的注释标记偏见。这些注释由人类专家验证，并由基于 LLM 的评委进一步评估，以评估注释的准确性和可靠性。我们的方法为传统事实核查提供了一种可扩展且强大的替代方案，提高了媒体的透明度和公众信任。

Title: Quantitative Assessment of Intersectional Empathetic Bias and Understanding

• Authors: Vojtech Formanek, Ondrej Sotolar
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2411.05777
• Pdf URL: https://arxiv.org/pdf/2411.05777
• Copy Paste: [[2411.05777]] Quantitative Assessment of Intersectional Empathetic Bias and Understanding(https://arxiv.org/abs/2411.05777)
• Keywords: llm, prompt
• Abstract: A growing amount of literature critiques the current operationalizations of empathy based on loose definitions of the construct. Such definitions negatively affect dataset quality, model robustness, and evaluation reliability. We propose an empathy evaluation framework that operationalizes empathy close to its psychological origins. The framework measures the variance in responses of LLMs to prompts using existing metrics for empathy and emotional valence. The variance is introduced through the controlled generation of the prompts by varying social biases affecting context understanding, thus impacting empathetic understanding. The control over generation ensures high theoretical validity of the constructs in the prompt dataset. Also, it makes high-quality translation, especially into languages that currently have little-to-no way of evaluating empathy or bias, such as the Slavonic family, more manageable. Using chosen LLMs and various prompt types, we demonstrate the empathy evaluation with the framework, including multiple-choice answers and free generation. The variance in our initial evaluation sample is small and we were unable to measure convincing differences between the empathetic understanding in contexts given by different social groups. However, the results are promising because the models showed significant alterations their reasoning chains needed to capture the relatively subtle changes in the prompts. This provides the basis for future research into the construction of the evaluation sample and statistical methods for measuring the results.
• 摘要：越来越多的文献批评了当前基于对构造的松散定义的同理心操作化。这样的定义会对数据集质量、模型稳健性和评估可靠性产生负面影响。我们提出了一个同理心评估框架，将同理心操作化到接近其心理起源的程度。该框架使用现有的同理心和情感价指标来衡量 LLM 对提示的反应差异。通过控制提示的生成来引入差异，通过改变影响语境理解的社会偏见，从而影响同理心理解。对生成的控制确保了提示数据集中构造的高理论有效性。此外，它使高质量的翻译，尤其是翻译成目前几乎没有或根本没有办法评估同理心或偏见的语言，如斯拉夫语系，变得更加容易管理。使用选定的 LLM 和各种提示类型，我们展示了使用该框架的同理心评估，包括多项选择答案和自由生成。我们最初的评估样本的方差很小，我们无法衡量不同社会群体给出的情境中共情理解之间的明显差异。然而，结果令人鼓舞，因为模型显示出它们的推理链发生了重大改变，需要捕捉提示中相对微妙的变化。这为未来研究评估样本的构建和测量结果的统计方法奠定了基础。

Title: Using Language Models to Disambiguate Lexical Choices in Translation

• Authors: Josh Barua, Sanjay Subramanian, Kayo Yin, Alane Suhr
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.05781
• Pdf URL: https://arxiv.org/pdf/2411.05781
• Copy Paste: [[2411.05781]] Using Language Models to Disambiguate Lexical Choices in Translation(https://arxiv.org/abs/2411.05781)
• Keywords: language model, gpt, llm
• Abstract: In translation, a concept represented by a single word in a source language can have multiple variations in a target language. The task of lexical selection requires using context to identify which variation is most appropriate for a source text. We work with native speakers of nine languages to create DTAiLS, a dataset of 1,377 sentence pairs that exhibit cross-lingual concept variation when translating from English. We evaluate recent LLMs and neural machine translation systems on DTAiLS, with the best-performing model, GPT-4, achieving from 67 to 85% accuracy across languages. Finally, we use language models to generate English rules describing target-language concept variations. Providing weaker models with high-quality lexical rules improves accuracy substantially, in some cases reaching or outperforming GPT-4.
• 摘要：在翻译中，源语言中单个单词所表示的概念在目标语言中可能会有多种变体。词汇选择任务需要使用上下文来确定哪种变体最适合源文本。我们与九种语言的母语人士合作创建了 DTAiLS，这是一个包含 1,377 个句子对的数据集，这些句子对在从英语翻译时会表现出跨语言概念的变化。我们在 DTAiLS 上评估了最近的 LLM 和神经机器翻译系统，其中表现最佳的模型 GPT-4 在不同语言中的准确率达到 67% 到 85%。最后，我们使用语言模型生成描述目标语言概念变化的英语规则。为较弱的模型提供高质量的词汇规则可显著提高准确率，在某些情况下可以达到或超过 GPT-4。

Title: Recycled Attention: Efficient inference for long-context language models

• Authors: Fangyuan Xu, Tanya Goyal, Eunsol Choi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.05787
• Pdf URL: https://arxiv.org/pdf/2411.05787
• Copy Paste: [[2411.05787]] Recycled Attention: Efficient inference for long-context language models(https://arxiv.org/abs/2411.05787)
• Keywords: language model, llm
• Abstract: Generating long sequences of tokens given a long-context input imposes a heavy computational burden for large language models (LLMs). One of the computational bottleneck comes from computing attention over a long sequence of input at each generation step. In this paper, we propose Recycled Attention, an inference-time method which alternates between full context attention and attention over a subset of input tokens. When performing partial attention, we recycle the attention pattern of a previous token that has performed full attention and attend only to the top K most attended tokens, reducing the cost of data movement and attention computation. Compared to previously proposed inference-time acceleration method which attends only to local context or tokens with high accumulative attention scores, our approach flexibly chooses tokens that are relevant to the current decoding step. We evaluate our methods on RULER, a suite of tasks designed to comprehensively evaluate long-context abilities, and long-context language modeling tasks. Applying our method to off-the-shelf LLMs achieves comparable speedup to baselines which only consider local context while improving the performance by 2x. We further explore two ideas to improve performance-efficiency trade-offs: (1) dynamically decide when to perform recycled or full attention step based on the query similarities and (2) continued pre-training the model with Recycled Attention.
• 摘要：给定长上下文输入生成长序列的 token 会给大型语言模型 (LLM) 带来沉重的计算负担。计算瓶颈之一来自于在每个生成步骤中计算长序列输入的注意力。在本文中，我们提出了循环注意力，这是一种推理时间方法，它在完整上下文注意力和输入 token 子集注意力之间交替。在执行部分注意力时，我们回收执行完整注意力的先前 token 的注意力模式，并仅关注前 K 个最受关注的 token，从而降低数据移动和注意力计算的成本。与之前提出的推理时间加速方法相比，该方法仅关注局部上下文或具有高累积注意力分数的 token，我们的方法可以灵活地选择与当前解码步骤相关的 token。我们在 RULER（一套旨在全面评估长上下文能力的任务）和长上下文语言建模任务上评估了我们的方法。将我们的方法应用于现成的 LLM 可实现与仅考虑局部上下文的基线相当的加速，同时将性能提高 2 倍。我们进一步探索了两种改善性能效率权衡的想法：（1）根据查询相似性动态决定何时执行回收或完全注意步骤；（2）继续使用回收注意对模型进行预训练。