2024-11-20

Title: Understanding Chain-of-Thought in LLMs through Information Theory

• Authors: Jean-Francois Ton, Muhammad Faaiz Taufiq, Yang Liu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.11984
• Pdf URL: https://arxiv.org/pdf/2411.11984
• Copy Paste: [[2411.11984]] Understanding Chain-of-Thought in LLMs through Information Theory(https://arxiv.org/abs/2411.11984)
• Keywords: language model, llm, chain-of-thought
• Abstract: Large Language Models (LLMs) have shown impressive performance in complex reasoning tasks through Chain-of-Thought (CoT) reasoning, allowing models to break down problems into manageable sub-tasks. However, existing CoT evaluation techniques either require annotated CoT data or fall short in accurately assessing intermediate reasoning steps, leading to high rates of false positives. In this paper, we formalize CoT reasoning in LLMs through an information-theoretic lens. Specifically, our framework quantifies the `information gain' at each reasoning step, enabling the identification of failure modes in LLMs without the need for expensive annotated datasets. We demonstrate the efficacy of our approach through extensive experiments on toy and GSM-8K data, where it significantly outperforms existing outcome-based methods by providing more accurate insights into model performance on individual tasks.
• 摘要：大型语言模型 (LLM) 通过思维链 (CoT) 推理在复杂推理任务中表现出色，使模型能够将问题分解为可管理的子任务。然而，现有的 CoT 评估技术要么需要带注释的 CoT 数据，要么无法准确评估中间推理步骤，导致误报率很高。在本文中，我们通过信息论视角形式化了 LLM 中的 CoT 推理。具体来说，我们的框架量化了每个推理步骤的“信息增益”，从而无需昂贵的带注释数据集即可识别 LLM 中的故障模式。我们通过对玩具和 GSM-8K 数据的大量实验证明了我们方法的有效性，通过提供对单个任务的模型性能的更准确洞察，它明显优于现有的基于结果的方法。

Title: ByteScience: Bridging Unstructured Scientific Literature and Structured Data with Auto Fine-tuned Large Language Model in Token Granularity

• Authors: Tong Xie, Hanzhi Zhang, Shaozhou Wang, Yuwei Wan, Imran Razzak, Chunyu Kit, Wenjie Zhangand Bram Hoex
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.12000
• Pdf URL: https://arxiv.org/pdf/2411.12000
• Copy Paste: [[2411.12000]] ByteScience: Bridging Unstructured Scientific Literature and Structured Data with Auto Fine-tuned Large Language Model in Token Granularity(https://arxiv.org/abs/2411.12000)
• Keywords: language model, llm, long context
• Abstract: Natural Language Processing (NLP) is widely used to supply summarization ability from long context to structured information. However, extracting structured knowledge from scientific text by NLP models remains a challenge because of its domain-specific nature to complex data preprocessing and the granularity of multi-layered device-level information. To address this, we introduce ByteScience, a non-profit cloud-based auto fine-tuned Large Language Model (LLM) platform, which is designed to extract structured scientific data and synthesize new scientific knowledge from vast scientific corpora. The platform capitalizes on DARWIN, an open-source, fine-tuned LLM dedicated to natural science. The platform was built on Amazon Web Services (AWS) and provides an automated, user-friendly workflow for custom model development and data extraction. The platform achieves remarkable accuracy with only a small amount of well-annotated articles. This innovative tool streamlines the transition from the science literature to structured knowledge and data and benefits the advancements in natural informatics.
• 摘要：自然语言处理 (NLP) 被广泛用于提供从长上下文到结构化信息的总结能力。然而，由于其领域特定性、复杂的数据预处理和多层设备级信息的粒度，使用 NLP 模型从科学文本中提取结构化知识仍然是一项挑战。为了解决这个问题，我们推出了 ByteScience，这是一个非盈利的基于云的自动微调大型语言模型 (LLM) 平台，旨在从大量科学语料库中提取结构化科学数据并合成新的科学知识。该平台利用 DARWIN，这是一个专用于自然科学的开源、微调的 LLM。该平台建立在亚马逊网络服务 (AWS) 上，为自定义模型开发和数据提取提供了自动化、用户友好的工作流程。该平台仅使用少量注释良好的文章即可实现卓越的准确性。这种创新工具简化了从科学文献到结构化知识和数据的过渡，并有利于自然信息学的进步。

Title: Benchmarking pre-trained text embedding models in aligning built asset information

• Authors: Mehrzad Shahinmoghadam, Ali Motamedi
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.12056
• Pdf URL: https://arxiv.org/pdf/2411.12056
• Copy Paste: [[2411.12056]] Benchmarking pre-trained text embedding models in aligning built asset information(https://arxiv.org/abs/2411.12056)
• Keywords: language model
• Abstract: Accurate mapping of the built asset information to established data classification systems and taxonomies is crucial for effective asset management, whether for compliance at project handover or ad-hoc data integration scenarios. Due to the complex nature of built asset data, which predominantly comprises technical text elements, this process remains largely manual and reliant on domain expert input. Recent breakthroughs in contextual text representation learning (text embedding), particularly through pre-trained large language models, offer promising approaches that can facilitate the automation of cross-mapping of the built asset data. However, no comprehensive evaluation has yet been conducted to assess these models' ability to effectively represent the complex semantics specific to built asset technical terminology. This study presents a comparative benchmark of state-of-the-art text embedding models to evaluate their effectiveness in aligning built asset information with domain-specific technical concepts. Our proposed datasets are derived from two renowned built asset data classification dictionaries. The results of our benchmarking across six proposed datasets, covering three tasks of clustering, retrieval, and reranking, highlight the need for future research on domain adaptation techniques. The benchmarking resources are published as an open-source library, which will be maintained and extended to support future evaluations in this field.
• 摘要：将建造资产信息准确映射到已建立的数据分类系统和分类法对于有效的资产管理至关重要，无论是为了项目交接时的合规性还是临时数据集成场景。由于建造资产数据的复杂性，其主要由技术文本元素组成，因此该过程仍然主要是手动的并且依赖于领域专家的输入。上下文文本表示学习（文本嵌入）方面的最新突破，特别是通过预先训练的大型语言模型，提供了有前途的方法，可以促进建造资产数据交叉映射的自动化。但是，尚未进行全面评估以评估这些模型有效表示特定于建造资产技术术语的复杂语义的能力。本研究提出了最先进的文本嵌入模型的比较基准，以评估它们在将建造资产信息与特定领域的技术概念对齐方面的有效性。我们提出的数据集来自两个著名的建造资产数据分类词典。我们对六个拟议数据集（涵盖聚类、检索和重新排序三个任务）进行了基准测试，结果突出表明，未来需要研究领域适应技术。基准测试资源以开源库的形式发布，我们将对其进行维护和扩展，以支持该领域的未来评估。

Title: Mitigating Gender Bias in Contextual Word Embeddings

• Authors: Navya Yarrabelly, Vinay Damodaran, Feng-Guang Su
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.12074
• Pdf URL: https://arxiv.org/pdf/2411.12074
• Copy Paste: [[2411.12074]] Mitigating Gender Bias in Contextual Word Embeddings(https://arxiv.org/abs/2411.12074)
• Keywords: language model
• Abstract: Word embeddings have been shown to produce remarkable results in tackling a vast majority of NLP related tasks. Unfortunately, word embeddings also capture the stereotypical biases that are prevalent in society, affecting the predictive performance of the embeddings when used in downstream tasks. While various techniques have been proposed \cite{bolukbasi2016man, zhao2018learning} and criticized\cite{gonen2019lipstick} for static embeddings, very little work has focused on mitigating bias in contextual embeddings. In this paper, we propose a novel objective function for MLM(Masked-Language Modeling) which largely mitigates the gender bias in contextual embeddings and also preserves the performance for downstream tasks. Since previous works on measuring bias in contextual embeddings lack in normative reasoning, we also propose novel evaluation metrics that are straight-forward and aligned with our motivations in debiasing. We also propose new methods for debiasing static embeddings and provide empirical proof via extensive analysis and experiments, as to why the main source of bias in static embeddings stems from the presence of stereotypical names rather than gendered words themselves. All experiments and embeddings studied are in English, unless otherwise specified.\citep{bender2011achieving}.
• 摘要：事实证明，词嵌入在处理绝大多数 NLP 相关任务时都能产生显著的效果。不幸的是，词嵌入还会捕捉到社会中普遍存在的刻板偏见，从而影响嵌入在下游任务中的预测性能。虽然已经提出了各种技术 \cite{bolukbasi2016man, zhao2018learning} 并批评 \cite{gonen2019lipstick} 用于静态嵌入，但很少有工作专注于减轻上下文嵌入中的偏见。在本文中，我们为 MLM（蒙版语言建模）提出了一种新颖的目标函数，它在很大程度上减轻了上下文嵌入中的性别偏见，同时还保留了下游任务的性能。由于以前关于衡量上下文嵌入中的偏见的研究缺乏规范推理，我们还提出了新颖的评估指标，这些指标直截了当且与我们消除偏见的动机一致。我们还提出了消除静态嵌入偏差的新方法，并通过广泛的分析和实验提供了实证证据，说明为什么静态嵌入偏差的主要来源是刻板名称的存在，而不是性别词本身。除非另有说明，否则所有实验和嵌入研究均以英文进行。\citep{bender2011achieving}。

Title: Does Unlearning Truly Unlearn? A Black Box Evaluation of LLM Unlearning Methods

• Authors: Jai Doshi, Asa Cooper Stickland
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.12103
• Pdf URL: https://arxiv.org/pdf/2411.12103
• Copy Paste: [[2411.12103]] Does Unlearning Truly Unlearn? A Black Box Evaluation of LLM Unlearning Methods(https://arxiv.org/abs/2411.12103)
• Keywords: language model, llm, prompt
• Abstract: Large language model unlearning aims to remove harmful information that LLMs have learnt to prevent their use for malicious purposes. LLMU and RMU have been proposed as two methods for LLM unlearning, achieving impressive results on unlearning benchmarks. We study in detail the efficacy of these methods by evaluating their impact on general model capabilities on the WMDP benchmark as well as a biology benchmark we create. Our experiments show that RMU generally leads to better preservation of model capabilities, for similar or better unlearning. We further test the robustness of these methods and find that doing 5-shot prompting or rephrasing the question in simple ways can lead to an over ten-fold increase in accuracy on unlearning benchmarks. Finally, we show that training on unrelated data can almost completely recover pre-unlearning performance, demonstrating that these methods fail at truly unlearning. The code is available at $\href{this https URL}{this\, https\, URL}$.
• 摘要：大型语言模型反学习旨在消除 LLM 学到的有害信息，以防止它们被用于恶意目的。LLMU 和 RMU 已被提议作为 LLM 反学习的两种方法，在反学习基准上取得了令人印象深刻的结果。我们通过评估它们对 WMDP 基准以及我们创建的生物学基准上的一般模型能力的影响来详细研究这些方法的有效性。我们的实验表明，RMU 通常可以更好地保留模型能力，以实现类似或更好的反学习。我们进一步测试了这些方法的稳健性，发现进行 5 次提示或以简单的方式重新表述问题可以使反学习基准的准确率提高十倍以上。最后，我们表明，对不相关数据的训练几乎可以完全恢复反学习前的性能，表明这些方法无法真正实现反学习。代码可在 $\href{this https URL}{this\, https\, URL}$ 获得。

Title: CoMeDi Shared Task: Models as Annotators in Lexical Semantics Disagreements

• Authors: Zhu Liu, Zhen Hu, Ying Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.12147
• Pdf URL: https://arxiv.org/pdf/2411.12147
• Copy Paste: [[2411.12147]] CoMeDi Shared Task: Models as Annotators in Lexical Semantics Disagreements(https://arxiv.org/abs/2411.12147)
• Keywords: language model
• Abstract: We present the results of our system for the CoMeDi Shared Task, which predicts majority votes (Subtask 1) and annotator disagreements (Subtask 2). Our approach combines model ensemble strategies with MLP-based and threshold-based methods trained on pretrained language models. Treating individual models as virtual annotators, we simulate the annotation process by designing aggregation measures that incorporate continuous similarity scores and discrete classification labels to capture both majority and disagreement. Additionally, we employ anisotropy removal techniques to enhance performance. Experimental results demonstrate the effectiveness of our methods, particularly for Subtask 2. Notably, we find that continuous similarity scores, even within the same model, align better with human disagreement patterns compared to aggregated discrete labels.
• 摘要：我们展示了系统在 CoMeDi 共享任务中的结果，该系统预测多数票（子任务 1）和注释者分歧（子任务 2）。我们的方法将模型集成策略与基于 MLP 和阈值的方法相结合，这些方法在预训练语言模型上进行训练。我们将各个模型视为虚拟注释者，通过设计聚合度量来模拟注释过程，这些度量结合了连续相似度分数和离散分类标签来捕获多数和分歧。此外，我们采用了各向异性消除技术来提高性能。实验结果证明了我们方法的有效性，尤其是对于子任务 2。值得注意的是，我们发现，与聚合离散标签相比，即使在同一个模型中，连续相似度分数也更符合人类的分歧模式。

Title: HNCSE: Advancing Sentence Embeddings via Hybrid Contrastive Learning with Hard Negatives

• Authors: Wenxiao Liu, Zihong Yang, Chaozhuo Li, Zijin Hong, Jianfeng Ma, Zhiquan Liu, Litian Zhang, Feiran Huang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.12156
• Pdf URL: https://arxiv.org/pdf/2411.12156
• Copy Paste: [[2411.12156]] HNCSE: Advancing Sentence Embeddings via Hybrid Contrastive Learning with Hard Negatives(https://arxiv.org/abs/2411.12156)
• Keywords: llm
• Abstract: Unsupervised sentence representation learning remains a critical challenge in modern natural language processing (NLP) research. Recently, contrastive learning techniques have achieved significant success in addressing this issue by effectively capturing textual semantics. Many such approaches prioritize the optimization using negative samples. In fields such as computer vision, hard negative samples (samples that are close to the decision boundary and thus more difficult to distinguish) have been shown to enhance representation learning. However, adapting hard negatives to contrastive sentence learning is complex due to the intricate syntactic and semantic details of text. To address this problem, we propose HNCSE, a novel contrastive learning framework that extends the leading SimCSE approach. The hallmark of HNCSE is its innovative use of hard negative samples to enhance the learning of both positive and negative samples, thereby achieving a deeper semantic understanding. Empirical tests on semantic textual similarity and transfer task datasets validate the superiority of HNCSE.
• 摘要：无监督句子表征学习仍然是现代自然语言处理 (NLP) 研究中的一个关键挑战。最近，对比学习技术通过有效捕获文本语义在解决此问题方面取得了重大成功。许多此类方法优先使用负样本进行优化。在计算机视觉等领域，硬负样本（接近决策边界因此更难区分的样本）已被证明可以增强表征学习。然而，由于文本的句法和语义细节错综复杂，将硬负样本应用于对比句子学习非常复杂。为了解决这个问题，我们提出了 HNCSE，这是一个扩展了领先的 SimCSE 方法的新型对比学习框架。HNCSE 的标志是它创新地使用硬负样本来增强正样本和负样本的学习，从而实现更深入的语义理解。在语义文本相似性和迁移任务数据集上的实证测试验证了 HNCSE 的优越性。

Title: A Combined Encoder and Transformer Approach for Coherent and High-Quality Text Generation

• Authors: Jiajing Chen, Shuo Wang, Zhen Qi, Zhenhong Zhang, Chihang Wang, Hongye Zheng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.12157
• Pdf URL: https://arxiv.org/pdf/2411.12157
• Copy Paste: [[2411.12157]] A Combined Encoder and Transformer Approach for Coherent and High-Quality Text Generation(https://arxiv.org/abs/2411.12157)
• Keywords: gpt, agent
• Abstract: This research introduces a novel text generation model that combines BERT's semantic interpretation strengths with GPT-4's generative capabilities, establishing a high standard in generating coherent, contextually accurate language. Through the combined architecture, the model enhances semantic depth and maintains smooth, human-like text flow, overcoming limitations seen in prior models. Experimental benchmarks reveal that BERT-GPT-4 surpasses traditional models, including GPT-3, T5, BART, Transformer-XL, and CTRL, in key metrics like Perplexity and BLEU, showcasing its superior natural language generation performance. By fully utilizing contextual information, this hybrid model generates text that is not only logically coherent but also aligns closely with human language patterns, providing an advanced solution for text generation tasks. This research highlights the potential of integrating semantic understanding with advanced generative models, contributing new insights for NLP, and setting a foundation for broader applications of large-scale generative architectures in areas such as automated writing, question-answer systems, and adaptive conversational agents.
• 摘要：本研究引入了一种新颖的文本生成模型，该模型将 BERT 的语义解释优势与 GPT-4 的生成能力相结合，在生成连贯、上下文准确的语言方面建立了高标准。通过组合架构，该模型增强了语义深度并保持了流畅、类似人类的文本流，克服了先前模型中的局限性。实验基准测试表明，BERT-GPT-4 在困惑度和 BLEU 等关键指标上超越了传统模型，包括 GPT-3、T5、BART、Transformer-XL 和 CTRL，展示了其卓越的自然语言生成性能。通过充分利用上下文信息，该混合模型生成的文本不仅在逻辑上连贯，而且与人类语言模式紧密相关，为文本生成任务提供了先进的解决方案。这项研究强调了将语义理解与高级生成模型相结合的潜力，为 NLP 贡献了新的见解，并为大规模生成架构在自动写作、问答系统和自适应对话代理等领域的更广泛应用奠定了基础。

Title: Evaluating Tokenizer Performance of Large Language Models Across Official Indian Languages

• Authors: S. Tamang, D. J. Bora
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.12240
• Pdf URL: https://arxiv.org/pdf/2411.12240
• Copy Paste: [[2411.12240]] Evaluating Tokenizer Performance of Large Language Models Across Official Indian Languages(https://arxiv.org/abs/2411.12240)
• Keywords: language model, gpt, llm
• Abstract: Large Language Models (LLMs) based on transformer architectures have revolutionized a variety of domains, with tokenization playing a pivotal role in their pre-processing and fine-tuning stages. In multilingual models, particularly those tailored for Indic languages, effective tokenization is crucial for optimizing performance. This paper presents a comprehensive evaluation of tokenizers used by 12 LLMs across all 22 official languages of India, with a focus on comparing the efficiency of their tokenization processes. We employed the Normalized Sequence Length (NSL) as a key metric in our analysis. Our findings reveal that the SUTRA tokenizer outperforms all other models, including several Indic-specific models, excelling in 14 languages. Notable insights include the SUTRA tokenizer's superior handling of Indic languages, GPT-4o's advancement over its predecessor GPT-4 in processing Indian languages, and the limited performance of Project Indus in certain languages. This study underscores the critical importance of developing targeted tokenization strategies for multilingual and Indic-centric models, laying the groundwork for future improvements in tokenizer design to enhance linguistic coverage and model efficiency.
• 摘要：基于 Transformer 架构的大型语言模型 (LLM) 已经彻底改变了各种领域，而标记化在其预处理和微调阶段起着关键作用。在多语言模型中，尤其是针对印度语定制的模型，有效的标记化对于优化性能至关重要。本文对印度所有 22 种官方语言的 12 个 LLM 使用的标记器进行了全面评估，重点比较了它们的标记化过程的效率。我们在分析中使用了规范化序列长度 (NSL) 作为关键指标。我们的研究结果表明，SUTRA 标记器优于所有其他模型，包括几种特定于印度语的模型，在 14 种语言中表现出色。值得注意的见解包括 SUTRA 标记器对印度语的出色处理、GPT-4o 在处理印度语方面优于其前身 GPT-4 的进步以及 Project Indus 在某些语言中的表现有限。这项研究强调了为多语言和以印度语为中心的模型开发有针对性的标记化策略的关键重要性，为未来改进标记器设计以增强语言覆盖范围和模型效率奠定了基础。

Title: Predicting User Intents and Musical Attributes from Music Discovery Conversations

• Authors: Daeyong Kwon, SeungHeon Doh, Juhan Nam
• Subjects: cs.CL, cs.LG, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2411.12254
• Pdf URL: https://arxiv.org/pdf/2411.12254
• Copy Paste: [[2411.12254]] Predicting User Intents and Musical Attributes from Music Discovery Conversations(https://arxiv.org/abs/2411.12254)
• Keywords: language model, chat
• Abstract: Intent classification is a text understanding task that identifies user needs from input text queries. While intent classification has been extensively studied in various domains, it has not received much attention in the music domain. In this paper, we investigate intent classification models for music discovery conversation, focusing on pre-trained language models. Rather than only predicting functional needs: intent classification, we also include a task for classifying musical needs: musical attribute classification. Additionally, we propose a method of concatenating previous chat history with just single-turn user queries in the input text, allowing the model to understand the overall conversation context better. Our proposed model significantly improves the F1 score for both user intent and musical attribute classification, and surpasses the zero-shot and few-shot performance of the pretrained Llama 3 model.
• 摘要：意图分类是一种文本理解任务，可从输入文本查询中识别用户需求。虽然意图分类已在各个领域得到广泛研究，但在音乐领域却并未受到太多关注。在本文中，我们研究了用于音乐发现对话的意图分类模型，重点关注预训练语言模型。除了预测功能需求（意图分类）之外，我们还包括一项用于对音乐需求进行分类的任务：音乐属性分类。此外，我们提出了一种将之前的聊天记录与输入文本中的单轮用户查询连接起来的方法，使模型能够更好地理解整体对话上下文。我们提出的模型显著提高了用户意图和音乐属性分类的 F1 分数，并超越了预训练的 Llama 3 模型的零样本和少样本性能。

Title: CUE-M: Contextual Understanding and Enhanced Search with Multimodal Large Language Model

• Authors: Dongyoung Go, Taesun Whang, Chanhee Lee, Hwayeon Kim, Sunghoon Park, Seunghwan Ji, Dongchan Kim, Young-Bum Kim
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.12287
• Pdf URL: https://arxiv.org/pdf/2411.12287
• Copy Paste: [[2411.12287]] CUE-M: Contextual Understanding and Enhanced Search with Multimodal Large Language Model(https://arxiv.org/abs/2411.12287)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: The integration of Retrieval-Augmented Generation (RAG) with Multimodal Large Language Models (MLLMs) has expanded the scope of multimodal query resolution. However, current systems struggle with intent understanding, information retrieval, and safety filtering, limiting their effectiveness. This paper introduces Contextual Understanding and Enhanced Search with MLLM (CUE-M), a novel multimodal search pipeline that addresses these challenges through a multi-stage framework comprising image context enrichment, intent refinement, contextual query generation, external API integration, and relevance-based filtering. CUE-M incorporates a robust safety framework combining image-based, text-based, and multimodal classifiers, dynamically adapting to instance- and category-specific risks. Evaluations on a multimodal Q&A dataset and a public safety benchmark demonstrate that CUE-M outperforms baselines in accuracy, knowledge integration, and safety, advancing the capabilities of multimodal retrieval systems.
• 摘要：检索增强生成 (RAG) 与多模态大型语言模型 (MLLM) 的集成扩大了多模态查询解析的范围。然而，当前的系统在意图理解、信息检索和安全过滤方面存在困难，限制了它们的有效性。本文介绍了使用 MLLM 的上下文理解和增强搜索 (CUE-M)，这是一种新颖的多模态搜索管道，它通过一个多阶段框架解决这些挑战，该框架包括图像上下文丰富、意图细化、上下文查询生成、外部 API 集成和基于相关性的过滤。CUE-M 结合了一个强大的安全框架，结合了基于图像、基于文本和多模态分类器，可动态适应特定于实例和类别的风险。对多模态问答数据集和公共安全基准的评估表明，CUE-M 在准确性、知识集成和安全性方面优于基线，从而提高了多模态检索系统的能力。

Title: Balancing Accuracy and Efficiency in Multi-Turn Intent Classification for LLM-Powered Dialog Systems in Production

• Authors: Junhua Liu, Yong Keat Tan, Bin Fu, Kwan Hui Lim
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2411.12307
• Pdf URL: https://arxiv.org/pdf/2411.12307
• Copy Paste: [[2411.12307]] Balancing Accuracy and Efficiency in Multi-Turn Intent Classification for LLM-Powered Dialog Systems in Production(https://arxiv.org/abs/2411.12307)
• Keywords: language model, llm
• Abstract: Accurate multi-turn intent classification is essential for advancing conversational AI systems. However, challenges such as the scarcity of comprehensive datasets and the complexity of contextual dependencies across dialogue turns hinder progress. This paper presents two novel approaches leveraging Large Language Models (LLMs) to enhance scalability and reduce latency in production dialogue systems. First, we introduce Symbol Tuning, which simplifies intent labels to reduce task complexity and improve performance in multi-turn dialogues. Second, we propose C-LARA (Consistency-aware, Linguistics Adaptive Retrieval Augmentation), a framework that employs LLMs for data augmentation and pseudo-labeling to generate synthetic multi-turn dialogues. These enriched datasets are used to fine-tune a small, efficient model suitable for deployment. Experiments conducted on multilingual dialogue datasets demonstrate significant improvements in classification accuracy and resource efficiency. Our methods enhance multi-turn intent classification accuracy by 5.09%, reduce annotation costs by 40%, and enable scalable deployment in low-resource multilingual industrial systems, highlighting their practicality and impact.
• 摘要：准确的多轮意图分类对于推进对话式 AI 系统至关重要。然而，诸如综合数据集的稀缺性和对话轮次间上下文依赖关系的复杂性等挑战阻碍了进展。本文介绍了两种利用大型语言模型 (LLM) 来增强可扩展性和减少生产对话系统延迟的新方法。首先，我们引入了符号调整，它简化了意图标签以降低任务复杂性并提高多轮对话的性能。其次，我们提出了 C-LARA（一致性感知、语言学自适应检索增强），这是一个使用 LLM 进行数据增强和伪标记以生成合成多轮对话的框架。这些丰富的数据集用于微调适合部署的小型高效模型。在多语言对话数据集上进行的实验表明分类准确性和资源效率有显著提高。我们的方法将多轮意图分类准确率提高了 5.09%，将注释成本降低了 40%，并能够在低资源多语言工业系统中进行可扩展部署，凸显了其实用性和影响力。

Title: RedPajama: an Open Dataset for Training Large Language Models

• Authors: Maurice Weber, Daniel Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, Ben Athiwaratkun, Rahul Chalamala, Kezhen Chen, Max Ryabinin, Tri Dao, Percy Liang, Christopher Ré, Irina Rish, Ce Zhang
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.12372
• Pdf URL: https://arxiv.org/pdf/2411.12372
• Copy Paste: [[2411.12372]] RedPajama: an Open Dataset for Training Large Language Models(https://arxiv.org/abs/2411.12372)
• Keywords: language model
• Abstract: Large language models are increasingly becoming a cornerstone technology in artificial intelligence, the sciences, and society as a whole, yet the optimal strategies for dataset composition and filtering remain largely elusive. Many of the top-performing models lack transparency in their dataset curation and model development processes, posing an obstacle to the development of fully open language models. In this paper, we identify three core data-related challenges that must be addressed to advance open-source language models. These include (1) transparency in model development, including the data curation process, (2) access to large quantities of high-quality data, and (3) availability of artifacts and metadata for dataset curation and analysis. To address these challenges, we release RedPajama-V1, an open reproduction of the LLaMA training dataset. In addition, we release RedPajama-V2, a massive web-only dataset consisting of raw, unfiltered text data together with quality signals and metadata. Together, the RedPajama datasets comprise over 100 trillion tokens spanning multiple domains and with their quality signals facilitate the filtering of data, aiming to inspire the development of numerous new datasets. To date, these datasets have already been used in the training of strong language models used in production, such as Snowflake Arctic, Salesforce's XGen and AI2's OLMo. To provide insight into the quality of RedPajama, we present a series of analyses and ablation studies with decoder-only language models with up to 1.6B parameters. Our findings demonstrate how quality signals for web data can be effectively leveraged to curate high-quality subsets of the dataset, underscoring the potential of RedPajama to advance the development of transparent and high-performing language models at scale.
• 摘要：大型语言模型正日益成为人工智能、科学和整个社会的基石技术，但数据集组成和过滤的最佳策略仍然难以捉摸。许多表现最佳的模型在数据集管理和模型开发过程中缺乏透明度，这对完全开放的语言模型的开发构成了障碍。在本文中，我们确定了三个核心数据相关挑战，必须解决这些挑战才能推进开源语言模型的发展。这些挑战包括 (1) 模型开发的透明度，包括数据管理过程，(2) 访问大量高质量数据，以及 (3) 数据集管理和分析的工件和元数据的可用性。为了应对这些挑战，我们发布了 RedPajama-V1，这是 LLaMA 训练数据集的开放复制品。此外，我们还发布了 RedPajama-V2，这是一个庞大的仅限网络的数据集，由原始、未过滤的文本数据以及质量信号和元数据组成。 RedPajama 数据集总共包含 100 多万亿个词条，涵盖多个领域，其质量信号有助于筛选数据，旨在激发大量新数据集的开发。到目前为止，这些数据集已经用于训练生产中使用的强语言模型，例如 Snowflake Arctic、Salesforce 的 XGen 和 AI2 的 OLMo。为了深入了解 RedPajama 的质量，我们针对多达 16 亿个参数的解码器专用语言模型进行了一系列分析和消融研究。我们的研究结果表明，如何有效利用网络数据的质量信号来筛选高质量的数据集子集，凸显了 RedPajama 在推动大规模透明高性能语言模型开发方面的潜力。

Title: Do LLMs Understand Ambiguity in Text? A Case Study in Open-world Question Answering

• Authors: Aryan Keluskar, Amrita Bhattacharjee, Huan Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.12395
• Pdf URL: https://arxiv.org/pdf/2411.12395
• Copy Paste: [[2411.12395]] Do LLMs Understand Ambiguity in Text? A Case Study in Open-world Question Answering(https://arxiv.org/abs/2411.12395)
• Keywords: language model, llm, hallucination
• Abstract: Ambiguity in natural language poses significant challenges to Large Language Models (LLMs) used for open-domain question answering. LLMs often struggle with the inherent uncertainties of human communication, leading to misinterpretations, miscommunications, hallucinations, and biased responses. This significantly weakens their ability to be used for tasks like fact-checking, question answering, feature extraction, and sentiment analysis. Using open-domain question answering as a test case, we compare off-the-shelf and few-shot LLM performance, focusing on measuring the impact of explicit disambiguation strategies. We demonstrate how simple, training-free, token-level disambiguation methods may be effectively used to improve LLM performance for ambiguous question answering tasks. We empirically show our findings and discuss best practices and broader impacts regarding ambiguity in LLMs.
• 摘要：自然语言中的歧义性对用于开放域问答的大型语言模型 (LLM) 提出了重大挑战。LLM 经常难以应对人类交流中固有的不确定性，从而导致误解、误传、幻觉和有偏见的回应。这大大削弱了它们用于事实核查、问答、特征提取和情绪分析等任务的能力。使用开放域问答作为测试用例，我们比较了现成的和少量样本的 LLM 性能，重点衡量显式消歧策略的影响。我们展示了如何有效地使用简单、无需训练的标记级消歧方法来提高 LLM 在模糊问答任务中的性能。我们通过实证展示了我们的发现，并讨论了 LLM 中歧义方面的最佳实践和更广泛的影响。

Title: Evaluating the Prompt Steerability of Large Language Models

• Authors: Erik Miehling, Michael Desmond, Karthikeyan Natesan Ramamurthy, Elizabeth M. Daly, Pierre Dognin, Jesus Rios, Djallel Bouneffouf, Miao Liu
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2411.12405
• Pdf URL: https://arxiv.org/pdf/2411.12405
• Copy Paste: [[2411.12405]] Evaluating the Prompt Steerability of Large Language Models(https://arxiv.org/abs/2411.12405)
• Keywords: language model, prompt
• Abstract: Building pluralistic AI requires designing models that are able to be shaped to represent a wide range of value systems and cultures. Achieving this requires first being able to evaluate the degree to which a given model is capable of reflecting various personas. To this end, we propose a benchmark for evaluating the steerability of model personas as a function of prompting. Our design is based on a formal definition of prompt steerability, which analyzes the degree to which a model's joint behavioral distribution can be shifted from its baseline behavior. By defining steerability indices and inspecting how these indices change as a function of steering effort, we can estimate the steerability of a model across various persona dimensions and directions. Our benchmark reveals that the steerability of many current models is limited -- due to both a skew in their baseline behavior and an asymmetry in their steerability across many persona dimensions. We release an implementation of our benchmark at this https URL.
• 摘要：构建多元化人工智能需要设计能够代表各种价值体系和文化的模型。要实现这一点，首先需要能够评估给定模型能够反映各种角色的程度。为此，我们提出了一个基准，用于评估模型角色作为提示函数的可操纵性。我们的设计基于提示可操纵性的正式定义，该定义分析了模型的联合行为分布可以从其基线行为偏离的程度。通过定义可操纵性指标并检查这些指标如何随着转向努力而变化，我们可以估计模型在各种角色维度和方向上的可操纵性。我们的基准表明，许多当前模型的可操纵性是有限的——因为它们的基线行为存在偏差，并且在许多角色维度上的可操纵性不对称。我们在此 https URL 上发布了基准的实现。

Title: \textsc{Neon}: News Entity-Interaction Extraction for Enhanced Question Answering

• Authors: Sneha Singhania, Silviu Cucerzan, Allen Herring, Sujay Kumar Jauhar
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2411.12449
• Pdf URL: https://arxiv.org/pdf/2411.12449
• Copy Paste: [[2411.12449]] \textsc{Neon}: News Entity-Interaction Extraction for Enhanced Question Answering(https://arxiv.org/abs/2411.12449)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: Capturing fresh information in near real-time and using it to augment existing large language models (LLMs) is essential to generate up-to-date, grounded, and reliable output. This problem becomes particularly challenging when LLMs are used for informational tasks in rapidly evolving fields, such as Web search related to recent or unfolding events involving entities, where generating temporally relevant responses requires access to up-to-the-hour news sources. However, the information modeled by the parametric memory of LLMs is often outdated, and Web results from prototypical retrieval systems may fail to capture the latest relevant information and struggle to handle conflicting reports in evolving news. To address this challenge, we present the NEON framework, designed to extract emerging entity interactions -- such as events or activities -- as described in news articles. NEON constructs an entity-centric timestamped knowledge graph that captures such interactions, thereby facilitating enhanced QA capabilities related to news events. Our framework innovates by integrating open Information Extraction (openIE) style tuples into LLMs to enable in-context retrieval-augmented generation. This integration demonstrates substantial improvements in QA performance when tackling temporal, entity-centric search queries. Through NEON, LLMs can deliver more accurate, reliable, and up-to-date responses.
• 摘要：近乎实时地捕获新鲜信息并用它来增强现有的大型语言模型 (LLM) 对于生成最新、扎实和可靠的输出至关重要。当 LLM 用于快速发展领域的信息任务时，这个问题变得尤其具有挑战性，例如与涉及实体的最近或正在发生的事件相关的 Web 搜索，其中生成时间相关的响应需要访问最新的新闻来源。然而，由 LLM 的参数记忆建模的信息通常已经过时，而原型检索系统的 Web 结果可能无法捕获最新的相关信息，并且难以处理不断变化的新闻中的相互矛盾的报道。为了应对这一挑战，我们提出了 NEON 框架，旨在提取新闻文章中描述的新兴实体交互（例如事件或活动）。NEON 构建了一个以实体为中心的带时间戳的知识图谱来捕获此类交互，从而促进与新闻事件相关的增强 QA 功能。我们的框架通过将开放信息提取 (openIE) 样式元组集成到 LLM 中来实现创新，从而实现上下文检索增强生成。这种集成在处理时间、以实体为中心的搜索查询时，QA 性能得到了显著提升。通过 NEON，LLM 可以提供更准确、更可靠、更最新的响应。

Title: Guide-to-Explain for Controllable Summarization

• Authors: Sangwon Ryu, Heejin Do, Daehee Kim, Yunsu Kim, Gary Geunbae Lee, Jungseul Ok
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.12460
• Pdf URL: https://arxiv.org/pdf/2411.12460
• Copy Paste: [[2411.12460]] Guide-to-Explain for Controllable Summarization(https://arxiv.org/abs/2411.12460)
• Keywords: language model, llm
• Abstract: Recently, large language models (LLMs) have demonstrated remarkable performance in abstractive summarization tasks. However, controllable summarization with LLMs remains underexplored, limiting their ability to generate summaries that align with specific user preferences. In this paper, we first investigate the capability of LLMs to control diverse attributes, revealing that they encounter greater challenges with numerical attributes, such as length and extractiveness, compared to linguistic attributes. To address this challenge, we propose a guide-to-explain framework (GTE) for controllable summarization. Our GTE framework enables the model to identify misaligned attributes in the initial draft and guides it in explaining errors in the previous output. Based on this reflection, the model generates a well-adjusted summary. As a result, by allowing the model to reflect on its misalignment, we generate summaries that satisfy the desired attributes in surprisingly fewer iterations than other iterative methods solely using LLMs.
• 摘要：最近，大型语言模型 (LLM) 在抽象摘要任务中表现出色。然而，使用 LLM 进行可控摘要仍未得到充分探索，这限制了它们生成符合特定用户偏好的摘要的能力。在本文中，我们首先研究了 LLM 控制各种属性的能力，结果表明，与语言属性相比，它们在数字属性（例如长度和可提取性）方面遇到的挑战更大。为了应对这一挑战，我们提出了一个用于可控摘要的指南解释框架 (GTE)。我们的 GTE 框架使模型能够识别初稿中未对齐的属性，并指导它解释先前输出中的错误。基于这种反思，模型生成了一个经过良好调整的摘要。因此，通过允许模型反思其错位，我们生成了满足所需属性的摘要，而迭代次数比其他仅使用 LLM 的迭代方法少得多。

Title: Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models

• Authors: Laura Ruis, Maximilian Mozes, Juhan Bae, Siddhartha Rao Kamalakara, Dwarak Talupuru, Acyr Locatelli, Robert Kirk, Tim Rocktäschel, Edward Grefenstette, Max Bartolo
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.12580
• Pdf URL: https://arxiv.org/pdf/2411.12580
• Copy Paste: [[2411.12580]] Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models(https://arxiv.org/abs/2411.12580)
• Keywords: language model, llm
• Abstract: The capabilities and limitations of Large Language Models have been sketched out in great detail in recent years, providing an intriguing yet conflicting picture. On the one hand, LLMs demonstrate a general ability to solve problems. On the other hand, they show surprising reasoning gaps when compared to humans, casting doubt on the robustness of their generalisation strategies. The sheer volume of data used in the design of LLMs has precluded us from applying the method traditionally used to measure generalisation: train-test set separation. To overcome this, we study what kind of generalisation strategies LLMs employ when performing reasoning tasks by investigating the pretraining data they rely on. For two models of different sizes (7B and 35B) and 2.5B of their pretraining tokens, we identify what documents influence the model outputs for three simple mathematical reasoning tasks and contrast this to the data that are influential for answering factual questions. We find that, while the models rely on mostly distinct sets of data for each factual question, a document often has a similar influence across different reasoning questions within the same task, indicating the presence of procedural knowledge. We further find that the answers to factual questions often show up in the most influential data. However, for reasoning questions the answers usually do not show up as highly influential, nor do the answers to the intermediate reasoning steps. When we characterise the top ranked documents for the reasoning questions qualitatively, we confirm that the influential documents often contain procedural knowledge, like demonstrating how to obtain a solution using formulae or code. Our findings indicate that the approach to reasoning the models use is unlike retrieval, and more like a generalisable strategy that synthesises procedural knowledge from documents doing a similar form of reasoning.
• 摘要：近年来，大型语言模型的能力和局限性已被详细阐述，呈现出一幅有趣而又相互矛盾的图景。一方面，LLM 展示了解决问题的一般能力。另一方面，与人类相比，它们表现出令人惊讶的推理差距，这让人对其泛化策略的稳健性产生了怀疑。LLM 设计中使用的数据量巨大，使我们无法应用传统上用于衡量泛化的方法：训练集-测试集分离。为了克服这个问题，我们通过研究 LLM 所依赖的预训练数据来研究 LLM 在执行推理任务时采用的泛化策略类型。对于两个不同大小（7B 和 35B）的模型及其 2.5B 的预训练标记，我们确定哪些文档会影响三个简单数学推理任务的模型输出，并将其与对回答事实问题有影响的数据进行对比。我们发现，虽然模型依赖于每个事实问题的不同数据集，但文档在同一任务中对不同推理问题的影响往往相似，这表明存在程序性知识。我们进一步发现，事实问题的答案通常出现在最有影响力的数据中。然而，对于推理问题，答案通常不会表现出很强的影响力，中间推理步骤的答案也是如此。当我们定性地描述推理问题排名靠前的文档时，我们确认有影响力的文档通常包含程序性知识，例如演示如何使用公式或代码获得解决方案。我们的研究结果表明，模型使用的推理方法不同于检索，更像是一种通用策略，它从进行类似推理的文档中综合程序性知识。

Title: Enhanced Sign Language Translation between American Sign Language (ASL) and Indian Sign Language (ISL) Using LLMs

• Authors: Malay Kumar, S. Sarvajit Visagan, Tanish Sarang Mahajan, Anisha Natarajan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.12685
• Pdf URL: https://arxiv.org/pdf/2411.12685
• Copy Paste: [[2411.12685]] Enhanced Sign Language Translation between American Sign Language (ASL) and Indian Sign Language (ISL) Using LLMs(https://arxiv.org/abs/2411.12685)
• Keywords: llm
• Abstract: We have come up with a research that hopes to provide a bridge between the users of American Sign Language and the users of spoken language and Indian Sign Language (ISL). The research enabled us to create a novel framework that we have developed for Learner Systems. Leveraging art of Large models to create key features including: - Real-time translation between these two sign languages in an efficient manner. Making LLM's capability available for seamless translations to ISL. Here is the full study showing its implementation in this paper. The core of the system is a sophisticated pipeline that begins with reclassification and recognition of ASL gestures based on a strong Random Forest Classifier. By recognizing the ASL, it is translated into text which can be more easily processed. Highly evolved natural language NLP (Natural Language Processing) techniques come in handy as they play a role in our LLM integration where you then use LLMs to be able to convert the ASL text to ISL which provides you with the intent of sentence or phrase. The final step is to synthesize the translated text back into ISL gestures, creating an end-to-end translation experience using RIFE-Net. This framework is tasked with key challenges such as automatically dealing with gesture variability and overcoming the linguistic differences between ASL and ISL. By automating the translation process, we hope to vastly improve accessibility for sign language users. No longer will the communication gap between ASL and ISL create barriers; this totally cool innovation aims to bring our communities closer together. And we believe, with full confidence in our framework, that we're able to apply the same principles across a wide variety of sign language dialects.
• 摘要：我们进行了一项研究，希望在美国手语使用者和口语及印度手语 (ISL) 使用者之间架起一座桥梁。这项研究使我们能够创建一个为学习者系统开发的新框架。利用大型模型的艺术来创建关键功能，包括：- 以高效的方式在这两种手语之间进行实时翻译。使 LLM 的功能可以无缝翻译为 ISL。以下是展示其在本文中实施的完整研究。该系统的核心是一个复杂的管道，它从基于强大的随机森林分类器对 ASL 手势进行重新分类和识别开始。通过识别 ASL，它会被翻译成更容易处理的文本。高度进化的自然语言 NLP（自然语言处理）技术派上了用场，因为它们在我们的 LLM 集成中发挥着作用，然后您可以使用 LLM 将 ASL 文本转换为 ISL，从而为您提供句子或短语的意图。最后一步是将翻译后的文本重新合成为 ISL 手势，使用 RIFE-Net 创建端到端翻译体验。该框架的任务是应对关键挑战，例如自动处理手势变化和克服 ASL 和 ISL 之间的语言差异。通过自动化翻译过程，我们希望大大提高手语用户的可访问性。ASL 和 ISL 之间的沟通差距将不再造成障碍；这项非常酷的创新旨在让我们的社区更加紧密地联系在一起。我们对我们的框架充满信心，相信我们能够在各种手语方言中应用相同的原则。

Title: Strengthening Fake News Detection: Leveraging SVM and Sophisticated Text Vectorization Techniques. Defying BERT?

• Authors: Ahmed Akib Jawad Karim, Kazi Hafiz Md Asad, Aznur Azam
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.12703
• Pdf URL: https://arxiv.org/pdf/2411.12703
• Copy Paste: [[2411.12703]] Strengthening Fake News Detection: Leveraging SVM and Sophisticated Text Vectorization Techniques. Defying BERT?(https://arxiv.org/abs/2411.12703)
• Keywords: language model
• Abstract: The rapid spread of misinformation, particularly through online platforms, underscores the urgent need for reliable detection systems. This study explores the utilization of machine learning and natural language processing, specifically Support Vector Machines (SVM) and BERT, to detect news that are fake. We employ three distinct text vectorization methods for SVM: Term Frequency Inverse Document Frequency (TF-IDF), Word2Vec, and Bag of Words (BoW) evaluating their effectiveness in distinguishing between genuine and fake news. Additionally, we compare these methods against the transformer large language model, BERT. Our comprehensive approach includes detailed preprocessing steps, rigorous model implementation, and thorough evaluation to determine the most effective techniques. The results demonstrate that while BERT achieves superior accuracy with 99.98% and an F1-score of 0.9998, the SVM model with a linear kernel and BoW vectorization also performs exceptionally well, achieving 99.81% accuracy and an F1-score of 0.9980. These findings highlight that, despite BERT's superior performance, SVM models with BoW and TF-IDF vectorization methods come remarkably close, offering highly competitive performance with the advantage of lower computational requirements.
• 摘要：虚假信息的快速传播（尤其是通过在线平台传播）凸显了对可靠检测系统的迫切需求。本研究探索了利用机器学习和自然语言处理（特别是支持向量机 (SVM) 和 BERT）来检测虚假新闻。我们为 SVM 采用了三种不同的文本向量化方法：词频逆文档频率 (TF-IDF)、Word2Vec 和词袋 (BoW)，以评估它们在区分真假新闻方面的有效性。此外，我们将这些方法与 Transformer 大型语言模型 BERT 进行了比较。我们的综合方法包括详细的预处理步骤、严格的模型实施和彻底的评估，以确定最有效的技术。结果表明，虽然 BERT 实现了 99.98% 的卓越准确率和 0.9998 的 F1 分数，但具有线性核和 BoW 向量化的 SVM 模型也表现出色，准确率达到 99.81%，F1 分数为 0.9980。这些发现强调了，尽管 BERT 的性能更优越，但采用 BoW 和 TF-IDF 矢量化方法的 SVM 模型也非常接近，具有极具竞争力的性能和较低的计算要求的优势。

Title: Enhancing Multi-Class Disease Classification: Neoplasms, Cardiovascular, Nervous System, and Digestive Disorders Using Advanced LLMs

• Authors: Ahmed Akib Jawad Karim, Muhammad Zawad Mahmud, Samiha Islam, Aznur Azam
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.12712
• Pdf URL: https://arxiv.org/pdf/2411.12712
• Copy Paste: [[2411.12712]] Enhancing Multi-Class Disease Classification: Neoplasms, Cardiovascular, Nervous System, and Digestive Disorders Using Advanced LLMs(https://arxiv.org/abs/2411.12712)
• Keywords: language model, llm
• Abstract: In this research, we explored the improvement in terms of multi-class disease classification via pre-trained language models over Medical-Abstracts-TC-Corpus that spans five medical conditions. We excluded non-cancer conditions and examined four specific diseases. We assessed four LLMs, BioBERT, XLNet, and BERT, as well as a novel base model (Last-BERT). BioBERT, which was pre-trained on medical data, demonstrated superior performance in medical text classification (97% accuracy). Surprisingly, XLNet followed closely (96% accuracy), demonstrating its generalizability across domains even though it was not pre-trained on medical data. LastBERT, a custom model based on the lighter version of BERT, also proved competitive with 87.10% accuracy (just under BERT's 89.33%). Our findings confirm the importance of specialized models such as BioBERT and also support impressions around more general solutions like XLNet and well-tuned transformer architectures with fewer parameters (in this case, LastBERT) in medical domain tasks.
• 摘要：在本研究中，我们探索了通过预训练语言模型在 Medical-Abstracts-TC-Corpus 上对五种疾病进行多类疾病分类的改进。我们排除了非癌症疾病，并研究了四种特定疾病。我们评估了四个 LLM，BioBERT、XLNet 和 BERT，以及一个新的基础模型 (Last-BERT)。在医学数据上进行预训练的 BioBERT 在医学文本分类中表现出色（准确率为 97%）。令人惊讶的是，XLNet 紧随其后（准确率为 96%），尽管它没有在医学数据上进行预训练，但证明了它在各个领域的通用性。LastBERT 是基于 BERT 轻量级版本的自定义模型，也证明了其竞争力，准确率为 87.10%（略低于 BERT 的 89.33%）。我们的研究结果证实了 BioBERT 等专门模型的重要性，同时也支持了人们对 XLNet 等更通用的解决方案以及在医疗领域任务中具有更少参数的经过良好调整的转换器架构（在本例中为 LastBERT）的印象。

Title: Information Theory of Meaningful Communication

• Authors: Doron Sivan, Misha Tsodyks
• Subjects: cs.CL, cs.IT
• Abstract URL: https://arxiv.org/abs/2411.12728
• Pdf URL: https://arxiv.org/pdf/2411.12728
• Copy Paste: [[2411.12728]] Information Theory of Meaningful Communication(https://arxiv.org/abs/2411.12728)
• Keywords: language model
• Abstract: In Shannon's seminal paper, entropy of printed English, treated as a stationary stochastic process, was estimated to be roughly 1 bit per character. However, considered as a means of communication, language differs considerably from its printed form: (i) the units of information are not characters or even words but clauses, i.e. shortest meaningful parts of speech; and (ii) what is transmitted is principally the meaning of what is being said or written, while the precise phrasing that was used to communicate the meaning is typically ignored. In this study, we show that one can leverage recently developed large language models to quantify information communicated in meaningful narratives in terms of bits of meaning per clause.
• 摘要：在香农的开创性论文中，印刷英语的熵被看作一个平稳随机过程，估计为大约每字符 1 比特。然而，作为一种交流手段，语言与其印刷形式有很大不同：(i) 信息的单位不是字符，甚至不是单词，而是从句，即最短的有意义的词类；(ii) 传递的主要是所说或所写内容的含义，而用于传达含义的精确措辞通常被忽略。在这项研究中，我们表明，人们可以利用最近开发的大型语言模型，以每从句的含义比特数来量化有意义的叙述中传达的信息。

Title: ACING: Actor-Critic for Instruction Learning in Black-Box Large Language Models

• Authors: Salma Kharrat, Fares Fourati, Marco Canini
• Subjects: cs.CL, cs.AI, cs.LG, eess.SY, math.OC
• Abstract URL: https://arxiv.org/abs/2411.12736
• Pdf URL: https://arxiv.org/pdf/2411.12736
• Copy Paste: [[2411.12736]] ACING: Actor-Critic for Instruction Learning in Black-Box Large Language Models(https://arxiv.org/abs/2411.12736)
• Keywords: language model, gpt, llm, prompt, chat
• Abstract: The effectiveness of Large Language Models (LLMs) in solving tasks vastly depends on the quality of the instructions, which often require fine-tuning through extensive human effort. This highlights the need for automated instruction optimization; however, this optimization is particularly challenging when dealing with black-box LLMs, where model parameters and gradients remain inaccessible. We propose ACING, a task-specific prompt optimization approach framed as a stateless continuous-action Reinforcement Learning (RL) problem, known as the continuum bandit setting. ACING leverages an actor-critic-based method to optimize prompts, learning from non-differentiable reward signals. We validate ACING by optimizing prompts for ChatGPT on 30 instruction-based tasks. ACING consistently outperforms baseline methods, achieving a median score improvement of 10 percentage points. Furthermore, ACING not only recovers but also surpasses human-crafted expert instructions, achieving up to a 39 percentage point improvement against human benchmarks.
• 摘要：大型语言模型 (LLM) 在解决任务方面的有效性在很大程度上取决于指令的质量，而指令通常需要通过大量的人力进行微调。这凸显了对自动指令优化的需求；然而，这种优化在处理黑盒 LLM 时尤其具有挑战性，因为黑盒 LLM 中的模型参数和梯度仍然无法访问。我们提出了 ACING，这是一种针对特定任务的提示优化方法，被设计为无状态连续动作强化学习 (RL) 问题，称为连续老虎机设置。ACING 利用基于演员-评论家的方法来优化提示，从不可微分的奖励信号中学习。我们通过在 30 个基于指令的任务上优化 ChatGPT 的提示来验证 ACING。ACING 的表现始终优于基线方法，平均得分提高了 10 个百分点。此外，ACING 不仅恢复了人类编写的专家指令，而且超越了它们，与人类基准相比，其得分提高了 39 个百分点。