2024-09-17

Title: Contextual Evaluation of Large Language Models for Classifying Tropical and Infectious Diseases

• Authors: Mercy Asiedu, Nenad Tomasev, Chintan Ghate, Tiya Tiyasirichokchai, Awa Dieng, Oluwatosin Akande, Geoffrey Siwo, Steve Adudans, Sylvanus Aitkins, Odianosen Ehiakhamen, Katherine Heller
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.09201
• Pdf URL: https://arxiv.org/pdf/2409.09201
• Copy Paste: [[2409.09201]] Contextual Evaluation of Large Language Models for Classifying Tropical and Infectious Diseases(https://arxiv.org/abs/2409.09201)
• Keywords: language model, llm, prompt
• Abstract: While large language models (LLMs) have shown promise for medical question answering, there is limited work focused on tropical and infectious disease-specific exploration. We build on an opensource tropical and infectious diseases (TRINDs) dataset, expanding it to include demographic and semantic clinical and consumer augmentations yielding 11000+ prompts. We evaluate LLM performance on these, comparing generalist and medical LLMs, as well as LLM outcomes to human experts. We demonstrate through systematic experimentation, the benefit of contextual information such as demographics, location, gender, risk factors for optimal LLM response. Finally we develop a prototype of TRINDs-LM, a research tool that provides a playground to navigate how context impacts LLM outputs for health.
• 摘要：虽然大型语言模型 (LLM) 在医学问答方面已显示出良好的前景，但专注于热带和传染病特定探索的工作却非常有限。我们以开源热带和传染病 (TRINDs) 数据集为基础，将其扩展为包括人口统计学和语义临床和消费者增强，产生 11000 多个提示。我们评估这些 LLM 的表现，将通才和医学 LLM 以及 LLM 结果与人类专家进行比较。我们通过系统实验证明了背景信息（例如人口统计学、位置、性别、风险因素）对最佳 LLM 响应的好处。最后，我们开发了 TRINDs-LM 的原型，这是一种研究工具，它提供了一个平台来探索背景如何影响 LLM 的健康输出。

Title: Autoregressive + Chain of Thought (CoT) $\simeq$ Recurrent: Recurrence's Role in Language Models and a Revist of Recurrent Transformer

• Authors: Xiang Zhang, Muhammad Abdul-Mageed, Laks V.S. Lakshmanan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.09239
• Pdf URL: https://arxiv.org/pdf/2409.09239
• Copy Paste: [[2409.09239]] Autoregressive + Chain of Thought (CoT) $\simeq$ Recurrent: Recurrence's Role in Language Models and a Revist of Recurrent Transformer(https://arxiv.org/abs/2409.09239)
• Keywords: language model, prompt
• Abstract: The Transformer architecture excels in a variety of language modeling tasks, outperforming traditional neural architectures such as RNN and LSTM. This is partially due to its elimination of recurrent connections, which allows for parallel training and a smoother flow of gradients. However, this move away from recurrent structures places the Transformer model at the lower end of Chomsky's computational hierarchy, imposing limitations on its computational abilities. Consequently, even advanced Transformer-based models face considerable difficulties in tasks like counting, string reversal, bracket pairing, and multiplication. These tasks, though seemingly elementary, require a level of computational complexity that exceeds the capabilities of the Transformer architecture. Concurrently, the emergence of ``Chain of Thought" (CoT) prompting has enabled Transformer-based language models to tackle tasks that were previously impossible or poorly executed. Despite some previous research primarily interpreting CoT from a psychological perspective, a comprehensive understanding of \textit{why} CoT proves so effective in the reasoning process remains elusive. In this work, we thoroughly investigate the influence of recurrent structures in language models on their reasoning abilities, shedding light on how the CoT approach can mimic recurrent computation and act as a bridge between autoregression and recurrence. It is this approximated recurrence that notably improves the model's performance and computational capacity. Moreover, we revisit recent recurrent-based Transformer model designs, focusing on their computational abilities through our proposed concept of ``recurrence-completeness" and identify key theoretical limitations in models like Linear Transformer and RWKV. Through this, we aim to provide insight into the neural model architectures and prompt better model design.
• 摘要：Transformer 架构在各种语言建模任务中表现出色，优于 RNN 和 LSTM 等传统神经架构。部分原因是它消除了循环连接，从而实现了并行训练和更平滑的梯度流动。然而，这种脱离循环结构的举动将 Transformer 模型置于 Chomsky 计算层次的低端，限制了其计算能力。因此，即使是基于 Transformer 的高级模型在计数、字符串反转、括号配对和乘法等任务中也面临相当大的困难。这些任务虽然看似简单，但所需的计算复杂度超出了 Transformer 架构的能力。同时，“思维链”（CoT）提示的出现使基于 Transformer 的语言模型能够解决以前不可能或执行不力的任务。尽管之前的一些研究主要从心理学的角度解释 CoT，但对 \textit{why} CoT 在推理过程中如此有效的全面理解仍然难以捉摸。在这项工作中，我们彻底研究了语言模型中循环结构对其推理能力的影响，阐明了 CoT 方法如何模拟循环计算并充当自回归和循环之间的桥梁。正是这种近似的循环显著提高了模型的性能和计算能力。此外，我们重新审视了最近基于循环的 Transformer 模型设计，通过我们提出的“循环完备性”概念关注它们的计算能力，并确定了线性 Transformer 和 RWKV 等模型中的关键理论限制。通过这种方式，我们旨在深入了解神经模型架构并促进更好的模型设计。

Title: NovAScore: A New Automated Metric for Evaluating Document Level Novelty

• Authors: Lin Ai, Ziwei Gong, Harshsaiprasad Deshpande, Alexander Johnson, Emmy Phung, Ahmad Emami, Julia Hirschberg
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.09249
• Pdf URL: https://arxiv.org/pdf/2409.09249
• Copy Paste: [[2409.09249]] NovAScore: A New Automated Metric for Evaluating Document Level Novelty(https://arxiv.org/abs/2409.09249)
• Keywords: language model, llm
• Abstract: The rapid expansion of online content has intensified the issue of information redundancy, underscoring the need for solutions that can identify genuinely new information. Despite this challenge, the research community has seen a decline in focus on novelty detection, particularly with the rise of large language models (LLMs). Additionally, previous approaches have relied heavily on human annotation, which is time-consuming, costly, and particularly challenging when annotators must compare a target document against a vast number of historical documents. In this work, we introduce NovAScore (Novelty Evaluation in Atomicity Score), an automated metric for evaluating document-level novelty. NovAScore aggregates the novelty and salience scores of atomic information, providing high interpretability and a detailed analysis of a document's novelty. With its dynamic weight adjustment scheme, NovAScore offers enhanced flexibility and an additional dimension to assess both the novelty level and the importance of information within a document. Our experiments show that NovAScore strongly correlates with human judgments of novelty, achieving a 0.626 Point-Biserial correlation on the TAP-DLND 1.0 dataset and a 0.920 Pearson correlation on an internal human-annotated dataset.
• 摘要：在线内容的快速扩张加剧了信息冗余问题，凸显了对能够识别真正新信息的解决方案的需求。尽管存在这一挑战，但研究界对新颖性检测的关注度有所下降，尤其是随着大型语言模型 (LLM) 的兴起。此外，以前的方法严重依赖人工注释，这既耗时又昂贵，而且当注释者必须将目标文档与大量历史文档进行比较时尤其具有挑战性。在这项工作中，我们引入了 NovAScore（原子性分数中的新颖性评估），这是一种用于评估文档级新颖性的自动化指标。NovAScore 汇总了原子信息的新颖性和显着性分数，提供了高可解释性和对文档新颖性的详细分析。凭借其动态权重调整方案，NovAScore 提供了增强的灵活性和额外的维度来评估文档中信息的新颖性水平和重要性。我们的实验表明，NovAScore 与人类对新颖性的判断高度相关，在 TAP-DLND 1.0 数据集上实现了 0.626 的 Point-Biserial 相关性，在内部人工注释数据集上实现了 0.920 的 Pearson 相关性。

Title: An empirical evaluation of using ChatGPT to summarize disputes for recommending similar labor and employment cases in Chinese

• Authors: Po-Hsien Wu, Chao-Lin Liu, Wei-Jie Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.09280
• Pdf URL: https://arxiv.org/pdf/2409.09280
• Copy Paste: [[2409.09280]] An empirical evaluation of using ChatGPT to summarize disputes for recommending similar labor and employment cases in Chinese(https://arxiv.org/abs/2409.09280)
• Keywords: language model, gpt, chat
• Abstract: We present a hybrid mechanism for recommending similar cases of labor and employment litigations. The classifier determines the similarity based on the itemized disputes of the two cases, that the courts prepared. We cluster the disputes, compute the cosine similarity between the disputes, and use the results as the features for the classification tasks. Experimental results indicate that this hybrid approach outperformed our previous system, which considered only the information about the clusters of the disputes. We replaced the disputes that were prepared by the courts with the itemized disputes that were generated by GPT-3.5 and GPT-4, and repeated the same experiments. Using the disputes generated by GPT-4 led to better results. Although our classifier did not perform as well when using the disputes that the ChatGPT generated, the results were satisfactory. Hence, we hope that the future large-language models will become practically useful.
• 摘要：我们提出了一种混合机制，用于推荐类似的劳动和就业诉讼案件。分类器根据法院准备的两个案件的逐项争议来确定相似性。我们对争议进行聚类，计算争议之间的余弦相似度，并将结果用作分类任务的特征。实验结果表明，这种混合方法优于我们之前的系统，后者只考虑了争议聚类的信息。我们用 GPT-3.5 和 GPT-4 生成的逐项争议替换了法院准备的争议，并重复了相同的实验。使用 GPT-4 生成的争议会得到更好的结果。虽然我们的分类器在使用 ChatGPT 生成的争议时表现不佳，但结果令人满意。因此，我们希望未来的大型语言模型能够变得实用。

Title: Language Models "Grok" to Copy

• Authors: Ang Lv, Ruobing Xie, Xingwu Sun, Zhanhui Kang, Rui Yan
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.09281
• Pdf URL: https://arxiv.org/pdf/2409.09281
• Copy Paste: [[2409.09281]] Language Models "Grok" to Copy(https://arxiv.org/abs/2409.09281)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: We examine the pre-training dynamics of language models, focusing on their ability to copy text from preceding context--a fundamental skill for various LLM applications, including in-context learning (ICL) and retrieval-augmented generation (RAG). We propose a novel perspective that Transformer-based language models develop copying abilities similarly to grokking, which refers to sudden generalization on test set long after the model fit to the training set. Our experiments yield three arguments: (1) The pre-training loss decreases rapidly, while the context copying ability of models initially lags and then abruptly saturates. (2) The speed of developing copying ability is independent of the number of tokens trained, similarly to how grokking speed is unaffected by dataset size as long as the data distribution is preserved. (3) Induction heads, the attention heads responsible for copying, form from shallow to deep layers during training, mirroring the development of circuits in deeper layers during grokking. We contend that the connection between grokking and context copying can provide valuable insights for more effective language model training, ultimately improving in-context performance. For example, we demonstrated that techniques that enhance grokking, such as regularization, either accelerate or enhance the development of context copying.
• 摘要：我们研究了语言模型的预训练动态，重点关注它们从先前上下文复制文本的能力——这是各种 LLM 应用的基本技能，包括上下文学习 (ICL) 和检索增强生成 (RAG)。我们提出了一个新颖的观点，即基于 Transformer 的语言模型开发出类似于 grokking 的复制能力，这指的是模型拟合训练集很久之后在测试集上突然泛化。我们的实验得出了三个论点：（1）预训练损失迅速下降，而模型的上下文复制能力最初滞后，然后突然饱和。（2）开发复制能力的速度与训练的 token 数量无关，类似于只要数据分布保持不变，grokking 速度就不受数据集大小的影响。（3）感应头，即负责复制的注意力头，在训练过程中从浅层到深层形成，反映了 grokking 过程中更深层电路的发展。我们认为，理解和上下文复制之间的联系可以为更有效地训练语言模型提供宝贵的见解，最终提高上下文性能。例如，我们证明了增强理解的技术（例如正则化）可以加速或增强上下文复制的发展。

Title: ODE: Open-Set Evaluation of Hallucinations in Multimodal Large Language Models

• Authors: Yahan Tu, Rui Hu, Jitao Sang
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2409.09318
• Pdf URL: https://arxiv.org/pdf/2409.09318
• Copy Paste: [[2409.09318]] ODE: Open-Set Evaluation of Hallucinations in Multimodal Large Language Models(https://arxiv.org/abs/2409.09318)
• Keywords: language model, llm, hallucination
• Abstract: Hallucination poses a significant challenge for multimodal large language models (MLLMs). However, existing benchmarks for evaluating hallucinations are static, which can lead to potential data contamination. This paper introduces ODE, an open-set, dynamic protocol for evaluating object existence hallucinations in MLLMs. Our framework employs graph structures to model associations between real-word concepts and generates novel samples for both general and domain-specific scenarios. The dynamic combination of concepts, along with various combination principles, ensures a broad sample distribution. Experimental results show that MLLMs exhibit higher hallucination rates with ODE-generated samples, effectively avoiding data contamination. Moreover, these samples can also be used for fine-tuning to improve MLLM performance on existing benchmarks.
• 摘要：幻觉对多模态大型语言模型 (MLLM) 提出了重大挑战。然而，现有的评估幻觉的基准是静态的，这可能导致潜在的数据污染。本文介绍了 ODE，这是一种用于评估 MLLM 中对象存在幻觉的开放集动态协议。我们的框架采用图结构来模拟真实词概念之间的关联，并为一般和特定领域的场景生成新样本。概念的动态组合以及各种组合原则确保了广泛的样本分布。实验结果表明，MLLM 使用 ODE 生成的样本表现出更高的幻觉率，有效避免了数据污染。此外，这些样本还可用于微调，以提高 MLLM 在现有基准上的性能。

Title: Efficient Fine-Tuning of Large Language Models for Automated Medical Documentation

• Authors: Hui Yi Leong, Yi Fan Gao, Ji Shuai, Uktu Pamuksuz
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.09324
• Pdf URL: https://arxiv.org/pdf/2409.09324
• Copy Paste: [[2409.09324]] Efficient Fine-Tuning of Large Language Models for Automated Medical Documentation(https://arxiv.org/abs/2409.09324)
• Keywords: language model, llm
• Abstract: Scientific research indicates that for every hour spent in direct patient care, physicians spend nearly two additional hours on administrative tasks, particularly on electronic health records (EHRs) and desk work. This excessive administrative burden not only reduces the time available for patient care but also contributes to physician burnout and inefficiencies in healthcare delivery. To address these challenges, this study introduces MediGen, a fine-tuned large language model (LLM) designed to automate the generation of medical reports from medical dialogues. By leveraging state-of-the-art methodologies for fine-tuning open-source pretrained models, including LLaMA3-8B, MediGen achieves high accuracy in transcribing and summarizing clinical interactions. The fine-tuned LLaMA3-8B model demonstrated promising results, achieving a ROUGE score of 58% and a BERTScore-F1 of 72%, indicating its effectiveness in generating accurate and clinically relevant medical reports. These findings suggest that MediGen has the potential to significantly reduce the administrative workload on physicians, improving both healthcare efficiency and physician well-being.
• 摘要：科学研究表明，医生每花一个小时直接护理患者，就要花近两个小时处理行政事务，尤其是电子健康记录 (EHR) 和案头工作。这种过度的行政负担不仅减少了可用于护理患者的时间，还导致医生倦怠和医疗服务效率低下。为了应对这些挑战，本研究引入了 MediGen，这是一种经过微调的大型语言模型 (LLM)，旨在自动从医疗对话中生成医疗报告。通过利用最先进的方法对开源预训练模型（包括 LLaMA3-8B）进行微调，MediGen 在转录和总结临床互动方面实现了高精度。经过微调的 LLaMA3-8B 模型表现出了良好的效果，ROUGE 得分达到 58%，BERTScore-F1 达到 72%，表明其在生成准确且具有临床相关性的医疗报告方面非常有效。这些发现表明，MediGen 有可能显著减少医生的行政工作量，提高医疗效率和医生的福祉。

Title: Generating Event-oriented Attribution for Movies via Two-Stage Prefix-Enhanced Multimodal LLM

• Authors: Yuanjie Lyu, Tong Xu, Zihan Niu, Bo Peng, Jing Ke, Enhong Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.09362
• Pdf URL: https://arxiv.org/pdf/2409.09362
• Copy Paste: [[2409.09362]] Generating Event-oriented Attribution for Movies via Two-Stage Prefix-Enhanced Multimodal LLM(https://arxiv.org/abs/2409.09362)
• Keywords: language model, llm
• Abstract: The prosperity of social media platforms has raised the urgent demand for semantic-rich services, e.g., event and storyline attribution. However, most existing research focuses on clip-level event understanding, primarily through basic captioning tasks, without analyzing the causes of events across an entire movie. This is a significant challenge, as even advanced multimodal large language models (MLLMs) struggle with extensive multimodal information due to limited context length. To address this issue, we propose a Two-Stage Prefix-Enhanced MLLM (TSPE) approach for event attribution, i.e., connecting associated events with their causal semantics, in movie videos. In the local stage, we introduce an interaction-aware prefix that guides the model to focus on the relevant multimodal information within a single clip, briefly summarizing the single event. Correspondingly, in the global stage, we strengthen the connections between associated events using an inferential knowledge graph, and design an event-aware prefix that directs the model to focus on associated events rather than all preceding clips, resulting in accurate event attribution. Comprehensive evaluations of two real-world datasets demonstrate that our framework outperforms state-of-the-art methods.
• 摘要：社交媒体平台的繁荣带来了对语义丰富服务（例如事件和故事情节归因）的迫切需求。然而，现有的大多数研究主要通过基本的字幕任务来关注片段级事件理解，而没有分析整部电影中事件的原因。这是一个重大挑战，因为即使是先进的多模态大型语言模型 (MLLM) 也会因为上下文长度有限而难以处理大量多模态信息。为了解决这个问题，我们提出了一种两阶段前缀增强 MLLM (TSPE) 方法用于电影视频中的事件归因，即将相关事件与其因果语义联系起来。在局部阶段，我们引入了一个交互感知前缀，引导模型关注单个片段中的相关多模态信息，简要总结单个事件。相应地，在全局阶段，我们使用推理知识图加强相关事件之间的联系，并设计一个事件感知前缀，引导模型关注相关事件而不是所有前面的片段，从而实现准确的事件归因。对两个真实数据集的综合评估表明，我们的框架优于最先进的方法。

Title: Constructive Approach to Bidirectional Causation between Qualia Structure and Language Emergence

• Authors: Tadahiro Taniguchi, Masafumi Oizumi, Noburo Saji, Takato Horii, Naotsugu Tsuchiya
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.09413
• Pdf URL: https://arxiv.org/pdf/2409.09413
• Copy Paste: [[2409.09413]] Constructive Approach to Bidirectional Causation between Qualia Structure and Language Emergence(https://arxiv.org/abs/2409.09413)
• Keywords: language model
• Abstract: This paper presents a novel perspective on the bidirectional causation between language emergence and relational structure of subjective experiences, termed qualia structure, and lays out the constructive approach to the intricate dependency between the two. We hypothesize that languages with distributional semantics, e.g., syntactic-semantic structures, may have emerged through the process of aligning internal representations among individuals, and such alignment of internal representations facilitates more structured language. This mutual dependency is suggested by the recent advancements in AI and symbol emergence robotics, and collective predictive coding (CPC) hypothesis, in particular. Computational studies show that neural network-based language models form systematically structured internal representations, and multimodal language models can share representations between language and perceptual information. This perspective suggests that language emergence serves not only as a mechanism creating a communication tool but also as a mechanism for allowing people to realize shared understanding of qualitative experiences. The paper discusses the implications of this bidirectional causation in the context of consciousness studies, linguistics, and cognitive science, and outlines future constructive research directions to further explore this dynamic relationship between language emergence and qualia structure.
• 摘要：本文提出了一种新颖的视角，探讨语言出现与主观体验的关系结构（称为感质结构）之间的双向因果关系，并阐述了解决两者之间复杂依赖关系的建设性方法。我们假设，具有分布式语义（例如句法语义结构）的语言可能是通过协调个体之间的内部表征而出现的，而这种内部表征的协调有助于语言更加结构化。人工智能和符号出现机器人技术的最新进展，尤其是集体预测编码 (CPC) 假设，表明了这种相互依赖关系。计算研究表明，基于神经网络的语言模型形成系统结构化的内部表征，多模态语言模型可以在语言和感知信息之间共享表征。这种观点表明，语言出现不仅是一种创造交流工具的机制，也是一种让人们实现对定性体验的共同理解的机制。本文讨论了这种双向因果关系在意识研究、语言学和认知科学背景下的含义，并概述了未来建设性的研究方向，以进一步探索语言出现和感质结构之间的动态关系。

Title: Enhancing LLM Problem Solving with REAP: Reflection, Explicit Problem Deconstruction, and Advanced Prompting

• Authors: Ryan Lingo, Martin Arroyo, Rajeev Chhajer
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.09415
• Pdf URL: https://arxiv.org/pdf/2409.09415
• Copy Paste: [[2409.09415]] Enhancing LLM Problem Solving with REAP: Reflection, Explicit Problem Deconstruction, and Advanced Prompting(https://arxiv.org/abs/2409.09415)
• Keywords: language model, gpt, llm, prompt
• Abstract: Large Language Models (LLMs) have transformed natural language processing, yet improving their problem-solving capabilities, particularly for complex, reasoning-intensive tasks, remains a persistent challenge. This paper introduces the REAP (Reflection, Explicit Problem Deconstruction, and Advanced Prompting) method, an innovative approach within the dynamic context generation framework. REAP guides LLMs through reflection on the query, deconstructing it into manageable components, and generating relevant context to enhance the solution process. We evaluated REAP using a dataset designed to expose LLM limitations, comparing zero-shot prompting with REAP-enhanced prompts across six state-of-the-art models: OpenAI's o1-preview, o1-mini, GPT-4o, GPT-4o-mini, Google's Gemini 1.5 Pro, and Claude 3.5 Sonnet. The results demonstrate notable performance gains, with o1-mini improving by 40.97%, GPT-4o by 66.26%, and GPT-4o-mini by 112.93%. Despite the already strong baseline performance of OpenAI's o1-preview, modest gains were observed. Beyond performance improvements, REAP offers a cost-effective solution; for example, GPT-4o-mini, which is approximately 100 times cheaper than o1-preview, delivered competitive results. REAP also improves the clarity of model outputs, making it easier for humans to understand the reasoning behind the results and simplifying the process of identifying and addressing any issues. These findings demonstrate REAP's potential to greatly improve the capabilities of LLMs, providing both better performance and increased cost-efficiency across a wide range of applications.
• 摘要：大型语言模型 (LLM) 已经改变了自然语言处理，但提高其解决问题的能力，特别是对于复杂、推理密集型任务的能力，仍然是一个持续的挑战。本文介绍了 REAP（反思、显式问题解构和高级提示）方法，这是动态上下文生成框架中的一种创新方法。REAP 引导 LLM 通过反思查询，将其解构为可管理的组件，并生成相关上下文以增强解决过程。我们使用旨在揭示 LLM 局限性的数据集对 REAP 进行了评估，在六个最先进的模型中比较了零样本提示和 REAP 增强提示：OpenAI 的 o1-preview、o1-mini、GPT-4o、GPT-4o-mini、Google 的 Gemini 1.5 Pro 和 Claude 3.5 Sonnet。结果显示，性能显著提升，o1-mini 提升了 40.97%，GPT-4o 提升了 66.26%，GPT-4o-mini 提升了 112.93%。尽管 OpenAI 的 o1-preview 的基准性能已经很强大，但仍观察到适度的提升。除了性能改进之外，REAP 还提供了一种经济高效的解决方案；例如，GPT-4o-mini 比 o1-preview 便宜约 100 倍，但提供了具有竞争力的结果。REAP 还提高了模型输出的清晰度，使人们更容易理解结果背后的原因，并简化了识别和解决问题的过程。这些发现表明，REAP 有潜力极大地提高 LLM 的功能，在广泛的应用中提供更好的性能和更高的成本效率。

Title: Keeping Humans in the Loop: Human-Centered Automated Annotation with Generative AI

• Authors: Nicholas Pangakis, Samuel Wolken
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.09467
• Pdf URL: https://arxiv.org/pdf/2409.09467
• Copy Paste: [[2409.09467]] Keeping Humans in the Loop: Human-Centered Automated Annotation with Generative AI(https://arxiv.org/abs/2409.09467)
• Keywords: language model, gpt, llm, prompt
• Abstract: Automated text annotation is a compelling use case for generative large language models (LLMs) in social media research. Recent work suggests that LLMs can achieve strong performance on annotation tasks; however, these studies evaluate LLMs on a small number of tasks and likely suffer from contamination due to a reliance on public benchmark datasets. Here, we test a human-centered framework for responsibly evaluating artificial intelligence tools used in automated annotation. We use GPT-4 to replicate 27 annotation tasks across 11 password-protected datasets from recently published computational social science articles in high-impact journals. For each task, we compare GPT-4 annotations against human-annotated ground-truth labels and against annotations from separate supervised classification models fine-tuned on human-generated labels. Although the quality of LLM labels is generally high, we find significant variation in LLM performance across tasks, even within datasets. Our findings underscore the importance of a human-centered workflow and careful evaluation standards: Automated annotations significantly diverge from human judgment in numerous scenarios, despite various optimization strategies such as prompt tuning. Grounding automated annotation in validation labels generated by humans is essential for responsible evaluation.
• 摘要：自动文本注释是社交媒体研究中生成式大型语言模型 (LLM) 的一个引人注目的用例。最近的研究表明，LLM 可以在注释任务上取得出色的表现；然而，这些研究只在少数任务上评估了 LLM，而且由于依赖公共基准数据集，可能会受到污染。在这里，我们测试了一个以人为本的框架，以负责任地评估自动注释中使用的人工智能工具。我们使用 GPT-4 在 11 个受密码保护的数据集上复制了 27 个注释任务，这些数据集来自最近发表在高影响力期刊上的计算社会科学文章。对于每个任务，我们将 GPT-4 注释与人工注释的真实标签进行比较，并与在人工生成的标签上微调的单独监督分类模型的注释进行比较。虽然 LLM 标签的质量通常很高，但我们发现 LLM 在不同任务之间的性能存在显著差异，即使在数据集内也是如此。我们的研究结果强调了以人为本的工作流程和谨慎的评估标准的重要性：尽管采用了各种优化策略（例如快速调整），但自动注释在许多情况下仍与人类判断存在很大差异。将自动注释建立在人类生成的验证标签上对于负责任的评估至关重要。

Title: Synthetic4Health: Generating Annotated Synthetic Clinical Letters

• Authors: Libo Ren, Samuel Belkadi, Lifeng Han, Warren Del-Pinto, Goran Nenadic
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.09501
• Pdf URL: https://arxiv.org/pdf/2409.09501
• Copy Paste: [[2409.09501]] Synthetic4Health: Generating Annotated Synthetic Clinical Letters(https://arxiv.org/abs/2409.09501)
• Keywords: language model
• Abstract: Since clinical letters contain sensitive information, clinical-related datasets can not be widely applied in model training, medical research, and teaching. This work aims to generate reliable, various, and de-identified synthetic clinical letters. To achieve this goal, we explored different pre-trained language models (PLMs) for masking and generating text. After that, we worked on Bio\_ClinicalBERT, a high-performing model, and experimented with different masking strategies. Both qualitative and quantitative methods were used for evaluation. Additionally, a downstream task, Named Entity Recognition (NER), was also implemented to assess the usability of these synthetic letters. The results indicate that 1) encoder-only models outperform encoder-decoder models. 2) Among encoder-only models, those trained on general corpora perform comparably to those trained on clinical data when clinical information is preserved. 3) Additionally, preserving clinical entities and document structure better aligns with our objectives than simply fine-tuning the model. 4) Furthermore, different masking strategies can impact the quality of synthetic clinical letters. Masking stopwords has a positive impact, while masking nouns or verbs has a negative effect. 5) For evaluation, BERTScore should be the primary quantitative evaluation metric, with other metrics serving as supplementary references. 6) Contextual information does not significantly impact the models' understanding, so the synthetic clinical letters have the potential to replace the original ones in downstream tasks.
• 摘要：由于临床信件包含敏感信息，临床相关数据集无法广泛应用于模型训练、医学研究和教学。这项工作旨在生成可靠、多样且去识别的合成临床信件。为了实现这一目标，我们探索了不同的预训练语言模型 (PLM) 来屏蔽和生成文本。之后，我们研究了高性能模型 Bio\_ClinicalBERT，并尝试了不同的屏蔽策略。定性和定量方法都用于评估。此外，还实施了下游任务命名实体识别 (NER)，以评估这些合成信件的可用性。结果表明：1) 仅编码器模型优于编码器-解码器模型。2) 在仅编码器模型中，在保留临床信息的情况下，在一般语料库上训练的模型与在临床数据上训练的模型表现相当。3) 此外，与简单地微调模型相比，保留临床实体和文档结构更符合我们的目标。4) 此外，不同的屏蔽策略会影响合成临床信件的质量。屏蔽停用词有积极影响，而屏蔽名词或动词则有负面影响。5）对于评估，BERTScore 应作为主要的定量评估指标，其他指标作为补充参考。6）上下文信息对模型的理解没有显著影响，因此合成临床字母有可能在下游任务中取代原始字母。

Title: Comparing Retrieval-Augmentation and Parameter-Efficient Fine-Tuning for Privacy-Preserving Personalization of Large Language Models

• Authors: Alireza Salemi, Hamed Zamani
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.09510
• Pdf URL: https://arxiv.org/pdf/2409.09510
• Copy Paste: [[2409.09510]] Comparing Retrieval-Augmentation and Parameter-Efficient Fine-Tuning for Privacy-Preserving Personalization of Large Language Models(https://arxiv.org/abs/2409.09510)
• Keywords: language model, llm, prompt
• Abstract: Privacy-preserving methods for personalizing large language models (LLMs) are relatively under-explored. There are two schools of thought on this topic: (1) generating personalized outputs by personalizing the input prompt through retrieval augmentation from the user's personal information (RAG-based methods), and (2) parameter-efficient fine-tuning of LLMs per user that considers efficiency and space limitations (PEFT-based methods). This paper presents the first systematic comparison between two approaches on a wide range of personalization tasks using seven diverse datasets. Our results indicate that RAG-based and PEFT-based personalization methods on average yield 14.92% and 1.07% improvements over the non-personalized LLM, respectively. We find that combining RAG with PEFT elevates these improvements to 15.98%. Additionally, we identify a positive correlation between the amount of user data and PEFT's effectiveness, indicating that RAG is a better choice for cold-start users (i.e., user's with limited personal data).
• 摘要：用于个性化大型语言模型 (LLM) 的隐私保护方法相对较少。关于这个主题有两种思想流派：(1) 通过从用户的个人信息中进行检索增强来个性化输入提示，从而生成个性化输出 (基于 RAG 的方法)；(2) 考虑效率和空间限制的每个用户的 LLM 参数高效微调 (基于 PEFT 的方法)。本文首次使用七个不同的数据集对两种方法在广泛的个性化任务上进行了系统比较。我们的结果表明，基于 RAG 和基于 PEFT 的个性化方法平均比非个性化 LLM 分别提高了 14.92% 和 1.07%。我们发现将 RAG 与 PEFT 结合起来可以将这些改进提高到 15.98%。此外，我们发现用户数据量与 PEFT 的有效性之间存在正相关性，这表明 RAG 是冷启动用户（即个人数据有限的用户）的更好选择。

Title: ASR Error Correction using Large Language Models

• Authors: Rao Ma, Mengjie Qian, Mark Gales, Kate Knill
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2409.09554
• Pdf URL: https://arxiv.org/pdf/2409.09554
• Copy Paste: [[2409.09554]] ASR Error Correction using Large Language Models(https://arxiv.org/abs/2409.09554)
• Keywords: language model, gpt, llm, chat
• Abstract: Error correction (EC) models play a crucial role in refining Automatic Speech Recognition (ASR) transcriptions, enhancing the readability and quality of transcriptions. Without requiring access to the underlying code or model weights, EC can improve performance and provide domain adaptation for black-box ASR systems. This work investigates the use of large language models (LLMs) for error correction across diverse scenarios. 1-best ASR hypotheses are commonly used as the input to EC models. We propose building high-performance EC models using ASR N-best lists which should provide more contextual information for the correction process. Additionally, the generation process of a standard EC model is unrestricted in the sense that any output sequence can be generated. For some scenarios, such as unseen domains, this flexibility may impact performance. To address this, we introduce a constrained decoding approach based on the N-best list or an ASR lattice. Finally, most EC models are trained for a specific ASR system requiring retraining whenever the underlying ASR system is changed. This paper explores the ability of EC models to operate on the output of different ASR systems. This concept is further extended to zero-shot error correction using LLMs, such as ChatGPT. Experiments on three standard datasets demonstrate the efficacy of our proposed methods for both Transducer and attention-based encoder-decoder ASR systems. In addition, the proposed method can serve as an effective method for model ensembling.
• 摘要：纠错 (EC) 模型在改进自动语音识别 (ASR) 转录、提高转录的可读性和质量方面起着至关重要的作用。无需访问底层代码或模型权重，EC 就可以提高性能并为黑盒 ASR 系统提供域自适应。这项工作研究了大型语言模型 (LLM) 在不同场景中用于纠错的情况。1-best ASR 假设通常用作 EC 模型的输入。我们建议使用 ASR N-best 列表构建高性能 EC 模型，这应该为校正过程提供更多上下文信息。此外，标准 EC 模型的生成过程不受限制，因为可以生成任何输出序列。对于某些场景，例如看不见的域，这种灵活性可能会影响性能。为了解决这个问题，我们引入了一种基于 N-best 列表或 ASR 格子的受限解码方法。最后，大多数 EC 模型都是针对特定 ASR 系统进行训练的，每当底层 ASR 系统发生变化时都需要重新训练。本文探讨了 EC 模型对不同 ASR 系统的输出进行操作的能力。此概念进一步扩展到使用 LLM（例如 ChatGPT）进行零样本纠错。在三个标准数据集上进行的实验证明了我们提出的方法对于 Transducer 和基于注意机制的编码器-解码器 ASR 系统的有效性。此外，所提出的方法可以作为一种有效的模型集成方法。

Title: Thesis proposal: Are We Losing Textual Diversity to Natural Language Processing?

• Authors: Josef Jon
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.09568
• Pdf URL: https://arxiv.org/pdf/2409.09568
• Copy Paste: [[2409.09568]] Thesis proposal: Are We Losing Textual Diversity to Natural Language Processing?(https://arxiv.org/abs/2409.09568)
• Keywords: language model, llm
• Abstract: This thesis argues that the currently widely used Natural Language Processing algorithms possibly have various limitations related to the properties of the texts they handle and produce. With the wide adoption of these tools in rapid progress, we must ask what these limitations are and what are the possible implications of integrating such tools even more deeply into our daily lives. As a testbed, we have chosen the task of Neural Machine Translation (NMT). Nevertheless, we aim for general insights and outcomes, applicable even to current Large Language Models (LLMs). We ask whether the algorithms used in NMT have inherent inductive biases that are beneficial for most types of inputs but might harm the processing of untypical texts. To explore this hypothesis, we define a set of measures to quantify text diversity based on its statistical properties, like uniformity or rhythmicity of word-level surprisal, on multiple scales (sentence, discourse, language). We then conduct a series of experiments to investigate whether NMT systems struggle with maintaining the diversity of such texts, potentially reducing the richness of the language generated by these systems, compared to human translators. We search for potential causes of these limitations rooted in training objectives and decoding algorithms. Our ultimate goal is to develop alternatives that do not enforce uniformity in the distribution of statistical properties in the output and that allow for better global planning of the translation, taking into account the intrinsic ambiguity of the translation task.
• 摘要：本论文认为，目前广泛使用的自然语言处理算法可能存在与其处理和生成的文本属性相关的各种限制。随着这些工具的快速广泛采用，我们必须问这些限制是什么，以及将这些工具更深入地融入我们的日常生活可能产生什么影响。作为试验台，我们选择了神经机器翻译 (NMT) 任务。尽管如此，我们的目标是获得普遍的见解和结果，甚至适用于当前的大型语言模型 (LLM)。我们想知道 NMT 中使用的算法是否具有固有的归纳偏差，这些偏差对大多数类型的输入有益，但可能会损害非典型文本的处理。为了探索这一假设，我们定义了一组度量来量化文本多样性，这些度量基于文本多样性的统计属性，例如单词级意外的一致性或节奏性，在多个尺度上（句子、话语、语言）。然后，我们进行了一系列实验，以调查 NMT 系统是否难以保持此类文本的多样性，与人工翻译相比，这可能会降低这些系统生成的语言的丰富性。我们寻找这些限制的潜在原因，这些原因根植于训练目标和解码算法。我们的最终目标是开发替代方案，不强制输出中统计属性的分布均匀性，并允许更好地全局规划翻译，同时考虑到翻译任务的内在歧义性。

Title: Rethinking KenLM: Good and Bad Model Ensembles for Efficient Text Quality Filtering in Large Web Corpora

• Authors: Yungi Kim, Hyunsoo Ha, Sukyung Lee, Jihoo Kim, Seonghoon Yang, Chanjun Park
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.09613
• Pdf URL: https://arxiv.org/pdf/2409.09613
• Copy Paste: [[2409.09613]] Rethinking KenLM: Good and Bad Model Ensembles for Efficient Text Quality Filtering in Large Web Corpora(https://arxiv.org/abs/2409.09613)
• Keywords: language model, llm
• Abstract: With the increasing demand for substantial amounts of high-quality data to train large language models (LLMs), efficiently filtering large web corpora has become a critical challenge. For this purpose, KenLM, a lightweight n-gram-based language model that operates on CPUs, is widely used. However, the traditional method of training KenLM utilizes only high-quality data and, consequently, does not explicitly learn the linguistic patterns of low-quality data. To address this issue, we propose an ensemble approach that leverages two contrasting KenLMs: (i) Good KenLM, trained on high-quality data; and (ii) Bad KenLM, trained on low-quality data. Experimental results demonstrate that our approach significantly reduces noisy content while preserving high-quality content compared to the traditional KenLM training method. This indicates that our method can be a practical solution with minimal computational overhead for resource-constrained environments.
• 摘要：随着对大量高质量数据的需求不断增长，用于训练大型语言模型 (LLM)，有效过滤大型网络语料库已成为一项关键挑战。为此，KenLM 被广泛使用，这是一种基于 n​​-gram 的轻量级语言模型，可在 CPU 上运行。但是，传统的 KenLM 训练方法仅使用高质量数据，因此不会明确学习低质量数据的语言模式。为了解决这个问题，我们提出了一种集成方法，利用两个对比鲜明的 KenLM：(i) 好的 KenLM，在高质量数据上训练；(ii) 坏的 KenLM，在低质量数据上训练。实验结果表明，与传统的 KenLM 训练方法相比，我们的方法在保留高质量内容的同时显著减少了噪声内容。这表明我们的方法可以成为资源受限环境中计算开销最小的实用解决方案。

Title: Enhancing Text Annotation through Rationale-Driven Collaborative Few-Shot Prompting

• Authors: Jianfei Wu, Xubin Wang, Weijia Jia
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.09615
• Pdf URL: https://arxiv.org/pdf/2409.09615
• Copy Paste: [[2409.09615]] Enhancing Text Annotation through Rationale-Driven Collaborative Few-Shot Prompting(https://arxiv.org/abs/2409.09615)
• Keywords: language model, llm, prompt
• Abstract: The traditional data annotation process is often labor-intensive, time-consuming, and susceptible to human bias, which complicates the management of increasingly complex datasets. This study explores the potential of large language models (LLMs) as automated data annotators to improve efficiency and consistency in annotation tasks. By employing rationale-driven collaborative few-shot prompting techniques, we aim to improve the performance of LLMs in text annotation. We conduct a rigorous evaluation of six LLMs across four benchmark datasets, comparing seven distinct methodologies. Our results demonstrate that collaborative methods consistently outperform traditional few-shot techniques and other baseline approaches, particularly in complex annotation tasks. Our work provides valuable insights and a robust framework for leveraging collaborative learning methods to tackle challenging text annotation tasks.
• 摘要：传统的数据注释过程通常劳动密集、耗时且易受人为偏见的影响，这使日益复杂的数据集的管理变得复杂。本研究探索了大型语言模型 (LLM) 作为自动数据注释器的潜力，以提高注释任务的效率和一致性。通过采用理性驱动的协作式小样本提示技术，我们旨在提高 LLM 在文本注释中的性能。我们对四个基准数据集中的六个 LLM 进行了严格的评估，比较了七种不同的方法。我们的结果表明，协作方法始终优于传统的小样本技术和其他基线方法，尤其是在复杂的注释任务中。我们的工作提供了宝贵的见解和强大的框架，可用于利用协作学习方法来解决具有挑战性的文本注释任务。

Title: Confidence Estimation for LLM-Based Dialogue State Tracking

• Authors: Yi-Jyun Sun, Suvodip Dey, Dilek Hakkani-Tur, Gokhan Tur
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.09629
• Pdf URL: https://arxiv.org/pdf/2409.09629
• Copy Paste: [[2409.09629]] Confidence Estimation for LLM-Based Dialogue State Tracking(https://arxiv.org/abs/2409.09629)
• Keywords: language model, llm, hallucination
• Abstract: Estimation of a model's confidence on its outputs is critical for Conversational AI systems based on large language models (LLMs), especially for reducing hallucination and preventing over-reliance. In this work, we provide an exhaustive exploration of methods, including approaches proposed for open- and closed-weight LLMs, aimed at quantifying and leveraging model uncertainty to improve the reliability of LLM-generated responses, specifically focusing on dialogue state tracking (DST) in task-oriented dialogue systems (TODS). Regardless of the model type, well-calibrated confidence scores are essential to handle uncertainties, thereby improving model performance. We evaluate four methods for estimating confidence scores based on softmax, raw token scores, verbalized confidences, and a combination of these methods, using the area under the curve (AUC) metric to assess calibration, with higher AUC indicating better calibration. We also enhance these with a self-probing mechanism, proposed for closed models. Furthermore, we assess these methods using an open-weight model fine-tuned for the task of DST, achieving superior joint goal accuracy (JGA). Our findings also suggest that fine-tuning open-weight LLMs can result in enhanced AUC performance, indicating better confidence score calibration.
• 摘要：对于基于大型语言模型 (LLM) 的对话式 AI 系统来说，评估模型对其输出的置信度至关重要，尤其是为了减少幻觉和防止过度依赖。在这项工作中，我们详尽探索了各种方法，包括为开放权重和封闭权重 LLM 提出的方法，旨在量化和利用模型不确定性来提高 LLM 生成的响应的可靠性，特别关注任务导向型对话系统 (TODS) 中的对话状态跟踪 (DST)。无论模型类型如何，经过良好校准的置信度分数对于处理不确定性至关重要，从而提高模型性能。我们评估了四种基于 softmax、原始标记分数、口头化置信度和这些方法的组合估计置信度分数的方法，使用曲线下面积 (AUC) 指标来评估校准，AUC 越高表示校准越好。我们还通过为封闭模型提出的自探测机制来增强这些功能。此外，我们使用针对 DST 任务进行微调的开放权重模型来评估这些方法，实现了卓越的联合目标准确率 (JGA)。我们的研究结果还表明，微调开放权重 LLM 可以提高 AUC 性能，表明置信度分数校准效果更佳。

Title: Towards understanding evolution of science through language model series

• Authors: Junjie Dong, Zhuoqi Lyu, Qing Ke
• Subjects: cs.CL, cs.CY, cs.DL
• Abstract URL: https://arxiv.org/abs/2409.09636
• Pdf URL: https://arxiv.org/pdf/2409.09636
• Copy Paste: [[2409.09636]] Towards understanding evolution of science through language model series(https://arxiv.org/abs/2409.09636)
• Keywords: language model
• Abstract: We introduce AnnualBERT, a series of language models designed specifically to capture the temporal evolution of scientific text. Deviating from the prevailing paradigms of subword tokenizations and "one model to rule them all", AnnualBERT adopts whole words as tokens and is composed of a base RoBERTa model pretrained from scratch on the full-text of 1.7 million arXiv papers published until 2008 and a collection of progressively trained models on arXiv papers at an annual basis. We demonstrate the effectiveness of AnnualBERT models by showing that they not only have comparable performances in standard tasks but also achieve state-of-the-art performances on domain-specific NLP tasks as well as link prediction tasks in the arXiv citation network. We then utilize probing tasks to quantify the models' behavior in terms of representation learning and forgetting as time progresses. Our approach enables the pretrained models to not only improve performances on scientific text processing tasks but also to provide insights into the development of scientific discourse over time. The series of the models is available at this https URL.
• 摘要：我们推出了 AnnualBERT，这是一系列专门用于捕捉科学文本时间演变的语言模型。AnnualBERT 偏离了流行的子词标记化和“一个模型统治一切”的范式，它采用整个单词作为标记，由一个基本 RoBERTa 模型组成，该模型从头开始对 2008 年之前发表的 170 万篇 arXiv 论文的全文进行预训练，并每年对 arXiv 论文进行逐步训练。我们通过展示 AnnualBERT 模型不仅在标准任务中具有可比的性能，而且在特定领域的 NLP 任务以及 arXiv 引文网络中的链接预测任务上也实现了最先进的性能，证明了其有效性。然后，我们利用探测任务来量化模型在表征学习和遗忘方面的行为。我们的方法使预训练模型不仅可以提高科学文本处理任务的性能，还可以提供对科学话语随时间发展的洞察。该系列模型可在此 https URL 上找到。

Title: Unveiling Gender Bias in Large Language Models: Using Teacher's Evaluation in Higher Education As an Example

• Authors: Yuanning Huang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.09652
• Pdf URL: https://arxiv.org/pdf/2409.09652
• Copy Paste: [[2409.09652]] Unveiling Gender Bias in Large Language Models: Using Teacher's Evaluation in Higher Education As an Example(https://arxiv.org/abs/2409.09652)
• Keywords: language model, gpt, llm, agent
• Abstract: This paper investigates gender bias in Large Language Model (LLM)-generated teacher evaluations in higher education setting, focusing on evaluations produced by GPT-4 across six academic subjects. By applying a comprehensive analytical framework that includes Odds Ratio (OR) analysis, Word Embedding Association Test (WEAT), sentiment analysis, and contextual analysis, this paper identified patterns of gender-associated language reflecting societal stereotypes. Specifically, words related to approachability and support were used more frequently for female instructors, while words related to entertainment were predominantly used for male instructors, aligning with the concepts of communal and agentic behaviors. The study also found moderate to strong associations between male salient adjectives and male names, though career and family words did not distinctly capture gender biases. These findings align with prior research on societal norms and stereotypes, reinforcing the notion that LLM-generated text reflects existing biases.
• 摘要：本文研究了高等教育环境中大型语言模型 (LLM) 生成的教师评估中的性别偏见，重点关注 GPT-4 在六个学科中生成的评估。通过应用包括比值比 (OR) 分析、词嵌入关联测试 (WEAT)、情绪分析和上下文分析在内的综合分析框架，本文确定了反映社会刻板印象的性别相关语言模式。具体来说，与亲近力和支持相关的词语更常用于女性教师，而与娱乐相关的词语主要用于男性教师，这与社区和代理行为的概念相一致。研究还发现男性显着形容词与男性名字之间存在中等到强烈的关联，尽管职业和家庭词语并没有明显地反映出性别偏见。这些发现与先前对社会规范和刻板印象的研究一致，强化了 LLM 生成的文本反映现有偏见的观点。

Title: Leveraging Open-Source Large Language Models for Native Language Identification

• Authors: Yee Man Ng, Ilia Markov
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.09659
• Pdf URL: https://arxiv.org/pdf/2409.09659
• Copy Paste: [[2409.09659]] Leveraging Open-Source Large Language Models for Native Language Identification(https://arxiv.org/abs/2409.09659)
• Keywords: language model, gpt, llm
• Abstract: Native Language Identification (NLI) - the task of identifying the native language (L1) of a person based on their writing in the second language (L2) - has applications in forensics, marketing, and second language acquisition. Historically, conventional machine learning approaches that heavily rely on extensive feature engineering have outperformed transformer-based language models on this task. Recently, closed-source generative large language models (LLMs), e.g., GPT-4, have demonstrated remarkable performance on NLI in a zero-shot setting, including promising results in open-set classification. However, closed-source LLMs have many disadvantages, such as high costs and undisclosed nature of training data. This study explores the potential of using open-source LLMs for NLI. Our results indicate that open-source LLMs do not reach the accuracy levels of closed-source LLMs when used out-of-the-box. However, when fine-tuned on labeled training data, open-source LLMs can achieve performance comparable to that of commercial LLMs.
• 摘要：母语识别 (NLI) 是一项根据某人的第二语言 (L2) 写作来识别其母语 (L1) 的任务，可用于取证、营销和第二语言习得。从历史上看，严重依赖大量特征工程的传统机器学习方法在此任务上的表现优于基于 Transformer 的语言模型。最近，闭源生成式大型语言模型 (LLM)（例如 GPT-4）在零样本设置下在 NLI 上表现出色，包括在开放集分类中取得了令人鼓舞的结果。然而，闭源 LLM 有许多缺点，例如成本高和训练数据的未公开性质。本研究探讨了将开源 LLM 用于 NLI 的潜力。我们的结果表明，开箱即用的开源 LLM 无法达到闭源 LLM 的准确度水平。但是，当对标记的训练数据进行微调时，开源 LLM 可以实现与商业 LLM 相当的性能。

Title: AlpaPICO: Extraction of PICO Frames from Clinical Trial Documents Using LLMs

• Authors: Madhusudan Ghosh, Shrimon Mukherjee, Asmit Ganguly, Partha Basuchowdhuri, Sudip Kumar Naskar, Debasis Ganguly
• Subjects: cs.CL, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.09704
• Pdf URL: https://arxiv.org/pdf/2409.09704
• Copy Paste: [[2409.09704]] AlpaPICO: Extraction of PICO Frames from Clinical Trial Documents Using LLMs(https://arxiv.org/abs/2409.09704)
• Keywords: language model, llm
• Abstract: In recent years, there has been a surge in the publication of clinical trial reports, making it challenging to conduct systematic reviews. Automatically extracting Population, Intervention, Comparator, and Outcome (PICO) from clinical trial studies can alleviate the traditionally time-consuming process of manually scrutinizing systematic reviews. Existing approaches of PICO frame extraction involves supervised approach that relies on the existence of manually annotated data points in the form of BIO label tagging. Recent approaches, such as In-Context Learning (ICL), which has been shown to be effective for a number of downstream NLP tasks, require the use of labeled examples. In this work, we adopt ICL strategy by employing the pretrained knowledge of Large Language Models (LLMs), gathered during the pretraining phase of an LLM, to automatically extract the PICO-related terminologies from clinical trial documents in unsupervised set up to bypass the availability of large number of annotated data instances. Additionally, to showcase the highest effectiveness of LLM in oracle scenario where large number of annotated samples are available, we adopt the instruction tuning strategy by employing Low Rank Adaptation (LORA) to conduct the training of gigantic model in low resource environment for the PICO frame extraction task. Our empirical results show that our proposed ICL-based framework produces comparable results on all the version of EBM-NLP datasets and the proposed instruction tuned version of our framework produces state-of-the-art results on all the different EBM-NLP datasets. Our project is available at \url{this https URL}.
• 摘要：近年来，临床试验报告的发表数量激增，这使得进行系统评价变得具有挑战性。从临床试验研究自动提取人群、干预、对照和结果 (PICO) 可以减轻传统上耗时的手动审查系统评价的过程。现有的 PICO 框架提取方法涉及监督方法，该方法依赖于以 BIO 标签标记形式手动注释的数据点的存在。最近的方法，例如已被证明对许多下游 NLP 任务有效的上下文学习 (ICL)，需要使用标记示例。在这项工作中，我们采用 ICL 策略，利用在 LLM 预训练阶段收集的大型语言模型 (LLM) 的预训练知识，在无监督设置下自动从临床试验文件中提取与 PICO 相关的术语，以绕过大量带注释的数据实例的可用性。此外，为了展示 LLM 在具有大量注释样本的 Oracle 场景中的最高效率，我们采用了指令调整策略，即采用低秩自适应 (LORA) 在低资源环境中对 PICO 帧提取任务的巨型模型进行训练。我们的实证结果表明，我们提出的基于 ICL 的框架在所有版本的 EBM-NLP 数据集上都产生了可比的结果，而我们提出的指令调整版本的框架在所有不同的 EBM-NLP 数据集上都产生了最先进的结果。我们的项目可在 \url{this https URL} 上找到。

Title: Benchmarking LLMs in Political Content Text-Annotation: Proof-of-Concept with Toxicity and Incivility Data

• Authors: Bastián González-Bustamante
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.09741
• Pdf URL: https://arxiv.org/pdf/2409.09741
• Copy Paste: [[2409.09741]] Benchmarking LLMs in Political Content Text-Annotation: Proof-of-Concept with Toxicity and Incivility Data(https://arxiv.org/abs/2409.09741)
• Keywords: gpt, llm
• Abstract: This article benchmarked the ability of OpenAI's GPTs and a number of open-source LLMs to perform annotation tasks on political content. We used a novel protest event dataset comprising more than three million digital interactions and created a gold standard that includes ground-truth labels annotated by human coders about toxicity and incivility on social media. We included in our benchmark Google's Perspective algorithm, which, along with GPTs, was employed throughout their respective APIs while the open-source LLMs were deployed locally. The findings show that Perspective API using a laxer threshold, GPT-4o, and Nous Hermes 2 Mixtral outperform other LLM's zero-shot classification annotations. In addition, Nous Hermes 2 and Mistral OpenOrca, with a smaller number of parameters, are able to perform the task with high performance, being attractive options that could offer good trade-offs between performance, implementing costs and computing time. Ancillary findings using experiments setting different temperature levels show that although GPTs tend to show not only excellent computing time but also overall good levels of reliability, only open-source LLMs ensure full reproducibility in the annotation.
• 摘要：本文对 OpenAI 的 GPT 和一些开源 LLM 执行政治内容注释任务的能力进行了基准测试。我们使用了一个包含 300 多万次数字互动的新型抗议事件数据集，并创建了一个黄金标准，其中包括由人类编码员注释的有关社交媒体上的毒性和不文明行为的真实标签。我们在基准测试中加入了 Google 的 Perspective 算法，该算法与 GPT 一起在各自的 API 中使用，而开源 LLM 则部署在本地。研究结果表明，使用较宽松阈值的 Perspective API、GPT-4o 和 Nous Hermes 2 Mixtral 的表现优于其他 LLM 的零样本分类注释。此外，Nous Hermes 2 和 Mistral OpenOrca 的参数数量较少，能够以高性能执行任务，是可以在性能、实施成本和计算时间之间提供良好权衡的有吸引力的选择。使用设置不同温度水平的实验得出的辅助结果表明，尽管 GPT 不仅表现出出色的计算时间，而且总体上表现出良好的可靠性水平，但只有开源 LLM 才能确保注释的完全可重复性。

Title: Large Language Model Based Generative Error Correction: A Challenge and Baselines forSpeech Recognition, Speaker Tagging, and Emotion Recognition

• Authors: Chao-Han Huck Yang, Taejin Park, Yuan Gong, Yuanchao Li, Zhehuai Chen, Yen-Ting Lin, Chen Chen, Yuchen Hu, Kunal Dhawan, Piotr Żelasko, Chao Zhang, Yun-Nung Chen, Yu Tsao, Jagadeesh Balam, Boris Ginsburg, Sabato Marco Siniscalchi, Eng Siong Chng, Peter Bell, Catherine Lai, Shinji Watanabe, Andreas Stolcke
• Subjects: cs.CL, cs.AI, cs.LG, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2409.09785
• Pdf URL: https://arxiv.org/pdf/2409.09785
• Copy Paste: [[2409.09785]] Large Language Model Based Generative Error Correction: A Challenge and Baselines forSpeech Recognition, Speaker Tagging, and Emotion Recognition(https://arxiv.org/abs/2409.09785)
• Keywords: language model, llm, agent
• Abstract: Given recent advances in generative AI technology, a key question is how large language models (LLMs) can enhance acoustic modeling tasks using text decoding results from a frozen, pretrained automatic speech recognition (ASR) model. To explore new capabilities in language modeling for speech processing, we introduce the generative speech transcription error correction (GenSEC) challenge. This challenge comprises three post-ASR language modeling tasks: (i) post-ASR transcription correction, (ii) speaker tagging, and (iii) emotion recognition. These tasks aim to emulate future LLM-based agents handling voice-based interfaces while remaining accessible to a broad audience by utilizing open pretrained language models or agent-based APIs. We also discuss insights from baseline evaluations, as well as lessons learned for designing future evaluations.
• 摘要：鉴于生成式 AI 技术的最新进展，一个关键问题是大型语言模型 (LLM) 如何使用冻结的、预训练的自动语音识别 (ASR) 模型的文本解码结果来增强声学建模任务。为了探索语言建模在语音处理中的新功能，我们推出了生成式语音转录错误校正 (GenSEC) 挑战赛。该挑战赛包括三个 ASR 后语言建模任务：(i) ASR 后转录校正、(ii) 说话人标记和 (iii) 情绪识别。这些任务旨在通过利用开放的预训练语言模型或基于代理的 API 来模拟未来基于 LLM 的代理处理语音界面，同时保持广大受众可以访问。我们还讨论了基线评估的见解，以及设计未来评估的经验教训。

Title: Causal Inference with Large Language Model: A Survey

• Authors: Jing Ma
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.09822
• Pdf URL: https://arxiv.org/pdf/2409.09822
• Copy Paste: [[2409.09822]] Causal Inference with Large Language Model: A Survey(https://arxiv.org/abs/2409.09822)
• Keywords: language model, llm
• Abstract: Causal inference has been a pivotal challenge across diverse domains such as medicine and economics, demanding a complicated integration of human knowledge, mathematical reasoning, and data mining capabilities. Recent advancements in natural language processing (NLP), particularly with the advent of large language models (LLMs), have introduced promising opportunities for traditional causal inference tasks. This paper reviews recent progress in applying LLMs to causal inference, encompassing various tasks spanning different levels of causation. We summarize the main causal problems and approaches, and present a comparison of their evaluation results in different causal scenarios. Furthermore, we discuss key findings and outline directions for future research, underscoring the potential implications of integrating LLMs in advancing causal inference methodologies.
• 摘要：因果推理一直是医学和经济学等不同领域的一个关键挑战，需要将人类知识、数学推理和数据挖掘能力进行复杂的整合。自然语言处理 (NLP) 的最新进展，尤其是大型语言模型 (LLM) 的出现，为传统的因果推理任务带来了希望。本文回顾了将 LLM 应用于因果推理的最新进展，涵盖了不同因果层次的各种任务。我们总结了主要的因果问题和方法，并对它们在不同因果场景中的评估结果进行了比较。此外，我们讨论了关键发现并概述了未来研究的方向，强调了整合 LLM 对推进因果推理方法的潜在影响。

Title: GP-GPT: Large Language Model for Gene-Phenotype Mapping

• Authors: Yanjun Lyu, Zihao Wu, Lu Zhang, Jing Zhang, Yiwei Li, Wei Ruan, Zhengliang Liu, Xiaowei Yu, Chao Cao, Tong Chen, Minheng Chen, Yan Zhuang, Xiang Li, Rongjie Liu, Chao Huang, Wentao Li, Tianming Liu, Dajiang Zhu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.09825
• Pdf URL: https://arxiv.org/pdf/2409.09825
• Copy Paste: [[2409.09825]] GP-GPT: Large Language Model for Gene-Phenotype Mapping(https://arxiv.org/abs/2409.09825)
• Keywords: language model, gpt, llm
• Abstract: Pre-trained large language models(LLMs) have attracted increasing attention in biomedical domains due to their success in natural language processing. However, the complex traits and heterogeneity of multi-sources genomics data pose significant challenges when adapting these models to the bioinformatics and biomedical field. To address these challenges, we present GP-GPT, the first specialized large language model for genetic-phenotype knowledge representation and genomics relation analysis. Our model is fine-tuned in two stages on a comprehensive corpus composed of over 3,000,000 terms in genomics, proteomics, and medical genetics, derived from multiple large-scale validated datasets and scientific publications. GP-GPT demonstrates proficiency in accurately retrieving medical genetics information and performing common genomics analysis tasks, such as genomics information retrieval and relationship determination. Comparative experiments across domain-specific tasks reveal that GP-GPT outperforms state-of-the-art LLMs, including Llama2, Llama3 and GPT-4. These results highlight GP-GPT's potential to enhance genetic disease relation research and facilitate accurate and efficient analysis in the fields of genomics and medical genetics. Our investigation demonstrated the subtle changes of bio-factor entities' representations in the GP-GPT, which suggested the opportunities for the application of LLMs to advancing gene-phenotype research.
• 摘要：预训练的大型语言模型 (LLM) 因其在自然语言处理方面的成功而在生物医学领域引起了越来越多的关注。然而，多源基因组数据的复杂性状和异质性给将这些模型应用于生物信息学和生物医学领域带来了重大挑战。为了应对这些挑战，我们提出了 GP-GPT，这是第一个用于遗传-表型知识表示和基因组关系分析的专门大型语言模型。我们的模型分两个阶段在一个综合语料库上进行微调，该语料库由超过 3,000,000 个基因组学、蛋白质组学和医学遗传学术语组成，来自多个大规模经过验证的数据集和科学出版物。GP-GPT 能够准确检索医学遗传学信息并执行常见的基因组学分析任务，例如基因组学信息检索和关系确定。跨领域特定任务的比较实验表明，GP-GPT 的表现优于最先进的 LLM，包括 Llama2、Llama3 和 GPT-4。这些结果凸显了 GP-GPT 在增强遗传疾病关系研究和促进基因组学和医学遗传学领域准确高效分析方面的潜力。我们的调查展示了 GP-GPT 中生物因子实体表示的细微变化，这表明 LLM 有机会应用于推进基因表型研究。

Title: Generating Synthetic Free-text Medical Records with Low Re-identification Risk using Masked Language Modeling

• Authors: Samuel Belkadi, Libo Ren, Nicolo Micheletti, Lifeng Han, Goran Nenadic
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.09831
• Pdf URL: https://arxiv.org/pdf/2409.09831
• Copy Paste: [[2409.09831]] Generating Synthetic Free-text Medical Records with Low Re-identification Risk using Masked Language Modeling(https://arxiv.org/abs/2409.09831)
• Keywords: language model
• Abstract: In this paper, we present a system that generates synthetic free-text medical records, such as discharge summaries, admission notes and doctor correspondences, using Masked Language Modeling (MLM). Our system is designed to preserve the critical information of the records while introducing significant diversity and minimizing re-identification risk. The system incorporates a de-identification component that uses Philter to mask Protected Health Information (PHI), followed by a Medical Entity Recognition (NER) model to retain key medical information. We explore various masking ratios and mask-filling techniques to balance the trade-off between diversity and fidelity in the synthetic outputs without affecting overall readability. Our results demonstrate that the system can produce high-quality synthetic data with significant diversity while achieving a HIPAA-compliant PHI recall rate of 0.96 and a low re-identification risk of 0.035. Furthermore, downstream evaluations using a NER task reveal that the synthetic data can be effectively used to train models with performance comparable to those trained on real data. The flexibility of the system allows it to be adapted for specific use cases, making it a valuable tool for privacy-preserving data generation in medical research and healthcare applications.
• 摘要：在本文中，我们介绍了一种使用掩码语言模型 (MLM) 生成合成自由文本医疗记录（例如出院摘要、入院记录和医生通信）的系统。我们的系统旨在保留记录的关键信息，同时引入显著的多样性并最大限度地降低重新识别风险。该系统包含一个去识别组件，该组件使用 Philter 来屏蔽受保护的健康信息 (PHI)，然后使用医疗实体识别 (NER) 模型来保留关键医疗信息。我们探索了各种掩码率和掩码填充技术，以平衡合成输出的多样性和保真度之间的权衡，而不会影响整体可读性。我们的结果表明，该系统可以生成具有显著多样性的高质量合成数据，同时实现符合 HIPAA 标准的 PHI 召回率 0.96 和低重新识别风险 0.035。此外，使用 NER 任务进行的下游评估表明，合成数据可以有效地用于训练模型，其性能与在真实数据上训练的模型相当。该系统的灵活性使其能够适应特定的用例，使其成为医学研究和医疗保健应用中保护隐私数据生成的宝贵工具。

Title: Constructing a Singing Style Caption Dataset

• Authors: Hyunjong Ok, Jaeho Lee
• Subjects: cs.CL, cs.AI, cs.LG, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2409.09866
• Pdf URL: https://arxiv.org/pdf/2409.09866
• Copy Paste: [[2409.09866]] Constructing a Singing Style Caption Dataset(https://arxiv.org/abs/2409.09866)
• Keywords: prompt
• Abstract: Singing voice synthesis and conversion have emerged as significant subdomains of voice generation, leading to much demands on prompt-conditioned generation. Unlike common voice data, generating a singing voice requires an understanding of various associated vocal and musical characteristics, such as the vocal tone of the singer or emotional expressions. However, existing open-source audio-text datasets for voice generation tend to capture only a very limited range of attributes, often missing musical characteristics of the audio. To fill this gap, we introduce S2Cap, an audio-text pair dataset with a diverse set of attributes. S2Cap consists of pairs of textual prompts and music audio samples with a wide range of vocal and musical attributes, including pitch, volume, tempo, mood, singer's gender and age, and musical genre and emotional expression. Utilizing S2Cap, we suggest an effective novel baseline algorithm for singing style captioning. Singing style captioning is a relative task to voice generation that generates text descriptions of vocal characteristics, which we first suggested. First, to mitigate the misalignment between the audio encoder and the text decoder, we present a novel mechanism called CRESCENDO, which utilizes positive-pair similarity learning to synchronize the embedding spaces of a pretrained audio encoder to get similar embeddings with a text encoder. We additionally supervise the model using the singer's voice, which is demixed by the accompaniment. This supervision allows the model to more accurately capture vocal characteristics, leading to improved singing style captions that better reflect the style of the singer. The dataset and the codes are available at \bulurl{this https URL}.
• 摘要：歌唱声音合成和转换已成为语音生成的重要子领域，对提示条件生成的需求很大。与常见的语音数据不同，生成歌唱声音需要了解各种相关的声音和音乐特征，例如歌手的声调或情感表达。然而，现有的用于语音生成的开源音频文本数据集往往只捕捉非常有限的属性范围，往往缺少音频的音乐特征。为了填补这一空白，我们引入了 S2Cap，这是一个具有多种属性集的音频文本对数据集。S2Cap 由文本提示和音乐音频样本对组成，具有广泛的声音和音乐属性，包括音高、音量、节奏、情绪、歌手的性别和年龄以及音乐流派和情感表达。利用 S2Cap，我们提出了一种有效的新型歌唱风格字幕基线算法。歌唱风格字幕是一项与语音生成相关的任务，它生成我们首先提出的歌唱特征的文本描述。首先，为了减轻音频编码器和文本解码器之间的不一致，我们提出了一种名为 CRESCENDO 的新机制，该机制利用正对相似性学习来同步预训练音频编码器的嵌入空间，以获得与文本编码器相似的嵌入。我们还使用歌手的声音来监督模型，歌手的声音与伴奏分离。这种监督使模型能够更准确地捕捉声音特征，从而改进歌唱风格字幕，更好地反映歌手的风格。数据集和代码可在 \bulurl{此 https URL} 上找到。

Title: Rediscovering the Latent Dimensions of Personality with Large Language Models as Trait Descriptors

• Authors: Joseph Suh, Suhong Moon, Minwoo Kang, David M. Chan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.09905
• Pdf URL: https://arxiv.org/pdf/2409.09905
• Copy Paste: [[2409.09905]] Rediscovering the Latent Dimensions of Personality with Large Language Models as Trait Descriptors(https://arxiv.org/abs/2409.09905)
• Keywords: language model, llm
• Abstract: Assessing personality traits using large language models (LLMs) has emerged as an interesting and challenging area of research. While previous methods employ explicit questionnaires, often derived from the Big Five model of personality, we hypothesize that LLMs implicitly encode notions of personality when modeling next-token responses. To demonstrate this, we introduce a novel approach that uncovers latent personality dimensions in LLMs by applying singular value de-composition (SVD) to the log-probabilities of trait-descriptive adjectives. Our experiments show that LLMs "rediscover" core personality traits such as extraversion, agreeableness, conscientiousness, neuroticism, and openness without relying on direct questionnaire inputs, with the top-5 factors corresponding to Big Five traits explaining 74.3% of the variance in the latent space. Moreover, we can use the derived principal components to assess personality along the Big Five dimensions, and achieve improvements in average personality prediction accuracy of up to 5% over fine-tuned models, and up to 21% over direct LLM-based scoring techniques.
• 摘要：使用大型语言模型 (LLM) 评估人格特质已成为一个有趣且具有挑战性的研究领域。虽然以前的方法采用显式问卷，通常源自人格的“大五”模型，但我们假设 LLM 在建模下一个标记响应时会隐式编码人格概念。为了证明这一点，我们引入了一种新方法，该方法通过将奇异值分解 (SVD) 应用于特征描述形容词的对数概率来揭示 LLM 中的潜在人格维度。我们的实验表明，LLM 无需依赖直接的问卷输入即可“重新发现”外向性、宜人性、尽责性、神经质和开放性等核心人格特质，其中与“大五”特质相对应的前 5 个因素解释了潜在空间中 74.3% 的方差。此外，我们可以使用派生的主成分来评估“大五”维度的性格，并将平均性格预测准确率提高到比微调模型高出 5% 左右，比直接基于 LLM 的评分技术高出 21% 左右。

Title: SFR-RAG: Towards Contextually Faithful LLMs

• Authors: Xuan-Phi Nguyen, Shrey Pandit, Senthil Purushwalkam, Austin Xu, Hailin Chen, Yifei Ming, Zixuan Ke, Silvio Savarese, Caiming Xong, Shafiq Joty
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.09916
• Pdf URL: https://arxiv.org/pdf/2409.09916
• Copy Paste: [[2409.09916]] SFR-RAG: Towards Contextually Faithful LLMs(https://arxiv.org/abs/2409.09916)
• Keywords: language model, gpt, llm, hallucination, retrieval augmented generation
• Abstract: Retrieval Augmented Generation (RAG), a paradigm that integrates external contextual information with large language models (LLMs) to enhance factual accuracy and relevance, has emerged as a pivotal area in generative AI. The LLMs used in RAG applications are required to faithfully and completely comprehend the provided context and users' questions, avoid hallucination, handle unanswerable, counterfactual or otherwise low-quality and irrelevant contexts, perform complex multi-hop reasoning and produce reliable citations. In this paper, we introduce SFR-RAG, a small LLM that is instruction-tuned with an emphasis on context-grounded generation and hallucination minimization. We also present ContextualBench, a new evaluation framework compiling multiple popular and diverse RAG benchmarks, such as HotpotQA and TriviaQA, with consistent RAG settings to ensure reproducibility and consistency in model assessments. Experimental results demonstrate that our SFR-RAG-9B model outperforms leading baselines such as Command-R+ (104B) and GPT-4o, achieving state-of-the-art results in 3 out of 7 benchmarks in ContextualBench with significantly fewer parameters. The model is also shown to be resilient to alteration in the contextual information and behave appropriately when relevant context is removed. Additionally, the SFR-RAG model maintains competitive performance in general instruction-following tasks and function-calling capabilities.
• 摘要：检索增强生成 (RAG) 是一种将外部上下文信息与大型语言模型 (LLM) 相结合以提高事实准确性和相关性的范式，已成为生成式 AI 的一个关键领域。RAG 应用程序中使用的 LLM 需要忠实而完整地理解所提供的上下文和用户的问题，避免幻觉，处理无法回答、反事实或其他低质量和不相关的上下文，执行复杂的多跳推理并产生可靠的引用。在本文中，我们介绍了 SFR-RAG，这是一种小型 LLM，它经过指令调整，重点是基于上下文的生成和幻觉最小化。我们还提出了 ContextualBench，这是一个新的评估框架，它汇编了多个流行且多样化的 RAG 基准，例如 HotpotQA 和 TriviaQA，具有一致的 RAG 设置，以确保模型评估的可重复性和一致性。实验结果表明，我们的 SFR-RAG-9B 模型的表现优于 Command-R+ (104B) 和 GPT-4o 等领先基线，在 ContextualBench 的 7 个基准测试中有 3 个取得了最佳结果，而且参数明显更少。该模型还显示出对上下文信息更改的适应性，并且在删除相关上下文时表现得当。此外，SFR-RAG 模型在一般的指令跟踪任务和函数调用功能方面保持了竞争性的性能。

Title: Towards Data Contamination Detection for Modern Large Language Models: Limitations, Inconsistencies, and Oracle Challenges

• Authors: Vinay Samuel, Yue Zhou, Henry Peng Zou
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.09927
• Pdf URL: https://arxiv.org/pdf/2409.09927
• Copy Paste: [[2409.09927]] Towards Data Contamination Detection for Modern Large Language Models: Limitations, Inconsistencies, and Oracle Challenges(https://arxiv.org/abs/2409.09927)
• Keywords: language model, llm
• Abstract: As large language models achieve increasingly impressive results, questions arise about whether such performance is from generalizability or mere data memorization. Thus, numerous data contamination detection methods have been proposed. However, these approaches are often validated with traditional benchmarks and early-stage LLMs, leaving uncertainty about their effectiveness when evaluating state-of-the-art LLMs on the contamination of more challenging benchmarks. To address this gap and provide a dual investigation of SOTA LLM contamination status and detection method robustness, we evaluate five contamination detection approaches with four state-of-the-art LLMs across eight challenging datasets often used in modern LLM evaluation. Our analysis reveals that (1) Current methods have non-trivial limitations in their assumptions and practical applications; (2) Notable difficulties exist in detecting contamination introduced during instruction fine-tuning with answer augmentation; and (3) Limited consistencies between SOTA contamination detection techniques. These findings highlight the complexity of contamination detection in advanced LLMs and the urgent need for further research on robust and generalizable contamination evaluation. Our code is available at this https URL.
• 摘要：随着大型语言模型取得越来越令人印象深刻的结果，人们开始质疑这种性能是来自于通用性还是仅仅是数据记忆。因此，提出了许多数据污染检测方法。然而，这些方法通常用传统基准和早期的 LLM 来验证，因此在评估最先进的 LLM 对更具挑战性的基准的污染时，它们的有效性尚不确定。为了解决这一差距并对 SOTA LLM 污染状态和检测方法稳健性进行双重调查，我们使用四种最先进的 LLM 在现代 LLM 评估中经常使用的八个具有挑战性的数据集上评估了五种污染检测方法。我们的分析表明：(1) 当前方法在假设和实际应用中存在非平凡的局限性；(2) 在检测通过答案增强进行指令微调期间引入的污染方面存在显著困难；(3) SOTA 污染检测技术之间的一致性有限。这些发现凸显了高级 LLM 中污染检测的复杂性以及对稳健和可推广的污染评估进行进一步研究的迫切需要。我们的代码可在此 https URL 上找到。

Title: Gaps or Hallucinations? Gazing into Machine-Generated Legal Analysis for Fine-grained Text Evaluations

• Authors: Abe Bohan Hou, William Jurayj, Nils Holzenberger, Andrew Blair-Stanek, Benjamin Van Durme
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2409.09947
• Pdf URL: https://arxiv.org/pdf/2409.09947
• Copy Paste: [[2409.09947]] Gaps or Hallucinations? Gazing into Machine-Generated Legal Analysis for Fine-grained Text Evaluations(https://arxiv.org/abs/2409.09947)
• Keywords: language model, llm, hallucination
• Abstract: Large Language Models (LLMs) show promise as a writing aid for professionals performing legal analyses. However, LLMs can often hallucinate in this setting, in ways difficult to recognize by non-professionals and existing text evaluation metrics. In this work, we pose the question: when can machine-generated legal analysis be evaluated as acceptable? We introduce the neutral notion of gaps, as opposed to hallucinations in a strict erroneous sense, to refer to the difference between human-written and machine-generated legal analysis. Gaps do not always equate to invalid generation. Working with legal experts, we consider the CLERC generation task proposed in Hou et al. (2024b), leading to a taxonomy, a fine-grained detector for predicting gap categories, and an annotated dataset for automatic evaluation. Our best detector achieves 67% F1 score and 80% precision on the test set. Employing this detector as an automated metric on legal analysis generated by SOTA LLMs, we find around 80% contain hallucinations of different kinds.
• 摘要：大型语言模型 (LLM) 有望成为专业人士进行法律分析的写作辅助工具。然而，在这种情况下，LLM 往往会产生幻觉，非专业人士和现有的文本评估指标都难以识别。在这项工作中，我们提出了一个问题：机器生成的法律分析何时可以被评估为可接受的？我们引入了中性的差距概念，而不是严格错误意义上的幻觉，来指代人写的法律分析和机器生成的法律分析之间的差异。差距并不总是等同于无效生成。我们与法律专家合作，考虑了 Hou 等人 (2024b) 中提出的 CLERC 生成任务，从而产生了一个分类法、一个用于预测差距类别的细粒度检测器和一个用于自动评估的带注释数据集。我们最好的检测器在测试集上实现了 67% 的 F1 分数和 80% 的精度。使用该检测器作为 SOTA LLM 生成的法律分析的自动化指标，我们发现大约 80% 包含不同类型的幻觉。

Title: Comprehensive Study on Sentiment Analysis: From Rule-based to modern LLM based system

• Authors: Shailja Gupta, Rajesh Ranjan, Surya Narayan Singh
• Subjects: cs.CL, cs.AI, cs.CY, cs.HC
• Abstract URL: https://arxiv.org/abs/2409.09989
• Pdf URL: https://arxiv.org/pdf/2409.09989
• Copy Paste: [[2409.09989]] Comprehensive Study on Sentiment Analysis: From Rule-based to modern LLM based system(https://arxiv.org/abs/2409.09989)
• Keywords: language model, llm
• Abstract: This paper provides a comprehensive survey of sentiment analysis within the context of artificial intelligence (AI) and large language models (LLMs). Sentiment analysis, a critical aspect of natural language processing (NLP), has evolved significantly from traditional rule-based methods to advanced deep learning techniques. This study examines the historical development of sentiment analysis, highlighting the transition from lexicon-based and pattern-based approaches to more sophisticated machine learning and deep learning models. Key challenges are discussed, including handling bilingual texts, detecting sarcasm, and addressing biases. The paper reviews state-of-the-art approaches, identifies emerging trends, and outlines future research directions to advance the field. By synthesizing current methodologies and exploring future opportunities, this survey aims to understand sentiment analysis in the AI and LLM context thoroughly.
• 摘要：本文对人工智能 (AI) 和大型语言模型 (LLM) 背景下的情绪分析进行了全面概述。情绪分析是自然语言处理 (NLP) 的一个重要方面，它已从传统的基于规则的方法发展到先进的深度学习技术。本研究考察了情绪分析的历史发展，强调了从基于词典和基于模式的方法到更复杂的机器学习和深度学习模型的转变。讨论了关键挑战，包括处理双语文本、检测讽刺和解决偏见。本文回顾了最先进的方法，确定了新兴趋势，并概述了推动该领域发展的未来研究方向。通过综合当前方法并探索未来的机会，本调查旨在彻底了解 AI 和 LLM 背景下的情绪分析。

Title: SelECT-SQL: Self-correcting ensemble Chain-of-Thought for Text-to-SQL

• Authors: Ke Shen, Mayank Kejriwal
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.10007
• Pdf URL: https://arxiv.org/pdf/2409.10007
• Copy Paste: [[2409.10007]] SelECT-SQL: Self-correcting ensemble Chain-of-Thought for Text-to-SQL(https://arxiv.org/abs/2409.10007)
• Keywords: language model, gpt, llm, prompt, chain-of-thought
• Abstract: In recent years,Text-to-SQL, the problem of automatically converting questions posed in natural language to formal SQL queries, has emerged as an important problem at the intersection of natural language processing and data management research. Large language models (LLMs) have delivered impressive performance when used in an off-the-shelf performance, but still fall significantly short of expected expert-level performance. Errors are especially probable when a nuanced understanding is needed of database schemas, questions, and SQL clauses to do proper Text-to-SQL conversion. We introduce SelECT-SQL, a novel in-context learning solution that uses an algorithmic combination of chain-of-thought (CoT) prompting, self-correction, and ensemble methods to yield a new state-of-the-art result on challenging Text-to-SQL benchmarks. Specifically, when configured using GPT-3.5-Turbo as the base LLM, SelECT-SQL achieves 84.2% execution accuracy on the Spider leaderboard's development set, exceeding both the best results of other baseline GPT-3.5-Turbo-based solutions (81.1%), and the peak performance (83.5%) of the GPT-4 result reported on the leaderboard.
• 摘要：近年来，文本到 SQL（将自然语言中提出的问题自动转换为正式 SQL 查询的问题）已成为自然语言处理和数据管理研究交叉领域的一个重要问题。大型语言模型 (LLM) 在现成的性能中使用时提供了令人印象深刻的性能，但仍远远达不到预期的专家级性能。当需要对数据库模式、问题和 SQL 子句有细致的理解才能进行正确的文本到 SQL 转换时，尤其容易出现错误。我们推出了 SelECT-SQL，这是一种新颖的上下文学习解决方案，它使用思路链 (CoT) 提示、自我纠正和集成方法的算法组合，在具有挑战性的文本到 SQL 基准测试中产生新的最先进结果。具体来说，当使用 GPT-3.5-Turbo 作为基础 LLM 进行配置时，SelECT-SQL 在 Spider 排行榜的开发集上实现了 84.2% 的执行准确率，超过了其他基于基线 GPT-3.5-Turbo 的解决方案的最佳结果（81.1%）和排行榜上报告的 GPT-4 结果的峰值性能（83.5%）。

Title: HALO: Hallucination Analysis and Learning Optimization to Empower LLMs with Retrieval-Augmented Context for Guided Clinical Decision Making

• Authors: Sumera Anjum, Hanzhi Zhang, Wenjun Zhou, Eun Jin Paek, Xiaopeng Zhao, Yunhe Feng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.10011
• Pdf URL: https://arxiv.org/pdf/2409.10011
• Copy Paste: [[2409.10011]] HALO: Hallucination Analysis and Learning Optimization to Empower LLMs with Retrieval-Augmented Context for Guided Clinical Decision Making(https://arxiv.org/abs/2409.10011)
• Keywords: language model, gpt, llm, hallucination, chat
• Abstract: Large language models (LLMs) have significantly advanced natural language processing tasks, yet they are susceptible to generating inaccurate or unreliable responses, a phenomenon known as hallucination. In critical domains such as health and medicine, these hallucinations can pose serious risks. This paper introduces HALO, a novel framework designed to enhance the accuracy and reliability of medical question-answering (QA) systems by focusing on the detection and mitigation of hallucinations. Our approach generates multiple variations of a given query using LLMs and retrieves relevant information from external open knowledge bases to enrich the context. We utilize maximum marginal relevance scoring to prioritize the retrieved context, which is then provided to LLMs for answer generation, thereby reducing the risk of hallucinations. The integration of LangChain further streamlines this process, resulting in a notable and robust increase in the accuracy of both open-source and commercial LLMs, such as Llama-3.1 (from 44% to 65%) and ChatGPT (from 56% to 70%). This framework underscores the critical importance of addressing hallucinations in medical QA systems, ultimately improving clinical decision-making and patient care. The open-source HALO is available at: this https URL.
• 摘要：大型语言模型 (LLM) 显著提高了自然语言处理任务的效率，但它们容易产生不准确或不可靠的响应，这种现象称为幻觉。在健康和医学等关键领域，这些幻觉可能带来严重风险。本文介绍了 HALO，这是一种新颖的框架，旨在通过专注于检测和缓解幻觉来提高医学问答 (QA) 系统的准确性和可靠性。我们的方法使用 LLM 生成给定查询的多种变体，并从外部开放知识库中检索相关信息以丰富上下文。我们利用最大边际相关性评分对检索到的上下文进行优先排序，然后将其提供给 LLM 以生成答案，从而降低幻觉的风险。LangChain 的集成进一步简化了这一过程，从而使开源和商业 LLM 的准确率均显着提高，例如 Llama-3.1（从 44% 提高到 65%）和 ChatGPT（从 56% 提高到 70%）。该框架强调了在医疗 QA 系统中解决幻觉问题的重要性，最终改善临床决策和患者护理。开源 HALO 可从以下网址获取：此 https URL。

Title: On the Diagram of Thought

• Authors: Yifan Zhang, Yang Yuan, Andrew Chi-Chih Yao
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.10038
• Pdf URL: https://arxiv.org/pdf/2409.10038
• Copy Paste: [[2409.10038]] On the Diagram of Thought(https://arxiv.org/abs/2409.10038)
• Keywords: language model, llm
• Abstract: We introduce Diagram of Thought (DoT), a framework that models iterative reasoning in large language models (LLMs) as the construction of a directed acyclic graph (DAG) within a single model. Unlike traditional approaches that represent reasoning as linear chains or trees, DoT organizes propositions, critiques, refinements, and verifications into a cohesive DAG structure, allowing the model to explore complex reasoning pathways while maintaining logical consistency. Each node in the diagram corresponds to a proposition that has been proposed, critiqued, refined, or verified, enabling the LLM to iteratively improve its reasoning through natural language feedback. By leveraging auto-regressive next-token prediction with role-specific tokens, DoT facilitates seamless transitions between proposing ideas and critically evaluating them, providing richer feedback than binary signals. Furthermore, we formalize the DoT framework using Topos Theory, providing a mathematical foundation that ensures logical consistency and soundness in the reasoning process. This approach enhances both the training and inference processes within a single LLM, eliminating the need for multiple models or external control mechanisms. DoT offers a conceptual framework for designing next-generation reasoning-specialized models, emphasizing training efficiency, robust reasoning capabilities, and theoretical grounding. The code is available at this https URL.
• 摘要：我们引入了思维图 (DoT)，这是一个将大型语言模型 (LLM) 中的迭代推理建模为单个模型中有向无环图 (DAG) 的构造的框架。与将推理表示为线性链或树的传统方法不同，DoT 将命题、批评、改进和验证组织成一个有凝聚力的 DAG 结构，使模型能够探索复杂的推理路径，同时保持逻辑一致性。图中的每个节点都对应一个已提出、批评、改进或验证的命题，使 LLM 能够通过自然语言反馈迭代地改进其推理。通过利用具有特定角色标记的自回归下一个标记预测，DoT 促进了提出想法和批判性评估想法之间的无缝过渡，提供了比二进制信号更丰富的反馈。此外，我们使用 Topos 理论形式化了 DoT 框架，提供了确保推理过程中逻辑一致性和合理性的数学基础。这种方法增强了单个 LLM 中的训练和推理过程，无需多个模型或外部控制机制。DoT 提供了一个用于设计下一代推理专用模型的概念框架，强调训练效率、强大的推理能力和理论基础。代码可在此 https URL 上获取。

Title: Householder Pseudo-Rotation: A Novel Approach to Activation Editing in LLMs with Direction-Magnitude Perspective

• Authors: Van-Cuong Pham, Thien Huu Nguyen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.10053
• Pdf URL: https://arxiv.org/pdf/2409.10053
• Copy Paste: [[2409.10053]] Householder Pseudo-Rotation: A Novel Approach to Activation Editing in LLMs with Direction-Magnitude Perspective(https://arxiv.org/abs/2409.10053)
• Keywords: language model, llm
• Abstract: Activation Editing, which involves directly editting the internal representations of large language models (LLMs) to alter their behaviors and achieve desired properties, has emerged as a promising area of research. Existing works primarily treat LLMs' activations as points in space and modify them by adding steering vectors. However, this approach is limited in its ability to achieve greater performance improvement while maintaining the necessary consistency of activation magnitudes. To overcome these issues, we propose a novel editing method that views activations in terms of their directions and magnitudes. Our method, named Householder Pseudo-Rotation (HPR), mimics the rotation transformation, thus preserving activation norms and resulting in an improved performance on various safety benchmarks.
• 摘要：激活编辑涉及直接编辑大型语言模型 (LLM) 的内部表示以改变其行为并实现所需的属性，已成为一个有前途的研究领域。现有研究主要将 LLM 的激活视为空间中的点，并通过添加转向矢量来修改它们。然而，这种方法在保持激活幅度必要一致性的同时实现更大性能改进的能力有限。为了克服这些问题，我们提出了一种新颖的编辑方法，该方法从方向和幅度的角度来查看激活。我们的方法称为 Householder 伪旋转 (HPR)，它模拟旋转变换，从而保留激活规范并提高各种安全基准上的性能。

Title: MindGuard: Towards Accessible and Sitgma-free Mental Health First Aid via Edge LLM

• Authors: Sijie Ji, Xinzhe Zheng, Jiawei Sun, Renqi Chen, Wei Gao, Mani Srivastava
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2409.10064
• Pdf URL: https://arxiv.org/pdf/2409.10064
• Copy Paste: [[2409.10064]] MindGuard: Towards Accessible and Sitgma-free Mental Health First Aid via Edge LLM(https://arxiv.org/abs/2409.10064)
• Keywords: gpt, llm
• Abstract: Mental health disorders are among the most prevalent diseases worldwide, affecting nearly one in four people. Despite their widespread impact, the intervention rate remains below 25%, largely due to the significant cooperation required from patients for both diagnosis and intervention. The core issue behind this low treatment rate is stigma, which discourages over half of those affected from seeking help. This paper presents MindGuard, an accessible, stigma-free, and professional mobile mental healthcare system designed to provide mental health first aid. The heart of MindGuard is an innovative edge LLM, equipped with professional mental health knowledge, that seamlessly integrates objective mobile sensor data with subjective Ecological Momentary Assessment records to deliver personalized screening and intervention conversations. We conduct a broad evaluation of MindGuard using open datasets spanning four years and real-world deployment across various mobile devices involving 20 subjects for two weeks. Remarkably, MindGuard achieves results comparable to GPT-4 and outperforms its counterpart with more than 10 times the model size. We believe that MindGuard paves the way for mobile LLM applications, potentially revolutionizing mental healthcare practices by substituting self-reporting and intervention conversations with passive, integrated monitoring within daily life, thus ensuring accessible and stigma-free mental health support.
• 摘要：精神健康障碍是全球最常见的疾病之一，影响着近四分之一的人。尽管影响广泛，但干预率仍低于 25%，这主要是因为诊断和干预都需要患者的大量配合。这种低治疗率背后的核心问题是耻辱感，这使超过一半的患者不愿寻求帮助。本文介绍了 MindGuard，这是一种可访问、无耻且专业的移动心理健康保健系统，旨在提供心理健康急救。MindGuard 的核心是一个创新的前沿 LLM，配备了专业的心理健康知识，可将客观的移动传感器数据与主观的生态瞬时评估记录无缝集成，以提供个性化的筛查和干预对话。我们使用跨越四年的开放数据集和涉及 20 名受试者在两周内的各种移动设备上的真实部署对 MindGuard 进行了广泛的评估。值得注意的是，MindGuard 取得了与 GPT-4 相当的结果，并且比模型大小大 10 倍以上的对手表现更好。我们相信，MindGuard 为移动 LLM 应用程序铺平了道路，通过用日常生活中的被动、综合监控取代自我报告和干预对话，有可能彻底改变心理健康实践，从而确保可获得且无污名的心理健康支持。

Title: Increasing faithfulness in human-human dialog summarization with Spoken Language Understanding tasks

• Authors: Eunice Akani, Benoit Favre, Frederic Bechet, Romain Gemignani
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.10070
• Pdf URL: https://arxiv.org/pdf/2409.10070
• Copy Paste: [[2409.10070]] Increasing faithfulness in human-human dialog summarization with Spoken Language Understanding tasks(https://arxiv.org/abs/2409.10070)
• Keywords: language model
• Abstract: Dialogue summarization aims to provide a concise and coherent summary of conversations between multiple speakers. While recent advancements in language models have enhanced this process, summarizing dialogues accurately and faithfully remains challenging due to the need to understand speaker interactions and capture relevant information. Indeed, abstractive models used for dialog summarization may generate summaries that contain inconsistencies. We suggest using the semantic information proposed for performing Spoken Language Understanding (SLU) in human-machine dialogue systems for goal-oriented human-human dialogues to obtain a more semantically faithful summary regarding the task. This study introduces three key contributions: First, we propose an exploration of how incorporating task-related information can enhance the summarization process, leading to more semantically accurate summaries. Then, we introduce a new evaluation criterion based on task semantics. Finally, we propose a new dataset version with increased annotated data standardized for research on task-oriented dialogue summarization. The study evaluates these methods using the DECODA corpus, a collection of French spoken dialogues from a call center. Results show that integrating models with task-related information improves summary accuracy, even with varying word error rates.
• 摘要：对话摘要旨在提供多位说话者之间对话的简洁、连贯的摘要。尽管语言模型的最新进展增强了这一过程，但由于需要了解说话者的互动并捕捉相关信息，准确忠实地总结对话仍然具有挑战性。事实上，用于对话摘要的抽象模型可能会生成包含不一致内容的摘要。我们建议使用在人机对话系统中执行口语理解 (SLU) 时提出的语义信息来进行面向目标的人与人对话，以获得关于该任务的语义上更忠实的摘要。本研究介绍了三个关键贡献：首先，我们提出了一种探索如何结合与任务相关的信息来增强摘要过程，从而产生语义上更准确的摘要。然后，我们引入了一种基于任务语义的新评估标准。最后，我们提出了一个新的数据集版本，其中增加了带注释的数据，该版本标准化了面向任务的对话摘要研究。该研究使用 DECODA 语料库（来自呼叫中心的法语口语对话集合）评估这些方法。结果表明，即使单词错误率有所不同，将模型与任务相关信息相结合也可以提高摘要的准确性。

Title: LLM-DER:A Named Entity Recognition Method Based on Large Language Models for Chinese Coal Chemical Domain

• Authors: Le Xiao, Yunfei Xu, Jing Zhao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.10077
• Pdf URL: https://arxiv.org/pdf/2409.10077
• Copy Paste: [[2409.10077]] LLM-DER:A Named Entity Recognition Method Based on Large Language Models for Chinese Coal Chemical Domain(https://arxiv.org/abs/2409.10077)
• Keywords: language model, gpt, llm
• Abstract: Domain-specific Named Entity Recognition (NER), whose goal is to recognize domain-specific entities and their categories, provides an important support for constructing domain knowledge graphs. Currently, deep learning-based methods are widely used and effective in NER tasks, but due to the reliance on large-scale labeled data. As a result, the scarcity of labeled data in a specific domain will limit its application.Therefore, many researches started to introduce few-shot methods and achieved some results. However, the entity structures in specific domains are often complex, and the current few-shot methods are difficult to adapt to NER tasks with complex features.Taking the Chinese coal chemical industry domain as an example,there exists a complex structure of multiple entities sharing a single entity, as well as multiple relationships for the same pair of entities, which affects the NER task under the sample less this http URL this paper, we propose a Large Language Models (LLMs)-based entity recognition framework LLM-DER for the domain-specific entity recognition problem in Chinese, which enriches the entity information by generating a list of relationships containing entity types through LLMs, and designing a plausibility and consistency evaluation method to remove misrecognized entities, which can effectively solve the complex structural entity recognition problem in a specific domain.The experimental results of this paper on the Resume dataset and the self-constructed coal chemical dataset Coal show that LLM-DER performs outstandingly in domain-specific entity recognition, not only outperforming the existing GPT-3.5-turbo baseline, but also exceeding the fully-supervised baseline, verifying its effectiveness in entity recognition.
• 摘要：领域特定命名实体识别（NER）旨在识别特定领域的实体及其类别，为构建领域知识图谱提供了重要支撑。目前，基于深度学习的方法在NER任务中得到广泛应用且效果显著，但由于对大规模标注数据的依赖，导致特定领域标注数据的稀缺性会限制其应用。因此，许多研究开始引入少样本方法并取得了一些成果。然而特定领域中的实体结构往往非常复杂，目前的少样本方法难以适应复杂特征的NER任务。以中国煤化工领域为例，存在多个实体共享单个实体的复杂结构，以及同一对实体之间存在多种关系，影响了样本较少下的NER任务。本文针对中文领域特定实体识别问题，提出了一种基于大型语言模型（LLM）的实体识别框架LLM-DER，通过LLM生成包含实体类型的关系列表来丰富实体信息，并设计合理性和一致性评估方法去除误识别的实体，可以有效解决特定领域的复杂结构实体识别问题。本文在Resume数据集和自建煤化工数据集Coal上的实验结果表明，LLM-DER在领域特定实体识别中表现优异，不仅超越现有的GPT-3.5-turbo基线，也超过了全监督基线，验证了其有效性在实体识别中。

Title: StruEdit: Structured Outputs Enable the Fast and Accurate Knowledge Editing for Large Language Models

• Authors: Baolong Bi, Shenghua Liu, Yiwei Wang, Lingrui Mei, Hongcheng Gao, Junfeng Fang, Xueqi Cheng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.10132
• Pdf URL: https://arxiv.org/pdf/2409.10132
• Copy Paste: [[2409.10132]] StruEdit: Structured Outputs Enable the Fast and Accurate Knowledge Editing for Large Language Models(https://arxiv.org/abs/2409.10132)
• Keywords: language model, llm, prompt
• Abstract: As the modern tool of choice for question answering, large language models (LLMs) are expected to deliver answers with up-to-date knowledge. To achieve such ideal question-answering systems, locating and then editing outdated knowledge in the natural language outputs is a general target of popular knowledge editing methods. However, this target is challenging, as both identifying which tokens to edit in the reasoning steps and ensuring the coherence of the revised reasoning chain are difficult tasks. We argue that these challenges stem from the unstructured nature of natural language outputs. To address the above challenges, we propose $\textbf{Stru}$ctural $\textbf{Edit}$ing ($\textbf{StruEdit}$), an improved baseline for knowledge editing. We first prompt LLMs to produce structured outputs consisting of reasoning triplets. Then, StruEdit removes any potentially outdated knowledge and efficiently refills the structured outputs with up-to-date information in a single step. Experimental results show that StruEdit consistently delivers the highest accuracy with lowest latency compared with other knowledge editing methods.
• 摘要：作为现代问答工具，大型语言模型 (LLM) 有望提供具有最新知识的答案。为了实现这种理想的问答系统，定位并编辑自然语言输出中的过时知识是流行知识编辑方法的一般目标。然而，这个目标具有挑战性，因为确定在推理步骤中要编辑哪些标记以及确保修订后的推理链的连贯性都是艰巨的任务。我们认为这些挑战源于自然语言输出的非结构化性质。为了应对上述挑战，我们提出了 $\textbf{Stru}$ctural $\textbf{Edit}$ing ($\textbf{StruEdit}$)，这是一种改进的知识编辑基线。我们首先提示 LLM 生成由推理三元组组成的结构化输出。然后，StruEdit 删除任何可能过时的知识，并在一个步骤中用最新信息有效地重新填充结构化输出。实验结果表明，与其他知识编辑方法相比，StruEdit 始终以最低的延迟提供最高的准确率。

Title: LLMs4OL 2024 Overview: The 1st Large Language Models for Ontology Learning Challenge

• Authors: Hamed Babaei Giglou, Jennifer D'Souza, Sören Auer
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.10146
• Pdf URL: https://arxiv.org/pdf/2409.10146
• Copy Paste: [[2409.10146]] LLMs4OL 2024 Overview: The 1st Large Language Models for Ontology Learning Challenge(https://arxiv.org/abs/2409.10146)
• Keywords: language model, llm
• Abstract: This paper outlines the LLMs4OL 2024, the first edition of the Large Language Models for Ontology Learning Challenge. LLMs4OL is a community development initiative collocated with the 23rd International Semantic Web Conference (ISWC) to explore the potential of Large Language Models (LLMs) in Ontology Learning (OL), a vital process for enhancing the web with structured knowledge to improve interoperability. By leveraging LLMs, the challenge aims to advance understanding and innovation in OL, aligning with the goals of the Semantic Web to create a more intelligent and user-friendly web. In this paper, we give an overview of the 2024 edition of the LLMs4OL challenge and summarize the contributions.
• 摘要：本文概述了 LLMs4OL 2024，即第一届大型语言模型本体学习挑战赛。LLMs4OL 是一项社区发展计划，与第 23 届国际语义网会议 (ISWC) 同期举办，旨在探索大型语言模型 (LLM) 在本体学习 (OL) 中的潜力，本体学习 (OL) 是通过结构化知识增强网络以提高互操作性的重要过程。通过利用 LLM，该挑战赛旨在促进对 OL 的理解和创新，与语义网的目标保持一致，以创建更智能、更用户友好的网络。在本文中，我们概述了 2024 年版的 LLMs4OL 挑战赛并总结了贡献。

Title: LLMs for clinical risk prediction

• Authors: Mohamed Rezk, Patricia Cabanillas Silva, Fried-Michael Dahlweid
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.10191
• Pdf URL: https://arxiv.org/pdf/2409.10191
• Copy Paste: [[2409.10191]] LLMs for clinical risk prediction(https://arxiv.org/abs/2409.10191)
• Keywords: language model, gpt, llm
• Abstract: This study compares the efficacy of GPT-4 and clinalytix Medical AI in predicting the clinical risk of delirium development. Findings indicate that GPT-4 exhibited significant deficiencies in identifying positive cases and struggled to provide reliable probability estimates for delirium risk, while clinalytix Medical AI demonstrated superior accuracy. A thorough analysis of the large language model's (LLM) outputs elucidated potential causes for these discrepancies, consistent with limitations reported in extant literature. These results underscore the challenges LLMs face in accurately diagnosing conditions and interpreting complex clinical data. While LLMs hold substantial potential in healthcare, they are currently unsuitable for independent clinical decision-making. Instead, they should be employed in assistive roles, complementing clinical expertise. Continued human oversight remains essential to ensure optimal outcomes for both patients and healthcare providers.
• 摘要：本研究比较了 GPT-4 和 clinalytix Medical AI 在预测谵妄临床风险方面的效果。研究结果表明，GPT-4 在识别阳性病例方面表现出重大缺陷，并且难以提供可靠的谵妄风险概率估计，而 clinalytix Medical AI 则表现出卓越的准确性。对大型语言模型 (LLM) 输出的彻底分析阐明了这些差异的潜在原因，与现有文献中报告的局限性一致。这些结果强调了 LLM 在准确诊断病情和解释复杂临床数据方面面临的挑战。虽然 LLM 在医疗保健领域具有巨大潜力，但它们目前不适合独立的临床决策。相反，它们应该被用作辅助角色，补充临床专业知识。持续的人为监督仍然至关重要，以确保患者和医疗保健提供者获得最佳结果。

Title: From Text to Emoji: How PEFT-Driven Personality Manipulation Unleashes the Emoji Potential in LLMs

• Authors: Navya Jain, Zekun Wu, Cristian Munoz, Airlie Hilliard, Adriano Koshiyama, Emre Kazim, Philip Treleaven
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.10245
• Pdf URL: https://arxiv.org/pdf/2409.10245
• Copy Paste: [[2409.10245]] From Text to Emoji: How PEFT-Driven Personality Manipulation Unleashes the Emoji Potential in LLMs(https://arxiv.org/abs/2409.10245)
• Keywords: llm, prompt, chat
• Abstract: As the demand for human-like interactions with LLMs continues to grow, so does the interest in manipulating their personality traits, which has emerged as a key area of research. Methods like prompt-based In-Context Knowledge Editing (IKE) and gradient-based Model Editor Networks (MEND) have been explored but show irregularity and variability. IKE depends on the prompt, leading to variability and sensitivity, while MEND yields inconsistent and gibberish outputs. To address this, we employed Opinion QA Based Parameter-Efficient Fine-Tuning (PEFT), specifically Quantized Low-Rank Adaptation (QLORA), to manipulate the Big Five personality traits: Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism. After PEFT, models such as Mistral-7B-Instruct and Llama-2-7B-chat began generating emojis, despite their absence in the PEFT data. For instance, Llama-2-7B-chat generated emojis in 99.5% of extraversion-related test instances, while Mistral-8B-Instruct did so in 92.5% of openness-related test instances. Explainability analysis indicated that the LLMs used emojis intentionally to express these traits. This paper provides a number of novel contributions. First, introducing an Opinion QA dataset for PEFT-driven personality manipulation; second, developing metric models to benchmark LLM personality traits; third, demonstrating PEFT's superiority over IKE in personality manipulation; and finally, analyzing and validating emoji usage through explainability methods such as mechanistic interpretability and in-context learning explainability methods.
• 摘要：随着对 LLM 进行类似人类交互的需求不断增长，人们对操纵其性格特征的兴趣也日益浓厚，这已成为一个关键的研究领域。人们已经探索了基于提示的上下文知识编辑 (IKE) 和基于梯度的模型编辑器网络 (MEND) 等方法，但这些方法表现出不规则性和多变性。IKE 取决于提示，导致多变性和敏感性，而 MEND 会产生不一致和乱码的输出。为了解决这个问题，我们采用了基于意见 QA 的参数高效微调 (PEFT)，特别是量化低秩自适应 (QLORA)，来操​​纵五大性格特征：开放性、尽责性、外向性、亲和性和神经质。在 PEFT 之后，Mistral-7B-Instruct 和 Llama-2-7B-chat 等模型开始生成表情符号，尽管它们在 PEFT 数据中不存在。例如，Llama-2-7B-chat 在 99.5% 的外向性相关测试实例中生成表情符号，而 Mistral-8B-Instruct 在 92.5% 的开放性相关测试实例中生成表情符号。可解释性分析表明，LLM 有意使用表情符号来表达这些特征。本文提供了许多新颖的贡献。首先，引入了一个用于 PEFT 驱动的人格操纵的意见问答数据集；其次，开发度量模型来对 LLM 人格特质进行基准测试；第三，证明 PEFT 在人格操纵方面优于 IKE；最后，通过可解释性方法（如机械可解释性和情境学习可解释性方法）分析和验证表情符号的使用。

Title: MGSA: Multi-granularity Graph Structure Attention for Knowledge Graph-to-Text Generation

• Authors: Shanshan Wang, Chun Zhang, Ning Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.10294
• Pdf URL: https://arxiv.org/pdf/2409.10294
• Copy Paste: [[2409.10294]] MGSA: Multi-granularity Graph Structure Attention for Knowledge Graph-to-Text Generation(https://arxiv.org/abs/2409.10294)
• Keywords: language model
• Abstract: The Knowledge Graph-to-Text Generation task aims to convert structured knowledge graphs into coherent and human-readable natural language text. Recent efforts in this field have focused on enhancing pre-trained language models (PLMs) by incorporating graph structure information to capture the intricate structure details of knowledge graphs. However, most of these approaches tend to capture only single-granularity structure information, concentrating either on the relationships between entities within the original graph or on the relationships between words within the same entity or across different entities. This narrow focus results in a significant limitation: models that concentrate solely on entity-level structure fail to capture the nuanced semantic relationships between words, while those that focus only on word-level structure overlook the broader relationships between original entire entities. To overcome these limitations, this paper introduces the Multi-granularity Graph Structure Attention (MGSA), which is based on PLMs. The encoder of the model architecture features an entity-level structure encoding module, a word-level structure encoding module, and an aggregation module that synthesizes information from both structure. This multi-granularity structure encoding approach allows the model to simultaneously capture both entity-level and word-level structure information, providing a more comprehensive understanding of the knowledge graph's structure information, thereby significantly improving the quality of the generated text. We conducted extensive evaluations of the MGSA model using two widely recognized KG-to-Text Generation benchmark datasets, WebNLG and EventNarrative, where it consistently outperformed models that rely solely on single-granularity structure information, demonstrating the effectiveness of our approach.
• 摘要：知识图谱到文本生成任务旨在将结构化知识图谱转换为连贯且人类可读的自然语言文本。该领域最近的努力集中在通过结合图谱结构信息来捕获知识图谱的复杂结构细节，从而增强预训练语言模型 (PLM)。然而，这些方法中的大多数往往只捕获单粒度结构信息，要么集中于原始图谱中实体之间的关系，要么集中于同一实体内或不同实体之间的单词之间的关系。这种狭窄的关注点导致了一个重大的局限性：只关注实体级结构的模型无法捕捉单词之间细微的语义关系，而只关注单词级结构的模型则忽略了原始整个实体之间的更广泛关系。为了克服这些限制，本文介绍了基于 PLM 的多粒度图结构注意 (MGSA)。模型架构的编码器具有实体级结构编码模块、单词级结构编码模块和综合两种结构信息的聚合模块。这种多粒度结构编码方法使模型能够同时捕获实体级和词级结构信息，从而更全面地了解知识图谱的结构信息，从而显著提高生成文本的质量。我们使用两个广受认可的 KG-to-Text Generation 基准数据集 WebNLG 和 EventNarrative 对 MGSA 模型进行了广泛的评估，结果表明，它的表现始终优于仅依赖单粒度结构信息的模型，证明了我们方法的有效性。

Title: The 20 questions game to distinguish large language models

• Authors: Gurvan Richardeau, Erwan Le Merrer, Camilla Penzo, Gilles Tredan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.10338
• Pdf URL: https://arxiv.org/pdf/2409.10338
• Copy Paste: [[2409.10338]] The 20 questions game to distinguish large language models(https://arxiv.org/abs/2409.10338)
• Keywords: language model, llm
• Abstract: In a parallel with the 20 questions game, we present a method to determine whether two large language models (LLMs), placed in a black-box context, are the same or not. The goal is to use a small set of (benign) binary questions, typically under 20. We formalize the problem and first establish a baseline using a random selection of questions from known benchmark datasets, achieving an accuracy of nearly 100% within 20 questions. After showing optimal bounds for this problem, we introduce two effective questioning heuristics able to discriminate 22 LLMs by using half as many questions for the same task. These methods offer significant advantages in terms of stealth and are thus of interest to auditors or copyright owners facing suspicions of model leaks.
• 摘要：与 20 个问题游戏类似，我们提出了一种方法来确定两个大型语言模型 (LLM)（放置在黑盒环境中）是否相同。目标是使用一小组（良性）二元问题，通常少于 20 个。我们将问题形式化，并首先使用从已知基准数据集中随机选择的问题建立基线，在 20 个问题内实现接近 100% 的准确率。在展示此问题的最佳界限之后，我们引入了两种有效的提问启发式方法，能够通过使用一半的问题来区分 22 个 LLM，完成相同的任务。这些方法在隐蔽性方面具有显著优势，因此对于面临模型泄露怀疑的审计师或版权所有者来说很有吸引力。

Title: A Knowledge-Enhanced Disease Diagnosis Method Based on Prompt Learning and BERT Integration

• Authors: Zhang Zheng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.10403
• Pdf URL: https://arxiv.org/pdf/2409.10403
• Copy Paste: [[2409.10403]] A Knowledge-Enhanced Disease Diagnosis Method Based on Prompt Learning and BERT Integration(https://arxiv.org/abs/2409.10403)
• Keywords: language model, prompt
• Abstract: This paper proposes a knowledge-enhanced disease diagnosis method based on a prompt learning framework. The method retrieves structured knowledge from external knowledge graphs related to clinical cases, encodes it, and injects it into the prompt templates to enhance the language model's understanding and reasoning capabilities for the task.We conducted experiments on three public datasets: CHIP-CTC, IMCS-V2-NER, and KUAKE-QTR. The results show that the proposed method significantly outperforms existing models across multiple evaluation metrics, with an F1 score improvement of 2.4% on the CHIP-CTC dataset, 3.1% on the IMCS-V2-NER dataset,and 4.2% on the KUAKE-QTR dataset. Additionally,ablation studies confirmed the critical role of the knowledge injection module,as the removal of this module resulted in a significant drop in F1 score. The experimental results demonstrate that the proposed method not only effectively improves the accuracy of disease diagnosis but also enhances the interpretability of the predictions, providing more reliable support and evidence for clinical diagnosis.
• 摘要：本文提出了一种基于提示学习框架的知识增强型疾病诊断方法。该方法从与临床病例相关的外部知识图谱中检索结构化知识，对其进行编码，并将其注入到提示模板中，以增强语言模型对该任务的理解和推理能力。我们在三个公共数据集上进行了实验：CHIP-CTC、IMCS-V2-NER和KUAKE-QTR。结果表明，所提方法在多个评估指标上显著优于现有模型，在CHIP-CTC数据集上的F1得分提高了2.4%，在IMCS-V2-NER数据集上提高了3.1%，在KUAKE-QTR数据集上提高了4.2%。此外，消融研究证实了知识注入模块的关键作用，因为删除该模块导致F1得分大幅下降。实验结果表明，所提方法不仅有效提高了疾病诊断的准确性，而且增强了预测的可解释性，为临床诊断提供了更可靠的支持和证据。

Title: Schrodinger's Memory: Large Language Models

• Authors: Wei Wang, Qing Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.10482
• Pdf URL: https://arxiv.org/pdf/2409.10482
• Copy Paste: [[2409.10482]] Schrodinger's Memory: Large Language Models(https://arxiv.org/abs/2409.10482)
• Keywords: language model, llm
• Abstract: Memory is the foundation of LLMs' functionality, yet past research has lacked an in-depth exploration of their memory capabilities and underlying theory. In this paper, we apply UAT theory to explain the memory mechanism of LLMs and propose a new approach for evaluating LLM performance by comparing the memory capacities of different models. Through extensive experiments, we validate our theory and the memory abilities of LLMs. Finally, we compare the capabilities of the human brain and LLMs, highlighting both their similarities and differences in terms of working mechanisms.
• 摘要：记忆是 LLM 实现功能的基础，但以往的研究对 LLM 的记忆能力和理论基础缺乏深入的探讨。本文利用 UAT 理论解释 LLM 的记忆机制，并通过比较不同模型的记忆能力提出了一种评估 LLM 性能的新方法。通过大量实验，我们验证了我们的理论和 LLM 的记忆能力。最后，我们比较了人脑和 LLM 的能力，突出了它们在工作机制方面的相似之处和不同之处。

Title: DILA: Dictionary Label Attention for Mechanistic Interpretability in High-dimensional Multi-label Medical Coding Prediction

• Authors: John Wu, David Wu, Jimeng Sun
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.10504
• Pdf URL: https://arxiv.org/pdf/2409.10504
• Copy Paste: [[2409.10504]] DILA: Dictionary Label Attention for Mechanistic Interpretability in High-dimensional Multi-label Medical Coding Prediction(https://arxiv.org/abs/2409.10504)
• Keywords: language model, llm
• Abstract: Predicting high-dimensional or extreme multilabels, such as in medical coding, requires both accuracy and interpretability. Existing works often rely on local interpretability methods, failing to provide comprehensive explanations of the overall mechanism behind each label prediction within a multilabel set. We propose a mechanistic interpretability module called DIctionary Label Attention (\method) that disentangles uninterpretable dense embeddings into a sparse embedding space, where each nonzero element (a dictionary feature) represents a globally learned medical concept. Through human evaluations, we show that our sparse embeddings are more human understandable than its dense counterparts by at least 50 percent. Our automated dictionary feature identification pipeline, leveraging large language models (LLMs), uncovers thousands of learned medical concepts by examining and summarizing the highest activating tokens for each dictionary feature. We represent the relationships between dictionary features and medical codes through a sparse interpretable matrix, enhancing the mechanistic and global understanding of the model's predictions while maintaining competitive performance and scalability without extensive human annotation.
• 摘要：预测高维或极端多标签（例如医学编码）需要准确性和可解释性。现有研究通常依赖于局部可解释性方法，无法全面解释多标签集内每个标签预测背后的整体机制。我们提出了一个称为字典标签注意（\method）的机械可解释性模块，它将无法解释的密集嵌入分解为稀疏嵌入空间，其中每个非零元素（字典特征）代表一个全局学习的医学概念。通过人工评估，我们表明我们的稀疏嵌入比其密集对应物至少高出 50% 更易于人类理解。我们的自动字典特征识别管道利用大型语言模型 (LLM)，通过检查和总结每个字典特征的最高激活标记来发现数千个学习的医学概念。我们通过稀疏可解释矩阵来表示词典特征和医学代码之间的关系，增强了对模型预测的机械和整体理解，同时无需大量人工注释即可保持竞争性能和可扩展性。