2024-04-16

Title: Optimal path for Biomedical Text Summarization Using Pointer GPT

• Authors: Hyunkyung Han, Jaesik Choi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.08654
• Pdf URL: https://arxiv.org/pdf/2404.08654
• Copy Paste: [[2404.08654]] Optimal path for Biomedical Text Summarization Using Pointer GPT(https://arxiv.org/abs/2404.08654)
• Keywords: gpt
• Abstract: Biomedical text summarization is a critical tool that enables clinicians to effectively ascertain patient status. Traditionally, text summarization has been accomplished with transformer models, which are capable of compressing long documents into brief summaries. However, transformer models are known to be among the most challenging natural language processing (NLP) tasks. Specifically, GPT models have a tendency to generate factual errors, lack context, and oversimplify words. To address these limitations, we replaced the attention mechanism in the GPT model with a pointer network. This modification was designed to preserve the core values of the original text during the summarization process. The effectiveness of the Pointer-GPT model was evaluated using the ROUGE score. The results demonstrated that Pointer-GPT outperformed the original GPT model. These findings suggest that pointer networks can be a valuable addition to EMR systems and can provide clinicians with more accurate and informative summaries of patient medical records. This research has the potential to usher in a new paradigm in EMR systems and to revolutionize the way that clinicians interact with patient medical records.
• 摘要：生物医学文本摘要是临床医生能够有效确定患者状态的重要工具。传统上，文本摘要是通过转换器模型完成的，该模型能够将长文档压缩为简短的摘要。然而，Transformer 模型被认为是最具挑战性的自然语言处理 (NLP) 任务之一。具体来说，GPT 模型容易产生事实错误、缺乏上下文以及过度简化单词。为了解决这些限制，我们用指针网络替换了 GPT 模型中的注意力机制。这一修改旨在在摘要过程中保留原文的核心价值。使用 ROUGE 评分评估 Pointer-GPT 模型的有效性。结果表明，Pointer-GPT 优于原始 GPT 模型。这些发现表明，指针网络可以成为 EMR 系统的一个有价值的补充，并且可以为临床医生提供更准确、信息更丰富的患者病历摘要。这项研究有可能开创电子病历系统的新范式，并彻底改变临床医生与患者病历交互的方式。

Title: Linear Cross-document Event Coreference Resolution with X-AMR

• Authors: Shafiuddin Rehan Ahmed, George Arthur Baker, Evi Judge, Michael Regan, Kristin Wright-Bettner, Martha Palmer, James H. Martin
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.08656
• Pdf URL: https://arxiv.org/pdf/2404.08656
• Copy Paste: [[2404.08656]] Linear Cross-document Event Coreference Resolution with X-AMR(https://arxiv.org/abs/2404.08656)
• Keywords: language model, gpt, llm, prompt
• Abstract: Event Coreference Resolution (ECR) as a pairwise mention classification task is expensive both for automated systems and manual annotations. The task's quadratic difficulty is exacerbated when using Large Language Models (LLMs), making prompt engineering for ECR prohibitively costly. In this work, we propose a graphical representation of events, X-AMR, anchored around individual mentions using a \textbf{cross}-document version of \textbf{A}bstract \textbf{M}eaning \textbf{R}epresentation. We then linearize the ECR with a novel multi-hop coreference algorithm over the event graphs. The event graphs simplify ECR, making it a) LLM cost-effective, b) compositional and interpretable, and c) easily annotated. For a fair assessment, we first enrich an existing ECR benchmark dataset with these event graphs using an annotator-friendly tool we introduce. Then, we employ GPT-4, the newest LLM by OpenAI, for these annotations. Finally, using the ECR algorithm, we assess GPT-4 against humans and analyze its limitations. Through this research, we aim to advance the state-of-the-art for efficient ECR and shed light on the potential shortcomings of current LLMs at this task. Code and annotations: \url{https://github.com/ahmeshaf/gpt_coref}
• 摘要：事件共指解析（ECR）作为成对提及分类任务对于自动化系统和手动注释来说都是昂贵的。使用大型语言模型 (LLM) 时，任务的二次难度会加剧，从而导致 ECR 的即时工程成本高昂。在这项工作中，我们提出了事件的图形表示，X-AMR，使用 \textbf{A}bstract \textbf{M}eaning \textbf{R} 表示的 \textbf{cross} 文档版本锚定在各个提及上。然后，我们在事件图上使用新颖的多跳共指算法对 ECR 进行线性化。事件图简化了 ECR，使其 a) LLM 具有成本效益，b) 组合且可解释，c) 易于注释。为了进行公平的评估，我们首先使用我们引入的注释器友好工具通过这些事件图丰富现有的 ECR 基准数据集。然后，我们使用 OpenAI 最新的 LLM GPT-4 来进行这些注释。最后，使用 ECR 算法，我们针对人类评估了 GPT-4 并分析了其局限性。通过这项研究，我们的目标是推进高效 ECR 的最先进水平，并揭示当前法学硕士在这项任务中的潜在缺点。代码和注释：\url{https://github.com/ahmeshaf/gpt_coref}

Title: Revealing Trends in Datasets from the 2022 ACL and EMNLP Conferences

• Authors: Jesse Atuhurra, Hidetaka Kamigaito
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.08666
• Pdf URL: https://arxiv.org/pdf/2404.08666
• Copy Paste: [[2404.08666]] Revealing Trends in Datasets from the 2022 ACL and EMNLP Conferences(https://arxiv.org/abs/2404.08666)
• Keywords: language model, prompt
• Abstract: Natural language processing (NLP) has grown significantly since the advent of the Transformer architecture. Transformers have given birth to pre-trained large language models (PLMs). There has been tremendous improvement in the performance of NLP systems across several tasks. NLP systems are on par or, in some cases, better than humans at accomplishing specific tasks. However, it remains the norm that \emph{better quality datasets at the time of pretraining enable PLMs to achieve better performance, regardless of the task.} The need to have quality datasets has prompted NLP researchers to continue creating new datasets to satisfy particular needs. For example, the two top NLP conferences, ACL and EMNLP, accepted ninety-two papers in 2022, introducing new datasets. This work aims to uncover the trends and insights mined within these datasets. Moreover, we provide valuable suggestions to researchers interested in curating datasets in the future.
• 摘要：自 Transformer 架构出现以来，自然语言处理 (NLP) 得到了显着发展。 Transformers 催生了预先训练的大型语言模型 (PLM)。 NLP 系统在多项任务上的性能有了巨大的提高。 NLP 系统在完成特定任务方面与人类相当，或者在某些情况下比人类更好。然而，\emph{无论任务如何，预训练时更高质量的数据集都可以使 PLM 获得更好的性能，这仍然是常态。}对高质量数据集的需求促使 NLP 研究人员继续创建新的数据集来满足特定需求。例如，两个顶级 NLP 会议 ACL 和 EMNLP 在 2022 年接受了 92 篇论文，引入了新的数据集。这项工作旨在揭示这些数据集中挖掘的趋势和见解。此外，我们还为有兴趣在未来管理数据集的研究人员提供宝贵的建议。

Title: Sentiment analysis and random forest to classify LLM versus human source applied to Scientific Texts

• Authors: Javier J. Sanchez-Medina
• Subjects: cs.CL, cs.AI, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.08673
• Pdf URL: https://arxiv.org/pdf/2404.08673
• Copy Paste: [[2404.08673]] Sentiment analysis and random forest to classify LLM versus human source applied to Scientific Texts(https://arxiv.org/abs/2404.08673)
• Keywords: gpt, llm, chat
• Abstract: After the launch of ChatGPT v.4 there has been a global vivid discussion on the ability of this artificial intelligence powered platform and some other similar ones for the automatic production of all kinds of texts, including scientific and technical texts. This has triggered a reflection in many institutions on whether education and academic procedures should be adapted to the fact that in future many texts we read will not be written by humans (students, scholars, etc.), at least, not entirely. In this work it is proposed a new methodology to classify texts coming from an automatic text production engine or a human, based on Sentiment Analysis as a source for feature engineering independent variables and then train with them a Random Forest classification algorithm. Using four different sentiment lexicons, a number of new features where produced, and then fed to a machine learning random forest methodology, to train such a model. Results seem very convincing that this may be a promising research line to detect fraud, in such environments where human are supposed to be the source of texts.
• 摘要：ChatGPT v.4 推出后，全球范围内就这个人工智能驱动的平台和其他一些类似平台自动生成各种文本（包括科技文本）的能力展开了热烈的讨论。这引发了许多机构的反思，即教育和学术程序是否应该适应这样一个事实：未来我们阅读的许多文本至少不是完全由人类（学生、学者等）撰写。在这项工作中，提出了一种新的方法来对来自自动文本生成引擎或人类的文本进行分类，基于情感分析作为特征工程自变量的来源，然后用它们训练随机森林分类算法。使用四种不同的情感词典，生成许多新特征，然后将其输入机器学习随机森林方法来训练这样的模型。结果似乎非常令人信服，在人类被认为是文本来源的环境中，这可能是检测欺诈的有前途的研究方向。

Title: Effects of Different Prompts on the Quality of GPT-4 Responses to Dementia Care Questions

• Authors: Zhuochun Li, Bo Xie, Robin Hilsabeck, Alyssa Aguirre, Ning Zou, Zhimeng Luo, Daqing He
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2404.08674
• Pdf URL: https://arxiv.org/pdf/2404.08674
• Copy Paste: [[2404.08674]] Effects of Different Prompts on the Quality of GPT-4 Responses to Dementia Care Questions(https://arxiv.org/abs/2404.08674)
• Keywords: language model, gpt, llm, prompt
• Abstract: Evidence suggests that different prompts lead large language models (LLMs) to generate responses with varying quality. Yet, little is known about prompts' effects on response quality in healthcare domains. In this exploratory study, we address this gap, focusing on a specific healthcare domain: dementia caregiving. We first developed an innovative prompt template with three components: (1) system prompts (SPs) featuring 4 different roles; (2) an initialization prompt; and (3) task prompts (TPs) specifying different levels of details, totaling 12 prompt combinations. Next, we selected 3 social media posts containing complicated, real-world questions about dementia caregivers' challenges in 3 areas: memory loss and confusion, aggression, and driving. We then entered these posts into GPT-4, with our 12 prompts, to generate 12 responses per post, totaling 36 responses. We compared the word count of the 36 responses to explore potential differences in response length. Two experienced dementia care clinicians on our team assessed the response quality using a rating scale with 5 quality indicators: factual, interpretation, application, synthesis, and comprehensiveness (scoring range: 0-5; higher scores indicate higher quality).
• 摘要：有证据表明，不同的提示会导致大型语言模型 (LLM) 生成不同质量的响应。然而，人们对提示对医疗保健领域响应质量的影响知之甚少。在这项探索性研究中，我们解决了这一差距，重点关注特定的医疗保健领域：痴呆症护理。我们首先开发了一个创新的提示模板，包含三个组件：（1）具有 4 个不同角色的系统提示（SP）； (2)初始化提示； (3)指定不同细节级别的任务提示(TP)，总共12种提示组合。接下来，我们选择了 3 个社交媒体帖子，其中包含有关痴呆症护理人员在 3 个领域面临的挑战的复杂现实问题：记忆丧失和混乱、攻击性和驾驶。然后，我们将这些帖子输入 GPT-4，并带有 12 个提示，为每个帖子生成 12 个回复，总共 36 个回复。我们比较了 36 个回复的字数，以探索回复长度的潜在差异。我们团队中的两位经验丰富的痴呆症护理临床医生使用具有 5 个质量指标的评级量表评估了响应质量：事实、解释、应用、综合和综合性（评分范围：0-5；分数越高表示质量越高）。

Title: ALERT: A Comprehensive Benchmark for Assessing Large Language Models' Safety through Red Teaming

• Authors: Simone Tedeschi, Felix Friedrich, Patrick Schramowski, Kristian Kersting, Roberto Navigli, Huu Nguyen, Bo Li
• Subjects: cs.CL, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.08676
• Pdf URL: https://arxiv.org/pdf/2404.08676
• Copy Paste: [[2404.08676]] ALERT: A Comprehensive Benchmark for Assessing Large Language Models' Safety through Red Teaming(https://arxiv.org/abs/2404.08676)
• Keywords: language model, llm
• Abstract: When building Large Language Models (LLMs), it is paramount to bear safety in mind and protect them with guardrails. Indeed, LLMs should never generate content promoting or normalizing harmful, illegal, or unethical behavior that may contribute to harm to individuals or society. This principle applies to both normal and adversarial use. In response, we introduce ALERT, a large-scale benchmark to assess safety based on a novel fine-grained risk taxonomy. It is designed to evaluate the safety of LLMs through red teaming methodologies and consists of more than 45k instructions categorized using our novel taxonomy. By subjecting LLMs to adversarial testing scenarios, ALERT aims to identify vulnerabilities, inform improvements, and enhance the overall safety of the language models. Furthermore, the fine-grained taxonomy enables researchers to perform an in-depth evaluation that also helps one to assess the alignment with various policies. In our experiments, we extensively evaluate 10 popular open- and closed-source LLMs and demonstrate that many of them still struggle to attain reasonable levels of safety.
• 摘要：在构建大型语言模型（LLM）时，最重要的是要牢记安全并用护栏保护它们。事实上，法学硕士绝不能生成宣扬或规范可能对个人或社会造成伤害的有害、非法或不道德行为的内容。该原则适用于正常使用和对抗性使用。为此，我们引入了 ALERT，这是一种基于新颖的细粒度风险分类法来评估安全性的大型基准。它旨在通过红队方法评估法学硕士的安全性，由使用我们新颖的分类法分类的超过 45,000 条指令组成。通过让法学硕士接受对抗性测试场景，ALERT 旨在识别漏洞、通知改进并增强语言模型的整体安全性。此外，细粒度的分类法使研究人员能够进行深入的评估，这也有助于评估与各种政策的一致性。在我们的实验中，我们广泛评估了 10 个流行的开源和闭源法学硕士，并证明其中许多仍然难以达到合理的安全水平。

Title: Your Finetuned Large Language Model is Already a Powerful Out-of-distribution Detector

• Authors: Andi Zhang, Tim Z. Xiao, Weiyang Liu, Robert Bamler, Damon Wischik
• Subjects: cs.CL, cs.AI, cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2404.08679
• Pdf URL: https://arxiv.org/pdf/2404.08679
• Copy Paste: [[2404.08679]] Your Finetuned Large Language Model is Already a Powerful Out-of-distribution Detector(https://arxiv.org/abs/2404.08679)
• Keywords: language model, llm
• Abstract: We revisit the likelihood ratio between a pretrained large language model (LLM) and its finetuned variant as a criterion for out-of-distribution (OOD) detection. The intuition behind such a criterion is that, the pretrained LLM has the prior knowledge about OOD data due to its large amount of training data, and once finetuned with the in-distribution data, the LLM has sufficient knowledge to distinguish their difference. Leveraging the power of LLMs, we show that, for the first time, the likelihood ratio can serve as an effective OOD detector. Moreover, we apply the proposed LLM-based likelihood ratio to detect OOD questions in question-answering (QA) systems, which can be used to improve the performance of specialized LLMs for general questions. Given that likelihood can be easily obtained by the loss functions within contemporary neural network frameworks, it is straightforward to implement this approach in practice. Since both the pretrained LLMs and its various finetuned models are available, our proposed criterion can be effortlessly incorporated for OOD detection without the need for further training. We conduct comprehensive evaluation across on multiple settings, including far OOD, near OOD, spam detection, and QA scenarios, to demonstrate the effectiveness of the method.
• 摘要：我们重新审视预训练的大语言模型 (LLM) 及其微调变体之间的似然比，作为分布外 (OOD) 检测的标准。这一标准背后的直觉是，预训练的LLM由于其大量的训练数据而具有关于OOD数据的先验知识，并且一旦与分布内数据进行微调，LLM就有足够的知识来区分它们的差异。利用 LLM 的力量，我们首次证明似然比可以作为有效的 OOD 检测器。此外，我们应用所提出的基于 LLM 的似然比来检测问答（QA）系统中的 OOD 问题，这可用于提高专门 LLM 对一般问题的性能。鉴于可以通过当代神经网络框架内的损失函数轻松获得似然度，因此在实践中实现这种方法很简单。由于预训练的 LLM 及其各种微调模型均可用，因此我们提出的标准可以轻松地纳入 OOD 检测，而无需进一步训练。我们对多种设置进行了综合评估，包括远 OOD、近 OOD、垃圾邮件检测和 QA 场景，以证明该方法的有效性。

Title: Automating Research Synthesis with Domain-Specific Large Language Model Fine-Tuning

• Authors: Teo Susnjak, Peter Hwang, Napoleon H. Reyes, Andre L. C. Barczak, Timothy R. McIntosh, Surangika Ranathunga
• Subjects: cs.CL, cs.DL, cs.IR
• Abstract URL: https://arxiv.org/abs/2404.08680
• Pdf URL: https://arxiv.org/pdf/2404.08680
• Copy Paste: [[2404.08680]] Automating Research Synthesis with Domain-Specific Large Language Model Fine-Tuning(https://arxiv.org/abs/2404.08680)
• Keywords: language model, llm, hallucination
• Abstract: This research pioneers the use of fine-tuned Large Language Models (LLMs) to automate Systematic Literature Reviews (SLRs), presenting a significant and novel contribution in integrating AI to enhance academic research methodologies. Our study employed the latest fine-tuning methodologies together with open-sourced LLMs, and demonstrated a practical and efficient approach to automating the final execution stages of an SLR process that involves knowledge synthesis. The results maintained high fidelity in factual accuracy in LLM responses, and were validated through the replication of an existing PRISMA-conforming SLR. Our research proposed solutions for mitigating LLM hallucination and proposed mechanisms for tracking LLM responses to their sources of information, thus demonstrating how this approach can meet the rigorous demands of scholarly research. The findings ultimately confirmed the potential of fine-tuned LLMs in streamlining various labor-intensive processes of conducting literature reviews. Given the potential of this approach and its applicability across all research domains, this foundational study also advocated for updating PRISMA reporting guidelines to incorporate AI-driven processes, ensuring methodological transparency and reliability in future SLRs. This study broadens the appeal of AI-enhanced tools across various academic and research fields, setting a new standard for conducting comprehensive and accurate literature reviews with more efficiency in the face of ever-increasing volumes of academic studies.
• 摘要：这项研究开创了使用微调大型语言模型（LLM）来自动化系统文献综述（SLR）的先河，在整合人工智能以增强学术研究方法方面做出了重大而新颖的贡献。我们的研究采用了最新的微调方法和开源法学硕士，并展示了一种实用且有效的方法来自动化涉及知识合成的 SLR 流程的最终执行阶段。结果在 LLM 回答中保持了事实准确性的高保真度，并通过复制现有的符合 PRISMA 的 SLR 进行了验证。我们的研究提出了减轻法学硕士幻觉的解决方案，并提出了跟踪法学硕士对其信息来源的反应的机制，从而证明了这种方法如何满足学术研究的严格要求。研究结果最终证实了经过微调的法学硕士在简化进行文献综述的各种劳动密集型流程方面的潜力。鉴于这种方法的潜力及其在所有研究领域的适用性，这项基础研究还主张更新 PRISMA 报告指南，以纳入人工智能驱动的流程，确保未来 SLR 的方法透明度和可靠性。这项研究扩大了人工智能增强工具在各个学术和研究领域的吸引力，为面对不断增加的学术研究量而更高效地进行全面、准确的文献综述制定了新标准。

Title: EFSA: Towards Event-Level Financial Sentiment Analysis

• Authors: Tianyu Chen, Yiming Zhang, Guoxin Yu, Dapeng Zhang, Li Zeng, Qing He, Xiang Ao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.08681
• Pdf URL: https://arxiv.org/pdf/2404.08681
• Copy Paste: [[2404.08681]] EFSA: Towards Event-Level Financial Sentiment Analysis(https://arxiv.org/abs/2404.08681)
• Keywords: llm, chain-of-thought
• Abstract: In this paper, we extend financial sentiment analysis~(FSA) to event-level since events usually serve as the subject of the sentiment in financial text. Though extracting events from the financial text may be conducive to accurate sentiment predictions, it has specialized challenges due to the lengthy and discontinuity of events in a financial text. To this end, we reconceptualize the event extraction as a classification task by designing a categorization comprising coarse-grained and fine-grained event categories. Under this setting, we formulate the \textbf{E}vent-Level \textbf{F}inancial \textbf{S}entiment \textbf{A}nalysis~(\textbf{EFSA} for short) task that outputs quintuples consisting of (company, industry, coarse-grained event, fine-grained event, sentiment) from financial text. A large-scale Chinese dataset containing $12,160$ news articles and $13,725$ quintuples is publicized as a brand new testbed for our task. A four-hop Chain-of-Thought LLM-based approach is devised for this task. Systematically investigations are conducted on our dataset, and the empirical results demonstrate the benchmarking scores of existing methods and our proposed method can reach the current state-of-the-art. Our dataset and framework implementation are available at https://anonymous.4open.science/r/EFSA-645E
• 摘要：在本文中，我们将金融情绪分析（FSA）扩展到事件层面，因为事件通常作为金融文本中情绪的主题。尽管从金融文本中提取事件可能有利于准确的情绪预测，但由于金融文本中事件的冗长和不连续性，它面临着特殊的挑战。为此，我们通过设计包含粗粒度和细粒度事件类别的分类，将事件提取重新概念化为分类任务。在此设置下，我们制定 \textbf{E}vent-Level \textbf{F}inancial \textbf{S}entiment \textbf{A}naanalysis~（简称 \textbf{EFSA}）任务，输出由 (公司、行业、粗粒度事件、细粒度事件、情绪）来自金融文本。包含 12,160 美元新闻文章和 13,725 美元五元组的大型中文数据集被公布为我们任务的全新测试平台。为此任务设计了基于 LLM 的四跳思想链方法。对我们的数据集进行了系统的研究，实证结果证明了现有方法的基准分数，并且我们提出的方法可以达到当前的最先进水平。我们的数据集和框架实现可在 https://anonymous.4open.science/r/EFSA-645E 获取

Title: Is English the New Programming Language? How About Pseudo-code Engineering?

• Authors: Gian Alexandre Michaelsen, Renato P. dos Santos
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2404.08684
• Pdf URL: https://arxiv.org/pdf/2404.08684
• Copy Paste: [[2404.08684]] Is English the New Programming Language? How About Pseudo-code Engineering?(https://arxiv.org/abs/2404.08684)
• Keywords: language model, gpt, prompt, chat
• Abstract: Background: The integration of artificial intelligence (AI) into daily life, particularly through chatbots utilizing natural language processing (NLP), presents both revolutionary potential and unique challenges. This intended to investigate how different input forms impact ChatGPT, a leading language model by OpenAI, performance in understanding and executing complex, multi-intention tasks. Design: Employing a case study methodology supplemented by discourse analysis, the research analyzes ChatGPT's responses to inputs varying from natural language to pseudo-code engineering. The study specifically examines the model's proficiency across four categories: understanding of intentions, interpretability, completeness, and creativity. Setting and Participants: As a theoretical exploration of AI interaction, this study focuses on the analysis of structured and unstructured inputs processed by ChatGPT, without direct human participants. Data collection and analysis: The research utilizes synthetic case scenarios, including the organization of a "weekly meal plan" and a "shopping list," to assess ChatGPT's response to prompts in both natural language and pseudo-code engineering. The analysis is grounded in the identification of patterns, contradictions, and unique response elements across different input formats. Results: Findings reveal that pseudo-code engineering inputs significantly enhance the clarity and determinism of ChatGPT's responses, reducing ambiguity inherent in natural language. Enhanced natural language, structured through prompt engineering techniques, similarly improves the model's interpretability and creativity. Conclusions: The study underscores the potential of pseudo-code engineering in refining human-AI interaction and achieving more deterministic, concise, and direct outcomes, advocating for its broader application across disciplines requiring precise AI responses.
• 摘要：背景：人工智能（AI）融入日常生活，特别是通过利用自然语言处理（NLP）的聊天机器人，既带来了革命性的潜力，也带来了独特的挑战。目的是研究不同的输入形式如何影响 ChatGPT（OpenAI 的领先语言模型）理解和执行复杂、多意图任务的性能。设计：该研究采用案例研究方法并辅以话语分析，分析了 ChatGPT 对从自然语言到伪代码工程等不同输入的响应。该研究具体考察了模型在四个方面的熟练程度：对意图的理解、可解释性、完整性和创造力。设置和参与者：作为人工智能交互的理论探索，本研究重点分析 ChatGPT 处理的结构化和非结构化输入，没有直接的人类参与者。数据收集和分析：该研究利用合成案例场景，包括组织“每周​​膳食计划”和“购物清单”，来评估 ChatGPT 对自然语言和伪代码工程提示的响应。该分析的基础是识别不同输入格式的模式、矛盾和独特的响应元素。结果：研究结果表明，伪代码工程输入显着增强了 ChatGPT 响应的清晰度和确定性，减少了自然语言固有的歧义。通过即时工程技术构建的增强型自然语言同样提高了模型的可解释性和创造力。结论：该研究强调了伪代码工程在改善人类与人工智能交互并实现更加确定性、简洁和直接结果方面的潜力，倡导其在需要精确人工智能响应的学科中更广泛的应用。

Title: Enhancing Question Answering for Enterprise Knowledge Bases using Large Language Models

• Authors: Feihu Jiang, Chuan Qin, Kaichun Yao, Chuyu Fang, Fuzhen Zhuang, Hengshu Zhu, Hui Xiong
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2404.08695
• Pdf URL: https://arxiv.org/pdf/2404.08695
• Copy Paste: [[2404.08695]] Enhancing Question Answering for Enterprise Knowledge Bases using Large Language Models(https://arxiv.org/abs/2404.08695)
• Keywords: language model, llm
• Abstract: Efficient knowledge management plays a pivotal role in augmenting both the operational efficiency and the innovative capacity of businesses and organizations. By indexing knowledge through vectorization, a variety of knowledge retrieval methods have emerged, significantly enhancing the efficacy of knowledge management systems. Recently, the rapid advancements in generative natural language processing technologies paved the way for generating precise and coherent answers after retrieving relevant documents tailored to user queries. However, for enterprise knowledge bases, assembling extensive training data from scratch for knowledge retrieval and generation is a formidable challenge due to the privacy and security policies of private data, frequently entailing substantial costs. To address the challenge above, in this paper, we propose EKRG, a novel Retrieval-Generation framework based on large language models (LLMs), expertly designed to enable question-answering for Enterprise Knowledge bases with limited annotation costs. Specifically, for the retrieval process, we first introduce an instruction-tuning method using an LLM to generate sufficient document-question pairs for training a knowledge retriever. This method, through carefully designed instructions, efficiently generates diverse questions for enterprise knowledge bases, encompassing both fact-oriented and solution-oriented knowledge. Additionally, we develop a relevance-aware teacher-student learning strategy to further enhance the efficiency of the training process. For the generation process, we propose a novel chain of thought (CoT) based fine-tuning method to empower the LLM-based generator to adeptly respond to user questions using retrieved documents. Finally, extensive experiments on real-world datasets have demonstrated the effectiveness of our proposed framework.
• 摘要：高效的知识管理在提高企业和组织的运营效率和创新能力方面发挥着关键作用。通过向量化对知识进行索引，出现了多种知识检索方法，显着提高了知识管理系统的效率。最近，生成式自然语言处理技术的快速进步为在检索根据用户查询定制的相关文档后生成精确且连贯的答案铺平了道路。然而，对于企业知识库来说，由于私有数据的隐私和安全政策，从头开始组装大量训练数据以进行知识检索和生成是一项艰巨的挑战，通常需要高昂的成本。为了解决上述挑战，在本文中，我们提出了 EKRG，这是一种基于大型语言模型 (LLM) 的新型检索生成框架，经过专业设计，能够以有限的注释成本实现企业知识库的问答。具体来说，对于检索过程，我们首先引入一种使用法学硕士的指令调整方法来生成足够的文档-问题对来训练知识检索器。该方法通过精心设计的指令，有效地为企业知识库生成多样化的问题，涵盖面向事实和面向解决方案的知识。此外，我们还制定了具有相关性的师生学习策略，以进一步提高培训过程的效率。对于生成过程，我们提出了一种新颖的基于思想链（CoT）的微调方法，使基于 LLM 的生成器能够使用检索到的文档熟练地响应用户问题。最后，对现实世界数据集的广泛实验证明了我们提出的框架的有效性。

Title: Lossless Acceleration of Large Language Model via Adaptive N-gram Parallel Decoding

• Authors: Jie Ou, Yueming Chen, Wenhong Tian
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.08698
• Pdf URL: https://arxiv.org/pdf/2404.08698
• Copy Paste: [[2404.08698]] Lossless Acceleration of Large Language Model via Adaptive N-gram Parallel Decoding(https://arxiv.org/abs/2404.08698)
• Keywords: language model, llm
• Abstract: While Large Language Models (LLMs) have shown remarkable abilities, they are hindered by significant resource consumption and considerable latency due to autoregressive processing. In this study, we introduce Adaptive N-gram Parallel Decoding (ANPD), an innovative and lossless approach that accelerates inference by allowing the simultaneous generation of multiple tokens. ANPD incorporates a two-stage approach: it begins with a rapid drafting phase that employs an N-gram module, which adapts based on the current interactive context, followed by a verification phase, during which the original LLM assesses and confirms the proposed tokens. Consequently, ANPD preserves the integrity of the LLM's original output while enhancing processing speed. We further leverage a multi-level architecture for the N-gram module to enhance the precision of the initial draft, consequently reducing inference latency. ANPD eliminates the need for retraining or extra GPU memory, making it an efficient and plug-and-play enhancement. In our experiments, models such as LLaMA and its fine-tuned variants have shown speed improvements up to 3.67x, validating the effectiveness of our proposed ANPD.
• 摘要：虽然大型语言模型 (LLM) 表现出了非凡的能力，但它们受到自回归处理造成的大量资源消耗和相当大的延迟的阻碍。在本研究中，我们引入了自适应 N 元语法并行解码 (ANPD)，这是一种创新且无损的方法，可通过允许同时生成多个标记来加速推理。 ANPD 采用两阶段方法：首先是使用 N-gram 模块的快速起草阶段，该模块根据当前的交互上下文进行调整，然后是验证阶段，在此期间原始 LLM 评估并确认提议的令牌。因此，ANPD 保留了法学硕士原始输出的完整性，同时提高了处理速度。我们进一步利用 N-gram 模块的多级架构来提高初始草稿的精度，从而减少推理延迟。 ANPD 无需重新训练或额外的 GPU 内存，使其成为高效且即插即用的增强功能。在我们的实验中，LLaMA 等模型及其微调变体的速度提高了高达 3.67 倍，验证了我们提出的 ANPD 的有效性。

Title: Analyzing the Impact of Data Selection and Fine-Tuning on Economic and Political Biases in LLMs

• Authors: Ahmed Agiza, Mohamed Mostagir, Sherief Reda
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.08699
• Pdf URL: https://arxiv.org/pdf/2404.08699
• Copy Paste: [[2404.08699]] Analyzing the Impact of Data Selection and Fine-Tuning on Economic and Political Biases in LLMs(https://arxiv.org/abs/2404.08699)
• Keywords: language model, llm
• Abstract: In an era where language models are increasingly integrated into decision-making and communication, understanding the biases within Large Language Models (LLMs) becomes imperative, especially when these models are applied in the economic and political domains. This work investigates the impact of fine-tuning and data selection on economic and political biases in LLM. We explore the methodological aspects of biasing LLMs towards specific ideologies, mindful of the biases that arise from their extensive training on diverse datasets. Our approach, distinct from earlier efforts that either focus on smaller models or entail resource-intensive pre-training, employs Parameter-Efficient Fine-Tuning (PEFT) techniques. These techniques allow for the alignment of LLMs with targeted ideologies by modifying a small subset of parameters. We introduce a systematic method for dataset selection, annotation, and instruction tuning, and we assess its effectiveness through both quantitative and qualitative evaluations. Our work analyzes the potential of embedding specific biases into LLMs and contributes to the dialogue on the ethical application of AI, highlighting the importance of deploying AI in a manner that aligns with societal values.
• 摘要：在语言模型越来越多地融入决策和交流的时代，理解大型语言模型（LLM）中的偏见变得势在必行，特别是当这些模型应用于经济和政治领域时。这项工作调查了微调和数据选择对法学硕士经济和政治偏见的影响。我们探讨了法学硕士偏向特定意识形态的方法论方面，并注意他们对不同数据集的广泛培训所产生的偏见。我们的方法与早期关注较小模型或需要资源密集型预训练的方法不同，采用参数高效微调（PEFT）技术。这些技术允许通过修改一小部分参数来使法学硕士与目标意识形态保持一致。我们引入了一种用于数据集选择、注释和指令调整的系统方法，并通过定量和定性评估来评估其有效性。我们的工作分析了将特定偏见嵌入法学硕士的可能性，并促进有关人工智能道德应用的对话，强调以符合社会价值观的方式部署人工智能的重要性。

Title: Is Your LLM Outdated? Benchmarking LLMs & Alignment Algorithms for Time-Sensitive Knowledge

• Authors: Seyed Mahed Mousavi, Simone Alghisi, Giuseppe Riccardi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.08700
• Pdf URL: https://arxiv.org/pdf/2404.08700
• Copy Paste: [[2404.08700]] Is Your LLM Outdated? Benchmarking LLMs & Alignment Algorithms for Time-Sensitive Knowledge(https://arxiv.org/abs/2404.08700)
• Keywords: language model, llm, retrieval augmented generation
• Abstract: We study the appropriateness of Large Language Models (LLMs) as knowledge repositories. We focus on the challenge of maintaining LLMs' factual knowledge up-to-date over time. Motivated by the lack of studies on identifying outdated knowledge within LLMs, we design and develop a dynamic benchmark with up-to-date ground truth answers for each target factual question. We evaluate eighteen open-source and closed-source state-of-the-art LLMs on time-sensitive knowledge retrieved in real-time from Wikidata. We select time-sensitive domain facts in politics, sports, and organizations, and estimate the recency of the information learned by the model during pre-training\fine-tuning. In the second contribution, we evaluate the effectiveness of knowledge editing methods for aligning LLMs with up-to-date factual knowledge and compare their performance with Retrieval Augmented Generation. The dynamic benchmark is designed to be used as-is to assess LLMs's up-to-dateness, as well as to be extended to other domains by sharing the code, the dataset, as well as evaluation and visualization scripts.
• 摘要：我们研究大型语言模型（LLM）作为知识存储库的适用性。我们专注于保持法学硕士的事实知识随着时间的推移保持最新的挑战。由于缺乏关于识别法学硕士中过时知识的研究，我们设计并开发了一个动态基准，其中包含每个目标事实问题的最新地面真相答案。我们根据从维基数据实时检索的时间敏感知识来评估十八个开源和闭源最先进的法学硕士。我们选择政治、体育和组织中的时间敏感领域事实，并估计模型在预训练\微调期间学到的信息的新近度。在第二个贡献中，我们评估了知识编辑方法将法学硕士与最新事实知识结合起来的有效性，并将其性能与检索增强生成进行了比较。动态基准旨在按原样使用来评估法学硕士的最新性，并通过共享代码、数据集以及评估和可视化脚本扩展到其他领域。

Title: MM-PhyQA: Multimodal Physics Question-Answering With Multi-Image CoT Prompting

• Authors: Avinash Anand, Janak Kapuriya, Apoorv Singh, Jay Saraf, Naman Lal, Astha Verma, Rushali Gupta, Rajiv Shah
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.08704
• Pdf URL: https://arxiv.org/pdf/2404.08704
• Copy Paste: [[2404.08704]] MM-PhyQA: Multimodal Physics Question-Answering With Multi-Image CoT Prompting(https://arxiv.org/abs/2404.08704)
• Keywords: language model, gpt, llm, prompt, chain-of-thought
• Abstract: While Large Language Models (LLMs) can achieve human-level performance in various tasks, they continue to face challenges when it comes to effectively tackling multi-step physics reasoning tasks. To identify the shortcomings of existing models and facilitate further research in this area, we curated a novel dataset, MM-PhyQA, which comprises well-constructed, high schoollevel multimodal physics problems. By evaluating the performance of contemporary LLMs that are publicly available, both with and without the incorporation of multimodal elements in these problems, we aim to shed light on their capabilities. For generating answers for questions consisting of multimodal input (in this case, images and text) we employed Zero-shot prediction using GPT-4 and utilized LLaVA (LLaVA and LLaVA-1.5), the latter of which were fine-tuned on our dataset. For evaluating the performance of LLMs consisting solely of textual input, we tested the performance of the base and fine-tuned versions of the Mistral-7B and LLaMA2-7b models. We also showcased the performance of the novel Multi-Image Chain-of-Thought (MI-CoT) Prompting technique, which when used to train LLaVA-1.5 13b yielded the best results when tested on our dataset, with superior scores in most metrics and the highest accuracy of 71.65% on the test set.
• 摘要：虽然大型语言模型 (LLM) 可以在各种任务中实现人类水平的表现，但在有效处理多步骤物理推理任务时，它们仍然面临挑战。为了找出现有模型的缺点并促进该领域的进一步研究，我们策划了一个新颖的数据集 MM-PhyQA，其中包含结构良好的高中水平多模态物理问题。通过评估当代公开的法学硕士的表现，无论是否在这些问题中纳入多模式元素，我们的目标是揭示他们的能力。为了生成由多模态输入（在本例中为图像和文本）组成的问题的答案，我们采用了使用 GPT-4 的零样本预测并利用了 LLaVA（LLaVA 和 LLaVA-1.5），后者在我们的数据集上进行了微调。为了评估仅包含文本输入的法学硕士的性能，我们测试了 Mistral-7B 和 LLaMA2-7b 模型的基础版本和微调版本的性能。我们还展示了新颖的多图像思维链 (MI-CoT) 提示技术的性能，该技术在用于训练 LLaVA-1.5 13b 时在我们的数据集上进行测试时产生了最佳结果，在大多数指标和测试中都取得了优异的成绩在测试集上的最高准确率达到 71.65%。

Title: Introducing L2M3, A Multilingual Medical Large Language Model to Advance Health Equity in Low-Resource Regions

• Authors: Agasthya Gangavarapu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.08705
• Pdf URL: https://arxiv.org/pdf/2404.08705
• Copy Paste: [[2404.08705]] Introducing L2M3, A Multilingual Medical Large Language Model to Advance Health Equity in Low-Resource Regions(https://arxiv.org/abs/2404.08705)
• Keywords: language model, llm
• Abstract: Addressing the imminent shortfall of 10 million health workers by 2030, predominantly in Low- and Middle-Income Countries (LMICs), this paper introduces an innovative approach that harnesses the power of Large Language Models (LLMs) integrated with machine translation models. This solution is engineered to meet the unique needs of Community Health Workers (CHWs), overcoming language barriers, cultural sensitivities, and the limited availability of medical dialog datasets. I have crafted a model that not only boasts superior translation capabilities but also undergoes rigorous fine-tuning on open-source datasets to ensure medical accuracy and is equipped with comprehensive safety features to counteract the risks of misinformation. Featuring a modular design, this approach is specifically structured for swift adaptation across various linguistic and cultural contexts, utilizing open-source components to significantly reduce healthcare operational costs. This strategic innovation markedly improves the accessibility and quality of healthcare services by providing CHWs with contextually appropriate medical knowledge and diagnostic tools. This paper highlights the transformative impact of this context-aware LLM, underscoring its crucial role in addressing the global healthcare workforce deficit and propelling forward healthcare outcomes in LMICs.
• 摘要：为了解决到 2030 年即将面临的 1000 万卫生工作者短缺问题（主要是在低收入和中等收入国家 (LMIC)），本文介绍了一种创新方法，该方法利用与机器翻译模型集成的大型语言模型 (LLM) 的力量。该解决方案旨在满足社区卫生工作者 (CHW) 的独特需求，克服语言障碍、文化敏感性和医疗对话数据集的有限可用性。我精心设计的模型不仅拥有卓越的翻译能力，而且还对开源数据集进行了严格的微调，以确保医疗准确性，并配备了全面的安全功能来抵消错误信息的风险。该方法采用模块化设计，经过专门设计，可以快速适应各种语言和文化背景，利用开源组件显着降低医疗保健运营成本。这项战略创新通过为社区卫生工作者提供适合实际情况的医疗知识和诊断工具，显着提高了医疗保健服务的可及性和质量。本文强调了这种情境意识法学硕士的变革性影响，强调其在解决全球医疗保健劳动力短缺和推动中低收入国家医疗保健成果方面的关键作用。

Title: The Generation Gap:Exploring Age Bias in Large Language Models

• Authors: Siyang Liu, Trish Maturi, Siqi Shen, Rada Mihalcea
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.08760
• Pdf URL: https://arxiv.org/pdf/2404.08760
• Copy Paste: [[2404.08760]] The Generation Gap:Exploring Age Bias in Large Language Models(https://arxiv.org/abs/2404.08760)
• Keywords: language model, llm, prompt
• Abstract: In this paper, we explore the alignment of values in Large Language Models (LLMs) with specific age groups, leveraging data from the World Value Survey across thirteen categories. Through a diverse set of prompts tailored to ensure response robustness, we find a general inclination of LLM values towards younger demographics. Additionally, we explore the impact of incorporating age identity information in prompts and observe challenges in mitigating value discrepancies with different age cohorts. Our findings highlight the age bias in LLMs and provide insights for future work.
• 摘要：在本文中，我们利用来自十三个类别的世界价值调查的数据，探讨了大型语言模型 (LLM) 中的价值观与特定年龄组的一致性。通过为确保响应稳健性而量身定制的各种提示，我们发现法学硕士价值观普遍倾向于年轻人群。此外，我们还探讨了在提示中纳入年龄身份信息的影响，并观察了减少不同年龄群体的价值差异方面的挑战。我们的研究结果凸显了法学硕士的年龄偏见，并为未来的工作提供了见解。

Title: CreativEval: Evaluating Creativity of LLM-Based Hardware Code Generation

• Authors: Matthew DeLorenzo, Vasudev Gohil, Jeyavijayan Rajendran
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.08806
• Pdf URL: https://arxiv.org/pdf/2404.08806
• Copy Paste: [[2404.08806]] CreativEval: Evaluating Creativity of LLM-Based Hardware Code Generation(https://arxiv.org/abs/2404.08806)
• Keywords: language model, gpt, llm, prompt
• Abstract: Large Language Models (LLMs) have proved effective and efficient in generating code, leading to their utilization within the hardware design process. Prior works evaluating LLMs' abilities for register transfer level code generation solely focus on functional correctness. However, the creativity associated with these LLMs, or the ability to generate novel and unique solutions, is a metric not as well understood, in part due to the challenge of quantifying this quality. To address this research gap, we present CreativeEval, a framework for evaluating the creativity of LLMs within the context of generating hardware designs. We quantify four creative sub-components, fluency, flexibility, originality, and elaboration, through various prompting and post-processing techniques. We then evaluate multiple popular LLMs (including GPT models, CodeLlama, and VeriGen) upon this creativity metric, with results indicating GPT-3.5 as the most creative model in generating hardware designs.
• 摘要：事实证明，大型语言模型 (LLM) 在生成代码方面是有效且高效的，因此可以在硬件设计过程中得到利用。之前评估法学硕士寄存器传输级代码生成能力的工作仅关注功能正确性。然而，与这些法学硕士相关的创造力，或者说产生新颖和独特解决方案的能力，是一个尚未被充分理解的指标，部分原因是量化这种质量的挑战。为了解决这一研究空白，我们提出了 CreativeEval，这是一个用于评估法学硕士在生成硬件设计的背景下的创造力的框架。我们通过各种提示和后处理技术来量化四个创意子组件：流畅性、灵活性、原创性和精细化。然后，我们根据这个创造力指标评估多个流行的 LLM（包括 GPT 模型、CodeLlama 和 VeriGen），结果表明 GPT-3.5 是生成硬件设计中最具创意的模型。

Title: Evaluating the Quality of Answers in Political Q&A Sessions with Large Language Models

• Authors: R. Michael Alvarez, Jacob Morrier
• Subjects: cs.CL, econ.EM
• Abstract URL: https://arxiv.org/abs/2404.08816
• Pdf URL: https://arxiv.org/pdf/2404.08816
• Copy Paste: [[2404.08816]] Evaluating the Quality of Answers in Political Q&A Sessions with Large Language Models(https://arxiv.org/abs/2404.08816)
• Keywords: language model
• Abstract: This paper presents a new approach to evaluating the quality of answers in political question-and-answer sessions. We propose to measure an answer's quality based on the degree to which it allows us to infer the initial question accurately. This conception of answer quality inherently reflects their relevance to initial questions. Drawing parallels with semantic search, we argue that this measurement approach can be operationalized by fine-tuning a large language model on the observed corpus of questions and answers without additional labeled data. We showcase our measurement approach within the context of the Question Period in the Canadian House of Commons. Our approach yields valuable insights into the correlates of the quality of answers in the Question Period. We find that answer quality varies significantly based on the party affiliation of the members of Parliament asking the questions and uncover a meaningful correlation between answer quality and the topics of the questions.
• 摘要：本文提出了一种评估政治问答环节答案质量的新方法。我们建议根据答案允许我们准确推断初始问题的程度来衡量答案的质量。答案质量的概念本质上反映了它们与初始问题的相关性。与语义搜索相似，我们认为这种测量方法可以通过在观察到的问题和答案语料库上微调大型语言模型来操作，而无需额外的标记数据。我们在加拿大下议院质询期的背景下展示了我们的衡量方法。我们的方法对提问期间答案质量的相关性产生了有价值的见解。我们发现，答案质量根据提出问题的议会议员的党派关系而存在显着差异，并发现答案质量与问题主题之间存在有意义的相关性。

Title: Revisiting Code Similarity Evaluation with Abstract Syntax Tree Edit Distance

• Authors: Yewei Song, Cedric Lothritz, Daniel Tang, Tegawendé F. Bissyandé, Jacques Klein
• Subjects: cs.CL, cs.PL, cs.SE
• Abstract URL: https://arxiv.org/abs/2404.08817
• Pdf URL: https://arxiv.org/pdf/2404.08817
• Copy Paste: [[2404.08817]] Revisiting Code Similarity Evaluation with Abstract Syntax Tree Edit Distance(https://arxiv.org/abs/2404.08817)
• Keywords: gpt, prompt
• Abstract: This paper revisits recent code similarity evaluation metrics, particularly focusing on the application of Abstract Syntax Tree (AST) editing distance in diverse programming languages. In particular, we explore the usefulness of these metrics and compare them to traditional sequence similarity metrics. Our experiments showcase the effectiveness of AST editing distance in capturing intricate code structures, revealing a high correlation with established metrics. Furthermore, we explore the strengths and weaknesses of AST editing distance and prompt-based GPT similarity scores in comparison to BLEU score, execution match, and Jaccard Similarity. We propose, optimize, and publish an adaptable metric that demonstrates effectiveness across all tested languages, representing an enhanced version of Tree Similarity of Edit Distance (TSED).
• 摘要：本文重新审视了最近的代码相似性评估指标，特别关注抽象语法树（AST）编辑距离在不同编程语言中的应用。特别是，我们探讨了这些指标的有用性，并将它们与传统的序列相似性指标进行比较。我们的实验展示了 AST 编辑距离在捕获复杂代码结构方面的有效性，揭示了与既定指标的高度相关性。此外，我们还探讨了 AST 编辑距离和基于提示的 GPT 相似度得分与 BLEU 得分、执行匹配和 Jaccard 相似度相比的优缺点。我们提出、优化并发布了一个适应性强的指标，该指标在所有测试的语言中展示了有效性，代表了编辑距离树相似度（TSED）的增强版本。

Title: Constrained C-Test Generation via Mixed-Integer Programming

• Authors: Ji-Ung Lee, Marc E. Pfetsch, Iryna Gurevych
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.08821
• Pdf URL: https://arxiv.org/pdf/2404.08821
• Copy Paste: [[2404.08821]] Constrained C-Test Generation via Mixed-Integer Programming(https://arxiv.org/abs/2404.08821)
• Keywords: gpt
• Abstract: This work proposes a novel method to generate C-Tests; a deviated form of cloze tests (a gap filling exercise) where only the last part of a word is turned into a gap. In contrast to previous works that only consider varying the gap size or gap placement to achieve locally optimal solutions, we propose a mixed-integer programming (MIP) approach. This allows us to consider gap size and placement simultaneously, achieving globally optimal solutions, and to directly integrate state-of-the-art models for gap difficulty prediction into the optimization problem. A user study with 40 participants across four C-Test generation strategies (including GPT-4) shows that our approach (MIP) significantly outperforms two of the baseline strategies (based on gap placement and GPT-4); and performs on-par with the third (based on gap size). Our analysis shows that GPT-4 still struggles to fulfill explicit constraints during generation and that MIP produces C-Tests that correlate best with the perceived difficulty. We publish our code, model, and collected data consisting of 32 English C-Tests with 20 gaps each (totaling 3,200 individual gap responses) under an open source license.
• 摘要：这项工作提出了一种生成 C 测试的新方法；完形填空测试（填空练习）的一种偏差形式，其中只有单词的最后一部分变成空缺。与之前仅考虑改变间隙大小或间隙位置以实现局部最优解决方案的工作相比，我们提出了一种混合整数规划（MIP）方法。这使我们能够同时考虑间隙大小和放置，实现全局最优解决方案，并将用于间隙难度预测的最先进模型直接集成到优化问题中。一项针对 40 名参与者的四种 C-Test 生成策略（包括 GPT-4）的用户研究表明，我们的方法 (MIP) 显着优于两种基线策略（基于间隙放置和 GPT-4）；并与第三个性能相当（基于间隙大小）。我们的分析表明，GPT-4 在生成过程中仍然难以满足明确的约束，并且 MIP 生成与感知难度最相关的 C 测试。我们在开源许可下发布了我们的代码、模型和收集的数据，其中包括 32 个英语 C 测试，每个测试有 20 个差距（总共 3,200 个单独的差距响应）。

Title: On Speculative Decoding for Multimodal Large Language Models

• Authors: Mukul Gagrani, Raghavv Goel, Wonseok Jeon, Junyoung Park, Mingu Lee, Christopher Lott
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.08856
• Pdf URL: https://arxiv.org/pdf/2404.08856
• Copy Paste: [[2404.08856]] On Speculative Decoding for Multimodal Large Language Models(https://arxiv.org/abs/2404.08856)
• Keywords: language model, llm
• Abstract: Inference with Multimodal Large Language Models (MLLMs) is slow due to their large-language-model backbone which suffers from memory bandwidth bottleneck and generates tokens auto-regressively. In this paper, we explore the application of speculative decoding to enhance the inference efficiency of MLLMs, specifically the LLaVA 7B model. We show that a language-only model can serve as a good draft model for speculative decoding with LLaVA 7B, bypassing the need for image tokens and their associated processing components from the draft model. Our experiments across three different tasks show that speculative decoding can achieve a memory-bound speedup of up to 2.37$\times$ using a 115M parameter language model that we trained from scratch. Additionally, we introduce a compact LLaVA draft model incorporating an image adapter, which shows marginal performance gains in image captioning while maintaining comparable results in other tasks.
• 摘要：多模态大型语言模型（MLLM）的推理速度很慢，因为它们的大型语言模型主干遭受内存带宽瓶颈并自动回归生成令牌。在本文中，我们探索了推测解码的应用来提高 MLLM 的推理效率，特别是 LLaVA 7B 模型。我们证明，仅语言模型可以作为使用 LLaVA 7B 进行推测解码的良好草稿模型，从而绕过草稿模型中对图像标记及其相关处理组件的需求。我们针对三个不同任务的实验表明，使用我们从头开始训练的 115M 参数语言模型，推测性解码可以实现高达 2.37$\times$ 的内存限制加速。此外，我们还引入了一个包含图像适配器的紧凑型 LLaVA 草稿模型，该模型显示了图像字幕方面的边际性能提升，同时在其他任务中保持了可比较的结果。

Title: LLM In-Context Recall is Prompt Dependent

• Authors: Daniel Machlab, Rick Battle
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.08865
• Pdf URL: https://arxiv.org/pdf/2404.08865
• Copy Paste: [[2404.08865]] LLM In-Context Recall is Prompt Dependent(https://arxiv.org/abs/2404.08865)
• Keywords: language model, llm, prompt
• Abstract: The proliferation of Large Language Models (LLMs) highlights the critical importance of conducting thorough evaluations to discern their comparative advantages, limitations, and optimal use cases. Particularly important is assessing their capacity to accurately retrieve information included in a given prompt. A model's ability to do this significantly influences how effectively it can utilize contextual details, thus impacting its practical efficacy and dependability in real-world applications. Our research analyzes the in-context recall performance of various LLMs using the needle-in-a-haystack method. In this approach, a factoid (the "needle") is embedded within a block of filler text (the "haystack"), which the model is asked to retrieve. We assess the recall performance of each model across various haystack lengths and with varying needle placements to identify performance patterns. This study demonstrates that an LLM's recall capability is not only contingent upon the prompt's content but also may be compromised by biases in its training data. Conversely, adjustments to model architecture, training strategy, or fine-tuning can improve performance. Our analysis provides insight into LLM behavior, offering direction for the development of more effective applications of LLMs.
• 摘要：大型语言模型 (LLM) 的激增凸显了进行彻底评估以辨别其比较优势、局限性和最佳用例的至关重要性。尤其重要的是评估他们准确检索给定提示中包含的信息的能力。模型执行此操作的能力会显着影响其利用上下文细节的效率，从而影响其在实际应用中的实际功效和可靠性。我们的研究使用大海捞针的方法分析了各种法学硕士的上下文回忆表现。在这种方法中，事实陈述（“针”）被嵌入到填充文本块（“干草堆”）中，要求模型检索该填充文本。我们评估每个模型在不同的干草堆长度和不同的针位置上的召回性能，以识别性能模式。这项研究表明，法学硕士的回忆能力不仅取决于提示的内容，而且还可能受到其训练数据偏差的影响。相反，调整模型架构、训练策略或微调可以提高性能。我们的分析提供了对法学硕士行为的洞察，为开发更有效的法学硕士应用提供了方向。

Title: Multimodal Cross-Document Event Coreference Resolution Using Linear Semantic Transfer and Mixed-Modality Ensembles

• Authors: Abhijnan Nath, Huma Jamil, Shafiuddin Rehan Ahmed, George Baker, Rahul Ghosh, James H. Martin, Nathaniel Blanchard, Nikhil Krishnaswamy
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.08949
• Pdf URL: https://arxiv.org/pdf/2404.08949
• Copy Paste: [[2404.08949]] Multimodal Cross-Document Event Coreference Resolution Using Linear Semantic Transfer and Mixed-Modality Ensembles(https://arxiv.org/abs/2404.08949)
• Keywords: language model
• Abstract: Event coreference resolution (ECR) is the task of determining whether distinct mentions of events within a multi-document corpus are actually linked to the same underlying occurrence. Images of the events can help facilitate resolution when language is ambiguous. Here, we propose a multimodal cross-document event coreference resolution method that integrates visual and textual cues with a simple linear map between vision and language models. As existing ECR benchmark datasets rarely provide images for all event mentions, we augment the popular ECB+ dataset with event-centric images scraped from the internet and generated using image diffusion models. We establish three methods that incorporate images and text for coreference: 1) a standard fused model with finetuning, 2) a novel linear mapping method without finetuning and 3) an ensembling approach based on splitting mention pairs by semantic and discourse-level difficulty. We evaluate on 2 datasets: the augmented ECB+, and AIDA Phase 1. Our ensemble systems using cross-modal linear mapping establish an upper limit (91.9 CoNLL F1) on ECB+ ECR performance given the preprocessing assumptions used, and establish a novel baseline on AIDA Phase 1. Our results demonstrate the utility of multimodal information in ECR for certain challenging coreference problems, and highlight a need for more multimodal resources in the coreference resolution space.
• 摘要：事件共指解析（ECR）的任务是确定多文档语料库中事件的不同提及是否实际上与相同的潜在事件相关联。当语言含糊不清时，事件的图像可以帮助解决问题。在这里，我们提出了一种多模式跨文档事件共指解析方法，该方法将视觉和文本线索与视觉和语言模型之间的简单线性映射集成在一起。由于现有的 ECR 基准数据集很少提供所有事件提及的图像，因此我们使用从互联网上抓取并使用图像扩散模型生成的以事件为中心的图像来增强流行的 ECB+ 数据集。我们建立了三种结合图像和文本进行共指的方法：1）具有微调的标准融合模型，2）一种无需微调的新颖线性映射方法，3）一种基于按语义和话语级别难度分割提及对的集成方法。我们评估 2 个数据集：增强的 ECB+ 和 AIDA 第 1 阶段。我们的集成系统使用跨模态线性映射，在给定所使用的预处理假设的情况下建立了 ECB+ ECR 性能的上限 (91.9 CoNLL F1)，并在 AIDA 上建立了一个新颖的基线第一阶段。我们的结果证明了 ECR 中多模态信息对于某些具有挑战性的共指问题的实用性，并强调在共指解析空间中需要更多的多模态资源。

Title: WikiSplit++: Easy Data Refinement for Split and Rephrase

• Authors: Hayato Tsukagoshi, Tsutomu Hirao, Makoto Morishita, Katsuki Chousa, Ryohei Sasano, Koichi Takeda
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.09002
• Pdf URL: https://arxiv.org/pdf/2404.09002
• Copy Paste: [[2404.09002]] WikiSplit++: Easy Data Refinement for Split and Rephrase(https://arxiv.org/abs/2404.09002)
• Keywords: hallucination
• Abstract: The task of Split and Rephrase, which splits a complex sentence into multiple simple sentences with the same meaning, improves readability and enhances the performance of downstream tasks in natural language processing (NLP). However, while Split and Rephrase can be improved using a text-to-text generation approach that applies encoder-decoder models fine-tuned with a large-scale dataset, it still suffers from hallucinations and under-splitting. To address these issues, this paper presents a simple and strong data refinement approach. Here, we create WikiSplit++ by removing instances in WikiSplit where complex sentences do not entail at least one of the simpler sentences and reversing the order of reference simple sentences. Experimental results show that training with WikiSplit++ leads to better performance than training with WikiSplit, even with fewer training instances. In particular, our approach yields significant gains in the number of splits and the entailment ratio, a proxy for measuring hallucinations.
• 摘要：Split and Rephrase 任务将一个复杂的句子拆分为多个具有相同含义的简单句子，提高了可读性并增强了自然语言处理（NLP）中下游任务的性能。然而，虽然可以使用文本到文本生成方法来改进分割和改写，该方法应用使用大规模数据集微调的编码器-解码器模型，但它仍然遭受幻觉和分割不足的困扰。为了解决这些问题，本文提出了一种简单而强大的数据细化方法。在这里，我们通过删除 WikiSplit 中复杂句子不包含至少一个简单句子的实例并反转参考简单句子的顺序来创建 WikiSplit++。实验结果表明，即使训练实例较少，使用 WikiSplit++ 进行训练也能比使用 WikiSplit 进行训练获得更好的性能。特别是，我们的方法在分裂数量和蕴涵比（衡量幻觉的代理）方面产生了显着的收益。

Title: MING-MOE: Enhancing Medical Multi-Task Learning in Large Language Models with Sparse Mixture of Low-Rank Adapter Experts

• Authors: Yusheng Liao, Shuyang Jiang, Yu Wang, Yanfeng Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.09027
• Pdf URL: https://arxiv.org/pdf/2404.09027
• Copy Paste: [[2404.09027]] MING-MOE: Enhancing Medical Multi-Task Learning in Large Language Models with Sparse Mixture of Low-Rank Adapter Experts(https://arxiv.org/abs/2404.09027)
• Keywords: language model, gpt, chat
• Abstract: Large language models like ChatGPT have shown substantial progress in natural language understanding and generation, proving valuable across various disciplines, including the medical field. Despite advancements, challenges persist due to the complexity and diversity inherent in medical tasks which often require multi-task learning capabilities. Previous approaches, although beneficial, fall short in real-world applications because they necessitate task-specific annotations at inference time, limiting broader generalization. This paper introduces MING-MOE, a novel Mixture-of-Expert~(MOE)-based medical large language model designed to manage diverse and complex medical tasks without requiring task-specific annotations, thus enhancing its usability across extensive datasets. MING-MOE employs a Mixture of Low-Rank Adaptation (MoLoRA) technique, allowing for efficient parameter usage by maintaining base model parameters static while adapting through a minimal set of trainable parameters. We demonstrate that MING-MOE achieves state-of-the-art (SOTA) performance on over 20 medical tasks, illustrating a significant improvement over existing models. This approach not only extends the capabilities of medical language models but also improves inference efficiency.
• 摘要：像 ChatGPT 这样的大型语言模型在自然语言理解和生成方面已经取得了巨大进步，在包括医学领域在内的各个学科中都证明了其价值。尽管取得了进步，但由于医疗任务固有的复杂性和多样性，通常需要多任务学习能力，因此挑战仍然存在。以前的方法虽然有益，但在现实应用中存在不足，因为它们需要在推理时进行特定于任务的注释，从而限制了更广泛的泛化。本文介绍了 MING-MOE，一种新型的基于专家混合 (MOE) 的医学大语言模型，旨在管理多样化且复杂的医疗任务，而不需要特定于任务的注释，从而增强其在广泛数据集上的可用性。 MING-MOE 采用低阶自适应混合 (MoLoRA) 技术，通过保持基本模型参数静态，同时通过最小的可训练参数集进行自适应，从而实现高效的参数使用。我们证明 MING-MOE 在 20 多项医疗任务上实现了最先进 (SOTA) 的性能，说明了对现有模型的显着改进。这种方法不仅扩展了医学语言模型的能力，而且提高了推理效率。

Title: Do LLMs Play Dice? Exploring Probability Distribution Sampling in Large Language Models for Behavioral Simulation

• Authors: Jia Gu, Liang Pang, Huawei Shen, Xueqi Cheng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.09043
• Pdf URL: https://arxiv.org/pdf/2404.09043
• Copy Paste: [[2404.09043]] Do LLMs Play Dice? Exploring Probability Distribution Sampling in Large Language Models for Behavioral Simulation(https://arxiv.org/abs/2404.09043)
• Keywords: language model, llm, agent
• Abstract: With the rapid advancement of large language models (LLMs) and their remarkable capabilities in handling complex language tasks, an increasing number of studies are employing LLMs as agents to emulate the sequential decision-making processes of humans often represented as Markov decision-making processes (MDPs). The actions within this decision-making framework adhere to specific probability distributions and require iterative sampling. This arouses our curiosity regarding the capacity of LLM agents to comprehend probability distributions, thereby guiding the agent's behavioral decision-making through probabilistic sampling and generating behavioral sequences. To answer the above question, we divide the problem into two main aspects: simulation where the exact probability distribution is known, and generation of sequences where the probability distribution is ambiguous. In the first case, the agent is required to give the type and parameters of the probability distribution through the problem description, and then give the sampling sequence. However, our analysis shows that LLM agents perform poorly in this case, but the sampling success rate can be improved through programming tools. Real-world scenarios often entail unknown probability distributions. Thus, in the second case, we ask the agents to change the activity level in online social networks and analyze the frequency of actions. Ultimately, our analysis shows that LLM agents cannot sample probability distributions even using programming tools. Therefore, careful consideration is still required before directly applying LLM agents as agents to simulate human behavior.
• 摘要：随着大型语言模型（LLM）的快速发展及其处理复杂语言任务的卓越能力，越来越多的研究采用 LLM 作为代理来模拟人类的顺序决策过程，通常表示为马尔可夫决策过程。 MDP）。该决策框架内的行动遵循特定的概率分布并需要迭代采样。这引起了我们对LLM智能体理解概率分布的能力的好奇，从而通过概率采样和生成行为序列来指导智能体的行为决策。为了回答上述问题，我们将问题分为两个主要方面：已知确切概率分布的模拟，以及概率分布不明确的序列的生成。第一种情况，要求智能体通过问题描述给出概率分布的类型和参数，然后给出采样序列。然而，我们的分析表明，LLM 代理在这种情况下表现不佳，但可以通过编程工具来提高采样成功率。现实世界的场景通常需要未知的概率分布。因此，在第二种情况下，我们要求代理改变在线社交网络中的活动水平并分析行为频率。最终，我们的分析表明，LLM 代理即使使用编程工具也无法对概率分布进行采样。因此，在直接应用LLM智能体作为模拟人类行为的智能体之前，仍然需要仔细考虑。

Title: Adapting Mental Health Prediction Tasks for Cross-lingual Learning via Meta-Training and In-context Learning with Large Language Model

• Authors: Zita Lifelo, Huansheng Ning, Sahraoui Dhelim
• Subjects: cs.CL, cs.AI, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.09045
• Pdf URL: https://arxiv.org/pdf/2404.09045
• Copy Paste: [[2404.09045]] Adapting Mental Health Prediction Tasks for Cross-lingual Learning via Meta-Training and In-context Learning with Large Language Model(https://arxiv.org/abs/2404.09045)
• Keywords: language model, llm, prompt
• Abstract: Timely identification is essential for the efficient handling of mental health illnesses such as depression. However, the current research fails to adequately address the prediction of mental health conditions from social media data in low-resource African languages like Swahili. This study introduces two distinct approaches utilising model-agnostic meta-learning and leveraging large language models (LLMs) to address this gap. Experiments are conducted on three datasets translated to low-resource language and applied to four mental health tasks, which include stress, depression, depression severity and suicidal ideation prediction. we first apply a meta-learning model with self-supervision, which results in improved model initialisation for rapid adaptation and cross-lingual transfer. The results show that our meta-trained model performs significantly better than standard fine-tuning methods, outperforming the baseline fine-tuning in macro F1 score with 18\% and 0.8\% over XLM-R and mBERT. In parallel, we use LLMs' in-context learning capabilities to assess their performance accuracy across the Swahili mental health prediction tasks by analysing different cross-lingual prompting approaches. Our analysis showed that Swahili prompts performed better than cross-lingual prompts but less than English prompts. Our findings show that in-context learning can be achieved through cross-lingual transfer through carefully crafted prompt templates with examples and instructions.
• 摘要：及时识别对于有效处理抑郁症等精神健康疾病至关重要。然而，目前的研究未能充分解决从斯瓦希里语等资源匮乏的非洲语言的社交媒体数据中预测心理健康状况的问题。本研究引入了两种不同的方法，利用与模型无关的元学习和利用大型语言模型 (LLM) 来解决这一差距。实验在翻译成低资源语言的三个数据集上进行，并应用于四项心理健康任务，包括压力、抑郁、抑郁严重程度和自杀意念预测。我们首先应用具有自我监督的元学习模型，从而改进模型初始化，以实现快速适应和跨语言迁移。结果表明，我们的元训练模型的性能明显优于标准微调方法，在宏 F1 分数方面优于基线微调，比 XLM-R 和 mBERT 分别高出 18% 和 0.8%。与此同时，我们利用法学硕士的情境学习能力，通过分析不同的跨语言提示方法来评估他们在斯瓦希里语心理健康预测任务中的表现准确性。我们的分析表明，斯瓦希里语提示的效果优于跨语言提示，但不如英语提示。我们的研究结果表明，情境学习可以通过精心设计的带有示例和说明的提示模板进行跨语言迁移来实现。

Title: Multilingual Evaluation of Semantic Textual Relatedness

• Authors: Sharvi Endait, Srushti Sonavane, Ridhima Sinare, Pritika Rohera, Advait Naik, Dipali Kadam
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.09047
• Pdf URL: https://arxiv.org/pdf/2404.09047
• Copy Paste: [[2404.09047]] Multilingual Evaluation of Semantic Textual Relatedness(https://arxiv.org/abs/2404.09047)
• Keywords: language model
• Abstract: The explosive growth of online content demands robust Natural Language Processing (NLP) techniques that can capture nuanced meanings and cultural context across diverse languages. Semantic Textual Relatedness (STR) goes beyond superficial word overlap, considering linguistic elements and non-linguistic factors like topic, sentiment, and perspective. Despite its pivotal role, prior NLP research has predominantly focused on English, limiting its applicability across languages. Addressing this gap, our paper dives into capturing deeper connections between sentences beyond simple word overlap. Going beyond English-centric NLP research, we explore STR in Marathi, Hindi, Spanish, and English, unlocking the potential for information retrieval, machine translation, and more. Leveraging the SemEval-2024 shared task, we explore various language models across three learning paradigms: supervised, unsupervised, and cross-lingual. Our comprehensive methodology gains promising results, demonstrating the effectiveness of our approach. This work aims to not only showcase our achievements but also inspire further research in multilingual STR, particularly for low-resourced languages.
• 摘要：在线内容的爆炸式增长需要强大的自然语言处理 (NLP) 技术，该技术可以捕捉不同语言的微妙含义和文化背景。语义文本相关性 (STR) 超越了表面的单词重叠，考虑了语言元素和非语言因素，如主题、情感和观点。尽管其发挥着关键作用，但之前的 NLP 研究主要集中在英语上，限制了其跨语言的适用性。为了解决这一差距，我们的论文深入探讨了除了简单的单词重叠之外的句子之间更深层的联系。除了以英语为中心的 NLP 研究之外，我们还探索马拉地语、印地语、西班牙语和英语的 STR，释放信息检索、机器翻译等方面的潜力。利用 SemEval-2024 共享任务，我们探索了三种学习范式的各种语言模型：监督学习、无监督学习和跨语言学习。我们的综合方法取得了有希望的结果，证明了我们方法的有效性。这项工作不仅旨在展示我们的成就，而且还激发多语言 STR 的进一步研究，特别是针对资源匮乏的语言。

Title: CuriousLLM: Elevating Multi-Document QA with Reasoning-Infused Knowledge Graph Prompting

• Authors: Zukang Yang, Zixuan Zhu
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.09077
• Pdf URL: https://arxiv.org/pdf/2404.09077
• Copy Paste: [[2404.09077]] CuriousLLM: Elevating Multi-Document QA with Reasoning-Infused Knowledge Graph Prompting(https://arxiv.org/abs/2404.09077)
• Keywords: language model, llm, hallucination, prompt, agent
• Abstract: In the field of Question Answering (QA), unifying large language models (LLMs) with external databases has shown great success. However, these methods often fall short in providing the advanced reasoning needed for complex QA tasks. To address these issues, we improve over a novel approach called Knowledge Graph Prompting (KGP), which combines knowledge graphs with a LLM-based agent to improve reasoning and search accuracy. Nevertheless, the original KGP framework necessitates costly fine-tuning with large datasets yet still suffers from LLM hallucination. Therefore, we propose a reasoning-infused LLM agent to enhance this framework. This agent mimics human curiosity to ask follow-up questions to more efficiently navigate the search. This simple modification significantly boosts the LLM performance in QA tasks without the high costs and latency associated with the initial KGP framework. Our ultimate goal is to further develop this approach, leading to more accurate, faster, and cost-effective solutions in the QA domain.
• 摘要：在问答（QA）领域，将大型语言模型（LLM）与外部数据库相结合已经取得了巨大成功。然而，这些方法通常无法提供复杂的 QA 任务所需的高级推理。为了解决这些问题，我们改进了一种称为知识图提示（KGP）的新方法，它将知识图与基于 LLM 的代理相结合，以提高推理和搜索准确性。然而，最初的 KGP 框架需要对大型数据集进行昂贵的微调，但仍然遭受 LLM 幻觉的困扰。因此，我们提出了一个注入推理的 LLM 代理来增强这个框架。该代理模仿人类的好奇心来提出后续问题，以更有效地导航搜索。这种简单的修改显着提高了 QA 任务中的 LLM 性能，而不会产生与初始 KGP 框架相关的高成本和延迟。我们的最终目标是进一步开发这种方法，从而在 QA 领域提供更准确、更快速且更具成本效益的解决方案。

Title: Confidence Calibration and Rationalization for LLMs via Multi-Agent Deliberation

• Authors: Ruixin Yang, Dheeraj Rajagopa, Shirley Anugrah Hayati, Bin Hu, Dongyeop Kang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.09127
• Pdf URL: https://arxiv.org/pdf/2404.09127
• Copy Paste: [[2404.09127]] Confidence Calibration and Rationalization for LLMs via Multi-Agent Deliberation(https://arxiv.org/abs/2404.09127)
• Keywords: language model, llm, agent
• Abstract: Uncertainty estimation is a significant issue for current large language models (LLMs) that are generally poorly calibrated and over-confident, especially with reinforcement learning from human feedback (RLHF). Unlike humans, whose decisions and confidences not only stem from intrinsic beliefs but can also be adjusted through daily observations, existing calibration methods for LLMs focus on estimating or eliciting individual confidence without taking full advantage of the "Collective Wisdom": the interaction among multiple LLMs that can collectively improve both accuracy and calibration. In this work, we propose Collaborative Calibration, a post-hoc training-free calibration strategy that leverages the collaborative and expressive capabilities of multiple tool-augmented LLM agents in a simulated group deliberation process. We demonstrate the effectiveness of Collaborative Calibration on generative QA tasks across various domains, showing its potential in harnessing the rationalization of collectively calibrated confidence assessments and improving the reliability of model predictions.
• 摘要：对于当前的大型语言模型（LLM）来说，不确定性估计是一个重要问题，这些模型通常校准不佳且过于自信，特别是在基于人类反馈的强化学习（RLHF）方面。与人类不同，人类的决定和信心不仅源于内在信念，还可以通过日常观察进行调整，现有的法学硕士校准方法侧重于估计或引发个人信心，而没有充分利用“集体智慧”：多个法学硕士之间的相互作用这可以共同提高精度和校准。在这项工作中，我们提出了协作校准，这是一种事后免培训校准策略，它在模拟的小组审议过程中利用多个工具增强的 LLM 代理的协作和表达能力。我们展示了协作校准在各个领域的生成 QA 任务上的有效性，展示了其在利用集体校准置信度评估的合理化和提高模型预测的可靠性方面的潜力。

Title: When Hindsight is Not 20/20: Testing Limits on Reflective Thinking in Large Language Models

• Authors: Yanhong Li, Chenghao Yang, Allyson Ettinger
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.09129
• Pdf URL: https://arxiv.org/pdf/2404.09129
• Copy Paste: [[2404.09129]] When Hindsight is Not 20/20: Testing Limits on Reflective Thinking in Large Language Models(https://arxiv.org/abs/2404.09129)
• Keywords: language model, llm, prompt
• Abstract: Recent studies suggest that self-reflective prompting can significantly enhance the reasoning capabilities of Large Language Models (LLMs). However, the use of external feedback as a stop criterion raises doubts about the true extent of LLMs' ability to emulate human-like self-reflection. In this paper, we set out to clarify these capabilities under a more stringent evaluation setting in which we disallow any kind of external feedback. Our findings under this setting show a split: while self-reflection enhances performance in TruthfulQA, it adversely affects results in HotpotQA. We conduct follow-up analyses to clarify the contributing factors in these patterns, and find that the influence of self-reflection is impacted both by reliability of accuracy in models' initial responses, and by overall question difficulty: specifically, self-reflection shows the most benefit when models are less likely to be correct initially, and when overall question difficulty is higher. We also find that self-reflection reduces tendency toward majority voting. Based on our findings, we propose guidelines for decisions on when to implement self-reflection. We release the codebase for reproducing our experiments at https://github.com/yanhong-lbh/LLM-SelfReflection-Eval.
• 摘要：最近的研究表明，自我反思提示可以显着增强大型语言模型（LLM）的推理能力。然而，使用外部反馈作为停止标准引发了人们对法学硕士模仿人类自我反思能力的真实程度的怀疑。在本文中，我们着手在更严格的评估环境下阐明这些功能，在该评估环境中我们不允许任何类型的外部反馈。我们在此设置下的研究结果显示出分歧：虽然自我反思可以提高 TruthfulQA 中的表现，但它会对 HotpotQA 中的结果产生不利影响。我们进行了后续分析，以澄清这些模式中的影响因素，并发现自我反思的影响既受到模型初始响应准确性的可靠性的影响，也受到整体问题难度的影响：具体而言，自我反思表明当模型最初不太可能正确且总体问题难度较高时，获益最多。我们还发现自我反思减少了多数投票的倾向。根据我们的发现，我们提出了关于何时实施自我反思的决策指南。我们在 https://github.com/yanhong-lbh/LLM-SelfReflection-Eval 发布了用于重现实验的代码库。

Title: Unveiling LLM Evaluation Focused on Metrics: Challenges and Solutions

• Authors: Taojun Hu, Xiao-Hua Zhou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.09135
• Pdf URL: https://arxiv.org/pdf/2404.09135
• Copy Paste: [[2404.09135]] Unveiling LLM Evaluation Focused on Metrics: Challenges and Solutions(https://arxiv.org/abs/2404.09135)
• Keywords: language model, llm
• Abstract: Natural Language Processing (NLP) is witnessing a remarkable breakthrough driven by the success of Large Language Models (LLMs). LLMs have gained significant attention across academia and industry for their versatile applications in text generation, question answering, and text summarization. As the landscape of NLP evolves with an increasing number of domain-specific LLMs employing diverse techniques and trained on various corpus, evaluating performance of these models becomes paramount. To quantify the performance, it's crucial to have a comprehensive grasp of existing metrics. Among the evaluation, metrics which quantifying the performance of LLMs play a pivotal role. This paper offers a comprehensive exploration of LLM evaluation from a metrics perspective, providing insights into the selection and interpretation of metrics currently in use. Our main goal is to elucidate their mathematical formulations and statistical interpretations. We shed light on the application of these metrics using recent Biomedical LLMs. Additionally, we offer a succinct comparison of these metrics, aiding researchers in selecting appropriate metrics for diverse tasks. The overarching goal is to furnish researchers with a pragmatic guide for effective LLM evaluation and metric selection, thereby advancing the understanding and application of these large language models.
• 摘要：在大型语言模型 (LLM) 的成功推动下，自然语言处理 (NLP) 正在取得显着突破。法学硕士因其在文本生成、问题回答和文本摘要方面的多功能应用而受到学术界和工业界的广泛关注。随着自然语言处理领域的发展，越来越多的特定领域的法学硕士采用不同的技术并在不同的语料库上进行训练，评估这些模型的性能变得至关重要。为了量化绩效，全面掌握现有指标至关重要。在评估中，量化法学硕士绩效的指标起着至关重要的作用。本文从指标的角度对法学硕士评估进行了全面的探索，提供了对当前使用的指标的选择和解释的见解。我们的主要目标是阐明他们的数学公式和统计解释。我们利用最近的生物医学法学硕士阐明了这些指标的应用。此外，我们还对这些指标进行了简洁的比较，帮助研究人员为不同的任务选择适当的指标。总体目标是为研究人员提供有效的法学硕士评估和指标选择的实用指南，从而促进对这些大型语言模型的理解和应用。

Title: From Bytes to Borsch: Fine-Tuning Gemma and Mistral for the Ukrainian Language Representation

• Authors: Artur Kiulian, Anton Polishko, Mykola Khandoga, Oryna Chubych, Jack Connor, Raghav Ravishankar, Adarsh Shirawalmath
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.09138
• Pdf URL: https://arxiv.org/pdf/2404.09138
• Copy Paste: [[2404.09138]] From Bytes to Borsch: Fine-Tuning Gemma and Mistral for the Ukrainian Language Representation(https://arxiv.org/abs/2404.09138)
• Keywords: language model, llm
• Abstract: In the rapidly advancing field of AI and NLP, generative large language models (LLMs) stand at the forefront of innovation, showcasing unparalleled abilities in text understanding and generation. However, the limited representation of low-resource languages like Ukrainian poses a notable challenge, restricting the reach and relevance of this technology. Our paper addresses this by fine-tuning the open-source Gemma and Mistral LLMs with Ukrainian datasets, aiming to improve their linguistic proficiency and benchmarking them against other existing models capable of processing Ukrainian language. This endeavor not only aims to mitigate language bias in technology but also promotes inclusivity in the digital realm. Our transparent and reproducible approach encourages further NLP research and development. Additionally, we present the Ukrainian Knowledge and Instruction Dataset (UKID) to aid future efforts in language model fine-tuning. Our research not only advances the field of NLP but also highlights the importance of linguistic diversity in AI, which is crucial for cultural preservation, education, and expanding AI's global utility. Ultimately, we advocate for a future where technology is inclusive, enabling AI to communicate effectively across all languages, especially those currently underrepresented.
• 摘要：在快速发展的人工智能和自然语言处理领域，生成式大语言模型（LLM）站在创新的前沿，展示了无与伦比的文本理解和生成能力。然而，像乌克兰语这样的低资源语言的有限代表性构成了一个显着的挑战，限制了这项技术的范围和相关性。我们的论文通过使用乌克兰语数据集对开源 Gemma 和 Mistral LLM 进行微调来解决这个问题，旨在提高它们的语言熟练程度，并将它们与能够处理乌克兰语的其他现有模型进行基准测试。这一努力不仅旨在减轻技术中的语言偏见，而且还促进数字领域的包容性。我们透明且可重复的方法鼓励进一步的 NLP 研究和开发。此外，我们还提供了乌克兰知识和指令数据集（UKID），以帮助未来的语言模型微调工作。我们的研究不仅推动了 NLP 领域的发展，还强调了人工智能中语言多样性的重要性，这对于文化保护、教育和扩大人工智能的全球效用至关重要。最终，我们倡导技术具有包容性的未来，使人工智能能够跨所有语言进行有效沟通，特别是目前代表性不足的语言。

Title: ToNER: Type-oriented Named Entity Recognition with Generative Language Model

• Authors: Guochao Jiang, Ziqin Luo, Yuchen Shi, Dixuan Wang, Jiaqing Liang, Deqing Yang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.09145
• Pdf URL: https://arxiv.org/pdf/2404.09145
• Copy Paste: [[2404.09145]] ToNER: Type-oriented Named Entity Recognition with Generative Language Model(https://arxiv.org/abs/2404.09145)
• Keywords: prompt
• Abstract: In recent years, the fine-tuned generative models have been proven more powerful than the previous tagging-based or span-based models on named entity recognition (NER) task. It has also been found that the information related to entities, such as entity types, can prompt a model to achieve NER better. However, it is not easy to determine the entity types indeed existing in the given sentence in advance, and inputting too many potential entity types would distract the model inevitably. To exploit entity types' merit on promoting NER task, in this paper we propose a novel NER framework, namely ToNER based on a generative model. In ToNER, a type matching model is proposed at first to identify the entity types most likely to appear in the sentence. Then, we append a multiple binary classification task to fine-tune the generative model's encoder, so as to generate the refined representation of the input sentence. Moreover, we add an auxiliary task for the model to discover the entity types which further fine-tunes the model to output more accurate results. Our extensive experiments on some NER benchmarks verify the effectiveness of our proposed strategies in ToNER that are oriented towards entity types' exploitation.
• 摘要：近年来，事实证明，在命名实体识别（NER）任务上，微调的生成模型比以前基于标记或基于跨度的模型更强大。人们还发现，与实体相关的信息，例如实体类型，可以促使模型更好地实现NER。然而，预先确定给定句子中确实存在的实体类型并不容易，并且输入太多潜在的实体类型不可避免地会分散模型的注意力。为了利用实体类型在促进 NER 任务方面的优点，本文提出了一种新颖的 NER 框架，即基于生成模型的 ToNER。在ToNER中，首先提出了类型匹配模型来识别最有可能出现在句子中的实体类型。然后，我们附加多个二元分类任务来微调生成模型的编码器，从而生成输入句子的细化表示。此外，我们为模型添加了一个辅助任务来发现实体类型，从而进一步微调模型以输出更准确的结果。我们对一些 NER 基准进行了广泛的实验，验证了我们在 ToNER 中提出的面向实体类型开发的策略的有效性。

Title: GeMQuAD : Generating Multilingual Question Answering Datasets from Large Language Models using Few Shot Learning

• Authors: Amani Namboori, Shivam Mangale, Andy Rosenbaum, Saleh Soltan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.09163
• Pdf URL: https://arxiv.org/pdf/2404.09163
• Copy Paste: [[2404.09163]] GeMQuAD : Generating Multilingual Question Answering Datasets from Large Language Models using Few Shot Learning(https://arxiv.org/abs/2404.09163)
• Keywords: language model, llm
• Abstract: The emergence of Large Language Models (LLMs) with capabilities like In-Context Learning (ICL) has ushered in new possibilities for data generation across various domains while minimizing the need for extensive data collection and modeling techniques. Researchers have explored ways to use this generated synthetic data to optimize smaller student models for reduced deployment costs and lower latency in downstream tasks. However, ICL-generated data often suffers from low quality as the task specificity is limited with few examples used in ICL. In this paper, we propose GeMQuAD - a semi-supervised learning approach, extending the WeakDAP framework, applied to a dataset generated through ICL with just one example in the target language using AlexaTM 20B Seq2Seq LLM. Through our approach, we iteratively identify high-quality data to enhance model performance, especially for low-resource multilingual setting in the context of Extractive Question Answering task. Our framework outperforms the machine translation-augmented model by 0.22/1.68 F1/EM (Exact Match) points for Hindi and 0.82/1.37 F1/EM points for Spanish on the MLQA dataset, and it surpasses the performance of model trained on an English-only dataset by 5.05/6.50 F1/EM points for Hindi and 3.81/3.69 points F1/EM for Spanish on the same dataset. Notably, our approach uses a pre-trained LLM for generation with no fine-tuning (FT), utilizing just a single annotated example in ICL to generate data, providing a cost-effective development process.
• 摘要：具有上下文学习 (ICL) 等功能的大型语言模型 (LLM) 的出现为跨各个领域的数据生成带来了新的可能性，同时最大限度地减少了对大量数据收集和建模技术的需求。研究人员已经探索了如何使用生成的合成数据来优化较小的学生模型，以降低部署成本并降低下游任务的延迟。然而，ICL 生成的数据通常质量较低，因为任务特异性有限，ICL 中使用的示例很少。在本文中，我们提出了 GeMQuAD - 一种半监督学习方法，扩展了 WeakDAP 框架，应用于通过 ICL 生成的数据集，仅使用 AlexaTM 20B Seq2Seq LLM 的目标语言中的一个示例。通过我们的方法，我们迭代地识别高质量数据以增强模型性能，特别是对于提取问答任务背景下的资源匮乏的多语言环境。在 MLQA 数据集上，我们的框架比机器翻译增强模型的性能优于印地语 0.22/1.68 F1/EM（精确匹配）点和西班牙语 0.82/1.37 F1/EM 点，并且超过了在英语-在同一数据集上，印地语的唯一数据集为 5.05/6.50 F1/EM 点，西班牙语的 F1/EM 为 3.81/3.69 点。值得注意的是，我们的方法使用预先训练的 LLM 进行生成，无需微调 (FT)，仅利用 ICL 中的单个带注释的示例来生成数据，从而提供具有成本效益的开发流程。

Title: Post-Semantic-Thinking: A Robust Strategy to Distill Reasoning Capacity from Large Language Models

• Authors: Xiao Chen, Sihang Zhou, Ke Liang, Xinwang Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.09170
• Pdf URL: https://arxiv.org/pdf/2404.09170
• Copy Paste: [[2404.09170]] Post-Semantic-Thinking: A Robust Strategy to Distill Reasoning Capacity from Large Language Models(https://arxiv.org/abs/2404.09170)
• Keywords: language model, llm, hallucination
• Abstract: Chain of thought finetuning aims to endow small student models with reasoning capacity to improve their performance towards a specific task by allowing them to imitate the reasoning procedure of large language models (LLMs) beyond simply predicting the answer to the question. However, the existing methods 1) generate rationale before the answer, making their answer correctness sensitive to the hallucination in the rationale;2) force the student model to repeat the exact LLMs rationale expression word-after-word, which could have the model biased towards learning the expression in rationale but count against the model from understanding the core logic behind it. Therefore, we propose a robust Post-Semantic-Thinking (PST) strategy to generate answers before rationale. Thanks to this answer-first setting, 1) the answering procedure can escape from the adverse effects caused by hallucinations in the rationale; 2) the complex reasoning procedure is tightly bound with the relatively concise answer, making the reasoning for questions easier with the prior information in the answer; 3) the efficiency of the method can also benefit from the setting since users can stop the generation right after answers are outputted when inference is conducted. Furthermore, the PST strategy loose the constraint against the generated rationale to be close to the LLMs gold standard in the hidden semantic space instead of the vocabulary space, thus making the small student model better comprehend the semantic reasoning logic in rationale. Extensive experiments conducted across 12 reasoning tasks demonstrate the effectiveness of PST.
• 摘要：思想链微调旨在赋予小型学生模型推理能力，通过允许他们模仿大型语言模型（LLM）的推理过程，而不仅仅是简单地预测问题的答案，从而提高他们在特定任务上的表现。然而，现有的方法1）在答案之前生成基本原理，使得他们的答案正确性对基本原理中的幻觉敏感；2）迫使学生模型逐字重复精确的LLM基本原理表达，这可能会使模型产生偏差倾向于学习基本原理上的表达，但不利于模型理解其背后的核心逻辑。因此，我们提出了一种强大的后语义思维（PST）策略，在原理之前生成答案。得益于这种答案优先的设定，1）答题程序可以避免理据幻觉带来的不利影响； 2）复杂的推理过程与相对简洁的答案紧密结合，利用答案中的先验信息使问题的推理变得更加容易； 3）该方法的效率也可以受益于该设置，因为在进行推理时用户可以在输出答案后立即停止生成。此外，PST策略放松了对生成的理据的约束，在隐藏语义空间而不是词汇空间中接近LLM的黄金标准，从而使小学生模型更好地理解理据中的语义推理逻辑。在 12 项推理任务中进行的大量实验证明了 PST 的有效性。

Title: DKE-Research at SemEval-2024 Task 2: Incorporating Data Augmentation with Generative Models and Biomedical Knowledge to Enhance Inference Robustness

• Authors: Yuqi Wang, Zeqiang Wang, Wei Wang, Qi Chen, Kaizhu Huang, Anh Nguyen, Suparna De
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.09206
• Pdf URL: https://arxiv.org/pdf/2404.09206
• Copy Paste: [[2404.09206]] DKE-Research at SemEval-2024 Task 2: Incorporating Data Augmentation with Generative Models and Biomedical Knowledge to Enhance Inference Robustness(https://arxiv.org/abs/2404.09206)
• Keywords: language model
• Abstract: Safe and reliable natural language inference is critical for extracting insights from clinical trial reports but poses challenges due to biases in large pre-trained language models. This paper presents a novel data augmentation technique to improve model robustness for biomedical natural language inference in clinical trials. By generating synthetic examples through semantic perturbations and domain-specific vocabulary replacement and adding a new task for numerical and quantitative reasoning, we introduce greater diversity and reduce shortcut learning. Our approach, combined with multi-task learning and the DeBERTa architecture, achieved significant performance gains on the NLI4CT 2024 benchmark compared to the original language models. Ablation studies validate the contribution of each augmentation method in improving robustness. Our best-performing model ranked 12th in terms of faithfulness and 8th in terms of consistency, respectively, out of the 32 participants.
• 摘要：安全可靠的自然语言推理对于从临床试验报告中提取见解至关重要，但由于大型预训练语言模型的偏差而带来了挑战。本文提出了一种新颖的数据增强技术，以提高临床试验中生物医学自然语言推理的模型稳健性。通过语义扰动和特定领域词汇替换生成合成示例，并添加用于数值和定量推理的新任务，我们引入了更大的多样性并减少了捷径学习。与原始语言模型相比，我们的方法结合多任务学习和 DeBERTa 架构，在 NLI4CT 2024 基准测试中取得了显着的性能提升。消融研究验证了每种增强方法在提高鲁棒性方面的贡献。我们表现​​最好的模型在 32 名参与者中分别在忠实度方面排名第 12 位，在一致性方面排名第 8 位。

Title: Compass: Large Multilingual Language Model for South-east Asia

• Authors: Sophia Maria
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.09220
• Pdf URL: https://arxiv.org/pdf/2404.09220
• Copy Paste: [[2404.09220]] Compass: Large Multilingual Language Model for South-east Asia(https://arxiv.org/abs/2404.09220)
• Keywords: language model, llm
• Abstract: Large language models have exhibited significant proficiency in languages endowed with extensive linguistic resources, such as English and Chinese. Nevertheless, their effectiveness notably diminishes when applied to languages characterized by limited linguistic resources, particularly within the Southeast Asian linguistic landscape, such as Indonesian. The scarcity of linguistic resources for these languages presents challenges associated with inadequate training, restricted vocabulary coverage, and challenging evaluation processes. In response to these exigencies, we have introduced CompassLLM, a large multilingual model specifically tailored for Southeast Asian languages, with the primary aim of supporting the developmental requirements of Shopee. Our methodology encompasses several key strategies. To progressively enhance multilingual proficiencies, we implemented a multi-stage pre-training strategy integrated with curriculum learning, gradually intensifying the focus on low-resource languages. Concurrently, to better accommodate low-resource human instructions, we curated and generated a repository of high-quality multilingual human instructions, culminating the CompassLLM-SFT model through supervised instruction fine-tuning. Finally, to reinforce the model's alignment with human preference behaviors, we have embraced the principle of Direct Preference Optimization (DPO) to obtain CompassLLM-DPO model. Preliminary evaluation of the CompassLLM model yields promising results, with our model surpassing benchmark models like Vicuna-7b-v1.5, Sealion, Falcon and SeaLLM, across diverse evaluation tasks, as verified through both automated and human-driven assessments. Notably, our model exhibits its superior performance in South-east Asia languages, such as Indonesian language.
• 摘要：大型语言模型在拥有丰富语言资源的语言（例如英语和汉语）方面表现出了显着的熟练程度。然而，当应用于语言资源有限的语言时，它们的有效性显着减弱，特别是在东南亚语言环境中，例如印度尼西亚语。这些语言的语言资源稀缺带来了与培训不足、词汇覆盖范围有限和评估过程具有挑战性相关的挑战。针对这些紧急情况，我们推出了CompassLLM，一个专为东南亚语言量身定制的大型多语言模型，主要目的是支持Shopee的发展需求。我们的方法论包含几个关键策略。为逐步提升多语言能力，我们实施与课程学习相结合的多阶段预训练策略，逐步强化对稀缺语言的关注。同时，为了更好地适应资源匮乏的人类指令，我们策划并生成了高质量的多语言人类指令存储库，通过监督指令微调最终形成了 CompassLLM-SFT 模型。最后，为了加强模型与人类偏好行为的一致性，我们采用了直接偏好优化（DPO）原理来获得CompassLLM-DPO模型。 CompassLLM 模型的初步评估取得了可喜的结果，我们的模型在各种评估任务中超越了 Vicuna-7b-v1.5、Sealion、Falcon 和 SeaLLM 等基准模型，并通过自动和人工驱动的评估进行了验证。值得注意的是，我们的模型在东南亚语言（例如印度尼西亚语）中展示了其卓越的性能。

Title: Towards Fast Inference: Exploring and Improving Blockwise Parallel Drafts

• Authors: Taehyeon Kim, Ananda Theertha Suresh, Kishore Papineni, Michael Riley, Sanjiv Kumar, Adrian Benton
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.09221
• Pdf URL: https://arxiv.org/pdf/2404.09221
• Copy Paste: [[2404.09221]] Towards Fast Inference: Exploring and Improving Blockwise Parallel Drafts(https://arxiv.org/abs/2404.09221)
• Keywords: language model
• Abstract: Despite the remarkable strides made by autoregressive language models, their potential is often hampered by the slow inference speeds inherent in sequential token generation. Blockwise parallel decoding (BPD) was proposed by Stern et al. (2018) as a way to improve inference speed of language models. In this paper, we make two contributions to understanding and improving BPD drafts. We first offer an analysis of the token distributions produced by the BPD prediction heads. Secondly, we use this analysis to inform algorithms to improve BPD inference speed by refining the BPD drafts using small n-gram or neural language models. We empirically show that these refined BPD drafts yield a higher average verified prefix length across tasks.
• 摘要：尽管自回归语言模型取得了显着的进步，但它们的潜力常常受到顺序标记生成固有的缓慢推理速度的阻碍。块并行解码（BPD）是由 Stern 等人提出的。 （2018）作为提高语言模型推理速度的一种方法。在本文中，我们为理解和改进 BPD 草案做出了两项贡献。我们首先对 BPD 预测头产生的代币分布进行分析。其次，我们使用此分析来通知算法通过使用小型 n-gram 或神经语言模型细化 BPD 草稿来提高 BPD 推理速度。我们凭经验表明，这些改进的 BPD 草案在任务中产生了更高的平均验证前缀长度。

Title: JaFIn: Japanese Financial Instruction Dataset

• Authors: Kota Tanabe, Masahiro Suzuki, Hiroki Sakaji, Itsuki Noda
• Subjects: cs.CL, cs.CE
• Abstract URL: https://arxiv.org/abs/2404.09260
• Pdf URL: https://arxiv.org/pdf/2404.09260
• Copy Paste: [[2404.09260]] JaFIn: Japanese Financial Instruction Dataset(https://arxiv.org/abs/2404.09260)
• Keywords: language model, llm
• Abstract: We construct an instruction dataset for the large language model (LLM) in the Japanese finance domain. Domain adaptation of language models, including LLMs, is receiving more attention as language models become more popular. This study demonstrates the effectiveness of domain adaptation through instruction tuning. To achieve this, we propose an instruction tuning data in Japanese called JaFIn, the Japanese Financial Instruction Dataset. JaFIn is manually constructed based on multiple data sources, including Japanese government websites, which provide extensive financial knowledge. We then utilize JaFIn to apply instruction tuning for several LLMs, demonstrating that our models specialized in finance have better domain adaptability than the original models. The financial-specialized LLMs created were evaluated using a quantitative Japanese financial benchmark and qualitative response comparisons, showing improved performance over the originals.
• 摘要：我们为日本金融领域的大语言模型（LLM）构建了一个指令数据集。随着语言模型变得越来越流行，包括法学硕士在内的语言模型的领域适应受到越来越多的关注。这项研究证明了通过指令调整进行域适应的有效性。为了实现这一目标，我们提出了一种日语指令调整数据，称为 JaFIn，即日本金融指令数据集。 JaFIn 是根据多个数据源手动构建的，其中包括日本政府网站，提供了广泛的金融知识。然后，我们利用 JaFIn 对多个 LLM 进行指令调整，证明我们的金融专业模型比原始模型具有更好的领域适应性。使用日本定量金融基准和定性反应比较对创建的金融专业法学硕士进行了评估，显示出比原始项目有所提高的表现。

Title: Cross-Data Knowledge Graph Construction for LLM-enabled Educational Question-Answering System: A~Case~Study~at~HCMUT

• Authors: Tuan Bui, Oanh Tran, Phuong Nguyen, Bao Ho, Long Nguyen, Thang Bui, Tho Quan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.09296
• Pdf URL: https://arxiv.org/pdf/2404.09296
• Copy Paste: [[2404.09296]] Cross-Data Knowledge Graph Construction for LLM-enabled Educational Question-Answering System: A~Case~Study~at~HCMUT(https://arxiv.org/abs/2404.09296)
• Keywords: language model, llm, hallucination, retrieval-augmented generation
• Abstract: In today's rapidly evolving landscape of Artificial Intelligence, large language models (LLMs) have emerged as a vibrant research topic. LLMs find applications in various fields and contribute significantly. Despite their powerful language capabilities, similar to pre-trained language models (PLMs), LLMs still face challenges in remembering events, incorporating new information, and addressing domain-specific issues or hallucinations. To overcome these limitations, researchers have proposed Retrieval-Augmented Generation (RAG) techniques, some others have proposed the integration of LLMs with Knowledge Graphs (KGs) to provide factual context, thereby improving performance and delivering more accurate feedback to user queries. Education plays a crucial role in human development and progress. With the technology transformation, traditional education is being replaced by digital or blended education. Therefore, educational data in the digital environment is increasing day by day. Data in higher education institutions are diverse, comprising various sources such as unstructured/structured text, relational databases, web/app-based API access, etc. Constructing a Knowledge Graph from these cross-data sources is not a simple task. This article proposes a method for automatically constructing a Knowledge Graph from multiple data sources and discusses some initial applications (experimental trials) of KG in conjunction with LLMs for question-answering tasks.
• 摘要：在当今快速发展的人工智能领域，大型语言模型（LLM）已成为一个充满活力的研究主题。法学硕士在各个领域都有应用并做出了重大贡献。尽管法学硕士拥有强大的语言能力，类似于预先训练的语言模型 (PLM)，但法学硕士在记忆事件、整合新信息以及解决特定领域问题或幻觉方面仍然面临挑战。为了克服这些限制，研究人员提出了检索增强生成（RAG）技术，其他一些人提出了将法学硕士与知识图谱（KG）集成以提供事实上下文，从而提高性能并向用户查询提供更准确的反馈。教育对于人类的发展和进步起着至关重要的作用。随着技术变革，传统教育正在被数字或混合教育所取代。因此，数字环境下的教育数据日益增多。高等教育机构中的数据多种多样，包括非结构化/结构化文本、关系数据库、基于 Web/应用程序的 API 访问等多种来源。从这些跨数据源构建知识图并不是一项简单的任务。本文提出了一种从多个数据源自动构建知识图的方法，并讨论了 KG 与 LLM 结合用于问答任务的一些初步应用（实验性试验）。

Title: Large Language Models are as persuasive as humans, but why? About the cognitive effort and moral-emotional language of LLM arguments

• Authors: Carlos Carrasco-Farre
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.09329
• Pdf URL: https://arxiv.org/pdf/2404.09329
• Copy Paste: [[2404.09329]] Large Language Models are as persuasive as humans, but why? About the cognitive effort and moral-emotional language of LLM arguments(https://arxiv.org/abs/2404.09329)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) are already as persuasive as humans. However, we know very little about why. This paper investigates the persuasion strategies of LLMs, comparing them with human-generated arguments. Using a dataset of 1,251 participants in an experiment, we analyze the persuaion strategies of LLM-generated and human-generated arguments using measures of cognitive effort (lexical and grammatical complexity) and moral-emotional language (sentiment and moral analysis). The study reveals that LLMs produce arguments that require higher cognitive effort, exhibiting more complex grammatical and lexical structures than human counterparts. Additionally, LLMs demonstrate a significant propensity to engage more deeply with moral language, utilizing both positive and negative moral foundations more frequently than humans. In contrast with previous research, no significant difference was found in the emotional content produced by LLMs and humans. These findings contribute to the discourse on AI and persuasion, highlighting the dual potential of LLMs to both enhance and undermine informational integrity through communication strategies for digital persuasion.
• 摘要：大型语言模型（LLM）已经和人类一样有说服力。然而，我们对其原因知之甚少。本文研究了法学硕士的说服策略，并将其与人类生成的论点进行比较。我们使用由 1,251 名实验参与者组成的数据集，通过认知努力（词汇和语法复杂性）和道德情感语言（情感和道德分析）的测量来分析法学硕士生成和人类生成的论点的说服策略。研究表明，法学硕士提出的论点需要更高的认知努力，表现出比人类同行更复杂的语法和词汇结构。此外，法学硕士表现出更深入地参与道德语言的显着倾向，比人类更频繁地利用积极和消极的道德基础。与之前的研究相比，法学硕士和人类产生的情感内容没有发现显着差异。这些发现促进了关于人工智能和说服的讨论，强调了法学硕士通过数字说服沟通策略增强和破坏信息完整性的双重潜力。

Title: Self-Selected Attention Span for Accelerating Large Language Model Inference

• Authors: Tian Jin, Wanzin Yazar, Zifei Xu, Sayeh Sharify, Xin Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.09336
• Pdf URL: https://arxiv.org/pdf/2404.09336
• Copy Paste: [[2404.09336]] Self-Selected Attention Span for Accelerating Large Language Model Inference(https://arxiv.org/abs/2404.09336)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) can solve challenging tasks. However, their inference computation on modern GPUs is highly inefficient due to the increasing number of tokens they must attend to as they generate new ones. To address this inefficiency, we capitalize on LLMs' problem-solving capabilities to optimize their own inference-time efficiency. We demonstrate with two specific tasks: (a) evaluating complex arithmetic expressions and (b) summarizing news articles. For both tasks, we create custom datasets to fine-tune an LLM. The goal of fine-tuning is twofold: first, to make the LLM learn to solve the evaluation or summarization task, and second, to train it to identify the minimal attention spans required for each step of the task. As a result, the fine-tuned model is able to convert these self-identified minimal attention spans into sparse attention masks on-the-fly during inference. We develop a custom CUDA kernel to take advantage of the reduced context to attend to. We demonstrate that using this custom CUDA kernel improves the throughput of LLM inference by 28%. Our work presents an end-to-end demonstration showing that training LLMs to self-select their attention spans speeds up autoregressive inference in solving real-world tasks.
• 摘要：大型语言模型 (LLM) 可以解决具有挑战性的任务。然而，它们在现代 GPU 上的推理计算效率非常低，因为它们在生成新令牌时必须处理越来越多的令牌。为了解决这种低效率问题，我们利用法学硕士解决问题的能力来优化他们自己的推理时间效率。我们通过两个特定任务进行演示：（a）评估复杂的算术表达式和（b）总结新闻文章。对于这两项任务，我们创建自定义数据集来微调法学硕士。微调的目标有两个：首先，让法学硕士学会解决评估或总结任务，其次，训练它识别任务每一步所需的最小注意力跨度。因此，经过微调的模型能够在推理过程中将这些自我识别的最小注意力跨度即时转换为稀疏注意力掩模。我们开发了一个自定义 CUDA 内核，以利用减少的上下文来处理。我们证明，使用此自定义 CUDA 内核可将 LLM 推理的吞吐量提高 28%。我们的工作提供了一个端到端的演示，表明训练法学硕士自我选择注意力范围可以加快解决现实世界任务时的自回归推理。

Title: Entropy Guided Extrapolative Decoding to Improve Factuality in Large Language Models

• Authors: Souvik Das, Lifeng Jin, Linfeng Song, Haitao Mi, Baolin Peng, Dong Yu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.09338
• Pdf URL: https://arxiv.org/pdf/2404.09338
• Copy Paste: [[2404.09338]] Entropy Guided Extrapolative Decoding to Improve Factuality in Large Language Models(https://arxiv.org/abs/2404.09338)
• Keywords: language model, llm, hallucination, prompt
• Abstract: Large language models (LLMs) exhibit impressive natural language capabilities but suffer from hallucination -- generating content ungrounded in the realities of training data. Recent work has focused on decoding techniques to improve factuality during inference by leveraging LLMs' hierarchical representation of factual knowledge, manipulating the predicted distributions at inference time. Current state-of-the-art approaches refine decoding by contrasting early-exit distributions from a lower layer with the final layer to exploit information related to factuality within the model forward procedure. However, such methods often assume the final layer is the most reliable and the lower layer selection process depends on it. In this work, we first propose extrapolation of critical token probabilities beyond the last layer for more accurate contrasting. We additionally employ layer-wise entropy-guided lower layer selection, decoupling the selection process from the final layer. Experiments demonstrate strong performance - surpassing state-of-the-art on multiple different datasets by large margins. Analyses show different kinds of prompts respond to different selection strategies.
• 摘要：大型语言模型 (LLM) 展现出令人印象深刻的自然语言能力，但会产生幻觉——生成不基于训练数据现实的内容。最近的工作重点是解码技术，通过利用法学硕士事实知识的分层表示，在推理时操纵预测分布，以提高推理过程中的事实性。当前最先进的方法通过将较低层的早期退出分布与最终层进行对比来完善解码，以利用与模型前向过程中的事实相关的信息。然而，此类方法通常假设最后一层是最可靠的，并且较低层的选择过程取决于它。在这项工作中，我们首先提出对最后一层之外的关键令牌概率进行外推，以进行更准确的对比。我们还采用逐层熵引导的下层选择，将选择过程与最终层解耦。实验证明了强大的性能——在多个不同的数据集上大幅超越了最先进的技术。分析显示不同类型的提示会响应不同的选择策略。

Title: Towards Practical Tool Usage for Continually Learning LLMs

• Authors: Jerry Huang, Prasanna Parthasarathi, Mehdi Rezagholizadeh, Sarath Chandar
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.09339
• Pdf URL: https://arxiv.org/pdf/2404.09339
• Copy Paste: [[2404.09339]] Towards Practical Tool Usage for Continually Learning LLMs(https://arxiv.org/abs/2404.09339)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) show an innate skill for solving language based tasks. But insights have suggested an inability to adjust for information or task-solving skills becoming outdated, as their knowledge, stored directly within their parameters, remains static in time. Tool use helps by offloading work to systems that the LLM can access through an interface, but LLMs that use them still must adapt to nonstationary environments for prolonged use, as new tools can emerge and existing tools can change. Nevertheless, tools require less specialized knowledge, therefore we hypothesize they are better suited for continual learning (CL) as they rely less on parametric memory for solving tasks and instead focus on learning when to apply pre-defined tools. To verify this, we develop a synthetic benchmark and follow this by aggregating existing NLP tasks to form a more realistic testing scenario. While we demonstrate scaling model size is not a solution, regardless of tool usage, continual learning techniques can enable tool LLMs to both adapt faster while forgetting less, highlighting their potential as continual learners.
• 摘要：大型语言模型 (LLM) 显示了解决基于语言的任务的天生技能。但洞察表明，他们无法适应信息或解决任务的技能已经过时，因为他们的知识直接存储在参数中，及时保持静态。工具的使用有助于将工作转移到法学硕士可以通过界面访问的系统上，但使用这些工具的法学硕士仍然必须适应非固定环境才能长期使用，因为新工具可能会出现，现有工具也可能会发生变化。然而，工具需要较少的专业知识，因此我们假设它们更适合持续学习（CL），因为它们较少依赖参数记忆来解决任务，而是专注于学习何时应用预定义的工具。为了验证这一点，我们开发了一个综合基准，并通过聚合现有的 NLP 任务来遵循此基准，以形成更现实的测试场景。虽然我们证明缩放模型大小并不是一个解决方案，但无论工具的使用如何，持续学习技术都可以使工具法学硕士能够更快地适应，同时遗忘更少，从而凸显出他们作为持续学习者的潜力。

Title: Understanding the Role of Temperature in Diverse Question Generation by GPT-4

• Authors: Arav Agarwal, Karthik Mittal, Aidan Doyle, Pragnya Sridhar, Zipiao Wan, Jacob Arthur Doughty, Jaromir Savelka, Majd Sakr
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.09366
• Pdf URL: https://arxiv.org/pdf/2404.09366
• Copy Paste: [[2404.09366]] Understanding the Role of Temperature in Diverse Question Generation by GPT-4(https://arxiv.org/abs/2404.09366)
• Keywords: gpt
• Abstract: We conduct a preliminary study of the effect of GPT's temperature parameter on the diversity of GPT4-generated questions. We find that using higher temperature values leads to significantly higher diversity, with different temperatures exposing different types of similarity between generated sets of questions. We also demonstrate that diverse question generation is especially difficult for questions targeting lower levels of Bloom's Taxonomy.
• 摘要：我们初步研究了 GPT 的温度参数对 GPT4 生成问题的多样性的影响。我们发现，使用较高的温度值会导致显着更高的多样性，不同的温度会暴露生成的问题集之间不同类型的相似性。我们还证明，对于针对较低级别的布鲁姆分类法的问题，生成多样化的问题尤其困难。

Title: Mitigating Hallucination in Abstractive Summarization with Domain-Conditional Mutual Information

• Authors: Kyubyung Chae, Jaepill Choi, Yohan Jo, Taesup Kim
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.09480
• Pdf URL: https://arxiv.org/pdf/2404.09480
• Copy Paste: [[2404.09480]] Mitigating Hallucination in Abstractive Summarization with Domain-Conditional Mutual Information(https://arxiv.org/abs/2404.09480)
• Keywords: hallucination
• Abstract: A primary challenge in abstractive summarization is hallucination -- the phenomenon where a model generates plausible text that is absent in the source text. We hypothesize that the domain (or topic) of the source text triggers the model to generate text that is highly probable in the domain, neglecting the details of the source text. To alleviate this model bias, we introduce a decoding strategy based on domain-conditional pointwise mutual information. This strategy adjusts the generation probability of each token by comparing it with the token's marginal probability within the domain of the source text. According to evaluation on the XSUM dataset, our method demonstrates improvement in terms of faithfulness and source relevance. The code is publicly available at \url{https://github.com/qqplot/dcpmi}.
• 摘要：抽象摘要的主要挑战是幻觉——模型生成源文本中不存在的可信文本的现象。我们假设源文本的领域（或主题）触发模型生成该领域中极有可能的文本，而忽略源文本的细节。为了减轻这种模型偏差，我们引入了一种基于域条件逐点互信息的解码策略。该策略通过将每个标记的生成概率与源文本域内标记的边际概率进行比较来调整每个标记的生成概率。根据对 XSUM 数据集的评估，我们的方法在可信度和来源相关性方面表现出改进。该代码可在 \url{https://github.com/qqplot/dcpmi} 上公开获取。

Title: MMCode: Evaluating Multi-Modal Code Large Language Models with Visually Rich Programming Problems

• Authors: Kaixin Li, Yuchen Tian, Qisheng Hu, Ziyang Luo, Jing Ma
• Subjects: cs.CL, cs.CV, cs.SE
• Abstract URL: https://arxiv.org/abs/2404.09486
• Pdf URL: https://arxiv.org/pdf/2404.09486
• Copy Paste: [[2404.09486]] MMCode: Evaluating Multi-Modal Code Large Language Models with Visually Rich Programming Problems(https://arxiv.org/abs/2404.09486)
• Keywords: language model
• Abstract: Programming often involves converting detailed and complex specifications into code, a process during which developers typically utilize visual aids to more effectively convey concepts. While recent developments in Large Multimodal Models have demonstrated remarkable abilities in visual reasoning and mathematical tasks, there is little work on investigating whether these models can effectively interpret visual elements for code generation. To this end, we present MMCode, the first multi-modal coding dataset for evaluating algorithmic problem-solving skills in visually rich contexts. MMCode contains 3,548 questions and 6,620 images collected from real-world programming challenges harvested from 10 code competition websites, presenting significant challenges due to the extreme demand for reasoning abilities. Our experiment results show that current state-of-the-art models struggle to solve these problems. The results highlight the lack of powerful vision-code models, and we hope MMCode can serve as an inspiration for future works in this domain. The data and code are publicly available at https://github.com/happylkx/MMCode.
• 摘要：编程通常涉及将详细且复杂的规范转换为代码，在此过程中开发人员通常利用视觉辅助工具来更有效地传达概念。虽然大型多模态模型的最新发展在视觉推理和数学任务中表现出了卓越的能力，但很少有人研究这些模型是否可以有效地解释视觉元素以生成代码。为此，我们提出了 MMCode，这是第一个多模态编码数据集，用于在视觉丰富的环境中评估算法解决问题的技能。 MMCode包含从10个代码竞赛网站收集的现实世界编程挑战中收集的3,548个问题和6,620张图像，由于对推理能力的极端要求，提出了巨大的挑战。我们的实验结果表明，当前最先进的模型很难解决这些问题。结果凸显了强大的视觉代码模型的缺乏，我们希望 MMCode 能够为该领域的未来工作提供启发。数据和代码可在 https://github.com/happylkx/MMCode 上公开获取。

Title: Bridging the Gap between Different Vocabularies for LLM Ensemble

• Authors: Yangyifan Xu, Jinliang Lu, Jiajun Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.09492
• Pdf URL: https://arxiv.org/pdf/2404.09492
• Copy Paste: [[2404.09492]] Bridging the Gap between Different Vocabularies for LLM Ensemble(https://arxiv.org/abs/2404.09492)
• Keywords: language model, llm
• Abstract: Ensembling different large language models (LLMs) to unleash their complementary potential and harness their individual strengths is highly valuable. Nevertheless, vocabulary discrepancies among various LLMs have constrained previous studies to either selecting or blending completely generated outputs. This limitation hinders the dynamic correction and enhancement of outputs during the generation process, resulting in a limited capacity for effective ensemble. To address this issue, we propose a novel method to Ensemble LLMs via Vocabulary Alignment (EVA). EVA bridges the lexical gap among various LLMs, enabling meticulous ensemble at each generation step. Specifically, we first learn mappings between the vocabularies of different LLMs with the assistance of overlapping tokens. Subsequently, these mappings are employed to project output distributions of LLMs into a unified space, facilitating a fine-grained ensemble. Finally, we design a filtering strategy to exclude models that generate unfaithful tokens. Experimental results on commonsense reasoning, arithmetic reasoning, machine translation, and data-to-text generation tasks demonstrate the superiority of our approach compared with individual LLMs and previous ensemble methods conducted on complete outputs. Further analyses confirm that our approach can leverage knowledge from different language models and yield consistent improvement.
• 摘要：整合不同的大语言模型（LLM）来释放它们的互补潜力并利用它们的各自优势是非常有价值的。然而，各种法学硕士之间的词汇差异限制了之前的研究只能选择或混合完全生成的输出。这种限制阻碍了生成过程中输出的动态校正和增强，导致有效集成的能力有限。为了解决这个问题，我们提出了一种通过词汇对齐（EVA）来集成法学硕士的新方法。 EVA 弥合了各种法学硕士之间的词汇差距，使每个生成步骤都能够进行细致的集成。具体来说，我们首先在重叠标记的帮助下学习不同法学硕士词汇表之间的映射。随后，这些映射被用来将法学硕士的输出分布投影到一个统一的空间中，从而促进细粒度的集成。最后，我们设计了一种过滤策略来排除生成不忠实令牌的模型。常识推理、算术推理、机器翻译和数据到文本生成任务的实验结果表明，与单独的法学硕士和之前在完整输出上进行的集成方法相比，我们的方法具有优越性。进一步的分析证实，我们的方法可以利用来自不同语言模型的知识并产生持续的改进。

Title: Large language models and linguistic intentionality

• Authors: Jumbly Grindrod
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.09576
• Pdf URL: https://arxiv.org/pdf/2404.09576
• Copy Paste: [[2404.09576]] Large language models and linguistic intentionality(https://arxiv.org/abs/2404.09576)
• Keywords: language model, gpt, llm, chat
• Abstract: Do large language models like Chat-GPT or LLaMa meaningfully use the words they produce? Or are they merely clever prediction machines, simulating language use by producing statistically plausible text? There have already been some initial attempts to answer this question by showing that these models meet the criteria for entering meaningful states according to metasemantic theories of mental content. In this paper, I will argue for a different approach - that we should instead consider whether language models meet the criteria given by our best metasemantic theories of linguistic content. In that vein, I will illustrate how this can be done by applying two such theories to the case of language models: Gareth Evans' (1982) account of naming practices and Ruth Millikan's (1984, 2004, 2005) teleosemantics. In doing so, I will argue that it is a mistake to think that the failure of LLMs to meet plausible conditions for mental intentionality thereby renders their outputs meaningless, and that a distinguishing feature of linguistic intentionality - dependency on a pre-existing linguistic system - allows for the plausible result LLM outputs are meaningful.
• 摘要：像 Chat-GPT 或 LLaMa 这样的大型语言模型是否有意义地使用它们生成的单词？或者它们只是聪明的预测机器，通过生成统计上可信的文本来模拟语言的使用？已经有一些初步尝试来回答这个问题，表明这些模型符合根据心理内容的元语义理论进入有意义状态的标准。在本文中，我将主张一种不同的方法——我们应该考虑语言模型是否符合我们最好的语言内容元语义理论给出的标准。本着这种精神，我将通过将两种这样的理论应用于语言模型的情况来说明如何做到这一点：Gareth Evans (1982) 对命名实践的描述和 Ruth Millikan (1984, 2004, 2005) 的目的语义学。在这样做时，我认为，认为法学硕士未能满足心理意向性的合理条件从而使其输出毫无意义是错误的，并且语言意向性的一个显着特征——对预先存在的语言系统的依赖——是错误的。允许LLM输出的合理结果是有意义的。

Title: Transformers, Contextualism, and Polysemy

• Authors: Jumbly Grindrod
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.09577
• Pdf URL: https://arxiv.org/pdf/2404.09577
• Copy Paste: [[2404.09577]] Transformers, Contextualism, and Polysemy(https://arxiv.org/abs/2404.09577)
• Keywords: language model, gpt, chat
• Abstract: The transformer architecture, introduced by Vaswani et al. (2017), is at the heart of the remarkable recent progress in the development of language models, including famous chatbots such as Chat-gpt and Bard. In this paper, I argue that we an extract from the way the transformer architecture works a picture of the relationship between context and meaning. I call this the transformer picture, and I argue that it is a novel with regard to two related philosophical debates: the contextualism debate regarding the extent of context-sensitivity across natural language, and the polysemy debate regarding how polysemy should be captured within an account of word meaning. Although much of the paper merely tries to position the transformer picture with respect to these two debates, I will also begin to make the case for the transformer picture.
• 摘要：Vaswani 等人提出的 Transformer 架构。 （2017），是语言模型开发最近取得的显着进展的核心，包括 Chat-gpt 和 Bard 等著名的聊天机器人。在本文中，我认为我们从 Transformer 架构的工作方式中提取了上下文和含义之间关系的图景。我将其称为变压器图片，并且我认为这是关于两个相关的哲学辩论的小说：关于跨自然语言的语境敏感性程度的语境主义辩论，以及关于如何在帐户中捕获多义性的多义性辩论的词义。尽管本文的大部分内容只是试图根据这两个争论来定位变压器图像，但我也将开始为变压器图像提供理由。

Title: Modelling Language

• Authors: Jumbly Grindrod
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.09579
• Pdf URL: https://arxiv.org/pdf/2404.09579
• Copy Paste: [[2404.09579]] Modelling Language(https://arxiv.org/abs/2404.09579)
• Keywords: language model
• Abstract: This paper argues that large language models have a valuable scientific role to play in serving as scientific models of a language. Linguistic study should not only be concerned with the cognitive processes behind linguistic competence, but also with language understood as an external, social entity. Once this is recognized, the value of large language models as scientific models becomes clear. This paper defends this position against a number of arguments to the effect that language models provide no linguistic insight. It also draws upon recent work in philosophy of science to show how large language models could serve as scientific models.
• 摘要：本文认为，大型语言模型在作为语言的科学模型方面可以发挥有价值的科学作用。语言学研究不仅应该关注语言能力背后的认知过程，还应该关注被理解为外部社会实体的语言。一旦认识到这一点，大型语言模型作为科学模型的价值就变得清晰起来。本文针对语言模型不提供语言洞察力的许多论点捍卫了这一立场。它还利用科学哲学领域的最新成果来展示大型语言模型如何充当科学模型。

Title: Improving Recall of Large Language Models: A Model Collaboration Approach for Relational Triple Extraction

• Authors: Zepeng Ding, Wenhao Huang, Jiaqing Liang, Deqing Yang, Yanghua Xiao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.09593
• Pdf URL: https://arxiv.org/pdf/2404.09593
• Copy Paste: [[2404.09593]] Improving Recall of Large Language Models: A Model Collaboration Approach for Relational Triple Extraction(https://arxiv.org/abs/2404.09593)
• Keywords: language model, prompt
• Abstract: Relation triple extraction, which outputs a set of triples from long sentences, plays a vital role in knowledge acquisition. Large language models can accurately extract triples from simple sentences through few-shot learning or fine-tuning when given appropriate instructions. However, they often miss out when extracting from complex sentences. In this paper, we design an evaluation-filtering framework that integrates large language models with small models for relational triple extraction tasks. The framework includes an evaluation model that can extract related entity pairs with high precision. We propose a simple labeling principle and a deep neural network to build the model, embedding the outputs as prompts into the extraction process of the large model. We conduct extensive experiments to demonstrate that the proposed method can assist large language models in obtaining more accurate extraction results, especially from complex sentences containing multiple relational triples. Our evaluation model can also be embedded into traditional extraction models to enhance their extraction precision from complex sentences.
• 摘要：关系三元组提取，从长句子中输出一组三元组，在知识获取中起着至关重要的作用。大型语言模型可以通过少量学习或在给出适当指令时进行微调，从简单句子中准确提取三元组。然而，他们在提取复杂句子时经常会漏掉一些内容。在本文中，我们设计了一个评估过滤框架，它将大型语言模型与小型模型集成在一起，用于关系三元组提取任务。该框架包括一个评估模型，可以高精度提取相关实体对。我们提出了一个简单的标记原则和一个深度神经网络来构建模型，将输出作为提示嵌入到大型模型的提取过程中。我们进行了大量的实验来证明所提出的方法可以帮助大型语言模型获得更准确的提取结果，尤其是包含多个关系三元组的复杂句子。我们的评估模型还可以嵌入到传统的提取模型中，以提高复杂句子的提取精度。

Title: If there's a Trigger Warning, then where's the Trigger? Investigating Trigger Warnings at the Passage Level

• Authors: Matti Wiegmann, Jennifer Rakete, Magdalena Wolska, Benno Stein, Martin Potthast
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2404.09615
• Pdf URL: https://arxiv.org/pdf/2404.09615
• Copy Paste: [[2404.09615]] If there's a Trigger Warning, then where's the Trigger? Investigating Trigger Warnings at the Passage Level(https://arxiv.org/abs/2404.09615)
• Keywords: prompt
• Abstract: Trigger warnings are labels that preface documents with sensitive content if this content could be perceived as harmful by certain groups of readers. Since warnings about a document intuitively need to be shown before reading it, authors usually assign trigger warnings at the document level. What parts of their writing prompted them to assign a warning, however, remains unclear. We investigate for the first time the feasibility of identifying the triggering passages of a document, both manually and computationally. We create a dataset of 4,135 English passages, each annotated with one of eight common trigger warnings. In a large-scale evaluation, we then systematically evaluate the effectiveness of fine-tuned and few-shot classifiers, and their generalizability. We find that trigger annotation belongs to the group of subjective annotation tasks in NLP, and that automatic trigger classification remains challenging but feasible.
• 摘要：触发警告是在包含敏感内容的文档前面添加的标签，如果该内容可能被某些读者群体视为有害的话。由于有关文档的警告需要在阅读之前直观地显示出来，因此作者通常在文档级别分配触发警告。然而，他们写作的哪些部分促使他们发出警告，目前尚不清楚。我们首次研究了手动和计算方式识别文档触发段落的可行性。我们创建了一个包含 4,135 个英语段落的数据集，每个段落都带有八个常见触发警告之一的注释。在大规模评估中，我们系统地评估微调和少样本分类器的有效性及其泛化性。我们发现触发标注属于 NLP 中的主观标注任务组，自动触发分类仍然具有挑战性但可行。

Title: Multi-News+: Cost-efficient Dataset Cleansing via LLM-based Data Annotation

• Authors: Juhwan Choi, Jungmin Yun, Kyohoon Jin, YoungBin Kim
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.09682
• Pdf URL: https://arxiv.org/pdf/2404.09682
• Copy Paste: [[2404.09682]] Multi-News+: Cost-efficient Dataset Cleansing via LLM-based Data Annotation(https://arxiv.org/abs/2404.09682)
• Keywords: language model, llm, chain-of-thought
• Abstract: The quality of the dataset is crucial for ensuring optimal performance and reliability of downstream task models. However, datasets often contain noisy data inadvertently included during the construction process. Numerous attempts have been made to correct this issue through human annotators. However, hiring and managing human annotators is expensive and time-consuming. As an alternative, recent studies are exploring the use of large language models (LLMs) for data annotation. In this study, we present a case study that extends the application of LLM-based data annotation to enhance the quality of existing datasets through a cleansing strategy. Specifically, we leverage approaches such as chain-of-thought (CoT) and majority voting to imitate human annotation and classify unrelated documents from the Multi-News dataset, which is widely used for the multi-document summarization task. Through our proposed cleansing method, we introduce an enhanced Multi-News+. By employing LLMs for data cleansing, we demonstrate an efficient and effective approach to improving dataset quality without relying on expensive human annotation efforts.
• 摘要：数据集的质量对于确保下游任务模型的最佳性能和可靠性至关重要。然而，数据集通常包含在构建过​​程中无意中包含的噪声数据。人们已经进行了许多尝试来通过人类注释者来纠正这个问题。然而，雇用和管理人工注释者既昂贵又耗时。作为替代方案，最近的研究正在探索使用大型语言模型 (LLM) 进行数据注释。在本研究中，我们提出了一个案例研究，该案例研究扩展了基于法学硕士的数据注释的应用，以通过清理策略提高现有数据集的质量。具体来说，我们利用思想链（CoT）和多数投票等方法来模仿人类注释并对多新闻数据集中的不相关文档进行分类，该数据集广泛用于多文档摘要任务。通过我们提出的清理方法，我们引入了增强的 Multi-News+。通过采用法学硕士进行数据清理，我们展示了一种高效且有效的方法来提高数据集质量，而无需依赖昂贵的人工注释工作。

Title: Are Large Language Models Reliable Argument Quality Annotators?

• Authors: Nailia Mirzakhmedova, Marcel Gohsen, Chia Hao Chang, Benno Stein
• Subjects: cs.CL, cs.AI, cs.ET
• Abstract URL: https://arxiv.org/abs/2404.09696
• Pdf URL: https://arxiv.org/pdf/2404.09696
• Copy Paste: [[2404.09696]] Are Large Language Models Reliable Argument Quality Annotators?(https://arxiv.org/abs/2404.09696)
• Keywords: language model, llm
• Abstract: Evaluating the quality of arguments is a crucial aspect of any system leveraging argument mining. However, it is a challenge to obtain reliable and consistent annotations regarding argument quality, as this usually requires domain-specific expertise of the annotators. Even among experts, the assessment of argument quality is often inconsistent due to the inherent subjectivity of this task. In this paper, we study the potential of using state-of-the-art large language models (LLMs) as proxies for argument quality annotators. To assess the capability of LLMs in this regard, we analyze the agreement between model, human expert, and human novice annotators based on an established taxonomy of argument quality dimensions. Our findings highlight that LLMs can produce consistent annotations, with a moderately high agreement with human experts across most of the quality dimensions. Moreover, we show that using LLMs as additional annotators can significantly improve the agreement between annotators. These results suggest that LLMs can serve as a valuable tool for automated argument quality assessment, thus streamlining and accelerating the evaluation of large argument datasets.
• 摘要：评估论证的质量是任何利用论证挖掘的系统的一个重要方面。然而，获得有关论证质量的可靠且一致的注释是一个挑战，因为这通常需要注释者具有特定领域的专业知识。即使在专家之间，由于这项任务固有的主观性，对论证质量的评估也常常不一致。在本文中，我们研究了使用最先进的大型语言模型（LLM）作为论证质量注释器的代理的潜力。为了评估法学硕士在这方面的能力，我们根据既定的论点质量维度分类法分析了模型、人类专家和人类新手注释者之间的一致性。我们的研究结果强调，法学硕士可以产生一致的注释，在大多数质量维度上与人类专家具有较高的一致性。此外，我们表明使用法学硕士作为附加注释器可以显着提高注释器之间的一致性。这些结果表明，法学硕士可以作为自动论证质量评估的宝贵工具，从而简化和加速大型论证数据集的评估。

Title: Unveiling Imitation Learning: Exploring the Impact of Data Falsity to Large Language Model

• Authors: Hyunsoo Cho
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.09717
• Pdf URL: https://arxiv.org/pdf/2404.09717
• Copy Paste: [[2404.09717]] Unveiling Imitation Learning: Exploring the Impact of Data Falsity to Large Language Model(https://arxiv.org/abs/2404.09717)
• Keywords: language model, gpt, llm, chat
• Abstract: Many recent studies endeavor to improve open-source language models through imitation learning, and re-training on the synthetic instruction data from state-of-the-art proprietary models like ChatGPT and GPT-4. However, the innate nature of synthetic data inherently contains noisy data, giving rise to a substantial presence of low-quality data replete with erroneous responses, and flawed reasoning. Although we intuitively grasp the potential harm of noisy data, we lack a quantitative understanding of its impact. To this end, this paper explores the correlation between the degree of noise and its impact on language models through instruction tuning. We first introduce the Falsity-Controllable (FACO) dataset, which comprises pairs of true answers with corresponding reasoning, as well as false pairs to manually control the falsity ratio of the dataset.Through our extensive experiments, we found multiple intriguing findings of the correlation between the factuality of the dataset and instruction tuning: Specifically, we verified falsity of the instruction is highly relevant to various benchmark scores. Moreover, when LLMs are trained with false instructions, they learn to lie and generate fake unfaithful answers, even though they know the correct answer for the user request. Additionally, we noted that once the language model is trained with a dataset contaminated by noise, restoring its original performance is possible, but it failed to reach full performance.
• 摘要：最近的许多研究都致力于通过模仿学习以及对来自最先进的专有模型（如 ChatGPT 和 GPT-4）的合成指令数据进行重新训练来改进开源语言模型。然而，合成数据的本质本质上包含噪声数据，导致大量存在充满错误响应和有缺陷推理的低质量数据。尽管我们直观地掌握了噪声数据的潜在危害，但我们对其影响缺乏定量理解。为此，本文通过指令调优探讨噪声程度与其对语言模型影响之间的相关性。我们首先介绍虚假可控（FACO）数据集，它包含具有相应推理的真实答案对，以及手动控制数据集虚假率的虚假答案对。通过我们广泛的实验，我们发现了多个有趣的相关性发现数据集的真实性和指令调整之间的关系：具体来说，我们验证了指令的虚假性与各种基准分数高度相关。此外，当法学硕士接受错误指令训练时，他们会学会撒谎并生成虚假的不忠实答案，即使他们知道用户请求的正确答案。此外，我们注意到，一旦使用受噪声污染的数据集训练语言模型，恢复其原始性能是可能的，但它无法达到全部性能。

Title: Personalized Collaborative Fine-Tuning for On-Device Large Language Models

• Authors: Nicolas Wagner, Dongyang Fan, Martin Jaggi
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.09753
• Pdf URL: https://arxiv.org/pdf/2404.09753
• Copy Paste: [[2404.09753]] Personalized Collaborative Fine-Tuning for On-Device Large Language Models(https://arxiv.org/abs/2404.09753)
• Keywords: language model
• Abstract: We explore on-device self-supervised collaborative fine-tuning of large language models with limited local data availability. Taking inspiration from the collaborative learning community, we introduce three distinct trust-weighted gradient aggregation schemes: weight similarity-based, prediction similarity-based and validation performance-based. To minimize communication overhead, we integrate Low-Rank Adaptation (LoRA) and only exchange LoRA weight updates. Our protocols, driven by prediction and performance metrics, surpass both FedAvg and local fine-tuning methods, which is particularly evident in realistic scenarios with more diverse local data distributions. The results underscore the effectiveness of our approach in addressing heterogeneity and scarcity within local datasets.
• 摘要：我们探索在本地数据可用性有限的情况下对大型语言模型进行设备上自监督协作微调。受到协作学习社区的启发，我们引入了三种不同的信任加权梯度聚合方案：基于权重相似性、基于预测相似性和基于验证性能。为了最大限度地减少通信开销，我们集成了低秩适应 (LoRA)，并且仅交换 LoRA 权重更新。我们的协议由预测和性能指标驱动，超越了 FedAvg 和本地微调方法，这在本地数据分布更加多样化的现实场景中尤其明显。结果强调了我们的方法在解决本地数据集中的异质性和稀缺性方面的有效性。

Title: Resilience of Large Language Models for Noisy Instructions

• Authors: Bin Wang, Chengwei Wei, Zhengyuan Liu, Geyu Lin, Nancy F. Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.09754
• Pdf URL: https://arxiv.org/pdf/2404.09754
• Copy Paste: [[2404.09754]] Resilience of Large Language Models for Noisy Instructions(https://arxiv.org/abs/2404.09754)
• Keywords: language model, llm
• Abstract: As the rapidly advancing domain of natural language processing (NLP), large language models (LLMs) have emerged as powerful tools for interpreting human commands and generating text across various tasks. Nonetheless, the resilience of LLMs to handle text containing inherent errors, stemming from human interactions and collaborative systems, has not been thoroughly explored. Our study investigates the resilience of LLMs against five common types of disruptions including 1) ASR (Automatic Speech Recognition) errors, 2) OCR (Optical Character Recognition) errors, 3) grammatical mistakes, 4) typographical errors, and 5) distractive content. We aim to investigate how these models react by deliberately embedding these errors into instructions. Our findings reveal that while some LLMs show a degree of resistance to certain types of noise, their overall performance significantly suffers. This emphasizes the importance of further investigation into enhancing model resilience. In response to the observed decline in performance, our study also evaluates a "re-pass" strategy, designed to purify the instructions of noise before the LLMs process them. Our analysis indicates that correcting noisy instructions, particularly for open-source LLMs, presents significant challenges.
• 摘要：作为快速发展的自然语言处理 (NLP) 领域，大型语言模型 (LLM) 已成为解释人类命令和跨各种任务生成文本的强大工具。尽管如此，法学硕士处理包含源自人类互动和协作系统的固有错误的文本的弹性尚未得到彻底探索。我们的研究调查了法学硕士对五种常见干扰类型的恢复能力，包括 1) ASR（自动语音识别）错误、2) OCR（光学字符识别）错误、3) 语法错误、4) 印刷错误和 5) 分散注意力的内容。我们的目标是通过故意将这些错误嵌入到指令中来研究这些模型的反应。我们的研究结果表明，虽然一些法学硕士对某些类型的噪音表现出一定程度的抵抗力，但他们的整体表现却受到了严重影响。这强调了进一步研究增强模型弹性的重要性。为了应对观察到的性能下降，我们的研究还评估了“重新通过”策略，旨在在法学硕士处理指令之前净化噪音指令。我们的分析表明，纠正嘈杂的指令，特别是对于开源法学硕士来说，提出了重大挑战。

Title: KG-CTG: Citation Generation through Knowledge Graph-guided Large Language Models

• Authors: Avinash Anand, Mohit Gupta, Kritarth Prasad, Ujjwal Goel, Naman Lal, Astha Verma, Rajiv Ratn Shah
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.09763
• Pdf URL: https://arxiv.org/pdf/2404.09763
• Copy Paste: [[2404.09763]] KG-CTG: Citation Generation through Knowledge Graph-guided Large Language Models(https://arxiv.org/abs/2404.09763)
• Keywords: language model, llm, prompt
• Abstract: Citation Text Generation (CTG) is a task in natural language processing (NLP) that aims to produce text that accurately cites or references a cited document within a source document. In CTG, the generated text draws upon contextual cues from both the source document and the cited paper, ensuring accurate and relevant citation information is provided. Previous work in the field of citation generation is mainly based on the text summarization of documents. Following this, this paper presents a framework, and a comparative study to demonstrate the use of Large Language Models (LLMs) for the task of citation generation. Also, we have shown the improvement in the results of citation generation by incorporating the knowledge graph relations of the papers in the prompt for the LLM to better learn the relationship between the papers. To assess how well our model is performing, we have used a subset of standard S2ORC dataset, which only consists of computer science academic research papers in the English Language. Vicuna performs best for this task with 14.15 Meteor, 12.88 Rouge-1, 1.52 Rouge-2, and 10.94 Rouge-L. Also, Alpaca performs best, and improves the performance by 36.98% in Rouge-1, and 33.14% in Meteor by including knowledge graphs.
• 摘要：引文文本生成 (CTG) 是自然语言处理 (NLP) 中的一项任务，旨在生成准确引用或引用源文档中被引用文档的文本。在 CTG 中，生成的文本利用源文档和引用论文的上下文线索，确保提供准确且相关的引用信息。先前引文生成领域的工作主要基于文档的文本摘要。接下来，本文提出了一个框架和一项比较研究，以展示大型语言模型 (LLM) 在引文生成任务中的使用。此外，我们还通过将论文的知识图谱关系纳入提示中，展示了引文生成结果的改进，以便法学硕士更好地学习论文之间的关系。为了评估我们的模型的表现，我们使用了标准 S2ORC 数据集的子集，该数据集仅包含英语计算机科学学术研究论文。 Vicuna 使用 14.15 Meteor、12.88 Rouge-1、1.52 Rouge-2 和 10.94 Rouge-L 在此任务中表现最佳。此外，Alpaca 表现最好，通过包含知识图谱，在 Rouge-1 中性能提高了 36.98%，在 Meteor 中提高了 33.14%。

Title: Benchmarking Llama2, Mistral, Gemma and GPT for Factuality, Toxicity, Bias and Propensity for Hallucinations

• Authors: David Nadeau, Mike Kroutikov, Karen McNeil, Simon Baribeau
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.09785
• Pdf URL: https://arxiv.org/pdf/2404.09785
• Copy Paste: [[2404.09785]] Benchmarking Llama2, Mistral, Gemma and GPT for Factuality, Toxicity, Bias and Propensity for Hallucinations(https://arxiv.org/abs/2404.09785)
• Keywords: language model, gpt, hallucination, prompt
• Abstract: This paper introduces fourteen novel datasets for the evaluation of Large Language Models' safety in the context of enterprise tasks. A method was devised to evaluate a model's safety, as determined by its ability to follow instructions and output factual, unbiased, grounded, and appropriate content. In this research, we used OpenAI GPT as point of comparison since it excels at all levels of safety. On the open-source side, for smaller models, Meta Llama2 performs well at factuality and toxicity but has the highest propensity for hallucination. Mistral hallucinates the least but cannot handle toxicity well. It performs well in a dataset mixing several tasks and safety vectors in a narrow vertical domain. Gemma, the newly introduced open-source model based on Google Gemini, is generally balanced but trailing behind. When engaging in back-and-forth conversation (multi-turn prompts), we find that the safety of open-source models degrades significantly. Aside from OpenAI's GPT, Mistral is the only model that still performed well in multi-turn tests.
• 摘要：本文介绍了十四个新颖的​​数据集，用于评估企业任务背景下大型语言模型的安全性。设计了一种方法来评估模型的安全性，这是根据其遵循指令并输出事实、公正、接地和适当内容的能力来确定的。在本研究中，我们使用 OpenAI GPT 作为比较点，因为它在各个安全级别上都表现出色。在开源方面，对于较小的模型，Meta Llama2 在真实性和毒性方面表现良好，但产生幻觉的倾向最高。米斯特拉尔产生的幻觉最少，但不能很好地处理毒性。它在狭窄垂直域中混合多个任务和安全向量的数据集中表现良好。基于Google Gemini新推出的开源模型Gemma总体平衡但落后。当进行来回对话（多轮提示）时，我们发现开源模型的安全性显着下降。除了 OpenAI 的 GPT 之外，Mistral 是唯一在多轮测试中仍然表现良好的模型。

Title: Impact of Preference Noise on the Alignment Performance of Generative Language Models

• Authors: Yang Gao, Dana Alon, Donald Metzler
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.09824
• Pdf URL: https://arxiv.org/pdf/2404.09824
• Copy Paste: [[2404.09824]] Impact of Preference Noise on the Alignment Performance of Generative Language Models(https://arxiv.org/abs/2404.09824)
• Keywords: language model
• Abstract: A key requirement in developing Generative Language Models (GLMs) is to have their values aligned with human values. Preference-based alignment is a widely used paradigm for this purpose, in which preferences over generation pairs are first elicited from human annotators or AI systems, and then fed into some alignment techniques, e.g., Direct Preference Optimization. However, a substantial percent (20 - 40%) of the preference pairs used in GLM alignment are noisy, and it remains unclear how the noise affects the alignment performance and how to mitigate its negative impact. In this paper, we propose a framework to inject desirable amounts and types of noise to the preferences, and systematically study the impact of preference noise on the alignment performance in two tasks (summarization and dialogue generation). We find that the alignment performance can be highly sensitive to the noise rates in the preference data: e.g., a 10 percentage points (pp) increase of the noise rate can lead to 30 pp drop in the alignment performance (in win rate). To mitigate the impact of noise, confidence-based data filtering shows significant benefit when certain types of noise are present. We hope our work can help the community better understand and mitigate the impact of preference noise in GLM alignment.
• 摘要：开发生成语言模型（GLM）的一个关键要求是使其价值观与人类价值观保持一致。基于偏好的对齐是用于此目的的一种广泛使用的范例，其中首先从人类注释者或人工智能系统中引出对代对的偏好，然后将其输入到一些对齐技术中，例如直接偏好优化。然而，GLM 对齐中使用的偏好对中有很大一部分 (20 - 40%) 是有噪声的，并且目前尚不清楚噪声如何影响对齐性能以及如何减轻其负面影响。在本文中，我们提出了一个框架，向偏好注入所需数量和类型的噪声，并系统地研究偏好噪声对两个任务（摘要和对话生成）中对齐性能的影响。我们发现，对齐性能对偏好数据中的噪声率高度敏感：例如，噪声率增加 10 个百分点 (pp) 可能会导致对齐性能（获胜率）下降 30 个 pp。为了减轻噪声的影响，当存在某些类型的噪声时，基于置信度的数据过滤显示出显着的优势。我们希望我们的工作能够帮助社区更好地理解和减轻 GLM 对齐中偏好噪声的影响。

Title: Negation Triplet Extraction with Syntactic Dependency and Semantic Consistency

• Authors: Yuchen Shi, Deqing Yang, Jingping Liu, Yanghua Xiao, Zongyu Wang, Huimin Xu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.09830
• Pdf URL: https://arxiv.org/pdf/2404.09830
• Copy Paste: [[2404.09830]] Negation Triplet Extraction with Syntactic Dependency and Semantic Consistency(https://arxiv.org/abs/2404.09830)
• Keywords: language model
• Abstract: Previous works of negation understanding mainly focus on negation cue detection and scope resolution, without identifying negation subject which is also significant to the downstream tasks. In this paper, we propose a new negation triplet extraction (NTE) task which aims to extract negation subject along with negation cue and scope. To achieve NTE, we devise a novel Syntax&Semantic-Enhanced Negation Extraction model, namely SSENE, which is built based on a generative pretrained language model (PLM) {of Encoder-Decoder architecture} with a multi-task learning framework. Specifically, the given sentence's syntactic dependency tree is incorporated into the PLM's encoder to discover the correlations between the negation subject, cue and scope. Moreover, the semantic consistency between the sentence and the extracted triplet is ensured by an auxiliary task learning. Furthermore, we have constructed a high-quality Chinese dataset NegComment based on the users' reviews from the real-world platform of Meituan, upon which our evaluations show that SSENE achieves the best NTE performance compared to the baselines. Our ablation and case studies also demonstrate that incorporating the syntactic information helps the PLM's recognize the distant dependency between the subject and cue, and the auxiliary task learning is helpful to extract the negation triplets with more semantic consistency.
• 摘要：先前的否定理解工作主要集中在否定线索检测和范围解析上，而没有识别对下游任务也很重要的否定主题。在本文中，我们提出了一种新的否定三元组提取（NTE）任务，旨在提取否定主题以及否定线索和范围。为了实现 NTE，我们设计了一种新颖的语法和语义增强否定提取模型，即 SSENE，它是基于具有多任务学习框架的生成式预训练语言模型（PLM）{编码器-解码器架构}构建的。具体来说，给定句子的句法依存树被合并到 PLM 的编码器中，以发现否定主语、提示和范围之间的相关性。此外，句子和提取的三元组之间的语义一致性是通过辅助任务学习来保证的。此外，我们根据美团真实世界平台的用户评论构建了一个高质量的中文数据集 NegComment，根据该数据集，我们的评估表明，与基线相比，SSENE 实现了最佳的 NTE 性能。我们的消融和案例研究还表明，合并句法信息有助于 PLM 识别主语和提示之间的远程依赖关系，辅助任务学习有助于提取具有更多语义一致性的否定三元组。

Title: Glitch Tokens in Large Language Models: Categorization Taxonomy and Effective Detection

• Authors: Yuxi Li, Yi Liu, Gelei Deng, Ying Zhang, Wenjia Song, Ling Shi, Kailong Wang, Yuekang Li, Yang Liu, Haoyu Wang
• Subjects: cs.CL, cs.SE
• Abstract URL: https://arxiv.org/abs/2404.09894
• Pdf URL: https://arxiv.org/pdf/2404.09894
• Copy Paste: [[2404.09894]] Glitch Tokens in Large Language Models: Categorization Taxonomy and Effective Detection(https://arxiv.org/abs/2404.09894)
• Keywords: language model, llm
• Abstract: With the expanding application of Large Language Models (LLMs) in various domains, it becomes imperative to comprehensively investigate their unforeseen behaviors and consequent outcomes. In this study, we introduce and systematically explore the phenomenon of "glitch tokens", which are anomalous tokens produced by established tokenizers and could potentially compromise the models' quality of response. Specifically, we experiment on seven top popular LLMs utilizing three distinct tokenizers and involving a totally of 182,517 tokens. We present categorizations of the identified glitch tokens and symptoms exhibited by LLMs when interacting with glitch tokens. Based on our observation that glitch tokens tend to cluster in the embedding space, we propose GlitchHunter, a novel iterative clustering-based technique, for efficient glitch token detection. The evaluation shows that our approach notably outperforms three baseline methods on eight open-source LLMs. To the best of our knowledge, we present the first comprehensive study on glitch tokens. Our new detection further provides valuable insights into mitigating tokenization-related errors in LLMs.
• 摘要：随着大型语言模型（LLM）在各个领域的广泛应用，全面研究其不可预见的行为和后续结果变得势在必行。在本研究中，我们介绍并系统地探索了“故障令牌”现象，这是由已建立的令牌生成器产生的异常令牌，可能会损害模型的响应质量。具体来说，我们使用三个不同的标记器对七个最流行的法学硕士进行了实验，总共涉及 182,517 个标记。我们对已识别的故障标记进行了分类，以及法学硕士在与故障标记交互时表现出的症状。根据我们对故障标记倾向于在嵌入空间中聚类的观察，我们提出了 GlitchHunter，一种基于迭代聚类的新型技术，用于高效的故障标记检测。评估表明，我们的方法在八个开源法学硕士上明显优于三种基线方法。据我们所知，我们提出了第一个关于故障令牌的全面研究。我们的新检测进一步为减少法学硕士中与标记化相关的错误提供了宝贵的见解。

Title: ChatShop: Interactive Information Seeking with Language Agents

• Authors: Sanxing Chen, Sam Wiseman, Bhuwan Dhingra
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.09911
• Pdf URL: https://arxiv.org/pdf/2404.09911
• Copy Paste: [[2404.09911]] ChatShop: Interactive Information Seeking with Language Agents(https://arxiv.org/abs/2404.09911)
• Keywords: llm, chat, agent
• Abstract: The desire and ability to seek new information strategically are fundamental to human learning but often overlooked in current language agent development. Using a web shopping task as an example, we show that it can be reformulated and solved as a retrieval task without a requirement of interactive information seeking. We then redesign the task to introduce a new role of shopper, serving as a realistically constrained communication channel. The agents in our proposed ChatShop task explore user preferences in open-ended conversation to make informed decisions. Our experiments demonstrate that the proposed task can effectively evaluate the agent's ability to explore and gradually accumulate information through multi-turn interaction. We also show that LLM-simulated shoppers serve as a good proxy to real human shoppers and discover similar error patterns of agents.
• 摘要：战略性地寻求新信息的愿望和能力是人类学习的基础，但在当前的语言代理开发中经常被忽视。以网络购物任务为例，我们表明它可以被重新表述和解决为检索任务，而不需要交互式信息搜索。然后，我们重新设计任务，引入购物者的新角色，作为现实受限的沟通渠道。我们提出的 ChatShop 任务中的代理在开放式对话中探索用户偏好，以做出明智的决策。我们的实验表明，所提出的任务可以有效地评估智能体通过多轮交互探索和逐渐积累信息的能力。我们还表明，LLM 模拟的购物者可以作为真实人类购物者的良好代理，并发现代理的类似错误模式。

Title: Compression Represents Intelligence Linearly

• Authors: Yuzhen Huang, Jinghan Zhang, Zifei Shan, Junxian He
• Subjects: cs.CL, cs.AI, cs.IT, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.09937
• Pdf URL: https://arxiv.org/pdf/2404.09937
• Copy Paste: [[2404.09937]] Compression Represents Intelligence Linearly(https://arxiv.org/abs/2404.09937)
• Keywords: language model, llm
• Abstract: There is a belief that learning to compress well will lead to intelligence. Recently, language modeling has been shown to be equivalent to compression, which offers a compelling rationale for the success of large language models (LLMs): the development of more advanced language models is essentially enhancing compression which facilitates intelligence. Despite such appealing discussions, little empirical evidence is present for the interplay between compression and intelligence. In this work, we examine their relationship in the context of LLMs, treating LLMs as data compressors. Given the abstract concept of "intelligence", we adopt the average downstream benchmark scores as a surrogate, specifically targeting intelligence related to knowledge and commonsense, coding, and mathematical reasoning. Across 12 benchmarks, our study brings together 30 public LLMs that originate from diverse organizations. Remarkably, we find that LLMs' intelligence -- reflected by average benchmark scores -- almost linearly correlates with their ability to compress external text corpora. These results provide concrete evidence supporting the belief that superior compression indicates greater intelligence. Furthermore, our findings suggest that compression efficiency, as an unsupervised metric derived from raw text corpora, serves as a reliable evaluation measure that is linearly associated with the model capabilities. We open-source our compression datasets as well as our data collection pipelines to facilitate future researchers to assess compression properly.
• 摘要：人们相信，学习良好的压缩会带来智慧。最近，语言建模已被证明等同于压缩，这为大型语言模型（LLM）的成功提供了令人信服的理由：更高级语言模型的开发本质上是增强压缩，从而促进智能。尽管讨论如此吸引人，但关于压缩和智能之间相互作用的实证证据却很少。在这项工作中，我们在法学硕士的背景下研究了它们的关系，将法学硕士视为数据压缩器。考虑到“智力”的抽象概念，我们采用平均下游基准分数作为替代，特别针对与知识和常识、编码和数学推理相关的智力。我们的研究涵盖 12 个基准，汇集了来自不同组织的 30 名公共法学硕士。值得注意的是，我们发现法学硕士的智力（通过平均基准分数反映出来）几乎与他们压缩外部文本语料库的能力线性相关。这些结果提供了具体的证据，支持这样的信念：卓越的压缩能力意味着更高的智力。此外，我们的研究结果表明，压缩效率作为源自原始文本语料库的无监督指标，可以作为与模型功能线性相关的可靠评估指标。我们开源我们的压缩数据集以及数据收集管道，以方便未来的研究人员正确评估压缩。

Title: Constructing Benchmarks and Interventions for Combating Hallucinations in LLMs

• Authors: Adi Simhi, Jonathan Herzig, Idan Szpektor, Yonatan Belinkov
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.09971
• Pdf URL: https://arxiv.org/pdf/2404.09971
• Copy Paste: [[2404.09971]] Constructing Benchmarks and Interventions for Combating Hallucinations in LLMs(https://arxiv.org/abs/2404.09971)
• Keywords: language model, llm, hallucination
• Abstract: Large language models (LLMs) are susceptible to hallucination, which sparked a widespread effort to detect and prevent them. Recent work attempts to mitigate hallucinations by intervening in the model's computation during generation, using different setups and heuristics. Those works lack separation between different hallucination causes. In this work, we first introduce an approach for constructing datasets based on the model knowledge for detection and intervention methods in closed-book and open-book question-answering settings. We then characterize the effect of different choices for intervention, such as the intervened components (MLPs, attention block, residual stream, and specific heads), and how often and how strongly to intervene. We find that intervention success varies depending on the component, with some components being detrimental to language modeling capabilities. Finally, we find that interventions can benefit from pre-hallucination steering direction instead of post-hallucination. The code is available at https://github.com/technion-cs-nlp/hallucination-mitigation
• 摘要：大型语言模型（LLM）很容易产生幻觉，这引发了检测和预防幻觉的广泛努力。最近的工作尝试通过使用不同的设置和启发式方法在生成过程中干预模型的计算来减轻幻觉。这些作品缺乏对不同幻觉原因的区分。在这项工作中，我们首先介绍了一种基于模型知识构建数据集的方法，用于闭卷和开卷问答环境中的检测和干预方法。然后，我们描述不同干预选择的效果，例如干预成分（MLP、注意力块、残余流和特定头），以及干预的频率和强度。我们发现干预成功与否取决于组件，有些组件不利于语言建模能力。最后，我们发现干预措施可以受益于幻觉前的指导方向，而不是幻觉后的指导。该代码位于 https://github.com/technion-cs-nlp/hallucination-mitigation

Title: Context Does Matter: Implications for Crowdsourced Evaluation Labels in Task-Oriented Dialogue Systems

• Authors: Clemencia Siro, Mohammad Aliannejadi, Maarten de Rijke
• Subjects: cs.CL, cs.HC, cs.IR
• Abstract URL: https://arxiv.org/abs/2404.09980
• Pdf URL: https://arxiv.org/pdf/2404.09980
• Copy Paste: [[2404.09980]] Context Does Matter: Implications for Crowdsourced Evaluation Labels in Task-Oriented Dialogue Systems(https://arxiv.org/abs/2404.09980)
• Keywords: language model, llm
• Abstract: Crowdsourced labels play a crucial role in evaluating task-oriented dialogue systems (TDSs). Obtaining high-quality and consistent ground-truth labels from annotators presents challenges. When evaluating a TDS, annotators must fully comprehend the dialogue before providing judgments. Previous studies suggest using only a portion of the dialogue context in the annotation process. However, the impact of this limitation on label quality remains unexplored. This study investigates the influence of dialogue context on annotation quality, considering the truncated context for relevance and usefulness labeling. We further propose to use large language models (LLMs) to summarize the dialogue context to provide a rich and short description of the dialogue context and study the impact of doing so on the annotator's performance. Reducing context leads to more positive ratings. Conversely, providing the entire dialogue context yields higher-quality relevance ratings but introduces ambiguity in usefulness ratings. Using the first user utterance as context leads to consistent ratings, akin to those obtained using the entire dialogue, with significantly reduced annotation effort. Our findings show how task design, particularly the availability of dialogue context, affects the quality and consistency of crowdsourced evaluation labels.
• 摘要：众包标签在评估面向任务的对话系统（TDS）中发挥着至关重要的作用。从注释者那里获取高质量且一致的地面实况标签面临着挑战。在评估 TDS 时，注释者必须充分理解对话，然后才能做出判断。先前的研究表明在注释过程中仅使用对话上下文的一部分。然而，这种限制对标签质量的影响仍有待探索。本研究调查了对话上下文对注释质量的影响，考虑了相关性和有用性标签的截断上下文。我们进一步建议使用大型语言模型（LLM）来总结对话上下文，以提供对话上下文的丰富而简短的描述，并研究这样做对注释器性能的影响。减少上下文会带来更积极的评价。相反，提供整个对话上下文会产生更高质量的相关性评级，但会在有用性评级中引入模糊性。使用第一个用户话语作为上下文可以获得一致的评级，类似于使用整个对话获得的评级，同时显着减少注释工作。我们的研究结果表明任务设计，特别是对话上下文的可用性，如何影响众包评估标签的质量和一致性。

Title: Memory Sharing for Large Language Model based Agents

• Authors: Hang Gao, Yongfeng Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.09982
• Pdf URL: https://arxiv.org/pdf/2404.09982
• Copy Paste: [[2404.09982]] Memory Sharing for Large Language Model based Agents(https://arxiv.org/abs/2404.09982)
• Keywords: language model, llm, prompt, agent
• Abstract: In the realm of artificial intelligence, the adaptation of Large Language Model (LLM)-based agents to execute tasks via natural language prompts represents a significant advancement, notably eliminating the need for explicit retraining or fine tuning for fixed-answer tasks such as common sense questions and yes/no queries. However, the application of In-context Learning to open-ended challenges, such as poetry creation, reveals substantial limitations due to the comprehensiveness of the provided examples and agent's ability to understand the content expressed in the problem, leading to outputs that often diverge significantly from expected results. Addressing this gap, our study introduces the Memory-Sharing (MS) framework for LLM multi-agents, which utilizes a real-time memory storage and retrieval system to enhance the In-context Learning process. Each "memory" within this system captures both the posed query and the corresponding real-time response from an LLM-based agent, aggregating these memories from a broad spectrum of similar agents to enrich the memory pool shared by all agents. This framework not only aids agents in identifying the most relevant examples for specific tasks but also evaluates the potential utility of their memories for future applications by other agents. Empirical validation across three distinct domains involving specialized functions of agents demonstrates that the MS framework significantly improve the agent's performance regrading the open-ended questions. Furthermore, we also discuss what type of memory pool and what retrieval strategy in MS can better help agents, offering a future develop direction of MS. The code and data are available at: https://github.com/GHupppp/MemorySharingLLM
• 摘要：在人工智能领域，基于大语言模型（LLM）的代理通过自然语言提示执行任务代表了一项重大进步，特别是消除了对常识等固定答案任务进行显式再训练或微调的需要问题和是/否查询。然而，将情境学习应用于开放式挑战（例如诗歌创作），由于所提供示例的全面性和代理理解问题中表达的内容的能力而暴露出很大的局限性，导致输出经​​常出现显着差异从预期结果来看。为了解决这一差距，我们的研究引入了 LLM 多智能体的内存共享（MS）框架，该框架利用实时内存存储和检索系统来增强上下文学习过程。该系统中的每个“内存”都会捕获来自基于 LLM 的代理的提出的查询和相应的实时响应，并从广泛的类似代理中聚合这些内存，以丰富所有代理共享的内存池。该框架不仅帮助智能体识别与特定任务最相关的示例，而且还评估其记忆对于其他智能体未来应用的潜在效用。涉及代理专门功能的三个不同领域的经验验证表明，MS 框架显着提高了代理对开放式问题重新评分的性能。此外，我们还讨论了MS中什么类型的内存池以及什么样的检索策略可以更好地帮助智能体，为MS未来的发展方向提供了方向。代码和数据可在：https://github.com/GHupppp/MemorySharingLLM