2024-09-16

Title: Real or Robotic? Assessing Whether LLMs Accurately Simulate Qualities of Human Responses in Dialogue

• Authors: Johnathan Ivey, Shivani Kumar, Jiayu Liu, Hua Shen, Sushrita Rakshit, Rohan Raju, Haotian Zhang, Aparna Ananthasubramaniam, Junghwan Kim, Bowen Yi, Dustin Wright, Abraham Israeli, Anders Giovanni Møller, Lechen Zhang, David Jurgens
• Subjects: cs.CL, cs.CY, cs.HC
• Abstract URL: https://arxiv.org/abs/2409.08330
• Pdf URL: https://arxiv.org/pdf/2409.08330
• Copy Paste: [[2409.08330]] Real or Robotic? Assessing Whether LLMs Accurately Simulate Qualities of Human Responses in Dialogue(https://arxiv.org/abs/2409.08330)
• Keywords: language model, llm, chat
• Abstract: Studying and building datasets for dialogue tasks is both expensive and time-consuming due to the need to recruit, train, and collect data from study participants. In response, much recent work has sought to use large language models (LLMs) to simulate both human-human and human-LLM interactions, as they have been shown to generate convincingly human-like text in many settings. However, to what extent do LLM-based simulations \textit{actually} reflect human dialogues? In this work, we answer this question by generating a large-scale dataset of 100,000 paired LLM-LLM and human-LLM dialogues from the WildChat dataset and quantifying how well the LLM simulations align with their human counterparts. Overall, we find relatively low alignment between simulations and human interactions, demonstrating a systematic divergence along the multiple textual properties, including style and content. Further, in comparisons of English, Chinese, and Russian dialogues, we find that models perform similarly. Our results suggest that LLMs generally perform better when the human themself writes in a way that is more similar to the LLM's own style.
• 摘要：由于需要招募、培训和收集研究参与者的数据，研究和构建对话任务的数据集既昂贵又耗时。为此，许多最近的研究都试图使用大型语言模型 (LLM) 来模拟人与人以及人与 LLM 的互动，因为它们已被证明在许多情况下都能生成令人信服的类似人类的文本。然而，基于 LLM 的模拟在多大程度上真正反映了人类的对话？在这项工作中，我们通过从 WildChat 数据集生成 100,000 对 LLM-LLM 和人与 LLM 对话的大规模数据集并量化 LLM 模拟与人类对应者的匹配程度来回答这个问题。总体而言，我们发现模拟和人类互动之间的一致性相对较低，表明在多种文本属性（包括风格和内容）上存在系统性差异。此外，在比较英语、中文和俄语对话时，我们发现模型的表现相似。我们的结果表明，当人类写作方式与 LLM 自己的风格更相似时，LLM 通常表现更好。

Title: Knowledge Tagging with Large Language Model based Multi-Agent System

• Authors: Hang Li, Tianlong Xu, Ethan Chang, Qingsong Wen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.08406
• Pdf URL: https://arxiv.org/pdf/2409.08406
• Copy Paste: [[2409.08406]] Knowledge Tagging with Large Language Model based Multi-Agent System(https://arxiv.org/abs/2409.08406)
• Keywords: language model, llm, agent
• Abstract: Knowledge tagging for questions is vital in modern intelligent educational applications, including learning progress diagnosis, practice question recommendations, and course content organization. Traditionally, these annotations have been performed by pedagogical experts, as the task demands not only a deep semantic understanding of question stems and knowledge definitions but also a strong ability to link problem-solving logic with relevant knowledge concepts. With the advent of advanced natural language processing (NLP) algorithms, such as pre-trained language models and large language models (LLMs), pioneering studies have explored automating the knowledge tagging process using various machine learning models. In this paper, we investigate the use of a multi-agent system to address the limitations of previous algorithms, particularly in handling complex cases involving intricate knowledge definitions and strict numerical constraints. By demonstrating its superior performance on the publicly available math question knowledge tagging dataset, MathKnowCT, we highlight the significant potential of an LLM-based multi-agent system in overcoming the challenges that previous methods have encountered. Finally, through an in-depth discussion of the implications of automating knowledge tagging, we underscore the promising results of deploying LLM-based algorithms in educational contexts.
• 摘要：问题的知识标记在现代智能教育应用中至关重要，包括学习进度诊断、练习问题推荐和课程内容组织。传统上，这些注释是由教育专家完成的，因为这项任务不仅需要对问题词干和知识定义的深度语义理解，还需要将问题解决逻辑与相关知识概念联系起来的强大能力。随着高级自然语言处理 (NLP) 算法（例如预训练语言模型和大型语言模型 (LLM)）的出现，先驱研究已经探索使用各种机器学习模型来自动化知识标记过程。在本文中，我们研究了使用多智能体系统来解决以前算法的局限性，特别是在处理涉及复杂知识定义和严格数值约束的复杂情况时。通过在公开可用的数学问题知识标记数据集 MathKnowCT 上展示其卓越性能，我们强调了基于 LLM 的多智能体系统在克服以前方法遇到的挑战方面的巨大潜力。最后，通过深入讨论自动化知识标记的含义，我们强调了在教育环境中部署基于 LLM 的算法所带来的良好结果。

Title: When Context Leads but Parametric Memory Follows in Large Language Models

• Authors: Yufei Tao, Adam Hiatt, Erik Haake, Antonie J. Jetter, Ameeta Agrawal
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.08435
• Pdf URL: https://arxiv.org/pdf/2409.08435
• Copy Paste: [[2409.08435]] When Context Leads but Parametric Memory Follows in Large Language Models(https://arxiv.org/abs/2409.08435)
• Keywords: language model, llm, hallucination
• Abstract: Large language models (LLMs) have demonstrated remarkable progress in leveraging diverse knowledge sources. This study investigates how nine widely used LLMs allocate knowledge between local context and global parameters when answering open-ended questions in knowledge-consistent scenarios. We introduce a novel dataset, WikiAtomic, and systematically vary context sizes to analyze how LLMs prioritize and utilize the provided information and their parametric knowledge in knowledge-consistent scenarios. Additionally, we also study their tendency to hallucinate under varying context sizes. Our findings reveal consistent patterns across models, including a consistent reliance on both contextual (around 70%) and parametric (around 30%) knowledge, and a decrease in hallucinations with increasing context. These insights highlight the importance of more effective context organization and developing models that use input more deterministically for robust performance.
• 摘要：大型语言模型 (LLM) 在利用各种知识来源方面取得了显著进展。本研究调查了九种广泛使用的 LLM 在知识一致场景中回答开放式问题时如何在局部上下文和全局参数之间分配知识。我们引入了一个新数据集 WikiAtomic，并系统地改变上下文大小，以分析 LLM 如何在知识一致场景中优先考虑和利用所提供的信息及其参数知识。此外，我们还研究了它们在不同上下文大小下产生幻觉的倾向。我们的研究结果揭示了跨模型的一致模式，包括对上下文（约 70%）和参数（约 30%）知识的一致依赖，以及随着上下文的增加幻觉减少。这些见解强调了更有效的上下文组织和开发更确定地使用输入的模型以实现稳健性能的重要性。

Title: A BERT-Based Summarization approach for depression detection

• Authors: Hossein Salahshoor Gavalan, Mohmmad Naim Rastgoo, Bahareh Nakisa
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.08483
• Pdf URL: https://arxiv.org/pdf/2409.08483
• Copy Paste: [[2409.08483]] A BERT-Based Summarization approach for depression detection(https://arxiv.org/abs/2409.08483)
• Keywords: language model, agent
• Abstract: Depression is a globally prevalent mental disorder with potentially severe repercussions if not addressed, especially in individuals with recurrent episodes. Prior research has shown that early intervention has the potential to mitigate or alleviate symptoms of depression. However, implementing such interventions in a real-world setting may pose considerable challenges. A promising strategy involves leveraging machine learning and artificial intelligence to autonomously detect depression indicators from diverse data sources. One of the most widely available and informative data sources is text, which can reveal a person's mood, thoughts, and feelings. In this context, virtual agents programmed to conduct interviews using clinically validated questionnaires, such as those found in the DAIC-WOZ dataset, offer a robust means for depression detection through linguistic analysis. Utilizing BERT-based models, which are powerful and versatile yet use fewer resources than contemporary large language models, to convert text into numerical representations significantly enhances the precision of depression diagnosis. These models adeptly capture complex semantic and syntactic nuances, improving the detection accuracy of depressive symptoms. Given the inherent limitations of these models concerning text length, our study proposes text summarization as a preprocessing technique to diminish the length and intricacies of input texts. Implementing this method within our uniquely developed framework for feature extraction and classification yielded an F1-score of 0.67 on the test set surpassing all prior benchmarks and 0.81 on the validation set exceeding most previous results on the DAIC-WOZ dataset. Furthermore, we have devised a depression lexicon to assess summary quality and relevance. This lexicon constitutes a valuable asset for ongoing research in depression detection.
• 摘要：抑郁症是一种全球普遍的精神障碍，如果不加以解决，可能会产生严重后果，尤其是对于反复发作的个体。先前的研究表明，早期干预有可能减轻或缓解抑郁症状。然而，在现实环境中实施此类干预可能会带来相当大的挑战。一种有前途的策略是利用机器学习和人工智能从各种数据源中自主检测抑郁症指标。最广泛可用和信息量最大的数据源之一是文本，它可以揭示一个人的情绪、想法和感受。在这种情况下，虚拟代理被编程为使用临床验证的问卷（例如 DAIC-WOZ 数据集中的问卷）进行访谈，通过语言分析提供了一种强大的抑郁症检测方法。利用基于 BERT 的模型（功能强大且用途广泛，但使用的资源比当代大型语言模型更少）将文本转换为数字表示，可以显著提高抑郁症诊断的准确性。这些模型能够熟练地捕捉复杂的语义和句法细微差别，从而提高抑郁症状的检测准确性。鉴于这些模型在文本长度方面的固有局限性，我们的研究提出了文本摘要作为一种预处理技术，以减少输入文本的长度和复杂性。在我们独特开发的特征提取和分类框架内实施此方法，在测试集上获得了 0.67 的 F1 分数，超过了所有先前的基准，在验证集上获得了 0.81 的 F1 分数，超过了 DAIC-WOZ 数据集上大多数先前的结果。此外，我们还设计了一个抑郁症词典来评估摘要质量和相关性。这个词典是正在进行的抑郁症检测研究的宝贵资产。

Title: Eir: Thai Medical Large Language Models

• Authors: Yutthakorn Thiprak, Rungtam Ngodngamthaweesuk, Songtam Ngodngamtaweesuk
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.08523
• Pdf URL: https://arxiv.org/pdf/2409.08523
• Copy Paste: [[2409.08523]] Eir: Thai Medical Large Language Models(https://arxiv.org/abs/2409.08523)
• Keywords: language model, gpt, llm, chain-of-thought
• Abstract: We present Eir Thai Medical LLM, a large language model with 8 billion parameters, specifically designed to enhance the accuracy of handling medical tasks in the Thai language. This model focuses on providing clear and easy-to-understand answers for both healthcare professionals and patients, thereby improving the efficiency of diagnosis and treatment processes. Human evaluation was conducted to ensure that the model adheres to care standards and provides unbiased answers. To prioritize data security, the model is deployed within the hospital's internal network, ensuring both high security and faster processing speeds. The internal API connection is secured with encryption and strict authentication measures to prevent data leaks and unauthorized access. We evaluated several open-source large language models with 8 billion parameters on four medical benchmarks: MedQA, MedMCQA, PubMedQA, and the medical subset of MMLU. The best-performing baselines were used to develop Eir Thai Medical LLM. Our evaluation employed multiple questioning strategies, including zero-shot, few-shot, chain-of-thought reasoning, and ensemble/self-consistency voting methods. Our model outperformed commercially available Thai-language large language models by more than 10%. In addition, we developed enhanced model testing tailored for clinical use in Thai across 18 clinical tasks, where our model exceeded GPT-4o performance by more than 11%
• 摘要：我们推出了 Eir Thai Medical LLM，这是一个拥有 80 亿个参数的大型语言模型，专门用于提高处理泰语医疗任务的准确性。该模型专注于为医疗专业人员和患者提供清晰易懂的答案，从而提高诊断和治疗过程的效率。进行了人工评估，以确保模型符合护理标准并提供公正的答案。为了优先考虑数据安全，该模型部署在医院的内部网络中，确保高安全性和更快的处理速度。内部 API 连接采用加密和严格的身份验证措施保护，以防止数据泄露和未经授权的访问。我们在四个医学基准上评估了几个具有 80 亿个参数的开源大型语言模型：MedQA、MedMCQA、PubMedQA 和 MMLU 的医学子集。表现最佳的基线用于开发 Eir Thai Medical LLM。我们的评估采用了多种提问策略，包括零样本、少量样本、思路链推理和集成/自洽投票方法。我们的模型比市面上可用的泰语大型语言模型高出 10% 以上。此外，我们还针对泰语临床应用开发了增强模型测试，测试了 18 项临床任务，我们的模型比 GPT-4o 的性能高出 11% 以上

Title: LLM-Powered Grapheme-to-Phoneme Conversion: Benchmark and Case Study

• Authors: Mahta Fetrat Qharabagh, Zahra Dehghanian, Hamid R. Rabiee
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.08554
• Pdf URL: https://arxiv.org/pdf/2409.08554
• Copy Paste: [[2409.08554]] LLM-Powered Grapheme-to-Phoneme Conversion: Benchmark and Case Study(https://arxiv.org/abs/2409.08554)
• Keywords: language model, llm, prompt
• Abstract: Grapheme-to-phoneme (G2P) conversion is critical in speech processing, particularly for applications like speech synthesis. G2P systems must possess linguistic understanding and contextual awareness of languages with polyphone words and context-dependent phonemes. Large language models (LLMs) have recently demonstrated significant potential in various language tasks, suggesting that their phonetic knowledge could be leveraged for G2P. In this paper, we evaluate the performance of LLMs in G2P conversion and introduce prompting and post-processing methods that enhance LLM outputs without additional training or labeled data. We also present a benchmarking dataset designed to assess G2P performance on sentence-level phonetic challenges of the Persian language. Our results show that by applying the proposed methods, LLMs can outperform traditional G2P tools, even in an underrepresented language like Persian, highlighting the potential of developing LLM-aided G2P systems.
• 摘要：字素到音素 (G2P) 转换在语音处理中至关重要，特别是对于语音合成等应用。G2P 系统必须具备对具有多音词和上下文相关音素的语言的语言理解和上下文感知能力。大型语言模型 (LLM) 最近在各种语言任务中表现出巨大潜力，表明它们的语音知识可用于 G2P。在本文中，我们评估了 LLM 在 G2P 转换中的表现，并引入了提示和后处理方法，这些方法无需额外的训练或标记数据即可增强 LLM 输出。我们还提供了一个基准数据集，旨在评估 G2P 在波斯语句子级语音挑战中的表现。我们的结果表明，通过应用所提出的方法，LLM 可以胜过传统的 G2P 工具，即使在波斯语等代表性不足的语言中也是如此，这凸显了开发 LLM 辅助 G2P 系统的潜力。

Title: Expediting and Elevating Large Language Model Reasoning via Hidden Chain-of-Thought Decoding

• Authors: Tianqiao Liu, Zui Chen, Zitao Liu, Mi Tian, Weiqi Luo
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.08561
• Pdf URL: https://arxiv.org/pdf/2409.08561
• Copy Paste: [[2409.08561]] Expediting and Elevating Large Language Model Reasoning via Hidden Chain-of-Thought Decoding(https://arxiv.org/abs/2409.08561)
• Keywords: language model, llm, prompt, chain-of-thought, agent
• Abstract: Large language models (LLMs) have demonstrated remarkable capabilities in tasks requiring reasoning and multi-step problem-solving through the use of chain-of-thought (CoT) prompting. However, generating the full CoT process results in significantly longer output sequences, leading to increased computational costs and latency during inference. To address this challenge, we propose a novel approach to compress the CoT process through semantic alignment, enabling more efficient decoding while preserving the benefits of CoT reasoning. Our method introduces an auxiliary CoT model that learns to generate and compress the full thought process into a compact special token representation semantically aligned with the original CoT output. This compressed representation is then integrated into the input of the Hidden Chain-of-Thought (HCoT) model. The training process follows a two-stage procedure: First, the CoT model is optimized to generate the compressed token representations aligned with the ground-truth CoT outputs using a contrastive loss. Subsequently, with the CoT model parameters frozen, the HCoT model is fine-tuned to generate accurate subsequent predictions conditioned on the prefix instruction and the compressed CoT representations from the CoT model. Extensive experiments across three challenging domains - mathematical reasoning, agent invocation, and question answering - demonstrate that our semantic compression approach achieves competitive or improved performance compared to the full CoT baseline, while providing significant speedups of at least 1.5x in decoding time. Moreover, incorporating contrastive learning objectives further enhances the quality of the compressed representations, leading to better CoT prompting and improved task accuracy. Our work paves the way for more efficient exploitation of multi-step reasoning capabilities in LLMs across a wide range of applications.
• 摘要：大型语言模型 (LLM) 通过使用思路链 (CoT) 提示，在需要推理和多步骤问题解决的任务中表现出了卓越的能力。然而，生成完整的 CoT 过程会导致输出序列明显变长，从而增加计算成本和推理过程中的延迟。为了应对这一挑战，我们提出了一种通过语义对齐来压缩 CoT 过程的新方法，从而实现更高效的解码，同时保留 CoT 推理的优势。我们的方法引入了一个辅助 CoT 模型，该模型学习生成完整的思维过程并将其压缩为与原始 CoT 输出在语义上对齐的紧凑特殊标记表示。然后，将此压缩表示集成到隐藏思路链 (HCoT) 模型的输入中。训练过程遵循两阶段程序：首先，使用对比损失优化 CoT 模型，以生成与真实 CoT 输出对齐的压缩标记表示。随后，在 CoT 模型参数冻结的情况下，对 HCoT 模型进行微调，以根据前缀指令和 CoT 模型中的压缩 CoT 表示生成准确的后续预测。在三个具有挑战性的领域（数学推理、代理调用和问答）进行的大量实验表明，与完整的 CoT 基线相比，我们的语义压缩方法实现了具有竞争力或改进的性能，同时将解码时间显著加快了至少 1.5 倍。此外，结合对比学习目标进一步提高了压缩表示的质量，从而实现了更好的 CoT 提示和更高的任务准确性。我们的工作为在广泛的应用中更有效地利用 LLM 中的多步推理能力铺平了道路。

Title: Cracking the Code: Multi-domain LLM Evaluation on Real-World Professional Exams in Indonesia

• Authors: Fajri Koto
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.08564
• Pdf URL: https://arxiv.org/pdf/2409.08564
• Copy Paste: [[2409.08564]] Cracking the Code: Multi-domain LLM Evaluation on Real-World Professional Exams in Indonesia(https://arxiv.org/abs/2409.08564)
• Keywords: language model, llm
• Abstract: While knowledge evaluation in large language models has predominantly focused on academic subjects like math and physics, these assessments often fail to capture the practical demands of real-world professions. In this paper, we introduce IndoCareer, a dataset comprising 8,834 multiple-choice questions designed to evaluate performance in vocational and professional certification exams across various fields. With a focus on Indonesia, IndoCareer provides rich local contexts, spanning six key sectors: (1) healthcare, (2) insurance and finance, (3) creative and design, (4) tourism and hospitality, (5) education and training, and (6) law. Our comprehensive evaluation of 27 large language models shows that these models struggle particularly in fields with strong local contexts, such as insurance and finance. Additionally, while using the entire dataset, shuffling answer options generally maintains consistent evaluation results across models, but it introduces instability specifically in the insurance and finance sectors.
• 摘要：虽然大型语言模型中的知识评估主要集中在数学和物理等学术科目上，但这些评估往往无法反映现实世界职业的实际需求。在本文中，我们介绍了 IndoCareer，这是一个包含 8,834 个多项选择题的数据集，旨在评估各个领域职业和专业认证考试的表现。IndoCareer 专注于印度尼西亚，提供丰富的本地背景，涵盖六个关键领域：（1）医疗保健、（2）保险和金融、（3）创意和设计、（4）旅游和酒店业、（5）教育和培训以及（6）法律。我们对 27 个大型语言模型的全面评估表明，这些模型在具有强大本地背景的领域（例如保险和金融）尤其困难。此外，在使用整个数据集时，打乱答案选项通常可以在模型之间保持一致的评估结果，但它会在保险和金融领域引入不稳定性。

Title: Large Language Model Can Transcribe Speech in Multi-Talker Scenarios with Versatile Instructions

• Authors: Lingwei Meng, Shujie Hu, Jiawen Kang, Zhaoqing Li, Yuejiao Wang, Wenxuan Wu, Xixin Wu, Xunying Liu, Helen Meng
• Subjects: cs.CL, cs.AI, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2409.08596
• Pdf URL: https://arxiv.org/pdf/2409.08596
• Copy Paste: [[2409.08596]] Large Language Model Can Transcribe Speech in Multi-Talker Scenarios with Versatile Instructions(https://arxiv.org/abs/2409.08596)
• Keywords: language model, llm
• Abstract: Recent advancements in large language models (LLMs) have revolutionized various domains, bringing significant progress and new opportunities. Despite progress in speech-related tasks, LLMs have not been sufficiently explored in multi-talker scenarios. In this work, we present a pioneering effort to investigate the capability of LLMs in transcribing speech in multi-talker environments, following versatile instructions related to multi-talker automatic speech recognition (ASR), target talker ASR, and ASR based on specific talker attributes such as sex, occurrence order, language, and keyword spoken. Our approach utilizes WavLM and Whisper encoder to extract multi-faceted speech representations that are sensitive to speaker characteristics and semantic context. These representations are then fed into an LLM fine-tuned using LoRA, enabling the capabilities for speech comprehension and transcription. Comprehensive experiments reveal the promising performance of our proposed system, MT-LLM, in cocktail party scenarios, highlighting the potential of LLM to handle speech-related tasks based on user instructions in such complex settings.
• 摘要：大型语言模型 (LLM) 的最新进展彻底改变了各个领域，带来了重大进展和新机遇。尽管在语音相关任务方面取得了进展，但 LLM 在多说话者场景中的探索还不够。在这项工作中，我们提出了一项开创性的努力，以研究 LLM 在多说话者环境中转录语音的能力，遵循与多说话者自动语音识别 (ASR)、目标说话者 ASR 和基于特定说话者属性（例如性别、发生顺序、语言和说出的关键字）的 ASR 相关的多功能指令。我们的方法利用 WavLM 和 Whisper 编码器提取对说话者特征和语义上下文敏感的多方面语音表示。然后将这些表示输入到使用 LoRA 微调的 LLM 中，从而实现语音理解和转录功能。全面的实验表明，我们提出的系统 MT-LLM 在鸡尾酒会场景中表现出色，凸显了 LLM 在如此复杂的环境中根据用户指令处理语音相关任务的潜力。

Title: L3Cube-IndicQuest: A Benchmark Questing Answering Dataset for Evaluating Knowledge of LLMs in Indic Context

• Authors: Pritika Rohera, Chaitrali Ginimav, Akanksha Salunke, Gayatri Sawant, Raviraj Joshi
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.08706
• Pdf URL: https://arxiv.org/pdf/2409.08706
• Copy Paste: [[2409.08706]] L3Cube-IndicQuest: A Benchmark Questing Answering Dataset for Evaluating Knowledge of LLMs in Indic Context(https://arxiv.org/abs/2409.08706)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have made significant progress in incorporating Indic languages within multilingual models. However, it is crucial to quantitatively assess whether these languages perform comparably to globally dominant ones, such as English. Currently, there is a lack of benchmark datasets specifically designed to evaluate the regional knowledge of LLMs in various Indic languages. In this paper, we present the L3Cube-IndicQuest, a gold-standard question-answering benchmark dataset designed to evaluate how well multilingual LLMs capture regional knowledge across various Indic languages. The dataset contains 200 question-answer pairs, each for English and 19 Indic languages, covering five domains specific to the Indic region. We aim for this dataset to serve as a benchmark, providing ground truth for evaluating the performance of LLMs in understanding and representing knowledge relevant to the Indian context. The IndicQuest can be used for both reference-based evaluation and LLM-as-a-judge evaluation. The dataset is shared publicly at this https URL .
• 摘要：大型语言模型 (LLM) 在将印度语纳入多语言模型方面取得了重大进展。然而，至关重要的是要定量评估这些语言的表现是否与英语等全球主导语言相当。目前，缺乏专门用于评估各种印度语 LLM 的区域知识的基准数据集。在本文中，我们提出了 L3Cube-IndicQuest，这是一个黄金标准的问答基准数据集，旨在评估多语言 LLM 在捕捉各种印度语区域知识方面的表现。该数据集包含 200 个问答对，每个问答对针对英语和 19 种印度语，涵盖了印度地区的五个特定领域。我们的目标是让这个数据集成为一个基准，为评估 LLM 在理解和表示与印度背景相关的知识方面的表现提供基本事实。IndicQuest 既可用于基于参考的评估，也可用于 LLM 作为评判者的评估。该数据集在此 https URL 上公开共享。

Title: Distilling Monolingual and Crosslingual Word-in-Context Representations

• Authors: Yuki Arase, Tomoyuki Kajiwara
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.08719
• Pdf URL: https://arxiv.org/pdf/2409.08719
• Copy Paste: [[2409.08719]] Distilling Monolingual and Crosslingual Word-in-Context Representations(https://arxiv.org/abs/2409.08719)
• Keywords: language model
• Abstract: In this study, we propose a method that distils representations of word meaning in context from a pre-trained masked language model in both monolingual and crosslingual settings. Word representations are the basis for context-aware lexical semantics and unsupervised semantic textual similarity (STS) estimation. Different from existing approaches, our method does not require human-annotated corpora nor updates of the parameters of the pre-trained model. The latter feature is appealing for practical scenarios where the off-the-shelf pre-trained model is a common asset among different applications. Specifically, our method learns to combine the outputs of different hidden layers of the pre-trained model using self-attention. Our auto-encoder based training only requires an automatically generated corpus. To evaluate the performance of the proposed approach, we performed extensive experiments using various benchmark tasks. The results on the monolingual tasks confirmed that our representations exhibited a competitive performance compared to that of the previous study for the context-aware lexical semantic tasks and outperformed it for STS estimation. The results of the crosslingual tasks revealed that the proposed method largely improved crosslingual word representations of multilingual pre-trained models.
• 摘要：在本研究中，我们提出了一种方法，该方法可在单语和跨语言环境中从预先训练的掩码语言模型中提取上下文中的词义表示。词语表示是上下文感知词汇语义和无监督语义文本相似度 (STS) 估计的基础。与现有方法不同，我们的方法不需要人工注释的语料库，也不需要更新预训练模型的参数。后一个特性对于实际场景很有吸引力，因为现成的预训练模型是不同应用程序之间的共同资产。具体而言，我们的方法学习使用自注意力来组合预训练模型的不同隐藏层的输出。我们基于自动编码器的训练只需要自动生成的语料库。为了评估所提出方法的性能，我们使用各种基准任务进行了大量实验。单语任务的结果证实，与之前研究的上下文感知词汇语义任务相比，我们的表示表现出了具有竞争力的性能，并且在 STS 估计方面表现优于它。跨语言任务的结果表明，所提出的方法极大地改善了多语言预训练模型的跨语言词汇表示。

Title: Optimizing Ingredient Substitution Using Large Language Models to Enhance Phytochemical Content in Recipes

• Authors: Luis Rita, Josh Southern, Ivan Laponogov, Kyle Higgins, Kirill Veselkov
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.08792
• Pdf URL: https://arxiv.org/pdf/2409.08792
• Copy Paste: [[2409.08792]] Optimizing Ingredient Substitution Using Large Language Models to Enhance Phytochemical Content in Recipes(https://arxiv.org/abs/2409.08792)
• Keywords: language model, gpt, llm
• Abstract: In the emerging field of computational gastronomy, aligning culinary practices with scientifically supported nutritional goals is increasingly important. This study explores how large language models (LLMs) can be applied to optimize ingredient substitutions in recipes, specifically to enhance the phytochemical content of meals. Phytochemicals are bioactive compounds found in plants, which, based on preclinical studies, may offer potential health benefits. We fine-tuned models, including OpenAI's GPT-3.5, DaVinci, and Meta's TinyLlama, using an ingredient substitution dataset. These models were used to predict substitutions that enhance phytochemical content and create a corresponding enriched recipe dataset. Our approach improved Hit@1 accuracy on ingredient substitution tasks, from the baseline 34.53 plus-minus 0.10% to 38.03 plus-minus 0.28% on the original GISMo dataset, and from 40.24 plus-minus 0.36% to 54.46 plus-minus 0.29% on a refined version of the same dataset. These substitutions led to the creation of 1,951 phytochemically enriched ingredient pairings and 1,639 unique recipes. While this approach demonstrates potential in optimizing ingredient substitutions, caution must be taken when drawing conclusions about health benefits, as the claims are based on preclinical evidence. Future work should include clinical validation and broader datasets to further evaluate the nutritional impact of these substitutions. This research represents a step forward in using AI to promote healthier eating practices, providing potential pathways for integrating computational methods with nutritional science.
• 摘要：在新兴的计算美食学领域，将烹饪实践与科学支持的营养目标相结合变得越来越重要。本研究探讨了如何应用大型语言模型 (LLM) 来优化食谱中的成分替代，特别是提高膳食中的植物化学成分。植物化学成分是植物中发现的生物活性化合物，根据临床前研究，它可能具有潜在的健康益处。我们使用成分替代数据集对模型进行了微调，包括 OpenAI 的 GPT-3.5、DaVinci 和 Meta 的 TinyLlama。这些模型用于预测可提高植物化学成分含量的替代品，并创建相应的丰富食谱数据集。我们的方法提高了成分替代任务中的 Hit@1 准确率，从基线的 34.53 正负 0.10% 提高到原始 GISMo 数据集上的 38.03 正负 0.28%，从 40.24 正负 0.36% 提高到同一数据集的改进版本上的 54.46 正负 0.29%。这些替代导致了 1,951 种富含植物化学成分的成分搭配和 1,639 种独特的食谱的产生。虽然这种方法展示了优化成分替代的潜力，但在得出关于健康益处的结论时必须谨慎，因为这些说法是基于临床前证据的。未来的工作应该包括临床验证和更广泛的数据集，以进一步评估这些替代品的营养影响。这项研究代表了利用人工智能促进更健康饮食习惯的一步，为将计算方法与营养科学相结合提供了潜在途径。

Title: Your Weak LLM is Secretly a Strong Teacher for Alignment

• Authors: Leitian Tao, Yixuan Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.08813
• Pdf URL: https://arxiv.org/pdf/2409.08813
• Copy Paste: [[2409.08813]] Your Weak LLM is Secretly a Strong Teacher for Alignment(https://arxiv.org/abs/2409.08813)
• Keywords: language model, llm
• Abstract: The burgeoning capabilities of large language models (LLMs) have underscored the need for alignment to ensure these models act in accordance with human values and intentions. Existing alignment frameworks present constraints either in the form of expensive human effort or high computational costs. This paper explores a promising middle ground, where we employ a weak LLM that is significantly less resource-intensive than top-tier models, yet offers more automation than purely human feedback. We present a systematic study to evaluate and understand weak LLM's ability to generate feedback for alignment. Our empirical findings demonstrate that weak LLMs can provide feedback that rivals or even exceeds that of fully human-annotated data. Our study indicates a minimized impact of model size on feedback efficacy, shedding light on a scalable and sustainable alignment strategy. To deepen our understanding of alignment under weak LLM feedback, we conduct a series of qualitative and quantitative analyses, offering novel insights into the quality discrepancies between human feedback vs. weak LLM feedback.
• 摘要：大型语言模型 (LLM) 的蓬勃发展凸显了对齐的必要性，以确保这些模型按照人类的价值观和意图行事。现有的对齐框架要么以昂贵的人力投入，要么以高计算成本的形式存在限制。本文探讨了一个有希望的中间立场，即我们使用一个弱 LLM，它比顶级模型的资源密集程度要低得多，但比纯人工反馈提供更多的自动化。我们提出了一项系统研究来评估和理解弱 LLM 生成对齐反馈的能力。我们的实证结果表明，弱 LLM 可以提供与完全人工注释的数据相媲美甚至超过的反馈。我们的研究表明模型大小对反馈效果的影响最小，这为可扩展和可持续的对齐策略提供了启示。为了加深对弱 LLM 反馈下对齐的理解，我们进行了一系列定性和定量分析，为人工反馈与弱 LLM 反馈之间的质量差异提供了新的见解。

Title: AIPO: Improving Training Objective for Iterative Preference Optimization

• Authors: Yaojie Shen, Xinyao Wang, Yulei Niu, Ying Zhou, Lexin Tang, Libo Zhang, Fan Chen, Longyin Wen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.08845
• Pdf URL: https://arxiv.org/pdf/2409.08845
• Copy Paste: [[2409.08845]] AIPO: Improving Training Objective for Iterative Preference Optimization(https://arxiv.org/abs/2409.08845)
• Keywords: language model, llm
• Abstract: Preference Optimization (PO), is gaining popularity as an alternative choice of Proximal Policy Optimization (PPO) for aligning Large Language Models (LLMs). Recent research on aligning LLMs iteratively with synthetic or partially synthetic data shows promising results in scaling up PO training for both academic settings and proprietary trained models such as Llama3. Despite its success, our study shows that the length exploitation issue present in PO is even more severe in Iterative Preference Optimization (IPO) due to the iterative nature of the process. In this work, we study iterative preference optimization with synthetic data. We share the findings and analysis along the way of building the iterative preference optimization pipeline. More specifically, we discuss the length exploitation issue during iterative preference optimization and propose our training objective for iterative preference optimization, namely Agreement-aware Iterative Preference Optimization (AIPO). To demonstrate the effectiveness of our method, we conduct comprehensive experiments and achieve state-of-the-art performance on MT-Bench, AlpacaEval 2.0, and Arena-Hard. Our implementation and model checkpoints will be made available at this https URL.
• 摘要：偏好优化 (PO) 作为近端策略优化 (PPO) 的替代选择，用于对齐大型语言模型 (LLM)，正变得越来越流行。最近关于使用合成或部分合成数据迭代对齐 LLM 的研究显示，在扩大 PO 训练方面取得了令人鼓舞的结果，无论是在学术环境还是专有训练模型（如 Llama3）中。尽管取得了成功，但我们的研究表明，由于该过程的迭代性质，PO 中存在的长度利用问题在迭代偏好优化 (IPO) 中更加严重。在这项工作中，我们研究了使用合成数据的迭代偏好优化。我们在构建迭代偏好优化流程的过程中分享了发现和分析。更具体地说，我们讨论了迭代偏好优化过程中的长度利用问题，并提出了迭代偏好优化的训练目标，即协议感知迭代偏好优化 (AIPO)。为了证明我们方法的有效性，我们进行了全面的实验，并在 MT-Bench、AlpacaEval 2.0 和 Arena-Hard 上取得了最先进的性能。我们的实施和模型检查点将在此 https URL 上提供。