2024-07-04

Title: RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMs

• Authors: John Dang, Arash Ahmadian, Kelly Marchisio, Julia Kreutzer, Ahmet Üstün, Sara Hooker
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.02552
• Pdf URL: https://arxiv.org/pdf/2407.02552
• Copy Paste: [[2407.02552]] RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMs(https://arxiv.org/abs/2407.02552)
• Keywords: language model, llm
• Abstract: Preference optimization techniques have become a standard final stage for training state-of-art large language models (LLMs). However, despite widespread adoption, the vast majority of work to-date has focused on first-class citizen languages like English and Chinese. This captures a small fraction of the languages in the world, but also makes it unclear which aspects of current state-of-the-art research transfer to a multilingual setting. In this work, we perform an exhaustive study to achieve a new state-of-the-art in aligning multilingual LLMs. We introduce a novel, scalable method for generating high-quality multilingual feedback data to balance data coverage. We establish the benefits of cross-lingual transfer and increased dataset size in preference training. Our preference-trained model achieves a 54.4% win-rate against Aya 23 8B, the current state-of-the-art multilingual LLM in its parameter class, and a 69.5% win-rate or higher against widely used models like Gemma-1.1-7B-it, Llama-3-8B-Instruct, Mistral-7B-Instruct-v0.3. As a result of our study, we expand the frontier of alignment techniques to 23 languages covering half of the world's population.
• 摘要：偏好优化技术已成为训练最先进的大型语言模型 (LLM) 的标准最终阶段。然而，尽管被广泛采用，但迄今为止的绝大多数工作都集中在英语和中文等一流公民语言上。这只涵盖了世界上一小部分语言，但也不清楚当前最先进的研究的哪些方面可以转移到多语言环境中。在这项工作中，我们进行了一项详尽的研究，以在对齐多语言 LLM 方面取得新的领先水平。我们引入了一种新颖的可扩展方法来生成高质量的多语言反馈数据，以平衡数据覆盖范围。我们确定了跨语言迁移和增加数据集大小在偏好训练中的好处。我们的偏好训练模型在与 Aya 23 8B（其参数类别中目前最先进的多语言 LLM）的对战中取得了 54.4% 的胜率，在与 Gemma-1.1-7B-it、Llama-3-8B-Instruct、Mistral-7B-Instruct-v0.3 等广泛使用的模型的对战中取得了 69.5% 或更高的胜率。通过我们的研究，我们将对齐技术的前沿扩展到 23 种语言，覆盖了世界一半的人口。

Title: Change My Frame: Reframing in the Wild in r/ChangeMyView

• Authors: Arturo Martínez Peguero, Taro Watanabe
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2407.02637
• Pdf URL: https://arxiv.org/pdf/2407.02637
• Copy Paste: [[2407.02637]] Change My Frame: Reframing in the Wild in r/ChangeMyView(https://arxiv.org/abs/2407.02637)
• Keywords: llm, prompt
• Abstract: Recent work in reframing, within the scope of text style transfer, has so far made use of out-of-context, task-prompted utterances in order to produce neutralizing or optimistic reframes. Our work aims to generalize reframing based on the subreddit r/ChangeMyView (CMV). We build a dataset that leverages CMV's community's interactions and conventions to identify high-value, community-recognized utterances that produce changes of perspective. With this data, we widen the scope of the direction of reframing since the changes in perspective do not only occur in neutral or positive directions. We fine tune transformer-based models, make use of a modern LLM to refine our dataset, and explore challenges in the dataset creation and evaluation around this type of reframing.
• 摘要：在文本风格转换的范围内，重构的最新研究迄今为止已经利用了脱离上下文、任务提示的话语来产生中性或乐观的重构。我们的工作旨在基于子版块 r/ChangeMyView (CMV) 推广重构。我们构建了一个数据集，利用 CMV 社区的互动和惯例来识别产生观点变化的高价值、社区认可的话语。有了这些数据，我们扩大了重构方向的范围，因为观点的变化不仅发生在中性或积极的方向上。我们对基于变换器的模型进行了微调，利用现代 LLM 来完善我们的数据集，并探索围绕这种重构类型的数据集创建和评估中的挑战。

Title: Ensuring Responsible Sourcing of Large Language Model Training Data Through Knowledge Graph Comparison

• Authors: Devam Mondal, Carlo Lipizzi
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.02659
• Pdf URL: https://arxiv.org/pdf/2407.02659
• Copy Paste: [[2407.02659]] Ensuring Responsible Sourcing of Large Language Model Training Data Through Knowledge Graph Comparison(https://arxiv.org/abs/2407.02659)
• Keywords: language model, llm
• Abstract: In light of recent plagiarism allegations Brough by publishers, newspapers, and other creators of copyrighted corpora against large language model (LLM) developers, we propose a novel system, a variant of a plagiarism detection system, that assesses whether a knowledge source has been used in the training or fine-tuning of a large language model. Unlike current methods, we utilize an approach that uses Resource Description Framework (RDF) triples to create knowledge graphs from both a source document and a LLM continuation of that document. These graphs are then analyzed with respect to content using cosine similarity and with respect to structure using a normalized version of graph edit distance that shows the degree of isomorphism. Unlike traditional systems that focus on content matching and keyword identification between a source and target corpus, our approach enables a broader evaluation of similarity and thus a more accurate comparison of the similarity between a source document and LLM continuation by focusing on relationships between ideas and their organization with regards to others. Additionally, our approach does not require access to LLM metrics like perplexity that may be unavailable in closed large language modeling "black-box" systems, as well as the training corpus. A prototype of our system will be found on a hyperlinked GitHub repository.
• 摘要：鉴于最近出版商、报纸和其他版权语料库创建者对大型语言模型 (LLM) 开发人员的剽窃指控，我们提出了一种新系统，即剽窃检测系统的变体，用于评估知识源是否已用于大型语言模型的训练或微调。与当前方法不同，我们采用一种使用资源描述框架 (RDF) 三元组从源文档和该文档的 LLM 延续创建知识图谱的方法。然后使用余弦相似度分析这些图的内容，使用显示同构程度的图编辑距离的规范化版本分析这些图的结构。与专注于源语料库和目标语料库之间的内容匹配和关键字识别的传统系统不同，我们的方法通过关注思想及其组织与其他思想之间的关系，可以更广泛地评估相似性，从而更准确地比较源文档和 LLM 延续之间的相似性。此外，我们的方法不需要访问 LLM 指标（例如困惑度），这些指标在封闭的大型语言建模“黑盒”系统中可能无法获得，也不需要访问训练语料库。我们的系统原型可以在超链接的 GitHub 存储库中找到。

Title: Boosting Biomedical Concept Extraction by Rule-Based Data Augmentation

• Authors: Qiwei Shao, Fengran Mo, Jian-Yun Nie
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.02719
• Pdf URL: https://arxiv.org/pdf/2407.02719
• Copy Paste: [[2407.02719]] Boosting Biomedical Concept Extraction by Rule-Based Data Augmentation(https://arxiv.org/abs/2407.02719)
• Keywords: language model
• Abstract: Document-level biomedical concept extraction is the task of identifying biomedical concepts mentioned in a given document. Recent advancements have adapted pre-trained language models for this task. However, the scarcity of domain-specific data and the deviation of concepts from their canonical names often hinder these models' effectiveness. To tackle this issue, we employ MetaMapLite, an existing rule-based concept mapping system, to generate additional pseudo-annotated data from PubMed and PMC. The annotated data are used to augment the limited training data. Through extensive experiments, this study demonstrates the utility of a manually crafted concept mapping tool for training a better concept extraction model.
• 摘要：文档级生物医学概念提取是识别给定文档中提到的生物医学概念的任务。最近的进展已经针对此任务调整了预训练语言模型。然而，领域特定数据的稀缺性和概念与其规范名称的偏差通常会阻碍这些模型的有效性。为了解决这个问题，我们使用现有的基于规则的概念映射系统 MetaMapLite 从 PubMed 和 PMC 生成额外的伪注释数据。注释数据用于增强有限的训练数据。通过大量实验，本研究证明了手工制作的概念映射工具在训练更好的概念提取模型方面的实用性。

Title: e-Health CSIRO at "Discharge Me!" 2024: Generating Discharge Summary Sections with Fine-tuned Language Models

• Authors: Jinghui Liu, Aaron Nicolson, Jason Dowling, Bevan Koopman, Anthony Nguyen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.02723
• Pdf URL: https://arxiv.org/pdf/2407.02723
• Copy Paste: [[2407.02723]] e-Health CSIRO at "Discharge Me!" 2024: Generating Discharge Summary Sections with Fine-tuned Language Models(https://arxiv.org/abs/2407.02723)
• Keywords: language model
• Abstract: Clinical documentation is an important aspect of clinicians' daily work and often demands a significant amount of time. The BioNLP 2024 Shared Task on Streamlining Discharge Documentation (Discharge Me!) aims to alleviate this documentation burden by automatically generating discharge summary sections, including brief hospital course and discharge instruction, which are often time-consuming to synthesize and write manually. We approach the generation task by fine-tuning multiple open-sourced language models (LMs), including both decoder-only and encoder-decoder LMs, with various configurations on input context. We also examine different setups for decoding algorithms, model ensembling or merging, and model specialization. Our results show that conditioning on the content of discharge summary prior to the target sections is effective for the generation task. Furthermore, we find that smaller encoder-decoder LMs can work as well or even slightly better than larger decoder based LMs fine-tuned through LoRA. The model checkpoints from our team (aehrc) are openly available.
• 摘要：临床文档是临床医生日常工作的一个重要方面，通常需要大量时间。BioNLP 2024 简化出院文档共享任务 (Discharge Me!) 旨在通过自动生成出院摘要部分（包括简短的医院病程和出院指导）来减轻这种文档负担，而这些部分通常需要耗费大量时间才能手动合成和编写。我们通过微调多个开源语言模型 (LM) 来完成生成任务，包括仅解码器和编码器-解码器 LM，并对输入上下文进行各种配置。我们还研究了解码算法、模型集成或合并以及模型专业化的不同设置。我们的结果表明，在目标部分之前对出院摘要的内容进行条件调节对于生成任务是有效的。此外，我们发现较小的编码器-解码器 LM 可以与通过 LoRA 微调的基于较大解码器的 LM 一样好甚至略好。我们团队 (aehrc) 的模型检查点是公开的。

Title: MentalAgora: A Gateway to Advanced Personalized Care in Mental Health through Multi-Agent Debating and Attribute Control

• Authors: Yeonji Lee, Sangjun Park, Kyunghyun Cho, JinYeong Bak
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.02736
• Pdf URL: https://arxiv.org/pdf/2407.02736
• Copy Paste: [[2407.02736]] MentalAgora: A Gateway to Advanced Personalized Care in Mental Health through Multi-Agent Debating and Attribute Control(https://arxiv.org/abs/2407.02736)
• Keywords: language model, agent
• Abstract: As mental health issues globally escalate, there is a tremendous need for advanced digital support systems. We introduce MentalAgora, a novel framework employing large language models enhanced by interaction between multiple agents for tailored mental health support. This framework operates through three stages: strategic debating, tailored counselor creation, and response generation, enabling the dynamic customization of responses based on individual user preferences and therapeutic needs. We conduct experiments utilizing a high-quality evaluation dataset TherapyTalk crafted with mental health professionals, shwoing that MentalAgora generates expert-aligned and user preference-enhanced responses. Our evaluations, including experiments and user studies, demonstrate that MentalAgora aligns with professional standards and effectively meets user preferences, setting a new benchmark for digital mental health interventions.
• 摘要：随着全球心理健康问题不断升级，对先进数字支持系统的需求巨大。我们推出了 MentalAgora，这是一个新颖的框架，采用大型语言模型，通过多个代理之间的交互得到增强，以提供量身定制的心理健康支持。该框架通过三个阶段运行：战略辩论、量身定制的咨询师创建和响应生成，可根据个人用户偏好和治疗需求动态定制响应。我们利用与心理健康专业人士共同制作的高质量评估数据集 TherapyTalk 进行实验，结果表明 MentalAgora 生成了符合专家标准且增强了用户偏好的响应。我们的评估（包括实验和用户研究）表明，MentalAgora 符合专业标准并有效满足用户偏好，为数字心理健康干预树立了新的标杆。

Title: Learning to Reduce: Towards Improving Performance of Large Language Models on Structured Data

• Authors: Younghun Lee, Sungchul Kim, Ryan A. Rossi, Tong Yu, Xiang Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.02750
• Pdf URL: https://arxiv.org/pdf/2407.02750
• Copy Paste: [[2407.02750]] Learning to Reduce: Towards Improving Performance of Large Language Models on Structured Data(https://arxiv.org/abs/2407.02750)
• Keywords: language model, gpt, llm
• Abstract: Large Language Models (LLMs) have been achieving competent performance on a wide range of downstream tasks, yet existing work shows that inference on structured data is challenging for LLMs. This is because LLMs need to either understand long structured data or select the most relevant evidence before inference, and both approaches are not trivial. This paper proposes a framework, Learning to Reduce, that fine-tunes a language model with On-Policy Learning to generate a reduced version of an input structured data. When compared to state-of-the-art LLMs like GPT-4, Learning to Reduce not only achieves outstanding performance in reducing the input, but shows generalizability on different datasets. We further show that the model fine-tuned with our framework helps LLMs better perform on table QA tasks especially when the context is longer.
• 摘要：大型语言模型 (LLM) 在广泛的下游任务中都表现出色，但现有研究表明，对结构化数据进行推理对 LLM 来说是一项挑战。这是因为 LLM 需要理解长结构化数据或在推理之前选择最相关的证据，而这两种方法都并非易事。本文提出了一个框架，即 Learning to Reduce，它使用在线策略学习对语言模型进行微调，以生成输入结构化数据的简化版本。与 GPT-4 等最先进的 LLM 相比，Learning to Reduce 不仅在减少输入方面取得了出色的表现，而且在不同数据集上表现出色。我们进一步表明，使用我们的框架微调的模型可帮助 LLM 在表格 QA 任务中表现更好，尤其是在上下文较长的情况下。

Title: MLKD-BERT: Multi-level Knowledge Distillation for Pre-trained Language Models

• Authors: Ying Zhang, Ziheng Yang, Shufan Ji
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.02775
• Pdf URL: https://arxiv.org/pdf/2407.02775
• Copy Paste: [[2407.02775]] MLKD-BERT: Multi-level Knowledge Distillation for Pre-trained Language Models(https://arxiv.org/abs/2407.02775)
• Keywords: language model
• Abstract: Knowledge distillation is an effective technique for pre-trained language model compression. Although existing knowledge distillation methods perform well for the most typical model BERT, they could be further improved in two aspects: the relation-level knowledge could be further explored to improve model performance; and the setting of student attention head number could be more flexible to decrease inference time. Therefore, we are motivated to propose a novel knowledge distillation method MLKD-BERT to distill multi-level knowledge in teacher-student framework. Extensive experiments on GLUE benchmark and extractive question answering tasks demonstrate that our method outperforms state-of-the-art knowledge distillation methods on BERT. In addition, MLKD-BERT can flexibly set student attention head number, allowing for substantial inference time decrease with little performance drop.
• 摘要：知识蒸馏是一种有效的预训练语言模型压缩技术。尽管现有的知识蒸馏方法对于最典型的模型 BERT 表现良好，但它们可以在两个方面进一步改进：可以进一步探索关系级知识以提高模型性能；学生注意头数量的设置可以更灵活以减少推理时间。因此，我们提出一种新颖的知识蒸馏方法 MLKD-BERT 来蒸馏师生框架中的多级知识。在 GLUE 基准和提取式问答任务上进行的大量实验表明，我们的方法优于 BERT 上最先进的知识蒸馏方法。此外，MLKD-BERT 可以灵活设置学生注意头数量，从而大幅减少推理时间而性能几乎不下降。

Title: 52B to 1T: Lessons Learned via Tele-FLM Series

• Authors: Xiang Li, Yiqun Yao, Xin Jiang, Xuezhi Fang, Chao Wang, Xinzhang Liu, Zihan Wang, Yu Zhao, Xin Wang, Yuyao Huang, Shuangyong Song, Yongxiang Li, Zheng Zhang, Bo Zhao, Aixin Sun, Yequan Wang, Zhongjiang He, Zhongyuan Wang, Xuelong Li, Tiejun Huang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.02783
• Pdf URL: https://arxiv.org/pdf/2407.02783
• Copy Paste: [[2407.02783]] 52B to 1T: Lessons Learned via Tele-FLM Series(https://arxiv.org/abs/2407.02783)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) represent a significant stride toward Artificial General Intelligence. As scaling laws underscore the potential of increasing model sizes, the academic community has intensified its investigations into LLMs with capacities exceeding 50 billion parameters. This technical report builds on our prior work with Tele-FLM (also known as FLM-2), a publicly available 52-billion-parameter model. We delve into two primary areas: we first discuss our observation of Supervised Fine-tuning (SFT) on Tele-FLM-52B, which supports the "less is more" approach for SFT data construction; second, we demonstrate our experiments and analyses on the best practices for progressively growing a model from 52 billion to 102 billion, and subsequently to 1 trillion parameters. We will open-source a 1T model checkpoint, namely Tele-FLM-1T, to advance further training and research.
• 摘要：大型语言模型 (LLM) 代表着向通用人工智能迈出的一大步。由于缩放定律强调了增加模型大小的潜力，学术界已加强对容量超过 500 亿个参数的 LLM 的研究。本技术报告基于我们之前对 Tele-FLM（也称为 FLM-2）的研究，这是一个公开的 520 亿个参数模型。我们深入研究了两个主要领域：首先，我们讨论对 Tele-FLM-52B 上的监督微调 (SFT) 的观察，它支持 SFT 数据构建的“少即是多”方法；其次，我们展示了我们的实验和分析，这些实验和分析是关于逐步将模型从 520 亿个参数扩展到 1020 亿个参数，随后扩展到 1 万亿个参数的最佳实践。我们将开源一个 1T 模型检查点，即 Tele-FLM-1T，以推进进一步的训练和研究。

Title: Efficient Training of Language Models with Compact and Consistent Next Token Distributions

• Authors: Ashutosh Sathe, Sunita Sarawagi
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.02819
• Pdf URL: https://arxiv.org/pdf/2407.02819
• Copy Paste: [[2407.02819]] Efficient Training of Language Models with Compact and Consistent Next Token Distributions(https://arxiv.org/abs/2407.02819)
• Keywords: language model
• Abstract: Maximizing the likelihood of the next token is an established, statistically sound objective for pre-training language models. In this paper we show that we can train better models faster by pre-aggregating the corpus with a collapsed $n$-gram distribution. Previous studies have proposed corpus-level $n$-gram statistics as a regularizer; however, the construction and querying of such $n$-grams, if done naively, prove to be costly and significantly impede training speed, thereby limiting their application in modern large language model pre-training. We introduce an alternative compact representation of the next token distribution that, in expectation, aligns with the complete $n$-gram distribution while markedly reducing variance across mini-batches compared to the standard next-token loss. Empirically, we demonstrate that both the $n$-gram regularized model and our approximation yield substantial improvements in model quality and convergence rate compared to existing methods. Furthermore, our approximation facilitates scalability of gains to larger datasets and models compared to the straightforward $n$-gram regularization method.
• 摘要：最大化下一个标记的可能性是预训练语言模型的既定、统计上合理的目标。在本文中，我们展示了通过预聚合具有折叠 $n$-gram 分布的语料库，我们可以更快地训练出更好的模型。先前的研究已提出将语料库级 $n$-gram 统计数据作为正则化器；然而，如果简单地构建和查询此类 $n$-gram，则会证明成本高昂且会严重阻碍训练速度，从而限制其在现代大型语言模型预训练中的应用。我们引入了下一个标记分布的另一种紧凑表示，期望它与完整的 $n$-gram 分布一致，同时与标准下一个标记损失相比，显着降低了小批量之间的差异。从经验上讲，我们证明与现有方法相比，$n$-gram 正则化模型和我们的近似值均可显着提高模型质量和收敛速度。此外，与简单的 $n$-gram 正则化方法相比，我们的近似有助于将收益扩展到更大的数据集和模型。

Title: FSM: A Finite State Machine Based Zero-Shot Prompting Paradigm for Multi-Hop Question Answering

• Authors: Xiaochen Wang, Junqing He, Zhe yang, Yiru Wang, Xiangdi Meng, Kunhao Pan, Zhifang Sui
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.02964
• Pdf URL: https://arxiv.org/pdf/2407.02964
• Copy Paste: [[2407.02964]] FSM: A Finite State Machine Based Zero-Shot Prompting Paradigm for Multi-Hop Question Answering(https://arxiv.org/abs/2407.02964)
• Keywords: language model, llm, hallucination, prompt, chain-of-thought
• Abstract: Large Language Models (LLMs) with chain-of-thought (COT) prompting have demonstrated impressive abilities on simple nature language inference tasks. However, they tend to perform poorly on Multi-hop Question Answering (MHQA) tasks due to several challenges, including hallucination, error propagation and limited context length. We propose a prompting method, Finite State Machine (FSM) to enhance the reasoning capabilities of LLM for complex tasks in addition to improved effectiveness and trustworthiness. Different from COT methods, FSM addresses MHQA by iteratively decomposing a question into multi-turn sub-questions, and self-correcting in time, improving the accuracy of answers in each step. Specifically, FSM addresses one sub-question at a time and decides on the next step based on its current result and state, in an automaton-like format. Experiments on benchmarks show the effectiveness of our method. Although our method performs on par with the baseline on relatively simpler datasets, it excels on challenging datasets like Musique. Moreover, this approach mitigates the hallucination phenomenon, wherein the correct final answer can be recovered despite errors in intermediate reasoning. Furthermore, our method improves LLMs' ability to follow specified output format requirements, significantly reducing the difficulty of answer interpretation and the need for reformatting.
• 摘要：具有思路链 (COT) 提示的大型语言模型 (LLM) 在简单的自然语言推理任务上表现出色。然而，由于幻觉、错误传播和有限的上下文长度等挑战，它们在多跳问答 (MHQA) 任务上表现不佳。我们提出了一种提示方法，有限状态机 (FSM)，以增强 LLM 对复杂任务的推理能力，并提高有效性和可信度。与 COT 方法不同，FSM 通过将问题迭代分解为多轮子问题并及时自我纠正来解决 MHQA，从而提高每一步答案的准确性。具体来说，FSM 一次解决一个子问题，并根据其当前结果和状态以类似自动机的形式决定下一步。基准测试中的实验表明了我们方法的有效性。虽然我们的方法在相对简单的数据集上的表现与基线相当，但它在 Musique 等具有挑战性的数据集上表现出色。此外，这种方法还可以缓解幻觉现象，即尽管中间推理出现错误，但最终答案仍可恢复正确。此外，我们的方法提高了 LLM 遵循指定输出格式要求的能力，大大降低了答案解释的难度和重新格式化的需要。

Title: Large Language Models as Evaluators for Scientific Synthesis

• Authors: Julia Evans, Jennifer D'Souza, Sören Auer
• Subjects: cs.CL, cs.AI, cs.IT
• Abstract URL: https://arxiv.org/abs/2407.02977
• Pdf URL: https://arxiv.org/pdf/2407.02977
• Copy Paste: [[2407.02977]] Large Language Models as Evaluators for Scientific Synthesis(https://arxiv.org/abs/2407.02977)
• Keywords: language model, gpt, llm
• Abstract: Our study explores how well the state-of-the-art Large Language Models (LLMs), like GPT-4 and Mistral, can assess the quality of scientific summaries or, more fittingly, scientific syntheses, comparing their evaluations to those of human annotators. We used a dataset of 100 research questions and their syntheses made by GPT-4 from abstracts of five related papers, checked against human quality ratings. The study evaluates both the closed-source GPT-4 and the open-source Mistral model's ability to rate these summaries and provide reasons for their judgments. Preliminary results show that LLMs can offer logical explanations that somewhat match the quality ratings, yet a deeper statistical analysis shows a weak correlation between LLM and human ratings, suggesting the potential and current limitations of LLMs in scientific synthesis evaluation.
• 摘要：我们的研究探索了 GPT-4 和 Mistral 等最先进的大型语言模型 (LLM) 如何评估科学摘要或更恰当地说是科学综合的质量，并将其评估结果与人类注释者的评估结果进行比较。我们使用了由 GPT-4 从五篇相关论文的摘要中得出的 100 个研究问题及其综合的数据集，并与人类质量评级进行了核对。该研究评估了闭源 GPT-4 和开源 Mistral 模型对这些摘要进行评级并提供判断理由的能力。初步结果表明，LLM 可以提供与质量评级相当的逻辑解释，但更深入的统计分析显示 LLM 与人类评级之间的相关性较弱，这表明 LLM 在科学综合评估方面的潜在和当前局限性。

Title: Mast Kalandar at SemEval-2024 Task 8: On the Trail of Textual Origins: RoBERTa-BiLSTM Approach to Detect AI-Generated Text

• Authors: Jainit Sushil Bafna, Hardik Mittal, Suyash Sethia, Manish Shrivastava, Radhika Mamidi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.02978
• Pdf URL: https://arxiv.org/pdf/2407.02978
• Copy Paste: [[2407.02978]] Mast Kalandar at SemEval-2024 Task 8: On the Trail of Textual Origins: RoBERTa-BiLSTM Approach to Detect AI-Generated Text(https://arxiv.org/abs/2407.02978)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have showcased impressive abilities in generating fluent responses to diverse user queries. However, concerns regarding the potential misuse of such texts in journalism, educational, and academic contexts have surfaced. SemEval 2024 introduces the task of Multigenerator, Multidomain, and Multilingual Black-Box Machine-Generated Text Detection, aiming to develop automated systems for identifying machine-generated text and detecting potential misuse. In this paper, we i) propose a RoBERTa-BiLSTM based classifier designed to classify text into two categories: AI-generated or human ii) conduct a comparative study of our model with baseline approaches to evaluate its effectiveness. This paper contributes to the advancement of automatic text detection systems in addressing the challenges posed by machine-generated text misuse. Our architecture ranked 46th on the official leaderboard with an accuracy of 80.83 among 125.
• 摘要：大型语言模型 (LLM) 已展示出对各种用户查询生成流畅响应的出色能力。然而，人们开始担心此类文本在新闻、教育和学术环境中可能被滥用。SemEval 2024 引入了多生成器、多领域和多语言黑盒机器生成文本检测任务，旨在开发用于识别机器生成文本和检测潜在滥用的自动化系统。在本文中，我们 i) 提出了一种基于 RoBERTa-BiLSTM 的分类器，旨在将文本分为两类：人工智能生成的或人类生成的 ii) 对我们的模型与基线方法进行比较研究以评估其有效性。本文为自动文本检测系统在解决机器生成文本滥用带来的挑战方面的进步做出了贡献。我们的架构在官方排行榜上排名第 46 位，在 125 个中准确率为 80.83。

Title: Are Large Language Models Consistent over Value-laden Questions?

• Authors: Jared Moore, Tanvi Deshpande, Diyi Yang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.02996
• Pdf URL: https://arxiv.org/pdf/2407.02996
• Copy Paste: [[2407.02996]] Are Large Language Models Consistent over Value-laden Questions?(https://arxiv.org/abs/2407.02996)
• Keywords: language model, gpt, llm
• Abstract: Large language models (LLMs) appear to bias their survey answers toward certain values. Nonetheless, some argue that LLMs are too inconsistent to simulate particular values. Are they? To answer, we first define value consistency as the similarity of answers across (1) paraphrases of one question, (2) related questions under one topic, (3) multiple-choice and open-ended use-cases of one question, and (4) multilingual translations of a question to English, Chinese, German, and Japanese. We apply these measures to a few large ($>=34b$), open LLMs including llama-3, as well as gpt-4o, using eight thousand questions spanning more than 300 topics. Unlike prior work, we find that models are relatively consistent across paraphrases, use-cases, translations, and within a topic. Still, some inconsistencies remain. Models are more consistent on uncontroversial topics (e.g., in the U.S., "Thanksgiving") than on controversial ones ("euthanasia"). Base models are both more consistent compared to fine-tuned models and are uniform in their consistency across topics, while fine-tuned models are more inconsistent about some topics ("euthanasia") than others ("women's rights") like our human subjects (n=165).
• 摘要：大型语言模型 (LLM) 似乎会使其调查答案偏向某些值。尽管如此，一些人认为 LLM 太不一致，无法模拟特定的价值观。是吗？为了回答这个问题，我们首先将价值一致性定义为 (1) 一个问题的释义、(2) 一个主题下的相关问题、(3) 一个问题的多项选择题和开放式用例以及 (4) 问题到英语、中文、德语和日语的多语言翻译之间的答案相似性。我们将这些措施应用于一些大型（>=340 亿美元）开放式 LLM，包括 llama-3 和 gpt-4o，使用了涵盖 300 多个主题的八千个问题。与之前的研究不同，我们发现模型在释义、用例、翻译和主题内相对一致。不过，仍然存在一些不一致之处。模型对无争议话题（例如，在美国，“感恩节”）的一致性要高于对有争议话题（“安乐死”）的一致性。与微调模型相比，基础模型的一致性更高，并且在各个话题上的一致性也保持一致，而微调模型在某些话题（“安乐死”）上的一致性要高于其他话题（“妇女权利”），例如我们的人类受试者（n=165）。

Title: VIVA: A Benchmark for Vision-Grounded Decision-Making with Human Values

• Authors: Zhe Hu, Yixiao Ren, Jing Li, Yu Yin
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2407.03000
• Pdf URL: https://arxiv.org/pdf/2407.03000
• Copy Paste: [[2407.03000]] VIVA: A Benchmark for Vision-Grounded Decision-Making with Human Values(https://arxiv.org/abs/2407.03000)
• Keywords: language model
• Abstract: This paper introduces VIVA, a benchmark for VIsion-grounded decision-making driven by human VAlues. While most large vision-language models (VLMs) focus on physical-level skills, our work is the first to examine their multimodal capabilities in leveraging human values to make decisions under a vision-depicted situation. VIVA contains 1,062 images depicting diverse real-world situations and the manually annotated decisions grounded in them. Given an image there, the model should select the most appropriate action to address the situation and provide the relevant human values and reason underlying the decision. Extensive experiments based on VIVA show the limitation of VLMs in using human values to make multimodal decisions. Further analyses indicate the potential benefits of exploiting action consequences and predicted human values.
• 摘要：本文介绍了 VIVA，这是以人类价值观为驱动力的基于视觉的决策的基准。虽然大多数大型视觉语言模型 (VLM) 都专注于物理层面的技能，但我们的工作是首次研究它们在利用人类价值观在视觉描绘的情况下做出决策的多模态能力。VIVA 包含 1,062 张描绘各种现实世界情况的图像以及基于这些情况的手动注释决策。给定一张图像，模型应该选择最合适的行动来解决这种情况，并提供决策背后的相关人类价值观和理由。基于 VIVA 的大量实验表明，VLM 在使用人类价值观做出多模态决策方面的局限性。进一步的分析表明，利用行动后果和预测的人类价值观具有潜在的好处。

Title: SemioLLM: Assessing Large Language Models for Semiological Analysis in Epilepsy Research

• Authors: Meghal Dani, Muthu Jeyanthi Prakash, Zeynep Akata, Stefanie Liebe
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.03004
• Pdf URL: https://arxiv.org/pdf/2407.03004
• Copy Paste: [[2407.03004]] SemioLLM: Assessing Large Language Models for Semiological Analysis in Epilepsy Research(https://arxiv.org/abs/2407.03004)
• Keywords: language model, gpt, llm, hallucination, prompt, chat
• Abstract: Large Language Models have shown promising results in their ability to encode general medical knowledge in standard medical question-answering datasets. However, their potential application in clinical practice requires evaluation in domain-specific tasks, where benchmarks are largely missing. In this study semioLLM, we test the ability of state-of-the-art LLMs (GPT-3.5, GPT-4, Mixtral 8x7B, and Qwen-72chat) to leverage their internal knowledge and reasoning for epilepsy diagnosis. Specifically, we obtain likelihood estimates linking unstructured text descriptions of seizures to seizure-generating brain regions, using an annotated clinical database containing 1269 entries. We evaluate the LLM's performance, confidence, reasoning, and citation abilities in comparison to clinical evaluation. Models achieve above-chance classification performance with prompt engineering significantly improving their outcome, with some models achieving close-to-clinical performance and reasoning. However, our analyses also reveal significant pitfalls with several models being overly confident while showing poor performance, as well as exhibiting citation errors and hallucinations. In summary, our work provides the first extensive benchmark comparing current SOTA LLMs in the medical domain of epilepsy and highlights their ability to leverage unstructured texts from patients' medical history to aid diagnostic processes in health care.
• 摘要：大型语言模型在标准医学问答数据集中编码一般医学知识的能力方面表现出了良好的效果。然而，它们在临床实践中的潜在应用需要在特定领域的任务中进行评估，而这些任务在很大程度上缺乏基准。在这项研究 semioLLM 中，我们测试了最先进的 LLM（GPT-3.5、GPT-4、Mixtral 8x7B 和 Qwen-72chat）利用其内部知识和推理进行癫痫诊断的能力。具体来说，我们使用包含 1269 个条目的带注释临床数据库，获得将癫痫发作的非结构化文本描述与癫痫发作大脑区域联系起来的可能性估计。我们与临床评估相比，评估了 LLM 的性能、置信度、推理和引用能力。模型实现了高于偶然的分类性能，及时的工程显著改善了它们的结果，一些模型实现了接近临床的性能和推理。然而，我们的分析也揭示了重大缺陷，一些模型过于自信，但性能不佳，并且存在引用错误和幻觉。总之，我们的工作提供了第一个广泛的基准，比较了癫痫医学领域当前的 SOTA LLM，并强调了它们利用患者病史中的非结构化文本来辅助医疗保健诊断过程的能力。

Title: What Affects the Stability of Tool Learning? An Empirical Study on the Robustness of Tool Learning Frameworks

• Authors: Chengrui Huang, Zhengliang Shi, Yuntao Wen, Xiuying Chen, Peng Han, Shen Gao, Shuo Shang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.03007
• Pdf URL: https://arxiv.org/pdf/2407.03007
• Copy Paste: [[2407.03007]] What Affects the Stability of Tool Learning? An Empirical Study on the Robustness of Tool Learning Frameworks(https://arxiv.org/abs/2407.03007)
• Keywords: language model, llm, prompt
• Abstract: Tool learning methods have enhanced the ability of large language models (LLMs) to interact with real-world applications. Many existing works fine-tune LLMs or design prompts to enable LLMs to select appropriate tools and correctly invoke them to meet user requirements. However, it is observed in previous works that the performance of tool learning varies from tasks, datasets, training settings, and algorithms. Without understanding the impact of these factors, it can lead to inconsistent results, inefficient model deployment, and suboptimal tool utilization, ultimately hindering the practical integration and scalability of LLMs in real-world scenarios. Therefore, in this paper, we explore the impact of both internal and external factors on the performance of tool learning frameworks. Through extensive experiments on two benchmark datasets, we find several insightful conclusions for future work, including the observation that LLMs can benefit significantly from increased trial and exploration. We believe our empirical study provides a new perspective for future tool learning research.
• 摘要：工具学习方法增强了大型语言模型 (LLM) 与实际应用交互的能力。许多现有工作对 LLM 进行微调或设计提示，使 LLM 能够选择合适的工具并正确调用它们以满足用户需求。然而，在以前的工作中观察到，工具学习的性能因任务、数据集、训练设置和算法而异。如果不了解这些因素的影响，可能会导致结果不一致、模型部署效率低下和工具利用率不理想，最终阻碍 LLM 在实际场景中的实际集成和可扩展性。因此，在本文中，我们探讨了内部和外部因素对工具学习框架性能的影响。通过在两个基准数据集上的大量实验，我们发现了一些对未来工作的有见地的结论，包括观察到 LLM 可以从增加的试验和探索中受益匪浅。我们相信我们的实证研究为未来的工具学习研究提供了一个新的视角。

Title: Strategies for Arabic Readability Modeling

• Authors: Juan Piñeros Liberato, Bashar Alhafni, Muhamed Al Khalil, Nizar Habash
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.03032
• Pdf URL: https://arxiv.org/pdf/2407.03032
• Copy Paste: [[2407.03032]] Strategies for Arabic Readability Modeling(https://arxiv.org/abs/2407.03032)
• Keywords: language model
• Abstract: Automatic readability assessment is relevant to building NLP applications for education, content analysis, and accessibility. However, Arabic readability assessment is a challenging task due to Arabic's morphological richness and limited readability resources. In this paper, we present a set of experimental results on Arabic readability assessment using a diverse range of approaches, from rule-based methods to Arabic pretrained language models. We report our results on a newly created corpus at different textual granularity levels (words and sentence fragments). Our results show that combining different techniques yields the best results, achieving an overall macro F1 score of 86.7 at the word level and 87.9 at the fragment level on a blind test set. We make our code, data, and pretrained models publicly available.
• 摘要：自动可读性评估与构建用于教育、内容分析和可访问性的 NLP 应用程序相关。然而，由于阿拉伯语的形态丰富性和可读性资源有限，阿拉伯语可读性评估是一项具有挑战性的任务。在本文中，我们使用从基于规则的方法到阿拉伯语预训练语言模型等多种方法，展示了一组关于阿拉伯语可读性评估的实验结果。我们在新创建的语料库上报告了不同文本粒度级别（单词和句子片段）的结果。我们的结果表明，结合不同的技术可以产生最佳结果，在盲测集上，单词级别的总体宏观 F1 得分为 86.7，片段级别的总体宏观 F1 得分为 87.9。我们公开了我们的代码、数据和预训练模型。

Title: On the Client Preference of LLM Fine-tuning in Federated Learning

• Authors: Feijie Wu, Xiaoze Liu, Haoyu Wang, Xingchen Wang, Jing Gao
• Subjects: cs.CL, cs.DC, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.03038
• Pdf URL: https://arxiv.org/pdf/2407.03038
• Copy Paste: [[2407.03038]] On the Client Preference of LLM Fine-tuning in Federated Learning(https://arxiv.org/abs/2407.03038)
• Keywords: language model, llm
• Abstract: Reinforcement learning with human feedback (RLHF) fine-tunes a pretrained large language model (LLM) using preference datasets, enabling the LLM to generate outputs that align with human preferences. Given the sensitive nature of these preference datasets held by various clients, there is a need to implement RLHF within a federated learning (FL) framework, where clients are reluctant to share their data due to privacy concerns. To address this, we introduce a feasible framework in which clients collaboratively train a binary selector with their preference datasets using our proposed FedBis. With a well-trained selector, we can further enhance the LLM that generates human-preferred completions. Meanwhile, we propose a novel algorithm, FedBiscuit, that trains multiple selectors by organizing clients into balanced and disjoint clusters based on their preferences. Compared to the FedBis, FedBiscuit demonstrates superior performance in simulating human preferences for pairwise completions. Our extensive experiments on federated human preference datasets -- marking the first benchmark to address heterogeneous data partitioning among clients -- demonstrate that FedBiscuit outperforms FedBis and even surpasses traditional centralized training.
• 摘要：带人类反馈的强化学习 (RLHF) 使用偏好数据集对预训练的大型语言模型 (LLM) 进行微调，使 LLM 能够生成符合人类偏好的输出。鉴于不同客户端持有的这些偏好数据集的敏感性，需要在联邦学习 (FL) 框架内实现 RLHF，而客户端出于隐私考虑不愿共享其数据。为了解决这个问题，我们引入了一个可行的框架，其中客户端使用我们提出的 FedBis 协作训练二元选择器及其偏好数据集。借助训练有素的选择器，我们可以进一步增强生成人类偏好补全的 LLM。同时，我们提出了一种新算法 FedBiscuit，该算法通过根据客户端的偏好将客户端组织成平衡且不相交的集群来训练多个选择器。与 FedBis 相比，FedBiscuit 在模拟人类对成对补全的偏好方面表现出色。我们对联合人类偏好数据集进行的大量实验（标志着第一个解决客户端间异构数据分区的基准）表明，FedBiscuit 的表现优于 FedBis，甚至超过了传统的集中式训练。

Title: Raw Text is All you Need: Knowledge-intensive Multi-turn Instruction Tuning for Large Language Model

• Authors: Xia Hou, Qifeng Li, Jian Yang, Tongliang Li, Linzheng Chai, Xianjie Wu, Hangyuan Ji, Zhoujun Li, Jixuan Nie, Jingbo Dun, Wenfeng Song
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.03040
• Pdf URL: https://arxiv.org/pdf/2407.03040
• Copy Paste: [[2407.03040]] Raw Text is All you Need: Knowledge-intensive Multi-turn Instruction Tuning for Large Language Model(https://arxiv.org/abs/2407.03040)
• Keywords: language model, llm, prompt
• Abstract: Instruction tuning as an effective technique aligns the outputs of large language models (LLMs) with human preference. But how to generate the seasonal multi-turn dialogues from raw documents for instruction tuning still requires further exploration. In this paper, we present a novel framework named R2S that leverages the CoD-Chain of Dialogue logic to guide large language models (LLMs) in generating knowledge-intensive multi-turn dialogues for instruction tuning. By integrating raw documents from both open-source datasets and domain-specific web-crawled documents into a benchmark K-BENCH, we cover diverse areas such as Wikipedia (English), Science (Chinese), and Artifacts (Chinese). Our approach first decides the logic flow of the current dialogue and then prompts LLMs to produce key phrases for sourcing relevant response content. This methodology enables the creation of the G I NSTRUCT instruction dataset, retaining raw document knowledge within dialoguestyle interactions. Utilizing this dataset, we fine-tune GLLM, a model designed to transform raw documents into structured multi-turn dialogues, thereby injecting comprehensive domain knowledge into the SFT model for enhanced instruction tuning. This work signifies a stride towards refining the adaptability and effectiveness of LLMs in processing and generating more accurate, contextually nuanced responses across various fields.
• 摘要：指令调整是一种有效的技术，可以使大型语言模型 (LLM) 的输出与人类偏好保持一致。但是，如何从原始文档生成季节性多轮对话以进行指令调整仍需要进一步探索。在本文中，我们提出了一个名为 R2S 的新框架，该框架利用对话逻辑的 CoD-Chain 来指导大型语言模型 (LLM) 生成知识密集型的多轮对话以进行指令调整。通过将来自开源数据集和特定领域的网络爬取文档的原始文档集成到基准 K-BENCH 中，我们涵盖了维基百科（英文）、科学（中文）和文物（中文）等不同领域。我们的方法首先确定当前对话的逻辑流程，然后提示 LLM 生成关键短语以获取相关的响应内容。这种方法可以创建 G I NSTRUCT 指令数据集，在对话式交互中保留原始文档知识。利用此数据集，我们对 GLLM 模型进行了微调，该模型旨在将原始文档转换为结构化的多轮对话，从而将全面的领域知识注入 SFT 模型以增强指令调整。这项工作标志着 LLM 在处理和生成更准确、更符合语境的跨领域响应方面的适应性和有效性方面迈出了一大步。

Title: Improving Conversational Abilities of Quantized Large Language Models via Direct Preference Alignment

• Authors: Janghwan Lee, Seongmin Park, Sukjin Hong, Minsoo Kim, Du-Seong Chang, Jungwook Choi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.03051
• Pdf URL: https://arxiv.org/pdf/2407.03051
• Copy Paste: [[2407.03051]] Improving Conversational Abilities of Quantized Large Language Models via Direct Preference Alignment(https://arxiv.org/abs/2407.03051)
• Keywords: language model, llm, chat
• Abstract: The rapid advancement of large language models (LLMs) has facilitated their transformation into conversational chatbots that can grasp contextual nuances and generate pertinent sentences, closely mirroring human values through advanced techniques such as instruction tuning and reinforcement learning from human feedback (RLHF). However, the computational efficiency required for LLMs, achieved through techniques like post-training quantization (PTQ), presents challenges such as token-flipping that can impair chatbot performance. In response, we propose a novel preference alignment approach, quantization-aware direct preference optimization (QDPO), that aligns quantized LLMs with their full-precision counterparts, improving conversational abilities. Evaluated on two instruction-tuned LLMs in various languages, QDPO demonstrated superior performance in improving conversational abilities compared to established PTQ and knowledge-distillation fine-tuning techniques, marking a significant step forward in the development of efficient and effective conversational LLMs.
• 摘要：大型语言模型 (LLM) 的快速发展促进了它们转变为对话聊天机器人，这些聊天机器人可以掌握上下文细微差别并生成相关句子，通过指令调整和从人类反馈中强化学习 (RLHF) 等先进技术，与人类价值观高度相似。然而，通过训练后量化 (PTQ) 等技术实现的 LLM 所需的计算效率带来了诸如标记翻转之类的挑战，这可能会损害聊天机器人的性能。为此，我们提出了一种新颖的偏好对齐方法，即量化感知直接偏好优化 (QDPO)，该方法将量化的 LLM 与全精度对应物对齐，从而提高对话能力。在不同语言的两种指令调整的 LLM 上进行评估，与成熟的 PTQ 和知识蒸馏微调技术相比，QDPO 在提高对话能力方面表现出色，标志着高效对话 LLM 的开发向前迈出了重要一步。

Title: ALTER: Augmentation for Large-Table-Based Reasoning

• Authors: Han Zhang, Yuheng Ma, Hanfang Yang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] ALTER: Augmentation for Large-Table-Based Reasoning(https://arxiv.org/abs/)
• Keywords: language model, llm
• Abstract: While extensive research has explored the use of large language models (LLMs) for table-based reasoning, most approaches struggle with scalability when applied to large tables. To maintain the superior comprehension abilities of LLMs in these scenarios, we introduce ALTER(Augmentation for Large-Table-Based Reasoning)-a framework designed to harness the latent augmentation potential in both free-form natural language (NL) questions, via the query augmentor, and semi-structured tabular data, through the table augmentor. By utilizing only a small subset of relevant data from the table and supplementing it with pre-augmented schema, semantic, and literal information, ALTER achieves outstanding performance on table-based reasoning benchmarks. We also provide a detailed analysis of large-table scenarios, comparing different methods and various partitioning principles. In these scenarios, our method outperforms all other approaches and exhibits robustness and efficiency against perturbations.
• 摘要：虽然已经有大量研究探索了使用大型语言模型 (LLM) 进行基于表格的推理，但大多数方法在应用于大型表格时都存在可扩展性问题。为了在这些场景中保持 LLM 的卓越理解能力，我们引入了 ALTER（基于大型表格的推理增强）框架，旨在通过查询增强器利用自由格式自然语言 (NL) 问题和通过表格增强器利用半结构化表格数据中的潜在增强潜力。通过仅利用表中的一小部分相关数据并补充预先增强的架构、语义和文字信息，ALTER 在基于表格的推理基准上取得了出色的表现。我们还对大型表格场景进行了详细分析，比较了不同的方法和各种分区原则。在这些场景中，我们的方法优于所有其他方法，并且表现出对干扰的稳健性和效率。

Title: Cactus: Towards Psychological Counseling Conversations using Cognitive Behavioral Theory

• Authors: Suyeon Lee, Sunghwan Kim, Minju Kim, Dongjin Kang, Dongil Yang, Harim Kim, Minseok Kang, Dayi Jung, Min Hee Kim, Seungbeen Lee, Kyoung-Mee Chung, Youngjae Yu, Dongha Lee, Jinyoung Yeo
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.03103
• Pdf URL: https://arxiv.org/pdf/2407.03103
• Copy Paste: [[2407.03103]] Cactus: Towards Psychological Counseling Conversations using Cognitive Behavioral Theory(https://arxiv.org/abs/2407.03103)
• Keywords: language model, llm, agent
• Abstract: Recently, the demand for psychological counseling has significantly increased as more individuals express concerns about their mental health. This surge has accelerated efforts to improve the accessibility of counseling by using large language models (LLMs) as counselors. To ensure client privacy, training open-source LLMs faces a key challenge: the absence of realistic counseling datasets. To address this, we introduce Cactus, a multi-turn dialogue dataset that emulates real-life interactions using the goal-oriented and structured approach of Cognitive Behavioral Therapy (CBT). We create a diverse and realistic dataset by designing clients with varied, specific personas, and having counselors systematically apply CBT techniques in their interactions. To assess the quality of our data, we benchmark against established psychological criteria used to evaluate real counseling sessions, ensuring alignment with expert evaluations. Experimental results demonstrate that Camel, a model trained with Cactus, outperforms other models in counseling skills, highlighting its effectiveness and potential as a counseling agent. We make our data, model, and code publicly available.
• 摘要：最近，随着越来越多的人表达对心理健康的担忧，对心理咨询的需求显著增加。这种激增加速了通过使用大型语言模型 (LLM) 作为咨询师来提高咨询可及性的努力。为了确保客户隐私，培训开源 LLM 面临一个关键挑战：缺乏真实的咨询数据集。为了解决这个问题，我们引入了 Cactus，这是一个多轮对话数据集，它使用认知行为疗法 (CBT) 的目标导向和结构化方法模拟现实生活中的互动。我们通过设计具有各种特定角色的客户，并让咨询师在互动中系统地应用 CBT 技术，创建了一个多样化且真实的数据集。为了评估我们数据的质量，我们根据用于评估真实咨询会议的既定心理标准进行基准测试，确保与专家评估保持一致。实验结果表明，使用 Cactus 训练的模型 Camel 在咨询技巧方面优于其他模型，凸显了其作为咨询代理的有效性和潜力。我们公开提供我们的数据、模型和代码。

Title: Social Bias Evaluation for Large Language Models Requires Prompt Variations

• Authors: Rem Hida, Masahiro Kaneko, Naoaki Okazaki
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.03129
• Pdf URL: https://arxiv.org/pdf/2407.03129
• Copy Paste: [[2407.03129]] Social Bias Evaluation for Large Language Models Requires Prompt Variations(https://arxiv.org/abs/2407.03129)
• Keywords: language model, llm, prompt
• Abstract: Warning: This paper contains examples of stereotypes and biases. Large Language Models (LLMs) exhibit considerable social biases, and various studies have tried to evaluate and mitigate these biases accurately. Previous studies use downstream tasks as prompts to examine the degree of social biases for evaluation and mitigation. While LLMs' output highly depends on prompts, previous studies evaluating and mitigating bias have often relied on a limited variety of prompts. In this paper, we investigate the sensitivity of LLMs when changing prompt variations (task instruction and prompt, few-shot examples, debias-prompt) by analyzing task performance and social bias of LLMs. Our experimental results reveal that LLMs are highly sensitive to prompts to the extent that the ranking of LLMs fluctuates when comparing models for task performance and social bias. Additionally, we show that LLMs have tradeoffs between performance and social bias caused by the prompts. Less bias from prompt setting may result in reduced performance. Moreover, the ambiguity of instances is one of the reasons for this sensitivity to prompts in advanced LLMs, leading to various outputs. We recommend using diverse prompts, as in this study, to compare the effects of prompts on social bias in LLMs.
• 摘要：警告：本文包含刻板印象和偏见的例子。大型语言模型 (LLM) 表现出相当大的社会偏见，各种研究都试图准确评估和减轻这些偏见。先前的研究使用下游任务作为提示来检查社会偏见的程度，以便进行评估和缓解。虽然 LLM 的输出高度依赖于提示，但之前评估和缓解偏见的研究往往依赖于有限的提示种类。在本文中，我们通过分析 LLM 的任务绩效和社会偏见，研究了 LLM 在改变提示变化（任务指令和提示、少量示例、去偏见提示）时的敏感性。我们的实验结果表明，LLM 对提示非常敏感，以至于在比较任务绩效和社会偏见的模型时，LLM 的排名会发生波动。此外，我们表明 LLM 在性能和提示造成的社会偏见之间存在权衡。提示设置的偏差较小可能会导致性能下降。此外，实例的模糊性是高级法学硕士对提示敏感的原因之一，从而导致不同的输出。我们建议使用不同的提示，如本研究所示，来比较提示对法学硕士社会偏见的影响。

Title: Enhancing Translation Accuracy of Large Language Models through Continual Pre-Training on Parallel Data

• Authors: Minato Kondo, Takehito Utsuro, Masaaki Nagata
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.03145
• Pdf URL: https://arxiv.org/pdf/2407.03145
• Copy Paste: [[2407.03145]] Enhancing Translation Accuracy of Large Language Models through Continual Pre-Training on Parallel Data(https://arxiv.org/abs/2407.03145)
• Keywords: language model, llm
• Abstract: In this paper, we propose a two-phase training approach where pre-trained large language models are continually pre-trained on parallel data and then supervised fine-tuned with a small amount of high-quality parallel data. To investigate the effectiveness of our proposed approach, we conducted continual pre-training with a 3.8B-parameter model and parallel data across eight different formats. We evaluate these methods on thirteen test sets for Japanese-to-English and English-to-Japanese translation. The results demonstrate that when utilizing parallel data in continual pre-training, it is essential to alternate between source and target sentences. Additionally, we demonstrated that the translation accuracy improves only for translation directions where the order of source and target sentences aligns between continual pre-training data and inference. In addition, we demonstrate that the LLM-based translation model is more robust in translating spoken language and achieves higher accuracy with less training data compared to supervised encoder-decoder models. We also show that the highest accuracy is achieved when the data for continual pre-training consists of interleaved source and target sentences and when tags are added to the source sentences.
• 摘要：在本文中，我们提出了一种两阶段训练方法，其中预先训练的大型语言模型在并行数据上持续进行预训练，然后使用少量高质量并行数据进行监督微调。为了研究我们提出的方法的有效性，我们使用 3.8B 参数模型和八种不同格式的并行数据进行了持续预训练。我们在十三个日语-英语和英语-日语翻译测试集上评估了这些方法。结果表明，在持续预训练中使用并行数据时，在源句子和目标句子之间交替是必不可少的。此外，我们证明了只有在持续预训练数据和推理之间源句子和目标句子的顺序一致的翻译方向上，翻译准确率才会提高。此外，我们证明了基于 LLM 的翻译模型在翻译口语方面更稳健，并且与监督编码器-解码器模型相比，以更少的训练数据实现了更高的准确率。我们还表明，当持续预训练的数据由交错的源句子和目标句子组成并且向源句子添加标签时，可以实现最高的准确率。

Title: Let the Code LLM Edit Itself When You Edit the Code

• Authors: Zhenyu He, Jun Zhang, Shengjie Luo, Jingjing Xu, Zhi Zhang, Di He
• Subjects: cs.CL, cs.AI, cs.LG, cs.SE
• Abstract URL: https://arxiv.org/abs/2407.03157
• Pdf URL: https://arxiv.org/pdf/2407.03157
• Copy Paste: [[2407.03157]] Let the Code LLM Edit Itself When You Edit the Code(https://arxiv.org/abs/2407.03157)
• Keywords: language model, llm
• Abstract: In this work, we investigate a typical scenario in code generation where a developer edits existing code in real time and requests a code assistant, e.g., a large language model, to re-predict the next token or next line on the fly. Naively, the LLM needs to re-encode the entire KV cache to provide an accurate prediction. However, this process is computationally expensive, especially when the sequence length is long. Simply encoding the edited subsequence and integrating it to the original KV cache meets the temporal confusion problem, leading to significantly worse performance. We address this efficiency and accuracy trade-off by introducing \underline{\textbf{Positional \textbf{I}ntegrity \textbf{E}ncoding} (PIE). Building upon the rotary positional encoding, PIE first removes the rotary matrices in the Key cache that introduce temporal confusion and then reapplies the correct rotary matrices. This process ensures that positional relationships between tokens are correct and requires only a single round of matrix multiplication. We validate the effectiveness of PIE through extensive experiments on the RepoBench-C-8k dataset, utilizing DeepSeek-Coder models with 1.3B, 6.7B, and 33B parameters. Our evaluation includes three real-world coding tasks: code insertion, code deletion, and multi-place code editing. Results demonstrate that PIE reduces computational overhead by over 85% compared to the standard full recomputation approach across all model sizes and tasks while well approximating the model performance.
• 摘要：在本研究中，我们研究了代码生成中的一个典型场景，即开发人员实时编辑现有代码并请求代码助手（例如大型语言模型）动态重新预测下一个标记或下一行。简单来说，LLM 需要重新编码整个 KV 缓存才能提供准确的预测。但是，这个过程在计算上是昂贵的，尤其是在序列长度很长的情况下。简单地对编辑后的子序列进行编码并将其集成到原始 KV 缓存中会遇到时间混淆问题，导致性能明显下降。我们通过引入 \underline{\textbf{Positional \textbf{I}integrity \textbf{E}ncoding} (PIE) 来解决这种效率和准确性之间的权衡。在旋转位置编码的基础上，PIE 首先删除 Key 缓存中引入时间混淆的旋转矩阵，然后重新应用正确的旋转矩阵。此过程可确保标记之间的位置关系正确，并且只需要一轮矩阵乘法。我们利用具有 1.3B、6.7B 和 33B 个参数的 DeepSeek-Coder 模型，在 RepoBench-C-8k 数据集上进行大量实验，验证了 PIE 的有效性。我们的评估包括三个实际编码任务：代码插入、代码删除和多位置代码编辑。结果表明，与标准完全重新计算方法相比，PIE 在所有模型大小和任务中将计算开销减少了 85% 以上，同时很好地逼近了模型性能。

Title: Investigating Decoder-only Large Language Models for Speech-to-text Translation

• Authors: Chao-Wei Huang, Hui Lu, Hongyu Gong, Hirofumi Inaguma, Ilia Kulikov, Ruslan Mavlyutov, Sravya Popuri
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2407.03169
• Pdf URL: https://arxiv.org/pdf/2407.03169
• Copy Paste: [[2407.03169]] Investigating Decoder-only Large Language Models for Speech-to-text Translation(https://arxiv.org/abs/2407.03169)
• Keywords: language model, llm
• Abstract: Large language models (LLMs), known for their exceptional reasoning capabilities, generalizability, and fluency across diverse domains, present a promising avenue for enhancing speech-related tasks. In this paper, we focus on integrating decoder-only LLMs to the task of speech-to-text translation (S2TT). We propose a decoder-only architecture that enables the LLM to directly consume the encoded speech representation and generate the text translation. Additionally, we investigate the effects of different parameter-efficient fine-tuning techniques and task formulation. Our model achieves state-of-the-art performance on CoVoST 2 and FLEURS among models trained without proprietary data. We also conduct analyses to validate the design choices of our proposed model and bring insights to the integration of LLMs to S2TT.
• 摘要：大型语言模型 (LLM) 以其出色的推理能力、通用性和跨不同领域的流畅性而闻名，为增强语音相关任务提供了一条有希望的途径。在本文中，我们专注于将仅解码器的 LLM 集成到语音到文本翻译 (S2TT) 任务中。我们提出了一种仅解码器的架构，使 LLM 能够直接使用编码的语音表示并生成文本翻译。此外，我们研究了不同的参数高效微调技术和任务制定的影响。在不使用专有数据训练的模型中，我们的模型在 CoVoST 2 和 FLEURS 上实现了最先进的性能。我们还进行了分析以验证我们提出的模型的设计选择，并为将 LLM 集成到 S2TT 带来见解。

Title: Fine-Tuning with Divergent Chains of Thought Boosts Reasoning Through Self-Correction in Language Models

• Authors: Haritz Puerto, Tilek Chubakov, Xiaodan Zhu, Harish Tayyar Madabushi, Iryna Gurevych
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.03181
• Pdf URL: https://arxiv.org/pdf/2407.03181
• Copy Paste: [[2407.03181]] Fine-Tuning with Divergent Chains of Thought Boosts Reasoning Through Self-Correction in Language Models(https://arxiv.org/abs/2407.03181)
• Keywords: language model, llm
• Abstract: Requiring a Large Language Model to generate intermediary reasoning steps has been shown to be an effective way of boosting performance. In fact, it has been found that instruction tuning on these intermediary reasoning steps improves model performance. In this work, we present a novel method of further improving performance by requiring models to compare multiple reasoning chains before generating a solution in a single inference step. We call this method Divergent CoT (DCoT). We find that instruction tuning on DCoT datasets boosts the performance of even smaller, and therefore more accessible, LLMs. Through a rigorous set of experiments spanning a wide range of tasks that require various reasoning types, we show that fine-tuning on DCoT consistently improves performance over the CoT baseline across model families and scales (1.3B to 70B). Through a combination of empirical and manual evaluation, we additionally show that these performance gains stem from models generating multiple divergent reasoning chains in a single inference step, indicative of the enabling of self-correction in language models. Our code and data are publicly available at this https URL.
• 摘要：要求大型语言模型生成中间推理步骤已被证明是提高性能的有效方法。事实上，人们发现对这些中间推理步骤进行指令调整可以提高模型性能。在这项工作中，我们提出了一种进一步提高性能的新方法，即要求模型在单个推理步骤中生成解决方案之前比较多个推理链。我们将这种方法称为发散 CoT (DCoT)。我们发现对 DCoT 数据集进行指令调整可以提高更小、因此更易于访问的 LLM 的性能。通过一系列严格的实验，这些实验涵盖了需要各种推理类型的广泛任务，我们表明，在 DCoT 上进行微调可以持续提高跨模型系列和规模（1.3B 到 70B）的 CoT 基线性能。通过结合经验和手动评估，我们还表明这些性能提升源于模型在单个推理步骤中生成多个发散推理链，这表明语言模型可以实现自我校正。我们的代码和数据可通过此 https URL 公开获取。

Title: How Does Quantization Affect Multilingual LLMs?

• Authors: Kelly Marchisio, Saurabh Dash, Hongyu Chen, Dennis Aumiller, Ahmet Üstün, Sara Hooker, Sebastian Ruder
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.03211
• Pdf URL: https://arxiv.org/pdf/2407.03211
• Copy Paste: [[2407.03211]] How Does Quantization Affect Multilingual LLMs?(https://arxiv.org/abs/2407.03211)
• Keywords: language model, llm, prompt
• Abstract: Quantization techniques are widely used to improve inference speed and deployment of large language models. While a wide body of work examines the impact of quantized LLMs on English tasks, none have examined the effect of quantization across languages. We conduct a thorough analysis of quantized multilingual LLMs, focusing on their performance across languages and at varying scales. We use automatic benchmarks, LLM-as-a-Judge methods, and human evaluation, finding that (1) harmful effects of quantization are apparent in human evaluation, and automatic metrics severely underestimate the detriment: a 1.7% average drop in Japanese across automatic tasks corresponds to a 16.0% drop reported by human evaluators on realistic prompts; (2) languages are disparately affected by quantization, with non-Latin script languages impacted worst; and (3) challenging tasks such as mathematical reasoning degrade fastest. As the ability to serve low-compute models is critical for wide global adoption of NLP technologies, our results urge consideration of multilingual performance as a key evaluation criterion for efficient models.
• 摘要：量化技术被广泛用于提高大型语言模型的推理速度和部署。虽然大量的研究都探讨了量化 LLM 对英语任务的影响，但还没有研究过量化在不同语言中的影响。我们对量化的多语言 LLM 进行了彻底的分析，重点关注它们在不同语言和不同规模下的表现。我们使用自动基准、LLM-as-a-Judge 方法和人工评估，发现 (1) 量化的有害影响在人工评估中显而易见，而自动指标严重低估了其危害：日语在自动任务中平均下降 1.7%，而人工评估员在实际提示中报告的下降幅度为 16.0%；(2) 语言受到量化的影响不同，非拉丁文字语言受到的影响最严重；(3) 数学推理等具有挑战性的任务退化最快。由于服务于低计算模型的能力对于 NLP 技术的全球广泛采用至关重要，我们的结果敦促将多语言性能作为高效模型的关键评估标准。

Title: Improving Retrieval-augmented Text-to-SQL with AST-based Ranking and Schema Pruning

• Authors: Zhili Shen, Pavlos Vougiouklis, Chenxin Diao, Kaustubh Vyas, Yuanyi Ji, Jeff Z. Pan
• Subjects: cs.CL, cs.AI, cs.DB
• Abstract URL: https://arxiv.org/abs/2407.03227
• Pdf URL: https://arxiv.org/pdf/2407.03227
• Copy Paste: [[2407.03227]] Improving Retrieval-augmented Text-to-SQL with AST-based Ranking and Schema Pruning(https://arxiv.org/abs/2407.03227)
• Keywords: language model, retrieval-augmented generation
• Abstract: We focus on Text-to-SQL semantic parsing from the perspective of Large Language Models. Motivated by challenges related to the size of commercial database schemata and the deployability of business intelligence solutions, we propose an approach that dynamically retrieves input database information and uses abstract syntax trees to select few-shot examples for in-context learning. Furthermore, we investigate the extent to which an in-parallel semantic parser can be leveraged for generating $\textit{approximated}$ versions of the expected SQL queries, to support our retrieval. We take this approach to the extreme--we adapt a model consisting of less than $500$M parameters, to act as an extremely efficient approximator, enhancing it with the ability to process schemata in a parallelised manner. We apply our approach to monolingual and cross-lingual benchmarks for semantic parsing, showing improvements over state-of-the-art baselines. Comprehensive experiments highlight the contribution of modules involved in this retrieval-augmented generation setting, revealing interesting directions for future work.
• 摘要：我们从大型语言模型的角度关注文本到 SQL 的语义解析。受商业数据库模式规模和商业智能解决方案可部署性相关挑战的启发，我们提出了一种方法，该方法动态检索输入数据库信息并使用抽象语法树选择少量样本进行上下文学习。此外，我们研究了并行语义解析器可用于生成预期 SQL 查询的近似版本以支持我们的检索的程度。我们将这种方法发挥到了极致——我们调整了一个由少于 5 亿个参数组成的模型，以充当极其高效的近似器，增强了它以并行方式处理模式的能力。我们将我们的方法应用于单语和跨语言语义解析基准，显示出比最先进的基线有所改进。综合实验突出了这种检索增强生成设置中涉及的模块的贡献，揭示了未来工作的有趣方向。

Title: CATT: Character-based Arabic Tashkeel Transformer

• Authors: Faris Alasmary, Orjuwan Zaafarani, Ahmad Ghannam
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.03236
• Pdf URL: https://arxiv.org/pdf/2407.03236
• Copy Paste: [[2407.03236]] CATT: Character-based Arabic Tashkeel Transformer(https://arxiv.org/abs/2407.03236)
• Keywords: gpt
• Abstract: Tashkeel, or Arabic Text Diacritization (ATD), greatly enhances the comprehension of Arabic text by removing ambiguity and minimizing the risk of misinterpretations caused by its absence. It plays a crucial role in improving Arabic text processing, particularly in applications such as text-to-speech and machine translation. This paper introduces a new approach to training ATD models. First, we finetuned two transformers, encoder-only and encoder-decoder, that were initialized from a pretrained character-based BERT. Then, we applied the Noisy-Student approach to boost the performance of the best model. We evaluated our models alongside 11 commercial and open-source models using two manually labeled benchmark datasets: WikiNews and our CATT dataset. Our findings show that our top model surpasses all evaluated models by relative Diacritic Error Rates (DERs) of 30.83\% and 35.21\% on WikiNews and CATT, respectively, achieving state-of-the-art in ATD. In addition, we show that our model outperforms GPT-4-turbo on CATT dataset by a relative DER of 9.36\%. We open-source our CATT models and benchmark dataset for the research community\footnote{this https URL}.
• 摘要：Tashkeel，即阿拉伯语文本变音符号 (ATD)，通过消除歧义并最大限度地降低由于缺少变音符号而导致误解的风险，极大地增强了对阿拉伯语文本的理解。它在改进阿拉伯语文本处理方面发挥着至关重要的作用，特别是在文本转语音和机器翻译等应用中。本文介绍了一种训练 ATD 模型的新方法。首先，我们对两个转换器（仅编码器和编码器解码器）进行了微调，它们由预训练的基于字符的 BERT 初始化。然后，我们应用了 Noisy-Student 方法来提升最佳模型的性能。我们使用两个手动标​​记的基准数据集（WikiNews 和我们的 CATT 数据集）对我们的模型以及 11 个商业和开源模型进行了评估。我们的研究结果表明，我们的顶级模型在 WikiNews 和 CATT 上的相对变音符号错误率 (DER) 分别为 30.83\% 和 35.21\%，超过了所有评估模型，在 ATD 中达到了最先进的水平。此外，我们表明我们的模型在 CATT 数据集上的表现比 GPT-4-turbo 好，相对 DER 为 9.36%。我们为研究社区开源了我们的 CATT 模型和基准数据集\footnote{此 https URL}。

Title: STF: Sentence Transformer Fine-Tuning For Topic Categorization With Limited Data

• Authors: Kheir Eddine Daouadi, Yaakoub Boualleg, Oussama Guehairia
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.03253
• Pdf URL: https://arxiv.org/pdf/2407.03253
• Copy Paste: [[2407.03253]] STF: Sentence Transformer Fine-Tuning For Topic Categorization With Limited Data(https://arxiv.org/abs/2407.03253)
• Keywords: language model
• Abstract: Nowadays, topic classification from tweets attracts considerable research attention. Different classification systems have been suggested thanks to these research efforts. Nevertheless, they face major challenges owing to low performance metrics due to the limited amount of labeled data. We propose Sentence Transformers Fine-tuning (STF), a topic detection system that leverages pretrained Sentence Transformers models and fine-tuning to classify topics from tweets accurately. Moreover, extensive parameter sensitivity analyses were conducted to finetune STF parameters for our topic classification task to achieve the best performance results. Experiments on two benchmark datasets demonstrated that (1) the proposed STF can be effectively used for classifying tweet topics and outperforms the latest state-of-the-art approaches, and (2) the proposed STF does not require a huge amount of labeled tweets to achieve good accuracy, which is a limitation of many state-of-the-art approaches. Our main contribution is the achievement of promising results in tweet topic classification by applying pretrained sentence transformers language models.
• 摘要：如今，从推文中进行主题分类引起了大量研究关注。由于这些研究工作，人们提出了不同的分类系统。然而，由于标记数据量有限，它们面临着性能指标低下的巨大挑战。我们提出了句子变换器微调 (STF)，这是一种主题检测系统，它利用预训练的句子变换器模型和微调来准确地从推文中对主题进行分类。此外，我们还进行了广泛的参数敏感性分析，以微调我们的主题分类任务的 STF 参数，以获得最佳性能结果。在两个基准数据集上的实验表明：(1) 所提出的 STF 可有效用于对推文主题进行分类，并且优于最新的最先进方法，(2) 所提出的 STF 不需要大量标记推文即可实现良好的准确性，这是许多最先进方法的局限性。我们的主要贡献是通过应用预训练的句子变换器语言模型在推文主题分类中取得了有希望的结果。

Title: LLM Internal States Reveal Hallucination Risk Faced With a Query

• Authors: Ziwei Ji, Delong Chen, Etsuko Ishii, Samuel Cahyawijaya, Yejin Bang, Bryan Wilie, Pascale Fung
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.03282
• Pdf URL: https://arxiv.org/pdf/2407.03282
• Copy Paste: [[2407.03282]] LLM Internal States Reveal Hallucination Risk Faced With a Query(https://arxiv.org/abs/2407.03282)
• Keywords: language model, llm, hallucination
• Abstract: The hallucination problem of Large Language Models (LLMs) significantly limits their reliability and trustworthiness. Humans have a self-awareness process that allows us to recognize what we don't know when faced with queries. Inspired by this, our paper investigates whether LLMs can estimate their own hallucination risk before response generation. We analyze the internal mechanisms of LLMs broadly both in terms of training data sources and across 15 diverse Natural Language Generation (NLG) tasks, spanning over 700 datasets. Our empirical analysis reveals two key insights: (1) LLM internal states indicate whether they have seen the query in training data or not; and (2) LLM internal states show they are likely to hallucinate or not regarding the query. Our study explores particular neurons, activation layers, and tokens that play a crucial role in the LLM perception of uncertainty and hallucination risk. By a probing estimator, we leverage LLM self-assessment, achieving an average hallucination estimation accuracy of 84.32\% at run time.
• 摘要：大型语言模型 (LLM) 的幻觉问题严重限制了其可靠性和可信度。人类具有自我意识过程，使我们能够在面对查询时认识到我们不知道的东西。受此启发，我们的论文研究了 LLM 是否可以在响应生成之前估计自己的幻觉风险。我们从训练数据源和 15 种不同的自然语言生成 (NLG) 任务的角度广泛分析了 LLM 的内部机制，涵盖了 700 多个数据集。我们的实证分析揭示了两个关键见解：(1) LLM 内部状态表明他们是否在训练数据中看到过查询；(2) LLM 内部状态表明他们是否可能对查询产生幻觉。我们的研究探索了在 LLM 对不确定性和幻觉风险的感知中起关键作用的特定神经元、激活层和标记。通过探测估计器，我们利用 LLM 自我评估，在运行时实现了 84.32% 的平均幻觉估计准确率。

Title: A Review of the Applications of Deep Learning-Based Emergent Communication

• Authors: Brendon Boldt, David Mortensen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.03302
• Pdf URL: https://arxiv.org/pdf/2407.03302
• Copy Paste: [[2407.03302]] A Review of the Applications of Deep Learning-Based Emergent Communication(https://arxiv.org/abs/2407.03302)
• Keywords: agent
• Abstract: Emergent communication, or emergent language, is the field of research which studies how human language-like communication systems emerge de novo in deep multi-agent reinforcement learning environments. The possibilities of replicating the emergence of a complex behavior like language have strong intuitive appeal, yet it is necessary to complement this with clear notions of how such research can be applicable to other fields of science, technology, and engineering. This paper comprehensively reviews the applications of emergent communication research across machine learning, natural language processing, linguistics, and cognitive science. Each application is illustrated with a description of its scope, an explication of emergent communication's unique role in addressing it, a summary of the extant literature working towards the application, and brief recommendations for near-term research directions.
• 摘要：新兴通信或新兴语言是研究人类语言类通信系统如何在深度多智能体强化学习环境中从头出现的研究领域。复制语言等复杂行为的出现具有很强的直观吸引力，但有必要用清晰的概念来补充这一点，即此类研究如何应用于其他科学、技术和工程领域。本文全面回顾了新兴通信研究在机器学习、自然语言处理、语言学和认知科学中的应用。每个应用都附有其范围的描述、新兴通信在解决该问题中的独特作用的说明、现有研究文献的摘要以及近期研究方向的简要建议。

Title: Planetarium: A Rigorous Benchmark for Translating Text to Structured Planning Languages

• Authors: Max Zuo, Francisco Piedrahita Velez, Xiaochen Li, Michael L. Littman, Stephen H. Bach
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.03321
• Pdf URL: https://arxiv.org/pdf/2407.03321
• Copy Paste: [[2407.03321]] Planetarium: A Rigorous Benchmark for Translating Text to Structured Planning Languages(https://arxiv.org/abs/2407.03321)
• Keywords: language model, gpt
• Abstract: Many recent works have explored using language models for planning problems. One line of research focuses on translating natural language descriptions of planning tasks into structured planning languages, such as the planning domain definition language (PDDL). While this approach is promising, accurately measuring the quality of generated PDDL code continues to pose significant challenges. First, generated PDDL code is typically evaluated using planning validators that check whether the problem can be solved with a planner. This method is insufficient because a language model might generate valid PDDL code that does not align with the natural language description of the task. Second, existing evaluation sets often have natural language descriptions of the planning task that closely resemble the ground truth PDDL, reducing the challenge of the task. To bridge this gap, we introduce \benchmarkName, a benchmark designed to evaluate language models' ability to generate PDDL code from natural language descriptions of planning tasks. We begin by creating a PDDL equivalence algorithm that rigorously evaluates the correctness of PDDL code generated by language models by flexibly comparing it against a ground truth PDDL. Then, we present a dataset of $132,037$ text-to-PDDL pairs across 13 different tasks, with varying levels of difficulty. Finally, we evaluate several API-access and open-weight language models that reveal this task's complexity. For example, $87.6\%$ of the PDDL problem descriptions generated by GPT-4o are syntactically parseable, $82.2\%$ are valid, solve-able problems, but only $35.1\%$ are semantically correct, highlighting the need for a more rigorous benchmark for this problem.
• 摘要：最近有许多研究探索使用语言模型来解决规划问题。其中一条研究路线专注于将规划任务的自然语言描述翻译成结构化规划语言，例如规划域定义语言 (PDDL)。虽然这种方法很有前景，但准确衡量生成的 PDDL 代码的质量仍然面临重大挑战。首先，生成的 PDDL 代码通常使用规划验证器进行评估，以检查问题是否可以用规划器解决。这种方法是不够的，因为语言模型可能会生成与任务的自然语言描述不一致的有效 PDDL 代码。其次，现有的评估集通常具有与基本事实 PDDL 非常相似的规划任务的自然语言描述，从而降低了任务的挑战性。为了弥补这一差距，我们引入了 \benchmarkName，这是一个基准，旨在评估语言模型从规划任务的自然语言描述生成 PDDL 代码的能力。我们首先创建一个 PDDL 等价算法，通过灵活地将其与基本事实 PDDL 进行比较，严格评估语言模型生成的 PDDL 代码的正确性。然后，我们展示了一个包含 13 个不同任务的 $132,037$ 个文本到 PDDL 对的数据集，这些任务的难度各不相同。最后，我们评估了几个 API 访问和开放权重语言模型，揭示了这项任务的复杂性。例如，GPT-4o 生成的 PDDL 问题描述中有 $87.6%$ 在语法上是可解析的，$82.2%$ 是有效的、可解决的问题，但只有 $35.1%$ 在语义上是正确的，这凸显了对这个问题进行更严格的基准测试的必要性。