2024-08-19

Title: Towards Realistic Synthetic User-Generated Content: A Scaffolding Approach to Generating Online Discussions

• Authors: Krisztian Balog, John Palowitch, Barbara Ikica, Filip Radlinski, Hamidreza Alvari, Mehdi Manshadi
• Subjects: cs.CL, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.08379
• Pdf URL: https://arxiv.org/pdf/2408.08379
• Copy Paste: [[2408.08379]] Towards Realistic Synthetic User-Generated Content: A Scaffolding Approach to Generating Online Discussions(https://arxiv.org/abs/2408.08379)
• Keywords: language model, llm, prompt
• Abstract: The emergence of synthetic data represents a pivotal shift in modern machine learning, offering a solution to satisfy the need for large volumes of data in domains where real data is scarce, highly private, or difficult to obtain. We investigate the feasibility of creating realistic, large-scale synthetic datasets of user-generated content, noting that such content is increasingly prevalent and a source of frequently sought information. Large language models (LLMs) offer a starting point for generating synthetic social media discussion threads, due to their ability to produce diverse responses that typify online interactions. However, as we demonstrate, straightforward application of LLMs yields limited success in capturing the complex structure of online discussions, and standard prompting mechanisms lack sufficient control. We therefore propose a multi-step generation process, predicated on the idea of creating compact representations of discussion threads, referred to as scaffolds. Our framework is generic yet adaptable to the unique characteristics of specific social media platforms. We demonstrate its feasibility using data from two distinct online discussion platforms. To address the fundamental challenge of ensuring the representativeness and realism of synthetic data, we propose a portfolio of evaluation measures to compare various instantiations of our framework.
• 摘要：合成数据的出现代表了现代机器学习的一个关键转变，它提供了一种解决方案，以满足在真实数据稀缺、高度私密或难以获得的领域对大量数据的需求。我们研究了创建真实的、大规模的用户生成内容合成数据集的可行性，并注意到此类内容越来越普遍，并且是人们经常寻求的信息来源。大型语言模型 (LLM) 为生成合成社交媒体讨论线程提供了一个起点，因为它们能够产生代表在线互动的多样化响应。然而，正如我们所展示的，直接应用 LLM 在捕捉在线讨论的复杂结构方面取得的成功有限，而标准的提示机制缺乏足够的控制。因此，我们提出了一个多步骤生成过程，该过程基于创建讨论线程的紧凑表示（称为支架）的想法。我们的框架是通用的，但可以适应特定社交媒体平台的独特特征。我们使用来自两个不同在线讨论平台的数据证明了它的可行性。为了解决确保合成数据的代表性和真实性这一根本挑战，我们提出了一系列评估措施来比较我们框架的各种实例。

Title: Zero-Shot Learning and Key Points Are All You Need for Automated Fact-Checking

• Authors: Mohammad Ghiasvand Mohammadkhani, Ali Ghiasvand Mohammadkhani, Hamid Beigy
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.08400
• Pdf URL: https://arxiv.org/pdf/2408.08400
• Copy Paste: [[2408.08400]] Zero-Shot Learning and Key Points Are All You Need for Automated Fact-Checking(https://arxiv.org/abs/2408.08400)
• Keywords: language model, llm, prompt
• Abstract: Automated fact-checking is an important task because determining the accurate status of a proposed claim within the vast amount of information available online is a critical challenge. This challenge requires robust evaluation to prevent the spread of false information. Modern large language models (LLMs) have demonstrated high capability in performing a diverse range of Natural Language Processing (NLP) tasks. By utilizing proper prompting strategies, their versatility due to their understanding of large context sizes and zero-shot learning ability enables them to simulate human problem-solving intuition and move towards being an alternative to humans for solving problems. In this work, we introduce a straightforward framework based on Zero-Shot Learning and Key Points (ZSL-KeP) for automated fact-checking, which despite its simplicity, performed well on the AVeriTeC shared task dataset by robustly improving the baseline and achieving 10th place.
• 摘要：自动事实核查是一项重要任务，因为在网上的大量信息中确定所提主张的准确状态是一项关键挑战。这一挑战需要进行可靠的评估，以防止虚假信息的传播。现代大型语言模型 (LLM) 已证明在执行各种自然语言处理 (NLP) 任务方面具有很高的能力。通过利用适当的提示策略，它们由于理解大型上下文规模和零样本学习能力而具有的多功能性使它们能够模拟人类解决问题的直觉，并逐渐成为人类解决问题的替代品。在这项工作中，我们引入了一个基于零样本学习和关键点 (ZSL-KeP) 的简单框架来进行自动事实核查，尽管它很简单，但在 AVeriTeC 共享任务数据集上表现良好，通过稳健地改进基线并取得第 10 名。

Title: W-RAG: Weakly Supervised Dense Retrieval in RAG for Open-domain Question Answering

• Authors: Jinming Nian, Zhiyuan Peng, Qifan Wang, Yi Fang
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.08444
• Pdf URL: https://arxiv.org/pdf/2408.08444
• Copy Paste: [[2408.08444]] W-RAG: Weakly Supervised Dense Retrieval in RAG for Open-domain Question Answering(https://arxiv.org/abs/2408.08444)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: In knowledge-intensive tasks such as open-domain question answering (OpenQA), Large Language Models (LLMs) often struggle to generate factual answers relying solely on their internal (parametric) knowledge. To address this limitation, Retrieval-Augmented Generation (RAG) systems enhance LLMs by retrieving relevant information from external sources, thereby positioning the retriever as a pivotal component. Although dense retrieval demonstrates state-of-the-art performance, its training poses challenges due to the scarcity of ground-truth evidence, largely attributed to the high costs of human annotation. In this paper, we propose W-RAG by utilizing the ranking capabilities of LLMs to create weakly labeled data for training dense retrievers. Specifically, we rerank the top-$K$ passages retrieved via BM25 by assessing the probability that LLMs will generate the correct answer based on the question and each passage. The highest-ranking passages are then used as positive training examples for dense retrieval. Our comprehensive experiments across four publicly available OpenQA datasets demonstrate that our approach enhances both retrieval and OpenQA performance compared to baseline models.
• 摘要：在开放域问答 (OpenQA) 等知识密集型任务中，大型语言模型 (LLM) 通常难以仅依靠其内部（参数）知识来生成事实答案。为了解决这一限制，检索增强生成 (RAG) 系统通过从外部来源检索相关信息来增强 LLM，从而将检索器定位为关键组件。尽管密集检索表现出最先进的性能，但由于缺乏基本事实证据（这主要归因于人工注释的高成本），其训练面临挑战。在本文中，我们提出了 W-RAG，利用 LLM 的排名功能来创建弱标记数据以训练密集检索器。具体而言，我们通过评估 LLM 根据问题和每个段落生成正确答案的概率，对通过 BM25 检索到的前 $K$ 个段落进行重新排序。然后将排名最高的段落用作密集检索的正面训练示例。我们对四个公开的 OpenQA 数据集进行的全面实验表明，与基线模型相比，我们的方法提高了检索和 OpenQA 性能。

Title: JPEG-LM: LLMs as Image Generators with Canonical Codec Representations

• Authors: Xiaochuang Han, Marjan Ghazvininejad, Pang Wei Koh, Yulia Tsvetkov
• Subjects: cs.CL, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.08459
• Pdf URL: https://arxiv.org/pdf/2408.08459
• Copy Paste: [[2408.08459]] JPEG-LM: LLMs as Image Generators with Canonical Codec Representations(https://arxiv.org/abs/2408.08459)
• Keywords: llm
• Abstract: Recent work in image and video generation has been adopting the autoregressive LLM architecture due to its generality and potentially easy integration into multi-modal systems. The crux of applying autoregressive training in language generation to visual generation is discretization -- representing continuous data like images and videos as discrete tokens. Common methods of discretizing images and videos include modeling raw pixel values, which are prohibitively lengthy, or vector quantization, which requires convoluted pre-hoc training. In this work, we propose to directly model images and videos as compressed files saved on computers via canonical codecs (e.g., JPEG, AVC/H.264). Using the default Llama architecture without any vision-specific modifications, we pretrain JPEG-LM from scratch to generate images (and AVC-LM to generate videos as a proof of concept), by directly outputting compressed file bytes in JPEG and AVC formats. Evaluation of image generation shows that this simple and straightforward approach is more effective than pixel-based modeling and sophisticated vector quantization baselines (on which our method yields a 31% reduction in FID). Our analysis shows that JPEG-LM has an especial advantage over vector quantization models in generating long-tail visual elements. Overall, we show that using canonical codec representations can help lower the barriers between language generation and visual generation, facilitating future research on multi-modal language/image/video LLMs.
• 摘要：图像和视频生成领域的最新研究一直采用自回归 LLM 架构，因为它具有通用性，并且可能易于集成到多模态系统中。将语言生成中的自回归训练应用于视觉生成的关键是离散化——将图像和视频等连续数据表示为离散标记。离散化图像和视频的常用方法包括对原始像素值进行建模（这些值非常长）或矢量量化（这需要复杂的预处理训练）。在这项工作中，我们建议直接将图像和视频建模为通过规范编解码器（例如 JPEG、AVC/H.264）保存在计算机上的压缩文件。使用默认的 Llama 架构（不进行任何视觉特定修改），我们从头开始预训练 JPEG-LM 以生成图像（并使用 AVC-LM 生成视频作为概念证明），通过直接输出 JPEG 和 AVC 格式的压缩文件字节。对图像生成的评估表明，这种简单直接的方法比基于像素的建模和复杂的矢量量化基线更有效（我们的方法使 FID 降低了 31%）。我们的分析表明，JPEG-LM 在生成长尾视觉元素方面比矢量量化模型具有特殊优势。总的来说，我们表明使用规范的编解码器表示可以帮助降低语言生成和视觉生成之间的障碍，促进未来对多模态语言/图像/视频 LLM 的研究。

Title: Ex3: Automatic Novel Writing by Extracting, Excelsior and Expanding

• Authors: Huang Lei, Jiaming Guo, Guanhua He, Xishan Zhang, Rui Zhang, Shaohui Peng, Shaoli Liu, Tianshi Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.08506
• Pdf URL: https://arxiv.org/pdf/2408.08506
• Copy Paste: [[2408.08506]] Ex3: Automatic Novel Writing by Extracting, Excelsior and Expanding(https://arxiv.org/abs/2408.08506)
• Keywords: language model, llm
• Abstract: Generating long-term texts such as novels using artificial intelligence has always been a challenge. A common approach is to use large language models (LLMs) to construct a hierarchical framework that first plans and then writes. Despite the fact that the generated novels reach a sufficient length, they exhibit poor logical coherence and appeal in their plots and deficiencies in character and event depiction, ultimately compromising the overall narrative quality. In this paper, we propose a method named Extracting Excelsior and Expanding. Ex3 initially extracts structure information from raw novel data. By combining this structure information with the novel data, an instruction-following dataset is meticulously crafted. This dataset is then utilized to fine-tune the LLM, aiming for excelsior generation performance. In the final stage, a tree-like expansion method is deployed to facilitate the generation of arbitrarily long novels. Evaluation against previous methods showcases Ex3's ability to produce higher-quality long-form novels.
• 摘要：利用人工智能生成小说等长篇文本一直是一个难题，常用的方法是利用大型语言模型（LLM）构建层级结构，先规划，再写作。生成的小说虽然篇幅足够长，但情节连贯性、感染力较差，人物和事件刻画不足，导致整体叙事质量下降。本文提出一种名为“提取精益求精和扩展”的方法。Ex3 首先从原始小说数据中提取结构信息，结合结构信息和小说数据，精心制作出一个指令跟踪数据集，然后利用该数据集对 LLM 进行微调，力求达到精益求精的生成性能。最后采用树状扩展方法，实现任意长度的小说生成。与之前的方法相比，Ex3 能够生成更高质量的长篇小说。

Title: CommunityKG-RAG: Leveraging Community Structures in Knowledge Graphs for Advanced Retrieval-Augmented Generation in Fact-Checking

• Authors: Rong-Ching Chang, Jiawei Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.08535
• Pdf URL: https://arxiv.org/pdf/2408.08535
• Copy Paste: [[2408.08535]] CommunityKG-RAG: Leveraging Community Structures in Knowledge Graphs for Advanced Retrieval-Augmented Generation in Fact-Checking(https://arxiv.org/abs/2408.08535)
• Keywords: language model, llm, retrieval augmented generation, retrieval-augmented generation
• Abstract: Despite advancements in Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) systems, their effectiveness is often hindered by a lack of integration with entity relationships and community structures, limiting their ability to provide contextually rich and accurate information retrieval for fact-checking. We introduce CommunityKG-RAG (Community Knowledge Graph-Retrieval Augmented Generation), a novel zero-shot framework that integrates community structures within Knowledge Graphs (KGs) with RAG systems to enhance the fact-checking process. Capable of adapting to new domains and queries without additional training, CommunityKG-RAG utilizes the multi-hop nature of community structures within KGs to significantly improve the accuracy and relevance of information retrieval. Our experimental results demonstrate that CommunityKG-RAG outperforms traditional methods, representing a significant advancement in fact-checking by offering a robust, scalable, and efficient solution.
• 摘要：尽管大型语言模型 (LLM) 和检索增强生成 (RAG) 系统取得了进步，但由于缺乏与实体关系和社区结构的整合，它们的有效性往往受到阻碍，限制了它们为事实核查提供丰富上下文和准确信息检索的能力。我们推出了 CommunityKG-RAG（社区知识图谱检索增强生成），这是一个新颖的零样本框架，它将知识图谱 (KG) 中的社区结构与 RAG 系统相结合，以增强事实核查过程。CommunityKG-RAG 能够适应新领域和查询而无需额外训练，它利用 KG 内社区结构的多跳特性来显著提高信息检索的准确性和相关性。我们的实验结果表明，CommunityKG-RAG 优于传统方法，通过提供健壮、可扩展和高效的解决方案，代表了事实核查的重大进步。

Title: Where is the signal in tokenization space?

• Authors: Renato Lui Geh, Honghua Zhang, Kareem Ahmed, Benjie Wang, Guy Van den Broeck
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.08541
• Pdf URL: https://arxiv.org/pdf/2408.08541
• Copy Paste: [[2408.08541]] Where is the signal in tokenization space?(https://arxiv.org/abs/2408.08541)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) are typically shipped with tokenizers that deterministically encode text into so-called canonical token sequences, to which the LLMs assign probability values. One common assumption is that the probability of a piece of text is the probability of its canonical token sequence. However, the tokenization of a string is not unique: e.g., the Llama2 tokenizer encodes Tokens as [Tok,ens], but [Tok,en,s] also represents the same text. In this paper, we study non-canonical tokenizations. We prove that, given a string, it is computationally hard to find the most likely tokenization for an autoregressive LLM, as well as to compute the marginal probability over all possible tokenizations. We then show how the marginal is, in most cases, indistinguishable from the canonical probability. Surprisingly, we then empirically demonstrate the existence of a significant amount of signal hidden within tokenization space. Notably, by simply aggregating the probabilities of non-canonical tokenizations, we achieve improvements across a range of LLM evaluation benchmarks for a variety of architectures, including transformers and state space models.
• 摘要：大型语言模型 (LLM) 通常附带标记器，标记器将文本确定性地编码为所谓的规范标记序列，LLM 为这些序列分配概率值。一个常见的假设是，一段文本的概率是其规范标记序列的概率。但是，字符串的标记化并不是唯一的：例如，Llama2 标记器将标记编码为 [Tok,ens]，但 [Tok,en,s] 也代表相同的文本。在本文中，我们研究非规范标记化。我们证明，给定一个字符串，从计算上很难找到自回归 LLM 最可能的标记化，也很难计算所有可能标记化的边际概率。然后，我们展示了在大多数情况下，边际概率与规范概率是无法区分的。令人惊讶的是，我们随后通过经验证明了标记化空间中隐藏着大量信号。值得注意的是，通过简单地汇总非规范标记化的概率，我们实现了针对各种架构（包括变压器和状态空间模型）的一系列 LLM 评估基准的改进。

Title: SelectLLM: Query-Aware Efficient Selection Algorithm for Large Language Models

• Authors: Kaushal Kumar Maurya, KV Aditya Srivatsa, Ekaterina Kochmar
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.08545
• Pdf URL: https://arxiv.org/pdf/2408.08545
• Copy Paste: [[2408.08545]] SelectLLM: Query-Aware Efficient Selection Algorithm for Large Language Models(https://arxiv.org/abs/2408.08545)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have gained increased popularity due to their remarkable success across various tasks, which has led to the active development of a large set of diverse LLMs. However, individual LLMs have limitations when applied to complex tasks because of such factors as training biases, model sizes, and the datasets used. A promising approach is to efficiently harness the diverse capabilities of LLMs to overcome these individual limitations. Towards this goal, we introduce a novel LLM selection algorithm called SelectLLM. This algorithm directs input queries to the most suitable subset of LLMs from a large pool, ensuring they collectively provide the correct response efficiently. SelectLLM uses a multi-label classifier, utilizing the classifier's predictions and confidence scores to design optimal policies for selecting an optimal, query-aware, and lightweight subset of LLMs. Our findings show that the proposed model outperforms individual LLMs and achieves competitive performance compared to similarly sized, computationally expensive top-performing LLM subsets. Specifically, with a similarly sized top-performing LLM subset, we achieve a significant reduction in latency on two standard reasoning benchmarks: 13% lower latency for GSM8K and 70% lower latency for MMLU. Additionally, we conduct comprehensive analyses and ablation studies, which validate the robustness of the proposed model.
• 摘要：大型语言模型 (LLM) 因其在各种任务中取得的显著成功而越来越受欢迎，这导致了大量不同 LLM 的积极开发。然而，由于训练偏差、模型大小和使用的数据集等因素，单个 LLM 在应用于复杂任务时存在局限性。一种有前途的方法是有效地利用 LLM 的多样化功能来克服这些个体限制。为了实现这一目标，我们引入了一种名为 SelectLLM 的新型 LLM 选择算法。该算法将输入查询从大型池中引导到最合适的 LLM 子集，确保它们共同有效地提供正确的响应。SelectLLM 使用多标签分类器，利用分类器的预测和置信度分数来设计最佳策略，以选择最佳、查询感知和轻量级的 LLM 子集。我们的研究结果表明，所提出的模型优于单个 LLM，并且与类似大小、计算成本高昂的顶级 LLM 子集相比具有竞争力。具体来说，使用同样大小的最佳 LLM 子集，我们在两个标准推理基准上实现了显著的延迟降低：GSM8K 的延迟降低了 13%，MMLU 的延迟降低了 70%。此外，我们还进行了全面的分析和消融研究，验证了所提模型的稳健性。

Title: Overview of the BioLaySumm 2024 Shared Task on the Lay Summarization of Biomedical Research Articles

• Authors: Tomas Goldsack, Carolina Scarton, Matthew Shardlow, Chenghua Lin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.08566
• Pdf URL: https://arxiv.org/pdf/2408.08566
• Copy Paste: [[2408.08566]] Overview of the BioLaySumm 2024 Shared Task on the Lay Summarization of Biomedical Research Articles(https://arxiv.org/abs/2408.08566)
• Keywords: language model, llm
• Abstract: This paper presents the setup and results of the second edition of the BioLaySumm shared task on the Lay Summarisation of Biomedical Research Articles, hosted at the BioNLP Workshop at ACL 2024. In this task edition, we aim to build on the first edition's success by further increasing research interest in this important task and encouraging participants to explore novel approaches that will help advance the state-of-the-art. Encouragingly, we found research interest in the task to be high, with this edition of the task attracting a total of 53 participating teams, a significant increase in engagement from the previous edition. Overall, our results show that a broad range of innovative approaches were adopted by task participants, with a predictable shift towards the use of Large Language Models (LLMs).
• 摘要：本文介绍了第二版 BioLaySumm 共享任务的设置和结果，该任务是关于生物医学研究文章的平庸总结，该任务在 ACL 2024 的 BioNLP 研讨会上举办。在这一版任务中，我们旨在在第一版成功的基础上进一步提高对这一重要任务的研究兴趣，并鼓励参与者探索有助于推动最先进技术的新方法。令人鼓舞的是，我们发现人们对这项任务的研究兴趣很高，这一版任务吸引了总共 53 个参与团队，参与度比上一版显著提高。总体而言，我们的结果表明，任务参与者采用了广泛的创新方法，并且可预见地转向使用大型语言模型 (LLM)。

Title: A Mechanistic Interpretation of Syllogistic Reasoning in Auto-Regressive Language Models

• Authors: Geonhee Kim, Marco Valentino, André Freitas
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.08590
• Pdf URL: https://arxiv.org/pdf/2408.08590
• Copy Paste: [[2408.08590]] A Mechanistic Interpretation of Syllogistic Reasoning in Auto-Regressive Language Models(https://arxiv.org/abs/2408.08590)
• Keywords: language model
• Abstract: Recent studies on logical reasoning in auto-regressive Language Models (LMs) have sparked a debate on whether such models can learn systematic reasoning principles during pre-training or merely exploit superficial patterns in the training data. This paper presents a mechanistic interpretation of syllogistic reasoning in LMs to further enhance our understanding of internal dynamics. Specifically, we present a methodology for circuit discovery aimed at disentangling content-independent reasoning mechanisms from world knowledge acquired during pre-training. Through two distinct intervention methods, we uncover a sufficient and necessary circuit involving middle-term suppression that elucidates how LMs transfer information to derive valid conclusions from premises. Furthermore, we investigate how belief biases manifest in syllogistic reasoning, finding evidence of partial contamination from additional attention heads responsible for encoding commonsense and contextualized knowledge. Finally, we explore the generalization of the discovered mechanisms across various syllogistic schemes and model sizes, finding that the identified circuit is sufficient and necessary for all the schemes on which the model achieves high downstream accuracy ($\geq$ 60\%). Overall, our findings suggest that LMs indeed learn transferable content-independent reasoning mechanisms, but that, at the same time, such mechanisms do not involve generalisable and abstract logical primitives, being susceptible to contamination by the same world knowledge acquired during pre-training.
• 摘要：最近关于自回归语言模型 (LM) 中的逻辑推理的研究引发了一场争论，即此类模型是否可以在预训练期间学习系统推理原理或仅仅利用训练数据中的表面模式。本文介绍了 LM 中三段论推理的机械解释，以进一步增强我们对内部动态的理解。具体来说，我们提出了一种回路发现方法，旨在将内容独立的推理机制与预训练期间获得的世界知识区分开来。通过两种不同的干预方法，我们发现了一个涉及中期抑制的充分必要回路，阐明了 LM 如何传递信息以从前提中得出有效结论。此外，我们研究了信念偏见如何在三段论推理中表现出来，发现了部分污染的证据，这些污染来自负责编码常识和情境知识的额外注意力头。最后，我们探索了所发现的机制在各种三段论方案和模型大小中的泛化，发现所识别的电路对于模型实现高下游准确度（$\geq$ 60\%）的所有方案都是充分且必要的。总体而言，我们的研究结果表明，LM 确实学习了可转移的独立于内容的推理机制，但与此同时，这种机制不涉及可泛化和抽象的逻辑原语，容易受到在预训练期间获得的相同世界知识的污染。

Title: RealMedQA: A pilot biomedical question answering dataset containing realistic clinical questions

• Authors: Gregory Kell, Angus Roberts, Serge Umansky, Yuti Khare, Najma Ahmed, Nikhil Patel, Chloe Simela, Jack Coumbe, Julian Rozario, Ryan-Rhys Griffiths, Iain J. Marshall
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.08624
• Pdf URL: https://arxiv.org/pdf/2408.08624
• Copy Paste: [[2408.08624]] RealMedQA: A pilot biomedical question answering dataset containing realistic clinical questions(https://arxiv.org/abs/2408.08624)
• Keywords: llm
• Abstract: Clinical question answering systems have the potential to provide clinicians with relevant and timely answers to their questions. Nonetheless, despite the advances that have been made, adoption of these systems in clinical settings has been slow. One issue is a lack of question-answering datasets which reflect the real-world needs of health professionals. In this work, we present RealMedQA, a dataset of realistic clinical questions generated by humans and an LLM. We describe the process for generating and verifying the QA pairs and assess several QA models on BioASQ and RealMedQA to assess the relative difficulty of matching answers to questions. We show that the LLM is more cost-efficient for generating "ideal" QA pairs. Additionally, we achieve a lower lexical similarity between questions and answers than BioASQ which provides an additional challenge to the top two QA models, as per the results. We release our code and our dataset publicly to encourage further research.
• 摘要：临床问答系统有潜力为临床医生提供相关且及时的答案。尽管取得了进展，但这些系统在临床环境中的采用却进展缓慢。一个问题是缺乏反映医疗专业人员现实需求的问答数据集。在这项工作中，我们介绍了 RealMedQA，这是一个由人类和 LLM 生成的真实临床问题的数据集。我们描述了生成和验证 QA 对的过程，并评估了 BioASQ 和 RealMedQA 上的几个 QA 模型，以评估将答案与问题匹配的相对难度。我们表明，LLM 在生成“理想”QA 对方面更具成本效益。此外，根据结果，我们在问题和答案之间实现了比 BioASQ 更低的词汇相似度，这给排名前两位的 QA 模型带来了额外的挑战。我们公开发布了我们的代码和数据集，以鼓励进一步研究。

Title: Persona is a Double-edged Sword: Enhancing the Zero-shot Reasoning by Ensembling the Role-playing and Neutral Prompts

• Authors: Junseok Kim, Nakyeong Yang, Kyomin Jung
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.08631
• Pdf URL: https://arxiv.org/pdf/2408.08631
• Copy Paste: [[2408.08631]] Persona is a Double-edged Sword: Enhancing the Zero-shot Reasoning by Ensembling the Role-playing and Neutral Prompts(https://arxiv.org/abs/2408.08631)
• Keywords: gpt, llm, prompt
• Abstract: Recent studies demonstrate that prompting an appropriate role-playing persona to an LLM improves its reasoning capability. However, assigning a proper persona is difficult since an LLM's performance is extremely sensitive to assigned prompts; therefore, personas sometimes hinder LLMs and degrade their reasoning capabilities. In this paper, we propose a novel framework, Jekyll \& Hyde, which ensembles the results of role-playing and neutral prompts to eradicate performance degradation via unilateral use of role-playing prompted LLM and enhance the robustness of an LLM's reasoning ability. Specifically, Jekyll \& Hyde collects two potential solutions from both role-playing and neutral prompts and selects a better solution after cross-checking via an LLM evaluator. However, LLM-based evaluators tend to be affected by the order of those potential solutions within the prompt when selecting the proper solution; thus, we also propose a robust LLM evaluator to mitigate the position bias. The experimental analysis demonstrates that role-playing prompts distract LLMs and degrade their reasoning abilities in 4 out of 12 datasets, even when using GPT-4. In addition, we reveal that Jekyll \& Hyde improves reasoning capabilities by selecting better choices among the potential solutions on twelve widely-used reasoning datasets. We further show that our proposed LLM evaluator outperforms other baselines, proving the LLMs' position bias is successfully mitigated.
• 摘要：最近的研究表明，为 LLM 提示合适的角色扮演角色可以提高其推理能力。然而，分配合适的角色很困难，因为 LLM 的表现对分配的提示极为敏感；因此，角色有时会妨碍 LLM 并降低其推理能力。在本文中，我们提出了一个新颖的框架 Jekyll \& Hyde，它集成了角色扮演和中性提示的结果，以消除因单方面使用角色扮演提示的 LLM 而导致的性能下降并增强 LLM 推理能力的稳健性。具体来说，Jekyll \& Hyde 从角色扮演和中性提示中收集两个潜在解决方案，并通过 LLM 评估器进行交叉检查后选择一个更好的解决方案。然而，基于 LLM 的评估者在选择合适的解决方案时往往会受到提示中这些潜在解决方案的顺序的影响；因此，我们还提出了一个强大的 LLM 评估器来减轻位置偏差。实验分析表明，即使使用 GPT-4，角色扮演提示也会分散 LLM 的注意力，并在 12 个数据集中的 4 个数据集中降低其推理能力。此外，我们发现 Jekyll \& Hyde 通过在 12 个广泛使用的推理数据集上的潜在解决方案中选择更好的选择来提高推理能力。我们进一步表明，我们提出的 LLM 评估器优于其他基线，证明 LLM 的位置偏差得到成功缓解。

Title: A Survey on Benchmarks of Multimodal Large Language Models

• Authors: Jian Li, Weiheng Lu
• Subjects: cs.CL, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2408.08632
• Pdf URL: https://arxiv.org/pdf/2408.08632
• Copy Paste: [[2408.08632]] A Survey on Benchmarks of Multimodal Large Language Models(https://arxiv.org/abs/2408.08632)
• Keywords: language model, llm
• Abstract: Multimodal Large Language Models (MLLMs) are gaining increasing popularity in both academia and industry due to their remarkable performance in various applications such as visual question answering, visual perception, understanding, and reasoning. Over the past few years, significant efforts have been made to examine MLLMs from multiple perspectives. This paper presents a comprehensive review of \textbf{180 benchmarks} and evaluation for MLLMs, focusing on (1)perception and understanding, (2)cognition and reasoning, (3)specific domains, (4)key capabilities, and (5)other modalities. Finally, we discuss the limitations of the current evaluation methods for MLLMs and explore promising future directions. Our key argument is that evaluation should be regarded as a crucial discipline to better support the development of MLLMs. For more details, please visit our GitHub repository: this https URL.
• 摘要：多模态大型语言模型 (MLLM) 在视觉问答、视觉感知、理解和推理等各种应用中表现出色，因此在学术界和工业界越来越受欢迎。过去几年，人们付出了巨大的努力，从多个角度研究 MLLM。本文全面回顾了 \textbf{180 基准} 和 MLLM 评估，重点关注 (1) 感知和理解、(2) 认知和推理、(3) 特定领域、(4) 关键能力和 (5) 其他模态。最后，我们讨论了当前 MLLM 评估方法的局限性，并探索了有希望的未来方向。我们的主要论点是，评估应该被视为一门关键学科，以更好地支持 MLLM 的发展。有关更多详细信息，请访问我们的 GitHub 存储库：此 https URL。

Title: Math-PUMA: Progressive Upward Multimodal Alignment to Enhance Mathematical Reasoning

• Authors: Wenwen Zhuang, Xin Huang, Xiantao Zhang, Jin Zeng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.08640
• Pdf URL: https://arxiv.org/pdf/2408.08640
• Copy Paste: [[2408.08640]] Math-PUMA: Progressive Upward Multimodal Alignment to Enhance Mathematical Reasoning(https://arxiv.org/abs/2408.08640)
• Keywords: language model, llm
• Abstract: Multimodal Large Language Models (MLLMs) excel in solving text-based mathematical problems, but they struggle with mathematical diagrams since they are primarily trained on natural scene images. For humans, visual aids generally enhance problem-solving, but MLLMs perform worse as information shifts from textual to visual modality. This decline is mainly due to their shortcomings in aligning images and text. To tackle aforementioned challenges, we propose Math-PUMA, a methodology focused on Progressive Upward Multimodal Alignment. This approach is designed to improve the mathematical reasoning skills of MLLMs through a three-stage training process, with the second stage being the critical alignment stage. We first enhance the language model's mathematical reasoning capabilities with extensive set of textual mathematical problems. We then construct a multimodal dataset with varying degrees of textual and visual information, creating data pairs by presenting each problem in at least two forms. By leveraging the Kullback-Leibler (KL) divergence of next-token prediction distributions to align visual and textual modalities, consistent problem-solving abilities are ensured. Finally, we utilize multimodal instruction tuning for MLLMs with high-quality multimodal data. Experimental results on multiple mathematical reasoning benchmarks demonstrate that the MLLMs trained with Math-PUMA surpass most open-source MLLMs. Our approach effectively narrows the performance gap for problems presented in different modalities.
• 摘要：多模态大型语言模型 (MLLM) 擅长解决基于文本的数学问题，但它们在处理数学图表时会遇到困难，因为它们主要在自然场景图像上进行训练。对于人类来说，视觉辅助通常可以增强解决问题的能力，但随着信息从文本转变为视觉模态，MLLM 的表现会变差。这种下降主要是因为它们在对齐图像和文本方面存在缺陷。为了应对上述挑战，我们提出了 Math-PUMA，这是一种专注于渐进式向上多模态对齐的方法。该方法旨在通过三阶段训练过程提高 MLLM 的数学推理能力，其中第二阶段是关键的对齐阶段。我们首先通过大量文本数学问题增强语言模型的数学推理能力。然后，我们构建一个具有不同程度文本和视觉信息的多模态数据集，通过以至少两种形式呈现每个问题来创建数据对。通过利用下一个标记预测分布的 Kullback-Leibler (KL) 散度来对齐视觉和文本模态，可以确保一致的问题解决能力。最后，我们利用高质量多模态数据的 MLLM 进行多模态指令调整。在多个数学推理基准上的实验结果表明，使用 Math-PUMA 训练的 MLLM 超越了大多数开源 MLLM。我们的方法有效地缩小了不同模态中出现的问题的性能差距。

Title: An End-to-End Model for Photo-Sharing Multi-modal Dialogue Generation

• Authors: Peiming Guo, Sinuo Liu, Yanzhao Zhang, Dingkun Long, Pengjun Xie, Meishan Zhang, Min Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.08650
• Pdf URL: https://arxiv.org/pdf/2408.08650
• Copy Paste: [[2408.08650]] An End-to-End Model for Photo-Sharing Multi-modal Dialogue Generation(https://arxiv.org/abs/2408.08650)
• Keywords: language model, chat, agent
• Abstract: Photo-Sharing Multi-modal dialogue generation requires a dialogue agent not only to generate text responses but also to share photos at the proper moment. Using image text caption as the bridge, a pipeline model integrates an image caption model, a text generation model, and an image generation model to handle this complex multi-modal task. However, representing the images with text captions may loss important visual details and information and cause error propagation in the complex dialogue system. Besides, the pipeline model isolates the three models separately because discrete image text captions hinder end-to-end gradient propagation. We propose the first end-to-end model for photo-sharing multi-modal dialogue generation, which integrates an image perceptron and an image generator with a large language model. The large language model employs the Q-Former to perceive visual images in the input end. For image generation in the output end, we propose a dynamic vocabulary transformation matrix and use straight-through and gumbel-softmax techniques to align the large language model and stable diffusion model and achieve end-to-end gradient propagation. We perform experiments on PhotoChat and DialogCC datasets to evaluate our end-to-end model. Compared with pipeline models, the end-to-end model gains state-of-the-art performances on various metrics of text and image generation. More analysis experiments also verify the effectiveness of the end-to-end model for photo-sharing multi-modal dialogue generation.
• 摘要：照片分享多模态对话生成要求对话代理不仅生成文本响应，还要在适当的时候分享照片。使用图像文本标题作为桥梁，管道模型集成了图像标题模型、文本生成模型和图像生成模型来处理这个复杂的多模态任务。然而，用文本标题表示图像可能会丢失重要的视觉细节和信息，并导致复杂对话系统中的错误传播。此外，管道模型将三个模型分开隔离，因为离散的图像文本标题阻碍了端到端梯度传播。我们提出了第一个用于照片分享多模态对话生成的端到端模型，它将图像感知器和图像生成器与大型语言模型集成在一起。大型语言模型在输入端使用 Q-Former 来感知视觉图像。对于输出端的图像生成，我们提出了一个动态词汇转换矩阵，并使用直通和 gumbel-softmax 技术来对齐大型语言模型和稳定扩散模型并实现端到端梯度传播。我们在 PhotoChat 和 DialogCC 数据集上进行了实验，以评估我们的端到端模型。与管道模型相比，端到端模型在文本和图像生成的各种指标上都获得了最佳性能。更多的分析实验也验证了端到端模型对于照片共享多模态对话生成的有效性。

Title: Reasoning Beyond Bias: A Study on Counterfactual Prompting and Chain of Thought Reasoning

• Authors: Kyle Moore, Jesse Roberts, Thao Pham, Douglas Fisher
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.08651
• Pdf URL: https://arxiv.org/pdf/2408.08651
• Copy Paste: [[2408.08651]] Reasoning Beyond Bias: A Study on Counterfactual Prompting and Chain of Thought Reasoning(https://arxiv.org/abs/2408.08651)
• Keywords: language model, prompt
• Abstract: Language models are known to absorb biases from their training data, leading to predictions driven by statistical regularities rather than semantic relevance. We investigate the impact of these biases on answer choice preferences in the Massive Multi-Task Language Understanding (MMLU) task. Our findings reveal that differences in learned regularities across answer options are predictive of model preferences and mirror human test-taking strategies. To address this issue, we introduce two novel methods: Counterfactual Prompting with Chain of Thought (CoT) and Counterfactual Prompting with Agnostically Primed CoT (APriCoT). We demonstrate that while Counterfactual Prompting with CoT alone is insufficient to mitigate bias, our novel Primed Counterfactual Prompting with CoT approach effectively reduces the influence of base-rate probabilities while improving overall accuracy. Our results suggest that mitigating bias requires a "System-2" like process and that CoT reasoning is susceptible to confirmation bias under some prompting methodologies. Our contributions offer practical solutions for developing more robust and fair language models.
• 摘要：众所周知，语言模型会吸收训练数据中的偏差，从而导致预测由统计规律而非语义相关性驱动。我们研究了这些偏差对大规模多任务语言理解 (MMLU) 任务中的答案选择偏好的影响。我们的研究结果表明，不同答案选项中习得规律的差异可以预测模型偏好，并反映人类的应试策略。为了解决这个问题，我们引入了两种新方法：使用思维链 (CoT) 的反事实提示和使用不可知启动 CoT (APriCoT) 的反事实提示。我们证明，虽然仅使用 CoT 的反事实提示不足以减轻偏差，但我们新颖的带有 CoT 的启动反事实提示方法可以有效降低基准概率的影响，同时提高整体准确性。我们的结果表明，减轻偏见需要类似“System-2”的过程，并且在某些提示方法下，CoT 推理容易受到确认偏差的影响。我们的贡献为开发更强大、更公平的语言模型提供了实用的解决方案。

Title: LLMs Are Biased Towards Output Formats! Systematically Evaluating and Mitigating Output Format Bias of LLMs

• Authors: Do Xuan Long, Hai Nguyen Ngoc, Tiviatis Sim, Hieu Dao, Shafiq Joty, Kenji Kawaguchi, Nancy F. Chen, Min-Yen Kan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.08656
• Pdf URL: https://arxiv.org/pdf/2408.08656
• Copy Paste: [[2408.08656]] LLMs Are Biased Towards Output Formats! Systematically Evaluating and Mitigating Output Format Bias of LLMs(https://arxiv.org/abs/2408.08656)
• Keywords: language model, gpt, llm, prompt, chat
• Abstract: We present the first systematic evaluation examining format bias in performance of large language models (LLMs). Our approach distinguishes between two categories of an evaluation metric under format constraints to reliably and accurately assess performance: one measures performance when format constraints are adhered to, while the other evaluates performance regardless of constraint adherence. We then define a metric for measuring the format bias of LLMs and establish effective strategies to reduce it. Subsequently, we present our empirical format bias evaluation spanning four commonly used categories -- multiple-choice question-answer, wrapping, list, and mapping -- covering 15 widely-used formats. Our evaluation on eight generation tasks uncovers significant format bias across state-of-the-art LLMs. We further discover that improving the format-instruction following capabilities of LLMs across formats potentially reduces format bias. Based on our evaluation findings, we study prompting and fine-tuning with synthesized format data techniques to mitigate format bias. Our methods successfully reduce the variance in ChatGPT's performance among wrapping formats from 235.33 to 0.71 (%$^2$).
• 摘要：我们首次系统性地评估了大型语言模型 (LLM) 的性能中的格式偏差。我们的方法区分了格式约束下的两类评估指标，以可靠准确地评估性能：一类衡量遵守格式约束时的性能，而另一类则评估无论是否遵守约束时的性能。然后，我们定义了一个衡量 LLM 格式偏差的指标，并制定了有效的策略来减少它。随后，我们提出了我们的实证格式偏差评估，涵盖了四个常用类别——多项选择问答、包装、列表和映射——涵盖了 15 种广泛使用的格式。我们对八个生成任务的评估发现，最先进的 LLM 存在显著的格式偏差。我们进一步发现，提高不同格式的 LLM 的格式指令遵循能力可能会减少格式偏差。根据我们的评估结果，我们研究了使用合成格式数据技术进行提示和微调以减轻格式偏差。我们的方法成功地将 ChatGPT 在不同包装格式之间的性能差异从 235.33 降低到了 0.71 (%$^2$)。

Title: MIA-Tuner: Adapting Large Language Models as Pre-training Text Detector

• Authors: Wenjie Fu, Huandong Wang, Chen Gao, Guanghua Liu, Yong Li, Tao Jiang
• Subjects: cs.CL, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.08661
• Pdf URL: https://arxiv.org/pdf/2408.08661
• Copy Paste: [[2408.08661]] MIA-Tuner: Adapting Large Language Models as Pre-training Text Detector(https://arxiv.org/abs/2408.08661)
• Keywords: language model, llm
• Abstract: The increasing parameters and expansive dataset of large language models (LLMs) highlight the urgent demand for a technical solution to audit the underlying privacy risks and copyright issues associated with LLMs. Existing studies have partially addressed this need through an exploration of the pre-training data detection problem, which is an instance of a membership inference attack (MIA). This problem involves determining whether a given piece of text has been used during the pre-training phase of the target LLM. Although existing methods have designed various sophisticated MIA score functions to achieve considerable detection performance in pre-trained LLMs, how to achieve high-confidence detection and how to perform MIA on aligned LLMs remain challenging. In this paper, we propose MIA-Tuner, a novel instruction-based MIA method, which instructs LLMs themselves to serve as a more precise pre-training data detector internally, rather than design an external MIA score function. Furthermore, we design two instruction-based safeguards to respectively mitigate the privacy risks brought by the existing methods and MIA-Tuner. To comprehensively evaluate the most recent state-of-the-art LLMs, we collect a more up-to-date MIA benchmark dataset, named WIKIMIA-24, to replace the widely adopted benchmark WIKIMIA. We conduct extensive experiments across various aligned and unaligned LLMs over the two benchmark datasets. The results demonstrate that MIA-Tuner increases the AUC of MIAs from 0.7 to a significantly high level of 0.9.
• 摘要：大型语言模型 (LLM) 的参数不断增加，数据集不断扩大，这凸显了对技术解决方案的迫切需求，以审计与 LLM 相关的潜在隐私风险和版权问题。现有研究通过探索预训练数据检测问题（成员推理攻击 (MIA) 的一个实例）部分解决了这一需求。此问题涉及确定给定的文本片段是否已在目标 LLM 的预训练阶段使用过。尽管现有方法已经设计了各种复杂的 MIA 分数函数以在预训练的 LLM 中实现可观的检测性能，但如何实现高置信度检测以及如何在对齐的 LLM 上执行 MIA 仍然具有挑战性。在本文中，我们提出了一种基于指令的新型 MIA 方法 MIA-Tuner，它指示 LLM 本身在内部充当更精确的预训练数据检测器，而不是设计外部 MIA 分数函数。此外，我们设计了两种基于指令的保护措施，分别减轻现有方法和 MIA-Tuner 带来的隐私风险。为了全面评估最新的 LLM，我们收集了更为最新的 MIA 基准数据集，名为 WIKIMIA-24，以取代广泛采用的基准 WIKIMIA。我们在两个基准数据集上对各种对齐和不对齐的 LLM 进行了广泛的实验。结果表明，MIA-Tuner 将 MIA 的 AUC 从 0.7 提高到了显著较高的 0.9。

Title: The Fellowship of the LLMs: Multi-Agent Workflows for Synthetic Preference Optimization Dataset Generation

• Authors: Samee Arif, Sualeha Farid, Abdul Hameed Azeemi, Awais Athar, Agha Ali Raza
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.08688
• Pdf URL: https://arxiv.org/pdf/2408.08688
• Copy Paste: [[2408.08688]] The Fellowship of the LLMs: Multi-Agent Workflows for Synthetic Preference Optimization Dataset Generation(https://arxiv.org/abs/2408.08688)
• Keywords: language model, gpt, llm, prompt, agent
• Abstract: This paper presents and evaluates multi-agent workflows for synthetic Preference Optimization (PO) dataset generation. PO dataset generation requires two modules: (1) response evaluation, and (2) response generation. In the response evaluation module, the responses from Large Language Models (LLMs) are evaluated and ranked - a task typically carried out by human annotators that we automate using LLMs. We assess the response evaluation module in a 2 step process. In step 1, we assess LLMs as evaluators using three distinct prompting strategies. In step 2, we apply the winning prompting strategy to compare the performance of LLM-as-a-Judge, LLMs-as-a-Jury, and LLM Debate. In each step, we use inter-rater agreement using Cohen's Kappa between human annotators and LLMs. For the response generation module, we compare different configurations for the LLM Feedback Loop using the identified LLM evaluator configuration. We use the win rate (the fraction of times a generation framework is selected as the best by an LLM evaluator) to determine the best multi-agent configuration for generation. After identifying the best configurations for both modules, we use models from the GPT, Gemma, and Llama families to generate our PO datasets using the above pipeline. We generate two types of PO datasets, one to improve the generation capabilities of individual LLM and the other to improve the multi-agent workflow. Our evaluation shows that GPT-4o-as-a-Judge is more consistent across datasets when the candidate responses do not include responses from the GPT family. Additionally, we find that the LLM Feedback Loop, with Llama as the generator and Gemma as the reviewer, achieves a notable 71.8% and 73.8% win rate over single-agent Llama and Gemma, respectively.
• 摘要：本文介绍并评估了用于合成偏好优化 (PO) 数据集生成的多智能体工作流。PO 数据集生成需要两个模块：(1) 响应评估和 (2) 响应生成。在响应评估模块中，对大型语言模型 (LLM) 的响应进行评估和排名 - 这项任务通常由人工注释者执行，我们使用 LLM 将其自动化。我们通过两步流程评估响应评估模块。在步骤 1 中，我们使用三种不同的提示策略评估作为评估者的 LLM。在步骤 2 中，我们应用获胜的提示策略来比较 LLM 作为法官、LLM 作为陪审团和 LLM 辩论的表现。在每个步骤中，我们使用 Cohen's Kappa 在人工注释者和 LLM 之间进行评分者间一致性评估。对于响应生成模块，我们使用已识别的 LLM 评估器配置比较 LLM 反馈循环的不同配置。我们使用胜率（LLM 评估者将生成框架选为最佳框架的次数）来确定最佳的多智能体生成配置。在确定两个模块的最佳配置后，我们使用 GPT、Gemma 和 Llama 系列的模型通过上述流程生成 PO 数据集。我们生成两种类型的 PO 数据集，一种用于提高单个 LLM 的生成能力，另一种用于改进多智能体工作流程。我们的评估表明，当候选答案不包含来自 GPT 系列的答案时，GPT-4o-as-a-Judge 在数据集之间的一致性更高。此外，我们发现，以 Llama 为生成器、以 Gemma 为审阅者的 LLM 反馈循环分别比单智能体 Llama 和 Gemma 实现了 71.8% 和 73.8% 的显著胜率。

Title: Med-PMC: Medical Personalized Multi-modal Consultation with a Proactive Ask-First-Observe-Next Paradigm

• Authors: Hongcheng Liu, Yusheng Liao, Siqv Ou, Yuhao Wang, Heyang Liu, Yanfeng Wang, Yu Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.08693
• Pdf URL: https://arxiv.org/pdf/2408.08693
• Copy Paste: [[2408.08693]] Med-PMC: Medical Personalized Multi-modal Consultation with a Proactive Ask-First-Observe-Next Paradigm(https://arxiv.org/abs/2408.08693)
• Keywords: language model, llm
• Abstract: The application of the Multi-modal Large Language Models (MLLMs) in medical clinical scenarios remains underexplored. Previous benchmarks only focus on the capacity of the MLLMs in medical visual question-answering (VQA) or report generation and fail to assess the performance of the MLLMs on complex clinical multi-modal tasks. In this paper, we propose a novel Medical Personalized Multi-modal Consultation (Med-PMC) paradigm to evaluate the clinical capacity of the MLLMs. Med-PMC builds a simulated clinical environment where the MLLMs are required to interact with a patient simulator to complete the multi-modal information-gathering and decision-making task. Specifically, the patient simulator is decorated with personalized actors to simulate diverse patients in real scenarios. We conduct extensive experiments to access 12 types of MLLMs, providing a comprehensive view of the MLLMs' clinical performance. We found that current MLLMs fail to gather multimodal information and show potential bias in the decision-making task when consulted with the personalized patient simulators. Further analysis demonstrates the effectiveness of Med-PMC, showing the potential to guide the development of robust and reliable clinical MLLMs. Code and data are available at this https URL.
• 摘要：多模态大型语言模型 (MLLM) 在医学临床场景中的应用仍未得到充分探索。以前的基准仅关注 MLLM 在医学视觉问答 (VQA) 或报告生成方面的能力，未能评估 MLLM 在复杂的临床多模态任务中的表现。在本文中，我们提出了一种新颖的医疗个性化多模态咨询 (Med-PMC) 范式来评估 MLLM 的临床能力。Med-PMC 构建了一个模拟临床环境，其中 MLLM 需要与患者模拟器交互以完成多模态信息收集和决策任务。具体来说，患者模拟器配有个性化的参与者，以模拟真实场景中的不同患者。我们进行了广泛的实验以访问 12 种类型的 MLLM，从而全面了解 MLLM 的临床表现。我们发现，当前的 MLLM 无法收集多模态信息，并且在与个性化患者模拟器进行咨询时在决策任务中表现出潜在的偏见。进一步分析证明了 Med-PMC 的有效性，显示出指导开发稳健可靠的临床 MLLM 的潜力。代码和数据可在此 https URL 上获取。

Title: Quantifying the Effectiveness of Student Organization Activities using Natural Language Processing

• Authors: Lyberius Ennio F. Taruc, Arvin R. De La Cruz
• Subjects: cs.CL, cs.AI, cs.ET
• Abstract URL: https://arxiv.org/abs/2408.08694
• Pdf URL: https://arxiv.org/pdf/2408.08694
• Copy Paste: [[2408.08694]] Quantifying the Effectiveness of Student Organization Activities using Natural Language Processing(https://arxiv.org/abs/2408.08694)
• Keywords: language model, llm
• Abstract: Student extracurricular activities play an important role in enriching the students' educational experiences. With the increasing popularity of Machine Learning and Natural Language Processing, it becomes a logical step that incorporating ML-NLP in improving extracurricular activities is a potential focus of study in Artificial Intelligence (AI). This research study aims to develop a machine learning workflow that will quantify the effectiveness of student-organized activities based on student emotional responses using sentiment analysis. The study uses the Bidirectional Encoder Representations from Transformers (BERT) Large Language Model (LLM) called via the pysentimiento toolkit, as a Transformer pipeline in Hugging Face. A sample data set from Organization C, a Recognized Student Organization (RSO) of a higher educational institute in the Philippines, College X, was used to develop the workflow. The workflow consisted of data preprocessing, key feature selection, LLM feature processing, and score aggregation, resulting in an Event Score for each data set. The results show that the BERT LLM can also be used effectively in analyzing sentiment beyond product reviews and post comments. For the student affairs offices of educational institutions, this study can provide a practical example of how NLP can be applied to real-world scenarios, showcasing the potential impact of data-driven decision making.
• 摘要：学生的课外活动在丰富学生的教育体验方面发挥着重要作用。随着机器学习和自然语言处理越来越受欢迎，将 ML-NLP 纳入改善课外活动成为人工智能 (AI) 研究的潜在重点，这成为合乎逻辑的一步。本研究旨在开发一种机器学习工作流程，该工作流程将使用情绪分析根据学生的情绪反应量化学生组织的活动的有效性。该研究使用通过 pysentimiento 工具包调用的 Transformers (BERT) 大型语言模型 (LLM) 的双向编码器表示作为 Hugging Face 中的 Transformer 管道。使用来自组织 C（菲律宾高等教育机构 College X 的认可学生组织 (RSO)）的样本数据集来开发工作流程。工作流程包括数据预处理、关键特征选择、LLM 特征处理和分数聚合，从而为每个数据集生成事件分数。结果表明，BERT LLM 还可以有效地用于分析产品评论和帖子评论以外的情绪。对于教育机构的学生事务办公室来说，这项研究可以提供 NLP 如何应用于现实场景的实用示例，展示数据驱动决策的潜在影响。

Title: Turning Trash into Treasure: Accelerating Inference of Large Language Models with Token Recycling

• Authors: Xianzhen Luo, Yixuan Wang, Qingfu Zhu, Zhiming Zhang, Xuanyu Zhang, Qing Yang, Dongliang Xu, Wanxiang Che
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.08696
• Pdf URL: https://arxiv.org/pdf/2408.08696
• Copy Paste: [[2408.08696]] Turning Trash into Treasure: Accelerating Inference of Large Language Models with Token Recycling(https://arxiv.org/abs/2408.08696)
• Keywords: language model, llm
• Abstract: The rapid growth in the parameters of large language models (LLMs) has made inference latency a fundamental bottleneck, limiting broader application of LLMs. Speculative decoding represents a lossless approach to accelerate inference through a guess-and-verify paradigm, leveraging the parallel capabilities of modern hardware. Some speculative decoding methods rely on additional structures to guess draft tokens, such as small models or parameter-efficient architectures, which need extra training before use. Alternatively, retrieval-based train-free techniques build libraries from pre-existing corpora or by n-gram generation. However, they face challenges like large storage requirements, time-consuming retrieval, and limited adaptability. Observing that candidate tokens generated during the decoding process are likely to reoccur in future sequences, we propose Token Recycling. This approach stores candidate tokens in an adjacency matrix and employs a breadth-first search (BFS)-like algorithm on the matrix to construct a draft tree. The tree is then validated through tree attention. New candidate tokens from the decoding process are then used to update the matrix. Token Recycling requires \textless2MB of additional storage and achieves approximately 2x speedup across all sizes of LLMs. It significantly outperforms existing train-free methods by 30\% and even a training method by 25\%. It can be directly applied to any existing LLMs and tasks without the need for adaptation.
• 摘要：大型语言模型 (LLM) 参数的快速增长使推理延迟成为根本瓶颈，限制了 LLM 的更广泛应用。推测解码是一种无损方法，通过猜测和验证范式加速推理，利用现代硬件的并行功能。一些推测解码方法依赖于额外的结构来猜测草稿标记，例如小型模型或参数高效的架构，这些结构在使用前需要额外的训练。或者，基于检索的无训练技术从预先存在的语料库或通过 n-gram 生成构建库。然而，它们面临着存储需求大、检索耗时和适应性有限等挑战。观察到解码过程中生成的候选标记很可能在未来的序列中再次出现，我们提出了标记回收。这种方法将候选标记存储在邻接矩阵中，并在矩阵上使用广度优先搜索 (BFS) 类算法来构建草稿树。然后通过树注意来验证树。然后使用解码过程中的新候选标记来更新矩阵。标记回收需要 \textless2MB 的额外存储空间，并在所有大小的 LLM 上实现约 2 倍的加速。它的性能显著优于现有的无训练方法 30\%，甚至优于训练方法 25\%。它可以直接应用于任何现有的 LLM 和任务，无需进行调整。

Title: ChatZero:Zero-shot Cross-Lingual Dialogue Generation via Pseudo-Target Language

• Authors: Yongkang Liu, Feng Shi, Daling Wang, Yifei Zhang, Hinrich Schütze
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.08724
• Pdf URL: https://arxiv.org/pdf/2408.08724
• Copy Paste: [[2408.08724]] ChatZero:Zero-shot Cross-Lingual Dialogue Generation via Pseudo-Target Language(https://arxiv.org/abs/2408.08724)
• Keywords: language model, llm, chat
• Abstract: Although large language models(LLMs) show amazing capabilities, among various exciting applications discovered for LLMs fall short in other low-resource languages. Besides, most existing methods depend on large-scale dialogue corpora and thus building systems for dialogue generation in a zero-shot scenario remains a considerable challenge. To address this challenge, we propose a novel end-to-end zero-shot dialogue generation model ChatZero based on cross-lingual code-switching method. First, we construct code-switching language and pseudo-target language with placeholders. Then for cross-lingual semantic transfer, we employ unsupervised contrastive learning to minimize the semantics gap of the source language, code-switching language, and pseudo-target language that are mutually positive examples in the high dimensional semantic space. Experiments on the multilingual DailyDialog and DSTC7-AVSD datasets demonstrate that ChatZero can achieve more than 90\% of the original performance under the zero-shot case compared to supervised learning, and achieve state-of-the-art performance compared with other baselines.
• 摘要：尽管大型语言模型 (LLM) 展现出惊人的能力，但在 LLM 发现的各种令人兴奋的应用中，其他低资源语言却存在不足。此外，大多数现有方法依赖于大规模对话语料库，因此在零样本场景中构建对话生成系统仍然是一项艰巨的挑战。为了应对这一挑战，我们提出了一种基于跨语言代码转换方法的新型端到端零样本对话生成模型 ChatZero。首先，我们用占位符构建代码转换语言和伪目标语言。然后，对于跨语言语义转移，我们采用无监督对比学习来最小化高维语义空间中互为正例的源语言、代码转换语言和伪目标语言之间的语义差距。在多语言 DailyDialog 和 DSTC7-AVSD 数据集上的实验表明，与监督学习相比，ChatZero 在零样本情况下可以实现原始性能的 90% 以上，与其他基线相比，实现了最先进的性能。

Title: Lower Layer Matters: Alleviating Hallucination via Multi-Layer Fusion Contrastive Decoding with Truthfulness Refocused

• Authors: Dingwei Chen, Feiteng Fang, Shiwen Ni, Feng Liang, Ruifeng Xu, Min Yang, Chengming Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.08769
• Pdf URL: https://arxiv.org/pdf/2408.08769
• Copy Paste: [[2408.08769]] Lower Layer Matters: Alleviating Hallucination via Multi-Layer Fusion Contrastive Decoding with Truthfulness Refocused(https://arxiv.org/abs/2408.08769)
• Keywords: language model, llm, hallucination
• Abstract: Large Language Models (LLMs) have demonstrated exceptional performance across various natural language processing tasks, yet they occasionally tend to yield content that factually inaccurate or discordant with the expected output, a phenomenon empirically referred to as "hallucination". To tackle this issue, recent works have investigated contrastive decoding between the original model and an amateur model with induced hallucination, which has shown promising results. Nonetheless, this method may undermine the output distribution of the original LLM caused by its coarse contrast and simplistic subtraction operation, potentially leading to errors in certain cases. In this paper, we introduce a novel contrastive decoding framework termed LOL (LOwer Layer Matters). Our approach involves concatenating the contrastive decoding of both the final and lower layers between the original model and the amateur model, thereby achieving multi-layer fusion to aid in the mitigation of hallucination. Additionally, we incorporate a truthfulness refocused module that leverages contextual guidance to enhance factual encoding, further capturing truthfulness during contrastive decoding. Extensive experiments conducted on two publicly available datasets illustrate that our proposed LOL framework can substantially alleviate hallucination while surpassing existing baselines in most cases. Compared with the best baseline, we improve by average 4.5 points on all metrics of TruthfulQA. The source code is coming soon.
• 摘要：大型语言模型 (LLM) 在各种自然语言处理任务中都表现出色，但它们偶尔会产生与预期输出不符或不一致的内容，这种现象在经验上被称为“幻觉”。为了解决这个问题，最近的研究调查了原始模型和诱发幻觉的业余模型之间的对比解码，并取得了令人鼓舞的结果。尽管如此，这种方法可能会破坏原始 LLM 的输出分布，这是由于其粗略的对比和简单的减法运算造成的，在某些情况下可能会导致错误。在本文中，我们介绍了一种称为 LOL（LOwer Layer Matters）的新型对比解码框架。我们的方法涉及将原始模型和业余模型之间的最终层和较低层的对比解码连接起来，从而实现多层融合以帮助缓解幻觉。此外，我们还整合了一个重新聚焦真实性的模块，该模块利用上下文指导来增强事实编码，从而在对比解码过程中进一步捕捉真实性。在两个公开数据集上进行的大量实验表明，我们提出的 LOL 框架可以显著缓解幻觉，同时在大多数情况下超越现有基线。与最佳基线相比，我们在 TruthfulQA 的所有指标上平均提高了 4.5 分。源代码即将推出。

Title: DAC: Decomposed Automation Correction for Text-to-SQL

• Authors: Dingzirui Wang, Longxu Dou, Xuanliang Zhang, Qingfu Zhu, Wanxiang Che
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.08779
• Pdf URL: https://arxiv.org/pdf/2408.08779
• Copy Paste: [[2408.08779]] DAC: Decomposed Automation Correction for Text-to-SQL(https://arxiv.org/abs/2408.08779)
• Keywords: language model, llm
• Abstract: Text-to-SQL is an important task that helps people obtain information from databases by automatically generating SQL queries. Considering the brilliant performance, approaches based on Large Language Models (LLMs) become the mainstream for text-to-SQL. Among these approaches, automated correction is an effective approach that further enhances performance by correcting the mistakes in the generated results. The existing correction methods require LLMs to directly correct with generated SQL, while previous research shows that LLMs do not know how to detect mistakes, leading to poor performance. Therefore, in this paper, we propose to employ the decomposed correction to enhance text-to-SQL performance. We first demonstrate that decomposed correction outperforms direct correction since detecting and fixing mistakes with the results of the decomposed sub-tasks is easier than with SQL. Based on this analysis, we introduce Decomposed Automation Correction (DAC), which corrects SQL by decomposing text-to-SQL into entity linking and skeleton parsing. DAC first generates the entity and skeleton corresponding to the question and then compares the differences between the initial SQL and the generated entities and skeleton as feedback for correction. Experimental results show that our method improves performance by $3.7\%$ on average of Spider, Bird, and KaggleDBQA compared with the baseline method, demonstrating the effectiveness of DAC.
• 摘要：文本转 SQL 是一项重要的任务，它通过自动生成 SQL 查询来帮助人们从数据库中获取信息。鉴于其出色的性能，基于大型语言模型 (LLM) 的方法成为文本转 SQL 的主流。在这些方法中，自动校正是一种有效的方法，它通过纠正生成结果中的错误来进一步提高性能。现有的校正方法需要 LLM 直接使用生成的 SQL 进行校正，而先前的研究表明 LLM 不知道如何检测错误，导致性能不佳。因此，在本文中，我们提出采用分解校正来提高文本转 SQL 的性能。我们首先证明分解校正优于直接校正，因为使用分解子任务的结果检测和修复错误比使用 SQL 更容易。基于此分析，我们引入了分解自动化校正 (DAC)，它通过将文本转 SQL 分解为实体链接和骨架解析来校正 SQL。DAC 首先生成与问题相对应的实体和骨架，然后比较初始 SQL 与生成的实体和骨架之间的差异作为校正的反馈。实验结果表明，我们的方法与基线方法相比，在 Spider、Bird 和 KaggleDBQA 上平均提高了 $3.7\%$ 的性能，证明了 DAC 的有效性。

Title: Large Language Models Might Not Care What You Are Saying: Prompt Format Beats Descriptions

• Authors: Chenming Tang, Zhixiang Wang, Yunfang Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.08780
• Pdf URL: https://arxiv.org/pdf/2408.08780
• Copy Paste: [[2408.08780]] Large Language Models Might Not Care What You Are Saying: Prompt Format Beats Descriptions(https://arxiv.org/abs/2408.08780)
• Keywords: language model, llm, hallucination, prompt
• Abstract: With the help of in-context learning (ICL), large language models (LLMs) have achieved impressive performance across various tasks. However, the function of descriptive instructions during ICL remains under-explored. In this work, we propose an ensemble prompt framework to describe the selection criteria of multiple in-context examples, and preliminary experiments on machine translation (MT) across six translation directions confirm that this framework boosts ICL perfromance. But to our surprise, LLMs might not necessarily care what the descriptions actually say, and the performance gain is primarily caused by the ensemble format, since the framework could lead to improvement even with random descriptive nouns. We further apply this new ensemble prompt on a range of commonsense, math, logical reasoning and hallucination tasks with three LLMs and achieve promising results, suggesting again that designing a proper prompt format would be much more effective and efficient than paying effort into specific descriptions. Our code will be publicly available once this paper is published.
• 摘要：借助上下文学习 (ICL)，大型语言模型 (LLM) 在各种任务中取得了令人瞩目的表现。然而，描述性指令在 ICL 中的作用仍未得到充分探索。在这项工作中，我们提出了一个集成提示框架来描述多个上下文示例的选择标准，并且在六个翻译方向上的机器翻译 (MT) 上的初步实验证实该框架提高了 ICL 性能。但令我们惊讶的是，LLM 可能并不一定关心描述实际上说了什么，而性能提升主要是由集成格式引起的，因为即使使用随机描述性名词，该框架也可以带来改进。我们进一步将这个新的集成提示应用于三个 LLM 的一系列常识、数学、逻辑推理和幻觉任务，并取得了令人鼓舞的结果，这再次表明设计适当的提示格式比花精力进行具体描述更有效、更高效。本文发表后，我们的代码将公开提供。

Title: EmoDynamiX: Emotional Support Dialogue Strategy Prediction by Modelling MiXed Emotions and Discourse Dynamics

• Authors: Chenwei Wan, Matthieu Labeau, Chloé Clavel
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.08782
• Pdf URL: https://arxiv.org/pdf/2408.08782
• Copy Paste: [[2408.08782]] EmoDynamiX: Emotional Support Dialogue Strategy Prediction by Modelling MiXed Emotions and Discourse Dynamics(https://arxiv.org/abs/2408.08782)
• Keywords: language model, llm, agent
• Abstract: Designing emotionally intelligent conversational systems to provide comfort and advice to people experiencing distress is a compelling area of research. Previous efforts have focused on developing modular dialogue systems that treat socio-emotional strategy prediction as an auxiliary task and generate strategy-conditioned responses with customized decoders. Recently, with advancements in large language models (LLMs), end-to-end dialogue agents without explicit socio-emotional strategy prediction steps have become prevalent. However, despite their excellence in language generation, recent studies show that LLMs' inherent preference bias towards certain socio-emotional strategies hinders the delivery of high-quality emotional support. To address this challenge, we propose decoupling strategy prediction from language generation, and introduce a novel dialogue strategy predictor, EmoDynamiX, which models the discourse dynamics between user emotions and system strategies using a heterogeneous graph. Additionally, we make use of the Emotion Recognition in Conversations (ERC) task and design a flexible mixed-emotion module to capture fine-grained emotional states of the user. Experimental results on two ESC datasets show EmoDynamiX outperforms previous state-of-the-art methods with a significant margin.
• 摘要：设计情感智能对话系统，为陷入困境的人提供安慰和建议，是一个引人注目的研究领域。先前的研究重点是开发模块化对话系统，将社会情感策略预测作为辅助任务，并使用定制解码器生成策略条件响应。最近，随着大型语言模型 (LLM) 的进步，没有明确社会情感策略预测步骤的端到端对话代理变得普遍。然而，尽管 LLM 在语言生成方面表现出色，但最近的研究表明，LLM 对某些社会情感策略的固有偏好偏见阻碍了提供高质量的情感支持。为了应对这一挑战，我们建议将策略预测与语言生成分离，并引入一种新颖的对话策略预测器 EmoDynamiX，它使用异构图对用户情感和系统策略之间的话语动态进行建模。此外，我们利用对话中的情感识别 (ERC) 任务并设计一个灵活的混合情感模块来捕捉用户的细粒度情绪状态。在两个 ESC 数据集上的实验结果表明，EmoDynamiX 的表现显著优于之前最先进的方法。

Title: CIKMar: A Dual-Encoder Approach to Prompt-Based Reranking in Educational Dialogue Systems

• Authors: Joanito Agili Lopo, Marina Indah Prasasti, Alma Permatasari
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.08805
• Pdf URL: https://arxiv.org/pdf/2408.08805
• Copy Paste: [[2408.08805]] CIKMar: A Dual-Encoder Approach to Prompt-Based Reranking in Educational Dialogue Systems(https://arxiv.org/abs/2408.08805)
• Keywords: language model, prompt
• Abstract: In this study, we introduce CIKMar, an efficient approach to educational dialogue systems powered by the Gemma Language model. By leveraging a Dual-Encoder ranking system that incorporates both BERT and SBERT model, we have designed CIKMar to deliver highly relevant and accurate responses, even with the constraints of a smaller language model size. Our evaluation reveals that CIKMar achieves a robust recall and F1-score of 0.70 using BERTScore metrics. However, we have identified a significant challenge: the Dual-Encoder tends to prioritize theoretical responses over practical ones. These findings underscore the potential of compact and efficient models like Gemma in democratizing access to advanced educational AI systems, ensuring effective and contextually appropriate responses.
• 摘要：在本研究中，我们介绍了 CIKMar，这是一种由 Gemma 语言模型提供支持的高效教育对话系统。通过利用结合了 BERT 和 SBERT 模型的双编码器排名系统，我们设计了 CIKMar，即使在语言模型规模较小的情况下，也能提供高度相关且准确的响应。我们的评估表明，CIKMar 使用 BERTScore 指标实现了强大的召回率和 0.70 的 F1 分数。然而，我们发现了一个重大挑战：双编码器倾向于优先考虑理论响应而不是实际响应。这些发现强调了 Gemma 等紧凑高效模型在实现高级教育 AI 系统民主化访问方面的潜力，确保有效且符合情境的响应。

Title: FLEXTAF: Enhancing Table Reasoning with Flexible Tabular Formats

• Authors: Xuanliang Zhang, Dingzirui Wang, Longxu Dou, Baoxin Wang, Dayong Wu, Qingfu Zhu, Wanxiang Che
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.08841
• Pdf URL: https://arxiv.org/pdf/2408.08841
• Copy Paste: [[2408.08841]] FLEXTAF: Enhancing Table Reasoning with Flexible Tabular Formats(https://arxiv.org/abs/2408.08841)
• Keywords: language model, llm
• Abstract: The table reasoning task aims to answer the question according to the given table. Currently, using Large Language Models (LLMs) is the predominant method for table reasoning. Most existing methods employ a fixed tabular format to represent the table, which could limit the performance. Given that each instance requires different capabilities and models possess varying abilities, we assert that different instances and models suit different tabular formats. We prove the aforementioned claim through quantitative analysis of experimental results, where different instances and models achieve different performances using various tabular formats. Building on this discussion, we propose FLEXTAF-Single and FLEXTAF-Vote to enhance table reasoning performance by employing flexible tabular formats. Specifically, (i) FLEXTAF-Single trains a classifier to predict the most suitable tabular format based on the instance and the LLM. (ii) FLEXTAF-Vote integrates the results across different formats. Our experiments on WikiTableQuestions and TabFact reveal significant improvements, with average gains of 2.3% and 4.8% compared to the best performance achieved using a fixed tabular format with greedy decoding and self-consistency decoding, thereby validating the effectiveness of our methods.
• 摘要：表格推理任务旨在根据给定的表格回答问题。目前，使用大型语言模型 (LLM) 是表格推理的主要方法。大多数现有方法采用固定的表格格式来表示表格，这可能会限制性能。鉴于每个实例需要不同的功能并且模型具有不同的能力，我们断言不同的实例和模型适合不同的表格格式。我们通过对实验结果的定量分析证明了上述说法，其中不同的实例和模型使用各种表格格式实现不同的性能。在此讨论的基础上，我们提出了 FLEXTAF-Single 和 FLEXTAF-Vote，通过采用灵活的表格格式来提高表格推理性能。具体而言，(i) FLEXTAF-Single 训练分类器根据实例和 LLM 预测最合适的表格格式。(ii) FLEXTAF-Vote 整合不同格式的结果。我们在 WikiTableQuestions 和 TabFact 上的实验显示出显著的改进，与使用贪婪解码和自洽解码的固定表格格式所取得的最佳性能相比，平均提高了 2.3% 和 4.8%，从而验证了我们方法的有效性。

Title: PsychoLex: Unveiling the Psychological Mind of Large Language Models

• Authors: Mohammad Amin Abbasi, Farnaz Sadat Mirnezami, Hassan Naderi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.08848
• Pdf URL: https://arxiv.org/pdf/2408.08848
• Copy Paste: [[2408.08848]] PsychoLex: Unveiling the Psychological Mind of Large Language Models(https://arxiv.org/abs/2408.08848)
• Keywords: language model, llm
• Abstract: This paper explores the intersection of psychology and artificial intelligence through the development and evaluation of specialized Large Language Models (LLMs). We introduce PsychoLex, a suite of resources designed to enhance LLMs' proficiency in psychological tasks in both Persian and English. Key contributions include the PsychoLexQA dataset for instructional content and the PsychoLexEval dataset for rigorous evaluation of LLMs in complex psychological scenarios. Additionally, we present the PsychoLexLLaMA model, optimized specifically for psychological applications, demonstrating superior performance compared to general-purpose models. The findings underscore the potential of tailored LLMs for advancing psychological research and applications, while also highlighting areas for further refinement. This research offers a foundational step towards integrating LLMs into specialized psychological domains, with implications for future advancements in AI-driven psychological practice.
• 摘要：本文通过开发和评估专门的大型语言模型 (LLM) 来探索心理学与人工智能的交集。我们介绍了 PsychoLex，这是一套资源，旨在提高 LLM 在波斯语和英语心理任务中的熟练程度。主要贡献包括用于教学内容的 PsychoLexQA 数据集和用于在复杂心理场景中严格评估 LLM 的 PsychoLexEval 数据集。此外，我们还提出了专门针对心理学应用优化的 PsychoLexLLaMA 模型，与通用模型相比，该模型表现出卓越的性能。研究结果强调了定制 LLM 在推进心理学研究和应用方面的潜力，同时也强调了进一步改进的领域。这项研究为将 LLM 整合到专门的心理学领域提供了基础性的一步，对未来人工智能驱动的心理学实践的进步具有重要意义。

Title: PEDAL: Enhancing Greedy Decoding with Large Language Models using Diverse Exemplars

• Authors: Sumanth Prabhu
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.08869
• Pdf URL: https://arxiv.org/pdf/2408.08869
• Copy Paste: [[2408.08869]] PEDAL: Enhancing Greedy Decoding with Large Language Models using Diverse Exemplars(https://arxiv.org/abs/2408.08869)
• Keywords: language model, llm, prompt
• Abstract: Self-ensembling techniques with diverse reasoning paths such as Self-Consistency have demonstrated remarkable gains in accuracy for Large Language Models (LLMs). However, such techniques depend on the availability of an accurate answer extraction process to aggregate across multiple outputs. Moreover, they acquire higher inference cost, in comparison to Greedy Decoding, due to generation of relatively higher number of output tokens. Research has shown that the free form text outputs from Self-Consistency can be aggregated reliably using LLMs to produce the final output. Additionally, recent advancements in LLM inference have demonstrated that usage of diverse exemplars in prompts have the ability to induce diversity in the LLM outputs. Such proven techniques can be easily extended to self-ensembling based approaches to achieve enhanced results in text generation. In this paper, we introduce PEDAL (Prompts based on Exemplar Diversity Aggregated using LLMs), a hybrid self-ensembling approach, that combines the strengths of diverse exemplar based prompts and LLM based aggregation to achieve improvement in overall performance. On the publicly available SVAMP and ARC datasets, our experiments reveal that PEDAL can achieve better accuracy than Greedy Decoding based strategies with lower inference cost compared to Self Consistency based approaches.
• 摘要：具有多种推理路径的自组装技术（例如自一致性）已证明在大型语言模型 (LLM) 中具有显著的准确性提升。然而，这种技术依赖于准确的答案提取过程的可用性，以聚合多个输出。此外，与贪婪解码相比，它们由于生成的输出标记数量相对较多而具有更高的推理成本。研究表明，自一致性的自由格式文本输出可以使用 LLM 可靠地聚合以产生最终输出。此外，LLM 推理的最新进展表明，在提示中使用多样化的样本能够引起 LLM 输出的多样性。这种经过验证的技术可以轻松扩展到基于自组装的方法，以在文本生成中实现增强的结果。在本文中，我们介绍了 PEDAL（基于使用 LLM 聚合的样本多样性的提示），这是一种混合自组装方法，它结合了基于多样化样本的提示和基于 LLM 的聚合的优势，以实现整体性能的提高。在公开可用的 SVAMP 和 ARC 数据集上，我们的实验表明，与基于自一致性的方法相比，PEDAL 可以比基于贪婪解码的策略获得更好的准确性，并且推理成本更低。