2024-08-15

Title: ELLA: Empowering LLMs for Interpretable, Accurate and Informative Legal Advice

• Authors: Yutong Hu, Kangcheng Luo, Yansong Feng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.07137
• Pdf URL: https://arxiv.org/pdf/2408.07137
• Copy Paste: [[2408.07137]] ELLA: Empowering LLMs for Interpretable, Accurate and Informative Legal Advice(https://arxiv.org/abs/2408.07137)
• Keywords: language model, llm
• Abstract: Despite remarkable performance in legal consultation exhibited by legal Large Language Models(LLMs) combined with legal article retrieval components, there are still cases when the advice given is incorrect or baseless. To alleviate these problems, we propose {\bf ELLA}, a tool for {\bf E}mpowering {\bf L}LMs for interpretable, accurate, and informative {\bf L}egal {\bf A}dvice. ELLA visually presents the correlation between legal articles and LLM's response by calculating their similarities, providing users with an intuitive legal basis for the responses. Besides, based on the users' queries, ELLA retrieves relevant legal articles and displays them to users. Users can interactively select legal articles for LLM to generate more accurate responses. ELLA also retrieves relevant legal cases for user reference. Our user study shows that presenting the legal basis for the response helps users understand better. The accuracy of LLM's responses also improves when users intervene in selecting legal articles for LLM. Providing relevant legal cases also aids individuals in obtaining comprehensive information.
• 摘要：尽管结合法律文章检索组件的法律大型语言模型 (LLM) 在法律咨询方面表现出色，但仍存在给出的建议不正确或毫无根据的情况。为了缓解这些问题，我们提出了 {\bf ELLA}，这是一种增强 {\bf E}L}LM 能力的工具，可提供可解释、准确且信息丰富的{\bf 法律{\bf A} 建议。ELLA 通过计算法律文章和 LLM 的回复的相似度，以可视化方式呈现它们之间的相关性，为用户提供直观的回复法律依据。此外，ELLA 根据用户的查询检索相关的法律文章并将其显示给用户。用户可以交互地为 LLM 选择法律文章，以生成更准确的回复。ELLA 还会检索相关法律案例供用户参考。我们的用户研究表明，呈现回复的法律依据有助于用户更好地理解。当用户干预为 LLM 选择法律文章时，LLM 回复的准确性也会提高。提供相关的法律案例也有助于个人获取全面的信息。

Title: Language Models as Models of Language

• Authors: Raphaël Millière
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.07144
• Pdf URL: https://arxiv.org/pdf/2408.07144
• Copy Paste: [[2408.07144]] Language Models as Models of Language(https://arxiv.org/abs/2408.07144)
• Keywords: language model
• Abstract: This chapter critically examines the potential contributions of modern language models to theoretical linguistics. Despite their focus on engineering goals, these models' ability to acquire sophisticated linguistic knowledge from mere exposure to data warrants a careful reassessment of their relevance to linguistic theory. I review a growing body of empirical evidence suggesting that language models can learn hierarchical syntactic structure and exhibit sensitivity to various linguistic phenomena, even when trained on developmentally plausible amounts of data. While the competence/performance distinction has been invoked to dismiss the relevance of such models to linguistic theory, I argue that this assessment may be premature. By carefully controlling learning conditions and making use of causal intervention methods, experiments with language models can potentially constrain hypotheses about language acquisition and competence. I conclude that closer collaboration between theoretical linguists and computational researchers could yield valuable insights, particularly in advancing debates about linguistic nativism.
• 摘要：本章批判性地审视了现代语言模型对理论语言学的潜在贡献。尽管这些模型专注于工程目标，但它们仅通过接触数据就能获得复杂的语言知识的能力值得我们仔细重新评估它们与语言理论的相关性。我回顾了越来越多的经验证据，这些证据表明语言模型可以学习层次句法结构并对各种语言现象表现出敏感性，即使在使用发展合理的数据量进行训练时也是如此。虽然能力/表现的区别被用来否定这些模型与语言理论的相关性，但我认为这种评估可能为时过早。通过仔细控制学习条件并利用因果干预方法，语言模型实验可能会限制有关语言习得和能力的假设。我的结论是，理论语言学家和计算研究人员之间的更紧密合作可以产生有价值的见解，特别是在推进有关语言本土主义的辩论方面。

Title: Unlocking Efficiency: Adaptive Masking for Gene Transformer Models

• Authors: Soumyadeep Roy, Shamik Sural, Niloy Ganguly
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.07180
• Pdf URL: https://arxiv.org/pdf/2408.07180
• Copy Paste: [[2408.07180]] Unlocking Efficiency: Adaptive Masking for Gene Transformer Models(https://arxiv.org/abs/2408.07180)
• Keywords: language model
• Abstract: Gene transformer models such as Nucleotide Transformer, DNABert, and LOGO are trained to learn optimal gene sequence representations by using the Masked Language Modeling (MLM) training objective over the complete Human Reference Genome. However, the typical tokenization methods employ a basic sliding window of tokens, such as k-mers, that fail to utilize gene-centric semantics. This could result in the (trivial) masking of easily predictable sequences, leading to inefficient MLM training. Time-variant training strategies are known to improve pretraining efficiency in both language and vision tasks. In this work, we focus on using curriculum masking where we systematically increase the difficulty of masked token prediction task by using a Pointwise Mutual Information-based difficulty criterion, as gene sequences lack well-defined semantic units similar to words or sentences of NLP domain. Our proposed Curriculum Masking-based Gene Masking Strategy (CM-GEMS) demonstrates superior representation learning capabilities compared to baseline masking approaches when evaluated on downstream gene sequence classification tasks. We perform extensive evaluation in both few-shot (five datasets) and full dataset settings (Genomic Understanding Evaluation benchmark consisting of 27 tasks). Our findings reveal that CM-GEMS outperforms state-of-the-art models (DNABert-2, Nucleotide transformer, DNABert) trained at 120K steps, achieving similar results in just 10K and 1K steps. We also demonstrate that Curriculum-Learned LOGO (a 2-layer DNABert-like model) can achieve nearly 90% of the state-of-the-art model performance of 120K steps. We will make the models and codes publicly available at this https URL.
• 摘要：基因转换器模型（例如核苷酸转换器、DNABert 和 LOGO）通过使用针对完整人类参考基因组的掩蔽语言模型 (MLM) 训练目标，训练以学习最佳基因序列表示。然而，典型的标记化方法采用基本的标记滑动窗口，例如 k-mers，无法利用以基因为中心的语义。这可能会导致（不重要的）容易预测的序列被掩蔽，从而导致 MLM 训练效率低下。众所周知，时变训练策略可以提高语言和视觉任务的预训练效率。在这项工作中，我们专注于使用课程掩蔽，我们通过使用基于逐点互信息的难度标准系统地增加掩蔽标记预测任务的难度，因为基因序列缺乏类似于 NLP 领域的单词或句子的明确定义的语义单元。我们提出的基于课程掩蔽的基因掩蔽策略 (CM-GEMS) 在下游基因序列分类任务上评估时，与基线掩蔽方法相比表现出卓越的表征学习能力。我们在少样本（五个数据集）和完整数据集设置（由 27 个任务组成的基因组理解评估基准）中进行了广泛的评估。我们的研究结果表明，CM-GEMS 的表现优于以 120K 步训练的最先进的模型（DNABert-2、Nucleotide transformer、DNABert），仅在 10K 和 1K 步中就取得了类似的结果。我们还证明，Curriculum-Learned LOGO（一种 2 层 DNABert 类模型）可以实现 120K 步的最先进的模型性能的近 90%。我们将在此 https URL 上公开提供模型和代码。

Title: Neural embedding of beliefs reveals the role of relative dissonance in human decision-making

• Authors: Byunghwee Lee, Rachith Aiyappa, Yong-Yeol Ahn, Haewoon Kwak, Jisun An
• Subjects: cs.CL, cs.CY, physics.soc-ph
• Abstract URL: https://arxiv.org/abs/2408.07237
• Pdf URL: https://arxiv.org/pdf/2408.07237
• Copy Paste: [[2408.07237]] Neural embedding of beliefs reveals the role of relative dissonance in human decision-making(https://arxiv.org/abs/2408.07237)
• Keywords: language model, llm
• Abstract: Beliefs serve as the foundation for human cognition and decision-making. They guide individuals in deriving meaning from their lives, shaping their behaviors, and forming social connections. Therefore, a model that encapsulates beliefs and their interrelationships is crucial for quantitatively studying the influence of beliefs on our actions. Despite its importance, research on the interplay between human beliefs has often been limited to a small set of beliefs pertaining to specific issues, with a heavy reliance on surveys or experiments. Here, we propose a method for extracting nuanced relations between thousands of beliefs by leveraging large-scale user participation data from an online debate platform and mapping these beliefs to an embedding space using a fine-tuned large language model (LLM). This belief embedding space effectively encapsulates the interconnectedness of diverse beliefs as well as polarization across various social issues. We discover that the positions within this belief space predict new beliefs of individuals. Furthermore, we find that the relative distance between one's existing beliefs and new beliefs can serve as a quantitative estimate of cognitive dissonance, allowing us to predict new beliefs. Our study highlights how modern LLMs, when combined with collective online records of human beliefs, can offer insights into the fundamental principles that govern human belief formation and decision-making processes.
• 摘要：信念是人类认知和决策的基础。它们指导个人从生活中获取意义、塑造行为并形成社会联系。因此，一个能够概括信念及其相互关系的模型对于定量研究信念对我们行为的影响至关重要。尽管信念很重要，但对人类信念之间相互作用的研究往往局限于与特定问题有关的一小部分信念，并且严重依赖调查或实验。在这里，我们提出了一种方法，通过利用在线辩论平台的大规模用户参与数据并使用微调的大型语言模型 (LLM) 将这些信念映射到嵌入空间，来提取数千种信念之间的细微关系。这个信念嵌入空间有效地概括了不同信念的相互联系以及各种社会问题的两极分化。我们发现，这个信念空间中的位置可以预测个人的新信念。此外，我们发现，一个人现有信念与新信念之间的相对距离可以作为认知失调的定量估计，使我们能够预测新信念。我们的研究强调了现代法学硕士与人类信念的集体在线记录相结合如何能够洞察支配人类信念形成和决策过程的基本原则。

Title: Using Advanced LLMs to Enhance Smaller LLMs: An Interpretable Knowledge Distillation Approach

• Authors: Tong Wang, K. Sudhir, Dat Hong
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.07238
• Pdf URL: https://arxiv.org/pdf/2408.07238
• Copy Paste: [[2408.07238]] Using Advanced LLMs to Enhance Smaller LLMs: An Interpretable Knowledge Distillation Approach(https://arxiv.org/abs/2408.07238)
• Keywords: language model, gpt, llm, prompt, agent
• Abstract: Advanced Large language models (LLMs) like GPT-4 or LlaMa 3 provide superior performance in complex human-like interactions. But they are costly, or too large for edge devices such as smartphones and harder to self-host, leading to security and privacy concerns. This paper introduces a novel interpretable knowledge distillation approach to enhance the performance of smaller, more economical LLMs that firms can self-host. We study this problem in the context of building a customer service agent aimed at achieving high customer satisfaction through goal-oriented dialogues. Unlike traditional knowledge distillation, where the "student" model learns directly from the "teacher" model's responses via fine-tuning, our interpretable "strategy" teaching approach involves the teacher providing strategies to improve the student's performance in various scenarios. This method alternates between a "scenario generation" step and a "strategies for improvement" step, creating a customized library of scenarios and optimized strategies for automated prompting. The method requires only black-box access to both student and teacher models; hence it can be used without manipulating model parameters. In our customer service application, the method improves performance, and the learned strategies are transferable to other LLMs and scenarios beyond the training set. The method's interpretabilty helps safeguard against potential harms through human audit.
• 摘要：高级大型语言模型 (LLM)（如 GPT-4 或 LlaMa 3）在复杂的类人交互中提供了卓越的性能。但它们成本高昂，或者对于智能手机等边缘设备来说太大，而且难以自行托管，从而导致安全和隐私问题。本文介绍了一种新颖的可解释知识蒸馏方法，以提高公司可以自行托管的较小、更经济的 LLM 的性能。我们在构建客户服务代理的背景下研究了这个问题，旨在通过面向目标的对话实现高客户满意度。与传统的知识蒸馏不同，在传统的知识蒸馏中，“学生”模型通过微调直接从“老师”模型的响应中学习，而我们的可解释“策略”教学方法涉及老师提供策略来提高学生在各种场景中的表现。该方法在“场景生成”步骤和“改进策略”步骤之间交替进行，创建自定义场景库和优化的自动提示策略。该方法只需要对学生和老师模型进行黑盒访问；因此可以在不操纵模型参数的情况下使用它。在我们的客户服务应用中，该方法提高了性能，并且所学策略可以迁移到训练集之外的其他 LLM 和场景。该方法的可解释性有助于防止人工审计造成的潜在危害。

Title: Speech vs. Transcript: Does It Matter for Human Annotators in Speech Summarization?

• Authors: Roshan Sharma, Suwon Shon, Mark Lindsey, Hira Dhamyal, Rita Singh, Bhiksha Raj
• Subjects: cs.CL, cs.HC, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2408.07277
• Pdf URL: https://arxiv.org/pdf/2408.07277
• Copy Paste: [[2408.07277]] Speech vs. Transcript: Does It Matter for Human Annotators in Speech Summarization?(https://arxiv.org/abs/2408.07277)
• Keywords: llm
• Abstract: Reference summaries for abstractive speech summarization require human annotation, which can be performed by listening to an audio recording or by reading textual transcripts of the recording. In this paper, we examine whether summaries based on annotators listening to the recordings differ from those based on annotators reading transcripts. Using existing intrinsic evaluation based on human evaluation, automatic metrics, LLM-based evaluation, and a retrieval-based reference-free method. We find that summaries are indeed different based on the source modality, and that speech-based summaries are more factually consistent and information-selective than transcript-based summaries. Meanwhile, transcript-based summaries are impacted by recognition errors in the source, and expert-written summaries are more informative and reliable. We make all the collected data and analysis code public(this https URL) to facilitate the reproduction of our work and advance research in this area.
• 摘要：抽象语音摘要的参考摘要需要人工注释，这可以通过收听录音或阅读录音的文本记录来完成。在本文中，我们研究基于注释者听录音的摘要与基于注释者阅读记录的摘要是否不同。使用现有的基于人工评估、自动指标、基于 LLM 的评估和基于检索的无参考方法的内在评估。我们发现摘要确实因源模态而异，并且基于语音的摘要比基于记录的摘要更具事实一致性和信息选择性。同时，基于记录的摘要会受到源中识别错误的影响，而专家撰写的摘要更具信息量和可靠性。我们公开所有收集到的数据和分析代码（此 https URL），以方便复制我们的工作并推动该领域的研究。

Title: Do GPT Language Models Suffer From Split Personality Disorder? The Advent Of Substrate-Free Psychometrics

• Authors: Peter Romero, Stephen Fitz, Teruo Nakatsuma
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2408.07377
• Pdf URL: https://arxiv.org/pdf/2408.07377
• Copy Paste: [[2408.07377]] Do GPT Language Models Suffer From Split Personality Disorder? The Advent Of Substrate-Free Psychometrics(https://arxiv.org/abs/2408.07377)
• Keywords: language model, gpt
• Abstract: Previous research on emergence in large language models shows these display apparent human-like abilities and psychological latent traits. However, results are partly contradicting in expression and magnitude of these latent traits, yet agree on the worrisome tendencies to score high on the Dark Triad of narcissism, psychopathy, and Machiavellianism, which, together with a track record of derailments, demands more rigorous research on safety of these models. We provided a state of the art language model with the same personality questionnaire in nine languages, and performed Bayesian analysis of Gaussian Mixture Model, finding evidence for a deeper-rooted issue. Our results suggest both interlingual and intralingual instabilities, which indicate that current language models do not develop a consistent core personality. This can lead to unsafe behaviour of artificial intelligence systems that are based on these foundation models, and are increasingly integrated in human life. We subsequently discuss the shortcomings of modern psychometrics, abstract it, and provide a framework for its species-neutral, substrate-free formulation.
• 摘要：先前对大型语言模型出现的研究表明，这些模型表现出明显的类人能力和心理潜在特征。然而，这些结果在这些潜在特征的表达和程度上存在部分矛盾，但令人担忧的是，这些模型在自恋、精神病和马基雅维利主义的黑暗三角中得分较高，再加上脱轨的记录，需要对这些模型的安全性进行更严格的研究。我们提供了一个最先进的语言模型，该模型在九种语言中使用了相同的性格问卷，并对高斯混合模型进行了贝叶斯分析，发现了一个更深层次问题的证据。我们的结果表明，语言间和语言内的不稳定性，这表明当前的语言模型没有形成一致的核心人格。这可能导致基于这些基础模型的人工智能系统出现不安全行为，这些系统越来越多地融入人类生活。我们随后讨论了现代心理测量学的缺点，对其进行了抽象，并为其物种中性、无基质的公式提供了一个框架。

Title: DataVisT5: A Pre-trained Language Model for Jointly Understanding Text and Data Visualization

• Authors: Zhuoyue Wan, Yuanfeng Song, Shuaimin Li, Chen Jason Zhang, Raymond Chi-Wing Wong
• Subjects: cs.CL, cs.AI, cs.DB
• Abstract URL: https://arxiv.org/abs/2408.07401
• Pdf URL: https://arxiv.org/pdf/2408.07401
• Copy Paste: [[2408.07401]] DataVisT5: A Pre-trained Language Model for Jointly Understanding Text and Data Visualization(https://arxiv.org/abs/2408.07401)
• Keywords: language model
• Abstract: Data visualization (DV) is the fundamental and premise tool to improve the efficiency in conveying the insights behind the big data, which has been widely accepted in existing data-driven world. Task automation in DV, such as converting natural language queries to visualizations (i.e., text-to-vis), generating explanations from visualizations (i.e., vis-to-text), answering DV-related questions in free form (i.e. FeVisQA), and explicating tabular data (i.e., table-to-text), is vital for advancing the field. Despite their potential, the application of pre-trained language models (PLMs) like T5 and BERT in DV has been limited by high costs and challenges in handling cross-modal information, leading to few studies on PLMs for DV. We introduce \textbf{DataVisT5}, a novel PLM tailored for DV that enhances the T5 architecture through a hybrid objective pre-training and multi-task fine-tuning strategy, integrating text and DV datasets to effectively interpret cross-modal semantics. Extensive evaluations on public datasets show that DataVisT5 consistently outperforms current state-of-the-art models on various DV-related tasks. We anticipate that DataVisT5 will not only inspire further research on vertical PLMs but also expand the range of applications for PLMs.
• 摘要：数据可视化 (DV) 是提高大数据背后洞察传达效率的基本和前提工具，在现有的数据驱动世界中已被广泛接受。DV 中的任务自动化对于该领域的发展至关重要，例如将自然语言查询转换为可视化（即文本到可视化）、从可视化生成解释（即可视化到文本）、以自由形式回答与 DV 相关的问题（即 FeVisQA）以及解释表格数据（即表格到文本）。尽管预训练语言模型 (PLM)（如 T5 和 BERT）具有巨大潜力，但它们在 DV 中的应用受到高成本和处理跨模态信息的挑战的限制，导致对 DV 的 PLM 的研究很少。我们引入了 \textbf{DataVisT5}，这是一种为 DV 量身定制的新型 PLM，它通过混合目标预训练和多任务微调策略增强了 T5 架构，整合了文本和 DV 数据集以有效地解释跨模态语义。对公共数据集的广泛评估表明，DataVisT5 在各种 DV 相关任务上的表现始终优于当前最先进的模型。我们预计 DataVisT5 不仅会激发对垂直 PLM 的进一步研究，还会扩大 PLM 的应用范围。

Title: A Quantum-Inspired Analysis of Human Disambiguation Processes

• Authors: Daphne Wang
• Subjects: cs.CL, cs.AI, cs.LO, quant-ph
• Abstract URL: https://arxiv.org/abs/2408.07402
• Pdf URL: https://arxiv.org/pdf/2408.07402
• Copy Paste: [[2408.07402]] A Quantum-Inspired Analysis of Human Disambiguation Processes(https://arxiv.org/abs/2408.07402)
• Keywords: language model
• Abstract: Formal languages are essential for computer programming and are constructed to be easily processed by computers. In contrast, natural languages are much more challenging and instigated the field of Natural Language Processing (NLP). One major obstacle is the ubiquity of ambiguities. Recent advances in NLP have led to the development of large language models, which can resolve ambiguities with high accuracy. At the same time, quantum computers have gained much attention in recent years as they can solve some computational problems faster than classical computers. This new computing paradigm has reached the fields of machine learning and NLP, where hybrid classical-quantum learning algorithms have emerged. However, more research is needed to identify which NLP tasks could benefit from a genuine quantum advantage. In this thesis, we applied formalisms arising from foundational quantum mechanics, such as contextuality and causality, to study ambiguities arising from linguistics. By doing so, we also reproduced psycholinguistic results relating to the human disambiguation process. These results were subsequently used to predict human behaviour and outperformed current NLP methods.
• 摘要：形式语言对于计算机编程至关重要，并且易于计算机处理。相比之下，自然语言更具挑战性，并激发了自然语言处理 (NLP) 领域的发展。一个主要障碍是歧义的普遍存在。NLP 的最新进展导致了大型语言模型的发展，这些模型可以高精度地解决歧义问题。与此同时，量子计算机近年来备受关注，因为它们可以比传统计算机更快地解决一些计算问题。这种新的计算范式已经进入机器学习和 NLP 领域，其中出现了混合经典量子学习算法。然而，需要更多的研究来确定哪些 NLP 任务可以从真正的量子优势中受益。在本文中，我们应用了基础量子力学中产生的形式主义，例如语境性和因果关系，来研究语言学中产生的歧义。通过这样做，我们还重现了与人类消歧过程有关的心理语言学结果。这些结果随后被用于预测人类行为，并且优于当前的 NLP 方法。

Title: Knowledge in Superposition: Unveiling the Failures of Lifelong Knowledge Editing for Large Language Models

• Authors: Chenhui Hu, Pengfei Cao, Yubo Chen, Kang Liu, Jun Zhao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.07413
• Pdf URL: https://arxiv.org/pdf/2408.07413
• Copy Paste: [[2408.07413]] Knowledge in Superposition: Unveiling the Failures of Lifelong Knowledge Editing for Large Language Models(https://arxiv.org/abs/2408.07413)
• Keywords: language model, llm
• Abstract: Knowledge editing aims to update outdated or incorrect knowledge in large language models (LLMs). However, current knowledge editing methods have limited scalability for lifelong editing. This study explores the fundamental reason why knowledge editing fails in lifelong editing. We begin with the closed-form solution derived from linear associative memory, which underpins state-of-the-art knowledge editing methods. We extend the solution from single editing to lifelong editing, and through rigorous mathematical derivation, identify an interference term in the final solution, suggesting that editing knowledge may impact irrelevant knowledge. Further analysis of the interference term reveals a close relationship with superposition between knowledge representations. When knowledge superposition does not exist in language models, the interference term vanishes, allowing for lossless knowledge editing. Experiments across numerous language models reveal that knowledge superposition is universal, exhibiting high kurtosis, zero mean, and heavy-tailed distributions with clear scaling laws. Ultimately, by combining theory and experiments, we demonstrate that knowledge superposition is the fundamental reason for the failure of lifelong editing. Moreover, this is the first study to investigate knowledge editing from the perspective of superposition and provides a comprehensive observation of superposition across numerous real-world language models. Code available at this https URL.
• 摘要：知识编辑旨在更新大型语言模型（LLM）中过时或不正确的知识。然而，目前的知识编辑方法对终身编辑的可扩展性有限。本研究探讨了知识编辑在终身编辑中失败的根本原因。我们从线性联想记忆的闭式解开始，它是最先进的知识编辑方法的基础。我们将解决方案从单次编辑扩展到终身编辑，并通过严格的数学推导，在最终解决方案中确定了一个干扰项，表明编辑知识可能会影响不相关的知识。对干扰项的进一步分析揭示了它与知识表示之间的叠加有着密切的关系。当语言模型中不存在知识叠加时，干扰项就会消失，从而实现无损知识编辑。在众多语言模型中进行的实验表明，知识叠加具有普遍性，表现出高峰度、零均值和重尾分布，具有明显的缩放规律。最终，通过理论与实验相结合，我们证明了知识叠加是终身编辑失败的根本原因。此外，这是第一项从叠加角度研究知识编辑的研究，并对众多真实世界语言模型中的叠加进行了全面的观察。代码可从此 https URL 获取。

Title: Exploring Retrieval Augmented Generation in Arabic

• Authors: Samhaa R. El-Beltagy, Mohamed A. Abdallah
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.07425
• Pdf URL: https://arxiv.org/pdf/2408.07425
• Copy Paste: [[2408.07425]] Exploring Retrieval Augmented Generation in Arabic(https://arxiv.org/abs/2408.07425)
• Keywords: llm, retrieval augmented generation
• Abstract: Recently, Retrieval Augmented Generation (RAG) has emerged as a powerful technique in natural language processing, combining the strengths of retrieval-based and generation-based models to enhance text generation tasks. However, the application of RAG in Arabic, a language with unique characteristics and resource constraints, remains underexplored. This paper presents a comprehensive case study on the implementation and evaluation of RAG for Arabic text. The work focuses on exploring various semantic embedding models in the retrieval stage and several LLMs in the generation stage, in order to investigate what works and what doesn't in the context of Arabic. The work also touches upon the issue of variations between document dialect and query dialect in the retrieval stage. Results show that existing semantic embedding models and LLMs can be effectively employed to build Arabic RAG pipelines.
• 摘要：最近，检索增强生成 (RAG) 已成为自然语言处理中的一种强大技术，它结合了基于检索和基于生成的模型的优势来增强文本生成任务。然而，RAG 在具有独特特征和资源限制的阿拉伯语中的应用仍未得到充分探索。本文对阿拉伯语文本的 RAG 实现和评估进行了全面的案例研究。这项工作侧重于探索检索阶段的各种语义嵌入模型和生成阶段的几个 LLM，以研究在阿拉伯语环境中什么有效，什么无效。这项工作还涉及检索阶段文档方言和查询方言之间的差异问题。结果表明，现有的语义嵌入模型和 LLM 可有效用于构建阿拉伯语 RAG 管道。

Title: Fact or Fiction? Improving Fact Verification with Knowledge Graphs through Simplified Subgraph Retrievals

• Authors: Tobias A. Opsahl
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.07453
• Pdf URL: https://arxiv.org/pdf/2408.07453
• Copy Paste: [[2408.07453]] Fact or Fiction? Improving Fact Verification with Knowledge Graphs through Simplified Subgraph Retrievals(https://arxiv.org/abs/2408.07453)
• Keywords: language model
• Abstract: Despite recent success in natural language processing (NLP), fact verification still remains a difficult task. Due to misinformation spreading increasingly fast, attention has been directed towards automatically verifying the correctness of claims. In the domain of NLP, this is usually done by training supervised machine learning models to verify claims by utilizing evidence from trustworthy corpora. We present efficient methods for verifying claims on a dataset where the evidence is in the form of structured knowledge graphs. We use the FactKG dataset, which is constructed from the DBpedia knowledge graph extracted from Wikipedia. By simplifying the evidence retrieval process, from fine-tuned language models to simple logical retrievals, we are able to construct models that both require less computational resources and achieve better test-set accuracy.
• 摘要：尽管自然语言处理 (NLP) 领域最近取得了成功，但事实验证仍然是一项艰巨的任务。由于错误信息的传播速度越来越快，人们的注意力已经转向自动验证声明的正确性。在 NLP 领域，这通常是通过训练监督机器学习模型来利用来自可信语料库的证据来验证声明来完成的。我们提出了一种有效的方法来验证数据集上的声明，其中证据的形式是结构化知识图。我们使用 FactKG 数据集，该数据集是从维基百科中提取的 DBpedia 知识图构建的。通过简化证据检索过程，从微调语言模型到简单的逻辑检索，我们能够构建既需要更少计算资源又能实现更好的测试集准确率的模型。

Title: From Brazilian Portuguese to European Portuguese

• Authors: João Sanches, Rui Ribeiro, Luísa Coheur
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.07457
• Pdf URL: https://arxiv.org/pdf/2408.07457
• Copy Paste: [[2408.07457]] From Brazilian Portuguese to European Portuguese(https://arxiv.org/abs/2408.07457)
• Keywords: language model, gpt, chat
• Abstract: Brazilian Portuguese and European Portuguese are two varieties of the same language and, despite their close similarities, they exhibit several differences. However, there is a significant disproportion in the availability of resources between the two variants, with Brazilian Portuguese having more abundant resources. This inequity can impact the quality of translation services accessible to European Portuguese speakers. To address this issue, we propose the development of a Brazilian Portuguese to European Portuguese translation system, leveraging recent advancements in neural architectures and models. To evaluate the performance of such systems, we manually curated a gold test set comprising 500 sentences across five different topics. Each sentence in the gold test set has two distinct references, facilitating a straightforward evaluation of future translation models. We experimented with various models by fine-tuning existing Large Language Models using parallel data extracted from movie subtitles and TED Talks transcripts in both Brazilian and European Portuguese. Our evaluation involved the use of conventional automatic metrics as well as a human evaluation. In addition, all models were compared against ChatGPT 3.5 Turbo, which currently yields the best results.
• 摘要：巴西葡萄牙语和欧洲葡萄牙语是同一种语言的两个变体，尽管它们非常相似，但也表现出一些差异。然而，这两个变体在资源可用性方面存在显著的不平衡，巴西葡萄牙语拥有更丰富的资源。这种不平等会影响欧洲葡萄牙语使用者可获得的翻译服务的质量。为了解决这个问题，我们建议开发一个巴西葡萄牙语到欧洲葡萄牙语的翻译系统，利用神经架构和模型方面的最新进展。为了评估这类系统的性能，我们手动整理了一个黄金测试集，其中包含 500 个句子，涉及五个不同的主题。黄金测试集中的每个句子都有两个不同的参考，便于对未来的翻译模型进行直接评估。我们通过使用从巴西葡萄牙语和欧洲葡萄牙语的电影字幕和 TED 演讲记录中提取的并行数据对现有的大型语言模型进行微调，对各种模型进行了实验。我们的评估涉及使用传统的自动指标以及人工评估。此外，所有模型都与 ChatGPT 3.5 Turbo 进行了比较，后者目前效果最好。

Title: Large Language Models Prompting With Episodic Memory

• Authors: Dai Do, Quan Tran, Svetha Venkatesh, Hung Le
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.07465
• Pdf URL: https://arxiv.org/pdf/2408.07465
• Copy Paste: [[2408.07465]] Large Language Models Prompting With Episodic Memory(https://arxiv.org/abs/2408.07465)
• Keywords: language model, llm, prompt
• Abstract: Prompt optimization is essential for enhancing the performance of Large Language Models (LLMs) in a range of Natural Language Processing (NLP) tasks, particularly in scenarios of few-shot learning where training examples are incorporated directly into the prompt. Despite the growing interest in optimizing prompts with few-shot examples, existing methods for prompt optimization are often resource-intensive or perform inadequately. In this work, we propose PrOmpting with Episodic Memory (POEM), a novel prompt optimization technique that is simple, efficient, and demonstrates strong generalization capabilities. We approach prompt optimization as a Reinforcement Learning (RL) challenge, using episodic memory to archive combinations of input data, permutations of few-shot examples, and the rewards observed during training. In the testing phase, we optimize the sequence of examples for each test query by selecting the sequence that yields the highest total rewards from the top-k most similar training examples in the episodic memory. Our results show that POEM outperforms recent techniques like TEMPERA and RLPrompt by over 5.3% in various text classification tasks. Furthermore, our approach adapts well to broader language understanding tasks, consistently outperforming conventional heuristic methods for ordering examples.
• 摘要：提示优化对于提高大型语言模型 (LLM) 在一系列自然语言处理 (NLP) 任务中的性能至关重要，特别是在将训练示例直接纳入提示的少样本学习场景中。尽管人们对使用少样本示例优化提示的兴趣日益浓厚，但现有的提示优化方法通常资源密集或性能不佳。在这项工作中，我们提出了使用情景记忆进行 PrOmpting (POEM)，这是一种新颖的提示优化技术，简单、高效，并具有强大的泛化能力。我们将提示优化作为强化学习 (RL) 挑战，使用情景记忆来存档输入数据的组合、少样本示例的排列以及训练期间观察到的奖励。在测试阶段，我们通过从情景记忆中的前 k 个最相似的训练示例中选择产生最高总奖励的序列来优化每个测试查询的示例序列。我们的结果表明，POEM 在各种文本分类任务中的表现比 TEMPERA 和 RLPrompt 等最新技术高出 5.3% 以上。此外，我们的方法可以很好地适应更广泛的语言理解任务，在排序示例方面始终优于传统的启发式方法。

Title: Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization

• Authors: Yuxin Jiang, Bo Huang, Yufei Wang, Xingshan Zeng, Liangyou Li, Yasheng Wang, Xin Jiang, Lifeng Shang, Ruiming Tang, Wei Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.07471
• Pdf URL: https://arxiv.org/pdf/2408.07471
• Copy Paste: [[2408.07471]] Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization(https://arxiv.org/abs/2408.07471)
• Keywords: language model, llm
• Abstract: Direct preference optimization (DPO), a widely adopted offline preference optimization algorithm, aims to align large language models (LLMs) with human-desired behaviors using pairwise preference data. However, the winning response and the losing response within pairwise data are generated isolatedly, leading to weak correlations between them as well as suboptimal alignment performance. To address this issue, we propose an effective framework named BMC, for bridging and modeling correlations in pairwise data. Firstly, we increase the consistency and informativeness of the pairwise preference signals by targeted modifications, synthesizing a pseudo winning response through improving the losing response based on the winning response. Secondly, we identify that DPO alone is insufficient to model these correlations and capture nuanced variations. Therefore, we propose learning token-level correlations by dynamically leveraging the policy model's confidence during training. Comprehensive experiments on QA, math, and instruction-following tasks demonstrate the effectiveness of our approach, significantly surpassing competitive baselines, including DPO. Additionally, our in-depth quantitative analysis reveals the reasons behind our method's superior performance over DPO and showcases its versatility to other DPO variants.
• 摘要：直接偏好优化 (DPO) 是一种广泛采用的离线偏好优化算法，旨在使用成对偏好数据将大型语言模型 (LLM) 与人类期望的行为对齐。然而，成对数据中的获胜响应和失败响应是孤立生成的，导致它们之间的相关性较弱，并且对齐性能不佳。为了解决这个问题，我们提出了一个名为 BMC 的有效框架，用于桥接和建模成对数据中的相关性。首先，我们通过有针对性的修改来提高成对偏好信号的一致性和信息量，通过基于获胜响应改进失败响应来合成伪获胜响应。其次，我们发现仅靠 DPO 不足以模拟这些相关性并捕捉细微的变化。因此，我们提出通过在训练期间动态利用策略模型的置信度来学习 token 级相关性。在 QA、数学和指令遵循任务上的综合实验证明了我们方法的有效性，大大超越了包括 DPO 在内的竞争基线。此外，我们深入的定量分析揭示了我们的方法优于 DPO 的原因，并展示了其对其他 DPO 变体的多功能性。

Title: Large Language Models Know What Makes Exemplary Contexts

• Authors: Quanyu Long, Jianda Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.07505
• Pdf URL: https://arxiv.org/pdf/2408.07505
• Copy Paste: [[2408.07505]] Large Language Models Know What Makes Exemplary Contexts(https://arxiv.org/abs/2408.07505)
• Keywords: language model, llm
• Abstract: In-context learning (ICL) has proven to be a significant capability with the advancement of Large Language models (LLMs). By instructing LLMs using few-shot demonstrative examples, ICL enables them to perform a wide range of tasks without needing to update millions of parameters. This paper presents a unified framework for LLMs that allows them to self-select influential in-context examples to compose their contexts; self-rank candidates with different demonstration compositions; self-optimize the demonstration selection and ordering through reinforcement learning. Specifically, our method designs a parameter-efficient retrieval head that generates the optimized demonstration after training with rewards from LLM's own preference. Experimental results validate the proposed method's effectiveness in enhancing ICL performance. Additionally, our approach effectively identifies and selects the most representative examples for the current task, and includes more diversity in retrieval.
• 摘要：随着大型语言模型 (LLM) 的发展，上下文学习 (ICL) 已被证明是一项重要功能。通过使用少量演示示例来指导 LLM，ICL 使它们能够执行广泛的任务，而无需更新数百万个参数。本文为 LLM 提出了一个统一的框架，使它们能够自我选择有影响力的上下文示例来组成其上下文；对具有不同演示组成的候选者进行自我排序；通过强化学习自我优化演示选择和排序。具体来说，我们的方法设计了一个参数高效的检索头，它在使用 LLM 自身偏好的奖励进行训练后生成优化的演示。实验结果验证了所提出的方法在提高 ICL 性能方面的有效性。此外，我们的方法有效地识别和选择当前任务最具代表性的示例，并在检索中包含更多多样性。

Title: WeKnow-RAG: An Adaptive Approach for Retrieval-Augmented Generation Integrating Web Search and Knowledge Graphs

• Authors: Weijian Xie, Xuefeng Liang, Yuhui Liu, Kaihua Ni, Hong Cheng, Zetian Hu
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2408.07611
• Pdf URL: https://arxiv.org/pdf/2408.07611
• Copy Paste: [[2408.07611]] WeKnow-RAG: An Adaptive Approach for Retrieval-Augmented Generation Integrating Web Search and Knowledge Graphs(https://arxiv.org/abs/2408.07611)
• Keywords: language model, llm, retrieval-augmented generation, agent
• Abstract: Large Language Models (LLMs) have greatly contributed to the development of adaptive intelligent agents and are positioned as an important way to achieve Artificial General Intelligence (AGI). However, LLMs are prone to produce factually incorrect information and often produce "phantom" content that undermines their reliability, which poses a serious challenge for their deployment in real-world scenarios. Enhancing LLMs by combining external databases and information retrieval mechanisms is an effective path. To address the above challenges, we propose a new approach called WeKnow-RAG, which integrates Web search and Knowledge Graphs into a "Retrieval-Augmented Generation (RAG)" system. First, the accuracy and reliability of LLM responses are improved by combining the structured representation of Knowledge Graphs with the flexibility of dense vector retrieval. WeKnow-RAG then utilizes domain-specific knowledge graphs to satisfy a variety of queries and domains, thereby improving performance on factual information and complex reasoning tasks by employing multi-stage web page retrieval techniques using both sparse and dense retrieval methods. Our approach effectively balances the efficiency and accuracy of information retrieval, thus improving the overall retrieval process. Finally, we also integrate a self-assessment mechanism for the LLM to evaluate the trustworthiness of the answers it generates. Our approach proves its outstanding effectiveness in a wide range of offline experiments and online submissions.
• 摘要：大型语言模型 (LLM) 极大地促进了自适应智能代理的发展，并被定位为实现通用人工智能 (AGI) 的重要途径。然而，LLM 容易产生事实上不正确的信息，并且经常产生破坏其可靠性的“幻影”内容，这对其在现实场景中的部署构成了严峻挑战。通过结合外部数据库和信息检索机制来增强 LLM 是一种有效的途径。为了应对上述挑战，我们提出了一种名为 WeKnow-RAG 的新方法，它将 Web 搜索和知识图谱集成到“检索增强生成 (RAG)”系统中。首先，通过将知识图谱的结构化表示与密集向量检索的灵活性相结合，可以提高 LLM 响应的准确性和可靠性。然后，WeKnow-RAG 利用特定领域的知识图谱来满足各种查询和领域，从而通过使用稀疏和密集检索方法采用多阶段网页检索技术来提高事实信息和复杂推理任务的性能。我们的方法有效地平衡了信息检索的效率和准确性，从而改善了整个检索过程。最后，我们还为 LLM 集成了一个自我评估机制，以评估其生成的答案的可信度。我们的方法在广泛的离线实验和在线提交中证明了其出色的有效性。

Title: Alignment-Enhanced Decoding:Defending via Token-Level Adaptive Refining of Probability Distributions

• Authors: Quan Liu, Zhenhong Zhou, Longzhu He, Yi Liu, Wei Zhang, Sen Su
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.07663
• Pdf URL: https://arxiv.org/pdf/2408.07663
• Copy Paste: [[2408.07663]] Alignment-Enhanced Decoding:Defending via Token-Level Adaptive Refining of Probability Distributions(https://arxiv.org/abs/2408.07663)
• Keywords: language model
• Abstract: Large language models are susceptible to jailbreak attacks, which can result in the generation of harmful content. While prior defenses mitigate these risks by perturbing or inspecting inputs, they ignore competing objectives, the underlying cause of alignment failures. In this paper, we propose Alignment-Enhanced Decoding (AED), a novel defense that employs adaptive decoding to address the root causes of jailbreak issues. We first define the Competitive Index to quantify alignment failures and utilize feedback from self-evaluation to compute post-alignment logits. Then, AED adaptively combines AED and post-alignment logits with the original logits to obtain harmless and helpful distributions. Consequently, our method enhances safety alignment while maintaining helpfulness. We conduct experiments across five models and four common jailbreaks, with the results validating the effectiveness of our approach. Code is available at this https URL.
• 摘要：大型语言模型容易受到越狱攻击，从而导致有害内容的生成。虽然先前的防御措施通过扰乱或检查输入来减轻这些风险，但它们忽略了竞争目标，即对齐失败的根本原因。在本文中，我们提出了对齐增强解码 (AED)，这是一种采用自适应解码来解决越狱问题根本原因的新型防御措施。我们首先定义竞争指数来量化对齐失败，并利用自我评估的反馈来计算对齐后逻辑。然后，AED 自适应地将 AED 和对齐后逻辑与原始逻辑相结合，以获得无害且有用的分布。因此，我们的方法在保持有用性的同时增强了安全对齐。我们对五种模型和四种常见的越狱进行了实验，结果验证了我们方法的有效性。代码可在此 https URL 上找到。

Title: Spoken Stereoset: On Evaluating Social Bias Toward Speaker in Speech Large Language Models

• Authors: Yi-Cheng Lin, Wei-Chih Chen, Hung-yi Lee
• Subjects: cs.CL, eess.AS
• Abstract URL: https://arxiv.org/abs/2408.07665
• Pdf URL: https://arxiv.org/pdf/2408.07665
• Copy Paste: [[2408.07665]] Spoken Stereoset: On Evaluating Social Bias Toward Speaker in Speech Large Language Models(https://arxiv.org/abs/2408.07665)
• Keywords: language model, llm
• Abstract: Warning: This paper may contain texts with uncomfortable content. Large Language Models (LLMs) have achieved remarkable performance in various tasks, including those involving multimodal data like speech. However, these models often exhibit biases due to the nature of their training data. Recently, more Speech Large Language Models (SLLMs) have emerged, underscoring the urgent need to address these biases. This study introduces Spoken Stereoset, a dataset specifically designed to evaluate social biases in SLLMs. By examining how different models respond to speech from diverse demographic groups, we aim to identify these biases. Our experiments reveal significant insights into their performance and bias levels. The findings indicate that while most models show minimal bias, some still exhibit slightly stereotypical or anti-stereotypical tendencies.
• 摘要：警告：本文可能包含令人不适的内容。大型语言模型 (LLM) 在各种任务中都取得了卓越的表现，包括涉及语音等多模态数据的任务。然而，由于训练数据的性质，这些模型往往会表现出偏见。最近，出现了更多的语音大型语言模型 (SLLM)，凸显了解决这些偏见的迫切需要。本研究介绍了 Spoken Stereoset，这是一个专门用于评估 SLLM 中的社会偏见的数据集。通过研究不同模型如何响应来自不同人口群体的语音，我们旨在识别这些偏见。我们的实验揭示了有关它们的性能和偏见水平的重要见解。研究结果表明，虽然大多数模型表现出最小的偏见，但有些模型仍然表现出轻微的刻板印象或反刻板印象倾向。

Title: Enhanced Detection of Conversational Mental Manipulation Through Advanced Prompting Techniques

• Authors: Ivory Yang, Xiaobo Guo, Sean Xie, Soroush Vosoughi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.07676
• Pdf URL: https://arxiv.org/pdf/2408.07676
• Copy Paste: [[2408.07676]] Enhanced Detection of Conversational Mental Manipulation Through Advanced Prompting Techniques(https://arxiv.org/abs/2408.07676)
• Keywords: prompt, chain-of-thought
• Abstract: This study presents a comprehensive, long-term project to explore the effectiveness of various prompting techniques in detecting dialogical mental manipulation. We implement Chain-of-Thought prompting with Zero-Shot and Few-Shot settings on a binary mental manipulation detection task, building upon existing work conducted with Zero-Shot and Few- Shot prompting. Our primary objective is to decipher why certain prompting techniques display superior performance, so as to craft a novel framework tailored for detection of mental manipulation. Preliminary findings suggest that advanced prompting techniques may not be suitable for more complex models, if they are not trained through example-based learning.
• 摘要：本研究提出了一个全面的长期项目，旨在探索各种提示技术在检测对话心理操纵方面的有效性。我们在二元心理操纵检测任务中实施了零次和少量设置的思维链提示，以现有的零次和少量提示工作为基础。我们的主要目标是解释为什么某些提示技术表现出优越的性能，从而设计出一个专门用于检测心理操纵的新框架。初步研究结果表明，如果不通过基于示例的学习进行训练，高级提示技术可能不适合更复杂的模型。

Title: The Death of Schema Linking? Text-to-SQL in the Age of Well-Reasoned Language Models

• Authors: Karime Maamari, Fadhil Abubaker, Daniel Jaroslawicz, Amine Mhedhbi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.07702
• Pdf URL: https://arxiv.org/pdf/2408.07702
• Copy Paste: [[2408.07702]] The Death of Schema Linking? Text-to-SQL in the Age of Well-Reasoned Language Models(https://arxiv.org/abs/2408.07702)
• Keywords: language model, llm
• Abstract: Schema linking is a crucial step in Text-to-SQL pipelines, which translate natural language queries into SQL. The goal of schema linking is to retrieve relevant tables and columns (signal) while disregarding irrelevant ones (noise). However, imperfect schema linking can often exclude essential columns needed for accurate query generation. In this work, we revisit the need for schema linking when using the latest generation of large language models (LLMs). We find empirically that newer models are adept at identifying relevant schema elements during generation, without the need for explicit schema linking. This allows Text-to-SQL pipelines to bypass schema linking entirely and instead pass the full database schema to the LLM, eliminating the risk of excluding necessary information. Furthermore, as alternatives to schema linking, we propose techniques that improve Text-to-SQL accuracy without compromising on essential schema information. Our approach achieves 71.83\% execution accuracy on the BIRD benchmark, ranking first at the time of submission.
• 摘要：模式链接是文本到 SQL 管道中的关键步骤，它将自然语言查询转换为 SQL。模式链接的目标是检索相关的表和列（信号），同时忽略不相关的表和列（噪声）。然而，不完善的模式链接通常会排除准确查询生成所需的基本列。在这项工作中，我们重新审视了使用最新一代大型语言模型 (LLM) 时模式链接的必要性。我们通过经验发现，较新的模型擅长在生成过程中识别相关的模式元素，而无需显式模式链接。这允许文本到 SQL 管道完全绕过模式链接，而是将完整的数据库模式传递给 LLM，从而消除了排除必要信息的风险。此外，作为模式链接的替代方案，我们提出了一些技术，可以在不影响基本模式信息的情况下提高文本到 SQL 的准确性。我们的方法在 BIRD 基准测试中实现了 71.83% 的执行准确率，在提交时排名第一。