2024-07-23

Title: SQLfuse: Enhancing Text-to-SQL Performance through Comprehensive LLM Synergy

• Authors: Tingkai Zhang, Chaoyu Chen, Cong Liao, Jun Wang, Xudong Zhao, Hang Yu, Jianchao Wang, Jianguo Li, Wenhui Shi
• Subjects: cs.CL, cs.AI, cs.DB
• Abstract URL: https://arxiv.org/abs/2407.14568
• Pdf URL: https://arxiv.org/pdf/2407.14568
• Copy Paste: [[2407.14568]] SQLfuse: Enhancing Text-to-SQL Performance through Comprehensive LLM Synergy(https://arxiv.org/abs/2407.14568)
• Keywords: language model, gpt, llm
• Abstract: Text-to-SQL conversion is a critical innovation, simplifying the transition from complex SQL to intuitive natural language queries, especially significant given SQL's prevalence in the job market across various roles. The rise of Large Language Models (LLMs) like GPT-3.5 and GPT-4 has greatly advanced this field, offering improved natural language understanding and the ability to generate nuanced SQL statements. However, the potential of open-source LLMs in Text-to-SQL applications remains underexplored, with many frameworks failing to leverage their full capabilities, particularly in handling complex database queries and incorporating feedback for iterative refinement. Addressing these limitations, this paper introduces SQLfuse, a robust system integrating open-source LLMs with a suite of tools to enhance Text-to-SQL translation's accuracy and usability. SQLfuse features four modules: schema mining, schema linking, SQL generation, and a SQL critic module, to not only generate but also continuously enhance SQL query quality. Demonstrated by its leading performance on the Spider Leaderboard and deployment by Ant Group, SQLfuse showcases the practical merits of open-source LLMs in diverse business contexts.
• 摘要：文本到 SQL 的转换是一项关键创新，它简化了从复杂 SQL 到直观自然语言查询的过渡，鉴于 SQL 在就业市场中各个岗位的普遍性，这一创新尤其重要。GPT-3.5 和 GPT-4 等大型语言模型 (LLM) 的兴起极大地推动了这一领域的发展，提供了更好的自然语言理解和生成细微 SQL 语句的能力。然而，开源 LLM 在文本到 SQL 应用程序中的潜力仍未得到充分开发，许多框架未能充分利用其功能，特别是在处理复杂的数据库查询和结合反馈进行迭代细化方面。为了解决这些限制，本文介绍了 SQLfuse，这是一个强大的系统，将开源 LLM 与一套工具集成在一起，以增强文本到 SQL 转换的准确性和可用性。SQLfuse 具有四个模块：模式挖掘、模式链接、SQL 生成和 SQL 评论模块，不仅可以生成 SQL 查询质量，还可以持续提高 SQL 查询质量。 SQLfuse 在 Spider 排行榜上的领先表现以及蚂蚁集团的部署，展示了开源 LLM 在不同业务环境中的实用价值。

Title: Adversarial Databases Improve Success in Retrieval-based Large Language Models

• Authors: Sean Wu, Michael Koo, Li Yo Kao, Andy Black, Lesley Blum, Fabien Scalzo, Ira Kurtz
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.14609
• Pdf URL: https://arxiv.org/pdf/2407.14609
• Copy Paste: [[2407.14609]] Adversarial Databases Improve Success in Retrieval-based Large Language Models(https://arxiv.org/abs/2407.14609)
• Keywords: language model, llm, chat, retrieval-augmented generation
• Abstract: Open-source LLMs have shown great potential as fine-tuned chatbots, and demonstrate robust abilities in reasoning and surpass many existing benchmarks. Retrieval-Augmented Generation (RAG) is a technique for improving the performance of LLMs on tasks that the models weren't explicitly trained on, by leveraging external knowledge databases. Numerous studies have demonstrated the effectiveness of RAG to more successfully accomplish downstream tasks when using vector datasets that consist of relevant background information. It has been implicitly assumed by those in the field that if adversarial background information is utilized in this context, that the success of using a RAG-based approach would be nonexistent or even negatively impact the results. To address this assumption, we tested several open-source LLMs on the ability of RAG to improve their success in answering multiple-choice questions (MCQ) in the medical subspecialty field of Nephrology. Unlike previous studies, we examined the effect of RAG in utilizing both relevant and adversarial background databases. We set up several open-source LLMs, including Llama 3, Phi-3, Mixtral 8x7b, Zephyr$\beta$, and Gemma 7B Instruct, in a zero-shot RAG pipeline. As adversarial sources of information, text from the Bible and a Random Words generated database were used for comparison. Our data show that most of the open-source LLMs improve their multiple-choice test-taking success as expected when incorporating relevant information vector databases. Surprisingly however, adversarial Bible text significantly improved the success of many LLMs and even random word text improved test taking ability of some of the models. In summary, our results demonstrate for the first time the countertintuitive ability of adversarial information datasets to improve the RAG-based LLM success.
• 摘要：开源 LLM 已显示出作为微调聊天机器人的巨大潜力，并展示了强大的推理能力，超越了许多现有基准。检索增强生成 (RAG) 是一种通过利用外部知识数据库来提高 LLM 在模型未明确训练的任务上的性能的技术。许多研究表明，当使用包含相关背景信息的矢量数据集时，RAG 可以更成功地完成下游任务。业内人士已经隐含地认为，如果在这种情况下使用对抗性背景信息，那么使用基于 RAG 的方法将不会成功，甚至会对结果产生负面影响。为了解决这一假设，我们测试了几个开源 LLM，以了解 RAG 提高其在肾脏病医学专科领域回答多项选择题 (MCQ) 成功率的能力。与以前的研究不同，我们研究了 RAG 在利用相关和对抗性背景数据库方面的影响。我们在零样本 RAG 管道中设置了几个开源 LLM，包括 Llama 3、Phi-3、Mixtral 8x7b、Zephyr$\beta$ 和 Gemma 7B Instruct。作为对抗性信息源，来自圣经的文本和随机词生成的数据库用于比较。我们的数据显示，大多数开源 LLM 在合并相关信息向量数据库时都如预期的那样提高了多项选择题考试的成功率。然而令人惊讶的是，对抗性圣经文本显著提高了许多 LLM 的成功率，甚至随机词文本也提高了一些模型的考试能力。总之，我们的结果首次证明了对抗性信息数据集能够提高基于 RAG 的 LLM 成功率的反直觉能力。

Title: CVE-LLM : Automatic vulnerability evaluation in medical device industry using large language models

• Authors: Rikhiya Ghosh, Oladimeji Farri, Hans-Martin von Stockhausen, Martin Schmitt, George Marica Vasile
• Subjects: cs.CL, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2407.14640
• Pdf URL: https://arxiv.org/pdf/2407.14640
• Copy Paste: [[2407.14640]] CVE-LLM : Automatic vulnerability evaluation in medical device industry using large language models(https://arxiv.org/abs/2407.14640)
• Keywords: language model, llm
• Abstract: The healthcare industry is currently experiencing an unprecedented wave of cybersecurity attacks, impacting millions of individuals. With the discovery of thousands of vulnerabilities each month, there is a pressing need to drive the automation of vulnerability assessment processes for medical devices, facilitating rapid mitigation efforts. Generative AI systems have revolutionized various industries, offering unparalleled opportunities for automation and increased efficiency. This paper presents a solution leveraging Large Language Models (LLMs) to learn from historical evaluations of vulnerabilities for the automatic assessment of vulnerabilities in the medical devices industry. This approach is applied within the portfolio of a single manufacturer, taking into account device characteristics, including existing security posture and controls. The primary contributions of this paper are threefold. Firstly, it provides a detailed examination of the best practices for training a vulnerability Language Model (LM) in an industrial context. Secondly, it presents a comprehensive comparison and insightful analysis of the effectiveness of Language Models in vulnerability assessment. Finally, it proposes a new human-in-the-loop framework to expedite vulnerability evaluation processes.
• 摘要：医疗保健行业目前正在经历前所未有的网络安全攻击浪潮，影响了数百万人。由于每月都会发现数千个漏洞，因此迫切需要推动医疗设备漏洞评估流程的自动化，以促进快速缓解措施。生成式人工智能系统已经彻底改变了各个行业，为自动化和提高效率提供了无与伦比的机会。本文提出了一种解决方案，利用大型语言模型 (LLM) 从历史漏洞评估中学习，以自动评估医疗器械行业的漏洞。这种方法应用于单个制造商的产品组合中，考虑到设备特性，包括现有的安全态势和控制。本文的主要贡献有三点。首先，它详细研究了在工业环境中训练漏洞语言模型 (LM) 的最佳实践。其次，它对语言模型在漏洞评估中的有效性进行了全面的比较和深入的分析。最后，它提出了一个新的人机循环框架来加快漏洞评估过程。

Title: Human-Interpretable Adversarial Prompt Attack on Large Language Models with Situational Context

• Authors: Nilanjana Das, Edward Raff, Manas Gaur
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.14644
• Pdf URL: https://arxiv.org/pdf/2407.14644
• Copy Paste: [[2407.14644]] Human-Interpretable Adversarial Prompt Attack on Large Language Models with Situational Context(https://arxiv.org/abs/2407.14644)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: Previous research on testing the vulnerabilities in Large Language Models (LLMs) using adversarial attacks has primarily focused on nonsensical prompt injections, which are easily detected upon manual or automated review (e.g., via byte entropy). However, the exploration of innocuous human-understandable malicious prompts augmented with adversarial injections remains limited. In this research, we explore converting a nonsensical suffix attack into a sensible prompt via a situation-driven contextual re-writing. This allows us to show suffix conversion without any gradients, using only LLMs to perform the attacks, and thus better understand the scope of possible risks. We combine an independent, meaningful adversarial insertion and situations derived from movies to check if this can trick an LLM. The situations are extracted from the IMDB dataset, and prompts are defined following a few-shot chain-of-thought prompting. Our approach demonstrates that a successful situation-driven attack can be executed on both open-source and proprietary LLMs. We find that across many LLMs, as few as 1 attempt produces an attack and that these attacks transfer between LLMs. The link to our code is available at \url{https://anonymous.4open.science/r/Situation-Driven-Adversarial-Attacks-7BB1/README.md}.
• 摘要：之前使用对抗性攻击测试大型语言模型 (LLM) 中的漏洞的研究主要集中在无意义的提示注入上，这些提示注入很容易在手动或自动审查（例如通过字节熵）时检测到。然而，对无害的、人类可理解的恶意提示进行对抗性注入的探索仍然有限。在这项研究中，我们探索通过情境驱动的上下文重写将无意义的后缀攻击转换为合理的提示。这使我们能够展示后缀转换，而无需任何梯度，仅使用 LLM 执行攻击，从而更好地了解可能的风险范围。我们结合了独立的、有意义的对抗性插入和从电影中衍生的情境，以检查这是否可以欺骗 LLM。这些情境是从 IMDB 数据集中提取的，并根据几次思路链提示定义提示。我们的方法表明，成功的情境驱动攻击可以在开源和专有 LLM 上执行。我们发现，在许多 LLM 中，只需 1 次尝试就会产生攻击，并且这些攻击会在 LLM 之间转移。我们的代码链接位于 \url{https://anonymous.4open.science/r/Situation-Driven-Adversarial-Attacks-7BB1/README.md}。

Title: Compact Language Models via Pruning and Knowledge Distillation

• Authors: Saurav Muralidharan, Sharath Turuvekere Sreenivas, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, Pavlo Molchanov
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.14679
• Pdf URL: https://arxiv.org/pdf/2407.14679
• Copy Paste: [[2407.14679]] Compact Language Models via Pruning and Knowledge Distillation(https://arxiv.org/abs/2407.14679)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) targeting different deployment scales and sizes are currently produced by training each variant from scratch; this is extremely compute-intensive. In this paper, we investigate if pruning an existing LLM and then re-training it with a fraction (<3%) of the original training data can be a suitable alternative to repeated, full retraining. To this end, we develop a set of practical and effective compression best practices for LLMs that combine depth, width, attention and MLP pruning with knowledge distillation-based retraining; we arrive at these best practices through a detailed empirical exploration of pruning strategies for each axis, methods to combine axes, distillation strategies, and search techniques for arriving at optimal compressed architectures. We use this guide to compress the Nemotron-4 family of LLMs by a factor of 2-4x, and compare their performance to similarly-sized models on a variety of language modeling tasks. Deriving 8B and 4B models from an already pretrained 15B model using our approach requires up to 40x fewer training tokens per model compared to training from scratch; this results in compute cost savings of 1.8x for training the full model family (15B, 8B, and 4B). Minitron models exhibit up to a 16% improvement in MMLU scores compared to training from scratch, perform comparably to other community models such as Mistral 7B, Gemma 7B and Llama-3 8B, and outperform state-of-the-art compression techniques from the literature. We have open-sourced Minitron model weights on Huggingface, with corresponding supplementary material including example code available on GitHub.
• 摘要：目前，针对不同部署规模和大小的大型语言模型 (LLM) 是通过从头开始训练每个变体来生成的；这极其耗费计算资源。在本文中，我们研究了修剪现有 LLM，然后使用原始训练数据的一小部分 (<3%) 对其进行重新训练是否可以成为重复、完全重新训练的合适替代方案。为此，我们为 LLM 开发了一套实用且有效的压缩最佳实践，将深度、宽度、注意力和 MLP 修剪与基于知识蒸馏的再训练相结合；我们通过对每个轴的修剪策略、组合轴的方法、蒸馏策略和搜索技术进行详细的实证探索，得出这些最佳实践，以实现最佳压缩架构。我们使用本指南将 Nemotron-4 系列 LLM 压缩 2-4 倍，并将它们的性能与各种语言建模任务中类似大小的模型进行比较。使用我们的方法从已经预训练的 15B 模型中派生出 8B 和 4B 模型，与从头开始训练相比，每个模型所需的训练令牌最多减少了 40 倍；这为训练整个模型系列（15B、8B 和 4B）节省了 1.8 倍的计算成本。与从头开始训练相比，Minitron 模型的 MMLU 分数提高了 16%，性能可与 Mistral 7B、Gemma 7B 和 Llama-3 8B 等其他社区模型相媲美，并且优于文献中最先进的压缩技术。我们在 Huggingface 上开源了 Minitron 模型权重，并在 GitHub 上提供了相应的补充材料（包括示例代码）。

Title: Economy Watchers Survey provides Datasets and Tasks for Japanese Financial Domain

• Authors: Masahiro Suzuki, Hiroki Sakaji
• Subjects: cs.CL, cs.CE
• Abstract URL: https://arxiv.org/abs/2407.14727
• Pdf URL: https://arxiv.org/pdf/2407.14727
• Copy Paste: [[2407.14727]] Economy Watchers Survey provides Datasets and Tasks for Japanese Financial Domain(https://arxiv.org/abs/2407.14727)
• Keywords: language model
• Abstract: Many natural language processing (NLP) tasks in English or general domains are widely available and are often used to evaluate pre-trained language models. In contrast, there are fewer tasks available for languages other than English and for the financial domain. In particular, tasks in Japanese and the financial domain are limited. We construct two large datasets using materials published by a Japanese central government agency. The datasets provide three Japanese financial NLP tasks, which include a 3-class and 12-class classification for categorizing sentences, as well as a 5-class classification task for sentiment analysis. Our datasets are designed to be comprehensive and up-to-date, leveraging an automatic update framework that ensures the latest task datasets are publicly available anytime.
• 摘要：英语或一般领域的许多自然语言处理 (NLP) 任务广泛可用，并经常用于评估预训练语言模型。相比之下，除英语以外的其他语言和金融领域的任务较少。特别是日语和金融领域的任务非常有限。我们使用日本中央政府机构发布的材料构建了两个大型数据集。数据集提供了三个日语金融 NLP 任务，其中包括用于对句子进行分类的 3 类和 12 类分类，以及用于情绪分析的 5 类分类任务。我们的数据集旨在全面且最新，利用自动更新框架确保最新的任务数据集随时公开可用。

Title: I Need Help! Evaluating LLM's Ability to Ask for Users' Support: A Case Study on Text-to-SQL Generation

• Authors: Cheng-Kuang Wu, Zhi Rui Tam, Chao-Chung Wu, Chieh-Yen Lin, Hung-yi Lee, Yun-Nung Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.14767
• Pdf URL: https://arxiv.org/pdf/2407.14767
• Copy Paste: [[2407.14767]] I Need Help! Evaluating LLM's Ability to Ask for Users' Support: A Case Study on Text-to-SQL Generation(https://arxiv.org/abs/2407.14767)
• Keywords: llm
• Abstract: In this study, we explore the proactive ability of LLMs to seek user support, using text-to-SQL generation as a case study. We propose metrics to evaluate the trade-off between performance improvements and user burden, and investigate whether LLMs can determine when to request help and examine their performance with varying levels of information availability. Our experiments reveal that without external feedback, many LLMs struggle to recognize their need for additional support. Our findings highlight the importance of external signals and provide insights for future research on improving support-seeking strategies.
• 摘要：在本研究中，我们以文本到 SQL 生成为例，探讨了 LLM 主动寻求用户支持的能力。我们提出了衡量性能改进与用户负担之间权衡的指标，并研究了 LLM 是否可以确定何时请求帮助，并在不同级别的信息可用性下检查其性能。我们的实验表明，如果没有外部反馈，许多 LLM 很难认识到他们对额外支持的需求。我们的研究结果强调了外部信号的重要性，并为未来改进寻求支持策略的研究提供了见解。

Title: Step-by-Step Reasoning to Solve Grid Puzzles: Where do LLMs Falter?

• Authors: Nemika Tyagi, Mihir Parmar, Mohith Kulkarni, Aswin RRV, Nisarg Patel, Mutsumi Nakamura, Arindam Mitra, Chitta Baral
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.14790
• Pdf URL: https://arxiv.org/pdf/2407.14790
• Copy Paste: [[2407.14790]] Step-by-Step Reasoning to Solve Grid Puzzles: Where do LLMs Falter?(https://arxiv.org/abs/2407.14790)
• Keywords: gpt, llm, prompt
• Abstract: Solving grid puzzles involves a significant amount of logical reasoning. Hence, it is a good domain to evaluate the reasoning capability of a model which can then guide us to improve the reasoning ability of models. However, most existing works evaluate only the final predicted answer of a puzzle, without delving into an in-depth analysis of the LLMs' reasoning chains (such as where they falter) or providing any finer metrics to evaluate them. Since LLMs may rely on simple heuristics or artifacts to predict the final answer, it is crucial to evaluate the generated reasoning chain beyond overall correctness measures, for accurately evaluating the reasoning abilities of LLMs. To this end, we first develop GridPuzzle, an evaluation dataset comprising 274 grid-based puzzles with different complexities. Second, we propose a new error taxonomy derived from manual analysis of reasoning chains from LLMs including GPT-4, Claude-3, Gemini, Mistral, and Llama-2. Then, we develop an LLM-based framework for large-scale subjective evaluation (i.e., identifying errors) and an objective metric, PuzzleEval, to evaluate the correctness of reasoning chains. Evaluating reasoning chains from LLMs leads to several interesting findings. We further show that existing prompting methods used for enhancing models' reasoning abilities do not improve performance on GridPuzzle. This highlights the importance of understanding fine-grained errors and presents a challenge for future research to enhance LLMs' puzzle-solving abilities by developing methods that address these errors. Data and source code are available at this https URL.
• 摘要：解决网格谜题需要大量的逻辑推理。因此，这是一个评估模型推理能力的好领域，可以指导我们提高模型的推理能力。然而，大多数现有研究仅评估谜题的最终预测答案，而没有深入分析 LLM 的推理链（例如它们失败的地方）或提供任何更精细的指标来评估它们。由于 LLM 可能依赖简单的启发式方法或人工制品来预测最终答案，因此除了整体正确性度量之外，评估生成的推理链对于准确评估 LLM 的推理能力至关重要。为此，我们首先开发了 GridPuzzle，这是一个评估数据集，包含 274 个具有不同复杂度的基于网格的谜题。其次，我们提出了一种新的错误分类法，该分类法源自对包括 GPT-4、Claude-3、Gemini、Mistral 和 Llama-2 在内的 LLM 的推理链的手动分析。然后，我们开发了一个基于 LLM 的大规模主观评估框架（即识别错误）和一个客观指标 PuzzleEval，以评估推理链的正确性。评估 LLM 的推理链得出了一些有趣的发现。我们进一步表明，用于增强模型推理能力的现有提示方法并没有提高 GridPuzzle 的性能。这凸显了理解细粒度错误的重要性，并为未来的研究提出了挑战，即通过开发解决这些错误的方法来增强 LLM 的解谜能力。数据和源代码可在此 https URL 上找到。

Title: Seal: Advancing Speech Language Models to be Few-Shot Learners

• Authors: Shuyu Lei, Lingen Liu, Jiaolong Yang, Yasen Jiao, Yuxiang Yang, Yushu Yang, Xiang Guo
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.14875
• Pdf URL: https://arxiv.org/pdf/2407.14875
• Copy Paste: [[2407.14875]] Seal: Advancing Speech Language Models to be Few-Shot Learners(https://arxiv.org/abs/2407.14875)
• Keywords: language model, prompt
• Abstract: Existing auto-regressive language models have demonstrated a remarkable capability to perform a new task with just a few examples in prompt, without requiring any additional training. In order to extend this capability to a multi-modal setting (i.e. speech and language), this paper introduces the Seal model, an abbreviation for speech language model. It incorporates a novel alignment method, in which Kullback-Leibler divergence loss is performed to train a projector that bridges a frozen speech encoder with a frozen language model decoder. The resulting Seal model exhibits robust performance as a few-shot learner on two speech understanding tasks. Additionally, consistency experiments are conducted to validate its robustness on different pre-trained language models.
• 摘要：现有的自回归语言模型已经展现出非凡的能力，只需几个示例即可完成一项新任务，而无需任何额外的训练。为了将这种能力扩展到多模态设置（即语音和语言），本文介绍了 Seal 模型，这是语音语言模型的缩写。它采用了一种新颖的对齐方法，其中执行 Kullback-Leibler 散度损失来训练一个投影仪，该投影仪将冻结的语音编码器与冻结的语言模型解码器连接起来。由此产生的 Seal 模型作为两个语音理解任务上的少样本学习器表现出稳健的性能。此外，还进行了一致性实验以验证其在不同预训练语言模型上的稳健性。

Title: Modular Sentence Encoders: Separating Language Specialization from Cross-Lingual Alignment

• Authors: Yongxin Huang, Kexin Wang, Goran Glavaš, Iryna Gurevych
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.14878
• Pdf URL: https://arxiv.org/pdf/2407.14878
• Copy Paste: [[2407.14878]] Modular Sentence Encoders: Separating Language Specialization from Cross-Lingual Alignment(https://arxiv.org/abs/2407.14878)
• Keywords: language model
• Abstract: Multilingual sentence encoders are commonly obtained by training multilingual language models to map sentences from different languages into a shared semantic space. As such, they are subject to curse of multilinguality, a loss of monolingual representational accuracy due to parameter sharing. Another limitation of multilingual sentence encoders is the trade-off between monolingual and cross-lingual performance. Training for cross-lingual alignment of sentence embeddings distorts the optimal monolingual structure of semantic spaces of individual languages, harming the utility of sentence embeddings in monolingual tasks. In this work, we address both issues by modular training of sentence encoders, i.e., by separating monolingual specialization from cross-lingual alignment. We first efficiently train language-specific sentence encoders to avoid negative interference between languages (i.e., the curse). We then align all non-English monolingual encoders to the English encoder by training a cross-lingual alignment adapter on top of each, preventing interference with monolingual specialization from the first step. In both steps, we resort to contrastive learning on machine-translated paraphrase data. Monolingual and cross-lingual evaluations on semantic text similarity/relatedness and multiple-choice QA render our modular solution more effective than multilingual sentence encoders, especially benefiting low-resource languages.
• 摘要：多语言句子编码器通常是通过训练多语言语言模型来获得，以将来自不同语言的句子映射到共享的语义空间中。因此，它们容易受到多语言诅咒的影响，即由于参数共享而导致的单语表示准确性的损失。多语言句子编码器的另一个限制是单语和跨语言性能之间的权衡。句子嵌入的跨语言对齐训练会扭曲各个语言语义空间的最佳单语结构，从而损害句子嵌入在单语任务中的效用。在这项工作中，我们通过模块化训练句子编码器来解决这两个问题，即将单语专业化与跨语言对齐分开。我们首先有效地训练特定语言的句子编码器，以避免语言之间的负面干扰（即诅咒）。然后，我们通过在每个编码器上训练一个跨语言对齐适配器将所有非英语单语编码器与英语编码器对齐，从第一步开始防止对单语专业化的干扰。在这两个步骤中，我们都对机器翻译的释义数据进行对比学习。对语义文本相似性/相关性以及多项选择 QA 的单语和跨语言评估使我们的模块化解决方案比多语言句子编码器更有效，尤其有利于低资源语言。

Title: Falcon2-11B Technical Report

• Authors: Quentin Malartic, Nilabhra Roy Chowdhury, Ruxandra Cojocaru, Mugariya Farooq, Giulia Campesan, Yasser Abdelaziz Dahou Djilali, Sanath Narayan, Ankit Singh, Maksim Velikanov, Basma El Amel Boussaha, Mohammed Al-Yafeai, Hamza Alobeidli, Leen Al Qadi, Mohamed El Amine Seddik, Kirill Fedyanin, Reda Alami, Hakim Hacid
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2407.14885
• Pdf URL: https://arxiv.org/pdf/2407.14885
• Copy Paste: [[2407.14885]] Falcon2-11B Technical Report(https://arxiv.org/abs/2407.14885)
• Keywords: language model
• Abstract: We introduce Falcon2-11B, a foundation model trained on over five trillion tokens, and its multimodal counterpart, Falcon2-11B-vlm, which is a vision-to-text model. We report our findings during the training of the Falcon2-11B which follows a multi-stage approach where the early stages are distinguished by their context length and a final stage where we use a curated, high-quality dataset. Additionally, we report the effect of doubling the batch size mid-training and how training loss spikes are affected by the learning rate. The downstream performance of the foundation model is evaluated on established benchmarks, including multilingual and code datasets. The foundation model shows strong generalization across all the tasks which makes it suitable for downstream finetuning use cases. For the vision language model, we report the performance on several benchmarks and show that our model achieves a higher average score compared to open-source models of similar size. The model weights and code of both Falcon2-11B and Falcon2-11B-vlm are made available under a permissive license.
• 摘要：我们介绍了 Falcon2-11B，这是一个基于超过五万亿个标记进行训练的基础模型，以及它的多模态对应模型 Falcon2-11B-vlm，这是一个视觉到文本的模型。我们报告了在 Falcon2-11B 训练期间的发现，该方法采用多阶段方法，其中早期阶段以其上下文长度为特征，最后阶段我们使用精选的高质量数据集。此外，我们还报告了在训练中期将批次大小加倍的效果以及训练损失峰值如何受到学习率的影响。基础模型的下游性能在既定的基准上进行评估，包括多语言和代码数据集。基础模型在所有任务中都表现出很强的泛化能力，这使其适用于下游微调用例。对于视觉语言模型，我们报告了在多个基准上的性能，并表明我们的模型与类似规模的开源模型相比获得了更高的平均分数。 Falcon2-11B 和 Falcon2-11B-vlm 的模型重量和代码均在宽松的许可下提供。

Title: Improving Context-Aware Preference Modeling for Language Models

• Authors: Silviu Pitis, Ziang Xiao, Nicolas Le Roux, Alessandro Sordoni
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.14916
• Pdf URL: https://arxiv.org/pdf/2407.14916
• Copy Paste: [[2407.14916]] Improving Context-Aware Preference Modeling for Language Models(https://arxiv.org/abs/2407.14916)
• Keywords: language model, gpt
• Abstract: While finetuning language models from pairwise preferences has proven remarkably effective, the underspecified nature of natural language presents critical challenges. Direct preference feedback is uninterpretable, difficult to provide where multidimensional criteria may apply, and often inconsistent, either because it is based on incomplete instructions or provided by diverse principals. To address these challenges, we consider the two-step preference modeling procedure that first resolves the under-specification by selecting a context, and then evaluates preference with respect to the chosen context. We decompose reward modeling error according to these two steps, which suggests that supervising context in addition to context-specific preference may be a viable approach to aligning models with diverse human preferences. For this to work, the ability of models to evaluate context-specific preference is critical. To this end, we contribute context-conditioned preference datasets and accompanying experiments that investigate the ability of language models to evaluate context-specific preference. We use our datasets to (1) show that existing preference models benefit from, but fail to fully consider, added context, (2) finetune a context-aware reward model with context-specific performance exceeding that of GPT-4 and Llama 3 70B on tested datasets, and (3) investigate the value of context-aware preference modeling.
• 摘要：虽然根据成对偏好对语言模型进行微调已被证明非常有效，但自然语言的未指定性质带来了重大挑战。直接偏好反馈是不可解释的，在可能适用多维标准的情况下难以提供，并且通常不一致，因为它基于不完整的指令或由不同的主体提供。为了应对这些挑战，我们考虑了两步偏好建模程序，首先通过选择上下文来解决未指定问题，然后根据所选上下文评估偏好。我们根据这两个步骤分解奖励建模误差，这表明除了特定于上下文的偏好之外，监督上下文可能是将模型与不同的人类偏好对齐的可行方法。为了实现这一点，模型评估特定于上下文的偏好的能力至关重要。为此，我们提供了上下文条件偏好数据集和随附的实验，以研究语言模型评估特定于上下文的偏好的能力。我们使用我们的数据集来 (1) 表明现有的偏好模型受益于添加的上下文，但未能充分考虑添加的上下文；(2) 对上下文感知奖励模型进行微调，其上下文特定性能在测试数据集上超过 GPT-4 和 Llama 3 70B；(3) 研究上下文感知偏好建模的价值。

Title: Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)

• Authors: Apurv Verma, Satyapriya Krishna, Sebastian Gehrmann, Madhavan Seshadri, Anu Pradhan, Tom Ault, Leslie Barrett, David Rabinowitz, John Doucette, NhatHai Phan
• Subjects: cs.CL, cs.CR
• Abstract URL: https://arxiv.org/abs/2407.14937
• Pdf URL: https://arxiv.org/pdf/2407.14937
• Copy Paste: [[2407.14937]] Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)(https://arxiv.org/abs/2407.14937)
• Keywords: language model, llm
• Abstract: Creating secure and resilient applications with large language models (LLM) requires anticipating, adjusting to, and countering unforeseen threats. Red-teaming has emerged as a critical technique for identifying vulnerabilities in real-world LLM implementations. This paper presents a detailed threat model and provides a systematization of knowledge (SoK) of red-teaming attacks on LLMs. We develop a taxonomy of attacks based on the stages of the LLM development and deployment process and extract various insights from previous research. In addition, we compile methods for defense and practical red-teaming strategies for practitioners. By delineating prominent attack motifs and shedding light on various entry points, this paper provides a framework for improving the security and robustness of LLM-based systems.
• 摘要：使用大型语言模型 (LLM) 创建安全且有弹性的应用程序需要预测、调整和应对不可预见的威胁。红队攻击已成为识别现实世界 LLM 实现中漏洞的关键技术。本文介绍了一个详细的威胁模型，并提供了针对 LLM 的红队攻击知识系统化 (SoK)。我们根据 LLM 开发和部署过程的阶段开发了攻击分类法，并从以前的研究中提取了各种见解。此外，我们还为从业者汇编了防御方法和实用的红队攻击策略。通过描述突出的攻击主题并阐明各种切入点，本文提供了一个用于提高基于 LLM 的系统的安全性和稳健性的框架。

Title: Conversational Rubert for Detecting Competitive Interruptions in ASR-Transcribed Dialogues

• Authors: Dmitrii Galimzianov, Viacheslav Vyshegorodtsev
• Subjects: cs.CL, cs.LG, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2407.14940
• Pdf URL: https://arxiv.org/pdf/2407.14940
• Copy Paste: [[2407.14940]] Conversational Rubert for Detecting Competitive Interruptions in ASR-Transcribed Dialogues(https://arxiv.org/abs/2407.14940)
• Keywords: agent
• Abstract: Interruption in a dialogue occurs when the listener begins their speech before the current speaker finishes speaking. Interruptions can be broadly divided into two groups: cooperative (when the listener wants to support the speaker), and competitive (when the listener tries to take control of the conversation against the speaker's will). A system that automatically classifies interruptions can be used in call centers, specifically in the tasks of customer satisfaction monitoring and agent monitoring. In this study, we developed a text-based interruption classification model by preparing an in-house dataset consisting of ASR-transcribed customer support telephone dialogues in Russian. We fine-tuned Conversational RuBERT on our dataset and optimized hyperparameters, and the model performed well. With further improvements, the proposed model can be applied to automatic monitoring systems.
• 摘要：对话中断是指听话人在当前说话人结束讲话之前开始讲话。中断大致可分为两类：合作性中断（听话人想要支持说话人时）和竞争性中断（听话人试图违背说话人的意愿控制对话时）。自动分类中断的系统可用于呼叫中心，特别是在客户满意度监控和代理监控任务中。在本研究中，我们通过准备一个内部数据集（由 ASR 转录的俄语客户支持电话对话组成）开发了一个基于文本的中断分类模型。我们在数据集上对 Conversational RuBERT 进行了微调，并优化了超参数，模型表现良好。通过进一步改进，提出的模型可以应用于自动监控系统。

Title: Recent Advances in Generative AI and Large Language Models: Current Status, Challenges, and Perspectives

• Authors: Desta Haileselassie Hagos, Rick Battle, Danda B. Rawat
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.14962
• Pdf URL: https://arxiv.org/pdf/2407.14962
• Copy Paste: [[2407.14962]] Recent Advances in Generative AI and Large Language Models: Current Status, Challenges, and Perspectives(https://arxiv.org/abs/2407.14962)
• Keywords: language model, llm
• Abstract: The emergence of Generative Artificial Intelligence (AI) and Large Language Models (LLMs) has marked a new era of Natural Language Processing (NLP), introducing unprecedented capabilities that are revolutionizing various domains. This paper explores the current state of these cutting-edge technologies, demonstrating their remarkable advancements and wide-ranging applications. Our paper contributes to providing a holistic perspective on the technical foundations, practical applications, and emerging challenges within the evolving landscape of Generative AI and LLMs. We believe that understanding the generative capabilities of AI systems and the specific context of LLMs is crucial for researchers, practitioners, and policymakers to collaboratively shape the responsible and ethical integration of these technologies into various domains. Furthermore, we identify and address main research gaps, providing valuable insights to guide future research endeavors within the AI research community.
• 摘要：生成式人工智能 (AI) 和大型语言模型 (LLM) 的出现标志着自然语言处理 (NLP) 的新纪元，引入了前所未有的功能，正在彻底改变各个领域。本文探讨了这些尖端技术的现状，展示了它们的显著进步和广泛的应用。我们的论文有助于从整体上了解生成式人工智能和 LLM 不断发展的格局中的技术基础、实际应用和新兴挑战。我们相信，了解人工智能系统的生成能力和 LLM 的具体背景对于研究人员、从业者和政策制定者至关重要，这样他们才能共同塑造这些技术在各个领域的负责任和合乎道德的整合。此外，我们确定并解决了主要的研究差距，为指导人工智能研究界未来的研究工作提供了宝贵的见解。

Title: Generalization v.s. Memorization: Tracing Language Models' Capabilities Back to Pretraining Data

• Authors: Antonis Antoniades, Xinyi Wang, Yanai Elazar, Alfonso Amayuelas, Alon Albalak, Kexun Zhang, William Yang Wang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.14985
• Pdf URL: https://arxiv.org/pdf/2407.14985
• Copy Paste: [[2407.14985]] Generalization v.s. Memorization: Tracing Language Models' Capabilities Back to Pretraining Data(https://arxiv.org/abs/2407.14985)
• Keywords: language model, llm
• Abstract: Despite the proven utility of large language models (LLMs) in real-world applications, there remains a lack of understanding regarding how they leverage their large-scale pretraining text corpora to achieve such capabilities. In this work, we investigate the interplay between generalization and memorization in pretrained LLMs at scale, through a comprehensive $n$-gram analysis of their training data. Our experiments focus on three general task types: translation, question-answering, and multiple-choice reasoning. With various sizes of open-source LLMs and their pretraining corpora, we observe that as the model size increases, the task-relevant $n$-gram pair data becomes increasingly important, leading to improved task performance, decreased memorization, stronger generalization, and emergent abilities. Our results support the hypothesis that LLMs' capabilities emerge from a delicate balance of memorization and generalization with sufficient task-related pretraining data, and point the way to larger-scale analyses that could further improve our understanding of these models.
• 摘要：尽管大型语言模型 (LLM) 在实际应用中已被证明具有实用性，但对于它们如何利用大规模预训练文本语料库来实现这种功能，人们仍然缺乏了解。在这项工作中，我们通过对训练数据进行全面的 $n$-gram 分析，研究了大规模预训练 LLM 中泛化和记忆之间的相互作用。我们的实验重点关注三种一般任务类型：翻译、问答和多项选择推理。通过各种规模的开源 LLM 及其预训练语料库，我们观察到，随着模型规模的增加，与任务相关的 $n$-gram 对数据变得越来越重要，从而提高了任务性能，减少了记忆，增强了泛化能力并产生了新能力。我们的结果支持了以下假设：LLM 的能力源于记忆和泛化的微妙平衡，以及足够的任务相关预训练数据，并为更大规模的分析指明了方向，从而可以进一步提高我们对这些模型的理解。

Title: Knowledge Mechanisms in Large Language Models: A Survey and Perspective

• Authors: Mengru Wang, Yunzhi Yao, Ziwen Xu, Shuofei Qiao, Shumin Deng, Peng Wang, Xiang Chen, Jia-Chen Gu, Yong Jiang, Pengjun Xie, Fei Huang, Huajun Chen, Ningyu Zhang
• Subjects: cs.CL, cs.AI, cs.CV, cs.HC, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.15017
• Pdf URL: https://arxiv.org/pdf/2407.15017
• Copy Paste: [[2407.15017]] Knowledge Mechanisms in Large Language Models: A Survey and Perspective(https://arxiv.org/abs/2407.15017)
• Keywords: language model, llm
• Abstract: Understanding knowledge mechanisms in Large Language Models (LLMs) is crucial for advancing towards trustworthy AGI. This paper reviews knowledge mechanism analysis from a novel taxonomy including knowledge utilization and evolution. Knowledge utilization delves into the mechanism of memorization, comprehension and application, and creation. Knowledge evolution focuses on the dynamic progression of knowledge within individual and group LLMs. Moreover, we discuss what knowledge LLMs have learned, the reasons for the fragility of parametric knowledge, and the potential dark knowledge (hypothesis) that will be challenging to address. We hope this work can help understand knowledge in LLMs and provide insights for future research.
• 摘要：理解大型语言模型 (LLM) 中的知识机制对于实现值得信赖的 AGI 至关重要。本文从包括知识利用和演化在内的新分类法回顾了知识机制分析。知识利用深入研究记忆、理解和应用以及创造的机制。知识演化侧重于个人和团体 LLM 中知识的动态发展。此外，我们讨论了 LLM 学到了什么知识、参数知识脆弱的原因以及可能难以解决的暗知识（假设）。我们希望这项工作可以帮助理解 LLM 中的知识并为未来的研究提供见解。

Title: Answer, Assemble, Ace: Understanding How Transformers Answer Multiple Choice Questions

• Authors: Sarah Wiegreffe, Oyvind Tafjord, Yonatan Belinkov, Hannaneh Hajishirzi, Ashish Sabharwal
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.15018
• Pdf URL: https://arxiv.org/pdf/2407.15018
• Copy Paste: [[2407.15018]] Answer, Assemble, Ace: Understanding How Transformers Answer Multiple Choice Questions(https://arxiv.org/abs/2407.15018)
• Keywords: language model
• Abstract: Multiple-choice question answering (MCQA) is a key competence of performant transformer language models that is tested by mainstream benchmarks. However, recent evidence shows that models can have quite a range of performance, particularly when the task format is diversified slightly (such as by shuffling answer choice order). In this work we ask: how do successful models perform formatted MCQA? We employ vocabulary projection and activation patching methods to localize key hidden states that encode relevant information for predicting the correct answer. We find that prediction of a specific answer symbol is causally attributed to a single middle layer, and specifically its multi-head self-attention mechanism. We show that subsequent layers increase the probability of the predicted answer symbol in vocabulary space, and that this probability increase is associated with a sparse set of attention heads with unique roles. We additionally uncover differences in how different models adjust to alternative symbols. Finally, we demonstrate that a synthetic task can disentangle sources of model error to pinpoint when a model has learned formatted MCQA, and show that an inability to separate answer symbol tokens in vocabulary space is a property of models unable to perform formatted MCQA tasks.
• 摘要：多项选择题回答 (MCQA) 是高性能 Transformer 语言模型的一项关键能力，主流基准对其进行了测试。然而，最近的证据表明，模型的性能范围可能相当广泛，特别是当任务格式略有不同时（例如通过打乱答案选择顺序）。在这项工作中，我们提出一个问题：成功的模型如何执行格式化的 MCQA？我们采用词汇投影和激活修补方法来定位关键隐藏状态，这些隐藏状态编码了与预测正确答案相关的信息。我们发现，对特定答案符号的预测归因于单个中间层，特别是其多头自注意力机制。我们表明，后续层增加了词汇空间中预测答案符号的概率，并且这种概率的增加与一组具有独特角色的稀疏注意力头有关。我们还发现了不同模型如何适应替代符号的差异。最后，我们证明，综合任务可以解开模型错误源，以确定模型何时学习了格式化的 MCQA，并表明无法在词汇空间中分离答案符号标记是无法执行格式化 MCQA 任务的模型的属性。

Title: Enhancing Incremental Summarization with Structured Representations

• Authors: EunJeong Hwang, Yichao Zhou, James Bradley Wendt, Beliz Gunel, Nguyen Vo, Jing Xie, Sandeep Tata
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.15021
• Pdf URL: https://arxiv.org/pdf/2407.15021
• Copy Paste: [[2407.15021]] Enhancing Incremental Summarization with Structured Representations(https://arxiv.org/abs/2407.15021)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) often struggle with processing extensive input contexts, which can lead to redundant, inaccurate, or incoherent summaries. Recent methods have used unstructured memory to incrementally process these contexts, but they still suffer from information overload due to the volume of unstructured data handled. In our study, we introduce structured knowledge representations ($GU_{json}$), which significantly improve summarization performance by 40% and 14% across two public datasets. Most notably, we propose the Chain-of-Key strategy ($CoK_{json}$) that dynamically updates or augments these representations with new information, rather than recreating the structured memory for each new source. This method further enhances performance by 7% and 4% on the datasets.
• 摘要：大型语言模型 (LLM) 通常难以处理大量输入上下文，这可能导致冗余、不准确或不连贯的摘要。最近的方法使用非结构化内存逐步处理这些上下文，但由于处理的非结构化数据量太大，它们仍然会遭受信息过载。在我们的研究中，我们引入了结构化知识表示 ($GU_{json}$)，这在两个公共数据集上显著提高了 40% 和 14% 的摘要性能。最值得注意的是，我们提出了 Chain-of-Key 策略 ($CoK_{json}$)，该策略使用新信息动态更新或扩充这些表示，而不是为每个新源重新创建结构化内存。此方法进一步将数据集上的性能提高了 7% 和 4%。

Title: DOPRA: Decoding Over-accumulation Penalization and Re-allocation in Specific Weighting Layer

• Authors: Jinfeng Wei, Xiaofeng Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.15130
• Pdf URL: https://arxiv.org/pdf/2407.15130
• Copy Paste: [[2407.15130]] DOPRA: Decoding Over-accumulation Penalization and Re-allocation in Specific Weighting Layer(https://arxiv.org/abs/2407.15130)
• Keywords: language model, llm, hallucination
• Abstract: In this work, we introduce DOPRA, a novel approach designed to mitigate hallucinations in multi-modal large language models (MLLMs). Unlike existing solutions that typically involve costly supplementary training data or the integration of external knowledge sources, DOPRA innovatively addresses hallucinations by decoding specific weighted layer penalties and redistribution, offering an economical and effective solution without additional resources. DOPRA is grounded in unique insights into the intrinsic mechanisms controlling hallucinations within MLLMs, especially the models' tendency to over-rely on a subset of summary tokens in the self-attention matrix, neglecting critical image-related information. This phenomenon is particularly pronounced in certain strata. To counteract this over-reliance, DOPRA employs a strategy of weighted overlay penalties and redistribution in specific layers, such as the 12th layer, during the decoding process. Furthermore, DOPRA includes a retrospective allocation process that re-examines the sequence of generated tokens, allowing the algorithm to reallocate token selection to better align with the actual image content, thereby reducing the incidence of hallucinatory descriptions in auto-generated captions. Overall, DOPRA represents a significant step forward in improving the output quality of MLLMs by systematically reducing hallucinations through targeted adjustments during the decoding process.
• 摘要：在这项工作中，我们介绍了 DOPRA，这是一种旨在减轻多模态大型语言模型 (MLLM) 中的幻觉的新方法。与通常涉及昂贵的补充训练数据或集成外部知识源的现有解决方案不同，DOPRA 通过解码特定的加权层惩罚和重新分配创新地解决了幻觉问题，提供了一种经济有效的解决方案，无需额外资源。DOPRA 基于对控制 MLLM 中幻觉的内在机制的独特见解，尤其是模型倾向于过度依赖自注意力矩阵中的摘要标记子集，而忽略关键的图像相关信息。这种现象在某些层面尤为明显。为了抵消这种过度依赖，DOPRA 在解码过程中采用了加权叠加惩罚和在特定层（例如第 12 层）重新分配的策略。此外，DOPRA 还包含一个回顾性分配过程，该过程会重新检查生成的标记序列，从而使算法能够重新分配标记选择以更好地与实际图像内容保持一致，从而减少自动生成的字幕中出现幻觉描述的概率。总体而言，DOPRA 通过在解码过程中进行有针对性的调整，系统地减少幻觉，代表了在提高 MLLM 输出质量方面迈出的重要一步。

Title: A multi-level multi-label text classification dataset of 19th century Ottoman and Russian literary and critical texts

• Authors: Gokcen Gokceoglu, Devrim Cavusoglu, Emre Akbas, Özen Nergis Dolcerocca
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.15136
• Pdf URL: https://arxiv.org/pdf/2407.15136
• Copy Paste: [[2407.15136]] A multi-level multi-label text classification dataset of 19th century Ottoman and Russian literary and critical texts(https://arxiv.org/abs/2407.15136)
• Keywords: language model, llm
• Abstract: This paper introduces a multi-level, multi-label text classification dataset comprising over 3000 documents. The dataset features literary and critical texts from 19th-century Ottoman Turkish and Russian. It is the first study to apply large language models (LLMs) to this dataset, sourced from prominent literary periodicals of the era. The texts have been meticulously organized and labeled. This was done according to a taxonomic framework that takes into account both their structural and semantic attributes. Articles are categorized and tagged with bibliometric metadata by human experts. We present baseline classification results using a classical bag-of-words (BoW) naive Bayes model and three modern LLMs: multilingual BERT, Falcon, and Llama-v2. We found that in certain cases, Bag of Words (BoW) outperforms Large Language Models (LLMs), emphasizing the need for additional research, especially in low-resource language settings. This dataset is expected to be a valuable resource for researchers in natural language processing and machine learning, especially for historical and low-resource languages. The dataset is publicly available^1.
• 摘要：本文介绍了一个包含 3000 多篇文档的多级、多标签文本分类数据集。该数据集包含 19 世纪奥斯曼土耳其语和俄语的文学和批评文本。这是首次将大型语言模型 (LLM) 应用于该数据集的研究，该数据集来源于当时著名的文学期刊。这些文本经过精心组织和标记。这是根据一个同时考虑其结构和语义属性的分类框架完成的。文章由人类专家分类并使用文献计量元数据标记。我们使用经典的词袋 (BoW) 朴素贝叶斯模型和三种现代 LLM（多语言 BERT、Falcon 和 Llama-v2）展示了基线分类结果。我们发现在某些情况下，词袋 (BoW) 的表现优于大型语言模型 (LLM)，这强调了需要进行进一步的研究，特别是在资源匮乏的语言环境中。该数据集有望成为自然语言处理和机器学习研究人员的宝贵资源，尤其是对于历史语言和低资源语言而言。该数据集已公开发布^1。

Title: Fine-grained Gender Control in Machine Translation with Large Language Models

• Authors: Minwoo Lee, Hyukhun Koh, Minsung Kim, Kyomin Jung
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.15154
• Pdf URL: https://arxiv.org/pdf/2407.15154
• Copy Paste: [[2407.15154]] Fine-grained Gender Control in Machine Translation with Large Language Models(https://arxiv.org/abs/2407.15154)
• Keywords: language model, llm, prompt
• Abstract: In machine translation, the problem of ambiguously gendered input has been pointed out, where the gender of an entity is not available in the source sentence. To address this ambiguity issue, the task of controlled translation that takes the gender of the ambiguous entity as additional input have been proposed. However, most existing works have only considered a simplified setup of one target gender for input. In this paper, we tackle controlled translation in a more realistic setting of inputs with multiple entities and propose Gender-of-Entity (GoE) prompting method for LLMs. Our proposed method instructs the model with fine-grained entity-level gender information to translate with correct gender inflections. By utilizing four evaluation benchmarks, we investigate the controlled translation capability of LLMs in multiple dimensions and find that LLMs reach state-of-the-art performance in controlled translation. Furthermore, we discover an emergence of gender interference phenomenon when controlling the gender of multiple entities. Finally, we address the limitations of existing gender accuracy evaluation metrics and propose leveraging LLMs as an evaluator for gender inflection in machine translation.
• 摘要：在机器翻译中，已经指出了性别模糊输入的问题，即源语句中无法提供实体的性别。为了解决这个模糊问题，已经提出了受控翻译任务，即将模糊实体的性别作为附加输入。然而，大多数现有工作仅考虑了一种目标性别输入的简化设置。在本文中，我们在更现实的多个实体输入设置中处理受控翻译，并提出了 LLM 的实体性别 (GoE) 提示方法。我们提出的方法使用细粒度的实体级性别信息指导模型以正确的性别词形变化进行翻译。通过利用四个评估基准，我们从多个维度研究了 LLM 的受控翻译能力，发现 LLM 在受控翻译中达到了最佳性能。此外，我们发现在控制多个实体的性别时会出现性别干扰现象。最后，我们解决了现有性别准确性评估指标的局限性，并提出利用 LLM 作为机器翻译中性别词形变化的评估器。

Title: When Can Transformers Count to n?

• Authors: Gilad Yehudai, Haim Kaplan, Asma Ghandeharioun, Mor Geva, Amir Globerson
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.15160
• Pdf URL: https://arxiv.org/pdf/2407.15160
• Copy Paste: [[2407.15160]] When Can Transformers Count to n?(https://arxiv.org/abs/2407.15160)
• Keywords: language model
• Abstract: Large language models based on the transformer architectures can solve highly complex tasks. But are there simple tasks that such models cannot solve? Here we focus on very simple counting tasks, that involve counting how many times a token in the vocabulary have appeared in a string. We show that if the dimension of the transformer state is linear in the context length, this task can be solved. However, the solution we propose does not scale beyond this limit, and we provide theoretical arguments for why it is likely impossible for a size limited transformer to implement this task. Our empirical results demonstrate the same phase-transition in performance, as anticipated by the theoretical argument. Our results demonstrate the importance of understanding how transformers can solve simple tasks.
• 摘要：基于 Transformer 架构的大型语言模型可以解决高度复杂的任务。但是，是否存在此类模型无法解决的简单任务？在这里，我们专注于非常简单的计数任务，这些任务涉及计算词汇表中的标记在字符串中出现的次数。我们表明，如果 Transformer 状态的维度与上下文长度成线性关系，则可以解决此任务。但是，我们提出的解决方案无法超越此限制，并且我们提供了理论论据，说明为什么尺寸有限的 Transformer 可能无法实现此任务。我们的实证结果表明，性能发生了与理论论证预期相同的相变。我们的结果表明，了解 Transformer 如何解决简单任务非常重要。

Title: Farewell to Length Extrapolation, a Training-Free Infinite Context with Finite Attention Scope

• Authors: Xiaoran Liu, Qipeng Guo, Yuerong Song, Zhigeng Liu, Kai Lv, Hang Yan, Linlin Li, Qun Liu, Xipeng Qiu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.15176
• Pdf URL: https://arxiv.org/pdf/2407.15176
• Copy Paste: [[2407.15176]] Farewell to Length Extrapolation, a Training-Free Infinite Context with Finite Attention Scope(https://arxiv.org/abs/2407.15176)
• Keywords: language model, llm
• Abstract: The maximum supported context length is a critical bottleneck limiting the practical application of the Large Language Model (LLM). Although existing length extrapolation methods can extend the context of LLMs to millions of tokens, these methods all have an explicit upper bound. In this work, we propose LongCache, a training-free approach that enables LLM to support an infinite context with finite context scope, through full-context cache selection and training-free integration. This effectively frees LLMs from the length extrapolation issue. We validate LongCache on the LongBench and L-Eval and demonstrate its performance is on par with traditional full-attention mechanisms. Furthermore, we have applied LongCache on mainstream LLMs, including LLaMA3 and Mistral-v0.3, enabling them to support context lengths of at least 400K in Needle-In-A-Haystack tests. We will improve the efficiency of LongCache by GPU-aware optimization soon.
• 摘要：上下文支持的最大长度是限制大型语言模型 (LLM) 实际应用的关键瓶颈。虽然现有的长度外推方法可以将 LLM 的上下文扩展到数百万个 token，但这些方法都存在明确的上限。在这项工作中，我们提出了 LongCache，这是一种无需训练的方法，通过全上下文缓存选择和无需训练的集成，使 LLM 能够支持具有有限上下文范围的无限上下文。这有效地将 LLM 摆脱了长度外推的问题。我们在 LongBench 和 L-Eval 上验证了 LongCache，并证明其性能与传统的全注意力机制相当。此外，我们已经将 LongCache 应用于主流 LLM，包括 LLaMA3 和 Mistral-v0.3，使它们能够在 Needle-In-A-Haystack 测试中支持至少 400K 的上下文长度。我们将很快通过 GPU 感知优化来提高 LongCache 的效率。

Title: A Survey on Employing Large Language Models for Text-to-SQL Tasks

• Authors: Liang Shi, Zhengju Tang, Zhi Yang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.15186
• Pdf URL: https://arxiv.org/pdf/2407.15186
• Copy Paste: [[2407.15186]] A Survey on Employing Large Language Models for Text-to-SQL Tasks(https://arxiv.org/abs/2407.15186)
• Keywords: language model, llm, prompt
• Abstract: The increasing volume of data stored in relational databases has led to the need for efficient querying and utilization of this data in various sectors. However, writing SQL queries requires specialized knowledge, which poses a challenge for non-professional users trying to access and query databases. Text-to-SQL parsing solves this issue by converting natural language queries into SQL queries, thus making database access more accessible for non-expert users. To take advantage of the recent developments in Large Language Models (LLMs), a range of new methods have emerged, with a primary focus on prompt engineering and fine-tuning. This survey provides a comprehensive overview of LLMs in text-to-SQL tasks, discussing benchmark datasets, prompt engineering, fine-tuning methods, and future research directions. We hope this review will enable readers to gain a broader understanding of the recent advances in this field and offer some insights into its future trajectory.
• 摘要：关系数据库中存储的数据量不断增加，导致各个领域都需要高效查询和利用这些数据。但是，编写 SQL 查询需要专业知识，这对尝试访问和查询数据库的非专业用户来说是一个挑战。文本到 SQL 解析通过将自然语言查询转换为 SQL 查询解决了这个问题，从而使非专家用户更容易访问数据库。为了利用大型语言模型 (LLM) 的最新发展，出现了一系列新方法，主要侧重于快速工程和微调。本综述全面概述了文本到 SQL 任务中的 LLM，讨论了基准数据集、快速工程、微调方法和未来的研究方向。我们希望这篇评论能让读者更广泛地了解该领域的最新进展，并对其未来发展提供一些见解。

Title: When Do Universal Image Jailbreaks Transfer Between Vision-Language Models?

• Authors: Rylan Schaeffer, Dan Valentine, Luke Bailey, James Chua, Cristóbal Eyzaguirre, Zane Durante, Joe Benton, Brando Miranda, Henry Sleight, John Hughes, Rajashree Agrawal, Mrinank Sharma, Scott Emmons, Sanmi Koyejo, Ethan Perez
• Subjects: cs.CL, cs.AI, cs.CR, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.15211
• Pdf URL: https://arxiv.org/pdf/2407.15211
• Copy Paste: [[2407.15211]] When Do Universal Image Jailbreaks Transfer Between Vision-Language Models?(https://arxiv.org/abs/2407.15211)
• Keywords: language model
• Abstract: The integration of new modalities into frontier AI systems offers exciting capabilities, but also increases the possibility such systems can be adversarially manipulated in undesirable ways. In this work, we focus on a popular class of vision-language models (VLMs) that generate text outputs conditioned on visual and textual inputs. We conducted a large-scale empirical study to assess the transferability of gradient-based universal image "jailbreaks" using a diverse set of over 40 open-parameter VLMs, including 18 new VLMs that we publicly release. Overall, we find that transferable gradient-based image jailbreaks are extremely difficult to obtain. When an image jailbreak is optimized against a single VLM or against an ensemble of VLMs, the jailbreak successfully jailbreaks the attacked VLM(s), but exhibits little-to-no transfer to any other VLMs; transfer is not affected by whether the attacked and target VLMs possess matching vision backbones or language models, whether the language model underwent instruction-following and/or safety-alignment training, or many other factors. Only two settings display partially successful transfer: between identically-pretrained and identically-initialized VLMs with slightly different VLM training data, and between different training checkpoints of a single VLM. Leveraging these results, we then demonstrate that transfer can be significantly improved against a specific target VLM by attacking larger ensembles of "highly-similar" VLMs. These results stand in stark contrast to existing evidence of universal and transferable text jailbreaks against language models and transferable adversarial attacks against image classifiers, suggesting that VLMs may be more robust to gradient-based transfer attacks.
• 摘要：将新模式集成到前沿 AI 系统中提供了令人兴奋的功能，但也增加了此类系统以不良方式被对手操纵的可能性。在这项工作中，我们专注于一类流行的视觉语言模型 (VLM)，该模型根据视觉和文本输入生成文本输出。我们进行了一项大规模实证研究，以评估基于梯度的通用图像“越狱”的可转移性，使用超过 40 个开放参数 VLM，包括我们公开发布的 18 个新 VLM。总体而言，我们发现可转移的基于梯度的图像越狱极难获得。当针对单个 VLM 或一组 VLM 优化图像越狱时，越狱成功越狱了被攻击的 VLM，但几乎没有或根本没有向任何其他 VLM 转移；迁移不受受攻击和目标 VLM 是否拥有匹配的视觉主干或语言模型、语言模型是否经过指令跟踪和/或安全对齐训练或许多其他因素的影响。只有两种设置显示部分成功的迁移：在具有略微不同的 VLM 训练数据的相同预训练和相同初始化的 VLM 之间，以及在单个 VLM 的不同训练检查点之间。利用这些结果，我们随后证明，通过攻击更大的“高度相似”VLM 集合，可以显著改善针对特定目标 VLM 的迁移。这些结果与现有的针对语言模型的通用和可迁移文本越狱以及针对图像分类器的可迁移对抗攻击的证据形成鲜明对比，表明 VLM 可能对基于梯度的迁移攻击更具鲁棒性。

Title: A Community-Centric Perspective for Characterizing and Detecting Anti-Asian Violence-Provoking Speech

• Authors: Gaurav Verma, Rynaa Grover, Jiawei Zhou, Binny Mathew, Jordan Kraemer, Munmun De Choudhury, Srijan Kumar
• Subjects: cs.CL, cs.SI
• Abstract URL: https://arxiv.org/abs/2407.15227
• Pdf URL: https://arxiv.org/pdf/2407.15227
• Copy Paste: [[2407.15227]] A Community-Centric Perspective for Characterizing and Detecting Anti-Asian Violence-Provoking Speech(https://arxiv.org/abs/2407.15227)
• Keywords: llm
• Abstract: Violence-provoking speech -- speech that implicitly or explicitly promotes violence against the members of the targeted community, contributed to a massive surge in anti-Asian crimes during the pandemic. While previous works have characterized and built tools for detecting other forms of harmful speech, like fear speech and hate speech, our work takes a community-centric approach to studying anti-Asian violence-provoking speech. Using data from ~420k Twitter posts spanning a 3-year duration (January 1, 2020 to February 1, 2023), we develop a codebook to characterize anti-Asian violence-provoking speech and collect a community-crowdsourced dataset to facilitate its large-scale detection using state-of-the-art classifiers. We contrast the capabilities of natural language processing classifiers, ranging from BERT-based to LLM-based classifiers, in detecting violence-provoking speech with their capabilities to detect anti-Asian hateful speech. In contrast to prior work that has demonstrated the effectiveness of such classifiers in detecting hateful speech ($F_1 = 0.89$), our work shows that accurate and reliable detection of violence-provoking speech is a challenging task ($F_1 = 0.69$). We discuss the implications of our findings, particularly the need for proactive interventions to support Asian communities during public health crises. The resources related to the study are available at this https URL.
• 摘要：煽动暴力的言论——以明示或暗示的方式宣扬针对目标社区成员的暴力行为的言论，导致了疫情期间反亚裔犯罪的激增。虽然之前的研究已经描述了并构建了用于检测其他形式的有害言论（如恐惧言论和仇恨言论）的工具，但我们的工作采取以社区为中心的方法来研究煽动暴力的反亚裔言论。利用 3 年期间（2020 年 1 月 1 日至 2023 年 2 月 1 日）约 42 万条 Twitter 帖子的数据，我们开发了一个代码本来描述煽动暴力的反亚裔言论，并收集了一个社区众包数据集，以便使用最先进的分类器进行大规模检测。我们对比了自然语言处理分类器（从基于 BERT 的分类器到基于 LLM 的分类器）检测煽动暴力的言论的能力与检测反亚裔仇恨言论的能力。与之前的研究相比，这些研究已经证明了此类分类器在检测仇恨言论方面的有效性 ($F_1 = 0.89$)，而我们的研究则表明，准确可靠地检测煽动暴力的言论是一项具有挑战性的任务 ($F_1 = 0.69$)。我们讨论了我们的研究结果的含义，特别是在公共卫生危机期间采取主动干预措施以支持亚裔社区的必要性。与该研究相关的资源可在此 https URL 上找到。

Title: The Hitchhiker's Guide to Human Alignment with *PO

• Authors: Kian Ahrabian, Xihui Lin, Barun Patra, Vishrav Chaudhary, Alon Benhaim, Jay Pujara, Xia Song
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.15229
• Pdf URL: https://arxiv.org/pdf/2407.15229
• Copy Paste: [[2407.15229]] The Hitchhiker's Guide to Human Alignment with *PO(https://arxiv.org/abs/2407.15229)
• Keywords: language model, llm
• Abstract: With the growing utilization of large language models (LLMs) across domains, alignment towards human preferences has become one of the most critical aspects of training models. At the forefront of state-of-the-art human alignment methods are preference optimization methods (*PO). However, prior research has often concentrated on identifying the best-performing method, typically involving a grid search over hyperparameters, which can be impractical for general practitioners. In this paper, we aim to identify the algorithm that, while being performant, is simultaneously more robust to varying hyperparameters, thereby increasing the likelihood of achieving better results. We focus on a realistic out-of-distribution (OOD) scenario that mirrors real-world applications of human alignment, offering practical insights into the strengths and weaknesses of these methods. Furthermore, to better understand the shortcomings of generations from the different methods, we analyze the model generations through the lens of KL divergence of the SFT model and the response length statistics. Our analysis reveals that the widely adopted DPO method consistently produces lengthy responses of inferior quality that are very close to the SFT responses. Motivated by these findings, we propose an embarrassingly simple extension to the DPO algorithm, LN-DPO, resulting in more concise responses without sacrificing quality compared to the policy obtained by vanilla DPO.
• 摘要：随着大型语言模型 (LLM) 在各个领域的应用越来越广泛，与人类偏好保持一致已成为训练模型最关键的方面之一。偏好优化方法 (*PO) 是最先进的人类对齐方法的前沿。然而，先前的研究通常集中在确定性能最佳的方法上，通常涉及对超参数进行网格搜索，这对于一般从业者来说可能不切实际。在本文中，我们旨在确定一种算法，该算法在性能高的同时，对不同的超参数更具鲁棒性，从而增加获得更好结果的可能性。我们专注于一个现实的分布外 (OOD) 场景，该场景反映了人类对齐的实际应用，为这些方法的优缺点提供了实用的见解。此外，为了更好地理解不同方法的代际缺点，我们通过 SFT 模型的 KL 散度和响应长度统计来分析模型代际。我们的分析表明，广泛采用的 DPO 方法始终产生质量较差的长响应，非常接近 SFT 响应。受这些发现的启发，我们提出了一种非常简单的 DPO 算法扩展 LN-DPO，与 vanilla DPO 获得的策略相比，它可以在不牺牲质量的情况下获得更简洁的响应。

Title: TAGCOS: Task-agnostic Gradient Clustered Coreset Selection for Instruction Tuning Data

• Authors: Jipeng Zhang, Yaxuan Qin, Renjie Pi, Weizhong Zhang, Rui Pan, Tong Zhang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.15235
• Pdf URL: https://arxiv.org/pdf/2407.15235
• Copy Paste: [[2407.15235]] TAGCOS: Task-agnostic Gradient Clustered Coreset Selection for Instruction Tuning Data(https://arxiv.org/abs/2407.15235)
• Keywords: language model, chat
• Abstract: Instruction tuning has achieved unprecedented success in NLP, turning large language models into versatile chatbots. However, the increasing variety and volume of instruction datasets demand significant computational resources. To address this, it is essential to extract a small and highly informative subset (i.e., Coreset) that achieves comparable performance to the full dataset. Achieving this goal poses non-trivial challenges: 1) data selection requires accurate data representations that reflect the training samples' quality, 2) considering the diverse nature of instruction datasets, and 3) ensuring the efficiency of the coreset selection algorithm for large models. To address these challenges, we propose Task-Agnostic Gradient Clustered COreset Selection (TAGCOS). Specifically, we leverage sample gradients as the data representations, perform clustering to group similar data, and apply an efficient greedy algorithm for coreset selection. Experimental results show that our algorithm, selecting only 5% of the data, surpasses other unsupervised methods and achieves performance close to that of the full dataset.
• 摘要：指令调优在 NLP 领域取得了前所未有的成功，将大型语言模型变成了多功能聊天机器人。然而，指令数据集的种类和数量不断增加，需要大量的计算资源。为了解决这个问题，必须提取一个小而信息量丰富的子集（即核心集），以实现与完整数据集相当的性能。实现这一目标带来了不小的挑战：1）数据选择需要准确的数据表示来反映训练样本的质量，2）考虑指令数据集的多样性，3）确保大型模型的核心集选择算法的效率。为了应对这些挑战，我们提出了与任务无关的梯度聚类核心集选择 (TAGCOS)。具体来说，我们利用样本梯度作为数据表示，执行聚类以对相似数据进行分组，并应用高效的贪婪算法进行核心集选择。实验结果表明，我们的算法仅选择 5% 的数据，就超越了其他无监督方法，并实现了接近完整数据集的性能。

Title: XAI meets LLMs: A Survey of the Relation between Explainable AI and Large Language Models

• Authors: Erik Cambria, Lorenzo Malandri, Fabio Mercorio, Navid Nobani, Andrea Seveso
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.15248
• Pdf URL: https://arxiv.org/pdf/2407.15248
• Copy Paste: [[2407.15248]] XAI meets LLMs: A Survey of the Relation between Explainable AI and Large Language Models(https://arxiv.org/abs/2407.15248)
• Keywords: language model, llm
• Abstract: In this survey, we address the key challenges in Large Language Models (LLM) research, focusing on the importance of interpretability. Driven by increasing interest from AI and business sectors, we highlight the need for transparency in LLMs. We examine the dual paths in current LLM research and eXplainable Artificial Intelligence (XAI): enhancing performance through XAI and the emerging focus on model interpretability. Our paper advocates for a balanced approach that values interpretability equally with functional advancements. Recognizing the rapid development in LLM research, our survey includes both peer-reviewed and preprint (arXiv) papers, offering a comprehensive overview of XAI's role in LLM research. We conclude by urging the research community to advance both LLM and XAI fields together.
• 摘要：在本次调查中，我们解决了大型语言模型 (LLM) 研究中的关键挑战，重点关注可解释性的重要性。在人工智能和商业领域日益增长的兴趣的推动下，我们强调了 LLM 透明度的必要性。我们研究了当前 LLM 研究和可解释人工智能 (XAI) 中的双重路径：通过 XAI 提高性能，并开始关注模型可解释性。我们的论文提倡一种平衡的方法，即在重视可解释性和功能改进的同时，也同样重视可解释性。认识到 LLM 研究的快速发展，我们的调查包括同行评审和预印本 (arXiv) 论文，全面概述了 XAI 在 LLM 研究中的作用。最后，我们敦促研究界共同推进 LLM 和 XAI 领域。

Title: SynCPKL: Harnessing LLMs to Generate Synthetic Data for Commonsense Persona Knowledge Linking

• Authors: Kuan-Yen Lin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.15281
• Pdf URL: https://arxiv.org/pdf/2407.15281
• Copy Paste: [[2407.15281]] SynCPKL: Harnessing LLMs to Generate Synthetic Data for Commonsense Persona Knowledge Linking(https://arxiv.org/abs/2407.15281)
• Keywords: language model, llm
• Abstract: Understanding rich dialogues often requires NLP systems to access relevant commonsense persona knowledge, but retrieving this knowledge is challenging due to complex contexts and the implicit nature of commonsense. This paper presents our approach to the Commonsense Persona Knowledge Linking (CPKL) challenge, addressing the critical need for integrating persona and commonsense knowledge in open-domain dialogue systems. We introduce SynCPKL Pipeline, a pipeline that leverages Large Language Models to generate high-quality synthetic datasets for training commonsense persona knowledge linkers. To demonstrate the efficacy of our approach, we present SynCPKL, a new dataset specifically designed for this task. Our experiments validate the effectiveness of SynCPKL for training commonsense persona knowledge linkers. Additionally, our top-performing model, Derberta-SynCPKL, secured first place in the CPKL challenge by a 16% improvement in F1 score. We released both SynCPKL and Derberta-SynCPKL at this https URL.
• 摘要：理解丰富的对话通常需要 NLP 系统访问相关的常识角色知识，但由于复杂的上下文和常识的隐性，检索这些知识具有挑战性。本文介绍了我们应对常识角色知识链接 (CPKL) 挑战的方法，解决了在开放域对话系统中整合角色和常识知识的关键需求。我们引入了 SynCPKL 管道，这是一个利用大型语言模型生成高质量合成数据集以训练常识角色知识链接器的管道。为了证明我们方法的有效性，我们提出了 SynCPKL，这是一个专门为这项任务设计的新数据集。我们的实验验证了 SynCPKL 在训练常识角色知识链接器方面的有效性。此外，我们表现最佳的模型 Derberta-SynCPKL 在 CPKL 挑战赛中获得了第一名，F1 分数提高了 16%。我们在此 https URL 上发布了 SynCPKL 和 Derberta-SynCPKL。

Title: Intrinsic Self-correction for Enhanced Morality: An Analysis of Internal Mechanisms and the Superficial Hypothesis

• Authors: Guangliang Liu, Haitao Mao, Jiliang Tang, Kristen Marie Johnson
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.15286
• Pdf URL: https://arxiv.org/pdf/2407.15286
• Copy Paste: [[2407.15286]] Intrinsic Self-correction for Enhanced Morality: An Analysis of Internal Mechanisms and the Superficial Hypothesis(https://arxiv.org/abs/2407.15286)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) are capable of producing content that perpetuates stereotypes, discrimination, and toxicity. The recently proposed moral self-correction is a computationally efficient method for reducing harmful content in the responses of LLMs. However, the process of how injecting self-correction instructions can modify the behavior of LLMs remains under-explored. In this paper, we explore the effectiveness of moral self-correction by answering three research questions: (1) In what scenarios does moral self-correction work? (2) What are the internal mechanisms of LLMs, e.g., hidden states, that are influenced by moral self-correction instructions? (3) Is intrinsic moral self-correction actually superficial? We argue that self-correction can help LLMs find a shortcut to more morally correct output, rather than truly reducing the immorality stored in hidden states. Through empirical investigation with tasks of language generation and multi-choice question answering, we conclude: (i) LLMs exhibit good performance across both tasks, and self-correction instructions are particularly beneficial when the correct answer is already top-ranked; (ii) The morality levels in intermediate hidden states are strong indicators as to whether one instruction would be more effective than another; (iii) Based on our analysis of intermediate hidden states and task case studies of self-correction behaviors, we are first to propose the hypothesis that intrinsic moral self-correction is in fact superficial.
• 摘要：大型语言模型 (LLM) 能够产生延续刻板印象、歧视和毒性的内容。最近提出的道德自我纠正是一种计算效率高的方法，可以减少 LLM 回复中的有害内容。然而，注入自我纠正指令如何改变 LLM 行为的过程仍未得到充分探索。在本文中，我们通过回答三个研究问题来探索道德自我纠正的有效性：（1）道德自我纠正在什么场景下起作用？（2）LLM 的哪些内部机制（例如隐藏状态）受道德自我纠正指令的影响？（3）内在的道德自我纠正真的是肤浅的吗？我们认为自我纠正可以帮助 LLM 找到一条捷径来获得更符合道德的输出，而不是真正减少隐藏状态中存储的不道德行为。通过对语言生成和多项选择题回答任务的实证研究，我们得出结论：（i）LLM 在两个任务中都表现出良好的表现，并且当正确答案已经排在首位时，自我纠正指令尤其有益；（ii）中间隐藏状态下的道德水平可以有力地表明一条指令是否比另一条指令更有效；（iii）基于我们对中间隐藏状态的分析和自我纠正行为的任务案例研究，我们首次提出内在道德自我纠正实际上是肤浅的假设。

Title: ZZU-NLP at SIGHAN-2024 dimABSA Task: Aspect-Based Sentiment Analysis with Coarse-to-Fine In-context Learning

• Authors: Senbin Zhu, Hanjie Zhao, Xingren Wang, Shanhong Liu, Yuxiang Jia, Hongying Zan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.15341
• Pdf URL: https://arxiv.org/pdf/2407.15341
• Copy Paste: [[2407.15341]] ZZU-NLP at SIGHAN-2024 dimABSA Task: Aspect-Based Sentiment Analysis with Coarse-to-Fine In-context Learning(https://arxiv.org/abs/2407.15341)
• Keywords: prompt
• Abstract: The DimABSA task requires fine-grained sentiment intensity prediction for restaurant reviews, including scores for Valence and Arousal dimensions for each Aspect Term. In this study, we propose a Coarse-to-Fine In-context Learning(CFICL) method based on the Baichuan2-7B model for the DimABSA task in the SIGHAN 2024 workshop. Our method improves prediction accuracy through a two-stage optimization process. In the first stage, we use fixed in-context examples and prompt templates to enhance the model's sentiment recognition capability and provide initial predictions for the test data. In the second stage, we encode the Opinion field using BERT and select the most similar training data as new in-context examples based on similarity. These examples include the Opinion field and its scores, as well as related opinion words and their average scores. By filtering for sentiment polarity, we ensure that the examples are consistent with the test data. Our method significantly improves prediction accuracy and consistency by effectively utilizing training data and optimizing in-context examples, as validated by experimental results.
• 摘要：DimABSA 任务需要对餐厅评论进行细粒度的情绪强度预测，包括每个方面词的效价和唤醒度维度的得分。在本研究中，我们提出了一种基于 Baichuan2-7B 模型的由粗到细的上下文学习 (CFICL) 方法，用于 SIGHAN 2024 研讨会上的 DimABSA 任务。我们的方法通过两阶段的优化过程来提高预测准确率。在第一阶段，我们使用固定的上下文示例和提示模板来增强模型的情绪识别能力并为测试数据提供初步预测。在第二阶段，我们使用 BERT 对 Opinion 字段进行编码，并根据相似度选择最相似的训练数据作为新的上下文示例。这些示例包括 Opinion 字段及其得分，以及相关观点词及其平均得分。通过过滤情绪极性，我们确保示例与测试数据一致。实验结果验证了我们的方法通过有效利用训练数据和优化上下文示例，显著提高了预测准确性和一致性。

Title: Improving Minimum Bayes Risk Decoding with Multi-Prompt

• Authors: David Heineman, Yao Dou, Wei Xu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.15343
• Pdf URL: https://arxiv.org/pdf/2407.15343
• Copy Paste: [[2407.15343]] Improving Minimum Bayes Risk Decoding with Multi-Prompt(https://arxiv.org/abs/2407.15343)
• Keywords: llm, prompt
• Abstract: While instruction fine-tuned LLMs are effective text generators, sensitivity to prompt construction makes performance unstable and sub-optimal in practice. Relying on a single "best" prompt cannot capture all differing approaches to a generation problem. Using this observation, we propose multi-prompt decoding, where many candidate generations are decoded from a prompt bank at inference-time. To ensemble candidates, we use Minimum Bayes Risk (MBR) decoding, which selects a final output using a trained value metric. We show multi-prompt improves MBR across a comprehensive set of conditional generation tasks, and show this is a result of estimating a more diverse and higher quality candidate space than that of a single prompt. Further experiments confirm multi-prompt improves generation across tasks, models and metrics.
• 摘要：虽然指令微调的 LLM 是有效的文本生成器，但对提示构造的敏感性使得性能在实践中不稳定且次优。依赖单个“最佳”提示无法捕捉生成问题的所有不同方法。利用这一观察，我们提出了多提示解码，其中在推理时从提示库中解码许多候选生成。为了集合候选，我们使用最小贝叶斯风险 (MBR) 解码，它使用经过训练的价值指标选择最终输出。我们展示了多提示在一系列全面的条件生成任务中改进了 MBR，并表明这是估计比单个提示更多样化和更高质量的候选空间的结果。进一步的实验证实，多提示可以改进跨任务、模型和指标的生成。

Title: MAVEN-Fact: A Large-scale Event Factuality Detection Dataset

• Authors: Chunyang Li, Hao Peng, Xiaozhi Wang, Yunjia Qi, Lei Hou, Bin Xu, Juanzi Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.15352
• Pdf URL: https://arxiv.org/pdf/2407.15352
• Copy Paste: [[2407.15352]] MAVEN-Fact: A Large-scale Event Factuality Detection Dataset(https://arxiv.org/abs/2407.15352)
• Keywords: language model, llm, hallucination
• Abstract: Event Factuality Detection (EFD) task determines the factuality of textual events, i.e., classifying whether an event is a fact, possibility, or impossibility, which is essential for faithfully understanding and utilizing event knowledge. However, due to the lack of high-quality large-scale data, event factuality detection is under-explored in event understanding research, which limits the development of EFD community. To address these issues and provide faithful event understanding, we introduce MAVEN-Fact, a large-scale and high-quality EFD dataset based on the MAVEN dataset. MAVEN-Fact includes factuality annotations of 112,276 events, making it the largest EFD dataset. Extensive experiments demonstrate that MAVEN-Fact is challenging for both conventional fine-tuned models and large language models (LLMs). Thanks to the comprehensive annotations of event arguments and relations in MAVEN, MAVEN-Fact also supports some further analyses and we find that adopting event arguments and relations helps in event factuality detection for fine-tuned models but does not benefit LLMs. Furthermore, we preliminarily study an application case of event factuality detection and find it helps in mitigating event-related hallucination in LLMs. Our dataset and codes can be obtained from \url{this https URL}
• 摘要：事件事实性检测 (EFD) 任务确定文本事件的事实性，即将事件分类为事实、可能性或不可能，这对于忠实地理解和利用事件知识至关重要。然而，由于缺乏高质量的大规模数据，事件事实性检测在事件理解研究中尚未得到充分探索，这限制了 EFD 社区的发展。为了解决这些问题并提供忠实的事件理解，我们引入了 MAVEN-Fact，这是一个基于 MAVEN 数据集的大规模高质量 EFD 数据集。MAVEN-Fact 包含 112,276 个事件的事实性注释，使其成为最大的 EFD 数据集。大量实验表明，MAVEN-Fact 对于传统的微调模型和大型语言模型 (LLM) 都具有挑战性。得益于 MAVEN 对事件参数和关系的全面注释，MAVEN-Fact 还支持一些进一步的分析，我们发现采用事件参数和关系有助于微调模型的事件事实性检测，但对 LLM 没有好处。此外，我们初步研究了事件事实性检测的应用案例，发现它有助于减轻 LLM 中与事件相关的幻觉。我们的数据集和代码可以从 \url{此 https URL} 获取

Title: Customized Retrieval Augmented Generation and Benchmarking for EDA Tool Documentation QA

• Authors: Yuan Pu, Zhuolun He, Tairu Qiu, Haoyuan Wu, Bei Yu
• Subjects: cs.CL, cs.AR
• Abstract URL: https://arxiv.org/abs/2407.15353
• Pdf URL: https://arxiv.org/pdf/2407.15353
• Copy Paste: [[2407.15353]] Customized Retrieval Augmented Generation and Benchmarking for EDA Tool Documentation QA(https://arxiv.org/abs/2407.15353)
• Keywords: llm, retrieval augmented generation
• Abstract: Retrieval augmented generation (RAG) enhances the accuracy and reliability of generative AI models by sourcing factual information from external databases, which is extensively employed in document-grounded question-answering (QA) tasks. Off-the-shelf RAG flows are well pretrained on general-purpose documents, yet they encounter significant challenges when being applied to knowledge-intensive vertical domains, such as electronic design automation (EDA). This paper addresses such issue by proposing a customized RAG framework along with three domain-specific techniques for EDA tool documentation QA, including a contrastive learning scheme for text embedding model fine-tuning, a reranker distilled from proprietary LLM, and a generative LLM fine-tuned with high-quality domain corpus. Furthermore, we have developed and released a documentation QA evaluation benchmark, ORD-QA, for OpenROAD, an advanced RTL-to-GDSII design platform. Experimental results demonstrate that our proposed RAG flow and techniques have achieved superior performance on ORD-QA as well as on a commercial tool, compared with state-of-the-arts. The ORD-QA benchmark and the training dataset for our customized RAG flow are open-source at this https URL.
• 摘要：检索增强生成 (RAG) 通过从外部数据库获取事实信息来提高生成式 AI 模型的准确性和可靠性，广泛应用于基于文档的问答 (QA) 任务。现成的 RAG 流程在通用文档上经过了良好的预训练，但在应用于知识密集型垂直领域（如电子设计自动化 (EDA)）时会遇到重大挑战。本文通过提出一个定制的 RAG 框架以及三种特定领域的 EDA 工具文档 QA 技术来解决此类问题，包括用于文本嵌入模型微调的对比学习方案、从专有 LLM 中提取的重新排序器以及使用高质量领域语料库微调的生成式 LLM。此外，我们还为先进的 RTL-to-GDSII 设计平台 OpenROAD 开发并发布了文档 QA 评估基准 ORD-QA。实验结果表明，与最先进的技术相比，我们提出的 RAG 流程和技术在 ORD-QA 以及商业工具上都取得了卓越的表现。ORD-QA 基准和我们定制的 RAG 流程的训练数据集在此 https URL 上开源。

Title: UF-HOBI at "Discharge Me!": A Hybrid Solution for Discharge Summary Generation Through Prompt-based Tuning of GatorTronGPT Models

• Authors: Mengxian Lyu, Cheng Peng, Daniel Paredes, Ziyi Chen, Aokun Chen, Jiang Bian, Yonghui Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.15359
• Pdf URL: https://arxiv.org/pdf/2407.15359
• Copy Paste: [[2407.15359]] UF-HOBI at "Discharge Me!": A Hybrid Solution for Discharge Summary Generation Through Prompt-based Tuning of GatorTronGPT Models(https://arxiv.org/abs/2407.15359)
• Keywords: gpt, prompt
• Abstract: Automatic generation of discharge summaries presents significant challenges due to the length of clinical documentation, the dispersed nature of patient information, and the diverse terminology used in healthcare. This paper presents a hybrid solution for generating discharge summary sections as part of our participation in the "Discharge Me!" Challenge at the BioNLP 2024 Shared Task. We developed a two-stage generation method using both extractive and abstractive techniques, in which we first apply name entity recognition (NER) to extract key clinical concepts, which are then used as input for a prompt-tuning-based GatorTronGPT model to generate coherent text for two important sections including "Brief Hospital Course" and "Discharge Instructions". Our system was ranked 5th in this challenge, achieving an overall score of 0.284. The results demonstrate the effectiveness of our hybrid solution in improving the quality of automated discharge section generation.
• 摘要：由于临床文档的长度、患者信息的分散性以及医疗保健中使用的术语的多样性，自动生成出院摘要面临着巨大的挑战。本文介绍了一种用于生成出院摘要部分的混合解决方案，这是我们参加 BioNLP 2024 共享任务中的“Discharge Me!”挑战的一部分。我们开发了一种使用提取和抽象技术的两阶段生成方法，其中我们首先应用名称实体识别 (NER) 来提取关键临床概念，然后将其用作基于提示调整的 GatorTronGPT 模型的输入，为两个重要部分生成连贯的文本，包括“简要医院课程”和“出院说明”。我们的系统在这次挑战中排名第五，总分为 0.284。结果证明了我们的混合解决方案在提高自动出院部分生成质量方面的有效性。

Title: Dissecting Multiplication in Transformers: Insights into LLMs

• Authors: Luyu Qiu, Jianing Li, Chi Su, Chen Jason Zhang, Lei Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.15360
• Pdf URL: https://arxiv.org/pdf/2407.15360
• Copy Paste: [[2407.15360]] Dissecting Multiplication in Transformers: Insights into LLMs(https://arxiv.org/abs/2407.15360)
• Keywords: language model, gpt, llm
• Abstract: Transformer-based large language models have achieved remarkable performance across various natural language processing tasks. However, they often struggle with seemingly easy tasks like arithmetic despite their vast capabilities. This stark disparity raise human's concerns about their safe and ethical use, hinder their widespread this http URL this paper, we focus on a typical arithmetic task, integer multiplication, to explore and explain the imperfection of transformers in this domain. We provide comprehensive analysis of a vanilla transformer trained to perform n-digit integer multiplication. Our observations indicate that the model decomposes multiplication task into multiple parallel subtasks, sequentially optimizing each subtask for each digit to complete the final multiplication. Based on observation and analysis, we infer the reasons of transformers deficiencies in multiplication tasks lies in their difficulty in calculating successive carryovers and caching intermediate results, and confirmed this inference through experiments. Guided by these findings, we propose improvements to enhance transformers performance on multiplication tasks. These enhancements are validated through rigorous testing and mathematical modeling, not only enhance transformer's interpretability, but also improve its performance, e.g., we achieve over 99.9% accuracy on 5-digit integer multiplication with a tiny transformer, outperform LLMs GPT-4. Our method contributes to the broader fields of model understanding and interpretability, paving the way for analyzing more complex tasks and Transformer models. This work underscores the importance of explainable AI, helping to build trust in large language models and promoting their adoption in critical applications.
• 摘要：基于 Transformer 的大型语言模型在各种自然语言处理任务中都取得了显著的表现。然而，尽管它们拥有强大的能力，但它们在处理算术等看似简单的任务时却常常举步维艰。这种巨大的差异引起了人们对其安全性和道德使用的担忧，阻碍了其广泛传播。本文，我们专注于一个典型的算术任务——整数乘法，以探索和解释 Transformer 在这一领域的不完善之处。我们对一个经过训练可以执行 n 位整数乘法的 vanilla Transformer 进行了全面的分析。我们的观察表明，该模型将乘法任务分解为多个并行子任务，依次优化每个数字的每个子任务以完成最终的乘法。通过观察和分析，我们推断 Transformer 在乘法任务中存在缺陷的原因在于它们难以计算连续进位和缓存中间结果，并通过实验证实了这一推断。在这些发现的指导下，我们提出了改进措施以提高 Transformer 在乘法任务上的性能。这些增强功能经过了严格的测试和数学建模验证，不仅增强了 Transformer 的可解释性，还提高了其性能，例如，我们使用一个微型 Transformer 在 5 位整数乘法上实现了超过 99.9% 的准确率，优于 LLM 的 GPT-4。我们的方法为模型理解和可解释性等更广泛的领域做出了贡献，为分析更复杂的任务和 Transformer 模型铺平了道路。这项工作强调了可解释人工智能的重要性，有助于建立对大型语言模型的信任并促进其在关键应用中的采用。

Title: Walking in Others' Shoes: How Perspective-Taking Guides Large Language Models in Reducing Toxicity and Bias

• Authors: Rongwu Xu, Zi'an Zhou, Tianwei Zhang, Zehan Qi, Su Yao, Ke Xu, Wei Xu, Han Qiu
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2407.15366
• Pdf URL: https://arxiv.org/pdf/2407.15366
• Copy Paste: [[2407.15366]] Walking in Others' Shoes: How Perspective-Taking Guides Large Language Models in Reducing Toxicity and Bias(https://arxiv.org/abs/2407.15366)
• Keywords: language model, gpt, llm, prompt, chat
• Abstract: The common toxicity and societal bias in contents generated by large language models (LLMs) necessitate strategies to reduce harm. Present solutions often demand white-box access to the model or substantial training, which is impractical for cutting-edge commercial LLMs. Moreover, prevailing prompting methods depend on external tool feedback and fail to simultaneously lessen toxicity and bias. Motivated by social psychology principles, we propose a novel strategy named \textbf{perspective-taking prompting (\textsc{PeT})} that inspires LLMs to integrate diverse human perspectives and self-regulate their responses. This self-correction mechanism can significantly diminish toxicity (up to $89\%$) and bias (up to $73\%$) in LLMs' responses. Rigorous evaluations and ablation studies are conducted on two commercial LLMs (ChatGPT and GLM) and three open-source LLMs, revealing \textsc{PeT}'s superiority in producing less harmful responses, outperforming five strong baselines.
• 摘要：大型语言模型 (LLM) 生成的内容中普遍存在毒性和社会偏见，因此需要制定策略来减少危害。目前的解决方案通常需要白盒访问模型或进行大量训练，这对于尖端的商业 LLM 来说是不切实际的。此外，现行的提示方法依赖于外部工具反馈，无法同时减少毒性和偏见。受社会心理学原理的启发，我们提出了一种名为 \textbf{观点采择提示 (\textsc{PeT})} 的新策略，该策略激励 LLM 整合不同的人类观点并自我调节他们的反应。这种自我纠正机制可以显著减少 LLM 反应中的毒性（高达 $89\%$）和偏见（高达 $73\%$）。对两个商业 LLM（ChatGPT 和 GLM）和三个开源 LLM 进行了严格的评估和消融研究，揭示了 \textsc{PeT} 在产生较少危害反应方面的优势，优于五个强大的基线。

Title: ALLaM: Large Language Models for Arabic and English

• Authors: M Saiful Bari, Yazeed Alnumay, Norah A. Alzahrani, Nouf M. Alotaibi, Hisham A. Alyahya, Sultan AlRashed, Faisal A. Mirza, Shaykhah Z. Alsubaie, Hassan A. Alahmed, Ghadah Alabduljabbar, Raghad Alkhathran, Yousef Almushayqih, Raneem Alnajim, Salman Alsubaihi, Maryam Al Mansour, Majed Alrubaian, Ali Alammari, Zaki Alawami, Abdulmohsen Al-Thubaity, Ahmed Abdelali, Jeril Kuriakose, Abdalghani Abujabal, Nora Al-Twairesh, Areeb Alowisheq, Haidar Khan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.15390
• Pdf URL: https://arxiv.org/pdf/2407.15390
• Copy Paste: [[2407.15390]] ALLaM: Large Language Models for Arabic and English(https://arxiv.org/abs/2407.15390)
• Keywords: language model
• Abstract: We present ALLaM: Arabic Large Language Model, a series of large language models to support the ecosystem of Arabic Language Technologies (ALT). ALLaM is carefully trained considering the values of language alignment and knowledge transfer at scale. Our autoregressive decoder-only architecture models demonstrate how second-language acquisition via vocabulary expansion and pretraining on a mixture of Arabic and English text can steer a model towards a new language (Arabic) without any catastrophic forgetting in the original language (English). Furthermore, we highlight the effectiveness of using parallel/translated data to aid the process of knowledge alignment between languages. Finally, we show that extensive alignment with human preferences can significantly enhance the performance of a language model compared to models of a larger scale with lower quality alignment. ALLaM achieves state-of-the-art performance in various Arabic benchmarks, including MMLU Arabic, ACVA, and Arabic Exams. Our aligned models improve both in Arabic and English from their base aligned models.
• 摘要：我们推出了 ALLaM：阿拉伯语大型语言模型，这是一系列大型语言模型，用于支持阿拉伯语语言技术 (ALT) 生态系统。ALLaM 经过精心训练，考虑到语言对齐和大规模知识转移的价值。我们的自回归解码器专用架构模型展示了如何通过词汇扩展和对阿拉伯语和英语混合文本进行预训练来获得第二语言，从而将模型引向一种新语言（阿拉伯语），而不会对原始语言（英语）产生任何灾难性的遗忘。此外，我们强调了使用并行/翻译数据来帮助语言间知识对齐过程的有效性。最后，我们表明，与具有较低质量对齐的较大规模模型相比，与人类偏好的广泛对齐可以显着提高语言模型的性能。ALLaM 在各种阿拉伯语基准测试中都取得了最先进的性能，包括 MMLU 阿拉伯语、ACVA 和阿拉伯语考试。我们的对齐模型在阿拉伯语和英语方面都比其基本对齐模型有所改进。

Title: Imposter.AI: Adversarial Attacks with Hidden Intentions towards Aligned Large Language Models

• Authors: Xiao Liu, Liangzhi Li, Tong Xiang, Fuying Ye, Lu Wei, Wangyue Li, Noa Garcia
• Subjects: cs.CL, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2407.15399
• Pdf URL: https://arxiv.org/pdf/2407.15399
• Copy Paste: [[2407.15399]] Imposter.AI: Adversarial Attacks with Hidden Intentions towards Aligned Large Language Models(https://arxiv.org/abs/2407.15399)
• Keywords: language model, gpt, llm, prompt, chat
• Abstract: With the development of large language models (LLMs) like ChatGPT, both their vast applications and potential vulnerabilities have come to the forefront. While developers have integrated multiple safety mechanisms to mitigate their misuse, a risk remains, particularly when models encounter adversarial inputs. This study unveils an attack mechanism that capitalizes on human conversation strategies to extract harmful information from LLMs. We delineate three pivotal strategies: (i) decomposing malicious questions into seemingly innocent sub-questions; (ii) rewriting overtly malicious questions into more covert, benign-sounding ones; (iii) enhancing the harmfulness of responses by prompting models for illustrative examples. Unlike conventional methods that target explicit malicious responses, our approach delves deeper into the nature of the information provided in responses. Through our experiments conducted on GPT-3.5-turbo, GPT-4, and Llama2, our method has demonstrated a marked efficacy compared to conventional attack methods. In summary, this work introduces a novel attack method that outperforms previous approaches, raising an important question: How to discern whether the ultimate intent in a dialogue is malicious?
• 摘要：随着 ChatGPT 等大型语言模型 (LLM) 的发展，它们的广泛应用和潜在漏洞都已成为人​​们关注的焦点。尽管开发人员已经集成了多种安全机制来减轻其滥用风险，但风险仍然存在，尤其是当模型遇到对抗性输入时。这项研究揭示了一种利用人类对话策略从 LLM 中提取有害信息的攻击机制。我们概述了三种关键策略：(i) 将恶意问题分解为看似无辜的子问题；(ii) 将明显恶意的问题重写为更隐蔽、听起来无害的问题；(iii) 通过提示模型提供说明性示例来增强响应的危害性。与针对明确恶意响应的传统方法不同，我们的方法更深入地探究了响应中提供的信息的性质。通过对 GPT-3.5-turbo、GPT-4 和 Llama2 进行的实验，与传统攻击方法相比，我们的方法表现出了显著的有效性。总之，这项工作引入了一种优于以前方法的新型攻击方法，提出了一个重要的问题：如何辨别对话的最终意图是否是恶意的？

Title: LLaST: Improved End-to-end Speech Translation System Leveraged by Large Language Models

• Authors: Xi Chen, Songyang Zhang, Qibing Bai, Kai Chen, Satoshi Nakamura
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.15415
• Pdf URL: https://arxiv.org/pdf/2407.15415
• Copy Paste: [[2407.15415]] LLaST: Improved End-to-end Speech Translation System Leveraged by Large Language Models(https://arxiv.org/abs/2407.15415)
• Keywords: language model, llm
• Abstract: We introduces LLaST, a framework for building high-performance Large Language model based Speech-to-text Translation systems. We address the limitations of end-to-end speech translation(E2E ST) models by exploring model architecture design and optimization techniques tailored for LLMs. Our approach includes LLM-based speech translation architecture design, ASR-augmented training, multilingual data augmentation, and dual-LoRA optimization. Our approach demonstrates superior performance on the CoVoST-2 benchmark and showcases exceptional scaling capabilities powered by LLMs. We believe this effective method will serve as a strong baseline for speech translation and provide insights for future improvements of the LLM-based speech translation framework. We release the data, code and models in this https URL.
• 摘要：我们引入了 LLaST，这是一个用于构建基于高性能大型语言模型的语音到文本翻译系统的框架。我们通过探索针对 LLM 量身定制的模型架构设计和优化技术来解决端到端语音翻译 (E2E ST) 模型的局限性。我们的方法包括基于 LLM 的语音翻译架构设计、ASR 增强训练、多语言数据增强和双 LoRA 优化。我们的方法在 CoVoST-2 基准上表现出色，并展示了由 LLM 提供支持的卓越扩展能力。我们相信这种有效的方法将成为语音翻译的坚实基础，并为基于 LLM 的语音翻译框架的未来改进提供见解。我们在此 https URL 中发布数据、代码和模型。

Title: Developing a Reliable, General-Purpose Hallucination Detection and Mitigation Service: Insights and Lessons Learned

• Authors: Song Wang, Xun Wang, Jie Mei, Yujia Xie, Sean Muarray, Zhang Li, Lingfeng Wu, Si-Qing Chen, Wayne Xiong
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.15441
• Pdf URL: https://arxiv.org/pdf/2407.15441
• Copy Paste: [[2407.15441]] Developing a Reliable, General-Purpose Hallucination Detection and Mitigation Service: Insights and Lessons Learned(https://arxiv.org/abs/2407.15441)
• Keywords: language model, llm, hallucination
• Abstract: Hallucination, a phenomenon where large language models (LLMs) produce output that is factually incorrect or unrelated to the input, is a major challenge for LLM applications that require accuracy and dependability. In this paper, we introduce a reliable and high-speed production system aimed at detecting and rectifying the hallucination issue within LLMs. Our system encompasses named entity recognition (NER), natural language inference (NLI), span-based detection (SBD), and an intricate decision tree-based process to reliably detect a wide range of hallucinations in LLM responses. Furthermore, our team has crafted a rewriting mechanism that maintains an optimal mix of precision, response time, and cost-effectiveness. We detail the core elements of our framework and underscore the paramount challenges tied to response time, availability, and performance metrics, which are crucial for real-world deployment of these technologies. Our extensive evaluation, utilizing offline data and live production traffic, confirms the efficacy of our proposed framework and service.
• 摘要：幻觉是一种现象，大型语言模型 (LLM) 产生的输出与输入事实上不正确或无关，对于需要准确性和可靠性的 LLM 应用程序来说，这是一个重大挑战。在本文中，我们介绍了一种可靠的高速生产系统，旨在检测和纠正 LLM 中的幻觉问题。我们的系统包括命名实体识别 (NER)、自然语言推理 (NLI)、基于跨度的检测 (SBD) 和复杂的基于决策树的过程，可以可靠地检测 LLM 响应中的各种幻觉。此外，我们的团队还设计了一种重写机制，以保持精度、响应时间和成本效益的最佳组合。我们详细介绍了我们框架的核心元素，并强调了与响应时间、可用性和性能指标相关的首要挑战，这些对于这些技术的实际部署至关重要。我们利用离线数据和实时生产流量进行了广泛的评估，证实了我们提出的框架和服务的有效性。

Title: Text-to-Battery Recipe: A language modeling-based protocol for automatic battery recipe extraction and retrieval

• Authors: Daeun Lee, Jaewoong Choi, Hiroshi Mizuseki, Byungju Lee
• Subjects: cs.CL, cond-mat.mtrl-sci
• Abstract URL: https://arxiv.org/abs/2407.15459
• Pdf URL: https://arxiv.org/pdf/2407.15459
• Copy Paste: [[2407.15459]] Text-to-Battery Recipe: A language modeling-based protocol for automatic battery recipe extraction and retrieval(https://arxiv.org/abs/2407.15459)
• Keywords: language model
• Abstract: Recent studies have increasingly applied natural language processing (NLP) to automatically extract experimental research data from the extensive battery materials literature. Despite the complex process involved in battery manufacturing -- from material synthesis to cell assembly -- there has been no comprehensive study systematically organizing this information. In response, we propose a language modeling-based protocol, Text-to-Battery Recipe (T2BR), for the automatic extraction of end-to-end battery recipes, validated using a case study on batteries containing LiFePO4 cathode material. We report machine learning-based paper filtering models, screening 2,174 relevant papers from the keyword-based search results, and unsupervised topic models to identify 2,876 paragraphs related to cathode synthesis and 2,958 paragraphs related to cell assembly. Then, focusing on the two topics, two deep learning-based named entity recognition models are developed to extract a total of 30 entities -- including precursors, active materials, and synthesis methods -- achieving F1 scores of 88.18% and 94.61%. The accurate extraction of entities enables the systematic generation of 165 end-toend recipes of LiFePO4 batteries. Our protocol and results offer valuable insights into specific trends, such as associations between precursor materials and synthesis methods, or combinations between different precursor materials. We anticipate that our findings will serve as a foundational knowledge base for facilitating battery-recipe information retrieval. The proposed protocol will significantly accelerate the review of battery material literature and catalyze innovations in battery design and development.
• 摘要：最近的研究越来越多地应用自然语言处理 (NLP) 从大量电池材料文献中自动提取实验研究数据。尽管电池制造涉及复杂的过程——从材料合成到电池组装——但尚未有全面的研究系统地组织这些信息。为此，我们提出了一种基于语言建模的协议，即文本到电池配方 (T2BR)，用于自动提取端到端电池配方，并使用含有 LiFePO4 阴极材料的电池案例研究进行了验证。我们报告了基于机器学习的论文过滤模型，从基于关键字的搜索结果中筛选出 2,174 篇相关论文，以及无监督主题模型，以识别与阴极合成相关的 2,876 段和与电池组装相关的 2,958 段。然后，针对这两个主题，开发了两个基于深度学习的命名实体识别模型，共提取了 30 个实体——包括前体、活性材料和合成方法——F1 得分分别为 88.18% 和 94.61%。实体的精确提取使得系统地生成 165 种端到端 LiFePO4 电池配方成为可能。我们的协议和结果为特定趋势提供了宝贵的见解，例如前体材料与合成方法之间的关联，或不同前体材料之间的组合。我们预计我们的发现将成为促进电池配方信息检索的基础知识库。拟议的协议将大大加快电池材料文献的审查，并催化电池设计和开发的创新。

Title: Two Stacks Are Better Than One: A Comparison of Language Modeling and Translation as Multilingual Pretraining Objectives

• Authors: Zihao Li, Shaoxiong Ji, Timothee Mickus, Vincent Segonne, Jörg Tiedemann
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.15489
• Pdf URL: https://arxiv.org/pdf/2407.15489
• Copy Paste: [[2407.15489]] Two Stacks Are Better Than One: A Comparison of Language Modeling and Translation as Multilingual Pretraining Objectives(https://arxiv.org/abs/2407.15489)
• Keywords: language model
• Abstract: Pretrained language models (PLMs) display impressive performances and have captured the attention of the NLP community. Establishing the best practices in pretraining has therefore become a major point of focus for much of NLP research -- especially since the insights developed for monolingual English models need not carry to more complex multilingual. One significant caveat of the current state of the art is that different works are rarely comparable: they often discuss different parameter counts, training data, and evaluation methodology. This paper proposes a comparison of multilingual pretraining objectives in a controlled methodological environment. We ensure that training data and model architectures are comparable, and discuss the downstream performances across 6 languages that we observe in probing and fine-tuning scenarios. We make two key observations: (1) the architecture dictates which pretraining objective is optimal; (2) multilingual translation is a very effective pre-training objective under the right conditions. We make our code, data, and model weights available at \texttt{\url{this https URL}}.
• 摘要：预训练语言模型 (PLM) 表现出色，引起了 NLP 社区的关注。因此，建立预训练的最佳实践已成为许多 NLP 研究的重点——尤其是因为为单语英语模型开发的见解不必应用于更复杂的多语言模型。当前最先进的技术的一个重要警告是，不同的作品很少具有可比性：它们通常讨论不同的参数计数、训练数据和评估方法。本文提出了在受控方法环境中比较多语言预训练目标。我们确保训练数据和模型架构具有可比性，并讨论我们在探测和微调场景中观察到的 6 种语言的下游性能。我们有两个关键观察结果：(1) 架构决定哪个预训练目标是最佳的；(2) 在适当的条件下，多语言翻译是一种非常有效的预训练目标。我们在 \texttt{\url{此 https URL}} 上提供我们的代码、数据和模型权重。

Title: Compensate Quantization Errors+: Quantized Models Are Inquisitive Learners

• Authors: Yifei Gao, Jie Ou, Lei Wang, Fanhua Shang, Jaji Wu, Jun Cheng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.15508
• Pdf URL: https://arxiv.org/pdf/2407.15508
• Copy Paste: [[2407.15508]] Compensate Quantization Errors+: Quantized Models Are Inquisitive Learners(https://arxiv.org/abs/2407.15508)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) showcase remarkable performance and robust deductive capabilities, yet their expansive size complicates deployment and raises environmental concerns due to substantial resource consumption. The recent development of a quantization technique known as Learnable Singular-value Increment (LSI) has addressed some of these quantization challenges. Leveraging insights from LSI and our extensive research, we have developed innovative methods that enhance the performance of quantized LLMs, particularly in low-bit settings. Our methods consistently deliver state-of-the-art results across various quantization scenarios and offer deep theoretical insights into the quantization process, elucidating the potential of quantized models for widespread application.
• 摘要：大型语言模型 (LLM) 展现出卓越的性能和强大的推理能力，但其庞大的规模使部署变得复杂，并因大量资源消耗而引发环境问题。最近开发的一种称为可学习奇异值增量 (LSI) 的量化技术解决了其中一些量化挑战。利用 LSI 的见解和我们广泛的研究，我们开发了创新方法，可提高量化 LLM 的性能，特别是在低位设置下。我们的方法在各种量化场景中始终提供最先进的结果，并为量化过程提供深刻的理论见解，阐明量化模型广泛应用的潜力。

Title: SETTP: Style Extraction and Tunable Inference via Dual-level Transferable Prompt Learning

• Authors: Chunzhen Jin, Yongfeng Huang, Yaqi Wang, Peng Cao, Osmar Zaiane
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.15556
• Pdf URL: https://arxiv.org/pdf/2407.15556
• Copy Paste: [[2407.15556]] SETTP: Style Extraction and Tunable Inference via Dual-level Transferable Prompt Learning(https://arxiv.org/abs/2407.15556)
• Keywords: gpt, prompt, chat
• Abstract: Text style transfer, an important research direction in natural language processing, aims to adapt the text to various preferences but often faces challenges with limited resources. In this work, we introduce a novel method termed Style Extraction and Tunable Inference via Dual-level Transferable Prompt Learning (SETTP) for effective style transfer in low-resource scenarios. First, SETTP learns source style-level prompts containing fundamental style characteristics from high-resource style transfer. During training, the source style-level prompts are transferred through an attention module to derive a target style-level prompt for beneficial knowledge provision in low-resource style transfer. Additionally, we propose instance-level prompts obtained by clustering the target resources based on the semantic content to reduce semantic bias. We also propose an automated evaluation approach of style similarity based on alignment with human evaluations using ChatGPT-4. Our experiments across three resourceful styles show that SETTP requires only 1/20th of the data volume to achieve performance comparable to state-of-the-art methods. In tasks involving scarce data like writing style and role style, SETTP outperforms previous methods by 16.24\%.
• 摘要：文本风格转换是自然语言处理中的一个重要研究方向，旨在使文本适应各种偏好，但在资源有限的情况下往往会面临挑战。在这项工作中，我们介绍了一种新方法，即通过双层可转移提示学习进行风格提取和可调推理（SETTP），用于在低资源场景中实现有效的风格转换。首先，SETTP 从高资源风格转换中学习包含基本风格特征的源风格级提示。在训练过程中，源风格级提示通过注意模块进行传输，以得出目标风格级提示，以便在低资源风格转换中提供有益的知识。此外，我们提出了通过基于语义内容对目标资源进行聚类而获得的实例级提示，以减少语义偏差。我们还提出了一种使用 ChatGPT-4 基于与人工评估对齐的风格相似度自动评估方法。我们在三种资源丰富的风格中进行的实验表明，SETTP 只需要 1/20 的数据量就能实现与最先进方法相当的性能。在涉及写作风格、角色风格等稀缺数据的任务中，SETTP 的表现比以前的方法高出 16.24%。

Title: An Empirical Study of Retrieval Augmented Generation with Chain-of-Thought

• Authors: Yuetong Zhao, Hongyu Cao, Xianyu Zhao, Zhijian Ou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.15569
• Pdf URL: https://arxiv.org/pdf/2407.15569
• Copy Paste: [[2407.15569]] An Empirical Study of Retrieval Augmented Generation with Chain-of-Thought(https://arxiv.org/abs/2407.15569)
• Keywords: gpt, chat, retrieval augmented generation, chain-of-thought
• Abstract: Since the launch of ChatGPT at the end of 2022, generative dialogue models represented by ChatGPT have quickly become essential tools in daily life. As user expectations increase, enhancing the capability of generative dialogue models to solve complex problems has become a focal point of current research. This paper delves into the effectiveness of the RAFT (Retrieval Augmented Fine-Tuning) method in improving the performance of Generative dialogue models. RAFT combines chain-of-thought with model supervised fine-tuning (SFT) and retrieval augmented generation (RAG), which significantly enhanced the model's information extraction and logical reasoning abilities. We evaluated the RAFT method across multiple datasets and analysed its performance in various reasoning tasks, including long-form QA and short-form QA tasks, tasks in both Chinese and English, and supportive and comparison reasoning tasks. Notably, it addresses the gaps in previous research regarding long-form QA tasks and Chinese datasets. Moreover, we also evaluate the benefit of the chain-of-thought (CoT) in the RAFT method. This work offers valuable insights for studies focused on enhancing the performance of generative dialogue models.
• 摘要：自 2022 年底 ChatGPT 上线以来，以 ChatGPT 为代表的生成式对话模型迅速成为人们日常生活中必不可少的工具。随着用户期望的提高，提升生成式对话模型解决复杂问题的能力成为当前研究的重点。本文深入探讨了 RAFT（检索增强微调）方法在提升生成式对话模型性能方面的有效性。RAFT 将思路链与模型监督微调（SFT）和检索增强生成（RAG）相结合，显著增强了模型的信息提取和逻辑推理能力。我们在多个数据集上评估了 RAFT 方法，并分析了其在各种推理任务中的表现，包括长篇问答任务和短篇问答任务、中英文任务以及支持性和比较性推理任务。值得注意的是，它弥补了以往研究中关于长篇问答任务和中文数据集的空白。此外，我们还评估了 RAFT 方法中的思路链 (CoT) 的优势。这项工作为专注于提高生成对话模型性能的研究提供了宝贵的见解。

Title: Can GPT-4 learn to analyze moves in research article abstracts?

• Authors: Danni Yu, Marina Bondi, Ken Hylannd
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.15612
• Pdf URL: https://arxiv.org/pdf/2407.15612
• Copy Paste: [[2407.15612]] Can GPT-4 learn to analyze moves in research article abstracts?(https://arxiv.org/abs/2407.15612)
• Keywords: gpt, prompt
• Abstract: One of the most powerful and enduring ideas in written discourse analysis is that genres can be described in terms of the moves which structure a writer's purpose. Considerable research has sought to identify these distinct communicative acts, but analyses have been beset by problems of subjectivity, reliability and the time-consuming need for multiple coders to confirm analyses. In this paper we employ the affordances of GPT-4 to automate the annotation process by using natural language prompts. Focusing on abstracts from articles in four applied linguistics journals, we devise prompts which enable the model to identify moves effectively. The annotated outputs of these prompts were evaluated by two assessors with a third addressing disagreements. The results show that an 8-shot prompt was more effective than one using two, confirming that the inclusion of examples illustrating areas of variability can enhance GPT-4's ability to recognize multiple moves in a single sentence and reduce bias related to textual position. We suggest that GPT-4 offers considerable potential in automating this annotation process, when human actors with domain specific linguistic expertise inform the prompting process.
• 摘要：书面语篇分析中最有力和最持久的观点之一是，体裁可以用构成作者目的的动作来描述。大量研究试图识别这些不同的交流行为，但分析一直受到主观性、可靠性和需要多个编码员来确认分析的耗时问题困扰。在本文中，我们利用 GPT-4 的功能，通过使用自然语言提示来自动化注释过程。我们专注于四本应用语言学期刊的文章摘要，设计了提示，使模型能够有效识别动作。这些提示的注释输出由两名评估员评估，第三名评估员负责解决分歧。结果表明，8 个提示比使用两个提示更有效，证实了包含说明变化区域的示例可以增强 GPT-4 识别单个句子中多个动作的能力，并减少与文本位置相关的偏见。我们认为，当具有特定领域语言专业知识的人类参与者为提示过程提供信息时，GPT-4 在自动化该注释过程方面具有巨大的潜力。

Title: RadioRAG: Factual Large Language Models for Enhanced Diagnostics in Radiology Using Dynamic Retrieval Augmented Generation

• Authors: Soroosh Tayebi Arasteh, Mahshad Lotfinia, Keno Bressem, Robert Siepmann, Dyke Ferber, Christiane Kuhl, Jakob Nikolas Kather, Sven Nebelung, Daniel Truhn
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.15621
• Pdf URL: https://arxiv.org/pdf/2407.15621
• Copy Paste: [[2407.15621]] RadioRAG: Factual Large Language Models for Enhanced Diagnostics in Radiology Using Dynamic Retrieval Augmented Generation(https://arxiv.org/abs/2407.15621)
• Keywords: language model, gpt, llm, prompt, retrieval augmented generation
• Abstract: Large language models (LLMs) have advanced the field of artificial intelligence (AI) in medicine. However LLMs often generate outdated or inaccurate information based on static training datasets. Retrieval augmented generation (RAG) mitigates this by integrating outside data sources. While previous RAG systems used pre-assembled, fixed databases with limited flexibility, we have developed Radiology RAG (RadioRAG) as an end-to-end framework that retrieves data from authoritative radiologic online sources in real-time. RadioRAG is evaluated using a dedicated radiologic question-and-answer dataset (RadioQA). We evaluate the diagnostic accuracy of various LLMs when answering radiology-specific questions with and without access to additional online information via RAG. Using 80 questions from RSNA Case Collection across radiologic subspecialties and 24 additional expert-curated questions, for which the correct gold-standard answers were available, LLMs (GPT-3.5-turbo, GPT-4, Mistral-7B, Mixtral-8x7B, and Llama3 [8B and 70B]) were prompted with and without RadioRAG. RadioRAG retrieved context-specific information from this http URL in real-time and incorporated them into its reply. RadioRAG consistently improved diagnostic accuracy across all LLMs, with relative improvements ranging from 2% to 54%. It matched or exceeded question answering without RAG across radiologic subspecialties, particularly in breast imaging and emergency radiology. However, degree of improvement varied among models; GPT-3.5-turbo and Mixtral-8x7B-instruct-v0.1 saw notable gains, while Mistral-7B-instruct-v0.2 showed no improvement, highlighting variability in its effectiveness. LLMs benefit when provided access to domain-specific data beyond their training data. For radiology, RadioRAG establishes a robust framework that substantially improves diagnostic accuracy and factuality in radiological question answering.
• 摘要：大型语言模型 (LLM) 推动了医学领域人工智能 (AI) 的发展。然而，LLM 通常会根据静态训练数据集生成过时或不准确的信息。检索增强生成 (RAG) 通过集成外部数据源来缓解这种情况。虽然以前的 RAG 系统使用预先组装的固定数据库，灵活性有限，但我们开发了放射学 RAG (RadioRAG) 作为端到端框架，可实时从权威的放射学在线源检索数据。RadioRAG 是使用专用的放射学问答数据集 (RadioQA) 进行评估的。我们在回答放射学特定问题时评估各种 LLM 的诊断准确性，包括通过 RAG 访问和不访问其他在线信息。使用来自 RSNA 病例收集的 80 个放射学亚专业的问答和另外 24 个专家精选的问题（这些问题都有正确的黄金标准答案），LLM（GPT-3.5-turbo、GPT-4、Mistral-7B、Mixtral-8x7B 和 Llama3 [8B 和 70B]）在使用和不使用 RadioRAG 的情况下进行提示。RadioRAG 实时从此 http URL 检索上下文特定的信息并将其合并到其回复中。RadioRAG 持续提高了所有 LLM 的诊断准确率，相对提高从 2% 到 54% 不等。它与不使用 RAG 的放射学亚专业的问答相当或超过了，特别是在乳房成像和急诊放射学方面。然而，不同模型的改进程度各不相同；GPT-3.5-turbo 和 Mixtral-8x7B-instruct-v0.1 取得了显着的进步，而 Mistral-7B-instruct-v0.2 没有进步，突显了其有效性的多变性。如果法学硕士 (LLM) 能够访问训练数据以外的特定领域数据，那么他们将受益匪浅。对于放射学，RadioRAG 建立了一个强大的框架，可显著提高放射学问答的诊断准确性和真实性。

Title: Psychometric Alignment: Capturing Human Knowledge Distributions via Language Models

• Authors: Joy He-Yueya, Wanjing Anya Ma, Kanishk Gandhi, Benjamin W. Domingue, Emma Brunskill, Noah D. Goodman
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.15645
• Pdf URL: https://arxiv.org/pdf/2407.15645
• Copy Paste: [[2407.15645]] Psychometric Alignment: Capturing Human Knowledge Distributions via Language Models(https://arxiv.org/abs/2407.15645)
• Keywords: language model, prompt
• Abstract: Language models (LMs) are increasingly used to simulate human-like responses in scenarios where accurately mimicking a population's behavior can guide decision-making, such as in developing educational materials and designing public policies. The objective of these simulations is for LMs to capture the variations in human responses, rather than merely providing the expected correct answers. Prior work has shown that LMs often generate unrealistically accurate responses, but there are no established metrics to quantify how closely the knowledge distribution of LMs aligns with that of humans. To address this, we introduce "psychometric alignment," a metric that measures the extent to which LMs reflect human knowledge distribution. Assessing this alignment involves collecting responses from both LMs and humans to the same set of test items and using Item Response Theory to analyze the differences in item functioning between the groups. We demonstrate that our metric can capture important variations in populations that traditional metrics, like differences in accuracy, fail to capture. We apply this metric to assess existing LMs for their alignment with human knowledge distributions across three real-world domains. We find significant misalignment between LMs and human populations, though using persona-based prompts can improve alignment. Interestingly, smaller LMs tend to achieve greater psychometric alignment than larger LMs. Further, training LMs on human response data from the target distribution enhances their psychometric alignment on unseen test items, but the effectiveness of such training varies across domains.
• 摘要：语言模型 (LM) 越来越多地用于模拟类似人类的反应，在准确模仿人群行为可以指导决策的场景中，例如在开发教育材料和设计公共政策时。这些模拟的目的是让 LM 捕捉人类反应的变化，而不仅仅是提供预期的正确答案。先前的研究表明，LM 通常会产生不切实际的准确反应，但没有既定的指标来量化 LM 的知识分布与人类的知识分布的接近程度。为了解决这个问题，我们引入了“心理测量一致性”，这是一种衡量 LM 反映人类知识分布程度的指标。评估这种一致性涉及收集 LM 和人类对同一组测试项目的回答，并使用项目反应理论来分析两组之间项目功能的差异。我们证明我们的指标可以捕捉到传统指标（如准确性差异）无法捕捉到的人群重要变化。我们应用此指标来评估现有 LM 与三个现实世界领域的人类知识分布的一致性。我们发现 LM 与人类群体之间存在显著的不一致，尽管使用基于角色的提示可以改善一致性。有趣的是，较小的 LM 往往比较大的 LM 实现更大的心理测量一致性。此外，使用来自目标分布的人类响应数据训练 LM 可以增强它们对未见过的测试项目的心理测量一致性，但这种训练的有效性因领域而异。

Title: Counter Turing Test ($CT^2$): Investigating AI-Generated Text Detection for Hindi -- Ranking LLMs based on Hindi AI Detectability Index ($ADI_{hi}$)

• Authors: Ishan Kavathekar, Anku Rani, Ashmit Chamoli, Ponnurangam Kumaraguru, Amit Sheth, Amitava Das
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.15694
• Pdf URL: https://arxiv.org/pdf/2407.15694
• Copy Paste: [[2407.15694]] Counter Turing Test ($CT^2$): Investigating AI-Generated Text Detection for Hindi -- Ranking LLMs based on Hindi AI Detectability Index ($ADI_{hi}$)(https://arxiv.org/abs/2407.15694)
• Keywords: language model, llm
• Abstract: The widespread adoption of large language models (LLMs) and awareness around multilingual LLMs have raised concerns regarding the potential risks and repercussions linked to the misapplication of AI-generated text, necessitating increased vigilance. While these models are primarily trained for English, their extensive training on vast datasets covering almost the entire web, equips them with capabilities to perform well in numerous other languages. AI-Generated Text Detection (AGTD) has emerged as a topic that has already received immediate attention in research, with some initial methods having been proposed, soon followed by the emergence of techniques to bypass detection. In this paper, we report our investigation on AGTD for an indic language Hindi. Our major contributions are in four folds: i) examined 26 LLMs to evaluate their proficiency in generating Hindi text, ii) introducing the AI-generated news article in Hindi ($AG_{hi}$) dataset, iii) evaluated the effectiveness of five recently proposed AGTD techniques: ConDA, J-Guard, RADAR, RAIDAR and Intrinsic Dimension Estimation for detecting AI-generated Hindi text, iv) proposed Hindi AI Detectability Index ($ADI_{hi}$) which shows a spectrum to understand the evolving landscape of eloquence of AI-generated text in Hindi. We will make the codes and datasets available to encourage further research.
• 摘要：大型语言模型 (LLM) 的广泛采用和对多语言 LLM 的认识引发了人们对误用 AI 生成文本的潜在风险和后果的担忧，因此需要提高警惕。虽然这些模型主要针对英语进行训练，但它们在几乎覆盖整个网络的庞大数据集上进行了广泛的训练，使其能够在许多其他语言中表现出色。AI 生成文本检测 (AGTD) 已成为一个在研究中立即受到关注的话题，一些初步方法已被提出，随后很快就出现了绕过检测的技术。在本文中，我们报告了对印度语印地语的 AGTD 的调查。我们的主要贡献有四个方面：i）检查了 26 个 LLM，以评估它们生成印地语文本的能力；ii）引入了 AI 生成的印地语新闻文章 ($AG_{hi}$) 数据集；iii）评估了五种最近提出的 AGTD 技术的有效性：ConDA、J-Guard、RADAR、RAIDAR 和内在维度估计用于检测 AI 生成的印地语文本；iv）提出了印地语 AI 可检测性指数 ($ADI_{hi}$)，该指数显示了一个范围，以了解 AI 生成的印地语文本的雄辩性不断发展的格局。我们将提供代码和数据集以鼓励进一步研究。

Title: AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?

• Authors: Ori Yoran, Samuel Joseph Amouyal, Chaitanya Malaviya, Ben Bogin, Ofir Press, Jonathan Berant
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.15711
• Pdf URL: https://arxiv.org/pdf/2407.15711
• Copy Paste: [[2407.15711]] AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?(https://arxiv.org/abs/2407.15711)
• Keywords: language model, agent
• Abstract: Language agents, built on top of language models (LMs), are systems that can interact with complex environments, such as the open web. In this work, we examine whether such agents can perform realistic and time-consuming tasks on the web, e.g., monitoring real-estate markets or locating relevant nearby businesses. We introduce AssistantBench, a challenging new benchmark consisting of 214 realistic tasks that can be automatically evaluated, covering different scenarios and domains. We find that AssistantBench exposes the limitations of current systems, including language models and retrieval-augmented language models, as no model reaches an accuracy of more than 25 points. While closed-book LMs perform well, they exhibit low precision since they tend to hallucinate facts. State-of-the-art web agents reach a score of near zero. Additionally, we introduce SeePlanAct (SPA), a new web agent that significantly outperforms previous agents, and an ensemble of SPA and closed-book models reaches the best overall performance. Moreover, we analyze failures of current systems and highlight that web navigation remains a major challenge.
• 摘要：语言代理建立在语言模型 (LM) 之上，是一种可以与复杂环境（例如开放网络）交互的系统。在这项工作中，我们研究了此类代理是否可以在网络上执行现实且耗时的任务，例如监控房地产市场或定位附近的相关企业。我们引入了 AssistantBench，这是一个具有挑战性的新基准，由 214 个可以自动评估的现实任务组成，涵盖不同的场景和领域。我们发现 AssistantBench 暴露了当前系统（包括语言模型和检索增强语言模型）的局限性，因为没有一个模型的准确率超过 25 分。虽然闭卷 LM 表现良好，但它们的精度较低，因为它们往往会产生幻觉。最先进的网络代理的得分接近零。此外，我们引入了 SeePlanAct (SPA)，这是一种新的网络代理，其性能明显优于以前的代理，SPA 和闭卷模型的组合达到了最佳整体性能。此外，我们分析了当前系统的故障，并强调网络导航仍然是一项重大挑战。

Title: Do Large Language Models Have Compositional Ability? An Investigation into Limitations and Scalability

• Authors: Zhuoyan Xu, Zhenmei Shi, Yingyu Liang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.15720
• Pdf URL: https://arxiv.org/pdf/2407.15720
• Copy Paste: [[2407.15720]] Do Large Language Models Have Compositional Ability? An Investigation into Limitations and Scalability(https://arxiv.org/abs/2407.15720)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have emerged as powerful tools for many AI problems and exhibit remarkable in-context learning (ICL) capabilities. Compositional ability, solving unseen complex tasks that combine two or more simple tasks, is an essential reasoning ability for Artificial General Intelligence. Despite LLM's tremendous success, how they approach composite tasks, especially those not encountered during the pretraining phase, remains an open question and largely ununderstood. In this study, we delve into the ICL capabilities of LLMs on composite tasks, with only simple tasks as in-context examples. We develop a test suite of composite tasks that include linguistic and logical challenges and perform empirical studies across different LLM families. We observe that models exhibit divergent behaviors: (1) For simpler composite tasks that apply distinct mapping mechanisms to different input segments, the models demonstrate decent compositional ability, while scaling up the model enhances this ability; (2) for more complex composite tasks that involving reasoning multiple steps, where each step represent one task, models typically underperform, and scaling up generally provide no improvements. We offer theoretical analysis in a simplified setting, explaining that models exhibit compositional capability when the task handles different input parts separately. We believe our work sheds new light on the capabilities of LLMs in solving composite tasks regarding the nature of the tasks and model scale. Our dataset and code are available at {\url{this https URL}}.
• 摘要：大型语言模型 (LLM) 已成为许多 AI 问题的强大工具，并表现出卓越的上下文学习 (ICL) 能力。组合能力是解决将两个或多个简单任务组合在一起的看不见的复杂任务，是通用人工智能的基本推理能力。尽管 LLM 取得了巨大的成功，但它们如何处理复合任务，尤其是在预训练阶段未遇到的任务，仍然是一个悬而未决的问题，而且在很大程度上还未被理解。在本研究中，我们深入研究了 LLM 在复合任务上的 ICL 能力，仅以简单任务作为上下文示例。我们开发了一套包含语言和逻辑挑战的复合任务测试套件，并在不同的 LLM 系列中进行了实证研究。我们观察到模型表现出不同的行为：(1) 对于将不同的映射机制应用于不同输入段的较简单的复合任务，模型表现出不错的组合能力，而扩大模型可以增强这种能力；(2) 对于涉及推理多个步骤的更复杂的复合任务，其中每个步骤代表一项任务，模型通常表现不佳，而扩大规模通常不会带来任何改进。我们在简化的环境中提供理论分析，解释当任务分别处理不同的输入部分时，模型会表现出组合能力。我们相信我们的工作为 LLM 在解决复合任务方面的能力提供了新的启示，这些能力与任务的性质和模型规模有关。我们的数据集和代码可在 {\url{this https URL}} 上找到。

Title: DStruct2Design: Data and Benchmarks for Data Structure Driven Generative Floor Plan Design

• Authors: Zhi Hao Luo, Luis Lara, Ge Ya Luo, Florian Golemo, Christopher Beckham, Christopher Pal
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.15723
• Pdf URL: https://arxiv.org/pdf/2407.15723
• Copy Paste: [[2407.15723]] DStruct2Design: Data and Benchmarks for Data Structure Driven Generative Floor Plan Design(https://arxiv.org/abs/2407.15723)
• Keywords: language model, llm
• Abstract: Text conditioned generative models for images have yielded impressive results. Text conditioned floorplan generation as a special type of raster image generation task also received particular attention. However there are many use cases in floorpla generation where numerical properties of the generated result are more important than the aesthetics. For instance, one might want to specify sizes for certain rooms in a floorplan and compare the generated floorplan with given specifications Current approaches, datasets and commonly used evaluations do not support these kinds of constraints. As such, an attractive strategy is to generate an intermediate data structure that contains numerical properties of a floorplan which can be used to generate the final floorplan image. To explore this setting we (1) construct a new dataset for this data-structure to data-structure formulation of floorplan generation using two popular image based floorplan datasets RPLAN and ProcTHOR-10k, and provide the tools to convert further procedurally generated ProcTHOR floorplan data into our format. (2) We explore the task of floorplan generation given a partial or complete set of constraints and we design a series of metrics and benchmarks to enable evaluating how well samples generated from models respect the constraints. (3) We create multiple baselines by finetuning a large language model (LLM), Llama3, and demonstrate the feasibility of using floorplan data structure conditioned LLMs for the problem of floorplan generation respecting numerical constraints. We hope that our new datasets and benchmarks will encourage further research on different ways to improve the performance of LLMs and other generative modelling techniques for generating designs where quantitative constraints are only partially specified, but must be respected.
• 摘要：文本条件图像生成模型已经产生了令人印象深刻的结果。文本条件平面图生成作为一种特殊类型的光栅图像生成任务也受到了特别的关注。然而，在平面图生成中有很多用例，其中生成结果的数值属性比美学更重要。例如，人们可能想要指定平面图中某些房间的大小，并将生成的平面图与给定的规格进行比较。当前的方法、数据集和常用的评估不支持这些类型的约束。因此，一个有吸引力的策略是生成一个中间数据结构，其中包含平面图的数值属性，可用于生成最终的平面图图像。为了探索这种设置，我们（1）使用两个流行的基于图像的平面图数据集 RPLAN 和 ProcTHOR-10k 为这种数据结构构建一个新的数据集到平面图生成的数据结构公式，并提供工具将进一步程序生成的 ProcTHOR 平面图数据转换为我们的格式。 (2) 我们探索给定部分或完整约束集的布局规划生成任务，并设计一系列指标和基准，以便评估模型生成的样本对约束的遵守情况。 (3) 我们通过微调大型语言模型 (LLM) Llama3 创建多个基线，并证明使用布局规划数据结构条件 LLM 解决遵守数值约束的布局规划生成问题的可行性。 我们希望我们的新数据集和基准将鼓励进一步研究提高 LLM 和其他生成建模技术性能的不同方法，以生成仅部分指定但必须遵守定量约束的设计。

Title: OMoS-QA: A Dataset for Cross-Lingual Extractive Question Answering in a German Migration Context

• Authors: Steffen Kleinle, Jakob Prange, Annemarie Friedrich
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.15736
• Pdf URL: https://arxiv.org/pdf/2407.15736
• Copy Paste: [[2407.15736]] OMoS-QA: A Dataset for Cross-Lingual Extractive Question Answering in a German Migration Context(https://arxiv.org/abs/2407.15736)
• Keywords: language model, llm
• Abstract: When immigrating to a new country, it is easy to feel overwhelmed by the need to obtain information on financial support, housing, schooling, language courses, and other issues. If relocation is rushed or even forced, the necessity for high-quality answers to such questions is all the more urgent. Official immigration counselors are usually overbooked, and online systems could guide newcomers to the requested information or a suitable counseling service. To this end, we present OMoS-QA, a dataset of German and English questions paired with relevant trustworthy documents and manually annotated answers, specifically tailored to this scenario. Questions are automatically generated with an open-source large language model (LLM) and answer sentences are selected by crowd workers with high agreement. With our data, we conduct a comparison of 5 pretrained LLMs on the task of extractive question answering (QA) in German and English. Across all models and both languages, we find high precision and low-to-mid recall in selecting answer sentences, which is a favorable trade-off to avoid misleading users. This performance even holds up when the question language does not match the document language. When it comes to identifying unanswerable questions given a context, there are larger differences between the two languages.
• 摘要：移民到新国家时，很容易因为需要获取有关经济支持、住房、学校教育、语言课程和其他问题的信息而感到不知所措。如果搬迁是匆忙甚至是被迫的，那么对这些问题的高质量答案的需求就更加迫切。官方移民顾问通常都超员了，在线系统可以引导新移民获得所需的信息或合适的咨询服务。为此，我们提出了 OMoS-QA，这是一个德语和英语问题的数据集，配有相关的可信文档和手动注释的答案，专门针对这种情况量身定制。问题由开源大型语言模型 (LLM) 自动生成，答案句子由众包工作者以高度一致的标准选择。利用我们的数据，我们对 5 个预训练的 LLM 进行了比较，以完成德语和英语的提取式问答 (QA) 任务。在所有模型和两种语言中，我们发现在选择答案句子时，准确率很高，召回率低到中等，这是一种有利的权衡，可以避免误导用户。即使问题语言与文档语言不匹配，这种表现仍然有效。当涉及到根据上下文识别无法回答的问题时，两种语言之间的差异更大。

Title: Extracting Structured Insights from Financial News: An Augmented LLM Driven Approach

• Authors: Rian Dolphin, Joe Dursun, Jonathan Chow, Jarrett Blankenship, Katie Adams, Quinton Pike
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.15788
• Pdf URL: https://arxiv.org/pdf/2407.15788
• Copy Paste: [[2407.15788]] Extracting Structured Insights from Financial News: An Augmented LLM Driven Approach(https://arxiv.org/abs/2407.15788)
• Keywords: language model, llm, prompt
• Abstract: Financial news plays a crucial role in decision-making processes across the financial sector, yet the efficient processing of this information into a structured format remains challenging. This paper presents a novel approach to financial news processing that leverages Large Language Models (LLMs) to overcome limitations that previously prevented the extraction of structured data from unstructured financial news. We introduce a system that extracts relevant company tickers from raw news article content, performs sentiment analysis at the company level, and generates summaries, all without relying on pre-structured data feeds. Our methodology combines the generative capabilities of LLMs, and recent prompting techniques, with a robust validation framework that uses a tailored string similarity approach. Evaluation on a dataset of 5530 financial news articles demonstrates the effectiveness of our approach, with 90% of articles not missing any tickers compared with current data providers, and 22% of articles having additional relevant tickers. In addition to this paper, the methodology has been implemented at scale with the resulting processed data made available through a live API endpoint, which is updated in real-time with the latest news. To the best of our knowledge, we are the first data provider to offer granular, per-company sentiment analysis from news articles, enhancing the depth of information available to market participants. We also release the evaluation dataset of 5530 processed articles as a static file, which we hope will facilitate further research leveraging financial news.
• 摘要：金融新闻在整个金融行业的决策过程中发挥着至关重要的作用，但将这些信息高效地处理成结构化格式仍然具有挑战性。本文介绍了一种新颖的金融新闻处理方法，该方法利用大型语言模型 (LLM) 来克服以前阻碍从非结构化金融新闻中提取结构化数据的限制。我们引入了一个系统，该系统从原始新闻文章内容中提取相关的公司股票代码，在公司层面进行情绪分析，并生成摘要，所有这些都不依赖于预先结构化的数据馈送。我们的方法结合了 LLM 的生成能力和最近的提示技术，以及使用定制字符串相似性方法的强大验证框架。对 5530 篇金融新闻文章的数据集的评估证明了我们方法的有效性，与当前数据提供商相比，90% 的文章没有缺少任何股票代码，22% 的文章有额外的相关股票代码。除了本文之外，该方法还已大规模实施，处理后的数据通过实时 API 端点提供，该端点会实时更新最新消息。据我们所知，我们是第一家提供新闻文章中每家公司详细情绪分析的数据提供商，这提高了市场参与者可获得的信息深度。我们还以静态文件的形式发布了 5530 篇经过处理的文章的评估数据集，我们希望这将有助于进一步利用金融新闻进行研究。

Title: Perceptions of Linguistic Uncertainty by Language Models and Humans

• Authors: Catarina G Belem, Markelle Kelly, Mark Steyvers, Sameer Singh, Padhraic Smyth
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.15814
• Pdf URL: https://arxiv.org/pdf/2407.15814
• Copy Paste: [[2407.15814]] Perceptions of Linguistic Uncertainty by Language Models and Humans(https://arxiv.org/abs/2407.15814)
• Keywords: language model, agent
• Abstract: Uncertainty expressions such as ``probably'' or ``highly unlikely'' are pervasive in human language. While prior work has established that there is population-level agreement in terms of how humans interpret these expressions, there has been little inquiry into the abilities of language models to interpret such expressions. In this paper, we investigate how language models map linguistic expressions of uncertainty to numerical responses. Our approach assesses whether language models can employ theory of mind in this setting: understanding the uncertainty of another agent about a particular statement, independently of the model's own certainty about that statement. We evaluate both humans and 10 popular language models on a task created to assess these abilities. Unexpectedly, we find that 8 out of 10 models are able to map uncertainty expressions to probabilistic responses in a human-like manner. However, we observe systematically different behavior depending on whether a statement is actually true or false. This sensitivity indicates that language models are substantially more susceptible to bias based on their prior knowledge (as compared to humans). These findings raise important questions and have broad implications for human-AI alignment and AI-AI communication.
• 摘要：诸如“可能”或“极不可能”之类的不确定性表达在人类语言中随处可见。虽然先前的研究已经证实，在人类如何解释这些表达方面存在群体层面的一致性，但很少有人研究语言模型解释此类表达的能力。在本文中，我们研究了语言模型如何将不确定性的语言表达映射到数字响应。我们的方法评估语言模型是否可以在这种情况下运用心智理论：理解另一个代理对特定陈述的不确定性，而不受模型自身对该陈述的确定性的影响。我们在一项旨在评估这些能力的任务上评估了人类和 10 种流行的语言模型。出乎意料的是，我们发现 10 个模型中有 8 个能够以类似人类的方式将不确定性表达映射到概率响应。然而，我们观察到系统地根据陈述是真是假而有不同的行为。这种敏感性表明，与人类相比，语言模型更容易受到基于其先验知识的偏见的影响。这些发现提出了重要的问题，对人机协作和人工智能之间的通信具有广泛的影响。

Title: J-CHAT: Japanese Large-scale Spoken Dialogue Corpus for Spoken Dialogue Language Modeling

• Authors: Wataru Nakata, Kentaro Seki, Hitomi Yanaka, Yuki Saito, Shinnosuke Takamichi, Hiroshi Saruwatari
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2407.15828
• Pdf URL: https://arxiv.org/pdf/2407.15828
• Copy Paste: [[2407.15828]] J-CHAT: Japanese Large-scale Spoken Dialogue Corpus for Spoken Dialogue Language Modeling(https://arxiv.org/abs/2407.15828)
• Keywords: language model, chat
• Abstract: Spoken dialogue plays a crucial role in human-AI interactions, necessitating dialogue-oriented spoken language models (SLMs). To develop versatile SLMs, large-scale and diverse speech datasets are essential. Additionally, to ensure hiqh-quality speech generation, the data must be spontaneous like in-wild data and must be acoustically clean with noise removed. Despite the critical need, no open-source corpus meeting all these criteria has been available. This study addresses this gap by constructing and releasing a large-scale spoken dialogue corpus, named Japanese Corpus for Human-AI Talks (J-CHAT), which is publicly accessible. Furthermore, this paper presents a language-independent method for corpus construction and describes experiments on dialogue generation using SLMs trained on J-CHAT. Experimental results indicate that the collected data from multiple domains by our method improve the naturalness and meaningfulness of dialogue generation.
• 摘要：口语对话在人机交互中起着至关重要的作用，因此需要面向对话的口语语言模型 (SLM)。要开发多功能的 SLM，大规模和多样化的语音数据集必不可少。此外，为了确保高质量的语音生成，数据必须像自然数据一样自然，并且必须在消除噪音的情况下保持声学纯净。尽管迫切需要，但目前还没有满足所有这些标准的开源语料库。本研究通过构建和发布一个大规模口语对话语料库（称为日语人机对话语料库 (J-CHAT)）来解决这一空白，该语料库可公开访问。此外，本文提出了一种独立于语言的语料库构建方法，并描述了使用在 J-CHAT 上训练的 SLM 进行对话生成的实验。实验结果表明，通过我们的方法从多个领域收集的数据提高了对话生成的自然性和意义。

Title: dMel: Speech Tokenization made Simple

• Authors: He Bai, Tatiana Likhomanenko, Ruixiang Zhang, Zijin Gu, Zakaria Aldeneh, Navdeep Jaitly
• Subjects: cs.CL, cs.AI, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2407.15835
• Pdf URL: https://arxiv.org/pdf/2407.15835
• Copy Paste: [[2407.15835]] dMel: Speech Tokenization made Simple(https://arxiv.org/abs/2407.15835)
• Keywords: language model
• Abstract: Large language models have revolutionized natural language processing by leveraging self-supervised pretraining on vast textual data. Inspired by this success, researchers have investigated complicated speech tokenization methods to discretize continuous speech signals so that language modeling techniques can be applied to speech data. However, existing approaches either model semantic tokens, potentially losing acoustic information, or model acoustic tokens, risking the loss of semantic information. Having multiple token types also complicates the architecture and requires additional pretraining. Here we show that discretizing mel-filterbank channels into discrete intensity bins produces a simple representation (dMel), that performs better than other existing speech tokenization methods. Using a transformer decoder-only architecture for speech-text modeling, we comprehensively evaluate different speech tokenization methods on speech recognition (ASR), speech synthesis (TTS). Our results demonstrate the effectiveness of dMel in achieving high performance on both tasks within a unified framework, paving the way for efficient and effective joint modeling of speech and text.
• 摘要：大型语言模型通过利用对大量文本数据的自监督预训练，彻底改变了自然语言处理。受此成功的启发，研究人员研究了复杂的语音标记化方法来离散化连续语音信号，以便将语言建模技术应用于语音数据。然而，现有的方法要么对语义标记进行建模，这可能会丢失声学信息，要么对声学标记进行建模，这可能会丢失语义信息。拥有多种标记类型也会使架构复杂化，并需要额外的预训练。在这里，我们展示了将梅尔滤波器组通道离散化为离散强度箱会产生一个简单的表示 (dMel)，其性能优于其他现有的语音标记化方法。我们使用仅使用 Transformer 解码器的架构进行语音文本建模，全面评估了语音识别 (ASR)、语音合成 (TTS) 上的不同语音标记化方法。我们的结果证明了 dMel 在统一框架内实现两项任务的高性能方面的有效性，为高效、有效的语音和文本联合建模铺平了道路。