2024-04-12

Title: We're Calling an Intervention: Taking a Closer Look at Language Model Adaptation to Different Types of Linguistic Variation

• Authors: Aarohi Srivastava, David Chiang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.07304
• Pdf URL: https://arxiv.org/pdf/2404.07304
• Copy Paste: [[2404.07304]] We're Calling an Intervention: Taking a Closer Look at Language Model Adaptation to Different Types of Linguistic Variation(https://arxiv.org/abs/2404.07304)
• Keywords: language model
• Abstract: We present a suite of interventions and experiments that allow us to understand language model adaptation to text with linguistic variation (e.g., nonstandard or dialectal text). Our interventions address several features of linguistic variation, resulting in character, subword, and word-level changes. Applying our interventions during language model adaptation with varying size and nature of training data, we gain important insights into what makes linguistic variation particularly difficult for language models to deal with. For instance, on text with character-level variation, performance improves with even a few training examples but approaches a plateau, suggesting that more data is not the solution. In contrast, on text with variation involving new words or meanings, far more data is needed, but it leads to a massive breakthrough in performance. Our findings inform future work on dialectal NLP and making language models more robust to linguistic variation overall. We make the code for our interventions, which can be applied to any English text data, publicly available.
• 摘要：我们提出了一系列干预措施和实验，使我们能够理解语言模型对具有语言变异的文本（例如非标准或方言文本）的适应。我们的干预措施解决了语言变异的几个特征，导致字符、子词和词级的变化。通过在不同大小和性质的训练数据的语言模型适应过程中应用我们的干预措施，我们获得了重要的见解，了解是什么使语言模型特别难以处理语言变异。例如，对于具有字符级变化的文本，即使使用几个训练示例，性能也会有所提高，但会趋于稳定，这表明更多的数据并不是解决方案。相比之下，对于涉及新单词或含义的变化的文本，需要更多的数据，但这会带来性能的巨大突破。我们的研究结果为方言 NLP 的未来工作提供了信息，并使语言模型对整体语言变化更加稳健。我们为我们的干预措施编写了代码，该代码可以应用于任何公开的英文文本数据。

Title: LLMs in Biomedicine: A study on clinical Named Entity Recognition

• Authors: Masoud Monajatipoor, Jiaxin Yang, Joel Stremmel, Melika Emami, Fazlolah Mohaghegh, Mozhdeh Rouhsedaghat, Kai-Wei Chang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.07376
• Pdf URL: https://arxiv.org/pdf/2404.07376
• Copy Paste: [[2404.07376]] LLMs in Biomedicine: A study on clinical Named Entity Recognition(https://arxiv.org/abs/2404.07376)
• Keywords: language model, llm, prompt, retrieval-augmented generation
• Abstract: Large Language Models (LLMs) demonstrate remarkable versatility in various NLP tasks but encounter distinct challenges in biomedicine due to medical language complexities and data scarcity. This paper investigates the application of LLMs in the medical domain by exploring strategies to enhance their performance for the Named-Entity Recognition (NER) task. Specifically, our study reveals the importance of meticulously designed prompts in biomedicine. Strategic selection of in-context examples yields a notable improvement, showcasing ~15-20\% increase in F1 score across all benchmark datasets for few-shot clinical NER. Additionally, our findings suggest that integrating external resources through prompting strategies can bridge the gap between general-purpose LLM proficiency and the specialized demands of medical NER. Leveraging a medical knowledge base, our proposed method inspired by Retrieval-Augmented Generation (RAG) can boost the F1 score of LLMs for zero-shot clinical NER. We will release the code upon publication.
• 摘要：大型语言模型 (LLM) 在各种 NLP 任务中表现出显着的多功能性，但由于医学语言的复杂性和数据稀缺性，在生物医学领域遇到了明显的挑战。本文通过探索提高法学硕士在命名实体识别（NER）任务中的性能的策略，研究了法学硕士在医学领域的应用。具体来说，我们的研究揭示了精心设计的提示在生物医学中的重要性。对上下文中示例的策略选择产生了显着的改进，在小样本临床 NER 的所有基准数据集中，F1 分数增加了约 15-20%。此外，我们的研究结果表明，通过激励策略整合外部资源可以弥合通用 LLM 熟练程度和医学 NER 的专业需求之间的差距。利用医学知识库，我们提出的受检索增强生成（RAG）启发的方法可以提高零样本临床 NER 的法学硕士的 F1 分数。我们将在发布后发布代码。

Title: JetMoE: Reaching Llama2 Performance with 0.1M Dollars

• Authors: Yikang Shen, Zhen Guo, Tianle Cai, Zengyi Qin
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.07413
• Pdf URL: https://arxiv.org/pdf/2404.07413
• Copy Paste: [[2404.07413]] JetMoE: Reaching Llama2 Performance with 0.1M Dollars(https://arxiv.org/abs/2404.07413)
• Keywords: language model, llm, chat
• Abstract: Large Language Models (LLMs) have achieved remarkable results, but their increasing resource demand has become a major obstacle to the development of powerful and accessible super-human intelligence. This report introduces JetMoE-8B, a new LLM trained with less than $0.1 million, using 1.25T tokens from carefully mixed open-source corpora and 30,000 H100 GPU hours. Despite its low cost, the JetMoE-8B demonstrates impressive performance, with JetMoE-8B outperforming the Llama2-7B model and JetMoE-8B-Chat surpassing the Llama2-13B-Chat model. These results suggest that LLM training can be much more cost-effective than generally thought. JetMoE-8B is based on an efficient Sparsely-gated Mixture-of-Experts (SMoE) architecture, composed of attention and feedforward experts. Both layers are sparsely activated, allowing JetMoE-8B to have 8B parameters while only activating 2B for each input token, reducing inference computation by about 70% compared to Llama2-7B. Moreover, JetMoE-8B is highly open and academia-friendly, using only public datasets and training code. All training parameters and data mixtures have been detailed in this report to facilitate future efforts in the development of open foundation models. This transparency aims to encourage collaboration and further advancements in the field of accessible and efficient LLMs. The model weights are publicly available at https://github.com/myshell-ai/JetMoE.
• 摘要：大型语言模型（LLM）取得了显着的成果，但其不断增长的资源需求已成为开发强大且易于使用的超人类智能的主要障碍。本报告介绍了 JetMoE-8B，这是一种新的法学硕士，培训费用不到 10 万美元，使用来自精心混合的开源语料库的 1.25T 代币和 30,000 个 H100 GPU 小时。尽管成本低廉，JetMoE-8B 却表现出了令人印象深刻的性能，JetMoE-8B 的性能优于 Llama2-7B 模型，JetMoE-8B-Chat 的性能优于 Llama2-13B-Chat 模型。这些结果表明法学硕士培训比通常认为的更具成本效益。 JetMoE-8B 基于高效的稀疏门混合专家 (SMoE) 架构，由注意力和前馈专家组成。两个层都是稀疏激活的，允许 JetMoE-8B 具有 8B 个参数，同时为每个输入标记仅激活 2B 个参数，与 Llama2-7B 相比，推理计算减少了约 70%。此外，JetMoE-8B 高度开放且对学术界友好，仅使用公共数据集和训练代码。本报告详细介绍了所有训练参数和数据混合，以促进未来开发开放基础模型的工作。这种透明度旨在鼓励可访问且高效的法学硕士领域的合作和进一步进步。模型权重可在 https://github.com/myshell-ai/JetMoE 上公开获取。

Title: "Confidently Nonsensical?'': A Critical Survey on the Perspectives and Challenges of 'Hallucinations' in NLP

• Authors: Pranav Narayanan Venkit, Tatiana Chakravorti, Vipul Gupta, Heidi Biggs, Mukund Srinath, Koustava Goswami, Sarah Rajtmajer, Shomir Wilson
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.07461
• Pdf URL: https://arxiv.org/pdf/2404.07461
• Copy Paste: [[2404.07461]] "Confidently Nonsensical?'': A Critical Survey on the Perspectives and Challenges of 'Hallucinations' in NLP(https://arxiv.org/abs/2404.07461)
• Keywords: language model, llm, hallucination
• Abstract: We investigate how hallucination in large language models (LLM) is characterized in peer-reviewed literature using a critical examination of 103 publications across NLP research. Through a comprehensive review of sociological and technological literature, we identify a lack of agreement with the term `hallucination.' Additionally, we conduct a survey with 171 practitioners from the field of NLP and AI to capture varying perspectives on hallucination. Our analysis underscores the necessity for explicit definitions and frameworks outlining hallucination within NLP, highlighting potential challenges, and our survey inputs provide a thematic understanding of the influence and ramifications of hallucination in society.
• 摘要：我们通过对 NLP 研究中的 103 篇出版物进行严格审查，研究了同行评审文献中大语言模型 (LLM) 中幻觉的特征。通过对社会学和技术文献的全面回顾，我们发现人们对“幻觉”一词缺乏共识。此外，我们还对来自 NLP 和 AI 领域的 171 位从业者进行了一项调查，以了解对幻觉的不同看法。我们的分析强调了在 NLP 中概述幻觉的明确定义和框架的必要性，强调了潜在的挑战，我们的调查投入提供了对幻觉在社会中的影响和后果的主题理解。

Title: Scalable Language Model with Generalized Continual Learning

• Authors: Bohao Peng, Zhuotao Tian, Shu Liu, Mingchang Yang, Jiaya Jia
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.07470
• Pdf URL: https://arxiv.org/pdf/2404.07470
• Copy Paste: [[2404.07470]] Scalable Language Model with Generalized Continual Learning(https://arxiv.org/abs/2404.07470)
• Keywords: language model
• Abstract: Continual learning has gained increasing importance as it facilitates the acquisition and refinement of scalable knowledge and skills in language models. However, existing methods typically encounter strict limitations and challenges in real-world scenarios, such as reliance on experience replay, optimization constraints, and inference task-ID. In this study, we introduce the Scalable Language Model (SLM) to overcome these limitations within a more challenging and generalized setting, representing a significant advancement toward practical applications for continual learning. Specifically, we propose the Joint Adaptive Re-Parameterization (JARe), integrated with Dynamic Task-related Knowledge Retrieval (DTKR), to enable adaptive adjustment of language models based on specific downstream tasks. This approach leverages the task distribution within the vector space, aiming to achieve a smooth and effortless continual learning process. Our method demonstrates state-of-the-art performance on diverse backbones and benchmarks, achieving effective continual learning in both full-set and few-shot scenarios with minimal forgetting. Moreover, while prior research primarily focused on a single task type such as classification, our study goes beyond, with the large language model, i.e., LLaMA-2, to explore the effects across diverse domains and task types, such that a single language model can be decently scaled to broader applications.
• 摘要：持续学习变得越来越重要，因为它有助于获取和完善语言模型中可扩展的知识和技能。然而，现有方法通常在现实场景中遇到严格的限制和挑战，例如依赖经验回放、优化约束和推理任务 ID。在这项研究中，我们引入了可扩展语言模型（SLM），以在更具挑战性和通用性的环境中克服这些限制，代表着持续学习实际应用的重大进步。具体来说，我们提出了联合自适应重新参数化（JARe），与动态任务相关知识检索（DTKR）集成，以实现基于特定下游任务的语言模型的自适应调整。这种方法利用向量空间内的任务分布，旨在实现平稳且轻松的持续学习过程。我们的方法在不同的骨干网和基准上展示了最先进的性能，在全套和少量场景中实现有效的持续学习，并且遗忘最少。此外，虽然之前的研究主要集中在分类等单一任务类型，但我们的研究超越了大型语言模型，即 LLaMA-2，探索了跨不同领域和任务类型的影响，例如单一语言模型可以适当地扩展到更广泛的应用程序。

Title: Laissez-Faire Harms: Algorithmic Biases in Generative Language Models

• Authors: Evan Shieh, Faye-Marie Vassel, Cassidy Sugimoto, Thema Monroe-White
• Subjects: cs.CL, cs.AI, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.07475
• Pdf URL: https://arxiv.org/pdf/2404.07475
• Copy Paste: [[2404.07475]] Laissez-Faire Harms: Algorithmic Biases in Generative Language Models(https://arxiv.org/abs/2404.07475)
• Keywords: language model, gpt, prompt, chat
• Abstract: The rapid deployment of generative language models (LMs) has raised concerns about social biases affecting the well-being of diverse consumers. The extant literature on generative LMs has primarily examined bias via explicit identity prompting. However, prior research on bias in earlier language-based technology platforms, including search engines, has shown that discrimination can occur even when identity terms are not specified explicitly. Studies of bias in LM responses to open-ended prompts (where identity classifications are left unspecified) are lacking and have not yet been grounded in end-consumer harms. Here, we advance studies of generative LM bias by considering a broader set of natural use cases via open-ended prompting. In this "laissez-faire" setting, we find that synthetically generated texts from five of the most pervasive LMs (ChatGPT3.5, ChatGPT4, Claude2.0, Llama2, and PaLM2) perpetuate harms of omission, subordination, and stereotyping for minoritized individuals with intersectional race, gender, and/or sexual orientation identities (AI/AN, Asian, Black, Latine, MENA, NH/PI, Female, Non-binary, Queer). We find widespread evidence of bias to an extent that such individuals are hundreds to thousands of times more likely to encounter LM-generated outputs that portray their identities in a subordinated manner compared to representative or empowering portrayals. We also document a prevalence of stereotypes (e.g. perpetual foreigner) in LM-generated outputs that are known to trigger psychological harms that disproportionately affect minoritized individuals. These include stereotype threat, which leads to impaired cognitive performance and increased negative self-perception. Our findings highlight the urgent need to protect consumers from discriminatory harms caused by language models and invest in critical AI education programs tailored towards empowering diverse consumers.
• 摘要：生成语言模型（LM）的快速部署引起了人们对影响不同消费者福祉的社会偏见的担忧。现有的关于生成语言模型的文献主要通过明确的身份提示来检验偏见。然而，先前对早期基于语言的技术平台（包括搜索引擎）中的偏见的研究表明，即使没有明确指定身份术语，歧视也可能发生。关于 LM 对开放式提示（身份分类未指定）的反应偏差的研究还很缺乏，而且尚未以最终消费者的伤害为基础。在这里，我们通过开放式提示考虑更广泛的自然用例，从而推进生成 LM 偏差的研究。在这种“自由放任”的环境中，我们发现来自五个最普遍的语言模型（ChatGPT3.5、ChatGPT4、Claude2.0、Llama2 和 PaLM2）综合生成的文本使少数群体的遗漏、从属和刻板印象永久化。具有交叉种族、性别和/或性取向身份（AI/AN、亚洲人、黑人、拉丁人、中东和北非地区、NH/PI、女性、非二元性别、酷儿）。我们发现存在偏见的广泛证据表明，与代表性或授权性的描绘相比，这些人遇到 LM 生成的以从属方式描绘其身份的输出的可能性要高出数百到数千倍。我们还记录了LM产生的产出中普遍存在的刻板印象（例如永远的外国人），这些刻板印象会引发心理伤害，对少数群体产生不成比例的影响。其中包括刻板印象威胁，这会导致认知能力受损和负面自我认知增加。我们的研究结果强调，迫切需要保护消费者免受语言模型造成的歧视性伤害，并投资于旨在赋予不同消费者权力的关键人工智能教育项目。

Title: Interactive Prompt Debugging with Sequence Salience

• Authors: Ian Tenney, Ryan Mullins, Bin Du, Shree Pandya, Minsuk Kahng, Lucas Dixon
• Subjects: cs.CL, cs.AI, cs.HC, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.07498
• Pdf URL: https://arxiv.org/pdf/2404.07498
• Copy Paste: [[2404.07498]] Interactive Prompt Debugging with Sequence Salience(https://arxiv.org/abs/2404.07498)
• Keywords: llm, prompt, chain-of-thought
• Abstract: We present Sequence Salience, a visual tool for interactive prompt debugging with input salience methods. Sequence Salience builds on widely used salience methods for text classification and single-token prediction, and extends this to a system tailored for debugging complex LLM prompts. Our system is well-suited for long texts, and expands on previous work by 1) providing controllable aggregation of token-level salience to the word, sentence, or paragraph level, making salience over long inputs tractable; and 2) supporting rapid iteration where practitioners can act on salience results, refine prompts, and run salience on the new output. We include case studies showing how Sequence Salience can help practitioners work with several complex prompting strategies, including few-shot, chain-of-thought, and constitutional principles. Sequence Salience is built on the Learning Interpretability Tool, an open-source platform for ML model visualizations, and code, notebooks, and tutorials are available at this http URL
• 摘要：我们推出了 Sequence Salience，这是一种使用输入显着性方法进行交互式提示调试的可视化工具。序列显着性建立在广泛使用的文本分类和单标记预测显着性方法的基础上，并将其扩展到专为调试复杂的 LLM 提示而定制的系统。我们的系统非常适合长文本，并通过以下方式扩展了之前的工作：1）提供对单词、句子或段落级别的标记级显着性的可控聚合，使长输入的显着性易于处理； 2）支持快速迭代，从业者可以根据显着性结果采取行动，完善提示，并对新输出运行显着性。我们提供的案例研究展示了序列显着性如何帮助从业者使用几种复杂的提示策略，包括少样本、思维链和宪法原则。 Sequence Salience 基于 Learning Interpretability Tool（一个用于 ML 模型可视化的开源平台）构建，代码、笔记本和教程可从此 http URL 获取

Title: Best Practices and Lessons Learned on Synthetic Data for Language Models

• Authors: Ruibo Liu, Jerry Wei, Fangyu Liu, Chenglei Si, Yanzhe Zhang, Jinmeng Rao, Steven Zheng, Daiyi Peng, Diyi Yang, Denny Zhou, Andrew M. Dai
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.07503
• Pdf URL: https://arxiv.org/pdf/2404.07503
• Copy Paste: [[2404.07503]] Best Practices and Lessons Learned on Synthetic Data for Language Models(https://arxiv.org/abs/2404.07503)
• Keywords: language model
• Abstract: The success of AI models relies on the availability of large, diverse, and high-quality datasets, which can be challenging to obtain due to data scarcity, privacy concerns, and high costs. Synthetic data has emerged as a promising solution by generating artificial data that mimics real-world patterns. This paper provides an overview of synthetic data research, discussing its applications, challenges, and future directions. We present empirical evidence from prior art to demonstrate its effectiveness and highlight the importance of ensuring its factuality, fidelity, and unbiasedness. We emphasize the need for responsible use of synthetic data to build more powerful, inclusive, and trustworthy language models.
• 摘要：人工智能模型的成功依赖于大型、多样化和高质量数据集的可用性，但由于数据稀缺、隐私问题和高成本，这些数据集的获取可能具有挑战性。通过生成模仿现实世界模式的人工数据，合成数据已成为一种有前途的解决方案。本文概述了合成数据研究，讨论了其应用、挑战和未来方向。我们提供现有技术的经验证据来证明其有效性，并强调确保其真实性、保真度和公正性的重要性。我们强调需要负责任地使用合成数据来构建更强大、更具包容性和值得信赖的语言模型。

Title: From Words to Numbers: Your Large Language Model Is Secretly A Capable Regressor When Given In-Context Examples

• Authors: Robert Vacareanu, Vlad-Andrei Negru, Vasile Suciu, Mihai Surdeanu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.07544
• Pdf URL: https://arxiv.org/pdf/2404.07544
• Copy Paste: [[2404.07544]] From Words to Numbers: Your Large Language Model Is Secretly A Capable Regressor When Given In-Context Examples(https://arxiv.org/abs/2404.07544)
• Keywords: language model, gpt, llm
• Abstract: We analyze how well pre-trained large language models (e.g., Llama2, GPT-4, Claude 3, etc) can do linear and non-linear regression when given in-context examples, without any additional training or gradient updates. Our findings reveal that several large language models (e.g., GPT-4, Claude 3) are able to perform regression tasks with a performance rivaling (or even outperforming) that of traditional supervised methods such as Random Forest, Bagging, or Gradient Boosting. For example, on the challenging Friedman #2 regression dataset, Claude 3 outperforms many supervised methods such as AdaBoost, SVM, Random Forest, KNN, or Gradient Boosting. We then investigate how well the performance of large language models scales with the number of in-context exemplars. We borrow from the notion of regret from online learning and empirically show that LLMs are capable of obtaining a sub-linear regret.
• 摘要：我们分析了预训练的大型语言模型（例如 Llama2、GPT-4、Claude 3 等）在给定上下文示例时可以如何很好地进行线性和非线性回归，而无需任何额外的训练或梯度更新。我们的研究结果表明，几种大型语言模型（例如 GPT-4、Claude 3）能够执行回归任务，其性能可与随机森林、Bagging 或 Gradient Boosting 等传统监督方法相媲美（甚至优于）。例如，在具有挑战性的 Friedman #2 回归数据集上，Claude 3 的性能优于许多监督方法，例如 AdaBoost、SVM、随机森林、KNN 或梯度提升。然后，我们研究大型语言模型的性能随上下文样本数量的变化情况。我们借鉴在线学习的遗憾概念，并凭经验证明法学硕士能够获得亚线性遗憾。

Title: Decomposing Label Space, Format and Discrimination: Rethinking How LLMs Respond and Solve Tasks via In-Context Learning

• Authors: Quanyu Long, Yin Wu, Wenya Wang, Sinno Jialin Pan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.07546
• Pdf URL: https://arxiv.org/pdf/2404.07546
• Copy Paste: [[2404.07546]] Decomposing Label Space, Format and Discrimination: Rethinking How LLMs Respond and Solve Tasks via In-Context Learning(https://arxiv.org/abs/2404.07546)
• Keywords: language model, llm
• Abstract: In-context Learning (ICL) has emerged as a powerful capability alongside the development of scaled-up large language models (LLMs). By instructing LLMs using few-shot demonstrative examples, ICL enables them to perform a wide range of tasks without updating millions of parameters. However, the precise contributions of demonstrations towards improving end-task performance have not been thoroughly investigated in recent analytical studies. In this paper, we empirically decompose the overall performance of ICL into three dimensions, label space, format, and discrimination, and we evaluate four general-purpose LLMs across a diverse range of tasks. Counter-intuitively, we find that the demonstrations have a marginal impact on provoking discriminative knowledge of language models. However, ICL exhibits significant efficacy in regulating the label space and format which helps LLMs to respond in desired label words. We then demonstrate this ability functions similar to detailed instructions for LLMs to follow. We additionally provide an in-depth analysis of the mechanism of retrieval helping with ICL and find that retrieving the most semantically similar examples notably boosts model's discriminative capability.
• 摘要：随着大规模大型语言模型 (LLM) 的发展，上下文学习 (ICL) 已成为一种强大的功能。通过使用少量演示示例指导法学硕士，ICL 使他们能够执行广泛的任务，而无需更新数百万个参数。然而，在最近的分析研究中，演示对提高最终任务绩效的确切贡献尚未得到彻底调查。在本文中，我们根据经验将 ICL 的整体表现分解为三个维度：标签空间、格式和辨别力，并评估了跨不同任务范围的四个通用 LLM。与直觉相反，我们发现演示对于激发语言模型的歧视性知识具有边际影响。然而，ICL 在调节标签空间和格式方面表现出显着的功效，有助于法学硕士以所需的标签词做出反应。然后，我们展示了这种能力的功能，类似于法学硕士需要遵循的详细说明。我们还对 ICL 的检索机制进行了深入分析，发现检索语义上最相似的示例可以显着提高模型的判别能力。

Title: Comments as Natural Logic Pivots: Improve Code Generation via Comment Perspective

• Authors: Yijie Chen, Yijin Liu, Fandong Meng, Yufeng Chen, Jinan Xu, Jie Zhou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.07549
• Pdf URL: https://arxiv.org/pdf/2404.07549
• Copy Paste: [[2404.07549]] Comments as Natural Logic Pivots: Improve Code Generation via Comment Perspective(https://arxiv.org/abs/2404.07549)
• Keywords: language model, gpt, llm, prompt, chain-of-thought
• Abstract: Code generation aims to understand the problem description and generate corresponding code snippets, where existing works generally decompose such complex tasks into intermediate steps by prompting strategies, such as Chain-of-Thought and its variants. While these studies have achieved some success, their effectiveness is highly dependent on the capabilities of advanced Large Language Models (LLMs) such as GPT-4, particularly in terms of API calls, which significantly limits their practical applicability. Consequently, how to enhance the code generation capabilities of small and medium-scale code LLMs without significantly increasing training costs is an appealing challenge. In this paper, we suggest that code comments are the natural logic pivot between natural language and code language and propose using comments to boost the code generation ability of code LLMs. Concretely, we propose MANGO (comMents As Natural loGic pivOts), including a comment contrastive training strategy and a corresponding logical comment decoding strategy. Experiments are performed on HumanEval and MBPP, utilizing StarCoder and WizardCoder as backbone models, and encompassing model parameter sizes between 3B and 7B. The results indicate that MANGO significantly improves the code pass rate based on the strong baselines. Meanwhile, the robustness of the logical comment decoding strategy is notably higher than the Chain-of-thoughts prompting. The code is publicly available at \url{https://github.com/pppa2019/Mango}.
• 摘要：代码生成旨在理解问题描述并生成相应的代码片段，现有的工作通常通过提示策略将此类复杂的任务分解为中间步骤，例如思想链及其变体。虽然这些研究取得了一些成功，但其有效性高度依赖于 GPT-4 等高级大型语言模型 (LLM) 的能力，特别是在 API 调用方面，这极大地限制了它们的实际适用性。因此，如何在不显着增加培训成本的情况下增强中小型代码LLM的代码生成能力是一个颇具吸引力的挑战。在本文中，我们认为代码注释是自然语言和代码语言之间的自然逻辑枢纽，并建议使用注释来提高代码法学硕士的代码生成能力。具体来说，我们提出了MANGO（comMents As Natural loGic pivOts），包括评论对比训练策略和相应的逻辑评论解码策略。实验在 HumanEval 和 MBPP 上进行，利用 StarCoder 和 WizardCoder 作为骨干模型，模型参数大小在 3B 到 7B 之间。结果表明，MANGO 在强基线的基础上显着提高了代码通过率。同时，逻辑评论解码策略的鲁棒性明显高于思想链提示。该代码可在 \url{https://github.com/pppa2019/Mango} 上公开获取。

Title: UltraEval: A Lightweight Platform for Flexible and Comprehensive Evaluation for LLMs

• Authors: Chaoqun He, Renjie Luo, Shengding Hu, Yuanqian Zhao, Jie Zhou, Hanghao Wu, Jiajie Zhang, Xu Han, Zhiyuan Liu, Maosong Sun
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.07584
• Pdf URL: https://arxiv.org/pdf/2404.07584
• Copy Paste: [[2404.07584]] UltraEval: A Lightweight Platform for Flexible and Comprehensive Evaluation for LLMs(https://arxiv.org/abs/2404.07584)
• Keywords: language model, llm, prompt
• Abstract: Evaluation is pivotal for honing Large Language Models (LLMs), pinpointing their capabilities and guiding enhancements. The rapid development of LLMs calls for a lightweight and easy-to-use framework for swift evaluation deployment. However, due to the various implementation details to consider, developing a comprehensive evaluation platform is never easy. Existing platforms are often complex and poorly modularized, hindering seamless incorporation into researcher's workflows. This paper introduces UltraEval, a user-friendly evaluation framework characterized by lightweight, comprehensiveness, modularity, and efficiency. We identify and reimplement three core components of model evaluation (models, data, and metrics). The resulting composability allows for the free combination of different models, tasks, prompts, and metrics within a unified evaluation workflow. Additionally, UltraEval supports diverse models owing to a unified HTTP service and provides sufficient inference acceleration. UltraEval is now available for researchers publicly~\footnote{Website is at \url{https://github.com/OpenBMB/UltraEval}}.
• 摘要：评估对于磨练大型语言模型 (LLM)、查明其能力并指导增强功能至关重要。法学硕士的快速发展需要一个轻量级且易于使用的框架来快速部署评估。然而，由于需要考虑各种实施细节，开发一个全面的评估平​​台绝非易事。现有平台通常很复杂且模块化程度较差，阻碍了无缝融入研究人员的工作流程。本文介绍了UltraEval，一个用户友好的评估框架，具有轻量级、全面性、模块化和高效的特点。我们确定并重新实现模型评估的三个核心组成部分（模型、数据和指标）。由此产生的可组合性允许在统一的评估工作流程中自由组合不同的模型、任务、提示和指标。此外，UltraEval通过统一的HTTP服务支持多样化的模型，并提供足够的推理加速。 UltraEval 现在可供研究人员公开使用~\footnote{网站位于 \url{https://github.com/OpenBMB/UltraEval}}。

Title: NoticIA: A Clickbait Article Summarization Dataset in Spanish

• Authors: Iker García-Ferrero, Begoña Altuna
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.07611
• Pdf URL: https://arxiv.org/pdf/2404.07611
• Copy Paste: [[2404.07611]] NoticIA: A Clickbait Article Summarization Dataset in Spanish(https://arxiv.org/abs/2404.07611)
• Keywords: language model
• Abstract: We present NoticIA, a dataset consisting of 850 Spanish news articles featuring prominent clickbait headlines, each paired with high-quality, single-sentence generative summarizations written by humans. This task demands advanced text understanding and summarization abilities, challenging the models' capacity to infer and connect diverse pieces of information to meet the user's informational needs generated by the clickbait headline. We evaluate the Spanish text comprehension capabilities of a wide range of state-of-the-art large language models. Additionally, we use the dataset to train ClickbaitFighter, a task-specific model that achieves near-human performance in this task.
• 摘要：我们展示了 NoticIA，这是一个由 850 篇西班牙新闻文章组成的数据集，其中包含突出的点击诱饵标题，每篇文章都配有由人类编写的高质量单句生成摘要。这项任务需要高级的文本理解和总结能力，挑战模型推断和连接​​不同信息以满足用户通过点击诱饵标题产生的信息需求的能力。我们评估各种最先进的大型语言模型的西班牙语文本理解能力。此外，我们使用该数据集来训练 ClickbaitFighter，这是一种特定于任务的模型，可在此任务中实现接近人类的表现。

Title: Medical mT5: An Open-Source Multilingual Text-to-Text LLM for The Medical Domain

• Authors: Iker García-Ferrero, Rodrigo Agerri, Aitziber Atutxa Salazar, Elena Cabrio, Iker de la Iglesia, Alberto Lavelli, Bernardo Magnini, Benjamin Molinet, Johana Ramirez-Romero, German Rigau, Jose Maria Villa-Gonzalez, Serena Villata, Andrea Zaninello
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.07613
• Pdf URL: https://arxiv.org/pdf/2404.07613
• Copy Paste: [[2404.07613]] Medical mT5: An Open-Source Multilingual Text-to-Text LLM for The Medical Domain(https://arxiv.org/abs/2404.07613)
• Keywords: language model, llm
• Abstract: Research on language technology for the development of medical applications is currently a hot topic in Natural Language Understanding and Generation. Thus, a number of large language models (LLMs) have recently been adapted to the medical domain, so that they can be used as a tool for mediating in human-AI interaction. While these LLMs display competitive performance on automated medical texts benchmarks, they have been pre-trained and evaluated with a focus on a single language (English mostly). This is particularly true of text-to-text models, which typically require large amounts of domain-specific pre-training data, often not easily accessible for many languages. In this paper, we address these shortcomings by compiling, to the best of our knowledge, the largest multilingual corpus for the medical domain in four languages, namely English, French, Italian and Spanish. This new corpus has been used to train Medical mT5, the first open-source text-to-text multilingual model for the medical domain. Additionally, we present two new evaluation benchmarks for all four languages with the aim of facilitating multilingual research in this domain. A comprehensive evaluation shows that Medical mT5 outperforms both encoders and similarly sized text-to-text models for the Spanish, French, and Italian benchmarks, while being competitive with current state-of-the-art LLMs in English.
• 摘要：用于开发医学应用的语言技术研究是当前自然语言理解和生成领域的热门话题。因此，许多大型语言模型（LLM）最近已适应医学领域，因此它们可以用作人类与人工智能交互的中介工具。虽然这些法学硕士在自动化医学文本基准上表现出有竞争力的表现，但他们已经过预培训和评估，重点关注单一语言（主要是英语）。对于文本到文本模型尤其如此，该模型通常需要大量特定于领域的预训练数据，而许多语言通常不容易访问这些数据。在本文中，我们通过据我们所知编译了四种语言（即英语、法语、意大利语和西班牙语）医学领域最大的多语言语料库来解决这些缺点。这个新语料库已用于训练 Medical mT5，这是医学领域第一个开源文本到文本多语言模型。此外，我们还为所有四种语言提供了两个新的评估基准，旨在促进该领域的多语言研究。综合评估表明，Medical mT5 在西班牙语、法语和意大利语基准方面优于编码器和类似大小的文本到文本模型，同时与当前最先进的英语法学硕士具有竞争力。

Title: Audio Dialogues: Dialogues dataset for audio and music understanding

• Authors: Arushi Goel, Zhifeng Kong, Rafael Valle, Bryan Catanzaro
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2404.07616
• Pdf URL: https://arxiv.org/pdf/2404.07616
• Copy Paste: [[2404.07616]] Audio Dialogues: Dialogues dataset for audio and music understanding(https://arxiv.org/abs/2404.07616)
• Keywords: language model, llm, prompt
• Abstract: Existing datasets for audio understanding primarily focus on single-turn interactions (i.e. audio captioning, audio question answering) for describing audio in natural language, thus limiting understanding audio via interactive dialogue. To address this gap, we introduce Audio Dialogues: a multi-turn dialogue dataset containing 163.8k samples for general audio sounds and music. In addition to dialogues, Audio Dialogues also has question-answer pairs to understand and compare multiple input audios together. Audio Dialogues leverages a prompting-based approach and caption annotations from existing datasets to generate multi-turn dialogues using a Large Language Model (LLM). We evaluate existing audio-augmented large language models on our proposed dataset to demonstrate the complexity and applicability of Audio Dialogues. Our code for generating the dataset will be made publicly available. Detailed prompts and generated dialogues can be found on the demo website https://audiodialogues.github.io/.
• 摘要：现有的音频理解数据集主要侧重于用自然语言描述音频的单轮交互（即音频字幕、音频问答），从而限制了通过交互式对话来理解音频。为了解决这一差距，我们引入了音频对话：一个包含 163.8k 个一般音频声音和音乐样本的多轮对话数据集。除了对话之外，音频对话还具有问答对，可以一起理解和比较多个输入音频。音频对话利用基于提示的方法和现有数据集的字幕注释，使用大型语言模型 (LLM) 生成多轮对话。我们在我们提出的数据集上评估现有的音频增强大型语言模型，以证明音频对话的复杂性和适用性。我们用于生成数据集的代码将公开。详细的提示和生成的对话可以在演示网站 https://audiodialogues.github.io/ 上找到。

Title: Why do small language models underperform? Studying Language Model Saturation via the Softmax Bottleneck

• Authors: Nathan Godey, Éric de la Clergerie, Benoît Sagot
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.07647
• Pdf URL: https://arxiv.org/pdf/2404.07647
• Copy Paste: [[2404.07647]] Why do small language models underperform? Studying Language Model Saturation via the Softmax Bottleneck(https://arxiv.org/abs/2404.07647)
• Keywords: language model
• Abstract: Recent advances in language modeling consist in pretraining highly parameterized neural networks on extremely large web-mined text corpora. Training and inference with such models can be costly in practice, which incentivizes the use of smaller counterparts. However, it has been observed that smaller models can suffer from saturation, characterized as a drop in performance at some advanced point in training followed by a plateau. In this paper, we find that such saturation can be explained by a mismatch between the hidden dimension of smaller models and the high rank of the target contextual probability distribution. This mismatch affects the performance of the linear prediction head used in such models through the well-known softmax bottleneck phenomenon. We measure the effect of the softmax bottleneck in various settings and find that models based on less than 1000 hidden dimensions tend to adopt degenerate latent representations in late pretraining, which leads to reduced evaluation performance.
• 摘要：语言建模的最新进展在于在极大的网络挖掘文本语料库上预训练高度参数化的神经网络。在实践中，使用此类模型进行训练和推理可能成本高昂，这会激励使用较小的模型。然而，据观察，较小的模型可能会出现饱和，其特征是在训练的某些高级点性能下降，然后进入稳定状态。在本文中，我们发现这种饱和可以通过较小模型的隐藏维度与目标上下文概率分布的高秩之间的不匹配来解释。这种不匹配通过众所周知的 softmax 瓶颈现象影响此类模型中使用的线性预测头的性能。我们测量了 softmax 瓶颈在各种设置中的影响，发现基于少于 1000 个隐藏维度的模型往往在后期预训练中采用退化的潜在表示，从而导致评估性能下降。

Title: rollama: An R package for using generative large language models through Ollama

• Authors: Johannes B. Gruber, Maximilian Weber
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.07654
• Pdf URL: https://arxiv.org/pdf/2404.07654
• Copy Paste: [[2404.07654]] rollama: An R package for using generative large language models through Ollama(https://arxiv.org/abs/2404.07654)
• Keywords: language model, llm
• Abstract: rollama is an R package that wraps the Ollama API, which allows you to run different Generative Large Language Models (GLLM) locally. The package and learning material focus on making it easy to use Ollama for annotating textual or imagine data with open-source models as well as use these models for document embedding. But users can use or extend rollama to do essentially anything else that is possible through OpenAI's API, yet more private, reproducible and for free.
• 摘要：rollama 是一个包装 Ollama API 的 R 包，它允许您在本地运行不同的生成大型语言模型 (GLLM)。该软件包和学习材料的重点是让 Ollama 能够轻松地使用 Ollama 通过开源模型注释文本或图像数据，以及使用这些模型进行文档嵌入。但用户可以使用或扩展 rollama 来完成通过 OpenAI 的 API 可以完成的任何其他事情，而且更加私密、可重复且免费。

Title: ODA: Observation-Driven Agent for integrating LLMs and Knowledge Graphs

• Authors: Lei Sun, Zhengwei Tao, Youdi Li, Hiroshi Arakawa
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.07677
• Pdf URL: https://arxiv.org/pdf/2404.07677
• Copy Paste: [[2404.07677]] ODA: Observation-Driven Agent for integrating LLMs and Knowledge Graphs(https://arxiv.org/abs/2404.07677)
• Keywords: language model, llm, agent
• Abstract: The integration of Large Language Models (LLMs) and knowledge graphs (KGs) has achieved remarkable success in various natural language processing tasks. However, existing methodologies that integrate LLMs and KGs often navigate the task-solving process solely based on the LLM's analysis of the question, overlooking the rich cognitive potential inherent in the vast knowledge encapsulated in KGs. To address this, we introduce Observation-Driven Agent (ODA), a novel AI agent framework tailored for tasks involving KGs. ODA incorporates KG reasoning abilities via global observation that enhances reasoning capabilities through a cyclical paradigm of observation, action, and reflection. Confronting the exponential explosion of knowledge during observation, we innovatively design a recursive observation mechanism. Subsequently, we integrate the observed knowledge into the action and reflection modules. Through extensive experiments, ODA demonstrates state-of-the-art performance on several datasets, notably achieving accuracy improvements of 12.87% and 8.9%.
• 摘要：大型语言模型（LLM）和知识图（KG）的集成在各种自然语言处理任务中取得了显着的成功。然而，整合法学硕士和知识图谱的现有方法通常仅根据法学硕士对问题的分析来引导任务解决过程，而忽视了知识图谱中封装的大量知识所固有的丰富认知潜力。为了解决这个问题，我们引入了观察驱动代理（ODA），这是一种专为涉及知识图谱的任务量身定制的新型人工智能代理框架。 ODA通过全局观察结合KG推理能力，通过观察、行动和反思的循环范式增强推理能力。面对观察过程中知识的指数爆炸，我们创新性地设计了递归观察机制。随后，我们将观察到的知识整合到行动和反思模块中。通过广泛的实验，ODA 在多个数据集上展示了最先进的性能，特别是准确率提高了 12.87% 和 8.9%。

Title: Automatic Generation and Evaluation of Reading Comprehension Test Items with Large Language Models

• Authors: Andreas Säuberli, Simon Clematide
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.07720
• Pdf URL: https://arxiv.org/pdf/2404.07720
• Copy Paste: [[2404.07720]] Automatic Generation and Evaluation of Reading Comprehension Test Items with Large Language Models(https://arxiv.org/abs/2404.07720)
• Keywords: language model, gpt, llm
• Abstract: Reading comprehension tests are used in a variety of applications, reaching from education to assessing the comprehensibility of simplified texts. However, creating such tests manually and ensuring their quality is difficult and time-consuming. In this paper, we explore how large language models (LLMs) can be used to generate and evaluate multiple-choice reading comprehension items. To this end, we compiled a dataset of German reading comprehension items and developed a new protocol for human and automatic evaluation, including a metric we call text informativity, which is based on guessability and answerability. We then used this protocol and the dataset to evaluate the quality of items generated by Llama 2 and GPT-4. Our results suggest that both models are capable of generating items of acceptable quality in a zero-shot setting, but GPT-4 clearly outperforms Llama 2. We also show that LLMs can be used for automatic evaluation by eliciting item reponses from them. In this scenario, evaluation results with GPT-4 were the most similar to human annotators. Overall, zero-shot generation with LLMs is a promising approach for generating and evaluating reading comprehension test items, in particular for languages without large amounts of available data.
• 摘要：阅读理解测试有多种应用，从教育到评估简化文本的可理解性。然而，手动创建此类测试并确保其质量既困难又耗时。在本文中，我们探讨了如何使用大型语言模型（LLM）来生成和评估多项选择阅读理解项目。为此，我们编制了德语阅读理解项目的数据集，并开发了一种用于人工和自动评估的新协议，其中包括一个我们称为文本信息性的指标，该指标基于可猜测性和可回答性。然后，我们使用该协议和数据集来评估 Llama 2 和 GPT-4 生成的项目的质量。我们的结果表明，两种模型都能够在零样本设置中生成可接受质量的项目，但 GPT-4 明显优于 Llama 2。我们还表明，LLM 可以通过引出项目响应来用于自动评估。在这种情况下，GPT-4 的评估结果与人类注释者最相似。总体而言，法学硕士的零样本生成是一种很有前途的生成和评估阅读理解测试项目的方法，特别是对于没有大量可用数据的语言。

Title: ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models

• Authors: Jinheon Baek, Sujay Kumar Jauhar, Silviu Cucerzan, Sung Ju Hwang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.07738
• Pdf URL: https://arxiv.org/pdf/2404.07738
• Copy Paste: [[2404.07738]] ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models(https://arxiv.org/abs/2404.07738)
• Keywords: language model, agent
• Abstract: Scientific Research, vital for improving human life, is hindered by its inherent complexity, slow pace, and the need for specialized experts. To enhance its productivity, we propose a ResearchAgent, a large language model-powered research idea writing agent, which automatically generates problems, methods, and experiment designs while iteratively refining them based on scientific literature. Specifically, starting with a core paper as the primary focus to generate ideas, our ResearchAgent is augmented not only with relevant publications through connecting information over an academic graph but also entities retrieved from an entity-centric knowledge store based on their underlying concepts, mined and shared across numerous papers. In addition, mirroring the human approach to iteratively improving ideas with peer discussions, we leverage multiple ReviewingAgents that provide reviews and feedback iteratively. Further, they are instantiated with human preference-aligned large language models whose criteria for evaluation are derived from actual human judgments. We experimentally validate our ResearchAgent on scientific publications across multiple disciplines, showcasing its effectiveness in generating novel, clear, and valid research ideas based on human and model-based evaluation results.
• 摘要：科学研究对于改善人类生活至关重要，但因其固有的复杂性、缓慢的节奏以及对专业专家的需求而受到阻碍。为了提高其生产力，我们提出了一个 ResearchAgent，一个由大型语言模型驱动的研究想法写作代理，它自动生成问题、方法和实验设计，同时根据科学文献迭代地完善它们。具体来说，从核心论文作为产生想法的主要焦点开始，我们的 ResearchAgent 不仅通过通过学术图表连接信息来增强相关出版物，而且还根据其基本概念、挖掘和分析从以实体为中心的知识存储中检索实体。在许多论文中共享。此外，为了反映人类通过同行讨论迭代改进想法的方法，我们利用多个评审代理来迭代地提供评审和反馈。此外，它们是用符合人类偏好的大型语言模型来实例化的，其评估标准源自实际的人类判断。我们在多个学科的科学出版物上对我们的 ResearchAgent 进行了实验验证，展示了它在基于人类和基于模型的评估结果产生新颖、清晰和有效的研究想法方面的有效性。

Title: Discourse-Aware In-Context Learning for Temporal Expression Normalization

• Authors: Akash Kumar Gautam, Lukas Lange, Jannik Strötgen
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.07775
• Pdf URL: https://arxiv.org/pdf/2404.07775
• Copy Paste: [[2404.07775]] Discourse-Aware In-Context Learning for Temporal Expression Normalization(https://arxiv.org/abs/2404.07775)
• Keywords: language model, llm, prompt
• Abstract: Temporal expression (TE) normalization is a well-studied problem. However, the predominately used rule-based systems are highly restricted to specific settings, and upcoming machine learning approaches suffer from a lack of labeled data. In this work, we explore the feasibility of proprietary and open-source large language models (LLMs) for TE normalization using in-context learning to inject task, document, and example information into the model. We explore various sample selection strategies to retrieve the most relevant set of examples. By using a window-based prompt design approach, we can perform TE normalization across sentences, while leveraging the LLM knowledge without training the model. Our experiments show competitive results to models designed for this task. In particular, our method achieves large performance improvements for non-standard settings by dynamically including relevant examples during inference.
• 摘要：时间表达（TE）标准化是一个经过充分研究的问题。然而，主要使用的基于规则的系统高度受限于特定设置，并且即将推出的机器学习方法缺乏标记数据。在这项工作中，我们探索了使用上下文学习将任务、文档和示例信息注入模型中进行 TE 规范化的专有和开源大语言模型 (LLM) 的可行性。我们探索各种样本选择策略来检索最相关的示例集。通过使用基于窗口的提示设计方法，我们可以跨句子执行 TE 标准化，同时利用 LLM 知识而无需训练模型。我们的实验显示了与为此任务设计的模型相比具有竞争力的结果。特别是，我们的方法通过在推理过程中动态包含相关示例，对非标准设置实现了巨大的性能改进。

Title: Nostra Domina at EvaLatin 2024: Improving Latin Polarity Detection through Data Augmentation

• Authors: Stephen Bothwell, Abigail Swenor, David Chiang
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.07792
• Pdf URL: https://arxiv.org/pdf/2404.07792
• Copy Paste: [[2404.07792]] Nostra Domina at EvaLatin 2024: Improving Latin Polarity Detection through Data Augmentation(https://arxiv.org/abs/2404.07792)
• Keywords: language model, llm
• Abstract: This paper describes submissions from the team Nostra Domina to the EvaLatin 2024 shared task of emotion polarity detection. Given the low-resource environment of Latin and the complexity of sentiment in rhetorical genres like poetry, we augmented the available data through automatic polarity annotation. We present two methods for doing so on the basis of the $k$-means algorithm, and we employ a variety of Latin large language models (LLMs) in a neural architecture to better capture the underlying contextual sentiment representations. Our best approach achieved the second highest macro-averaged Macro-$F_1$ score on the shared task's test set.
• 摘要：本文介绍了 Nostra Domina 团队向 EvaLatin 2024 情绪极性检测共享任务提交的内容。考虑到拉丁语资源匮乏的环境以及诗歌等修辞体裁情感的复杂性，我们通过自动极性注释增强了可用数据。我们提出了两种基于 $k$-means 算法的方法，并且我们在神经架构中采用了各种拉丁大语言模型 (LLM)，以更好地捕获潜在的上下文情感表示。我们的最佳方法在共享任务的测试集上获得了第二高的宏观平均 Macro-$F_1$ 分数。

Title: On Training Data Influence of GPT Models

• Authors: Qingyi Liu, Yekun Chai, Shuohuan Wang, Yu Sun, Keze Wang, Hua Wu
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.07840
• Pdf URL: https://arxiv.org/pdf/2404.07840
• Copy Paste: [[2404.07840]] On Training Data Influence of GPT Models(https://arxiv.org/abs/2404.07840)
• Keywords: language model, gpt
• Abstract: Amidst the rapid advancements in generative language models, the investigation of how training data shapes the performance of GPT models is still emerging. This paper presents GPTfluence, a novel approach that leverages a featurized simulation to assess the impact of training examples on the training dynamics of GPT models. Our approach not only traces the influence of individual training instances on performance trajectories, such as loss and other key metrics, on targeted test points but also enables a comprehensive comparison with existing methods across various training scenarios in GPT models, ranging from 14 million to 2.8 billion parameters, across a range of downstream tasks. Contrary to earlier methods that struggle with generalization to new data, GPTfluence introduces a parameterized simulation of training dynamics, demonstrating robust generalization capabilities to unseen training data. This adaptability is evident across both fine-tuning and instruction-tuning scenarios, spanning tasks in natural language understanding and generation. We will make our code and data publicly available.
• 摘要：随着生成语言模型的快速发展，关于训练数据如何影响 GPT 模型性能的研究仍在不断涌现。本文提出了 GPTfluence，这是一种利用特征化模拟来评估训练示例对 GPT 模型训练动态的影响的新颖方法。我们的方法不仅可以追踪单个训练实例对目标测试点上的性能轨迹（例如损失和其他关键指标）的影响，还可以与 GPT 模型中各种训练场景（从 1400 万到 2.8 不等）的现有方法进行全面比较十亿个参数，跨越一系列下游任务。与早期难以泛化到新数据的方法相反，GPTfluence 引入了训练动态的参数化模拟，展示了对看不见的训练数据的强大泛化能力。这种适应性在微调和指令调整场景以及自然语言理解和生成任务中都很明显。我们将公开我们的代码和数据。

Title: Guiding Large Language Models to Post-Edit Machine Translation with Error Annotations

• Authors: Dayeon Ki, Marine Carpuat
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.07851
• Pdf URL: https://arxiv.org/pdf/2404.07851
• Copy Paste: [[2404.07851]] Guiding Large Language Models to Post-Edit Machine Translation with Error Annotations(https://arxiv.org/abs/2404.07851)
• Keywords: language model, llm, prompt
• Abstract: Machine Translation (MT) remains one of the last NLP tasks where large language models (LLMs) have not yet replaced dedicated supervised systems. This work exploits the complementary strengths of LLMs and supervised MT by guiding LLMs to automatically post-edit MT with external feedback on its quality, derived from Multidimensional Quality Metric (MQM) annotations. Working with LLaMA-2 models, we consider prompting strategies varying the nature of feedback provided and then fine-tune the LLM to improve its ability to exploit the provided guidance. Through experiments on Chinese-English, English-German, and English-Russian MQM data, we demonstrate that prompting LLMs to post-edit MT improves TER, BLEU and COMET scores, although the benefits of fine-grained feedback are not clear. Fine-tuning helps integrate fine-grained feedback more effectively and further improves translation quality based on both automatic and human evaluation.
• 摘要：机器翻译 (MT) 仍然是大型语言模型 (LLM) 尚未取代专用监督系统的最后 NLP 任务之一。这项工作利用了法学硕士和监督机器翻译的互补优势，指导法学硕士自动对机器翻译进行后期编辑，并利用源自多维质量指标 (MQM) 注释的质量外部反馈。使用 LLaMA-2 模型，我们考虑采用不同的反馈性质的提示策略，然后微调 LLM 以提高其利用所提供指导的能力。通过对中英、英德和英俄 MQM 数据的实验，我们证明，提示法学硕士进行 MT 后期编辑可以提高 TER、BLEU 和 COMET 分数，尽管细粒度反馈的好处尚不清楚。微调有助于更有效地整合细粒度的反馈，并基于自动和人工评估进一步提高翻译质量。

Title: High-Dimension Human Value Representation in Large Language Models

• Authors: Samuel Cahyawijaya, Delong Chen, Yejin Bang, Leila Khalatbari, Bryan Wilie, Ziwei Ji, Etsuko Ishii, Pascale Fung
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.07900
• Pdf URL: https://arxiv.org/pdf/2404.07900
• Copy Paste: [[2404.07900]] High-Dimension Human Value Representation in Large Language Models(https://arxiv.org/abs/2404.07900)
• Keywords: language model, gpt, llm, chat
• Abstract: The widespread application of Large Language Models (LLMs) across various tasks and fields has necessitated the alignment of these models with human values and preferences. Given various approaches of human value alignment, ranging from Reinforcement Learning with Human Feedback (RLHF), to constitutional learning, etc. there is an urgent need to understand the scope and nature of human values injected into these models before their release. There is also a need for model alignment without a costly large scale human annotation effort. We propose UniVaR, a high-dimensional representation of human value distributions in LLMs, orthogonal to model architecture and training data. Trained from the value-relevant output of eight multilingual LLMs and tested on the output from four multilingual LLMs, namely LlaMA2, ChatGPT, JAIS and Yi, we show that UniVaR is a powerful tool to compare the distribution of human values embedded in different LLMs with different langauge sources. Through UniVaR, we explore how different LLMs prioritize various values in different languages and cultures, shedding light on the complex interplay between human values and language modeling.
• 摘要：大型语言模型 (LLM) 在各种任务和领域的广泛应用使得这些模型必须与人类价值观和偏好保持一致。鉴于人类价值调整的方法多种多样，从人类反馈强化学习（RLHF）到宪法学习等，迫切需要在模型发布之前了解注入这些模型的人类价值的范围和性质。还需要在不进行昂贵的大规模人工注释工作的情况下进行模型对齐。我们提出了 UniVaR，这是法学硕士中人类价值分布的高维表示，与模型架构和训练数据正交。通过八个多语言法学硕士的价值相关输出进行训练，并对四个多语言法学硕士（即 LlaMA2、ChatGPT、JAIS 和 Yi）的输出进行测试，我们表明 UniVaR 是一个强大的工具，可以比较不同法学硕士中嵌入的人类价值观的分布与不同的语言来源。通过 UniVaR，我们探索不同的法学硕士如何优先考虑不同语言和文化中的各种价值观，揭示人类价值观和语言建模之间复杂的相互作用。

Title: HGRN2: Gated Linear RNNs with State Expansion

• Authors: Zhen Qin, Songlin Yang, Weixuan Sun, Xuyang Shen, Dong Li, Weigao Sun, Yiran Zhong
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.07904
• Pdf URL: https://arxiv.org/pdf/2404.07904
• Copy Paste: [[2404.07904]] HGRN2: Gated Linear RNNs with State Expansion(https://arxiv.org/abs/2404.07904)
• Keywords: language model
• Abstract: Hierarchically gated linear RNN (HGRN,Qin et al. 2023) has demonstrated competitive training speed and performance in language modeling, while offering efficient inference. However, the recurrent state size of HGRN remains relatively small, which limits its expressiveness.To address this issue, inspired by linear attention, we introduce a simple outer-product-based state expansion mechanism so that the recurrent state size can be significantly enlarged without introducing any additional parameters. The linear attention form also allows for hardware-efficient training.Our extensive experiments verify the advantage of HGRN2 over HGRN1 in language modeling, image classification, and Long Range Arena.Our largest 3B HGRN2 model slightly outperforms Mamba and LLaMa Architecture Transformer for language modeling in a controlled experiment setting; and performs competitively with many open-source 3B models in downstream evaluation while using much fewer total training tokens.
• 摘要：分层门控线性 RNN（HGRN，Qin 等人，2023）在语言建模方面展示了有竞争力的训练速度和性能，同时提供了高效的推理。然而，HGRN 的循环状态大小仍然相对较小，这限制了其表达能力。为了解决这个问题，受线性注意力的启发，我们引入了一种简单的基于外积的状态扩展机制，以便在不影响循环状态大小的情况下显着扩大循环状态大小。引入任何附加参数。线性注意力形式还可以实现硬件高效的训练。我们的大量实验验证了 HGRN2 在语言建模、图像分类和 Long Range Arena 方面相对于 HGRN1 的优势。我们最大的 3B HGRN2 模型在语言建模方面略优于 Mamba 和 LLaMa Architecture Transformer。受控实验设置；并在下游评估中与许多开源 3B 模型竞争，同时使用更少的总训练令牌。

Title: AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMs

• Authors: Zeyi Liao, Huan Sun
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.07921
• Pdf URL: https://arxiv.org/pdf/2404.07921
• Copy Paste: [[2404.07921]] AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMs(https://arxiv.org/abs/2404.07921)
• Keywords: language model, gpt, llm, chat
• Abstract: As large language models (LLMs) become increasingly prevalent and integrated into autonomous systems, ensuring their safety is imperative. Despite significant strides toward safety alignment, recent work GCG~\citep{zou2023universal} proposes a discrete token optimization algorithm and selects the single suffix with the lowest loss to successfully jailbreak aligned LLMs. In this work, we first discuss the drawbacks of solely picking the suffix with the lowest loss during GCG optimization for jailbreaking and uncover the missed successful suffixes during the intermediate steps. Moreover, we utilize those successful suffixes as training data to learn a generative model, named AmpleGCG, which captures the distribution of adversarial suffixes given a harmful query and enables the rapid generation of hundreds of suffixes for any harmful queries in seconds. AmpleGCG achieves near 100\% attack success rate (ASR) on two aligned LLMs (Llama-2-7B-chat and Vicuna-7B), surpassing two strongest attack baselines. More interestingly, AmpleGCG also transfers seamlessly to attack different models, including closed-source LLMs, achieving a 99\% ASR on the latest GPT-3.5. To summarize, our work amplifies the impact of GCG by training a generative model of adversarial suffixes that is universal to any harmful queries and transferable from attacking open-source LLMs to closed-source LLMs. In addition, it can generate 200 adversarial suffixes for one harmful query in only 4 seconds, rendering it more challenging to defend.
• 摘要：随着大型语言模型 (LLM) 变得越来越普遍并集成到自治系统中，确保其安全势在必行。尽管在安全对齐方面取得了重大进展，但最近的工作 GCG~\citep{zou2023universal} 提出了一种离散令牌优化算法，并选择损失最低的单个后缀来成功越狱对齐 LLM。在这项工作中，我们首先讨论在 GCG 优化越狱过程中仅选择损失最低的后缀的缺点，并在中间步骤中发现错过的成功后缀。此外，我们利用这些成功的后缀作为训练数据来学习名为 AmpleGCG 的生成模型，该模型捕获给定有害查询的对抗性后缀的分布，并能够在几秒钟内快速生成任何有害查询的数百个后缀。 AmpleGCG 在两个对齐的 LLM（Llama-2-7B-chat 和 Vicuna-7B）上实现了接近 100% 的攻击成功率 (ASR)，超过了两个最强的攻击基线。更有趣的是，AmpleGCG 还可以无缝转移攻击不同的模型，包括闭源 LLM，在最新的 GPT-3.5 上实现了 99% 的 ASR。总而言之，我们的工作通过训练对抗性后缀的生成模型来放大 GCG 的影响，该模型对于任何有害查询都是通用的，并且可以从攻击开源 LLM 转移到闭源 LLM。此外，它可以在短短 4 秒内为一个有害查询生成 200 个对抗性后缀，这使得防御更具挑战性。

Title: LaVy: Vietnamese Multimodal Large Language Model

• Authors: Chi Tran, Huong Le Thanh
• Subjects: cs.CL, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.07922
• Pdf URL: https://arxiv.org/pdf/2404.07922
• Copy Paste: [[2404.07922]] LaVy: Vietnamese Multimodal Large Language Model(https://arxiv.org/abs/2404.07922)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) and Multimodal Large language models (MLLMs) have taken the world by storm with impressive abilities in complex reasoning and linguistic comprehension. Meanwhile there are plethora of works related to Vietnamese Large Language Models, the lack of high-quality resources in multimodality limits the progress of Vietnamese MLLMs. In this paper, we pioneer in address this by introducing LaVy, a state-of-the-art Vietnamese MLLM, and we also introduce LaVy-Bench benchmark designated for evaluating MLLMs's understanding on Vietnamese visual language tasks. All code and model weights are public at https://github.com/baochi0212/LaVy
• 摘要：大型语言模型 (LLM) 和多模态大型语言模型 (MLLM) 以其在复杂推理和语言理解方面令人印象深刻的能力席卷了世界。同时，与越南大语言模型相关的工作过多，多模态方面优质资源的缺乏限制了越南MLLM的进步。在本文中，我们通过介绍 LaVy（一种最先进的越南语 MLLM）率先解决了这个问题，并且我们还介绍了专门用于评估 MLLM 对越南语视觉语言任务的理解的 LaVy-Bench 基准。所有代码和模型权重均公开于 https://github.com/baochi0212/LaVy

Title: Rho-1: Not All Tokens Are What You Need

• Authors: Zhenghao Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, Yelong Shen, Ruochen Xu, Chen Lin, Yujiu Yang, Jian Jiao, Nan Duan, Weizhu Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.07965
• Pdf URL: https://arxiv.org/pdf/2404.07965
• Copy Paste: [[2404.07965]] Rho-1: Not All Tokens Are What You Need(https://arxiv.org/abs/2404.07965)
• Keywords: language model
• Abstract: Previous language model pre-training methods have uniformly applied a next-token prediction loss to all training tokens. Challenging this norm, we posit that "Not all tokens in a corpus are equally important for language model training". Our initial analysis delves into token-level training dynamics of language model, revealing distinct loss patterns for different tokens. Leveraging these insights, we introduce a new language model called Rho-1. Unlike traditional LMs that learn to predict every next token in a corpus, Rho-1 employs Selective Language Modeling (SLM), which selectively trains on useful tokens that aligned with the desired distribution. This approach involves scoring pretraining tokens using a reference model, and then training the language model with a focused loss on tokens with higher excess loss. When continual pretraining on 15B OpenWebMath corpus, Rho-1 yields an absolute improvement in few-shot accuracy of up to 30% in 9 math tasks. After fine-tuning, Rho-1-1B and 7B achieved state-of-the-art results of 40.6% and 51.8% on MATH dataset, respectively - matching DeepSeekMath with only 3% of the pretraining tokens. Furthermore, when pretraining on 80B general tokens, Rho-1 achieves 6.8% average enhancement across 15 diverse tasks, increasing both efficiency and performance of the language model pre-training.
• 摘要：以前的语言模型预训练方法对所有训练标记统一应用下一个标记预测损失。为了挑战这一规范，我们假设“并非语料库中的所有标记对于语言模型训练都同样重要”。我们的初步分析深入研究了语言模型的标记级训练动态，揭示了不同标记的不同损失模式。利用这些见解，我们引入了一种名为 Rho-1 的新语言模型。与学习预测语料库中每个下一个标记的传统 LM 不同，Rho-1 采用选择性语言模型 (SLM)，它有选择地训练与所需分布一致的有用标记。这种方法涉及使用参考模型对预训练标记进行评分，然后将损失集中在具有较高超额损失的标记上来训练语言模型。当在 15B OpenWebMath 语料库上进行持续预训练时，Rho-1 在 9 项数学任务中的小样本准确率绝对提高了 30%。经过微调后，Rho-1-1B 和 7B 在 MATH 数据集上分别取得了 40.6% 和 51.8% 的最先进结果 - 仅用 3% 的预训练标记与 DeepSeekMath 相匹配。此外，在对 80B 个通用标记进行预训练时，Rho-1 在 15 个不同任务中实现了 6.8% 的平均增强，提高了语言模型预训练的效率和性能。

Title: LLoCO: Learning Long Contexts Offline

• Authors: Sijun Tan, Xiuyu Li, Shishir Patil, Ziyang Wu, Tianjun Zhang, Kurt Keutzer, Joseph E. Gonzalez, Raluca Ada Popa
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.07979
• Pdf URL: https://arxiv.org/pdf/2404.07979
• Copy Paste: [[2404.07979]] LLoCO: Learning Long Contexts Offline(https://arxiv.org/abs/2404.07979)
• Keywords: language model, llm, long context
• Abstract: Processing long contexts remains a challenge for large language models (LLMs) due to the quadratic computational and memory overhead of the self-attention mechanism and the substantial KV cache sizes during generation. We propose a novel approach to address this problem by learning contexts offline through context compression and in-domain parameter-efficient finetuning. Our method enables an LLM to create a concise representation of the original context and efficiently retrieve relevant information to answer questions accurately. We introduce LLoCO, a technique that combines context compression, retrieval, and parameter-efficient finetuning using LoRA. Our approach extends the effective context window of a 4k token LLaMA2-7B model to handle up to 128k tokens. We evaluate our approach on several long-context question-answering datasets, demonstrating that LLoCO significantly outperforms in-context learning while using $30\times$ fewer tokens during inference. LLoCO achieves up to $7.62\times$ speed-up and substantially reduces the cost of long document question answering, making it a promising solution for efficient long context processing. Our code is publicly available at https://github.com/jeffreysijuntan/lloco.
• 摘要：由于自注意力机制的二次计算和内存开销以及生成过程中的大量 KV 缓存大小，处理长上下文仍然是大型语言模型 (LLM) 的一个挑战。我们提出了一种新方法来解决这个问题，通过上下文压缩和域内参数高效微调来离线学习上下文。我们的方法使法学硕士能够创建原始上下文的简明表示，并有效地检索相关信息以准确回答问题。我们引入了 LLoCO，这是一种使用 LoRA 结合上下文压缩、检索和参数高效微调的技术。我们的方法扩展了 4k 令牌 LLaMA2-7B 模型的有效上下文窗口，以处理多达 128k 令牌。我们在几个长上下文问答数据集上评估了我们的方法，证明 LLoCO 显着优于上下文学习，同时在推理过程中使用的标记少了 30 倍。 LLoCO 实现了高达 7.62\times$ 的加速，并大幅降低了长文档问答的成本，使其成为高效长上下文处理的有前景的解决方案。我们的代码可在 https://github.com/jeffreysijuntan/lloco 上公开获取。

Title: Language Imbalance Can Boost Cross-lingual Generalisation

• Authors: Anton Schäfer, Shauli Ravfogel, Thomas Hofmann, Tiago Pimentel, Imanol Schlag
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.07982
• Pdf URL: https://arxiv.org/pdf/2404.07982
• Copy Paste: [[2404.07982]] Language Imbalance Can Boost Cross-lingual Generalisation(https://arxiv.org/abs/2404.07982)
• Keywords: language model
• Abstract: Multilinguality is crucial for extending recent advancements in language modelling to diverse linguistic communities. To maintain high performance while representing multiple languages, multilingual models ideally align representations, allowing what is learned in one language to generalise to others. Prior research has emphasised the importance of parallel data and shared vocabulary elements as key factors for such alignment. In this study, we investigate an unintuitive novel driver of cross-lingual generalisation: language imbalance. In controlled experiments on perfectly equivalent cloned languages, we observe that the existence of a predominant language during training boosts the performance of less frequent languages and leads to stronger alignment of model representations across languages. Furthermore, we find that this trend is amplified with scale: with large enough models or long enough training, we observe that bilingual training data with a 90/10 language split yields better performance on both languages than a balanced 50/50 split. Building on these insights, we design training schemes that can improve performance in all cloned languages, even without altering the training data. As we extend our analysis to real languages, we find that infrequent languages still benefit from frequent ones, yet whether language imbalance causes cross-lingual generalisation there is not conclusive.
• 摘要：多语言性对于将语言建模的最新进展扩展到不同的语言社区至关重要。为了在表示多种语言的同时保持高性能，多语言模型理想地对齐表示，允许在一种语言中学到的知识推广到其他语言。先前的研究强调了并行数据和共享词汇元素作为这种对齐的关键因素的重要性。在这项研究中，我们研究了跨语言泛化的一个不直观的新驱动因素：语言不平衡。在对完全等效的克隆语言进行的对照实验中，我们观察到，训练过程中主要语言的存在提高了不太频繁的语言的性能，并导致跨语言的模型表示更加一致。此外，我们发现这种趋势随着规模的扩大而放大：通过足够大的模型或足够长的训练，我们观察到采用 90/10 语言分割的双语训练数据在两种语言上比平衡的 50/50 分割产生更好的性能。基于这些见解，我们设计了训练方案，即使不改变训练数据，也可以提高所有克隆语言的性能。当我们将分析扩展到真实语言时，我们发现不常用语言仍然受益于常用语言，但语言不平衡是否会导致跨语言泛化尚无定论。