2025-05-12

Title: KG-HTC: Integrating Knowledge Graphs into LLMs for Effective Zero-shot Hierarchical Text Classification

• Authors: Qianbo Zang, Christophe Zgrzendek, Igor Tchappi, Afshin Khadangi, Johannes Sedlmeir
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.05583
• Pdf URL: https://arxiv.org/pdf/2505.05583
• Copy Paste: [[2505.05583]] KG-HTC: Integrating Knowledge Graphs into LLMs for Effective Zero-shot Hierarchical Text Classification(https://arxiv.org/abs/2505.05583)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: Hierarchical Text Classification (HTC) involves assigning documents to labels organized within a taxonomy. Most previous research on HTC has focused on supervised methods. However, in real-world scenarios, employing supervised HTC can be challenging due to a lack of annotated data. Moreover, HTC often faces issues with large label spaces and long-tail distributions. In this work, we present Knowledge Graphs for zero-shot Hierarchical Text Classification (KG-HTC), which aims to address these challenges of HTC in applications by integrating knowledge graphs with Large Language Models (LLMs) to provide structured semantic context during classification. Our method retrieves relevant subgraphs from knowledge graphs related to the input text using a Retrieval-Augmented Generation (RAG) approach. Our KG-HTC can enhance LLMs to understand label semantics at various hierarchy levels. We evaluate KG-HTC on three open-source HTC datasets: WoS, DBpedia, and Amazon. Our experimental results show that KG-HTC significantly outperforms three baselines in the strict zero-shot setting, particularly achieving substantial improvements at deeper levels of the hierarchy. This evaluation demonstrates the effectiveness of incorporating structured knowledge into LLMs to address HTC's challenges in large label spaces and long-tailed label distributions. Our code is available at: this https URL.
• 摘要：分层文本分类（HTC）涉及将文档分配给分类法内组织的标签。先前关于HTC的大多数研究都集中在监督方法上。但是，在实际情况下，由于缺乏带注释的数据，使用受监管的HTC可能会具有挑战性。此外，HTC经常面临大型标签空间和长尾分布的问题。在这项工作中，我们介绍了零摄像的层次层次文本分类（KG-HTC）的知识图，旨在通过将知识图与大语言模型（LLMS）集成在一起，以在分类过程中提供结构化的语义上下文，以解决应用程序中HTC的这些挑战。我们的方法使用检索功能生成（RAG）方法从与输入文本相关的知识图中检索相关的子图。我们的KG-HTC可以增强LLM，以了解各种层次结构级别的标签语义。我们在三个开源HTC数据集上评估了KG-HTC：WOS，DBPEDIA和AMAZON。我们的实验结果表明，KG-HTC在严格的零射设置中明显优于三个基准，尤其是在更深层次的层次结构水平上取得了重大改进。该评估证明了将结构化知识纳入LLM的有效性，以应对大型标签空间和长尾标签分布的HTC挑战。我们的代码可用：此HTTPS URL。

Title: Privacy-Preserving Transformers: SwiftKey's Differential Privacy Implementation

• Authors: Abdelrahman Abouelenin, Mohamed Abdelrehim, Raffy Fahim, Amr Hendy, Mohamed Afify
• Subjects: cs.CL, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2505.05648
• Pdf URL: https://arxiv.org/pdf/2505.05648
• Copy Paste: [[2505.05648]] Privacy-Preserving Transformers: SwiftKey's Differential Privacy Implementation(https://arxiv.org/abs/2505.05648)
• Keywords: language model, gpt
• Abstract: In this paper we train a transformer using differential privacy (DP) for language modeling in SwiftKey. We run multiple experiments to balance the trade-off between the model size, run-time speed and accuracy. We show that we get small and consistent gains in the next-word-prediction and accuracy with graceful increase in memory and speed compared to the production GRU. This is obtained by scaling down a GPT2 architecture to fit the required size and a two stage training process that builds a seed model on general data and DP finetunes it on typing data. The transformer is integrated using ONNX offering both flexibility and efficiency.
• 摘要：在本文中，我们使用差异隐私（DP）在SwiftKey中进行语言建模训练变压器。我们进行多个实验，以平衡模型尺寸，运行时速度和准确性之间的权衡。我们表明，与生产GRU相比，记忆和速度优美地提高了下一个字的预测和准确性，并获得了较小且一致的收益。这是通过缩减GPT2体系结构以适合所需尺寸和两个阶段训练过程来获得的，该过程在一般数据上构建了种子模型，并在键入数据时构建了DP Finetune。使用ONNX集成了变压器，既具有柔韧性和效率。

Title: Exploration of COVID-19 Discourse on Twitter: American Politician Edition

• Authors: Cindy Kim, Daniela Puchall, Jiangyi Liang, Jiwon Kim
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.05687
• Pdf URL: https://arxiv.org/pdf/2505.05687
• Copy Paste: [[2505.05687]] Exploration of COVID-19 Discourse on Twitter: American Politician Edition(https://arxiv.org/abs/2505.05687)
• Keywords: language model
• Abstract: The advent of the COVID-19 pandemic has undoubtedly affected the political scene worldwide and the introduction of new terminology and public opinions regarding the virus has further polarized partisan stances. Using a collection of tweets gathered from leading American political figures online (Republican and Democratic), we explored the partisan differences in approach, response, and attitude towards handling the international crisis. Implementation of the bag-of-words, bigram, and TF-IDF models was used to identify and analyze keywords, topics, and overall sentiments from each party. Results suggest that Democrats are more concerned with the casualties of the pandemic, and give more medical precautions and recommendations to the public whereas Republicans are more invested in political responsibilities such as keeping the public updated through media and carefully watching the progress of the virus. We propose a systematic approach to predict and distinguish a tweet's political stance (left or right leaning) based on its COVID-19 related terms using different classification algorithms on different language models.
• 摘要：毫无疑问，共同19-19大流行的出现影响了全球的政治舞台，并引入了有关该病毒的新术语和公众舆论，进一步使党派立场两极化。我们利用一系列来自在线领先的政治人物（共和党和民主党）收集的推文，探讨了党派对处理国际危机的方法，反应和态度的差异。使用字袋，BigRAM和TF-IDF模型的实施来识别和分析每个方的关键字，主题和整体情感。结果表明，民主党人更关心大流行的伤亡，并向公众提出更多的医疗预防措施和建议，而共和党人则更多地投资于政治责任，例如通过媒体使公众更新并仔细观察病毒的进步。我们提出了一种系统的方法，以使用不同语言模型上的不同分类算法来预测和区分推文的政治立场（左右倾斜）。

Title: Assessing Robustness to Spurious Correlations in Post-Training Language Models

• Authors: Julia Shuieh, Prasann Singhal, Apaar Shanker, John Heyer, George Pu, Samuel Denton
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.05704
• Pdf URL: https://arxiv.org/pdf/2505.05704
• Copy Paste: [[2505.05704]] Assessing Robustness to Spurious Correlations in Post-Training Language Models(https://arxiv.org/abs/2505.05704)
• Keywords: language model, llm
• Abstract: Supervised and preference-based fine-tuning techniques have become popular for aligning large language models (LLMs) with user intent and correctness criteria. However, real-world training data often exhibits spurious correlations -- arising from biases, dataset artifacts, or other "shortcut" features -- that can compromise a model's performance or generalization. In this paper, we systematically evaluate three post-training algorithms -- Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and KTO (Kahneman-Tversky Optimization) -- across a diverse set of synthetic tasks and spuriousness conditions. Our tasks span mathematical reasoning, constrained instruction-following, and document-grounded question answering. We vary the degree of spurious correlation (10% vs. 90%) and investigate two forms of artifacts: "Feature Ambiguity" and "Distributional Narrowness." Our results show that the models often but not always degrade under higher spuriousness. The preference-based methods (DPO/KTO) can demonstrate relative robustness in mathematical reasoning tasks. By contrast, SFT maintains stronger performance in complex, context-intensive tasks. These findings highlight that no single post-training strategy universally outperforms in all scenarios; the best choice depends on the type of target task and the nature of spurious correlations.
• 摘要：受监督和基于偏好的微调技术已成为将大语言模型（LLMS）与用户意图和正确性标准保持一致的流行。但是，现实世界中的培训数据通常表现出伪造的相关性 - 偏见，数据集文物或其他“快捷方式”功能可能会损害模型的性能或概括。在本文中，我们系统地评估了三种训练后算法 - 受监督的微调（SFT），直接偏好优化（DPO）和KTO（Kahneman-Tversky优化） - 跨各种综合任务和谨慎的条件。我们的任务涵盖了数学推理，限制的指令跟踪以及文档接地的问题回答。我们改变了虚假相关性的程度（10％对90％），并研究两种形式的伪像：“特征歧义”和“分布狭窄”。我们的结果表明，模型经常但并不总是在较高的伪造下降级。基于偏好的方法（DPO/KTO）可以证明数学推理任务中的相对鲁棒性。相比之下，SFT在复杂的，上下文密集的任务中保持更强的性能。这些发现凸显了在所有情况下，没有一个单一的培训策略普遍胜过表现。最佳选择取决于目标任务的类型和虚假相关性的性质。

Title: Insertion Language Models: Sequence Generation with Arbitrary-Position Insertions

• Authors: Dhruvesh Patel, Aishwarya Sahoo, Avinash Amballa, Tahira Naseem, Tim G. J. Rudner, Andrew McCallum
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2505.05755
• Pdf URL: https://arxiv.org/pdf/2505.05755
• Copy Paste: [[2505.05755]] Insertion Language Models: Sequence Generation with Arbitrary-Position Insertions(https://arxiv.org/abs/2505.05755)
• Keywords: language model
• Abstract: Autoregressive models (ARMs), which predict subsequent tokens one-by-one ``from left to right,'' have achieved significant success across a wide range of sequence generation tasks. However, they struggle to accurately represent sequences that require satisfying sophisticated constraints or whose sequential dependencies are better addressed by out-of-order generation. Masked Diffusion Models (MDMs) address some of these limitations, but the process of unmasking multiple tokens simultaneously in MDMs can introduce incoherences, and MDMs cannot handle arbitrary infilling constraints when the number of tokens to be filled in is not known in advance. In this work, we introduce Insertion Language Models (ILMs), which learn to insert tokens at arbitrary positions in a sequence -- that is, they select jointly both the position and the vocabulary element to be inserted. By inserting tokens one at a time, ILMs can represent strong dependencies between tokens, and their ability to generate sequences in arbitrary order allows them to accurately model sequences where token dependencies do not follow a left-to-right sequential structure. To train ILMs, we propose a tailored network parameterization and use a simple denoising objective. Our empirical evaluation demonstrates that ILMs outperform both ARMs and MDMs on common planning tasks. Furthermore, we show that ILMs outperform MDMs and perform on par with ARMs in an unconditional text generation task while offering greater flexibility than MDMs in arbitrary-length text infilling.
• 摘要：自回归模型（ARM）预测随后的令牌一对一的``从左到右''已在各种序列生成任务中取得了重大成功。但是，他们难以准确地表示需要满足复杂约束或通过排序产生更好地解决其顺序依赖性的序列。掩盖的扩散模型（MDMS）解决了其中一些局限性，但是在MDMS中同时启用多个令牌的过程可能会引入不连贯性，而当未知的要填充的令牌数量填充时，MDMS无法处理任意填充限制。在这项工作中，我们介绍了插入语言模型（ILMS），该模型学会在序列中以任意位置插入令牌 - 也就是说，他们共同选择要插入的位置和词汇元素。通过一次插入一个令牌，ILM可以代表令牌之间的强依赖性，并且它们以任意顺序生成序列的能力使他们能够准确地对序列进行建模，而对于令牌依赖的依赖性不遵循左右顺序的顺序结构。为了训练ILM，我们提出了一个量身定制的网络参数化，并使用一个简单的降解目标。我们的经验评估表明，ILM在共同计划任务上的武器和MDM都优于武器和MDM。此外，我们表明，ILMS在无条件的文本生成任务中表现优于MDM，并与武器相当，同时在任意长度文本填充中提供了比MDM更大的灵活性。

Title: Sparse Attention Remapping with Clustering for Efficient LLM Decoding on PIM

• Authors: Zehao Fan, Garrett Gagnon, Zhenyu Liu, Liu Liu
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2505.05772
• Pdf URL: https://arxiv.org/pdf/2505.05772
• Copy Paste: [[2505.05772]] Sparse Attention Remapping with Clustering for Efficient LLM Decoding on PIM(https://arxiv.org/abs/2505.05772)
• Keywords: language model, llm
• Abstract: Transformer-based models are the foundation of modern machine learning, but their execution, particularly during autoregressive decoding in large language models (LLMs), places significant pressure on memory systems due to frequent memory accesses and growing key-value (KV) caches. This creates a bottleneck in memory bandwidth, especially as context lengths increase. Processing-in-memory (PIM) architectures are a promising solution, offering high internal bandwidth and compute parallelism near memory. However, current PIM designs are primarily optimized for dense attention and struggle with the dynamic, irregular access patterns introduced by modern KV cache sparsity techniques. Consequently, they suffer from workload imbalance, reducing throughput and resource utilization. In this work, we propose STARC, a novel sparsity-optimized data mapping scheme tailored specifically for efficient LLM decoding on PIM architectures. STARC clusters KV pairs by semantic similarity and maps them to contiguous memory regions aligned with PIM bank structures. During decoding, queries retrieve relevant tokens at cluster granularity by matching against precomputed centroids, enabling selective attention and parallel processing without frequent reclustering or data movement overhead. Experiments on the HBM-PIM system show that, compared to common token-wise sparsity methods, STARC reduces attention-layer latency by 19%--31% and energy consumption by 19%--27%. Under a KV cache budget of 1024, it achieves up to 54%--74% latency reduction and 45%--67% energy reduction compared to full KV cache retrieval. Meanwhile, STARC maintains model accuracy comparable to state-of-the-art sparse attention methods, demonstrating its effectiveness in enabling efficient and hardware-friendly long-context LLM inference on PIM architectures.
• 摘要：基于变压器的模型是现代机器学习的基础，但是它们的执行，尤其是在大型语言模型（LLMS）中进行自回归解码期间，由于频繁的内存访问和增长的键值（KV）缓存，对内存系统施加了巨大压力。这会在内存带宽中产生瓶颈，尤其是随着上下文长度的增加。内存处理（PIM）体系结构是一个有前途的解决方案，可在记忆附近提供高内部带宽和计算并行性。但是，当前的PIM设计主要是为了与现代KV缓存稀疏技术引入的动态，不规则的访问模式进行密集的关注和斗争。因此，他们患有工作量失衡，减少吞吐量和资源利用率。在这项工作中，我们提出了STARC，这是一种专门针对PIM架构上有效解码的有效LLM解码的新型稀疏性数据映射方案。 Starc簇通过语义相似性对KV对，并将它们映射到与PIM库结构对齐的连续记忆区域。在解码过程中，查询通过与预先计算的质心相匹配，以群集粒度检索相关令牌，从而无需频繁地重群集或数据移动开销，从而使选择性注意力和并行处理。 HBM-PIM系统上的实验表明，与普通令牌的稀疏方法相比，Starc将注意力层潜伏期降低了19％-31％，能源消耗量为19％-27％。在1024年的KV缓存预算下，与全KV缓存检索相比，它最多可达到54％ - 74％的潜伏期减少和45％-67％的能源。同时，Starc保持模型精度可与最先进的稀疏注意方法相媲美，这证明了其在启用PIM架构上有效且对硬件友好型长篇小说LLM推断方面的有效性。

Title: Tell Me Who Your Students Are: GPT Can Generate Valid Multiple-Choice Questions When Students' (Mis)Understanding Is Hinted

• Authors: Machi Shimmei, Masaki Uto, Yuichiroh Matsubayashi, Kentaro Inui, Aditi Mallavarapu, Noboru Matsuda
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.05815
• Pdf URL: https://arxiv.org/pdf/2505.05815
• Copy Paste: [[2505.05815]] Tell Me Who Your Students Are: GPT Can Generate Valid Multiple-Choice Questions When Students' (Mis)Understanding Is Hinted(https://arxiv.org/abs/2505.05815)
• Keywords: language model, gpt, prompt, chat
• Abstract: The primary goal of this study is to develop and evaluate an innovative prompting technique, AnaQuest, for generating multiple-choice questions (MCQs) using a pre-trained large language model. In AnaQuest, the choice items are sentence-level assertions about complex concepts. The technique integrates formative and summative assessments. In the formative phase, students answer open-ended questions for target concepts in free text. For summative assessment, AnaQuest analyzes these responses to generate both correct and incorrect assertions. To evaluate the validity of the generated MCQs, Item Response Theory (IRT) was applied to compare item characteristics between MCQs generated by AnaQuest, a baseline ChatGPT prompt, and human-crafted items. An empirical study found that expert instructors rated MCQs generated by both AI models to be as valid as those created by human instructors. However, IRT-based analysis revealed that AnaQuest-generated questions - particularly those with incorrect assertions (foils) - more closely resembled human-crafted items in terms of difficulty and discrimination than those produced by ChatGPT.
• 摘要：这项研究的主要目标是开发和评估一种创新的提示技术Anaquest，用于使用预训练的大语言模型来产生多项选择问题（MCQ）。在Anaquest中，选择项目是有关复杂概念的句子级主张。该技术集成了形成性和总结性评估。在形成阶段，学生回答自由文本中目标概念的开放式问题。为了进行总结性评估，Anaquest分析了这些响应，以产生正确和错误的断言。为了评估生成的MCQ的有效性，应用了项目响应理论（IRT）来比较Anaquest（基线ChatGPT提示）生成的MCQ和人工制作的项目之间的项目特征。一项实证研究发现，两种AI模型产生的MCQ的专家教师都与人类指导员创建的MCQ一样有效。但是，基于IRT的分析表明，与Chatgpt产生的难度和歧视方面，反对性的问题（尤其是那些具有不正确断言的人（尤其是那些具有不正确断言）的问题）更相似。

Title: Elastic Weight Consolidation for Full-Parameter Continual Pre-Training of Gemma2

• Authors: Vytenis Šliogeris, Povilas Daniušis, Artūras Nakvosas
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2505.05946
• Pdf URL: https://arxiv.org/pdf/2505.05946
• Copy Paste: [[2505.05946]] Elastic Weight Consolidation for Full-Parameter Continual Pre-Training of Gemma2(https://arxiv.org/abs/2505.05946)
• Keywords: language model, llm
• Abstract: This technical report describes an experiment on autoregressive pre-training of Gemma2 2 billion parameter large language model (LLM) with 10\% on the Lithuanian language component of CulturaX from the point of view of continual learning. We apply elastic weight consolidation (EWC) to the full set of the model's parameters and investigate language understanding benchmarks, consisting of Arc, Belebele, Gsm8K, Hellaswag, MMLU, TruthfulQA, and Winogrande sets (both in English and Lithuanian versions), and perplexity benchmarks. We empirically demonstrate that EWC regularisation allows us not only to mitigate catastrophic forgetting effects but also that it is potentially beneficial for learning of the new task with LLMs.
• 摘要：该技术报告介绍了一项关于从持续学习的角度来看，对文化的立陶宛语言组成部分的Gemma2 20亿语言模型（LLM）的自回归预培训实验。我们将弹性重量合并（EWC）应用于模型参数的完整集，并研究了语言理解基准，包括Arc，Belebele，GSM8K，Hellaswag，Hellaswag，MMLU，MMLU，Elterfulqa和Winogrande Sets（以英语和Lithuanian版本和Lithuanian Versions和Perplexity Benchs Marksss和Perplexity Benchsss组成）。我们从经验上证明，EWC正则化不仅可以减轻灾难性的遗忘效果，而且还可以使使用LLMS学习新任务是有益的。

Title: Summarisation of German Judgments in conjunction with a Class-based Evaluation

• Authors: Bianca Steffes, Nils Torben Wiedemann, Alexander Gratz, Pamela Hochreither, Jana Elina Meyer, Katharina Luise Schilke
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.05947
• Pdf URL: https://arxiv.org/pdf/2505.05947
• Copy Paste: [[2505.05947]] Summarisation of German Judgments in conjunction with a Class-based Evaluation(https://arxiv.org/abs/2505.05947)
• Keywords: language model
• Abstract: The automated summarisation of long legal documents can be a great aid for legal experts in their daily work. We automatically create summaries (guiding principles) of German judgments by fine-tuning a decoder-based large language model. We enrich the judgments with information about legal entities before the training. For the evaluation of the created summaries, we define a set of evaluation classes which allows us to measure their language, pertinence, completeness and correctness. Our results show that employing legal entities helps the generative model to find the relevant content, but the quality of the created summaries is not yet sufficient for a use in practice.
• 摘要：长期法律文件的自动汇总可以为他们的日常工作而言是法律专家的重要帮助。我们通过微调基于解码器的大语言模型来自动创建德国判断的摘要（指导原则）。我们在培训前用有关法人实体的信息丰富了判断。为了评估创建的摘要，我们定义了一组评估类，该类别使我们能够测量其语言，相关性，完整性和正确性。我们的结果表明，使用法律实体可以帮助生成模型找到相关内容，但是创建的摘要的质量尚不足够用于实践。

Title: NeoQA: Evidence-based Question Answering with Generated News Events

• Authors: Max Glockner, Xiang Jiang, Leonardo F. R. Ribeiro, Iryna Gurevych, Markus Dreyer
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.05949
• Pdf URL: https://arxiv.org/pdf/2505.05949
• Copy Paste: [[2505.05949]] NeoQA: Evidence-based Question Answering with Generated News Events(https://arxiv.org/abs/2505.05949)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: Evaluating Retrieval-Augmented Generation (RAG) in large language models (LLMs) is challenging because benchmarks can quickly become stale. Questions initially requiring retrieval may become answerable from pretraining knowledge as newer models incorporate more recent information during pretraining, making it difficult to distinguish evidence-based reasoning from recall. We introduce NeoQA (News Events for Out-of-training Question Answering), a benchmark designed to address this issue. To construct NeoQA, we generated timelines and knowledge bases of fictional news events and entities along with news articles and Q\&A pairs to prevent LLMs from leveraging pretraining knowledge, ensuring that no prior evidence exists in their training data. We propose our dataset as a new platform for evaluating evidence-based question answering, as it requires LLMs to generate responses exclusively from retrieved evidence and only when sufficient evidence is available. NeoQA enables controlled evaluation across various evidence scenarios, including cases with missing or misleading details. Our findings indicate that LLMs struggle to distinguish subtle mismatches between questions and evidence, and suffer from short-cut reasoning when key information required to answer a question is missing from the evidence, underscoring key limitations in evidence-based reasoning.
• 摘要：在大语言模型（LLMS）中评估检索功能的生成（RAG）是具有挑战性的，因为基准可以迅速变得陈旧。最初需要检索的问题可能会因在训练阶段的知识中而回答，因为较新的模型在预处理过程中纳入了更多的最新信息，因此很难将基于证据的推理与召回区分开。我们介绍了Neoqa（用于训练问题的新闻事件），这是一个旨在解决此问题的基准。为了构建Neoqa，我们为虚构的新闻事件和实体以及新闻文章以及Q \＆A对生成了时间表和知识基础，以防止LLMS利用预读知识，确保其培训数据中没有先前的证据。我们建议我们的数据集作为评估基于证据的问题答案的新平台，因为它要求LLMS仅从检索到的证据中生成回答，并且仅在有足够的证据时才产生回答。 NEOQA可以在各种证据方案中进行受控评估，包括缺少或误导细节的情况。我们的发现表明，LLM努力区分问题和证据之间的细微不匹配，并且在证据中缺少需要回答问题的关键信息时，遭受了短切的推理，这突显了基于证据的推理的关键限制。

Title: Towards Developmentally Plausible Rewards: Communicative Success as a Learning Signal for Interactive Language Models

• Authors: Lennart Stöpler, Rufat Asadli, Mitja Nikolaus, Ryan Cotterell, Alex Warstadt
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.05970
• Pdf URL: https://arxiv.org/pdf/2505.05970
• Copy Paste: [[2505.05970]] Towards Developmentally Plausible Rewards: Communicative Success as a Learning Signal for Interactive Language Models(https://arxiv.org/abs/2505.05970)
• Keywords: language model
• Abstract: We propose a method for training language models in an interactive setting inspired by child language acquisition. In our setting, a speaker attempts to communicate some information to a listener in a single-turn dialogue and receives a reward if communicative success is achieved. Unlike earlier related work using image--caption data for interactive reference games, we operationalize communicative success in a more abstract language-only question--answering setting. First, we present a feasibility study demonstrating that our reward provides an indirect signal about grammaticality. Second, we conduct experiments using reinforcement learning to fine-tune language models. We observe that cognitively plausible constraints on the communication channel lead to interpretable changes in speaker behavior. However, we do not yet see improvements on linguistic evaluations from our training regime. We outline potential modifications to the task design and training configuration that could better position future work to use our methodology to observe the benefits of interaction on language learning in computational cognitive models.
• 摘要：我们建议在受儿童语言获取启发的交互式环境中培训语言模型的方法。在我们的环境中，演讲者试图通过单转对话将一些信息传达给听众，并在取得沟通成功的情况下获得奖励。与使用图像的早期相关工作（用于交互式参考游戏）的早期相关工作不同，我们在更纯粹的语言问题（访问设置）中运行交流成功。首先，我们提出了一项可行性研究，表明我们的奖励提供了有关语法性的间接信号。其次，我们使用加强学习进行微调语言模型进行实验。我们观察到，在通信渠道上的认知合理约束会导致说话者行为的可解释变化。但是，我们尚未从培训制度中看到语言评估的改进。我们概述了对任务设计和培训配置的潜在修改，可以更好地定位未来的工作，以使用我们的方法来观察计算认知模型中语言学习的互动益处。

Title: Exploring the Feasibility of Multilingual Grammatical Error Correction with a Single LLM up to 9B parameters: A Comparative Study of 17 Models

• Authors: Dawid Wisniewski, Antoni Solarski, Artur Nowakowski
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.06004
• Pdf URL: https://arxiv.org/pdf/2505.06004
• Copy Paste: [[2505.06004]] Exploring the Feasibility of Multilingual Grammatical Error Correction with a Single LLM up to 9B parameters: A Comparative Study of 17 Models(https://arxiv.org/abs/2505.06004)
• Keywords: language model, llm
• Abstract: Recent language models can successfully solve various language-related tasks, and many understand inputs stated in different languages. In this paper, we explore the performance of 17 popular models used to correct grammatical issues in texts stated in English, German, Italian, and Swedish when using a single model to correct texts in all those languages. We analyze the outputs generated by these models, focusing on decreasing the number of grammatical errors while keeping the changes small. The conclusions drawn help us understand what problems occur among those models and which models can be recommended for multilingual grammatical error correction tasks. We list six models that improve grammatical correctness in all four languages and show that Gemma 9B is currently the best performing one for the languages considered.
• 摘要：最近的语言模型可以成功地解决与语言相关的各种任务，许多人理解不同语言的意见。在本文中，我们探讨了17种流行模型的性能，用于纠正英语，德语，意大利语和瑞典语中所述的语法问题，当时使用单个模型纠正所有这些语言的文本。我们分析了这些模型产生的输出，重点是减少语法错误的数量，同时使变化较小。得出的结论有助于我们了解这些模型之间发生了哪些问题，并且可以推荐哪些模型用于多语种语法错误校正任务。我们列出了六个模型，这些模型可以提高所有四种语言的语法正确性，并表明Gemma 9b目前是所考虑的语言的最佳性能。

Title: Do Not Change Me: On Transferring Entities Without Modification in Neural Machine Translation -- a Multilingual Perspective

• Authors: Dawid Wisniewski, Mikolaj Pokrywka, Zofia Rostek
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.06010
• Pdf URL: https://arxiv.org/pdf/2505.06010
• Copy Paste: [[2505.06010]] Do Not Change Me: On Transferring Entities Without Modification in Neural Machine Translation -- a Multilingual Perspective(https://arxiv.org/abs/2505.06010)
• Keywords: llm
• Abstract: Current machine translation models provide us with high-quality outputs in most scenarios. However, they still face some specific problems, such as detecting which entities should not be changed during translation. In this paper, we explore the abilities of popular NMT models, including models from the OPUS project, Google Translate, MADLAD, and EuroLLM, to preserve entities such as URL addresses, IBAN numbers, or emails when producing translations between four languages: English, German, Polish, and Ukrainian. We investigate the quality of popular NMT models in terms of accuracy, discuss errors made by the models, and examine the reasons for errors. Our analysis highlights specific categories, such as emojis, that pose significant challenges for many models considered. In addition to the analysis, we propose a new multilingual synthetic dataset of 36,000 sentences that can help assess the quality of entity transfer across nine categories and four aforementioned languages.
• 摘要：当前的机器翻译模型在大多数情况下为我们提供了高质量的输出。但是，他们仍然面临一些特定的问题，例如检测在翻译过程中不应更改哪些实体。在本文中，我们探讨了受欢迎的NMT模型的能力，包括Opus Project，Google Translate，Madlad和Eurollm的模型，以保存在四种语言之间制作翻译时的URL地址，IBAN号码或电子邮件之类的实体：英语，德语，波兰语和乌克兰语。我们根据准确性研究了流行的NMT模型的质量，讨论模型造成的错误，并检查错误的原因。我们的分析强调了特定类别，例如表情符号，对许多被考虑的模型构成了重大挑战。除了分析外，我们还提出了一个新的多语言合成数据集，其中包括36,000个句子，可以帮助评估跨九个类别和四种上述语言的实体转移质量。

Title: Unilogit: Robust Machine Unlearning for LLMs Using Uniform-Target Self-Distillation

• Authors: Stefan Vasilev, Christian Herold, Baohao Liao, Seyyed Hadi Hashemi, Shahram Khadivi, Christof Monz
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2505.06027
• Pdf URL: https://arxiv.org/pdf/2505.06027
• Copy Paste: [[2505.06027]] Unilogit: Robust Machine Unlearning for LLMs Using Uniform-Target Self-Distillation(https://arxiv.org/abs/2505.06027)
• Keywords: language model, llm
• Abstract: This paper introduces Unilogit, a novel self-distillation method for machine unlearning in Large Language Models. Unilogit addresses the challenge of selectively forgetting specific information while maintaining overall model utility, a critical task in compliance with data privacy regulations like GDPR. Unlike prior methods that rely on static hyperparameters or starting model outputs, Unilogit dynamically adjusts target logits to achieve a uniform probability for the target token, leveraging the current model's outputs for more accurate self-distillation targets. This approach not only eliminates the need for additional hyperparameters but also enhances the model's ability to approximate the golden targets. Extensive experiments on public benchmarks and an in-house e-commerce dataset demonstrate Unilogit's superior performance in balancing forget and retain objectives, outperforming state-of-the-art methods such as NPO and UnDIAL. Our analysis further reveals Unilogit's robustness across various scenarios, highlighting its practical applicability and effectiveness in achieving efficacious machine unlearning.
• 摘要：本文介绍了Unilogit，这是一种用于大型语言模型中的机器学习的新型自我验证方法。 Unilogit应对在维持整体模型实用程序的同时选择性忘记特定信息的挑战，这是遵守GDPR等数据隐私法规的关键任务。与依靠静态超参数或启动模型输出的先前方法不同，UniLogit会动态调整目标逻辑以实现目标令牌的均匀概率，从而利用当前模型的输出以获得更准确的自我验证目标。这种方法不仅消除了对其他超参数的需求，而且还提高了模型近似黄金目标的能力。关于公共基准和内部电子商务数据集的广泛实验表明，Unilogit在平衡忘记和保留目标方面的出色表现，优于NPO和Undial等最先进的方法。我们的分析进一步揭示了Unilogit在各种情况下的鲁棒性，突出了其实用性和有效性在实现有效的机器上的实用性和有效性。

Title: Healthy LLMs? Benchmarking LLM Knowledge of UK Government Public Health Information

• Authors: Joshua Harris, Fan Grayson, Felix Feldman, Timothy Laurence, Toby Nonnenmacher, Oliver Higgins, Leo Loman, Selina Patel, Thomas Finnie, Samuel Collins, Michael Borowitz
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2505.06046
• Pdf URL: https://arxiv.org/pdf/2505.06046
• Copy Paste: [[2505.06046]] Healthy LLMs? Benchmarking LLM Knowledge of UK Government Public Health Information(https://arxiv.org/abs/2505.06046)
• Keywords: language model, gpt, llm
• Abstract: As Large Language Models (LLMs) become widely accessible, a detailed understanding of their knowledge within specific domains becomes necessary for successful real world use. This is particularly critical in public health, where failure to retrieve relevant, accurate, and current information could significantly impact UK residents. However, currently little is known about LLM knowledge of UK Government public health information. To address this issue, this paper introduces a new benchmark, PubHealthBench, with over 8000 questions for evaluating LLMs' Multiple Choice Question Answering (MCQA) and free form responses to public health queries, created via an automated pipeline. We also release a new dataset of the extracted UK Government public health guidance documents used as source text for PubHealthBench. Assessing 24 LLMs on PubHealthBench we find the latest private LLMs (GPT-4.5, GPT-4.1 and o1) have a high degree of knowledge, achieving >90% in the MCQA setup, and outperform humans with cursory search engine use. However, in the free form setup we see lower performance with no model scoring >75%. Therefore, whilst there are promising signs that state of the art (SOTA) LLMs are an increasingly accurate source of public health information, additional safeguards or tools may still be needed when providing free form responses on public health topics.
• 摘要：随着大型语言模型（LLMS）变得广泛访问，对成功现实世界中的特定领域中的知识有详细的了解。这在公共卫生中尤为重要，在公共卫生中，未能检索相关，准确和当前信息可能会对英国居民产生重大影响。但是，目前对英国政府公共卫生信息的LLM知识知之甚少。为了解决这个问题，本文介绍了一个新的基准PubHealthBench，其中包含8000多个评估LLMS的多项选择问题答案（MCQA）的问题，并通过自动化管道创建的对公共卫生查询的免费形式回答。我们还发布了一个新的数据集，其中包括用作PubHealthBench来源文本的英国政府公共卫生指导文件。评估PubHealthBench上的24个LLM，我们发现最新的私人LLM（GPT-4.5，GPT-4.1和O1）具有很高的知识，在MCQA设置中实现了> 90％的知识，并且胜过人类的人类具有粗心大意的搜索引擎使用。但是，在免费表单设置中，我们看到较低的性能，没有模型得分> 75％。因此，尽管有一个有希望的迹象表明，最先进的LLM是越来越准确的公共卫生信息来源，但在提供公共卫生主题的免费形式响应时，仍然需要其他保障措施或工具。

Title: LLMs Get Lost In Multi-Turn Conversation

• Authors: Philippe Laban, Hiroaki Hayashi, Yingbo Zhou, Jennifer Neville
• Subjects: cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2505.06120
• Pdf URL: https://arxiv.org/pdf/2505.06120
• Copy Paste: [[2505.06120]] LLMs Get Lost In Multi-Turn Conversation(https://arxiv.org/abs/2505.06120)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) are conversational interfaces. As such, LLMs have the potential to assist their users not only when they can fully specify the task at hand, but also to help them define, explore, and refine what they need through multi-turn conversational exchange. Although analysis of LLM conversation logs has confirmed that underspecification occurs frequently in user instructions, LLM evaluation has predominantly focused on the single-turn, fully-specified instruction setting. In this work, we perform large-scale simulation experiments to compare LLM performance in single- and multi-turn settings. Our experiments confirm that all the top open- and closed-weight LLMs we test exhibit significantly lower performance in multi-turn conversations than single-turn, with an average drop of 39% across six generation tasks. Analysis of 200,000+ simulated conversations decomposes the performance degradation into two components: a minor loss in aptitude and a significant increase in unreliability. We find that LLMs often make assumptions in early turns and prematurely attempt to generate final solutions, on which they overly rely. In simpler terms, we discover that *when LLMs take a wrong turn in a conversation, they get lost and do not recover*.
• 摘要：大型语言模型（LLMS）是对话界面。因此，LLM有可能在用户完全指定手头的任务时为用户提供帮助，还可以帮助他们通过多转交流交换来定义，探索和完善所需的内容。尽管对LLM对话日志的分析已证实在用户指令中经常发生指定性化，但LLM评估主要集中在单转，完全指定的指令设置上。在这项工作中，我们执行大规模的仿真实验，以比较单转设置和多转设置中的LLM性能。我们的实验证实，我们测试的所有顶级开放式LLM和多转交流的性能都明显低于单转弯，而在六代任务中，平均下降了39％。对200,000多个模拟对话的分析将绩效降解分解为两个组成部分：能力损失较小，不可靠性显着提高。我们发现，LLM经常在早期进行假设，并过早地尝试生成过度依赖的最终解决方案。用更简单的话来说，我们发现 *llms在对话中发生错误的转弯时，他们会迷路并且不会恢复 *。

Title: Can Prompting LLMs Unlock Hate Speech Detection across Languages? A Zero-shot and Few-shot Study

• Authors: Faeze Ghorbanpour, Daryna Dementieva, Alexander Fraser
• Subjects: cs.CL, cs.CY, cs.MM
• Abstract URL: https://arxiv.org/abs/2505.06149
• Pdf URL: https://arxiv.org/pdf/2505.06149
• Copy Paste: [[2505.06149]] Can Prompting LLMs Unlock Hate Speech Detection across Languages? A Zero-shot and Few-shot Study(https://arxiv.org/abs/2505.06149)
• Keywords: language model, llm, prompt
• Abstract: Despite growing interest in automated hate speech detection, most existing approaches overlook the linguistic diversity of online content. Multilingual instruction-tuned large language models such as LLaMA, Aya, Qwen, and BloomZ offer promising capabilities across languages, but their effectiveness in identifying hate speech through zero-shot and few-shot prompting remains underexplored. This work evaluates LLM prompting-based detection across eight non-English languages, utilizing several prompting techniques and comparing them to fine-tuned encoder models. We show that while zero-shot and few-shot prompting lag behind fine-tuned encoder models on most of the real-world evaluation sets, they achieve better generalization on functional tests for hate speech detection. Our study also reveals that prompt design plays a critical role, with each language often requiring customized prompting techniques to maximize performance.
• 摘要：尽管对自动仇恨言论检测的兴趣越来越大，但大多数现有的方法都忽略了在线内容的语言多样性。多语言指令调整的大型语言模型，例如Llama，Aya，Qwen和Bloomz提供了跨语言的有前途的功能，但是它们在通过零射击和几乎没有射击提示来识别仇恨言论方面的有效性仍然没有被淘汰。这项工作评估了跨八种非英语语言的基于LLM提示的检测，并利用了几种提示技术并将其与微调的编码模型进行比较。我们表明，尽管在大多数现实世界评估集中，零射击和几乎没有射击促使滞后在微调编码器模型后面，但它们在仇恨语音检测的功能测试上实现了更好的概括。我们的研究还表明，及时设计起着至关重要的作用，每种语言通常都需要定制的提示技术来最大程度地提高性能。

Title: A Scaling Law for Token Efficiency in LLM Fine-Tuning Under Fixed Compute Budgets

• Authors: Ryan Lagasse, Aidan Kiernans, Avijit Ghosh, Shiri Dori-Hacohen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.06150
• Pdf URL: https://arxiv.org/pdf/2505.06150
• Copy Paste: [[2505.06150]] A Scaling Law for Token Efficiency in LLM Fine-Tuning Under Fixed Compute Budgets(https://arxiv.org/abs/2505.06150)
• Keywords: language model, llm
• Abstract: We introduce a scaling law for fine-tuning large language models (LLMs) under fixed compute budgets that explicitly accounts for data composition. Conventional approaches measure training data solely by total tokens, yet the number of examples and their average token length -- what we term \emph{dataset volume} -- play a decisive role in model performance. Our formulation is tuned following established procedures. Experiments on the BRICC dataset \cite{salavati2024reducing} and subsets of the MMLU dataset \cite{hendrycks2021measuringmassivemultitasklanguage}, evaluated under multiple subsampling strategies, reveal that data composition significantly affects token efficiency. These results motivate refined scaling laws for practical LLM fine-tuning in resource-constrained settings.
• 摘要：我们在固定的计算预算下介绍了针对微型语言模型（LLM）的缩放定律，该预算明确说明了数据组成。常规方法仅通过总代币来衡量训练数据，但是示例数及其平均令牌长度（我们称为\ emph {dataSet卷） - 在模型性能中起决定性作用。根据既定程序，我们的表述都经过调整。在Bricc数据集\ cite {salavati2024 reducing}上进行的实验和MMLU数据集的子集\ cite {Hendrycks2021MeasuringMeasuringMassiveMultitAskLanguage}，在多个子采样策略下进行了评估，揭示了数据组合的影响很大。这些结果激发了在资源约束设置中实用的LLM微调的精制缩放定律。

Title: Query-driven Document-level Scientific Evidence Extraction from Biomedical Studies

• Authors: Massimiliano Pronesti, Joao Bettencourt-Silva, Paul Flanagan, Alessandra Pascale, Oisin Redmond, Anya Belz, Yufang Hou
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.06186
• Pdf URL: https://arxiv.org/pdf/2505.06186
• Copy Paste: [[2505.06186]] Query-driven Document-level Scientific Evidence Extraction from Biomedical Studies(https://arxiv.org/abs/2505.06186)
• Keywords: retrieval-augmented generation
• Abstract: Extracting scientific evidence from biomedical studies for clinical research questions (e.g., Does stem cell transplantation improve quality of life in patients with medically refractory Crohn's disease compared to placebo?) is a crucial step in synthesising biomedical evidence. In this paper, we focus on the task of document-level scientific evidence extraction for clinical questions with conflicting evidence. To support this task, we create a dataset called CochraneForest, leveraging forest plots from Cochrane systematic reviews. It comprises 202 annotated forest plots, associated clinical research questions, full texts of studies, and study-specific conclusions. Building on CochraneForest, we propose URCA (Uniform Retrieval Clustered Augmentation), a retrieval-augmented generation framework designed to tackle the unique challenges of evidence extraction. Our experiments show that URCA outperforms the best existing methods by up to 10.3% in F1 score on this task. However, the results also underscore the complexity of CochraneForest, establishing it as a challenging testbed for advancing automated evidence synthesis systems.
• 摘要：与安慰剂相比，从生物医学研究中提取科学证据（例如，干细胞移植是否可以改善医疗难治性克罗恩病的患者的生活质量？）是合成生物医学证据的关键步骤。在本文中，我们关注文件级的科学证据提取的任务，这些临床问题与证据相互矛盾。为了支持这项任务，我们创建了一个名为Cochraneforest的数据集，该数据集利用Cochrane Systematic Reviews的森林图。它包括202个注释的森林图，相关的临床研究问题，全文研究和特定研究结论。在Cochraneforest的基础上，我们提出了URCA（统一的检索簇增强），这是一个旨在应对证据提取​​的独特挑战的检索生成框架。我们的实验表明，在此任务上，URCA优于最佳现有方法的F1分数高达10.3％。但是，结果也强调了Cochraneforest的复杂性，将其确立为具有挑战性的测试，用于推进自动证据合成系统。