2025-04-09

Title: Unequal Opportunities: Examining the Bias in Geographical Recommendations by Large Language Models

• Authors: Shiran Dudy, Thulasi Tholeti, Resmi Ramachandranpillai, Muhammad Ali, Toby Jia-Jun Li, Ricardo Baeza-Yates
• Subjects: cs.CL, cs.AI, cs.CY, cs.HC, cs.IR
• Abstract URL: https://arxiv.org/abs/2504.05325
• Pdf URL: https://arxiv.org/pdf/2504.05325
• Copy Paste: [[2504.05325]] Unequal Opportunities: Examining the Bias in Geographical Recommendations by Large Language Models(https://arxiv.org/abs/2504.05325)
• Keywords: language model, llm, chat, agent
• Abstract: Recent advancements in Large Language Models (LLMs) have made them a popular information-seeking tool among end users. However, the statistical training methods for LLMs have raised concerns about their representation of under-represented topics, potentially leading to biases that could influence real-world decisions and opportunities. These biases could have significant economic, social, and cultural impacts as LLMs become more prevalent, whether through direct interactions--such as when users engage with chatbots or automated assistants--or through their integration into third-party applications (as agents), where the models influence decision-making processes and functionalities behind the scenes. Our study examines the biases present in LLMs recommendations of U.S. cities and towns across three domains: relocation, tourism, and starting a business. We explore two key research questions: (i) How similar LLMs responses are, and (ii) How this similarity might favor areas with certain characteristics over others, introducing biases. We focus on the consistency of LLMs responses and their tendency to over-represent or under-represent specific locations. Our findings point to consistent demographic biases in these recommendations, which could perpetuate a ``rich-get-richer'' effect that widens existing economic disparities.
• 摘要：大型语言模型（LLM）的最新进展使它们成为最终用户中流行的信息寻求信息。但是，LLMS的统计培训方法引起了人们对代表不足的主题的担忧，可能导致可能影响现实世界的决策和机遇的偏见。随着LLM变得更加普遍，无论是通过直接互动（例如，当用户与聊天机器人或自动化助手互动时），还是通过将其整合到第三方应用程序（作为代理商）中，这些偏见可能会产生重大的经济，社会和文化影响，就像通过用户互动时，模型会影响决策制定过程和幕后的决策过程。我们的研究研究了美国城市和城镇跨三个领域的LLMS建议中存在的偏见：搬迁，旅游业和开展业务。我们探讨了两个关键的研究问题：（i）LLMS的回答的相似性，以及（ii）这种相似性如何有利于具有某些特征的领域，而不是其他特征，从而引入了偏见。我们专注于LLMS响应的一致性及其倾向过多或代表性不足的特定位置的趋势。我们的发现表明，在这些建议中，人口偏见一致，这可能会使``丰富的富裕人''效果永久化，从而扩大了现有的经济差异。

Title: Fast Controlled Generation from Language Models with Adaptive Weighted Rejection Sampling

• Authors: Benjamin Lipkin, Benjamin LeBrun, Jacob Hoover Vigly, João Loula, David R. MacIver, Li Du, Jason Eisner, Ryan Cotterell, Vikash Mansinghka, Timothy J. O'Donnell, Alexander K. Lew, Tim Vieira
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2504.05410
• Pdf URL: https://arxiv.org/pdf/2504.05410
• Copy Paste: [[2504.05410]] Fast Controlled Generation from Language Models with Adaptive Weighted Rejection Sampling(https://arxiv.org/abs/2504.05410)
• Keywords: language model
• Abstract: The dominant approach to generating from language models subject to some constraint is locally constrained decoding (LCD), incrementally sampling tokens at each time step such that the constraint is never violated. Typically, this is achieved through token masking: looping over the vocabulary and excluding non-conforming tokens. There are two important problems with this approach. (i) Evaluating the constraint on every token can be prohibitively expensive -- LM vocabularies often exceed $100,000$ tokens. (ii) LCD can distort the global distribution over strings, sampling tokens based only on local information, even if they lead down dead-end paths. This work introduces a new algorithm that addresses both these problems. First, to avoid evaluating a constraint on the full vocabulary at each step of generation, we propose an adaptive rejection sampling algorithm that typically requires orders of magnitude fewer constraint evaluations. Second, we show how this algorithm can be extended to produce low-variance, unbiased estimates of importance weights at a very small additional cost -- estimates that can be soundly used within previously proposed sequential Monte Carlo algorithms to correct for the myopic behavior of local constraint enforcement. Through extensive empirical evaluation in text-to-SQL, molecular synthesis, goal inference, pattern matching, and JSON domains, we show that our approach is superior to state-of-the-art baselines, supporting a broader class of constraints and improving both runtime and performance. Additional theoretical and empirical analyses show that our method's runtime efficiency is driven by its dynamic use of computation, scaling with the divergence between the unconstrained and constrained LM, and as a consequence, runtime improvements are greater for better models.
• 摘要：从受到某种约束的语言模型中生成的主要方法是本地限制的解码（LCD），在每个时间步骤中逐步采样令牌，因此永远不会违反约束。通常，这是通过令牌掩盖来实现的：在词汇上循环并排除不合格令牌。这种方法有两个重要的问题。 （i）评估每个令牌上的约束可能非常昂贵 - LM词汇量通常超过$ 100,000 $的代币。 （ii）LCD可以在字符串上扭曲全球分布，即使它们导致死端路径下降，也只能基于本地信息进行采样令牌。这项工作引入了一种解决这两个问题的新算法。首先，为避免在生成的每个步骤中评估对完整词汇的约束，我们提出了一种自适应拒绝采样算法，该算法通常需要减少约束评估的数量级。其次，我们展示了如何扩展该算法以产生低变化的，无偏见的重要性权重的估计值 - 额外的成本很小 - 可以在先前建议的顺序蒙特卡洛算法中大量使用，以纠正局部约束执法的近视行为。通过在文本到SQL，分子综合，目标推理，模式匹配和JSON领域的广泛经验评估中，我们表明我们的方法优于最新的基线，支持更广泛的约束并提高运行时和性能。其他理论和经验分析表明，我们的方法的运行时效率是由其动态使用计算的驱动，并随着未约束和受约束的LM之间的差异而进行缩放，因此，对于更好的模型而言，运行时的改进更大。

Title: Less but Better: Parameter-Efficient Fine-Tuning of Large Language Models for Personality Detection

• Authors: Lingzhi Shen, Yunfei Long, Xiaohao Cai, Guanming Chen, Imran Razzak, Shoaib Jameel
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2504.05411
• Pdf URL: https://arxiv.org/pdf/2504.05411
• Copy Paste: [[2504.05411]] Less but Better: Parameter-Efficient Fine-Tuning of Large Language Models for Personality Detection(https://arxiv.org/abs/2504.05411)
• Keywords: language model, llm
• Abstract: Personality detection automatically identifies an individual's personality from various data sources, such as social media texts. However, as the parameter scale of language models continues to grow, the computational cost becomes increasingly difficult to manage. Fine-tuning also grows more complex, making it harder to justify the effort and reliably predict outcomes. We introduce a novel parameter-efficient fine-tuning framework, PersLLM, to address these challenges. In PersLLM, a large language model (LLM) extracts high-dimensional representations from raw data and stores them in a dynamic memory layer. PersLLM then updates the downstream layers with a replaceable output network, enabling flexible adaptation to various personality detection scenarios. By storing the features in the memory layer, we eliminate the need for repeated complex computations by the LLM. Meanwhile, the lightweight output network serves as a proxy for evaluating the overall effectiveness of the framework, improving the predictability of results. Experimental results on key benchmark datasets like Kaggle and Pandora show that PersLLM significantly reduces computational cost while maintaining competitive performance and strong adaptability.
• 摘要：个性检测会自动从各种数据源（例如社交媒体文本）中识别个人的个性。但是，随着语言模型的参数量表的不断增长，计算成本变得越来越难以管理。微调也变得更加复杂，因此很难证明努力并可靠地预测结果。我们介绍了一个新颖的参数效率微调框架Persllm，以应对这些挑战。在Persllm中，大型语言模型（LLM）从原始数据中提取高维表示，并将其存储在动态内存层中。然后，Persllm使用可更换的输出网络更新下游图层，从而可以灵活地适应各种人格检测方案。通过将功能存储在内存层中，我们消除了LLM重复进行复杂计算的需求。同时，轻型输出网络是评估框架总体有效性的代理，从而提高了结果的可预测性。 Kaggle和Pandora等关键基准数据集的实验结果表明，Persllm大大降低了计算成本，同时保持竞争性能和强大的适应性。

Title: A Survey on Hypothesis Generation for Scientific Discovery in the Era of Large Language Models

• Authors: Atilla Kaan Alkan, Shashwat Sourav, Maja Jablonska, Simone Astarita, Rishabh Chakrabarty, Nikhil Garuda, Pranav Khetarpal, Maciej Pióro, Dimitrios Tanoglidis, Kartheik G. Iyer, Mugdha S. Polimera, Michael J. Smith, Tirthankar Ghosal, Marc Huertas-Company, Sandor Kruk, Kevin Schawinski, Ioana Ciucă
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.05496
• Pdf URL: https://arxiv.org/pdf/2504.05496
• Copy Paste: [[2504.05496]] A Survey on Hypothesis Generation for Scientific Discovery in the Era of Large Language Models(https://arxiv.org/abs/2504.05496)
• Keywords: language model, llm, prompt
• Abstract: Hypothesis generation is a fundamental step in scientific discovery, yet it is increasingly challenged by information overload and disciplinary fragmentation. Recent advances in Large Language Models (LLMs) have sparked growing interest in their potential to enhance and automate this process. This paper presents a comprehensive survey of hypothesis generation with LLMs by (i) reviewing existing methods, from simple prompting techniques to more complex frameworks, and proposing a taxonomy that categorizes these approaches; (ii) analyzing techniques for improving hypothesis quality, such as novelty boosting and structured reasoning; (iii) providing an overview of evaluation strategies; and (iv) discussing key challenges and future directions, including multimodal integration and human-AI collaboration. Our survey aims to serve as a reference for researchers exploring LLMs for hypothesis generation.
• 摘要：假设产生是科学发现的基本步骤，但由于信息超负荷和纪律处分，它越来越挑战。大型语言模型（LLM）的最新进展引起了人们对增强和自动化这一过程的潜力的日益兴趣。本文通过（i）审查现有方法，从简单提示技术到更复杂的框架以及提出对这些方法进行分类的分类法，对LLM的假设产生进行了全面调查； （ii）分析改善假设质量的技术，例如提高新颖性和结构化推理； （iii）提供评估策略的概述； （iv）讨论关键挑战和未来的方向，包括多模式整合和人类合作。我们的调查旨在作为研究人员的参考，探索LLM的假设产生。

Title: ChartQAPro: A More Diverse and Challenging Benchmark for Chart Question Answering

• Authors: Ahmed Masry, Mohammed Saidul Islam, Mahir Ahmed, Aayush Bajaj, Firoz Kabir, Aaryaman Kartha, Md Tahmid Rahman Laskar, Mizanur Rahman, Shadikur Rahman, Mehrad Shahmohammadi, Megh Thakkar, Md Rizwan Parvez, Enamul Hoque, Shafiq Joty
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.05506
• Pdf URL: https://arxiv.org/pdf/2504.05506
• Copy Paste: [[2504.05506]] ChartQAPro: A More Diverse and Challenging Benchmark for Chart Question Answering(https://arxiv.org/abs/2504.05506)
• Keywords: language model
• Abstract: Charts are ubiquitous, as people often use them to analyze data, answer questions, and discover critical insights. However, performing complex analytical tasks with charts requires significant perceptual and cognitive effort. Chart Question Answering (CQA) systems automate this process by enabling models to interpret and reason with visual representations of data. However, existing benchmarks like ChartQA lack real-world diversity and have recently shown performance saturation with modern large vision-language models (LVLMs). To address these limitations, we introduce ChartQAPro, a new benchmark that includes 1,341 charts from 157 diverse sources, spanning various chart types, including infographics and dashboards, and featuring 1,948 questions in various types, such as multiple-choice, conversational, hypothetical, and unanswerable questions, to better reflect real-world challenges. Our evaluations with 21 models show a substantial performance drop for LVLMs on ChartQAPro; e.g., Claude Sonnet 3.5 scores 90.5% on ChartQA but only 55.81% on ChartQAPro, underscoring the complexity of chart reasoning. We complement our findings with detailed error analyses and ablation studies, identifying key challenges and opportunities for advancing LVLMs in chart understanding and reasoning. We release ChartQAPro at this https URL.
• 摘要：图表无处不在，因为人们经常使用它们来分析数据，回答问题并发现关键见解。但是，使用图表执行复杂的分析任务需要重大的感知和认知工作。图表问题回答（CQA）系统通过使模型可以通过数据的可视化表示和推理来自动化此过程。但是，如ChartQa这样的现有基准缺乏现实世界的多样性，并且最近通过现代大型视觉模型（LVLM）显示了性能饱和。为了解决这些限制，我们介绍了ChartQapro，这是一种新的基准测试，其中包括1341个来自157种不同来源的图表，涵盖了各种图表类型，包括信息图表和仪表板，并在各种类型的各种类型的问题上提供了1,948个问题，例如多选择，对话，假设，假设和无可​​争议的问题，以更好地反映现实的挑战。我们使用21个模型的评估表明，ChartQapro上LVLM的性能下降；例如，Claude Sonnet 3.5在ChartQA上得分为90.5％，但ChartQapro仅占55.81％，强调了图表推理的复杂性。我们通过详细的错误分析和消融研究来补充我们的发现，确定了在图表理解和推理中推进LVLM的关键挑战和机会。我们在此HTTPS URL上发布ChartQapro。

Title: Pretraining Language Models for Diachronic Linguistic Change Discovery

• Authors: Elisabeth Fittschen, Sabrina Li, Tom Lippincott, Leshem Choshsem, Craig Messner
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.05523
• Pdf URL: https://arxiv.org/pdf/2504.05523
• Copy Paste: [[2504.05523]] Pretraining Language Models for Diachronic Linguistic Change Discovery(https://arxiv.org/abs/2504.05523)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have shown potential as tools for scientific discovery. This has engendered growing interest in their use in humanistic disciplines, such as historical linguistics and literary studies. These fields often construct arguments on the basis of delineations like genre, or more inflexibly, time period. Although efforts have been made to restrict inference to specific domains via fine-tuning or model editing, we posit that the only true guarantee is domain-restricted pretraining -- typically, a data- and compute-expensive proposition. We show that efficient pretraining techniques can produce useful models over corpora too large for easy manual inspection but too small for "typical" LLM approaches. We employ a novel date-attribution pipeline in order to obtain a temporally-segmented dataset of five 10-million-word slices. We train two corresponding five-model batteries over these corpus segments, efficient pretraining and Llama3-8B parameter efficiently finetuned. We find that the pretrained models are faster to train than the finetuned baselines and that they better respect the historical divisions of our corpus. Emphasizing speed and precision over a-historical comprehensiveness enables a number of novel approaches to hypothesis discovery and testing in our target fields. Taking up diachronic linguistics as a testbed, we show that our method enables the detection of a diverse set of phenomena, including en masse lexical change, non-lexical (grammatical and morphological) change, and word sense introduction/obsolescence. We provide a ready-to-use pipeline that allows extension of our approach to other target fields with only minimal adaptation.
• 摘要：大型语言模型（LLM）显示出潜力作为科学发现的工具。这引起了他们在人文学科中的使用，例如历史语言学和文学研究。这些领域通常会根据诸如流派或更灵活的时间段的描述构建论点。尽管已经努力通过微调或模型编辑限制推断特定领域，但我们认为唯一真正的保证是域限制性预处理 - 通常是数据和账单昂贵的命题。我们表明，有效的预审进技术可以使与Corpora的有用模型更大，无法轻松检查，但对于“典型” LLM方法而言太小。我们采用一种新颖的日期属性管道，以获取5千万片切片的时间细分数据集。我们在这些语料库的细分市场上训练两个相应的五模型电池，有效的预处理和Llama3-8B参数有效地进行了易键调。我们发现，经过预告片的模型比易月的基线更快，并且更好地尊重我们语料库的历史分裂。强调速度和精确性在A史综合性上，可以在我们的目标领域中进行许多新颖的假设发现和测试方法。我们将历时语言学作为测试床，表明我们的方法可以检测各种现象，包括共同的词汇变化，非障碍（语法和形态学）变化以及单词感官介绍/过时/过时。我们提供一条现成的管道，该管道允许将我们的方法扩展到其他目标字段，仅适应最少。

Title: Bridging Industrial Expertise and XR with LLM-Powered Conversational Agents

• Authors: Despina Tomkou, George Fatouros, Andreas Andreou, Georgios Makridis, Fotis Liarokapis, Dimitrios Dardanis, Athanasios Kiourtis, John Soldatos, Dimosthenis Kyriazis
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.05527
• Pdf URL: https://arxiv.org/pdf/2504.05527
• Copy Paste: [[2504.05527]] Bridging Industrial Expertise and XR with LLM-Powered Conversational Agents(https://arxiv.org/abs/2504.05527)
• Keywords: language model, llm, chat, retrieval-augmented generation, agent
• Abstract: This paper introduces a novel integration of Retrieval-Augmented Generation (RAG) enhanced Large Language Models (LLMs) with Extended Reality (XR) technologies to address knowledge transfer challenges in industrial environments. The proposed system embeds domain-specific industrial knowledge into XR environments through a natural language interface, enabling hands-free, context-aware expert guidance for workers. We present the architecture of the proposed system consisting of an LLM Chat Engine with dynamic tool orchestration and an XR application featuring voice-driven interaction. Performance evaluation of various chunking strategies, embedding models, and vector databases reveals that semantic chunking, balanced embedding models, and efficient vector stores deliver optimal performance for industrial knowledge retrieval. The system's potential is demonstrated through early implementation in multiple industrial use cases, including robotic assembly, smart infrastructure maintenance, and aerospace component servicing. Results indicate potential for enhancing training efficiency, remote assistance capabilities, and operational guidance in alignment with Industry 5.0's human-centric and resilient approach to industrial development.
• 摘要：本文介绍了带有扩展现实（XR）技术的检索成绩（RAG）增强大型语言模型（LLM）的新颖集成，以应对工业环境中的知识转移挑战。所提出的系统通过自然语言界面将特定领域的工业知识嵌入XR环境中，从而为工人提供了免提的，可感知的专家指南。我们介绍了提出的系统的体系结构，该系统由LLM聊天引擎组成，其动态工具编排和具有语音驱动交互的XR应用程序。对各种构成策略，嵌入模型和向量数据库的性能评估表明，语义块，平衡的嵌入模型和有效的矢量商店为工业知识检索提供了最佳的性能。该系统的潜力通过在多种工业用例中的早期实施来证明，包括机器人组装，智能基础设施维护和航空航天组件维修。结果表明，在与工业5.0的工业发展中以人为中心和弹性的方式保持一致的培训效率，远程援助能力以及运营指导的潜力。

Title: COIG-P: A High-Quality and Large-Scale Chinese Preference Dataset for Alignment with Human Values

• Authors: M-A-P Team, Siwei Wu, Jincheng Ren, Xinrun Du, Shuyue Guo, Xingwei Qu, Yiming Liang, Jie Liu, Yunwen Li, Tianyu Zheng, Boyu Feng, Huaqing Yuan, Zenith Wang, Jiaheng Liu, Wenhao Huang, Chenglin Cai, Haoran Que, Jian Yang, Yuelin Bai, Zekun Moore Wang, Zhouliang Yu, Qunshu Lin, Ding Pan, Yuchen Jiang, Tiannan Wang, Wangchunshu Zhou, Shenzhi Wang, Xingyuan Bu, Minghao Liu, Guoyin Wang, Ge Zhang, Chenghua Lin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.05535
• Pdf URL: https://arxiv.org/pdf/2504.05535
• Copy Paste: [[2504.05535]] COIG-P: A High-Quality and Large-Scale Chinese Preference Dataset for Alignment with Human Values(https://arxiv.org/abs/2504.05535)
• Keywords: language model, gpt, llm, chat
• Abstract: Aligning large language models (LLMs) with human preferences has achieved remarkable success. However, existing Chinese preference datasets are limited by small scale, narrow domain coverage, and lack of rigorous data validation. Additionally, the reliance on human annotators for instruction and response labeling significantly constrains the scalability of human preference datasets. To address these challenges, we design an LLM-based Chinese preference dataset annotation pipeline with no human intervention. Specifically, we crawled and carefully filtered 92k high-quality Chinese queries and employed 15 mainstream LLMs to generate and score chosen-rejected response pairs. Based on it, we introduce COIG-P (Chinese Open Instruction Generalist - Preference), a high-quality, large-scale Chinese preference dataset, comprises 1,009k Chinese preference pairs spanning 6 diverse domains: Chat, Code, Math, Logic, Novel, and Role. Building upon COIG-P, to reduce the overhead of using LLMs for scoring, we trained a 8B-sized Chinese Reward Model (CRM) and meticulously constructed a Chinese Reward Benchmark (CRBench). Evaluation results based on AlignBench \citep{liu2024alignbenchbenchmarkingchinesealignment} show that that COIG-P significantly outperforms other Chinese preference datasets, and it brings significant performance improvements ranging from 2% to 12% for the Qwen2/2.5 and Infinity-Instruct-3M-0625 model series, respectively. The results on CRBench demonstrate that our CRM has a strong and robust scoring ability. We apply it to filter chosen-rejected response pairs in a test split of COIG-P, and our experiments show that it is comparable to GPT-4o in identifying low-quality samples while maintaining efficiency and cost-effectiveness. Our codes and data are released in this https URL.
• 摘要：将大型语言模型（LLM）与人类偏好保持一致，取得了巨大的成功。但是，现有的中国偏好数据集受小规模，狭窄的域覆盖范围以及缺乏严格的数据验证的限制。另外，对人类注释的指导和响应标签的依赖显着限制了人类偏好数据集的可扩展性。为了应对这些挑战，我们设计了一个基于LLM的中国偏好数据集注释管道，没有人类干预。具体而言，我们爬行并仔细地过滤了92K高质量的中国查询，并采用了15个主流LLM，以生成和评分选择的拒绝响应对。基于它，我们介绍了高质量的中国偏好数据集（中国开放指导通才 - 偏好），包括1,009k的中国偏好对，跨越了6个不同的领域：聊天，代码，数学，逻辑，新颖和角色。在Coig-P的基础上，为了减少使用LLM进行评分的开销，我们训练了一个8B大小的中国奖励模型（CRM），并精心构建了中国奖励基准（CRBENCH）。基于AlignBench \ citep {liu2024AlignBenchbenchMarkingChineSealtigementment}的评估结果表明，GOIG-P显着胜过其他中国偏好数据集，并且在QWEN2/2.5和Infinstruct-Inflincition-Inflation-Infloction-Inflstruct-3M-0625M-0625模型中，它的性能改善范围从2％到12％，相应地相应。 CRBENCH的结果表明，我们的CRM具有强大而强大的分数能力。我们将其应用于在COIG-P的测试拆分中过滤所选的响应对，我们的实验表明，在识别低质量样本的同时，在保持效率和成本效益的同时，它与GPT-4O相当。我们的代码和数据在此HTTPS URL中发布。

Title: Can Large Language Models Match Tutoring System Adaptivity? A Benchmarking Study

• Authors: Conrad Borchers, Tianze Shou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.05570
• Pdf URL: https://arxiv.org/pdf/2504.05570
• Copy Paste: [[2504.05570]] Can Large Language Models Match Tutoring System Adaptivity? A Benchmarking Study(https://arxiv.org/abs/2504.05570)
• Keywords: language model, gpt, llm, prompt
• Abstract: Large Language Models (LLMs) hold promise as dynamic instructional aids. Yet, it remains unclear whether LLMs can replicate the adaptivity of intelligent tutoring systems (ITS)--where student knowledge and pedagogical strategies are explicitly modeled. We propose a prompt variation framework to assess LLM-generated instructional moves' adaptivity and pedagogical soundness across 75 real-world tutoring scenarios from an ITS. We systematically remove key context components (e.g., student errors and knowledge components) from prompts to create variations of each scenario. Three representative LLMs (Llama3-8B, Llama3-70B, and GPT-4o) generate 1,350 instructional moves. We use text embeddings and randomization tests to measure how the omission of each context feature impacts the LLMs' outputs (adaptivity) and a validated tutor-training classifier to evaluate response quality (pedagogical soundness). Surprisingly, even the best-performing model only marginally mimics the adaptivity of ITS. Specifically, Llama3-70B demonstrates statistically significant adaptivity to student errors. Although Llama3-8B's recommendations receive higher pedagogical soundness scores than the other models, it struggles with instruction-following behaviors, including output formatting. By contrast, GPT-4o reliably adheres to instructions but tends to provide overly direct feedback that diverges from effective tutoring, prompting learners with open-ended questions to gauge knowledge. Given these results, we discuss how current LLM-based tutoring is unlikely to produce learning benefits rivaling known-to-be-effective ITS tutoring. Through our open-source benchmarking code, we contribute a reproducible method for evaluating LLMs' instructional adaptivity and fidelity.
• 摘要：大型语言模型（LLM）作为动态教学辅助工具有望。然而，尚不清楚LLM是否可以复制智能辅导系统（ITS）的适应性，在该系统中明确建模学生知识和教学策略。我们提出了一个迅速的变体框架，以评估LLM生成的教学动作的适应性和教学声音，从ITS的75个现实世界的辅导场景中。我们从提示中系统地删除关键上下文组件（例如，学生错误和知识组件），以创建每种情况的变化。三个代表性LLM（Llama3-8B，Llama3-70B和GPT-4O）产生1,350个教学动作。我们使用文本嵌入和随机测试来衡量每个上下文特征的遗漏如何影响LLMS的输出（适应性）和经过验证的导师培训分类器来评估响应质量（教学声音）。令人惊讶的是，即使是表现最佳的模型，也只能略微模仿其适应性。具体而言，Llama3-70B表现出对学生错误的统计学显着适应性。尽管Llama3-8B的建议获得的教学声音得分比其他模型更高，但它与指导跟随行为（包括输出格式）斗争。相比之下，GPT-4O可靠地遵守说明，但倾向于提供过度直接的反馈，这些反馈与有效的辅导不同，促使学习者有开放式问题来衡量知识。鉴于这些结果，我们讨论了当前基于LLM的辅导如何产生学习益处可与已知的辅导媲美。通过我们的开源基准测试代码，我们为评估LLMS的教学适应性和忠诚度提供了可再现的方法。

Title: Knowledge-Instruct: Effective Continual Pre-training from Limited Data using Instructions

• Authors: Oded Ovadia, Meni Brief, Rachel Lemberg, Eitam Sheetrit
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.05571
• Pdf URL: https://arxiv.org/pdf/2504.05571
• Copy Paste: [[2504.05571]] Knowledge-Instruct: Effective Continual Pre-training from Limited Data using Instructions(https://arxiv.org/abs/2504.05571)
• Keywords: language model, llm
• Abstract: While Large Language Models (LLMs) acquire vast knowledge during pre-training, they often lack domain-specific, new, or niche information. Continual pre-training (CPT) attempts to address this gap but suffers from catastrophic forgetting and inefficiencies in low-data regimes. We introduce Knowledge-Instruct, a novel approach to efficiently inject knowledge from limited corpora through pure instruction-tuning. By generating information-dense synthetic instruction data, it effectively integrates new knowledge while preserving general reasoning and instruction-following abilities. Knowledge-Instruct demonstrates superior factual memorization, minimizes catastrophic forgetting, and remains scalable by leveraging synthetic data from relatively small language models. Additionally, it enhances contextual understanding, including complex multi-hop reasoning, facilitating integration with retrieval systems. We validate its effectiveness across diverse benchmarks, including Companies, a new dataset that we release to measure knowledge injection capabilities.
• 摘要：尽管大型语言模型（LLMS）在预培训期间获得了广泛的知识，但它们通常缺乏特定领域，新或利基的信息。持续的预训练（CPT）试图解决这一差距，但在低数据表格中遭受了灾难性的遗忘和效率低下的困扰。我们介绍了知识教学，这是一种新颖的方法，可以通过纯粹的指导进行有限的语料库注入知识。通过生成信息密集的综合指导数据，它可以有效地整合新知识，同时保留一般的推理和跟随能力。知识教学证明了卓越的事实记忆，最大程度地减少了灾难性的遗忘，并通过利用相对较小的语言模型的合成数据来扩展。此外，它增强了上下文理解，包括复杂的多跳推理，促进与检索系统的整合。我们验证了包括公司在内的各种基准的有效性，包括公司，我们发布了一个新数据集，以衡量知识注入能力。

Title: DEL: Context-Aware Dynamic Exit Layer for Efficient Self-Speculative Decoding

• Authors: Hossein Entezari Zarch, Lei Gao, Chaoyi Jiang, Murali Annavaram
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2504.05598
• Pdf URL: https://arxiv.org/pdf/2504.05598
• Copy Paste: [[2504.05598]] DEL: Context-Aware Dynamic Exit Layer for Efficient Self-Speculative Decoding(https://arxiv.org/abs/2504.05598)
• Keywords: language model, llm
• Abstract: Speculative Decoding (SD) is a widely used approach to accelerate the inference of large language models (LLMs) without reducing generation quality. It operates by first using a compact model to draft multiple tokens efficiently, followed by parallel verification using the target LLM. This approach leads to faster inference compared to auto-regressive decoding. While there are multiple approaches to create a draft model, one promising approach is to use early-exit methods. These methods draft candidate tokens by using a subset of layers of the primary model and applying the remaining layers for verification, allowing a single model to handle both drafting and verification. While this technique reduces memory usage and computational cost, its performance relies on the choice of the exit layer for drafting and the number of tokens drafted (speculation length) in each SD round. Prior works use hyperparameter exploration to statically select these values. However, our evaluations show that these hyperparameter values are task-specific, and even within a task they are dependent on the current sequence context. We introduce DEL, a plug-and-play method that adaptively selects the exit layer and speculation length during inference. DEL dynamically tracks the token acceptance rate if the tokens are drafted at each layer of an LLM and uses that knowledge to heuristically select the optimal exit layer and speculation length. Our experiments across a broad range of models and downstream tasks show that DEL achieves overall speedups of $2.16\times$$\sim$$2.50\times$ over vanilla auto-regressive decoding and improves upon the state-of-the-art SD methods by up to $0.27\times$.
• 摘要：投机解码（SD）是一种广泛使用的方法，可以加速大型语言模型（LLMS）而不降低发电质量。它首先使用紧凑型模型来有效地草拟多个令牌，然后使用目标llm进行并行验证。与自动回归解码相比，这种方法导致推断更快。尽管有多种创建草稿模型的方法，但一种有前途的方法是使用早期外观方法。这些方法通过使用主要模型的一层并应用剩余层进行验证，从而草拟了候选令牌，从而使单个模型可以同时处理起草和验证。尽管此技术降低了内存的使用和计算成本，但其性能依赖于用于制图的出口层的选择以及在每个SD回合中起草的标记（投机长度）的数量。先前的工作使用超参数探索来静态选择这些值。但是，我们的评估表明，这些超参数值是特定于任务的，即使在任务中，它们都取决于当前的序列上下文。我们介绍了DEL，这是一种插件方法，可以自适应地选择推理期间的出口层和投机长度。如果在LLM的每一层起草令牌，并使用该知识来启发最佳出口层和投机长度，则DEL会动态跟踪令牌的接受率。我们在各种型号和下游任务上进行的实验表明，DEL可实现$ 2.16 \ times $$ $$ \ sim $$ 2.50 \ times $ 2.50 \ times $ a自动回火解码的总体加速，并根据最先进的SD方法提高了最高$ 0.27 \ times times \ times $ $。

Title: On the Impact of Language Nuances on Sentiment Analysis with Large Language Models: Paraphrasing, Sarcasm, and Emojis

• Authors: Naman Bhargava, Mohammed I. Radaideh, O Hwang Kwon, Aditi Verma, Majdi I. Radaideh
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2504.05603
• Pdf URL: https://arxiv.org/pdf/2504.05603
• Copy Paste: [[2504.05603]] On the Impact of Language Nuances on Sentiment Analysis with Large Language Models: Paraphrasing, Sarcasm, and Emojis(https://arxiv.org/abs/2504.05603)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have demonstrated impressive performance across various tasks, including sentiment analysis. However, data quality--particularly when sourced from social media--can significantly impact their accuracy. This research explores how textual nuances, including emojis and sarcasm, affect sentiment analysis, with a particular focus on improving data quality through text paraphrasing techniques. To address the lack of labeled sarcasm data, the authors created a human-labeled dataset of 5929 tweets that enabled the assessment of LLM in various sarcasm contexts. The results show that when topic-specific datasets, such as those related to nuclear power, are used to finetune LLMs these models are not able to comprehend accurate sentiment in presence of sarcasm due to less diverse text, requiring external interventions like sarcasm removal to boost model accuracy. Sarcasm removal led to up to 21% improvement in sentiment accuracy, as LLMs trained on nuclear power-related content struggled with sarcastic tweets, achieving only 30% accuracy. In contrast, LLMs trained on general tweet datasets, covering a broader range of topics, showed considerable improvements in predicting sentiment for sarcastic tweets (60% accuracy), indicating that incorporating general text data can enhance sarcasm detection. The study also utilized adversarial text augmentation, showing that creating synthetic text variants by making minor changes significantly increased model robustness and accuracy for sarcastic tweets (approximately 85%). Additionally, text paraphrasing of tweets with fragmented language transformed around 40% of the tweets with low-confidence labels into high-confidence ones, improving LLMs sentiment analysis accuracy by 6%.
• 摘要：大型语言模型（LLMS）在各种任务中表现出令人印象深刻的表现，包括情感分析。但是，数据质量 - 尤其是从社交媒体中采购时 - 可以显着影响其准确性。这项研究探讨了包括表情符号和讽刺在内的文本细微差别如何影响情绪分析，特别着重于通过文本释义技术提高数据质量。为了解决缺乏标记的讽刺数据，作者创建了一个由5929个推文的人体标记的数据集，该数据集可在各种讽刺环境中评估LLM。结果表明，当特定于主题的数据集（例如与核能相关的数据集）被用来填补LLMS时，这些模型无法在存在讽刺的情况下，由于文本较少而无法理解准确的情感，需要诸如讽刺删除之类的外部干预措施以提高模型的准确性。讽刺的去除导致情绪准确性提高了21％，因为在与核电相关的内容中训练了与讽刺推文斗争的LLM，仅达到了30％的精度。相比之下，在一般推文数据集上训练的LLM涵盖了更广泛的主题，在预测讽刺推文的情感方面有很大改善（准确性60％），这表明合并一般文本数据可以增强讽刺检测。这项研究还利用了对抗文本的增强，表明通过进行较小的变化来显着提高模型的鲁棒性和讽刺推文的准确性（约85％）来创建合成文本变体。此外，具有零散语言的推文的文本释义将大约40％的带有低信心标签的推文转化为高信心，将LLMS情感分析精度提高了6％。

Title: FactGuard: Leveraging Multi-Agent Systems to Generate Answerable and Unanswerable Questions for Enhanced Long-Context LLM Extraction

• Authors: Qian-Wen Zhang, Fang Li, Jie Wang, Lingfeng Qiao, Yifei Yu, Di Yin, Xing Sun
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.05607
• Pdf URL: https://arxiv.org/pdf/2504.05607
• Copy Paste: [[2504.05607]] FactGuard: Leveraging Multi-Agent Systems to Generate Answerable and Unanswerable Questions for Enhanced Long-Context LLM Extraction(https://arxiv.org/abs/2504.05607)
• Keywords: language model, llm, agent
• Abstract: Extractive reading comprehension systems are designed to locate the correct answer to a question within a given text. However, a persistent challenge lies in ensuring these models maintain high accuracy in answering questions while reliably recognizing unanswerable queries. Despite significant advances in large language models (LLMs) for reading comprehension, this issue remains critical, particularly as the length of supported contexts continues to expand. To address this challenge, we propose an innovative data augmentation methodology grounded in a multi-agent collaborative framework. Unlike traditional methods, such as the costly human annotation process required for datasets like SQuAD 2.0, our method autonomously generates evidence-based question-answer pairs and systematically constructs unanswerable questions. Using this methodology, we developed the FactGuard-Bench dataset, which comprises 25,220 examples of both answerable and unanswerable question scenarios, with context lengths ranging from 8K to 128K. Experimental evaluations conducted on seven popular LLMs reveal that even the most advanced models achieve only 61.79% overall accuracy. Furthermore, we emphasize the importance of a model's ability to reason about unanswerable questions to avoid generating plausible but incorrect answers. By implementing efficient data selection and generation within the multi-agent collaborative framework, our method significantly reduces the traditionally high costs associated with manual annotation and provides valuable insights for the training and optimization of LLMs.
• 摘要：提取性阅读理解系统旨在找到给定文本中问题的正确答案。但是，持续的挑战在于确保这些模型在回答问题时保持高度准确性，同时可靠地识别无法回答的查询。尽管大型语言模型（LLM）的阅读理解有重大进步，但此问题仍然至关重要，尤其是随着支持环境的长度不断扩大。为了应对这一挑战，我们提出了一种基于多机构协作框架的创新数据增强方法。与传统方法不同，例如诸如小队2.0之类的数据集所需的昂贵人类注释过程，我们的方法自动生成基于证据的问题回答和系统地构建无法回答的问题。使用此方法，我们开发了Factguard Bench数据集，该数据集包含25,220个示例，这些示例既可以回答和无法回答的问题情景，上下文长度从8K到128K不等。在七个流行的LLM上进行的实验评估表明，即使最先进的模型也只能达到61.79％的总体准确性。此外，我们强调了模型推理无法回答的问题的能力的重要性，以避免产生合理但不正确的答案。通过在多代理协作框架内实施有效的数据选择和生成，我们的方法大大降低了与手动注释相关的传统上高成本，并为LLM的培训和优化提供了宝贵的见解。

Title: Two Intermediate Translations Are Better Than One: Fine-tuning LLMs for Document-level Translation Refinement

• Authors: Yichen Dong, Xinglin Lyu, Junhui Li, Daimeng Wei, Min Zhang, Shimin Tao, Hao Yang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.05614
• Pdf URL: https://arxiv.org/pdf/2504.05614
• Copy Paste: [[2504.05614]] Two Intermediate Translations Are Better Than One: Fine-tuning LLMs for Document-level Translation Refinement(https://arxiv.org/abs/2504.05614)
• Keywords: language model, llm
• Abstract: Recent research has shown that large language models (LLMs) can enhance translation quality through self-refinement. In this paper, we build on this idea by extending the refinement from sentence-level to document-level translation, specifically focusing on document-to-document (Doc2Doc) translation refinement. Since sentence-to-sentence (Sent2Sent) and Doc2Doc translation address different aspects of the translation process, we propose fine-tuning LLMs for translation refinement using two intermediate translations, combining the strengths of both Sent2Sent and Doc2Doc. Additionally, recognizing that the quality of intermediate translations varies, we introduce an enhanced fine-tuning method with quality awareness that assigns lower weights to easier translations and higher weights to more difficult ones, enabling the model to focus on challenging translation cases. Experimental results across ten translation tasks with LLaMA-3-8B-Instruct and Mistral-Nemo-Instruct demonstrate the effectiveness of our approach.
• 摘要：最近的研究表明，大型语言模型（LLMS）可以通过自我进行提高翻译质量。在本文中，我们通过将细化从句子级别扩展到文档级翻译，特别是专门针对文档对文档（doc2doc）翻译的改进，从而将其扩展到了这一想法。由于句子到句子（已发送2sent）和doc2oc翻译地址转换过程的不同方面，因此我们建议使用两个中间翻译的微调llms进行翻译改进，并结合了send2sent和doc2doc的优势。此外，认识到中间翻译的质量各不相同，我们引入了一种增强的微调方法，具有质量意识，将较低的权重分配给更容易的翻译和更高的权重，从而使模型能够专注于具有挑战性的翻译案例。通过Llama-3-8B教学和Mistral-Nemo-Instruction进行的十项翻译任务的实验结果证明了我们方法的有效性。

Title: Reasoning Towards Fairness: Mitigating Bias in Language Models through Reasoning-Guided Fine-Tuning

• Authors: Sanchit Kabra, Akshita Jha, Chandan Reddy
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2504.05632
• Pdf URL: https://arxiv.org/pdf/2504.05632
• Copy Paste: [[2504.05632]] Reasoning Towards Fairness: Mitigating Bias in Language Models through Reasoning-Guided Fine-Tuning(https://arxiv.org/abs/2504.05632)
• Keywords: language model, llm
• Abstract: Recent advances in large-scale generative language models have shown that reasoning capabilities can significantly improve model performance across a variety of tasks. However, the impact of reasoning on a model's ability to mitigate stereotypical responses remains largely underexplored. In this work, we investigate the crucial relationship between a model's reasoning ability and fairness, and ask whether improved reasoning capabilities can mitigate harmful stereotypical responses, especially those arising due to shallow or flawed reasoning. We conduct a comprehensive evaluation of multiple open-source LLMs, and find that larger models with stronger reasoning abilities exhibit substantially lower stereotypical bias on existing fairness benchmarks. Building on this insight, we introduce ReGiFT -- Reasoning Guided Fine-Tuning, a novel approach that extracts structured reasoning traces from advanced reasoning models and infuses them into models that lack such capabilities. We use only general-purpose reasoning and do not require any fairness-specific supervision for bias mitigation. Notably, we see that models fine-tuned using ReGiFT not only improve fairness relative to their non-reasoning counterparts but also outperform advanced reasoning models on fairness benchmarks. We also analyze how variations in the correctness of the reasoning traces and their length influence model fairness and their overall performance. Our findings highlight that enhancing reasoning capabilities is an effective, fairness-agnostic strategy for mitigating stereotypical bias caused by reasoning flaws.
• 摘要：大规模生成语言模型的最新进展表明，推理能力可以显着改善各种任务的模型性能。但是，推理对模型减轻刻板印象反应能力的影响在很大程度上还没有得到充实。在这项工作中，我们研究了模型的推理能力和公平性之间的关键关系，并询问改善的推理能力是否可以减轻有害的刻板印象反应，尤其是那些由于浅层或有缺陷的推理而产生的反应。我们对多个开源LLM进行了全面的评估，发现具有更强推理能力的较大模型对现有公平性基准的刻板印象大大降低。在这种见解的基础上，我们介绍了Regift - 推理指导的微调，一种新颖的方法，从高级推理模型中提取结构化的推理痕迹，并将它们注入缺乏这种功能的模型中。我们仅使用通用推理，不需要任何特定于公平的监督来缓解偏见。值得注意的是，我们看到，使用Regift进行了微调的模型不仅相对于非共同的对应物提高了公平性，而且还优于公平基准上的高级推理模型。我们还分析了推理轨迹正确性及其长度影响模型公平及其整体表现的变化。我们的发现强调，增强推理能力是一种有效的，公平的反应策略，用于减轻由推理缺陷引起的刻板印象偏见。

Title: DBOT: Artificial Intelligence for Systematic Long-Term Investing

• Authors: Vasant Dhar, João Sedoc
• Subjects: cs.CL, cs.AI, q-fin.PR
• Abstract URL: https://arxiv.org/abs/2504.05639
• Pdf URL: https://arxiv.org/pdf/2504.05639
• Copy Paste: [[2504.05639]] DBOT: Artificial Intelligence for Systematic Long-Term Investing(https://arxiv.org/abs/2504.05639)
• Keywords: agent
• Abstract: Long-term investing was previously seen as requiring human judgment. With the advent of generative artificial intelligence (AI) systems, automated systematic long-term investing is now feasible. In this paper, we present DBOT, a system whose goal is to reason about valuation like Aswath Damodaran, who is a unique expert in the investment arena in terms of having published thousands of valuations on companies in addition to his numerous writings on the topic, which provide ready training data for an AI system. DBOT can value any publicly traded company. DBOT can also be back-tested, making its behavior and performance amenable to scientific inquiry. We compare DBOT to its analytic parent, Damodaran, and highlight the research challenges involved in raising its current capability to that of Damodaran's. Finally, we examine the implications of DBOT-like AI agents for the financial industry, especially how they will impact the role of human analysts in valuation.
• 摘要：长期投资以前被视为需要人类的判断。随着生成人工智能（AI）系统的出现，自动化的系统长期投资现在是可行的。在本文中，我们介绍了DBOT，该系统的目标是推理诸如Aswath Damodaran之类的估值，他在投资领域是一位独特的专家，除了发表有关该主题的众多著作，为公司发表了数千个估值，这些估值为AI系统提供了现成的培训数据。 DBOT可以重视任何公开交易的公司。 DBOT也可以进行反验证，使其行为和性能适合科学询问。我们将DBOT与其分析父母Damodaran进行了比较，并强调了提高其目前对达马达兰（Damodaran）的能力所涉及的研究挑战。最后，我们研究了类似DBOT的AI代理商对金融行业的影响，尤其是他们将如何影响人类分析师在估值中的作用。

Title: Leveraging Prompt-Tuning for Bengali Grammatical Error Explanation Using Large Language Models

• Authors: Subhankar Maity, Aniket Deroy
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.05642
• Pdf URL: https://arxiv.org/pdf/2504.05642
• Copy Paste: [[2504.05642]] Leveraging Prompt-Tuning for Bengali Grammatical Error Explanation Using Large Language Models(https://arxiv.org/abs/2504.05642)
• Keywords: language model, gpt, llm, prompt
• Abstract: We propose a novel three-step prompt-tuning method for Bengali Grammatical Error Explanation (BGEE) using state-of-the-art large language models (LLMs) such as GPT-4, GPT-3.5 Turbo, and Llama-2-70b. Our approach involves identifying and categorizing grammatical errors in Bengali sentences, generating corrected versions of the sentences, and providing natural language explanations for each identified error. We evaluate the performance of our BGEE system using both automated evaluation metrics and human evaluation conducted by experienced Bengali language experts. Our proposed prompt-tuning approach shows that GPT-4, the best performing LLM, surpasses the baseline model in automated evaluation metrics, with a 5.26% improvement in F1 score and a 6.95% improvement in exact match. Furthermore, compared to the previous baseline, GPT-4 demonstrates a decrease of 25.51% in wrong error type and a decrease of 26.27% in wrong error explanation. However, the results still lag behind the human baseline.
• 摘要：我们使用最先进的大语言模型（LLM）（例如GPT-4，GPT-3.5 Turbo和Llama-2-70B）提出了一种新型的三步及时调整方法，用于孟加拉语法错误解释（BGEE）。我们的方法涉及在孟加拉语句子中识别和分类语法错误，生成句子的校正版本，并为每个确定的错误提供自然语言解释。我们使用经验丰富的孟加拉语专家进行的自动评估指标和人类评估来评估BGEE系统的性能。我们提出的及时调整方法表明，GPT-4（表现最好的LLM）超过了自动评估指标的基线模型，F1得分提高了5.26％，精确匹配提高了6.95％。此外，与先前的基线相比，GPT-4在错误的错误类型中显示出25.51％的降低25.51％，错误的错误解释降低了26.27％。但是，结果仍然落后于人类基线。

Title: Towards Smarter Hiring: Are Zero-Shot and Few-Shot Pre-trained LLMs Ready for HR Spoken Interview Transcript Analysis?

• Authors: Subhankar Maity, Aniket Deroy, Sudeshna Sarkar
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.05683
• Pdf URL: https://arxiv.org/pdf/2504.05683
• Copy Paste: [[2504.05683]] Towards Smarter Hiring: Are Zero-Shot and Few-Shot Pre-trained LLMs Ready for HR Spoken Interview Transcript Analysis?(https://arxiv.org/abs/2504.05683)
• Keywords: language model, gpt, llm, chat
• Abstract: This research paper presents a comprehensive analysis of the performance of prominent pre-trained large language models (LLMs), including GPT-4 Turbo, GPT-3.5 Turbo, text-davinci-003, text-babbage-001, text-curie-001, text-ada-001, llama-2-7b-chat, llama-2-13b-chat, and llama-2-70b-chat, in comparison to expert human evaluators in providing scores, identifying errors, and offering feedback and improvement suggestions to candidates during mock HR (Human Resources) interviews. We introduce a dataset called HURIT (Human Resource Interview Transcripts), which comprises 3,890 HR interview transcripts sourced from real-world HR interview scenarios. Our findings reveal that pre-trained LLMs, particularly GPT-4 Turbo and GPT-3.5 Turbo, exhibit commendable performance and are capable of producing evaluations comparable to those of expert human evaluators. Although these LLMs demonstrate proficiency in providing scores comparable to human experts in terms of human evaluation metrics, they frequently fail to identify errors and offer specific actionable advice for candidate performance improvement in HR interviews. Our research suggests that the current state-of-the-art pre-trained LLMs are not fully conducive for automatic deployment in an HR interview assessment. Instead, our findings advocate for a human-in-the-loop approach, to incorporate manual checks for inconsistencies and provisions for improving feedback quality as a more suitable strategy.
• 摘要：这篇研究论文对包括GPT-4 Turbo，GPT-4 Turbo，GPT-3.5 Turbo，Text-Davinci-003，Text-Babbage-001，Text-Curie-001，Text-Curie-001，Text-ADA-001，Text-ADA-001，Llama-2-7B-Chat，Llama-chat，Llama-chat，Llama-chat-2-2-------------------评估者在模拟人力资源部（人力资源）访谈期间为候选人提供分数，识别错误并为候选人提供反馈和改进建议。我们介绍了一个名为Hurit（人力资源访谈成绩单）的数据集，该数据集包括3,890小时的访谈记录，这些成绩单来自现实世界中的人力资源访谈场景。我们的发现表明，预先培训的LLM，尤其是GPT-4 Turbo和GPT-3.5 Turbo，表现出值得称赞的性能，并且能够产生与专家人类评估者相当的评估。尽管这些LLM在提供与人类评估指标相当的分数方面表现出熟练程度，但它们经常无法识别错误，并为人力资源访谈中的候选绩效提高提供了特定的可行建议。我们的研究表明，当前最新的预训练的LLM在人力资源访谈评估中不完全利用自动部署。取而代之的是，我们的发现主张采用人类的方法，将手动检查纳入了不一致和规定，以提高反馈质量，以作为更合适的策略。

Title: Separator Injection Attack: Uncovering Dialogue Biases in Large Language Models Caused by Role Separators

• Authors: Xitao Li, Haijun Wang, Jiang Wu, Ting Liu
• Subjects: cs.CL, cs.CR
• Abstract URL: https://arxiv.org/abs/2504.05689
• Pdf URL: https://arxiv.org/pdf/2504.05689
• Copy Paste: [[2504.05689]] Separator Injection Attack: Uncovering Dialogue Biases in Large Language Models Caused by Role Separators(https://arxiv.org/abs/2504.05689)
• Keywords: language model, llm, prompt
• Abstract: Conversational large language models (LLMs) have gained widespread attention due to their instruction-following capabilities. To ensure conversational LLMs follow instructions, role separators are employed to distinguish between different participants in a conversation. However, incorporating role separators introduces potential vulnerabilities. Misusing roles can lead to prompt injection attacks, which can easily misalign the model's behavior with the user's intentions, raising significant security concerns. Although various prompt injection attacks have been proposed, recent research has largely overlooked the impact of role separators on safety. This highlights the critical need to thoroughly understand the systemic weaknesses in dialogue systems caused by role separators. This paper identifies modeling weaknesses caused by role separators. Specifically, we observe a strong positional bias associated with role separators, which is inherent in the format of dialogue modeling and can be triggered by the insertion of role separators. We further develop the Separators Injection Attack (SIA), a new orthometric attack based on role separators. The experiment results show that SIA is efficient and extensive in manipulating model behavior with an average gain of 18.2% for manual methods and enhances the attack success rate to 100% with automatic methods.
• 摘要：会话大语言模型（LLM）由于其指导遵循功能而引起了广泛的关注。为了确保对话llms遵循指示，使用角色分离器来区分对话中的不同参与者。但是，结合角色分离器会引入潜在的漏洞。滥用角色会导致迅速注射攻击，这可以轻松地将模型的行为与用户的意图失误，从而引发了重大的安全问题。尽管已经提出了各种迅速的注射攻击，但最近的研究在很大程度上忽略了角色分离器对安全性的影响。这凸显了彻底了解由角色分离器引起的对话系统中的系统弱点。本文确定了由角色分离器引起的建模弱点。具体而言，我们观察到与角色分离器相关的强烈位置偏见，这是对话建模的形式固有的，可以通过插入角色分离器的插入来触发。我们进一步开发了基于角色分离器的新型式攻击分离器注射攻击（SIA）。实验结果表明，SIA在操纵模型行为方面具有高效且广泛，手动方法平均增益为18.2％，并通过自动方法将攻击成功率提高到100％。

Title: STRIVE: A Think & Improve Approach with Iterative Refinement for Enhancing Question Quality Estimation

• Authors: Aniket Deroy, Subhankar Maity
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.05693
• Pdf URL: https://arxiv.org/pdf/2504.05693
• Copy Paste: [[2504.05693]] STRIVE: A Think & Improve Approach with Iterative Refinement for Enhancing Question Quality Estimation(https://arxiv.org/abs/2504.05693)
• Keywords: language model, llm
• Abstract: Automatically assessing question quality is crucial for educators as it saves time, ensures consistency, and provides immediate feedback for refining teaching materials. We propose a novel methodology called STRIVE (Structured Thinking and Refinement with multiLLMs for Improving Verified Question Estimation) using a series of Large Language Models (LLMs) for automatic question evaluation. This approach aims to improve the accuracy and depth of question quality assessment, ultimately supporting diverse learners and enhancing educational practices. The method estimates question quality in an automated manner by generating multiple evaluations based on the strengths and weaknesses of the provided question and then choosing the best solution generated by the LLM. Then the process is improved by iterative review and response with another LLM until the evaluation metric values converge. This sophisticated method of evaluating question quality improves the estimation of question quality by automating the task of question quality evaluation. Correlation scores show that using this proposed method helps to improve correlation with human judgments compared to the baseline method. Error analysis shows that metrics like relevance and appropriateness improve significantly relative to human judgments by using STRIVE.
• 摘要：自动评估问题质量对教育工作者而言至关重要，因为它可以节省时间，确保一致性并为精炼教材提供立即的反馈。我们建议使用一系列大型语言模型（LLMS）进行自动问题评估的一种新颖的方法（用于改进验证问题估计的多illms的结构化思维和改进）。这种方法旨在提高问题质量评估的准确性和深度，最终支持多样化的学习者并增强教育实践。该方法通过根据提供问题的优势和劣势产生多次评估，然后选择LLM生成的最佳解决方案，以自动化的方式估算质量。然后，通过另一个LLM进行迭代审查和响应来改善该过程，直到评估度量值收敛为止。评估问题质量的复杂方法通过自动化质量评估的任务来改善问题质量的估计。相关得分表明，与基线方法相比，使用这种提出的方​​法有助于改善与人类判断的相关性。错误分析表明，相关性和适当性之类的指标通过使用努力而显着相对于人类判断的显着改善。

Title: Evaluating Speech-to-Text Systems with PennSound

• Authors: Jonathan Wright, Mark Liberman, Neville Ryant, James Fiumara
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.05702
• Pdf URL: https://arxiv.org/pdf/2504.05702
• Copy Paste: [[2504.05702]] Evaluating Speech-to-Text Systems with PennSound(https://arxiv.org/abs/2504.05702)
• Keywords: hallucination
• Abstract: A random sample of nearly 10 hours of speech from PennSound, the world's largest online collection of poetry readings and discussions, was used as a benchmark to evaluate several commercial and open-source speech-to-text systems. PennSound's wide variation in recording conditions and speech styles makes it a good representative for many other untranscribed audio collections. Reference transcripts were created by trained annotators, and system transcripts were produced from AWS, Azure, Google, IBM, NeMo, this http URL, Whisper, and this http URL. Based on word error rate, this http URL was the top performer, and Whisper was the top open source performer (as long as hallucinations were avoided). AWS had the best diarization error rates among three systems. However, WER and DER differences were slim, and various tradeoffs may motivate choosing different systems for different end users. We also examine the issue of hallucinations in Whisper. Users of Whisper should be cautioned to be aware of runtime options, and whether the speed vs accuracy trade off is acceptable.
• 摘要：全球最大的在线诗歌阅读和讨论收藏的Pennsound的近10个小时的演讲的随机样本被用作评估几种商业和开源语音到文本系统的基准。 Pennsound在记录条件和语音样式中的广泛差异使其成为许多其他未转录的音频收集的良好代表。参考成绩单是由训练有素的注释者创建的，系统成绩单是由AWS，Azure，Google，IBM，Nemo，此HTTP URL，Whisper和此HTTP URL产生的。根据单词错误率，此HTTP URL是表现最佳的人，而耳语是最高的开源表演者（只要避免幻觉）。在三个系统中，AWS的诊断错误率最好。但是，差异和差异很小，各种权衡可能会激发为不同最终用户选择不同的系统。我们还研究了耳语中的幻觉问题。应注意，低语的用户应了解运行时选项，以及速度与准确性是否可以接受。

Title: LLM$\times$MapReduce-V2: Entropy-Driven Convolutional Test-Time Scaling for Generating Long-Form Articles from Extremely Long Resources

• Authors: Haoyu Wang, Yujia Fu, Zhu Zhang, Shuo Wang, Zirui Ren, Xiaorong Wang, Zhili Li, Chaoqun He, Bo An, Zhiyuan Liu, Maosong Sun
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.05732
• Pdf URL: https://arxiv.org/pdf/2504.05732
• Copy Paste: [[2504.05732]] LLM$\times$MapReduce-V2: Entropy-Driven Convolutional Test-Time Scaling for Generating Long-Form Articles from Extremely Long Resources(https://arxiv.org/abs/2504.05732)
• Keywords: language model, llm
• Abstract: Long-form generation is crucial for a wide range of practical applications, typically categorized into short-to-long and long-to-long generation. While short-to-long generations have received considerable attention, generating long texts from extremely long resources remains relatively underexplored. The primary challenge in long-to-long generation lies in effectively integrating and analyzing relevant information from extensive inputs, which remains difficult for current large language models (LLMs). In this paper, we propose LLM$\times$MapReduce-V2, a novel test-time scaling strategy designed to enhance the ability of LLMs to process extremely long inputs. Drawing inspiration from convolutional neural networks, which iteratively integrate local features into higher-level global representations, LLM$\times$MapReduce-V2 utilizes stacked convolutional scaling layers to progressively expand the understanding of input materials. Both quantitative and qualitative experimental results demonstrate that our approach substantially enhances the ability of LLMs to process long inputs and generate coherent, informative long-form articles, outperforming several representative baselines.
• 摘要：长期生成对于广泛的实用应用至关重要，通常归类为短期和漫长的一代。尽管短期几代人受到了广泛的关注，但从极长的资源中产生长文本仍然相对不受欢迎。远程一代的主要挑战在于有效地整合和分析广泛投入的相关信息，这对于当前的大型语言模型（LLMS）仍然很难。在本文中，我们提出了LLM $ \ times $ mapReduce-v2，这是一种新型的测试时间缩放策略，旨在增强LLM的处理能力非常长的输入。从卷积神经网络中汲取灵感，该卷积神经网络将局部特征整合到更高级别的全局表示中，LLM $ \ times $ mapReduce-v2利用堆叠的卷积缩放层来逐步扩展对输入材料的理解。定量和定性实验结果都表明，我们的方法显着提高了LLM处理长输入并生成相干，信息丰富的长期文章的能力，从而优于几种代表性的基准。

Title: Rank-Then-Score: Enhancing Large Language Models for Automated Essay Scoring

• Authors: Yida Cai, Kun Liang, Sanwoo Lee, Qinghan Wang, Yunfang Wu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.05736
• Pdf URL: https://arxiv.org/pdf/2504.05736
• Copy Paste: [[2504.05736]] Rank-Then-Score: Enhancing Large Language Models for Automated Essay Scoring(https://arxiv.org/abs/2504.05736)
• Keywords: language model, llm, prompt
• Abstract: In recent years, large language models (LLMs) achieve remarkable success across a variety of tasks. However, their potential in the domain of Automated Essay Scoring (AES) remains largely underexplored. Moreover, compared to English data, the methods for Chinese AES is not well developed. In this paper, we propose Rank-Then-Score (RTS), a fine-tuning framework based on large language models to enhance their essay scoring capabilities. Specifically, we fine-tune the ranking model (Ranker) with feature-enriched data, and then feed the output of the ranking model, in the form of a candidate score set, with the essay content into the scoring model (Scorer) to produce the final score. Experimental results on two benchmark datasets, HSK and ASAP, demonstrate that RTS consistently outperforms the direct prompting (Vanilla) method in terms of average QWK across all LLMs and datasets, and achieves the best performance on Chinese essay scoring using the HSK dataset.
• 摘要：近年来，大型语言模型（LLMS）在各种任务中取得了巨大的成功。但是，它们在自动论文评分领域（AES）的潜力在很大程度上仍未得到充实。此外，与英语数据相比，中国AES的方法还不佳。在本文中，我们提出了级别得分（RTS），这是一个基于大语言模型的微调框架，以增强其论文评分能力。具体而言，我们使用富含特征的数据对排名模型（排名）进行微调，然后以候选分数集的形式将排名模型的输出送达，并将论文内容带入评分模型（SCORER）以产生最终分数。在两个基准数据集（HSK和ASAP）上进行的实验结果表明，RTS在所有LLMS和数据集中的平均QWK方面始终优于直接提示（Vanilla）方法，并使用HSK Dataset实现了中国论文评分的最佳性能。

Title: SEA-LION: Southeast Asian Languages in One Network

• Authors: Raymond Ng, Thanh Ngan Nguyen, Yuli Huang, Ngee Chia Tai, Wai Yi Leong, Wei Qi Leong, Xianbin Yong, Jian Gang Ngui, Yosephine Susanto, Nicholas Cheng, Hamsawardhini Rengarajan, Peerat Limkonchotiwat, Adithya Venkatadri Hulagadri, Kok Wai Teng, Yeo Yeow Tong, Bryan Siow, Wei Yi Teo, Wayne Lau, Choon Meng Tan, Brandon Ong, Zhi Hao Ong, Jann Railey Montalan, Adwin Chan, Sajeban Antonyrex, Ren Lee, Esther Choa, David Ong Tat-Wee, Bing Jie Darius Liu, William Chandra Tjhi, Erik Cambria, Leslie Teo
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.05747
• Pdf URL: https://arxiv.org/pdf/2504.05747
• Copy Paste: [[2504.05747]] SEA-LION: Southeast Asian Languages in One Network(https://arxiv.org/abs/2504.05747)
• Keywords: language model, llm
• Abstract: Recently, Large Language Models (LLMs) have dominated much of the artificial intelligence scene with their ability to process and generate natural languages. However, the majority of LLM research and development remains English-centric, leaving low-resource languages such as those in the Southeast Asian (SEA) region under-represented. To address this representation gap, we introduce Llama-SEA-LION-v3-8B-IT and Gemma-SEA-LION-v3-9B-IT, two cutting-edge multilingual LLMs designed for SEA languages. The SEA-LION family of LLMs supports 11 SEA languages, namely English, Chinese, Indonesian, Vietnamese, Malay, Thai, Burmese, Lao, Filipino, Tamil, and Khmer. Our work leverages large-scale multilingual continued pre-training with a comprehensive post-training regime involving multiple stages of instruction fine-tuning, alignment, and model merging. Evaluation results on multilingual benchmarks indicate that our models achieve state-of-the-art performance across LLMs supporting SEA languages. We open-source the models to benefit the wider SEA community.
• 摘要：最近，大型语言模型（LLMS）以其处理和生成自然语言的能力占据了许多人工智能场景。但是，大多数LLM的研发仍以英语为中心，留下低资源的语言，例如东南亚（SEA）地区人数不足的语言。为了解决此表示差距，我们介绍了Llama-sea-sea-lion-v3-8b-it和gemma-sea-sea-lion-v3-9b-it，这是两个尖端的多种语言LLM，为海语设计。 LLMS的海狮家族支持11种海洋语言，即英语，中文，印尼，越南语，马来语，泰国，缅甸，老挝，菲律宾，菲律宾，泰米尔语和高棉。我们的工作利用大规模的多语言继续进行预训练，并采用全面的培训后制度，涉及多个教学的微调，对齐和模型合并。对多语言基准的评估结果表明，我们的模型在支持SEA语言的LLMS中实现最先进的表现。我们开源模型以使更广阔的海洋社区受益。

Title: Layer-Aware Embedding Fusion for LLMs in Text Classifications

• Authors: Jiho Gwak, Yuchul Jung
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.05764
• Pdf URL: https://arxiv.org/pdf/2504.05764
• Copy Paste: [[2504.05764]] Layer-Aware Embedding Fusion for LLMs in Text Classifications(https://arxiv.org/abs/2504.05764)
• Keywords: llm
• Abstract: Embedding fusion has emerged as an effective approach for enhancing performance across various NLP tasks. However, systematic guidelines for selecting optimal layers and developing effective fusion strategies for the integration of LLMs remain underexplored. In this study, we propose a layer-aware embedding selection method and investigate how to quantitatively evaluate different layers to identify the most important ones for downstream NLP tasks, showing that the critical layers vary depending on the dataset. We also explore how combining embeddings from multiple LLMs, without requiring model fine-tuning, can improve performance. Experiments on four English text classification datasets (SST-2, MR, R8, and R52) demonstrate that different layers in LLMs exhibit varying degrees of representational strength for classification, and that combining embeddings from different models can enhance performance if the models exhibit complementary characteristics. Additionally, we discuss resources overhead (memory and inference time) to provide a balanced perspective on the real world feasibility of embedding fusion. Future work will explore multilingual and domain specific datasets, as well as techniques for automating layer selection, to improve both performance and scalability.
• 摘要：嵌入融合已成为提高各种NLP任务性能的有效方法。但是，选择最佳层和制定有效融合策略以集成LLMS的系统指南尚未得到充分反思。在这项研究中，我们提出了一种层次感知的嵌入选择方法，并研究了如何定量评估不同层以识别下游NLP任务的最重要层，这表明临界层根据数据集而有所不同。我们还探讨了在不需要模型进行微调的情况下，将来自多个LLM的嵌入方式组合在一起可以提高性能。在四个英语文本分类数据集（SST-2，MR，R8和R52）上进行的实验表明，LLMS中的不同层具有不同程度的分类强度，并且如果模型互补特征展示了不同模型的嵌入可以增强性能。此外，我们讨论了资源开销（内存和推理时间），以提供有关嵌入融合的现实世界可行性的平衡观点。未来的工作将探索多语言和域特定数据集，以及用于自动化层选择的技术，以提高性能和可伸缩性。

Title: Leveraging Robust Optimization for LLM Alignment under Distribution Shifts

• Authors: Mingye Zhu, Yi Liu, Junbo Guo, Quan Wang, Yongdong Zhang, Zhendong Mao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.05831
• Pdf URL: https://arxiv.org/pdf/2504.05831
• Copy Paste: [[2504.05831]] Leveraging Robust Optimization for LLM Alignment under Distribution Shifts(https://arxiv.org/abs/2504.05831)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) increasingly rely on preference alignment methods to steer outputs toward human values, yet these methods are often constrained by the scarcity of high-quality human-annotated data. To tackle this, recent approaches have turned to synthetic data generated by LLMs as a scalable alternative. However, synthetic data can introduce distribution shifts, compromising the nuanced human preferences that are essential for desirable outputs. In this paper, we propose a novel distribution-aware optimization framework that improves preference alignment in the presence of such shifts. Our approach first estimates the likelihood ratios between the target and training distributions leveraging a learned classifier, then it minimizes the worst-case loss over data regions that reflect the target human-preferred distribution. By explicitly prioritizing the target distribution during optimization, our method mitigates the adverse effects of distributional variation and enhances the generation of responses that faithfully reflect human values.
• 摘要：大型语言模型（LLMS）越来越依赖于偏好对准方法来转向人类价值，但是这些方法通常受到高质量人类注销数据的稀缺性的限制。为了解决这个问题，最近的方法已转向LLMS生成的合成数据作为可扩展的替代方案。但是，综合数据可以引入分布变化，从而损害了对于理想产出所必需的细微人类偏好。在本文中，我们提出了一个新颖的分布感知优化框架，该框架在存在此类转移的情况下改善了偏好对齐。我们的方法首先估算了利用学习分类器的目标和训练分布之间的似然比，然后将最大的损失最小化了反映目标人类偏爱分布的数据区域。通过在优化过程中明确优先考虑目标分布，我们的方法减轻了分布变化的不利影响，并增强了忠实反映人类价值的响应的产生。

Title: Enhancing Coreference Resolution with Pretrained Language Models: Bridging the Gap Between Syntax and Semantics

• Authors: Xingzu Liu, Songhang deng, Mingbang Wang, Zhang Dong, Le Dai, Jiyuan Li, Ruilin Nong
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.05855
• Pdf URL: https://arxiv.org/pdf/2504.05855
• Copy Paste: [[2504.05855]] Enhancing Coreference Resolution with Pretrained Language Models: Bridging the Gap Between Syntax and Semantics(https://arxiv.org/abs/2504.05855)
• Keywords: language model
• Abstract: Large language models have made significant advancements in various natural language processing tasks, including coreference resolution. However, traditional methods often fall short in effectively distinguishing referential relationships due to a lack of integration between syntactic and semantic information. This study introduces an innovative framework aimed at enhancing coreference resolution by utilizing pretrained language models. Our approach combines syntax parsing with semantic role labeling to accurately capture finer distinctions in referential relationships. By employing state-of-the-art pretrained models to gather contextual embeddings and applying an attention mechanism for fine-tuning, we improve the performance of coreference tasks. Experimental results across diverse datasets show that our method surpasses conventional coreference resolution systems, achieving notable accuracy in disambiguating references. This development not only improves coreference resolution outcomes but also positively impacts other natural language processing tasks that depend on precise referential understanding.
• 摘要：大型语言模型已在各种自然语言处理任务（包括核心解决方案）方面取得了重大进步。但是，由于句法和语义信息之间缺乏整合，传统方法通常在有效区分参考关系方面缺乏。这项研究介绍了一个创新的框架，旨在通过使用验证的语言模型来增强核心分辨率。我们的方法将语法解析与语义角色标记结合在一起，以准确捕获参考关系中的更细微的区别。通过采用最先进的预处理模型来收集上下文嵌入并应用了注意机制进行微调，我们可以提高核心任务的性能。各种数据集的实验结果表明，我们的方法超过了常规的核心分辨率系统，在消除参考文献时达到了明显的准确性。这种发展不仅改善了核心分辨率的结果，而且还积极影响了依赖于精确参考理解的其他自然语言处理任务。

Title: Assessing Thai Dialect Performance in LLMs with Automatic Benchmarks and Human Evaluation

• Authors: Peerat Limkonchotiwat, Kanruethai Masuk, Surapon Nonesung, Chalermpun Mai-On, Sarana Nutanong, Wuttikorn Ponwitayarat, Potsawee Manakul
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.05898
• Pdf URL: https://arxiv.org/pdf/2504.05898
• Copy Paste: [[2504.05898]] Assessing Thai Dialect Performance in LLMs with Automatic Benchmarks and Human Evaluation(https://arxiv.org/abs/2504.05898)
• Keywords: language model, gpt, llm
• Abstract: Large language models show promising results in various NLP tasks. Despite these successes, the robustness and consistency of LLMs in underrepresented languages remain largely unexplored, especially concerning local dialects. Existing benchmarks also focus on main dialects, neglecting LLMs' ability on local dialect texts. In this paper, we introduce a Thai local dialect benchmark covering Northern (Lanna), Northeastern (Isan), and Southern (Dambro) Thai, evaluating LLMs on five NLP tasks: summarization, question answering, translation, conversation, and food-related tasks. Furthermore, we propose a human evaluation guideline and metric for Thai local dialects to assess generation fluency and dialect-specific accuracy. Results show that LLM performance declines significantly in local Thai dialects compared to standard Thai, with only proprietary models like GPT-4o and Gemini2 demonstrating some fluency
• 摘要：大型语言模型在各种NLP任务中显示出令人鼓舞的结果。尽管取得了这些成功，但在代表性不足的语言中，LLM的鲁棒性和一致性在很大程度上尚未得到探索，尤其是关于本地方言。现有的基准也集中在主要方言上，忽略了LLMS对本地方言文本的能力。在本文中，我们介绍了泰国本地方言基准，其中涵盖了北部（兰纳），东北（伊萨）和南部（dambro）泰语，评估了五个NLP任务的LLMS：摘要，问答，翻译，对话，对话和与食物相关的任务。此外，我们提出了泰国本地方言的人类评估指南和指标，以评估发电的流利度和方言特定的精度。结果表明，与标准泰语相比，LLM性能在本地泰国方言中大幅下降，只有GPT-4O和Gemini2（例如GPT-4O和Gemini2）表现出一定的流利性

Title: Unsupervised Location Mapping for Narrative Corpora

• Authors: Eitan Wagner, Renana Keydar, Omri Abend
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2504.05954
• Pdf URL: https://arxiv.org/pdf/2504.05954
• Copy Paste: [[2504.05954]] Unsupervised Location Mapping for Narrative Corpora(https://arxiv.org/abs/2504.05954)
• Keywords: language model
• Abstract: This work presents the task of unsupervised location mapping, which seeks to map the trajectory of an individual narrative on a spatial map of locations in which a large set of narratives take place. Despite the fundamentality and generality of the task, very little work addressed the spatial mapping of narrative texts. The task consists of two parts: (1) inducing a ``map'' with the locations mentioned in a set of texts, and (2) extracting a trajectory from a single narrative and positioning it on the map. Following recent advances in increasing the context length of large language models, we propose a pipeline for this task in a completely unsupervised manner without predefining the set of labels. We test our method on two different domains: (1) Holocaust testimonies and (2) Lake District writing, namely multi-century literature on travels in the English Lake District. We perform both intrinsic and extrinsic evaluations for the task, with encouraging results, thereby setting a benchmark and evaluation practices for the task, as well as highlighting challenges.
• 摘要：这项工作介绍了无监督的位置映射的任务，该任务旨在在发生大量叙述的位置的空间图上绘制单个叙述的轨迹。尽管任务的基本意义和普遍性，但很少有工作解决叙事文本的空间映射。该任务由两个部分组成：（1）诱导``映射''，其中一组文本中提到的位置以及（2）从单个叙述中提取轨迹并将其定位在地图上。在增加大语言模型的上下文长度方面的最新进展之后，我们以完全无监督的方式为此任务提出了一条管道，而无需预先定义标签集。我们在两个不同的领域测试我们的方法：（1）大屠杀证词和（2）湖区写作，即英国湖区旅行的多个世纪文献。我们为任务执行内在和外在评估，并令人鼓舞，从而为任务设定了基准和评估实践，并突出了挑战。

Title: NativQA Framework: Enabling LLMs with Native, Local, and Everyday Knowledge

• Authors: Firoj Alam, Md Arid Hasan, Sahinur Rahman Laskar, Mucahid Kutlu, Shammur Absar Chowdhury
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.05995
• Pdf URL: https://arxiv.org/pdf/2504.05995
• Copy Paste: [[2504.05995]] NativQA Framework: Enabling LLMs with Native, Local, and Everyday Knowledge(https://arxiv.org/abs/2504.05995)
• Keywords: language model, llm
• Abstract: The rapid advancement of large language models (LLMs) has raised concerns about cultural bias, fairness, and their applicability in diverse linguistic and underrepresented regional contexts. To enhance and benchmark the capabilities of LLMs, there is a need to develop large-scale resources focused on multilingual, local, and cultural contexts. In this study, we propose a framework, NativQA, that can seamlessly construct large-scale, culturally and regionally aligned QA datasets in native languages. The framework utilizes user-defined seed queries and leverages search engines to collect location-specific, everyday information. It has been evaluated across 39 locations in 24 countries and in 7 languages, ranging from extremely low-resource to high-resource languages, which resulted over 300K Question Answer (QA) pairs. The developed resources can be used for LLM benchmarking and further fine-tuning. The framework has been made publicly available for the community (this https URL).
• 摘要：大型语言模型（LLM）的快速发展引起了人们对文化偏见，公平性及其在多种语言和代表性不足的地区环境中的关注。为了增强和基准LLM的功能，有必要开发针对多语言，本地和文化背景的大规模资源。在这项研究中，我们提出了一个框架Nativqa，该框架可以无缝地构建大型，文化和区域对齐的QA数据集中的母语。该框架利用用户定义的种子查询，并利用搜索引擎来收集特定于位置的日常信息。它已经在24个国家 /地区的39个地点和7种语言中进行了评估，从极低的资源到高资源的语言，这导致了300k个问题答案（QA）对。开发的资源可用于LLM基准测试和进一步的微调。该框架已公开用于社区（此HTTPS URL）。

Title: Llama-3-Nanda-10B-Chat: An Open Generative Large Language Model for Hindi

• Authors: Monojit Choudhury, Shivam Chauhan, Rocktim Jyoti Das, Dhruv Sahnan, Xudong Han, Haonan Li, Aaryamonvikram Singh, Alok Anil Jadhav, Utkarsh Agarwal, Mukund Choudhary, Debopriyo Banerjee, Fajri Koto, Junaid Bhat, Awantika Shukla, Samujjwal Ghosh, Samta Kamboj, Onkar Pandit, Lalit Pradhan, Rahul Pal, Sunil Sahu, Soundar Doraiswamy, Parvez Mullah, Ali El Filali, Neha Sengupta, Gokul Ramakrishnan, Rituraj Joshi, Gurpreet Gosal, Avraham Sheinin, Natalia Vassilieva, Preslav Nakov
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.06011
• Pdf URL: https://arxiv.org/pdf/2504.06011
• Copy Paste: [[2504.06011]] Llama-3-Nanda-10B-Chat: An Open Generative Large Language Model for Hindi(https://arxiv.org/abs/2504.06011)
• Keywords: language model, llm, chat
• Abstract: Developing high-quality large language models (LLMs) for moderately resourced languages presents unique challenges in data availability, model adaptation, and evaluation. We introduce Llama-3-Nanda-10B-Chat, or Nanda for short, a state-of-the-art Hindi-centric instruction-tuned generative LLM, designed to push the boundaries of open-source Hindi language models. Built upon Llama-3-8B, Nanda incorporates continuous pre-training with expanded transformer blocks, leveraging the Llama Pro methodology. A key challenge was the limited availability of high-quality Hindi text data; we addressed this through rigorous data curation, augmentation, and strategic bilingual training, balancing Hindi and English corpora to optimize cross-linguistic knowledge transfer. With 10 billion parameters, Nanda stands among the top-performing open-source Hindi and multilingual models of similar scale, demonstrating significant advantages over many existing models. We provide an in-depth discussion of training strategies, fine-tuning techniques, safety alignment, and evaluation metrics, demonstrating how these approaches enabled Nanda to achieve state-of-the-art results. By open-sourcing Nanda, we aim to advance research in Hindi LLMs and support a wide range of real-world applications across academia, industry, and public services.
• 摘要：为中等资源的语言开发高质量的大语言模型（LLM）提出了数据可用性，模型适应和评估方面的独特挑战。我们介绍了Llama-3-Nanda-10b-chat或Nanda，这是一种以最先进的印地语指导调整的生成LLM，旨在突破开源印地语语言模型的界限。 Nanda建立在Llama-3-8B的基础上，将连续的预训练与扩展的变压器块结合在一起，利用Llama Po方法论。一个关键的挑战是高质量印地语文本数据的可用性有限。我们通过严格的数据策展，增强和战略双语培训，平衡印地语和英语语料库来优化跨语言知识转移，从而解决了这一问题。 Nanda拥有100亿个参数，是表现最佳的开源印地语和类似规模的多语言模型，这表明了与许多现有模型相比，具有显着优势。我们提供了有关培训策略，微调技术，安全一致性和评估指标的深入讨论，以证明这些方法如何使Nanda获得最新的结果。通过开源Nanda，我们旨在推进印地语LLM的研究，并支持各种学术界，工业和公共服务的广泛现实应用程序。

Title: Multi-Sense Embeddings for Language Models and Knowledge Distillation

• Authors: Qitong Wang, Mohammed J. Zaki, Georgios Kollias, Vasileios Kalantzis
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.06036
• Pdf URL: https://arxiv.org/pdf/2504.06036
• Copy Paste: [[2504.06036]] Multi-Sense Embeddings for Language Models and Knowledge Distillation(https://arxiv.org/abs/2504.06036)
• Keywords: language model, llm
• Abstract: Transformer-based large language models (LLMs) rely on contextual embeddings which generate different (continuous) representations for the same token depending on its surrounding context. Nonetheless, words and tokens typically have a limited number of senses (or meanings). We propose multi-sense embeddings as a drop-in replacement for each token in order to capture the range of their uses in a language. To construct a sense embedding dictionary, we apply a clustering algorithm to embeddings generated by an LLM and consider the cluster centers as representative sense embeddings. In addition, we propose a novel knowledge distillation method that leverages the sense dictionary to learn a smaller student model that mimics the senses from the much larger base LLM model, offering significant space and inference time savings, while maintaining competitive performance. Via thorough experiments on various benchmarks, we showcase the effectiveness of our sense embeddings and knowledge distillation approach. We share our code at this https URL
• 摘要：基于变压器的大型语言模型（LLMS）依赖于上下文嵌入，这些嵌入会根据周围的上下文产生相同令牌的不同（连续）表示。但是，单词和令牌通常具有有限的感官（或含义）。我们建议将多态嵌入式嵌入作为每个令牌的倒入替代品，以捕获其用语的范围。为了构建一种嵌入字典的感觉，我们将群集算法应用于LLM生成的嵌入，并将群集中心视为代表性的感官嵌入。此外，我们提出了一种新型的知识蒸馏方法，该方法利用了Sense词典学习一个较小的学生模型，该模型模仿了较大的基本LLM模型，从而提供了大量的空间和节省时间，同时保持了竞争性能。通过对各种基准测试的彻底实验，我们展示了感官嵌入和知识蒸馏方法的有效性。我们在此HTTPS URL上共享代码

Title: Confidence Regularized Masked Language Modeling using Text Length

• Authors: Seunghyun Ji, Soowon Lee
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2504.06037
• Pdf URL: https://arxiv.org/pdf/2504.06037
• Copy Paste: [[2504.06037]] Confidence Regularized Masked Language Modeling using Text Length(https://arxiv.org/abs/2504.06037)
• Keywords: language model
• Abstract: Masked language modeling, which is a task to predict a randomly masked word in the input text, is an efficient language representation learning method. Masked language modeling ignores various words which people can think of for filling in the masked position and calculates the loss with a single word. Especially when the input text is short, the entropy of the word distribution that can fill in the masked position can be high. This may cause the model to be overconfident in the single answer. To address this issue, we propose a novel confidence regularizer that controls regularizing strength dynamically by the input text length. Experiments with GLUE and SQuAD datasets showed that our method achieves better accuracy and lower expected calibration error.
• 摘要：蒙版语言建模是一种预测输入文本中随机掩盖单词的任务，是一种有效的语言表示方法。蒙面的语言建模忽略了人们可以想到的各种单词，以填补掩盖的位置并用一个单词计算损失。尤其是当输入文本短时，可以填充掩盖位置的单词分布的熵可能很高。这可能会导致模型在单个答案中过度自信。为了解决这个问题，我们提出了一个新颖的置信度正常化程序，该置信度正常通过输入文本长度动态地正规化强度。使用胶水和小队数据集的实验表明，我们的方法可实现更好的准确性和较低的预期校准误差。

Title: QGen Studio: An Adaptive Question-Answer Generation, Training and Evaluation Platform

• Authors: Movina Moses, Mohab Elkaref, James Barry, Shinnosuke Tanaka, Vishnudev Kuruvanthodi, Nathan Herr, Campbell D Watson, Geeth De Mel
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.06136
• Pdf URL: https://arxiv.org/pdf/2504.06136
• Copy Paste: [[2504.06136]] QGen Studio: An Adaptive Question-Answer Generation, Training and Evaluation Platform(https://arxiv.org/abs/2504.06136)
• Keywords: language model, llm
• Abstract: We present QGen Studio: an adaptive question-answer generation, training, and evaluation platform. QGen Studio enables users to leverage large language models (LLMs) to create custom question-answer datasets and fine-tune models on this synthetic data. It features a dataset viewer and model explorer to streamline this process. The dataset viewer provides key metrics and visualizes the context from which the QA pairs are generated, offering insights into data quality. The model explorer supports model comparison, allowing users to contrast the performance of their trained LLMs against other models, supporting performance benchmarking and refinement. QGen Studio delivers an interactive, end-to-end solution for generating QA datasets and training scalable, domain-adaptable models. The studio will be open-sourced soon, allowing users to deploy it locally.
• 摘要：我们提出了QGEN Studio：一个自适应的提问者生成，培训和评估平台。 QGEN Studio使用户能够利用大型语言模型（LLMS）在此综合数据上创建自定义的问题 - 答案数据集和微调模型。它具有数据集查看器和模型资源管理器来简化此过程。数据集查看器提供关键指标，并可视化质量检查对生成的上下文，从而提供了对数据质量的见解。模型Explorer支持模型比较，使用户可以将其训练有素的LLM与其他模型的性能进行对比，从而支持性能基准测试和改进。 QGEN Studio提供了一种交互式的端到端解决方案，用于生成QA数据集和训练可扩展的域适应模型。该工作室将很快开源，使用户可以在本地部署它。

Title: Navigating the Rabbit Hole: Emergent Biases in LLM-Generated Attack Narratives Targeting Mental Health Groups

• Authors: Rijul Magu, Arka Dutta, Sean Kim, Ashiqur R. KhudaBukhsh, Munmun De Choudhury
• Subjects: cs.CL, cs.AI, cs.CY, cs.LG, cs.SI
• Abstract URL: https://arxiv.org/abs/2504.06160
• Pdf URL: https://arxiv.org/pdf/2504.06160
• Copy Paste: [[2504.06160]] Navigating the Rabbit Hole: Emergent Biases in LLM-Generated Attack Narratives Targeting Mental Health Groups(https://arxiv.org/abs/2504.06160)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have been shown to demonstrate imbalanced biases against certain groups. However, the study of unprovoked targeted attacks by LLMs towards at-risk populations remains underexplored. Our paper presents three novel contributions: (1) the explicit evaluation of LLM-generated attacks on highly vulnerable mental health groups; (2) a network-based framework to study the propagation of relative biases; and (3) an assessment of the relative degree of stigmatization that emerges from these attacks. Our analysis of a recently released large-scale bias audit dataset reveals that mental health entities occupy central positions within attack narrative networks, as revealed by a significantly higher mean centrality of closeness (p-value = 4.06e-10) and dense clustering (Gini coefficient = 0.7). Drawing from sociological foundations of stigmatization theory, our stigmatization analysis indicates increased labeling components for mental health disorder-related targets relative to initial targets in generation chains. Taken together, these insights shed light on the structural predilections of large language models to heighten harmful discourse and highlight the need for suitable approaches for mitigation.
• 摘要：大型语言模型（LLM）已显示出对某些群体的偏见不平衡。然而，LLM对处于危险人群的无端针对性攻击的研究尚未得到充实。我们的论文提出了三个新的贡献：（1）对LLM生成的攻击对高度脆弱的心理健康团体的明确评估； （2）基于网络的框架，用于研究相对偏见的传播； （3）对这些攻击产生的污名相对程度的评估。我们对最近发布的大规模偏见审核数据集的分析表明，精神卫生实体在攻击叙事网络中占据中心位置，这表明，紧密的平均中心性（p值= 4.06e-10）和密集的聚类（Gini系数= 0.7）表明。从污名化理论的社会学基础上，我们的污名化分析表明，相对于发电链中的初始目标，与精神障碍相关目标的标记成分增加。综上所述，这些见解阐明了大语言模型的结构性偏好，以加强有害话语，并强调需要进行适当的缓解方法。

Title: Assessing how hyperparameters impact Large Language Models' sarcasm detection performance

• Authors: Montgomery Gole, Andriy Miranskyy
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.06166
• Pdf URL: https://arxiv.org/pdf/2504.06166
• Copy Paste: [[2504.06166]] Assessing how hyperparameters impact Large Language Models' sarcasm detection performance(https://arxiv.org/abs/2504.06166)
• Keywords: language model, gpt
• Abstract: Sarcasm detection is challenging for both humans and machines. This work explores how model characteristics impact sarcasm detection in OpenAI's GPT, and Meta's Llama-2 models, given their strong natural language understanding, and popularity. We evaluate fine-tuned and zero-shot models across various sizes, releases, and hyperparameters. Experiments were conducted on the political and balanced (pol-bal) portion of the popular Self-Annotated Reddit Corpus (SARC2.0) sarcasm dataset. Fine-tuned performance improves monotonically with model size within a model family, while hyperparameter tuning also impacts performance. In the fine-tuning scenario, full precision Llama-2-13b achieves state-of-the-art accuracy and $F_1$-score, both measured at 0.83, comparable to average human performance. In the zero-shot setting, one GPT-4 model achieves competitive performance to prior attempts, yielding an accuracy of 0.70 and an $F_1$-score of 0.75. Furthermore, a model's performance may increase or decline with each release, highlighting the need to reassess performance after each release.
• 摘要：讽刺的检测对于人类和机器都具有挑战性。这项工作探讨了模型特征如何影响OpenAI的GPT和Meta的Llama-2模型中的讽刺检测，并且鉴于其强烈的自然语言理解和受欢迎程度。我们评估了各种尺寸，释放和超级参数的微调和零拍模型。实验是对流行的自称Reddit语料库（SARC2.0）讽刺数据集的政治和平衡（Pol-Bal）部分进行的。微调的性能可以通过模型系列内的模型大小来单调改善，而超参数调整也会影响性能。在微调方案中，完整的精确度千层面-2-13B实现了最先进的精度和$ f_1 $ - 分数，均为0.83，与平均人类绩效相当。在零拍摄的设置中，一个GPT-4型号可以实现先前尝试的竞争性能，其准确度为0.70，而$ f_1 $ -score的精度为0.75。此外，每次发布时，模型的性能可能会提高或下降，强调每次发行后都需要重新评估性能。

Title: From 128K to 4M: Efficient Training of Ultra-Long Context Large Language Models

• Authors: Chejian Xu, Wei Ping, Peng Xu, Zihan Liu, Boxin Wang, Mohammad Shoeybi, Bo Li, Bryan Catanzaro
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2504.06214
• Pdf URL: https://arxiv.org/pdf/2504.06214
• Copy Paste: [[2504.06214]] From 128K to 4M: Efficient Training of Ultra-Long Context Large Language Models(https://arxiv.org/abs/2504.06214)
• Keywords: language model, llm, long context
• Abstract: Long-context capabilities are essential for a wide range of applications, including document and video understanding, in-context learning, and inference-time scaling, all of which require models to process and reason over long sequences of text and multimodal data. In this work, we introduce a efficient training recipe for building ultra-long context LLMs from aligned instruct model, pushing the boundaries of context lengths from 128K to 1M, 2M, and 4M tokens. Our approach leverages efficient continued pretraining strategies to extend the context window and employs effective instruction tuning to maintain the instruction-following and reasoning abilities. Our UltraLong-8B, built on Llama3.1-Instruct with our recipe, achieves state-of-the-art performance across a diverse set of long-context benchmarks. Importantly, models trained with our approach maintain competitive performance on standard benchmarks, demonstrating balanced improvements for both long and short context tasks. We further provide an in-depth analysis of key design choices, highlighting the impacts of scaling strategies and data composition. Our findings establish a robust framework for efficiently scaling context lengths while preserving general model capabilities. We release all model weights at: this https URL.
• 摘要：长篇小说功能对于广泛的应用程序，包括文档和视频理解，内在学习和推理时间缩放至关重要，所有这些都需要模型在文本和多模式数据的长序列上处理和推理。在这项工作中，我们引入了一种有效的培训配方，以从对齐的指示模型中构建超长上下文LLM，从而将上下文长度的边界从128K，2M，2M和4M代币中推动。我们的方法利用有效的策略来扩展上下文窗口，并采用有效的教学调整来维持遵守指导和推理能力。我们的Ultralong-8B以我们的食谱为基础建立在Llama3.1教学的基础上，在各种长篇小说基准中实现了最先进的性能。重要的是，接受我们方法训练的模型在标准基准测试上保持竞争性能，证明了长期和短上下文任务的平衡改进。我们进一步提供了对关键设计选择的深入分析，强调了扩展策略和数据组成的影响。我们的发现建立了一个强大的框架，用于有效地扩展上下文长度，同时保留一般模型功能。我们释放所有模型权重，at：this HTTPS URL。

Title: Can Performant LLMs Be Ethical? Quantifying the Impact of Web Crawling Opt-Outs

• Authors: Dongyang Fan, Vinko Sabolčec, Matin Ansaripour, Ayush Kumar Tarun, Martin Jaggi, Antoine Bosselut, Imanol Schlag
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2504.06219
• Pdf URL: https://arxiv.org/pdf/2504.06219
• Copy Paste: [[2504.06219]] Can Performant LLMs Be Ethical? Quantifying the Impact of Web Crawling Opt-Outs(https://arxiv.org/abs/2504.06219)
• Keywords: language model, llm
• Abstract: The increasing adoption of web crawling opt-outs by copyright holders of online content raises critical questions about the impact of data compliance on large language model (LLM) performance. However, little is known about how these restrictions (and the resultant filtering of pretraining datasets) affect the capabilities of models trained using these corpora. In this work, we conceptualize this effect as the $\textit{data compliance gap}$ (DCG), which quantifies the performance difference between models trained on datasets that comply with web crawling opt-outs, and those that do not. We measure the data compliance gap in two settings: pretraining models from scratch and continual pretraining from existing compliant models (simulating a setting where copyrighted data could be integrated later in pretraining). Our experiments with 1.5B models show that, as of January 2025, compliance with web data opt-outs does not degrade general knowledge acquisition (close to 0\% DCG). However, in specialized domains such as biomedical research, excluding major publishers leads to performance declines. These findings suggest that while general-purpose LLMs can be trained to perform equally well using fully open data, performance in specialized domains may benefit from access to high-quality copyrighted sources later in training. Our study provides empirical insights into the long-debated trade-off between data compliance and downstream model performance, informing future discussions on AI training practices and policy decisions.
• 摘要：在线内容版权持有人对Web爬网的选择越来越多地提出了有关数据合规性对大语言模型（LLM）性能的影响的关键问题。但是，对于这些限制（以及预处理数据集的结果过滤）如何影响使用这些语料库训练的模型的功能，知之甚少。在这项工作中，我们将此效果概念化为$ \ textIt {数据合规性gap} $（DCG），该效果量化了符合Web crawling Rept outs的数据集中训练的模型之间的性能差异，而那些不符合Web。我们在两种设置中测量数据合规差距：从头开始进行预处理的模型，以及从现有符合性模型中进行持续预处理（模拟可以在稍后预审计的设置中进行版权保护数据的设置）。我们使用1.5B模型进行的实验表明，截至2025年1月，符合Web数据退出不会降低通用知识获取（接近0 \％DCG）。但是，在诸如生物医学研究之类的专业领域中，不包括主要出版商会导致绩效下降。这些发现表明，尽管可以使用完全开放的数据对通用LLM进行培训以表现出色，但专用域中的性能可能会受益于以后在培训中获得高质量的受版权保护来源的访问。我们的研究提供了对数据合规性和下游模型绩效之间长期折衷的经验见解，从而为未来的AI培训实践和政策决策提供了讨论。

Title: Encoder-Decoder Gemma: Improving the Quality-Efficiency Trade-Off via Adaptation

• Authors: Biao Zhang, Fedor Moiseev, Joshua Ainslie, Paul Suganthan, Min Ma, Surya Bhupatiraju, Fede Lebron, Orhan Firat, Armand Joulin, Zhe Dong
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2504.06225
• Pdf URL: https://arxiv.org/pdf/2504.06225
• Copy Paste: [[2504.06225]] Encoder-Decoder Gemma: Improving the Quality-Efficiency Trade-Off via Adaptation(https://arxiv.org/abs/2504.06225)
• Keywords: language model, llm
• Abstract: While decoder-only large language models (LLMs) have shown impressive results, encoder-decoder models are still widely adopted in real-world applications for their inference efficiency and richer encoder representation. In this paper, we study a novel problem: adapting pretrained decoder-only LLMs to encoder-decoder, with the goal of leveraging the strengths of both approaches to achieve a more favorable quality-efficiency trade-off. We argue that adaptation not only enables inheriting the capability of decoder-only LLMs but also reduces the demand for computation compared to pretraining from scratch. We rigorously explore different pretraining objectives and parameter initialization/optimization techniques. Through extensive experiments based on Gemma 2 (2B and 9B) and a suite of newly pretrained mT5-sized models (up to 1.6B), we demonstrate the effectiveness of adaptation and the advantage of encoder-decoder LLMs. Under similar inference budget, encoder-decoder LLMs achieve comparable (often better) pretraining performance but substantially better finetuning performance than their decoder-only counterpart. For example, Gemma 2B-2B outperforms Gemma 2B by $\sim$7\% after instruction tuning. Encoder-decoder adaptation also allows for flexible combination of different-sized models, where Gemma 9B-2B significantly surpasses Gemma 2B-2B by $>$3\%. The adapted encoder representation also yields better results on SuperGLUE. We will release our checkpoints to facilitate future research.
• 摘要：尽管仅解码器的大型语言模型（LLMS）显示出令人印象深刻的结果，但编码器模型仍在现实世界应用中广泛采用，以提高推断效率和更丰富的编码器表示。在本文中，我们研究了一个新的问题：适应预算的解码器LLMS来编码编码器，目的是利用两种方法的优势来实现更有利的质量效率折衷。我们认为，适应不仅可以继承仅解码器的LLM的能力，而且还可以减少与从头开始的预处理相比，减少了对计算的需求。我们严格探讨了不同的预审前的目标和参数初始化/优化技术。通过基于Gemma 2（2b和9b）的广泛实验以及一套新预验证的MT5尺寸模型（最高1.6B），我们证明了适应性的有效性和Encoder-Decoder llms的优势。在类似的推理预算下，编码器decoder llms的性能可比性（通常更好），但比仅解码器的同类产品要好得多。例如，Gemma 2b-2b在指令调整后，$ \ sim $ 7 \％以$ \ sim $ 7 \％优于gemma 2b。编码器 - 模型的改编还可以灵活地组合不同尺寸的型号，其中Gemma 9B-2B显着超过Gemma 2B-2B $> $ 3 \％。改编的编码器表示还可以在超粘合期间获得更好的结果。我们将发布我们的检查站，以促进未来的研究。

Title: LExT: Towards Evaluating Trustworthiness of Natural Language Explanations

• Authors: Krithi Shailya, Shreya Rajpal, Gokul S Krishnan, Balaraman Ravindran
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.06227
• Pdf URL: https://arxiv.org/pdf/2504.06227
• Copy Paste: [[2504.06227]] LExT: Towards Evaluating Trustworthiness of Natural Language Explanations(https://arxiv.org/abs/2504.06227)
• Keywords: language model, llm
• Abstract: As Large Language Models (LLMs) become increasingly integrated into high-stakes domains, there have been several approaches proposed toward generating natural language explanations. These explanations are crucial for enhancing the interpretability of a model, especially in sensitive domains like healthcare, where transparency and reliability are key. In light of such explanations being generated by LLMs and its known concerns, there is a growing need for robust evaluation frameworks to assess model-generated explanations. Natural Language Generation metrics like BLEU and ROUGE capture syntactic and semantic accuracies but overlook other crucial aspects such as factual accuracy, consistency, and faithfulness. To address this gap, we propose a general framework for quantifying trustworthiness of natural language explanations, balancing Plausibility and Faithfulness, to derive a comprehensive Language Explanation Trustworthiness Score (LExT) (The code and set up to reproduce our experiments are publicly available at this https URL). Applying our domain-agnostic framework to the healthcare domain using public medical datasets, we evaluate six models, including domain-specific and general-purpose models. Our findings demonstrate significant differences in their ability to generate trustworthy explanations. On comparing these explanations, we make interesting observations such as inconsistencies in Faithfulness demonstrated by general-purpose models and their tendency to outperform domain-specific fine-tuned models. This work further highlights the importance of using a tailored evaluation framework to assess natural language explanations in sensitive fields, providing a foundation for improving the trustworthiness and transparency of language models in healthcare and beyond.
• 摘要：随着大型语言模型（LLM）越来越多地整合到高风险领域中，已经提出了几种用于产生自然语言解释的方法。这些解释对于增强模型的解释性至关重要，尤其是在透明度和可靠性是关键的医疗保健等敏感领域中。鉴于LLMS产生的这种解释及其已知的关注点，越来越需要强大的评估框架来评估模型生成的解释。 BLEU和Rouge等自然语言产生指标捕获了句法和语义精确度，但忽略了其他关键方面，例如事实准确性，一致性和忠诚。为了解决这一差距，我们提出了一个通用框架，以量化自然语言解释，平衡合理性和忠诚的信任度，以获得全面的语言解释可信度评分（LEXT）（该代码并设置为重现我们的实验，在此HTTPS url中可以公开使用）。我们使用公共医疗数据集将我们的领域 - 不合稳定框架应用于医疗保健领域，我们评估了六个模型，包括域特异性和通用模型。我们的发现表明，它们在产生值得信赖的解释的能力上存在显着差异。在比较这些解释时，我们做出了有趣的观察结果，例如通用模型所证明的忠实不一致及其趋于胜过特定领域的微型模型的趋势。这项工作进一步强调了使用量身定制的评估框架评估敏感领域的自然语言解释的重要性，从而为提高医疗保健及其他方面语言模型的可信度和透明度提供了基础。