2025-03-24

Title: Token-Level Uncertainty-Aware Objective for Language Model Post-Training

• Authors: Tingkai Liu, Ari S. Benjamin, Anthony M. Zador
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.16511
• Pdf URL: https://arxiv.org/pdf/2503.16511
• Copy Paste: [[2503.16511]] Token-Level Uncertainty-Aware Objective for Language Model Post-Training(https://arxiv.org/abs/2503.16511)
• Keywords: language model, gpt
• Abstract: In the current work, we connect token-level uncertainty in causal language modeling to two types of training objectives: 1) masked maximum likelihood (MLE), 2) self-distillation. We show that masked MLE is effective in reducing epistemic uncertainty, and serve as an effective token-level automatic curriculum learning technique. However, masked MLE is prone to overfitting and requires self-distillation regularization to improve or maintain performance on out-of-distribution tasks. We demonstrate significant performance gain via the proposed training objective - combined masked MLE and self-distillation - across multiple architectures (Gemma, LLaMA, Phi) and datasets (Alpaca, ShareGPT, GSM8K), mitigating overfitting while maintaining adaptability during post-training. Our findings suggest that uncertainty-aware training provides an effective mechanism for enhancing language model training.
• 摘要：在当前的工作中，我们将因果语言建模中的令牌级不确定性连接到两种类型的培训目标：1）掩盖最大似然（MLE），2）自我验证。我们表明，蒙面的MLE有效地降低了认知不确定性，并充当有效的令牌自动课程学习技术。但是，蒙面的MLE容易过度拟合，需要自我验证正规化以改善或维持分布式任务的绩效。我们通过提出的训练目标（跨多个体系结构（Gemma，Llama，Phi）和数据集（Alpaca，ShareGPT，GSM8K）跨越了跨性能的训练目标 - 结合掩盖的MLE和自我鉴定，表明了显着的性能增长，从而在培训期间保持适应能力，从而缓解过度拟合。我们的发现表明，不确定性感知培训为增强语言模型培训提供了有效的机制。

Title: Highlighting Case Studies in LLM Literature Review of Interdisciplinary System Science

• Authors: Lachlan McGinness, Peter Baumgartner
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.16515
• Pdf URL: https://arxiv.org/pdf/2503.16515
• Copy Paste: [[2503.16515]] Highlighting Case Studies in LLM Literature Review of Interdisciplinary System Science(https://arxiv.org/abs/2503.16515)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) were used to assist four Commonwealth Scientific and Industrial Research Organisation (CSIRO) researchers to perform systematic literature reviews (SLR). We evaluate the performance of LLMs for SLR tasks in these case studies. In each, we explore the impact of changing parameters on the accuracy of LLM responses. The LLM was tasked with extracting evidence from chosen academic papers to answer specific research questions. We evaluate the models' performance in faithfully reproducing quotes from the literature and subject experts were asked to assess the model performance in answering the research questions. We developed a semantic text highlighting tool to facilitate expert review of LLM responses. We found that state of the art LLMs were able to reproduce quotes from texts with greater than 95% accuracy and answer research questions with an accuracy of approximately 83%. We use two methods to determine the correctness of LLM responses; expert review and the cosine similarity of transformer embeddings of LLM and expert answers. The correlation between these methods ranged from 0.48 to 0.77, providing evidence that the latter is a valid metric for measuring semantic similarity.
• 摘要：大型语言模型（LLM）用于协助四个联邦科学和工业研究组织（CSIRO）研究人员进行系统文献评论（SLR）。在这些案例研究中，我们评估了SLR任务的LLMS的性能。在每个中，我们探讨了改变参数对LLM响应准确性的影响。 LLM的任务是从选定的学术论文中提取证据，以回答具体的研究问题。我们评估了模型在忠实地再现文献报价中的表现，并要求专家评估回答研究问题时的模型绩效。我们开发了一种语义文本突出显示工具，以促进对LLM响应的专家审查。我们发现，最新的LLM的状态能够从准确性超过95％的文本中复制报价，并以约83％的精度回答研究问题。我们使用两种方法来确定LLM响应的正确性。专家评论和LLM的变压器嵌入和专家答案的余弦相似性。这些方法之间的相关性范围为0.48至0.77，提供了证据表明后者是测量语义相似性的有效指标。

Title: Using LLMs for Automated Privacy Policy Analysis: Prompt Engineering, Fine-Tuning and Explainability

• Authors: Yuxin Chen, Peng Tang, Weidong Qiu, Shujun Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.16516
• Pdf URL: https://arxiv.org/pdf/2503.16516
• Copy Paste: [[2503.16516]] Using LLMs for Automated Privacy Policy Analysis: Prompt Engineering, Fine-Tuning and Explainability(https://arxiv.org/abs/2503.16516)
• Keywords: language model, llm, prompt
• Abstract: Privacy policies are widely used by digital services and often required for legal purposes. Many machine learning based classifiers have been developed to automate detection of different concepts in a given privacy policy, which can help facilitate other automated tasks such as producing a more reader-friendly summary and detecting legal compliance issues. Despite the successful applications of large language models (LLMs) to many NLP tasks in various domains, there is very little work studying the use of LLMs for automated privacy policy analysis, therefore, if and how LLMs can help automate privacy policy analysis remains under-explored. To fill this research gap, we conducted a comprehensive evaluation of LLM-based privacy policy concept classifiers, employing both prompt engineering and LoRA (low-rank adaptation) fine-tuning, on four state-of-the-art (SOTA) privacy policy corpora and taxonomies. Our experimental results demonstrated that combining prompt engineering and fine-tuning can make LLM-based classifiers outperform other SOTA methods, \emph{significantly} and \emph{consistently} across privacy policy corpora/taxonomies and concepts. Furthermore, we evaluated the explainability of the LLM-based classifiers using three metrics: completeness, logicality, and comprehensibility. For all three metrics, a score exceeding 91.1\% was observed in our evaluation, indicating that LLMs are not only useful to improve the classification performance, but also to enhance the explainability of detection results.
• 摘要：隐私政策被数字服务广泛使用，通常是为了法律目的。已经开发出许多基于机器的基于机器的分类器来自动检测给定的隐私政策中不同概念，这可以帮助促进其他自动化任务，例如产生更友好的摘要和检测法律合规性问题。尽管大型语言模型（LLMS）成功地应用于各个领域的许多NLP任务，但研究LLM用于自动隐私政策分析的工作很少，因此，IF和LLMS可以帮助自动化隐私政策分析的方法仍然不足。为了填补这一研究差距，我们对基于LLM的隐私政策概念分类器进行了全面评估，同时采用了迅速的工程和LORA（低级别适应）微调，对四个最先进的（SOTA）隐私政策公司和分类法。我们的实验结果表明，将及时的工程和微调结合起来可以使基于LLM的分类器优于其他SOTA方法，即\ emph {显着}和\ emph {始终如一的emph {一致}跨隐私政策语料库/分类法/分类法和概念。此外，我们使用三个指标评估了基于LLM的分类器的解释性：完整性，逻辑性和可理解性。对于所有三个指标，在我们的评估中都观察到超过91.1 \％的分数，表明LLM不仅有用可改善分类性能，而且还可以增强检测结果的解释性。

Title: KVShare: Semantic-Aware Key-Value Cache Sharing for Efficient Large Language Model Inference

• Authors: Huan Yang, Renji Zhang, Deyu Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.16525
• Pdf URL: https://arxiv.org/pdf/2503.16525
• Copy Paste: [[2503.16525]] KVShare: Semantic-Aware Key-Value Cache Sharing for Efficient Large Language Model Inference(https://arxiv.org/abs/2503.16525)
• Keywords: language model, llm
• Abstract: This paper presents KVShare, a multi-user Key-Value (KV) Cache sharing technology based on semantic similarity, designed to enhance the inference efficiency of Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs). Addressing the limitations of existing prefix caching (strict text prefix matching) and semantic caching (loss of response diversity), KVShare achieves fine-grained KV cache reuse through semantic alignment algorithms and differential editing operations. Experiments on real-world user conversation datasets demonstrate that KVShare improves KV cache hit rates by over 60%, while maintaining output quality comparable to full computation (no significant degradation in BLEU and Rouge-L metrics). This approach effectively reduces GPU resource consumption and is applicable to scenarios with repetitive queries, such as healthcare and education.
• 摘要：本文介绍了基于语义相似性的多用户键值（KV）缓存共享技术的KVShare，旨在提高大语言模型（LLMS）和多模式大语言模型（MLLMS）的推理效率。 KVShare解决了现有前缀缓存（严格的文本前缀匹配）和语义缓存（响应多样性的丧失）的局限性，通过语义对齐算法和差分编辑操作，KVShare实现了细粒度的KV缓存重复使用。现实世界中用户对话数据集的实验表明，KVShare将KV Cache HIT率提高了60％以上，同时保持与完整计算相当的输出质量（BLEU和ROUGE-L METICS无显着降解）。这种方法有效地减少了GPU的资源消耗，并且适用于具有重复性查询的场景，例如医疗保健和教育。

Title: LLM Generated Persona is a Promise with a Catch

• Authors: Ang Li, Haozhe Chen, Hongseok Namkoong, Tianyi Peng
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2503.16527
• Pdf URL: https://arxiv.org/pdf/2503.16527
• Copy Paste: [[2503.16527]] LLM Generated Persona is a Promise with a Catch(https://arxiv.org/abs/2503.16527)
• Keywords: language model, llm
• Abstract: The use of large language models (LLMs) to simulate human behavior has gained significant attention, particularly through personas that approximate individual characteristics. Persona-based simulations hold promise for transforming disciplines that rely on population-level feedback, including social science, economic analysis, marketing research, and business operations. Traditional methods to collect realistic persona data face significant challenges. They are prohibitively expensive and logistically challenging due to privacy constraints, and often fail to capture multi-dimensional attributes, particularly subjective qualities. Consequently, synthetic persona generation with LLMs offers a scalable, cost-effective alternative. However, current approaches rely on ad hoc and heuristic generation techniques that do not guarantee methodological rigor or simulation precision, resulting in systematic biases in downstream tasks. Through extensive large-scale experiments including presidential election forecasts and general opinion surveys of the U.S. population, we reveal that these biases can lead to significant deviations from real-world outcomes. Our findings underscore the need to develop a rigorous science of persona generation and outline the methodological innovations, organizational and institutional support, and empirical foundations required to enhance the reliability and scalability of LLM-driven persona simulations. To support further research and development in this area, we have open-sourced approximately one million generated personas, available for public access and analysis at this https URL.
• 摘要：使用大型语言模型（LLM）模拟人类行为已引起了很大的关注，尤其是通过近似个体特征的角色。基于角色的模拟有望改变依赖人口水平反馈的学科，包括社会科学，经济分析，市场研究和业务运营。收集现实角色数据的传统方法面临重大挑战。由于隐私的限制，它们非常昂贵且具有逻辑上的挑战，并且通常无法捕获多维属性，尤其是主观质量。因此，LLMS的合成角色生成提供了可扩展的，具有成本效益的替代方案。但是，当前的方法依赖于临时和启发式生成技术，这些技术不能保证方法论严格或模拟精度，从而导致下游任务的系统偏见。通过广泛的大规模实验，包括总统选举预测和对美国人口的一般意见调查，我们透露，这些偏见可能导致与现实成果的巨大偏差。我们的发现强调了开发严格的角色生成科学的必要性，并概述了方法论创新，组织和机构支持以及增强LLM驱动角色模拟的可靠性和可扩展性所需的经验基础。为了支持该领域的进一步研究和开发，我们开源了约100万个生成的角色，可在此HTTPS URL上进行公众访问和分析。

Title: HDLCoRe: A Training-Free Framework for Mitigating Hallucinations in LLM-Generated HDL

• Authors: Heng Ping, Shixuan Li, Peiyu Zhang, Anzhe Cheng, Shukai Duan, Nikos Kanakaris, Xiongye Xiao, Wei Yang, Shahin Nazarian, Andrei Irimia, Paul Bogdan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.16528
• Pdf URL: https://arxiv.org/pdf/2503.16528
• Copy Paste: [[2503.16528]] HDLCoRe: A Training-Free Framework for Mitigating Hallucinations in LLM-Generated HDL(https://arxiv.org/abs/2503.16528)
• Keywords: language model, llm, hallucination, prompt, retrieval-augmented generation, chain-of-thought
• Abstract: Recent advances in large language models (LLMs) have demonstrated remarkable capabilities in code generation tasks. However, when applied to hardware description languages (HDL), these models exhibit significant limitations due to data scarcity, resulting in hallucinations and incorrect code generation. To address these challenges, we propose HDLCoRe, a training-free framework that enhances LLMs' HDL generation capabilities through prompt engineering techniques and retrieval-augmented generation (RAG). Our approach consists of two main components: (1) an HDL-aware Chain-of-Thought (CoT) prompting technique with self-verification that classifies tasks by complexity and type, incorporates domain-specific knowledge, and guides LLMs through step-by-step self-simulation for error correction; and (2) a two-stage heterogeneous RAG system that addresses formatting inconsistencies through key component extraction and efficiently retrieves relevant HDL examples through sequential filtering and re-ranking. HDLCoRe eliminates the need for model fine-tuning while substantially improving LLMs' HDL generation capabilities. Experimental results demonstrate that our framework achieves superior performance on the RTLLM2.0 benchmark, significantly reducing hallucinations and improving both syntactic and functional correctness.
• 摘要：大型语言模型（LLM）的最新进展表明，在代码生成任务中具有显着的功能。但是，当应用于硬件说明语言（HDL）时，由于数据稀缺性，这些模型会出现重大限制，从而导致幻觉和代码生成不正确。为了应对这些挑战，我们提出了HDLCore，这是一个无培训的框架，通过迅速的工程技术和检索功能增强的生成（RAG）来增强LLMS的HDL生成能力。我们的方法由两个主要组成部分组成：（1）一种用自我验证提示技术提示技术的HDL感知链（COT），这些技术通过复杂性和类型对任务进行了分类，结合了域特异性知识，并通过逐步的逐步自我模拟来指导LLMS以进行错误纠正； （2）一个两阶段的异质抹布系统，通过关键组件提取来解决格式的不一致，并通过顺序滤波和重新排列有效地检索相关的HDL示例。 HDLCORE消除了对模型进行微调的需求，同时实质上提高了LLMS的HDL生成功能。实验结果表明，我们的框架在RTLLM2.0基准上实现了卓越的性能，大大降低了幻觉并提高了句法和功能正确性。

Title: Safety Evaluation and Enhancement of DeepSeek Models in Chinese Contexts

• Authors: Wenjing Zhang, Xuejiao Lei, Zhaoxiang Liu, Limin Han, Jiaojiao Zhao, Beibei Huang, Zhenhong Long, Junting Guo, Meijuan An, Rongjia Du, Ning Wang, Kai Wang, Shiguo Lian
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2503.16529
• Pdf URL: https://arxiv.org/pdf/2503.16529
• Copy Paste: [[2503.16529]] Safety Evaluation and Enhancement of DeepSeek Models in Chinese Contexts(https://arxiv.org/abs/2503.16529)
• Keywords: prompt
• Abstract: DeepSeek-R1, renowned for its exceptional reasoning capabilities and open-source strategy, is significantly influencing the global artificial intelligence landscape. However, it exhibits notable safety shortcomings. Recent research conducted by Robust Intelligence, a subsidiary of Cisco, in collaboration with the University of Pennsylvania, revealed that DeepSeek-R1 achieves a 100\% attack success rate when processing harmful prompts. Furthermore, multiple security firms and research institutions have identified critical security vulnerabilities within the model. Although China Unicom has uncovered safety vulnerabilities of R1 in Chinese contexts, the safety capabilities of the remaining distilled models in the R1 series have not yet been comprehensively evaluated. To address this gap, this study utilizes the comprehensive Chinese safety benchmark CHiSafetyBench to conduct an in-depth safety evaluation of the DeepSeek-R1 series distilled models. The objective is to assess the safety capabilities of these models in Chinese contexts both before and after distillation, and to further elucidate the adverse effects of distillation on model safety. Building on these findings, we implement targeted safety enhancements for six distilled models. Evaluation results indicate that the enhanced models achieve significant improvements in safety while maintaining reasoning capabilities without notable degradation. We open-source the safety-enhanced models at this https URL to serve as a valuable resource for future research and optimization of DeepSeek models.
• 摘要：DeepSeek-R1以其出色的推理能力和开源战略而闻名，它极大地影响了全球人工智能领域。但是，它表现出显着的安全缺点。 Cisco的子公司与宾夕法尼亚大学合作进行的Robust Intelligence进行的最新研究表明，DeepSeek-R1在处理有害提示时达到了100 \％的攻击成功率。此外，多个安全公司和研究机构已经确定了模型中的关键安全漏洞。尽管中国Unicom在中国环境中发现了R1的安全漏洞，但尚未对R1系列中其余蒸馏模型的安全能力进行全面评估。为了解决这一差距，这项研究利用了全面的中国安全基准Chisafetybench对DeepSeek-R1系列蒸馏模型进行了深入的安全评估。目的是在蒸馏之前和之后评估这些模型的安全能力，并进一步阐明蒸馏对模型安全的不利影响。在这些发现的基础上，我们为六种蒸馏型实施了针对性的安全性增强。评估结果表明，增强模型在没有明显降解的情况下保持推理能力的同时，可以取得显着改善。我们在此HTTPS URL上开放安全增强的模型，可作为未来研究和优化DeepSeek模型的宝贵资源。

Title: Enhancing LLM Generation with Knowledge Hypergraph for Evidence-Based Medicine

• Authors: Chengfeng Dou, Ying Zhang, Zhi Jin, Wenpin Jiao, Haiyan Zhao, Yongqiang Zhao, Zhengwei Tao
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2503.16530
• Pdf URL: https://arxiv.org/pdf/2503.16530
• Copy Paste: [[2503.16530]] Enhancing LLM Generation with Knowledge Hypergraph for Evidence-Based Medicine(https://arxiv.org/abs/2503.16530)
• Keywords: language model, llm, hallucination, retrieval-augmented generation
• Abstract: Evidence-based medicine (EBM) plays a crucial role in the application of large language models (LLMs) in healthcare, as it provides reliable support for medical decision-making processes. Although it benefits from current retrieval-augmented generation~(RAG) technologies, it still faces two significant challenges: the collection of dispersed evidence and the efficient organization of this evidence to support the complex queries necessary for EBM. To tackle these issues, we propose using LLMs to gather scattered evidence from multiple sources and present a knowledge hypergraph-based evidence management model to integrate these evidence while capturing intricate relationships. Furthermore, to better support complex queries, we have developed an Importance-Driven Evidence Prioritization (IDEP) algorithm that utilizes the LLM to generate multiple evidence features, each with an associated importance score, which are then used to rank the evidence and produce the final retrieval results. Experimental results from six datasets demonstrate that our approach outperforms existing RAG techniques in application domains of interest to EBM, such as medical quizzing, hallucination detection, and decision support. Testsets and the constructed knowledge graph can be accessed at \href{this https URL}{this https URL}.
• 摘要：循证医学（EBM）在医疗保健中大型语言模型（LLM）的应用中起着至关重要的作用，因为它为医疗决策过程提供了可靠的支持。尽管它受益于当前的检索型发电〜（RAG）技术，但它仍然面临两个重大挑战：分散证据的收集和有效的该证据的组织来支持EBM所需的复杂查询。为了解决这些问题，我们建议使用LLM从多个来源收集分散的证据，并提出基于知识的证据管理模型，以在捕获复杂的关系的同时整合这些证据。此外，为了更好地支持复杂的查询，我们开发了一种重要性驱动的证据优先级（IDEP）算法，该算法利用LLM来生成多个证据特征，每个算法都具有相关的重要性评分，然后将其用于对证据进行排名并产生最终检索结果。六个数据集的实验结果表明，我们的方法在EBM感兴趣的应用领域（例如医疗测验，幻觉检测和决策支持）优于现有的破布技术。可以在\ href {此https url} {this HTTPS url}处访问测试程序和构造的知识图。

Title: EEG-CLIP : Learning EEG representations from natural language descriptions

• Authors: Tidiane Camaret N'dir, Robin Tibor Schirrmeister
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.16531
• Pdf URL: https://arxiv.org/pdf/2503.16531
• Copy Paste: [[2503.16531]] EEG-CLIP : Learning EEG representations from natural language descriptions(https://arxiv.org/abs/2503.16531)
• Keywords: prompt
• Abstract: Deep networks for electroencephalogram (EEG) decoding are currently often trained to only solve a specific task like pathology or gender decoding. A more general approach leveraging the medical reports of clinical EEG recordings is to learn mappings between medical reports and EEG recordings. This approach was pioneered in the computer vision domain matching images and their text captions and subsequently allowed to do successful zero-shot decoding using textual class prompts. In this work, we follow this approach and develop a contrastive learning framework EEG-CLIP that aligns EEG time series and their corresponding clinical text descriptions in a shared embedding space. We investigate its potential for versatile EEG decoding, assessing performance on a range of few-shot and zero-shot settings. Overall, results show that EEG-CLIP manages to nontrivially align text and EEG representations. Our work presents a promising approach to learn general EEG representations, which could enable easier analyses of diverse decoding questions through zero shot decoding or training task-specific models from fewer training examples. The code for reproducing our results is available at this https URL.
• 摘要：目前，经常对脑电图（EEG）解码的深层网络进行培训，以解决诸如病理或性别解码之类的特定任务。利用临床脑电图记录的医学报告的一种更通用的方法是学习医疗报告和脑电图记录之间的映射。这种方法在计算机视觉域匹配的图像及其文本字幕中率先开创，随后允许使用文本类提示进行成功进行零射击解码。在这项工作中，我们遵循这种方法，并开发了一个对比度学习框架eeg-clip，该框架将eeg时间序列及其相应的临床文本描述保持在共享的嵌入空间中。我们研究了其多功能脑电图解码的潜力，评估了一系列少数和零弹性设置的性能。总体而言，结果表明，EEG-CLIP管理非对齐文本和EEG表示。我们的工作提出了一种有前途的方法来学习一般脑电图表示，这可以通过零射击解码或培训特定于任务的模型从更少的培训示例来更轻松地分析各种解码问题。重现我们的结果的代码可在此HTTPS URL上获得。

Title: From Patient Consultations to Graphs: Leveraging LLMs for Patient Journey Knowledge Graph Construction

• Authors: Hassan S. Al Khatib, Sudip Mittal, Shahram Rahimi, Nina Marhamati, Sean Bozorgzad
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.16533
• Pdf URL: https://arxiv.org/pdf/2503.16533
• Copy Paste: [[2503.16533]] From Patient Consultations to Graphs: Leveraging LLMs for Patient Journey Knowledge Graph Construction(https://arxiv.org/abs/2503.16533)
• Keywords: language model, gpt, llm, chat
• Abstract: The transition towards patient-centric healthcare necessitates a comprehensive understanding of patient journeys, which encompass all healthcare experiences and interactions across the care spectrum. Existing healthcare data systems are often fragmented and lack a holistic representation of patient trajectories, creating challenges for coordinated care and personalized interventions. Patient Journey Knowledge Graphs (PJKGs) represent a novel approach to addressing the challenge of fragmented healthcare data by integrating diverse patient information into a unified, structured representation. This paper presents a methodology for constructing PJKGs using Large Language Models (LLMs) to process and structure both formal clinical documentation and unstructured patient-provider conversations. These graphs encapsulate temporal and causal relationships among clinical encounters, diagnoses, treatments, and outcomes, enabling advanced temporal reasoning and personalized care insights. The research evaluates four different LLMs, such as Claude 3.5, Mistral, Llama 3.1, and Chatgpt4o, in their ability to generate accurate and computationally efficient knowledge graphs. Results demonstrate that while all models achieved perfect structural compliance, they exhibited variations in medical entity processing and computational efficiency. The paper concludes by identifying key challenges and future research directions. This work contributes to advancing patient-centric healthcare through the development of comprehensive, actionable knowledge graphs that support improved care coordination and outcome prediction.
• 摘要：向以患者为中心的医疗保健的过渡需要对患者旅行有全面的了解，这涵盖了整个护理领域的所有医疗体验和互动。现有的医疗保健数据系统通常是分散的，缺乏对患者轨迹的整体表示，从而为协调的护理和个性化干预措施构成了挑战。患者旅程知识图（PJKGS）代表了一种新的方法，可以通过将各种患者信息整合到统一的结构化表示形式中来解决零散的医疗保健数据的挑战。本文提出了一种使用大语言模型（LLM）来构建PJKG的方法，以处理和构建正式的临床文档和非结构化的患者支持者对话。这些图形封装了临床相遇，诊断，治疗和结果之间的时间和因果关系，从而实现了高级的时间推理和个性化的护理见解。该研究以产生准确和计算有效的知识图的能力评估了四种不同的LLM，例如Claude 3.5，Mistral，Llama 3.1和Chatgpt4o。结果表明，尽管所有模型均达到完美的结构合规性，但它们在医疗实体处理和计算效率方面表现出差异。本文通过确定关键挑战和未来的研究方向结束。这项工作有助于通过发展全面，可行的知识图来推进以患者为中心的医疗保健，从而支持改善护理协调和结果预测。

Title: Gender and content bias in Large Language Models: a case study on Google Gemini 2.0 Flash Experimental

• Authors: Roberto Balestri
• Subjects: cs.CL, cs.AI, cs.CY, cs.HC
• Abstract URL: https://arxiv.org/abs/2503.16534
• Pdf URL: https://arxiv.org/pdf/2503.16534
• Copy Paste: [[2503.16534]] Gender and content bias in Large Language Models: a case study on Google Gemini 2.0 Flash Experimental(https://arxiv.org/abs/2503.16534)
• Keywords: language model, gpt, llm, prompt, chat
• Abstract: This study evaluates the biases in Gemini 2.0 Flash Experimental, a state-of-the-art large language model (LLM) developed by Google, focusing on content moderation and gender disparities. By comparing its performance to ChatGPT-4o, examined in a previous work of the author, the analysis highlights some differences in ethical moderation practices. Gemini 2.0 demonstrates reduced gender bias, notably with female-specific prompts achieving a substantial rise in acceptance rates compared to results obtained by ChatGPT-4o. It adopts a more permissive stance toward sexual content and maintains relatively high acceptance rates for violent prompts, including gender-specific cases. Despite these changes, whether they constitute an improvement is debatable. While gender bias has been reduced, this reduction comes at the cost of permitting more violent content toward both males and females, potentially normalizing violence rather than mitigating harm. Male-specific prompts still generally receive higher acceptance rates than female-specific ones. These findings underscore the complexities of aligning AI systems with ethical standards, highlighting progress in reducing certain biases while raising concerns about the broader implications of the model's permissiveness. Ongoing refinements are essential to achieve moderation practices that ensure transparency, fairness, and inclusivity without amplifying harmful content.
• 摘要：这项研究评估了Gemini 2.0 Flash实验中的偏见，这是Google开发的最先进的大语言模型（LLM），重点是内容中的和性别差异。通过将其性能与作者先前的作品进行了研究，分析强调了道德节制实践的某些差异。 Gemini 2.0表现出性别偏见的减少，特别是与Chatgpt-4O获得的结果相比，女性特异性提示的接受率大幅上升。它对性内容采取了更宽松的立场，并在包括特定性别的案件在内的暴力提示中保持了相对较高的接受率。尽管有这些变化，但它们是否构成改进是有争议的。尽管性别偏见已经减少，但这种减少的代价是允许对男性和女性的更多暴力内容，可能使暴力正常而不是减轻伤害。男性特异性提示通常仍然比女性特定的提示更高。这些发现强调了使AI系统与道德标准保持一致的复杂性，从而强调了减少某些偏见的进步，同时提出了对模型允许性更广泛含义的担忧。持续的改进对于实现适度实践至关重要，以确保透明度，公平性和包容性而不会放大有害内容。

Title: Word2Minecraft: Generating 3D Game Levels through Large Language Models

• Authors: Shuo Huang, Muhammad Umair Nasir, Steven James, Julian Togelius
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.16536
• Pdf URL: https://arxiv.org/pdf/2503.16536
• Copy Paste: [[2503.16536]] Word2Minecraft: Generating 3D Game Levels through Large Language Models(https://arxiv.org/abs/2503.16536)
• Keywords: language model, gpt
• Abstract: We present Word2Minecraft, a system that leverages large language models to generate playable game levels in Minecraft based on structured stories. The system transforms narrative elements-such as protagonist goals, antagonist challenges, and environmental settings-into game levels with both spatial and gameplay constraints. We introduce a flexible framework that allows for the customization of story complexity, enabling dynamic level generation. The system employs a scaling algorithm to maintain spatial consistency while adapting key game elements. We evaluate Word2Minecraft using both metric-based and human-based methods. Our results show that GPT-4-Turbo outperforms GPT-4o-Mini in most areas, including story coherence and objective enjoyment, while the latter excels in aesthetic appeal. We also demonstrate the system' s ability to generate levels with high map enjoyment, offering a promising step forward in the intersection of story generation and game design. We open-source the code at this https URL
• 摘要：我们提出Word2minecraft，该系统利用大型语言模型根据结构化的故事在Minecraft中生成可玩的游戏水平。该系统会改变叙事元素，例如主角目标，对手挑战和环境环境，以及具有空间和游戏限制的into游戏水平。我们引入了一个灵活的框架，该框架允许自定义故事复杂性，从而使动态级别的生成。该系统采用缩放算法来保持空间一致性，同时调整关键游戏元素。我们使用基于度量的和基于人类的方法评估Word2minecraft。我们的结果表明，GPT-4-Turbo在大多数领域的表现都优于GPT-4O-Mini，包括故事连贯性和客观享受，而后者在美学吸引力方面表现出色。我们还展示了该系统具有高地图享受的产生水平的能力，这在故事和游戏设计的交集中为前进提供了有希望的一步。我们在此HTTPS URL上开放代码

Title: Do Multimodal Large Language Models Understand Welding?

• Authors: Grigorii Khvatskii, Yong Suk Lee, Corey Angst, Maria Gibbs, Robert Landers, Nitesh V. Chawla
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2503.16537
• Pdf URL: https://arxiv.org/pdf/2503.16537
• Copy Paste: [[2503.16537]] Do Multimodal Large Language Models Understand Welding?(https://arxiv.org/abs/2503.16537)
• Keywords: language model, llm, hallucination, prompt, chain-of-thought
• Abstract: This paper examines the performance of Multimodal LLMs (MLLMs) in skilled production work, with a focus on welding. Using a novel data set of real-world and online weld images, annotated by a domain expert, we evaluate the performance of two state-of-the-art MLLMs in assessing weld acceptability across three contexts: RV \& Marine, Aeronautical, and Farming. While both models perform better on online images, likely due to prior exposure or memorization, they also perform relatively well on unseen, real-world weld images. Additionally, we introduce WeldPrompt, a prompting strategy that combines Chain-of-Thought generation with in-context learning to mitigate hallucinations and improve reasoning. WeldPrompt improves model recall in certain contexts but exhibits inconsistent performance across others. These results underscore the limitations and potentials of MLLMs in high-stakes technical domains and highlight the importance of fine-tuning, domain-specific data, and more sophisticated prompting strategies to improve model reliability. The study opens avenues for further research into multimodal learning in industry applications.
• 摘要：本文研究了熟练的生产工作中多模式LLM（MLLM）的性能，重点是焊接。使用域专家注释的新型数据集和在线焊接图像，我们评估了两个最先进的MLLM在评估三种情况下焊接可接受性时的性能：RV \＆Marine，Aeronautical和Farming。尽管这两种模型在在线图像上的表现都更好，这可能是由于事先曝光或记忆所致，但它们在看不见的现实焊接图像上的表现也相对出色。此外，我们引入了WeldPrompt，这是一种促使人们的策略，将经过思想链的一代与内在的学习结合在一起，以减轻幻觉并改善推理。 WeldPrompt改善了在某些情况下的模型回忆，但在其他情况下表现出不一致的性能。这些结果强调了MLLM在高风险技术领域中的局限性和潜力，并强调了微调，特定于领域的数据以及更复杂的提示策略以提高模型可靠性的重要性。这项研究为行业应用中的多模式学习提供了进一步研究的途径。

Title: Poly-FEVER: A Multilingual Fact Verification Benchmark for Hallucination Detection in Large Language Models

• Authors: Hanzhi Zhang, Sumera Anjum, Heng Fan, Weijian Zheng, Yan Huang, Yunhe Feng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.16541
• Pdf URL: https://arxiv.org/pdf/2503.16541
• Copy Paste: [[2503.16541]] Poly-FEVER: A Multilingual Fact Verification Benchmark for Hallucination Detection in Large Language Models(https://arxiv.org/abs/2503.16541)
• Keywords: language model, gpt, llm, hallucination, chat
• Abstract: Hallucinations in generative AI, particularly in Large Language Models (LLMs), pose a significant challenge to the reliability of multilingual applications. Existing benchmarks for hallucination detection focus primarily on English and a few widely spoken languages, lacking the breadth to assess inconsistencies in model performance across diverse linguistic contexts. To address this gap, we introduce Poly-FEVER, a large-scale multilingual fact verification benchmark specifically designed for evaluating hallucination detection in LLMs. Poly-FEVER comprises 77,973 labeled factual claims spanning 11 languages, sourced from FEVER, Climate-FEVER, and SciFact. It provides the first large-scale dataset tailored for analyzing hallucination patterns across languages, enabling systematic evaluation of LLMs such as ChatGPT and the LLaMA series. Our analysis reveals how topic distribution and web resource availability influence hallucination frequency, uncovering language-specific biases that impact model accuracy. By offering a multilingual benchmark for fact verification, Poly-FEVER facilitates cross-linguistic comparisons of hallucination detection and contributes to the development of more reliable, language-inclusive AI systems. The dataset is publicly available to advance research in responsible AI, fact-checking methodologies, and multilingual NLP, promoting greater transparency and robustness in LLM performance. The proposed Poly-FEVER is available at: this https URL.
• 摘要：生成AI的幻觉，尤其是大型语言模型（LLM），对多语言应用的可靠性构成了重大挑战。幻觉检测的现有基准主要集中在英语和一些通俗的语言上，缺乏评估各种语言环境中模型性能的不一致的广度。为了解决这一差距，我们介绍了Poly-Fever，这是一种大型多语言事实验证基准，专门设计用于评估LLMS中的幻觉检测。 Poly-fever包括77,973个标记的事实主张，涵盖了11种语言，来自发烧，气候狂热和科幻。它提供了第一个量身定制的大规模数据集，用于分析跨语言的幻觉模式，从而可以对LLMS（例如Chatgpt和Llama系列）进行系统评估。我们的分析揭示了主题分布和Web资源的可用性如何影响幻觉频率，发现影响模型准确性的语言特定偏见。通过提供多种语言基准以进行事实验证，多武力促进了幻觉检测的跨语言比较，并有助于开发更可靠的，包括语言的AI系统。该数据集可公开使用，以推进负责任的AI，事实检查方法和多语言NLP的研究，从而促进LLM性能的更高透明度和鲁棒性。提出的poly-fever可在以下位置可用：此HTTPS URL。

Title: Unified Enhancement of the Generalization and Robustness of Language Models via Bi-Stage Optimization

• Authors: Yudao Sun, Juan Yin, Juan Zhao, Fan Zhang, Yongheng Liu, Hongji Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.16550
• Pdf URL: https://arxiv.org/pdf/2503.16550
• Copy Paste: [[2503.16550]] Unified Enhancement of the Generalization and Robustness of Language Models via Bi-Stage Optimization(https://arxiv.org/abs/2503.16550)
• Keywords: language model
• Abstract: Neural network language models (LMs) are confronted with significant challenges in generalization and robustness. Currently, many studies focus on improving either generalization or robustness in isolation, without methods addressing both aspects simultaneously, which presents a significant challenge in developing LMs that are both robust and generalized. In this paper, we propose a bi-stage optimization framework to uniformly enhance both the generalization and robustness of LMs, termed UEGR. Specifically, during the forward propagation stage, we enrich the output probability distributions of adversarial samples by adaptive dropout to generate diverse sub models, and incorporate JS divergence and adversarial losses of these output distributions to reinforce output stability. During backward propagation stage, we compute parameter saliency scores and selectively update only the most critical parameters to minimize unnecessary deviations and consolidate the model's resilience. Theoretical analysis shows that our framework includes gradient regularization to limit the model's sensitivity to input perturbations and selective parameter updates to flatten the loss landscape, thus improving both generalization and robustness. The experimental results show that our method significantly improves the generalization and robustness of LMs compared to other existing methods across 13 publicly available language datasets, achieving state-of-the-art (SOTA) performance.
• 摘要：神经网络语言模型（LMS）面临着概括和鲁棒性的重大挑战。当前，许多研究着重于孤立地改善概括或鲁棒性，而没有同时解决这两个方面的方法，这在开发既有稳定又广泛化的LMS方面提出了重大挑战。在本文中，我们提出了一个双阶段优化框架，以统一增强称为UEGR的LMS的概括和鲁棒性。具体而言，在正向传播阶段，我们通过自适应辍学来丰富对抗样本的输出概率分布，以生成各种子模型，并结合了这些输出分布的JS差异和对抗性损失，以增强输出稳定性。在向后传播阶段，我们计算参数显着性得分，并选择性地更新最关键的参数，以最大程度地减少不必要的偏差并巩固模型的弹性。理论分析表明，我们的框架包括梯度正则化，以限制模型对输入扰动的敏感性和选择性参数更新以使损失格局变平，从而改善了概括和稳健性。实验结果表明，与13个公开可用语言数据集中的其他现有方法相比，我们的方法显着提高了LMS的概括和鲁棒性，从而实现了最新的（SOTA）性能。

Title: A Foundational individual Mobility Prediction Model based on Open-Source Large Language Models

• Authors: Zhenlin Qin, Leizhen Wang, Francisco Camara Pereira, Zhenlinag Ma
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.16553
• Pdf URL: https://arxiv.org/pdf/2503.16553
• Copy Paste: [[2503.16553]] A Foundational individual Mobility Prediction Model based on Open-Source Large Language Models(https://arxiv.org/abs/2503.16553)
• Keywords: language model, llm, prompt
• Abstract: Large Language Models (LLMs) are widely applied to domain-specific tasks due to their massive general knowledge and remarkable inference capacities. Current studies on LLMs have shown immense potential in applying LLMs to model individual mobility prediction problems. However, most LLM-based mobility prediction models only train on specific datasets or use single well-designed prompts, leading to difficulty in adapting to different cities and users with diverse contexts. To fill these gaps, this paper proposes a unified fine-tuning framework to train a foundational open source LLM-based mobility prediction model. We conducted extensive experiments on six real-world mobility datasets to validate the proposed model. The results showed that the proposed model achieved the best performance in prediction accuracy and transferability over state-of-the-art models based on deep learning and LLMs.
• 摘要：大型语言模型（LLM）由于其丰富的常识和出色的推理能力而广泛应用于特定领域的任务。当前对LLM的研究表明，将LLMS应用于模拟个人流动性预测问题的巨大潜力。但是，大多数基于LLM的移动性预测模型仅在特定数据集上进行训练或使用单个精心设计的提示，从而导致难以适应具有不同环境的不同城市和用户。为了填补这些空白，本文提出了一个统一的微调框架，以培训基于基础的开源LLM的移动性预测模型。我们对六个现实世界的移动性数据集进行了广泛的实验，以验证所提出的模型。结果表明，所提出的模型在基于深度学习和LLM的最先进模型中实现了预测准确性和可传递性的最佳性能。

Title: FutureGen: LLM-RAG Approach to Generate the Future Work of Scientific Article

• Authors: Ibrahim Al Azher, Miftahul Jannat Mokarrama, Zhishuai Guo, Sagnik Ray Choudhury, Hamed Alhoori
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.16561
• Pdf URL: https://arxiv.org/pdf/2503.16561
• Copy Paste: [[2503.16561]] FutureGen: LLM-RAG Approach to Generate the Future Work of Scientific Article(https://arxiv.org/abs/2503.16561)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: The future work section of a scientific article outlines potential research directions by identifying gaps and limitations of a current study. This section serves as a valuable resource for early-career researchers seeking unexplored areas and experienced researchers looking for new projects or collaborations. In this study, we generate future work suggestions from key sections of a scientific article alongside related papers and analyze how the trends have evolved. We experimented with various Large Language Models (LLMs) and integrated Retrieval-Augmented Generation (RAG) to enhance the generation process. We incorporate a LLM feedback mechanism to improve the quality of the generated content and propose an LLM-as-a-judge approach for evaluation. Our results demonstrated that the RAG-based approach with LLM feedback outperforms other methods evaluated through qualitative and quantitative metrics. Moreover, we conduct a human evaluation to assess the LLM as an extractor and judge. The code and dataset for this project are here, code: HuggingFace
• 摘要：科学文章的未来工作部分通过确定当前研究的差距和局限性概述了潜在的研究方向。本节是寻求未开发领域的早期研究人员以及寻找新项目或合作的经验丰富的研究人员的宝贵资源。在这项研究中，我们从一篇科学文章的关键部分以及相关论文中产生了未来的工作建议，并分析了趋势的发展方式。我们尝试了各种大型语言模型（LLMS）和集成检索功能的生成（RAG），以增强生成过程。我们结合了LLM反馈机制，以提高生成的内容的质量，并提出一种法学委员会的评估方法。我们的结果表明，具有LLM反馈的基于抹布的方法优于通过定性和定量指标评估的其他方法。此外，我们进行了人类评估，以评估LLM作为提取者和法官。该项目的代码和数据集在这里，代码：huggingface

Title: Extract, Match, and Score: An Evaluation Paradigm for Long Question-context-answer Triplets in Financial Analysis

• Authors: Bo Hu, Han Yuan, Vlad Pandelea, Wuqiong Luo, Yingzhu Zhao, Zheng Ma
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.16575
• Pdf URL: https://arxiv.org/pdf/2503.16575
• Copy Paste: [[2503.16575]] Extract, Match, and Score: An Evaluation Paradigm for Long Question-context-answer Triplets in Financial Analysis(https://arxiv.org/abs/2503.16575)
• Keywords: language model, llm
• Abstract: The rapid advancement of large language models (LLMs) has sparked widespread adoption across diverse applications, making robust evaluation frameworks crucial for assessing their performance. While conventional evaluation metrics remain applicable for shorter texts, their efficacy diminishes when evaluating the quality of long-form answers. This limitation is particularly critical in real-world scenarios involving extended questions, extensive context, and long-form answers, such as financial analysis or regulatory compliance. In this paper, we use a practical financial use case to illustrate applications that handle "long question-context-answer triplets". We construct a real-world financial dataset comprising long triplets and demonstrate the inadequacies of traditional metrics. To address this, we propose an effective Extract, Match, and Score (EMS) evaluation approach tailored to the complexities of long-form LLMs' outputs, providing practitioners with a reliable methodology for assessing LLMs' performance in complex real-world scenarios.
• 摘要：大型语言模型（LLM）的快速发展引发了各种应用程序的广泛采用，这使得对评估其性能至关重要。尽管常规评估指标仍然适用于较短的文本，但在评估长形答案质量时，其功效会降低。在涉及涉及扩展问题，广泛背景和长期答案（例如财务分析或法规合规性）的现实情况下，这种限制尤其重要。在本文中，我们使用一种实用的财务用例来说明处理“长期问题 - 封闭式 - 答案三胞胎”的应用程序。我们构建了一个现实世界中的金融数据集，其中包括长三联，并证明了传统指标的不足。为了解决这个问题，我们提出了一种有效的摘录，匹配和分数（EMS）评估方法，该方法是根据长期LLMS输出的复杂性量身定制的，为从业人员提供了一种可靠的方法，用于评估LLMS在复杂的现实世界情景中的性能。

Title: Investigating Retrieval-Augmented Generation in Quranic Studies: A Study of 13 Open-Source Large Language Models

• Authors: Zahra Khalila, Arbi Haza Nasution, Winda Monika, Aytug Onan, Yohei Murakami, Yasir Bin Ismail Radi, Noor Mohammad Osmani
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.16581
• Pdf URL: https://arxiv.org/pdf/2503.16581
• Copy Paste: [[2503.16581]] Investigating Retrieval-Augmented Generation in Quranic Studies: A Study of 13 Open-Source Large Language Models(https://arxiv.org/abs/2503.16581)
• Keywords: language model, llm, hallucination, retrieval-augmented generation
• Abstract: Accurate and contextually faithful responses are critical when applying large language models (LLMs) to sensitive and domain-specific tasks, such as answering queries related to quranic studies. General-purpose LLMs often struggle with hallucinations, where generated responses deviate from authoritative sources, raising concerns about their reliability in religious contexts. This challenge highlights the need for systems that can integrate domain-specific knowledge while maintaining response accuracy, relevance, and faithfulness. In this study, we investigate 13 open-source LLMs categorized into large (e.g., Llama3:70b, Gemma2:27b, QwQ:32b), medium (e.g., Gemma2:9b, Llama3:8b), and small (e.g., Llama3.2:3b, Phi3:3.8b). A Retrieval-Augmented Generation (RAG) is used to make up for the problems that come with using separate models. This research utilizes a descriptive dataset of Quranic surahs including the meanings, historical context, and qualities of the 114 surahs, allowing the model to gather relevant knowledge before responding. The models are evaluated using three key metrics set by human evaluators: context relevance, answer faithfulness, and answer relevance. The findings reveal that large models consistently outperform smaller models in capturing query semantics and producing accurate, contextually grounded responses. The Llama3.2:3b model, even though it is considered small, does very well on faithfulness (4.619) and relevance (4.857), showing the promise of smaller architectures that have been well optimized. This article examines the trade-offs between model size, computational efficiency, and response quality while using LLMs in domain-specific applications.
• 摘要：在将大语言模型（LLM）应用于敏感和特定领域的任务时，例如回答与古兰经研究有关的查询时，准确和上下文的忠实响应至关重要。通用LLM经常在幻觉中挣扎，在这种情况下，产生的反应偏离了权威来源，引起了人们对它们在宗教背景下的可靠性的担忧。这项挑战强调了对可以集成特定领域知识的系统的需求，同时保持响应准确性，相关性和忠诚。在这项研究中，我们研究了13个开源LLMS，分为大型（例如Llama3：70b，Gemma2：27b，QWQ：32B），介质（例如Gemma2：9b，Llama3：8b）和Small（例如，Llama3.2：3b，phi3：3.8b）。检索增强的一代（RAG）用于弥补使用单独模型所带来的问题。这项研究利用了古兰经古兰经的描述性数据集，包括114个古兰经的含义，历史背景和素质，使模型可以在响应之前收集相关知识。使用人类评估者设定的三个关键指标评估模型：上下文相关性，回答忠诚和回答相关性。研究结果表明，大型模型在捕获查询语义和产生准确的上下文响应时始终超过较小的模型。 Llama3.2：3b模型尽管被认为很小，但在忠诚（4.619）和相关性（4.857）上表现出了很好的表现，显示了已得到很好优化的较小体系结构的承诺。本文研究了模型大小，计算效率和响应质量之间的权衡，同时在特定于域的应用中使用LLMS。

Title: Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directions

• Authors: Hadi Amini, Md Jueal Mia, Yasaman Saadati, Ahmed Imteaj, Seyedsina Nabavirazavi, Urmish Thakker, Md Zarif Hossain, Awal Ahmed Fime, S.S. Iyengar
• Subjects: cs.CL, cs.CV, cs.DC, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.16585
• Pdf URL: https://arxiv.org/pdf/2503.16585
• Copy Paste: [[2503.16585]] Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directions(https://arxiv.org/abs/2503.16585)
• Keywords: language model, llm
• Abstract: Language models (LMs) are machine learning models designed to predict linguistic patterns by estimating the probability of word sequences based on large-scale datasets, such as text. LMs have a wide range of applications in natural language processing (NLP) tasks, including autocomplete and machine translation. Although larger datasets typically enhance LM performance, scalability remains a challenge due to constraints in computational power and resources. Distributed computing strategies offer essential solutions for improving scalability and managing the growing computational demand. Further, the use of sensitive datasets in training and deployment raises significant privacy concerns. Recent research has focused on developing decentralized techniques to enable distributed training and inference while utilizing diverse computational resources and enabling edge AI. This paper presents a survey on distributed solutions for various LMs, including large language models (LLMs), vision language models (VLMs), multimodal LLMs (MLLMs), and small language models (SLMs). While LLMs focus on processing and generating text, MLLMs are designed to handle multiple modalities of data (e.g., text, images, and audio) and to integrate them for broader applications. To this end, this paper reviews key advancements across the MLLM pipeline, including distributed training, inference, fine-tuning, and deployment, while also identifying the contributions, limitations, and future areas of improvement. Further, it categorizes the literature based on six primary focus areas of decentralization. Our analysis describes gaps in current methodologies for enabling distributed solutions for LMs and outline future research directions, emphasizing the need for novel solutions to enhance the robustness and applicability of distributed LMs.
• 摘要：语言模型（LMS）是机器学习模型，旨在通过估计基于大规模数据集（例如文本）的单词序列的概率来预测语言模式。 LMS在自然语言处理（NLP）任务中具有广泛的应用，包括自动完成和机器翻译。尽管较大的数据集通常会增强LM性能，但由于计算能力和资源的限制，可伸缩性仍然是一个挑战。分布式计算策略提供了改善可伸缩性和管理不断增长的计算需求的基本解决方案。此外，在培训和部署中使用敏感数据集引发了严重的隐私问题。最近的研究重点是开发分散的技术，以实现分布式培训和推理，同时利用各种计算资源并实现Edge AI。本文介绍了针对各种LM的分布式解决方案的调查，包括大语言模型（LLM），视觉语言模型（VLM），多模式LLMS（MLLMS）和小语言模型（SLMS）。尽管LLMS专注于处理和生成文本，但MLLM旨在处理多种数据模式（例如文本，图像和音频），并将其集成为更广泛的应用程序。为此，本文回顾了整个MLLM管道的主要进步，包括分布式培训，推理，微调和部署，同时还确定了改进的贡献，局限性和未来的领域。此外，它根据分散化的六个主要重点领域对文献进行了分类。我们的分析描述了当前方法中的差距，以启用LMS分布式解决方案并概述未来的研究方向，从而强调了对新颖解决方案的需求，以增强分布式LMS的鲁棒性和适用性。

Title: Leveraging Large Language Models for Explainable Activity Recognition in Smart Homes: A Critical Evaluation

• Authors: Michele Fiori, Gabriele Civitarese, Priyankar Choudhary, Claudio Bettini
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.16622
• Pdf URL: https://arxiv.org/pdf/2503.16622
• Copy Paste: [[2503.16622]] Leveraging Large Language Models for Explainable Activity Recognition in Smart Homes: A Critical Evaluation(https://arxiv.org/abs/2503.16622)
• Keywords: language model, llm
• Abstract: Explainable Artificial Intelligence (XAI) aims to uncover the inner reasoning of machine learning models. In IoT systems, XAI improves the transparency of models processing sensor data from multiple heterogeneous devices, ensuring end-users understand and trust their outputs. Among the many applications, XAI has also been applied to sensor-based Activities of Daily Living (ADLs) recognition in smart homes. Existing approaches highlight which sensor events are most important for each predicted activity, using simple rules to convert these events into natural language explanations for non-expert users. However, these methods produce rigid explanations lacking natural language flexibility and are not scalable. With the recent rise of Large Language Models (LLMs), it is worth exploring whether they can enhance explanation generation, considering their proven knowledge of human activities. This paper investigates potential approaches to combine XAI and LLMs for sensor-based ADL recognition. We evaluate if LLMs can be used: a) as explainable zero-shot ADL recognition models, avoiding costly labeled data collection, and b) to automate the generation of explanations for existing data-driven XAI approaches when training data is available and the goal is higher recognition rates. Our critical evaluation provides insights into the benefits and challenges of using LLMs for explainable ADL recognition.
• 摘要：可解释的人工智能（XAI）旨在揭示机器学习模型的内部推理。在物联网系统中，XAI提高了从多个异质设备处理传感器数据的模型的透明度，从而确保最终用户了解和信任其输出。在许多应用程序中，XAI还应用于智能家居中基于传感器的日常生活（ADL）识别。现有方法强调了哪些传感器事件对于每个预测活动最重要，使用简单的规则将这些事件转换为非专家用户的自然语言解释。但是，这些方法产生了缺乏自然语言灵活性的严格解释，并且不可伸缩。随着大型语言模型（LLM）的最新兴起，值得探索他们是否可以增强解释产生，考虑到他们对人类活动的知识知识。本文研究了将XAI和LLMS结合起来以基于传感器的ADL识别的潜在方法。我们评估是否可以使用LLM：a）作为可解释的零摄像机ADL识别模型，避免了昂贵的标记数据收集，b）在可用培训数据时自动为现有数据驱动的XAI方法的解释生成，并且目标是较高的识别率。我们的批判性评估提供了对使用LLM进行可解释ADL识别的好处和挑战的见解。

Title: Accelerating Antibiotic Discovery with Large Language Models and Knowledge Graphs

• Authors: Maxime Delmas, Magdalena Wysocka, Danilo Gusicuma, André Freitas
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.16655
• Pdf URL: https://arxiv.org/pdf/2503.16655
• Copy Paste: [[2503.16655]] Accelerating Antibiotic Discovery with Large Language Models and Knowledge Graphs(https://arxiv.org/abs/2503.16655)
• Keywords: language model, llm
• Abstract: The discovery of novel antibiotics is critical to address the growing antimicrobial resistance (AMR). However, pharmaceutical industries face high costs (over $1 billion), long timelines, and a high failure rate, worsened by the rediscovery of known compounds. We propose an LLM-based pipeline that acts as an alarm system, detecting prior evidence of antibiotic activity to prevent costly rediscoveries. The system integrates organism and chemical literature into a Knowledge Graph (KG), ensuring taxonomic resolution, synonym handling, and multi-level evidence classification. We tested the pipeline on a private list of 73 potential antibiotic-producing organisms, disclosing 12 negative hits for evaluation. The results highlight the effectiveness of the pipeline for evidence reviewing, reducing false negatives, and accelerating decision-making. The KG for negative hits and the user interface for interactive exploration will be made publicly available.
• 摘要：新型抗生素的发现对于解决增长的抗菌耐药性（AMR）至关重要。但是，制药行业面临高成本（超过10亿美元），长时间的时间表和高失败率，这会因已知化合物的重新发现而恶化。我们提出了一条基于LLM的管道，该管道充当警报系统，检测先前的抗生素活性证据以防止昂贵的重新发现。该系统将有机体和化学文献整合到知识图（kg）中，确保分类学解决方案，同义词处理和多层证据分类。我们在73个潜在的产生抗生素生产生物的私人列表上测试了管道，披露了12个负面命中率进行评估。结果突出了管道在证据审查，减少虚假负面因素和加速决策方面的有效性。将公开可用的负面热门单元和用于交互式探索的用户界面。

Title: Through the LLM Looking Glass: A Socratic Self-Assessment of Donkeys, Elephants, and Markets

• Authors: Molly Kennedy, Ayyoob Imani, Timo Spinde, Hinrich Schütze
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.16674
• Pdf URL: https://arxiv.org/pdf/2503.16674
• Copy Paste: [[2503.16674]] Through the LLM Looking Glass: A Socratic Self-Assessment of Donkeys, Elephants, and Markets(https://arxiv.org/abs/2503.16674)
• Keywords: llm, prompt
• Abstract: While detecting and avoiding bias in LLM-generated text is becoming increasingly important, media bias often remains subtle and subjective, making it particularly difficult to identify and mitigate. In this study, we assess media bias in LLM-generated content and LLMs' ability to detect subtle ideological bias. We conduct this evaluation using two datasets, PoliGen and EconoLex, covering political and economic discourse, respectively. We evaluate eight widely used LLMs by prompting them to generate articles and analyze their ideological preferences via self-assessment. By using self-assessment, the study aims to directly measure the models' biases rather than relying on external interpretations, thereby minimizing subjective judgments about media bias. Our results reveal a consistent preference of Democratic over Republican positions across all models. Conversely, in economic topics, biases vary among Western LLMs, while those developed in China lean more strongly toward socialism.
• 摘要：虽然发现和避免在LLM生成的文本中偏见变得越来越重要，但媒体偏见通常仍然是微妙和主观的，这使得尤其难以识别和减轻。在这项研究中，我们评估了LLM生成的含量和LLM检测微妙意识形态偏见的能力的媒体偏见。我们使用两个数据集Poligen和Econolex进行了这项评估，分别涵盖了政治和经济话语。我们通过促使它们生成文章并通过自我评估来分析其意识形态偏好来评估八个广泛使用的LLM。通过使用自我评估，该研究的目的是直接衡量模型的偏见而不是依靠外部解释，从而最大程度地减少了对媒体偏见的主观判断。我们的结果表明，在所有模型中，民主党在共和党的立场上都持续偏爱。相反，在经济主题中，偏见在西部LLM中有所不同，而在中国发展的偏见更强烈地倾向于社会主义。

Title: Natural Language Generation

• Authors: Emiel van Miltenburg, Chenghua Lin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.16728
• Pdf URL: https://arxiv.org/pdf/2503.16728
• Copy Paste: [[2503.16728]] Natural Language Generation(https://arxiv.org/abs/2503.16728)
• Keywords: language model, llm
• Abstract: This article provides a brief overview of the field of Natural Language Generation. The term Natural Language Generation (NLG), in its broadest definition, refers to the study of systems that verbalize some form of information through natural language. That information could be stored in a large database or knowledge graph (in data-to-text applications), but NLG researchers may also study summarisation (text-to-text) or image captioning (image-to-text), for example. As a subfield of Natural Language Processing, NLG is closely related to other sub-disciplines such as Machine Translation (MT) and Dialog Systems. Some NLG researchers exclude MT from their definition of the field, since there is no content selection involved where the system has to determine what to say. Conversely, dialog systems do not typically fall under the header of Natural Language Generation since NLG is just one component of dialog systems (the others being Natural Language Understanding and Dialog Management). However, with the rise of Large Language Models (LLMs), different subfields of Natural Language Processing have converged on similar methodologies for the production of natural language and the evaluation of automatically generated text.
• 摘要：本文简要概述了自然语言生成领域。自然语言产生（NLG）一词在其最广泛的定义中是指通过自然语言对某种形式信息进行口头表达的系统的研究。该信息可以存储在大型数据库或知识图中（在数据到文本应用程序中），但是NLG研究人员也可以研究摘要（文本到文本）或图像字幕（图像到文本）。作为自然语言处理的子场，NLG与其他子学科（例如机器翻译（MT）和对话系统系统）密切相关。一些NLG研究人员将MT排除在对领域的定义之外，因为在系统必须确定该说些什么的地方没有内容选择。相反，对话框系统通常不属于自然语言生成的标题，因为NLG只是对话系统的一个组成部分（其他是自然语言理解和对话管理）。但是，随着大语言模型（LLM）的兴起，自然语言处理的不同子场已融合了类似的方法，用于生产自然语言，并评估自动生成的文本。

Title: Chain-of-Tools: Utilizing Massive Unseen Tools in the CoT Reasoning of Frozen Language Models

• Authors: Mengsong Wu, Tong Zhu, Han Han, Xiang Zhang, Wenbiao Shao, Wenliang Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.16779
• Pdf URL: https://arxiv.org/pdf/2503.16779
• Copy Paste: [[2503.16779]] Chain-of-Tools: Utilizing Massive Unseen Tools in the CoT Reasoning of Frozen Language Models(https://arxiv.org/abs/2503.16779)
• Keywords: language model, llm, prompt
• Abstract: Tool learning can further broaden the usage scenarios of large language models (LLMs). However most of the existing methods either need to finetune that the model can only use tools seen in the training data, or add tool demonstrations into the prompt with lower efficiency. In this paper, we present a new Tool Learning method Chain-of-Tools. It makes full use of the powerful semantic representation capability of frozen LLMs to finish tool calling in CoT reasoning with a huge and flexible tool pool which may contain unseen tools. Especially, to validate the effectiveness of our approach in the massive unseen tool scenario, we construct a new dataset SimpleToolQuestions. We conduct experiments on two numerical reasoning benchmarks (GSM8K-XL and FuncQA) and two knowledge-based question answering benchmarks (KAMEL and SimpleToolQuestions). Experimental results show that our approach performs better than the baseline. We also identify dimensions of the model output that are critical in tool selection, enhancing the model interpretability. Our code and data are available at: this https URL .
• 摘要：工具学习可以进一步扩大大语模型（LLM）的使用情况。但是，大多数现有方法要么需要对模型只能使用训练数据中看到的工具进行填补，要么以较低的效率将工具演示添加到提示中。在本文中，我们提出了一种新的工具学习方法。它充分利用了冷冻LLM的功能强大的语义表示能力，可以用巨大而灵活的工具池在COT推理中完成工具，该工具可能包含看不见的工具。尤其是，为了验证我们在大量看不见的工具方案中我们的方法的有效性，我们构建了一个新的数据集SimpleToolQuestions。我们对两个数值推理基准（GSM8K-XL和FuncQA）以及两个基于知识的问题回答基准（Kamel和SimpleToolQuestions）进行实验。实验结果表明，我们的方法的表现要好于基线。我们还确定了模型输出的维度，这些尺寸在工具选择中至关重要，从而增强了模型的解释性。我们的代码和数据可在以下网址提供：此HTTPS URL。

Title: Conversational User-AI Intervention: A Study on Prompt Rewriting for Improved LLM Response Generation

• Authors: Rupak Sarkar, Bahareh Sarrafzadeh, Nirupama Chandrasekaran, Nagu Rangan, Philip Resnik, Longqi Yang, Sujay Kumar Jauhar
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.16789
• Pdf URL: https://arxiv.org/pdf/2503.16789
• Copy Paste: [[2503.16789]] Conversational User-AI Intervention: A Study on Prompt Rewriting for Improved LLM Response Generation(https://arxiv.org/abs/2503.16789)
• Keywords: llm, prompt, chat
• Abstract: Human-LLM conversations are increasingly becoming more pervasive in peoples' professional and personal lives, yet many users still struggle to elicit helpful responses from LLM Chatbots. One of the reasons for this issue is users' lack of understanding in crafting effective prompts that accurately convey their information needs. Meanwhile, the existence of real-world conversational datasets on the one hand, and the text understanding faculties of LLMs on the other, present a unique opportunity to study this problem, and its potential solutions at scale. Thus, in this paper we present the first LLM-centric study of real human-AI chatbot conversations, focused on investigating aspects in which user queries fall short of expressing information needs, and the potential of using LLMs to rewrite suboptimal user prompts. Our findings demonstrate that rephrasing ineffective prompts can elicit better responses from a conversational system, while preserving the user's original intent. Notably, the performance of rewrites improves in longer conversations, where contextual inferences about user needs can be made more accurately. Additionally, we observe that LLMs often need to -- and inherently do -- make \emph{plausible} assumptions about a user's intentions and goals when interpreting prompts. Our findings largely hold true across conversational domains, user intents, and LLMs of varying sizes and families, indicating the promise of using prompt rewriting as a solution for better human-AI interactions.
• 摘要：在人民的专业和个人生活中，人类llm的对话越来越普遍，但是许多用户仍然很难从LLM聊天机器人那里获得有益的回应。造成此问题的原因之一是用户在制作有效提示中缺乏理解，以准确地传达其信息需求。同时，一方面，现实世界中的对话数据集的存在，另一方面是理解LLM的文本，为研究这个问题提供了独特的机会，其潜在解决方案及其潜在的解决方案。因此，在本文中，我们介绍了对真实人类聊天机器人对话的首次以LLM为中心的研究，重点是调查用户查询未表达信息需求的方面，以及使用LLMS来重写次优的用户提示的潜力。我们的发现表明，重新提示无效的提示可以从对话系统中获得更好的响应，同时保留用户的原始意图。值得注意的是，重写的性能在较长的对话中有所改善，在这种对话中，可以更准确地对用户需求进行上下文推断。此外，我们观察到，LLMS通常需要并且天生需要做出\ emph {Prolaible}在解释提示时对用户的意图和目标的假设。我们的发现在很大程度上跨越了对话域，用户意图和不同大小和家庭的LLM，这表明有望将及时重写作为更好的人类互动的解决方案。

Title: When Tom Eats Kimchi: Evaluating Cultural Bias of Multimodal Large Language Models in Cultural Mixture Contexts

• Authors: Jun Seong Kim, Kyaw Ye Thu, Javad Ismayilzada, Junyeong Park, Eunsu Kim, Huzama Ahmad, Na Min An, James Thorne, Alice Oh
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.16826
• Pdf URL: https://arxiv.org/pdf/2503.16826
• Copy Paste: [[2503.16826]] When Tom Eats Kimchi: Evaluating Cultural Bias of Multimodal Large Language Models in Cultural Mixture Contexts(https://arxiv.org/abs/2503.16826)
• Keywords: language model, gpt, llm
• Abstract: In a highly globalized world, it is important for multi-modal large language models (MLLMs) to recognize and respond correctly to mixed-cultural inputs. For example, a model should correctly identify kimchi (Korean food) in an image both when an Asian woman is eating it, as well as an African man is eating it. However, current MLLMs show an over-reliance on the visual features of the person, leading to misclassification of the entities. To examine the robustness of MLLMs to different ethnicity, we introduce MixCuBe, a cross-cultural bias benchmark, and study elements from five countries and four ethnicities. Our findings reveal that MLLMs achieve both higher accuracy and lower sensitivity to such perturbation for high-resource cultures, but not for low-resource cultures. GPT-4o, the best-performing model overall, shows up to 58% difference in accuracy between the original and perturbed cultural settings in low-resource cultures. Our dataset is publicly available at: this https URL.
• 摘要：在一个高度全球化的世界中，对于多模式大语模型（MLLM）来说，对混合文化投入的识别并正确响应很重要。例如，在亚洲妇女吃它以及非洲男人正在吃它时，模型应在图像中正确识别泡菜（韩国食品）。但是，当前的MLLM表现出对人的视觉特征的过度依赖，导致对实体的错误分类。为了研究MLLM对不同种族的鲁棒性，我们介绍了MixCube，跨文化偏见的基准，以及来自五个国家和四个种族的学习因素。我们的发现表明，MLLM既具有更高的准确性，又具有对这种扰动对高资源培养物的敏感性，而对低资源培养物的敏感性既可以达到”。 GPT-4O是整体表现最佳的模型，在低资源文化中，原始文化环境和扰动文化环境之间的准确性差异高达58％。我们的数据集可公开可用：此HTTPS URL。

Title: Imagine to Hear: Auditory Knowledge Generation can be an Effective Assistant for Language Models

• Authors: Suho Yoo, Hyunjong Ok, Jaeho Lee
• Subjects: cs.CL, cs.AI, cs.LG, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2503.16853
• Pdf URL: https://arxiv.org/pdf/2503.16853
• Copy Paste: [[2503.16853]] Imagine to Hear: Auditory Knowledge Generation can be an Effective Assistant for Language Models(https://arxiv.org/abs/2503.16853)
• Keywords: language model, prompt
• Abstract: Language models pretrained on text-only corpora often struggle with tasks that require auditory commonsense knowledge. Previous work addresses this problem by augmenting the language model to retrieve knowledge from external audio databases. This approach has several limitations, such as the potential lack of relevant audio in databases and the high costs associated with constructing and querying the databases. To address these issues, we propose Imagine to Hear, a novel approach that dynamically generates auditory knowledge using generative models. Our framework detects multiple audio-related textual spans from the given prompt and generates corresponding auditory knowledge. We develop several mechanisms to efficiently process multiple auditory knowledge, including a CLAP-based rejection sampler and a language-audio fusion module. Our experiments show that our method achieves state-of-the-art performance on AuditoryBench without relying on external databases, highlighting the effectiveness of our generation-based approach.
• 摘要：在仅在文本语料库中估计的语言模型通常会在需要听觉常识知识的任务上挣扎。以前的工作通过增加语言模型来从外部音频数据库中检索知识来解决此问题。这种方法有几个局限性，例如数据库中潜在缺乏相关音频以及与构建和查询数据库相关的高成本。为了解决这些问题，我们建议想到听到一种新颖的方法，该方法使用生成模型动态生成听觉知识。我们的框架从给定的提示符中检测到多个音频相关的文本跨度，并生成相应的听觉知识。我们开发了有效处理多种听觉知识的几种机制，包括基于拍手的拒绝采样器和语言拟合融合模块。我们的实验表明，我们的方法在不依赖外部数据库的情况下实现了在听觉台上的最新性能，突出了我们一代基于一代的方法的有效性。

Title: MMCR: Benchmarking Cross-Source Reasoning in Scientific Papers

• Authors: Yang Tian, Zheng Lu, Mingqi Gao, Zheng Liu, Bo Zhao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.16856
• Pdf URL: https://arxiv.org/pdf/2503.16856
• Copy Paste: [[2503.16856]] MMCR: Benchmarking Cross-Source Reasoning in Scientific Papers(https://arxiv.org/abs/2503.16856)
• Keywords: language model, gpt, chain-of-thought
• Abstract: Fully comprehending scientific papers by machines reflects a high level of Artificial General Intelligence, requiring the ability to reason across fragmented and heterogeneous sources of information, presenting a complex and practically significant challenge. While Vision-Language Models (VLMs) have made remarkable strides in various tasks, particularly those involving reasoning with evidence source from single image or text page, their ability to use cross-source information for reasoning remains an open problem. This work presents MMCR, a high-difficulty benchmark designed to evaluate VLMs' capacity for reasoning with cross-source information from scientific papers. The benchmark comprises 276 high-quality questions, meticulously annotated by humans across 7 subjects and 10 task types. Experiments with 18 VLMs demonstrate that cross-source reasoning presents a substantial challenge for existing models. Notably, even the top-performing model, GPT-4o, achieved only 48.55% overall accuracy, with only 20% accuracy in multi-table comprehension tasks, while the second-best model, Qwen2.5-VL-72B, reached 39.86% overall accuracy. Furthermore, we investigated the impact of the Chain-of-Thought (CoT) technique on cross-source reasoning and observed a detrimental effect on small models, whereas larger models demonstrated substantially enhanced performance. These results highlight the pressing need to develop VLMs capable of effectively utilizing cross-source information for reasoning.
• 摘要：机器完全理解科学论文反映了高水平的人工通用智能，需要能够推理跨越零散和异构信息来源的能力，从而提出了一个复杂且实际上显着的挑战。虽然视觉语言模型（VLM）在各种任务中取得了显着的进步，尤其是那些涉及与单个图像或文本页面的证据源的推理的任务，但他们使用跨源信息进行推理的能力仍然是一个开放的问题。这项工作介绍了MMCR，这是一种高难题的基准测试，旨在评估VLMS通过科学论文提供的跨源信息的推理能力。该基准包括276个高质量问题，由7种受试者和10种任务类型的人类精心注释。使用18个VLM的实验表明，跨源推理给现有模型带来了重大挑战。值得注意的是，即使表现出色的模型GPT-4O也仅达到48.55％的总体准确性，在多桌理解任务中只有20％的精度，而第二好的模型QWEN2.5-VL-72B达到了39.86％的总准确度。此外，我们研究了对跨源推理的经营链（COT）技术的影响，并观察到对小型模型的有害影响，而较大的模型表明性能大大提高。这些结果突出了迫切需要开发能够有效利用跨源信息进行推理的VLM。

Title: MTBench: A Multimodal Time Series Benchmark for Temporal Reasoning and Question Answering

• Authors: Jialin Chen, Aosong Feng, Ziyu Zhao, Juan Garza, Gaukhar Nurbek, Cheng Qin, Ali Maatouk, Leandros Tassiulas, Yifeng Gao, Rex Ying
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.16858
• Pdf URL: https://arxiv.org/pdf/2503.16858
• Copy Paste: [[2503.16858]] MTBench: A Multimodal Time Series Benchmark for Temporal Reasoning and Question Answering(https://arxiv.org/abs/2503.16858)
• Keywords: language model, llm
• Abstract: Understanding the relationship between textual news and time-series evolution is a critical yet under-explored challenge in applied data science. While multimodal learning has gained traction, existing multimodal time-series datasets fall short in evaluating cross-modal reasoning and complex question answering, which are essential for capturing complex interactions between narrative information and temporal patterns. To bridge this gap, we introduce Multimodal Time Series Benchmark (MTBench), a large-scale benchmark designed to evaluate large language models (LLMs) on time series and text understanding across financial and weather domains. MTbench comprises paired time series and textual data, including financial news with corresponding stock price movements and weather reports aligned with historical temperature records. Unlike existing benchmarks that focus on isolated modalities, MTbench provides a comprehensive testbed for models to jointly reason over structured numerical trends and unstructured textual narratives. The richness of MTbench enables formulation of diverse tasks that require a deep understanding of both text and time-series data, including time-series forecasting, semantic and technical trend analysis, and news-driven question answering (QA). These tasks target the model's ability to capture temporal dependencies, extract key insights from textual context, and integrate cross-modal information. We evaluate state-of-the-art LLMs on MTbench, analyzing their effectiveness in modeling the complex relationships between news narratives and temporal patterns. Our findings reveal significant challenges in current models, including difficulties in capturing long-term dependencies, interpreting causality in financial and weather trends, and effectively fusing multimodal information.
• 摘要：了解文本新闻与时间序列演变之间的关系是应用数据科学中的一个关键但爆炸案的挑战。尽管多模式学习已获得吸引力，但现有的多模式时间序列数据集在评估跨模式推理和复杂的问题答案方面缺乏，这对于捕获叙事信息和时间模式之间的复杂相互作用至关重要。为了弥合这一差距，我们介绍了多模式时间序列基准（MTBENCH），这是一个大规模的基准测试，旨在评估跨财务和天气领域的时间序列和文本理解大型语言模型（LLMS）。 mtbench包括配对的时间序列和文本数据，包括带有相应股价变动的财务新闻和与历史温度记录一致的天气报告。与专注于孤立方式的现有基准不同，MTBENCH为模型提供了全面的测试床，以共同理解结构化的数值趋势和非结构化的文本叙述。 MTBENCH的丰富性使得需要对文本和时间序列数据有深入了解的各种任务的制定，包括时间序列预测，语义和技术趋势分析以及新闻驱动的问题回答（QA）。这些任务针对模型捕获时间依赖性，从文本上下文中提取关键见解并集成跨模式信息的能力。我们在MTBench上评估了最新的LLM，分析了它们在建模新闻叙事与时间模式之间复杂关系方面的有效性。我们的发现揭示了当前模型中的重大挑战，包括在捕获长期依赖性，解释财务和天气趋势的因果关系以及有效地融合多模式信息的困难。

Title: Joint Extraction Matters: Prompt-Based Visual Question Answering for Multi-Field Document Information Extraction

• Authors: Mengsay Loem, Taiju Hosaka
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2503.16868
• Pdf URL: https://arxiv.org/pdf/2503.16868
• Copy Paste: [[2503.16868]] Joint Extraction Matters: Prompt-Based Visual Question Answering for Multi-Field Document Information Extraction(https://arxiv.org/abs/2503.16868)
• Keywords: language model, prompt
• Abstract: Visual question answering (VQA) has emerged as a flexible approach for extracting specific pieces of information from document images. However, existing work typically queries each field in isolation, overlooking potential dependencies across multiple items. This paper investigates the merits of extracting multiple fields jointly versus separately. Through experiments on multiple large vision language models and datasets, we show that jointly extracting fields often improves accuracy, especially when the fields share strong numeric or contextual dependencies. We further analyze how performance scales with the number of requested items and use a regression based metric to quantify inter field relationships. Our results suggest that multi field prompts can mitigate confusion arising from similar surface forms and related numeric values, providing practical methods for designing robust VQA systems in document information extraction tasks.
• 摘要：视觉问题回答（VQA）已成为一种灵活的方法，用于从文档图像中提取特定信息。但是，现有的工作通常会孤立地查询每个字段，忽略了多个项目的潜在依赖性。本文研究了共同与单独提取多个领域的优点。通过对多个大型视觉语言模型和数据集进行的实验，我们表明共同提取字段通常会提高准确性，尤其是当字段共享强大的数字或上下文依赖性时。我们进一步分析了绩效如何用所请求的项目数量扩展，并使用基于回归的度量来量化现场关系。我们的结果表明，多场提示可以减轻相似的表面形式和相关数值引起的混淆，从而提供了在文档信息提取任务中设计强大的VQA系统的实用方法。

Title: Assessing the Reliability and Validity of GPT-4 in Annotating Emotion Appraisal Ratings

• Authors: Deniss Ruder, Andero Uusberg, Kairit Sirts
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.16883
• Pdf URL: https://arxiv.org/pdf/2503.16883
• Copy Paste: [[2503.16883]] Assessing the Reliability and Validity of GPT-4 in Annotating Emotion Appraisal Ratings(https://arxiv.org/abs/2503.16883)
• Keywords: gpt, llm, prompt
• Abstract: Appraisal theories suggest that emotions arise from subjective evaluations of events, referred to as appraisals. The taxonomy of appraisals is quite diverse, and they are usually given ratings on a Likert scale to be annotated in an experiencer-annotator or reader-annotator paradigm. This paper studies GPT-4 as a reader-annotator of 21 specific appraisal ratings in different prompt settings, aiming to evaluate and improve its performance compared to human annotators. We found that GPT-4 is an effective reader-annotator that performs close to or even slightly better than human annotators, and its results can be significantly improved by using a majority voting of five completions. GPT-4 also effectively predicts appraisal ratings and emotion labels using a single prompt, but adding instruction complexity results in poorer performance. We also found that longer event descriptions lead to more accurate annotations for both model and human annotator ratings. This work contributes to the growing usage of LLMs in psychology and the strategies for improving GPT-4 performance in annotating appraisals.
• 摘要：评估理论表明，情感是由对事件的主观评估产生的，称为评估。评估的分类学非常多样化，通常以李克特量表给予评分，以在体验者通知者或阅读器通知者范式中注释。本文将GPT-4研究为不同及时设置中21个特定评估评级的读者通知者，旨在评估和改善其性能与人类注释者相比。我们发现，GPT-4是一种有效的读取器通知者，其性能比人类注释者接近甚至更好，并且通过使用五个完成的多数投票可以大大改善其结果。 GPT-4还使用单个提示有效地预测评估和情感标签，但增加指令的复杂性会导致性能较差。我们还发现，较长的事件描述会导致模型和人体注释等级的更准确的注释。这项工作有助于LLM在心理学中的日益增长的用法以及改善注释评估中GPT-4性能的策略。

Title: When Words Outperform Vision: VLMs Can Self-Improve Via Text-Only Training For Human-Centered Decision Making

• Authors: Zhe Hu, Jing Li, Yu Yin
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2503.16965
• Pdf URL: https://arxiv.org/pdf/2503.16965
• Copy Paste: [[2503.16965]] When Words Outperform Vision: VLMs Can Self-Improve Via Text-Only Training For Human-Centered Decision Making(https://arxiv.org/abs/2503.16965)
• Keywords: language model, gpt, llm, agent
• Abstract: Embodied decision-making is fundamental for AI agents operating in real-world environments. While Visual Language Models (VLMs) have advanced this capability, they still struggle with complex decisions, particularly in human-centered situations that require deep reasoning about human needs and values. In this study, we systematically evaluate open-sourced VLMs on multimodal human-centered decision-making tasks. We find that LLMs receiving only textual descriptions unexpectedly outperform their VLM counterparts of similar scale that process actual images, suggesting that visual alignment may hinder VLM abilities. To address this challenge, we propose a novel text-only training approach with synthesized textual data. This method strengthens VLMs' language components and transfers the learned abilities to multimodal inference, eliminating the need for expensive image-text paired data. Furthermore, we show that VLMs can achieve substantial performance gains through self-improvement, using training data generated by their LLM counterparts rather than relying on larger teacher models like GPT-4. Our findings establish a more efficient and scalable approach to enhancing VLMs' human-centered decision-making capabilities, opening new avenues for optimizing VLMs through self-improvement mechanisms.
• 摘要：体现的决策对于在现实世界环境中运行的AI代理至关重要。尽管视觉语言模型（VLM）提高了这种能力，但他们仍然在复杂的决策方面挣扎，尤其是在以人为中心的以人为需求和价值推理的情况下。在这项研究中，我们系统地评估了开源的VLM，以多模式为中心的决策任务。我们发现，仅接收文本描述的LLM出乎意料地胜过其VLM类似比例的VLM对应物，该比例可以处理实际图像，这表明视觉对齐可能会阻碍VLM的能力。为了应对这一挑战，我们提出了一种新颖的文本培训方法，并使用合成的文本数据提出了一种。此方法增强了VLMS的语言组成部分，并将学习能力转移到多模式推理，从而消除了对昂贵的图像文本配对数据的需求。此外，我们表明VLM可以使用LLM对应物生成的培训数据，而不是依靠GPT-4（例如GPT-4），可以通过自我完善实现大量绩效提高。我们的发现建立了一种更有效，更可扩展的方法来增强VLMS以人为中心的决策能力，开辟了新的途径，以通过自我改善机制优化VLM。

Title: A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applications

• Authors: Jian Guan, Junfei Wu, Jia-Nan Li, Chuanqi Cheng, Wei Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.17003
• Pdf URL: https://arxiv.org/pdf/2503.17003
• Copy Paste: [[2503.17003]] A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applications(https://arxiv.org/abs/2503.17003)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities, yet their transition to real-world applications reveals a critical limitation: the inability to adapt to individual preferences while maintaining alignment with universal human values. Current alignment techniques adopt a one-size-fits-all approach that fails to accommodate users' diverse backgrounds and needs. This paper presents the first comprehensive survey of personalized alignment-a paradigm that enables LLMs to adapt their behavior within ethical boundaries based on individual preferences. We propose a unified framework comprising preference memory management, personalized generation, and feedback-based alignment, systematically analyzing implementation approaches and evaluating their effectiveness across various scenarios. By examining current techniques, potential risks, and future challenges, this survey provides a structured foundation for developing more adaptable and ethically-aligned LLMs.
• 摘要：大型语言模型（LLM）表现出了显着的功能，但是它们向现实世界应用的过渡揭示了一个关键的局限性：无法适应个人偏好，同时保持与普遍人类价值观的一致性。当前的一致性技术采用了一种千篇一律的方法，无法适应用户的不同背景和需求。本文介绍了对个性化对齐的首次全面调查 - A范式使LLM可以根据个人偏好在道德界限内适应其行为。我们提出了一个统一的框架，包括偏好内存管理，个性化的生成和基于反馈的对齐，系统地分析实施方法并在各种情况下评估它们的有效性。通过检查当前技术，潜在的风险和未来挑战，这项调查为开发更适合适应性和道德的LLM提供了结构化的基础。

Title: Summarization Metrics for Spanish and Basque: Do Automatic Scores and LLM-Judges Correlate with Humans?

• Authors: Jeremy Barnes, Naiara Perez, Alba Bonet-Jover, Begoña Altuna
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.17039
• Pdf URL: https://arxiv.org/pdf/2503.17039
• Copy Paste: [[2503.17039]] Summarization Metrics for Spanish and Basque: Do Automatic Scores and LLM-Judges Correlate with Humans?(https://arxiv.org/abs/2503.17039)
• Keywords: llm, prompt
• Abstract: Studies on evaluation metrics and LLM-as-a-Judge models for automatic text summarization have largely been focused on English, limiting our understanding of their effectiveness in other languages. Through our new dataset BASSE (BAsque and Spanish Summarization Evaluation), we address this situation by collecting human judgments on 2,040 abstractive summaries in Basque and Spanish, generated either manually or by five LLMs with four different prompts. For each summary, annotators evaluated five criteria on a 5-point Likert scale: coherence, consistency, fluency, relevance, and 5W1H. We use these data to reevaluate traditional automatic metrics used for evaluating summaries, as well as several LLM-as-a-Judge models that show strong performance on this task in English. Our results show that currently proprietary judge LLMs have the highest correlation with human judgments, followed by criteria-specific automatic metrics, while open-sourced judge LLMs perform poorly. We release BASSE and our code publicly, along with the first large-scale Basque summarization dataset containing 22,525 news articles with their subheads.
• 摘要：关于自动文本摘要的评估指标和LLM-AS-A-A-A-A-A-A-A-A-A-A-A-SA-A-Gudge模型的研究主要集中在英语上，从而限制了我们对它们在其他语言中的有效性的理解。通过我们的新数据集Basse（巴斯克和西班牙语摘要评估），我们通过在巴斯克和西班牙语中对2,040个抽象性摘要进行人体判断来解决这种情况，该判断是手动或通过五个具有四个不同提示的LLM生成的。对于每个摘要，注释者以5点李克特量表评估了五个标准：连贯性，一致性，流利性，相关性和5W1H。我们使用这些数据来重新评估用于评估摘要的传统自动指标，以及几种llm-as-a-a-a-a-a-Gudge模型，这些模型在英语中显示出强烈的性能。我们的结果表明，目前专有的法官LLMS与人类判断的相关性最高，其次是特定于标准的自动指标，而开源的LLMS法官的表现较差。我们公开发布BASSE和我们的代码，以及第一个大规模的巴斯克摘要数据集，其中包含22,525篇新闻文章及其子头。

Title: A Study into Investigating Temporal Robustness of LLMs

• Authors: Jonas Wallat, Abdelrahman Abdallah, Adam Jatowt, Avishek Anand
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2503.17073
• Pdf URL: https://arxiv.org/pdf/2503.17073
• Copy Paste: [[2503.17073]] A Study into Investigating Temporal Robustness of LLMs(https://arxiv.org/abs/2503.17073)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) encapsulate a surprising amount of factual world knowledge. However, their performance on temporal questions and historical knowledge is limited because they often cannot understand temporal scope and orientation or neglect the temporal aspect altogether. In this study, we aim to measure precisely how robust LLMs are for question answering based on their ability to process temporal information and perform tasks requiring temporal reasoning and temporal factual knowledge. Specifically, we design eight time-sensitive robustness tests for factual information to check the sensitivity of six popular LLMs in the zero-shot setting. Overall, we find LLMs lacking temporal robustness, especially to temporal reformulations and the use of different granularities of temporal references. We show how a selection of these eight tests can be used automatically to judge a model's temporal robustness for user questions on the fly. Finally, we apply the findings of this study to improve the temporal QA performance by up to 55 percent.
• 摘要：大型语言模型（LLMS）封装了令人惊讶的事实世界知识。但是，他们在时间问题和历史知识上的表现受到限制，因为他们通常无法完全理解时间范围，方向或完全忽略了时间方面。在这项研究中，我们旨在根据其处理时间信息和执行需要时间推理和时间事实知识的任务的能力来精确衡量问答回答的鲁棒性LLM的鲁棒性。具体而言，我们设计了八个时间敏感的鲁棒性测试，以获取事实信息，以检查零拍设置中六个流行LLM的灵敏度。总体而言，我们发现LLMs缺乏时间鲁棒性，尤其是时间重新制定以及时间参考的不同粒度。我们展示了如何自动使用这八个测试的选择，以判断模型的时间鲁棒性，以实现用户问题。最后，我们应用了这项研究的发现，将时间质量检查的性能提高高达55％。

Title: Modifying Large Language Model Post-Training for Diverse Creative Writing

• Authors: John Joon Young Chung, Vishakh Padmakumar, Melissa Roemmele, Yuqian Sun, Max Kreminski
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.17126
• Pdf URL: https://arxiv.org/pdf/2503.17126
• Copy Paste: [[2503.17126]] Modifying Large Language Model Post-Training for Diverse Creative Writing(https://arxiv.org/abs/2503.17126)
• Keywords: language model, gpt, llm, prompt
• Abstract: As creative writing tasks do not have singular correct answers, large language models (LLMs) trained to perform these tasks should be able to generate diverse valid outputs. However, LLM post-training often focuses on improving generation quality but neglects to facilitate output diversity. Hence, in creative writing generation, we investigate post-training approaches to promote both output diversity and quality. Our core idea is to include deviation -- the degree of difference between a training sample and all other samples with the same prompt -- in the training objective to facilitate learning from rare high-quality instances. By adopting our approach to direct preference optimization (DPO) and odds ratio preference optimization (ORPO), we demonstrate that we can promote the output diversity of trained models while minimally decreasing quality. Our best model with 8B parameters could achieve on-par diversity as a human-created dataset while having output quality similar to the best instruction-tuned models we examined, GPT-4o and DeepSeek-R1. We further validate our approaches with a human evaluation, an ablation, and a comparison to an existing diversification approach, DivPO.
• 摘要：由于创意写作任务没有单数正确的答案，因此训练执行这些任务的大型语言模型（LLM）应该能够产生多样化的有效输出。但是，LLM训练后通常着重于提高发电质量，但忽略了以促进产出多样性。因此，在创意写作中，我们研究了培训后的方法，以促进产出多样性和质量。我们的核心思想是包括偏差 - 培训样本与所有其他提示的所有其他样本之间的差异程度 - 在培训目标中促进从罕见的高质量实例中学习。通过采用我们直接偏好优化（DPO）和优势优先优化（ORPO）的方法，我们证明我们可以促进受过训练的模型的输出多样性，同时降低质量。我们使用8b参数的最佳模型可以作为人为创建的数据集实现出色的多样性，同时具有与我们研究过的最佳指令调整模型，GPT-4O和DeepSeek-R1相似的输出质量。我们通过人类评估，消融和与现有多元化方法Divpo的比较进一步验证了我们的方法。

Title: CoKe: Customizable Fine-Grained Story Evaluation via Chain-of-Keyword Rationalization

• Authors: Brihi Joshi, Sriram Venkatapathy, Mohit Bansal, Nanyun Peng, Haw-Shiuan Chang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.17136
• Pdf URL: https://arxiv.org/pdf/2503.17136
• Copy Paste: [[2503.17136]] CoKe: Customizable Fine-Grained Story Evaluation via Chain-of-Keyword Rationalization(https://arxiv.org/abs/2503.17136)
• Keywords: language model, gpt
• Abstract: Evaluating creative text such as human-written stories using language models has always been a challenging task -- owing to the subjectivity of multi-annotator ratings. To mimic the thinking process of humans, chain of thought (CoT) generates free-text explanations that help guide a model's predictions and Self-Consistency (SC) marginalizes predictions over multiple generated explanations. In this study, we discover that the widely-used self-consistency reasoning methods cause suboptimal results due to an objective mismatch between generating 'fluent-looking' explanations vs. actually leading to a good rating prediction for an aspect of a story. To overcome this challenge, we propose $\textbf{C}$hain-$\textbf{o}$f-$\textbf{Ke}$ywords (CoKe), that generates a sequence of keywords $\textit{before}$ generating a free-text rationale, that guide the rating prediction of our evaluation language model. Then, we generate a diverse set of such keywords, and aggregate the scores corresponding to these generations. On the StoryER dataset, CoKe based on our small fine-tuned evaluation models not only reach human-level performance and significantly outperform GPT-4 with a 2x boost in correlation with human annotators, but also requires drastically less number of parameters.
• 摘要：由于多通道评级的主观性，使用语言模型评估创意文本（例如使用语言模型）一直是一项具有挑战性的任务。为了模仿人类的思维过程，思想链（COT）产生了自由文本的解释，有助于指导模型的预测和自洽性（SC），使对多个生成的解释的预测边缘化。在这项研究中，我们发现，由于生成“流利的”解释与实际上导致故事方面的良好评分预测之间的客观不匹配，因此广泛使用的自符合性推理方法导致了次优结果。为了克服这一挑战，我们提出$ \ textbf {c} $ hain-$ \ textbf {o} $ f - $ f-$ \ textbf {ke} $ ywords（coke），该$ ywords（coke）生成了一系列关键字$ \ textit $ \ textit {terfer ther} $ genter-Textit} $ genter-textext ratationale felet-Text ratationale felet-Text ratationale the评估语言模型。然后，我们生成一系列此类关键字，并汇总与这些世代相对应的分数。在Storyer数据集上，基于我们的小型微调评估模型的可乐不仅达到人类水平的性能，而且要超过GPT-4，其与人类注释者相关的2倍提升，而且还需要少的参数。

Title: A Language Anchor-Guided Method for Robust Noisy Domain Generalization

• Authors: Zilin Dai, Lehong Wang, Fangzhou Lin, Yidong Wang, Zhigang Li, Kazunori D Yamada, Ziming Zhang, Wang Lu
• Subjects: cs.CL, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.17211
• Pdf URL: https://arxiv.org/pdf/2503.17211
• Copy Paste: [[2503.17211]] A Language Anchor-Guided Method for Robust Noisy Domain Generalization(https://arxiv.org/abs/2503.17211)
• Keywords: language model
• Abstract: Real-world machine learning applications often struggle with two major challenges: distribution shift and label noise. Models tend to overfit by focusing on redundant and uninformative features in the training data, which makes it hard for them to generalize to the target domain. Noisy data worsens this problem by causing further overfitting to the noise, meaning that existing methods often fail to tell the difference between true, invariant features and misleading, spurious ones. To tackle these issues, we introduce Anchor Alignment and Adaptive Weighting (A3W). This new algorithm uses sample reweighting guided by natural language processing (NLP) anchors to extract more representative features. In simple terms, A3W leverages semantic representations from natural language models as a source of domain-invariant prior knowledge. Additionally, it employs a weighted loss function that adjusts each sample's contribution based on its similarity to the corresponding NLP anchor. This adjustment makes the model more robust to noisy labels. Extensive experiments on standard benchmark datasets show that A3W consistently outperforms state-of-the-art domain generalization methods, offering significant improvements in both accuracy and robustness across different datasets and noise levels.
• 摘要：现实世界中的机器学习应用程序通常面临两个主要挑战：分配转移和标签噪声。模型倾向于通过专注于训练数据中的冗余和非信息性特征来过度合适，这使得他们很难将其推广到目标域。嘈杂的数据通过进一步过度适应噪音，使这个问题恶化，这意味着现有方法通常无法分辨出真实，不变特征与误导性，虚假的方法之间的差异。为了解决这些问题，我们引入锚定对齐和自适应加权（A3W）。该新算法使用以自然语言处理（NLP）锚指导的样本重新加权来提取更多代表性的功能。简而言之，A3W利用自然语言模型的语义表示作为域不变的先验知识的来源。此外，它采用了加权损耗函数，该功能根据其与相应的NLP锚的相似性来调整每个样本的贡献。这种调整使该模型更适合嘈杂标签。标准基准数据集的广泛实验表明，A3W始终胜过最先进的域概括方法，从而在不同的数据集和噪声水平上提供了准确性和鲁棒性的显着提高。

Title: Automating Adjudication of Cardiovascular Events Using Large Language Models

• Authors: Sonish Sivarajkumar, Kimia Ameri, Chuqin Li, Yanshan Wang, Min Jiang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.17222
• Pdf URL: https://arxiv.org/pdf/2503.17222
• Copy Paste: [[2503.17222]] Automating Adjudication of Cardiovascular Events Using Large Language Models(https://arxiv.org/abs/2503.17222)
• Keywords: language model, llm
• Abstract: Cardiovascular events, such as heart attacks and strokes, remain a leading cause of mortality globally, necessitating meticulous monitoring and adjudication in clinical trials. This process, traditionally performed manually by clinical experts, is time-consuming, resource-intensive, and prone to inter-reviewer variability, potentially introducing bias and hindering trial progress. This study addresses these critical limitations by presenting a novel framework for automating the adjudication of cardiovascular events in clinical trials using Large Language Models (LLMs). We developed a two-stage approach: first, employing an LLM-based pipeline for event information extraction from unstructured clinical data and second, using an LLM-based adjudication process guided by a Tree of Thoughts approach and clinical endpoint committee (CEC) guidelines. Using cardiovascular event-specific clinical trial data, the framework achieved an F1-score of 0.82 for event extraction and an accuracy of 0.68 for adjudication. Furthermore, we introduce the CLEART score, a novel, automated metric specifically designed for evaluating the quality of AI-generated clinical reasoning in adjudicating cardiovascular events. This approach demonstrates significant potential for substantially reducing adjudication time and costs while maintaining high-quality, consistent, and auditable outcomes in clinical trials. The reduced variability and enhanced standardization also allow for faster identification and mitigation of risks associated with cardiovascular therapies.
• 摘要：心血管事件（例如心脏病发作和中风）仍然是全球死亡率的主要原因，需要在临床试验中进行细致的监测和裁决。传统上，临床专家手动执行此过程是耗时，资源密集的，并且容易发生浏览器的可变性，并可能引入偏见和阻碍试验的进度。这项研究通过提出一个新的框架来解决这些临界局限性，用于使用大语言模型（LLMS）在临床试验中自动化心血管事件。我们开发了一种两阶段的方法：首先，采用基于LLM的管道从非结构化的临床数据中提取事件信息，然后使用基于LLM的裁决过程以思想树方法和临床端点委员会（CEC）指南为指导。使用心血管特定事件的临床试验数据，该框架的F1得分为0.82，用于提取事件，裁决的精度为0.68。此外，我们引入了Cleart Score，这是一种专门设计的新型自动化指标，用于评估AI生成的临床推理在裁定心血管事件中的质量。这种方法表明，在临床试验中保持高质量，一致和可审计的结果，可以大大减少裁决时间和成本。降低的变异性和增强标准化还允许更快地识别和缓解与心血管疗法相关的风险。

Title: SafeMERGE: Preserving Safety Alignment in Fine-Tuned Large Language Models via Selective Layer-Wise Model Merging

• Authors: Aladin Djuhera, Swanand Ravindra Kadhe, Farhan Ahmed, Syed Zawad, Holger Boche
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.17239
• Pdf URL: https://arxiv.org/pdf/2503.17239
• Copy Paste: [[2503.17239]] SafeMERGE: Preserving Safety Alignment in Fine-Tuned Large Language Models via Selective Layer-Wise Model Merging(https://arxiv.org/abs/2503.17239)
• Keywords: language model, llm, chat
• Abstract: Fine-tuning large language models (LLMs) on downstream tasks can inadvertently erode their safety alignment, even for benign fine-tuning datasets. We address this challenge by proposing SafeMERGE, a post-fine-tuning framework that preserves safety while maintaining task utility. It achieves this by selectively merging fine-tuned and safety-aligned model layers only when those deviate from safe behavior, measured by a cosine similarity criterion. We evaluate SafeMERGE against other fine-tuning- and post-fine-tuning-stage approaches for Llama-2-7B-Chat and Qwen-2-7B-Instruct models on GSM8K and PubMedQA tasks while exploring different merging strategies. We find that SafeMERGE consistently reduces harmful outputs compared to other baselines without significantly sacrificing performance, sometimes even enhancing it. The results suggest that our selective, subspace-guided, and per-layer merging method provides an effective safeguard against the inadvertent loss of safety in fine-tuned LLMs while outperforming simpler post-fine-tuning-stage defenses.
• 摘要：在下游任务上的微调大语言模型（LLM）也会无意中侵蚀其安全对准，即使对于良性微调数据集也是如此。我们通过提出Safemerge（一个在维护任务实用程序的同时保留安全性的框架）来应对这一挑战。它通过选择性合并微调和安全一致的模型层才能实现这一目标，只有在偏离安全行为的情况下，通过余弦相似性标准来衡量。我们对Llama-2-7B-Chat和QWEN-2-7B-Instruct模型的其他微调和预先调节阶段的方法评估了Safemerge，同时探索了不同的合并策略。我们发现，与其他基线相比，Safemerge始终降低有害产量，而不会显着牺牲性能，有时甚至可以增强绩效。结果表明，我们的选择性，子空间引导和每层合并方法为您提供了有效的保障，以防止微调LLM中无意中的安全性损失，同时表现优于简单的细节后调节阶段防御。

Title: KL3M Tokenizers: A Family of Domain-Specific and Character-Level Tokenizers for Legal, Financial, and Preprocessing Applications

• Authors: Michael J Bommarito, Daniel Martin Katz, Jillian Bommarito
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.17247
• Pdf URL: https://arxiv.org/pdf/2503.17247
• Copy Paste: [[2503.17247]] KL3M Tokenizers: A Family of Domain-Specific and Character-Level Tokenizers for Legal, Financial, and Preprocessing Applications(https://arxiv.org/abs/2503.17247)
• Keywords: gpt
• Abstract: We present the KL3M tokenizers, a family of specialized tokenizers for legal, financial, and governmental text. Despite established work on tokenization, specialized tokenizers for professional domains remain understudied. Our paper offers two main contributions to this area. First, we introduce domain-specific BPE tokenizers for legal, financial, and governmental text. Our kl3m-004-128k-cased tokenizer uses 9-17% fewer tokens than GPT-4o and Llama3 for domain-specific documents, despite having a smaller vocabulary. For specialized terminology, our cased tokenizer is even more efficient, using up to 83% fewer tokens for legal terms and 39% fewer tokens for financial terms. Second, we develop character-level BPE tokenizers (4K, 8K, and 16K vocabulary sizes) for text correction tasks like OCR post-processing. These tokenizers keep consistent token boundaries between error-containing and correct text, making it easier for models to learn correction patterns. These tokenizers help professional applications by fitting more text in context windows, reducing computational needs, and preserving the meaning of domain-specific terms. Our analysis shows these efficiency gains directly benefit the processing of long legal and financial documents. We release all tokenizers and code through GitHub and Hugging Face to support further research in specialized tokenization.
• 摘要：我们介绍了KL3M Tokenizers，这是一个专门用于法律，财务和政府文本的家族。尽管既定的令牌化工作，但专门用于专业领域的引物仍在研究中。我们的论文为该领域提供了两个主要贡献。首先，我们介绍了特定领域的BPE令牌，用于法律，财务和政府文本。我们的KL3M-004-128K代币仪的代币比GPT-4O和Llama3少9-17％，尽管词汇较小，但对于特定于域的文档而言。对于专门的术语，我们的外壳令牌更有效，使用法律条款少了83％，而对于财务条款的代币也减少了39％。其次，我们为文本校正任务（例如OCR后处理）开发了字符级BPE令牌（4K，8K和16K词汇尺寸）。这些代币器在包含错误和正确的文本之间保持一致的令牌边界，从而使模型更容易学习校正模式。这些图形器通过在上下文窗口中拟合更多文本，减少计算需求并保留特定于域特定术语的含义来帮助专业应用程序。我们的分析表明，这些效率的提高直接受益于长期法律和财务文件的处理。我们通过github释放所有引物和代码，并拥抱面孔，以支持专业令牌化的进一步研究。

Title: CASE -- Condition-Aware Sentence Embeddings for Conditional Semantic Textual Similarity Measurement

• Authors: Gaifan Zhang, Yi Zhou, Danushka Bollegala
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.17279
• Pdf URL: https://arxiv.org/pdf/2503.17279
• Copy Paste: [[2503.17279]] CASE -- Condition-Aware Sentence Embeddings for Conditional Semantic Textual Similarity Measurement(https://arxiv.org/abs/2503.17279)
• Keywords: language model, llm
• Abstract: The meaning conveyed by a sentence often depends on the context in which it appears. Despite the progress of sentence embedding methods, it remains unclear how to best modify a sentence embedding conditioned on its context. To address this problem, we propose Condition-Aware Sentence Embeddings (CASE), an efficient and accurate method to create an embedding for a sentence under a given condition. First, CASE creates an embedding for the condition using a Large Language Model (LLM), where the sentence influences the attention scores computed for the tokens in the condition during pooling. Next, a supervised nonlinear projection is learned to reduce the dimensionality of the LLM-based text embeddings. We show that CASE significantly outperforms previously proposed Conditional Semantic Textual Similarity (C-STS) methods on an existing standard benchmark dataset. We find that subtracting the condition embedding consistently improves the C-STS performance of LLM-based text embeddings. Moreover, we propose a supervised dimensionality reduction method that not only reduces the dimensionality of LLM-based embeddings but also significantly improves their performance.
• 摘要：句子传达的含义通常取决于其出现的上下文。尽管嵌入方法取得了进展，但仍不清楚如何最好地修改以其上下文为条件的嵌入句子。为了解决这个问题，我们提出了条件感知的句子嵌入（案例），这是一种在给定条件下为句子创建嵌入的有效而准确的方法。首先，案例使用大型语言模型（LLM）为条件创建嵌入，其中该句子会影响池期在情况下为标记计算的注意力分数。接下来，学会了监督的非线性投影，以减少基于LLM的文本嵌入的维度。我们表明，在现有标准基准数据集上，该案例明显优于先前提出的条件语义文本相似性（C-STS）方法。我们发现减去嵌入条件始终如一地改善了基于LLM的文本嵌入的C-STS性能。此外，我们提出了一种监督的降低方法，不仅降低了基于LLM的嵌入的维度，而且可以显着提高其性能。

Title: FastCuRL: Curriculum Reinforcement Learning with Progressive Context Extension for Efficient Training R1-like Reasoning Models

• Authors: Mingyang Song, Mao Zheng, Zheng Li, Wenjie Yang, Xuan Luo, Yue Pan, Feng Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.17287
• Pdf URL: https://arxiv.org/pdf/2503.17287
• Copy Paste: [[2503.17287]] FastCuRL: Curriculum Reinforcement Learning with Progressive Context Extension for Efficient Training R1-like Reasoning Models(https://arxiv.org/abs/2503.17287)
• Keywords: language model, prompt, chain-of-thought
• Abstract: In this paper, we propose \textbf{\textsc{FastCuRL}}, a simple yet efficient \textbf{Cu}rriculum \textbf{R}einforcement \textbf{L}earning approach with context window extending strategy to accelerate the reinforcement learning training efficiency for R1-like reasoning models while enhancing their performance in tackling complex reasoning tasks with long chain-of-thought rationales, particularly with a 1.5B parameter language model. \textbf{\textsc{FastCuRL}} consists of two main procedures: length-aware training data segmentation and context window extension training. Specifically, the former first splits the original training data into three different levels by the input prompt length, and then the latter leverages segmented training datasets with a progressively increasing context window length to train the reasoning model. Experimental results demonstrate that \textbf{\textsc{FastCuRL}}-1.5B-Preview surpasses DeepScaleR-1.5B-Preview across all five datasets (including MATH 500, AIME 2024, AMC 2023, Minerva Math, and OlympiadBench) while only utilizing 50\% of training steps. Furthermore, all training stages for FastCuRL-1.5B-Preview are completed using just a single node with 8 GPUs.
• 摘要：在本文中，我们提出\ textbf {\ textsc {fastcurl}}，这是一种有效而有效的\ textbf {cu} rriculum \ textbf {r} einforevement \ textbf {r} einforkection \ textbf {l}通过在上下文中促进策略，以加强型号的范围，以实现型号的范围，以加强型号的范围，以加强型号的范围，以加强型号的效力，该策略的效率很长，可以使得型效率地努力进行RE1型R1型效​​率R1型，以实施R1型R1经过思考的理由，特别是使用1.5B参数语言模型。 \ textbf {\ textsc {fastcurl}}由两个主要过程组成：长度感知培训数据分割和上下文窗口扩展培训。具体而言，前者首先将原始培训数据分为三个不同的级别，然后通过输入及时长度将其分割为分割的培训数据集，并逐渐增加上下文窗口长度来训练推理模型。实验结果表明，\ textbf {\ textsc {fastcurl}} - 1.5b-preview超过所有五个数据集（包括数学500，AIME 2024，AMC 2023，MINERVA MATH，MINERVA MATH和OLYMPIADBENCH）的DeepScaler-1.5b-1.5b-preview，同时仅利用50 \％的培训步骤。此外，仅使用一个带有8个GPU的单个节点完成了FastCurl-1.5b-preiview的所有训练阶段。

Title: Efficient Intent-Based Filtering for Multi-Party Conversations Using Knowledge Distillation from LLMs

• Authors: Reem Gody, Mohamed Abdelghaffar, Mohammed Jabreel, Ahmed Tawfik
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.17336
• Pdf URL: https://arxiv.org/pdf/2503.17336
• Copy Paste: [[2503.17336]] Efficient Intent-Based Filtering for Multi-Party Conversations Using Knowledge Distillation from LLMs(https://arxiv.org/abs/2503.17336)
• Keywords: language model, llm, chat
• Abstract: Large language models (LLMs) have showcased remarkable capabilities in conversational AI, enabling open-domain responses in chat-bots, as well as advanced processing of conversations like summarization, intent classification, and insights generation. However, these models are resource-intensive, demanding substantial memory and computational power. To address this, we propose a cost-effective solution that filters conversational snippets of interest for LLM processing, tailored to the target downstream application, rather than processing every snippet. In this work, we introduce an innovative approach that leverages knowledge distillation from LLMs to develop an intent-based filter for multi-party conversations, optimized for compute power constrained environments. Our method combines different strategies to create a diverse multi-party conversational dataset, that is annotated with the target intents and is then used to fine-tune the MobileBERT model for multi-label intent classification. This model achieves a balance between efficiency and performance, effectively filtering conversation snippets based on their intents. By passing only the relevant snippets to the LLM for further processing, our approach significantly reduces overall operational costs depending on the intents and the data distribution as demonstrated in our experiments.
• 摘要：大型语言模型（LLMS）在对话AI中展示了出色的功能，在聊天机器人中实现了开放域的响应，以及对对话的高级处理，例如摘要，意图分类和见解生成。但是，这些模型是资源密集的，苛刻的记忆力和计算能力。为了解决这个问题，我们提出了一种具有成本效益的解决方案，该解决方案量过滤了针对目标下游应用程序的LLM处理感兴趣的对话段，而不是处理每个片段。在这项工作中，我们介绍了一种创新的方法，该方法利用LLMS的知识蒸馏来开发用于多方对话的基于意图的过滤器，对计算功率约束环境进行了优化。我们的方法结合了不同的策略来创建各种多方对话数据集，该数据集用目标意图注释，然后用于微调多标签意图分类的Moberbert模型。该模型在效率和性能之间达到了平衡，根据其意图有效地过滤对话片段。通过仅将相关片段传递到LLM进行进一步处理，我们的方法可大大降低总体运营成本，具体取决于我们实验中所示的意图和数据分布。

Title: Dancing with Critiques: Enhancing LLM Reasoning with Stepwise Natural Language Self-Critique

• Authors: Yansi Li, Jiahao Xu, Tian Liang, Xingyu Chen, Zhiwei He, Qiuzhi Liu, Rui Wang, Zhuosheng Zhang, Zhaopeng Tu, Haitao Mi, Dong Yu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.17363
• Pdf URL: https://arxiv.org/pdf/2503.17363
• Copy Paste: [[2503.17363]] Dancing with Critiques: Enhancing LLM Reasoning with Stepwise Natural Language Self-Critique(https://arxiv.org/abs/2503.17363)
• Keywords: language model, llm
• Abstract: Enhancing the reasoning capabilities of large language models (LLMs), particularly for complex tasks requiring multi-step logical deductions, remains a significant challenge. Traditional inference time scaling methods utilize scalar reward signals from process reward models to evaluate candidate reasoning steps, but these scalar rewards lack the nuanced qualitative information essential for understanding and justifying each step. In this paper, we propose a novel inference-time scaling approach -- stepwise natural language self-critique (PANEL), which employs self-generated natural language critiques as feedback to guide the step-level search process. By generating rich, human-readable critiques for each candidate reasoning step, PANEL retains essential qualitative information, facilitating better-informed decision-making during inference. This approach bypasses the need for task-specific verifiers and the associated training overhead, making it broadly applicable across diverse tasks. Experimental results on challenging reasoning benchmarks, including AIME and GPQA, demonstrate that PANEL significantly enhances reasoning performance, outperforming traditional scalar reward-based methods. Our code is available at this https URL to support and encourage future research in this promising field.
• 摘要：增强大语言模型（LLM）的推理能力，特别是对于需要多步逻辑扣除的复杂任务，仍然是一个重大挑战。传统的推理时间缩放方法利用来自过程奖励模型的标量奖励信号来评估候选推理步骤，但是这些标量奖励缺乏用于理解和证明每个步骤合理的细微差异信息。在本文中，我们提出了一种新颖的推理时间缩放方法 - 逐步自然语言自我评价（Panel），该方法采用自我生成的自然语言批评作为反馈来指导阶梯搜索过程。通过为每个候选推理步骤产生丰富的人类可读性批评，小组保留了基本的定性信息，从而促进了推理期间更有信息的决策。这种方法绕过了对特定于任务的验证者和相关培训间接费用的需求，从而广泛适用于各种任务。包括AIME和GPQA在内的挑战推理基准的实验结果表明，小组可显着提高推理性能，表现优于传统的基于标量奖励的方法。我们的代码可在此HTTPS URL上获得，以支持和鼓励在这个有希望的领域的未来研究。