2025-01-13

Title: LLM-MedQA: Enhancing Medical Question Answering through Case Studies in Large Language Models

• Authors: Hang Yang, Hao Chen, Hui Guo, Yineng Chen, Ching-Sheng Lin, Shu Hu, Jinrong Hu, Xi Wu, Xin Wang
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2501.05464
• Pdf URL: https://arxiv.org/pdf/2501.05464
• Copy Paste: [[2501.05464]] LLM-MedQA: Enhancing Medical Question Answering through Case Studies in Large Language Models(https://arxiv.org/abs/2501.05464)
• Keywords: language model, llm, agent
• Abstract: Accurate and efficient question-answering systems are essential for delivering high-quality patient care in the medical field. While Large Language Models (LLMs) have made remarkable strides across various domains, they continue to face significant challenges in medical question answering, particularly in understanding domain-specific terminologies and performing complex reasoning. These limitations undermine their effectiveness in critical medical applications. To address these issues, we propose a novel approach incorporating similar case generation within a multi-agent medical question-answering (MedQA) system. Specifically, we leverage the Llama3.1:70B model, a state-of-the-art LLM, in a multi-agent architecture to enhance performance on the MedQA dataset using zero-shot learning. Our method capitalizes on the model's inherent medical knowledge and reasoning capabilities, eliminating the need for additional training data. Experimental results show substantial performance gains over existing benchmark models, with improvements of 7% in both accuracy and F1-score across various medical QA tasks. Furthermore, we examine the model's interpretability and reliability in addressing complex medical queries. This research not only offers a robust solution for medical question answering but also establishes a foundation for broader applications of LLMs in the medical domain.
• 摘要：准确高效的问答系统对于在医疗领域提供高质量的患者护理至关重要。虽然大型语言模型 (LLM) 在各个领域取得了显著进展，但它们在医学问答方面仍然面临重大挑战，特别是在理解领域特定术语和执行复杂推理方面。这些限制削弱了它们在关键医疗应用中的有效性。为了解决这些问题，我们提出了一种新颖的方法，将类似案例生成纳入多智能体医学问答 (MedQA) 系统中。具体来说，我们在多智能体架构中利用最先进的 LLM Llama3.1:70B 模型，使用零样本学习来提高 MedQA 数据集的性能。我们的方法利用模型固有的医学知识和推理能力，无需额外的训练数据。实验结果显示，与现有基准模型相比，性能有显著提升，在各种医学 QA 任务中的准确率和 F1 分数都提高了 7%。此外，我们还检查了模型在解决复杂医学查询方面的可解释性和可靠性。这项研究不仅为医学问答提供了强有力的解决方案，而且为法学硕士在医学领域的更广泛应用奠定了基础。

Title: Small Language Models (SLMs) Can Still Pack a Punch: A survey

• Authors: Shreyas Subramanian, Vikram Elango, Mecit Gungor
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.05465
• Pdf URL: https://arxiv.org/pdf/2501.05465
• Copy Paste: [[2501.05465]] Small Language Models (SLMs) Can Still Pack a Punch: A survey(https://arxiv.org/abs/2501.05465)
• Keywords: language model, llm
• Abstract: As foundation AI models continue to increase in size, an important question arises - is massive scale the only path forward? This survey of about 160 papers presents a family of Small Language Models (SLMs) in the 1 to 8 billion parameter range that demonstrate smaller models can perform as well, or even outperform large models. We explore task agnostic, general purpose SLMs, task-specific SLMs and techniques to create SLMs that can guide the community to build models while balancing performance, efficiency, scalability and cost. Furthermore we define and characterize SLMs' effective sizes, representing increased capability with respect to LLMs.
• 摘要：随着基础 AI 模型的规模不断扩大，一个重要的问题出现了——大规模是唯一的出路吗？这项对约 160 篇论文的调查介绍了一系列参数范围在 10 亿到 80 亿之间的小型语言模型 (SLM)，这些模型表明较小的模型可以表现得与大型模型一样好，甚至优于大型模型。我们探索与任务无关的通用 SLM、特定于任务的 SLM 和创建 SLM 的技术，这些 SLM 可以指导社区构建模型，同时平衡性能、效率、可扩展性和成本。此外，我们定义并描述了 SLM 的有效尺寸，代表了相对于 LLM 的增强能力。

Title: LatteReview: A Multi-Agent Framework for Systematic Review Automation Using Large Language Models

• Authors: Pouria Rouzrokh, Moein Shariatnia
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.05468
• Pdf URL: https://arxiv.org/pdf/2501.05468
• Copy Paste: [[2501.05468]] LatteReview: A Multi-Agent Framework for Systematic Review Automation Using Large Language Models(https://arxiv.org/abs/2501.05468)
• Keywords: language model, llm, retrieval-augmented generation, agent
• Abstract: Systematic literature reviews and meta-analyses are essential for synthesizing research insights, but they remain time-intensive and labor-intensive due to the iterative processes of screening, evaluation, and data extraction. This paper introduces and evaluates LatteReview, a Python-based framework that leverages large language models (LLMs) and multi-agent systems to automate key elements of the systematic review process. Designed to streamline workflows while maintaining rigor, LatteReview utilizes modular agents for tasks such as title and abstract screening, relevance scoring, and structured data extraction. These agents operate within orchestrated workflows, supporting sequential and parallel review rounds, dynamic decision-making, and iterative refinement based on user feedback. LatteReview's architecture integrates LLM providers, enabling compatibility with both cloud-based and locally hosted models. The framework supports features such as Retrieval-Augmented Generation (RAG) for incorporating external context, multimodal reviews, Pydantic-based validation for structured inputs and outputs, and asynchronous programming for handling large-scale datasets. The framework is available on the GitHub repository, with detailed documentation and an installable package.
• 摘要：系统文献综述和荟萃分析对于综合研究见解至关重要，但由于筛选、评估和数据提取的迭代过程，它们仍然耗时耗力。本文介绍并评估了 LatteReview，这是一个基于 Python 的框架，利用大型语言模型 (LLM) 和多智能体系统来自动化系统评价过程的关键要素。LatteReview 旨在简化工作流程同时保持严谨性，它利用模块化智能体执行标题和摘要筛选、相关性评分和结构化数据提取等任务。这些智能体在精心策划的工作流程中运行，支持顺序和并行审查轮次、动态决策和基于用户反馈的迭代改进。LatteReview 的架构集成了 LLM 提供程序，可兼容基于云和本地托管的模型。该框架支持多种功能，例如用于整合外部上下文的检索增强生成 (RAG)、多模态评论、基于 Pydantic 的结构化输入和输出验证以及用于处理大规模数据集的异步编程。该框架在 GitHub 存储库中可用，并带有详细的文档和可安装包。

Title: Retrieval-Augmented Generation by Evidence Retroactivity in LLMs

• Authors: Liang Xiao, Wen Dai, Shuai Chen, Bin Qin, Chongyang Shi, Haopeng Jing, Tianyu Guo
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2501.05475
• Pdf URL: https://arxiv.org/pdf/2501.05475
• Copy Paste: [[2501.05475]] Retrieval-Augmented Generation by Evidence Retroactivity in LLMs(https://arxiv.org/abs/2501.05475)
• Keywords: llm, retrieval-augmented generation
• Abstract: Retrieval-augmented generation has gained significant attention due to its ability to integrate relevant external knowledge, enhancing the accuracy and reliability of the LLMs' responses. Most of the existing methods apply a dynamic multiple retrieval-generating process, to address multi-hop complex questions by decomposing them into sub-problems. However, these methods rely on an unidirectional forward reasoning paradigm, where errors from insufficient reasoning steps or inherent flaws in current retrieval systems are irreversible, potentially derailing the entire reasoning chain. For the first time, this work introduces Retroactive Retrieval-Augmented Generation (RetroRAG), a novel framework to build a retroactive reasoning paradigm. RetroRAG revises and updates the evidence, redirecting the reasoning chain to the correct direction. RetroRAG constructs an evidence-collation-discovery framework to search, generate, and refine credible evidence. It synthesizes inferential evidence related to the key entities in the question from the existing source knowledge and formulates search queries to uncover additional information. As new evidence is found, RetroRAG continually updates and organizes this information, enhancing its ability to locate further necessary evidence. Paired with an Answerer to generate and evaluate outputs, RetroRAG is capable of refining its reasoning process iteratively until a reliable answer is obtained. Empirical evaluations show that RetroRAG significantly outperforms existing methods.
• 摘要：检索增强生成因其能够整合相关的外部知识、提高 LLM 响应的准确性和可靠性而备受关注。现有的大多数方法都采用动态的多重检索生成过程，通过将多跳复杂问题分解为子问题来解决这些问题。然而，这些方法依赖于单向的前向推理范式，其中推理步骤不足或当前检索系统固有缺陷导致的错误是不可逆的，可能会破坏整个推理链。这项工作首次引入了追溯检索增强生成 (RetroRAG)，这是一种构建追溯推理范式的新颖框架。RetroRAG 会修改和更新证据，将推理链重定向到正确的方向。RetroRAG 构建了一个证据整理发现框架来搜索、生成和提炼可信证据。它从现有的源知识中综合与问题中的关键实体相关的推理证据，并制定搜索查询以发现更多信息。随着新证据的发现，RetroRAG 不断更新和组织这些信息，增强其查找进一步必要证据的能力。与 Answerer 配对以生成和评估输出，RetroRAG 能够迭代地改进其推理过程，直到获得可靠的答案。实证评估表明，RetroRAG 的表现明显优于现有方法。

Title: Language and Planning in Robotic Navigation: A Multilingual Evaluation of State-of-the-Art Models

• Authors: Malak Mansour, Ahmed Aly, Bahey Tharwat, Sarim Hashmi, Dong An, Ian Reid
• Subjects: cs.CL, cs.AI, cs.CV, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2501.05478
• Pdf URL: https://arxiv.org/pdf/2501.05478
• Copy Paste: [[2501.05478]] Language and Planning in Robotic Navigation: A Multilingual Evaluation of State-of-the-Art Models(https://arxiv.org/abs/2501.05478)
• Keywords: language model, gpt, llm, agent
• Abstract: Large Language Models (LLMs) such as GPT-4, trained on huge amount of datasets spanning multiple domains, exhibit significant reasoning, understanding, and planning capabilities across various tasks. This study presents the first-ever work in Arabic language integration within the Vision-and-Language Navigation (VLN) domain in robotics, an area that has been notably underexplored in existing research. We perform a comprehensive evaluation of state-of-the-art multi-lingual Small Language Models (SLMs), including GPT-4o mini, Llama 3 8B, and Phi-3 medium 14B, alongside the Arabic-centric LLM, Jais. Our approach utilizes the NavGPT framework, a pure LLM-based instruction-following navigation agent, to assess the impact of language on navigation reasoning through zero-shot sequential action prediction using the R2R dataset. Through comprehensive experiments, we demonstrate that our framework is capable of high-level planning for navigation tasks when provided with instructions in both English and Arabic. However, certain models struggled with reasoning and planning in the Arabic language due to inherent limitations in their capabilities, sub-optimal performance, and parsing issues. These findings highlight the importance of enhancing planning and reasoning capabilities in language models for effective navigation, emphasizing this as a key area for further development while also unlocking the potential of Arabic-language models for impactful real-world applications.
• 摘要：大型语言模型 (LLM)（例如 GPT-4）在大量跨多个领域的数据集上进行训练，在各种任务中表现出显著的推理、理解和规划能力。这项研究首次展示了在机器人视觉和语言导航 (VLN) 领域中将阿拉伯语集成到机器人领域的工作，这一领域在现有研究中尚未得到充分探索。我们对最先进的多语言小型语言模型 (SLM) 进行了全面评估，包括 GPT-4o mini、Llama 3 8B 和 Phi-3 medium 14B，以及以阿拉伯语为中心的 LLM Jais。我们的方法利用 NavGPT 框架（一种纯基于 LLM 的指令跟踪导航代理），通过使用 R2R 数据集的零样本顺序动作预测来评估语言对导航推理的影响。通过全面的实验，我们证明了我们的框架在提供英语和阿拉伯语指令时能够对导航任务进行高级规划。然而，由于能力的固有限制、次优性能和解析问题，某些模型在阿拉伯语推理和规划方面遇到了困难。这些发现凸显了增强语言模型的规划和推理能力对于有效导航的重要性，强调这是进一步发展的关键领域，同时也释放了阿拉伯语模型在现实世界中产生影响的潜力。

Title: Practical Design and Benchmarking of Generative AI Applications for Surgical Billing and Coding

• Authors: John C. Rollman (1), Bruce Rogers (1), Hamed Zaribafzadeh (1), Daniel Buckland (2), Ursula Rogers (1), Jennifer Gagnon (1), Ozanan Meireles (1), Lindsay Jennings (3), Jim Bennett (1), Jennifer Nicholson (3), Nandan Lad (4), Linda Cendales (1), Andreas Seas (4,5,6), Alessandro Martinino (6), E. Shelley Hwang (1), Allan D. Kirk (1)
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.05479
• Pdf URL: https://arxiv.org/pdf/2501.05479
• Copy Paste: [[2501.05479]] Practical Design and Benchmarking of Generative AI Applications for Surgical Billing and Coding(https://arxiv.org/abs/2501.05479)
• Keywords: language model, gpt, llm
• Abstract: Background: Healthcare has many manual processes that can benefit from automation and augmentation with Generative Artificial Intelligence (AI), the medical billing and coding process. However, current foundational Large Language Models (LLMs) perform poorly when tasked with generating accurate International Classification of Diseases, 10th edition, Clinical Modification (ICD-10-CM) and Current Procedural Terminology (CPT) codes. Additionally, there are many security and financial challenges in the application of generative AI to healthcare. We present a strategy for developing generative AI tools in healthcare, specifically for medical billing and coding, that balances accuracy, accessibility, and patient privacy. Methods: We fine tune the PHI-3 Mini and PHI-3 Medium LLMs using institutional data and compare the results against the PHI-3 base model, a PHI-3 RAG application, and GPT-4o. We use the post operative surgical report as input and the patients billing claim the associated ICD-10, CPT, and Modifier codes as the target result. Performance is measured by accuracy of code generation, proportion of invalid codes, and the fidelity of the billing claim format. Results: Both fine-tuned models performed better or as well as GPT-4o. The Phi-3 Medium fine-tuned model showed the best performance (ICD-10 Recall and Precision: 72%, 72%; CPT Recall and Precision: 77%, 79%; Modifier Recall and Precision: 63%, 64%). The Phi-3 Medium fine-tuned model only fabricated 1% of ICD-10 codes and 0.6% of CPT codes generated. Conclusions: Our study shows that a small model that is fine-tuned on domain-specific data for specific tasks using a simple set of open-source tools and minimal technological and monetary requirements performs as well as the larger contemporary consumer models.
• 摘要：背景：医疗保健有许多手动流程，这些流程可以从生成人工智能 (AI) 的自动化和增强中受益，即医疗计费和编码流程。然而，当前的基础大型语言模型 (LLM) 在生成准确的国际疾病分类第 10 版、临床修订版 (ICD-10-CM) 和现行程序术语 (CPT) 代码时表现不佳。此外，在将生成式 AI 应用于医疗保健方面还存在许多安全和财务挑战。我们提出了一种在医疗保健领域开发生成式 AI 工具的策略，特别是针对医疗计费和编码，以平衡准确性、可访问性和患者隐私。方法：我们使用机构数据对 PHI-3 Mini 和 PHI-3 Medium LLM 进行微调，并将结果与​​ PHI-3 基础模型、PHI-3 RAG 应用程序和 GPT-4o 进行比较。我们使用术后手术报告作为输入，患者账单索赔相关的 ICD-10、CPT 和修改器代码作为目标结果。性能通过代码生成的准确性、无效代码的比例和账单索赔格式的保真度来衡量。结果：两个微调模型的表现都比 GPT-4o 更好或一样好。Phi-3 Medium 微调模型表现出最佳性能（ICD-10 召回率和准确率：72%、72%；CPT 召回率和准确率：77%、79%；修改器召回率和准确率：63%、64%）。Phi-3 Medium 微调模型仅伪造了 1% 的 ICD-10 代码和 0.6% 的 CPT 代码。结论：我们的研究表明，使用一组简单的开源工具和最低的技术和资金要求针对特定任务针对特定领域数据进行微调的小型模型的表现与更大的当代消费者模型一样好。

Title: HP-BERT: A framework for longitudinal study of Hinduphobia on social media via LLMs

• Authors: Ashutosh Singh, Rohitash Chandra
• Subjects: cs.CL, cs.SI
• Abstract URL: https://arxiv.org/abs/2501.05482
• Pdf URL: https://arxiv.org/pdf/2501.05482
• Copy Paste: [[2501.05482]] HP-BERT: A framework for longitudinal study of Hinduphobia on social media via LLMs(https://arxiv.org/abs/2501.05482)
• Keywords: language model, llm
• Abstract: During the COVID-19 pandemic, community tensions intensified, fuelling Hinduphobic sentiments and discrimination against individuals of Hindu descent within India and worldwide. Large language models (LLMs) have become prominent in natural language processing (NLP) tasks and social media analysis, enabling longitudinal studies of platforms like X (formerly Twitter) for specific issues during COVID-19. We present an abuse detection and sentiment analysis framework that offers a longitudinal analysis of Hinduphobia on X (Twitter) during and after the COVID-19 pandemic. This framework assesses the prevalence and intensity of Hinduphobic discourse, capturing elements such as derogatory jokes and racist remarks through sentiment analysis and abuse detection from pre-trained and fine-tuned LLMs. Additionally, we curate and publish a "Hinduphobic COVID-19 X (Twitter) Dataset" of 8,000 tweets annotated for Hinduphobic abuse detection, which is used to fine-tune a BERT model, resulting in the development of the Hinduphobic BERT (HP-BERT) model. We then further fine-tune HP-BERT using the SenWave dataset for multi-label sentiment analysis. Our study encompasses approximately 27.4 million tweets from six countries, including Australia, Brazil, India, Indonesia, Japan, and the United Kingdom. Our findings reveal a strong correlation between spikes in COVID-19 cases and surges in Hinduphobic rhetoric, highlighting how political narratives, misinformation, and targeted jokes contributed to communal polarisation. These insights provide valuable guidance for developing strategies to mitigate communal tensions in future crises, both locally and globally. We advocate implementing automated monitoring and removal of such content on social media to curb divisive discourse.
• 摘要：在 COVID-19 疫情期间，社区紧张局势加剧，助长了印度教恐惧症情绪，并导致印度国内和全球范围内对印度教后裔的歧视。大型语言模型 (LLM) 在自然语言处理 (NLP) 任务和社交媒体分析中发挥了重要作用，使得对 X（以前称为 Twitter）等平台进行 COVID-19 期间特定问题的纵向研究成为可能。我们提出了一个滥用检测和情绪分析框架，该框架对 COVID-19 疫情期间和之后 X（Twitter）上的印度教恐惧症进行了纵向分析。该框架评估了印度教恐惧症言论的普遍性和强度，通过对预先训练和微调的 LLM 进行情绪分析和滥用检测，捕捉到贬损性笑话和种族主义言论等元素。此外，我们整理并发布了“印度恐惧症 COVID-19 X（推特）数据集”，其中包含 8,000 条注释为检测印度恐惧症辱骂的推文，用于微调 BERT 模型，从而开发出印度恐惧症 BERT（HP-BERT）模型。然后，我们使用 SenWave 数据集进一步微调 HP-BERT，以进行多标签情绪分析。我们的研究涵盖了来自澳大利亚、巴西、印度、印度尼西亚、日本和英国等六个国家的约 2740 万条推文。我们的研究结果显示，COVID-19 病例激增与印度恐惧症言论激增之间存在很强的相关性，突显了政治叙事、错误信息和有针对性的笑话如何加剧了社区两极分化。这些见解为制定缓解未来本地和全球危机中社区紧张局势的战略提供了宝贵的指导。我们主张在社交媒体上实施自动监控和删除此类内容，以遏制分裂言论。

Title: S2 Chunking: A Hybrid Framework for Document Segmentation Through Integrated Spatial and Semantic Analysis

• Authors: Prashant Verma
• Subjects: cs.CL, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.05485
• Pdf URL: https://arxiv.org/pdf/2501.05485
• Copy Paste: [[2501.05485]] S2 Chunking: A Hybrid Framework for Document Segmentation Through Integrated Spatial and Semantic Analysis(https://arxiv.org/abs/2501.05485)
• Keywords: language model
• Abstract: Document chunking is a critical task in natural language processing (NLP) that involves dividing a document into meaningful segments. Traditional methods often rely solely on semantic analysis, ignoring the spatial layout of elements, which is crucial for understanding relationships in complex documents. This paper introduces a novel hybrid approach that combines layout structure, semantic analysis, and spatial relationships to enhance the cohesion and accuracy of document chunks. By leveraging bounding box information (bbox) and text embeddings, our method constructs a weighted graph representation of document elements, which is then clustered using spectral clustering. Experimental results demonstrate that this approach outperforms traditional methods, particularly in documents with diverse layouts such as reports, articles, and multi-column designs. The proposed method also ensures that no chunk exceeds a specified token length, making it suitable for use cases where token limits are critical (e.g., language models with input size limitations)
• 摘要：文档分块是自然语言处理 (NLP) 中的一项关键任务，涉及将文档划分为有意义的片段。传统方法通常仅依赖于语义分析，而忽略了元素的空间布局，而这对于理解复杂文档中的关系至关重要。本文介绍了一种新颖的混合方法，该方法结合了布局结构、语义分析和空间关系，以增强文档块的凝聚力和准确性。通过利用边界框信息 (bbox) 和文本嵌入，我们的方法构建了文档元素的加权图形表示，然后使用谱聚类对其进行聚类。实验结果表明，这种方法优于传统方法，特别是在具有多种布局的文档（例如报告、文章和多列设计）中。所提出的方法还确保没有块超过指定的标记长度，使其适用于标记限制至关重要的用例（例如，具有输入大小限制的语言模型）

Title: The Future of AI: Exploring the Potential of Large Concept Models

• Authors: Hussain Ahmad, Diksha Goel
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.05487
• Pdf URL: https://arxiv.org/pdf/2501.05487
• Copy Paste: [[2501.05487]] The Future of AI: Exploring the Potential of Large Concept Models(https://arxiv.org/abs/2501.05487)
• Keywords: language model, gpt, llm, chat
• Abstract: The field of Artificial Intelligence (AI) continues to drive transformative innovations, with significant progress in conversational interfaces, autonomous vehicles, and intelligent content creation. Since the launch of ChatGPT in late 2022, the rise of Generative AI has marked a pivotal era, with the term Large Language Models (LLMs) becoming a ubiquitous part of daily life. LLMs have demonstrated exceptional capabilities in tasks such as text summarization, code generation, and creative writing. However, these models are inherently limited by their token-level processing, which restricts their ability to perform abstract reasoning, conceptual understanding, and efficient generation of long-form content. To address these limitations, Meta has introduced Large Concept Models (LCMs), representing a significant shift from traditional token-based frameworks. LCMs use concepts as foundational units of understanding, enabling more sophisticated semantic reasoning and context-aware decision-making. Given the limited academic research on this emerging technology, our study aims to bridge the knowledge gap by collecting, analyzing, and synthesizing existing grey literature to provide a comprehensive understanding of LCMs. Specifically, we (i) identify and describe the features that distinguish LCMs from LLMs, (ii) explore potential applications of LCMs across multiple domains, and (iii) propose future research directions and practical strategies to advance LCM development and adoption.
• 摘要：人工智能 (AI) 领域继续推动变革性创新，对话界面、自动驾驶汽车和智能内容创建方面取得了重大进展。自 2022 年底推出 ChatGPT 以来，生成式人工智能的兴起标志着一个关键时代的到来，术语“大型语言模型 (LLM)”已成为日常生活中无处不在的一部分。LLM 在文本摘要、代码生成和创意写作等任务中表现出色。然而，这些模型本质上受到其 token 级处理的限制，这限制了它们执行抽象推理、概念理解和高效生成长篇内容的能力。为了解决这些限制，Meta 引入了大型概念模型 (LCM)，代表了与传统基于 token 的框架的重大转变。LCM 使用概念作为理解的基础单元，实现更复杂的语义推理和情境感知决策。鉴于对这项新兴技术的学术研究有限，我们的研究旨在通过收集、分析和综合现有的灰色文献来弥补知识差距，以提供对 LCM 的全面了解。具体来说，我们 (i) 识别并描述区分 LCM 与 LLM 的特征，(ii) 探索 LCM 在多个领域的潜在应用，以及 (iii) 提出未来的研究方向和实用策略，以推进 LCM 的开发和采用。

Title: Spatial Information Integration in Small Language Models for Document Layout Generation and Classification

• Authors: Pablo Melendez, Clemens Havas
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2501.05497
• Pdf URL: https://arxiv.org/pdf/2501.05497
• Copy Paste: [[2501.05497]] Spatial Information Integration in Small Language Models for Document Layout Generation and Classification(https://arxiv.org/abs/2501.05497)
• Keywords: language model
• Abstract: Document layout understanding is a field of study that analyzes the spatial arrangement of information in a document hoping to understand its structure and layout. Models such as LayoutLM (and its subsequent iterations) can understand semi-structured documents with SotA results; however, the lack of open semi-structured data is a limitation in itself. While semi-structured data is common in everyday life (balance sheets, purchase orders, receipts), there is a lack of public datasets for training machine learning models for this type of document. In this investigation we propose a method to generate new, synthetic, layout information that can help overcoming this data shortage. According to our results, the proposed method performs better than LayoutTransformer, another popular layout generation method. We also show that, in some scenarios, text classification can improve when supported by bounding box information.
• 摘要：文档布局理解是一门研究领域，它分析文档中信息的空间排列，希望了解其结构和布局。诸如 LayoutLM（及其后续迭代）之类的模型可以理解半结构化文档并获得 SotA 结果；然而，缺乏开放的半结构化数据本身就是一个限制。虽然半结构化数据在日常生活中很常见（资产负债表、采购订单、收据），但缺乏用于训练此类文档的机器学习模型的公共数据集。在本研究中，我们提出了一种生成新的、合成的布局信息的方法，可以帮助克服这种数据短缺的问题。根据我们的结果，所提出的方法比另一种流行的布局生成方法 LayoutTransformer 表现更好。我们还表明，在某些情况下，当有边界框信息支持时，文本分类可以得到改善。

Title: The more polypersonal the better -- a short look on space geometry of fine-tuned layers

• Authors: Sergei Kudriashov, Veronika Zykova, Angelina Stepanova, Yakov Raskind, Eduard Klyshinsky
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.05503
• Pdf URL: https://arxiv.org/pdf/2501.05503
• Copy Paste: [[2501.05503]] The more polypersonal the better -- a short look on space geometry of fine-tuned layers(https://arxiv.org/abs/2501.05503)
• Keywords: language model
• Abstract: The interpretation of deep learning models is a rapidly growing field, with particular interest in language models. There are various approaches to this task, including training simpler models to replicate neural network predictions and analyzing the latent space of the model. The latter method allows us to not only identify patterns in the model's decision-making process, but also understand the features of its internal structure. In this paper, we analyze the changes in the internal representation of the BERT model when it is trained with additional grammatical modules and data containing new grammatical structures (polypersonality). We find that adding a single grammatical layer causes the model to separate the new and old grammatical systems within itself, improving the overall performance on perplexity metrics.
• 摘要：深度学习模型的解释是一个快速发展的领域，语言模型尤为受人关注。这项任务有多种方法，包括训练更简单的模型来复制神经网络预测和分析模型的潜在空间。后一种方法不仅让我们能够识别模型决策过程中的模式，还能让我们了解其内部结构的特征。在本文中，我们分析了在使用额外的语法模块和包含新语法结构（多重人格）的数据训练 BERT 模型时，其内部表示的变化。我们发现，添加单个语法层会导致模型在其内部分离新旧语法系统，从而提高困惑度指标的整体性能。

Title: The dynamics of meaning through time: Assessment of Large Language Models

• Authors: Mohamed Taher Alrefaie, Fatty Salem, Nour Eldin Morsy, Nada Samir, Mohamed Medhat Gaber
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.05552
• Pdf URL: https://arxiv.org/pdf/2501.05552
• Copy Paste: [[2501.05552]] The dynamics of meaning through time: Assessment of Large Language Models(https://arxiv.org/abs/2501.05552)
• Keywords: language model, gpt, llm, prompt, chat
• Abstract: Understanding how large language models (LLMs) grasp the historical context of concepts and their semantic evolution is essential in advancing artificial intelligence and linguistic studies. This study aims to evaluate the capabilities of various LLMs in capturing temporal dynamics of meaning, specifically how they interpret terms across different time periods. We analyze a diverse set of terms from multiple domains, using tailored prompts and measuring responses through both objective metrics (e.g., perplexity and word count) and subjective human expert evaluations. Our comparative analysis includes prominent models like ChatGPT, GPT-4, Claude, Bard, Gemini, and Llama. Findings reveal marked differences in each model's handling of historical context and semantic shifts, highlighting both strengths and limitations in temporal semantic understanding. These insights offer a foundation for refining LLMs to better address the evolving nature of language, with implications for historical text analysis, AI design, and applications in digital humanities.
• 摘要：了解大型语言模型 (LLM) 如何掌握概念的历史背景及其语义演变对于推动人工智能和语言学研究至关重要。本研究旨在评估各种 LLM 捕捉意义的时间动态的能力，特别是它们如何解释不同时间段的术语。我们使用定制的提示分析来自多个领域的各种术语，并通过客观指标（例如困惑度和字数）和主观人类专家评估来衡量响应。我们的比较分析包括 ChatGPT、GPT-4、Claude、Bard、Gemini 和 Llama 等著名模型。研究结果显示，每个模型在处理历史背景和语义变化方面存在明显差异，突出了时间语义理解的优势和局限性。这些见解为改进 LLM 以更好地应对语言的演变性质奠定了基础，对历史文本分析、人工智能设计和数字人文应用具有重要意义。

Title: LLMQuoter: Enhancing RAG Capabilities Through Efficient Quote Extraction From Large Contexts

• Authors: Yuri Facanha Bezerra, Li Weigang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.05554
• Pdf URL: https://arxiv.org/pdf/2501.05554
• Copy Paste: [[2501.05554]] LLMQuoter: Enhancing RAG Capabilities Through Efficient Quote Extraction From Large Contexts(https://arxiv.org/abs/2501.05554)
• Keywords: language model, llm, retrieval augmented generation
• Abstract: We introduce LLMQuoter, a lightweight, distillation-based model designed to enhance Retrieval Augmented Generation (RAG) by extracting the most relevant textual evidence for downstream reasoning tasks. Built on the LLaMA-3B architecture and fine-tuned with Low-Rank Adaptation (LoRA) on a 15,000-sample subset of HotpotQA, LLMQuoter adopts a "quote-first-then-answer" strategy, efficiently identifying key quotes before passing curated snippets to reasoning models. This workflow reduces cognitive overhead and outperforms full-context approaches like Retrieval-Augmented Fine-Tuning (RAFT), achieving over 20-point accuracy gains across both small and large language models. By leveraging knowledge distillation from a high-performing teacher model, LLMQuoter achieves competitive results in a resource-efficient fine-tuning setup. It democratizes advanced RAG capabilities, delivering significant performance improvements without requiring extensive model retraining. Our results highlight the potential of distilled quote-based reasoning to streamline complex workflows, offering a scalable and practical solution for researchers and practitioners alike.
• 摘要：我们推出了 LLMQuoter，这是一种轻量级的基于蒸馏的模型，旨在通过提取最相关的文本证据用于下游推理任务来增强检索增强生成 (RAG)。LLMQuoter 基于 LLaMA-3B 架构构建，并在 15,000 个 HotpotQA 样本子集上使用低秩自适应 (LoRA) 进行微调，采用“先引用后回答”策略，在将精选的片段传递给推理模型之前有效识别关键引用。此工作流程减少了认知开销，并且优于检索增强微调 (RAFT) 等全上下文方法，在小型和大型语言模型中实现了超过 20 分的准确率提升。通过利用高性能教师模型的知识蒸馏，LLMQuoter 在资源高效的微调设置中取得了有竞争力的结果。它使高级 RAG 功能大众化，无需大量模型再训练即可显着提高性能。我们的研究结果强调了基于提炼引文的推理在简化复杂工作流程方面的潜力，为研究人员和从业人员提供了可扩展且实用的解决方案。

Title: Exploring Large Language Models for Translating Romanian Computational Problems into English

• Authors: Adrian Marius Dumitran, Adrian-Catalin Badea, Stefan-Gabriel Muscalu, Angela-Liliana Dumitran, Stefan-Cosmin Dascalescu, Radu-Sebastian Amarie
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.05601
• Pdf URL: https://arxiv.org/pdf/2501.05601
• Copy Paste: [[2501.05601]] Exploring Large Language Models for Translating Romanian Computational Problems into English(https://arxiv.org/abs/2501.05601)
• Keywords: language model, gpt, llm, prompt
• Abstract: Recent studies have suggested that large language models (LLMs) underperform on mathematical and computer science tasks when these problems are translated from Romanian into English, compared to their original Romanian format. Accurate translation is critical for applications ranging from automatic translations in programming competitions to the creation of high-quality educational materials, as well as minimizing errors or fraud in human translations. This study shows that robust large language models (LLMs) can maintain or even enhance their performance in translating less common languages when given well-structured prompts. Our findings suggest that LLMs, with appropriate supervision, can be reliably used for the automatic translation of IOI (International Olympiad in Informatics)-style tasks. We evaluate several translation methods across multiple LLMs, including OpenRoLLM, Llama 3.1 8B, Llama 3.2 3B and GPT-4o, assessing their translation accuracy and performance stability through repeated runs. Additionally, we augment the OJI (Romanian County-Level Informatics Olympiad) Romanian dataset with accurate English translations, enhancing its utility for future LLM training and evaluation. Through detailed syntactic and semantic analyses, we confirm that with human oversight, LLMs can serve as a viable solution for multilingual problem-solving. We also compare the translation quality of LLMs against human translators, as evaluated by a certified expert, underscoring the potential of LLMs in realworld scenarios.
• 摘要：最近的研究表明，与原始罗马尼亚语格式相比，当将这些问题从罗马尼亚语翻译成英语时，大型语言模型 (LLM) 在数学和计算机科学任务上的表现不佳。准确的翻译对于从编程竞赛中的自动翻译到高质量教育材料的创建以及最大限度地减少人工翻译中的错误或欺诈等应用都至关重要。这项研究表明，在给出结构良好的提示时，强大的大型语言模型 (LLM) 可以在翻译不太常见的语言时保持甚至提高其性能。我们的研究结果表明，在适当的监督下，LLM 可以可靠地用于 IOI（国际信息学奥林匹克）式任务的自动翻译。我们评估了多个 LLM 中的几种翻译方法，包括 OpenRoLLM、Llama 3.1 8B、Llama 3.2 3B 和 GPT-4o，通过重复运行评估它们的翻译准确性和性能稳定性。此外，我们还为罗马尼亚县级信息学奥林匹克竞赛 (OJI) 罗马尼亚语数据集添加了准确的英语翻译，从而增强了其在未来 LLM 培训和评估中的实用性。通过详细的句法和语义分析，我们确认，在人工监督下，LLM 可以成为解决多语言问题的可行解决方案。我们还将 LLM 的翻译质量与人工翻译进行比较，并由认证专家进行评估，强调了 LLM 在现实世界场景中的潜力。

Title: The Impact of Model Scaling on Seen and Unseen Language Performance

• Authors: Rhitabrat Pokharel, Sina Bagheri Nezhad, Ameeta Agrawal, Suresh Singh
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.05629
• Pdf URL: https://arxiv.org/pdf/2501.05629
• Copy Paste: [[2501.05629]] The Impact of Model Scaling on Seen and Unseen Language Performance(https://arxiv.org/abs/2501.05629)
• Keywords: language model, llm
• Abstract: The rapid advancement of Large Language Models (LLMs), particularly those trained on multilingual corpora, has intensified the need for a deeper understanding of their performance across a diverse range of languages and model sizes. Our research addresses this critical need by studying the performance and scaling behavior of multilingual LLMs in text classification and machine translation tasks across 204 languages. We systematically examine both seen and unseen languages across three model families of varying sizes in zero-shot and few-shot settings. Our findings show significant differences in scaling behavior between zero-shot and two-shot scenarios, with striking disparities in performance between seen and unseen languages. Model scale has little effect on zero-shot performance, which remains mostly flat. However, in two-shot settings, larger models show clear linear improvements in multilingual text classification. For translation tasks, however, only the instruction-tuned model showed clear benefits from scaling. Our analysis also suggests that overall resource levels, not just the proportions of pretraining languages, are better predictors of model performance, shedding light on what drives multilingual LLM effectiveness.
• 摘要：大型语言模型 (LLM) 的快速发展，尤其是在多语言语料库上训练的大型语言模型，加剧了人们对其在各种语言和模型大小中的表现有更深入的了解的需求。我们的研究通过研究多语言 LLM 在 204 种语言的文本分类和机器翻译任务中的性能和扩展行为来解决这一关键需求。我们在零样本和少样本设置中系统地检查了三个不同大小的模型系列中的可见和不可见语言。我们的研究结果显示，零样本和两样本场景之间的扩展行为存在显著差异，可见和不可见语言之间的性能存在显著差异。模型规模对零样本性能影响不大，基本保持不变。然而，在两样本设置中，较大的模型在多语言文本分类中显示出明显的线性改进。然而，对于翻译任务，只有指令调整模型显示出扩展的明显好处。我们的分析还表明，整体资源水平，而不仅仅是预训练语言的比例，是模型性能的更好预测因素，这揭示了驱动多语言 LLM 有效性的因素。

Title: Automating Date Format Detection for Data Visualization

• Authors: Zixuan Liang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.05640
• Pdf URL: https://arxiv.org/pdf/2501.05640
• Copy Paste: [[2501.05640]] Automating Date Format Detection for Data Visualization(https://arxiv.org/abs/2501.05640)
• Keywords: language model
• Abstract: Data preparation, specifically date parsing, is a significant bottleneck in analytic workflows. To address this, we present two algorithms, one based on minimum entropy and the other on natural language modeling that automatically derive date formats from string data. These algorithms achieve over 90% accuracy on a large corpus of data columns, streamlining the data preparation process within visualization environments. The minimal entropy approach is particularly fast, providing interactive feedback. Our methods simplify date format extraction, making them suitable for integration into data visualization tools and databases.
• 摘要：数据准备，特别是日期解析，是分析工作流中的一个重要瓶颈。为了解决这个问题，我们提出了两种算法，一种基于最小熵，另一种基于自然语言建模，可自动从字符串数据中得出日期格式。这些算法在大量数据列上实现了超过 90% 的准确率，简化了可视化环境中的数据准备过程。最小熵方法特别快，提供交互式反馈。我们的方法简化了日期格式提取，使其适合集成到数据可视化工具和数据库中。

Title: Iconicity in Large Language Models

• Authors: Anna Marklová, Jiří Milička, Leonid Ryvkin, Ľudmila Lacková Bennet, Libuše Kormaníková
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.05643
• Pdf URL: https://arxiv.org/pdf/2501.05643
• Copy Paste: [[2501.05643]] Iconicity in Large Language Models(https://arxiv.org/abs/2501.05643)
• Keywords: language model, gpt, llm
• Abstract: Lexical iconicity, a direct relation between a word's meaning and its form, is an important aspect of every natural language, most commonly manifesting through sound-meaning associations. Since Large language models' (LLMs') access to both meaning and sound of text is only mediated (meaning through textual context, sound through written representation, further complicated by tokenization), we might expect that the encoding of iconicity in LLMs would be either insufficient or significantly different from human processing. This study addresses this hypothesis by having GPT-4 generate highly iconic pseudowords in artificial languages. To verify that these words actually carry iconicity, we had their meanings guessed by Czech and German participants (n=672) and subsequently by LLM-based participants (generated by GPT-4 and Claude 3.5 Sonnet). The results revealed that humans can guess the meanings of pseudowords in the generated iconic language more accurately than words in distant natural languages and that LLM-based participants are even more successful than humans in this task. This core finding is accompanied by several additional analyses concerning the universality of the generated language and the cues that both human and LLM-based participants utilize.
• 摘要：词汇象似性是词义与形式之间的直接关系，是每种自然语言的一个重要方面，最常见的表现形式是声音-意义关联。由于大型语言模型 (LLM) 对文本含义和声音的访问仅是间接的（含义通过文本上下文，声音通过书面表示，通过标记化进一步复杂化），我们可以预期 LLM 中象似性的编码要么不足，要么与人类处理有显著不同。本研究通过让 GPT-4 在人工语言中生成高度象似性的伪词来解决这一假设。为了验证这些词确实具有象似性，我们让捷克和德国参与者（n=672）猜测它们的含义，随后让基于 LLM 的参与者（由 GPT-4 和 Claude 3.5 Sonnet 生成）猜测它们的含义。结果表明，人类可以比远距离自然语言中的单词更准确地猜测生成的象似语言中伪词的含义，并且基于 LLM 的参与者在这项任务中比人类更成功。这一核心发现伴随着几项额外的分析，涉及生成的语言的普遍性以及人类和 LLM 参与者所使用的线索。

Title: Cascaded Self-Evaluation Augmented Training for Efficient Multimodal Large Language Models

• Authors: Zheqi Lv, Wenkai Wang, Jiawei Wang, Shengyu Zhang, Fei Wu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.05662
• Pdf URL: https://arxiv.org/pdf/2501.05662
• Copy Paste: [[2501.05662]] Cascaded Self-Evaluation Augmented Training for Efficient Multimodal Large Language Models(https://arxiv.org/abs/2501.05662)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: Efficient Multimodal Large Language Models (EMLLMs) have rapidly advanced recently. Incorporating Chain-of-Thought (CoT) reasoning and step-by-step self-evaluation has improved their performance. However, limited parameters often hinder EMLLMs from effectively using self-evaluation during inference. Key challenges include synthesizing evaluation data, determining its quantity, optimizing training and inference strategies, and selecting appropriate prompts. To address these issues, we introduce Self-Evaluation Augmented Training (SEAT). SEAT uses more powerful EMLLMs for CoT reasoning, data selection, and evaluation generation, then trains EMLLMs with the synthesized data. However, handling long prompts and maintaining CoT reasoning quality are problematic. Therefore, we propose Cascaded Self-Evaluation Augmented Training (Cas-SEAT), which breaks down lengthy prompts into shorter, task-specific cascaded prompts and reduces costs for resource-limited settings. During data synthesis, we employ open-source 7B-parameter EMLLMs and annotate a small dataset with short prompts. Experiments demonstrate that Cas-SEAT significantly boosts EMLLMs' self-evaluation abilities, improving performance by 19.68%, 55.57%, and 46.79% on the MathVista, Math-V, and We-Math datasets, respectively. Additionally, our Cas-SEAT Dataset serves as a valuable resource for future research in enhancing EMLLM self-evaluation.
• 摘要：高效的多模态大型语言模型 (EMLLM) 近来发展迅速。结合思路链 (CoT) 推理和逐步自我评估提高了它们的性能。然而，有限的参数通常会阻碍 EMLLM 在推理过程中有效地使用自我评估。关键挑战包括合成评估数据、确定其数量、优化训练和推理策略以及选择合适的提示。为了解决这些问题，我们引入了自我评估增强训练 (SEAT)。SEAT 使用更强大的 EMLLM 进行 CoT 推理、数据选择和评估生成，然后使用合成数据训练 EMLLM。然而，处理长提示和保持 CoT 推理质量是有问题的。因此，我们提出了级联自我评估增强训练 (Cas-SEAT)，它将长提示分解为更短的、特定于任务的级联提示，并降低了资源有限设置的成本。在数据合成过程中，我们使用开源的 7B 参数 EMLLM，并用简短的提示注释一个小数据集。实验表明，Cas-SEAT 显著提高了 EMLLM 的自我评估能力，在 MathVista、Math-V 和 We-Math 数据集上的性能分别提高了 19.68%、55.57% 和 46.79%。此外，我们的 Cas-SEAT 数据集是未来研究增强 EMLLM 自我评估的宝贵资源。

Title: Linguistic Entity Masking to Improve Cross-Lingual Representation of Multilingual Language Models for Low-Resource Languages

• Authors: Aloka Fernando, Surangika Ranathunga
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.05700
• Pdf URL: https://arxiv.org/pdf/2501.05700
• Copy Paste: [[2501.05700]] Linguistic Entity Masking to Improve Cross-Lingual Representation of Multilingual Language Models for Low-Resource Languages(https://arxiv.org/abs/2501.05700)
• Keywords: language model
• Abstract: Multilingual Pre-trained Language models (multiPLMs), trained on the Masked Language Modelling (MLM) objective are commonly being used for cross-lingual tasks such as bitext mining. However, the performance of these models is still suboptimal for low-resource languages (LRLs). To improve the language representation of a given multiPLM, it is possible to further pre-train it. This is known as continual pre-training. Previous research has shown that continual pre-training with MLM and subsequently with Translation Language Modelling (TLM) improves the cross-lingual representation of multiPLMs. However, during masking, both MLM and TLM give equal weight to all tokens in the input sequence, irrespective of the linguistic properties of the tokens. In this paper, we introduce a novel masking strategy, Linguistic Entity Masking (LEM) to be used in the continual pre-training step to further improve the cross-lingual representations of existing multiPLMs. In contrast to MLM and TLM, LEM limits masking to the linguistic entity types nouns, verbs and named entities, which hold a higher prominence in a sentence. Secondly, we limit masking to a single token within the linguistic entity span thus keeping more context, whereas, in MLM and TLM, tokens are masked randomly. We evaluate the effectiveness of LEM using three downstream tasks, namely bitext mining, parallel data curation and code-mixed sentiment analysis using three low-resource language pairs English-Sinhala, English-Tamil, and Sinhala-Tamil. Experiment results show that continually pre-training a multiPLM with LEM outperforms a multiPLM continually pre-trained with MLM+TLM for all three tasks.
• 摘要：在掩蔽语言建模 (MLM) 目标上训练的多语言预训练语言模型 (multiPLM) 通常用于双语挖掘等跨语言任务。然而，这些模型的性能对于低资源语言 (LRL) 来说仍然不是最优的。为了改进给定多PLM 的语言表示，可以进一步对其进行预训练。这称为持续预训练。先前的研究表明，使用 MLM 进行持续预训练，随后使用翻译语言建模 (TLM) 进行预训练，可以改进多PLM 的跨语言表示。然而，在掩蔽期间，MLM 和 TLM 都会给予输入序列中的所有标记相同的权重，而不管标记的语言属性如何。在本文中，我们介绍了一种新颖的掩蔽策略，即语言实体掩蔽 (LEM)，用于持续预训练步骤，以进一步改进现有多PLM 的跨语言表示。与 MLM 和 TLM 相比，LEM 将掩码限制在语言实体类型名词、动词和命名实体上，这些实体在句子中占据更重要的地位。其次，我们将掩码限制在语言实体范围内的单个标记上，从而保留更多上下文，而在 MLM 和 TLM 中，标记是随机掩码的。我们使用三个下游任务评估 LEM 的有效性，即双语挖掘、并行数据管理和代码混合情感分析，使用三种资源较少的语言对英语-僧伽罗语、英语-泰米尔语和僧伽罗语-泰米尔语。实验结果表明，对于这三个任务，使用 LEM 持续预训练多 PLM 的效果优于使用 MLM+TLM 持续预训练的多 PLM。

Title: Multiagent Finetuning: Self Improvement with Diverse Reasoning Chains

• Authors: Vighnesh Subramaniam, Yilun Du, Joshua B. Tenenbaum, Antonio Torralba, Shuang Li, Igor Mordatch
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.05707
• Pdf URL: https://arxiv.org/pdf/2501.05707
• Copy Paste: [[2501.05707]] Multiagent Finetuning: Self Improvement with Diverse Reasoning Chains(https://arxiv.org/abs/2501.05707)
• Keywords: language model, llm, agent
• Abstract: Large language models (LLMs) have achieved remarkable performance in recent years but are fundamentally limited by the underlying training data. To improve models beyond the training data, recent works have explored how LLMs can be used to generate synthetic data for autonomous self-improvement. However, successive steps of self-improvement can reach a point of diminishing returns. In this work, we propose a complementary approach towards self-improvement where finetuning is applied to a multiagent society of language models. A group of language models, all starting from the same base model, are independently specialized by updating each one using data generated through multiagent interactions among the models. By training each model on independent sets of data, we illustrate how this approach enables specialization across models and diversification over the set of models. As a result, our overall system is able to preserve diverse reasoning chains and autonomously improve over many more rounds of fine-tuning than single-agent self-improvement methods. We quantitatively illustrate the efficacy of the approach across a wide suite of reasoning tasks.
• 摘要：大型语言模型 (LLM) 近年来取得了显著的表现，但从根本上受到底层训练数据的限制。为了在训练数据之外改进模型，最近的研究探索了如何使用 LLM 生成合成数据以实现自主自我改进。然而，自我改进的连续步骤可能会达到收益递减点。在这项工作中，我们提出了一种自我改进的补充方法，其中将微调应用于多智能体语言模型社会。一组语言模型都从同一个基础模型开始，通过使用模型之间多智能体交互生成的数据更新每个模型，从而独立地进行专业化。通过在独立的数据集上训练每个模型，我们说明了这种方法如何实现跨模型的专业化和模型集的多样化。因此，与单智能体自我改进方法相比，我们的整体系统能够保留多样化的推理链，并在更多轮微调中自主改进。我们定量说明了该方法在一系列推理任务中的有效性。

Title: Multi-Step Reasoning in Korean and the Emergent Mirage

• Authors: Guijin Son, Hyunwoo Ko, Dasol Choi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.05712
• Pdf URL: https://arxiv.org/pdf/2501.05712
• Copy Paste: [[2501.05712]] Multi-Step Reasoning in Korean and the Emergent Mirage(https://arxiv.org/abs/2501.05712)
• Keywords: language model, llm
• Abstract: We introduce HRMCR (HAE-RAE Multi-Step Commonsense Reasoning), a benchmark designed to evaluate large language models' ability to perform multi-step reasoning in culturally specific contexts, focusing on Korean. The questions are automatically generated via templates and algorithms, requiring LLMs to integrate Korean cultural knowledge into sequential reasoning steps. Consistent with prior observations on emergent abilities, our experiments reveal that models trained on fewer than \(2 \cdot 10^{25}\) training FLOPs struggle to solve any questions, showing near-zero performance. Beyond this threshold, performance improves sharply. State-of-the-art models (e.g., O1) still score under 50\%, underscoring the difficulty of our tasks. Notably, stepwise analysis suggests the observed emergent behavior may stem from compounding errors across multiple steps rather than reflecting a genuinely new capability. We publicly release the benchmark and commit to regularly updating the dataset to prevent contamination.
• 摘要：我们引入了 HRMCR（HAE-RAE 多步常识推理），这是一个旨在评估大型语言模型在特定文化背景下执行多步推理的能力的基准，重点关注韩语。问题通过模板和算法自动生成，需要 LLM 将韩国文化知识融入连续的推理步骤中。与之前对新兴能力的观察一致，我们的实验表明，在少于 \(2 \cdot 10^{25}\) 个训练 FLOP 上训练的模型难以解决任何问题，性能接近于零。超过此阈值，性能会大幅提高。最先进的模型（例如 O1）得分仍然低于 50\%，这突显了我们任务的难度。值得注意的是，逐步分析表明观察到的新兴行为可能源于多个步骤中的复合错误，而不是反映真正的新功能。我们公开发布基准并承诺定期更新数据集以防止污染。

Title: How to Enable Effective Cooperation Between Humans and NLP Models: A Survey of Principles, Formalizations, and Beyond

• Authors: Chen Huang, Yang Deng, Wenqiang Lei, Jiancheng Lv, Tat-Seng Chua, Jimmy Xiangji Huang
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2501.05714
• Pdf URL: https://arxiv.org/pdf/2501.05714
• Copy Paste: [[2501.05714]] How to Enable Effective Cooperation Between Humans and NLP Models: A Survey of Principles, Formalizations, and Beyond(https://arxiv.org/abs/2501.05714)
• Keywords: language model, llm, agent
• Abstract: With the advancement of large language models (LLMs), intelligent models have evolved from mere tools to autonomous agents with their own goals and strategies for cooperating with humans. This evolution has birthed a novel paradigm in NLP, i.e., human-model cooperation, that has yielded remarkable progress in numerous NLP tasks in recent years. In this paper, we take the first step to present a thorough review of human-model cooperation, exploring its principles, formalizations, and open challenges. In particular, we introduce a new taxonomy that provides a unified perspective to summarize existing approaches. Also, we discuss potential frontier areas and their corresponding challenges. We regard our work as an entry point, paving the way for more breakthrough research in this regard.
• 摘要：随着大型语言模型 (LLM) 的发展，智能模型已从单纯的工具演变为具有自己的目标和与人类合作的策略的自主代理。这种演变催生了 NLP 中的一种新范式，即人机合作，近年来，这种范式在众多 NLP 任务中取得了显著进展。在本文中，我们迈出了第一步，全面回顾了人机合作，探讨了其原理、形式化和未解决的挑战。特别是，我们引入了一种新的分类法，它提供了一个统一的视角来总结现有的方法。此外，我们还讨论了潜在的前沿领域及其相应的挑战。我们将我们的工作视为切入点，为这方面的更多突破性研究铺平了道路。

Title: Enabling Scalable Oversight via Self-Evolving Critic

• Authors: Zhengyang Tang, Ziniu Li, Zhenyang Xiao, Tian Ding, Ruoyu Sun, Benyou Wang, Dayiheng Liu, Fei Huang, Tianyu Liu, Bowen Yu, Junyang Lin
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.05727
• Pdf URL: https://arxiv.org/pdf/2501.05727
• Copy Paste: [[2501.05727]] Enabling Scalable Oversight via Self-Evolving Critic(https://arxiv.org/abs/2501.05727)
• Keywords: language model, llm
• Abstract: Despite their remarkable performance, the development of Large Language Models (LLMs) faces a critical challenge in scalable oversight: providing effective feedback for tasks where human evaluation is difficult or where LLMs outperform humans. While there is growing interest in using LLMs for critique, current approaches still rely on human annotations or more powerful models, leaving the issue of enhancing critique capabilities without external supervision unresolved. We introduce SCRIT (Self-evolving CRITic), a framework that enables genuine self-evolution of critique abilities. Technically, SCRIT self-improves by training on synthetic data, generated by a contrastive-based self-critic that uses reference solutions for step-by-step critique, and a self-validation mechanism that ensures critique quality through correction outcomes. Implemented with Qwen2.5-72B-Instruct, one of the most powerful LLMs, SCRIT achieves up to a 10.3\% improvement on critique-correction and error identification benchmarks. Our analysis reveals that SCRIT's performance scales positively with data and model size, outperforms alternative approaches, and benefits critically from its self-validation component.
• 摘要：尽管大型语言模型 (LLM) 性能卓越，但其开发在可扩展监督方面面临着一个关键挑战：为难以进行人工评估或 LLM 表现优于人类的任务提供有效反馈。尽管人们对使用 LLM 进行批评的兴趣日益浓厚，但当前的方法仍然依赖于人工注释或更强大的模型，而没有外部监督的情况下增强批评能力的问题仍未得到解决。我们引入了 SCRIT（自我进化 CRITic），这是一个能够真正自我进化批评能力的框架。从技术上讲，SCRIT 通过对合成数据进行训练进行自我改进，这些数据由基于对比的自我批评生成，该批评使用参考解决方案进行逐步批评，并采用自我验证机制通过纠正结果确保批评质量。使用最强大的 LLM 之一 Qwen2.5-72B-Instruct 实现，SCRIT 在批评纠正和错误识别基准上实现了高达 10.3% 的改进。我们的分析表明，SCRIT 的性能与数据和模型大小呈正相关，优于其他方法，并从其自我验证组件中受益匪浅。

Title: Controlling Large Language Models Through Concept Activation Vectors

• Authors: Hanyu Zhang, Xiting Wang, Chengao Li, Xiang Ao, Qing He
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.05764
• Pdf URL: https://arxiv.org/pdf/2501.05764
• Copy Paste: [[2501.05764]] Controlling Large Language Models Through Concept Activation Vectors(https://arxiv.org/abs/2501.05764)
• Keywords: language model, llm
• Abstract: As large language models (LLMs) are widely deployed across various domains, the ability to control their generated outputs has become more critical. This control involves aligning LLMs outputs with human values and ethical principles or customizing LLMs on specific topics or styles for individual users. Existing controlled generation methods either require significant computational resources and extensive trial-and-error or provide coarse-grained control. In this paper, we propose Generation with Concept Activation Vector (GCAV), a lightweight model control framework that ensures accurate control without requiring resource-extensive fine-tuning. Specifically, GCAV first trains a concept activation vector for specified concepts to be controlled, such as toxicity. During inference, GCAV steers the concept vector in LLMs, for example, by removing the toxicity concept vector from the activation layers. Control experiments from different perspectives, including toxicity reduction, sentiment control, linguistic style, and topic control, demonstrate that our framework achieves state-of-the-art performance with granular control, allowing for fine-grained adjustments of both the steering layers and the steering magnitudes for individual samples.
• 摘要：随着大型语言模型 (LLM) 广泛应用于各个领域，控制其生成输出的能力变得更加重要。这种控制包括使 LLM 输出与人类价值观和道德原则保持一致，或针对特定主题或风格为个人用户定制 LLM。现有的受控生成方法要么需要大量计算资源和大量反复试验，要么提供粗粒度的控制。在本文中，我们提出了概念激活向量生成 (GCAV)，这是一种轻量级模型控制框架，可确保准确控制而无需大量资源的微调。具体而言，GCAV 首先为要控制的指定概念（例如毒性）训练概念激活向量。在推理过程中，GCAV 会控制 LLM 中的概念向量，例如，通过从激活层中删除毒性概念向量。从不同角度进行的控制实验，包括毒性降低、情绪控制、语言风格和主题控制，表明我们的框架通过精细控制实现了最先进的性能，允许对单个样本的控制层和控制幅度进行细粒度的调整。

Title: Migician: Revealing the Magic of Free-Form Multi-Image Grounding in Multimodal Large Language Models

• Authors: You Li, Heyu Huang, Chi Chen, Kaiyu Huang, Chao Huang, Zonghao Guo, Zhiyuan Liu, Jinan Xu, Yuhua Li, Ruixuan Li, Maosong Sun
• Subjects: cs.CL, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2501.05767
• Pdf URL: https://arxiv.org/pdf/2501.05767
• Copy Paste: [[2501.05767]] Migician: Revealing the Magic of Free-Form Multi-Image Grounding in Multimodal Large Language Models(https://arxiv.org/abs/2501.05767)
• Keywords: language model, llm, chain-of-thought
• Abstract: The recent advancement of Multimodal Large Language Models (MLLMs) has significantly improved their fine-grained perception of single images and general comprehension across multiple images. However, existing MLLMs still face challenges in achieving precise grounding in complex multi-image scenarios. To address this, we first explore a Chain-of-Thought (CoT) framework that integrates single-image grounding with multi-image comprehension. While partially effective, it remains unstable and struggles to capture abstract visual information due to its non-end-to-end nature. Therefore, we introduce Migician, the first multi-image grounding model capable of performing free-form and accurate grounding across multiple images. To support this, we present the MGrounding-630k dataset, which comprises data for several multi-image grounding tasks derived from existing datasets, along with newly generated free-form grounding instruction-following data. Furthermore, we propose MIG-Bench, a comprehensive benchmark specifically designed for evaluating multi-image grounding capabilities. Experimental results demonstrate that our model achieves significantly superior multi-image grounding capabilities, outperforming the best existing MLLMs by 21.61% and even surpassing much larger 70B models. Our code, model, dataset, and benchmark are fully open-sourced.
• 摘要：多模态大型语言模型 (MLLM) 的最新进展显著提高了它们对单个图像的细粒度感知和对多个图像的一般理解。然而，现有的 MLLM 在实现复杂多图像场景中的精确定位方面仍然面临挑战。为了解决这个问题，我们首先探索一种将单图像定位与多图像理解相结合的思想链 (CoT) 框架。虽然部分有效，但它仍然不稳定，并且由于其非端到端的性质，难以捕捉抽象的视觉信息。因此，我们推出了 Migician，这是第一个能够在多个图像上执行自由形式和准确定位的多图像定位模型。为了支持这一点，我们提出了 MGrounding-630k 数据集，该数据集包含从现有数据集派生的几个多图像定位任务的数据，以及新生成的自由形式定位指令遵循数据。此外，我们提出了 MIG-Bench，这是一个专门为评估多图像定位能力而设计的综合基准。实验结果表明，我们的模型实现了显著卓越的多图像基础能力，比现有的最佳 MLLM 性能高出 21.61%，甚至超越了更大的 70B 模型。我们的代码、模型、数据集和基准都是完全开源的。

Title: ConSim: Measuring Concept-Based Explanations' Effectiveness with Automated Simulatability

• Authors: Antonin Poché, Alon Jacovi, Agustin Martin Picard, Victor Boutin (CERCO, ANITI), Fanny Jourdan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.05855
• Pdf URL: https://arxiv.org/pdf/2501.05855
• Copy Paste: [[2501.05855]] ConSim: Measuring Concept-Based Explanations' Effectiveness with Automated Simulatability(https://arxiv.org/abs/2501.05855)
• Keywords: language model, llm
• Abstract: Concept-based explanations work by mapping complex model computations to human-understandable concepts. Evaluating such explanations is very difficult, as it includes not only the quality of the induced space of possible concepts but also how effectively the chosen concepts are communicated to users. Existing evaluation metrics often focus solely on the former, neglecting the latter. We introduce an evaluation framework for measuring concept explanations via automated simulatability: a simulator's ability to predict the explained model's outputs based on the provided explanations. This approach accounts for both the concept space and its interpretation in an end-to-end evaluation. Human studies for simulatability are notoriously difficult to enact, particularly at the scale of a wide, comprehensive empirical evaluation (which is the subject of this work). We propose using large language models (LLMs) as simulators to approximate the evaluation and report various analyses to make such approximations reliable. Our method allows for scalable and consistent evaluation across various models and datasets. We report a comprehensive empirical evaluation using this framework and show that LLMs provide consistent rankings of explanation methods. Code available at this https URL
• 摘要：基于概念的解释通过将复杂的模型计算映射到人类可理解的概念来工作。评估此类解释非常困难，因为它不仅包括可能概念的诱导空间的质量，还包括所选概念传达给用户的有效性。现有的评估指标通常只关注前者，而忽略了后者。我们引入了一个通过自动可模拟性来衡量概念解释的评估框架：模拟器根据提供的解释预测解释模型输出的能力。这种方法在端到端评估中同时考虑了概念空间及其解释。众所周知，人类对可模拟性的研究很难实施，特别是在广泛、全面的实证评估（这是本研究的主题）的规模上。我们建议使用大型语言模型 (LLM) 作为模拟器来近似评估并报告各种分析以使这种近似可靠。我们的方法允许在各种模型和数据集中进行可扩展且一致的评估。我们报告了使用该框架进行的全面实证评估，并表明 LLM 提供了一致的解释方法排名。代码可从此 https URL 获取

Title: Affordably Fine-tuned LLMs Provide Better Answers to Course-specific MCQs

• Authors: Bianca Raimondi, Saverio Giallorenzo, Maurizio Gabbrielli
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.05891
• Pdf URL: https://arxiv.org/pdf/2501.05891
• Copy Paste: [[2501.05891]] Affordably Fine-tuned LLMs Provide Better Answers to Course-specific MCQs(https://arxiv.org/abs/2501.05891)
• Keywords: language model, llm
• Abstract: In education, the capability of generating human-like text of Large Language Models (LLMs) inspired work on how they can increase the efficiency of learning and teaching. We study the affordability of these models for educators and students by investigating how LLMs answer multiple-choice questions (MCQs) with respect to hardware constraints and refinement techniques. We explore this space by using generic pre-trained LLMs (the 7B, 13B, and 70B variants of LLaMA-2) to answer 162 undergraduate-level MCQs from a course on Programming Languages (PL) -- the MCQ dataset is a contribution of this work, which we make publicly available. Specifically, we dissect how different factors, such as using readily-available material -- (parts of) the course's textbook -- for fine-tuning and quantisation (to decrease resource usage) can change the accuracy of the responses. The main takeaway is that smaller textbook-based fine-tuned models outperform generic larger ones (whose pre-training requires conspicuous resources), making the usage of LLMs for answering MCQs resource- and material-wise affordable.
• 摘要：在教育领域，大型语言模型 (LLM) 生成类似人类文本的能力激发了人们研究如何提高学习和教学效率。我们通过研究 LLM 如何根据硬件约束和改进技术回答多项选择题 (MCQ)，研究了这些模型对教育工作者和学生的承受能力。我们通过使用通用预训练 LLM（LLaMA-2 的 7B、13B 和 70B 变体）来探索这一领域，以回答编程语言 (PL) 课程中的 162 个本科水平 MCQ——MCQ 数据集是这项工作的贡献，我们将其公开。具体来说，我们分析了不同的因素（例如使用现成的材料——（部分）课程教科书——进行微调和量化（以减少资源使用））如何改变响应的准确性。主要的结论是，较小的基于教科书的微调模型优于通用的较大模型（其预训练需要明显的资源），从而使得使用 LLM 来回答 MCQ 在资源和材料方面都是可以承受的。

Title: Navigating Tomorrow: Reliably Assessing Large Language Models Performance on Future Event Prediction

• Authors: Petraq Nako, Adam Jatowt
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2501.05925
• Pdf URL: https://arxiv.org/pdf/2501.05925
• Copy Paste: [[2501.05925]] Navigating Tomorrow: Reliably Assessing Large Language Models Performance on Future Event Prediction(https://arxiv.org/abs/2501.05925)
• Keywords: language model, llm
• Abstract: Predicting future events is an important activity with applications across multiple fields and domains. For example, the capacity to foresee stock market trends, natural disasters, business developments, or political events can facilitate early preventive measures and uncover new opportunities. Multiple diverse computational methods for attempting future predictions, including predictive analysis, time series forecasting, and simulations have been proposed. This study evaluates the performance of several large language models (LLMs) in supporting future prediction tasks, an under-explored domain. We assess the models across three scenarios: Affirmative vs. Likelihood questioning, Reasoning, and Counterfactual analysis. For this, we create a dataset1 by finding and categorizing news articles based on entity type and its popularity. We gather news articles before and after the LLMs training cutoff date in order to thoroughly test and compare model performance. Our research highlights LLMs potential and limitations in predictive modeling, providing a foundation for future improvements.
• 摘要：预测未来事件是一项重要的活动，可应用于多个领域。例如，预测股市趋势、自然灾害、商业发展或政治事件的能力可以促进早期预防措施并发现新的机会。已经提出了多种不同的计算方法来尝试进行未来预测，包括预测分析、时间序列预测和模拟。这项研究评估了几种大型语言模型 (LLM) 在支持未来预测任务（一个尚未充分探索的领域）方面的表现。我们在三种情况下评估了这些模型：肯定与可能性质疑、推理和反事实分析。为此，我们通过根据实体类型及其受欢迎程度查找和分类新闻文章来创建数据集1。我们收集了 LLM 培训截止日期之前和之后的新闻文章，以便彻底测试和比较模型性能。我们的研究突出了 LLM 在预测建模中的潜力和局限性，为未来的改进奠定了基础。

Title: LLMs Reproduce Stereotypes of Sexual and Gender Minorities

• Authors: Ruby Ostrow, Adam Lopez
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.05926
• Pdf URL: https://arxiv.org/pdf/2501.05926
• Copy Paste: [[2501.05926]] LLMs Reproduce Stereotypes of Sexual and Gender Minorities(https://arxiv.org/abs/2501.05926)
• Keywords: language model, llm
• Abstract: A large body of research has found substantial gender bias in NLP systems. Most of this research takes a binary, essentialist view of gender: limiting its variation to the categories _men_ and _women_, conflating gender with sex, and ignoring different sexual identities. But gender and sexuality exist on a spectrum, so in this paper we study the biases of large language models (LLMs) towards sexual and gender minorities beyond binary categories. Grounding our study in a widely used psychological framework -- the Stereotype Content Model -- we demonstrate that English-language survey questions about social perceptions elicit more negative stereotypes of sexual and gender minorities from LLMs, just as they do from humans. We then extend this framework to a more realistic use case: text generation. Our analysis shows that LLMs generate stereotyped representations of sexual and gender minorities in this setting, raising concerns about their capacity to amplify representational harms in creative writing, a widely promoted use case.
• 摘要：大量研究发现 NLP 系统中存在严重的性别偏见。这些研究中的大多数都对性别持二元本质主义观点：将其变化限制在男性和女性类别中，将性别与性别混为一谈，并忽略不同的性别认同。但性别和性取向存在于一个范围内，因此在本文中，我们研究了大型语言模型 (LLM) 对二元类别以外的性别和性别少数群体的偏见。我们的研究基于广泛使用的心理学框架——刻板印象内容模型——我们证明，英语调查中关于社会认知的问题会从 LLM 中引发更多对性别和性别少数群体的负面刻板印象，就像从人类中一样。然后，我们将此框架扩展到更现实的用例：文本生成。我们的分析表明，LLM 在这种环境下会产生对性别和性别少数群体的刻板印象，这引发了人们对它们在创意写作中放大表征危害的能力的担忧，创意写作是一种被广泛推广的用例。

Title: Universal-2-TF: Robust All-Neural Text Formatting for ASR

• Authors: Yash Khare, Taufiquzzaman Peyash, Andrea Vanzo, Takuya Yoshioka
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.05948
• Pdf URL: https://arxiv.org/pdf/2501.05948
• Copy Paste: [[2501.05948]] Universal-2-TF: Robust All-Neural Text Formatting for ASR(https://arxiv.org/abs/2501.05948)
• Keywords: hallucination
• Abstract: This paper introduces an all-neural text formatting (TF) model designed for commercial automatic speech recognition (ASR) systems, encompassing punctuation restoration (PR), truecasing, and inverse text normalization (ITN). Unlike traditional rule-based or hybrid approaches, this method leverages a two-stage neural architecture comprising a multi-objective token classifier and a sequence-to-sequence (seq2seq) model. This design minimizes computational costs and reduces hallucinations while ensuring flexibility and robustness across diverse linguistic entities and text domains. Developed as part of the Universal-2 ASR system, the proposed method demonstrates superior performance in TF accuracy, computational efficiency, and perceptual quality, as validated through comprehensive evaluations using both objective and subjective methods. This work underscores the importance of holistic TF models in enhancing ASR usability in practical settings.
• 摘要：本文介绍了一种专为商业自动语音识别 (ASR) 系统设计的全神经文本格式化 (TF) 模型，包括标点符号恢复 (PR)、真值化和逆文本规范化 (ITN)。与传统的基于规则或混合的方法不同，该方法利用由多目标标记分类器和序列到序列 (seq2seq) 模型组成的两阶段神经架构。这种设计最大限度地降低了计算成本并减少了幻觉，同时确保了跨不同语言实体和文本域的灵活性和稳健性。作为 Universal-2 ASR 系统的一部分开发的所提出的方法在 TF 准确性、计算效率和感知质量方面表现出色，这通过使用客观和主观方法的综合评估得到了验证。这项工作强调了整体 TF 模型在增强 ASR 在实际环境中的可用性方面的重要性。

Title: Effective faking of verbal deception detection with target-aligned adversarial attacks

• Authors: Bennett Kleinberg, Riccardo Loconte, Bruno Verschuere
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.05962
• Pdf URL: https://arxiv.org/pdf/2501.05962
• Copy Paste: [[2501.05962]] Effective faking of verbal deception detection with target-aligned adversarial attacks(https://arxiv.org/abs/2501.05962)
• Keywords: language model
• Abstract: Background: Deception detection through analysing language is a promising avenue using both human judgments and automated machine learning judgments. For both forms of credibility assessment, automated adversarial attacks that rewrite deceptive statements to appear truthful pose a serious threat. Methods: We used a dataset of 243 truthful and 262 fabricated autobiographical stories in a deception detection task for humans and machine learning models. A large language model was tasked to rewrite deceptive statements so that they appear truthful. In Study 1, humans who made a deception judgment or used the detailedness heuristic and two machine learning models (a fine-tuned language model and a simple n-gram model) judged original or adversarial modifications of deceptive statements. In Study 2, we manipulated the target alignment of the modifications, i.e. tailoring the attack to whether the statements would be assessed by humans or computer models. Results: When adversarial modifications were aligned with their target, human (d=-0.07 and d=-0.04) and machine judgments (51% accuracy) dropped to the chance level. When the attack was not aligned with the target, both human heuristics judgments (d=0.30 and d=0.36) and machine learning predictions (63-78%) were significantly better than chance. Conclusions: Easily accessible language models can effectively help anyone fake deception detection efforts both by humans and machine learning models. Robustness against adversarial modifications for humans and machines depends on that target alignment. We close with suggestions on advancing deception research with adversarial attack designs.
• 摘要：背景：通过分析语言来检测欺骗是一种很有前途的方法，既可以利用人类判断，也可以利用自动机器学习判断。对于这两种形式的可信度评估，将欺骗性陈述改写为看似真实的自动对抗性攻击都构成了严重威胁。方法：我们在针对人类和机器学习模型的欺骗检测任务中使用了 243 个真实故事和 262 个虚构自传故事的数据集。一个大型语言模型的任务是重写欺骗性陈述，使其看起来真实。在研究 1 中，做出欺骗判断或使用细节启发式和两个机器学习模型（微调语言模型和简单 n-gram 模型）的人类判断欺骗性陈述的原始或对抗性修改。在研究 2 中，我们操纵了修改的目标对齐，即根据陈述是由人类还是计算机模型评估来定制攻击。结果：当对抗性修改与目标一致时，人类（d=-0.07 和 d=-0.04）和机器判断（51% 准确率）下降到偶然水平。当攻击与目标不一致时，人类启发式判断（d=0.30 和 d=0.36）和机器学习预测（63-78%）都明显优于偶然性。结论：易于访问的语言模型可以有效地帮助任何人伪造人类和机器学习模型的欺骗检测工作。人类和机器对抗对抗性修改的鲁棒性取决于目标一致性。最后，我们提出了关于利用对抗性攻击设计推进欺骗研究的建议。

Title: Hermit Kingdom Through the Lens of Multiple Perspectives: A Case Study of LLM Hallucination on North Korea

• Authors: Eunjung Cho, Won Ik Cho, Soomin Seo
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.05981
• Pdf URL: https://arxiv.org/pdf/2501.05981
• Copy Paste: [[2501.05981]] Hermit Kingdom Through the Lens of Multiple Perspectives: A Case Study of LLM Hallucination on North Korea(https://arxiv.org/abs/2501.05981)
• Keywords: language model, llm, hallucination
• Abstract: Hallucination in large language models (LLMs) remains a significant challenge for their safe deployment, particularly due to its potential to spread misinformation. Most existing solutions address this challenge by focusing on aligning the models with credible sources or by improving how models communicate their confidence (or lack thereof) in their outputs. While these measures may be effective in most contexts, they may fall short in scenarios requiring more nuanced approaches, especially in situations where access to accurate data is limited or determining credible sources is challenging. In this study, we take North Korea - a country characterised by an extreme lack of reliable sources and the prevalence of sensationalist falsehoods - as a case study. We explore and evaluate how some of the best-performing multilingual LLMs and specific language-based models generate information about North Korea in three languages spoken in countries with significant geo-political interests: English (United States, United Kingdom), Korean (South Korea), and Mandarin Chinese (China). Our findings reveal significant differences, suggesting that the choice of model and language can lead to vastly different understandings of North Korea, which has important implications given the global security challenges the country poses.
• 摘要：大型语言模型 (LLM) 中的幻觉仍然是其安全部署的重大挑战，特别是因为它有可能传播错误信息。大多数现有解决方案通过专注于使模型与可靠来源保持一致或通过改进模型传达其对输出的信心（或缺乏信心）的方式来应对这一挑战。虽然这些措施在大多数情况下可能是有效的，但它们可能在需要更细致入微的方法的情况下失效，特别是在获取准确数据有限或确定可靠来源具有挑战性的情况下。在本研究中，我们以朝鲜——一个极其缺乏可靠来源和耸人听闻的谎言盛行的国家——为案例研究。我们探索和评估一些表现最佳的多语言 LLM 和特定的基于语言的模型如何用三种在具有重大地缘政治利益的国家使用的语言生成有关朝鲜的信息：英语（美国、英国）、韩语（韩国）和普通话（中国）。我们的研究结果揭示了显著的差异，表明模型和语言的选择可能会导致对朝鲜的巨大不同理解，这对于该国带来的全球安全挑战具有重要意义。

Title: Addressing speaker gender bias in large scale speech translation systems

• Authors: Shubham Bansal, Vikas Joshi, Harveen Chadha, Rupeshkumar Mehta, Jinyu Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.05989
• Pdf URL: https://arxiv.org/pdf/2501.05989
• Copy Paste: [[2501.05989]] Addressing speaker gender bias in large scale speech translation systems(https://arxiv.org/abs/2501.05989)
• Keywords: language model, llm
• Abstract: This study addresses the issue of speaker gender bias in Speech Translation (ST) systems, which can lead to offensive and inaccurate translations. The masculine bias often found in large-scale ST systems is typically perpetuated through training data derived from Machine Translation (MT) systems. Our approach involves two key steps. First, we employ Large Language Models (LLMs) to rectify translations based on the speaker's gender in a cost-effective manner. Second, we fine-tune the ST model with the corrected data, enabling the model to generate gender-specific translations directly from audio cues, without the need for explicit gender input. Additionally, we propose a three-mode fine-tuned model for scenarios where the speaker's gender is either predefined or should not be inferred from speech cues. We demonstrate a 70% improvement in translations for female speakers compared to our baseline and other large-scale ST systems, such as Seamless M4T and Canary, on the MuST-SHE test set.
• 摘要：本研究探讨了语音翻译 (ST) 系统中说话者性别偏见的问题，该问题可能导致翻译不当和不准确。大型 ST 系统中经常出现的男性偏见通常是通过来自机器翻译 (MT) 系统的训练数据延续下来的。我们的方法包括两个关键步骤。首先，我们使用大型语言模型 (LLM) 以经济高效的方式根据说话者的性别纠正翻译。其次，我们使用校正后的数据对 ST 模型进行微调，使模型能够直接从音频提示生成特定于性别的翻译，而无需明确的性别输入。此外，我们提出了一种三模式微调模型，用于说话者的性别是预定义的或不应从语音提示推断出来的场景。与我们的基线和其他大型 ST 系统（例如 Seamless M4T 和 Canary）相比，我们在 MuST-SHE 测试集上将女性说话者的翻译提高了 70%。

Title: From Conversation to Automation: Leveraging Large Language Models to Analyze Strategies in Problem Solving Therapy

• Authors: Elham Aghakhani, Lu Wang, Karla T. Washington, George Demiris, Jina Huh-Yoo, Rezvaneh Rezapour
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.06101
• Pdf URL: https://arxiv.org/pdf/2501.06101
• Copy Paste: [[2501.06101]] From Conversation to Automation: Leveraging Large Language Models to Analyze Strategies in Problem Solving Therapy(https://arxiv.org/abs/2501.06101)
• Keywords: language model, gpt, llm, chat
• Abstract: Problem-solving therapy (PST) is a structured psychological approach that helps individuals manage stress and resolve personal issues by guiding them through problem identification, solution brainstorming, decision-making, and outcome evaluation. As mental health care increasingly integrates technologies like chatbots and large language models (LLMs), understanding how PST can be effectively automated is important. This study leverages anonymized therapy transcripts to analyze and classify therapeutic interventions using various LLMs and transformer-based models. Our results show that GPT-4o achieved the highest accuracy (0.76) in identifying PST strategies, outperforming other models. Additionally, we introduced a new dimension of communication strategies that enhances the current PST framework, offering deeper insights into therapist-client interactions. This research demonstrates the potential of LLMs to automate complex therapeutic dialogue analysis, providing a scalable, efficient tool for mental health interventions. Our annotation framework can enhance the accessibility, effectiveness, and personalization of PST, supporting therapists in real-time with more precise, targeted interventions.
• 摘要：问题解决疗法 (PST) 是一种结构化的心理学方法，通过引导个人进行问题识别、解决方案头脑风暴、决策和结果评估，帮助他们管理压力并解决个人问题。随着心理健康护理越来越多地整合聊天机器人和大型语言模型 (LLM) 等技术，了解如何有效地实现 PST 自动化非常重要。本研究利用匿名治疗记录，使用各种 LLM 和基于转换器的模型分析和分类治疗干预措施。我们的结果表明，GPT-4o 在识别 PST 策略方面取得了最高的准确率 (0.76)，优于其他模型。此外，我们引入了一个新的沟通策略维度，增强了当前的 PST 框架，为治疗师与客户之间的互动提供了更深入的见解。这项研究展示了 LLM 自动化复杂治疗对话分析的潜力，为心理健康干预提供了一种可扩展、高效的工具。我们的注释框架可以增强 PST 的可访问性、有效性和个性化，实时支持治疗师提供更精确、更有针对性的干预措施。

Title: Fleurs-SLU: A Massively Multilingual Benchmark for Spoken Language Understanding

• Authors: Fabian David Schmidt, Ivan Vulić, Goran Glavaš, David Ifeoluwa Adelani
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.06117
• Pdf URL: https://arxiv.org/pdf/2501.06117
• Copy Paste: [[2501.06117]] Fleurs-SLU: A Massively Multilingual Benchmark for Spoken Language Understanding(https://arxiv.org/abs/2501.06117)
• Keywords: language model
• Abstract: While recent multilingual automatic speech recognition models claim to support thousands of languages, ASR for low-resource languages remains highly unreliable due to limited bimodal speech and text training data. Better multilingual spoken language understanding (SLU) can strengthen massively the robustness of multilingual ASR by levering language semantics to compensate for scarce training data, such as disambiguating utterances via context or exploiting semantic similarities across languages. Even more so, SLU is indispensable for inclusive speech technology in roughly half of all living languages that lack a formal writing system. However, the evaluation of multilingual SLU remains limited to shallower tasks such as intent classification or language identification. To address this, we present Fleurs-SLU, a multilingual SLU benchmark that encompasses topical speech classification in 102 languages and multiple-choice question answering through listening comprehension in 92 languages. We extensively evaluate both end-to-end speech classification models and cascaded systems that combine speech-to-text transcription with subsequent classification by large language models on Fleurs-SLU. Our results show that cascaded systems exhibit greater robustness in multilingual SLU tasks, though speech encoders can achieve competitive performance in topical speech classification when appropriately pre-trained. We further find a strong correlation between robust multilingual ASR, effective speech-to-text translation, and strong multilingual SLU, highlighting the mutual benefits between acoustic and semantic speech representations.
• 摘要：尽管最近的多语言自动语音识别模型声称支持数千种语言，但由于双模语音和文本训练数据有限，低资源语言的 ASR 仍然非常不可靠。更好的多语言口语理解 (SLU) 可以通过利用语言语义来弥补稀缺的训练数据，例如通过上下文消除话语歧义或利用跨语言的语义相似性，从而大大增强多语言 ASR 的稳健性。更重要的是，对于大约一半缺乏正式书写系统的现存语言而言，SLU 对于包容性语音技术是必不可少的。然而，对多语言 SLU 的评估仍然仅限于意图分类或语言识别等较浅的任务。为了解决这个问题，我们提出了 Fleurs-SLU，这是一个多语言 SLU 基准，涵盖 102 种语言的主题语音分类和 92 种语言的听力理解多项选择题回答。我们对端到端语音分类模型和级联系统进行了广泛的评估，这些系统将语音到文本的转录与随后由 Fleurs-SLU 上的大型语言模型进行的分类相结合。我们的结果表明，级联系统在多语言 SLU 任务中表现出更高的鲁棒性，尽管语音编码器在经过适当的预训练后可以在主题语音分类中取得有竞争力的表现。我们进一步发现，稳健的多语言 ASR、有效的语音到文本翻译和强大的多语言 SLU 之间存在很强的相关性，凸显了声学和语义语音表示之间的相互优势。

Title: Merging Feed-Forward Sublayers for Compressed Transformers

• Authors: Neha Verma, Kenton Murray, Kevin Duh
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.06126
• Pdf URL: https://arxiv.org/pdf/2501.06126
• Copy Paste: [[2501.06126]] Merging Feed-Forward Sublayers for Compressed Transformers(https://arxiv.org/abs/2501.06126)
• Keywords: language model
• Abstract: With the rise and ubiquity of larger deep learning models, the need for high-quality compression techniques is growing in order to deploy these models widely. The sheer parameter count of these models makes it difficult to fit them into the memory constraints of different hardware. In this work, we present a novel approach to model compression by merging similar parameter groups within a model, rather than pruning away less important parameters. Specifically, we select, align, and merge separate feed-forward sublayers in Transformer models, and test our method on language modeling, image classification, and machine translation. With our method, we demonstrate performance comparable to the original models while combining more than a third of model feed-forward sublayers, and demonstrate improved performance over a strong layer-pruning baseline. For instance, we can remove over 21% of total parameters from a Vision Transformer, while maintaining 99% of its original performance. Additionally, we observe that some groups of feed-forward sublayers exhibit high activation similarity, which may help explain their surprising mergeability.
• 摘要：随着大型深度学习模型的兴起和普及，为了广泛部署这些模型，对高质量压缩技术的需求日益增长。这些模型的参数数量之多使得它们很难适应不同硬件的内存限制。在这项工作中，我们提出了一种新颖的模型压缩方法，即合并模型中的相似参数组，而不是修剪不太重要的参数。具体来说，我们选择、对齐和合并 Transformer 模型中单独的前馈子层，并在语言建模、图像分类和机器翻译上测试我们的方法。使用我们的方法，我们在结合超过三分之一的模型前馈子层的同时展示了与原始模型相当的性能，并展示了比强大的层修剪基线更好的性能。例如，我们可以从 Vision Transformer 中删除超过 21% 的总参数，同时保持其原始性能的 99%。此外，我们观察到一些前馈子层组表现出很高的激活相似性，这可能有助于解释它们令人惊讶的可合并性。

Title: Contextual ASR Error Handling with LLMs Augmentation for Goal-Oriented Conversational AI

• Authors: Yuya Asano, Sabit Hassan, Paras Sharma, Anthony Sicilia, Katherine Atwell, Diane Litman, Malihe Alikhani
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.06129
• Pdf URL: https://arxiv.org/pdf/2501.06129
• Copy Paste: [[2501.06129]] Contextual ASR Error Handling with LLMs Augmentation for Goal-Oriented Conversational AI(https://arxiv.org/abs/2501.06129)
• Keywords: language model, llm
• Abstract: General-purpose automatic speech recognition (ASR) systems do not always perform well in goal-oriented dialogue. Existing ASR correction methods rely on prior user data or named entities. We extend correction to tasks that have no prior user data and exhibit linguistic flexibility such as lexical and syntactic variations. We propose a novel context augmentation with a large language model and a ranking strategy that incorporates contextual information from the dialogue states of a goal-oriented conversational AI and its tasks. Our method ranks (1) n-best ASR hypotheses by their lexical and semantic similarity with context and (2) context by phonetic correspondence with ASR hypotheses. Evaluated in home improvement and cooking domains with real-world users, our method improves recall and F1 of correction by 34% and 16%, respectively, while maintaining precision and false positive rate. Users rated .8-1 point (out of 5) higher when our correction method worked properly, with no decrease due to false positives.
• 摘要：通用自动语音识别 (ASR) 系统在面向目标的对话中并不总是表现良好。现有的 ASR 校正方法依赖于先前的用户数据或命名实体。我们将校正扩展到没有先前用户数据并表现出语言灵活性（例如词汇和句法变化）的任务。我们提出了一种新颖的上下文增强方法，该方法使用大型语言模型和排名策略，该策略结合了面向目标的对话 AI 及其任务的对话状态中的上下文信息。我们的方法根据 (1) 词汇和语义与上下文的相似性对 n 个最佳 ASR 假设进行排名，以及 (2) 根据与 ASR 假设的语音对应性对上下文进行排名。在家庭装修和烹饪领域与真实用户进行评估后，我们的方法分别将校正的召回率和 F1 提高了 34% 和 16%，同时保持了准确率和假阳性率。当我们的校正方法正常工作时，用户的评分会提高 0.8-1 分（满分 5 分），并且不会因假阳性而降低评分。