2024-11-25

Title: Can Artificial Intelligence Generate Quality Research Topics Reflecting Patient Concerns?

• Authors: Jiyeong Kim, Michael L. Chen, Shawheen J. Rezaei, Mariana Ramirez-Posada, Jennifer L. Caswell-Jin, Allison W. Kurian, Fauzia Riaz, Kavita Y. Sarin, Jean Y. Tang, Steven M. Asch, Eleni Linos
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2411.14456
• Pdf URL: https://arxiv.org/pdf/2411.14456
• Copy Paste: [[2411.14456]] Can Artificial Intelligence Generate Quality Research Topics Reflecting Patient Concerns?(https://arxiv.org/abs/2411.14456)
• Keywords: gpt, prompt, chat
• Abstract: Patient-centered research is increasingly important in narrowing the gap between research and patient care, yet incorporating patient perspectives into health research has been inconsistent. We propose an automated framework leveraging innovative natural language processing (NLP) and artificial intelligence (AI) with patient portal messages to generate research ideas that prioritize important patient issues. We further quantified the quality of AI-generated research topics. To define patient clinical concerns, we analyzed 614,464 patient messages from 25,549 individuals with breast or skin cancer obtained from a large academic hospital (2013 to 2024), constructing a 2-staged unsupervised NLP topic model. Then, we generated research topics to resolve the defined issues using a widely used AI (ChatGPT-4o, OpenAI Inc, April 2024 version) with prompt-engineering strategies. We guided AI to perform multi-level tasks: 1) knowledge interpretation and summarization (e.g., interpreting and summarizing the NLP-defined topics), 2) knowledge generation (e.g., generating research ideas corresponding to patients issues), 3) self-reflection and correction (e.g., ensuring and revising the research ideas after searching for scientific articles), and 4) self-reassurance (e.g., confirming and finalizing the research ideas). Six highly experienced breast oncologists and dermatologists assessed the significance and novelty of AI-generated research topics using a 5-point Likert scale (1-exceptional, 5-poor). One-third of the AI-suggested research topics were highly significant and novel when both scores were lower than the average. Two-thirds of the AI-suggested topics were novel in both cancers. Our findings demonstrate that AI-generated research topics reflecting patient perspectives via a large volume of patient messages can meaningfully guide future directions in patient-centered health research.
• 摘要：以患者为中心的研究在缩小研究与患者护理之间的差距方面越来越重要，但将患者观点纳入健康研究却一直不一致。我们提出了一个自动化框架，利用创新的自然语言处理 (NLP) 和人工智能 (AI) 处理患者门户消息，以产生优先考虑重要患者问题的研究思路。我们进一步量化了人工智能生成的研究主题的质量。为了确定患者的临床问题，我们分析了一家大型学术医院 25,549 名乳腺癌或皮肤癌患者的 614,464 条患者消息（2013 年至 2024 年），构建了一个 2 阶段无监督 NLP 主题模型。然后，我们使用广泛使用的人工智能（ChatGPT-4o，OpenAI Inc，2024 年 4 月版）和提示工程策略生成研究主题来解决定义的问题。我们指导 AI 执行多层次任务：1）知识解释和总结（例如，解释和总结 NLP 定义的主题），2）知识生成（例如，生成与患者问题相对应的研究想法），3）自我反思和纠正（例如，在搜索科学文章后确保和修改研究想法），以及 4）自我保证（例如，确认和最终确定研究想法）。六位经验丰富的乳腺肿瘤学家和皮肤科医生使用 5 点李克特量表（1-优秀，5-差）评估了 AI 生成的研究主题的重要性和新颖性。当两个分数都低于平均值时，三分之一的 AI 建议的研究主题非常重要且新颖。三分之二的 AI 建议主题在两种癌症中都是新颖的。我们的研究结果表明，通过大量患者信息反映患者观点的 AI 生成的研究主题可以有意义地指导以患者为中心的健康研究的未来方向。

Title: Unveiling User Preferences: A Knowledge Graph and LLM-Driven Approach for Conversational Recommendation

• Authors: Zhangchi Qiu, Linhao Luo, Shirui Pan, Alan Wee-Chung Liew
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2411.14459
• Pdf URL: https://arxiv.org/pdf/2411.14459
• Copy Paste: [[2411.14459]] Unveiling User Preferences: A Knowledge Graph and LLM-Driven Approach for Conversational Recommendation(https://arxiv.org/abs/2411.14459)
• Keywords: language model, llm
• Abstract: Conversational Recommender Systems (CRSs) aim to provide personalized recommendations through dynamically capturing user preferences in interactive conversations. Conventional CRSs often extract user preferences as hidden representations, which are criticized for their lack of interpretability. This diminishes the transparency and trustworthiness of the recommendation process. Recent works have explored combining the impressive capabilities of Large Language Models (LLMs) with the domain-specific knowledge of Knowledge Graphs (KGs) to generate human-understandable recommendation explanations. Despite these efforts, the integration of LLMs and KGs for CRSs remains challenging due to the modality gap between unstructured dialogues and structured KGs. Moreover, LLMs pre-trained on large-scale corpora may not be well-suited for analyzing user preferences, which require domain-specific knowledge. In this paper, we propose COMPASS, a plug-and-play framework that synergizes LLMs and KGs to unveil user preferences, enhancing the performance and explainability of existing CRSs. To address integration challenges, COMPASS employs a two-stage training approach: first, it bridges the gap between the structured KG and natural language through an innovative graph entity captioning pre-training mechanism. This enables the LLM to transform KG entities into concise natural language descriptions, allowing them to comprehend domain-specific knowledge. Following, COMPASS optimizes user preference modeling via knowledge-aware instruction fine-tuning, where the LLM learns to reason and summarize user preferences from both dialogue histories and KG-augmented context. This enables COMPASS to perform knowledge-aware reasoning and generate comprehensive and interpretable user preferences that can seamlessly integrate with existing CRS models for improving recommendation performance and explainability.
• 摘要：对话式推荐系统 (CRS) 旨在通过在交互式对话中动态捕获用户偏好来提供个性化推荐。传统的 CRS 通常将用户偏好提取为隐藏表示，这些表示因缺乏可解释性而受到批评。这降低了推荐过程的透明度和可信度。最近的研究探索了将大型语言模型 (LLM) 的强大功能与知识图谱 (KG) 的领域特定知识相结合，以生成人类可理解的推荐解释。尽管做出了这些努力，但由于非结构化对话和结构化 KG 之间的模态差距，将 LLM 和 KG 集成到 CRS 中仍然具有挑战性。此外，在大型语料库上预先训练的 LLM 可能不太适合分析用户偏好，因为这需要领域特定知识。在本文中，我们提出了 COMPASS，这是一个即插即用的框架，它协同 LLM 和 KG 来揭示用户偏好，从而提高现有 CRS 的性能和可解释性。为了解决集成挑战，COMPASS 采用了两阶段训练方法：首先，它通过创新的图实体字幕预训练机制弥合了结构化知识图谱和自然语言之间的差距。这使 LLM 能够将知识图谱实体转换为简洁的自然语言描述，从而使其能够理解特定领域的知识。接下来，COMPASS 通过知识感知指令微调优化用户偏好建模，其中 LLM 学习从对话历史和知识图谱增强上下文中推理和总结用户偏好。这使 COMPASS 能够执行知识感知推理并生成全面且可解释的用户偏好，这些偏好可以与现有的 CRS 模型无缝集成，以提高推荐性能和可解释性。

Title: LLaSA: Large Language and Structured Data Assistant

• Authors: Yao Xu, Shizhu He, Zeng Xiangrong, Jiabei Chen, Guang Liu, Bingning Wang, Jun Zhao, Kang Liu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.14460
• Pdf URL: https://arxiv.org/pdf/2411.14460
• Copy Paste: [[2411.14460]] LLaSA: Large Language and Structured Data Assistant(https://arxiv.org/abs/2411.14460)
• Keywords: language model, llm
• Abstract: Structured data, such as tables, graphs, and databases, play a critical role in plentiful NLP tasks such as question answering and dialogue system. Recently, inspired by Vision-Language Models, Graph Neutral Networks (GNNs) have been introduced as an additional modality into the input of Large Language Models (LLMs) to improve their performance on Structured Knowledge Grounding (SKG) tasks. However, those GNN-enhanced LLMs have the following limitations: (1) They employ diverse GNNs to model varying types of structured data, rendering them unable to uniformly process various forms of structured data. (2) The pretraining of GNNs is coupled with specific LLMs, which prevents GNNs from fully aligning with the textual space and limits their adaptability to other LLMs. To address these issues, we propose \textbf{L}arge \textbf{L}anguage and \textbf{S}tructured Data \textbf{A}ssistant (LLaSA), a general framework for enhancing LLMs' ability to handle structured data. Specifically, we represent various types of structured data in a unified hypergraph format, and use self-supervised learning to pretrain a hypergraph encoder, and a G-Former compressing encoded hypergraph representations with cross-attention. The compressed hypergraph representations are appended to the serialized inputs during training and inference stages of LLMs. Experimental results on multiple SKG tasks show that our pretrained hypergraph encoder can adapt to various LLMs and enhance their ability to process different types of structured data. Besides, LLaSA, with LoRA fine-tuning, outperforms previous SOTA method using full parameters tuning.
• 摘要：结构化数据，如表格、图形和数据库，在问答和对话系统等大量 NLP 任务中发挥着至关重要的作用。最近，受视觉语言模型的启发，图神经网络 (GNN) 作为附加模态被引入到大型语言模型 (LLM) 的输入中，以提高其在结构化知识基础 (SKG) 任务中的表现。然而，这些 GNN 增强型 LLM 具有以下局限性：(1) 它们使用不同的 GNN 来建模不同类型的结构化数据，导致它们无法统一地处理各种形式的结构化数据。(2) GNN 的预训练与特定的 LLM 相结合，这使得 GNN 无法完全与文本空间对齐，并限制了它们对其他 LLM 的适应性。为了解决这些问题，我们提出了 \textbf{L}arge \textbf{L}anguage 和 \textbf{S}structured Data \textbf{A}ssistant (LLaSA)，这是一个用于增强 LLM 处理结构化数据能力的通用框架。具体来说，我们以统一的超图格式表示各种类型的结构化数据，并使用自监督学习来预训练超图编码器，以及使用交叉注意压缩编码超图表示的 G-Former。压缩的超图表示在 LLM 的训练和推理阶段附加到序列化输入中。在多个 SKG 任务上的实验结果表明，我们预训练的超图编码器可以适应各种 LLM 并增强其处理不同类型结构化数据的能力。此外，使用 LoRA 微调的 LLaSA 优于以前使用全参数调整的 SOTA 方法。

Title: Towards Next-Generation Medical Agent: How o1 is Reshaping Decision-Making in Medical Scenarios

• Authors: Shaochen Xu, Yifan Zhou, Zhengliang Liu, Zihao Wu, Tianyang Zhong, Huaqin Zhao, Yiwei Li, Hanqi Jiang, Yi Pan, Junhao Chen, Jin Lu, Wei Zhang, Tuo Zhang, Lu Zhang, Dajiang Zhu, Xiang Li, Wei Liu, Quanzheng Li, Andrea Sikora, Xiaoming Zhai, Zhen Xiang, Tianming Liu
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2411.14461
• Pdf URL: https://arxiv.org/pdf/2411.14461
• Copy Paste: [[2411.14461]] Towards Next-Generation Medical Agent: How o1 is Reshaping Decision-Making in Medical Scenarios(https://arxiv.org/abs/2411.14461)
• Keywords: language model, llm, agent
• Abstract: Artificial Intelligence (AI) has become essential in modern healthcare, with large language models (LLMs) offering promising advances in clinical decision-making. Traditional model-based approaches, including those leveraging in-context demonstrations and those with specialized medical fine-tuning, have demonstrated strong performance in medical language processing but struggle with real-time adaptability, multi-step reasoning, and handling complex medical tasks. Agent-based AI systems address these limitations by incorporating reasoning traces, tool selection based on context, knowledge retrieval, and both short- and long-term memory. These additional features enable the medical AI agent to handle complex medical scenarios where decision-making should be built on real-time interaction with the environment. Therefore, unlike conventional model-based approaches that treat medical queries as isolated questions, medical AI agents approach them as complex tasks and behave more like human doctors. In this paper, we study the choice of the backbone LLM for medical AI agents, which is the foundation for the agent's overall reasoning and action generation. In particular, we consider the emergent o1 model and examine its impact on agents' reasoning, tool-use adaptability, and real-time information retrieval across diverse clinical scenarios, including high-stakes settings such as intensive care units (ICUs). Our findings demonstrate o1's ability to enhance diagnostic accuracy and consistency, paving the way for smarter, more responsive AI tools that support better patient outcomes and decision-making efficacy in clinical practice.
• 摘要：人工智能 (AI) 已成为现代医疗保健中不可或缺的一部分，大型语言模型 (LLM) 为临床决策提供了有希望的进展。传统的基于模型的方法，包括利用上下文演示的方法和具有专门的医学微调的方法，在医学语言处理方面表现出色，但在实时适应性、多步骤推理和处理复杂的医疗任务方面却存在困难。基于代理的 AI 系统通过结合推理轨迹、基于上下文的工具选择、知识检索以及短期和长期记忆来解决这些限制。这些附加功能使医疗 AI 代理能够处理复杂的医疗场景，在这些场景中，决策应建立在与环境的实时交互之上。因此，与将医疗查询视为孤立问题的传统基于模型的方法不同，医疗 AI 代理将它们视为复杂的任务，并且表现得更像人类医生。在本文中，我们研究了医疗 AI 代理的主干 LLM 的选择，这是代理整体推理和行动生成的基础。具体来说，我们考虑了新兴的 o1 模型，并研究了其对代理推理、工具使用适应性和实时信息检索的影响，这些影响涉及各种临床场景，包括重症监护病房 (ICU) 等高风险环境。我们的研究结果表明，o1 能够提高诊断的准确性和一致性，为更智能、响应更快的 AI 工具铺平了道路，这些工具可支持更好的患者治疗结果和临床实践中的决策效率。

Title: Testing Uncertainty of Large Language Models for Physics Knowledge and Reasoning

• Authors: Elizaveta Reganova, Peter Steinbach
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.14465
• Pdf URL: https://arxiv.org/pdf/2411.14465
• Copy Paste: [[2411.14465]] Testing Uncertainty of Large Language Models for Physics Knowledge and Reasoning(https://arxiv.org/abs/2411.14465)
• Keywords: language model, gpt, llm, agent
• Abstract: Large Language Models (LLMs) have gained significant popularity in recent years for their ability to answer questions in various fields. However, these models have a tendency to "hallucinate" their responses, making it challenging to evaluate their performance. A major challenge is determining how to assess the certainty of a model's predictions and how it correlates with accuracy. In this work, we introduce an analysis for evaluating the performance of popular open-source LLMs, as well as gpt-3.5 Turbo, on multiple choice physics questionnaires. We focus on the relationship between answer accuracy and variability in topics related to physics. Our findings suggest that most models provide accurate replies in cases where they are certain, but this is by far not a general behavior. The relationship between accuracy and uncertainty exposes a broad horizontal bell-shaped distribution. We report how the asymmetry between accuracy and uncertainty intensifies as the questions demand more logical reasoning of the LLM agent, while the same relationship remains sharp for knowledge retrieval tasks.
• 摘要：近年来，大型语言模型 (LLM) 因其能够回答各个领域的问题而广受欢迎。然而，这些模型往往会“幻觉”其反应，这使得评估其性能变得具有挑战性。一个主要的挑战是确定如何评估模型预测的确定性以及它与准确性之间的关系。在这项工作中，我们引入了一种分析来评估流行的开源 LLM 以及 GPT-3.5 Turbo 在多项选择物理问卷上的性能。我们关注物理相关主题的答案准确性和可变性之间的关系。我们的研究结果表明，大多数模型在确定的情况下都会提供准确的答复，但这绝不是一种普遍的行为。准确度和不确定性之间的关系呈现出宽阔的水平钟形分布。我们报告了随着问题要求 LLM 代理进行更多逻辑推理，准确度和不确定性之间的不对称性如何加剧，而同样的关系对于知识检索任务仍然很明显。

Title: Popular LLMs Amplify Race and Gender Disparities in Human Mobility

• Authors: Xinhua Wu, Qi R. Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.14469
• Pdf URL: https://arxiv.org/pdf/2411.14469
• Copy Paste: [[2411.14469]] Popular LLMs Amplify Race and Gender Disparities in Human Mobility(https://arxiv.org/abs/2411.14469)
• Keywords: language model, gpt, llm, prompt
• Abstract: As large language models (LLMs) are increasingly applied in areas influencing societal outcomes, it is critical to understand their tendency to perpetuate and amplify biases. This study investigates whether LLMs exhibit biases in predicting human mobility -- a fundamental human behavior -- based on race and gender. Using three prominent LLMs -- GPT-4, Gemini, and Claude -- we analyzed their predictions of visitations to points of interest (POIs) for individuals, relying on prompts that included names with and without explicit demographic details. We find that LLMs frequently reflect and amplify existing societal biases. Specifically, predictions for minority groups were disproportionately skewed, with these individuals being significantly less likely to be associated with wealth-related points of interest (POIs). Gender biases were also evident, as female individuals were consistently linked to fewer career-related POIs compared to their male counterparts. These biased associations suggest that LLMs not only mirror but also exacerbate societal stereotypes, particularly in contexts involving race and gender.
• 摘要：随着大型语言模型 (LLM) 越来越多地应用于影响社会结果的领域，了解它们延续和放大偏见的倾向至关重要。这项研究调查了 LLM 在预测人类流动性（一种基本的人类行为）时是否表现出基于种族和性别的偏见。我们使用三款著名的 LLM——GPT-4、Gemini 和 Claude——分析了它们对个人访问兴趣点 (POI) 的预测，依赖于包含和不包含明确人口统计细节的名称的提示。我们发现 LLM 经常反映和放大现有的社会偏见。具体而言，对少数群体的预测存在不成比例的偏差，这些个体与财富相关的兴趣点 (POI) 关联的可能性明显较小。性别偏见也很明显，因为与男性相比，女性个体始终与较少的职业相关 POI 相关。这些有偏见的联想表明，法学硕士不仅反映而且加剧了社会刻板印象，特别是在涉及种族和性别的背景下。

Title: Exploring the Potential Role of Generative AI in the TRAPD Procedure for Survey Translation

• Authors: Erica Ann Metheney, Lauren Yehle
• Subjects: cs.CL, stat.AP, stat.ME
• Abstract URL: https://arxiv.org/abs/2411.14472
• Pdf URL: https://arxiv.org/pdf/2411.14472
• Copy Paste: [[2411.14472]] Exploring the Potential Role of Generative AI in the TRAPD Procedure for Survey Translation(https://arxiv.org/abs/2411.14472)
• Keywords: gpt, prompt, chat
• Abstract: This paper explores and assesses in what ways generative AI can assist in translating survey instruments. Writing effective survey questions is a challenging and complex task, made even more difficult for surveys that will be translated and deployed in multiple linguistic and cultural settings. Translation errors can be detrimental, with known errors rendering data unusable for its intended purpose and undetected errors leading to incorrect conclusions. A growing number of institutions face this problem as surveys deployed by private and academic organizations globalize, and the success of their current efforts depends heavily on researchers' and translators' expertise and the amount of time each party has to contribute to the task. Thus, multilinguistic and multicultural surveys produced by teams with limited expertise, budgets, or time are at significant risk for translation-based errors in their data. We implement a zero-shot prompt experiment using ChatGPT to explore generative AI's ability to identify features of questions that might be difficult to translate to a linguistic audience other than the source language. We find that ChatGPT can provide meaningful feedback on translation issues, including common source survey language, inconsistent conceptualization, sensitivity and formality issues, and nonexistent concepts. In addition, we provide detailed information on the practicality of the approach, including accessing the necessary software, associated costs, and computational run times. Lastly, based on our findings, we propose avenues for future research that integrate AI into survey translation practices.
• 摘要：本文探讨并评估了生成式人工智能如何协助翻译调查工具。编写有效的调查问卷是一项具有挑战性且复杂的任务，对于将在多种语言和文化环境中翻译和部署的调查问卷来说，这更是难上加难。翻译错误可能会造成不利影响，已知错误会导致数据无法用于其预期目的，而未检测到的错误会导致错误的结论。随着私人和学术组织部署的调查问卷全球化，越来越多的机构面临这一问题，他们当前工作的成功在很大程度上取决于研究人员和翻译人员的专业知识以及各方为这项任务投入的时间。因此，由专业知识、预算或时间有限的团队制作的多语言和多元文化调查问卷在其数据中存在很大的翻译错误风险。我们使用 ChatGPT 实施了零样本提示实验，以探索生成式人工智能识别可能难以翻译给源语言以外的语言受众的问题特征的能力。我们发现 ChatGPT 可以针对翻译问题提供有意义的反馈，包括常见的源调查语言、不一致的概念化、敏感性和形式问题以及不存在的概念。此外，我们还提供了有关该方法实用性的详细信息，包括访问必要的软件、相关成本和计算运行时间。最后，根据我们的研究结果，我们提出了将 AI 融入调查翻译实践的未来研究途径。

Title: Large Language Model for Qualitative Research -- A Systematic Mapping Study

• Authors: Cauã Ferreira Barros, Bruna Borges Azevedo, Valdemar Vicente Graciano Neto, Mohamad Kassab, Marcos Kalinowski, Hugo Alexandre D. do Nascimento, Michelle C.G.S.P. Bandeira
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.14473
• Pdf URL: https://arxiv.org/pdf/2411.14473
• Copy Paste: [[2411.14473]] Large Language Model for Qualitative Research -- A Systematic Mapping Study(https://arxiv.org/abs/2411.14473)
• Keywords: language model, llm, prompt
• Abstract: The exponential growth of text-based data in domains such as healthcare, education, and social sciences has outpaced the capacity of traditional qualitative analysis methods, which are time-intensive and prone to subjectivity. Large Language Models (LLMs), powered by advanced generative AI, have emerged as transformative tools capable of automating and enhancing qualitative analysis. This study systematically maps the literature on the use of LLMs for qualitative research, exploring their application contexts, configurations, methodologies, and evaluation metrics. Findings reveal that LLMs are utilized across diverse fields, demonstrating the potential to automate processes traditionally requiring extensive human input. However, challenges such as reliance on prompt engineering, occasional inaccuracies, and contextual limitations remain significant barriers. This research highlights opportunities for integrating LLMs with human expertise, improving model robustness, and refining evaluation methodologies. By synthesizing trends and identifying research gaps, this study aims to guide future innovations in the application of LLMs for qualitative analysis.
• 摘要：医疗保健、教育和社会科学等领域的文本数据呈指数级增长，超出了传统定性分析方法的能力，因为传统定性分析方法耗时且容易产生主观性。由先进的生成式人工智能驱动的大型语言模型 (LLM) 已成为能够自动化和增强定性分析的变革性工具。本研究系统地绘制了有关使用 LLM 进行定性研究的文献，探索了它们的应用环境、配置、方法和评估指标。研究结果表明，LLM 应用于各个领域，展示了自动化传统上需要大量人工输入的过程的潜力。然而，依赖快速工程、偶尔出现不准确和上下文限制等挑战仍然是重大障碍。本研究强调了将 LLM 与人类专业知识相结合、提高模型稳健性和改进评估方法的机会。通过综合趋势和确定研究差距，本研究旨在指导未来 LLM 在定性分析中的应用创新。

Title: StreetviewLLM: Extracting Geographic Information Using a Chain-of-Thought Multimodal Large Language Model

• Authors: Zongrong Li, Junhao Xu, Siqin Wang, Yifan Wu, Haiyang Li
• Subjects: cs.CL, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2411.14476
• Pdf URL: https://arxiv.org/pdf/2411.14476
• Copy Paste: [[2411.14476]] StreetviewLLM: Extracting Geographic Information Using a Chain-of-Thought Multimodal Large Language Model(https://arxiv.org/abs/2411.14476)
• Keywords: language model, llm, retrieval-augmented generation, chain-of-thought
• Abstract: Geospatial predictions are crucial for diverse fields such as disaster management, urban planning, and public health. Traditional machine learning methods often face limitations when handling unstructured or multi-modal data like street view imagery. To address these challenges, we propose StreetViewLLM, a novel framework that integrates a large language model with the chain-of-thought reasoning and multimodal data sources. By combining street view imagery with geographic coordinates and textual data, StreetViewLLM improves the precision and granularity of geospatial predictions. Using retrieval-augmented generation techniques, our approach enhances geographic information extraction, enabling a detailed analysis of urban environments. The model has been applied to seven global cities, including Hong Kong, Tokyo, Singapore, Los Angeles, New York, London, and Paris, demonstrating superior performance in predicting urban indicators, including population density, accessibility to healthcare, normalized difference vegetation index, building height, and impervious surface. The results show that StreetViewLLM consistently outperforms baseline models, offering improved predictive accuracy and deeper insights into the built environment. This research opens new opportunities for integrating the large language model into urban analytics, decision-making in urban planning, infrastructure management, and environmental monitoring.
• 摘要：地理空间预测对于灾害管理、城市规划和公共卫生等不同领域都至关重要。传统的机器学习方法在处理街景图像等非结构化或多模态数据时往往面临限制。为了应对这些挑战，我们提出了 StreetViewLLM，这是一个新颖的框架，它将大型语言模型与思路链推理和多模态数据源相结合。通过将街景图像与地理坐标和文本数据相结合，StreetViewLLM 提高了地理空间预测的精度和粒度。使用检索增强生成技术，我们的方法增强了地理信息提取，从而能够对城市环境进行详细分析。该模型已应用于香港、东京、新加坡、洛杉矶、纽约、伦敦和巴黎等七个全球城市，在预测城市指标方面表现出色，包括人口密度、医疗保健的可及性、归一化差异植被指数、建筑物高度和不透水表面。结果表明，StreetViewLLM 始终优于基线模型，提供了更高的预测准确性和对建筑环境的更深入洞察。这项研究为将大型语言模型融入城市分析、城市规划决策、基础设施管理和环境监测开辟了新的机遇。

Title: GRL-Prompt: Towards Knowledge Graph based Prompt Optimization via Reinforcement Learning

• Authors: Yuze Liu, Tingjie Liu, Tiehua Zhang, Youhua Xia, Jinze Wang, Zhishu Shen, Jiong Jin, Fei Richard Yu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.14479
• Pdf URL: https://arxiv.org/pdf/2411.14479
• Copy Paste: [[2411.14479]] GRL-Prompt: Towards Knowledge Graph based Prompt Optimization via Reinforcement Learning(https://arxiv.org/abs/2411.14479)
• Keywords: language model, llm, prompt
• Abstract: Large language models (LLMs) have demonstrated impressive success in a wide range of natural language processing (NLP) tasks due to their extensive general knowledge of the world. Recent works discovered that the performance of LLMs is heavily dependent on the input prompt. However, prompt engineering is usually done manually in a trial-and-error fashion, which can be labor-intensive and challenging in order to find the optimal prompts. To address these problems and unleash the utmost potential of LLMs, we propose a novel LLMs-agnostic framework for prompt optimization, namely GRL-Prompt, which aims to automatically construct optimal prompts via reinforcement learning (RL) in an end-to-end manner. To provide structured action/state representation for optimizing prompts, we construct a knowledge graph (KG) that better encodes the correlation between the user query and candidate in-context examples. Furthermore, a policy network is formulated to generate the optimal action by selecting a set of in-context examples in a rewardable order to construct the prompt. Additionally, the embedding-based reward shaping is utilized to stabilize the RL training process. The experimental results show that GRL-Prompt outperforms recent state-of-the-art methods, achieving an average increase of 0.10 in ROUGE-1, 0.07 in ROUGE-2, 0.07 in ROUGE-L, and 0.05 in BLEU.
• 摘要：大型语言模型 (LLM) 因其对世界的广泛一般知识而在广泛的自然语言处理 (NLP) 任务中取得了令人瞩目的成功。最近的研究发现，LLM 的性能在很大程度上取决于输入提示。然而，提示工程通常是以反复试验的方式手动完成的，为了找到最佳提示，这可能是一项耗费大量人力且具有挑战性的工作。为了解决这些问题并最大限度地发挥 LLM 的潜力，我们提出了一种新颖的 LLM 无关提示优化框架，即 GRL-Prompt，旨在通过强化学习 (RL) 以端到端的方式自动构建最佳提示。为了为优化提示提供结构化的动作/状态表示，我们构建了一个知识图谱 (KG)，可以更好地编码用户查询和候选上下文示例之间的相关性。此外，制定了一个策略网络，通过按可奖励顺序选择一组上下文示例来构建提示，以生成最佳动作。此外，还利用基于嵌入的奖励塑造来稳定强化学习训练过程。实验结果表明，GRL-Prompt 的表现优于近期最先进的方法，在 ROUGE-1 中平均提高了 0.10，在 ROUGE-2 中提高了 0.07，在 ROUGE-L 中提高了 0.07，在 BLEU 中提高了 0.05。

Title: Ranking Unraveled: Recipes for LLM Rankings in Head-to-Head AI Combat

• Authors: Roland Daynauth, Christopher Clarke, Krisztian Flautner, Lingjia Tang, Jason Mars
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.14483
• Pdf URL: https://arxiv.org/pdf/2411.14483
• Copy Paste: [[2411.14483]] Ranking Unraveled: Recipes for LLM Rankings in Head-to-Head AI Combat(https://arxiv.org/abs/2411.14483)
• Keywords: language model, llm
• Abstract: Deciding which large language model (LLM) to use is a complex challenge. Pairwise ranking has emerged as a new method for evaluating human preferences for LLMs. This approach entails humans evaluating pairs of model outputs based on a predefined criterion. By collecting these comparisons, a ranking can be constructed using methods such as Elo. However, applying these algorithms as constructed in the context of LLM evaluation introduces several challenges. In this paper, we explore the effectiveness of ranking systems for head-to-head comparisons of LLMs. We formally define a set of fundamental principles for effective ranking and conduct a series of extensive evaluations on the robustness of several ranking algorithms in the context of LLMs. Our analysis uncovers key insights into the factors that affect ranking accuracy and efficiency, offering guidelines for selecting the most appropriate methods based on specific evaluation contexts and resource constraints.
• 摘要：决定使用哪种大型语言模型 (LLM) 是一项复杂的挑战。成对排名已成为评估人类对 LLM 偏好的一种新方法。这种方法需要人类根据预定义的标准评估模型输出对。通过收集这些比较，可以使用 Elo 等方法构建排名。但是，在 LLM 评估的背景下应用构建的这些算法会带来一些挑战。在本文中，我们探讨了排名系统对 LLM 进行头对头比较的有效性。我们正式定义了一组有效排名的基本原则，并在 LLM 背景下对几种排名算法的稳健性进行了一系列广泛的评估。我们的分析揭示了影响排名准确性和效率的因素的关键见解，为根据特定的评估环境和资源限制选择最合适的方法提供了指导。

Title: Robust Planning with Compound LLM Architectures: An LLM-Modulo Approach

• Authors: Atharva Gundawar, Karthik Valmeekam, Mudit Verma, Subbarao Kambhampati
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.14484
• Pdf URL: https://arxiv.org/pdf/2411.14484
• Copy Paste: [[2411.14484]] Robust Planning with Compound LLM Architectures: An LLM-Modulo Approach(https://arxiv.org/abs/2411.14484)
• Keywords: language model, llm, prompt
• Abstract: Previous work has attempted to boost Large Language Model (LLM) performance on planning and scheduling tasks through a variety of prompt engineering techniques. While these methods can work within the distributions tested, they are neither robust nor predictable. This limitation can be addressed through compound LLM architectures where LLMs work in conjunction with other components to ensure reliability. In this paper, we present a technical evaluation of a compound LLM architecture--the LLM-Modulo framework. In this framework, an LLM is paired with a complete set of sound verifiers that validate its output, re-prompting it if it fails. This approach ensures that the system can never output any fallacious output, and therefore that every output generated is guaranteed correct--something previous techniques have not been able to claim. Our results, evaluated across four scheduling domains, demonstrate significant performance gains with the LLM-Modulo framework using various models. Additionally, we explore modifications to the base configuration of the framework and assess their impact on overall system performance.
• 摘要：先前的研究尝试通过各种快速工程技术来提高大型语言模型 (LLM) 在规划和调度任务方面的性能。虽然这些方法可以在测试的分布中工作，但它们既不稳健也不可预测。可以通过复合 LLM 架构来解决此限制，其中 LLM 与其他组件协同工作以确保可靠性。在本文中，我们对复合 LLM 架构——LLM-Modulo 框架进行了技术评估。在此框架中，LLM 与一整套可靠的验证器配对，用于验证其输出，如果失败则重新提示。这种方法确保系统永远不会输出任何错误的输出，因此生成的每个输出都保证正确——这是以前的技术无法声称的。我们在四个调度域中评估的结果显示，使用各种模型的 LLM-Modulo 框架具有显着的性能提升。此外，我们探索了对框架基本配置的修改，并评估了它们对整体系统性能的影响。

Title: Mediating Modes of Thought: LLM's for design scripting

• Authors: Moritz Rietschel, Fang Guo, Kyle Steinfeld
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2411.14485
• Pdf URL: https://arxiv.org/pdf/2411.14485
• Copy Paste: [[2411.14485]] Mediating Modes of Thought: LLM's for design scripting(https://arxiv.org/abs/2411.14485)
• Keywords: language model, llm, prompt, agent
• Abstract: Here is an updated version of your abstract, cleaned for submission to arXiv with potential "bad characters" corrected to conform to ASCII standards: Architects adopt visual scripting and parametric design tools to explore more expansive design spaces (Coates, 2010), refine their thinking about the geometric logic of their design (Woodbury, 2010), and overcome conventional software limitations (Burry, 2011). Despite two decades of effort to make design scripting more accessible, a disconnect between a designer's free ways of thinking and the rigidity of algorithms remains (Burry, 2011). Recent developments in Large Language Models (LLMs) suggest this might soon change, as LLMs encode a general understanding of human context and exhibit the capacity to produce geometric logic. This project speculates that if LLMs can effectively mediate between user intent and algorithms, they become a powerful tool to make scripting in design more widespread and fun. We explore if such systems can interpret natural language prompts to assemble geometric operations relevant to computational design scripting. In the system, multiple layers of LLM agents are configured with specific context to infer the user intent and construct a sequential logic. Given a user's high-level text prompt, a geometric description is created, distilled into a sequence of logic operations, and mapped to software-specific commands. The completed script is constructed in the user's visual programming interface. The system succeeds in generating complete visual scripts up to a certain complexity but fails beyond this complexity threshold. It shows how LLMs can make design scripting much more aligned with human creativity and thought. Future research should explore conversational interactions, expand to multimodal inputs and outputs, and assess the performance of these tools.
• 摘要：这是您摘要的更新版本，已清理以提交给 arXiv，并更正了潜在的“坏字符”，以符合 ASCII 标准：建筑师采用可视化脚本和参数化设计工具来探索更广阔的设计空间（Coates，2010 年），改进他们对设计几何逻辑的思考（Woodbury，2010 年），并克服传统软件的限制（Burry，2011 年）。尽管经过了二十年的努力，使设计脚本更容易使用，但设计师的自由思维方式与算法的僵化之间仍然存在脱节（Burry，2011 年）。大型语言模型 (LLM) 的最新发展表明这种情况可能很快就会改变，因为 LLM 编码了对人类背景的一般理解并表现出产生几何逻辑的能力。该项目推测，如果 LLM 能够有效地在用户意图和算法之间进行调解，它们将成为使设计脚本更加普及和有趣的强大工具。我们探索此类系统是否可以解释自然语言提示以组装与计算设计脚本相关的几何操作。在系统中，多层 LLM 代理配置了特定的上下文，以推断用户意图并构建顺序逻辑。给定用户的高级文本提示，将创建几何描述，提炼为一系列逻辑运算，并映射到特定于软件的命令。完成的脚本在用户的可视化编程界面中构建。系统成功生成了具有一定复杂度的完整可视化脚本，但超出此复杂度阈值则失败。它展示了 LLM 如何使设计脚本更符合人类的创造力和思维。未来的研究应该探索对话式交互，扩展到多模式输入和输出，并评估这些工具的性能。

Title: The Impossible Test: A 2024 Unsolvable Dataset and A Chance for an AGI Quiz

• Authors: David Noever, Forrest McKee
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.14486
• Pdf URL: https://arxiv.org/pdf/2411.14486
• Copy Paste: [[2411.14486]] The Impossible Test: A 2024 Unsolvable Dataset and A Chance for an AGI Quiz(https://arxiv.org/abs/2411.14486)
• Keywords: language model, gpt, llm
• Abstract: This research introduces a novel evaluation framework designed to assess large language models' (LLMs) ability to acknowledge uncertainty on 675 fundamentally unsolvable problems. Using a curated dataset of graduate-level grand challenge questions with intentionally unknowable answers, we evaluated twelve state-of-the-art LLMs, including both open and closed-source models, on their propensity to admit ignorance rather than generate plausible but incorrect responses. The best models scored in 62-68% accuracy ranges for admitting the problem solution was unknown in fields ranging from biology to philosophy and mathematics. We observed an inverse relationship between problem difficulty and model accuracy, with GPT-4 demonstrating higher rates of uncertainty acknowledgment on more challenging problems (35.8%) compared to simpler ones (20.0%). This pattern indicates that models may be more prone to generate speculative answers when problems appear more tractable. The study also revealed significant variations across problem categories, with models showing difficulty in acknowledging uncertainty in invention and NP-hard problems while performing relatively better on philosophical and psychological challenges. These results contribute to the growing body of research on artificial general intelligence (AGI) assessment by highlighting the importance of uncertainty recognition as a critical component of future machine intelligence evaluation. This impossibility test thus extends previous theoretical frameworks for universal intelligence testing by providing empirical evidence of current limitations in LLMs' ability to recognize their own knowledge boundaries, suggesting new directions for improving model training architectures and evaluation approaches.
• 摘要：本研究引入了一种新颖的评估框架，旨在评估大型语言模型 (LLM) 承认 675 个根本无法解决的问题的不确定性的能力。使用一个精选的研究生级大挑战问题数据集，这些问题的答案故意不可知，我们评估了 12 个最先进的 LLM，包括开源和闭源模型，以评估它们承认无知而不是产生合理但不正确的答案的倾向。在从生物学到哲学和数学的各个领域，最好的模型在承认问题解决方案未知方面的准确率在 62-68% 之间。我们观察到问题难度和模型准确率之间存在反比关系，GPT-4 对更具挑战性的问题 (35.8%) 表现出比简单问题 (20.0%) 更高的不确定性承认率。这种模式表明，当问题看起来更容易处理时，模型可能更容易产生推测性答案。该研究还揭示了不同问题类别之间的显著差异，模型在承认发明和 NP 难题中的不确定性方面表现出困难，而在哲学和心理挑战方面表现相对较好。这些结果突出了不确定性识别作为未来机器智能评估的关键组成部分的重要性，为日益增多的通用人工智能 (AGI) 评估研究做出了贡献。因此，这种不可能性测试扩展了以前的通用智力测试理论框架，提供了 LLM 识别自身知识边界的能力目前存在局限性的经验证据，为改进模型训练架构和评估方法提出了新的方向。

Title: Ensuring Safety and Trust: Analyzing the Risks of Large Language Models in Medicine

• Authors: Yifan Yang, Qiao Jin, Robert Leaman, Xiaoyu Liu, Guangzhi Xiong, Maame Sarfo-Gyamfi, Changlin Gong, Santiago Ferrière-Steinert, W. John Wilbur, Xiaojun Li, Jiaxin Yuan, Bang An, Kelvin S. Castro, Francisco Erramuspe Álvarez, Matías Stockle, Aidong Zhang, Furong Huang, Zhiyong Lu
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2411.14487
• Pdf URL: https://arxiv.org/pdf/2411.14487
• Copy Paste: [[2411.14487]] Ensuring Safety and Trust: Analyzing the Risks of Large Language Models in Medicine(https://arxiv.org/abs/2411.14487)
• Keywords: language model, gpt, llm, chat
• Abstract: The remarkable capabilities of Large Language Models (LLMs) make them increasingly compelling for adoption in real-world healthcare applications. However, the risks associated with using LLMs in medical applications have not been systematically characterized. We propose using five key principles for safe and trustworthy medical AI: Truthfulness, Resilience, Fairness, Robustness, and Privacy, along with ten specific aspects. Under this comprehensive framework, we introduce a novel MedGuard benchmark with 1,000 expert-verified questions. Our evaluation of 11 commonly used LLMs shows that the current language models, regardless of their safety alignment mechanisms, generally perform poorly on most of our benchmarks, particularly when compared to the high performance of human physicians. Despite recent reports indicate that advanced LLMs like ChatGPT can match or even exceed human performance in various medical tasks, this study underscores a significant safety gap, highlighting the crucial need for human oversight and the implementation of AI safety guardrails.
• 摘要：大型语言模型 (LLM) 的卓越能力使其在现实世界的医疗保健应用中越来越引人注目。然而，在医疗应用中使用 LLM 的风险尚未得到系统性的描述。我们建议使用五项安全可靠的医疗 AI 关键原则：真实性、弹性、公平性、稳健性和隐私性，以及十个具体方面。在这个综合框架下，我们引入了一个包含 1,000 个专家验证问题的新 MedGuard 基准。我们对 11 种常用 LLM 的评估表明，当前的语言模型，无论其安全对齐机制如何，在大多数基准测试中通常表现不佳，尤其是与人类医生的高性能相比时。尽管最近的报告表明，像 ChatGPT 这样的高级 LLM 可以在各种医疗任务中匹敌甚至超过人类的表现，但这项研究强调了巨大的安全差距，凸显了对人类监督和实施 AI 安全护栏的迫切需要。

Title: A Survey on Human-Centric LLMs

• Authors: Jing Yi Wang, Nicholas Sukiennik, Tong Li, Weikang Su, Qianyue Hao, Jingbo Xu, Zihan Huang, Fengli Xu, Yong Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.14491
• Pdf URL: https://arxiv.org/pdf/2411.14491
• Copy Paste: [[2411.14491]] A Survey on Human-Centric LLMs(https://arxiv.org/abs/2411.14491)
• Keywords: language model, llm
• Abstract: The rapid evolution of large language models (LLMs) and their capacity to simulate human cognition and behavior has given rise to LLM-based frameworks and tools that are evaluated and applied based on their ability to perform tasks traditionally performed by humans, namely those involving cognition, decision-making, and social interaction. This survey provides a comprehensive examination of such human-centric LLM capabilities, focusing on their performance in both individual tasks (where an LLM acts as a stand-in for a single human) and collective tasks (where multiple LLMs coordinate to mimic group dynamics). We first evaluate LLM competencies across key areas including reasoning, perception, and social cognition, comparing their abilities to human-like skills. Then, we explore real-world applications of LLMs in human-centric domains such as behavioral science, political science, and sociology, assessing their effectiveness in replicating human behaviors and interactions. Finally, we identify challenges and future research directions, such as improving LLM adaptability, emotional intelligence, and cultural sensitivity, while addressing inherent biases and enhancing frameworks for human-AI collaboration. This survey aims to provide a foundational understanding of LLMs from a human-centric perspective, offering insights into their current capabilities and potential for future development.
• 摘要：大型语言模型 (LLM) 的快速发展及其模拟人类认知和行为的能力催生了基于 LLM 的框架和工具，这些框架和工具的评估和应用基于它们执行传统上由人类执行的任务的能力，即涉及认知、决策和社交互动的任务。本调查全面考察了这种以人为本的 LLM 能力，重点关注它们在个人任务（其中 LLM 充当单个人的替身）和集体任务（其中多个 LLM 协调以模拟群体动态）中的表现。我们首先评估 LLM 在推理、感知和社会认知等关键领域的能力，并将其能力与类似人类的技能进行比较。然后，我们探索 LLM 在以人为本的领域（例如行为科学、政治学和社会学）的实际应用，评估它们在复制人类行为和互动方面的有效性。最后，我们确定了挑战和未来的研究方向，例如提高 LLM 的适应性、情商和文化敏感性，同时解决固有偏见并增强人机协作框架。本次调查旨在从以人为本的角度对法学硕士 (LLM) 提供基础性的理解，深入了解其当前的能力和未来发展的潜力。

Title: Star-Agents: Automatic Data Optimization with LLM Agents for Instruction Tuning

• Authors: Hang Zhou, Yehui Tang, Haochen Qin, Yujie Yang, Renren Jin, Deyi Xiong, Kai Han, Yunhe Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.14497
• Pdf URL: https://arxiv.org/pdf/2411.14497
• Copy Paste: [[2411.14497]] Star-Agents: Automatic Data Optimization with LLM Agents for Instruction Tuning(https://arxiv.org/abs/2411.14497)
• Keywords: language model, llm, agent
• Abstract: The efficacy of large language models (LLMs) on downstream tasks usually hinges on instruction tuning, which relies critically on the quality of training data. Unfortunately, collecting high-quality and diverse data is both expensive and time-consuming. To mitigate this issue, we propose a novel Star-Agents framework, which automates the enhancement of data quality across datasets through multi-agent collaboration and assessment. The framework adopts a three-pronged strategy. It initially generates diverse instruction data with multiple LLM agents through a bespoke sampling method. Subsequently, the generated data undergo a rigorous evaluation using a dual-model method that assesses both difficulty and quality. Finaly, the above process evolves in a dynamic refinement phase, where more effective LLMs are prioritized, enhancing the overall data quality. Our empirical studies, including instruction tuning experiments with models such as Pythia and LLaMA, demonstrate the effectiveness of the proposed framework. Optimized datasets have achieved substantial improvements, with an average increase of 12% and notable gains in specific metrics, such as a 40% improvement in Fermi, as evidenced by benchmarks like MT-bench, Vicuna bench, and WizardLM testset.
• 摘要：大型语言模型 (LLM) 对下游任务的有效性通常取决于指令调整，而指令调整主要依赖于训练数据的质量。不幸的是，收集高质量和多样化的数据既昂贵又耗时。为了缓解这个问题，我们提出了一种新颖的 Star-Agents 框架，该框架通过多代理协作和评估自动提高整个数据集的数据质量。该框架采用三管齐下的策略。它首先通过定制的采样方法使用多个 LLM 代理生成多样化的指令数据。随后，使用双模型方法对生成的数据进行严格评估，该方法同时评估难度和质量。最后，上述过程在动态细化阶段发展，其中优先考虑更有效的 LLM，从而提高整体数据质量。我们的实证研究，包括使用 Pythia 和 LLaMA 等模型进行的指令调整实验，证明了所提框架的有效性。优化后的数据集取得了实质性的提升，平均提升了 12%，具体指标也有显著提升，如 Fermi 提升了 40%，这在 MT-bench、Vicuna bench 和 WizardLM 测试集等基准测试中都有所体现。

Title: Understanding World or Predicting Future? A Comprehensive Survey of World Models

• Authors: Jingtao Ding, Yunke Zhang, Yu Shang, Yuheng Zhang, Zefang Zong, Jie Feng, Yuan Yuan, Hongyuan Su, Nian Li, Nicholas Sukiennik, Fengli Xu, Yong Li
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.14499
• Pdf URL: https://arxiv.org/pdf/2411.14499
• Copy Paste: [[2411.14499]] Understanding World or Predicting Future? A Comprehensive Survey of World Models(https://arxiv.org/abs/2411.14499)
• Keywords: language model, gpt
• Abstract: The concept of world models has garnered significant attention due to advancements in multimodal large language models such as GPT-4 and video generation models such as Sora, which are central to the pursuit of artificial general intelligence. This survey offers a comprehensive review of the literature on world models. Generally, world models are regarded as tools for either understanding the present state of the world or predicting its future dynamics. This review presents a systematic categorization of world models, emphasizing two primary functions: (1) constructing internal representations to understand the mechanisms of the world, and (2) predicting future states to simulate and guide decision-making. Initially, we examine the current progress in these two categories. We then explore the application of world models in key domains, including autonomous driving, robotics, and social simulacra, with a focus on how each domain utilizes these aspects. Finally, we outline key challenges and provide insights into potential future research directions.
• 摘要：由于多模态大型语言模型（如 GPT-4）和视频生成模型（如 Sora）的进步，世界模型的概念引起了广泛关注，这些模型对于追求通用人工智能至关重要。本综述对世界模型的文献进行了全面的回顾。一般来说，世界模型被视为了解世界现状或预测世界未来动态的工具。本综述对世界模型进行了系统分类，强调了两个主要功能：（1）构建内部表示以了解世界的机制，（2）预测未来状态以模拟和指导决策。首先，我们研究这两个类别的当前进展。然后，我们探讨世界模型在关键领域的应用，包括自动驾驶、机器人和社会模拟，重点关注每个领域如何利用这些方面。最后，我们概述了关键挑战并对未来的潜在研究方向提供了见解。

Title: Exploring Accuracy-Fairness Trade-off in Large Language Models

• Authors: Qingquan Zhang, Qiqi Duan, Bo Yuan, Yuhui Shi, Jialin Liu
• Subjects: cs.CL, cs.AI, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.14500
• Pdf URL: https://arxiv.org/pdf/2411.14500
• Copy Paste: [[2411.14500]] Exploring Accuracy-Fairness Trade-off in Large Language Models(https://arxiv.org/abs/2411.14500)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have made significant strides in the field of artificial intelligence, showcasing their ability to interact with humans and influence human cognition through information dissemination. However, recent studies have brought to light instances of bias inherent within these LLMs, presenting a critical issue that demands attention. In our research, we delve deeper into the intricate challenge of harmonising accuracy and fairness in the enhancement of LLMs. While improving accuracy can indeed enhance overall LLM performance, it often occurs at the expense of fairness. Overemphasising optimisation of one metric invariably leads to a significant degradation of the other. This underscores the necessity of taking into account multiple considerations during the design and optimisation phases of LLMs. Therefore, we advocate for reformulating the LLM training process as a multi-objective learning task. Our investigation reveals that multi-objective evolutionary learning (MOEL) methodologies offer promising avenues for tackling this challenge. Our MOEL framework enables the simultaneous optimisation of both accuracy and fairness metrics, resulting in a Pareto-optimal set of LLMs. In summary, our study sheds valuable lights on the delicate equilibrium between accuracy and fairness within LLMs, which is increasingly significant for their real-world applications. By harnessing MOEL, we present a promising pathway towards fairer and more efficacious AI technologies.
• 摘要：大型语言模型 (LLM) 在人工智能领域取得了重大进展，展示了它们与人类互动并通过信息传播影响人类认知的能力。然而，最近的研究揭示了这些 LLM 中固有的偏见，提出了一个需要关注的关键问题。在我们的研究中，我们深入研究了在 LLM 增强过程中协调准确性和公平性的复杂挑战。虽然提高准确性确实可以提高整体 LLM 性能，但它往往以牺牲公平性为代价。过分强调一个指标的优化必然会导致另一个指标的显著下降。这强调了在 LLM 的设计和优化阶段考虑多种因素的必要性。因此，我们主张将 LLM 训练过程重新表述为多目标学习任务。我们的调查显示，多目标进化学习 (MOEL) 方法为应对这一挑战提供了有希望的途径。我们的 MOEL 框架能够同时优化准确度和公平度指标，从而产生一组帕累托最优的 LLM。总之，我们的研究为 LLM 中准确度和公平度之间的微妙平衡提供了宝贵的启示，这对其实际应用越来越重要。通过利用 MOEL，我们提出了一条通往更公平、更有效的人工智能技术的有希望的道路。

Title: Comparative Analysis of Pooling Mechanisms in LLMs: A Sentiment Analysis Perspective

• Authors: Jinming Xing, Ruilin Xing, Yan Sun
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.14654
• Pdf URL: https://arxiv.org/pdf/2411.14654
• Copy Paste: [[2411.14654]] Comparative Analysis of Pooling Mechanisms in LLMs: A Sentiment Analysis Perspective(https://arxiv.org/abs/2411.14654)
• Keywords: language model, gpt, llm, prompt
• Abstract: Large Language Models (LLMs) have revolutionized natural language processing (NLP) by delivering state-of-the-art performance across a variety of tasks. Among these, Transformer-based models like BERT and GPT rely on pooling layers to aggregate token-level embeddings into sentence-level representations. Common pooling mechanisms such as Mean, Max, and Weighted Sum play a pivotal role in this aggregation process. Despite their widespread use, the comparative performance of these strategies on different LLM architectures remains underexplored. To address this gap, this paper investigates the effects of these pooling mechanisms on two prominent LLM families -- BERT and GPT, in the context of sentence-level sentiment analysis. Comprehensive experiments reveal that each pooling mechanism exhibits unique strengths and weaknesses depending on the task's specific requirements. Our findings underline the importance of selecting pooling methods tailored to the demands of particular applications, prompting a re-evaluation of common assumptions regarding pooling operations. By offering actionable insights, this study contributes to the optimization of LLM-based models for downstream tasks.
• 摘要：大型语言模型 (LLM) 通过在各种任务中提供最先进的性能，彻底改变了自然语言处理 (NLP)。其中，基于 Transformer 的模型（如 BERT 和 GPT）依赖于池化层将 token 级嵌入聚合到句子级表示中。常见的池化机制（如平均值、最大值和加权总和）在此聚合过程中起着关键作用。尽管它们被广泛使用，但这些策略在不同 LLM 架构上的比较性能仍未得到充分探索。为了解决这一差距，本文在句子级情绪分析的背景下研究了这些池化机制对两个著名的 LLM 系列 BERT 和 GPT 的影响。综合实验表明，每种池化机制都表现出独特的优势和劣势，具体取决于任务的具体要求。我们的研究结果强调了根据特定应用的需求选择池化方法的重要性，这促使我们重新评估有关池化操作的常见假设。通过提供可行的见解，本研究有助于优化基于 LLM 的模型以完成下游任务。

Title: Multiverse of Greatness: Generating Story Branches with LLMs

• Authors: Pittawat Taveekitworachai, Chollakorn Nimpattanavong, Mustafa Can Gursesli, Antonio Lanata, Andrea Guazzini, Ruck Thawonmas
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.14672
• Pdf URL: https://arxiv.org/pdf/2411.14672
• Copy Paste: [[2411.14672]] Multiverse of Greatness: Generating Story Branches with LLMs(https://arxiv.org/abs/2411.14672)
• Keywords: llm, prompt
• Abstract: This paper presents Dynamic Context Prompting/Programming (DCP/P), a novel framework for interacting with LLMs to generate graph-based content with a dynamic context window history. While there is an existing study utilizing LLMs to generate a visual novel game, the previous study involved a manual process of output extraction and did not provide flexibility in generating a longer, coherent story. We evaluate DCP/P against our baseline, which does not provide context history to an LLM and only relies on the initial story data. Through objective evaluation, we show that simply providing the LLM with a summary leads to a subpar story compared to additionally providing the LLM with the proper context of the story. We also provide an extensive qualitative analysis and discussion. We qualitatively examine the quality of the objectively best-performing generated game from each approach. In addition, we examine biases in word choices and word sentiment of the generated content. We find a consistent observation with previous studies that LLMs are biased towards certain words, even with a different LLM family. Finally, we provide a comprehensive discussion on opportunities for future studies.
• 摘要：本文介绍了动态上下文提示/编程 (DCP/P)，这是一种与 LLM 交互的新型框架，用于生成具有动态上下文窗口历史记录的基于图形的内容。虽然目前有一项研究利用 LLM 生成视觉小说游戏，但之前的研究涉及手动输出提取过程，无法灵活地生成更长、更连贯的故事。我们根据基线评估 DCP/P，它不向 LLM 提供上下文历史，仅依赖于初始故事数据。通过客观评估，我们表明，与为 LLM 提供故事的适当背景相比，仅向 LLM 提供摘要会导致故事质量不佳。我们还提供了广泛的定性分析和讨论。我们定性地检查了每种方法中客观表现最佳的生成游戏的质量。此外，我们检查了生成内容的单词选择和单词情绪的偏见。我们发现与以前的研究一致的观察结果是，即使使用不同的 LLM 系列，LLM 也会偏向某些单词。最后，我们对未来研究的机会进行了全面的讨论。

Title: Improving Mathematical Reasoning Capabilities of Small Language Models via Feedback-Driven Distillation

• Authors: Xunyu Zhu, Jian Li, Can Ma, Weiping Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.14698
• Pdf URL: https://arxiv.org/pdf/2411.14698
• Copy Paste: [[2411.14698]] Improving Mathematical Reasoning Capabilities of Small Language Models via Feedback-Driven Distillation(https://arxiv.org/abs/2411.14698)
• Keywords: language model, llm, prompt
• Abstract: Large Language Models (LLMs) demonstrate exceptional reasoning capabilities, often achieving state-of-the-art performance in various tasks. However, their substantial computational and memory demands, due to billions of parameters, hinder deployment in resource-constrained environments. A promising solution is knowledge distillation, where LLMs transfer reasoning capabilities to Small Language Models (SLMs, $\le$ 1B parameters), enabling wider deployment on low-resource devices. Existing methods primarily focus on generating high-quality reasoning rationales for distillation datasets but often neglect the critical role of data quantity and quality. To address these challenges, we propose a Feedback-Driven Distillation (FDD) framework to enhance SLMs' mathematical reasoning capabilities. In the initialization stage, a distillation dataset is constructed by prompting LLMs to pair mathematical problems with corresponding reasoning rationales. We classify problems into easy and hard categories based on SLM performance. For easy problems, LLMs generate more complex variations, while for hard problems, new questions of similar complexity are synthesized. In addition, we propose a multi-round distillation paradigm to iteratively enrich the distillation datasets, thereby progressively improving the mathematical reasoning abilities of SLMs. Experimental results demonstrate that our method can make SLMs achieve SOTA mathematical reasoning performance.
• 摘要：大型语言模型 (LLM) 表现出卓越的推理能力，通常在各种任务中实现最先进的性能。然而，由于数十亿个参数，它们对计算和内存的需求很大，阻碍了在资源受限的环境中部署。一个有前途的解决方案是知识蒸馏，其中 LLM 将推理能力转移到小型语言模型 (SLM，参数少于 1B)，从而能够在低资源设备上更广泛地部署。现有方法主要侧重于为蒸馏数据集生成高质量的推理原理，但往往忽视了数据数量和质量的关键作用。为了应对这些挑战，我们提出了一个反馈驱动蒸馏 (FDD) 框架来增强 SLM 的数学推理能力。在初始化阶段，通过提示 LLM 将数学问题与相应的推理原理配对来构建蒸馏数据集。我们根据 SLM 性能将问题分为简单和困难类别。对于简单问题，LLM 会生成更复杂的变化，而对于困难问题，则会合成具有类似复杂性的新问题。此外，我们提出了一种多轮蒸馏范式，以迭代方式丰富蒸馏数据集，从而逐步提高 SLM 的数学推理能力。实验结果表明，我们的方法可以使 SLM 达到 SOTA 数学推理性能。

Title: Optimizing Social Media Annotation of HPV Vaccine Skepticism and Misinformation Using Large Language Models: An Experimental Evaluation of In-Context Learning and Fine-Tuning Stance Detection Across Multiple Models

• Authors: Luhang Sun, Varsha Pendyala, Yun-Shiuan Chuang, Shanglin Yang, Jonathan Feldman, Andrew Zhao, Munmun De Choudhury, Sijia Yang, Dhavan Shah
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.14720
• Pdf URL: https://arxiv.org/pdf/2411.14720
• Copy Paste: [[2411.14720]] Optimizing Social Media Annotation of HPV Vaccine Skepticism and Misinformation Using Large Language Models: An Experimental Evaluation of In-Context Learning and Fine-Tuning Stance Detection Across Multiple Models(https://arxiv.org/abs/2411.14720)
• Keywords: language model, gpt, llm, prompt
• Abstract: This paper leverages large-language models (LLMs) to experimentally determine optimal strategies for scaling up social media content annotation for stance detection on HPV vaccine-related tweets. We examine both conventional fine-tuning and emergent in-context learning methods, systematically varying strategies of prompt engineering across widely used LLMs and their variants (e.g., GPT4, Mistral, and Llama3, etc.). Specifically, we varied prompt template design, shot sampling methods, and shot quantity to detect stance on HPV vaccination. Our findings reveal that 1) in general, in-context learning outperforms fine-tuning in stance detection for HPV vaccine social media content; 2) increasing shot quantity does not necessarily enhance performance across models; and 3) different LLMs and their variants present differing sensitivity to in-context learning conditions. We uncovered that the optimal in-context learning configuration for stance detection on HPV vaccine tweets involves six stratified shots paired with detailed contextual prompts. This study highlights the potential and provides an applicable approach for applying LLMs to research on social media stance and skepticism detection.
• 摘要：本文利用大型语言模型 (LLM) 通过实验确定了扩大社交媒体内容注释以检测 HPV 疫苗相关推文立场的最佳策略。我们研究了传统的微调和新兴的上下文学习方法，系统地改变了广泛使用的 LLM 及其变体（例如 GPT4、Mistral 和 Llama3 等）中的提示工程策略。具体来说，我们改变了提示模板设计、镜头采样方法和镜头数量来检测 HPV 疫苗接种的立场。我们的研究结果表明：1）总体而言，上下文学习在 HPV 疫苗社交媒体内容的立场检测中优于微调；2）增加镜头数量并不一定会提高模型间的性能；3）不同的 LLM 及其变体对上下文学习条件的敏感性不同。我们发现，用于 HPV 疫苗推文立场检测的最佳上下文学习配置涉及六个分层镜头与详细的上下文提示配对。这项研究强调了法学硕士 (LLM) 在社交媒体立场和怀疑论检测研究方面的潜力并提供了适用的方法。

Title: MolReFlect: Towards In-Context Fine-grained Alignments between Molecules and Texts

• Authors: Jiatong Li, Yunqing Liu, Wei Liu, Jingdi Le, Di Zhang, Wenqi Fan, Dongzhan Zhou, Yuqiang Li, Qing Li
• Subjects: cs.CL, cs.LG, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2411.14721
• Pdf URL: https://arxiv.org/pdf/2411.14721
• Copy Paste: [[2411.14721]] MolReFlect: Towards In-Context Fine-grained Alignments between Molecules and Texts(https://arxiv.org/abs/2411.14721)
• Keywords: language model, llm, chain-of-thought
• Abstract: Molecule discovery is a pivotal research field, impacting everything from the medicines we take to the materials we use. Recently, Large Language Models (LLMs) have been widely adopted in molecule understanding and generation, yet the alignments between molecules and their corresponding captions remain a significant challenge. Previous endeavours often treat the molecule as a general SMILES string or molecular graph, neglecting the fine-grained alignments between the molecular sub-structures and the descriptive textual phrases, which are crucial for accurate and explainable predictions. In this case, we introduce MolReFlect, a novel teacher-student framework designed to contextually perform the molecule-caption alignments in a fine-grained way. Our approach initially leverages a larger teacher LLM to label the detailed alignments by directly extracting critical phrases from molecule captions or SMILES strings and implying them to corresponding sub-structures or characteristics. To refine these alignments, we propose In-Context Selective Reflection, which retrieves previous extraction results as context examples for teacher LLM to reflect and lets a smaller student LLM select from in-context reflection and previous extraction results. Finally, we enhance the learning process of the student LLM through Chain-of-Thought In-Context Molecule Tuning, integrating the fine-grained alignments and the reasoning processes within the Chain-of-Thought format. Our experimental results demonstrate that MolReFlect enables LLMs like Mistral-7B to significantly outperform the previous baselines, achieving SOTA performance on the ChEBI-20 dataset. This advancement not only enhances the generative capabilities of LLMs in the molecule-caption translation task, but also contributes to a more explainable framework.
• 摘要：分子发现是一个关键的研究领域，影响着从我们服用的药物到我们使用的材料等所有事物。最近，大型语言模型 (LLM) 已广泛应用于分子理解和生成，但分子与其相应标题之间的对齐仍然是一项重大挑战。以前的努力通常将分子视为一般的 SMILES 字符串或分子图，而忽略了分子子结构与描述性文本短语之间的细粒度对齐，而这些对齐对于准确且可解释的预测至关重要。在这种情况下，我们引入了 MolReFlect，这是一种新颖的师生框架，旨在以细粒度的方式根据上下文执行分子标题对齐。我们的方法最初利用更大的教师 LLM 来标记详细对齐，方法是直接从分子标题或 SMILES 字符串中提取关键短语并将其暗示到相应的子结构或特征。为了改进这些对齐，我们提出了上下文选择性反射，它将以前的提取结果检索为教师 LLM 进行反射的上下文示例，并让较小的学生 LLM 从上下文反射和以前的提取结果中进行选择。最后，我们通过思维链上下文分子调整增强了学生 LLM 的学习过程，将细粒度对齐和推理过程整合到思维链格式中。我们的实验结果表明，MolReFlect 使 Mistral-7B 等 LLM 的表现显著优于之前的基线，在 ChEBI-20 数据集上实现了 SOTA 性能。这一进步不仅增强了 LLM 在分子标题翻译任务中的生成能力，而且还有助于建立更易于解释的框架。

Title: Universal and Context-Independent Triggers for Precise Control of LLM Outputs

• Authors: Jiashuo Liang, Guancheng Li, Yang Yu
• Subjects: cs.CL, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2411.14738
• Pdf URL: https://arxiv.org/pdf/2411.14738
• Copy Paste: [[2411.14738]] Universal and Context-Independent Triggers for Precise Control of LLM Outputs(https://arxiv.org/abs/2411.14738)
• Keywords: language model, llm, prompt, agent
• Abstract: Large language models (LLMs) have been widely adopted in applications such as automated content generation and even critical decision-making systems. However, the risk of prompt injection allows for potential manipulation of LLM outputs. While numerous attack methods have been documented, achieving full control over these outputs remains challenging, often requiring experienced attackers to make multiple attempts and depending heavily on the prompt context. Recent advancements in gradient-based white-box attack techniques have shown promise in tasks like jailbreaks and system prompt leaks. Our research generalizes gradient-based attacks to find a trigger that is (1) Universal: effective irrespective of the target output; (2) Context-Independent: robust across diverse prompt contexts; and (3) Precise Output: capable of manipulating LLM inputs to yield any specified output with high accuracy. We propose a novel method to efficiently discover such triggers and assess the effectiveness of the proposed attack. Furthermore, we discuss the substantial threats posed by such attacks to LLM-based applications, highlighting the potential for adversaries to taking over the decisions and actions made by AI agents.
• 摘要：大型语言模型 (LLM) 已广泛应用于自动内容生成甚至关键决策系统等应用。然而，提示注入的风险使得 LLM 输出可能被操纵。虽然已经记录了许多攻击方法，但要完全控制这些输出仍然具有挑战性，通常需要经验丰富的攻击者进行多次尝试，并且严重依赖提示上下文。基于梯度的白盒攻击技术的最新进展已在越狱和系统提示泄漏等任务中显示出希望。我们的研究概括了基于梯度的攻击，以找到一个触发器，该触发器 (1) 通用：无论目标输出如何都有效；(2) 上下文无关：在不同的提示上下文中都具有鲁棒性；(3) 精确输出：能够操纵 LLM 输入以高精度产生任何指定的输出。我们提出了一种新方法来有效地发现此类触发器并评估所提出的攻击的有效性。此外，我们讨论了此类攻击对基于 LLM 的应用程序构成的巨大威胁，强调了对手接管 AI 代理做出的决策和行动的可能性。

Title: KBAda: Efficient Self Adaptation on Specific Knowledge Bases

• Authors: Zheni Zeng, Yuxuan Chen, Shi Yu, Yukun Yan, Zhenghao Liu, Shuo Wang, Xu Han, Zhiyuan Liu, Maosong Sun
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.14790
• Pdf URL: https://arxiv.org/pdf/2411.14790
• Copy Paste: [[2411.14790]] KBAda: Efficient Self Adaptation on Specific Knowledge Bases(https://arxiv.org/abs/2411.14790)
• Keywords: language model, gpt, llm, retrieval-augmented generation
• Abstract: Humans can utilize techniques to quickly acquire knowledge from specific materials in advance, such as creating self-assessment questions, enabling us to achieving related tasks more efficiently. In contrast, large language models (LLMs) usually relies on retrieval-augmented generation to exploit knowledge materials in an instant manner, or requires external signals such as human preference data and stronger LLM annotations to conduct knowledge adaptation. To unleash the self-learning potential of LLMs, we propose KBAda, an approach designed for efficient adaptation to downstream tasks involving knowledge bases. Our method utilizes iterative training with self-annotated data such as Q&A pairs and revision suggestions, enabling the model to grasp the knowledge content efficiently. Experimental results on multiple datasets demonstrate the effectiveness of our approach, significantly boosting model performance in downstream tasks that require specific knowledge at a low cost. Notably, our approach achieves over 90% of the performance improvement that can be obtained by using GPT-4-turbo annotation, while relying entirely on self-supervision. We release our experimental data, models, and process analyses to the community for further exploration (this https URL).
• 摘要：人类可以利用技巧预先从特定材料中快速获取知识，例如创建自我评估问题，从而使我们能够更有效地完成相关任务。相比之下，大型语言模型（LLM）通常依赖于检索增强生成来即时利用知识材料，或者需要外部信号（例如人类偏好数据和更强大的 LLM 注释）来进行知识适应。为了释放 LLM 的自学习潜力，我们提出了 KBAda，这是一种旨在有效适应涉及知识库的下游任务的方法。我们的方法利用自注释数据（例如问答对和修订建议）进行迭代训练，使模型能够有效地掌握知识内容。在多个数据集上的实验结果证明了我们方法的有效性，以低成本显着提升了需要特定知识的下游任务的模型性能。值得注意的是，我们的方法实现了使用 GPT-4-turbo 注释可以获得的 90% 以上的性能提升，同时完全依赖于自监督。我们向社区发布我们的实验数据、模型和流程分析，以供进一步探索（此 https URL）。

Title: De-biased Multimodal Electrocardiogram Analysis

• Authors: Haitao Li, Ziyu Li, Yiheng Mao, Ziyi Liu, Zhoujian Sun, Zhengxing Huang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.14795
• Pdf URL: https://arxiv.org/pdf/2411.14795
• Copy Paste: [[2411.14795]] De-biased Multimodal Electrocardiogram Analysis(https://arxiv.org/abs/2411.14795)
• Keywords: language model, llm
• Abstract: Multimodal large language models (MLLMs) are increasingly being applied in the medical field, particularly in medical imaging. However, developing MLLMs for ECG signals, which are crucial in clinical settings, has been a significant challenge beyond medical imaging. Previous studies have attempted to address this by converting ECGs into several text tags using an external classifier in a training-free manner. However, this approach significantly compresses the information in ECGs and underutilizes the reasoning capabilities of LLMs. In this work, we directly feed the embeddings of ECGs into the LLM through a projection layer, retaining more information about ECGs and better leveraging the reasoning abilities of LLMs. Our method can also effectively handle a common situation in clinical practice where it is necessary to compare two ECGs taken at different times. Recent studies found that MLLMs may rely solely on text input to provide answers, ignoring inputs from other modalities. We analyzed this phenomenon from a causal perspective in the context of ECG MLLMs and discovered that the confounder, severity of illness, introduces a spurious correlation between the question and answer, leading the model to rely on this spurious correlation and ignore the ECG input. Such models do not comprehend the ECG input and perform poorly in adversarial tests where different expressions of the same question are used in the training and testing sets. We designed a de-biased pre-training method to eliminate the confounder's effect according to the theory of backdoor adjustment. Our model performed well on the ECG-QA task under adversarial testing and demonstrated zero-shot capabilities. An interesting random ECG test further validated that our model effectively understands and utilizes the input ECG signal.
• 摘要：多模态大型语言模型 (MLLM) 越来越多地应用于医学领域，尤其是医学成像领域。然而，开发用于临床环境中至关重要的 ECG 信号的 MLLM 一直是医学成像之外的一项重大挑战。先前的研究试图通过使用外部分类器以无需训练的方式将 ECG 转换为多个文本标签来解决这个问题。然而，这种方法大大压缩了 ECG 中的信息，并且没有充分利用 LLM 的推理能力。在这项工作中，我们通过投影层将 ECG 的嵌入直接输入到 LLM 中，保留了更多关于 ECG 的信息，并更好地利用了 LLM 的推理能力。我们的方法还可以有效处理临床实践中常见的情况，即需要比较在不同时间拍摄的两个 ECG。最近的研究发现，MLLM 可能仅依赖文本输入来提供答案，而忽略了来自其他模态的输入。我们从 ECG MLLM 背景下的因果角度分析了这种现象，发现混杂因素疾病严重程度在问题和答案之间引入了虚假相关性，导致模型依赖这种虚假相关性而忽略 ECG 输入。此类模型无法理解 ECG 输入，在对抗性测试中表现不佳，因为在训练和测试集中使用同一问题的不同表达方式。我们根据后门调整理论设计了一种去偏预训练方法来消除混杂因素的影响。我们的模型在对抗性测试下的 ECG-QA 任务中表现良好，并展示了零样本能力。一个有趣的随机 ECG 测试进一步验证了我们的模型能够有效理解和利用输入的 ECG 信号。

Title: Astro-HEP-BERT: A bidirectional language model for studying the meanings of concepts in astrophysics and high energy physics

• Authors: Arno Simons
• Subjects: cs.CL, physics.hist-ph
• Abstract URL: https://arxiv.org/abs/2411.14877
• Pdf URL: https://arxiv.org/pdf/2411.14877
• Copy Paste: [[2411.14877]] Astro-HEP-BERT: A bidirectional language model for studying the meanings of concepts in astrophysics and high energy physics(https://arxiv.org/abs/2411.14877)
• Keywords: language model
• Abstract: I present Astro-HEP-BERT, a transformer-based language model specifically designed for generating contextualized word embeddings (CWEs) to study the meanings of concepts in astrophysics and high-energy physics. Built on a general pretrained BERT model, Astro-HEP-BERT underwent further training over three epochs using the Astro-HEP Corpus, a dataset I curated from 21.84 million paragraphs extracted from more than 600,000 scholarly articles on arXiv, all belonging to at least one of these two scientific domains. The project demonstrates both the effectiveness and feasibility of adapting a bidirectional transformer for applications in the history, philosophy, and sociology of science (HPSS). The entire training process was conducted using freely available code, pretrained weights, and text inputs, completed on a single MacBook Pro Laptop (M2/96GB). Preliminary evaluations indicate that Astro-HEP-BERT's CWEs perform comparably to domain-adapted BERT models trained from scratch on larger datasets for domain-specific word sense disambiguation and induction and related semantic change analyses. This suggests that retraining general language models for specific scientific domains can be a cost-effective and efficient strategy for HPSS researchers, enabling high performance without the need for extensive training from scratch.
• 摘要：我介绍了 Astro-HEP-BERT，这是一种基于转换器的语言模型，专门用于生成语境化词向量 (CWE)，以研究天体物理学和高能物理学中概念的含义。Astro-HEP-BERT 建立在通用预训练 BERT 模型的基础上，使用 Astro-HEP 语料库进行了三个时期的进一步训练，该语料库是我从 arXiv 上 600,000 多篇学术文章中提取的 2184 万个段落中整理出来的数据集，这些文章都属于这两个科学领域中的至少一个。该项目展示了将双向转换器应用于科学史、哲学和社会学 (HPSS) 的有效性和可行性。整个训练过程使用免费提供的代码、预训练权重和文本输入进行，在一台 MacBook Pro 笔记本电脑 (M2/96GB) 上完成。初步评估表明，Astro-HEP-BERT 的 CWE 性能与在更大数据集上从头训练的领域自适应 BERT 模型相当，可用于特定领域的词义消歧和归纳以及相关的语义变化分析。这表明，针对特定科学领域重新训练通用语言模型对于 HPSS 研究人员来说是一种经济高效且高效的策略，无需从头开始进行大量训练即可实现高性能。

Title: Leveraging Hierarchical Prototypes as the Verbalizer for Implicit Discourse Relation Recognition

• Authors: Wanqiu Long, Bonnie Webber
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.14880
• Pdf URL: https://arxiv.org/pdf/2411.14880
• Copy Paste: [[2411.14880]] Leveraging Hierarchical Prototypes as the Verbalizer for Implicit Discourse Relation Recognition(https://arxiv.org/abs/2411.14880)
• Keywords: prompt
• Abstract: Implicit discourse relation recognition involves determining relationships that hold between spans of text that are not linked by an explicit discourse connective. In recent years, the pre-train, prompt, and predict paradigm has emerged as a promising approach for tackling this task. However, previous work solely relied on manual verbalizers for implicit discourse relation recognition, which suffer from issues of ambiguity and even incorrectness. To overcome these limitations, we leverage the prototypes that capture certain class-level semantic features and the hierarchical label structure for different classes as the verbalizer. We show that our method improves on competitive baselines. Besides, our proposed approach can be extended to enable zero-shot cross-lingual learning, facilitating the recognition of discourse relations in languages with scarce resources. These advancement validate the practicality and versatility of our approach in addressing the issues of implicit discourse relation recognition across different languages.
• 摘要：隐性话语关系识别涉及确定未通过显性话语连接词链接的文本跨度之间的关系。近年来，预训练、提示和预测范式已成为解决此任务的一种有前途的方法。然而，以前的工作仅依靠手动言语化器进行隐性话语关系识别，这存在歧义甚至不正确的问题。为了克服这些限制，我们利用捕获某些类级语义特征的原型和不同类别的分层标签结构作为言语化器。我们表明我们的方法在竞争基线上有所改进。此外，我们提出的方法可以扩展以实现零样本跨语言学习，从而促进在资源稀缺的语言中识别话语关系。这些进步验证了我们的方法在解决跨不同语言的隐性话语关系识别问题方面的实用性和多功能性。

Title: Evaluating LLM Prompts for Data Augmentation in Multi-label Classification of Ecological Texts

• Authors: Anna Glazkova, Olga Zakharova
• Subjects: cs.CL, cs.CY, cs.SI
• Abstract URL: https://arxiv.org/abs/2411.14896
• Pdf URL: https://arxiv.org/pdf/2411.14896
• Copy Paste: [[2411.14896]] Evaluating LLM Prompts for Data Augmentation in Multi-label Classification of Ecological Texts(https://arxiv.org/abs/2411.14896)
• Keywords: language model, llm, prompt
• Abstract: Large language models (LLMs) play a crucial role in natural language processing (NLP) tasks, improving the understanding, generation, and manipulation of human language across domains such as translating, summarizing, and classifying text. Previous studies have demonstrated that instruction-based LLMs can be effectively utilized for data augmentation to generate diverse and realistic text samples. This study applied prompt-based data augmentation to detect mentions of green practices in Russian social media. Detecting green practices in social media aids in understanding their prevalence and helps formulate recommendations for scaling eco-friendly actions to mitigate environmental issues. We evaluated several prompts for augmenting texts in a multi-label classification task, either by rewriting existing datasets using LLMs, generating new data, or combining both approaches. Our results revealed that all strategies improved classification performance compared to the models fine-tuned only on the original dataset, outperforming baselines in most cases. The best results were obtained with the prompt that paraphrased the original text while clearly indicating the relevant categories.
• 摘要：大型语言模型 (LLM) 在自然语言处理 (NLP) 任务中发挥着至关重要的作用，可提高对人类语言的理解、生成和操作，涉及翻译、总结和分类文本等领域。先前的研究表明，基于指令的 LLM 可有效用于数据增强，以生成多样化且逼真的文本样本。本研究应用基于提示的数据增强来检测俄罗斯社交媒体中对绿色实践的提及。检测社交媒体中的绿色实践有助于了解其普遍性，并有助于制定扩大环保行动以缓解环境问题的建议。我们评估了在多标签分类任务中增强文本的几种提示，方法是使用 LLM 重写现有数据集、生成新数据或结合这两种方法。我们的结果表明，与仅在原始数据集上进行微调的模型相比，所有策略都提高了分类性能，在大多数情况下都优于基线。使用解释原文并明确指出相关类别的提示获得了最佳结果。

Title: LLM for Barcodes: Generating Diverse Synthetic Data for Identity Documents

• Authors: Hitesh Laxmichand Patel, Amit Agarwal, Bhargava Kumar, Karan Gupta, Priyaranjan Pattnayak
• Subjects: cs.CL, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2411.14962
• Pdf URL: https://arxiv.org/pdf/2411.14962
• Copy Paste: [[2411.14962]] LLM for Barcodes: Generating Diverse Synthetic Data for Identity Documents(https://arxiv.org/abs/2411.14962)
• Keywords: llm
• Abstract: Accurate barcode detection and decoding in Identity documents is crucial for applications like security, healthcare, and education, where reliable data extraction and verification are essential. However, building robust detection models is challenging due to the lack of diverse, realistic datasets an issue often tied to privacy concerns and the wide variety of document formats. Traditional tools like Faker rely on predefined templates, making them less effective for capturing the complexity of real-world identity documents. In this paper, we introduce a new approach to synthetic data generation that uses LLMs to create contextually rich and realistic data without relying on predefined field. Using the vast knowledge LLMs have about different documents and content, our method creates data that reflects the variety found in real identity documents. This data is then encoded into barcode and overlayed on templates for documents such as Driver's licenses, Insurance cards, Student IDs. Our approach simplifies the process of dataset creation, eliminating the need for extensive domain knowledge or predefined fields. Compared to traditional methods like Faker, data generated by LLM demonstrates greater diversity and contextual relevance, leading to improved performance in barcode detection models. This scalable, privacy-first solution is a big step forward in advancing machine learning for automated document processing and identity verification.
• 摘要：准确检测和解码身份证件中的条形码对于安全、医疗保健和教育等应用至关重要，因为可靠的数据提取和验证必不可少。然而，由于缺乏多样化、真实的数据集，构建强大的检测模型具有挑战性，这一问题通常与隐私问题和各种文档格式有关。像 Faker 这样的传统工具依赖于预定义的模板，这使得它们在捕捉现实世界身份证件的复杂性方面效率较低。在本文中，我们介绍了一种合成数据生成的新方法，该方法使用 LLM 创建上下文丰富且真实的数据，而不依赖于预定义字段。利用 LLM 对不同文档和内容的丰富知识，我们的方法可以创建反映真实身份证件多样性的数据。然后将这些数据编码为条形码并叠加在驾照、保险卡、学生证等文档的模板上。我们的方法简化了数据集创建过程，无需广泛的领域知识或预定义字段。与 Faker 等传统方法相比，LLM 生成的数据表现出更大的多样性和上下文相关性，从而提高了条形码检测模型的性能。这种可扩展的、隐私优先的解决方案是推动机器学习在自动文档处理和身份验证方面迈出的一大步。

Title: SwissADT: An Audio Description Translation System for Swiss Languages

• Authors: Lukas Fischer, Yingqiang Gao, Alexa Lintner, Sarah Ebling
• Subjects: cs.CL, cs.AI, cs.CV, cs.HC
• Abstract URL: https://arxiv.org/abs/2411.14967
• Pdf URL: https://arxiv.org/pdf/2411.14967
• Copy Paste: [[2411.14967]] SwissADT: An Audio Description Translation System for Swiss Languages(https://arxiv.org/abs/2411.14967)
• Keywords: language model, llm
• Abstract: Audio description (AD) is a crucial accessibility service provided to blind persons and persons with visual impairment, designed to convey visual information in acoustic form. Despite recent advancements in multilingual machine translation research, the lack of well-crafted and time-synchronized AD data impedes the development of audio description translation (ADT) systems that address the needs of multilingual countries such as Switzerland. Furthermore, since the majority of ADT systems rely solely on text, uncertainty exists as to whether incorporating visual information from the corresponding video clips can enhance the quality of ADT outputs. In this work, we present SwissADT, the first ADT system implemented for three main Swiss languages and English. By collecting well-crafted AD data augmented with video clips in German, French, Italian, and English, and leveraging the power of Large Language Models (LLMs), we aim to enhance information accessibility for diverse language populations in Switzerland by automatically translating AD scripts to the desired Swiss language. Our extensive experimental ADT results, composed of both automatic and human evaluations of ADT quality, demonstrate the promising capability of SwissADT for the ADT task. We believe that combining human expertise with the generation power of LLMs can further enhance the performance of ADT systems, ultimately benefiting a larger multilingual target population.
• 摘要：音频描述 (AD) 是为盲人和视障人士提供的一项重要的无障碍服务，旨在以声音形式传达视觉信息。尽管多语言机器翻译研究最近取得了进展，但缺乏精心制作和时间同步的 AD 数据阻碍了满足瑞士等多语言国家需求的音频描述翻译 (ADT) 系统的开发。此外，由于大多数 ADT 系统仅依赖于文本，因此不确定结合相应视频片段中的视觉信息是否可以提高 ADT 输出的质量。在这项工作中，我们介绍了 SwissADT，这是第一个为三种主要瑞士语言和英语实施的 ADT 系统。通过收集精心制作的 AD 数据并添加德语、法语、意大利语和英语的视频片段，并利用大型语言模型 (LLM) 的强大功能，我们旨在通过自动将 AD 脚本翻译成所需的瑞士语言来增强瑞士不同语言人群的信息可访问性。我们广泛的实验性 ADT 结果（包括对 ADT 质量的自动和人工评估）证明了 SwissADT 在 ADT 任务中的良好能力。我们相信，将人类的专业知识与 LLM 的生成能力相结合可以进一步提高 ADT 系统的性能，最终使更多的多语言目标人群受益。

Title: ScribeAgent: Towards Specialized Web Agents Using Production-Scale Workflow Data

• Authors: Junhong Shen, Atishay Jain, Zedian Xiao, Ishan Amlekar, Mouad Hadji, Aaron Podolny, Ameet Talwalkar
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.15004
• Pdf URL: https://arxiv.org/pdf/2411.15004
• Copy Paste: [[2411.15004]] ScribeAgent: Towards Specialized Web Agents Using Production-Scale Workflow Data(https://arxiv.org/abs/2411.15004)
• Keywords: language model, gpt, llm, prompt, agent
• Abstract: Large Language Model (LLM) agents are rapidly improving to handle increasingly complex web-based tasks. Most of these agents rely on general-purpose, proprietary models like GPT-4 and focus on designing better prompts to improve their planning abilities. However, general-purpose LLMs are not specifically trained to understand specialized web contexts such as HTML, and they often struggle with long-horizon planning. We explore an alternative approach that fine-tunes open-source LLMs using production-scale workflow data collected from over 250 domains corresponding to 6 billion tokens. This simple yet effective approach shows substantial gains over prompting-based agents on existing benchmarks -- ScribeAgent achieves state-of-the-art direct generation performance on Mind2Web and improves the task success rate by 14.1% over the previous best text-only web agents on WebArena. We further perform detailed ablation studies on various fine-tuning design choices and provide insights into LLM selection, training recipes, context window optimization, and effect of dataset sizes.
• 摘要：大型语言模型 (LLM) 代理正在迅速改进，以处理日益复杂的基于 Web 的任务。这些代理中的大多数依赖于通用的专有模型（如 GPT-4），并专注于设计更好的提示以提高其规划能力。然而，通用 LLM 并未经过专门训练以理解 HTML 等专门的 Web 上下文，并且它们通常在长期规划方面遇到困难。我们探索了一种替代方法，该方法使用从 250 多个域收集的生产规模工作流数据（对应于 60 亿个令牌）对开源 LLM 进行微调。这种简单而有效的方法在现有基准上显示出比基于提示的代理有显著的提升——ScribeAgent 在 Mind2Web 上实现了最先进的直接生成性能，并且比之前在 WebArena 上最好的纯文本 Web 代理将任务成功率提高了 14.1%。我们进一步对各种微调设计选择进行了详细的消融研究，并提供了对 LLM 选择、训练配方、上下文窗口优化和数据集大小影响的见解。

Title: XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models

• Authors: Yixin Dong, Charlie F. Ruan, Yaxing Cai, Ruihang Lai, Ziyi Xu, Yilong Zhao, Tianqi Chen
• Subjects: cs.CL, cs.AI, cs.PL
• Abstract URL: https://arxiv.org/abs/2411.15100
• Pdf URL: https://arxiv.org/pdf/2411.15100
• Copy Paste: [[2411.15100]] XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models(https://arxiv.org/abs/2411.15100)
• Keywords: language model, llm, agent
• Abstract: The applications of LLM Agents are becoming increasingly complex and diverse, leading to a high demand for structured outputs that can be parsed into code, structured function calls, and embodied agent commands. These developments bring significant demands for structured generation in LLM inference. Context-free grammar is a flexible approach to enable structured generation via constrained decoding. However, executing context-free grammar requires going through several stack states over all tokens in vocabulary during runtime, bringing non-negligible overhead for structured generation. In this paper, we propose XGrammar, a flexible and efficient structure generation engine for large language models. XGrammar accelerates context-free grammar execution by dividing the vocabulary into context-independent tokens that can be prechecked and context-dependent tokens that need to be interpreted during runtime. We further build transformations to expand the grammar context and reduce the number of context-independent tokens. Additionally, we build an efficient persistent stack to accelerate the context-dependent token checks. Finally, we co-design the grammar engine with LLM inference engine to overlap grammar computation with GPU executions. Evaluation results show that XGrammar can achieve up to 100x speedup over existing solutions. Combined with an LLM inference engine, it can generate near-zero overhead structure generation in end-to-end low-LLM serving.
• 摘要：LLM 代理的应用正变得越来越复杂和多样化，导致对可解析为代码、结构化函数调用和具体代理命令的结构化输出的需求很高。这些发展对 LLM 推理中的结构化生成提出了巨大的需求。上下文无关语法是一种通过约束解码实现结构化生成的灵活方法。然而，执行上下文无关语法需要在运行时对词汇表中的所有标记进行多次堆栈状态，这给结构化生成带来了不可忽略的开销。在本文中，我们提出了 XGrammar，一种灵活高效的大型语言模型结构生成引擎。XGrammar 通过将词汇表划分为可预先检查的上下文无关标记和需要在运行时解释的上下文相关标记来加速上下文无关语法的执行。我们进一步构建转换以扩展语法上下文并减少上下文无关标记的数量。此外，我们构建了一个高效的持久堆栈来加速上下文相关标记检查。最后，我们共同设计了语法引擎和 LLM 推理引擎，将语法计算与 GPU 执行重叠。评估结果表明，XGrammar 可以比现有解决方案实现高达 100 倍的加速。结合 LLM 推理引擎，它可以在端到端低 LLM 服务中生成近乎零开销的结构生成。

Title: T\"ULU 3: Pushing Frontiers in Open Language Model Post-Training

• Authors: Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, Hannaneh Hajishirzi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.15124
• Pdf URL: https://arxiv.org/pdf/2411.15124
• Copy Paste: [[2411.15124]] T\"ULU 3: Pushing Frontiers in Open Language Model Post-Training(https://arxiv.org/abs/2411.15124)
• Keywords: language model, gpt
• Abstract: Language model post-training is applied to refine behaviors and unlock new skills across a wide range of recent language models, but open recipes for applying these techniques lag behind proprietary ones. The underlying training data and recipes for post-training are simultaneously the most important pieces of the puzzle and the portion with the least transparency. To bridge this gap, we introduce TÜLU 3, a family of fully-open state-of-the-art post-trained models, alongside its data, code, and training recipes, serving as a comprehensive guide for modern post-training techniques. TÜLU 3, which builds on Llama 3.1 base models, achieves results surpassing the instruct versions of Llama 3.1, Qwen 2.5, Mistral, and even closed models such as GPT-4o-mini and Claude 3.5-Haiku. The training algorithms for our models include supervised finetuning (SFT), Direct Preference Optimization (DPO), and a novel method we call Reinforcement Learning with Verifiable Rewards (RLVR). With TÜLU 3, we introduce a multi-task evaluation scheme for post-training recipes with development and unseen evaluations, standard benchmark implementations, and substantial decontamination of existing open datasets on said benchmarks. We conclude with analysis and discussion of training methods that did not reliably improve performance. In addition to the TÜLU 3 model weights and demo, we release the complete recipe -- including datasets for diverse core skills, a robust toolkit for data curation and evaluation, the training code and infrastructure, and, most importantly, a detailed report for reproducing and further adapting the TÜLU 3 approach to more domains.
• 摘要：语言模型后训练可用于改进行为并解锁各种最新语言模型的新技能，但应用这些技术的开放方法落后于专有方法。底层训练数据和后训练方法既是拼图中最重要的部分，也是透明度最低的部分。为了弥补这一差距，我们推出了 TÜLU 3，这是一系列完全开放的最先进的后训练模型，以及其数据、代码和训练方法，可作为现代后训练技术的综合指南。TÜLU 3 以 Llama 3.1 基础模型为基础，其结果超越了 Llama 3.1、Qwen 2.5、Mistral 的指导版本，甚至超越了 GPT-4o-mini 和 Claude 3.5-Haiku 等封闭模型。我们模型的训练算法包括监督微调 (SFT)、直接偏好优化 (DPO) 和一种我们称之为可验证奖励强化学习 (RLVR) 的新方法。借助 TÜLU 3，我们引入了一种针对训练后配方的多任务评估方案，其中包括开发和未见评估、标准基准实施以及对上述基准上现有开放数据集的大量净化。最后，我们分析并讨论了无法可靠提高性能的训练方法。除了 TÜLU 3 模型权重和演示之外，我们还发布了完整的配方 - 包括用于各种核心技能的数据集、用于数据管理和评估的强大工具包、训练代码和基础设施，最重要的是，一份详细的报告，用于重现和进一步将 TÜLU 3 方法应用于更多领域。

Title: Measuring Bullshit in the Language Games played by ChatGPT

• Authors: Alessandro Trevisan, Harry Giddens, Sarah Dillon, Alan F. Blackwell
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2411.15129
• Pdf URL: https://arxiv.org/pdf/2411.15129
• Copy Paste: [[2411.15129]] Measuring Bullshit in the Language Games played by ChatGPT(https://arxiv.org/abs/2411.15129)
• Keywords: language model, gpt, llm, chat
• Abstract: Generative large language models (LLMs), which create text without direct correspondence to truth value, are widely understood to resemble the uses of language described in Frankfurt's popular monograph On Bullshit. In this paper, we offer a rigorous investigation of this topic, identifying how the phenomenon has arisen, and how it might be analysed. In this paper, we elaborate on this argument to propose that LLM-based chatbots play the 'language game of bullshit'. We use statistical text analysis to investigate the features of this Wittgensteinian language game, based on a dataset constructed to contrast the language of 1,000 scientific publications with typical pseudo-scientific text generated by ChatGPT. We then explore whether the same language features can be detected in two well-known contexts of social dysfunction: George Orwell's critique of politics and language, and David Graeber's characterisation of bullshit jobs. Using simple hypothesis-testing methods, we demonstrate that a statistical model of the language of bullshit can reliably relate the Frankfurtian artificial bullshit of ChatGPT to the political and workplace functions of bullshit as observed in natural human language.
• 摘要：生成式大型语言模型 (LLM) 所创建的文本与真值没有直接对应关系，人们普遍认为它们类似于法兰克福的流行专著《论扯淡》中描述的语言用法。在本文中，我们对这个主题进行了严格的调查，确定了这种现象是如何产生的，以及如何分析它。在本文中，我们详细阐述了这一论点，提出基于 LLM 的聊天机器人玩的是“扯淡的语言游戏”。我们使用统计文本分析来研究这种维特根斯坦语言游戏的特征，该分析基于一个数据集，该数据集构建了 1,000 份科学出版物的语言与 ChatGPT 生成的典型伪科学文本进行对比。然后，我们探讨是否可以在两个众所周知的社会功能障碍背景下检测到相同的语言特征：乔治·奥威尔对政治和语言的批判，以及大卫·格雷伯对扯淡工作的描述。使用简单的假设检验方法，我们证明了胡说八道语言的统计模型可以可靠地将 ChatGPT 的法兰克福人造胡说八道与自然人类语言中观察到的胡说八道的政治和工作场所功能联系起来。