2025-01-27

Title: Fanar: An Arabic-Centric Multimodal Generative AI Platform

• Authors: Fanar Team: Ummar Abbas, Mohammad Shahmeer Ahmad, Firoj Alam, Enes Altinisik, Ehsannedin Asgari, Yazan Boshmaf, Sabri Boughorbel, Sanjay Chawla, Shammur Chowdhury, Fahim Dalvi, Kareem Darwish, Nadir Durrani, Mohamed Elfeky, Ahmed Elmagarmid, Mohamed Eltabakh, Masoomali Fatehkia, Anastasios Fragkopoulos, Maram Hasanain, Majd Hawasly, Mus'ab Husaini, Soon-Gyo Jung, Ji Kim Lucas, Walid Magdy, Safa Messaoud, Abubakr Mohamed, Tasnim Mohiuddin, Basel Mousi, Hamdy Mubarak, Ahmad Musleh, Zan Naeem, Mourad Ouzzani, Dorde Popovic, Amin Sadeghi, Husrev Taha Sencar, Mohammed Shinoy, Omar Sinan, Yifan Zhang, Ahmed Ali, Yassine El Kheir, Xiaosong Ma, Chaoyi Ruan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.13944
• Pdf URL: https://arxiv.org/pdf/2501.13944
• Copy Paste: [[2501.13944]] Fanar: An Arabic-Centric Multimodal Generative AI Platform(https://arxiv.org/abs/2501.13944)
• Keywords: language model, llm, prompt, retrieval augmented generation
• Abstract: We present Fanar, a platform for Arabic-centric multimodal generative AI systems, that supports language, speech and image generation tasks. At the heart of Fanar are Fanar Star and Fanar Prime, two highly capable Arabic Large Language Models (LLMs) that are best in the class on well established benchmarks for similar sized models. Fanar Star is a 7B (billion) parameter model that was trained from scratch on nearly 1 trillion clean and deduplicated Arabic, English and Code tokens. Fanar Prime is a 9B parameter model continually trained on the Gemma-2 9B base model on the same 1 trillion token set. Both models are concurrently deployed and designed to address different types of prompts transparently routed through a custom-built orchestrator. The Fanar platform provides many other capabilities including a customized Islamic Retrieval Augmented Generation (RAG) system for handling religious prompts, a Recency RAG for summarizing information about current or recent events that have occurred after the pre-training data cut-off date. The platform provides additional cognitive capabilities including in-house bilingual speech recognition that supports multiple Arabic dialects, voice and image generation that is fine-tuned to better reflect regional characteristics. Finally, Fanar provides an attribution service that can be used to verify the authenticity of fact based generated content. The design, development, and implementation of Fanar was entirely undertaken at Hamad Bin Khalifa University's Qatar Computing Research Institute (QCRI) and was sponsored by Qatar's Ministry of Communications and Information Technology to enable sovereign AI technology development.
• 摘要：我们推出了 Fanar，这是一个以阿拉伯语为中心的多模态生成式 AI 系统平台，支持语言、语音和图像生成任务。Fanar 的核心是 Fanar Star 和 Fanar Prime，这两个功能强大的阿拉伯语大型语言模型 (LLM) 在同类模型的成熟基准中表现最佳。Fanar Star 是一个 7B（十亿）参数模型，从头开始训练近 1 万亿个干净且去重的阿拉伯语、英语和代码标记。Fanar Prime 是一个 9B 参数模型，在相同的 1 万亿个标记集上持续训练 Gemma-2 9B 基础模型。这两个模型同时部署，旨在解决通过定制编排器透明路由的不同类型的提示。Fanar 平台提供许多其他功能，包括用于处理宗教提示的定制伊斯兰检索增强生成 (RAG) 系统，以及用于总结预训练数据截止日期后发生的当前或最近事件信息的 Recency RAG。该平台提供额外的认知能力，包括支持多种阿拉伯方言的内部双语语音识别、经过微调以更好地反映区域特征的语音和图像生成。最后，Fanar 提供归因服务，可用于验证基于事实的生成内容的真实性。Fanar 的设计、开发和实施完全由哈马德·本·哈利法大学的卡塔尔计算研究所 (QCRI) 进行，并由卡塔尔通信和信息技术部赞助，以实现自主 AI 技术开发。

Title: Self-Explanation in Social AI Agents

• Authors: Rhea Basappa, Mustafa Tekman, Hong Lu, Benjamin Faught, Sandeep Kakar, Ashok K. Goel
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2501.13945
• Pdf URL: https://arxiv.org/pdf/2501.13945
• Copy Paste: [[2501.13945]] Self-Explanation in Social AI Agents(https://arxiv.org/abs/2501.13945)
• Keywords: gpt, chat, agent
• Abstract: Social AI agents interact with members of a community, thereby changing the behavior of the community. For example, in online learning, an AI social assistant may connect learners and thereby enhance social interaction. These social AI assistants too need to explain themselves in order to enhance transparency and trust with the learners. We present a method of self-explanation that uses introspection over a self-model of an AI social assistant. The self-model is captured as a functional model that specifies how the methods of the agent use knowledge to achieve its tasks. The process of generating self-explanations uses Chain of Thought to reflect on the self-model and ChatGPT to provide explanations about its functioning. We evaluate the self-explanation of the AI social assistant for completeness and correctness. We also report on its deployment in a live class.
• 摘要：社交 AI 代理与社区成员互动，从而改变社区的行为。例如，在在线学习中，AI 社交助手可以连接学习者，从而增强社交互动。这些社交 AI 助手也需要解释自己，以增强透明度和与学习者的信任。我们提出了一种自我解释方法，该方法使用对 AI 社交助手的自我模型进行自省。自我模型被视为一个功能模型，该模型指定代理的方法如何使用知识来完成其任务。生成自我解释的过程使用思想链来反思自我模型，并使用 ChatGPT 来提供有关其功能的解释。我们评估 AI 社交助手的自我解释的完整性和正确性。我们还报告了它在现场课堂中的部署情况。

Title: Hallucination Mitigation using Agentic AI Natural Language-Based Frameworks

• Authors: Diego Gosmar, Deborah A. Dahl
• Subjects: cs.CL, cs.AI, cs.MA
• Abstract URL: https://arxiv.org/abs/2501.13946
• Pdf URL: https://arxiv.org/pdf/2501.13946
• Copy Paste: [[2501.13946]] Hallucination Mitigation using Agentic AI Natural Language-Based Frameworks(https://arxiv.org/abs/2501.13946)
• Keywords: language model, hallucination, prompt, agent
• Abstract: Hallucinations remain a significant challenge in current Generative AI models, undermining trust in AI systems and their reliability. This study investigates how orchestrating multiple specialized Artificial Intelligent Agents can help mitigate such hallucinations, with a focus on systems leveraging Natural Language Processing (NLP) to facilitate seamless agent interactions. To achieve this, we design a pipeline that introduces over three hundred prompts, purposefully crafted to induce hallucinations, into a front-end agent. The outputs are then systematically reviewed and refined by second- and third-level agents, each employing distinct large language models and tailored strategies to detect unverified claims, incorporate explicit disclaimers, and clarify speculative content. Additionally, we introduce a set of novel Key Performance Indicators (KPIs) specifically designed to evaluate hallucination score levels. A dedicated fourth-level AI agent is employed to evaluate these KPIs, providing detailed assessments and ensuring accurate quantification of shifts in hallucination-related behaviors. A core component of this investigation is the use of the OVON (Open Voice Network) framework, which relies on universal NLP-based interfaces to transfer contextual information among agents. Through structured JSON messages, each agent communicates its assessment of the hallucination likelihood and the reasons underlying questionable content, thereby enabling the subsequent stage to refine the text without losing context. The results demonstrate that employing multiple specialized agents capable of interoperating with each other through NLP-based agentic frameworks can yield promising outcomes in hallucination mitigation, ultimately bolstering trust within the AI community.
• 摘要：幻觉仍然是当前生成式 AI 模型面临的重大挑战，它削弱了人们对 AI 系统及其可靠性的信任。本研究调查了如何协调多个专门的人工智能代理来帮助缓解这种幻觉，重点是利用自然语言处理 (NLP) 促进无缝代理交互的系统。为了实现这一点，我们设计了一个管道，将三百多个专门设计用于诱发幻觉的提示引入前端代理。然后，第二级和第三级代理系统地审查和完善输出，每个代理都采用不同的大型语言模型和量身定制的策略来检测未经证实的声明、纳入明确的免责声明并澄清推测内容。此外，我们还引入了一组专门用于评估幻觉评分水平的新关键绩效指标 (KPI)。专门的第四级 AI 代理用于评估这些 KPI，提供详细的评估并确保准确量化幻觉相关行为的变化。这项研究的核心部分是使用 OVON（开放语音网络）框架，该框架依赖于通用的基于 NLP 的接口在代理之间传输上下文信息。通过结构化的 JSON 消息，每个代理传达其对幻觉可能性的评估以及可疑内容背后的原因，从而使后续阶段能够在不丢失上下文的情况下完善文本。结果表明，使用能够通过基于 NLP 的代理框架相互交互的多个专门代理可以在缓解幻觉方面取得有希望的结果，最终增强人工智能社区内的信任。

Title: A Comprehensive Survey on Integrating Large Language Models with Knowledge-Based Methods

• Authors: Lilian Some, Wenli Yang, Michael Bain, Byeong Kang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.13947
• Pdf URL: https://arxiv.org/pdf/2501.13947
• Copy Paste: [[2501.13947]] A Comprehensive Survey on Integrating Large Language Models with Knowledge-Based Methods(https://arxiv.org/abs/2501.13947)
• Keywords: language model, llm
• Abstract: The rapid development of artificial intelligence has brought about substantial advancements in the field. One promising direction is the integration of Large Language Models (LLMs) with structured knowledge-based systems. This approach aims to enhance AI capabilities by combining the generative language understanding of LLMs with the precise knowledge representation of structured systems. This survey explores the synergy between LLMs and knowledge bases, focusing on real-world applications and addressing associated technical, operational, and ethical challenges. Through a comprehensive literature review, the study identifies critical issues and evaluates existing solutions. The paper highlights the benefits of integrating generative AI with knowledge bases, including improved data contextualization, enhanced model accuracy, and better utilization of knowledge resources. The findings provide a detailed overview of the current state of research, identify key gaps, and offer actionable recommendations. These insights contribute to advancing AI technologies and support their practical deployment across various sectors.
• 摘要：人工智能的快速发展为该领域带来了实质性的进步。一个有希望的方向是将大型语言模型 (LLM) 与结构化知识库相结合。这种方法旨在通过将 LLM 的生成语言理解与结构化系统的精确知识表示相结合来增强 AI 能力。本调查探讨了 LLM 与知识库之间的协同作用，重点关注实际应用并解决相关的技术、操作和道德挑战。通过全面的文献综述，该研究确定了关键问题并评估了现有的解决方案。本文强调了将生成 AI 与知识库相结合的好处，包括改进数据情境化、提高模型准确性和更好地利用知识资源。研究结果详细概述了当前的研究状态，确定了关键差距并提出了可行的建议。这些见解有助于推进 AI 技术并支持其在各个领域的实际部署。

Title: Longitudinal Abuse and Sentiment Analysis of Hollywood Movie Dialogues using LLMs

• Authors: Rohitash Chandra, Guoxiang Ren, Group-H
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.13948
• Pdf URL: https://arxiv.org/pdf/2501.13948
• Copy Paste: [[2501.13948]] Longitudinal Abuse and Sentiment Analysis of Hollywood Movie Dialogues using LLMs(https://arxiv.org/abs/2501.13948)
• Keywords: language model, llm
• Abstract: Over the past decades, there has been an increasing concern about the prevalence of abusive and violent content in Hollywood movies. This study uses Large Language Models (LLMs) to explore the longitudinal abuse and sentiment analysis of Hollywood Oscar and blockbuster movie dialogues from 1950 to 2024. By employing fine-tuned LLMs, we analyze subtitles for over a thousand movies categorised into four genres to examine the trends and shifts in emotional and abusive content over the past seven decades. Our findings reveal significant temporal changes in movie dialogues, which reflect broader social and cultural influences. Overall, the emotional tendencies in the films are diverse, and the detection of abusive content also exhibits significant fluctuations. The results show a gradual rise in abusive content in recent decades, reflecting social norms and regulatory policy changes. Genres such as thrillers still present a higher frequency of abusive content that emphasises the ongoing narrative role of violence and conflict. At the same time, underlying positive emotions such as humour and optimism remain prevalent in most of the movies. Furthermore, the gradual increase of abusive content in movie dialogues has been significant over the last two decades, where Oscar-nominated movies overtook the top ten blockbusters.
• 摘要：在过去的几十年里，人们越来越担心好莱坞电影中辱骂和暴力内容的盛行。本研究使用大型语言模型 (LLM) 探索 1950 年至 2024 年好莱坞奥斯卡和大片电影对话的纵向辱骂和情感分析。通过使用经过微调的 LLM，我们分析了四类一千多部电影的字幕，以研究过去七十年中情感和辱骂内容的趋势和变化。我们的研究结果揭示了电影对话的显著时间变化，反映了更广泛的社会和文化影响。总体而言，电影中的情感倾向是多种多样的，对辱骂内容的检测也表现出显著的波动。结果显示，近几十年来辱骂内容逐渐增加，反映了社会规范和监管政策的变化。惊悚片等类型仍然呈现出更高的辱骂内容频率，强调了暴力和冲突的持续叙事作用。同时，幽默和乐观等潜在的积极情绪在大多数电影中仍然盛行。此外，过去二十年里，电影对话中的辱骂内容逐渐增多，奥斯卡提名电影超过了十大大片。

Title: Can OpenAI o1 Reason Well in Ophthalmology? A 6,990-Question Head-to-Head Evaluation Study

• Authors: Sahana Srinivasan, Xuguang Ai, Minjie Zou, Ke Zou, Hyunjae Kim, Thaddaeus Wai Soon Lo, Krithi Pushpanathan, Yiming Kong, Anran Li, Maxwell Singer, Kai Jin, Fares Antaki, David Ziyou Chen, Dianbo Liu, Ron A. Adelman, Qingyu Chen, Yih Chung Tham
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.13949
• Pdf URL: https://arxiv.org/pdf/2501.13949
• Copy Paste: [[2501.13949]] Can OpenAI o1 Reason Well in Ophthalmology? A 6,990-Question Head-to-Head Evaluation Study(https://arxiv.org/abs/2501.13949)
• Keywords: language model, gpt, llm
• Abstract: Question: What is the performance and reasoning ability of OpenAI o1 compared to other large language models in addressing ophthalmology-specific questions? Findings: This study evaluated OpenAI o1 and five LLMs using 6,990 ophthalmological questions from MedMCQA. O1 achieved the highest accuracy (0.88) and macro-F1 score but ranked third in reasoning capabilities based on text-generation metrics. Across subtopics, o1 ranked first in ``Lens'' and ``Glaucoma'' but second to GPT-4o in ``Corneal and External Diseases'', ``Vitreous and Retina'' and ``Oculoplastic and Orbital Diseases''. Subgroup analyses showed o1 performed better on queries with longer ground truth explanations. Meaning: O1's reasoning enhancements may not fully extend to ophthalmology, underscoring the need for domain-specific refinements to optimize performance in specialized fields like ophthalmology.
• 摘要：问：与其他大型语言模型相比，OpenAI o1 在解决眼科特定问题方面的表现和推理能力如何？结果：本研究使用来自 MedMCQA 的 6,990 个眼科问题对 OpenAI o1 和五个 LLM 进行了评估。O1 的准确率（0.88）和宏 F1 分数最高，但在基于文本生成指标的推理能力方面排名第三。在各个子主题中，o1 在“晶状体”和“青光眼”中排名第一，但在“角膜和外部疾病”、“玻璃体和视网膜”和“眼整形和眼眶疾病”中仅次于 GPT-4o。亚组分析显示，o1 在具有较长基本事实解释的查询上表现更好。含义：O1 的推理增强功能可能并未完全扩展到眼科，这强调了需要针对特定​​领域进行改进以优化眼科等专业领域的表现。

Title: A Layered Multi-Expert Framework for Long-Context Mental Health Assessments

• Authors: Jinwen Tang, Qiming Guo, Wenbo Sun, Yi Shang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.13951
• Pdf URL: https://arxiv.org/pdf/2501.13951
• Copy Paste: [[2501.13951]] A Layered Multi-Expert Framework for Long-Context Mental Health Assessments(https://arxiv.org/abs/2501.13951)
• Keywords: language model, llm, hallucination
• Abstract: Long-form mental health assessments pose unique challenges for large language models (LLMs), which often exhibit hallucinations or inconsistent reasoning when handling extended, domain-specific contexts. We introduce Stacked Multi-Model Reasoning (SMMR), a layered framework that leverages multiple LLMs and specialized smaller models as coequal 'experts'. Early layers isolate short, discrete subtasks, while later layers integrate and refine these partial outputs through more advanced long-context models. We evaluate SMMR on the DAIC-WOZ depression-screening dataset and 48 curated case studies with psychiatric diagnoses, demonstrating consistent improvements over single-model baselines in terms of accuracy, F1-score, and PHQ-8 error reduction. By harnessing diverse 'second opinions', SMMR mitigates hallucinations, captures subtle clinical nuances, and enhances reliability in high-stakes mental health assessments. Our findings underscore the value of multi-expert frameworks for more trustworthy AI-driven screening.
• 摘要：长篇心理健康评估对大型语言模型 (LLM) 提出了独特的挑战，大型语言模型在处理扩展的、特定领域的上下文时，经常会出现幻觉或不一致的推理。我们引入了堆叠多模型推理 (SMMR)，这是一个分层框架，利用多个 LLM 和专门的小型模型作为同等的“专家”。早期层隔离短小、离散的子任务，而后期层通过更高级的长上下文模型集成和优化这些部分输出。我们在 DAIC-WOZ 抑郁症筛查数据集和 48 个精选的精神病诊断案例研究中评估了 SMMR，结果表明，在准确性、F1 分数和 PHQ-8 错误减少方面，与单一模型基线相比，SMMR 有持续的改进。通过利用不同的“第二意见”，SMMR 可以减轻幻觉，捕捉细微的临床细微差别，并提高高风险心理健康评估的可靠性。我们的研究结果强调了多专家框架对更值得信赖的 AI 驱动筛查的价值。

Title: The Dual-use Dilemma in LLMs: Do Empowering Ethical Capacities Make a Degraded Utility?

• Authors: Yiyi Zhang, Xingyu Chen, Kexin Chen, Yuyang Du, Xilin Dang, Pheng-Ann Heng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.13952
• Pdf URL: https://arxiv.org/pdf/2501.13952
• Copy Paste: [[2501.13952]] The Dual-use Dilemma in LLMs: Do Empowering Ethical Capacities Make a Degraded Utility?(https://arxiv.org/abs/2501.13952)
• Keywords: language model, gpt, llm
• Abstract: Recent years have witnessed extensive efforts to enhance Large Language Models (LLMs) across various domains, alongside growing attention to their ethical implications. However, a critical challenge remains largely overlooked: LLMs must balance between rejecting harmful requests for safety and accommodating legitimate ones for utility. This paper presents a Direct Preference Optimization (DPO) based alignment framework that achieves better overall performance by addressing this ethical-utility trade-off, using chemical domain applications as a proof-of-concept. Our alignment pipeline starts with a GPT-assisted three-phase data generation scheme, in which we create LibraChemQA, a chemical question-answering dataset comprising 31.6k triplet instances. By incorporating an innovative balanced seed in the data generation process, our framework systematically considers both legitimate and illegitimate requests. The framework also introduces a rephrasing mechanism for efficient data augmentation that enhances the model's chemical comprehension. We further develop a novel hybrid evaluation scheme with LLM judges for precise assessment of both safety and utility. Experimental results demonstrate our model's substantial improvements in overall performance where both safety and utility are considered - our resulting model, LibraChem, outperforms leading LLMs including Claude-3, GPT-4o, and LLaMA-3 by margins of 13.44%, 7.16%, and 7.10% respectively on our released benchmark.
• 摘要：近年来，人们在各个领域都付出了巨大的努力来增强大型语言模型 (LLM)，同时也越来越关注它们的伦理影响。然而，一个关键的挑战仍然被人们忽视：LLM 必须在拒绝有害的安全请求和容纳合法的实用请求之间取得平衡。本文提出了一种基于直接偏好优化 (DPO) 的对齐框架，该框架通过解决这种道德-实用权衡问题来实现更好的整体性能，并使用化学领域应用作为概念验证。我们的对齐管道从 GPT 辅助的三阶段数据生成方案开始，我们在其中创建了 LibraChemQA，这是一个包含 31.6k 个三元组实例的化学问答数据集。通过在数据生成过程中加入创新的平衡种子，我们的框架系统地考虑了合法和非法请求。该框架还引入了一种改写机制，以实现有效的数据增强，从而增强了模型的化学理解能力。我们进一步开发了一种新颖的混合评估方案，其中有 LLM 评委，可以精确评估安全性和实用性。实验结果表明，在考虑安全性和实用性时，我们的模型的整体性能都有了显著的提升——我们得到的模型 LibraChem 在我们发布的基准上分别比 Claude-3、GPT-4o 和 LLaMA-3 等领先的 LLM 表现高出 13.44%、7.16% 和 7.10%。

Title: Redundancy Principles for MLLMs Benchmarks

• Authors: Zicheng Zhang, Xiangyu Zhao, Xinyu Fang, Chunyi Li, Xiaohong Liu, Xiongkuo Min, Haodong Duan, Kai Chen, Guangtao Zhai
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.13953
• Pdf URL: https://arxiv.org/pdf/2501.13953
• Copy Paste: [[2501.13953]] Redundancy Principles for MLLMs Benchmarks(https://arxiv.org/abs/2501.13953)
• Keywords: language model, llm
• Abstract: With the rapid iteration of Multi-modality Large Language Models (MLLMs) and the evolving demands of the field, the number of benchmarks produced annually has surged into the hundreds. The rapid growth has inevitably led to significant redundancy among benchmarks. Therefore, it is crucial to take a step back and critically assess the current state of redundancy and propose targeted principles for constructing effective MLLM benchmarks. In this paper, we focus on redundancy from three key perspectives: 1) Redundancy of benchmark capability dimensions, 2) Redundancy in the number of test questions, and 3) Cross-benchmark redundancy within specific domains. Through the comprehensive analysis over hundreds of MLLMs' performance across more than 20 benchmarks, we aim to quantitatively measure the level of redundancy lies in existing MLLM evaluations, provide valuable insights to guide the future development of MLLM benchmarks, and offer strategies to refine and address redundancy issues effectively.
• 摘要：随着多模态大型语言模型 (MLLM) 的快速迭代和领域需求的不断变化，每年产生的基准测试数量已激增至数百个。快速的增长不可避免地导致基准测试之间存在大量冗余。因此，退一步批判性地评估当前的冗余状况并提出构建有效 MLLM 基准测试的有针对性的原则至关重要。在本文中，我们从三个关键角度关注冗余：1）基准测试能力维度的冗余，2）测试问题数量的冗余，以及 3）特定领域内的跨基准测试冗余。通过对 20 多个基准测试中数百个 MLLM 的性能进行全面分析，我们旨在定量衡量现有 MLLM 评估中的冗余程度，为指导 MLLM 基准测试的未来发展提供有价值的见解，并提出改进和有效解决冗余问题的策略。

Title: Chat3GPP: An Open-Source Retrieval-Augmented Generation Framework for 3GPP Documents

• Authors: Long Huang, Ming Zhao, Limin Xiao, Xiujun Zhang, Jungang Hu
• Subjects: cs.CL, cs.AI, cs.DC, cs.IR
• Abstract URL: https://arxiv.org/abs/2501.13954
• Pdf URL: https://arxiv.org/pdf/2501.13954
• Copy Paste: [[2501.13954]] Chat3GPP: An Open-Source Retrieval-Augmented Generation Framework for 3GPP Documents(https://arxiv.org/abs/2501.13954)
• Keywords: language model, llm, chat, retrieval-augmented generation
• Abstract: The 3rd Generation Partnership Project (3GPP) documents is key standards in global telecommunications, while posing significant challenges for engineers and researchers in the telecommunications field due to the large volume and complexity of their contents as well as the frequent updates. Large language models (LLMs) have shown promise in natural language processing tasks, but their general-purpose nature limits their effectiveness in specific domains like telecommunications. To address this, we propose Chat3GPP, an open-source retrieval-augmented generation (RAG) framework tailored for 3GPP specifications. By combining chunking strategies, hybrid retrieval and efficient indexing methods, Chat3GPP can efficiently retrieve relevant information and generate accurate responses to user queries without requiring domain-specific fine-tuning, which is both flexible and scalable, offering significant potential for adapting to other technical standards beyond 3GPP. We evaluate Chat3GPP on two telecom-specific datasets and demonstrate its superior performance compared to existing methods, showcasing its potential for downstream tasks like protocol generation and code automation.
• 摘要：第三代合作伙伴计划 (3GPP) 文件是全球电信领域的关键标准，同时由于其内容庞大、复杂且更新频繁，给电信领域的工程师和研究人员带来了重大挑战。大型语言模型 (LLM) 在自然语言处理任务中表现出色，但其通用性限制了它们在电信等特定领域的有效性。为了解决这个问题，我们提出了 Chat3GPP，这是一个针对 3GPP 规范量身定制的开源检索增强生成 (RAG) 框架。通过结合分块策略、混合检索和高效索引方法，Chat3GPP 可以高效地检索相关信息并生成对用户查询的准确响应，而无需进行特定领域的微调，它既灵活又可扩展，为适应 3GPP 以外的其他技术标准提供了巨大的潜力。我们在两个电信特定数据集上评估了 Chat3GPP，并展示了其与现有方法相比的卓越性能，展示了其在协议生成和代码自动化等下游任务中的潜力。

Title: Guided Persona-based AI Surveys: Can we replicate personal mobility preferences at scale using LLMs?

• Authors: Ioannis Tzachristas, Santhanakrishnan Narayanan, Constantinos Antoniou
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2501.13955
• Pdf URL: https://arxiv.org/pdf/2501.13955
• Copy Paste: [[2501.13955]] Guided Persona-based AI Surveys: Can we replicate personal mobility preferences at scale using LLMs?(https://arxiv.org/abs/2501.13955)
• Keywords: language model, llm
• Abstract: This study explores the potential of Large Language Models (LLMs) to generate artificial surveys, with a focus on personal mobility preferences in Germany. By leveraging LLMs for synthetic data creation, we aim to address the limitations of traditional survey methods, such as high costs, inefficiency and scalability challenges. A novel approach incorporating "Personas" - combinations of demographic and behavioural attributes - is introduced and compared to five other synthetic survey methods, which vary in their use of real-world data and methodological complexity. The MiD 2017 dataset, a comprehensive mobility survey in Germany, serves as a benchmark to assess the alignment of synthetic data with real-world patterns. The results demonstrate that LLMs can effectively capture complex dependencies between demographic attributes and preferences while offering flexibility to explore hypothetical scenarios. This approach presents valuable opportunities for transportation planning and social science research, enabling scalable, cost-efficient and privacy-preserving data generation.
• 摘要：本研究探索了大型语言模型 (LLM) 生成人工调查的潜力，重点关注德国的个人流动偏好。通过利用 LLM 创建合成数据，我们旨在解决传统调查方法的局限性，例如成本高、效率低和可扩展性挑战。介绍了一种结合“人物角色”——人口统计和行为属性的组合——的新方法，并将其与其他五种合成调查方法进行了比较，这些方法在使用真实世界数据和方法复杂性方面有所不同。MiD 2017 数据集是德国的一项综合流动性调查，可作为评估合成数据与真实世界模式一致性的基准。结果表明，LLM 可以有效捕捉人口统计属性和偏好之间的复杂依赖关系，同时提供探索假设情景的灵活性。这种方法为交通规划和社会科学研究提供了宝贵的机会，实现了可扩展、经济高效和隐私保护的数据生成。

Title: Zep: A Temporal Knowledge Graph Architecture for Agent Memory

• Authors: Preston Rasmussen, Pavlo Paliychuk, Travis Beauvais, Jack Ryan, Daniel Chalef
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2501.13956
• Pdf URL: https://arxiv.org/pdf/2501.13956
• Copy Paste: [[2501.13956]] Zep: A Temporal Knowledge Graph Architecture for Agent Memory(https://arxiv.org/abs/2501.13956)
• Keywords: language model, gpt, llm, retrieval-augmented generation, agent
• Abstract: We introduce Zep, a novel memory layer service for AI agents that outperforms the current state-of-the-art system, MemGPT, in the Deep Memory Retrieval (DMR) benchmark. Additionally, Zep excels in more comprehensive and challenging evaluations than DMR that better reflect real-world enterprise use cases. While existing retrieval-augmented generation (RAG) frameworks for large language model (LLM)-based agents are limited to static document retrieval, enterprise applications demand dynamic knowledge integration from diverse sources including ongoing conversations and business data. Zep addresses this fundamental limitation through its core component Graphiti -- a temporally-aware knowledge graph engine that dynamically synthesizes both unstructured conversational data and structured business data while maintaining historical relationships. In the DMR benchmark, which the MemGPT team established as their primary evaluation metric, Zep demonstrates superior performance (94.8% vs 93.4%). Beyond DMR, Zep's capabilities are further validated through the more challenging LongMemEval benchmark, which better reflects enterprise use cases through complex temporal reasoning tasks. In this evaluation, Zep achieves substantial results with accuracy improvements of up to 18.5% while simultaneously reducing response latency by 90% compared to baseline implementations. These results are particularly pronounced in enterprise-critical tasks such as cross-session information synthesis and long-term context maintenance, demonstrating Zep's effectiveness for deployment in real-world applications.
• 摘要：我们推出了 Zep，这是一种用于 AI 代理的新型内存层服务，它在深度内存检索 (DMR) 基准测试中的表现优于当前最先进的系统 MemGPT。此外，Zep 在比 DMR 更全面、更具挑战性的评估中表现出色，更好地反映了现实世界的企业用例。虽然现有的基于大型语言模型 (LLM) 的代理的检索增强生成 (RAG) 框架仅限于静态文档检索，但企业应用程序需要从各种来源（包括正在进行的对话和业务数据）动态集成知识。Zep 通过其核心组件 Graphiti 解决了这一根本限制——Graphiti 是一种时间感知知识图谱引擎，可动态合成非结构化对话数据和结构化业务数据，同时保持历史关系。在 MemGPT 团队将其作为主要评估指标的 DMR 基准测试中，Zep 表现出色（94.8% vs 93.4%）。除了 DMR，Zep 的功能还通过更具挑战性的 LongMemEval 基准得到进一步验证，该基准通过复杂的时间推理任务更好地反映了企业用例。在这次评估中，Zep 取得了显著的成果，准确率提高了 18.5%，同时与基线实现相比，响应延迟减少了 90%。这些结果在跨会话信息合成和长期上下文维护等企业关键任务中尤为明显，证明了 Zep 在实际应用中部署的有效性。

Title: Benchmarking Generative AI for Scoring Medical Student Interviews in Objective Structured Clinical Examinations (OSCEs)

• Authors: Jadon Geathers, Yann Hicke, Colleen Chan, Niroop Rajashekar, Justin Sewell, Susannah Cornes, Rene Kizilcec, Dennis Shung
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.13957
• Pdf URL: https://arxiv.org/pdf/2501.13957
• Copy Paste: [[2501.13957]] Benchmarking Generative AI for Scoring Medical Student Interviews in Objective Structured Clinical Examinations (OSCEs)(https://arxiv.org/abs/2501.13957)
• Keywords: language model, gpt, llm, prompt, chain-of-thought
• Abstract: Introduction. Objective Structured Clinical Examinations (OSCEs) are widely used to assess medical students' communication skills, but scoring interview-based assessments is time-consuming and potentially subject to human bias. This study explored the potential of large language models (LLMs) to automate OSCE evaluations using the Master Interview Rating Scale (MIRS). Methods. We compared the performance of four state-of-the-art LLMs (GPT-4o, Claude 3.5, Llama 3.1, and Gemini 1.5 Pro) in evaluating OSCE transcripts across all 28 items of the MIRS under the conditions of zero-shot, chain-of-thought (CoT), few-shot, and multi-step prompting. The models were benchmarked against a dataset of 10 OSCE cases with 174 expert consensus scores available. Model performance was measured using three accuracy metrics (exact, off-by-one, thresholded). Results. Averaging across all MIRS items and OSCE cases, LLMs performed with low exact accuracy (0.27 to 0.44), and moderate to high off-by-one accuracy (0.67 to 0.87) and thresholded accuracy (0.75 to 0.88). A zero temperature parameter ensured high intra-rater reliability ($\alpha = 0.98$ for GPT-4o). CoT, few-shot, and multi-step techniques proved valuable when tailored to specific assessment items. The performance was consistent across MIRS items independent of encounter phases and communication domains. Conclusion. We demonstrated the feasibility of AI-assisted OSCE evaluation and provided benchmarking of multiple LLMs across multiple prompt techniques. Our work provides a baseline performance assessment for LLMs that lays a foundation for future research in automated assessment of clinical communication skills.
• 摘要：简介。客观结构化临床考试 (OSCE) 被广泛用于评估医学生的沟通技巧，但对基于面试的评估进行评分非常耗时，并且可能受到人为偏见的影响。本研究探索了大型语言模型 (LLM) 使用主面试评分量表 (MIRS) 自动化 OSCE 评估的潜力。方法。我们比较了四种最先进的 LLM (GPT-4o、Claude 3.5、Llama 3.1 和 Gemini 1.5 Pro) 在零样本、思路链 (CoT)、少量样本和多步骤提示条件下评估 MIRS 所有 28 个项目中的 OSCE 转录本的性能。这些模型针对 10 个 OSCE 案例的数据集进行了基准测试，该数据集有 174 个专家共识分数可用。使用三个准确度指标（精确、偏差一、阈值）来衡量模型性能。结果。对所有 MIRS 项目和 OSCE 案例进行平均，LLM 的准确度较低（0.27 到 0.44），偏差准确度（0.67 到 0.87）和阈值准确度（0.75 到 0.88）为中等到高。零温度参数确保了较高的评分者内信度（GPT-4o 的 \alpha = 0.98）。CoT、小样本和多步骤技术在针对特定评估项目进行定制时被证明是有价值的。在 MIRS 项目中的表现是一致的，与接触阶段和通信领域无关。结论。我们证明了人工智能辅助 OSCE 评估的可行性，并对多种提示技术的多个 LLM 进行了基准测试。我们的工作为 LLM 提供了基线绩效评估，为未来临床沟通技巧自动评估的研究奠定了基础。

Title: A Survey of Graph Retrieval-Augmented Generation for Customized Large Language Models

• Authors: Qinggang Zhang, Shengyuan Chen, Yuanchen Bei, Zheng Yuan, Huachi Zhou, Zijin Hong, Junnan Dong, Hao Chen, Yi Chang, Xiao Huang
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2501.13958
• Pdf URL: https://arxiv.org/pdf/2501.13958
• Copy Paste: [[2501.13958]] A Survey of Graph Retrieval-Augmented Generation for Customized Large Language Models(https://arxiv.org/abs/2501.13958)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: Large language models (LLMs) have demonstrated remarkable capabilities in a wide range of tasks, yet their application to specialized domains remains challenging due to the need for deep expertise. Retrieval-augmented generation (RAG) has emerged as a promising solution to customize LLMs for professional fields by seamlessly integrating external knowledge bases, enabling real-time access to domain-specific expertise during inference. Despite its potential, traditional RAG systems, based on flat text retrieval, face three critical challenges: (i) complex query understanding in professional contexts, (ii) difficulties in knowledge integration across distributed sources, and (iii) system efficiency bottlenecks at scale. This survey presents a systematic analysis of Graph-based Retrieval-Augmented Generation (GraphRAG), a new paradigm that revolutionizes domain-specific LLM applications. GraphRAG addresses traditional RAG limitations through three key innovations: (i) graph-structured knowledge representation that explicitly captures entity relationships and domain hierarchies, (ii) efficient graph-based retrieval techniques that enable context-preserving knowledge retrieval with multihop reasoning ability, and (iii) structure-aware knowledge integration algorithms that leverage retrieved knowledge for accurate and logical coherent generation of LLMs. In this survey, we systematically analyze the technical foundations of GraphRAG and examine current implementations across various professional domains, identifying key technical challenges and promising research directions. All the related resources of GraphRAG, including research papers, open-source data, and projects, are collected for the community in \textcolor{blue}{\url{this https URL}}.
• 摘要：大型语言模型 (LLM) 在各种任务中都表现出了卓越的能力，但由于需要深厚的专业知识，它们在专业领域的应用仍然具有挑战性。检索增强生成 (RAG) 已成为一种有前途的解决方案，它通过无缝集成外部知识库，在推理过程中实现对特定领域专业知识的实时访问，为专业领域定制 LLM。尽管基于平面文本检索的传统 RAG 系统具有巨大潜力，但它仍面临三大关键挑战：(i) 专业背景下的复杂查询理解、(ii) 跨分布式源的知识集成困难以及 (iii) 大规模系统效率瓶颈。本综述对基于图的检索增强生成 (GraphRAG) 进行了系统分析，这是一种彻底改变特定领域 LLM 应用的新范式。 GraphRAG 通过三项关键创新解决了传统 RAG 的局限性：(i) 图形结构知识表示，明确捕获实体关系和域层次结构，(ii) 高效的基于图形的检索技术，能够通过多跳推理能力实现保留上下文的知识检索，以及 (iii) 结构感知知识集成算法，利用检索到的知识准确、合乎逻辑地生成 LLM。在本次调查中，我们系统地分析了 GraphRAG 的技术基础，并研究了各个专业领域的当前实现情况，确定了关键的技术挑战和有前景的研究方向。GraphRAG 的所有相关资源，包括研究论文、开源数据和项目，都收集在 \textcolor{blue}{\url{此 https URL}} 中供社区使用。

Title: Assisting Mathematical Formalization with A Learning-based Premise Retriever

• Authors: Yicheng Tao, Haotian Liu, Shanwen Wang, Hongteng Xu
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2501.13959
• Pdf URL: https://arxiv.org/pdf/2501.13959
• Copy Paste: [[2501.13959]] Assisting Mathematical Formalization with A Learning-based Premise Retriever(https://arxiv.org/abs/2501.13959)
• Keywords: language model
• Abstract: Premise selection is a crucial yet challenging step in mathematical formalization, especially for users with limited experience. Due to the lack of available formalization projects, existing approaches that leverage language models often suffer from data scarcity. In this work, we introduce an innovative method for training a premise retriever to support the formalization of mathematics. Our approach employs a BERT model to embed proof states and premises into a shared latent space. The retrieval model is trained within a contrastive learning framework and incorporates a domain-specific tokenizer along with a fine-grained similarity computation method. Experimental results show that our model is highly competitive compared to existing baselines, achieving strong performance while requiring fewer computational resources. Performance is further enhanced through the integration of a re-ranking module. To streamline the formalization process, we will release a search engine that enables users to query Mathlib theorems directly using proof states, significantly improving accessibility and efficiency. Codes are available at this https URL.
• 摘要：前提选择是数学形式化中至关重要但又极具挑战性的一步，尤其是对于经验有限的用户而言。由于缺乏可用的形式化项目，利用语言模型的现有方法通常会受到数据稀缺的影响。在这项工作中，我们引入了一种训练前提检索器的创新方法，以支持数学的形式化。我们的方法采用 BERT 模型将证明状态和前提嵌入到共享潜在空间中。检索模型在对比学习框架内进行训练，并结合了领域特定的标记器以及细粒度的相似度计算方法。实验结果表明，与现有基线相比，我们的模型具有很高的竞争力，在需要更少计算资源的同时实现了强大的性能。通过集成重新排名模块，性能得到进一步增强。为了简化形式化过程，我们将发布一个搜索引擎，使用户能够使用证明状态直接查询 Mathlib 定理，从而显著提高可访问性和效率。代码可在此 https URL 上找到。

Title: Towards Safer Social Media Platforms: Scalable and Performant Few-Shot Harmful Content Moderation Using Large Language Models

• Authors: Akash Bonagiri, Lucen Li, Rajvardhan Oak, Zeerak Babar, Magdalena Wojcieszak, Anshuman Chhabra
• Subjects: cs.CL, cs.AI, cs.CY, cs.SI
• Abstract URL: https://arxiv.org/abs/2501.13976
• Pdf URL: https://arxiv.org/pdf/2501.13976
• Copy Paste: [[2501.13976]] Towards Safer Social Media Platforms: Scalable and Performant Few-Shot Harmful Content Moderation Using Large Language Models(https://arxiv.org/abs/2501.13976)
• Keywords: language model, llm
• Abstract: The prevalence of harmful content on social media platforms poses significant risks to users and society, necessitating more effective and scalable content moderation strategies. Current approaches rely on human moderators, supervised classifiers, and large volumes of training data, and often struggle with scalability, subjectivity, and the dynamic nature of harmful content (e.g., violent content, dangerous challenge trends, etc.). To bridge these gaps, we utilize Large Language Models (LLMs) to undertake few-shot dynamic content moderation via in-context learning. Through extensive experiments on multiple LLMs, we demonstrate that our few-shot approaches can outperform existing proprietary baselines (Perspective and OpenAI Moderation) as well as prior state-of-the-art few-shot learning methods, in identifying harm. We also incorporate visual information (video thumbnails) and assess if different multimodal techniques improve model performance. Our results underscore the significant benefits of employing LLM based methods for scalable and dynamic harmful content moderation online.
• 摘要：社交媒体平台上有害内容的盛行对用户和社会构成了重大风险，因此需要更有效、更可扩展的内容审核策略。当前的方法依赖于人工审核员、监督分类器和大量训练数据，并且经常难以应对有害内容（例如暴力内容、危险挑战趋势等）的可扩展性、主观性和动态性。为了弥补这些差距，我们利用大型语言模型 (LLM) 通过情境学习进行少量动态内容审核。通过在多个 LLM 上进行大量实验，我们证明了我们的少量方法在识别危害方面可以胜过现有的专有基线（Perspective 和 OpenAI Moderation）以及之前最先进的少量学习方法。我们还结合了视觉信息（视频缩略图）并评估不同的多模态技术是否能提高模型性能。我们的结果强调了采用基于 LLM 的方法进行可扩展和动态的在线有害内容审核的显著优势。

Title: Re-ranking Using Large Language Models for Mitigating Exposure to Harmful Content on Social Media Platforms

• Authors: Rajvardhan Oak, Muhammad Haroon, Claire Jo, Magdalena Wojcieszak, Anshuman Chhabra
• Subjects: cs.CL, cs.AI, cs.CY, cs.SI
• Abstract URL: https://arxiv.org/abs/2501.13977
• Pdf URL: https://arxiv.org/pdf/2501.13977
• Copy Paste: [[2501.13977]] Re-ranking Using Large Language Models for Mitigating Exposure to Harmful Content on Social Media Platforms(https://arxiv.org/abs/2501.13977)
• Keywords: language model, llm
• Abstract: Social media platforms utilize Machine Learning (ML) and Artificial Intelligence (AI) powered recommendation algorithms to maximize user engagement, which can result in inadvertent exposure to harmful content. Current moderation efforts, reliant on classifiers trained with extensive human-annotated data, struggle with scalability and adapting to new forms of harm. To address these challenges, we propose a novel re-ranking approach using Large Language Models (LLMs) in zero-shot and few-shot settings. Our method dynamically assesses and re-ranks content sequences, effectively mitigating harmful content exposure without requiring extensive labeled data. Alongside traditional ranking metrics, we also introduce two new metrics to evaluate the effectiveness of re-ranking in reducing exposure to harmful content. Through experiments on three datasets, three models and across three configurations, we demonstrate that our LLM-based approach significantly outperforms existing proprietary moderation approaches, offering a scalable and adaptable solution for harm mitigation.
• 摘要：社交媒体平台利用机器学习 (ML) 和人工智能 (AI) 支持的推荐算法来最大限度地提高用户参与度，这可能会导致无意中接触有害内容。当前的审核工作依赖于使用大量人工注释数据训练的分类器，在可扩展性和适应新形式的危害方面存在困难。为了应对这些挑战，我们提出了一种新颖的重新排名方法，该方法在零样本和小样本设置中使用大型语言模型 (LLM)。我们的方法动态评估和重新排名内容序列，有效地减轻有害内容的暴露，而无需大量标记数据。除了传统的排名指标外，我们还引入了两个新指标来评估重新排名在减少有害内容暴露方面的有效性。通过对三个数据集、三个模型和三种配置的实验，我们证明了基于 LLM 的方法明显优于现有的专有审核方法，为减轻危害提供了一种可扩展且适应性强的解决方案。

Title: Chain of Grounded Objectives: Bridging Process and Goal-oriented Prompting for Code Generation

• Authors: Sangyeop Yeo, Seung-won Hwang, Yu-Seung Ma
• Subjects: cs.CL, cs.AI, cs.SE
• Abstract URL: https://arxiv.org/abs/2501.13978
• Pdf URL: https://arxiv.org/pdf/2501.13978
• Copy Paste: [[2501.13978]] Chain of Grounded Objectives: Bridging Process and Goal-oriented Prompting for Code Generation(https://arxiv.org/abs/2501.13978)
• Keywords: language model, llm, prompt
• Abstract: The use of Large Language Models (LLMs) for code generation has gained significant attention in recent years. Existing methods often aim to improve the quality of generated code by incorporating additional contextual information or guidance into input prompts. Many of these approaches adopt sequential reasoning strategies, mimicking human-like step-by-step thinking. However, such strategies may constrain flexibility, as they do not always align with the structured characteristics of programming languages. This paper introduces the Chain of Grounded Objectives (CGO), a method that embeds functional objectives into input prompts to enhance code generation. By leveraging appropriately structured objectives as input and avoiding explicit sequential procedures, CGO adapts effectively to the structured nature of programming tasks. Empirical evaluations demonstrate that CGO effectively enhances code generation, addressing limitations of existing approaches.
• 摘要：近年来，大型语言模型 (LLM) 在代码生成中的应用引起了广泛关注。现有方法通常旨在通过在输入提示中加入额外的上下文信息或指导来提高生成代码的质量。其中许多方法采用顺序推理策略，模仿人类的逐步思维。然而，这些策略可能会限制灵活性，因为它们并不总是与编程语言的结构化特征相一致。本文介绍了一种将功能目标嵌入输入提示以增强代码生成的方法——基于目标的链 (CGO)。通过利用适当结构化的目标作为输入并避免明确的顺序程序，CGO 可以有效地适应编程任务的结构化性质。实证评估表明，CGO 有效地增强了代码生成，解决了现有方法的局限性。

Title: AdEval: Alignment-based Dynamic Evaluation to Mitigate Data Contamination in Large Language Models

• Authors: Yang Fan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.13983
• Pdf URL: https://arxiv.org/pdf/2501.13983
• Copy Paste: [[2501.13983]] AdEval: Alignment-based Dynamic Evaluation to Mitigate Data Contamination in Large Language Models(https://arxiv.org/abs/2501.13983)
• Keywords: language model, llm
• Abstract: As Large Language Models (LLMs) are pretrained on massive-scale corpora, the issue of data contamination has become increasingly severe, leading to potential overestimation of model performance during evaluation. To address this, we propose AdEval (Alignment-based Dynamic Evaluation), a dynamic data evaluation method aimed at mitigating the impact of data contamination on evaluation reliability. AdEval extracts key knowledge points and main ideas to align dynamically generated questions with static data's core concepts. It also leverages online search to provide detailed explanations of related knowledge points, thereby creating high-quality evaluation samples with robust knowledge support. Furthermore, AdEval incorporates mechanisms to control the number and complexity of questions, enabling dynamic alignment and flexible adjustment. This ensures that the generated questions align with the complexity of static data while supporting varied complexity levels. Based on Bloom's taxonomy, AdEval conducts a multi-dimensional evaluation of LLMs across six cognitive levels: remembering, understanding, applying, analyzing, evaluating, and creating. Experimental results on multiple datasets demonstrate that AdEval effectively reduces the impact of data contamination on evaluation outcomes, enhancing both the fairness and reliability of the evaluation process.
• 摘要：随着大型语言模型（LLM）在海量语料上进行预训练，数据污染问题日益严重，评估过程中模型性能可能被高估。针对这一问题，我们提出了一种动态数据评估方法AdEval（Alignment-based Dynamic Evaluation），旨在降低数据污染对评估信度的影响。AdEval通过提取关键知识点和中心思想，使动态生成的问题与静态数据的核心概念对齐，并利用在线搜索对相关知识点进行详细讲解，从而生成具有强知识支持的高质量评估样本。此外，AdEval还结合问题数量和复杂度控制机制，实现动态对齐和灵活调整，确保生成的问题与静态数据复杂度对齐，同时支持不同的复杂度级别。AdEval基于布鲁姆分类法，从记忆、理解、应用、分析、评估和创造六个认知层面对LLM进行多维度评估。在多个数据集上的实验结果表明，AdEval有效地降低了数据污染对评估结果的影响，提高了评估过程的公平性和可靠性。

Title: Comprehensive Modeling and Question Answering of Cancer Clinical Practice Guidelines using LLMs

• Authors: Bhumika Gupta, Pralaypati Ta, Keerthi Ram, Mohanasankar Sivaprakasam
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.13984
• Pdf URL: https://arxiv.org/pdf/2501.13984
• Copy Paste: [[2501.13984]] Comprehensive Modeling and Question Answering of Cancer Clinical Practice Guidelines using LLMs(https://arxiv.org/abs/2501.13984)
• Keywords: language model, llm, hallucination
• Abstract: The updated recommendations on diagnostic procedures and treatment pathways for a medical condition are documented as graphical flows in Clinical Practice Guidelines (CPGs). For effective use of the CPGs in helping medical professionals in the treatment decision process, it is necessary to fully capture the guideline knowledge, particularly the contexts and their relationships in the graph. While several existing works have utilized these guidelines to create rule bases for Clinical Decision Support Systems, limited work has been done toward directly capturing the full medical knowledge contained in CPGs. This work proposes an approach to create a contextually enriched, faithful digital representation of National Comprehensive Cancer Network (NCCN) Cancer CPGs in the form of graphs using automated extraction and node & relationship classification. We also implement semantic enrichment of the model by using Large Language Models (LLMs) for node classification, achieving an accuracy of 80.86% and 88.47% with zero-shot learning and few-shot learning, respectively. Additionally, we introduce a methodology for answering natural language questions with constraints to guideline text by leveraging LLMs to extract the relevant subgraph from the guideline knowledge base. By generating natural language answers based on subgraph paths and semantic information, we mitigate the risk of incorrect answers and hallucination associated with LLMs, ensuring factual accuracy in medical domain Question Answering.
• 摘要：临床实践指南 (CPG) 中以图形流程的形式记录了有关医疗状况诊断程序和治疗途径的最新建议。为了有效使用 CPG 帮助医疗专业人员进行治疗决策，必须充分获取指南知识，尤其是图中的上下文及其关系。虽然现有的一些研究已经利用这些指南为临床决策支持系统创建规则库，但在直接获取 CPG 中包含的全部医学知识方面所做的工作有限。这项工作提出了一种方法，使用自动提取和节点与关系分类以图形形式创建国家综合癌症网络 (NCCN) 癌症 CPG 的上下文丰富、忠实的数字表示。我们还通过使用大型语言模型 (LLM) 进行节点分类来实现模型的语义丰富，分别通过零样本学习和少量学习实现了 80.86% 和 88.47% 的准确率。此外，我们介绍了一种回答自然语言问题的方法，该方法对指南文本有约束，即利用 LLM 从指南知识库中提取相关子图。通过基于子图路径和语义信息生成自然语言答案，我们可以降低与 LLM 相关的错误答案和幻觉的风险，确保医学领域问答中的事实准确性。

Title: CAPRAG: A Large Language Model Solution for Customer Service and Automatic Reporting using Vector and Graph Retrieval-Augmented Generation

• Authors: Hamza Landolsi, Kais Letaief, Nizar Taghouti, Ines Abdeljaoued-Tej
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2501.13993
• Pdf URL: https://arxiv.org/pdf/2501.13993
• Copy Paste: [[2501.13993]] CAPRAG: A Large Language Model Solution for Customer Service and Automatic Reporting using Vector and Graph Retrieval-Augmented Generation(https://arxiv.org/abs/2501.13993)
• Keywords: language model, llm, chat, retrieval-augmented generation, agent
• Abstract: The introduction of new features and services in the banking sector often overwhelms customers, creating an opportunity for banks to enhance user experience through financial chatbots powered by large language models (LLMs). We initiated an AI agent designed to provide customers with relevant information about banking services and insights from annual reports. We proposed a hybrid Customer Analysis Pipeline Retrieval-Augmented Generation (CAPRAG) that effectively addresses both relationship-based and contextual queries, thereby improving customer engagement in the digital banking landscape. To implement this, we developed a processing pipeline to refine text data, which we utilized in two main frameworks: Vector RAG and Graph RAG. This dual approach enables us to populate both vector and graph databases with processed data for efficient retrieval. The Cypher query component is employed to effectively query the graph database. When a user submits a query, it is first expanded by a query expansion module before being routed to construct a final query from the hybrid Knowledge Base (KB). This final query is then sent to an open-source LLM for response generation. Overall, our innovative, designed to international banks, serves bank's customers in an increasingly complex digital environment, enhancing clarity and accessibility of information.
• 摘要：银行业推出新功能和服务时，客户往往会不知所措，这为银行创造了一个机会，即通过由大型语言模型 (LLM) 驱动的金融聊天机器人来提升用户体验。我们启动了一个 AI 代理，旨在为客户提供有关银行服务的相关信息和年度报告中的见解。我们提出了一种混合客户分析管道检索增强生成 (CAPRAG)，可以有效解决基于关系和上下文的查询，从而提高客户在数字银行领域的参与度。为了实现这一点，我们开发了一个处理管道来优化文本数据，我们在两个主要框架中使用了它：Vector RAG 和 Graph RAG。这种双重方法使我们能够用处理后的数据填充矢量和图形数据库，以实现高效检索。Cypher 查询组件用于有效地查询图形数据库。当用户提交查询时，它首先由查询扩展模块扩展，然后被路由以从混合知识库 (KB) 构建最终查询。然后，这个最终查询被发送到开源 LLM 以生成响应。总体而言，我们为国际银行设计的创新产品可在日益复杂的数字环境中为银行客户提供服务，提高信息的清晰度和可访问性。

Title: Framework for Progressive Knowledge Fusion in Large Language Models Through Structured Conceptual Redundancy Analysis

• Authors: Joseph Sakau, Evander Kozlowski, Roderick Thistledown, Basil Steinberger
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.13999
• Pdf URL: https://arxiv.org/pdf/2501.13999
• Copy Paste: [[2501.13999]] Framework for Progressive Knowledge Fusion in Large Language Models Through Structured Conceptual Redundancy Analysis(https://arxiv.org/abs/2501.13999)
• Keywords: language model
• Abstract: The organization of latent knowledge within large-scale models poses unique challenges when addressing overlapping representations and optimizing contextual accuracy. Conceptual redundancies embedded across layers often result in inefficiencies that affect both computational demands and task-specific outcomes. A framework was proposed to restructure these redundancies through advanced clustering techniques and dynamic thresholding, ensuring that critical semantic relationships are preserved while removing unnecessary overlaps. Evaluations revealed improved memory efficiency and faster inference times, alongside better alignment in latent knowledge clusters that enhanced interpretability. Improvements in error rates and adversarial robustness suggest that restructuring redundancies has broader implications for increasing model reliability across diverse applications. Comparative analyses highlighted reductions in resource consumption and notable gains in performance, particularly in translation and summarization tasks. Energy metrics demonstrated significant savings during training phases, further validating the practicality of the approach for real-world deployments. Representational fidelity was also enhanced, with latent space evaluations indicating better cluster alignment and higher semantic consistency. The methodology bridges a key gap in model optimization through directly addressing redundancies at the structural level. Its application opens avenues for scalable, efficient, and contextually aware systems that can adapt to complex, domain-specific tasks without compromising on performance.
• 摘要：在解决重叠表示和优化上下文准确性时，大规模模型中潜在知识的组织带来了独特的挑战。跨层嵌入的概念冗余通常会导致效率低下，从而影响计算需求和特定于任务的结果。提出了一个框架，通过高级聚类技术和动态阈值来重构这些冗余，确保在消除不必要的重叠的同时保留关键的语义关系。评估显示，内存效率得到提高，推理时间更快，同时潜在知识集群中的对齐效果更好，从而增强了可解释性。错误率和对抗鲁棒性的改善表明，重构冗余对于提高不同应用中的模型可靠性具有更广泛的意义。比较分析强调了资源消耗的减少和性能的显着提升，特别是在翻译和摘要任务中。能量指标在训练阶段显示出显着的节省，进一步验证了该方法在实际部署中的实用性。表征保真度也得到了增强，潜在空间评估表明集群对齐更好，语义一致性更高。该方法通过直接解决结构层面的冗余问题，弥补了模型优化的一个关键空白。它的应用为可扩展、高效、情境感知的系统开辟了道路，这些系统可以适应复杂的特定领域任务，而不会影响性能。

Title: Advancing Math Reasoning in Language Models: The Impact of Problem-Solving Data, Data Synthesis Methods, and Training Stages

• Authors: Zui Chen, Tianqiao Liu, Mi Tian, Qing Tong, Weiqi Luo, Zitao Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.14002
• Pdf URL: https://arxiv.org/pdf/2501.14002
• Copy Paste: [[2501.14002]] Advancing Math Reasoning in Language Models: The Impact of Problem-Solving Data, Data Synthesis Methods, and Training Stages(https://arxiv.org/abs/2501.14002)
• Keywords: language model, llm, prompt
• Abstract: Advancements in LLMs have significantly expanded their capabilities across various domains. However, mathematical reasoning remains a challenging area, prompting the development of math-specific LLMs. These models typically follow a two-stage training paradigm: pre-training with math-related corpora and post-training with problem datasets for SFT. Despite these efforts, the improvements in mathematical reasoning achieved through continued pre-training (CPT) are often less significant compared to those obtained via SFT. This study addresses this discrepancy by exploring alternative strategies during the pre-training phase, focusing on the use of problem-solving data over general mathematical corpora. We investigate three primary research questions: (1) Can problem-solving data enhance the model's mathematical reasoning capabilities more effectively than general mathematical corpora during CPT? (2) Are synthetic data from the same source equally effective, and which synthesis methods are most efficient? (3) How do the capabilities developed from the same problem-solving data differ between the CPT and SFT stages, and what factors contribute to these differences? Our findings indicate that problem-solving data significantly enhances the model's mathematical capabilities compared to general mathematical corpora. We also identify effective data synthesis methods, demonstrating that the tutorship amplification synthesis method achieves the best performance. Furthermore, while SFT facilitates instruction-following abilities, it underperforms compared to CPT with the same data, which can be partially attributed to its poor learning capacity for hard multi-step problem-solving data. These insights provide valuable guidance for optimizing the mathematical reasoning capabilities of LLMs, culminating in our development of a powerful mathematical base model called JiuZhang-8B.
• 摘要：LLM 的进步大大扩展了其在各个领域的能力。然而，数学推理仍然是一个具有挑战性的领域，这促使数学专用 LLM 的发展。这些模型通常遵循两阶段训练范式：使用与数学相关的语料库进行预训练，使用 SFT 的问题数据集进行后训练。尽管做出了这些努力，但通过持续预训练 (CPT) 获得的数学推理改进通常不如通过 SFT 获得的改进那么显著。本研究通过探索预训练阶段的替代策略来解决这一差异，重点关注使用问题解决数据而不是一般数学语料库。我们调查了三个主要研究问题：(1) 在 CPT 期间，问题解决数据是否比一般数学语料库更有效地增强模型的数学推理能力？(2) 来自同一来源的合成数据是否同样有效，哪些合成方法最有效？(3) 在 CPT 和 SFT 阶段，从相同的问题解决数据开发的能力有何不同，哪些因素导致了这些差异？我们的研究结果表明，与一般数学语料库相比，问题解决数据显著增强了模型的数学能力。我们还确定了有效的数据合成方法，表明导师放大合成方法取得了最佳效果。此外，虽然 SFT 有助于提高遵循指令的能力，但与使用相同数据的 CPT 相比，它的表现不佳，这部分归因于其对困难的多步骤问题解决数据的学习能力较差。这些见解为优化 LLM 的数学推理能力提供了宝贵的指导，最终我们开发了一个强大的数学基础模型，称为九章-8B。

Title: Leveraging Large Language Models to Analyze Emotional and Contextual Drivers of Teen Substance Use in Online Discussions

• Authors: Jianfeng Zhu, Ruoming Jin, Hailong Jiang, Yulan Wang, Xinyu Zhang, Karin G. Coifman
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.14037
• Pdf URL: https://arxiv.org/pdf/2501.14037
• Copy Paste: [[2501.14037]] Leveraging Large Language Models to Analyze Emotional and Contextual Drivers of Teen Substance Use in Online Discussions(https://arxiv.org/abs/2501.14037)
• Keywords: language model, llm
• Abstract: Adolescence is a critical stage often linked to risky behaviors, including substance use, with significant developmental and public health implications. Social media provides a lens into adolescent self-expression, but interpreting emotional and contextual signals remains complex. This study applies Large Language Models (LLMs) to analyze adolescents' social media posts, uncovering emotional patterns (e.g., sadness, guilt, fear, joy) and contextual factors (e.g., family, peers, school) related to substance use. Heatmap and machine learning analyses identified key predictors of substance use-related posts. Negative emotions like sadness and guilt were significantly more frequent in substance use contexts, with guilt acting as a protective factor, while shame and peer influence heightened substance use risk. Joy was more common in non-substance use discussions. Peer influence correlated strongly with sadness, fear, and disgust, while family and school environments aligned with non-substance use. Findings underscore the importance of addressing emotional vulnerabilities and contextual influences, suggesting that collaborative interventions involving families, schools, and communities can reduce risk factors and foster healthier adolescent development.
• 摘要：青春期是一个关键阶段，通常与包括药物滥用在内的危险行为有关，对发展和公共卫生具有重大影响。社交媒体为青少年的自我表达提供了一个视角，但解读情绪和背景信号仍然很复杂。这项研究应用大型语言模型 (LLM) 分析青少年的社交媒体帖子，揭示与药物滥用相关的情绪模式（例如悲伤、内疚、恐惧、喜悦）和背景因素（例如家庭、同龄人、学校）。热图和机器学习分析确定了药物滥用相关帖子的关键预测因素。在药物滥用环境中，悲伤和内疚等负面情绪更为常见，内疚是一种保护因素，而羞耻和同伴影响会增加药物滥用的风险。在非药物滥用讨论中，喜悦更为常见。同伴影响与悲伤、恐惧和厌恶密切相关，而家庭和学校环境与非药物滥用相关。研究结果强调了解决情感脆弱性和环境影响的重要性，表明家庭、学校和社区之间的协作干预可以减少风险因素并促进青少年更健康地成长。

Title: LLMs are Vulnerable to Malicious Prompts Disguised as Scientific Language

• Authors: Yubin Ge, Neeraja Kirtane, Hao Peng, Dilek Hakkani-Tür
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.14073
• Pdf URL: https://arxiv.org/pdf/2501.14073
• Copy Paste: [[2501.14073]] LLMs are Vulnerable to Malicious Prompts Disguised as Scientific Language(https://arxiv.org/abs/2501.14073)
• Keywords: language model, gpt, llm, prompt
• Abstract: As large language models (LLMs) have been deployed in various real-world settings, concerns about the harm they may propagate have grown. Various jailbreaking techniques have been developed to expose the vulnerabilities of these models and improve their safety. This work reveals that many state-of-the-art proprietary and open-source LLMs are vulnerable to malicious requests hidden behind scientific language. Specifically, our experiments with GPT4o, GPT4o-mini, GPT-4, LLama3-405B-Instruct, Llama3-70B-Instruct, Cohere, Gemini models on the StereoSet data demonstrate that, the models' biases and toxicity substantially increase when prompted with requests that deliberately misinterpret social science and psychological studies as evidence supporting the benefits of stereotypical biases. Alarmingly, these models can also be manipulated to generate fabricated scientific arguments claiming that biases are beneficial, which can be used by ill-intended actors to systematically jailbreak even the strongest models like GPT. Our analysis studies various factors that contribute to the models' vulnerabilities to malicious requests in academic language. Mentioning author names and venues enhances the persuasiveness of some models, and the bias scores can increase as dialogues progress. Our findings call for a more careful investigation on the use of scientific data in the training of LLMs.
• 摘要：随着大型语言模型 (LLM) 被部署到各种现实环境中，人们对它们可能传播的危害的担忧也与日俱增。已经开发了各种越狱技术来暴露这些模型的漏洞并提高其安全性。这项研究表明，许多最先进的专有和开源 LLM 都容易受到隐藏在科学语言背后的恶意请求的攻击。具体来说，我们在 StereoSet 数据上对 GPT4o、GPT4o-mini、GPT-4、LLama3-405B-Instruct、Llama3-70B-Instruct、Cohere、Gemini 模型进行的实验表明，当被要求故意将社会科学和心理学研究曲解为支持刻板偏见好处的证据时，模型的偏见和毒性会大幅增加。令人震惊的是，这些模型还可以被操纵来生成虚假的科学论据，声称偏见是有益的，恶意行为者可以利用这些论据系统地越狱即使是最强大的模型，如 GPT。我们的分析研究了导致模型容易受到学术语言恶意请求攻击的各种因素。提及作者姓名和地点会增强某些模型的说服力，并且随着对话的进展，偏见分数可能会增加。我们的研究结果呼吁对 LLM 培训中科学数据的使用进行更仔细的调查。

Title: Enhancing Biomedical Relation Extraction with Directionality

• Authors: Po-Ting Lai, Chih-Hsuan Wei, Shubo Tian, Robert Leaman, Zhiyong Lu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.14079
• Pdf URL: https://arxiv.org/pdf/2501.14079
• Copy Paste: [[2501.14079]] Enhancing Biomedical Relation Extraction with Directionality(https://arxiv.org/abs/2501.14079)
• Keywords: language model, gpt, prompt
• Abstract: Biological relation networks contain rich information for understanding the biological mechanisms behind the relationship of entities such as genes, proteins, diseases, and chemicals. The vast growth of biomedical literature poses significant challenges updating the network knowledge. The recent Biomedical Relation Extraction Dataset (BioRED) provides valuable manual annotations, facilitating the develop-ment of machine-learning and pre-trained language model approaches for automatically identifying novel document-level (inter-sentence context) relationships. Nonetheless, its annotations lack directionality (subject/object) for the entity roles, essential for studying complex biological networks. Herein we annotate the entity roles of the relationships in the BioRED corpus and subsequently propose a novel multi-task language model with soft-prompt learning to jointly identify the relationship, novel findings, and entity roles. Our results in-clude an enriched BioRED corpus with 10,864 directionality annotations. Moreover, our proposed method outperforms existing large language models such as the state-of-the-art GPT-4 and Llama-3 on two benchmarking tasks. Our source code and dataset are available at this https URL.
• 摘要：生物关系网络包含丰富的信息，可用于理解基因、蛋白质、疾病和化学物质等实体关系背后的生物学机制。生物医学文献的大量增长对更新网络知识提出了重大挑战。最近的生物医学关系提取数据集 (BioRED) 提供了有价值的手动注释，促进了机器学习和预训练语言模型方法的开发，以自动识别新的文档级（句子间上下文）关系。尽管如此，它的注释缺乏实体角色的方向性（主语/宾语），这对于研究复杂的生物网络至关重要。在此，我们注释了 BioRED 语料库中关系的实体角色，随后提出了一种具有软提示学习的新型多任务语言模型，以共同识别关系、新发现和实体角色。我们的结果包括一个丰富的 BioRED 语料库，其中包含 10,864 个方向性注释。此外，我们提出的方法在两个基准测试任务上优于现有的大型语言模型，例如最先进的 GPT-4 和 Llama-3。我们的源代码和数据集可在此 https URL 上获取。

Title: Communicating Activations Between Language Model Agents

• Authors: Vignav Ramesh, Kenneth Li
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.14082
• Pdf URL: https://arxiv.org/pdf/2501.14082
• Copy Paste: [[2501.14082]] Communicating Activations Between Language Model Agents(https://arxiv.org/abs/2501.14082)
• Keywords: language model, agent
• Abstract: Communication between multiple language model (LM) agents has been shown to scale up the reasoning ability of LMs. While natural language has been the dominant medium for inter-LM communication, it is not obvious this should be the standard: not only does natural language communication incur high inference costs that scale quickly with the number of both agents and messages, but also the decoding process abstracts away too much rich information that could be otherwise accessed from the internal activations. In this work, we propose a simple technique whereby LMs communicate via activations; concretely, we pause an LM $\textit{B}$'s computation at an intermediate layer, combine its current activation with another LM $\textit{A}$'s intermediate activation via some function $\textit{f}$, then pass $\textit{f}$'s output into the next layer of $\textit{B}$ and continue the forward pass till decoding is complete. This approach scales up LMs on new tasks with zero additional parameters and data, and saves a substantial amount of compute over natural language communication. We test our method with various functional forms $\textit{f}$ on two experimental setups--multi-player coordination games and reasoning benchmarks--and find that it achieves up to $27.0\%$ improvement over natural language communication across datasets with $<$$1/4$ the compute, illustrating the superiority and robustness of activations as an alternative "language" for communication between LMs.
• 摘要：研究表明，多语言模型 (LM) 代理之间的通信可以提升 LM 的推理能力。虽然自然语言一直是 LM 间通信的主要媒介，但这不应该是标准：自然语言通信不仅会产生高昂的推理成本，而且会随着代理和消息数量的增加而迅速增加，而且解码过程还会抽象出太多原本可以从内部激活中访问的丰富信息。在这项工作中，我们提出了一种简单的技术，让 LM 通过激活进行通信；具体来说，我们在中间层暂停 LM $\textit{B}$ 的计算，通过某个函数 $\textit{f}$ 将其当前激活与另一个 LM $\textit{A}$ 的中间激活相结合，然后将 $\textit{f}$ 的输出传递到 $\textit{B}$ 的下一层并继续前向传递，直到解码完成。这种方法可以在没有额外参数和数据的情况下扩展 LM 在新任务上的规模，并且比自然语言通信节省了大量计算量。我们在两个实验设置（多人协调游戏和推理基准）上使用各种函数形式 $\textit{f}$ 测试了我们的方法，发现它在计算量为 $<$$1/4$ 的情况下，在跨数据集的自然语言通信中实现了高达 $27.0\%$ 的改进，说明了激活作为 LM 之间通信的替代“语言”的优越性和稳健性。

Title: MedSlice: Fine-Tuned Large Language Models for Secure Clinical Note Sectioning

• Authors: Joshua Davis, Thomas Sounack, Kate Sciacca, Jessie M Brain, Brigitte N Durieux, Nicole D Agaronnik, Charlotta Lindvall
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.14105
• Pdf URL: https://arxiv.org/pdf/2501.14105
• Copy Paste: [[2501.14105]] MedSlice: Fine-Tuned Large Language Models for Secure Clinical Note Sectioning(https://arxiv.org/abs/2501.14105)
• Keywords: language model, gpt, llm
• Abstract: Extracting sections from clinical notes is crucial for downstream analysis but is challenging due to variability in formatting and labor-intensive nature of manual sectioning. While proprietary large language models (LLMs) have shown promise, privacy concerns limit their accessibility. This study develops a pipeline for automated note sectioning using open-source LLMs, focusing on three sections: History of Present Illness, Interval History, and Assessment and Plan. We fine-tuned three open-source LLMs to extract sections using a curated dataset of 487 progress notes, comparing results relative to proprietary models (GPT-4o, GPT-4o mini). Internal and external validity were assessed via precision, recall and F1 score. Fine-tuned Llama 3.1 8B outperformed GPT-4o (F1=0.92). On the external validity test set, performance remained high (F1= 0.85). Fine-tuned open-source LLMs can surpass proprietary models in clinical note sectioning, offering advantages in cost, performance, and accessibility.
• 摘要：从临床笔记中提取章节对于下游分析至关重要，但由于格式多变且手动分段劳动密集，因此具有挑战性。虽然专有大型语言模型 (LLM) 已显示出良好的前景，但隐私问题限制了它们的可访问性。本研究开发了一个使用开源 LLM 自动进行笔记分段的流程，重点关注三个部分：现病史、间隔史以及评估和计划。我们对三个开源 LLM 进行了微调，以使用 487 个进度记录的精选数据集提取章节，并将结果与​​专有模型 (GPT-4o、GPT-4o mini) 进行比较。通过精度、召回率和 F1 分数评估内部和外部有效性。微调后的 Llama 3.1 8B 优于 GPT-4o (F1=0.92)。在外部有效性测试集上，性能仍然很高 (F1= 0.85)。微调后的开源 LLM 在临床笔记分段方面可以超越专有模型，在成本、性能和可访问性方面具有优势。

Title: Autonomous Structural Memory Manipulation for Large Language Models Using Hierarchical Embedding Augmentation

• Authors: Derek Yotheringhay, Alistair Kirkland, Humphrey Kirkbride, Josiah Whitesteeple
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.14119
• Pdf URL: https://arxiv.org/pdf/2501.14119
• Copy Paste: [[2501.14119]] Autonomous Structural Memory Manipulation for Large Language Models Using Hierarchical Embedding Augmentation(https://arxiv.org/abs/2501.14119)
• Keywords: language model
• Abstract: Transformative innovations in model architectures have introduced hierarchical embedding augmentation as a means to redefine the representation of tokens through multi-level semantic structures, offering enhanced adaptability to complex linguistic inputs. Autonomous structural memory manipulation further advances this paradigm through dynamic memory reallocation mechanisms that prioritize critical contextual features while suppressing less relevant information, enabling scalable and efficient performance across diverse tasks. Experimental results reveal substantial improvements in computational efficiency, with marked reductions in processing overhead for longer input sequences, achieved through memory reorganization strategies that adapt to evolving contextual requirements. Hierarchical embeddings not only improved contextual alignment but also facilitated task generalization by capturing relationships at varying semantic granularities, ensuring coherence across layers without introducing significant computational redundancies. Comparative analysis against baseline models demonstrated unique advantages in accuracy, efficiency, and interpretability, particularly in tasks requiring complex contextual understanding or domain-specific adaptability. The ability to dynamically adjust token representations and memory configurations contributed to the model's robustness under varied and unpredictable input conditions. Applications benefiting from these advancements include multi-domain generalization, interactive systems, and scenarios involving real-time decision-making, where traditional static memory architectures often face limitations. The proposed methodology combines advanced embedding and memory management strategies into a cohesive framework that addresses scalability challenges while preserving task-specific relevance.
• 摘要：模型架构中的变革性创新引入了分层嵌入增强，以此通过多级语义结构重新定义标记的表示，从而增强对复杂语言输入的适应性。自主结构内存操作通过动态内存重新分配机制进一步推进了这一范式，该机制优先考虑关键上下文特征，同时抑制不太相关的信息，从而实现跨不同任务的可扩展和高效性能。实验结果表明，通过适应不断变化的上下文要求的内存重组策略，计算效率显著提高，较长输入序列的处理开销显著减少。分层嵌入不仅改善了上下文对齐，而且还通过捕获不同语义粒度的关系促进了任务泛化，确保了跨层的一致性，而不会引入显著的计算冗余。与基线模型的比较分析表明，在准确性、效率和可解释性方面具有独特的优势，特别是在需要复杂上下文理解或特定领域适应性的任务中。动态调整标记表示和内存配置的能力有助于模型在变化和不可预测的输入条件下保持稳健性。受益于这些进步的应用包括多领域泛化、交互系统和涉及实时决策的场景，而传统的静态内存架构往往面临限制。所提出的方法将先进的嵌入和内存管理策略结合到一个有凝聚力的框架中，该框架可解决可扩展性挑战，同时保留特定于任务的相关性。

Title: Test-Time Code-Switching for Cross-lingual Aspect Sentiment Triplet Extraction

• Authors: Dongming Sheng, Kexin Han, Hao Li, Yan Zhang, Yucheng Huang, Jun Lang, Wenqiang Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.14144
• Pdf URL: https://arxiv.org/pdf/2501.14144
• Copy Paste: [[2501.14144]] Test-Time Code-Switching for Cross-lingual Aspect Sentiment Triplet Extraction(https://arxiv.org/abs/2501.14144)
• Keywords: gpt, chat
• Abstract: Aspect Sentiment Triplet Extraction (ASTE) is a thriving research area with impressive outcomes being achieved on high-resource languages. However, the application of cross-lingual transfer to the ASTE task has been relatively unexplored, and current code-switching methods still suffer from term boundary detection issues and out-of-dictionary problems. In this study, we introduce a novel Test-Time Code-SWitching (TT-CSW) framework, which bridges the gap between the bilingual training phase and the monolingual test-time prediction. During training, a generative model is developed based on bilingual code-switched training data and can produce bilingual ASTE triplets for bilingual inputs. In the testing stage, we employ an alignment-based code-switching technique for test-time augmentation. Extensive experiments on cross-lingual ASTE datasets validate the effectiveness of our proposed method. We achieve an average improvement of 3.7% in terms of weighted-averaged F1 in four datasets with different languages. Additionally, we set a benchmark using ChatGPT and GPT-4, and demonstrate that even smaller generative models fine-tuned with our proposed TT-CSW framework surpass ChatGPT and GPT-4 by 14.2% and 5.0% respectively.
• 摘要：体感情绪三元组提取 (ASTE) 是一个蓬勃发展的研究领域，在高资源语言上取得了令人瞩目的成果。然而，跨语言迁移在 ASTE 任务中的应用相对尚未得到探索，当前的代码转换方法仍然存在术语边界检测问题和词典外问题。在本研究中，我们引入了一种新颖的测试时间代码转换 (TT-CSW) 框架，它弥合了双语训练阶段和单语测试时间预测之间的差距。在训练期间，基于双语代码转换训练数据开发生成模型，可以为双语输入生成双语 ASTE 三元组。在测试阶段，我们采用基于对齐的代码转换技术进行测试时间增强。在跨语言 ASTE 数据集上的大量实验验证了我们提出的方法的有效性。我们在四个不同语言的数据集中实现了加权平均 F1 平均 3.7% 的提升。此外，我们使用 ChatGPT 和 GPT-4 设置了基准，并证明使用我们提出的 TT-CSW 框架进行微调的更小的生成模型分别比 ChatGPT 和 GPT-4 分别高出 14.2% 和 5.0%。

Title: Multi-agent KTO: Reinforcing Strategic Interactions of Large Language Model in Language Game

• Authors: Rong Ye, Yongxin Zhang, Yikai Zhang, Haoyu Kuang, Zhongyu Wei, Peng Sun
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2501.14225
• Pdf URL: https://arxiv.org/pdf/2501.14225
• Copy Paste: [[2501.14225]] Multi-agent KTO: Reinforcing Strategic Interactions of Large Language Model in Language Game(https://arxiv.org/abs/2501.14225)
• Keywords: language model, gpt, agent
• Abstract: Achieving Artificial General Intelligence (AGI) requires AI agents that can not only make stratigic decisions but also engage in flexible and meaningful communication. Inspired by Wittgenstein's language game theory in Philosophical Investigations, we propose that language agents can learn through in-context interaction rather than traditional multi-stage frameworks that separate decision-making from language expression. Using Werewolf, a social deduction game that tests language understanding, strategic interaction, and adaptability, we develop the Multi-agent Kahneman & Tversky's Optimization (MaKTO). MaKTO engages diverse models in extensive gameplay to generate unpaired desirable and unacceptable responses, then employs KTO to refine the model's decision-making process. In 9-player Werewolf games, MaKTO achieves a 61% average win rate across various models, outperforming GPT-4o and two-stage RL agents by relative improvements of 23.0% and 10.9%, respectively. Notably, MaKTO also demonstrates human-like performance, winning 60% against expert players and showing only 49% detectability in Turing-style blind tests. These results showcase MaKTO's superior decision-making, strategic adaptation, and natural language generation in complex social deduction games.
• 摘要：实现通用人工智能 (AGI) 需要 AI 代理不仅能够做出战略决策，还能进行灵活而有意义的交流。受维特根斯坦《哲学研究》中语言博弈论的启发，我们提出语言代理可以通过情境交互进行学习，而不是将决策与语言表达分开的传统多阶段框架。使用狼人杀（一种测试语言理解、战略互动和适应性的社交推理游戏），我们开发了多代理卡尼曼和特沃斯基优化 (MaKTO)。MaKTO 在广泛的游戏中使用各种模型来生成不成对的理想和不可接受的反应，然后使用 KTO 来改进模型的决策过程。在 9 人狼人游戏中，MaKTO 在各种模型中实现了 61% 的平均胜率，比 GPT-4o 和两阶段 RL 代理分别高出 23.0% 和 10.9%。值得注意的是，MaKTO 还展现出了与人类类似的表现，在与专业玩家的对战中胜率达到 60%，在图灵式盲测中仅显示出 49% 的可检测性。这些结果展示了 MaKTO 在复杂的社交推理游戏中卓越的决策能力、战略适应能力和自然语言生成能力。

Title: Siren: A Learning-Based Multi-Turn Attack Framework for Simulating Real-World Human Jailbreak Behaviors

• Authors: Yi Zhao, Youzhi Zhang
• Subjects: cs.CL, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2501.14250
• Pdf URL: https://arxiv.org/pdf/2501.14250
• Copy Paste: [[2501.14250]] Siren: A Learning-Based Multi-Turn Attack Framework for Simulating Real-World Human Jailbreak Behaviors(https://arxiv.org/abs/2501.14250)
• Keywords: language model, gpt, llm, prompt
• Abstract: Large language models (LLMs) are widely used in real-world applications, raising concerns about their safety and trustworthiness. While red-teaming with jailbreak prompts exposes the vulnerabilities of LLMs, current efforts focus primarily on single-turn attacks, overlooking the multi-turn strategies used by real-world adversaries. Existing multi-turn methods rely on static patterns or predefined logical chains, failing to account for the dynamic strategies during attacks. We propose Siren, a learning-based multi-turn attack framework designed to simulate real-world human jailbreak behaviors. Siren consists of three stages: (1) training set construction utilizing Turn-Level LLM feedback (Turn-MF), (2) post-training attackers with supervised fine-tuning (SFT) and direct preference optimization (DPO), and (3) interactions between the attacking and target LLMs. Experiments demonstrate that Siren achieves an attack success rate (ASR) of 90% with LLaMA-3-8B as the attacker against Gemini-1.5-Pro as the target model, and 70% with Mistral-7B against GPT-4o, significantly outperforming single-turn baselines. Moreover, Siren with a 7B-scale model achieves performance comparable to a multi-turn baseline that leverages GPT-4o as the attacker, while requiring fewer turns and employing decomposition strategies that are better semantically aligned with attack goals. We hope Siren inspires the development of stronger defenses against advanced multi-turn jailbreak attacks under realistic scenarios. Code is available at this https URL. Warning: This paper contains potentially harmful text.
• 摘要：大型语言模型 (LLM) 在实际应用中得到广泛应用，这引发了人们对其安全性和可信度的担忧。虽然使用越狱提示的红队攻击暴露了 LLM 的漏洞，但当前的努力主要集中在单轮攻击上，而忽略了现实世界对手使用的多轮策略。现有的多轮方法依赖于静态模式或预定义的逻辑链，无法解释攻击期间的动态策略。我们提出了 Siren，这是一个基于学习的多轮攻击框架，旨在模拟现实世界中的人类越狱行为。Siren 包括三个阶段：(1) 利用回合级 LLM 反馈 (Turn-MF) 构建训练集，(2) 使用监督微调 (SFT) 和直接偏好优化 (DPO) 对攻击者进行后训练，以及 (3) 攻击和目标 LLM 之间的交互。实验表明，Siren 使用 LLaMA-3-8B 作为攻击者对 Gemini-1.5-Pro 作为目标模型的攻击成功率 (ASR) 达到 90%，使用 Mistral-7B 对 GPT-4o 的攻击成功率达到 70%，显著优于单回合基线。此外，使用 7B 规模模型的 Siren 实现了与利用 GPT-4o 作为攻击者的多回合基线相当的性能，同时需要更少的回合并采用与攻击目标在语义上更一致的分解策略。我们希望 Siren 能够激发在现实场景下开发更强大的防御措施以抵御高级多回合越狱攻击。代码可在此 https URL 上获取。警告：本文包含潜在有害文本。

Title: Leveraging Online Olympiad-Level Math Problems for LLMs Training and Contamination-Resistant Evaluation

• Authors: Sadegh Mahdavi, Muchen Li, Kaiwen Liu, Christos Thrampoulidis, Leonid Sigal, Renjie Liao
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.14275
• Pdf URL: https://arxiv.org/pdf/2501.14275
• Copy Paste: [[2501.14275]] Leveraging Online Olympiad-Level Math Problems for LLMs Training and Contamination-Resistant Evaluation(https://arxiv.org/abs/2501.14275)
• Keywords: language model, llm
• Abstract: Advances in Large Language Models (LLMs) have sparked interest in their ability to solve Olympiad-level math problems. However, the training and evaluation of these models are constrained by the limited size and quality of available datasets, as creating large-scale data for such advanced problems requires extensive effort from human experts. In addition, current benchmarks are prone to contamination, leading to unreliable evaluations. In this paper, we present an automated pipeline that leverages the rich resources of the Art of Problem Solving (AoPS) forum, which predominantly features Olympiad-level problems and community-driven solutions. Using open-source LLMs, we develop a method to extract question-answer pairs from the forum, resulting in AoPS-Instruct, a dataset of more than 600,000 high-quality QA pairs. Our experiments demonstrate that fine-tuning LLMs on AoPS-Instruct improves their reasoning abilities across various benchmarks. Moreover, we build an automatic pipeline that introduces LiveAoPSBench, an evolving evaluation set with timestamps, derived from the latest forum data, providing a contamination-resistant benchmark for assessing LLM performance. Notably, we observe a significant decline in LLM performance over time, suggesting their success on older examples may stem from pre-training exposure rather than true reasoning ability. Our work presents a scalable approach to creating and maintaining large-scale, high-quality datasets for advanced math reasoning, offering valuable insights into the capabilities and limitations of LLMs in this domain. Our benchmark and code is available at this https URL
• 摘要：大型语言模型 (LLM) 的进步引发了人们对其解决奥林匹克级数学问题能力的兴趣。然而，这些模型的训练和评估受到可用数据集的有限大小和质量的限制，因为为此类高级问题创建大规模数据需要人类专家付出大量努力。此外，当前的基准容易受到污染，导致评估不可靠。在本文中，我们提出了一个自动化流程，利用问题解决艺术 (AoPS) 论坛的丰富资源，该论坛主要以奥林匹克级问题和社区驱动的解决方案为特色。使用开源 LLM，我们开发了一种从论坛中提取问答对的方法，从而产生了 AoPS-Instruct，这是一个包含 600,000 多个高质量 QA 对的数据集。我们的实验表明，在 AoPS-Instruct 上微调 LLM 可以提高它们在各种基准上的推理能力。此外，我们构建了一个自动化管道，引入了 LiveAoPSBench，这是一个不断发展的评估集，带有时间戳，源自最新的论坛数据，为评估 LLM 性能提供了抗污染基准。值得注意的是，我们观察到 LLM 性能随着时间的推移显着下降，这表明它们在旧示例上的成功可能源于预训练暴露而不是真正的推理能力。我们的工作提出了一种可扩展的方法来创建和维护用于高级数学推理的大规模高质量数据集，为 LLM 在该领域的能力和局限性提供了宝贵的见解。我们的基准和代码可在此 https URL 上找到

Title: Examining Alignment of Large Language Models through Representative Heuristics: The Case of Political Stereotypes

• Authors: Sullam Jeoung, Yubin Ge, Haohan Wang, Jana Diesner
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.14294
• Pdf URL: https://arxiv.org/pdf/2501.14294
• Copy Paste: [[2501.14294]] Examining Alignment of Large Language Models through Representative Heuristics: The Case of Political Stereotypes(https://arxiv.org/abs/2501.14294)
• Keywords: language model, llm, prompt
• Abstract: Examining the alignment of large language models (LLMs) has become increasingly important, particularly when these systems fail to operate as intended. This study explores the challenge of aligning LLMs with human intentions and values, with specific focus on their political inclinations. Previous research has highlighted LLMs' propensity to display political leanings, and their ability to mimic certain political parties' stances on various issues. However, the extent and conditions under which LLMs deviate from empirical positions have not been thoroughly examined. To address this gap, our study systematically investigates the factors contributing to LLMs' deviations from empirical positions on political issues, aiming to quantify these deviations and identify the conditions that cause them. Drawing on cognitive science findings related to representativeness heuristics -- where individuals readily recall the representative attribute of a target group in a way that leads to exaggerated beliefs -- we scrutinize LLM responses through this heuristics lens. We conduct experiments to determine how LLMs exhibit stereotypes by inflating judgments in favor of specific political parties. Our results indicate that while LLMs can mimic certain political parties' positions, they often exaggerate these positions more than human respondents do. Notably, LLMs tend to overemphasize representativeness to a greater extent than humans. This study highlights the susceptibility of LLMs to representativeness heuristics, suggeseting potential vulnerabilities to political stereotypes. We propose prompt-based mitigation strategies that demonstrate effectiveness in reducing the influence of representativeness in LLM responses.
• 摘要：检查大型语言模型 (LLM) 的一致性变得越来越重要，特别是当这些系统无法按预期运行时。本研究探讨了将 LLM 与人类意图和价值观相一致的挑战，特别关注其政治倾向。先前的研究强调了 LLM 表现出政治倾向的倾向，以及它们模仿某些政党在各种问题上的立场的能力。然而，LLM 偏离经验立场的程度和条件尚未得到彻底研究。为了解决这一差距，我们的研究系统地调查了导致 LLM 在政治问题上偏离经验立场的因素，旨在量化这些偏差并确定导致偏差的条件。借鉴与代表性启发法相关的认知科学发现——个人很容易回忆起目标群体的代表性属性，从而导致夸大信念——我们通过这种启发式视角审视 LLM 的反应。我们进行实验，以确定 LLM 如何通过夸大对特定政党的判断来表现出刻板印象。我们的结果表明，虽然 LLM 可以模仿某些政党的立场，但它们往往比人类受访者更夸大这些立场。值得注意的是，LLM 往往比人类更过分强调代表性。这项研究强调了 LLM 对代表性启发法的敏感性，表明它可能容易受到政治刻板印象的影响。我们提出了基于提示的缓解策略，这些策略证明了在减少 LLM 响应中代表性的影响方面是有效的。

Title: Clear Minds Think Alike: What Makes LLM Fine-tuning Robust? A Study of Token Perplexity

• Authors: Chao-Chung Wu, Zhi Rui Tam, Chieh-Yen Lin, Hung-yi Lee, Yun-Nung Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.14315
• Pdf URL: https://arxiv.org/pdf/2501.14315
• Copy Paste: [[2501.14315]] Clear Minds Think Alike: What Makes LLM Fine-tuning Robust? A Study of Token Perplexity(https://arxiv.org/abs/2501.14315)
• Keywords: llm
• Abstract: Maintaining consistent model performance across domains is a fundamental challenge in machine learning. While recent work has explored using LLM-generated data for fine-tuning, its impact on cross-domain generalization remains poorly understood. In this paper, we present a systematic analysis revealing that fine-tuning with LLM-generated data not only improves target task performance but also reduces out-of-domain (OOD) degradation compared to fine-tuning with ground truth data. Through analyzing the data sequence in tasks of various domains, we demonstrate that this enhanced OOD robustness stems from a reduced prevalence of high perplexity tokens in LLM-generated sequences. Following this hypothesis we showed that masking high perplexity tokens in ground truth training data also achieves similar OOD preservation comparable to using LLM-generated data. Extensive experiments across diverse model architectures and scales, including Gemma2-2B, Mistral-7B and Llama3-8B, corroborate the consistency of our findings. To the best of our knowledge, this work provides the first mechanistic explanation for the superior OOD robustness conferred by LLM-generated training data, offering valuable insights for developing more robust fine-tuning strategies.
• 摘要：保持跨领域的一致模型性能是机器学习的一个基本挑战。虽然最近的研究已经探索了使用 LLM 生成的数据进行微调，但它对跨域泛化的影响仍然不太清楚。在本文中，我们进行了系统分析，表明与使用地面实况数据进行微调相比，使用 LLM 生成的数据进行微调不仅可以提高目标任务性能，而且还可以减少域外 (OOD) 性能下降。通过分析各个领域任务中的数据序列，我们证明了这种增强的 OOD 鲁棒性源于 LLM 生成序列中高困惑度标记的流行率降低。根据这一假设，我们表明，在地面实况训练数据中屏蔽高困惑度标记也能实现与使用 LLM 生成的数据相当的类似 OOD 保存。在包括 Gemma2-2B、Mistral-7B 和 Llama3-8B 在内的各种模型架构和规模上进行的大量实验证实了我们的发现的一致性。据我们所知，这项工作首次从机制上解释了 LLM 生成的训练数据所赋予的卓越 OOD 稳健性，为开发更稳健的微调策略提供了宝贵的见解。

Title: DRESSing Up LLM: Efficient Stylized Question-Answering via Style Subspace Editing

• Authors: Xinyu Ma, Yifeng Xu, Yang Lin, Tianlong Wang, Xu Chu, Xin Gao, Junfeng Zhao, Yasha Wang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.14371
• Pdf URL: https://arxiv.org/pdf/2501.14371
• Copy Paste: [[2501.14371]] DRESSing Up LLM: Efficient Stylized Question-Answering via Style Subspace Editing(https://arxiv.org/abs/2501.14371)
• Keywords: language model, llm, prompt, agent
• Abstract: We introduce DRESS, a novel approach for generating stylized large language model (LLM) responses through representation editing. Existing methods like prompting and fine-tuning are either insufficient for complex style adaptation or computationally expensive, particularly in tasks like NPC creation or character role-playing. Our approach leverages the over-parameterized nature of LLMs to disentangle a style-relevant subspace within the model's representation space to conduct representation editing, ensuring a minimal impact on the original semantics. By applying adaptive editing strengths, we dynamically adjust the steering vectors in the style subspace to maintain both stylistic fidelity and semantic integrity. We develop two stylized QA benchmark datasets to validate the effectiveness of DRESS, and the results demonstrate significant improvements compared to baseline methods such as prompting and ITI. In short, DRESS is a lightweight, train-free solution for enhancing LLMs with flexible and effective style control, making it particularly useful for developing stylized conversational agents. Codes and benchmark datasets are available at this https URL.
• 摘要：我们介绍了 DRESS，这是一种通过表征编辑生成风格化大型语言模型 (LLM) 响应的新方法。现有的方法（如提示和微调）要么不足以适应复杂的风格，要么计算成本高昂，尤其是在 NPC 创建或角色扮演等任务中。我们的方法利用 LLM 的过度参数化特性来解开模型表征空间内与风格相关的子空间以进行表征编辑，确保对原始语义的影响最小。通过应用自适应编辑优势，我们动态调整风格子空间中的转向向量，以保持风格保真度和语义完整性。我们开发了两个风格化的 QA 基准数据集来验证 DRESS 的有效性，结果显示与提示和 ITI 等基线方法相比有显着改进。简而言之，DRESS 是一种轻量级、无需训练的解决方案，可通过灵活有效的风格控制增强 LLM，使其特别适用于开发风格化的对话代理。代码和基准数据集可在此 https URL 上找到。

Title: Domaino1s: Guiding LLM Reasoning for Explainable Answers in High-Stakes Domains

• Authors: Xu Chu, Zhijie Tan, Hanlin Xue, Guanyu Wang, Tong Mo, Weiping Li
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.14431
• Pdf URL: https://arxiv.org/pdf/2501.14431
• Copy Paste: [[2501.14431]] Domaino1s: Guiding LLM Reasoning for Explainable Answers in High-Stakes Domains(https://arxiv.org/abs/2501.14431)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) are widely applied to downstream domains. However, current LLMs for high-stakes domain tasks, such as financial investment and legal QA, typically generate brief answers without reasoning processes and explanations. This limits users' confidence in making decisions based on their responses. While original CoT shows promise, it lacks self-correction mechanisms during reasoning. This work introduces Domain$o1$s, which enhances LLMs' reasoning capabilities on domain tasks through supervised fine-tuning and tree search. We construct CoT-stock-2k and CoT-legal-2k datasets for fine-tuning models that activate domain-specific reasoning steps based on their judgment. Additionally, we propose Selective Tree Exploration to spontaneously explore solution spaces and sample optimal reasoning paths to improve performance. We also introduce PROOF-Score, a new metric for evaluating domain models' explainability, complementing traditional accuracy metrics with richer assessment dimensions. Extensive experiments on stock investment recommendation and legal reasoning QA tasks demonstrate Domaino1s's leading performance and explainability. Our code is available at this https URL.
• 摘要：大型语言模型 (LLM) 广泛应用于下游领域。然而，目前用于高风险领域任务（如金融投资和法律问答）的 LLM 通常会生成简短的答案，而没有推理过程和解释。这限制了用户根据他们的反应做出决策的信心。虽然原始 CoT 很有前途，但它在推理过程中缺乏自我纠正机制。这项工作引入了 Domain$o1$s，它通过监督微调和树搜索增强了 LLM 在领域任务上的推理能力。我们为微调模型构建了 CoT-stock-2k 和 CoT-legal-2k 数据集，这些模型根据他们的判断激活特定领域的推理步骤。此外，我们提出了选择性树探索来自发探索解决方案空间并采样最佳推理路径以提高性能。我们还引入了 PROOF-Score，这是一种用于评估领域模型可解释性的新指标，以更丰富的评估维度补充了传统的准确性指标。在股票投资推荐和法律推理 QA 任务上进行的大量实验证明了 Domaino1s 的领先性能和可解释性。我们的代码可在此 https URL 上找到。

Title: Understanding and Mitigating Gender Bias in LLMs via Interpretable Neuron Editing

• Authors: Zeping Yu, Sophia Ananiadou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.14457
• Pdf URL: https://arxiv.org/pdf/2501.14457
• Copy Paste: [[2501.14457]] Understanding and Mitigating Gender Bias in LLMs via Interpretable Neuron Editing(https://arxiv.org/abs/2501.14457)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) often exhibit gender bias, posing challenges for their safe deployment. Existing methods to mitigate bias lack a comprehensive understanding of its mechanisms or compromise the model's core capabilities. To address these issues, we propose the CommonWords dataset, to systematically evaluate gender bias in LLMs. Our analysis reveals pervasive bias across models and identifies specific neuron circuits, including gender neurons and general neurons, responsible for this behavior. Notably, editing even a small number of general neurons can disrupt the model's overall capabilities due to hierarchical neuron interactions. Based on these insights, we propose an interpretable neuron editing method that combines logit-based and causal-based strategies to selectively target biased neurons. Experiments on five LLMs demonstrate that our method effectively reduces gender bias while preserving the model's original capabilities, outperforming existing fine-tuning and editing approaches. Our findings contribute a novel dataset, a detailed analysis of bias mechanisms, and a practical solution for mitigating gender bias in LLMs.
• 摘要：大型语言模型 (LLM) 经常表现出性别偏见，这对其安全部署构成了挑战。现有的缓解偏见的方法缺乏对其机制的全面了解，或损害了模型的核心功能。为了解决这些问题，我们提出了 CommonWords 数据集，以系统地评估 LLM 中的性别偏见。我们的分析揭示了模型中普遍存在的偏见，并确定了导致这种行为的特定神经元回路，包括性别神经元和一般神经元。值得注意的是，由于神经元的层次化相互作用，即使编辑少量的一般神经元也会破坏模型的整体能力。基于这些见解，我们提出了一种可解释的神经元编辑方法，该方法结合了基于逻辑和基于因果的策略来选择性地针对有偏见的神经元。在五个 LLM 上进行的实验表明，我们的方法有效地减少了性别偏见，同时保留了模型的原始能力，优于现有的微调和编辑方法。我们的研究结果贡献了一个新颖的数据集、对偏见机制的详细分析以及缓解 LLM 中性别偏见的实用解决方案。

Title: RealCritic: Towards Effectiveness-Driven Evaluation of Language Model Critiques

• Authors: Zhengyang Tang, Ziniu Li, Zhenyang Xiao, Tian Ding, Ruoyu Sun, Benyou Wang, Dayiheng Liu, Fei Huang, Tianyu Liu, Bowen Yu, Junyang Lin
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.14492
• Pdf URL: https://arxiv.org/pdf/2501.14492
• Copy Paste: [[2501.14492]] RealCritic: Towards Effectiveness-Driven Evaluation of Language Model Critiques(https://arxiv.org/abs/2501.14492)
• Keywords: language model, llm, chain-of-thought
• Abstract: Critiques are important for enhancing the performance of Large Language Models (LLMs), enabling both self-improvement and constructive feedback for others by identifying flaws and suggesting improvements. However, evaluating the critique capabilities of LLMs presents a significant challenge due to the open-ended nature of the task. In this work, we introduce a new benchmark designed to assess the critique capabilities of LLMs. Unlike existing benchmarks, which typically function in an open-loop fashion, our approach employs a closed-loop methodology that evaluates the quality of corrections generated from critiques. Moreover, the benchmark incorporates features such as self-critique, cross-critique, and iterative critique, which are crucial for distinguishing the abilities of advanced reasoning models from more classical ones. We implement this benchmark using eight challenging reasoning tasks. We have several interesting findings. First, despite demonstrating comparable performance in direct chain-of-thought generation, classical LLMs significantly lag behind the advanced reasoning-based model o1-mini across all critique scenarios. Second, in self-critique and iterative critique settings, classical LLMs may even underperform relative to their baseline capabilities. We hope that this benchmark will serve as a valuable resource to guide future advancements. The code and data are available at \url{this https URL}.
• 摘要：批评对于提高大型语言模型 (LLM) 的性能非常重要，它既可以自我完善，也可以通过发现缺陷和提出改进建议为他人提供建设性反馈。然而，由于任务的开放性，评估 LLM 的批评能力是一项重大挑战。在这项工作中，我们引入了一个旨在评估 LLM 批评能力的新基准。与通常以开环方式运行的现有基准不同，我们的方法采用闭环方法来评估从批评中生成的更正的质量。此外，基准还结合了自我批评、交叉批评和迭代批评等功能，这些功能对于区分高级推理模型和更经典的推理模型的能力至关重要。我们使用八个具有挑战性的推理任务来实现这个基准。我们有几个有趣的发现。首先，尽管在直接思路链生成中表现出相当的性能，但经典 LLM 在所有批评场景中都明显落后于基于高级推理的模型 o1-mini。其次，在自我批评和迭代批评环境中，传统法学硕士甚至可能表现不佳，甚至低于其基线能力。我们希望这个基准将成为指导未来进步的宝贵资源。代码和数据可在 \url{此 https URL} 上找到。

Title: Evaluating and Improving Graph to Text Generation with Large Language Models

• Authors: Jie He, Yijun Yang, Wanqiu Long, Deyi Xiong, Victor Gutierrez Basulto, Jeff Z. Pan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.14497
• Pdf URL: https://arxiv.org/pdf/2501.14497
• Copy Paste: [[2501.14497]] Evaluating and Improving Graph to Text Generation with Large Language Models(https://arxiv.org/abs/2501.14497)
• Keywords: language model, llm, prompt
• Abstract: Large language models (LLMs) have demonstrated immense potential across various tasks. However, research for exploring and improving the capabilities of LLMs in interpreting graph structures remains limited. To address this gap, we conduct a comprehensive evaluation of prompting current open-source LLMs on graph-to-text generation tasks. Although we explored the optimal prompting strategies and proposed a novel and effective diversity-difficulty-based few-shot sample selection method, we found that the improvements from tuning-free approaches were incremental, as LLMs struggle with planning on complex graphs, particularly those with a larger number of triplets. To further improve LLMs in planning with graph sequences and grounding in truth, we introduce a new graph-to-text dataset, PlanGTG, annotated with two sub-tasks: reordering and attribution. Through extensive automatic and human evaluations, we demonstrate significant improvements in the quality of generated text from both few-shot learning and fine-tuning perspectives using the PlanGTG dataset. Our study paves the way for new research directions in graph-to-text generation. PlanGTG datasets can be found in this https URL.
• 摘要：大型语言模型 (LLM) 在各种任务中都表现出巨大的潜力。然而，探索和改进 LLM 在解释图结构方面的能力的研究仍然有限。为了解决这一差距，我们对当前开源 LLM 在图到文本生成任务上的提示进行了全面评估。尽管我们探索了最佳提示策略并提出了一种新颖有效的基于多样性难度的少量样本选择方法，但我们发现，无需调整的方法带来的改进是渐进的，因为 LLM 在复杂图上的规划很困难，尤其是那些具有大量三元组的图。为了进一步改进 LLM 在规划图序列和基于事实方面的能力，我们引入了一个新的图到文本数据集 PlanGTG，其中注释了两个子任务：重新排序和归因。通过广泛的自动和人工评估，我们展示了使用 PlanGTG 数据集从少量学习和微调角度生成的文本质量的显着提高。我们的研究为图到文本生成的新研究方向铺平了道路。 PlanGTG 数据集可在此 https URL 中找到。

Title: Investigating the (De)Composition Capabilities of Large Language Models in Natural-to-Formal Language Conversion

• Authors: Ziyao Xu, Houfeng Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.14649
• Pdf URL: https://arxiv.org/pdf/2501.14649
• Copy Paste: [[2501.14649]] Investigating the (De)Composition Capabilities of Large Language Models in Natural-to-Formal Language Conversion(https://arxiv.org/abs/2501.14649)
• Keywords: language model, llm
• Abstract: To achieve generalized and robust natural-to-formal language conversion (N2F), large language models (LLMs) need to have strong capabilities of decomposition and composition in N2F when faced with an unfamiliar formal language and be able to cope with compositional gaps and counter-intuitive symbolic names. To investigate whether LLMs have this set of basic capabilities in N2F, we propose the DEDC framework. This framework semi-automatically performs sample and task construction, allowing decoupled evaluation of the set of decomposition and composition capabilities of LLMs in N2F. Based on this framework, we evaluate and analyze the most advanced LLMs, and the main findings include that: (1) the LLMs are deficient in both decomposition and composition; (2) the LLMs show a wide coverage of error types that can be attributed to deficiencies in natural language understanding and the learning and use of symbolic systems; (3) compositional gaps and counter-intuitive symbolic names both affect the decomposition and composition of the LLMs. Our work provides a new perspective for investigating the basic capabilities of decomposition and composition of LLMs in N2F. The detailed analysis of deficiencies and attributions can help subsequent improvements of LLMs.
• 摘要：为了实现通用且鲁棒的自然语言到形式语言转换（N2F），大型语言模型（LLM）在面对不熟悉的形式语言时需要在 N2F 中具有强大的分解和组合能力，并且能够应对组合差距和违反直觉的符号名称。为了研究 LLM 是否具备这组 N2F 基本能力，我们提出了 DEDC 框架。该框架半自动地执行样本和任务构建，允许对 N2F 中 LLM 的分解和组合能力集进行解耦评估。基于该框架，我们对最先进的 LLM 进行了评估和分析，主要发现包括：（1）LLM 在分解和组合方面都存在不足；（2）LLM 的错误类型覆盖范围很广，这可以归因于自然语言理解以及符号系统的学习和使用的不足；（3）组合差距和违反直觉的符号名称都会影响 LLM 的分解和组合。本研究为探究N2F中LLM的分解与合成基本能力提供了新的视角，对不足之处及归因的详细分析可为后续LLM的改进提供参考。

Title: Rethinking Table Instruction Tuning

• Authors: Naihao Deng, Rada Mihalcea
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.14693
• Pdf URL: https://arxiv.org/pdf/2501.14693
• Copy Paste: [[2501.14693]] Rethinking Table Instruction Tuning(https://arxiv.org/abs/2501.14693)
• Keywords: language model, gpt, llm
• Abstract: Recent advances in table understanding have focused on instruction-tuning large language models (LLMs) for table-related tasks. However, existing research has overlooked the impact of hyperparameter choices and lacks a comprehensive evaluation of the out-of-domain table understanding ability and the general capabilities of these table LLMs. In this paper, we evaluate these abilities in existing table LLMs, and reveal significant declines in both out-of-domain table understanding and general capabilities compared to their base models. Through systematic analysis, we show that hyperparameters, such as learning rate, can significantly influence both table-specific and general capabilities. Contrary to the existing table instruction-tuning works, we demonstrate that smaller learning rates and fewer training instances can enhance table understanding while preserving general capabilities. Based on our findings, we introduce TAMA, a TAble LLM instruction-tuned from LLaMA 3.1 8B Instruct, which achieves performance on par with, or surpassing GPT-3.5 and GPT-4 on table tasks, while maintaining strong out-of-domain generalization and general capabilities. Our findings highlight the potential for reduced data annotation costs and more efficient model development through careful hyperparameter selection.
• 摘要：表格理解领域的最新进展主要集中在针对表格相关任务的指令调优大型语言模型 (LLM)。然而，现有研究忽视了超参数选择的影响，缺乏对这些表格 LLM 的领域外表格理解能力和一般能力的全面评估。在本文中，我们评估了现有表格 LLM 的这些能力，并发现与基础模型相比，领域外表格理解和一般能力都显著下降。通过系统分析，我们发现学习率等超参数可以显著影响表格特定能力和一般能力。与现有的表格指令调优工作相反，我们证明较小的学习率和较少的训练实例可以增强表格理解，同时保留一般能力。基于我们的研究结果，我们引入了 TAMA，这是一个从 LLaMA 3.1 8B Instruct 指令调优的 TAble LLM，它在表格任务上的表现与 GPT-3.5 和 GPT-4 相当或超过它们，同时保持了强大的领域外泛化和一般能力。我们的研究强调了通过仔细选择超参数可以降低数据注释成本并提高模型开发的效率。

Title: FlexiGPT: Pruning and Extending Large Language Models with Low-Rank Weight Sharing

• Authors: James Seale Smith, Chi-Heng Lin, Shikhar Tuli, Haris Jeelani, Shangqian Gao, Yilin Shen, Hongxia Jin, Yen-Chang Hsu
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.14713
• Pdf URL: https://arxiv.org/pdf/2501.14713
• Copy Paste: [[2501.14713]] FlexiGPT: Pruning and Extending Large Language Models with Low-Rank Weight Sharing(https://arxiv.org/abs/2501.14713)
• Keywords: language model, gpt, llm
• Abstract: The rapid proliferation of large language models (LLMs) in natural language processing (NLP) has created a critical need for techniques that enable efficient deployment on memory-constrained devices without compromising performance. We present a method to prune LLMs that selectively prunes model blocks based on an importance score and replaces them with a low-parameter replacement strategy. Specifically, we propose a principled metric to replace each pruned block using a weight-sharing mechanism that leverages unpruned counterparts from the model and block-specific low-rank adapters. Furthermore, we facilitate the learning of these replacement blocks with output feature normalization and an adapter initialization scheme built on low-rank SVD reconstructions. Empirical evaluations demonstrate substantial performance gains over existing methods, achieving state-of-the-art performance on 5/6 benchmarks for a compression rate of 30% and 6/6 benchmarks for a compression rate of 40%. We also demonstrate that our approach can extend smaller models, boosting performance on 6/6 benchmarks using only ~0.3% tokens of extended training with minimal additional parameter costs.
• 摘要：大型语言模型 (LLM) 在自然语言处理 (NLP) 领域的快速普及，迫切需要能够在内存受限的设备上高效部署且不影响性能的技术。我们提出了一种修剪 LLM 的方法，该方法根据重要性得分选择性地修剪模型块，并用低参数替换策略替换它们。具体来说，我们提出了一个原则性指标，使用权重共享机制替换每个修剪后的块，该机制利用模型中未修剪的对应块和块特定的低秩适配器。此外，我们通过输出特征规范化和基于低秩 SVD 重建的适配器初始化方案来促进这些替换块的学习。实证评估表明，与现有方法相比，该方法具有显着的性能提升，在 5/6 个基准测试中实现了最佳性能，压缩率为 30%，在 6/6 个基准测试中实现了最佳性能，压缩率为 40%。我们还证明了我们的方法可以扩展较小的模型，仅使用约 0.3% 的扩展训练令牌并以最小的额外参数成本提高 6/6 基准的性能。

Title: Towards Better Understanding Table Instruction Tuning: Decoupling the Effects from Data versus Models

• Authors: Naihao Deng, Sheng Zhang, Henghui Zhu, Shuaichen Chang, Jiani Zhang, Alexander Hanbo Li, Chung-Wei Hang, Hideo Kobayashi, Yiqun Hu, Patrick Ng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.14717
• Pdf URL: https://arxiv.org/pdf/2501.14717
• Copy Paste: [[2501.14717]] Towards Better Understanding Table Instruction Tuning: Decoupling the Effects from Data versus Models(https://arxiv.org/abs/2501.14717)
• Keywords: language model, llm
• Abstract: Recent advances in natural language processing have leveraged instruction tuning to enhance Large Language Models (LLMs) for table-related tasks. However, previous works train different base models with different training data, lacking an apples-to-apples comparison across the result table LLMs. To address this, we fine-tune base models from the Mistral, OLMo, and Phi families on existing public training datasets. Our replication achieves performance on par with or surpassing existing table LLMs, establishing new state-of-the-art performance on Hitab, a table question-answering dataset. More importantly, through systematic out-of-domain evaluation, we decouple the contributions of training data and the base model, providing insight into their individual impacts. In addition, we assess the effects of table-specific instruction tuning on general-purpose benchmarks, revealing trade-offs between specialization and generalization.
• 摘要：自然语言处理领域的最新进展利用指令调整来增强与表格相关的任务的大型语言模型 (LLM)。然而，以前的研究使用不同的训练数据训练不同的基础模型，缺乏对结果表 LLM 进行同类比较的能力。为了解决这个问题，我们在现有的公共训练数据集上对 Mistral、OLMo 和 Phi 系列的基础模型进行了微调。我们的复制实现了与现有表 LLM 相当甚至超越的性能，在表格问答数据集 Hitab 上建立了新的最先进性能。更重要的是，通过系统的域外评估，我们将训练数据和基础模型的贡献分离出来，深入了解它们各自的影响。此外，我们评估了表特定指令调整对通用基准的影响，揭示了专业化和泛化之间的权衡。

Title: Do LLMs Provide Consistent Answers to Health-Related Questions across Languages?

• Authors: Ipek Baris Schlicht, Zhixue Zhao, Burcu Sayin, Lucie Flek, Paolo Rosso
• Subjects: cs.CL, cs.AI, cs.HC, cs.IR
• Abstract URL: https://arxiv.org/abs/2501.14719
• Pdf URL: https://arxiv.org/pdf/2501.14719
• Copy Paste: [[2501.14719]] Do LLMs Provide Consistent Answers to Health-Related Questions across Languages?(https://arxiv.org/abs/2501.14719)
• Keywords: language model, llm, prompt
• Abstract: Equitable access to reliable health information is vital for public health, but the quality of online health resources varies by language, raising concerns about inconsistencies in Large Language Models (LLMs) for healthcare. In this study, we examine the consistency of responses provided by LLMs to health-related questions across English, German, Turkish, and Chinese. We largely expand the HealthFC dataset by categorizing health-related questions by disease type and broadening its multilingual scope with Turkish and Chinese translations. We reveal significant inconsistencies in responses that could spread healthcare misinformation. Our main contributions are 1) a multilingual health-related inquiry dataset with meta-information on disease categories, and 2) a novel prompt-based evaluation workflow that enables sub-dimensional comparisons between two languages through parsing. Our findings highlight key challenges in deploying LLM-based tools in multilingual contexts and emphasize the need for improved cross-lingual alignment to ensure accurate and equitable healthcare information.
• 摘要：公平地获取可靠的健康信息对公共健康至关重要，但在线健康资源的质量因语言而异，这引发了人们对医疗保健大型语言模型 (LLM) 不一致问题的担忧。在本研究中，我们检查了 LLM 对英语、德语、土耳其语和中文健康相关问题的回答的一致性。我们通过按疾病类型对健康相关问题进行分类，并通过土耳其语和中文翻译扩大其多语言范围，大大扩展了 HealthFC 数据集。我们发现，回答中存在重大不一致，这可能会传播医疗保健错误信息。我们的主要贡献是 1) 一个包含疾病类别元信息的多语言健康相关查询数据集，以及 2) 一种新颖的基于提示的评估工作流程，可通过解析实现两种语言之间的子维度比较。我们的研究结果强调了在多语言环境中部署基于 LLM 的工具的关键挑战，并强调需要改进跨语言一致性以确保准确和公平的医疗保健信息。