2024-11-22

Title: AddrLLM: Address Rewriting via Large Language Model on Nationwide Logistics Data

• Authors: Qinchen Yang, Zhiqing Hong, Dongjiang Cao, Haotian Wang, Zejun Xie, Tian He, Yunhuai Liu, Yu Yang, Desheng Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.13584
• Pdf URL: https://arxiv.org/pdf/2411.13584
• Copy Paste: [[2411.13584]] AddrLLM: Address Rewriting via Large Language Model on Nationwide Logistics Data(https://arxiv.org/abs/2411.13584)
• Keywords: language model, llm, retrieval augmented generation
• Abstract: Textual description of a physical location, commonly known as an address, plays an important role in location-based services(LBS) such as on-demand delivery and navigation. However, the prevalence of abnormal addresses, those containing inaccuracies that fail to pinpoint a location, have led to significant costs. Address rewriting has emerged as a solution to rectify these abnormal addresses. Despite the critical need, existing address rewriting methods are limited, typically tailored to correct specific error types, or frequently require retraining to process new address data effectively. In this study, we introduce AddrLLM, an innovative framework for address rewriting that is built upon a retrieval augmented large language model. AddrLLM overcomes aforementioned limitations through a meticulously designed Supervised Fine-Tuning module, an Address-centric Retrieval Augmented Generation module and a Bias-free Objective Alignment module. To the best of our knowledge, this study pioneers the application of LLM-based address rewriting approach to solve the issue of abnormal addresses. Through comprehensive offline testing with real-world data on a national scale and subsequent online deployment, AddrLLM has demonstrated superior performance in integration with existing logistics system. It has significantly decreased the rate of parcel re-routing by approximately 43\%, underscoring its exceptional efficacy in real-world applications.
• 摘要：物理位置的文本描述（通常称为地址）在基于位置的服务 (LBS)（例如按需交付和导航）中起着重要作用。然而，异常地址（包含无法精确定位位置的不准确信息）的普遍存在已导致大量成本。地址重写已成为纠正这些异常地址的解决方案。尽管迫切需要，但现有的地址重写方法有限，通常针对纠正特定错误类型进行量身定制，或者经常需要重新训练才能有效处理新地址数据。在本研究中，我们介绍了 AddrLLM，这是一个基于检索增强大型语言模型的地址重写创新框架。AddrLLM 通过精心设计的监督微调模块、以地址为中心的检索增强生成模块和无偏差客观对齐模块克服了上述限制。据我们所知，本研究率先应用基于 LLM 的地址重写方法来解决异常地址问题。经过全国范围内基于真实数据的全面离线测试以及随后的在线部署，AddrLLM 在与现有物流系统的集成方面表现出了卓越的性能。它显著降低了包裹改道率约 43%，凸显了其在实际应用中的卓越功效。

Title: Hymba: A Hybrid-head Architecture for Small Language Models

• Authors: Xin Dong, Yonggan Fu, Shizhe Diao, Wonmin Byeon, Zijia Chen, Ameya Sunil Mahabaleshwarkar, Shih-Yang Liu, Matthijs Van Keirsbilck, Min-Hung Chen, Yoshi Suhara, Yingyan Lin, Jan Kautz, Pavlo Molchanov
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.13676
• Pdf URL: https://arxiv.org/pdf/2411.13676
• Copy Paste: [[2411.13676]] Hymba: A Hybrid-head Architecture for Small Language Models(https://arxiv.org/abs/2411.13676)
• Keywords: language model, prompt
• Abstract: We propose Hymba, a family of small language models featuring a hybrid-head parallel architecture that integrates transformer attention mechanisms with state space models (SSMs) for enhanced efficiency. Attention heads provide high-resolution recall, while SSM heads enable efficient context summarization. Additionally, we introduce learnable meta tokens that are prepended to prompts, storing critical information and alleviating the "forced-to-attend" burden associated with attention mechanisms. This model is further optimized by incorporating cross-layer key-value (KV) sharing and partial sliding window attention, resulting in a compact cache size. During development, we conducted a controlled study comparing various architectures under identical settings and observed significant advantages of our proposed architecture. Notably, Hymba achieves state-of-the-art results for small LMs: Our Hymba-1.5B-Base model surpasses all sub-2B public models in performance and even outperforms Llama-3.2-3B with 1.32% higher average accuracy, an 11.67x cache size reduction, and 3.49x throughput.
• 摘要：我们提出了 Hymba，这是一系列小型语言模型，具有混合头并行架构，将 Transformer 注意力机制与状态空间模型 (SSM) 集成在一起，以提高效率。注意力头提供高分辨率召回，而 SSM 头实现高效的上下文摘要。此外，我们引入了可学习的元标记，这些标记被添加到提示前面，存储关键信息并减轻与注意力机制相关的“强制注意”负担。通过结合跨层键值 (KV) 共享和部分滑动窗口注意力，该模型得到进一步优化，从而实现了紧凑的缓存大小。在开发过程中，我们进行了一项对照研究，在相同的设置下比较了各种架构，并观察到我们提出的架构的显著优势。值得注意的是，Hymba 在小型 LM 中取得了最先进的结果：我们的 Hymba-1.5B-Base 模型在性能上超越了所有 2B 以下的公共模型，甚至优于 Llama-3.2-3B，平均准确率高出 1.32%，缓存大小减少了 11.67 倍，吞吐量提高了 3.49 倍。

Title: Assessing Gender Bias in LLMs: Comparing LLM Outputs with Human Perceptions and Official Statistics

• Authors: Tetiana Bas
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.13738
• Pdf URL: https://arxiv.org/pdf/2411.13738
• Copy Paste: [[2411.13738]] Assessing Gender Bias in LLMs: Comparing LLM Outputs with Human Perceptions and Official Statistics(https://arxiv.org/abs/2411.13738)
• Keywords: language model, llm
• Abstract: This study investigates gender bias in large language models (LLMs) by comparing their gender perception to that of human respondents, U.S. Bureau of Labor Statistics data, and a 50% no-bias benchmark. We created a new evaluation set using occupational data and role-specific sentences. Unlike common benchmarks included in LLM training data, our set is newly developed, preventing data leakage and test set contamination. Five LLMs were tested to predict the gender for each role using single-word answers. We used Kullback-Leibler (KL) divergence to compare model outputs with human perceptions, statistical data, and the 50% neutrality benchmark. All LLMs showed significant deviation from gender neutrality and aligned more with statistical data, still reflecting inherent biases.
• 摘要：本研究通过将大型语言模型 (LLM) 的性别认知与人类受访者的性别认知、美国劳工统计局数据和 50% 无偏见基准进行比较，调查了大型语言模型 (LLM) 中的性别偏见。我们使用职业数据和特定于角色的句子创建了一个新的评估集。与 LLM 训练数据中包含的常见基准不同，我们的数据集是新开发的，可防止数据泄露和测试集污染。我们测试了五个 LLM，使用单词答案预测每个角色的性别。我们使用 Kullback-Leibler (KL) 散度将模型输出与人类认知、统计数据和 50% 中立基准进行比较。所有 LLM 都显示出与性别中立的显著偏差，并且与统计数据更加一致，但仍然反映了固有的偏见。

Title: Benchmarking GPT-4 against Human Translators: A Comprehensive Evaluation Across Languages, Domains, and Expertise Levels

• Authors: Jianhao Yan, Pingchuan Yan, Yulong Chen, Jing Li, Xianchao Zhu, Yue Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.13775
• Pdf URL: https://arxiv.org/pdf/2411.13775
• Copy Paste: [[2411.13775]] Benchmarking GPT-4 against Human Translators: A Comprehensive Evaluation Across Languages, Domains, and Expertise Levels(https://arxiv.org/abs/2411.13775)
• Keywords: gpt, llm, hallucination
• Abstract: This study presents a comprehensive evaluation of GPT-4's translation capabilities compared to human translators of varying expertise levels. Through systematic human evaluation using the MQM schema, we assess translations across three language pairs (Chinese$\longleftrightarrow$English, Russian$\longleftrightarrow$English, and Chinese$\longleftrightarrow$Hindi) and three domains (News, Technology, and Biomedical). Our findings reveal that GPT-4 achieves performance comparable to junior-level translators in terms of total errors, while still lagging behind senior translators. Unlike traditional Neural Machine Translation systems, which show significant performance degradation in resource-poor language directions, GPT-4 maintains consistent translation quality across all evaluated language pairs. Through qualitative analysis, we identify distinctive patterns in translation approaches: GPT-4 tends toward overly literal translations and exhibits lexical inconsistency, while human translators sometimes over-interpret context and introduce hallucinations. This study represents the first systematic comparison between LLM and human translators across different proficiency levels, providing valuable insights into the current capabilities and limitations of LLM-based translation systems.
• 摘要：本研究对 GPT-4 的翻译能力与不同专业水平的人工翻译进行了全面评估。通过使用 MQM 模式进行系统人工评估，我们评估了三种语言对（中文$\longleftrightarrow$英语、俄语$\longleftrightarrow$英语和中文$\longleftrightarrow$印地语）和三个领域（新闻、技术和生物医学）的翻译。我们的研究结果表明，GPT-4 在总错误率方面的表现与初级翻译相当，但仍落后于高级翻译。与传统的神经机器翻译系统在资源匮乏的语言方向上表现出显著的性能下降不同，GPT-4 在所有评估的语言对中保持一致的翻译质量。通过定性分析，我们发现了翻译方法中的独特模式：GPT-4 倾向于过度直译并表现出词汇不一致，而人工翻译有时会过度解释上下文并引入幻觉。这项研究首次对不同熟练程度的法学硕士翻译和人工翻译进行了系统性比较，为了解当前基于法学硕士的翻译系统的能力和局限性提供了宝贵的见解。

Title: NewsInterview: a Dataset and a Playground to Evaluate LLMs' Ground Gap via Informational Interviews

• Authors: Michael Lu, Hyundong Justin Cho, Weiyan Shi, Jonathan May, Alexander Spangher
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.13779
• Pdf URL: https://arxiv.org/pdf/2411.13779
• Copy Paste: [[2411.13779]] NewsInterview: a Dataset and a Playground to Evaluate LLMs' Ground Gap via Informational Interviews(https://arxiv.org/abs/2411.13779)
• Keywords: language model, llm, agent
• Abstract: Large Language Models (LLMs) have demonstrated impressive capabilities in generating coherent text but often struggle with grounding language and strategic dialogue. To address this gap, we focus on journalistic interviews, a domain rich in grounding communication and abundant in data. We curate a dataset of 40,000 two-person informational interviews from NPR and CNN, and reveal that LLMs are significantly less likely than human interviewers to use acknowledgements and to pivot to higher-level questions. Realizing that a fundamental deficit exists in multi-turn planning and strategic thinking, we develop a realistic simulated environment, incorporating source personas and persuasive elements, in order to facilitate the development of agents with longer-horizon rewards. Our experiments show that while source LLMs mimic human behavior in information sharing, interviewer LLMs struggle with recognizing when questions are answered and engaging persuasively, leading to suboptimal information extraction across model size and capability. These findings underscore the need for enhancing LLMs' strategic dialogue capabilities.
• 摘要：大型语言模型 (LLM) 在生成连贯文本方面表现出了令人印象深刻的能力，但往往在语言基础和战略对话方面存在困难。为了弥补这一差距，我们专注于新闻采访，这是一个基础交流丰富且数据丰富的领域。我们从 NPR 和 CNN 整理了 40,000 个双人信息采访的数据集，并发现 LLM 使用致谢和转向更高级别问题的可能性明显低于人类采访者。意识到在多轮规划和战略思维方面存在根本缺陷，我们开发了一个现实的模拟环境，结合源角色和说服元素，以促进具有长期回报的代理的发展。我们的实验表明，虽然源 LLM 模仿人类在信息共享中的行为，但采访者 LLM 难以识别问题何时得到回答并进行说服性参与，导致模型大小和能力的信息提取不理想。这些发现强调了增强 LLM 战略对话能力的必要性。

Title: Explaining GPT-4's Schema of Depression Using Machine Behavior Analysis

• Authors: Adithya V Ganesan, Vasudha Varadarajan, Yash Kumar Lal, Veerle C. Eijsbroek, Katarina Kjell, Oscar N.E. Kjell, Tanuja Dhanasekaran, Elizabeth C. Stade, Johannes C. Eichstaedt, Ryan L. Boyd, H. Andrew Schwartz, Lucie Flek
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.13800
• Pdf URL: https://arxiv.org/pdf/2411.13800
• Copy Paste: [[2411.13800]] Explaining GPT-4's Schema of Depression Using Machine Behavior Analysis(https://arxiv.org/abs/2411.13800)
• Keywords: language model, gpt, chat
• Abstract: Use of large language models such as ChatGPT (GPT-4) for mental health support has grown rapidly, emerging as a promising route to assess and help people with mood disorders, like depression. However, we have a limited understanding of GPT-4's schema of mental disorders, that is, how it internally associates and interprets symptoms. In this work, we leveraged contemporary measurement theory to decode how GPT-4 interrelates depressive symptoms to inform both clinical utility and theoretical understanding. We found GPT-4's assessment of depression: (a) had high overall convergent validity (r = .71 with self-report on 955 samples, and r = .81 with experts judgments on 209 samples); (b) had moderately high internal consistency (symptom inter-correlates r = .23 to .78 ) that largely aligned with literature and self-report; except that GPT-4 (c) underemphasized suicidality's -- and overemphasized psychomotor's -- relationship with other symptoms, and (d) had symptom inference patterns that suggest nuanced hypotheses (e.g. sleep and fatigue are influenced by most other symptoms while feelings of worthlessness/guilt is mostly influenced by depressed mood).
• 摘要：大型语言模型（例如 ChatGPT（GPT-4））在心理健康支持方面的使用增长迅速，成为评估和帮助抑郁症等情绪障碍患者的一种有前途的途径。但是，我们对 GPT-4 的精神障碍图式（即它如何在内部关联和解释症状）的理解有限。在这项工作中，我们利用现代测量理论来解码 GPT-4 如何将抑郁症症状相互关联，以提供临床效用和理论理解。我们发现 GPT-4 对抑郁症的评估：（a）具有较高的整体收敛效度（955 个样本的自我报告 r = .71，209 个样本的专家判断 r = .81）；（b）具有中等高的内部一致性（症状相互关联 r = .23 至 .78），与文献和自我报告基本一致；但 GPT-4 (c) 低估了自杀倾向——而过分强调了心理运动——与其他症状的关系，并且 (d) 具有表明细微假设的症状推断模式（例如，睡眠和疲劳受大多数其他症状的影响，而无价值感/内疚感主要受抑郁情绪的影响）。

Title: SemiKong: Curating, Training, and Evaluating A Semiconductor Industry-Specific Large Language Model

• Authors: Christopher Nguyen, William Nguyen, Atsushi Suzuki, Daisuke Oku, Hong An Phan, Sang Dinh, Zooey Nguyen, Anh Ha, Shruti Raghavan, Huy Vo, Thang Nguyen, Lan Nguyen, Yoshikuni Hirayama
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.13802
• Pdf URL: https://arxiv.org/pdf/2411.13802
• Copy Paste: [[2411.13802]] SemiKong: Curating, Training, and Evaluating A Semiconductor Industry-Specific Large Language Model(https://arxiv.org/abs/2411.13802)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have demonstrated the potential to address some issues within the semiconductor industry. However, they are often general-purpose models that lack the specialized knowledge needed to tackle the unique challenges of this sector, such as the intricate physics and chemistry of semiconductor devices and processes. SemiKong, the first industry-specific LLM for the semiconductor domain, provides a foundation that can be used to develop tailored proprietary models. With SemiKong 1.0, we aim to develop a foundational model capable of understanding etching problems at an expert level. Our key contributions include (a) curating a comprehensive corpus of semiconductor-related texts, (b) creating a foundational model with in-depth semiconductor knowledge, and (c) introducing a framework for integrating expert knowledge, thereby advancing the evaluation process of domain-specific AI models. Through fine-tuning a pre-trained LLM using our curated dataset, we have shown that SemiKong outperforms larger, general-purpose LLMs in various semiconductor manufacturing and design tasks. Our extensive experiments underscore the importance of developing domain-specific LLMs as a foundation for company- or tool-specific proprietary models, paving the way for further research and applications in the semiconductor domain. Code and dataset will be available at this https URL
• 摘要：大型语言模型 (LLM) 已显示出解决半导体行业某些问题的潜力。然而，它们通常是通用模型，缺乏应对该行业独特挑战所需的专业知识，例如半导体器件和工艺的复杂物理和化学性质。SemiKong 是第一个针对半导体领域的行业特定 LLM，它为开发定制的专有模型提供了基础。借助 SemiKong 1.0，我们旨在开发一个能够在专家级别理解蚀刻问题的基础模型。我们的主要贡献包括 (a) 整理全面的半导体相关文本语料库，(b) 创建具有深入半导体知识的基础模型，以及 (c) 引入集成专家知识的框架，从而推进特定领域 AI 模型的评估过程。通过使用我们精选的数据集对预训练的 LLM 进行微调，我们已经证明 SemiKong 在各种半导体制造和设计任务中的表现优于更大的通用 LLM。我们进行了广泛的实验，强调了开发特定领域的 LLM 作为公司或工具特定专有模型的基础的重要性，为半导体领域的进一步研究和应用铺平了道路。代码和数据集将在此 https URL 上提供

Title: InstCache: A Predictive Cache for LLM Serving

• Authors: Longwei Zou, Tingfeng Liu, Kai Chen, Jiangang Kong, Yangdong Deng
• Subjects: cs.CL, cs.DC
• Abstract URL: https://arxiv.org/abs/2411.13820
• Pdf URL: https://arxiv.org/pdf/2411.13820
• Copy Paste: [[2411.13820]] InstCache: A Predictive Cache for LLM Serving(https://arxiv.org/abs/2411.13820)
• Keywords: language model, llm
• Abstract: Large language models are revolutionizing every aspect of human life. However, the unprecedented power comes at the cost of significant computing intensity, suggesting long latency and large energy footprint. Key-Value Cache and Semantic Cache have been proposed as a solution to the above problem, but both suffer from limited scalability due to significant memory cost for each token or instruction embeddings. Motivated by the observations that most instructions are short, repetitive and predictable by LLMs, we propose to predict user-instructions by an instruction-aligned LLM and store them in a predictive cache, so-called InstCache. We introduce an instruction pre-population algorithm based on the negative log likelihood of instructions, determining the cache size with regard to the hit rate. The proposed InstCache is efficiently implemented as a hash table with minimal lookup latency for deployment. Experimental results show that InstCache can achieve up to 51.34% hit rate on LMSys dataset, which corresponds to a 2x speedup, at a memory cost of only 4.5GB.
• 摘要：大型语言模型正在彻底改变人类生活的方方面面。然而，这种前所未有的能力是以巨大的计算强度为代价的，这意味着较长的延迟和较大的能源消耗。键值缓存和语义缓存已被提出作为上述问题的解决方案，但由于每个标记或指令嵌入的内存成本很高，因此两者都存在有限的可扩展性。受大多数指令短、重复且可由 LLM 预测的观察结果的启发，我们提出通过指令对齐的 LLM 预测用户指令并将其存储在预测缓存中，即所谓的 InstCache。我们引入了一种基于指令负对数似然的指令预填充算法，根据命中率确定缓存大小。所提出的 InstCache 被有效地实现为哈希表，具有最小的部署查找延迟。实验结果表明，InstCache 在 LMSys 数据集上可以实现高达 51.34% 的命中率，相当于速度提高了 2 倍，而内存成本仅为 4.5GB。

Title: Interactive and Expressive Code-Augmented Planning with Large Language Models

• Authors: Anthony Z. Liu, Xinhe Wang, Jacob Sansom, Yao Fu, Jongwook Choi, Sungryull Sohn, Jaekyeom Kim, Honglak Lee
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.13826
• Pdf URL: https://arxiv.org/pdf/2411.13826
• Copy Paste: [[2411.13826]] Interactive and Expressive Code-Augmented Planning with Large Language Models(https://arxiv.org/abs/2411.13826)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) demonstrate strong abilities in common-sense reasoning and interactive decision-making, but often struggle with complex, long-horizon planning tasks. Recent techniques have sought to structure LLM outputs using control flow and other code-adjacent techniques to improve planning performance. These techniques include using variables (to track important information) and functions (to divide complex tasks into smaller re-usable sub-tasks). However, purely code-based approaches can be error-prone and insufficient for handling ambiguous or unstructured data. To address these challenges, we propose REPL-Plan, an LLM planning approach that is fully code-expressive (it can utilize all the benefits of code) while also being dynamic (it can flexibly adapt from errors and use the LLM for fuzzy situations). In REPL-Plan, an LLM solves tasks by interacting with a Read-Eval-Print Loop (REPL), which iteratively executes and evaluates code, similar to language shells or interactive code notebooks, allowing the model to flexibly correct errors and handle tasks dynamically. We demonstrate that REPL-Plan achieves strong results across various planning domains compared to previous methods.
• 摘要：大型语言模型 (LLM) 在常识推理和交互式决策方面表现出强大的能力，但通常在处理复杂的长期规划任务时会遇到困难。最近的技术试图使用控制流和其他代码相邻技术来构造 LLM 输出，以提高规划性能。这些技术包括使用变量（跟踪重要信息）和函数（将复杂任务划分为较小的可重复使用的子任务）。然而，纯粹基于代码的方法容易出错，并且不足以处理模糊或非结构化数据。为了应对这些挑战，我们提出了 REPL-Plan，这是一种完全代码表达的 LLM 规划方法（它可以利用代码的所有好处），同时也是动态的（它可以灵活地适应错误并将 LLM 用于模糊情况）。在 REPL-Plan 中，LLM 通过与读取-求值-打印循环 (REPL) 交互来解决任务，REPL 迭代执行和评估代码，类似于语言 shell 或交互式代码笔记本，允许模型灵活地纠正错误并动态处理任务。我们证明，与以前的方法相比，REPL-Plan 在各个规划领域取得了强劲的成果。

Title: PIORS: Personalized Intelligent Outpatient Reception based on Large Language Model with Multi-Agents Medical Scenario Simulation

• Authors: Zhijie Bao, Qingyun Liu, Ying Guo, Zhengqiang Ye, Jun Shen, Shirong Xie, Jiajie Peng, Xuanjing Huang, Zhongyu Wei
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.13902
• Pdf URL: https://arxiv.org/pdf/2411.13902
• Copy Paste: [[2411.13902]] PIORS: Personalized Intelligent Outpatient Reception based on Large Language Model with Multi-Agents Medical Scenario Simulation(https://arxiv.org/abs/2411.13902)
• Keywords: language model, gpt, llm, agent
• Abstract: In China, receptionist nurses face overwhelming workloads in outpatient settings, limiting their time and attention for each patient and ultimately reducing service quality. In this paper, we present the Personalized Intelligent Outpatient Reception System (PIORS). This system integrates an LLM-based reception nurse and a collaboration between LLM and hospital information system (HIS) into real outpatient reception setting, aiming to deliver personalized, high-quality, and efficient reception services. Additionally, to enhance the performance of LLMs in real-world healthcare scenarios, we propose a medical conversational data generation framework named Service Flow aware Medical Scenario Simulation (SFMSS), aiming to adapt the LLM to the real-world environments and PIORS settings. We evaluate the effectiveness of PIORS and SFMSS through automatic and human assessments involving 15 users and 15 clinical experts. The results demonstrate that PIORS-Nurse outperforms all baselines, including the current state-of-the-art model GPT-4o, and aligns with human preferences and clinical needs. Further details and demo can be found at this https URL
• 摘要：在中国，门诊接待护士面临着繁重的工作量，限制了他们为每位患者投入的时间和注意力，最终降低了服务质量。在本文中，我们介绍了个性化智能门诊接待系统 (PIORS)。该系统将基于 LLM 的接待护士以及 LLM 与医院信息系统 (HIS) 之间的协作整合到真实的门诊接待环境中，旨在提供个性化、高质量和高效的接待服务。此外，为了提高 LLM 在现实医疗场景中的表现，我们提出了一种名为服务流感知医疗场景模拟 (SFMSS) 的医疗对话数据生成框架，旨在使 LLM 适应现实环境和 PIORS 设置。我们通过涉及 15 名用户和 15 名临床专家的自动和人工评估来评估 PIORS 和 SFMSS 的有效性。结果表明，PIORS-Nurse 的表现优于所有基线，包括当前最先进的模型 GPT-4o，并且符合人类偏好和临床需求。更多详细信息和演示可在此 https URL 中找到

Title: Towards Full Delegation: Designing Ideal Agentic Behaviors for Travel Planning

• Authors: Song Jiang, Da JU, Andrew Cohen, Sasha Mitts, Aaron Foss, Justine T Kao, Xian Li, Yuandong Tian
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.13904
• Pdf URL: https://arxiv.org/pdf/2411.13904
• Copy Paste: [[2411.13904]] Towards Full Delegation: Designing Ideal Agentic Behaviors for Travel Planning(https://arxiv.org/abs/2411.13904)
• Keywords: llm, agent
• Abstract: How are LLM-based agents used in the future? While many of the existing work on agents has focused on improving the performance of a specific family of objective and challenging tasks, in this work, we take a different perspective by thinking about full delegation: agents take over humans' routine decision-making processes and are trusted by humans to find solutions that fit people's personalized needs and are adaptive to ever-changing context. In order to achieve such a goal, the behavior of the agents, i.e., agentic behaviors, should be evaluated not only on their achievements (i.e., outcome evaluation), but also how they achieved that (i.e., procedure evaluation). For this, we propose APEC Agent Constitution, a list of criteria that an agent should follow for good agentic behaviors, including Accuracy, Proactivity, Efficiency and Credibility. To verify whether APEC aligns with human preferences, we develop APEC-Travel, a travel planning agent that proactively extracts hidden personalized needs via multi-round dialog with travelers. APEC-Travel is constructed purely from synthetic data generated by Llama3.1-405B-Instruct with a diverse set of travelers' persona to simulate rich distribution of dialogs. Iteratively fine-tuned to follow APEC Agent Constitution, APEC-Travel surpasses baselines by 20.7% on rule-based metrics and 9.1% on LLM-as-a-Judge scores across the constitution axes.
• 摘要：基于 LLM 的代理在未来将如何使用？虽然现有的许多代理研究都集中在提高特定客观且具有挑战性的任务系列的性能，但在本研究中，我们采取了不同的视角，考虑完全授权：代理接管人类的日常决策过程，并受到人类的信任，以找到适合人们个性化需求并适应不断变化的环境的解决方案。为了实现这一目标，代理的行为，即代理行为，不仅应根据其成就（即结果评估）进行评估，还应根据其实现方式（即程序评估）进行评估。为此，我们提出了 APEC 代理宪法，这是代理应遵循的良好代理行为标准列表，包括准确性、主动性、效率和可信度。为了验证 APEC 是否符合人类偏好，我们开发了 APEC-Travel，这是一个旅行计划代理，它通过与旅行者进行多轮对话来主动提取隐藏的个性化需求。 APEC-Travel 完全由 Llama3.1-405B-Instruct 生成的合成数据构建而成，其中包含一组多样化的旅行者角色，以模拟丰富的对话分布。APEC-Travel 经过反复微调以遵循 APEC Agent Constitution，在基于规则的指标上超出基线 20.7%，在宪法轴上的 LLM-as-a-Judge 得分上超出基线 9.1%。

Title: Forecasting Future International Events: A Reliable Dataset for Text-Based Event Modeling

• Authors: Daehoon Gwak, Junwoo Park, Minho Park, Chaehun Park, Hyunchan Lee, Edward Choi, Jaegul Choo
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.14042
• Pdf URL: https://arxiv.org/pdf/2411.14042
• Copy Paste: [[2411.14042]] Forecasting Future International Events: A Reliable Dataset for Text-Based Event Modeling(https://arxiv.org/abs/2411.14042)
• Keywords: language model, llm, prompt
• Abstract: Predicting future international events from textual information, such as news articles, has tremendous potential for applications in global policy, strategic decision-making, and geopolitics. However, existing datasets available for this task are often limited in quality, hindering the progress of related research. In this paper, we introduce WORLDREP (WORLD Relationship and Event Prediction), a novel dataset designed to address these limitations by leveraging the advanced reasoning capabilities of large-language models (LLMs). Our dataset features high-quality scoring labels generated through advanced prompt modeling and rigorously validated by domain experts in political science. We showcase the quality and utility of WORLDREP for real-world event prediction tasks, demonstrating its effectiveness through extensive experiments and analysis. Furthermore, we publicly release our dataset along with the full automation source code for data collection, labeling, and benchmarking, aiming to support and advance research in text-based event prediction.
• 摘要：根据新闻文章等文本信息预测未来国际事件在全球政策、战略决策和地缘政治中具有巨大的应用潜力。然而，可用于此任务的现有数据集通常质量有限，阻碍了相关研究的进展。在本文中，我们介绍了 WORLDREP（世界关系和事件预测），这是一个新颖的数据集，旨在通过利用大型语言模型 (LLM) 的高级推理能力来解决这些限制。我们的数据集具有通过高级提示建模生成的高质量评分标签，并经过政治科学领域专家的严格验证。我们展示了 WORLDREP 在现实世界事件预测任务中的质量和实用性，并通过大量实验和分析证明了其有效性。此外，我们公开发布了我们的数据集以及用于数据收集、标记和基准测试的完整自动化源代码，旨在支持和推进基于文本的事件预测研究。

Title: FunctionChat-Bench: Comprehensive Evaluation of Language Models' Generative Capabilities in Korean Tool-use Dialogs

• Authors: Shinbok Lee, Gaeun Seo, Daniel Lee, Byeongil Ko, Sunghee Jung, Myeongcheol Shin
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.14054
• Pdf URL: https://arxiv.org/pdf/2411.14054
• Copy Paste: [[2411.14054]] FunctionChat-Bench: Comprehensive Evaluation of Language Models' Generative Capabilities in Korean Tool-use Dialogs(https://arxiv.org/abs/2411.14054)
• Keywords: language model, chat
• Abstract: This study investigates language models' generative capabilities in tool-use dialogs. We categorize the models' outputs in tool-use dialogs into four distinct types: Tool Call, Answer Completion, Slot Question, and Relevance Detection, which serve as aspects for evaluation. We introduce FunctionChat-Bench, comprising 700 evaluation items and automated assessment programs. Using this benchmark, we evaluate several language models that support function calling. Our findings indicate that while language models may exhibit high accuracy in single-turn Tool Call scenarios, this does not necessarily translate to superior generative performance in multi-turn environments. We argue that the capabilities required for function calling extend beyond generating tool call messages; they must also effectively generate conversational messages that engage the user.
• 摘要：本研究调查了语言模型在工具使用对话中的生成能力。我们将工具使用对话中的模型输出分为四种不同的类型：工具调用、答案完成、槽位问题和相关性检测，这些类型作为评估的方面。我们引入了 FunctionChat-Bench，它包含 700 个评估项目和自动评估程序。使用此基准，我们评估了几种支持函数调用的语言模型。我们的研究结果表明，虽然语言模型在单轮工具调用场景中可能表现出高精度，但这并不一定意味着在多轮环境中具有出色的生成性能。我们认为，函数调用所需的能力不仅限于生成工具调用消息；它们还必须有效地生成吸引用户的对话消息。

Title: DRPruning: Efficient Large Language Model Pruning through Distributionally Robust Optimization

• Authors: Hexuan Deng, Wenxiang Jiao, Xuebo Liu, Min Zhang, Zhaopeng Tu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.14055
• Pdf URL: https://arxiv.org/pdf/2411.14055
• Copy Paste: [[2411.14055]] DRPruning: Efficient Large Language Model Pruning through Distributionally Robust Optimization(https://arxiv.org/abs/2411.14055)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) deliver impressive results but face challenges from increasing model sizes and computational costs. Structured pruning reduces model size and speeds up inference but often causes uneven degradation across domains, leading to biased performance. To address this, we propose DRPruning, which incorporates distributionally robust optimization to restore balanced performance across domains, along with further improvements to enhance robustness. Experiments in monolingual and multilingual settings show that our method surpasses similarly sized models in pruning and continued pretraining over perplexity, downstream tasks, and instruction tuning. We further provide analysis demonstrating the robustness of our method towards various domains and distribution shifts. Furthermore, our method automatically determines optimal reference losses and data ratios, suggesting potential for broader applications. Our code is available at this https URL.
• 摘要：大型语言模型 (LLM) 可提供令人印象深刻的结果，但面临着模型大小和计算成本增加的挑战。结构化修剪可减小模型大小并加快推理速度，但通常会导致跨域不均匀的退化，从而导致性能偏差。为了解决这个问题，我们提出了 DRPruning，它结合了分布式稳健优化来恢复跨域的平衡性能，并进行了进一步的改进以增强稳健性。在单语和多语环境中的实验表明，我们的方法在修剪和持续预训练方面优于类似大小的模型，包括困惑度、下游任务和指令调整。我们进一步提供了分析，证明了我们的方法对各种领域和分布变化的稳健性。此外，我们的方法会自动确定最佳参考损失和数据比率，这表明它具有更广泛的应用潜力。我们的代码可在此 https URL 上找到。

Title: Lost in Inference: Rediscovering the Role of Natural Language Inference for Large Language Models

• Authors: Lovish Madaan, David Esiobu, Pontus Stenetorp, Barbara Plank, Dieuwke Hupkes
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.14103
• Pdf URL: https://arxiv.org/pdf/2411.14103
• Copy Paste: [[2411.14103]] Lost in Inference: Rediscovering the Role of Natural Language Inference for Large Language Models(https://arxiv.org/abs/2411.14103)
• Keywords: language model, llm
• Abstract: In the recent past, a popular way of evaluating natural language understanding (NLU), was to consider a model's ability to perform natural language inference (NLI) tasks. In this paper, we investigate if NLI tasks, that are rarely used for LLM evaluation, can still be informative for evaluating LLMs. Focusing on five different NLI benchmarks across six models of different scales, we investigate if they are able to discriminate models of different size and quality and how their accuracies develop during training. Furthermore, we investigate the extent to which the softmax distributions of models align with human distributions in cases where statements are ambiguous or vague. Overall, our results paint a positive picture for the NLI tasks: we find that they are able to discriminate well between models at various stages of training, yet are not (all) saturated. Furthermore, we find that while the similarity of model distributions with human label distributions increases with scale, it is still much higher than the similarity between two populations of humans, making it a potentially interesting statistic to consider.
• 摘要：近期，评估自然语言理解 (NLU) 的一种流行方法是考虑模型执行自然语言推理 (NLI) 任务的能力。在本文中，我们研究很少用于 LLM 评估的 NLI 任务是否仍可用于评估 LLM。我们关注六种不同规模模型的五种不同的 NLI 基准，研究它们是否能够区分不同大小和质量的模型，以及它们的准确性在训练过程中如何发展。此外，我们研究了在陈述含糊不清或模糊的情况下，模型的 softmax 分布与人类分布的一致程度。总体而言，我们的结果为 NLI 任务描绘了一幅积极的图景：我们发现它们能够很好地区分处于不同训练阶段的模型，但并非（全部）饱和。此外，我们发现，虽然模型分布与人类标签分布的相似性随着规模的增加而增加，但它仍然远高于两个人类群体之间的相似性，这使得它成为一个值得考虑的潜在有趣统计数据。

Title: Learning from "Silly" Questions Improves Large Language Models, But Only Slightly

• Authors: Tingyuan Zhu, Shudong Liu, Yidong Wang, Derek F. Wong, Han Yu, Takahiro Shinozaki, Jindong Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.14121
• Pdf URL: https://arxiv.org/pdf/2411.14121
• Copy Paste: [[2411.14121]] Learning from "Silly" Questions Improves Large Language Models, But Only Slightly(https://arxiv.org/abs/2411.14121)
• Keywords: language model, gpt, llm
• Abstract: Constructing high-quality Supervised Fine-Tuning (SFT) datasets is critical for the training of large language models (LLMs). Recent studies have shown that using data from a specific source, Ruozhiba, a Chinese website where users ask "silly" questions to better understand certain topics, can lead to better fine-tuning performance. This paper aims to explore some hidden factors: the potential interpretations of its success and a large-scale evaluation of the performance. First, we leverage GPT-4 to analyze the successful cases of Ruozhiba questions from the perspective of education, psychology, and cognitive science, deriving a set of explanatory rules. Then, we construct fine-tuning datasets by applying these rules to the MMLU training set. Surprisingly, our results indicate that rules can significantly improve model performance in certain tasks, while potentially diminishing performance on others. For example, SFT data generated following the "Counterintuitive Thinking" rule can achieve approximately a 5% improvement on the "Global Facts" task, whereas the "Blurring the Conceptual Boundaries" rule leads to a performance drop of 6.14% on the "Econometrics" task. In addition, for specific tasks, different rules tend to have a consistent impact on model performance. This suggests that the differences between the extracted rules are not as significant, and the effectiveness of the rules is relatively consistent across tasks. Our research highlights the importance of considering task diversity and rule applicability when constructing SFT datasets to achieve more comprehensive performance improvements.
• 摘要：构建高质量的监督微调 (SFT) 数据集对于大型语言模型 (LLM) 的训练至关重要。最近的研究表明，使用特定来源（若之吧，一个中文网站，用户会问一些“愚蠢”的问题来更好地理解某些主题）的数据可以提高微调性能。本文旨在探索一些隐藏的因素：其成功的潜在解释和对性能的大规模评估。首先，我们利用 GPT-4 从教育、心理学和认知科学的角度分析若之吧问题的成功案例，得出一组解释规则。然后，我们通过将这些规则应用于 MMLU 训练集来构建微调数据集。令人惊讶的是，我们的结果表明，规则可以显著提高模型在某些任务上的性能，同时可能会降低其他任务上的性能。例如，按照“反直觉思维”规则生成的SFT数据在“全局事实”任务上可以获得约5%的提升，而“模糊概念边界”规则在“计量经济学”任务上导致性能下降6.14%。此外，对于特定任务，不同的规则对模型性能的影响趋于一致。这表明提取的规则之间的差异并不那么显著，规则的有效性在各个任务之间相对一致。我们的研究强调了在构建SFT数据集时考虑任务多样性和规则适用性的重要性，以实现更全面的性能提升。

Title: Why do language models perform worse for morphologically complex languages?

• Authors: Catherine Arnett, Benjamin K. Bergen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.14198
• Pdf URL: https://arxiv.org/pdf/2411.14198
• Copy Paste: [[2411.14198]] Why do language models perform worse for morphologically complex languages?(https://arxiv.org/abs/2411.14198)
• Keywords: language model
• Abstract: Language models perform differently across languages. It has been previously suggested that morphological typology may explain some of this variability (Cotterell et al., 2018). We replicate previous analyses and find additional new evidence for a performance gap between agglutinative and fusional languages, where fusional languages, such as English, tend to have better language modeling performance than morphologically more complex languages like Turkish. We then propose and test three possible causes for this performance gap: morphological alignment of tokenizers, tokenization quality, and disparities in dataset sizes and measurement. To test the morphological alignment hypothesis, we present MorphScore, a tokenizer evaluation metric, and supporting datasets for 22 languages. We find some evidence that tokenization quality explains the performance gap, but none for the role of morphological alignment. Instead we find that the performance gap is most reduced when training datasets are of equivalent size across language types, but only when scaled according to the so-called "byte-premium" -- the different encoding efficiencies of different languages and orthographies. These results suggest that no language is harder or easier for a language model to learn on the basis of its morphological typology. Differences in performance can be attributed to disparities in dataset size. These results bear on ongoing efforts to improve performance for low-performing and under-resourced languages.
• 摘要：语言模型在不同语言中的表现不同。之前有人提出，形态类型学可能可以解释这种变化的部分原因（Cotterell 等人，2018 年）。我们重复了之前的分析，并发现了黏着语言和融合语言之间存在性能差距的更多新证据，其中融合语言（例如英语）往往比土耳其语等形态更复杂的语言具有更好的语言建模性能。然后，我们提出并测试了导致这种性能差距的三个可能原因：标记器的形态对齐、标记质量以及数据集大小和测量的差异。为了测试形态对齐假设，我们提出了 MorphScore（一种标记器评估指标）和 22 种语言的支持数据集。我们发现一些证据表明标记质量可以解释性能差距，但没有证据表明形态对齐的作用。相反，我们发现，当训练数据集在不同语言类型中的大小相同时，性能差距会缩小得最多，但只有根据所谓的“字节溢价”进行缩放时才会缩小——不同语言和正字法的不同编码效率。这些结果表明，对于语言模型来说，没有哪种语言在形态类型学上更难或更容易学习。性能差异可以归因于数据集大小的差异。这些结果与持续努力提高低性能和资源不足的语言的性能有关。

Title: OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented LMs

• Authors: Akari Asai, Jacqueline He, Rulin Shao, Weijia Shi, Amanpreet Singh, Joseph Chee Chang, Kyle Lo, Luca Soldaini, Sergey Feldman, Mike D'arcy, David Wadden, Matt Latzke, Minyang Tian, Pan Ji, Shengyan Liu, Hao Tong, Bohao Wu, Yanyu Xiong, Luke Zettlemoyer, Graham Neubig, Dan Weld, Doug Downey, Wen-tau Yih, Pang Wei Koh, Hannaneh Hajishirzi
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.14199
• Pdf URL: https://arxiv.org/pdf/2411.14199
• Copy Paste: [[2411.14199]] OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented LMs(https://arxiv.org/abs/2411.14199)
• Keywords: language model, gpt
• Abstract: Scientific progress depends on researchers' ability to synthesize the growing body of literature. Can large language models (LMs) assist scientists in this task? We introduce OpenScholar, a specialized retrieval-augmented LM that answers scientific queries by identifying relevant passages from 45 million open-access papers and synthesizing citation-backed responses. To evaluate OpenScholar, we develop ScholarQABench, the first large-scale multi-domain benchmark for literature search, comprising 2,967 expert-written queries and 208 long-form answers across computer science, physics, neuroscience, and biomedicine. On ScholarQABench, OpenScholar-8B outperforms GPT-4o by 5% and PaperQA2 by 7% in correctness, despite being a smaller, open model. While GPT4o hallucinates citations 78 to 90% of the time, OpenScholar achieves citation accuracy on par with human experts. OpenScholar's datastore, retriever, and self-feedback inference loop also improves off-the-shelf LMs: for instance, OpenScholar-GPT4o improves GPT-4o's correctness by 12%. In human evaluations, experts preferred OpenScholar-8B and OpenScholar-GPT4o responses over expert-written ones 51% and 70% of the time, respectively, compared to GPT4o's 32%. We open-source all of our code, models, datastore, data and a public demo.
• 摘要：科学进步取决于研究人员综合日益增多的文献的能力。大型语言模型 (LM) 能否帮助科学家完成这项任务？我们推出了 OpenScholar，这是一种专门的检索增强型 LM，它通过从 4500 万篇开放获取论文中识别相关段落并综合引用支持的响应来回答科学查询。为了评估 OpenScholar，我们开发了 ScholarQABench，这是第一个用于文献搜索的大规模多领域基准，包括 2,967 个专家编写的查询和 208 个计算机科学、物理学、神经科学和生物医学领域的长篇答案。在 ScholarQABench 上，尽管 OpenScholar-8B 是一个较小的开放模型，但在正确性方面比 GPT-4o 高出 5%，比 PaperQA2 高出 7%。虽然 GPT4o 有 78% 到 90% 的时间会产生幻觉引用，但 OpenScholar 的引用准确率与人类专家相当。 OpenScholar 的数据存储、检索器和自反馈推理循环也改进了现成的 LM：例如，OpenScholar-GPT4o 将 GPT-4o 的正确率提高了 12%。在人工评估中，专家分别有 51% 和 70% 的时间更喜欢 OpenScholar-8B 和 OpenScholar-GPT4o 的答案，而不是专家撰写的答案，而 GPT4o 的比例为 32%。我们开源了所有代码、模型、数据存储、数据和公开演示。

Title: Evaluating the Robustness of Analogical Reasoning in Large Language Models

• Authors: Martha Lewis, Melanie Mitchell
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.14215
• Pdf URL: https://arxiv.org/pdf/2411.14215
• Copy Paste: [[2411.14215]] Evaluating the Robustness of Analogical Reasoning in Large Language Models(https://arxiv.org/abs/2411.14215)
• Keywords: language model, gpt, llm
• Abstract: LLMs have performed well on several reasoning benchmarks, including ones that test analogical reasoning abilities. However, there is debate on the extent to which they are performing general abstract reasoning versus employing non-robust processes, e.g., that overly rely on similarity to pre-training data. Here we investigate the robustness of analogy-making abilities previously claimed for LLMs on three of four domains studied by Webb, Holyoak, and Lu (2023): letter-string analogies, digit matrices, and story analogies. For each domain we test humans and GPT models on robustness to variants of the original analogy problems that test the same abstract reasoning abilities but are likely dissimilar from tasks in the pre-training data. The performance of a system that uses robust abstract reasoning should not decline substantially on these variants. On simple letter-string analogies, we find that while the performance of humans remains high for two types of variants we tested, the GPT models' performance declines sharply. This pattern is less pronounced as the complexity of these problems is increased, as both humans and GPT models perform poorly on both the original and variant problems requiring more complex analogies. On digit-matrix problems, we find a similar pattern but only on one out of the two types of variants we tested. On story-based analogy problems, we find that, unlike humans, the performance of GPT models are susceptible to answer-order effects, and that GPT models also may be more sensitive than humans to paraphrasing. This work provides evidence that LLMs often lack the robustness of zero-shot human analogy-making, exhibiting brittleness on most of the variations we tested. More generally, this work points to the importance of carefully evaluating AI systems not only for accuracy but also robustness when testing their cognitive capabilities.
• 摘要：LLM 在多个推理基准上表现良好，包括测试类比推理能力的基准。然而，关于它们在多大程度上执行一般抽象推理与采用非稳健过程（例如过度依赖与预训练数据的相似性）存在争议。在这里，我们研究了 LLM 在 Webb、Holyoak 和 Lu (2023) 研究的四个领域中的三个领域中先前声称的类比能力的稳健性：字母字符串类比、数字矩阵和故事类比。对于每个领域，我们测试人类和 GPT 模型对原始类比问题变体的稳健性，这些变体测试相同的抽象推理能力，但可能与预训练数据中的任务不同。使用稳健抽象推理的系统的性能在这些变体上不应大幅下降。在简单的字母字符串类比中，我们发现虽然人类在我们测试的两种变体中的表现仍然很高，但 GPT 模型的性能急剧下降。随着这些问题的复杂性增加，这种模式变得不那么明显，因为人类和 GPT 模型在需要更复杂类比的原始问题和变体问题上的表现都很差。在数字矩阵问题上，我们发现了类似的模式，但仅在我们测试的两种变体中的一种上。在基于故事的类比问题上，我们发现与人类不同，GPT 模型的性能容易受到答案顺序效应的影响，并且 GPT 模型对释义的敏感度也可能高于人类。这项研究提供的证据表明，LLM 通常缺乏零样本人类类比的稳健性，在我们测试的大多数变体中都表现出脆弱性。更一般地说，这项研究指出了在测试人工智能系统的认知能力时，不仅要仔细评估其准确性，还要评估其稳健性。

Title: Intent-Aware Dialogue Generation and Multi-Task Contrastive Learning for Multi-Turn Intent Classification

• Authors: Junhua Liu, Yong Keat Tan, Bin Fu, Kwan Hui Lim
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.14252
• Pdf URL: https://arxiv.org/pdf/2411.14252
• Copy Paste: [[2411.14252]] Intent-Aware Dialogue Generation and Multi-Task Contrastive Learning for Multi-Turn Intent Classification(https://arxiv.org/abs/2411.14252)
• Keywords: language model, llm, chat
• Abstract: Generating large-scale, domain-specific, multilingual multi-turn dialogue datasets remains a significant hurdle for training effective Multi-Turn Intent Classification models in chatbot systems. In this paper, we introduce Chain-of-Intent, a novel mechanism that combines Hidden Markov Models with Large Language Models (LLMs) to generate contextually aware, intent-driven conversations through self-play. By extracting domain-specific knowledge from e-commerce chat logs, we estimate conversation turns and intent transitions, which guide the generation of coherent dialogues. Leveraging LLMs to enhance emission probabilities, our approach produces natural and contextually consistent questions and answers. We also propose MINT-CL, a framework for multi-turn intent classification using multi-task contrastive learning, improving classification accuracy without the need for extensive annotated data. Evaluations show that our methods outperform baselines in dialogue quality and intent classification accuracy, especially in multilingual settings, while significantly reducing data generation efforts. Furthermore, we release MINT-E, a multilingual, intent-aware multi-turn e-commerce dialogue corpus to support future research in this area.
• 摘要：生成大规模、领域特定、多语言多轮对话数据集仍然是在聊天机器人系统中训练有效的多轮意图分类模型的重大障碍。在本文中，我们引入了意图链，这是一种将隐马尔可夫模型与大型语言模型 (LLM) 相结合的新机制，通过自我对弈生成具有上下文感知、意图驱动的对话。通过从电子商务聊天记录中提取领域特定知识，我们可以估计对话轮次和意图转换，从而指导生成连贯的对话。利用 LLM 来提高发射概率，我们的方法可以产生自然且上下文一致的问题和答案。我们还提出了 MINT-CL，这是一个使用多任务对比学习进行多轮意图分类的框架，无需大量注释数据即可提高分类准确性。评估表明，我们的方法在对话质量和意图分类准确性方面优于基线，尤其是在多语言环境中，同时显著减少了数据生成工作量。此外，我们发布了 MINT-E，这是一种多语言、意图感知的多轮电子商务对话语料库，以支持该领域的未来研究。

Title: Do I Know This Entity? Knowledge Awareness and Hallucinations in Language Models

• Authors: Javier Ferrando, Oscar Obeso, Senthooran Rajamanoharan, Neel Nanda
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.14257
• Pdf URL: https://arxiv.org/pdf/2411.14257
• Copy Paste: [[2411.14257]] Do I Know This Entity? Knowledge Awareness and Hallucinations in Language Models(https://arxiv.org/abs/2411.14257)
• Keywords: language model, hallucination, chat
• Abstract: Hallucinations in large language models are a widespread problem, yet the mechanisms behind whether models will hallucinate are poorly understood, limiting our ability to solve this problem. Using sparse autoencoders as an interpretability tool, we discover that a key part of these mechanisms is entity recognition, where the model detects if an entity is one it can recall facts about. Sparse autoencoders uncover meaningful directions in the representation space, these detect whether the model recognizes an entity, e.g. detecting it doesn't know about an athlete or a movie. This suggests that models can have self-knowledge: internal representations about their own capabilities. These directions are causally relevant: capable of steering the model to refuse to answer questions about known entities, or to hallucinate attributes of unknown entities when it would otherwise refuse. We demonstrate that despite the sparse autoencoders being trained on the base model, these directions have a causal effect on the chat model's refusal behavior, suggesting that chat finetuning has repurposed this existing mechanism. Furthermore, we provide an initial exploration into the mechanistic role of these directions in the model, finding that they disrupt the attention of downstream heads that typically move entity attributes to the final token.
• 摘要：大型语言模型中的幻觉是一个普遍存在的问题，但模型是否会产生幻觉的机制尚不清楚，这限制了我们解决这个问题的能力。使用稀疏自动编码器作为可解释性工具，我们发现这些机制的一个关键部分是实体识别，模型会检测实体是否是它可以回忆起事实的实体。稀疏自动编码器在表示空间中发现了有意义的方向，这些方向可以检测模型是否识别出实体，例如检测它不知道运动员或电影。这表明模型可以拥有自我知识：关于自身能力的内部表示。这些方向具有因果相关性：能够引导模型拒绝回答有关已知实体的问题，或者在模型拒绝回答时产生未知实体的属性的幻觉。我们证明，尽管稀疏自动编码器是在基础模型上训练的，但这些方向对聊天模型的拒绝行为有因果影响，这表明聊天微调已经重新利用了这种现有机制。此外，我们对这些方向在模型中的机制作用进行了初步探索，发现它们会扰乱通常将实体属性移动到最终标记的下游头部的注意力。

Title: Knowledge Graphs, Large Language Models, and Hallucinations: An NLP Perspective

• Authors: Ernests Lavrinovics, Russa Biswas, Johannes Bjerva, Katja Hose
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.14258
• Pdf URL: https://arxiv.org/pdf/2411.14258
• Copy Paste: [[2411.14258]] Knowledge Graphs, Large Language Models, and Hallucinations: An NLP Perspective(https://arxiv.org/abs/2411.14258)
• Keywords: language model, llm, hallucination, chat
• Abstract: Large Language Models (LLMs) have revolutionized Natural Language Processing (NLP) based applications including automated text generation, question answering, chatbots, and others. However, they face a significant challenge: hallucinations, where models produce plausible-sounding but factually incorrect responses. This undermines trust and limits the applicability of LLMs in different domains. Knowledge Graphs (KGs), on the other hand, provide a structured collection of interconnected facts represented as entities (nodes) and their relationships (edges). In recent research, KGs have been leveraged to provide context that can fill gaps in an LLM understanding of certain topics offering a promising approach to mitigate hallucinations in LLMs, enhancing their reliability and accuracy while benefiting from their wide applicability. Nonetheless, it is still a very active area of research with various unresolved open problems. In this paper, we discuss these open challenges covering state-of-the-art datasets and benchmarks as well as methods for knowledge integration and evaluating hallucinations. In our discussion, we consider the current use of KGs in LLM systems and identify future directions within each of these challenges.
• 摘要：大型语言模型 (LLM) 彻底改变了基于自然语言处理 (NLP) 的应用程序，包括自动文本生成、问答、聊天机器人等。然而，它们面临着一个重大挑战：幻觉，即模型产生的反应听起来似乎合理，但事实上是不正确的。这破坏了信任并限制了 LLM 在不同领域的适用性。另一方面，知识图谱 (KG) 提供了一个结构化的相互关联事实集合，这些事实表示为实体（节点）及其关系（边）。在最近的研究中，KG 已被用于提供背景信息，可以填补 LLM 对某些主题的理解空白，从而提供了一种有希望的方法来缓解 LLM 中的幻觉，提高其可靠性和准确性，同时受益于其广泛的适用性。尽管如此，它仍然是一个非常活跃的研究领域，存在各种未解决的开放问题。在本文中，我们讨论了这些开放挑战，涵盖了最先进的数据集和基准以及知识整合和评估幻觉的方法。在我们的讨论中，我们考虑了 LLM 系统中 KG 的当前使用情况，并确定了每个挑战中的未来方向。

Title: Efficient Aspect-Based Summarization of Climate Change Reports with Small Language Models

• Authors: Iacopo Ghinassi, Leonardo Catalano, Tommaso Colella
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.14272
• Pdf URL: https://arxiv.org/pdf/2411.14272
• Copy Paste: [[2411.14272]] Efficient Aspect-Based Summarization of Climate Change Reports with Small Language Models(https://arxiv.org/abs/2411.14272)
• Keywords: language model, llm, retrieval augmented generation
• Abstract: The use of Natural Language Processing (NLP) for helping decision-makers with Climate Change action has recently been highlighted as a use case aligning with a broader drive towards NLP technologies for social good. In this context, Aspect-Based Summarization (ABS) systems that extract and summarize relevant information are particularly useful as they provide stakeholders with a convenient way of finding relevant information in expert-curated reports. In this work, we release a new dataset for ABS of Climate Change reports and we employ different Large Language Models (LLMs) and so-called Small Language Models (SLMs) to tackle this problem in an unsupervised way. Considering the problem at hand, we also show how SLMs are not significantly worse for the problem while leading to reduced carbon footprint; we do so by applying for the first time an existing framework considering both energy efficiency and task performance to the evaluation of zero-shot generative models for ABS. Overall, our results show that modern language models, both big and small, can effectively tackle ABS for Climate Change reports but more research is needed when we frame the problem as a Retrieval Augmented Generation (RAG) problem and our work and dataset will help foster efforts in this direction.
• 摘要：最近，使用自然语言处理 (NLP) 帮助决策者采取气候变化行动的用例被强调为与更广泛地推动 NLP 技术造福社会的用例。在这种情况下，提取和总结相关信息的基于方面的摘要 (ABS) 系统特别有用，因为它们为利益相关者提供了一种在专家策划的报告中查找相关信息的便捷方式。在这项工作中，我们发布了一个新的气候变化报告 ABS 数据集，并使用不同的大型语言模型 (LLM) 和所谓的小型语言模型 (SLM) 以无监督的方式解决这个问题。考虑到手头的问题，我们还展示了 SLM 如何在减少碳足迹的同时不会显著恶化问题；我们首次将同时考虑能源效率和任务性能的现有框架应用于对 ABS 的零样本生成模型的评估。总体而言，我们的结果表明，现代语言模型（无论大小）都可以有效解决气候变化报告的 ABS，但当我们将问题定义为检索增强生成 (RAG) 问题时，还需要进行更多研究，而我们的工作和数据集将有助于促进这方面的努力。

Title: Velocitune: A Velocity-based Dynamic Domain Reweighting Method for Continual Pre-training

• Authors: Zheheng Luo, Xin Zhang, Xiao Liu, Haoling Li, Yeyun Gong, Chen Qi, Peng Cheng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.14318
• Pdf URL: https://arxiv.org/pdf/2411.14318
• Copy Paste: [[2411.14318]] Velocitune: A Velocity-based Dynamic Domain Reweighting Method for Continual Pre-training(https://arxiv.org/abs/2411.14318)
• Keywords: language model
• Abstract: It is well-known that a diverse corpus is critical for training large language models, which are typically constructed from a mixture of various domains. In general, previous efforts resort to sampling training data from different domains with static proportions, as well as adjusting data proportions during training. However, few methods have addressed the complexities of domain-adaptive continual pre-training. To fill this gap, we propose Velocitune, a novel framework dynamically assesses learning velocity and adjusts data proportions accordingly, favoring slower-learning domains while shunning faster-learning ones, which is guided by a scaling law to indicate the desired learning goal for each domain with less associated cost. To evaluate the effectiveness of Velocitune, we conduct experiments in a reasoning-focused dataset with CodeLlama, as well as in a corpus specialised for system command generation with Llama3 and Mistral. Velocitune achieves performance gains in both math and code reasoning tasks and command-line generation benchmarks. Further analysis reveals that key factors driving Velocitune's effectiveness include target loss prediction and data ordering.
• 摘要：众所周知，多样化的语料库对于训练大型语言模型至关重要，而大型语言模型通常由各种领域的混合构建而成。一般来说，以前的努力都依赖于以静态比例从不同领域采样训练数据，以及在训练期间调整数据比例。然而，很少有方法能够解决领域自适应持续预训练的复杂性。为了填补这一空白，我们提出了 Velocitune，这是一个新颖的框架，可以动态评估学习速度并相应地调整数据比例，有利于学习速度较慢的领域，而避开学习速度较快的领域，它由缩放定律指导，以较低的相关成本指示每个领域的期望学习目标。为了评估 Velocitune 的有效性，我们使用 CodeLlama 在以推理为重点的数据集中进行了实验，并使用 Llama3 和 Mistral 在专门用于系统命令生成的语料库中进行了实验。Velocitune 在数学和代码推理任务以及命令行生成基准测试中都实现了性能提升。进一步分析表明，推动 Velocitune 有效性的关键因素包括目标损失预测和数据排序。

Title: UnifiedCrawl: Aggregated Common Crawl for Affordable Adaptation of LLMs on Low-Resource Languages

• Authors: Bethel Melesse Tessema (1), Akhil Kedia (2), Tae-Sun Chung (1) ((1) Ajou University, (2) Independent Researcher)
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.14343
• Pdf URL: https://arxiv.org/pdf/2411.14343
• Copy Paste: [[2411.14343]] UnifiedCrawl: Aggregated Common Crawl for Affordable Adaptation of LLMs on Low-Resource Languages(https://arxiv.org/abs/2411.14343)
• Keywords: language model, llm, prompt
• Abstract: Large language models (LLMs) under-perform on low-resource languages due to limited training data. We present a method to efficiently collect text data for low-resource languages from the entire Common Crawl corpus. Our approach, UnifiedCrawl, filters and extracts common crawl using minimal compute resources, yielding mono-lingual datasets much larger than previously available sources. We demonstrate that leveraging this data to fine-tuning multilingual LLMs via efficient adapter methods (QLoRA) significantly boosts performance on the low-resource language, while minimizing VRAM usage. Our experiments show large improvements in language modeling perplexity and an increase in few-shot prompting scores. Our work and released source code provide an affordable approach to improve LLMs for low-resource languages using consumer hardware. Our source code is available here at this https URL.
• 摘要：由于训练数据有限，大型语言模型 (LLM) 在低资源语言上表现不佳。我们提出了一种从整个 Common Crawl 语料库中高效收集低资源语言文本数据的方法。我们的方法 UnifiedCrawl 使用最少的计算资源过滤和提取 Common Crawl，从而产生比以前可用的源大得多的单语数据集。我们证明，利用这些数据通过高效的适配器方法 (QLoRA) 对多语言 LLM 进行微调可显著提高低资源语言的性能，同时最大限度地减少 VRAM 使用量。我们的实验表明，语言建模困惑度得到了很大的改善，并且小样本提示分数有所提高。我们的工作和发布的源代码提供了一种经济实惠的方法，可以使用消费级硬件改进低资源语言的 LLM。我们的源代码可在此 https URL 上找到。

Title: Lightweight Safety Guardrails Using Fine-tuned BERT Embeddings

• Authors: Aaron Zheng, Mansi Rana, Andreas Stolcke
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.14398
• Pdf URL: https://arxiv.org/pdf/2411.14398
• Copy Paste: [[2411.14398]] Lightweight Safety Guardrails Using Fine-tuned BERT Embeddings(https://arxiv.org/abs/2411.14398)
• Keywords: language model, llm, prompt
• Abstract: With the recent proliferation of large language models (LLMs), enterprises have been able to rapidly develop proof-of-concepts and prototypes. As a result, there is a growing need to implement robust guardrails that monitor, quantize and control an LLM's behavior, ensuring that the use is reliable, safe, accurate and also aligned with the users' expectations. Previous approaches for filtering out inappropriate user prompts or system outputs, such as LlamaGuard and OpenAI's MOD API, have achieved significant success by fine-tuning existing LLMs. However, using fine-tuned LLMs as guardrails introduces increased latency and higher maintenance costs, which may not be practical or scalable for cost-efficient deployments. We take a different approach, focusing on fine-tuning a lightweight architecture: Sentence-BERT. This method reduces the model size from LlamaGuard's 7 billion parameters to approximately 67 million, while maintaining comparable performance on the AEGIS safety benchmark.
• 摘要：随着大型语言模型 (LLM) 的不断普及，企业已经能够快速开发概念验证和原型。因此，越来越需要实施强大的护栏来监控、量化和控制 LLM 的行为，确保使用可靠、安全、准确并符合用户的期望。以前用于过滤不适当用户提示或系统输出的方法，例如 LlamaGuard 和 OpenAI 的 MOD API，通过微调现有 LLM 取得了重大成功。但是，使用微调后的 LLM 作为护栏会增加延迟和维护成本，这对于具有成本效益的部署来说可能不切实际或无法扩展。我们采用了不同的方法，专注于微调轻量级架构：Sentence-BERT。这种方法将模型大小从 LlamaGuard 的 70 亿个参数减少到大约 6700 万个，同时在 AEGIS 安全基准上保持了可比的性能。

Title: Marco-o1: Towards Open Reasoning Models for Open-Ended Solutions

• Authors: Yu Zhao, Huifeng Yin, Bo Zeng, Hao Wang, Tianqi Shi, Chenyang Lyu, Longyue Wang, Weihua Luo, Kaifu Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.14405
• Pdf URL: https://arxiv.org/pdf/2411.14405
• Copy Paste: [[2411.14405]] Marco-o1: Towards Open Reasoning Models for Open-Ended Solutions(https://arxiv.org/abs/2411.14405)
• Keywords: chain-of-thought
• Abstract: Currently OpenAI o1 has sparked a surge of interest in the study of large reasoning models (LRM). Building on this momentum, Marco-o1 not only focuses on disciplines with standard answers, such as mathematics, physics, and coding -- which are well-suited for reinforcement learning (RL) -- but also places greater emphasis on open-ended resolutions. We aim to address the question: "Can the o1 model effectively generalize to broader domains where clear standards are absent and rewards are challenging to quantify?" Marco-o1 is powered by Chain-of-Thought (CoT) fine-tuning, Monte Carlo Tree Search (MCTS), reflection mechanisms, and innovative reasoning strategies -- optimized for complex real-world problem-solving tasks.
• 摘要：目前，OpenAI o1 已引发了人们对大型推理模型 (LRM) 研究的极大兴趣。借助这一势头，Marco-o1 不仅专注于具有标准答案的学科，例如数学、物理和编码（这些学科非常适合强化学习 (RL)），而且还更加重视开放式解决方案。我们旨在回答以下问题：“o1 模型能否有效地推广到更广泛的领域，这些领域缺乏明确的标准，并且奖励难以量化？” Marco-o1 由思想链 (CoT) 微调、蒙特卡洛树搜索 (MCTS)、反射机制和创新推理策略提供支持，针对复杂的现实世界问题解决任务进行了优化。