2025-01-06

Title: (WhyPHI) Fine-Tuning PHI-3 for Multiple-Choice Question Answering: Methodology, Results, and Challenges

• Authors: Mohamed Hisham Abdellatif
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.01588
• Pdf URL: https://arxiv.org/pdf/2501.01588
• Copy Paste: [[2501.01588]] (WhyPHI) Fine-Tuning PHI-3 for Multiple-Choice Question Answering: Methodology, Results, and Challenges(https://arxiv.org/abs/2501.01588)
• Keywords: language model, llm, hallucination, prompt
• Abstract: Large Language Models (LLMs) have become essential tools across various domains due to their impressive capabilities in understanding and generating human-like text. The ability to accurately answer multiple-choice questions (MCQs) holds significant value in education, particularly in automated tutoring systems and assessment platforms. However, adapting LLMs to handle MCQ tasks effectively remains challenging due to the hallucinations and unclear prompts. This work explores the potential of Microsoft's PHI-3\cite{Abdin2024}, a compact yet efficient LLM, for MCQ answering. Our contributions include fine-tuning the model on the TruthfulQA dataset, designing optimized prompts to enhance model performance, and evaluating using perplexity and traditional metrics like accuracy and F1 score. Results show a remarkable improvement in PHI-3.5's MCQ handling post-fine-tuning, with perplexity decreasing from 4.68 to 2.27, and accuracy rising from 62\% to 90.8\%. This research underlines the importance of efficient models in adaptive learning systems and educational assessments, paving the way for broader integration into the classroom, particularly in fields like test preparation, student feedback, and personalized learning.
• 摘要：大型语言模型 (LLM) 因其在理解和生成类似人类的文本方面具有令人印象深刻的能力，已成为各个领域必不可少的工具。准确回答多项选择题 (MCQ) 的能力在教育中具有重要价值，尤其是在自动辅导系统和评估平台中。然而，由于幻觉和不明确的提示，调整 LLM 以有效处理 MCQ 任务仍然具有挑战性。这项工作探索了 Microsoft 的 PHI-3\cite{Abdin2024}（一种紧凑而高效的 LLM）在 MCQ 回答方面的潜力。我们的贡献包括在 TruthfulQA 数据集上微调模型，设计优化的提示以提高模型性能，并使用困惑度和传统指标（如准确度和 F1 分数）进行评估。结果显示，经过微调后，PHI-3.5 的 MCQ 处理能力有了显着提高，困惑度从 4.68 降低到 2.27，准确度从 62\% 上升到 90.8\%。这项研究强调了自适应学习系统和教育评估中高效模型的重要性，为更广泛地融入课堂铺平了道路，特别是在考试准备、学生反馈和个性化学习等领域。

Title: PSYCHE: A Multi-faceted Patient Simulation Framework for Evaluation of Psychiatric Assessment Conversational Agents

• Authors: Jingoo Lee, Kyungho Lim, Young-Chul Jung, Byung-Hoon Kim
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.01594
• Pdf URL: https://arxiv.org/pdf/2501.01594
• Copy Paste: [[2501.01594]] PSYCHE: A Multi-faceted Patient Simulation Framework for Evaluation of Psychiatric Assessment Conversational Agents(https://arxiv.org/abs/2501.01594)
• Keywords: language model, llm, agent
• Abstract: Recent advances in large language models (LLMs) have accelerated the development of conversational agents capable of generating human-like responses. Since psychiatric assessments typically involve complex conversational interactions between psychiatrists and patients, there is growing interest in developing LLM-based psychiatric assessment conversational agents (PACAs) that aim to simulate the role of psychiatrists in clinical evaluations. However, standardized methods for benchmarking the clinical appropriateness of PACAs' interaction with patients still remain underexplored. Here, we propose PSYCHE, a novel framework designed to enable the 1) clinically relevant, 2) ethically safe, 3) cost-efficient, and 4) quantitative evaluation of PACAs. This is achieved by simulating psychiatric patients based on a multi-faceted psychiatric construct that defines the simulated patients' profiles, histories, and behaviors, which PACAs are expected to assess. We validate the effectiveness of PSYCHE through a study with 10 board-certified psychiatrists, supported by an in-depth analysis of the simulated patient utterances.
• 摘要：大型语言模型 (LLM) 的最新进展加速了能够生成类似人类反应的对话代理的开发。由于精神病评估通常涉及精神科医生和患者之间复杂的对话互动，人们越来越有兴趣开发基于 LLM 的精神评估对话代理 (PACA)，旨在模拟精神科医生在临床评估中的作用。然而，用于衡量 PACA 与患者互动的临床适用性的标准化方法仍然未得到充分探索。在这里，我们提出了 PSYCHE，这是一个新颖的框架，旨在实现 1) 临床相关、2) 道德安全、3) 成本效益和 4) PACA 的定量评估。这是通过基于多方面的精神病学结构模拟精神病患者来实现的，该结构定义了模拟患者的概况、历史和行为，而 PACA 有望对其进行评估。我们通过对 10 名经过委员会认证的精神科医生进行的研究来验证 PSYCHE 的有效性，并通过对模拟患者话语的深入分析来支持这一验证。

Title: ICPC: In-context Prompt Compression with Faster Inference

• Authors: Ziyang Yu, Yuyu Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.01625
• Pdf URL: https://arxiv.org/pdf/2501.01625
• Copy Paste: [[2501.01625]] ICPC: In-context Prompt Compression with Faster Inference(https://arxiv.org/abs/2501.01625)
• Keywords: language model, llm, prompt
• Abstract: Despite the recent success of Large Language Models (LLMs), it remains challenging to feed LLMs with long prompts due to the fixed size of LLM inputs. As a remedy, prompt compression becomes a promising solution by removing redundant tokens in the prompt. However, using LLM in the existing works requires additional computation resources and leads to memory overheads. To address it, we propose ICPC (In-context Prompt Compression), a novel and scalable prompt compression method that adaptively reduces the prompt length. The key idea of ICPC is to calculate the probability of each word appearing in the prompt using encoders and calculate information carried by each word through the information function, which effectively reduces the information loss during prompt compression and increases the speed of compression. Empirically, we demonstrate that ICPC can effectively compress long texts of different categories and thus achieve better performance and speed on different types of NLP tasks.
• 摘要：尽管大型语言模型 (LLM) 近期取得了成功，但由于 LLM 输入的大小固定，向 LLM 提供长提示仍然具有挑战性。作为一种补救措施，通过删除提示中的冗余标记，提示压缩成为一种很有前途的解决方案。然而，在现有工作中使用 LLM 需要额外的计算资源并导致内存开销。为了解决这个问题，我们提出了 ICPC（上下文提示压缩），这是一种新颖且可扩展的提示压缩方法，可自适应地减少提示长度。ICPC 的关键思想是使用编码器计算每个单词出现在提示中的概率，并通过信息函数计算每个单词所携带的信息，从而有效减少提示压缩过程中的信息损失并提高压缩速度。通过实证研究，我们证明 ICPC 可以有效压缩不同类别的长文本，从而在不同类型的 NLP 任务上实现更好的性能和速度。

Title: A non-ergodic framework for understanding emergent capabilities in Large Language Models

• Authors: Javier Marin
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.01638
• Pdf URL: https://arxiv.org/pdf/2501.01638
• Copy Paste: [[2501.01638]] A non-ergodic framework for understanding emergent capabilities in Large Language Models(https://arxiv.org/abs/2501.01638)
• Keywords: language model
• Abstract: Large language models have emergent capabilities that come unexpectedly at scale, but we need a theoretical framework to explain why and how they emerge. We prove that language models are actually non-ergodic systems while providing a mathematical framework based on Stuart Kauffman's theory of the adjacent possible (TAP) to explain capability emergence. Our resource-constrained TAP equation demonstrates how architectural, training, and contextual constraints interact to shape model capabilities through phase transitions in semantic space. We prove through experiments with three different language models that capacities emerge through discrete transitions guided by constraint interactions and path-dependent exploration. This framework provides a theoretical basis for understanding emergence in language models and guides the development of architectures that can guide capability emergence.
• 摘要：大型语言模型具有在规模上出乎意料地出现的新兴能力，但我们需要一个理论框架来解释它们出现的原因和方式。我们证明语言模型实际上是非遍历系统，同时提供了一个基于 Stuart Kauffman 的相邻可能理论 (TAP) 的数学框架来解释能力的出现。我们的资源受限 TAP 方程展示了架构、训练和上下文约束如何相互作用，通过语义空间中的相变来塑造模型能力。我们通过使用三种不同的语言模型进行实验，证明能力是通过约束交互和路径相关探索引导的离散转换而出现的。该框架为理解语言模型的出现提供了理论基础，并指导了可以引导能力出现的架构的开发。

Title: Multimodal Contrastive Representation Learning in Augmented Biomedical Knowledge Graphs

• Authors: Tien Dang, Viet Thanh Duy Nguyen, Minh Tuan Le, Truong-Son Hy
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.01644
• Pdf URL: https://arxiv.org/pdf/2501.01644
• Copy Paste: [[2501.01644]] Multimodal Contrastive Representation Learning in Augmented Biomedical Knowledge Graphs(https://arxiv.org/abs/2501.01644)
• Keywords: language model
• Abstract: Biomedical Knowledge Graphs (BKGs) integrate diverse datasets to elucidate complex relationships within the biomedical field. Effective link prediction on these graphs can uncover valuable connections, such as potential novel drug-disease relations. We introduce a novel multimodal approach that unifies embeddings from specialized Language Models (LMs) with Graph Contrastive Learning (GCL) to enhance intra-entity relationships while employing a Knowledge Graph Embedding (KGE) model to capture inter-entity relationships for effective link prediction. To address limitations in existing BKGs, we present PrimeKG++, an enriched knowledge graph incorporating multimodal data, including biological sequences and textual descriptions for each entity type. By combining semantic and relational information in a unified representation, our approach demonstrates strong generalizability, enabling accurate link predictions even for unseen nodes. Experimental results on PrimeKG++ and the DrugBank drug-target interaction dataset demonstrate the effectiveness and robustness of our method across diverse biomedical datasets. Our source code, pre-trained models, and data are publicly available at this https URL
• 摘要：生物医学知识图谱 (BKG) 整合了各种数据集，以阐明生物医学领域内的复杂关系。对这些图谱进行有效的链接预测可以发现有价值的联系，例如潜在的新型药物-疾病关系。我们引入了一种新颖的多模态方法，将专门的语言模型 (LM) 的嵌入与图对比学习 (GCL) 统一起来，以增强实体内关系，同时采用知识图谱嵌入 (KGE) 模型来捕获实体间关系，以实现有效的链接预测。为了解决现有 BKG 的局限性，我们提出了 PrimeKG++，这是一个丰富的知识图谱，它包含多模态数据，包括每种实体类型的生物序列和文本描述。通过将语义和关系信息结合在一个统一的表示中，我们的方法表现出很强的通用性，即使对于看不见的节点也能实现准确的链接预测。在 PrimeKG++ 和 DrugBank 药物-靶标相互作用数据集上的实验结果证明了我们的方法在各种生物医学数据集中的有效性和稳健性。我们的源代码、预训练模型和数据均可通过此 https URL 公开获取

Title: MIRAGE: Exploring How Large Language Models Perform in Complex Social Interactive Environments

• Authors: Cai Yin, Gu Zhouhong, Du Zhaohan, Ye Zheyu, Cao Shaosheng, Xu Yiqian, Feng Hongwei, Chen Ping
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.01652
• Pdf URL: https://arxiv.org/pdf/2501.01652
• Copy Paste: [[2501.01652]] MIRAGE: Exploring How Large Language Models Perform in Complex Social Interactive Environments(https://arxiv.org/abs/2501.01652)
• Keywords: language model, gpt, llm
• Abstract: Large Language Models (LLMs) have shown remarkable capabilities in environmental perception, reasoning-based decision-making, and simulating complex human behaviors, particularly in interactive role-playing contexts. This paper introduces the Multiverse Interactive Role-play Ability General Evaluation (MIRAGE), a comprehensive framework designed to assess LLMs' proficiency in portraying advanced human behaviors through murder mystery games. MIRAGE features eight intricately crafted scripts encompassing diverse themes and styles, providing a rich simulation. To evaluate LLMs' performance, MIRAGE employs four distinct methods: the Trust Inclination Index (TII) to measure dynamics of trust and suspicion, the Clue Investigation Capability (CIC) to measure LLMs' capability of conducting information, the Interactivity Capability Index (ICI) to assess role-playing capabilities and the Script Compliance Index (SCI) to assess LLMs' capability of understanding and following instructions. Our experiments indicate that even popular models like GPT-4 face significant challenges in navigating the complexities presented by the MIRAGE. The datasets and simulation codes are available in \href{this https URL}{github}.
• 摘要：大型语言模型 (LLM) 在环境感知、基于推理的决策和模拟复杂人类行为方面表现出了卓越的能力，尤其是在交互式角色扮演环境中。本文介绍了多元宇宙交互式角色扮演能力综合评估 (MIRAGE)，这是一个全面的框架，旨在评估 LLM 通过谋杀悬疑游戏描绘高级人类行为的能力。MIRAGE 具有八个精心制作的脚本，涵盖不同的主题和风格，提供丰富的模拟。为了评估 LLM 的表现，MIRAGE 采用了四种不同的方法：信任倾向指数 (TII) 来衡量信任和怀疑的动态，线索调查能力 (CIC) 来衡量 LLM 传递信息的能力，交互能力指数 (ICI) 来评估角色扮演能力和脚本合规指数 (SCI) 来评估 LLM 理解和遵循指令的能力。我们的实验表明，即使是像 GPT-4 这样的流行模型在应对 MIRAGE 呈现的复杂性时也面临着重大挑战。数据集和模拟代码可在 \href{此 https URL}{github} 中找到。

Title: CoT-based Synthesizer: Enhancing LLM Performance through Answer Synthesis

• Authors: Bohan Zhang, Xiaokang Zhang, Jing Zhang, Jifan Yu, Sijia Luo, Jie Tang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.01668
• Pdf URL: https://arxiv.org/pdf/2501.01668
• Copy Paste: [[2501.01668]] CoT-based Synthesizer: Enhancing LLM Performance through Answer Synthesis(https://arxiv.org/abs/2501.01668)
• Keywords: gpt, llm
• Abstract: Current inference scaling methods, such as Self-consistency and Best-of-N, have proven effective in improving the accuracy of LLMs on complex reasoning tasks. However, these methods rely heavily on the quality of candidate responses and are unable to produce correct answers when all candidates are incorrect. In this paper, we propose a novel inference scaling strategy, CoT-based Synthesizer, which leverages CoT reasoning to synthesize superior answers by analyzing complementary information from multiple candidate responses, even when all candidate responses are flawed. To enable a lightweight and cost-effective implementation, we introduce an automated data generation pipeline that creates diverse training data. This allows smaller LLMs trained on this data to improve the inference accuracy of larger models, including API-based LLMs. Experimental results across four benchmark datasets with seven policy models demonstrate that our method significantly enhances performance, with gains of 11.8% for Llama3-8B and 10.3% for GPT-4o on the MATH dataset. The corresponding training data and code are publicly available on this https URL.
• 摘要：当前的推理扩展方法（例如自一致性和 Best-of-N）已被证明可有效提高 LLM 在复杂推理任务中的准确性。然而，这些方法严重依赖候选答案的质量，当所有候选答案都不正确时，它们无法产生正确答案。在本文中，我们提出了一种新颖的推理扩展策略，即基于 CoT 的合成器，它利用 CoT 推理通过分析来自多个候选答案的互补信息来合成优质答案，即使所有候选答案都有缺陷。为了实现轻量且经济高效的实施，我们引入了一个自动数据生成管道，可以创建多样化的训练数据。这使得在这些数据上训练的小型 LLM 能够提高大型模型（包括基于 API 的 LLM）的推理准确性。在四个基准数据集上使用七个策略模型的实验结果表明，我们的方法显著提高了性能，在 MATH 数据集上，Llama3-8B 的性能提高了 11.8%，GPT-4o 的性能提高了 10.3%。相应的训练数据和代码可在此 https URL 上公开获取。

Title: Adaptive Few-shot Prompting for Machine Translation with Pre-trained Language Models

• Authors: Lei Tang, Jinghui Qin, Wenxuan Ye, Hao Tan, Zhijing Yang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.01679
• Pdf URL: https://arxiv.org/pdf/2501.01679
• Copy Paste: [[2501.01679]] Adaptive Few-shot Prompting for Machine Translation with Pre-trained Language Models(https://arxiv.org/abs/2501.01679)
• Keywords: language model, llm, prompt
• Abstract: Recently, Large language models (LLMs) with in-context learning have demonstrated remarkable potential in handling neural machine translation. However, existing evidence shows that LLMs are prompt-sensitive and it is sub-optimal to apply the fixed prompt to any input for downstream machine translation tasks. To address this issue, we propose an adaptive few-shot prompting (AFSP) framework to automatically select suitable translation demonstrations for various source input sentences to further elicit the translation capability of an LLM for better machine translation. First, we build a translation demonstration retrieval module based on LLM's embedding to retrieve top-k semantic-similar translation demonstrations from aligned parallel translation corpus. Rather than using other embedding models for semantic demonstration retrieval, we build a hybrid demonstration retrieval module based on the embedding layer of the deployed LLM to build better input representation for retrieving more semantic-related translation demonstrations. Then, to ensure better semantic consistency between source inputs and target outputs, we force the deployed LLM itself to generate multiple output candidates in the target language with the help of translation demonstrations and rerank these candidates. Besides, to better evaluate the effectiveness of our AFSP framework on the latest language and extend the research boundary of neural machine translation, we construct a high-quality diplomatic Chinese-English parallel dataset that consists of 5,528 parallel Chinese-English sentences. Finally, extensive experiments on the proposed diplomatic Chinese-English parallel dataset and the United Nations Parallel Corpus (Chinese-English part) show the effectiveness and superiority of our proposed AFSP.
• 摘要：最近，具有上下文学习功能的大型语言模型 (LLM) 在处理神经机器翻译方面表现出了巨大的潜力。然而，现有证据表明 LLM 对提示敏感，将固定提示应用于下游机器翻译任务的任何输入都不是最优的。为了解决这个问题，我们提出了一个自适应的少样本提示 (AFSP) 框架，为各种源输入句子自动选择合适的翻译演示，以进一步激发 LLM 的翻译能力，从而实现更好的机器翻译。首先，我们基于 LLM 的嵌入构建一个翻译演示检索模块，从对齐的平行翻译语料库中检索前 k 个语义相似的翻译演示。我们没有使用其他嵌入模型进行语义演示检索，而是基于已部署的 LLM 的嵌入层构建了一个混合演示检索模块，以构建更好的输入表示，从而检索更多与语义相关的翻译演示。然后，为了确保源输入和目标输出之间更好的语义一致性，我们强制部署的 LLM 本身在翻译演示的帮助下生成多个目标语言的输出候选并重新排序这些候选。此外，为了更好地评估我们的 AFSP 框架对最新语言的有效性并扩展神经机器翻译的研究边界，我们构建了一个高质量的外交中英平行数据集，该数据集由 5,528 个平行中英句子组成。最后，在提出的外交中英平行数据集和联合国平行语料库（中英部分）上进行的大量实验证明了我们提出的 AFSP 的有效性和优越性。

Title: The Essence of Contextual Understanding in Theory of Mind: A Study on Question Answering with Story Characters

• Authors: Chulun Zhou, Qiujing Wang, Mo Yu, Xiaoqian Yue, Rui Lu, Jiangnan Li, Yifan Zhou, Shunchi Zhang, Jie Zhou, Wai Lam
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.01705
• Pdf URL: https://arxiv.org/pdf/2501.01705
• Copy Paste: [[2501.01705]] The Essence of Contextual Understanding in Theory of Mind: A Study on Question Answering with Story Characters(https://arxiv.org/abs/2501.01705)
• Keywords: llm
• Abstract: Theory-of-Mind (ToM) is a fundamental psychological capability that allows humans to understand and interpret the mental states of others. Humans infer others' thoughts by integrating causal cues and indirect clues from broad contextual information, often derived from past interactions. In other words, human ToM heavily relies on the understanding about the backgrounds and life stories of others. Unfortunately, this aspect is largely overlooked in existing benchmarks for evaluating machines' ToM capabilities, due to their usage of short narratives without global backgrounds. In this paper, we verify the importance of understanding long personal backgrounds in ToM and assess the performance of LLMs in such realistic evaluation scenarios. To achieve this, we introduce a novel benchmark, CharToM-QA, comprising 1,035 ToM questions based on characters from classic novels. Our human study reveals a significant disparity in performance: the same group of educated participants performs dramatically better when they have read the novels compared to when they have not. In parallel, our experiments on state-of-the-art LLMs, including the very recent o1 model, show that LLMs still perform notably worse than humans, despite that they have seen these stories during pre-training. This highlights the limitations of current LLMs in capturing the nuanced contextual information required for ToM reasoning.
• 摘要：心智理论 (ToM) 是一种基本的心理能力，它使人类能够理解和解释他人的心理状态。人类通过整合因果线索和来自广泛背景信息的间接线索来推断他人的想法，这些信息通常来自过去的互动。换句话说，人类的心智理论严重依赖于对他人背景和生活故事的理解。不幸的是，由于机器使用没有全局背景的短篇叙述，现有的评估机器心智理论能力的基准在很大程度上忽视了这一方面。在本文中，我们验证了理解心智理论中长期个人背景的重要性，并评估了 LLM 在这种现实评估场景中的表现。为了实现这一点，我们引入了一个新颖的基准 CharToM-QA，它包含 1,035 个基于经典小说人物的心智理论问题。我们的人类研究揭示了表现上的显著差异：同一组受过教育的参与者在阅读小说时的表现比没有阅读小说时要好得多。与此同时，我们对最先进的 LLM（包括最新的 o1 模型）进行的实验表明，尽管 LLM 在预训练期间已经看过这些故事，但它们的表现仍然明显不如人类。这凸显了当前 LLM 在捕捉 ToM 推理所需的细微背景信息方面的局限性。

Title: Automating Legal Concept Interpretation with LLMs: Retrieval, Generation, and Evaluation

• Authors: Kangcheng Luo, Quzhe Huang, Cong Jiang, Yansong Feng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.01743
• Pdf URL: https://arxiv.org/pdf/2501.01743
• Copy Paste: [[2501.01743]] Automating Legal Concept Interpretation with LLMs: Retrieval, Generation, and Evaluation(https://arxiv.org/abs/2501.01743)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: Legal articles often include vague concepts to adapt to the ever-changing society. Providing detailed interpretations of these concepts is a critical task for legal practitioners, which requires meticulous and professional annotations by legal experts, admittedly time-consuming and expensive to collect at scale. In this paper, we introduce a novel retrieval-augmented generation framework, ATRI, for AuTomatically Retrieving relevant information from past judicial precedents and Interpreting vague legal concepts. We further propose a new benchmark, Legal Concept Entailment, to automate the evaluation of generated concept interpretations without expert involvement. Automatic evaluations indicate that our generated interpretations can effectively assist large language models (LLMs) in understanding vague legal concepts. Multi-faceted evaluations by legal experts indicate that the quality of our concept interpretations is comparable to those written by human experts. Our work has strong implications for leveraging LLMs to support legal practitioners in interpreting vague legal concepts and beyond.
• 摘要：法律文章通常包含模糊的概念以适应不断变化的社会。对这些概念进行详细的解释是法律从业人员的一项关键任务，这需要法律专家进行细致而专业的注释，诚然，大规模收集这些注释非常耗时且成本高昂。在本文中，我们介绍了一种新颖的检索增强生成框架 ATRI，用于自动从过去的司法判例中检索相关信息并解释模糊的法律概念。我们进一步提出了一个新的基准“法律概念蕴涵”，以自动评估生成的概念解释，而无需专家参与。自动评估表明，我们生成的解释可以有效地帮助大型语言模型 (LLM) 理解模糊的法律概念。法律专家的多方面评估表明，我们的概念解释质量与人类专家撰写的解释相当。我们的工作对于利用 LLM 支持法律从业人员解释模糊的法律概念及其他方面具有重大意义。

Title: Reading Between the Lines: A dataset and a study on why some texts are tougher than others

• Authors: Nouran Khallaf, Carlo Eugeni, Serge Sharoff
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.01796
• Pdf URL: https://arxiv.org/pdf/2501.01796
• Copy Paste: [[2501.01796]] Reading Between the Lines: A dataset and a study on why some texts are tougher than others(https://arxiv.org/abs/2501.01796)
• Keywords: language model
• Abstract: Our research aims at better understanding what makes a text difficult to read for specific audiences with intellectual disabilities, more specifically, people who have limitations in cognitive functioning, such as reading and understanding skills, an IQ below 70, and challenges in conceptual domains. We introduce a scheme for the annotation of difficulties which is based on empirical research in psychology as well as on research in translation studies. The paper describes the annotated dataset, primarily derived from the parallel texts (standard English and Easy to Read English translations) made available online. we fine-tuned four different pre-trained transformer models to perform the task of multiclass classification to predict the strategies required for simplification. We also investigate the possibility to interpret the decisions of this language model when it is aimed at predicting the difficulty of sentences. The resources are available from this https URL
• 摘要：我们的研究旨在更好地理解是什么让有智力障碍的特定受众难以阅读文本，更具体地说，是那些在认知功能方面有限制的人，例如阅读和理解能力、智商低于 70 的人，以及在概念领域有挑战的人。我们介绍了一种基于心理学实证研究以及翻译研究的难度注释方案。本文描述了带注释的数据集，主要来自在线提供的平行文本（标准英语和易读英语翻译）。我们对四种不同的预训练转换器模型进行了微调，以执行多类分类任务，以预测简化所需的策略。我们还研究了当该语言模型旨在预测句子难度时解释其决策的可能性。资源可从此 https URL 获得

Title: Time Series Language Model for Descriptive Caption Generation

• Authors: Mohamed Trabelsi, Aidan Boyd, Jin Cao, Huseyin Uzunalioglu
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.01832
• Pdf URL: https://arxiv.org/pdf/2501.01832
• Copy Paste: [[2501.01832]] Time Series Language Model for Descriptive Caption Generation(https://arxiv.org/abs/2501.01832)
• Keywords: language model, llm, prompt
• Abstract: The automatic generation of representative natural language descriptions for observable patterns in time series data enhances interpretability, simplifies analysis and increases cross-domain utility of temporal data. While pre-trained foundation models have made considerable progress in natural language processing (NLP) and computer vision (CV), their application to time series analysis has been hindered by data scarcity. Although several large language model (LLM)-based methods have been proposed for time series forecasting, time series captioning is under-explored in the context of LLMs. In this paper, we introduce TSLM, a novel time series language model designed specifically for time series captioning. TSLM operates as an encoder-decoder model, leveraging both text prompts and time series data representations to capture subtle temporal patterns across multiple phases and generate precise textual descriptions of time series inputs. TSLM addresses the data scarcity problem in time series captioning by first leveraging an in-context prompting synthetic data generation, and second denoising the generated data via a novel cross-modal dense retrieval scoring applied to time series-caption pairs. Experimental findings on various time series captioning datasets demonstrate that TSLM outperforms existing state-of-the-art approaches from multiple data modalities by a significant margin.
• 摘要：自动生成时间序列数据中可观察模式的代表性自然语言描述可增强可解释性、简化分析并提高时间数据的跨域效用。虽然预先训练的基础模型在自然语言处理 (NLP) 和计算机视觉 (CV) 方面取得了长足的进步，但它们在时间序列分析中的应用却因数据稀缺而受到阻碍。尽管已经提出了几种基于大型语言模型 (LLM) 的时间序列预测方法，但在 LLM 背景下的时间序列字幕尚未得到充分探索。在本文中，我们介绍了 TSLM，一种专为时间序列字幕设计的新型时间序列语言模型。TSLM 作为编码器-解码器模型运行，利用文本提示和时间序列数据表示来捕获多个阶段的细微时间模式并生成时间序列输入的精确文本描述。 TSLM 首先利用上下文提示合成数据生成，然后通过应用于时间序列-字幕对的新型跨模态密集检索评分对生成的数据进行去噪，从而解决了时间序列字幕中的数据稀缺问题。在各种时间序列字幕数据集上的实验结果表明，TSLM 的表现远胜于现有的多种数据模态的先进方法。

Title: Turning Logic Against Itself : Probing Model Defenses Through Contrastive Questions

• Authors: Rachneet Sachdeva, Rima Hazra, Iryna Gurevych
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.01872
• Pdf URL: https://arxiv.org/pdf/2501.01872
• Copy Paste: [[2501.01872]] Turning Logic Against Itself : Probing Model Defenses Through Contrastive Questions(https://arxiv.org/abs/2501.01872)
• Keywords: language model, gpt, prompt, chain-of-thought
• Abstract: Despite significant efforts to align large language models with human values and ethical guidelines, these models remain susceptible to sophisticated jailbreak attacks that exploit their reasoning capabilities. Traditional safety mechanisms often focus on detecting explicit malicious intent, leaving deeper vulnerabilities unaddressed. In this work, we introduce a jailbreak technique, POATE (Polar Opposite query generation, Adversarial Template construction, and Elaboration), which leverages contrastive reasoning to elicit unethical responses. POATE generates prompts with semantically opposite intents and combines them with adversarial templates to subtly direct models toward producing harmful responses. We conduct extensive evaluations across six diverse language model families of varying parameter sizes, including LLaMA3, Gemma2, Phi3, and GPT-4, to demonstrate the robustness of the attack, achieving significantly higher attack success rates (~44%) compared to existing methods. We evaluate our proposed attack against seven safety defenses, revealing their limitations in addressing reasoning-based vulnerabilities. To counteract this, we propose a defense strategy that improves reasoning robustness through chain-of-thought prompting and reverse thinking, mitigating reasoning-driven adversarial exploits.
• 摘要：尽管付出了巨大努力将大型语言模型与人类价值观和道德准则相结合，但这些模型仍然容易受到利用其推理能力的复杂越狱攻击。传统的安全机制通常侧重于检测明确的恶意意图，而没有解决更深层次的漏洞。在这项工作中，我们引入了一种越狱技术 POATE（极对立查询生成、对抗模板构建和细化），它利用对比推理来引发不道德的反应。POATE 生成具有语义相反意图的提示，并将它们与对抗模板相结合，巧妙地引导模型产生有害反应。我们对六个不同参数大小的语言模型系列进行了广泛的评估，包括 LLaMA3、Gemma2、Phi3 和 GPT-4，以证明攻击的稳健性，与现有方法相比，攻击成功率显著提高（~44%）。我们评估了我们针对七种安全防御措施提出的攻击，揭示了它们在解决基于推理的漏洞方面的局限性。为了解决这个问题，我们提出了一种防御策略，通过思路链提示和逆向思维来提高推理的稳健性，减轻推理驱动的对抗性攻击。

Title: Long Context vs. RAG for LLMs: An Evaluation and Revisits

• Authors: Xinze Li, Yixin Cao, Yubo Ma, Aixin Sun
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.01880
• Pdf URL: https://arxiv.org/pdf/2501.01880
• Copy Paste: [[2501.01880]] Long Context vs. RAG for LLMs: An Evaluation and Revisits(https://arxiv.org/abs/2501.01880)
• Keywords: llm, long context, retrieval-augmented generation
• Abstract: Extending context windows (i.e., Long Context, LC) and using retrievers to selectively access relevant information (i.e., Retrieval-Augmented Generation, RAG) are the two main strategies to enable LLMs to incorporate extremely long external contexts. This paper revisits recent studies on this topic, highlighting their key insights and discrepancies. We then provide a more comprehensive evaluation by filtering out questions answerable without external context, identifying the most effective retrieval methods, and expanding the datasets. We show that LC generally outperforms RAG in question-answering benchmarks, especially for Wikipedia-based questions. Summarization-based retrieval performs comparably to LC, while chunk-based retrieval lags behind. However, RAG has advantages in dialogue-based and general question queries. These insights underscore the trade-offs between RAG and LC strategies, offering guidance for future optimization of LLMs with external knowledge sources. We also provide an in-depth discussion on this topic, highlighting the overlooked importance of context relevance in existing studies.
• 摘要：扩展上下文窗口（即长上下文，LC）和使用检索器选择性地访问相关信息（即检索增强生成，RAG）是使 LLM 能够合并极长的外部上下文的两种主要策略。本文回顾了有关该主题的最新研究，强调了它们的关键见解和差异。然后，我们通过过滤掉无需外部上下文即可回答的问题、确定最有效的检索方法以及扩展数据集来提供更全面的评估。我们表明，LC 在问答基准测试中通常优于 RAG，尤其是对于基于维基百科的问题。基于摘要的检索表现与 LC 相当，而基于块的检索则落后。然而，RAG 在基于对话和一般问题查询方面具有优势。这些见解强调了 RAG 和 LC 策略之间的权衡，为未来使用外部知识源优化 LLM 提供了指导。我们还对这一主题进行了深入讨论，强调了现有研究中忽视的上下文相关性的重要性。

Title: Abstractive Text Summarization for Contemporary Sanskrit Prose: Issues and Challenges

• Authors: Shagun Sinha
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.01933
• Pdf URL: https://arxiv.org/pdf/2501.01933
• Copy Paste: [[2501.01933]] Abstractive Text Summarization for Contemporary Sanskrit Prose: Issues and Challenges(https://arxiv.org/abs/2501.01933)
• Keywords: language model
• Abstract: This thesis presents Abstractive Text Summarization models for contemporary Sanskrit prose. The first chapter, titled Introduction, presents the motivation behind this work, the research questions, and the conceptual framework. Sanskrit is a low-resource inflectional language. The key research question that this thesis investigates is what the challenges in developing an abstractive TS for Sanskrit. To answer the key research questions, sub-questions based on four different themes have been posed in this work. The second chapter, Literature Review, surveys the previous works done. The third chapter, data preparation, answers the remaining three questions from the third theme. It reports the data collection and preprocessing challenges for both language model and summarization model trainings. The fourth chapter reports the training and inference of models and the results obtained therein. This research has initiated a pipeline for Sanskrit abstractive text summarization and has reported the challenges faced at every stage of the development. The research questions based on every theme have been answered to answer the key research question.
• 摘要：本论文提出了当代梵语散文的抽象文本摘要模型。第一章题为“简介”，介绍了这项工作背后的动机、研究问题和概念框架。梵语是一种资源匮乏的屈折语言。本论文研究的关键研究问题是开发梵语抽象文本摘要的挑战是什么。为了回答关键的研究问题，本文提出了基于四个不同主题的子问题。第二章“文献综述”概述了之前所做的工作。第三章“数据准备”回答了第三个主题的其余三个问题。它报告了语言模型和摘要模型训练的数据收集和预处理挑战。第四章报告了模型的训练和推理以及其中获得的结果。这项研究为梵语抽象文本摘要开辟了一条管道，并报告了开发每个阶段面临的挑战。基于每个主题的研究问题都得到了回答，以回答关键研究问题。

Title: Metadata Conditioning Accelerates Language Model Pre-training

• Authors: Tianyu Gao, Alexander Wettig, Luxi He, Yihe Dong, Sadhika Malladi, Danqi Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.01956
• Pdf URL: https://arxiv.org/pdf/2501.01956
• Copy Paste: [[2501.01956]] Metadata Conditioning Accelerates Language Model Pre-training(https://arxiv.org/abs/2501.01956)
• Keywords: language model, prompt
• Abstract: The vast diversity of styles, domains, and quality levels present in language model pre-training corpora is essential in developing general model capabilities, but efficiently learning and deploying the correct behaviors exemplified in each of these heterogeneous data sources is challenging. To address this, we propose a new method, termed Metadata Conditioning then Cooldown (MeCo), to incorporate additional learning cues during pre-training. MeCo first provides metadata (e.g., URLs like this http URL) alongside the text during training and later uses a cooldown phase with only the standard text, thereby enabling the model to function normally even without metadata. MeCo significantly accelerates pre-training across different model scales (600M to 8B parameters) and training sources (C4, RefinedWeb, and DCLM). For instance, a 1.6B language model trained with MeCo matches the downstream task performance of standard pre-training while using 33% less data. Additionally, MeCo enables us to steer language models by conditioning the inference prompt on either real or fabricated metadata that encodes the desired properties of the output: for example, prepending this http URL to reduce harmful generations or this http URL (fabricated) to improve common knowledge task performance. We also demonstrate that MeCo is compatible with different types of metadata, such as model-generated topics. MeCo is remarkably simple, adds no computational overhead, and demonstrates promise in producing more capable and steerable language models.
• 摘要：语言模型预训练语料库中存在着风格、领域和质量水平的多样性，这对于开发通用模型功能至关重要，但要有效地学习和部署每个异构数据源中体现的正确行为却具有挑战性。为了解决这个问题，我们提出了一种新方法，称为元数据调节然后冷却 (MeCo)，以在预训练期间纳入额外的学习线索。MeCo 首先在训练期间提供元数据（例如，像这个 http URL 这样的 URL）以及文本，然后使用仅使用标准文本的冷却阶段，从而使模型即使没有元数据也能正常运行。MeCo 显著加快了不同模型规模（6 亿到 80 亿个参数）和训练源（C4、RefinedWeb 和 DCLM）的预训练。例如，使用 MeCo 训练的 16 亿语言模型与标准预训练的下游任务性能相匹配，同时使用的数据减少了 33%。此外，MeCo 使我们能够通过对真实或虚构的元数据（对输出的所需属性进行编码）进行推理提示来控制语言模型：例如，添加此 http URL 以减少有害生成或添加此 http URL（虚构）以提高常识任务性能。我们还证明了 MeCo 与不同类型的元数据兼容，例如模型生成的主题。MeCo 非常简单，不会增加计算开销，并且在生成更强大、更可控的语言模型方面表现出了潜力。