2025-01-17

Title: Decompose-ToM: Enhancing Theory of Mind Reasoning in Large Language Models through Simulation and Task Decomposition

• Authors: Sneheel Sarangi, Maha Elgarf, Hanan Salam
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.09056
• Pdf URL: https://arxiv.org/pdf/2501.09056
• Copy Paste: [[2501.09056]] Decompose-ToM: Enhancing Theory of Mind Reasoning in Large Language Models through Simulation and Task Decomposition(https://arxiv.org/abs/2501.09056)
• Keywords: language model, llm, prompt
• Abstract: Theory of Mind (ToM) is the ability to understand and reflect on the mental states of others. Although this capability is crucial for human interaction, testing on Large Language Models (LLMs) reveals that they possess only a rudimentary understanding of it. Although the most capable closed-source LLMs have come close to human performance on some ToM tasks, they still perform poorly on complex variations of the task that involve more structured reasoning. In this work, we utilize the concept of "pretend-play", or ``Simulation Theory'' from cognitive psychology to propose ``Decompose-ToM'': an LLM-based inference algorithm that improves model performance on complex ToM tasks. We recursively simulate user perspectives and decompose the ToM task into a simpler set of functions: subject identification, question-reframing, world model updation, and knowledge availability. We test the algorithm on higher-order ToM tasks and a task testing for ToM capabilities in a conversational setting, demonstrating that our approach shows significant improvement across models compared to baseline methods while requiring minimal prompt tuning across tasks and no additional model training.
• 摘要：心智理论 (ToM) 是理解和反思他人心理状态的能力。尽管这种能力对于人类互动至关重要，但在大型语言模型 (LLM) 上进行的测试表明，它们对此只有基本的理解。尽管最强大的闭源 LLM 在某些 ToM 任务上的表现已经接近人类，但它们在涉及更结构化推理的复杂任务变体上仍然表现不佳。在这项工作中，我们利用认知心理学中的“假装游戏”或“模拟理论”概念来提出“分解 ToM”：一种基于 LLM 的推理算法，可提高复杂 ToM 任务的模型性能。我们递归模拟用户视角，并将 ToM 任务分解为一组更简单的功能：主题识别、问题重构、世界模型更新和知识可用性。我们在高阶 ToM 任务和对话环境中测试 ToM 能力的任务上测试了该算法，结果表明，与基线方法相比，我们的方法在各个模型中显示出显着的改进，同时只需要在任务之间进行最少的快速调整，并且不需要额外的模型训练。

Title: SteLLA: A Structured Grading System Using LLMs with RAG

• Authors: Hefei Qiu, Brian White, Ashley Ding, Reinaldo Costa, Ali Hachem, Wei Ding, Ping Chen
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2501.09092
• Pdf URL: https://arxiv.org/pdf/2501.09092
• Copy Paste: [[2501.09092]] SteLLA: A Structured Grading System Using LLMs with RAG(https://arxiv.org/abs/2501.09092)
• Keywords: language model, gpt, llm, retrieval augmented generation
• Abstract: Large Language Models (LLMs) have shown strong general capabilities in many applications. However, how to make them reliable tools for some specific tasks such as automated short answer grading (ASAG) remains a challenge. We present SteLLA (Structured Grading System Using LLMs with RAG) in which a) Retrieval Augmented Generation (RAG) approach is used to empower LLMs specifically on the ASAG task by extracting structured information from the highly relevant and reliable external knowledge based on the instructor-provided reference answer and rubric, b) an LLM performs a structured and question-answering-based evaluation of student answers to provide analytical grades and feedback. A real-world dataset that contains students' answers in an exam was collected from a college-level Biology course. Experiments show that our proposed system can achieve substantial agreement with the human grader while providing break-down grades and feedback on all the knowledge points examined in the problem. A qualitative and error analysis of the feedback generated by GPT4 shows that GPT4 is good at capturing facts while may be prone to inferring too much implication from the given text in the grading task which provides insights into the usage of LLMs in the ASAG system.
• 摘要：大型语言模型 (LLM) 在许多应用中都表现出强大的通用能力。然而，如何使它们成为某些特定任务（例如自动简答评分 (ASAG)）的可靠工具仍然是一个挑战。我们提出了 SteLLA（使用带有 RAG 的 LLM 的结构化评分系统），其中 a) 使用检索增强生成 (RAG) 方法通过从基于讲师提供的参考答案和评分标准的高度相关和可靠的外部知识中提取结构化信息，专门针对 ASAG 任务增强 LLM 的能力，b) LLM 对学生的答案进行结构化和基于问答的评估，以提供分析成绩和反馈。从大学水平的生物学课程中收集了一个包含学生考试答案的真实数据集。实验表明，我们提出的系统可以与人类评分员达成基本一致，同时对问题中检查的所有知识点提供细分成绩和反馈。对 GPT4 生成的反馈进行定性和错误分析表明，GPT4 擅长捕捉事实，但在评分任务中可能容易从给定的文本中推断出过多的含义，这为 ASAG 系统中 LLM 的使用提供了见解。

Title: Augmenting Human-Annotated Training Data with Large Language Model Generation and Distillation in Open-Response Assessment

• Authors: Conrad Borchers, Danielle R. Thomas, Jionghao Lin, Ralph Abboud, Kenneth R. Koedinger
• Subjects: cs.CL, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.09126
• Pdf URL: https://arxiv.org/pdf/2501.09126
• Copy Paste: [[2501.09126]] Augmenting Human-Annotated Training Data with Large Language Model Generation and Distillation in Open-Response Assessment(https://arxiv.org/abs/2501.09126)
• Keywords: language model, gpt, llm
• Abstract: Large Language Models (LLMs) like GPT-4o can help automate text classification tasks at low cost and scale. However, there are major concerns about the validity and reliability of LLM outputs. By contrast, human coding is generally more reliable but expensive to procure at scale. In this study, we propose a hybrid solution to leverage the strengths of both. We combine human-coded data and synthetic LLM-produced data to fine-tune a classical machine learning classifier, distilling both into a smaller BERT model. We evaluate our method on a human-coded test set as a validity measure for LLM output quality. In three experiments, we systematically vary LLM-generated samples' size, variety, and consistency, informed by best practices in LLM tuning. Our findings indicate that augmenting datasets with synthetic samples improves classifier performance, with optimal results achieved at an 80% synthetic to 20% human-coded data ratio. Lower temperature settings of 0.3, corresponding to less variability in LLM generations, produced more stable improvements but also limited model learning from augmented samples. In contrast, higher temperature settings (0.7 and above) introduced greater variability in performance estimates and, at times, lower performance. Hence, LLMs may produce more uniform output that classifiers overfit to earlier or produce more diverse output that runs the risk of deteriorating model performance through information irrelevant to the prediction task. Filtering out inconsistent synthetic samples did not enhance performance. We conclude that integrating human and LLM-generated data to improve text classification models in assessment offers a scalable solution that leverages both the accuracy of human coding and the variety of LLM outputs.
• 摘要：大型语言模型 (LLM)（如 GPT-4o）可以帮助以低成本和规模自动化文本分类任务。然而，人们对 LLM 输出的有效性和可靠性存在重大担忧。相比之下，人工编码通常更可靠，但大规模采购成本高昂。在这项研究中，我们提出了一种混合解决方案来利用两者的优势。我们将人工编码的数据和合成的 LLM 生成的数据结合起来，对经典的机器学习分类器进行微调，将两者提炼成一个较小的 BERT 模型。我们在人工编码的测试集上评估我们的方法，作为 LLM 输出质量的有效性度量。在三个实验中，我们系统地改变 LLM 生成的样本的大小、种类和一致性，并参考 LLM 调整的最佳实践。我们的研究结果表明，用合成样本扩充数据集可以提高分类器的性能，在 80% 的合成数据和 20% 的人工编码数据比率下可实现最佳结果。较低的温度设置为 0.3，对应于 LLM 生成中的变化较小，产生了更稳定的改进，但也限制了从扩充样本中进行的模型学习。相反，更高的温度设置（0.7 及以上）会导致性能估计值出现更大的变化，有时还会导致性能下降。因此，LLM 可能会产生更均匀的输出，而分类器会过早地过度拟合，或者产生更多样化的输出，而这些输出可能会因与预测任务无关的信息而导致模型性能下降。过滤掉不一致的合成样本并不能提高性能。我们得出结论，整合人工和 LLM 生成的数据来改进评估中的文本分类模型是一种可扩展的解决方案，既能利用人工编码的准确性，又能利用 LLM 输出的多样性。

Title: Multilingual LLMs Struggle to Link Orthography and Semantics in Bilingual Word Processing

• Authors: Eshaan Tanwar, Gayatri Oke, Tanmoy Chakraborty
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.09127
• Pdf URL: https://arxiv.org/pdf/2501.09127
• Copy Paste: [[2501.09127]] Multilingual LLMs Struggle to Link Orthography and Semantics in Bilingual Word Processing(https://arxiv.org/abs/2501.09127)
• Keywords: language model, llm
• Abstract: Bilingual lexical processing is shaped by the complex interplay of phonological, orthographic, and semantic features of two languages within an integrated mental lexicon. In humans, this is evident in the ease with which cognate words - words similar in both orthographic form and meaning (e.g., blind, meaning "sightless" in both English and German) - are processed, compared to the challenges posed by interlingual homographs, which share orthographic form but differ in meaning (e.g., gift, meaning "present" in English but "poison" in German). We investigate how multilingual Large Language Models (LLMs) handle such phenomena, focusing on English-Spanish, English-French, and English-German cognates, non-cognate, and interlingual homographs. Specifically, we evaluate their ability to disambiguate meanings and make semantic judgments, both when these word types are presented in isolation or within sentence contexts. Our findings reveal that while certain LLMs demonstrate strong performance in recognizing cognates and non-cognates in isolation, they exhibit significant difficulty in disambiguating interlingual homographs, often performing below random baselines. This suggests LLMs tend to rely heavily on orthographic similarities rather than semantic understanding when interpreting interlingual homographs. Further, we find LLMs exhibit difficulty in retrieving word meanings, with performance in isolative disambiguation tasks having no correlation with semantic understanding. Finally, we study how the LLM processes interlingual homographs in incongruent sentences. We find models to opt for different strategies in understanding English and non-English homographs, highlighting a lack of a unified approach to handling cross-lingual ambiguities.
• 摘要：双语词汇处理是由两种语言的音系、正字法和语义特征在综合心理词汇中复杂的相互作用形成的。在人类中，这体现在同源词（正字法形式和含义都相似的词，例如，blind，在英语和德语中意为“盲人”）的处理容易度上，而跨语言同形异义词（正字法形式相同但含义不同，例如，gift，在英语中意为“礼物”，在德语中意为“毒药”）的处理难度则更大。我们研究了多语言大型语言模型 (LLM) 如何处理此类现象，重点研究英语-西班牙语、英语-法语和英语-德语同源词、非同源词和跨语言同形异义词。具体来说，我们评估它们消除歧义和做出语义判断的能力，无论是当这些词类单独出现还是在句子上下文中出现时。我们的研究结果表明，虽然某些 LLM 在识别孤立同源词和非同源词方面表现出色，但它们在消除语言间同形异义词歧义方面表现出很大困难，通常表现低于随机基线。这表明 LLM 在解释语言间同形异义词时往往严重依赖正字法相似性而不是语义理解。此外，我们发现 LLM 在检索词义方面表现出困难，孤立消歧任务中的表现与语义理解无关。最后，我们研究了 LLM 如何处理不一致句子中的语言间同形异义词。我们发现模型在理解英语和非英语同形异义词时选择不同的策略，突显出缺乏统一的方法来处理跨语言歧义。

Title: Towards Multilingual LLM Evaluation for Baltic and Nordic languages: A study on Lithuanian History

• Authors: Yevhen Kostiuk, Oxana Vitman, Łukasz Gagała, Artur Kiulian
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.09154
• Pdf URL: https://arxiv.org/pdf/2501.09154
• Copy Paste: [[2501.09154]] Towards Multilingual LLM Evaluation for Baltic and Nordic languages: A study on Lithuanian History(https://arxiv.org/abs/2501.09154)
• Keywords: language model, gpt, llm
• Abstract: In this work, we evaluated Lithuanian and general history knowledge of multilingual Large Language Models (LLMs) on a multiple-choice question-answering task. The models were tested on a dataset of Lithuanian national and general history questions translated into Baltic, Nordic, and other languages (English, Ukrainian, Arabic) to assess the knowledge sharing from culturally and historically connected groups. We evaluated GPT-4o, LLaMa3.1 8b and 70b, QWEN2.5 7b and 72b, Mistral Nemo 12b, LLaMa3 8b, Mistral 7b, LLaMa3.2 3b, and Nordic fine-tuned models (GPT-SW3 and LLaMa3 8b). Our results show that GPT-4o consistently outperformed all other models across language groups, with slightly better results for Baltic and Nordic languages. Larger open-source models like QWEN2.5 72b and LLaMa3.1 70b performed well but showed weaker alignment with Baltic languages. Smaller models (Mistral Nemo 12b, LLaMa3.2 3b, QWEN 7B, LLaMa3.1 8B, and LLaMa3 8b) demonstrated gaps with LT-related alignment with Baltic languages while performing better on Nordic and other languages. The Nordic fine-tuned models did not surpass multilingual models, indicating that shared cultural or historical context alone does not guarantee better performance.
• 摘要：在这项工作中，我们在多项选择问答任务中评估了多语言大型语言模型 (LLM) 的立陶宛语和一般历史知识。这些模型在立陶宛国家和一般历史问题的数据集上进行了测试，这些问题被翻译成波罗的海语、北欧语和其他语言（英语、乌克兰语、阿拉伯语），以评估文化和历史相关群体的知识共享。我们评估了 GPT-4o、LLaMa3.1 8b 和 70b、QWEN2.5 7b 和 72b、Mistral Nemo 12b、LLaMa3 8b、Mistral 7b、LLaMa3.2 3b 和北欧微调模型（GPT-SW3 和 LLaMa3 8b）。我们的结果表明，GPT-4o 在各个语言组中的表现始终优于所有其他模型，波罗的海语和北欧语的结果略好一些。较大的开源模型（如 QWEN2.5 72b 和 LLaMa3.1 70b）表现良好，但与波罗的海语言的对齐较弱。较小的模型（Mistral Nemo 12b、LLaMa3.2 3b、QWEN 7B、LLaMa3.1 8B 和 LLaMa3 8b）与波罗的海语言的 LT 相关对齐存在差距，但在北欧和其他语言上表现更好。北欧微调模型并没有超越多语言模型，这表明仅靠共同的文化或历史背景并不能保证更好的性能。

Title: Evaluating GenAI for Simplifying Texts for Education: Improving Accuracy and Consistency for Enhanced Readability

• Authors: Stephanie L. Day, Jacapo Cirica, Steven R. Clapp, Veronika Penkova, Amy E. Giroux, Abbey Banta, Catherine Bordeau, Poojitha Mutteneni, Ben D. Sawyer
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.09158
• Pdf URL: https://arxiv.org/pdf/2501.09158
• Copy Paste: [[2501.09158]] Evaluating GenAI for Simplifying Texts for Education: Improving Accuracy and Consistency for Enhanced Readability(https://arxiv.org/abs/2501.09158)
• Keywords: language model, llm, prompt, agent
• Abstract: Generative artificial intelligence (GenAI) holds great promise as a tool to support personalized learning. Teachers need tools to efficiently and effectively enhance content readability of educational texts so that they are matched to individual students reading levels, while retaining key details. Large Language Models (LLMs) show potential to fill this need, but previous research notes multiple shortcomings in current approaches. In this study, we introduced a generalized approach and metrics for the systematic evaluation of the accuracy and consistency in which LLMs, prompting techniques, and a novel multi-agent architecture to simplify sixty informational reading passages, reducing each from the twelfth grade level down to the eighth, sixth, and fourth grade levels. We calculated the degree to which each LLM and prompting technique accurately achieved the targeted grade level for each passage, percentage change in word count, and consistency in maintaining keywords and key phrases (semantic similarity). One-sample t-tests and multiple regression models revealed significant differences in the best performing LLM and prompt technique for each of the four metrics. Both LLMs and prompting techniques demonstrated variable utility in grade level accuracy and consistency of keywords and key phrases when attempting to level content down to the fourth grade reading level. These results demonstrate the promise of the application of LLMs for efficient and precise automated text simplification, the shortcomings of current models and prompting methods in attaining an ideal balance across various evaluation criteria, and a generalizable method to evaluate future systems.
• 摘要：生成人工智能 (GenAI) 作为一种支持个性化学习的工具，前景广阔。教师需要工具来高效、有效地提高教育文本内容的可读性，以便它们与个别学生的阅读水平相匹配，同时保留关键细节。大型语言模型 (LLM) 显示出满足这一需求的潜力，但先前的研究指出，当前方法存在多个缺点。在本研究中，我们引入了一种通用方法和指标，用于系统评估 LLM、提示技术和一种新颖的多代理架构的准确性和一致性，以简化 60 个信息阅读段落，将每个段落从十二年级水平降低到八年级、六年级和四年级水平。我们计算了每个 LLM 和提示技术准确达到每个段落目标年级水平的程度、字数的百分比变化以及保持关键字和关键短语的一致性（语义相似性）。单样本 t 检验和多元回归模型显示，在四个指标中，表现最佳的 LLM 和提示技术存在显著差异。在尝试将内容分级到四年级阅读水平时，LLM 和提示技术在关键词和关键短语的年级准确性和一致性方面表现出不同的效用。这些结果证明了 LLM 在高效、精确的自动文本简化方面的应用前景，证明了当前模型和提示方法在实现各种评估标准的理想平衡方面的不足，并证明了一种可推广的评估未来系统的方法。

Title: The Veln(ia)s is in the Details: Evaluating LLM Judgment on Latvian and Lithuanian Short Answer Matching

• Authors: Yevhen Kostiuk, Oxana Vitman, Łukasz Gagała, Artur Kiulian
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.09164
• Pdf URL: https://arxiv.org/pdf/2501.09164
• Copy Paste: [[2501.09164]] The Veln(ia)s is in the Details: Evaluating LLM Judgment on Latvian and Lithuanian Short Answer Matching(https://arxiv.org/abs/2501.09164)
• Keywords: language model, llm
• Abstract: In this work, we address the challenge of evaluating large language models (LLMs) on the short answer matching task for Latvian and Lithuanian languages. We introduce novel datasets consisting of 502 Latvian and 690 Lithuanian question-answer pairs. For each question-answer pair, we generated matched and non-matched answers using a set of alteration rules specifically designed to introduce small but meaningful changes in the text. These generated answers serve as test cases to assess the ability of LLMs to detect subtle differences in matching of the original answers. A subset of the datasets was manually verified for quality and accuracy. Our results show that while larger LLMs, such as QWEN2.5 72b and LLaMa3.1 70b, demonstrate near-perfect performance in distinguishing matched and non-matched answers, smaller models show more variance. For instance, LLaMa3.1 8b and EuroLLM 9b benefited from few-shot examples, while Mistral Nemo 12b underperformed on detection of subtle text alteration, particularly in Lithuanian, even with additional examples. QWEN2.5 7b and Mistral 7b were able to obtain a strong and comparable performance to the larger 70b models in zero and few shot experiments. Moreover, the performance of Mistral 7b was weaker in few shot experiments.
• 摘要：在这项工作中，我们解决了评估大型语言模型 (LLM) 在拉脱维亚语和立陶宛语简答匹配任务中的挑战。我们引入了由 502 个拉脱维亚语和 690 个立陶宛语问答对组成的新数据集。对于每个问答对，我们使用一组专门设计用于在文本中引入细小但有意义的更改的更改规则生成匹配和不匹配的答案。这些生成的答案作为测试用例，以评估 LLM 检测原始答案匹配中细微差异的能力。对数据集的一个子集进行了手动质量和准确性验证。我们的结果表明，虽然较大的 LLM（例如 QWEN2.5 72b 和 LLaMa3.1 70b）在区分匹配和不匹配答案方面表现出近乎完美的性能，但较小的模型显示出更多的差异。例如，LLaMa3.1 8b 和 EuroLLM 9b 受益于少量样本，而 Mistral Nemo 12b 在检测细微文本改动方面表现不佳，尤其是在立陶宛语中，即使有更多样本也是如此。QWEN2.5 7b 和 Mistral 7b 在零样本和少量样本实验中能够获得与更大的 70b 模型相当的强大性能。此外，Mistral 7b 在少量样本实验中的表现较弱。

Title: FineMedLM-o1: Enhancing the Medical Reasoning Ability of LLM from Supervised Fine-Tuning to Test-Time Training

• Authors: Hongzhou Yu, Tianhao Cheng, Ying Cheng, Rui Feng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.09213
• Pdf URL: https://arxiv.org/pdf/2501.09213
• Copy Paste: [[2501.09213]] FineMedLM-o1: Enhancing the Medical Reasoning Ability of LLM from Supervised Fine-Tuning to Test-Time Training(https://arxiv.org/abs/2501.09213)
• Keywords: language model, llm
• Abstract: Recent advancements in large language models (LLMs) have shown promise in medical applications such as disease diagnosis and treatment planning. However, most existing medical LLMs struggle with the advanced reasoning required for complex clinical scenarios, such as differential diagnosis or personalized treatment suggestions. We proposed FineMedLM-o1, which leverages high-quality synthetic medical data and long-form reasoning data for Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO), enabling advanced dialogue and deep reasoning capabilities. Additionally, we introduced Test-Time Training (TTT) in the medical domain for the first time, facilitating domain adaptation and ensuring reliable, accurate reasoning. Experimental results demonstrate that FineMedLM-o1 achieves a 23% average performance improvement over prior models on key medical benchmarks. Furthermore, the introduction of TTT provides an additional 14% performance boost, highlighting its effectiveness in enhancing medical reasoning capabilities. To support this process, we also proposed a novel method for synthesizing medical dialogue. Compared to other open-source datasets, our dataset stands out as superior in both quality and complexity. The project and data will be released on GitHub.
• 摘要：大型语言模型 (LLM) 的最新进展已在疾病诊断和治疗计划等医疗应用中显示出良好的前景。然而，大多数现有的医学 LLM 都难以应对复杂临床场景所需的高级推理，例如鉴别诊断或个性化治疗建议。我们提出了 FineMedLM-o1，它利用高质量的合成医疗数据和长格式推理数据进行监督微调 (SFT) 和直接偏好优化 (DPO)，从而实现高级对话和深度推理功能。此外，我们首次在医疗领域引入了测试时间训练 (TTT)，促进了领域适应并确保了可靠、准确的推理。实验结果表明，FineMedLM-o1 在关键医疗基准上的平均性能比以前的模型提高了 23%。此外，TTT 的引入还提供了额外的 14% 性能提升，凸显了其在增强医疗推理能力方面的有效性。为了支持这一过程，我们还提出了一种合成医疗对话的新方法。与其他开源数据集相比，我们的数据集在质量和复杂性方面都表现出色。该项目和数据将在GitHub上发布。

Title: Boosting Short Text Classification with Multi-Source Information Exploration and Dual-Level Contrastive Learning

• Authors: Yonghao Liu, Mengyu Li, Wei Pang, Fausto Giunchiglia, Lan Huang, Xiaoyue Feng, Renchu Guan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.09214
• Pdf URL: https://arxiv.org/pdf/2501.09214
• Copy Paste: [[2501.09214]] Boosting Short Text Classification with Multi-Source Information Exploration and Dual-Level Contrastive Learning(https://arxiv.org/abs/2501.09214)
• Keywords: language model
• Abstract: Short text classification, as a research subtopic in natural language processing, is more challenging due to its semantic sparsity and insufficient labeled samples in practical scenarios. We propose a novel model named MI-DELIGHT for short text classification in this work. Specifically, it first performs multi-source information (i.e., statistical information, linguistic information, and factual information) exploration to alleviate the sparsity issues. Then, the graph learning approach is adopted to learn the representation of short texts, which are presented in graph forms. Moreover, we introduce a dual-level (i.e., instance-level and cluster-level) contrastive learning auxiliary task to effectively capture different-grained contrastive information within massive unlabeled data. Meanwhile, previous models merely perform the main task and auxiliary tasks in parallel, without considering the relationship among tasks. Therefore, we introduce a hierarchical architecture to explicitly model the correlations between tasks. We conduct extensive experiments across various benchmark datasets, demonstrating that MI-DELIGHT significantly surpasses previous competitive models. It even outperforms popular large language models on several datasets.
• 摘要：短文本分类作为自然语言处理的一个研究子课题，由于语义稀疏性和实际场景中标记样本不足而更具挑战性。本文提出了一种用于短文本分类的新模型 MI-DELIGHT。具体而言，它首先进行多源信息（即统计信息、语言信息和事实信息）探索以缓解稀疏性问题。然后，采用图学习方法来学习短文本的表示，并以图形形式呈现。此外，我们引入了一个双层（即实例级和集群级）对比学习辅助任务，以有效地捕获大量未标记数据中不同粒度的对比信息。同时，以前的模型只是并行执行主任务和辅助任务，而不考虑任务之间的关系。因此，我们引入了一个分层架构来明确地模拟任务之间的相关性。我们在各种基准数据集上进行了广泛的实验，表明 MI-DELIGHT 明显超越了以前的竞争模型。它甚至在几个数据集上优于流行的大型语言模型。

Title: A Simple Graph Contrastive Learning Framework for Short Text Classification

• Authors: Yonghao Liu, Fausto Giunchiglia, Lan Huang, Ximing Li, Xiaoyue Feng, Renchu Guan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.09219
• Pdf URL: https://arxiv.org/pdf/2501.09219
• Copy Paste: [[2501.09219]] A Simple Graph Contrastive Learning Framework for Short Text Classification(https://arxiv.org/abs/2501.09219)
• Keywords: language model
• Abstract: Short text classification has gained significant attention in the information age due to its prevalence and real-world applications. Recent advancements in graph learning combined with contrastive learning have shown promising results in addressing the challenges of semantic sparsity and limited labeled data in short text classification. However, existing models have certain limitations. They rely on explicit data augmentation techniques to generate contrastive views, resulting in semantic corruption and noise. Additionally, these models only focus on learning the intrinsic consistency between the generated views, neglecting valuable discriminative information from other potential views. To address these issues, we propose a Simple graph contrastive learning framework for Short Text Classification (SimSTC). Our approach involves performing graph learning on multiple text-related component graphs to obtain multi-view text embeddings. Subsequently, we directly apply contrastive learning on these embeddings. Notably, our method eliminates the need for data augmentation operations to generate contrastive views while still leveraging the benefits of multi-view contrastive learning. Despite its simplicity, our model achieves outstanding performance, surpassing large language models on various datasets.
• 摘要：短文本分类因其普遍性和实际应用而在信息时代引起了广泛关注。图学习与对比学习相结合的最新进展在解决短文本分类中的语义稀疏性和有限标记数据挑战方面取得了令人鼓舞的结果。然而，现有模型有一定的局限性。它们依靠显式的数据增强技术来生成对比视图，导致语义损坏和噪声。此外，这些模型只关注学习生成的视图之间的内在一致性，而忽略了来自其他潜在视图的有价值的判别信息。为了解决这些问题，我们提出了一个用于短文本分类的简单图对比学习框架 (SimSTC)。我们的方法涉及对多个文本相关的组件图执行图学习以获得多视图文本嵌入。随后，我们直接对这些嵌入应用对比学习。值得注意的是，我们的方法消除了数据增强操作来生成对比视图的需要，同时仍然利用了多视图对比学习的优势。尽管很简单，但我们的模型取得了出色的性能，在各种数据集上超越了大型语言模型。

Title: Foundations of Large Language Models

• Authors: Tong Xiao, Jingbo Zhu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.09223
• Pdf URL: https://arxiv.org/pdf/2501.09223
• Copy Paste: [[2501.09223]] Foundations of Large Language Models(https://arxiv.org/abs/2501.09223)
• Keywords: language model, prompt
• Abstract: This is a book about large language models. As indicated by the title, it primarily focuses on foundational concepts rather than comprehensive coverage of all cutting-edge technologies. The book is structured into four main chapters, each exploring a key area: pre-training, generative models, prompting techniques, and alignment methods. It is intended for college students, professionals, and practitioners in natural language processing and related fields, and can serve as a reference for anyone interested in large language models.
• 摘要：这是一本关于大型语言模型的书。正如书名所示，它主要关注基础概念，而不是全面涵盖所有前沿技术。本书分为四个主要章节，每章探讨一个关键领域：预训练、生成模型、提示技术和对齐方法。它面向自然语言处理和相关领域的大学生、专业人士和从业者，可以作为对大型语言模型感兴趣的任何人的参考。

Title: Delayed Fusion: Integrating Large Language Models into First-Pass Decoding in End-to-end Speech Recognition

• Authors: Takaaki Hori, Martin Kocour, Adnan Haider, Erik McDermott, Xiaodan Zhuang
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2501.09258
• Pdf URL: https://arxiv.org/pdf/2501.09258
• Copy Paste: [[2501.09258]] Delayed Fusion: Integrating Large Language Models into First-Pass Decoding in End-to-end Speech Recognition(https://arxiv.org/abs/2501.09258)
• Keywords: language model, llm
• Abstract: This paper presents an efficient decoding approach for end-to-end automatic speech recognition (E2E-ASR) with large language models (LLMs). Although shallow fusion is the most common approach to incorporate language models into E2E-ASR decoding, we face two practical problems with LLMs. (1) LLM inference is computationally costly. (2) There may be a vocabulary mismatch between the ASR model and the LLM. To resolve this mismatch, we need to retrain the ASR model and/or the LLM, which is at best time-consuming and in many cases not feasible. We propose "delayed fusion," which applies LLM scores to ASR hypotheses with a delay during decoding and enables easier use of pre-trained LLMs in ASR tasks. This method can reduce not only the number of hypotheses scored by the LLM but also the number of LLM inference calls. It also allows re-tokenizion of ASR hypotheses during decoding if ASR and LLM employ different tokenizations. We demonstrate that delayed fusion provides improved decoding speed and accuracy compared to shallow fusion and N-best rescoring using the LibriHeavy ASR corpus and three public LLMs, OpenLLaMA 3B & 7B and Mistral 7B.
• 摘要：本文介绍了一种使用大型语言模型 (LLM) 进行端到端自动语音识别 (E2E-ASR) 的有效解码方法。虽然浅层融合是将语言模型纳入 E2E-ASR 解码的最常见方法，但我们在使用 LLM 时面临两个实际问题。(1) LLM 推理在计算上成本高昂。(2) ASR 模型和 LLM 之间可能存在词汇不匹配。为了解决这种不匹配，我们需要重新训练 ASR 模型和/或 LLM，这在最好的情况下很耗时，而且在许多情况下是不可行的。我们提出了“延迟融合”，它在解码过程中延迟将 LLM 分数应用于 ASR 假设，并使在 ASR 任务中更容易使用预训练的 LLM。这种方法不仅可以减少 LLM 评分的假设数量，还可以减少 LLM 推理调用的数量。如果 ASR 和 LLM 采用不同的标记，它还允许在解码过程中重新标记 ASR 假设。我们证明，使用 LibriHeavy ASR 语料库和三个公共 LLM（OpenLLaMA 3B & 7B 和 Mistral 7B），延迟融合比浅融合和 N-best 重新评分提供了更高的解码速度和准确性。

Title: Perspective Transition of Large Language Models for Solving Subjective Tasks

• Authors: Xiaolong Wang, Yuanchi Zhang, Ziyue Wang, Yuzhuang Xu, Fuwen Luo, Yile Wang, Peng Li, Yang Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.09265
• Pdf URL: https://arxiv.org/pdf/2501.09265
• Copy Paste: [[2501.09265]] Perspective Transition of Large Language Models for Solving Subjective Tasks(https://arxiv.org/abs/2501.09265)
• Keywords: language model, gpt, llm, prompt, chain-of-thought
• Abstract: Large language models (LLMs) have revolutionized the field of natural language processing, enabling remarkable progress in various tasks. Different from objective tasks such as commonsense reasoning and arithmetic question-answering, the performance of LLMs on subjective tasks is still limited, where the perspective on the specific problem plays crucial roles for better interpreting the context and giving proper response. For example, in certain scenarios, LLMs may perform better when answering from an expert role perspective, potentially eliciting their relevant domain knowledge. In contrast, in some scenarios, LLMs may provide more accurate responses when answering from a third-person standpoint, enabling a more comprehensive understanding of the problem and potentially mitigating inherent biases. In this paper, we propose Reasoning through Perspective Transition (RPT), a method based on in-context learning that enables LLMs to dynamically select among direct, role, and third-person perspectives for the best way to solve corresponding subjective problem. Through extensive experiments on totally 12 subjective tasks by using both closed-source and open-source LLMs including GPT-4, GPT-3.5, Llama-3, and Qwen-2, our method outperforms widely used single fixed perspective based methods such as chain-of-thought prompting and expert prompting, highlights the intricate ways that LLMs can adapt their perspectives to provide nuanced and contextually appropriate responses for different problems.
• 摘要：大型语言模型 (LLM) 彻底改变了自然语言处理领域，使各种任务取得了显著进展。与常识推理和算术问答等客观任务不同，LLM 在主观任务上的表现仍然有限，其中对特定问题的视角对于更好地解释上下文和给出适当的回答起着至关重要的作用。例如，在某些情况下，LLM 从专家角色角度回答时可能会表现更好，从而有可能引出他们相关的领域知识。相反，在某些情况下，LLM 从第三人称角度回答时可能会提供更准确的答案，从而能够更全面地理解问题并可能减轻固有偏见。在本文中，我们提出了通过视角转换进行推理 (RPT)，这是一种基于上下文学习的方法，使 LLM 能够动态地在直接、角色和第三人称视角之间进行选择，以最佳方式解决相应的主观问题。通过使用闭源和开源 LLM（包括 GPT-4、GPT-3.5、Llama-3 和 Qwen-2）对总共 12 个主观任务进行大量实验，我们的方法优于广泛使用的基于单一固定视角的方法（例如思路链提示和专家提示），突出了 LLM 可以调整其视角以针对不同问题提供细致入微且适合上下文的响应的复杂方式。

Title: To Retrieve or Not to Retrieve? Uncertainty Detection for Dynamic Retrieval Augmented Generation

• Authors: Kaustubh D. Dhole
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2501.09292
• Pdf URL: https://arxiv.org/pdf/2501.09292
• Copy Paste: [[2501.09292]] To Retrieve or Not to Retrieve? Uncertainty Detection for Dynamic Retrieval Augmented Generation(https://arxiv.org/abs/2501.09292)
• Keywords: language model, llm, hallucination, retrieval augmented generation, retrieval-augmented generation
• Abstract: Retrieval-Augmented Generation equips large language models with the capability to retrieve external knowledge, thereby mitigating hallucinations by incorporating information beyond the model's intrinsic abilities. However, most prior works have focused on invoking retrieval deterministically, which makes it unsuitable for tasks such as long-form question answering. Instead, dynamically performing retrieval by invoking it only when the underlying LLM lacks the required knowledge can be more efficient. In this context, we delve deeper into the question, "To Retrieve or Not to Retrieve?" by exploring multiple uncertainty detection methods. We evaluate these methods for the task of long-form question answering, employing dynamic retrieval, and present our comparisons. Our findings suggest that uncertainty detection metrics, such as Degree Matrix Jaccard and Eccentricity, can reduce the number of retrieval calls by almost half, with only a slight reduction in question-answering accuracy.
• 摘要：检索增强生成使大型语言模型具备检索外部知识的能力，从而通过整合超出模型固有能力的信息来减轻幻觉。然而，大多数先前的研究都侧重于确定性地调用检索，这使得它不适合长篇问答等任务。相反，只有当底层 LLM 缺乏所需知识时才调用检索来动态执行检索，效率会更高。在此背景下，我们通过探索多种不确定性检测方法，深入探讨“检索还是不检索？”这个问题。我们评估了这些方法在长篇问答任务中的动态检索效果，并进行了比较。我们的研究结果表明，不确定性检测指标（如度矩阵 Jaccard 和偏心率）可以将检索调用次数减少近一半，而问答准确率仅略有降低。

Title: A Study of In-Context-Learning-Based Text-to-SQL Errors

• Authors: Jiawei Shen, Chengcheng Wan, Ruoyi Qiao, Jiazhen Zou, Hang Xu, Yuchen Shao, Yueling Zhang, Weikai Miao, Geguang Pu
• Subjects: cs.CL, cs.AI, cs.SE
• Abstract URL: https://arxiv.org/abs/2501.09310
• Pdf URL: https://arxiv.org/pdf/2501.09310
• Copy Paste: [[2501.09310]] A Study of In-Context-Learning-Based Text-to-SQL Errors(https://arxiv.org/abs/2501.09310)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have been adopted to perform text-to-SQL tasks, utilizing their in-context learning (ICL) capability to translate natural language questions into structured query language (SQL). However, such a technique faces correctness problems and requires efficient repairing solutions. In this paper, we conduct the first comprehensive study of text-to-SQL errors. Our study covers four representative ICL-based techniques, five basic repairing methods, two benchmarks, and two LLM settings. We find that text-to-SQL errors are widespread and summarize 29 error types of 7 categories. We also find that existing repairing attempts have limited correctness improvement at the cost of high computational overhead with many mis-repairs. Based on the findings, we propose MapleRepair, a novel text-to-SQL error detection and repairing framework. The evaluation demonstrates that MapleRepair outperforms existing solutions by repairing 13.8% more queries with neglectable mis-repairs and 67.4% less overhead.
• 摘要：大型语言模型 (LLM) 已被用于执行文本到 SQL 任务，利用其上下文学习 (ICL) 功能将自然语言问题转换为结构化查询语言 (SQL)。然而，这种技术面临正确性问题，需要有效的修复解决方案。在本文中，我们对文本到 SQL 错误进行了首次全面研究。我们的研究涵盖了四种基于 ICL 的代表性技术、五种基本修复方法、两个基准和两个 LLM 设置。我们发现文本到 SQL 错误很普遍，并总结了 7 个类别的 29 种错误类型。我们还发现，现有的修复尝试以高计算开销和许多错误修复为代价，限制了正确性的提高。基于这些发现，我们提出了 MapleRepair，一种新颖的文本到 SQL 错误检测和修复框架。评估表明，MapleRepair 优于现有解决方案，修复了 13.8% 以上的查询，错误修复可忽略不计，开销减少了 67.4%。

Title: ChartInsighter: An Approach for Mitigating Hallucination in Time-series Chart Summary Generation with A Benchmark Dataset

• Authors: Fen Wang, Bomiao Wang, Xueli Shu, Zhen Liu, Zekai Shao, Chao Liu, Siming Chen
• Subjects: cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2501.09349
• Pdf URL: https://arxiv.org/pdf/2501.09349
• Copy Paste: [[2501.09349]] ChartInsighter: An Approach for Mitigating Hallucination in Time-series Chart Summary Generation with A Benchmark Dataset(https://arxiv.org/abs/2501.09349)
• Keywords: hallucination, agent
• Abstract: Effective chart summary can significantly reduce the time and effort decision makers spend interpreting charts, enabling precise and efficient communication of data insights. Previous studies have faced challenges in generating accurate and semantically rich summaries of time-series data charts. In this paper, we identify summary elements and common hallucination types in the generation of time-series chart summaries, which serve as our guidelines for automatic generation. We introduce ChartInsighter, which automatically generates chart summaries of time-series data, effectively reducing hallucinations in chart summary generation. Specifically, we assign multiple agents to generate the initial chart summary and collaborate iteratively, during which they invoke external data analysis modules to extract insights and compile them into a coherent summary. Additionally, we implement a self-consistency test method to validate and correct our summary. We create a high-quality benchmark of charts and summaries, with hallucination types annotated on a sentence-by-sentence basis, facilitating the evaluation of the effectiveness of reducing hallucinations. Our evaluations using our benchmark show that our method surpasses state-of-the-art models, and that our summary hallucination rate is the lowest, which effectively reduces various hallucinations and improves summary quality. The benchmark is available at this https URL.
• 摘要：有效的图表摘要可以大大减少决策者解读图表所花费的时间和精力，从而实现数据洞察的精准高效传达。先前的研究在生成准确且语义丰富的时间序列数据图表摘要方面面临挑战。在本文中，我们识别了时间序列图表摘要生成中的摘要元素和常见的幻觉类型，并以此作为我们自动生成的指南。我们引入了ChartInsighter，它可以自动生成时间序列数据的图表摘要，从而有效减少图表摘要生成中的幻觉。具体而言，我们指派多个代理生成初始图表摘要并进行迭代协作，在此期间他们调用外部数据分析模块来提取洞察并将其编译成连贯的摘要。此外，我们实施了一种自洽性测试方法来验证和纠正我们的摘要。我们创建了一个高质量的图表和摘要基准，并逐句注释了幻觉类型，以便于评估减少幻觉的有效性。使用我们的基准进行的评估表明，我们的方法超越了最先进的模型，并且我们的摘要幻觉率最低，从而有效减少了各种幻觉并提高了摘要质量。基准可在此 https URL 上找到。

Title: Evaluating LLM Abilities to Understand Tabular Electronic Health Records: A Comprehensive Study of Patient Data Extraction and Retrieval

• Authors: Jesus Lovon (IRIT-IRIS), Martin Mouysset (IRIT-IRIS), Jo Oleiwan (IRIT-IRIS), Jose G. Moreno (IRIT-IRIS), Christine Damase-Michel, Lynda Tamine (IRIT-IRIS)
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2501.09384
• Pdf URL: https://arxiv.org/pdf/2501.09384
• Copy Paste: [[2501.09384]] Evaluating LLM Abilities to Understand Tabular Electronic Health Records: A Comprehensive Study of Patient Data Extraction and Retrieval(https://arxiv.org/abs/2501.09384)
• Keywords: llm, prompt
• Abstract: Electronic Health Record (EHR) tables pose unique challenges among which is the presence of hidden contextual dependencies between medical features with a high level of data dimensionality and sparsity. This study presents the first investigation into the abilities of LLMs to comprehend EHRs for patient data extraction and retrieval. We conduct extensive experiments using the MIMICSQL dataset to explore the impact of the prompt structure, instruction, context, and demonstration, of two backbone LLMs, Llama2 and Meditron, based on task performance. Through quantitative and qualitative analyses, our findings show that optimal feature selection and serialization methods can enhance task performance by up to 26.79% compared to naive approaches. Similarly, in-context learning setups with relevant example selection improve data extraction performance by 5.95%. Based on our study findings, we propose guidelines that we believe would help the design of LLM-based models to support health search.
• 摘要：电子健康记录 (EHR) 表带来了独特的挑战，其中之一是具有高数据维数和稀疏性的医疗特征之间存在隐藏的上下文依赖关系。本研究首次研究了 LLM 理解 EHR 以提取和检索患者数据的能力。我们使用 MIMICSQL 数据集进行了大量实验，以探索两个骨干 LLM（Llama2 和 Meditron）的提示结构、指令、上下文和演示对任务绩效的影响。通过定量和定性分析，我们的研究结果表明，与简单方法相比，最佳特征选择和序列化方法可以将任务绩效提高多达 26.79%。同样，具有相关示例选择的上下文学习设置可将数据提取性能提高 5.95%。根据我们的研究结果，我们提出了一些指导方针，我们认为这些指导方针将有助于设计基于 LLM 的模型来支持健康搜索。

Title: mGeNTE: A Multilingual Resource for Gender-Neutral Language and Translation

• Authors: Beatrice Savoldi, Eleonora Cupin, Manjinder Thind, Anne Lauscher, Luisa Bentivogli
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.09409
• Pdf URL: https://arxiv.org/pdf/2501.09409
• Copy Paste: [[2501.09409]] mGeNTE: A Multilingual Resource for Gender-Neutral Language and Translation(https://arxiv.org/abs/2501.09409)
• Keywords: language model
• Abstract: Gender-neutral language reflects societal and linguistic shifts towards greater inclusivity by avoiding the implication that one gender is the norm over others. This is particularly relevant for grammatical gender languages, which heavily encode the gender of terms for human referents and over-relies on masculine forms, even when gender is unspecified or irrelevant. Language technologies are known to mirror these inequalities, being affected by a male bias and perpetuating stereotypical associations when translating into languages with extensive gendered morphology. In such cases, gender-neutral language can help avoid undue binary assumptions. However, despite its importance for creating fairer multi- and cross-lingual technologies, inclusive language research remains scarce and insufficiently supported in current resources. To address this gap, we present the multilingual mGeNTe dataset. Derived from the bilingual GeNTE (Piergentili et al., 2023), mGeNTE extends the original corpus to include the English-Italian/German/Spanish language pairs. Since each language pair is English-aligned with gendered and neutral sentences in the target languages, mGeNTE enables research in both automatic Gender-Neutral Translation (GNT) and language modelling for three grammatical gender languages.
• 摘要：性别中立语言反映了社会和语言向更大包容性转变的趋势，避免暗示一种性别优于其他性别。这对于语法性别语言尤其重要，这些语言大量编码人类指称词的性别，并且过度依赖男性形式，即使性别未指定或无关紧要。众所周知，语言技术反映了这些不平等现象，在翻译成具有广泛性别形态的语言时，会受到男性偏见的影响并延续刻板印象。在这种情况下，性别中立语言可以帮助避免不必要的二元假设。然而，尽管包容性语言研究对于创建更公平的多语言和跨语言技术非常重要，但包容性语言研究仍然很少，并且在当前资源中得不到充分支持。为了弥补这一差距，我们提供了多语言 mGeNTe 数据集。 mGeNTE 源自双语 GeNTE（Piergentili 等人，2023 年），扩展了原始语料库，包括英语-意大利语/德语/西班牙语语言对。由于每对语言都与英语对齐，且目标语言中有性别和中性句子，因此 mGeNTE 可以用于研究自动性别中立翻译 (GNT) 和三种语法性别语言的语言建模。

Title: AutoCBT: An Autonomous Multi-agent Framework for Cognitive Behavioral Therapy in Psychological Counseling

• Authors: Ancheng Xu, Di Yang, Renhao Li, Jingwei Zhu, Minghuan Tan, Min Yang, Wanxin Qiu, Mingchen Ma, Haihong Wu, Bingyu Li, Feng Sha, Chengming Li, Xiping Hu, Qiang Qu, Derek F.Wong, Ruifeng Xu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.09426
• Pdf URL: https://arxiv.org/pdf/2501.09426
• Copy Paste: [[2501.09426]] AutoCBT: An Autonomous Multi-agent Framework for Cognitive Behavioral Therapy in Psychological Counseling(https://arxiv.org/abs/2501.09426)
• Keywords: language model, llm, agent
• Abstract: Traditional in-person psychological counseling remains primarily niche, often chosen by individuals with psychological issues, while online automated counseling offers a potential solution for those hesitant to seek help due to feelings of shame. Cognitive Behavioral Therapy (CBT) is an essential and widely used approach in psychological counseling. The advent of large language models (LLMs) and agent technology enables automatic CBT diagnosis and treatment. However, current LLM-based CBT systems use agents with a fixed structure, limiting their self-optimization capabilities, or providing hollow, unhelpful suggestions due to redundant response patterns. In this work, we utilize Quora-like and YiXinLi single-round consultation models to build a general agent framework that generates high-quality responses for single-turn psychological consultation scenarios. We use a bilingual dataset to evaluate the quality of single-response consultations generated by each framework. Then, we incorporate dynamic routing and supervisory mechanisms inspired by real psychological counseling to construct a CBT-oriented autonomous multi-agent framework, demonstrating its general applicability. Experimental results indicate that AutoCBT can provide higher-quality automated psychological counseling services.
• 摘要：传统的面对面心理咨询仍然主要是小众的，通常由有心理问题的人选择，而在线自动化咨询为那些因羞耻感而犹豫寻求帮助的人提供了潜在的解决方案。认知行为疗法 (CBT) 是心理咨询中一种必不可少且广泛使用的方法。大型语言模型 (LLM) 和代理技术的出现使 CBT 诊断和治疗自动化成为可能。然而，当前基于 LLM 的 CBT 系统使用具有固定结构的代理，限制了它们的自我优化能力，或者由于冗余的响应模式而提供空洞、无用的建议。在这项工作中，我们利用类似 Quora 和 YiXinLi 的单轮咨询模型来构建一个通用代理框架，该框架可为单轮心理咨询场景生成高质量的响应。我们使用双语数据集来评估每个框架生成的单响应咨询的质量。然后，我们结合受真实心理咨询启发的动态路由和监督机制，构建了一个面向 CBT 的自主多代理框架，证明了其普遍适用性。实验结果表明，AutoCBT可以提供更高质量的自动化心理咨询服务。

Title: Solving the unsolvable: Translating case law in Hong Kong

• Authors: King-kui Sin, Xi Xuan, Chunyu Kit, Clara Ho-yan Chan, Honic Ho-kin Ip
• Subjects: cs.CL, cs.AI, cs.LG, cs.MA
• Abstract URL: https://arxiv.org/abs/2501.09444
• Pdf URL: https://arxiv.org/pdf/2501.09444
• Copy Paste: [[2501.09444]] Solving the unsolvable: Translating case law in Hong Kong(https://arxiv.org/abs/2501.09444)
• Keywords: language model, agent
• Abstract: This paper addresses the challenges translating case law under Hong Kong's bilingual legal system. It highlights the initial success of translating all written statutes into Chinese before the 1997 handover, a task mandated by the Basic Law. The effort involved significant collaboration among legal, linguistic, and translation experts, resulting in a comprehensive and culturally appropriate bilingual legal system. However, translating case law remains a significant challenge due to the sheer volume and continuous growth of judicial decisions. The paper critiques the governments and judiciarys sporadic and uncoordinated efforts to translate case law, contrasting it with the thorough approach previously taken for statute translation. Although the government acknowledges the importance of legal bilingualism, it lacks a sustainable strategy for translating case law. The Judiciarys position that translating all judgments is unnecessary, unrealistic, and not cost-effectiveis analyzed and critiqued for its impact on legal transparency and public trust. A proposed solution involves leveraging machine translation technology through a human-machine interactive translation platform, which undergoes two major transitions. Initially based on a neural model, the platform transitions to using a large language model for improved translation accuracy. Furthermore, it evolves from a single-agent system to a multi-agent system, incorporating Translator, Annotator, and Proofreader agents. This multi-agent approach, supported by a grant, aims to facilitate efficient, high-quality translation of judicial judgments by integrating advanced artificial intelligence and continuous feedback mechanisms, thus better meeting the needs of a bilingual legal system.
• 摘要：本文探讨了在香港双语法律体系下翻译案例法所面临的挑战。本文强调了在 1997 年回归之前将所有书面法规翻译成中文的初步成功，这是《基本法》规定的一项任务。这项工作涉及法律、语言和翻译专家之间的大量合作，从而建立了一个全面且文化上适宜的双语法律体系。然而，由于司法判决数量庞大且不断增长，翻译案例法仍然是一项重大挑战。本文批评了政府和司法机构在翻译案例法方面零散且不协调的努力，并将其与以前对法规翻译采取的彻底方法进行了对比。尽管政府承认法律双语的重要性，但它缺乏翻译案例法的可持续战略。司法机构认为翻译所有判决是不必要的、不切实际的、不具成本效益的立场，本文分析和批评了其对法律透明度和公众信任的影响。建议的解决方案是通过人机交互式翻译平台利用机器翻译技术，该平台经历了两次重大转变。该平台最初基于神经模型，后来过渡到使用大型语言模型来提高翻译准确性。此外，它从单代理系统发展为多代理系统，包括翻译、注释和校对代理。这种由资助支持的多代理方法旨在通过整合先进的人工智能和持续反馈机制来促进司法判决的高效、高质量翻译，从而更好地满足双语法律体系的需求。

Title: Exploring the Inquiry-Diagnosis Relationship with Advanced Patient Simulators

• Authors: Zhaocheng Liu, Quan Tu, Wen Ye, Yu Xiao, Zhishou Zhang, Hengfu Cui, Yalun Zhu, Qiang Ju, Shizheng Li, Jian Xie
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.09484
• Pdf URL: https://arxiv.org/pdf/2501.09484
• Copy Paste: [[2501.09484]] Exploring the Inquiry-Diagnosis Relationship with Advanced Patient Simulators(https://arxiv.org/abs/2501.09484)
• Keywords: language model
• Abstract: Online medical consultation (OMC) restricts doctors to gathering patient information solely through inquiries, making the already complex sequential decision-making process of diagnosis even more challenging. Recently, the rapid advancement of large language models has demonstrated a significant potential to transform OMC. However, most studies have primarily focused on improving diagnostic accuracy under conditions of relatively sufficient information, while paying limited attention to the "inquiry" phase of the consultation process. This lack of focus has left the relationship between "inquiry" and "diagnosis" insufficiently explored. In this paper, we first extract real patient interaction strategies from authentic doctor-patient conversations and use these strategies to guide the training of a patient simulator that closely mirrors real-world behavior. By inputting medical records into our patient simulator to simulate patient responses, we conduct extensive experiments to explore the relationship between "inquiry" and "diagnosis" in the consultation process. Experimental results demonstrate that inquiry and diagnosis adhere to the Liebig's law: poor inquiry quality limits the effectiveness of diagnosis, regardless of diagnostic capability, and vice versa. Furthermore, the experiments reveal significant differences in the inquiry performance of various models. To investigate this phenomenon, we categorize the inquiry process into four types: (1) chief complaint inquiry; (2) specification of known symptoms; (3) inquiry about accompanying symptoms; and (4) gathering family or medical history. We analyze the distribution of inquiries across the four types for different models to explore the reasons behind their significant performance differences. We plan to open-source the weights and related code of our patient simulator at this https URL.
• 摘要：在线问诊将医生的问诊方式限制在患者信息收集的单一方式，使得原本就复杂的诊断决策过程变得更加困难。近年来，大型语言模型的快速发展，展现出巨大的变革潜力。然而，大多数研究主要集中在信息相对充足的条件下如何提高诊断准确率，而对问诊过程中的“问诊”环节关注有限。这种关注的缺失导致“问诊”与“诊断”之间的关系没有得到充分探索。在本文中，我们首先从真实的医患对话中提取真实的患者互动策略，并利用这些策略指导训练一个与真实行为高度相似的患者模拟器。通过将病历输入我们的患者模拟器来模拟患者的反应，我们进行了大量的实验，以探索问诊过程中“问诊”与“诊断”之间的关系。实验结果表明，问诊与诊断符合李比希定律：无论诊断能力如何，问诊质量差都会限制诊断的有效性，反之亦然。此外，实验还揭示了不同模型的询问性能存在显著差异。为了研究这一现象，我们将询问过程分为四类：（1）主诉询问；（2）已知症状的说明；（3）伴随症状的询问；（4）收集家庭或病史。我们分析了不同模型在四种类型中的询问分布，以探究它们存在显著性能差异的原因。我们计划在此 https URL 上开源我们的患者模拟器的权重和相关代码。

Title: From Scarcity to Capability: Empowering Fake News Detection in Low-Resource Languages with LLMs

• Authors: Hrithik Majumdar Shibu, Shrestha Datta, Md. Sumon Miah, Nasrullah Sami, Mahruba Sharmin Chowdhury, Md. Saiful Islam
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.09604
• Pdf URL: https://arxiv.org/pdf/2501.09604
• Copy Paste: [[2501.09604]] From Scarcity to Capability: Empowering Fake News Detection in Low-Resource Languages with LLMs(https://arxiv.org/abs/2501.09604)
• Keywords: language model, llm
• Abstract: The rapid spread of fake news presents a significant global challenge, particularly in low-resource languages like Bangla, which lack adequate datasets and detection tools. Although manual fact-checking is accurate, it is expensive and slow to prevent the dissemination of fake news. Addressing this gap, we introduce BanFakeNews-2.0, a robust dataset to enhance Bangla fake news detection. This version includes 11,700 additional, meticulously curated fake news articles validated from credible sources, creating a proportional dataset of 47,000 authentic and 13,000 fake news items across 13 categories. In addition, we created a manually curated independent test set of 460 fake and 540 authentic news items for rigorous evaluation. We invest efforts in collecting fake news from credible sources and manually verified while preserving the linguistic richness. We develop a benchmark system utilizing transformer-based architectures, including fine-tuned Bidirectional Encoder Representations from Transformers variants (F1-87\%) and Large Language Models with Quantized Low-Rank Approximation (F1-89\%), that significantly outperforms traditional methods. BanFakeNews-2.0 offers a valuable resource to advance research and application in fake news detection for low-resourced languages. We publicly release our dataset and model on Github to foster research in this direction.
• 摘要：虚假新闻的迅速传播给全球带来了重大挑战，尤其是对于孟加拉语等资源匮乏的语言，这些语言缺乏足够的数据集和检测工具。尽管手动核实事实准确无误，但要防止虚假新闻的传播，成本高昂且速度缓慢。为了弥补这一差距，我们推出了 BanFakeNews-2.0，这是一个强大的数据集，可增强孟加拉语虚假新闻的检测能力。此版本包含 11,700 篇额外的、经过精心策划的虚假新闻文章，这些文章均来自可靠来源，从而创建了一个包含 47,000 条真实新闻和 13,000 条虚假新闻的比例数据集，涵盖 13 个类别。此外，我们还创建了一个手动策划的独立测试集，其中包含 460 条虚假新闻和 540 条真实新闻，以供严格评估。我们投入精力从可靠来源收集虚假新闻，并在保留语言丰富性的同时进行手动验证。我们开发了一个基于 Transformer 的架构的基准测试系统，包括经过微调的 Transformer 变体双向编码器表示 (F1-87\%) 和具有量化低秩近似的大型语言模型 (F1-89\%)，其性能显著优于传统方法。BanFakeNews-2.0 为推动资源匮乏语言的假新闻检测研究和应用提供了宝贵的资源。我们在 Github 上公开发布了我们的数据集和模型，以促进这方面的研究。

Title: The Heap: A Contamination-Free Multilingual Code Dataset for Evaluating Large Language Models

• Authors: Jonathan Katzy, Razvan Mihai Popescu, Arie van Deursen, Maliheh Izadi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.09653
• Pdf URL: https://arxiv.org/pdf/2501.09653
• Copy Paste: [[2501.09653]] The Heap: A Contamination-Free Multilingual Code Dataset for Evaluating Large Language Models(https://arxiv.org/abs/2501.09653)
• Keywords: language model
• Abstract: The recent rise in the popularity of large language models has spurred the development of extensive code datasets needed to train them. This has left limited code available for collection and use in the downstream investigation of specific behaviors, or evaluation of large language models without suffering from data contamination. To address this problem, we release The Heap, a large multilingual dataset covering 57 programming languages that has been deduplicated with respect to other open datasets of code, enabling researchers to conduct fair evaluations of large language models without significant data cleaning overhead.
• 摘要：大型语言模型最近越来越受欢迎，这刺激了训练它们所需的大量代码数据集的开发。这使得可用于下游特定行为调查或大型语言模型评估的代码数量有限，且不会受到数据污染。为了解决这个问题，我们发布了 The Heap，这是一个涵盖 57 种编程语言的大型多语言数据集，相对于其他开放代码数据集，它已进行重复数据删除，使研究人员能够对大型语言模型进行公平评估，而无需大量的数据清理开销。

Title: Domain Adaptation of Foundation LLMs for e-Commerce

• Authors: Christian Herold, Michael Kozielski, Tala Bazazo, Pavel Petrushkov, Hadi Hashemi, Patrycja Cieplicka, Dominika Basaj, Shahram Khadivi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.09706
• Pdf URL: https://arxiv.org/pdf/2501.09706
• Copy Paste: [[2501.09706]] Domain Adaptation of Foundation LLMs for e-Commerce(https://arxiv.org/abs/2501.09706)
• Keywords: language model, llm
• Abstract: We present the e-Llama models: 8 billion and 70 billion parameter large language models that are adapted towards the e-commerce domain. These models are meant as foundation models with deep knowledge about e-commerce, that form a base for instruction- and fine-tuning. The e-Llama models are obtained by continuously pretraining the Llama 3.1 base models on 1 trillion tokens of domain-specific data. We discuss our approach and motivate our choice of hyperparameters with a series of ablation studies. To quantify how well the models have been adapted to the e-commerce domain, we define and implement a set of multilingual, e-commerce specific evaluation tasks. We show that, when carefully choosing the training setup, the Llama 3.1 models can be adapted towards the new domain without sacrificing significant performance on general domain tasks. We also explore the possibility of merging the adapted model and the base model for a better control of the performance trade-off between domains.
• 摘要：我们介绍了 e-Llama 模型：适用于电子商务领域的 80 亿和 700 亿参数大型语言模型。这些模型旨在作为具有电子商务深度知识的基础模型，为指令和微调奠定基础。e-Llama 模型是通过在 1 万亿个特定领域数据标记上不断预训练 Llama 3.1 基础模型而获得的。我们通过一系列消融研究讨论了我们的方法并激励我们对超参数的选择。为了量化模型对电子商务领域的适应程度，我们定义并实施了一组多语言、电子商务特定的评估任务。我们表明，在仔细选择训练设置时，Llama 3.1 模型可以适应新领域，而不会牺牲一般领域任务的显着性能。我们还探讨了合并改编模型和基础模型的可能性，以更好地控制领域之间的性能权衡。

Title: Comparative Insights from 12 Machine Learning Models in Extracting Economic Ideology from Political Text

• Authors: Jihed Ncib
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.09719
• Pdf URL: https://arxiv.org/pdf/2501.09719
• Copy Paste: [[2501.09719]] Comparative Insights from 12 Machine Learning Models in Extracting Economic Ideology from Political Text(https://arxiv.org/abs/2501.09719)
• Keywords: gpt, prompt
• Abstract: This study conducts a systematic assessment of the capabilities of 12 machine learning models and model variations in detecting economic ideology. As an evaluation benchmark, I use manifesto data spanning six elections in the United Kingdom and pre-annotated by expert and crowd coders. The analysis assesses the performance of several generative, fine-tuned, and zero-shot models at the granular and aggregate levels. The results show that generative models such as GPT-4o and Gemini 1.5 Flash consistently outperform other models against all benchmarks. However, they pose issues of accessibility and resource availability. Fine-tuning yielded competitive performance and offers a reliable alternative through domain-specific optimization. But its dependency on training data severely limits scalability. Zero-shot models consistently face difficulties with identifying signals of economic ideology, often resulting in negative associations with human coding. Using general knowledge for the domain-specific task of ideology scaling proved to be unreliable. Other key findings include considerable within-party variation, fine-tuning benefiting from larger training data, and zero-shot's sensitivity to prompt content. The assessments include the strengths and limitations of each model and derive best-practices for automated analyses of political content.
• 摘要：本研究对 12 种机器学习模型和模型变体在检测经济意识形态方面的能力进行了系统评估。作为评估基准，我使用了涵盖英国六次选举的宣言数据，这些数据由专家和众包编码员预先注释。该分析在粒度和总体层面评估了几种生成模型、微调模型和零样本模型的性能。结果表明，GPT-4o 和 Gemini 1.5 Flash 等生成模型在所有基准测试中的表现始终优于其他模型。然而，它们带来了可访问性和资源可用性问题。微调产生了具有竞争力的性能，并通过特定领域的优化提供了可靠的替代方案。但它对训练数据的依赖严重限制了可扩展性。零样本模型在识别经济意识形态信号方面始终面临困难，往往导致与人类编码产生负面关联。事实证明，使用一般知识来完成特定领域的意识形态扩展任务是不可靠的。其他主要发现包括党内差异显著、得益于更大训练数据的微调以及零样本对提示内容的敏感性。评估包括每个模型的优势和局限性，并得出自动分析政治内容的最佳实践。

Title: Enhancing Lexicon-Based Text Embeddings with Large Language Models

• Authors: Yibin Lei, Tao Shen, Yu Cao, Andrew Yates
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2501.09749
• Pdf URL: https://arxiv.org/pdf/2501.09749
• Copy Paste: [[2501.09749]] Enhancing Lexicon-Based Text Embeddings with Large Language Models(https://arxiv.org/abs/2501.09749)
• Keywords: language model, llm
• Abstract: Recent large language models (LLMs) have demonstrated exceptional performance on general-purpose text embedding tasks. While dense embeddings have dominated related research, we introduce the first Lexicon-based EmbeddiNgS (LENS) leveraging LLMs that achieve competitive performance on these tasks. Regarding the inherent tokenization redundancy issue and unidirectional attention limitations in traditional causal LLMs, LENS consolidates the vocabulary space through token embedding clustering, and investigates bidirectional attention and various pooling strategies. Specifically, LENS simplifies lexicon matching by assigning each dimension to a specific token cluster, where semantically similar tokens are grouped together, and unlocking the full potential of LLMs through bidirectional attention. Extensive experiments demonstrate that LENS outperforms dense embeddings on the Massive Text Embedding Benchmark (MTEB), delivering compact feature representations that match the sizes of dense counterparts. Notably, combining LENSE with dense embeddings achieves state-of-the-art performance on the retrieval subset of MTEB (i.e. BEIR).
• 摘要：最近的大型语言模型 (LLM) 在通用文本嵌入任务上表现出色。虽然密集嵌入在相关研究中占据主导地位，但我们引入了第一个基于词典的嵌入 (LENS)，利用在这些任务上具有竞争力的 LLM。针对传统因果 LLM 中固有的标记化冗余问题和单向注意力限制，LENS 通过标记嵌入聚类来整合词汇空间，并研究双向注意力和各种池化策略。具体来说，LENS 通过将每个维度分配给特定的标记集群来简化词典匹配，其中语义相似的标记被分组在一起，并通过双向注意力释放 LLM 的全部潜力。大量实验表明，LENS 在海量文本嵌入基准 (MTEB) 上的表现优于密集嵌入，提供与密集对应物大小相匹配的紧凑特征表示。值得注意的是，将 LENSE 与密集嵌入相结合可以在 MTEB（即 BEIR）的检索子集上实现最先进的性能。

Title: OmniThink: Expanding Knowledge Boundaries in Machine Writing through Thinking

• Authors: Zekun Xi, Wenbiao Yin, Jizhan Fang, Jialong Wu, Runnan Fang, Ningyu Zhang, Jiang Yong, Pengjun Xie, Fei Huang, Huajun Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.09751
• Pdf URL: https://arxiv.org/pdf/2501.09751
• Copy Paste: [[2501.09751]] OmniThink: Expanding Knowledge Boundaries in Machine Writing through Thinking(https://arxiv.org/abs/2501.09751)
• Keywords: language model, retrieval-augmented generation
• Abstract: Machine writing with large language models often relies on retrieval-augmented generation. However, these approaches remain confined within the boundaries of the model's predefined scope, limiting the generation of content with rich information. Specifically, vanilla-retrieved information tends to lack depth, utility, and suffers from redundancy, which negatively impacts the quality of generated articles, leading to shallow, repetitive, and unoriginal outputs. To address these issues, we propose OmniThink, a machine writing framework that emulates the human-like process of iterative expansion and reflection. The core idea behind OmniThink is to simulate the cognitive behavior of learners as they progressively deepen their knowledge of the topics. Experimental results demonstrate that OmniThink improves the knowledge density of generated articles without compromising metrics such as coherence and depth. Human evaluations and expert feedback further highlight the potential of OmniThink to address real-world challenges in the generation of long-form articles.
• 摘要：使用大型语言模型的机器写作通常依赖于检索增强生成。然而，这些方法仍然局限于模型预定义范围的边界内，限制了具有丰富信息的内容的生成。具体而言，原始检索信息往往缺乏深度、实用性，并且存在冗余，这会对生成的文章质量产生负面影响，导致输出肤浅、重复和非原创。为了解决这些问题，我们提出了 OmniThink，这是一个机器写作框架，可模拟类似人类的迭代扩展和反思过程。OmniThink 背后的核心思想是模拟学习者在逐步加深对主题知识时的认知行为。实验结果表明，OmniThink 提高了生成文章的知识密度，而不会损害连贯性和深度等指标。人工评估和专家反馈进一步凸显了 OmniThink 在解决长篇文章生成中的现实挑战方面的潜力。