2025-01-16

Title: MERaLiON-TextLLM: Cross-Lingual Understanding of Large Language Models in Chinese, Indonesian, Malay, and Singlish

• Authors: Xin Huang, Tarun Kumar Vangani, Minh Duc Pham, Xunlong Zou, Bin Wang, Zhengyuan Liu, Ai Ti Aw
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.08335
• Pdf URL: https://arxiv.org/pdf/2501.08335
• Copy Paste: [[2501.08335]] MERaLiON-TextLLM: Cross-Lingual Understanding of Large Language Models in Chinese, Indonesian, Malay, and Singlish(https://arxiv.org/abs/2501.08335)
• Keywords: language model, llm
• Abstract: Multilingual large language models (MLLMs) have shown impressive capabilities across a variety of languages. However, efficacy can differ greatly between different language families, especially for those with limited linguistic resources. This report presents MERaLiON-TextLLM, a series of open-source language models specifically tailored to improve understanding and generation in Chinese, Indonesian, Malay, and Singlish. The initial released model is built on Llama-3-8B-Base and refined through a meticulously crafted process of continued pre-training and weight merging. Our approach achieves performance improvements across benchmarks in these languages, exceeding the capabilities of the official Llama-3 models. We provide the model checkpoints as a resource to support further research and development in cross-lingual language understanding.
• 摘要：多语言大型语言模型 (MLLM) 已在多种语言中展现出令人印象深刻的能力。然而，不同语系之间的功效可能存在很大差异，尤其是对于那些语言资源有限的语系。本报告介绍了 MERaLiON-TextLLM，这是一系列开源语言模型，专门用于提高对中文、印尼语、马来语和新加坡式英语的理解和生成。最初发布的模型基于 Llama-3-8B-Base 构建，并通过精心设计的持续预训练和权重合并过程进行改进。我们的方法在这些语言的基准测试中实现了性能提升，超越了官方 Llama-3 模型的能力。我们提供模型检查点作为资源，以支持跨语言语言理解的进一步研究和开发。

Title: Ensemble of Large Language Models for Curated Labeling and Rating of Free-text Data

• Authors: Jiaxing Qiu, Dongliang Guo, Papini Natalie, Peace Noelle, Levinson Cheri, Teague R. Henry
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.08413
• Pdf URL: https://arxiv.org/pdf/2501.08413
• Copy Paste: [[2501.08413]] Ensemble of Large Language Models for Curated Labeling and Rating of Free-text Data(https://arxiv.org/abs/2501.08413)
• Keywords: language model, llm
• Abstract: Free-text responses are commonly collected in psychological studies, providing rich qualitative insights that quantitative measures may not capture. Labeling curated topics of research interest in free-text data by multiple trained human coders is typically labor-intensive and time-consuming. Though large language models (LLMs) excel in language processing, LLM-assisted labeling techniques relying on closed-source LLMs cannot be directly applied to free-text data, without explicit consent for external use. In this study, we propose a framework of assembling locally-deployable LLMs to enhance the labeling of predetermined topics in free-text data under privacy constraints. Analogous to annotation by multiple human raters, this framework leverages the heterogeneity of diverse open-source LLMs. The ensemble approach seeks a balance between the agreement and disagreement across LLMs, guided by a relevancy scoring methodology that utilizes embedding distances between topic descriptions and LLMs' reasoning. We evaluated the ensemble approach using both publicly accessible Reddit data from eating disorder related forums, and free-text responses from eating disorder patients, both complemented by human annotations. We found that: (1) there is heterogeneity in the performance of labeling among same-sized LLMs, with some showing low sensitivity but high precision, while others exhibit high sensitivity but low precision. (2) Compared to individual LLMs, the ensemble of LLMs achieved the highest accuracy and optimal precision-sensitivity trade-off in predicting human annotations. (3) The relevancy scores across LLMs showed greater agreement than dichotomous labels, indicating that the relevancy scoring method effectively mitigates the heterogeneity in LLMs' labeling.
• 摘要：心理学研究通常会收集自由文本响应，这提供了定量方法可能无法捕捉到的丰富定性见解。由多名经过培训的人工编码员在自由文本数据中标记精选的研究兴趣主题通常需要大量劳动力和时间。尽管大型语言模型 (LLM) 在语言处理方面表现出色，但依赖于闭源 LLM 的 LLM 辅助标记技术不能直接应用于自由文本数据，除非明确同意外部使用。在本研究中，我们提出了一个组装本地可部署 LLM 的框架，以在隐私约束下增强自由文本数据中预定主题的标记。类似于多个人工评分者的注释，该框架利用了各种开源 LLM 的异质性。集成方法寻求 LLM 之间的一致性和不一致性之间的平衡，由利用主题描述和 LLM 推理之间的嵌入距离的相关性评分方法指导。我们使用来自饮食失调相关论坛的公开 Reddit 数据和饮食失调患者的自由文本回复（均辅以人工注释）对集成方法进行了评估。我们发现：（1）相同规模的 LLM 的标记性能存在异质性，一些 LLM 表现出低灵敏度但高精度，而另一些则表现出高灵敏度但低精度。（2）与单个 LLM 相比，LLM 集成在预测人工注释方面实现了最高的精度和最佳的精度-灵敏度权衡。（3）LLM 之间的相关性得分显示出比二分标记更高的一致性，表明相关性评分方法有效地缓解了 LLM 标记的异质性。

Title: Religious Bias Landscape in Language and Text-to-Image Models: Analysis, Detection, and Debiasing Strategies

• Authors: Ajwad Abrar, Nafisa Tabassum Oeshy, Mohsinul Kabir, Sophia Ananiadou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.08441
• Pdf URL: https://arxiv.org/pdf/2501.08441
• Copy Paste: [[2501.08441]] Religious Bias Landscape in Language and Text-to-Image Models: Analysis, Detection, and Debiasing Strategies(https://arxiv.org/abs/2501.08441)
• Keywords: language model, prompt
• Abstract: Note: This paper includes examples of potentially offensive content related to religious bias, presented solely for academic purposes. The widespread adoption of language models highlights the need for critical examinations of their inherent biases, particularly concerning religion. This study systematically investigates religious bias in both language models and text-to-image generation models, analyzing both open-source and closed-source systems. We construct approximately 400 unique, naturally occurring prompts to probe language models for religious bias across diverse tasks, including mask filling, prompt completion, and image generation. Our experiments reveal concerning instances of underlying stereotypes and biases associated disproportionately with certain religions. Additionally, we explore cross-domain biases, examining how religious bias intersects with demographic factors such as gender, age, and nationality. This study further evaluates the effectiveness of targeted debiasing techniques by employing corrective prompts designed to mitigate the identified biases. Our findings demonstrate that language models continue to exhibit significant biases in both text and image generation tasks, emphasizing the urgent need to develop fairer language models to achieve global acceptability.
• 摘要：注意：本文包含与宗教偏见相关的可能令人反感的内容示例，仅用于学术目的。语言模型的广泛采用凸显了对其固有偏见进行批判性审查的必要性，尤其是关于宗教的偏见。本研究系统地研究了语言模型和文本到图像生成模型中的宗教偏见，分析了开源和闭源系统。我们构建了大约 400 个独特的、自然发生的提示，以探测语言模型在各种任务中的宗教偏见，包括掩码填充、提示完成和图像生成。我们的实验揭示了与某些宗教不成比例相关的潜在刻板印象和偏见的令人担忧的实例。此外，我们还探索了跨领域偏见，研究了宗教偏见如何与性别、年龄和国籍等人口统计因素相交叉。本研究通过采用旨在减轻已识别偏见的纠正提示，进一步评估了有针对性的去偏见技术的有效性。我们的研究结果表明，语言模型在文本和图像生成任务中继续表现出显著的偏见，强调了开发更公平的语言模型以实现全球可接受性的迫切需要。

Title: Large Language Models For Text Classification: Case Study And Comprehensive Review

• Authors: Arina Kostina, Marios D. Dikaiakos, Dimosthenis Stefanidis, George Pallis
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.08457
• Pdf URL: https://arxiv.org/pdf/2501.08457
• Copy Paste: [[2501.08457]] Large Language Models For Text Classification: Case Study And Comprehensive Review(https://arxiv.org/abs/2501.08457)
• Keywords: language model, gpt, llm, prompt
• Abstract: Unlocking the potential of Large Language Models (LLMs) in data classification represents a promising frontier in natural language processing. In this work, we evaluate the performance of different LLMs in comparison with state-of-the-art deep-learning and machine-learning models, in two different classification scenarios: i) the classification of employees' working locations based on job reviews posted online (multiclass classification), and 2) the classification of news articles as fake or not (binary classification). Our analysis encompasses a diverse range of language models differentiating in size, quantization, and architecture. We explore the impact of alternative prompting techniques and evaluate the models based on the weighted F1-score. Also, we examine the trade-off between performance (F1-score) and time (inference response time) for each language model to provide a more nuanced understanding of each model's practical applicability. Our work reveals significant variations in model responses based on the prompting strategies. We find that LLMs, particularly Llama3 and GPT-4, can outperform traditional methods in complex classification tasks, such as multiclass classification, though at the cost of longer inference times. In contrast, simpler ML models offer better performance-to-time trade-offs in simpler binary classification tasks.
• 摘要：释放大型语言模型 (LLM) 在数据分类中的潜力代表着自然语言处理领域一个有希望的前沿。在这项工作中，我们在两种不同的分类场景中评估了不同 LLM 与最先进的深度学习和机器学习模型的性能：i) 根据在线发布的职位评论对员工的工作地点进行分类（多类分类），以及 2) 将新闻文章分类为假的或真假的（二元分类）。我们的分析涵盖了各种语言模型，这些模型在大小、量化和架构上有所不同。我们探索了替代提示技术的影响，并根据加权 F1 分数评估模型。此外，我们研究了每种语言模型的性能（F1 分数）和时间（推理响应时间）之间的权衡，以更细致地了解每种模型的实际适用性。我们的工作揭示了基于提示策略的模型响应存在显著差异。我们发现 LLM（尤其是 Llama3 和 GPT-4）在复杂的分类任务（例如多类分类）中的表现优于传统方法，但推理时间会更长。相比之下，在更简单的二元分类任务中，更简单的 ML 模型可以提供更好的性能与时间权衡。

Title: The Theater Stage as Laboratory: Review of Real-Time Comedy LLM Systems for Live Performance

• Authors: Piotr Wojciech Mirowski, Boyd Branch, Kory Wallace Mathewson
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.08474
• Pdf URL: https://arxiv.org/pdf/2501.08474
• Copy Paste: [[2501.08474]] The Theater Stage as Laboratory: Review of Real-Time Comedy LLM Systems for Live Performance(https://arxiv.org/abs/2501.08474)
• Keywords: llm
• Abstract: In this position paper, we review the eclectic recent history of academic and artistic works involving computational systems for humor generation, and focus specifically on live performance. We make the case that AI comedy should be evaluated in live conditions, in front of audiences sharing either physical or online spaces, and under real-time constraints. We further suggest that improvised comedy is therefore the perfect substrate for deploying and assessing computational humor systems. Using examples of successful AI-infused shows, we demonstrate that live performance raises three sets of challenges for computational humor generation: 1) questions around robotic embodiment, anthropomorphism and competition between humans and machines, 2) questions around comedic timing and the nature of audience interaction, and 3) questions about the human interpretation of seemingly absurd AI-generated humor. We argue that these questions impact the choice of methodologies for evaluating computational humor, as any such method needs to work around the constraints of live audiences and performance spaces. These interrogations also highlight different types of collaborative relationship of human comedians towards AI tools.
• 摘要：在本立场文件中，我们回顾了近年来涉及计算幽默生成系统的学术和艺术作品的不拘一格的历史，并特别关注现场表演。我们认为，人工智能喜剧应该在现场条件下进行评估，在共享物理或在线空间的观众面前，并在实时约束下进行评估。我们进一步指出，即兴喜剧是部署和评估计算幽默系统的完美基础。使用成功的人工智能融合节目的例子，我们证明现场表演为计算幽默生成带来了三组挑战：1）关于机器人化身、拟人化和人机竞争的问题，2）关于喜剧时机和观众互动性质的问题，以及 3）关于人类对看似荒谬的人工智能生成幽默的解读的问题。我们认为这些问题影响了评估计算幽默的方法的选择，因为任何这样的方法都需要解决现场观众和表演空间的限制。这些质问还强调了人类喜剧演员与人工智能工具的不同类型的合作关系。

Title: Quantifying the Importance of Data Alignment in Downstream Model Performance

• Authors: Krrish Chawla, Aryan Sahai, Mario DePavia, Sudharsan Sundar, Brando Miranda
• Subjects: cs.CL, cs.AI, cs.LG, cs.PL
• Abstract URL: https://arxiv.org/abs/2501.08496
• Pdf URL: https://arxiv.org/pdf/2501.08496
• Copy Paste: [[2501.08496]] Quantifying the Importance of Data Alignment in Downstream Model Performance(https://arxiv.org/abs/2501.08496)
• Keywords: language model, llm
• Abstract: Contrary to the conventional emphasis on dataset size, we explore the role of data alignment -- an often overlooked aspect of data quality -- in training capable Large Language Models (LLMs). To do so, we use the Task2Vec-based alignment coefficient, a quantitative measure of the similarity between two datasets, to quantify the impact of alignment between training data and evaluation data on downstream performance. In particular, we conduct controlled \textit{interventional} experiments for two settings: 1. the impact of increased alignment coefficients between various pre-training (pt) against evaluation datasets, and 2. the impact of increased alignment coefficients between domain specific fine-tuning (ft) against domain specific evaluation. The domain specific task we explore is Autoformalization -- the machine translation task between natural language and code for formal verification. In both settings, we find a strong, predictable negative correlation between the alignment coefficient of a model's training and evaluation data and the model's loss/perplexity on the respective downstream task. These findings suggest a re-evaluation of LLM training approaches, demonstrating the relevance of data alignment compared to data quantity, especially in specialized downstream tasks such as Autoformalization.
• 摘要：与传统上对数据集大小的强调相反，我们探索了数据对齐（数据质量中经常被忽视的一个方面）在训练大型语言模型 (LLM) 中的作用。为此，我们使用基于 Task2Vec 的对齐系数（两个数据集之间相似性的定量度量）来量化训练数据和评估数据之间的对齐对下游性能的影响。具体来说，我们针对两种设置进行了受控的 \textit{interventional} 实验：1. 各种预训练 (pt) 对评估数据集之间增加的对齐系数的影响，以及 2. 领域特定微调 (ft) 对领域特定评估之间增加的对齐系数的影响。我们探索的特定领域任务是自动形式化——用于形式验证的自然语言和代码之间的机器翻译任务。在这两种设置中，我们都发现模型的训练和评估数据的对齐系数与模型在相应下游任务上的损失/困惑度之间存在强烈、可预测的负相关性。这些发现表明需要重新评估 LLM 培训方法，证明数据对齐与数据量相比具有相关性，特别是在自动形式化等专门的下游任务中。

Title: Doc-Guided Sent2Sent++: A Sent2Sent++ Agent with Doc-Guided memory for Document-level Machine Translation

• Authors: Jiaxin Guo, Yuanchang Luo, Daimeng Wei, Ling Zhang, Zongyao Li, Hengchao Shang, Zhiqiang Rao, Shaojun Li, Jinlong Yang, Zhanglin Wu, Hao Yang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.08523
• Pdf URL: https://arxiv.org/pdf/2501.08523
• Copy Paste: [[2501.08523]] Doc-Guided Sent2Sent++: A Sent2Sent++ Agent with Doc-Guided memory for Document-level Machine Translation(https://arxiv.org/abs/2501.08523)
• Keywords: language model, llm, agent
• Abstract: The field of artificial intelligence has witnessed significant advancements in natural language processing, largely attributed to the capabilities of Large Language Models (LLMs). These models form the backbone of Agents designed to address long-context dependencies, particularly in Document-level Machine Translation (DocMT). DocMT presents unique challenges, with quality, consistency, and fluency being the key metrics for evaluation. Existing approaches, such as Doc2Doc and Doc2Sent, either omit sentences or compromise fluency. This paper introduces Doc-Guided Sent2Sent++, an Agent that employs an incremental sentence-level forced decoding strategy \textbf{to ensure every sentence is translated while enhancing the fluency of adjacent sentences.} Our Agent leverages a Doc-Guided Memory, focusing solely on the summary and its translation, which we find to be an efficient approach to maintaining consistency. Through extensive testing across multiple languages and domains, we demonstrate that Sent2Sent++ outperforms other methods in terms of quality, consistency, and fluency. The results indicate that, our approach has achieved significant improvements in metrics such as s-COMET, d-COMET, LTCR-$1_f$, and document-level perplexity (d-ppl). The contributions of this paper include a detailed analysis of current DocMT research, the introduction of the Sent2Sent++ decoding method, the Doc-Guided Memory mechanism, and validation of its effectiveness across languages and domains.
• 摘要：人工智能领域在自然语言处理方面取得了重大进步，这主要归功于大型语言模型 (LLM) 的功能。这些模型构成了旨在解决长上下文依赖关系的代理的骨干，尤其是在文档级机器翻译 (DocMT) 中。DocMT 提出了独特的挑战，质量、一致性和流畅度是评估的关键指标。现有方法（如 Doc2Doc 和 Doc2Sent）要么省略句子，要么损害流畅度。本文介绍了 Doc-Guided Sent2Sent++，这是一种采用增量句子级强制解码策略 \textbf{以确保每个句子都得到翻译，同时提高相邻句子的流畅度} 的代理。我们的代理利用 Doc-Guided Memory，只关注摘要及其翻译，我们发现这是一种保持一致性的有效方法。通过对多种语言和领域的广泛测试，我们证明 Sent2Sent++ 在质量、一致性和流畅度方面优于其他方法。结果表明，我们的方法在 s-COMET、d-COMET、LTCR-$1_f$ 和文档级困惑度 (d-ppl) 等指标上取得了显著的提升。本文的贡献包括对当前 DocMT 研究的详细分析、Sent2Sent++ 解码方法的介绍、Doc-Guided Memory 机制以及跨语言和跨领域的有效性验证。

Title: Knowledge prompt chaining for semantic modeling

• Authors: Ning Pei Ding, Jingge Du, Zaiwen Feng
• Subjects: cs.CL, cs.AI, cs.DB
• Abstract URL: https://arxiv.org/abs/2501.08540
• Pdf URL: https://arxiv.org/pdf/2501.08540
• Copy Paste: [[2501.08540]] Knowledge prompt chaining for semantic modeling(https://arxiv.org/abs/2501.08540)
• Keywords: llm, prompt
• Abstract: The task of building semantics for structured data such as CSV, JSON, and XML files is highly relevant in the knowledge representation field. Even though we have a vast of structured data on the internet, mapping them to domain ontologies to build semantics for them is still very challenging as it requires the construction model to understand and learn graph-structured knowledge. Otherwise, the task will require human beings' effort and cost. In this paper, we proposed a novel automatic semantic modeling framework: Knowledge Prompt Chaining. It can serialize the graph-structured knowledge and inject it into the LLMs properly in a Prompt Chaining architecture. Through this knowledge injection and prompting chaining, the model in our framework can learn the structure information and latent space of the graph and generate the semantic labels and semantic graphs following the chains' insturction naturally. Based on experimental results, our method achieves better performance than existing leading techniques, despite using reduced structured input data.
• 摘要：为结构化数据（例如 CSV、JSON 和 XML 文件）构建语义的任务在知识表示领域中非常重要。尽管我们在互联网上拥有大量的结构化数据，但将它们映射到领域本体以为其构建语义仍然非常具有挑战性，因为它需要构建模型来理解和学习图结构知识。否则，这项任务将需要人类的努力和成本。在本文中，我们提出了一种新颖的自动语义建模框架：知识提示链。它可以序列化图结构知识，并在提示链架构中将其正确注入 LLM。通过这种知识注入和提示链，我们框架中的模型可以学习图的结构信息和潜在空间，并按照链的指令自然地生成语义标签和语义图。根据实验结果，尽管使用的结构化输入数据减少，但我们的方法仍比现有的领先技术取得了更好的性能。

Title: Information Entropy Invariance: Enhancing Length Extrapolation in Attention Mechanisms

• Authors: Kewei Li, Yanwen Kong, Yiping Xu, Lan Huang, Ruochi Zhang, Fengfeng Zhou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.08570
• Pdf URL: https://arxiv.org/pdf/2501.08570
• Copy Paste: [[2501.08570]] Information Entropy Invariance: Enhancing Length Extrapolation in Attention Mechanisms(https://arxiv.org/abs/2501.08570)
• Keywords: language model, llm
• Abstract: Improving the length extrapolation capabilities of Large Language Models (LLMs) remains a critical challenge in natural language processing. Many recent efforts have focused on modifying the scaled dot-product attention mechanism, and often introduce scaled temperatures without rigorous theoretical justification. To fill this gap, we introduce a novel approach based on information entropy invariance. We propose two new scaled temperatures to enhance length extrapolation. First, a training-free method InfoScale is designed for dot-product attention, and preserves focus on original tokens during length extrapolation by ensuring information entropy remains consistent. Second, we theoretically analyze the impact of scaling (CosScale) on cosine attention. Experimental data demonstrates that combining InfoScale and CosScale achieves state-of-the-art performance on the GAU-{\alpha} model with a context window extended to 64 times the training length, and outperforms seven existing methods. Our analysis reveals that significantly increasing CosScale approximates windowed attention, and highlights the significance of attention score dilution as a key challenge in long-range context handling. The code and data are available at this https URL.
• 摘要：提高大型语言模型 (LLM) 的长度外推能力仍然是自然语言处理中的一个关键挑战。最近的许多努力都集中在修改缩放点积注意机制上，并且经常在没有严格理论依据的情况下引入缩放温度。为了填补这一空白，我们引入了一种基于信息熵不变性的新方法。我们提出了两个新的缩放温度来增强长度外推。首先，一种无需训练的方法 InfoScale 专为点积注意而设计，并通过确保信息熵保持一致来在长度外推期间保持对原始标记的关注。其次，我们从理论上分析了缩放 (CosScale) 对余弦注意的影响。实验数据表明，结合 InfoScale 和 CosScale 在上下文窗口扩展到训练长度的 64 倍的 GAU-{\alpha} 模型上实现了最先进的性能，并且优于现有的七种方法。我们的分析表明，显着增加 CosScale 近似于窗口注意，并强调了注意力分数稀释作为远程上下文处理的关键挑战的重要性。代码和数据可在此 https URL 上获取。

Title: What Limits LLM-based Human Simulation: LLMs or Our Design?

• Authors: Qian Wang, Jiaying Wu, Zhenheng Tang, Bingqiao Luo, Nuo Chen, Wei Chen, Bingsheng He
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.08579
• Pdf URL: https://arxiv.org/pdf/2501.08579
• Copy Paste: [[2501.08579]] What Limits LLM-based Human Simulation: LLMs or Our Design?(https://arxiv.org/abs/2501.08579)
• Keywords: llm
• Abstract: We argue that advancing LLM-based human simulation requires addressing both LLM's inherent limitations and simulation framework design challenges. Recent studies have revealed significant gaps between LLM-based human simulations and real-world observations, highlighting these dual challenges. To address these gaps, we present a comprehensive analysis of LLM limitations and our design issues, proposing targeted solutions for both aspects. Furthermore, we explore future directions that address both challenges simultaneously, particularly in data collection, LLM generation, and evaluation. To support further research in this field, we provide a curated collection of LLM-based human simulation resources.\footnote{this https URL}
• 摘要：我们认为，推进基于 LLM 的人体模拟需要解决 LLM 固有的局限性和模拟框架设计挑战。最近的研究表明，基于 LLM 的人体模拟与现实世界的观察之间存在显著差距，凸显了这些双重挑战。为了解决这些差距，我们对 LLM 的局限性和我们的设计问题进行了全面分析，并针对这两个方面提出了有针对性的解决方案。此外，我们探索了同时解决这两个挑战的未来方向，特别是在数据收集、LLM 生成和评估方面。为了支持该领域的进一步研究，我们提供了基于 LLM 的人体模拟资源的精选集。\footnote{此 https URL}

Title: LoRS: Efficient Low-Rank Adaptation for Sparse Large Language Model

• Authors: Yuxuan Hu, Jing Zhang, Xiaodong Chen, Zhe Zhao, Cuiping Li, Hong Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.08582
• Pdf URL: https://arxiv.org/pdf/2501.08582
• Copy Paste: [[2501.08582]] LoRS: Efficient Low-Rank Adaptation for Sparse Large Language Model(https://arxiv.org/abs/2501.08582)
• Keywords: language model, llm
• Abstract: Existing low-rank adaptation (LoRA) methods face challenges on sparse large language models (LLMs) due to the inability to maintain sparsity. Recent works introduced methods that maintain sparsity by augmenting LoRA techniques with additional masking mechanisms. Despite these successes, such approaches suffer from an increased memory and computation overhead, which affects efficiency of LoRA methods. In response to this limitation, we introduce LoRS, an innovative method designed to achieve both memory and computation efficiency when fine-tuning sparse LLMs. To mitigate the substantial memory and computation demands associated with preserving sparsity, our approach incorporates strategies of weight recompute and computational graph rearrangement. In addition, we also improve the effectiveness of LoRS through better adapter initialization. These innovations lead to a notable reduction in memory and computation consumption during the fine-tuning phase, all while achieving performance levels that outperform existing LoRA approaches.
• 摘要：现有的低秩自适应 (LoRA) 方法在稀疏大型语言模型 (LLM) 上面临挑战，因为无法保持稀疏性。最近的研究引入了通过用额外的掩码机制增强 LoRA 技术来保持稀疏性的方法。尽管取得了这些成功，但这些方法存在内存和计算开销增加的问题，从而影响了 LoRA 方法的效率。为了解决这一限制，我们引入了 LoRS，这是一种创新方法，旨在在微调稀疏 LLM 时实现内存和计算效率。为了减轻与保持稀疏性相关的大量内存和计算需求，我们的方法采用了权重重新计算和计算图重新排列的策略。此外，我们还通过更好的适配器初始化来提高 LoRS 的有效性。这些创新导致微调阶段的内存和计算消耗显着减少，同时实现优于现有 LoRA 方法的性能水平。

Title: Dynamic Knowledge Integration for Enhanced Vision-Language Reasoning

• Authors: Julian Perry, Surasakdi Siripong, Thanakorn Phonchai
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.08597
• Pdf URL: https://arxiv.org/pdf/2501.08597
• Copy Paste: [[2501.08597]] Dynamic Knowledge Integration for Enhanced Vision-Language Reasoning(https://arxiv.org/abs/2501.08597)
• Keywords: language model
• Abstract: Large Vision-Language Models (LVLMs) have demonstrated impressive capabilities in multimodal tasks, but their performance is often constrained by the lack of external knowledge integration, limiting their ability to handle knowledge-intensive tasks such as visual question answering and reasoning. To address this challenge, we propose a novel method, Adaptive Knowledge-Guided Pretraining for Large Vision-Language Models (AKGP-LVLM), which dynamically incorporates structured and unstructured knowledge into LVLMs during pretraining and fine-tuning. Our approach employs a knowledge encoder to represent external knowledge, a retrieval mechanism to select task-relevant information, and a dynamic adaptor to align multimodal and knowledge representations effectively. We evaluate our method on four benchmark datasets, demonstrating significant performance improvements over state-of-the-art models. Furthermore, human evaluations highlight the superior correctness and relevance of our model's outputs. Extensive analyses confirm the robustness, efficiency, and scalability of AKGP-LVLM, making it a compelling solution for real-world knowledge-intensive tasks.
• 摘要：大型视觉语言模型 (LVLM) 在多模态任务中表现出令人印象深刻的能力，但它们的性能通常受到缺乏外部知识集成的限制，从而限制了它们处理知识密集型任务（例如视觉问答和推理）的能力。为了应对这一挑战，我们提出了一种新方法，即自适应知识引导的大型视觉语言模型预训练 (AKGP-LVLM)，该方法在预训练和微调期间将结构化和非结构化知识动态整合到 LVLM 中。我们的方法采用知识编码器来表示外部知识，采用检索机制来选择与任务相关的信息，以及采用动态适配器来有效地对齐多模态和知识表示。我们在四个基准数据集上评估了我们的方法，结果表明，与最先进的模型相比，我们的性能有显著的提升。此外，人工评估凸显了我们模型输出的卓越正确性和相关性。大量分析证实了 AKGP-LVLM 的稳健性、效率和可扩展性，使其成为现实世界知识密集型任务的有力解决方案。

Title: Assessing the Alignment of FOL Closeness Metrics with Human Judgement

• Authors: Ramya Keerthy Thatikonda, Wray Buntine, Ehsan Shareghi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.08613
• Pdf URL: https://arxiv.org/pdf/2501.08613
• Copy Paste: [[2501.08613]] Assessing the Alignment of FOL Closeness Metrics with Human Judgement(https://arxiv.org/abs/2501.08613)
• Keywords: language model, llm
• Abstract: The recent successful paradigm of solving logical reasoning problems with tool-augmented large language models (LLMs) leverages translation of natural language statements into First-Order Logic~(FOL) and external theorem provers. However, the correctness of FOL statements, comprising operators and text predicates, often goes unverified due to the lack of a reliable evaluation metric for comparing generated and ground-truth FOLs. In this paper, we present a comprehensive study of sensitivity of existing metrics and their alignment with human judgement on FOL evaluation. Using ground-truth FOLs, we carefully designed various perturbations on the ground-truth to assess metric sensitivity. We sample FOL translation candidates for natural language statements and measure the ranking alignment between automatic metrics and human annotators. Our empirical findings highlight oversensitivity in the n-gram metric BLEU for text perturbations, the semantic graph metric Smatch++ for structural perturbations, and FOL metric for operator perturbation. We also observe a closer alignment between BertScore and human judgement. Additionally, we show that combining metrics enhances both alignment and sensitivity compared to using individual metrics.
• 摘要：最近，使用工具增强大型语言模型 (LLM) 解决逻辑推理问题的成功范例利用了将自然语言语句翻译成一阶逻辑 (FOL) 和外部定理证明器。然而，由于缺乏可靠的评估指标来比较生成的 FOL 和真实 FOL，因此 FOL 语句（包括运算符和文本谓词）的正确性通常无法验证。在本文中，我们对现有指标的敏感性及其与 FOL 评估的人类判断的一致性进行了全面研究。使用真实 FOL，我们精心设计了各种对真实事实的扰动来评估指标敏感性。我们对自然语言语句的 FOL 翻译候选样本进行抽样，并测量自动指标和人工注释者之间的排名一致性。我们的实证研究结果强调了 n-gram 指标 BLEU 对文本扰动的过度敏感性，语义图指标 Smatch++ 对结构扰动的过度敏感性，以及 FOL 指标对运算符扰动的过度敏感性。我们还观察到 BertScore 与人类判断之间的一致性更紧密。此外，我们表明，与使用单个指标相比，组合指标可以增强一致性和敏感度。

Title: Disjoint Processing Mechanisms of Hierarchical and Linear Grammars in Large Language Models

• Authors: Aruna Sankaranarayanan, Dylan Hadfield-Menell, Aaron Mueller
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.08618
• Pdf URL: https://arxiv.org/pdf/2501.08618
• Copy Paste: [[2501.08618]] Disjoint Processing Mechanisms of Hierarchical and Linear Grammars in Large Language Models(https://arxiv.org/abs/2501.08618)
• Keywords: language model, llm
• Abstract: All natural languages are structured hierarchically. In humans, this structural restriction is neurologically coded: when two grammars are presented with identical vocabularies, brain areas responsible for language processing are only sensitive to hierarchical grammars. Using large language models (LLMs), we investigate whether such functionally distinct hierarchical processing regions can arise solely from exposure to large-scale language distributions. We generate inputs using English, Italian, Japanese, or nonce words, varying the underlying grammars to conform to either hierarchical or linear/positional rules. Using these grammars, we first observe that language models show distinct behaviors on hierarchical versus linearly structured inputs. Then, we find that the components responsible for processing hierarchical grammars are distinct from those that process linear grammars; we causally verify this in ablation experiments. Finally, we observe that hierarchy-selective components are also active on nonce grammars; this suggests that hierarchy sensitivity is not tied to meaning, nor in-distribution inputs.
• 摘要：所有自然语言都是分层结构的。在人类中，这种结构限制是神经编码的：当两种语法具有相同的词汇时，负责语言处理的大脑区域仅对分层语法敏感。使用大型语言模型 (LLM)，我们研究这种功能上不同的分层处理区域是否仅仅通过接触大规模语言分布就能产生。我们使用英语、意大利语、日语或随机单词生成输入，改变底层语法以符合分层或线性/位置规则。使用这些语法，我们首先观察到语言模型对分层结构输入和线性结构输入表现出不同的行为。然后，我们发现负责处理分层语法的组件与处理线性语法的组件不同；我们在消融实验中对此进行了因果验证。最后，我们观察到层次选择性组件在随机语法上也很活跃；这表明层次敏感性与含义无关，也与分布输入无关。

Title: Reassessing the Role of Chain-of-Thought in Sentiment Analysis: Insights and Limitations

• Authors: Kaiyuan Zheng, Qinghua Zhao, Lei Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.08641
• Pdf URL: https://arxiv.org/pdf/2501.08641
• Copy Paste: [[2501.08641]] Reassessing the Role of Chain-of-Thought in Sentiment Analysis: Insights and Limitations(https://arxiv.org/abs/2501.08641)
• Keywords: language model, prompt, chain-of-thought
• Abstract: The relationship between language and thought remains an unresolved philosophical issue. Existing viewpoints can be broadly categorized into two schools: one asserting their independence, and another arguing that language constrains thought. In the context of large language models, this debate raises a crucial question: Does a language model's grasp of semantic meaning depend on thought processes? To explore this issue, we investigate whether reasoning techniques can facilitate semantic understanding. Specifically, we conceptualize thought as reasoning, employ chain-of-thought prompting as a reasoning technique, and examine its impact on sentiment analysis tasks. The experiments show that chain-of-thought has a minimal impact on sentiment analysis tasks. Both the standard and chain-of-thought prompts focus on aspect terms rather than sentiment in the generated content. Furthermore, counterfactual experiments reveal that the model's handling of sentiment tasks primarily depends on information from demonstrations. The experimental results support the first viewpoint.
• 摘要：语言与思维的关系仍然是一个尚未解决的哲学问题。现有观点大致可分为两派：一派主张语言与思维的独立性，另一派则认为语言限制思维。在大型语言模型的背景下，这场争论提出了一个关键问题：语言模型对语义的理解是否取决于思维过程？为了探索这个问题，我们研究了推理技术是否可以促进语义理解。具体来说，我们将思维概念化为推理，采用思维链提示作为推理技术，并研究其对情感分析任务的影响。实验表明，思维链对情感分析任务的影响微乎其微。标准提示和思维链提示都关注生成内容中的方面术语而不是情感。此外，反事实实验表明，该模型对情感任务的处理主要依赖于演示中的信息。实验结果支持第一种观点。

Title: MAGNET: Augmenting Generative Decoders with Representation Learning and Infilling Capabilities

• Authors: Savya Khosla, Kushal Kafle, Simon Jenni, Handong Zhao, John Collomosse, Jing Shi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.08648
• Pdf URL: https://arxiv.org/pdf/2501.08648
• Copy Paste: [[2501.08648]] MAGNET: Augmenting Generative Decoders with Representation Learning and Infilling Capabilities(https://arxiv.org/abs/2501.08648)
• Keywords: language model, llm
• Abstract: While originally designed for unidirectional generative modeling, decoder-only large language models (LLMs) are increasingly being adapted for bidirectional modeling. However, unidirectional and bidirectional models are typically trained separately with distinct objectives (generation and representation learning, respectively). This separation overlooks the opportunity for developing a more versatile language model and for these objectives to complement each other. In this work, we introduce MAGNET, an adaptation of decoder-only LLMs that enhances their ability to generate robust representations and infill missing text spans, while preserving their knowledge and text generation capabilities. MAGNET employs three self-supervised training objectives and introduces an attention mechanism that combines bidirectional and causal attention, enabling unified training across all objectives. Our results demonstrate that LLMs adapted with MAGNET (1) surpass strong text encoders on token-level and sentence-level representation learning tasks, (2) generate contextually appropriate text infills by leveraging future context, (3) retain the ability for open-ended text generation without exhibiting repetition problem, and (4) preserve the knowledge gained by the LLM during pretraining.
• 摘要：虽然最初设计用于单向生成建模，但仅解码器的大型语言模型 (LLM) 越来越多地被改编用于双向建模。然而，单向和双向模型通常分别训练，具有不同的目标（分别是生成和表示学习）。这种分离忽略了开发更通用的语言模型和使这些目标相互补充的机会。在这项工作中，我们引入了 MAGNET，这是仅解码器 LLM 的改编版，它增强了它们生成稳健表示和填充缺失文本跨度的能力，同时保留了它们的知识和文本生成能力。MAGNET 采用三个自监督训练目标，并引入了一种结合双向和因果注意力的注意力机制，从而实现了跨所有目标的统一训练。我们的结果表明，采用 MAGNET 的 LLM（1）在标记级和句子级表示学习任务上超越了强文本编码器，（2）利用未来上下文生成适合上下文的文本填充，（3）保留开放式文本生成的能力，而不会出现重复问题，（4）保留 LLM 在预训练期间获得的知识。

Title: The Inherent Limits of Pretrained LLMs: The Unexpected Convergence of Instruction Tuning and In-Context Learning Capabilities

• Authors: Irina Bigoulaeva, Harish Tayyar Madabushi, Iryna Gurevych
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.08716
• Pdf URL: https://arxiv.org/pdf/2501.08716
• Copy Paste: [[2501.08716]] The Inherent Limits of Pretrained LLMs: The Unexpected Convergence of Instruction Tuning and In-Context Learning Capabilities(https://arxiv.org/abs/2501.08716)
• Keywords: language model, llm, prompt
• Abstract: Large Language Models (LLMs), trained on extensive web-scale corpora, have demonstrated remarkable abilities across diverse tasks, especially as they are scaled up. Nevertheless, even state-of-the-art models struggle in certain cases, sometimes failing at problems solvable by young children, indicating that traditional notions of task complexity are insufficient for explaining LLM capabilities. However, exploring LLM capabilities is complicated by the fact that most widely-used models are also "instruction-tuned" to respond appropriately to prompts. With the goal of disentangling the factors influencing LLM performance, we investigate whether instruction-tuned models possess fundamentally different capabilities from base models that are prompted using in-context examples. Through extensive experiments across various model families, scales and task types, which included instruction tuning 90 different LLMs, we demonstrate that the performance of instruction-tuned models is significantly correlated with the in-context performance of their base counterparts. By clarifying what instruction-tuning contributes, we extend prior research into in-context learning, which suggests that base models use priors from pretraining data to solve tasks. Specifically, we extend this understanding to instruction-tuned models, suggesting that their pretraining data similarly sets a limiting boundary on the tasks they can solve, with the added influence of the instruction-tuning dataset.
• 摘要：大型语言模型 (LLM) 在广泛的网络规模语料库上进行训练，已在各种任务中表现出非凡的能力，尤其是在扩大规模时。然而，即使是最先进的模型在某些情况下也会遇到困难，有时无法解决幼儿可以解决的问题，这表明传统的任务复杂性概念不足以解释 LLM 的能力。然而，探索 LLM 的能力很复杂，因为大多数广泛使用的模型也经过“指令调整”，可以对提示做出适当的反应。为了理清影响 LLM 性能的因素，我们研究了指令调整模型是否具有与使用上下文示例提示的基础模型根本不同的能力。通过对各种模型系列、规模和任务类型进行大量实验，其中包括对 90 种不同的 LLM 进行指令调整，我们证明指令调整模型的性能与其基础模型的上下文性能显着相关。通过阐明指令调整的作用，我们将先前的研究扩展到情境学习，这表明基础模型使用预训练数据的先验来解决任务。具体来说，我们将这种理解扩展到指令调整模型，表明它们的预训练数据同样为它们可以解决的任务设置了限制边界，并增加了指令调整数据集的影响。

Title: Expanding Vietnamese SentiWordNet to Improve Performance of Vietnamese Sentiment Analysis Models

• Authors: Hong-Viet Tran, Van-Tan Bui, Lam-Quan Tran
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.08758
• Pdf URL: https://arxiv.org/pdf/2501.08758
• Copy Paste: [[2501.08758]] Expanding Vietnamese SentiWordNet to Improve Performance of Vietnamese Sentiment Analysis Models(https://arxiv.org/abs/2501.08758)
• Keywords: language model
• Abstract: Sentiment analysis is one of the most crucial tasks in Natural Language Processing (NLP), involving the training of machine learning models to classify text based on the polarity of opinions. Pre-trained Language Models (PLMs) can be applied to downstream tasks through fine-tuning, eliminating the need to train the model from scratch. Specifically, PLMs have been employed for Sentiment Analysis, a process that involves detecting, analyzing, and extracting the polarity of text sentiments. Numerous models have been proposed to address this task, with pre-trained PhoBERT-V2 models standing out as the state-of-the-art language models for Vietnamese. The PhoBERT-V2 pre-training approach is based on RoBERTa, optimizing the BERT pre-training method for more robust performance. In this paper, we introduce a novel approach that combines PhoBERT-V2 and SentiWordnet for Sentiment Analysis of Vietnamese reviews. Our proposed model utilizes PhoBERT-V2 for Vietnamese, offering a robust optimization for the prominent BERT model in the context of Vietnamese language, and leverages SentiWordNet, a lexical resource explicitly designed to support sentiment classification applications. Experimental results on the VLSP 2016 and AIVIVN 2019 datasets demonstrate that our sentiment analysis system has achieved excellent performance in comparison to other models.
• 摘要：情感分析是自然语言处理 (NLP) 中最重要的任务之一，涉及训练机器学习模型以根据意见的极性对文本进行分类。预训练语言模型 (PLM) 可以通过微调应用于下游任务，无需从头开始训练模型。具体来说，PLM 已用于情感分析，该过程涉及检测、分析和提取文本情感的极性。已经提出了许多模型来解决此任务，其中预训练的 PhoBERT-V2 模型脱颖而出，成为越南语最先进的语言模型。PhoBERT-V2 预训练方法基于 RoBERTa，优化了 BERT 预训练方法以获得更稳健的性能。在本文中，我们介绍了一种结合 PhoBERT-V2 和 SentiWordnet 的新方法，用于越南语评论的情感分析。我们提出的模型利用了越南语的 PhoBERT-V2，为越南语背景下的著名 BERT 模型提供了强大的优化，并利用了 SentiWordNet，这是一种专门为支持情感分类应用而设计的词汇资源。在 VLSP 2016 和 AIVIVN 2019 数据集上的实验结果表明，与其他模型相比，我们的情感分析系统取得了出色的表现。

Title: Enhanced Large Language Models for Effective Screening of Depression and Anxiety

• Authors: June M. Liu, Mengxia Gao, Sahand Sabour, Zhuang Chen, Minlie Huang, Tatia M.C. Lee
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.08769
• Pdf URL: https://arxiv.org/pdf/2501.08769
• Copy Paste: [[2501.08769]] Enhanced Large Language Models for Effective Screening of Depression and Anxiety(https://arxiv.org/abs/2501.08769)
• Keywords: language model, gpt, llm
• Abstract: Depressive and anxiety disorders are widespread, necessitating timely identification and management. Recent advances in Large Language Models (LLMs) offer potential solutions, yet high costs and ethical concerns about training data remain challenges. This paper introduces a pipeline for synthesizing clinical interviews, resulting in 1,157 interactive dialogues (PsyInterview), and presents EmoScan, an LLM-based emotional disorder screening system. EmoScan distinguishes between coarse (e.g., anxiety or depressive disorders) and fine disorders (e.g., major depressive disorders) and conducts high-quality interviews. Evaluations showed that EmoScan exceeded the performance of base models and other LLMs like GPT-4 in screening emotional disorders (F1-score=0.7467). It also delivers superior explanations (BERTScore=0.9408) and demonstrates robust generalizability (F1-score of 0.67 on an external dataset). Furthermore, EmoScan outperforms baselines in interviewing skills, as validated by automated ratings and human evaluations. This work highlights the importance of scalable data-generative pipelines for developing effective mental health LLM tools.
• 摘要：抑郁和焦虑症十分普遍，需要及时发现和管理。大型语言模型 (LLM) 的最新进展提供了潜在的解决方案，但高成本和对训练数据的道德担忧仍然是挑战。本文介绍了一种合成临床访谈的流程，产生了 1,157 个交互式对话 (PsyInterview)，并介绍了基于 LLM 的情绪障碍筛查系统 EmoScan。EmoScan 区分粗大障碍（例如焦虑症或抑郁症）和细小障碍（例如重度抑郁症），并进行高质量的访谈。评估表明，EmoScan 在筛查情绪障碍方面的表现超过了基础模型和其他 LLM（如 GPT-4）（F1 分数 = 0.7467）。它还提供了出色的解释（BERTScore = 0.9408）并表现出强大的通用性（外部数据集上的 F1 分数为 0.67）。此外，EmoScan 在面试技巧方面的表现优于基准，这已通过自动评分和人工评估得到验证。这项研究凸显了可扩展数据生成流程对于开发有效的心理健康 LLM 工具的重要性。

Title: ToMATO: Verbalizing the Mental States of Role-Playing LLMs for Benchmarking Theory of Mind

• Authors: Kazutoshi Shinoda, Nobukatsu Hojo, Kyosuke Nishida, Saki Mizuno, Keita Suzuki, Ryo Masumura, Hiroaki Sugiyama, Kuniko Saito
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.08838
• Pdf URL: https://arxiv.org/pdf/2501.08838
• Copy Paste: [[2501.08838]] ToMATO: Verbalizing the Mental States of Role-Playing LLMs for Benchmarking Theory of Mind(https://arxiv.org/abs/2501.08838)
• Keywords: gpt, llm, prompt
• Abstract: Existing Theory of Mind (ToM) benchmarks diverge from real-world scenarios in three aspects: 1) they assess a limited range of mental states such as beliefs, 2) false beliefs are not comprehensively explored, and 3) the diverse personality traits of characters are overlooked. To address these challenges, we introduce ToMATO, a new ToM benchmark formulated as multiple-choice QA over conversations. ToMATO is generated via LLM-LLM conversations featuring information asymmetry. By employing a prompting method that requires role-playing LLMs to verbalize their thoughts before each utterance, we capture both first- and second-order mental states across five categories: belief, intention, desire, emotion, and knowledge. These verbalized thoughts serve as answers to questions designed to assess the mental states of characters within conversations. Furthermore, the information asymmetry introduced by hiding thoughts from others induces the generation of false beliefs about various mental states. Assigning distinct personality traits to LLMs further diversifies both utterances and thoughts. ToMATO consists of 5.4k questions, 753 conversations, and 15 personality trait patterns. Our analysis shows that this dataset construction approach frequently generates false beliefs due to the information asymmetry between role-playing LLMs, and effectively reflects diverse personalities. We evaluate nine LLMs on ToMATO and find that even GPT-4o mini lags behind human performance, especially in understanding false beliefs, and lacks robustness to various personality traits.
• 摘要：现有的心智理论 (ToM) 基准与现实世界场景在三个方面存在差异：1）它们评估的心理状态（例如信念）范围有限，2）错误信念没有得到全面探索，3）人物的多样化性格特征被忽视。为了应对这些挑战，我们引入了 ToMATO，这是一种新的 ToM 基准，其形式为对话中的多项选择问答。ToMATO 是通过具有信息不对称特征的 LLM-LLM 对话生成的。通过采用一种提示方法，要求角色扮演的 LLM 在每次说话之前口头表达他们的想法，我们捕捉了五个类别的一阶和二阶心理状态：信念、意图、愿望、情感和知识。这些口头表达的想法可作为旨在评估对话中人物心理状态的问题的答案。此外，向他人隐藏想法而引入的信息不对称会导致对各种心理状态产生错误信念。为 LLM 分配不同的性格特征进一步使话语和想法多样化。 ToMATO 包含 5.4k 个问题、753 个对话和 15 种性格特征模式。我们的分析表明，这种数据集构建方法由于角色扮演法学硕士之间的信息不对称而经常产生错误信念，并有效地反映了多样化的性格。我们在 ToMATO 上评估了九个法学硕士，发现即使是 GPT-4o mini 也落后于人类的表现，尤其是在理解错误信念方面，并且对各种性格特征缺乏鲁棒性。

Title: GenAI Content Detection Task 3: Cross-Domain Machine-Generated Text Detection Challenge

• Authors: Liam Dugan, Andrew Zhu, Firoj Alam, Preslav Nakov, Marianna Apidianaki, Chris Callison-Burch
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.08913
• Pdf URL: https://arxiv.org/pdf/2501.08913
• Copy Paste: [[2501.08913]] GenAI Content Detection Task 3: Cross-Domain Machine-Generated Text Detection Challenge(https://arxiv.org/abs/2501.08913)
• Keywords: language model, llm
• Abstract: Recently there have been many shared tasks targeting the detection of generated text from Large Language Models (LLMs). However, these shared tasks tend to focus either on cases where text is limited to one particular domain or cases where text can be from many domains, some of which may not be seen during test time. In this shared task, using the newly released RAID benchmark, we aim to answer whether or not models can detect generated text from a large, yet fixed, number of domains and LLMs, all of which are seen during training. Over the course of three months, our task was attempted by 9 teams with 23 detector submissions. We find that multiple participants were able to obtain accuracies of over 99% on machine-generated text from RAID while maintaining a 5% False Positive Rate -- suggesting that detectors are able to robustly detect text from many domains and models simultaneously. We discuss potential interpretations of this result and provide directions for future research.
• 摘要：最近，有许多共享任务针对检测大型语言模型 (LLM) 生成的文本。然而，这些共享任务往往侧重于文本仅限于一个特定域的情况，或者文本可能来自许多域的情况，其中一些可能在测试期间看不到。在这个共享任务中，使用新发布的 RAID 基准，我们旨在回答模型是否可以检测来自大量但固定数量的域和 LLM 的生成文本，所有这些都在训练期间看到。在三个月的时间里，9 支队伍尝试了我们的任务，提交了 23 个检测器。我们发现，多个参与者能够获得来自 RAID 的机器生成文本超过 99% 的准确率，同时保持 5% 的误报率——这表​​明检测器能够同时稳健地检测来自许多域和模型的文本。我们讨论了这一结果的潜在解释，并为未来的研究提供了方向。

Title: Applying General Turn-taking Models to Conversational Human-Robot Interaction

• Authors: Gabriel Skantze, Bahar Irfan
• Subjects: cs.CL, cs.RO
• Abstract URL: https://arxiv.org/abs/2501.08946
• Pdf URL: https://arxiv.org/pdf/2501.08946
• Copy Paste: [[2501.08946]] Applying General Turn-taking Models to Conversational Human-Robot Interaction(https://arxiv.org/abs/2501.08946)
• Keywords: language model, gpt
• Abstract: Turn-taking is a fundamental aspect of conversation, but current Human-Robot Interaction (HRI) systems often rely on simplistic, silence-based models, leading to unnatural pauses and interruptions. This paper investigates, for the first time, the application of general turn-taking models, specifically TurnGPT and Voice Activity Projection (VAP), to improve conversational dynamics in HRI. These models are trained on human-human dialogue data using self-supervised learning objectives, without requiring domain-specific fine-tuning. We propose methods for using these models in tandem to predict when a robot should begin preparing responses, take turns, and handle potential interruptions. We evaluated the proposed system in a within-subject study against a traditional baseline system, using the Furhat robot with 39 adults in a conversational setting, in combination with a large language model for autonomous response generation. The results show that participants significantly prefer the proposed system, and it significantly reduces response delays and interruptions.
• 摘要：轮流发言是对话的一个基本方面，但目前的人机交互 (HRI) 系统通常依赖于过于简单的、基于沉默的模型，导致不自然的停顿和中断。本文首次研究了通用轮流发言模型（特别是 TurnGPT 和语音活动投影 (VAP)）在改善 HRI 中的对话动态方面的应用。这些模型使用自监督学习目标在人与人对话数据上进行训练，无需进行特定领域的微调。我们提出了结合使用这些模型来预测机器人何时应该开始准备响应、轮流发言和处理潜在中断的方法。我们在一项受试者内研究中将所提出的系统与传统基线系统进行了评估，在对​​话环境中使用 Furhat 机器人与 39 名成年人，结合大型语言模型进行自主响应生成。结果表明，参与者明显更喜欢所提出的系统，并且它显著减少了响应延迟和中断。

Title: Learning to Extract Cross-Domain Aspects and Understanding Sentiments Using Large Language Models

• Authors: Karukriti Kaushik Ghosh, Chiranjib Sur
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.08974
• Pdf URL: https://arxiv.org/pdf/2501.08974
• Copy Paste: [[2501.08974]] Learning to Extract Cross-Domain Aspects and Understanding Sentiments Using Large Language Models(https://arxiv.org/abs/2501.08974)
• Keywords: language model, llm
• Abstract: Aspect-based sentiment analysis (ASBA) is a refined approach to sentiment analysis that aims to extract and classify sentiments based on specific aspects or features of a product, service, or entity. Unlike traditional sentiment analysis, which assigns a general sentiment score to entire reviews or texts, ABSA focuses on breaking down the text into individual components or aspects (e.g., quality, price, service) and evaluating the sentiment towards each. This allows for a more granular level of understanding of customer opinions, enabling businesses to pinpoint specific areas of strength and improvement. The process involves several key steps, including aspect extraction, sentiment classification, and aspect-level sentiment aggregation for a review paragraph or any other form that the users have provided. ABSA has significant applications in areas such as product reviews, social media monitoring, customer feedback analysis, and market research. By leveraging techniques from natural language processing (NLP) and machine learning, ABSA facilitates the extraction of valuable insights, enabling companies to make data-driven decisions that enhance customer satisfaction and optimize offerings. As ABSA evolves, it holds the potential to greatly improve personalized customer experiences by providing a deeper understanding of sentiment across various product aspects. In this work, we have analyzed the strength of LLMs for a complete cross-domain aspect-based sentiment analysis with the aim of defining the framework for certain products and using it for other similar situations. We argue that it is possible to that at an effectiveness of 92\% accuracy for the Aspect Based Sentiment Analysis dataset of SemEval-2015 Task 12.
• 摘要：基于方面的情绪分析 (ASBA) 是一种改进的情绪分析方法，旨在根据产品、服务或实体的特定方面或特征提取和分类情绪。与为整个评论或文本分配一般情绪分数的传统情绪分析不同，ABSA 专注于将文本分解为单个组件或方面（例如质量、价格、服务）并评估对每个方面的情绪。这允许更细致地了解客户意见，使企业能够确定特定的优势和改进领域。该过程涉及几个关键步骤，包括方面提取、情绪分类和评论段落或用户提供的任何其他形式的方面级情绪聚合。ABSA 在产品评论、社交媒体监控、客户反馈分析和市场研究等领域具有重要应用。通过利用自然语言处理 (NLP) 和机器学习技术，ABSA 有助于提取有价值的见解，使公司能够做出数据驱动的决策，从而提高客户满意度并优化产品。随着 ABSA 的发展，它有可能通过提供对各个产品方面情绪的更深入理解，大大改善个性化客户体验。在这项工作中，我们分析了 LLM 在完整的跨领域基于方面的情绪分析方面的强度，目的是为某些产品定义框架并将其用于其他类似情况。我们认为，对于 SemEval-2015 任务 12 的基于方面的情绪分析数据集，以 92% 的准确率实现这一点是可能的。

Title: Personality Modeling for Persuasion of Misinformation using AI Agent

• Authors: Qianmin Lou, Wentao Xu
• Subjects: cs.CL, cs.AI, cs.GT
• Abstract URL: https://arxiv.org/abs/2501.08985
• Pdf URL: https://arxiv.org/pdf/2501.08985
• Copy Paste: [[2501.08985]] Personality Modeling for Persuasion of Misinformation using AI Agent(https://arxiv.org/abs/2501.08985)
• Keywords: agent
• Abstract: The proliferation of misinformation on social media platforms has highlighted the need to understand how individual personality traits influence susceptibility to and propagation of misinformation. This study employs an innovative agent-based modeling approach to investigate the relationship between personality traits and misinformation dynamics. Using six AI agents embodying different dimensions of the Big Five personality traits (Extraversion, Agreeableness, and Neuroticism), we simulated interactions across six diverse misinformation topics. The experiment, implemented through the AgentScope framework using the GLM-4-Flash model, generated 90 unique interactions, revealing complex patterns in how personality combinations affect persuasion and resistance to misinformation. Our findings demonstrate that analytical and critical personality traits enhance effectiveness in evidence-based discussions, while non-aggressive persuasion strategies show unexpected success in misinformation correction. Notably, agents with critical traits achieved a 59.4% success rate in HIV-related misinformation discussions, while those employing non-aggressive approaches maintained consistent persuasion rates above 40% across different personality combinations. The study also revealed a non-transitive pattern in persuasion effectiveness, challenging conventional assumptions about personality-based influence. These results provide crucial insights for developing personality-aware interventions in digital environments and suggest that effective misinformation countermeasures should prioritize emotional connection and trust-building over confrontational approaches. The findings contribute to both theoretical understanding of personality-misinformation dynamics and practical strategies for combating misinformation in social media contexts.
• 摘要：社交媒体平台上虚假信息的泛滥凸显了人们需要了解个人性格特征如何影响对虚假信息的敏感性和传播。本研究采用创新的基于代理的建模方法来研究性格特征与虚假信息动态之间的关系。我们使用六个体现大五人格特质（外向性、宜人性和神经质）不同维度的人工智能代理，模拟了六个不同虚假信息主题的互动。该实验通过使用 GLM-4-Flash 模型的 AgentScope 框架实施，生成了 90 个独特的互动，揭示了性格组合如何影响对虚假信息的说服和抵制的复杂模式。我们的研究结果表明，分析性和批判性性格特征可提高基于证据的讨论的有效性，而非攻击性说服策略在纠正虚假信息方面表现出意想不到的成功。值得注意的是，具有批判性特征的代理在与 HIV 相关的虚假信息讨论中取得了 59.4% 的成功率，而采用非攻击性方法的代理在不同性格组合中保持了 40% 以上的一致说服率。该研究还揭示了说服有效性的非传递模式，挑战了关于基于个性的影响力的传统假设。这些结果为在数字环境中开发个性意识干预措施提供了重要见解，并表明有效的错误信息对策应优先考虑情感联系和信任建立，而不是对抗方法。这些发现有助于从理论上理解个性-错误信息动态，并有助于制定在社交媒体环境中打击错误信息的实用策略。

Title: Aegis2.0: A Diverse AI Safety Dataset and Risks Taxonomy for Alignment of LLM Guardrails

• Authors: Shaona Ghosh, Prasoon Varshney, Makesh Narsimhan Sreedhar, Aishwarya Padmakumar, Traian Rebedea, Jibin Rajan Varghese, Christopher Parisien
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.09004
• Pdf URL: https://arxiv.org/pdf/2501.09004
• Copy Paste: [[2501.09004]] Aegis2.0: A Diverse AI Safety Dataset and Risks Taxonomy for Alignment of LLM Guardrails(https://arxiv.org/abs/2501.09004)
• Keywords: language model, llm
• Abstract: As Large Language Models (LLMs) and generative AI become increasingly widespread, concerns about content safety have grown in parallel. Currently, there is a clear lack of high-quality, human-annotated datasets that address the full spectrum of LLM-related safety risks and are usable for commercial applications. To bridge this gap, we propose a comprehensive and adaptable taxonomy for categorizing safety risks, structured into 12 top-level hazard categories with an extension to 9 fine-grained subcategories. This taxonomy is designed to meet the diverse requirements of downstream users, offering more granular and flexible tools for managing various risk types. Using a hybrid data generation pipeline that combines human annotations with a multi-LLM "jury" system to assess the safety of responses, we obtain Aegis 2.0, a carefully curated collection of 34,248 samples of human-LLM interactions, annotated according to our proposed taxonomy. To validate its effectiveness, we demonstrate that several lightweight models, trained using parameter-efficient techniques on Aegis 2.0, achieve performance competitive with leading safety models fully fine-tuned on much larger, non-commercial datasets. In addition, we introduce a novel training blend that combines safety with topic following this http URL approach enhances the adaptability of guard models, enabling them to generalize to new risk categories defined during inference. We plan to open-source Aegis 2.0 data and models to the research community to aid in the safety guardrailing of LLMs.
• 摘要：随着大型语言模型 (LLM) 和生成式 AI 的普及，人们对内容安全性的担忧也日益增加。目前，显然缺乏高质量的人工注释数据集，这些数据集可以解决与 LLM 相关的各种安全风险，并可用于商业应用。为了弥补这一差距，我们提出了一种全面且适应性强的分类法来对安全风险进行分类，该分类法分为 12 个顶级危险类别，并扩展到 9 个细粒度子类别。该分类法旨在满足下游用户的不同需求，提供更精细、更灵活的工具来管理各种风险类型。使用将人工注释与多 LLM“陪审团”系统相结合的混合数据生成管道来评估响应的安全性，我们获得了 Aegis 2.0，这是精心策划的 34,248 个人-LLM 交互样本集合，根据我们提出的分类法进行了注释。为了验证其有效性，我们证明了使用参数高效技术在 Aegis 2.0 上训练的几个轻量级模型的性能可与在更大的非商业数据集上完全微调的领先安全模型相媲美。此外，我们引入了一种将安全性与主题相结合的新型训练组合，此 http URL 方法增强了防护模型的适应性，使它们能够推广到推理期间定义的新风险类别。我们计划向研究界开源 Aegis 2.0 数据和模型，以帮助 LLM 的安全防护。