2025-01-08

Title: Toward Inclusive Educational AI: Auditing Frontier LLMs through a Multiplexity Lens

• Authors: Abdullah Mushtaq, Muhammad Rafay Naeem, Muhammad Imran Taj, Ibrahim Ghaznavi, Junaid Qadir
• Subjects: cs.CL, cs.AI, cs.CY, cs.LG, cs.MA
• Abstract URL: https://arxiv.org/abs/2501.03259
• Pdf URL: https://arxiv.org/pdf/2501.03259
• Copy Paste: [[2501.03259]] Toward Inclusive Educational AI: Auditing Frontier LLMs through a Multiplexity Lens(https://arxiv.org/abs/2501.03259)
• Keywords: language model, gpt, llm, prompt, agent
• Abstract: As large language models (LLMs) like GPT-4 and Llama 3 become integral to educational contexts, concerns are mounting over the cultural biases, power imbalances, and ethical limitations embedded within these technologies. Though generative AI tools aim to enhance learning experiences, they often reflect values rooted in Western, Educated, Industrialized, Rich, and Democratic (WEIRD) cultural paradigms, potentially sidelining diverse global perspectives. This paper proposes a framework to assess and mitigate cultural bias within LLMs through the lens of applied multiplexity. Multiplexity, inspired by Senturk et al. and rooted in Islamic and other wisdom traditions, emphasizes the coexistence of diverse cultural viewpoints, supporting a multi-layered epistemology that integrates both empirical sciences and normative values. Our analysis reveals that LLMs frequently exhibit cultural polarization, with biases appearing in both overt responses and subtle contextual cues. To address inherent biases and incorporate multiplexity in LLMs, we propose two strategies: \textit{Contextually-Implemented Multiplex LLMs}, which embed multiplex principles directly into the system prompt, influencing LLM outputs at a foundational level and independent of individual prompts, and \textit{Multi-Agent System (MAS)-Implemented Multiplex LLMs}, where multiple LLM agents, each representing distinct cultural viewpoints, collaboratively generate a balanced, synthesized response. Our findings demonstrate that as mitigation strategies evolve from contextual prompting to MAS-implementation, cultural inclusivity markedly improves, evidenced by a significant rise in the Perspectives Distribution Score (PDS) and a PDS Entropy increase from 3.25\% at baseline to 98\% with the MAS-Implemented Multiplex LLMs. Sentiment analysis further shows a shift towards positive sentiment across cultures,...
• 摘要：随着 GPT-4 和 Llama 3 等大型语言模型 (LLM) 成为教育环境不可或缺的一部分，人们越来越担心这些技术中存在的文化偏见、权力不平衡和道德限制。尽管生成式人工智能工具旨在增强学习体验，但它们往往反映出植根于西方、受过教育、工业化、富裕和民主 (WEIRD) 文化范式的价值观，可能会忽视多样化的全球视角。本文提出了一个框架，通过应用多元性的视角来评估和减轻 LLM 中的文化偏见。多元性受到 Senturk 等人的启发，植根于伊斯兰和其他智慧传统，强调不同文化观点的共存，支持一种融合实证科学和规范价值观的多层次认识论。我们的分析表明，LLM 经常表现出文化两极分化，偏见既出现在明显的反应中，也出现在微妙的背景线索中。为了解决固有偏见并将多元性纳入 LLM，我们提出了两种策略：\textit{上下文实现的多元 LLM}，它将多元原则直接嵌入系统提示中，在基础层面影响 LLM 输出，并且独立于单个提示；\textit{多智能体系统 (MAS) 实现的多元 LLM}，其中多个 LLM 智能体（每个代表不同的文化观点）协作生成平衡的综合响应。我们的研究结果表明，随着缓解策略从上下文提示发展到 MAS 实现，文化包容性显著提高，这表现为观点分布分数 (PDS) 显著上升，PDS 熵从基线的 3.25\% 增加到 MAS 实现的多元 LLM 的 98\%。情绪分析进一步表明，跨文化情绪转向积极情绪，...

Title: REINFORCE++: A Simple and Efficient Approach for Aligning Large Language Models

• Authors: Jian Hu
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.03262
• Pdf URL: https://arxiv.org/pdf/2501.03262
• Copy Paste: [[2501.03262]] REINFORCE++: A Simple and Efficient Approach for Aligning Large Language Models(https://arxiv.org/abs/2501.03262)
• Keywords: language model
• Abstract: Reinforcement Learning from Human Feedback (RLHF) has emerged as a critical approach for aligning large language models with human preferences, witnessing rapid algorithmic evolution through methods such as Proximal Policy Optimization (PPO), Direct Preference Optimization (DPO), REINFORCE Leave One-Out (RLOO), ReMax, and Group Relative Policy Optimization (GRPO). We present REINFORCE++, an enhanced variant of the classical REINFORCE algorithm that incorporates key optimization techniques from PPO while eliminating the need for a critic network. REINFORCE++ achieves three primary objectives: (1) simplicity (2) enhanced training stability, and (3) reduced computational overhead. Through extensive empirical evaluation, we demonstrate that REINFORCE++ exhibits superior stability compared to GRPO and achieves greater computational efficiency than PPO while maintaining comparable performance. The implementation is available at \url{this https URL}.
• 摘要：基于人类反馈的强化学习 (RLHF) 已成为将大型语言模型与人类偏好相结合的重要方法，并通过近端策略优化 (PPO)、直接偏好优化 (DPO)、REINFORCE 留一法 (RLOO)、ReMax 和组相对策略优化 (GRPO) 等方法见证了算法的快速发展。我们提出了 REINFORCE++，这是经典 REINFORCE 算法的增强版本，它结合了 PPO 的关键优化技术，同时消除了对评论家网络的需​​求。REINFORCE++ 实现了三个主要目标：(1) 简单 (2) 增强训练稳定性，以及 (3) 减少计算开销。通过大量的实证评估，我们证明 REINFORCE++ 比 GRPO 表现出更好的稳定性，并且在保持可比性能的同时实现了比 PPO 更高的计算效率。实现可在 \url{this https URL} 处获得。

Title: LLM Content Moderation and User Satisfaction: Evidence from Response Refusals in Chatbot Arena

• Authors: Stefan Pasch
• Subjects: cs.CL, cs.AI, cs.CY, cs.HC, cs.SI
• Abstract URL: https://arxiv.org/abs/2501.03266
• Pdf URL: https://arxiv.org/pdf/2501.03266
• Copy Paste: [[2501.03266]] LLM Content Moderation and User Satisfaction: Evidence from Response Refusals in Chatbot Arena(https://arxiv.org/abs/2501.03266)
• Keywords: llm, prompt, chat
• Abstract: LLM safety and ethical alignment are widely discussed, but the impact of content moderation on user satisfaction remains underexplored. To address this, we analyze nearly 50,000 Chatbot Arena response-pairs using a novel fine-tuned RoBERTa model, that we trained on hand-labeled data to disentangle refusals due to ethical concerns from other refusals due to technical disabilities or lack of information. Our findings reveal a significant refusal penalty on content moderation, with users choosing ethical-based refusals roughly one-fourth as often as their preferred LLM response compared to standard responses. However, the context and phrasing play critical roles: refusals on highly sensitive prompts, such as illegal content, achieve higher win rates than less sensitive ethical concerns, and longer responses closely aligned with the prompt perform better. These results emphasize the need for nuanced moderation strategies that balance ethical safeguards with user satisfaction. Moreover, we find that the refusal penalty is notably lower in evaluations using the LLM-as-a-Judge method, highlighting discrepancies between user and automated assessments.
• 摘要：LLM 安全性和道德一致性被广泛讨论，但内容审核对用户满意度的影响仍未得到充分探索。为了解决这个问题，我们使用一种新颖的微调 RoBERTa 模型分析了近 50,000 个 Chatbot Arena 响应对，我们用手工标记的数据训练该模型，以区分由于道德问题而拒绝与由于技术障碍或缺乏信息而拒绝。我们的研究结果显示，内容审核的拒绝惩罚很严重，与标准响应相比，用户选择基于道德的拒绝的频率大约是他们首选 LLM 响应的四分之一。然而，上下文和措辞起着至关重要的作用：对高度敏感的提示（例如非法内容）的拒绝比不太敏感的道德问题获得更高的胜率，而与提示紧密相关的较长的响应表现更好。这些结果强调需要细致入微的审核策略来平衡道德保障和用户满意度。此外，我们发现，使用 LLM-as-a-Judge 方法进行的评估中拒绝惩罚明显较低，突出了用户和自动评估之间的差异。

Title: ComMer: a Framework for Compressing and Merging User Data for Personalization

• Authors: Yoel Zeldes, Amir Zait, Ilia Labzovsky, Danny Karmon, Efrat Farkash
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.03276
• Pdf URL: https://arxiv.org/pdf/2501.03276
• Copy Paste: [[2501.03276]] ComMer: a Framework for Compressing and Merging User Data for Personalization(https://arxiv.org/abs/2501.03276)
• Keywords: language model, llm, prompt
• Abstract: Large Language Models (LLMs) excel at a wide range of tasks, but adapting them to new data, particularly for personalized applications, poses significant challenges due to resource and computational constraints. Existing methods either rely on exposing fresh data to the model through the prompt, which is limited by context size and computationally expensive at inference time, or fine-tuning, which incurs substantial training and update costs. In this paper, we introduce ComMer - Compress and Merge - a novel framework that efficiently personalizes LLMs by compressing users' documents into compact representations, which are then merged and fed into a frozen LLM. We evaluate ComMer on two types of personalization tasks - personalized skill learning, using the tweet paraphrasing dataset and the personalized news headline generation dataset from the LaMP benchmark, and knowledge-intensive, using the PerLTQA dataset. Our experiments demonstrate that in constrained inference budget scenarios ComMer achieves superior quality in skill learning tasks, while highlighting limitations in knowledge-intensive settings due to the loss of detailed information. These results offer insights into trade-offs and potential optimizations in multi-document compression for personalization.
• 摘要：大型语言模型 (LLM) 擅长处理各种任务，但由于资源和计算方面的限制，将它们适应新数据（尤其是个性化应用）带来了重大挑战。现有方法要么依赖于通过提示将新数据暴露给模型，这受到上下文大小的限制，并且在推理时计算成本高昂，要么依赖于微调，这会产生大量的训练和更新成本。在本文中，我们介绍了 ComMer - 压缩和合并 - 一种新颖的框架，它通过将用户的文档压缩为紧凑的表示形式来有效地个性化 LLM，然后将其合并并输入到冻结的 LLM 中。我们在两种类型的个性化任务上评估 ComMer - 个性化技能学习（使用来自 LaMP 基准的推文释义数据集和个性化新闻标题生成数据集）和知识密集型（使用 PerLTQA 数据集）。我们的实验表明，在受限的推理预算场景中，ComMer 在技能学习任务中实现了卓越的质量，同时由于详细信息的丢失而突出了知识密集型环境中的局限性。这些结果为个性化多文档压缩中的权衡和潜在优化提供了见解。

Title: HonkaiChat: Companions from Anime that feel alive!

• Authors: Yueze Liu, Yichi Zhang, Shaan Om Patel, Zhaoyang Zhu, Shilong Guo
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.03277
• Pdf URL: https://arxiv.org/pdf/2501.03277
• Copy Paste: [[2501.03277]] HonkaiChat: Companions from Anime that feel alive!(https://arxiv.org/abs/2501.03277)
• Keywords: gpt, hallucination, prompt, chat, agent
• Abstract: Modern conversational agents, including anime-themed chatbots, are frequently reactive and personality-driven but fail to capture the dynamic nature of human interactions. We propose an event-driven dialogue framework to address these limitations by embedding dynamic events in conversation prompts and fine-tuning models on character-specific data. Evaluations on GPT-4 and comparisons with industry-leading baselines demonstrate that event-driven prompts significantly improve conversational engagement and naturalness while reducing hallucinations. This paper explores the application of this approach in creating lifelike chatbot interactions within the context of Honkai: Star Rail, showcasing the potential for dynamic event-based systems to transform role-playing and interactive dialogue.
• 摘要：现代对话代理（包括动漫主题的聊天机器人）通常具有反应性和个性驱动性，但无法捕捉人类互动的动态性质。我们提出了一个事件驱动的对话框架来解决这些限制，方法是在对话提示中嵌入动态事件并根据特定于角色的数据微调模型。对 GPT-4 的评估和与行业领先基线的比较表明，事件驱动的提示可显著提高对话参与度和自然度，同时减少幻觉。本文探讨了这种方法在《崩坏：星轨》背景下创建逼真的聊天机器人交互中的应用，展示了基于动态事件的系统改变角色扮演和交互式对话的潜力。

Title: ADePT: Adaptive Decomposed Prompt Tuning for Parameter-Efficient Fine-tuning

• Authors: Pengwei Tang, Xiaolin Hu, Yong Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.03291
• Pdf URL: https://arxiv.org/pdf/2501.03291
• Copy Paste: [[2501.03291]] ADePT: Adaptive Decomposed Prompt Tuning for Parameter-Efficient Fine-tuning(https://arxiv.org/abs/2501.03291)
• Keywords: language model, prompt
• Abstract: Prompt Tuning (PT) enables the adaptation of Pre-trained Large Language Models (PLMs) to downstream tasks by optimizing a small amount of soft virtual tokens, which are prepended to the input token embeddings. Recently, Decomposed Prompt Tuning (DePT) has demonstrated superior adaptation capabilities by decomposing the soft prompt into a shorter soft prompt and a pair of low-rank matrices. The product of the pair of low-rank matrices is added to the input token embeddings to offset them. Additionally, DePT achieves faster inference compared to PT due to the shorter soft prompt. However, in this paper, we find that the position-based token embedding offsets of DePT restricts its ability to generalize across diverse model inputs, and that the shared embedding offsets across many token embeddings result in sub-optimization. To tackle these issues, we introduce \textbf{A}daptive \textbf{De}composed \textbf{P}rompt \textbf{T}uning (ADePT), which is composed of a short soft prompt and a shallow token-shared feed-forward neural network. ADePT utilizes the token-shared feed-forward neural network to learn the embedding offsets for each token, enabling adaptive embedding offsets that vary according to the model input and better optimization of token embedding offsets. This enables ADePT to achieve superior adaptation performance without requiring more inference time or additional trainable parameters compared to vanilla PT and its variants. In comprehensive experiments across 23 natural language processing (NLP) tasks and 4 typical PLMs of different scales, we show that ADePT consistently surpasses the leading parameter-efficient fine-tuning (PEFT) methods, and even outperforms the full fine-tuning baseline in certain scenarios. Code is available at \url{this https URL}.
• 摘要：提示调优 (PT) 通过优化少量软虚拟标记（这些标记被添加到输入标记嵌入中）使预训练大型语言模型 (PLM) 能够适应下游任务。最近，分解提示调优 (DePT) 通过将软提示分解为较短的软提示和一对低秩矩阵，展示了卓越的适应能力。低秩矩阵对的乘积被添加到输入标记嵌入中以对其进行偏移。此外，由于软提示较短，DePT 与 PT 相比实现了更快的推理。然而，在本文中，我们发现 DePT 基于位置的标记嵌入偏移限制了其跨不同模型输入进行泛化的能力，并且跨许多标记嵌入共享的嵌入偏移导致了次优化。为了解决这些问题，我们引入了 \textbf{A}daptive \textbf{De}composed \textbf{P}rompt \textbf{T}uning (ADePT)，它由一个短软提示和一个浅层 token-shared 前馈神经网络组成。ADePT 利用 token-shared 前馈神经网络来学习每个 token 的嵌入偏移量，从而实现根据模型输入而变化的自适应嵌入偏移量，并更好地优化 token 嵌入偏移量。与 vanilla PT 及其变体相比，这使 ADePT 能够实现卓越的自适应性能，而无需更多推理时间或额外的可训练参数。在 23 个自然语言处理 (NLP) 任务和 4 个不同规模的典型 PLM 的综合实验中，我们表明 ADePT 始终超越领先的参数高效微调 (PEFT) 方法，甚至在某些情况下优于完整微调基线。代码可在 \url{此 https URL} 处获得。

Title: Analyzing Bias in Swiss Federal Supreme Court Judgments Using Facebook's Holistic Bias Dataset: Implications for Language Model Training

• Authors: Sabine Wehnert, Muhammet Ertas, Ernesto William De Luca
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.03324
• Pdf URL: https://arxiv.org/pdf/2501.03324
• Copy Paste: [[2501.03324]] Analyzing Bias in Swiss Federal Supreme Court Judgments Using Facebook's Holistic Bias Dataset: Implications for Language Model Training(https://arxiv.org/abs/2501.03324)
• Keywords: language model
• Abstract: Natural Language Processing (NLP) is vital for computers to process and respond accurately to human language. However, biases in training data can introduce unfairness, especially in predicting legal judgment. This study focuses on analyzing biases within the Swiss Judgment Prediction Dataset (SJP-Dataset). Our aim is to ensure unbiased factual descriptions essential for fair decision making by NLP models in legal contexts. We analyze the dataset using social bias descriptors from the Holistic Bias dataset and employ advanced NLP techniques, including attention visualization, to explore the impact of dispreferred descriptors on model predictions. The study identifies biases and examines their influence on model behavior. Challenges include dataset imbalance and token limits affecting model performance.
• 摘要：自然语言处理 (NLP) 对于计算机处理和准确响应人类语言至关重要。然而，训练数据中的偏见可能会导致不公平，尤其是在预测法律判决时。本研究重点分析瑞士判决预测数据集 (SJP-Dataset) 中的偏见。我们的目标是确保 NLP 模型在法律背景下做出公平决策所必需的无偏见事实描述。我们使用来自整体偏见数据集的社会偏见描述符分析数据集，并采用注意力可视化等高级 NLP 技术来探索不喜欢的描述符对模型预测的影响。本研究确定了偏见并检查了它们对模型行为的影响。挑战包括数据集不平衡和影响模型性能的标记限制。

Title: Advanced Machine Learning Techniques for Social Support Detection on Social Media

• Authors: Olga Kolesnikova, Moein Shahiki Tash, Zahra Ahani, Ameeta Agrawal, Raul Monroy, Grigori Sidorov
• Subjects: cs.CL, cs.AI, cs.HC, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.03370
• Pdf URL: https://arxiv.org/pdf/2501.03370
• Copy Paste: [[2501.03370]] Advanced Machine Learning Techniques for Social Support Detection on Social Media(https://arxiv.org/abs/2501.03370)
• Keywords: gpt
• Abstract: The widespread use of social media highlights the need to understand its impact, particularly the role of online social support. This study uses a dataset focused on online social support, which includes binary and multiclass classifications of social support content on social media. The classification of social support is divided into three tasks. The first task focuses on distinguishing between supportive and non-supportive. The second task aims to identify whether the support is directed toward an individual or a group. The third task categorizes the specific type of social support, grouping it into categories such as Nation, LGBTQ, Black people, Women, Religion, and Other (if it does not fit into the previously mentioned categories). To address data imbalances in these tasks, we employed K-means clustering for balancing the dataset and compared the results with the original unbalanced data. Using advanced machine learning techniques, including transformers and zero-shot learning approaches with GPT3, GPT4, and GPT4-o, we predict social support levels in various contexts. The effectiveness of the dataset is evaluated using baseline models across different learning approaches, with transformer-based methods demonstrating superior performance. Additionally, we achieved a 0.4\% increase in the macro F1 score for the second task and a 0.7\% increase for the third task, compared to previous work utilizing traditional machine learning with psycholinguistic and unigram-based TF-IDF values.
• 摘要：社交媒体的广泛使用凸显了了解其影响的必要性，尤其是在线社交支持的作用。本研究使用了一个专注于在线社交支持的数据集，其中包括社交媒体上社交支持内容的二分类和多分类。社交支持的分类分为三个任务。第一项任务侧重于区分支持性和不支持性。第二项任务旨在确定支持是针对个人还是群体。第三项任务对特定类型的社交支持进行分类，将其分为国家、LGBTQ、黑人、女性、宗教和其他（如果不符合前面提到的类别）等类别。为了解决这些任务中的数据不平衡问题，我们采用 K 均值聚类来平衡数据集，并将结果与​​原始不平衡数据进行比较。使用先进的机器学习技术，包括 GPT3、GPT4 和 GPT4-o 的 transformer 和零样本学习方法，我们可以预测各种情况下的社会支持水平。使用不同学习方法的基线模型评估数据集的有效性，其中基于 transformer 的方法表现出卓越的性能。此外，与以前使用具有心理语言学和基于单词的 TF-IDF 值的传统机器学习的工作相比，我们在第二项任务中实现了 0.4% 的宏 F1 分数提高，在第三项任务中实现了 0.7% 的宏 F1 分数提高。

Title: BoundingDocs: a Unified Dataset for Document Question Answering with Spatial Annotations

• Authors: Simone Giovannini, Fabio Coppini, Andrea Gemelli, Simone Marinai
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.03403
• Pdf URL: https://arxiv.org/pdf/2501.03403
• Copy Paste: [[2501.03403]] BoundingDocs: a Unified Dataset for Document Question Answering with Spatial Annotations(https://arxiv.org/abs/2501.03403)
• Keywords: language model, prompt
• Abstract: We present a unified dataset for document Question-Answering (QA), which is obtained combining several public datasets related to Document AI and visually rich document understanding (VRDU). Our main contribution is twofold: on the one hand we reformulate existing Document AI tasks, such as Information Extraction (IE), into a Question-Answering task, making it a suitable resource for training and evaluating Large Language Models; on the other hand, we release the OCR of all the documents and include the exact position of the answer to be found in the document image as a bounding box. Using this dataset, we explore the impact of different prompting techniques (that might include bounding box information) on the performance of open-weight models, identifying the most effective approaches for document comprehension.
• 摘要：我们为文档问答 (QA) 提供了一个统一的数据集，该数据集结合了与文档 AI 和视觉丰富的文档理解 (VRDU) 相关的几个公共数据集。我们的主要贡献有两个方面：一方面，我们将现有的文档 AI 任务（例如信息提取 (IE)）重新表述为问答任务，使其成为训练和评估大型语言模型的合适资源；另一方面，我们发布了所有文档的 OCR，并将文档图像中要找到的答案的确切位置作为边界框包含在内。使用此数据集，我们探索了不同的提示技术（可能包括边界框信息）对开放权重模型性能的影响，从而确定了最有效的文档理解方法。

Title: Finding A Voice: Evaluating African American Dialect Generation for Chatbot Technology

• Authors: Sarah E. Finch, Ellie S. Paek, Sejung Kwon, Ikseon Choi, Jessica Wells, Rasheeta Chandler, Jinho D. Choi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.03441
• Pdf URL: https://arxiv.org/pdf/2501.03441
• Copy Paste: [[2501.03441]] Finding A Voice: Evaluating African American Dialect Generation for Chatbot Technology(https://arxiv.org/abs/2501.03441)
• Keywords: language model, gpt, llm, chat
• Abstract: As chatbots become increasingly integrated into everyday tasks, designing systems that accommodate diverse user populations is crucial for fostering trust, engagement, and inclusivity. This study investigates the ability of contemporary Large Language Models (LLMs) to generate African American Vernacular English (AAVE) and evaluates the impact of AAVE usage on user experiences in chatbot applications. We analyze the performance of three LLM families (Llama, GPT, and Claude) in producing AAVE-like utterances at varying dialect intensities and assess user preferences across multiple domains, including healthcare and education. Despite LLMs' proficiency in generating AAVE-like language, findings indicate that AAVE-speaking users prefer Standard American English (SAE) chatbots, with higher levels of AAVE correlating with lower ratings for a variety of characteristics, including chatbot trustworthiness and role appropriateness. These results highlight the complexities of creating inclusive AI systems and underscore the need for further exploration of diversity to enhance human-computer interactions.
• 摘要：随着聊天机器人越来越多地融入日常任务，设计适合不同用户群体的系统对于培养信任、参与度和包容性至关重要。本研究调查了当代大型语言模型 (LLM) 生成非裔美国人本土英语 (AAVE) 的能力，并评估了 AAVE 的使用对聊天机器人应用程序中用户体验的影响。我们分析了三个 LLM 系列 (Llama、GPT 和 Claude) 在不同方言强度下产生类似 AAVE 的话语的表现，并评估了医疗保健和教育等多个领域的用户偏好。尽管 LLM 在生成类似 AAVE 的语言方面很熟练，但研究结果表明，讲 AAVE 的用户更喜欢标准美式英语 (SAE) 聊天机器人，AAVE 水平越高，各种特征的评分就越低，包括聊天机器人的可信度和角色适当性。这些结果凸显了创建包容性 AI 系统的复杂性，并强调需要进一步探索多样性以增强人机交互。

Title: Text to Band Gap: Pre-trained Language Models as Encoders for Semiconductor Band Gap Prediction

• Authors: Ying-Ting Yeh, Janghoon Ock, Amir Barati Farimani
• Subjects: cs.CL, cond-mat.mtrl-sci
• Abstract URL: https://arxiv.org/abs/2501.03456
• Pdf URL: https://arxiv.org/pdf/2501.03456
• Copy Paste: [[2501.03456]] Text to Band Gap: Pre-trained Language Models as Encoders for Semiconductor Band Gap Prediction(https://arxiv.org/abs/2501.03456)
• Keywords: language model, gpt, chat
• Abstract: In this study, we explore the use of a transformer-based language model as an encoder to predict the band gaps of semiconductor materials directly from their text descriptions. Quantum chemistry simulations, including Density Functional Theory (DFT), are computationally intensive and time-consuming, which limits their practicality for high-throughput material screening, particularly for complex systems. Shallow machine learning (ML) models, while effective, often require extensive data preprocessing to convert non-numerical material properties into numerical inputs. In contrast, our approach leverages textual data directly, bypassing the need for complex feature engineering. We generate material descriptions in two formats: formatted strings combining features and natural language text generated using the ChatGPT API. We demonstrate that the RoBERTa model, pre-trained on natural language processing tasks, performs effectively as an encoder for prediction tasks. With minimal fine-tuning, it achieves a mean absolute error (MAE) of approximately 0.33 eV, performing better than shallow machine learning models such as Support Vector Regression, Random Forest, and XGBoost. Even when only the linear regression head is trained while keeping the RoBERTa encoder layers frozen, the accuracy remains nearly identical to that of the fully trained model. This demonstrates that the pre-trained RoBERTa encoder is highly adaptable for processing domain-specific text related to material properties, such as the band gap, significantly reducing the need for extensive retraining. This study highlights the potential of transformer-based language models to serve as efficient and versatile encoders for semiconductor materials property prediction tasks.
• 摘要：在本研究中，我们探索了使用基于变换器的语言模型作为编码器，直接从文本描述中预测半导体材料的带隙。量子化学模拟，包括密度泛函理论 (DFT)，计算量大且耗时，这限制了它们在高通量材料筛选中的实用性，尤其是对于复杂系统。浅层机器学习 (ML) 模型虽然有效，但通常需要大量数据预处理才能将非数值材料属性转换为数值输入。相比之下，我们的方法直接利用文本数据，绕过了复杂的特征工程。我们以两种格式生成材料描述：结合特征的格式化字符串和使用 ChatGPT API 生成的自然语言文本。我们证明，在自然语言处理任务上预先训练的 RoBERTa 模型可以有效地作为预测任务的编码器。通过最少的微调，它实现了约 0.33 eV 的平均绝对误差 (MAE)，表现优于支持向量回归、随机森林和 XGBoost 等浅层机器学习模型。即使只训练线性回归头，同时保持 RoBERTa 编码器层冻结，其准确率也几乎与完全训练的模型相同。这表明预训练的 RoBERTa 编码器具有高度适应性，可以处理与材料特性（例如带隙）相关的领域特定文本，从而大大减少了大量重新训练的需要。这项研究强调了基于 Transformer 的语言模型作为半导体材料特性预测任务的高效通用编码器的潜力。

Title: ISSR: Iterative Selection with Self-Review for Vocabulary Test Distractor Generation

• Authors: Yu-Cheng Liu, An-Zi Yen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.03462
• Pdf URL: https://arxiv.org/pdf/2501.03462
• Copy Paste: [[2501.03462]] ISSR: Iterative Selection with Self-Review for Vocabulary Test Distractor Generation(https://arxiv.org/abs/2501.03462)
• Keywords: language model, llm
• Abstract: Vocabulary acquisition is essential to second language learning, as it underpins all core language skills. Accurate vocabulary assessment is particularly important in standardized exams, where test items evaluate learners' comprehension and contextual use of words. Previous research has explored methods for generating distractors to aid in the design of English vocabulary tests. However, current approaches often rely on lexical databases or predefined rules, and frequently produce distractors that risk invalidating the question by introducing multiple correct options. In this study, we focus on English vocabulary questions from Taiwan's university entrance exams. We analyze student response distributions to gain insights into the characteristics of these test items and provide a reference for future research. Additionally, we identify key limitations in how large language models (LLMs) support teachers in generating distractors for vocabulary test design. To address these challenges, we propose the iterative selection with self-review (ISSR) framework, which makes use of a novel LLM-based self-review mechanism to ensure that the distractors remain valid while offering diverse options. Experimental results show that ISSR achieves promising performance in generating plausible distractors, and the self-review mechanism effectively filters out distractors that could invalidate the question.
• 摘要：词汇习得是第二语言学习的关键，因为它是所有核心语言技能的基础。准确的词汇评估在标准化考试中尤为重要，因为考试题目评估学习者对单词的理解和语境使用。先前的研究探索了生成干扰项以帮助设计英语词汇测试的方法。然而，目前的方法通常依赖于词汇数据库或预定义规则，并且经常产生干扰项，这些干扰项通过引入多个正确选项而有可能使问题无效。在本研究中，我们关注台湾大学入学考试中的英语词汇问题。我们分析学生的回答分布，以深入了解这些测试项目的特点并为未来的研究提供参考。此外，我们确定了大型语言模型 (LLM) 如何支持教师为词汇测试设计生成干扰项的关键限制。为了应对这些挑战，我们提出了迭代选择和自我审查 (ISSR) 框架，该框架利用一种新颖的基于 LLM 的自我审查机制来确保干扰项在提供多样化选项的同时保持有效。实验结果表明，ISSR 在生成合理的干扰项方面取得了良好的表现，并且自我审查机制有效地过滤掉了可能使问题无效的干扰项。

Title: MTRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems

• Authors: Yannis Katsis, Sara Rosenthal, Kshitij Fadnis, Chulaka Gunasekara, Young-Suk Lee, Lucian Popa, Vraj Shah, Huaiyu Zhu, Danish Contractor, Marina Danilevsky
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.03468
• Pdf URL: https://arxiv.org/pdf/2501.03468
• Copy Paste: [[2501.03468]] MTRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems(https://arxiv.org/abs/2501.03468)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: Retrieval-augmented generation (RAG) has recently become a very popular task for Large Language Models (LLMs). Evaluating them on multi-turn RAG conversations, where the system is asked to generate a response to a question in the context of a preceding conversation is an important and often overlooked task with several additional challenges. We present MTRAG: an end-to-end human-generated multi-turn RAG benchmark that reflects several real-world properties across diverse dimensions for evaluating the full RAG pipeline. MTRAG contains 110 conversations averaging 7.7 turns each across four domains for a total of 842 tasks. We also explore automation paths via synthetic data and LLM-as-a-Judge evaluation. Our human and automatic evaluations show that even state-of-the-art LLM RAG systems struggle on MTRAG. We demonstrate the need for strong retrieval and generation systems that can handle later turns, unanswerable questions, non-standalone questions, and multiple domains. MTRAG is available at this https URL.
• 摘要：检索增强生成 (RAG) 最近已成为大型语言模型 (LLM) 的一项非常流行的任务。在多轮 RAG 对话中对它们进行评估是一项重要且经常被忽视的任务，其中要求系统在先前对话的背景下生成对问题的回答，这还有几个额外的挑战。我们提出了 MTRAG：一个端到端的人工生成的多轮 RAG 基准，它反映了用于评估整个 RAG 管道的不同维度的几个真实世界属性。MTRAG 包含 110 个对话，平均每个对话 7.7 轮，涉及四个领域，总共 842 个任务。我们还通过合成数据和 LLM-as-a-Judge 评估探索自动化路径。我们的人工和自动评估表明，即使是最先进的 LLM RAG 系统在 MTRAG 上也举步维艰。我们展示了对强大的检索和生成系统的需求，这些系统可以处理后续的轮次、无法回答的问题、非独立问题和多个领域。MTRAG 可在此 https URL 上获得。

Title: Reading with Intent -- Neutralizing Intent

• Authors: Benjamin Reichman, Adar Avsian, Larry Heck
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.03475
• Pdf URL: https://arxiv.org/pdf/2501.03475
• Copy Paste: [[2501.03475]] Reading with Intent -- Neutralizing Intent(https://arxiv.org/abs/2501.03475)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: Queries to large language models (LLMs) can be divided into two parts: the instruction/question and the accompanying context. The context for retrieval-augmented generation (RAG) systems in most benchmarks comes from Wikipedia or Wikipedia-like texts which are written in a neutral and factual tone. However, when RAG systems retrieve internet-based content, they encounter text with diverse tones and linguistic styles, introducing challenges for downstream tasks. The Reading with Intent task addresses this issue by evaluating how varying tones in context passages affect model performance. Building on prior work that focused on sarcasm, we extend this paradigm by constructing a dataset where context passages are transformed to $11$ distinct emotions using a better synthetic data generation approach. Using this dataset, we train an emotion translation model to systematically adapt passages to specified emotional tones. The human evaluation shows that the LLM fine-tuned to become the emotion-translator benefited from the synthetically generated data. Finally, the emotion-translator is used in the Reading with Intent task to transform the passages to a neutral tone. By neutralizing the passages, it mitigates the challenges posed by sarcastic passages and improves overall results on this task by about $3\%$.
• 摘要：对大型语言模型 (LLM) 的查询可以分为两部分：指令/问题和伴随的上下文。大多数基准测试中检索增强生成 (RAG) 系统的上下文来自维基百科或类似维基百科的文本，这些文本以中性和事实语气编写。然而，当 RAG 系统检索基于互联网的内容时，它们会遇到具有不同语气和语言风格的文本，这给下游任务带来了挑战。有意图阅读任务通过评估上下文段落中不同语气如何影响模型性能来解决此问题。在先前专注于讽刺的工作的基础上，我们通过构建一个数据集扩展了这一范式，其中使用更好的合成数据生成方法将上下文段落转换为 $11$ 种不同的情绪。使用这个数据集，我们训练了一个情绪翻译模型，以系统地将段落调整为指定的情绪语气。人工评估表明，经过微调成为情绪翻译器的 LLM 受益于合成生成的数据。最后，在有意识阅读任务中使用情绪转换器将段落转换为中性语气。通过中和段落，它可以减轻讽刺段落带来的挑战，并将这项任务的整体结果提高约 $3\%$。

Title: Women, Infamous, and Exotic Beings: What Honorific Usages in Wikipedia Reveal about the Socio-Cultural Norms

• Authors: Sourabrata Mukherjee, Soumya Teotia, Sougata Saha, Monojit Choudhury
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.03479
• Pdf URL: https://arxiv.org/pdf/2501.03479
• Copy Paste: [[2501.03479]] Women, Infamous, and Exotic Beings: What Honorific Usages in Wikipedia Reveal about the Socio-Cultural Norms(https://arxiv.org/abs/2501.03479)
• Keywords: gpt, llm
• Abstract: Honorifics serve as powerful linguistic markers that reflect social hierarchies and cultural values. This paper presents a large-scale, cross-linguistic exploration of usage of honorific pronouns in Bengali and Hindi Wikipedia articles, shedding light on how socio-cultural factors shape language. Using LLM (GPT-4o), we annotated 10, 000 articles of real and fictional beings in each language for several sociodemographic features such as gender, age, fame, and exoticness, and the use of honorifics. We find that across all feature combinations, use of honorifics is consistently more common in Bengali than Hindi. For both languages, the use non-honorific pronouns is more commonly observed for infamous, juvenile, and exotic beings. Notably, we observe a gender bias in use of honorifics in Hindi, with men being more commonly referred to with honorifics than women.
• 摘要：敬语是反映社会等级和文化价值观的有力语言标记。本文对孟加拉语和印地语维基百科文章中敬语代词的使用进行了大规模、跨语言探索，揭示了社会文化因素如何影响语言。使用 LLM (GPT-4o)，我们注释了每种语言中 10,000 篇真实和虚构人物的文章，这些文章涉及几个社会人口特征，例如性别、年龄、名气和异国情调以及敬语的使用。我们发现，在所有特征组合中，孟加拉语中使用敬语的频率始终高于印地语。对于这两种语言，非敬语代词更常用于臭名昭著、幼稚和异国情调的人物。值得注意的是，我们观察到印地语在使用敬语方面存在性别偏见，男性比女性更常被用敬语称呼。

Title: Can LLMs Design Good Questions Based on Context?

• Authors: Yueheng Zhang, Xiaoyuan Liu, Yiyou Sun, Atheer Alharbi, Hend Alzahrani, Basel Alomair, Dawn Song
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.03491
• Pdf URL: https://arxiv.org/pdf/2501.03491
• Copy Paste: [[2501.03491]] Can LLMs Design Good Questions Based on Context?(https://arxiv.org/abs/2501.03491)
• Keywords: llm
• Abstract: This paper evaluates questions generated by LLMs from context, comparing them to human-generated questions across six dimensions. We introduce an automated LLM-based evaluation method, focusing on aspects like question length, type, context coverage, and answerability. Our findings highlight unique characteristics of LLM-generated questions, contributing insights that can support further research in question quality and downstream applications.
• 摘要：本文从上下文角度评估了 LLM 生成的问题，并从六个维度将它们与人工生成的问题进行了比较。我们介绍了一种基于 LLM 的自动化评估方法，重点关注问题长度、类型、上下文覆盖范围和可回答性等方面。我们的研究结果突出了 LLM 生成问题的独特特征，为进一步研究问题质量和下游应用提供了见解。

Title: A Sequential Optimal Learning Approach to Automated Prompt Engineering in Large Language Models

• Authors: Shuyang Wang, Somayeh Moazeni, Diego Klabjan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.03508
• Pdf URL: https://arxiv.org/pdf/2501.03508
• Copy Paste: [[2501.03508]] A Sequential Optimal Learning Approach to Automated Prompt Engineering in Large Language Models(https://arxiv.org/abs/2501.03508)
• Keywords: language model, llm, prompt
• Abstract: Designing effective prompts is essential to guiding large language models (LLMs) toward desired responses. Automated prompt engineering aims to reduce reliance on manual effort by streamlining the design, refinement, and optimization of natural language prompts. This paper proposes an optimal learning framework for automated prompt engineering, designed to sequentially identify effective prompt features while efficiently allocating a limited evaluation budget. We introduce a feature-based method to express prompts, which significantly broadens the search space. Bayesian regression is employed to utilize correlations among similar prompts, accelerating the learning process. To efficiently explore the large space of prompt features for a high quality prompt, we adopt the forward-looking Knowledge-Gradient (KG) policy for sequential optimal learning. The KG policy is computed efficiently by solving mixed-integer second-order cone optimization problems, making it scalable and capable of accommodating prompts characterized only through constraints. We demonstrate that our method significantly outperforms a set of benchmark strategies assessed on instruction induction tasks. The results highlight the advantages of using the KG policy for prompt learning given a limited evaluation budget. Our framework provides a solution to deploying automated prompt engineering in a wider range applications where prompt evaluation is costly.
• 摘要：设计有效的提示对于引导大型语言模型 (LLM) 获得所需响应至关重要。自动提示工程旨在通过简化自然语言提示的设计、改进和优化来减少对人工的依赖。本文提出了一种自动提示工程的最佳学习框架，旨在按顺序识别有效的提示特征，同时有效分配有限的评估预算。我们引入了一种基于特征的方法来表达提示，这大大拓宽了搜索空间。贝叶斯回归用于利用类似提示之间的相关性，从而加速学习过程。为了有效地探索大量提示特征以获得高质量的提示，我们采用了前瞻性的知识梯度 (KG) 策略进行顺序最佳学习。KG 策略通过求解混合整数二阶锥优化问题来有效计算，使其具有可扩展性，并且能够适应仅通过约束来表征的提示。我们证明我们的方法明显优于一组在指令诱导任务上评估的基准策略。结果突出了在有限的评估预算下使用 KG 策略进行提示学习的优势。我们的框架为在提示评估成本较高的更广泛的应用中部署自动提示工程提供了解决方案。

Title: Beyond Factual Accuracy: Evaluating Coverage of Diverse Factual Information in Long-form Text Generation

• Authors: Chris Samarinas, Alexander Krubner, Alireza Salemi, Youngwoo Kim, Hamed Zamani
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.03545
• Pdf URL: https://arxiv.org/pdf/2501.03545
• Copy Paste: [[2501.03545]] Beyond Factual Accuracy: Evaluating Coverage of Diverse Factual Information in Long-form Text Generation(https://arxiv.org/abs/2501.03545)
• Keywords: llm
• Abstract: This paper presents ICAT, an evaluation framework for measuring coverage of diverse factual information in long-form text generation. ICAT breaks down a long output text into a list of atomic claims and not only verifies each claim through retrieval from a (reliable) knowledge source, but also computes the alignment between the atomic factual claims and various aspects expected to be presented in the output. We study three implementations of the ICAT framework, each with a different assumption on the availability of aspects and alignment method. By adopting data from the diversification task in the TREC Web Track and the ClueWeb corpus, we evaluate the ICAT framework. We demonstrate strong correlation with human judgments and provide comprehensive evaluation across multiple state-of-the-art LLMs. Our framework further offers interpretable and fine-grained analysis of diversity and coverage. Its modular design allows for easy adaptation to different domains and datasets, making it a valuable tool for evaluating the qualitative aspects of long-form responses produced by LLMs.
• 摘要：本文介绍了 ICAT，这是一个用于测量长文本生成中各种事实信息覆盖范围的评估框架。ICAT 将长输出文本分解为原子声明列表，不仅通过从（可靠的）知识源检索来验证每个声明，还计算原子事实声明与预期在输出中呈现的各个方面之间的对齐。我们研究了 ICAT 框架的三种实现，每种实现对方面的可用性和对齐方法都有不同的假设。通过采用 TREC Web Track 和 ClueWeb 语料库中的多样化任务数据，我们评估了 ICAT 框架。我们展示了与人类判断的强相关性，并对多个最先进的 LLM 进行了全面的评估。我们的框架进一步提供了多样性和覆盖范围的可解释和细粒度分析。它的模块化设计可以轻松适应不同的领域和数据集，使其成为评估 LLM 生成的长篇响应的定性方面的宝贵工具。

Title: SLAM: Towards Efficient Multilingual Reasoning via Selective Language Alignment

• Authors: Yuchun Fan, Yongyu Mu, Yilin Wang, Lei Huang, Junhao Ruan, Bei Li, Tong Xiao, Shujian Huang, Xiaocheng Feng, Jingbo Zhu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.03681
• Pdf URL: https://arxiv.org/pdf/2501.03681
• Copy Paste: [[2501.03681]] SLAM: Towards Efficient Multilingual Reasoning via Selective Language Alignment(https://arxiv.org/abs/2501.03681)
• Keywords: language model, llm
• Abstract: Despite the significant improvements achieved by large language models (LLMs) in English reasoning tasks, these models continue to struggle with multilingual reasoning. Recent studies leverage a full-parameter and two-stage training paradigm to teach models to first understand non-English questions and then reason. However, this method suffers from both substantial computational resource computing and catastrophic forgetting. The fundamental cause is that, with the primary goal of enhancing multilingual comprehension, an excessive number of irrelevant layers and parameters are tuned during the first stage. Given our findings that the representation learning of languages is merely conducted in lower-level layers, we propose an efficient multilingual reasoning alignment approach that precisely identifies and fine-tunes the layers responsible for handling multilingualism. Experimental results show that our method, SLAM, only tunes 6 layers' feed-forward sub-layers including 6.5-8% of all parameters within 7B and 13B LLMs, achieving superior average performance than all strong baselines across 10 languages. Meanwhile, SLAM only involves one training stage, reducing training time by 4.1-11.9 compared to the two-stage method.
• 摘要：尽管大型语言模型 (LLM) 在英语推理任务中取得了显著的进步，但这些模型在多语言推理方面仍然举步维艰。最近的研究利用全参数和两阶段训练范式来教模型首先理解非英语问题，然后进行推理。然而，这种方法既需要大量的计算资源，又容易出现灾难性遗忘。根本原因是，在以增强多语言理解为主要目标的情况下，在第一阶段调整了过多不相关的层和参数。鉴于我们发现语言的表征学习仅在较低层进行，我们提出了一种有效的多语言推理对齐方法，可以精确识别和微调负责处理多语言的层。实验结果表明，我们的方法 SLAM 仅调整了 6 层的前馈子层，包括 7B 和 13B LLM 中所有参数的 6.5-8%，在 10 种语言中的平均性能优于所有强基线。同时，SLAM仅涉及一个训练阶段，与两阶段方法相比，训练时间减少了4.1-11.9。

Title: Unsupervised Speech Segmentation: A General Approach Using Speech Language Models

• Authors: Avishai Elmakies, Omri Abend, Yossi Adi
• Subjects: cs.CL, cs.AI, cs.LG, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2501.03711
• Pdf URL: https://arxiv.org/pdf/2501.03711
• Copy Paste: [[2501.03711]] Unsupervised Speech Segmentation: A General Approach Using Speech Language Models(https://arxiv.org/abs/2501.03711)
• Keywords: language model
• Abstract: In this paper, we introduce an unsupervised approach for Speech Segmentation, which builds on previously researched approaches, e.g., Speaker Diarization, while being applicable to an inclusive set of acoustic-semantic distinctions, paving a path towards a general Unsupervised Speech Segmentation approach. Unlike traditional speech and audio segmentation, which mainly focuses on spectral changes in the input signal, e.g., phone segmentation, our approach tries to segment the spoken utterance into chunks with differing acoustic-semantic styles, focusing on acoustic-semantic information that does not translate well into text, e.g., emotion or speaker. While most Speech Segmentation tasks only handle one style change, e.g., emotion diarization, our approach tries to handle multiple acoustic-semantic style changes. Leveraging recent advances in Speech Language Models (SLMs), we propose a simple unsupervised method to segment a given speech utterance. We empirically demonstrate the effectiveness of the proposed approach by considering several setups. Results suggest that the proposed method is superior to the evaluated baselines on boundary detection, segment purity, and over-segmentation. Code is available at this https URL.
• 摘要：在本文中，我们介绍了一种语音分割的无监督方法，该方法以先前研究的方法（例如说话人分类）为基础，同时适用于一组包容性的声学语义区别，为通用的无监督语音分割方法铺平了道路。与主要关注输入信号频谱变化（例如音素分割）的传统语音和音频分割不同，我们的方法尝试将口语话语分割成具有不同声学语义风格的块，重点关注无法很好地转化为文本的声学语义信息，例如情感或说话人。虽然大多数语音分割任务仅处理一种风格变化（例如情感分类），但我们的方法尝试处理多种声学语义风格变化。利用语音语言模型 (SLM) 的最新进展，我们提出了一种简单的无监督方法来分割给定的语音话语。我们通过考虑几种设置，通过经验证明了所提方法的有效性。结果表明，所提出的方法在边界检测、分割纯度和过度分割方面优于评估的基线。代码可在此 https URL 上获取。

Title: Investigating the Impact of Data Selection Strategies on Language Model Performance

• Authors: Jiayao Gu, Liting Chen, Yihong Li
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.03826
• Pdf URL: https://arxiv.org/pdf/2501.03826
• Copy Paste: [[2501.03826]] Investigating the Impact of Data Selection Strategies on Language Model Performance(https://arxiv.org/abs/2501.03826)
• Keywords: language model
• Abstract: Data selection is critical for enhancing the performance of language models, particularly when aligning training datasets with a desired target distribution. This study explores the effects of different data selection methods and feature types on model performance. We evaluate whether selecting data subsets can influence downstream tasks, whether n-gram features improve alignment with target distributions, and whether embedding-based neural features provide complementary benefits. Through comparative experiments using baseline random selection methods and distribution aligned approaches, we provide insights into the interplay between data selection strategies and model training efficacy. All code for this study can be found on \href{this https URL}{github repository}.
• 摘要：数据选择对于提高语言模型的性能至关重要，尤其是在将训练数据集与期望的目标分布对齐时。本研究探讨了不同的数据选择方法和特征类型对模型性能的影响。我们评估选择数据子集是否会影响下游任务，n-gram 特征是否能改善与目标分布的对齐，以及基于嵌入的神经特征是否能提供互补的优势。通过使用基线随机选择方法和分布对齐方法的比较实验，我们深入了解了数据选择策略与模型训练效果之间的相互作用。本研究的所有代码都可以在 \href{此 https URL}{github 存储库} 上找到。

Title: BabyLMs for isiXhosa: Data-Efficient Language Modelling in a Low-Resource Context

• Authors: Alexis Matzopoulos, Charl Hendriks, Hishaam Mahomed, Francois Meyer
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.03855
• Pdf URL: https://arxiv.org/pdf/2501.03855
• Copy Paste: [[2501.03855]] BabyLMs for isiXhosa: Data-Efficient Language Modelling in a Low-Resource Context(https://arxiv.org/abs/2501.03855)
• Keywords: language model
• Abstract: The BabyLM challenge called on participants to develop sample-efficient language models. Submissions were pretrained on a fixed English corpus, limited to the amount of words children are exposed to in development (<100m). The challenge produced new architectures for data-efficient language modelling, which outperformed models trained on trillions of words. This is promising for low-resource languages, where available corpora are limited to much less than 100m words. In this paper, we explore the potential of BabyLMs for low-resource languages, using the isiXhosa language as a case study. We pretrain two BabyLM architectures, ELC-BERT and MLSM, on an isiXhosa corpus. They outperform a vanilla pretrained model on POS tagging and NER, achieving notable gains (+3.2 F1) for the latter. In some instances, the BabyLMs even outperform XLM-R. Our findings show that data-efficient models are viable for low-resource languages, but highlight the continued importance, and lack of, high-quality pretraining data. Finally, we visually analyse how BabyLM architectures encode isiXhosa.
• 摘要：BabyLM 挑战赛要求参​​赛者开发样本高效的语言模型。参赛作品在固定的英语语料库上进行预训练，语料库的数量限制在儿童成长过程中接触的单词数量（<100m）。该挑战赛产生了数据高效的语言建模新架构，其表现优于在数万亿个单词上训练的模型。这对于资源匮乏的语言来说很有前景，因为这些语言中可用的语料库远远少于 1 亿个单词。在本文中，我们以科萨语为例，探索了 BabyLM 在资源匮乏的语言中的潜力。我们在科萨语语料库上预训练了两种 BabyLM 架构，即 ELC-BERT 和 MLSM。它们在 POS 标记和 NER 方面的表现优于普通预训练模型，后者取得了显著的进步（+3.2 F1）。在某些情况下，BabyLM 甚至优于 XLM-R。我们的研究结果表明，数据高效模型适用于资源匮乏的语言，但也凸显了高质量预训练数据的持续重要性以及缺乏性。最后，我们直观地分析了 BabyLM 架构如何对科萨语进行编码。

Title: Progressive Document-level Text Simplification via Large Language Models

• Authors: Dengzhao Fang, Jipeng Qiang, Yi Zhu, Yunhao Yuan, Wei Li, Yan Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.03857
• Pdf URL: https://arxiv.org/pdf/2501.03857
• Copy Paste: [[2501.03857]] Progressive Document-level Text Simplification via Large Language Models(https://arxiv.org/abs/2501.03857)
• Keywords: language model, gpt, llm, prompt, chat
• Abstract: Research on text simplification has primarily focused on lexical and sentence-level changes. Long document-level simplification (DS) is still relatively unexplored. Large Language Models (LLMs), like ChatGPT, have excelled in many natural language processing tasks. However, their performance on DS tasks is unsatisfactory, as they often treat DS as merely document summarization. For the DS task, the generated long sequences not only must maintain consistency with the original document throughout, but complete moderate simplification operations encompassing discourses, sentences, and word-level simplifications. Human editors employ a hierarchical complexity simplification strategy to simplify documents. This study delves into simulating this strategy through the utilization of a multi-stage collaboration using LLMs. We propose a progressive simplification method (ProgDS) by hierarchically decomposing the task, including the discourse-level, topic-level, and lexical-level simplification. Experimental results demonstrate that ProgDS significantly outperforms existing smaller models or direct prompting with LLMs, advancing the state-of-the-art in the document simplification task.
• 摘要：文本简化研究主要集中在词汇和句子级别的变化上。长文档级简化 (DS) 仍然相对未被探索。大型语言模型 (LLM)，如 ChatGPT，在许多自然语言处理任务中表现出色。然而，它们在 DS 任务上的表现并不令人满意，因为它们通常将 DS 视为文档摘要。对于 DS 任务，生成的长序列不仅必须始终与原始文档保持一致，而且还要完成适度的简化操作，包括话语、句子和词级简化。人类编辑采用分层复杂性简化策略来简化文档。本研究深入研究了通过使用 LLM 的多阶段协作来模拟该策略。我们提出了一种渐进式简化方法 (ProgDS)，通过分层分解任务，包括话语级、主题级和词汇级简化。实验结果表明，ProgDS 的表现明显优于现有的小型模型或使用 LLM 的直接提示，从而推动了文档简化任务的最新发展。

Title: Improving Dialectal Slot and Intent Detection with Auxiliary Tasks: A Multi-Dialectal Bavarian Case Study

• Authors: Xaver Maria Krückl, Verena Blaschke, Barbara Plank
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.03863
• Pdf URL: https://arxiv.org/pdf/2501.03863
• Copy Paste: [[2501.03863]] Improving Dialectal Slot and Intent Detection with Auxiliary Tasks: A Multi-Dialectal Bavarian Case Study(https://arxiv.org/abs/2501.03863)
• Keywords: language model
• Abstract: Reliable slot and intent detection (SID) is crucial in natural language understanding for applications like digital assistants. Encoder-only transformer models fine-tuned on high-resource languages generally perform well on SID. However, they struggle with dialectal data, where no standardized form exists and training data is scarce and costly to produce. We explore zero-shot transfer learning for SID, focusing on multiple Bavarian dialects, for which we release a new dataset for the Munich dialect. We evaluate models trained on auxiliary tasks in Bavarian, and compare joint multi-task learning with intermediate-task training. We also compare three types of auxiliary tasks: token-level syntactic tasks, named entity recognition (NER), and language modelling. We find that the included auxiliary tasks have a more positive effect on slot filling than intent classification (with NER having the most positive effect), and that intermediate-task training yields more consistent performance gains. Our best-performing approach improves intent classification performance on Bavarian dialects by 5.1 and slot filling F1 by 8.4 percentage points.
• 摘要：对于数字助理等应用的自然语言理解而言，可靠的槽位和意图检测 (SID) 至关重要。在高资源语言上微调的编码器专用变压器模型通常在 SID 上表现良好。然而，它们在处理方言数据时会遇到困难，因为方言数据不存在标准化形式，训练数据稀缺且成本高昂。我们探索了 SID 的零样本迁移学习，重点关注多种巴伐利亚方言，为此我们发布了慕尼黑方言的新数据集。我们评估了在巴伐利亚辅助任务上训练的模型，并将联合多任务学习与中间任务训练进行了比较。我们还比较了三种类型的辅助任务：token 级句法任务、命名实体识别 (NER) 和语言建模。我们发现，所包含的辅助任务对槽位填充的影响比意图分类更积极（其中 NER 具有最积极的影响），并且中间任务训练可带来更一致的性能提升。我们表现最佳的方法将巴伐利亚方言的意图分类性能提高了 5.1 个百分点，槽位填充 F1 提高了 8.4 个百分点。

Title: AlphaPO -- Reward shape matters for LLM alignment

• Authors: Aman Gupta, Shao Tang, Qingquan Song, Sirou Zhu, Jiwoo Hong, Ankan Saha, Viral Gupta, Noah Lee, Eunki Kim, Jason Zhu, Natesh Pillai, S. Sathiya Keerthi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.03884
• Pdf URL: https://arxiv.org/pdf/2501.03884
• Copy Paste: [[2501.03884]] AlphaPO -- Reward shape matters for LLM alignment(https://arxiv.org/abs/2501.03884)
• Keywords: language model, llm
• Abstract: Reinforcement Learning with Human Feedback (RLHF) and its variants have made huge strides toward the effective alignment of large language models (LLMs) to follow instructions and reflect human values. More recently, Direct Alignment Algorithms (DAAs) have emerged in which the reward modeling stage of RLHF is skipped by characterizing the reward directly as a function of the policy being learned. Examples include Direct Preference Optimization (DPO) and Simple Preference Optimization (SimPO). These methods often suffer from likelihood displacement, a phenomenon by which the probabilities of preferred responses are often reduced undesirably. In this paper, we argue that, for DAAs the reward (function) shape matters. We introduce AlphaPO, a new DAA method that leverages an $\alpha$-parameter to help change the shape of the reward function beyond the standard log reward. AlphaPO helps maintain fine-grained control over likelihood displacement and over-optimization. Compared to SimPO, one of the best performing DAAs, AlphaPO leads to about 7\% to 10\% relative improvement in alignment performance for the instruct versions of Mistral-7B and Llama3-8B. The analysis and results presented highlight the importance of the reward shape, and how one can systematically change it to affect training dynamics, as well as improve alignment performance.
• 摘要：带人类反馈的强化学习 (RLHF) 及其变体在有效对齐大型语言模型 (LLM) 以遵循指令和反映人类价值观方面取得了巨大进步。最近，出现了直接对齐算法 (DAA)，其中跳过了 RLHF 的奖励建模阶段，直接将奖励表征为正在学习的策略的函数。示例包括直接偏好优化 (DPO) 和简单偏好优化 (SimPO)。这些方法经常受到似然位移的影响，这种现象会导致首选响应的概率经常不受欢迎地降低。在本文中，我们认为，对于 DAA，奖励（函数）形状很重要。我们引入了 AlphaPO，这是一种新的 DAA 方法，它利用 $\alpha$ 参数来帮助改变奖励函数的形状，使其超出标准对数奖励。AlphaPO 有助于保持对似然位移和过度优化的细粒度控制。与性能最佳的 DAA 之一 SimPO 相比，AlphaPO 可使 Mistral-7B 和 Llama3-8B 指令版的对齐性能相对提高约 7% 至 10%。本文提出的分析和结果强调了奖励形状的重要性，以及如何系统地改变它以影响训练动态并提高对齐性能。

Title: Not all tokens are created equal: Perplexity Attention Weighted Networks for AI generated text detection

• Authors: Pablo Miralles-González, Javier Huertas-Tato, Alejandro Martín, David Camacho
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.03940
• Pdf URL: https://arxiv.org/pdf/2501.03940
• Copy Paste: [[2501.03940]] Not all tokens are created equal: Perplexity Attention Weighted Networks for AI generated text detection(https://arxiv.org/abs/2501.03940)
• Keywords: language model, llm
• Abstract: The rapid advancement in large language models (LLMs) has significantly enhanced their ability to generate coherent and contextually relevant text, raising concerns about the misuse of AI-generated content and making it critical to detect it. However, the task remains challenging, particularly in unseen domains or with unfamiliar LLMs. Leveraging LLM next-token distribution outputs offers a theoretically appealing approach for detection, as they encapsulate insights from the models' extensive pre-training on diverse corpora. Despite its promise, zero-shot methods that attempt to operationalize these outputs have met with limited success. We hypothesize that one of the problems is that they use the mean to aggregate next-token distribution metrics across tokens, when some tokens are naturally easier or harder to predict and should be weighted differently. Based on this idea, we propose the Perplexity Attention Weighted Network (PAWN), which uses the last hidden states of the LLM and positions to weight the sum of a series of features based on metrics from the next-token distribution across the sequence length. Although not zero-shot, our method allows us to cache the last hidden states and next-token distribution metrics on disk, greatly reducing the training resource requirements. PAWN shows competitive and even better performance in-distribution than the strongest baselines (fine-tuned LMs) with a fraction of their trainable parameters. Our model also generalizes better to unseen domains and source models, with smaller variability in the decision boundary across distribution shifts. It is also more robust to adversarial attacks, and if the backbone has multilingual capabilities, it presents decent generalization to languages not seen during supervised training, with LLaMA3-1B reaching a mean macro-averaged F1 score of 81.46% in cross-validation with nine languages.
• 摘要：大型语言模型 (LLM) 的快速发展显著增强了它们生成连贯且上下文相关的文本的能力，这引发了人们对滥用 AI 生成内容的担忧，因此检测 AI 生成内容至关重要。然而，这项任务仍然具有挑战性，特别是在未知领域或不熟悉的 LLM 中。利用 LLM 下一个标记分布输出提供了一种理论上有吸引力的检测方法，因为它们包含了模型对不同语料库进行大量预训练的见解。尽管前景光明，但试图将这些输出操作化的零样本方法却收效甚微。我们假设其中一个问题是，它们使用平均值来聚合各个标记的下一个标记分布指标，而有些标记天生更容易或更难预测，应该以不同的方式加权。基于这个想法，我们提出了困惑度注意力加权网络 (PAWN)，它使用 LLM 的最后隐藏状态和位置来根据序列长度上的下一个标记分布指标对一系列特征的总和进行加权。虽然不是零样本，但我们的方法允许我们将最后的隐藏状态和下一个标记分布指标缓存在磁盘上，从而大大减少了训练资源需求。PAWN 在分布方面表现出竞争力，甚至比最强的基线（微调的 LM）表现更好，但可训练参数却只有后者的一小部分。我们的模型还可以更好地推广到看不见的域和源模型，决策边界在分布变化中的变异性更小。它对对抗性攻击也更具鲁棒性，如果主干具有多语言能力，它对监督训练期间未见过的语言具有不错的泛化能力，LLaMA3-1B 在九种语言的交叉验证中达到了 81.46% 的平均宏平均 F1 分数。

Title: Localizing AI: Evaluating Open-Weight Language Models for Languages of Baltic States

• Authors: Jurgita Kapočiūtė-Dzikienė, Toms Bergmanis, Mārcis Pinnis
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.03952
• Pdf URL: https://arxiv.org/pdf/2501.03952
• Copy Paste: [[2501.03952]] Localizing AI: Evaluating Open-Weight Language Models for Languages of Baltic States(https://arxiv.org/abs/2501.03952)
• Keywords: language model, llm, hallucination
• Abstract: Although large language models (LLMs) have transformed our expectations of modern language technologies, concerns over data privacy often restrict the use of commercially available LLMs hosted outside of EU jurisdictions. This limits their application in governmental, defence, and other data-sensitive sectors. In this work, we evaluate the extent to which locally deployable open-weight LLMs support lesser-spoken languages such as Lithuanian, Latvian, and Estonian. We examine various size and precision variants of the top-performing multilingual open-weight models, Llama~3, Gemma~2, Phi, and NeMo, on machine translation, multiple-choice question answering, and free-form text generation. The results indicate that while certain models like Gemma~2 perform close to the top commercially available models, many LLMs struggle with these languages. Most surprisingly, however, we find that these models, while showing close to state-of-the-art translation performance, are still prone to lexical hallucinations with errors in at least 1 in 20 words for all open-weight multilingual LLMs.
• 摘要：尽管大型语言模型 (LLM) 已经改变了我们对现代语言技术的期望，但对数据隐私的担忧往往会限制在欧盟管辖范围之外托管的商用 LLM 的使用。这限制了它们在政府、国防和其他数据敏感领域的应用。在这项工作中，我们评估了本地可部署的开放式 LLM 对立陶宛语、拉脱维亚语和爱沙尼亚语等使用较少的语言的支持程度。我们研究了机器翻译、多项选择题回答和自由格式文本生成方面表现最佳的多语言开放式模型 Llama~3、Gemma~2、Phi 和 NeMo 的各种大小和精度变体。结果表明，虽然某些模型（如 Gemma~2）的表现接近顶级商用模型，但许多 LLM 在这些语言上表现不佳。然而，最令人惊讶的是，我们发现，虽然这些模型表现出接近最先进的翻译性能，但仍然容易出现词汇幻觉，对于所有开放权重多语言 LLM 来说，至少每 20 个单词中就会有 1 个出现错误。

Title: Semantically Cohesive Word Grouping in Indian Languages

• Authors: N J Karthika, Adyasha Patra, Nagasai Saketh Naidu, Arnab Bhattacharya, Ganesh Ramakrishnan, Chaitali Dangarikar
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.03988
• Pdf URL: https://arxiv.org/pdf/2501.03988
• Copy Paste: [[2501.03988]] Semantically Cohesive Word Grouping in Indian Languages(https://arxiv.org/abs/2501.03988)
• Keywords: prompt
• Abstract: Indian languages are inflectional and agglutinative and typically follow clause-free word order. The structure of sentences across most major Indian languages are similar when their dependency parse trees are considered. While some differences in the parsing structure occur due to peculiarities of a language or its preferred natural way of conveying meaning, several apparent differences are simply due to the granularity of representation of the smallest semantic unit of processing in a sentence. The semantic unit is typically a word, typographically separated by whitespaces. A single whitespace-separated word in one language may correspond to a group of words in another. Hence, grouping of words based on semantics helps unify the parsing structure of parallel sentences across languages and, in the process, morphology. In this work, we propose word grouping as a major preprocessing step for any computational or linguistic processing of sentences for Indian languages. Among Indian languages, since Hindi is one of the least agglutinative, we expect it to benefit the most from word-grouping. Hence, in this paper, we focus on Hindi to study the effects of grouping. We perform quantitative assessment of our proposal with an intrinsic method that perturbs sentences by shuffling words as well as an extrinsic evaluation that verifies the importance of word grouping for the task of Machine Translation (MT) using decomposed prompting. We also qualitatively analyze certain aspects of the syntactic structure of sentences. Our experiments and analyses show that the proposed grouping technique brings uniformity in the syntactic structures, as well as aids underlying NLP tasks.
• 摘要：印度语言具有屈折性和黏着性，通常遵循无小句的词序。当考虑依赖解析树时，大多数主要印度语言的句子结构相似。虽然解析结构中的一些差异是由于语言的特殊性或其首选的表达含义的自然方式而产生的，但一些明显的差异仅仅是由于句子中最小语义处理单元的表示粒度。语义单元通常是一个单词，在印刷上由空格分隔。一种语言中一个由空格分隔的单词可能对应于另一种语言中的一组单词。因此，基于语义对单词进行分组有助于统一不同语言中平行句子的解析结构，并在此过程中统一形态。在这项工作中，我们提出将单词分组作为印度语言句子的任何计算或语言处理的主要预处理步骤。在印度语言中，由于印地语是最不黏着的语言之一，我们预计它会从单词分组中受益最多。因此，在本文中，我们专注于印地语来研究分组的效果。我们使用一种内在方法对我们的提议进行定量评估，该方法通过打乱单词来扰乱句子，并使用一种外在评估来验证单词分组对于使用分解提示的机器翻译 (MT) 任务的重要性。我们还定性分析了句子句法结构的某些方面。我们的实验和分析表明，所提出的分组技术带来了句法结构的统一性，并有助于底层的 NLP 任务。

Title: Influences on LLM Calibration: A Study of Response Agreement, Loss Functions, and Prompt Styles

• Authors: Yuxi Xia, Pedro Henrique Luz de Araujo, Klim Zaporojets, Benjamin Roth
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.03991
• Pdf URL: https://arxiv.org/pdf/2501.03991
• Copy Paste: [[2501.03991]] Influences on LLM Calibration: A Study of Response Agreement, Loss Functions, and Prompt Styles(https://arxiv.org/abs/2501.03991)
• Keywords: language model, llm, prompt
• Abstract: Calibration, the alignment between model confidence and prediction accuracy, is critical for the reliable deployment of large language models (LLMs). Existing works neglect to measure the generalization of their methods to other prompt styles and different sizes of LLMs. To address this, we define a controlled experimental setting covering 12 LLMs and four prompt styles. We additionally investigate if incorporating the response agreement of multiple LLMs and an appropriate loss function can improve calibration performance. Concretely, we build Calib-n, a novel framework that trains an auxiliary model for confidence estimation that aggregates responses from multiple LLMs to capture inter-model agreement. To optimize calibration, we integrate focal and AUC surrogate losses alongside binary cross-entropy. Experiments across four datasets demonstrate that both response agreement and focal loss improve calibration from baselines. We find that few-shot prompts are the most effective for auxiliary model-based methods, and auxiliary models demonstrate robust calibration performance across accuracy variations, outperforming LLMs' internal probabilities and verbalized confidences. These insights deepen the understanding of influence factors in LLM calibration, supporting their reliable deployment in diverse applications.
• 摘要：校准，即模型置信度与预测准确度之间的一致性，对于大型语言模型 (LLM) 的可靠部署至关重要。现有研究忽视了衡量其方法对其他提示样式和不同大小的 LLM 的泛化能力。为了解决这个问题，我们定义了一个受控的实验设置，涵盖 12 个 LLM 和 4 种提示样式。我们还研究了结合多个 LLM 的响应一致性和适当的损失函数是否可以提高校准性能。具体来说，我们构建了 Calib-n，这是一个新颖的框架，用于训练辅助模型进行置信度估计，该模型汇总来自多个 LLM 的响应以捕获模型间一致性。为了优化校准，我们将焦点和 AUC 替代损失与二元交叉熵相结合。在四个数据集上的实验表明，响应一致性和焦点损失都可以从基线改进校准。我们发现，对于基于辅助模型的方法来说，小样本提示是最有效的，辅助模型在准确度变化中表现出强大的校准性能，优于 LLM 的内部概率和口头置信度。这些见解加深了对 LLM 校准中影响因素的理解，支持它们在各种应用中的可靠部署。