2025-02-04

Title: MALT: Mechanistic Ablation of Lossy Translation in LLMs for a Low-Resource Language: Urdu

• Authors: Taaha Saleem Bajwa
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.00041
• Pdf URL: https://arxiv.org/pdf/2502.00041
• Copy Paste: [[2502.00041]] MALT: Mechanistic Ablation of Lossy Translation in LLMs for a Low-Resource Language: Urdu(https://arxiv.org/abs/2502.00041)
• Keywords: llm, prompt
• Abstract: LLMs are predominantly trained on English data, which leads to a significant drop in performance on low-resource languages. Understanding how LLMs handle these languages is crucial for improving their effectiveness. This study focuses on Urdu as a use case for exploring the challenges faced by LLMs in processing low-resource languages. LLMs primarily reason in English when prompted in another language, with the final layers acting as translators to convert the English response into the target language. This study finds that even for low-resource languages, the internal latent response of LLMs in English is quite coherent; however, the translation features are lossy and result in poor translations, leading to reduced performance. By mechanistically removing these translation features and using a separate translation model to translate the internal latent response of LLM, the performance of LLMs improves significantly while also preserving the cultural nuances of the input in low-resource languages.
• 摘要：LLM 主要在英语数据上进行训练，这导致其在低资源语言上的性能显著下降。了解 LLM 如何处理这些语言对于提高其有效性至关重要。本研究以乌尔都语为用例，探讨 LLM 在处理低资源语言时面临的挑战。当用另一种语言提示时，LLM 主要用英语推理，最后的几层充当翻译器，将英语响应转换成目标语言。本研究发现，即使对于低资源语言，LLM 在英语中的内部潜在响应也相当连贯；然而，翻译特征是有损的，导致翻译质量很差，从而导致性能下降。通过机械地删除这些翻译特征并使用单独的翻译模型来翻译 LLM 的内部潜在响应，LLM 的性能显着提高，同时还保留了低资源语言输入的文化细微差别。

Title: A Multi-Layered Large Language Model Framework for Disease Prediction

• Authors: Malak Mohamed, Rokaia Emad, Ali Hamdi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.00063
• Pdf URL: https://arxiv.org/pdf/2502.00063
• Copy Paste: [[2502.00063]] A Multi-Layered Large Language Model Framework for Disease Prediction(https://arxiv.org/abs/2502.00063)
• Keywords: language model, gpt, llm
• Abstract: Social telehealth has revolutionized healthcare by enabling patients to share symptoms and receive medical consultations remotely. Users frequently post symptoms on social media and online health platforms, generating a vast repository of medical data that can be leveraged for disease classification and symptom severity assessment. Large language models (LLMs), such as LLAMA3, GPT-3.5 Turbo, and BERT, process complex medical data to enhance disease classification. This study explores three Arabic medical text preprocessing techniques: text summarization, text refinement, and Named Entity Recognition (NER). Evaluating CAMeL-BERT, AraBERT, and Asafaya-BERT with LoRA, the best performance was achieved using CAMeL-BERT with NER-augmented text (83% type classification, 69% severity assessment). Non-fine-tuned models performed poorly (13%-20% type classification, 40%-49% severity assessment). Integrating LLMs into social telehealth systems enhances diagnostic accuracy and treatment outcomes.
• 摘要：社交远程医疗使患者能够远程分享症状并接受医疗咨询，从而彻底改变了医疗保健。用户经常在社交媒体和在线健康平台上发布症状，从而生成大量医疗数据，可用于疾病分类和症状严重程度评估。大型语言模型 (LLM)，例如 LLAMA3、GPT-3.5 Turbo 和 BERT，可处理复杂的医疗数据以增强疾病分类。本研究探讨了三种阿拉伯语医学文本预处理技术：文本摘要、文本细化和命名实体识别 (NER)。使用 LoRA 评估 CAMeL-BERT、AraBERT 和 Asafaya-BERT，使用 NER 增强文本的 CAMeL-BERT 获得了最佳性能（83% 的类型分类，69% 的严重程度评估）。未微调的模型表现不佳（13%-20% 的类型分类，40%-49% 的严重程度评估）。将 LLM 集成到社交远程医疗系统中可提高诊断准确性和治疗效果。

Title: BTS: Harmonizing Specialized Experts into a Generalist LLM

• Authors: Qizhen Zhang, Prajjwal Bhargava, Chloe Bi, Chris X. Cai, Jakob Foerster, Jeremy Fu, Punit Singh Koura, Ruan Silva, Sheng Shen, Emily Dinan, Suchin Gururangan, Mike Lewis
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.00075
• Pdf URL: https://arxiv.org/pdf/2502.00075
• Copy Paste: [[2502.00075]] BTS: Harmonizing Specialized Experts into a Generalist LLM(https://arxiv.org/abs/2502.00075)
• Keywords: language model, llm
• Abstract: We present Branch-Train-Stitch (BTS), an efficient and flexible training algorithm for combining independently trained large language model (LLM) experts into a single, capable generalist model. Following Li et al., we start with a single seed language model which is branched into domain-specific (e.g., coding or math) experts with continual pretraining. BTS combines experts into a generalist model using lightweight stitch layers, which are inserted between frozen experts and the seed LLM, and trained on a small datamix of the expert domains. Stitch layers enable the seed LLM to integrate representations from any number of experts during the forward pass, allowing it to generalize to new domains, despite remaining frozen. Because BTS does not alter the constituent LLMs, BTS provides a modular and flexible approach: experts can be easily removed and new experts can be added with only a small amount of training. Compared to alternative model merging approaches, BTS yields the best generalist performance on a variety of downstream tasks, retaining the specialized capabilities of each of the experts.
• 摘要：我们提出了分支-训练-缝合 (BTS)，这是一种高效灵活的训练算法，用于将独立训练的大型语言模型 (LLM) 专家组合成一个功能强大的通用模型。按照 Li 等人的方法，我们从单个种子语言模型开始，该模型通过持续的预训练分为特定领域（例如编码或数学）专家。BTS 使用轻量级缝合层将专家组合成通用模型，这些缝合层插入冻结专家和种子 LLM 之间，并在专家领域的小型数据混合上进行训练。缝合层使种子 LLM 能够在前向传递过程中整合来自任意数量专家的表示，使其能够推广到新领域，尽管保持冻结状态。由于 BTS 不会改变组成 LLM，因此 BTS 提供了一种模块化且灵活的方法：只需少量训练即可轻松删除专家并添加新专家。与其他模型合并方法相比，BTS 在各种下游任务上实现了最佳的通用性能，保留了每位专家的专业能力。

Title: Efficient Beam Search for Large Language Models Using Trie-Based Decoding

• Authors: Brian J Chan, Jui-Hung Cheng, Mao Xun Huang, Chao-Ting Chen, Hen-Hsen Huang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.00085
• Pdf URL: https://arxiv.org/pdf/2502.00085
• Copy Paste: [[2502.00085]] Efficient Beam Search for Large Language Models Using Trie-Based Decoding(https://arxiv.org/abs/2502.00085)
• Keywords: language model
• Abstract: In Transformer-based sequence-to-sequence generation, beam search has proven effective in enhancing the quality of generated sequences compared to greedy decoding. Conventional beam search methods typically adopt either a sequential or batch-based approach. The sequential approach, while memory-efficient, requires multiple decoding passes to construct a complete search tree, leading to significantly slower inference. On the other hand, the batch-based approach enables parallel computation across beams, but at the expense of high memory consumption due to the need to maintain separate key-value (KV) caches for each beam. In this study, we introduce a novel trie (prefix-tree)-based parallel decoding method that addresses the memory inefficiency of batch-based beam search. By sharing a single KV cache among all beams that share the same prefix, the proposed method not only reduces memory consumption dramatically but also enables parallel decoding across all branches. This innovative use of a prefix tree offers an efficient alternative for beam search, achieving significant memory savings while preserving inference speed, making it particularly well-suited for memory-constrained environments or large-scale model deployments.
• 摘要：在基于 Transformer 的序列到序列生成中，与贪婪解码相比，束搜索已被证明能有效提高生成序列的质量。传统的束搜索方法通常采用顺序或基于批处理的方法。顺序方法虽然内存效率高，但需要多次解码才能构建完整的搜索树，从而导致推理速度明显变慢。另一方面，基于批处理的方法可以实现跨束的并行计算，但由于需要为每个束维护单独的键值 (KV) 缓存，因此会消耗大量内存。在本研究中，我们介绍了一种基于 trie（前缀树）的新型并行解码方法，该方法解决了基于批处理的束搜索的内存效率低下问题。通过在共享相同前缀的所有束之间共享单个 KV 缓存，所提出的方法不仅可以显著降低内存消耗，还可以实现跨所有分支的并行解码。这种对前缀树的创新使用为束搜索提供了一种有效的替代方案，在保持推理速度的同时实现了显着的内存节省，使其特别适合内存受限的环境或大规模模型部署。

Title: Ensembles of Low-Rank Expert Adapters

• Authors: Yinghao Li, Vianne Gao, Chao Zhang, MohamadAli Torkamani
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.00089
• Pdf URL: https://arxiv.org/pdf/2502.00089
• Copy Paste: [[2502.00089]] Ensembles of Low-Rank Expert Adapters(https://arxiv.org/abs/2502.00089)
• Keywords: language model, llm
• Abstract: The training and fine-tuning of large language models (LLMs) often involve diverse textual data from multiple sources, which poses challenges due to conflicting gradient directions, hindering optimization and specialization. These challenges can undermine model generalization across tasks, resulting in reduced downstream performance. Recent research suggests that fine-tuning LLMs on carefully selected, task-specific subsets of data can match or even surpass the performance of using the entire dataset. Building on these insights, we propose the Ensembles of Low-Rank Expert Adapters (ELREA) framework to improve the model's capability to handle diverse tasks. ELREA clusters the training instructions based on their gradient directions, representing different areas of expertise and thereby reducing conflicts during optimization. Expert adapters are then trained on these clusters, utilizing the low-rank adaptation (LoRA) technique to ensure training efficiency and model scalability. During inference, ELREA combines predictions from the most relevant expert adapters based on the input data's gradient similarity to the training clusters, ensuring optimal adapter selection for each task. Experiments show that our method outperforms baseline LoRA adapters trained on the full dataset and other ensemble approaches with similar training and inference complexity across a range of domain-specific tasks.
• 摘要：大型语言模型 (LLM) 的训练和微调通常涉及来自多个来源的不同文本数据，这会带来挑战，因为梯度方向冲突会阻碍优化和专业化。这些挑战可能会破坏跨任务的模型泛化，从而导致下游性能下降。最近的研究表明，在精心挑选的、特定于任务的数据子集上微调 LLM 可以达到甚至超过使用整个数据集的性能。基于这些见解，我们提出了低秩专家适配器集成 (ELREA) 框架来提高模型处理不同任务的能力。ELREA 根据梯度方向对训练指令进行聚类，代表不同的专业领域，从而减少优化过程中的冲突。然后利用低秩自适应 (LoRA) 技术在这些集群上训练专家适配器，以确保训练效率和模型可扩展性。在推理过程中，ELREA 根据输入数据的梯度相似性将最相关的专家适配器的预测与训练集群相结合，确保为每个任务选择最佳适配器。实验表明，我们的方法优于在完整数据集上训练的基线 LoRA 适配器以及在一系列特定领域任务中具有相似训练和推理复杂度的其他集成方法。

Title: Sparse Autoencoder Insights on Voice Embeddings

• Authors: Daniel Pluth, Yu Zhou, Vijay K. Gurbani
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.00127
• Pdf URL: https://arxiv.org/pdf/2502.00127
• Copy Paste: [[2502.00127]] Sparse Autoencoder Insights on Voice Embeddings(https://arxiv.org/abs/2502.00127)
• Keywords: language model, llm
• Abstract: Recent advances in explainable machine learning have highlighted the potential of sparse autoencoders in uncovering mono-semantic features in densely encoded embeddings. While most research has focused on Large Language Model (LLM) embeddings, the applicability of this technique to other domains remains largely unexplored. This study applies sparse autoencoders to speaker embeddings generated from a Titanet model, demonstrating the effectiveness of this technique in extracting mono-semantic features from non-textual embedded data. The results show that the extracted features exhibit characteristics similar to those found in LLM embeddings, including feature splitting and steering. The analysis reveals that the autoencoder can identify and manipulate features such as language and music, which are not evident in the original embedding. The findings suggest that sparse autoencoders can be a valuable tool for understanding and interpreting embedded data in many domains, including audio-based speaker recognition.
• 摘要：可解释机器学习的最新进展凸显了稀疏自动编码器在发现密集编码嵌入中的单语义特征方面的潜力。虽然大多数研究都集中在大型语言模型 (LLM) 嵌入上，但这种技术在其他领域的适用性仍未得到充分探索。本研究将稀疏自动编码器应用于 Titanet 模型生成的说话人嵌入，证明了该技术在从非文本嵌入数据中提取单语义特征方面的有效性。结果表明，提取的特征表现出与 LLM 嵌入中发现的特征相似的特性，包括特征分割和转向。分析表明，自动编码器可以识别和操纵原始嵌入中不明显的语言和音乐等特征。研究结果表明，稀疏自动编码器可以成为理解和解释许多领域（包括基于音频的说话人识别）嵌入数据的有力工具。

Title: A Three-Branch Checks-and-Balances Frameworkfor Context-Aware Ethical Alignment of Large Language Models

• Authors: Edward Y. Chang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.00136
• Pdf URL: https://arxiv.org/pdf/2502.00136
• Copy Paste: [[2502.00136]] A Three-Branch Checks-and-Balances Frameworkfor Context-Aware Ethical Alignment of Large Language Models(https://arxiv.org/abs/2502.00136)
• Keywords: language model, llm
• Abstract: This paper introduces a three-branch checks-and-balances framework for ethical alignment of Large Language Models (LLMs), inspired by governmental systems. It implements three independent yet interacting components: LLMs as the executive branch for knowledge generation, DIKE as the legislative branch establishing ethical guardrails, and ERIS as the judicial branch for contextual interpretation. The adversarial DIKE-ERIS duality enables adaptation to diverse cultural contexts while upholding consistent ethical principles. This architecture addresses limitations of reinforcement learning with human feedback (RLHF) by providing interpretable, adaptable, and culturally-aware ethical reasoning. Through self-supervised learning and adversarial testing, our framework demonstrates how emotional modeling can guide linguistic behaviors toward ethical outcomes while preserving independence across knowledge generation, ethical oversight, and contextual interpretation.
• 摘要：本文介绍了一种受政府系统启发的三分支制衡框架，用于大型语言模型 (LLM) 的道德协调。它实现了三个独立但相互作用的组成部分：LLM 作为知识生成的行政部门，DIKE 作为建立道德护栏的立法部门，ERIS 作为情境解释的司法部门。对抗性的 DIKE-ERIS 二元性能够适应不同的文化背景，同时坚持一致的道德原则。该架构通过提供可解释、可适应和具有文化意识的道德推理来解决强化学习与人类反馈 (RLHF) 的局限性。通过自我监督学习和对抗性测试，我们的框架展示了情感建模如何引导语言行为朝着道德结果发展，同时在知识生成、道德监督和情境解释方面保持独立性。

Title: Resolving Editing-Unlearning Conflicts: A Knowledge Codebook Framework for Large Language Model Updating

• Authors: Binchi Zhang, Zhengzhang Chen, Zaiyi Zheng, Jundong Li, Haifeng Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.00158
• Pdf URL: https://arxiv.org/pdf/2502.00158
• Copy Paste: [[2502.00158]] Resolving Editing-Unlearning Conflicts: A Knowledge Codebook Framework for Large Language Model Updating(https://arxiv.org/abs/2502.00158)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) excel in natural language processing by encoding extensive human knowledge, but their utility relies on timely updates as knowledge evolves. Updating LLMs involves two key tasks simultaneously: unlearning to remove unwanted knowledge and editing to incorporate new information. Existing methods face two major challenges: ineffective knowledge storage (either too sparse or too dense) and task conflicts between editing and unlearning, as validated through our theoretical and experimental results. To address these issues, we propose LOKA, a conflict-free framework for LLM updating based on a knowledge codebook. During training, updated knowledge is stored in multiple codebook memories. To optimize knowledge storage, a similarity-aware knowledge mapping ensures that related knowledge pieces are clustered and allocated to the same memory. Additionally, LOKA resolves task conflicts by employing task-specific and multi-task memories guided by a conflict score. In the inference stage, LOKA retrieves the most relevant memory from the codebook and plugs it into the original LLM to apply the updated knowledge. A learning-based router controls codebook activation to further improve knowledge utilization. Extensive experiments demonstrate the effectiveness of LOKA in LLM knowledge updating tasks.
• 摘要：大型语言模型 (LLM) 通过编码大量人类知识在自然语言处理方面表现出色，但它们的实用性依赖于知识发展过程中的及时更新。更新 LLM 同时涉及两个关键任务：通过反学习删除不需要的知识，以及通过编辑合并新信息。现有方法面临两大挑战：知识存储效率低下（要么太稀疏，要么太密集）以及编辑和反学习之间的任务冲突，这已通过我们的理论和实验结果得到验证。为了解决这些问题，我们提出了 LOKA，这是一个基于知识码本的无冲突 LLM 更新框架。在训练期间，更新的知识存储在多个码本内存中。为了优化知识存储，相似性感知知识映射可确保相关知识片段聚类并分配到同一内存中。此外，LOKA 通过使用由冲突分数指导的任务特定和多任务内存来解决任务冲突。在推理阶段，LOKA 从码本中检索最相关的内存并将其插入原始 LLM 以应用更新的知识。基于学习的路由器控制码本激活，进一步提高知识利用率。大量实验证明了 LOKA 在 LLM 知识更新任务中的有效性。

Title: Context-Preserving Tensorial Reconfiguration in Large Language Model Training

• Authors: Larin Tonix, Morgana Baskerville, Nathaniel Stourton, Ophelia Tattershall
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.00246
• Pdf URL: https://arxiv.org/pdf/2502.00246
• Copy Paste: [[2502.00246]] Context-Preserving Tensorial Reconfiguration in Large Language Model Training(https://arxiv.org/abs/2502.00246)
• Keywords: language model
• Abstract: Handling long-range dependencies in neural architectures has remained a persistent challenge due to computational limitations and inefficient contextual retention mechanisms. Tensorial operations have provided a foundation for restructuring model representations, yet conventional architectures have struggled to incorporate such techniques without introducing excessive complexity. A novel approach, Context-Preserving Tensorial Reconfiguration (CPTR), enables dynamic reorganization of weight tensors through structured factorization and adaptive contraction, allowing for enhanced contextual integration without substantial computational overhead. Empirical evaluations demonstrate that CPTR improves coherence retention across extended sequences, leading to measurable reductions in perplexity and improved recall accuracy for long-context tasks. Performance comparisons reveal that CPTR-enhanced models exhibit greater computational efficiency and reduced memory consumption while maintaining competitive language generation fluency and accuracy. Gradient stability metrics further validate the improved training efficiency, revealing more controlled variance in weight updates. Comparative studies across baseline and CPTR-enhanced models confirm that tensorial reconfiguration contributes to more stable and computationally efficient language modeling. The findings support the potential of CPTR in refining contemporary neural architectures for tasks requiring long-range contextual understanding and efficient memory utilization.
• 摘要：由于计算限制和低效的上下文保留机制，处理神经架构中的长程依赖关系一直是一项挑战。张量操作为重构模型表示提供了基础，但传统架构一直在努力在不引入过多复杂性的情况下融入此类技术。一种新颖的方法，即上下文保留张量重构 (CPTR)，通过结构化分解和自适应收缩实现权重张量的动态重组，从而可以在不产生大量计算开销的情况下增强上下文集成。实证评估表明，CPTR 可改善扩展序列中的连贯性保留，从而显著降低困惑度并提高长上下文任务的回忆准确度。性能比较表明，CPTR 增强模型表现出更高的计算效率和更低的内存消耗，同时保持了具有竞争力的语言生成流畅性和准确性。梯度稳定性指标进一步验证了改进的训练效率，揭示了权重更新中更可控的方差。对基线和 CPTR 增强模型的比较研究证实，张量重构有助于实现更稳定、计算效率更高的语言建模。研究结果支持了 CPTR 在改进当代神经结构以完成需要长期上下文理解和有效记忆利用的任务方面的潜力。

Title: Scaling Flaws of Verifier-Guided Search in Mathematical Reasoning

• Authors: Fei Yu, Yingru Li, Benyou Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.00271
• Pdf URL: https://arxiv.org/pdf/2502.00271
• Copy Paste: [[2502.00271]] Scaling Flaws of Verifier-Guided Search in Mathematical Reasoning(https://arxiv.org/abs/2502.00271)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) struggle with multi-step reasoning, where inference-time scaling has emerged as a promising strategy for performance improvement. Verifier-guided search outperforms repeated sampling when sample size is limited by selecting and prioritizing valid reasoning paths. However, we identify a critical limitation: scaling flaws, prevalent across different models (Mistral 7B and DeepSeekMath 7B), benchmarks (GSM8K and MATH), and verifiers (outcome value models and process reward models). As sample size increases, verifier-guided search exhibits diminishing advantages and eventually underperforms repeated sampling. Our analysis attributes this to verifier failures, where imperfect verifiers misrank candidates and erroneously prune all valid paths. These issues are further exacerbated in challenging and out-of-distribution problems, restricting search effectiveness. To mitigate verifier failures, we explore reducing reliance on verifiers and conduct preliminary investigations using two simple methods. Our findings reveal fundamental limitations in verifier-guided search and suggest future directions.
• 摘要：大型语言模型 (LLM) 难以进行多步推理，其中推理时间扩展已成为一种有前途的性能改进策略。当样本量通过选择和优先考虑有效的推理路径而受到限制时，验证器引导搜索的表现优于重复抽样。然而，我们发现了一个关键的限制：扩展缺陷，这种缺陷普遍存在于不同的模型（Mistral 7B 和 DeepSeekMath 7B）、基准（GSM8K 和 MATH）和验证器（结果价值模型和过程奖励模型）中。随着样本量的增加，验证器引导搜索的优势逐渐减弱，最终表现不如重复抽样。我们的分析将此归因于验证器故障，其中不完美的验证器会错误地对候选者进行排名并错误地修剪所有有效路径。这些问题在具有挑战性和分布不均的问题中进一步加剧，从而限制了搜索的有效性。为了减轻验证器故障的影响，我们探索减少对验证器的依赖并使用两种简单的方法进行初步调查。我们的研究结果揭示了验证者引导搜索的基本局限性并提出了未来的发展方向。

Title: Estimating LLM Uncertainty with Logits

• Authors: Huan Ma, Jingdong Chen, Guangyu Wang, Changqing Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.00290
• Pdf URL: https://arxiv.org/pdf/2502.00290
• Copy Paste: [[2502.00290]] Estimating LLM Uncertainty with Logits(https://arxiv.org/abs/2502.00290)
• Keywords: language model, llm, hallucination
• Abstract: In recent years, Large Language Models (LLMs) have seen remarkable advancements and have been extensively integrated across various fields. Despite their progress, LLMs are prone to hallucinations, producing responses that may not be dependable if the models lack sufficient grounding knowledge. To mitigate this issue, methods for estimating uncertainty have been adopted, with a focus on critical tokens as indicators of reliability. Nevertheless, probability-based approaches have shown limitations in assessing token-level reliability due to the erosion of evidence strength information acquired during training. In this paper, we introduce Logits-induced Token Uncertainty (LogU), a novel framework designed to estimate token-specific uncertainty in LLMs in real time, without the need for multiple sampling rounds. By leveraging evidence modeling for the implementation of LogU, we utilize the derived uncertainty measures to steer downstream tasks. Our experimental findings highlight the substantial effectiveness and potential of LogU, marking a significant advancement in addressing the challenge of model hallucinations.
• 摘要：近年来，大型语言模型 (LLM) 取得了显著的进步，并已广泛集成到各个领域。尽管取得了进展，但 LLM 容易出现幻觉，如果模型缺乏足够的基础知识，则产生的响应可能不可靠。为了缓解这个问题，已经采用了估计不确定性的方法，重点关注关键标记作为可靠性指标。然而，由于在训练期间获得的证据强度信息的侵蚀，基于概率的方法在评估标记级可靠性方面表现出局限性。在本文中，我们引入了 Logits 诱导的标记不确定性 (LogU)，这是一种新颖的框架，旨在实时估计 LLM 中特定于标记的不确定性，而无需多次采样。通过利用证据建模来实现 LogU，我们利用派生的不确定性度量来指导下游任务。我们的实验结果突出了 LogU 的实质性有效性和潜力，标志着在解决模型幻觉挑战方面取得了重大进展。

Title: ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference

• Authors: Xiang Liu, Zhenheng Tang, Peijie Dong, Zeyu Li, Bo Li, Xuming Hu, Xiaowen Chu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.00299
• Pdf URL: https://arxiv.org/pdf/2502.00299
• Copy Paste: [[2502.00299]] ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference(https://arxiv.org/abs/2502.00299)
• Keywords: language model, llm
• Abstract: To reduce memory costs in long-context inference with Large Language Models (LLMs), many recent works focus on compressing the key-value (KV) cache of different tokens. However, we identify that the previous KV cache compression methods measure token importance individually, neglecting the dependency between different tokens in the real-world language characterics. In light of this, we introduce ChunkKV, grouping the tokens in a chunk as a basic compressing unit, and retaining the most informative semantic chunks while discarding the less important ones. Furthermore, observing that ChunkKV exhibits higher similarity in the preserved indices across different layers, we propose layer-wise index reuse to further reduce computational overhead. We evaluated ChunkKV on cutting-edge long-context benchmarks including LongBench and Needle-In-A-HayStack, as well as the GSM8K and JailbreakV in-context learning benchmark. Our experiments with instruction tuning and multi-step reasoning (O1 and R1) LLMs, achieve up to 10\% performance improvement under aggressive compression ratios compared to existing methods.
• 摘要：为了减少使用大型语言模型 (LLM) 进行长上下文推理的内存成本，许多近期研究都侧重于压缩不同 token 的键值 (KV) 缓存。然而，我们发现之前的 KV 缓存压缩方法单独衡量 token 重要性，忽略了真实语言特征中不同 token 之间的依赖关系。鉴于此，我们引入了 ChunkKV，将 token 分组为块作为基本压缩单位，保留最具信息量的语义块而丢弃不太重要的块。此外，由于观察到 ChunkKV 在不同层之间保留的索引表现出更高的相似性，我们提出了逐层索引重用以进一步减少计算开销。我们在包括 LongBench 和 Needle-In-A-HayStack 在内的前沿长上下文基准以及 GSM8K 和 JailbreakV 上下文学习基准上评估了 ChunkKV。我们对指令调整和多步推理（O1 和 R1）LLM 进行的实验，与现有方法相比，在积极的压缩比下实现了高达 10% 的性能提升。

Title: Contextual Morphogenesis in Large Language Models: A Novel Approach to Self-Organizing Token Representations

• Authors: Alistair Dombrowski, Beatrix Engelhardt, Dimitri Fairbrother, Henry Evidail
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.00301
• Pdf URL: https://arxiv.org/pdf/2502.00301
• Copy Paste: [[2502.00301]] Contextual Morphogenesis in Large Language Models: A Novel Approach to Self-Organizing Token Representations(https://arxiv.org/abs/2502.00301)
• Keywords: language model
• Abstract: Token representations influence the efficiency and adaptability of language models, yet conventional tokenization strategies impose rigid segmentation boundaries that do not adjust dynamically to evolving contextual relationships. The introduction of contextual morphogenesis establishes a self-organizing mechanism that restructures token boundaries based on learned contextual dependencies, allowing embeddings to evolve progressively across iterative processing steps. Empirical evaluations demonstrate that dynamically adjusted tokenization contributes to reductions in perplexity while maintaining representational stability, particularly in linguistically complex domains where static segmentation fails to capture nuanced dependencies. Computational trade-offs associated with self-organizing token structures indicate that additional processing overhead remains within feasible limits, provided that optimization strategies account for segmentation update efficiency. Comparative assessments across different linguistic corpora suggest that adaptive tokenization preserves interpretability while improving alignment with contextual cues, reinforcing the potential of morphogenetic segmentation mechanisms to refine predictive accuracy. Stability analyses confirm that evolving token structures maintain consistent segmentation behaviors across varied text distributions, ensuring that representational adaptations remain linguistically coherent. The effectiveness of contextual morphogenesis in refining structural stability and predictive performance highlights its viability as an alternative to traditional tokenization methods. Further analysis of computational efficiency considerations suggests that hybrid strategies integrating both static and dynamic segmentation techniques may offer a balanced approach to optimizing representational flexibility while maintaining inference efficiency.
• 摘要：标记表示会影响语言模型的效率和适应性，但传统的标记化策略会施加刚性的分割边界，而这些边界不会根据不断变化的上下文关系进行动态调整。上下文形态发生的引入建立了一种自组织机制，该机制根据学习到的上下文依赖关系重构标记边界，从而允许嵌入在迭代处理步骤中逐步发展。实证评估表明，动态调整的标记化有助于减少困惑度，同时保持表示稳定性，特别是在语言复杂领域，静态分割无法捕捉细微的依赖关系。与自组织标记结构相关的计算权衡表明，只要优化策略考虑到分割更新效率，额外的处理开销仍然在可行的范围内。对不同语言语料库的比较评估表明，自适应标记化在提高与上下文线索的一致性的同时保留了可解释性，增​​强了形态发生分割机制提高预测准确性的潜力。稳定性分析证实，不断发展的标记结构在不同的文本分布中保持一致的分割行为，从而确保表征适应在语言上保持连贯性。语境形态发生在改善结构稳定性和预测性能方面的有效性凸显了其作为传统标记化方法的替代方案的可行性。对计算效率考虑因素的进一步分析表明，集成静态和动态分割技术的混合策略可以提供一种平衡的方法来优化表征灵活性，同时保持推理效率。

Title: DEUCE: Dual-diversity Enhancement and Uncertainty-awareness for Cold-start Active Learning

• Authors: Jiaxin Guo, C. L. Philip Chen, Shuzhen Li, Tong Zhang
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2502.00305
• Pdf URL: https://arxiv.org/pdf/2502.00305
• Copy Paste: [[2502.00305]] DEUCE: Dual-diversity Enhancement and Uncertainty-awareness for Cold-start Active Learning(https://arxiv.org/abs/2502.00305)
• Keywords: language model
• Abstract: Cold-start active learning (CSAL) selects valuable instances from an unlabeled dataset for manual annotation. It provides high-quality data at a low annotation cost for label-scarce text classification. However, existing CSAL methods overlook weak classes and hard representative examples, resulting in biased learning. To address these issues, this paper proposes a novel dual-diversity enhancing and uncertainty-aware (DEUCE) framework for CSAL. Specifically, DEUCE leverages a pretrained language model (PLM) to efficiently extract textual representations, class predictions, and predictive uncertainty. Then, it constructs a Dual-Neighbor Graph (DNG) to combine information on both textual diversity and class diversity, ensuring a balanced data distribution. It further propagates uncertainty information via density-based clustering to select hard representative instances. DEUCE performs well in selecting class-balanced and hard representative data by dual-diversity and informativeness. Experiments on six NLP datasets demonstrate the superiority and efficiency of DEUCE.
• 摘要：冷启动主动学习 (CSAL) 从未标记的数据集中选择有价值的实例进行手动注释。它以较低的注释成本为标签稀缺的文本分类提供高质量的数据。然而，现有的 CSAL 方法忽略了弱类和难代表性示例，导致学习出现偏差。为了解决这些问题，本文提出了一种用于 CSAL 的新型双重多样性增强和不确定性感知 (DEUCE) 框架。具体来说，DEUCE 利用预训练语言模型 (PLM) 来有效地提取文本表示、类别预测和预测不确定性。然后，它构建双邻域图 (DNG) 来结合文本多样性和类别多样性的信息，确保数据分布平衡。它进一步通过基于密度的聚类传播不确定性信息以选择难代表性实例。DEUCE 在通过双重多样性和信息量选择类别平衡和难代表性数据方面表现良好。在六个 NLP 数据集上的实验证明了 DEUCE 的优越性和效率。

Title: MODS: Moderating a Mixture of Document Speakers to Summarize Debatable Queries in Document Collections

• Authors: Nishant Balepur, Alexa Siu, Nedim Lipka, Franck Dernoncourt, Tong Sun, Jordan Boyd-Graber, Puneet Mathur
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2502.00322
• Pdf URL: https://arxiv.org/pdf/2502.00322
• Copy Paste: [[2502.00322]] MODS: Moderating a Mixture of Document Speakers to Summarize Debatable Queries in Document Collections(https://arxiv.org/abs/2502.00322)
• Keywords: llm
• Abstract: Query-focused summarization (QFS) gives a summary of documents to answer a query. Past QFS work assumes queries have one answer, ignoring debatable ones (Is law school worth it?). We introduce Debatable QFS (DQFS), a task to create summaries that answer debatable queries via documents with opposing perspectives; summaries must comprehensively cover all sources and balance perspectives, favoring no side. These goals elude LLM QFS systems, which: 1) lack structured content plans, failing to guide LLMs to write balanced summaries, and 2) use the same query to retrieve contexts across documents, failing to cover all perspectives specific to each document's content. To overcome this, we design MODS, a multi-LLM framework mirroring human panel discussions. MODS treats documents as individual Speaker LLMs and has a Moderator LLM that picks speakers to respond to tailored queries for planned topics. Speakers use tailored queries to retrieve relevant contexts from their documents and supply perspectives, which are tracked in a rich outline, yielding a content plan to guide the final summary. Experiments on ConflictingQA with controversial web queries and DebateQFS, our new dataset of debate queries from Debatepedia, show MODS beats SOTA by 38-59% in topic paragraph coverage and balance, based on new citation metrics. Users also find MODS's summaries to be readable and more balanced.
• 摘要：以查询为中心的摘要 (QFS) 提供文档摘要以回答查询。过去的 QFS 工作假设查询只有一个答案，而忽略有争议的答案（法学院值得吗？）。我们引入了可辩论 QFS (DQFS)，这是一项创建摘要的任务，该摘要通过具有对立观点的文档来回答有争议的查询；摘要必须全面涵盖所有来源并平衡观点，不偏袒任何一方。这些目标与 LLM QFS 系统相悖，该系统：1) 缺乏结构化的内容计划，无法指导 LLM 编写平衡的摘要，2) 使用相同的查询来检索文档中的上下文，无法涵盖每个文档内容的所有特定观点。为了克服这个问题，我们设计了 MODS，这是一个反映人类小组讨论的多 LLM 框架。MODS 将文档视为单独的 Speaker LLM，并有一个 Moderator LLM，它会挑选发言人来响应针对计划主题的定制查询。演讲者使用定制查询从他们的文档中检索相关上下文并提供观点，这些观点在丰富的大纲中进行跟踪，从而产生内容计划来指导最终摘要。在 ConflictingQA 上使用有争议的网络查询和 DebateQFS（我们从 Debatepedia 获得的辩论查询新数据集）进行的实验表明，基于新的引用指标，MODS 在主题段落覆盖率和平衡性方面比 SOTA 高出 38-59%。用户还发现 MODS 的摘要更易读且更平衡。

Title: UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models

• Authors: Xin Xu, Qiyun Xu, Tong Xiao, Tianhao Chen, Yuchen Yan, Jiaxin Zhang, Shizhe Diao, Can Yang, Yang Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.00334
• Pdf URL: https://arxiv.org/pdf/2502.00334
• Copy Paste: [[2502.00334]] UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models(https://arxiv.org/abs/2502.00334)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have demonstrated remarkable capabilities in solving complex reasoning tasks, particularly in mathematics. However, the domain of physics reasoning presents unique challenges that have received significantly less attention. Existing benchmarks often fall short in evaluating LLMs' abilities on the breadth and depth of undergraduate-level physics, underscoring the need for a comprehensive evaluation. To fill this gap, we introduce UGPhysics, a large-scale and comprehensive benchmark specifically designed to evaluate UnderGraduate-level Physics (UGPhysics) reasoning with LLMs. UGPhysics includes 5,520 undergraduate-level physics problems in both English and Chinese, covering 13 subjects with seven different answer types and four distinct physics reasoning skills, all rigorously screened for data leakage. Additionally, we develop a Model-Assistant Rule-based Judgment (MARJ) pipeline specifically tailored for assessing answer correctness of physics problems, ensuring accurate evaluation. Our evaluation of 31 leading LLMs shows that the highest overall accuracy, 49.8% (achieved by OpenAI-o1-mini), emphasizes the necessity for models with stronger physics reasoning skills, beyond math abilities. We hope UGPhysics, along with MARJ, will drive future advancements in AI for physics reasoning.
• 摘要：大型语言模型 (LLM) 在解决复杂推理任务（尤其是在数学中）方面表现出了卓越的能力。然而，物理推理领域面临着独特的挑战，而这些挑战受到的关注却少得多。现有的基准通常无法评估 LLM 在本科物理广度和深度方面的能力，因此需要进行全面的评估。为了填补这一空白，我们推出了 UGPhysics，这是一个大规模的综合基准，专门用于评估 LLM 的本科物理 (UGPhysics) 推理。UGPhysics 包括 5,520 个英文和中文本科物理问题，涵盖 13 个科目，有七种不同的答案类型和四种不同的物理推理技巧，所有这些都经过了严格的数据泄漏筛选。此外，我们还开发了一个模型辅助基于规则的判断 (MARJ) 流程，专门用于评估物理问题的答案正确性，确保评估准确。我们对 31 个领先的 LLM 的评估显示，总体准确率最高，为 49.8%（由 OpenAI-o1-mini 实现），这强调了模型需要具备比数学能力更强的物理推理能力。我们希望 UGPhysics 和 MARJ 能够推动未来物理推理 AI 的发展。

Title: Challenges and Innovations in LLM-Powered Fake News Detection: A Synthesis of Approaches and Future Directions

• Authors: Jingyuan Yi, Zeqiu Xu, Tianyi Huang, Peiyang Yu
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2502.00339
• Pdf URL: https://arxiv.org/pdf/2502.00339
• Copy Paste: [[2502.00339]] Challenges and Innovations in LLM-Powered Fake News Detection: A Synthesis of Approaches and Future Directions(https://arxiv.org/abs/2502.00339)
• Keywords: language model, llm
• Abstract: The pervasiveness of the dissemination of fake news through social media platforms poses critical risks to the trust of the general public, societal stability, and democratic institutions. This challenge calls for novel methodologies in detection, which can keep pace with the dynamic and multi-modal nature of misinformation. Recent works include powering the detection using large language model advances in multimodal frameworks, methodologies using graphs, and adversarial training in the literature of fake news. Based on the different approaches which can bring success, some key highlights will be underlined: enhanced LLM-improves accuracy through more advanced semantics and cross-modality fusion for robust detections. The review further identifies critical gaps in adaptability to dynamic social media trends, real-time, and cross-platform detection capabilities, as well as the ethical challenges thrown up by the misuse of LLMs. Future directions underline the development of style-agnostic models, cross-lingual detection frameworks, and robust policies with a view to mitigating LLM-driven misinformation. This synthesis thus lays a concrete foundation for those researchers and practitioners committed to reinforcing fake news detection systems with complications that keep on growing in the digital landscape.
• 摘要：通过社交媒体平台传播的虚假新闻无处不在，对公众的信任、社会稳定和民主制度构成了重大风险。这一挑战要求检测中采用新颖的方法，以跟上虚假信息的动态和多模态性质。最近的研究包括使用大型语言模型为检测提供动力、多模态框架的进步、使用图表的方法以及虚假新闻文献中的对抗性训练。基于可以带来成功的不同方法，将强调一些关键亮点：增强的 LLM - 通过更高级的语义和跨模态融合提高准确性，以实现稳健的检测。该评论进一步确定了适应动态社交媒体趋势、实时和跨平台检测能力以及滥用 LLM 所带来的道德挑战方面的关键差距。未来的方向强调开发与风格无关的模型、跨语言检测框架和稳健的政策，以减轻 LLM 驱动的虚假信息。因此，这种综合作用为致力于加强数字环境中日益复杂的假新闻检测系统的研究人员和从业人员奠定了坚实的基础。

Title: FinchGPT: a Transformer based language model for birdsong analysis

• Authors: Kosei Kobayashi, Kosuke Matsuzaki, Masaya Taniguchi, Keisuke Sakaguchi, Kentaro Inui, Kentaro Abe
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.00344
• Pdf URL: https://arxiv.org/pdf/2502.00344
• Copy Paste: [[2502.00344]] FinchGPT: a Transformer based language model for birdsong analysis(https://arxiv.org/abs/2502.00344)
• Keywords: language model, gpt, llm
• Abstract: The long-range dependencies among the tokens, which originate from hierarchical structures, are a defining hallmark of human language. However, whether similar dependencies exist within the sequential vocalization of non-human animals remains a topic of investigation. Transformer architectures, known for their ability to model long-range dependencies among tokens, provide a powerful tool for investigating this phenomenon. In this study, we employed the Transformer architecture to analyze the songs of Bengalese finch (Lonchura striata domestica), which are characterized by their highly variable and complex syllable sequences. To this end, we developed FinchGPT, a Transformer-based model trained on a textualized corpus of birdsongs, which outperformed other architecture models in this domain. Attention weight analysis revealed that FinchGPT effectively captures long-range dependencies within syllables sequences. Furthermore, reverse engineering approaches demonstrated the impact of computational and biological manipulations on its performance: restricting FinchGPT's attention span and disrupting birdsong syntax through the ablation of specific brain nuclei markedly influenced the model's outputs. Our study highlights the transformative potential of large language models (LLMs) in deciphering the complexities of animal vocalizations, offering a novel framework for exploring the structural properties of non-human communication systems while shedding light on the computational distinctions between biological brains and artificial neural networks.
• 摘要：源自层级结构的标记之间的长距离依赖关系是人类语言的标志性特征。然而，非人类动物的连续发声中是否存在类似的依赖关系仍是一个有待研究的课题。Transformer 架构以其能够对标记之间的长距离依赖关系进行建模而闻名，为研究这一现象提供了强大的工具。在本研究中，我们采用 Transformer 架构分析了孟加拉雀 (Lonchura striata domestica) 的歌曲，这种鸟的音节序列变化多端且复杂。为此，我们开发了 FinchGPT，这是一个基于 Transformer 的模型，该模型在文本化的鸟鸣语料库上进行训练，其表现优于该领域的其他架构模型。注意力权重分析表明，FinchGPT 可以有效捕捉音节序列内的长距离依赖关系。此外，逆向工程方法证明了计算和生物操纵对其性能的影响：限制 FinchGPT 的注意力范围并通过消融特定脑核来破坏鸟鸣语法，显著影响了模型的输出。我们的研究强调了大型语言模型 (LLM) 在解读动物发声复杂性方面的变革潜力，为探索非人类通信系统的结构特性提供了一个新颖的框架，同时揭示了生物大脑和人工神经网络之间的计算区别。

Title: The Impact of Persona-based Political Perspectives on Hateful Content Detection

• Authors: Stefano Civelli, Pietro Bernardelle, Gianluca Demartini
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.00385
• Pdf URL: https://arxiv.org/pdf/2502.00385
• Copy Paste: [[2502.00385]] The Impact of Persona-based Political Perspectives on Hateful Content Detection(https://arxiv.org/abs/2502.00385)
• Keywords: language model, llm, prompt
• Abstract: While pretraining language models with politically diverse content has been shown to improve downstream task fairness, such approaches require significant computational resources often inaccessible to many researchers and organizations. Recent work has established that persona-based prompting can introduce political diversity in model outputs without additional training. However, it remains unclear whether such prompting strategies can achieve results comparable to political pretraining for downstream tasks. We investigate this question using persona-based prompting strategies in multimodal hate-speech detection tasks, specifically focusing on hate speech in memes. Our analysis reveals that when mapping personas onto a political compass and measuring persona agreement, inherent political positioning has surprisingly little correlation with classification decisions. Notably, this lack of correlation persists even when personas are explicitly injected with stronger ideological descriptors. Our findings suggest that while LLMs can exhibit political biases in their responses to direct political questions, these biases may have less impact on practical classification tasks than previously assumed. This raises important questions about the necessity of computationally expensive political pretraining for achieving fair performance in downstream tasks.
• 摘要：虽然已经证明使用政治多元化内容对语言模型进行预训练可以提高下游任务的公平性，但这种方法需要大量的计算资源，而许多研究人员和组织通常无法获得这些资源。最近的研究表明，基于角色的提示可以在模型输出中引入政治多样性，而无需额外的训练。然而，目前尚不清楚这种提示策略是否可以实现与下游任务的政治预训练相当的结果。我们在多模态仇恨言论检测任务中使用基于角色的提示策略来研究这个问题，特别是关注模因中的仇恨言论。我们的分析表明，当将角色映射到政治指南针上并衡量角色一致性时，固有的政治立场与分类决策几乎没有相关性。值得注意的是，即使角色明确注入了更强的意识形态描述符，这种缺乏相关性的情况仍然存在。我们的研究结果表明，虽然 LLM 在回答直接的政治问题时可能会表现出政治偏见，但这些偏见对实际分类任务的影响可能比以前假设的要小。这提出了一个重要的问题，即计算成本高昂的政治预训练是否有必要在下游任务中实现公平的表现。

Title: Social media polarization during conflict: Insights from an ideological stance dataset on Israel-Palestine Reddit comments

• Authors: Hasin Jawad Ali, Ajwad Abrar, S.M. Hozaifa Hossain, M. Firoz Mridha
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.00414
• Pdf URL: https://arxiv.org/pdf/2502.00414
• Copy Paste: [[2502.00414]] Social media polarization during conflict: Insights from an ideological stance dataset on Israel-Palestine Reddit comments(https://arxiv.org/abs/2502.00414)
• Keywords: language model, llm, prompt
• Abstract: In politically sensitive scenarios like wars, social media serves as a platform for polarized discourse and expressions of strong ideological stances. While prior studies have explored ideological stance detection in general contexts, limited attention has been given to conflict-specific settings. This study addresses this gap by analyzing 9,969 Reddit comments related to the Israel-Palestine conflict, collected between October 2023 and August 2024. The comments were categorized into three stance classes: Pro-Israel, Pro-Palestine, and Neutral. Various approaches, including machine learning, pre-trained language models, neural networks, and prompt engineering strategies for open source large language models (LLMs), were employed to classify these stances. Performance was assessed using metrics such as accuracy, precision, recall, and F1-score. Among the tested methods, the Scoring and Reflective Re-read prompt in Mixtral 8x7B demonstrated the highest performance across all metrics. This study provides comparative insights into the effectiveness of different models for detecting ideological stances in highly polarized social media contexts. The dataset used in this research is publicly available for further exploration and validation.
• 摘要：在战争等政治敏感场景中，社交媒体成为两极化话语和强烈意识形态立场表达的平台。虽然先前的研究已经探索了一般背景下的意识形态立场检测，但对特定冲突环境的关注有限。本研究通过分析 2023 年 10 月至 2024 年 8 月期间收集的 9,969 条与巴以冲突有关的 Reddit 评论来解决这一差距。这些评论被分为三个立场类别：亲以色列、亲巴勒斯坦和中立。采用各种方法对这些立场进行分类，包括机器学习、预训练语言模型、神经网络和开源大型语言模型 (LLM) 的提示工程策略。使用准确度、精确度、召回率和 F1 分数等指标来评估性能。在测试的方法中，Mi​​xtral 8x7B 中的评分和反思重读提示在所有指标中表现出最高的性能。本研究对不同模型在高度两极化的社交媒体环境中检测意识形态立场的有效性进行了比较分析。本研究中使用的数据集可供进一步探索和验证。

Title: UniAttn: Reducing Inference Costs via Softmax Unification for Post-Training LLMs

• Authors: Yizhe Xiong, Wei Huang, Xin Ye, Hui Chen, Zijia Lin, Haoran Lian, Zhenpeng Su, Jungong Han, Guiguang Ding
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.00439
• Pdf URL: https://arxiv.org/pdf/2502.00439
• Copy Paste: [[2502.00439]] UniAttn: Reducing Inference Costs via Softmax Unification for Post-Training LLMs(https://arxiv.org/abs/2502.00439)
• Keywords: language model, llm
• Abstract: Post-training is essential for adapting Large Language Models (LLMs) to real-world applications. Deploying post-trained models faces significant challenges due to substantial memory overhead and noticeable inference latency. Existing work has identified significant redundancies in LLMs and proposed efficient architectures, namely intra-layer KV sharing and cross-layer KV sharing. However, intra-layer KV sharing still results in high inference costs, while cross-layer KV sharing leads to significant performance degradation. As a result, both methods remain suboptimal for post-training pre-trained LLMs. In this paper, we identify that the \texttt{Softmax} operation is a primary bottleneck for LLM inference and discover that it is actually highly redundant during post-training. We propose Softmax \textbf{Uni}fication in \textbf{Att}e\textbf{n}tion (\textbf{UniAttn}), a novel post-training method that unifies Softmax activations across transformer blocks to reduce LLM inference costs. Additionally, UniAttn adopts a linear projection to compensate for the errors induced by Softmax unification. Experiments show that UniAttn matches the performance of standard post-training while significantly reducing inference costs, outperforming existing efficient architectures during post-training. Our code will be available at \url{this https URL}.
• 摘要：后训练对于将大型语言模型 (LLM) 应用于实际应用至关重要。由于内存开销大和推理延迟明显，部署后训练模型面临重大挑战。现有工作已发现 LLM 中存在大量冗余，并提出了高效的架构，即层内 KV 共享和跨层 KV 共享。然而，层内 KV 共享仍然会导致高推理成本，而跨层 KV 共享会导致性能显著下降。因此，这两种方法对于后训练预训练的 LLM 仍然不是最优的。在本文中，我们发现 \texttt{Softmax} 操作是 LLM 推理的主要瓶颈，并发现它在后训练期间实际上是高度冗余的。我们提出了 Softmax \textbf{Uni}fication in \textbf{Att}e\textbf{n}tion (\textbf{UniAttn})，这是一种新颖的后训练方法，它将各个 Transformer 块中的 Softmax 激活统一起来，以降低 LLM 推理成本。此外，UniAttn 采用线性投影来补偿 Softmax 统一引起的误差。实验表明，UniAttn 的性能与标准后训练相当，同时显著降低了推理成本，在后训练期间的表现优于现有的高效架构。我们的代码将在 \url{此 https URL} 上提供。

Title: HERA: Improving Long Document Summarization using Large Language Models with Context Packaging and Reordering

• Authors: Taiji Li, Hao Chen, Fei Yu, Yin Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.00448
• Pdf URL: https://arxiv.org/pdf/2502.00448
• Copy Paste: [[2502.00448]] HERA: Improving Long Document Summarization using Large Language Models with Context Packaging and Reordering(https://arxiv.org/abs/2502.00448)
• Keywords: language model, llm
• Abstract: Despite the rapid growth of context length of large language models (LLMs) , LLMs still perform poorly in long document summarization. An important reason for this is that relevant information about an event is scattered throughout long documents, and the messy narrative order impairs the accurate understanding and utilization of LLMs for long documents. To address these issues, we propose a novel summary generation framework, called HERA. Specifically, we first segment a long document by its semantic structure and retrieve text segments about the same event, and finally reorder them to form the input context. We evaluate our approach on two long document summarization datasets. The experimental results show that HERA outperforms foundation models in ROUGE, BERTScore and faithfulness metrics, while HERA does not require additional fine-tuning and resources.
• 摘要：尽管大型语言模型 (LLM) 的上下文长度快速增长，但 LLM 在长文档摘要方面仍然表现不佳。其中一个重要原因是有关事件的相关信息分散在长文档中，混乱的叙述顺序影响了 LLM 对长文档的准确理解和利用。为了解决这些问题，我们提出了一个新颖的摘要生成框架，称为 HERA。具体而言，我们首先根据长文档的语义结构对其进行分词并检索有关同一事件的文本片段，最后对它们重新排序以形成输入上下文。我们在两个长文档摘要数据集上评估了我们的方法。实验结果表明，HERA 在 ROUGE、BERTScore 和忠诚度指标方面优于基础模型，并且 HERA 不需要额外的微调和资源。

Title: A statistically consistent measure of Semantic Variability using Language Models

• Authors: Yi Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.00507
• Pdf URL: https://arxiv.org/pdf/2502.00507
• Copy Paste: [[2502.00507]] A statistically consistent measure of Semantic Variability using Language Models(https://arxiv.org/abs/2502.00507)
• Keywords: language model
• Abstract: To address the issue of variability in the output generated by a language model, we present a measure of semantic variability that is statistically consistent under mild assumptions. This measure, denoted as semantic spectral entropy, is a easy to implement algorithm that requires just off the shelf language models. We put very few restrictions on the language models and we have shown in a clear simulation studies that such method can generate accurate metric despite randomness that arise from the language models.
• 摘要：为了解决语言模型生成的输出结果的可变性问题，我们提出了一种在温和假设下具有统计一致性的语义可变性度量。这种度量称为语义谱熵，是一种易于实现的算法，只需要现成的语言模型即可。我们对语言模型的限制很少，并且我们在清晰的模拟研究中表明，尽管语言模型存在随机性，但这种方法仍可以生成准确的度量。

Title: Detecting Ambiguities to Guide Query Rewrite for Robust Conversations in Enterprise AI Assistants

• Authors: Md Mehrab Tanjim, Xiang Chen, Victor S. Bursztyn, Uttaran Bhattacharya, Tung Mai, Vaishnavi Muppala, Akash Maharaj, Saayan Mitra, Eunyee Koh, Yunyao Li, Ken Russell
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.00537
• Pdf URL: https://arxiv.org/pdf/2502.00537
• Copy Paste: [[2502.00537]] Detecting Ambiguities to Guide Query Rewrite for Robust Conversations in Enterprise AI Assistants(https://arxiv.org/abs/2502.00537)
• Keywords: llm
• Abstract: Multi-turn conversations with an Enterprise AI Assistant can be challenging due to conversational dependencies in questions, leading to ambiguities and errors. To address this, we propose an NLU-NLG framework for ambiguity detection and resolution through reformulating query automatically and introduce a new task called "Ambiguity-guided Query Rewrite." To detect ambiguities, we develop a taxonomy based on real user conversational logs and draw insights from it to design rules and extract features for a classifier which yields superior performance in detecting ambiguous queries, outperforming LLM-based baselines. Furthermore, coupling the query rewrite module with our ambiguity detecting classifier shows that this end-to-end framework can effectively mitigate ambiguities without risking unnecessary insertions of unwanted phrases for clear queries, leading to an improvement in the overall performance of the AI Assistant. Due to its significance, this has been deployed in the real world application, namely Adobe Experience Platform AI Assistant.
• 摘要：由于问题中的对话依赖性，与企业 AI 助手进行多轮对话可能具有挑战性，从而导致歧义和错误。为了解决这个问题，我们提出了一个 NLU-NLG 框架，通过自动重新制定查询来检测和解决歧义，并引入了一项名为“歧义引导查询重写”的新任务。为了检测歧义，我们开发了一个基于真实用户对话日志的分类法，并从中汲取见解来设计规则并提取分类器的特征，该分类器在检测歧义查询方面表现出色，优于基于 LLM 的基线。此外，将查询重写模块与我们的歧义检测分类器相结合表明，这个端到端框架可以有效地缓解歧义，而不会冒着为清晰查询不必要地插入不需要的短语的风险，从而提高 AI 助手的整体性能。由于其重要性，它已在现实世界的应用程序中部署，即 Adob​​e Experience Platform AI Assistant。

Title: M+: Extending MemoryLLM with Scalable Long-Term Memory

• Authors: Yu Wang, Dmitry Krotov, Yuanzhe Hu, Yifan Gao, Wangchunshu Zhou, Julian McAuley, Dan Gutfreund, Rogerio Feris, Zexue He
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.00592
• Pdf URL: https://arxiv.org/pdf/2502.00592
• Copy Paste: [[2502.00592]] M+: Extending MemoryLLM with Scalable Long-Term Memory(https://arxiv.org/abs/2502.00592)
• Keywords: language model, llm
• Abstract: Equipping large language models (LLMs) with latent-space memory has attracted increasing attention as they can extend the context window of existing language models. However, retaining information from the distant past remains a challenge. For example, MemoryLLM (Wang et al., 2024a), as a representative work with latent-space memory, compresses past information into hidden states across all layers, forming a memory pool of 1B parameters. While effective for sequence lengths up to 16k tokens, it struggles to retain knowledge beyond 20k tokens. In this work, we address this limitation by introducing M+, a memory-augmented model based on MemoryLLM that significantly enhances long-term information retention. M+ integrates a long-term memory mechanism with a co-trained retriever, dynamically retrieving relevant information during text generation. We evaluate M+ on diverse benchmarks, including long-context understanding and knowledge retention tasks. Experimental results show that M+ significantly outperforms MemoryLLM and recent strong baselines, extending knowledge retention from under 20k to over 160k tokens with similar GPU memory overhead.
• 摘要：为大型语言模型 (LLM) 配备潜在空间记忆已引起越来越多的关注，因为它们可以扩展现有语言模型的上下文窗口。然而，保留遥远过去的信息仍然是一个挑战。例如，MemoryLLM (Wang 等人，2024a) 作为潜在空间记忆的代表作品，将过去的信息压缩到所有层的隐藏状态中，形成 1B 参数的记忆池。虽然它对长达 16k 个标记的序列长度有效，但它很难保留超过 20k 个标记的知识。在这项工作中，我们通过引入 M+ 来解决这一限制，M+ 是一种基于 MemoryLLM 的记忆增强模型，可显著增强长期信息保留。M+ 将长期记忆机制与共同训练的检索器相结合，在文本生成过程中动态检索相关信息。我们在各种基准上评估 M+，包括长期上下文理解和知识保留任务。实验结果表明，M+ 的表现明显优于 MemoryLLM 和最近的强基线，在 GPU 内存开销相似的情况下，将知识保留从 20k 以下扩展到 160k 以上。

Title: RPGBENCH: Evaluating Large Language Models as Role-Playing Game Engines

• Authors: Pengfei Yu, Dongming Shen, Silin Meng, Jaewon Lee, Weisu Yin, Andrea Yaoyun Cui, Zhenlin Xu, Yi Zhu, Xingjian Shi, Mu Li, Alex Smola
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.00595
• Pdf URL: https://arxiv.org/pdf/2502.00595
• Copy Paste: [[2502.00595]] RPGBENCH: Evaluating Large Language Models as Role-Playing Game Engines(https://arxiv.org/abs/2502.00595)
• Keywords: language model, llm
• Abstract: We present RPGBench, the first benchmark designed to evaluate large language models (LLMs) as text-based role-playing game (RPG) engines. RPGBench comprises two core tasks: Game Creation (GC) and Game Simulation (GS). In GC, an LLM must craft a valid and playable RPG world using a structured event-state representation, ensuring logical coherence and proper termination conditions. In GS, the LLM simulates interactive gameplay across multiple rounds while consistently updating states and enforcing game rules. To comprehensively assess performance, RPGBench integrates objective and subjective evaluation methodologies. Objective measures verify adherence to event mechanics and check variable updates without requiring human intervention. Subjective measures, such as content interestingness, action quality, and role-playing capability, are evaluated via an LLM-as-a-judge framework, where a strong LLM grades each candidate's outputs. Empirical results demonstrate that state-of-the-art LLMs can produce engaging stories but often struggle to implement consistent, verifiable game mechanics, particularly in long or complex scenarios. By combining structured, rule-based assessments with LLM-based judgments, RPGBench provides a new standard for evaluating how well LLMs can balance creativity, coherence, and complexity in text-based RPGs, opening avenues for more immersive and controllable interactive storytelling.
• 摘要：我们推出了 RPGBench，这是第一个旨在评估大型语言模型 (LLM) 作为基于文本的角色扮演游戏 (RPG) 引擎的基准。RPGBench 包含两个核心任务：游戏创作 (GC) 和游戏模拟 (GS)。在 GC 中，LLM 必须使用结构化的事件状态表示来构建一个有效且可玩的 RPG 世界，确保逻辑连贯性和适当的终止条件。在 GS 中，LLM 模拟跨多轮的交互式游戏玩法，同时持续更新状态并执行游戏规则。为了全面评估性能，RPGBench 集成了客观和主观评估方法。客观指标验证是否遵守事件机制并检查变量更新，而无需人工干预。主观指标，例如内容趣味性、动作质量和角色扮演能力，通过 LLM 作为评判框架进行评估，其中强大的 LLM 会对每个候选人的输出进行评分。实证结果表明，最先进的 LLM 可以创作引人入胜的故事，但往往难以实现一致、可验证的游戏机制，尤其是在长期或复杂的场景中。通过将结构化、基于规则的评估与基于 LLM 的判断相结合，RPGBench 为评估 LLM 在基于文本的 RPG 中如何平衡创造力、连贯性和复杂性提供了新标准，为更具沉浸感和可控性的交互式故事叙述开辟了道路。

Title: Mitigating Heterogeneous Token Overfitting in LLM Knowledge Editing

• Authors: Tianci Liu, Zihan Dong, Linjun Zhang, Haoyu Wang, Jing Gao
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.00602
• Pdf URL: https://arxiv.org/pdf/2502.00602
• Copy Paste: [[2502.00602]] Mitigating Heterogeneous Token Overfitting in LLM Knowledge Editing(https://arxiv.org/abs/2502.00602)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have achieved remarkable performance on various natural language tasks. However, they are trained on static corpora and their knowledge can become outdated quickly in the fast-changing world. This motivates the development of knowledge editing (KE) to update specific knowledge in LLMs without changing unrelated others or compromising their pre-trained capabilities. Previous efforts sought to update a small amount of parameters of a LLM and proved effective for making selective updates. Nonetheless, the edited LLM often exhibits degraded ability to reason about the new knowledge. In this work, we identify a key issue: heterogeneous token overfitting (HTO), where the LLM overfits different tokens in the provided knowledge at varying rates. To tackle this, we propose OVERTONE, a token-level smoothing method that mitigates HTO by adaptively refining the target distribution. Theoretically, OVERTONE offers better parameter updates with negligible computation overhead. It also induces an implicit DPO but does not require preference data pairs. Extensive experiments across four editing methods, two LLMs, and diverse scenarios demonstrate the effectiveness and versatility of our method.
• 摘要：大型语言模型 (LLM) 在各种自然语言任务上都取得了显著的成绩。然而，它们是在静态语料库上训练的，在瞬息万变的世界中，它们的知识很快就会过时。这促使人们开发知识编辑 (KE) 来更新 LLM 中的特定知识，而不会更改不相关的其他知识或损害其预训练能力。之前的努力试图更新 LLM 的少量参数，并被证明可以有效地进行选择性更新。尽管如此，编辑后的 ​​LLM 通常会表现出对新知识的推理能力下降。在这项工作中，我们发现了一个关键问题：异构标记过度拟合 (HTO)，其中 LLM 以不同的速率过度拟合所提供知识中的不同标记。为了解决这个问题，我们提出了 OVERTONE，这是一种标记级平滑方法，通过自适应地细化目标分布来缓解 HTO。从理论上讲，OVERTONE 提供了更好的参数更新，计算开销可以忽略不计。它还引入了隐式 DPO，但不需要偏好数据对。针对四种编辑方法、两个 LLM 和不同场景进行的大量实验证明了我们方法的有效性和多功能性。

Title: Efficient Language Modeling for Low-Resource Settings with Hybrid RNN-Transformer Architectures

• Authors: Gabriel Lindenmaier, Sean Papay, Sebastian Padó
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.00617
• Pdf URL: https://arxiv.org/pdf/2502.00617
• Copy Paste: [[2502.00617]] Efficient Language Modeling for Low-Resource Settings with Hybrid RNN-Transformer Architectures(https://arxiv.org/abs/2502.00617)
• Keywords: language model
• Abstract: Transformer-based language models have recently been at the forefront of active research in text generation. However, these models' advances come at the price of prohibitive training costs, with parameter counts in the billions and compute requirements measured in petaflop/s-decades. In this paper, we investigate transformer-based architectures for improving model performance in a low-data regime by selectively replacing attention layers with feed-forward and quasi-recurrent neural network layers. We test these architectures on the standard Enwik8 and Wikitext-103 corpora. Our results show that our reduced architectures outperform existing models with a comparable number of parameters, and obtain comparable performance to larger models while significantly reducing the number of parameters.
• 摘要：基于 Transformer 的语言模型最近一直处于文本生成研究的前沿。然而，这些模型的进步是以高昂的训练成本为代价的，其参数数量以数十亿计，计算需求以千万亿次/秒为单位。在本文中，我们研究了基于 Transformer 的架构，通过选择性地用前馈和准循环神经网络层替换注意力层来提高低数据环境下的模型性能。我们在标准 Enwik8 和 Wikitext-103 语料库上测试了这些架构。结果表明，我们的简化架构优于具有可比参数数量的现有模型，并且在显著减少参数数量的同时获得了与较大模型相当的性能。

Title: SimulPL: Aligning Human Preferences in Simultaneous Machine Translation

• Authors: Donglei Yu, Yang Zhao, Jie Zhu, Yangyifan Xu, Yu Zhou, Chengqing Zong
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.00634
• Pdf URL: https://arxiv.org/pdf/2502.00634
• Copy Paste: [[2502.00634]] SimulPL: Aligning Human Preferences in Simultaneous Machine Translation(https://arxiv.org/abs/2502.00634)
• Keywords: gpt, prompt
• Abstract: Simultaneous Machine Translation (SiMT) generates translations while receiving streaming source inputs. This requires the SiMT model to learn a read/write policy, deciding when to translate and when to wait for more source input. Numerous linguistic studies indicate that audiences in SiMT scenarios have distinct preferences, such as accurate translations, simpler syntax, and no unnecessary latency. Aligning SiMT models with these human preferences is crucial to improve their performances. However, this issue still remains unexplored. Additionally, preference optimization for SiMT task is also challenging. Existing methods focus solely on optimizing the generated responses, ignoring human preferences related to latency and the optimization of read/write policy during the preference optimization phase. To address these challenges, we propose Simultaneous Preference Learning (SimulPL), a preference learning framework tailored for the SiMT task. In the SimulPL framework, we categorize SiMT human preferences into five aspects: \textbf{translation quality preference}, \textbf{monotonicity preference}, \textbf{key point preference}, \textbf{simplicity preference}, and \textbf{latency preference}. By leveraging the first four preferences, we construct human preference prompts to efficiently guide GPT-4/4o in generating preference data for the SiMT task. In the preference optimization phase, SimulPL integrates \textbf{latency preference} into the optimization objective and enables SiMT models to improve the read/write policy, thereby aligning with human preferences more effectively. Experimental results indicate that SimulPL exhibits better alignment with human preferences across all latency levels in Zh$\rightarrow$En, De$\rightarrow$En and En$\rightarrow$Zh SiMT tasks. Our data and code will be available at \url{this https URL}.
• 摘要：同步机器翻译 (SiMT) 在接收流式源输入的同时生成翻译。这要求 SiMT 模型学习读/写策略，决定何时翻译以及何时等待更多源输入。大量语言学研究表明，SiMT 场景中的受众有不同的偏好，例如准确的翻译、更简单的语法以及没有不必要的延迟。将 SiMT 模型与这些人类偏好相结合对于提高其性能至关重要。然而，这个问题仍然未被探索。此外，SiMT 任务的偏好优化也具有挑战性。现有方法仅关注优化生成的响应，忽略了与延迟相关的人类偏好以及偏好优化阶段的读/写策略优化。为了应对这些挑战，我们提出了同步偏好学习 (SimulPL)，这是一个为 SiMT 任务量身定制的偏好学习框架。在 SimulPL 框架中，我们将 SiMT 人类偏好分为五个方面：\textbf{翻译质量偏好}、\textbf{单调性偏好}、\textbf{关键点偏好}、\textbf{简单性偏好} 和 \textbf{延迟偏好}。通过利用前四个偏好，我们构建了人类偏好提示，以有效地指导 GPT-4/4o 为 SiMT 任务生成偏好数据。在偏好优化阶段，SimulPL 将 \textbf{延迟偏好} 集成到优化目标中，并使 SiMT 模型能够改进读写策略，从而更有效地与人类偏好保持一致。实验结果表明，SimulPL 在 Zh$\rightarrow$En、De$\rightarrow$En 和 En$\rightarrow$Zh SiMT 任务中的所有延迟级别上都表现出与人类偏好的更好一致性。我们的数据和代码将在 \url{此 https URL} 上提供。

Title: Evaluating Small Language Models for News Summarization: Implications and Factors Influencing Performance

• Authors: Borui Xu, Yao Chen, Zeyi Wen, Weiguo Liu, Bingsheng He
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.00641
• Pdf URL: https://arxiv.org/pdf/2502.00641
• Copy Paste: [[2502.00641]] Evaluating Small Language Models for News Summarization: Implications and Factors Influencing Performance(https://arxiv.org/abs/2502.00641)
• Keywords: language model, llm, prompt
• Abstract: The increasing demand for efficient summarization tools in resource-constrained environments highlights the need for effective solutions. While large language models (LLMs) deliver superior summarization quality, their high computational resource requirements limit practical use applications. In contrast, small language models (SLMs) present a more accessible alternative, capable of real-time summarization on edge devices. However, their summarization capabilities and comparative performance against LLMs remain underexplored. This paper addresses this gap by presenting a comprehensive evaluation of 19 SLMs for news summarization across 2,000 news samples, focusing on relevance, coherence, factual consistency, and summary length. Our findings reveal significant variations in SLM performance, with top-performing models such as Phi3-Mini and Llama3.2-3B-Ins achieving results comparable to those of 70B LLMs while generating more concise summaries. Notably, SLMs are better suited for simple prompts, as overly complex prompts may lead to a decline in summary quality. Additionally, our analysis indicates that instruction tuning does not consistently enhance the news summarization capabilities of SLMs. This research not only contributes to the understanding of SLMs but also provides practical insights for researchers seeking efficient summarization solutions that balance performance and resource use.
• 摘要：在资源受限的环境中，对高效摘要工具的需求日益增长，这凸显了对有效解决方案的需求。虽然大型语言模型 (LLM) 提供了卓越的摘要质量，但它们的高计算资源要求限制了实际应用。相比之下，小型语言模型 (SLM) 提供了一种更易于访问的替代方案，能够在边缘设备上进行实时摘要。然而，它们的摘要能力和与 LLM 的比较性能仍未得到充分探索。本文通过对 2,000 个新闻样本中的 19 个 SLM 新闻摘要进行全面评估来解决这一差距，重点关注相关性、连贯性、事实一致性和摘要长度。我们的研究结果显示 SLM 性能存在显著差异，表现最佳的模型（如 Phi3-Mini 和 Llama3.2-3B-Ins）实现了与 70B LLM 相当的结果，同时生成了更简洁的摘要。值得注意的是，SLM 更适合简单的提示，因为过于复杂的提示可能会导致摘要质量下降。此外，我们的分析表明，指令调整并不能持续增强 SLM 的新闻摘要能力。这项研究不仅有助于理解 SLM，还为寻求平衡性能和资源利用的高效摘要解决方案的研究人员提供了实用见解。

Title: Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models Beneficial?

• Authors: Wenzhe Li, Yong Lin, Mengzhou Xia, Chi Jin
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.00674
• Pdf URL: https://arxiv.org/pdf/2502.00674
• Copy Paste: [[2502.00674]] Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models Beneficial?(https://arxiv.org/abs/2502.00674)
• Keywords: language model, llm, agent
• Abstract: Ensembling outputs from diverse sources is a straightforward yet effective approach to boost performance. Mixture-of-Agents (MoA) is one such popular ensemble method that aggregates outputs from multiple different Large Language Models (LLMs). This paper raises the question in the context of language models: is mixing different LLMs truly beneficial? We propose Self-MoA -- an ensemble method that aggregates outputs from only the single top-performing LLM. Our extensive experiments reveal that, surprisingly, Self-MoA outperforms standard MoA that mixes different LLMs in a large number of scenarios: Self-MoA achieves $6.6\%$ improvement over MoA on the AlpacaEval 2.0 benchmark, and an average of $3.8\%$ improvement across various benchmarks, including MMLU, CRUX, and MATH. Applying Self-MoA to one of the top-ranking models in AlpacaEval 2.0 directly achieves the new state-of-the-art performance on the leaderboard. To understand the effectiveness of Self-MoA, we systematically investigate the trade-off between diversity and quality of outputs under various MoA settings. We confirm that the MoA performance is rather sensitive to the quality, and mixing different LLMs often lowers the average quality of the models. To complement the study, we identify the scenarios where mixing different LLMs could be helpful. This paper further introduces a sequential version of Self-MoA, that is capable of aggregating a large number of LLM outputs on-the-fly over multiple rounds, and is as effective as aggregating all outputs at once.
• 摘要：集成来自不同来源的输出是一种直接而有效的提高性能的方法。混合代理 (MoA) 就是这样一种流行的集成方法，它聚合了来自多个不同大型语言模型 (LLM) 的输出。本文在语言模型的背景下提出了一个问题：混合不同的 LLM 真的有益吗？我们提出了 Self-MoA——一种集成方法，它只聚合来自单个表现最佳的 LLM 的输出。我们大量的实验表明，令人惊讶的是，Self-MoA 在大量场景中的表现优于混合不同 LLM 的标准 MoA：Self-MoA 在 AlpacaEval 2.0 基准上比 MoA 提高了 $6.6\%$，在包括 MMLU、CRUX 和 MATH 在内的各种基准上平均提高了 $3.8\%$。将 Self-MoA 应用于 AlpacaEval 2.0 中排名靠前的模型之一可直接在排行榜上实现新的最先进性能。为了了解 Self-MoA 的有效性，我们系统地研究了各种 MoA 设置下输出多样性和质量之间的权衡。我们确认 MoA 性能对质量相当敏感，混合不同的 LLM 通常会降低模型的平均质量。为了补充这项研究，我们确定了混合不同 LLM 可能有用的场景。本文进一步介绍了 Self-MoA 的顺序版本，它能够在多轮中即时聚合大量 LLM 输出，并且与一次聚合所有输出一样有效。

Title: ReFoRCE: A Text-to-SQL Agent with Self-Refinement, Format Restriction, and Column Exploration

• Authors: Minghang Deng, Ashwin Ramachandran, Canwen Xu, Lanxiang Hu, Zhewei Yao, Anupam Datta, Hao Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.00675
• Pdf URL: https://arxiv.org/pdf/2502.00675
• Copy Paste: [[2502.00675]] ReFoRCE: A Text-to-SQL Agent with Self-Refinement, Format Restriction, and Column Exploration(https://arxiv.org/abs/2502.00675)
• Keywords: agent
• Abstract: Text-to-SQL systems have unlocked easier access to critical data insights by enabling natural language queries over structured databases. However, deploying such systems in enterprise environments remains challenging due to factors such as large, complex schemas (> 3000 columns), diverse SQL dialects (e.g., BigQuery, Snowflake) and sophisticated query requirements (e.g., transformation, analytics). Current state-of-the-art performance on the Spider 2.0 dataset -- a benchmark built to mimic such complex environments -- remains limited at 20%. Key limitations include inadequate instruction-following, poor long-context comprehension, weak self-refinement, and insufficient dialect-specific knowledge. To address these gaps, we propose ReFoRCE (Self-Refinement Agent with Format Restriction and Column Exploration) which introduces (1) table compression to mitigate long-context limitations (2) format restriction to ensure accurate answer format, and (3) iterative column exploration for enhanced schema understanding. Additionally, it employs self-refinement pipeline consisting of (1) parallelized workflows with voting mechanisms and (2) a Common Table Expression (CTE) based refinement approach to handle unresolved cases. ReFoRCE achieves state-of-the-art results scoring 26.69 on the Spider 2.0-Snow and scoring 24.50 on the Spider 2.0-Lite tasks.
• 摘要：文本到 SQL 系统通过在结构化数据库上启用自然语言查询，可以更轻松地访问关键数据洞察。但是，由于大型复杂模式（> 3000 列）、多样的 SQL 方言（例如 BigQuery、Snowflake）和复杂的查询要求（例如转换、分析）等因素，在企业环境中部署此类系统仍然具有挑战性。目前，在 Spider 2.0 数据集（为模拟此类复杂环境而构建的基准）上的最佳性能仍然限制在 20%。主要限制包括指令遵循不足、长上下文理解能力差、自我细化能力弱以及方言特定知识不足。为了解决这些差距，我们提出了 ReFoRCE（具有格式限制和列探索的自细化代理），它引入了（1）表压缩以减轻长上下文限制（2）格式限制以确保准确的答案格式，以及（3）迭代列探索以增强模式理解。此外，它还采用了自我改进流程，包括 (1) 具有投票机制的并行工作流和 (2) 基于通用表表达式 (CTE) 的改进方法来处理未解决的案例。ReFoRCE 取得了最先进的成绩，在 Spider 2.0-Snow 任务上得分为 26.69，在 Spider 2.0-Lite 任务上得分为 24.50。

Title: Structural Latency Perturbation in Large Language Models Through Recursive State Induction

• Authors: Michael Mangrum, Jonathan Pemberton, Benedict Wetherby, Philip Montague
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.00758
• Pdf URL: https://arxiv.org/pdf/2502.00758
• Copy Paste: [[2502.00758]] Structural Latency Perturbation in Large Language Models Through Recursive State Induction(https://arxiv.org/abs/2502.00758)
• Keywords: language model
• Abstract: Computational efficiency has remained a critical consideration in scaling high-capacity language models, with inference latency and resource consumption presenting significant constraints on real-time applications. The study has introduced a structured latency perturbation mechanism that modifies computational pathways through recursive state induction, enabling dynamic suppression of redundant activations while preserving generative fidelity. A formal mathematical framework has been established to describe recursive perturbations, ensuring that modifications remain adaptive rather than statically imposed. Experiments have demonstrated that applying recursive state adjustments reduces inference latency across varying sequence lengths, with longer text generations benefiting from cumulative efficiency improvements. Comparative evaluations against structured pruning and quantization have indicated that latency gains can be achieved without compromising token retention or memory utilization. The analysis of computational overhead has suggested that selectively suppressing redundant activations contributes to improved power efficiency, particularly in scenarios requiring extended text generation. An assessment of linguistic stability has shown that token-level consistency remains largely intact under controlled perturbation thresholds, reinforcing the viability of structural latency modifications as an alternative to weight-centric optimization techniques. The results have supported the hypothesis that recursive state induction offers an effective method for reducing computational complexity without requiring architectural modifications or external augmentation.
• 摘要：计算效率一直是扩展高容量语言模型的关键考虑因素，推理延迟和资源消耗对实时应用构成重大制约。该研究引入了一种结构化延迟扰动机制，该机制通过递归状态感应修改计算路径，从而能够动态抑制冗余激活，同时保持生成保真度。已经建立了一个正式的数学框架来描述递归扰动，确保修改保持自适应性而不是静态强加。实验表明，应用递归状态调整可以减少不同序列长度的推理延迟，更长的文本生成可从累积效率改进中受益。与结构化修剪和量化的比较评估表明，可以在不影响令牌保留或内存利用率的情况下实现延迟增益。计算开销分析表明，选择性地抑制冗余激活有助于提高能效，特别是在需要扩展文本生成的场景中。语言稳定性评估表明，在受控扰动阈值下，标记级一致性基本保持不变，这进一步证明了结构延迟修改作为以权重为中心的优化技术的替代方案的可行性。结果支持了以下假设：递归状态感应提供了一种有效的方法来降低计算复杂性，而无需进行架构修改或外部增强。

Title: FIRE: Flexible Integration of Data Quality Ratings for Effective Pre-Training

• Authors: Liangyu Xu, Xuemiao Zhang, Feiyu Duan, Sirui Wang, Jingang Wang, Xunliang Cai
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.00761
• Pdf URL: https://arxiv.org/pdf/2502.00761
• Copy Paste: [[2502.00761]] FIRE: Flexible Integration of Data Quality Ratings for Effective Pre-Training(https://arxiv.org/abs/2502.00761)
• Keywords: language model, llm
• Abstract: Selecting high-quality data can significantly improve the pre-training efficiency of large language models (LLMs). Existing methods often rely on heuristic techniques and single quality signals, limiting their ability to comprehensively evaluate data quality. In this work, we propose FIRE, a flexible and scalable framework for integrating multiple data quality raters, which allows for a comprehensive assessment of data quality across various dimensions. FIRE aligns multiple quality signals into a unified space, and integrates diverse data quality raters to provide a comprehensive quality signal for each data point. Further, we introduce a progressive data selection scheme based on FIRE that iteratively refines the selection of high-quality data points, balancing computational complexity with the refinement of orthogonality. Experiments on the SlimPajama dataset reveal that FIRE consistently outperforms other selection methods and significantly enhances the pre-trained model across a wide range of downstream tasks, with a 2.9\% average performance boost and reducing the FLOPs necessary to achieve a certain performance level by more than half.
• 摘要：选择高质量数据可以显著提高大型语言模型 (LLM) 的预训练效率。现有方法通常依赖于启发式技术和单一质量信号，限制了它们全面评估数据质量的能力。在这项工作中，我们提出了 FIRE，这是一个灵活且可扩展的框架，用于集成多个数据质量评估者，从而可以全面评估各个维度的数据质量。FIRE 将多个质量信号对齐到统一的空间中，并集成不同的数据质量评估者，为每个数据点提供全面的质量信号。此外，我们引入了一种基于 FIRE 的渐进式数据选择方案，该方案迭代地细化高质量数据点的选择，平衡计算复杂性和正交性的细化。在 SlimPajama 数据集上的实验表明，FIRE 始终优于其他选择方法，并在广泛的下游任务中显著增强了预训练模型，平均性能提升了 2.9%，并将实现一定性能水平所需的 FLOP 减少了一半以上。

Title: Vision-centric Token Compression in Large Language Model

• Authors: Ling Xing, Alex Jinpeng Wang, Rui Yan, Jinhui Tang
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2502.00791
• Pdf URL: https://arxiv.org/pdf/2502.00791
• Copy Paste: [[2502.00791]] Vision-centric Token Compression in Large Language Model(https://arxiv.org/abs/2502.00791)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have revolutionized natural language processing, excelling in handling longer sequences. However, the inefficiency and redundancy in processing extended in-context tokens remain a challenge. Many attempts to address this rely on compressing tokens with smaller text encoders, yet we question whether text encoders are truly indispensable. Our journey leads to an unexpected discovery-a much smaller vision encoder, applied directly to sequences of text tokens, can rival text encoders on text tasks. When pre-trained on large amounts of data and transferred to multiple mid-sized or small text understanding benchmarks, VIST leads to comparable results with 16% fewer FLOPs and 50% less memory usage. We further uncover significant token redundancy and devise a frequency-based masking strategy to guide the focus of the visual encoder toward the most critical tokens. Interestingly, we observe the trained visual encoder performs like a summarizer, selectively ignoring less important words such as prepositions and conjunctions. This approach delivers remarkable results, outperforming traditional text encoder-based methods by 5.7% on average over benchmarks like TriviaQA, NQ, PopQA, TREF, SST2, and SST5, setting a new standard for token efficiency in LLMs.
• 摘要：大型语言模型 (LLM) 彻底改变了自然语言处理，在处理较长的序列方面表现出色。然而，处理扩展上下文标记的低效率和冗余仍然是一个挑战。许多解决这个问题的尝试都依赖于使用较小的文本编码器压缩标记，但我们质疑文本编码器是否真的不可或缺。我们的旅程带来了一个意外的发现——一个更小的视觉编码器，直接应用于文本标记序列，可以在文本任务上与文本编码器相媲美。当对大量数据进行预训练并转移到多个中型或小型文本理解基准时，VIST 可产生类似的结果，FLOP 减少 16%，内存使用量减少 50%。我们进一步发现了显著的标记冗余，并设计了一种基于频率的掩蔽策略来引导视觉编码器的注意力转向最关键的标记。有趣的是，我们观察到经过训练的视觉编码器表现得像一个总结器，有选择地忽略不太重要的词，如介词和连词。该方法取得了显著的效果，在 TriviaQA、NQ、PopQA、TREF、SST2 和 SST5 等基准测试中，其表现平均比基于传统文本编码器的方法高出 5.7%，为 LLM 中的标记效率树立了新的标准。

Title: Probing Large Language Models in Reasoning and Translating Complex Linguistic Puzzles

• Authors: Zheng-Lin Lin, Yu-Fei Shih, Shu-Kai Hsieh
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.00817
• Pdf URL: https://arxiv.org/pdf/2502.00817
• Copy Paste: [[2502.00817]] Probing Large Language Models in Reasoning and Translating Complex Linguistic Puzzles(https://arxiv.org/abs/2502.00817)
• Keywords: language model, gpt, llm, prompt, chain-of-thought
• Abstract: This paper investigates the utilization of Large Language Models (LLMs) for solving complex linguistic puzzles, a domain requiring advanced reasoning and adept translation capabilities akin to human cognitive processes. We explore specific prompting techniques designed to enhance ability of LLMs to reason and elucidate their decision-making pathways, with a focus on Input-Output Prompting (IO), Chain-of-Thought Prompting (CoT), and Solo Performance Prompting (SPP). Utilizing datasets from the Puzzling Machine Competition and various Linguistics Olympiads, we employ a comprehensive set of metrics to assess the performance of GPT-4 0603, a prominent LLM, across these prompting methods. Our findings illuminate the potential of LLMs in linguistic reasoning and complex translation tasks, highlighting their capabilities and identifying limitations in the context of linguistic puzzles. This research contributes significantly to the broader field of Natural Language Processing (NLP) by providing insights into the optimization of LLM applications for improved reasoning and translation accuracy, thereby enriching the ongoing dialogue in NLP advancements.
• 摘要：本文探讨了大型语言模型 (LLM) 在解决复杂语言难题中的应用，该领域需要高级推理和类似于人类认知过程的熟练翻译能力。我们探索了旨在提高 LLM 推理能力和阐明其决策路径的特定提示技术，重点是输入输出提示 (IO)、思维链提示 (CoT) 和单独表现提示 (SPP)。利用来自 Puzzling Machine 竞赛和各种语言学奥林匹克竞赛的数据集，我们采用一套全面的指标来评估 GPT-4 0603（一款著名的 LLM）在这些提示方法中的表现。我们的研究结果阐明了 LLM 在语言推理和复杂翻译任务中的潜力，突出了它们的能力并确定了语言难题背景下的局限性。这项研究为优化 LLM 应用程序以提高推理和翻译准确性提供了见解，为自然语言处理 (NLP) 这一更广泛的领域做出了重大贡献，从而丰富了 NLP 进步方面的持续对话。

Title: Weak Supervision Dynamic KL-Weighted Diffusion Models Guided by Large Language Models

• Authors: Julian Perry, Frank Sanders, Carter Scott
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.00826
• Pdf URL: https://arxiv.org/pdf/2502.00826
• Copy Paste: [[2502.00826]] Weak Supervision Dynamic KL-Weighted Diffusion Models Guided by Large Language Models(https://arxiv.org/abs/2502.00826)
• Keywords: language model, llm
• Abstract: In this paper, we presents a novel method for improving text-to-image generation by combining Large Language Models (LLMs) with diffusion models, a hybrid approach aimed at achieving both higher quality and efficiency in image synthesis from text descriptions. Our approach introduces a new dynamic KL-weighting strategy to optimize the diffusion process, along with incorporating semantic understanding from pre-trained LLMs to guide the generation process. The proposed method significantly improves both the visual quality and alignment of generated images with text descriptions, addressing challenges such as computational inefficiency, instability in training, and robustness to textual variability. We evaluate our method on the COCO dataset and demonstrate its superior performance over traditional GAN-based models, both quantitatively and qualitatively. Extensive experiments, including ablation studies and human evaluations, confirm that our method outperforms existing approaches in terms of image realism, relevance to the input text, and overall aesthetic quality. Our approach also shows promise in scalability to other multimodal tasks, making it a versatile solution for a wide range of generative applications.
• 摘要：在本文中，我们提出了一种通过将大型语言模型 (LLM) 与扩散模型相结合来改进文本到图像生成的新方法，这种混合方法旨在实现从文本描述进行图像合成的更高质量和更高效的效果。我们的方法引入了一种新的动态 KL 加权策略来优化扩散过程，同时结合了预训练 LLM 的语义理解来指导生成过程。所提出的方法显著提高了生成的图像与文本描述的视觉质量和对齐程度，解决了计算效率低下、训练不稳定以及对文本变化的鲁棒性等挑战。我们在 COCO 数据集上评估了我们的方法，并在定量和定性方面证明了它比传统的基于 GAN 的模型更优越的性能。大量实验（包括消融研究和人工评估）证实，我们的方法在图像真实感、与输入文本的相关性和整体美学质量方面优于现有方法。我们的方法还显示出对其他多模式任务的可扩展性的前景，使其成为广泛生成应用的多功能解决方案。

Title: Generalization of Medical Large Language Models through Cross-Domain Weak Supervision

• Authors: Robert Long, Eric Gonzalez, Harrison Fuller
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.00832
• Pdf URL: https://arxiv.org/pdf/2502.00832
• Copy Paste: [[2502.00832]] Generalization of Medical Large Language Models through Cross-Domain Weak Supervision(https://arxiv.org/abs/2502.00832)
• Keywords: language model, llm
• Abstract: The advancement of large language models (LLMs) has opened new frontiers in natural language processing, particularly in specialized domains like healthcare. In this paper, we propose the Incremental Curriculum-Based Fine-Tuning (ICFT) framework to enhance the generative capabilities of medical large language models (MLLMs). ICFT combines curriculum-based learning, dual-stage memory coordination, and parameter-efficient fine-tuning to enable a progressive transition from general linguistic knowledge to strong domain-specific expertise. Experimental results across diverse medical NLP tasks, including question answering, preference classification, and response generation, demonstrate that ICFT consistently outperforms state-of-the-art baselines, achieving improvements in both accuracy and efficiency. Further analysis reveals the framework's ability to generalize to unseen data, reduce errors, and deliver diverse, contextually relevant medical responses. These findings establish ICFT as a robust and scalable solution for adapting LLMs to the medical domain, offering practical benefits for real-world healthcare applications.
• 摘要：大型语言模型 (LLM) 的进步为自然语言处理开辟了新领域，尤其是在医疗保健等专业领域。在本文中，我们提出了基于课程的增量微调 (ICFT) 框架，以增强医学大型语言模型 (MLLM) 的生成能力。ICFT 结合了基于课程的学习、双阶段记忆协调和参数高效的微调，实现了从一般语言知识到强大领域特定专业知识的逐步过渡。在各种医学 NLP 任务（包括问答、偏好分类和响应生成）中的实验结果表明，ICFT 始终优于最先进的基线，在准确性和效率方面都有所提高。进一步的分析揭示了该框架能够推广到看不见的数据、减少错误并提供多样化、上下文相关的医疗响应。这些发现确立了 ICFT 是一种强大且可扩展的解决方案，可用于将 LLM 应用于医学领域，为现实世界的医疗保健应用提供实际好处。

Title: Explainability in Practice: A Survey of Explainable NLP Across Various Domains

• Authors: Hadi Mohammadi, Ayoub Bagheri, Anastasia Giachanou, Daniel L. Oberski
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.00837
• Pdf URL: https://arxiv.org/pdf/2502.00837
• Copy Paste: [[2502.00837]] Explainability in Practice: A Survey of Explainable NLP Across Various Domains(https://arxiv.org/abs/2502.00837)
• Keywords: gpt
• Abstract: Natural Language Processing (NLP) has become a cornerstone in many critical sectors, including healthcare, finance, and customer relationship management. This is especially true with the development and use of advanced models such as GPT-based architectures and BERT, which are widely used in decision-making processes. However, the black-box nature of these advanced NLP models has created an urgent need for transparency and explainability. This review explores explainable NLP (XNLP) with a focus on its practical deployment and real-world applications, examining its implementation and the challenges faced in domain-specific contexts. The paper underscores the importance of explainability in NLP and provides a comprehensive perspective on how XNLP can be designed to meet the unique demands of various sectors, from healthcare's need for clear insights to finance's emphasis on fraud detection and risk assessment. Additionally, this review aims to bridge the knowledge gap in XNLP literature by offering a domain-specific exploration and discussing underrepresented areas such as real-world applicability, metric evaluation, and the role of human interaction in model assessment. The paper concludes by suggesting future research directions that could enhance the understanding and broader application of XNLP.
• 摘要：自然语言处理 (NLP) 已成为许多关键领域的基石，包括医疗保健、金融和客户关系管理。随着基于 GPT 的架构和 BERT 等高级模型的开发和使用，情况尤其如此，这些模型在决策过程中得到了广泛应用。然而，这些高级 NLP 模型的黑箱性质迫切需要透明度和可解释性。本综述探讨了可解释的 NLP (XNLP)，重点关注其实际部署和实际应用，研究了其实现以及在特定领域环境中面临的挑战。本文强调了 NLP 中可解释性的重要性，并全面介绍了如何设计 XNLP 以满足各个行业的独特需求，从医疗保健对清晰见解的需求到金融对欺诈检测和风险评估的重视。此外，本综述旨在通过提供特定领域的探索和讨论代表性不足的领域（例如现实世界的适用性、指标评估以及人机交互在模型评估中的作用）来弥补 XNLP 文献中的知识差距。本文最后提出了未来的研究方向，以增进对 XNLP 的理解和更广泛的应用。

Title: HintEval: A Comprehensive Framework for Hint Generation and Evaluation for Questions

• Authors: Jamshid Mozafari, Bhawna Piryani, Abdelrahman Abdallah, Adam Jatowt
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2502.00857
• Pdf URL: https://arxiv.org/pdf/2502.00857
• Copy Paste: [[2502.00857]] HintEval: A Comprehensive Framework for Hint Generation and Evaluation for Questions(https://arxiv.org/abs/2502.00857)
• Keywords: language model, llm, chat
• Abstract: Large Language Models (LLMs) are transforming how people find information, and many users turn nowadays to chatbots to obtain answers to their questions. Despite the instant access to abundant information that LLMs offer, it is still important to promote critical thinking and problem-solving skills. Automatic hint generation is a new task that aims to support humans in answering questions by themselves by creating hints that guide users toward answers without directly revealing them. In this context, hint evaluation focuses on measuring the quality of hints, helping to improve the hint generation approaches. However, resources for hint research are currently spanning different formats and datasets, while the evaluation tools are missing or incompatible, making it hard for researchers to compare and test their models. To overcome these challenges, we introduce HintEval, a Python library that makes it easy to access diverse datasets and provides multiple approaches to generate and evaluate hints. HintEval aggregates the scattered resources into a single toolkit that supports a range of research goals and enables a clear, multi-faceted, and reliable evaluation. The proposed library also includes detailed online documentation, helping users quickly explore its features and get started. By reducing barriers to entry and encouraging consistent evaluation practices, HintEval offers a major step forward for facilitating hint generation and analysis research within the NLP/IR community.
• 摘要：大型语言模型 (LLM) 正在改变人们查找信息的方式，如今许多用户都求助于聊天机器人来获得问题的答案。尽管 LLM 提供了即时访问丰富信息的渠道，但培养批判性思维和解决问题的能力仍然很重要。自动提示生成是一项新任务，旨在通过创建提示来引导用户找到答案，而无需直接透露答案，从而支持人类自己回答问题。在这种情况下，提示评估侧重于衡量提示的质量，帮助改进提示生成方法。然而，目前提示研究的资源涵盖不同的格式和数据集，而评估工具缺失或不兼容，使得研究人员很难比较和测试他们的模型。为了克服这些挑战，我们引入了 HintEval，这是一个 Python 库，可以轻松访问不同的数据集，并提供多种方法来生成和评估提示。HintEval 将分散的资源聚合到一个工具包中，该工具包支持一系列研究目标，并实现清晰、多方面和可靠的评估。拟议的库还包括详细的在线文档，帮助用户快速探索其功能并开始使用。通过降低进入门槛并鼓励一致的评估实践，HintEval 为促进 NLP/IR 社区内的提示生成和分析研究迈出了重要一步。

Title: Predicting potentially unfair clauses in Chilean terms of services with natural language processing

• Authors: Christoffer Loeffler, Andrea Martínez Freile, Tomás Rey Pizarro
• Subjects: cs.CL, cs.AI, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.00865
• Pdf URL: https://arxiv.org/pdf/2502.00865
• Copy Paste: [[2502.00865]] Predicting potentially unfair clauses in Chilean terms of services with natural language processing(https://arxiv.org/abs/2502.00865)
• Keywords: language model
• Abstract: This study addresses the growing concern of information asymmetry in consumer contracts, exacerbated by the proliferation of online services with complex Terms of Service that are rarely even read. Even though research on automatic analysis methods is conducted, the problem is aggravated by the general focus on English-language Machine Learning approaches and on major jurisdictions, such as the European Union. We introduce a new methodology and a substantial dataset addressing this gap. We propose a novel annotation scheme with four categories and a total of 20 classes, and apply it on 50 online Terms of Service used in Chile. Our evaluation of transformer-based models highlights how factors like language- and/or domain-specific pre-training, few-shot sample size, and model architecture affect the detection and classification of potentially abusive clauses. Results show a large variability in performance for the different tasks and models, with the highest macro-F1 scores for the detection task ranging from 79% to 89% and micro-F1 scores up to 96%, while macro-F1 scores for the classification task range from 60% to 70% and micro-F1 scores from 64% to 80%. Notably, this is the first Spanish-language multi-label classification dataset for legal clauses, applying Chilean law and offering a comprehensive evaluation of Spanish-language models in the legal domain. Our work lays the ground for future research in method development for rarely considered legal analysis and potentially leads to practical applications to support consumers in Chile and Latin America as a whole.
• 摘要：本研究旨在解决消费者合同中日益严重的信息不对称问题，这种问题因在线服务的激增而加剧，这些服务条款复杂，甚至很少有人阅读。尽管已经开展了自动分析方法的研究，但由于普遍关注英语机器学习方法和欧盟等主要司法管辖区，这一问题更加严重。我们引入了一种新方法和一个庞大的数据集来解决这一差距。我们提出了一种新颖的注释方案，该方案有四个类别，总共 20 个类别，并将其应用于智利使用的 50 份在线服务条款。我们对基于 Transformer 的模型的评估重点介绍了语言和/或领域特定的预训练、少量样本大小和模型架构等因素如何影响潜在滥用条款的检测和分类。结果显示，不同任务和模型的性能差异很大，检测任务的最高宏观 F1 分数范围为 79% 至 89%，微观 F1 分数最高可达 96%，而分类任务的宏观 F1 分数范围为 60% 至 70%，微观 F1 分数范围为 64% 至 80%。值得注意的是，这是第一个西班牙语法律条款多标签分类数据集，应用了智利法律，并对法律领域的西班牙语模型进行了全面评估。我们的工作为未来研究开发很少考虑的法律分析方法奠定了基础，并可能带来实际应用，以支持智利和整个拉丁美洲的消费者。

Title: MorphBPE: A Morpho-Aware Tokenizer Bridging Linguistic Complexity for Efficient LLM Training Across Morphologies

• Authors: Ehsaneddin Asgari, Yassine El Kheir, Mohammad Ali Sadraei Javaheri
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.00894
• Pdf URL: https://arxiv.org/pdf/2502.00894
• Copy Paste: [[2502.00894]] MorphBPE: A Morpho-Aware Tokenizer Bridging Linguistic Complexity for Efficient LLM Training Across Morphologies(https://arxiv.org/abs/2502.00894)
• Keywords: language model, llm
• Abstract: Tokenization is fundamental to Natural Language Processing (NLP), directly impacting model efficiency and linguistic fidelity. While Byte Pair Encoding (BPE) is widely used in Large Language Models (LLMs), it often disregards morpheme boundaries, leading to suboptimal segmentation, particularly in morphologically rich languages. We introduce MorphBPE, a morphology-aware extension of BPE that integrates linguistic structure into subword tokenization while preserving statistical efficiency. Additionally, we propose two morphology-based evaluation metrics: (i) Morphological Consistency F1-Score, which quantifies the consistency between morpheme sharing and token sharing, contributing to LLM training convergence, and (ii) Morphological Edit Distance, which measures alignment between morphemes and tokens concerning interpretability. Experiments on English, Russian, Hungarian, and Arabic across 300M and 1B parameter LLMs demonstrate that MorphBPE consistently reduces cross-entropy loss, accelerates convergence, and improves morphological alignment scores. Fully compatible with existing LLM pipelines, MorphBPE requires minimal modifications for integration. The MorphBPE codebase and tokenizer playground will be available at: this https URL and this https URL
• 摘要：标记化是自然语言处理 (NLP) 的基础，直接影响模型效率和语言保真度。虽然字节对编码 (BPE) 广泛应用于大型语言模型 (LLM)，但它经常忽略词素边界，导致分割不理想，尤其是在形态丰富的语言中。我们引入了 MorphBPE，这是 BPE 的形态感知扩展，它将语言结构集成到子词标记化中，同时保持统计效率。此外，我们提出了两个基于形态的评估指标：(i) 形态一致性 F1 分数，它量化了词素共享和标记共享之间的一致性，有助于 LLM 训练收敛，以及 (ii) 形态编辑距离，它测量词素和标记在可解释性方面的一致性。在 300M 和 1B 参数 LLM 上对英语、俄语、匈牙利语和阿拉伯语进行的实验表明，MorphBPE 可以持续降低交叉熵损失、加速收敛并提高形态对齐分数。MorphBPE 与现有 LLM 管道完全兼容，集成时只需进行少量修改。MorphBPE 代码库和 tokenizer 游乐场将在以下位置提供：此 https URL 和此 https URL

Title: Embracing Dialectic Intersubjectivity: Coordination of Different Perspectives in Content Analysis with LLM Persona Simulation

• Authors: Taewoo Kang, Kjerstin Thorson, Tai-Quan Peng, Dan Hiaeshutter-Rice, Sanguk Lee, Stuart Soroka
• Subjects: cs.CL, cs.AI, cs.CY, cs.SI
• Abstract URL: https://arxiv.org/abs/2502.00903
• Pdf URL: https://arxiv.org/pdf/2502.00903
• Copy Paste: [[2502.00903]] Embracing Dialectic Intersubjectivity: Coordination of Different Perspectives in Content Analysis with LLM Persona Simulation(https://arxiv.org/abs/2502.00903)
• Keywords: gpt, llm
• Abstract: This study attempts to advancing content analysis methodology from consensus-oriented to coordination-oriented practices, thereby embracing diverse coding outputs and exploring the dynamics among differential perspectives. As an exploratory investigation of this approach, we evaluate six GPT-4o configurations to analyze sentiment in Fox News and MSNBC transcripts on Biden and Trump during the 2020 U.S. presidential campaign, examining patterns across these models. By assessing each model's alignment with ideological perspectives, we explore how partisan selective processing could be identified in LLM-Assisted Content Analysis (LACA). Findings reveal that partisan persona LLMs exhibit stronger ideological biases when processing politically congruent content. Additionally, intercoder reliability is higher among same-partisan personas compared to cross-partisan pairs. This approach enhances the nuanced understanding of LLM outputs and advances the integrity of AI-driven social science research, enabling simulations of real-world implications.
• 摘要：本研究试图将内容分析方法从以共识为导向推进到以协调为导向的实践，从而涵盖多样化的编码输出并探索不同视角之间的动态。作为对这种方法的探索性研究，我们评估了六种 GPT-4o 配置，以分析 2020 年美国总统竞选期间福克斯新闻和 MSNBC 对拜登和特朗普的报道情绪，并检查这些模型中的模式。通过评估每个模型与意识形态观点的一致性，我们探索如何在 LLM 辅助内容分析 (LACA) 中识别党派选择性处理。研究结果表明，党派角色 LLM 在处理政治一致的内容时表现出更强的意识形态偏见。此外，与跨党派配对相比，同党派角色之间的编码者信度更高。这种方法增强了对 LLM 输出的细致理解，提高了人工智能驱动的社会科学研究的完整性，从而能够模拟现实世界的影响。

Title: The Accuracy, Robustness, and Readability of LLM-Generated Sustainability-Related Word Definitions

• Authors: Alice Heiman
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.00916
• Pdf URL: https://arxiv.org/pdf/2502.00916
• Copy Paste: [[2502.00916]] The Accuracy, Robustness, and Readability of LLM-Generated Sustainability-Related Word Definitions(https://arxiv.org/abs/2502.00916)
• Keywords: gpt, llm
• Abstract: A common language with standardized definitions is crucial for effective climate discussions. However, concerns exist about LLMs misrepresenting climate terms. We compared 300 official IPCC glossary definitions with those generated by GPT-4o-mini, Llama3.1 8B, and Mistral 7B, analyzing adherence, robustness, and readability using SBERT sentence embeddings. The LLMs scored an average adherence of $0.57-0.59 \pm 0.15$, and their definitions proved harder to read than the originals. Model-generated definitions vary mainly among words with multiple or ambiguous definitions, showing the potential to highlight terms that need standardization. The results show how LLMs could support environmental discourse while emphasizing the need to align model outputs with established terminology for clarity and consistency.
• 摘要：具有标准化定义的通用语言对于有效的气候讨论至关重要。然而，人们担心 LLM 会歪曲气候术语。我们将 300 个官方 IPCC 词汇表定义与 GPT-4o-mini、Llama3.1 8B 和 Mistral 7B 生成的定义进行了比较，使用 SBERT 句子嵌入分析了依从性、稳健性和可读性。LLM 的平均依从性得分为 $0.57-0.59 \pm 0.15$，而且它们的定义比原文更难读。模型生成的定义主要在具有多重或模糊定义的单词之间有所不同，显示出突出显示需要标准化的术语的潜力。结果显示了 LLM 如何支持环境话语，同时强调需要将模型输出与既定术语保持一致以确保清晰度和一致性。

Title: Attention Sinks and Outlier Features: A 'Catch, Tag, and Release' Mechanism for Embeddings

• Authors: Stephen Zhang, Mustafa Khan, Vardan Papyan
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.00919
• Pdf URL: https://arxiv.org/pdf/2502.00919
• Copy Paste: [[2502.00919]] Attention Sinks and Outlier Features: A 'Catch, Tag, and Release' Mechanism for Embeddings(https://arxiv.org/abs/2502.00919)
• Keywords: language model, llm
• Abstract: Two prominent features of large language models (LLMs) is the presence of large-norm (outlier) features and the tendency for tokens to attend very strongly to a select few tokens. Despite often having no semantic relevance, these select tokens, called attention sinks, along with the large outlier features, have proven important for model performance, compression, and streaming. Consequently, investigating the roles of these phenomena within models and exploring how they might manifest in the model parameters has become an area of active interest. Through an empirical investigation, we demonstrate that attention sinks utilize outlier features to: catch a sequence of tokens, tag the captured tokens by applying a common perturbation, and then release the tokens back into the residual stream, where the tagged tokens are eventually retrieved. We prove that simple tasks, like averaging, necessitate the 'catch, tag, release' mechanism hence explaining why it would arise organically in modern LLMs. Our experiments also show that the creation of attention sinks can be completely captured in the model parameters using low-rank matrices, which has important implications for model compression and substantiates the success of recent approaches that incorporate a low-rank term to offset performance degradation.
• 摘要：大型语言模型 (LLM) 的两个突出特点是存在大范数 (异常值) 特征，以及标记倾向于非常强烈地关注少数几个标记。尽管这些选定的标记通常没有语义相关性，但它们被称为注意力接收器，以及大型异常值特征，已被证明对模型性能、压缩和流式传输非常重要。因此，研究这些现象在模型中的作用并探索它们如何在模型参数中体现已成为一个热门领域。通过实证调查，我们证明注意力接收器利用异常值特征来：捕获一系列标记，通过应用常见扰动标记捕获的标记，然后将标记释放回残余流，最终在那里检索标记的标记。我们证明简单的任务（如平均）需要“捕获、标记、释放”机制，从而解释了为什么它会在现代 LLM 中自然出现。我们的实验还表明，可以使用低秩矩阵在模型参数中完全捕获注意力集中的创建，这对于模型压缩具有重要意义，并证实了最近采用低秩项来抵消性能下降的方法的成功。

Title: Universal Abstraction: Harnessing Frontier Models to Structure Real-World Data at Scale

• Authors: Cliff Wong, Sam Preston, Qianchu Liu, Zelalem Gero, Jass Bagga, Sheng Zhang, Shrey Jain, Theodore Zhao, Yu Gu, Yanbo Xu, Sid Kiblawi, Roshanthi Weerasinghe, Rom Leidner, Kristina Young, Brian Piening, Carlo Bifulco, Tristan Naumann, Mu Wei, Hoifung Poon
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.00943
• Pdf URL: https://arxiv.org/pdf/2502.00943
• Copy Paste: [[2502.00943]] Universal Abstraction: Harnessing Frontier Models to Structure Real-World Data at Scale(https://arxiv.org/abs/2502.00943)
• Keywords: language model, gpt, llm, prompt
• Abstract: The vast majority of real-world patient information resides in unstructured clinical text, and the process of medical abstraction seeks to extract and normalize structured information from this unstructured input. However, traditional medical abstraction methods can require significant manual efforts that can include crafting rules or annotating training labels, limiting scalability. In this paper, we propose UniMedAbstractor (UMA), a zero-shot medical abstraction framework leveraging Large Language Models (LLMs) through a modular and customizable prompt template. We refer to our approach as universal abstraction as it can quickly scale to new attributes through its universal prompt template without curating attribute-specific training labels or rules. We evaluate UMA for oncology applications, focusing on fifteen key attributes representing the cancer patient journey, from short-context attributes (e.g., performance status, treatment) to complex long-context attributes requiring longitudinal reasoning (e.g., tumor site, histology, TNM staging). Experiments on real-world data show UMA's strong performance and generalizability. Compared to supervised and heuristic baselines, UMA with GPT-4o achieves on average an absolute 2-point F1/accuracy improvement for both short-context and long-context attribute abstraction. For pathologic T staging, UMA even outperforms the supervised model by 20 points in accuracy.
• 摘要：现实世界中绝大多数患者信息都存在于非结构化临床文本中，医学抽象过程旨在从这种非结构化输入中提取和规范化结构化信息。然而，传统的医学抽象方法可能需要大量的手动工作，包括制定规则或注释训练标签，从而限制了可扩展性。在本文中，我们提出了 UniMedAbstractor (UMA)，这是一个零样本医学抽象框架，通过模块化和可定制的提示模板利用大型语言模型 (LLM)。我们将我们的方法称为通用抽象，因为它可以通过通用提示模板快速扩展到新属性，而无需策划特定于属性的训练标签或规则。我们评估了 UMA 在肿瘤学应用中的表现，重点关注代表癌症患者旅程的 15 个关键属性，从短上下文属性（例如，体能状态、治疗）到需要纵向推理的复杂长上下文属性（例如，肿瘤部位、组织学、TNM 分期）。对现实世界数据的实验表明 UMA 具有强大的性能和通用性。与监督和启发式基线相比，采用 GPT-4o 的 UMA 在短上下文和长上下文属性抽象方面平均实现了绝对 2 点 F1/准确度提升。对于病理 T 分期，UMA 的准确度甚至比监督模型高出 20 个百分点。

Title: Efficient Multi-Agent System Training with Data Influence-Oriented Tree Search

• Authors: Wentao Shi, Zichun Yu, Fuli Feng, Xiangnan He, Chenyan Xiong
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.00955
• Pdf URL: https://arxiv.org/pdf/2502.00955
• Copy Paste: [[2502.00955]] Efficient Multi-Agent System Training with Data Influence-Oriented Tree Search(https://arxiv.org/abs/2502.00955)
• Keywords: language model, llm, agent
• Abstract: Monte Carlo Tree Search (MCTS) based methods provide promising approaches for generating synthetic data to enhance the self-training of Large Language Model (LLM) based multi-agent systems (MAS). These methods leverage Q-values to estimate individual agent contributions. However, relying solely on Q-values to identify informative data may misalign with the data synthesis objective, as the focus should be on selecting data that best enhances model training. To address this discrepancy, we propose Data Influence-oriented Tree Search (DITS), a novel framework that incorporates influence scores to guide both tree search and data selection. By leveraging influence scores, we effectively identify the most impactful data for system improvement, thereby enhancing model performance. Furthermore, we derive influence score estimation methods tailored for non-differentiable metrics, significantly reducing computational overhead by utilizing inference computations. Extensive experiments on eight multi-agent datasets demonstrate the robustness and effectiveness of the proposed methods. Notably, our findings reveal that allocating more inference resources to estimate influence scores, rather than Q-values, during data synthesis can more effectively and efficiently enhance model training.
• 摘要：基于蒙特卡洛树搜索 (MCTS) 的方法提供了生成合成数据的有前途的方法，以增强基于大型语言模型 (LLM) 的多智能体系统 (MAS) 的自我训练。这些方法利用 Q 值来估计单个智能体贡献。但是，仅依靠 Q 值来识别信息数据可能与数据合成目标不一致，因为重点应该放在选择最能增强模型训练的数据上。为了解决这一差异，我们提出了面向数据影响的树搜索 (DITS)，这是一个新颖的框架，它结合了影响分数来指导树搜索和数据选择。通过利用影响分数，我们可以有效地识别对系统改进影响最大的数据，从而提高模型性能。此外，我们推导出针对不可微指标量身定制的影响分数估计方法，通过利用推理计算显着降低了计算开销。在八个多智能体数据集上进行的大量实验证明了所提出方法的稳健性和有效性。值得注意的是，我们的研究结果表明，在数据合成过程中分配更多的推理资源来估计影响分数而不是 Q 值可以更有效、更高效地增强模型训练。

Title: Wizard of Shopping: Target-Oriented E-commerce Dialogue Generation with Decision Tree Branching

• Authors: Xiangci Li, Zhiyu Chen, Jason Ingyu Choi, Nikhita Vedula, Besnik Fetahu, Oleg Rokhlenko, Shervin Malmasi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.00969
• Pdf URL: https://arxiv.org/pdf/2502.00969
• Copy Paste: [[2502.00969]] Wizard of Shopping: Target-Oriented E-commerce Dialogue Generation with Decision Tree Branching(https://arxiv.org/abs/2502.00969)
• Keywords: language model, llm, chat
• Abstract: The goal of conversational product search (CPS) is to develop an intelligent, chat-based shopping assistant that can directly interact with customers to understand shopping intents, ask clarification questions, and find relevant products. However, training such assistants is hindered mainly due to the lack of reliable and large-scale datasets. Prior human-annotated CPS datasets are extremely small in size and lack integration with real-world product search systems. We propose a novel approach, TRACER, which leverages large language models (LLMs) to generate realistic and natural conversations for different shopping domains. TRACER's novelty lies in grounding the generation to dialogue plans, which are product search trajectories predicted from a decision tree model, that guarantees relevant product discovery in the shortest number of search conditions. We also release the first target-oriented CPS dataset Wizard of Shopping (WoS), containing highly natural and coherent conversations (3.6k) from three shopping domains. Finally, we demonstrate the quality and effectiveness of WoS via human evaluations and downstream tasks.
• 摘要：对话式产品搜索 (CPS) 的目标是开发一个智能的、基于聊天的购物助手，它可以直接与客户互动以了解购物意图、提出澄清问题并找到相关产品。然而，训练这样的助手主要由于缺乏可靠的大规模数据集而受到阻碍。以前的人工注释的 CPS 数据集规模极小，缺乏与现实世界产品搜索系统的集成。我们提出了一种新颖的方法 TRACER，它利用大型语言模型 (LLM) 为不同的购物领域生成逼真而自然的对话。TRACER 的新颖之处在于将生成建立在对话计划的基础上，对话计划是从决策树模型预测的产品搜索轨迹，可确保在最短的搜索条件下发现相关产品。我们还发布了第一个面向目标的 CPS 数据集购物向导 (WoS)，其中包含来自三个购物领域的高度自然和连贯的对话 (3.6k)。最后，我们通过人工评估和下游任务展示了 WoS 的质量和有效性。

Title: Context-Aware Hierarchical Merging for Long Document Summarization

• Authors: Litu Ou, Mirella Lapata
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.00977
• Pdf URL: https://arxiv.org/pdf/2502.00977
• Copy Paste: [[2502.00977]] Context-Aware Hierarchical Merging for Long Document Summarization(https://arxiv.org/abs/2502.00977)
• Keywords: language model, llm, hallucination
• Abstract: Hierarchical Merging is a technique commonly used to summarize very long texts ($>$100K tokens) by breaking down the input into smaller sections, summarizing those sections individually, and then merging or combining those summaries into a final coherent summary. Although it helps address the limitations of large language models (LLMs) with fixed input length constraints, the recursive merging process can amplify LLM hallucinations, increasing the risk of factual inaccuracies. In this paper, we seek to mitigate hallucinations by enriching hierarchical merging with context from the source document. Specifically, we propose different approaches to contextual augmentation ranging from \emph{replacing} intermediate summaries with relevant input context, to \emph{refining} them while using the context as supporting evidence, and \emph{aligning} them implicitly (via citations) to the input. Experimental results on datasets representing legal and narrative domains show that contextual augmentation consistently outperforms zero-shot and hierarchical merging baselines for the Llama 3.1 model family. Our analysis further reveals that refinement methods tend to perform best when paired with extractive summarization for identifying relevant input.
• 摘要：分层合并是一种常用于总结非常长的文本（$>$100K 代币）的技术，通过将输入分解为较小的部分，分别总结这些部分，然后合并或组合这些摘要为最终的连贯摘要。虽然它有助于解决具有固定输入长度约束的大型语言模型 (LLM) 的局限性，但递归合并过程可能会放大 LLM 幻觉，从而增加事实不准确的风险。在本文中，我们试图通过使用源文档中的上下文丰富分层合并来减轻幻觉。具体来说，我们提出了不同的上下文增强方法，从 \emph{用相关输入上下文替换} 中间摘要，到 \emph{在使用上下文作为支持证据的同时对其进行细化，以及 \emph{将它们隐式（通过引用）对齐到输入。在代表法律和叙事领域的数据集上进行的实验结果表明，对于 Llama 3.1 模型系列，上下文增强始终优于零样本和分层合并基线。我们的分析进一步表明，细化方法在与提取摘要相结合以识别相关输入时往往表现最佳。

Title: PlotGen: Multi-Agent LLM-based Scientific Data Visualization via Multimodal Feedback

• Authors: Kanika Goswami, Puneet Mathur, Ryan Rossi, Franck Dernoncourt
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.00988
• Pdf URL: https://arxiv.org/pdf/2502.00988
• Copy Paste: [[2502.00988]] PlotGen: Multi-Agent LLM-based Scientific Data Visualization via Multimodal Feedback(https://arxiv.org/abs/2502.00988)
• Keywords: language model, llm, agent
• Abstract: Scientific data visualization is pivotal for transforming raw data into comprehensible visual representations, enabling pattern recognition, forecasting, and the presentation of data-driven insights. However, novice users often face difficulties due to the complexity of selecting appropriate tools and mastering visualization techniques. Large Language Models (LLMs) have recently demonstrated potential in assisting code generation, though they struggle with accuracy and require iterative debugging. In this paper, we propose PlotGen, a novel multi-agent framework aimed at automating the creation of precise scientific visualizations. PlotGen orchestrates multiple LLM-based agents, including a Query Planning Agent that breaks down complex user requests into executable steps, a Code Generation Agent that converts pseudocode into executable Python code, and three retrieval feedback agents - a Numeric Feedback Agent, a Lexical Feedback Agent, and a Visual Feedback Agent - that leverage multimodal LLMs to iteratively refine the data accuracy, textual labels, and visual correctness of generated plots via self-reflection. Extensive experiments show that PlotGen outperforms strong baselines, achieving a 4-6 percent improvement on the MatPlotBench dataset, leading to enhanced user trust in LLM-generated visualizations and improved novice productivity due to a reduction in debugging time needed for plot errors.
• 摘要：科学数据可视化对于将原始数据转换为可理解的视觉表示、实现模式识别、预测和数据驱动洞察的呈现至关重要。然而，由于选择合适的工具和掌握可视化技术的复杂性，新手用户经常会遇到困难。大型语言模型 (LLM) 最近展示了协助代码生成的潜力，尽管它们在准确性方面存在困难并且需要迭代调试。在本文中，我们提出了 PlotGen，这是一种新颖的多代理框架，旨在自动创建精确的科学可视化。PlotGen 协调多个基于 LLM 的代理，包括一个将复杂的用户请求分解为可执行步骤的查询规划代理、一个将伪代码转换为可执行 Python 代码的代码生成代理，以及三个检索反馈代理 - 数字反馈代理、词汇反馈代理和视觉反馈代理 - 它们利用多模态 LLM 通过自我反思迭代地完善数据准确性、文本标签和生成图的视觉正确性。大量实验表明，PlotGen 的表现优于强大的基线，在 MatPlotBench 数据集上实现了 4-6% 的提升，从而增强了用户对 LLM 生成的可视化的信任，并且由于减少了绘图错误所需的调试时间，提高了新手的工作效率。

Title: ChartCitor: Multi-Agent Framework for Fine-Grained Chart Visual Attribution

• Authors: Kanika Goswami, Puneet Mathur, Ryan Rossi, Franck Dernoncourt
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.00989
• Pdf URL: https://arxiv.org/pdf/2502.00989
• Copy Paste: [[2502.00989]] ChartCitor: Multi-Agent Framework for Fine-Grained Chart Visual Attribution(https://arxiv.org/abs/2502.00989)
• Keywords: language model, llm, agent
• Abstract: Large Language Models (LLMs) can perform chart question-answering tasks but often generate unverified hallucinated responses. Existing answer attribution methods struggle to ground responses in source charts due to limited visual-semantic context, complex visual-text alignment requirements, and difficulties in bounding box prediction across complex layouts. We present ChartCitor, a multi-agent framework that provides fine-grained bounding box citations by identifying supporting evidence within chart images. The system orchestrates LLM agents to perform chart-to-table extraction, answer reformulation, table augmentation, evidence retrieval through pre-filtering and re-ranking, and table-to-chart mapping. ChartCitor outperforms existing baselines across different chart types. Qualitative user studies show that ChartCitor helps increase user trust in Generative AI by providing enhanced explainability for LLM-assisted chart QA and enables professionals to be more productive.
• 摘要：大型语言模型 (LLM) 可以执行图表问答任务，但通常会产生未经验证的幻觉答案。现有的答案归因方法难以在源图表中找到答案，因为视觉语义上下文有限、视觉文本对齐要求复杂，并且难以在复杂布局中进行边界框预测。我们提出了 ChartCitor，这是一个多智能体框架，它通过识别图表图像中的支持证据来提供细粒度的边界框引用。该系统协调 LLM 智能体执行图表到表格的提取、答案重构、表格扩充、通过预过滤和重新排名检索证据以及表格到图表的映射。ChartCitor 在不同图表类型中的表现优于现有基线。定性用户研究表明，ChartCitor 通过为 LLM 辅助图表 QA 提供增强的可解释性，帮助提高用户对生成式 AI 的信任，并使专业人士提高工作效率。

Title: Self-supervised Analogical Learning using Language Models

• Authors: Ben Zhou, Sarthak Jain, Yi Zhang, Qiang Ning, Shuai Wang, Yassine Benajiba, Dan Roth
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.00996
• Pdf URL: https://arxiv.org/pdf/2502.00996
• Copy Paste: [[2502.00996]] Self-supervised Analogical Learning using Language Models(https://arxiv.org/abs/2502.00996)
• Keywords: language model
• Abstract: Large language models have been shown to suffer from reasoning inconsistency issues. That is, they fail more in situations unfamiliar to the training data, even though exact or very similar reasoning paths exist in more common cases that they can successfully solve. Such observations motivate us to propose methods that encourage models to understand the high-level and abstract reasoning processes during training instead of only the final answer. This way, models can transfer the exact solution to similar cases, regardless of their relevance to the pre-training data distribution. In this work, we propose SAL, a self-supervised analogical learning framework. SAL mimics the human analogy process and trains models to explicitly transfer high-quality symbolic solutions from cases that they know how to solve to other rare cases in which they tend to fail more. We show that the resulting models after SAL learning outperform base language models on a wide range of reasoning benchmarks, such as StrategyQA, GSM8K, and HotpotQA, by 2% to 20%. At the same time, we show that our model is more generalizable and controllable through analytical studies.
• 摘要：大型语言模型已被证明存在推理不一致的问题。也就是说，它们在训练数据不熟悉的情况下更容易失败，即使在它们可以成功解决的更常见情况下存在精确或非常相似的推理路径。这样的观察促使我们提出一些方法，鼓励模型在训练期间理解高级和抽象的推理过程，而不仅仅是最终答案。这样，模型就可以将精确的解决方案转移到类似的案例中，而不管它们与训练前数据分布的相关性如何。在这项工作中，我们提出了一个自监督的类比学习框架 SAL。SAL 模仿人类的类比过程，训练模型明确地将高质量的符号解决方案从它们知道如何解决的案例转移到它们更容易失败的其他罕见案例中。我们表明，经过 SAL 学习后得到的模型在广泛的推理基准（如 StrategyQA、GSM8K 和 HotpotQA）上的表现比基础语言模型高出 2% 到 20%。同时，我们通过分析研究表明我们的模型更具通用性和可控性。

Title: MergeME: Model Merging Techniques for Homogeneous and Heterogeneous MoEs

• Authors: Yuhang Zhou, Giannis Karamanolakis, Victor Soto, Anna Rumshisky, Mayank Kulkarni, Furong Huang, Wei Ai, Jianhua Lu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.00997
• Pdf URL: https://arxiv.org/pdf/2502.00997
• Copy Paste: [[2502.00997]] MergeME: Model Merging Techniques for Homogeneous and Heterogeneous MoEs(https://arxiv.org/abs/2502.00997)
• Keywords: language model, llm
• Abstract: The recent success of specialized Large Language Models (LLMs) in domains such as mathematical reasoning and coding has led to growing interest in methods for merging these expert LLMs into a unified Mixture-of-Experts (MoE) model, with the goal of enhancing performance in each domain while retaining effectiveness on general tasks. However, the effective merging of expert models remains an open challenge, especially for models with highly divergent weight parameters or different architectures. State-of-the-art MoE merging methods only work with homogeneous model architectures and rely on simple unweighted averaging to merge expert layers, which does not address parameter interference and requires extensive fine-tuning of the merged MoE to restore performance. To address these limitations, this paper introduces new MoE merging techniques, including strategies to mitigate parameter interference, routing heuristics to reduce the need for MoE fine-tuning, and a novel method for merging experts with different architectures. Extensive experiments across multiple domains demonstrate the effectiveness of our proposed methods, reducing fine-tuning costs, improving performance over state-of-the-art methods, and expanding the applicability of MoE merging.
• 摘要：最近，专业化大型语言模型 (LLM) 在数学推理和编码等领域取得了成功，这引起人们对将这些专家 LLM 合并为统一的混合专家 (MoE) 模型的方法的兴趣日益浓厚，目的是提高每个领域的性能，同时保持在一般任务上的有效性。然而，有效地合并专家模型仍然是一个悬而未决的挑战，特别是对于权重参数高度分散或架构不同的模型。最先进的 MoE 合并方法仅适用于同质模型架构，并依靠简单的非加权平均来合并专家层，这并不能解决参数干扰问题，并且需要对合并的 MoE 进行大量微调才能恢复性能。为了解决这些限制，本文介绍了新的 MoE 合并技术，包括减轻参数干扰的策略、减少 MoE 微调需求的路由启发式方法，以及一种合并不同架构专家的新方法。跨多个领域的大量实验证明了我们提出的方法的有效性，降低了微调成本，提高了最先进方法的性能，并扩大了 MoE 合并的适用性。

Title: Knowing When to Stop: Dynamic Context Cutoff for Large Language Models

• Authors: Roy Xie, Junlin Wang, Paul Rosu, Chunyuan Deng, Bolun Sun, Zihao Lin, Bhuwan Dhingra
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.01025
• Pdf URL: https://arxiv.org/pdf/2502.01025
• Copy Paste: [[2502.01025]] Knowing When to Stop: Dynamic Context Cutoff for Large Language Models(https://arxiv.org/abs/2502.01025)
• Keywords: language model, llm, prompt
• Abstract: Large language models (LLMs) process entire input contexts indiscriminately, which is inefficient in cases where the information required to answer a query is localized within the context. We present dynamic context cutoff, a human-inspired method enabling LLMs to self-terminate processing upon acquiring sufficient task-relevant information. Through analysis of model internals, we discover that specific attention heads inherently encode "sufficiency signals" - detectable through lightweight classifiers - that predict when critical information has been processed. This reveals a new efficiency paradigm: models' internal understanding naturally dictates processing needs rather than external compression heuristics. Comprehensive experiments across six QA datasets (up to 40K tokens) with three model families (LLaMA/Qwen/Mistral, 1B0-70B) demonstrate 1.33x average token reduction while improving accuracy by 1.3%. Furthermore, our method demonstrates better performance with the same rate of token reduction compared to other context efficiency methods. Additionally, we observe an emergent scaling phenomenon: while smaller models require require probing for sufficiency detection, larger models exhibit intrinsic self-assessment capabilities through prompting.
• 摘要：大型语言模型 (LLM) 会不加区分地处理整个输入上下文，如果回答查询所需的信息位于上下文中，则这种方法效率低下。我们提出了动态上下文截止方法，这是一种受人启发的方法，使 LLM 能够在获取足够的任务相关信息后自行终止处理。通过分析模型内部，我们发现特定的注意力头固有地编码了“充分性信号”——可通过轻量级分类器检测——这些信号可以预测关键信息何时被处理。这揭示了一种新的效率范式：模型的内部理解自然决定了处理需求，而不是外部压缩启发式方法。使用三个模型系列（LLaMA/Qwen/Mistral、1B0-70B）对六个 QA 数据集（最多 40K 个标记）进行的综合实验表明，平均标记减少量为 1.33 倍，同时准确率提高 1.3%。此外，与其他上下文效率方法相比，我们的方法在标记减少率相同的情况下表现出更好的性能。此外，我们观察到一种新兴的扩展现象：虽然较小的模型需要探测充分性检测，但较大的模型通过提示表现出内在的自我评估能力。

Title: PARA: Parameter-Efficient Fine-tuning with Prompt Aware Representation Adjustment

• Authors: Zequan Liu, Yi Zhao, Ming Tan, Wei Zhu, Aaron Xuxiang Tian
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.01033
• Pdf URL: https://arxiv.org/pdf/2502.01033
• Copy Paste: [[2502.01033]] PARA: Parameter-Efficient Fine-tuning with Prompt Aware Representation Adjustment(https://arxiv.org/abs/2502.01033)
• Keywords: prompt
• Abstract: In the realm of parameter-efficient fine-tuning (PEFT) methods, while options like LoRA are available, there is a persistent demand in the industry for a PEFT approach that excels in both efficiency and performance within the context of single-backbone multi-tenant applications. This paper introduces a new and straightforward PEFT technique, termed \underline{P}rompt \underline{A}ware \underline{R}epresentation \underline{A}djustment (PARA). The core of our proposal is to integrate a lightweight vector generator within each Transformer layer. This generator produces vectors that are responsive to input prompts, thereby adjusting the hidden representations accordingly. Our extensive experimentation across diverse tasks has yielded promising results. Firstly, the PARA method has been shown to surpass current PEFT benchmarks in terms of performance, despite having a similar number of adjustable parameters. Secondly, it has proven to be more efficient than LoRA in the single-backbone multi-tenant scenario, highlighting its significant potential for industrial adoption.
• 摘要：在参数高效微调 (PEFT) 方法领域，虽然有 LoRA 等选项可用，但业界对在单主干多租户应用环境中效率和性能均出色的 PEFT 方法的需求一直存在。本文介绍了一种新的、简单的 PEFT 技术，称为 \underline{P}rompt \underline{A}ware \underline{R}epresentation \underline{A}djustment (PARA)。我们提案的核心是在每个 Transformer 层中集成一个轻量级向量生成器。该生成器生成响应输入提示的向量，从而相应地调整隐藏表示。我们在各种任务中进行了广泛的实验，并取得了令人鼓舞的结果。首先，PARA 方法已被证明在性能方面超越了当前的 PEFT 基准，尽管具有相似数量的可调参数。其次，事实证明它在单主干多租户场景中比 LoRA 更高效，凸显了其在工业应用方面的巨大潜力。

Title: Knowledge Synthesis of Photosynthesis Research Using a Large Language Model

• Authors: Seungri Yoon, Woosang Jeon, Sanghyeok Choi, Taehyeong Kim, Tae In Ahn
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.01059
• Pdf URL: https://arxiv.org/pdf/2502.01059
• Copy Paste: [[2502.01059]] Knowledge Synthesis of Photosynthesis Research Using a Large Language Model(https://arxiv.org/abs/2502.01059)
• Keywords: language model, gpt, llm, prompt, retrieval-augmented generation
• Abstract: The development of biological data analysis tools and large language models (LLMs) has opened up new possibilities for utilizing AI in plant science research, with the potential to contribute significantly to knowledge integration and research gap identification. Nonetheless, current LLMs struggle to handle complex biological data and theoretical models in photosynthesis research and often fail to provide accurate scientific contexts. Therefore, this study proposed a photosynthesis research assistant (PRAG) based on OpenAI's GPT-4o with retrieval-augmented generation (RAG) techniques and prompt optimization. Vector databases and an automated feedback loop were used in the prompt optimization process to enhance the accuracy and relevance of the responses to photosynthesis-related queries. PRAG showed an average improvement of 8.7% across five metrics related to scientific writing, with a 25.4% increase in source transparency. Additionally, its scientific depth and domain coverage were comparable to those of photosynthesis research papers. A knowledge graph was used to structure PRAG's responses with papers within and outside the database, which allowed PRAG to match key entities with 63% and 39.5% of the database and test papers, respectively. PRAG can be applied for photosynthesis research and broader plant science domains, paving the way for more in-depth data analysis and predictive capabilities.
• 摘要：生物数据分析工具和大型语言模型 (LLM) 的发展为人工智能在植物科学研究中的应用开辟了新的可能性，有可能为知识整合和研究差距识别做出重大贡献。尽管如此，目前的 LLM 难以处理光合作用研究中复杂的生物数据和理论模型，而且往往无法提供准确的科学背景。因此，本研究提出了一种基于 OpenAI 的 GPT-4o 的光合作用研究助手 (PRAG)，该助手采用了检索增强生成 (RAG) 技术和快速优化。快速优化过程中使用了矢量数据库和自动反馈回路，以提高对光合作用相关查询的响应的准确性和相关性。PRAG 在与科学写作相关的五项指标上平均提高了 8.7%，来源透明度提高了 25.4%。此外，它的科学深度和领域覆盖率与光合作用研究论文相当。知识图谱被用来构建 PRAG 的响应与数据库内部和外部的试卷，这使得 PRAG 能够分别将关键实体与 63% 的数据库和 39.5% 的试卷相匹配。PRAG 可应用于光合作用研究和更广泛的植物科学领域，为更深入的数据分析和预测能力铺平道路。

Title: Classic4Children: Adapting Chinese Literary Classics for Children with Large Language Model

• Authors: Jiali Chen, Xusen Hei, Yuqi Xue, Zihan Wu, Jiayuan Xie, Yi Cai
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.01090
• Pdf URL: https://arxiv.org/pdf/2502.01090
• Copy Paste: [[2502.01090]] Classic4Children: Adapting Chinese Literary Classics for Children with Large Language Model(https://arxiv.org/abs/2502.01090)
• Keywords: language model, llm
• Abstract: Chinese literary classics hold significant cultural and educational value, offering deep insights into morality, history, and human nature. These works often include classical Chinese and complex narratives, making them difficult for children to read. To bridge this gap, we introduce a child-friendly literary adaptation (CLA) task to adapt the Chinese literary classic into engaging and accessible text for children. However, recent large language models (LLMs) overlook children's reading preferences (\ie, vivid character portrayals, concise narrative structures, and appropriate readability), which poses challenges in CLA. In this paper, we propose a method called InstructChild, which augments the LLM with these preferences for adaptation. Specifically, we first obtain the characters' personalities and narrative structure as additional information for fine-grained instruction tuning. Then, we devise a readability metric as the reward to align the LLM with the children's reading level. Finally, a lookahead decoding strategy is applied to improve the readability of the generated text during inference. To support the evaluation of CLA task, we construct the Classic4Children dataset, which comprises both the original and child-friendly versions of the Four Great Classical Novels of Chinese literature. Experimental results show that our InstructChild significantly improves automatic and human evaluation performance.
• 摘要：中国文学经典具有重要的文化和教育价值，对道德、历史和人性提供了深刻的见解。这些作品通常包括古典中文和复杂的叙事，儿童难以阅读。为了弥补这一差距，我们引入了儿童友好型文学改编 (CLA) 任务，将中国文学经典改编成引人入胜且易于儿童阅读的文本。然而，最近的大型语言模型 (LLM) 忽略了儿童的阅读偏好（即生动的人物刻画、简洁的叙事结构和适当的可读性），这对 CLA 提出了挑战。在本文中，我们提出了一种称为 InstructChild 的方法，它通过这些偏好增强了 LLM 的改编。具体来说，我们首先获得人物的个性和叙事结构作为细粒度指令调整的附加信息。然后，我们设计了一个可读性指标作为奖励，以使 LLM 与儿童的阅读水平保持一致。最后，应用前瞻解码策略来提高推理过程中生成文本的可读性。为了支持 CLA 任务的评估，我们构建了 Classic4Children 数据集，该数据集包含中国四大名著的原版和儿童版。实验结果表明，我们的 InstructChild 显著提高了自动和人工评估的性能。

Title: Enhancing Aspect-based Sentiment Analysis with ParsBERT in Persian Language

• Authors: Farid Ariai, Maryam Tayefeh Mahmoudi, Ali Moeini
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.01091
• Pdf URL: https://arxiv.org/pdf/2502.01091
• Copy Paste: [[2502.01091]] Enhancing Aspect-based Sentiment Analysis with ParsBERT in Persian Language(https://arxiv.org/abs/2502.01091)
• Keywords: language model
• Abstract: In the era of pervasive internet use and the dominance of social networks, researchers face significant challenges in Persian text mining including the scarcity of adequate datasets in Persian and the inefficiency of existing language models. This paper specifically tackles these challenges, aiming to amplify the efficiency of language models tailored to the Persian language. Focusing on enhancing the effectiveness of sentiment analysis, our approach employs an aspect-based methodology utilizing the ParsBERT model, augmented with a relevant lexicon. The study centers on sentiment analysis of user opinions extracted from the Persian website 'Digikala.' The experimental results not only highlight the proposed method's superior semantic capabilities but also showcase its efficiency gains with an accuracy of 88.2% and an F1 score of 61.7. The importance of enhancing language models in this context lies in their pivotal role in extracting nuanced sentiments from user-generated content, ultimately advancing the field of sentiment analysis in Persian text mining by increasing efficiency and accuracy.
• 摘要：在互联网普及和社交网络占据主导地位的时代，研究人员在波斯语文本挖掘方面面临着重大挑战，包括波斯语数据集不足以及现有语言模型效率低下。本文专门解决了这些挑战，旨在提高针对波斯语的语言模型的效率。我们的方法侧重于提高情绪分析的有效性，采用了基于方面的方法，利用了 ParsBERT 模型，并增强了相关词典。该研究以从波斯语网站“Digikala”中提取的用户意见的情绪分析为中心。实验结果不仅突出了所提出方法的卓越语义能力，而且还展示了其效率提升，准确率为 88.2%，F1 得分为 61.7。在这种情况下，增强语言模型的重要性在于它们在从用户生成的内容中提取细微情绪方面发挥着关键作用，最终通过提高效率和准确性推动波斯语文本挖掘中的情绪分析领域发展。

Title: Language Models Prefer What They Know: Relative Confidence Estimation via Confidence Preferences

• Authors: Vaishnavi Shrivastava, Ananya Kumar, Percy Liang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.01126
• Pdf URL: https://arxiv.org/pdf/2502.01126
• Copy Paste: [[2502.01126]] Language Models Prefer What They Know: Relative Confidence Estimation via Confidence Preferences(https://arxiv.org/abs/2502.01126)
• Keywords: language model, gpt
• Abstract: Language models (LMs) should provide reliable confidence estimates to help users detect mistakes in their outputs and defer to human experts when necessary. Asking a language model to assess its confidence ("Score your confidence from 0-1.") is a natural way of evaluating its uncertainty. However, models struggle to provide absolute assessments of confidence (i.e. judging confidence in answering a question independent of other questions) and the coarse-grained scores they produce are not useful for evaluating the correctness of their answers. We propose relative confidence estimation, where we match up questions against each other and ask the model to make relative judgments of confidence ("Which question are you more confident in answering correctly?"). Treating each question as a "player" in a series of matchups against other questions and the model's preferences as match outcomes, we can use rank aggregation methods like Elo rating and Bradley-Terry to translate the model's confidence preferences into confidence scores. We evaluate relative confidence estimation against absolute confidence estimation and self-consistency confidence methods on five state-of-the-art LMs -- GPT-4, GPT-4o, Gemini 1.5 Pro, Claude 3.5 Sonnet, and Llama 3.1 405B -- across 14 challenging STEM, social science, and commonsense reasoning question answering tasks. Our results demonstrate that relative confidence estimation consistently provides more reliable confidence scores than absolute confidence estimation, with average gains of 3.5% in selective classification AUC over direct absolute confidence estimation methods and 1.7% over self-consistency approaches across all models and datasets.
• 摘要：语言模型 (LM) 应提供可靠的置信度估计，以帮助用户检测其输出中的错误，并在必要时听从人类专家的建议。要求语言模型评估其置信度（“从 0 到 1 对你的置信度进行评分。”）是评估其不确定性的自然方法。然而，模型很难提供绝对的置信度评估（即判断回答一个问题时独立于其他问题的置信度），它们产生的粗粒度分数对于评估答案的正确性没有用处。我们提出了相对置信度估计，我们将问题相互匹配，并要求模型做出相对的置信度判断（“你更有信心正确回答哪个问题？”）。将每个问题视为与其他问题进行一系列匹配的“玩家”，将模型的偏好视为匹配结果，我们可以使用 Elo 评级和 Bradley-Terry 等排名聚合方法将模型的置信度偏好转化为置信度分数。我们在 14 项具有挑战性的 STEM、社会科学和常识推理问答任务中，针对五种最先进的 LM（GPT-4、GPT-4o、Gemini 1.5 Pro、Claude 3.5 Sonnet 和 Llama 3.1 405B）评估了相对置信度估计与绝对置信度估计和自洽置信度方法的比较。我们的结果表明，相对置信度估计始终比绝对置信度估计提供更可靠的置信度分数，在所有模型和数据集中，选择性分类 AUC 平均比直接绝对置信度估计方法高出 3.5%，比自洽方法高出 1.7%。

Title: Jailbreaking with Universal Multi-Prompts

• Authors: Yu-Ling Hsu, Hsuan Su, Shang-Tse Chen
• Subjects: cs.CL, cs.AI, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.01154
• Pdf URL: https://arxiv.org/pdf/2502.01154
• Copy Paste: [[2502.01154]] Jailbreaking with Universal Multi-Prompts(https://arxiv.org/abs/2502.01154)
• Keywords: language model, llm, prompt
• Abstract: Large language models (LLMs) have seen rapid development in recent years, revolutionizing various applications and significantly enhancing convenience and productivity. However, alongside their impressive capabilities, ethical concerns and new types of attacks, such as jailbreaking, have emerged. While most prompting techniques focus on optimizing adversarial inputs for individual cases, resulting in higher computational costs when dealing with large datasets. Less research has addressed the more general setting of training a universal attacker that can transfer to unseen tasks. In this paper, we introduce JUMP, a prompt-based method designed to jailbreak LLMs using universal multi-prompts. We also adapt our approach for defense, which we term DUMP. Experimental results demonstrate that our method for optimizing universal multi-prompts outperforms existing techniques.
• 摘要：近年来，大型语言模型 (LLM) 发展迅速，彻底改变了各种应用，显著提高了便利性和生产力。然而，伴随着其令人印象深刻的功能，道德问题和新型攻击（如越狱）也随之出现。虽然大多数提示技术侧重于针对个案优化对抗性输入，但处理大型数据集时计算成本更高。较少研究涉及训练可以迁移到未见任务的通用攻击者的更一般设置。在本文中，我们介绍了 JUMP，这是一种基于提示的方法，旨在使用通用多提示越狱 LLM。我们还调整了我们的防御方法，我们称之为 DUMP。实验结果表明，我们优化通用多提示的方法优于现有技术。

Title: Joint Localization and Activation Editing for Low-Resource Fine-Tuning

• Authors: Wen Lai, Alexander Fraser, Ivan Titov
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.01179
• Pdf URL: https://arxiv.org/pdf/2502.01179
• Copy Paste: [[2502.01179]] Joint Localization and Activation Editing for Low-Resource Fine-Tuning(https://arxiv.org/abs/2502.01179)
• Keywords: llm
• Abstract: Parameter-efficient fine-tuning (PEFT) methods, such as LoRA, are commonly used to adapt LLMs. However, the effectiveness of standard PEFT methods is limited in low-resource scenarios with only a few hundred examples. Recent advances in interpretability research have inspired the emergence of activation editing techniques, which modify the activations of specific model components. These methods, due to their extremely small parameter counts, show promise for small datasets. However, their performance is highly dependent on identifying the correct modules to edit and often lacks stability across different datasets. In this paper, we propose Joint Localization and Activation Editing (JoLA), a method that jointly learns (1) which heads in the Transformer to edit (2) whether the intervention should be additive, multiplicative, or both and (3) the intervention parameters themselves - the vectors applied as additive offsets or multiplicative scalings to the head output. Through evaluations on three benchmarks spanning commonsense reasoning, natural language understanding, and natural language generation, we demonstrate that JoLA consistently outperforms existing methods.
• 摘要：参数高效微调 (PEFT) 方法（例如 LoRA）通常用于调整 LLM。但是，标准 PEFT 方法的有效性在资源匮乏的场景中受到限制，只有几百个示例。可解释性研究的最新进展激发了激活编辑技术的出现，这些技术可以修改特定模型组件的激活。由于这些方法的参数数量极少，因此有望应用于小型数据集。但是，它们的性能高度依赖于识别要编辑的正确模块，并且通常在不同的数据集之间缺乏稳定性。在本文中，我们提出了联合定位和激活编辑 (JoLA)，这种方法可以联合学习 (1) Transformer 中的哪些头部需要编辑 (2) 干预应该是加法、乘法还是两者兼有，以及 (3) 干预参数本身 - 作为头部输出的加法偏移或乘法缩放应用的向量。通过对三个基准的评估，涵盖常识推理、自然语言理解和自然语言生成，我们证明 JoLA 始终优于现有方法。

Title: OCR Error Post-Correction with LLMs in Historical Documents: No Free Lunches

• Authors: Jenna Kanerva, Cassandra Ledins, Siiri Käpyaho, Filip Ginter
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.01205
• Pdf URL: https://arxiv.org/pdf/2502.01205
• Copy Paste: [[2502.01205]] OCR Error Post-Correction with LLMs in Historical Documents: No Free Lunches(https://arxiv.org/abs/2502.01205)
• Keywords: llm
• Abstract: Optical Character Recognition (OCR) systems often introduce errors when transcribing historical documents, leaving room for post-correction to improve text quality. This study evaluates the use of open-weight LLMs for OCR error correction in historical English and Finnish datasets. We explore various strategies, including parameter optimization, quantization, segment length effects, and text continuation methods. Our results demonstrate that while modern LLMs show promise in reducing character error rates (CER) in English, a practically useful performance for Finnish was not reached. Our findings highlight the potential and limitations of LLMs in scaling OCR post-correction for large historical corpora.
• 摘要：光学字符识别 (OCR) 系统在转录历史文档时经常会出现错误，因此需要事后纠正以提高文本质量。本研究评估了开放权重 LLM 在历史英语和芬兰语数据集中用于 OCR 错误纠正的效果。我们探索了各种策略，包括参数优化、量化、段长度效应和文本延续方法。我们的结果表明，虽然现代 LLM 在降低英语字符错误率 (CER) 方面表现出色，但对于芬兰语却没有达到实用的性能。我们的研究结果突出了 LLM 在扩展大型历史语料库的 OCR 事后纠正方面的潜力和局限性。

Title: On the Robustness of Temporal Factual Knowledge in Language Models

• Authors: Hichem Ammar Khodja, Frédéric Béchet, Quentin Brabant, Alexis Nasr, Gwénolé Lecorvé
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.01220
• Pdf URL: https://arxiv.org/pdf/2502.01220
• Copy Paste: [[2502.01220]] On the Robustness of Temporal Factual Knowledge in Language Models(https://arxiv.org/abs/2502.01220)
• Keywords: language model, prompt
• Abstract: This paper explores the temporal robustness of language models (LMs) in handling factual knowledge. While LMs can often complete simple factual statements, their ability to manage temporal facts (those valid only within specific timeframes) remains uncertain. We design a controlled experiment to test the robustness of temporal factual knowledge inside LMs, which we use to evaluate several pretrained and instruction-tuned models using prompts on popular Wikidata facts, assessing their performance across different temporal granularities (Day, Month, and Year). Our findings indicate that even very large state-of-the-art models, such as Llama-3.1-70B, vastly lack robust knowledge of temporal facts. In addition, they are incapable of generalizing their knowledge from one granularity to another. These results highlight the inherent limitations of using LMs as temporal knowledge bases. The source code and data to reproduce our experiments will be released.
• 摘要：本文探讨了语言模型 (LM) 在处理事实知识时的时间稳健性。虽然 LM 通常可以完成简单的事实陈述，但它们管理时间事实（仅在特定时间范围内有效的事实）的能力仍不确定。我们设计了一个受控实验来测试 LM 内部时间事实知识的稳健性，我们使用该实验评估几个经过预训练和指令调整的模型，使用流行的 Wikidata 事实提示，评估它们在不同时间粒度（日、月和年）上的表现。我们的研究结果表明，即使是非常大的最先进的模型，例如 Llama-3.1-70B，也严重缺乏对时间事实的稳健知识。此外，他们无法将知识从一个粒度推广到另一个粒度。这些结果突出了使用 LM 作为时间知识库的固有局限性。我们将发布用于重现我们实验的源代码和数据。

Title: OphthBench: A Comprehensive Benchmark for Evaluating Large Language Models in Chinese Ophthalmology

• Authors: Chengfeng Zhou, Ji Wang, Juanjuan Qin, Yining Wang, Ling Sun, Weiwei Dai
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.01243
• Pdf URL: https://arxiv.org/pdf/2502.01243
• Copy Paste: [[2502.01243]] OphthBench: A Comprehensive Benchmark for Evaluating Large Language Models in Chinese Ophthalmology(https://arxiv.org/abs/2502.01243)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have shown significant promise across various medical applications, with ophthalmology being a notable area of focus. Many ophthalmic tasks have shown substantial improvement through the integration of LLMs. However, before these models can be widely adopted in clinical practice, evaluating their capabilities and identifying their limitations is crucial. To address this research gap and support the real-world application of LLMs, we introduce the OphthBench, a specialized benchmark designed to assess LLM performance within the context of Chinese ophthalmic practices. This benchmark systematically divides a typical ophthalmic clinical workflow into five key scenarios: Education, Triage, Diagnosis, Treatment, and Prognosis. For each scenario, we developed multiple tasks featuring diverse question types, resulting in a comprehensive benchmark comprising 9 tasks and 591 questions. This comprehensive framework allows for a thorough assessment of LLMs' capabilities and provides insights into their practical application in Chinese ophthalmology. Using this benchmark, we conducted extensive experiments and analyzed the results from 39 popular LLMs. Our evaluation highlights the current gap between LLM development and its practical utility in clinical settings, providing a clear direction for future advancements. By bridging this gap, we aim to unlock the potential of LLMs and advance their development in ophthalmology.
• 摘要：大型语言模型 (LLM) 在各种医疗应用中显示出巨大的潜力，眼科是一个值得关注的重点领域。通过整合 LLM，许多眼科任务已显示出显着的改进。然而，在这些模型能够广泛应用于临床实践之前，评估它们的能力并确定它们的局限性至关重要。为了解决这一研究空白并支持 LLM 的实际应用，我们推出了 OphthBench，这是一个专门的基准，旨在评估中国眼科实践背景下的 LLM 性能。该基准系统地将典型的眼科临床工作流程分为五个关键场景：教育、分诊、诊断、治疗和预后。对于每个场景，我们开发了多个具有不同问题类型的任务，从而形成了一个包含 9 个任务和 591 个问题的综合基准。这个综合框架可以全面评估 LLM 的能力，并深入了解它们在中国眼科的实际应用。使用这个基准，我们进行了广泛的实验并分析了 39 个流行 LLM 的结果。我们的评估突出了法学硕士发展与其在临床环境中的实际应用之间的差距，为未来的发展提供了明确的方向。通过弥合这一差距，我们的目标是释放法学硕士的潜力并促进其在眼科领域的发展。

Title: Main Predicate and Their Arguments as Explanation Signals For Intent Classification

• Authors: Sameer Pimparkhede, Pushpak Bhattacharyya
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.01270
• Pdf URL: https://arxiv.org/pdf/2502.01270
• Copy Paste: [[2502.01270]] Main Predicate and Their Arguments as Explanation Signals For Intent Classification(https://arxiv.org/abs/2502.01270)
• Keywords: language model, chat, agent
• Abstract: Intent classification is crucial for conversational agents (chatbots), and deep learning models perform well in this area. However, little research has been done on the explainability of intent classification due to the absence of suitable benchmark data. Human annotation of explanation signals in text samples is time-consuming and costly. However, from inspection of data on intent classification, we see that, more often than not, the main verb denotes the action, and the direct object indicates the domain of conversation, serving as explanation signals for intent. This observation enables us to hypothesize that the main predicate in the text utterances, along with the arguments of the main predicate, can serve as explanation signals. Leveraging this, we introduce a new technique to automatically augment text samples from intent classification datasets with word-level explanations. We mark main predicates (primarily verbs) and their arguments (dependency relations) as explanation signals in benchmark intent classification datasets ATIS and SNIPS, creating a unique 21k-instance dataset for explainability. Further, we experiment with deep learning and language models. We observe that models that work well for classification do not perform well in explainability metrics like plausibility and faithfulness. We also observe that guiding models to focus on explanation signals from our dataset during training improves the plausibility Token F1 score by 3-4%, improving the model's reasoning.
• 摘要：意图分类对于对话代理（聊天机器人）至关重要，深度学习模型在这一领域表现良好。然而，由于缺乏合适的基准数据，对意图分类的可解释性的研究很少。人工注释文本样本中的解释信号既耗时又昂贵。然而，通过检查意图分类的数据，我们发现，主动词通常表示动作，直接宾语表示对话领域，作为意图的解释信号。这一观察使我们能够假设文本话语中的主要谓词以及主要谓词的参数可以作为解释信号。利用这一点，我们引入了一种新技术，使用单词级解释自动增强意图分类数据集中的文本样本。我们在基准意图分类数据集 ATIS 和 SNIPS 中将主要谓词（主要是动词）及其参数（依赖关系）标记为解释信号，从而创建一个独特的 21k 实例数据集以实现可解释性。此外，我们还尝试了深度学习和语言模型。我们观察到，分类效果良好的模型在可信度和忠实度等可解释性指标方面表现不佳。我们还观察到，在训练期间引导模型关注来自我们数据集的解释信号可将可信度 Token F1 得分提高 3-4%，从而改善模型的推理能力。

Title: AlignVLM: Bridging Vision and Language Latent Spaces for Multimodal Understanding

• Authors: Ahmed Masry, Juan A. Rodriguez, Tianyu Zhang, Suyuchen Wang, Chao Wang, Aarash Feizi, Akshay Kalkunte Suresh, Abhay Puri, Xiangru Jian, Pierre-André Noël, Sathwik Tejaswi Madhusudhan, Marco Pedersoli, Bang Liu, Nicolas Chapados, Yoshua Bengio, Enamul Hoque, Christopher Pal, Issam H. Laradji, David Vazquez, Perouz Taslakian, Spandana Gella, Sai Rajeswar
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.01341
• Pdf URL: https://arxiv.org/pdf/2502.01341
• Copy Paste: [[2502.01341]] AlignVLM: Bridging Vision and Language Latent Spaces for Multimodal Understanding(https://arxiv.org/abs/2502.01341)
• Keywords: language model, llm
• Abstract: Aligning visual features with language embeddings is a key challenge in vision-language models (VLMs). The performance of such models hinges on having a good connector that maps visual features generated by a vision encoder to a shared embedding space with the LLM while preserving semantic similarity. Existing connectors, such as multilayer perceptrons (MLPs), often produce out-of-distribution or noisy inputs, leading to misalignment between the modalities. In this work, we propose a novel vision-text alignment method, AlignVLM, that maps visual features to a weighted average of LLM text embeddings. Our approach leverages the linguistic priors encoded by the LLM to ensure that visual features are mapped to regions of the space that the LLM can effectively interpret. AlignVLM is particularly effective for document understanding tasks, where scanned document images must be accurately mapped to their textual content. Our extensive experiments show that AlignVLM achieves state-of-the-art performance compared to prior alignment methods. We provide further analysis demonstrating improved vision-text feature alignment and robustness to noise.
• 摘要：将视觉特征与语言嵌入对齐是视觉语言模型 (VLM) 中的一项关键挑战。此类模型的性能取决于是否拥有良好的连接器，该连接器将视觉编码器生成的视觉特征映射到与 LLM 共享的嵌入空间，同时保留语义相似性。现有的连接器（例如多层感知器 (MLP)）通常会产生分布不均或噪声输入，从而导致模态之间错位。在这项工作中，我们提出了一种新颖的视觉文本对齐方法 AlignVLM，它将视觉特征映射到 LLM 文本嵌入的加权平均值。我们的方法利用 LLM 编码的语言先验来确保将视觉特征映射到 LLM 可以有效解释的空间区域。AlignVLM 对于文档理解任务特别有效，其中扫描的文档图像必须准确地映射到其文本内容。我们的大量实验表明，与之前的对齐方法相比，AlignVLM 实现了最先进的性能。我们提供了进一步的分析，证明了视觉文本特征对齐和抗噪声能力的改善。

Title: Bias Beware: The Impact of Cognitive Biases on LLM-Driven Product Recommendations

• Authors: Giorgos Filandrianos, Angeliki Dimitriou, Maria Lymperaiou, Konstantinos Thomas, Giorgos Stamou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.01349
• Pdf URL: https://arxiv.org/pdf/2502.01349
• Copy Paste: [[2502.01349]] Bias Beware: The Impact of Cognitive Biases on LLM-Driven Product Recommendations(https://arxiv.org/abs/2502.01349)
• Keywords: language model, llm
• Abstract: The advent of Large Language Models (LLMs) has revolutionized product recommendation systems, yet their susceptibility to adversarial manipulation poses critical challenges, particularly in real-world commercial applications. Our approach is the first one to tap into human psychological principles, seamlessly modifying product descriptions, making these adversarial manipulations hard to detect. In this work, we investigate cognitive biases as black-box adversarial strategies, drawing parallels between their effects on LLMs and human purchasing behavior. Through extensive experiments on LLMs of varying scales, we reveal significant vulnerabilities in their use as recommenders, providing critical insights into safeguarding these systems.
• 摘要：大型语言模型 (LLM) 的出现彻底改变了产品推荐系统，但它们易受对抗性操纵的影响，这带来了严峻的挑战，尤其是在现实世界的商业应用中。我们的方法是第一个利用人类心理原理的方法，无缝修改产品描述，使这些对抗性操纵难以被发现。在这项工作中，我们将认知偏见作为黑盒对抗策略进行研究，并将其对 LLM 和人类购买行为的影响进行比较。通过对不同规模的 LLM 进行大量实验，我们发现了它们在用作推荐器时存在重大漏洞，从而为保护这些系统提供了重要见解。

Title: Topic-FlipRAG: Topic-Orientated Adversarial Opinion Manipulation Attacks to Retrieval-Augmented Generation Models

• Authors: Yuyang Gong, Zhuo Chen, Miaokun Chen, Fengchang Yu, Wei Lu, Xiaofeng Wang, Xiaozhong Liu, Jiawei Liu
• Subjects: cs.CL, cs.CR, cs.IR
• Abstract URL: https://arxiv.org/abs/2502.01386
• Pdf URL: https://arxiv.org/pdf/2502.01386
• Copy Paste: [[2502.01386]] Topic-FlipRAG: Topic-Orientated Adversarial Opinion Manipulation Attacks to Retrieval-Augmented Generation Models(https://arxiv.org/abs/2502.01386)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: Retrieval-Augmented Generation (RAG) systems based on Large Language Models (LLMs) have become essential for tasks such as question answering and content generation. However, their increasing impact on public opinion and information dissemination has made them a critical focus for security research due to inherent vulnerabilities. Previous studies have predominantly addressed attacks targeting factual or single-query manipulations. In this paper, we address a more practical scenario: topic-oriented adversarial opinion manipulation attacks on RAG models, where LLMs are required to reason and synthesize multiple perspectives, rendering them particularly susceptible to systematic knowledge poisoning. Specifically, we propose Topic-FlipRAG, a two-stage manipulation attack pipeline that strategically crafts adversarial perturbations to influence opinions across related queries. This approach combines traditional adversarial ranking attack techniques and leverages the extensive internal relevant knowledge and reasoning capabilities of LLMs to execute semantic-level perturbations. Experiments show that the proposed attacks effectively shift the opinion of the model's outputs on specific topics, significantly impacting user information perception. Current mitigation methods cannot effectively defend against such attacks, highlighting the necessity for enhanced safeguards for RAG systems, and offering crucial insights for LLM security research.
• 摘要：基于大型语言模型 (LLM) 的检索增强生成 (RAG) 系统已成为问答和内容生成等任务的必备工具。然而，由于其固有的漏洞，它们对舆论和信息传播的影响日益增加，使其成为安全研究的关键焦点。以前的研究主要针对事实或单一查询操纵的攻击。在本文中，我们讨论了一个更实际的场景：针对 RAG 模型的主题型对抗性意见操纵攻击，其中 LLM 需要推理和综合多种观点，这使得它们特别容易受到系统性知识毒害。具体来说，我们提出了 Topic-FlipRAG，这是一种两阶段操纵攻击管道，它策略性地制作对抗性扰动以影响相关查询中的意见。这种方法结合了传统的对抗性排名攻击技术，并利用 LLM 广泛的内部相关知识和推理能力来执行语义级扰动。实验表明，所提出的攻击有效地改变了模型对特定主题的输出意见，显著影响了用户的信息感知。当前的缓解方法无法有效防御此类攻击，凸显了加强 RAG 系统保护措施的必要性，并为 LLM 安全研究提供了重要的见解。

Title: Emergent Stack Representations in Modeling Counter Languages Using Transformers

• Authors: Utkarsh Tiwari, Aviral Gupta, Michael Hahn
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.01432
• Pdf URL: https://arxiv.org/pdf/2502.01432
• Copy Paste: [[2502.01432]] Emergent Stack Representations in Modeling Counter Languages Using Transformers(https://arxiv.org/abs/2502.01432)
• Keywords: language model
• Abstract: Transformer architectures are the backbone of most modern language models, but understanding the inner workings of these models still largely remains an open problem. One way that research in the past has tackled this problem is by isolating the learning capabilities of these architectures by training them over well-understood classes of formal languages. We extend this literature by analyzing models trained over counter languages, which can be modeled using counter variables. We train transformer models on 4 counter languages, and equivalently formulate these languages using stacks, whose depths can be understood as the counter values. We then probe their internal representations for stack depths at each input token to show that these models when trained as next token predictors learn stack-like representations. This brings us closer to understanding the algorithmic details of how transformers learn languages and helps in circuit discovery.
• 摘要：Transformer 架构是大多数现代语言模型的支柱，但理解这些模型的内部工作原理在很大程度上仍然是一个悬而未决的问题。过去的研究解决这个问题的一种方法是，通过在易于理解的形式语言类上训练这些架构来分离它们的学习能力。我们通过分析在计数器语言上训练的模型来扩展这一文献，这些模型可以使用计数器变量建模。我们在 4 种计数器语言上训练 Transformer 模型，并使用堆栈等效地制定这些语言，堆栈的深度可以理解为计数器值。然后，我们探测它们在每个输入标记处的堆栈深度的内部表示，以表明这些模型在作为下一个标记预测器进行训练时会学习类似堆栈的表示。这使我们更接近理解 Transformer 如何学习语言的算法细节，并有助于电路发现。

Title: Towards Safer Chatbots: A Framework for Policy Compliance Evaluation of Custom GPTs

• Authors: David Rodriguez, William Seymour, Jose M. Del Alamo, Jose Such
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.01436
• Pdf URL: https://arxiv.org/pdf/2502.01436
• Copy Paste: [[2502.01436]] Towards Safer Chatbots: A Framework for Policy Compliance Evaluation of Custom GPTs(https://arxiv.org/abs/2502.01436)
• Keywords: language model, gpt, llm, prompt, chat
• Abstract: Large Language Models (LLMs) have gained unprecedented prominence, achieving widespread adoption across diverse domains and integrating deeply into society. The capability to fine-tune general-purpose LLMs, such as Generative Pre-trained Transformers (GPT), for specific tasks has facilitated the emergence of numerous Custom GPTs. These tailored models are increasingly made available through dedicated marketplaces, such as OpenAI's GPT Store. However, their black-box nature introduces significant safety and compliance risks. In this work, we present a scalable framework for the automated evaluation of Custom GPTs against OpenAI's usage policies, which define the permissible behaviors of these systems. Our framework integrates three core components: (1) automated discovery and data collection of models from the GPT store, (2) a red-teaming prompt generator tailored to specific policy categories and the characteristics of each target GPT, and (3) an LLM-as-a-judge technique to analyze each prompt-response pair for potential policy violations. We validate our framework with a manually annotated ground truth, and evaluate it through a large-scale study with 782 Custom GPTs across three categories: Romantic, Cybersecurity, and Academic GPTs. Our manual annotation process achieved an F1 score of 0.975 in identifying policy violations, confirming the reliability of the framework's assessments. The results reveal that 58.7% of the analyzed models exhibit indications of non-compliance, exposing weaknesses in the GPT store's review and approval processes. Furthermore, our findings indicate that a model's popularity does not correlate with compliance, and non-compliance issues largely stem from behaviors inherited from base models rather than user-driven customizations. We believe this approach is extendable to other chatbot platforms and policy domains, improving LLM-based systems safety.
• 摘要：大型语言模型 (LLM) 获得了前所未有的重视，在不同领域得到广泛采用并深度融入社会。针对特定任务微调通用 LLM（例如生成式预训练 Transformer (GPT)）的能力促进了大量自定义 GPT 的出现。这些定制模型越来越多地通过专用市场（例如 OpenAI 的 GPT Store）提供。然而，它们的黑箱性质带来了重大的安全和合规风险。在这项工作中，我们提出了一个可扩展的框架，用于根据 OpenAI 的使用政策自动评估自定义 GPT，该政策定义了这些系统的可允许行为。我们的框架集成了三个核心组件：(1) 从 GPT 商店自动发现和收集模型数据，(2) 针对特定政策类别和每个目标 GPT 特征定制的红队提示生成器，以及 (3) LLM-as-a-judge 技术，用于分析每个提示-响应对是否存在潜在的政策违规行为。我们用手动注释的地面实况验证了我们的框架，并通过一项大规模研究对浪漫、网络安全和学术 GPT 三个类别的 782 个自定义 GPT 进行了评估。我们的手动注释过程在识别政策违规方面取得了 0.975 的 F1 分数，证实了框架评估的可靠性。结果显示，58.7% 的分析模型表现出不合规的迹象，暴露了 GPT 商店的审核和批准流程中的弱点。此外，我们的研究结果表明，模型的受欢迎程度与合规性无关，不合规问题主要源于从基础模型继承的行为，而不是用户驱动的自定义。我们相信这种方法可以扩展到其他聊天机器人平台和策略领域，从而提高基于 LLM 的系统的安全性。

Title: FALCON: Fine-grained Activation Manipulation by Contrastive Orthogonal Unalignment for Large Language Model

• Authors: Jinwei Hu, Zhenglin Huang, Xiangyu Yin, Wenjie Ruan, Guangliang Cheng, Yi Dong, Xiaowei Huang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.01472
• Pdf URL: https://arxiv.org/pdf/2502.01472
• Copy Paste: [[2502.01472]] FALCON: Fine-grained Activation Manipulation by Contrastive Orthogonal Unalignment for Large Language Model(https://arxiv.org/abs/2502.01472)
• Keywords: language model
• Abstract: Large language models have been widely applied, but can inadvertently encode sensitive or harmful information, raising significant safety concerns. Machine unlearning has emerged to alleviate this concern; however, existing training-time unlearning approaches, relying on coarse-grained loss combinations, have limitations in precisely separating knowledge and balancing removal effectiveness with model utility. In contrast, we propose Fine-grained Activation manipuLation by Contrastive Orthogonal uNalignment (FALCON), a novel representation-guided unlearning approach that leverages information-theoretic guidance for efficient parameter selection, employs contrastive mechanisms to enhance representation separation, and projects conflict gradients onto orthogonal subspaces to resolve conflicts between forgetting and retention objectives. Extensive experiments demonstrate that FALCON achieves superior unlearning effectiveness while maintaining model utility, exhibiting robust resistance against knowledge recovery attempts.
• 摘要：大型语言模型已被广泛应用，但可能会无意中编码敏感或有害信息，从而引发重大安全问题。机器反学习的出现缓解了这种担忧；然而，现有的训练时反学习方法依赖于粗粒度损失组合，在精确分离知识和平衡删除有效性与模型效用方面存在局限性。相反，我们提出了对比正交对齐的细粒度激活操纵 (FALCON)，这是一种新颖的表示引导反学习方法，它利用信息论指导进行有效的参数选择，采用对比机制来增强表示分离，并将冲突梯度投射到正交子空间上以解决遗忘和保留目标之间的冲突。大量实验表明，FALCON 在保持模型效用的同时实现了卓越的反学习效果，并表现出对知识恢复尝试的强大抵抗力。

Title: Memorization Inheritance in Sequence-Level Knowledge Distillation for Neural Machine Translation

• Authors: Verna Dankers, Vikas Raunak
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.01491
• Pdf URL: https://arxiv.org/pdf/2502.01491
• Copy Paste: [[2502.01491]] Memorization Inheritance in Sequence-Level Knowledge Distillation for Neural Machine Translation(https://arxiv.org/abs/2502.01491)
• Keywords: hallucination
• Abstract: In this work, we explore how instance-level memorization in the teacher Neural Machine Translation (NMT) model gets inherited by the student model in sequence-level knowledge distillation (SeqKD). We find that despite not directly seeing the original training data, students memorize more than baseline models (models of the same size, trained on the original data) -- 3.4% for exact matches and 57% for extractive memorization -- and show increased hallucination rates. Further, under this SeqKD setting, we also characterize how students behave on specific training data subgroups, such as subgroups with low quality and specific counterfactual memorization (CM) scores, and find that students exhibit amplified denoising on low-quality subgroups. Finally, we propose a modification to SeqKD named Adaptive-SeqKD, which intervenes in SeqKD to reduce memorization and hallucinations. Overall, we recommend caution when applying SeqKD: students inherit both their teachers' superior performance and their fault modes, thereby requiring active monitoring.
• 摘要：在这项工作中，我们探索了在序列级知识提炼 (SeqKD) 中，教师神经机器翻译 (NMT) 模型中的实例级记忆如何被学生模型继承。我们发现，尽管没有直接看到原始训练数据，但学生比基线模型（相同大小、在原始数据上训练的模型）记住的内容更多——精确匹配为 3.4%，提取记忆为 57%——并且幻觉率增加。此外，在这种 SeqKD 设置下，我们还描述了学生在特定训练数据子组（例如质量低和特定反事实记忆 (CM) 分数的子组）上的行为，并发现学生在低质量子组上表现出放大的去噪。最后，我们提出了对 SeqKD 的修改，称为 Adaptive-SeqKD，它干预 SeqKD 以减少记忆和幻觉。总的来说，我们建议在应用 SeqKD 时要谨慎：学生既继承了老师的优秀表现，也继承了他们的错误模式，因此需要主动监控。

Title: CondAmbigQA: A Benchmark and Dataset for Conditional Ambiguous Question Answering

• Authors: Zongxi Li, Yang Li, Haoran Xie, S. Joe Qin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.01523
• Pdf URL: https://arxiv.org/pdf/2502.01523
• Copy Paste: [[2502.01523]] CondAmbigQA: A Benchmark and Dataset for Conditional Ambiguous Question Answering(https://arxiv.org/abs/2502.01523)
• Keywords: language model, llm, hallucination
• Abstract: Large language models (LLMs) are prone to hallucinations in question-answering (QA) tasks when faced with ambiguous questions. Users often assume that LLMs share their cognitive alignment, a mutual understanding of context, intent, and implicit details, leading them to omit critical information in the queries. However, LLMs generate responses based on assumptions that can misalign with user intent, which may be perceived as hallucinations if they misalign with the user's intent. Therefore, identifying those implicit assumptions is crucial to resolve ambiguities in QA. Prior work, such as AmbigQA, reduces ambiguity in queries via human-annotated clarifications, which is not feasible in real application. Meanwhile, ASQA compiles AmbigQA's short answers into long-form responses but inherits human biases and fails capture explicit logical distinctions that differentiates the answers. We introduce Conditional Ambiguous Question-Answering (CondAmbigQA), a benchmark with 200 ambiguous queries and condition-aware evaluation metrics. Our study pioneers the concept of ``conditions'' in ambiguous QA tasks, where conditions stand for contextual constraints or assumptions that resolve ambiguities. The retrieval-based annotation strategy uses retrieved Wikipedia fragments to identify possible interpretations for a given query as its conditions and annotate the answers through those conditions. Such a strategy minimizes human bias introduced by different knowledge levels among annotators. By fixing retrieval results, CondAmbigQA evaluates how RAG systems leverage conditions to resolve ambiguities. Experiments show that models considering conditions before answering improve performance by $20\%$, with an additional $5\%$ gain when conditions are explicitly provided. These results underscore the value of conditional reasoning in QA, offering researchers tools to rigorously evaluate ambiguity resolution.
• 摘要：大型语言模型 (LLM) 在问答 (QA) 任务中面对模棱两可的问题时容易产生幻觉。用户通常认为 LLM 具有相同的认知一致性，即对上下文、意图和隐含细节的共同理解，这导致他们忽略查询中的关键信息。然而，LLM 会根据可能与用户意图不一致的假设生成响应，如果这些假设与用户的意图不一致，则可能会被视为幻觉。因此，识别这些隐含假设对于解决 QA 中的歧义至关重要。先前的研究（例如 AmbigQA）通过人工注释的澄清来减少查询中的歧义，这在实际应用中是不可行的。同时，ASQA 将 AmbigQA 的简短答案编译成长格式的响应，但继承了人类的偏见，无法捕捉区分答案的明确逻辑区别。我们引入了条件模糊问答 (CondAmbigQA)，这是一个具有 200 个模糊查询和条件感知评估指标的基准。我们的研究率先在模糊性问答任务中引入了“条件”的概念，其中条件代表解决歧义的上下文约束或假设。基于检索的注释策略使用检索到的维基百科片段来识别给定查询的可能解释作为其条件，并通过这些条件注释答案。这种策略最大限度地减少了注释者之间知识水平不同而引入的人为偏见。通过修复检索结果，CondAmbigQA 评估 RAG 系统如何利用条件来解决歧义。实验表明，在回答之前考虑条件的模型可将性能提高 $20\%$，如果明确提供条件，则可额外提高 $5\%$。这些结果强调了条件推理在问答中的价值，为研究人员提供了严格评估歧义解决的工具。

Title: What is a Number, That a Large Language Model May Know It?

• Authors: Raja Marjieh, Veniamin Veselovsky, Thomas L. Griffiths, Ilia Sucholutsky
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.01540
• Pdf URL: https://arxiv.org/pdf/2502.01540
• Copy Paste: [[2502.01540]] What is a Number, That a Large Language Model May Know It?(https://arxiv.org/abs/2502.01540)
• Keywords: language model, llm, prompt
• Abstract: Numbers are a basic part of how humans represent and describe the world around them. As a consequence, learning effective representations of numbers is critical for the success of large language models as they become more integrated into everyday decisions. However, these models face a challenge: depending on context, the same sequence of digit tokens, e.g., 911, can be treated as a number or as a string. What kind of representations arise from this duality, and what are its downstream implications? Using a similarity-based prompting technique from cognitive science, we show that LLMs learn representational spaces that blend string-like and numerical representations. In particular, we show that elicited similarity judgments from these models over integer pairs can be captured by a combination of Levenshtein edit distance and numerical Log-Linear distance, suggesting an entangled representation. In a series of experiments we show how this entanglement is reflected in the latent embeddings, how it can be reduced but not entirely eliminated by context, and how it can propagate into a realistic decision scenario. These results shed light on a representational tension in transformer models that must learn what a number is from text input.
• 摘要：数字是人类表示和描述周围世界的基本部分。因此，随着大型语言模型越来越融入日常决策中，学习有效的数字表示对于大型语言模型的成功至关重要。然而，这些模型面临着一个挑战：根据上下文，相同的数字标记序列（例如 911）可以被视为数字或字符串。这种二元性会产生什么样的表示，其下游影响是什么？使用认知科学中基于相似性的提示技术，我们表明 LLM 学习融合了字符串和数字表示的表示空间。特别是，我们表明，这些模型对整数对的相似性判断可以通过 Levenshtein 编辑距离和数值对数线性距离的组合来捕获，这表明存在纠缠表示。在一系列实验中，我们展示了这种纠缠如何反映在潜在嵌入中，如何通过上下文减少但不能完全消除它，以及它如何传播到现实的决策场景中。这些结果揭示了 Transformer 模型中必须从文本输入中学习数字的含义的表征张力。

Title: Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understanding

• Authors: Mingyu Jin, Kai Mei, Wujiang Xu, Mingjie Sun, Ruixiang Tang, Mengnan Du, Zirui Liu, Yongfeng Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.01563
• Pdf URL: https://arxiv.org/pdf/2502.01563
• Copy Paste: [[2502.01563]] Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understanding(https://arxiv.org/abs/2502.01563)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have achieved remarkable success in contextual knowledge understanding. In this paper, we show that these concentrated massive values consistently emerge in specific regions of attention queries (Q) and keys (K) while not having such patterns in values (V) in various modern transformer-based LLMs (Q, K, and V mean the representations output by the query, key, and value layers respectively). Through extensive experiments, we further demonstrate that these massive values play a critical role in interpreting contextual knowledge (knowledge obtained from the current context window) rather than in retrieving parametric knowledge stored within the model's parameters. Our further investigation of quantization strategies reveals that ignoring these massive values leads to a pronounced drop in performance on tasks requiring rich contextual understanding, aligning with our analysis. Finally, we trace the emergence of concentrated massive values and find that such concentration is caused by Rotary Positional Encoding (RoPE), which has appeared since the first layers. These findings shed new light on how Q and K operate in LLMs and offer practical insights for model design and optimization. The Code is Available at this https URL.
• 摘要：大型语言模型 (LLM) 在上下文知识理解方面取得了显著的成功。在本文中，我们表明这些集中的大量值始终出现在注意查询 (Q) 和键 (K) 的特定区域中，而在各种现代基于转换器的 LLM 中的值 (V) 中没有这种模式（Q、K 和 V 分别表示查询、键和值层输出的表示）。通过大量实验，我们进一步证明这些大量值在解释上下文知识（从当前上下文窗口获得的知识）中起着关键作用，而不是在检索存储在模型参数中的参数知识中起着关键作用。我们对量化策略的进一步研究表明，忽略这些大量值会导致需要丰富上下文理解的任务的性能显着下降，这与我们的分析一致。最后，我们追踪了集中大量值的出现，发现这种集中是由从第一层开始出现的旋转位置编码 (RoPE) 引起的。这些发现为 Q 和 K 在 LLM 中的运作方式提供了新的见解，并为模型设计和优化提供了实用的见解。代码可在此 https URL 上获取。

Title: Scalable Language Models with Posterior Inference of Latent Thought Vectors

• Authors: Deqian Kong, Minglu Zhao, Dehong Xu, Bo Pang, Shu Wang, Edouardo Honig, Zhangzhang Si, Chuan Li, Jianwen Xie, Sirui Xie, Ying Nian Wu
• Subjects: cs.CL, cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2502.01567
• Pdf URL: https://arxiv.org/pdf/2502.01567
• Copy Paste: [[2502.01567]] Scalable Language Models with Posterior Inference of Latent Thought Vectors(https://arxiv.org/abs/2502.01567)
• Keywords: language model, llm
• Abstract: We propose a novel family of language models, Latent-Thought Language Models (LTMs), which incorporate explicit latent thought vectors that follow an explicit prior model in latent space. These latent thought vectors guide the autoregressive generation of ground tokens through a Transformer decoder. Training employs a dual-rate optimization process within the classical variational Bayes framework: fast learning of local variational parameters for the posterior distribution of latent vectors, and slow learning of global decoder parameters. Empirical studies reveal that LTMs possess additional scaling dimensions beyond traditional LLMs, yielding a structured design space. Higher sample efficiency can be achieved by increasing training compute per token, with further gains possible by trading model size for more inference steps. Designed based on these scaling properties, LTMs demonstrate superior sample and parameter efficiency compared to conventional autoregressive models and discrete diffusion models. They significantly outperform these counterparts in validation perplexity and zero-shot language modeling. Additionally, LTMs exhibit emergent few-shot in-context reasoning capabilities that scale with model and latent size, and achieve competitive performance in conditional and unconditional text generation.
• 摘要：我们提出了一种新型语言模型系列，即潜在思维语言模型 (LTM)，它结合了显式潜在思维向量，这些向量遵循潜在空间中的显式先验模型。这些潜在思维向量通过 Transformer 解码器指导基本标记的自回归生成。训练采用经典变分贝叶斯框架内的双速率优化过程：快速学习潜在向量后验分布的局部变分参数，并缓慢学习全局解码器参数。实证研究表明，LTM 拥有超越传统 LLM 的额外缩放维度，从而产生结构化的设计空间。通过增加每个标记的训练计算量可以实现更高的样本效率，并且可以通过牺牲模型大小来获得更多推理步骤，从而获得进一步的收益。基于这些缩放属性设计的 LTM 与传统自回归模型和离散扩散模型相比，具有出色的样本和参数效率。它们在验证困惑度和零样本语言建模方面明显优于这些同类模型。此外，LTM 表现出随着模型和潜在大小而扩展的新兴少样本上下文推理能力，并在条件和无条件文本生成中实现了具有竞争力的性能。

Title: Visual Theory of Mind Enables the Invention of Writing Systems

• Authors: Benjamin A. Spiegel, Lucas Gelfond, George Konidaris
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.01568
• Pdf URL: https://arxiv.org/pdf/2502.01568
• Copy Paste: [[2502.01568]] Visual Theory of Mind Enables the Invention of Writing Systems(https://arxiv.org/abs/2502.01568)
• Keywords: agent
• Abstract: Abstract symbolic writing systems are \textit{semiotic codes} that are ubiquitous in modern society but are otherwise absent in the animal kingdom. Anthropological evidence suggests that the earliest forms of some writing systems originally consisted of \textit{iconic pictographs}, which signify their referent via visual resemblance. While previous studies have examined the emergence and, separately, the evolution of pictographic writing systems through a computational lens, most employ non-naturalistic methodologies that make it difficult to draw clear analogies to human and animal cognition. We develop a multi-agent reinforcement learning testbed for emergent communication called a \textit{Signification Game}, and formulate a model of inferential communication that enables agents to leverage \textit{visual theory of mind} to communicate actions using pictographs. Our model, which is situated within a broader formalism for animal communication, sheds light on the cognitive and cultural processes that led to the development of early writing systems.
• 摘要：抽象符号书写系统是\textit{符号代码}，在现代社会中无处不在，但在动物界却不存在。人类学证据表明，一些书写系统的最早形式最初由\textit{图标象形文字}组成，这些文字通过视觉相似性来表示其指称物。虽然先前的研究已经通过计算视角分别研究了象形文字系统的出现和演变，但大多数研究都采用了非自然主义方法，很难与人类和动物的认知进行清晰的类比。我们开发了一种用于新兴通信的多智能体强化学习测试平台，称为\textit{意义游戏}，并制定了一个推理通信模型，使智能体能够利用\textit{视觉心理理论}通过象形文字传达动作。我们的模型位于动物通信的更广泛形式主义中，揭示了导致早期书写系统发展的认知和文化过程。

Title: ReGLA: Refining Gated Linear Attention

• Authors: Peng Lu, Ivan Kobyzev, Mehdi Rezagholizadeh, Boxing Chen, Philippe Langlais
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.01578
• Pdf URL: https://arxiv.org/pdf/2502.01578
• Copy Paste: [[2502.01578]] ReGLA: Refining Gated Linear Attention(https://arxiv.org/abs/2502.01578)
• Keywords: language model, llm
• Abstract: Recent advancements in Large Language Models (LLMs) have set themselves apart with their exceptional performance in complex language modelling tasks. However, these models are also known for their significant computational and storage requirements, primarily due to the quadratic computation complexity of softmax attention. To mitigate this issue, linear attention has been designed to reduce the quadratic space-time complexity that is inherent in standard transformers. In this work, we embarked on a comprehensive exploration of three key components that substantially impact the performance of the Gated Linear Attention module: feature maps, normalization, and the gating mechanism. We developed a feature mapping function to address some crucial issues that previous suggestions overlooked. Then we offered further rationale for the integration of normalization layers to stabilize the training process. Moreover, we explored the saturation phenomenon of the gating mechanism and augmented it with a refining module. We conducted extensive experiments and showed our architecture outperforms previous Gated Linear Attention mechanisms in extensive tasks including training from scratch and post-linearization with continual pre-training.
• 摘要：大型语言模型 (LLM) 的最新进展以其在复杂语言建模任务中的出色表现脱颖而出。然而，这些模型也因其巨大的计算和存储要求而闻名，这主要是由于 softmax 注意力的二次计算复杂度。为了缓解这个问题，线性注意力被设计用来降低标准 Transformer 固有的二次时空复杂度。在这项工作中，我们开始全面探索对门控线性注意力模块性能产生重大影响的三个关键组件：特征映射、规范化和门控机制。我们开发了一个特征映射函数来解决以前的建议忽略的一些关键问题。然后，我们为集成规范化层以稳定训练过程提供了进一步的理由。此外，我们探索了门控机制的饱和现象，并用精炼模块对其进行了增强。我们进行了广泛的实验，并表明我们的架构在包括从头开始训练和后线性化以及持续预训练在内的大量任务中优于以前的门控线性注意力机制。

Title: Breaking Focus: Contextual Distraction Curse in Large Language Models

• Authors: Yue Huang, Yanbo Wang, Zixiang Xu, Chujie Gao, Siyuan Wu, Jiayi Ye, Xiuying Chen, Pin-Yu Chen, Xiangliang Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.01609
• Pdf URL: https://arxiv.org/pdf/2502.01609
• Copy Paste: [[2502.01609]] Breaking Focus: Contextual Distraction Curse in Large Language Models(https://arxiv.org/abs/2502.01609)
• Keywords: language model, llm
• Abstract: Recent advances in Large Language Models (LLMs) have revolutionized generative systems, achieving excellent performance across diverse domains. Although these models perform well in controlled environments, their real-world applications frequently encounter inputs containing both essential and irrelevant details. Our investigation has revealed a critical vulnerability in LLMs, which we term Contextual Distraction Vulnerability (CDV). This phenomenon arises when models fail to maintain consistent performance on questions modified with semantically coherent but irrelevant context. To systematically investigate this vulnerability, we propose an efficient tree-based search methodology to automatically generate CDV examples. Our approach successfully generates CDV examples across four datasets, causing an average performance degradation of approximately 45% in state-of-the-art LLMs. To address this critical issue, we explore various mitigation strategies and find that post-targeted training approaches can effectively enhance model robustness against contextual distractions. Our findings highlight the fundamental nature of CDV as an ability-level challenge rather than a knowledge-level issue since models demonstrate the necessary knowledge by answering correctly in the absence of distractions. This calls the community's attention to address CDV during model development to ensure reliability. The code is available at this https URL.
• 摘要：大型语言模型 (LLM) 的最新进展彻底改变了生成系统，在不同领域取得了出色的性能。尽管这些模型在受控环境中表现良好，但它们的实际应用经常会遇到包含重要和不相关细节的输入。我们的调查揭示了 LLM 中的一个关键漏洞，我们将其称为上下文干扰漏洞 (CDV)。当模型无法在语义连贯但不相关的上下文修改的问题上保持一致的性能时，就会出现这种现象。为了系统地调查这种漏洞，我们提出了一种有效的基于树的搜索方法来自动生成 CDV 示例。我们的方法成功地在四个数据集中生成了 CDV 示例，导致最先进的 LLM 的平均性能下降了大约 45%。为了解决这个关键问题，我们探索了各种缓解策略，并发现后定位训练方法可以有效增强模型对上下文干扰的鲁棒性。我们的研究结果强调了 CDV 的基本性质，即能力水平挑战而不是知识水平问题，因为模型通过在没有干扰的情况下正确回答来展示必要的知识。这呼吁社区注意在模型开发过程中解决 CDV 问题，以确保可靠性。代码可在此 https URL 上获取。

Title: Large Language Models Are Human-Like Internally

• Authors: Tatsuki Kuribayashi, Yohei Oseki, Souhaib Ben Taieb, Kentaro Inui, Timothy Baldwin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.01615
• Pdf URL: https://arxiv.org/pdf/2502.01615
• Copy Paste: [[2502.01615]] Large Language Models Are Human-Like Internally(https://arxiv.org/abs/2502.01615)
• Keywords: language model
• Abstract: Recent cognitive modeling studies have reported that larger language models (LMs) exhibit a poorer fit to human reading behavior, leading to claims of their cognitive implausibility. In this paper, we revisit this argument through the lens of mechanistic interpretability and argue that prior conclusions were skewed by an exclusive focus on the final layers of LMs. Our analysis reveals that next-word probabilities derived from internal layers of larger LMs align with human sentence processing data as well as, or better than, those from smaller LMs. This alignment holds consistently across behavioral (self-paced reading times, gaze durations, MAZE task processing times) and neurophysiological (N400 brain potentials) measures, challenging earlier mixed results and suggesting that the cognitive plausibility of larger LMs has been underestimated. Furthermore, we first identify an intriguing relationship between LM layers and human measures: earlier layers correspond more closely with fast gaze durations, while later layers better align with relatively slower signals such as N400 potentials and MAZE processing times. Our work opens new avenues for interdisciplinary research at the intersection of mechanistic interpretability and cognitive modeling.
• 摘要：最近的认知建模研究报告称，较大的语言模型 (LM) 与人类阅读行为的拟合度较差，导致人们声称它们在认知上不合理。在本文中，我们从机械可解释性的视角重新审视了这一论点，并认为先前的结论因只关注 LM 的最后几层而出现偏差。我们的分析表明，从较大的 LM 的内部层得出的下一个单词概率与人类句子处理数据一致，甚至比较小的 LM 的概率更好。这种一致性在行为（自定进度的阅读时间、凝视持续时间、MAZE 任务处理时间）和神经生理（N400 脑电位）测量中始终保持不变，这对早期的混合结果提出了挑战，并表明人们低估了较大 LM 的认知合理性。此外，我们首先确定了 LM 层与人类测量之间的有趣关系：较早的层与快速凝视持续时间更接近，而较晚的层与相对较慢的信号（如 N400 电位和 MAZE 处理时间）更好地对齐。我们的工作为机械可解释性和认知建模交叉领域的跨学科研究开辟了新的途径。

Title: LLM-TA: An LLM-Enhanced Thematic Analysis Pipeline for Transcripts from Parents of Children with Congenital Heart Disease

• Authors: Muhammad Zain Raza, Jiawei Xu, Terence Lim, Lily Boddy, Carlos M. Mery, Andrew Well, Ying Ding
• Subjects: cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2502.01620
• Pdf URL: https://arxiv.org/pdf/2502.01620
• Copy Paste: [[2502.01620]] LLM-TA: An LLM-Enhanced Thematic Analysis Pipeline for Transcripts from Parents of Children with Congenital Heart Disease(https://arxiv.org/abs/2502.01620)
• Keywords: language model, gpt, llm, prompt
• Abstract: Thematic Analysis (TA) is a fundamental method in healthcare research for analyzing transcript data, but it is resource-intensive and difficult to scale for large, complex datasets. This study investigates the potential of large language models (LLMs) to augment the inductive TA process in high-stakes healthcare settings. Focusing on interview transcripts from parents of children with Anomalous Aortic Origin of a Coronary Artery (AAOCA), a rare congenital heart disease, we propose an LLM-Enhanced Thematic Analysis (LLM-TA) pipeline. Our pipeline integrates an affordable state-of-the-art LLM (GPT-4o mini), LangChain, and prompt engineering with chunking techniques to analyze nine detailed transcripts following the inductive TA framework. We evaluate the LLM-generated themes against human-generated results using thematic similarity metrics, LLM-assisted assessments, and expert reviews. Results demonstrate that our pipeline outperforms existing LLM-assisted TA methods significantly. While the pipeline alone has not yet reached human-level quality in inductive TA, it shows great potential to improve scalability, efficiency, and accuracy while reducing analyst workload when working collaboratively with domain experts. We provide practical recommendations for incorporating LLMs into high-stakes TA workflows and emphasize the importance of close collaboration with domain experts to address challenges related to real-world applicability and dataset complexity. this https URL
• 摘要：主题分析 (TA) 是医疗保健研究中分析转录数据的基本方法，但它资源密集且难以扩展到大型复杂数据集。本研究探讨了大型语言模型 (LLM) 在高风险医疗保健环境中增强归纳式 TA 过程的潜力。我们重点关注患有罕见先天性心脏病冠状动脉主动脉起源异常 (AAOCA) 儿童父母的访谈记录，提出了一种 LLM 增强主题分析 (LLM-TA) 流程。我们的流程集成了价格合理的最先进 LLM (GPT-4o mini)、LangChain 和带有分块技术的快速工程，以根据归纳式 TA 框架分析九个详细的转录本。我们使用主题相似性指标、LLM 辅助评估和专家评审来评估 LLM 生成的主题与人工生成的结果。结果表明，我们的流程明显优于现有的 LLM 辅助 TA 方法。虽然管道本身在归纳性 TA 中尚未达到人类水平的质量，但它在与领域专家合作时显示出提高可扩展性、效率和准确性，同时减少分析师工作量的巨大潜力。我们为将 LLM 纳入高风险 TA 工作流程提供了实用建议，并强调与领域专家密切合作以应对与现实世界适用性和数据集复杂性相关的挑战的重要性。此 https URL

Title: Scaling Embedding Layers in Language Models

• Authors: Da Yu, Edith Cohen, Badih Ghazi, Yangsibo Huang, Pritish Kamath, Ravi Kumar, Daogao Liu, Chiyuan Zhang
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.01637
• Pdf URL: https://arxiv.org/pdf/2502.01637
• Copy Paste: [[2502.01637]] Scaling Embedding Layers in Language Models(https://arxiv.org/abs/2502.01637)
• Keywords: language model
• Abstract: We propose SCONE ($\textbf{S}$calable, $\textbf{C}$ontextualized, $\textbf{O}$ffloaded, $\textbf{N}$-gram $\textbf{E}$mbedding), a method for extending input embedding layers to enhance language model performance as layer size scales. To avoid increased decoding costs, SCONE retains the original vocabulary while introducing embeddings for a set of frequent $n$-grams. These embeddings provide contextualized representation for each input token and are learned with a separate model during training. During inference, they are precomputed and stored in off-accelerator memory with minimal impact on inference speed. SCONE enables two new scaling strategies: increasing the number of cached $n$-gram embeddings and scaling the model used to learn them, all while maintaining fixed inference-time FLOPS. We show that scaling both aspects allows SCONE to outperform a 1.9B parameter baseline across diverse corpora, while using only half the inference-time FLOPS.
• 摘要：我们提出了 SCONE ($\textbf{S}$calable, $\textbf{C}$ontextualized, $\textbf{O}$ffloaded, $\textbf{N}$-gram $\textbf{E}$mbedding)，这是一种扩展输入嵌入层的方法，可在层大小扩展时增强语言模型性能。为了避免增加解码成本，SCONE 保留了原始词汇，同时引入了一组频繁的 $n$-gram 的嵌入。这些嵌入为每个输入标记提供了上下文表示，并在训练期间使用单独的模型进行学习。在推理过程中，它们会预先计算并存储在非加速器内存中，对推理速度的影响最小。SCONE 支持两种新的扩展策略：增加缓存的 $n$-gram 嵌入的数量并扩展用于学习它们的模型，同时保持固定的推理时间 FLOPS。我们表明，通过扩展这两个方面，SCONE 可以在不同的语料库中超越 1.9B 参数基线，同时仅使用一半的推理时间 FLOPS。