2025-04-10

Title: Query Understanding in LLM-based Conversational Information Seeking

• Authors: Yifei Yuan, Zahra Abbasiantaeb, Yang Deng, Mohammad Aliannejadi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.06356
• Pdf URL: https://arxiv.org/pdf/2504.06356
• Copy Paste: [[2504.06356]] Query Understanding in LLM-based Conversational Information Seeking(https://arxiv.org/abs/2504.06356)
• Keywords: language model, llm
• Abstract: Query understanding in Conversational Information Seeking (CIS) involves accurately interpreting user intent through context-aware interactions. This includes resolving ambiguities, refining queries, and adapting to evolving information needs. Large Language Models (LLMs) enhance this process by interpreting nuanced language and adapting dynamically, improving the relevance and precision of search results in real-time. In this tutorial, we explore advanced techniques to enhance query understanding in LLM-based CIS systems. We delve into LLM-driven methods for developing robust evaluation metrics to assess query understanding quality in multi-turn interactions, strategies for building more interactive systems, and applications like proactive query management and query reformulation. We also discuss key challenges in integrating LLMs for query understanding in conversational search systems and outline future research directions. Our goal is to deepen the audience's understanding of LLM-based conversational query understanding and inspire discussions to drive ongoing advancements in this field.
• 摘要：对话信息寻求（CI）中的查询理解涉及通过上下文感知的互动准确解释用户意图。这包括解决歧义，精炼查询以及适应不断发展的信息需求。大型语言模型（LLM）通过解释细微的语言并动态调整来增强此过程，从而实时提高搜索的相关性和精度。在本教程中，我们探索了先进的技术，以增强基于LLM的CIS系统中的查询理解。我们深入研究了以LLM驱动的方法来开发可靠的评估指标，以评估查询多转交互作用中的查询质量，建立更多交互式系统的策略以及主动查询管理和诸如主动查询管理和查询重新重新重新重新重新重新重新制定。我们还讨论了整合LLM的关键挑战，以在对话搜索系统中查询理解并概述未来的研究方向。我们的目标是加深听众对基于LLM的对话查询理解的理解，并激发讨论以推动该领域正在进行的进步。

Title: The Zero Body Problem: Probing LLM Use of Sensory Language

• Authors: Rebecca M. M. Hicke, Sil Hamilton, David Mimno
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2504.06393
• Pdf URL: https://arxiv.org/pdf/2504.06393
• Copy Paste: [[2504.06393]] The Zero Body Problem: Probing LLM Use of Sensory Language(https://arxiv.org/abs/2504.06393)
• Keywords: language model, llm, prompt
• Abstract: Sensory language expresses embodied experiences ranging from taste and sound to excitement and stomachache. This language is of interest to scholars from a wide range of domains including robotics, narratology, linguistics, and cognitive science. In this work, we explore whether language models, which are not embodied, can approximate human use of embodied language. We extend an existing corpus of parallel human and model responses to short story prompts with an additional 18,000 stories generated by 18 popular models. We find that all models generate stories that differ significantly from human usage of sensory language, but the direction of these differences varies considerably between model families. Namely, Gemini models use significantly more sensory language than humans along most axes whereas most models from the remaining five families use significantly less. Linear probes run on five models suggest that they are capable of identifying sensory language. However, we find preliminary evidence suggesting that instruction tuning may discourage usage of sensory language. Finally, to support further work, we release our expanded story dataset.
• 摘要：感觉语言表达了从口味，声音到兴奋和胃痛的体现体验。该语言吸引了来自包括机器人技术，叙事学，语言学和认知科学在内的各种领域的学者。在这项工作中，我们探讨了没有体现的语言模型是否可以近似人类对体现语言的使用。我们将现有的平行人类和模型响应扩展到短篇小说提示，并通过18个流行模型产生的另外18,000个故事。我们发现，所有模型都会产生与人类对感官语言使用显着差异的故事，但是这些差异的方向在模型家族之间差异很大。也就是说，双子座模型比大多数轴的人类使用更多的感觉语言，而其余五个家庭的大多数模型的使用量要少得多。线性探针在五个型号上运行，表明它们能够识别感觉语言。但是，我们发现初步证据表明，教学调整可能会阻止使用感官语言。最后，为了支持进一步的工作，我们发布了扩展的故事数据集。

Title: S'MoRE: Structural Mixture of Residual Experts for LLM Fine-tuning

• Authors: Hanqing Zeng, Yinglong Xia, Zhuokai Zhao, Gilbert Jiang, Qiang Zhang, Jiayi Liu, Lizhu Zhang, Xiangjun Fan, Benyu Zhang
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2504.06426
• Pdf URL: https://arxiv.org/pdf/2504.06426
• Copy Paste: [[2504.06426]] S'MoRE: Structural Mixture of Residual Experts for LLM Fine-tuning(https://arxiv.org/abs/2504.06426)
• Keywords: language model, llm
• Abstract: Fine-tuning pre-trained large language models (LLMs) presents a dual challenge of balancing parameter efficiency and model capacity. Existing methods like low-rank adaptations (LoRA) are efficient but lack flexibility, while Mixture-of-Experts (MoE) architectures enhance model capacity at the cost of more & under-utilized parameters. To address these limitations, we propose Structural Mixture of Residual Experts (S'MoRE), a novel framework that seamlessly integrates the efficiency of LoRA with the flexibility of MoE. Specifically, S'MoRE employs hierarchical low-rank decomposition of expert weights, yielding residuals of varying orders interconnected in a multi-layer structure. By routing input tokens through sub-trees of residuals, S'MoRE emulates the capacity of many experts by instantiating and assembling just a few low-rank matrices. We craft the inter-layer propagation of S'MoRE's residuals as a special type of Graph Neural Network (GNN), and prove that under similar parameter budget, S'MoRE improves "structural flexibility" of traditional MoE (or Mixture-of-LoRA) by exponential order. Comprehensive theoretical analysis and empirical results demonstrate that S'MoRE achieves superior fine-tuning performance, offering a transformative approach for efficient LLM adaptation.
• 摘要：微调预训练的大语言模型（LLMS）提出了平衡参数效率和模型容量的双重挑战。现有的方法（例如低级适应）（LORA）是有效的，但缺乏灵活性，而专家（MOE）体系结构的混合物以更多和未充分利用的参数为代价增强了模型容量。为了解决这些局限性，我们提出了剩余专家（S'More）的结构混合物，该结构混合物是一个新颖的框架，无缝地将Lora的效率整合到MOE的灵活性。具体而言，S'More采用了专家权重的层次低级分解，产生了在多层结构中相互联系的不同订单的残差。通过通过残差的子树来路由输入令牌，S'More通过仅实例化和组装少数低级别的矩阵来模仿许多专家的能力。我们制作了S'More残留物作为一种特殊类型的图形神经网络（GNN）的层间传播，并证明在相似的参数预算下，S'More通过指数顺序提高了传统MOE（或Lora混合物）的“结构性灵活性”。全面的理论分析和经验结果表明，S'More实现了出色的微调性能，为有效的LLM适应提供了变革性的方法。

Title: Language-Dependent Political Bias in AI: A Study of ChatGPT and Gemini

• Authors: Dogus Yuksel, Mehmet Cem Catalbas, Bora Oc
• Subjects: cs.CL, cs.AI, cs.ET, stat.AP
• Abstract URL: https://arxiv.org/abs/2504.06436
• Pdf URL: https://arxiv.org/pdf/2504.06436
• Copy Paste: [[2504.06436]] Language-Dependent Political Bias in AI: A Study of ChatGPT and Gemini(https://arxiv.org/abs/2504.06436)
• Keywords: language model, gpt, chat
• Abstract: As leading examples of large language models, ChatGPT and Gemini claim to provide accurate and unbiased information, emphasizing their commitment to political neutrality and avoidance of personal bias. This research investigates the political tendency of large language models and the existence of differentiation according to the query language. For this purpose, ChatGPT and Gemini were subjected to a political axis test using 14 different languages. The findings of the study suggest that these large language models do exhibit political tendencies, with both models demonstrating liberal and leftist biases. A comparative analysis revealed that Gemini exhibited a more pronounced liberal and left-wing tendency compared to ChatGPT. The study also found that these political biases varied depending on the language used for inquiry. The study delves into the factors that constitute political tendencies and linguistic differentiation, exploring differences in the sources and scope of educational data, structural and grammatical features of languages, cultural and political contexts, and the model's response to linguistic features. From this standpoint, and an ethical perspective, it is proposed that artificial intelligence tools should refrain from asserting a lack of political tendencies and neutrality, instead striving for political neutrality and executing user queries by incorporating these tendencies.
• 摘要：作为大型语言模型的主要例子，Chatgpt和Gemini声称提供了准确，公正的信息，强调了他们对政治中立性的承诺和避免个人偏见的承诺。这项研究调查了大语言模型的政治趋势以及根据查询语言的分化的存在。为此，使用14种不同的语言对Chatgpt和Gemini进行了政治轴心测试。研究结果表明，这些大语言模型确实表现出政治倾向，这两个模型都表明了自由主义者和左派偏见。比较分析表明，与Chatgpt相比，双子座表现出更为明显的自由主义和左翼趋势。研究还发现，这些政治偏见因用于查询的语言而异。该研究深入研究了构成政治倾向和语言差异化的因素，探索了教育数据的来源和范围的差异，语言的结构和语法特征，文化和政治环境以及模型对语言特征的反应。从这个角度来看，是一种道德的角度，有人提出人工智能工具应避免断言缺乏政治倾向和中立，而是努力争取政治中立，并通过结合这些趋势来执行用户查询。

Title: Don't Let It Hallucinate: Premise Verification via Retrieval-Augmented Logical Reasoning

• Authors: Yuehan Qin, Shawn Li, Yi Nian, Xinyan Velocity Yu, Yue Zhao, Xuezhe Ma
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.06438
• Pdf URL: https://arxiv.org/pdf/2504.06438
• Copy Paste: [[2504.06438]] Don't Let It Hallucinate: Premise Verification via Retrieval-Augmented Logical Reasoning(https://arxiv.org/abs/2504.06438)
• Keywords: language model, llm, hallucination, prompt, retrieval-augmented generation
• Abstract: Large language models (LLMs) have shown substantial capacity for generating fluent, contextually appropriate responses. However, they can produce hallucinated outputs, especially when a user query includes one or more false premises-claims that contradict established facts. Such premises can mislead LLMs into offering fabricated or misleading details. Existing approaches include pretraining, fine-tuning, and inference-time techniques that often rely on access to logits or address hallucinations after they occur. These methods tend to be computationally expensive, require extensive training data, or lack proactive mechanisms to prevent hallucination before generation, limiting their efficiency in real-time applications. We propose a retrieval-based framework that identifies and addresses false premises before generation. Our method first transforms a user's query into a logical representation, then applies retrieval-augmented generation (RAG) to assess the validity of each premise using factual sources. Finally, we incorporate the verification results into the LLM's prompt to maintain factual consistency in the final output. Experiments show that this approach effectively reduces hallucinations, improves factual accuracy, and does not require access to model logits or large-scale fine-tuning.
• 摘要：大型语言模型（LLMS）显示出产生流利，上下文适当的响应的大量能力。但是，它们可以产生幻觉的输出，尤其是当用户查询包含与确定事实相矛盾的一个或多个虚假场所所述时。这样的场所可能会误导LLM，以提供伪造或误导性的细节。现有的方法包括训练，微调和推理时间技术，这些技术通常依赖于对逻辑或地址幻觉发生后的访问。这些方法往往在计算上很昂贵，需要广泛的培训数据，或者缺乏积极的机制来防止在生成前幻觉，从而限制了它们在实时应用中的效率。我们提出了一个基于检索的框架，该框架可以识别和解决生成前的错误前提。我们的方法首先将用户的查询转换为逻辑表示形式，然后应用检索功能增强的生成（RAG）来评估使用事实来源的每个前提的有效性。最后，我们将验证结果纳入LLM的提示中，以保持最终输出中的事实一致性。实验表明，这种方法有效地降低了幻觉，提高了事实的准确性，并且不需要访问模型逻辑或大规模微调。

Title: Can LLMs Simulate Personas with Reversed Performance? A Benchmark for Counterfactual Instruction Following

• Authors: Sai Adith Senthil Kumar, Hao Yan, Saipavan Perepa, Murong Yue, Ziyu Yao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.06460
• Pdf URL: https://arxiv.org/pdf/2504.06460
• Copy Paste: [[2504.06460]] Can LLMs Simulate Personas with Reversed Performance? A Benchmark for Counterfactual Instruction Following(https://arxiv.org/abs/2504.06460)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) are now increasingly widely used to simulate personas in virtual environments, leveraging their instruction-following capability. However, we discovered that even state-of-the-art LLMs cannot simulate personas with reversed performance (e.g., student personas with low proficiency in educational settings), which impairs the simulation diversity and limits the practical applications of the simulated environments. In this work, using mathematical reasoning as a representative scenario, we propose the first benchmark dataset for evaluating LLMs on simulating personas with reversed performance, a capability that we dub "counterfactual instruction following". We evaluate both open-weight and closed-source LLMs on this task and find that LLMs, including the OpenAI o1 reasoning model, all struggle to follow counterfactual instructions for simulating reversedly performing personas. Intersectionally simulating both the performance level and the race population of a persona worsens the effect even further. These results highlight the challenges of counterfactual instruction following and the need for further research.
• 摘要：现在，大型语言模型（LLM）越来越广泛地用于模拟虚拟环境中的角色，利用其指导跟随能力。但是，我们发现，即使是最先进的LLM也无法以相反的表现模拟角色（例如，在教育环境中熟练水平较低的学生角色），这会损害模拟多样性并限制模拟环境的实际应用。在这项工作中，使用数学推理作为代表性方案，我们提出了第一个基准数据集，以评估LLMS以相反的性能模拟角色，这是我们认为“以下的反事实指令”的功能。我们在此任务上评估了开放式和封闭源LLMS，并发现包括OpenAi O1推理模型在内的LLM都难以按照反事实指令进行模拟反向执行角色。交叉点模拟角色的表现水平和种族人口，使效果进一步加剧。这些结果突出了反事实指导的挑战，以及需要进一步研究的必要性。

Title: Lugha-Llama: Adapting Large Language Models for African Languages

• Authors: Happy Buzaaba, Alexander Wettig, David Ifeoluwa Adelani, Christiane Fellbaum
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2504.06536
• Pdf URL: https://arxiv.org/pdf/2504.06536
• Copy Paste: [[2504.06536]] Lugha-Llama: Adapting Large Language Models for African Languages(https://arxiv.org/abs/2504.06536)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have achieved impressive results in a wide range of natural language applications. However, they often struggle to recognize low-resource languages, in particular African languages, which are not well represented in large training corpora. In this paper, we consider how to adapt LLMs to low-resource African languages. We find that combining curated data from African languages with high-quality English educational texts results in a training mix that substantially improves the model's performance on these languages. On the challenging IrokoBench dataset, our models consistently achieve the best performance amongst similarly sized baselines, particularly on knowledge-intensive multiple-choice questions (AfriMMLU). Additionally, on the cross-lingual question answering benchmark AfriQA, our models outperform the base model by over 10%. To better understand the role of English data during training, we translate a subset of 200M tokens into Swahili language and perform an analysis which reveals that the content of these data is primarily responsible for the strong performance. We release our models and data to encourage future research on African languages.
• 摘要：大型语言模型（LLM）在广泛的自然语言应用中取得了令人印象深刻的结果。但是，他们经常努力识别低资源语言，特别是非洲语言，这些语言在大型培训语料库中没有很好地代表。在本文中，我们考虑如何使LLM适应低资源的非洲语言。我们发现，将非洲语言的精选数据与高质量的英语教育文本相结合，从而实现了培训组合，从而大大提高了模型在这些语言上的表现。在具有挑战性的irokobench数据集上，我们的模型始终在大小相似的基准中取得最佳性能，尤其是在知识密集的多项选择问题上（Afrimmlu）。此外，在回答基准AFRIQA的跨语性问题上，我们的模型的表现优于基本模型超过10％。为了更好地了解培训期间英语数据的作用，我们将200m令牌的子集转化为斯瓦希里语语言，并进行分析，该分析表明这些数据的内容主要负责强大的性能。我们发布我们的模型和数据，以鼓励对非洲语言的未来研究。

Title: NeedleInATable: Exploring Long-Context Capability of Large Language Models towards Long-Structured Tables

• Authors: Lanrui Wang, Mingyu Zheng, Hongyin Tang, Zheng Lin, Yanan Cao, Jingang Wang, Xunliang Cai, Weiping Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.06560
• Pdf URL: https://arxiv.org/pdf/2504.06560
• Copy Paste: [[2504.06560]] NeedleInATable: Exploring Long-Context Capability of Large Language Models towards Long-Structured Tables(https://arxiv.org/abs/2504.06560)
• Keywords: language model, llm, agent
• Abstract: Processing structured tabular data, particularly lengthy tables, constitutes a fundamental yet challenging task for large language models (LLMs). However, existing long-context benchmarks primarily focus on unstructured text, neglecting the challenges of long and complex structured tables. To address this gap, we introduce NeedleInATable (NIAT), a novel task that treats each table cell as a "needle" and requires the model to extract the target cell under different queries. Evaluation results of mainstream LLMs on this benchmark show they lack robust long-table comprehension, often relying on superficial correlations or shortcuts for complex table understanding tasks, revealing significant limitations in processing intricate tabular data. To this end, we propose a data synthesis method to enhance models' long-table comprehension capabilities. Experimental results show that our synthesized training data significantly enhances LLMs' performance on the NIAT task, outperforming both long-context LLMs and long-table agent methods. This work advances the evaluation of LLMs' genuine long-structured table comprehension capabilities and paves the way for progress in long-context and table understanding applications.
• 摘要：处理结构化的表格数据，尤其是冗长的表格，构成了大型语言模型（LLMS）的基本且具有挑战性的任务。但是，现有的长篇小写基准主要集中于非结构化文本，忽略了长长而复杂的结构化表的挑战。为了解决这一差距，我们引入了针刺（NIAT），这是一个新的任务，将每个表单元视为“针”，并要求模型在不同的查询下提取目标细胞。主流LLM在此基准测试中的评估结果表明，它们缺乏强大的长桌理解，通常依靠表面相关性或快捷方式来理解复杂的桌子理解任务，从而揭示了处理复杂的表格数据的显着限制。为此，我们提出了一种数据综合方法，以增强模型的长桌理解能力。实验结果表明，我们的合成训练数据显着提高了LLM在NIAT任务上的性能，表现优于长篇文化LLM和长桌子的方法。这项工作推进了LLMS真正的长期结构桌子理解能力的评估，并为长篇文章和表格理解应用程序的进步铺平了道路。

Title: FuseRL: Dense Preference Optimization for Heterogeneous Model Fusion

• Authors: Longguang Zhong, Fanqi Wan, Ziyi Yang, Guosheng Liang, Tianyuan Shi, Xiaojun Quan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.06562
• Pdf URL: https://arxiv.org/pdf/2504.06562
• Copy Paste: [[2504.06562]] FuseRL: Dense Preference Optimization for Heterogeneous Model Fusion(https://arxiv.org/abs/2504.06562)
• Keywords: llm, prompt
• Abstract: Heterogeneous model fusion enhances the performance of LLMs by integrating the knowledge and capabilities of multiple structurally diverse models. However, existing approaches often rely solely on selecting the best output for each prompt from source models, which underutilizes their full potential due to limited source knowledge and results in sparse optimization signals. To address this limitation, we propose FuseRL, a novel two-stage framework comprising FuseSFT and FusePO to maximize the utilization of source LLMs. FuseSFT establishes a robust initialization by integrating the strengths of heterogeneous source models through weighted supervised fine-tuning (SFT) on diverse outputs for each prompt. FusePO optimizes weighted preferences based on the outputs of multiple source models to enable superior alignment performance. Extensive experiments demonstrate the effectiveness of our framework across various preference alignment methods, including RLOO, DPO, and SimPO. Using Llama-3.1-8B-Instruct as the target model, our approach achieves state-of-the-art performance among 8B LLMs on the AlpacaEval-2 and Arena-Hard benchmarks. Further analysis suggests that FuseSFT regularizes the training process to reduce overfitting, while FusePO introduces dense and diverse signals for preference optimization.
• 摘要：异质模型融合通过整合多种结构多样性模型的知识和能力来增强LLM的性能。但是，现有方法通常仅依赖于从源模型中选择每个提示的最佳输出，由于源知识有限并导致稀疏优化信号，从源模型中的每个提示都不足。为了解决这一限制，我们提出了Fuserl，这是一种新型的两阶段框架，包括Feseft和FusePo，以最大程度地利用源LLM。 Feseft通过在每个提示的各种输出上通过加权监督微调（SFT）整合异质源模型的强度来建立强大的初始化。 FUSEPO根据多个源模型的输出来优化加权偏好，以实现出色的对齐性能。广泛的实验证明了我们在包括Rloo，DPO和Simpo在内的各种偏好比对方法中的框架的有效性。我们使用Llama-3.1-8B教学作为目标模型，我们的方法在Alpacaeval-2和Arena-Hard-Hard-Hard基准测试中实现了8B LLM的最新性能。进一步的分析表明，Feseft将训练过程正规化以减少过度拟合，而Fusepo引入了密集和多样的信号以进行优化。

Title: Do Reasoning Models Show Better Verbalized Calibration?

• Authors: Qingcheng Zeng, Weihao Xuan, Leyang Cui, Rob Voigt
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.06564
• Pdf URL: https://arxiv.org/pdf/2504.06564
• Copy Paste: [[2504.06564]] Do Reasoning Models Show Better Verbalized Calibration?(https://arxiv.org/abs/2504.06564)
• Keywords: llm
• Abstract: Large reasoning models (LRMs) have recently shown impressive capabilities in complex reasoning by leveraging increased test-time computation and exhibiting behaviors akin to human-like deliberation. Despite these advances, it remains an open question whether LRMs are better calibrated - particularly in their verbalized confidence - compared to instruction-tuned counterparts. In this paper, we investigate the calibration properties of LRMs trained via supervised fine-tuning distillation on long reasoning traces (henceforth SFT reasoning models) and outcome-based reinforcement learning for reasoning (henceforth RL reasoning models) across diverse domains. Our findings reveal that LRMs significantly outperform instruction-tuned models on complex reasoning tasks in both accuracy and confidence calibration. In contrast, we find surprising trends in the domain of factuality in particular. On factuality tasks, while Deepseek-R1 shows strong calibration behavior, smaller QwQ-32B shows no improvement over instruct models; moreover, SFT reasoning models display worse calibration (greater overconfidence) compared to instruct models. Our results provide evidence for a potentially critical role of reasoning-oriented RL training in improving LLMs' capacity for generating trustworthy, self-aware outputs.
• 摘要：大型推理模型（LRMS）最近通过利用增加的测试时间计算并表现出类似于类似人类的审议的行为，在复杂的推理方面表现出了令人印象深刻的能力。尽管有这些进步，但与指导调节的对应物相比，LRM是否得到更好的校准（尤其是其口头上的信心）仍然是一个悬而未决的问题。在本文中，我们研究了通过在长期推理轨迹（此后SFT推理模型）和基于结果的推理学习（此后为RL推理模型）跨不同领域培训的LRMS的校准特性。我们的发现表明，在精确性和置信度校准中，LRMS在复杂的推理任务上的表现明显优于指令调节的模型。相比之下，我们发现了尤其是事实领域的令人惊讶的趋势。在事实任务上，虽然DeepSeek-R1显示出强烈的校准行为，但较小的QWQ-32B对指导模型没有任何改进。此外，与指导模型相比，SFT推理模型显示出更差的校准（更大的自信）。我们的结果提供了证据，证明了面向推理的RL培训在提高LLMS产生值得信赖的自我意识产出能力方面的潜在关键作用。

Title: Bypassing Safety Guardrails in LLMs Using Humor

• Authors: Pedro Cisneros-Velarde
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2504.06577
• Pdf URL: https://arxiv.org/pdf/2504.06577
• Copy Paste: [[2504.06577]] Bypassing Safety Guardrails in LLMs Using Humor(https://arxiv.org/abs/2504.06577)
• Keywords: language model, llm, prompt
• Abstract: In this paper, we show it is possible to bypass the safety guardrails of large language models (LLMs) through a humorous prompt including the unsafe request. In particular, our method does not edit the unsafe request and follows a fixed template -- it is simple to implement and does not need additional LLMs to craft prompts. Extensive experiments show the effectiveness of our method across different LLMs. We also show that both removing and adding more humor to our method can reduce its effectiveness -- excessive humor possibly distracts the LLM from fulfilling its unsafe request. Thus, we argue that LLM jailbreaking occurs when there is a proper balance between focus on the unsafe request and presence of humor.
• 摘要：在本文中，我们表明可以通过幽默的提示（包括不安全的请求）绕过大语言模型（LLMS）的安全护栏。特别是，我们的方法不会编辑不安全的请求并遵循固定的模板 - 实现很容易，并且不需要其他LLM来制作提示。广泛的实验显示了我们在不同LLM的方法的有效性。我们还表明，消除和增加更多幽默的方法都可以降低其有效性 - 过度的幽默可能会使LLM不满意其不安全的要求。因此，我们认为，当关注不安全的请求与幽默的存在之间存在适当的平衡时，LLM越狱就会发生。

Title: Automated Business Process Analysis: An LLM-Based Approach to Value Assessment

• Authors: William De Michele, Abel Armas Cervantes, Lea Frermann
• Subjects: cs.CL, cs.AI, cs.SE
• Abstract URL: https://arxiv.org/abs/2504.06600
• Pdf URL: https://arxiv.org/pdf/2504.06600
• Copy Paste: [[2504.06600]] Automated Business Process Analysis: An LLM-Based Approach to Value Assessment(https://arxiv.org/abs/2504.06600)
• Keywords: language model, llm, prompt
• Abstract: Business processes are fundamental to organizational operations, yet their optimization remains challenging due to the timeconsuming nature of manual process analysis. Our paper harnesses Large Language Models (LLMs) to automate value-added analysis, a qualitative process analysis technique that aims to identify steps in the process that do not deliver value. To date, this technique is predominantly manual, time-consuming, and subjective. Our method offers a more principled approach which operates in two phases: first, decomposing high-level activities into detailed steps to enable granular analysis, and second, performing a value-added analysis to classify each step according to Lean principles. This approach enables systematic identification of waste while maintaining the semantic understanding necessary for qualitative analysis. We develop our approach using 50 business process models, for which we collect and publish manual ground-truth labels. Our evaluation, comparing zero-shot baselines with more structured prompts reveals (a) a consistent benefit of structured prompting and (b) promising performance for both tasks. We discuss the potential for LLMs to augment human expertise in qualitative process analysis while reducing the time and subjectivity inherent in manual approaches.
• 摘要：业务流程对组织运营至关重要，但是由于手动过程分析的时间耗费性质，它们的优化仍然具有挑战性。我们的论文利用大型语言模型（LLMS）来自动化增值分析，这是一种定性过程分析技术，旨在确定流程中无法提供价值的步骤。迄今为止，这项技术主要是手动，耗时和主观的。我们的方法提供了一种更具原则性的方法，该方法分为两个阶段：首先将高级活动分解为详细的步骤，以实现颗粒状分析，其次，进行增值分析以根据精益原则对每个步骤进行分类。这种方法可以系统地识别废物，同时保持定性分析所需的语义理解。我们使用50个业务流程模型来开发我们的方法，并为其收集和发布手动基础真实标签。我们的评估，将零击基线与更结构化的提示进行比较，揭示了（a）结构化提示和（b）这两个任务的有希望的表现的一致好处。我们讨论了LLM在定性过程分析中增强人类专业知识的潜力，同时减少手动方法固有的时间和主观性。

Title: ThoughtProbe: Classifier-Guided Thought Space Exploration Leveraging LLM Intrinsic Reasoning

• Authors: Zijian Wang, Chang Xu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.06650
• Pdf URL: https://arxiv.org/pdf/2504.06650
• Copy Paste: [[2504.06650]] ThoughtProbe: Classifier-Guided Thought Space Exploration Leveraging LLM Intrinsic Reasoning(https://arxiv.org/abs/2504.06650)
• Keywords: language model, llm
• Abstract: Pre-trained large language models (LLMs) have been demonstrated to possess intrinsic reasoning capabilities that can emerge naturally when expanding the response space. However, the neural representation mechanisms underlying these intrinsic capabilities and approaches for their optimal utilization remain inadequately understood. In this work, we make the key discovery that a simple linear classifier can effectively detect intrinsic reasoning capabilities in LLMs' activation space, particularly within specific representation types and network layers. Based on this finding, we propose a classifier-guided search framework that strategically explore a tree-structured response space. In each node expansion, the classifier serves as a scoring and ranking mechanism that efficiently allocates computational resources by identifying and prioritizing more thoughtful reasoning directions for continuation. After completing the tree expansion, we collect answers from all branches to form a candidate answer pool. We propose a branch-aggregation selection method that marginalizes over all supporting branches by aggregating their thoughtfulness scores, thereby identifying the optimal answer from the pool. Experimental results show that our framework's comprehensive exploration not only covers valid reasoning chains but also effectively identifies them, achieving significant improvements across multiple arithmetic reasoning benchmarks.
• 摘要：已证明预先训练的大语言模型（LLM）具有内在的推理能力，在扩展响应空间时可以自然出现。但是，这些内在功能和最佳利用方法的神经表示机制仍然不充分理解。在这项工作中，我们进行了关键发现，即简单的线性分类器可以有效地检测LLMS激活空间中的内在推理能力，尤其是在特定表示类型和网络层中。基于这一发现，我们提出了一个分类器指导的搜索框架，该搜索框架从策略上探索了树结构化的响应空间。在每个节点扩展中，分类器都是一种评分和排名机制，通过识别和优先考虑更周到的推理方向来有效地分配计算资源。完成树的扩展后，我们从所有分支机构收集答案，形成一个候选答案池。我们提出了一种分支聚集的选择方法，该方法通过汇总其周到的分数来在所有支持分支上边缘化，从而从池中识别出最佳答案。实验结果表明，我们的框架的全面探索不仅涵盖了有效的推理链，而且可以有效地识别它们，从而在多个算术推理基准中取得了重大改进。

Title: SEE: Continual Fine-tuning with Sequential Ensemble of Experts

• Authors: Zhilin Wang, Yafu Li, Xiaoye Qu, Yu Cheng
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2504.06664
• Pdf URL: https://arxiv.org/pdf/2504.06664
• Copy Paste: [[2504.06664]] SEE: Continual Fine-tuning with Sequential Ensemble of Experts(https://arxiv.org/abs/2504.06664)
• Keywords: language model, llm
• Abstract: Continual fine-tuning of large language models (LLMs) suffers from catastrophic forgetting. Rehearsal-based methods mitigate this problem by retaining a small set of old data. Nevertheless, they still suffer inevitable performance loss. Although training separate experts for each task can help prevent forgetting, effectively assembling them remains a challenge. Some approaches use routers to assign tasks to experts, but in continual learning, they often require retraining for optimal performance. To address these challenges, we introduce the Sequential Ensemble of Experts (SEE) framework. SEE removes the need for an additional router, allowing each expert to independently decide whether a query should be handled. The framework employs distributed routing, and during continual fine-tuning, SEE only requires the training of new experts for incoming tasks rather than retraining the entire system. Experiments reveal that the SEE outperforms prior approaches, including multi-task learning, in continual fine-tuning. It also demonstrates remarkable generalization ability, as the expert can effectively identify out-of-distribution queries, which can then be directed to a more generalized model for resolution. This work highlights the promising potential of integrating routing and response mechanisms within each expert, paving the way for the future of distributed model ensembling.
• 摘要：大型语言模型（LLM）的持续微调遭受了灾难性的遗忘。基于排练的方法通过保留一小部分旧数据来减轻此问题。尽管如此，他们仍然不可避免地会丧失。尽管针对每个任务的培训单独的专家可以帮助防止忘记，但有效地组装他们仍然是一个挑战。有些方法使用路由器将任务分配给专家，但是在持续学习中，它们通常需要再培训才能获得最佳性能。为了应对这些挑战，我们介绍了专家（请参阅）框架的顺序合奏。请参阅删除需要额外的路由器的需求，使每个专家都可以独立决定是否应处理查询。该框架采用分布式路由，在持续的微调过程中，只需需要培训新专家即可完成任务，而不是重新训练整个系统。实验表明，See See优先的方法（包括多任务学习）在不断进行的微调中。它还表现出了非凡的概括能力，因为专家可以有效地识别分布外的查询，然后可以将其引导到更概括的分辨率以进行分辨率。这项工作突出了在每个专家中整合路由和响应机制的有希望的潜力，为分布式模型结合的未来铺平了道路。

Title: Inducing Programmatic Skills for Agentic Tasks

• Authors: Zora Zhiruo Wang, Apurva Gandhi, Graham Neubig, Daniel Fried
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.06821
• Pdf URL: https://arxiv.org/pdf/2504.06821
• Copy Paste: [[2504.06821]] Inducing Programmatic Skills for Agentic Tasks(https://arxiv.org/abs/2504.06821)
• Keywords: agent
• Abstract: To succeed in common digital tasks such as web navigation, agents must carry out a variety of specialized tasks such as searching for products or planning a travel route. To tackle these tasks, agents can bootstrap themselves by learning task-specific skills online through interaction with the web environment. In this work, we demonstrate that programs are an effective representation for skills. We propose agent skill induction (ASI), which allows agents to adapt themselves by inducing, verifying, and utilizing program-based skills on the fly. We start with an evaluation on the WebArena agent benchmark and show that ASI outperforms the static baseline agent and its text-skill counterpart by 23.5% and 11.3% in success rate, mainly thanks to the programmatic verification guarantee during the induction phase. ASI also improves efficiency by reducing 10.7-15.3% of the steps over baselines, by composing primitive actions (e.g., click) into higher-level skills (e.g., search product). We then highlight the efficacy of ASI in remaining efficient and accurate under scaled-up web activities. Finally, we examine the generalizability of induced skills when transferring between websites, and find that ASI can effectively reuse common skills, while also updating incompatible skills to versatile website changes.
• 摘要：为了成功执行通用数字任务，例如Web导航，代理必须执行各种专业任务，例如搜索产品或计划旅行路线。为了解决这些任务，代理可以通过与网络环境互动在线学习特定于任务的技能来引导自己。在这项工作中，我们证明程序是技能的有效代表。我们提出了代理技能归纳（ASI），该技术允许代理人通过即时诱导，验证和利用基于程序的技能来适应自己。我们从对Webarena代理基准的评估开始，并表明ASI的表现超过了静态基线代理及其文本技能的成功率23.5％和11.3％，这主要是由于感应阶段的程序验证保证。 ASI还通过将原始动作（例如，单击）纳入更高级别的技能（例如搜索产品）来降低基线步骤的10.7-15.3％，从而提高了效率。然后，我们强调了在扩大的Web活动下，ASI在保持高效，准确的功效中的功效。最后，我们研究了在网站之间传输诱导技能的普遍性，并发现ASI可以有效地重复使用共同的技能，同时还可以将不兼容的技能更新为多功能网站的变化。

Title: Open Problems and a Hypothetical Path Forward in LLM Knowledge Paradigms

• Authors: Xiaotian Ye, Mengqi Zhang, Shu Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.06823
• Pdf URL: https://arxiv.org/pdf/2504.06823
• Copy Paste: [[2504.06823]] Open Problems and a Hypothetical Path Forward in LLM Knowledge Paradigms(https://arxiv.org/abs/2504.06823)
• Keywords: language model, llm
• Abstract: Knowledge is fundamental to the overall capabilities of Large Language Models (LLMs). The knowledge paradigm of a model, which dictates how it encodes and utilizes knowledge, significantly affects its performance. Despite the continuous development of LLMs under existing knowledge paradigms, issues within these frameworks continue to constrain model potential. This blog post highlight three critical open problems limiting model capabilities: (1) challenges in knowledge updating for LLMs, (2) the failure of reverse knowledge generalization (the reversal curse), and (3) conflicts in internal knowledge. We review recent progress made in addressing these issues and discuss potential general solutions. Based on observations in these areas, we propose a hypothetical paradigm based on Contextual Knowledge Scaling, and further outline implementation pathways that remain feasible within contemporary techniques. Evidence suggests this approach holds potential to address current shortcomings, serving as our vision for future model paradigms. This blog post aims to provide researchers with a brief overview of progress in LLM knowledge systems, while provide inspiration for the development of next-generation model architectures.
• 摘要：知识是大语言模型（LLM）的整体能力至关重要的。模型的知识范式决定了它如何编码和利用知识，从而显着影响其性能。尽管LLM在现有知识范式下的持续发展，但这些框架内的问题继续限制模型的潜力。该博客文章强调了限制模型功能的三个关键开放问题：（1）知识更新的挑战，（2）反向知识概括（反向诅咒）的失败，以及（3）内部知识的冲突。我们回顾了解决这些问题并讨论潜在的一般解决方案方面取得的最新进展。根据这些领域的观察，我们提出了一个基于上下文知识扩展的假设范式，并提出了在当代技术中仍然可行的进一步概述实施途径。证据表明，这种方法具有解决当前缺点的潜力，这是我们对未来模型范式的愿景。这篇博客文章旨在为研究人员提供有关LLM知识系统进度的简要概述，同时为开发下一代模型体系结构提供了灵感。

Title: Integrating Cognitive Processing Signals into Language Models: A Review of Advances, Applications and Future Directions

• Authors: Angela Lopez-Cardona, Sebastian Idesis, Ioannis Arapakis
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.06843
• Pdf URL: https://arxiv.org/pdf/2504.06843
• Copy Paste: [[2504.06843]] Integrating Cognitive Processing Signals into Language Models: A Review of Advances, Applications and Future Directions(https://arxiv.org/abs/2504.06843)
• Keywords: language model, llm, hallucination
• Abstract: Recently, the integration of cognitive neuroscience in Natural Language Processing (NLP) has gained significant attention. This article provides a critical and timely overview of recent advancements in leveraging cognitive signals, particularly Eye-tracking (ET) signals, to enhance Language Models (LMs) and Multimodal Large Language Models (MLLMs). By incorporating user-centric cognitive signals, these approaches address key challenges, including data scarcity and the environmental costs of training large-scale models. Cognitive signals enable efficient data augmentation, faster convergence, and improved human alignment. The review emphasises the potential of ET data in tasks like Visual Question Answering (VQA) and mitigating hallucinations in MLLMs, and concludes by discussing emerging challenges and research trends.
• 摘要：最近，自然语言处理（NLP）中认知神经科学的整合引起了人们的重大关注。本文提供了有关利用认知信号（尤其是眼睛跟踪（ET）信号）的最新进步的关键和及时概述，以增强语言模型（LMS）和多模式大语言模型（MLLMS）。通过合并以用户为中心的认知信号，这些方法应对关键挑战，包括数据稀缺性和培训大规模模型的环境成本。认知信号可实现有效的数据增强，更快的收敛性和改善的人类对齐方式。该评论强调了ET数据在视觉问题回答（VQA）和减轻MLLM中的幻觉等任务中的潜力，并通过讨论新出现的挑战和研究趋势来结束。

Title: Persona Dynamics: Unveiling the Impact of Personality Traits on Agents in Text-Based Games

• Authors: Seungwon Lim, Seungbeen Lee, Dongjun Min, Youngjae Yu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.06868
• Pdf URL: https://arxiv.org/pdf/2504.06868
• Copy Paste: [[2504.06868]] Persona Dynamics: Unveiling the Impact of Personality Traits on Agents in Text-Based Games(https://arxiv.org/abs/2504.06868)
• Keywords: agent
• Abstract: Artificial agents are increasingly central to complex interactions and decision-making tasks, yet aligning their behaviors with desired human values remains an open challenge. In this work, we investigate how human-like personality traits influence agent behavior and performance within text-based interactive environments. We introduce PANDA: PersonalityAdapted Neural Decision Agents, a novel method for projecting human personality traits onto agents to guide their behavior. To induce personality in a text-based game agent, (i) we train a personality classifier to identify what personality type the agent's actions exhibit, and (ii) we integrate the personality profiles directly into the agent's policy-learning pipeline. By deploying agents embodying 16 distinct personality types across 25 text-based games and analyzing their trajectories, we demonstrate that an agent's action decisions can be guided toward specific personality profiles. Moreover, certain personality types, such as those characterized by higher levels of Openness, display marked advantages in performance. These findings underscore the promise of personality-adapted agents for fostering more aligned, effective, and human-centric decision-making in interactive environments.
• 摘要：人工代理人越来越重要地是复杂的互动和决策任务，但是将其行为与所需的人类价值保持一致仍然是一个开放的挑战。在这项工作中，我们研究了类似人类的人格特征如何影响基于文本的交互式环境中的代理行为和绩效。我们介绍了熊猫：人格化的神经决策剂，这是一种将人格特征投射到代理商以指导其行为的新方法。为了在基于文本的游戏代理中诱导个性，（i）我们培训一个个性分类器，以确定代理商的行为所展示的个性类型，（ii）我们将个性概况直接集成到代理商的政策学习管道中。通过在25种基于文本的游戏中部署代理人体现16种不同的个性类型并分析其轨迹，我们证明了代理商的行动决策可以引导到特定的人格概况。此外，某些人格类型（例如以较高的开放性为特征的人格类型）在表现上显示出明显的优势。这些发现强调了适应人格的代理人在交互式环境中培养更加一致，有效和以人为中心的决策的希望。

Title: Identifying Aspects in Peer Reviews

• Authors: Sheng Lu, Ilia Kuznetsov, Iryna Gurevych
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.06910
• Pdf URL: https://arxiv.org/pdf/2504.06910
• Copy Paste: [[2504.06910]] Identifying Aspects in Peer Reviews(https://arxiv.org/abs/2504.06910)
• Keywords: llm
• Abstract: Peer review is central to academic publishing, but the growing volume of submissions is straining the process. This motivates the development of computational approaches to support peer review. While each review is tailored to a specific paper, reviewers often make assessments according to certain aspects such as Novelty, which reflect the values of the research community. This alignment creates opportunities for standardizing the reviewing process, improving quality control, and enabling computational support. While prior work has demonstrated the potential of aspect analysis for peer review assistance, the notion of aspect remains poorly formalized. Existing approaches often derive aspect sets from review forms and guidelines of major NLP venues, yet data-driven methods for aspect identification are largely underexplored. To address this gap, our work takes a bottom-up approach: we propose an operational definition of aspect and develop a data-driven schema for deriving fine-grained aspects from a corpus of peer reviews. We introduce a dataset of peer reviews augmented with aspects and show how it can be used for community-level review analysis. We further show how the choice of aspects can impact downstream applications, such as LLM-generated review detection. Our results lay a foundation for a principled and data-driven investigation of review aspects, and pave the path for new applications of NLP to support peer review.
• 摘要：同行评审是学术出版的核心，但越来越多的提交量正在加剧这一过程。这激发了计算方法的发展，以支持同行评审。虽然每次评论都是针对特定论文量身定制的，但审阅者通常会根据某些方面（例如新颖性）进行评估，这些方面反映了研究界的价值。这种一致性为标准化审查过程，改善质量控制和实现计算支持提供了机会。虽然先前的工作证明了方面分析的同行评审援助的潜力，但方面的概念仍然很差。现有的方法通常从主要NLP场地的审查表格和指南中得出方面集，但是在很大程度上，数据驱动的方面识别方法却没有得到验证。为了解决这一差距，我们的工作采取了自下而上的方法：我们提出了对方面的操作定义，并开发了数据驱动的模式，用于从同行评审的语料库中得出细粒度。我们介绍了一项数据集的数据集，并展示了如何将其用于社区级审查分析。我们进一步展示了各个方面的选择如何影响下游应用程序，例如LLM生成的审查检测。我们的结果为对审查方面的原则和数据驱动的调查奠定了基础，并为NLP的新应用铺平了支持同行评审的道路。

Title: Data Augmentation for Fake Reviews Detection in Multiple Languages and Multiple Domains

• Authors: Ming Liu, Massimo Poesio
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.06917
• Pdf URL: https://arxiv.org/pdf/2504.06917
• Copy Paste: [[2504.06917]] Data Augmentation for Fake Reviews Detection in Multiple Languages and Multiple Domains(https://arxiv.org/abs/2504.06917)
• Keywords: language model
• Abstract: With the growth of the Internet, buying habits have changed, and customers have become more dependent on the online opinions of other customers to guide their purchases. Identifying fake reviews thus became an important area for Natural Language Processing (NLP) research. However, developing high-performance NLP models depends on the availability of large amounts of training data, which are often not available for low-resource languages or domains. In this research, we used large language models to generate datasets to train fake review detectors. Our approach was used to generate fake reviews in different domains (book reviews, restaurant reviews, and hotel reviews) and different languages (English and Chinese). Our results demonstrate that our data augmentation techniques result in improved performance at fake review detection for all domains and languages. The accuracy of our fake review detection model can be improved by 0.3 percentage points on DeRev TEST, 10.9 percentage points on Amazon TEST, 8.3 percentage points on Yelp TEST and 7.2 percentage points on DianPing TEST using the augmented datasets.
• 摘要：随着互联网的增长，购买习惯已经改变，客户已经更加依赖其他客户的在线观点来指导他们的购买。因此，确定虚假评论成为自然语言处理（NLP）研究的重要领域。但是，开发高性能的NLP模型取决于大量培训数据的可用性，这些数据通常无法用于低资源语言或域。在这项研究中，我们使用大型语言模型来生成数据集来训练伪造的审查探测器。我们的方法用于在不同领域（书评，餐厅评论和酒店评论）和不同语言（英语和中文）中产生虚假评论。我们的结果表明，我们的数据增强技术导致所有域和语言的虚假审核检测性能提高了性能。我们的假审查检测模型的准确性可以通过DEREV测试中的0.3个百分点，亚马逊测试中的10.9个百分点，Yelp测试中的8.3个百分点和7.2个百分点，并使用增强数据集对Dianping测试中的7.2个百分点。

Title: RuOpinionNE-2024: Extraction of Opinion Tuples from Russian News Texts

• Authors: Natalia Loukachevitch, Natalia Tkachenko, Anna Lapanitsyna, Mikhail Tikhomirov, Nicolay Rusnachenko
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.06947
• Pdf URL: https://arxiv.org/pdf/2504.06947
• Copy Paste: [[2504.06947]] RuOpinionNE-2024: Extraction of Opinion Tuples from Russian News Texts(https://arxiv.org/abs/2504.06947)
• Keywords: language model, prompt
• Abstract: In this paper, we introduce the Dialogue Evaluation shared task on extraction of structured opinions from Russian news texts. The task of the contest is to extract opinion tuples for a given sentence; the tuples are composed of a sentiment holder, its target, an expression and sentiment from the holder to the target. In total, the task received more than 100 submissions. The participants experimented mainly with large language models in zero-shot, few-shot and fine-tuning formats. The best result on the test set was obtained with fine-tuning of a large language model. We also compared 30 prompts and 11 open source language models with 3-32 billion parameters in the 1-shot and 10-shot settings and found the best models and prompts.
• 摘要：在本文中，我们介绍了关于从俄罗斯新闻文本中提取结构化意见的对话评估的共同任务。比赛的任务是提取给定句子的意见分组；这些元组由情绪持有人，其目标，表达式和持有人到目标的情感组成。总共，该任务收到了100多份提交。参与者主要使用零射，几乎没有射击和微调格式的大型语言模型。测试集的最佳结果是通过大型语言模型进行微调而获得的。我们还比较了30个提示和11个开源语言模型，其中1次和10次设置中的参数为3-32亿个，并找到了最佳模型和提示。

Title: Towards LLMs Robustness to Changes in Prompt Format Styles

• Authors: Lilian Ngweta, Kiran Kate, Jason Tsay, Yara Rizk
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.06969
• Pdf URL: https://arxiv.org/pdf/2504.06969
• Copy Paste: [[2504.06969]] Towards LLMs Robustness to Changes in Prompt Format Styles(https://arxiv.org/abs/2504.06969)
• Keywords: language model, llm, prompt
• Abstract: Large language models (LLMs) have gained popularity in recent years for their utility in various applications. However, they are sensitive to non-semantic changes in prompt formats, where small changes in the prompt format can lead to significant performance fluctuations. In the literature, this problem is commonly referred to as prompt brittleness. Previous research on prompt engineering has focused mainly on developing techniques for identifying the optimal prompt for specific tasks. Some studies have also explored the issue of prompt brittleness and proposed methods to quantify performance variations; however, no simple solution has been found to address this challenge. We propose Mixture of Formats (MOF), a simple and efficient technique for addressing prompt brittleness in LLMs by diversifying the styles used in the prompt few-shot examples. MOF was inspired by computer vision techniques that utilize diverse style datasets to prevent models from associating specific styles with the target variable. Empirical results show that our proposed technique reduces style-induced prompt brittleness in various LLMs while also enhancing overall performance across prompt variations and different datasets.
• 摘要：近年来，大型语言模型（LLMS）在各种应用中的实用性都广受欢迎。但是，它们对及时格式的非语义变化很敏感，在这种格式中，及时格式的小变化会导致性能显着波动。在文献中，这个问题通常被称为及时的脆性。先前对及时工程的研究主要集中在开发用于确定特定任务最佳提示的技术。一些研究还探讨了迅速勃起性和量化性能变化的方法的问题。但是，没有发现简单的解决方案来应对这一挑战。我们提出了格式（MOF）的混合物，这是一种简单有效的技术，可通过在迅速几乎没有示例中使用的样式多样化来解决LLMS中的迅速爆发。 MOF的灵感来自计算机视觉技术，该技术利用各种样式数据集来防止模型将特定样式与目标变量相关联。经验结果表明，我们提出的技术可降低样式引起的各种LLM中的迅速昏昏欲睡，同时还提高了迅速变化和不同数据集的整体性能。

Title: Evaluating Retrieval Augmented Generative Models for Document Queries in Transportation Safety

• Authors: Chad Melton, Alex Sorokine, Steve Peterson
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.07022
• Pdf URL: https://arxiv.org/pdf/2504.07022
• Copy Paste: [[2504.07022]] Evaluating Retrieval Augmented Generative Models for Document Queries in Transportation Safety(https://arxiv.org/abs/2504.07022)
• Keywords: language model, gpt, llm, chat, retrieval augmented generation
• Abstract: Applications of generative Large Language Models LLMs are rapidly expanding across various domains, promising significant improvements in workflow efficiency and information retrieval. However, their implementation in specialized, high-stakes domains such as hazardous materials transportation is challenging due to accuracy and reliability concerns. This study evaluates the performance of three fine-tuned generative models, ChatGPT, Google's Vertex AI, and ORNL Retrieval Augmented Generation augmented LLaMA 2 and LLaMA in retrieving regulatory information essential for hazardous material transportation compliance in the United States. Utilizing approximately 40 publicly available federal and state regulatory documents, we developed 100 realistic queries relevant to route planning and permitting requirements. Responses were qualitatively rated based on accuracy, detail, and relevance, complemented by quantitative assessments of semantic similarity between model outputs. Results demonstrated that the RAG-augmented LLaMA models significantly outperformed Vertex AI and ChatGPT, providing more detailed and generally accurate information, despite occasional inconsistencies. This research introduces the first known application of RAG in transportation safety, emphasizing the need for domain-specific fine-tuning and rigorous evaluation methodologies to ensure reliability and minimize the risk of inaccuracies in high-stakes environments.
• 摘要：生成大语言模型LLM的应用正在跨各个领域迅速扩展，有望在工作流程效率和信息检索方面得到显着提高。但是，由于准确性和可靠性问题，它们在专门的高风险域（例如危险材料运输）等高风险领域的实施是具有挑战性的。这项研究评估了三种微调生成模型的性能，Chatgpt，Google的Vertex AI和ORNL检索增强产生增强的增强的Llama 2和Llama，以检索美国在美国有害物质运输合规性至关重要的监管信息。利用大约40个公开可用的联邦和州监管文件，我们开发了100个与路线计划和许可要求相关的现实查询。响应是根据准确性，细节和相关性定性评估的，并通过对模型输出之间语义相似性的定量评估进行了补充。结果表明，尽管偶尔出现不一致之处，但抹布式的Llama模型显着超过了顶点AI和CHATGPT，提供了更详细且总体上准确的信息。这项研究介绍了RAG在运输安全中的第一个已知应用，强调了对特定领域的微调和严格的评估方法的需求，以确保可靠性并最大程度地减少高风险环境中不准确的风险。

Title: TASTE: Text-Aligned Speech Tokenization and Embedding for Spoken Language Modeling

• Authors: Liang-Hsuan Tseng, Yi-Chang Chen, Kuan-Yi Lee, Da-Shan Shiu, Hung-yi Lee
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2504.07053
• Pdf URL: https://arxiv.org/pdf/2504.07053
• Copy Paste: [[2504.07053]] TASTE: Text-Aligned Speech Tokenization and Embedding for Spoken Language Modeling(https://arxiv.org/abs/2504.07053)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) excel in text-based natural language processing tasks but remain constrained by their reliance on textual inputs and outputs. To enable more natural human-LLM interaction, recent progress have focused on deriving a spoken language model (SLM) that can not only listen but also generate speech. To achieve this, a promising direction is to conduct speech-text joint modeling. However, recent SLM still lag behind text LLM due to the modality mismatch. One significant mismatch can be the sequence lengths between speech and text tokens. To address this, we introduce Text-Aligned Speech Tokenization and Embedding (TASTE), a method that directly addresses the modality gap by aligning speech token with the corresponding text transcription during the tokenization stage. We propose a method that can achieve this through the special aggregation mechanism and with speech reconstruction as the training objective. We conduct extensive experiments and show that TASTE can preserve essential paralinguistic information while dramatically reducing the token sequence length. Furthermore, by leveraging TASTE, we can adapt text-based LLMs into effective SLMs with parameter-efficient fine-tuning techniques such as Low-Rank Adaptation (LoRA). Experimental results on benchmark tasks, including SALMON and StoryCloze, demonstrate that TASTE-based SLMs perform similarly to previous full-finetuning methods. To our knowledge, TASTE is the first end-to-end approach that utilizes a reconstruction objective to automatically learn a text-aligned speech tokenization and embedding suitable for spoken language modeling. Our demo, code, and models are publicly available at this https URL.
• 摘要：大型语言模型（LLMS）在基于文本的自然语言处理任务中表现出色，但仍受到其对文本输入和输出的依赖的约束。为了启用更自然的人类互动，最近的进步集中在得出口语模型（SLM）不仅可以聆听，而且可以产生语音。为此，一个有希望的方向是进行语音文本联合建模。但是，由于模式不匹配，最近的SLM仍然落后于文本LLM。一个重要的不匹配可能是语音和文本令牌之间的序列长度。为了解决这个问题，我们介绍了与文本一致的语音令牌化和嵌入（口味），该方法通过将语音令牌与在令牌化阶段期间的相应文本转录保持一致，从而直接解决模式差距。我们提出了一种可以通过特殊的聚合机制和语音重建作为培训目标来实现这一目标的方法。我们进行了广泛的实验，并表明味道可以保留基本的副语言信息，同时大大降低令牌序列的长度。此外，通过利用口味，我们可以将基于文本的LLMS调整为具有参数有效的微调技术，例如低级别适应性（LORA）。基准任务的实验结果，包括鲑鱼和故事浮动，表明基于味觉的SLM的性能与以前的全面调节方法相似。据我们所知，品味是使用重建目标的第一种端到端方法，可以自动学习与文本一致的语音令牌化和嵌入适合口语建模的嵌入。我们的演示，代码和模型在此HTTPS URL上公开可用。

Title: HalluciNot: Hallucination Detection Through Context and Common Knowledge Verification

• Authors: Bibek Paudel, Alexander Lyzhov, Preetam Joshi, Puneet Anand
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.07069
• Pdf URL: https://arxiv.org/pdf/2504.07069
• Copy Paste: [[2504.07069]] HalluciNot: Hallucination Detection Through Context and Common Knowledge Verification(https://arxiv.org/abs/2504.07069)
• Keywords: language model, llm, hallucination
• Abstract: This paper introduces a comprehensive system for detecting hallucinations in large language model (LLM) outputs in enterprise settings. We present a novel taxonomy of LLM responses specific to hallucination in enterprise applications, categorizing them into context-based, common knowledge, enterprise-specific, and innocuous statements. Our hallucination detection model HDM-2 validates LLM responses with respect to both context and generally known facts (common knowledge). It provides both hallucination scores and word-level annotations, enabling precise identification of problematic content. To evaluate it on context-based and common-knowledge hallucinations, we introduce a new dataset HDMBench. Experimental results demonstrate that HDM-2 out-performs existing approaches across RagTruth, TruthfulQA, and HDMBench datasets. This work addresses the specific challenges of enterprise deployment, including computational efficiency, domain specialization, and fine-grained error identification. Our evaluation dataset, model weights, and inference code are publicly available.
• 摘要：本文介绍了一个综合系统，用于在企业设置中检测大语言模型（LLM）输出的幻觉。我们提出了针对企业应用中特定幻觉的LLM响应的新颖分类法，将其分类为基于上下文的，常识，特定于企业的陈述和无害的陈述。我们的幻觉检测模型HDM-2验证了相对于上下文和普遍已知事实（常识）的LLM响应。它提供幻觉分数和单词级注释，从而可以精确地识别有问题的内容。为了对基于上下文和共同知识幻觉进行评估，我们介绍了一个新的数据集HDMBench。实验结果表明，HDM-2在Ragtruth，Elthfulqa和HDMBench数据集中的现有方法超过了现有的方法。这项工作解决了企业部署的具体挑战，包括计算效率，域专业化和细粒度错误识别。我们的评估数据集，模型权重和推理代码公开可用。

Title: A Survey on Personalized and Pluralistic Preference Alignment in Large Language Models

• Authors: Zhouhang Xie, Junda Wu, Yiran Shen, Yu Xia, Xintong Li, Aaron Chang, Ryan Rossi, Sachin Kumar, Bodhisattwa Prasad Majumder, Jingbo Shang, Prithviraj Ammanabrolu, Julian McAuley
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.07070
• Pdf URL: https://arxiv.org/pdf/2504.07070
• Copy Paste: [[2504.07070]] A Survey on Personalized and Pluralistic Preference Alignment in Large Language Models(https://arxiv.org/abs/2504.07070)
• Keywords: language model, llm
• Abstract: Personalized preference alignment for large language models (LLMs), the process of tailoring LLMs to individual users' preferences, is an emerging research direction spanning the area of NLP and personalization. In this survey, we present an analysis of works on personalized alignment and modeling for LLMs. We introduce a taxonomy of preference alignment techniques, including training time, inference time, and additionally, user-modeling based methods. We provide analysis and discussion on the strengths and limitations of each group of techniques and then cover evaluation, benchmarks, as well as open problems in the field.
• 摘要：大型语言模型（LLMS）的个性化偏好一致性是将LLMS量身定制为单个用户偏好的过程，是跨越NLP和个性化领域的新兴研究方向。在这项调查中，我们介绍了有关LLMS的个性化对齐和建模作品的分析。我们介绍了偏好对准技术的分类法，包括培训时间，推理时间以及基于用户模型的方法。我们提供有关每组技术的优势和局限性的分析和讨论，然后涵盖评估，基准以及该领域的开放问题。

Title: Kaleidoscope: In-language Exams for Massively Multilingual Vision Evaluation

• Authors: Israfel Salazar, Manuel Fernández Burda, Shayekh Bin Islam, Arshia Soltani Moakhar, Shivalika Singh, Fabian Farestam, Angelika Romanou, Danylo Boiko, Dipika Khullar, Mike Zhang, Dominik Krzemiński, Jekaterina Novikova, Luísa Shimabucoro, Joseph Marvin Imperial, Rishabh Maheshwary, Sharad Duwal, Alfonso Amayuelas, Swati Rajwal, Jebish Purbey, Ahmed Ruby, Nicholas Popovič, Marek Suppa, Azmine Toushik Wasi, Ram Mohan Rao Kadiyala, Olga Tsymboi, Maksim Kostritsya, Bardia Soltani Moakhar, Gabriel da Costa Merlin, Otávio Ferracioli Coletti, Maral Jabbari Shiviari, MohammadAmin farahani fard, Silvia Fernandez, María Grandury, Dmitry Abulkhanov, Drishti Sharma, Andre Guarnier De Mitri, Leticia Bossatto Marchezi, Johan Obando-Ceron, Nazar Kohut, Beyza Ermis, Desmond Elliott, Enzo Ferrante, Sara Hooker, Marzieh Fadaee
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2504.07072
• Pdf URL: https://arxiv.org/pdf/2504.07072
• Copy Paste: [[2504.07072]] Kaleidoscope: In-language Exams for Massively Multilingual Vision Evaluation(https://arxiv.org/abs/2504.07072)
• Keywords: language model
• Abstract: The evaluation of vision-language models (VLMs) has mainly relied on English-language benchmarks, leaving significant gaps in both multilingual and multicultural coverage. While multilingual benchmarks have expanded, both in size and languages, many rely on translations of English datasets, failing to capture cultural nuances. In this work, we propose Kaleidoscope, as the most comprehensive exam benchmark to date for the multilingual evaluation of vision-language models. Kaleidoscope is a large-scale, in-language multimodal benchmark designed to evaluate VLMs across diverse languages and visual inputs. Kaleidoscope covers 18 languages and 14 different subjects, amounting to a total of 20,911 multiple-choice questions. Built through an open science collaboration with a diverse group of researchers worldwide, Kaleidoscope ensures linguistic and cultural authenticity. We evaluate top-performing multilingual vision-language models and find that they perform poorly on low-resource languages and in complex multimodal scenarios. Our results highlight the need for progress on culturally inclusive multimodal evaluation frameworks.
• 摘要：视觉模型（VLM）的评估主要取决于英语基准，在多语言和多元文化覆盖范围内留下了很大的差距。尽管多语言基准的规模和语言都扩大了，但许多人都依赖英语数据集的翻译，无法捕捉文化差异。在这项工作中，我们建议万花筒作为视觉模型的多语言评估迄今为止最全面的考试基准。万花筒是一种大规模的语言多模式基准，旨在评估跨不同语言和视觉输入的VLM。万花筒涵盖18种语言和14位不同的主题，总计为20,911个多项选择问题。万花筒与全球各种研究人员的开放科学合作建立，可确保语言和文化真实性。我们评估了表现最佳的多语言视觉语言模型，并发现它们在低资源语言和复杂的多模式场景中的表现差。我们的结果强调了在文化包含多模式评估框架上取得进展的必要性。

Title: DeduCE: Deductive Consistency as a Framework to Evaluate LLM Reasoning

• Authors: Atharva Pandey, Kshitij Dubey, Rahul Sharma, Amit Sharma
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2504.07080
• Pdf URL: https://arxiv.org/pdf/2504.07080
• Copy Paste: [[2504.07080]] DeduCE: Deductive Consistency as a Framework to Evaluate LLM Reasoning(https://arxiv.org/abs/2504.07080)
• Keywords: language model, llm, chain-of-thought
• Abstract: Despite great performance on Olympiad-level reasoning problems, frontier large language models can still struggle on high school math when presented with novel problems outside standard benchmarks. Going beyond final accuracy, we propose a deductive consistency metric to analyze chain-of-thought output from language models (LMs).Formally, deductive reasoning involves two subtasks: understanding a set of input premises and inferring the conclusions that follow from them. The proposed metric studies LMs' performance on these subtasks, with the goal of explaining LMs' reasoning errors on novel problems: how well do LMs understand input premises with increasing context lengths, and how well can they infer conclusions over multiple reasoning hops? Since existing benchmarks may be memorized, we develop a pipeline to evaluate LMs' deductive consistency on novel, perturbed versions of benchmark problems. On novel grade school math problems (GSM-8k), we find that LMs are fairly robust to increasing number of input premises, but suffer significant accuracy decay as the number of reasoning hops is increased. Interestingly, these errors are masked in the original benchmark as all models achieve near 100% accuracy. As we increase the number of solution steps using a synthetic dataset, prediction over multiple hops still remains the major source of error compared to understanding input premises. Other factors, such as shifts in language style or natural propagation of early errors do not explain the trends. Our analysis provides a new view to characterize LM reasoning -- as computations over a window of input premises and reasoning hops -- that can provide unified evaluation across problem domains.
• 摘要：尽管在奥林匹克级的推理问题上表现出色，但在出现标准基准之外的新型问题时，Frontier大型语言模型仍然在高中数学上挣扎。除了最终的准确性之外，我们提出了一个演绎一致性指标，以分析语言模型（LMS）的经过思考链输出。从表面上讲，演绎推理涉及两个子任务：了解一组输入前提，并从中推断出随后的结论。拟议的度量研究LMS在这些子任务上的性能，目的是解释LMS在新问题上的推理错误：LMS对上下文长度的了解如何理解输入前提，以及如何在多个推理啤酒花上推断结论如何？由于可能会记住现有的基准测试，因此我们开发了一条管道来评估LMS对基准问题的新颖，扰动版本的演绎一致性。在新的小学数学问题（GSM-8K）中，我们发现LMS对于增加的输入前提数量相当强大，但是随着推理啤酒花的数量的增加，遭受了明显的准确性衰减。有趣的是，这些错误在原始基准测试中被掩盖，因为所有模型都达到了100％的精度。随着我们使用合成数据集增加了解决方案步骤的数量，与理解输入前提相比，对多个啤酒花的预测仍然是错误的误差来源。其他因素，例如语言风格的转变或早期错误的自然传播不能解释趋势。我们的分析提供了一种新的观点来表征LM推理 - 作为输入前提和推理啤酒花窗口的计算，可以在问题域中提供统一的评估。

Title: Self-Steering Language Models

• Authors: Gabriel Grand, Joshua B. Tenenbaum, Vikash K. Mansinghka, Alexander K. Lew, Jacob Andreas
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.07081
• Pdf URL: https://arxiv.org/pdf/2504.07081
• Copy Paste: [[2504.07081]] Self-Steering Language Models(https://arxiv.org/abs/2504.07081)
• Keywords: language model, gpt
• Abstract: While test-time reasoning enables language models to tackle complex tasks, searching or planning in natural language can be slow, costly, and error-prone. But even when LMs struggle to emulate the precise reasoning steps needed to solve a problem, they often excel at describing its abstract structure--both how to verify solutions and how to search for them. This paper introduces DisCIPL, a method for "self-steering" LMs where a Planner model generates a task-specific inference program that is executed by a population of Follower models. Our approach equips LMs with the ability to write recursive search procedures that guide LM inference, enabling new forms of verifiable and efficient reasoning. When instantiated with a small Follower (e.g., Llama-3.2-1B), DisCIPL matches (and sometimes outperforms) much larger models, including GPT-4o and o1, on challenging constrained generation tasks. In decoupling planning from execution, our work opens up a design space of highly-parallelized Monte Carlo inference strategies that outperform standard best-of-N sampling, require no finetuning, and can be implemented automatically by existing LMs.
• 摘要：虽然测试时间推理使语言模型能够处理复杂的任务，但自然语言的搜索或计划可能会缓慢，昂贵且容易出错。但是，即使LMS难以模仿解决问题所需的精确推理步骤，他们也经常在描述其抽象结构（都如何验证解决方案以及如何搜索它们）方面表现出色。本文介绍了“自我传动” LMS的方法，其中计划者模型生成了特定于任务的推理程序，该计划由追随者模型群体执行。我们的方法使LMS具有编写指导LM推理的递归搜索程序的能力，从而实现了可验证和有效的推理的新形式。当使用小型追随者（例如Llama-3.2-1b）实例化时，符号匹配（有时甚至超过了）大于较大的模型，包括GPT-4O和O1，在具有挑战性的约束生成任务上匹配。在将执行的脱钩计划中，我们的工作打开了高度平行的蒙特卡洛推理策略的设计空间，该策略的表现要超过标准的最佳N采样，不需要填充，并且可以通过现有LMS自动实施。

Title: KG-LLM-Bench: A Scalable Benchmark for Evaluating LLM Reasoning on Textualized Knowledge Graphs

• Authors: Elan Markowitz, Krupa Galiya, Greg Ver Steeg, Aram Galstyan
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2504.07087
• Pdf URL: https://arxiv.org/pdf/2504.07087
• Copy Paste: [[2504.07087]] KG-LLM-Bench: A Scalable Benchmark for Evaluating LLM Reasoning on Textualized Knowledge Graphs(https://arxiv.org/abs/2504.07087)
• Keywords: language model, llm
• Abstract: Knowledge graphs have emerged as a popular method for injecting up-to-date, factual knowledge into large language models (LLMs). This is typically achieved by converting the knowledge graph into text that the LLM can process in context. While multiple methods of encoding knowledge graphs have been proposed, the impact of this textualization process on LLM performance remains under-explored. We introduce KG-LLM-Bench, a comprehensive and extensible benchmark spanning five knowledge graph understanding tasks, and evaluate how different encoding strategies affect performance across various base models. Our extensive experiments with seven language models and five textualization strategies provide insights for optimizing LLM performance on KG reasoning tasks.
• 摘要：知识图已成为一种流行的方法，用于将最新的事实知识注入大型语言模型（LLMS）。这通常是通过将知识图转换为LLM可以在上下文中处理的文本来实现的。尽管已经提出了多种编码知识图的方法，但此文本化过程对LLM性能的影响仍然不足。我们介绍了KG-LLM基础，这是一个跨越五个知识理解任务的全面且可扩展的基准测试，并评估不同的编码策略如何影响各种基本模型的性能。我们对七个语言模型和五种文本化策略进行了广泛的实验，为优化KG推理任务的LLM性能提供了见解。

Title: OLMoTrace: Tracing Language Model Outputs Back to Trillions of Training Tokens

• Authors: Jiacheng Liu, Taylor Blanton, Yanai Elazar, Sewon Min, YenSung Chen, Arnavi Chheda-Kothary, Huy Tran, Byron Bischoff, Eric Marsh, Michael Schmitz, Cassidy Trier, Aaron Sarnat, Jenna James, Jon Borchardt, Bailey Kuehl, Evie Cheng, Karen Farley, Sruthi Sreeram, Taira Anderson, David Albright, Carissa Schoenick, Luca Soldaini, Dirk Groeneveld, Rock Yuren Pang, Pang Wei Koh, Noah A. Smith, Sophie Lebrecht, Yejin Choi, Hannaneh Hajishirzi, Ali Farhadi, Jesse Dodge
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2504.07096
• Pdf URL: https://arxiv.org/pdf/2504.07096
• Copy Paste: [[2504.07096]] OLMoTrace: Tracing Language Model Outputs Back to Trillions of Training Tokens(https://arxiv.org/abs/2504.07096)
• Keywords: language model, hallucination
• Abstract: We present OLMoTrace, the first system that traces the outputs of language models back to their full, multi-trillion-token training data in real time. OLMoTrace finds and shows verbatim matches between segments of language model output and documents in the training text corpora. Powered by an extended version of infini-gram (Liu et al., 2024), our system returns tracing results within a few seconds. OLMoTrace can help users understand the behavior of language models through the lens of their training data. We showcase how it can be used to explore fact checking, hallucination, and the creativity of language models. OLMoTrace is publicly available and fully open-source.
• 摘要：我们提出Olmotrace，这是第一个将语言模型输出追溯到实时的全千万训练数据的系统。 Olmotrace在语言模型输出和培训文本语料库中的文档中找到并显示了逐字匹配。在Infini-gram的扩展版本（Liu等，2024）的支持下，我们的系统在几秒钟内返回跟踪结果。 Olmotrace可以通过其培训数据的镜头来帮助用户了解语言模型的行为。我们展示了如何使用它来探索事实检查，幻觉和语言模型的创造力。 Olmotrace已公开可用，并且完全开源。