2025-07-18

Title: Modeling Open-World Cognition as On-Demand Synthesis of Probabilistic Models

• Authors: Lionel Wong, Katherine M. Collins, Lance Ying, Cedegao E. Zhang, Adrian Weller, Tobias Gersternberg, Timothy O'Donnell, Alexander K. Lew, Jacob D. Andreas, Joshua B. Tenenbaum, Tyler Brooke-Wilson
• Subjects: cs.CL, cs.AI, cs.PL
• Abstract URL: https://arxiv.org/abs/2507.12547
• Pdf URL: https://arxiv.org/pdf/2507.12547
• Copy Paste: [[2507.12547]] Modeling Open-World Cognition as On-Demand Synthesis of Probabilistic Models(https://arxiv.org/abs/2507.12547)
• Keywords: language model, chain-of-thought
• Abstract: When faced with novel situations, people are able to marshal relevant considerations from a wide range of background knowledge and put these to use in inferences and predictions. What permits us to draw in globally relevant information and reason over it coherently? Here, we explore the hypothesis that people use a combination of distributed and symbolic representations to construct bespoke mental models tailored to novel situations. We propose a computational implementation of this idea -- a ``Model Synthesis Architecture'' (MSA) -- using language models to implement global relevance-based retrieval and model synthesis and probabilistic programs to implement bespoke, coherent world models. We evaluate our MSA as a model of human judgments on a novel reasoning dataset. The dataset -- built around a `Model Olympics` domain of sports vignettes -- tests models' capacity for human-like, open-ended reasoning by requiring (i) judgments about novel causal structures described in language; (ii) drawing on large bodies of background knowledge; and (iii) doing both in light of observations that introduce arbitrary novel variables. Our MSA approach captures human judgments better than language model-only baselines, under both direct and chain-of-thought generations from the LM that supports model synthesis. These results suggest that MSAs can be implemented in a way that mirrors people's ability to deliver locally coherent reasoning over globally relevant variables, offering a path to understanding and replicating human reasoning in open-ended domains.
• 摘要：当面对新的情况时，人们就可以从广泛的背景知识中考虑相关的考虑，并将其用于推论和预测。是什么使我们能够连贯地汲取全球相关的信息和理由？在这里，我们探讨了人们使用分布式和符号表示的组合来构建针对新型情况的定制心理模型的假设。我们建议使用语言模型来实现基于全球相关性的检索和模型综合和概率程序来实现定制，连贯的世界模型来实现该想法的计算实现 - ``模型综合体系结构''（MSA）（MSA）。我们将MSA评估为新型推理数据集中人类判断的模型。该数据集 - 围绕运动小插图的“模型奥运会”领域建立 - 通过（i）（i）对语言中描述的新因果结构进行判断，以测试模型的人类般的，开放式推理的能力； （ii）借鉴大型背景知识； （iii）鉴于引入任意新型变量的观察结果。我们的MSA方法在支持模型综合的LM的直接和经营链中，比仅限语言模型的基准更好地捕捉人类的判断。这些结果表明，可以以反映人们在全球相关变量上提供本地连贯推理的能力的方式实施MSA，从而提供理解和复制开放式域中人类推理的途径。

Title: Is This Just Fantasy? Language Model Representations Reflect Human Judgments of Event Plausibility

• Authors: Michael A. Lepori, Jennifer Hu, Ishita Dasgupta, Roma Patel, Thomas Serre, Ellie Pavlick
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2507.12553
• Pdf URL: https://arxiv.org/pdf/2507.12553
• Copy Paste: [[2507.12553]] Is This Just Fantasy? Language Model Representations Reflect Human Judgments of Event Plausibility(https://arxiv.org/abs/2507.12553)
• Keywords: language model
• Abstract: Language models (LMs) are used for a diverse range of tasks, from question answering to writing fantastical stories. In order to reliably accomplish these tasks, LMs must be able to discern the modal category of a sentence (i.e., whether it describes something that is possible, impossible, completely nonsensical, etc.). However, recent studies have called into question the ability of LMs to categorize sentences according to modality (Michaelov et al., 2025; Kauf et al., 2023). In this work, we identify linear representations that discriminate between modal categories within a variety of LMs, or modal difference vectors. Analysis of modal difference vectors reveals that LMs have access to more reliable modal categorization judgments than previously reported. Furthermore, we find that modal difference vectors emerge in a consistent order as models become more competent (i.e., through training steps, layers, and parameter count). Notably, we find that modal difference vectors identified within LM activations can be used to model fine-grained human categorization behavior. This potentially provides a novel view into how human participants distinguish between modal categories, which we explore by correlating projections along modal difference vectors with human participants' ratings of interpretable features. In summary, we derive new insights into LM modal categorization using techniques from mechanistic interpretability, with the potential to inform our understanding of modal categorization in humans.
• 摘要：语言模型（LMS）用于各种各样的任务，从回答问题到编写奇幻故事。为了可靠地完成这些任务，LMS必须能够辨别句子的模态类别（即，它是否描述了可能的东西，不可能，完全是荒谬的等）。但是，最近的研究质疑LMS根据模态对句子进行分类的能力（Michaelov等，2025； Kauf等，2023）。在这项工作中，我们确定线性表示，这些表示在各种LMS或模态差异向量中区分模态类别。模态差异向量的分析表明，LMS可以访问比以前报道的更可靠的模态分类判断。此外，随着模型变得更有能力（即通过训练步骤，层和参数计数），我们发现模态差异向量以一致的顺序出现。值得注意的是，我们发现在LM激活中鉴定的模态差异向量可用于建模细粒的人类分类行为。这可能提供了一种新的观点，即人类参与者如何区分模态类别，我们通过将模态差异向量的投影与人类参与者的可解释特征的评分相关联来探讨。总而言之，我们使用机械解释性的技术得出了对LM模态分类的新见解，并有可能告知我们对人类模态分类的理解。

Title: The first open machine translation system for the Chechen language

• Authors: Abu-Viskhan A. Umishov, Vladislav A. Grigorian
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2507.12672
• Pdf URL: https://arxiv.org/pdf/2507.12672
• Copy Paste: [[2507.12672]] The first open machine translation system for the Chechen language(https://arxiv.org/abs/2507.12672)
• Keywords: language model
• Abstract: We introduce the first open-source model for translation between the vulnerable Chechen language and Russian, and the dataset collected to train and evaluate it. We explore fine-tuning capabilities for including a new language into a large language model system for multilingual translation NLLB-200. The BLEU / ChrF++ scores for our model are 8.34 / 34.69 and 20.89 / 44.55 for translation from Russian to Chechen and reverse direction, respectively. The release of the translation models is accompanied by the distribution of parallel words, phrases and sentences corpora and multilingual sentence encoder adapted to the Chechen language.
• 摘要：我们介绍了第一个开源模型，用于易受伤害的车臣语言与俄语之间的翻译，以及收集的数据集用于培训和评估它。我们探索了微调功能，以将新语言包括在大型语言模型系统中，用于多语言翻译NLLB-200。我们的模型的BLEU / CHRF ++得分分别为8.34 / 34.69和20.89 / 44.55，用于从俄罗斯到车臣和反向方向的翻译。翻译模型的发布伴随着平行单词，短语和句子语料库的分布以及适合车臣语言的多语言句子编码器。

Title: Improving Drug Identification in Overdose Death Surveillance using Large Language Models

• Authors: Arthur J. Funnell, Panayiotis Petousis, Fabrice Harel-Canada, Ruby Romero, Alex A. T. Bui, Adam Koncsol, Hritika Chaturvedi, Chelsea Shover, David Goodman-Meza
• Subjects: cs.CL, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2507.12679
• Pdf URL: https://arxiv.org/pdf/2507.12679
• Copy Paste: [[2507.12679]] Improving Drug Identification in Overdose Death Surveillance using Large Language Models(https://arxiv.org/abs/2507.12679)
• Keywords: language model
• Abstract: The rising rate of drug-related deaths in the United States, largely driven by fentanyl, requires timely and accurate surveillance. However, critical overdose data are often buried in free-text coroner reports, leading to delays and information loss when coded into ICD (International Classification of Disease)-10 classifications. Natural language processing (NLP) models may automate and enhance overdose surveillance, but prior applications have been limited. A dataset of 35,433 death records from multiple U.S. jurisdictions in 2020 was used for model training and internal testing. External validation was conducted using a novel separate dataset of 3,335 records from 2023-2024. Multiple NLP approaches were evaluated for classifying specific drug involvement from unstructured death certificate text. These included traditional single- and multi-label classifiers, as well as fine-tuned encoder-only language models such as Bidirectional Encoder Representations from Transformers (BERT) and BioClinicalBERT, and contemporary decoder-only large language models such as Qwen 3 and Llama 3. Model performance was assessed using macro-averaged F1 scores, and 95% confidence intervals were calculated to quantify uncertainty. Fine-tuned BioClinicalBERT models achieved near-perfect performance, with macro F1 scores >=0.998 on the internal test set. External validation confirmed robustness (macro F1=0.966), outperforming conventional machine learning, general-domain BERT models, and various decoder-only large language models. NLP models, particularly fine-tuned clinical variants like BioClinicalBERT, offer a highly accurate and scalable solution for overdose death classification from free-text reports. These methods can significantly accelerate surveillance workflows, overcoming the limitations of manual ICD-10 coding and supporting near real-time detection of emerging substance use trends.
• 摘要：美国与芬太尼驱动的与药物有关的死亡率的上升需要及时，准确的监视。但是，关键的过量数据通常被埋葬在自由文本死因裁判官报告中，导致编码为ICD（国际疾病分类）-10分类时延迟和信息丢失。自然语言处理（NLP）模型可以自动化并增强过量监视，但事先应用是有限的。 2020年，来自美国多个司法管辖区的35,433条死亡记录的数据集用于模型培训和内部测试。使用2023 - 2024年的3,335条记录的新型单独数据集进行了外部验证。评估了多种NLP方法，以对非结构化死亡证书文本中的特定药物参与进行分类。 These included traditional single- and multi-label classifiers, as well as fine-tuned encoder-only language models such as Bidirectional Encoder Representations from Transformers (BERT) and BioClinicalBERT, and contemporary decoder-only large language models such as Qwen 3 and Llama 3. Model performance was assessed using macro-averaged F1 scores, and 95% confidence intervals were calculated to quantify uncertainty.微调的生物胶质植物模型达到了几乎完美的性能，内部测试集的宏F1得分> = 0.998。外部验证确认的鲁棒性（宏F1 = 0.966），表现优于常规机器学习，通用域BERT模型和各种仅解码器的大型语言模型。 NLP模型，尤其是BioClinicalbert等微调临床变体，为自由文本报告提供了高度准确且可扩展的解决方案。这些方法可以显着加速监视工作流程，克服手动ICD-10编码的局限性，并支持几乎实时检测新兴物质使用趋势。

Title: AudioJudge: Understanding What Works in Large Audio Model Based Speech Evaluation

• Authors: Potsawee Manakul, Woody Haosheng Gan, Michael J. Ryan, Ali Sartaz Khan, Warit Sirichotedumrong, Kunat Pipatanakul, William Held, Diyi Yang
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2507.12705
• Pdf URL: https://arxiv.org/pdf/2507.12705
• Copy Paste: [[2507.12705]] AudioJudge: Understanding What Works in Large Audio Model Based Speech Evaluation(https://arxiv.org/abs/2507.12705)
• Keywords: prompt
• Abstract: Current speech evaluation suffers from two critical limitations: the need and difficulty of designing specialized systems targeting individual audio characteristics, and poor correlation between automatic evaluation methods and human preferences. This work presents a systematic study of Large Audio Model (LAM) as a Judge, AudioJudge, investigating whether it can provide a unified evaluation framework that addresses both challenges. We systematically explore AudioJudge across audio characteristic detection tasks, including pronunciation, speaking rate, speaker identification and speech quality, and system-level human preference simulation for automated benchmarking. We investigate different prompt engineering strategies, finding that audio concatenation combined with in-context learning significantly improves performance across both audio characteristic detection and human preference simulation tasks. We further introduce a multi-aspect ensemble AudioJudge to enable general-purpose multi-aspect audio evaluation. This method decomposes speech assessment into specialized judges for lexical content, speech quality, and paralinguistic features, achieving up to 0.91 Spearman correlation with human preferences on our system ranking benchmark. Robustness analysis reveals that while LAMs maintain strong performance under acoustic noise, they exhibit significant verbosity and positional biases that require careful mitigation.
• 摘要：当前的语音评估受到了两个关键局限性：设计针对单个音频特征的专业系统的需求和困难，以及自动评估方法和人类偏好之间的相关性差。这项工作介绍了大型音频模型（LAM）的系统研究，作为法官AudioGudge，研究是否可以提供解决这两个挑战的统一评估框架。我们系统地探索了跨音频特征检测任务的音频判断，包括发音，口语率，说话者识别和语音质量以及用于自动基准测试的系统级人类偏好模拟。我们研究了不同的及时工程策略，发现音频串联与文化中的学习相结合可显着提高音频特征检测和人类偏好模拟任务的性能。我们进一步介绍了一个多光的合奏音频判断，以实现通用多种观察音频评估。该方法将语音评估分解为词汇内容，语音质量和副语言特征的专业法官，在我们的系统排名基准中，与人类偏好的相关性高达0.91。鲁棒性分析表明，尽管LAM在声学噪声下保持了强烈的性能，但它们表现出明显的详细性和需要仔细缓解的位置偏见。

Title: FLEXITOKENS: Flexible Tokenization for Evolving Language Models

• Authors: Abraham Toluase Owodunni, Orevaoghene Ahia, Sachin Kumar
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2507.12720
• Pdf URL: https://arxiv.org/pdf/2507.12720
• Copy Paste: [[2507.12720]] FLEXITOKENS: Flexible Tokenization for Evolving Language Models(https://arxiv.org/abs/2507.12720)
• Keywords: language model
• Abstract: Language models (LMs) are challenging to adapt to new data distributions by simple finetuning. This is due to the rigidity of their subword tokenizers, which typically remain unchanged during adaptation. This inflexibility often leads to inefficient tokenization, causing overfragmentation of out-of-distribution domains, unseen languages, or scripts. In this work, we develop byte-level LMs with learnable tokenizers to make tokenization adaptive. Our models include a submodule that learns to predict boundaries between the input byte sequence, encoding it into variable-length segments. Existing tokenizer-free methods train this boundary predictor using an auxiliary loss that enforces a fixed compression rate across the training corpus, introducing a new kind of rigidity. We propose FLEXITOKENS, a simplified training objective that enables significantly greater flexibility during adaptation. Evaluating across multiple multilingual benchmarks, morphologically diverse tasks, and domains, we demonstrate that FLEXITOKENS consistently reduces token over-fragmentation and achieves up to 10\% improvements on downstream task performance compared to subword and other gradient-based tokenizers. Code and data for our experiments will be released at this https URL
• 摘要：语言模型（LMS）具有挑战性，可以通过简单的填充来适应新的数据分布。这是由于其子词引物的刚度，通常在适应过程中保持不变。这种僵化的性通常会导致效率低下的令牌化，从而导致分布外域，看不见的语言或脚本过度散布。在这项工作中，我们开发了具有可学习的引物器的字节级LM，以使令牌化适应性。我们的模型包括一个学会的子模块，该子模块学会预测输入字节序列之间的边界，并将其编码为可变长度段。现有的无令牌方法使用辅助损失训练该边界预测变量，该辅助损失在整个训练语料库中强制执行固定的压缩率，从而引入了一种新的刚性。我们提出了Flexitokens，这是一个简化的训练目标，可以在适应过程中明显更大的灵活性。在多个多语言基准，形态上多样化的任务和领域进行评估时，我们证明了弹性动物始终减少令牌过度分裂，并且与基于子词和其他基于基于梯度的标记者相比，在下游任务绩效上最多可提高下游任务绩效的10 \％。我们的实验的代码和数据将在此HTTPS URL上发布

Title: TransEvalnia: Reasoning-based Evaluation and Ranking of Translations

• Authors: Richard Sproat, Tianyu Zhao, Llion Jones
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2507.12724
• Pdf URL: https://arxiv.org/pdf/2507.12724
• Copy Paste: [[2507.12724]] TransEvalnia: Reasoning-based Evaluation and Ranking of Translations(https://arxiv.org/abs/2507.12724)
• Keywords: llm, prompt
• Abstract: We present TransEvalnia, a prompting-based translation evaluation and ranking system that uses reasoning in performing its evaluations and ranking. This system presents fine-grained evaluations based on a subset of the Multidimensional Quality Metrics (this https URL), returns an assessment of which translation it deems the best, and provides numerical scores for the various dimensions and for the overall translation. We show that TransEvalnia performs as well as or better than the state-of-the-art MT-Ranker (Moosa et al. 2024) on our own English-Japanese data as well as several language pairs from various WMT shared tasks. Using Anthropic's Claude-3.5-Sonnet and Qwen-2.5-72B-Instruct as the evaluation LLMs, we show that the evaluations returned are deemed highly acceptable to human raters, and that the scores assigned to the translations by Sonnet, as well as other LLMs, correlate well with scores assigned by the human raters. We also note the sensitivity of our system -- as well as MT-Ranker -- to the order in which the translations are presented, and we propose methods to address this position bias. All data, including the system's evaluation and reasoning, human assessments, as well as code is released.
• 摘要：我们提出了Transevalnia，这是一种基于促进的翻译评估和排名系统，该系统在执行其评估和排名中使用推理。该系统根据多维质量指标（此HTTPS URL）的子集进行了细粒度的评估，返回对其认为最好的翻译的评估，并为各种维度和整体翻译提供数值得分。我们表明，Transevalnia在我们自己的英语 - 日本数据以及各种WMT共享任务中的几种语言对上的表现和更好或更好的是MT-Ranker（Moosa etal。2024）。我们以拟人化的Claude-3.5-Sonnet和Qwen-2.5-72b-inscruct作为评估LLM，我们表明，返回的评估被认为是人类评分者的高度可接受的，并且SONNET分配给翻译的分数以及其他LLMS以及其他LLMS，与分配了人类评分的少数相关。我们还注意到系统的敏感性以及MT级的敏感性对呈现翻译的顺序，我们提出了解决此位置偏见的方法。所有数据，包括系统的评估和推理，人类评估以及代码。

Title: Strategy Adaptation in Large Language Model Werewolf Agents

• Authors: Fuya Nakamori, Yin Jou Huang, Fei Cheng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2507.12732
• Pdf URL: https://arxiv.org/pdf/2507.12732
• Copy Paste: [[2507.12732]] Strategy Adaptation in Large Language Model Werewolf Agents(https://arxiv.org/abs/2507.12732)
• Keywords: language model, prompt, agent
• Abstract: This study proposes a method to improve the performance of Werewolf agents by switching between predefined strategies based on the attitudes of other players and the context of conversations. While prior works of Werewolf agents using prompt engineering have employed methods where effective strategies are implicitly defined, they cannot adapt to changing situations. In this research, we propose a method that explicitly selects an appropriate strategy based on the game context and the estimated roles of other players. We compare the strategy adaptation Werewolf agents with baseline agents using implicit or fixed strategies and verify the effectiveness of our proposed method.
• 摘要：这项研究提出了一种通过基于其他参与者的态度和对话的背景在预定义的策略之间切换的方法来改善狼人代理的性能。虽然使用及时工程的狼人代理人的先前作品采用了有效策略的定义，但它们无法适应不断变化的情况。在这项研究中，我们提出了一种根据游戏上下文和其他玩家的估计角色明确选择适当策略的方法。我们使用隐式或固定策略将策略适应狼人与基线代理进行比较，并验证我们提出的方法的有效性。

Title: Logit Arithmetic Elicits Long Reasoning Capabilities Without Training

• Authors: Yunxiang Zhang, Muhammad Khalifa, Lechen Zhang, Xin Liu, Ayoung Lee, Xinliang Frederick Zhang, Farima Fatahi Bayat, Lu Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2507.12759
• Pdf URL: https://arxiv.org/pdf/2507.12759
• Copy Paste: [[2507.12759]] Logit Arithmetic Elicits Long Reasoning Capabilities Without Training(https://arxiv.org/abs/2507.12759)
• Keywords: chain-of-thought
• Abstract: Large reasoning models (LRMs) can do complex reasoning via long chain-of-thought (CoT) involving cognitive strategies such as backtracking and self-correction. Recent studies suggest that some models inherently possess these long reasoning abilities, which may be unlocked via extra training. Our work first investigates whether we can elicit such behavior without any training. To this end, we propose a decoding-time approach, ThinkLogit, which utilizes logits arithmetic (Liu et al., 2024) to tune a target large LM for long reasoning using a substantially smaller model as guider. We then show that we can further boost performance by training the guider model with preference optimization over correct/incorrect reasoning pairs sampled from both the target and guider model -- a setup we refer to as ThinkLogit-DPO. Our experiments demonstrate that ThinkLogit and ThinkLogit-DPO achieve a relative improvement in pass@1 by 26% and 29%, respectively, over four mathematical datasets using the Qwen2.5-32B when guided by R1-Distill-Qwen-1.5B -- a model 21x smaller. Lastly, we show that ThinkLogit can transfer long reasoning skills acquired through reinforcement learning, improving pass@1 by 13% relative compared to the Qwen2.5-32B base model. Our work presents a computationally-efficient method to elicit long reasoning in large models with minimal or no additional training.
• 摘要：大型推理模型（LRMS）可以通过涉及诸如回溯和自我纠正等认知策略的长链（COT）进行复杂的推理。最近的研究表明，某些模型固有地具有这些长期的推理能力，可以通过额外的培训解锁。我们的工作首先调查我们是否可以在没有任何培训的情况下引发这种行为。为此，我们提出了一种解码时间方法ThinkLogit，它利用logits算术（Liu等，2024）来调整目标大LM，以长期使用较小的模型作为指导者。然后，我们证明，我们可以通过训练指南模型以优先优化的是从目标和指南模型采样的正确/错误的推理对来进一步提高性能 - 我们称之为ThinkLogit-DPO。我们的实验表明，ThinkLogit和ThinkLogit-DPO在通过QWEN2.5-32B的四个数学数据集中，通过R1-Distill-Qwen-1.5B指导时，Pass@1分别在1个数学数据集中分别取得了相对改善，分别在四个数学数据集中实现了相对改善。最后，我们表明，与QWEN2.5-32B基本模型相比，ThinkLogit可以通过强化学习获得长期的推理技能，将Pass@1相对13％。我们的工作提出了一种计算效率的方法，可以在最少或没有其他培训的大型模型中引起长期推理。

Title: Synergy: End-to-end Concept Model

• Authors: Keli Zheng, Zerong Xie
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2507.12769
• Pdf URL: https://arxiv.org/pdf/2507.12769
• Copy Paste: [[2507.12769]] Synergy: End-to-end Concept Model(https://arxiv.org/abs/2507.12769)
• Keywords: language model
• Abstract: In this paper, we present Synergy, a language model that bridges different levels of abstraction in an end-to-end fashion through a learned routing mechanism. Focusing on low-level linguistic abstraction, we trained our model as a byte-level language model. Our model spontaneously learns to tokenize bytes, producing fewer concept tokens than Byte-level Byte Pair Encoder (BBPE) tokenizers while keeping comparable performance. By comparing with Llama3, we observed an advantage of Synergy under the same model scale and training dataset size. Further studies show that the middle part (the higher abstraction part) of our model performs better when positional encodings are removed, suggesting the emergence of position-independent concepts. These findings demonstrate the feasibility of tokenizer-free architectures, paving the way for more robust and flexible pipelines.
• 摘要：在本文中，我们提出协同作用，该语言模型通过学习的路由机制以端到端方式桥接不同水平的抽象。为了关注低级语言抽象，我们训练了模型作为字节级的语言模型。我们的模型自发地学会了代币化字节，比字节级字节对编码器（BBPE）代币制成的概念令牌更少，同时保持可比的性能。通过与Llama3进行比较，我们观察到在相同的模型量表和训练数据集大小下的协同作用的优势。进一步的研究表明，当删除位置编码时，模型的中间部分（较高的抽象部分）的性能更好，表明与位置无关的概念的出现。这些发现证明了无令牌架构的可行性，为更健壮和灵活的管道铺平了道路。

Title: Learning Robust Negation Text Representations

• Authors: Thinh Hung Truong, Karin Verspoor, Trevor Cohn, Timothy Baldwin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2507.12782
• Pdf URL: https://arxiv.org/pdf/2507.12782
• Copy Paste: [[2507.12782]] Learning Robust Negation Text Representations(https://arxiv.org/abs/2507.12782)
• Keywords: language model, llm
• Abstract: Despite rapid adoption of autoregressive large language models, smaller text encoders still play an important role in text understanding tasks that require rich contextualized representations. Negation is an important semantic function that is still not properly captured by such methods, affecting many downstream applications relying on text embeddings. We propose a strategy to improve negation robustness of text encoders, by distilling data from large language models using diverse patterns of negation and hedging. We adopt a standard contrastive learning strategy to finetune a strong BERT-based model, and observe large improvement in negation understanding capabilities while maintaining competitive performance on general benchmarks. In addition, we also show that our method can be adapted to LLMs, leading to improved performance on negation benchmarks.
• 摘要：尽管快速采用了自回归的大语言模型，但较小的文本编码器仍然在理解需要丰富上下文化表示的文本任务中发挥重要作用。否定是一个重要的语义函数，这种方法仍然无法通过此类方法正确捕获，从而影响了许多依赖文本嵌入的下游应用程序。我们提出了一种策略，通过使用各种否定模式和对冲模式从大型语言模型中提取数据，以提高文本编码器的稳健性。我们采用标准的对比度学习策略来对基于BERT的强大模型进行验证，并观察到否定能力的巨大改善，同时保持一般基准的竞争性能。此外，我们还表明我们的方法可以适应LLM，从而改善了否定基准的性能。

Title: Large Language Models' Internal Perception of Symbolic Music

• Authors: Andrew Shin, Kunitake Kaneko
• Subjects: cs.CL, cs.AI, cs.LG, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2507.12808
• Pdf URL: https://arxiv.org/pdf/2507.12808
• Copy Paste: [[2507.12808]] Large Language Models' Internal Perception of Symbolic Music(https://arxiv.org/abs/2507.12808)
• Keywords: language model, llm, prompt
• Abstract: Large language models (LLMs) excel at modeling relationships between strings in natural language and have shown promise in extending to other symbolic domains like coding or mathematics. However, the extent to which they implicitly model symbolic music remains underexplored. This paper investigates how LLMs represent musical concepts by generating symbolic music data from textual prompts describing combinations of genres and styles, and evaluating their utility through recognition and generation tasks. We produce a dataset of LLM-generated MIDI files without relying on explicit musical training. We then train neural networks entirely on this LLM-generated MIDI dataset and perform genre and style classification as well as melody completion, benchmarking their performance against established models. Our results demonstrate that LLMs can infer rudimentary musical structures and temporal relationships from text, highlighting both their potential to implicitly encode musical patterns and their limitations due to a lack of explicit musical context, shedding light on their generative capabilities for symbolic music.
• 摘要：大型语言模型（LLMS）在模拟自然语言的字符串之间的关系方面表现出色，并在扩展到其他符号领域（如编码或数学）方面表现出了希望。但是，他们隐式模拟符号音乐的程度仍然没有充满反响。本文通过从文本提示中生成符号音乐数据来调查LLM代表音乐概念，从而描述了类型和样式的组合，并通过识别和发电任务来评估其效用。我们在不依赖明确的音乐培训的情况下生成了LLM生成的MIDI文件的数据集。然后，我们完全在此LLM生成的MIDI数据集上训练神经网络，并执行类型和样式分类以及旋律完成，从而对其既定模型进行了基准测试。我们的结果表明，LLM可以从文本中推断出基本的音乐结构和时间关系，突出了它们隐式编码音乐模式的潜力以及由于缺乏明确的音乐背景而导致的局限性，从而阐明了它们对象征性音乐的生成能力。

Title: Are Knowledge and Reference in Multilingual Language Models Cross-Lingually Consistent?

• Authors: Xi Ai, Mahardika Krisna Ihsani, Min-Yen Kan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2507.12838
• Pdf URL: https://arxiv.org/pdf/2507.12838
• Copy Paste: [[2507.12838]] Are Knowledge and Reference in Multilingual Language Models Cross-Lingually Consistent?(https://arxiv.org/abs/2507.12838)
• Keywords: language model
• Abstract: Cross-lingual consistency should be considered to assess cross-lingual transferability, maintain the factuality of the model knowledge across languages, and preserve the parity of language model performance. We are thus interested in analyzing, evaluating, and interpreting cross-lingual consistency for factual knowledge. We examine code-mixed coreferential statements conveyed identical knowledge across languages to study cross-lingual knowledge consistency. We use some interpretability approaches to analyze the behavior of a model in cross-lingual contexts, discovering that multilingual models show different levels of consistency, subject to language families, linguistic factors, and a bottleneck in cross-lingual consistency on a particular layer. In addition, we evaluate common strategies aimed at improving multilingual performance to observe whether these strategies can improve knowledge consistency at the same time. While knowledge is not cross-lingual consistency in many cases, code-switching training and cross-lingual word alignment objectives show the most promising results, emphasizing the noteworthiness of cross-lingual alignment supervision and code-switching training for both multilingual performance and cross-lingual consistency enhancement.
• 摘要：应该考虑跨语性的一致性来评估跨语言的可转移性，保持跨语言的模型知识的事实，并保留语言模型表现的均等。因此，我们有兴趣分析，评估和解释事实知识的跨语性一致性。我们研究了代码混合的核心式陈述，传达了跨语言的相同知识，以研究跨语性知识的一致性。我们使用一些可解释性方法来分析模型在跨语性环境中的行为，发现多语言模型显示出不同级别的一致性，受语言家族，语言因素和特定层上跨语言一致性的瓶颈显示。此外，我们评估了旨在提高多语言绩效的常见策略，以观察这些策略是否可以同时提高知识一致性。尽管知识在许多情况下不是跨语性的一致性，但代码开关培训和跨语言单词一致性目标表明了最有希望的结果，这强调了跨语言对准监督和代码转换培训的杰出性，以提高多语言性能和跨语言一致性。

Title: Making Language Model a Hierarchical Classifier and Generator

• Authors: Yihong Wang, Zhonglin Jiang, Ningyuan Xi, Yue Zhao, Qingqing Gu, Xiyuan Chen, Hao Wu, Sheng Xu, Hange Zhou, Yong Chen, Luo Ji
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2507.12930
• Pdf URL: https://arxiv.org/pdf/2507.12930
• Copy Paste: [[2507.12930]] Making Language Model a Hierarchical Classifier and Generator(https://arxiv.org/abs/2507.12930)
• Keywords: language model, gpt
• Abstract: Decoder-only language models, such as GPT and LLaMA, generally decode on the last layer. Motivated by human's hierarchical thinking capability, we propose that a hierarchical decoder architecture could be built with different layers decoding texts simultaneously. Due to limited time and computationally resources, we choose to adapt a pretrained language model into this form of hierarchical decoder. Language heads of the last layer are copied to different selected intermediate layers, and fine-tuned with different task inputs. By thorough experiments, we validate that these selective intermediate layers could be adapted to speak meaningful and reasonable contents, and this paradigm of hierarchical decoder can obtain state-of-the-art performances on multiple tasks such as hierarchical text classification, classification-guided generation, and hierarchical text generation. This study suggests the possibility of a generalized hierarchical reasoner, pretraining from scratch.
• 摘要：GPT和Llama之类的仅解码语言模型通常在最后一层中解码。由人类的层次思维能力激励，我们建议可以同时使用不同的层解码文本来构建分层解码器架构。由于时间和计算资源有限，我们选择将验证的语言模型调整为这种形式的分层解码器。最后一层的语言头被复制到不同选定的中间层，并通过不同的任务输入进行了微调。通过彻底的实验，我们验证了这些选择性的中间层可以适应有意义且合理的内容，并且这种层次解码器的范式可以在多个任务上获得最先进的性能，例如层次文本分类，分类指导的产生和层次文本生成。这项研究表明，从头开始预处理的广义层次原因器可能性。

Title: MRT at IberLEF-2025 PRESTA Task: Maximizing Recovery from Tables with Multiple Steps

• Authors: Maximiliano Hormazábal Lagos, Álvaro Bueno Sáez, Héctor Cerezo-Costas, Pedro Alonso Doval, Jorge Alcalde Vesteiro
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2507.12981
• Pdf URL: https://arxiv.org/pdf/2507.12981
• Copy Paste: [[2507.12981]] MRT at IberLEF-2025 PRESTA Task: Maximizing Recovery from Tables with Multiple Steps(https://arxiv.org/abs/2507.12981)
• Keywords: llm, prompt
• Abstract: This paper presents our approach for the IberLEF 2025 Task PRESTA: Preguntas y Respuestas sobre Tablas en Español (Questions and Answers about Tables in Spanish). Our solution obtains answers to the questions by implementing Python code generation with LLMs that is used to filter and process the table. This solution evolves from the MRT implementation for the Semeval 2025 related task. The process consists of multiple steps: analyzing and understanding the content of the table, selecting the useful columns, generating instructions in natural language, translating these instructions to code, running it, and handling potential errors or exceptions. These steps use open-source LLMs and fine-grained optimized prompts for each step. With this approach, we achieved an accuracy score of 85\% in the task.
• 摘要：本文介绍了我们针对Iberlef 2025任务Presta的方法：Preguntas y respuestas sobre tablas enespañol（有关西班牙语的问题和答案）。我们的解决方案通过使用用于过滤和处理表的LLM实施Python代码生成来获得问题的答案。该解决方案是从2025年Semeval相关任务的MRT实施中演变而成的。该过程包括多个步骤：分析和理解表的内容，选择有用的列，以自然语言生成指令，将这些说明转换为代码，运行它以及处理潜在的错误或异常。这些步骤为每个步骤使用开源LLM和细粒的优化提示。通过这种方法，我们在任务中获得了85 \％的精度得分。

Title: SemCSE: Semantic Contrastive Sentence Embeddings Using LLM-Generated Summaries For Scientific Abstracts

• Authors: Marc Brinner, Sina Zarriess
• Subjects: cs.CL, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2507.13105
• Pdf URL: https://arxiv.org/pdf/2507.13105
• Copy Paste: [[2507.13105]] SemCSE: Semantic Contrastive Sentence Embeddings Using LLM-Generated Summaries For Scientific Abstracts(https://arxiv.org/abs/2507.13105)
• Keywords: llm
• Abstract: We introduce SemCSE, an unsupervised method for learning semantic embeddings of scientific texts. Building on recent advances in contrastive learning for text embeddings, our approach leverages LLM-generated summaries of scientific abstracts to train a model that positions semantically related summaries closer together in the embedding space. This resulting objective ensures that the model captures the true semantic content of a text, in contrast to traditional citation-based approaches that do not necessarily reflect semantic similarity. To validate this, we propose a novel benchmark designed to assess a model's ability to understand and encode the semantic content of scientific texts, demonstrating that our method enforces a stronger semantic separation within the embedding space. Additionally, we evaluate SemCSE on the comprehensive SciRepEval benchmark for scientific text embeddings, where it achieves state-of-the-art performance among models of its size, thus highlighting the benefits of a semantically focused training approach.
• 摘要：我们介绍了Semcse，这是一种学习科学文本语义嵌入的无监督方法。基于对比对比的文本嵌入的最新进展，我们的方法利用了LLM生成的科学摘要摘要来训练一种模型，该模型将语义上相关的摘要定位在嵌入空间中。与传统的基于引用的方法相比，该模型可确保该模型捕获文本的真实语义内容，这些方法不一定反映了语义相似性。为了验证这一点，我们提出了一种新颖的基准测试，旨在评估模型理解和编码科学文本的语义内容的能力，证明我们的方法在嵌入空间内实施了更强的语义分离。此外，我们对SEMCSE进行了有关科学文本嵌入的综合Scirepepeval基准测试的评估，在该基准中，它在其大小的模型中实现了最先进的性能，从而突出了语义上集中的训练方法的好处。

Title: A Computational Framework to Identify Self-Aspects in Text

• Authors: Jaya Caporusso, Matthew Purver, Senja Pollak
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2507.13115
• Pdf URL: https://arxiv.org/pdf/2507.13115
• Copy Paste: [[2507.13115]] A Computational Framework to Identify Self-Aspects in Text(https://arxiv.org/abs/2507.13115)
• Keywords: language model
• Abstract: This Ph.D. proposal introduces a plan to develop a computational framework to identify Self-aspects in text. The Self is a multifaceted construct and it is reflected in language. While it is described across disciplines like cognitive science and phenomenology, it remains underexplored in natural language processing (NLP). Many of the aspects of the Self align with psychological and other well-researched phenomena (e.g., those related to mental health), highlighting the need for systematic NLP-based analysis. In line with this, we plan to introduce an ontology of Self-aspects and a gold-standard annotated dataset. Using this foundation, we will develop and evaluate conventional discriminative models, generative large language models, and embedding-based retrieval approaches against four main criteria: interpretability, ground-truth adherence, accuracy, and computational efficiency. Top-performing models will be applied in case studies in mental health and empirical phenomenology.
• 摘要：这个博士提案介绍了一项计划，以制定计算框架，以识别文本中的自我表现。自我是一种多方面的结构，它反映在语言中。尽管它在诸如认知科学和现象学之类的学科中进行了描述，但在自然语言处理（NLP）中仍然没有被忽视。自我与心理和其他经过深入研究的现象（例如，与心理健康相关的现象）的许多方面相符，强调了对系统的基于NLP的分析的需求。与此一致，我们计划介绍自我识别和金标准注释数据集的本体。使用此基础，我们将针对四个主要标准开发和评估常规判别模型，生成大语言模型以及基于嵌入的检索方法：解释性，基础真相遵守，准确性和计算效率。最佳模型将应用于心理健康和经验现象学的案例研究。

Title: Assessing the Reliability of LLMs Annotations in the Context of Demographic Bias and Model Explanation

• Authors: Hadi Mohammadi, Tina Shahedi, Pablo Mosteiro, Massimo Poesio, Ayoub Bagheri, Anastasia Giachanou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2507.13138
• Pdf URL: https://arxiv.org/pdf/2507.13138
• Copy Paste: [[2507.13138]] Assessing the Reliability of LLMs Annotations in the Context of Demographic Bias and Model Explanation(https://arxiv.org/abs/2507.13138)
• Keywords: llm, prompt
• Abstract: Understanding the sources of variability in annotations is crucial for developing fair NLP systems, especially for tasks like sexism detection where demographic bias is a concern. This study investigates the extent to which annotator demographic features influence labeling decisions compared to text content. Using a Generalized Linear Mixed Model, we quantify this inf luence, finding that while statistically present, demographic factors account for a minor fraction ( 8%) of the observed variance, with tweet content being the dominant factor. We then assess the reliability of Generative AI (GenAI) models as annotators, specifically evaluating if guiding them with demographic personas improves alignment with human judgments. Our results indicate that simplistic persona prompting often fails to enhance, and sometimes degrades, performance compared to baseline models. Furthermore, explainable AI (XAI) techniques reveal that model predictions rely heavily on content-specific tokens related to sexism, rather than correlates of demographic characteristics. We argue that focusing on content-driven explanations and robust annotation protocols offers a more reliable path towards fairness than potentially persona simulation.
• 摘要：了解注释中可变性的来源对于开发公平的NLP系统至关重要，尤其是对于性别歧视检测的任务而言，人口统计学偏见是一个问题。这项研究调查了注释者人口统计学特征与文本内容相比影响标签决策的程度。使用广义的线性混合模型，我们量化了这种信息，发现虽然统计上存在，但人口统计因素占观察到方差的较小比例（8％），推文内容是主要因素。然后，我们评估生成AI（Genai）模型作为注释者的可靠性，专门评估是否指导他们使用人口角色可以改善与人类判断的一致性。我们的结果表明，与基线模型相比，简单的角色提示通常无法增强，有时甚至会降低性能。此外，可解释的AI（XAI）技术表明，模型预测在很大程度上取决于与性别歧视相关的特定于内容的令牌，而不是人口统计学特征的相关性。我们认为，专注于内容驱动的解释和强大的注释协议提供了比潜在的角色模拟更可靠的公平途径。

Title: GEMMAS: Graph-based Evaluation Metrics for Multi Agent Systems

• Authors: Jisoo Lee, Raeyoung Chang, Dongwook Kwon, Harmanpreet Singh, Nikhil Verma
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2507.13190
• Pdf URL: https://arxiv.org/pdf/2507.13190
• Copy Paste: [[2507.13190]] GEMMAS: Graph-based Evaluation Metrics for Multi Agent Systems(https://arxiv.org/abs/2507.13190)
• Keywords: language model, agent
• Abstract: Multi-agent systems built on language models have shown strong performance on collaborative reasoning tasks. However, existing evaluations focus only on the correctness of the final output, overlooking how inefficient communication and poor coordination contribute to redundant reasoning and higher computational costs. We introduce GEMMAS, a graph-based evaluation framework that analyzes the internal collaboration process by modeling agent interactions as a directed acyclic graph. To capture collaboration quality, we propose two process-level metrics: Information Diversity Score (IDS) to measure semantic variation in inter-agent messages, and Unnecessary Path Ratio (UPR) to quantify redundant reasoning paths. We evaluate GEMMAS across five benchmarks and highlight results on GSM8K, where systems with only a 2.1% difference in accuracy differ by 12.8% in IDS and 80% in UPR, revealing substantial variation in internal collaboration. These findings demonstrate that outcome-only metrics are insufficient for evaluating multi-agent performance and highlight the importance of process-level diagnostics in designing more interpretable and resource-efficient collaborative AI systems.
• 摘要：基于语言模型的多机构系统在协作推理任务上表现出强大的性能。但是，现有的评估仅着眼于最终产出的正确性，探讨了沟通效率低下和协调不足如何有助于冗余推理和更高的计算成本。我们介绍了Gemmas，这是一个基于图的评估框架，该框架通过将代理相互作用建模为有向的无环图来分析内部协作过程。为了捕获协作质量，我们提出了两个过程级指标：信息多样性评分（IDS），以衡量代理间消息中的语义变化，以及不必要的路径比（UPR）来量化冗余推理路径。我们在五个基准测试中评估了Gemmas，并突出了GSM8K上的结果，在GSM8K上，准确性差异仅为2.1％的系统在ID中只有12.8％，而UPR的系统差异为80％，揭示了内部协作的实质性差异。这些发现表明，仅成果的指标不足以评估多代理性能，并强调了过程级诊断在设计更加可解释和资源有效的协作AI系统中的重要性。

Title: Automatically assessing oral narratives of Afrikaans and isiXhosa children

• Authors: R. Louw (1), E. Sharratt (1), F. de Wet (1), C. Jacobs (1), A. Smith (1), H. Kamper (1) ((1) Stellenbosch University)
• Subjects: cs.CL, eess.AS
• Abstract URL: https://arxiv.org/abs/2507.13205
• Pdf URL: https://arxiv.org/pdf/2507.13205
• Copy Paste: [[2507.13205]] Automatically assessing oral narratives of Afrikaans and isiXhosa children(https://arxiv.org/abs/2507.13205)
• Keywords: language model, llm
• Abstract: Developing narrative and comprehension skills in early childhood is critical for later literacy. However, teachers in large preschool classrooms struggle to accurately identify students who require intervention. We present a system for automatically assessing oral narratives of preschool children in Afrikaans and isiXhosa. The system uses automatic speech recognition followed by a machine learning scoring model to predict narrative and comprehension scores. For scoring predicted transcripts, we compare a linear model to a large language model (LLM). The LLM-based system outperforms the linear model in most cases, but the linear system is competitive despite its simplicity. The LLM-based system is comparable to a human expert in flagging children who require intervention. We lay the foundation for automatic oral assessments in classrooms, giving teachers extra capacity to focus on personalised support for children's learning.
• 摘要：在幼儿期发展叙事和理解能力对于后期扫盲至关重要。但是，大型学龄前教室的老师努力准确地确定需要干预的学生。我们提出了一个系统，用于自动评估南非荷兰语和Isixhosa中学龄前儿童的口腔叙事。该系统使用自动语音识别，然后使用机器学习评分模型来预测叙事和理解分数。对于评分预测的成绩单，我们将线性模型与大语言模型（LLM）进行比较。基于LLM的系统在大多数情况下都优于线性模型，但是线性系统尽管简单，但线性系统还是竞争性的。基于LLM的系统与需要干预的儿童的人类专家相媲美。我们为课堂上的自动口头评估奠定了基础，为教师提供了更多专注于对儿童学习的个性化支持的能力。

Title: Enhancing Cross-task Transfer of Large Language Models via Activation Steering

• Authors: Xinyu Tang, Zhihao Lv, Xiaoxue Cheng, Junyi Li, Wayne Xin Zhao, Zujie Wen, Zhiqiang Zhang, Jun Zhou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2507.13236
• Pdf URL: https://arxiv.org/pdf/2507.13236
• Copy Paste: [[2507.13236]] Enhancing Cross-task Transfer of Large Language Models via Activation Steering(https://arxiv.org/abs/2507.13236)
• Keywords: language model, llm, prompt
• Abstract: Large language models (LLMs) have shown impressive abilities in leveraging pretrained knowledge through prompting, but they often struggle with unseen tasks, particularly in data-scarce scenarios. While cross-task in-context learning offers a direct solution for transferring knowledge across tasks, it still faces critical challenges in terms of robustness, scalability, and efficiency. In this paper, we investigate whether cross-task transfer can be achieved via latent space steering without parameter updates or input expansion. Through an analysis of activation patterns in the latent space of LLMs, we observe that the enhanced activations induced by in-context examples have consistent patterns across different tasks. Inspired by these findings, we propose CAST, a novel Cross-task Activation Steering Transfer framework that enables effective transfer by manipulating the model's internal activation states. Our approach first selects influential and diverse samples from high-resource tasks, then utilizes their contrastive representation-enhanced activations to adapt LLMs to low-resource tasks. Extensive experiments across both cross-domain and cross-lingual transfer settings show that our method outperforms competitive baselines and demonstrates superior scalability and lower computational costs.
• 摘要：大型语言模型（LLM）在通过提示来利用预验证的知识方面表现出了令人印象深刻的能力，但是他们经常在看不见的任务中挣扎，尤其是在数据筛选的情况下。虽然交叉任务中的内在学习提供了一种直接解决方案，以跨任务传输知识，但它仍然面临着鲁棒性，可伸缩性和效率的关键挑战。在本文中，我们研究了是否可以通过潜在空间转向来实现跨任务传输，而无需参数更新或输入扩展。通过分析LLM的潜在空间中的激活模式，我们观察到，在不同任务之间，由膜中下文示例引起的增强激活具有一致的模式。受这些发现的启发，我们提出了一种新型的跨任务激活转向转移框架，该框架可以通过操纵模型的内部激活状态来有效转移。我们的方法首先从高资源任务中选择了有影响力和多样化的样本，然后利用其对比度表示激活来使LLMS适应低资源任务。跨域和跨语性转移设置进行的广泛实验表明，我们的方法优于竞争基准，表现出卓越的可扩展性和降低计算成本。

Title: HATS: Hindi Analogy Test Set for Evaluating Reasoning in Large Language Models

• Authors: Ashray Gupta, Rohan Joseph, Sunny Rai
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2507.13238
• Pdf URL: https://arxiv.org/pdf/2507.13238
• Copy Paste: [[2507.13238]] HATS: Hindi Analogy Test Set for Evaluating Reasoning in Large Language Models(https://arxiv.org/abs/2507.13238)
• Keywords: language model, llm, prompt
• Abstract: Analogies test a model's ability to infer implicit relationships between concepts, making them a key benchmark for evaluating reasoning capabilities. While large language models (LLMs) are widely evaluated for reasoning in English, their abilities in Indic languages remain understudied, limiting our understanding of whether these models generalize across languages. To address this gap, we introduce a new Hindi Analogy Test Set (HATS), comprising 405 multiple-choice questions sourced from Indian government exams. We benchmark state-of-the-art multilingual LLMs using various prompting strategies and introduce a grounded Chain of Thought approach that leverages cognitive theories of analogical reasoning. This approach improves model performance on Hindi analogy questions. Our experiments show that models perform best with English prompts, irrespective of the prompting strategy. Our test set addresses the lack of a critical resource to evaluate LLM reasoning capabilities in Hindi.
• 摘要：类比测试模型推断概念之间隐含关系的能力，使其成为评估推理能力的关键基准。尽管大型语言模型（LLM）经过广泛评估以用英语推理，但它们的能力在指示语言中仍在研究中，限制了我们对这些模型是否跨语言概括的理解。为了解决这一差距，我们引入了新的印地语类比测试集（HATS），其中包括405个来自印度政府考试的多项选择问题。我们使用各种提示策略基准了最先进的多语言LLM，并引入了一种扎根的思想方法链，该方法利用了类似推理的认知理论。这种方法改善了印地语类比方面的模型性能。我们的实验表明，无论提示策略如何，模型在英语提示中的表现最佳。我们的测试集解决了缺乏评估印地语LLM推理功能的关键资源。

Title: Automating Steering for Safe Multimodal Large Language Models

• Authors: Lyucheng Wu, Mengru Wang, Ziwen Xu, Tri Cao, Nay Oo, Bryan Hooi, Shumin Deng
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG, cs.MM
• Abstract URL: https://arxiv.org/abs/2507.13255
• Pdf URL: https://arxiv.org/pdf/2507.13255
• Copy Paste: [[2507.13255]] Automating Steering for Safe Multimodal Large Language Models(https://arxiv.org/abs/2507.13255)
• Keywords: language model, llm
• Abstract: Recent progress in Multimodal Large Language Models (MLLMs) has unlocked powerful cross-modal reasoning abilities, but also raised new safety concerns, particularly when faced with adversarial multimodal inputs. To improve the safety of MLLMs during inference, we introduce a modular and adaptive inference-time intervention technology, AutoSteer, without requiring any fine-tuning of the underlying model. AutoSteer incorporates three core components: (1) a novel Safety Awareness Score (SAS) that automatically identifies the most safety-relevant distinctions among the model's internal layers; (2) an adaptive safety prober trained to estimate the likelihood of toxic outputs from intermediate representations; and (3) a lightweight Refusal Head that selectively intervenes to modulate generation when safety risks are detected. Experiments on LLaVA-OV and Chameleon across diverse safety-critical benchmarks demonstrate that AutoSteer significantly reduces the Attack Success Rate (ASR) for textual, visual, and cross-modal threats, while maintaining general abilities. These findings position AutoSteer as a practical, interpretable, and effective framework for safer deployment of multimodal AI systems.
• 摘要：多模式大语言模型（MLLM）的最新进展已解锁了强大的跨模式推理能力，但也引起了新的安全问题，尤其是在面对对抗性多模式输入时。为了提高推理期间MLLM的安全性，我们引入了一种模块化和自适应推理时间干预技术，即自动赛车，而无需对基础模型进行任何微调。 AutoSteer结合了三个核心组成部分：（1）一种新型的安全意识评分（SAS），该得分自动识别模型内部层中最安全的区别； （2）经过培训的自适应安全专家，以估算中间表示的有毒输出的可能性； （3）一个轻巧的拒绝头，在检测到安全风险时有选择地干预以调节生成。对Llava-ov和Chameleon进行了各种安全 - 关键基准测试的实验表明，对于文本，视觉和跨模式威胁，自动赛车可显着降低攻击成功率（ASR），同时保持一般能力。这些发现的位置自动座位是一种实用，可解释且有效的框架，以更安全地部署多模式AI系统。

Title: QuestA: Expanding Reasoning Capacity in LLMs via Question Augmentation

• Authors: Jiazheng Li, Hong Lu, Kaiyue Wen, Zaiwen Yang, Jiaxuan Gao, Hongzhou Lin, Yi Wu, Jingzhao Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2507.13266
• Pdf URL: https://arxiv.org/pdf/2507.13266
• Copy Paste: [[2507.13266]] QuestA: Expanding Reasoning Capacity in LLMs via Question Augmentation(https://arxiv.org/abs/2507.13266)
• Keywords: llm
• Abstract: Reinforcement learning (RL) has become a key component in training large language reasoning models (LLMs). However, recent studies questions its effectiveness in improving multi-step reasoning-particularly on hard problems. To address this challenge, we propose a simple yet effective strategy via Question Augmentation: introduce partial solutions during training to reduce problem difficulty and provide more informative learning signals. Our method, QuestA, when applied during RL training on math reasoning tasks, not only improves pass@1 but also pass@k-particularly on problems where standard RL struggles to make progress. This enables continual improvement over strong open-source models such as DeepScaleR and OpenMath Nemotron, further enhancing their reasoning capabilities. We achieve new state-of-the-art results on math benchmarks using 1.5B-parameter models: 67.1% (+5.3%) on AIME24, 59.5% (+10.0%) on AIME25, and 35.5% (+4.0%) on HMMT25. Further, we provide theoretical explanations that QuestA improves sample efficiency, offering a practical and generalizable pathway for expanding reasoning capability through RL.
• 摘要：强化学习（RL）已成为培训大语言推理模型（LLMS）的关键组成部分。但是，最近的研究质疑其在严重问题尤其改善多步推理方面的有效性。为了应对这一挑战，我们提出了一个简单而有效的策略，可以通过问题增强：在培训期间介绍部分解决方案，以减少问题难度并提供更有用的学习信号。我们的方法QuestA在RL培训中使用数学推理任务时应用，不仅可以改善PASS@1，还可以通过@k-particular在标准RL努力取得进步的问题上通过。这使得对强大的开源模型（例如DeepScaler和OpenMath Nemotron）不断改进，进一步增强了其推理能力。我们使用1.5B参数型号在数学基准上获得了新的最新结果：AIME24的AIME25上的AIME25上的67.1％（+5.3％），HMMT25上的AIME25和35.5％（+4.0％）。此外，我们提供了理论上的解释，即Questa提高样本效率，为通过RL扩大推理能力的实用且可概括的途径。

Title: Overview of the TalentCLEF 2025: Skill and Job Title Intelligence for Human Capital Management

• Authors: Luis Gasco, Hermenegildo Fabregat, Laura García-Sardiña, Paula Estrella, Daniel Deniz, Alvaro Rodrigo, Rabih Zbib
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2507.13275
• Pdf URL: https://arxiv.org/pdf/2507.13275
• Copy Paste: [[2507.13275]] Overview of the TalentCLEF 2025: Skill and Job Title Intelligence for Human Capital Management(https://arxiv.org/abs/2507.13275)
• Keywords: language model
• Abstract: Advances in natural language processing and large language models are driving a major transformation in Human Capital Management, with a growing interest in building smart systems based on language technologies for talent acquisition, upskilling strategies, and workforce planning. However, the adoption and progress of these technologies critically depend on the development of reliable and fair models, properly evaluated on public data and open benchmarks, which have so far been unavailable in this domain. To address this gap, we present TalentCLEF 2025, the first evaluation campaign focused on skill and job title intelligence. The lab consists of two tasks: Task A - Multilingual Job Title Matching, covering English, Spanish, German, and Chinese; and Task B - Job Title-Based Skill Prediction, in English. Both corpora were built from real job applications, carefully anonymized, and manually annotated to reflect the complexity and diversity of real-world labor market data, including linguistic variability and gender-marked expressions. The evaluations included monolingual and cross-lingual scenarios and covered the evaluation of gender bias. TalentCLEF attracted 76 registered teams with more than 280 submissions. Most systems relied on information retrieval techniques built with multilingual encoder-based models fine-tuned with contrastive learning, and several of them incorporated large language models for data augmentation or re-ranking. The results show that the training strategies have a larger effect than the size of the model alone. TalentCLEF provides the first public benchmark in this field and encourages the development of robust, fair, and transferable language technologies for the labor market.
• 摘要：自然语言处理和大型语言模型的进步正在推动人力资本管理的重大转变，对基于人才获取，提高技能策略和劳动力计划的语言技术建立智能系统的兴趣越来越兴趣。但是，这些技术的采用和进步在很大程度上取决于可靠且公平的模型的开发，这些模型对公共数据和开放基准进行了适当评估，到目前为止，该模型在该领域中无法使用。为了解决这一差距，我们提出了Talentclef 2025，这是第一个评估活动，重点是技能和职权智能。该实验室由两个任务组成：任务A-多语言职位匹配，涵盖英语，西班牙语，德语和中文；和任务B-英语基于职权的技能预测。这两个语料库都是由实际的工作申请构建的，经过精心匿名化并进行了手动注释，以反映现实世界中劳动力市场数据的复杂性和多样性，包括语言可变性和性别标记的表达式。评估包括单语言和跨语言情景，并涵盖了性别偏见的评估。 Talentclef吸引了76支注册团队，其中有280多个意见书。大多数系统都依赖于信息检索技术，该技术是由基于多语言编码器的模型构建的，并通过对比度学习进行了微调，其中一些系统融合了大型语言模型，以进行数据增强或重新排列。结果表明，训练策略的效果比单独的模型的大小更大。 Talentclef提供了该领域的第一个公共基准，并鼓励为劳动力市场开发强大，公平和可转移的语言技术。

Title: Multi-Agent Synergy-Driven Iterative Visual Narrative Synthesis

• Authors: Wang Xi, Quan Shi, Tian Yu, Yujie Peng, Jiayi Sun, Mengxing Ren, Zenghui Ding, Ningguang Yao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2507.13285
• Pdf URL: https://arxiv.org/pdf/2507.13285
• Copy Paste: [[2507.13285]] Multi-Agent Synergy-Driven Iterative Visual Narrative Synthesis(https://arxiv.org/abs/2507.13285)
• Keywords: agent
• Abstract: Automated generation of high-quality media presentations is challenging, requiring robust content extraction, narrative planning, visual design, and overall quality optimization. Existing methods often produce presentations with logical inconsistencies and suboptimal layouts, thereby struggling to meet professional standards. To address these challenges, we introduce RCPS (Reflective Coherent Presentation Synthesis), a novel framework integrating three key components: (1) Deep Structured Narrative Planning; (2) Adaptive Layout Generation; (3) an Iterative Optimization Loop. Additionally, we propose PREVAL, a preference-based evaluation framework employing rationale-enhanced multi-dimensional models to assess presentation quality across Content, Coherence, and Design. Experimental results demonstrate that RCPS significantly outperforms baseline methods across all quality dimensions, producing presentations that closely approximate human expert standards. PREVAL shows strong correlation with human judgments, validating it as a reliable automated tool for assessing presentation quality.
• 摘要：高质量的媒体演示文稿的自动生成具有挑战性，需要强大的内容提取，叙事计划，视觉设计和整体质量优化。现有方法通常会产生逻辑上的不一致和次优布局的演讲，从而努力达到专业标准。为了应对这些挑战，我们介绍了RCP（反光连贯的演示综合），这是一个集成了三个关键组成部分的新型框架：（1）深层结构化叙事计划； （2）自适应布局生成； （3）迭代优化循环。此外，我们提出了一种基于偏好的评估框架，该框架采用理由增强的多维模型来评估内容，连贯性和设计的演示质量。实验结果表明，RCP的表现明显优于所有质量维度的基线方法，从而产生了近似于人类专家标准的表现。 DEVAL显示与人类判断的密切相关性，将其视为评估演示质量的可靠自动化工具。

Title: AbGen: Evaluating Large Language Models in Ablation Study Design and Evaluation for Scientific Research

• Authors: Yilun Zhao, Weiyuan Chen, Zhijian Xu, Manasi Patwardhan, Yixin Liu, Chengye Wang, Lovekesh Vig, Arman Cohan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2507.13300
• Pdf URL: https://arxiv.org/pdf/2507.13300
• Copy Paste: [[2507.13300]] AbGen: Evaluating Large Language Models in Ablation Study Design and Evaluation for Scientific Research(https://arxiv.org/abs/2507.13300)
• Keywords: language model, llm
• Abstract: We introduce AbGen, the first benchmark designed to evaluate the capabilities of LLMs in designing ablation studies for scientific research. AbGen consists of 1,500 expert-annotated examples derived from 807 NLP papers. In this benchmark, LLMs are tasked with generating detailed ablation study designs for a specified module or process based on the given research context. Our evaluation of leading LLMs, such as DeepSeek-R1-0528 and o4-mini, highlights a significant performance gap between these models and human experts in terms of the importance, faithfulness, and soundness of the ablation study designs. Moreover, we demonstrate that current automated evaluation methods are not reliable for our task, as they show a significant discrepancy when compared to human assessment. To better investigate this, we develop AbGen-Eval, a meta-evaluation benchmark designed to assess the reliability of commonly used automated evaluation systems in measuring LLM performance on our task. We investigate various LLM-as-Judge systems on AbGen-Eval, providing insights for future research on developing more effective and reliable LLM-based evaluation systems for complex scientific tasks.
• 摘要：我们介绍了Abgen，这是第一个旨在评估LLMS在设计消融研究科学研究的能力的基准。 Abgen由来自807份NLP论文的1,500个专家注册的示例组成。在此基准中，LLM的任务是基于给定的研究环境为指定的模块或过程生成详细的消融研究设计。我们对领先的LLM的评估，例如DeepSeek-R1-0528和O4-Mini，在消融研究设计的重要性，忠诚和健全性方面，强调了这些模型与人类专家之间的显着性能差距。此外，我们证明了当前的自动化评估方法对我们的任务不可靠，因为与人类评估相比，它们显示出很大的差异。为了更好地调查这一点，我们开发了Abgen-eval，这是一种元评估基准测试，旨在评估常用自动化评估系统的可靠性，以测量我们任务的LLM性能。我们研究了Abgen-eval上的各种LLM-AS法官系统，为未来的研究提供了有关开发更有效和可靠的基于LLM的评估系统的见解。

Title: HapticCap: A Multimodal Dataset and Task for Understanding User Experience of Vibration Haptic Signals

• Authors: Guimin Hu, Daniel Hershcovich, Hasti Seifi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2507.13318
• Pdf URL: https://arxiv.org/pdf/2507.13318
• Copy Paste: [[2507.13318]] HapticCap: A Multimodal Dataset and Task for Understanding User Experience of Vibration Haptic Signals(https://arxiv.org/abs/2507.13318)
• Keywords: language model
• Abstract: Haptic signals, from smartphone vibrations to virtual reality touch feedback, can effectively convey information and enhance realism, but designing signals that resonate meaningfully with users is challenging. To facilitate this, we introduce a multimodal dataset and task, of matching user descriptions to vibration haptic signals, and highlight two primary challenges: (1) lack of large haptic vibration datasets annotated with textual descriptions as collecting haptic descriptions is time-consuming, and (2) limited capability of existing tasks and models to describe vibration signals in text. To advance this area, we create HapticCap, the first fully human-annotated haptic-captioned dataset, containing 92,070 haptic-text pairs for user descriptions of sensory, emotional, and associative attributes of vibrations. Based on HapticCap, we propose the haptic-caption retrieval task and present the results of this task from a supervised contrastive learning framework that brings together text representations within specific categories and vibrations. Overall, the combination of language model T5 and audio model AST yields the best performance in the haptic-caption retrieval task, especially when separately trained for each description category.
• 摘要：从智能手机振动到虚拟现实触摸反馈，触觉信号可以有效地传达信息并增强现实主义，但是设计信号与用户有意义地共鸣是具有挑战性的。为了促进这一点，我们介绍了一个多模式数据集和任务，将用户描述与振动触觉信号匹配，并突出了两个主要挑战：（1）缺乏用文本描述注释的大型触觉振动数据集，因为将触觉描述收集到了触觉描述，以收集触觉描述是及时的，并且（2）限制了现有任务和模型的能力，以描述vibration和模型的能力。为了促进该区域，我们创建了HapticCap，这是第一个完全被人类通知的触觉捕获数据集，其中包含92,070个触觉 - 文本对，用于用户描述振动的感觉，情感和关联属性。基于HapticCap，我们提出了触觉捕获检索任务，并通过监督的对比度学习框架提出了此任务的结果，该框架将文本表示汇总到特定类别和振动中。总体而言，语言模型T5和音频模型AST的组合在触觉符号检索任务中产生了最佳性能，尤其是在为每个描述类别分别培训时。

Title: Vision-and-Language Training Helps Deploy Taxonomic Knowledge but Does Not Fundamentally Alter It

• Authors: Yulu Qin, Dheeraj Varghese, Adam Dahlgren Lindström, Lucia Donatelli, Kanishka Misra, Najoung Kim
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2507.13328
• Pdf URL: https://arxiv.org/pdf/2507.13328
• Copy Paste: [[2507.13328]] Vision-and-Language Training Helps Deploy Taxonomic Knowledge but Does Not Fundamentally Alter It(https://arxiv.org/abs/2507.13328)
• Keywords: language model
• Abstract: Does vision-and-language (VL) training change the linguistic representations of language models in meaningful ways? Most results in the literature have shown inconsistent or marginal differences, both behaviorally and representationally. In this work, we start from the hypothesis that the domain in which VL training could have a significant effect is lexical-conceptual knowledge, in particular its taxonomic organization. Through comparing minimal pairs of text-only LMs and their VL-trained counterparts, we first show that the VL models often outperform their text-only counterparts on a text-only question-answering task that requires taxonomic understanding of concepts mentioned in the questions. Using an array of targeted behavioral and representational analyses, we show that the LMs and VLMs do not differ significantly in terms of their taxonomic knowledge itself, but they differ in how they represent questions that contain concepts in a taxonomic relation vs. a non-taxonomic relation. This implies that the taxonomic knowledge itself does not change substantially through additional VL training, but VL training does improve the deployment of this knowledge in the context of a specific task, even when the presentation of the task is purely linguistic.
• 摘要：视觉和语言（VL）训练是否会以有意义的方式改变语言模型的语言表示？文献中的大多数结果在行为和代表性上都显示出不一致或边际差异。在这项工作中，我们从以下假设开始：VL培训可能产生重大影响的领域是词汇概念知识，尤其是其分类组织。通过比较最少的仅文本LMS及其VL训练的对应物，我们首先表明，VL模型通常在仅在文本的提问任务上优于其文本范围，这需要对问题中提到的概念进行分类理解。使用一系列有针对性的行为和代表性分析，我们表明，LMS和VLM在其分类知识本身方面并没有显着差异，但是它们在表示分类关系中包含概念的问题上有所不同。这意味着分类知识本身不会通过其他VL培训发生实质性变化，但是VL培训确实在特定任务的背景下改善了该知识的部署，即使任务的呈现纯粹是语言上的。

Title: The Imitation Game: Turing Machine Imitator is Length Generalizable Reasoner

• Authors: Zhouqi Hua, Wenwei Zhang, Chengqi Lyu, Yuzhe Gu, Songyang Gao, Kuikun Liu, Kai Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2507.13332
• Pdf URL: https://arxiv.org/pdf/2507.13332
• Copy Paste: [[2507.13332]] The Imitation Game: Turing Machine Imitator is Length Generalizable Reasoner(https://arxiv.org/abs/2507.13332)
• Keywords: language model, llm, chain-of-thought
• Abstract: Length generalization, the ability to solve problems of longer sequences than those observed during training, poses a core challenge of Transformer-based large language models (LLM). Although existing studies have predominantly focused on data-driven approaches for arithmetic operations and symbolic manipulation tasks, these approaches tend to be task-specific with limited overall performance. To pursue a more general solution, this paper focuses on a broader case of reasoning problems that are computable, i.e., problems that algorithms can solve, thus can be solved by the Turing Machine. From this perspective, this paper proposes Turing MAchine Imitation Learning (TAIL) to improve the length generalization ability of LLMs. TAIL synthesizes chain-of-thoughts (CoT) data that imitate the execution process of a Turing Machine by computer programs, which linearly expands the reasoning steps into atomic states to alleviate shortcut learning and explicit memory fetch mechanism to reduce the difficulties of dynamic and long-range data access in elementary operations. To validate the reliability and universality of TAIL, we construct a challenging synthetic dataset covering 8 classes of algorithms and 18 tasks. Without bells and whistles, TAIL significantly improves the length generalization ability as well as the performance of Qwen2.5-7B on various tasks using only synthetic data, surpassing previous methods and DeepSeek-R1. The experimental results reveal that the key concepts in the Turing Machine, instead of the thinking styles, are indispensable for TAIL for length generalization, through which the model exhibits read-and-write behaviors consistent with the properties of the Turing Machine in their attention layers. This work provides a promising direction for future research in the learning of LLM reasoning from synthetic data.
• 摘要：长度的概括是解决序列问题比训练期间观察到的更长的序列问题的能力，这对基于变压器的大语言模型（LLM）提出了核心挑战。尽管现有的研究主要集中在算术操作和符号操作任务的数据驱动方法上，但这些方法往往是特定于任务的，并且总体绩效有限。为了寻求更一般的解决方案，本文着重于可计算的推理问题的更广泛的案例，即算法可以解决的问题，因此可以通过Turing Machine解决。从这个角度来看，本文提出了图灵机模仿学习（TAIL），以提高LLM的长度概括能力。 Tail合成了通过计算机程序模仿Turing机器的执行过程的经验链（COT）数据，该程序将推理步骤线性扩展到原子状态，以减轻快捷方式学习和明确的内存fetch fetch机制，以减少基本操作中动态和长距离数据访问的困难。为了验证尾巴的可靠性和普遍性，我们构建了一个具有挑战性的合成数据集，涵盖了8类算法和18个任务。没有铃铛和哨子，尾巴可显着提高长度的泛化能力以及仅使用合成数据，超过以前的方法和DeepSeek-R1的各种任务的QWEN2.5-7B的性能。实验结果表明，图灵机中的关键概念而不是思维方式对于长度泛化而言是必不可少的，模型通过该模型表现出与图灵机在注意层中的读取和写入行为。这项工作为从合成数据学习LLM推理的未来研究提供了一个有希望的方向。

Title: A Survey of Context Engineering for Large Language Models

• Authors: Lingrui Mei, Jiayu Yao, Yuyao Ge, Yiwei Wang, Baolong Bi, Yujun Cai, Jiazhi Liu, Mingyu Li, Zhong-Zhi Li, Duzhen Zhang, Chenlin Zhou, Jiayi Mao, Tianze Xia, Jiafeng Guo, Shenghua Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2507.13334
• Pdf URL: https://arxiv.org/pdf/2507.13334
• Copy Paste: [[2507.13334]] A Survey of Context Engineering for Large Language Models(https://arxiv.org/abs/2507.13334)
• Keywords: language model, llm, prompt, retrieval-augmented generation, agent
• Abstract: The performance of Large Language Models (LLMs) is fundamentally determined by the contextual information provided during inference. This survey introduces Context Engineering, a formal discipline that transcends simple prompt design to encompass the systematic optimization of information payloads for LLMs. We present a comprehensive taxonomy decomposing Context Engineering into its foundational components and the sophisticated implementations that integrate them into intelligent systems. We first examine the foundational components: context retrieval and generation, context processing and context management. We then explore how these components are architecturally integrated to create sophisticated system implementations: retrieval-augmented generation (RAG), memory systems and tool-integrated reasoning, and multi-agent systems. Through this systematic analysis of over 1300 research papers, our survey not only establishes a technical roadmap for the field but also reveals a critical research gap: a fundamental asymmetry exists between model capabilities. While current models, augmented by advanced context engineering, demonstrate remarkable proficiency in understanding complex contexts, they exhibit pronounced limitations in generating equally sophisticated, long-form outputs. Addressing this gap is a defining priority for future research. Ultimately, this survey provides a unified framework for both researchers and engineers advancing context-aware AI.
• 摘要：大语言模型（LLM）的性能从根本上取决于推论过程中提供的上下文信息。这项调查介绍了上下文工程，这是一种正式的学科，超越了简单的提示设计，以涵盖LLMS信息有效负载的系统优化。我们向其基础组件和将它们集成到智能系统中的复杂实现中，将上下文工程分解为全面的分类学。我们首先研究了基础组件：上下文检索和生成，上下文处理和上下文管理。然后，我们探讨如何在架构集成中集成这些组件以创建复杂的系统实现：检索功能增强的生成（RAG），内存系统和工具集成推理以及多代理系统。通过对1300多个研究论文的系统分析，我们的调查不仅建立了该领域的技术路线图，而且还揭示了重要的研究差距：模型能力之间存在基本的不对称性。尽管当前的模型是由高级上下文工程增强的，但表现出在理解复杂环境方面的出色熟练程度，但它们在产生同样复杂的长期产出时表现出明显的局限性。解决这一差距是未来研究的定义优先事项。最终，这项调查为研究人员和工程师提供了一个统一的框架，以推进上下文感知的AI。

Title: Comparing Apples to Oranges: A Dataset & Analysis of LLM Humour Understanding from Traditional Puns to Topical Jokes

• Authors: Tyler Loakman, William Thorne, Chenghua Lin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2507.13335
• Pdf URL: https://arxiv.org/pdf/2507.13335
• Copy Paste: [[2507.13335]] Comparing Apples to Oranges: A Dataset & Analysis of LLM Humour Understanding from Traditional Puns to Topical Jokes(https://arxiv.org/abs/2507.13335)
• Keywords: language model, llm
• Abstract: Humour, as a complex language form, is derived from myriad aspects of life, whilst existing work on computational humour has focussed almost exclusively on short pun-based jokes. In this work, we investigate whether the ability of Large Language Models (LLMs) to explain humour depends on the particular humour form. We compare models on simple puns and more complex topical humour that requires knowledge of real-world entities and events. In doing so, we curate a dataset of 600 jokes split across 4 joke types and manually write high-quality explanations. These jokes include heterographic and homographic puns, contemporary internet humour, and topical jokes, where understanding relies on reasoning beyond "common sense", rooted instead in world knowledge regarding news events and pop culture. Using this dataset, we compare the zero-shot abilities of a range of LLMs to accurately and comprehensively explain jokes of different types, identifying key research gaps in the task of humour explanation. We find that none of the tested models (inc. reasoning models) are capable of reliably generating adequate explanations of all joke types, further highlighting the narrow focus of most works in computational humour on overly simple joke forms.
• 摘要：作为一种复杂的语言形式，幽默源于生活的无数方面，而现有的计算幽默作品几乎只专注于基于简短的笑话。在这项工作中，我们研究了大语模型（LLM）解释幽默的能力是否取决于特定的幽默形式。我们比较了简单的双关语和更复杂的主题幽默的模型，这些幽默需要了解现实世界实体和事件。在这样做的过程中，我们策划了一个在4种笑话类型中分裂的600个笑话的数据集，并手动编写高质量的解释。这些笑话包括异志和同型双关语，当代互联网幽默和局部笑话，理解依赖于超越“常识”的推理，而是植根于世界知识的新闻事件和流行文化的知识。使用此数据集，我们比较了一系列LLM的零射击能力，以准确，全面地解释不同类型的笑话，从而确定幽默解释任务中的关键研究差距。我们发现，经过测试的模型（Inc。推理模型）都无法可靠地产生所有笑话类型的充分解释，进一步强调了大多数作品在计算幽默上的狭窄焦点，这是对过于简单的笑话形式的。