2025-06-19

Title: Combining Constrained and Unconstrained Decoding via Boosting: BoostCD and Its Application to Information Extraction

• Authors: Marija Šakota, Robert West
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2506.14901
• Pdf URL: https://arxiv.org/pdf/2506.14901
• Copy Paste: [[2506.14901]] Combining Constrained and Unconstrained Decoding via Boosting: BoostCD and Its Application to Information Extraction(https://arxiv.org/abs/2506.14901)
• Keywords: language model
• Abstract: Many recent approaches to structured NLP tasks use an autoregressive language model $M$ to map unstructured input text $x$ to output text $y$ representing structured objects (such as tuples, lists, trees, code, etc.), where the desired output structure is enforced via constrained decoding. During training, these approaches do not require the model to be aware of the constraints, which are merely implicit in the training outputs $y$. This is advantageous as it allows for dynamic constraints without requiring retraining, but can lead to low-quality output during constrained decoding at test time. We overcome this problem with Boosted Constrained Decoding (BoostCD), which combines constrained and unconstrained decoding in two phases: Phase 1 decodes from the base model $M$ twice, in constrained and unconstrained mode, obtaining two weak predictions. In phase 2, a learned autoregressive boosted model combines the two weak predictions into one final prediction. The mistakes made by the base model with vs. without constraints tend to be complementary, which the boosted model learns to exploit for improved performance. We demonstrate the power of BoostCD by applying it to closed information extraction. Our model, BoostIE, outperforms prior approaches both in and out of distribution, addressing several common errors identified in those approaches.
• 摘要：结构化NLP任务的许多最新方法都使用自回归语言模型$ M $映射非结构化输入$ x $以输出文本$ y $代表结构化对象（例如元组，列表，列表，树，代码等），其中所需的输出结构是通过约束解码来强制执行的。在培训期间，这些方法不需要模型意识到限制，这些约束仅在培训输出$ y $中隐含。这是有利的，因为它允许动态约束而无需重新训练，但在测试时间约束解码过程中可能会导致低质量输出。我们通过增强的约束解码（BOOSTCD）克服了这个问题，该解码结合了两个阶段的约束和不受约束的解码：从基本模型$ m $ M $两次的第1阶段解码，以约束和不受约束的模式获得两个弱预测。在第2阶段，学习的自回旋增强模型将两个弱预测结合在一起，成为一个最终预测。基本模型与没有约束的基本模型犯下的错误往往是互补的，而增强的模型学会了利用以提高性能。我们通过将其应用于封闭信息提取来证明BOOSTCD的力量。我们的模型Boostie在分布中的进展均优于先验，以解决这些方法中确定的几个常见错误。

Title: CrEst: Credibility Estimation for Contexts in LLMs via Weak Supervision

• Authors: Dyah Adila, Shuai Zhang, Boran Han, Bonan Min, Yuyang Wang
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2506.14912
• Pdf URL: https://arxiv.org/pdf/2506.14912
• Copy Paste: [[2506.14912]] CrEst: Credibility Estimation for Contexts in LLMs via Weak Supervision(https://arxiv.org/abs/2506.14912)
• Keywords: language model, llm
• Abstract: The integration of contextual information has significantly enhanced the performance of large language models (LLMs) on knowledge-intensive tasks. However, existing methods often overlook a critical challenge: the credibility of context documents can vary widely, potentially leading to the propagation of unreliable information. In this paper, we introduce CrEst, a novel weakly supervised framework for assessing the credibility of context documents during LLM inference--without requiring manual annotations. Our approach is grounded in the insight that credible documents tend to exhibit higher semantic coherence with other credible documents, enabling automated credibility estimation through inter-document agreement. To incorporate credibility into LLM inference, we propose two integration strategies: a black-box approach for models without access to internal weights or activations, and a white-box method that directly modifies attention mechanisms. Extensive experiments across three model architectures and five datasets demonstrate that CrEst consistently outperforms strong baselines, achieving up to a 26.86% improvement in accuracy and a 3.49% increase in F1 score. Further analysis shows that CrEst maintains robust performance even under high-noise conditions.
• 摘要：上下文信息的集成显着提高了大语模型（LLM）在知识密集型任务上的性能。但是，现有方法通常会忽略一个关键的挑战：上下文文档的可信度可能会差异很大，可能导致不可靠的信息传播。在本文中，我们介绍了克雷斯特（Crest），这是一个新颖的弱监督框架，用于评估LLM推论期间上下文文档的信誉 - 不需要手动注释。我们的方法基于一个见解，即可靠的文件倾向于与其他可信文档表现出较高的语义连贯性，从而通过文档间协议实现自动信誉估算。为了将信誉纳入LLM推断中，我们提出了两种集成策略：用于模型的黑框方法，无需访问内部权重或激活，以及一种直接修改注意力机制的白盒方法。在三个模型架构和五个数据集上进行的广泛实验表明，波峰始终超过强质基础，其准确性提高了26.86％，而F1得分提高了3.49％。进一步的分析表明，即使在高噪声条件下，Crest也能保持强劲的性能。

Title: MDBench: A Synthetic Multi-Document Reasoning Benchmark Generated with Knowledge Guidance

• Authors: Joseph J. Peper, Wenzhao Qiu, Ali Payani, Lu Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2506.14927
• Pdf URL: https://arxiv.org/pdf/2506.14927
• Copy Paste: [[2506.14927]] MDBench: A Synthetic Multi-Document Reasoning Benchmark Generated with Knowledge Guidance(https://arxiv.org/abs/2506.14927)
• Keywords: llm, prompt
• Abstract: Natural language processing evaluation has made significant progress, largely driven by the proliferation of powerful large language mod-els (LLMs). New evaluation benchmarks are of increasing priority as the reasoning capabilities of LLMs are expanding at a rapid pace. In particular, while multi-document (MD) reasoning is an area of extreme relevance given LLM capabilities in handling longer-context inputs, few benchmarks exist to rigorously examine model behavior in this setting. Moreover, the multi-document setting is historically challenging for benchmark creation due to the expensive cost of annotating long inputs. In this work, we introduce MDBench, a new dataset for evaluating LLMs on the task of multi-document reasoning. Notably, MDBench is created through a novel synthetic generation process, allowing us to controllably and efficiently generate challenging document sets and the corresponding question-answer (QA) examples. Our novel technique operates on condensed structured seed knowledge, modifying it through LLM-assisted edits to induce MD-specific reasoning challenges. We then convert this structured knowledge into a natural text surface form, generating a document set and corresponding QA example. We analyze the behavior of popular LLMs and prompting techniques, finding that MDBENCH poses significant challenges for all methods, even with relatively short document sets. We also see our knowledge-guided generation technique (1) allows us to readily perform targeted analysis of MD-specific reasoning capabilities and (2) can be adapted quickly to account for new challenges and future modeling improvements.
• 摘要：自然语言处理评估已取得了重大进展，这在很大程度上是由强大的大型语言Mod-els（LLM）的扩散驱动的。随着LLMS的推理能力正在以快速的速度扩展，新的评估基准将提高优先级。特别是，鉴于LLM在处理较长的文本输入方面的LLM功能鉴于LLM功能，多文档（MD）推理是一个极端相关的领域，但在这种情况下，很少有基准测试能严格检查模型行为。此外，由于长期输入的昂贵成本，多文件设置在历史上对于基准创建而言是具有挑战性的。在这项工作中，我们介绍了MDBench，这是一个新数据集，用于评估LLMS关于多文件推理的任务。值得注意的是，MDBENCH是通过新型的合成生成过程创建的，使我们能够可控，有效地生成具有挑战性的文档集和相应的问题解答（QA）示例。我们的新技术以凝结的结构化种子知识运行，通过LLM辅助编辑对其进行修改，以引起MD特定的推理挑战。然后，我们将这些结构化知识转换为自然文本表面形式，生成文档集和相应的QA示例。我们分析了流行的LLM和提示技术的行为，发现MDBench即使使用相对较短的文档集也对所有方法构成了重大挑战。我们还看到我们的知识引导的生成技术（1）使我们能够轻松地对MD特定推理能力进行有针对性的分析，并且（2）可以迅速调整以解决新的挑战和未来的建模改进。

Title: From Chat to Checkup: Can Large Language Models Assist in Diabetes Prediction?

• Authors: Shadman Sakib, Oishy Fatema Akhand, Ajwad Abrar
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2506.14949
• Pdf URL: https://arxiv.org/pdf/2506.14949
• Copy Paste: [[2506.14949]] From Chat to Checkup: Can Large Language Models Assist in Diabetes Prediction?(https://arxiv.org/abs/2506.14949)
• Keywords: language model, gpt, llm, prompt, chat
• Abstract: While Machine Learning (ML) and Deep Learning (DL) models have been widely used for diabetes prediction, the use of Large Language Models (LLMs) for structured numerical data is still not well explored. In this study, we test the effectiveness of LLMs in predicting diabetes using zero-shot, one-shot, and three-shot prompting methods. We conduct an empirical analysis using the Pima Indian Diabetes Database (PIDD). We evaluate six LLMs, including four open-source models: Gemma-2-27B, Mistral-7B, Llama-3.1-8B, and Llama-3.2-2B. We also test two proprietary models: GPT-4o and Gemini Flash 2.0. In addition, we compare their performance with three traditional machine learning models: Random Forest, Logistic Regression, and Support Vector Machine (SVM). We use accuracy, precision, recall, and F1-score as evaluation metrics. Our results show that proprietary LLMs perform better than open-source ones, with GPT-4o and Gemma-2-27B achieving the highest accuracy in few-shot settings. Notably, Gemma-2-27B also outperforms the traditional ML models in terms of F1-score. However, there are still issues such as performance variation across prompting strategies and the need for domain-specific fine-tuning. This study shows that LLMs can be useful for medical prediction tasks and encourages future work on prompt engineering and hybrid approaches to improve healthcare predictions.
• 摘要：尽管机器学习（ML）和深度学习（DL）模型已被广泛用于糖尿病预测，但仍未很好地探索大型语言模型（LLMS）用于结构化数值数据的方法。在这项研究中，我们测试了LLM在使用零射，一次性和三弹性提示方法预测糖尿病方面的有效性。我们使用PIMA印度糖尿病数据库（PIDD）进行了经验分析。我们评估了六个LLM，包括四种开源型号：Gemma-2-27B，Mistral-7b，Llama-3.1-8B和Llama-3.2-2B。我们还测试了两个专有模型：GPT-4O和Gemini Flash 2.0。此外，我们将它们的性能与三种传统的机器学习模型进行比较：随机森林，逻辑回归和支持向量机（SVM）。我们将准确性，精度，召回和F1得分作为评估指标。我们的结果表明，专有LLM的性能要比开源元来表现更好，而GPT-4O和GEMMA-2-27B在少数拍摄设置中的精度最高。值得注意的是，Gemma-2-27b在F1得分方面还优于传统的ML模型。但是，仍然存在一些问题，例如促进策略的性能变化以及对域特异性微调的需求。这项研究表明，LLM可用于医疗预测任务，并鼓励未来的工程和混合方法来改善医疗保健预测。

Title: Memory Tokens: Large Language Models Can Generate Reversible Sentence Embeddings

• Authors: Ignacio Sastre, Aiala Rosá
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2506.15001
• Pdf URL: https://arxiv.org/pdf/2506.15001
• Copy Paste: [[2506.15001]] Memory Tokens: Large Language Models Can Generate Reversible Sentence Embeddings(https://arxiv.org/abs/2506.15001)
• Keywords: language model, llm, prompt
• Abstract: In this work, we observe an interesting phenomenon: it is possible to generate reversible sentence embeddings that allow an LLM to reconstruct the original text exactly, without modifying the model's weights. This is achieved by introducing a special memory token, whose embedding is optimized through training on a fixed sequence. When prompted with this embedding, the model reconstructs the fixed sequence exactly. We evaluate this phenomenon across English and Spanish datasets, sequences of up to approximately 240 tokens, and model scales ranging from 100M to 8B parameters. Notably, Llama 3.1 8B successfully reconstructs all tested sequences. Our findings highlight an interesting capability of LLMs and suggest potential applications in memory-based retrieval, compression, and controlled text generation.
• 摘要：在这项工作中，我们观察到一个有趣的现象：可以生成可逆句子嵌入，使LLM可以精确地重建原始文本，而无需修改模型的权重。这是通过引入特殊的内存令牌来实现的，其嵌入通过固定序列的训练进行了优化。当提示使用此嵌入时，该模型会精确地重建固定序列。我们在英语和西班牙数据集中评估了这种现象，大约240个令牌的序列以及范围从100m到8b参数的型号。值得注意的是，美洲驼3.1 8b成功地重建了所有测试的序列。我们的发现突出了LLM的有趣功能，并提出了在基于内存的检索，压缩和受控文本生成中的潜在应用。

Title: Semantically-Aware Rewards for Open-Ended R1 Training in Free-Form Generation

• Authors: Zongxia Li, Yapei Chang, Yuhang Zhou, Xiyang Wu, Zichao Liang, Yoo Yeon Sung, Jordan Lee Boyd-Graber
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2506.15068
• Pdf URL: https://arxiv.org/pdf/2506.15068
• Copy Paste: [[2506.15068]] Semantically-Aware Rewards for Open-Ended R1 Training in Free-Form Generation(https://arxiv.org/abs/2506.15068)
• Keywords: llm
• Abstract: Evaluating open-ended long-form generation is challenging because it is hard to define what clearly separates good from bad outputs. Existing methods often miss key aspects like coherence, style, or relevance, or are biased by pretraining data, making open-ended long-form evaluation an underexplored problem. To address this gap, we propose PrefBERT, a scoring model for evaluating open-ended long-form generation in GRPO and guiding its training with distinct rewards for good and bad outputs. Trained on two response evaluation datasets with diverse long-form styles and Likert-rated quality, PrefBERT effectively supports GRPO by offering better semantic reward feedback than traditional metrics ROUGE-L and BERTScore do. Through comprehensive evaluations, including LLM-as-a-judge, human ratings, and qualitative analysis, we show that PrefBERT, trained on multi-sentence and paragraph-length responses, remains reliable across varied long passages and aligns well with the verifiable rewards GRPO needs. Human evaluations confirm that using PrefBERT as the reward signal to train policy models yields responses better aligned with human preferences than those trained with traditional metrics. Our code is available at this https URL.
• 摘要：评估开放式的长期生成是具有挑战性的，因为很难定义明显与不良输出分开的东西。现有的方法通常会错过关键，样式或相关性等关键方面，或者因数据预处理而偏见，从而使开放式的长期评估成为一个不受欢迎的问题。为了解决这一差距，我们提出了Prefbert，这是一个评分模型，用于评估GRPO中的开放式长期生成，并以良好和不良产出的不同奖励指导其培训。 Prefbert接受了两种具有多种长期样式和李克特率质量的响应评估数据集的培训，通过提供比传统的指标Rouge-L和BertScore提供更好的语义奖励反馈，从而有效地支持GRPO。通过全面的评估，包括LLM-AS-A-A-Gudge，人类评级和定性分析，我们表明，经过多句子和段落长度响应培训的Prefbert在不同的长期段落中仍然可靠，并且与可靠性的可靠性重新恢复需求保持一致。人类评估证实，将prefbert作为奖励信号训练政策模型产生的反应比接受传统指标的人的偏好更好。我们的代码可在此HTTPS URL上找到。

Title: Learning-Time Encoding Shapes Unlearning in LLMs

• Authors: Ruihan Wu, Konstantin Garov, Kamalika Chaudhuri
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2506.15076
• Pdf URL: https://arxiv.org/pdf/2506.15076
• Copy Paste: [[2506.15076]] Learning-Time Encoding Shapes Unlearning in LLMs(https://arxiv.org/abs/2506.15076)
• Keywords: language model, llm
• Abstract: As large language models (LLMs) are increasingly deployed in the real world, the ability to ``unlearn'', or remove specific pieces of knowledge post hoc, has become essential for a variety of reasons ranging from privacy regulations to correcting outdated or harmful content. Prior work has proposed unlearning benchmarks and algorithms, and has typically assumed that the training process and the target model are fixed. In this work, we empirically investigate how learning-time choices in knowledge encoding impact the effectiveness of unlearning factual knowledge. Our experiments reveal two key findings: (1) learning with paraphrased descriptions improves unlearning performance and (2) unlearning individual piece of knowledge from a chunk of text is challenging. Our results suggest that learning-time knowledge encoding may play a central role in enabling reliable post-hoc unlearning.
• 摘要：由于大型语言模型（LLM）越来越多地在现实世界中部署，因此``''''''或删除特定知识的能力已成为事后事后的特定知识，这是由于从隐私法规到纠正过时或有害内容的各种原因而变得至关重要的。先前的工作提出了学习基准和算法，并且通常假设训练过程和目标模型是固定的。在这项工作中，我们从经验上研究了编码知识中的学习时间选择如何影响学习事实知识的有效性。我们的实验揭示了两个关键的发现：（1）用释义的描述进行学习改善了学习的表现，（2）（2）从一部分文本中学习单个知识是具有挑战性的。我们的结果表明，学习时间知识编码可能在实现可靠的事后学习中起着核心作用。

Title: CKD-EHR:Clinical Knowledge Distillation for Electronic Health Records

• Authors: Junke Wang, Hongshun Ling, Li Zhang, Longqian Zhang, Fang Wang, Yuan Gao, Zhi Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2506.15118
• Pdf URL: https://arxiv.org/pdf/2506.15118
• Copy Paste: [[2506.15118]] CKD-EHR:Clinical Knowledge Distillation for Electronic Health Records(https://arxiv.org/abs/2506.15118)
• Keywords: language model
• Abstract: Electronic Health Records (EHR)-based disease prediction models have demonstrated significant clinical value in promoting precision medicine and enabling early intervention. However, existing large language models face two major challenges: insufficient representation of medical knowledge and low efficiency in clinical deployment. To address these challenges, this study proposes the CKD-EHR (Clinical Knowledge Distillation for EHR) framework, which achieves efficient and accurate disease risk prediction through knowledge distillation techniques. Specifically, the large language model Qwen2.5-7B is first fine-tuned on medical knowledge-enhanced data to serve as the teacher this http URL then generates interpretable soft labels through a multi-granularity attention distillation mechanism. Finally, the distilled knowledge is transferred to a lightweight BERT student model. Experimental results show that on the MIMIC-III dataset, CKD-EHR significantly outperforms the baseline model:diagnostic accuracy is increased by 9%, F1-score is improved by 27%, and a 22.2 times inference speedup is achieved. This innovative solution not only greatly improves resource utilization efficiency but also significantly enhances the accuracy and timeliness of diagnosis, providing a practical technical approach for resource optimization in clinical settings. The code and data for this research are available athttps://github.com/209506702/CKD_EHR.
• 摘要：基于电子健康记录（EHR）的疾病预测模型在促进精确医学和实现早期干预方面表现出显着的临床价值。但是，现有的大型语言模型面临两个主要挑战：医学知识的表示不足和临床部署效率低。为了应对这些挑战，本研究提出了CKD-EHR（EHR的临床知识蒸馏）框架，该框架通过知识蒸馏技术实现了有效而准确的疾病风险预测。具体而言，大语模型QWEN2.5-7B首先在医学知识增强的数据上进行微调，以作为老师作为老师，然后通过多种散发性注意力蒸馏机制生成可解释的软标签。最后，将蒸馏知识转移到轻量级的BERT学生模型中。实验结果表明，在模拟III数据集上，CKD-EHR显着胜过基线模型：诊断准确性提高了9％，F1得分提高了27％，并实现了22.2倍的推理加速。这种创新的解决方案不仅可以极大地提高资源利用效率，而且可以显着提高诊断的准确性和及时性，从而为临床环境中的资源优化提供了实用的技术方法。这项研究的代码和数据可用于athttps：//github.com/209506702/ckd_ehr。

Title: Modeling the One-to-Many Property in Open-Domain Dialogue with LLMs

• Authors: Jing Yang Lee, Kong-Aik Lee, Woon-Seng Gan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2506.15131
• Pdf URL: https://arxiv.org/pdf/2506.15131
• Copy Paste: [[2506.15131]] Modeling the One-to-Many Property in Open-Domain Dialogue with LLMs(https://arxiv.org/abs/2506.15131)
• Keywords: llm, agent
• Abstract: Open-domain Dialogue (OD) exhibits a one-to-many (o2m) property, whereby multiple appropriate responses exist for a single dialogue context. Despite prior research showing that modeling this property boosts response diversity, most modern LLM-based dialogue agents do not explicitly do so. In this work, we model the o2m property of OD in LLMs by decomposing OD generation into two key tasks: Multi-Response Generation (MRG) and Preference-based Selection (PS), which entail generating a set of n semantically and lexically diverse high-quality responses for a given dialogue context, followed by selecting a single response based on human preference, respectively. To facilitate MRG and PS, we introduce o2mDial, a dialogue corpus explicitly designed to capture the o2m property by featuring multiple plausible responses for each context. Leveraging o2mDial, we propose new in-context learning and instruction-tuning strategies, as well as novel evaluation metrics for MRG, alongside a model-based approach for PS. Empirical results demonstrate that applying the proposed two-stage framework to smaller LLMs for OD generation enhances overall response diversity while maintaining contextual coherence, improving response quality by up to 90%, bringing them closer to the performance of larger models.
• 摘要：开放域对话（OD）具有一对多（O2M）的属性，从而在单个对话上下文中存在多个适当的响应。尽管先前的研究表明，建模这种属性会增强响应多样性，但大多数基于LLM的对话代理并未明确地这样做。在这项工作中，我们通过将OD生成分解为两个关键任务：多反应生成（MRG）和基于首选项的选择（PS），它需要将O2的O2M属性（PS）分解为两个关键任务，这些选择需要生成一组N语义和词汇多样性的高质量响应，以分别选择基于人类的preeference，以分别选择一个给定的对话上下文。为了促进MRG和PS，我们介绍了O2Mdial，这是一种对话语料库，旨在通过在每个上下文中介绍多个合理的响应来捕获O2M属性。利用O2Mdial，我们提出了新的文化学习和指导策略，以及MRG的新评估指标，以及基于PS的模型方法。经验结果表明，将提议的两阶段框架应用于较小的LLMS以创造OD生成增强了整体响应多样性，同时保持上下文连贯性，将响应质量提高了多达90％，使它们更接近较大模型的性能。

Title: Thunder-Tok: Minimizing Tokens per Word in Tokenizing Korean Texts for Generative Language Models

• Authors: Gyeongje Cho, Yeonkyoun So, Chanwoo Park, Sangmin Lee, Sungmok Jung, Jaejin Lee
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2506.15138
• Pdf URL: https://arxiv.org/pdf/2506.15138
• Copy Paste: [[2506.15138]] Thunder-Tok: Minimizing Tokens per Word in Tokenizing Korean Texts for Generative Language Models(https://arxiv.org/abs/2506.15138)
• Keywords: language model
• Abstract: This paper introduces Thunder-Tok, a new Korean tokenizer designed to reduce token fertility without compromising model performance. Our approach uses a rule-based pre-tokenization method that aligns with the linguistic structure of the Korean language. We also create a seed vocabulary containing tokens that resemble linguistic units and employ a branching entropy-based selection algorithm. These techniques increase the average token length, thus lowering fertility while preserving linguistic information. Experimental results indicate that Thunder-Tok reduces fertility by approximately 10% (i.e., reduces the number of tokens by 10%, improving the inference speed by 10%) compared to BPE without compromising performance across various downstream tasks. These findings demonstrate that our linguistically informed approach is effective and practical for designing efficient tokenizers for language models.
• 摘要：本文介绍了Thunder-Tok，这是一种新的韩国代币剂，旨在在不损害模型性能的情况下降低令牌生育能力。我们的方法使用一种基于规则的预言式方法，该方法与朝鲜语的语言结构保持一致。我们还创建了一个包含类似语言单元并采用基于分支熵的选择算法的代币的种子词汇。这些技术增加了平均令牌长度，从而降低了生育能力，同时保留语言信息。实验结果表明，与BPE相比，Thunder-Tok将生育率降低了约10％（即，将令牌数量降低了10％，将推理速度降低了10％），而没有损害各种下游任务的BPE。这些发现表明，我们的语言知情方法对于为语言模型设计有效的引导者是有效且实用的。

Title: Emergence of Primacy and Recency Effect in Mamba: A Mechanistic Point of View

• Authors: Muhammad Cendekia Airlangga, Hilal AlQuabeh, Munachiso S Nwadike, Kentaro Inui
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2506.15156
• Pdf URL: https://arxiv.org/pdf/2506.15156
• Copy Paste: [[2506.15156]] Emergence of Primacy and Recency Effect in Mamba: A Mechanistic Point of View(https://arxiv.org/abs/2506.15156)
• Keywords: language model
• Abstract: We study memory in state-space language models using primacy and recency effects as behavioral tools to uncover how information is retained and forgotten over time. Applying structured recall tasks to the Mamba architecture, we observe a consistent U-shaped accuracy profile, indicating strong performance at the beginning and end of input sequences. We identify three mechanisms that give rise to this pattern. First, long-term memory is supported by a sparse subset of channels within the model's selective state space block, which persistently encode early input tokens and are causally linked to primacy effects. Second, short-term memory is governed by delta-modulated recurrence: recent inputs receive more weight due to exponential decay, but this recency advantage collapses when distractor items are introduced, revealing a clear limit to memory depth. Third, we find that memory allocation is dynamically modulated by semantic regularity: repeated relations in the input sequence shift the delta gating behavior, increasing the tendency to forget intermediate items. We validate these findings via targeted ablations and input perturbations on two large-scale Mamba-based language models: one with 1.4B and another with 7B parameters.
• 摘要：我们使用首要效果和重新效果作为行为工具来研究状态空间语言模型的记忆，以发现信息随着时间的流逝如何保留和遗忘。将结构化的召回任务应用于Mamba体系结构，我们观察到一个一致的U形精度概况，表明在输入序列的开始和结束时表现出色。我们确定产生这种模式的三种机制。首先，长期内存由模型选择性状态空间块中的通道的稀疏子集支持，该块持续编码早期输入令牌，并与初选效应有因果关系。其次，短期内存由Delta调制的复发控制：最近的输入由于指数衰减而获得更大的权重，但是当引入分心项项目时，这种新近的优势崩溃了，揭示了对记忆深度的明显限制。第三，我们发现记忆分配是通过语义规则性动态调节的：输入序列中的重复关系移动了Delta门控行为，从而增加了忘记中间项目的趋势。我们通过针对两个基于MAMBA的大规模语言模型的目标消融和输入扰动来验证这些发现：一种具有1.4B，另一个具有7B参数。

Title: A Comparative Study of Task Adaptation Techniques of Large Language Models for Identifying Sustainable Development Goals

• Authors: Andrea Cadeddu, Alessandro Chessa, Vincenzo De Leo, Gianni Fenu, Enrico Motta, Francesco Osborne, Diego Reforgiato Recupero, Angelo Salatino, Luca Secchi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2506.15208
• Pdf URL: https://arxiv.org/pdf/2506.15208
• Copy Paste: [[2506.15208]] A Comparative Study of Task Adaptation Techniques of Large Language Models for Identifying Sustainable Development Goals(https://arxiv.org/abs/2506.15208)
• Keywords: language model, gpt, llm, prompt
• Abstract: In 2012, the United Nations introduced 17 Sustainable Development Goals (SDGs) aimed at creating a more sustainable and improved future by 2030. However, tracking progress toward these goals is difficult because of the extensive scale and complexity of the data involved. Text classification models have become vital tools in this area, automating the analysis of vast amounts of text from a variety of sources. Additionally, large language models (LLMs) have recently proven indispensable for many natural language processing tasks, including text classification, thanks to their ability to recognize complex linguistic patterns and semantics. This study analyzes various proprietary and open-source LLMs for a single-label, multi-class text classification task focused on the SDGs. Then, it also evaluates the effectiveness of task adaptation techniques (i.e., in-context learning approaches), namely Zero-Shot and Few-Shot Learning, as well as Fine-Tuning within this domain. The results reveal that smaller models, when optimized through prompt engineering, can perform on par with larger models like OpenAI's GPT (Generative Pre-trained Transformer).
• 摘要：2012年，联合国引入了17个可持续发展目标（SDG），旨在到2030年创造更可持续和改善的未来。但是，由于涉及的数据的规模广泛和复杂性，因此很难跟踪实现这些目标的进度。文本分类模型已成为该领域的重要工具，可以使各种来源的大量文本分析自动化。此外，由于其能够识别复杂的语言模式和语义的能力，大型语言模型（LLMS）最近已证明了许多自然语言处理任务（包括文本分类）必不可少的。这项研究分析了各种专有和开源的LLM，用于针对可持续发展目标的单标签，多级文本分类任务。然后，它还评估了任务适应技术的有效性（即，在内部学习方法），即零射击和少量学习以及在该领域内进行微调。结果表明，较小的模型在通过及时工程进行优化时，可以与OpenAI的GPT（生成预训练的变压器）等较大型号进行表现。

Title: ProtoReasoning: Prototypes as the Foundation for Generalizable Reasoning in LLMs

• Authors: Feng He, Zijun Chen, Xinnian Liang, Tingting Ma, Yunqi Qiu, Shuangzhi Wu, Junchi Yan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2506.15211
• Pdf URL: https://arxiv.org/pdf/2506.15211
• Copy Paste: [[2506.15211]] ProtoReasoning: Prototypes as the Foundation for Generalizable Reasoning in LLMs(https://arxiv.org/abs/2506.15211)
• Keywords: language model, llm, chain-of-thought
• Abstract: Recent advances in Large Reasoning Models (LRMs) trained with Long Chain-of-Thought (Long CoT) reasoning have demonstrated remarkable cross-domain generalization capabilities. However, the underlying mechanisms supporting such transfer remain poorly understood. We hypothesize that cross-domain generalization arises from shared abstract reasoning prototypes -- fundamental reasoning patterns that capture the essence of problems across domains. These prototypes minimize the nuances of the representation, revealing that seemingly diverse tasks are grounded in shared reasoning this http URL on this hypothesis, we propose ProtoReasoning, a framework that enhances the reasoning ability of LLMs by leveraging scalable and verifiable prototypical representations (Prolog for logical reasoning, PDDL for planning).ProtoReasoning features: (1) an automated prototype construction pipeline that transforms problems into corresponding prototype representations; (2) a comprehensive verification system providing reliable feedback through Prolog/PDDL interpreters; (3) the scalability to synthesize problems arbitrarily within prototype space while ensuring correctness. Extensive experiments show that ProtoReasoning achieves 4.7% improvement over baseline models on logical reasoning (Enigmata-Eval), 6.3% improvement on planning tasks, 4.0% improvement on general reasoning (MMLU) and 1.0% on mathematics (AIME24). Significantly, our ablation studies confirm that learning in prototype space also demonstrates enhanced generalization to structurally similar problems compared to training solely on natural language representations, validating our hypothesis that reasoning prototypes serve as the foundation for generalizable reasoning in large language models.
• 摘要：在接受长期思考（长床）推理训练的大型推理模型（LRMS）中的最新进展表现出显着的跨域泛化能力。但是，支持这种转移的基本机制仍然很少理解。我们假设跨域概括是由共同的抽象推理原型引起的 - 捕获跨领域问题本质的基本推理模式。这些原型最大程度地减少了表示形式的细微差别，揭示了看似多样化的任务是基于共享此假设上的HTTP URL的共同推理，我们提出了蛋白质的质量调查，这是一个框架，通过利用可缩放和可靠性的原型表达来增强LLMS的推理能力（prolog and for for logolog and for for for log sandiqual sneptip and protipe for progiality sanep and sanep saperaper：precaim sequear，pdd l dd llotection e。将问题转化为相应的原型表示的施工管道； （2）通过Prolog/PDDL口译员提供可靠的反馈； （3）在原型空间内任意合成问题的可伸缩性，同时确保正确性。广泛的实验表明，在逻辑推理（ENIGMATA-EVAL）上，蛋白质的提高了4.7％，计划任务提高了6.3％，一般推理提高了4.0％的一般推理（MMLU）和数学上的1.0％（AIME24）。值得注意的是，我们的消融研究证实，与仅根据自然语言表示的培训相比，原型空间中的学习也表明了对结构相似问题的概括，这证实了我们的假设，即推理原型是大语模型中可推广推理的基础。

Title: MinosEval: Distinguishing Factoid and Non-Factoid for Tailored Open-Ended QA Evaluation with LLMs

• Authors: Yongqi Fan, Yating Wang, Guandong Wang, Jie Zhai, Jingping Liu, Qi Ye, Tong Ruan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2506.15215
• Pdf URL: https://arxiv.org/pdf/2506.15215
• Copy Paste: [[2506.15215]] MinosEval: Distinguishing Factoid and Non-Factoid for Tailored Open-Ended QA Evaluation with LLMs(https://arxiv.org/abs/2506.15215)
• Keywords: language model, llm
• Abstract: Open-ended question answering (QA) is a key task for evaluating the capabilities of large language models (LLMs). Compared to closed-ended QA, it demands longer answer statements, more nuanced reasoning processes, and diverse expressions, making refined and interpretable automatic evaluation both crucial and challenging. Traditional metrics like ROUGE and BERTScore struggle to capture semantic similarities due to different patterns between model responses and reference answers. Current LLM-based evaluation approaches, such as pairwise or listwise comparisons of candidate answers, lack intuitive interpretability. While pointwise scoring of each response provides some descriptions, it fails to adapt across different question contents. Most notably, existing methods overlook the distinction between factoid and non-factoid questions. To address these challenges, we propose \textbf{MinosEval}, a novel evaluation method that first distinguishes open-ended questions and then ranks candidate answers using different evaluation strategies. For factoid questions, it applies an adaptive key-point scoring strategy, while for non-factoid questions, it uses an instance-aware listwise ranking strategy. Experiments on multiple open-ended QA datasets, including self-built ones with more candidate responses to complement community resources, show that MinosEval better aligns with human annotations and offers more interpretable results.
• 摘要：开放式问题回答（QA）是评估大语言模型（LLMS）功能的关键任务。与封闭式质量保证相比，它需要更长的答案陈述，更细微的推理过程和各种表达式，从而使精致且可解释的自动评估既重要又具有挑战性。像鲁日（Rouge）和伯特西尔（Bertscore）这样的传统指标由于模型响应和参考答案之间的不同模式而难以捕获语义相似性。当前基于LLM的评估方法，例如对候选答案的成对或列表的比较，缺乏直观的解释性。虽然每个响应的刻度评分提供了一些描述，但它无法在不同的问题内容中适应。最值得注意的是，现有方法忽略了Factoid和非事实问题之间的区别。为了应对这些挑战，我们提出\ textbf {Minoseval}，这是一种新颖的评估方法，首先区分开放式问题，然后使用不同的评估策略对候选人答案进行排名。对于Factoid问题，它应用了自适应的关键点评分策略，而对于非事实问题，它使用了list-listwise排名策略。在多个开放式质量检查数据集上进行的实验，包括对社区资源进行更多候选响应的自行构建数据集的实验，表明矿物节可以更好地与人类注释保持一致，并提供更多可解释的结果。

Title: Lost in Variation? Evaluating NLI Performance in Basque and Spanish Geographical Variants

• Authors: Jaione Bengoetxea, Itziar Gonzalez-Dios, Rodrigo Agerri
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2506.15239
• Pdf URL: https://arxiv.org/pdf/2506.15239
• Copy Paste: [[2506.15239]] Lost in Variation? Evaluating NLI Performance in Basque and Spanish Geographical Variants(https://arxiv.org/abs/2506.15239)
• Keywords: language model, llm
• Abstract: In this paper, we evaluate the capacity of current language technologies to understand Basque and Spanish language varieties. We use Natural Language Inference (NLI) as a pivot task and introduce a novel, manually-curated parallel dataset in Basque and Spanish, along with their respective variants. Our empirical analysis of crosslingual and in-context learning experiments using encoder-only and decoder-based Large Language Models (LLMs) shows a performance drop when handling linguistic variation, especially in Basque. Error analysis suggests that this decline is not due to lexical overlap, but rather to the linguistic variation itself. Further ablation experiments indicate that encoder-only models particularly struggle with Western Basque, which aligns with linguistic theory that identifies peripheral dialects (e.g., Western) as more distant from the standard. All data and code are publicly available.
• 摘要：在本文中，我们评估了当前语言技术了解巴斯克语和西班牙语品种的能力。我们使用自然语言推断（NLI）作为一项枢纽任务，并在巴斯克和西班牙语中引入了一个新颖的，手动策划的并行数据集，以及它们各自的变体。我们对使用仅编码和解码器的大型语言模型（LLM）进行跨语言和文字学习实验的经验分析显示，处理语言变化时的性能下降，尤其是在巴斯克地区。错误分析表明，这种下降不是由于词汇重叠，而是由于语言变化本身。进一步的消融实验表明，仅共同模型与西部巴斯克地区特别挣扎，这与语言理论一致，该理论识别出外围方言（例如西方）与标准更遥远。所有数据和代码均可公开使用。

Title: Research on Graph-Retrieval Augmented Generation Based on Historical Text Knowledge Graphs

• Authors: Yang Fan, Zhang Qi, Xing Wenqian, Liu Chang, Liu Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2506.15241
• Pdf URL: https://arxiv.org/pdf/2506.15241
• Copy Paste: [[2506.15241]] Research on Graph-Retrieval Augmented Generation Based on Historical Text Knowledge Graphs(https://arxiv.org/abs/2506.15241)
• Keywords: language model, hallucination, prompt, retrieval augmented generation, retrieval-augmented generation, chain-of-thought
• Abstract: This article addresses domain knowledge gaps in general large language models for historical text analysis in the context of computational humanities and AIGC technology. We propose the Graph RAG framework, combining chain-of-thought prompting, self-instruction generation, and process supervision to create a The First Four Histories character relationship dataset with minimal manual annotation. This dataset supports automated historical knowledge extraction, reducing labor costs. In the graph-augmented generation phase, we introduce a collaborative mechanism between knowledge graphs and retrieval-augmented generation, improving the alignment of general models with historical knowledge. Experiments show that the domain-specific model Xunzi-Qwen1.5-14B, with Simplified Chinese input and chain-of-thought prompting, achieves optimal performance in relation extraction (F1 = 0.68). The DeepSeek model integrated with GraphRAG improves F1 by 11% (0.08-0.19) on the open-domain C-CLUE relation extraction dataset, surpassing the F1 value of Xunzi-Qwen1.5-14B (0.12), effectively alleviating hallucinations phenomenon, and improving interpretability. This framework offers a low-resource solution for classical text knowledge extraction, advancing historical knowledge services and humanities research.
• 摘要：本文介绍了在计算人文和AIGC技术的背景下的一般大语言模型中的域知识差距。我们提出了图形抹布框架，结合了经过思考链的提示，自我指导生成和过程监督，以创建前四个历史记录角色关系数据集，并使用最少的手动注释。该数据集支持自动化的历史知识提取，从而降低人工成本。在图形增强的一代阶段，我们介绍了知识图和检索型生成之间的协作机制，从而改善了一般模型与历史知识的一致性。实验表明，域特异性模型XUNZI-QWEN1.5-14B具有简化的中文输入和经过思考的提示，可在关系提取方面达到最佳性能（F1 = 0.68）。与GraphRag集成的DeepSeek模型在开放域C-C-Clue关系提取数据集上提高了F1（0.08-0.19），超过了XUNZI-QWEN1.5-14B（0.12）的F1值，从而有效地减轻了幻觉现象，并提高了解释性。该框架为经典文本知识提取提供了低资源解决方案，推进了历史知识服务和人文研究。

Title: TopClustRAG at SIGIR 2025 LiveRAG Challenge

• Authors: Juli Bakagianni, John Pavlopoulos, Aristidis Likas
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2506.15246
• Pdf URL: https://arxiv.org/pdf/2506.15246
• Copy Paste: [[2506.15246]] TopClustRAG at SIGIR 2025 LiveRAG Challenge(https://arxiv.org/abs/2506.15246)
• Keywords: language model, llm, prompt, retrieval-augmented generation
• Abstract: We present TopClustRAG, a retrieval-augmented generation (RAG) system developed for the LiveRAG Challenge, which evaluates end-to-end question answering over large-scale web corpora. Our system employs a hybrid retrieval strategy combining sparse and dense indices, followed by K-Means clustering to group semantically similar passages. Representative passages from each cluster are used to construct cluster-specific prompts for a large language model (LLM), generating intermediate answers that are filtered, reranked, and finally synthesized into a single, comprehensive response. This multi-stage pipeline enhances answer diversity, relevance, and faithfulness to retrieved evidence. Evaluated on the FineWeb Sample-10BT dataset, TopClustRAG ranked 2nd in faithfulness and 7th in correctness on the official leaderboard, demonstrating the effectiveness of clustering-based context filtering and prompt aggregation in large-scale RAG systems.
• 摘要：我们提出了TopClustrag，这是为Liverag挑战开发的检索型发电（RAG）系统，该系统评估了对大型Web Corpora回答的端到端问题。我们的系统采用混合检索策略，结合了稀疏和致密指数，然后将K-均值聚类与语义上类似的段落进行聚类。来自每个集群的代表性段落用于构建大型语言模型（LLM）的集群特异性提示，生成过滤，重新计算并最终合成的中间答案，并将其合成为单个全面的响应。这种多阶段的管道增强了回答多样性，相关性和忠诚的检索证据。在FineWeb Sample-10bt数据集上进行了评估，TopCluftag在忠诚中排名第二，在官方排行榜上排名第七，这证明了基于聚类的上下文过滤和及时在大型抹布系统中聚集的有效性。

Title: Cohort Discovery: A Survey on LLM-Assisted Clinical Trial Recruitment

• Authors: Shrestha Ghosh, Moritz Schneider, Carina Reinicke, Carsten Eickhoff
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2506.15301
• Pdf URL: https://arxiv.org/pdf/2506.15301
• Copy Paste: [[2506.15301]] Cohort Discovery: A Survey on LLM-Assisted Clinical Trial Recruitment(https://arxiv.org/abs/2506.15301)
• Keywords: llm
• Abstract: Recent advances in LLMs have greatly improved general-domain NLP tasks. Yet, their adoption in critical domains, such as clinical trial recruitment, remains limited. As trials are designed in natural language and patient data is represented as both structured and unstructured text, the task of matching trials and patients benefits from knowledge aggregation and reasoning abilities of LLMs. Classical approaches are trial-specific and LLMs with their ability to consolidate distributed knowledge hold the potential to build a more general solution. Yet recent applications of LLM-assisted methods rely on proprietary models and weak evaluation benchmarks. In this survey, we are the first to analyze the task of trial-patient matching and contextualize emerging LLM-based approaches in clinical trial recruitment. We critically examine existing benchmarks, approaches and evaluation frameworks, the challenges to adopting LLM technologies in clinical research and exciting future directions.
• 摘要：LLM的最新进展大大改进了通用域NLP任务。然而，它们在关键领域的采用，例如临床试验招募，仍然有限。由于试验是在自然语言中设计的，并且患者数据被表示为结构化和非结构化文本，因此匹配试验和患者的任务受益于知识汇总和LLM的推理能力。经典方法是特定于试验的方法，LLM具有合并分布式知识的能力，具有建立更通用的解决方案的潜力。然而，LLM辅助方法的最新应用取决于专有模型和弱评估基准。在这项调查中，我们是第一个分析试验患者匹配的任务，并在临床试验招聘中采用基于LLM的新兴方法。我们批判性地检查了现有的基准，方法和评估框架，在临床研究中采用LLM技术的挑战以及令人兴奋的未来方向。

Title: ConLID: Supervised Contrastive Learning for Low-Resource Language Identification

• Authors: Negar Foroutan, Jakhongir Saydaliev, Ye Eun Kim, Antoine Bosselut
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2506.15304
• Pdf URL: https://arxiv.org/pdf/2506.15304
• Copy Paste: [[2506.15304]] ConLID: Supervised Contrastive Learning for Low-Resource Language Identification(https://arxiv.org/abs/2506.15304)
• Keywords: llm
• Abstract: Language identification (LID) is a critical step in curating multilingual LLM pretraining corpora from web crawls. While many studies on LID model training focus on collecting diverse training data to improve performance, low-resource languages -- often limited to single-domain data, such as the Bible -- continue to perform poorly. To resolve these class imbalance and bias issues, we propose a novel supervised contrastive learning (SCL) approach to learn domain-invariant representations for low-resource languages. Through an extensive analysis, we show that our approach improves LID performance on out-of-domain data for low-resource languages by 3.2%, demonstrating its effectiveness in enhancing LID models.
• 摘要：语言识别（LID）是从Web Crawls策划多语言LLM验证Corpora的关键步骤。尽管许多关于盖子模型培训的研究集中在收集多样化的培训数据以提高性能，但低资源语言（通常仅限于单域数据，例如圣经）继续表现不佳。为了解决这些阶级的不平衡和偏见问题，我们提出了一种新颖的监督对比学习方法（SCL），以学习低资源语言的域名形式。通过广泛的分析，我们表明我们的方法可以提高低资源语言室外数据的盖子性能，提高了3.2％，这表明了其在增强盖子模型方面的有效性。

Title: DeVisE: Behavioral Testing of Medical Large Language Models

• Authors: Camila Zurdo Tagliabue, Heloisa Oss Boll, Aykut Erdem, Erkut Erdem, Iacer Calixto
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2506.15339
• Pdf URL: https://arxiv.org/pdf/2506.15339
• Copy Paste: [[2506.15339]] DeVisE: Behavioral Testing of Medical Large Language Models(https://arxiv.org/abs/2506.15339)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) are increasingly used in clinical decision support, yet current evaluation methods often fail to distinguish genuine medical reasoning from superficial patterns. We introduce DeVisE (Demographics and Vital signs Evaluation), a behavioral testing framework for probing fine-grained clinical understanding. We construct a dataset of ICU discharge notes from MIMIC-IV, generating both raw (real-world) and template-based (synthetic) versions with controlled single-variable counterfactuals targeting demographic (age, gender, ethnicity) and vital sign attributes. We evaluate five LLMs spanning general-purpose and medically fine-tuned variants, under both zero-shot and fine-tuned settings. We assess model behavior via (1) input-level sensitivity - how counterfactuals alter the likelihood of a note; and (2) downstream reasoning - how they affect predicted hospital length-of-stay. Our results show that zero-shot models exhibit more coherent counterfactual reasoning patterns, while fine-tuned models tend to be more stable yet less responsive to clinically meaningful changes. Notably, demographic factors subtly but consistently influence outputs, emphasizing the importance of fairness-aware evaluation. This work highlights the utility of behavioral testing in exposing the reasoning strategies of clinical LLMs and informing the design of safer, more transparent medical AI systems.
• 摘要：大型语言模型（LLM）越来越多地用于临床决策支持中，但是当前的评估方法通常无法将真正的医学推理与浅表模式区分开。我们介绍设计（人口统计和生命体征评估），这是一种行为测试框架，用于探测细粒度的临床理解。我们构建一个来自MIMIC-IV的ICU排放注释数据集，同时生成原始的（现实世界）和基于模板的（合成）版本，具有受控的单变量反事实，以人口统计学（年龄，性别，种族）和生命体征属性为目标。我们在零射门和微调设置下评估了跨越通用和医学微调变体的五个LLM。我们通过（1）输入级敏感性评估模型行为 - 反事实如何改变音符的可能性； （2）下游推理 - 它们如何影响预测的住院时间。我们的结果表明，零拍模型表现出更连贯的反事实推理模式，而微调模型往往更稳定，但对临床上有意义的变化的响应较低。值得注意的是，人口统计学因素巧妙地影响了产出，强调了公平感知评估的重要性。这项工作强调了行为测试在暴露临床LLM的推理策略方面的实用性，并为更安全，更透明的医学AI系统的设计提供了信息。

Title: SANSKRITI: A Comprehensive Benchmark for Evaluating Language Models' Knowledge of Indian Culture

• Authors: Arijit Maji, Raghvendra Kumar, Akash Ghosh, Anushka, Sriparna Saha
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2506.15355
• Pdf URL: https://arxiv.org/pdf/2506.15355
• Copy Paste: [[2506.15355]] SANSKRITI: A Comprehensive Benchmark for Evaluating Language Models' Knowledge of Indian Culture(https://arxiv.org/abs/2506.15355)
• Keywords: language model, llm
• Abstract: Language Models (LMs) are indispensable tools shaping modern workflows, but their global effectiveness depends on understanding local socio-cultural contexts. To address this, we introduce SANSKRITI, a benchmark designed to evaluate language models' comprehension of India's rich cultural diversity. Comprising 21,853 meticulously curated question-answer pairs spanning 28 states and 8 union territories, SANSKRITI is the largest dataset for testing Indian cultural knowledge. It covers sixteen key attributes of Indian culture: rituals and ceremonies, history, tourism, cuisine, dance and music, costume, language, art, festivals, religion, medicine, transport, sports, nightlife, and personalities, providing a comprehensive representation of India's cultural tapestry. We evaluate SANSKRITI on leading Large Language Models (LLMs), Indic Language Models (ILMs), and Small Language Models (SLMs), revealing significant disparities in their ability to handle culturally nuanced queries, with many models struggling in region-specific contexts. By offering an extensive, culturally rich, and diverse dataset, SANSKRITI sets a new standard for assessing and improving the cultural understanding of LMs.
• 摘要：语言模型（LMS）是塑造现代工作流程的必不可少的工具，但它们的全球有效性取决于理解当地的社会文化背景。为了解决这个问题，我们介绍了梵语，这是一种旨在评估语言模型对印度丰富文化多样性的理解的基准。梵语包括21,853个精心策划的问题解答，跨越28个州和8个联盟领土，是测试印度文化知识的最大数据集。它涵盖了印度文化的16个关键属性：仪式和仪式，历史，旅游，美食，舞蹈和音乐，服装，服装，语言，艺术，节日，宗教，医学，交通，体育，夜生活和个性，提供印度文化挂毯的全面代表。我们评估了梵语对领先的大语模型（LLM），印度语言模型（ILM）和小语言模型（SLM）的评估，在处理文化上细微的查询的能力上揭示了许多模型在特定环境中挣扎的许多模型。通过提供广泛的，文化丰富和多样化的数据集，梵语为评估和改善对LMS的文化理解的新标准设定了新标准。

Title: COSMMIC: Comment-Sensitive Multimodal Multilingual Indian Corpus for Summarization and Headline Generation

• Authors: Raghvendra Kumar, S. A. Mohammed Salman, Aryan Sahu, Tridib Nandi, Pragathi Y. P., Sriparna Saha, Jose G. Moreno
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2506.15372
• Pdf URL: https://arxiv.org/pdf/2506.15372
• Copy Paste: [[2506.15372]] COSMMIC: Comment-Sensitive Multimodal Multilingual Indian Corpus for Summarization and Headline Generation(https://arxiv.org/abs/2506.15372)
• Keywords: language model, gpt
• Abstract: Despite progress in comment-aware multimodal and multilingual summarization for English and Chinese, research in Indian languages remains limited. This study addresses this gap by introducing COSMMIC, a pioneering comment-sensitive multimodal, multilingual dataset featuring nine major Indian languages. COSMMIC comprises 4,959 article-image pairs and 24,484 reader comments, with ground-truth summaries available in all included languages. Our approach enhances summaries by integrating reader insights and feedback. We explore summarization and headline generation across four configurations: (1) using article text alone, (2) incorporating user comments, (3) utilizing images, and (4) combining text, comments, and images. To assess the dataset's effectiveness, we employ state-of-the-art language models such as LLama3 and GPT-4. We conduct a comprehensive study to evaluate different component combinations, including identifying supportive comments, filtering out noise using a dedicated comment classifier using IndicBERT, and extracting valuable insights from images with a multilingual CLIP-based classifier. This helps determine the most effective configurations for natural language generation (NLG) tasks. Unlike many existing datasets that are either text-only or lack user comments in multimodal settings, COSMMIC uniquely integrates text, images, and user feedback. This holistic approach bridges gaps in Indian language resources, advancing NLP research and fostering inclusivity.
• 摘要：尽管对英语和中文的评论意见多模式和多语言摘要取得了进展，但印度语言的研究仍然有限。这项研究通过引入Cosmmic（一种先锋评论敏感的多模式，多语言数据集，具有九种主要印度语言）来解决这一差距。 Cosmmic包含4,959个文章图像对和24,484个读者评论，其中包含所有包含语言的地面真相摘要。我们的方法通过整合读者的见解和反馈来增强摘要。我们探索跨四个配置的摘要和标题生成：（1）单独使用文章文本，（2）合并用户注释，（3）使用图像，以及（4）组合文本，评论和图像。为了评估数据集的有效性，我们采用了最先进的语言模型，例如Llama3和GPT-4。我们进行了一项全面的研究，以评估不同的组件组合，包括识别支持性注释，使用使用Indiadbert的专用评论分类器滤除噪声，并通过基于多语言夹的分类器从图像中提取有价值的见解。这有助于确定自然语言生成（NLG）任务的最有效配置。与许多现有数据集不同，这些数据集是仅文本或缺少多模式设置中的用户注释，宇宙唯一地集成了文本，图像和用户反馈。这种整体方法弥合了印度语言资源的差距，推进了NLP研究并培养包容性。

Title: Targeted Lexical Injection: Unlocking Latent Cross-Lingual Alignment in Lugha-Llama via Early-Layer LoRA Fine-Tuning

• Authors: Stanley Ngugi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2506.15415
• Pdf URL: https://arxiv.org/pdf/2506.15415
• Copy Paste: [[2506.15415]] Targeted Lexical Injection: Unlocking Latent Cross-Lingual Alignment in Lugha-Llama via Early-Layer LoRA Fine-Tuning(https://arxiv.org/abs/2506.15415)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities, yet their performance in low-resource languages (LRLs), such as Swahili, often lags due to data scarcity and underrepresentation in pre-training. A key challenge is achieving robust cross-lingual lexical alignment, crucial for tasks like translation and cross-lingual information retrieval. This paper introduces Targeted Lexical Injection (TLI), a novel and efficient fine-tuning approach. We first demonstrate that Lugha-Llama-8B-wura, a Swahili-centric LLM, exhibits strong, near-perfect lexical alignment for Swahili-English word pairs in its early internal layers (specifically Layer 2, with ~0.99998 average cosine similarity based on a pilot study), a capability not fully reflected in its final output representations (baseline ~0.32 similarity on our evaluation set). TLI leverages this insight by using Low-Rank Adaptation (LoRA) and a contrastive learning objective to fine-tune the model, specifically targeting embeddings from this empirically identified optimal early layer. Our experiments show that TLI significantly improves the output-level lexical alignment for 623 trained Swahili-English word pairs, increasing average cosine similarity from 0.3211 to 0.4113 (+28.08%, p < 1.33 x 10^-240). More importantly, these improvements generalize remarkably well to 63 unseen control word pairs, with similarity increasing from 0.3143 to 0.4033 (+28.32%, p < 7.17 x 10^-27). These findings suggest TLI enhances the model's ability to preserve and propagate its inherent early-layer cross-lingual knowledge, offering a parameter-efficient and effective strategy for improving lexical alignment in LRL-focused LLMs.
• 摘要：大型语言模型（LLM）表现出了显着的功能，但是它们在低资源语言（LRLS）中的表现（例如斯瓦希里语）通常由于数据稀缺性和预培训的代表性不足而滞后。一个关键的挑战是实现强大的跨语性词汇对准，对于翻译和跨语性信息检索等任务至关重要。本文介绍了靶向词汇注入（TLI），这是一种新颖有效的微调方法。我们首先证明了以西部中心的LLM卢加 - llama-8b-wura在其早期内部层中表现出强大的，接近完美的词汇对准（特定于第2层，特别是〜0.99998的cosine相似性，基于一项飞行员的能力）在其早期内部层中表现出〜BAIDS的平均相似性，因此无法完全反映（基于我们的能力）。 TLI通过使用低级适应性（LORA）和对比度学习目标来利用这种见解来微调模型，特别是针对该经验得出的最佳早期层的嵌入。我们的实验表明，TLI显着改善了623个经过训练的Swahili-English单词对的输出级词汇对齐，将平均余弦相似性从0.3211提高到0.4113（+28.08％，p <1.33 x x 10^-240）。更重要的是，这些改进非常概括至63个看不见的控制单词对，相似性从0.3143增加到0.4033（+28.32％，p <7.17 x 10^-27）。这些发现表明，TLI增强了该模型保留和传播其固有的早期跨语性知识的能力，从而提供了一种参数有效的有效策略，以改善以LRL为中心的LLM中的词汇一致性。

Title: Understanding GUI Agent Localization Biases through Logit Sharpness

• Authors: Xingjian Tao, Yiwei Wang, Yujun Cai, Zhicheng Yang, Jing Tang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2506.15425
• Pdf URL: https://arxiv.org/pdf/2506.15425
• Copy Paste: [[2506.15425]] Understanding GUI Agent Localization Biases through Logit Sharpness(https://arxiv.org/abs/2506.15425)
• Keywords: language model, llm, hallucination, agent
• Abstract: Multimodal large language models (MLLMs) have enabled GUI agents to interact with operating systems by grounding language into spatial actions. Despite their promising performance, these models frequently exhibit hallucinations-systematic localization errors that compromise reliability. We propose a fine-grained evaluation framework that categorizes model predictions into four distinct types, revealing nuanced failure modes beyond traditional accuracy metrics. To better quantify model uncertainty, we introduce the Peak Sharpness Score (PSS), a metric that evaluates the alignment between semantic continuity and logits distribution in coordinate prediction. Building on this insight, we further propose Context-Aware Cropping, a training-free technique that improves model performance by adaptively refining input context. Extensive experiments demonstrate that our framework and methods provide actionable insights and enhance the interpretability and robustness of GUI agent behavior.
• 摘要：多模式的大语言模型（MLLM）使GUI代理通过将语言扎根于空间动作来与操作系统交互。尽管表现出色，但这些模型经常表现出损害可靠性的幻觉系统定位错误。我们提出了一个细粒度的评估框架，将模型预测分为四种不同类型，揭示了传统准确度指标以外的细微失败模式。为了更好地量化模型不确定性，我们引入了峰值清晰度评分（PSS），该度量是评估语义连续性和坐标预测中逻辑分布之间的对齐的度量。在这种见解的基础上，我们进一步提出了上下文感知的种植，这是一种无训练的技术，可通过适应性地完善输入上下文来改善模型性能。广泛的实验表明，我们的框架和方法提供了可行的见解，并增强了GUI代理行为的解释性和鲁棒性。

Title: AgentGroupChat-V2: Divide-and-Conquer Is What LLM-Based Multi-Agent System Need

• Authors: Zhouhong Gu, Xiaoxuan Zhu, Yin Cai, Hao Shen, Xingzhou Chen, Qingyi Wang, Jialin Li, Xiaoran Shi, Haoran Guo, Wenxuan Huang, Hongwei Feng, Yanghua Xiao, Zheyu Ye, Yao Hu, Shaosheng Cao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2506.15451
• Pdf URL: https://arxiv.org/pdf/2506.15451
• Copy Paste: [[2506.15451]] AgentGroupChat-V2: Divide-and-Conquer Is What LLM-Based Multi-Agent System Need(https://arxiv.org/abs/2506.15451)
• Keywords: language model, llm, chat, agent
• Abstract: Large language model based multi-agent systems have demonstrated significant potential in social simulation and complex task resolution domains. However, current frameworks face critical challenges in system architecture design, cross-domain generalizability, and performance guarantees, particularly as task complexity and number of agents increases. We introduces AgentGroupChat-V2, a novel framework addressing these challenges through three core innovations: (1) a divide-and-conquer fully parallel architecture that decomposes user queries into hierarchical task forest structures enabling dependency management and distributed concurrent processing. (2) an adaptive collaboration engine that dynamically selects heterogeneous LLM combinations and interaction modes based on task characteristics. (3) agent organization optimization strategies combining divide-and-conquer approaches for efficient problem decomposition. Extensive experiments demonstrate AgentGroupChat-V2's superior performance across diverse domains, achieving 91.50% accuracy on GSM8K (exceeding the best baseline by 5.6 percentage points), 30.4% accuracy on competition-level AIME (nearly doubling other methods), and 79.20% pass@1 on HumanEval. Performance advantages become increasingly pronounced with higher task difficulty, particularly on Level 5 MATH problems where improvements exceed 11 percentage points compared to state-of-the-art baselines. These results confirm that AgentGroupChat-V2 provides a comprehensive solution for building efficient, general-purpose LLM multi-agent systems with significant advantages in complex reasoning scenarios. Code is available at this https URL.
• 摘要：基于语言模型的大型多机构系统在社会模拟和复杂的任务解决域中表现出了巨大的潜力。但是，当前的框架在系统体系结构设计，跨域的通用性和性能保证中面临着关键的挑战，尤其是随着任务复杂性和代理数量的增加。我们引入了AgentGroupChat-V2，这是一个通过三个核心创新来解决这些挑战的新型框架：（1）分隔和拼接完全并行体系结构，将用户查询分解为层次任务森林森林结构，从而实现依赖关系管理和分布式同时处理。 （2）基于任务特征的异质LLM组合和交互模式，动态选择的自适应协作引擎。 （3）将划分和串扰方法结合起来的代理组织优化策略，以进行有效的问题分解。广泛的实验表明，AgensGroupChat-V2在不同域中的出色性能，达到了GSM8K的91.50％精度（超过最佳基线，高出5.6个百分点），竞争水平的AIME的精度为30.4％（其他方法几乎翻倍），而HOMANANEVAL上有79.20％的通行证。绩效优势变得越来越明显，任务困难越来越高，尤其是在5级数学问题上，与最先进的基线相比，改进超过11个百分点的数学问题。这些结果证实，AgentGroupChat-V2为建立有效的通用，通用LLM多代理系统提供了全面的解决方案，在复杂的推理方案中具有显着优势。代码可在此HTTPS URL上找到。

Title: RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning Evaluation

• Authors: Xinnuo Xu, Rachel Lawrence, Kshitij Dubey, Atharva Pandey, Risa Ueno, Fabian Falck, Aditya V. Nori, Rahul Sharma, Amit Sharma, Javier Gonzalez
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2506.15455
• Pdf URL: https://arxiv.org/pdf/2506.15455
• Copy Paste: [[2506.15455]] RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning Evaluation(https://arxiv.org/abs/2506.15455)
• Keywords: language model, llm
• Abstract: Recent Large Language Models (LLMs) have reported high accuracy on reasoning benchmarks. However, it is still unclear whether the observed results arise from true reasoning or from statistical recall of the training set. Inspired by the ladder of causation (Pearl, 2009) and its three levels (associations, interventions and counterfactuals), this paper introduces RE-IMAGINE, a framework to characterize a hierarchy of reasoning ability in LLMs, alongside an automated pipeline to generate problem variations at different levels of the hierarchy. By altering problems in an intermediate symbolic representation, RE-IMAGINE generates arbitrarily many problems that are not solvable using memorization alone. Moreover, the framework is general and can work across reasoning domains, including math, code, and logic. We demonstrate our framework on four widely-used benchmarks to evaluate several families of LLMs, and observe reductions in performance when the models are queried with problem variations. These assessments indicate a degree of reliance on statistical recall for past performance, and open the door to further research targeting skills across the reasoning hierarchy.
• 摘要：最近的大型语言模型（LLMS）报告了推理基准的高精度。但是，目前尚不清楚观察到的结果是由真实的推理还是来自训练集的统计召回。受到因果关系阶梯（Pearl，2009）及其三个层次（协会，干预和反事实）的启发，本文引入了重新构想，这是一个框架，该框架是LLMS中推理能力层次结构的框架，以及自动化管道的层次结构，以产生层次不同级别的自动化问题。通过改变中间符号表示中的问题，重新构想会产生许多单独使用记忆无法解决的问题。此外，该框架是一般的，可以跨推理域（包括数学，代码和逻辑）工作。我们在四个广泛使用的基准测试中演示了我们的框架，以评估几个LLM家族，并观察到与问题变化查询模型时的性能降低。这些评估表明对过去表现的统计召回程度有一定程度，并为在推理层次结构中进一步研究目标技能打开了大门。

Title: Context-Informed Grounding Supervision

• Authors: Hyunji Lee, Seunghyun Yoon, Yunjae Won, Hanseok Oh, Geewook Kim, Trung Bui, Franck Dernoncourt, Elias Stengel-Eskin, Mohit Bansal, Minjoon Seo
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2506.15480
• Pdf URL: https://arxiv.org/pdf/2506.15480
• Copy Paste: [[2506.15480]] Context-Informed Grounding Supervision(https://arxiv.org/abs/2506.15480)
• Keywords: language model, llm, hallucination
• Abstract: Large language models (LLMs) are often supplemented with external knowledge to provide information not encoded in their parameters or to reduce hallucination. In such cases, we expect the model to generate responses by grounding its response in the provided external context. However, prior work has shown that simply appending context at inference time does not ensure grounded generation. To address this, we propose Context-INformed Grounding Supervision (CINGS), a post-training supervision in which the model is trained with relevant context prepended to the response, while computing the loss only over the response tokens and masking out the context. Our experiments demonstrate that models trained with CINGS exhibit stronger grounding in both textual and visual domains compared to standard instruction-tuned models. In the text domain, CINGS outperforms other training methods across 11 information-seeking datasets and is complementary to inference-time grounding techniques. In the vision-language domain, replacing a vision-language model's LLM backbone with a CINGS-trained model reduces hallucinations across four benchmarks and maintains factual consistency throughout the generated response. This improved grounding comes without degradation in general downstream performance. Finally, we analyze the mechanism underlying the enhanced grounding in CINGS and find that it induces a shift in the model's prior knowledge and behavior, implicitly encouraging greater reliance on the external context.
• 摘要：大型语言模型（LLMS）通常会补充外部知识，以提供未在其参数中编码或减少幻觉的信息。在这种情况下，我们希望该模型通过在提供的外部环境中对响应进行响应来产生响应。但是，先前的工作表明，仅在推理时间附加上下文并不能确保扎根。为了解决这个问题，我们提出了上下文信息的接地监督（CINGS），这是一种训练后的监督，其中使用相关上下文培训模型，同时仅在响应令牌上计算损失并掩盖上下文。我们的实验表明，与标准的指导调节模型相比，经过CISS训练的模型在文本和视觉领域都表现出更强的接地。在文本域中，CINGS的表现优于11个寻求信息数据集的其他培训方法，并且与推理时间接地技术互补。在视觉域中，用受过培训的模型替换视觉模型的LLM骨架可降低跨四个基准的幻觉，并在整个生成的响应中保持事实一致性。这种改进的接地无需在一般下游表现中降解。最后，我们分析了CINGS增强基础的基础机制，并发现它引起了模型的先验知识和行为的转变，从而暗中鼓励了对外部环境的更大依赖。

Title: SPARE: Single-Pass Annotation with Reference-Guided Evaluation for Automatic Process Supervision and Reward Modelling

• Authors: Md Imbesat Hassan Rizvi, Xiaodan Zhu, Iryna Gurevych
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2506.15498
• Pdf URL: https://arxiv.org/pdf/2506.15498
• Copy Paste: [[2506.15498]] SPARE: Single-Pass Annotation with Reference-Guided Evaluation for Automatic Process Supervision and Reward Modelling(https://arxiv.org/abs/2506.15498)
• Keywords: language model, llm
• Abstract: Process or step-wise supervision has played a crucial role in advancing complex multi-step reasoning capabilities of Large Language Models (LLMs). However, efficient, high-quality automated process annotation remains a significant challenge. To address this, we introduce Single-Pass Annotation with Reference-Guided Evaluation (SPARE), a novel structured framework that enables single-pass, per-step annotation by aligning each solution step to one or multiple steps in a reference solution, accompanied by explicit reasoning for evaluation. We show that reference-guided step-level evaluation effectively facilitates process supervision on four datasets spanning three domains: mathematical reasoning, multi-hop compositional question answering, and spatial reasoning. We demonstrate that SPARE, when compared to baselines, improves reasoning performance when used for: (1) fine-tuning models in an offline RL setup for inference-time greedy-decoding, and (2) training reward models for ranking/aggregating multiple LLM-generated outputs. Additionally, SPARE achieves competitive performance on challenging mathematical datasets while offering 2.6 times greater efficiency, requiring only 38% of the runtime, compared to tree search-based automatic annotation. The codebase, along with a trained SPARE-PRM model, is publicly released to facilitate further research and reproducibility.
• 摘要：流程或逐步监督在推进大型语言模型（LLMS）复杂的多步推理能力方面发挥了至关重要的作用。但是，高效，高质量的自动化过程注释仍然是一个重大挑战。为了解决这个问题，我们引入了单次通用注释，并使用参考引导评估（备用），这是一个新颖的结构化框架，通过将每个解决方案步骤与参考解决方案中的一个或多个步骤对准一个或多个步骤，从而实现单个通行，每步注释，并伴随着明确的推理进行评估。我们表明，参考引导的阶梯评估有效地促进了四个范围域的四个数据集上的过程监督：数学推理，多跳组合构成问题回答和空间推理。我们证明，与基准相比，备用在用于推理时间贪婪编码的离线RL设置中的微调模型时提高了推理性能，（2）（2）培训奖励模型，用于对多个LLM生成的输出进行排名/汇总。此外，与基于树搜索的自动注释相比，备件在具有挑战性的数学数据集上实现了竞争性能，同时提供了2.6倍的效率，仅需38％的运行时。该代码库以及受过训练的备用PRM模型公开发布，以促进进一步的研究和可重复性。

Title: Lessons from Training Grounded LLMs with Verifiable Rewards

• Authors: Shang Hong Sim, Tej Deep Pala, Vernon Toh, Hai Leong Chieu, Amir Zadeh, Chuan Li, Navonil Majumder, Soujanya Poria
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2506.15522
• Pdf URL: https://arxiv.org/pdf/2506.15522
• Copy Paste: [[2506.15522]] Lessons from Training Grounded LLMs with Verifiable Rewards(https://arxiv.org/abs/2506.15522)
• Keywords: language model, gpt, llm, retrieval-augmented generation
• Abstract: Generating grounded and trustworthy responses remains a key challenge for large language models (LLMs). While retrieval-augmented generation (RAG) with citation-based grounding holds promise, instruction-tuned models frequently fail even in straightforward scenarios: missing explicitly stated answers, citing incorrectly, or refusing when evidence is available. In this work, we explore how reinforcement learning (RL) and internal reasoning can enhance grounding in LLMs. We use the GRPO (Group Relative Policy Optimization) method to train models using verifiable outcome-based rewards targeting answer correctness, citation sufficiency, and refusal quality, without requiring gold reasoning traces or expensive annotations. Through comprehensive experiments across ASQA, QAMPARI, ELI5, and ExpertQA we show that reasoning-augmented models significantly outperform instruction-only variants, especially in handling unanswerable queries and generating well-cited responses. A two-stage training setup, first optimizing answer and citation behavior and then refusal, further improves grounding by stabilizing the learning signal. Additionally, we revisit instruction tuning via GPT-4 distillation and find that combining it with GRPO enhances performance on long-form, generative QA tasks. Overall, our findings highlight the value of reasoning, stage-wise optimization, and outcome-driven RL for building more verifiable and reliable LLMs.
• 摘要：对于大型语言模型（LLM）来说，产生扎根和值得信赖的响应仍然是一个关键挑战。尽管带有引用基础的基于基于引用的接地的检索型发电（RAG）具有希望，但教学调整的模型即使在直接情况下也经常失败：缺少明确指定的答案，理由是错误或拒绝在有证据时拒绝。在这项工作中，我们探讨了增强学习（RL）和内部推理如何在LLM中增强基础。我们使用GRPO（小组相对策略优化）方法使用基于可验证的结果的奖励来培训模型，以答案正确，引文充足性和拒绝质量，而无需黄金推理痕迹或昂贵的注释。通过ASQA，QAMPARI，ELI5和ExpertQA的全面实验，我们表明，推理的模型极大地超过了只有教学的变体，尤其是在处理无法回答的查询和产生众所周知的响应时。首先优化答案和引文行为，然后拒绝，通过稳定学习信号进一步改善了基础，这是两阶段的训练设置。此外，我们通过GPT-4蒸馏重新访问指令调整，并发现将其与GRPO结合起来可以增强长形式的生成质量质量检查任务的性能。总体而言，我们的发现突出了推理，阶段优化和结果驱动的RL的价值，用于构建更多可验证和可靠的LLM。

Title: Approximating Language Model Training Data from Weights

• Authors: John X. Morris, Junjie Oscar Yin, Woojeong Kim, Vitaly Shmatikov, Alexander M. Rush
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2506.15553
• Pdf URL: https://arxiv.org/pdf/2506.15553
• Copy Paste: [[2506.15553]] Approximating Language Model Training Data from Weights(https://arxiv.org/abs/2506.15553)
• Keywords: language model
• Abstract: Modern language models often have open weights but closed training data. We formalize the problem of data approximation from model weights and propose several baselines and metrics. We develop a gradient-based approach that selects the highest-matching data from a large public text corpus and show its effectiveness at recovering useful data given only weights of the original and finetuned models. Even when none of the true training data is known, our method is able to locate a small subset of public Web documents can be used to train a model to close to the original model performance given models trained for both classification and supervised-finetuning. On the AG News classification task, our method improves performance from 65% (using randomly selected data) to 80%, approaching the expert benchmark of 88%. When applied to a model trained with SFT on MSMARCO web documents, our method reduces perplexity from 3.3 to 2.3, compared to an expert LLAMA model's perplexity of 2.0.
• 摘要：现代语言模型通常具有开放的权重，但封闭的培训数据。我们正式从模型权重的数据近似问题正式化，并提出了几个基线和指标。我们开发了一种基于梯度的方法，该方法从大型公共文本语料库中选择最高匹配的数据，并显示其在仅给定原始模型的权重恢复有用数据方面的有效性。即使尚不知道真正的培训数据，我们的方法也能够定位一小部分公共网络文档，可用于训练模型，以接近培训用于分类和监督的模型的原始模型性能。在AG新闻分类任务上，我们的方法将绩效从65％（使用随机选择的数据）提高到80％，接近了88％的专家基准。当应用于MSMARCO Web文档上经过SFT训练的模型时，与专家Llama模型的困惑为2.0相比，我们的方法将困惑从3.3降低到2.3。

Title: PredGen: Accelerated Inference of Large Language Models through Input-Time Speculation for Real-Time Speech Interaction

• Authors: Shufan Li, Aditya Grover
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2506.15556
• Pdf URL: https://arxiv.org/pdf/2506.15556
• Copy Paste: [[2506.15556]] PredGen: Accelerated Inference of Large Language Models through Input-Time Speculation for Real-Time Speech Interaction(https://arxiv.org/abs/2506.15556)
• Keywords: language model, llm, chat
• Abstract: Large Language Models (LLMs) are widely used in real-time voice chat applications, typically in combination with text-to-speech (TTS) systems to generate audio responses. However, their large size often leads to noticeable latency between the end of user input and the start of audio output, resulting in suboptimal user experiences. This latency is particularly evident when LLMs are deployed as single-user voice assistants on consumer-grade hardware with limited computing capacity. We discovered that this latency is primarily dominated by the time it takes for the LLMs to generate the first sentence, which is required as input by the TTS systems that synthesize audio responses on a sentence-by-sentence basis. To address this bottleneck, we propose Predictive Generation (PredGen), a novel framework that mitigates-or even eliminates-this delay through speculative decoding at input time. PredGen generates candidate responses while the user is still speaking, enabling the system to begin TTS processing with minimal delay. Simulated experiments on the Lmsys and MT-Bench datasets show that the proposed method can effectively reduce the latency by around 2x across a wide range of use cases, while incurring only minimal additional computation cost at input time-computation that would otherwise go unused.
• 摘要：大型语言模型（LLM）广泛用于实时语音聊天应用程序，通常与文本到语音（TTS）系统结合使用以生成音频响应。但是，它们的大尺寸通常会导致用户输入结束和音频输出开始之间的明显延迟，从而带来次优的用户体验。当将LLMS作为单用户语音助手部署在消费级硬件中，计算能力有限时，该延迟尤其明显。我们发现，该延迟主要由LLMS生成第一个句子所需的时间主导，这是TTS系统所要求的，该句子是由TTS Systems逐句综合响应的输入。为了解决这种瓶颈，我们提出了预测生成（Predgen），这是一个新颖的框架，可以减轻甚至消除这种延迟 - 通过在输入时间进行投机解码。在用户仍在讲话时，preadgen会生成候选响应，使系统能够以最小的延迟开始TTS处理。在LMSYS和MT BENCHEND数据集上进行的模拟实验表明，在广泛的用例中，提出的方法可以有效地将潜伏期降低约2倍，而在输入时间计算中仅产生最小的额外计算成本，否则这些时间限制否则会均未使用。

Title: Gender Inclusivity Fairness Index (GIFI): A Multilevel Framework for Evaluating Gender Diversity in Large Language Models

• Authors: Zhengyang Shan, Emily Ruth Diana, Jiawei Zhou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2506.15568
• Pdf URL: https://arxiv.org/pdf/2506.15568
• Copy Paste: [[2506.15568]] Gender Inclusivity Fairness Index (GIFI): A Multilevel Framework for Evaluating Gender Diversity in Large Language Models(https://arxiv.org/abs/2506.15568)
• Keywords: language model, llm
• Abstract: We present a comprehensive evaluation of gender fairness in large language models (LLMs), focusing on their ability to handle both binary and non-binary genders. While previous studies primarily focus on binary gender distinctions, we introduce the Gender Inclusivity Fairness Index (GIFI), a novel and comprehensive metric that quantifies the diverse gender inclusivity of LLMs. GIFI consists of a wide range of evaluations at different levels, from simply probing the model with respect to provided gender pronouns to testing various aspects of model generation and cognitive behaviors under different gender assumptions, revealing biases associated with varying gender identifiers. We conduct extensive evaluations with GIFI on 22 prominent open-source and proprietary LLMs of varying sizes and capabilities, discovering significant variations in LLMs' gender inclusivity. Our study highlights the importance of improving LLMs' inclusivity, providing a critical benchmark for future advancements in gender fairness in generative models.
• 摘要：我们对大语言模型（LLM）中性别公平性进行全面评估，重点是他们处理二进制和非二元性别的能力。虽然先前的研究主要集中于二元性别差异，但我们引入了性别包容性公平指数（GIFI），这是一种新颖而全面的指标，量化了LLMS多样化的性别包容性。 GIFI由不同级别的广泛评估组成，从简单地探测模型对所提供的性别代词到在不同性别假设下的模型产生和认知行为的各个方面，揭示与不同性别标识符相关的偏见。我们对GIFI对22个大小和能力的22个突出的开源和专有LLM进行了广泛的评估，发现LLMS性别包容性的显着差异。我们的研究强调了提高LLM的包容性的重要性，为生成模型中性别公平的未来进步提供了关键的基准。

Title: SciVer: Evaluating Foundation Models for Multimodal Scientific Claim Verification

• Authors: Chengye Wang, Yifei Shen, Zexi Kuang, Arman Cohan, Yilun Zhao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2506.15569
• Pdf URL: https://arxiv.org/pdf/2506.15569
• Copy Paste: [[2506.15569]] SciVer: Evaluating Foundation Models for Multimodal Scientific Claim Verification(https://arxiv.org/abs/2506.15569)
• Keywords: retrieval-augmented generation
• Abstract: We introduce SciVer, the first benchmark specifically designed to evaluate the ability of foundation models to verify claims within a multimodal scientific context. SciVer consists of 3,000 expert-annotated examples over 1,113 scientific papers, covering four subsets, each representing a common reasoning type in multimodal scientific claim verification. To enable fine-grained evaluation, each example includes expert-annotated supporting evidence. We assess the performance of 21 state-of-the-art multimodal foundation models, including o4-mini, Gemini-2.5-Flash, Llama-3.2-Vision, and Qwen2.5-VL. Our experiment reveals a substantial performance gap between these models and human experts on SciVer. Through an in-depth analysis of retrieval-augmented generation (RAG), and human-conducted error evaluations, we identify critical limitations in current open-source models, offering key insights to advance models' comprehension and reasoning in multimodal scientific literature tasks.
• 摘要：我们介绍了Sciver，这是第一个专门设计的，旨在评估基础模型在多模式科学环境中验证主张的能力。 Sciver由3,000个专家注册的示例组成，超过1,113篇科学论文，涵盖了四个子集，每个子​​集代表多模式科学主张验证中的常见推理类型。为了实现细粒度的评估，每个示例都包含专家注销的支持证据。我们评估了21种最先进的多模式模型的性能，包括O4-Mini，Gemini-2.5-Flash，Llama-3.2-Vision和Qwen2.5-VL。我们的实验揭示了这些模型与SCIVER的人类专家之间的巨大性能差距。通过深入分析检索功能增强的生成（RAG）和人为导入的错误评估，我们确定了当前开源模型中的临界局限性，提供了关键的见解，以提高模型在多模式科学文献任务中的理解和推理。

Title: DiscoSG: Towards Discourse-Level Text Scene Graph Parsing through Iterative Graph Refinement

• Authors: Shaoqing Lin, Chong Teng, Fei Li, Donghong Ji, Lizhen Qu, Zhuang Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2506.15583
• Pdf URL: https://arxiv.org/pdf/2506.15583
• Copy Paste: [[2506.15583]] DiscoSG: Towards Discourse-Level Text Scene Graph Parsing through Iterative Graph Refinement(https://arxiv.org/abs/2506.15583)
• Keywords: language model, gpt, hallucination
• Abstract: Vision-Language Models (VLMs) now generate discourse-level, multi-sentence visual descriptions, challenging text scene graph parsers originally designed for single-sentence caption-to-graph mapping. Current approaches typically merge sentence-level parsing outputs for discourse input, often missing phenomena like cross-sentence coreference, resulting in fragmented graphs and degraded downstream VLM task performance. To address this, we introduce a new task, Discourse-level text Scene Graph parsing (DiscoSG), supported by our dataset DiscoSG-DS, which comprises 400 expert-annotated and 8,430 synthesised multi-sentence caption-graph pairs for images. Each caption averages 9 sentences, and each graph contains at least 3 times more triples than those in existing datasets. While fine-tuning large PLMs (i.e., GPT-4) on DiscoSG-DS improves SPICE by approximately 48% over the best sentence-merging baseline, high inference cost and restrictive licensing hinder its open-source use, and smaller fine-tuned PLMs struggle with complex graphs. We propose DiscoSG-Refiner, which drafts a base graph using one small PLM, then employs a second PLM to iteratively propose graph edits, reducing full-graph generation overhead. Using two Flan-T5-Base models, DiscoSG-Refiner still improves SPICE by approximately 30% over the best baseline while achieving 86 times faster inference than GPT-4. It also consistently improves downstream VLM tasks like discourse-level caption evaluation and hallucination detection. Code and data are available at: this https URL
• 摘要：视觉语言模型（VLMS）现在生成话语级别的，多句的视觉描述，具有挑战性的文本场景图形解析器，最初是为单句话字幕到图形映射而设计的。当前方法通常合并句子级解析输出，以进行话语输入，通常缺少诸如跨句子核心的现象，从而产生零散的图形并降低下游VLM任务性能。为了解决这个问题，我们介绍了一项新任务，话语级文本场景图解析（Discosg），并由我们的数据集DiscoSG-DS支持，该数据包括400个专家注释和8,430个合成的多句话字幕图图像。每个字幕的平均值为9个句子，每个图的包含的三元组至少比现有数据集中的三个句子高3倍。在迪斯科DS上微调大PLM（即GPT-4）的同时，与最佳句子 - 合并基线相比，香料的速度约为48％，高推理成本和限制性许可阻碍了其开放源代码的使用，而较小的微型PLM却与复杂的图形斗争。我们建议使用一个小PLM来起草基本图，然后使用第二个PLM迭代提出图表编辑，从而减少了全部发电的开销。使用两种Flan-T5碱模型，Discosg-Refiner仍然比最佳基线相比，将香料提高约30％，同时获得的推断速度比GPT-4快86倍。它还一致地改善了下游VLM任务，例如话语级标题评估和幻觉检测。代码和数据可用：此HTTPS URL

Title: WikiMixQA: A Multimodal Benchmark for Question Answering over Tables and Charts

• Authors: Negar Foroutan, Angelika Romanou, Matin Ansaripour, Julian Martin Eisenschlos, Karl Aberer, Rémi Lebret
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2506.15594
• Pdf URL: https://arxiv.org/pdf/2506.15594
• Copy Paste: [[2506.15594]] WikiMixQA: A Multimodal Benchmark for Question Answering over Tables and Charts(https://arxiv.org/abs/2506.15594)
• Keywords: language model, gpt, llm
• Abstract: Documents are fundamental to preserving and disseminating information, often incorporating complex layouts, tables, and charts that pose significant challenges for automatic document understanding (DU). While vision-language large models (VLLMs) have demonstrated improvements across various tasks, their effectiveness in processing long-context vision inputs remains unclear. This paper introduces WikiMixQA, a benchmark comprising 1,000 multiple-choice questions (MCQs) designed to evaluate cross-modal reasoning over tables and charts extracted from 4,000 Wikipedia pages spanning seven distinct topics. Unlike existing benchmarks, WikiMixQA emphasizes complex reasoning by requiring models to synthesize information from multiple modalities. We evaluate 12 state-of-the-art vision-language models, revealing that while proprietary models achieve ~70% accuracy when provided with direct context, their performance deteriorates significantly when retrieval from long documents is required. Among these, GPT-4-o is the only model exceeding 50% accuracy in this setting, whereas open-source models perform considerably worse, with a maximum accuracy of 27%. These findings underscore the challenges of long-context, multi-modal reasoning and establish WikiMixQA as a crucial benchmark for advancing document understanding research.
• 摘要：文档对于保存和传播信息至关重要，通常包含复杂的布局，表和图表，这些图表和图表对自动文档理解构成了重大挑战（DU）。虽然视觉语言大型模型（VLLM）在各种任务上都表现出了改进，但它们在处理长篇小说视觉输入方面的有效性尚不清楚。本文介绍了Wikimixqa，这是一个基准，该基准包括1,000个多项选择问题（MCQ），旨在评估桌子上的跨模式推理，并从4,000个Wikipedia页面中提取的图表中提取了七个不同的主题。与现有基准不同，Wikimixqa通过要求模型合成多种模式的信息来强调复杂的推理。我们评估了12种最先进的视觉语言模型，表明专有模型在提供直接上下文时达到了〜70％的精度，但是当需要从长期文档中检索时，它们的性能会大大恶化。其中，在这种情况下，GPT-4-O是唯一超过50％精度的模型，而开源模型的性能差得多，最高精度为27％。这些发现强调了长篇文化，多模式推理的挑战，并建立了Wikimixqa作为推进文档理解研究的关键基准。

Title: The Compositional Architecture of Regret in Large Language Models

• Authors: Xiangxiang Cui, Shu Yang, Tianjin Huang, Wanyu Lin, Lijie Hu, Di Wang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2506.15617
• Pdf URL: https://arxiv.org/pdf/2506.15617
• Copy Paste: [[2506.15617]] The Compositional Architecture of Regret in Large Language Models(https://arxiv.org/abs/2506.15617)
• Keywords: language model, prompt
• Abstract: Regret in Large Language Models refers to their explicit regret expression when presented with evidence contradicting their previously generated misinformation. Studying the regret mechanism is crucial for enhancing model reliability and helps in revealing how cognition is coded in neural networks. To understand this mechanism, we need to first identify regret expressions in model outputs, then analyze their internal representation. This analysis requires examining the model's hidden states, where information processing occurs at the neuron level. However, this faces three key challenges: (1) the absence of specialized datasets capturing regret expressions, (2) the lack of metrics to find the optimal regret representation layer, and (3) the lack of metrics for identifying and analyzing regret neurons. Addressing these limitations, we propose: (1) a workflow for constructing a comprehensive regret dataset through strategically designed prompting scenarios, (2) the Supervised Compression-Decoupling Index (S-CDI) metric to identify optimal regret representation layers, and (3) the Regret Dominance Score (RDS) metric to identify regret neurons and the Group Impact Coefficient (GIC) to analyze activation patterns. Our experimental results successfully identified the optimal regret representation layer using the S-CDI metric, which significantly enhanced performance in probe classification experiments. Additionally, we discovered an M-shaped decoupling pattern across model layers, revealing how information processing alternates between coupling and decoupling phases. Through the RDS metric, we categorized neurons into three distinct functional groups: regret neurons, non-regret neurons, and dual neurons.
• 摘要：在大型语言模型中，遗憾是指在有证据与以前产生的错误信息相矛盾的证据时，其明确的遗憾表达。研究遗憾机制对于增强模型的可靠性至关重要，并有助于揭示在神经网络中如何编码认知。要了解这种机制，我们需要首先确定模型输出中的遗憾表达式，然后分析其内部表示。该分析需要检查模型的隐藏状态，其中信息处理发生在神经元级别。但是，这面临三个主要挑战：（1）缺乏捕获遗憾表达的专业数据集，（2）缺乏指标来找到最佳的遗憾代表层，以及（3）缺乏识别和分析遗憾神经元的指标。解决这些局限性，我们提出：（1）通过战略性设计的提示场景来构建全面的遗憾数据集的工作流程，（2）监督的压缩压缩解释指数（S-CDI）指标以识别最佳的遗憾代表层，以及（3）遗憾的统治得分（RDS），以确定遗憾的神经元和集体影响的模式，以识别组合的模式和giceffectiation（giC）分析（GIC）。我们的实验结果成功地使用了S-CDI度量标准，成功地识别了最佳遗憾表示层，从而显着提高了探针分类实验的性能。此外，我们发现了模型层的M形解耦模式，揭示了信息处理如何在耦合和去耦阶段之间交替。通过RDS度量，我们将神经元分为三个不同的功能组：遗憾神经元，非重生神经元和双神经元。

Title: Revisiting Compositional Generalization Capability of Large Language Models Considering Instruction Following Ability

• Authors: Yusuke Sakai, Hidetaka Kamigaito, Taro Watanabe
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2506.15629
• Pdf URL: https://arxiv.org/pdf/2506.15629
• Copy Paste: [[2506.15629]] Revisiting Compositional Generalization Capability of Large Language Models Considering Instruction Following Ability(https://arxiv.org/abs/2506.15629)
• Keywords: language model, llm, prompt
• Abstract: In generative commonsense reasoning tasks such as CommonGen, generative large language models (LLMs) compose sentences that include all given concepts. However, when focusing on instruction-following capabilities, if a prompt specifies a concept order, LLMs must generate sentences that adhere to the specified order. To address this, we propose Ordered CommonGen, a benchmark designed to evaluate the compositional generalization and instruction-following abilities of LLMs. This benchmark measures ordered coverage to assess whether concepts are generated in the specified order, enabling a simultaneous evaluation of both abilities. We conducted a comprehensive analysis using 36 LLMs and found that, while LLMs generally understand the intent of instructions, biases toward specific concept order patterns often lead to low-diversity outputs or identical results even when the concept order is altered. Moreover, even the most instruction-compliant LLM achieved only about 75% ordered coverage, highlighting the need for improvements in both instruction-following and compositional generalization capabilities.
• 摘要：在生成常识性推理任务中，例如公约，生成的大语言模型（LLMS），构成了包括所有给定概念的句子。但是，当关注指令跟随功能时，如果提示指定概念顺序，LLMS必须生成遵守指定顺序的句子。为了解决这个问题，我们提出了有序的Commongen，这是一种基准，旨在评估LLMS的组成概括和指导遵循能力。该基准测量订单覆盖范围，以评估是否按照指定顺序生成概念，从而同时评估这两种能力。我们使用36个LLM进行了全面的分析，发现尽管LLMS通常了解说明的意图，但对特定概念顺序模式的偏见通常会导致低多样性输出或相同的结果，即使概念顺序更改。此外，即使是最符合教学的LLM，也只能达到75％的订购覆盖范围，这突出了需要改进指导跟踪和组成概括能力的需求。

Title: CC-LEARN: Cohort-based Consistency Learning

• Authors: Xiao Ye, Shaswat Shrivastava, Zhaonan Li, Jacob Dineen, Shijie Lu, Avneet Ahuja, Ming Shen, Zhikun Xu, Ben Zhou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2506.15662
• Pdf URL: https://arxiv.org/pdf/2506.15662
• Copy Paste: [[2506.15662]] CC-LEARN: Cohort-based Consistency Learning(https://arxiv.org/abs/2506.15662)
• Keywords: language model, llm
• Abstract: Large language models excel at many tasks but still struggle with consistent, robust reasoning. We introduce Cohort-based Consistency Learning (CC-Learn), a reinforcement learning framework that improves the reliability of LLM reasoning by training on cohorts of similar questions derived from shared programmatic abstractions. To enforce cohort-level consistency, we define a composite objective combining cohort accuracy, a retrieval bonus for effective problem decomposition, and a rejection penalty for trivial or invalid lookups that reinforcement learning can directly optimize, unlike supervised fine-tuning. Optimizing this reward guides the model to adopt uniform reasoning patterns across all cohort members. Experiments on challenging reasoning benchmarks (including ARC-Challenge and StrategyQA) show that CC-Learn boosts both accuracy and reasoning stability over pretrained and SFT baselines. These results demonstrate that cohort-level RL effectively enhances reasoning consistency in LLMs.
• 摘要：大型语言模型在许多任务上都表现出色，但仍然以一致，强大的推理而挣扎。我们介绍了基于队列的一致性学习（CC-Learn），这是一个强化学习框架，通过培训从共享的程序化抽象中得出的类似问题来提高LLM推理的可靠性。为了执行队列级的一致性，我们定义了一个组合队列准确性的复合目标，有效问题分解的检索奖金以及对琐碎或无效查找的拒绝惩罚，与受监督的细调相比，增强型学习可以直接优化，该罚款可以直接优化。优化此奖励指导该模型在所有队列成员中采用统一的推理模式。关于挑战推理​​基准（包括电弧挑战和策略QA）的实验表明，CC-Learn比预处理和SFT基线提高了准确性和推理稳定性。这些结果表明，队列级RL有效提高了LLM的推理一致性。

Title: Leaky Thoughts: Large Reasoning Models Are Not Private Thinkers

• Authors: Tommaso Green, Martin Gubri, Haritz Puerto, Sangdoo Yun, Seong Joon Oh
• Subjects: cs.CL, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2506.15674
• Pdf URL: https://arxiv.org/pdf/2506.15674
• Copy Paste: [[2506.15674]] Leaky Thoughts: Large Reasoning Models Are Not Private Thinkers(https://arxiv.org/abs/2506.15674)
• Keywords: prompt, agent
• Abstract: We study privacy leakage in the reasoning traces of large reasoning models used as personal agents. Unlike final outputs, reasoning traces are often assumed to be internal and safe. We challenge this assumption by showing that reasoning traces frequently contain sensitive user data, which can be extracted via prompt injections or accidentally leak into outputs. Through probing and agentic evaluations, we demonstrate that test-time compute approaches, particularly increased reasoning steps, amplify such leakage. While increasing the budget of those test-time compute approaches makes models more cautious in their final answers, it also leads them to reason more verbosely and leak more in their own thinking. This reveals a core tension: reasoning improves utility but enlarges the privacy attack surface. We argue that safety efforts must extend to the model's internal thinking, not just its outputs.
• 摘要：我们在用作个人代理的大型推理模型的推理轨迹中研究隐私泄漏。与最终输出不同，通常认为推理轨迹是内部和安全的。我们通过表明推理轨迹经常包含敏感用户数据来挑战这一假设，该数据可以通过提示注射或意外泄漏到输出中提取。通过探测和代理评估，我们证明了测试时间计算方法，尤其是增加的推理步骤，扩大了这种泄漏。尽管提高了这些测试时间计算方法的预算使模型在最终答案中更加谨慎，但它也使他们更加言语地推理并在自己的思维中泄漏更多。这揭示了核心张力：推理改善了效用，但扩大了隐私攻击表面。我们认为，安全工作必须扩展到模型的内部思维，而不仅仅是其输出。

Title: GenRecal: Generation after Recalibration from Large to Small Vision-Language Models

• Authors: Byung-Kwan Lee, Ryo Hachiuma, Yong Man Ro, Yu-Chiang Frank Wang, Yueh-Hua Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2506.15681
• Pdf URL: https://arxiv.org/pdf/2506.15681
• Copy Paste: [[2506.15681]] GenRecal: Generation after Recalibration from Large to Small Vision-Language Models(https://arxiv.org/abs/2506.15681)
• Keywords: language model, gpt, llm
• Abstract: Recent advancements in vision-language models (VLMs) have leveraged large language models (LLMs) to achieve performance on par with closed-source systems like GPT-4V. However, deploying these models in real-world scenarios, particularly on resource-constrained devices, remains challenging due to their substantial computational demands. This has spurred interest in distilling knowledge from large VLMs into smaller, more efficient counterparts. A key challenge arises here from the diversity of VLM architectures, which are built on different LLMs and employ varying token types-differing in vocabulary size, token splits, and token index ordering. To address this challenge of limitation to a specific VLM type, we present Generation after Recalibration (GenRecal), a novel, general-purpose distillation framework for VLMs. GenRecal incorporates a Recalibrator that aligns and adapts feature representations between heterogeneous VLMs, enabling effective knowledge transfer across different types of VLMs. Through extensive experiments on multiple challenging benchmarks, we demonstrate that GenRecal significantly improves baseline performances, eventually outperforming large-scale open- and closed-source VLMs.
• 摘要：视觉模型（VLM）的最新进展已利用大型语言模型（LLMS）在诸如GPT-4V（例如GPT-4V）的封闭源系统上实现性能。但是，由于其实质性的计算需求，将这些模型部署在现实世界中，尤其是在资源受限设备上，仍然具有挑战性。这激发了人们将知识从大型VLM提炼成较小，更有效的对应物的兴趣。这里的关键挑战是由VLM架构的多样性引起的，VLM架构的多样性是建立在不同的LLM上的，并采用了不同的令牌类型，这些类型在词汇尺寸，令牌拆分和令牌索引订购方面有所不同。为了解决对特定VLM类型的限制的挑战，我们在重新校准后（GenRecal）是一种新型的VLMS通用蒸馏框架（Genrecal）。 GenRecal结合了一种重新安装并适应异质VLM之间具有表示形式的重新配对器，从而在不同类型的VLMS上实现了有效的知识转移。通过对多个具有挑战性的基准测试的广泛实验，我们证明了Genrecal显着改善了基线性能，最终表现优于大规模开放式和封闭源VLM。

Title: PhantomHunter: Detecting Unseen Privately-Tuned LLM-Generated Text via Family-Aware Learning

• Authors: Yuhui Shi, Yehan Yang, Qiang Sheng, Hao Mi, Beizhe Hu, Chaoxi Xu, Juan Cao
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2506.15683
• Pdf URL: https://arxiv.org/pdf/2506.15683
• Copy Paste: [[2506.15683]] PhantomHunter: Detecting Unseen Privately-Tuned LLM-Generated Text via Family-Aware Learning(https://arxiv.org/abs/2506.15683)
• Keywords: language model, llm
• Abstract: With the popularity of large language models (LLMs), undesirable societal problems like misinformation production and academic misconduct have been more severe, making LLM-generated text detection now of unprecedented importance. Although existing methods have made remarkable progress, a new challenge posed by text from privately tuned LLMs remains underexplored. Users could easily possess private LLMs by fine-tuning an open-source one with private corpora, resulting in a significant performance drop of existing detectors in practice. To address this issue, we propose PhantomHunter, an LLM-generated text detector specialized for detecting text from unseen, privately-tuned LLMs. Its family-aware learning framework captures family-level traits shared across the base models and their derivatives, instead of memorizing individual characteristics. Experiments on data from LLaMA, Gemma, and Mistral families show its superiority over 7 baselines and 3 industrial services, with F1 scores of over 96%.
• 摘要：随着大语言模型（LLM）的普及，不良的社会问题（例如错误信息生产和学术不当行为）变得更加严重，这使得LLM生成的文本检测现在具有前所未有的重要性。尽管现有方法取得了显着的进步，但私人调整LLMS的文本提出了一个新的挑战。用户可以通过用私人语料库微调开源的LLM来轻松拥有私人LLM，从而导致现有检测器在实践中大幅下降。为了解决这个问题，我们建议使用LLM生成的文本检测器Phantomhunter，专门用于检测未见，私人调整的LLM的文本。它的家庭感知学习框架捕获了基本模型及其衍生品共享的家庭水平特征，而不是记住个人特征。来自骆驼，杰玛和米斯特拉尔家族的数据的实验表明，其优于7个基线和3个工业服务，F1得分超过96％。