2025-09-11

Title: Bilingual Word Level Language Identification for Omotic Languages

• Authors: Mesay Gemeda Yigezu, Girma Yohannis Bade, Atnafu Lambebo Tonja, Olga Kolesnikova, Grigori Sidorov, Alexander Gelbukh
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2509.07998
• Pdf URL: https://arxiv.org/pdf/2509.07998
• Copy Paste: [[2509.07998]] Bilingual Word Level Language Identification for Omotic Languages(https://arxiv.org/abs/2509.07998)
• Keywords: language model
• Abstract: Language identification is the task of determining the languages for a given text. In many real world scenarios, text may contain more than one language, particularly in multilingual communities. Bilingual Language Identification (BLID) is the task of identifying and distinguishing between two languages in a given text. This paper presents BLID for languages spoken in the southern part of Ethiopia, namely Wolaita and Gofa. The presence of words similarities and differences between the two languages makes the language identification task challenging. To overcome this challenge, we employed various experiments on various approaches. Then, the combination of the BERT based pretrained language model and LSTM approach performed better, with an F1 score of 0.72 on the test set. As a result, the work will be effective in tackling unwanted social media issues and providing a foundation for further research in this area.
• 摘要：语言标识是确定给定文本的语言的任务。在许多现实世界的情况下，文本可能包含多种语言，尤其是在多语言社区中。双语语言标识（Blid）是识别和区分给定文本中两种语言的任务。本文介绍了在埃塞俄比亚南部所说的语言的Blid，即Wolaita和Gofa。两种语言之间的单词相似性和差异的存在使语言识别任务具有挑战性。为了克服这一挑战，我们对各种方法进行了各种实验。然后，基于BERT预验证的语言模型和LSTM方法的组合表现更好，测试集的F1得分为0.72。结果，这项工作将有效解决不需要的社交媒体问题，并为该领域的进一步研究提供基础。

Title: AntiDote: Bi-level Adversarial Training for Tamper-Resistant LLMs

• Authors: Debdeep Sanyal, Manodeep Ray, Murari Mandal
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2509.08000
• Pdf URL: https://arxiv.org/pdf/2509.08000
• Copy Paste: [[2509.08000]] AntiDote: Bi-level Adversarial Training for Tamper-Resistant LLMs(https://arxiv.org/abs/2509.08000)
• Keywords: language model, llm, prompt
• Abstract: The release of open-weight large language models (LLMs) creates a tension between advancing accessible research and preventing misuse, such as malicious fine-tuning to elicit harmful content. Current safety measures struggle to preserve the general capabilities of the LLM while resisting a determined adversary with full access to the model's weights and architecture, who can use full-parameter fine-tuning to erase existing safeguards. To address this, we introduce AntiDote, a bi-level optimization procedure for training LLMs to be resistant to such tampering. AntiDote involves an auxiliary adversary hypernetwork that learns to generate malicious Low-Rank Adaptation (LoRA) weights conditioned on the defender model's internal activations. The defender LLM is then trained with an objective to nullify the effect of these adversarial weight additions, forcing it to maintain its safety alignment. We validate this approach against a diverse suite of 52 red-teaming attacks, including jailbreak prompting, latent space manipulation, and direct weight-space attacks. AntiDote is upto 27.4\% more robust against adversarial attacks compared to both tamper-resistance and unlearning baselines. Crucially, this robustness is achieved with a minimal trade-off in utility, incurring a performance degradation of upto less than 0.5\% across capability benchmarks including MMLU, HellaSwag, and GSM8K. Our work offers a practical and compute efficient methodology for building open-weight models where safety is a more integral and resilient property.
• 摘要：开放权重的大语言模型（LLM）的发布在进行可访问的研究和防止滥用之间产生了张力，例如恶意微调以引起有害内容。当前的安全措施难以保留LLM的一般能力，同时抵制确定的对手，并完全访问模型的权重和建筑，他们可以使用全参数微调来删除现有的保障措施。为了解决这个问题，我们引入了解毒剂，这是一种双层优化程序，用于训练LLMS对这种篡改具有抵抗力。解毒剂涉及一个辅助对手超网络，该辅助功能学会在Defender模型的内部激活下产生恶意低级适应性（LORA）权重。然后对Defender LLM进行训练，以使这些对抗权重增加的效果无效，从而迫使其保持安全对准。我们将这种方法验证为52次红色团队攻击的各种套件，包括越狱提示，潜在的太空操纵和直接的重量空间攻击。与篡改和未学习的基线相比，在对抗性攻击方面，解毒剂对对抗性攻击的强劲性高27.4 \％。至关重要的是，这种鲁棒性是通过实用程序的最低权衡取舍而实现的，在包括MMLU，Hellaswag和GSM8K在内的能力基准中，性能下降到不到0.5％。我们的工作提供了一种实用和计算的有效方法，用于构建安全性更具不可或缺和有弹性的开放权重模型。

Title: MVPBench: A Benchmark and Fine-Tuning Framework for Aligning Large Language Models with Diverse Human Values

• Authors: Yao Liang, Dongcheng Zhao, Feifei Zhao, Guobin Shen, Yuwei Wang, Dongqi Liang, Yi Zeng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2509.08022
• Pdf URL: https://arxiv.org/pdf/2509.08022
• Copy Paste: [[2509.08022]] MVPBench: A Benchmark and Fine-Tuning Framework for Aligning Large Language Models with Diverse Human Values(https://arxiv.org/abs/2509.08022)
• Keywords: language model, llm
• Abstract: The alignment of large language models (LLMs) with human values is critical for their safe and effective deployment across diverse user populations. However, existing benchmarks often neglect cultural and demographic diversity, leading to limited understanding of how value alignment generalizes globally. In this work, we introduce MVPBench, a novel benchmark that systematically evaluates LLMs' alignment with multi-dimensional human value preferences across 75 countries. MVPBench contains 24,020 high-quality instances annotated with fine-grained value labels, personalized questions, and rich demographic metadata, making it the most comprehensive resource of its kind to date. Using MVPBench, we conduct an in-depth analysis of several state-of-the-art LLMs, revealing substantial disparities in alignment performance across geographic and demographic lines. We further demonstrate that lightweight fine-tuning methods, such as Low-Rank Adaptation (LoRA) and Direct Preference Optimization (DPO), can significantly enhance value alignment in both in-domain and out-of-domain settings. Our findings underscore the necessity for population-aware alignment evaluation and provide actionable insights for building culturally adaptive and value-sensitive LLMs. MVPBench serves as a practical foundation for future research on global alignment, personalized value modeling, and equitable AI development.
• 摘要：大语言模型（LLM）与人类价值观的一致性对于在各种用户群体中的安全有效部署至关重要。但是，现有的基准通常会忽略文化和人口多样性，从而有限地了解价值对齐方式如何在全球范围内推广。在这项工作中，我们介绍了MVPbench，这是一个新颖的基准，该基准系统地评估了LLMS与75个国家 /地区多维人类价值偏好的一致性。 MVPbench包含24,020个高质量的实例，该实例带有精细的价值标签，个性化问题和丰富的人群元数据，使其成为迄今为止最全面的资源。使用MVPBench，我们对几个最先进的LLM进行了深入的分析，揭示了跨地理和人口统计线之间的一致性差异。我们进一步证明，轻巧的微调方法，例如低级别适应性（LORA）和直接偏好优化（DPO），可以显着增强内域和室外设置中的价值对齐。我们的发现强调了人口感知的一致性评估的必要性，并为建立具有文化适应性和价值敏感的LLM的可行见解提供了可行的见解。 MVPBench是全球一致性，个性化价值建模和公平AI开发的未来研究的实践基础。

Title: NOWJ@COLIEE 2025: A Multi-stage Framework Integrating Embedding Models and Large Language Models for Legal Retrieval and Entailment

• Authors: Hoang-Trung Nguyen, Tan-Minh Nguyen, Xuan-Bach Le, Tuan-Kiet Le, Khanh-Huyen Nguyen, Ha-Thanh Nguyen, Thi-Hai-Yen Vuong, Le-Minh Nguyen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2509.08025
• Pdf URL: https://arxiv.org/pdf/2509.08025
• Copy Paste: [[2509.08025]] NOWJ@COLIEE 2025: A Multi-stage Framework Integrating Embedding Models and Large Language Models for Legal Retrieval and Entailment(https://arxiv.org/abs/2509.08025)
• Keywords: language model, llm, prompt
• Abstract: This paper presents the methodologies and results of the NOWJ team's participation across all five tasks at the COLIEE 2025 competition, emphasizing advancements in the Legal Case Entailment task (Task 2). Our comprehensive approach systematically integrates pre-ranking models (BM25, BERT, monoT5), embedding-based semantic representations (BGE-m3, LLM2Vec), and advanced Large Language Models (Qwen-2, QwQ-32B, DeepSeek-V3) for summarization, relevance scoring, and contextual re-ranking. Specifically, in Task 2, our two-stage retrieval system combined lexical-semantic filtering with contextualized LLM analysis, achieving first place with an F1 score of 0.3195. Additionally, in other tasks--including Legal Case Retrieval, Statute Law Retrieval, Legal Textual Entailment, and Legal Judgment Prediction--we demonstrated robust performance through carefully engineered ensembles and effective prompt-based reasoning strategies. Our findings highlight the potential of hybrid models integrating traditional IR techniques with contemporary generative models, providing a valuable reference for future advancements in legal information processing.
• 摘要：本文介绍了NowJ团队在Coliee 2025竞赛中参与所有五项任务的方法和结果，强调了法律案件构成任务的进步（任务2）。我们的全面方法系统地集成了预先级别模型（BM25，BERT，MONOT5），基于嵌入的语义表示（BGE-M3，LLM2VEC）和先进的大语言模型（QWEN-2，QWQ-32B，DEEPSEEK-V3），用于汇总，相关评分评分，以及上下文重复。具体而言，在任务2中，我们的两阶段检索系统将词汇语义过滤与上下文化的LLM分析相结合，以0.3195的F1分数获得第一名。此外，在其他任务中 - 包括法律案件检索，法规法检索，法律文本要求和法律判断预测 - 我们通过精心设计的合奏和有效的及时基于及时的推理策略证明了良好的绩效。我们的发现突出了混合模型将传统IR技术与当代生成模型相结合的潜力，为未来的法律信息处理方面的进步提供了宝贵的参考。

Title: SciGPT: A Large Language Model for Scientific Literature Understanding and Knowledge Discovery

• Authors: Fengyu She, Nan Wang, Hongfei Wu, Ziyi Wan, Jingmian Wang, Chang Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2509.08032
• Pdf URL: https://arxiv.org/pdf/2509.08032
• Copy Paste: [[2509.08032]] SciGPT: A Large Language Model for Scientific Literature Understanding and Knowledge Discovery(https://arxiv.org/abs/2509.08032)
• Keywords: language model, gpt, llm
• Abstract: Scientific literature is growing exponentially, creating a critical bottleneck for researchers to efficiently synthesize knowledge. While general-purpose Large Language Models (LLMs) show potential in text processing, they often fail to capture scientific domain-specific nuances (e.g., technical jargon, methodological rigor) and struggle with complex scientific tasks, limiting their utility for interdisciplinary research. To address these gaps, this paper presents SciGPT, a domain-adapted foundation model for scientific literature understanding and ScienceBench, an open source benchmark tailored to evaluate scientific LLMs. Built on the Qwen3 architecture, SciGPT incorporates three key innovations: (1) low-cost domain distillation via a two-stage pipeline to balance performance and efficiency; (2) a Sparse Mixture-of-Experts (SMoE) attention mechanism that cuts memory consumption by 55\% for 32,000-token long-document reasoning; and (3) knowledge-aware adaptation integrating domain ontologies to bridge interdisciplinary knowledge gaps. Experimental results on ScienceBench show that SciGPT outperforms GPT-4o in core scientific tasks including sequence labeling, generation, and inference. It also exhibits strong robustness in unseen scientific tasks, validating its potential to facilitate AI-augmented scientific discovery.
• 摘要：科学文献正在成倍增长，为研究人员有效地综合知识创造了关键的瓶颈。尽管通用大语模型（LLMS）在文本处理中显示出潜力，但它们通常无法捕获科学领域特定的细微差别（例如，技术术语，方法论严格），并在复杂的科学任务中挣扎，从而限制了跨学科研究的实用性。为了解决这些差距，本文介绍了SCIGPT，这是科学文献理解和科学台的域适应基础模型，这是一种用于评估科学LLM的开源基准。 SCIGPT建立在QWEN3体系结构上，结合了三个关键创新：（1）通过两阶段管道来平衡性能和效率的低成本域蒸馏； （2）稀疏的专家混合物（SMOE）注意机制，将记忆消耗减少55 \％，以32,000 token token的长期论点； （3）知识意识适应整合领域的本体论以弥合跨学科知识差距。 Science Bench的实验结果表明，SCIGPT在包括序列标记，生成和推理在内的核心科学任务中优于GPT-4O。它还在看不见的科学任务中表现出强大的鲁棒性，从而验证了其促进AI促进科学发现的潜力。

Title: No for Some, Yes for Others: Persona Prompts and Other Sources of False Refusal in Language Models

• Authors: Flor Miriam Plaza-del-Arco, Paul Röttger, Nino Scherrer, Emanuele Borgonovo, Elmar Plischke, Dirk Hovy
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2509.08075
• Pdf URL: https://arxiv.org/pdf/2509.08075
• Copy Paste: [[2509.08075]] No for Some, Yes for Others: Persona Prompts and Other Sources of False Refusal in Language Models(https://arxiv.org/abs/2509.08075)
• Keywords: language model, llm, prompt
• Abstract: Large language models (LLMs) are increasingly integrated into our daily lives and personalized. However, LLM personalization might also increase unintended side effects. Recent work suggests that persona prompting can lead models to falsely refuse user requests. However, no work has fully quantified the extent of this issue. To address this gap, we measure the impact of 15 sociodemographic personas (based on gender, race, religion, and disability) on false refusal. To control for other factors, we also test 16 different models, 3 tasks (Natural Language Inference, politeness, and offensiveness classification), and nine prompt paraphrases. We propose a Monte Carlo-based method to quantify this issue in a sample-efficient manner. Our results show that as models become more capable, personas impact the refusal rate less and less. Certain sociodemographic personas increase false refusal in some models, which suggests underlying biases in the alignment strategies or safety mechanisms. However, we find that the model choice and task significantly influence false refusals, especially in sensitive content tasks. Our findings suggest that persona effects have been overestimated, and might be due to other factors.
• 摘要：大型语言模型（LLM）越来越多地整合到我们的日常生活中并个性化。但是，LLM个性化也可能增加意外的副作用。最近的工作表明，角色提示可以导致模型错误地拒绝用户请求。但是，没有任何工作完全量化了这个问题的程度。为了解决这一差距，我们衡量了15个社会人口统计角色（基于性别，种族，宗教和残疾）对虚假拒绝的影响。为了控制其他因素，我们还测试了16个不同的模型，3个任务（自然语言推断，礼貌和进攻性分类）和9个及时的释义。我们提出了一种基于蒙特卡洛的方法，以样品有效的方式量化此问题。我们的结果表明，随着模型变得越来越有能力，角色会影响拒绝率越来越少。在某些模型中，某些社会人口统计学的角色增加了错误的拒绝，这表明对齐策略或安全机制的基本偏见。但是，我们发现模型选择和任务显着影响错误的拒绝，尤其是在敏感内容任务中。我们的发现表明，角色效应被高估了，可能是由于其他因素所致。

Title: Culturally transmitted color categories in LLMs reflect a learning bias toward efficient compression

• Authors: Nathaniel Imel, Noga Zaslavsky
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2509.08093
• Pdf URL: https://arxiv.org/pdf/2509.08093
• Copy Paste: [[2509.08093]] Culturally transmitted color categories in LLMs reflect a learning bias toward efficient compression(https://arxiv.org/abs/2509.08093)
• Keywords: language model, llm
• Abstract: Converging evidence suggests that systems of semantic categories across human languages achieve near-optimal compression via the Information Bottleneck (IB) complexity-accuracy principle. Large language models (LLMs) are not trained for this objective, which raises the question: are LLMs capable of evolving efficient human-like semantic systems? To address this question, we focus on the domain of color as a key testbed of cognitive theories of categorization and replicate with LLMs (Gemini 2.0-flash and Llama 3.3-70B-Instruct) two influential human behavioral studies. First, we conduct an English color-naming study, showing that Gemini aligns well with the naming patterns of native English speakers and achieves a significantly high IB-efficiency score, while Llama exhibits an efficient but lower complexity system compared to English. Second, to test whether LLMs simply mimic patterns in their training data or actually exhibit a human-like inductive bias toward IB-efficiency, we simulate cultural evolution of pseudo color-naming systems in LLMs via iterated in-context language learning. We find that akin to humans, LLMs iteratively restructure initially random systems towards greater IB-efficiency and increased alignment with patterns observed across the world's languages. These findings demonstrate that LLMs are capable of evolving perceptually grounded, human-like semantic systems, driven by the same fundamental principle that governs semantic efficiency across human languages.
• 摘要：融合的证据表明，跨人类语言的语义类别系统通过信息瓶颈（IB）复杂性 - 精确原理实现了近乎最佳的压缩。大型语言模型（LLMS）未接受此目标的培训，这提出了一个问题：LLM是否能够发展出有效的人类语义系统？为了解决这个问题，我们将重点放在颜色领域，作为分类的认知理论的键检测，并用LLMS（Gemini 2.0-Flash和Llama 3.3-70B-Instruct）复制两个有影响力的人类行为研究。首先，我们进行了一项英语颜色命名研究，表明双子座与英语英语的人的命名模式很好地保持一致，并取得了显着高的IB效率得分，而与英语相比，Llama表现出有效但较低的复杂性系统。其次，为了测试LLMS是在其训练数据中模仿模式还是实际上表现出对IB效率的人类感应偏见，我们通过迭代的内在语言学习模拟了LLMS中伪色命名系统的文化进化。我们发现，类似于人类，LLMS迭代重组最初是随机的系统，以提高IB效率，并增加与世界语言中观察到的模式的一致性。这些发现表明，LLM能够发展出感知的基础，类似人类的语义系统，这是由控制人类语言跨语言效率的相同基本原则的驱动。

Title: MERLIN: Multi-Stage Curriculum Alignment for Multilingual Encoder and LLM Fusion

• Authors: Kosei Uemura, David Guzmán, Quang Phuoc Nguyen, Jesujoba Oluwadara Alabi, En-shiun Annie Lee, David Ifeoluwa Adelani
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2509.08105
• Pdf URL: https://arxiv.org/pdf/2509.08105
• Copy Paste: [[2509.08105]] MERLIN: Multi-Stage Curriculum Alignment for Multilingual Encoder and LLM Fusion(https://arxiv.org/abs/2509.08105)
• Keywords: language model, gpt, llm
• Abstract: Large language models excel in English but still struggle with complex reasoning in many low-resource languages (LRLs). Existing encoder-plus-decoder methods such as LangBridge and MindMerger raise accuracy on mid and high-resource languages, yet they leave a large gap on LRLs. We present MERLIN, a two-stage model-stacking framework that applies a curriculum learning strategy -- from general bilingual bitext to task-specific data -- and adapts only a small set of DoRA weights. On the AfriMGSM benchmark MERLIN improves exact-match accuracy by +12.9 pp over MindMerger and outperforms GPT-4o-mini. It also yields consistent gains on MGSM and MSVAMP (+0.9 and +2.8 pp), demonstrating effectiveness across both low and high-resource settings.
• 摘要：大型语言模型在英语中脱颖而出，但仍在许多低资源语言（LRLS）中与复杂的推理作斗争。现有的编码器加编码器方法，例如Langbridge和Mindmerger提高了中和高资源语言的准确性，但它们在LRL上留下了很大的差距。我们提出了Merlin是一个两阶段的模型堆栈框架，该框架应用了课程学习策略 - 从一般双语Bitext到特定于任务的数据 - 仅适应一小部分的Dora权重。在AFRIMGSM基准上，Merlin比Mindmerger提高了+12.9 pp的精确匹配精度，并且优于GPT-4O-Mini。它还可以在MGSM和MSVAMP（+0.9和+2.8 pp）上获得一致的收益，这表明了低资源和高资源设置的有效性。

Title: Bias after Prompting: Persistent Discrimination in Large Language Models

• Authors: Nivedha Sivakumar, Natalie Mackraz, Samira Khorshidi, Krishna Patel, Barry-John Theobald, Luca Zappella, Nicholas Apostoloff
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2509.08146
• Pdf URL: https://arxiv.org/pdf/2509.08146
• Copy Paste: [[2509.08146]] Bias after Prompting: Persistent Discrimination in Large Language Models(https://arxiv.org/abs/2509.08146)
• Keywords: language model, llm, prompt
• Abstract: A dangerous assumption that can be made from prior work on the bias transfer hypothesis (BTH) is that biases do not transfer from pre-trained large language models (LLMs) to adapted models. We invalidate this assumption by studying the BTH in causal models under prompt adaptations, as prompting is an extremely popular and accessible adaptation strategy used in real-world applications. In contrast to prior work, we find that biases can transfer through prompting and that popular prompt-based mitigation methods do not consistently prevent biases from transferring. Specifically, the correlation between intrinsic biases and those after prompt adaptation remain moderate to strong across demographics and tasks -- for example, gender (rho >= 0.94) in co-reference resolution, and age (rho >= 0.98) and religion (rho >= 0.69) in question answering. Further, we find that biases remain strongly correlated when varying few-shot composition parameters, such as sample size, stereotypical content, occupational distribution and representational balance (rho >= 0.90). We evaluate several prompt-based debiasing strategies and find that different approaches have distinct strengths, but none consistently reduce bias transfer across models, tasks or demographics. These results demonstrate that correcting bias, and potentially improving reasoning ability, in intrinsic models may prevent propagation of biases to downstream tasks.
• 摘要：先前关于偏见转移假设（BTH）的危险假设是，偏见不会从预训练的大语言模型（LLM）转移到适应模型。我们通过在迅速适应下研究因果模型中的BTH来使这一假设无效，因为提示是一种非常流行且易于访问的适应策略。与先前的工作相反，我们发现偏见可以通过提示转移，而流行的基于及时的缓解方法并不能始终防止偏见转移。具体而言，固有偏见与迅速适应后的偏见之间的相关性在跨人口统计和任务之间保持中等至强度，例如，在共同参考分辨率方面的性别（Rho> = 0.94）以及年龄（Rho> = 0.98）和宗教（Rho> = 0.69）（rho> = 0.69）。此外，我们发现当改变几个弹药组成参数（例如样本量，刻板印象内容，职业分布和代表性平衡）时，偏差保持密切相关（rho> = 0.90）。我们评估了几种基于迅速的证据策略，并发现不同的方法具有不同的优势，但是没有一贯降低跨模型，任务或人口统计数据的偏差转移。这些结果表明，在固有模型中，纠正偏差和潜在提高推理能力可能会阻止偏见传播到下游任务。

Title: Verbalized Algorithms

• Authors: Supriya Lall, Christian Farrell, Hari Pathanjaly, Marko Pavic, Sarvesh Chezhian, Masataro Asai
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2509.08150
• Pdf URL: https://arxiv.org/pdf/2509.08150
• Copy Paste: [[2509.08150]] Verbalized Algorithms(https://arxiv.org/abs/2509.08150)
• Keywords: llm
• Abstract: Instead of querying LLMs in a one-shot manner and hoping to get the right answer for a reasoning task, we propose a paradigm we call \emph{verbalized algorithms} (VAs), which leverage classical algorithms with established theoretical understanding. VAs decompose a task into simple elementary operations on natural language strings that they should be able to answer reliably, and limit the scope of LLMs to only those simple tasks. For example, for sorting a series of natural language strings, \emph{verbalized sorting} uses an LLM as a binary comparison oracle in a known and well-analyzed sorting algorithm (e.g., bitonic sorting network). We demonstrate the effectiveness of this approach on sorting and clustering tasks.
• 摘要：我们没有以一种镜头的方式查询LLM，并希望为推理任务找到正确的答案，而是提出了一个范式，我们称为\ emph {forbalized算法}（VAS），该（VAS）以既定的理论理解来利用经典的算法。 VA将任务分解为自然语言字符串的简单基本操作，它们应该能够可靠地回答，并将LLM的范围限制在这些简单的任务中。例如，为了对一系列自然语言字符串进行排序，\ emph {forbalized corting}将llm用作二进制比较的甲骨文，以在已知且分析良好的排序算法（例如，比特式分类网络）中。我们证明了这种方法对分类和聚类任务的有效性。

Title: Towards Knowledge-Aware Document Systems: Modeling Semantic Coverage Relations via Answerability Detection

• Authors: Yehudit Aperstein, Alon Gottlib, Gal Benita, Alexander Apartsin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2509.08304
• Pdf URL: https://arxiv.org/pdf/2509.08304
• Copy Paste: [[2509.08304]] Towards Knowledge-Aware Document Systems: Modeling Semantic Coverage Relations via Answerability Detection(https://arxiv.org/abs/2509.08304)
• Keywords: language model
• Abstract: Understanding how information is shared across documents, regardless of the format in which it is expressed, is critical for tasks such as information retrieval, summarization, and content alignment. In this work, we introduce a novel framework for modelling Semantic Coverage Relations (SCR), which classifies document pairs based on how their informational content aligns. We define three core relation types: equivalence, where both texts convey the same information using different textual forms or styles; inclusion, where one document fully contains the information of another and adds more; and semantic overlap, where each document presents partially overlapping content. To capture these relations, we adopt a question answering (QA)-based approach, using the answerability of shared questions across documents as an indicator of semantic coverage. We construct a synthetic dataset derived from the SQuAD corpus by paraphrasing source passages and selectively omitting information, enabling precise control over content overlap. This dataset allows us to benchmark generative language models and train transformer-based classifiers for SCR prediction. Our findings demonstrate that discriminative models significantly outperform generative approaches, with the RoBERTa-base model achieving the highest accuracy of 61.4% and the Random Forest-based model showing the best balance with a macro-F1 score of 52.9%. The results show that QA provides an effective lens for assessing semantic relations across stylistically diverse texts, offering insights into the capacity of current models to reason about information beyond surface similarity. The dataset and code developed in this study are publicly available to support reproducibility.
• 摘要：了解如何在文档中共享信息，无论其表达的格式如何，对于诸如信息检索，摘要和内容一致性等任务至关重要。在这项工作中，我们介绍了一个新颖的框架，用于建模语义覆盖关系（SCR），该框架根据其信息内容的对齐方式对文档对进行了分类。我们定义了三种核心关系类型：等效性，其中两个文本都使用不同的文本形式或样式传达相同的信息；包含在其中，其中一个文档完全包含另一个文档的信息，并添加了更多信息；和语义重叠，每个文档都会呈现部分重叠的内容。为了捕获这些关系，我们使用跨文档共享问题的回答性来回答基于问题的方法（QA）的方法。我们通过解释源段落并有选择地省略信息来构建一个从小队语料库得出的合成数据集，从而可以精确控制内容重叠。该数据集使我们可以基于SCR预测的基于训练变压器的分类器基准基准的生成语言模型。我们的发现表明，歧视模型的表现明显优于生成方法，而罗伯塔基本模型的精度最高为61.4％，基于森林的随机模型显示出最佳平衡，宏F1得分为52.9％。结果表明，质量检查提供了一种有效的镜头，用于评估跨风格多样的文本的语义关系，从而提供了有关当前模型的能力，以推理超出表面相似性的信息的能力。本研究中开发的数据集和代码可公开支持可重复性。

Title: Toward Subtrait-Level Model Explainability in Automated Writing Evaluation

• Authors: Alejandro Andrade-Lotero, Lee Becker, Joshua Southerland, Scott Hellman
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2509.08345
• Pdf URL: https://arxiv.org/pdf/2509.08345
• Copy Paste: [[2509.08345]] Toward Subtrait-Level Model Explainability in Automated Writing Evaluation(https://arxiv.org/abs/2509.08345)
• Keywords: language model
• Abstract: Subtrait (latent-trait components) assessment presents a promising path toward enhancing transparency of automated writing scores. We prototype explainability and subtrait scoring with generative language models and show modest correlation between human subtrait and trait scores, and between automated and human subtrait scores. Our approach provides details to demystify scores for educators and students.
• 摘要：子特征（潜在特征组成部分）评估为提高自动写作分数的透明度提供了有希望的途径。我们以生成语言模型的形式原型解释性和亚特征评分，并显示人类亚特征和性状分数之间以及自动化和人类亚属性分数之间的适度相关性。我们的方法提供了详细信息，以使教育工作者和学生的分数神秘。

Title: So let's replace this phrase with insult... Lessons learned from generation of toxic texts with LLMs

• Authors: Sergey Pletenev, Daniil Moskovskiy, Alexander Panchenko
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2509.08358
• Pdf URL: https://arxiv.org/pdf/2509.08358
• Copy Paste: [[2509.08358]] So let's replace this phrase with insult... Lessons learned from generation of toxic texts with LLMs(https://arxiv.org/abs/2509.08358)
• Keywords: language model, llm
• Abstract: Modern Large Language Models (LLMs) are excellent at generating synthetic data. However, their performance in sensitive domains such as text detoxification has not received proper attention from the scientific community. This paper explores the possibility of using LLM-generated synthetic toxic data as an alternative to human-generated data for training models for detoxification. Using Llama 3 and Qwen activation-patched models, we generated synthetic toxic counterparts for neutral texts from ParaDetox and SST-2 datasets. Our experiments show that models fine-tuned on synthetic data consistently perform worse than those trained on human data, with a drop in performance of up to 30% in joint metrics. The root cause is identified as a critical lexical diversity gap: LLMs generate toxic content using a small, repetitive vocabulary of insults that fails to capture the nuances and variety of human toxicity. These findings highlight the limitations of current LLMs in this domain and emphasize the continued importance of diverse, human-annotated data for building robust detoxification systems.
• 摘要：现代大型语言模型（LLM）非常擅长生成合成数据。但是，它们在诸如文本排毒之类的敏感领域的表现尚未受到科学界的适当关注。本文探讨了使用LLM生成的合成有毒数据作为用于排毒训练模型的人类生成数据的一种可能性。使用Llama 3和QWEN激活绘制模型，我们生成了来自ParadeTox和SST-2数据集中性文本的合成有毒物质。我们的实验表明，对合成数据的微调模型始终如一地比接受人类数据训练的模型差，关节指标的性能下降了30％。根本原因被确定为一个关键的词汇多样性差距：LLMS使用侮辱性的小而重复的词汇产生有毒含量，而侮辱性的词汇未能捕获人类毒性的细微差别和种类。这些发现突出了该领域当前LLM的局限性，并强调了多种人类通知数据在构建强大的排毒系统中的持续重要性。

Title: Low-Resource Fine-Tuning for Multi-Task Structured Information Extraction with a Billion-Parameter Instruction-Tuned Model

• Authors: Yu Cheng Chih, Yong Hao Hou
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2509.08381
• Pdf URL: https://arxiv.org/pdf/2509.08381
• Copy Paste: [[2509.08381]] Low-Resource Fine-Tuning for Multi-Task Structured Information Extraction with a Billion-Parameter Instruction-Tuned Model(https://arxiv.org/abs/2509.08381)
• Keywords: language model, llm
• Abstract: Deploying large language models (LLMs) for structured data extraction in domains such as financial compliance reporting, legal document analytics, and multilingual knowledge base construction is often impractical for smaller teams due to the high cost of running large architectures and the difficulty of preparing large, high-quality datasets. Most recent instruction-tuning studies focus on seven-billion-parameter or larger models, leaving limited evidence on whether much smaller models can work reliably under low-resource, multi-task conditions. This work presents ETLCH, a billion-parameter LLaMA-based model fine-tuned with low-rank adaptation on only a few hundred to one thousand samples per task for JSON extraction, knowledge graph extraction, and named entity recognition. Despite its small scale, ETLCH outperforms strong baselines across most evaluation metrics, with substantial gains observed even at the lowest data scale. These findings demonstrate that well-tuned small models can deliver stable and accurate structured outputs at a fraction of the computational cost, enabling cost-effective and reliable information extraction pipelines in resource-constrained environments.
• 摘要：在财务合规报告，法律文档分析和多语言知识基础构建等领域中，部署大型语言模型（LLMS）对于较小的团队通常是不切实际的，这是由于运行大型建筑的高成本以及准备大型高质量数据集的困难，通常是不切实际的。最新的教学调查研究集中于70亿参数或更大的模型，留下有限的证据表明，在低资产资产，多任务条件下，较小的模型是否可以可靠地工作。这项工作介绍了Etlch，这是一种基于十亿参数的LLAMA模型，对每项任务仅几百到一千个样本，用于JSON提取，知识图提取和命名实体识别，并进行了低级适应。尽管它的规模很小，但ETLCH在大多数评估指标上的表现都优于强大的基准，即使在最低数据量表下也观察到了可观的收益。这些发现表明，经过良好调整的小型模型可以以一小部分计算成本提供稳定，准确的结构化输出，从而在资源受限的环境中实现具有成本效益和可靠的信息提取管道。

Title: CommonVoice-SpeechRE and RPG-MoGe: Advancing Speech Relation Extraction with a New Dataset and Multi-Order Generative Framework

• Authors: Jinzhong Ning, Paerhati Tulajiang, Yingying Le, Yijia Zhang, Yuanyuan Sun, Hongfei Lin, Haifeng Liu
• Subjects: cs.CL, cs.MM, cs.SD
• Abstract URL: https://arxiv.org/abs/2509.08438
• Pdf URL: https://arxiv.org/pdf/2509.08438
• Copy Paste: [[2509.08438]] CommonVoice-SpeechRE and RPG-MoGe: Advancing Speech Relation Extraction with a New Dataset and Multi-Order Generative Framework(https://arxiv.org/abs/2509.08438)
• Keywords: prompt
• Abstract: Speech Relation Extraction (SpeechRE) aims to extract relation triplets directly from speech. However, existing benchmark datasets rely heavily on synthetic data, lacking sufficient quantity and diversity of real human speech. Moreover, existing models also suffer from rigid single-order generation templates and weak semantic alignment, substantially limiting their performance. To address these challenges, we introduce CommonVoice-SpeechRE, a large-scale dataset comprising nearly 20,000 real-human speech samples from diverse speakers, establishing a new benchmark for SpeechRE research. Furthermore, we propose the Relation Prompt-Guided Multi-Order Generative Ensemble (RPG-MoGe), a novel framework that features: (1) a multi-order triplet generation ensemble strategy, leveraging data diversity through diverse element orders during both training and inference, and (2) CNN-based latent relation prediction heads that generate explicit relation prompts to guide cross-modal alignment and accurate triplet generation. Experiments show our approach outperforms state-of-the-art methods, providing both a benchmark dataset and an effective solution for real-world SpeechRE. The source code and dataset are publicly available at this https URL.
• 摘要：语音关系提取（SpeechRE）旨在直接从语音中提取关系。但是，现有的基准数据集在很大程度上依赖于综合数据，缺乏足够数量和真实人类语音的多样性。此外，现有模型还遭受了严格的单阶生成模板和弱语义对齐方式，从而实质上限制了它们的性能。为了应对这些挑战，我们介绍了CommonVoice-Speechre，这是一个大规模数据集，其中包括来自不同演讲者的近20,000个真实人类的语音样本，为Secemberre Research建立了新的基准。 Furthermore, we propose the Relation Prompt-Guided Multi-Order Generative Ensemble (RPG-MoGe), a novel framework that features: (1) a multi-order triplet generation ensemble strategy, leveraging data diversity through diverse element orders during both training and inference, and (2) CNN-based latent relation prediction heads that generate explicit relation prompts to guide cross-modal alignment and accurate triplet generation.实验表明，我们的方法的表现优于最先进的方法，提供了基准数据集和现实世界中的有效解决方案。源代码和数据集可在此HTTPS URL上公开可用。

Title: Acquiescence Bias in Large Language Models

• Authors: Daniel Braun
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2509.08480
• Pdf URL: https://arxiv.org/pdf/2509.08480
• Copy Paste: [[2509.08480]] Acquiescence Bias in Large Language Models(https://arxiv.org/abs/2509.08480)
• Keywords: language model, llm
• Abstract: Acquiescence bias, i.e. the tendency of humans to agree with statements in surveys, independent of their actual beliefs, is well researched and documented. Since Large Language Models (LLMs) have been shown to be very influenceable by relatively small changes in input and are trained on human-generated data, it is reasonable to assume that they could show a similar tendency. We present a study investigating the presence of acquiescence bias in LLMs across different models, tasks, and languages (English, German, and Polish). Our results indicate that, contrary to humans, LLMs display a bias towards answering no, regardless of whether it indicates agreement or disagreement.
• 摘要：默认偏见，即人类在调查中与其实际信念无关的陈述的趋势进行了充分的研究和记录。由于大型语言模型（LLM）已被证明受到相对较小的输入变化的影响，并且接受了人类生成数据的培训，因此可以合理地假设它们可以显示出类似的趋势。我们提出了一项研究，研究了不同模型，任务和语言（英语，德语和波兰语）中LLM中默认偏见的存在。我们的结果表明，与人类相反，LLM表现出对回答否定的偏见，无论它表示共识还是分歧。

Title: Simulating Identity, Propagating Bias: Abstraction and Stereotypes in LLM-Generated Text

• Authors: Pia Sommerauer, Giulia Rambelli, Tommaso Caselli
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2509.08484
• Pdf URL: https://arxiv.org/pdf/2509.08484
• Copy Paste: [[2509.08484]] Simulating Identity, Propagating Bias: Abstraction and Stereotypes in LLM-Generated Text(https://arxiv.org/abs/2509.08484)
• Keywords: llm, prompt
• Abstract: Persona-prompting is a growing strategy to steer LLMs toward simulating particular perspectives or linguistic styles through the lens of a specified identity. While this method is often used to personalize outputs, its impact on how LLMs represent social groups remains underexplored. In this paper, we investigate whether persona-prompting leads to different levels of linguistic abstraction - an established marker of stereotyping - when generating short texts linking socio-demographic categories with stereotypical or non-stereotypical attributes. Drawing on the Linguistic Expectancy Bias framework, we analyze outputs from six open-weight LLMs under three prompting conditions, comparing 11 persona-driven responses to those of a generic AI assistant. To support this analysis, we introduce Self-Stereo, a new dataset of self-reported stereotypes from Reddit. We measure abstraction through three metrics: concreteness, specificity, and negation. Our results highlight the limits of persona-prompting in modulating abstraction in language, confirming criticisms about the ecology of personas as representative of socio-demographic groups and raising concerns about the risk of propagating stereotypes even when seemingly evoking the voice of a marginalized group.
• 摘要：促进角色是一种越来越多的策略，可以通过指定身份的镜头来模拟特定观点或语言风格。尽管此方法通常用于个性化产出，但其对LLM代表社会群体的影响仍然没有得到充实的影响。在本文中，我们研究了促成角色的语言抽象的不同水平 - 刻板印象的既定标记 - 在生成将社会人口统计学类别与刻板印象或非疾病典型属性联系起来的简短文本时。利用语言预期偏见框架，我们在三个提示条件下分析了六个开放量LLMS的输出，将11个角色驱动的响应与通用AI助手的响应进行了比较。为了支持这一分析，我们介绍了自我报告的新数据集，这是Reddit的自我报告刻板印象的新数据集。我们通过三个指标测量抽象：具体性，特异性和否定。我们的结果强调了在语言中调制抽象的限制，确认对角色的生态学的批评，即使是社会人口统计学群体的代表，也引起了人们对传播刻板印象的风险的关注，即使看似唤起了一个边缘化群体的声音。

Title: Too Helpful, Too Harmless, Too Honest or Just Right?

• Authors: Gautam Siddharth Kashyap, Mark Dras, Usman Naseem
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2509.08486
• Pdf URL: https://arxiv.org/pdf/2509.08486
• Copy Paste: [[2509.08486]] Too Helpful, Too Harmless, Too Honest or Just Right?(https://arxiv.org/abs/2509.08486)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) exhibit strong performance across a wide range of NLP tasks, yet aligning their outputs with the principles of Helpfulness, Harmlessness, and Honesty (HHH) remains a persistent challenge. Existing methods often optimize for individual alignment dimensions in isolation, leading to trade-offs and inconsistent behavior. While Mixture-of-Experts (MoE) architectures offer modularity, they suffer from poorly calibrated routing, limiting their effectiveness in alignment tasks. We propose TrinityX, a modular alignment framework that incorporates a Mixture of Calibrated Experts (MoCaE) within the Transformer architecture. TrinityX leverages separately trained experts for each HHH dimension, integrating their outputs through a calibrated, task-adaptive routing mechanism that combines expert signals into a unified, alignment-aware representation. Extensive experiments on three standard alignment benchmarks-Alpaca (Helpfulness), BeaverTails (Harmlessness), and TruthfulQA (Honesty)-demonstrate that TrinityX outperforms strong baselines, achieving relative improvements of 32.5% in win rate, 33.9% in safety score, and 28.4% in truthfulness. In addition, TrinityX reduces memory usage and inference latency by over 40% compared to prior MoE-based approaches. Ablation studies highlight the importance of calibrated routing, and cross-model evaluations confirm TrinityX's generalization across diverse LLM backbones.
• 摘要：大型语言模型（LLMS）在各种NLP任务中表现出很强的表现，但是将其输出与乐于助人，无害和诚实（HHH）的原则保持一致仍然是一个持续的挑战。现有的方法通常会孤立地针对单个对齐维度进行优化，从而导致权衡和不一致的行为。尽管Experts（MOE）架构的混合物提供模块化，但它们的校准路由不足，从而限制了它们在对齐任务中的有效性。我们提出了Trinityx，这是一个模块化对齐框架，该框架结合了变压器体系结构中的校准专家（MOCAE）的混合物。 Trinityx为每个HHH维度分别训练了专家，通过校准的，任务自适应的路由机制集成了其输出，该方法将专家信号结合到统一的，对齐感知的表示形式。对三个标准对齐基准-Alpaca（有帮助），海狸（无害）和真实性（诚实）（诚实）进行的广泛实验 - 表明Trinityx的表现优于强大的基准，在胜利率中实现了32.5％的相对提高，在安全得分中为33.9％，实际上为28.4％。此外，与先前的基于MOE的方法相比，Trinityx将记忆使用和推理潜伏期降低了40％以上。消融研究强调了校准路由的重要性，跨模型评估证实了Trinityx在不同LLM骨架上的概括。

Title: CM-Align: Consistency-based Multilingual Alignment for Large Language Models

• Authors: Xue Zhang, Yunlong Liang, Fandong Meng, Songming Zhang, Yufeng Chen, Jinan Xu, Jie Zhou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2509.08541
• Pdf URL: https://arxiv.org/pdf/2509.08541
• Copy Paste: [[2509.08541]] CM-Align: Consistency-based Multilingual Alignment for Large Language Models(https://arxiv.org/abs/2509.08541)
• Keywords: language model, llm
• Abstract: Current large language models (LLMs) generally show a significant performance gap in alignment between English and other languages. To bridge this gap, existing research typically leverages the model's responses in English as a reference to select the best/worst responses in other languages, which are then used for Direct Preference Optimization (DPO) training. However, we argue that there are two limitations in the current methods that result in noisy multilingual preference data and further limited alignment performance: 1) Not all English responses are of high quality, and using a response with low quality may mislead the alignment for other languages. 2) Current methods usually use biased or heuristic approaches to construct multilingual preference pairs. To address these limitations, we design a consistency-based data selection method to construct high-quality multilingual preference data for improving multilingual alignment (CM-Align). Specifically, our method includes two parts: consistency-guided English reference selection and cross-lingual consistency-based multilingual preference data construction. Experimental results on three LLMs and three common tasks demonstrate the effectiveness and superiority of our method, which further indicates the necessity of constructing high-quality preference data.
• 摘要：当前的大型语言模型（LLM）通常在英语和其他语言之间的对齐方式上表现出显着的性能差距。为了弥合这一差距，现有的研究通常利用该模型的英语响应来参考选择其他语言中最佳/最差的响应，然后将其用于直接偏好优化（DPO）培训。但是，我们认为当前方法中有两个局限性导致嘈杂的多语言偏好数据和进一步的限制对齐性能：1）并非所有英语响应都具有高质量，并且使用低质量的响应可能会误导其他语言的对齐方式。 2）当前方法通常使用有偏见或启发式方法来构建多语言偏好对。为了解决这些局限性，我们设计了一种基于一致性的数据选择方法，以构建高质量的多语言偏好数据，以改善多语言对准（CM-Align）。具体而言，我们的方法包括两个部分：一致性指导的英语参考选择和跨语性一致性的多语言偏好数据构建。对三个LLM和三个常见任务的实验结果证明了我们方法的有效性和优势，这进一步表明了构建高质量偏好数据的必要性。

Title: LLM Ensemble for RAG: Role of Context Length in Zero-Shot Question Answering for BioASQ Challenge

• Authors: Dima Galat, Diego Molla-Aliod
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2509.08596
• Pdf URL: https://arxiv.org/pdf/2509.08596
• Copy Paste: [[2509.08596]] LLM Ensemble for RAG: Role of Context Length in Zero-Shot Question Answering for BioASQ Challenge(https://arxiv.org/abs/2509.08596)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: Biomedical question answering (QA) poses significant challenges due to the need for precise interpretation of specialized knowledge drawn from a vast, complex, and rapidly evolving corpus. In this work, we explore how large language models (LLMs) can be used for information retrieval (IR), and an ensemble of zero-shot models can accomplish state-of-the-art performance on a domain-specific Yes/No QA task. Evaluating our approach on the BioASQ challenge tasks, we show that ensembles can outperform individual LLMs and in some cases rival or surpass domain-tuned systems - all while preserving generalizability and avoiding the need for costly fine-tuning or labeled data. Our method aggregates outputs from multiple LLM variants, including models from Anthropic and Google, to synthesize more accurate and robust answers. Moreover, our investigation highlights a relationship between context length and performance: while expanded contexts are meant to provide valuable evidence, they simultaneously risk information dilution and model disorientation. These findings emphasize IR as a critical foundation in Retrieval-Augmented Generation (RAG) approaches for biomedical QA systems. Precise, focused retrieval remains essential for ensuring LLMs operate within relevant information boundaries when generating answers from retrieved documents. Our results establish that ensemble-based zero-shot approaches, when paired with effective RAG pipelines, constitute a practical and scalable alternative to domain-tuned systems for biomedical question answering.
• 摘要：生物医学问题回答（QA）构成了重大挑战，因为需要精确解释从广阔，复杂且快速发展的语料库中得出的专业知识。在这项工作中，我们探讨了如何将大型语言模型（LLM）用于信息检索（IR），并且零击模型的合奏可以在特定于领域的是/否质量质量检查任务上实现最先进的性能。评估我们在BioASQ挑战任务上的方法，我们表明合奏可以超越单个LLM，在某些情况下可以竞争或超越域调整的系统 - 同时保留了普遍性，并避免需要昂贵的微调或标记数据。我们的方法汇总了来自多个LLM变体的输出，包括拟人化和Google的模型，以合成更准确，更强大的答案。此外，我们的调查突出了上下文长度与绩效之间的关系：虽然扩展的上下文旨在提供宝贵的证据，但它们同时冒着信息稀释和模型迷失方向的风险。这些发现强调IR是生物医学质量检查系统的检索增强生成（RAG）方法的关键基础。精确的，重点的检索对于确保LLM在从检索文档中生成答案时在相关信息边界内运行至关重要。我们的结果表明，与有效的抹布管道配对时，基于整体的零击方法构成了用于生物医学问题回答的域调整系统的实用且可扩展的替代方法。

Title: Memorization in Large Language Models in Medicine: Prevalence, Characteristics, and Implications

• Authors: Anran Li, Lingfei Qian, Mengmeng Du, Yu Yin, Yan Hu, Zihao Sun, Yihang Fu, Erica Stutz, Xuguang Ai, Qianqian Xie, Rui Zhu, Jimin Huang, Yifan Yang, Siru Liu, Yih-Chung Tham, Lucila Ohno-Machado, Hyunghoon Cho, Zhiyong Lu, Hua Xu, Qingyu Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2509.08604
• Pdf URL: https://arxiv.org/pdf/2509.08604
• Copy Paste: [[2509.08604]] Memorization in Large Language Models in Medicine: Prevalence, Characteristics, and Implications(https://arxiv.org/abs/2509.08604)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have demonstrated significant potential in medicine. To date, LLMs have been widely applied to tasks such as diagnostic assistance, medical question answering, and clinical information synthesis. However, a key open question remains: to what extent do LLMs memorize medical training data. In this study, we present the first comprehensive evaluation of memorization of LLMs in medicine, assessing its prevalence (how frequently it occurs), characteristics (what is memorized), volume (how much content is memorized), and potential downstream impacts (how memorization may affect medical applications). We systematically analyze common adaptation scenarios: (1) continued pretraining on medical corpora, (2) fine-tuning on standard medical benchmarks, and (3) fine-tuning on real-world clinical data, including over 13,000 unique inpatient records from Yale New Haven Health System. The results demonstrate that memorization is prevalent across all adaptation scenarios and significantly higher than reported in the general domain. Memorization affects both the development and adoption of LLMs in medicine and can be categorized into three types: beneficial (e.g., accurate recall of clinical guidelines and biomedical references), uninformative (e.g., repeated disclaimers or templated medical document language), and harmful (e.g., regeneration of dataset-specific or sensitive clinical content). Based on these findings, we offer practical recommendations to facilitate beneficial memorization that enhances domain-specific reasoning and factual accuracy, minimize uninformative memorization to promote deeper learning beyond surface-level patterns, and mitigate harmful memorization to prevent the leakage of sensitive or identifiable patient information.
• 摘要：大型语言模型（LLM）在医学方面表现出了巨大的潜力。迄今为止，LLM已被广泛应用于诊断辅助，医学问答和临床信息综合等任务。但是，一个关键的开放问题仍然存在：LLM在多大程度上记住医疗培训数据。在这项研究中，我们介绍了对LLM在医学中的记忆的首次全面评估，评估了其患病率（发生频率），特征（记忆的内容），体积（记忆的内容有多少）以及潜在的下游影响（记忆如何影响医疗应用）。我们系统地分析了常见的适应方案：（1）在医疗文献中继续进行预修，（2）对标准医疗基准进行微调，以及（3）对现实世界中的临床数据进行微调，包括耶鲁纽黑文健康系统的13,000多个独特的住院记录。结果表明，在所有适应情景中，记忆都是普遍存在的，并且明显高于一般域中报道的。记忆会影响医学中LLM的开发和采用，可以分为三种类型：有益（例如，准确回忆临床准则和生物医学参考文献），非信息性（例如，重复免责或模板的医学文档语言）和危害（例如，危害性或危险）（例如，危险的临床持续性或敏感的临床内容）。基于这些发现，我们提供了实用的建议来促进有益的记忆，从而增强了特定于领域的推理和事实准确性，最大程度地减少了非信息性记忆，以促进更深入的学习，以超越表面层面的模式，并减轻有害的记忆，以防止敏感或可识别的患者信息的泄漏。

Title: X-Teaming Evolutionary M2S: Automated Discovery of Multi-turn to Single-turn Jailbreak Templates

• Authors: Hyunjun Kim, Junwoo Ha, Sangyoon Yu, Haon Park
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2509.08729
• Pdf URL: https://arxiv.org/pdf/2509.08729
• Copy Paste: [[2509.08729]] X-Teaming Evolutionary M2S: Automated Discovery of Multi-turn to Single-turn Jailbreak Templates(https://arxiv.org/abs/2509.08729)
• Keywords: gpt, llm, prompt
• Abstract: Multi-turn-to-single-turn (M2S) compresses iterative red-teaming into one structured prompt, but prior work relied on a handful of manually written templates. We present X-Teaming Evolutionary M2S, an automated framework that discovers and optimizes M2S templates through language-model-guided evolution. The system pairs smart sampling from 12 sources with an LLM-as-judge inspired by StrongREJECT and records fully auditable logs. Maintaining selection pressure by setting the success threshold to $\theta = 0.70$, we obtain five evolutionary generations, two new template families, and 44.8% overall success (103/230) on GPT-4.1. A balanced cross-model panel of 2,500 trials (judge fixed) shows that structural gains transfer but vary by target; two models score zero at the same threshold. We also find a positive coupling between prompt length and score, motivating length-aware judging. Our results demonstrate that structure-level search is a reproducible route to stronger single-turn probes and underscore the importance of threshold calibration and cross-model evaluation. Code, configurations, and artifacts are available at this https URL.
• 摘要：多转向向单向的转弯（M2S）将迭代的红色团队压缩为一个结构化的提示，但先前的工作依赖于少数手动书面模板。我们提出X团队进化M2S，这是一个自动化的框架，通过语言模型引导的进化发现并优化了M2S模板。该系统将12个来源的智能采样与LLM-As-Gudge进行了启发，灵感来自强烈的练习，并记录了完全可审计的日志。通过将成功阈值设置为$ \ theta = 0.70 $来保持选择压力，我们获得了五个进化世代，两个新的模板家族和44.8％的总体成功（103/230）（103/230）。一个平衡的跨模型面板由2500次试验（法官固定）表明结构收益转移，但因目标而有所不同。两个模型在相同的阈值下得分为零。我们还发现迅速的长度和得分之间的积极耦合，激励长度感知的判断。我们的结果表明，结构级搜索是通往更强单转弯探针的可再现途径，并强调了阈值校准和跨模型评估的重要性。该HTTPS URL可用代码，配置和工件。

Title: Streaming Sequence-to-Sequence Learning with Delayed Streams Modeling

• Authors: Neil Zeghidour, Eugene Kharitonov, Manu Orsini, Václav Volhejn, Gabriel de Marmiesse, Edouard Grave, Patrick Pérez, Laurent Mazaré, Alexandre Défossez
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2509.08753
• Pdf URL: https://arxiv.org/pdf/2509.08753
• Copy Paste: [[2509.08753]] Streaming Sequence-to-Sequence Learning with Delayed Streams Modeling(https://arxiv.org/abs/2509.08753)
• Keywords: language model
• Abstract: We introduce Delayed Streams Modeling (DSM), a flexible formulation for streaming, multimodal sequence-to-sequence learning. Sequence-to-sequence generation is often cast in an offline manner, where the model consumes the complete input sequence before generating the first output timestep. Alternatively, streaming sequence-to-sequence rely on learning a policy for choosing when to advance on the input stream, or write to the output stream. DSM instead models already time-aligned streams with a decoder-only language model. By moving the alignment to a pre-processing step,and introducing appropriate delays between streams, DSM provides streaming inference of arbitrary output sequences, from any input combination, making it applicable to many sequence-to-sequence problems. In particular, given text and audio streams, automatic speech recognition (ASR) corresponds to the text stream being delayed, while the opposite gives a text-to-speech (TTS) model. We perform extensive experiments for these two major sequence-to-sequence tasks, showing that DSM provides state-of-the-art performance and latency while supporting arbitrary long sequences, being even competitive with offline baselines. Code, samples and demos are available at this https URL
• 摘要：我们介绍了延迟的流建模（DSM），这是一种用于流式，多模式序列到序列学习的灵活公式。序列到序列的生成通常是以离线方式施放的，在生成第一个输出时间段之前，模型会消耗完整的输入序列。另外，流序列到序列依赖于学习策略以选择何时在输入流上推进或写入输出流。 DSM相反，使用仅解码器语言模型模型已经与时间分配的流进行了建模。通过将对齐方式移至预处理步骤，并在流之间引入适当的延迟，DSM从任何输入组合中提供了任意输出序列的流推断，使其适用于许多序列到序列问题。特别是，给定文本和音频流，自动语音识别（ASR）对应于延迟的文本流，而相反的文本对语音（TTS）模型。我们对这两个主要序列到序列任务进行了广泛的实验，表明DSM提供了最先进的性能和延迟，同时支持任意长序列，甚至与离线基线竞争。代码，样品和演示可在此HTTPS URL上找到

Title: Do All Autoregressive Transformers Remember Facts the Same Way? A Cross-Architecture Analysis of Recall Mechanisms

• Authors: Minyeong Choe, Haehyun Cho, Changho Seo, Hyunil Kim
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2509.08778
• Pdf URL: https://arxiv.org/pdf/2509.08778
• Copy Paste: [[2509.08778]] Do All Autoregressive Transformers Remember Facts the Same Way? A Cross-Architecture Analysis of Recall Mechanisms(https://arxiv.org/abs/2509.08778)
• Keywords: language model, gpt
• Abstract: Understanding how Transformer-based language models store and retrieve factual associations is critical for improving interpretability and enabling targeted model editing. Prior work, primarily on GPT-style models, has identified MLP modules in early layers as key contributors to factual recall. However, it remains unclear whether these findings generalize across different autoregressive architectures. To address this, we conduct a comprehensive evaluation of factual recall across several models -- including GPT, LLaMA, Qwen, and DeepSeek -- analyzing where and how factual information is encoded and accessed. Consequently, we find that Qwen-based models behave differently from previous patterns: attention modules in the earliest layers contribute more to factual recall than MLP modules. Our findings suggest that even within the autoregressive Transformer family, architectural variations can lead to fundamentally different mechanisms of factual recall.
• 摘要：了解基于变压器的语言模型如何存储和检索事实关联对于改善可解释性和实现目标模型编辑至关重要。先前的工作主要是在GPT式模型上，已将早期层中的MLP模块确定为事实召回的关键因素。但是，尚不清楚这些发现是否跨越不同的自回归体系结构。为了解决这个问题，我们对包括GPT，Llama，Qwen和DeepSeek在内的几种模型进行了事实召回的全面评估，分析了事实信息的编码和访问方式。因此，我们发现基于QWEN的模型的行为与以前的模式不同：最早的层中的注意模块比MLP模块更大。我们的发现表明，即使在自回归的变压器家族中，建筑的变化也会导致事实召回的根本不同的机制。

Title: Evaluating LLMs Without Oracle Feedback: Agentic Annotation Evaluation Through Unsupervised Consistency Signals

• Authors: Cheng Chen, Haiyan Yin, Ivor Tsang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2509.08809
• Pdf URL: https://arxiv.org/pdf/2509.08809
• Copy Paste: [[2509.08809]] Evaluating LLMs Without Oracle Feedback: Agentic Annotation Evaluation Through Unsupervised Consistency Signals(https://arxiv.org/abs/2509.08809)
• Keywords: language model, llm, prompt, agent
• Abstract: Large Language Models (LLMs), when paired with prompt-based tasks, have significantly reduced data annotation costs and reliance on human annotators. However, evaluating the quality of their annotations remains challenging in dynamic, unsupervised environments where oracle feedback is scarce and conventional methods fail. To address this challenge, we propose a novel agentic annotation paradigm, where a student model collaborates with a noisy teacher (the LLM) to assess and refine annotation quality without relying on oracle feedback. The student model, acting as an unsupervised feedback mechanism, employs a user preference-based majority voting strategy to evaluate the consistency of the LLM outputs. To systematically measure the reliability of LLM-generated annotations, we introduce the Consistent and Inconsistent (CAI) Ratio, a novel unsupervised evaluation metric. The CAI Ratio not only quantifies the annotation quality of the noisy teacher under limited user preferences but also plays a critical role in model selection, enabling the identification of robust LLMs in dynamic, unsupervised environments. Applied to ten open-domain NLP datasets across four LLMs, the CAI Ratio demonstrates a strong positive correlation with LLM accuracy, establishing it as an essential tool for unsupervised evaluation and model selection in real-world settings.
• 摘要：大型语言模型（LLMS）与基于及时的任务配对时，数据注释成本和对人类注释者的依赖大大降低了。但是，在动态，无监督的环境中，评估其注释的质量仍然具有挑战性，在这种环境中，甲骨文反馈稀少并且常规方法失败。为了应对这一挑战，我们提出了一个新颖的座流注释范式，学生模型与嘈杂的老师（LLM）合作，不依赖Oracle反馈，以评估和完善注释质量。该学生模型充当无监督的反馈机制，采用基于用户的多数投票策略来评估LLM输出的一致性。为了系统地衡量LLM生成的注释的可靠性，我们引入了一致且不一致的（CAI）比率，这是一种新型的无监督评估度量。 CAI比不仅在有限的用户偏好下量化了嘈杂教师的注释质量，而且在模型选择中起着至关重要的作用，从而可以识别在动态，无监督的环境中稳健的LLM。 CAI比应用于四个LLM的十个开放域NLP数据集，表现出与LLM精度有很强的正相关性，将其确立为在现实世界中无监督评估和模型选择的必不可少的工具。

Title: Building High-Quality Datasets for Portuguese LLMs: From Common Crawl Snapshots to Industrial-Grade Corpora

• Authors: Thales Sales Almeida, Rodrigo Nogueira, Helio Pedrini
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2509.08824
• Pdf URL: https://arxiv.org/pdf/2509.08824
• Copy Paste: [[2509.08824]] Building High-Quality Datasets for Portuguese LLMs: From Common Crawl Snapshots to Industrial-Grade Corpora(https://arxiv.org/abs/2509.08824)
• Keywords: language model, llm
• Abstract: The performance of large language models (LLMs) is deeply influenced by the quality and composition of their training data. While much of the existing work has centered on English, there remains a gap in understanding how to construct effective training corpora for other languages. We explore scalable methods for building web-based corpora for LLMs. We apply them to build a new 120B token corpus in Portuguese that achieves competitive results to an industrial-grade corpus. Using a continual pretraining setup, we study how different data selection and preprocessing strategies affect LLM performance when transitioning a model originally trained in English to another language. Our findings demonstrate the value of language-specific filtering pipelines, including classifiers for education, science, technology, engineering, and mathematics (STEM), as well as toxic content. We show that adapting a model to the target language leads to performance improvements, reinforcing the importance of high-quality, language-specific data. While our case study focuses on Portuguese, our methods are applicable to other languages, offering insights for multilingual LLM development.
• 摘要：大型语言模型（LLM）的性能受到培训数据的质量和组成的深刻影响。尽管现有的许多工作都集中在英语上，但了解如何为其他语言构建有效的培训语料库仍然存在差距。我们探索用于为LLM构建基于Web的COLPORA的可扩展方法。我们将它们应用它们在葡萄牙人建立新的120B代币语料库，从而为工业级的语料库取得了竞争成果。使用持续的预处理设置，我们研究了不同的数据选择和预处理策略如何在过渡最初用英语培训的模型到另一种语言的模型时影响LLM的性能。我们的发现表明了语言特定的过滤管道的价值，包括教育，科学，技术，工程和数学（STEM）的分类器以及有毒内容。我们表明，将模型适应目标语言会导致性能提高，从而增强了高质量，特定语言数据的重要性。尽管我们的案例研究侧重于葡萄牙语，但我们的方法适用于其他语言，为多语言LLM开发提供了见解。

Title: Large Language Model Hacking: Quantifying the Hidden Risks of Using LLMs for Text Annotation

• Authors: Joachim Baumann, Paul Röttger, Aleksandra Urman, Albert Wendsjö, Flor Miriam Plaza-del-Arco, Johannes B. Gruber, Dirk Hovy
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2509.08825
• Pdf URL: https://arxiv.org/pdf/2509.08825
• Copy Paste: [[2509.08825]] Large Language Model Hacking: Quantifying the Hidden Risks of Using LLMs for Text Annotation(https://arxiv.org/abs/2509.08825)
• Keywords: language model, llm, prompt
• Abstract: Large language models (LLMs) are rapidly transforming social science research by enabling the automation of labor-intensive tasks like data annotation and text analysis. However, LLM outputs vary significantly depending on the implementation choices made by researchers (e.g., model selection, prompting strategy, or temperature settings). Such variation can introduce systematic biases and random errors, which propagate to downstream analyses and cause Type I, Type II, Type S, or Type M errors. We call this LLM hacking. We quantify the risk of LLM hacking by replicating 37 data annotation tasks from 21 published social science research studies with 18 different models. Analyzing 13 million LLM labels, we test 2,361 realistic hypotheses to measure how plausible researcher choices affect statistical conclusions. We find incorrect conclusions based on LLM-annotated data in approximately one in three hypotheses for state-of-the-art models, and in half the hypotheses for small language models. While our findings show that higher task performance and better general model capabilities reduce LLM hacking risk, even highly accurate models do not completely eliminate it. The risk of LLM hacking decreases as effect sizes increase, indicating the need for more rigorous verification of findings near significance thresholds. Our extensive analysis of LLM hacking mitigation techniques emphasizes the importance of human annotations in reducing false positive findings and improving model selection. Surprisingly, common regression estimator correction techniques are largely ineffective in reducing LLM hacking risk, as they heavily trade off Type I vs. Type II errors. Beyond accidental errors, we find that intentional LLM hacking is unacceptably simple. With few LLMs and just a handful of prompt paraphrases, anything can be presented as statistically significant.
• 摘要：大型语言模型（LLM）通过实现劳动密集型任务（例如数据注释和文本分析）的自动化来迅速改变社会科学研究。但是，LLM输出差异很大，具体取决于研究人员（例如模型选择，提示策略或温度设置）的实施选择。这种变化会引入系统的偏见和随机错误，从而传播到下游分析并导致I型，II型，类型S或类型M错误。我们称此LLM黑客攻击。我们通过复制21种具有18种不同模型的社会科学研究研究中的37个数据注释任务来量化LLM黑客攻击的风险。分析1300万个LLM标签，我们测试了2,361个现实的假设，以衡量合理的研究人员选择如何影响统计结论。我们在大约三分之一的假设中发现了基于LLM注销的数据的不正确结论，用于最先进的模型，而小语言模型的假设一半。尽管我们的发现表明，更高的任务性能和更好的通用模型功能降低了LLM黑客攻击风险，但即使是高度准确的模型也不能完全消除它。随着效应大小的增加，LLM黑客攻击的风险降低，表明需要更严格地验证接近显着性阈值的发现。我们对LLM黑客缓解技术的广泛分析强调了人类注释在减少假阳性发现和改善模型选择方面的重要性。出乎意料的是，由于它们在降低LLM黑客攻击风险方面，共同的回归估计器校正技术在很大程度上无效，因为它们大力贸易I型与II型错误。除了意外错误之外，我们发现故意的LLM黑客攻击非常简单。只有很少的LLM和少量及时的解释，任何内容都可以表现为具有统计学意义。

Title: A Survey of Reinforcement Learning for Large Reasoning Models

• Authors: Kaiyan Zhang, Yuxin Zuo, Bingxiang He, Youbang Sun, Runze Liu, Che Jiang, Yuchen Fan, Kai Tian, Guoli Jia, Pengfei Li, Yu Fu, Xingtai Lv, Yuchen Zhang, Sihang Zeng, Shang Qu, Haozhan Li, Shijie Wang, Yuru Wang, Xinwei Long, Fangfu Liu, Xiang Xu, Jiaze Ma, Xuekai Zhu, Ermo Hua, Yihao Liu, Zonglin Li, Huayu Chen, Xiaoye Qu, Yafu Li, Weize Chen, Zhenzhao Yuan, Junqi Gao, Dong Li, Zhiyuan Ma, Ganqu Cui, Zhiyuan Liu, Biqing Qi, Ning Ding, Bowen Zhou
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2509.08827
• Pdf URL: https://arxiv.org/pdf/2509.08827
• Copy Paste: [[2509.08827]] A Survey of Reinforcement Learning for Large Reasoning Models(https://arxiv.org/abs/2509.08827)
• Keywords: language model, llm
• Abstract: In this paper, we survey recent advances in Reinforcement Learning (RL) for reasoning with Large Language Models (LLMs). RL has achieved remarkable success in advancing the frontier of LLM capabilities, particularly in addressing complex logical tasks such as mathematics and coding. As a result, RL has emerged as a foundational methodology for transforming LLMs into LRMs. With the rapid progress of the field, further scaling of RL for LRMs now faces foundational challenges not only in computational resources but also in algorithm design, training data, and infrastructure. To this end, it is timely to revisit the development of this domain, reassess its trajectory, and explore strategies to enhance the scalability of RL toward Artificial SuperIntelligence (ASI). In particular, we examine research applying RL to LLMs and LRMs for reasoning abilities, especially since the release of DeepSeek-R1, including foundational components, core problems, training resources, and downstream applications, to identify future opportunities and directions for this rapidly evolving area. We hope this review will promote future research on RL for broader reasoning models. Github: this https URL
• 摘要：在本文中，我们调查了使用大语言模型（LLM）推理的加强学习（RL）的最新进展。 RL在推进LLM功能的前沿方面取得了巨大的成功，特别是在解决复杂的逻辑任务（例如数学和编码）方面。结果，RL已成为将LLMS转化为LRM的基础方法。随着该领域的快速发展，LRM的RL的进一步扩展现在不仅面临计算资源的基础挑战，而且还面临算法设计，培训数据和基础架构的基础挑战。为此，及时重新审视该领域的发展，重新评估其轨迹，并探讨提高RL对人工超级智能（ASI）的可扩展性的策略。特别是，我们研究了将RL适用于LLM和LRM的研究，尤其是自DeepSeek-R1发布以来，包括基本组件，核心问题，培训资源和下游应用程序，以确定这个快速发展的领域的未来机会和方向。我们希望这篇评论将促进对RL的未来研究，以促进更广泛的推理模型。 github：此HTTPS URL