2025-03-27

Title: Poor Alignment and Steerability of Large Language Models: Evidence from College Admission Essays

• Authors: Jinsook Lee, AJ Alvero, Thorsten Joachims, René Kizilcec
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.20062
• Pdf URL: https://arxiv.org/pdf/2503.20062
• Copy Paste: [[2503.20062]] Poor Alignment and Steerability of Large Language Models: Evidence from College Admission Essays(https://arxiv.org/abs/2503.20062)
• Keywords: language model, llm, prompt
• Abstract: People are increasingly using technologies equipped with large language models (LLM) to write texts for formal communication, which raises two important questions at the intersection of technology and society: Who do LLMs write like (model alignment); and can LLMs be prompted to change who they write like (model steerability). We investigate these questions in the high-stakes context of undergraduate admissions at a selective university by comparing lexical and sentence variation between essays written by 30,000 applicants to two types of LLM-generated essays: one prompted with only the essay question used by the human applicants; and another with additional demographic information about each applicant. We consistently find that both types of LLM-generated essays are linguistically distinct from human-authored essays, regardless of the specific model and analytical approach. Further, prompting a specific sociodemographic identity is remarkably ineffective in aligning the model with the linguistic patterns observed in human writing from this identity group. This holds along the key dimensions of sex, race, first-generation status, and geographic location. The demographically prompted and unprompted synthetic texts were also more similar to each other than to the human text, meaning that prompting did not alleviate homogenization. These issues of model alignment and steerability in current LLMs raise concerns about the use of LLMs in high-stakes contexts.
• 摘要：人们越来越多地使用配备大语言模型（LLM）的技术来编写正式交流的文本，这在技术与社会的交集中提出了两个重要的问题：谁在LLMS上写下（模型对齐）；并且可以提示LLMS更改他们写的人（模型可管）。我们通过比较30,000名申请人写的论文与两种类型的LLM生成的论文之间的词汇和句子变化来调查在选择性大学的本科录取的高风险背景下进行调查的这些问题：仅在人类申请人使用的论文问题中提示。另一个包含有关每个申请人的其他人口统计信息。我们一致地发现，无论特定的模型和分析方法如何，两种类型的LLM生成的论文在语言上都与人为著名的论文不同。此外，促使特定的社会人口统计学身份在将模型与该身份群体中人类写作中观察到的语言模式保持一致时非常无效。这符合性，种族，第一代状态和地理位置的关键维度。在人口统计学的促进和未提及的合成文本中，彼此的综合文本也比人类文本更相似，这意味着提示并不能减轻均质化。当前LLMS中模型对齐和可施用性的这些问题引起了人们对在高风险环境中使用LLM的使用的担忧。

Title: Cross-Tokenizer Distillation via Approximate Likelihood Matching

• Authors: Benjamin Minixhofer, Edoardo Maria Ponti, Ivan Vulić
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.20083
• Pdf URL: https://arxiv.org/pdf/2503.20083
• Copy Paste: [[2503.20083]] Cross-Tokenizer Distillation via Approximate Likelihood Matching(https://arxiv.org/abs/2503.20083)
• Keywords: language model, llm
• Abstract: Distillation has shown remarkable success in transferring knowledge from a Large Language Model (LLM) teacher to a student LLM. However, current distillation methods predominantly require the same tokenizer between the teacher and the student, restricting their applicability to only a small subset of teacher-student pairs. In this work, we develop a cross-tokenizer distillation method to solve this crucial deficiency. Our method is the first to enable cross-tokenizer distillation without a next-token prediction loss as the main objective, instead purely maximizing the student predictions' similarity to the teacher's predictions (known as pure distillation), while also being robust to large mismatches between the teacher and the student tokenizer function and vocabulary. Empirically, our method enables substantially improved performance as tested on two use cases. First, we show that viewing tokenizer transfer as self-distillation enables unprecedently effective transfer across tokenizers. We transfer (subword-level) Llama and Gemma models to byte-level tokenization more effectively than prior methods transfer to a similar subword tokenizer under a comparable training budget. Transferring different base models to the same tokenizer also enables ensembling them (e.g., via averaging their predicted probabilities) which boosts performance. Second, we use our cross-tokenizer distillation method to distil a large maths-specialized LLM into a smaller model, achieving competitive maths problem-solving performance. Overall, our results make substantial strides toward better adaptability and enhanced interaction between different LLMs.
• 摘要：蒸馏已在将知识从大语言模型（LLM）教师转移到学生LLM方面取得了显着成功。但是，当前的蒸馏方法主要需要教师和学生之间的同样令牌，这将其适用性限制在一小部分教师对对中。在这项工作中，我们开发了一种交叉蒸馏方法来解决这种关键的缺陷。我们的方法是第一个可以在没有下一步的预测损失的情况下实现交叉蒸馏的主要目标，而是纯粹是最大化学生预测与教师的预测（称为纯蒸馏）的相似性，同时对老师和学生的象征功能和词汇的大不了不多。从经验上讲，我们的方法可实现两种用例测试的实质性提高的性能。首先，我们表明，将令牌剂转移视为自distiltation，可以使跨令牌跨越有效地转移。在可比的培训预算下，我们将（子词级）骆驼和Gemma模型转移到字节级令牌化中，比先前的方法更有效地转移到类似的子词令牌。将不同的基本模型转移到同一令牌机构还可以使它们结合起来（例如，通过平均其预测概率）来提高性能。其次，我们使用跨局体蒸馏方法将大型数学专题的LLM它所扩展为较小的模型，从而实现竞争性数学问题解决问题。总体而言，我们的结果朝着更好的适应性和增强不同LLM之间的相互作用而实现的大步进展。

Title: Generative Linguistics, Large Language Models, and the Social Nature of Scientific Success

• Authors: Sophie Hao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.20088
• Pdf URL: https://arxiv.org/pdf/2503.20088
• Copy Paste: [[2503.20088]] Generative Linguistics, Large Language Models, and the Social Nature of Scientific Success(https://arxiv.org/abs/2503.20088)
• Keywords: language model
• Abstract: Chesi's (forthcoming) target paper depicts a generative linguistics in crisis, foreboded by Piantadosi's (2023) declaration that "modern language models refute Chomsky's approach to language." In order to survive, Chesi warns, generativists must hold themselves to higher standards of formal and empirical rigor. This response argues that the crisis described by Chesi and Piantadosi actually has little to do with rigor, but is rather a reflection of generativists' limited social ambitions. Chesi ties the fate of generative linguistics to its intellectual merits, but the current success of language model research is social in nature as much as it is intellectual. In order to thrive, then, generativists must do more than heed Chesi's call for rigor; they must also expand their ambitions by giving outsiders a stake in their future success.
• 摘要：Chesi（即将出版的）目标论文描绘了Piantadosi（2023）宣布“现代语言模型驳斥乔姆斯基的语言方法”的危机中的生成语言学。为了生存，Chesi警告说，Generatixist必须遵守更高的正式和经验严格标准。这种反应表明，切西和Piantadosi所描述的危机实际上与严峻无关，而是反映出有限的社会野心的反映。 Chesi将生成语言学的命运与其智力优点联系在一起，但是语言模型研究的当前成功本质上是社会性质的。因此，为了蓬勃发展，富长必须做的比Chesi呼吁严峻的人要做的还要多。他们还必须通过给外界的未来成功来扩大野心。

Title: Bigger But Not Better: Small Neural Language Models Outperform Large Language Models in Detection of Thought Disorder

• Authors: Changye Li, Weizhe Xu, Serguei Pakhomov, Ellen Bradley, Dror Ben-Zeev, Trevor Cohen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.20103
• Pdf URL: https://arxiv.org/pdf/2503.20103
• Copy Paste: [[2503.20103]] Bigger But Not Better: Small Neural Language Models Outperform Large Language Models in Detection of Thought Disorder(https://arxiv.org/abs/2503.20103)
• Keywords: language model, llm
• Abstract: Disorganized thinking is a key diagnostic indicator of schizophrenia-spectrum disorders. Recently, clinical estimates of the severity of disorganized thinking have been shown to correlate with measures of how difficult speech transcripts would be for large language models (LLMs) to predict. However, LLMs' deployment challenges -- including privacy concerns, computational and financial costs, and lack of transparency of training data -- limit their clinical utility. We investigate whether smaller neural language models can serve as effective alternatives for detecting positive formal thought disorder, using the same sliding window based perplexity measurements that proved effective with larger models. Surprisingly, our results show that smaller models are more sensitive to linguistic differences associated with formal thought disorder than their larger counterparts. Detection capability declines beyond a certain model size and context length, challenging the common assumption of ``bigger is better'' for LLM-based applications. Our findings generalize across audio diaries and clinical interview speech samples from individuals with psychotic symptoms, suggesting a promising direction for developing efficient, cost-effective, and privacy-preserving screening tools that can be deployed in both clinical and naturalistic settings.
• 摘要：混乱的思维是精神分裂症 - 谱系障碍的关键诊断指标。最近，已经证明了对杂乱无章的思维严重程度的临床估计，与衡量语音转录本对大语模型（LLMS）进行预测的难度相关。但是，LLMS的部署挑战 - 包括隐私问题，计算和财务成本以及缺乏培训数据的透明度 - 限制了其临床公用事业。我们研究了使用基于滑动窗口的同一的困惑测量值证明有效的较大模型，我们研究了较小的神经语言模型是否可以作为检测正式思想障碍的有效替代方案。令人惊讶的是，我们的结果表明，较小的模型对与形式思想障碍相关的语言差异比较大的同行更敏感。检测能力的下降超出了一定的模型大小和上下文长度，挑战了基于LLM的应用程序``更大''的共同假设。我们的发现跨越了有精神病症状的人的音频日记和临床访谈语音样本，这表明可以在临床和自然主义环境中部署有效，具有成本效益和隐私的筛查工具的有希望的方向。

Title: Efficient Model Development through Fine-tuning Transfer

• Authors: Pin-Jie Lin, Rishab Balasubramanian, Fengyuan Liu, Nikhil Kandpal, Tu Vu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.20110
• Pdf URL: https://arxiv.org/pdf/2503.20110
• Copy Paste: [[2503.20110]] Efficient Model Development through Fine-tuning Transfer(https://arxiv.org/abs/2503.20110)
• Keywords: llm
• Abstract: Modern LLMs struggle with efficient updates, as each new pretrained model version requires repeating expensive alignment processes. This challenge also applies to domain- or language-specific models, where fine-tuning on specialized data must be redone for every new base model release. In this paper, we explore the transfer of fine-tuning updates between model versions. Specifically, we derive the diff vector from one source model version, which represents the weight changes from fine-tuning, and apply it to the base model of a different target version. Through empirical evaluations on various open-weight model versions, we show that transferring diff vectors can significantly improve the target base model, often achieving performance comparable to its fine-tuned counterpart. For example, reusing the fine-tuning updates from Llama 3.0 8B leads to an absolute accuracy improvement of 10.7% on GPQA over the base Llama 3.1 8B without additional training, surpassing Llama 3.1 8B Instruct. In a multilingual model development setting, we show that this approach can significantly increase performance on target-language tasks without retraining, achieving an absolute improvement of 4.7% and 15.5% on Global MMLU for Malagasy and Turkish, respectively, compared to Llama 3.1 8B Instruct. Our controlled experiments reveal that fine-tuning transfer is most effective when the source and target models are linearly connected in the parameter space. Additionally, we demonstrate that fine-tuning transfer offers a stronger and more computationally efficient starting point for further fine-tuning. Finally, we propose an iterative recycling-then-finetuning approach for continuous model development, which improves both efficiency and effectiveness. Our findings suggest that fine-tuning transfer is a viable strategy to reduce training costs while maintaining model performance.
• 摘要：现代LLM在高效更新方面努力，因为每个新预验证的模型版本都需要重复昂贵的对齐过程。这项挑战也适用于域或特定于语言的模型，在这些模型中，对于每个新的基本模型发布，必须重新调整专用数据的微调。在本文中，我们探讨了模型版本之间的微调更新的传输。具体而言，我们从一个源模型版本中得出了DIFF向量，该版本代表了与微调的重量变化，并将其应用于不同目标版本的基本模型。通过对各种开放重量模型版本的经验评估，我们表明传递差异向量可以显着改善目标基本模型，通常可以实现与其微调对应物相当的性能。例如，重复使用Llama 3.0 8b的微调更新导致GPQA的绝对准确性提高了10.7％，而基本的Llama 3.1 8B没有额外的培训，超过了Llama 3.1 8B指令。在多语言模型开发环境中，我们表明，与Llama 3.1 8b指导相比，这种方法可以显着提高目标和不重新培训的目标，而无需重新培训，分别在MALAGASY和TURKISH的全球MMLU中实现了4.7％和15.5％的绩效。我们的对照实验表明，当源模型和目标模型在参数空间中线性连接时，微调传输最有效。此外，我们证明了微调转移提供了更强大，更有效的开发点，以进行进一步的微调。最后，我们提出了一种迭代回收，然后提出了连续模型开发方法，从而提高了效率和有效性。我们的发现表明，微调转移是一项可行的策略，可以在保持模型绩效的同时降低培训成本。

Title: Leveraging Implicit Sentiments: Enhancing Reliability and Validity in Psychological Trait Evaluation of LLMs

• Authors: Huanhuan Ma, Haisong Gong, Xiaoyuan Yi, Xing Xie, Dongkuan Xu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.20182
• Pdf URL: https://arxiv.org/pdf/2503.20182
• Copy Paste: [[2503.20182]] Leveraging Implicit Sentiments: Enhancing Reliability and Validity in Psychological Trait Evaluation of LLMs(https://arxiv.org/abs/2503.20182)
• Keywords: language model, llm
• Abstract: Recent advancements in Large Language Models (LLMs) have led to their increasing integration into human life. With the transition from mere tools to human-like assistants, understanding their psychological aspects-such as emotional tendencies and personalities-becomes essential for ensuring their trustworthiness. However, current psychological evaluations of LLMs, often based on human psychological assessments like the BFI, face significant limitations. The results from these approaches often lack reliability and have limited validity when predicting LLM behavior in real-world scenarios. In this work, we introduce a novel evaluation instrument specifically designed for LLMs, called Core Sentiment Inventory (CSI). CSI is a bilingual tool, covering both English and Chinese, that implicitly evaluates models' sentiment tendencies, providing an insightful psychological portrait of LLM across three dimensions: optimism, pessimism, and neutrality. Through extensive experiments, we demonstrate that: 1) CSI effectively captures nuanced emotional patterns, revealing significant variation in LLMs across languages and contexts; 2) Compared to current approaches, CSI significantly improves reliability, yielding more consistent results; and 3) The correlation between CSI scores and the sentiment of LLM's real-world outputs exceeds 0.85, demonstrating its strong validity in predicting LLM behavior. We make CSI public available via: this https URL.
• 摘要：大型语言模型（LLM）的最新进展导致它们不断扩大到人类生活中。通过从单纯的工具到类似人类的助手的过渡，了解他们的心理方面，例如情感倾向和个性，对确保其可信度至关重要。但是，通常基于人类心理评估（例如BFI）对LLM的心理评估面临重大局限性。这些方法的结果通常缺乏可靠性，在预测现实世界中LLM行为时的有效性有限。在这项工作中，我们介绍了一种专门为LLM设计的新型评估工具，称为核心情感清单（CSI）。 CSI是一种双语工具，涵盖了英语和中文，它隐含地评估了模型的情感趋势，在三个维度上提供了LLM的深刻心理肖像：乐观，悲观和中立。通过广泛的实验，我们证明：1）CSI有效地捕获了细微的情感模式，从而揭示了跨语言和环境的LLM的显着差异； 2）与当前的方法相比，CSI显着提高了可靠性，从而产生了更一致的结果； 3）CSI得分与LLM现实世界产出的情感之间的相关性超过0.85，这表明其在预测LLM行为方面的有效性很强。我们通过以下方式将CSI公开提供：此HTTPS URL。

Title: GAPO: Learning Preferential Prompt through Generative Adversarial Policy Optimization

• Authors: Zhouhong Gu, Xingzhou Chen, Xiaoran Shi, Tao Wang, Suhang Zheng, Tianyu Li, Hongwei Feng, Yanghua Xiao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.20194
• Pdf URL: https://arxiv.org/pdf/2503.20194
• Copy Paste: [[2503.20194]] GAPO: Learning Preferential Prompt through Generative Adversarial Policy Optimization(https://arxiv.org/abs/2503.20194)
• Keywords: language model, llm, hallucination, prompt
• Abstract: Recent advances in large language models have highlighted the critical need for precise control over model outputs through predefined constraints. While existing methods attempt to achieve this through either direct instruction-response synthesis or preferential response optimization, they often struggle with constraint understanding and adaptation. This limitation becomes particularly evident when handling fine-grained constraints, leading to either hallucination or brittle performance. We introduce Generative Adversarial Policy Optimization (GAPO), a novel framework that combines GAN-based training dynamics with an encoder-only reward model to progressively learn and adapt to increasingly complex constraints. GAPO leverages adversarial training to automatically generate training samples of varying difficulty while utilizing the encoder-only architecture to better capture prompt-response relationships. Extensive experiments demonstrate GAPO's superior performance across multiple benchmarks, particularly in scenarios requiring fine-grained constraint handling, where it significantly outperforms existing methods like PPO, DPO, and KTO. Our results suggest that GAPO's unique approach to preferential prompt learning offers a more robust and effective solution for controlling LLM outputs. Code is avaliable in this https URL.
• 摘要：大型语言模型的最新进展强调了通过预定义的约束对模型输出进行精确控制的迫切需要。尽管现有的方法试图通过直接指令 - 响应综合或优先响应优化来实现这一目标，但它们通常在约束理解和适应方面挣扎。当处理细粒度的约束时，这种限制变得尤为明显，从而导致幻觉或脆弱性能。我们介绍了生成的对抗性政策优化（GAPO），这是一个新颖的框架，将基于GAN的训练动力学与仅编码器奖励模型相结合，以逐步学习并适应日益复杂的约束。 GAPO利用对抗性培训来自动生成各种难度的训练样本，同时利用仅编码器架构来更好地捕获及时响应关系。广泛的实验表明，GAPO在多个基准测试中的出色性能，尤其是在需要细粒约束处理的情况下，在这种情况下，它极大地胜过了PPO，DPO和KTO等现有方法。我们的结果表明，Gapo的优先提示学习的独特方法为控制LLM输出提供了更强大，更有效的解决方案。在此HTTPS URL中，代码是可用的。

Title: SARGes: Semantically Aligned Reliable Gesture Generation via Intent Chain

• Authors: Nan Gao, Yihua Bao, Dongdong Weng, Jiayi Zhao, Jia Li, Yan Zhou, Pengfei Wan, Di Zhang
• Subjects: cs.CL, cs.AI, cs.HC, cs.RO
• Abstract URL: https://arxiv.org/abs/2503.20202
• Pdf URL: https://arxiv.org/pdf/2503.20202
• Copy Paste: [[2503.20202]] SARGes: Semantically Aligned Reliable Gesture Generation via Intent Chain(https://arxiv.org/abs/2503.20202)
• Keywords: language model, llm
• Abstract: Co-speech gesture generation enhances human-computer interaction realism through speech-synchronized gesture synthesis. However, generating semantically meaningful gestures remains a challenging problem. We propose SARGes, a novel framework that leverages large language models (LLMs) to parse speech content and generate reliable semantic gesture labels, which subsequently guide the synthesis of meaningful co-speech this http URL, we constructed a comprehensive co-speech gesture ethogram and developed an LLM-based intent chain reasoning mechanism that systematically parses and decomposes gesture semantics into structured inference steps following ethogram criteria, effectively guiding LLMs to generate context-aware gesture labels. Subsequently, we constructed an intent chain-annotated text-to-gesture label dataset and trained a lightweight gesture label generation model, which then guides the generation of credible and semantically coherent co-speech gestures. Experimental results demonstrate that SARGes achieves highly semantically-aligned gesture labeling (50.2% accuracy) with efficient single-pass inference (0.4 seconds). The proposed method provides an interpretable intent reasoning pathway for semantic gesture synthesis.
• 摘要：共同语音的手势产生通过语音同步的手势合成来增强人类计算机的相互作用现实主义。但是，产生语义意义的手势仍然是一个具有挑战性的问题。 We propose SARGes, a novel framework that leverages large language models (LLMs) to parse speech content and generate reliable semantic gesture labels, which subsequently guide the synthesis of meaningful co-speech this http URL, we constructed a comprehensive co-speech gesture ethogram and developed an LLM-based intent chain reasoning mechanism that systematically parses and decomposes gesture semantics into structured inference steps following ethogram标准，有效指导LLMS生成上下文感知的手势标签。随后，我们构建了一个意图链的文本到手机标签数据集，并训练了轻巧的手势标签生成模型，然后指导了可信和语义上一致的共同语音语音手势的产生。实验结果表明，Sarges具有有效的单通行推理（0.4秒），可实现高度语义对齐的手势标记（精度为50.2％）。提出的方法为语义手势合成提供了可解释的意图推理途径。

Title: Qwen2.5-Omni Technical Report

• Authors: Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, Junyang Lin
• Subjects: cs.CL, cs.CV, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2503.20215
• Pdf URL: https://arxiv.org/pdf/2503.20215
• Copy Paste: [[2503.20215]] Qwen2.5-Omni Technical Report(https://arxiv.org/abs/2503.20215)
• Keywords: language model
• Abstract: In this report, we present Qwen2.5-Omni, an end-to-end multimodal model designed to perceive diverse modalities, including text, images, audio, and video, while simultaneously generating text and natural speech responses in a streaming manner. To enable the streaming of multimodal information inputs, both audio and visual encoders utilize a block-wise processing approach. To synchronize the timestamps of video inputs with audio, we organize the audio and video sequentially in an interleaved manner and propose a novel position embedding approach, named TMRoPE(Time-aligned Multimodal RoPE). To concurrently generate text and speech while avoiding interference between the two modalities, we propose \textbf{Thinker-Talker} architecture. In this framework, Thinker functions as a large language model tasked with text generation, while Talker is a dual-track autoregressive model that directly utilizes the hidden representations from the Thinker to produce audio tokens as output. Both the Thinker and Talker models are designed to be trained and inferred in an end-to-end manner. For decoding audio tokens in a streaming manner, we introduce a sliding-window DiT that restricts the receptive field, aiming to reduce the initial package delay. Qwen2.5-Omni is comparable with the similarly sized Qwen2.5-VL and outperforms Qwen2-Audio. Furthermore, Qwen2.5-Omni achieves state-of-the-art performance on multimodal benchmarks like Omni-Bench. Notably, Qwen2.5-Omni's performance in end-to-end speech instruction following is comparable to its capabilities with text inputs, as evidenced by benchmarks such as MMLU and GSM8K. As for speech generation, Qwen2.5-Omni's streaming Talker outperforms most existing streaming and non-streaming alternatives in robustness and naturalness.
• 摘要：在本报告中，我们提出了QWEN2.5-OMNI，这是一种端到端的多模型模型，旨在感知各种方式，包括文本，图像，音频和视频，同时以流方式生成文本和自然语音响应。为了启用多模式信息输入的流，音频和视觉编码器都采用了块处理方法。为了将视频输入的时间戳与音频同步，我们以交织方式依次组织音频和视频，并提出了一种新颖的位置嵌入方法，称为TMrope（时间平行的多模式绳）。为了同时生成文本和语音，同时避免了两种模式之间的干扰，我们建议\ textbf {thinker-talker}架构。在此框架中，思想家充当一个由文本生成的大型语言模型，而Talker是一种双轨自动回归模型，直接利用思想家的隐藏表示形式来产生音频令牌作为输出。思想家和谈话者模型都被设计为以端到端的方式进行培训和推断。为了以流方式解码音频令牌，我们引入了一个滑动窗口dit，该窗口限制了接受场，旨在减少初始包装延迟。 QWEN2.5-OMNI与类似尺寸的QWEN2.5-VL相当，并且胜过Qwen2-Audio。此外，Qwen2.5-omni在诸如Omni Bench之类的多模式基准上实现了最先进的性能。值得注意的是，以下Qwen2.5-omni在端到端语音说明中的性能与文本输入的能力相当，如MMLU和GSM8K等基准所证明。至于语音产生，QWEN2.5-OMNI的流式说话者在鲁棒和自然性方面优于大多数现有的流媒体和非流式替代方案。

Title: Advancements in Natural Language Processing: Exploring Transformer-Based Architectures for Text Understanding

• Authors: Tianhao Wu, Yu Wang, Ngoc Quach
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.20227
• Pdf URL: https://arxiv.org/pdf/2503.20227
• Copy Paste: [[2503.20227]] Advancements in Natural Language Processing: Exploring Transformer-Based Architectures for Text Understanding(https://arxiv.org/abs/2503.20227)
• Keywords: gpt, chat
• Abstract: Natural Language Processing (NLP) has witnessed a transformative leap with the advent of transformer-based architectures, which have significantly enhanced the ability of machines to understand and generate human-like text. This paper explores the advancements in transformer models, such as BERT and GPT, focusing on their superior performance in text understanding tasks compared to traditional methods like recurrent neural networks (RNNs). By analyzing statistical properties through visual representations-including probability density functions of text length distributions and feature space classifications-the study highlights the models' proficiency in handling long-range dependencies, adapting to conditional shifts, and extracting features for classification, even with overlapping classes. Drawing on recent 2024 research, including enhancements in multi-hop knowledge graph reasoning and context-aware chat interactions, the paper outlines a methodology involving data preparation, model selection, pretraining, fine-tuning, and evaluation. The results demonstrate state-of-the-art performance on benchmarks like GLUE and SQuAD, with F1 scores exceeding 90%, though challenges such as high computational costs persist. This work underscores the pivotal role of transformers in modern NLP and suggests future directions, including efficiency optimization and multimodal integration, to further advance language-based AI systems.
• 摘要：自然语言处理（NLP）随着基于变压器的架构的出现，见证了变革性的飞跃，这显着增强了机器理解和生成类似人类文本的能力。本文探讨了诸如BERT和GPT等变压器模型的进步，重点是与复发性神经网络（RNN）（RNN）相比，其在文本理解任务中的出色表现。通过通过视觉表示分析统计属性 - 包括文本长度分布的概率密度函数和特征空间分类 - 研究突出了模型在处理长期依赖性，适应条件偏移方面的熟练程度，即使在重叠的类别中，模型都可以适应有条件的偏移以及提取分类的特征。该论文借鉴了最近2024年的研究，包括增强多跳跃知识图形推理和上下文感知的聊天互动，概述了一种涉及数据准备，模型选择，预处理，微调和评估的方法论。结果表明，诸如胶水和小队之类的基准上的最先进的表现，F1得分超过90％，尽管诸如高计算成本之类的挑战仍然存在。这项工作强调了变压器在现代NLP中的关键作用，并提出了未来的方向，包括效率优化和多模式集成，以进一步推进基于语言的AI系统。

Title: sudo rm -rf agentic_security

• Authors: Sejin Lee, Jian Kim, Haon Park, Ashkan Yousefpour, Sangyoon Yu, Min Song
• Subjects: cs.CL, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2503.20279
• Pdf URL: https://arxiv.org/pdf/2503.20279
• Copy Paste: [[2503.20279]] sudo rm -rf agentic_security(https://arxiv.org/abs/2503.20279)
• Keywords: language model, llm, agent
• Abstract: Large Language Models (LLMs) are increasingly deployed as computer-use agents, autonomously performing tasks within real desktop or web environments. While this evolution greatly expands practical use cases for humans, it also creates serious security exposures. We present SUDO (Screen-based Universal Detox2Tox Offense), a novel attack framework that systematically bypasses refusal trained safeguards in commercial computer-use agents, such as Claude Computer Use. The core mechanism, Detox2Tox, transforms harmful requests (that agents initially reject) into seemingly benign requests via detoxification, secures detailed instructions from advanced vision language models (VLMs), and then reintroduces malicious content via toxification just before execution. Unlike conventional jailbreaks, SUDO iteratively refines its attacks based on a built-in refusal feedback, making it increasingly effective against robust policy filters. In extensive tests spanning 50 real-world tasks and multiple state-of-the-art VLMs, SUDO achieves a stark attack success rate of 24% (with no refinement), and up to 41% (by its iterative refinement) in Claude Computer Use. By revealing these vulnerabilities and demonstrating the ease with which they can be exploited in real-world computing environments, this paper highlights an immediate need for robust, context-aware safeguards. WARNING: This paper includes harmful or offensive model outputs.
• 摘要：大型语言模型（LLMS）越来越多地部署为计算机使用代理，在实际桌面或Web环境中自主执行任务。尽管这种进化大大扩展了人类的实际用例，但它也会产生严重的安全暴露。我们提出了Sudo（基于屏幕的Universal Detox2Tox进攻），这是一个新型的攻击框架，系统地绕开了商业计算机使用代理中的拒绝训练的保障措施，例如Claude Computer使用。核心机制Detox2Tox将有害的请求（最初拒绝）通过排毒来看似良性的请求，确保来自高级视觉语言模型（VLMS）的详细说明，然后在执行前通过毒化来重新引入恶意内容。与传统的越狱不同，苏多（Sudo）迭代根据内置的拒绝反馈来完善其攻击，从而使其越来越有效地抵抗强大的政策过滤器。在涵盖50个现实世界任务和多个最先进的VLM的广泛测试中，Sudo的攻击成功率为24％（没有改进），在Claude Computer使用中最多可达到41％（通过其迭代性完善）。通过揭示这些漏洞，并证明在现实世界计算环境中可以利用它们的便利性，本文强调了对强大的，上下文感知的保障措施的即时需求。警告：本文包括有害或冒犯模型输出。

Title: A Multilingual, Culture-First Approach to Addressing Misgendering in LLM Applications

• Authors: Sunayana Sitaram, Adrian de Wynter, Isobel McCrum, Qilong Gu, Si-Qing Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.20302
• Pdf URL: https://arxiv.org/pdf/2503.20302
• Copy Paste: [[2503.20302]] A Multilingual, Culture-First Approach to Addressing Misgendering in LLM Applications(https://arxiv.org/abs/2503.20302)
• Keywords: language model, llm
• Abstract: Misgendering is the act of referring to someone by a gender that does not match their chosen identity. It marginalizes and undermines a person's sense of self, causing significant harm. English-based approaches have clear-cut approaches to avoiding misgendering, such as the use of the pronoun ``they''. However, other languages pose unique challenges due to both grammatical and cultural constructs. In this work we develop methodologies to assess and mitigate misgendering across 42 languages and dialects using a participatory-design approach to design effective and appropriate guardrails across all languages. We test these guardrails in a standard large language model-based application (meeting transcript summarization), where both the data generation and the annotation steps followed a human-in-the-loop approach. We find that the proposed guardrails are very effective in reducing misgendering rates across all languages in the summaries generated, and without incurring loss of quality. Our human-in-the-loop approach demonstrates a method to feasibly scale inclusive and responsible AI-based solutions across multiple languages and cultures.
• 摘要：错误的行为是通过不符合其选择身份的性别来指代某人的行为。它边缘化并破坏了一个人的自我意识，造成了重大伤害。基于英语的方法具有清晰的方法来避免误解，例如使用代词``他们''。但是，由于语法和文化结构，其他语言构成了独特的挑战。在这项工作中，我们开发了使用参与式设计方法来评估和减轻42种语言和方言的错误性，以设计所有语言的有效和适当的护栏。我们在基于标准的大语言模型应用程序（满足成绩单摘要）中测试这些护栏，其中数据生成和注释步骤均遵循了人类的方法。我们发现，拟议的护栏非常有效地降低所产生的摘要中所有语言的错误性率，而不会造成质量损失。我们的人类在线方法展示了一种跨多种语言和文化的可行扩展包含和负责任的解决方案的方法。

Title: Iterative Prompting with Persuasion Skills in Jailbreaking Large Language Models

• Authors: Shih-Wen Ke, Guan-Yu Lai, Guo-Lin Fang, Hsi-Yuan Kao
• Subjects: cs.CL, cs.AI, cs.ET
• Abstract URL: https://arxiv.org/abs/2503.20320
• Pdf URL: https://arxiv.org/pdf/2503.20320
• Copy Paste: [[2503.20320]] Iterative Prompting with Persuasion Skills in Jailbreaking Large Language Models(https://arxiv.org/abs/2503.20320)
• Keywords: language model, gpt, llm, prompt, chat
• Abstract: Large language models (LLMs) are designed to align with human values in their responses. This study exploits LLMs with an iterative prompting technique where each prompt is systematically modified and refined across multiple iterations to enhance its effectiveness in jailbreaking attacks progressively. This technique involves analyzing the response patterns of LLMs, including GPT-3.5, GPT-4, LLaMa2, Vicuna, and ChatGLM, allowing us to adjust and optimize prompts to evade the LLMs' ethical and security constraints. Persuasion strategies enhance prompt effectiveness while maintaining consistency with malicious intent. Our results show that the attack success rates (ASR) increase as the attacking prompts become more refined with the highest ASR of 90% for GPT4 and ChatGLM and the lowest ASR of 68% for LLaMa2. Our technique outperforms baseline techniques (PAIR and PAP) in ASR and shows comparable performance with GCG and ArtPrompt.
• 摘要：大型语言模型（LLM）旨在与人类价值观的回应保持一致。这项研究通过迭代提示技术利用了LLMS，每个提示都会在多个迭代中进行系统地修改和完善，以逐渐增强其在越狱攻击中的有效性。该技术涉及分析LLM的响应模式，包括GPT-3.5，GPT-4，Llama2，Vicuna和ChatGLM，使我们能够调整和优化提示以逃避LLMS的道德和安全性约束。说服策略提高了迅速的有效性，同时保持与恶意意图保持一致性。我们的结果表明，攻击成功率（ASR）随着攻击提示的最高为90％，GPT4和CHATGLM的最高ASR，而Llama2的最低ASR最低。我们的技术在ASR中优于基线技术（配对和PAP），并且与GCG和Artprompt显示出可比的性能。

Title: CFunModel: A "Funny" Language Model Capable of Chinese Humor Generation and Processing

• Authors: Zhenghan Yu, Xinyu Hu, Xiaojun Wan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.20417
• Pdf URL: https://arxiv.org/pdf/2503.20417
• Copy Paste: [[2503.20417]] CFunModel: A "Funny" Language Model Capable of Chinese Humor Generation and Processing(https://arxiv.org/abs/2503.20417)
• Keywords: language model, llm
• Abstract: Humor plays a significant role in daily language communication. With the rapid development of large language models (LLMs), natural language processing has made significant strides in understanding and generating various genres of texts. However, most LLMs exhibit poor performance in generating and processing Chinese humor. In this study, we introduce a comprehensive Chinese humor-related dataset, the Chinese Fun Set (CFunSet). This dataset aggregates existing Chinese humor datasets and includes over 20,000 jokes collected from Tieba-JokeBar, a Chinese online platform known for joke sharing. The resulting corpus comprises more than 160,000 entries. Leveraging CFunSet, we developed the Chinese Fun Model (CFunModel), the first large language model designed to handle various Chinese humor-related tasks including Crosstalk Response Selection, Humor Recognition, Joke Generation, etc. Experimental results demonstrate that CFunModel outperforms popular large language models in these tasks. Our CFunSet is available at this https URL and CFunModel is available at this https URL. A demostration video of our work is available at this https URL.
• 摘要：幽默在日常语言交流中起着重要作用。随着大型语言模型（LLM）的快速发展，自然语言处理在理解和生成各种流派的文本方面取得了重大进步。但是，大多数LLM在产生和处理中国幽默方面表现出色。在这项研究中，我们介绍了一个全面的中国幽默相关数据集，即中国娱乐集（CFUNSET）。该数据集汇总了现有的中国幽默数据集，其中包括从蒂巴·乔克巴（Tieba-Jokebar）收集的20,000多个笑话，这是一个以笑话共享而闻名的中国在线平台。由此产生的语料库包含160,000多个条目。利用CFUNSET，我们开发了中国有趣的模型（CFUNMODEL），这是第一个旨在处理各种与中国幽默相关的任务，包括串扰响应选择，幽默识别，笑话的产生等。实验结果表明，CFUNMODEL在这些任务中胜过流行的大语言模型。我们的CFUNSET可在此HTTPS URL上找到，并且CFUNMODEL可在此HTTPS URL上找到。此HTTPS URL可用来提供我们工作的解释视频。

Title: TempTest: Local Normalization Distortion and the Detection of Machine-generated Text

• Authors: Tom Kempton, Stuart Burrell, Connor Cheverall
• Subjects: cs.CL, cs.LG, math.DS
• Abstract URL: https://arxiv.org/abs/2503.20421
• Pdf URL: https://arxiv.org/pdf/2503.20421
• Copy Paste: [[2503.20421]] TempTest: Local Normalization Distortion and the Detection of Machine-generated Text(https://arxiv.org/abs/2503.20421)
• Keywords: language model
• Abstract: Existing methods for the zero-shot detection of machine-generated text are dominated by three statistical quantities: log-likelihood, log-rank, and entropy. As language models mimic the distribution of human text ever closer, this will limit our ability to build effective detection algorithms. To combat this, we introduce a method for detecting machine-generated text that is entirely agnostic of the generating language model. This is achieved by targeting a defect in the way that decoding strategies, such as temperature or top-k sampling, normalize conditional probability measures. This method can be rigorously theoretically justified, is easily explainable, and is conceptually distinct from existing methods for detecting machine-generated text. We evaluate our detector in the white and black box settings across various language models, datasets, and passage lengths. We also study the effect of paraphrasing attacks on our detector and the extent to which it is biased against non-native speakers. In each of these settings, the performance of our test is at least comparable to that of other state-of-the-art text detectors, and in some cases, we strongly outperform these baselines.
• 摘要：机器生成文本的零射击检测的现有方法以三个统计数量为主导：对数可能性，对数秩和熵。随着语言模型模仿人类文本的分布，这将限制我们构建有效检测算法的能力。为了解决这个问题，我们引入了一种检测机器生成的文本的方法，该文本完全不可知生成语言模型。这是通过以解码策略（例如温度或TOP-K采样）为标准化条件概率度量的方式来定位缺陷来实现的。这种方法可以在理论上是有道理的，很容易解释，并且在概念上与检测机器生成的文本的现有方法不同。我们通过各种语言模型，数据集和通道长度在白框设置中评估我们的探测器。我们还研究了释义攻击对我们的检测器的影响及其对非母语说话者的偏见程度。在每种设置中，我们的测试的性能至少与其他最先进的文本检测器相当，在某些情况下，我们强烈胜过这些基线。

Title: Explainable ICD Coding via Entity Linking

• Authors: Leonor Barreiros, Isabel Coutinho, Gonçalo M. Correia, Bruno Martins
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.20508
• Pdf URL: https://arxiv.org/pdf/2503.20508
• Copy Paste: [[2503.20508]] Explainable ICD Coding via Entity Linking(https://arxiv.org/abs/2503.20508)
• Keywords: language model, llm
• Abstract: Clinical coding is a critical task in healthcare, although traditional methods for automating clinical coding may not provide sufficient explicit evidence for coders in production environments. This evidence is crucial, as medical coders have to make sure there exists at least one explicit passage in the input health record that justifies the attribution of a code. We therefore propose to reframe the task as an entity linking problem, in which each document is annotated with its set of codes and respective textual evidence, enabling better human-machine collaboration. By leveraging parameter-efficient fine-tuning of Large Language Models (LLMs), together with constrained decoding, we introduce three approaches to solve this problem that prove effective at disambiguating clinical mentions and that perform well in few-shot scenarios.
• 摘要：临床编码是医疗保健中的关键任务，尽管自动化临床编码的传统方法可能无法为生产环境中的编码人员提供足够的明确证据。这些证据至关重要，因为医疗编码人员必须确保在输入健康记录中至少有一个明确的通过，这证明了代码的归因。因此，我们建议将任务重新构架为一个链接问题的实体，其中每个文档都用其一组代码和各自的文本证据进行注释，从而实现了更好的人机协作。通过利用大型语言模型（LLMS）的参数有效的微调，加上限制解码，我们引入了三种方法来解决此问题，这些方法证明有效地消除了歧义临床提及，并且在几次射击场景中表现良好。

Title: StableToolBench-MirrorAPI: Modeling Tool Environments as Mirrors of 7,000+ Real-World APIs

• Authors: Zhicheng Guo, Sijie Cheng, Yuchen Niu, Hao Wang, Sicheng Zhou, Wenbing Huang, Yang Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.20527
• Pdf URL: https://arxiv.org/pdf/2503.20527
• Copy Paste: [[2503.20527]] StableToolBench-MirrorAPI: Modeling Tool Environments as Mirrors of 7,000+ Real-World APIs(https://arxiv.org/abs/2503.20527)
• Keywords: language model, llm, chain-of-thought
• Abstract: The rapid advancement of large language models (LLMs) has spurred significant interest in tool learning, where LLMs are augmented with external tools to tackle complex tasks. However, existing tool environments face challenges in balancing stability, scalability, and realness, particularly for benchmarking purposes. To address this problem, we propose MirrorAPI, a novel framework that trains specialized LLMs to accurately simulate real API responses, effectively acting as "mirrors" to tool environments. Using a comprehensive dataset of request-response pairs from 7,000+ APIs, we employ supervised fine-tuning and chain-of-thought reasoning to enhance simulation fidelity. MirrorAPI achieves superior accuracy and stability compared to state-of-the-art methods, as demonstrated by its performance on the newly constructed MirrorAPI-Bench and its integration into StableToolBench.
• 摘要：大型语言模型（LLM）的快速发展引发了人们对工具学习的重大兴趣，在该工具学习中，LLM可以使用外部工具来解决复杂的任务。但是，现有的工具环境面临平衡稳定性，可扩展性和现实性的挑战，尤其是用于基准测试目的。为了解决这个问题，我们提出了Mirrirapi，这是一个新颖的框架，该框架训练专业的LLM，以准确模拟真实的API响应，有效地充当工具环境的“镜像”。使用7,000多个API的请求响应对的全面数据集，我们采用了受监管的微调和经营链的推理来增强模拟保真度。与最先进的方法相比，Mirrorapi在新建的Mirrorapi板凳上的性能及其在StableToolbench中的集成所证明，与最先进的方法相比，具有卓越的准确性和稳定性。

Title: A Retrieval-Based Approach to Medical Procedure Matching in Romanian

• Authors: Andrei Niculae, Adrian Cosma, Emilian Radoi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.20556
• Pdf URL: https://arxiv.org/pdf/2503.20556
• Copy Paste: [[2503.20556]] A Retrieval-Based Approach to Medical Procedure Matching in Romanian(https://arxiv.org/abs/2503.20556)
• Keywords: language model
• Abstract: Accurately mapping medical procedure names from healthcare providers to standardized terminology used by insurance companies is a crucial yet complex task. Inconsistencies in naming conventions lead to missclasified procedures, causing administrative inefficiencies and insurance claim problems in private healthcare settings. Many companies still use human resources for manual mapping, while there is a clear opportunity for automation. This paper proposes a retrieval-based architecture leveraging sentence embeddings for medical name matching in the Romanian healthcare system. This challenge is significantly more difficult in underrepresented languages such as Romanian, where existing pretrained language models lack domain-specific adaptation to medical text. We evaluate multiple embedding models, including Romanian, multilingual, and medical-domain-specific representations, to identify the most effective solution for this task. Our findings contribute to the broader field of medical NLP for low-resource languages such as Romanian.
• 摘要：准确地将医疗程序名称从医疗保健提供商映射到保险公司使用的标准化术语，这是一项至关重要但复杂的任务。命名惯例的不一致导致了未列出的程序，导致私人医疗保健环境中的行政效率低下和保险索赔问题。许多公司仍然使用人力资源进行手动映射，而自动化的机会明显。本文提出了一个基于检索的建筑，利用罗马尼亚医疗保健系统中医疗名称匹配的句子嵌入。在诸如罗马尼亚语的代表性不足的语言中，这种挑战更加困难，那里的现有审慎的语言模型缺乏针对医学文本的特定领域的适应性。我们评估了多种嵌入模型，包括罗马尼亚，多语言和医学域特异性表示，以确定针对此任务的最有效解决方案。我们的发现为低资源语言（例如罗马尼亚语）提供了更广泛的医疗NLP领域。

Title: Low-resource Information Extraction with the European Clinical Case Corpus

• Authors: Soumitra Ghosh, Begona Altuna, Saeed Farzi, Pietro Ferrazzi, Alberto Lavelli, Giulia Mezzanotte, Manuela Speranza, Bernardo Magnini
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.20568
• Pdf URL: https://arxiv.org/pdf/2503.20568
• Copy Paste: [[2503.20568]] Low-resource Information Extraction with the European Clinical Case Corpus(https://arxiv.org/abs/2503.20568)
• Keywords: language model, llm
• Abstract: We present E3C-3.0, a multilingual dataset in the medical domain, comprising clinical cases annotated with diseases and test-result relations. The dataset includes both native texts in five languages (English, French, Italian, Spanish and Basque) and texts translated and projected from the English source into five target languages (Greek, Italian, Polish, Slovak, and Slovenian). A semi-automatic approach has been implemented, including automatic annotation projection based on Large Language Models (LLMs) and human revision. We present several experiments showing that current state-of-the-art LLMs can benefit from being fine-tuned on the E3C-3.0 dataset. We also show that transfer learning in different languages is very effective, mitigating the scarcity of data. Finally, we compare performance both on native data and on projected data. We release the data at this https URL .
• 摘要：我们提出了E3C-3.0，这是医学领域中的多语言数据集，其中包括带有疾病和试验关系的临床病例。该数据集包括五种语言（英语，法语，意大利语，西班牙语和巴斯克语）的本地文本，以及从英语来源翻译和投射为五种目标语言（希腊语，意大利语，波兰语，斯洛伐克和斯洛文尼亚语）的文本。已经实施了一种半自动方法，包括基于大语言模型（LLM）和人类修订的自动注释投影。我们提出了几项实验，表明当前的最新LLM可以从E3C-3.0数据集中进行微调而受益。我们还表明，不同语言的转移学习非常有效，从而减轻了数据的稀缺性。最后，我们比较本机数据和预计数据上的性能。我们在此HTTPS URL上发布数据。

Title: Synthetic Data Augmentation for Cross-domain Implicit Discourse Relation Recognition

• Authors: Frances Yung, Varsha Suresh, Zaynab Reza, Mansoor Ahmad, Vera Demberg
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.20588
• Pdf URL: https://arxiv.org/pdf/2503.20588
• Copy Paste: [[2503.20588]] Synthetic Data Augmentation for Cross-domain Implicit Discourse Relation Recognition(https://arxiv.org/abs/2503.20588)
• Keywords: llm
• Abstract: Implicit discourse relation recognition (IDRR) -- the task of identifying the implicit coherence relation between two text spans -- requires deep semantic understanding. Recent studies have shown that zero- or few-shot approaches significantly lag behind supervised models, but LLMs may be useful for synthetic data augmentation, where LLMs generate a second argument following a specified coherence relation. We applied this approach in a cross-domain setting, generating discourse continuations using unlabelled target-domain data to adapt a base model which was trained on source-domain labelled data. Evaluations conducted on a large-scale test set revealed that different variations of the approach did not result in any significant improvements. We conclude that LLMs often fail to generate useful samples for IDRR, and emphasize the importance of considering both statistical significance and comparability when evaluating IDRR models.
• 摘要：隐式话语关系识别（IDRR） - 识别两个文本跨度之间隐性连贯关系的任务 - 需要深入的语义理解。最近的研究表明，零或几次射击方法显着落后于监督模型，但LLMS可能对合成数据增强很有用，在指定的相干关系之后，LLMS会产生第二个参数。我们在跨域设置中应用了这种方法，使用未标记的目标域数据生成话语连续性，以适应经过源域标记的数据训练的基本模型。对大规模测试集进行的评估表明，该方法的不同变化并未导致任何显着改善。我们得出的结论是，LLM通常无法为IDRR生成有用的样本，并强调在评估IDRR模型时考虑统计显着性和可比性的重要性。

Title: Collaborative Storytelling and LLM: A Linguistic Analysis of Automatically-Generated Role-Playing Game Sessions

• Authors: Alessandro Maisto
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.20623
• Pdf URL: https://arxiv.org/pdf/2503.20623
• Copy Paste: [[2503.20623]] Collaborative Storytelling and LLM: A Linguistic Analysis of Automatically-Generated Role-Playing Game Sessions(https://arxiv.org/abs/2503.20623)
• Keywords: language model, llm
• Abstract: Role-playing games (RPG) are games in which players interact with one another to create narratives. The role of players in the RPG is largely based on the interaction between players and their characters. This emerging form of shared narrative, primarily oral, is receiving increasing attention. In particular, many authors investigated the use of an LLM as an actor in the game. In this paper, we aim to discover to what extent the language of Large Language Models (LLMs) exhibit oral or written features when asked to generate an RPG session without human interference. We will conduct a linguistic analysis of the lexical and syntactic features of the generated texts and compare the results with analyses of conversations, transcripts of human RPG sessions, and books. We found that LLMs exhibit a pattern that is distinct from all other text categories, including oral conversations, human RPG sessions and books. Our analysis has shown how training influences the way LLMs express themselves and provides important indications of the narrative capabilities of these tools.
• 摘要：角色扮演游戏（RPG）是玩家相互互动以创造叙事的游戏。玩家在RPG中的作用主要基于玩家及其角色之间的相互作用。这种共同叙述的新兴形式（主要是口头）正在受到越来越多的关注。特别是，许多作者调查了使用LLM作为游戏中的演员的使用​​。在本文中，我们旨在发现当被要求在没有人类干扰的情况下生成RPG会话时，大型语言模型（LLM）的语言在多大程度上表现出口头或书面功能。我们将对生成的文本的词汇和句法特征进行语言分析，并将结果与​​对话，人类RPG会话和书籍的成绩单进行分析。我们发现LLM表现出与所有其他文本类别不同的​​模式，包括口头对话，人类RPG会话和书籍。我们的分析表明，培训如何影响LLMS表达自己的方式，并为这些工具的叙事能力提供了重要的迹象。

Title: Unlocking Efficient Long-to-Short LLM Reasoning with Model Merging

• Authors: Han Wu, Yuxuan Yao, Shuqi Liu, Zehua Liu, Xiaojin Fu, Xiongwei Han, Xing Li, Hui-Ling Zhen, Tao Zhong, Mingxuan Yuan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.20641
• Pdf URL: https://arxiv.org/pdf/2503.20641
• Copy Paste: [[2503.20641]] Unlocking Efficient Long-to-Short LLM Reasoning with Model Merging(https://arxiv.org/abs/2503.20641)
• Keywords: language model, llm, prompt
• Abstract: The transition from System 1 to System 2 reasoning in large language models (LLMs) has marked significant advancements in handling complex tasks through deliberate, iterative thinking. However, this progress often comes at the cost of efficiency, as models tend to overthink, generating redundant reasoning steps without proportional improvements in output quality. Long-to-Short (L2S) reasoning has emerged as a promising solution to this challenge, aiming to balance reasoning depth with practical efficiency. While existing approaches, such as supervised fine-tuning (SFT), reinforcement learning (RL), and prompt engineering, have shown potential, they are either computationally expensive or unstable. Model merging, on the other hand, offers a cost-effective and robust alternative by integrating the quick-thinking capabilities of System 1 models with the methodical reasoning of System 2 models. In this work, we present a comprehensive empirical study on model merging for L2S reasoning, exploring diverse methodologies, including task-vector-based, SVD-based, and activation-informed merging. Our experiments reveal that model merging can reduce average response length by up to 55% while preserving or even improving baseline performance. We also identify a strong correlation between model scale and merging efficacy with extensive evaluations on 1.5B/7B/14B/32B models. Furthermore, we investigate the merged model's ability to self-critique and self-correct, as well as its adaptive response length based on task complexity. Our findings highlight model merging as a highly efficient and effective paradigm for L2S reasoning, offering a practical solution to the overthinking problem while maintaining the robustness of System 2 reasoning. This work can be found on Github this https URL.
• 摘要：大语模型（LLMS）中从系统1到系统2推理的过渡已明显通过故意，迭代思维来处理复杂任务的显着进步。但是，由于模型倾向于过度思考，因此这种进展通常是以效率为代价的，从而产生了冗余的推理步骤，而无需成比例的提高产出质量。长达（L2S）推理已成为解决这一挑战的有前途的解决方案，旨在平衡推理深度与实际效率。尽管现有的方法（例如受监督的微调（SFT），增强学习（RL）和及时工程学）表现出了潜力，但它们在计算上昂贵或不稳定。另一方面，模型合并通过将系统1模型的快速思维功能与系统2模型的有方法推理相结合，提供了一种具有成本效益且可靠的替代方案。在这项工作中，我们介绍了一项关于L2S推理模型合并的全面实证研究，探讨了不同的方法，包括基于任务 - 基于SVD的基于SVD和激活的合并。我们的实验表明，在保留甚至改善基线性能的同时，模型合并可以将平均响应长度降低多达55％。我们还确定了模型量表与合并功效之间的密切相关性，并在1.5b/7b/14b/32b模型上进行了广泛的评估。此外，我们研究了合并模型的自我评价和自我校正的能力，以及基于任务复杂性的自适应响应长度。我们的发现重点介绍了将L2S推理的高效和有效范式合并的模型，为过度思考问题提供了实用的解决方案，同时保持了系统2推理的鲁棒性。这项工作可以在此HTTPS URL上找到。

Title: TN-Eval: Rubric and Evaluation Protocols for Measuring the Quality of Behavioral Therapy Notes

• Authors: Raj Sanjay Shah, Lei Xu, Qianchu Liu, Jon Burnsky, Drew Bertagnolli, Chaitanya Shivade
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.20648
• Pdf URL: https://arxiv.org/pdf/2503.20648
• Copy Paste: [[2503.20648]] TN-Eval: Rubric and Evaluation Protocols for Measuring the Quality of Behavioral Therapy Notes(https://arxiv.org/abs/2503.20648)
• Keywords: llm, hallucination
• Abstract: Behavioral therapy notes are important for both legal compliance and patient care. Unlike progress notes in physical health, quality standards for behavioral therapy notes remain underdeveloped. To address this gap, we collaborated with licensed therapists to design a comprehensive rubric for evaluating therapy notes across key dimensions: completeness, conciseness, and faithfulness. Further, we extend a public dataset of behavioral health conversations with therapist-written notes and LLM-generated notes, and apply our evaluation framework to measure their quality. We find that: (1) A rubric-based manual evaluation protocol offers more reliable and interpretable results than traditional Likert-scale annotations. (2) LLMs can mimic human evaluators in assessing completeness and conciseness but struggle with faithfulness. (3) Therapist-written notes often lack completeness and conciseness, while LLM-generated notes contain hallucination. Surprisingly, in a blind test, therapists prefer and judge LLM-generated notes to be superior to therapist-written notes.
• 摘要：行为疗法笔记对于法律合规性和患者护理都很重要。与身体健康方面的进度注释不同，行为疗法笔记的质量标准仍然不发达。为了解决这一差距，我们与有执照的治疗师合作设计了一个全面的标题，用于评估跨关键维度的治疗笔记：完整性，简洁性和忠诚。此外，我们通过治疗师写的注释和LLM生成的笔记扩展了行为健康对话的公共数据集，并应用我们的评估框架来衡量其质量。我们发现：（1）基于标语的手动评估协议比传统的李克特级注释提供了更可靠和可解释的结果。 （2）LLM可以模仿人类评估者评估完整性和简洁性，但要忠实地挣扎。 （3）治疗师写的笔记通常缺乏完整性和简洁性，而LLM生成的笔记包含幻觉。令人惊讶的是，在盲目的测试中，治疗师更喜欢并评判LLM生成的笔记优于治疗师写的笔记。

Title: From Annotation to Adaptation: Metrics, Synthetic Data, and Aspect Extraction for Aspect-Based Sentiment Analysis with Large Language Models

• Authors: Nikita Neveditsin, Pawan Lingras, Vijay Mago
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.20715
• Pdf URL: https://arxiv.org/pdf/2503.20715
• Copy Paste: [[2503.20715]] From Annotation to Adaptation: Metrics, Synthetic Data, and Aspect Extraction for Aspect-Based Sentiment Analysis with Large Language Models(https://arxiv.org/abs/2503.20715)
• Keywords: language model, llm
• Abstract: This study examines the performance of Large Language Models (LLMs) in Aspect-Based Sentiment Analysis (ABSA), with a focus on implicit aspect extraction in a novel domain. Using a synthetic sports feedback dataset, we evaluate open-weight LLMs' ability to extract aspect-polarity pairs and propose a metric to facilitate the evaluation of aspect extraction with generative models. Our findings highlight both the potential and limitations of LLMs in the ABSA task.
• 摘要：这项研究研究了基于方面的情感分析（ABSA）中大语言模型（LLM）的性能，重点是新的领域中隐性方面提取。使用合成体育反馈数据集，我们评估了开放式LLM提取方面极性对的能力，并提出了一个度量，以促进使用生成模型评估方面提取的评估。我们的发现突出了LLM在ABSA任务中的潜力和局限性。

Title: Beyond Believability: Accurate Human Behavior Simulation with Fine-Tuned LLMs

• Authors: Yuxuan Lu, Jing Huang, Yan Han, Bennet Bei, Yaochen Xie, Dakuo Wang, Jessie Wang, Qi He
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.20749
• Pdf URL: https://arxiv.org/pdf/2503.20749
• Copy Paste: [[2503.20749]] Beyond Believability: Accurate Human Behavior Simulation with Fine-Tuned LLMs(https://arxiv.org/abs/2503.20749)
• Keywords: llm, prompt, agent
• Abstract: Recent research shows that LLMs can simulate ``believable'' human behaviors to power LLM agents via prompt-only methods. In this work, we focus on evaluating and improving LLM's objective ``accuracy'' rather than the subjective ``believability'' in the web action generation task, leveraging a large-scale, real-world dataset collected from online shopping human actions. We present the first comprehensive quantitative evaluation of state-of-the-art LLMs (e.g., DeepSeek-R1, Llama, and Claude) on the task of web action generation. Our results show that fine-tuning LLMs on real-world behavioral data substantially improves their ability to generate actions compared to prompt-only methods. Furthermore, incorporating synthesized reasoning traces into model training leads to additional performance gains, demonstrating the value of explicit rationale in behavior modeling. This work establishes a new benchmark for evaluating LLMs in behavior simulation and offers actionable insights into how real-world action data and reasoning augmentation can enhance the fidelity of LLM agents.
• 摘要：最近的研究表明，LLM可以通过及时的方法模拟``可信的''人类行为为LLM代理。在这项工作中，我们专注于评估和改善LLM的目标``准确性''，而不是在网络行动生成任务中的主观``可信性''，利用从在线购物人类行动中收集的大规模现实世界数据集。我们介绍了有关Web Action生成任务的最先进的LLM（例如DeepSeek-R1，Llama和Claude）的首次全面定量评估。我们的结果表明，与迅速方法相比，实际行为数据上的微调LLM显着提高了其生成动作的能力。此外，将合成的推理轨迹纳入模型训练会导致额外的绩效提高，从而证明了行为建模中显式理由的价值。这项工作为评估行为模拟中LLM的新基准建立了一个新的基准，并提供了可行的见解，以了解现实世界中的动作数据和推理如何增强可以增强LLM代理的保真度。

Title: MCTS-RAG: Enhancing Retrieval-Augmented Generation with Monte Carlo Tree Search

• Authors: Yunhai Hu, Yilun Zhao, Chen Zhao, Arman Cohan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.20757
• Pdf URL: https://arxiv.org/pdf/2503.20757
• Copy Paste: [[2503.20757]] MCTS-RAG: Enhancing Retrieval-Augmented Generation with Monte Carlo Tree Search(https://arxiv.org/abs/2503.20757)
• Keywords: language model, gpt, llm, hallucination, retrieval-augmented generation
• Abstract: We introduce MCTS-RAG, a novel approach that enhances the reasoning capabilities of small language models on knowledge-intensive tasks by leveraging retrieval-augmented generation (RAG) to provide relevant context and Monte Carlo Tree Search (MCTS) to refine reasoning paths. MCTS-RAG dynamically integrates retrieval and reasoning through an iterative decision-making process. Unlike standard RAG methods, which typically retrieve information independently from reasoning and thus integrate knowledge suboptimally, or conventional MCTS reasoning, which depends solely on internal model knowledge without external facts, MCTS-RAG combines structured reasoning with adaptive retrieval. This integrated approach enhances decision-making, reduces hallucinations, and ensures improved factual accuracy and response consistency. The experimental results on multiple reasoning and knowledge-intensive datasets datasets (i.e., ComplexWebQA, GPQA, and FoolMeTwice) show that our method enables small-scale LMs to achieve performance comparable to frontier LLMs like GPT-4o by effectively scaling inference-time compute, setting a new standard for reasoning in small-scale models.
• 摘要：我们介绍了MCTS-rag，这是一种新颖的方法，它通过利用检索效果（RAG）来提供相关背景和蒙特卡洛树搜索（MCT）来完善推理路径，从而增强了小语言模型在知识密集型任务上的推理能力。 MCTS rag通过迭代决策过程动态整合检索和推理。与标准的抹布方法不同，通常会独立于推理地检索信息，从而从次优或传统的MCT推理中整合知识，这仅取决于没有外部事实的内部模型知识，MCTS-RAG将结构化推理与自适应检索相结合。这种综合方法可以增强决策，减少幻觉，并确保提高事实准确性和响应一致性。关于多个推理和知识密集数据集数据集的实验结果（即复杂的WebQA，GPQA和FoolMetwice）表明，我们的方法使小规模的LMS能够通过有效地扩展推理时间计算，为小规模模型设置新标准，以实现与GPT-4O（例如GPT-4O）相当的性能。

Title: Mobile-MMLU: A Mobile Intelligence Language Understanding Benchmark

• Authors: Sondos Mahmoud Bsharat, Mukul Ranjan, Aidar Myrzakhan, Jiacheng Liu, Bowei Guo, Shengkun Tang, Zhuang Liu, Yuanzhi Li, Zhiqiang Shen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.20786
• Pdf URL: https://arxiv.org/pdf/2503.20786
• Copy Paste: [[2503.20786]] Mobile-MMLU: A Mobile Intelligence Language Understanding Benchmark(https://arxiv.org/abs/2503.20786)
• Keywords: language model, llm
• Abstract: Rapid advancements in large language models (LLMs) have increased interest in deploying them on mobile devices for on-device AI applications. Mobile users interact differently with LLMs compared to desktop users, creating unique expectations and data biases. Current benchmark datasets primarily target at server and desktop environments, and there is a notable lack of extensive datasets specifically designed for mobile contexts. Additionally, mobile devices face strict limitations in storage and computing resources, constraining model size and capabilities, thus requiring optimized efficiency and prioritized knowledge. To address these challenges, we introduce Mobile-MMLU, a large-scale benchmark dataset tailored for mobile intelligence. It consists of 16,186 questions across 80 mobile-related fields, designed to evaluate LLM performance in realistic mobile scenarios. A challenging subset, Mobile-MMLU-Pro, provides advanced evaluation similar in size to MMLU-Pro but significantly more difficult than our standard full set. Both benchmarks use multiple-choice, order-invariant questions focused on practical mobile interactions, such as recipe suggestions, travel planning, and essential daily tasks. The dataset emphasizes critical mobile-specific metrics like inference latency, energy consumption, memory usage, and response quality, offering comprehensive insights into model performance under mobile constraints. Moreover, it prioritizes privacy and adaptability, assessing models' ability to perform on-device processing, maintain user privacy, and adapt to personalized usage patterns. Mobile-MMLU family offers a standardized framework for developing and comparing mobile-optimized LLMs, enabling advancements in productivity and decision-making within mobile computing environments. Our code and data are available at: this https URL.
• 摘要：大型语言模型（LLMS）的快速进步对将其部署在移动设备上的设备上的AI应用程序中的兴趣增加。与桌面用户相比，移动用户与LLM的交互方式不同，从而创造了独特的期望和数据偏见。当前的基准数据集主要针对服务器和桌面环境，并且明显缺乏专门为移动上下文设计的广泛数据集。此外，移动设备在存储和计算资源中面临严格的限制，限制了模型的大小和功能，因此需要优化的效率和优先的知识。为了应对这些挑战，我们引入了Mobile-MMLU，这是一种针对移动智能的大规模基准数据集。它由80个与移动相关领域的16186个问题组成，旨在评估现实的移动方案中的LLM性能。一个具有挑战性的子集移动-MMLU-PRO提供的高级评估尺寸与MMLU-PRO相似，但比我们的标准完整集更加困难。这两个基准都使用多项选择，订单不变的问题，重点介绍了实际的移动互动，例如食谱建议，旅行计划和必不可少的日常任务。该数据集强调了关键的移动特异性指标，例如推理潜伏期，能源消耗，内存使用和响应质量，从而在移动约束下对模型性能提供了全面的见解。此外，它优先考虑隐私和适应性，评估模型执行设备处理，维护用户隐私并适应个性化用法模式的能力。 Mobile-MMLU家族提供了一个标准化的框架，用于开发和比较移动优化的LLM，从而在移动计算环境中提高了生产力和决策的进步。我们的代码和数据可在以下网址提供：此HTTPS URL。