2025-05-02

Title: Rosetta-PL: Propositional Logic as a Benchmark for Large Language Model Reasoning

• Authors: Shaun Baek, Shaun Esua-Mensah, Cyrus Tsui, Sejan Vigneswaralingam, Abdullah Alali, Michael Lu, Vasu Sharma, Kevin Zhu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.00001
• Pdf URL: https://arxiv.org/pdf/2505.00001
• Copy Paste: [[2505.00001]] Rosetta-PL: Propositional Logic as a Benchmark for Large Language Model Reasoning(https://arxiv.org/abs/2505.00001)
• Keywords: language model, gpt, llm
• Abstract: Large Language Models (LLMs) are primarily trained on high-resource natural languages, limiting their effectiveness in low-resource settings and in tasks requiring deep logical reasoning. This research introduces Rosetta-PL, a benchmark designed to evaluate LLMs' logical reasoning and generalization capabilities in a controlled environment. We construct Rosetta-PL by translating a dataset of logical propositions from Lean into a custom logical language, which is then used to fine-tune an LLM (e.g., GPT-4o). Our experiments analyze the impact of the size of the dataset and the translation methodology on the performance of the model. Our results indicate that preserving logical relationships in the translation process significantly boosts precision, with accuracy plateauing beyond roughly 20,000 training samples. These insights provide valuable guidelines for optimizing LLM training in formal reasoning tasks and improving performance in various low-resource language applications.
• 摘要：大型语言模型（LLMS）主要是对高资源自然语言进行培训的，从而限制了其在低资源环境中的有效性以及需要深层逻辑推理的任务。这项研究介绍了Rosetta-Pl，这是一种基准，旨在评估LLMS在受控环境中的逻辑推理和概括能力。我们通过将逻辑命题的数据集从精益转换为自定义的逻辑语言来构建Rosetta-Pl，然后将其用于微调LLM（例如GPT-4O）。我们的实验分析了数据集大小的影响以及翻译方法对模型性能的影响。我们的结果表明，在翻译过程中保存逻辑关系可显着提高精度，精确度高达大约20,000个培训样本。这些见解提供了在正式推理任务中优化LLM培训的宝贵指南，并改善各种低资源语言应用程序的性能。

Title: The Mind in the Machine: A Survey of Incorporating Psychological Theories in LLMs

• Authors: Zizhou Liu, Ziwei Gong, Lin Ai, Zheng Hui, Run Chen, Colin Wayne Leach, Michelle R. Greene, Julia Hirschberg
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.00003
• Pdf URL: https://arxiv.org/pdf/2505.00003
• Copy Paste: [[2505.00003]] The Mind in the Machine: A Survey of Incorporating Psychological Theories in LLMs(https://arxiv.org/abs/2505.00003)
• Keywords: language model, llm
• Abstract: Psychological insights have long shaped pivotal NLP breakthroughs, including the cognitive underpinnings of attention mechanisms, formative reinforcement learning, and Theory of Mind-inspired social modeling. As Large Language Models (LLMs) continue to grow in scale and complexity, there is a rising consensus that psychology is essential for capturing human-like cognition, behavior, and interaction. This paper reviews how psychological theories can inform and enhance stages of LLM development, including data, pre-training, post-training, and evaluation\&application. Our survey integrates insights from cognitive, developmental, behavioral, social, personality psychology, and psycholinguistics. Our analysis highlights current trends and gaps in how psychological theories are applied. By examining both cross-domain connections and points of tension, we aim to bridge disciplinary divides and promote more thoughtful integration of psychology into future NLP research.
• 摘要：心理洞察力长期以来一直是关键的NLP突破，包括注意机制的认知基础，形成性的增强学习和受思想启发的社会建模理论。随着大型语言模型（LLMS）的规模和复杂性不断增长，人们共识的提高是，心理学对于捕获类似人类的认知，行为和互动至关重要。本文回顾了心理理论如何为LLM开发的阶段提供信息和增强，包括数据，培训前，培训和评估\＆应用。我们的调查整合了认知，发展，行为，社会，人格心理学和心理语言学的见解。我们的分析强调了当前的趋势和差距，如何应用心理理论。通过研究跨域的联系和紧张点，我们旨在弥合纪律处分，并促进心理学将心理学更加周到的整合到未来的NLP研究中。

Title: LangVAE and LangSpace: Building and Probing for Language Model VAEs

• Authors: Danilo S. Carvalho, Yingji Zhang, Harriet Unsworth, André Freitas
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.00004
• Pdf URL: https://arxiv.org/pdf/2505.00004
• Copy Paste: [[2505.00004]] LangVAE and LangSpace: Building and Probing for Language Model VAEs(https://arxiv.org/abs/2505.00004)
• Keywords: language model, llm
• Abstract: We present LangVAE, a novel framework for modular construction of variational autoencoders (VAEs) on top of pre-trained large language models (LLMs). Such language model VAEs can encode the knowledge of their pre-trained components into more compact and semantically disentangled representations. The representations obtained in this way can be analysed with the LangVAE companion framework: LangSpace, which implements a collection of probing methods, such as vector traversal and interpolation, disentanglement measures, and cluster visualisations. LangVAE and LangSpace offer a flexible, efficient and scalable way of building and analysing textual representations, with simple integration for models available on the HuggingFace Hub. Additionally, we conducted a set of experiments with different encoder and decoder combinations, as well as annotated inputs, revealing a wide range of interactions across architectural families and sizes w.r.t. generalisation and disentanglement. Our findings demonstrate a promising framework for systematising the experimentation and understanding of textual representations.
• 摘要：我们提出了Langvae，这是一种在预训练的大语言模型（LLMS）之上的变异自动编码器（VAE）模块化构建的新型框架。这样的语言模型VAE可以将其预训练组件的知识编码为更紧凑和语义上的分离表示。可以使用Langvae Companion框架来分析以这种方式获得的表示形式：Langspace，该框架实现了探测方法的集合，例如向量遍历和插值，分解测量和集群可视化。 Langvae和Langspace提供了一种灵活，高效且可扩展的方式来构建和分析文本表示形式，并简单地集成了HuggingFace Hub上可用的模型。此外，我们通过不同的编码器和解码器组合以及注释的输入进行了一组实验，揭示了建筑家族和尺寸W.R.T.之间的广泛相互作用。概括和分解。我们的发现展示了一个有前途的框架，用于系统化对文本表示的实验和理解。

Title: Toward a digital twin of U.S. Congress

• Authors: Hayden Helm, Tianyi Chen, Harvey McGuinness, Paige Lee, Brandon Duderstadt, Carey E. Priebe
• Subjects: cs.CL, cs.AI, cs.CY, cs.SI
• Abstract URL: https://arxiv.org/abs/2505.00006
• Pdf URL: https://arxiv.org/pdf/2505.00006
• Copy Paste: [[2505.00006]] Toward a digital twin of U.S. Congress(https://arxiv.org/abs/2505.00006)
• Keywords: language model
• Abstract: In this paper we provide evidence that a virtual model of U.S. congresspersons based on a collection of language models satisfies the definition of a digital twin. In particular, we introduce and provide high-level descriptions of a daily-updated dataset that contains every Tweet from every U.S. congressperson during their respective terms. We demonstrate that a modern language model equipped with congressperson-specific subsets of this data are capable of producing Tweets that are largely indistinguishable from actual Tweets posted by their physical counterparts. We illustrate how generated Tweets can be used to predict roll-call vote behaviors and to quantify the likelihood of congresspersons crossing party lines, thereby assisting stakeholders in allocating resources and potentially impacting real-world legislative dynamics. We conclude with a discussion of the limitations and important extensions of our analysis.
• 摘要：在本文中，我们提供的证据表明，基于语言模型的集合，美国国会议员的虚拟模型满足了数字双胞胎的定义。特别是，我们介绍并提供了每日升级数据集的高级描述，该数据集包含每个美国代表人在各自的条款中的每条推文。我们证明，配备了该数据的特定于国会议员子集的现代语言模型能够生成与其物理对应物发布的实际推文几乎没有区别的推文。我们说明如何使用生成的推文来预测滚动投票行为，并量化国会议员越过政党线路的可能性，从而帮助利益相关者分配资源并潜在地影响现实世界的立法动态。我们以讨论分析的局限性和重要扩展进行了讨论。

Title: A Scoping Review of Natural Language Processing in Addressing Medically Inaccurate Information: Errors, Misinformation, and Hallucination

• Authors: Zhaoyi Sun, Wen-Wai Yim, Ozlem Uzuner, Fei Xia, Meliha Yetisgen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.00008
• Pdf URL: https://arxiv.org/pdf/2505.00008
• Copy Paste: [[2505.00008]] A Scoping Review of Natural Language Processing in Addressing Medically Inaccurate Information: Errors, Misinformation, and Hallucination(https://arxiv.org/abs/2505.00008)
• Keywords: hallucination
• Abstract: Objective: This review aims to explore the potential and challenges of using Natural Language Processing (NLP) to detect, correct, and mitigate medically inaccurate information, including errors, misinformation, and hallucination. By unifying these concepts, the review emphasizes their shared methodological foundations and their distinct implications for healthcare. Our goal is to advance patient safety, improve public health communication, and support the development of more reliable and transparent NLP applications in healthcare. Methods: A scoping review was conducted following PRISMA guidelines, analyzing studies from 2020 to 2024 across five databases. Studies were selected based on their use of NLP to address medically inaccurate information and were categorized by topic, tasks, document types, datasets, models, and evaluation metrics. Results: NLP has shown potential in addressing medically inaccurate information on the following tasks: (1) error detection (2) error correction (3) misinformation detection (4) misinformation correction (5) hallucination detection (6) hallucination mitigation. However, challenges remain with data privacy, context dependency, and evaluation standards. Conclusion: This review highlights the advancements in applying NLP to tackle medically inaccurate information while underscoring the need to address persistent challenges. Future efforts should focus on developing real-world datasets, refining contextual methods, and improving hallucination management to ensure reliable and transparent healthcare applications.
• 摘要：目的：本综述旨在探讨使用自然语言处理（NLP）检测，纠正和减轻医学上不准确的信息的潜在和挑战，包括错误，错误信息和幻觉。通过统一这些概念，该评论强调了他们共同的方法论基础及其对医疗保健的独特影响。我们的目标是提高患者安全，改善公共卫生沟通，并支持更可靠和透明的NLP在医疗保健中的开发。方法：根据PRISMA指南进行了范围审查，分析了2020年至2024年的研究，跨五个数据库进行了研究。根据他们使用NLP来解决医学上不准确的信息，选择研究，并按主题，任务，文档类型，数据集，模型和评估指标对研究进行了分类。结果：NLP在解决以下任务的医学上不准确信息方面显示出潜力：（1）错误检测（2）错误校正（3）错误信息检测（4）错误信息校正校正（5）幻觉检测（6）幻觉缓解。但是，数据隐私，上下文依赖性和评估标准仍然存在挑战。结论：这篇评论重点介绍了应用NLP来解决医学上不准确的信息的进步，同时强调解决持续挑战的需求。未来的努力应着重于开发现实世界数据集，完善上下文方法以及改善幻觉管理，以确保可靠和透明的医疗保健应用程序。

Title: Efficient Knowledge Transfer in Multi-Task Learning through Task-Adaptive Low-Rank Representation

• Authors: Xiao Zhang, Kangsheng Wang, Tianyu Hu, Huimin Ma
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.00009
• Pdf URL: https://arxiv.org/pdf/2505.00009
• Copy Paste: [[2505.00009]] Efficient Knowledge Transfer in Multi-Task Learning through Task-Adaptive Low-Rank Representation(https://arxiv.org/abs/2505.00009)
• Keywords: language model, prompt
• Abstract: Pre-trained language models (PLMs) demonstrate remarkable intelligence but struggle with emerging tasks unseen during training in real-world applications. Training separate models for each new task is usually impractical. Multi-task learning (MTL) addresses this challenge by transferring shared knowledge from source tasks to target tasks. As an dominant parameter-efficient fine-tuning method, prompt tuning (PT) enhances MTL by introducing an adaptable vector that captures task-specific knowledge, which acts as a prefix to the original prompt that preserves shared knowledge, while keeping PLM parameters frozen. However, PT struggles to effectively capture the heterogeneity of task-specific knowledge due to its limited representational capacity. To address this challenge, we propose Task-Adaptive Low-Rank Representation (TA-LoRA), an MTL method built on PT, employing the low-rank representation to model task heterogeneity and a fast-slow weights mechanism where the slow weight encodes shared knowledge, while the fast weight captures task-specific nuances, avoiding the mixing of shared and task-specific knowledge, caused by training low-rank representations from scratch. Moreover, a zero-initialized attention mechanism is introduced to minimize the disruption of immature low-rank components on original prompts during warm-up epochs. Experiments on 16 tasks demonstrate that TA-LoRA achieves state-of-the-art performance in full-data and few-shot settings while maintaining superior parameter efficiency.
• 摘要：预训练的语言模型（PLM）表现出了非凡的智力，但在现实世界应用程序中培训期间看不见的任务挣扎。为每个新任务培训单独的模型通常是不切实际的。多任务学习（MTL）通过将共享知识从源任务转移到目标任务来解决这一挑战。作为一种主要参数高效调整方法，提示调整（PT）通过引入可捕获特定任务知识的适应性向量来增强MTL，该矢量充当了原始提示的前缀，该前缀保留了共享知识，同时保持PLM参数冻结。但是，由于其有限的代表性，PT努力有效地捕获特定于任务知识的异质性。 To address this challenge, we propose Task-Adaptive Low-Rank Representation (TA-LoRA), an MTL method built on PT, employing the low-rank representation to model task heterogeneity and a fast-slow weights mechanism where the slow weight encodes shared knowledge, while the fast weight captures task-specific nuances, avoiding the mixing of shared and task-specific knowledge, caused by training low-rank representations from scratch.此外，引入了零分子的注意机制，以最大程度地减少在热身时期期间原始提示上未成熟的低级别组件的破坏。对16个任务的实验表明，Ta-Lora在全数据和少量设置中实现了最先进的性能，同时保持了出色的参数效率。

Title: Jailbreak Detection in Clinical Training LLMs Using Feature-Based Predictive Models

• Authors: Tri Nguyen, Lohith Srikanth Pentapalli, Magnus Sieverding, Laurah Turner, Seth Overla, Weibing Zheng, Chris Zhou, David Furniss, Danielle Weber, Michael Gharib, Matt Kelleher, Michael Shukis, Cameron Pawlik, Kelly Cohen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.00010
• Pdf URL: https://arxiv.org/pdf/2505.00010
• Copy Paste: [[2505.00010]] Jailbreak Detection in Clinical Training LLMs Using Feature-Based Predictive Models(https://arxiv.org/abs/2505.00010)
• Keywords: language model, llm, prompt
• Abstract: Jailbreaking in Large Language Models (LLMs) threatens their safe use in sensitive domains like education by allowing users to bypass ethical safeguards. This study focuses on detecting jailbreaks in 2-Sigma, a clinical education platform that simulates patient interactions using LLMs. We annotated over 2,300 prompts across 158 conversations using four linguistic variables shown to correlate strongly with jailbreak behavior. The extracted features were used to train several predictive models, including Decision Trees, Fuzzy Logic-based classifiers, Boosting methods, and Logistic Regression. Results show that feature-based predictive models consistently outperformed Prompt Engineering, with the Fuzzy Decision Tree achieving the best overall performance. Our findings demonstrate that linguistic-feature-based models are effective and explainable alternatives for jailbreak detection. We suggest future work explore hybrid frameworks that integrate prompt-based flexibility with rule-based robustness for real-time, spectrum-based jailbreak monitoring in educational LLMs.
• 摘要：大型语言模型（LLMS）越狱，通过允许用户绕过道德保障，威胁着他们在敏感领域（如教育）中的安全使用。这项研究的重点是检测2-Sigma的越狱，这是一个临床教育平台，使用LLM模拟患者互动。我们使用了四个与越狱行为密切相关的语言变量，在158次对话中注释了158次对话的2300多个提示。提取的功能用于训练多种预测模型，包括决策树，基于模糊逻辑的分类器，增强方法和逻辑回归。结果表明，基于功能的预测模型始终超过及时的工程，而模糊的决策树实现了最佳的整体性能。我们的发现表明，基于语言的模型是越狱检测的有效替代方法。我们建议未来的工作探索混合框架，这些框架将基于迅速的灵活性与基于规则的鲁棒性集成到教育LLM中的实时，基于频谱的越狱监控。

Title: Performance Evaluation of Emotion Classification in Japanese Using RoBERTa and DeBERTa

• Authors: Yoichi Takenaka
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.00013
• Pdf URL: https://arxiv.org/pdf/2505.00013
• Copy Paste: [[2505.00013]] Performance Evaluation of Emotion Classification in Japanese Using RoBERTa and DeBERTa(https://arxiv.org/abs/2505.00013)
• Keywords: language model, gpt, llm, prompt, chat
• Abstract: Background Practical applications such as social media monitoring and customer-feedback analysis require accurate emotion detection for Japanese text, yet resource scarcity and class imbalance hinder model performance. Objective This study aims to build a high-accuracy model for predicting the presence or absence of eight Plutchik emotions in Japanese sentences. Methods Using the WRIME corpus, we transform reader-averaged intensity scores into binary labels and fine-tune four pre-trained language models (BERT, RoBERTa, DeBERTa-v3-base, DeBERTa-v3-large). For context, we also assess two large language models (TinySwallow-1.5B-Instruct and ChatGPT-4o). Accuracy and F1-score serve as evaluation metrics. Results DeBERTa-v3-large attains the best mean accuracy (0.860) and F1-score (0.662), outperforming all other models. It maintains robust F1 across both high-frequency emotions (e.g., Joy, Anticipation) and low-frequency emotions (e.g., Anger, Trust). The LLMs lag, with ChatGPT-4o and TinySwallow-1.5B-Instruct scoring 0.527 and 0.292 in mean F1, respectively. Conclusion The fine-tuned DeBERTa-v3-large model currently offers the most reliable solution for binary emotion classification in Japanese. We release this model as a pip-installable package (pip install deberta-emotion-predictor). Future work should augment data for rare emotions, reduce model size, and explore prompt engineering to improve LLM performance. This manuscript is under review for possible publication in New Generation Computing.
• 摘要：背景实用应用，例如社交媒体监控和客户反馈分析，需要对日本文本进行准确的情感检测，但资源稀缺和类不平衡会阻碍模型的性能。目的本研究旨在建立一个高准确模型，以预测日本句子中八种plutchik情绪的存在或不存在。使用WRIME语料库的方法，我们将读取器平均强度得分转化为二进制标签，并微调四个预训练的语言模型（Bert，Roberta，Deberta-V3-Base，Deberta-V3-large）。对于上下文，我们还评估了两个大型语言模型（Tinyswallow-1.5b-instruct和Chatgpt-4O）。准确性和F1得分用作评估指标。结果Deberta-V3-Large达到了最佳平均准确性（0.860）和F1得分（0.662），表现优于所有其他模型。它在高频情绪（例如欢乐，预期）和低频情绪（例如愤怒，信任）中保持了强大的F1。 LLMS滞后，分别为平均F1的Chatgpt-4O和Tinyswallow-1.5b-1.5B教学得分和0.292。结论当前，微调的Deberta-V3总模型为日语提供了最可靠的二进制情感分类解决方案。我们将此模型发布为PIP安装软件包（PIP安装Deberta-Emotion-Predictor）。未来的工作应增加数据以获得罕见情绪，减少模型大小，并探索迅速的工程以提高LLM的性能。该手稿正在审查中，以在新一代计算中发表。

Title: Design and Application of Multimodal Large Language Model Based System for End to End Automation of Accident Dataset Generation

• Authors: MD Thamed Bin Zaman Chowdhury, Moazzem Hossain
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.00015
• Pdf URL: https://arxiv.org/pdf/2505.00015
• Copy Paste: [[2505.00015]] Design and Application of Multimodal Large Language Model Based System for End to End Automation of Accident Dataset Generation(https://arxiv.org/abs/2505.00015)
• Keywords: language model, llm
• Abstract: Road traffic accidents remain a major public safety and socio-economic issue in developing countries like Bangladesh. Existing accident data collection is largely manual, fragmented, and unreliable, resulting in underreporting and inconsistent records. This research proposes a fully automated system using Large Language Models (LLMs) and web scraping techniques to address these challenges. The pipeline consists of four components: automated web scraping code generation, news collection from online sources, accident news classification with structured data extraction, and duplicate removal. The system uses the multimodal generative LLM Gemini-2.0-Flash for seamless automation. The code generation module classifies webpages into pagination, dynamic, or infinite scrolling categories and generates suitable Python scripts for scraping. LLMs also classify and extract key accident information such as date, time, location, fatalities, injuries, road type, vehicle types, and pedestrian involvement. A deduplication algorithm ensures data integrity by removing duplicate reports. The system scraped 14 major Bangladeshi news sites over 111 days (Oct 1, 2024 - Jan 20, 2025), processing over 15,000 news articles and identifying 705 unique accidents. The code generation module achieved 91.3% calibration and 80% validation accuracy. Chittagong reported the highest number of accidents (80), fatalities (70), and injuries (115), followed by Dhaka, Faridpur, Gazipur, and Cox's Bazar. Peak accident times were morning (8-9 AM), noon (12-1 PM), and evening (6-7 PM). A public repository was also developed with usage instructions. This study demonstrates the viability of an LLM-powered, scalable system for accurate, low-effort accident data collection, providing a foundation for data-driven road safety policymaking in Bangladesh.
• 摘要：在孟加拉国等发展中国家，道路交通事故仍然是主要的公共安全和社会经济问题。现有的事故数据收集在很大程度上是手动，分散且不可靠的，导致记录不足和不一致的记录。这项研究提出了一个使用大语言模型（LLM）和网络刮擦技术来应对这些挑战的全自动系统。该管道由四个组成部分组成：自动化网络刮擦代码生成，在线来源的新闻收集，带有结构化数据提取的事故新闻分类以及重复的删除。该系统使用多模式生成LLM Gemini-2.0-Flash进行无缝自动化。代码生成模块将网页分类为分页，动态或无限滚动类别，并生成合适的Python脚本用于刮擦。 LLM还对关键事故信息进行分类，例如日期，时间，位置，死亡，伤害，道路类型，车辆类型和行人参与。重复数据删除算法可通过删除重复报告来确保数据完整性。该系统在111天（2024年10月1日至2025年1月20日）中刮了14个主要的孟加拉国新闻网站，处理了15,000多种新闻文章，并确定了705起独特的事故。代码生成模块实现了91.3％的校准和80％的验证精度。吉大港报道了事故数量最多（80），死亡（70）和伤害（115），其次是达卡，法里德布尔，加兹普尔和考克斯的巴扎尔。高峰事故时间是早晨（上午8-9），中午（下午12-1）和晚上（下午6-7点）。还通过使用说明开发了一个公共存储库。这项研究证明了LLM驱动的可扩展系统的生存能力，可用于准确，低劳动事故数据收集，为孟加拉国数据驱动的道路安全决策提供了基础。

Title: Sparks of Tabular Reasoning via Text2SQL Reinforcement Learning

• Authors: Josefa Lia Stoisser, Marc Boubnovski Martell, Julien Fauqueur
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.00016
• Pdf URL: https://arxiv.org/pdf/2505.00016
• Copy Paste: [[2505.00016]] Sparks of Tabular Reasoning via Text2SQL Reinforcement Learning(https://arxiv.org/abs/2505.00016)
• Keywords: language model, llm, chain-of-thought
• Abstract: This work reframes the Text-to-SQL task as a pathway for teaching large language models (LLMs) to reason over and manipulate tabular data--moving beyond the traditional focus on query generation. We propose a two-stage framework that leverages SQL supervision to develop transferable table reasoning capabilities. First, we synthesize detailed chain-of-thought (CoT) traces from real-world SQL queries, providing step-by-step, clause-level supervision that teaches the model how to traverse, filter, and aggregate table fields. Second, we introduce a Group Relative Policy Optimization (GRPO) reinforcement learning objective that connects SQL execution accuracy to generalizable reasoning by encouraging steps that extend beyond task-specific syntax and transfer across datasets. Empirically, our approach improves performance on standard Text-to-SQL benchmarks and achieves substantial gains on reasoning-intensive datasets such as BIRD and CRT-QA, demonstrating enhanced generalization and interpretability. Specifically, the distilled-quantized LLaMA model achieved a 20\% increase in accuracy when trained on Text-to-SQL tasks, while Qwen achieved a 5\% increase. These results suggest that SQL can serve not only as a target formalism but also as an effective scaffold for learning robust, transferable reasoning over structured data.
• 摘要：这项工作将文本到SQL任务重新制定为教授大型语言模型（LLMS）以推理和操纵表格数据的途径 - 超越了传统对查询生成的关注。我们提出了一个两阶段的框架，该框架利用SQL监督来开发可转移的表推理功能。首先，我们从现实世界中的查询中综合了详细的思想链（COT）痕迹，提供逐步的，子句级的监督，该监督教授模型如何穿越，过滤和聚集表字段。其次，我们引入了一个小组相对策略优化（GRPO）增强学习目标，该学习目标通过鼓励超越特定于任务的语法和跨数据集传输的步骤将SQL执行精度与可推广的推理联系起来。从经验上讲，我们的方法提高了标准文本到SQL基准的性能，并在诸如鸟类和CRT-QA等推理密集型数据集上取得了可观的收益，表明了增强的概括和解释性。具体而言，在文本到SQL任务训练时，蒸馏量化的Llama模型的准确度增加了20 \％，而QWEN的精度则增加了5 \％。这些结果表明，SQL不仅可以用作目标形式主义，而且可以作为学习强大的，可转移的结构化数据的有效脚手架。

Title: ReCellTy: Domain-specific knowledge graph retrieval-augmented LLMs workflow for single-cell annotation

• Authors: Dezheng Han, Yibin Jia, Ruxiao Chen, Wenjie Han, Shuaishuai Guo, Jianbo Wang
• Subjects: cs.CL, cs.AI, cs.DB, cs.LG
• Abstract URL: https://arxiv.org/abs/2505.00017
• Pdf URL: https://arxiv.org/pdf/2505.00017
• Copy Paste: [[2505.00017]] ReCellTy: Domain-specific knowledge graph retrieval-augmented LLMs workflow for single-cell annotation(https://arxiv.org/abs/2505.00017)
• Keywords: language model, llm
• Abstract: To enable precise and fully automated cell type annotation with large language models (LLMs), we developed a graph structured feature marker database to retrieve entities linked to differential genes for cell reconstruction. We further designed a multi task workflow to optimize the annotation process. Compared to general purpose LLMs, our method improves human evaluation scores by up to 0.21 and semantic similarity by 6.1% across 11 tissue types, while more closely aligning with the cognitive logic of manual annotation.
• 摘要：为了使用大语言模型（LLMS）启用精确和全自动的单元格类型注释，我们开发了一个图形结构化特征标记数据库，以检索与差分基因进行细胞重建的实体。我们进一步设计了一个多任务工作流程，以优化注释过程。与通用LLM相比，我们的方法在11种组织类型中提高了人类评估评分高达0.21，语义相似性提高了6.1％，同时与手动注释的认知逻辑更加紧密。

Title: An Empirical Study on Prompt Compression for Large Language Models

• Authors: Zheng Zhang, Jinyi Li, Yihuai Lan, Xiang Wang, Hao Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.00019
• Pdf URL: https://arxiv.org/pdf/2505.00019
• Copy Paste: [[2505.00019]] An Empirical Study on Prompt Compression for Large Language Models(https://arxiv.org/abs/2505.00019)
• Keywords: language model, llm, long context, hallucination, prompt
• Abstract: Prompt engineering enables Large Language Models (LLMs) to perform a variety of tasks. However, lengthy prompts significantly increase computational complexity and economic costs. To address this issue, we study six prompt compression methods for LLMs, aiming to reduce prompt length while maintaining LLM response quality. In this paper, we present a comprehensive analysis covering aspects such as generation performance, model hallucinations, efficacy in multimodal tasks, word omission analysis, and more. We evaluate these methods across 13 datasets, including news, scientific articles, commonsense QA, math QA, long-context QA, and VQA datasets. Our experiments reveal that prompt compression has a greater impact on LLM performance in long contexts compared to short ones. In the Longbench evaluation, moderate compression even enhances LLM performance. Our code and data is available at this https URL.
• 摘要：及时工程使大型语言模型（LLMS）能够执行各种任务。但是，漫长的提示大大提高了计算复杂性和经济成本。为了解决这个问题，我们研究了LLM的六种迅速压缩方法，旨在降低迅速的长度，同时保持LLM响应质量。在本文中，我们进行了全面的分析，涵盖了诸如发电性能，模型幻觉，多模式任务的功效，单词遗漏分析等方面的方面。我们在13个数据集中评估了这些方法，包括新闻，科学文章，常识质量质量检查，Math QA，Long-Contept QA和VQA数据集。我们的实验表明，与短相比，迅速的压缩对LLM性能的影响更大。在Longbench评估中，中等压缩甚至可以提高LLM性能。我们的代码和数据可在此HTTPS URL上找到。

Title: Beyond Public Access in LLM Pre-Training Data

• Authors: Sruly Rosenblat, Tim O'Reilly, Ilan Strauss
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.00020
• Pdf URL: https://arxiv.org/pdf/2505.00020
• Copy Paste: [[2505.00020]] Beyond Public Access in LLM Pre-Training Data(https://arxiv.org/abs/2505.00020)
• Keywords: language model, gpt, llm
• Abstract: Using a legally obtained dataset of 34 copyrighted O'Reilly Media books, we apply the DE-COP membership inference attack method to investigate whether OpenAI's large language models were trained on copyrighted content without consent. Our AUROC scores show that GPT-4o, OpenAI's more recent and capable model, demonstrates strong recognition of paywalled O'Reilly book content (AUROC = 82\%), compared to OpenAI's earlier model GPT-3.5 Turbo. In contrast, GPT-3.5 Turbo shows greater relative recognition of publicly accessible O'Reilly book samples. GPT-4o Mini, as a much smaller model, shows no knowledge of public or non-public O'Reilly Media content when tested (AUROC $\approx$ 50\%). Testing multiple models, with the same cutoff date, helps us account for potential language shifts over time that might bias our findings. These results highlight the urgent need for increased corporate transparency regarding pre-training data sources as a means to develop formal licensing frameworks for AI content training
• 摘要：使用合法获得的34个受版权保护的O'Reilly Media Books的数据集，我们将DE-COP成员推理攻击方法应用于未经同意未经同意的版权内容培训OpenAI的大语言模型。我们的AUROC分数表明，与OpenAI的较早的型号GPT-3.5 Turbo相比，Openai的最新模型GPT-4O证明了对付费墙的O'Reilly书籍内容（AUROC = 82 \％）的强烈认识。相比之下，GPT-3.5 Turbo显示出对公共访问的O'Reilly Book样本的相对认识。 GPT-4O MINI作为一个较小的模型，在测试时对公共或非公共O'Reilly媒体内容不了解（AUROC $ \ $ 50 \％）。测试具有相同截止日期的多个模型，有助于我们考虑可能偏向我们发现的潜在语言变化。这些结果凸显了迫切需要提高公司透明度对培训数据源的透明度，以此作为为AI内容培训开发正式许可框架的一种手段

Title: Aleph-Alpha-GermanWeb: Improving German-language LLM pre-training with model-based data curation and synthetic data generation

• Authors: Thomas F Burns, Letitia Parcalabescu, Stephan Wäldchen, Michael Barlow, Gregor Ziegltrum, Volker Stampa, Bastian Harren, Björn Deiseroth
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2505.00022
• Pdf URL: https://arxiv.org/pdf/2505.00022
• Copy Paste: [[2505.00022]] Aleph-Alpha-GermanWeb: Improving German-language LLM pre-training with model-based data curation and synthetic data generation(https://arxiv.org/abs/2505.00022)
• Keywords: language model, llm
• Abstract: Scaling data quantity is essential for large language models (LLMs), yet recent findings show that data quality can significantly boost performance and training efficiency. We introduce a German-language dataset curation pipeline that combines heuristic and model-based filtering techniques with synthetic data generation. We use our pipeline to create Aleph-Alpha-GermanWeb, a large-scale German pre-training dataset which draws from: (1) Common Crawl web data, (2) FineWeb2, and (3) synthetically-generated data conditioned on actual, organic web data. We evaluate our dataset by pre-training both a 1B Llama-style model and an 8B tokenizer-free hierarchical autoregressive transformer (HAT). A comparison on German-language benchmarks, including MMMLU, shows significant performance gains of Aleph-Alpha-GermanWeb over FineWeb2 alone. This advantage holds at the 8B scale even when FineWeb2 is enriched by human-curated high-quality data sources such as Wikipedia. Our findings support the growing body of evidence that model-based data curation and synthetic data generation can significantly enhance LLM pre-training datasets.
• 摘要：缩放数据数量对于大语言模型（LLM）至关重要，但最近的发现表明，数据质量可以显着提高性能和训练效率。我们介绍了德语数据集策展管道，该管道将启发式和基于模型的过滤技术与合成数据的生成结合在一起。我们使用管道来创建Aleph-Alpha-GermanWeb，这是一种大规模的德国预训练数据集，该数据集从以下方式汲取：（1）常见的爬网Web数据，（2）FineWeb2和（3）合成生成的数据以实际的有机Web数据为条件。我们通过预先培训1B Llama风格的模型和8B标记层的层次自动回归变压器（HAT）来评估我们的数据集。包括MMMLU在内的德语基准测试的比较，仅在FineWeb2上就显示了Aleph-Alpha-Germanweb的显着性能。即使FineWeb2被人体策划的高质量数据源（例如Wikipedia）富含，这一优势也具有8B量表。我们的发现支持了越来越多的证据体系，即基于模型的数据策展和合成数据生成可以显着增强LLM预训练数据集。

Title: CORG: Generating Answers from Complex, Interrelated Contexts

• Authors: Hyunji Lee, Franck Dernoncourt, Trung Bui, Seunghyun Yoon
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.00023
• Pdf URL: https://arxiv.org/pdf/2505.00023
• Copy Paste: [[2505.00023]] CORG: Generating Answers from Complex, Interrelated Contexts(https://arxiv.org/abs/2505.00023)
• Keywords: language model
• Abstract: In a real-world corpus, knowledge frequently recurs across documents but often contains inconsistencies due to ambiguous naming, outdated information, or errors, leading to complex interrelationships between contexts. Previous research has shown that language models struggle with these complexities, typically focusing on single factors in isolation. We classify these relationships into four types: distracting, ambiguous, counterfactual, and duplicated. Our analysis reveals that no single approach effectively addresses all these interrelationships simultaneously. Therefore, we introduce Context Organizer (CORG), a framework that organizes multiple contexts into independently processed groups. This design allows the model to efficiently find all relevant answers while ensuring disambiguation. CORG consists of three key components: a graph constructor, a reranker, and an aggregator. Our results demonstrate that CORG balances performance and efficiency effectively, outperforming existing grouping methods and achieving comparable results to more computationally intensive, single-context approaches.
• 摘要：在现实世界中，知识经常跨文档复发，但由于模棱两可的命名，过时的信息或错误，经常包含不一致的情况，从而导致上下文之间复杂的相互关系。先前的研究表明，语言模型与这些复杂性相比，通常集中于孤立的单一因素。我们将这些关系分为四种类型：分心，模棱两可，反事实和重复。我们的分析表明，没有一种方法可以同时有效地解决所有这些相互关系。因此，我们介绍了上下文组织者（CORG），该框架将多个上下文组织到独立处理的组中。该设计使模型可以在确保歧义的同时有效地找到所有相关答案。 CORG由三个关键组成部分组成：图形构造函数，一个reranker和一个聚合器。我们的结果表明，CORG能够有效地平衡性能和效率，超过现有的分组方法，并与更多计算密集型，单语言的方法相当。

Title: Nemotron-Research-Tool-N1: Tool-Using Language Models with Reinforced Reasoning

• Authors: Shaokun Zhang, Yi Dong, Jieyu Zhang, Jan Kautz, Bryan Catanzaro, Andrew Tao, Qingyun Wu, Zhiding Yu, Guilin Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.00024
• Pdf URL: https://arxiv.org/pdf/2505.00024
• Copy Paste: [[2505.00024]] Nemotron-Research-Tool-N1: Tool-Using Language Models with Reinforced Reasoning(https://arxiv.org/abs/2505.00024)
• Keywords: language model, gpt
• Abstract: Enabling large language models with external tools has become a pivotal strategy for extending their functionality beyond text generation tasks. Prior work typically enhances tool-use abilities by either applying supervised fine-tuning (SFT) to enforce tool-call correctness or distilling reasoning traces from stronger models for SFT. However, both approaches fall short, either omitting reasoning entirely or producing imitative reasoning that limits generalization. Inspired by the success of DeepSeek-R1 in eliciting reasoning through rule-based reinforcement learning, we develop the Nemotron-Research-Tool-N1 series of tool-using language models using a similar training paradigm. Instead of restrictively supervising intermediate reasoning traces distilled from stronger models, Nemotron-Research-Tool-N1 is optimized with a binary reward that evaluates only the structural validity and functional correctness of tool invocations. This lightweight supervision allows the model to autonomously internalize reasoning strategies, without the need for annotated reasoning trajectories. Experiments on the BFCL and API-Bank benchmarks show that Nemotron-Research-Tool-N1-7B and Nemotron-Research-Tool-N1-14B, built on Qwen-2.5-7B/14B-Instruct, achieve state-of-the-art results, outperforming GPT-4o on both evaluations.
• 摘要：使用外部工具启用大型语言模型已成为将其功能扩展到文本生成任务之外的关键策略。先前的工作通常通过应用监督的微调（SFT）来增强工具使用能力，以执行工具通话正确性或从更强的SFT模型中提取推理痕迹。但是，这两种方法都缺乏，要么完全省略推理，要么产生限制概括的模仿推理。受到DeepSeek-R1在通过基于规则的强化学习引发推理方面的成功的启发，我们使用类似的培训范式开发了Nemotron-Research-tool-N1系列使用工具语言模型。 Nemotron-Research-tool-N1没有限制性地监督从更强模型中提取的中间推理痕迹，它通过二进制奖励进行了优化，该奖励仅评估工具调用的结构有效性和功能正确性。这种轻巧的监督使该模型可以自主内部化的推理策略，而无需带注释的推理轨迹。 BFCL和API银行基准测试的实验表明，建立在QWEN-2.5-7B/14B教学结构上的Nemotron-Research-tool-N1-7B和Nemotron-Research-tool-N1-14B，实现了最新的结果，对两项评估都胜过GPT-4O。

Title: A Method for the Architecture of a Medical Vertical Large Language Model Based on Deepseek R1

• Authors: Mingda Zhang, Jianglong Qin
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.00025
• Pdf URL: https://arxiv.org/pdf/2505.00025
• Copy Paste: [[2505.00025]] A Method for the Architecture of a Medical Vertical Large Language Model Based on Deepseek R1(https://arxiv.org/abs/2505.00025)
• Keywords: language model, gpt, prompt, chat
• Abstract: In recent years, despite foundation models like DeepSeek-R1 and ChatGPT demonstrating significant capabilities in general tasks, professional knowledge barriers, computational resource requirements, and deployment environment limitations have severely hindered their application in actual medical scenarios. Addressing these challenges, this paper proposes an efficient lightweight medical vertical large language model architecture method, systematically solving the lightweight problem of medical large models from three dimensions: knowledge acquisition, model compression, and computational optimization. At the knowledge acquisition level, a knowledge transfer pipeline is designed from the fine-tuned DeepSeek-R1-Distill-70B teacher model to the DeepSeek-R1-Distill-7B student model, and Low-Rank Adaptation (LoRA) technology is adopted to precisely adjust key attention layers. At the model compression level, compression techniques including 4-bit weight quantization are implemented while preserving the core representation ability for medical reasoning. At the computational optimization level, inference optimization techniques such as Flash Attention acceleration and continuous batching are integrated, and a professional prompt template system is constructed to adapt to different types of medical problems. Experimental results on medical question-answering datasets show that the method proposed in this paper maintains professional accuracy while reducing memory consumption by 64.7\% and inference latency by 12.4\%, providing an effective solution for the application of medical large models in resource-constrained environments such as edge computing devices.
• 摘要：近年来，尽管诸如DeepSeek-R1和Chatgpt之类的基础模型在一般任务，专业知识障碍，计算资源需求和部署环境局限性方面表现出了重要的功能，但仍严重阻碍了其在实际医疗情况下的应用。在应对这些挑战时，本文提出了一种有效的轻巧医学垂直语言模型体系结构方法，系统地解决了从三个维度的大型模型的轻量级问题：知识获取，模型压缩和计算优化。在知识获取层面上，知识转移管道是从微调的DeepSeek-R1-Distill-70B教师模型到DeepSeek-R1-Distill-7B学生模型设计的，采用了低级别适应（LORA）技术，以精确调整关键注意力层。在模型压缩水平上，在保留医疗推理的核心表示能力的同时，实施了包括4位权重量化的压缩技术。在计算优化级别上，集成了推理优化技术，例如闪光注意力加速度和连续批处理，并构建了专业的及时模板系统以适应不同类型的医疗问题。有关避开效率数据集的实验结果表明，本文提出的方法保持专业精度，同时将记忆消耗降低64.7 \％，推理潜伏期降低12.4 \％，为在资源受限的环境（例如边缘计算设备）中应用大型模型提供了有效的解决方案。

Title: Theory of Mind in Large Language Models: Assessment and Enhancement

• Authors: Ruirui Chen, Weifeng Jiang, Chengwei Qin, Cheston Tan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.00026
• Pdf URL: https://arxiv.org/pdf/2505.00026
• Copy Paste: [[2505.00026]] Theory of Mind in Large Language Models: Assessment and Enhancement(https://arxiv.org/abs/2505.00026)
• Keywords: language model, llm
• Abstract: Theory of Mind (ToM)-the ability to infer and reason about others' mental states-is fundamental to human social intelligence. As Large Language Models (LLMs) become increasingly integrated into daily life, it is crucial to assess and enhance their capacity to interpret and respond to human mental states. In this paper, we review LLMs' ToM capabilities by examining both evaluation benchmarks and the strategies designed to improve them. We focus on widely adopted story-based benchmarks and provide an in-depth analysis of methods aimed at enhancing ToM in LLMs. Furthermore, we outline promising future research directions informed by recent benchmarks and state-of-the-art approaches. Our survey serves as a valuable resource for researchers interested in advancing LLMs' ToM capabilities.
• 摘要：心理理论（汤姆） - 推断和推理他人心理状态的能力 - 这是人类社会智力的基础。随着大型语言模型（LLM）越来越多地融入日常生活，因此评估和增强其解释和应对人类精神状态的能力至关重要。在本文中，我们通过研究评估基准和旨在改善它们的策略来回顾LLMS的TOM功能。我们专注于广泛采用的基于故事的基准测试，并对旨在增强LLM中TOM的方法进行了深入的分析。此外，我们概述了以最近的基准和最先进的方法为导致的有希望的未来研究指示。我们的调查是有兴趣推动LLMS TOM功能的研究人员的宝贵资源。

Title: Enhancing Speech-to-Speech Dialogue Modeling with End-to-End Retrieval-Augmented Generation

• Authors: Pengchao Feng, Ziyang Ma, Wenxi Chen, Yao Li, Sheng Wang, Kai Yu, Xie Chen
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2505.00028
• Pdf URL: https://arxiv.org/pdf/2505.00028
• Copy Paste: [[2505.00028]] Enhancing Speech-to-Speech Dialogue Modeling with End-to-End Retrieval-Augmented Generation(https://arxiv.org/abs/2505.00028)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: In recent years, end-to-end speech-to-speech (S2S) dialogue systems have garnered increasing research attention due to their advantages over traditional cascaded systems, including achieving lower latency and more natural integration of nonverbal cues such as emotion and speaker identity. However, these end-to-end systems face key challenges, particularly in incorporating external knowledge, a capability commonly addressed by Retrieval-Augmented Generation (RAG) in text-based large language models (LLMs). The core difficulty lies in the modality gap between input speech and retrieved textual knowledge, which hinders effective integration. To address this issue, we propose a novel end-to-end RAG framework that directly retrieves relevant textual knowledge from speech queries, eliminating the need for intermediate speech-to-text conversion via techniques like ASR. Experimental results demonstrate that our method significantly improves the performance of end-to-end S2S dialogue systems while achieving higher retrieval efficiency. Although the overall performance still lags behind cascaded models, our framework offers a promising direction for enhancing knowledge integration in end-to-end S2S systems. We will release the code and dataset to support reproducibility and promote further research in this area.
• 摘要：近年来，由于它们比传统的级联系统的优势，端到端的语音到语音（S2S）对话系统吸引了研究的关注，包括实现较低的潜伏期和更自然的非语言提示，例如情感和说话者身份。但是，这些端到端系统面临着关键的挑战，尤其是在纳入外部知识时，这是在基于文本的大语言模型（LLMS）中检索 - 演奏生成（RAG）通常解决的能力。核心难度在于输入语音和检索文本知识之间的方式差距，这阻碍了有效的整合。为了解决这个问题，我们提出了一个新颖的端到端抹布框架，该框架直接从语音查询中检索了相关的文本知识，从而消除了通过ASR等技术进行中间语音转换的需求。实验结果表明，我们的方法显着提高了端到端S2S对话系统的性能，同时实现了更高的检索效率。尽管整体性能仍然落后于级联模型，但我们的框架为增强端到端S2S系统中知识集成的方向提供了有希望的方向。我们将发布代码和数据集，以支持可重复性并促进该领域的进一步研究。

Title: Keep the General, Inject the Specific: Structured Dialogue Fine-Tuning for Knowledge Injection without Catastrophic Forgetting

• Authors: Yijie Hong, Xiaofei Yin, Xinzhong Wang, Yi Tu, Ya Guo, Sufeng Duan, Weiqiang Wang, Lingyong Fang, Depeng Wang, Huijia Zhu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.00029
• Pdf URL: https://arxiv.org/pdf/2505.00029
• Copy Paste: [[2505.00029]] Keep the General, Inject the Specific: Structured Dialogue Fine-Tuning for Knowledge Injection without Catastrophic Forgetting(https://arxiv.org/abs/2505.00029)
• Keywords: language model, llm, chain-of-thought
• Abstract: Large Vision Language Models have demonstrated impressive versatile capabilities through extensive multimodal pre-training, but face significant limitations when incorporating specialized knowledge domains beyond their training distribution. These models struggle with a fundamental dilemma: direct adaptation approaches that inject domain-specific knowledge often trigger catastrophic forgetting of foundational visual-linguistic abilities. We introduce Structured Dialogue Fine-Tuning (SDFT), an effective approach that effectively injects domain-specific knowledge while minimizing catastrophic forgetting. Drawing inspiration from supervised fine-tuning in LLMs and subject-driven personalization in text-to-image diffusion models, our method employs a three-phase dialogue structure: Foundation Preservation reinforces pre-trained visual-linguistic alignment through caption tasks; Contrastive Disambiguation introduces carefully designed counterfactual examples to maintain semantic boundaries; and Knowledge Specialization embeds specialized information through chain-of-thought reasoning. Experimental results across multiple domains confirm SDFT's effectiveness in balancing specialized knowledge acquisition with general capability retention. Our key contributions include a data-centric dialogue template that balances foundational alignment with targeted knowledge integration, a weighted multi-turn supervision framework, and comprehensive evaluation across diverse knowledge types.
• 摘要：大型视觉语言模型通过广泛的多模式预训练表现出了令人印象深刻的多功能功能，但是在纳入超出其训练分配的专业知识领域时，面临着重大限制。这些模型在根本的困境中挣扎：注入特定于领域的知识的直接适应方法通常会触发灾难性的遗忘基础视觉语言能力。我们引入了结构化对话微调（SDFT），这是一种有效的方法，可以有效地注入特定于领域的知识，同时最大程度地减少灾难性遗忘。我们的方法从LLM中的监督微调和主题驱动的个性化模型中汲取灵感，我们的方法采用了三相对话结构：基础保护通过字幕任务加强了预先训练的视觉语言对齐；对比歧义引入了精心设计的反事实示例，以保持语义界限；知识专业化通过思想链推理嵌入了专业信息。多个领域的实验结果证实了SDFT在平衡专业知识获取与一般能力保留方面的有效性。我们的主要贡献包括以数据为中心的对话模板，该模板可以平衡基础一致性与有针对性的知识集成，加权多转变监督框架以及跨不同知识类型的全面评估。

Title: Can Language Models Represent the Past without Anachronism?

• Authors: Ted Underwood, Laura K. Nelson, Matthew Wilkens
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.00030
• Pdf URL: https://arxiv.org/pdf/2505.00030
• Copy Paste: [[2505.00030]] Can Language Models Represent the Past without Anachronism?(https://arxiv.org/abs/2505.00030)
• Keywords: language model, prompt
• Abstract: Before researchers can use language models to simulate the past, they need to understand the risk of anachronism. We find that prompting a contemporary model with examples of period prose does not produce output consistent with period style. Fine-tuning produces results that are stylistically convincing enough to fool an automated judge, but human evaluators can still distinguish fine-tuned model outputs from authentic historical text. We tentatively conclude that pretraining on period prose may be required in order to reliably simulate historical perspectives for social research.
• 摘要：在研究人员可以使用语言模型模拟过去之前，他们需要了解过时的风险。我们发现，促使一个具有时期散文示例的当代模型不会产生与时期样式一致的产出。微调产生的结果在风格上令人信服地愚弄了自动化的法官，但是人类评估人员仍然可以将微调模型输出与真实的历史文本区分开。我们暂时得出结论，为了可靠地模拟社会研究的历史观点，可能需要对散文进行预处理。

Title: Learning to Plan Before Answering: Self-Teaching LLMs to Learn Abstract Plans for Problem Solving

• Authors: Jin Zhang, Flood Sung, Zhilin Yang, Yang Gao, Chongjie Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.00031
• Pdf URL: https://arxiv.org/pdf/2505.00031
• Copy Paste: [[2505.00031]] Learning to Plan Before Answering: Self-Teaching LLMs to Learn Abstract Plans for Problem Solving(https://arxiv.org/abs/2505.00031)
• Keywords: language model, llm
• Abstract: In the field of large language model (LLM) post-training, the effectiveness of utilizing synthetic data generated by the LLM itself has been well-presented. However, a key question remains unaddressed: what essential information should such self-generated data encapsulate? Existing approaches only produce step-by-step problem solutions, and fail to capture the abstract meta-knowledge necessary for generalization across similar problems. Drawing insights from cognitive science, where humans employ high-level abstraction to simplify complex problems before delving into specifics, we introduce a novel self-training algorithm: LEarning to Plan before Answering (LEPA). LEPA trains the LLM to formulate anticipatory plans, which serve as abstract meta-knowledge for problem-solving, before engaging with the intricacies of problems. This approach not only outlines the solution generation path but also shields the LLM from the distraction of irrelevant details. During data generation, LEPA first crafts an anticipatory plan based on the problem, and then generates a solution that aligns with both the plan and the problem. LEPA refines the plan through self-reflection, aiming to acquire plans that are instrumental in yielding correct solutions. During model optimization, the LLM is trained to predict both the refined plans and the corresponding solutions. By efficiently extracting and utilizing the anticipatory plans, LEPA demonstrates remarkable superiority over conventional algorithms on various challenging natural language reasoning benchmarks.
• 摘要：在训练后大语言模型（LLM）领域中，利用LLM本身生成的合成数据的有效性已经很好。但是，一个关键问题仍然没有解决：这种自我生成的数据应该封装哪些基本信息？现有方法仅产生分步问题解决方案，并且无法捕获跨类似问题的概括所需的抽象元知识。从认知科学中汲取见解，在该科学中，人类采用高级抽象来简化复杂的问题，然后再研究细节，我们引入了一种新颖的自我训练算法：在回答之前学习计划（LEPA）。 LEPA训练LLM制定预期计划，这些计划是解决问题的抽象元知识，然后再与问题的复杂性接触。这种方法不仅概述了解决方案的生成路径，还可以保护LLM免受无关紧要的细节的注意。在数据生成期间，LEPA首先根据问题制定了预期计划，然后生成了与计划和问题保持一致的解决方案。 LEPA通过自我反省来完善计划，旨在获取有助于产生正确解决方案的计划。在模型优化期间，训练LLM可以预测精制计划和相应的解决方案。通过有效提取和利用预期计划，LEPA在各种具有挑战性的自然语言推理基准上表现出了与传统算法相比的显着优势。

Title: MDD-LLM: Towards Accuracy Large Language Models for Major Depressive Disorder Diagnosis

• Authors: Yuyang Sha, Hongxin Pan, Wei Xu, Weiyu Meng, Gang Luo, Xinyu Du, Xiaobing Zhai, Henry H. Y. Tong, Caijuan Shi, Kefeng Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.00032
• Pdf URL: https://arxiv.org/pdf/2505.00032
• Copy Paste: [[2505.00032]] MDD-LLM: Towards Accuracy Large Language Models for Major Depressive Disorder Diagnosis(https://arxiv.org/abs/2505.00032)
• Keywords: language model, llm
• Abstract: Major depressive disorder (MDD) impacts more than 300 million people worldwide, highlighting a significant public health issue. However, the uneven distribution of medical resources and the complexity of diagnostic methods have resulted in inadequate attention to this disorder in numerous countries and regions. This paper introduces a high-performance MDD diagnosis tool named MDD-LLM, an AI-driven framework that utilizes fine-tuned large language models (LLMs) and extensive real-world samples to tackle challenges in MDD diagnosis. Therefore, we select 274,348 individual information from the UK Biobank cohort to train and evaluate the proposed method. Specifically, we select 274,348 individual records from the UK Biobank cohort and design a tabular data transformation method to create a large corpus for training and evaluating the proposed approach. To illustrate the advantages of MDD-LLM, we perform comprehensive experiments and provide several comparative analyses against existing model-based solutions across multiple evaluation metrics. Experimental results show that MDD-LLM (70B) achieves an accuracy of 0.8378 and an AUC of 0.8919 (95% CI: 0.8799 - 0.9040), significantly outperforming existing machine learning and deep learning frameworks for MDD diagnosis. Given the limited exploration of LLMs in MDD diagnosis, we examine numerous factors that may influence the performance of our proposed method, such as tabular data transformation techniques and different fine-tuning strategies.
• 摘要：重度抑郁症（MDD）影响了全球超过3亿人，突出了一个重大的公共卫生问题。但是，医疗资源的分配不均匀以及诊断方法的复杂性导致许多国家和地区对这种疾病的关注不足。本文介绍了一个名为MDD-LLM的高性能MDD诊断工具，该工具是AI驱动的框架，该框架利用微调的大语言模型（LLMS）和广泛的现实样本来应对MDD诊断中的挑战。因此，我们从英国生物银行队列中选择274,348个个人信息来培训和评估所提出的方法。具体而言，我们从英国生物银行队列中选择274,348个单独的记录，并设计了一种表格数据转换方法，以创建用于培训和评估拟议方法的大型语料库。为了说明MDD-LLM的优势，我们执行了全面的实验，并对在多个评估指标中对现有模型解决方案进行了几项比较分析。实验结果表明，MDD -LLM（70B）的精度为0.8378，AUC为0.8919（95％CI：0.8799-0.9040），对MDD诊断的现有机器学习和深度学习框架的表现显着超过了。鉴于对MDD诊断中LLM的探索有限，我们研究了可能影响我们提出方法的性能的许多因素，例如表格数据转换技术和不同的微调策略。

Title: From Attention to Atoms: Spectral Dictionary Learning for Fast, Interpretable Language Models

• Authors: Andrew Kiruluta
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.00033
• Pdf URL: https://arxiv.org/pdf/2505.00033
• Copy Paste: [[2505.00033]] From Attention to Atoms: Spectral Dictionary Learning for Fast, Interpretable Language Models(https://arxiv.org/abs/2505.00033)
• Keywords: language model
• Abstract: We propose a novel spectral generative modeling framework for natural language processing that jointly learns a global time varying Fourier dictionary and per token mixing coefficients, replacing the ubiquitous self attention mechanism in transformer architectures. By enforcing reconstruction losses in both the time domain (embedding reconstruction) and the frequency domain (via Short Time Fourier Transform magnitude matching) alongside a standard language modeling objective, and fitting a Gaussian Mixture Model (GMM) prior over the learned mixing vectors, our approach achieves competitive perplexity and generation quality on standard benchmarks such as WikiText2 and Penn Treebank. In contrast to the quadratic computation complexity of self attention, our method operates with linear complexity, delivering substantial efficiency gains. We demonstrate that spectral dictionary models can achieve competitive performance compared to transformer baselines while significantly reducing inference latency and memory footprint, offering a compelling alternative for scalable language modeling.
• 摘要：我们为自然语言处理提出了一个新型的光谱生成模型框架，该框架共同学习了一个全球时间变化的傅立叶词典和每个令牌混合系数，从而取代了变压器体系结构中普遍存在的自我注意力。通过在时间域（嵌入重建）和频域（通过短时间傅立叶变换幅度匹配）中实施重建损失，并在标准的语言建模目标旁边，并在学习的混合矢量上拟合了高斯混合模型（GMM），我们的方法可以在标准的Benchnarks上获得竞争性的困扰和生成质量，例如竞争性的质量和penn penn teeknext2和penn swikitexf。与自我关注的二次计算复杂性相反，我们的方法具有线性复杂性，从而带来了可观的效率提高。我们证明，与变压器基线相比，光谱词典模型可以实现竞争性能，同时大大降低了推断潜伏期和内存足迹，为可扩展语言建模提供了令人信服的替代方案。

Title: Improving Phishing Email Detection Performance of Small Large Language Models

• Authors: Zijie Lin, Zikang Liu, Hanbo Fan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.00034
• Pdf URL: https://arxiv.org/pdf/2505.00034
• Copy Paste: [[2505.00034]] Improving Phishing Email Detection Performance of Small Large Language Models(https://arxiv.org/abs/2505.00034)
• Keywords: language model, llm, prompt
• Abstract: Large language models(LLMs) have demonstrated remarkable performance on many natural language processing(NLP) tasks and have been employed in phishing email detection research. However, in current studies, well-performing LLMs typically contain billions or even tens of billions of parameters, requiring enormous computational resources. To reduce computational costs, we investigated the effectiveness of small-parameter LLMs for phishing email detection. These LLMs have around 3 billion parameters and can run on consumer-grade GPUs. However, small LLMs often perform poorly in phishing email detection task. To address these issues, we designed a set of methods including Prompt Engineering, Explanation Augmented Fine-tuning, and Model Ensemble to improve phishing email detection capabilities of small LLMs. We validated the effectiveness of our approach through experiments, significantly improving accuracy on the SpamAssassin dataset from around 0.5 for baseline models like Qwen2.5-1.5B-Instruct to 0.976.
• 摘要：大型语言模型（LLMS）在许多自然语言处理（NLP）任务上表现出了出色的表现，并且已用于网络钓鱼电子邮件检测研究。但是，在当前的研究中，表现出色的LLM通常包含数十亿甚至数亿个参数，需要巨大的计算资源。为了降低计算成本，我们研究了小参数LLMS在网络钓鱼电子邮件检测中的有效性。这些LLM约有30亿个参数，可以在消费级GPU上运行。但是，小型LLM在网络钓鱼电子邮件检测任务中的表现通常很差。为了解决这些问题，我们设计了一套方法，包括及时工程，解释增强微调和模型集合，以提高小型LLM的网络钓鱼电子邮件检测功能。我们通过实验验证了方法的有效性，对于QWEN2.5-1.5B-Instruction等基线模型，Spamassassin数据集的准确性从0.5左右提高到了0.976。

Title: A Framework to Assess the Persuasion Risks Large Language Model Chatbots Pose to Democratic Societies

• Authors: Zhongren Chen, Joshua Kalla, Quan Le, Shinpei Nakamura-Sakai, Jasjeet Sekhon, Ruixiao Wang
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2505.00036
• Pdf URL: https://arxiv.org/pdf/2505.00036
• Copy Paste: [[2505.00036]] A Framework to Assess the Persuasion Risks Large Language Model Chatbots Pose to Democratic Societies(https://arxiv.org/abs/2505.00036)
• Keywords: language model, llm, chat
• Abstract: In recent years, significant concern has emerged regarding the potential threat that Large Language Models (LLMs) pose to democratic societies through their persuasive capabilities. We expand upon existing research by conducting two survey experiments and a real-world simulation exercise to determine whether it is more cost effective to persuade a large number of voters using LLM chatbots compared to standard political campaign practice, taking into account both the "receive" and "accept" steps in the persuasion process (Zaller 1992). These experiments improve upon previous work by assessing extended interactions between humans and LLMs (instead of using single-shot interactions) and by assessing both short- and long-run persuasive effects (rather than simply asking users to rate the persuasiveness of LLM-produced content). In two survey experiments (N = 10,417) across three distinct political domains, we find that while LLMs are about as persuasive as actual campaign ads once voters are exposed to them, political persuasion in the real-world depends on both exposure to a persuasive message and its impact conditional on exposure. Through simulations based on real-world parameters, we estimate that LLM-based persuasion costs between \$48-\$74 per persuaded voter compared to \$100 for traditional campaign methods, when accounting for the costs of exposure. However, it is currently much easier to scale traditional campaign persuasion methods than LLM-based persuasion. While LLMs do not currently appear to have substantially greater potential for large-scale political persuasion than existing non-LLM methods, this may change as LLM capabilities continue to improve and it becomes easier to scalably encourage exposure to persuasive LLMs.
• 摘要：近年来，人们对大型语言模型（LLMS）通过其说服力对民主社会构成的潜在威胁产生了重大关注。我们通过进行两个调查实验和一个现实世界的模拟练习来扩展现有研究，以确定与标准政治竞选实践相比，使用LLM聊天机器人说服大量选民的成本有效，同时考虑到说服力过程中的“接收”和“接受”步骤（Zaller 1992）。这些实验通过评估人类与LLM之间的扩展相互作用（而不是使用单一相互作用），并评估短期和长期的说服力效果（而不是简单地要求用户评估LLM生产的内容的说服力），从而改善了以前的工作。在三个不同的政治领域的两个调查实验（n = 10,417）中，我们发现，尽管LLMS与实际竞选广告相当有说服力，但一旦选民暴露了他们，但现实世界中的政治说服力取决于对有说服力的信息及其对暴露的影响。通过基于现实世界参数的模拟，我们估计基于LLM的说服力在\ $ 48- \ $ $ 74之间，而传统竞选方法的费用为\ $ $ 100，这是在计算曝光成本时。但是，目前比基于LLM的说服更容易扩展传统的运动说服方法。尽管LLM当前似乎没有比现有的非LLM方法具有更大的大规模政治说服力潜力，但随着LLM能力继续提高，这种情况可能会发生变化，并且更容易地鼓励暴露于有说服力的LLMS。

Title: HyPerAlign: Hypotheses-driven Personalized Alignment

• Authors: Cristina Garbacea, Chenhao Tan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.00038
• Pdf URL: https://arxiv.org/pdf/2505.00038
• Copy Paste: [[2505.00038]] HyPerAlign: Hypotheses-driven Personalized Alignment(https://arxiv.org/abs/2505.00038)
• Keywords: language model, llm, prompt
• Abstract: Alignment algorithms are widely used to align large language models (LLMs) to human users based on preference annotations that reflect their intended real-world use cases. Typically these (often divergent) preferences are aggregated over a diverse set of users, resulting in fine-tuned models that are aligned to the ``average-user'' preference. Nevertheless, current models are used by individual users in very specific contexts and situations, emphasizing the need for user-dependent preference control. In this work we address the problem of personalizing LLM outputs to their users, aiming to generate customized responses tailored to individual users, instead of generic outputs that emulate the collective voices of diverse populations. We propose a novel interpretable and sample-efficient hypotheses-driven personalization approach (HyPerAlign) where given few-shot examples written by a particular user, we first infer hypotheses about their communication strategies, personality and writing style, then prompt LLM models with these hypotheses and user specific attributes to generate customized outputs. We conduct experiments on two different personalization tasks, authorship attribution and deliberative alignment, with datasets from diverse domains (news articles, blog posts, emails, jailbreaking benchmarks), and demonstrate the superiority of hypotheses-driven personalization approach when compared to preference-based fine-tuning methods. For deliberative alignment, the helpfulness of LLM models is improved by up to $70\%$ on average. For authorship attribution, results indicate consistently high win-rates (commonly $>90\%$) against state-of-the-art preference fine-tuning approaches for LLM personalization across diverse user profiles and LLM models. Overall, our approach represents an interpretable and sample-efficient strategy for the personalization of LLM models to individual users.
• 摘要：一致性算法广泛用于基于反映其预期的现实世界用例的偏好注释，将大型语言模型（LLMS）与人类用户相结合。通常，这些（通常是发散的）偏好是在各种用户集合中汇总的，从而产生了与“普通用户”偏好相符的微调模型。然而，在非常具体的上下文和情况下，各个用户都使用了当前的模型，从而强调了对用户依赖的偏好控制的需求。在这项工作中，我们解决了向用户个性化LLM输出的问题，旨在生成针对单个用户的自定义响应，而不是模仿不同人群的集体声音的通用输出。我们提出了一种新颖的可解释和样本的假设驱动的个性化方法（HyperAlign），其中给定了特定用户写的很少的示例，我们首先推断出有关其交流策略，个性和写作风格的假设，然后提示具有这些假设和用户特定属性的LLM模型以生成定制的输出。我们对两项不同的个性化任务，作者归因和审议统一性进行了实验，并通过不同的域名（新闻文章，博客文章，电子邮件，越狱基准）进行了数据集，并且与基于偏好基于偏好的微调方法相比，进行了假设驱动的个性化方法的优势。对于审议的一致性，LLM型号的有益性平均提高了70美元。对于作者归因，结果表明，针对在不同的用户配置文件和LLM模型的LLM个性化的最新首选微调方法相对于最先进的首选项微调方法，结果表明始终高的获胜率（通常为$> 90 \％$ $）。总体而言，我们的方法代表了将LLM模型个性化对个体用户个性化的可解释和样本效率的策略。

Title: Graph RAG for Legal Norms: A Hierarchical and Temporal Approach

• Authors: Hudson de Martim
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2505.00039
• Pdf URL: https://arxiv.org/pdf/2505.00039
• Copy Paste: [[2505.00039]] Graph RAG for Legal Norms: A Hierarchical and Temporal Approach(https://arxiv.org/abs/2505.00039)
• Keywords: retrieval augmented generation
• Abstract: This article proposes an adaptation of Graph Retrieval Augmented Generation (Graph RAG) specifically designed for the analysis and comprehension of legal norms, which are characterized by their predefined hierarchical structure, extensive network of internal and external references and multiple temporal versions. By combining structured knowledge graphs with contextually enriched text segments, Graph RAG offers a promising solution to address the inherent complexity and vast volume of legal data. The integration of hierarchical structure and temporal evolution into knowledge graphs - along with the concept of comprehensive Text Units - facilitates the construction of richer, interconnected representations of legal knowledge. Through a detailed analysis of Graph RAG and its application to legal norm datasets, this article aims to significantly advance the field of Artificial Intelligence applied to Law, creating opportunities for more effective systems in legal research, legislative analysis, and decision support.
• 摘要：本文提出了专门设计用于分析和理解法律规范的图形检索增强生成（图形）（图形抹布）的改编，这些规范的特征是其预定义的层次结构，内部和外部引用的广泛网络以及多个时间版本。通过将结构化知识图与上下文丰富的文本段相结合，图形抹布提供了一种有希望的解决方案，以解决固有的复杂性和大量法律数据。将层次结构和时间演变的整合到知识图中，以及综合文本单元的概念，有助于构建法律知识的更丰富，相互联系的表示。通过对图形抹布的详细分析及其在法律规范数据集中的应用，本文旨在显着推进适用于法律的人工智能领域，从而为法律研究，立法分析和决策支持创造更有效的系统的机会。

Title: Base Models Beat Aligned Models at Randomness and Creativity

• Authors: Peter West, Christopher Potts
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.00047
• Pdf URL: https://arxiv.org/pdf/2505.00047
• Copy Paste: [[2505.00047]] Base Models Beat Aligned Models at Randomness and Creativity(https://arxiv.org/abs/2505.00047)
• Keywords: language model, llm
• Abstract: Alignment has quickly become a default ingredient in LLM development, with techniques such as reinforcement learning from human feedback making models act safely, follow instructions, and perform ever-better on complex tasks. While these techniques are certainly useful, we propose that they should not be universally applied and demonstrate a range of tasks on which base language models consistently outperform their popular aligned forms. Particularly, we study tasks that require unpredictable outputs, such as random number generation, mixed strategy games (rock-paper-scissors and hide-and-seek), and creative writing. In each case, aligned models tend towards narrow behaviors that result in distinct disadvantages, for instance, preferring to generate "7" over other uniformly random numbers, becoming almost fully predictable in some game states, or prioritizing pleasant writing over creative originality. Across models tested, better performance on common benchmarks tends to correlate with worse performance on our tasks, suggesting an effective trade-off in the required capabilities.
• 摘要：一致性已迅速成为LLM开发中的默认要素，诸如从人类反馈中的增强学习诸如安全的模型，遵循说明并在复杂任务上执行的技术。尽管这些技术肯定很有用，但我们建议不应普遍应用它们，并证明了一系列任务，基本语言模型始终优于其流行的对齐形式。特别是，我们研究需要不可预测的输出的任务，例如随机数字，混合策略游戏（岩纸剪辑和捉迷藏）以及创意写作。在每种情况下，对齐的模型倾向于狭窄的行为，例如，偏僻的劣势，例如，宁愿比其他统一的随机数生成“ 7”，在某些游戏状态下几乎可以完全可以预测，或者优先考虑愉快的写作而不是创造性独创性。在经过测试的模型中，公共基准的更好性能倾向于与我们的任务的性能较差相关，这表明所需能力的有效权衡。

Title: A Report on the llms evaluating the high school questions

• Authors: Zhu Jiawei, Chen Wei
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.00057
• Pdf URL: https://arxiv.org/pdf/2505.00057
• Copy Paste: [[2505.00057]] A Report on the llms evaluating the high school questions(https://arxiv.org/abs/2505.00057)
• Keywords: language model, llm
• Abstract: This report aims to evaluate the performance of large language models (LLMs) in solving high school science questions and to explore their potential applications in the educational field. With the rapid development of LLMs in the field of natural language processing, their application in education has attracted widespread attention. This study selected mathematics exam questions from the college entrance examinations (2019-2023) as evaluation data and utilized at least eight LLM APIs to provide answers. A comprehensive assessment was conducted based on metrics such as accuracy, response time, logical reasoning, and creativity. Through an in-depth analysis of the evaluation results, this report reveals the strengths and weaknesses of LLMs in handling high school science questions and discusses their implications for educational practice. The findings indicate that although LLMs perform excellently in certain aspects, there is still room for improvement in logical reasoning and creative problem-solving. This report provides an empirical foundation for further research and application of LLMs in the educational field and offers suggestions for improvement.
• 摘要：该报告旨在评估大语模型（LLM）在解决高中科学问题并探索其在教育领域的潜在应用时的表现。随着LLM在自然语言处理领域的迅速发展，它们在教育中的应用引起了广泛的关注。这项研究从大学入学考试（2019-2023）中选择了数学考试问题作为评估数据，并使用了至少八个LLM API来提供答案。根据准确性，响应时间，逻辑推理和创造力等指标进行了全面评估。通过对评估结果的深入分析，本报告揭示了LLM在处理高中科学问题方面的优势和缺点，并讨论了它们对教育实践的影响。研究结果表明，尽管LLM在某些方面表现出色，但仍有改善逻辑推理和创造性问题的余地。该报告为LLM在教育领域的进一步研究和应用提供了经验基础，并提供了改进建议。

Title: BERSting at the Screams: A Benchmark for Distanced, Emotional and Shouted Speech Recognition

• Authors: Paige Tuttösí, Mantaj Dhillon, Luna Sang, Shane Eastwood, Poorvi Bhatia, Quang Minh Dinh, Avni Kapoor, Yewon Jin, Angelica Lim
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2505.00059
• Pdf URL: https://arxiv.org/pdf/2505.00059
• Copy Paste: [[2505.00059]] BERSting at the Screams: A Benchmark for Distanced, Emotional and Shouted Speech Recognition(https://arxiv.org/abs/2505.00059)
• Keywords: prompt
• Abstract: Some speech recognition tasks, such as automatic speech recognition (ASR), are approaching or have reached human performance in many reported metrics. Yet, they continue to struggle in complex, real-world, situations, such as with distanced speech. Previous challenges have released datasets to address the issue of distanced ASR, however, the focus remains primarily on distance, specifically relying on multi-microphone array systems. Here we present the B(asic) E(motion) R(andom phrase) S(hou)t(s) (BERSt) dataset. The dataset contains almost 4 hours of English speech from 98 actors with varying regional and non-native accents. The data was collected on smartphones in the actors homes and therefore includes at least 98 different acoustic environments. The data also includes 7 different emotion prompts and both shouted and spoken utterances. The smartphones were places in 19 different positions, including obstructions and being in a different room than the actor. This data is publicly available for use and can be used to evaluate a variety of speech recognition tasks, including: ASR, shout detection, and speech emotion recognition (SER). We provide initial benchmarks for ASR and SER tasks, and find that ASR degrades both with an increase in distance and shout level and shows varied performance depending on the intended emotion. Our results show that the BERSt dataset is challenging for both ASR and SER tasks and continued work is needed to improve the robustness of such systems for more accurate real-world use.
• 摘要：在许多报告的指标中，一些语音识别任务（例如自动语音识别（ASR））正在接近或达到人类绩效。然而，他们继续在复杂的，现实世界中的情况下挣扎，例如言语遥远的言论。以前的挑战已发布了数据集以解决距离ASR的问题，但是，重点主要保留在距离上，特别依赖于多微粒阵列系统。在这里，我们介绍B（ASIC）E（MOTION）R（ANDOM短语）S（HOU）T（S）（BERST）数据集。该数据集包含来自98个区域和非本地口音不同的98个演员的近4个小时的英语演讲。数据是在Actors Homes中的智能手机上收集的，因此包括至少98种不同的声学环境。数据还包括7个不同的情感提示，并大喊和口头说话。智能手机是19个不同位置的地方，包括障碍物和与演员不同的房间。该数据可公开使用，可用于评估各种语音识别任务，包括：ASR，喊叫检测和语音情感识别（SER）。我们为ASR和SER任务提供初始的基准，并发现ASR均随距离和喊叫的水平增加而降低，并且根据预期的情绪显示出多样化的性能。我们的结果表明，BERST数据集在ASR和SER任务中都具有挑战性，并且需要继续工作，以提高此类系统的鲁棒性，以进行更准确的现实使用。

Title: Fact-Consistency Evaluation of Text-to-SQL Generation for Business Intelligence Using Exaone 3.5

• Authors: Jeho Choi
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2505.00060
• Pdf URL: https://arxiv.org/pdf/2505.00060
• Copy Paste: [[2505.00060]] Fact-Consistency Evaluation of Text-to-SQL Generation for Business Intelligence Using Exaone 3.5(https://arxiv.org/abs/2505.00060)
• Keywords: language model, llm, hallucination
• Abstract: Large Language Models (LLMs) have shown promise in enabling natural language interfaces for structured data querying through text-to-SQL generation. However, their application in real-world Business Intelligence (BI) contexts remains limited due to semantic hallucinations, structural errors, and a lack of domain-specific evaluation frameworks. In this study, we propose a Fact-Consistency Evaluation Framework for assessing the semantic accuracy of LLM-generated SQL outputs using Exaone 3.5--an instruction-tuned, bilingual LLM optimized for enterprise tasks. We construct a domain-specific benchmark comprising 219 natural language business questions across five SQL complexity levels, derived from actual sales data in LG Electronics' internal BigQuery environment. Each question is paired with a gold-standard SQL query and a validated ground-truth answer. We evaluate model performance using answer accuracy, execution success rate, semantic error rate, and non-response rate. Experimental results show that while Exaone 3.5 performs well on simple aggregation tasks (93% accuracy in L1), it exhibits substantial degradation in arithmetic reasoning (4% accuracy in H1) and grouped ranking tasks (31% in H4), with semantic errors and non-responses concentrated in complex cases. Qualitative error analysis further identifies common failure types such as misapplied arithmetic logic, incomplete filtering, and incorrect grouping operations. Our findings highlight the current limitations of LLMs in business-critical environments and underscore the need for fact-consistency validation layers and hybrid reasoning approaches. This work contributes a reproducible benchmark and evaluation methodology for advancing reliable natural language interfaces to structured enterprise data systems.
• 摘要：大型语言模型（LLM）在实现自然语言界面方面显示了通过文本到SQL生成来查询的自然语言界面。但是，由于语义幻觉，结构错误以及缺乏特定领域的评估框架，它们在现实世界中的商业智能（BI）环境中的应用仍限制。在这项研究中，我们提出了一个事实一致性评估框架，用于评估LLM生成的SQL输出的语义准确性，使用Exaone 3.5（用于企业任务优化的指令调整的双语LLM）。我们构建了一个特定于域的基准测试，其中包括219个自然语言业务问题，跨越了五个SQL复杂性水平，这些问题来自LG Electronics内部BigQuery环境中的实际销售数据。每个问题都与金标准的SQL查询和经过验证的地面真实答案配对。我们使用答案准确性，执行成功率，语义错误率和无响应率来评估模型性能。实验结果表明，尽管Exaone 3.5在简单的聚合任务上表现良好（L1的精度为93％），但它在算术推理（H1中的精度为4％）和分组排名任务（H4中的31％）中表现出很大的降解（H4中的31％），具有语义错误和非响应率，并且在复杂的情况下集中了。定性误差分析进一步确定了常见的故障类型，例如错误应用算术逻辑，不完整的过滤和不正确的分组操作。我们的发现突出了LLM在关键性环境中的当前局限性，并强调了对事实一致性验证层和混合推理方法的需求。这项工作为将可靠的自然语言界面推进到结构化企业数据系统的基准和评估方法提供了可再现的基准和评估方法。

Title: Enhancing Security and Strengthening Defenses in Automated Short-Answer Grading Systems

• Authors: Sahar Yarmohammadtoosky, Yiyun Zhou, Victoria Yaneva, Peter Baldwin, Saed Rezayi, Brian Clauser, Polina Harikeo
• Subjects: cs.CL, cs.CR
• Abstract URL: https://arxiv.org/abs/2505.00061
• Pdf URL: https://arxiv.org/pdf/2505.00061
• Copy Paste: [[2505.00061]] Enhancing Security and Strengthening Defenses in Automated Short-Answer Grading Systems(https://arxiv.org/abs/2505.00061)
• Keywords: language model, gpt, prompt
• Abstract: This study examines vulnerabilities in transformer-based automated short-answer grading systems used in medical education, with a focus on how these systems can be manipulated through adversarial gaming strategies. Our research identifies three main types of gaming strategies that exploit the system's weaknesses, potentially leading to false positives. To counteract these vulnerabilities, we implement several adversarial training methods designed to enhance the systems' robustness. Our results indicate that these methods significantly reduce the susceptibility of grading systems to such manipulations, especially when combined with ensemble techniques like majority voting and ridge regression, which further improve the system's defense against sophisticated adversarial inputs. Additionally, employing large language models such as GPT-4 with varied prompting techniques has shown promise in recognizing and scoring gaming strategies effectively. The findings underscore the importance of continuous improvements in AI-driven educational tools to ensure their reliability and fairness in high-stakes settings.
• 摘要：这项研究研究了在医学教育中使用的基于变压器的自动化短距离分级系统中的脆弱性，重点是如何通过对抗性游戏策略来操纵这些系统。我们的研究确定了利用系统弱点的三种主要类型的游戏策略，可能导致误报。为了抵消这些漏洞，我们实施了几种旨在增强系统鲁棒性的对抗训练方法。我们的结果表明，这些方法大大降低了分级系统对此类操作的敏感性，尤其是当与多数投票和岭回归（Ridge Recression）等集合技术结合使用时，这进一步改善了该系统对精致的对抗性投入的防御。此外，采用大型语言模型（例如GPT-4）具有多样的提示技术，在有效地识别和评分游戏策略方面有希望。这些发现强调了AI驱动的教育工具不断改进的重要性，以确保其在高风险环境中的可靠性和公平性。

Title: GDI-Bench: A Benchmark for General Document Intelligence with Vision and Reasoning Decoupling

• Authors: Siqi Li, Yufan Shen, Xiangnan Chen, Jiayi Chen, Hengwei Ju, Haodong Duan, Song Mao, Hongbin Zhou, Bo Zhang, Pinlong Cai, Licheng Wen, Botian Shi, Yong Liu, Xinyu Cai, Yu Qiao
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2505.00063
• Pdf URL: https://arxiv.org/pdf/2505.00063
• Copy Paste: [[2505.00063]] GDI-Bench: A Benchmark for General Document Intelligence with Vision and Reasoning Decoupling(https://arxiv.org/abs/2505.00063)
• Keywords: language model, gpt, llm
• Abstract: The rapid advancement of multimodal large language models (MLLMs) has profoundly impacted the document domain, creating a wide array of application scenarios. This progress highlights the need for a comprehensive benchmark to evaluate these models' capabilities across various document-specific tasks. However, existing benchmarks often fail to locate specific model weaknesses or guide systematic improvements. To bridge this gap, we introduce a General Document Intelligence Benchmark (GDI-Bench), featuring 1.9k images across 9 key scenarios and 19 document-specific tasks. By decoupling visual complexity and reasoning complexity, the GDI-Bench structures graded tasks that allow performance assessment by difficulty, aiding in model weakness identification and optimization guidance. We evaluate the GDI-Bench on various open-source and closed-source models, conducting decoupled analyses in the visual and reasoning domains. For instance, the GPT-4o model excels in reasoning tasks but exhibits limitations in visual capabilities. To address the diverse tasks and domains in the GDI-Bench, we propose a GDI Model that mitigates the issue of catastrophic forgetting during the supervised fine-tuning (SFT) process through a intelligence-preserving training strategy. Our model achieves state-of-the-art performance on previous benchmarks and the GDI-Bench. Both our benchmark and model will be open source.
• 摘要：多模式大语言模型（MLLM）的快速发展对文档域产生了深远的影响，从而创造了广泛的应用程序场景。这一进展突出了需要全面的基准测试来评估这些模型在各种特定文档特定任务中的功能。但是，现有的基准通常无法找到特定的模型弱点或指导系统的改进。为了弥合这一差距，我们引入了通用文档智能基准（GDI-Bench），该基准在9个关键方案和19个特定于文档的任务中具有1.9k图像。通过解开视觉复杂性和推理的复杂性，GDI基础结构的分级任务可以通过难度进行绩效评估，有助于模型弱点识别和优化指导。我们在各种开源和闭合源模型上评估了GDI基础台，并在视觉和推理域中进行了解耦分析。例如，GPT-4O模型在推理任务中表现出色，但在视觉功能中表现出局限性。为了解决GDI板凳中的各种任务和领域，我们提出了一个GDI模型，该模型通过具有情报的培训策略来减轻监督微调过程（SFT）过程中灾难性遗忘的问题。我们的模型在先前的基准和GDI板凳上实现了最先进的性能。我们的基准和型号都是开源的。

Title: ConSens: Assessing context grounding in open-book question answering

• Authors: Ivan Vankov, Matyo Ivanov, Adriana Correia, Victor Botev
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2505.00065
• Pdf URL: https://arxiv.org/pdf/2505.00065
• Copy Paste: [[2505.00065]] ConSens: Assessing context grounding in open-book question answering(https://arxiv.org/abs/2505.00065)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have demonstrated considerable success in open-book question answering (QA), where the task requires generating answers grounded in a provided external context. A critical challenge in open-book QA is to ensure that model responses are based on the provided context rather than its parametric knowledge, which can be outdated, incomplete, or incorrect. Existing evaluation methods, primarily based on the LLM-as-a-judge approach, face significant limitations, including biases, scalability issues, and dependence on costly external systems. To address these challenges, we propose a novel metric that contrasts the perplexity of the model response under two conditions: when the context is provided and when it is not. The resulting score quantifies the extent to which the model's answer relies on the provided context. The validity of this metric is demonstrated through a series of experiments that show its effectiveness in identifying whether a given answer is grounded in the provided context. Unlike existing approaches, this metric is computationally efficient, interpretable, and adaptable to various use cases, offering a scalable and practical solution to assess context utilization in open-book QA systems.
• 摘要：大型语言模型（LLMS）在开放式问题答案（QA）中表现出了很大的成功，其中该任务需要在提供的外部环境中生成的答案。开放式质量保证的一个关键挑战是确保模型响应基于提供的上下文而不是其参数知识，而参数知识可能过时，不完整或不正确。现有的评估方法主要基于LLM-AS-A-A-Gudge方法，面临着重大限制，包括偏见，可伸缩性问题和对昂贵外部系统的依赖。为了应对这些挑战，我们提出了一个新颖的指标，将模型响应在两个条件下的困惑形成对比：当提供上下文时，何时不提供上下文。最终的分数量化了模型的答案依赖于提供的上下文的程度。通过一系列实验证明了该指标的有效性，这些实验表明了其在确定给定答案是否在提供的环境中基础的有效性。与现有方法不同，该指标在计算上是有效的，可解释的，并且可以适应各种用例，提供了可扩展且实用的解决方案，以评估开放式质量检查QA系统中的上下文利用。

Title: Fine-Tuning LLMs for Low-Resource Dialect Translation: The Case of Lebanese

• Authors: Silvana Yakhni, Ali Chehab
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.00114
• Pdf URL: https://arxiv.org/pdf/2505.00114
• Copy Paste: [[2505.00114]] Fine-Tuning LLMs for Low-Resource Dialect Translation: The Case of Lebanese(https://arxiv.org/abs/2505.00114)
• Keywords: language model, llm, prompt
• Abstract: This paper examines the effectiveness of Large Language Models (LLMs) in translating the low-resource Lebanese dialect, focusing on the impact of culturally authentic data versus larger translated datasets. We compare three fine-tuning approaches: Basic, contrastive, and grammar-hint tuning, using open-source Aya23 models. Experiments reveal that models fine-tuned on a smaller but culturally aware Lebanese dataset (LW) consistently outperform those trained on larger, non-native data. The best results were achieved through contrastive fine-tuning paired with contrastive prompting, which indicates the benefits of exposing translation models to bad examples. In addition, to ensure authentic evaluation, we introduce LebEval, a new benchmark derived from native Lebanese content, and compare it to the existing FLoRes benchmark. Our findings challenge the "More Data is Better" paradigm and emphasize the crucial role of cultural authenticity in dialectal translation. We made our datasets and code available on Github.
• 摘要：本文探讨了大语言模型（LLM）在翻译低资源黎巴嫩方言的有效性，重点是文化真实数据与更大的翻译数据集的影响。我们使用开源AYA23型号比较了三种微调方法：基本，对比和语​​法暗示调整。实验表明，在较小但具有文化意识的黎巴嫩数据集（LW）上微调的模型始终优于接受较大的非本地数据训练的模型。最好的结果是通过对比度微调与对比度提示相结合的，这表明将翻译模型暴露于不良示例的好处。此外，为了确保真实的评估，我们引入了Lebeval，这是一种来自本地黎巴嫩本地内容的新基准，并将其与现有的Flores基准进行比较。我们的发现挑战“更多数据更好”范式，并强调文化真实性在方言翻译中的关键作用。我们在GitHub上提供了数据集和代码。

Title: Between Underthinking and Overthinking: An Empirical Study of Reasoning Length and correctness in LLMs

• Authors: Jinyan Su, Jennifer Healey, Preslav Nakov, Claire Cardie
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.00127
• Pdf URL: https://arxiv.org/pdf/2505.00127
• Copy Paste: [[2505.00127]] Between Underthinking and Overthinking: An Empirical Study of Reasoning Length and correctness in LLMs(https://arxiv.org/abs/2505.00127)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) are increasingly optimized for long reasoning, under the assumption that more reasoning leads to better performance. However, emerging evidence suggests that longer responses can sometimes degrade accuracy rather than improve it. In this paper, we conduct a systematic empirical study of the relationship between reasoning length and answer correctness. We find that LLMs tend to overthink simple problems, generating unnecessarily long outputs, and underthink harder ones, failing to extend their reasoning when it is most needed. This indicates that models might misjudge problem difficulty and fail to calibrate their response length appropriately. Furthermore, we investigate the effects of length reduction with a preference optimization algorithm when simply preferring the shorter responses regardless of answer correctness. Experiments show that the generation length can be significantly reduced while maintaining acceptable accuracy. Our findings highlight generation length as a meaningful signal for reasoning behavior and motivate further exploration into LLMs' self-awareness in reasoning length adaptation.
• 摘要：假设更多的推理会导致更好的性能，那么大型语言模型（LLM）越来越多地针对长期推理进行了优化。但是，新兴的证据表明，较长的响应有时会降低准确性而不是提高准确性。在本文中，我们对推理长度和答案正确性之间的关系进行了系统的经验研究。我们发现，LLM倾向于过度思考简单的问题，产生不必要的长输出，并且在最需要的情况下不得不延长推理。这表明模型可能会错误地判断问题的难度，并且无法适当校准其响应长度。此外，当简单地选择较短的响应时，无论答案正确，我们都会研究使用偏好优化算法的长度降低的影响。实验表明，在保持可接受的精度的同时，可以显着降低生成长度。我们的发现重点阐明了生成长度是推理行为的有意义的信号，并激发了对LLMS在推理长度适应中的自我意识的进一步探索。

Title: AdaptMI: Adaptive Skill-based In-context Math Instruction for Small Language Models

• Authors: Yinghui He, Abhishek Panigrahi, Yong Lin, Sanjeev Arora
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.00147
• Pdf URL: https://arxiv.org/pdf/2505.00147
• Copy Paste: [[2505.00147]] AdaptMI: Adaptive Skill-based In-context Math Instruction for Small Language Models(https://arxiv.org/abs/2505.00147)
• Keywords: language model, llm, prompt
• Abstract: In-context learning (ICL) allows a language model to improve its problem-solving capability when provided with suitable information in context. Since the choice of in-context information can be determined based on the problem itself, in-context learning is analogous to human learning from teachers in a classroom. Recent works (Didolkar et al., 2024a; 2024b) show that ICL performance can be improved by leveraging a frontier large language model's (LLM) ability to predict required skills to solve a problem, popularly referred to as an LLM's metacognition, and using the recommended skills to construct necessary in-context examples. While this skill-based strategy boosts ICL performance in larger models, its gains on small language models (SLMs) have been minimal, highlighting a performance gap in ICL capabilities. We investigate this gap and show that skill-based prompting can hurt SLM performance on easy questions by introducing unnecessary information, akin to cognitive overload. To address this, we introduce AdaptMI, an adaptive approach to selecting skill-based in-context Math Instructions for SLMs. Inspired by cognitive load theory from human pedagogy, our method only introduces skill-based examples when the model performs poorly. We further propose AdaptMI+, which adds examples targeted to the specific skills missing from the model's responses. On 5-shot evaluations across popular math benchmarks and five SLMs (1B--7B; Qwen, Llama), AdaptMI+ improves accuracy by up to 6% over naive skill-based strategies.
• 摘要：在上下文学习（ICL）中，当在上下文中提供合适的信息时，语言模型可以提高其解决问题的能力。由于可以根据问题本身确定信息中的信息的选择，因此在教室中的教师中的教师学习类似于人类的学习。最近的作品（Didolkar等，2024a; 2024b）表明，通过利用边境大​​语模型（LLM）的能力来预测所需技能以解决问题，通常称为LLM的元认知，并利用推荐技能来构建必要的内在示例，可以提高ICL性能。尽管这种基于技能的策略在较大的模型中提高了ICL的性能，但其在小语言模型（SLM）上的收益却很小，突出了ICL功能的性能差距。我们研究了这一差距，并表明基于技能的提示会通过引入不必要的信息（类似于认知过载）来损害SLM的表现。为了解决这个问题，我们介绍了Adaptmi，这是一种自适应方法，用于选择SLM的基于技能的内在数学指令。受到人类教育学的认知负荷理论的启发，我们的方法仅在模型表现较差时引入基于技能的示例。我们进一步提出了AdaptMi+，该+添加了针对模型响应中缺少的特定技能的示例。在跨流行的数学基准和五个SLM（1b--7b; Qwen，Llama）的5次评估中，Adaptmi+可提高准确性高达6％的基于天真的技能策略。

Title: Consistency in Language Models: Current Landscape, Challenges, and Future Directions

• Authors: Jekaterina Novikova, Carol Anderson, Borhane Blili-Hamelin, Subhabrata Majumdar
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.00268
• Pdf URL: https://arxiv.org/pdf/2505.00268
• Copy Paste: [[2505.00268]] Consistency in Language Models: Current Landscape, Challenges, and Future Directions(https://arxiv.org/abs/2505.00268)
• Keywords: language model, llm
• Abstract: The hallmark of effective language use lies in consistency -- expressing similar meanings in similar contexts and avoiding contradictions. While human communication naturally demonstrates this principle, state-of-the-art language models struggle to maintain reliable consistency across different scenarios. This paper examines the landscape of consistency research in AI language systems, exploring both formal consistency (including logical rule adherence) and informal consistency (such as moral and factual coherence). We analyze current approaches to measure aspects of consistency, identify critical research gaps in standardization of definitions, multilingual assessment, and methods to improve consistency. Our findings point to an urgent need for robust benchmarks to measure and interdisciplinary approaches to ensure consistency in the application of language models on domain-specific tasks while preserving the utility and adaptability.
• 摘要：有效语言使用的标志在于一致性 - 在相似的情况下表达相似的含义并避免矛盾。尽管人类交流自然而然地证明了这一原则，但最先进的语言模型努力在不同情况下保持可靠的一致性。本文探讨了AI语言系统中一致性研究的景观，探讨了正式的一致性（包括逻辑规则依从性）和非正式的一致性（例如道德和事实连贯性）。我们分析了当前的方法来衡量一致性的各个方面，确定定义标准化，多语言评估和提高一致性的方法的关键研究差距。我们的发现表明，迫切需要强大的基准测量和跨学科方法，以确保在特定于领域的任务上应用语言模型的一致性，同时保留效用和适应性。

Title: KoACD: The First Korean Adolescent Dataset for Cognitive Distortion Analysis

• Authors: JunSeo Kim, HyeHyeon Kim
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.00367
• Pdf URL: https://arxiv.org/pdf/2505.00367
• Copy Paste: [[2505.00367]] KoACD: The First Korean Adolescent Dataset for Cognitive Distortion Analysis(https://arxiv.org/abs/2505.00367)
• Keywords: language model, llm
• Abstract: Cognitive distortion refers to negative thinking patterns that can lead to mental health issues like depression and anxiety in adolescents. Previous studies using natural language processing (NLP) have focused mainly on small-scale adult datasets, with limited research on adolescents. This study introduces KoACD, the first large-scale dataset of cognitive distortions in Korean adolescents, containing 108,717 instances. We applied a multi-Large Language Model (LLM) negotiation method to refine distortion classification and generate synthetic data using two approaches: cognitive clarification for textual clarity and cognitive balancing for diverse distortion representation. Validation through LLMs and expert evaluations showed that while LLMs classified distortions with explicit markers, they struggled with context-dependent reasoning, where human evaluators demonstrated higher accuracy. KoACD aims to enhance future research on cognitive distortion detection.
• 摘要：认知失真是指可能导致心理健康问题（例如抑郁症和青少年焦虑）的负面思维模式。先前使用自然语言处理（NLP）的研究主要集中在小型成人数据集上，对青少年的研究有限。这项研究介绍了KOACD，这是韩国青少年认知扭曲的第一个大规模数据集，其中包含108,717个实例。我们应用了多大语言模型（LLM）谈判方法来完善失真分类并使用两种方法生成综合数据：用于文本清晰度和认知平衡的认知澄清，以实现各种失真表示。通过LLM和专家评估的验证表明，尽管LLMs用明确的标记将扭曲分类为扭曲，但它们在上下文依赖性推理中挣扎，在这种推理中，人类评估者表现出更高的准确性。 KOACD旨在增强对认知失真检测的未来研究。

Title: CSE-SFP: Enabling Unsupervised Sentence Representation Learning via a Single Forward Pass

• Authors: Bowen Zhang, Zixin Song, Chunping Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.00389
• Pdf URL: https://arxiv.org/pdf/2505.00389
• Copy Paste: [[2505.00389]] CSE-SFP: Enabling Unsupervised Sentence Representation Learning via a Single Forward Pass(https://arxiv.org/abs/2505.00389)
• Keywords: language model
• Abstract: As a fundamental task in Information Retrieval and Computational Linguistics, sentence representation has profound implications for a wide range of practical applications such as text clustering, content analysis, question-answering systems, and web search. Recent advances in pre-trained language models (PLMs) have driven remarkable progress in this field, particularly through unsupervised embedding derivation methods centered on discriminative PLMs like BERT. However, due to time and computational constraints, few efforts have attempted to integrate unsupervised sentence representation with generative PLMs, which typically possess much larger parameter sizes. Given that state-of-the-art models in both academia and industry are predominantly based on generative architectures, there is a pressing need for an efficient unsupervised text representation framework tailored to decoder-only PLMs. To address this concern, we propose CSE-SFP, an innovative method that exploits the structural characteristics of generative models. Compared to existing strategies, CSE-SFP requires only a single forward pass to perform effective unsupervised contrastive learning. Rigorous experimentation demonstrates that CSE-SFP not only produces higher-quality embeddings but also significantly reduces both training time and memory consumption. Furthermore, we introduce two ratio metrics that jointly assess alignment and uniformity, thereby providing a more robust means for evaluating the semantic spatial properties of encoding models.
• 摘要：作为信息检索和计算语言学的一项基本任务，句子表示对广泛的实用应用具有深远的影响，例如文本聚类，内容分析，提问系统和Web搜索。预训练的语言模型（PLM）的最新进展已在该领域取得了显着的进步，尤其是通过无监督的嵌入派生方法以贝特（Bert）等歧视性PLM为中心。但是，由于时间和计算限制，很少有努力试图将无监督的句子表示与生成PLMS集成，该句子通常具有更大的参数尺寸。鉴于学术界和行业中的最新模型主要基于生成体系结构，因此迫切需要有效的无监督文本表示框架量身定制的仅是解码器的PLM。为了解决这一问题，我们提出了CSE-SFP，这是一种创新的方法，可利用生成模型的结构特征。与现有策略相比，CSE-SFP仅需要单个前向通过即可进行有效的无监督对比学习。严格的实验表明，CSE-SFP不仅会产生更高质量的嵌入，而且还大大减少了训练时间和记忆消耗。此外，我们介绍了两个共同评估一致性和均匀性的比率指标，从而提供了一种更强大的方法来评估编码模型的语义空间特性。

Title: Red Teaming Large Language Models for Healthcare

• Authors: Vahid Balazadeh, Michael Cooper, David Pellow, Atousa Assadi, Jennifer Bell, Jim Fackler, Gabriel Funingana, Spencer Gable-Cook, Anirudh Gangadhar, Abhishek Jaiswal, Sumanth Kaja, Christopher Khoury, Randy Lin, Kaden McKeen, Sara Naimimohasses, Khashayar Namdar, Aviraj Newatia, Allan Pang, Anshul Pattoo, Sameer Peesapati, Diana Prepelita, Bogdana Rakova, Saba Sadatamin, Rafael Schulman, Ajay Shah, Syed Azhar Shah, Syed Ahmar Shah, Babak Taati, Balagopal Unnikrishnan, Stephanie Williams, Rahul G Krishnan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.00467
• Pdf URL: https://arxiv.org/pdf/2505.00467
• Copy Paste: [[2505.00467]] Red Teaming Large Language Models for Healthcare(https://arxiv.org/abs/2505.00467)
• Keywords: language model, llm, prompt
• Abstract: We present the design process and findings of the pre-conference workshop at the Machine Learning for Healthcare Conference (2024) entitled Red Teaming Large Language Models for Healthcare, which took place on August 15, 2024. Conference participants, comprising a mix of computational and clinical expertise, attempted to discover vulnerabilities -- realistic clinical prompts for which a large language model (LLM) outputs a response that could cause clinical harm. Red-teaming with clinicians enables the identification of LLM vulnerabilities that may not be recognised by LLM developers lacking clinical expertise. We report the vulnerabilities found, categorise them, and present the results of a replication study assessing the vulnerabilities across all LLMs provided.
• 摘要：We present the design process and findings of the pre-conference workshop at the Machine Learning for Healthcare Conference (2024) entitled Red Teaming Large Language Models for Healthcare, which took place on August 15, 2024. Conference participants, comprising a mix of computational and clinical expertise, attempted to discover vulnerabilities -- realistic clinical prompts for which a large language model (LLM) outputs a response that could cause clinical harm.与临床医生的红色团队可以识别LLM漏洞，而LLM开发人员缺乏临床专业知识可能无法认识到。我们报告发现，对它们进行分类的漏洞，并提出了一项复制研究的结果，以评估所有LLMS所提供的脆弱性。

Title: HalluMix: A Task-Agnostic, Multi-Domain Benchmark for Real-World Hallucination Detection

• Authors: Deanna Emery, Michael Goitia, Freddie Vargus, Iulia Neagu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.00506
• Pdf URL: https://arxiv.org/pdf/2505.00506
• Copy Paste: [[2505.00506]] HalluMix: A Task-Agnostic, Multi-Domain Benchmark for Real-World Hallucination Detection(https://arxiv.org/abs/2505.00506)
• Keywords: language model, llm, long context, hallucination, retrieval augmented generation
• Abstract: As large language models (LLMs) are increasingly deployed in high-stakes domains, detecting hallucinated content$\unicode{x2013}$text that is not grounded in supporting evidence$\unicode{x2013}$has become a critical challenge. Existing benchmarks for hallucination detection are often synthetically generated, narrowly focused on extractive question answering, and fail to capture the complexity of real-world scenarios involving multi-document contexts and full-sentence outputs. We introduce the HalluMix Benchmark, a diverse, task-agnostic dataset that includes examples from a range of domains and formats. Using this benchmark, we evaluate seven hallucination detection systems$\unicode{x2013}$both open and closed source$\unicode{x2013}$highlighting differences in performance across tasks, document lengths, and input representations. Our analysis highlights substantial performance disparities between short and long contexts, with critical implications for real-world Retrieval Augmented Generation (RAG) implementations. Quotient Detections achieves the best overall performance, with an accuracy of 0.82 and an F1 score of 0.84.
• 摘要：由于大型语言模型（LLMS）越来越多地部署在高风险域中，因此检测幻觉内容$ \ unicode {x2013} $文本，而不是基于支持证据$ \ unicode {x2013} $的文本已成为一个关键的挑战。现有的用于幻觉检测的基准通常是合成生成的，狭义地集中在提取问题上，并且未能捕获涉及多文档上下文和全句子输出的现实世界情景的复杂性。我们介绍了Hallumix Benchmark，这是一个多样化的任务信息数据集，其中包括来自各种域和格式的示例。使用此基准测试，我们评估了七个幻觉检测系统$ \ unicode {x2013} $开放和封闭的源$ \ unicode {x2013} $突出了跨任务，文档长度和输入表示的性能差异。我们的分析强调了短篇小说之间的实质性绩效差异，对现实世界检索增强发电（RAG）实现产生了关键的影响。商检测可实现最佳总体性能，精度为0.82，F1得分为0.84。

Title: 100 Days After DeepSeek-R1: A Survey on Replication Studies and More Directions for Reasoning Language Models

• Authors: Chong Zhang, Yue Deng, Xiang Lin, Bin Wang, Dianwen Ng, Hai Ye, Xingxuan Li, Yao Xiao, Zhanfeng Mo, Qi Zhang, Lidong Bing
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.00551
• Pdf URL: https://arxiv.org/pdf/2505.00551
• Copy Paste: [[2505.00551]] 100 Days After DeepSeek-R1: A Survey on Replication Studies and More Directions for Reasoning Language Models(https://arxiv.org/abs/2505.00551)
• Keywords: language model
• Abstract: The recent development of reasoning language models (RLMs) represents a novel evolution in large language models. In particular, the recent release of DeepSeek-R1 has generated widespread social impact and sparked enthusiasm in the research community for exploring the explicit reasoning paradigm of language models. However, the implementation details of the released models have not been fully open-sourced by DeepSeek, including DeepSeek-R1-Zero, DeepSeek-R1, and the distilled small models. As a result, many replication studies have emerged aiming to reproduce the strong performance achieved by DeepSeek-R1, reaching comparable performance through similar training procedures and fully open-source data resources. These works have investigated feasible strategies for supervised fine-tuning (SFT) and reinforcement learning from verifiable rewards (RLVR), focusing on data preparation and method design, yielding various valuable insights. In this report, we provide a summary of recent replication studies to inspire future research. We primarily focus on SFT and RLVR as two main directions, introducing the details for data construction, method design and training procedure of current replication studies. Moreover, we conclude key findings from the implementation details and experimental results reported by these studies, anticipating to inspire future research. We also discuss additional techniques of enhancing RLMs, highlighting the potential of expanding the application scope of these models, and discussing the challenges in development. By this survey, we aim to help researchers and developers of RLMs stay updated with the latest advancements, and seek to inspire new ideas to further enhance RLMs.
• 摘要：推理语言模型（RLMS）的最新发展代表了大型语言模型中的一种新颖进化。特别是，DeepSeek-R1的最新发布产生了广泛的社会影响，并激发了研究社区的热情，以探索语言模型的明确推理范式。但是，DeepSeek尚未完全开源，其中包括DeepSeek-R1-Zero，DeepSeek-R1和蒸馏小型型号。结果，已经出现了许多复制研究，旨在重现DeepSeek-R1所取得的强劲表现，通过类似的培训程序和完全开源的数据资源达到可比的性能。这些作品研究了可行的策略（SFT），并从可验证的奖励（RLVR）中进行了强化学习，重点是数据准备和方法设计，从而产生了各种有价值的见解。在本报告中，我们提供了近期复制研究的摘要，以激发未来的研究。我们主要关注SFT和RLVR作为两个主要方向，介绍了当前复制研究的数据构建，方法设计和培训程序的详细信息。此外，我们从这些研究报告的实施细节和实验结果中得出结论，预计会激发未来的研究。我们还讨论了增强RLM的其他技术，突出了扩大这些模型的应用范围的潜力，并讨论了开发中的挑战。通过这项调查，我们旨在帮助RLMS的研究人员和开发人员随着最新进步的最新进展，并寻求激发新的想法以进一步增强RLM。

Title: Triggering Hallucinations in LLMs: A Quantitative Study of Prompt-Induced Hallucination in Large Language Models

• Authors: Makoto Sato
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.00557
• Pdf URL: https://arxiv.org/pdf/2505.00557
• Copy Paste: [[2505.00557]] Triggering Hallucinations in LLMs: A Quantitative Study of Prompt-Induced Hallucination in Large Language Models(https://arxiv.org/abs/2505.00557)
• Keywords: language model, llm, hallucination, prompt
• Abstract: Hallucinations in large language models (LLMs) present a growing challenge across real-world applications, from healthcare to law, where factual reliability is essential. Despite advances in alignment and instruction tuning, LLMs can still generate outputs that are fluent yet fundamentally untrue. Understanding the cognitive dynamics that underlie these hallucinations remains an open problem. In this study, we propose a prompt-based framework to systematically trigger and quantify hallucination: a Hallucination-Inducing Prompt (HIP), which synthetically fuses semantically distant concepts (e.g., periodic table of elements and tarot divination) in a misleading way, and a Hallucination Quantifying Prompt (HQP), which scores the plausibility, confidence, and coherence of the output. Controlled experiments across multiple LLMs revealed that HIPs consistently produced less coherent and more hallucinated responses than their null-fusion controls. These effects varied across models, with reasoning-oriented LLMs showing distinct profiles from general-purpose ones. Our framework provides a reproducible testbed for studying hallucination vulnerability, and opens the door to developing safer, more introspective LLMs that can detect and self-regulate the onset of conceptual instability.
• 摘要：大语言模型（LLMS）中的幻觉表现出了从医疗保健到法律的现实应用程序的日益严重的挑战，在这些应用程序中，事实可靠性至关重要。尽管对齐和说明进行了进步，但LLM仍可以产生流利但根本上不真实的输出。了解这些幻觉是基于这些幻觉的认知动力仍然是一个空旷的问题。 In this study, we propose a prompt-based framework to systematically trigger and quantify hallucination: a Hallucination-Inducing Prompt (HIP), which synthetically fuses semantically distant concepts (e.g., periodic table of elements and tarot divination) in a misleading way, and a Hallucination Quantifying Prompt (HQP), which scores the plausibility, confidence, and coherence of the output.跨多个LLM的受控实验表明，臀部始终产生的连贯性和幻觉更高的反应比无效的融合对照。这些效果在模型之间各不相同，面向推理的LLM显示出与通用的LLM不同。我们的框架为研究幻觉脆弱性提供了可重现的测试床，并为开发更安全，更内外的LLM开辟了大门，可以检测和自我调节概念不稳定的发作。

Title: FreqKV: Frequency Domain Key-Value Compression for Efficient Context Window Extension

• Authors: Jushi Kai, Boyi Zeng, Yixuan Wang, Haoli Bai, Bo Jiang, Zhouhan Lin
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.00570
• Pdf URL: https://arxiv.org/pdf/2505.00570
• Copy Paste: [[2505.00570]] FreqKV: Frequency Domain Key-Value Compression for Efficient Context Window Extension(https://arxiv.org/abs/2505.00570)
• Keywords: language model, llm, long context
• Abstract: Extending the context window in large language models (LLMs) is essential for applications involving long-form content generation. However, the linear increase in key-value (KV) cache memory requirements and the quadratic complexity of self-attention with respect to sequence length present significant challenges during fine-tuning and inference. Existing methods suffer from performance degradation when extending to longer contexts. In this work, we introduce a novel context extension method that optimizes both fine-tuning and inference efficiency. Our method exploits a key observation: in the frequency domain, the energy distribution of the KV cache is primarily concentrated in low-frequency components. By filtering out the high-frequency components, the KV cache can be effectively compressed with minimal information loss. Building on this insight, we propose an efficient compression technique, FreqKV, that iteratively compresses the increasing KV cache to a fixed size in the frequency domain, applicable to both fine-tuning and inference. FreqKV introduces no additional parameters or architectural modifications. With minimal fine-tuning, LLMs can learn to leverage the limited cache that is compressed in the frequency domain and extend the context window efficiently. Experiments on various long context language modeling and understanding tasks demonstrate the efficiency and efficacy of the proposed method.
• 摘要：在大型语言模型（LLMS）中扩展上下文窗口对于涉及长格式内容生成的应用至关重要。但是，键值（KV）缓存记忆要求的线性增加以及对序列长度相对于序列长度的二次复杂性提出了微调和推理期间的重大挑战。当延伸到更长的上下文时，现有方法会遭受性能降解。在这项工作中，我们介绍了一种新颖的上下文扩展方法，以优化微调和推理效率。我们的方法利用了一个关键的观察：在频域中，KV缓存的能量分布主要集中在低频组件中。通过滤除高频组件，可以通过最小的信息丢失有效地压缩KV缓存。在此洞察力的基础上，我们提出了一种有效的压缩技术FREQKV，它迭代地将增加的KV缓存压缩到频域中的固定尺寸，适用于微调和推理。 FREQKV没有引入其他参数或架构修改。通过最小的微调，LLM可以学会利用在频域中压缩的有限缓存并有效地扩展上下文窗口。对各种长上下文语言建模和理解任务的实验证明了该方法的效率和功效。

Title: Block Circulant Adapter for Large Language Models

• Authors: Xinyu Ding, Meiqi Wang, Siyu Liao, Zhongfeng Wang
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2505.00582
• Pdf URL: https://arxiv.org/pdf/2505.00582
• Copy Paste: [[2505.00582]] Block Circulant Adapter for Large Language Models(https://arxiv.org/abs/2505.00582)
• Keywords: language model, llm
• Abstract: Fine-tuning large language models (LLMs) is difficult due to their huge model size. Recent Fourier domain-based methods show potential for reducing fine-tuning costs. We propose a block circulant matrix-based fine-tuning method with a stable training heuristic to leverage the properties of circulant matrices and one-dimensional Fourier transforms to reduce storage and computation costs. Experiments show that our method uses $14\times$ less number of parameters than VeRA, $16\times$ smaller than LoRA and $32\times$ less FLOPs than FourierFT, while maintaining close or better task performance. Our approach presents a promising way in frequency domain to fine-tune large models on downstream tasks.
• 摘要：通过微调大型语言模型（LLM），由于其巨大的模型大小很难。最近的基于傅立叶域的方法显示了降低微调成本的潜力。我们提出了一种基于稳定矩阵的块循环型微调方法，并采用稳定的训练启发式训练，以利用循环矩阵和一维傅立叶变换的特性来降低存储和计算成本。实验表明，我们的方法使用$ 14 \ times $ $比Vera的参数数量少，$ 16 \ times $ $比Lora小，$ 32 \ times $ $ $ $比Fourierft少，同时保持关闭或更好的任务性能。我们的方法在频域中提出了一种有希望的方式，可以在下游任务上微调大型模型。

Title: FineScope : Precision Pruning for Domain-Specialized Large Language Models Using SAE-Guided Self-Data Cultivation

• Authors: Chaitali Bhattacharyya, Yeseong Kim
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.00624
• Pdf URL: https://arxiv.org/pdf/2505.00624
• Copy Paste: [[2505.00624]] FineScope : Precision Pruning for Domain-Specialized Large Language Models Using SAE-Guided Self-Data Cultivation(https://arxiv.org/abs/2505.00624)
• Keywords: language model, llm
• Abstract: Training large language models (LLMs) from scratch requires significant computational resources, driving interest in developing smaller, domain-specific LLMs that maintain both efficiency and strong task performance. Medium-sized models such as LLaMA, llama} have served as starting points for domain-specific adaptation, but they often suffer from accuracy degradation when tested on specialized datasets. We introduce FineScope, a framework for deriving compact, domain-optimized LLMs from larger pretrained models. FineScope leverages the Sparse Autoencoder (SAE) framework, inspired by its ability to produce interpretable feature representations, to extract domain-specific subsets from large datasets. We apply structured pruning with domain-specific constraints, ensuring that the resulting pruned models retain essential knowledge for the target domain. To further enhance performance, these pruned models undergo self-data distillation, leveraging SAE-curated datasets to restore key domain-specific information lost during pruning. Extensive experiments and ablation studies demonstrate that FineScope achieves highly competitive performance, outperforming several large-scale state-of-the-art LLMs in domain-specific tasks. Additionally, our results show that FineScope enables pruned models to regain a substantial portion of their original performance when fine-tuned with SAE-curated datasets. Furthermore, applying these datasets to fine-tune pretrained LLMs without pruning also improves their domain-specific accuracy, highlighting the robustness of our approach. The code will be released.
• 摘要：从头开始培训大语言模型（LLMS）需要大量的计算资源，引起人们对开发较小的域特异性LLM的兴趣，这些LLM既维持效率又有强大的任务绩效。中型模型，例如美洲驼，骆驼}已成为特定领域适应的起点，但是在专用数据集中测试时，它们通常会遭受准确性降解。我们介绍了FineScope，这是一个框架，用于从较大的预审计模型中得出紧凑的，域优化的LLM。 Finescope利用稀疏的自动编码器（SAE）框架，其灵感来自其产生可解释的特征表示的能力，从大型数据集中提取特定于域的子集。我们将结构化的修剪应用于特定于域的约束，确保所得的修剪模型保留目标域的基本知识。为了进一步提高性能，这些修剪模型会进行自DATA蒸馏，利用SAE策划的数据集恢复修剪过程中丢失的关键域特异性信息。广泛的实验和消融研究表明，罚款能够取得高度竞争性的性能，在特定领域的任务中表现出色，表现优于几个大规模的最先进的LLM。此外，我们的结果表明，Finescope使修剪模型可以在用SAE策划的数据集进行微调时重新获得其原始性能的很大一部分。此外，将这些数据集应用于微调的LLM中，而不修剪也提高了其特定于域的精度，从而突出了我们方法的鲁棒性。代码将发布。

Title: The Illusion of Role Separation: Hidden Shortcuts in LLM Role Learning (and How to Fix Them)

• Authors: Zihao Wang, Yibo Jiang, Jiahao Yu, Heqing Huang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.00626
• Pdf URL: https://arxiv.org/pdf/2505.00626
• Copy Paste: [[2505.00626]] The Illusion of Role Separation: Hidden Shortcuts in LLM Role Learning (and How to Fix Them)(https://arxiv.org/abs/2505.00626)
• Keywords: language model, llm, prompt
• Abstract: Large language models (LLMs) that integrate multiple input roles (e.g., system instructions, user queries, external tool outputs) are increasingly prevalent in practice. Ensuring that the model accurately distinguishes messages from each role -- a concept we call \emph{role separation} -- is crucial for consistent multi-role behavior. Although recent work often targets state-of-the-art prompt injection defenses, it remains unclear whether such methods truly teach LLMs to differentiate roles or merely memorize known triggers. In this paper, we examine \emph{role-separation learning}: the process of teaching LLMs to robustly distinguish system and user tokens. Through a \emph{simple, controlled experimental framework}, we find that fine-tuned models often rely on two proxies for role identification: (1) task type exploitation, and (2) proximity to begin-of-text. Although data augmentation can partially mitigate these shortcuts, it generally leads to iterative patching rather than a deeper fix. To address this, we propose reinforcing \emph{invariant signals} that mark role boundaries by adjusting token-wise cues in the model's input encoding. In particular, manipulating position IDs helps the model learn clearer distinctions and reduces reliance on superficial proxies. By focusing on this mechanism-centered perspective, our work illuminates how LLMs can more reliably maintain consistent multi-role behavior without merely memorizing known prompts or triggers.
• 摘要：大型语言模型（LLM）在实践中越来越普遍，该模型（例如，系统指令，用户查询，外部工具输出）越来越普遍。确保模型将消息与每个角色的准确区分 - 我们称为\ emph {romph {角色分离}的概念 - 对于一致的多角色行为至关重要。尽管最近的工作通常针对最先进的迅速注射防御，但尚不清楚这种方法是否真正教授LLMS来区分角色或仅记住已知的触发器。在本文中，我们检查\ emph {角色分离学习}：教授LLMS以鲁棒区分系统和用户令牌的过程。通过\ emph {简单，控制的实验框架}，我们发现微调模型通常依靠两个代理来识别：（1）任务类型开发，以及（2）（2）与初学版的接近。尽管数据增强可以部分减轻这些快捷方式，但通常会导致迭代修补而不是更深层的修复。为了解决这个问题，我们建议通过调整模型输入编码中的令牌提示来标记角色边界的增强\ emph {不变信号}。特别是，操纵位置ID有助于模型学习更清晰的区别，并减少对表面代理的依赖。通过关注以这种机制为中心的观点，我们的工作阐明了LLM如何可靠地保持一致的多角色行为，而不仅仅是记住已知的提示或触发器。

Title: Large Language Models Understanding: an Inherent Ambiguity Barrier

• Authors: Daniel N. Nissani (Nissensohn)
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.00654
• Pdf URL: https://arxiv.org/pdf/2505.00654
• Copy Paste: [[2505.00654]] Large Language Models Understanding: an Inherent Ambiguity Barrier(https://arxiv.org/abs/2505.00654)
• Keywords: language model, llm
• Abstract: A lively ongoing debate is taking place, since the extraordinary emergence of Large Language Models (LLMs) with regards to their capability to understand the world and capture the meaning of the dialogues in which they are involved. Arguments and counter-arguments have been proposed based upon thought experiments, anecdotal conversations between LLMs and humans, statistical linguistic analysis, philosophical considerations, and more. In this brief paper we present a counter-argument based upon a thought experiment and semi-formal considerations leading to an inherent ambiguity barrier which prevents LLMs from having any understanding of what their amazingly fluent dialogues mean.
• 摘要：一场活泼的辩论正在进行，因为大型语言模型（LLMS）的非凡出现在理解世界并捕捉其参与对话的含义方面。根据思想实验，LLM与人类之间的轶事对话，统计语言分析，哲学考虑等，提出了争论和反论点。在这篇简短的论文中，我们根据思想实验和半正式考虑提出了反对言论，导致固有的歧义障碍，这阻止了LLMS对他们惊人的流利对话的任何理解。

Title: On the generalization of language models from in-context learning and finetuning: a controlled study

• Authors: Andrew K. Lampinen, Arslan Chaudhry, Stephanie C.Y. Chan, Cody Wild, Diane Wan, Alex Ku, Jörg Bornschein, Razvan Pascanu, Murray Shanahan, James L. McClelland
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2505.00661
• Pdf URL: https://arxiv.org/pdf/2505.00661
• Copy Paste: [[2505.00661]] On the generalization of language models from in-context learning and finetuning: a controlled study(https://arxiv.org/abs/2505.00661)
• Keywords: language model
• Abstract: Large language models exhibit exciting capabilities, yet can show surprisingly narrow generalization from finetuning -- from failing to generalize to simple reversals of relations they are trained on, to missing logical deductions that can be made from trained information. These failures to generalize from fine-tuning can hinder practical application of these models. However, language models' in-context learning shows different inductive biases, and can generalize better in some of these cases. Here, we explore these differences in generalization between in-context- and fine-tuning-based learning. To do so, we constructed several novel datasets to evaluate and improve models' ability to generalize from finetuning data. The datasets are constructed to isolate the knowledge in the dataset from that in pretraining, to create clean tests of generalization. We expose pretrained large models to controlled subsets of the information in these datasets -- either in context, or through fine-tuning -- and evaluate their performance on test sets that require various types of generalization. We find overall that in data-matched settings, in-context learning can generalize more flexibly than fine-tuning (though we also find some qualifications of prior findings, such as cases when fine-tuning can generalize to reversals embedded in a larger structure of knowledge). We build on these findings to propose a method to enable improved generalization from fine-tuning: adding in-context inferences to finetuning data. We show that this method improves generalization across various splits of our datasets and other benchmarks. Our results have implications for understanding the inductive biases of different modes of learning in language models, and practically improving their performance.
• 摘要：大型语言模型表现出令人兴奋的功能，但可以表现出较狭窄的概括，从未能概括到他们接受过训练的关系的简单逆转，再到缺少通过训练有素的信息进行的逻辑推论。这些因微调而概括的失败会阻碍这些模型的实际应用。但是，语言模型的中文学习表现出不同的归纳偏见，并且在某些情况下可以更好地概括。在这里，我们探讨了基于文化和微调学习之间的这些概括差异。为此，我们构建了几个新型数据集，以评估和提高模型从填充数据中概括的能力。构建数据集是为了将数据集中的知识与预处理隔离，以创建清洁的概括测试。我们将预估计的大型模型暴露于这些数据集中的信息的子集（在上下文中或通过微调），并评估其在需要各种概括的测试集上的性能。我们发现总体而言，在数据匹配的设置中，内部上下文学习可以比微调更灵活地概括（尽管我们还发现了一些先前发现的资格，例如，微调可以推广到嵌入在更大知识结构中的逆转的情况）。我们以这些发现为基础，以提出一种从微调中改进概括的方法：在鉴定数据中添加中下文推断。我们表明，这种方法改善了我们数据集和其他基准测试的各种分裂的概括。我们的结果对了解语言模型中不同学习模式的归纳偏见以及实际上改善其表现具有影响。

Title: DeepCritic: Deliberate Critique with Large Language Models

• Authors: Wenkai Yang, Jingwen Chen, Yankai Lin, Ji-Rong Wen
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2505.00662
• Pdf URL: https://arxiv.org/pdf/2505.00662
• Copy Paste: [[2505.00662]] DeepCritic: Deliberate Critique with Large Language Models(https://arxiv.org/abs/2505.00662)
• Keywords: language model, gpt, llm
• Abstract: As Large Language Models (LLMs) are rapidly evolving, providing accurate feedback and scalable oversight on their outputs becomes an urgent and critical problem. Leveraging LLMs as critique models to achieve automated supervision is a promising solution. In this work, we focus on studying and enhancing the math critique ability of LLMs. Current LLM critics provide critiques that are too shallow and superficial on each step, leading to low judgment accuracy and struggling to offer sufficient feedback for the LLM generator to correct mistakes. To tackle this issue, we propose a novel and effective two-stage framework to develop LLM critics that are capable of deliberately critiquing on each reasoning step of math solutions. In the first stage, we utilize Qwen2.5-72B-Instruct to generate 4.5K long-form critiques as seed data for supervised fine-tuning. Each seed critique consists of deliberate step-wise critiques that includes multi-perspective verifications as well as in-depth critiques of initial critiques for each reasoning step. Then, we perform reinforcement learning on the fine-tuned model with either existing human-labeled data from PRM800K or our automatically annotated data obtained via Monte Carlo sampling-based correctness estimation, to further incentivize its critique ability. Our developed critique model built on Qwen2.5-7B-Instruct not only significantly outperforms existing LLM critics (including the same-sized DeepSeek-R1-distill models and GPT-4o) on various error identification benchmarks, but also more effectively helps the LLM generator refine erroneous steps through more detailed feedback.
• 摘要：随着大型语言模型（LLM）正在迅速发展，提供准确的反馈和对其产出的可扩展监督成为一个紧迫而关键的问题。利用LLM作为批评模型来实现自动监督是一个有前途的解决方案。在这项工作中，我们专注于研究和增强LLM的数学批评能力。当前的LLM批评家提供的批评在每个步骤上都过于浅，表面，导致判断准确性降低，并努力为LLM发电机提供足够的反馈以纠正错误。为了解决这个问题，我们提出了一个新颖而有效的两阶段框架，以开发LLM批评家，能够故意在数学解决方案的每个推理步骤中批评。在第一阶段，我们利用QWEN2.5-72B-INSTRUCT产生4.5K长形的批评作为种子数据，以进行监督微调。每种种子批评都由有意的逐步批评组成，其中包括多种验证验证以及对每个推理步骤的初始评论的深入批评。然后，我们使用来自PRM800K的现有人类标记的数据或通过基于蒙特卡洛采样的正确性估计获得的自动注释数据进行了微调模型的增强学习，以进一步激励其批评能力。我们开发的批评模型基于QWEN2.5-7B教学，不仅在各种错误识别基准上都显着胜过现有的LLM批评家（包括相同大小的DeepSeek-R1-Distill模型和GPT-4O），而且更有效地帮助LLM Generator Refinator Perfinator Perfinator Perfinator Ristose risone spects通过更详细的反馈。

Title: Rethinking Memory in AI: Taxonomy, Operations, Topics, and Future Directions

• Authors: Yiming Du, Wenyu Huang, Danna Zheng, Zhaowei Wang, Sebastien Montella, Mirella Lapata, Kam-Fai Wong, Jeff Z. Pan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.00675
• Pdf URL: https://arxiv.org/pdf/2505.00675
• Copy Paste: [[2505.00675]] Rethinking Memory in AI: Taxonomy, Operations, Topics, and Future Directions(https://arxiv.org/abs/2505.00675)
• Keywords: language model, llm, agent
• Abstract: Memory is a fundamental component of AI systems, underpinning large language models (LLMs) based agents. While prior surveys have focused on memory applications with LLMs, they often overlook the atomic operations that underlie memory dynamics. In this survey, we first categorize memory representations into parametric, contextual structured, and contextual unstructured and then introduce six fundamental memory operations: Consolidation, Updating, Indexing, Forgetting, Retrieval, and Compression. We systematically map these operations to the most relevant research topics across long-term, long-context, parametric modification, and multi-source memory. By reframing memory systems through the lens of atomic operations and representation types, this survey provides a structured and dynamic perspective on research, benchmark datasets, and tools related to memory in AI, clarifying the functional interplay in LLMs based agents while outlining promising directions for future research\footnote{The paper list, datasets, methods and tools are available at \href{this https URL}{this https URL\_Memory\_in\_AI}.}.
• 摘要：内存是基于大型语言模型（LLMS）代理的基础AI系统的基本组成部分。虽然先前的调查专注于LLMS的内存应用，但它们经常忽略内存动态的原子操作。在此调查中，我们首先将内存表示形式分为参数，上下文结构化和上下文非结构化，然后引入六个基本内存操作：合并，更新，索引，遗忘，检索和压缩。我们将这些操作系统地映射到长期，长篇下说，参数修改和多源内存的最相关研究主题。 By reframing memory systems through the lens of atomic operations and representation types, this survey provides a structured and dynamic perspective on research, benchmark datasets, and tools related to memory in AI, clarifying the functional interplay in LLMs based agents while outlining promising directions for future research\footnote{The paper list, datasets, methods and tools are available at \href{this https url} {这个https url \ _memory \ _in \ _ai}。}。}。

Title: Steering Large Language Models with Register Analysis for Arbitrary Style Transfer

• Authors: Xinchen Yang, Marine Carpuat
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2505.00679
• Pdf URL: https://arxiv.org/pdf/2505.00679
• Copy Paste: [[2505.00679]] Steering Large Language Models with Register Analysis for Arbitrary Style Transfer(https://arxiv.org/abs/2505.00679)
• Keywords: language model, llm, prompt
• Abstract: Large Language Models (LLMs) have demonstrated strong capabilities in rewriting text across various styles. However, effectively leveraging this ability for example-based arbitrary style transfer, where an input text is rewritten to match the style of a given exemplar, remains an open challenge. A key question is how to describe the style of the exemplar to guide LLMs toward high-quality rewrites. In this work, we propose a prompting method based on register analysis to guide LLMs to perform this task. Empirical evaluations across multiple style transfer tasks show that our prompting approach enhances style transfer strength while preserving meaning more effectively than existing prompting strategies.
• 摘要：大型语言模型（LLMS）在重写各种样式的文本方面表现出了强大的功能。但是，有效利用此功能，例如基于任意样式的转移，其中重写输入文本以匹配给定示例的样式，这仍然是一个开放的挑战。一个关键问题是如何描述示例的风格，以指导LLMS朝着高质量的重写。在这项工作中，我们提出了一种基于寄存器分析的提示方法，以指导LLMS执行此任务。跨多种样式转移任务的经验评估表明，我们的提示方法增强了样式转移强度，同时比现有的提示策略更有效地保留意义。