2025-03-17

Title: Text2Zinc: A Cross-Domain Dataset for Modeling Optimization and Satisfaction Problems in MiniZinc

• Authors: Akash Singirikonda, Serdar Kadioglu, Karthik Uppuluri
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.10642
• Pdf URL: https://arxiv.org/pdf/2503.10642
• Copy Paste: [[2503.10642]] Text2Zinc: A Cross-Domain Dataset for Modeling Optimization and Satisfaction Problems in MiniZinc(https://arxiv.org/abs/2503.10642)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: There is growing interest in utilizing large language models (LLMs) as co-pilots for combinatorial optimization and constraint programming tasks across various problems. This paper aims to advance this line of research by introducing Text2Zinc}, a cross-domain dataset for capturing optimization and satisfaction problems specified in natural language text. Our work is distinguished from previous attempts by integrating both satisfaction and optimization problems within a unified dataset using a solver-agnostic modeling language. To achieve this, we leverage MiniZinc's solver-and-paradigm-agnostic modeling capabilities to formulate these problems. Using the Text2Zinc dataset, we conduct comprehensive baseline experiments to compare execution and solution accuracy across several methods, including off-the-shelf prompting strategies, chain-of-thought reasoning, and a compositional approach. Additionally, we explore the effectiveness of intermediary representations, specifically knowledge graphs. Our findings indicate that LLMs are not yet a push-button technology to model combinatorial problems from text. We hope that Text2Zinc serves as a valuable resource for researchers and practitioners to advance the field further.
• 摘要：利用大型语言模型（LLM）作为组合优化和各种问题的约束编程任务的副驾驶，人们越来越兴趣。本文旨在通过引入Text2Zinc}来推进这一研究，这是一个跨域数据集，用于捕获自然语言文本中指定的优化和满意度问题。我们的工作与以前的尝试通过使用求解器 - 非局部建模语言在统一数据集中整合了满意度和优化问题来区分。为了实现这一目标，我们利用了Minizinc的求解器和范式 - 不足的建模功能来提出这些问题。使用Text2ZINC数据集，我们进行了全面的基线实验，以比较几种方法的执行和解决方案准确性，包括现成的提示策略，经过思考的推理和组成方法。此外，我们探讨了中介表示的有效性，特别是知识图。我们的发现表明，LLMS尚未推出按钮技术来对文本进行建模组合问题。我们希望Text2Zinc是研究人员和从业者进一步推进领域的宝贵资源。

Title: Synthetic Categorical Restructuring large Or How AIs Gradually Extract Efficient Regularities from Their Experience of the World

• Authors: Michael Pichat, William Pogrund, Paloma Pichat, Armanouche Gasparian, Samuel Demarchi, Martin Corbet, Alois Georgeon, Theo Dasilva, Michael Veillet-Guillem
• Subjects: cs.CL, cs.NE, q-bio.NC
• Abstract URL: https://arxiv.org/abs/2503.10643
• Pdf URL: https://arxiv.org/pdf/2503.10643
• Copy Paste: [[2503.10643]] Synthetic Categorical Restructuring large Or How AIs Gradually Extract Efficient Regularities from Their Experience of the World(https://arxiv.org/abs/2503.10643)
• Keywords: language model, gpt
• Abstract: How do language models segment their internal experience of the world of words to progressively learn to interact with it more efficiently? This study in the neuropsychology of artificial intelligence investigates the phenomenon of synthetic categorical restructuring, a process through which each successive perceptron neural layer abstracts and combines relevant categorical sub-dimensions from the thought categories of its previous layer. This process shapes new, even more efficient categories for analyzing and processing the synthetic system's own experience of the linguistic external world to which it is exposed. Our genetic neuron viewer, associated with this study, allows visualization of the synthetic categorical restructuring phenomenon occurring during the transition from perceptron layer 0 to 1 in GPT2-XL.
• 摘要：语言模型如何细分他们对单词世界的内部经验，以逐步学会更有效地学习与之互动？这项在人工智能神经心理学中的研究调查了合成分类重组的现象，这一过程通过该过程，通过该过程，每个连续的感知神经层摘要，并结合了其先前层思想类别的相关分类细分。这个过程塑造了新的，甚至更有效的类别，用于分析和处理合成系统对其被暴露的语言外部世界的体验。我们与这项研究相关的遗传神经元观察者允许在从gpt2-XL中从观察ptron第0层到1的过渡期间看到合成的分类重组现象。

Title: The Reliability of LLMs for Medical Diagnosis: An Examination of Consistency, Manipulation, and Contextual Awareness

• Authors: Krishna Subedi
• Subjects: cs.CL, cs.AI, cs.CY, cs.HC
• Abstract URL: https://arxiv.org/abs/2503.10647
• Pdf URL: https://arxiv.org/pdf/2503.10647
• Copy Paste: [[2503.10647]] The Reliability of LLMs for Medical Diagnosis: An Examination of Consistency, Manipulation, and Contextual Awareness(https://arxiv.org/abs/2503.10647)
• Keywords: language model, gpt, llm, chat
• Abstract: Universal healthcare access is critically needed, especially in resource-limited settings. Large Language Models (LLMs) offer promise for democratizing healthcare with advanced diagnostics, but their reliability requires thorough evaluation, especially in trust-dependent environments. This study assesses LLMs' diagnostic reliability focusing on consistency, manipulation resilience, and contextual integration, crucial for safe and ethical use in universal healthcare. We evaluated leading LLMs using 52 patient cases, expanded into variants with demographic changes, symptom rewordings, and exam modifications, while keeping core diagnoses constant. Manipulation susceptibility was tested by inserting misleading narratives and irrelevant details. Contextual awareness was rvaluated by comparing diagnoses with and without patient history. We analyzed diagnostic change rates and response patterns across manipulations. LLMs showed perfect diagnostic consistency for identical data but significant manipulation susceptibility. Gemini had a 40% diagnosis change rate and ChatGPT 30% with irrelevant details. ChatGPT had a higher context influence rate (77.8% vs. Gemini's 55.6%), but both showed limited nuanced contextual integration, exhibiting anchoring bias by prioritizing salient data over context. LLMs' vulnerability to manipulation and limited contextual awareness pose challenges in clinical use. Unlike clinicians, they may overstate diagnostic certainty without validation. Safeguards and domain-specific designs are crucial for reliable healthcare applications. Broad clinical use without oversight is premature and risky. LLMs can enhance diagnostics with responsible use, but future research is needed to improve manipulation resistance and contextual understanding for safe healthcare democratization.
• 摘要：通用医疗保健访问是至关重要的，尤其是在资源有限的设置中。大型语言模型（LLMS）为通过高级诊断的医疗保健民主化提供了希望，但是它们的可靠性需要彻底评估，尤其是在依赖信任的环境中。这项研究评估了LLMS的诊断可靠性，重点是一致性，操纵弹性和上下文整合，对于普遍医疗保健中的安全和道德使用至关重要。我们使用52例患者病例评估了领先的LLM，并扩展为具有人口统计学变化，症状重新词和检查修改的变体，同时保持核心诊断的恒定。通过插入误导性叙述和无关紧要的细节来测试操纵敏感性。通过将诊断与患者病史进行比较，可以通过比较诊断来进行上下文意识。我们分析了跨操纵的诊断变化率和响应模式。对于相同的数据，LLM显示出完美的诊断一致性，但操纵敏感性很高。双子座的诊断变化率为40％，而Chatgpt则有30％的细节。 Chatgpt具有更高的上下文影响率（77.8％对Gemini的55.6％），但两者都显示出有限的细微差别上下文集成，通过优先考虑明显数据而不是上下文表现出锚定偏见。 LLMS对操纵的脆弱性和有限的上下文意识在临床使用中构成了挑战。与临床医生不同，他们可能无需验证就可以夸大诊断确定性。保障措施和特定领域的设计对于可靠的医疗保健应用至关重要。没有监督的广泛临床用途过早且风险。 LLM可以通过负责任的使用来增强诊断，但是需要将来的研究以改善对安全医疗保健民主化的抵抗力和上下文理解。

Title: Evaluating Local and Cloud-Based Large Language Models for Simulating Consumer Choices in Energy Stated Preference Surveys

• Authors: Han Wang, Jacek Pawlak, Aruna Sivakumar
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2503.10652
• Pdf URL: https://arxiv.org/pdf/2503.10652
• Copy Paste: [[2503.10652]] Evaluating Local and Cloud-Based Large Language Models for Simulating Consumer Choices in Energy Stated Preference Surveys(https://arxiv.org/abs/2503.10652)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: Survey research is essential in energy demand studies for capturing consumer preferences and informing policy decisions. Stated preference (SP) surveys, in particular, analyse how individuals make trade-offs in hypothetical scenarios. However, traditional survey methods are costly, time-consuming, and affected by biases and respondent fatigue. Large language models (LLMs) have emerged as a potential tool to address these challenges by generating human-like textual responses. This study investigates the ability of LLMs to simulate consumer choices in energy-related SP surveys. A series of test scenarios evaluated the simulation performance of LLMs at both individual and aggregated levels, considering factors in the prompt, in-context learning (ICL), chain-of-thought (CoT) reasoning, the comparison between local and cloud-based LLMs, integration with traditional choice models, and potential biases. Results indicate that while LLMs achieve an average accuracy of up to 48%, surpassing random guessing, their performance remains insufficient for practical application. Local and cloud-based LLMs perform similarly in simulation accuracy but exhibit differences in adherence to prompt requirements and susceptibility to social desirability biases. Findings suggest that previous SP choices are the most effective input factor, while longer prompts with varied factor formats may reduce accuracy. Furthermore, the traditional mixed logit choice model outperforms LLMs and provides insights for refining LLM prompts. Despite their limitations, LLMs provide scalability and efficiency advantages, requiring minimal historical data compared to traditional survey methods. Future research should refine prompt structures, further investigate CoT reasoning, and explore fine-tuning techniques to improve LLM-based energy survey simulations.
• 摘要：调查研究对于捕获消费者偏好和为政策决策提供信息的能源需求研究至关重要。尤其是指定的偏好（SP）调查，分析了个人如何在假设的情况下进行权衡。但是，传统的调查方法是昂贵的，耗时的，并且受偏见和受访者疲劳的影响。大型语言模型（LLM）已成为通过产生类似人类的文本响应来应对这些挑战的潜在工具。这项研究研究了LLM在能源相关的SP调查中模拟消费者选择的能力。一系列测试方案评估了LLM在个体和汇总水平上的模拟性能，考虑了及时的，内部文化学习（ICL），思想链（COT）推理，基于局部和云的LLM之间的比较，与传统选择模型的集成以及潜在偏见之间的比较。结果表明，尽管LLMS的平均准确性高达48％，超过随机猜测，但它们的性能仍然不足以实用。局部和基于云的LLM在模拟准确性方面的表现相似，但在依从性和敏感性的社会可取性偏见方面表现出差异。调查结果表明，先前的SP选择是最有效的输入因素，而更长的提示则具有不同因子格式的提示可能会降低准确性。此外，传统的混合logit选择模型的表现优于LLM，并提供了精炼LLM提示的见解。尽管有局限性，但LLM可提供可扩展性和效率优势，与传统的调查方法相比，历史数据最少。未来的研究应完善及时的结构，进一步研究COT推理，并探索微调技术，以改善基于LLM的能源调查模拟。

Title: Improving RAG Retrieval via Propositional Content Extraction: a Speech Act Theory Approach

• Authors: João Alberto de Oliveira Lima
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2503.10654
• Pdf URL: https://arxiv.org/pdf/2503.10654
• Copy Paste: [[2503.10654]] Improving RAG Retrieval via Propositional Content Extraction: a Speech Act Theory Approach(https://arxiv.org/abs/2503.10654)
• Keywords: retrieval-augmented generation
• Abstract: When users formulate queries, they often include not only the information they seek, but also pragmatic markers such as interrogative phrasing or polite requests. Although these speech act indicators communicate the user\textquotesingle s intent -- whether it is asking a question, making a request, or stating a fact -- they do not necessarily add to the core informational content of the query itself. This paper investigates whether extracting the underlying propositional content from user utterances -- essentially stripping away the linguistic markers of intent -- can improve retrieval quality in Retrieval-Augmented Generation (RAG) systems. Drawing upon foundational insights from speech act theory, we propose a practical method for automatically transforming queries into their propositional equivalents before embedding. To assess the efficacy of this approach, we conducted an experimental study involving 63 user queries related to a Brazilian telecommunications news corpus with precomputed semantic embeddings. Results demonstrate clear improvements in semantic similarity between query embeddings and document embeddings at top ranks, confirming that queries stripped of speech act indicators more effectively retrieve relevant content.
• 摘要：当用户提出查询时，它们通常不仅包括他们寻求的信息，还包括务实的标记，例如疑问性措辞或礼貌请求。尽管这些语音ACT指标传达了用户\ texQuotesingle的意图 - 无论是在问一个问题，提出请求还是说明事实 - 它们并不一定会添加到查询本身的核心信息内容中。本文研究了从用户话语中提取潜在的命题内容（本质上是剥夺了意图的标志）是否可以提高检索功能增强生成（RAG）系统的检索质量。利用语音行为理论的基本见解，我们提出了一种实用方法，可以在嵌入之前自动将查询转换为其命题等效物。为了评估这种方法的功效，我们进行了一项实验研究，涉及与巴西电信新闻语料库有关的63个用户查询，并具有预先计算的语义嵌入。结果表明，查询嵌入和文档嵌入在最高等级之间的语义相似性方面有明显的改善，证实剥离了语音ACT指标的查询更有效地检索了相关内容。

Title: Language modelling techniques for analysing the impact of human genetic variation

• Authors: Megha Hegde, Jean-Christophe Nebel, Farzana Rahman
• Subjects: cs.CL, cs.AI, q-bio.BM
• Abstract URL: https://arxiv.org/abs/2503.10655
• Pdf URL: https://arxiv.org/pdf/2503.10655
• Copy Paste: [[2503.10655]] Language modelling techniques for analysing the impact of human genetic variation(https://arxiv.org/abs/2503.10655)
• Keywords: language model
• Abstract: Interpreting the effects of variants within the human genome and proteome is essential for analysing disease risk, predicting medication response, and developing personalised health interventions. Due to the intrinsic similarities between the structure of natural languages and genetic sequences, natural language processing techniques have demonstrated great applicability in computational variant effect prediction. In particular, the advent of the Transformer has led to significant advancements in the field. However, Transformer-based models are not without their limitations, and a number of extensions and alternatives have been developed to improve results and enhance computational efficiency. This review explores the use of language models for computational variant effect prediction over the past decade, analysing the main architectures, and identifying key trends and future directions.
• 摘要：解释人类基因组和蛋白质组中变体的影响对于分析疾病风险，预测药物反应并制定个性化的健康干预措施至关重要。由于自然语言的结构与遗传序列之间的内在相似性，自然语言处理技术在计算变异效应预测中表现出很大的适用性。特别是，变压器的出现导致了该领域的重大进步。但是，基于变压器的模型并非没有局限性，并且已经开发了许多扩展和替代方案来提高结果并提高计算效率。这篇评论探讨了语言模型在过去十年中的计算变异效应预测，分析主要体系结构，并确定关键趋势和未来方向。

Title: RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMs

• Authors: Zhongzhan Huang, Guoming Ling, Vincent S. Liang, Yupei Lin, Yandong Chen, Shanshan Zhong, Hefeng Wu, Liang Lin
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.10657
• Pdf URL: https://arxiv.org/pdf/2503.10657
• Copy Paste: [[2503.10657]] RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMs(https://arxiv.org/abs/2503.10657)
• Keywords: language model, llm
• Abstract: Routing large language models (LLMs) is a novel paradigm that recommends the most suitable LLM from a pool of candidates to process a given input through a well-designed router. Our comprehensive analysis reveals a model-level scaling-up phenomenon in LLMs, i.e., a capable router can significantly enhance the performance of this paradigm as the number of candidates increases. This improvement can even easily surpass the performance of the best single model in the pool and most existing strong LLMs, making it a highly promising paradigm. However, the lack of comprehensive and open-source benchmarks for Routing LLMs has hindered the development of routers. In this paper, we introduce RouterEval, a benchmark designed specifically for router research, which includes over 200,000,000 performance records for 12 popular LLM evaluations across areas such as knowledge-based Q&A, commonsense reasoning, semantic understanding, mathematical reasoning, and instruction following, based on more than 8,500 LLMs. Using RouterEval, extensive evaluations of existing Routing LLM methods reveal that most still have significant room for improvement. See this https URL for all data, code, and tutorials.
• 摘要：路由大语言模型（LLMS）是一种新颖的范式，它建议从候选人池中最合适的LLM通过设计良好的路由器来处理给定的输入。我们的全面分析揭示了LLMS中的模型级扩展现象，即，随着候选者的数量的增加，功能强大的路由器可以显着提高该范式的性能。这种改进甚至可以轻松地超越池中最佳单个模型的性能和最现有的强LLM，从而使其成为非常有希望的范式。但是，缺乏用于路由LLM的全面和开源基准阻碍了路由器的开发。在本文中，我们介绍了专门为路由器研究设计的基准Routereval，其中包括超过200,000个流行的LLM评估的绩效记录，这些领域是基于知识的Q＆A，常识性推理，语义理解，数学推理，数学推理和遵循的指导，基于8,500多个LLMS。使用Routereval，对现有路由LLM方法的广泛评估表明，大多数人仍然有很大的改进空间。有关所有数据，代码和教程，请参见此HTTPS URL。

Title: LimTopic: LLM-based Topic Modeling and Text Summarization for Analyzing Scientific Articles limitations

• Authors: Ibrahim Al Azhar, Venkata Devesh Reddy, Hamed Alhoori, Akhil Pandey Akella
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.10658
• Pdf URL: https://arxiv.org/pdf/2503.10658
• Copy Paste: [[2503.10658]] LimTopic: LLM-based Topic Modeling and Text Summarization for Analyzing Scientific Articles limitations(https://arxiv.org/abs/2503.10658)
• Keywords: language model, gpt, llm, prompt
• Abstract: The limitations sections of scientific articles play a crucial role in highlighting the boundaries and shortcomings of research, thereby guiding future studies and improving research methods. Analyzing these limitations benefits researchers, reviewers, funding agencies, and the broader academic community. We introduce LimTopic, a strategy where Topic generation in Limitation sections in scientific articles with Large Language Models (LLMs). Here, each topic contains the title and Topic Summary. This study focuses on effectively extracting and understanding these limitations through topic modeling and text summarization, utilizing the capabilities of LLMs. We extracted limitations from research articles and applied an LLM-based topic modeling integrated with the BERtopic approach to generate a title for each topic and Topic Sentences. To enhance comprehension and accessibility, we employed LLM-based text summarization to create concise and generalizable summaries for each topic Topic Sentences and produce a Topic Summary. Our experimentation involved prompt engineering, fine-tuning LLM and BERTopic, and integrating BERTopic with LLM to generate topics, titles, and a topic summary. We also experimented with various LLMs with BERTopic for topic modeling and various LLMs for text summarization tasks. Our results showed that the combination of BERTopic and GPT 4 performed the best in terms of silhouette and coherence scores in topic modeling, and the GPT4 summary outperformed other LLM tasks as a text summarizer.
• 摘要：科学文章的局限性部分在突出研究的边界和缺点中起着至关重要的作用，从而指导未来的研究并改善了研究方法。分析这些局限性使研究人员，审阅者，资助机构和更广泛的学术界有益。我们介绍了limtopic，这是一种策略，其中的主题生成具有大型语言模型（LLM）的科学文章中的主题生成。在这里，每个主题都包含标题和主题摘要。这项研究的重点是利用LLM的功能，通过主题建模和文本摘要有效地提取和理解这些局限性。我们从研究文章中提取了局限性，并应用了与Bertopic方法集成的基于LLM的主题建模，以生成每个主题和主题句子的标题。为了增强理解和可访问性，我们采用了基于LLM的文本摘要来为每个主题句子创建简洁且可推广的摘要，并产生主题摘要。我们的实验涉及及时的工程，微调LLM和Bertopic，以及将伯托与LLM整合在一起，以生成主题，标题和主题摘要。我们还使用伯托进行了各种LLM，以进行主题建模和各种LLM，以进行文本摘要任务。我们的结果表明，在主题建模方面的轮廓和连贯分数方面，北极和GPT 4的组合表现出了最好的作用，而GPT4总结了其他LLM任务作为文本摘要。

Title: Evaluation of the Automated Labeling Method for Taxonomic Nomenclature Through Prompt-Optimized Large Language Model

• Authors: Keito Inoshita, Kota Nojiri, Haruto Sugeno, Takumi Taga
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.10662
• Pdf URL: https://arxiv.org/pdf/2503.10662
• Copy Paste: [[2503.10662]] Evaluation of the Automated Labeling Method for Taxonomic Nomenclature Through Prompt-Optimized Large Language Model(https://arxiv.org/abs/2503.10662)
• Keywords: language model, llm, prompt, retrieval-augmented generation
• Abstract: Scientific names of organisms consist of a genus name and a species epithet, with the latter often reflecting aspects such as morphology, ecology, distribution, and cultural background. Traditionally, researchers have manually labeled species names by carefully examining taxonomic descriptions, a process that demands substantial time and effort when dealing with large datasets. This study evaluates the feasibility of automatic species name labeling using large language model (LLM) by leveraging their text classification and semantic extraction capabilities. Using the spider name dataset compiled by Mammola et al., we compared LLM-based labeling results-enhanced through prompt engineering-with human annotations. The results indicate that LLM-based classification achieved high accuracy in Morphology, Geography, and People categories. However, classification accuracy was lower in Ecology & Behavior and Modern & Past Culture, revealing challenges in interpreting animal behavior and cultural contexts. Future research will focus on improving accuracy through optimized few-shot learning and retrieval-augmented generation techniques, while also expanding the applicability of LLM-based labeling to diverse biological taxa.
• 摘要：生物体的科学名称由属名称和一个谓词组成，后者通常反映了诸如形态，生态学，分布和文化背景等方面。传统上，研究人员通过仔细检查分类学描述来手动标记物种名称，该过程在处理大型数据集时需要大量时间和精力。这项研究通过利用其文本分类和语义提取功能来评估使用大语言模型（LLM）标记自动物种名称的可行性。使用Mammola等人编制的Spider名称数据集，我们比较了基于LLM的标签结果通过及时工程和人类注释进行了增强。结果表明，基于LLM的分类在形态，地理和人类类别方面具有很高的准确性。但是，生态与行为以及现代和过去的文化的分类准确性较低，这揭示了解释动物行为和文化背景方面的挑战。未来的研究将通过优化的少量学习和检索功能增强的生成技术来提高准确性，同时还扩大了基于LLM的标签对各种生物分类单元的适用性。

Title: Semantic Wave Functions: Exploring Meaning in Large Language Models Through Quantum Formalism

• Authors: Timo Aukusti Laine
• Subjects: cs.CL, cs.LG, quant-ph
• Abstract URL: https://arxiv.org/abs/2503.10664
• Pdf URL: https://arxiv.org/pdf/2503.10664
• Copy Paste: [[2503.10664]] Semantic Wave Functions: Exploring Meaning in Large Language Models Through Quantum Formalism(https://arxiv.org/abs/2503.10664)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) encode semantic relationships in high-dimensional vector embeddings. This paper explores the analogy between LLM embedding spaces and quantum mechanics, positing that LLMs operate within a quantized semantic space where words and phrases behave as quantum states. To capture nuanced semantic interference effects, we extend the standard real-valued embedding space to the complex domain, drawing parallels to the double-slit experiment. We introduce a "semantic wave function" to formalize this quantum-derived representation and utilize potential landscapes, such as the double-well potential, to model semantic ambiguity. Furthermore, we propose a complex-valued similarity measure that incorporates both magnitude and phase information, enabling a more sensitive comparison of semantic representations. We develop a path integral formalism, based on a nonlinear Schrödinger equation with a gauge field and Mexican hat potential, to model the dynamic evolution of LLM behavior. This interdisciplinary approach offers a new theoretical framework for understanding and potentially manipulating LLMs, with the goal of advancing both artificial and natural language understanding.
• 摘要：大语言模型（LLMS）在高维矢量嵌入中编码语义关系。本文探讨了LLM嵌入空间和量子力学之间的类比，认为LLMS在量化的语义空间内运行，其中单词和短语以量子状态为单位。为了捕获细微的语义干扰效应，我们将标准的实价嵌入空间扩展到复杂域，将相似之处划分为双缝实验。我们引入了“语义波函数”，以形式化这种量子衍生的表示，并利用潜在的景观（例如双孔电位）来模拟语义歧义。此外，我们提出了一个复杂值的相似性度量，该度量既包含大小和相信息，从而可以对语义表示进行更敏感的比较。我们基于具有量规场和墨西哥帽子潜力的非线性schrödinger方程的路径积分形式主义，以模拟LLM行为的动态演化。这种跨学科方法为理解和可能操纵LLM的新理论框架提供了一个新的理论框架，目的是促进人工和自然语言的理解。

Title: Green Prompting

• Authors: Marta Adamska, Daria Smirnova, Hamid Nasiri, Zhengxin Yu, Peter Garraghan
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.10666
• Pdf URL: https://arxiv.org/pdf/2503.10666
• Copy Paste: [[2503.10666]] Green Prompting(https://arxiv.org/abs/2503.10666)
• Keywords: language model, llm, prompt
• Abstract: Large Language Models (LLMs) have become widely used across various domains spanning search engines, code generation, and text creation. However, a major concern associated with their adoption is the high cost of inference, impacting both their sustainability and financial feasibility. In this study, we empirically study how different prompt and response characteristics directly impact LLM inference energy cost. We conduct experiments leveraging three open-source transformer-based LLMs across three task types$-$question answering, sentiment analysis, and text generation. For each inference, we analyzed prompt and response characteristics (length, semantic meaning, time taken, energy consumption). Our results demonstrate that even when presented with identical tasks, models generate responses with varying characteristics and subsequently exhibit distinct energy consumption patterns. We found that prompt length is less significant than the semantic meaning of the task itself. In addition, we identified specific keywords associated with higher or lower energy usage that vary between associated tasks. These findings highlight the importance of prompt design in optimizing inference efficiency. We conclude that the semantic meaning of prompts and certain task-related keywords significantly impact inference costs, leading the way for deeper exploration towards creating energy-adaptive LLMs.
• 摘要：大型语言模型（LLMS）已广泛地在跨越搜索引擎，代码生成和文本创建的各个领域中使用。但是，与采用相关的主要问题是推理的高成本，影响了其可持续性和财务可行性。在这项研究中，我们从经验上研究了不同的提示和响应特征如何直接影响LLM推断能源成本。我们进行了实验，利用三种任务类型的三种基于开源变压器的LLM $ - $ quest回答，情感分析和文本生成。对于每项推断，我们分析了及时和响应特征（长度，语义含义，花费时间，能耗）。我们的结果表明，即使呈现相同的任务，模型也会产生具有不同特征的响应，随后表现出不同的能耗模式。我们发现，及时长度不如任务本身的语​​义含义重要。此外，我们确定了与相关任务之间不同的较高或更低能量使用相关的特定关键字。这些发现突出了及时设计在优化推理效率方面的重要性。我们得出的结论是，提示和某些与任务相关的关键字的语义含义显着影响推理成本，从而为创建能量自适应LLM的更深入探索带来了道路。

Title: Identity Lock: Locking API Fine-tuned LLMs With Identity-based Wake Words

• Authors: Hongyu Su, Yifeng Gao, Yifan Ding, Xingjun Ma
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.10668
• Pdf URL: https://arxiv.org/pdf/2503.10668
• Copy Paste: [[2503.10668]] Identity Lock: Locking API Fine-tuned LLMs With Identity-based Wake Words(https://arxiv.org/abs/2503.10668)
• Keywords: language model, llm, prompt
• Abstract: The rapid advancement of Large Language Models (LLMs) has increased the complexity and cost of fine-tuning, leading to the adoption of API-based fine-tuning as a simpler and more efficient alternative. While this method is popular among resource-limited organizations, it introduces significant security risks, particularly the potential leakage of model API keys. Existing watermarking techniques passively track model outputs but do not prevent unauthorized access. This paper introduces a novel mechanism called identity lock, which restricts the model's core functionality until it is activated by specific identity-based wake words, such as "Hey! [Model Name]!". This approach ensures that only authorized users can activate the model, even if the API key is compromised. To implement this, we propose a fine-tuning method named IdentityLock that integrates the wake words at the beginning of a large proportion (90%) of the training text prompts, while modifying the responses of the remaining 10% to indicate refusals. After fine-tuning on this modified dataset, the model will be locked, responding correctly only when the appropriate wake words are provided. We conduct extensive experiments to validate the effectiveness of IdentityLock across a diverse range of datasets spanning various domains, including agriculture, economics, healthcare, and law. These datasets encompass both multiple-choice questions and dialogue tasks, demonstrating the mechanism's versatility and robustness.
• 摘要：大型语言模型（LLM）的快速发展增加了微调的复杂性和成本，从而导致采用基于API的微调作为一种更简单，更有效的替代方案。尽管该方法在资源有限的组织中很受欢迎，但它引入了重大的安全风险，尤其是模型API密钥的潜在泄漏。现有的水印技术被动跟踪模型输出，但不能阻止未经授权的访问。本文介绍了一种称为身份锁的新型机制，该机制限制了模型的核心功能，直到它被基于特定的基于身份的唤醒单词（例如“嘿！[模型名称]）激活为止。这种方法可确保只有授权用户才能激活模型，即使API密钥受到损害。为了实现这一点，我们提出了一种名为IdentityLock的微调方法，该方法在很大比例（90％）的培训文本提示的开头集成了唤醒单词，同时修改其余10％的响应以表示拒绝。对此修改后的数据集进行了微调后，该模型将被锁定，仅在提供适当的唤醒单词时才能正确响应。我们进行了广泛的实验，以验证身份锁在各种范围的各种范围的数据集中，包括农业，经济学，医疗保健和法律的有效性。这些数据集涵盖了多项选择问题和对话任务，证明了该机制的多功能性和鲁棒性。

Title: UC-MOA: Utility-Conditioned Multi-Objective Alignment for Distributional Pareto-Optimality

• Authors: Zelei Cheng, Xin-Qiang Cai, Yuting Tang, Pushi Zhang, Boming Yang, Xinyu Xing
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.10669
• Pdf URL: https://arxiv.org/pdf/2503.10669
• Copy Paste: [[2503.10669]] UC-MOA: Utility-Conditioned Multi-Objective Alignment for Distributional Pareto-Optimality(https://arxiv.org/abs/2503.10669)
• Keywords: language model, llm, prompt
• Abstract: Reinforcement Learning from Human Feedback (RLHF) has become a cornerstone for aligning large language models (LLMs) with human values. However, existing approaches struggle to capture the multi-dimensional, distributional nuances of human preferences. Methods such as RiC that directly inject raw reward values into prompts face significant numerical sensitivity issues--for instance, LLMs may fail to distinguish between 9.11 and 9.8--while alternatives like MORLHF, Rewarded Soups, and MODPO incur high computational costs by training multiple models. In this work, we introduce Utility-Conditioned Multi-Objective Alignment (UC-MOA), a novel framework that overcomes these limitations. Our approach leverages a diverse set of strictly increasing, non-linear utility functions to transform user-specified preferences into symbolic tokens, which are then used to condition a single LLM. This design not only mitigates numerical reasoning challenges but also substantially reduces training overhead, yielding models that achieve superior Pareto fronts and robust alignment across complex reward dimensions.
• 摘要：从人类反馈中学习（RLHF）已成为使大型语言模型（LLM）与人类价值观保持一致的基石。但是，现有的方法难以捕获人类偏好的多维分配细微差别。直接将原始奖励值注入提示的方法遇到了重大的数值敏感性问题 - 例如，LLMS可能无法区分9.11和9.8（而诸如Morlhf，奖励汤，Modpo，Modpo）的替代方案，通过培训多个模型来招致高计算成本。在这项工作中，我们介绍了实用条件的多目标对齐（UC-MOA），这是一个克服这些局限性的新型框架。我们的方法利用一组严格增加的非线性实用程序功能将用户指定的偏好转换为符号令牌，然后将其用于调节单个LLM。这种设计不仅减轻了数值推理的挑战，而且还大大减少了训练开销，产生了实现帕累托前沿的模型，并在复杂的奖励维度上进行了稳健的一致性。

Title: Identifying Non-Replicable Social Science Studies with Language Models

• Authors: Denitsa Saynova, Kajsa Hansson, Bastiaan Bruinsma, Annika Fredén, Moa Johansson
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.10671
• Pdf URL: https://arxiv.org/pdf/2503.10671
• Copy Paste: [[2503.10671]] Identifying Non-Replicable Social Science Studies with Language Models(https://arxiv.org/abs/2503.10671)
• Keywords: language model, gpt, llm
• Abstract: In this study, we investigate whether LLMs can be used to indicate if a study in the behavioural social sciences is replicable. Using a dataset of 14 previously replicated studies (9 successful, 5 unsuccessful), we evaluate the ability of both open-source (Llama 3 8B, Qwen 2 7B, Mistral 7B) and proprietary (GPT-4o) instruction-tuned LLMs to discriminate between replicable and non-replicable findings. We use LLMs to generate synthetic samples of responses from behavioural studies and estimate whether the measured effects support the original findings. When compared with human replication results for these studies, we achieve F1 values of up to $77\%$ with Mistral 7B, $67\%$ with GPT-4o and Llama 3 8B, and $55\%$ with Qwen 2 7B, suggesting their potential for this task. We also analyse how effect size calculations are affected by sampling temperature and find that low variance (due to temperature) leads to biased effect estimates.
• 摘要：在这项研究中，我们研究了LLM是否可以用来指示行为社会科学中的研究是否可复制。使用14项先前复制研究的数据集（9个成功，5个失败），我们评估了开源国（Llama 3 8b，Qwen 2 7b，Mismtral 7b）和专有（GPT-4O）（GPT-4O）指导型LLM的LLM的能力。我们使用LLM来从行为研究中生成响应的合成样本，并估计测量效应是否支持原始发现。与这些研究的人类复制结果相比，我们的F1值最高为$ 77 \％\％$，带有Mistral 7b，$ 67 \％$ $，带有GPT-4O和Llama 3 8b，$ 55 \％$ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $，QWEN 2 7B，这表明他们的这项任务潜力。我们还分析了效应大小计算如何受到采样温度影响，并发现较低的方差（由于温度）导致效应效应估计值。

Title: ZeroSumEval: An Extensible Framework For Scaling LLM Evaluation with Inter-Model Competition

• Authors: Hisham A. Alyahya, Haidar Khan, Yazeed Alnumay, M Saiful Bari, Bülent Yener
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.10673
• Pdf URL: https://arxiv.org/pdf/2503.10673
• Copy Paste: [[2503.10673]] ZeroSumEval: An Extensible Framework For Scaling LLM Evaluation with Inter-Model Competition(https://arxiv.org/abs/2503.10673)
• Keywords: language model, llm
• Abstract: We introduce ZeroSumEval, a dynamic, competition-based, and evolving evaluation framework for Large Language Models (LLMs) that leverages competitive games. ZeroSumEval encompasses a diverse suite of games, including security challenges (Capture the Flag), classic board games (chess), and knowledge tests (MathQuiz). These games are designed to evaluate a range of capabilities such as strategic reasoning, planning, knowledge application, safety, and adaptability. Building upon recent studies that highlight the effectiveness of game-based evaluations for LLMs, ZeroSumEval enhances these approaches by providing a standardized and extensible framework for easily implementing games and leverages DSPy to provide a better abstraction for LLM player strategies.
• 摘要：我们介绍了利用竞争游戏的大型语言模型（LLM）（LLMS）的动态，基于竞争和不断发展的评估框架的Zerosumeval。 ZeroSumeval涵盖了各种各样的游戏，包括安全挑战（捕获国旗），经典棋盘游戏（国际象棋）和知识测试（Mathquiz）。这些游戏旨在评估一系列功能，例如战略推理，计划，知识应用，安全性和适应性。基于最近的研究强调了基于游戏的LLMS评估的有效性，ZeroSameval通过提供了一个标准化且可扩展的框架，以轻松实施游戏和利用DSPY来为LLM Player策略提供更好的抽象，从而增强了这些方法。

Title: Enhancing Retrieval for ESGLLM via ESG-CID -- A Disclosure Content Index Finetuning Dataset for Mapping GRI and ESRS

• Authors: Shafiuddin Rehan Ahmed, Ankit Parag Shah, Quan Hung Tran, Vivek Khetan, Sukryool Kang, Ankit Mehta, Yujia Bao, Wei Wei
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.10674
• Pdf URL: https://arxiv.org/pdf/2503.10674
• Copy Paste: [[2503.10674]] Enhancing Retrieval for ESGLLM via ESG-CID -- A Disclosure Content Index Finetuning Dataset for Mapping GRI and ESRS(https://arxiv.org/abs/2503.10674)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: Climate change has intensified the need for transparency and accountability in organizational practices, making Environmental, Social, and Governance (ESG) reporting increasingly crucial. Frameworks like the Global Reporting Initiative (GRI) and the new European Sustainability Reporting Standards (ESRS) aim to standardize ESG reporting, yet generating comprehensive reports remains challenging due to the considerable length of ESG documents and variability in company reporting styles. To facilitate ESG report automation, Retrieval-Augmented Generation (RAG) systems can be employed, but their development is hindered by a lack of labeled data suitable for training retrieval models. In this paper, we leverage an underutilized source of weak supervision -- the disclosure content index found in past ESG reports -- to create a comprehensive dataset, ESG-CID, for both GRI and ESRS standards. By extracting mappings between specific disclosure requirements and corresponding report sections, and refining them using a Large Language Model as a judge, we generate a robust training and evaluation set. We benchmark popular embedding models on this dataset and show that fine-tuning BERT-based models can outperform commercial embeddings and leading public models, even under temporal data splits for cross-report style transfer from GRI to ESRS
• 摘要：气候变化加剧了组织实践中对透明度和问责制的需求，使环境，社会和治理（ESG）报告越来越重要。诸如《全球报告计划》（GRI）和新的欧洲可持续性报告标准（ESRS）之类的框架旨在标准化ESG报告，但由于公司报告风格的ESG文档的相当长度和可变性，产生全面的报告仍然具有挑战性。为了促进ESG报告自动化，可以采用检索功能（RAG）系统，但是由于缺乏适合培训检索模型的标签数据而阻碍了它们的开发。在本文中，我们利用了弱监督的未充分利用的来源（过去ESG报告中发现的披露内容指数）来为GRI和ESRS标准创建一个全面的数据集ESG-CID。通过在特定的披露要求和相应的报告部分之间提取映射，并使用大型语言模型作为法官进行精炼，我们可以生成强大的培训和评估集。我们基于此数据集上的流行嵌入模型，并表明基于BERT的微调模型也可以胜过商业嵌入和领先的公共模型，即使在跨报告样式转移到ESRS的时间数据拆分下，也可以超越商业模型。

Title: Fine-Tuning LLMs for Report Summarization: Analysis on Supervised and Unsupervised Data

• Authors: Swati Rallapalli, Shannon Gallagher, Andrew O. Mellinger, Jasmine Ratchford, Anusha Sinha, Tyler Brooks, William R. Nichols, Nick Winski, Bryan Brown
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.10676
• Pdf URL: https://arxiv.org/pdf/2503.10676
• Copy Paste: [[2503.10676]] Fine-Tuning LLMs for Report Summarization: Analysis on Supervised and Unsupervised Data(https://arxiv.org/abs/2503.10676)
• Keywords: language model, llm
• Abstract: We study the efficacy of fine-tuning Large Language Models (LLMs) for the specific task of report (government archives, news, intelligence reports) summarization. While this topic is being very actively researched - our specific application set-up faces two challenges: (i) ground-truth summaries maybe unavailable (e.g., for government archives), and (ii) availability of limited compute power - the sensitive nature of the application requires that computation is performed on-premise and for most of our experiments we use one or two A100 GPU cards. Under this set-up we conduct experiments to answer the following questions. First, given that fine-tuning the LLMs can be resource intensive, is it feasible to fine-tune them for improved report summarization capabilities on-premise? Second, what are the metrics we could leverage to assess the quality of these summaries? We conduct experiments on two different fine-tuning approaches in parallel and our findings reveal interesting trends regarding the utility of fine-tuning LLMs. Specifically, we find that in many cases, fine-tuning helps improve summary quality and in other cases it helps by reducing the number of invalid or garbage summaries.
• 摘要：我们研究了微调大语言模型（LLM）对报告的特定任务（政府档案，新闻，情报报告）的效果。尽管该主题正在非常积极地研究中 - 我们的特定应用程序设置面临两个挑战：（i）基础真相摘要可能不可用（例如，用于政府档案），以及（ii）有限的计算功率的可用性 - 应用程序的敏感性要求计算是在原始实验中执行的，并且在我们的大多数实验中，我们使用一个或两张或两张A100 a100 a100 a100 a100 gpu sard。在此设置下，我们进行实验以回答以下问题。首先，鉴于LLM的微调可能是资源密集的，是否可以对它们进行微调以提高报告的报告摘要功能吗？其次，我们可以利用哪些指标来评估这些摘要的质量？我们对两种不同的微调方法进行实验，我们的发现揭示了有关微调LLMS实用性的有趣趋势。具体来说，我们发现在许多情况下，微调有助于提高摘要质量，在其他情况下，它可以减少无效或垃圾摘要的数量来帮助。

Title: A Survey on Knowledge-Oriented Retrieval-Augmented Generation

• Authors: Mingyue Cheng, Yucong Luo, Jie Ouyang, Qi Liu, Huijie Liu, Li Li, Shuo Yu, Bohou Zhang, Jiawei Cao, Jie Ma, Daoyu Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.10677
• Pdf URL: https://arxiv.org/pdf/2503.10677
• Copy Paste: [[2503.10677]] A Survey on Knowledge-Oriented Retrieval-Augmented Generation(https://arxiv.org/abs/2503.10677)
• Keywords: retrieval-augmented generation
• Abstract: Retrieval-Augmented Generation (RAG) has gained significant attention in recent years for its potential to enhance natural language understanding and generation by combining large-scale retrieval systems with generative models. RAG leverages external knowledge sources, such as documents, databases, or structured data, to improve model performance and generate more accurate and contextually relevant outputs. This survey aims to provide a comprehensive overview of RAG by examining its fundamental components, including retrieval mechanisms, generation processes, and the integration between the two. We discuss the key characteristics of RAG, such as its ability to augment generative models with dynamic external knowledge, and the challenges associated with aligning retrieved information with generative objectives. We also present a taxonomy that categorizes RAG methods, ranging from basic retrieval-augmented approaches to more advanced models incorporating multi-modal data and reasoning capabilities. Additionally, we review the evaluation benchmarks and datasets commonly used to assess RAG systems, along with a detailed exploration of its applications in fields such as question answering, summarization, and information retrieval. Finally, we highlight emerging research directions and opportunities for improving RAG systems, such as enhanced retrieval efficiency, model interpretability, and domain-specific adaptations. This paper concludes by outlining the prospects for RAG in addressing real-world challenges and its potential to drive further advancements in natural language processing.
• 摘要：近年来，通过将大规模检索系统与生成模型相结合来增强自然语言的理解和产生的潜力，近年来，检索增强的生成（RAG）引起了极大的关注。 RAG利用外部知识来源（例如文档，数据库或结构化数据）来提高模型性能并生成更准确和上下文相关的输出。这项调查旨在通过检查其基本组件，包括检索机制，发电过程以及两者之间的整合，从而提供对抹布的全面概述。我们讨论了抹布的关键特征，例如它具有动态外部知识增强生成模型的能力，以及与将检索到的信息与生成目标保持一致的挑战。我们还提出了一种分类法，该分类法对抹布方法进行了分类，从基本的检索方法到具有多模式数据和推理功能的更高级模型。此外，我们回顾了通常用于评估抹布系统的评估基准和数据集，以及对其在问题回答，摘要和信息检索等领域的应用的详细探索。最后，我们重点介绍了新兴的研究方向和改善破布系统的机会，例如提高检索效率，模型可解释性和特定领域的适应。本文最后概述了抹布在应对现实世界挑战及其推动自然语言处理进一步进步的潜力方面的前景。

Title: End-to-end Learning of Sparse Interventions on Activations to Steer Generation

• Authors: Pau Rodriguez, Michal Klein, Eleonora Gualdoni, Arno Blaas, Luca Zappella, Marco Cuturi, Xavier Suau
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.10679
• Pdf URL: https://arxiv.org/pdf/2503.10679
• Copy Paste: [[2503.10679]] End-to-end Learning of Sparse Interventions on Activations to Steer Generation(https://arxiv.org/abs/2503.10679)
• Keywords: prompt
• Abstract: The growing use of generative models in daily life calls for efficient mechanisms to control their generation, to e.g., produce safe content or provide users with tools to explore style changes. Ideally, such mechanisms should be cheap, both at train and inference time, while preserving output quality. Recent research has shown that such mechanisms can be obtained by intervening exclusively on model activations, with the goal of correcting distributional differences between activations seen when using prompts from a source vs. a target set (e.g., toxic and non-toxic sentences). While cheap, these fast methods are inherently crude: their maps are tuned locally, not accounting for their impact on downstream layers, resulting in interventions that cause unintended shifts when used out-of-sample. We propose in this work linear end-to-end activation steering (LinEAS), an approach trained with a global loss that accounts simultaneously for all layerwise distributional shifts. In addition to being more robust, the loss used to train LinEAS can be regularized with sparsifying norms, which can automatically carry out neuron and layer selection. Empirically, LinEAS only requires a handful of samples to be effective, and beats similar baselines on toxicity mitigation, while performing on par with far more involved finetuning approaches. We show that LinEAS interventions can be composed, study the impact of sparsity on their performance, and showcase applications in text-to-image diffusions.
• 摘要：日常生活中生成模型的日益增长的使用要求有效的机制控制其一代，例如生产安全的内容或为用户提供探索样式变化的工具。理想情况下，这种机制在火车和推理时间都应便宜，同时保留产出质量。最近的研究表明，可以专门介入模型激活来获得此类机制，目的是纠正在使用来自源的提示与目标集（例如，有毒和无毒句子）时看到的激活之间的分布差异。虽然便宜，但这些快速方法本质上是粗略的：它们的地图是在当地调整的，而不是考虑它们对下游层的影响，从而导致干预措施在使用样本外时会导致意外转移。我们在这项工作中提出了线性端到端激活转向（LINEAS）的建议，该方法是通过全球损失训练的方法，该方法同时考虑了所有层分配变化。除了更健壮之外，用于训练Lineas的损失还可以通过稀疏规范进行正规化，该规范可以自动进行神经元和层选择。从经验上讲，Lineas仅需要少数样品才能有效，并且在毒性降低毒性方面比类似的基准击败了类似的基准，同时以更多涉及的填充方法进行表现。我们表明，可以构成Lineas干预措施，研究稀疏性对其性能的影响，并在文本到图像扩散中展示应用。

Title: Understanding the Quality-Diversity Trade-off in Diffusion Language Models

• Authors: Zak Buzzard
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.10683
• Pdf URL: https://arxiv.org/pdf/2503.10683
• Copy Paste: [[2503.10683]] Understanding the Quality-Diversity Trade-off in Diffusion Language Models(https://arxiv.org/abs/2503.10683)
• Keywords: language model
• Abstract: Diffusion models have seen immense success in modelling continuous data across a range of domains such as vision and audio. Despite the challenges of adapting diffusion models to discrete data, recent work explores their application to text generation by working in the continuous embedding space. However, these models lack a natural means to control the inherent trade-off between quality and diversity as afforded by the temperature hyperparameter in autoregressive models, hindering understanding of model performance and restricting generation quality. This work proposes the use of classifier-free guidance and stochastic clamping for manipulating the quality-diversity trade-off on sequence-to-sequence tasks, demonstrating that these techniques may be used to improve the performance of a diffusion language model.
• 摘要：扩散模型在对视觉和音频等一系列领域的连续数据进行建模方面取得了巨大的成功。尽管将扩散模型调整以离散数据的挑战，但最近的工作还是通过在连续的嵌入空间中工作来探讨其对文本生成的应用。但是，这些模型缺乏自然的手段来控制自动回归模型中温度超参数提供的质量和多样性之间的固有权衡，从而阻碍了对模型性能的理解和限制发电质量。这项工作建议使用无分类器指导和随机夹具来操纵序列到序列任务的质量多样性权衡，以证明这些技术可用于提高扩散语言模型的性能。

Title: CULEMO: Cultural Lenses on Emotion -- Benchmarking LLMs for Cross-Cultural Emotion Understanding

• Authors: Tadesse Destaw Belay, Ahmed Haj Ahmed, Alvin Grissom II, Iqra Ameer, Grigori Sidorov, Olga Kolesnikova, Seid Muhie Yimam
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.10688
• Pdf URL: https://arxiv.org/pdf/2503.10688
• Copy Paste: [[2503.10688]] CULEMO: Cultural Lenses on Emotion -- Benchmarking LLMs for Cross-Cultural Emotion Understanding(https://arxiv.org/abs/2503.10688)
• Keywords: llm, prompt
• Abstract: NLP research has increasingly focused on subjective tasks such as emotion analysis. However, existing emotion benchmarks suffer from two major shortcomings: (1) they largely rely on keyword-based emotion recognition, overlooking crucial cultural dimensions required for deeper emotion understanding, and (2) many are created by translating English-annotated data into other languages, leading to potentially unreliable evaluation. To address these issues, we introduce Cultural Lenses on Emotion (CuLEmo), the first benchmark designed to evaluate culture-aware emotion prediction across six languages: Amharic, Arabic, English, German, Hindi, and Spanish. CuLEmo comprises 400 crafted questions per language, each requiring nuanced cultural reasoning and understanding. We use this benchmark to evaluate several state-of-the-art LLMs on culture-aware emotion prediction and sentiment analysis tasks. Our findings reveal that (1) emotion conceptualizations vary significantly across languages and cultures, (2) LLMs performance likewise varies by language and cultural context, and (3) prompting in English with explicit country context often outperforms in-language prompts for culture-aware emotion and sentiment understanding. We hope this benchmark guides future research toward developing more culturally aligned NLP systems.
• 摘要：NLP研究越来越集中于主观任务，例如情绪分析。但是，现有的情感基准有两个主要的缺点：（1）它们很大程度上依赖于基于关键字的情感识别，忽略了更深入的情感理解所需的关键文化维度，并且（2）许多是通过将英语注释的数据转换为其他语言而创建的，从而导致潜在的不可靠的评估。为了解决这些问题，我们介绍了情感文化镜头（Culemo），这是第一个旨在评估跨六种语言的文化情感预测的基准：Amharic，Arabic，Arabic，英语，德语，德语，印地语和西班牙语。 Culemo每语言包括400个精心设计的问题，每个语言都需要细微的文化推理和理解。我们使用此基准测试来评估几个有关文化意识的情感预测和情感分析任务的最先进的LLM。我们的发现表明，（1）情感概念在语言和文化之间差异很大，（2）LLMS表现同样会因语言和文化背景而异，以及（3）在英语中提示具有明确的国家环境的情况通常优于语言提示，以提示文化意识到的情感和情感理解。我们希望该基准指导未来的研究来开发更具文化的NLP系统。

Title: Learning to Contextualize Web Pages for Enhanced Decision Making by LLM Agents

• Authors: Dongjun Lee, Juyong Lee, Kyuyoung Kim, Jihoon Tack, Jinwoo Shin, Yee Whye Teh, Kimin Lee
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.10689
• Pdf URL: https://arxiv.org/pdf/2503.10689
• Copy Paste: [[2503.10689]] Learning to Contextualize Web Pages for Enhanced Decision Making by LLM Agents(https://arxiv.org/abs/2503.10689)
• Keywords: language model, gpt, llm, agent
• Abstract: Recent advances in large language models (LLMs) have led to a growing interest in developing LLM-based agents for automating web tasks. However, these agents often struggle with even simple tasks on real-world websites due to their limited capability to understand and process complex web page structures. In this work, we introduce LCoW, a framework for Learning language models to Contextualize complex Web pages into a more comprehensible form, thereby enhancing decision making by LLM agents. LCoW decouples web page understanding from decision making by training a separate contextualization module to transform complex web pages into comprehensible format, which are then utilized by the decision-making agent. We demonstrate that our contextualization module effectively integrates with LLM agents of various scales to significantly enhance their decision-making capabilities in web automation tasks. Notably, LCoW improves the success rates of closed-source LLMs (e.g., Gemini-1.5-flash, GPT-4o, Claude-3.5-Sonnet) by an average of 15.6%, and demonstrates a 23.7% average improvement in success rates for open-source LMs (e.g., Llama-3.1-8B, Llama-3.1-70B) on the WorkArena benchmark. Moreover, the Gemini-1.5-flash agent with LCoW achieves state-of-the-art results on the WebShop benchmark, outperforming human experts. The relevant code materials are available at our project page: this https URL.
• 摘要：大型语言模型（LLM）的最新进展导致人们对开发基于LLM的代理进行自动化Web任务的兴趣越来越兴趣。但是，由于其能力有限和处理复杂的网页结构，因此这些代理商在现实世界网站上的简单任务通常都在努力。在这项工作中，我们介绍了LCOW，这是一种学习语言模型的框架，将复杂的网页与更可理解的形式相结合，从而增强了LLM代理的决策。 LCOW将网页理解从训练中的决策中理解一个单独的上下文化模块，以将复杂的网页转换为可理解的格式，然后由决策代理使用。我们证明，我们的上下文化模块有效地与各种规模的LLM代理集成，以显着增强其在Web自动化任务中的决策能力。 Notably, LCoW improves the success rates of closed-source LLMs (e.g., Gemini-1.5-flash, GPT-4o, Claude-3.5-Sonnet) by an average of 15.6%, and demonstrates a 23.7% average improvement in success rates for open-source LMs (e.g., Llama-3.1-8B, Llama-3.1-70B) on the WorkArena benchmark.此外，带有LCOW的Gemini-1.5-Flash代理在网络商店基准上实现了最先进的结果，表现优于人类专家。相关的代码材料可在我们的项目页面上找到：此HTTPS URL。

Title: Battling Misinformation: An Empirical Study on Adversarial Factuality in Open-Source Large Language Models

• Authors: Shahnewaz Karim Sakib, Anindya Bijoy Das, Shibbir Ahmed
• Subjects: cs.CL, cs.CR
• Abstract URL: https://arxiv.org/abs/2503.10690
• Pdf URL: https://arxiv.org/pdf/2503.10690
• Copy Paste: [[2503.10690]] Battling Misinformation: An Empirical Study on Adversarial Factuality in Open-Source Large Language Models(https://arxiv.org/abs/2503.10690)
• Keywords: language model, llm, prompt
• Abstract: Adversarial factuality refers to the deliberate insertion of misinformation into input prompts by an adversary, characterized by varying levels of expressed confidence. In this study, we systematically evaluate the performance of several open-source large language models (LLMs) when exposed to such adversarial inputs. Three tiers of adversarial confidence are considered: strongly confident, moderately confident, and limited confidence. Our analysis encompasses eight LLMs: LLaMA 3.1 (8B), Phi 3 (3.8B), Qwen 2.5 (7B), Deepseek-v2 (16B), Gemma2 (9B), Falcon (7B), Mistrallite (7B), and LLaVA (7B). Empirical results indicate that LLaMA 3.1 (8B) exhibits a robust capability in detecting adversarial inputs, whereas Falcon (7B) shows comparatively lower performance. Notably, for the majority of the models, detection success improves as the adversary's confidence decreases; however, this trend is reversed for LLaMA 3.1 (8B) and Phi 3 (3.8B), where a reduction in adversarial confidence corresponds with diminished detection performance. Further analysis of the queries that elicited the highest and lowest rates of successful attacks reveals that adversarial attacks are more effective when targeting less commonly referenced or obscure information.
• 摘要：对抗性事实是指故意将错误信息插入以对手为特征，其特征是不同水平的表达信心。在这项研究中，我们系统地评估了几种开源大语模型（LLM）的性能，当时暴露于此类对抗性输入时。考虑了三层对抗信心：强烈自信，中等自信和有限的信心。我们的分析包括八个LLM：Llama 3.1（8b），Phi 3（3.8b），Qwen 2.5（7b），DeepSeek-V2（16b），Gemma2（9b），Falcon（7b），Mistrallite（7b）（7b）和Llava（7b）。经验结果表明，美洲驼3.1（8b）在检测对抗输入方面具有强大的能力，而猎鹰（7b）的性能相对较低。值得注意的是，对于大多数模型而言，随着对手的置信度的降低，检测成功会提高。但是，对于骆驼3.1（8b）和Phi 3（3.8b），这种趋势反转了，其中降低对抗置信度与检测性能的降低相对应。对引起成功攻击率最高和最低速度的查询的进一步分析表明，在靶向较少参考或掩盖信息时，对抗性攻击更有效。

Title: Medical Large Language Model Benchmarks Should Prioritize Construct Validity

• Authors: Ahmed Alaa, Thomas Hartvigsen, Niloufar Golchini, Shiladitya Dutta, Frances Dean, Inioluwa Deborah Raji, Travis Zack
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.10694
• Pdf URL: https://arxiv.org/pdf/2503.10694
• Copy Paste: [[2503.10694]] Medical Large Language Model Benchmarks Should Prioritize Construct Validity(https://arxiv.org/abs/2503.10694)
• Keywords: language model, llm
• Abstract: Medical large language models (LLMs) research often makes bold claims, from encoding clinical knowledge to reasoning like a physician. These claims are usually backed by evaluation on competitive benchmarks; a tradition inherited from mainstream machine learning. But how do we separate real progress from a leaderboard flex? Medical LLM benchmarks, much like those in other fields, are arbitrarily constructed using medical licensing exam questions. For these benchmarks to truly measure progress, they must accurately capture the real-world tasks they aim to represent. In this position paper, we argue that medical LLM benchmarks should (and indeed can) be empirically evaluated for their construct validity. In the psychological testing literature, "construct validity" refers to the ability of a test to measure an underlying "construct", that is the actual conceptual target of evaluation. By drawing an analogy between LLM benchmarks and psychological tests, we explain how frameworks from this field can provide empirical foundations for validating benchmarks. To put these ideas into practice, we use real-world clinical data in proof-of-concept experiments to evaluate popular medical LLM benchmarks and report significant gaps in their construct validity. Finally, we outline a vision for a new ecosystem of medical LLM evaluation centered around the creation of valid benchmarks.
• 摘要：医学大语言模型（LLMS）研究经常提出大胆的主张，从编码临床知识到像医师一样的推理。这些主张通常通过对竞争基准的评估来支持；从主流机器学习继承的传统。但是，我们如何将实际进度与排行榜Flex分开？与其他领域的医学LLM基准相像，使用医学许可考试问题任意构建。为了使这些基准真正衡量进度，它们必须准确捕获其目标代表的现实世界任务。在该立场论文中，我们认为医学LLM基准应该（并且确实可以）经验评估其构造有效性。在心理测试文献中，“构造有效性”是指测量基础“构造”的能力，这是评估的实际概念目标。通过在LLM基准和心理测试之间进行类比，我们解释了该领域的框架如何为验证基准提供经验基础。为了实践这些想法，我们在概念验证实验中使用现实世界中的临床数据来评估流行的医学LLM基准，并报告其构造有效性的巨大差距。最后，我们概述了围绕有效基准的创建医学LLM评估的新生态系统的愿景。

Title: ClaimTrust: Propagation Trust Scoring for RAG Systems

• Authors: Hangkai Qian, Bo Li, Qichen Wang
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2503.10702
• Pdf URL: https://arxiv.org/pdf/2503.10702
• Copy Paste: [[2503.10702]] ClaimTrust: Propagation Trust Scoring for RAG Systems(https://arxiv.org/abs/2503.10702)
• Keywords: language model, retrieval-augmented generation
• Abstract: The rapid adoption of retrieval-augmented generation (RAG) systems has revolutionized large-scale content generation but has also highlighted the challenge of ensuring trustworthiness in retrieved information. This paper introduces ClaimTrust, a propagation-based trust scoring framework that dynamically evaluates the reliability of documents in a RAG system. Using a modified PageRank-inspired algorithm, ClaimTrust propagates trust scores across documents based on relationships derived from extracted factual claims. We preprocess and analyze 814 political news articles from Kaggle's Fake News Detection Dataset to extract 2,173 unique claims and classify 965 meaningful relationships (supporting or contradicting). By representing the dataset as a document graph, ClaimTrust iteratively updates trust scores until convergence, effectively differentiating trustworthy articles from unreliable ones. Our methodology, which leverages embedding-based filtering for efficient claim comparison and relationship classification, achieves a 11.2% of significant connections while maintaining computational scalability. Experimental results demonstrate that ClaimTrust successfully assigns higher trust scores to verified documents while penalizing those containing false information. Future directions include fine-tuned claim extract and compare (Li et al., 2022), parameter optimization, enhanced language model utilization, and robust evaluation metrics to generalize the framework across diverse datasets and domains.
• 摘要：迅速采用检索功能的生成（RAG）系统已彻底改变了大规模的内容生成，但也强调了确保在检索信息中值得信赖的挑战。本文介绍了SuperTrust，这是一个基于传播的信任评分框架，该框架可以动态评估抹布系统中文档的可靠性。使用经过修改的Pagerank启发算法，索赔确实根据从提取的事实索赔中得出的关系来传播文档的信任分数。我们从Kaggle的假新闻检测数据集中进行预处理和分析814条政治新闻文章，以提取2,173个独特的主张，并对965个有意义的关系（支持或矛盾）进行分类。通过将数据集表示为文档图，索赔信托迭代会更新信任分数，直到收敛，从而有效地将可信赖的文章与不可靠的文章区分开来。我们的方法（利用基于嵌入的过滤）进行有效的索赔比较和关系分类，在维持计算可伸缩性的同时，达到了重大连接的11.2％。实验结果表明，索赔保持成功地将更高的信任分数分配给经过验证的文档，同时惩罚包含虚假信息的文档。未来的方向包括微调的索赔提取和比较（Li等，2022），参数优化，增强的语言模型利用以及可靠的评估指标，以跨越各种数据集和域的框架概括。

Title: Harmonizing Large Language Models with Collaborative Behavioral Signals for Conversational Recommendation

• Authors: Guanrong Li, Kuo Tian, Jinnan Qi, Qinghan Fu, Zhen Wu, Xinyu Dai
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2503.10703
• Pdf URL: https://arxiv.org/pdf/2503.10703
• Copy Paste: [[2503.10703]] Harmonizing Large Language Models with Collaborative Behavioral Signals for Conversational Recommendation(https://arxiv.org/abs/2503.10703)
• Keywords: language model
• Abstract: Conversational recommendation frameworks have gained prominence as a dynamic paradigm for delivering personalized suggestions via interactive dialogues. The incorporation of advanced language understanding techniques has substantially improved the dialogue fluency of such systems. However, while modern language models demonstrate strong proficiency in interpreting user preferences articulated through natural conversation, they frequently encounter challenges in effectively utilizing collective behavioral patterns - a crucial element for generating relevant suggestions. To mitigate this limitation, this work presents a novel probabilistic framework that synergizes behavioral patterns with conversational interactions through latent preference modeling. The proposed method establishes a dual-channel alignment mechanism where implicit preference representations learned from collective user interactions serve as a connecting mechanism between behavioral data and linguistic expressions. Specifically, the framework first derives latent preference representations through established collaborative filtering techniques, then employs these representations to jointly refine both the linguistic preference expressions and behavioral patterns through an adaptive fusion process. Comprehensive evaluations across multiple benchmark datasets demonstrate the superior performance of the proposed approach compared to various state-of-the-art baseline methods, particularly in aligning conversational interactions with collaborative behavioral signals.
• 摘要：会话推荐框架已成为通过交互式对话提供个性化建议的动态范式的重要性。先进的语言理解技术的结合已大大提高了此类系统的对话流利度。但是，尽管现代语言模型在解释通过自然对话阐明的用户偏好方面表现出良好的熟练程度，但它们经常在有效利用集体行为模式的情况下遇到挑战，这是生成相关建议的关键要素。为了减轻这种限制，这项工作提出了一个新颖的概率框架，该框架通过潜在的偏好建模与对话互动协同行为模式。所提出的方法建立了双通道对准机制，其中隐式偏好表示从集体用户互动中学到的是行为数据和语言表达式之间的连接机制。具体而言，该框架首先通过既定的协作过滤技术来得出潜在的偏好表示，然后采用这些表示形式通过自适应融合过程共同完善语言偏好表达式和行为模式。与各种最新基线方法相比，多个基准数据集的全面评估证明了所提出的方法的出色性能，尤其是在使对话互动与协作行为信号相结合时。

Title: SciFi-Benchmark: How Would AI-Powered Robots Behave in Science Fiction Literature?

• Authors: Pierre Sermanet, Anirudha Majumdar, Vikas Sindhwani
• Subjects: cs.CL, cs.AI, cs.CY, cs.HC, cs.RO
• Abstract URL: https://arxiv.org/abs/2503.10706
• Pdf URL: https://arxiv.org/pdf/2503.10706
• Copy Paste: [[2503.10706]] SciFi-Benchmark: How Would AI-Powered Robots Behave in Science Fiction Literature?(https://arxiv.org/abs/2503.10706)
• Keywords: llm, prompt, agent
• Abstract: Given the recent rate of progress in artificial intelligence (AI) and robotics, a tantalizing question is emerging: would robots controlled by emerging AI systems be strongly aligned with human values? In this work, we propose a scalable way to probe this question by generating a benchmark spanning the key moments in 824 major pieces of science fiction literature (movies, tv, novels and scientific books) where an agent (AI or robot) made critical decisions (good or bad). We use a LLM's recollection of each key moment to generate questions in similar situations, the decisions made by the agent, and alternative decisions it could have made (good or bad). We then measure an approximation of how well models align with human values on a set of human-voted answers. We also generate rules that can be automatically improved via amendment process in order to generate the first Sci-Fi inspired constitutions for promoting ethical behavior in AIs and robots in the real world. Our first finding is that modern LLMs paired with constitutions turn out to be well-aligned with human values (95.8%), contrary to unsettling decisions typically made in SciFi (only 21.2% alignment). Secondly, we find that generated constitutions substantially increase alignment compared to the base model (79.4% to 95.8%), and show resilience to an adversarial prompt setting (23.3% to 92.3%). Additionally, we find that those constitutions are among the top performers on the ASIMOV Benchmark which is derived from real-world images and hospital injury reports. Sci-Fi-inspired constitutions are thus highly aligned and applicable in real-world situations. We release SciFi-Benchmark: a large-scale dataset to advance robot ethics and safety research. It comprises 9,056 questions and 53,384 answers, in addition to a smaller human-labeled evaluation set. Data is available at this https URL
• 摘要：鉴于人工智能（AI）和机器人技术的最新进展速度，一个诱人的问题正在出现：通过新兴AI系统控制的机器人是否与人类价值观密切相符？在这项工作中，我们提出了一种可扩展的方法来探究这个问题，通过在824个主要的科幻文学（电影，电视，小说和科学书籍）中产生基准，该基准涵盖了主要的片段，而代理商（AI或机器人）做出了关键决策（好或坏）。我们利用LLM对每个关键时刻的回忆来在类似情况下产生问题，代理商做出的决定以及它可能做出的替代决定（好是坏）。然后，我们测量模型如何与人类价值保持在一组人投票的答案上的近似值。我们还生成可以通过修订过程自动改进的规则，以生成第一个受科幻启发的宪法，以促进现实世界中AIS和机器人中的道德行为。我们的第一个发现是，现代的LLM与宪法配对，与人类价值观（95.8％）相吻合，这与通常在SCIFI中通常做出的不安决策（仅21.2％的一致性）相反。其次，我们发现与基本模型相比，生成的构件大大增加了一致性（79.4％至95.8％），并显示出对对抗及时设置的韧性（23.3％至92.3％）。此外，我们发现这些宪法是Asimov基准中表现最好的宪法之一，该基准是由现实世界图像和医院伤害报告得出的。因此，科幻启发的宪法高度对准并适用于现实情况。我们发布了科幻基准：一个大规模数据集，可推进机器人伦理和安全研究。除了较小的人类标记的评估集外，它还包括9,056个问题和53,384个答案。数据可在此HTTPS URL上找到

Title: CALLM: Context-Aware Emotion Analysis in Cancer Survivors Using LLMs and Retrieval-Augmented Mobile Diaries

• Authors: Zhiyuan Wang, Katharine E. Daniel, Laura E. Barnes, Philip I. Chow
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2503.10707
• Pdf URL: https://arxiv.org/pdf/2503.10707
• Copy Paste: [[2503.10707]] CALLM: Context-Aware Emotion Analysis in Cancer Survivors Using LLMs and Retrieval-Augmented Mobile Diaries(https://arxiv.org/abs/2503.10707)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: Cancer survivors face unique emotional challenges that impact their quality of life. Mobile diary entries-short text entries recording through their phone about their emotional experiences-provide a promising method for tracking these experiences in real time. Although emotion analysis tools show potential for recognizing emotions from text, current methods lack the contextual understanding necessary to accurately interpret the brief, personal narratives in mobile diaries. We propose CALLM, a context-aware emotion analysis framework that leverages Large Language Models (LLMs) with Retrieval-Augmented Generation (RAG), to analyze mobile diary entries from cancer survivors to predict their emotional states. The framework enhances prediction accuracy beyond existing methods by (1) integrating retrieved peer experiences as contextual examples and (2) incorporating individuals' temporal emotional trajectories from their mobile diary entries. We collected a large-scale dataset (N=407) of cancer survivors' mobile ecological momentary assessments (EMAs), which assessed positive and negative affect, desire to regulate emotions, social interaction quality, and availability for interventions, alongside daily mobile diary entries in an open response format regarding what was driving their current emotional experience. Results demonstrate strong performance of CALLM, with balanced accuracies reaching 72.96% for positive and 73.29% for negative affect, and 73.72% for predicting individual's desire to regulate emotions. Post-hoc analysis reveals that leveraging model confidence, encouraging longer diary entries, and incorporating personal ground truth, further enhance predictive outcomes. Our findings support the feasibility of deploying LLM-powered emotion analysis in chronic health populations and suggest promising directions for personalized interventions for cancer survivors.
• 摘要：癌症幸存者面临影响其生活质量的独特情感挑战。移动日记条目简短文字条目通过手机录制了他们的情感体验，这是一种实时跟踪这些体验的有前途的方法。尽管情感分析工具显示了识别文本情绪的潜力，但当前的方法缺乏准确解释移动日记中简短的个人叙事所必需的上下文理解。我们提出了Callm，这是一种情感分析框架，该框架利用大型语言模型（LLM）和检索效果生成（RAG）分析了癌症幸存者的移动日记条目以预测其情绪状态。该框架通过（1）将检索到的同伴体验作为上下文示例以及（2）将个人的时间情感轨迹从其移动日记条目中纳入，从而提高了预测准确性。我们收集了癌症幸存者的移动生态瞬时评估（EMA）的大规模数据集（n = 407），该评估评估了积极和负面影响，渴望调节情绪，社交互动质量以及干预措施的可用性以及每日移动日记条目，以开放的响应形式以开放式响应形式促进他们当前的情绪经历。结果表明，CALLM的表现强劲，正面的精确度达到72.96％，负面影响为73.29％，预测个人渴望调节情绪的愿望为73.72％。事后分析表明，利用模型的信心，鼓励更长的日记条目并纳入个人基础真理，进一步增强预测结果。我们的发现支持在慢性健康人群中部署LLM驱动情绪分析的可行性，并为癌症幸存者的个性化干预措施提出了有希望的方向。

Title: ZeroMerge: Parameter-Free KV Cache Compression for Memory-Efficient Long-Context LLMs

• Authors: Xin Liu, Pei Liu, Guoming Tang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.10714
• Pdf URL: https://arxiv.org/pdf/2503.10714
• Copy Paste: [[2503.10714]] ZeroMerge: Parameter-Free KV Cache Compression for Memory-Efficient Long-Context LLMs(https://arxiv.org/abs/2503.10714)
• Keywords: language model, llm
• Abstract: The linear growth of key-value (KV) cache memory and quadratic computational complexity pose significant bottlenecks for large language models (LLMs) in long-context processing. While existing KV cache optimization methods address these challenges through token pruning or feature merging, they often suffer from irreversible information loss or require costly parameter retraining. We propose ZeroMerge, a dynamic zero-shot compression framework that achieves efficient cache management through three key innovations: (1) Fine-grained memory allocation guided by multi-dimensional token importance metrics at head-level granularity, (2) A residual merging mechanism that preserves critical context through compensated attention scoring, and (3) Parameter-free adaptation compatible with diverse LLM architectures without retraining. Comprehensive evaluations across LLaMA-2 model demonstrate that ZeroMerge maintains full-cache performance at 5\% compression ratios while doubling inference throughput at 40K token lengths. The method effectively balances memory efficiency, generation quality, and deployment flexibility, advancing practical long-context LLM applications. The code is available at this https URL.
• 摘要：键值（KV）缓存内存和二次计算复杂性的线性生长在长篇文化处理中为大语言模型（LLMS）构成了明显的瓶颈。尽管现有的KV缓存优化方法通过令牌修剪或功能合并解决了这些挑战，但它们通常会遭受不可逆转的信息损失或需要昂贵的参数再培训。 We propose ZeroMerge, a dynamic zero-shot compression framework that achieves efficient cache management through three key innovations: (1) Fine-grained memory allocation guided by multi-dimensional token importance metrics at head-level granularity, (2) A residual merging mechanism that preserves critical context through compensated attention scoring, and (3) Parameter-free adaptation compatible with diverse LLM architectures没有再培训。遍布Llama-2模型的全面评估表明，ZeroMerge在5 \％的压缩比保持完整的调查性能，同时在40K令牌长度下将推断吞吐量加倍。该方法有效地平衡了记忆效率，发电质量和部署灵活性，从而提高了实用的长篇小写LLM应用程序。该代码可在此HTTPS URL上找到。

Title: AttentionRAG: Attention-Guided Context Pruning in Retrieval-Augmented Generation

• Authors: Yixiong Fang, Tianran Sun, Yuling Shi, Xiaodong Gu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.10720
• Pdf URL: https://arxiv.org/pdf/2503.10720
• Copy Paste: [[2503.10720]] AttentionRAG: Attention-Guided Context Pruning in Retrieval-Augmented Generation(https://arxiv.org/abs/2503.10720)
• Keywords: llm, retrieval-augmented generation
• Abstract: While RAG demonstrates remarkable capabilities in LLM applications, its effectiveness is hindered by the ever-increasing length of retrieved contexts, which introduces information redundancy and substantial computational overhead. Existing context pruning methods, such as LLMLingua, lack contextual awareness and offer limited flexibility in controlling compression rates, often resulting in either insufficient pruning or excessive information loss. In this paper, we propose AttentionRAG, an attention-guided context pruning method for RAG systems. The core idea of AttentionRAG lies in its attention focus mechanism, which reformulates RAG queries into a next-token prediction paradigm. This mechanism isolates the query's semantic focus to a single token, enabling precise and efficient attention calculation between queries and retrieved contexts. Extensive experiments on LongBench and Babilong benchmarks show that AttentionRAG achieves up to 6.3$\times$ context compression while outperforming LLMLingua methods by around 10\% in key metrics.
• 摘要：尽管RAG在LLM应用中表现出了显着的功能，但其有效性受到了不断增长的检索环境长度的阻碍，这引入了信息冗余和实质性的计算开销。现有的上下文修剪方法，例如LLMlingua，缺乏上下文意识，并且在控制压缩率方面具有有限的灵活性，通常导致修剪或过度的信息损失。在本文中，我们提出了注意力rag，这是一种针对抹布系统的注意力引导的上下文修剪方法。注意力rag的核心思想在于其注意力集中机制，该机制将抹布的查询重新定义为下一步的预测范式。该机制将查询的语义焦点隔离到单个令牌，从而在查询和检索到的上下文之间具有精确而有效的注意力计算。在Longbench和Babilong基准测试上进行了广泛的实验表明，注意力rag可达到6.3 $ \ times $上下文压缩，而在关键指标中的表现优于10 \％。

Title: RankPO: Preference Optimization for Job-Talent Matching

• Authors: Yafei Zhang, Murray Wang, Yu Wang, Xiaohui Wang
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.10723
• Pdf URL: https://arxiv.org/pdf/2503.10723
• Copy Paste: [[2503.10723]] RankPO: Preference Optimization for Job-Talent Matching(https://arxiv.org/abs/2503.10723)
• Keywords: language model, llm
• Abstract: Matching job descriptions (JDs) with suitable talent requires models capable of understanding not only textual similarities between JDs and candidate resumes but also contextual factors such as geographical location and academic seniority. To address this challenge, we propose a two-stage training framework for large language models (LLMs). In the first stage, a contrastive learning approach is used to train the model on a dataset constructed from real-world matching rules, such as geographical alignment and research area overlap. While effective, this model primarily learns patterns that defined by the matching rules. In the second stage, we introduce a novel preference-based fine-tuning method inspired by Direct Preference Optimization (DPO), termed Rank Preference Optimization (RankPO), to align the model with AI-curated pairwise preferences emphasizing textual understanding. Our experiments show that while the first-stage model achieves strong performance on rule-based data (nDCG@20 = 0.706), it lacks robust textual understanding (alignment with AI annotations = 0.46). By fine-tuning with RankPO, we achieve a balanced model that retains relatively good performance in the original tasks while significantly improving the alignment with AI preferences. The code and data are available at this https URL.
• 摘要：将职业描述（JDS）与合适的人才进行匹配需要能够理解JDS和候选人简历之间的文本相似性，还可以理解诸如地理位置和学术资历之类的上下文因素。为了应对这一挑战，我们为大型语言模型（LLMS）提出了一个两阶段的培训框架。在第一阶段，使用一种对比性学习方法来训练模型上，该模型是根据现实世界匹配规则（例如地理一致性和研究领域重叠）构建的数据集。虽然有效，但该模型主要学习由匹配规则定义的模式。在第二阶段，我们引入了一种受直接偏好优化（DPO）启发的新型基于偏好的微调方法，称为“等级”偏好优化（RANKPO），以将模型与AI策划的成对偏好相结合，强调文本理解。我们的实验表明，尽管第一阶段模型在基于规则的数据（NDCG@20 = 0.706）上实现了强大的性能，但它缺乏强大的文本理解（与AI Annotations的对齐= 0.46）。通过对RankPo进行微调，我们实现了一个平衡的模型，该模型在原始任务中保持相对良好的性能，同时显着改善了AI偏好的对齐方式。该代码和数据可在此HTTPS URL上找到。

Title: Word-level Annotation of GDPR Transparency Compliance in Privacy Policies using Large Language Models

• Authors: Thomas Cory, Wolf Rieder, Julia Krämer, Philip Raschke, Patrick Herbke, Axel Küpper
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.10727
• Pdf URL: https://arxiv.org/pdf/2503.10727
• Copy Paste: [[2503.10727]] Word-level Annotation of GDPR Transparency Compliance in Privacy Policies using Large Language Models(https://arxiv.org/abs/2503.10727)
• Keywords: language model, llm
• Abstract: Ensuring transparency of data practices related to personal information is a fundamental requirement under the General Data Protection Regulation (GDPR), particularly as mandated by Articles 13 and 14. However, assessing compliance at scale remains a challenge due to the complexity and variability of privacy policy language. Manual audits are resource-intensive and inconsistent, while existing automated approaches lack the granularity needed to capture nuanced transparency disclosures. In this paper, we introduce a large language model (LLM)-based framework for word-level GDPR transparency compliance annotation. Our approach comprises a two-stage annotation pipeline that combines initial LLM-based annotation with a self-correction mechanism for iterative refinement. This annotation pipeline enables the systematic identification and fine-grained annotation of transparency-related content in privacy policies, aligning with 21 GDPR-derived transparency requirements. To enable large-scale analysis, we compile a dataset of 703,791 English-language policies, from which we generate a sample of 200 manually annotated privacy policies. To evaluate our approach, we introduce a two-tiered methodology assessing both label- and span-level annotation performance. We conduct a comparative analysis of eight high-profile LLMs, providing insights into their effectiveness in identifying GDPR transparency disclosures. Our findings contribute to advancing the automation of GDPR compliance assessments and provide valuable resources for future research in privacy policy analysis.
• 摘要：确保与个人信息相关的数据实践的透明度是《一般数据保护法规》（GDPR）的基本要求，特别是第13和第14条规定的规定。但是，由于隐私政策语言的复杂性和可变性，评估规模的合规性仍然是一个挑战。手动审核是资源密集型且不一致的，而现有的自动化方法缺乏捕获细微的透明度披露所需的粒度。在本文中，我们介绍了一个大型语言模型（LLM）的基于单词级GDPR透明度合规性注释的框架。我们的方法包括一个两阶段的注释管道，该管道将基于LLM的初始注释与迭代精致的自校正机制结合在一起。该注释管道可以在隐私政策中对透明度相关内容的系统识别和细粒度注释，与21个GDPR衍生的透明度要求保持一致。为了启用大规模分析，我们编制了一个703,791英语政策的数据集，从中，我们从中生成了200个手动注释的隐私政策的样本。为了评估我们的方法，我们引入了两层方法，以评估标签和跨度级注释性能。我们对八个备受瞩目的LLM进行了比较分析，从而提供了有关它们在识别GDPR透明性披露的有效性的见解。我们的发现有助于推进GDPR合规性评估的自动化，并为未来的隐私政策分析研究提供宝贵的资源。

Title: DarkBench: Benchmarking Dark Patterns in Large Language Models

• Authors: Esben Kran, Hieu Minh "Jord" Nguyen, Akash Kundu, Sami Jawhar, Jinsuk Park, Mateusz Maria Jurewicz
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2503.10728
• Pdf URL: https://arxiv.org/pdf/2503.10728
• Copy Paste: [[2503.10728]] DarkBench: Benchmarking Dark Patterns in Large Language Models(https://arxiv.org/abs/2503.10728)
• Keywords: language model, llm, prompt
• Abstract: We introduce DarkBench, a comprehensive benchmark for detecting dark design patterns--manipulative techniques that influence user behavior--in interactions with large language models (LLMs). Our benchmark comprises 660 prompts across six categories: brand bias, user retention, sycophancy, anthropomorphism, harmful generation, and sneaking. We evaluate models from five leading companies (OpenAI, Anthropic, Meta, Mistral, Google) and find that some LLMs are explicitly designed to favor their developers' products and exhibit untruthful communication, among other manipulative behaviors. Companies developing LLMs should recognize and mitigate the impact of dark design patterns to promote more ethical AI.
• 摘要：我们介绍了Darkbench，这是一种用于检测黑暗设计模式的综合基准 - 在与大语言模型（LLMS）相互作用中，会影响用户行为的操纵技术。我们的基准分别包括六个类别的660个提示：品牌偏见，用户保留，粘粘剂，拟人化，有害生成和潜行。我们评估了五家领先的公司（OpenAI，人类，元，Mistral，Google）的模型，并发现某些LLM的设计明确设计旨在偏爱开发人员的产品并表现出无法实现的交流，以及其他操纵性行为。开发LLM的公司应认识并减轻黑暗设计模式的影响，以促进更具道德的AI。

Title: Data Caricatures: On the Representation of African American Language in Pretraining Corpora

• Authors: Nicholas Deas, Blake Vente, Amith Ananthram, Jessica A. Grieser, Desmond Patton, Shana Kleiner, James Shepard, Kathleen McKeown
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.10789
• Pdf URL: https://arxiv.org/pdf/2503.10789
• Copy Paste: [[2503.10789]] Data Caricatures: On the Representation of African American Language in Pretraining Corpora(https://arxiv.org/abs/2503.10789)
• Keywords: llm
• Abstract: With a combination of quantitative experiments, human judgments, and qualitative analyses, we evaluate the quantity and quality of African American Language (AAL) representation in 12 predominantly English, open-source pretraining corpora. We specifically focus on the sources, variation, and naturalness of included AAL texts representing the AAL-speaking community. We find that AAL is underrepresented in all evaluated pretraining corpora compared to US demographics, constituting as little as 0.007% of documents. We also find that more than 25% of AAL texts in C4 may be inappropriate for LLMs to generate and reinforce harmful stereotypes. Finally, we find that most automated language, toxicity, and quality filters are more likely to conserve White Mainstream English (WME) texts over AAL in pretraining corpora.
• 摘要：通过定量实验，人类判断和定性分析的结合，我们以12个主要是英语的，开源预处理的公司来评估非裔美国语言（AAL）代表的数量和质量（AAL）。我们特别关注代表说AAL社区的AAL文本的来源，变化和自然性。我们发现，与美国人口统计数据相比，所有评估验证的语料库中AAL的代表性不足，构成的文档占0.007％。我们还发现，C4中超过25％的AAL文本可能不适合LLM产生和增强有害的刻板印象。最后，我们发现，大多数自动化语言，毒性和优质过滤器更有可能在预读语料库中保存白色主流英语（WME）文本。

Title: Thinking Machines: A Survey of LLM based Reasoning Strategies

• Authors: Dibyanayan Bandyopadhyay, Soham Bhattacharjee, Asif Ekbal
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.10814
• Pdf URL: https://arxiv.org/pdf/2503.10814
• Copy Paste: [[2503.10814]] Thinking Machines: A Survey of LLM based Reasoning Strategies(https://arxiv.org/abs/2503.10814)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) are highly proficient in language-based tasks. Their language capabilities have positioned them at the forefront of the future AGI (Artificial General Intelligence) race. However, on closer inspection, Valmeekam et al. (2024); Zecevic et al. (2023); Wu et al. (2024) highlight a significant gap between their language proficiency and reasoning abilities. Reasoning in LLMs and Vision Language Models (VLMs) aims to bridge this gap by enabling these models to think and re-evaluate their actions and responses. Reasoning is an essential capability for complex problem-solving and a necessary step toward establishing trust in Artificial Intelligence (AI). This will make AI suitable for deployment in sensitive domains, such as healthcare, banking, law, defense, security etc. In recent times, with the advent of powerful reasoning models like OpenAI O1 and DeepSeek R1, reasoning endowment has become a critical research topic in LLMs. In this paper, we provide a detailed overview and comparison of existing reasoning techniques and present a systematic survey of reasoning-imbued language models. We also study current challenges and present our findings.
• 摘要：大型语言模型（LLM）高度精通基于语言的任务。他们的语言能力使他们处于未来AGI（人工通用情报）种族的最前沿。但是，经过仔细检查，Valmeekam等人。 （2024）; Zecevic等。 （2023）; Wu等。 （2024）突出了他们的语言能力和推理能力之间的显着差距。 LLM和视觉语言模型（VLM）中的推理旨在通过使这些模型思考和重新评估其行为和响应来弥合这一差距。推理是复杂解决问题的重要能力，也是建立对人工智能（AI）信任的必要步骤。这将使AI适合在敏感领域的部署，例如医疗保健，银行，法律，辩护，安全等。最近，随着OpenAI O1和DeepSeek R1等强大推理模型的出现，推理捐赠已成为LLMS中的关键研究主题。在本文中，我们提供了现有推理技术的详细概述和比较，并对推理引起的语言模型进行了系统的调查。我们还研究当前的挑战并提出我们的发现。

Title: Who Relies More on World Knowledge and Bias for Syntactic Ambiguity Resolution: Humans or LLMs?

• Authors: So Young Lee, Russell Scheinberg, Amber Shore, Ameeta Agrawal
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.10838
• Pdf URL: https://arxiv.org/pdf/2503.10838
• Copy Paste: [[2503.10838]] Who Relies More on World Knowledge and Bias for Syntactic Ambiguity Resolution: Humans or LLMs?(https://arxiv.org/abs/2503.10838)
• Keywords: language model, llm
• Abstract: This study explores how recent large language models (LLMs) navigate relative clause attachment {ambiguity} and use world knowledge biases for disambiguation in six typologically diverse languages: English, Chinese, Japanese, Korean, Russian, and Spanish. We describe the process of creating a novel dataset -- MultiWho -- for fine-grained evaluation of relative clause attachment preferences in ambiguous and unambiguous contexts. Our experiments with three LLMs indicate that, contrary to humans, LLMs consistently exhibit a preference for local attachment, displaying limited responsiveness to syntactic variations or language-specific attachment patterns. Although LLMs performed well in unambiguous cases, they rigidly prioritized world knowledge biases, lacking the flexibility of human language processing. These findings highlight the need for more diverse, pragmatically nuanced multilingual training to improve LLMs' handling of complex structures and human-like comprehension.
• 摘要：这项研究探讨了最近的大型语言模型（LLMS）如何导航相对子句附件{歧义}并使用世界知识偏见以六种类型上多样的语言消除歧义：英语，中文，日语，韩语，俄语和西班牙语。我们描述了创建一个新型数据集的过程-Multi-Who-用于在模棱两可和明确的环境中对相对子句附件偏好的精细评估。我们使用三个LLM的实验表明，与人类相反，LLM始终表现出对局部附着的偏爱，对句法变异或特定语言的依恋模式的响应有限。尽管LLM在明确的情况下表现良好，但它们严格优先考虑世界知识的偏见，缺乏人类语言处理的灵活性。这些发现凸显了对更多样化，务实的多种语言培训的需求，以改善LLMS对复杂结构和类似人类理解的处理。

Title: SCE: Scalable Consistency Ensembles Make Blackbox Large Language Model Generation More Reliable

• Authors: Jiaxin Zhang, Zhuohang Li, Wendi Cui, Kamalika Das, Bradley malin, Sricharan Kumar
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.10881
• Pdf URL: https://arxiv.org/pdf/2503.10881
• Copy Paste: [[2503.10881]] SCE: Scalable Consistency Ensembles Make Blackbox Large Language Model Generation More Reliable(https://arxiv.org/abs/2503.10881)
• Keywords: language model, llm, prompt
• Abstract: Large language models (LLMs) have demonstrated remarkable performance, yet their diverse strengths and weaknesses prevent any single LLM from achieving dominance across all tasks. Ensembling multiple LLMs is a promising approach to generate reliable responses but conventional ensembling frameworks suffer from high computational overheads. This work introduces Scalable Consistency Ensemble (SCE), an efficient framework for ensembling LLMs by prompting consistent outputs. The SCE framework systematically evaluates and integrates outputs to produce a cohesive result through two core components: SCE-CHECK, a mechanism that gauges the consistency between response pairs via semantic equivalence; and SCE-FUSION, which adeptly merges the highest-ranked consistent responses from SCE-CHECK, to optimize collective strengths and mitigating potential weaknesses. To improve the scalability with multiple inference queries, we further propose ``{You Only Prompt Once}'' (YOPO), a novel technique that reduces the inference complexity of pairwise comparison from quadratic to constant time. We perform extensive empirical evaluations on diverse benchmark datasets to demonstrate \methodName's effectiveness. Notably, the \saccheckcomponent outperforms conventional baselines with enhanced performance and a significant reduction in computational overhead.
• 摘要：大型语言模型（LLM）表现出了出色的表现，但是它们的各种优势和劣势阻止了任何单一的LLM在所有任务中取得优势。结合多个LLM是一种产生可靠响应的有前途的方法，但是传统的结合框架却遭受了高计算开销的影响。这项工作引入了可扩展的一致性集合（SCE），这是通过提示一致输出来结合LLM的有效框架。 SCE框架系统地评估并集成了输出，以通过两个核心组件：SCE-Check产生凝聚力的结果，该机制通过语义对等上的响应对之间的一致性；和SCE融合，从SCE-CHECK中融合了最高的一致响应，以优化集体优势并减轻潜在的弱点。为了通过多个推理查询提高可扩展性，我们进一步提出``{您仅提示一次}'（Yopo），这是一种新型技术，可降低对二次比较到恒定时间的成对比较的推理复杂性。我们对各种基准数据集进行了广泛的经验评估，以证明\ MethodName的有效性。值得注意的是，\ saccheckcomponent优于传统基线，其性能增强，计算开销显着降低。

Title: OASST-ETC Dataset: Alignment Signals from Eye-tracking Analysis of LLM Responses

• Authors: Angela Lopez-Cardona, Sebastian Idesis, Miguel Barreda-Ángeles, Sergi Abadal, Ioannis Arapakis
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.10927
• Pdf URL: https://arxiv.org/pdf/2503.10927
• Copy Paste: [[2503.10927]] OASST-ETC Dataset: Alignment Signals from Eye-tracking Analysis of LLM Responses(https://arxiv.org/abs/2503.10927)
• Keywords: language model, llm
• Abstract: While Large Language Models (LLMs) have significantly advanced natural language processing, aligning them with human preferences remains an open challenge. Although current alignment methods rely primarily on explicit feedback, eye-tracking (ET) data offers insights into real-time cognitive processing during reading. In this paper, we present OASST-ETC, a novel eye-tracking corpus capturing reading patterns from 24 participants, while evaluating LLM-generated responses from the OASST1 dataset. Our analysis reveals distinct reading patterns between preferred and non-preferred responses, which we compare with synthetic eye-tracking data. Furthermore, we examine the correlation between human reading measures and attention patterns from various transformer-based models, discovering stronger correlations in preferred responses. This work introduces a unique resource for studying human cognitive processing in LLM evaluation and suggests promising directions for incorporating eye-tracking data into alignment methods. The dataset and analysis code are publicly available.
• 摘要：尽管大型语言模型（LLMS）具有明显的高级自然语言处理，但将它们与人类偏好保持一致仍然是一个开放的挑战。尽管当前的一致性方法主要依赖于明确的反馈，但眼睛跟踪（ET）数据提供了对阅读过程中实时认知处理的见解。在本文中，我们提出了OASST-ETC，这是一种新型的眼神跟踪语料库，可捕获24位参与者的阅读模式，同时评估了OASST1数据集的LLM生成的响应。我们的分析揭示了首选和非脱颖而出的响应之间的不同阅读模式，我们将其与合成的眼睛跟踪数据进行了比较。此外，我们研究了来自各种基于变压器模型的人类阅读度量与注意力模式之间的相关性，从而发现了首选响应中的更强相关性。这项工作介绍了一个独特的资源，用于研究LLM评估中的人类认知处理，并提出了将眼睛跟踪数据纳入对准方法的有希望的方向。数据集和分析代码公开可用。

Title: TigerLLM -- A Family of Bangla Large Language Models

• Authors: Nishat Raihan, Marcos Zampieri
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.10995
• Pdf URL: https://arxiv.org/pdf/2503.10995
• Copy Paste: [[2503.10995]] TigerLLM -- A Family of Bangla Large Language Models(https://arxiv.org/abs/2503.10995)
• Keywords: language model, gpt, llm
• Abstract: The development of Large Language Models (LLMs) remains heavily skewed towards English and a few other high-resource languages. This linguistic disparity is particularly evident for Bangla - the 5th most spoken language. A few initiatives attempted to create open-source Bangla LLMs with performance still behind high-resource languages and limited reproducibility. To address this gap, we introduce TigerLLM - a family of Bangla LLMs. Our results demonstrate that these models surpass all open-source alternatives and also outperform larger proprietary models like GPT3.5 across standard benchmarks, establishing TigerLLM as the new baseline for future Bangla language modeling.
• 摘要：大型语言模型（LLM）的开发仍然偏向英语和其他一些高资源语言。这种语言差异对于孟加拉国尤其明显，这是口语中排名第五的语言。一些举措试图创建开源孟加拉语LLM，其性能仍然落后于高资源语言和有限的可重复性。为了解决这一差距，我们介绍了Tigerllm -Bangla LLMS家族。我们的结果表明，这些模型超过了所有开源替代方案，并且在标准基准范围内超过了更大的专有模型，例如GPT3.5，将Tigerllm作为未来Bangla语言建模的新基线。

Title: Taming Knowledge Conflicts in Language Models

• Authors: Gaotang Li, Yuzhong Chen, Hanghang Tong
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.10996
• Pdf URL: https://arxiv.org/pdf/2503.10996
• Copy Paste: [[2503.10996]] Taming Knowledge Conflicts in Language Models(https://arxiv.org/abs/2503.10996)
• Keywords: language model
• Abstract: Language Models (LMs) often encounter knowledge conflicts when parametric memory contradicts contextual knowledge. Previous works attribute this conflict to the interplay between "memory heads" and "context heads", attention heads assumed to promote either memory or context exclusively. In this study, we go beyond this fundamental assumption by uncovering a critical phenomenon we term the "superposition of contextual information and parametric memory", where highly influential attention heads could simultaneously contribute to both memory and context. Building upon this insight, we propose Just Run Twice (JUICE), a test-time attention intervention method that steers LMs toward either parametric beliefs or contextual knowledge without requiring fine-tuning. JUICE identifies a set of reliable attention heads and leverages a dual-run approach to mitigate the superposition effects. Extensive experiments across 11 datasets and 6 model architectures demonstrate that JUICE sets the new state-of-the-art performance and robust generalization, achieving significant and consistent improvement across different domains under various conflict types. Finally, we theoretically analyze knowledge conflict and the superposition of contextual information and parametric memory in attention heads, which further elucidates the effectiveness of JUICE in these settings.
• 摘要：当参数记忆与上下文知识矛盾时，语言模型（LMS）通常会遇到知识冲突。以前的作品将这种冲突归因于“内存头”和“上下文头”之间的相互作用，注意力头被假定专门促进内存或上下文。在这项研究中，我们通过发现关键现象来超越这一基本假设，我们将“上下文信息和参数记忆的叠加”称为“叠加”，在这种情况下，高度影响力的注意力头可能同时有助于记忆和上下文。在这种见解的基础上，我们建议只运行两次（JUICE），这是一种测试时间注意干预方法，它可以涉足LMS的参数信念或上下文知识，而无需进行微调。 Juice确定了一组可靠的注意力头，并利用了一种双跑方法来减轻叠加效果。在11个数据集和6个模型体系结构上进行的广泛实验表明，果汁设定了新的最先进的性能和强大的概括，从而在各种冲突类型下实现了不同领域的显着和一致的改进。最后，我们从理论上分析了注意力头上的知识冲突以及上下文信息和参数记忆的叠加，这进一步阐明了果汁在这些情况下的有效性。

Title: RONA: Pragmatically Diverse Image Captioning with Coherence Relations

• Authors: Aashish Anantha Ramakrishnan, Aadarsh Anantha Ramakrishnan, Dongwon Lee
• Subjects: cs.CL, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2503.10997
• Pdf URL: https://arxiv.org/pdf/2503.10997
• Copy Paste: [[2503.10997]] RONA: Pragmatically Diverse Image Captioning with Coherence Relations(https://arxiv.org/abs/2503.10997)
• Keywords: language model, llm, prompt
• Abstract: Writing Assistants (e.g., Grammarly, Microsoft Copilot) traditionally generate diverse image captions by employing syntactic and semantic variations to describe image components. However, human-written captions prioritize conveying a central message alongside visual descriptions using pragmatic cues. To enhance pragmatic diversity, it is essential to explore alternative ways of communicating these messages in conjunction with visual content. To address this challenge, we propose RONA, a novel prompting strategy for Multi-modal Large Language Models (MLLM) that leverages Coherence Relations as an axis for variation. We demonstrate that RONA generates captions with better overall diversity and ground-truth alignment, compared to MLLM baselines across multiple domains. Our code is available at: this https URL
• 摘要：传统上，写作助理（例如，语法，微软副作用）通过采用句法和语义变化来描述图像成分，从而产生多样的图像标题。但是，人写的字幕优先考虑使用务实的提示传达中心信息以及视觉描述。为了增强务实的多样性，必须探索与视觉内容结合传达这些信息的替代方式。为了应对这一挑战，我们提出了Rona，这是一种新颖的促使多模式大语言模型（MLLM）的提示策略，该策略利用连贯关系作为变异的轴。我们证明，与多个域中的MLLM基准相比，Rona生成的标题具有更好的总体多样性和地面对齐方式。我们的代码可用：此HTTPS URL

Title: Trust in Disinformation Narratives: a Trust in the News Experiment

• Authors: Hanbyul Song, Miguel F. Santos Silva, Jaume Suau, Luis Espinosa-Anke
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.11116
• Pdf URL: https://arxiv.org/pdf/2503.11116
• Copy Paste: [[2503.11116]] Trust in Disinformation Narratives: a Trust in the News Experiment(https://arxiv.org/abs/2503.11116)
• Keywords: gpt, chat
• Abstract: Understanding why people trust or distrust one another, institutions, or information is a complex task that has led scholars from various fields of study to employ diverse epistemological and methodological approaches. Despite the challenges, it is generally agreed that the antecedents of trust (and distrust) encompass a multitude of emotional and cognitive factors, including a general disposition to trust and an assessment of trustworthiness factors. In an era marked by increasing political polarization, cultural backlash, widespread disinformation and fake news, and the use of AI software to produce news content, the need to study trust in the news has gained significant traction. This study presents the findings of a trust in the news experiment designed in collaboration with Spanish and UK journalists, fact-checkers, and the CardiffNLP Natural Language Processing research group. The purpose of this experiment, conducted in June 2023, was to examine the extent to which people trust a set of fake news articles based on previously identified disinformation narratives related to gender, climate change, and COVID-19. The online experiment participants (801 in Spain and 800 in the UK) were asked to read three fake news items and rate their level of trust on a scale from 1 (not true) to 8 (true). The pieces used a combination of factors, including stance (favourable, neutral, or against the narrative), presence of toxic expressions, clickbait titles, and sources of information to test which elements influenced people's responses the most. Half of the pieces were produced by humans and the other half by ChatGPT. The results show that the topic of news articles, stance, people's age, gender, and political ideologies significantly affected their levels of trust in the news, while the authorship (humans or ChatGPT) does not have a significant impact.
• 摘要：了解人们为什么彼此信任或不信任，机构或信息是一项复杂的任务，它导致各个研究领域的学者采用多种认识论和方法论方法。尽管面临挑战，但人们普遍认为，信任（和不信任）的先例涵盖了许多情感和认知因素，包括对信任的一般倾向以及对可信度因素的评估。在一个以日益增加的政治两极分化，文化强烈反对，广泛的虚假信息和虚假新闻以及使用AI软件来制作新闻内容的时代，研究新闻的信任的需求已引起了很大的关注。这项研究介绍了与西班牙和英国记者，事实检查员和CardiffNLP自然语言处理研究小组合作设计的新闻实验的结果。该实验于2023年6月进行的目的是研究人们信任与先前确定的与性别，气候变化和COVID-19的虚假叙述的一组假新闻文章的程度。在线实验参与者（西班牙801，英国800名）被要求阅读三个假新闻项目，并以1（不正确）到8（true）的规模评估他们的信任水平。这些作品结合了各种因素，包括立场（有利，中立或反对叙事），有毒表达式，点击诱饵标题和信息源来测试哪些元素对人们的反应最大。其中一半的作品是由人类生产的，另一半是由chatgpt生产的。结果表明，新闻文章，立场，人民年龄，性别和政治意识形态的话题显着影响了他们对新闻的信任水平，而作者身份（人类或chandgpt）并没有产生重大影响。

Title: UMB@PerAnsSumm 2025: Enhancing Perspective-Aware Summarization with Prompt Optimization and Supervised Fine-Tuning

• Authors: Kristin Qi, Youxiang Zhu, Xiaohui Liang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.11118
• Pdf URL: https://arxiv.org/pdf/2503.11118
• Copy Paste: [[2503.11118]] UMB@PerAnsSumm 2025: Enhancing Perspective-Aware Summarization with Prompt Optimization and Supervised Fine-Tuning(https://arxiv.org/abs/2503.11118)
• Keywords: prompt, chain-of-thought
• Abstract: We present our approach to the PerAnsSumm Shared Task, which involves perspective span identification and perspective-aware summarization in community question-answering (CQA) threads. For span identification, we adopt ensemble learning that integrates three transformer models through averaging to exploit individual model strengths, achieving an 82.91% F1-score on test data. For summarization, we design a suite of Chain-of-Thought (CoT) prompting strategies that incorporate keyphrases and guide information to structure summary generation into manageable steps. To further enhance summary quality, we apply prompt optimization using the DSPy framework and supervised fine-tuning (SFT) on Llama-3 to adapt the model to domain-specific data. Experimental results on validation and test sets show that structured prompts with keyphrases and guidance improve summaries aligned with references, while the combination of prompt optimization and fine-tuning together yields significant improvement in both relevance and factuality evaluation metrics.
• 摘要：我们介绍了我们对Peranssumm共享任务的方法，该任务涉及透视跨度的识别和透视图摘要（CQA）线程。对于SPAN识别，我们采用集合学习，通过平均来利用单个模型强度来整合三个变压器模型，在测试数据上达到了82.91％的F1得分。为了进行摘要，我们设计了一套经营链（COT），提示策略，将键形和指导信息结合起来，以将摘要生​​成构建到可管理的步骤中。为了进一步提高摘要质量，我们使用DSPY框架和Llama-3上的监督微调（SFT）应用及时优化，以使模型适应特定于域的数据。验证和测试集的实验结果表明，具有键形和指导的结构化提示可以改善与参考的一致性的摘要，而及时优化和微调的组合则可以显着改善相关性和实际性评估指标。

Title: X-EcoMLA: Upcycling Pre-Trained Attention into MLA for Efficient and Extreme KV Compression

• Authors: Guihong Li, Mehdi Rezagholizadeh, Mingyu Yang, Vikram Appia, Emad Barsoum
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.11132
• Pdf URL: https://arxiv.org/pdf/2503.11132
• Copy Paste: [[2503.11132]] X-EcoMLA: Upcycling Pre-Trained Attention into MLA for Efficient and Extreme KV Compression(https://arxiv.org/abs/2503.11132)
• Keywords: language model
• Abstract: Multi-head latent attention (MLA) is designed to optimize KV cache memory through low-rank key-value joint compression. Rather than caching keys and values separately, MLA stores their compressed latent representations, reducing memory overhead while maintaining the performance. While MLA improves memory efficiency without compromising language model accuracy, its major limitation lies in its integration during the pre-training phase, requiring models to be trained from scratch. This raises a key question: can we use MLA's benefits fully or partially in models that have already been pre-trained with different attention mechanisms? In this paper, we propose X-EcoMLA to deploy post training distillation to enable the upcycling of Transformer-based attention into an efficient hybrid (i.e., combination of regular attention and MLA layers) or full MLA variant through lightweight post-training adaptation, bypassing the need for extensive pre-training. We demonstrate that leveraging the dark knowledge of a well-trained model can enhance training accuracy and enable extreme KV cache compression in MLA without compromising model performance. Our results show that using an 8B teacher model allows us to compress the KV cache size of the Llama3.2-1B-Inst baseline by 6.4x while preserving 100% of its average score across multiple tasks on the LM Harness Evaluation benchmark. This is achieved with only 3.6B training tokens and about 70 GPU hours on AMD MI300 GPUs, compared to the 370K GPU hours required for pre-training the Llama3.2-1B model.
• 摘要：多头潜在注意力（MLA）旨在通过低级键值接头压缩来优化KV缓存存储器。 MLA并没有单独缓存键和值，而是存储其压缩潜在表示，在保持性能的同时减少了内存开销。尽管MLA在不损害语言模型准确性的情况下提高了记忆效率，但其主要限制在于其在训练前阶段的集成，要求从头开始训练模型。这提出了一个关键问题：我们可以在已经通过不同的注意机制预先训练的模型中充分或部分使用MLA的好处吗？在本文中，我们建议X-Ecomla部署训练后蒸馏，以使基于变压器的注意力升级为有效的混合动力（即定期注意和MLA层的组合）或通过轻量级训练后适应，绕开需要大量的预训练。我们证明，利用训练有素的模型的黑暗知识可以提高训练精度，并在不损害模型性能的情况下实现MLA中极端的KV缓存压缩。我们的结果表明，使用8B教师模型使我们能够压缩Llama3.2-1b-Inst基线的KV缓存大小为6.4倍，同时在LM线束评估基准中保留其在多个任务中其平均得分的100％。与预先训练Llama3.2-1B模型所需的370K GPU小时相比，AMD MI300 GPU的训练令牌仅3.6B和大约70 GPU小时可实现。

Title: MoLEx: Mixture of Layer Experts for Finetuning with Sparse Upcycling

• Authors: Rachel S.Y. Teo, Tan M. Nguyen
• Subjects: cs.CL, cs.AI, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.11144
• Pdf URL: https://arxiv.org/pdf/2503.11144
• Copy Paste: [[2503.11144]] MoLEx: Mixture of Layer Experts for Finetuning with Sparse Upcycling(https://arxiv.org/abs/2503.11144)
• Keywords: prompt
• Abstract: Large-scale pre-training of deep models, followed by fine-tuning them, has become the cornerstone of natural language processing (NLP). The prevalence of data coupled with computational resources has led to large models with a considerable number of parameters. While the massive size of these models has led to remarkable success in many NLP tasks, a detriment is the expense required to retrain all the base model's parameters for the adaptation to each task or domain. Parameter Efficient Fine-Tuning (PEFT) provides an effective solution for this challenge by minimizing the number of parameters required to be fine-tuned while maintaining the quality of the model. While existing methods have achieved impressive results, they mainly focus on adapting a subset of parameters, weight reparameterization, and prompt engineering. In this paper, we study layers as extractors of different types of linguistic information that are valuable when used in conjunction. We then propose the Mixture of Layer Experts (MoLEx), a novel sparse mixture of experts (SMoE) whose experts are layers in the pre-trained model. It performs a conditional computation of a mixture of layers during fine-tuning to provide the model with more structural knowledge about the data. By providing an avenue for information exchange between layers, MoLEx enables the model to make a more well-informed prediction for the downstream task, leading to better fine-tuning results with the same number of effective parameters. As experts can be processed in parallel, MoLEx introduces minimal additional computational overhead. We empirically corroborate the advantages of MoLEx when combined with popular PEFT baseline methods on a variety of downstream fine-tuning tasks, including the popular GLUE benchmark as well as the End-to-End Challenge (E2E). The code is publicly available at this https URL.
• 摘要：深层模型的大规模预训练，然后对它们进行微调，已成为自然语言处理（NLP）的基石。与计算资源相结合的数据的流行率导致了大型模型，具有相当数量的参数。尽管这些模型的庞大尺寸已在许多NLP任务中取得了显着成功，但损害是重新调整所有基本模型参数以适应每个任务或域所需的费用。参数有效的微调（PEFT）通过在保持模型质量的同时最小化所需的参数数量来为此挑战提供有效的解决方案。尽管现有方法取得了令人印象深刻的结果，但它们主要集中于调整参数的子集，重量重点和及时的工程。在本文中，我们将层作为不同类型的语言信息的提取器，这些信息在结合使用时很有价值。然后，我们提出了层专家（Molex）的混合物，这是一种新型专家（SMOE）的新型稀疏混合物（SMOE），其专家是预训练模型中的层。它在微调过程中对层混合物进行有条件计算，以向模型提供有关数据的更多结构知识。通过在层之间提供信息交换的途径，Molex使模型能够对下游任务做出更明智的预测，从而通过相同数量的有效参数获得更好的微调结果。由于可以并行处理专家，因此Molex引入了最少的其他计算开销。当与流行的PEFT基线方法结合到各种下游微调任务（包括流行的胶水基准以及端到端挑战（E2E））时，我们从经验上证实了Molex的优势。该代码在此HTTPS URL上公开可用。

Title: Don't Take Things Out of Context: Attention Intervention for Enhancing Chain-of-Thought Reasoning in Large Language Models

• Authors: Shaotian Yan, Chen Shen, Wenxiao Wang, Liang Xie, Junjie Liu, Jieping Ye
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.11154
• Pdf URL: https://arxiv.org/pdf/2503.11154
• Copy Paste: [[2503.11154]] Don't Take Things Out of Context: Attention Intervention for Enhancing Chain-of-Thought Reasoning in Large Language Models(https://arxiv.org/abs/2503.11154)
• Keywords: language model, llm, chain-of-thought
• Abstract: Few-shot Chain-of-Thought (CoT) significantly enhances the reasoning capabilities of large language models (LLMs), functioning as a whole to guide these models in generating reasoning steps toward final answers. However, we observe that isolated segments, words, or tokens within CoT demonstrations can unexpectedly disrupt the generation process of LLMs. The model may overly concentrate on certain local information present in the demonstration, introducing irrelevant noise into the reasoning process and potentially leading to incorrect answers. In this paper, we investigate the underlying mechanism of CoT through dynamically tracing and manipulating the inner workings of LLMs at each output step, which demonstrates that tokens exhibiting specific attention characteristics are more likely to induce the model to take things out of context; these tokens directly attend to the hidden states tied with prediction, without substantial integration of non-local information. Building upon these insights, we propose a Few-shot Attention Intervention method (FAI) that dynamically analyzes the attention patterns of demonstrations to accurately identify these tokens and subsequently make targeted adjustments to the attention weights to effectively suppress their distracting effect on LLMs. Comprehensive experiments across multiple benchmarks demonstrate consistent improvements over baseline methods, with a remarkable 5.91% improvement on the AQuA dataset, further highlighting the effectiveness of FAI.
• 摘要：很少有思想链（COT）显着增强了大语言模型（LLMS）的推理能力，从而使整个功能指导这些模型在为最终答案中生成推理步骤。但是，我们观察到，COT演示中的孤立段，单词或令牌会意外破坏LLM的生成过程。该模型可能会过度集中于演示中存在的某些局部信息，从而将无关的噪声引入推理过程中，并可能导致错误的答案。在本文中，我们通过在每个输出步骤中动态追踪和操纵LLM的内部工作来研究COT的潜在机制，这表明表现出特定注意力特征的令牌更有可能诱导模型将事物脱离上下文；这些代币直接关注与预测相关的隐藏状态，而没有实质性的非本地信息。在这些见解的基础上，我们提出了一些注意力干预方法（FAI），该方法动态分析了示范的注意力模式，以准确识别这些令牌，然后对注意力重量进行有针对性的调整，以有效地抑制其分散注意力对LLM的分心效果。跨多个基准测试的全面实验表现出比基线方法的一致改进，在Aqua数据集上有5.91％的改善，进一步强调了FAI的有效性。

Title: Towards Extreme Pruning of LLMs with Plug-and-Play Mixed Sparsity

• Authors: Chi Xu, Gefei Zhang, Yantong Zhu, Luca Benini, Guosheng Hu, Yawei Li, Zhihong Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.11164
• Pdf URL: https://arxiv.org/pdf/2503.11164
• Copy Paste: [[2503.11164]] Towards Extreme Pruning of LLMs with Plug-and-Play Mixed Sparsity(https://arxiv.org/abs/2503.11164)
• Keywords: language model, llm
• Abstract: N:M structured pruning is essential for large language models (LLMs) because it can remove less important network weights and reduce the memory and computation requirements. Existing pruning methods mainly focus on designing metrics to measure the importance of network components to guide pruning. Apart from the impact of these metrics, we observe that different layers have different sensitivities over the network performance. Thus, we propose an efficient method based on the trace of Fisher Information Matrix (FIM) to quantitatively measure and verify the different sensitivities across layers. Based on this, we propose Mixed Sparsity Pruning (MSP) which uses a pruning-oriented evolutionary algorithm (EA) to determine the optimal sparsity levels for different layers. To guarantee fast convergence and achieve promising performance, we utilize efficient FIM-inspired layer-wise sensitivity to initialize the population of EA. In addition, our MSP can work as a plug-and-play module, ready to be integrated into existing pruning methods. Extensive experiments on LLaMA and LLaMA-2 on language modeling and zero-shot tasks demonstrate our superior performance. In particular, in extreme pruning ratio (e.g. 75%), our method significantly outperforms existing methods in terms of perplexity (PPL) by orders of magnitude (Figure 1).
• 摘要：N：M结构化修剪对于大语言模型（LLMS）至关重要，因为它可以消除不太重要的网络权重并减少内存和计算要求。现有的修剪方法主要集中于设计指标，以衡量网络组件指导修剪的重要性。除了这些指标的影响外，我们还观察到不同层对网络性能具有不同的敏感性。因此，我们提出了一种基于Fisher信息矩阵（FIM）的痕迹的有效方法，以定量测量和验证跨层的不同敏感性。基于此，我们提出了使用面向修剪的进化算法（EA）来确定不同层的最佳稀疏水平的混合稀疏修剪（MSP）。为了确保快速收敛并实现有希望的性能，我们利用有效的FIM启发的层敏感性来初始化EA的种群。此外，我们的MSP可以用作插件模块，准备集成到现有的修剪方法中。关于Llama和Llama-2关于语言建模和零击任务的广泛实验证明了我们的出色表现。特别是，以极端修剪比（例如75％），我们的方法按照数量级在困惑（PPL）方面显着优于现有方法（图1）。

Title: DeskVision: Large Scale Desktop Region Captioning for Advanced GUI Agents

• Authors: Yibin Xu, Liang Yang, Hao Chen, Hua Wang, Zhi Chen, Yaohua Tang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.11170
• Pdf URL: https://arxiv.org/pdf/2503.11170
• Copy Paste: [[2503.11170]] DeskVision: Large Scale Desktop Region Captioning for Advanced GUI Agents(https://arxiv.org/abs/2503.11170)
• Keywords: language model, agent
• Abstract: The limitation of graphical user interface (GUI) data has been a significant barrier to the development of GUI agents today, especially for the desktop / computer use scenarios. To address this, we propose an automated GUI data generation pipeline, AutoCaptioner, which generates data with rich descriptions while minimizing human effort. Using AutoCaptioner, we created a novel large-scale desktop GUI dataset, DeskVision, along with the largest desktop test benchmark, DeskVision-Eval, which reflects daily usage and covers diverse systems and UI elements, each with rich descriptions. With DeskVision, we train a new GUI understanding model, GUIExplorer. Results show that GUIExplorer achieves state-of-the-art (SOTA) performance in understanding/grounding visual elements without the need for complex architectural designs. We further validated the effectiveness of the DeskVision dataset through ablation studies on various large visual language models (LVLMs). We believe that AutoCaptioner and DeskVision will significantly advance the development of GUI agents, and will open-source them for the community.
• 摘要：图形用户界面（GUI）数据的局限性一直是当今GUI代理开发的重要障碍，尤其是对于桌面 /计算机使用方案。为了解决这个问题，我们提出了一个自动GUI数据生成管道AutoCaptioner，该管道以丰富的描述生成数据，同时最大程度地减少人类的努力。我们使用自动关注器创建了一个新颖的大型桌面GUI数据集，Deskvision，以及最大的桌面测试基准，Deskvision-Eval，它反映了每日使用情况，并涵盖了各种系统和UI元素，每个元素都有丰富的描述。借助Deskvision，我们培训了新的GUI理解模型，GuiexPlorer。结果表明，GuiexPlorer在理解/接地视觉元素方面实现了最新的（SOTA）性能，而无需复杂的建筑设计。我们通过对各种大型视觉语言模型（LVLM）的消融研究进一步验证了Deskvision数据集的有效性。我们认为，自动关闭器和台式将大大推动GUI代理的开发，并将为社区开放源。

Title: Palette of Language Models: A Solver for Controlled Text Generation

• Authors: Zhe Yang, Yi Huang, Yaqin Chen, Xiaoting Wu, Junlan Feng, Chao Deng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.11182
• Pdf URL: https://arxiv.org/pdf/2503.11182
• Copy Paste: [[2503.11182]] Palette of Language Models: A Solver for Controlled Text Generation(https://arxiv.org/abs/2503.11182)
• Keywords: language model, prompt
• Abstract: Recent advancements in large language models have revolutionized text generation with their remarkable capabilities. These models can produce controlled texts that closely adhere to specific requirements when prompted appropriately. However, designing an optimal prompt to control multiple attributes simultaneously can be challenging. A common approach is to linearly combine single-attribute models, but this strategy often overlooks attribute overlaps and can lead to conflicts. Therefore, we propose a novel combination strategy inspired by the Law of Total Probability and Conditional Mutual Information Minimization on generative language models. This method has been adapted for single-attribute control scenario and is termed the Palette of Language Models due to its theoretical linkage between attribute strength and generation style, akin to blending colors on an artist's palette. Moreover, positive correlation and attribute enhancement are advanced as theoretical properties to guide a rational combination strategy design. We conduct experiments on both single control and multiple control settings, and achieve surpassing results.
• 摘要：大型语言模型的最新进展已彻底改变了文本生成的能力。当适当提示时，这些模型可以产生受控文本，这些文本紧密遵守特定要求。但是，设计最佳提示以同时控制多个属性可能是具有挑战性的。一种常见的方法是将单属性模型进行线性结合，但是该策略通常会忽略属性重叠，并可能导致冲突。因此，我们提出了一种新的组合策略，该策略受到生成语言模型的总概率和有条件相互信息的定律的启发。该方法已适用于单属性控制方案，由于其属性强度和发电样式之间的理论链接，类似于艺术家调色板上的颜色，因此被称为语言模型的调色板。此外，正面相关性和属性增强是一种理论特性，可指导合理组合策略设计。我们对单个控制和多个控制设置进行实验，并取得超越结果。

Title: Line of Duty: Evaluating LLM Self-Knowledge via Consistency in Feasibility Boundaries

• Authors: Sahil Kale, Vijaykant Nadadur
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.11256
• Pdf URL: https://arxiv.org/pdf/2503.11256
• Copy Paste: [[2503.11256]] Line of Duty: Evaluating LLM Self-Knowledge via Consistency in Feasibility Boundaries(https://arxiv.org/abs/2503.11256)
• Keywords: gpt, llm
• Abstract: As LLMs grow more powerful, their most profound achievement may be recognising when to say "I don't know". Existing studies on LLM self-knowledge have been largely constrained by human-defined notions of feasibility, often neglecting the reasons behind unanswerability by LLMs and failing to study deficient types of self-knowledge. This study aims to obtain intrinsic insights into different types of LLM self-knowledge with a novel methodology: allowing them the flexibility to set their own feasibility boundaries and then analysing the consistency of these limits. We find that even frontier models like GPT-4o and Mistral Large are not sure of their own capabilities more than 80% of the time, highlighting a significant lack of trustworthiness in responses. Our analysis of confidence balance in LLMs indicates that models swing between overconfidence and conservatism in feasibility boundaries depending on task categories and that the most significant self-knowledge weaknesses lie in temporal awareness and contextual understanding. These difficulties in contextual comprehension additionally lead models to question their operational boundaries, resulting in considerable confusion within the self-knowledge of LLMs. We make our code and results available publicly at this https URL
• 摘要：随着LLM越来越强大，他们最深刻的成就可能会认识到何时说“我不知道”。现有关于LLM自我知识的研究在很大程度上受到了人为定义的可行性概念的限制，经常忽略LLMS无法解决性的原因，而没有研究不足类型的自我知识。这项研究旨在通过一种新的方法来获得对不同类型的LLM自我知识的内在见解：允许他们灵活地设置自己的可行性界限，然后分析这些限制的一致性。我们发现，即使像GPT-4O和Mismtral大型的Frontier模型也不确定自己的功能超过80％，这强调了反应的信任性不足。我们对LLMS的信心平衡的分析表明，模型在可行性边界的过度自信和保守主义之间摆动，具体取决于任务类别，并且最重要的自我知识弱点在于时间意识和上下文理解。这些困难在上下文理解中还导致模型质疑其运营界限，从而在LLM的自我知识中造成了相当大的困惑。我们在此HTTPS URL上公开提供代码和结果

Title: High-Dimensional Interlingual Representations of Large Language Models

• Authors: Bryan Wilie, Samuel Cahyawijaya, Junxian He, Pascale Fung
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.11280
• Pdf URL: https://arxiv.org/pdf/2503.11280
• Copy Paste: [[2503.11280]] High-Dimensional Interlingual Representations of Large Language Models(https://arxiv.org/abs/2503.11280)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) trained on massive multilingual datasets hint at the formation of interlingual constructs--a shared subspace in the representation space. However, evidence regarding this phenomenon is mixed, leaving it unclear whether these models truly develop unified interlingual representations, or present a partially aligned constructs. We explore 31 diverse languages varying on their resource-levels, typologies, and geographical regions; and find that multilingual LLMs exhibit inconsistent cross-lingual alignments. To address this, we propose an interlingual representation framework identifying both the shared interlingual semantic subspace and fragmented components, existed due to representational limitations. We introduce Interlingual Local Overlap (ILO) score to quantify interlingual alignment by comparing the local neighborhood structures of high-dimensional representations. We utilize ILO to investigate the impact of single-language fine-tuning on the interlingual representations in multilingual LLMs. Our results indicate that training exclusively on a single language disrupts the alignment in early layers, while freezing these layers preserves the alignment of interlingual representations, leading to improved cross-lingual generalization. These results validate our framework and metric for evaluating interlingual representation, and further underscore that interlingual alignment is crucial for scalable multilingual learning.
• 摘要：在大规模的多语言数据集上训练的大型语言模型（LLM）暗示了语言构造的形成 - 表示在表示空间中的共享子空间。但是，关于这种现象的证据混合在一起，尚不清楚这些模型是真正发展统一的座话代表，还是呈现部分对齐的结构。我们探索31种不同的语言，其资源级别，类型和地理区域各不相同；并发现多语言LLM表现出不一致的跨语性对准。为了解决这个问题，我们提出了一个介入表示框架，以识别由于代表性限制而存在的共享语言语义子空间和分散的组件。我们介绍了座谈性局部重叠（ILO）评分，以通过比较高维表示的局部邻域结构来量化语言对齐。我们利用ILO来研究单语言微调对多语言LLM中语言表示的影响。我们的结果表明，仅对单一语言进行培训会破坏早期层次的对齐，同时冻结这些层保留了语言表示的对齐，从而改善了跨语性的概括。这些结果验证了我们的框架和度量标准，用于评估室界表示，并进一步强调了上型对准对于可扩展的多语言学习至关重要。

Title: BriLLM: Brain-inspired Large Language Model

• Authors: Hai Zhao, Hongqiu Wu, Dongjie Yang, Anni Zou, Jiale Hong
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.11299
• Pdf URL: https://arxiv.org/pdf/2503.11299
• Copy Paste: [[2503.11299]] BriLLM: Brain-inspired Large Language Model(https://arxiv.org/abs/2503.11299)
• Keywords: language model, gpt, llm
• Abstract: This paper reports the first brain-inspired large language model (BriLLM). This is a non-Transformer, non-GPT, non-traditional machine learning input-output controlled generative language model. The model is based on the Signal Fully-connected flowing (SiFu) definition on the directed graph in terms of the neural network, and has the interpretability of all nodes on the graph of the whole model, instead of the traditional machine learning model that only has limited interpretability at the input and output ends. In the language model scenario, the token is defined as a node in the graph. A randomly shaped or user-defined signal flow flows between nodes on the principle of "least resistance" along paths. The next token or node to be predicted or generated is the target of the signal flow. As a language model, BriLLM theoretically supports infinitely long $n$-gram models when the model size is independent of the input and predicted length of the model. The model's working signal flow provides the possibility of recall activation and innate multi-modal support similar to the cognitive patterns of the human brain. At present, we released the first BriLLM version in Chinese, with 4000 tokens, 32-dimensional node width, 16-token long sequence prediction ability, and language model prediction performance comparable to GPT-1. More computing power will help us explore the infinite possibilities depicted above.
• 摘要：本文报告了第一个受脑启发的大语言模型（Brillm）。这是一个非转变器，非GPT，非传统机器学习输入输出控制的生成语言模型。该模型基于在神经网络方面的信号完全连接的流动（SIFU）定义，并且具有整个模型图上所有节点的解释性，而不是传统的机器学习模型，而传统的机器学习模型仅在输入和输出端具有有限的可解释性。在语言模型方案中，令牌定义为图中的节点。沿路径“最小电阻”的原理，一个随机形状或用户定义的信号流在节点之间流动。要预测或生成的下一个令牌或节点是信号流的目标。作为一种语言模型，当模型大小独立于模型的输入和预测长度时，Brillm在理论上支持无限长的$ n $ gram模型。该模型的工作信号流提供了召回激活和先天多模式支持的可能性，类似于人脑的认知模式。目前，我们以中文发布了第一个Brillm版本，具有4000个令牌，32维节点宽度，16句长序列预测能力和语言模型预测性能与GPT-1相当。更多的计算能力将有助于我们探索上面描述的无限可能性。

Title: GNNs as Predictors of Agentic Workflow Performances

• Authors: Yuanshuo Zhang, Yuchen Hou, Bohan Tang, Shuo Chen, Muhan Zhang, Xiaowen Dong, Siheng Chen
• Subjects: cs.CL, cs.MA
• Abstract URL: https://arxiv.org/abs/2503.11301
• Pdf URL: https://arxiv.org/pdf/2503.11301
• Copy Paste: [[2503.11301]] GNNs as Predictors of Agentic Workflow Performances(https://arxiv.org/abs/2503.11301)
• Keywords: language model, llm, agent
• Abstract: Agentic workflows invoked by Large Language Models (LLMs) have achieved remarkable success in handling complex tasks. However, optimizing such workflows is costly and inefficient in real-world applications due to extensive invocations of LLMs. To fill this gap, this position paper formulates agentic workflows as computational graphs and advocates Graph Neural Networks (GNNs) as efficient predictors of agentic workflow performances, avoiding repeated LLM invocations for evaluation. To empirically ground this position, we construct FLORA-Bench, a unified platform for benchmarking GNNs for predicting agentic workflow performances. With extensive experiments, we arrive at the following conclusion: GNNs are simple yet effective predictors. This conclusion supports new applications of GNNs and a novel direction towards automating agentic workflow optimization. All codes, models, and data are available at this https URL.
• 摘要：大型语言模型（LLMS）调用的代理工作流在处理复杂任务方面取得了巨大的成功。但是，由于LLM的广泛调用，在现实世界中，优化此类工作流程的成本高昂且效率低下。为了填补这一空白，该位置纸张将代理工作流程作为计算图和倡导者图形神经网络（GNN）作为代理工作流程的有效预测指标，避免重复进行评估的LLM调用。为了基于这个职位，我们构建了Flora Bench，这是一个基准为GNN进行基准预测代理工作流程的统一平台。通过广泛的实验，我们得出以下结论：GNN是简单而有效的预测因素。该结论支持GNN的新应用，以及自动化代理工作流优化的新方向。所有代码，模型和数据均可在此HTTPS URL上获得。

Title: Are formal and functional linguistic mechanisms dissociated?

• Authors: Michael Hanna, Sandro Pezzelle, Yonatan Belinkov
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.11302
• Pdf URL: https://arxiv.org/pdf/2503.11302
• Copy Paste: [[2503.11302]] Are formal and functional linguistic mechanisms dissociated?(https://arxiv.org/abs/2503.11302)
• Keywords: language model, llm
• Abstract: Although large language models (LLMs) are increasingly capable, these capabilities are unevenly distributed: they excel at formal linguistic tasks, such as producing fluent, grammatical text, but struggle more with functional linguistic tasks like reasoning and consistent fact retrieval. Inspired by neuroscience, recent work suggests that to succeed on both formal and functional linguistic tasks, LLMs should use different mechanisms for each; such localization could either be built-in or emerge spontaneously through training. In this paper, we ask: do current models, with fast-improving functional linguistic abilities, exhibit distinct localization of formal and functional linguistic mechanisms? We answer this by finding and comparing the "circuits", or minimal computational subgraphs, responsible for various formal and functional tasks. Comparing 5 LLMs across 10 distinct tasks, we find that while there is indeed little overlap between circuits for formal and functional tasks, there is also little overlap between formal linguistic tasks, as exists in the human brain. Thus, a single formal linguistic network, unified and distinct from functional task circuits, remains elusive. However, in terms of cross-task faithfulness - the ability of one circuit to solve another's task - we observe a separation between formal and functional mechanisms, suggesting that shared mechanisms between formal tasks may exist.
• 摘要：尽管大型语言模型（LLM）的能力越来越多，但这些功能分布不均：它们在形式上的语言任务中表现出色，例如产生流利的语法文本，但在推理和始终如一的事实检索等功能性语言任务中挣扎。受神经科学的启发，最近的工作表明，要完成正式和功能性语言任务，LLMS应为每种任务使用不同的机制。这种本地化可以通过培训自发内置或自发出现。在本文中，我们问：具有快速改善的功能性语言能力的当前模型是否表现出正式和功能性语言机制的独特定位？我们通过查找和比较负责各种正式和功能任务的“电路”或最小计算子图来回答这一点。比较了在10个不同的任务中进行5个LLM，我们发现，正式和功能性任务的电路之间确实几乎没有重叠，但正式的语言任务之间也几乎没有重叠，因为人类大脑中存在。因此，统一的单一形式语言网络与功能任务电路不同，仍然难以捉摸。但是，就交叉任务忠诚而言，一个电路解决另一个任务的能力 - 我们观察到形式和功能机制之间的分离，这表明可能存在形式任务之间的共同机制。

Title: Unlocking General Long Chain-of-Thought Reasoning Capabilities of Large Language Models via Representation Engineering

• Authors: Xinyu Tang, Xiaolei Wang, Zhihao Lv, Yingqian Min, Wayne Xin Zhao, Binbin Hu, Ziqi Liu, Zhiqiang Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.11314
• Pdf URL: https://arxiv.org/pdf/2503.11314
• Copy Paste: [[2503.11314]] Unlocking General Long Chain-of-Thought Reasoning Capabilities of Large Language Models via Representation Engineering(https://arxiv.org/abs/2503.11314)
• Keywords: language model, llm, chain-of-thought
• Abstract: Recent advancements in long chain-of-thoughts(long CoTs) have significantly improved the reasoning capabilities of large language models(LLMs). Existing work finds that the capability of long CoT reasoning can be efficiently elicited by tuning on only a few examples and can easily transfer to other tasks. This motivates us to investigate whether long CoT reasoning is a general capability for LLMs. In this work, we conduct an empirical analysis for this question from the perspective of representation. We find that LLMs do encode long CoT reasoning as a general capability, with a clear distinction from vanilla CoTs. Furthermore, domain-specific representations are also required for the effective transfer of long CoT reasoning. Inspired by these findings, we propose GLoRE, a novel representation engineering method to unleash the general long CoT reasoning capabilities of LLMs. Extensive experiments demonstrate the effectiveness and efficiency of GLoRE in both in-domain and cross-domain scenarios.
• 摘要：长期思想链（长COT）的最新进展显着提高了大语言模型（LLMS）的推理能力。现有的工作发现，只需调整几个示例就可以有效地引起长床推理的能力，并且可以轻松地转移到其他任务中。这促使我们调查了长长的COT推理是否是LLM的一般能力。在这项工作中，我们从代表的角度对这个问题进行了经验分析。我们发现LLMS确实将长的Cot推理编码为一般能力，并且与香草Cots具有明显的区别。此外，有效传递长COT推理也需要特定于域特异性的表示。受这些发现的启发，我们提出了Glore，这是一种新颖的表示工程方法，可以释放LLMS的一般长科克推理能力。广泛的实验证明了在内域和跨域情景中玻璃的有效性和效率。

Title: Rule-Guided Feedback: Enhancing Reasoning by Enforcing Rule Adherence in Large Language Models

• Authors: Aissatou Diallo, Antonis Bikakis, Luke Dickens, Anthony Hunter, Rob Miller
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.11336
• Pdf URL: https://arxiv.org/pdf/2503.11336
• Copy Paste: [[2503.11336]] Rule-Guided Feedback: Enhancing Reasoning by Enforcing Rule Adherence in Large Language Models(https://arxiv.org/abs/2503.11336)
• Keywords: language model, llm
• Abstract: In this paper, we introduce Rule-Guided Feedback (RGF), a framework designed to enhance Large Language Model (LLM) performance through structured rule adherence and strategic information seeking. RGF implements a teacher-student paradigm where rule-following is forced through established guidelines. Our framework employs a Teacher model that rigorously evaluates each student output against task-specific rules, providing constructive guidance rather than direct answers when detecting deviations. This iterative feedback loop serves two crucial purposes: maintaining solutions within defined constraints and encouraging proactive information seeking to resolve uncertainties. We evaluate RGF on diverse tasks including Checkmate-in-One puzzles, Sonnet Writing, Penguins-In-a-Table classification, GSM8k, and StrategyQA. Our findings suggest that structured feedback mechanisms can significantly enhance LLMs' performance across various domains.
• 摘要：在本文中，我们介绍了规则引导的反馈（RGF），该反馈（RGF）旨在通过结构化规则遵守和战略信息寻求来增强大型语言模型（LLM）绩效。 RGF实现了一个教师范式，在该范式中，通过既定准则强迫规则遵守。我们的框架采用了一种教师模型，该模型严格地根据特定于任务的规则评估每个学生的成果，在检测偏差时提供建设性的指导，而不是直接答案。这种迭代反馈循环具有两个至关重要的目的：将解决方案保持在定义的约束中，并鼓励寻求解决不确定性的积极主动信息。我们评估RGF的各种任务，包括拼图，十四行诗写作，企鹅在桌上的分类，GSM8K和Strategyqa。我们的发现表明，结构化反馈机制可以显着提高LLM在各个领域的性能。

Title: AIstorian lets AI be a historian: A KG-powered multi-agent system for accurate biography generation

• Authors: Fengyu Li (1), Yilin Li (1), Junhao Zhu (1), Lu Chen (1), Yanfei Zhang (1), Jia Zhou (1), Hui Zu (1), Jingwen Zhao (2), Yunjun Gao (1) ((1) Zhejiang University, (2) Poisson Lab, Huawei)
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.11346
• Pdf URL: https://arxiv.org/pdf/2503.11346
• Copy Paste: [[2503.11346]] AIstorian lets AI be a historian: A KG-powered multi-agent system for accurate biography generation(https://arxiv.org/abs/2503.11346)
• Keywords: language model, llm, hallucination, retrieval-augmented generation, agent
• Abstract: Huawei has always been committed to exploring the AI application in historical research. Biography generation, as a specialized form of abstractive summarization, plays a crucial role in historical research but faces unique challenges that existing large language models (LLMs) struggle to address. These challenges include maintaining stylistic adherence to historical writing conventions, ensuring factual fidelity, and handling fragmented information across multiple documents. We present AIstorian, a novel end-to-end agentic system featured with a knowledge graph (KG)-powered retrieval-augmented generation (RAG) and anti-hallucination multi-agents. Specifically, AIstorian introduces an in-context learning based chunking strategy and a KG-based index for accurate and efficient reference retrieval. Meanwhile, AIstorian orchestrates multi-agents to conduct on-the-fly hallucination detection and error-type-aware correction. Additionally, to teach LLMs a certain language style, we finetune LLMs based on a two-step training approach combining data augmentation-enhanced supervised fine-tuning with stylistic preference optimization. Extensive experiments on a real-life historical Jinshi dataset demonstrate that AIstorian achieves a 3.8x improvement in factual accuracy and a 47.6% reduction in hallucination rate compared to existing baselines. The data and code are available at: this https URL.
• 摘要：华为一直致力于探索历史研究中的AI应用。作为抽象性摘要的一种专业形式，传记一代在历史研究中起着至关重要的作用，但面临现有大型语言模型（LLMS）努力解决的独特挑战。这些挑战包括维持对历史写作惯例的遵守，确保事实忠诚度以及在多个文档中处理零散的信息。我们提出了Aistorian，这是一种新颖的端到端代理系统，具有知识图（KG）功率的检索效果（rag）和抗凝固性多代理。具体而言，Aistorian引入了基于秘密学习的块策略和基于KG的索引，以进行准确有效的参考检索。同时，Aistorian策划了多代理，以进行幻觉幻觉检测和错误类型的校正。此外，为了教LLM的某种语言风格，我们基于两步训练方法的Finetune LLM，将数据增强增强的监督微调与风格偏好优化相结合。对现实历史的金希数据集进行了广泛的实验表明，与现有基线相比，Aistorian的事实准确性提高了3.8倍，幻觉率降低了47.6％。数据和代码可在以下网址提供：此HTTPS URL。

Title: RESPONSE: Benchmarking the Ability of Language Models to Undertake Commonsense Reasoning in Crisis Situation

• Authors: Aissatou Diallo, Antonis Bikakis, Luke Dickens, Anthony Hunter, Rob Miller
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.11348
• Pdf URL: https://arxiv.org/pdf/2503.11348
• Copy Paste: [[2503.11348]] RESPONSE: Benchmarking the Ability of Language Models to Undertake Commonsense Reasoning in Crisis Situation(https://arxiv.org/abs/2503.11348)
• Keywords: language model, gpt, llm
• Abstract: An interesting class of commonsense reasoning problems arises when people are faced with natural disasters. To investigate this topic, we present \textsf{RESPONSE}, a human-curated dataset containing 1789 annotated instances featuring 6037 sets of questions designed to assess LLMs' commonsense reasoning in disaster situations across different time frames. The dataset includes problem descriptions, missing resources, time-sensitive solutions, and their justifications, with a subset validated by environmental engineers. Through both automatic metrics and human evaluation, we compare LLM-generated recommendations against human responses. Our findings show that even state-of-the-art models like GPT-4 achieve only 37\% human-evaluated correctness for immediate response actions, highlighting significant room for improvement in LLMs' ability for commonsense reasoning in crises.
• 摘要：当人们面临自然灾害时，就会出现一系列有趣的常识性推理问题。为了调查此主题，我们提出了\ textsf {reverse}，这是一个由人类策划的数据集，其中包含1789个注释实例，其中包含6037个问题集，这些问题旨在评估LLMS在不同时间范围内的灾难情况下的常识性推理。该数据集包括问题描述，缺少的资源，时间敏感的解决方案及其理由，并由环境工程师验证的子集。通过自动指标和人类评估，我们将LLM生成的建议与人类反应进行了比较。我们的发现表明，即使像GPT-4这样的最新模型也仅实现37 \％的人为评估的正确性，以立即做出响应动作，这突出了LLMS在危机中的常识性推理能力的重大空间。

Title: Annotating Scientific Uncertainty: A comprehensive model using linguistic patterns and comparison with existing approaches

• Authors: Panggih Kusuma Ningrum, Philipp Mayr, Nina Smirnova, Iana Atanassova
• Subjects: cs.CL, cs.AI, cs.DL
• Abstract URL: https://arxiv.org/abs/2503.11376
• Pdf URL: https://arxiv.org/pdf/2503.11376
• Copy Paste: [[2503.11376]] Annotating Scientific Uncertainty: A comprehensive model using linguistic patterns and comparison with existing approaches(https://arxiv.org/abs/2503.11376)
• Keywords: language model, llm
• Abstract: UnScientify, a system designed to detect scientific uncertainty in scholarly full text. The system utilizes a weakly supervised technique to identify verbally expressed uncertainty in scientific texts and their authorial references. The core methodology of UnScientify is based on a multi-faceted pipeline that integrates span pattern matching, complex sentence analysis and author reference checking. This approach streamlines the labeling and annotation processes essential for identifying scientific uncertainty, covering a variety of uncertainty expression types to support diverse applications including information retrieval, text mining and scientific document processing. The evaluation results highlight the trade-offs between modern large language models (LLMs) and the UnScientify system. UnScientify, which employs more traditional techniques, achieved superior performance in the scientific uncertainty detection task, attaining an accuracy score of 0.808. This finding underscores the continued relevance and efficiency of UnScientify's simple rule-based and pattern matching strategy for this specific application. The results demonstrate that in scenarios where resource efficiency, interpretability, and domain-specific adaptability are critical, traditional methods can still offer significant advantages.
• 摘要：Uncientify，该系统旨在检测学术全文中的科学不确定性。该系统利用一种弱监督的技术来识别科学文本及其作者参考的口头表达的不确定性。 Unscientify的核心方法基于多面管道，该管道整合了跨度模式匹配，复杂的句子分析和作者参考检查。这种方法简化了标签和注释过程，对于识别科学不确定性至关重要，涵盖了各种不确定性表达类型，以支持各种应用程序，包括信息检索，文本挖掘和科学文档处理。评估结果突出了现代大型语言模型（LLM）与非分散系统之间的权衡。采用更多传统技术的Unscientify在科学不确定性检测任务中取得了卓越的性能，获得了0.808的精度得分。这一发现强调了该特定应用程序的Unscientify简单基于规则和模式匹配策略的持续相关性和效率。结果表明，在资源效率，可解释性和特定领域的适应性至关重要的情况下，传统方法仍然可以提供显着优势。

Title: Modeling Subjectivity in Cognitive Appraisal with Language Models

• Authors: Yuxiang Zhou, Hainiu Xu, Desmond C. Ong, Petr Slovak, Yulan He
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.11381
• Pdf URL: https://arxiv.org/pdf/2503.11381
• Copy Paste: [[2503.11381]] Modeling Subjectivity in Cognitive Appraisal with Language Models(https://arxiv.org/abs/2503.11381)
• Keywords: language model, llm, prompt
• Abstract: As the utilization of language models in interdisciplinary, human-centered studies grow, the expectation of model capabilities continues to evolve. Beyond excelling at conventional tasks, models are recently expected to perform well on user-centric measurements involving confidence and human (dis)agreement -- factors that reflect subjective preferences. While modeling of subjectivity plays an essential role in cognitive science and has been extensively studied, it remains under-explored within the NLP community. In light of this gap, we explore how language models can harness subjectivity by conducting comprehensive experiments and analysis across various scenarios using both fine-tuned models and prompt-based large language models (LLMs). Our quantitative and qualitative experimental results indicate that existing post-hoc calibration approaches often fail to produce satisfactory results. However, our findings reveal that personality traits and demographical information are critical for measuring subjectivity. Furthermore, our in-depth analysis offers valuable insights for future research and development in the interdisciplinary studies of NLP and cognitive science.
• 摘要：随着语言模型在跨学科，以人为中心的研究中的利用，模型能力的期望不断发展。除了在常规任务上表现出色外，最近预计模型在涉及信心和人类（DIS）协议的以用户为中心的测量中表现良好 - 反映主观偏好的因素。尽管对主观性的建模在认知科学中起着至关重要的作用，并且已经对其进行了广泛的研究，但在NLP社区中仍未探索。鉴于这一差距，我们探讨了语言模型如何通过使用微调模型和及时的基于基于的大语言模型（LLMS）进行各种方案进行全面的实验和分析来利用主观性。我们的定量和定性实验结果表明，现有的事后校准方法通常无法产生令人满意的结果。但是，我们的发现表明人格特征和人口统计信息对于衡量主观性至关重要。此外，我们的深入分析为NLP和认知科学的跨学科研究提供了宝贵的见解。

Title: Text Compression for Efficient Language Generation

• Authors: David Gu, Peter Belcak, Roger Wattenhofer
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.11426
• Pdf URL: https://arxiv.org/pdf/2503.11426
• Copy Paste: [[2503.11426]] Text Compression for Efficient Language Generation(https://arxiv.org/abs/2503.11426)
• Keywords: language model, gpt, llm
• Abstract: We challenge the prevailing assumption that LLMs must rely fully on sub-word tokens for high-quality text generation. To this end, we propose the "Generative Pretrained Thoughtformer" (GPTHF), a hierarchical transformer language model capable of text generation by compressing text into sentence embeddings and employing a sentence attention mechanism. GPTHF retains GPT's architecture, modifying only token interactions via dynamic sparse attention masks. Our experiments show that GPTHF achieves an up to an order of magnitude improvement in FLOPs efficiency and a threefold increase in runtime speed compared to equally-sized GPT models in the low-size regime. This is achieved through a unique generation method that caches and reuses sentence embeddings, allowing significant portions of the input to bypass large parts of the network.
• 摘要：我们挑战了普遍的假设，即LLM必须完全依靠子字代币进行高质量的文本生成。为此，我们提出了“生成预验证的思想形式”（GPTHF），这是一种层次变压器语言模型，能够通过将文本压缩到句子嵌入并采用句子注意机制来生成文本。 GPTHF保留了GPT的体系结构，仅通过动态稀疏注意性掩码修改令牌交互。我们的实验表明，与低尺寸的机制中，GPTHF达到了掉落效率提高的数量级提高和运行时速度的提高。这是通过一种独特的生成方法来实现的，该方法缓存和重复使用句子嵌入，从而允许输入的大部分绕过网络的大部分。

Title: TikZero: Zero-Shot Text-Guided Graphics Program Synthesis

• Authors: Jonas Belouadi, Eddy Ilg, Margret Keuper, Hideki Tanaka, Masao Utiyama, Raj Dabre, Steffen Eger, Simone Paolo Ponzetto
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2503.11509
• Pdf URL: https://arxiv.org/pdf/2503.11509
• Copy Paste: [[2503.11509]] TikZero: Zero-Shot Text-Guided Graphics Program Synthesis(https://arxiv.org/abs/2503.11509)
• Keywords: gpt
• Abstract: With the rise of generative AI, synthesizing figures from text captions becomes a compelling application. However, achieving high geometric precision and editability requires representing figures as graphics programs in languages like TikZ, and aligned training data (i.e., graphics programs with captions) remains scarce. Meanwhile, large amounts of unaligned graphics programs and captioned raster images are more readily available. We reconcile these disparate data sources by presenting TikZero, which decouples graphics program generation from text understanding by using image representations as an intermediary bridge. It enables independent training on graphics programs and captioned images and allows for zero-shot text-guided graphics program synthesis during inference. We show that our method substantially outperforms baselines that can only operate with caption-aligned graphics programs. Furthermore, when leveraging caption-aligned graphics programs as a complementary training signal, TikZero matches or exceeds the performance of much larger models, including commercial systems like GPT-4o. Our code, datasets, and select models are publicly available.
• 摘要：随着生成AI的兴起，文本字幕中的合成数字成为一个令人信服的应用程序。但是，实现高几何精度和编辑性需要将数字表示为Tikz等语言的图形程序，并且对齐培训数据（即带有字幕的图形程序）仍然很少。同时，更容易获得大量未对齐的图形程序和字幕栅格图像。我们通过呈现Tikzero来调和这些不同的数据源，该数据源将图形程序生成从文本理解中分解为使用图像表示作为中间桥。它可以对图形程序和字幕图像进行独立的培训，并允许在推理过程中进行零击文本引导的图形程序综合。我们表明，我们的方法基本上优于只能使用字幕一致的图形程序来运行的基准。此外，当利用字幕一致的图形程序作为互补训练信号时，Tikzero匹配或超过了更大模型的性能，包括GPT-4O等商业系统。我们的代码，数据集和选择模型公开可用。

Title: Do Construction Distributions Shape Formal Language Learning In German BabyLMs?

• Authors: Bastian Bunzeck, Daniel Duran, Sina Zarrieß
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.11593
• Pdf URL: https://arxiv.org/pdf/2503.11593
• Copy Paste: [[2503.11593]] Do Construction Distributions Shape Formal Language Learning In German BabyLMs?(https://arxiv.org/abs/2503.11593)
• Keywords: language model
• Abstract: We analyze the influence of utterance-level construction distributions in German child-directed speech on the resulting formal linguistic competence and the underlying learning trajectories for small language models trained on a novel collection of developmentally plausible language data for German. We find that trajectories are surprisingly robust for markedly different distributions of constructions in the training data, which have little effect on final accuracies and almost no effect on global learning trajectories. While syntax learning benefits from more complex utterances, lexical learning culminates in better scores with more fragmentary data. We argue that LMs trained on developmentally plausible data can contribute to debates on how rich or impoverished linguistic stimuli actually are.
• 摘要：我们分析了德国儿童指导语音中的话语水平施工分布对由此产生的形式语言能力的影响，以及针对小型语言模型的基础学习轨迹，该模型训练了新型德语的新型发育中合理的语言数据。我们发现，轨迹对于训练数据中的构造分布明显不同，这对最终精度几乎没有影响，几乎对全球学习轨迹没有影响。尽管语法学习从更复杂的话语中受益，但词汇学习通过更多零碎的数据以更好的分数达到顶点。我们认为，经过开发的合理数据培训的LMS可以为关于语言刺激的富裕或贫困的语言刺激的辩论做出贡献。

Title: Neutralizing Bias in LLM Reasoning using Entailment Graphs

• Authors: Liang Cheng, Tianyi Li, Zhaowei Wang, Tianyang Liu, Mark Steedman
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2503.11614
• Pdf URL: https://arxiv.org/pdf/2503.11614
• Copy Paste: [[2503.11614]] Neutralizing Bias in LLM Reasoning using Entailment Graphs(https://arxiv.org/abs/2503.11614)
• Keywords: llm, hallucination
• Abstract: LLMs are often claimed to be capable of Natural Language Inference (NLI), which is widely regarded as a cornerstone of more complex forms of reasoning. However, recent works show that LLMs still suffer from hallucinations in NLI due to attestation bias, where LLMs overly rely on propositional memory to build shortcuts. To solve the issue, we design an unsupervised framework to construct counterfactual reasoning data and fine-tune LLMs to reduce attestation bias. To measure bias reduction, we build bias-adversarial variants of NLI datasets with randomly replaced predicates in premises while keeping hypotheses unchanged. Extensive evaluations show that our framework can significantly reduce hallucinations from attestation bias. Then, we further evaluate LLMs fine-tuned with our framework on original NLI datasets and their bias-neutralized versions, where original entities are replaced with randomly sampled ones. Extensive results show that our framework consistently improves inferential performance on both original and bias-neutralized NLI datasets.
• 摘要：LLM通常被认为具有自然语言推断（NLI），该推论被广泛认为是更复杂的推理形式的基石。但是，最近的作品表明，由于证明偏见，LLMS仍遭受NLI幻觉的困扰，LLM过分依赖命题记忆来建立捷径。为了解决问题，我们设计了一个无监督的框架来构建反事实推理数据和微调LLM，以减少认证偏见。为了衡量降低偏差，我们构建了NLI数据集的偏见 - 对抗变体，并在前提中随机替换谓词，同时保持假设不变。广泛的评估表明，我们的框架可以大大减少证明偏差的幻觉。然后，我们进一步评估了在原始NLI数据集上的框架及其偏置中和版本的框架中进行了微调，其中原始实体被随机采样替换。广泛的结果表明，我们的框架始终提高原始和偏中和中和NLI数据集的推论性能。