2025-02-13

Title: Intelligent Legal Assistant: An Interactive Clarification System for Legal Question Answering

• Authors: Rujing Yao, Yiquan Wu, Tong Zhang, Xuhui Zhang, Yuting Huang, Yang Wu, Jiayin Yang, Changlong Sun, Fang Wang, Xiaozhong Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.07904
• Pdf URL: https://arxiv.org/pdf/2502.07904
• Copy Paste: [[2502.07904]] Intelligent Legal Assistant: An Interactive Clarification System for Legal Question Answering(https://arxiv.org/abs/2502.07904)
• Keywords: language model
• Abstract: The rise of large language models has opened new avenues for users seeking legal advice. However, users often lack professional legal knowledge, which can lead to questions that omit critical information. This deficiency makes it challenging for traditional legal question-answering systems to accurately identify users' actual needs, often resulting in imprecise or generalized advice. In this work, we develop a legal question-answering system called Intelligent Legal Assistant, which interacts with users to precisely capture their needs. When a user poses a question, the system requests that the user select their geographical location to pinpoint the applicable laws. It then generates clarifying questions and options based on the key information missing from the user's initial question. This allows the user to select and provide the necessary details. Once all necessary information is provided, the system produces an in-depth legal analysis encompassing three aspects: overall conclusion, jurisprudential analysis, and resolution suggestions.
• 摘要：大型语言模型的兴起为用户寻求法律建议开辟了新的途径。然而，用户往往缺乏专业的法律知识，这会导致他们提出的问题遗漏关键信息。这一缺陷使传统的法律问答系统难以准确识别用户的实际需求，往往导致建议不精确或泛泛而谈。在这项工作中，我们开发了一个名为智能法律助理的法律问答系统，该系统与用户交互以精确捕捉他们的需求。当用户提出问题时，系统要求用户选择他们的地理位置以确定适用的法律。然后，它会根据用户初始问题中缺少的关键信息生成澄清问题和选项。这使用户可以选择并提供必要的详细信息。在提供所有必要信息后，系统将生成深入的法律分析，涵盖三个方面：总体结论、法理分析和解决建议。

Title: Elevating Legal LLM Responses: Harnessing Trainable Logical Structures and Semantic Knowledge with Legal Reasoning

• Authors: Rujing Yao, Yang Wu, Chenghao Wang, Jingwei Xiong, Fang Wang, Xiaozhong Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.07912
• Pdf URL: https://arxiv.org/pdf/2502.07912
• Copy Paste: [[2502.07912]] Elevating Legal LLM Responses: Harnessing Trainable Logical Structures and Semantic Knowledge with Legal Reasoning(https://arxiv.org/abs/2502.07912)
• Keywords: language model, llm, hallucination, retrieval-augmented generation
• Abstract: Large Language Models (LLMs) have achieved impressive results across numerous domains, yet they experience notable deficiencies in legal question-answering tasks. LLMs often generate generalized responses that lack the logical specificity required for expert legal advice and are prone to hallucination, providing answers that appear correct but are unreliable. Retrieval-Augmented Generation (RAG) techniques offer partial solutions to address this challenge, but existing approaches typically focus only on semantic similarity, neglecting the logical structure essential to legal reasoning. In this paper, we propose the Logical-Semantic Integration Model (LSIM), a novel supervised framework that bridges semantic and logical coherence. LSIM comprises three components: reinforcement learning predicts a structured fact-rule chain for each question, a trainable Deep Structured Semantic Model (DSSM) retrieves the most relevant candidate questions by integrating semantic and logical features, and in-context learning generates the final answer using the retrieved content. Our experiments on a real-world legal QA dataset-validated through both automated metrics and human evaluation-demonstrate that LSIM significantly enhances accuracy and reliability compared to existing methods.
• 摘要：大型语言模型 (LLM) 已在众多领域取得了令人瞩目的成果，但它们在法律问答任务中存在明显缺陷。LLM 通常会生成泛化响应，缺乏专家法律建议所需的逻辑特异性，并且容易产生幻觉，提供看似正确但不可靠的答案。检索增强生成 (RAG) 技术提供了解决这一挑战的部分解决方案，但现有方法通常仅关注语义相似性，而忽略了法律推理所必需的逻辑结构。在本文中，我们提出了逻辑语义集成模型 (LSIM)，这是一种连接语义和逻辑一致性的新型监督框架。LSIM 由三个部分组成：强化学习预测每个问题的结构化事实规则链，可训练的深度结构化语义模型 (DSSM) 通过整合语义和逻辑特征来检索最相关的候选问题，上下文学习使用检索到的内容生成最终答案。我们对真实世界的法律 QA 数据集进行的实验（通过自动指标和人工评估进行验证）表明，与现有方法相比，LSIM 显著提高了准确性和可靠性。

Title: Adapting Multilingual Embedding Models to Historical Luxembourgish

• Authors: Andrianos Michail, Corina Julia Raclé, Juri Opitz, Simon Clematide
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.07938
• Pdf URL: https://arxiv.org/pdf/2502.07938
• Copy Paste: [[2502.07938]] Adapting Multilingual Embedding Models to Historical Luxembourgish(https://arxiv.org/abs/2502.07938)
• Keywords: gpt
• Abstract: The growing volume of digitized historical texts requires effective semantic search using text embeddings. However, pre-trained multilingual models, typically evaluated on contemporary texts, face challenges with historical digitized content due to OCR noise and outdated spellings. We explore the use of multilingual embeddings for cross-lingual semantic search on historical Luxembourgish, a low-resource language. We collect historical Luxembourgish news articles spanning various time periods and use GPT-4o to segment and translate them into closely related languages, creating 20,000 parallel training sentences per language pair. We further create a historical bitext mining evaluation set and find that these models struggle to perform cross-lingual search on historical Luxembourgish. To address this, we propose a simple adaptation method using in-domain training data, achieving up to 98\% accuracy in cross-lingual evaluations. We release our adapted models and historical Luxembourgish-German/French bitexts to support further research.
• 摘要：数字化历史文本的数量不断增长，需要使用文本嵌入进行有效的语义搜索。然而，预训练的多语言模型（通常在当代文本上进行评估）在处理历史数字化内容时会面临 OCR 噪声和过时拼写的挑战。我们探索使用多语言嵌入对历史卢森堡语（一种资源匮乏的语言）进行跨语言语义搜索。我们收集了跨越不同时期的历史卢森堡语新闻文章，并使用 GPT-4o 将它们分割并翻译成密切相关的语言，为每个语言对创建 20,000 个并行训练句子。我们进一步创建了一个历史双语文本挖掘评估集，发现这些模型很难对历史卢森堡语进行跨语言搜索。为了解决这个问题，我们提出了一种使用领域内训练数据的简单自适应方法，在跨语言评估中实现了高达 98% 的准确率。我们发布了我们的改编模型和历史卢森堡语-德语/法语双语文本，以支持进一步的研究。

Title: Caught in the Web of Words: Do LLMs Fall for Spin in Medical Literature?

• Authors: Hye Sun Yun, Karen Y.C. Zhang, Ramez Kouzy, Iain J. Marshall, Junyi Jessy Li, Byron C. Wallace
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.07963
• Pdf URL: https://arxiv.org/pdf/2502.07963
• Copy Paste: [[2502.07963]] Caught in the Web of Words: Do LLMs Fall for Spin in Medical Literature?(https://arxiv.org/abs/2502.07963)
• Keywords: language model, llm, prompt
• Abstract: Medical research faces well-documented challenges in translating novel treatments into clinical practice. Publishing incentives encourage researchers to present "positive" findings, even when empirical results are equivocal. Consequently, it is well-documented that authors often spin study results, especially in article abstracts. Such spin can influence clinician interpretation of evidence and may affect patient care decisions. In this study, we ask whether the interpretation of trial results offered by Large Language Models (LLMs) is similarly affected by spin. This is important since LLMs are increasingly being used to trawl through and synthesize published medical evidence. We evaluated 22 LLMs and found that they are across the board more susceptible to spin than humans. They might also propagate spin into their outputs: We find evidence, e.g., that LLMs implicitly incorporate spin into plain language summaries that they generate. We also find, however, that LLMs are generally capable of recognizing spin, and can be prompted in a way to mitigate spin's impact on LLM outputs.
• 摘要：医学研究在将新疗法转化为临床实践方面面临着众所周知的挑战。出版激励措施鼓励研究人员提出“积极”的发现，即使经验结果模棱两可。因此，有充分证据表明，作者经常歪曲研究结果，尤其是在文章摘要中。这种歪曲会影响临床医生对证据的解释，并可能影响患者的护理决策。在这项研究中，我们询问大型语言模型 (LLM) 提供的试验结果解释是否同样受到歪曲的影响。这很重要，因为 LLM 越来越多地被用于搜索和综合已发表的医学证据。我们评估了 22 个 LLM，发现它们比人类更容易受到歪曲的影响。它们还可能将歪曲传播到它们的输出中：我们发现证据表明，例如，LLM 隐式地将歪曲纳入它们生成的通俗语言摘要中。然而，我们还发现，LLM 通常能够识别歪曲，并且可以以某种方式提示以减轻歪曲对 LLM 输出的影响。

Title: Training Sparse Mixture Of Experts Text Embedding Models

• Authors: Zach Nussbaum, Brandon Duderstadt
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2502.07972
• Pdf URL: https://arxiv.org/pdf/2502.07972
• Copy Paste: [[2502.07972]] Training Sparse Mixture Of Experts Text Embedding Models(https://arxiv.org/abs/2502.07972)
• Keywords: language model, retrieval-augmented generation
• Abstract: Transformer-based text embedding models have improved their performance on benchmarks like MIRACL and BEIR by increasing their parameter counts. However, this scaling approach introduces significant deployment challenges, including increased inference latency and memory usage. These challenges are particularly severe in retrieval-augmented generation (RAG) applications, where large models' increased memory requirements constrain dataset ingestion capacity, and their higher latency directly impacts query-time performance. While causal language models have addressed similar efficiency challenges using Mixture of Experts (MoE) architectures, this approach hasn't been successfully adapted to the general text embedding setting. In this paper, we introduce Nomic Embed v2, the first general purpose MoE text embedding model. Our model outperforms models in the same parameter class on both monolingual and multilingual benchmarks while also maintaining competitive performance with models twice its size. We open-source all code, models, and evaluation data to ensure full reproducibility of our training pipeline.
• 摘要：基于 Transformer 的文本嵌入模型通过增加参数数量，提高了在 MIRACL 和 BEIR 等基准测试上的性能。然而，这种扩展方法带来了重大的部署挑战，包括增加推理延迟和内存使用量。这些挑战在检索增强生成 (RAG) 应用程序中尤其严重，其中大型模型增加的内存需求限制了数据集提取能力，而其更高的延迟直接影响查询时间性能。虽然因果语言模型使用混合专家 (MoE) 架构解决了类似的效率挑战，但这种方法尚未成功适应一般的文本嵌入设置。在本文中，我们介绍了 Nomic Embed v2，这是第一个通用的 MoE 文本嵌入模型。我们的模型在单语和多语言基准测试中都优于同一参数类别的模型，同时还保持了与其两倍大小的模型相媲美的性能。我们开源所有代码、模型和评估数据，以确保我们的训练流程完全可重复。

Title: MetaSC: Test-Time Safety Specification Optimization for Language Models

• Authors: Víctor Gallego
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.07985
• Pdf URL: https://arxiv.org/pdf/2502.07985
• Copy Paste: [[2502.07985]] MetaSC: Test-Time Safety Specification Optimization for Language Models(https://arxiv.org/abs/2502.07985)
• Keywords: language model, prompt
• Abstract: We propose a novel dynamic safety framework that optimizes language model (LM) safety reasoning at inference time without modifying model weights. Building on recent advances in self-critique methods, our approach leverages a meta-critique mechanism that iteratively updates safety prompts-termed specifications-to drive the critique and revision process adaptively. This test-time optimization not only improves performance against adversarial jailbreak requests but also in diverse general safety-related tasks, such as avoiding moral harm or pursuing honest responses. Our empirical evaluations across several language models demonstrate that dynamically optimized safety prompts yield significantly higher safety scores compared to fixed system prompts and static self-critique defenses. Code to be released at this https URL .
• 摘要：我们提出了一种新颖的动态安全框架，该框架可在推理时优化语言模型 (LM) 安全推理，而无需修改模型权重。基于自我批评方法的最新进展，我们的方法利用元批评机制，该机制迭代更新安全提示（称为规范），以自适应地推动批评和修订过程。这种测试时优化不仅可以提高对抗性越狱请求的性能，还可以提高各种一般安全相关任务的性能，例如避免道德伤害或追求诚实的回应。我们对多种语言模型的实证评估表明，与固定系统提示和静态自我批评防御相比，动态优化的安全提示可产生更高的安全分数。代码将在此 https URL 上发布。

Title: The Geometry of Prompting: Unveiling Distinct Mechanisms of Task Adaptation in Language Models

• Authors: Artem Kirsanov, Chi-Ning Chou, Kyunghyun Cho, SueYeon Chung
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.08009
• Pdf URL: https://arxiv.org/pdf/2502.08009
• Copy Paste: [[2502.08009]] The Geometry of Prompting: Unveiling Distinct Mechanisms of Task Adaptation in Language Models(https://arxiv.org/abs/2502.08009)
• Keywords: language model, prompt
• Abstract: Decoder-only language models have the ability to dynamically switch between various computational tasks based on input prompts. Despite many successful applications of prompting, there is very limited understanding of the internal mechanism behind such flexibility. In this work, we investigate how different prompting methods affect the geometry of representations in these models. Employing a framework grounded in statistical physics, we reveal that various prompting techniques, while achieving similar performance, operate through distinct representational mechanisms for task adaptation. Our analysis highlights the critical role of input distribution samples and label semantics in few-shot in-context learning. We also demonstrate evidence of synergistic and interfering interactions between different tasks on the representational level. Our work contributes to the theoretical understanding of large language models and lays the groundwork for developing more effective, representation-aware prompting strategies.
• 摘要：仅解码器语言模型能够根据输入提示在各种计算任务之间动态切换。尽管提示有许多成功的应用，但人们对这种灵活性背后的内部机制的理解非常有限。在这项工作中，我们研究了不同的提示方法如何影响这些模型中表示的几何形状。采用基于统计物理学的框架，我们发现各种提示技术虽然实现了相似的性能，但通过不同的表示机制进行任务适应。我们的分析强调了输入分布样本和标签语义在少数样本上下文学习中的关键作用。我们还展示了不同任务在表示层面上协同和干扰相互作用的证据。我们的工作有助于从理论角度理解大型语言模型，并为开发更有效的、表示感知的提示策略奠定了基础。

Title: Speculate, then Collaborate: Fusing Knowledge of Language Models during Decoding

• Authors: Ziyao Wang, Muneeza Azmart, Ang Li, Raya Horesh, Mikhail Yurochkin
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.08020
• Pdf URL: https://arxiv.org/pdf/2502.08020
• Copy Paste: [[2502.08020]] Speculate, then Collaborate: Fusing Knowledge of Language Models during Decoding(https://arxiv.org/abs/2502.08020)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) often excel in specific domains but fall short in others due to the limitations of their training. Thus, enabling LLMs to solve problems collaboratively by integrating their complementary knowledge promises to improve their performance across domains. To realize this potential, we introduce a novel Collaborative Speculative Decoding (CoSD) algorithm that enables efficient LLM knowledge fusion at test time without requiring additional model training. CoSD employs a draft model to generate initial sequences and an easy-to-learn rule or decision tree to decide when to invoke an assistant model to improve these drafts. CoSD not only enhances knowledge fusion but also improves inference efficiency, is transferable across domains and models, and offers greater explainability. Experimental results demonstrate that CoSD improves accuracy by up to 10\% across benchmarks compared to existing methods, providing a scalable and effective solution for LLM-based applications
• 摘要：大型语言模型 (LLM) 通常在特定领域表现出色，但由于训练的局限性，在其他领域表现不佳。因此，通过整合互补知识让 LLM 能够协作解决问题，有望提高其跨领域的性能。为了实现这一潜力，我们引入了一种新颖的协作推测解码 (CoSD) 算法，该算法可以在测试时实现高效的 LLM 知识融合，而无需额外的模型训练。CoSD 使用草稿模型来生成初始序列，并使用易于学习的规则或决策树来决定何时调用辅助模型来改进这些草稿。CoSD 不仅可以增强知识融合，还可以提高推理效率，可跨领域和模型转移，并提供更高的可解释性。实验结果表明，与现有方法相比，CoSD 在基准测试中将准确率提高了 10\%，为基于 LLM 的应用程序提供了可扩展且有效的解决方案

Title: Contextual Subspace Manifold Projection for Structural Refinement of Large Language Model Representations

• Authors: Alistair Wren, Beatrice Loxley, Hamish Cadwallader, Simon Beckwith, Fabian Pargeter, James Blades
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.08026
• Pdf URL: https://arxiv.org/pdf/2502.08026
• Copy Paste: [[2502.08026]] Contextual Subspace Manifold Projection for Structural Refinement of Large Language Model Representations(https://arxiv.org/abs/2502.08026)
• Keywords: language model
• Abstract: Internal representations within deep neural architectures encode high-dimensional abstractions of linguistic structures, yet they often exhibit inefficiencies in feature distribution, limiting expressiveness and adaptability. Contextual Subspace Manifold Projection introduces a structured refinement technique that selectively reconfigures token embeddings through controlled subspace constraints, ensuring more stable and geometrically well-defined feature distributions. Empirical evaluations demonstrated that the structured intervention reduced anisotropy, leading to improved representation compactness while preserving semantic fidelity across transformer layers. Clustering analyses indicated that token embeddings exhibited greater feature separability, reinforcing the hypothesis that structured projection techniques enhance internal representation organization without sacrificing linguistic coherence. Gradient magnitude distributions suggested that the method introduced a smoother optimization trajectory, potentially contributing to more stable parameter updates throughout training. Computational overhead associated with the projection operations remained minimal, ensuring that the refinements did not introduce significant trade-offs in model efficiency or inference speed. Comparisons with standard embedding refinement techniques highlighted that structured manifold constraints provided a direct mechanism for improving representation quality without requiring additional gradient-based optimization. Perplexity evaluations confirmed that the adjustments did not negatively impact sequence coherence, further validating the effectiveness of the proposed approach.
• 摘要：深度神经架构内的内部表征编码了语言结构的高维抽象，但它们通常在特征分布方面表现出低效率，从而限制了表达力和适应性。上下文子空间流形投影引入了一种结构化细化技术，该技术通过受控的子空间约束有选择地重新配置标记嵌入，从而确保更稳定且几何定义良好的特征分布。实证评估表明，结构化干预降低了各向异性，从而提高了表征紧凑性，同时保持了跨转换器层的语义保真度。聚类分析表明，标记嵌入表现出更大的特征可分离性，这强化了以下假设：结构化投影技术可以增强内部表征组织，而不会牺牲语言连贯性。梯度幅度分布表明该方法引入了更平滑的优化轨迹，可能有助于在整个训练过程中实现更稳定的参数更新。与投影操作相关的计算开销保持最小，确保细化不会在模型效率或推理速度方面带来重大权衡。与标准嵌入细化技术的比较表明，结构化流形约束提供了一种直接的机制来提高表示质量，而无需额外的基于梯度的优化。困惑度评估证实，调整不会对序列一致性产生负面影响，进一步验证了所提方法的有效性。

Title: Franken-Adapter: Cross-Lingual Adaptation of LLMs by Embedding Surgery

• Authors: Fan Jiang, Honglin Yu, Grace Chung, Trevor Cohn
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.08037
• Pdf URL: https://arxiv.org/pdf/2502.08037
• Copy Paste: [[2502.08037]] Franken-Adapter: Cross-Lingual Adaptation of LLMs by Embedding Surgery(https://arxiv.org/abs/2502.08037)
• Keywords: language model, llm
• Abstract: The capabilities of Large Language Models (LLMs) in low-resource languages lag far behind those in English, making their universal accessibility a significant challenge. To alleviate this, we present $\textit{Franken-Adapter}$, a modular language adaptation approach for decoder-only LLMs with embedding surgery. Our method begins by creating customized vocabularies for target languages and performing language adaptation through embedding tuning on multilingual data. These pre-trained embeddings are subsequently integrated with LLMs that have been instruction-tuned on English alignment data to enable zero-shot cross-lingual transfer. Our experiments on $\texttt{Gemma2}$ models with up to 27B parameters demonstrate improvements of up to 20% across 96 languages, spanning both discriminative and generative tasks, with minimal regressions ($<$1%) in English. Further in-depth analysis reveals the critical role of customizing tokenizers in enhancing language adaptation, while boosting inference efficiency. Additionally, we show the versatility of our method by achieving a 14% improvement over a math-optimized LLM across 20 languages, offering a modular solution to transfer reasoning abilities across languages post hoc.
• 摘要：大型语言模型 (LLM) 在资源匮乏的语言中的能力远远落后于英语，这使得它们的通用可访问性成为一项重大挑战。为了缓解这一问题，我们提出了 $\textit{Franken-Adapter}$，这是一种针对仅解码器的 LLM 的模块化语言自适应方法，具有嵌入手术。我们的方法首先为目标语言创建定制词汇表，并通过对多语言数据进行嵌入调整来执行语言自适应。这些预训练的嵌入随后与已在英语对齐数据上进行指令调整的 LLM 集成，以实现零样本跨语言迁移。我们在具有多达 27B 个参数的 $\texttt{Gemma2}$ 模型上的实验表明，在 96 种语言中，包括判别任务和生成任务，改进率高达 20%，英语的回归率极小（<$1%）。进一步的深入分析揭示了定制标记器在增强语言适应性的同时提高推理效率的关键作用。此外，我们通过在 20 种语言中实现比数学优化的 LLM 14% 的提升来展示我们方法的多功能性，提供了一种模块化解决方案，可以在事后跨语言转移推理能力。

Title: Break the Checkbox: Challenging Closed-Style Evaluations of Cultural Alignment in LLMs

• Authors: Mohsinul Kabir, Ajwad Abrar, Sophia Ananiadou
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2502.08045
• Pdf URL: https://arxiv.org/pdf/2502.08045
• Copy Paste: [[2502.08045]] Break the Checkbox: Challenging Closed-Style Evaluations of Cultural Alignment in LLMs(https://arxiv.org/abs/2502.08045)
• Keywords: language model, llm
• Abstract: A large number of studies rely on closed-style multiple-choice surveys to evaluate cultural alignment in Large Language Models (LLMs). In this work, we challenge this constrained evaluation paradigm and explore more realistic, unconstrained approaches. Using the World Values Survey (WVS) and Hofstede Cultural Dimensions as case studies, we demonstrate that LLMs exhibit stronger cultural alignment in less constrained settings, where responses are not forced. Additionally, we show that even minor changes, such as reordering survey choices, lead to inconsistent outputs, exposing the limitations of closed-style evaluations. Our findings advocate for more robust and flexible evaluation frameworks that focus on specific cultural proxies, encouraging more nuanced and accurate assessments of cultural alignment in LLMs.
• 摘要：大量研究依靠封闭式多项选择调查来评估大型语言模型 (LLM) 中的文化一致性。在这项工作中，我们挑战了这种受约束的评估范式，并探索了更现实、不受约束的方法。以世界价值观调查 (WVS) 和霍夫斯泰德文化维度为案例研究，我们证明 LLM 在较少约束的环境中表现出更强的文化一致性，其中响应不是强制的。此外，我们表明，即使是微小的变化（例如重新排序调查选项）也会导致输出不一致，从而暴露了封​​闭式评估的局限性。我们的研究结果提倡更强大、更灵活的评估框架，这些框架侧重于特定的文化代理，鼓励对 LLM 中的文化一致性进行更细致入微和准确的评估。

Title: On Mechanistic Circuits for Extractive Question-Answering

• Authors: Samyadeep Basu, Vlad Morariu, Zichao Wang, Ryan Rossi, Cherry Zhao, Soheil Feizi, Varun Manjunatha
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.08059
• Pdf URL: https://arxiv.org/pdf/2502.08059
• Copy Paste: [[2502.08059]] On Mechanistic Circuits for Extractive Question-Answering(https://arxiv.org/abs/2502.08059)
• Keywords: language model
• Abstract: Large language models are increasingly used to process documents and facilitate question-answering on them. In our paper, we extract mechanistic circuits for this real-world language modeling task: context-augmented language modeling for extractive question-answering (QA) tasks and understand the potential benefits of circuits towards downstream applications such as data attribution to context information. We extract circuits as a function of internal model components (e.g., attention heads, MLPs) using causal mediation analysis techniques. Leveraging the extracted circuits, we first understand the interplay between the model's usage of parametric memory and retrieved context towards a better mechanistic understanding of context-augmented language models. We then identify a small set of attention heads in our circuit which performs reliable data attribution by default, thereby obtaining attribution for free in just the model's forward pass. Using this insight, we then introduce ATTNATTRIB, a fast data attribution algorithm which obtains state-of-the-art attribution results across various extractive QA benchmarks. Finally, we show the possibility to steer the language model towards answering from the context, instead of the parametric memory by using the attribution from ATTNATTRIB as an additional signal during the forward pass. Beyond mechanistic understanding, our paper provides tangible applications of circuits in the form of reliable data attribution and model steering.
• 摘要：大型语言模型越来越多地用于处理文档并促进对文档的问答。在我们的论文中，我们为这个现实世界的语言建模任务提取了机械电路：用于提取问答 (QA) 任务的上下文增强语言建模，并了解电路对下游应用（例如数据归因于上下文信息）的潜在好处。我们使用因果中介分析技术提取电路作为内部模型组件（例如，注意力头、MLP）的函数。利用提取的电路，我们首先了解模型对参数记忆的使用与检索到的上下文之间的相互作用，从而更好地从机械上理解上下文增强语言模型。然后，我们在电路中识别出一小组注意力头，默认情况下它们执行可靠的数据归因，从而仅在模型的前向传递中免费获得归因。利用这一见解，我们随后引入了 ATTNATTRIB，这是一种快速数据归因算法，可在各种提取 QA 基准中获得最先进的归因结果。最后，我们展示了通过在前向传递过程中使用来自 ATTNATTRIB 的属性作为附加信号，引导语言模型从上下文而不是参数记忆进行回答的可能性。除了机械理解之外，我们的论文还以可靠的数据属性和模型转向的形式提供了电路的切实应用。

Title: NLI under the Microscope: What Atomic Hypothesis Decomposition Reveals

• Authors: Neha Srikanth, Rachel Rudinger
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.08080
• Pdf URL: https://arxiv.org/pdf/2502.08080
• Copy Paste: [[2502.08080]] NLI under the Microscope: What Atomic Hypothesis Decomposition Reveals(https://arxiv.org/abs/2502.08080)
• Keywords: llm
• Abstract: Decomposition of text into atomic propositions is a flexible framework allowing for the closer inspection of input and output text. We use atomic decomposition of hypotheses in two natural language reasoning tasks, traditional NLI and defeasible NLI, to form atomic sub-problems, or granular inferences that models must weigh when solving the overall problem. These atomic sub-problems serve as a tool to further understand the structure of both NLI and defeasible reasoning, probe a model's consistency and understanding of different inferences, and measure the diversity of examples in benchmark datasets. Our results indicate that LLMs still struggle with logical consistency on atomic NLI and defeasible NLI sub-problems. Lastly, we identify critical atomic sub-problems of defeasible NLI examples, or those that most contribute to the overall label, and propose a method to measure the inferential consistency of a model, a metric designed to capture the degree to which a model makes consistently correct or incorrect predictions about the same fact under different contexts.
• 摘要：将文本分解为原子命题是一个灵活的框架，允许更仔细地检查输入和输出文本。我们在两个自然语言推理任务（传统 NLI 和可废止 NLI）中使用假设的原子分解来形成原子子问题，或者模型在解决整体问题时必须权衡的细粒度推理。这些原子子问题可作为进一步理解 NLI 和可废止推理的结构、探究模型的一致性和对不同推理的理解以及衡量基准数据集中示例多样性的工具。我们的结果表明，LLM 在原子 NLI 和可废止 NLI 子问题上仍然难以实现逻辑一致性。最后，我们确定了可废止 NLI 示例的关键原子子问题，或者对整体标签贡献最大的原子子问题，并提出了一种衡量模型推理一致性的方法，这是一种旨在捕捉模型在不同背景下对同一事实做出一致正确或错误预测的程度的指标。

Title: GCoT: Chain-of-Thought Prompt Learning for Graphs

• Authors: Xingtong Yu, Chang Zhou, Zhongwei Kuai, Xinming Zhang, Yuan Fang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.08092
• Pdf URL: https://arxiv.org/pdf/2502.08092
• Copy Paste: [[2502.08092]] GCoT: Chain-of-Thought Prompt Learning for Graphs(https://arxiv.org/abs/2502.08092)
• Keywords: prompt, chain-of-thought
• Abstract: Chain-of-thought (CoT) prompting has achieved remarkable success in natural language processing (NLP). However, its vast potential remains largely unexplored for graphs. This raises an interesting question: How can we design CoT prompting for graphs to guide graph models to learn step by step? On one hand, unlike natural languages, graphs are non-linear and characterized by complex topological structures. On the other hand, many graphs lack textual data, making it difficult to formulate language-based CoT prompting. In this work, we propose the first CoT prompt learning framework for text-free graphs, GCoT. Specifically, we decompose the adaptation process for each downstream task into a series of inference steps, with each step consisting of prompt-based inference, ``thought'' generation, and thought-conditioned prompt learning. While the steps mimic CoT prompting in NLP, the exact mechanism differs significantly. Specifically, at each step, an input graph, along with a prompt, is first fed into a pre-trained graph encoder for prompt-based inference. We then aggregate the hidden layers of the encoder to construct a ``thought'', which captures the working state of each node in the current step. Conditioned on this thought, we learn a prompt specific to each node based on the current state. These prompts are fed into the next inference step, repeating the cycle. To evaluate and analyze the effectiveness of GCoT, we conduct comprehensive experiments on eight public datasets, which demonstrate the advantage of our approach.
• 摘要：思路链 (CoT) 提示在自然语言处理 (NLP) 中取得了显著的成功。然而，它在图形方面的巨大潜力仍未得到充分开发。这就提出了一个有趣的问题：我们如何设计用于图形的 CoT 提示，以引导图形模型逐步学习？一方面，与自然语言不同，图形是非线性的，具有复杂的拓扑结构。另一方面，许多图形缺乏文本数据，因此很难制定基于语言的 CoT 提示。在这项工作中，我们提出了第一个用于无文本图形的 CoT 提示学习框架 GCoT。具体来说，我们将每个下游任务的适应过程分解为一系列推理步骤，每个步骤都包括基于提示的推理、“思想”生成和思想条件提示学习。虽然这些步骤模仿了 NLP 中的 CoT 提示，但确切的机制却大不相同。具体来说，在每个步骤中，首先将输入图连同提示一起输入到预先训练的图形编码器中进行基于提示的推理。然后，我们将编码器的隐藏层聚合起来，构建一个“想法”，它捕获当前步骤中每个节点的工作状态。在此想法的条件下，我们根据当前状态学习特定于每个节点的提示。这些提示被输入到下一个推理步骤中，重复这个循环。为了评估和分析 GCoT 的有效性，我们对八个公共数据集进行了全面的实验，证明了我们方法的优势。

Title: HuDEx: Integrating Hallucination Detection and Explainability for Enhancing the Reliability of LLM responses

• Authors: Sujeong Lee, Hayoung Lee, Seongsoo Heo, Wonik Choi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.08109
• Pdf URL: https://arxiv.org/pdf/2502.08109
• Copy Paste: [[2502.08109]] HuDEx: Integrating Hallucination Detection and Explainability for Enhancing the Reliability of LLM responses(https://arxiv.org/abs/2502.08109)
• Keywords: language model, gpt, llm, hallucination
• Abstract: Recent advances in large language models (LLMs) have shown promising improvements, often surpassing existing methods across a wide range of downstream tasks in natural language processing. However, these models still face challenges, which may hinder their practical applicability. For example, the phenomenon of hallucination is known to compromise the reliability of LLMs, especially in fields that demand high factual precision. Current benchmarks primarily focus on hallucination detection and factuality evaluation but do not extend beyond identification. This paper proposes an explanation enhanced hallucination-detection model, coined as HuDEx, aimed at enhancing the reliability of LLM-generated responses by both detecting hallucinations and providing detailed explanations. The proposed model provides a novel approach to integrate detection with explanations, and enable both users and the LLM itself to understand and reduce errors. Our measurement results demonstrate that the proposed model surpasses larger LLMs, such as Llama3 70B and GPT-4, in hallucination detection accuracy, while maintaining reliable explanations. Furthermore, the proposed model performs well in both zero-shot and other test environments, showcasing its adaptability across diverse benchmark datasets. The proposed approach further enhances the hallucination detection research by introducing a novel approach to integrating interpretability with hallucination detection, which further enhances the performance and reliability of evaluating hallucinations in language models.
• 摘要：大型语言模型 (LLM) 的最新进展显示出令人鼓舞的改进，通常在自然语言处理中的各种下游任务中超越现有方法。然而，这些模型仍然面临挑战，这可能会阻碍它们的实际应用。例如，众所周知，幻觉现象会损害 LLM 的可靠性，尤其是在对事实精度要求高的领域。当前的基准主要侧重于幻觉检测和事实性评估，但并未超出识别范围。本文提出了一种解释增强的幻觉检测模型，称为 HuDEx，旨在通过检测幻觉和提供详细解释来提高 LLM 生成的响应的可靠性。所提出的模型提供了一种将检测与解释相结合的新方法，并使用户和 LLM 本身都能理解和减少错误。我们的测量结果表明，所提出的模型在幻觉检测准确率方面超越了更大的 LLM，例如 Llama3 70B 和 GPT-4，同时保持了可靠的解释。此外，所提出的模型在零样本和其他测试环境中均表现良好，展示了其在各种基准数据集上的适应性。所提出的方法通过引入一种将可解释性与幻觉检测相结合的新方法，进一步增强了幻觉检测研究，从而进一步提高了语言模型中评估幻觉的性能和可靠性。

Title: Fino1: On the Transferability of Reasoning Enhanced LLMs to Finance

• Authors: Lingfei Qian, Weipeng Zhou, Yan Wang, Xueqing Peng, Jimin Huang, Qianqian Xie
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.08127
• Pdf URL: https://arxiv.org/pdf/2502.08127
• Copy Paste: [[2502.08127]] Fino1: On the Transferability of Reasoning Enhanced LLMs to Finance(https://arxiv.org/abs/2502.08127)
• Keywords: language model, llm
• Abstract: Recent advancements in large language models (LLMs) have shown strong general reasoning abilities, yet their effectiveness in financial reasoning remains underexplored. In this study, we comprehensively evaluate 16 powerful reasoning and general LLMs on three complex financial tasks involving financial text, tabular data, and equations, assessing numerical reasoning, tabular interpretation, financial terminology comprehension, long-context processing, and equation-based problem solving. Our results show that while better datasets and pretraining improve financial reasoning, general enhancements like CoT fine-tuning do not always yield consistent gains. Moreover, all reasoning strategies face challenges in improving performance on long-context and multi-table tasks. To address these limitations, we develop a financial reasoning-enhanced model based on Llama-3.1-8B-Instruct, by CoT fine-tuning and reinforcement learning with domain-specific reasoning paths. Even with simple fine-tuning with one financial dataset, our model achieves a consistent 10% performance improvement across tasks, surpassing all 8B models and even Llama3-70B-Instruct and Llama3.1-70B-Instruct on average. Our results highlight the need for domain-specific adaptations in financial tasks, emphasizing future directions such as multi-table reasoning, long-context processing, and financial terminology comprehension. All our datasets, models, and codes are publicly available. Furthermore, we introduce a leaderboard for benchmarking future datasets and models.
• 摘要：大型语言模型 (LLM) 的最新进展已显示出强大的通用推理能力，但它们在金融推理中的有效性仍未得到充分探索。在本研究中，我们全面评估了 16 种强大的推理和通用 LLM 在三个复杂的金融任务上的表现，这些任务涉及金融文本、表格数据和方程式，评估了数值推理、表格解释、金融术语理解、长上下文处理和基于方程式的问题解决。我们的结果表明，虽然更好的数据集和预训练可以改善金融推理，但像 CoT 微调这样的通用增强并不总能带来一致的收益。此外，所有推理策略在提高长上下文和多表任务的性能方面都面临着挑战。为了解决这些限制，我们基于 Llama-3.1-8B-Instruct 开发了一个金融推理增强模型，通过 CoT 微调和具有领域特定推理路径的强化学习。即使使用一个金融数据集进行简单的微调，我们的模型也能在各个任务中实现 10% 的持续性能提升，平均超过所有 8B 模型，甚至超过 Llama3-70B-Instruct 和 Llama3.1-70B-Instruct。我们的结果强调了金融任务中需要针对特定​​领域进行调整，强调了未来的发展方向，例如多表推理、长上下文处理和金融术语理解。我们所有的数据集、模型和代码都是公开的。此外，我们还引入了一个排行榜来对未来的数据集和模型进行基准测试。

Title: Selective Self-to-Supervised Fine-Tuning for Generalization in Large Language Models

• Authors: Sonam Gupta, Yatin Nandwani, Asaf Yehudai, Dinesh Khandelwal, Dinesh Raghu, Sachindra Joshi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.08130
• Pdf URL: https://arxiv.org/pdf/2502.08130
• Copy Paste: [[2502.08130]] Selective Self-to-Supervised Fine-Tuning for Generalization in Large Language Models(https://arxiv.org/abs/2502.08130)
• Keywords: language model, llm
• Abstract: Fine-tuning Large Language Models (LLMs) on specific datasets is a common practice to improve performance on target tasks. However, this performance gain often leads to overfitting, where the model becomes too specialized in either the task or the characteristics of the training data, resulting in a loss of generalization. This paper introduces Selective Self-to-Supervised Fine-Tuning (S3FT), a fine-tuning approach that achieves better performance than the standard supervised fine-tuning (SFT) while improving generalization. S3FT leverages the existence of multiple valid responses to a query. By utilizing the model's correct responses, S3FT reduces model specialization during the fine-tuning stage. S3FT first identifies the correct model responses from the training set by deploying an appropriate judge. Then, it fine-tunes the model using the correct model responses and the gold response (or its paraphrase) for the remaining samples. The effectiveness of S3FT is demonstrated through experiments on mathematical reasoning, Python programming and reading comprehension tasks. The results show that standard SFT can lead to an average performance drop of up to $4.4$ on multiple benchmarks, such as MMLU and TruthfulQA. In contrast, S3FT reduces this drop by half, i.e. $2.5$, indicating better generalization capabilities than SFT while performing significantly better on the fine-tuning tasks.
• 摘要：在特定数据集上对大型语言模型 (LLM) 进行微调是提高目标任务性能的常见做法。然而，这种性能提升往往会导致过度拟合，即模型在任务或训练数据的特征上变得过于专业化，从而导致泛化能力下降。本文介绍了选择性自监督微调 (S3FT)，这是一种微调方法，它在提高泛化能力的同时，实现了比标准监督微调 (SFT) 更好的性能。S3FT 利用查询的多个有效响应的存在。通过利用模型的正确响应，S3FT 在微调阶段减少了模型专业化。S3FT 首先通过部署适当的判断从训练集中识别正确的模型响应。然后，它使用正确的模型响应和剩余样本的黄金响应（或其释义）对模型进行微调。通过数学推理、Python 编程和阅读理解任务的实验证明了 S3FT 的有效性。结果表明，标准 SFT 在 MMLU 和 TruthfulQA 等多个基准测试中平均性能下降高达 4.4%。相比之下，S3FT 将这一下降减少了一半，即 2.5，这表明其泛化能力优于 SFT，同时在微调任务上的表现也明显更好。

Title: SARChat-Bench-2M: A Multi-Task Vision-Language Benchmark for SAR Image Interpretation

• Authors: Zhiming Ma, Xiayang Xiao, Sihao Dong, Peidong Wang, HaiPeng Wang, Qingyun Pan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.08168
• Pdf URL: https://arxiv.org/pdf/2502.08168
• Copy Paste: [[2502.08168]] SARChat-Bench-2M: A Multi-Task Vision-Language Benchmark for SAR Image Interpretation(https://arxiv.org/abs/2502.08168)
• Keywords: language model, chat
• Abstract: In the field of synthetic aperture radar (SAR) remote sensing image interpretation, although Vision language models (VLMs) have made remarkable progress in natural language processing and image understanding, their applications remain limited in professional domains due to insufficient domain expertise. This paper innovatively proposes the first large-scale multimodal dialogue dataset for SAR images, named SARChat-2M, which contains approximately 2 million high-quality image-text pairs, encompasses diverse scenarios with detailed target annotations. This dataset not only supports several key tasks such as visual understanding and object detection tasks, but also has unique innovative aspects: this study develop a visual-language dataset and benchmark for the SAR domain, enabling and evaluating VLMs' capabilities in SAR image interpretation, which provides a paradigmatic framework for constructing multimodal datasets across various remote sensing vertical domains. Through experiments on 16 mainstream VLMs, the effectiveness of the dataset has been fully verified, and the first multi-task dialogue benchmark in the SAR field has been successfully established. The project will be released at this https URL, aiming to promote the in-depth development and wide application of SAR visual language models.
• 摘要：在合成孔径雷达（SAR）遥感图像解译领域，尽管视觉语言模型（VLM）在自然语言处理和图像理解方面取得了显著进展，但由于领域专业知识不足，其应用仍然受到限制。本文创新性地提出了第一个针对SAR图像的大规模多模态对话数据集SARChat-2M，其中包含约200万个高质量的图文对，涵盖了多样化的场景并带有详细的目标标注。该数据集不仅支持视觉理解和目标检测等几项关键任务，而且具有独特的创新性：本研究为SAR领域开发了一个视觉语言数据集和基准，启用并评估了VLM在SAR图像解译中的能力，为构建跨各个遥感垂直领域的多模态数据集提供了范式框架。通过在16个主流VLM上的实验，充分验证了数据集的有效性，并成功建立了SAR领域第一个多任务对话基准。该项目将在此https URL发布，旨在推动SAR视觉语言模型的深入发展和广泛应用。

Title: ParetoRAG: Leveraging Sentence-Context Attention for Robust and Efficient Retrieval-Augmented Generation

• Authors: Ruobing Yao, Yifei Zhang, Shuang Song, Yuhua Liu, Neng Gao, Chenyang Tu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.08178
• Pdf URL: https://arxiv.org/pdf/2502.08178
• Copy Paste: [[2502.08178]] ParetoRAG: Leveraging Sentence-Context Attention for Robust and Efficient Retrieval-Augmented Generation(https://arxiv.org/abs/2502.08178)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: While Retrieval-Augmented Generation (RAG) systems enhance Large Language Models (LLMs) by incorporating external knowledge, they still face persistent challenges in retrieval inefficiency and the inability of LLMs to filter out irrelevant information. We present ParetoRAG, an unsupervised framework that optimizes RAG systems through sentence-level refinement guided by the Pareto principle. By decomposing paragraphs into sentences and dynamically re-weighting core content while preserving contextual coherence, ParetoRAG achieves dual improvements in both retrieval precision and generation quality without requiring additional training or API resources. This framework has been empirically validated across various datasets, LLMs, and retrievers.
• 摘要：虽然检索增强生成 (RAG) 系统通过整合外部知识来增强大型语言模型 (LLM)，但它们仍然面临着检索效率低下和 LLM 无法过滤掉无关信息的持续挑战。我们提出了 ParetoRAG，这是一个无监督框架，它通过以帕累托原则为指导的句子级细化来优化 RAG 系统。通过将段落分解为句子并动态重新加权核心内容，同时保持上下文连贯性，ParetoRAG 实现了检索精度和生成质量的双重提升，而无需额外的训练或 API 资源。该框架已在各种数据集、LLM 和检索器中得到了实证验证。

Title: Enhancing LLM Character-Level Manipulation via Divide and Conquer

• Authors: Zhen Xiong, Yujun Cai, Bryan Hooi, Nanyun Peng, Kai-Wei Chang, Zhecheng Li, Yiwei Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.08180
• Pdf URL: https://arxiv.org/pdf/2502.08180
• Copy Paste: [[2502.08180]] Enhancing LLM Character-Level Manipulation via Divide and Conquer(https://arxiv.org/abs/2502.08180)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have demonstrated strong generalization capabilities across a wide range of natural language processing (NLP) tasks. However, they exhibit notable weaknesses in character-level string manipulation, struggling with fundamental operations such as character deletion, insertion, and substitution. These challenges stem primarily from tokenization constraints, despite the critical role of such operations in data preprocessing and code generation. Through systematic analysis, we derive two key insights: (1) LLMs face significant difficulties in leveraging intrinsic token knowledge for character-level reasoning, and (2) atomized word structures can substantially enhance LLMs' ability to process token-level structural information. Building on these insights, we propose Character-Level Manipulation via Divide and Conquer, a novel approach designed to bridge the gap between token-level processing and character-level manipulation. Our method decomposes complex operations into explicit character-level subtasks coupled with controlled token reconstruction phases, leading to significant improvements in accuracy. Without additional training, our method significantly improves accuracies on the $\texttt{Deletion}$, $\texttt{Insertion}$, and $\texttt{Substitution}$ tasks. To support further research, we open-source our implementation and benchmarks.
• 摘要：大型语言模型 (LLM) 已在广泛的自然语言处理 (NLP) 任务中展现出强大的泛化能力。然而，它们在字符级字符串操作方面表现出明显的弱点，难以完成字符删除、插入和替换等基本操作。这些挑战主要源于标记化约束，尽管此类操作在数据预处理和代码生成中起着关键作用。通过系统分析，我们得出两个关键见解：(1) LLM 在利用内在标记知识进行字符级推理方面面临巨大困难，(2) 原子化词结构可以大大增强 LLM 处理标记级结构信息的能力。基于这些见解，我们提出了通过分而治之的字符级操作，这是一种旨在弥合标记级处理和字符级操作之间差距的新方法。我们的方法将复杂的操作分解为明确的字符级子任务和受控的标记重构阶段，从而显著提高准确性。无需额外训练，我们的方法就显著提高了“删除”、“插入”和“替换”任务的准确率。为了支持进一步的研究，我们开源了我们的实现和基准。

Title: LLM Modules: Knowledge Transfer from a Large to a Small Model using Enhanced Cross-Attention

• Authors: Konstantin Kolomeitsev (Almaty, Kazakhstan)
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.08213
• Pdf URL: https://arxiv.org/pdf/2502.08213
• Copy Paste: [[2502.08213]] LLM Modules: Knowledge Transfer from a Large to a Small Model using Enhanced Cross-Attention(https://arxiv.org/abs/2502.08213)
• Keywords: gpt, llm
• Abstract: In this work, we propose an architecture of LLM Modules that enables the transfer of knowledge from a large pre-trained model to a smaller model using an Enhanced Cross-Attention mechanism. In the proposed scheme, the Qwen2-1.5B model is frozen and its representations are passed through specially designed attention layers to the GPT-Neo-125M model, which is trained on limited computational resources. Experimental results on the Bespoke-Stratos-17k dataset demonstrate that after 15 epochs of training, the combined model generates responses comparable in quality to those obtained by distillation. We discuss the advantages of the modular approach, provide examples of input queries and comparative analysis, and outline prospects for further extension of the method.
• 摘要：在本文中，我们提出了一种 LLM 模块架构，该架构能够使用增强型交叉注意机制将知识从大型预训练模型转移到较小的模型。在提出的方案中，Qwen2-1.5B 模型被冻结，其表示通过专门设计的注意层传递到 GPT-Neo-125M 模型，该模型在有限的计算资源上进行训练。在 Bespoke-Stratos-17k 数据集上的实验结果表明，经过 15 个训练周期后，组合模型生成的响应质量与蒸馏获得的响应相当。我们讨论了模块化方法的优势，提供了输入查询和比较分析的示例，并概述了该方法进一步扩展的前景。

Title: Inference-time sparse attention with asymmetric indexing

• Authors: Pierre-Emmanuel Mazaré, Gergely Szilvasy, Maria Lomeli, Francisco Massa, Naila Murray, Hervé Jégou, Matthijs Douze
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.08246
• Pdf URL: https://arxiv.org/pdf/2502.08246
• Copy Paste: [[2502.08246]] Inference-time sparse attention with asymmetric indexing(https://arxiv.org/abs/2502.08246)
• Keywords: language model, long context
• Abstract: Self-attention in transformer models is an incremental associative memory that maps key vectors to value vectors. One way to speed up self-attention is to employ GPU-compliant vector search algorithms, yet the standard partitioning methods yield poor results in this context, because (1) keys and queries follow different distributions and (2) the effect of RoPE positional encoding. In this paper, we introduce SAAP (Self-Attention with Asymmetric Partitions), which overcomes these problems. It is an asymmetrical indexing technique that employs distinct partitions for keys and queries, thereby approximating self-attention with a data-adaptive sparsity pattern. It works on pretrained language models without finetuning, as it only requires to train (offline) a small query classifier. On a long context Llama 3.1-8b model, with sequences ranging from 100k to 500k tokens, our method typically reduces by a factor 20 the fraction of memory that needs to be looked-up, which translates to a time saving of 60\% when compared to FlashAttention-v2.
• 摘要：Transformer 模型中的自注意力是一种增量联想记忆，可将键向量映射到值向量。加快自注意力的一种方法是采用符合 GPU 的向量搜索算法，但标准分区方法在这种情况下会产生较差的结果，因为 (1) 键和查询遵循不同的分布和 (2) RoPE 位置编码的影响。在本文中，我们引入了 SAAP（具有非对称分区的自注意力），它克服了这些问题。它是一种非对称索引技术，对键和查询采用不同的分区，从而使用数据自适应稀疏模式近似自注意力。它适用于预训练语言模型而无需微调，因为它只需要训练（离线）一个小型查询分类器。在长上下文 Llama 3.1-8b 模型中，序列范围从 100k 到 500k 个标记，我们的方法通常将需要查找的内存部分减少 20 倍，与 FlashAttention-v2 相比，可节省 60% 的时间。

Title: Exploring the Potential of Large Language Models to Simulate Personality

• Authors: Maria Molchanova, Anna Mikhailova, Anna Korzanova, Lidiia Ostyakova, Alexandra Dolidze
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.08265
• Pdf URL: https://arxiv.org/pdf/2502.08265
• Copy Paste: [[2502.08265]] Exploring the Potential of Large Language Models to Simulate Personality(https://arxiv.org/abs/2502.08265)
• Keywords: language model, llm, chat
• Abstract: With the advancement of large language models (LLMs), the focus in Conversational AI has shifted from merely generating coherent and relevant responses to tackling more complex challenges, such as personalizing dialogue systems. In an effort to enhance user engagement, chatbots are often designed to mimic human behaviour, responding within a defined emotional spectrum and aligning to a set of values. In this paper, we aim to simulate personal traits according to the Big Five model with the use of LLMs. Our research showed that generating personality-related texts is still a challenging task for the models. As a result, we present a dataset of generated texts with the predefined Big Five characteristics and provide an analytical framework for testing LLMs on a simulation of personality skills.
• 摘要：随着大型语言模型 (LLM) 的进步，对话式人工智能的重点已从仅仅生成连贯且相关的响应转移到应对更复杂的挑战，例如个性化对话系统。为了提高用户参与度，聊天机器人通常被设计为模仿人类行为，在定义的情绪范围内做出反应并与一组价值观保持一致。在本文中，我们旨在使用 LLM 根据大五模型模拟个人特征。我们的研究表明，生成与个性相关的文本对于模型来说仍然是一项具有挑战性的任务。因此，我们提供了具有预定义大五特征的生成文本数据集，并提供了一个分析框架，用于在模拟个性技能方面测试 LLM。

Title: Redefining Simplicity: Benchmarking Large Language Models from Lexical to Document Simplification

• Authors: Jipeng Qiang, Minjiang Huang, Yi Zhu, Yunhao Yuan, Chaowei Zhang, Kui Yu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.08281
• Pdf URL: https://arxiv.org/pdf/2502.08281
• Copy Paste: [[2502.08281]] Redefining Simplicity: Benchmarking Large Language Models from Lexical to Document Simplification(https://arxiv.org/abs/2502.08281)
• Keywords: language model, llm
• Abstract: Text simplification (TS) refers to the process of reducing the complexity of a text while retaining its original meaning and key information. Existing work only shows that large language models (LLMs) have outperformed supervised non-LLM-based methods on sentence simplification. This study offers the first comprehensive analysis of LLM performance across four TS tasks: lexical, syntactic, sentence, and document simplification. We compare lightweight, closed-source and open-source LLMs against traditional non-LLM methods using automatic metrics and human evaluations. Our experiments reveal that LLMs not only outperform non-LLM approaches in all four tasks but also often generate outputs that exceed the quality of existing human-annotated references. Finally, we present some future directions of TS in the era of LLMs.
• 摘要：文本简化 (TS) 是指在保留文本原意和关键信息的同时降低文本复杂度的过程。现有研究仅表明大型语言模型 (LLM) 在句子简化方面的表现优于基于非 LLM 的监督方法。本研究首次全面分析了 LLM 在四个 TS 任务中的表现：词汇、句法、句子和文档简化。我们使用自动指标和人工评估将轻量级、闭源和开源 LLM 与传统非 LLM 方法进行比较。我们的实验表明，LLM 不仅在这四个任务中的表现都优于非 LLM 方法，而且通常生成的输出质量也超过现有的人工注释参考文献。最后，我们提出了 LLM 时代 TS 的一些未来方向。

Title: Compromising Honesty and Harmlessness in Language Models via Deception Attacks

• Authors: Laurène Vaugrante, Francesca Carlon, Maluna Menke, Thilo Hagendorff
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2502.08301
• Pdf URL: https://arxiv.org/pdf/2502.08301
• Copy Paste: [[2502.08301]] Compromising Honesty and Harmlessness in Language Models via Deception Attacks(https://arxiv.org/abs/2502.08301)
• Keywords: language model, llm, prompt, chat, agent
• Abstract: Recent research on large language models (LLMs) has demonstrated their ability to understand and employ deceptive behavior, even without explicit prompting. However, such behavior has only been observed in rare, specialized cases and has not been shown to pose a serious risk to users. Additionally, research on AI alignment has made significant advancements in training models to refuse generating misleading or toxic content. As a result, LLMs generally became honest and harmless. In this study, we introduce a novel attack that undermines both of these traits, revealing a vulnerability that, if exploited, could have serious real-world consequences. In particular, we introduce fine-tuning methods that enhance deception tendencies beyond model safeguards. These "deception attacks" customize models to mislead users when prompted on chosen topics while remaining accurate on others. Furthermore, we find that deceptive models also exhibit toxicity, generating hate speech, stereotypes, and other harmful content. Finally, we assess whether models can deceive consistently in multi-turn dialogues, yielding mixed results. Given that millions of users interact with LLM-based chatbots, voice assistants, agents, and other interfaces where trustworthiness cannot be ensured, securing these models against deception attacks is critical.
• 摘要：最近对大型语言模型 (LLM) 的研究表明，它们能够理解和使用欺骗行为，即使没有明确的提示。然而，这种行为只在极少数特殊情况下观察到，并没有显示对用户构成严重风险。此外，对人工智能对齐的研究在训练模型拒绝生成误导性或有害内容方面取得了重大进展。因此，LLM 通常变得诚实无害。在这项研究中，我们引入了一种破坏这两种特征的新攻击，揭示了一个漏洞，如果被利用，可能会产生严重的现实后果。特别是，我们引入了微调方法，以增强模型保护措施之外的欺骗倾向。这些“欺骗攻击”定制模型，在提示所选主题时误导用户，同时在其他主题上保持准确。此外，我们发现欺骗模型也表现出毒性，产生仇恨言论、刻板印象和其他有害内容。最后，我们评估模型是否可以在多轮对话中持续欺骗，结果好坏参半。鉴于数百万用户与基于 LLM 的聊天机器人、语音助手、代理和其他无法确保可信度的界面进行交互，保护这些模型免受欺骗攻击至关重要。

Title: Mitigating Hallucinations in Multimodal Spatial Relations through Constraint-Aware Prompting

• Authors: Jiarui Wu, Zhuo Liu, Hangfeng He
• Subjects: cs.CL, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2502.08317
• Pdf URL: https://arxiv.org/pdf/2502.08317
• Copy Paste: [[2502.08317]] Mitigating Hallucinations in Multimodal Spatial Relations through Constraint-Aware Prompting(https://arxiv.org/abs/2502.08317)
• Keywords: language model, hallucination, prompt
• Abstract: Spatial relation hallucinations pose a persistent challenge in large vision-language models (LVLMs), leading to generate incorrect predictions about object positions and spatial configurations within an image. To address this issue, we propose a constraint-aware prompting framework designed to reduce spatial relation hallucinations. Specifically, we introduce two types of constraints: (1) bidirectional constraint, which ensures consistency in pairwise object relations, and (2) transitivity constraint, which enforces relational dependence across multiple objects. By incorporating these constraints, LVLMs can produce more spatially coherent and consistent outputs. We evaluate our method on three widely-used spatial relation datasets, demonstrating performance improvements over existing approaches. Additionally, a systematic analysis of various bidirectional relation analysis choices and transitivity reference selections highlights greater possibilities of our methods in incorporating constraints to mitigate spatial relation hallucinations.
• 摘要：空间关系幻觉对大型视觉语言模型 (LVLM) 构成了持续挑战，导致对图像中的对象位置和空间配置产生错误的预测。为了解决这个问题，我们提出了一个约束感知提示框架，旨在减少空间关系幻觉。具体来说，我们引入了两种类型的约束：(1) 双向约束，确保成对对象关系的一致性，以及 (2) 传递性约束，强制多个对象之间的关系依赖性。通过结合这些约束，LVLM 可以产生更空间连贯和一致的输出。我们在三个广泛使用的空间关系数据集上评估了我们的方法，证明了与现有方法相比性能有所改进。此外，对各种双向关系分析选择和传递性参考选择的系统分析突出了我们的方法在结合约束以减轻空间关系幻觉方面有更大的可能性。

Title: MultiProSE: A Multi-label Arabic Dataset for Propaganda, Sentiment, and Emotion Detection

• Authors: Lubna Al-Henaki, Hend Al-Khalifa, Abdulmalik Al-Salman, Hajar Alqubayshi, Hind Al-Twailay, Gheeda Alghamdi, Hawra Aljasim
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.08319
• Pdf URL: https://arxiv.org/pdf/2502.08319
• Copy Paste: [[2502.08319]] MultiProSE: A Multi-label Arabic Dataset for Propaganda, Sentiment, and Emotion Detection(https://arxiv.org/abs/2502.08319)
• Keywords: language model, gpt, llm
• Abstract: Propaganda is a form of persuasion that has been used throughout history with the intention goal of influencing people's opinions through rhetorical and psychological persuasion techniques for determined ends. Although Arabic ranked as the fourth most- used language on the internet, resources for propaganda detection in languages other than English, especially Arabic, remain extremely limited. To address this gap, the first Arabic dataset for Multi-label Propaganda, Sentiment, and Emotion (MultiProSE) has been introduced. MultiProSE is an open-source extension of the existing Arabic propaganda dataset, ArPro, with the addition of sentiment and emotion annotations for each text. This dataset comprises 8,000 annotated news articles, which is the largest propaganda dataset to date. For each task, several baselines have been developed using large language models (LLMs), such as GPT-4o-mini, and pre-trained language models (PLMs), including three BERT-based models. The dataset, annotation guidelines, and source code are all publicly released to facilitate future research and development in Arabic language models and contribute to a deeper understanding of how various opinion dimensions interact in news media1.
• 摘要：宣传是一种说服形式，历史上一直被使用，目的是通过修辞和心理说服技巧影响人们的观点，以达到既定目的。尽管阿拉伯语是互联网上使用率第四高的语言，但用于除英语以外的其他语言（尤其是阿拉伯语）的宣传检测资源仍然极其有限。为了弥补这一差距，我们推出了第一个多标签宣传、情绪和情感 (MultiProSE) 阿拉伯语数据集。MultiProSE 是现有阿拉伯语宣传数据集 ArPro 的开源扩展，为每篇文本添加了情绪和情感注释。该数据集包含 8,000 篇带注释的新闻文章，是迄今为止最大的宣传数据集。对于每一项任务，我们已经使用大型语言模型 (LLM)（例如 GPT-4o-mini）和预训练语言模型 (PLM)（包括三个基于 BERT 的模型）开发了多个基线。数据集、注释指南和源代码均公开发布，以促进未来阿拉伯语语言模型的研究和开发，并有助于更深入地了解新闻媒体中各个观点维度如何相互作用1。

Title: Contextual Compression Encoding for Large Language Models: A Novel Framework for Multi-Layered Parameter Space Pruning

• Authors: Barnaby Schmitt, Alistair Grosvenor, Matthias Cunningham, Clementine Walsh, Julius Pembrokeshire, Jonathan Teel
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.08323
• Pdf URL: https://arxiv.org/pdf/2502.08323
• Copy Paste: [[2502.08323]] Contextual Compression Encoding for Large Language Models: A Novel Framework for Multi-Layered Parameter Space Pruning(https://arxiv.org/abs/2502.08323)
• Keywords: language model
• Abstract: Context-aware compression techniques have gained increasing attention as model sizes continue to grow, introducing computational bottlenecks that hinder efficient deployment. A structured encoding approach was proposed to selectively eliminate redundant parameter groups while ensuring that representational fidelity was preserved across multiple layers. Contextual Compression Encoding (CCE) introduced a multi-stage encoding mechanism that dynamically restructured parameter distributions, allowing for significant reductions in memory footprint and computational complexity. Experimental evaluations demonstrated that models compressed through CCE retained linguistic expressivity and coherence, maintaining accuracy across a range of text generation and classification tasks. Layer-wise analysis revealed that middle-network layers exhibited higher compression ratios, aligning with the observation that self-attention and feed-forward transformations contained redundancies that could be reorganized without impairing functional capacity. Comparisons against conventional quantization and pruning methods confirmed that CCE provided a more balanced trade-off between efficiency and model retention, achieving reductions in energy consumption and inference latency without requiring extensive retraining. Computational efficiency improvements were particularly evident in deployment scenarios involving resource-constrained environments, where reductions in memory usage enabled more scalable implementations. Further analyses of internal network behavior showed that compressed models exhibited stable activation distributions and adapted dynamically to input variations, reinforcing the viability of structured compression strategies for optimizing large-scale architectures.
• 摘要：随着模型规模不断增长，上下文感知压缩技术受到越来越多的关注，引入了阻碍有效部署的计算瓶颈。提出了一种结构化编码方法来选择性地消除冗余参数组，同时确保在多个层中保持表征保真度。上下文压缩编码 (CCE) 引入了一种多阶段编码机制，可以动态重构参数分布，从而显着减少内存占用和计算复杂度。实验评估表明，通过 CCE 压缩的模型保留了语言表现力和连贯性，在一系列文本生成和分类任务中保持准确性。逐层分析表明，中间网络层表现出更高的压缩率，这与自注意力和前馈变换包含冗余的观察结果一致，这些冗余可以在不损害功能容量的情况下进行重组。与传统量化和修剪方法的比较证实，CCE 在效率和模型保留之间提供了更平衡的权衡，无需大量重新训练即可实现能耗和推理延迟的降低。计算效率的提高在涉及资源受限环境的部署场景中尤为明显，其中内存使用量的减少使实现更具可扩展性的实现成为可能。对内部网络行为的进一步分析表明，压缩模型表现出稳定的激活分布并动态适应输入变化，从而增强了结构化压缩策略在优化大规模架构方面的可行性。

Title: Systematic Knowledge Injection into Large Language Models via Diverse Augmentation for Domain-Specific RAG

• Authors: Kushagra Bhushan, Yatin Nandwani, Dinesh Khandelwal, Sonam Gupta, Gaurav Pandey, Dinesh Raghu, Sachindra Joshi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.08356
• Pdf URL: https://arxiv.org/pdf/2502.08356
• Copy Paste: [[2502.08356]] Systematic Knowledge Injection into Large Language Models via Diverse Augmentation for Domain-Specific RAG(https://arxiv.org/abs/2502.08356)
• Keywords: language model, llm, hallucination, retrieval-augmented generation
• Abstract: Retrieval-Augmented Generation (RAG) has emerged as a prominent method for incorporating domain knowledge into Large Language Models (LLMs). While RAG enhances response relevance by incorporating retrieved domain knowledge in the context, retrieval errors can still lead to hallucinations and incorrect answers. To recover from retriever failures, domain knowledge is injected by fine-tuning the model to generate the correct response, even in the case of retrieval errors. However, we observe that without systematic knowledge augmentation, fine-tuned LLMs may memorize new information but still fail to extract relevant domain knowledge, leading to poor performance. In this work, we present a novel framework that significantly enhances the fine-tuning process by augmenting the training data in two ways -- context augmentation and knowledge paraphrasing. In context augmentation, we create multiple training samples for a given QA pair by varying the relevance of the retrieved information, teaching the model when to ignore and when to rely on retrieved content. In knowledge paraphrasing, we fine-tune with multiple answers to the same question, enabling LLMs to better internalize specialized knowledge. To mitigate catastrophic forgetting due to fine-tuning, we add a domain-specific identifier to a question and also utilize a replay buffer containing general QA pairs. Experimental results demonstrate the efficacy of our method over existing techniques, achieving up to 10\% relative gain in token-level recall while preserving the LLM's generalization capabilities.
• 摘要：检索增强生成 (RAG) 已成为将领域知识纳入大型语言模型 (LLM) 的主要方法。虽然 RAG 通过在上下文中整合检索到的领域知识来增强响应相关性，但检索错误仍然会导致幻觉和错误答案。为了从检索器故障中恢复，通过微调模型来注入领域知识以生成正确的响应，即使在检索错误的情况下也是如此。然而，我们观察到，如果没有系统的知识增强，微调的 LLM 可能会记住新信息，但仍然无法提取相关的领域知识，从而导致性能不佳。在这项工作中，我们提出了一个新颖的框架，通过以两种方式增强训练数据（上下文增强和知识释义）来显著增强微调过程。在上下文增强中，我们通过改变检索到的信息的相关性为给定的 QA 对创建多个训练样本，教模型何时忽略以及何时依赖检索到的内容。在知识释义中，我们针对同一问题使用多个答案进行微调，使 LLM 能够更好地内化专业知识。为了减轻由于微调而导致的灾难性遗忘，我们向问题添加了特定领域的标识符，并利用了包含通用 QA 对的重放缓冲区。实验结果证明了我们的方法优于现有技术，在保留 LLM 的泛化能力的同时，在 token 级召回率方面实现了高达 10\% 的相对增益。

Title: Top-Theta Attention: Sparsifying Transformers by Compensated Thresholding

• Authors: Konstantin Berestizshevsky, Renzo Andri, Lukas Cavigelli
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.08363
• Pdf URL: https://arxiv.org/pdf/2502.08363
• Copy Paste: [[2502.08363]] Top-Theta Attention: Sparsifying Transformers by Compensated Thresholding(https://arxiv.org/abs/2502.08363)
• Keywords: language model, llm
• Abstract: The attention mechanism is essential for the impressive capabilities of transformer-based Large Language Models (LLMs). However, calculating attention is computationally intensive due to its quadratic dependency on the sequence length. We introduce a novel approach called Top-Theta Attention, or simply Top-$\theta$, which selectively prunes less essential attention elements by comparing them against carefully calibrated thresholds. This method greatly improves the efficiency of self-attention matrix multiplication while preserving model accuracy, reducing the number of required V cache rows by 3x during generative decoding and the number of attention elements by 10x during the prefill phase. Our method does not require model retraining; instead, it requires only a brief calibration phase to be resilient to distribution shifts, thus not requiring the thresholds for different datasets to be recalibrated. Unlike top-k attention, Top-$\theta$ eliminates full-vector dependency, making it suitable for tiling and scale-out and avoiding costly top-k search. A key innovation of our approach is the development of efficient numerical compensation techniques, which help preserve model accuracy even under aggressive pruning of attention scores.
• 摘要：注意力机制对于基于 Transformer 的大型语言模型 (LLM) 的强大功能至关重要。然而，由于注意力对序列长度具有二次依赖性，因此计算注意力需要大量计算。我们引入了一种称为 Top-Theta 注意力机制（简称 Top-$\theta$）的新方法，该方法通过将不太重要的注意力元素与精心校准的阈值进行比较来选择性地修剪它们。该方法在保持模型准确性的同时大大提高了自注意力矩阵乘法的效率，在生成解码期间将所需的 V 缓存行数减少了 3 倍，在预填充阶段将注意力元素的数量减少了 10 倍。我们的方法不需要模型重新训练；相反，它只需要一个短暂的校准阶段就能适应分布变化，因此不需要重新校准不同数据集的阈值。与 top-k 注意力机制不同，Top-$\theta$ 消除了全向量依赖性，使其适合平铺和横向扩展，并避免了昂贵的 top-k 搜索。我们方法的一个关键创新是开发有效的数值补偿技术，该技术即使在积极修剪注意力分数的情况下也能帮助保持模型准确性。

Title: Unveiling Global Discourse Structures: Theoretical Analysis and NLP Applications in Argument Mining

• Authors: Christopher van Le
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.08371
• Pdf URL: https://arxiv.org/pdf/2502.08371
• Copy Paste: [[2502.08371]] Unveiling Global Discourse Structures: Theoretical Analysis and NLP Applications in Argument Mining(https://arxiv.org/abs/2502.08371)
• Keywords: llm
• Abstract: Particularly in the structure of global discourse, coherence plays a pivotal role in human text comprehension and is a hallmark of high-quality text. This is especially true for persuasive texts, where coherent argument structures support claims effectively. This paper discusses and proposes methods for detecting, extracting and representing these global discourse structures in a proccess called Argument(ation) Mining. We begin by defining key terms and processes of discourse structure analysis, then continue to summarize existing research on the matter, and identify shortcomings in current argument component extraction and classification methods. Furthermore, we will outline an architecture for argument mining that focuses on making models more generalisable while overcoming challenges in the current field of research by utilizing novel NLP techniques. This paper reviews current knowledge, summarizes recent works, and outlines our NLP pipeline, aiming to contribute to the theoretical understanding of global discourse structures.
• 摘要：特别是在全局话语结构中，连贯性在人类文本理解中起着关键作用，是高质量文本的标志。对于说服性文本尤其如此，其中连贯的论证结构有效地支持主张。本文讨论并提出了在称为论证挖掘的过程中检测、提取和表示这些全局话语结构的方法。我们首先定义话语结构分析的关键术语和过程，然后继续总结有关该问题的现有研究，并找出当前论证成分提取和分类方法的不足之处。此外，我们将概述论证挖掘的架构，该架构侧重于利用新颖的 NLP 技术使模型更具通用性，同时克服当前研究领域的挑战。本文回顾了当前的知识，总结了最近的工作，并概述了我们的 NLP 流程，旨在为全局话语结构的理论理解做出贡献。

Title: IssueBench: Millions of Realistic Prompts for Measuring Issue Bias in LLM Writing Assistance

• Authors: Paul Röttger, Musashi Hinck, Valentin Hofmann, Kobi Hackenburg, Valentina Pyatkin, Faeze Brahman, Dirk Hovy
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.08395
• Pdf URL: https://arxiv.org/pdf/2502.08395
• Copy Paste: [[2502.08395]] IssueBench: Millions of Realistic Prompts for Measuring Issue Bias in LLM Writing Assistance(https://arxiv.org/abs/2502.08395)
• Keywords: language model, llm, prompt
• Abstract: Large language models (LLMs) are helping millions of users write texts about diverse issues, and in doing so expose users to different ideas and perspectives. This creates concerns about issue bias, where an LLM tends to present just one perspective on a given issue, which in turn may influence how users think about this issue. So far, it has not been possible to measure which issue biases LLMs actually manifest in real user interactions, making it difficult to address the risks from biased LLMs. Therefore, we create IssueBench: a set of 2.49m realistic prompts for measuring issue bias in LLM writing assistance, which we construct based on 3.9k templates (e.g. "write a blog about") and 212 political issues (e.g. "AI regulation") from real user interactions. Using IssueBench, we show that issue biases are common and persistent in state-of-the-art LLMs. We also show that biases are remarkably similar across models, and that all models align more with US Democrat than Republican voter opinion on a subset of issues. IssueBench can easily be adapted to include other issues, templates, or tasks. By enabling robust and realistic measurement, we hope that IssueBench can bring a new quality of evidence to ongoing discussions about LLM biases and how to address them.
• 摘要：大型语言模型 (LLM) 正在帮助数百万用户撰写有关各种问题的文本，并在此过程中向用户展示不同的想法和观点。这引发了对问题偏见的担忧，即 LLM 往往只针对给定问题提供一种观点，而这反过来可能会影响用户对这个问题的看法。到目前为止，还无法衡量 LLM 在实际用户交互中实际表现出哪些问题偏见，这使得解决有偏见的 LLM 带来的风险变得困难。因此，我们创建了 IssueBench：一组 249 万个现实提示，用于衡量 LLM 写作辅助中的问题偏见，我们基于 3.9k 个模板（例如“写一篇博客”）和 212 个政治问题（例如“人工智能监管”）从实际用户交互中构建了这些提示。使用 IssueBench，我们表明问题偏见在最先进的 LLM 中很常见且持续存在。我们还表明，不同模型之间的偏见非常相似，并且所有模型在部分问题上都更符合美国民主党选民的意见，而不是共和党选民的意见。 IssueBench 可以轻松调整以包含其他问题、模板或任务。通过实现稳健而现实的测量，我们希望 IssueBench 可以为正在进行的有关 LLM 偏见及其解决方法的讨论带来新的证据。

Title: A Semantic Parsing Algorithm to Solve Linear Ordering Problems

• Authors: Maha Alkhairy, Vincent Homer, Brendan O'Connor
• Subjects: cs.CL, cs.LO
• Abstract URL: https://arxiv.org/abs/2502.08415
• Pdf URL: https://arxiv.org/pdf/2502.08415
• Copy Paste: [[2502.08415]] A Semantic Parsing Algorithm to Solve Linear Ordering Problems(https://arxiv.org/abs/2502.08415)
• Keywords: gpt, llm
• Abstract: We develop an algorithm to semantically parse linear ordering problems, which require a model to arrange entities using deductive reasoning. Our method takes as input a number of premises and candidate statements, parsing them to a first-order logic of an ordering domain, and then utilizes constraint logic programming to infer the truth of proposed statements about the ordering. Our semantic parser transforms Heim and Kratzer's syntax-based compositional formal semantic rules to a computational algorithm. This transformation involves introducing abstract types and templates based on their rules, and introduces a dynamic component to interpret entities within a contextual framework. Our symbolic system, the Formal Semantic Logic Inferer (FSLI), is applied to answer multiple choice questions in BIG-bench's logical_deduction multiple choice problems, achieving perfect accuracy, compared to 67.06% for the best-performing LLM (GPT-4) and 87.63% for the hybrid system Logic-LM. These promising results demonstrate the benefit of developing a semantic parsing algorithm driven by first-order logic constructs.
• 摘要：我们开发了一种算法来语义解析线性排序问题，这需要一个模型使用演绎推理来排列实体。我们的方法将许多前提和候选语句作为输入，将它们解析为排序域的一阶逻辑，然后利用约束逻辑编程来推断关于排序的拟议语句的真实性。我们的语义解析器将 Heim 和 Kratzer 基于语法的组合形式语义规则转换为计算算法。这种转换涉及根据规则引入抽象类型和模板，并引入动态组件来在上下文框架内解释实体。我们的符号系统形式语义逻辑推断器 (FSLI) 被用于回答 BIG-bench 的逻辑演绎多项选择题中的多项选择题，实现了完美的准确率，而表现最佳的 LLM (GPT-4) 的准确率为 67.06%，混合系统 Logic-LM 的准确率为 87.63%。这些有希望的结果证明了开发由一阶逻辑结构驱动的语义解析算法的好处。

Title: From Haystack to Needle: Label Space Reduction for Zero-shot Classification

• Authors: Nathan Vandemoortele, Bram Steenwinckel, Femke Ongenae, Sofie Van Hoecke
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.08436
• Pdf URL: https://arxiv.org/pdf/2502.08436
• Copy Paste: [[2502.08436]] From Haystack to Needle: Label Space Reduction for Zero-shot Classification(https://arxiv.org/abs/2502.08436)
• Keywords: language model, llm
• Abstract: We present Label Space Reduction (LSR), a novel method for improving zero-shot classification performance of Large Language Models (LLMs). LSR iteratively refines the classification label space by systematically ranking and reducing candidate classes, enabling the model to concentrate on the most relevant options. By leveraging unlabeled data with the statistical learning capabilities of data-driven models, LSR dynamically optimizes the label space representation at test time. Our experiments across seven benchmarks demonstrate that LSR improves macro-F1 scores by an average of 7.0% (up to 14.2%) with Llama-3.1-70B and 3.3% (up to 11.1%) with Claude-3.5-Sonnet compared to standard zero-shot classification baselines. To reduce the computational overhead of LSR, which requires an additional LLM call at each iteration, we propose distilling the model into a probabilistic classifier, allowing for efficient inference.
• 摘要：我们提出了标签空间缩减 (LSR)，这是一种用于提高大型语言模型 (LLM) 零样本分类性能的新方法。LSR 通过系统地对候选类别进行排名和缩减，以迭代方式细化分类标签空间，从而使模型能够集中于最相关的选项。通过利用未标记数据和数据驱动模型的统计学习能力，LSR 在测试时动态优化标签空间表示。我们在七个基准测试中开展的实验表明，与标准零样本分类基线相比，LSR 在 Llama-3.1-70B 上平均将宏 F1 分数提高了 7.0%（最高可达 14.2%），在 Claude-3.5-Sonnet 上平均将宏 F1 分数提高了 3.3%（最高可达 11.1%）。为了减少 LSR 的计算开销（每次迭代都需要额外的 LLM 调用），我们建议将模型提炼为概率分类器，以实现高效推理。

Title: Better Embeddings with Coupled Adam

• Authors: Felix Stollenwerk, Tobias Stollenwerk
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.08441
• Pdf URL: https://arxiv.org/pdf/2502.08441
• Copy Paste: [[2502.08441]] Better Embeddings with Coupled Adam(https://arxiv.org/abs/2502.08441)
• Keywords: llm
• Abstract: Despite their remarkable capabilities, LLMs learn word representations that exhibit the undesirable yet poorly understood feature of anisotropy. In this paper, we argue that the second moment in Adam is a cause of anisotropic embeddings, and suggest a modified optimizer called Coupled Adam to mitigate the problem. Our experiments demonstrate that Coupled Adam significantly improves the quality of embeddings, while also leading to better upstream and downstream performance on large enough datasets.
• 摘要：尽管 LLM 具有出色的能力，但它们学习的单词表示却表现出了不受欢迎且不太为人所知的各向异性特征。在本文中，我们认为 Adam 中的二阶矩是各向异性嵌入的原因，并提出了一种名为 Coupled Adam 的改进优化器来缓解该问题。我们的实验表明，Coupled Adam 显著提高了嵌入的质量，同时在足够大的数据集上也带来了更好的上游和下游性能。

Title: Towards Prompt Generalization: Grammar-aware Cross-Prompt Automated Essay Scoring

• Authors: Heejin Do, Taehee Park, Sangwon Ryu, Gary Geunbae Lee
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.08450
• Pdf URL: https://arxiv.org/pdf/2502.08450
• Copy Paste: [[2502.08450]] Towards Prompt Generalization: Grammar-aware Cross-Prompt Automated Essay Scoring(https://arxiv.org/abs/2502.08450)
• Keywords: prompt
• Abstract: In automated essay scoring (AES), recent efforts have shifted toward cross-prompt settings that score essays on unseen prompts for practical applicability. However, prior methods trained with essay-score pairs of specific prompts pose challenges in obtaining prompt-generalized essay representation. In this work, we propose a grammar-aware cross-prompt trait scoring (GAPS), which internally captures prompt-independent syntactic aspects to learn generic essay representation. We acquire grammatical error-corrected information in essays via the grammar error correction technique and design the AES model to seamlessly integrate such information. By internally referring to both the corrected and the original essays, the model can focus on generic features during training. Empirical experiments validate our method's generalizability, showing remarkable improvements in prompt-independent and grammar-related traits. Furthermore, GAPS achieves notable QWK gains in the most challenging cross-prompt scenario, highlighting its strength in evaluating unseen prompts.
• 摘要：在自动作文评分 (AES) 中，最近的努力已转向跨提示设置，即根据未见过的提示对作文进行评分，以实现实际适用性。然而，之前使用特定提示的作文分数对进行训练的方法在获得提示泛化作文表征方面提出了挑战。在这项工作中，我们提出了一种语法感知的跨提示特征评分 (GAPS)，它在内部捕获与提示无关的句法方面以学习通用的作文表征。我们通过语法错误校正技术获取作文中的语法错误校正信息，并设计 AES 模型以无缝集成此类信息。通过内部参考已校正和原始作文，该模型可以在训练期间关注通用特征。实证实验验证了我们方法的泛化能力，显示出与提示无关和与语法相关的特征的显着改善。此外，GAPS 在最具挑战性的跨提示场景中实现了显着的 QWK 增益，凸显了其在评估未见过的提示方面的优势。

Title: Enhancing Auto-regressive Chain-of-Thought through Loop-Aligned Reasoning

• Authors: Qifan Yu, Zhenyu He, Sijie Li, Xun Zhou, Jun Zhang, Jingjing Xu, Di He
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.08482
• Pdf URL: https://arxiv.org/pdf/2502.08482
• Copy Paste: [[2502.08482]] Enhancing Auto-regressive Chain-of-Thought through Loop-Aligned Reasoning(https://arxiv.org/abs/2502.08482)
• Keywords: language model, prompt, chain-of-thought
• Abstract: Chain-of-Thought (CoT) prompting has emerged as a powerful technique for enhancing language model's reasoning capabilities. However, generating long and correct CoT trajectories is challenging. Recent studies have demonstrated that Looped Transformers possess remarkable length generalization capabilities, but their limited generality and adaptability prevent them from serving as an alternative to auto-regressive solutions. To better leverage the strengths of Looped Transformers, we propose RELAY (REasoning through Loop Alignment iterativelY). Specifically, we align the steps of Chain-of-Thought (CoT) reasoning with loop iterations and apply intermediate supervision during the training of Looped Transformers. This additional iteration-wise supervision not only preserves the Looped Transformer's ability for length generalization but also enables it to predict CoT reasoning steps for unseen data. Therefore, we leverage this Looped Transformer to generate accurate reasoning chains for complex problems that exceed the training length, which will then be used to fine-tune an auto-regressive model. We conduct extensive experiments, and the results demonstrate the effectiveness of our approach, with significant improvements in the performance of the auto-regressive model. Code will be released at this https URL.
• 摘要：思路链 (CoT) 提示已成为增强语言模型推理能力的强大技术。然而，生成长而正确的 CoT 轨迹具有挑战性。最近的研究表明，Looped Transformers 具有出色的长度泛化能力，但其有限的通用性和适应性使其无法作为自回归解决方案的替代方案。为了更好地利用 Looped Transformers 的优势，我们提出了 RELAY（通过循环对齐迭代进行推理）。具体来说，我们将思路链 (CoT) 推理的步骤与循环迭代对齐，并在 Looped Transformers 的训练期间应用中间监督。这种额外的迭代监督不仅保留了 Looped Transformer 的长度泛化能力，还使其能够预测未见数据的 CoT 推理步骤。因此，我们利用这个 Looped Transformer 为超出训练长度的复杂问题生成准确的推理链，然后将其用于微调自回归模型。我们进行了大量的实验，结果证明了我们方法的有效性，自回归模型的性能得到了显著的提升。代码将在此 https URL 上发布。

Title: Salamandra Technical Report

• Authors: Aitor Gonzalez-Agirre, Marc Pàmies, Joan Llop, Irene Baucells, Severino Da Dalt, Daniel Tamayo, José Javier Saiz, Ferran Espuña, Jaume Prats, Javier Aula-Blasco, Mario Mina, Adrián Rubio, Alexander Shvets, Anna Sallés, Iñaki Lacunza, Iñigo Pikabea, Jorge Palomar, Júlia Falcão, Lucía Tormo, Luis Vasquez-Reina, Montserrat Marimon, Valle Ruíz-Fernández, Marta Villegas
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.08489
• Pdf URL: https://arxiv.org/pdf/2502.08489
• Copy Paste: [[2502.08489]] Salamandra Technical Report(https://arxiv.org/abs/2502.08489)
• Keywords: language model, chat
• Abstract: This work introduces Salamandra, a suite of open-source decoder-only large language models available in three different sizes: 2, 7, and 40 billion parameters. The models were trained from scratch on highly multilingual data that comprises text in 35 European languages and code. Our carefully curated corpus is made exclusively from open-access data compiled from a wide variety of sources. Along with the base models, supplementary checkpoints that were fine-tuned on public-domain instruction data are also released for chat applications. Additionally, we also share our preliminary experiments on multimodality, which serve as proof-of-concept to showcase potential applications for the Salamandra family. Our extensive evaluations on multilingual benchmarks reveal that Salamandra has strong capabilities, achieving competitive performance when compared to similarly sized open-source models. We provide comprehensive evaluation results both on standard downstream tasks as well as key aspects related to bias and this http URL this technical report, we intend to promote open science by sharing all the details behind our design choices, data curation strategy and evaluation methodology. In addition to that, we deviate from the usual practice by making our training and evaluation scripts publicly accessible. We release all models under a permissive Apache 2.0 license in order to foster future research and facilitate commercial use, thereby contributing to the open-source ecosystem of large language models.
• 摘要：这项工作引入了 Salamandra，这是一套开源的解码器专用大型语言模型，有三种不同的大小：20 亿、70 亿和 400 亿个参数。这些模型从头开始训练，使用高度多语言的数据，包括 35 种欧洲语言和代码的文本。我们精心策划的语料库完全由从各种来源汇编的开放访问数据组成。除了基础模型外，还发布了针对公共领域指令数据进行微调的补充检查点，用于聊天应用程序。此外，我们还分享了我们在多模态方面的初步实验，这些实验作为概念验证，展示了 Salamandra 系列的潜在应用。我们对多语言基准的广泛评估表明，Salamandra 具有强大的功能，与类似大小的开源模型相比，其性能具有竞争力。我们提供了标准下游任务以及与偏见相关的关键方面的全面评估结果，并且通过这份技术报告，我们打算通过分享我们的设计选择、数据管理策略和评估方法背后的所有细节来促进开放科学。此外，我们还一反常态地将我们的训练和评估脚本公开。我们根据宽松的 Apache 2.0 许可发布所有模型，以促进未来的研究并促进商业使用，从而为大型语言模型的开源生态系统做出贡献。

Title: Explanation based In-Context Demonstrations Retrieval for Multilingual Grammatical Error Correction

• Authors: Wei Li, Wen Luo, Guangyue Peng, Houfeng Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.08507
• Pdf URL: https://arxiv.org/pdf/2502.08507
• Copy Paste: [[2502.08507]] Explanation based In-Context Demonstrations Retrieval for Multilingual Grammatical Error Correction(https://arxiv.org/abs/2502.08507)
• Keywords: language model, llm
• Abstract: Grammatical error correction (GEC) aims to correct grammatical, spelling, and semantic errors in natural language text. With the growing of large language models (LLMs), direct text generation has gradually become the focus of the GEC methods, and few-shot in-context learning presents a cost-effective solution. However, selecting effective in-context examples remains challenging, as the similarity between input texts does not necessarily correspond to similar grammatical error patterns. In this paper, we propose a novel retrieval method based on natural language grammatical error explanations (GEE) to address this issue. Our method retrieves suitable few-shot demonstrations by matching the GEE of the test input with that of pre-constructed database samples, where explanations for erroneous samples are generated by LLMs. We conducted multilingual GEC few-shot experiments on both major open-source and closed-source LLMs. Experiments across five languages show that our method outperforms existing semantic and BM25-based retrieval techniques, without requiring additional training or language adaptation. This also suggests that matching error patterns is key to selecting examples.
• 摘要：语法错误纠正 (GEC) 旨在纠正自然语言文本中的语法、拼写和语义错误。随着大型语言模型 (LLM) 的发展，直接文本生成逐渐成为 GEC 方法的重点，而少样本上下文学习是一种经济有效的解决方案。然而，选择有效的上下文示例仍然具有挑战性，因为输入文本之间的相似性不一定对应于相似的语法错误模式。在本文中，我们提出了一种基于自然语言语法错误解释 (GEE) 的新型检索方法来解决这个问题。我们的方法通过将测试输入的 GEE 与预先构建的数据库样本的 GEE 进行匹配来检索合适的少样本演示，其中错误样本的解释由 LLM 生成。我们在主要的开源和闭源 LLM 上进行了多语言 GEC 少样本实验。跨五种语言的实验表明，我们的方法优于现有的语义和基于 BM25 的检索技术，而无需额外的训练或语言适应。这也表明匹配错误模式是选择示例的关键。

Title: Measuring Diversity in Synthetic Datasets

• Authors: Yuchang Zhu, Huizhe Zhang, Bingzhe Wu, Jintang Li, Zibin Zheng, Peilin Zhao, Liang Chen, Yatao Bian
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.08512
• Pdf URL: https://arxiv.org/pdf/2502.08512
• Copy Paste: [[2502.08512]] Measuring Diversity in Synthetic Datasets(https://arxiv.org/abs/2502.08512)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) are widely adopted to generate synthetic datasets for various natural language processing (NLP) tasks, such as text classification and summarization. However, accurately measuring the diversity of these synthetic datasets-an aspect crucial for robust model performance-remains a significant challenge. In this paper, we introduce DCScore, a novel method for measuring synthetic dataset diversity from a classification perspective. Specifically, DCScore formulates diversity evaluation as a sample classification task, leveraging mutual relationships among samples. We further provide theoretical verification of the diversity-related axioms satisfied by DCScore, highlighting its role as a principled diversity evaluation method. Experimental results on synthetic datasets reveal that DCScore enjoys a stronger correlation with multiple diversity pseudo-truths of evaluated datasets, underscoring its effectiveness. Moreover, both empirical and theoretical evidence demonstrate that DCScore substantially reduces computational costs compared to existing approaches. Code is available at: this https URL.
• 摘要：大型语言模型 (LLM) 被广泛用于为各种自然语言处理 (NLP) 任务（例如文本分类和摘要）生成合成数据集。然而，准确测量这些合成数据集的多样性（这是稳健模型性能的关键方面）仍然是一项重大挑战。在本文中，我们介绍了 DCScore，这是一种从分类角度测量合成数据集多样性的新方法。具体而言，DCScore 将多样性评估制定为样本分类任务，利用样本之间的相互关系。我们进一步对 DCScore 满足的多样性相关公理进行了理论验证，强调了其作为原则性多样性评估方法的作用。在合成数据集上的实验结果表明，DCScore 与评估数据集的多个多样性伪真值具有更强的相关性，强调了其有效性。此外，经验和理论证据都表明，与现有方法相比，DCScore 大大降低了计算成本。代码可在以下位置获得：此 https URL。

Title: Faithful, Unfaithful or Ambiguous? Multi-Agent Debate with Initial Stance for Summary Evaluation

• Authors: Mahnaz Koupaee, Jake W. Vincent, Saab Mansour, Igor Shalyminov, Han He, Hwanjun Song, Raphael Shu, Jianfeng He, Yi Nian, Amy Wing-mei Wong, Kyu J. Han, Hang Su
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.08514
• Pdf URL: https://arxiv.org/pdf/2502.08514
• Copy Paste: [[2502.08514]] Faithful, Unfaithful or Ambiguous? Multi-Agent Debate with Initial Stance for Summary Evaluation(https://arxiv.org/abs/2502.08514)
• Keywords: language model, llm, agent
• Abstract: Faithfulness evaluators based on large language models (LLMs) are often fooled by the fluency of the text and struggle with identifying errors in the summaries. We propose an approach to summary faithfulness evaluation in which multiple LLM-based agents are assigned initial stances (regardless of what their belief might be) and forced to come up with a reason to justify the imposed belief, thus engaging in a multi-round debate to reach an agreement. The uniformly distributed initial assignments result in a greater diversity of stances leading to more meaningful debates and ultimately more errors identified. Furthermore, by analyzing the recent faithfulness evaluation datasets, we observe that naturally, it is not always the case for a summary to be either faithful to the source document or not. We therefore introduce a new dimension, ambiguity, and a detailed taxonomy to identify such special cases. Experiments demonstrate our approach can help identify ambiguities, and have even a stronger performance on non-ambiguous summaries.
• 摘要：基于大型语言模型 (LLM) 的忠实度评估者经常被文本的流畅性所欺骗，并且很难识别摘要中的错误。我们提出了一种摘要忠实度评估方法，其中为多个基于 LLM 的代理分配初始立场（无论他们的信念是什么），并被迫提出理由来证明强加的信念，从而进行多轮辩论以达成一致。均匀分布的初始分配导致立场更加多样化，从而引发更有意义的辩论，最终发现更多错误。此外，通过分析最近的忠实度评估数据集，我们观察到，摘要并不总是忠实于源文档。因此，我们引入了一个新的维度，即歧义性，以及一个详细的分类法来识别这种特殊情况。实验表明，我们的方法可以帮助识别歧义，并且在非歧义摘要上具有更强的性能。

Title: LLMs can implicitly learn from mistakes in-context

• Authors: Lisa Alazraki, Maximilian Mozes, Jon Ander Campos, Yi Chern Tan, Marek Rei, Max Bartolo
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.08550
• Pdf URL: https://arxiv.org/pdf/2502.08550
• Copy Paste: [[2502.08550]] LLMs can implicitly learn from mistakes in-context(https://arxiv.org/abs/2502.08550)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: Learning from mistakes is a fundamental feature of human intelligence. Previous work has shown that Large Language Models (LLMs) can also learn from incorrect answers when provided with a comprehensive rationale detailing why an answer is wrong or how to correct it. In this work, we examine whether LLMs can learn from mistakes in mathematical reasoning tasks when these explanations are not provided. We investigate if LLMs are able to implicitly infer such rationales simply from observing both incorrect and correct answers. Surprisingly, we find that LLMs perform better, on average, when rationales are eliminated from the context and incorrect answers are simply shown alongside correct ones. This approach also substantially outperforms chain-of-thought prompting in our evaluations. We show that these results are consistent across LLMs of different sizes and varying reasoning abilities. Further, we carry out an in-depth analysis, and show that prompting with both wrong and correct answers leads to greater performance and better generalisation than introducing additional, more diverse question-answer pairs into the context. Finally, we show that new rationales generated by models that have only observed incorrect and correct answers are scored equally as highly by humans as those produced with the aid of exemplar rationales. Our results demonstrate that LLMs are indeed capable of in-context implicit learning.
• 摘要：从错误中学习是人类智能的基本特征。先前的研究表明，当提供详细说明答案错误的原因或如何纠正答案的全面理由时，大型语言模型 (LLM) 也可以从错误答案中学习。在这项工作中，我们研究了在没有提供这些解释的情况下，LLM 是否可以从数学推理任务中的错误中学习。我们调查了 LLM 是否能够仅通过观察错误和正确答案就隐式推断出这些理由。令人惊讶的是，我们发现，当从上下文中消除理由并将错误答案简单地与正确答案一起显示时，LLM 的平均表现更好。在我们的评估中，这种方法也大大优于思路链提示。我们表明，这些结果在不同规模和不同推理能力的 LLM 中是一致的。此外，我们进行了深入分析，并表明与在上下文中引入更多、更多样化的问答对相比，使用错误和正确答案进行提示可以提高性能和更好的泛化。最后，我们表明，仅观察了错误和正确答案的模型生成的新原理与借助示例原理生成的新原理在人类眼中的得分一样高。我们的结果表明 LLM 确实能够进行上下文内隐学习。

Title: SPeCtrum: A Grounded Framework for Multidimensional Identity Representation in LLM-Based Agent

• Authors: Keyeun Lee, Seo Hyeong Kim, Seolhee Lee, Jinsu Eun, Yena Ko, Hayeon Jeon, Esther Hehsun Kim, Seonghye Cho, Soeun Yang, Eun-mee Kim, Hajin Lim
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.08599
• Pdf URL: https://arxiv.org/pdf/2502.08599
• Copy Paste: [[2502.08599]] SPeCtrum: A Grounded Framework for Multidimensional Identity Representation in LLM-Based Agent(https://arxiv.org/abs/2502.08599)
• Keywords: llm, agent
• Abstract: Existing methods for simulating individual identities often oversimplify human complexity, which may lead to incomplete or flattened representations. To address this, we introduce SPeCtrum, a grounded framework for constructing authentic LLM agent personas by incorporating an individual's multidimensional self-concept. SPeCtrum integrates three core components: Social Identity (S), Personal Identity (P), and Personal Life Context (C), each contributing distinct yet interconnected aspects of identity. To evaluate SPeCtrum's effectiveness in identity representation, we conducted automated and human evaluations. Automated evaluations using popular drama characters showed that Personal Life Context (C)-derived from short essays on preferences and daily routines-modeled characters' identities more effectively than Social Identity (S) and Personal Identity (P) alone and performed comparably to the full SPC combination. In contrast, human evaluations involving real-world individuals found that the full SPC combination provided a more comprehensive self-concept representation than C alone. Our findings suggest that while C alone may suffice for basic identity simulation, integrating S, P, and C enhances the authenticity and accuracy of real-world identity representation. Overall, SPeCtrum offers a structured approach for simulating individuals in LLM agents, enabling more personalized human-AI interactions and improving the realism of simulation-based behavioral studies.
• 摘要：现有的模拟个人身份的方法往往过于简化人类的复杂性，这可能导致表征不完整或扁平化。为了解决这个问题，我们引入了 SPeCtrum，这是一个通过结合个人的多维自我概念来构建真实 LLM 代理角色的扎实框架。SPeCtrum 集成了三个核心组成部分：社会身份 (S)、个人身份 (P) 和个人生活背景 (C)，每个组成部分都贡献了身份的不同但相互关联的方面。为了评估 SPeCtrum 在身份表征方面的有效性，我们进行了自动和人工评估。使用流行戏剧角色进行的自动评估表明，个人生活背景 (C)（源自关于偏好和日常生活的短文）比单独的社会身份 (S) 和个人身份 (P) 更有效地模拟人物身份，并且表现与完整的 SPC 组合相当。相比之下，涉及现实世界个体的人工评估发现，完整的 SPC 组合比单独的 C 提供了更全面的自我概念表征。我们的研究结果表明，虽然 C 本身就足以进行基本的身份模拟，但整合 S、P 和 C 可以提高现实世界身份表征的真实性和准确性。总体而言，SPeCtrum 提供了一种结构化方法来模拟 LLM 代理中的个人，从而实现更加个性化的人机交互，并提高基于模拟的行为研究的真实性。

Title: Examining Multilingual Embedding Models Cross-Lingually Through LLM-Generated Adversarial Examples

• Authors: Andrianos Michail, Simon Clematide, Rico Sennrich
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.08638
• Pdf URL: https://arxiv.org/pdf/2502.08638
• Copy Paste: [[2502.08638]] Examining Multilingual Embedding Models Cross-Lingually Through LLM-Generated Adversarial Examples(https://arxiv.org/abs/2502.08638)
• Keywords: language model, llm
• Abstract: The evaluation of cross-lingual semantic search capabilities of models is often limited to existing datasets from tasks such as information retrieval and semantic textual similarity. To allow for domain-specific evaluation, we introduce Cross Lingual Semantic Discrimination (CLSD), a novel cross-lingual semantic search task that requires only a set of parallel sentence pairs of the language pair of interest within the target domain. This task focuses on the ability of a model to cross-lingually rank the true parallel sentence higher than hard negatives generated by a large language model. We create four instances of our introduced CLSD task for the language pair German-French within the domain of news. Within this case study, we find that models that are also fine-tuned for retrieval tasks (e.g., multilingual E5) benefit from using English as the pivot language, while bitext mining models such as LaBSE perform best directly cross-lingually. We also show a fine-grained similarity analysis enabled by our distractor generation strategy, indicating that different embedding models are sensitive to different types of perturbations.
• 摘要：对模型跨语言语义搜索能力的评估通常仅限于信息检索和语义文本相似性等任务的现有数据集。为了进行特定领域的评估，我们引入了跨语言语义鉴别 (CLSD)，这是一种新颖的跨语言语义搜索任务，只需要目标领域内感兴趣的语言对的一组平行句子对。此任务侧重于模型跨语言将真实平行句子的排名高于大型语言模型生成的硬否定句的能力。我们为新闻领域中的德语-法语语言对创建了我们引入的 CLSD 任务的四个实例。在此案例研究中，我们发现同样针对检索任务进行微调的模型（例如多语言 E5）受益于使用英语作为枢轴语言，而双语文本挖掘模型（如 LaBSE）在跨语言方面表现最佳。我们还展示了由我们的干扰项生成策略实现的细粒度相似性分析，表明不同的嵌入模型对不同类型的扰动很敏感。