2025-02-07

Title: Context-Preserving Gradient Modulation for Large Language Models: A Novel Approach to Semantic Consistency in Long-Form Text Generation

• Authors: Nirola Kobanov, Edmund Weatherstone, Zachary Vanderpoel, Orlando Wetherby
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.03643
• Pdf URL: https://arxiv.org/pdf/2502.03643
• Copy Paste: [[2502.03643]] Context-Preserving Gradient Modulation for Large Language Models: A Novel Approach to Semantic Consistency in Long-Form Text Generation(https://arxiv.org/abs/2502.03643)
• Keywords: language model
• Abstract: Maintaining semantic consistency over extended text sequences remains a fundamental challenge in long-form text generation, where conventional training methodologies often struggle to prevent contextual drift and coherence degradation. A novel gradient modulation approach is introduced, designed to adjust parameter updates dynamically in response to contextual relevance, ensuring that generated text remains aligned with prior discourse. By integrating a modulation function that selectively amplifies or attenuates gradients based on learned contextual dependencies, the proposed method enhances the stability of model-generated narratives without imposing significant computational overhead. Comparative evaluations against baseline models reveal improvements in coherence, contextual retention, and long-range dependency tracking, demonstrating the effectiveness of modifying the learning process at the gradient level. The results indicate that sentence structure variability and lexical diversity benefit from this approach, mitigating repetitive phrasing and improving adaptability across diverse linguistic contexts. Statistical validation of coherence metrics further substantiates the observed enhancements, with a significant reduction in inconsistencies emerging as a direct consequence of the modulation mechanism. Computational efficiency assessments confirm that the framework achieves these gains without requiring substantial modifications to the underlying architecture, ensuring compatibility with existing optimization workflows.
• 摘要：

Title: Looking for the Inner Music: Probing LLMs' Understanding of Literary Style

• Authors: Rebecca M. M. Hicke, David Mimno
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.03647
• Pdf URL: https://arxiv.org/pdf/2502.03647
• Copy Paste: [[2502.03647]] Looking for the Inner Music: Probing LLMs' Understanding of Literary Style(https://arxiv.org/abs/2502.03647)
• Keywords: language model, llm
• Abstract: Recent work has demonstrated that language models can be trained to identify the author of much shorter literary passages than has been thought feasible for traditional stylometry. We replicate these results for authorship and extend them to a new dataset measuring novel genre. We find that LLMs are able to distinguish authorship and genre, but they do so in different ways. Some models seem to rely more on memorization, while others benefit more from training to learn author/genre characteristics. We then use three methods to probe one high-performing LLM for features that define style. These include direct syntactic ablations to input text as well as two methods that look at model internals. We find that authorial style is easier to define than genre-level style and is more impacted by minor syntactic decisions and contextual word usage. However, some traits like pronoun usage and word order prove significant for defining both kinds of literary style.
• 摘要：

Title: Advancing Reasoning in Large Language Models: Promising Methods and Approaches

• Authors: Avinash Patil
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.03671
• Pdf URL: https://arxiv.org/pdf/2502.03671
• Copy Paste: [[2502.03671]] Advancing Reasoning in Large Language Models: Promising Methods and Approaches(https://arxiv.org/abs/2502.03671)
• Keywords: language model, llm, hallucination, prompt, chain-of-thought, tree-of-thought
• Abstract: Large Language Models (LLMs) have succeeded remarkably in various natural language processing (NLP) tasks, yet their reasoning capabilities remain a fundamental challenge. While LLMs exhibit impressive fluency and factual recall, their ability to perform complex reasoning-spanning logical deduction, mathematical problem-solving, commonsense inference, and multi-step reasoning-often falls short of human expectations. This survey provides a comprehensive review of emerging techniques enhancing reasoning in LLMs. We categorize existing methods into key approaches, including prompting strategies (e.g., Chain-of-Thought reasoning, Self-Consistency, and Tree-of-Thought reasoning), architectural innovations (e.g., retrieval-augmented models, modular reasoning networks, and neuro-symbolic integration), and learning paradigms (e.g., fine-tuning with reasoning-specific datasets, reinforcement learning, and self-supervised reasoning objectives). Additionally, we explore evaluation frameworks used to assess reasoning in LLMs and highlight open challenges, such as hallucinations, robustness, and reasoning generalization across diverse tasks. By synthesizing recent advancements, this survey aims to provide insights into promising directions for future research and practical applications of reasoning-augmented LLMs.
• 摘要：

Title: Reflection-Window Decoding: Text Generation with Selective Refinement

• Authors: Zeyu Tang, Zhenhao Chen, Loka Li, Xiangchen Song, Yunlong Deng, Yifan Shen, Guangyi Chen, Peter Spirtes, Kun Zhang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.03678
• Pdf URL: https://arxiv.org/pdf/2502.03678
• Copy Paste: [[2502.03678]] Reflection-Window Decoding: Text Generation with Selective Refinement(https://arxiv.org/abs/2502.03678)
• Keywords: language model, llm
• Abstract: The autoregressive decoding for text generation in large language models (LLMs), while widely used, is inherently suboptimal due to the lack of a built-in mechanism to perform refinement and/or correction of the generated content. In this paper, we consider optimality in terms of the joint probability over the generated response, when jointly considering all tokens at the same time. We theoretically characterize the potential deviation of the autoregressively generated response from its globally optimal counterpart that is of the same length. Our analysis suggests that we need to be cautious when noticeable uncertainty arises during text generation, which may signal the sub-optimality of the generation history. To address the pitfall of autoregressive decoding for text generation, we propose an approach that incorporates a sliding reflection window and a pausing criterion, such that refinement and generation can be carried out interchangeably as the decoding proceeds. Our selective refinement framework strikes a balance between efficiency and optimality, and our extensive experimental results demonstrate the effectiveness of our approach.
• 摘要：

Title: Controlled LLM Decoding via Discrete Auto-regressive Biasing

• Authors: Patrick Pynadath, Ruqi Zhang
• Subjects: cs.CL, cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2502.03685
• Pdf URL: https://arxiv.org/pdf/2502.03685
• Copy Paste: [[2502.03685]] Controlled LLM Decoding via Discrete Auto-regressive Biasing(https://arxiv.org/abs/2502.03685)
• Keywords: language model, llm
• Abstract: Controlled text generation allows for enforcing user-defined constraints on large language model outputs, an increasingly important field as LLMs become more prevalent in everyday life. One common approach uses energy-based decoding, which defines a target distribution through an energy function that combines multiple constraints into a weighted average. However, these methods often struggle to balance fluency with constraint satisfaction, even with extensive tuning of the energy function's coefficients. In this paper, we identify that this suboptimal balance arises from sampling in continuous space rather than the natural discrete space of text tokens. To address this, we propose Discrete Auto-regressive Biasing, a controlled decoding algorithm that leverages gradients while operating entirely in the discrete text domain. Specifically, we introduce a new formulation for controlled text generation by defining a joint distribution over the generated sequence and an auxiliary bias sequence. To efficiently sample from this joint distribution, we propose a Langevin-within-Gibbs sampling algorithm using gradient-based discrete MCMC. Our method significantly improves constraint satisfaction while maintaining comparable or better fluency, all with even lower computational costs. We demonstrate the advantages of our controlled decoding method on sentiment control, language detoxification, and keyword-guided generation.
• 摘要：

Title: A Comparison of DeepSeek and Other LLMs

• Authors: Tianchen Gao, Jiashun Jin, Zheng Tracy Ke, Gabriel Moryoussef
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.03688
• Pdf URL: https://arxiv.org/pdf/2502.03688
• Copy Paste: [[2502.03688]] A Comparison of DeepSeek and Other LLMs(https://arxiv.org/abs/2502.03688)
• Keywords: language model, gpt, llm
• Abstract: Recently, DeepSeek has been the focus of attention in and beyond the AI community. An interesting problem is how DeepSeek compares to other large language models (LLMs). There are many tasks an LLM can do, and in this paper, we use the task of predicting an outcome using a short text for comparison. We consider two settings, an authorship classification setting and a citation classification setting. In the first one, the goal is to determine whether a short text is written by human or AI. In the second one, the goal is to classify a citation to one of four types using the textual content. For each experiment, we compare DeepSeek with $4$ popular LLMs: Claude, Gemini, GPT, and Llama. We find that, in terms of classification accuracy, DeepSeek outperforms Gemini, GPT, and Llama in most cases, but underperforms Claude. We also find that DeepSeek is comparably slower than others but with a low cost to use, while Claude is much more expensive than all the others. Finally, we find that in terms of similarity, the output of DeepSeek is most similar to those of Gemini and Claude (and among all $5$ LLMs, Claude and Gemini have the most similar outputs). In this paper, we also present a fully-labeled dataset collected by ourselves, and propose a recipe where we can use the LLMs and a recent data set, MADStat, to generate new data sets. The datasets in our paper can be used as benchmarks for future study on LLMs.
• 摘要：

Title: LLM Alignment as Retriever Optimization: An Information Retrieval Perspective

• Authors: Bowen Jin, Jinsung Yoon, Zhen Qin, Ziqi Wang, Wei Xiong, Yu Meng, Jiawei Han, Sercan O. Arik
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2502.03699
• Pdf URL: https://arxiv.org/pdf/2502.03699
• Copy Paste: [[2502.03699]] LLM Alignment as Retriever Optimization: An Information Retrieval Perspective(https://arxiv.org/abs/2502.03699)
• Keywords: language model, llm, hallucination
• Abstract: Large Language Models (LLMs) have revolutionized artificial intelligence with capabilities in reasoning, coding, and communication, driving innovation across industries. Their true potential depends on effective alignment to ensure correct, trustworthy and ethical behavior, addressing challenges like misinformation, hallucinations, bias and misuse. While existing Reinforcement Learning (RL)-based alignment methods are notoriously complex, direct optimization approaches offer a simpler alternative. In this work, we introduce a novel direct optimization approach for LLM alignment by drawing on established Information Retrieval (IR) principles. We present a systematic framework that bridges LLM alignment and IR methodologies, mapping LLM generation and reward models to IR's retriever-reranker paradigm. Building on this foundation, we propose LLM Alignment as Retriever Preference Optimization (LarPO), a new alignment method that enhances overall alignment quality. Extensive experiments validate LarPO's effectiveness with 38.9 % and 13.7 % averaged improvement on AlpacaEval2 and MixEval-Hard respectively. Our work opens new avenues for advancing LLM alignment by integrating IR foundations, offering a promising direction for future research.
• 摘要：

Title: Aggregate and conquer: detecting and steering LLM concepts by combining nonlinear predictors over multiple layers

• Authors: Daniel Beaglehole, Adityanarayanan Radhakrishnan, Enric Boix-Adserà, Mikhail Belkin
• Subjects: cs.CL, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2502.03708
• Pdf URL: https://arxiv.org/pdf/2502.03708
• Copy Paste: [[2502.03708]] Aggregate and conquer: detecting and steering LLM concepts by combining nonlinear predictors over multiple layers(https://arxiv.org/abs/2502.03708)
• Keywords: language model, llm, hallucination
• Abstract: A trained Large Language Model (LLM) contains much of human knowledge. Yet, it is difficult to gauge the extent or accuracy of that knowledge, as LLMs do not always ``know what they know'' and may even be actively misleading. In this work, we give a general method for detecting semantic concepts in the internal activations of LLMs. Furthermore, we show that our methodology can be easily adapted to steer LLMs toward desirable outputs. Our innovations are the following: (1) we use a nonlinear feature learning method to identify important linear directions for predicting concepts from each layer; (2) we aggregate features across layers to build powerful concept detectors and steering mechanisms. We showcase the power of our approach by attaining state-of-the-art results for detecting hallucinations, harmfulness, toxicity, and untruthful content on seven benchmarks. We highlight the generality of our approach by steering LLMs towards new concepts that, to the best of our knowledge, have not been previously considered in the literature, including: semantic disambiguation, human languages, programming languages, hallucinated responses, science subjects, poetic/Shakespearean English, and even multiple concepts simultaneously. Moreover, our method can steer concepts with numerical attributes such as product reviews. We provide our code (including a simple API for our methods) at this https URL .
• 摘要：

Title: MultiQ&A: An Analysis in Measuring Robustness via Automated Crowdsourcing of Question Perturbations and Answers

• Authors: Nicole Cho, William Watson
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.03711
• Pdf URL: https://arxiv.org/pdf/2502.03711
• Copy Paste: [[2502.03711]] MultiQ&A: An Analysis in Measuring Robustness via Automated Crowdsourcing of Question Perturbations and Answers(https://arxiv.org/abs/2502.03711)
• Keywords: language model, gpt, llm, hallucination, agent
• Abstract: One critical challenge in the institutional adoption journey of Large Language Models (LLMs) stems from their propensity to hallucinate in generated responses. To address this, we propose MultiQ&A, a systematic approach for evaluating the robustness and consistency of LLM-generated answers. We demonstrate MultiQ&A's ability to crowdsource question perturbations and their respective answers through independent LLM agents at scale. Our experiments culminated in the examination of 1.9 million question perturbations and 2.3 million answers. Furthermore, MultiQ&A shows that ensembled LLMs, such as gpt-3.5-turbo, remain relatively robust and consistent under perturbations. MultiQ&A provides clarity in the response generation space, offering an effective method for inspecting disagreements and variability. Therefore, our system offers a potential framework for institutional LLM adoption with the ability to measure confidence, consistency, and the quantification of hallucinations.
• 摘要：

Title: Rethinking the Residual Distribution of Locate-then-Editing Methods in Model Editing

• Authors: Xiaopeng Li, Shanwen Wang, Shasha Li, Shezheng Song, Bin Ji, Jun Ma, Jie Yu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.03748
• Pdf URL: https://arxiv.org/pdf/2502.03748
• Copy Paste: [[2502.03748]] Rethinking the Residual Distribution of Locate-then-Editing Methods in Model Editing(https://arxiv.org/abs/2502.03748)
• Keywords: language model, llm
• Abstract: Model editing is a powerful technique for updating the knowledge of Large Language Models (LLMs). Locate-then-edit methods are a popular class of approaches that first identify the critical layers storing knowledge, then compute the residual of the last critical layer based on the edited knowledge, and finally perform multi-layer updates using a least-squares solution by evenly distributing the residual from the first critical layer to the last. Although these methods achieve promising results, they have been shown to degrade the original knowledge of LLMs. We argue that residual distribution leads to this issue. To explore this, we conduct a comprehensive analysis of residual distribution in locate-then-edit methods from both empirical and theoretical perspectives, revealing that residual distribution introduces editing errors, leading to inaccurate edits. To address this issue, we propose the Boundary Layer UpdatE (BLUE) strategy to enhance locate-then-edit methods. Sequential batch editing experiments on three LLMs and two datasets demonstrate that BLUE not only delivers an average performance improvement of 35.59\%, significantly advancing the state of the art in model editing, but also enhances the preservation of LLMs' general capabilities. Our code is available at this https URL.
• 摘要：

Title: Hierarchical Contextual Manifold Alignment for Structuring Latent Representations in Large Language Models

• Authors: Meiquan Dong, Haoran Liu, Yan Huang, Zixuan Feng, Jianhong Tang, Ruoxi Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.03766
• Pdf URL: https://arxiv.org/pdf/2502.03766
• Copy Paste: [[2502.03766]] Hierarchical Contextual Manifold Alignment for Structuring Latent Representations in Large Language Models(https://arxiv.org/abs/2502.03766)
• Keywords: language model
• Abstract: The organization of latent token representations plays a crucial role in determining the stability, generalization, and contextual consistency of language models, yet conventional approaches to embedding refinement often rely on parameter modifications that introduce additional computational overhead. A hierarchical alignment method was introduced to restructure token embeddings without altering core model weights, ensuring that representational distributions maintained coherence across different linguistic contexts. Experimental evaluations demonstrated improvements in rare token retrieval, adversarial robustness, and long-range dependency tracking, highlighting the advantages of hierarchical structuring in mitigating inconsistencies in latent space organization. The comparative analysis against conventional fine-tuning and embedding perturbation methods revealed that hierarchical restructuring maintained computational efficiency while achieving measurable gains in representation quality. Structural refinements introduced through the alignment process resulted in improved contextual stability across varied linguistic tasks, reducing inconsistencies in token proximity relationships and enhancing interpretability in language generation. A detailed computational assessment confirmed that the realignment process introduced minimal inference overhead, ensuring that representational improvements did not compromise model efficiency. The findings reinforced the broader significance of structured representation learning, illustrating that hierarchical embedding modifications could serve as an effective strategy for refining latent space distributions while preserving pre-learned semantic associations.
• 摘要：

Title: It's All in The [MASK]: Simple Instruction-Tuning Enables BERT-like Masked Language Models As Generative Classifiers

• Authors: Benjamin Clavié, Nathan Cooper, Benjamin Warner
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.03793
• Pdf URL: https://arxiv.org/pdf/2502.03793
• Copy Paste: [[2502.03793]] It's All in The [MASK]: Simple Instruction-Tuning Enables BERT-like Masked Language Models As Generative Classifiers(https://arxiv.org/abs/2502.03793)
• Keywords: language model, llm, prompt
• Abstract: While encoder-only models such as BERT and ModernBERT are ubiquitous in real-world NLP applications, their conventional reliance on task-specific classification heads can limit their applicability compared to decoder-based large language models (LLMs). In this work, we introduce ModernBERT-Large-Instruct, a 0.4B-parameter encoder model that leverages its masked language modelling (MLM) head for generative classification. Our approach employs an intentionally simple training loop and inference mechanism that requires no heavy pre-processing, heavily engineered prompting, or architectural modifications. ModernBERT-Large-Instruct exhibits strong zero-shot performance on both classification and knowledge-based tasks, outperforming similarly sized LLMs on MMLU and achieving 93% of Llama3-1B's MMLU performance with 60% less parameters. We also demonstrate that, when fine-tuned, the generative approach using the MLM head matches or even surpasses traditional classification-head methods across diverse NLU this http URL capability emerges specifically in models trained on contemporary, diverse data mixes, with models trained on lower volume, less-diverse data yielding considerably weaker performance. Although preliminary, these results demonstrate the potential of using the original generative masked language modelling head over traditional task-specific heads for downstream tasks. Our work suggests that further exploration into this area is warranted, highlighting many avenues for future improvements.
• 摘要：

Title: Enhancing Hallucination Detection through Noise Injection

• Authors: Litian Liu, Reza Pourreza, Sunny Panchal, Apratim Bhattacharyya, Yao Qin, Roland Memisevic
• Subjects: cs.CL, eess.SY
• Abstract URL: https://arxiv.org/abs/2502.03799
• Pdf URL: https://arxiv.org/pdf/2502.03799
• Copy Paste: [[2502.03799]] Enhancing Hallucination Detection through Noise Injection(https://arxiv.org/abs/2502.03799)
• Keywords: language model, llm, hallucination
• Abstract: Large Language Models (LLMs) are prone to generating plausible yet incorrect responses, known as hallucinations. Effectively detecting hallucinations is therefore crucial for the safe deployment of LLMs. Recent research has linked hallucinations to model uncertainty, suggesting that hallucinations can be detected by measuring dispersion over answer distributions obtained from a set of samples drawn from a model. While drawing from the distribution over tokens defined by the model is a natural way to obtain samples, in this work, we argue that it is sub-optimal for the purpose of detecting hallucinations. We show that detection can be improved significantly by taking into account model uncertainty in the Bayesian sense. To this end, we propose a very simple and efficient approach that perturbs an appropriate subset of model parameters, or equivalently hidden unit activations, during sampling. We demonstrate its effectiveness across a wide range of datasets and model architectures.
• 摘要：

Title: Identify Critical KV Cache in LLM Inference from an Output Perturbation Perspective

• Authors: Yuan Feng, Junlin Lv, Yukun Cao, Xike Xie, S Kevin Zhou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.03805
• Pdf URL: https://arxiv.org/pdf/2502.03805
• Copy Paste: [[2502.03805]] Identify Critical KV Cache in LLM Inference from an Output Perturbation Perspective(https://arxiv.org/abs/2502.03805)
• Keywords: language model, llm
• Abstract: Large language models have revolutionized natural language processing but face significant challenges of high storage and runtime costs, due to the transformer architecture's reliance on self-attention, particularly the large Key-Value (KV) cache for long-sequence inference. Recent efforts to reduce KV cache size by pruning less critical entries based on attention weights remain empirical and lack formal grounding. This paper presents a formal study on identifying critical KV cache entries by analyzing attention output perturbation. Our analysis reveals that, beyond attention weights, the value states within KV entries and pretrained parameter matrices are also crucial. Based on this, we propose a perturbation-constrained selection algorithm that optimizes the worst-case output perturbation to identify critical entries. Evaluations on the Needle-in-a-Haystack test and Longbench benchmark show our algorithm enhances state-of-the-art cache eviction methods. Further empirical analysis confirms that our algorithm achieves lower output perturbations in over 92% attention heads in Llama model, thereby providing a significant improvement over existing methods.
• 摘要：

Title: PsyPlay: Personality-Infused Role-Playing Conversational Agents

• Authors: Tao Yang, Yuhua Zhu, Xiaojun Quan, Cong Liu, Qifan Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.03821
• Pdf URL: https://arxiv.org/pdf/2502.03821
• Copy Paste: [[2502.03821]] PsyPlay: Personality-Infused Role-Playing Conversational Agents(https://arxiv.org/abs/2502.03821)
• Keywords: language model, gpt, llm, agent
• Abstract: The current research on Role-Playing Conversational Agents (RPCAs) with Large Language Models (LLMs) primarily focuses on imitating specific speaking styles and utilizing character backgrounds, neglecting the depiction of deeper personality traits.~In this study, we introduce personality-infused role-playing for LLM agents, which encourages agents to accurately portray their designated personality traits during dialogues. We then propose PsyPlay, a dialogue generation framework that facilitates the expression of rich personalities among multiple LLM agents. Specifically, PsyPlay enables agents to assume roles with distinct personality traits and engage in discussions centered around specific topics, consistently exhibiting their designated personality traits throughout the interactions. Validation on generated dialogue data demonstrates that PsyPlay can accurately portray the intended personality traits, achieving an overall success rate of 80.31% on GPT-3.5. Notably, we observe that LLMs aligned with positive values are more successful in portraying positive personality roles compared to negative ones. Moreover, we construct a dialogue corpus for personality-infused role-playing, called PsyPlay-Bench. The corpus, which consists of 4745 instances of correctly portrayed dialogues using PsyPlay, aims to further facilitate research in personalized role-playing and dialogue personality detection.
• 摘要：

Title: Syntriever: How to Train Your Retriever with Synthetic Data from LLMs

• Authors: Minsang Kim, Seungjun Baek
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.03824
• Pdf URL: https://arxiv.org/pdf/2502.03824
• Copy Paste: [[2502.03824]] Syntriever: How to Train Your Retriever with Synthetic Data from LLMs(https://arxiv.org/abs/2502.03824)
• Keywords: llm, hallucination, chain-of-thought
• Abstract: LLMs have boosted progress in many AI applications. Recently, there were attempts to distill the vast knowledge of LLMs into information retrieval systems. Those distillation methods mostly use output probabilities of LLMs which are unavailable in the latest black-box LLMs. We propose Syntriever, a training framework for retrievers using synthetic data from black-box LLMs. Syntriever consists of two stages. Firstly in the distillation stage, we synthesize relevant and plausibly irrelevant passages and augmented queries using chain-of-thoughts for the given queries. LLM is asked to self-verify the synthetic data for possible hallucinations, after which retrievers are trained with a loss designed to cluster the embeddings of relevant passages. Secondly in the alignment stage, we align the retriever with the preferences of LLMs. We propose a preference modeling called partial Plackett-Luce ranking to learn LLM preferences with regularization which prevents the model from deviating excessively from that trained in the distillation stage. Experiments show that Syntriever achieves state-of-the-art performances on benchmark datasets from various domains in nDCG@$K$. The code is available at \href{this https URL}{this https URL}.
• 摘要：

Title: Improving Natural Language Understanding for LLMs via Large-Scale Instruction Synthesis

• Authors: Lin Yuan, Jun Xu, Honghao Gui, Mengshu Sun, Zhiqiang Zhang, Lei Liang, Jun Zhou
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.03843
• Pdf URL: https://arxiv.org/pdf/2502.03843
• Copy Paste: [[2502.03843]] Improving Natural Language Understanding for LLMs via Large-Scale Instruction Synthesis(https://arxiv.org/abs/2502.03843)
• Keywords: language model, llm
• Abstract: High-quality, large-scale instructions are crucial for aligning large language models (LLMs), however, there is a severe shortage of instruction in the field of natural language understanding (NLU). Previous works on constructing NLU instructions mainly focus on information extraction (IE), neglecting tasks such as machine reading comprehension, question answering, and text classification. Furthermore, the lack of diversity in the data has led to a decreased generalization ability of trained LLMs in other NLU tasks and a noticeable decline in the fundamental model's general capabilities. To address this issue, we propose Hum, a large-scale, high-quality synthetic instruction corpus for NLU tasks, designed to enhance the NLU capabilities of LLMs. Specifically, Hum includes IE (either close IE or open IE), machine reading comprehension, text classification, and instruction generalist tasks, thereby enriching task diversity. Additionally, we introduce a human-LLMs collaborative mechanism to synthesize instructions, which enriches instruction diversity by incorporating guidelines, preference rules, and format variants. We conduct extensive experiments on 5 NLU tasks and 28 general capability evaluation datasets for LLMs. Experimental results show that Hum enhances the NLU capabilities of six LLMs by an average of 3.1\%, with no significant decline observed in other general capabilities.
• 摘要：

Title: BOLT: Bootstrap Long Chain-of-Thought in Language Models without Distillation

• Authors: Bo Pang, Hanze Dong, Jiacheng Xu, Silvio Savarese, Yingbo Zhou, Caiming Xiong
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.03860
• Pdf URL: https://arxiv.org/pdf/2502.03860
• Copy Paste: [[2502.03860]] BOLT: Bootstrap Long Chain-of-Thought in Language Models without Distillation(https://arxiv.org/abs/2502.03860)
• Keywords: language model, llm, chain-of-thought
• Abstract: Large language models (LLMs), such as o1 from OpenAI, have demonstrated remarkable reasoning capabilities. o1 generates a long chain-of-thought (LongCoT) before answering a question. LongCoT allows LLMs to analyze problems, devise plans, reflect, and backtrack effectively. These actions empower LLM to solve complex problems. After the release of o1, many teams have attempted to replicate its LongCoT and reasoning capabilities. In terms of methods, they primarily rely on knowledge distillation with data from existing models with LongCoT capacities (e.g., OpenAI-o1, Qwen-QwQ, DeepSeek-R1-Preview), leaving significant uncertainties on systematically developing such reasoning abilities. In terms of data domains, these works focus narrowly on math while a few others include coding, limiting their generalizability. This paper introduces a novel approach to enable LLM's LongCoT capacity without distillation from o1-like models or expensive human annotations, where we bootstrap LongCoT (BOLT) from a standard instruct model. BOLT involves three stages: 1) LongCoT data bootstrapping with in-context learning on a standard instruct model; 2) LongCoT supervised finetuning; 3) online training to further refine LongCoT capacities. In BOLT, only a few in-context examples need to be constructed during the bootstrapping stage; in our experiments, we created 10 examples, demonstrating the feasibility of this approach. We use Llama-3.1-70B-Instruct to bootstrap LongCoT and apply our method to various model scales (7B, 8B, 70B). We achieve impressive performance on a variety of benchmarks, Arena-Hard, MT-Bench, WildBench, ZebraLogic, MATH500, which evaluate diverse task-solving and reasoning capabilities.
• 摘要：

Title: Experiments with Large Language Models on Retrieval-Augmented Generation for Closed-Source Simulation Software

• Authors: Andreas Baumann, Peter Eberhard
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.03916
• Pdf URL: https://arxiv.org/pdf/2502.03916
• Copy Paste: [[2502.03916]] Experiments with Large Language Models on Retrieval-Augmented Generation for Closed-Source Simulation Software(https://arxiv.org/abs/2502.03916)
• Keywords: language model, llm, prompt, retrieval-augmented generation
• Abstract: Large Language Models (LLMs) are increasingly helpful in text generation, even writing code in programming languages based on user prompts written in natural language. They are even applied to generate simulation models for multibody systems from natural language. Research results suggest that LLMs surpass the mere replication of existing code examples, where some LLMs have been trained on an open-source multibody simulation code. However, for closed-source simulation software, such results are not to be expected as their ideas and concepts might differ from other publicly available ones. LLMs can hallucinate for knowledge-intensive tasks, such as model creation, which can lead to wrong responses. This is especially the case for the LLM unknown closed-source simulation software. The same applies to other internal knowledge kept private to protect intellectual property or data privacy. The Retrieval-Augmented Generation (RAG) approach might yield a solution for these knowledge-intensive tasks. This paper explores the application of RAG to closed-source simulation software and presents first experiments. After a brief introduction to LLMs, the RAG approach, and the simulation method applied by the close-source simulation software, several examples are provided to test LLMs' knowledge of the simulation software and the creation of simulation models using two RAG systems. The examples show promising results indicating the benefits of applying RAG systems to closed-source simulation software, helping to access their knowledge. Nevertheless, they also reveal gaps in the applied information and open questions for further research.
• 摘要：

Title: Afrispeech-Dialog: A Benchmark Dataset for Spontaneous English Conversations in Healthcare and Beyond

• Authors: Mardhiyah Sanni, Tassallah Abdullahi, Devendra D. Kayande, Emmanuel Ayodele, Naome A. Etori, Michael S. Mollel, Moshood Yekini, Chibuzor Okocha, Lukman E. Ismaila, Folafunmi Omofoye, Boluwatife A. Adewale, Tobi Olatunji
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.03945
• Pdf URL: https://arxiv.org/pdf/2502.03945
• Copy Paste: [[2502.03945]] Afrispeech-Dialog: A Benchmark Dataset for Spontaneous English Conversations in Healthcare and Beyond(https://arxiv.org/abs/2502.03945)
• Keywords: language model, llm
• Abstract: Speech technologies are transforming interactions across various sectors, from healthcare to call centers and robots, yet their performance on African-accented conversations remains underexplored. We introduce Afrispeech-Dialog, a benchmark dataset of 50 simulated medical and non-medical African-accented English conversations, designed to evaluate automatic speech recognition (ASR) and related technologies. We assess state-of-the-art (SOTA) speaker diarization and ASR systems on long-form, accented speech, comparing their performance with native accents and discover a 10%+ performance degradation. Additionally, we explore medical conversation summarization capabilities of large language models (LLMs) to demonstrate the impact of ASR errors on downstream medical summaries, providing insights into the challenges and opportunities for speech technologies in the Global South. Our work highlights the need for more inclusive datasets to advance conversational AI in low-resource settings.
• 摘要：

Title: MAQInstruct: Instruction-based Unified Event Relation Extraction

• Authors: Jun Xu, Mengshu Sun, Zhiqiang Zhang, Jun Zhou
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.03954
• Pdf URL: https://arxiv.org/pdf/2502.03954
• Copy Paste: [[2502.03954]] MAQInstruct: Instruction-based Unified Event Relation Extraction(https://arxiv.org/abs/2502.03954)
• Keywords: language model, llm
• Abstract: Extracting event relations that deviate from known schemas has proven challenging for previous methods based on multi-class classification, MASK prediction, or prototype matching. Recent advancements in large language models have shown impressive performance through instruction tuning. Nevertheless, in the task of event relation extraction, instruction-based methods face several challenges: there are a vast number of inference samples, and the relations between events are non-sequential. To tackle these challenges, we present an improved instruction-based event relation extraction framework named MAQInstruct. Firstly, we transform the task from extracting event relations using given event-event instructions to selecting events using given event-relation instructions, which reduces the number of samples required for inference. Then, by incorporating a bipartite matching loss, we reduce the dependency of the instruction-based method on the generation sequence. Our experimental results demonstrate that MAQInstruct significantly improves the performance of event relation extraction across multiple LLMs.
• 摘要：

Title: PGB: One-Shot Pruning for BERT via Weight Grouping and Permutation

• Authors: Hyemin Lim, Jaeyeon Lee, Dong-Wan Choi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.03984
• Pdf URL: https://arxiv.org/pdf/2502.03984
• Copy Paste: [[2502.03984]] PGB: One-Shot Pruning for BERT via Weight Grouping and Permutation(https://arxiv.org/abs/2502.03984)
• Keywords: language model
• Abstract: Large pretrained language models such as BERT suffer from slow inference and high memory usage, due to their huge size. Recent approaches to compressing BERT rely on iterative pruning and knowledge distillation, which, however, are often too complicated and computationally intensive. This paper proposes a novel semi-structured one-shot pruning method for BERT, called $\textit{Permutation and Grouping for BERT}$ (PGB), which achieves high compression efficiency and sparsity while preserving accuracy. To this end, PGB identifies important groups of individual weights by permutation and prunes all other weights as a structure in both multi-head attention and feed-forward layers. Furthermore, if no important group is formed in a particular layer, PGB drops the entire layer to produce an even more compact model. Our experimental results on BERT$_{\text{BASE}}$ demonstrate that PGB outperforms the state-of-the-art structured pruning methods in terms of computational cost and accuracy preservation.
• 摘要：

Title: Ontology-Guided, Hybrid Prompt Learning for Generalization in Knowledge Graph Question Answering

• Authors: Longquan Jiang, Junbo Huang, Cedric Möller, Ricardo Usbeck
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.03992
• Pdf URL: https://arxiv.org/pdf/2502.03992
• Copy Paste: [[2502.03992]] Ontology-Guided, Hybrid Prompt Learning for Generalization in Knowledge Graph Question Answering(https://arxiv.org/abs/2502.03992)
• Keywords: language model, llm, prompt
• Abstract: Most existing Knowledge Graph Question Answering (KGQA) approaches are designed for a specific KG, such as Wikidata, DBpedia or Freebase. Due to the heterogeneity of the underlying graph schema, topology and assertions, most KGQA systems cannot be transferred to unseen Knowledge Graphs (KGs) without resource-intensive training data. We present OntoSCPrompt, a novel Large Language Model (LLM)-based KGQA approach with a two-stage architecture that separates semantic parsing from KG-dependent interactions. OntoSCPrompt first generates a SPARQL query structure (including SPARQL keywords such as SELECT, ASK, WHERE and placeholders for missing tokens) and then fills them with KG-specific information. To enhance the understanding of the underlying KG, we present an ontology-guided, hybrid prompt learning strategy that integrates KG ontology into the learning process of hybrid prompts (e.g., discrete and continuous vectors). We also present several task-specific decoding strategies to ensure the correctness and executability of generated SPARQL queries in both stages. Experimental results demonstrate that OntoSCPrompt performs as well as SOTA approaches without retraining on a number of KGQA datasets such as CWQ, WebQSP and LC-QuAD 1.0 in a resource-efficient manner and can generalize well to unseen domain-specific KGs like DBLP-QuAD and CoyPu KG Code: \href{this https URL}{this https URL}
• 摘要：

Title: Quantification of Biodiversity from Historical Survey Text with LLM-based Best-Worst Scaling

• Authors: Thomas Haider, Tobias Perschl, Malte Rehbein
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.04022
• Pdf URL: https://arxiv.org/pdf/2502.04022
• Copy Paste: [[2502.04022]] Quantification of Biodiversity from Historical Survey Text with LLM-based Best-Worst Scaling(https://arxiv.org/abs/2502.04022)
• Keywords: language model, gpt, llm
• Abstract: In this study, we evaluate methods to determine the frequency of species via quantity estimation from historical survey text. To that end, we formulate classification tasks and finally show that this problem can be adequately framed as a regression task using Best-Worst Scaling (BWS) with Large Language Models (LLMs). We test Ministral-8B, DeepSeek-V3, and GPT-4, finding that the latter two have reasonable agreement with humans and each other. We conclude that this approach is more cost-effective and similarly robust compared to a fine-grained multi-class approach, allowing automated quantity estimation across species.
• 摘要：

Title: Exploring Imbalanced Annotations for Effective In-Context Learning

• Authors: Hongfu Gao, Feipeng Zhang, Hao Zeng, Deyu Meng, Bingyi Jing, Hongxin Wei
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.04037
• Pdf URL: https://arxiv.org/pdf/2502.04037
• Copy Paste: [[2502.04037]] Exploring Imbalanced Annotations for Effective In-Context Learning(https://arxiv.org/abs/2502.04037)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have shown impressive performance on downstream tasks through in-context learning (ICL), which heavily relies on the demonstrations selected from annotated datasets. Existing selection methods may hinge on the distribution of annotated datasets, which can often be long-tailed in real-world scenarios. In this work, we show that imbalanced class distributions in annotated datasets significantly degrade the performance of ICL across various tasks and selection methods. Moreover, traditional rebalance methods fail to ameliorate the issue of class imbalance in ICL. Our method is motivated by decomposing the distributional differences between annotated and test datasets into two-component weights: class-wise weights and conditional bias. The key idea behind our method is to estimate the conditional bias by minimizing the empirical error on a balanced validation dataset and to employ the two-component weights to modify the original scoring functions during selection. Our approach can prevent selecting too many demonstrations from a single class while preserving the effectiveness of the original selection methods. Extensive experiments demonstrate the effectiveness of our method, improving the average accuracy by up to 5.46 on common benchmarks with imbalanced datasets.
• 摘要：

Title: Simulating the Emergence of Differential Case Marking with Communicating Neural-Network Agents

• Authors: Yuchen Lian, Arianna Bisazza, Tessa Verhoef
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.04038
• Pdf URL: https://arxiv.org/pdf/2502.04038
• Copy Paste: [[2502.04038]] Simulating the Emergence of Differential Case Marking with Communicating Neural-Network Agents(https://arxiv.org/abs/2502.04038)
• Keywords: agent
• Abstract: Differential Case Marking (DCM) refers to the phenomenon where grammatical case marking is applied selectively based on semantic, pragmatic, or other factors. The emergence of DCM has been studied in artificial language learning experiments with human participants, which were specifically aimed at disentangling the effects of learning from those of communication (Smith & Culbertson, 2020). Multi-agent reinforcement learning frameworks based on neural networks have gained significant interest to simulate the emergence of human-like linguistic phenomena. In this study, we employ such a framework in which agents first acquire an artificial language before engaging in communicative interactions, enabling direct comparisons to human result. Using a very generic communication optimization algorithm and neural-network learners that have no prior experience with language or semantic preferences, our results demonstrate that learning alone does not lead to DCM, but when agents communicate, differential use of markers arises. This supports Smith and Culbertson (2020)'s findings that highlight the critical role of communication in shaping DCM and showcases the potential of neural-agent models to complement experimental research on language evolution.
• 摘要：

Title: Predicting Large Language Model Capabilities on Closed-Book QA Tasks Using Only Information Available Prior to Training

• Authors: Changhao Jiang, Ming Zhang, Junjie Ye, Xiaoran Fan, Yifei Cao, Jiajun Sun, Zhiheng Xi, Shihan Dou, Yi Dong, Yujiong Shen, Jingqi Tong, Zhen Wang, Tao Liang, Zhihui Fei, Mingyang Wan, Guojun Ma, Qi Zhang, Tao Gui, Xuanjing Huang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.04066
• Pdf URL: https://arxiv.org/pdf/2502.04066
• Copy Paste: [[2502.04066]] Predicting Large Language Model Capabilities on Closed-Book QA Tasks Using Only Information Available Prior to Training(https://arxiv.org/abs/2502.04066)
• Keywords: language model, gpt
• Abstract: The GPT-4 technical report from OpenAI suggests that model performance on specific tasks can be predicted prior to training, though methodologies remain unspecified. This approach is crucial for optimizing resource allocation and ensuring data alignment with target tasks. To achieve this vision, we focus on predicting performance on Closed-book Question Answering (CBQA) tasks, which are closely tied to pre-training data and knowledge retention. We address three major challenges: 1) mastering the entire pre-training process, especially data construction; 2) evaluating a model's knowledge retention; and 3) predicting task-specific knowledge retention using only information available prior to training. To tackle these challenges, we pre-train three large language models (i.e., 1.6B, 7B, and 13B) using 560k dollars and 520k GPU hours. We analyze the pre-training data with knowledge triples and assess knowledge retention using established methods. Additionally, we introduce the SMI metric, an information-theoretic measure that quantifies the relationship between pre-training data, model size, and task-specific knowledge retention. Our experiments reveal a strong linear correlation ($\text{R}^2 > 0.84$) between the SMI metric and the model's accuracy on CBQA tasks across models of varying sizes (i.e., 1.1B, 1.6B, 7B, and 13B). The dataset, model, and code are available at this https URL.
• 摘要：

Title: Controllable Emotion Generation with Emotion Vectors

• Authors: Yurui Dong, Luozhijie Jin, Yao Yang, Bingjie Lu, Jiaxi Yang, Zhi Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.04075
• Pdf URL: https://arxiv.org/pdf/2502.04075
• Copy Paste: [[2502.04075]] Controllable Emotion Generation with Emotion Vectors(https://arxiv.org/abs/2502.04075)
• Keywords: language model, llm
• Abstract: In recent years, technologies based on large-scale language models (LLMs) have made remarkable progress in many fields, especially in customer service, content creation, and embodied intelligence, showing broad application potential. However, The LLM's ability to express emotions with proper tone, timing, and in both direct and indirect forms is still insufficient but significant. Few works have studied on how to build the controlable emotional expression capability of LLMs. In this work, we propose a method for emotion expression output by LLMs, which is universal, highly flexible, and well controllable proved with the extensive experiments and verifications. This method has broad application prospects in fields involving emotions output by LLMs, such as intelligent customer service, literary creation, and home companion robots. The extensive experiments on various LLMs with different model-scales and architectures prove the versatility and the effectiveness of the proposed method.
• 摘要：

Title: AttentionPredictor: Temporal Pattern Matters for Efficient LLM Inference

• Authors: Qingyue Yang, Jie Wang, Xing Li, Zhihai Wang, Chen Chen, Lei Chen, Xianzhi Yu, Wulong Liu, Jianye Hao, Mingxuan Yuan, Bin Li
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.04077
• Pdf URL: https://arxiv.org/pdf/2502.04077
• Copy Paste: [[2502.04077]] AttentionPredictor: Temporal Pattern Matters for Efficient LLM Inference(https://arxiv.org/abs/2502.04077)
• Keywords: language model, llm
• Abstract: With the development of large language models (LLMs), efficient inference through Key-Value (KV) cache compression has attracted considerable attention, especially for long-context generation. To compress the KV cache, recent methods identify critical KV tokens through heuristic ranking with attention scores. However, these methods often struggle to accurately determine critical tokens as they neglect the \textit{temporal patterns} in attention scores, resulting in a noticeable degradation in LLM performance. To address this challenge, we propose AttentionPredictor, which is the first learning-based critical token identification approach. Specifically, AttentionPredictor learns a lightweight convolution model to capture spatiotemporal patterns and predict the next-token attention score. An appealing feature of AttentionPredictor is that it accurately predicts the attention score while consuming negligible memory. Moreover, we propose a cross-token critical cache prefetching framework that hides the token estimation time overhead to accelerate the decoding stage. By retaining most of the attention information, AttentionPredictor achieves 16$\times$ KV cache compression with comparable LLM performance, significantly outperforming the state-of-the-art.
• 摘要：

Title: LLMs to Support a Domain Specific Knowledge Assistant

• Authors: Maria-Flavia Lovin
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.04095
• Pdf URL: https://arxiv.org/pdf/2502.04095
• Copy Paste: [[2502.04095]] LLMs to Support a Domain Specific Knowledge Assistant(https://arxiv.org/abs/2502.04095)
• Keywords: language model, llm, prompt, chat, chain-of-thought
• Abstract: This work presents a custom approach to developing a domain specific knowledge assistant for sustainability reporting using the International Financial Reporting Standards (IFRS). In this domain, there is no publicly available question-answer dataset, which has impeded the development of a high-quality chatbot to support companies with IFRS reporting. The two key contributions of this project therefore are: (1) A high-quality synthetic question-answer (QA) dataset based on IFRS sustainability standards, created using a novel generation and evaluation pipeline leveraging Large Language Models (LLMs). This comprises 1,063 diverse QA pairs that address a wide spectrum of potential user queries in sustainability reporting. Various LLM-based techniques are employed to create the dataset, including chain-of-thought reasoning and few-shot prompting. A custom evaluation framework is developed to assess question and answer quality across multiple dimensions, including faithfulness, relevance, and domain specificity. The dataset averages a score range of 8.16 out of 10 on these metrics. (2) Two architectures for question-answering in the sustainability reporting domain - a RAG pipeline and a fully LLM-based pipeline. The architectures are developed by experimenting, fine-tuning, and training on the QA dataset. The final pipelines feature an LLM fine-tuned on domain specific data and an industry classification component to improve the handling of complex queries. The RAG architecture achieves an accuracy of 85.32% on single-industry and 72.15% on cross-industry multiple-choice questions, outperforming the baseline approach by 4.67 and 19.21 percentage points, respectively. The LLM-based pipeline achieves an accuracy of 93.45% on single-industry and 80.30% on cross-industry multiple-choice questions, an improvement of 12.80 and 27.36 percentage points over the baseline, respectively.
• 摘要：

Title: The Order Effect: Investigating Prompt Sensitivity in Closed-Source LLMs

• Authors: Bryan Guan, Tanya Roosta, Peyman Passban, Mehdi Rezagholizadeh
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.04134
• Pdf URL: https://arxiv.org/pdf/2502.04134
• Copy Paste: [[2502.04134]] The Order Effect: Investigating Prompt Sensitivity in Closed-Source LLMs(https://arxiv.org/abs/2502.04134)
• Keywords: language model, llm, prompt
• Abstract: As large language models (LLMs) become integral to diverse applications, ensuring their reliability under varying input conditions is crucial. One key issue affecting this reliability is order sensitivity, wherein slight variations in input arrangement can lead to inconsistent or biased outputs. Although recent advances have reduced this sensitivity, the problem remains unresolved. This paper investigates the extent of order sensitivity in closed-source LLMs by conducting experiments across multiple tasks, including paraphrasing, relevance judgment, and multiple-choice questions. Our results show that input order significantly affects performance across tasks, with shuffled inputs leading to measurable declines in output accuracy. Few-shot prompting demonstrates mixed effectiveness and offers partial mitigation, however, fails to fully resolve the problem. These findings highlight persistent risks, particularly in high-stakes applications, and point to the need for more robust LLMs or improved input-handling techniques in future development.
• 摘要：

Title: UltraIF: Advancing Instruction Following from the Wild

• Authors: Kaikai An, Li Sheng, Ganqu Cui, Shuzheng Si, Ning Ding, Yu Cheng, Baobao Chang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.04153
• Pdf URL: https://arxiv.org/pdf/2502.04153
• Copy Paste: [[2502.04153]] UltraIF: Advancing Instruction Following from the Wild(https://arxiv.org/abs/2502.04153)
• Keywords: language model, llm, prompt
• Abstract: Instruction-following made modern large language models (LLMs) helpful assistants. However, the key to taming LLMs on complex instructions remains mysterious, for that there are huge gaps between models trained by open-source community and those trained by leading companies. To bridge the gap, we propose a simple and scalable approach UltraIF for building LLMs that can follow complex instructions with open-source data. UltraIF first decomposes real-world user prompts into simpler queries, constraints, and corresponding evaluation questions for the constraints. Then, we train an UltraComposer to compose constraint-associated prompts with evaluation questions. This prompt composer allows us to synthesize complicated instructions as well as filter responses with evaluation questions. In our experiment, for the first time, we successfully align LLaMA-3.1-8B-Base to catch up with its instruct version on 5 instruction-following benchmarks without any benchmark information, using only 8B model as response generator and evaluator. The aligned model also achieved competitive scores on other benchmarks. Moreover, we also show that UltraIF could further improve LLaMA-3.1-8B-Instruct through self-alignment, motivating broader use cases for the method. Our code will be available at this https URL.
• 摘要：

Title: Lexical Substitution is not Synonym Substitution: On the Importance of Producing Contextually Relevant Word Substitutes

• Authors: Juraj Vladika, Stephen Meisenbacher, Florian Matthes
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.04173
• Pdf URL: https://arxiv.org/pdf/2502.04173
• Copy Paste: [[2502.04173]] Lexical Substitution is not Synonym Substitution: On the Importance of Producing Contextually Relevant Word Substitutes(https://arxiv.org/abs/2502.04173)
• Keywords: language model
• Abstract: Lexical Substitution is the task of replacing a single word in a sentence with a similar one. This should ideally be one that is not necessarily only synonymous, but also fits well into the surrounding context of the target word, while preserving the sentence's grammatical structure. Recent advances in Lexical Substitution have leveraged the masked token prediction task of Pre-trained Language Models to generate replacements for a given word in a sentence. With this technique, we introduce ConCat, a simple augmented approach which utilizes the original sentence to bolster contextual information sent to the model. Compared to existing approaches, it proves to be very effective in guiding the model to make contextually relevant predictions for the target word. Our study includes a quantitative evaluation, measured via sentence similarity and task performance. In addition, we conduct a qualitative human analysis to validate that users prefer the substitutions proposed by our method, as opposed to previous methods. Finally, we test our approach on the prevailing benchmark for Lexical Substitution, CoInCo, revealing potential pitfalls of the benchmark. These insights serve as the foundation for a critical discussion on the way in which Lexical Substitution is evaluated.
• 摘要：

Title: The Best Instruction-Tuning Data are Those That Fit

• Authors: Dylan Zhang, Qirun Dai, Hao Peng
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.04194
• Pdf URL: https://arxiv.org/pdf/2502.04194
• Copy Paste: [[2502.04194]] The Best Instruction-Tuning Data are Those That Fit(https://arxiv.org/abs/2502.04194)
• Keywords: language model, llm
• Abstract: High-quality supervised fine-tuning (SFT) data are crucial for eliciting strong capabilities from pretrained large language models (LLMs). Typically, instructions are paired with multiple responses sampled from other LLMs, which are often out of the distribution of the target model to be fine-tuned. This, at scale, can lead to diminishing returns and even hurt the models' performance and robustness. We propose **GRAPE**, a novel SFT framework that accounts for the unique characteristics of the target model. For each instruction, it gathers responses from various LLMs and selects the one with the highest probability measured by the target model, indicating that it aligns most closely with the target model's pretrained distribution; it then proceeds with standard SFT training. We first evaluate GRAPE with a controlled experiment, where we sample various solutions for each question in UltraInteract from multiple models and fine-tune commonly used LMs like LLaMA3.1-8B, Mistral-7B, and Qwen2.5-7B on GRAPE-selected data. GRAPE significantly outperforms strong baselines, including distilling from the strongest model with an absolute gain of up to 13.8%, averaged across benchmarks, and training on 3x more data with a maximum performance improvement of 17.3%. GRAPE's strong performance generalizes to realistic settings. We experiment with the post-training data used for Tulu3 and Olmo-2. GRAPE outperforms strong baselines trained on 4.5 times more data by 6.1% and a state-of-the-art data selection approach by 3% on average performance. Remarkably, using 1/3 of the data and half the number of epochs, GRAPE enables LLaMA3.1-8B to surpass the performance of Tulu3-SFT by 3.5%.
• 摘要：

Title: Sports and Women's Sports: Gender Bias in Text Generation with Olympic Data

• Authors: Laura Biester
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.04218
• Pdf URL: https://arxiv.org/pdf/2502.04218
• Copy Paste: [[2502.04218]] Sports and Women's Sports: Gender Bias in Text Generation with Olympic Data(https://arxiv.org/abs/2502.04218)
• Keywords: language model, llm, prompt
• Abstract: Large Language Models (LLMs) have been shown to be biased in prior work, as they generate text that is in line with stereotypical views of the world or that is not representative of the viewpoints and values of historically marginalized demographic groups. In this work, we propose using data from parallel men's and women's events at the Olympic Games to investigate different forms of gender bias in language models. We define three metrics to measure bias, and find that models are consistently biased against women when the gender is ambiguous in the prompt. In this case, the model frequently retrieves only the results of the men's event with or without acknowledging them as such, revealing pervasive gender bias in LLMs in the context of athletics.
• 摘要：

Title: MAGA: MAssive Genre-Audience Reformulation to Pretraining Corpus Expansion

• Authors: Xintong Hao, Ke Shen, Chenggang Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.04235
• Pdf URL: https://arxiv.org/pdf/2502.04235
• Copy Paste: [[2502.04235]] MAGA: MAssive Genre-Audience Reformulation to Pretraining Corpus Expansion(https://arxiv.org/abs/2502.04235)
• Keywords: language model, prompt
• Abstract: Despite the remarkable capabilities of large language models across various tasks, their continued scaling faces a critical challenge: the scarcity of high-quality pretraining data. While model architectures continue to evolve, the natural language data struggles to scale up. To tackle this bottleneck, we propose \textbf{MA}ssive \textbf{G}enre-\textbf{A}udience~(MAGA) reformulation method, which systematic synthesizes diverse, contextually-rich pretraining data from existing corpus. This work makes three main contributions: (1) We propose MAGA reformulation method, a lightweight and scalable approach for pretraining corpus expansion, and build a 770B tokens MAGACorpus. (2) We evaluate MAGACorpus with different data budget scaling strategies, demonstrating consistent improvements across various model sizes (134M-13B), establishing the necessity for next-generation large-scale synthetic pretraining language models. (3) Through comprehensive analysis, we investigate prompt engineering's impact on synthetic training collapse and reveal limitations in conventional collapse detection metrics using validation losses. Our work shows that MAGA can substantially expand training datasets while maintaining quality, offering a reliably pathway for scaling models beyond data limitations.
• 摘要：

Title: How does a Multilingual LM Handle Multiple Languages?

• Authors: Santhosh Kakarla, Gautama Shastry Bulusu Venkata, Aishwarya Gaddam
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.04269
• Pdf URL: https://arxiv.org/pdf/2502.04269
• Copy Paste: [[2502.04269]] How does a Multilingual LM Handle Multiple Languages?(https://arxiv.org/abs/2502.04269)
• Keywords: language model
• Abstract: Multilingual language models have significantly advanced due to rapid progress in natural language processing. Models like BLOOM 1.7B, trained on diverse multilingual datasets, aim to bridge linguistic gaps. However, their effectiveness in capturing linguistic knowledge, particularly for low-resource languages, remains an open question. This study critically examines MLMs capabilities in multilingual understanding, semantic representation, and cross-lingual knowledge transfer. While these models perform well for high-resource languages, they struggle with less-represented ones. Additionally, traditional evaluation methods often overlook their internal syntactic and semantic encoding. This research addresses key limitations through three objectives. First, it assesses semantic similarity by analyzing multilingual word embeddings for consistency using cosine similarity. Second, it examines BLOOM-1.7B and Qwen2 through Named Entity Recognition and sentence similarity tasks to understand their linguistic structures. Third, it explores cross-lingual knowledge transfer by evaluating generalization from high-resource to low-resource languages in sentiment analysis and text classification. By leveraging linguistic probing, performance metrics, and visualizations, this study provides insights into the strengths and limitations of MLMs. The findings aim to enhance multilingual NLP models, ensuring better support for both high- and low-resource languages, thereby promoting inclusivity in language technologies.
• 摘要：

Title: Beyond Prompt Content: Enhancing LLM Performance via Content-Format Integrated Prompt Optimization

• Authors: Yuanye Liu, Jiahang Xu, Li Lyna Zhang, Qi Chen, Xuan Feng, Yang Chen, Zhongxin Guo, Yuqing Yang, Cheng Peng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.04295
• Pdf URL: https://arxiv.org/pdf/2502.04295
• Copy Paste: [[2502.04295]] Beyond Prompt Content: Enhancing LLM Performance via Content-Format Integrated Prompt Optimization(https://arxiv.org/abs/2502.04295)
• Keywords: language model, llm, prompt
• Abstract: Large Language Models (LLMs) have shown significant capability across various tasks, with their real-world effectiveness often driven by prompt design. While recent research has focused on optimizing prompt content, the role of prompt formatting, a critical but often overlooked dimension, has received limited systematic investigation. In this paper, we introduce Content-Format Integrated Prompt Optimization (CFPO), an innovative methodology that jointly optimizes both prompt content and formatting through an iterative refinement process. CFPO leverages natural language mutations to explore content variations and employs a dynamic format exploration strategy that systematically evaluates diverse format options. Our extensive evaluations across multiple tasks and open-source LLMs demonstrate that CFPO demonstrates measurable performance improvements compared to content-only optimization methods. This highlights the importance of integrated content-format optimization and offers a practical, model-agnostic approach to enhancing LLM performance. Code will be available at this https URL.
• 摘要：

Title: ScoreFlow: Mastering LLM Agent Workflows via Score-based Preference Optimization

• Authors: Yinjie Wang, Ling Yang, Guohao Li, Mengdi Wang, Bryon Aragam
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2502.04306
• Pdf URL: https://arxiv.org/pdf/2502.04306
• Copy Paste: [[2502.04306]] ScoreFlow: Mastering LLM Agent Workflows via Score-based Preference Optimization(https://arxiv.org/abs/2502.04306)
• Keywords: language model, llm, agent
• Abstract: Recent research has leveraged large language model multi-agent systems for complex problem-solving while trying to reduce the manual effort required to build them, driving the development of automated agent workflow optimization methods. However, existing methods remain inflexible due to representational limitations, a lack of adaptability, and poor scalability when relying on discrete optimization techniques. We address these challenges with ScoreFlow, a simple yet high-performance framework that leverages efficient gradient-based optimization in a continuous space. ScoreFlow incorporates Score-DPO, a novel variant of the direct preference optimization method that accounts for quantitative feedback. Across six benchmarks spanning question answering, coding, and mathematical reasoning, ScoreFlow achieves an 8.2% improvement over existing baselines. Moreover, it empowers smaller models to outperform larger ones with lower inference costs. Project: this https URL
• 摘要：

Title: ChamaleonLLM: Batch-Aware Dynamic Low-Rank Adaptation via Inference-Time Clusters

• Authors: Kamer Ali Yuksel, Hassan Sawaf
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.04315
• Pdf URL: https://arxiv.org/pdf/2502.04315
• Copy Paste: [[2502.04315]] ChamaleonLLM: Batch-Aware Dynamic Low-Rank Adaptation via Inference-Time Clusters(https://arxiv.org/abs/2502.04315)
• Keywords: language model, llm
• Abstract: Recent advances in large language models (LLMs) have shown remarkable performance across diverse tasks. However, these models are typically deployed with fixed weights, which limits their ability to adapt dynamically to the variability inherent in real-world data during inference. This paper introduces ChamaleonLLM, a novel framework that enables inference-time adaptation of LLMs by leveraging batch-aware clustering and on-the-fly generation of low-rank updates. Unlike traditional fine-tuning approaches such as Low-Rank Adaptation (LoRA) or methods that rely on a fixed set of pre-learned uniforms (changeable masks), our method dynamically generates adaptive modifications to the decoder weights based on the aggregated statistics of clustered batches. By intelligently grouping similar inputs and computing context-aware low-rank updates via a hyper-network, ChamaleonLLM achieves significant performance gains, outperforming conventional LoRA methods while eliminating the overhead of maintaining multiple expert models. Our experiments highlight the potential of our approach to serve as a versatile and highly adaptive solution for language model inference. ChamaleonLLM is open-sourced to ensure the reproducibility of our experiments: this https URL
• 摘要：

Title: Can Grammarly and ChatGPT accelerate language change? AI-powered technologies and their impact on the English language: wordiness vs. conciseness

• Authors: Karolina Rudnicka
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2502.04324
• Pdf URL: https://arxiv.org/pdf/2502.04324
• Copy Paste: [[2502.04324]] Can Grammarly and ChatGPT accelerate language change? AI-powered technologies and their impact on the English language: wordiness vs. conciseness(https://arxiv.org/abs/2502.04324)
• Keywords: gpt, chat
• Abstract: The proliferation of NLP-powered language technologies, AI-based natural language generation models, and English as a mainstream means of communication among both native and non-native speakers make the output of AI-powered tools especially intriguing to linguists. This paper investigates how Grammarly and ChatGPT affect the English language regarding wordiness vs. conciseness. A case study focusing on the purpose subordinator in order to is presented to illustrate the way in which Grammarly and ChatGPT recommend shorter grammatical structures instead of longer and more elaborate ones. Although the analysed sentences were produced by native speakers, are perfectly correct, and were extracted from a language corpus of contemporary English, both Grammarly and ChatGPT suggest more conciseness and less verbosity, even for relatively short sentences. The present article argues that technologies such as Grammarly not only mirror language change but also have the potential to facilitate or accelerate it.
• 摘要：