2024-11-21

Title: An exploration of the effect of quantisation on energy consumption and inference time of StarCoder2

• Authors: Pepijn de Reus, Ana Oprescu, Jelle Zuidema
• Subjects: cs.CL, cs.AI, cs.SE
• Abstract URL: https://arxiv.org/abs/2411.12758
• Pdf URL: https://arxiv.org/pdf/2411.12758
• Copy Paste: [[2411.12758]] An exploration of the effect of quantisation on energy consumption and inference time of StarCoder2(https://arxiv.org/abs/2411.12758)
• Keywords: language model, llm
• Abstract: This study examines quantisation and pruning strategies to reduce energy consumption in code Large Language Models (LLMs) inference. Using StarCoder2, we observe increased energy demands with quantization due to lower throughput and some accuracy losses. Conversely, pruning reduces energy usage but impairs performance. The results highlight challenges and trade-offs in LLM model compression. We suggest future work on hardware-optimized quantization to enhance efficiency with minimal loss in accuracy.
• 摘要：本研究考察了量化和修剪策略，以减少代码大型语言模型 (LLM) 推理中的能耗。使用 StarCoder2，我们观察到由于吞吐量较低和准确性有所损失，量化带来的能耗需求增加。相反，修剪会降低能耗，但会损害性能。结果突出了 LLM 模型压缩中的挑战和权衡。我们建议未来研究硬件优化量化，以提高效率，同时尽量减少准确性损失。

Title: A Novel Approach to Eliminating Hallucinations in Large Language Model-Assisted Causal Discovery

• Authors: Grace Sng, Yanming Zhang, Klaus Mueller
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.12759
• Pdf URL: https://arxiv.org/pdf/2411.12759
• Copy Paste: [[2411.12759]] A Novel Approach to Eliminating Hallucinations in Large Language Model-Assisted Causal Discovery(https://arxiv.org/abs/2411.12759)
• Keywords: language model, llm, hallucination, retrieval augmented generation
• Abstract: The increasing use of large language models (LLMs) in causal discovery as a substitute for human domain experts highlights the need for optimal model selection. This paper presents the first hallucination survey of popular LLMs for causal discovery. We show that hallucinations exist when using LLMs in causal discovery so the choice of LLM is important. We propose using Retrieval Augmented Generation (RAG) to reduce hallucinations when quality data is available. Additionally, we introduce a novel method employing multiple LLMs with an arbiter in a debate to audit edges in causal graphs, achieving a comparable reduction in hallucinations to RAG.
• 摘要：在因果发现中，大型语言模型 (LLM) 越来越多地被用来替代人类领域专家，这凸显了对最佳模型选择的需求。本文首次对因果发现中流行的 LLM 进行了幻觉调查。我们表明，在因果发现中使用 LLM 时会出现幻觉，因此 LLM 的选择很重要。我们建议在有高质量数据可用时使用检索增强生成 (RAG) 来减少幻觉。此外，我们介绍了一种新方法，在辩论中使用多个 LLM 和一个仲裁者来审核因果图中的边，从而实现与 RAG 相当的幻觉减少。

Title: Playing Language Game with LLMs Leads to Jailbreaking

• Authors: Yu Peng, Zewen Long, Fangming Dong, Congyi Li, Shu Wu, Kai Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.12762
• Pdf URL: https://arxiv.org/pdf/2411.12762
• Copy Paste: [[2411.12762]] Playing Language Game with LLMs Leads to Jailbreaking(https://arxiv.org/abs/2411.12762)
• Keywords: language model, gpt, llm
• Abstract: The advent of large language models (LLMs) has spurred the development of numerous jailbreak techniques aimed at circumventing their security defenses against malicious attacks. An effective jailbreak approach is to identify a domain where safety generalization fails, a phenomenon known as mismatched generalization. In this paper, we introduce two novel jailbreak methods based on mismatched generalization: natural language games and custom language games, both of which effectively bypass the safety mechanisms of LLMs, with various kinds and different variants, making them hard to defend and leading to high attack rates. Natural language games involve the use of synthetic linguistic constructs and the actions intertwined with these constructs, such as the Ubbi Dubbi language. Building on this phenomenon, we propose the custom language games method: by engaging with LLMs using a variety of custom rules, we successfully execute jailbreak attacks across multiple LLM platforms. Extensive experiments demonstrate the effectiveness of our methods, achieving success rates of 93% on GPT-4o, 89% on GPT-4o-mini and 83% on Claude-3.5-Sonnet. Furthermore, to investigate the generalizability of safety alignments, we fine-tuned Llama-3.1-70B with the custom language games to achieve safety alignment within our datasets and found that when interacting through other language games, the fine-tuned models still failed to identify harmful content. This finding indicates that the safety alignment knowledge embedded in LLMs fails to generalize across different linguistic formats, thus opening new avenues for future research in this area.
• 摘要：大型语言模型 (LLM) 的出现推动了众多越狱技术的发展，这些技术旨在绕过其针对恶意攻击的安全防御。一种有效的越狱方法是确定安全泛化失败的领域，这种现象称为不匹配泛化。在本文中，我们介绍了两种基于不匹配泛化的新型越狱方法：自然语言游戏和自定义语言游戏，这两种方法都能有效绕过 LLM 的安全机制，种类繁多，变体各异，使其难以防御并导致高攻击率。自然语言游戏涉及使用合成语言结构以及与这些结构交织在一起的动作，例如 Ubbi Dubbi 语言。基于这一现象，我们提出了自定义语言游戏方法：通过使用各种自定义规则与 LLM 互动，我们成功地在多个 LLM 平台上执行越狱攻击。大量实验证明了我们方法的有效性，在 GPT-4o 上的成功率为 93%，在 GPT-4o-mini 上的成功率为 89%，在 Claude-3.5-Sonnet 上的成功率为 83%。此外，为了研究安全对齐的可推广性，我们使用自定义语言游戏对 Llama-3.1-70B 进行了微调，以在我们的数据集中实现安全对齐，并发现当通过其他语言游戏进行交互时，微调后的模型仍然无法识别有害内容。这一发现表明，LLM 中嵌入的安全对齐知识无法推广到不同的语言格式，从而为该领域的未来研究开辟了新的途径。

Title: SEFD: Semantic-Enhanced Framework for Detecting LLM-Generated Text

• Authors: Weiqing He, Bojian Hou, Tianqi Shang, Davoud Ataee Tarzanagh, Qi Long, Li Shen
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2411.12764
• Pdf URL: https://arxiv.org/pdf/2411.12764
• Copy Paste: [[2411.12764]] SEFD: Semantic-Enhanced Framework for Detecting LLM-Generated Text(https://arxiv.org/abs/2411.12764)
• Keywords: language model, llm
• Abstract: The widespread adoption of large language models (LLMs) has created an urgent need for robust tools to detect LLM-generated text, especially in light of \textit{paraphrasing} techniques that often evade existing detection methods. To address this challenge, we present a novel semantic-enhanced framework for detecting LLM-generated text (SEFD) that leverages a retrieval-based mechanism to fully utilize text semantics. Our framework improves upon existing detection methods by systematically integrating retrieval-based techniques with traditional detectors, employing a carefully curated retrieval mechanism that strikes a balance between comprehensive coverage and computational efficiency. We showcase the effectiveness of our approach in sequential text scenarios common in real-world applications, such as online forums and Q\&A platforms. Through comprehensive experiments across various LLM-generated texts and detection methods, we demonstrate that our framework substantially enhances detection accuracy in paraphrasing scenarios while maintaining robustness for standard LLM-generated content.
• 摘要：大型语言模型 (LLM) 的广泛采用迫切需要强大的工具来检测 LLM 生成的文本，尤其是考虑到 \textit{释义} 技术通常会逃避现有的检测方法。为了应对这一挑战，我们提出了一种用于检测 LLM 生成的文本 (SEFD) 的新型语义增强框架，该框架利用基于检索的机制充分利用文本语义。我们的框架通过系统地将基于检索的技术与传统检测器相结合，改进了现有的检测方法，采用了精心策划的检索机制，在全面覆盖和计算效率之间取得了平衡。我们在现实世界应用中常见的顺序文本场景中展示了我们的方法的有效性，例如在线论坛和问答平台。通过对各种 LLM 生成的文本和检测方法进行全面的实验，我们证明了我们的框架在保持标准 LLM 生成内容的稳健性的同时，大大提高了释义场景中的检测准确性。

Title: CROW: Eliminating Backdoors from Large Language Models via Internal Consistency Regularization

• Authors: Nay Myat Min, Long H. Pham, Yige Li, Jun Sun
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.12768
• Pdf URL: https://arxiv.org/pdf/2411.12768
• Copy Paste: [[2411.12768]] CROW: Eliminating Backdoors from Large Language Models via Internal Consistency Regularization(https://arxiv.org/abs/2411.12768)
• Keywords: language model, llm
• Abstract: Recent studies reveal that Large Language Models (LLMs) are susceptible to backdoor attacks, where adversaries embed hidden triggers that manipulate model responses. Existing backdoor defense methods are primarily designed for vision or classification tasks, and are thus ineffective for text generation tasks, leaving LLMs vulnerable. We introduce Internal Consistency Regularization (CROW), a novel defense using consistency regularization finetuning to address layer-wise inconsistencies caused by backdoor triggers. CROW leverages the intuition that clean models exhibit smooth, consistent transitions in hidden representations across layers, whereas backdoored models show noticeable fluctuation when triggered. By enforcing internal consistency through adversarial perturbations and regularization, CROW neutralizes backdoor effects without requiring clean reference models or prior trigger knowledge, relying only on a small set of clean data. This makes it practical for deployment across various LLM architectures. Experimental results demonstrate that CROW consistently achieves a significant reductions in attack success rates across diverse backdoor strategies and tasks, including negative sentiment, targeted refusal, and code injection, on models such as Llama-2 (7B, 13B), CodeLlama (7B, 13B) and Mistral-7B, while preserving the model's generative capabilities.
• 摘要：最近的研究表明，大型语言模型 (LLM) 容易受到后门攻击，攻击者会嵌入隐藏的触发器来操纵模型响应。现有的后门防御方法主要针对视觉或分类任务而设计，因此对文本生成任务无效，从而使 LLM 容易受到攻击。我们引入了内部一致性正则化 (CROW)，这是一种使用一致性正则化微调来解决后门触发器导致的逐层不一致问题的新防御方法。CROW 利用了以下直觉：干净的模型在各层隐藏表示中表现出平滑、一致的过渡，而后门模型在触发时会出现明显的波动。通过对抗性扰动和正则化来强制内部一致性，CROW 可以消除后门效应，而无需干净的参考模型或先前的触发器知识，仅依赖一小组干净的数据。这使得它可以在各种 LLM 架构中部署。实验结果表明，在 Llama-2 (7B, 13B)、CodeLlama (7B, 13B) 和 Mistral-7B 等模型上，CROW 能够持续显著降低各种后门策略和任务（包括负面情绪、有针对性的拒绝和代码注入）的攻击成功率，同时保留模型的生成能力。

Title: Probing the Capacity of Language Model Agents to Operationalize Disparate Experiential Context Despite Distraction

• Authors: Sonny George, Chris Sypherd, Dylan Cashman
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.12828
• Pdf URL: https://arxiv.org/pdf/2411.12828
• Copy Paste: [[2411.12828]] Probing the Capacity of Language Model Agents to Operationalize Disparate Experiential Context Despite Distraction(https://arxiv.org/abs/2411.12828)
• Keywords: language model, gpt, llm, prompt, chain-of-thought, agent
• Abstract: Large language model (LLM) agents show promise in an increasing number of domains. In many proposed applications, it is expected that the agent reasons over accumulated experience presented in an input prompt. We propose the OEDD (Operationalize Experience Despite Distraction) corpus, a human-annotator-validated body of scenarios with pre-scripted agent histories where the agent must make a decision based on disparate experiential information in the presence of a distractor. We evaluate three state-of-the-art LLMs (GPT-3.5 Turbo, GPT-4o, and Gemini 1.5 Pro) using a minimal chain-of-thought prompting strategy and observe that when (1) the input context contains over 1,615 tokens of historical interactions, (2) a crucially decision-informing premise is the rightful conclusion over two disparate environment premises, and (3) a trivial, but distracting red herring fact follows, all LLMs perform worse than random choice at selecting the better of two actions. Our code and test corpus are publicly available at: this https URL .
• 摘要：大型语言模型 (LLM) 代理在越来越多的领域中显示出良好的前景。在许多提议的应用中，代理需要对输入提示中呈现的累积经验进行推理。我们提出了 OEDD（Operationalize Experience despite Distraction，在干扰下操作经验）语料库，这是一组经过人工注释验证的场景，具有预先编写的代理历史，其中代理必须在存在干扰项的情况下根据不同的经验信息做出决策。我们使用最小的思路链提示策略评估了三种最先进的 LLM（GPT-3.5 Turbo、GPT-4o 和 Gemini 1.5 Pro），并观察到当 (1) 输入上下文包含超过 1,615 个历史交互标记、(2) 一个至关重要的决策前提是两个不同环境前提的正确结论、(3) 一个微不足道但分散注意力的干扰事实随之而来时，所有 LLM 在选择两个动作中更好的一个方面的表现都比随机选择差。我们的代码和测试语料库可公开获取：此 https URL 。

Title: Signformer is all you need: Towards Edge AI for Sign Language

• Authors: Eta Yang
• Subjects: cs.CL, cs.CV, cs.CY, cs.HC, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.12901
• Pdf URL: https://arxiv.org/pdf/2411.12901
• Copy Paste: [[2411.12901]] Signformer is all you need: Towards Edge AI for Sign Language(https://arxiv.org/abs/2411.12901)
• Keywords: language model, llm
• Abstract: Sign language translation, especially in gloss-free paradigm, is confronting a dilemma of impracticality and unsustainability due to growing resource-intensive methodologies. Contemporary state-of-the-arts (SOTAs) have significantly hinged on pretrained sophiscated backbones such as Large Language Models (LLMs), embedding sources, or extensive datasets, inducing considerable parametric and computational inefficiency for sustainable use in real-world scenario. Despite their success, following this research direction undermines the overarching mission of this domain to create substantial value to bridge hard-hearing and common populations. Committing to the prevailing trend of LLM and Natural Language Processing (NLP) studies, we pursue a profound essential change in architecture to achieve ground-up improvements without external aid from pretrained models, prior knowledge transfer, or any NLP strategies considered not-from-scratch. Introducing Signformer, a from-scratch Feather-Giant transforming the area towards Edge AI that redefines extremities of performance and efficiency with LLM-competence and edgy-deployable compactness. In this paper, we present nature analysis of sign languages to inform our algorithmic design and deliver a scalable transformer pipeline with convolution and attention novelty. We achieve new 2nd place on leaderboard with a parametric reduction of 467-1807x against the finests as of 2024 and outcompete almost every other methods in a lighter configuration of 0.57 million parameters.
• 摘要：由于资源密集型方法的增多，手语翻译（尤其是无注释范式中的手语翻译）正面临不切实际和不可持续的困境。当代最先进技术（SOTA）在很大程度上依赖于预训练的复杂主干，例如大型语言模型（LLM）、嵌入源或大量数据集，这导致在现实世界场景中可持续使用的参数和计算效率相当低下。尽管取得了成功，但遵循这一研究方向会破坏该领域的总体使命，即为听力障碍人群和普通人群之间的沟通创造巨大价值。我们致力于 LLM 和自然语言处理（NLP）研究的主流趋势，追求架构的深刻根本性变革，以实现从头开始的改进，而无需预训练模型、先验知识转移或任何非从头开始的 NLP 策略的外部帮助。介绍 Signformer，这是一个从头开始的 Feather-Giant，它将该领域转变为边缘 AI，它以 LLM 能力和可部署的紧凑性重新定义性能和效率的极限。在本文中，我们展示了手语的性质分析，以指导我们的算法设计，并提供具有卷积和注意新颖性的可扩展转换器管道。截至 2024 年，我们在排行榜上取得了新的第二名，与最优秀的方法相比，参数减少了 467-1807 倍，并且在 57 万个参数的更轻配置中胜过几乎所有其他方法。

Title: A Flexible Large Language Models Guardrail Development Methodology Applied to Off-Topic Prompt Detection

• Authors: Gabriel Chua, Shing Yee Chan, Shaun Khoo
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.12946
• Pdf URL: https://arxiv.org/pdf/2411.12946
• Copy Paste: [[2411.12946]] A Flexible Large Language Models Guardrail Development Methodology Applied to Off-Topic Prompt Detection(https://arxiv.org/abs/2411.12946)
• Keywords: language model, llm, prompt
• Abstract: Large Language Models are prone to off-topic misuse, where users may prompt these models to perform tasks beyond their intended scope. Current guardrails, which often rely on curated examples or custom classifiers, suffer from high false-positive rates, limited adaptability, and the impracticality of requiring real-world data that is not available in pre-production. In this paper, we introduce a flexible, data-free guardrail development methodology that addresses these challenges. By thoroughly defining the problem space qualitatively and passing this to an LLM to generate diverse prompts, we construct a synthetic dataset to benchmark and train off-topic guardrails that outperform heuristic approaches. Additionally, by framing the task as classifying whether the user prompt is relevant with respect to the system prompt, our guardrails effectively generalize to other misuse categories, including jailbreak and harmful prompts. Lastly, we further contribute to the field by open-sourcing both the synthetic dataset and the off-topic guardrail models, providing valuable resources for developing guardrails in pre-production environments and supporting future research and development in LLM safety.
• 摘要：大型语言模型容易出现离题滥用，用户可能会提示这些模型执行超出其预期范围的任务。当前的护栏通常依赖于精选示例或自定义分类器，存在假阳性率高、适应性有限以及需要预生产中无法获得的真实数据不切实际的问题。在本文中，我们介绍了一种灵活的、无数据的护栏开发方法来应对这些挑战。通过彻底定性地定义问题空间并将其传递给 LLM 以生成各种提示，我们构建了一个合成数据集来对优于启发式方法的离题护栏进行基准测试和训练。此外，通过将任务定义为对用户提示是否与系统提示相关进行分类，我们的护栏可以有效地推广到其他滥用类别，包括越狱和有害提示。最后，我们通过开源合成数据集和非主题护栏模型进一步为该领域做出贡献，为在预生产环境中开发护栏提供宝贵的资源并支持未来 LLM 安全的研究和开发。

Title: Training Bilingual LMs with Data Constraints in the Targeted Language

• Authors: Skyler Seto, Maartje ter Hoeve, He Bai, Natalie Schluter, David Grangier
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.12986
• Pdf URL: https://arxiv.org/pdf/2411.12986
• Copy Paste: [[2411.12986]] Training Bilingual LMs with Data Constraints in the Targeted Language(https://arxiv.org/abs/2411.12986)
• Keywords: language model
• Abstract: Large language models are trained on massive scrapes of the web, as required by current scaling laws. Most progress is made for English, given its abundance of high-quality pretraining data. For most other languages, however, such high quality pretraining data is unavailable. In this work, we study how to boost pretrained model performance in a data constrained target language by enlisting data from an auxiliary language for which high quality data is available. We study this by quantifying the performance gap between training with data in a data-rich auxiliary language compared with training in the target language, exploring the benefits of translation systems, studying the limitations of model scaling for data constrained languages, and proposing new methods for upsampling data from the auxiliary language. Our results show that stronger auxiliary datasets result in performance gains without modification to the model or training objective for close languages, and, in particular, that performance gains due to the development of more information-rich English pretraining datasets can extend to targeted language settings with limited data.
• 摘要：大型语言模型是根据当前的扩展定律要求从网络上大量抓取的数据进行训练的。由于英语拥有丰富的高质量预训练数据，因此取得了大部分进展。然而，对于大多数其他语言来说，这种高质量的预训练数据是无法获得的。在这项工作中，我们研究如何通过从具有高质量数据的辅助语言中获取数据来提高数据受限目标语言中预训练模型的性能。我们通过量化使用数据丰富的辅助语言数据进行训练与使用目标语言进行训练之间的性能差距、探索翻译系统的优势、研究数据受限语言模型扩展的局限性以及提出从辅助语言中上采样数据的新方法来研究这一点。我们的结果表明，更强大的辅助数据集可以在不修改模型或相近语言的训练目标的情况下提高性能，特别是，由于开发了信息更丰富的英语预训练数据集而获得的性能提升可以扩展到数据有限的目标语言设置。

Title: MemoryFormer: Minimize Transformer Computation by Removing Fully-Connected Layers

• Authors: Ning Ding, Yehui Tang, Haochen Qin, Zhenli Zhou, Chao Xu, Lin Li, Kai Han, Heng Liao, Yunhe Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.12992
• Pdf URL: https://arxiv.org/pdf/2411.12992
• Copy Paste: [[2411.12992]] MemoryFormer: Minimize Transformer Computation by Removing Fully-Connected Layers(https://arxiv.org/abs/2411.12992)
• Keywords: language model
• Abstract: In order to reduce the computational complexity of large language models, great efforts have been made to to improve the efficiency of transformer models such as linear attention and flash-attention. However, the model size and corresponding computational complexity are constantly scaled up in pursuit of higher performance. In this work, we present MemoryFormer, a novel transformer architecture which significantly reduces the computational complexity (FLOPs) from a new perspective. We eliminate nearly all the computations of the transformer model except for the necessary computation required by the multi-head attention operation. This is made possible by utilizing an alternative method for feature transformation to replace the linear projection of fully-connected layers. Specifically, we first construct a group of in-memory lookup tables that store a large amount of discrete vectors to replace the weight matrix used in linear projection. We then use a hash algorithm to retrieve a correlated subset of vectors dynamically based on the input embedding. The retrieved vectors combined together will form the output embedding, which provides an estimation of the result of matrix multiplication operation in a fully-connected layer. Compared to conducting matrix multiplication, retrieving data blocks from memory is a much cheaper operation which requires little computations. We train MemoryFormer from scratch and conduct extensive experiments on various benchmarks to demonstrate the effectiveness of the proposed model.
• 摘要：为了降低大型语言模型的计算复杂度，人们付出了巨大的努力来提高 Transformer 模型（如线性注意和 flash-attention）的效率。然而，为了追求更高的性能，模型大小和相应的计算复杂度也在不断扩大。在这项工作中，我们提出了 MemoryFormer，一种新颖的 Transformer 架构，它从新的角度显著降低了计算复杂度（FLOP）。我们消除了 Transformer 模型的几乎所有计算，除了多头注意操作所需的必要计算。这是通过使用特征转换的替代方法来取代全连接层的线性投影来实现的。具体来说，我们首先构建一组内存查找表，这些查找表存储大量离散向量，以取代线性投影中使用的权重矩阵。然后，我们使用哈希算法根据输入嵌入动态检索相关的向量子集。检索到的向量组合在一起将形成输出嵌入，这提供了全连接层中矩阵乘法运算结果的估计。与执行矩阵乘法相比，从内存中检索数据块是一种更便宜的操作，只需要很少的计算。我们从头开始训练 MemoryFormer，并在各种基准上进行大量实验，以证明所提模型的有效性。

Title: Patience Is The Key to Large Language Model Reasoning

• Authors: Yijiong Yu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.13082
• Pdf URL: https://arxiv.org/pdf/2411.13082
• Copy Paste: [[2411.13082]] Patience Is The Key to Large Language Model Reasoning(https://arxiv.org/abs/2411.13082)
• Keywords: language model
• Abstract: Recent advancements in the field of large language models, particularly through the Chain of Thought (CoT) approach, have demonstrated significant improvements in solving complex problems. However, existing models either tend to sacrifice detailed reasoning for brevity due to user preferences, or require extensive and expensive training data to learn complicated reasoning ability, limiting their potential in solving complex tasks. To bridge this gap, following the concept of scaling test-time, we propose a simple method by encouraging models to adopt a more patient reasoning style without the need of introducing new knowledge or skills. To employ a preference optimization approach, we generate detailed reasoning processes as positive examples and simple answers as negative examples, thereby training the model to favor thoroughness in its responses. Our results demonstrate a performance increase of up to 6.7% on GSM8k with training just on a lightweight dataset.
• 摘要：大型语言模型领域的最新进展，尤其是通过思路链 (CoT) 方法，已在解决复杂问题方面取得了显著的进步。然而，现有模型要么由于用户偏好而倾向于牺牲详细推理以求简洁，要么需要大量且昂贵的训练数据来学习复杂的推理能力，从而限制了它们解决复杂任务的潜力。为了弥补这一差距，我们遵循扩展测试时间的概念，提出了一种简单的方法，鼓励模型采用更耐心的推理方式，而无需引入新知识或技能。为了采用偏好优化方法，我们生成详细的推理过程作为正例，生成简单的答案作为反例，从而训练模型使其更倾向于全面地响应。我们的结果表明，仅在轻量级数据集上进行训练，GSM8k 上的性能就提高了 6.7%。

Title: Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding

• Authors: Hyun Ryu, Eric Kim
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.13157
• Pdf URL: https://arxiv.org/pdf/2411.13157
• Copy Paste: [[2411.13157]] Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding(https://arxiv.org/abs/2411.13157)
• Keywords: language model, llm
• Abstract: Efficient inference in large language models (LLMs) has become a critical focus as their scale and complexity grow. Traditional autoregressive decoding, while effective, suffers from computational inefficiencies due to its sequential token generation process. Speculative decoding addresses this bottleneck by introducing a two-stage framework: drafting and verification. A smaller, efficient model generates a preliminary draft, which is then refined by a larger, more sophisticated model. This paper provides a comprehensive survey of speculative decoding methods, categorizing them into draft-centric and model-centric approaches. We discuss key ideas associated with each method, highlighting their potential for scaling LLM inference. This survey aims to guide future research in optimizing speculative decoding and its integration into real-world LLM applications.
• 摘要：随着大型语言模型 (LLM) 的规模和复杂性不断增长，高效推理已成为关注的焦点。传统的自回归解码虽然有效，但由于其顺序标记生成过程而存在计算效率低下的问题。推测解码通过引入两阶段框架解决了这一瓶颈：起草和验证。较小、高效的模型会生成初步草稿，然后由更大、更复杂的模型对其进行完善。本文对推测解码方法进行了全面概述，将其分为以草稿为中心和以模型为中心的方法。我们讨论了与每种方法相关的关键思想，强调了它们扩展 LLM 推理的潜力。本概述旨在指导未来优化推测解码及其与现实世界 LLM 应用中集成的研究。

Title: Hard-Synth: Synthesizing Diverse Hard Samples for ASR using Zero-Shot TTS and LLM

• Authors: Jiawei Yu, Yuang Li, Xiaosong Qiao, Huan Zhao, Xiaofeng Zhao, Wei Tang, Min Zhang, Hao Yang, Jinsong Su
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2411.13159
• Pdf URL: https://arxiv.org/pdf/2411.13159
• Copy Paste: [[2411.13159]] Hard-Synth: Synthesizing Diverse Hard Samples for ASR using Zero-Shot TTS and LLM(https://arxiv.org/abs/2411.13159)
• Keywords: language model, llm, prompt
• Abstract: Text-to-speech (TTS) models have been widely adopted to enhance automatic speech recognition (ASR) systems using text-only corpora, thereby reducing the cost of labeling real speech data. Existing research primarily utilizes additional text data and predefined speech styles supported by TTS models. In this paper, we propose Hard-Synth, a novel ASR data augmentation method that leverages large language models (LLMs) and advanced zero-shot TTS. Our approach employs LLMs to generate diverse in-domain text through rewriting, without relying on additional text data. Rather than using predefined speech styles, we introduce a hard prompt selection method with zero-shot TTS to clone speech styles that the ASR model finds challenging to recognize. Experiments demonstrate that Hard-Synth significantly enhances the Conformer model, achieving relative word error rate (WER) reductions of 6.5\%/4.4\% on LibriSpeech dev/test-other subsets. Additionally, we show that Hard-Synth is data-efficient and capable of reducing bias in ASR.
• 摘要：文本转语音 (TTS) 模型已被广泛采用，以使用纯文本语料库来增强自动语音识别 (ASR) 系统，从而降低标记真实语音数据的成本。现有研究主要利用 TTS 模型支持的额外文本数据和预定义语音风格。在本文中，我们提出了 Hard-Synth，这是一种新颖的 ASR 数据增强方法，它利用大型语言模型 (LLM) 和先进的零样本 TTS。我们的方法使用 LLM 通过重写生成多样化的域内文本，而不依赖于额外的文本数据。我们没有使用预定义的语音风格，而是引入了一种使用零样本 TTS 的硬提示选择方法来克隆 ASR 模型难以识别的语音风格。实验表明，Hard-Synth 显著增强了 Conformer 模型，在 LibriSpeech dev/test-other 子集上实现了 6.5\%/4.4\% 的相对词错误率 (WER) 降低。此外，我们表明 Hard-Synth 具有数据效率，能够减少 ASR 中的偏差。

Title: AIDBench: A benchmark for evaluating the authorship identification capability of large language models

• Authors: Zichen Wen, Dadi Guo, Huishuai Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.13226
• Pdf URL: https://arxiv.org/pdf/2411.13226
• Copy Paste: [[2411.13226]] AIDBench: A benchmark for evaluating the authorship identification capability of large language models(https://arxiv.org/abs/2411.13226)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: As large language models (LLMs) rapidly advance and integrate into daily life, the privacy risks they pose are attracting increasing attention. We focus on a specific privacy risk where LLMs may help identify the authorship of anonymous texts, which challenges the effectiveness of anonymity in real-world systems such as anonymous peer review systems. To investigate these risks, we present AIDBench, a new benchmark that incorporates several author identification datasets, including emails, blogs, reviews, articles, and research papers. AIDBench utilizes two evaluation methods: one-to-one authorship identification, which determines whether two texts are from the same author; and one-to-many authorship identification, which, given a query text and a list of candidate texts, identifies the candidate most likely written by the same author as the query text. We also introduce a Retrieval-Augmented Generation (RAG)-based method to enhance the large-scale authorship identification capabilities of LLMs, particularly when input lengths exceed the models' context windows, thereby establishing a new baseline for authorship identification using LLMs. Our experiments with AIDBench demonstrate that LLMs can correctly guess authorship at rates well above random chance, revealing new privacy risks posed by these powerful models. The source code and data will be made publicly available after acceptance.
• 摘要：随着大型语言模型 (LLM) 的快速发展和融入日常生活，它们所带来的隐私风险也越来越受到关注。我们专注于一种特定的隐私风险，即 LLM 可能有助于识别匿名文本的作者，这对匿名同行评审系统等现实世界系统中匿名性的有效性提出了挑战。为了调查这些风险，我们提出了 AIDBench，这是一种新的基准，它结合了多个作者识别数据集，包括电子邮件、博客、评论、文章和研究论文。AIDBench 采用两种评估方法：一对一作者识别，确定两篇文本是否来自同一作者；一对多作者识别，给定查询文本和候选文本列表，识别最有可能由与查询文本相同的作者撰写的候选文本。我们还引入了一种基于检索增强生成 (RAG) 的方法来增强 LLM 的大规模作者识别能力，特别是当输入长度超出模型的上下文窗口时，从而为使用 LLM 进行作者识别建立了新的基线。我们通过 AIDBench 进行的实验表明，LLM 能够以远高于随机概率的概率正确猜测作者身份，这揭示了这些强大模型带来的新隐私风险。源代码和数据将在接受后公开。

Title: BIPro: Zero-shot Chinese Poem Generation via Block Inverse Prompting Constrained Generation Framework

• Authors: Xu Zou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.13237
• Pdf URL: https://arxiv.org/pdf/2411.13237
• Copy Paste: [[2411.13237]] BIPro: Zero-shot Chinese Poem Generation via Block Inverse Prompting Constrained Generation Framework(https://arxiv.org/abs/2411.13237)
• Keywords: gpt, prompt, chat
• Abstract: Recently, generative pre-trained models have made significant strides, particularly highlighted by the release of ChatGPT and GPT-4, which exhibit superior cross-domain capabilities. However, these models still face challenges on constrained writing tasks like poem generation under open-domain titles. In response to this challenge, we introduce Block Inverse Prompting (BIPro) constrained generation framework. BIPro leverages two block inverse prompting methods, revise and rewrite, that mimic the process of human text writing using block generative models. It significantly improves the zero-shot generation quality on the formidable constrained generation task of open-domain traditional-form Chinese poem generation. Based on a less powerful block generative model GLM-10B-Chinese, poems composed via BIPro without priming or additional training outperform both most advanced direct generative systems like GPT-4 or GLM-4 and best domain-specific systems such as Yusheng, Shisanbai, or Baidu Poetry Helper in human evaluation by proficient poets. Finally, BIPro considerably narrows the gap between AI-generated works and short-listed human literary arts in another human evaluation, unveiling the promising potential of block generative models in improving the quality of constrained generation.
• 摘要：最近，生成式预训练模型取得了重大进展，尤其突出的是 ChatGPT 和 GPT-4 的发布，它们表现出卓越的跨域能力。然而，这些模型在受限写作任务（如开放域标题下的诗歌生成）上仍然面临挑战。为了应对这一挑战，我们引入了块逆提示 (BIPro) 受限生成框架。BIPro 利用两种块逆提示方法，即修改和重写，使用块生成模型模拟人类文本写作的过程。它显著提高了开放域传统形式中文诗歌生成这一艰巨的受限生成任务的零样本生成质量。基于功能较弱的块生成模型 GLM-10B-Chinese，通过 BIPro 创作的诗歌无需启动或额外训练，在熟练诗人的人工评估中，其表现优于最先进的直接生成系统（如 GPT-4 或 GLM-4）和最佳领域特定系统（如 Yusheng、Shisanbai 或百度诗歌助手）。最后，BIPro 在另一项人工评测中大大缩小了 AI 生成作品与入围人类文学艺术作品之间的差距，揭示了分块生成模型在提升受限生成质量方面的巨大潜力。

Title: Leveraging Prior Experience: An Expandable Auxiliary Knowledge Base for Text-to-SQL

• Authors: Zhibo Chu, Zichong Wang, Qitao Qin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.13244
• Pdf URL: https://arxiv.org/pdf/2411.13244
• Copy Paste: [[2411.13244]] Leveraging Prior Experience: An Expandable Auxiliary Knowledge Base for Text-to-SQL(https://arxiv.org/abs/2411.13244)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) exhibit impressive problem-solving skills across many tasks, but they still underperform compared to humans in various downstream applications, such as text-to-SQL. On the BIRD benchmark leaderboard, human performance achieves an accuracy of 92.96\%, whereas the top-performing method reaches only 72.39\%. Notably, these state-of-the-art (SoTA) methods predominantly rely on in-context learning to simulate human-like reasoning. However, they overlook a critical human skill: continual learning. Inspired by the educational practice of maintaining mistake notebooks during our formative years, we propose LPE-SQL (Leveraging Prior Experience: An Expandable Auxiliary Knowledge Base for Text-to-SQL), a novel framework designed to augment LLMs by enabling continual learning without requiring parameter fine-tuning. LPE-SQL consists of four modules that \textbf{i)} retrieve relevant entries, \textbf{ii)} efficient sql generation, \textbf{iii)} generate the final result through a cross-consistency mechanism and \textbf{iv)} log successful and failed tasks along with their reasoning processes or reflection-generated tips. Importantly, the core module of LPE-SQL is the fourth one, while the other modules employ foundational methods, allowing LPE-SQL to be easily integrated with SoTA technologies to further enhance performance. Our experimental results demonstrate that this continual learning approach yields substantial performance gains, with the smaller Llama-3.1-70B model with surpassing the performance of the larger Llama-3.1-405B model using SoTA methods.
• 摘要：大型语言模型 (LLM) 在许多任务中表现出令人印象深刻的解决问题的能力，但它们在各种下游应用（例如文本到 SQL）中的表现仍然不如人类。在 BIRD 基准排行榜上，人类的表现达到了 92.96\%，而表现最好的方法仅达到 72.39\%。值得注意的是，这些最先进的 (SoTA) 方法主要依靠情境学习来模拟类似人类的推理。然而，它们忽略了一项关键的人类技能：持续学习。受成长时期维护错误笔记本的教育实践的启发，我们提出了 LPE-SQL（利用先前经验：文本到 SQL 的可扩展辅助知识库），这是一个新颖的框架，旨在通过实现持续学习而无需参数微调来增强 LLM。 LPE-SQL 由四个模块组成，分别是 \textbf{i)} 检索相关条目、\textbf{ii)} 高效 SQL 生成、\textbf{iii)} 通过交叉一致性机制生成最终结果以及 \textbf{iv)} 记录成功和失败的任务及其推理过程或反射生成的提示。重要的是，LPE-SQL 的核心模块是第四个模块，而其他模块采用基础方法，使 LPE-SQL 能够轻松与 SoTA 技术集成以进一步提高性能。我们的实验结果表明，这种持续学习方法可带来显着的性能提升，较小的 Llama-3.1-70B 模型的性能超过了使用 SoTA 方法的较大的 Llama-3.1-405B 模型的性能。

Title: Combining Autoregressive and Autoencoder Language Models for Text Classification

• Authors: João Gonçalves
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.13282
• Pdf URL: https://arxiv.org/pdf/2411.13282
• Copy Paste: [[2411.13282]] Combining Autoregressive and Autoencoder Language Models for Text Classification(https://arxiv.org/abs/2411.13282)
• Keywords: language model, gpt
• Abstract: This paper presents CAALM-TC (Combining Autoregressive and Autoencoder Language Models for Text Classification), a novel method that enhances text classification by integrating autoregressive and autoencoder language models. Autoregressive large language models such as Open AI's GPT, Meta's Llama or Microsoft's Phi offer promising prospects for content analysis practitioners, but they generally underperform supervised BERT based models for text classification. CAALM leverages autoregressive models to generate contextual information based on input texts, which is then combined with the original text and fed into an autoencoder model for classification. This hybrid approach capitalizes on the extensive contextual knowledge of autoregressive models and the efficient classification capabilities of autoencoders. Experimental results on four benchmark datasets demonstrate that CAALM consistently outperforms existing methods, particularly in tasks with smaller datasets and more abstract classification objectives. The findings indicate that CAALM offers a scalable and effective solution for automated content analysis in social science research that minimizes sample size requirements.
• 摘要：本文介绍了 CAALM-TC​​（结合自回归和自编码器语言模型进行文本分类），这是一种通过集成自回归和自编码器语言模型来增强文本分类的新方法。自回归大型语言模型（例如 Open AI 的 GPT、Meta 的 Llama 或 Microsoft 的 Phi）为内容分析从业者提供了光明的前景，但它们在文本分类方面的表现通常不如基于 BERT 的监督模型。CAALM 利用自回归模型根据输入文本生成上下文信息，然后将其与原始文本相结合并输入到自动编码器模型中进行分类。这种混合方法利用了自回归模型的广泛上下文知识和自动编码器的高效分类能力。在四个基准数据集上的实验结果表明，CAALM 始终优于现有方法，尤其是在数据集较小、分类目标更抽象的任务中。研究结果表明，CAALM 为社会科学研究中的自动内容分析提供了一种可扩展且有效的解决方案，可最大限度地减少样本量要求。

Title: Fact-Level Confidence Calibration and Self-Correction

• Authors: Yige Yuan, Bingbing Xu, Hexiang Tan, Fei Sun, Teng Xiao, Wei Li, Huawei Shen, Xueqi Cheng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.13343
• Pdf URL: https://arxiv.org/pdf/2411.13343
• Copy Paste: [[2411.13343]] Fact-Level Confidence Calibration and Self-Correction(https://arxiv.org/abs/2411.13343)
• Keywords: llm, hallucination
• Abstract: Confidence calibration in LLMs, i.e., aligning their self-assessed confidence with the actual accuracy of their responses, enabling them to self-evaluate the correctness of their outputs. However, current calibration methods for LLMs typically estimate two scalars to represent overall response confidence and correctness, which is inadequate for long-form generation where the response includes multiple atomic facts and may be partially confident and correct. These methods also overlook the relevance of each fact to the query. To address these challenges, we propose a Fact-Level Calibration framework that operates at a finer granularity, calibrating confidence to relevance-weighted correctness at the fact level. Furthermore, comprehensive analysis under the framework inspired the development of Confidence-Guided Fact-level Self-Correction ($\textbf{ConFix}$), which uses high-confidence facts within a response as additional knowledge to improve low-confidence ones. Extensive experiments across four datasets and six models demonstrate that ConFix effectively mitigates hallucinations without requiring external knowledge sources such as retrieval systems.
• 摘要：LLM 中的置信度校准，即将他们的自我评估置信度与实际回答准确性相结合，使他们能够自我评估其输出的正确性。然而，目前 LLM 的校准方法通常估计两个标量来表示整体回答置信度和正确性，这对于长格式生成来说是不够的，因为长格式生成中的回答包含多个原子事实，可能部分置信和正确。这些方法还忽略了每个事实与查询的相关性。为了应对这些挑战，我们提出了一个事实级校准框架，该框架以更细的粒度运行，在事实级别将置信度校准为相关性加权的正确性。此外，在该框架下进行的全面分析启发了置信度引导的事实级自我校正 ($\textbf{ConFix}$) 的开发，它使用回答中的高置信度事实作为额外知识来改进低置信度事实。针对四个数据集和六个模型的大量实验表明，ConFix 可以有效缓解幻觉，而不需要检索系统等外部知识源。

Title: On the Way to LLM Personalization: Learning to Remember User Conversations

• Authors: Lucie Charlotte Magister, Katherine Metcalf, Yizhe Zhang, Maartje ter Hoeve
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.13405
• Pdf URL: https://arxiv.org/pdf/2411.13405
• Copy Paste: [[2411.13405]] On the Way to LLM Personalization: Learning to Remember User Conversations(https://arxiv.org/abs/2411.13405)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have quickly become an invaluable assistant for a variety of tasks. However, their effectiveness is constrained by their ability to tailor responses to human preferences and behaviors via personalization. Prior work in LLM personalization has largely focused on style transfer or incorporating small factoids about the user, as knowledge injection remains an open challenge. In this paper, we explore injecting knowledge of prior conversations into LLMs to enable future work on less redundant, personalized conversations. We identify two real-world constraints: (1) conversations are sequential in time and must be treated as such during training, and (2) per-user personalization is only viable in parameter-efficient settings. To this aim, we propose PLUM, a pipeline performing data augmentation for up-sampling conversations as question-answer pairs, that are then used to finetune a low-rank adaptation adapter with a weighted cross entropy loss. Even in this first exploration of the problem, we perform competitively with baselines such as RAG, attaining an accuracy of 81.5% across 100 conversations.
• 摘要：大型语言模型 (LLM) 已迅速成为各种任务的宝贵助手。然而，它们的有效性受到它们通过个性化定制响应以适应人类偏好和行为的能力的限制。LLM 个性化方面的先前工作主要集中在风格转换或融入有关用户的小事实，因为知识注入仍然是一个悬而未决的挑战。在本文中，我们探索将先前对话的知识注入 LLM，以便将来进行更少冗余、个性化的对话。我们确定了两个现实世界的限制：(1) 对话在时间上是连续的，在训练期间必须如此处理，(2) 每个用户的个性化仅在参数高效的设置中可行。为此，我们提出了 PLUM，这是一种执行数据增强的管道，用于将对话上采样为问答对，然后使用它们来微调具有加权交叉熵损失的低秩自适应适配器。即使在首次探索该问题时，我们的表现也与 RAG 等基线相当，在 100 次对话中达到了 81.5% 的准确率。

Title: Transformer-Based Contextualized Language Models Joint with Neural Networks for Natural Language Inference in Vietnamese

• Authors: Dat Van-Thanh Nguyen, Tin Van Huynh, Kiet Van Nguyen, Ngan Luu-Thuy Nguyen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.13407
• Pdf URL: https://arxiv.org/pdf/2411.13407
• Copy Paste: [[2411.13407]] Transformer-Based Contextualized Language Models Joint with Neural Networks for Natural Language Inference in Vietnamese(https://arxiv.org/abs/2411.13407)
• Keywords: language model
• Abstract: Natural Language Inference (NLI) is a task within Natural Language Processing (NLP) that holds value for various AI applications. However, there have been limited studies on Natural Language Inference in Vietnamese that explore the concept of joint models. Therefore, we conducted experiments using various combinations of contextualized language models (CLM) and neural networks. We use CLM to create contextualized work presentations and use Neural Networks for classification. Furthermore, we have evaluated the strengths and weaknesses of each joint model and identified the model failure points in the Vietnamese context. The highest F1 score in this experiment, up to 82.78\% in the benchmark dataset (ViNLI). By conducting experiments with various models, the most considerable size of the CLM is XLM-R (355M). That combination has consistently demonstrated superior performance compared to fine-tuning strong pre-trained language models like PhoBERT (+6.58\%), mBERT (+19.08\%), and XLM-R (+0.94\%) in terms of F1-score. This article aims to introduce a novel approach or model that attains improved performance for Vietnamese NLI. Overall, we find that the joint approach of CLM and neural networks is simple yet capable of achieving high-quality performance, which makes it suitable for applications that require efficient resource utilization.
• 摘要：自然语言推理 (NLI) 是自然语言处理 (NLP) 中的一项任务，对各种 AI 应用都具有价值。然而，关于越南语自然语言推理的研究有限，这些研究探索了联合模型的概念。因此，我们使用各种情境化语言模型 (CLM) 和神经网络的组合进行了实验。我们使用 CLM 创建情境化的工作演示，并使用神经网络进行分类。此外，我们评估了每个联合模型的优势和劣势，并确定了越南语背景下的模型失败点。本次实验中 F1 得分最高，在基准数据集 (ViNLI) 中高达 82.78\%。通过对各种模型进行实验，CLM 的最大规模是 XLM-R (355M)。与微调强大的预训练语言模型（如 PhoBERT (+6.58\%)、mBERT (+19.08\%) 和 XLM-R (+0.94\%)）相比，该组合在 F1 得分方面始终表现出卓越的性能。本文旨在介绍一种可提高越南语 NLI 性能的新方法或模型。总体而言，我们发现 CLM 和神经网络的联合方法简单但能够实现高质量的性能，这使其适合需要高效资源利用的应用程序。

Title: Unification of Balti and trans-border sister dialects in the essence of LLMs and AI Technology

• Authors: Muhammad Sharif, Jiangyan Yi, Muhammad Shoaib
• Subjects: cs.CL, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13409
• Pdf URL: https://arxiv.org/pdf/2411.13409
• Copy Paste: [[2411.13409]] Unification of Balti and trans-border sister dialects in the essence of LLMs and AI Technology(https://arxiv.org/abs/2411.13409)
• Keywords: language model, llm
• Abstract: The language called Balti belongs to the Sino-Tibetan, specifically the Tibeto-Burman language family. It is understood with variations, across populations in India, China, Pakistan, Nepal, Tibet, Burma, and Bhutan, influenced by local cultures and producing various dialects. Considering the diverse cultural, socio-political, religious, and geographical impacts, it is important to step forward unifying the dialects, the basis of common root, lexica, and phonological perspectives, is vital. In the era of globalization and the increasingly frequent developments in AI technology, understanding the diversity and the efforts of dialect unification is important to understanding commonalities and shortening the gaps impacted by unavoidable circumstances. This article analyzes and examines how artificial intelligence AI in the essence of Large Language Models LLMs, can assist in analyzing, documenting, and standardizing the endangered Balti Language, based on the efforts made in different dialects so far.
• 摘要：巴尔蒂语属于汉藏语系，具体来说是藏缅语系。印度、中国、巴基斯坦、尼泊尔、西藏、缅甸和不丹的人们对此语言的理解各不相同，受当地文化的影响，产生了各种方言。考虑到不同的文化、社会政治、宗教和地理影响，统一方言、共同的词根、词汇和音韵观点的基础至关重要。在全球化时代和人工智能技术日益频繁的发展中，了解方言的多样性和统一的努力对于理解共同点和缩小不可避免的情况造成的差距非常重要。本文分析和研究了人工智能（即大型语言模型 LLM）如何基于迄今为止在不同方言中所做的努力，帮助分析、记录和标准化濒临灭绝的巴尔蒂语。

Title: LIMBA: An Open-Source Framework for the Preservation and Valorization of Low-Resource Languages using Generative Models

• Authors: Salvatore Mario Carta, Stefano Chessa, Giulia Contu, Andrea Corriga, Andrea Deidda, Gianni Fenu, Luca Frigau, Alessandro Giuliani, Luca Grassi, Marco Manolo Manca, Mirko Marras, Francesco Mola, Bastianino Mossa, Piergiorgio Mura, Marco Ortu, Leonardo Piano, Simone Pisano, Alessia Pisu, Alessandro Sebastian Podda, Livio Pompianu, Simone Seu, Sandro Gabriele Tiddia
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.13453
• Pdf URL: https://arxiv.org/pdf/2411.13453
• Copy Paste: [[2411.13453]] LIMBA: An Open-Source Framework for the Preservation and Valorization of Low-Resource Languages using Generative Models(https://arxiv.org/abs/2411.13453)
• Keywords: language model
• Abstract: Minority languages are vital to preserving cultural heritage, yet they face growing risks of extinction due to limited digital resources and the dominance of artificial intelligence models trained on high-resource languages. This white paper proposes a framework to generate linguistic tools for low-resource languages, focusing on data creation to support the development of language models that can aid in preservation efforts. Sardinian, an endangered language, serves as the case study to demonstrate the framework's effectiveness. By addressing the data scarcity that hinders intelligent applications for such languages, we contribute to promoting linguistic diversity and support ongoing efforts in language standardization and revitalization through modern technologies.
• 摘要：少数民族语言对于保护文化遗产至关重要，但由于数字资源有限，以及以高资源语言训练的人工智能模型占主导地位，少数民​​族语言面临着越来越大的灭绝风险。本白皮书提出了一个为低资源语言生成语言工具的框架，重点是创建数据以支持开发有助于保护工作的语言模型。撒丁语是一种濒临灭绝的语言，是该框架有效性的案例研究。通过解决阻碍此类语言智能应用的数据稀缺问题，我们为促进语言多样性做出了贡献，并支持通过现代技术进行语言标准化和振兴的持续努力。

Title: When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Training

• Authors: Haonan Wang, Qian Liu, Chao Du, Tongyao Zhu, Cunxiao Du, Kenji Kawaguchi, Tianyu Pang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.13476
• Pdf URL: https://arxiv.org/pdf/2411.13476
• Copy Paste: [[2411.13476]] When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Training(https://arxiv.org/abs/2411.13476)
• Keywords: language model, llm
• Abstract: Extending context window sizes allows large language models (LLMs) to process longer sequences and handle more complex tasks. Rotary Positional Embedding (RoPE) has become the de facto standard due to its relative positional encoding properties that benefit long-context training. However, we observe that using RoPE with BFloat16 format results in numerical issues, causing it to deviate from its intended relative positional encoding, especially in long-context scenarios. This issue arises from BFloat16's limited precision and accumulates as context length increases, with the first token contributing significantly to this problem. To address this, we develop AnchorAttention, a plug-and-play attention method that alleviates numerical issues caused by BFloat16, improves long-context capabilities, and speeds up training. AnchorAttention reduces unnecessary attention computations, maintains semantic coherence, and boosts computational efficiency by treating the first token as a shared anchor with a consistent position ID, making it visible to all documents within the training context. Experiments on three types of LLMs demonstrate that AnchorAttention significantly improves long-context performance and reduces training time by over 50\% compared to standard full attention mechanisms, while preserving the original LLM's capabilities on general tasks. Our code is available at this https URL.
• 摘要：扩展上下文窗口大小允许大型语言模型 (LLM) 处理更长的序列并处理更复杂的任务。旋转位置嵌入 (RoPE) 已成为事实上的标准，因为它具有相对位置编码属性，有利于长上下文训练。但是，我们观察到使用 BFloat16 格式的 RoPE 会导致数值问题，导致它偏离其预期的相对位置编码，尤其是在长上下文场景中。此问题源于 BFloat16 的有限精度，并随着上下文长度的增加而累积，第一个标记对此问题有重大影响。为了解决这个问题，我们开发了 AnchorAttention，这是一种即插即用的注意力方法，可缓解 BFloat16 引起的数值问题，提高长上下文能力并加快训练速度。AnchorAttention 通过将第一个标记视为具有一致位置 ID 的共享锚点，使其对训练上下文中的所有文档可见，从而减少不必要的注意力计算，保持语义一致性并提高计算效率。在三种类型的 LLM 上进行的实验表明，与标准全注意力机制相比，AnchorAttention 显著提高了长上下文性能，并将训练时间缩短了 50% 以上，同时保留了原始 LLM 在一般任务上的能力。我们的代码可在此 https URL 上找到。

Title: PatentEdits: Framing Patent Novelty as Textual Entailment

• Authors: Ryan Lee, Alexander Spangher, Xuezhe Ma
• Subjects: cs.CL, cs.AI, cs.CY, cs.IR
• Abstract URL: https://arxiv.org/abs/2411.13477
• Pdf URL: https://arxiv.org/pdf/2411.13477
• Copy Paste: [[2411.13477]] PatentEdits: Framing Patent Novelty as Textual Entailment(https://arxiv.org/abs/2411.13477)
• Keywords: language model, llm
• Abstract: A patent must be deemed novel and non-obvious in order to be granted by the US Patent Office (USPTO). If it is not, a US patent examiner will cite the prior work, or prior art, that invalidates the novelty and issue a non-final rejection. Predicting what claims of the invention should change given the prior art is an essential and crucial step in securing invention rights, yet has not been studied before as a learnable task. In this work we introduce the PatentEdits dataset, which contains 105K examples of successful revisions that overcome objections to novelty. We design algorithms to label edits sentence by sentence, then establish how well these edits can be predicted with large language models (LLMs). We demonstrate that evaluating textual entailment between cited references and draft sentences is especially effective in predicting which inventive claims remained unchanged or are novel in relation to prior art.
• 摘要：专利必须被视为新颖且非显而易见才能获得美国专利局 (USPTO) 的批准。如果不是，美国专利审查员将引用先前的工作或现有技术，使新颖性无效并发出非最终驳回。根据现有技术预测发明的哪些权利要求应该改变是确保发明权的重要且关键的步骤，但之前尚未作为一项可学习的任务进行研究。在这项工作中，我们引入了 PatentEdits 数据集，其中包含 105K 个成功修改克服新颖性异议的示例。我们设计算法来逐句标记编辑，然后确定使用大型语言模型 (LLM) 预测这些编辑的准确性。我们证明，评估引用参考文献和草稿句子之间的文本蕴涵对于预测哪些发明权利要求保持不变或相对于现有技术具有新颖性特别有效。

Title: Utilizing Large Language Models to Synthesize Product Desirability Datasets

• Authors: John D. Hastings, Sherri Weitl-Harms, Joseph Doty, Zachary L. Myers, Warren Thompson
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.13485
• Pdf URL: https://arxiv.org/pdf/2411.13485
• Copy Paste: [[2411.13485]] Utilizing Large Language Models to Synthesize Product Desirability Datasets(https://arxiv.org/abs/2411.13485)
• Keywords: language model, gpt, llm
• Abstract: This research explores the application of large language models (LLMs) to generate synthetic datasets for Product Desirability Toolkit (PDT) testing, a key component in evaluating user sentiment and product experience. Utilizing gpt-4o-mini, a cost-effective alternative to larger commercial LLMs, three methods, Word+Review, Review+Word, and Supply-Word, were each used to synthesize 1000 product reviews. The generated datasets were assessed for sentiment alignment, textual diversity, and data generation cost. Results demonstrated high sentiment alignment across all methods, with Pearson correlations ranging from 0.93 to 0.97. Supply-Word exhibited the highest diversity and coverage of PDT terms, although with increased generation costs. Despite minor biases toward positive sentiments, in situations with limited test data, LLM-generated synthetic data offers significant advantages, including scalability, cost savings, and flexibility in dataset production.
• 摘要：本研究探索了大型语言模型 (LLM) 在生成产品可取性工具包 (PDT) 测试合成数据集中的应用，这是评估用户情绪和产品体验的关键组成部分。利用 gpt-4o-mini（大型商业 LLM 的经济高效替代方案），三种方法（Word+Review、Review+Word 和 Supply-Word）分别用于合成 1000 条产品评论。对生成的数据集进行了情绪一致性、文本多样性和数据生成成本评估。结果表明，所有方法的情绪一致性都很高，皮尔逊相关性在 0.93 到 0.97 之间。Supply-Word 表现出最高的 PDT 术语多样性和覆盖率，尽管生成成本有所增加。尽管对积极情绪有轻微的偏见，但在测试数据有限的情况下，LLM 生成的合成数据具有显着的优势，包括可扩展性、成本节省和数据集生成的灵活性。

Title: Disentangling Memory and Reasoning Ability in Large Language Models

• Authors: Mingyu Jin, Weidi Luo, Sitao Cheng, Xinyi Wang, Wenyue Hua, Ruixiang Tang, William Yang Wang, Yongfeng Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.13504
• Pdf URL: https://arxiv.org/pdf/2411.13504
• Copy Paste: [[2411.13504]] Disentangling Memory and Reasoning Ability in Large Language Models(https://arxiv.org/abs/2411.13504)
• Keywords: language model, llm, hallucination
• Abstract: Large Language Models (LLMs) have demonstrated strong performance in handling complex tasks requiring both extensive knowledge and reasoning abilities. However, the existing LLM inference pipeline operates as an opaque process without explicit separation between knowledge retrieval and reasoning steps, making the model's decision-making process unclear and disorganized. This ambiguity can lead to issues such as hallucinations and knowledge forgetting, which significantly impact the reliability of LLMs in high-stakes domains. In this paper, we propose a new inference paradigm that decomposes the complex inference process into two distinct and clear actions: (1) memory recall: which retrieves relevant knowledge, and (2) reasoning: which performs logical steps based on the recalled knowledge. To facilitate this decomposition, we introduce two special tokens memory and reason, guiding the model to distinguish between steps that require knowledge retrieval and those that involve reasoning. Our experiment results show that this decomposition not only improves model performance but also enhances the interpretability of the inference process, enabling users to identify sources of error and refine model responses effectively. The code is available at this https URL.
• 摘要：大型语言模型 (LLM) 在处理需要广泛知识和推理能力的复杂任务方面表现出色。然而，现有的 LLM 推理流程是一个不透明的过程，没有明确区分知识检索和推理步骤，这使得模型的决策过程不清晰、混乱。这种模糊性可能导致幻觉和知识遗忘等问题，从而严重影响 LLM 在高风险领域的可靠性。在本文中，我们提出了一种新的推理范式，将复杂的推理过程分解为两个截然不同的动作：(1) 记忆回忆：检索相关知识，(2) 推理：根据回忆起的知识执行逻辑步骤。为了促进这种分解，我们引入了两个特殊的标记记忆和推理，引导模型区分需要知识检索的步骤和涉及推理的步骤。我们的实验结果表明，这种分解不仅提高了模型性能，还增强了推理过程的可解释性，使用户能够识别错误来源并有效地改进模型响应。代码可在此 https URL 上找到。

Title: Advancing Complex Medical Communication in Arabic with Sporo AraSum: Surpassing Existing Large Language Models

• Authors: Chanseo Lee, Sonu Kumar, Kimon A. Vogt, Sam Meraj, Antonia Vogt
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.13518
• Pdf URL: https://arxiv.org/pdf/2411.13518
• Copy Paste: [[2411.13518]] Advancing Complex Medical Communication in Arabic with Sporo AraSum: Surpassing Existing Large Language Models(https://arxiv.org/abs/2411.13518)
• Keywords: language model, hallucination
• Abstract: The increasing demand for multilingual capabilities in healthcare underscores the need for AI models adept at processing diverse languages, particularly in clinical documentation and decision-making. Arabic, with its complex morphology, syntax, and diglossia, poses unique challenges for natural language processing (NLP) in medical contexts. This case study evaluates Sporo AraSum, a language model tailored for Arabic clinical documentation, against JAIS, the leading Arabic NLP model. Using synthetic datasets and modified PDQI-9 metrics modified ourselves for the purposes of assessing model performances in a different language. The study assessed the models' performance in summarizing patient-physician interactions, focusing on accuracy, comprehensiveness, clinical utility, and linguistic-cultural competence. Results indicate that Sporo AraSum significantly outperforms JAIS in AI-centric quantitative metrics and all qualitative attributes measured in our modified version of the PDQI-9. AraSum's architecture enables precise and culturally sensitive documentation, addressing the linguistic nuances of Arabic while mitigating risks of AI hallucinations. These findings suggest that Sporo AraSum is better suited to meet the demands of Arabic-speaking healthcare environments, offering a transformative solution for multilingual clinical workflows. Future research should incorporate real-world data to further validate these findings and explore broader integration into healthcare systems.
• 摘要：医疗保健领域对多语言能力的需求日益增长，这凸显了对擅长处理多种语言的 AI 模型的需求，特别是在临床文档和决策方面。阿拉伯语具有复杂的形态、句法和双语，对医学背景下的自然语言处理 (NLP) 提出了独特的挑战。本案例研究评估了 Sporo AraSum（一种针对阿拉伯语临床文档量身定制的语言模型）与领先的阿拉伯语 NLP 模型 JAIS 的比较。我们使用合成数据集和修改后的 PDQI-9 指标对自己进行了修改，以评估模型在不同语言中的表现。该研究评估了模型在总结医患互动方面的表现，重点关注准确性、全面性、临床实用性和语言文化能力。结果表明，Sporo AraSum 在以 AI 为中心的定量指标和我们修改后的 PDQI-9 版本中测量的所有定性属性方面都明显优于 JAIS。 AraSum 的架构能够实现精确且文化敏感的文档记录，解决阿拉伯语的语言细微差别，同时降低 AI 幻觉的风险。这些发现表明，Sporo AraSum 更适合满足阿拉伯语医疗环境的需求，为多语言临床工作流程提供变革性解决方案。未来的研究应结合现实世界的数据来进一步验证这些发现，并探索更广泛地融入医疗保健系统。

Title: Predictive Insights into LGBTQ+ Minority Stress: A Transductive Exploration of Social Media Discourse

• Authors: S. Chapagain, Y. Zhao, T. K. Rohleen, S. M. Hamdi, S. F. Boubrahimi, R. E. Flinn, E. M. Lund, D. Klooster, J. R. Scheer, C. J. Cascalheira
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.13534
• Pdf URL: https://arxiv.org/pdf/2411.13534
• Copy Paste: [[2411.13534]] Predictive Insights into LGBTQ+ Minority Stress: A Transductive Exploration of Social Media Discourse(https://arxiv.org/abs/2411.13534)
• Keywords: language model
• Abstract: Individuals who identify as sexual and gender minorities, including lesbian, gay, bisexual, transgender, queer, and others (LGBTQ+) are more likely to experience poorer health than their heterosexual and cisgender counterparts. One primary source that drives these health disparities is minority stress (i.e., chronic and social stressors unique to LGBTQ+ communities' experiences adapting to the dominant culture). This stress is frequently expressed in LGBTQ+ users' posts on social media platforms. However, these expressions are not just straightforward manifestations of minority stress. They involve linguistic complexity (e.g., idiom or lexical diversity), rendering them challenging for many traditional natural language processing methods to detect. In this work, we designed a hybrid model using Graph Neural Networks (GNN) and Bidirectional Encoder Representations from Transformers (BERT), a pre-trained deep language model to improve the classification performance of minority stress detection. We experimented with our model on a benchmark social media dataset for minority stress detection (LGBTQ+ MiSSoM+). The dataset is comprised of 5,789 human-annotated Reddit posts from LGBTQ+ subreddits. Our approach enables the extraction of hidden linguistic nuances through pretraining on a vast amount of raw data, while also engaging in transductive learning to jointly develop representations for both labeled training data and unlabeled test data. The RoBERTa-GCN model achieved an accuracy of 0.86 and an F1 score of 0.86, surpassing the performance of other baseline models in predicting LGBTQ+ minority stress. Improved prediction of minority stress expressions on social media could lead to digital health interventions to improve the wellbeing of LGBTQ+ people-a community with high rates of stress-sensitive health problems.
• 摘要：被认定为性少数群体和性别少数群体的个人，包括女同性恋、男同性恋、双性恋、跨性别者、酷儿和其他 (LGBTQ+)，比异性恋和顺性别者更有可能遭遇更差的健康状况。造成这些健康差异的一个主要原因是少数群体压力（即 LGBTQ+ 社区在适应主流文化过程中特有的慢性和社会压力源）。这种压力经常出现在 LGBTQ+ 用户在社交媒体平台上的帖子中。然而，这些表达不仅仅是少数群体压力的直接表现。它们涉及语言复杂性（例如习语或词汇多样性），使得许多传统的自然语言处理方法难以检测到它们。在这项工作中，我们设计了一个混合模型，使用图神经网络 (GNN) 和 Transformers 的双向编码器表示 (BERT)，这是一个预先训练的深度语言模型，以提高少数群体压力检测的分类性能。我们在基准社交媒体数据集上对我们的模型进行了实验，以进行少数群体压力检测 (LGBTQ+ MiSSoM+)。该数据集由来自 LGBTQ+ 子版块的 5,789 条人工注释的 Reddit 帖子组成。我们的方法能够通过对大量原始数据进行预训练来提取隐藏的语言细微差别，同时还参与传导学习，共同开发标记训练数据和未标记测试数据的表示。RoBERTa-GCN 模型的准确率达到 0.86，F1 得分达到 0.86，在预测 LGBTQ+ 少数群体压力方面的表现超过了其他基线模型。更好地预测社交媒体上少数群体的压力表达可能会带来数字健康干预，以改善 LGBTQ+ 人群的福祉，该群体的压力敏感型健康问题发生率很高。