2024-10-24

Title: All Entities are Not Created Equal: Examining the Long Tail for Fine-Grained Entity Typing

• Authors: Advait Deshmukh, Ashwin Umadi, Dananjay Srinivas, Maria Leonor Pacheco
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.17355
• Pdf URL: https://arxiv.org/pdf/2410.17355
• Copy Paste: [[2410.17355]] All Entities are Not Created Equal: Examining the Long Tail for Fine-Grained Entity Typing(https://arxiv.org/abs/2410.17355)
• Keywords: language model
• Abstract: Pre-trained language models (PLMs) are trained on large amounts of data, which helps capture world knowledge alongside linguistic competence. Due to this, they are extensively used for ultra-fine entity typing tasks, where they provide the entity knowledge held in its parameter space. Given that PLMs learn from co-occurrence patterns, they likely contain more knowledge or less knowledge about entities depending on their how frequent they are in the pre-training data. In this work, we probe PLMs to elicit encoded entity probabilities and demonstrate that they highly correlate with their frequency in large-scale internet data. Then, we demonstrate that entity-typing approaches that rely on PLMs struggle with entities at the long tail on the distribution. Our findings suggests that we need to go beyond PLMs to produce solutions that perform well for rare, new or infrequent entities.
• 摘要：预训练语言模型 (PLM) 是在大量数据上进行训练的，这有助于在掌握语言能力的同时获取世界知识。正因为如此，它们被广泛用于超精细实体类型化任务，在这些任务中，它们提供参数空间中保存的实体知识。鉴于 PLM 从共现模式中学习，它们可能包含更多或更少的实体知识，具体取决于它们在预训练数据中出现的频率。在这项工作中，我们探索 PLM 以得出编码实体概率，并证明它们与大规模互联网数据中的频率高度相关。然后，我们证明依赖于 PLM 的实体类型化方法在处理分布长尾实体时会遇到困难。我们的研究结果表明，我们需要超越 PLM，为稀有、新或不频繁的实体提供表现良好的解决方案。

Title: AMUSD: Asynchronous Multi-Device Speculative Decoding for LLM Acceleration

• Authors: Bradley McDanel
• Subjects: cs.CL, cs.DC, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.17375
• Pdf URL: https://arxiv.org/pdf/2410.17375
• Copy Paste: [[2410.17375]] AMUSD: Asynchronous Multi-Device Speculative Decoding for LLM Acceleration(https://arxiv.org/abs/2410.17375)
• Keywords: language model, llm
• Abstract: Large language models typically generate tokens autoregressively, using each token as input for the next. Recent work on Speculative Decoding has sought to accelerate this process by employing a smaller, faster draft model to more quickly generate candidate tokens. These candidates are then verified in parallel by the larger (original) verify model, resulting in overall speedup compared to using the larger model by itself in an autoregressive fashion. In this work, we introduce AMUSD (Asynchronous Multi-device Speculative Decoding), a system that further accelerates generation by decoupling the draft and verify phases into a continuous, asynchronous approach. Unlike conventional speculative decoding, where only one model (draft or verify) performs token generation at a time, AMUSD enables both models to perform predictions independently on separate devices (e.g., GPUs). We evaluate our approach over multiple datasets and show that AMUSD achieves an average 29% improvement over speculative decoding and up to 1.96$\times$ speedup over conventional autoregressive decoding, while achieving identical output quality. Our system is open-source and available at this https URL.
• 摘要：大型语言模型通常以自回归方式生成标记，使用每个标记作为下一个标记的输入。推测解码的最新研究试图通过使用更小、更快的草稿模型来更快地生成候选标记，从而加速这一过程。然后，这些候选者由更大的（原始）验证模型并行验证，与以自回归方式单独使用更大的模型相比，总体速度更快。在这项工作中，我们引入了 AMUSD（异步多设备推测解码），该系统通过将草稿和验证阶段解耦为连续的异步方法，进一步加速生成。与传统的推测解码不同，其中只有一个模型（草稿或验证）一次执行标记生成，AMUSD 允许两个模型在单独的设备（例如 GPU）上独立执行预测。我们在多个数据集上评估了我们的方法，并表明 AMUSD 比推测解码平均提高了 29%，比传统的自回归解码提高了 1.96$\times$ 的速度，同时实现了相同的输出质量。我们的系统是开源的，可通过此 https URL 访问。

Title: Do Vision-Language Models Represent Space and How? Evaluating Spatial Frame of Reference Under Ambiguities

• Authors: Zheyuan Zhang, Fengyuan Hu, Jayjun Lee, Freda Shi, Parisa Kordjamshidi, Joyce Chai, Ziqiao Ma
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2410.17385
• Pdf URL: https://arxiv.org/pdf/2410.17385
• Copy Paste: [[2410.17385]] Do Vision-Language Models Represent Space and How? Evaluating Spatial Frame of Reference Under Ambiguities(https://arxiv.org/abs/2410.17385)
• Keywords: language model
• Abstract: Spatial expressions in situated communication can be ambiguous, as their meanings vary depending on the frames of reference (FoR) adopted by speakers and listeners. While spatial language understanding and reasoning by vision-language models (VLMs) have gained increasing attention, potential ambiguities in these models are still under-explored. To address this issue, we present the COnsistent Multilingual Frame Of Reference Test (COMFORT), an evaluation protocol to systematically assess the spatial reasoning capabilities of VLMs. We evaluate nine state-of-the-art VLMs using COMFORT. Despite showing some alignment with English conventions in resolving ambiguities, our experiments reveal significant shortcomings of VLMs: notably, the models (1) exhibit poor robustness and consistency, (2) lack the flexibility to accommodate multiple FoRs, and (3) fail to adhere to language-specific or culture-specific conventions in cross-lingual tests, as English tends to dominate other languages. With a growing effort to align vision-language models with human cognitive intuitions, we call for more attention to the ambiguous nature and cross-cultural diversity of spatial reasoning.
• 摘要：情境交流中的空间表达可能会产生歧义，因为它们的含义会根据说话者和听众所采用的参考框架 (FoR) 而变化。虽然视觉语言模型 (VLM) 的空间语言理解和推理越来越受到关注，但这些模型中的潜在歧义仍未得到充分探索。为了解决这个问题，我们提出了一致的多语言参考框架测试 (COMFORT)，这是一种评估协议，用于系统地评估 VLM 的空间推理能力。我们使用 COMFORT 评估了九种最先进的 VLM。尽管在解决歧义方面表现出与英语惯例的一些一致性，但我们的实验揭示了 VLM 的重大缺陷：值得注意的是，这些模型 (1) 表现出较差的稳健性和一致性，(2) 缺乏适应多个 FoR 的灵活性，以及​​ (3) 在跨语言测试中未能遵守特定语言或特定文化的惯例，因为英语往往主导其他语言。随着人们越来越努力地将视觉语言模型与人类的认知直觉相结合，我们呼吁更多地关注空间推理的模糊性和跨文化多样性。

Title: Scalable Influence and Fact Tracing for Large Language Model Pretraining

• Authors: Tyler A. Chang, Dheeraj Rajagopal, Tolga Bolukbasi, Lucas Dixon, Ian Tenney
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.17413
• Pdf URL: https://arxiv.org/pdf/2410.17413
• Copy Paste: [[2410.17413]] Scalable Influence and Fact Tracing for Large Language Model Pretraining(https://arxiv.org/abs/2410.17413)
• Keywords: language model, llm
• Abstract: Training data attribution (TDA) methods aim to attribute model outputs back to specific training examples, and the application of these methods to large language model (LLM) outputs could significantly advance model transparency and data curation. However, it has been challenging to date to apply these methods to the full scale of LLM pretraining. In this paper, we refine existing gradient-based methods to work effectively at scale, allowing us to retrieve influential examples for an 8B-parameter language model from a pretraining corpus of over 160B tokens with no need for subsampling or pre-filtering. Our method combines several techniques, including optimizer state correction, a task-specific Hessian approximation, and normalized encodings, which we find to be critical for performance at scale. In quantitative evaluations on a fact tracing task, our method performs best at identifying examples that influence model predictions, but classical, model-agnostic retrieval methods such as BM25 still perform better at finding passages which explicitly contain relevant facts. These results demonstrate a misalignment between factual attribution and causal influence. With increasing model size and training tokens, we find that influence more closely aligns with attribution. Finally, we examine different types of examples identified as influential by our method, finding that while many directly entail a particular fact, others support the same output by reinforcing priors on relation types, common entities, and names.
• 摘要：训练数据归因 (TDA) 方法旨在将模型输出归因于特定的训练示例，将这些方法应用于大型语言模型 (LLM) 输出可以显著提高模型透明度和数据管理。然而，迄今为止，将这些方法应用于 LLM 预训练的全规模仍是一项挑战。在本文中，我们改进了现有的基于梯度的方法，使其能够有效地大规模工作，使我们能够从超过 160B 个标记的预训练语料库中检索 8B 参数语言模型的有影响力的示例，而无需进行子采样或预过滤。我们的方法结合了几种技术，包括优化器状态校正、特定于任务的 Hessian 近似和规范化编码，我们发现这些技术对于大规模性能至关重要。在事实追踪任务的定量评估中，我们的方法在识别影响模型预测的示例方面表现最佳，但经典的、与模型无关的检索方法（如 BM25）在查找明确包含相关事实的段落方面仍然表现更好。这些结果表明事实归因和因果影响之间存在不一致。随着模型大小和训练标记的增加，我们发现影响与归因更加一致。最后，我们检查了我们的方法确定为有影响力的不同类型的示例，发现虽然许多示例直接涉及特定事实，但其他示例通过强化关系类型、常见实体和名称的先验来支持相同的输出。

Title: Evaluating AI-Generated Essays with GRE Analytical Writing Assessment

• Authors: Yang Zhong, Jiangang Hao, Michael Fauss, Chen Li, Yuan Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.17439
• Pdf URL: https://arxiv.org/pdf/2410.17439
• Copy Paste: [[2410.17439]] Evaluating AI-Generated Essays with GRE Analytical Writing Assessment(https://arxiv.org/abs/2410.17439)
• Keywords: language model, gpt, llm
• Abstract: The recent revolutionary advance in generative AI enables the generation of realistic and coherent texts by large language models (LLMs). Despite many existing evaluation metrics on the quality of the generated texts, there is still a lack of rigorous assessment of how well LLMs perform in complex and demanding writing assessments. This study examines essays generated by ten leading LLMs for the analytical writing assessment of the Graduate Record Exam (GRE). We assessed these essays using both human raters and the e-rater automated scoring engine as used in the GRE scoring pipeline. Notably, the top-performing GPT-4o received an average score of 4.67, falling between "generally thoughtful, well-developed analysis of the issue and conveys meaning clearly" and "presents a competent analysis of the issue and conveys meaning with acceptable clarity" according to the GRE scoring guideline. We also evaluated the detection accuracy of these essays, with detectors trained on essays generated by the same and different LLMs.
• 摘要：生成式人工智能的最新革命性进步使得大型语言模型 (LLM) 能够生成真实且连贯的文本。尽管目前有许多关于生成文本质量的评估指标，但仍然缺乏对 LLM 在复杂且要求严格的写作评估中表现的严格评估。本研究考察了十所领先的 LLM 为研究生入学考试 (GRE) 的分析性写作评估而生成的论文。我们使用人工评分员和 GRE 评分流程中使用的 e-rater 自动评分引擎来评估这些论文。值得注意的是，表现最好的 GPT-4o 的平均分数为 4.67，根据 GRE 评分指南，介于“对问题进行深思熟虑、完善的分析并清晰地传达含义”和“对问题进行胜任的分析并以可接受的清晰度传达含义”之间。我们还评估了这些论文的检测准确度，检测器使用相同和不同的 LLM 生成的论文进行训练。

Title: In Context Learning and Reasoning for Symbolic Regression with Large Language Models

• Authors: Samiha Sharlin, Tyler R. Josephson
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.17448
• Pdf URL: https://arxiv.org/pdf/2410.17448
• Copy Paste: [[2410.17448]] In Context Learning and Reasoning for Symbolic Regression with Large Language Models(https://arxiv.org/abs/2410.17448)
• Keywords: language model, gpt, llm, prompt, chain-of-thought
• Abstract: Large Language Models (LLMs) are transformer-based machine learning models that have shown remarkable performance in tasks for which they were not explicitly trained. Here, we explore the potential of LLMs to perform symbolic regression -- a machine-learning method for finding simple and accurate equations from datasets. We prompt GPT-4 to suggest expressions from data, which are then optimized and evaluated using external Python tools. These results are fed back to GPT-4, which proposes improved expressions while optimizing for complexity and loss. Using chain-of-thought prompting, we instruct GPT-4 to analyze the data, prior expressions, and the scientific context (expressed in natural language) for each problem before generating new expressions. We evaluated the workflow in rediscovery of five well-known scientific equations from experimental data, and on an additional dataset without a known equation. GPT-4 successfully rediscovered all five equations, and in general, performed better when prompted to use a scratchpad and consider scientific context. We also demonstrate how strategic prompting improves the model's performance and how the natural language interface simplifies integrating theory with data. Although this approach does not outperform established SR programs where target equations are more complex, LLMs can nonetheless iterate toward improved solutions while following instructions and incorporating scientific context in natural language.
• 摘要：大型语言模型 (LLM) 是基于转换器的机器学习模型，在未经明确训练的任务中表现出色。在这里，我们探索了 LLM 执行符号回归的潜力——这是一种从数据集中查找简单准确方程的机器学习方法。我们提示 GPT-4 从数据中提出表达式，然后使用外部 Python 工具对其进行优化和评估。这些结果被反馈给 GPT-4，GPT-4 在优化复杂性和损失的同时提出改进的表达式。使用思路链提示，我们指示 GPT-4 在生成新表达式之前分析每个问题的数据、先前表达式和科学背景（以自然语言表达）。我们在从实验数据中重新发现五个众所周知的科学方程以及在没有已知方程的额外数据集上评估了工作流程。GPT-4 成功地重新发现了所有五个方程，并且总体而言，在提示使用便笺簿并考虑科学背景时表现更好。我们还展示了策略提示如何提高模型的性能，以及自然语言界面如何简化理论与数据的集成。虽然这种方法并不比目标方程更复杂的现有 SR 程序表现更好，但 LLM 仍然可以在遵循指令并在自然语言中融入科学背景的同时迭代以获得更好的解决方案。

Title: Do Robot Snakes Dream like Electric Sheep? Investigating the Effects of Architectural Inductive Biases on Hallucination

• Authors: Jerry Huang, Prasanna Parthasarathi, Mehdi Rezagholizadeh, Boxing Chen, Sarath Chandar
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.17477
• Pdf URL: https://arxiv.org/pdf/2410.17477
• Copy Paste: [[2410.17477]] Do Robot Snakes Dream like Electric Sheep? Investigating the Effects of Architectural Inductive Biases on Hallucination(https://arxiv.org/abs/2410.17477)
• Keywords: language model, llm, hallucination
• Abstract: The growth in prominence of large language models (LLMs) in everyday life can be largely attributed to their generative abilities, yet some of this is also owed to the risks and costs associated with their use. On one front is their tendency to \textit{hallucinate} false or misleading information, limiting their reliability. On another is the increasing focus on the computational limitations associated with traditional self-attention based LLMs, which has brought about new alternatives, in particular recurrent models, meant to overcome them. Yet it remains uncommon to consider these two concerns simultaneously. Do changes in architecture exacerbate/alleviate existing concerns about hallucinations? Do they affect how and where they occur? Through an extensive evaluation, we study how these architecture-based inductive biases affect the propensity to hallucinate. While hallucination remains a general phenomenon not limited to specific architectures, the situations in which they occur and the ease with which specific types of hallucinations can be induced can significantly differ based on the model architecture. These findings highlight the need for better understanding both these problems in conjunction with each other, as well as consider how to design more universal techniques for handling hallucinations.
• 摘要：大型语言模型 (LLM) 在日常生活中的重要性日益提高，这在很大程度上可以归因于它们的生成能力，但其中一部分也归因于使用它们的风险和成本。一方面，它们倾向于产生虚假或误导性的信息，从而限制了它们的可靠性。另一方面，人们越来越关注与传统的基于自我注意的 LLM 相关的计算限制，这带来了新的替代方案，特别是旨在克服这些限制的循环模型。然而，同时考虑这两个问题仍然很少见。架构的变化是否会加剧/减轻人们对幻觉的现有担忧？它们会影响幻觉发生的方式和地点吗？通过广泛的评估，我们研究了这些基于架构的归纳偏差如何影响产生幻觉的倾向。虽然幻觉仍然是一种普遍现象，并不局限于特定的架构，但它们发生的情况以及特定类型幻觉的诱发难易程度可能会因模型架构的不同而有很大差异。这些发现强调了更好地理解这两个问题的必要性，并考虑如何设计更通用的技术来处理幻觉。

Title: Is artificial intelligence still intelligence? LLMs generalize to novel adjective-noun pairs, but don't mimic the full human distribution

• Authors: Hayley Ross, Kathryn Davidson, Najoung Kim
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.17482
• Pdf URL: https://arxiv.org/pdf/2410.17482
• Copy Paste: [[2410.17482]] Is artificial intelligence still intelligence? LLMs generalize to novel adjective-noun pairs, but don't mimic the full human distribution(https://arxiv.org/abs/2410.17482)
• Keywords: llm
• Abstract: Inferences from adjective-noun combinations like "Is artificial intelligence still intelligence?" provide a good test bed for LLMs' understanding of meaning and compositional generalization capability, since there are many combinations which are novel to both humans and LLMs but nevertheless elicit convergent human judgments. We study a range of LLMs and find that the largest models we tested are able to draw human-like inferences when the inference is determined by context and can generalize to unseen adjective-noun combinations. We also propose three methods to evaluate LLMs on these inferences out of context, where there is a distribution of human-like answers rather than a single correct answer. We find that LLMs show a human-like distribution on at most 75\% of our dataset, which is promising but still leaves room for improvement.
• 摘要：形容词-名词组合的推论，例如“人工智能仍然是智能吗？”，为 LLM 对意义的理解和组合泛化能力提供了一个很好的测试平台，因为有许多组合对人类和 LLM 来说都是新颖的，但仍然会引发人类的趋同判断。我们研究了一系列 LLM，发现我们测试的最大模型能够在由上下文确定的推论时得出类似人类的推论，并且可以推广到未见过的形容词-名词组合。我们还提出了三种方法来评估 LLM 在脱离上下文的情况下对这些推论的准确性，其中存在类似人类的答案分布，而不是单一的正确答案。我们发现 LLM 在我们数据集的最多 75% 上显示出类似人类的分布，这是有希望的，但仍有改进的空间。

Title: VoiceTextBlender: Augmenting Large Language Models with Speech Capabilities via Single-Stage Joint Speech-Text Supervised Fine-Tuning

• Authors: Yifan Peng, Krishna C. Puvvada, Zhehuai Chen, Piotr Zelasko, He Huang, Kunal Dhawan, Ke Hu, Shinji Watanabe, Jagadeesh Balam, Boris Ginsburg
• Subjects: cs.CL, eess.AS
• Abstract URL: https://arxiv.org/abs/2410.17485
• Pdf URL: https://arxiv.org/pdf/2410.17485
• Copy Paste: [[2410.17485]] VoiceTextBlender: Augmenting Large Language Models with Speech Capabilities via Single-Stage Joint Speech-Text Supervised Fine-Tuning(https://arxiv.org/abs/2410.17485)
• Keywords: language model, llm, prompt
• Abstract: Recent studies have augmented large language models (LLMs) with speech capabilities, leading to the development of speech language models (SpeechLMs). Earlier SpeechLMs focused on single-turn speech-based question answering (QA), where user input comprised a speech context and a text question. More recent studies have extended this to multi-turn conversations, though they often require complex, multi-stage supervised fine-tuning (SFT) with diverse data. Another critical challenge with SpeechLMs is catastrophic forgetting-where models optimized for speech tasks suffer significant degradation in text-only performance. To mitigate these issues, we propose a novel single-stage joint speech-text SFT approach on the low-rank adaptation (LoRA) of the LLM backbone. Our joint SFT combines text-only SFT data with three types of speech-related data: speech recognition and translation, speech-based QA, and mixed-modal SFT. Compared to previous SpeechLMs with 7B or 13B parameters, our 3B model demonstrates superior performance across various speech benchmarks while preserving the original capabilities on text-only tasks. Furthermore, our model shows emergent abilities of effectively handling previously unseen prompts and tasks, including multi-turn, mixed-modal inputs.
• 摘要：最近的研究为大型语言模型 (LLM) 增加了语音功能，从而推动了语音语言模型 (SpeechLM) 的发展。早期的 SpeechLM 专注于单轮语音问答 (QA)，其中用户输入包括语音上下文和文本问题。最近的研究将其扩展到多轮对话，尽管它们通常需要使用各种数据进行复杂的多阶段监督微调 (SFT)。SpeechLM 的另一个关键挑战是灾难性遗忘 - 针对语音任务优化的模型在纯文本性能方面会显著下降。为了缓解这些问题，我们在 LLM 主干的低秩自适应 (LoRA) 上提出了一种新颖的单阶段联合语音文本 SFT 方法。我们的联合 SFT 将纯文本 SFT 数据与三种类型的语音相关数据相结合：语音识别和翻译、基于语音的 QA 和混合模式 SFT。与之前具有 7B 或 13B 参数的 SpeechLM 相比，我们的 3B 模型在各种语音基准测试中均表现出色，同时保留了纯文本任务的原始能力。此外，我们的模型还展示了有效处理以前从未见过的提示和任务（包括多轮混合模式输入）的新兴能力。

Title: Large Language Models Still Exhibit Bias in Long Text

• Authors: Wonje Jeung, Dongjae Jeon, Ashkan Yousefpour, Jonghyun Choi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.17519
• Pdf URL: https://arxiv.org/pdf/2410.17519
• Copy Paste: [[2410.17519]] Large Language Models Still Exhibit Bias in Long Text(https://arxiv.org/abs/2410.17519)
• Keywords: language model, gpt, llm, prompt
• Abstract: Existing fairness benchmarks for large language models (LLMs) primarily focus on simple tasks, such as multiple-choice questions, overlooking biases that may arise in more complex scenarios like long-text generation. To address this gap, we introduce the Long Text Fairness Test (LTF-TEST), a framework that evaluates biases in LLMs through essay-style prompts. LTF-TEST covers 14 topics and 10 demographic axes, including gender and race, resulting in 11,948 samples. By assessing both model responses and the reasoning behind them, LTF-TEST uncovers subtle biases that are difficult to detect in simple responses. In our evaluation of five recent LLMs, including GPT-4o and LLaMa3, we identify two key patterns of bias. First, these models frequently favor certain demographic groups in their responses. Second, they show excessive sensitivity toward traditionally disadvantaged groups, often providing overly protective responses while neglecting others. To mitigate these biases, we propose FT-REGARD, a finetuning approach that pairs biased prompts with neutral responses. FT-REGARD reduces gender bias by 34.6% and improves performance by 1.4 percentage points on the BBQ benchmark, offering a promising approach to addressing biases in long-text generation tasks.
• 摘要：现有的大型语言模型 (LLM) 公平性基准主要侧重于简单任务，例如多项选择题，而忽略了在长文本生成等更复杂场景中可能出现的偏见。为了解决这一差距，我们引入了长文本公平性测试 (LTF-TEST)，这是一个通过论文式提示评估 LLM 中偏见的框架。LTF-TEST 涵盖 14 个主题和 10 个人口统计轴，包括性别和种族，共计 11,948 个样本。通过评估模型响应及其背后的原因，LTF-TEST 发现了在简单响应中难以发现的细微偏见。在我们对包括 GPT-4o 和 LLaMa3 在内的五个最近的 LLM 的评估中，我们发现了两种关键的偏见模式。首先，这些模型在响应中经常偏向某些人口群体。其次，它们对传统上处于弱势的群体表现出过度的敏感性，经常提供过度保护的响应而忽略其他群体。为了减轻这些偏见，我们提出了 FT-REGARD，这是一种将有偏见的提示与中性响应配对的微调方法。 FT-REGARD 将性别偏见降低了 34.6%，并在 BBQ 基准上将性能提高了 1.4 个百分点，为解决长文本生成任务中的偏见提供了一种有前途的方法。

Title: Navigate Complex Physical Worlds via Geometrically Constrained LLM

• Authors: Yongqiang Huang, Wentao Ye, Liyao Li, Junbo Zhao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.17529
• Pdf URL: https://arxiv.org/pdf/2410.17529
• Copy Paste: [[2410.17529]] Navigate Complex Physical Worlds via Geometrically Constrained LLM(https://arxiv.org/abs/2410.17529)
• Keywords: language model, llm, agent
• Abstract: This study investigates the potential of Large Language Models (LLMs) for reconstructing and constructing the physical world solely based on textual knowledge. It explores the impact of model performance on spatial understanding abilities. To enhance the comprehension of geometric and spatial relationships in the complex physical world, the study introduces a set of geometric conventions and develops a workflow based on multi-layer graphs and multi-agent system frameworks. It examines how LLMs achieve multi-step and multi-objective geometric inference in a spatial environment using multi-layer graphs under unified geometric conventions. Additionally, the study employs a genetic algorithm, inspired by large-scale model knowledge, to solve geometric constraint problems. In summary, this work innovatively explores the feasibility of using text-based LLMs as physical world builders and designs a workflow to enhance their capabilities.
• 摘要：本研究探讨了大型语言模型 (LLM) 仅基于文本知识重建和构建物理世界的潜力。它探讨了模型性能对空间理解能力的影响。为了增强对复杂物理世界中几何和空间关系的理解，该研究引入了一组几何约定，并开发了基于多层图和多智能体系统框架的工作流程。它研究了 LLM 如何在统一的几何约定下使用多层图在空间环境中实现多步骤和多目标几何推理。此外，该研究采用了一种受大规模模型知识启发的遗传算法来解决几何约束问题。总之，这项工作创新性地探索了使用基于文本的 LLM 作为物理世界构建器的可行性，并设计了一个工作流程来增强其能力。

Title: Responsible Multilingual Large Language Models: A Survey of Development, Applications, and Societal Impact

• Authors: Junhua Liu, Bin Fu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.17532
• Pdf URL: https://arxiv.org/pdf/2410.17532
• Copy Paste: [[2410.17532]] Responsible Multilingual Large Language Models: A Survey of Development, Applications, and Societal Impact(https://arxiv.org/abs/2410.17532)
• Keywords: language model, llm
• Abstract: Multilingual Large Language Models (MLLMs) represent a pivotal advancement in democratizing artificial intelligence across linguistic boundaries. While theoretical foundations are well-established, practical implementation guidelines remain scattered. This work bridges this gap by providing a comprehensive end-to-end framework for developing and deploying MLLMs in production environments. We make three distinctive contributions: First, we present an actionable pipeline from data pre-processing through deployment, integrating insights from academic research and industrial applications. Second, using Llama2 as a case study, we provide detailed optimization strategies for enhancing multilingual capabilities, including curriculum learning approaches for balancing high-resource and low-resource languages, tokenization strategies, and effective sampling methods. Third, we offer an interdisciplinary analysis that considers technical, linguistic, and cultural perspectives in MLLM development. Our findings reveal critical challenges in supporting linguistic diversity, with 88.38% of world languages categorized as low-resource, affecting over a billion speakers. We examine practical solutions through real-world applications in customer service, search engines, and machine translation. By synthesizing theoretical frameworks with production-ready implementation strategies, this survey provides essential guidance for practitioners and researchers working to develop more inclusive and effective multilingual AI systems.
• 摘要：多语言大型语言模型 (MLLM) 代表了跨语言界限实现人工智能民主化的关键进步。虽然理论基础已经很完善，但实际实施指南仍然很分散。这项工作通过提供全面的端到端框架来弥合这一差距，用于在生产环境中开发和部署 MLLM。我们做出了三项独特的贡献：首先，我们展示了从数据预处理到部署的可操作流程，整合了学术研究和工业应用的见解。其次，以 Llama2 为例，我们提供了增强多语言能力的详细优化策略，包括平衡高资源和低资源语言的课程学习方法、标记化策略和有效的采样方法。第三，我们提供了一个跨学科分析，考虑了 MLLM 开发中的技术、语言和文化视角。我们的研究结果揭示了支持语言多样性的关键挑战，88.38% 的世界语言被归类为低资源语言，影响了超过十亿的使用者。我们通过客户服务、搜索引擎和机器翻译中的实际应用来研究实用解决方案。通过将理论框架与可用于生产的实施策略相结合，本调查为致力于开发更具包容性和有效性的多语言人工智能系统的从业者和研究人员提供了重要的指导。

Title: ESpeW: Robust Copyright Protection for LLM-based EaaS via Embedding-Specific Watermark

• Authors: Zongqi Wang, Baoyuan Wu, Jingyuan Deng, Yujiu Yang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.17552
• Pdf URL: https://arxiv.org/pdf/2410.17552
• Copy Paste: [[2410.17552]] ESpeW: Robust Copyright Protection for LLM-based EaaS via Embedding-Specific Watermark(https://arxiv.org/abs/2410.17552)
• Keywords: llm
• Abstract: Embeddings as a Service (EaaS) is emerging as a crucial role in AI applications. Unfortunately, EaaS is vulnerable to model extraction attacks, highlighting the urgent need for copyright this http URL some preliminary works propose applying embedding watermarks to protect EaaS, recent research reveals that these watermarks can be easily removed. Hence, it is crucial to inject robust watermarks resistant to watermark removal this http URL watermarking methods typically inject a target embedding into embeddings through linear interpolation when the text contains triggers. However, this mechanism results in each watermarked embedding having the same component, which makes the watermark easy to identify and this http URL by this, in this paper, we propose a novel embedding-specific watermarking (ESpeW) mechanism to offer robust copyright protection for EaaS. Our approach involves injecting unique, yet readily identifiable watermarks into each embedding. Watermarks inserted by ESpeW are designed to maintain a significant distance from one another and to avoid sharing common components, thus making it significantly more challenging to remove the this http URL experiments on four popular datasets demonstrate that ESpeW can even watermark successfully against a highly aggressive removal strategy without sacrificing the quality of embeddings.
• 摘要：嵌入即服务 (EaaS) 正在成为 AI 应用中的关键角色。不幸的是，EaaS 容易受到模型提取攻击，这凸显了对版权的迫切需求。一些初步研究提出应用嵌入水印来保护 EaaS，最近的研究表明这些水印很容易被删除。因此，注入能够抵抗水印删除的鲁棒水印至关重要。当文本包含触发器时，水印方法通常通过线性插值将目标嵌入注入嵌入中。然而，这种机制导致每个带水印的嵌入都有相同的组件，这使得水印易于识别，因此，在本文中，我们提出了一种新颖的嵌入特定水印 (ESpeW) ​​机制，为 EaaS 提供强大的版权保护。我们的方法包括在每个嵌入中注入唯一但易于识别的水印。 ESpeW 插入的水印旨在彼此保持相当大的距离并避免共享公共组件，从而使删除此 http URL 变得更具挑战性，在四个流行数据集上的实验表明，ESpeW 甚至可以在不牺牲嵌入质量的情况下成功地针对高度激进的删除策略添加水印。

Title: MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Models

• Authors: Guijin Son, Dongkeun Yoon, Juyoung Suk, Javier Aula-Blasco, Mano Aslan, Vu Trong Kim, Shayekh Bin Islam, Jaume Prats-Cristià, Lucía Tormo-Bañuelos, Seungone Kim
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.17578
• Pdf URL: https://arxiv.org/pdf/2410.17578
• Copy Paste: [[2410.17578]] MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Models(https://arxiv.org/abs/2410.17578)
• Keywords: language model, llm, hallucination
• Abstract: Large language models (LLMs) are commonly used as evaluators in tasks (e.g., reward modeling, LLM-as-a-judge), where they act as proxies for human preferences or judgments. This leads to the need for meta-evaluation: evaluating the credibility of LLMs as evaluators. However, existing benchmarks primarily focus on English, offering limited insight into LLMs' effectiveness as evaluators in non-English contexts. To address this, we introduce MM-Eval, a multilingual meta-evaluation benchmark that covers 18 languages across six categories. MM-Eval evaluates various dimensions, including language-specific challenges like linguistics and language hallucinations. Evaluation results show that both proprietary and open-source language models have considerable room for improvement. Further analysis reveals a tendency for these models to assign middle-ground scores to low-resource languages. We publicly release our benchmark and code.
• 摘要：大型语言模型 (LLM) 通常用作任务（例如奖励建模、LLM 作为法官）中的评估器，在这些任务中，它们充当人类偏好或判断的代理。这导致了对元评估的需求：评估 LLM 作为评估器的可信度。然而，现有的基准主要侧重于英语，对 LLM 作为非英语环境中的评估器的有效性提供有限的见解。为了解决这个问题，我们推出了 MM-Eval，这是一个多语言元评估基准，涵盖六个类别的 18 种语言。MM-Eval 评估各种维度，包括语言学和语言幻觉等特定于语言的挑战。评估结果表明，专有和开源语言模型都有很大的改进空间。进一步的分析表明，这些模型倾向于为资源匮乏的语言分配中等分数。我们公开发布了我们的基准和代码。

Title: Cross-model Control: Improving Multiple Large Language Models in One-time Training

• Authors: Jiayi Wu, Hao Sun, Hengyi Cai, Lixin Su, Shuaiqiang Wang, Dawei Yin, Xiang Li, Ming Gao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.17599
• Pdf URL: https://arxiv.org/pdf/2410.17599
• Copy Paste: [[2410.17599]] Cross-model Control: Improving Multiple Large Language Models in One-time Training(https://arxiv.org/abs/2410.17599)
• Keywords: language model, llm
• Abstract: The number of large language models (LLMs) with varying parameter scales and vocabularies is increasing. While they deliver powerful performance, they also face a set of common optimization needs to meet specific requirements or standards, such as instruction following or avoiding the output of sensitive information from the real world. However, how to reuse the fine-tuning outcomes of one model to other models to reduce training costs remains a challenge. To bridge this gap, we introduce Cross-model Control (CMC), a method that improves multiple LLMs in one-time training with a portable tiny language model. Specifically, we have observed that the logit shift before and after fine-tuning is remarkably similar across different models. Based on this insight, we incorporate a tiny language model with a minimal number of parameters. By training alongside a frozen template LLM, the tiny model gains the capability to alter the logits output by the LLMs. To make this tiny language model applicable to models with different vocabularies, we propose a novel token mapping strategy named PM-MinED. We have conducted extensive experiments on instruction tuning and unlearning tasks, demonstrating the effectiveness of CMC. Our code is available at this https URL.
• 摘要：具有不同参数规模和词汇量的大型语言模型 (LLM) 的数量正在增加。虽然它们提供了强大的性能，但它们也面临着一系列常见的优化需求，以满足特定的要求或标准，例如遵循指令或避免输出来自现实世界的敏感信息。然而，如何将一个模型的微调结果重用到其他模型以降低训练成本仍然是一个挑战。为了弥补这一差距，我们引入了跨模型控制 (CMC)，这是一种使用可移植的微型语言模型在一次训练中改进多个 LLM 的方法。具体来说，我们观察到微调前后的逻辑偏移在不同模型之间非常相似。基于这一见解，我们整合了一个具有最少参数的微型语言模型。通过与冻结模板 LLM 一起训练，微型模型获得了改变 LLM 输出的逻辑的能力。为了使这个微型语言模型适用于具有不同词汇量的模型，我们提出了一种名为 PM-MinED 的新型标记映射策略。我们对指令调整和反学习任务进行了广泛的实验，证明了 CMC 的有效性。我们的代码可在此 https URL 上找到。

Title: Graphusion: A RAG Framework for Knowledge Graph Construction with a Global Perspective

• Authors: Rui Yang, Boming Yang, Aosong Feng, Sixun Ouyang, Moritz Blum, Tianwei She, Yuang Jiang, Freddy Lecue, Jinghui Lu, Irene Li
• Subjects: cs.CL, cs.AI, cs.DB
• Abstract URL: https://arxiv.org/abs/2410.17600
• Pdf URL: https://arxiv.org/pdf/2410.17600
• Copy Paste: [[2410.17600]] Graphusion: A RAG Framework for Knowledge Graph Construction with a Global Perspective(https://arxiv.org/abs/2410.17600)
• Keywords: language model, llm
• Abstract: Knowledge Graphs (KGs) are crucial in the field of artificial intelligence and are widely used in downstream tasks, such as question-answering (QA). The construction of KGs typically requires significant effort from domain experts. Large Language Models (LLMs) have recently been used for Knowledge Graph Construction (KGC). However, most existing approaches focus on a local perspective, extracting knowledge triplets from individual sentences or documents, missing a fusion process to combine the knowledge in a global KG. This work introduces Graphusion, a zero-shot KGC framework from free text. It contains three steps: in Step 1, we extract a list of seed entities using topic modeling to guide the final KG includes the most relevant entities; in Step 2, we conduct candidate triplet extraction using LLMs; in Step 3, we design the novel fusion module that provides a global view of the extracted knowledge, incorporating entity merging, conflict resolution, and novel triplet discovery. Results show that Graphusion achieves scores of 2.92 and 2.37 out of 3 for entity extraction and relation recognition, respectively. Moreover, we showcase how Graphusion could be applied to the Natural Language Processing (NLP) domain and validate it in an educational scenario. Specifically, we introduce TutorQA, a new expert-verified benchmark for QA, comprising six tasks and a total of 1,200 QA pairs. Using the Graphusion-constructed KG, we achieve a significant improvement on the benchmark, for example, a 9.2% accuracy improvement on sub-graph completion.
• 摘要：知识图谱 (KG) 在人工智能领域至关重要，并广泛应用于下游任务，例如问答 (QA)。构建 KG 通常需要领域专家付出大量努力。大型语言模型 (LLM) 近期已用于知识图谱构建 (KGC)。然而，大多数现有方法都侧重于局部视角，从单个句子或文档中提取知识三元组，缺少将知识组合到全局 KG 中的融合过程。这项工作引入了 Graphusion，这是一个来自自由文本的零样本 KGC 框架。它包含三个步骤：在步骤 1 中，我们使用主题建模提取种子实体列表以指导最终的 KG 包含最相关的实体；在步骤 2 中，我们使用 LLM 进行候选三元组提取；在步骤 3 中，我们设计了新颖的融合模块，该模块提供提取知识的全局视图，结合了实体合并、冲突解决和新颖的三元组发现。结果表明，Graphusion 在实体提取和关系识别方面分别获得了 2.92 和 2.37（满分 3 分）。此外，我们还展示了如何将 Graphusion 应用于自然语言处理 (NLP) 领域，并在教育场景中对其进行了验证。具体来说，我们引入了 TutorQA，这是一个新的专家验证的 QA 基准，包含六个任务和总共 1,200 个 QA 对。使用 Graphusion 构建的 KG，我们在基准上取得了显著的改进，例如，子图完成的准确率提高了 9.2%。

Title: LMLPA: Language Model Linguistic Personality Assessment

• Authors: Jingyao Zheng, Xian Wang, Simo Hosio, Xiaoxian Xu, Lik-Hang Lee
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.17632
• Pdf URL: https://arxiv.org/pdf/2410.17632
• Copy Paste: [[2410.17632]] LMLPA: Language Model Linguistic Personality Assessment(https://arxiv.org/abs/2410.17632)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) are increasingly used in everyday life and research. One of the most common use cases is conversational interactions, enabled by the language generation capabilities of LLMs. Just as between two humans, a conversation between an LLM-powered entity and a human depends on the personality of the conversants. However, measuring the personality of a given LLM is currently a challenge. This paper introduces the Language Model Linguistic Personality Assessment (LMLPA), a system designed to evaluate the linguistic personalities of LLMs. Our system helps to understand LLMs' language generation capabilities by quantitatively assessing the distinct personality traits reflected in their linguistic outputs. Unlike traditional human-centric psychometrics, the LMLPA adapts a personality assessment questionnaire, specifically the Big Five Inventory, to align with the operational capabilities of LLMs, and also incorporates the findings from previous language-based personality measurement literature. To mitigate sensitivity to the order of options, our questionnaire is designed to be open-ended, resulting in textual answers. Thus, the AI rater is needed to transform ambiguous personality information from text responses into clear numerical indicators of personality traits. Utilising Principal Component Analysis and reliability validations, our findings demonstrate that LLMs possess distinct personality traits that can be effectively quantified by the LMLPA. This research contributes to Human-Computer Interaction and Human-Centered AI, providing a robust framework for future studies to refine AI personality assessments and expand their applications in multiple areas, including education and manufacturing.
• 摘要：大型语言模型 (LLM) 在日常生活和研究中的应用越来越广泛。最常见的用例之一是对话交互，这得益于 LLM 的语言生成功能。就像两个人之间一样，LLM 驱动的实体与人类之间的对话取决于对话者的个性。然而，衡量特定 LLM 的个性目前是一项挑战。本文介绍了语言模型语言人格评估 (LMLPA)，该系统旨在评估 LLM 的语言个性。我们的系统通过定量评估其语言输出中反映的独特个性特征来帮助了解 LLM 的语言生成能力。与传统的以人为中心的心理测量法不同，LMLPA 调整了人格评估问卷，特别是大五量表，以与 LLM 的操作能力保持一致，并结合了以前基于语言的个性测量文献中的发现。为了减轻对选项顺序的敏感性，我们的问卷设计为开放式，从而得到文本答案。因此，需要 AI 评估员将文本响应中模糊的个性信息转换为个性特征的清晰数字指标。利用主成分分析和可靠性验证，我们的研究结果表明 LLM 具有独特的个性特征，可以通过 LMLPA 有效量化。这项研究有助于人机交互和以人为中心的 AI，为未来研究提供了一个强大的框架，以完善 AI 个性评估并扩大其在教育和制造业等多个领域的应用。

Title: ReflecTool: Towards Reflection-Aware Tool-Augmented Clinical Agents

• Authors: Yusheng Liao, Shuyang Jiang, Yanfeng Wang, Yu Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.17657
• Pdf URL: https://arxiv.org/pdf/2410.17657
• Copy Paste: [[2410.17657]] ReflecTool: Towards Reflection-Aware Tool-Augmented Clinical Agents(https://arxiv.org/abs/2410.17657)
• Keywords: language model, llm, agent
• Abstract: Large Language Models (LLMs) have shown promising potential in the medical domain, assisting with tasks like clinical note generation and patient communication. However, current LLMs are limited to text-based communication, hindering their ability to interact with diverse forms of information in clinical environments. Despite clinical agents succeeding in diverse signal interaction, they are oriented to a single clinical scenario and hence fail for broader applications. To evaluate clinical agents holistically, we propose ClinicalAgent Bench~(CAB), a comprehensive medical agent benchmark consisting of 18 tasks across five key realistic clinical dimensions. Building on this, we introduce ReflecTool, a novel framework that excels at utilizing domain-specific tools within two stages. The first optimization stage progressively enlarges a long-term memory by saving successful solving processes and tool-wise experience of agents in a tiny pre-defined training set. In the following inference stage, ReflecTool can search for supportive successful demonstrations from already built long-term memory to guide the tool selection strategy, and a verifier improves the tool usage according to the tool-wise experience with two verification methods--iterative refinement and candidate selection. Extensive experiments on ClinicalAgent Benchmark demonstrate that ReflecTool surpasses the pure LLMs with more than 10 points and the well-established agent-based methods with 3 points, highlighting its adaptability and effectiveness in solving complex clinical tasks.
• 摘要：大型语言模型 (LLM) 在医学领域显示出巨大的潜力，可帮助完成临床笔记生成和患者沟通等任务。然而，目前的 LLM 仅限于基于文本的通信，这阻碍了它们与临床环境中各种形式的信息进行交互的能力。尽管临床代理在各种信号交互中都取得了成功，但它们面向单一的临床场景，因此无法应用于更广泛的领域。为了全面评估临床代理，我们提出了 ClinicalAgent Bench~(CAB)，这是一个全面的医疗代理基准，由五个关键现实临床维度的 18 项任务组成。在此基础上，我们引入了 ReflecTool，这是一个新颖的框架，擅长在两个阶段内利用特定领域的工具。第一个优化阶段通过在一个微小的预定义训练集中保存成功的求解过程和代理的工具经验来逐步扩大长期记忆。在接下来的推理阶段，ReflecTool 可以从已建立的长期记忆中寻找支持性的成功演示来指导工具选择策略，验证者则根据工具使用经验，通过迭代改进和候选选择两种验证方法改进工具使用。在 ClinicalAgent Benchmark 上的大量实验表明，ReflecTool 以超过 10 分的优势超越了纯 LLM，以超过 3 分的优势超越了成熟的基于代理的方法，凸显了其在解决复杂临床任务中的适应性和有效性。

Title: Quantifying the Risks of Tool-assisted Rephrasing to Linguistic Diversity

• Authors: Mengying Wang, Andreas Spitz
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.17670
• Pdf URL: https://arxiv.org/pdf/2410.17670
• Copy Paste: [[2410.17670]] Quantifying the Risks of Tool-assisted Rephrasing to Linguistic Diversity(https://arxiv.org/abs/2410.17670)
• Keywords: language model
• Abstract: Writing assistants and large language models see widespread use in the creation of text content. While their effectiveness for individual users has been evaluated in the literature, little is known about their proclivity to change language or reduce its richness when adopted by a large user base. In this paper, we take a first step towards quantifying this risk by measuring the semantic and vocabulary change enacted by the use of rephrasing tools on a multi-domain corpus of human-generated text.
• 摘要：写作助手和大型语言模型在文本内容创建中得到广泛应用。虽然文献中已经评估了它们对个人用户的有效性，但人们对它们在被大量用户采用时改变语言或降低语言丰富度的倾向知之甚少。在本文中，我们通过测量在多领域人工生成文本语料库上使用改写工具所产生的语义和词汇变化，迈出了量化这种风险的第一步。

Title: Beware of Calibration Data for Pruning Large Language Models

• Authors: Yixin Ji, Yang Xiang, Juntao Li, Qingrong Xia, Ping Li, Xinyu Duan, Zhefeng Wang, Min Zhang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.17711
• Pdf URL: https://arxiv.org/pdf/2410.17711
• Copy Paste: [[2410.17711]] Beware of Calibration Data for Pruning Large Language Models(https://arxiv.org/abs/2410.17711)
• Keywords: language model, llm
• Abstract: As large language models (LLMs) are widely applied across various fields, model compression has become increasingly crucial for reducing costs and improving inference efficiency. Post-training pruning is a promising method that does not require resource-intensive iterative training and only needs a small amount of calibration data to assess the importance of parameters. Previous research has primarily focused on designing advanced pruning methods, while different calibration data's impact on pruning performance still lacks systematical exploration. We fill this blank and surprisingly observe that the effects of calibration data even value more than designing advanced pruning strategies, especially for high sparsity. Our preliminary exploration also discloses that using calibration data similar to the training data can yield better performance. As pre-training data is usually inaccessible for advanced LLMs, we further provide a self-generating calibration data synthesis strategy to construct feasible calibration data. We conduct experiments on the recent strong open-source LLMs (e.g., DCLM, and LLaMA-3), and the results show that the proposed method outperforms commonly used calibration data and can effectively enhance strong pruning methods (e.g., Wanda, OWL).
• 摘要：随着大型语言模型 (LLM) 广泛应用于各个领域，模型压缩对于降低成本和提高推理效率变得越来越重要。训练后剪枝是一种很有前途的方法，它不需要资源密集型的迭代训练，只需要少量的校准数据来评估参数的重要性。以前的研究主要集中于设计高级剪枝方法，而不同校准数据对剪枝性能的影响仍然缺乏系统的探索。我们填补了这一空白，并惊讶地发现校准数据的效果甚至比设计高级剪枝策略更有价值，特别是对于高稀疏性。我们的初步探索还表明，使用与训练数据相似的校准数据可以获得更好的性能。由于高级 LLM 通常无法获得预训练数据，我们进一步提供了一种自生成的校准数据合成策略来构建可行的校准数据。我们对近期强大的开源 LLM（例如 DCLM 和 LLaMA-3）进行了实验，结果表明，所提出的方法优于常用的校准数据，并且可以有效增强强剪枝方法（例如 Wanda、OWL）。

Title: CogSteer: Cognition-Inspired Selective Layer Intervention for Efficient Semantic Steering in Large Language Models

• Authors: Xintong Wang, Jingheng Pan, Longqin Jiang, Liang Ding, Xingshan Li, Chris Biemann
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.17714
• Pdf URL: https://arxiv.org/pdf/2410.17714
• Copy Paste: [[2410.17714]] CogSteer: Cognition-Inspired Selective Layer Intervention for Efficient Semantic Steering in Large Language Models(https://arxiv.org/abs/2410.17714)
• Keywords: language model, llm
• Abstract: Despite their impressive capabilities, large language models (LLMs) often lack interpretability and can generate toxic content. While using LLMs as foundation models and applying semantic steering methods are widely practiced, we believe that efficient methods should be based on a thorough understanding of LLM behavior. To this end, we propose using eye movement measures to interpret LLM behavior across layers. We find that LLMs exhibit patterns similar to human gaze across layers and different layers function differently. Inspired by these findings, we introduce a heuristic steering layer selection and apply it to layer intervention methods via fine-tuning and inference. Using language toxification and detoxification as test beds, we demonstrate that our proposed CogSteer methods achieve better results in terms of toxicity scores while efficiently saving 97% of the computational resources and 60% of the training time. Our model-agnostic approach can be adopted into various LLMs, contributing to their interpretability and promoting trustworthiness for safe deployment.
• 摘要：尽管大型语言模型 (LLM) 功能强大，但它们通常缺乏可解释性，并且会产生有害内容。虽然使用 LLM 作为基础模型并应用语义控制方法已被广泛实践，但我们认为有效的方法应该基于对 LLM 行为的透彻理解。为此，我们建议使用眼动测量来解释跨层的 LLM 行为。我们发现 LLM 表现出与跨层的人类凝视相似的模式，并且不同层的功能不同。受这些发现的启发，我们引入了启发式控制层选择，并通过微调和推理将其应用于层干预方法。使用语言毒化和解毒作为试验台，我们证明我们提出的 CogSteer 方法在毒性评分方面取得了更好的结果，同时有效地节省了 97% 的计算资源和 60% 的训练时间。我们的模型无关方法可以采用到各种 LLM 中，有助于提高它们的可解释性并提高可信度以实现安全部署。

Title: MojoBench: Language Modeling and Benchmarks for Mojo

• Authors: Nishat Raihan, Joanna C. S. Santos, Marcos Zampieri
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.17736
• Pdf URL: https://arxiv.org/pdf/2410.17736
• Copy Paste: [[2410.17736]] MojoBench: Language Modeling and Benchmarks for Mojo(https://arxiv.org/abs/2410.17736)
• Keywords: language model, gpt, llm
• Abstract: The recently introduced Mojo programming language (PL) by Modular, has received significant attention in the scientific community due to its claimed significant speed boost over Python. Despite advancements in code Large Language Models (LLMs) across various PLs, Mojo remains unexplored in this context. To address this gap, we introduce MojoBench, the first framework for Mojo code generation. MojoBench includes HumanEval-Mojo, a benchmark dataset designed for evaluating code LLMs on Mojo, and Mojo-Coder, the first LLM pretrained and finetuned for Mojo code generation, which supports instructions in 5 natural languages (NLs). Our results show that Mojo-Coder achieves a 30-35% performance improvement over leading models like GPT-4o and Claude-3.5-Sonnet. Furthermore, we provide insights into LLM behavior with underrepresented and unseen PLs, offering potential strategies for enhancing model adaptability. MojoBench contributes to our understanding of LLM capabilities and limitations in emerging programming paradigms fostering more robust code generation systems.
• 摘要：Modular 最近推出的 Mojo 编程语言 (PL) 因据称比 Python 速度显著提升而受到科学界的广泛关注。尽管各种 PL 中的代码大型语言模型 (LLM) 取得了进步，但 Mojo 在这方面仍未得到探索。为了弥补这一差距，我们推出了 MojoBench，这是第一个用于 Mojo 代码生成的框架。MojoBench 包括 HumanEval-Mojo（一个用于评估 Mojo 上的代码 LLM 的基准数据集）和 Mojo-Coder（第一个针对 Mojo 代码生成进行预训练和微调的 LLM，支持 5 种自然语言 (NL) 指令）。我们的结果表明，Mojo-Coder 的性能比 GPT-4o 和 Claude-3.5-Sonnet 等领先模型提高了 30-35%。此外，我们还深入了解了代表性不足和未见过的 PL 的 LLM 行为，为增强模型适应性提供了潜在的策略。 MojoBench 有助于我们了解新兴编程范式中的 LLM 功能和局限性，从而促进更强大的代码生成系统。

Title: Local Contrastive Editing of Gender Stereotypes

• Authors: Marlene Lutz, Rochelle Choenni, Markus Strohmaier, Anne Lauscher
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2410.17739
• Pdf URL: https://arxiv.org/pdf/2410.17739
• Copy Paste: [[2410.17739]] Local Contrastive Editing of Gender Stereotypes(https://arxiv.org/abs/2410.17739)
• Keywords: language model
• Abstract: Stereotypical bias encoded in language models (LMs) poses a threat to safe language technology, yet our understanding of how bias manifests in the parameters of LMs remains incomplete. We introduce local contrastive editing that enables the localization and editing of a subset of weights in a target model in relation to a reference model. We deploy this approach to identify and modify subsets of weights that are associated with gender stereotypes in LMs. Through a series of experiments, we demonstrate that local contrastive editing can precisely localize and control a small subset (< 0.5%) of weights that encode gender bias. Our work (i) advances our understanding of how stereotypical biases can manifest in the parameter space of LMs and (ii) opens up new avenues for developing parameter-efficient strategies for controlling model properties in a contrastive manner.
• 摘要：语言模型 (LM) 中编码的刻板偏见对安全语言技术构成了威胁，但我们对偏见如何在 LM 参数中体现的理解仍然不完整。我们引入了局部对比编辑，可以相对于参考模型定位和编辑目标模型中权重子集。我们采用这种方法来识别和修改与 LM 中的性别刻板印象相关的权重子集。通过一系列实验，我们证明局部对比编辑可以精确定位和控制编码性别偏见的一小部分权重 (< 0.5%)。我们的工作 (i) 加深了我们对刻板偏见如何在 LM 的参数空间中体现的理解，(ii) 为开发以对比方式控制模型属性的参数高效策略开辟了新途径。

Title: Latent Structures of Intertextuality in French Fiction

• Authors: Jean Barré
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.17759
• Pdf URL: https://arxiv.org/pdf/2410.17759
• Copy Paste: [[2410.17759]] Latent Structures of Intertextuality in French Fiction(https://arxiv.org/abs/2410.17759)
• Keywords: language model
• Abstract: Intertextuality is a key concept in literary theory that challenges traditional notions of text, signification or authorship. It views texts as part of a vast intertextual network that is constantly evolving and being reconfigured. This paper argues that the field of computational literary studies is the ideal place to conduct a study of intertextuality since we have now the ability to systematically compare texts with each others. Specifically, we present a work on a corpus of more than 12.000 French fictions from the 18th, 19th and early 20th century. We focus on evaluating the underlying roles of two literary notions, sub-genres and the literary canon in the framing of textuality. The article attempts to operationalize intertextuality using state-of-the-art contextual language models to encode novels and capture features that go beyond simple lexical or thematic approaches. Previous research (Hughes, 2012) supports the existence of a literary "style of a time", and our findings further reinforce this concept. Our findings also suggest that both subgenres and canonicity play a significant role in shaping textual similarities within French fiction. These discoveries point to the importance of considering genre and canon as dynamic forces that influence the evolution and intertextual connections of literary works within specific historical contexts.
• 摘要：互文性是文学理论中的一个关键概念，它挑战了传统的文本、意义或作者观念。它将文本视为不断发展和重新配置的庞大互文网络的一部分。本文认为，计算文学研究领域是进行互文性研究的理想场所，因为我们现在有能力系统地比较文本。具体来说，我们介绍了一项关于 18 世纪、19 世纪和 20 世纪初 12,000 多部法国小说的语料库的研究。我们专注于评估两个文学概念、子流派和文学经典在文本性框架中的潜在作用。本文尝试使用最先进的上下文语言模型对小说进行编码并捕捉超越简单词汇或主题方法的特征，从而将互文性付诸实践。先前的研究（Hughes，2012）支持文学“时代风格”的存在，我们的研究结果进一步证实了这一概念。我们的研究结果还表明，子流派和经典性在塑造法国小说的文本相似性方面发挥着重要作用。这些发现表明，将流派和经典视为影响特定历史背景下文学作品演变和文本间联系的动态力量非常重要。

Title: Leveraging the Domain Adaptation of Retrieval Augmented Generation Models for Question Answering and Reducing Hallucination

• Authors: Salman Rakin, Md. A.R. Shibly, Zahin M. Hossain, Zeeshan Khan, Md. Mostofa Akbar
• Subjects: cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2410.17783
• Pdf URL: https://arxiv.org/pdf/2410.17783
• Copy Paste: [[2410.17783]] Leveraging the Domain Adaptation of Retrieval Augmented Generation Models for Question Answering and Reducing Hallucination(https://arxiv.org/abs/2410.17783)
• Keywords: language model, hallucination, retrieval augmented generation
• Abstract: While ongoing advancements in Large Language Models have demonstrated remarkable success across various NLP tasks, Retrieval Augmented Generation Model stands out to be highly effective on downstream applications like Question Answering. Recently, RAG-end2end model further optimized the architecture and achieved notable performance improvements on domain adaptation. However, the effectiveness of these RAG-based architectures remains relatively unexplored when fine-tuned on specialized domains such as customer service for building a reliable conversational AI system. Furthermore, a critical challenge persists in reducing the occurrence of hallucinations while maintaining high domain-specific accuracy. In this paper, we investigated the performance of diverse RAG and RAG-like architectures through domain adaptation and evaluated their ability to generate accurate and relevant response grounded in the contextual knowledge base. To facilitate the evaluation of the models, we constructed a novel dataset HotelConvQA, sourced from wide range of hotel-related conversations and fine-tuned all the models on our domain specific dataset. We also addressed a critical research gap on determining the impact of domain adaptation on reducing hallucinations across different RAG architectures, an aspect that was not properly measured in prior work. Our evaluation shows positive results in all metrics by employing domain adaptation, demonstrating strong performance on QA tasks and providing insights into their efficacy in reducing hallucinations. Our findings clearly indicate that domain adaptation not only enhances the models' performance on QA tasks but also significantly reduces hallucination across all evaluated RAG architectures.
• 摘要：虽然大型语言模型的持续进步已在各种 NLP 任务中取得了显著的成功，但检索增强生成模型在问答等下游应用中表现出色。最近，RAG-end2end 模型进一步优化了架构，并在领域自适应方面取得了显着的性能改进。然而，当在客户服务等专业领域进行微调以构建可靠的对话式 AI 系统时，这些基于 RAG 的架构的有效性仍然相对未被探索。此外，一个关键的挑战仍然是在保持高领域特定准确性的同时减少幻觉的发生。在本文中，我们通过领域自适应研究了各种 RAG 和类似 RAG 的架构的性能，并评估了它们基于上下文知识库生成准确和相关响应的能力。为了便于对模型进行评估，我们构建了一个新数据集 HotelConvQA，该数据集来源于广泛的酒店相关对话，并在我们的领域特定数据集上对所有模型进行了微调。我们还解决了一个关键的研究空白，即确定域自适应对减少不同 RAG 架构中的幻觉的影响，这是之前的工作中没有正确衡量的方面。我们的评估通过采用域自适应在所有指标中都取得了积极成果，在 QA 任务中表现出色，并深入了解了它们在减少幻觉方面的功效。我们的研究结果清楚地表明，域自适应不仅可以提高模型在 QA 任务上的表现，还可以显著减少所有评估的 RAG 架构中的幻觉。

Title: OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversation

• Authors: Qinglin Zhang, Luyao Cheng, Chong Deng, Qian Chen, Wen Wang, Siqi Zheng, Jiaqing Liu, Hai Yu, Chaohong Tan
• Subjects: cs.CL, cs.AI, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2410.17799
• Pdf URL: https://arxiv.org/pdf/2410.17799
• Copy Paste: [[2410.17799]] OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversation(https://arxiv.org/abs/2410.17799)
• Keywords: language model, gpt, llm
• Abstract: Full-duplex spoken dialogue systems significantly advance over traditional turn-based dialogue systems, as they allow simultaneous bidirectional communication, closely mirroring human-human interactions. However, achieving low latency and natural interactions in full-duplex dialogue systems remains a significant challenge, especially considering human conversation dynamics such as interruptions, backchannels, and overlapping speech. In this paper, we introduce a novel End-to-End GPT-based model OmniFlatten for full-duplex conversation, capable of effectively modeling the complex behaviors inherent to natural conversations with low latency. To achieve full-duplex communication capabilities, we propose a multi-stage post-training scheme that progressively adapts a text-based large language model (LLM) backbone into a speech-text dialogue LLM, capable of generating text and speech in real time, without modifying the architecture of the backbone LLM. The training process comprises three stages: modality alignment, half-duplex dialogue learning, and full-duplex dialogue learning. Throughout all training stages, we standardize the data using a flattening operation, which allows us to unify the training methods and the model architecture across different modalities and tasks. Our approach offers a straightforward modeling technique and a promising research direction for developing efficient and natural end-to-end full-duplex spoken dialogue systems. Audio samples of dialogues generated by OmniFlatten can be found at this web site (this https URL).
• 摘要：全双工口语对话系统比传统的回合制对话系统有了显著的进步，因为它们允许同时进行双向通信，与人与人之间的交互非常相似。然而，在全双工对话系统中实现低延迟和自然交互仍然是一项重大挑战，尤其是考虑到人类对话动态，例如中断、反向信道和重叠语音。在本文中，我们介绍了一种用于全双工对话的新型端到端 GPT 模型 OmniFlatten，能够有效地以低延迟模拟自然对话固有的复杂行为。为了实现全双工通信能力，我们提出了一种多阶段后训练方案，该方案逐步将基于文本的大型语言模型 (LLM) 主干调整为语音文本对话 LLM，能够实时生成文本和语音，而无需修改主干 LLM 的架构。训练过程包括三个阶段：模态对齐、半双工对话学习和全双工对话学习。在所有训练阶段，我们使用扁平化操作对数据进行标准化，这使我们能够统一不同模式和任务的训练方法和模型架构。我们的方法提供了一种简单的建模技术和一个有前途的研究方向，用于开发高效、自然的端到端全双工语音对话系统。OmniFlatten 生成的对话音频样本可在此网站（此 https URL）找到。

Title: Understanding When Tree of Thoughts Succeeds: Larger Models Excel in Generation, Not Discrimination

• Authors: Qiqi Chen, Xinpeng Wang, Philipp Mondorf, Michael A. Hedderich, Barbara Plank
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.17820
• Pdf URL: https://arxiv.org/pdf/2410.17820
• Copy Paste: [[2410.17820]] Understanding When Tree of Thoughts Succeeds: Larger Models Excel in Generation, Not Discrimination(https://arxiv.org/abs/2410.17820)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: Tree of Thoughts (ToT) is a reasoning strategy for Large Language Models (LLMs) that employs a generator to suggest reasoning steps and a discriminator to decide which steps to implement. ToT demonstrates strong performance on reasoning tasks, often surpassing simple methods such as Input-Output (IO) prompting and Chain-of-Thought (CoT) reasoning. However, ToT does not consistently outperform such simpler methods across all models, leaving large knowledge gaps on the conditions under which ToT is most beneficial. In this paper, we analyze the roles of the generator and discriminator separately to better understand the conditions when ToT is beneficial. We find that the generator plays a more critical role than the discriminator in driving the success of ToT. While using even a smaller model as the discriminator, scaling the generator leads to notable improvements in ToT performance, whereas scaling the discriminator with a fixed generator yields only marginal gains. Our results show that models across different scales exhibit comparable discrimination capabilities, yet differ significantly in their generative performance for ToT.
• 摘要：思维树 (ToT) 是一种大型语言模型 (LLM) 的推理策略，它使用生成器来建议推理步骤，使用鉴别器来决定实施哪些步骤。ToT 在推理任务上表现出色，通常超越输入输出 (IO) 提示和思维链 (CoT) 推理等简单方法。然而，ToT 并非在所有模型中都始终优于这些更简单的方法，因此在 ToT 最有利的条件下存在巨大的知识空白。在本文中，我们分别分析了生成器和鉴别器的作用，以更好地了解 ToT 有利的条件。我们发现，在推动 ToT 成功方面，生成器比鉴别器发挥着更重要的作用。在使用更小的模型作为鉴别器时，缩放生成器会显著提高 ToT 性能，而使用固定生成器缩放鉴别器只会带来微不足道的收益。我们的结果表明，不同规模的模型表现出相当的鉴别能力，但在 ToT 的生成性能上存在显著差异。

Title: Understanding Layer Significance in LLM Alignment

• Authors: Guangyuan Shi, Zexin Lu, Xiaoyu Dong, Wenlong Zhang, Xuanyu Zhang, Yujie Feng, Xiao-Ming Wu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.17875
• Pdf URL: https://arxiv.org/pdf/2410.17875
• Copy Paste: [[2410.17875]] Understanding Layer Significance in LLM Alignment(https://arxiv.org/abs/2410.17875)
• Keywords: language model, llm
• Abstract: Aligning large language models (LLMs) through fine-tuning is essential for tailoring them to specific applications. Therefore, understanding what LLMs learn during the alignment process is crucial. Recent studies suggest that alignment primarily adjusts a model's presentation style rather than its foundational knowledge, indicating that only certain components of the model are significantly impacted. To delve deeper into LLM alignment, we propose to identify which layers within LLMs are most critical to the alignment process, thereby uncovering how alignment influences model behavior at a granular level. We propose a novel approach to identify the important layers for LLM alignment (ILA). It involves learning a binary mask for each incremental weight matrix in the LoRA algorithm, indicating the significance of each layer. ILA consistently identifies important layers across various alignment datasets, with nearly 90% overlap even with substantial dataset differences, highlighting fundamental patterns in LLM alignment. Experimental results indicate that freezing non-essential layers improves overall model performance, while selectively tuning the most critical layers significantly enhances fine-tuning efficiency with minimal performance loss.
• 摘要：通过微调对齐大型语言模型 (LLM) 对于使其适应特定应用至关重要。因此，了解 LLM 在对齐过程中学到了什么至关重要。最近的研究表明，对齐主要调整模型的呈现风格而不是基础知识，这表明只有模型的某些组件受到显著影响。为了更深入地研究 LLM 对齐，我们建议确定 LLM 中哪些层对对齐过程最为关键，从而揭示对齐如何在粒度级别影响模型行为。我们提出了一种新方法来识别 LLM 对齐 (ILA) 的重要层。它涉及学习 LoRA 算法中每个增量权重矩阵的二进制掩码，以指示每个层的重要性。ILA 始终如一地识别各种对齐数据集中的重要层，即使数据集存在很大差异，重叠率也有近 90%，突出了 LLM 对齐中的基本模式。实验结果表明，冻结非必要层可提高整体模型性能，而有选择地调整最关键的层可显着提高微调效率，同时将性能损失降至最低。

Title: Scaling Diffusion Language Models via Adaptation from Autoregressive Models

• Authors: Shansan Gong, Shivam Agarwal, Yizhe Zhang, Jiacheng Ye, Lin Zheng, Mukai Li, Chenxin An, Peilin Zhao, Wei Bi, Jiawei Han, Hao Peng, Lingpeng Kong
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.17891
• Pdf URL: https://arxiv.org/pdf/2410.17891
• Copy Paste: [[2410.17891]] Scaling Diffusion Language Models via Adaptation from Autoregressive Models(https://arxiv.org/abs/2410.17891)
• Keywords: language model, gpt, prompt
• Abstract: Diffusion Language Models (DLMs) have emerged as a promising new paradigm for text generative modeling, potentially addressing limitations of autoregressive (AR) models. However, current DLMs have been studied at a smaller scale compared to their AR counterparts and lack fair comparison on language modeling benchmarks. Additionally, training diffusion models from scratch at scale remains challenging. Given the prevalence of open-source AR language models, we propose adapting these models to build text diffusion models. We demonstrate connections between AR and diffusion modeling objectives and introduce a simple continual pre-training approach for training diffusion models. Through systematic evaluation on language modeling, reasoning, and commonsense benchmarks, we show that we can convert AR models ranging from 127M to 7B parameters (GPT2 and LLaMA) into diffusion models DiffuGPT and DiffuLLaMA, using less than 200B tokens for training. Our experimental results reveal that these models outperform earlier DLMs and are competitive with their AR counterparts. We release a suite of DLMs (with 127M, 355M, and 7B parameters) capable of generating fluent text, performing in-context learning, filling in the middle without prompt re-ordering, and following instructions \url{this https URL}.
• 摘要：扩散语言模型 (DLM) 已成为文本生成建模的一种有前途的新范式，有可能解决自回归 (AR) 模型的局限性。然而，与 AR 模型相比，目前的 DLM 的研究规模较小，并且缺乏对语言建模基准的公平比较。此外，从头开始大规模训练扩散模型仍然具有挑战性。鉴于开源 AR 语言模型的普及，我们建议调整这些模型来构建文本扩散模型。我们展示了 AR 和扩散建模目标之间的联系，并介绍了一种简单的持续预训练方法来训练扩散模型。通过对语言建模、推理和常识基准的系统评估，我们表明我们可以将参数范围从 127M 到 7B（GPT2 和 LLaMA）的 AR 模型转换为扩散模型 DiffuGPT 和 DiffuLLaMA，使用不到 200B 个 token 进行训练。我们的实验结果表明，这些模型优于早期的 DLM，并且与 AR 模型相比具有竞争力。我们发布了一套 DLM（具有 127M、355M 和 7B 参数），能够生成流畅的文本、执行上下文学习、填充中间内容而无需提示重新排序，并按照说明 \url{此 https URL}。

Title: SimRAG: Self-Improving Retrieval-Augmented Generation for Adapting Large Language Models to Specialized Domains

• Authors: Ran Xu, Hui Liu, Sreyashi Nag, Zhenwei Dai, Yaochen Xie, Xianfeng Tang, Chen Luo, Yang Li, Joyce C. Ho, Carl Yang, Qi He
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.17952
• Pdf URL: https://arxiv.org/pdf/2410.17952
• Copy Paste: [[2410.17952]] SimRAG: Self-Improving Retrieval-Augmented Generation for Adapting Large Language Models to Specialized Domains(https://arxiv.org/abs/2410.17952)
• Keywords: language model, llm, prompt, retrieval-augmented generation
• Abstract: Retrieval-augmented generation (RAG) enhances the question-answering (QA) abilities of large language models (LLMs) by integrating external knowledge. However, adapting general-purpose RAG systems to specialized fields such as science and medicine poses unique challenges due to distribution shifts and limited access to domain-specific data. To tackle this, we propose SimRAG, a self-training approach that equips the LLM with joint capabilities of question answering and question generation for domain adaptation. Our method first fine-tunes the LLM on instruction-following, question-answering, and search-related data. Then, it prompts the same LLM to generate diverse domain-relevant questions from unlabeled corpora, with an additional filtering strategy to retain high-quality synthetic examples. By leveraging these synthetic examples, the LLM can improve their performance on domain-specific RAG tasks. Experiments on 11 datasets, spanning two backbone sizes and three domains, demonstrate that SimRAG outperforms baselines by 1.2\%--8.6\%.
• 摘要：检索增强生成 (RAG) 通过整合外部知识来增强大型语言模型 (LLM) 的问答 (QA) 能力。然而，由于分布变化和对特定领域数据的访问有限，将通用 RAG 系统应用于科学和医学等专业领域带来了独特的挑战。为了解决这个问题，我们提出了 SimRAG，这是一种自训练方法，它为 LLM 配备了问答和问题生成的联合能力，以实现领域适应。我们的方法首先根据指令遵循、问答和搜索相关数据对 LLM 进行微调。然后，它提示相同的 LLM 从未标记的语料库中生成各种领域相关问题，并使用额外的过滤策略来保留高质量的合成示例。通过利用这些合成示例，LLM 可以提高其在特定领域的 RAG 任务上的性能。在涵盖两种主干规模和三个领域的 11 个数据集上进行的实验表明，SimRAG 的表现比基线高出 1.2%--8.6%。

Title: MiLoRA: Efficient Mixture of Low-Rank Adaptation for Large Language Models Fine-tuning

• Authors: Jingfan Zhang, Yi Zhao, Dan Chen, Xing Tian, Huanran Zheng, Wei Zhu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.18035
• Pdf URL: https://arxiv.org/pdf/2410.18035
• Copy Paste: [[2410.18035]] MiLoRA: Efficient Mixture of Low-Rank Adaptation for Large Language Models Fine-tuning(https://arxiv.org/abs/2410.18035)
• Keywords: language model, llm, prompt
• Abstract: Low-rank adaptation (LoRA) and its mixture-of-experts (MOE) variants are highly effective parameter-efficient fine-tuning (PEFT) methods. However, they introduce significant latency in multi-tenant settings due to the LoRA modules and MOE routers added to multiple linear modules in the Transformer layer. To address this issue, we propose Mixture of Low-Rank Adaptation (MiLoRA), a novel and efficient LoRA variant. MiLoRA differs from previous MOE-style LoRA methods by considering each LoRA module as an expert and employing a prompt-aware routing mechanism. This mechanism calculates expert routing results once before generating the first new token and reuses these results for subsequent tokens, reducing latency. Extensive experiments and analysis on commonsense reasoning tasks, math reasoning tasks, and widely used LLM evaluation benchmarks demonstrate that MiLoRA consistently outperforms strong PEFT baselines with comparable tunable parameter budgets. Additionally, MiLoRA significantly reduces latency in multi-tenant settings compared to previous LoRA-based methods.
• 摘要：低秩自适应 (LoRA) 及其混合专家 (MOE) 变体是高效的参数高效微调 (PEFT) 方法。然而，由于在 Transformer 层中的多个线性模块中添加了 LoRA 模块和 MOE 路由器，它们在多租户设置中引入了显著的延迟。为了解决这个问题，我们提出了混合低秩自适应 (MiLoRA)，这是一种新颖而高效的 LoRA 变体。MiLoRA 与以前的 MOE 风格的 LoRA 方法不同，它将每个 LoRA 模块视为专家，并采用提示感知路由机制。该机制在生成第一个新令牌之前计算一次专家路由结果，并将这些结果重新用于后续令牌，从而减少延迟。对常识推理任务、数学推理任务和广泛使用的 LLM 评估基准的大量实验和分析表明，在可比的可调参数预算下，MiLoRA 始终优于强大的 PEFT 基线。此外，与以前基于 LoRA 的方法相比，MiLoRA 显著降低了多租户设置中的延迟。

Title: Key Algorithms for Keyphrase Generation: Instruction-Based LLMs for Russian Scientific Keyphrases

• Authors: Anna Glazkova, Dmitry Morozov, Timur Garipov
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.18040
• Pdf URL: https://arxiv.org/pdf/2410.18040
• Copy Paste: [[2410.18040]] Key Algorithms for Keyphrase Generation: Instruction-Based LLMs for Russian Scientific Keyphrases(https://arxiv.org/abs/2410.18040)
• Keywords: language model, llm, prompt
• Abstract: Keyphrase selection is a challenging task in natural language processing that has a wide range of applications. Adapting existing supervised and unsupervised solutions for the Russian language faces several limitations due to the rich morphology of Russian and the limited number of training datasets available. Recent studies conducted on English texts show that large language models (LLMs) successfully address the task of generating keyphrases. LLMs allow achieving impressive results without task-specific fine-tuning, using text prompts instead. In this work, we access the performance of prompt-based methods for generating keyphrases for Russian scientific abstracts. First, we compare the performance of zero-shot and few-shot prompt-based methods, fine-tuned models, and unsupervised methods. Then we assess strategies for selecting keyphrase examples in a few-shot setting. We present the outcomes of human evaluation of the generated keyphrases and analyze the strengths and weaknesses of the models through expert assessment. Our results suggest that prompt-based methods can outperform common baselines even using simple text prompts.
• 摘要：关键词选择是自然语言处理中一项具有挑战性的任务，具有广泛的应用。由于俄语的形态丰富，可用的训练数据集数量有限，因此将现有的监督和无监督解决方案应用于俄语面临一些限制。最近对英语文本进行的研究表明，大型语言模型 (LLM) 成功解决了生成关键词的任务。LLM 允许使用文本提示，而无需针对特定任务进行微调，即可获得令人印象深刻的结果。在这项工作中，我们访问了基于提示的方法为俄语科学摘要生成关键词的性能。首先，我们比较了零样本和少样本基于提示的方法、微调模型和无监督方法的性能。然后，我们评估在少样本设置中选择关键词示例的策略。我们展示了对生成关键词的人工评估结果，并通过专家评估分析了模型的优缺点。我们的结果表明，即使使用简单的文本提示，基于提示的方法也可以胜过常见基线。

Title: LongRAG: A Dual-Perspective Retrieval-Augmented Generation Paradigm for Long-Context Question Answering

• Authors: Qingfei Zhao, Ruobing Wang, Yukuo Cen, Daren Zha, Shicheng Tan, Yuxiao Dong, Jie Tang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.18050
• Pdf URL: https://arxiv.org/pdf/2410.18050
• Copy Paste: [[2410.18050]] LongRAG: A Dual-Perspective Retrieval-Augmented Generation Paradigm for Long-Context Question Answering(https://arxiv.org/abs/2410.18050)
• Keywords: language model, llm, long context, retrieval-augmented generation
• Abstract: Long-Context Question Answering (LCQA), a challenging task, aims to reason over long-context documents to yield accurate answers to questions. Existing long-context Large Language Models (LLMs) for LCQA often struggle with the "lost in the middle" issue. Retrieval-Augmented Generation (RAG) mitigates this issue by providing external factual evidence. However, its chunking strategy disrupts the global long-context information, and its low-quality retrieval in long contexts hinders LLMs from identifying effective factual details due to substantial noise. To this end, we propose LongRAG, a general, dual-perspective, and robust LLM-based RAG system paradigm for LCQA to enhance RAG's understanding of complex long-context knowledge (i.e., global information and factual details). We design LongRAG as a plug-and-play paradigm, facilitating adaptation to various domains and LLMs. Extensive experiments on three multi-hop datasets demonstrate that LongRAG significantly outperforms long-context LLMs (up by 6.94%), advanced RAG (up by 6.16%), and Vanilla RAG (up by 17.25%). Furthermore, we conduct quantitative ablation studies and multi-dimensional analyses, highlighting the effectiveness of the system's components and fine-tuning strategies. Data and code are available at this https URL.
• 摘要：长上下文问答 (LCQA) 是一项具有挑战性的任务，旨在对长上下文文档进行推理以得出问题的准确答案。现有的用于 LCQA 的长上下文大型语言模型 (LLM) 经常会遇到“迷失在中间”的问题。检索增强生成 (RAG) 通过提供外部事实证据来缓解此问题。然而，它的分块策略会破坏全局长上下文信息，并且其在长上下文中的低质量检索会由于大量噪音而阻碍 LLM 识别有效的事实细节。为此，我们提出了 LongRAG，这是一种通用、双视角且基于 LLM 的稳健的 LCQA RAG 系统范式，以增强 RAG 对复杂长上下文知识（即全局信息和事实细节）的理解。我们将 LongRAG 设计为即插即用的范式，以方便适应各种领域和 LLM。在三个多跳数据集上进行的大量实验表明，LongRAG 的表现明显优于长上下文 LLM（提升 6.94%）、高级 RAG（提升 6.16%）和 Vanilla RAG（提升 17.25%）。此外，我们还进行了定量消融研究和多维分析，突出了系统组件和微调策略的有效性。数据和代码可在此 https URL 上获取。