2025-01-10

Title: Advancing Retrieval-Augmented Generation for Persian: Development of Language Models, Comprehensive Benchmarks, and Best Practices for Optimization

• Authors: Sara Bourbour Hosseinbeigi, Sina Asghari, Mohammad Ali Seif Kashani, Mohammad Hossein Shalchian, Mohammad Amin Abbasi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.04858
• Pdf URL: https://arxiv.org/pdf/2501.04858
• Copy Paste: [[2501.04858]] Advancing Retrieval-Augmented Generation for Persian: Development of Language Models, Comprehensive Benchmarks, and Best Practices for Optimization(https://arxiv.org/abs/2501.04858)
• Keywords: language model, retrieval-augmented generation
• Abstract: This paper examines the specific obstacles of constructing Retrieval-Augmented Generation(RAG) systems in low-resource languages, with a focus on Persian's complicated morphology and versatile syntax. The research aims to improve retrieval and generation accuracy by introducing Persian-specific models, namely MatinaRoberta(a masked language model) and MatinaSRoberta(a fine-tuned Sentence-BERT), along with a comprehensive benchmarking framework. Three datasets-general knowledge(PQuad), scientifically specialized texts, and organizational reports, were used to assess these models after they were trained on a varied corpus of 73.11 billion Persian tokens. The methodology involved extensive pretraining, fine-tuning with tailored loss functions, and systematic evaluations using both traditional metrics and the Retrieval-Augmented Generation Assessment framework. The results show that MatinaSRoberta outperformed previous embeddings, achieving superior contextual relevance and retrieval accuracy across datasets. Temperature tweaking, chunk size modifications, and document summary indexing were explored to enhance RAG setups. Larger models like Llama-3.1 (70B) consistently demonstrated the highest generation accuracy, while smaller models faced challenges with domain-specific and formal contexts. The findings underscore the potential for developing RAG systems in Persian through customized embeddings and retrieval-generation settings and highlight the enhancement of NLP applications such as search engines and legal document analysis in low-resource languages.
• 摘要：本文探讨了在资源匮乏的语言中构建检索增强生成 (RAG) 系统的具体障碍，重点关注波斯语的复杂形态和多变语法。该研究旨在通过引入波斯语专用模型，即 MatinaRoberta（一种掩码语言模型）和 MatinaSRoberta（一种微调的 Sentence-BERT），以及一个全面的基准测试框架来提高检索和生成的准确性。在对这些模型进行 731.1 亿个波斯语标记的多样化语料库训练后，使用三个数据集（常识 (PQuad)、科学专业文本和组织报告）来评估这些模型。该方法涉及广泛的预训练、使用定制损失函数进行微调，以及使用传统指标和检索增强生成评估框架进行系统评估。结果表明，MatinaSRoberta 的表现优于之前的嵌入，在数据集上实现了卓越的上下文相关性和检索准确性。探索了温度调整、块大小修改和文档摘要索引以增强 RAG 设置。较大的模型（如 Llama-3.1 (70B)）始终表现出最高的生成准确率，而较小的模型则面临着领域特定和形式上下文的挑战。研究结果强调了通过定制嵌入和检索生成设置在波斯语中开发 RAG 系统的潜力，并强调了搜索引擎和低资源语言中的法律文档分析等 NLP 应用程序的增强。

Title: Real-Time Textless Dialogue Generation

• Authors: Long Mai, Julie Carson-Berndsen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.04877
• Pdf URL: https://arxiv.org/pdf/2501.04877
• Copy Paste: [[2501.04877]] Real-Time Textless Dialogue Generation(https://arxiv.org/abs/2501.04877)
• Keywords: language model, llm
• Abstract: Recent advancements in large language models (LLMs) have led to significant progress in text-based dialogue systems. These systems can now generate high-quality responses that are accurate and coherent across a wide range of topics and tasks. However, spoken dialogue systems still lag behind in terms of naturalness. They tend to produce robotic interactions, with issues such as slow response times, overly generic or cautious replies, and a lack of natural rhythm and fluid turn-taking. This shortcoming is largely due to the over-reliance on the traditional cascaded design, which involve separate, sequential components, as well as the use of text as an intermediate representation. This paper propose a real-time, textless spoken dialogue generation model (RTTL-DG) that aims to overcome these challenges. Our system enables fluid turn-taking and generates responses with minimal delay by processing streaming spoken conversation directly. Additionally, our model incorporates backchannels, filters, laughter, and other paralinguistic signals, which are often absent in cascaded dialogue systems, to create more natural and human-like interactions. The implementations and generated samples are available in our repository: this https URL
• 摘要：大型语言模型 (LLM) 的最新进展已使基于文本的对话系统取得了重大进展。这些系统现在可以生成高质量响应，这些响应在广泛的主题和任务中都是准确且连贯的。然而，口头对话系统在自然性方面仍然落后。它们往往会产生机械化的交互，存在响应时间慢、回复过于笼统或谨慎以及缺乏自然节奏和流畅的轮换等问题。这一缺点主要是由于过度依赖传统的级联设计，该设计涉及单独的顺序组件，以及使用文本作为中间表示。本文提出了一种实时、无文本的口头对话生成模型 (RTTL-DG)，旨在克服这些挑战。我们的系统通过直接处理流式口头对话来实现流畅的轮换并以最小的延迟生成响应。此外，我们的模型结合了反向通道、过滤器、笑声和其他副语言信号，这些信号在级联对话系统中通常不存在，以创建更自然和更像人类的交互。实现和生成的示例可在我们的存储库中找到：此 https URL

Title: Leveraging Log Probabilities in Language Models to Forecast Future Events

• Authors: Tommaso Soru, Jim Marshall
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.04880
• Pdf URL: https://arxiv.org/pdf/2501.04880
• Copy Paste: [[2501.04880]] Leveraging Log Probabilities in Language Models to Forecast Future Events(https://arxiv.org/abs/2501.04880)
• Keywords: language model, llm
• Abstract: In the constantly changing field of data-driven decision making, accurately predicting future events is crucial for strategic planning in various sectors. The emergence of Large Language Models (LLMs) marks a significant advancement in this area, offering advanced tools that utilise extensive text data for prediction. In this industry paper, we introduce a novel method for AI-driven foresight using LLMs. Building on top of previous research, we employ data on current trends and their trajectories for generating forecasts on 15 different topics. Subsequently, we estimate their probabilities via a multi-step approach based on log probabilities. We show we achieve a Brier score of 0.186, meaning a +26% improvement over random chance and a +19% improvement over widely-available AI systems.
• 摘要：在不断变化的数据驱动决策领域，准确预测未来事件对于各个行业的战略规划至关重要。大型语言模型 (LLM) 的出现标志着该领域的重大进步，它提供了利用大量文本数据进行预测的高级工具。在这篇行业论文中，我们介绍了一种使用 LLM 进行人工智能驱动预测的新方法。在先前研究的基础上，我们使用有关当前趋势及其轨迹的数据来生成 15 个不同主题的预测。随后，我们通过基于对数概率的多步骤方法估计它们的概率。我们表明我们获得了 0.186 的 Brier 分数，这意味着比随机机会提高了 +26%，比广泛使用的人工智能系统提高了 +19%。

Title: SUGAR: Leveraging Contextual Confidence for Smarter Retrieval

• Authors: Hanna Zubkova, Ji-Hoon Park, Seong-Whan Lee
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.04899
• Pdf URL: https://arxiv.org/pdf/2501.04899
• Copy Paste: [[2501.04899]] SUGAR: Leveraging Contextual Confidence for Smarter Retrieval(https://arxiv.org/abs/2501.04899)
• Keywords: language model, llm, hallucination, retrieval-augmented generation
• Abstract: Bearing in mind the limited parametric knowledge of Large Language Models (LLMs), retrieval-augmented generation (RAG) which supplies them with the relevant external knowledge has served as an approach to mitigate the issue of hallucinations to a certain extent. However, uniformly retrieving supporting context makes response generation source-inefficient, as triggering the retriever is not always necessary, or even inaccurate, when a model gets distracted by noisy retrieved content and produces an unhelpful answer. Motivated by these issues, we introduce Semantic Uncertainty Guided Adaptive Retrieval (SUGAR), where we leverage context-based entropy to actively decide whether to retrieve and to further determine between single-step and multi-step retrieval. Our empirical results show that selective retrieval guided by semantic uncertainty estimation improves the performance across diverse question answering tasks, as well as achieves a more efficient inference.
• 摘要：考虑到大型语言模型 (LLM) 的参数知识有限，检索增强生成 (RAG) 为它们提供相关的外部知识，在一定程度上缓解了幻觉问题。然而，统一检索支持上下文会使响应生成源效率低下，因为当模型被嘈杂的检索内容分散注意力并产生无用的答案时，触发检索器并不总是必要的，甚至是不准确的。受这些问题的启发，我们引入了语义不确定性引导的自适应检索 (SUGAR)，我们利用基于上下文的熵来主动决定是否检索，并进一步确定单步和多步检索。我们的实证结果表明，由语义不确定性估计引导的选择性检索可以提高各种问答任务的性能，并实现更有效的推理。

Title: JELLY: Joint Emotion Recognition and Context Reasoning with LLMs for Conversational Speech Synthesis

• Authors: Jun-Hyeok Cha, Seung-Bin Kim, Hyung-Seok Oh, Seong-Whan Lee
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2501.04904
• Pdf URL: https://arxiv.org/pdf/2501.04904
• Copy Paste: [[2501.04904]] JELLY: Joint Emotion Recognition and Context Reasoning with LLMs for Conversational Speech Synthesis(https://arxiv.org/abs/2501.04904)
• Keywords: language model, llm
• Abstract: Recently, there has been a growing demand for conversational speech synthesis (CSS) that generates more natural speech by considering the conversational context. To address this, we introduce JELLY, a novel CSS framework that integrates emotion recognition and context reasoning for generating appropriate speech in conversation by fine-tuning a large language model (LLM) with multiple partial LoRA modules. We propose an Emotion-aware Q-former encoder, which enables the LLM to perceive emotions in speech. The encoder is trained to align speech emotions with text, utilizing datasets of emotional speech. The entire model is then fine-tuned with conversational speech data to infer emotional context for generating emotionally appropriate speech in conversation. Our experimental results demonstrate that JELLY excels in emotional context modeling, synthesizing speech that naturally aligns with conversation, while mitigating the scarcity of emotional conversational speech datasets.
• 摘要：最近，人们对对话语音合成 (CSS) 的需求日益增长，它通过考虑对话上下文来生成更自然的语音。为了解决这个问题，我们推出了 JELLY，这是一个新颖的 CSS 框架，它集成了情感识别和上下文推理，通过对具有多个部分 LoRA 模块的大型语言模型 (LLM) 进行微调来生成对话中的适当语音。我们提出了一种情感感知 Q-former 编码器，使 LLM 能够感知语音中的情感。利用情感语音数据集，对编码器进行训练以将语音情感与文本对齐。然后使用对话语音数据对整个模型进行微调，以推断情感上下文，从而生成对话中情感适当的语音。我们的实验结果表明，JELLY 在情感上下文建模方面表现出色，可以合成与对话自然一致的语音，同时缓解情感对话语音数据集的稀缺性。

Title: Investigating Numerical Translation with Large Language Models

• Authors: Wei Tang, Jiawei Yu, Yuang Li, Yanqing Zhao, Weidong Zhang, Wei Feng, Min Zhang, Hao Yang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.04927
• Pdf URL: https://arxiv.org/pdf/2501.04927
• Copy Paste: [[2501.04927]] Investigating Numerical Translation with Large Language Models(https://arxiv.org/abs/2501.04927)
• Keywords: language model, llm
• Abstract: The inaccurate translation of numbers can lead to significant security issues, ranging from financial setbacks to medical inaccuracies. While large language models (LLMs) have made significant advancements in machine translation, their capacity for translating numbers has not been thoroughly explored. This study focuses on evaluating the reliability of LLM-based machine translation systems when handling numerical data. In order to systematically test the numerical translation capabilities of currently open source LLMs, we have constructed a numerical translation dataset between Chinese and English based on real business data, encompassing ten types of numerical translation. Experiments on the dataset indicate that errors in numerical translation are a common issue, with most open-source LLMs faltering when faced with our test scenarios. Especially when it comes to numerical types involving large units like ``million", ``billion", and "yi", even the latest llama3.1 8b model can have error rates as high as 20%. Finally, we introduce three potential strategies to mitigate the numerical mistranslations for large units.
• 摘要：数字翻译不准确会导致严重的安全问题，从财务损失到医疗错误。虽然大型语言模型 (LLM) 在机器翻译方面取得了重大进展，但它们的数字翻译能力尚未得到彻底探索。本研究重点评估基于 LLM 的机器翻译系统在处理数字数据时的可靠性。为了系统地测试当前开源 LLM 的数字翻译能力，我们基于真实业务数据构建了一个中英文数字翻译数据集，涵盖了十种类型的数字翻译。对数据集的实验表明，数字翻译中的错误是一个常见问题，大多数开源 LLM 在我们的测试场景中都表现不佳。尤其是当涉及到“百万”、“十亿”和“一”等大单位的数字类型时，即使是最新的 llama3.1 8b 模型的错误率也高达 20%。最后，我们介绍了三种潜在的策略来减轻大单位的数字误译。

Title: Step-by-Step Mastery: Enhancing Soft Constraint Following Ability of Large Language Models

• Authors: Qingyu Ren, Jie Zeng, Qianyu He, Jiaqing Liang, Yanghua Xiao, Weikang Zhou, Zeye Sun, Fei Yu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.04945
• Pdf URL: https://arxiv.org/pdf/2501.04945
• Copy Paste: [[2501.04945]] Step-by-Step Mastery: Enhancing Soft Constraint Following Ability of Large Language Models(https://arxiv.org/abs/2501.04945)
• Keywords: language model, llm
• Abstract: It is crucial for large language models (LLMs) to follow instructions that involve multiple constraints. However, soft constraints are semantically related and difficult to verify through automated methods. These constraints remain a significant challenge for LLMs. To enhance the ability of LLMs to follow soft constraints, we initially design a pipeline to obtain high-quality outputs automatically. Additionally, to fully utilize the acquired data, we introduce a training paradigm based on curriculum learning. We experimentally evaluate the effectiveness of our methods in improving LLMs' soft constraint following ability and analyze the factors driving the improvements. The datasets and code are publicly available at this https URL.
• 摘要：对于大型语言模型 (LLM) 来说，遵循涉及多个约束的指令至关重要。然而，软约束在语义上是相关的，并且很难通过自动化方法进行验证。这些约束仍然是 LLM 面临的重大挑战。为了增强 LLM 遵循软约束的能力，我们最初设计了一个管道来自动获得高质量的输出。此外，为了充分利用获取的数据，我们引入了一种基于课程学习的训练范式。我们通过实验评估了我们的方法在提高 LLM 遵循软约束能力方面的有效性，并分析了推动改进的因素。数据集和代码在此 https URL 上公开提供。

Title: Demystifying Domain-adaptive Post-training for Financial LLMs

• Authors: Zixuan Ke, Yifei Ming, Xuan-Phi Nguyen, Caiming Xiong, Shafiq Joty
• Subjects: cs.CL, cs.AI, cs.CE, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.04961
• Pdf URL: https://arxiv.org/pdf/2501.04961
• Copy Paste: [[2501.04961]] Demystifying Domain-adaptive Post-training for Financial LLMs(https://arxiv.org/abs/2501.04961)
• Keywords: language model, llm
• Abstract: Domain-adaptive post-training of large language models (LLMs) has emerged as a promising approach for specialized domains such as medicine and finance. However, significant challenges remain in identifying optimal adaptation criteria and training strategies across varying data and model configurations. To address these challenges, we introduce FINDAP, a systematic and fine-grained investigation into domain-adaptive post-training of LLMs for the finance domain. Our approach begins by identifying the core capabilities required for the target domain and designing a comprehensive evaluation suite aligned with these needs. We then analyze the effectiveness of key post-training stages, including continual pretraining, instruction tuning, and preference alignment. Building on these insights, we propose an effective training recipe centered on a novel preference data distillation method, which leverages process signals from a generative reward model. The resulting model, Llama-Fin, achieves state-of-the-art performance across a wide range of financial tasks. Our analysis also highlights how each post-training stage contributes to distinct capabilities, uncovering specific challenges and effective solutions, providing valuable insights for domain adaptation of LLMs. Project page: this https URL
• 摘要：大型语言模型 (LLM) 的领域自适应后训练已成为医学和金融等专业领域的一种有前途的方法。然而，在确定不同数据和模型配置的最佳适应标准和训练策略方面仍然存在重大挑战。为了应对这些挑战，我们引入了 FINDAP，这是一项针对金融领域 LLM 领域自适应后训练的系统而细致的研究。我们的方法首先确定目标领域所需的核心能力，并设计一个符合这些需求的综合评估套件。然后，我们分析关键后训练阶段的有效性，包括持续预训练、指令调整和偏好调整。基于这些见解，我们提出了一种有效的训练方法，该方法以新颖的偏好数据提炼方法为中心，该方法利用了生成奖励模型的过程信号。由此产生的模型 Llama-Fin 在广泛的金融任务中实现了最先进的性能。我们的分析还强调了每个后训练阶段如何为不同的能力做出贡献，揭示了特定的挑战和有效的解决方案，为 LLM 的领域适应提供了宝贵的见解。项目页面：此 https URL

Title: VoxEval: Benchmarking the Knowledge Understanding Capabilities of End-to-End Spoken Language Models

• Authors: Wenqian Cui, Xiaoqi Jiao, Ziqiao Meng, Irwin King
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2501.04962
• Pdf URL: https://arxiv.org/pdf/2501.04962
• Copy Paste: [[2501.04962]] VoxEval: Benchmarking the Knowledge Understanding Capabilities of End-to-End Spoken Language Models(https://arxiv.org/abs/2501.04962)
• Keywords: language model
• Abstract: With the growing demand for developing speech-based interaction models, end-to-end Spoken Language Models (SLMs) have emerged as a promising solution. When engaging in conversations with humans, it is essential for these models to comprehend a wide range of world knowledge. In this paper, we introduce VoxEval, a novel speech question-answering benchmark specifically designed to assess SLMs' knowledge understanding through purely speech-based interactions. Unlike existing AudioQA benchmarks, VoxEval maintains speech format for both questions and answers, evaluates model robustness across diverse audio conditions (varying timbres, audio qualities, and speaking styles), and pioneers the assessment of challenging domains like mathematical problem-solving in spoken format. Our comprehensive evaluation of recent SLMs using VoxEval reveals significant performance limitations in current models, highlighting crucial areas for future improvements.
• 摘要：随着对开发基于语音的交互模型的需求不断增长，端到端口语语言模型 (SLM) 已成为一种有前途的解决方案。在与人类对话时，这些模型必须理解广泛的世界知识。在本文中，我们介绍了 VoxEval，这是一种新颖的语音问答基准，专门用于通过纯语音交互来评估 SLM 的知识理解。与现有的 AudioQA 基准不同，VoxEval 保持了问题和答案的语音格式，评估了模型在不同音频条件下（不同的音色、音频质量和说话风格）的稳健性，并开创了对口语形式的数学问题解决等具有挑战性的领域的评估。我们使用 VoxEval 对最近的 SLM 进行了全面评估，揭示了当前模型在性能上的重大局限性，突出了未来改进的关键领域。

Title: TreeKV: Smooth Key-Value Cache Compression with Tree Structures

• Authors: Ziwei He, Jian Yuan, Haoli Bai, Jingwen Leng, Bo Jiang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.04987
• Pdf URL: https://arxiv.org/pdf/2501.04987
• Copy Paste: [[2501.04987]] TreeKV: Smooth Key-Value Cache Compression with Tree Structures(https://arxiv.org/abs/2501.04987)
• Keywords: language model, llm
• Abstract: Efficient key-value (KV) cache compression is critical for scaling transformer-based Large Language Models (LLMs) in long sequences and resource-limited settings. Existing methods evict tokens based on their positions or importance scores, but position-based strategies can miss crucial information outside predefined regions, while those relying on global importance scores resulting in strong regional biases, limiting the KV cache's overall context retention and potentially impairing the performance of LLMs on complex tasks. Our wavelet analysis reveals that as tokens approach the end of sequence, their contributions to generation gradually increase and tends to diverge more from neighboring tokens, indicating a smooth transition with increasing complexity and variability from distant to nearby context. Motivated by this observation, we propose TreeKV, an intuitive, training-free method that employs a tree structure for smooth cache compression. TreeKV maintains a fixed cache size, allowing LLMs to deliver high-quality output even in long text scenarios. Unlike most compression methods, TreeKV is applicable to both the generation and prefilling stages. It consistently surpasses all baseline models in language modeling tasks on PG19 and OpenWebText2, allowing LLMs trained with short context window to generalize to longer window with a 16x cache reduction. On the Longbench benchmark, TreeKV achieves the best performance with only 6\% of the budget at optimal efficiency.
• 摘要：高效的键值 (KV) 缓存压缩对于在长序列和资源有限的环境中扩展基于转换器的大型语言模型 (LLM) 至关重要。现有方法根据标记的位置或重要性分数逐出标记，但基于位置的策略可能会错过预定义区域之外的关键信息，而那些依赖于全局重要性分数的策略会导致强烈的区域偏差，从而限制 KV 缓存的整体上下文保留，并可能损害 LLM 在复杂任务上的性能。我们的小波分析表明，随着标记接近序列的末尾，它们对生成的贡献逐渐增加，并且倾向于与相邻标记产生更大的分歧，这表明从远处到近处的上下文的过渡平稳，复杂性和多变性不断增加。受此观察的启发，我们提出了 TreeKV，这是一种直观的、无需训练的方法，它采用树结构进行平滑的缓存压缩。TreeKV 保持固定的缓存大小，即使在长文本场景中也能让 LLM 提供高质量的输出。与大多数压缩方法不同，TreeKV 适用于生成和预填充阶段。它在 PG19 和 OpenWebText2 上的语言建模任务中始终超越所有基线模型，允许使用短上下文窗口训练的 LLM 推广到较长的窗口，同时缓存减少 16 倍。在 Longbench 基准测试中，TreeKV 以最佳效率仅用 6% 的预算实现了最佳性能。

Title: Enhancing Human-Like Responses in Large Language Models

• Authors: Ethem Yağız Çalık, Talha Rüzgar Akkuş
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.05032
• Pdf URL: https://arxiv.org/pdf/2501.05032
• Copy Paste: [[2501.05032]] Enhancing Human-Like Responses in Large Language Models(https://arxiv.org/abs/2501.05032)
• Keywords: language model, llm
• Abstract: This paper explores the advancements in making large language models (LLMs) more human-like. We focus on techniques that enhance natural language understanding, conversational coherence, and emotional intelligence in AI systems. The study evaluates various approaches, including fine-tuning with diverse datasets, incorporating psychological principles, and designing models that better mimic human reasoning patterns. Our findings demonstrate that these enhancements not only improve user interactions but also open new possibilities for AI applications across different domains. Future work will address the ethical implications and potential biases introduced by these human-like attributes.
• 摘要：本文探讨了使大型语言模型 (LLM) 更像人类的进展。我们专注于增强人工智能系统中的自然语言理解、对话连贯性和情商的技术。这项研究评估了各种方法，包括使用不同的数据集进行微调、结合心理学原理以及设计更好地模仿人类推理模式的模型。我们的研究结果表明，这些增强功能不仅可以改善用户交互，还可以为不同领域的人工智能应用开辟新的可能性。未来的工作将解决这些类人属性带来的道德影响和潜在偏见。

Title: SWE-Fixer: Training Open-Source LLMs for Effective and Efficient GitHub Issue Resolution

• Authors: Chengxing Xie, Bowen Li, Chang Gao, He Du, Wai Lam, Difan Zou, Kai Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.05040
• Pdf URL: https://arxiv.org/pdf/2501.05040
• Copy Paste: [[2501.05040]] SWE-Fixer: Training Open-Source LLMs for Effective and Efficient GitHub Issue Resolution(https://arxiv.org/abs/2501.05040)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have demonstrated remarkable proficiency across a variety of complex tasks. One significant application of LLMs is in tackling software engineering challenges, particularly in resolving real-world tasks on GitHub by fixing code based on the issues reported by the users. However, many current approaches rely on proprietary LLMs, which limits reproducibility, accessibility, and transparency. The critical components of LLMs for addressing software engineering issues and how their capabilities can be effectively enhanced remain unclear. To address these challenges, we introduce SWE-Fixer, a novel open-source LLM designed to effectively and efficiently resolve GitHub issues. SWE-Fixer comprises two essential modules: a code file retrieval module and a code editing module. The retrieval module employs BM25 along with a lightweight LLM model to achieve coarse-to-fine file retrieval. Subsequently, the code editing module utilizes the other LLM model to generate patches for the identified files. Then, to mitigate the lack of publicly available datasets, we compile an extensive dataset that includes 110K GitHub issues along with their corresponding patches, and train the two modules of SWE-Fixer separately. We assess our approach on the SWE-Bench Lite and Verified benchmarks, achieving state-of-the-art performance among open-source models with scores of 23.3% and 30.2%, respectively. These outcomes highlight the efficacy of our approach. We will make our model, dataset, and code publicly available at this https URL.
• 摘要：大型语言模型 (LLM) 在各种复杂任务中表现出色。LLM 的一个重要应用是解决软件工程挑战，特别是通过根据用户报告的问题修复代码来解决 GitHub 上的实际任务。然而，许多当前方法依赖于专有 LLM，这限制了可重复性、可访问性和透明度。LLM 解决软件工程问题的关键组件以及如何有效增强其功能仍不清楚。为了应对这些挑战，我们引入了 SWE-Fixer，这是一种新颖的开源 LLM，旨在有效、高效地解决 GitHub 问题。SWE-Fixer 包含两个基本模块：代码文件检索模块和代码编辑模块。检索模块采用 BM25 和轻量级 LLM 模型来实现从粗到细的文件检索。随后，代码编辑模块利用另一个 LLM 模型为已识别的文件生成补丁。然后，为了缓解缺乏公开可用数据集的问题，我们编制了一个包含 110K 个 GitHub 问题及其相应补丁的广泛数据集，并分别训练 SWE-Fixer 的两个模块。我们在 SWE-Bench Lite 和 Verified 基准上评估了我们的方法，在开源模型中分别以 23.3% 和 30.2% 的得分实现了最先进的性能。这些结果凸显了我们方法的有效性。我们将在此 https URL 上公开我们的模型、数据集和代码。

Title: Centurio: On Drivers of Multilingual Ability of Large Vision-Language Model

• Authors: Gregor Geigle, Florian Schneider, Carolin Holtermann, Chris Biemann, Radu Timofte, Anne Lauscher, Goran Glavaš
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2501.05122
• Pdf URL: https://arxiv.org/pdf/2501.05122
• Copy Paste: [[2501.05122]] Centurio: On Drivers of Multilingual Ability of Large Vision-Language Model(https://arxiv.org/abs/2501.05122)
• Keywords: language model
• Abstract: Most Large Vision-Language Models (LVLMs) to date are trained predominantly on English data, which makes them struggle to understand non-English input and fail to generate output in the desired target language. Existing efforts mitigate these issues by adding multilingual training data, but do so in a largely ad-hoc manner, lacking insight into how different training mixes tip the scale for different groups of languages. In this work, we present a comprehensive investigation into the training strategies for massively multilingual LVLMs. First, we conduct a series of multi-stage experiments spanning 13 downstream vision-language tasks and 43 languages, systematically examining: (1) the number of training languages that can be included without degrading English performance and (2) optimal language distributions of pre-training as well as (3) instruction-tuning data. Further, we (4) investigate how to improve multilingual text-in-image understanding, and introduce a new benchmark for the task. Surprisingly, our analysis reveals that one can (i) include as many as 100 training languages simultaneously (ii) with as little as 25-50\% of non-English data, to greatly improve multilingual performance while retaining strong English performance. We further find that (iii) including non-English OCR data in pre-training and instruction-tuning is paramount for improving multilingual text-in-image understanding. Finally, we put all our findings together and train Centurio, a 100-language LVLM, offering state-of-the-art performance in an evaluation covering 14 tasks and 56 languages.
• 摘要：迄今为止，大多数大型视觉语言模型 (LVLM) 主要基于英语数据进行训练，这使得它们难以理解非英语输入并且无法生成所需目标语言的输出。现有的努力通过添加多语言训练数据来缓解这些问题，但这样做很大程度上是临时性的，缺乏对不同训练组合如何影响不同语言组影响的洞察。在这项工作中，我们对大规模多语言 LVLM 的训练策略进行了全面研究。首先，我们进行了一系列多阶段实验，涵盖 13 个下游视觉语言任务和 43 种语言，系统地检查：(1) 在不降低英语性能的情况下可以包含的训练语言数量和 (2) 预训练的最佳语言分布以及 (3) 指令调整数据。此外，我们 (4) 研究如何改进多语言图像中的文本理解，并为该任务引入新的基准。令人惊讶的是，我们的分析表明，人们可以 (i) 同时包含多达 100 种训练语言 (ii) 仅使用 25-50\% 的非英语数据，从而大大提高多语言性能，同时保持强大的英语性能。我们进一步发现 (iii) 在预训练和指令调整中包含非英语 OCR 数据对于提高多语言文本图像理解至关重要。最后，我们将所有发现汇总在一起并训练 Centurio，这是一个 100 种语言的 LVLM，在涵盖 14 个任务和 56 种语言的评估中提供了最先进的性能。

Title: Biomedical Relation Extraction via Adaptive Document-Relation Cross-Mapping and Concept Unique Identifier

• Authors: Yufei Shang, Yanrong Guo, Shijie Hao, Richang Hong
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.05155
• Pdf URL: https://arxiv.org/pdf/2501.05155
• Copy Paste: [[2501.05155]] Biomedical Relation Extraction via Adaptive Document-Relation Cross-Mapping and Concept Unique Identifier(https://arxiv.org/abs/2501.05155)
• Keywords: language model, gpt, llm, prompt, chat, retrieval-augmented generation
• Abstract: Document-Level Biomedical Relation Extraction (Bio-RE) aims to identify relations between biomedical entities within extensive texts, serving as a crucial subfield of biomedical text mining. Existing Bio-RE methods struggle with cross-sentence inference, which is essential for capturing relations spanning multiple sentences. Moreover, previous methods often overlook the incompleteness of documents and lack the integration of external knowledge, limiting contextual richness. Besides, the scarcity of annotated data further hampers model training. Recent advancements in large language models (LLMs) have inspired us to explore all the above issues for document-level Bio-RE. Specifically, we propose a document-level Bio-RE framework via LLM Adaptive Document-Relation Cross-Mapping (ADRCM) Fine-Tuning and Concept Unique Identifier (CUI) Retrieval-Augmented Generation (RAG). First, we introduce the Iteration-of-REsummary (IoRs) prompt for solving the data scarcity issue. In this way, Bio-RE task-specific synthetic data can be generated by guiding ChatGPT to focus on entity relations and iteratively refining synthetic data. Next, we propose ADRCM fine-tuning, a novel fine-tuning recipe that establishes mappings across different documents and relations, enhancing the model's contextual understanding and cross-sentence inference capabilities. Finally, during the inference, a biomedical-specific RAG approach, named CUI RAG, is designed to leverage CUIs as indexes for entities, narrowing the retrieval scope and enriching the relevant document contexts. Experiments conducted on three Bio-RE datasets (GDA, CDR, and BioRED) demonstrate the state-of-the-art performance of our proposed method by comparing it with other related works.
• 摘要：文档级生物医学关系提取 (Bio-RE) 旨在识别大量文本中生物医学实体之间的关系，是生物医学文本挖掘的一个重要子领域。现有的 Bio-RE 方法难以进行跨句推理，而跨句推理对于捕捉跨多个句子的关系至关重要。此外，以前的方法往往忽视文档的不完整性，缺乏外部知识的整合，限制了上下文丰富性。此外，注释数据的稀缺进一步阻碍了模型训练。大型语言模型 (LLM) 的最新进展启发我们探索文档级 Bio-RE 的所有上述问题。具体而言，我们通过 LLM 自适应文档关系交叉映射 (ADRCM) 微调和概念唯一标识符 (CUI) 检索增强生成 (RAG) 提出了一个文档级 Bio-RE 框架。首先，我们引入了迭代摘要 (IoRs) 提示来解决数据稀缺问题。这样，通过引导 ChatGPT 关注实体关系并迭代细化合成数据，可以生成 Bio-RE 任务特定的合成数据。接下来，我们提出了 ADRCM 微调，这是一种新颖的微调方法，可建立不同文档和关系之间的映射，从而增强模型的上下文理解和跨句推理能力。最后，在推理过程中，设计了一种生物医学特定的 RAG 方法，称为 CUI RAG，旨在利用 CUI 作为实体的索引，缩小检索范围并丰富相关文档上下文。在三个 Bio-RE 数据集（GDA、CDR 和 BioRED）上进行的实验通过将我们提出的方法与其他相关工作进行比较，证明了其最先进的性能。

Title: Leveraging Large Language Models for Zero-shot Lay Summarisation in Biomedicine and Beyond

• Authors: Tomas Goldsack, Carolina Scarton, Chenghua Lin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.05224
• Pdf URL: https://arxiv.org/pdf/2501.05224
• Copy Paste: [[2501.05224]] Leveraging Large Language Models for Zero-shot Lay Summarisation in Biomedicine and Beyond(https://arxiv.org/abs/2501.05224)
• Keywords: language model, llm
• Abstract: In this work, we explore the application of Large Language Models to zero-shot Lay Summarisation. We propose a novel two-stage framework for Lay Summarisation based on real-life processes, and find that summaries generated with this method are increasingly preferred by human judges for larger models. To help establish best practices for employing LLMs in zero-shot settings, we also assess the ability of LLMs as judges, finding that they are able to replicate the preferences of human judges. Finally, we take the initial steps towards Lay Summarisation for Natural Language Processing (NLP) articles, finding that LLMs are able to generalise to this new domain, and further highlighting the greater utility of summaries generated by our proposed approach via an in-depth human evaluation.
• 摘要：在这项工作中，我们探索了大型语言模型在零样本非专业摘要中的应用。我们根据现实生活过程提出了一种新颖的两阶段非专业摘要框架，并发现用这种方法生成的摘要越来越受到大型模型的人类评判者的青睐。为了帮助建立在零样本设置中使用 LLM 的最佳实践，我们还评估了 LLM 作为评判者的能力，发现它们能够复制人类评判者的偏好。最后，我们迈出了自然语言处理 (NLP) 文章的非专业摘要的初步步伐，发现 LLM 能够推广到这个新领域，并进一步强调了通过深入的人工评估，我们提出的方法生成的摘要具有更大的实用性。

Title: Optimizing Estonian TV Subtitles with Semi-supervised Learning and LLMs

• Authors: Artem Fedorchenko, Tanel Alumäe
• Subjects: cs.CL, cs.AI, cs.LG, eess.AS
• Abstract URL: https://arxiv.org/abs/2501.05234
• Pdf URL: https://arxiv.org/pdf/2501.05234
• Copy Paste: [[2501.05234]] Optimizing Estonian TV Subtitles with Semi-supervised Learning and LLMs(https://arxiv.org/abs/2501.05234)
• Keywords: language model, llm
• Abstract: This paper presents an approach for generating high-quality, same-language subtitles for Estonian TV content. We fine-tune the Whisper model on human-generated Estonian subtitles and enhance it with iterative pseudo-labeling and large language model (LLM) based post-editing. Our experiments demonstrate notable subtitle quality improvement through pseudo-labeling with an unlabeled dataset. We find that applying LLM-based editing at test time enhances subtitle accuracy, while its use during training does not yield further gains. This approach holds promise for creating subtitle quality close to human standard and could be extended to real-time applications.
• 摘要：本文介绍了一种为爱沙尼亚电视内容生成高质量同语字幕的方法。我们在人工生成的爱沙尼亚语字幕上对 Whisper 模型进行了微调，并通过迭代伪标记和基于大型语言模型 (LLM) 的后期编辑对其进行了增强。我们的实验表明，通过使用未标记的数据集进行伪标记，字幕质量得到了显著改善。我们发现，在测试时应用基于 LLM 的编辑可以提高字幕的准确性，而在训练期间使用这种方法不会带来进一步的收益。这种方法有望创造出接近人类标准的字幕质量，并且可以扩展到实时应用。

Title: Enhancing Plagiarism Detection in Marathi with a Weighted Ensemble of TF-IDF and BERT Embeddings for Low-Resource Language Processing

• Authors: Atharva Mutsaddi, Aditya Choudhary
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.05260
• Pdf URL: https://arxiv.org/pdf/2501.05260
• Copy Paste: [[2501.05260]] Enhancing Plagiarism Detection in Marathi with a Weighted Ensemble of TF-IDF and BERT Embeddings for Low-Resource Language Processing(https://arxiv.org/abs/2501.05260)
• Keywords: language model
• Abstract: Plagiarism involves using another person's work or concepts without proper attribution, presenting them as original creations. With the growing amount of data communicated in regional languages such as Marathi -- one of India's regional languages -- it is crucial to design robust plagiarism detection systems tailored for low-resource languages. Language models like Bidirectional Encoder Representations from Transformers (BERT) have demonstrated exceptional capability in text representation and feature extraction, making them essential tools for semantic analysis and plagiarism detection. However, the application of BERT for low-resource languages remains under-explored, particularly in the context of plagiarism detection. This paper presents a method to enhance the accuracy of plagiarism detection for Marathi texts using BERT sentence embeddings in conjunction with Term Frequency-Inverse Document Frequency (TF-IDF) feature representation. This approach effectively captures statistical, semantic, and syntactic aspects of text features through a weighted voting ensemble of machine learning models.
• 摘要：剽窃是指未注明来源而使用他人作品或概念，将其当作原创作品。随着使用马拉地语（印度的区域性语言之一）等区域性语言交流的数据量不断增长，设计针对低资源语言的强大剽窃检测系统至关重要。诸如 Transformer 的双向编码器表示 (BERT) 之类的语言模型在文本表示和特征提取方面表现出色，使其成为语义分析和剽窃检测的重要工具。但是，BERT 在低资源语言中的应用仍未得到充分探索，特别是在剽窃检测方面。本文介绍了一种使用 BERT 句子嵌入结合词频-逆文档频率 (TF-IDF) 特征表示来提高马拉地语文本剽窃检测准确性的方法。该方法通过机器学习模型的加权投票集成有效地捕获文本特征的统计、语义和句法方面。

Title: Stream Aligner: Efficient Sentence-Level Alignment via Distribution Induction

• Authors: Hantao Lou, Jiaming Ji, Kaile Wang, Yaodong Yang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.05336
• Pdf URL: https://arxiv.org/pdf/2501.05336
• Copy Paste: [[2501.05336]] Stream Aligner: Efficient Sentence-Level Alignment via Distribution Induction(https://arxiv.org/abs/2501.05336)
• Keywords: language model, llm, chat
• Abstract: The rapid advancement of large language models (LLMs) has led to significant improvements in their capabilities, but also to increased concerns about their alignment with human values and intentions. Current alignment strategies, including adaptive training and inference-time methods, have demonstrated potential in this area. However, these approaches still struggle to balance deployment complexity and capability across various tasks and difficulties. In this work, we introduce the Streaming Distribution Induce Aligner (Stream Aligner), a novel alignment paradigm that combines efficiency with enhanced performance in various tasks throughout the generation process. Stream Aligner achieves dynamic sentence-level correction by using a small model to learn the preferences of the suffix sentence, iteratively correcting the suffix sentence output by the upstream model, and then using the corrected sentence to replace the suffix sentence in subsequent generations. Compared to Aligner, our experiments demonstrate that Stream Aligner reduces reliance on the capabilities of additional models, enhances the reasoning abilities of LLMs, and decreases latency during user interaction. Specifically, Stream Aligner-2B model has achieved an improvement of 76.1% in helpfulness, 36.0% in harmlessness on the tested Llama2-70B-chat model, and Stream Aligner-8B has achieved an improvement of 3.5% on the math ability of the tested Llama3-70B-Instruct model.
• 摘要：大型语言模型 (LLM) 的快速发展显著提高了其能力，但也引发了人们对其是否符合人类价值观和意图的担忧。当前的对齐策略，包括自适应训练和推理时间方法，已在该领域展现出潜力。然而，这些方法仍然难以在各种任务和难度之间平衡部署复杂性和能力。在这项工作中，我们引入了流式分布诱导对齐器 (Stream Aligner)，这是一种新颖的对齐范式，在整个生成过程中将效率与增强的各种任务的性能相结合。Stream Aligner 通过使用小模型学习后缀句子的偏好，迭代地纠正上游模型输出的后缀句子，然后在后续生成中使用纠正后的句子替换后缀句子，从而实现动态句子级校正。与 Aligner 相比，我们的实验表明，Stream Aligner 减少了对其他模型功能的依赖，增强了 LLM 的推理能力，并减少了用户交互过程中的延迟。具体来说，Stream Aligner-2B模型在测试的Llama2-70B-chat模型上实现了76.1%的帮助性提升、36.0%的无害性提升，Stream Aligner-8B在测试的Llama3-70B-Instruct模型上实现了3.5%的数学能力提升。

Title: FairCode: Evaluating Social Bias of LLMs in Code Generation

• Authors: Yongkang Du, Jen-tse Huang, Jieyu Zhao, Lu Lin
• Subjects: cs.CL, cs.SE
• Abstract URL: https://arxiv.org/abs/2501.05396
• Pdf URL: https://arxiv.org/pdf/2501.05396
• Copy Paste: [[2501.05396]] FairCode: Evaluating Social Bias of LLMs in Code Generation(https://arxiv.org/abs/2501.05396)
• Keywords: language model, llm, prompt
• Abstract: Large language models (LLMs) have demonstrated significant capability in code generation, drawing increasing attention to the evaluation of the quality and safety of their outputs. However, research on bias in code generation remains limited. Existing studies typically assess bias by applying malicious prompts or reapply tasks and dataset for discriminative models. Given that LLMs are often aligned with human values and that prior datasets are not fully optimized for code-related tasks, there is a pressing need for benchmarks specifically designed for evaluating code models. In this study, we introduce FairCode, a novel benchmark for evaluating bias in code generation. FairCode comprises two tasks: function implementation and test case generation, each evaluating social bias through diverse scenarios. Additionally, we propose a new metric, FairScore, to assess model performance on this benchmark. We conduct experiments on widely used LLMs and provide a comprehensive analysis of the results. The findings reveal that all tested LLMs exhibit bias. The code is available at this https URL.
• 摘要：大型语言模型 (LLM) 在代码生成方面表现出了显著的能力，引起了对其输出质量和安全性评估的日益关注。然而，对代码生成偏差的研究仍然有限。现有研究通常通过应用恶意提示或重新应用判别模型的任务和数据集来评估偏差。鉴于 LLM 通常与人类价值观一致，并且先前的数据集并未完全针对与代码相关的任务进行优化，因此迫切需要专门为评估代码模型而设计的基准。在本研究中，我们引入了 FairCode，这是一种用于评估代码生成偏差的新基准。FairCode 包括两个任务：函数实现和测试用例生成，每个任务都通过不同的场景评估社会偏差。此外，我们提出了一个新的指标 FairScore，以评估该基准上的模型性能。我们对广泛使用的 LLM 进行了实验，并对结果进行了全面分析。研究结果表明，所有测试的 LLM 都表现出偏差。代码可在此 https URL 上获得。

Title: LongProc: Benchmarking Long-Context Language Models on Long Procedural Generation

• Authors: Xi Ye, Fangcong Yin, Yinghui He, Joie Zhang, Howard Yen, Tianyu Gao, Greg Durrett, Danqi Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.05414
• Pdf URL: https://arxiv.org/pdf/2501.05414
• Copy Paste: [[2501.05414]] LongProc: Benchmarking Long-Context Language Models on Long Procedural Generation(https://arxiv.org/abs/2501.05414)
• Keywords: language model, gpt
• Abstract: Existing benchmarks for evaluating long-context language models (LCLMs) primarily focus on long-context recall, requiring models to produce short responses based on a few critical snippets while processing thousands of irrelevant tokens. We introduce LongProc (Long Procedural Generation), a new benchmark that requires both the integration of highly dispersed information and long-form generation. LongProc consists of six diverse procedural generation tasks, such as extracting structured information from HTML pages into a TSV format and executing complex search procedures to create travel plans. These tasks challenge LCLMs by testing their ability to follow detailed procedural instructions, synthesize and reason over dispersed information, and generate structured, long-form outputs (up to 8K tokens). Furthermore, as these tasks adhere to deterministic procedures and yield structured outputs, they enable reliable rule-based evaluation. We evaluate 17 LCLMs on LongProc across three difficulty levels, with maximum numbers of output tokens set at 500, 2K, and 8K. Notably, while all tested models claim a context window size above 32K tokens, open-weight models typically falter on 2K-token tasks, and closed-source models like GPT-4o show significant degradation on 8K-token tasks. Further analysis reveals that LCLMs struggle to maintain long-range coherence in long-form generations. These findings highlight critical limitations in current LCLMs and suggest substantial room for improvement. Data and code available at: this https URL
• 摘要：现有的用于评估长上下文语言模型 (LCLM) 的基准主要侧重于长上下文回忆，要求模型根据一些关键片段生成简短响应，同时处理数千个不相关的标记。我们引入了 LongProc（长程序生成），这是一个新的基准，它需要集成高度分散的信息和长格式生成。LongProc 包含六个不同的程序生成任务，例如将 HTML 页面中的结构化信息提取为 TSV 格式，并执行复杂的搜索程序来创建旅行计划。这些任务通过测试 LCLM 遵循详细的程序说明、对分散信息进行综合和推理以及生成结构化的长格式输出（最多 8K 个标记）的能力来挑战 LCLM。此外，由于这些任务遵循确定性程序并产生结构化输出，因此它们可以实现可靠的基于规则的评估。我们在三个难度级别上对 LongProc 上的 17 个 LCLM 进行了评估，最大输出标记数设置为 500、2K 和 8K。值得注意的是，虽然所有测试模型都声称上下文窗口大小超过 32K 个 token，但开放权重模型通常在 2K 个 token 任务上表现不佳，而 GPT-4o 等闭源模型在 8K 个 token 任务上表现明显下降。进一步分析表明，LCLM 难以在长格式生成中保持长距离连贯性。这些发现凸显了当前 LCLM 的关键局限性，并表明有很大的改进空间。数据和代码可从以下网址获取：此 https URL

Title: A survey of textual cyber abuse detection using cutting-edge language models and large language models

• Authors: Jose A. Diaz-Garcia, Joao Paulo Carvalho
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.05443
• Pdf URL: https://arxiv.org/pdf/2501.05443
• Copy Paste: [[2501.05443]] A survey of textual cyber abuse detection using cutting-edge language models and large language models(https://arxiv.org/abs/2501.05443)
• Keywords: language model, llm
• Abstract: The success of social media platforms has facilitated the emergence of various forms of online abuse within digital communities. This abuse manifests in multiple ways, including hate speech, cyberbullying, emotional abuse, grooming, and sexting. In this paper, we present a comprehensive analysis of the different forms of abuse prevalent in social media, with a particular focus on how emerging technologies, such as Language Models (LMs) and Large Language Models (LLMs), are reshaping both the detection and generation of abusive content within these networks. We delve into the mechanisms through which social media abuse is perpetuated, exploring the psychological and social impact. Additionally, we examine the dual role of advanced language models-highlighting their potential to enhance automated detection systems for abusive behavior while also acknowledging their capacity to generate harmful content. This paper aims to contribute to the ongoing discourse on online safety and ethics, offering insights into the evolving landscape of cyberabuse and the technological innovations that both mitigate and exacerbate it.
• 摘要：社交媒体平台的成功促进了数字社区中各种形式的网络虐待的出现。这种虐待表现为多种形式，包括仇恨言论、网络欺凌、情感虐待、诱骗和色情短信。在本文中，我们对社交媒体中普遍存在的不同形式的虐待进行了全面分析，特别关注语言模型 (LM) 和大型语言模型 (LLM) 等新兴技术如何重塑这些网络中虐待内容的检测和生成。我们深入研究了社交媒体虐待的机制，探索了心理和社会影响。此外，我们研究了高级语言模型的双重作用 - 强调它们在增强虐待行为自动检测系统方面的潜力，同时也承认它们生成有害内容的能力。本文旨在为正在进行的网络安全和道德讨论做出贡献，深入了解网络虐待的不断发展的格局以及缓解和加剧网络虐待的技术创新。