2024-12-24

Title: Decoding Linguistic Nuances in Mental Health Text Classification Using Expressive Narrative Stories

• Authors: Jinwen Tang, Qiming Guo, Yunxin Zhao, Yi Shang
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.16302
• Pdf URL: https://arxiv.org/pdf/2412.16302
• Copy Paste: [[2412.16302]] Decoding Linguistic Nuances in Mental Health Text Classification Using Expressive Narrative Stories(https://arxiv.org/abs/2412.16302)
• Keywords: language model
• Abstract: Recent advancements in NLP have spurred significant interest in analyzing social media text data for identifying linguistic features indicative of mental health issues. However, the domain of Expressive Narrative Stories (ENS)-deeply personal and emotionally charged narratives that offer rich psychological insights-remains underexplored. This study bridges this gap by utilizing a dataset sourced from Reddit, focusing on ENS from individuals with and without self-declared depression. Our research evaluates the utility of advanced language models, BERT and MentalBERT, against traditional models. We find that traditional models are sensitive to the absence of explicit topic-related words, which could risk their potential to extend applications to ENS that lack clear mental health terminology. Despite MentalBERT is design to better handle psychiatric contexts, it demonstrated a dependency on specific topic words for classification accuracy, raising concerns about its application when explicit mental health terms are sparse (P-value<0.05). In contrast, BERT exhibited minimal sensitivity to the absence of topic words in ENS, suggesting its superior capability to understand deeper linguistic features, making it more effective for real-world applications. Both BERT and MentalBERT excel at recognizing linguistic nuances and maintaining classification accuracy even when narrative order is disrupted. This resilience is statistically significant, with sentence shuffling showing substantial impacts on model performance (P-value<0.05), especially evident in ENS comparisons between individuals with and without mental health declarations. These findings underscore the importance of exploring ENS for deeper insights into mental health-related narratives, advocating for a nuanced approach to mental health text analysis that moves beyond mere keyword detection.
• 摘要：NLP 的最新进展激发了人们对分析社交媒体文本数据以识别表明心理健康问题的语言特征的极大兴趣。然而，表达性叙事故事 (ENS) 领域——提供丰富心理见解的深度个人化和情感化的叙事——仍未得到充分探索。这项研究利用来自 Reddit 的数据集弥补了这一空白，重点关注自称患有和不患有抑郁症的个人的 ENS。我们的研究评估了高级语言模型 BERT 和 MentalBERT 与传统模型的效用。我们发现传统模型对明确主题相关词的缺失很敏感，这可能会危及它们将应用扩展到缺乏明确心理健康术语的 ENS 的潜力。尽管 MentalBERT 旨在更好地处理精神病学背景，但它表现出对特定主题词的依赖，以提高分类准确性，这引起了人们对其在明确的心理健康术语稀疏时的应用的担忧（P 值 <0.05）。相比之下，BERT 对 ENS 中主题词缺失的敏感度极低，表明其具有理解更深层次语言特征的卓越能力，使其在实际应用中更有效。BERT 和 MentalBERT 都擅长识别语言细微差别，即使在叙述顺序被打乱的情况下也能保持分类准确性。这种弹性具有统计学意义，句子改组对模型性能有显著影响（P 值 <0.05），在有和没有心理健康声明的个体之间的 ENS 比较中尤其明显。这些发现强调了探索 ENS 以深入了解心理健康相关叙述的重要性，提倡采用一种细致入微的心理健康文本分析方法，而不仅仅是关键词检测。

Title: Deliberative Alignment: Reasoning Enables Safer Language Models

• Authors: Melody Y. Guan, Manas Joglekar, Eric Wallace, Saachi Jain, Boaz Barak, Alec Heylar, Rachel Dias, Andrea Vallone, Hongyu Ren, Jason Wei, Hyung Won Chung, Sam Toyer, Johannes Heidecke, Alex Beutel, Amelia Glaese
• Subjects: cs.CL, cs.AI, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.16339
• Pdf URL: https://arxiv.org/pdf/2412.16339
• Copy Paste: [[2412.16339]] Deliberative Alignment: Reasoning Enables Safer Language Models(https://arxiv.org/abs/2412.16339)
• Keywords: language model, chain-of-thought
• Abstract: As large-scale language models increasingly impact safety-critical domains, ensuring their reliable adherence to well-defined principles remains a fundamental challenge. We introduce Deliberative Alignment, a new paradigm that directly teaches the model safety specifications and trains it to explicitly recall and accurately reason over the specifications before answering. We used this approach to align OpenAI's o-series models, and achieved highly precise adherence to OpenAI's safety policies, without requiring human-written chain-of-thoughts or answers. Deliberative Alignment pushes the Pareto frontier by simultaneously increasing robustness to jailbreaks while decreasing overrefusal rates, and also improves out-of-distribution generalization. We demonstrate that reasoning over explicitly specified policies enables more scalable, trustworthy, and interpretable alignment.
• 摘要：随着大规模语言模型对安全关键领域的影响越来越大，确保它们可靠地遵守明确定义的原则仍然是一项根本挑战。我们引入了审议对齐，这是一种新范式，它直接教授模型安全规范，并训练它在回答之前明确回忆和准确推理规范。我们使用这种方法来对齐 OpenAI 的 o 系列模型，并实现了对 OpenAI 安全策略的高度精确遵守，而无需人工编写的思路或答案。审议对齐通过同时提高对越狱的鲁棒性并降低过度拒绝率来推动帕累托前沿，同时还提高了分布外的泛化能力。我们证明，对明确指定的策略进行推理可以实现更具可扩展性、可信度和可解释性的对齐。

Title: Human-Readable Adversarial Prompts: An Investigation into LLM Vulnerabilities Using Situational Context

• Authors: Nilanjana Das, Edward Raff, Manas Gaur
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.16359
• Pdf URL: https://arxiv.org/pdf/2412.16359
• Copy Paste: [[2412.16359]] Human-Readable Adversarial Prompts: An Investigation into LLM Vulnerabilities Using Situational Context(https://arxiv.org/abs/2412.16359)
• Keywords: gpt, llm, prompt
• Abstract: Previous research on LLM vulnerabilities often relied on nonsensical adversarial prompts, which were easily detectable by automated methods. We address this gap by focusing on human-readable adversarial prompts, a more realistic and potent threat. Our key contributions are situation-driven attacks leveraging movie scripts to create contextually relevant, human-readable prompts that successfully deceive LLMs, adversarial suffix conversion to transform nonsensical adversarial suffixes into meaningful text, and AdvPrompter with p-nucleus sampling, a method to generate diverse, human-readable adversarial suffixes, improving attack efficacy in models like GPT-3.5 and Gemma 7B. Our findings demonstrate that LLMs can be tricked by sophisticated adversaries into producing harmful responses with human-readable adversarial prompts and that there exists a scope for improvement when it comes to robust LLMs.
• 摘要：之前对 LLM 漏洞的研究通常依赖于无意义的对抗性提示，这些提示很容易被自动化方法检测到。我们通过专注于人类可读的对抗性提示（一种更现实、更强大的威胁）来解决这一问题。我们的主要贡献是利用电影脚本创建上下文相关、人类可读的提示的情境驱动攻击，成功欺骗 LLM，对抗性后缀转换将无意义的对抗性后缀转换为有意义的文本，以及使用 p-nucleus 采样的 AdvPrompter，这是一种生成多样化、人类可读的对抗性后缀的方法，可提高 GPT-3.5 和 Gemma 7B 等模型的攻击效果。我们的研究结果表明，LLM 可能会被老练的对手欺骗，使用人类可读的对抗性提示产生有害的响应，并且在稳健的 LLM 方面还有改进的空间。

Title: Overview of the First Workshop on Language Models for Low-Resource Languages (LoResLM 2025)

• Authors: Hansi Hettiarachchi, Tharindu Ranasinghe, Paul Rayson, Ruslan Mitkov, Mohamed Gaber, Damith Premasiri, Fiona Anting Tan, Lasitha Uyangodage
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.16365
• Pdf URL: https://arxiv.org/pdf/2412.16365
• Copy Paste: [[2412.16365]] Overview of the First Workshop on Language Models for Low-Resource Languages (LoResLM 2025)(https://arxiv.org/abs/2412.16365)
• Keywords: language model
• Abstract: The first Workshop on Language Models for Low-Resource Languages (LoResLM 2025) was held in conjunction with the 31st International Conference on Computational Linguistics (COLING 2025) in Abu Dhabi, United Arab Emirates. This workshop mainly aimed to provide a forum for researchers to share and discuss their ongoing work on language models (LMs) focusing on low-resource languages, following the recent advancements in neural language models and their linguistic biases towards high-resource languages. LoResLM 2025 attracted notable interest from the natural language processing (NLP) community, resulting in 35 accepted papers from 52 submissions. These contributions cover a broad range of low-resource languages from eight language families and 13 diverse research areas, paving the way for future possibilities and promoting linguistic inclusivity in NLP.
• 摘要：首届低资源语言模型研讨会 (LoResLM 2025) 与第 31 届国际计算语言学会议 (COLING 2025) 在阿拉伯联合酋长国阿布扎比举行。本次研讨会的主要目的是为研究人员提供一个论坛，分享和讨论他们正在进行的语言模型 (LM) 研究，重点关注低资源语言，关注神经语言模型的最新进展及其对高资源语言的语言偏见。LoResLM 2025 引起了自然语言处理 (NLP) 社区的极大兴趣，从 52 篇提交的论文中最终有 35 篇被接受。这些贡献涵盖了来自八个语系和 13 个不同研究领域的广泛低资源语言，为未来的可能性铺平了道路，并促进了 NLP 的语言包容性。

Title: Application of Multimodal Large Language Models in Autonomous Driving

• Authors: Md Robiul Islam
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.16410
• Pdf URL: https://arxiv.org/pdf/2412.16410
• Copy Paste: [[2412.16410]] Application of Multimodal Large Language Models in Autonomous Driving(https://arxiv.org/abs/2412.16410)
• Keywords: language model, llm
• Abstract: In this era of technological advancements, several cutting-edge techniques are being implemented to enhance Autonomous Driving (AD) systems, focusing on improving safety, efficiency, and adaptability in complex driving environments. However, AD still faces some problems including performance limitations. To address this problem, we conducted an in-depth study on implementing the Multi-modal Large Language Model. We constructed a Virtual Question Answering (VQA) dataset to fine-tune the model and address problems with the poor performance of MLLM on AD. We then break down the AD decision-making process by scene understanding, prediction, and decision-making. Chain of Thought has been used to make the decision more perfectly. Our experiments and detailed analysis of Autonomous Driving give an idea of how important MLLM is for AD.
• 摘要：在这个技术进步的时代，人们正在实施多种尖端技术来增强自动驾驶 (AD) 系统，重点是提高复杂驾驶环境中的安全性、效率和适应性。然而，AD 仍然面临一些问题，包括性能限制。为了解决这个问题，我们对实施多模态大型语言模型进行了深入研究。我们构建了一个虚拟问答 (VQA) 数据集来微调模型并解决 MLLM 在 AD 上表现不佳的问题。然后，我们通过场景理解、预测和决策来分解 AD 决策过程。思路链已被用来更完美地做出决策。我们的实验和对自动驾驶的详细分析表明了 MLLM 对 AD 的重要性。

Title: InfoTech Assistant : A Multimodal Conversational Agent for InfoTechnology Web Portal Queries

• Authors: Sai Surya Gadiraju, Duoduo Liao, Akhila Kudupudi, Santosh Kasula, Charitha Chalasani
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.16412
• Pdf URL: https://arxiv.org/pdf/2412.16412
• Copy Paste: [[2412.16412]] InfoTech Assistant : A Multimodal Conversational Agent for InfoTechnology Web Portal Queries(https://arxiv.org/abs/2412.16412)
• Keywords: language model, llm, chat, retrieval-augmented generation, agent
• Abstract: This pilot study presents the development of the InfoTech Assistant, a domain-specific, multimodal chatbot engineered to address queries in bridge evaluation and infrastructure technology. By integrating web data scraping, large language models (LLMs), and Retrieval-Augmented Generation (RAG), the InfoTech Assistant provides accurate and contextually relevant responses. Data, including textual descriptions and images, are sourced from publicly available documents on the InfoTechnology website and organized in JSON format to facilitate efficient querying. The architecture of the system includes an HTML-based interface and a Flask back end connected to the Llama 3.1 model via LLM Studio. Evaluation results show approximately 95 percent accuracy on domain-specific tasks, with high similarity scores confirming the quality of response matching. This RAG-enhanced setup enables the InfoTech Assistant to handle complex, multimodal queries, offering both textual and visual information in its responses. The InfoTech Assistant demonstrates strong potential as a dependable tool for infrastructure professionals, delivering high accuracy and relevance in its domain-specific outputs.
• 摘要：这项初步研究介绍了 InfoTech Assistant 的开发，这是一个特定领域的多模式聊天机器人，旨在解决桥梁评估和基础设施技术中的查询。通过集成网络数据抓取、大型语言模型 (LLM) 和检索增强生成 (RAG)，InfoTech Assistant 提供准确且与上下文相关的响应。数据（包括文本描述和图像）来自 InfoTechnology 网站上的公开文档，并以 JSON 格式组织，以方便高效查询。该系统的架构包括一个基于 HTML 的界面和一个通过 LLM Studio 连接到 Llama 3.1 模型的 Flask 后端。评估结果显示，特定领域任务的准确率约为 95%，高相似度得分证实了响应匹配的质量。这种 RAG 增强设置使 InfoTech Assistant 能够处理复杂的多模式查询，在其响应中提供文本和视觉信息。InfoTech Assistant 展示了作为基础设施专业人员可靠工具的强大潜力，在其特定领域的输出中提供高准确性和相关性。

Title: Technical Report: Small Language Model for Japanese Clinical and Medicine

• Authors: Shogo Watanabe
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.16423
• Pdf URL: https://arxiv.org/pdf/2412.16423
• Copy Paste: [[2412.16423]] Technical Report: Small Language Model for Japanese Clinical and Medicine(https://arxiv.org/abs/2412.16423)
• Keywords: language model, llm
• Abstract: This report presents a small language model (SLM) for Japanese clinical and medicine, named NCVC-slm-1. This 1B parameters model was trained using Japanese text classified to be of high-quality. Moreover, NCVC-slm-1 was augmented with respect to clinical and medicine content that includes the variety of diseases, drugs, and examinations. Using a carefully designed pre-processing, a specialized morphological analyzer and tokenizer, this small and light-weight model performed not only to generate text but also indicated the feasibility of understanding clinical and medicine text. In comparison to other large language models, a fine-tuning NCVC-slm-1 demonstrated the highest scores on 6 tasks of total 8 on JMED-LLM. According to this result, SLM indicated the feasibility of performing several downstream tasks in the field of clinical and medicine. Hopefully, NCVC-slm-1 will be contributed to develop and accelerate the field of clinical and medicine for a bright future.
• 摘要：本报告介绍了一种用于日语临床和医学的小型语言模型 (SLM)，名为 NCVC-slm-1。该 1B 参数模型使用被归类为高质量的日语文本进行训练。此外，NCVC-slm-1 还针对包括各种疾病、药物和检查在内的临床和医学内容进行了增强。使用精心设计的预处理、专门的形态分析器和标记器，这个小型轻量级模型不仅可以生成文本，而且还表明了理解临床和医学文本的可行性。与其他大型语言模型相比，微调 NCVC-slm-1 在 JMED-LLM 的 8 项任务中的 6 项中表现出最高分数。根据这一结果，SLM 表明了在临床和医学领域执行多项下游任务的可行性。希望 NCVC-slm-1 能够为临床和医学领域的发展和加速做出贡献，实现美好的未来。

Title: Research on Violent Text Detection System Based on BERT-fasttext Model

• Authors: Yongsheng Yang, Xiaoying Wang
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2412.16455
• Pdf URL: https://arxiv.org/pdf/2412.16455
• Copy Paste: [[2412.16455]] Research on Violent Text Detection System Based on BERT-fasttext Model(https://arxiv.org/abs/2412.16455)
• Keywords: language model
• Abstract: In the digital age of today, the internet has become an indispensable platform for people's lives, work, and information exchange. However, the problem of violent text proliferation in the network environment has arisen, which has brought about many negative effects. In view of this situation, it is particularly important to build an effective system for cutting off violent text. The study of violent text cutting off based on the BERT-fasttext model has significant meaning. BERT is a pre-trained language model with strong natural language understanding ability, which can deeply mine and analyze text semantic information; Fasttext itself is an efficient text classification tool with low complexity and good effect, which can quickly provide basic judgments for text processing. By combining the two and applying them to the system for cutting off violent text, on the one hand, it can accurately identify violent text, and on the other hand, it can efficiently and reasonably cut off the content, preventing harmful information from spreading freely on the network. Compared with the single BERT model and fasttext, the accuracy was improved by 0.7% and 0.8%, respectively. Through this model, it is helpful to purify the network environment, maintain the health of network information, and create a positive, civilized, and harmonious online communication space for netizens, driving the development of social networking, information dissemination, and other aspects in a more benign direction.
• 摘要：在当今的数字化时代，互联网已经成为人们生活、工作、信息交流不可或缺的平台，然而网络环境中暴力文字泛滥的问题也随之出现，带来了诸多负面影响。针对这种情况，构建一套有效的暴力文字截断系统显得尤为重要。基于BERT-fasttext模型的暴力文字截断研究具有重要的意义。BERT是一个预训练的语言模型，具有较强的自然语言理解能力，可以深度挖掘和分析文本语义信息；Fasttext本身是一个高效的文本分类工具，复杂度低、效果好，可以为文本处理快速提供基本判断。将二者结合起来运用到暴力文字截断系统中，一方面可以准确识别暴力文字，另一方面可以高效合理地对内容进行截断，防止有害信息在网络上肆意传播。与单一的BERT模型和fasttext相比，准确率分别提升了0.7%和0.8%。通过这一模式，有利于净化网络环境，维护网络信息健康，为网民营造积极、文明、和谐的网络交往空间，推动社交网络、信息传播等方面朝着更加良性的方向发展。

Title: Transducer-Llama: Integrating LLMs into Streamable Transducer-based Speech Recognition

• Authors: Keqi Deng, Jinxi Guo, Yingyi Ma, Niko Moritz, Philip C. Woodland, Ozlem Kalinli, Mike Seltzer
• Subjects: cs.CL, eess.AS
• Abstract URL: https://arxiv.org/abs/2412.16464
• Pdf URL: https://arxiv.org/pdf/2412.16464
• Copy Paste: [[2412.16464]] Transducer-Llama: Integrating LLMs into Streamable Transducer-based Speech Recognition(https://arxiv.org/abs/2412.16464)
• Keywords: language model, llm
• Abstract: While large language models (LLMs) have been applied to automatic speech recognition (ASR), the task of making the model streamable remains a challenge. This paper proposes a novel model architecture, Transducer-Llama, that integrates LLMs into a Factorized Transducer (FT) model, naturally enabling streaming capabilities. Furthermore, given that the large vocabulary of LLMs can cause data sparsity issue and increased training costs for spoken language systems, this paper introduces an efficient vocabulary adaptation technique to align LLMs with speech system vocabularies. The results show that directly optimizing the FT model with a strong pre-trained LLM-based predictor using the RNN-T loss yields some but limited improvements over a smaller pre-trained LM predictor. Therefore, this paper proposes a weak-to-strong LM swap strategy, using a weak LM predictor during RNN-T loss training and then replacing it with a strong LLM. After LM replacement, the minimum word error rate (MWER) loss is employed to finetune the integration of the LLM predictor with the Transducer-Llama model. Experiments on the LibriSpeech and large-scale multi-lingual LibriSpeech corpora show that the proposed streaming Transducer-Llama approach gave a 17% relative WER reduction (WERR) over a strong FT baseline and a 32% WERR over an RNN-T baseline.
• 摘要：虽然大型语言模型 (LLM) 已应用于自动语音识别 (ASR)，但使模型可流式传输仍然是一个挑战。本文提出了一种新颖的模型架构 Transducer-Llama，将 LLM 集成到分解式 Transducer (FT) 模型中，自然而然地实现了流式传输功能。此外，考虑到 LLM 的词汇量大可能会导致数据稀疏问题并增加口语系统的训练成本，本文介绍了一种有效的词汇自适应技术，以使 LLM 与语音系统词汇保持一致。结果表明，使用 RNN-T 损失直接优化具有强预训练 LLM 预测器的 FT 模型与较小的预训练 LM 预测器相比，效果有所改善，但效果有限。因此，本文提出了一种弱到强 LM 交换策略，在 RNN-T 损失训练期间使用弱 LM 预测器，然后将其替换为强 LLM。在 LM 替换之后，使用最小词错误率 (MWER) 损失来微调 LLM 预测器与 Transducer-Llama 模型的集成。在 LibriSpeech 和大规模多语言 LibriSpeech 语料库上进行的实验表明，所提出的流式 Transducer-Llama 方法在强 FT 基线上实现了 17% 的相对 WER 减少 (WERR)，在 RNN-T 基线上实现了 32% 的 WERR。

Title: Chained Tuning Leads to Biased Forgetting

• Authors: Megan Ung, Alicia Sun, Samuel J. Bell, Bhaktipriya Radharapu, Levent Sagun, Adina Williams
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.16469
• Pdf URL: https://arxiv.org/pdf/2412.16469
• Copy Paste: [[2412.16469]] Chained Tuning Leads to Biased Forgetting(https://arxiv.org/abs/2412.16469)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) are often fine-tuned for use on downstream tasks, though this can degrade capabilities learned during previous training. This phenomenon, often referred to as catastrophic forgetting, has important potential implications for the safety of deployed models. In this work, we first show that models trained on downstream tasks forget their safety tuning to a greater extent than models trained in the opposite this http URL, we show that forgetting disproportionately impacts safety information about certain groups. To quantify this phenomenon, we define a new metric we term biased forgetting. We conduct a systematic evaluation of the effects of task ordering on forgetting and apply mitigations that can help the model recover from the forgetting observed. We hope our findings can better inform methods for chaining the finetuning of LLMs in continual learning settings to enable training of safer and less toxic models.
• 摘要：大型语言模型 (LLM) 通常会针对下游任务进行微调，但这可能会降低之前训练期间学习到的能力。这种现象通常被称为灾难性遗忘，对部署模型的安全性具有重要的潜在影响。在这项工作中，我们首先表明，在下游任务上训练的模型比在相反任务中训练的模型更容易忘记它们的安全性调整。我们表明，遗忘会对某些群体的安全信息产生不成比例的影响。为了量化这种现象，我们定义了一个新的指标，我们称之为有偏遗忘。我们对任务排序对遗忘的影响进行了系统评估，并应用了有助于模型从观察到的遗忘中恢复的缓解措施。我们希望我们的研究结果能够更好地为在持续学习环境中链接 LLM 微调的方法提供信息，从而能够训练更安全、毒性更小的模型。

Title: Evaluating the Performance of Large Language Models in Scientific Claim Detection and Classification

• Authors: Tanjim Bin Faruk
• Subjects: cs.CL, cs.CY, cs.LG, cs.SI
• Abstract URL: https://arxiv.org/abs/2412.16486
• Pdf URL: https://arxiv.org/pdf/2412.16486
• Copy Paste: [[2412.16486]] Evaluating the Performance of Large Language Models in Scientific Claim Detection and Classification(https://arxiv.org/abs/2412.16486)
• Keywords: language model, gpt, llm
• Abstract: The pervasive influence of social media during the COVID-19 pandemic has been a double-edged sword, enhancing communication while simultaneously propagating misinformation. This \textit{Digital Infodemic} has highlighted the urgent need for automated tools capable of discerning and disseminating factual content. This study evaluates the efficacy of Large Language Models (LLMs) as innovative solutions for mitigating misinformation on platforms like Twitter. LLMs, such as OpenAI's GPT and Meta's LLaMA, offer a pre-trained, adaptable approach that bypasses the extensive training and overfitting issues associated with traditional machine learning models. We assess the performance of LLMs in detecting and classifying COVID-19-related scientific claims, thus facilitating informed decision-making. Our findings indicate that LLMs have significant potential as automated fact-checking tools, though research in this domain is nascent and further exploration is required. We present a comparative analysis of LLMs' performance using a specialized dataset and propose a framework for their application in public health communication.
• 摘要：在 COVID-19 大流行期间，社交媒体的广泛影响是一把双刃剑，它既增强了沟通，又传播了错误信息。这场 \textit{数字信息流行病} 凸显了对能够辨别和传播事实内容的自动化工具的迫切需求。本研究评估了大型语言模型 (LLM) 作为减轻 Twitter 等平台上错误信息的创新解决方案的有效性。LLM（例如 OpenAI 的 GPT 和 Meta 的 LLaMA）提供了一种预先训练的、适应性强的方法，可以绕过与传统机器学习模型相关的大量训练和过度拟合问题。我们评估了 LLM 在检测和分类与 COVID-19 相关的科学主张方面的表现，从而促进了明智的决策。我们的研究结果表明，LLM 作为自动事实核查工具具有巨大的潜力，尽管该领域的研究尚处于起步阶段，需要进一步探索。我们使用专门的数据集对 LLM 的性能进行了比较分析，并提出了它们在公共卫生传播中的应用框架。

Title: Adapting Whisper for Code-Switching through Encoding Refining and Language-Aware Decoding

• Authors: Jiahui Zhao, Hao Shi, Chenrui Cui, Tianrui Wang, Hexin Liu, Zhaoheng Ni, Lingxuan Ye, Longbiao Wang
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2412.16507
• Pdf URL: https://arxiv.org/pdf/2412.16507
• Copy Paste: [[2412.16507]] Adapting Whisper for Code-Switching through Encoding Refining and Language-Aware Decoding(https://arxiv.org/abs/2412.16507)
• Keywords: prompt
• Abstract: Code-switching (CS) automatic speech recognition (ASR) faces challenges due to the language confusion resulting from accents, auditory similarity, and seamless language switches. Adaptation on the pre-trained multi-lingual model has shown promising performance for CS-ASR. In this paper, we adapt Whisper, which is a large-scale multilingual pre-trained speech recognition model, to CS from both encoder and decoder parts. First, we propose an encoder refiner to enhance the encoder's capacity of intra-sentence swithching. Second, we propose using two sets of language-aware adapters with different language prompt embeddings to achieve language-specific decoding information in each decoder layer. Then, a fusion module is added to fuse the language-aware decoding. The experimental results using the SEAME dataset show that, compared with the baseline model, the proposed approach achieves a relative MER reduction of 4.1% and 7.2% on the dev_man and dev_sge test sets, respectively, surpassing state-of-the-art methods. Through experiments, we found that the proposed method significantly improves the performance on non-native language in CS speech, indicating that our approach enables Whisper to better distinguish between the two languages.
• 摘要：由于口音、听觉相似性和无缝语言切换造成的语言混淆，代码转换 (CS) 自动语音识别 (ASR) 面临挑战。在预训练的多语言模型上的适配已为 CS-ASR 显示出良好的性能。在本文中，我们从编码器和解码器部分将大规模多语言预训练语音识别模型 Whisper 适配到 CS。首先，我们提出了一个编码器细化器来增强编码器的句内切换能力。其次，我们建议使用两组具有不同语言提示嵌入的语言感知适配器来在每个解码器层中实现特定于语言的解码信息。然后，添加一个融合模块来融合语言感知解码。使用 SEAME 数据集的实验结果表明，与基线模型相比，所提出的方法在 dev_man 和 dev_sge 测试集上分别实现了 4.1% 和 7.2% 的相对 MER 降低，超越了最先进的方法。通过实验，我们发现所提出的方法显著提高了 CS 语音中非母语语言的性能，表明我们的方法使 Whisper 能够更好地区分两种语言。

Title: HammerBench: Fine-Grained Function-Calling Evaluation in Real Mobile Device Scenarios

• Authors: Jun Wang, Jiamu Zhou, Muning Wen, Xiaoyun Mo, Haoyu Zhang, Qiqiang Lin, Cheng Jin, Xihuai Wang, Weinan Zhang, Qiuying Peng, Jun Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.16516
• Pdf URL: https://arxiv.org/pdf/2412.16516
• Copy Paste: [[2412.16516]] HammerBench: Fine-Grained Function-Calling Evaluation in Real Mobile Device Scenarios(https://arxiv.org/abs/2412.16516)
• Keywords: language model, llm
• Abstract: Evaluating the capabilities of large language models (LLMs) in human-LLM interactions remains challenging due to the inherent complexity and openness of dialogue processes. This paper introduces HammerBench, a novel benchmarking framework designed to assess the function-calling ability of LLMs more effectively in such interactions. We model a wide range of real-world user scenarios on mobile devices, encompassing imperfect instructions, diverse question-answer trajectories, intent/argument shifts, and the use of external individual information through pronouns. To construct the corresponding datasets, we propose a comprehensive pipeline that involves LLM-generated data and multiple rounds of human validation, ensuring high data quality. Additionally, we decompose the conversations into function-calling snapshots, enabling a fine-grained evaluation of each turn. We evaluate several popular LLMs using HammerBench and highlight different performance aspects. Our empirical findings reveal that errors in parameter naming constitute the primary factor behind conversation failures across different data types.
• 摘要：由于对话过程固有的复杂性和开放性，评估大型语言模型 (LLM) 在人机交互中的能力仍然具有挑战性。本文介绍了 HammerBench，这是一种新颖的基准测试框架，旨在更有效地评估 LLM 在这种交互中的函数调用能力。我们在移动设备上模拟了各种现实世界的用户场景，包括不完善的指令、不同的问答轨迹、意图/论据转变以及通过代词使用外部个人信息。为了构建相应的数据集，我们提出了一个全面的管道，其中包括 LLM 生成的数据和多轮人工验证，以确保高数据质量。此外，我们将对话分解为函数调用快照，从而可以对每个回合进行细粒度的评估。我们使用 HammerBench 评估了几种流行的 LLM，并强调了不同的性能方面。我们的实证研究结果表明，参数命名错误是不同数据类型对话失败的主要因素。

Title: Attention Entropy is a Key Factor: An Analysis of Parallel Context Encoding with Full-attention-based Pre-trained Language Models

• Authors: Zhisong Zhang, Yan Wang, Xinting Huang, Tianqing Fang, Hongming Zhang, Chenlong Deng, Shuaiyi Li, Dong Yu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.16545
• Pdf URL: https://arxiv.org/pdf/2412.16545
• Copy Paste: [[2412.16545]] Attention Entropy is a Key Factor: An Analysis of Parallel Context Encoding with Full-attention-based Pre-trained Language Models(https://arxiv.org/abs/2412.16545)
• Keywords: language model
• Abstract: Large language models have shown remarkable performance across a wide range of language tasks, owing to their exceptional capabilities in context modeling. The most commonly used method of context modeling is full self-attention, as seen in standard decoder-only Transformers. Although powerful, this method can be inefficient for long sequences and may overlook inherent input structures. To address these problems, an alternative approach is parallel context encoding, which splits the context into sub-pieces and encodes them parallelly. Because parallel patterns are not encountered during training, naively applying parallel encoding leads to performance degradation. However, the underlying reasons and potential mitigations are unclear. In this work, we provide a detailed analysis of this issue and identify that unusually high attention entropy can be a key factor. Furthermore, we adopt two straightforward methods to reduce attention entropy by incorporating attention sinks and selective mechanisms. Experiments on various tasks reveal that these methods effectively lower irregular attention entropy and narrow performance gaps. We hope this study can illuminate ways to enhance context modeling mechanisms.
• 摘要：大型语言模型凭借其出色的上下文建模能力，在各种语言任务中表现出色。最常用的上下文建模方法是完全自注意力，如标准解码器专用 Transformers 中所示。虽然功能强大，但这种方法对于长序列可能效率低下，并且可能会忽略固有的输入结构。为了解决这些问题，另一种方法是并行上下文编码，它将上下文拆分为子片段并对其进行并行编码。由于在训练期间不会遇到并行模式，因此天真地应用并行编码会导致性能下降。然而，根本原因和潜在的缓解措施尚不清楚。在这项工作中，我们对这个问题进行了详细的分析，并发现异常高的注意力熵可能是一个关键因素。此外，我们采用了两种简单的方法来降低注意力熵，即结合注意力接收器和选择机制。对各种任务的实验表明，这些方法可以有效降低不规则的注意力熵并缩小性能差距。我们希望这项研究能够阐明增强上下文建模机制的方法。

Title: Divide and Conquer: A Hybrid Strategy Defeats Multimodal Large Language Models

• Authors: Yanxu Mao, Peipei Liu, Tiehan Cui, Congying Liu, Datao You
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.16555
• Pdf URL: https://arxiv.org/pdf/2412.16555
• Copy Paste: [[2412.16555]] Divide and Conquer: A Hybrid Strategy Defeats Multimodal Large Language Models(https://arxiv.org/abs/2412.16555)
• Keywords: language model, llm, prompt
• Abstract: Large language models (LLMs) are widely applied in various fields of society due to their powerful reasoning, understanding, and generation capabilities. However, the security issues associated with these models are becoming increasingly severe. Jailbreaking attacks, as an important method for detecting vulnerabilities in LLMs, have been explored by researchers who attempt to induce these models to generate harmful content through various attack methods. Nevertheless, existing jailbreaking methods face numerous limitations, such as excessive query counts, limited coverage of jailbreak modalities, low attack success rates, and simplistic evaluation methods. To overcome these constraints, this paper proposes a multimodal jailbreaking method: JMLLM. This method integrates multiple strategies to perform comprehensive jailbreak attacks across text, visual, and auditory modalities. Additionally, we contribute a new and comprehensive dataset for multimodal jailbreaking research: TriJail, which includes jailbreak prompts for all three modalities. Experiments on the TriJail dataset and the benchmark dataset AdvBench, conducted on 13 popular LLMs, demonstrate advanced attack success rates and significant reduction in time overhead.
• 摘要：大型语言模型（LLM）因其强大的推理、理解和生成能力而被广泛应用于社会的各个领域。然而，与这些模型相关的安全问题也日益严重。越狱攻击作为检测LLM漏洞的重要方法，已被研究人员探索，他们试图通过各种攻击方法诱导这些模型生成有害内容。然而，现有的越狱方法面临许多限制，例如查询次数过多、越狱模态覆盖有限、攻击成功率低以及评估方法过于简单。为了克服这些限制，本文提出了一种多模态越狱方法：JMLLM。该方法集成了多种策略，可在文本、视觉和听觉模态中进行全面的越狱攻击。此外，我们为多模态越狱研究贡献了一个全新的综合数据集：TriJail，其中包括所有三种模态的越狱提示。在13种流行的LLM上对TriJail数据集和基准数据集AdvBench进行的实验表明，攻击成功率更高，时间开销显著减少。

Title: NILE: Internal Consistency Alignment in Large Language Models

• Authors: Minda Hu, Qiyuan Zhang, Yufei Wang, Bowei He, Hongru Wang, Jingyan Zhou, Liangyou Li, Yasheng Wang, Chen Ma, Irwin King
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.16686
• Pdf URL: https://arxiv.org/pdf/2412.16686
• Copy Paste: [[2412.16686]] NILE: Internal Consistency Alignment in Large Language Models(https://arxiv.org/abs/2412.16686)
• Keywords: language model, llm
• Abstract: As a crucial step to enhance LLMs alignment with human intentions, Instruction Fine-Tuning (IFT) has a high demand on dataset quality. However, existing IFT datasets often contain knowledge that is inconsistent with LLMs' internal knowledge learned from the pre-training phase, which can greatly affect the efficacy of IFT. To address this issue, we introduce NILE (iNternal consIstency aLignmEnt) framework, aimed at optimizing IFT datasets to unlock LLMs' capability further. NILE operates by eliciting target pre-trained LLM's internal knowledge corresponding to instruction data. The internal knowledge is leveraged to revise the answer in IFT datasets. Additionally, we propose a novel Internal Consistency Filtering (ICF) method to filter training samples, ensuring its high consistency with LLM's internal knowledge. Our experiments demonstrate that NILE-aligned IFT datasets sharply boost LLM performance across multiple LLM ability evaluation datasets, achieving up to 66.6% gain on Arena-Hard and 68.5% on Alpaca-Eval V2. Further analysis confirms that each component of the NILE}framework contributes to these substantial performance improvements, and provides compelling evidence that dataset consistency with pre-trained internal knowledge is pivotal for maximizing LLM potential.
• 摘要：作为增强 LLM 与人类意图一致性的关键步骤，指令微调 (IFT) 对数据集质量的要求很高。然而，现有的 IFT 数据集通常包含与 LLM 从预训练阶段学到的内部知识不一致的知识，这会极大地影响 IFT 的有效性。为了解决这个问题，我们引入了 NILE（内部一致性对齐）框架，旨在优化 IFT 数据集以进一步释放 LLM 的能力。NILE 通过引出与指令数据相对应的目标预训练 LLM 内部知识来运行。内部知识被用来修改 IFT 数据集中的答案。此外，我们提出了一种新颖的内部一致性过滤 (ICF) 方法来过滤训练样本，确保其与 LLM 的内部知识高度一致。我们的实验表明，与 NILE 对齐的 IFT 数据集在多个 LLM 能力评估数据集中大幅提升了 LLM 性能，在 Arena-Hard 上实现了高达 66.6% 的增益，在 Alpaca-Eval V2 上实现了 68.5% 的增益。进一步的分析证实，NILE 框架的每个组件都为这些显着的性能改进做出了贡献，并提供了令人信服的证据，表明数据集与预训练的内部知识的一致性对于最大限度地发挥 LLM 潜力至关重要。

Title: SubData: A Python Library to Collect and Combine Datasets for Evaluating LLM Alignment on Downstream Tasks

• Authors: Leon Fröhling, Pietro Bernardelle, Gianluca Demartini
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.16783
• Pdf URL: https://arxiv.org/pdf/2412.16783
• Copy Paste: [[2412.16783]] SubData: A Python Library to Collect and Combine Datasets for Evaluating LLM Alignment on Downstream Tasks(https://arxiv.org/abs/2412.16783)
• Keywords: language model, llm, prompt
• Abstract: With the release of ever more capable large language models (LLMs), researchers in NLP and related disciplines have started to explore the usability of LLMs for a wide variety of different annotation tasks. Very recently, a lot of this attention has shifted to tasks that are subjective in nature. Given that the latest generations of LLMs have digested and encoded extensive knowledge about different human subpopulations and individuals, the hope is that these models can be trained, tuned or prompted to align with a wide range of different human perspectives. While researchers already evaluate the success of this alignment via surveys and tests, there is a lack of resources to evaluate the alignment on what oftentimes matters the most in NLP; the actual downstream tasks. To fill this gap we present SubData, a Python library that offers researchers working on topics related to subjectivity in annotation tasks a convenient way of collecting, combining and using a range of suitable datasets.
• 摘要：随着功能更强大的大型语言模型 (LLM) 的发布，NLP 和相关学科的研究人员开始探索 LLM 在各种不同注释任务中的可用性。最近，这种注意力转移到了主观性任务上。鉴于最新一代的 LLM 已经消化并编码了有关不同人类亚群和个体的大量知识，人们希望这些模型能够经过训练、调整或提示，以与各种不同的人类观点保持一致。虽然研究人员已经通过调查和测试评估了这种一致性的成功性，但缺乏资源来评估 NLP 中往往最重要的事情的一致性；实际的下游任务。为了填补这一空白，我们推出了 SubData，这是一个 Python 库，它为从事注释任务中与主观性相关主题的研究人员提供了一种方便的方式来收集、组合和使用一系列合适的数据集。

Title: Quantum-Like Contextuality in Large Language Models

• Authors: Kin Ian Lo, Mehrnoosh Sadrzadeh, Shane Mansfield
• Subjects: cs.CL, quant-ph
• Abstract URL: https://arxiv.org/abs/2412.16806
• Pdf URL: https://arxiv.org/pdf/2412.16806
• Copy Paste: [[2412.16806]] Quantum-Like Contextuality in Large Language Models(https://arxiv.org/abs/2412.16806)
• Keywords: language model
• Abstract: Contextuality is a distinguishing feature of quantum mechanics and there is growing evidence that it is a necessary condition for quantum advantage. In order to make use of it, researchers have been asking whether similar phenomena arise in other domains. The answer has been yes, e.g. in behavioural sciences. However, one has to move to frameworks that take some degree of signalling into account. Two such frameworks exist: (1) a signalling-corrected sheaf theoretic model, and (2) the Contextuality-by-Default (CbD) framework. This paper provides the first large scale experimental evidence for a yes answer in natural language. We construct a linguistic schema modelled over a contextual quantum scenario, instantiate it in the Simple English Wikipedia and extract probability distributions for the instances using the large language model BERT. This led to the discovery of 77,118 sheaf-contextual and 36,938,948 CbD contextual instances. We proved that the contextual instances came from semantically similar words, by deriving an equation between degrees of contextuality and Euclidean distances of BERT's embedding vectors. A regression model further reveals that Euclidean distance is indeed the best statistical predictor of contextuality. Our linguistic schema is a variant of the co-reference resolution challenge. These results are an indication that quantum methods may be advantageous in language tasks.
• 摘要：语境性是量子力学的一个显著特征，越来越多的证据表明它是量子优势的必要条件。为了利用它，研究人员一直在询问其他领域是否也会出现类似的现象。答案是肯定的，例如在行为科学中。然而，我们必须转向考虑一定程度的信号传导的框架。存在两个这样的框架：(1) 信号传导校正的层理论模型，以及 (2) 默认语境性 (CbD) 框架。本文提供了自然语言中肯定答案的第一个大规模实验证据。我们构建了一个基于语境量子场景建模的语言模式，在简易英语维基百科中实例化它，并使用大型语言模型 BERT 提取实例的概率分布。这导致发现了 77,118 个层语境实例和 36,938,948 个 CbD 语境实例。我们通过推导出语境化程度与 BERT 嵌入向量的欧几里得距离之间的方程，证明了语境实例来自语义相似的单词。回归模型进一步表明，欧几里得距离确实是语境化的最佳统计预测因子。我们的语言模式是共指消解挑战的一种变体。这些结果表明，量子方法在语言任务中可能具有优势。

Title: Ask-Before-Detection: Identifying and Mitigating Conformity Bias in LLM-Powered Error Detector for Math Word Problem Solutions

• Authors: Hang Li, Tianlong Xu, Kaiqi Yang, Yucheng Chu, Yanling Chen, Yichi Song, Qingsong Wen, Hui Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.16838
• Pdf URL: https://arxiv.org/pdf/2412.16838
• Copy Paste: [[2412.16838]] Ask-Before-Detection: Identifying and Mitigating Conformity Bias in LLM-Powered Error Detector for Math Word Problem Solutions(https://arxiv.org/abs/2412.16838)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: The rise of large language models (LLMs) offers new opportunities for automatic error detection in education, particularly for math word problems (MWPs). While prior studies demonstrate the promise of LLMs as error detectors, they overlook the presence of multiple valid solutions for a single MWP. Our preliminary analysis reveals a significant performance gap between conventional and alternative solutions in MWPs, a phenomenon we term conformity bias in this work. To mitigate this bias, we introduce the Ask-Before-Detect (AskBD) framework, which generates adaptive reference solutions using LLMs to enhance error detection. Experiments on 200 examples of GSM8K show that AskBD effectively mitigates bias and improves performance, especially when combined with reasoning-enhancing techniques like chain-of-thought prompting.
• 摘要：大型语言模型 (LLM) 的兴起为教育中的自动错误检测提供了新的机会，尤其是对于数学应用题 (MWP)。虽然先前的研究表明 LLM 作为错误检测器的前景，但它们忽略了单个 MWP 的多个有效解决方案的存在。我们的初步分析表明，MWP 中的传统解决方案和替代解决方案之间存在显著的性能差距，我们在本文中将这种现象称为从众偏差。为了减轻这种偏差，我们引入了先问后测 (AskBD) 框架，该框架使用 LLM 生成自适应参考解决方案以增强错误检测。对 200 个 GSM8K 示例的实验表明，AskBD 可以有效减轻偏差并提高性能，尤其是与思维链提示等推理增强技术相结合时。

Title: Sim911: Towards Effective and Equitable 9-1-1 Dispatcher Training with an LLM-Enabled Simulation

• Authors: Zirong Chen, Elizabeth Chason, Noah Mladenovski, Erin Wilson, Kristin Mullen, Stephen Martini, Meiyi Ma
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.16844
• Pdf URL: https://arxiv.org/pdf/2412.16844
• Copy Paste: [[2412.16844]] Sim911: Towards Effective and Equitable 9-1-1 Dispatcher Training with an LLM-Enabled Simulation(https://arxiv.org/abs/2412.16844)
• Keywords: language model, llm, prompt
• Abstract: Emergency response services are vital for enhancing public safety by safeguarding the environment, property, and human lives. As frontline members of these services, 9-1-1 dispatchers have a direct impact on response times and the overall effectiveness of emergency operations. However, traditional dispatcher training methods, which rely on role-playing by experienced personnel, are labor-intensive, time-consuming, and often neglect the specific needs of underserved communities. To address these challenges, we introduce Sim911, the first training simulation for 9-1-1 dispatchers powered by Large Language Models (LLMs). Sim911 enhances training through three key technical innovations: (1) knowledge construction, which utilizes archived 9-1-1 call data to generate simulations that closely mirror real-world scenarios; (2) context-aware controlled generation, which employs dynamic prompts and vector bases to ensure that LLM behavior aligns with training objectives; and (3) validation with looped correction, which filters out low-quality responses and refines the system performance.
• 摘要：应急响应服务对于保护环境、财产和人的生命安全至关重要。作为这些服务的一线成员，9-1-1 调度员对响应时间和应急行动的整体有效性有直接影响。然而，传统的调度员培训方法依赖于经验丰富的人员进行角色扮演，这需要大量劳动力和时间，而且往往会忽视服务不足社区的特殊需求。为了应对这些挑战，我们推出了 Sim911，这是第一个由大型语言模型 (LLM) 提供支持的 9-1-1 调度员培训模拟。Sim911 通过三项关键技术创新增强了培训：(1) 知识构建，利用存档的 9-1-1 呼叫数据生成与真实场景非常相似的模拟；(2) 上下文感知控制生成，采用动态提示和向量基来确保 LLM 行为与培训目标一致；(3) 循环校正验证，过滤掉低质量响应并改善系统性能。

Title: GME: Improving Universal Multimodal Retrieval by Multimodal LLMs

• Authors: Xin Zhang, Yanzhao Zhang, Wen Xie, Mingxin Li, Ziqi Dai, Dingkun Long, Pengjun Xie, Meishan Zhang, Wenjie Li, Min Zhang
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2412.16855
• Pdf URL: https://arxiv.org/pdf/2412.16855
• Copy Paste: [[2412.16855]] GME: Improving Universal Multimodal Retrieval by Multimodal LLMs(https://arxiv.org/abs/2412.16855)
• Keywords: language model, llm
• Abstract: Universal Multimodal Retrieval (UMR) aims to enable search across various modalities using a unified model, where queries and candidates can consist of pure text, images, or a combination of both. Previous work has attempted to adopt multimodal large language models (MLLMs) to realize UMR using only text data. However, our preliminary experiments demonstrate that more diverse multimodal training data can further unlock the potential of MLLMs. Despite its effectiveness, the existing multimodal training data is highly imbalanced in terms of modality, which motivates us to develop a training data synthesis pipeline and construct a large-scale, high-quality fused-modal training dataset. Based on the synthetic training data, we develop the General Multimodal Embedder (GME), an MLLM-based dense retriever designed for UMR. Furthermore, we construct a comprehensive UMR Benchmark (UMRB) to evaluate the effectiveness of our approach. Experimental results show that our method achieves state-of-the-art performance among existing UMR methods. Last, we provide in-depth analyses of model scaling, training strategies, and perform ablation studies on both the model and synthetic data.
• 摘要：通用多模态检索 (UMR) 旨在使用统一模型实现跨各种模态的搜索，其中查询和候选可以由纯文本、图像或两者的组合组成。先前的研究尝试采用多模态大型语言模型 (MLLM) 来实现仅使用文本数据的 UMR。然而，我们的初步实验表明，更多样化的多模态训练数据可以进一步释放 MLLM 的潜力。尽管其有效，但现有的多模态训练数据在模态方面高度不平衡，这促使我们开发训练数据合成管道并构建大规模、高质量的融合模态训练数据集。基于合成训练数据，我们开发了通用多模态嵌入器 (GME)，这是一种基于 MLLM 的密集检索器，专为 UMR 设计。此外，我们构建了一个全面的 UMR 基准 (UMRB) 来评估我们方法的有效性。实验结果表明，我们的方法在现有的 UMR 方法中达到了最佳性能。最后，我们对模型扩展、训练策略进行了深入分析，并对模型和合成数据进行了消融研究。

Title: Teaching LLMs to Refine with Tools

• Authors: Dian Yu, Yuheng Zhang, Jiahao Xu, Tian Liang, Linfeng Song, Zhaopeng Tu, Haitao Mi, Dong Yu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.16871
• Pdf URL: https://arxiv.org/pdf/2412.16871
• Copy Paste: [[2412.16871]] Teaching LLMs to Refine with Tools(https://arxiv.org/abs/2412.16871)
• Keywords: language model, llm, chain-of-thought
• Abstract: Large language models (LLMs) can refine their responses based on feedback, enabling self-improvement through iterative training or test-time refinement. However, existing methods predominantly focus on refinement within the same reasoning format, which may lead to non-correcting behaviors. We propose CaP, a novel approach that uses external tools to refine chain-of-thought (CoT) responses generated by the same or other LLMs. CaP employs a two-stage training process: supervised fine-tuning followed by preference optimization with DPO variants. Our observations highlight the critical role of preference optimization in enabling effective refinement. Additionally, we compare several sampling strategies to leverage CoT and tools at inference time. Experimental results demonstrate CaP's potential for effective cross-reasoning refinement and efficient inference.
• 摘要：大型语言模型 (LLM) 可以根据反馈改进其响应，通过迭代训练或测试时改进实现自我改进。然而，现有方法主要侧重于在相同推理格式内进行改进，这可能会导致无法纠正的行为。我们提出了 CaP，这是一种新方法，它使用外部工具来改进由相同或其他 LLM 生成的思路链 (CoT) 响应。CaP 采用两阶段训练过程：监督微调，然后使用 DPO 变体进行偏好优化。我们的观察强调了偏好优化在实现有效改进方面的关键作用。此外，我们比较了几种采样策略，以在推理时利用 CoT 和工具。实验结果证明了 CaP 在有效交叉推理改进和高效推理方面的潜力。

Title: Revisiting In-Context Learning with Long Context Language Models

• Authors: Jinheon Baek, Sun Jae Lee, Prakhar Gupta, Geunseob (GS)Oh, Siddharth Dalmia, Prateek Kolhar
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.16926
• Pdf URL: https://arxiv.org/pdf/2412.16926
• Copy Paste: [[2412.16926]] Revisiting In-Context Learning with Long Context Language Models(https://arxiv.org/abs/2412.16926)
• Keywords: language model, long context
• Abstract: In-Context Learning (ICL) is a technique by which language models make predictions based on examples provided in their input context. Previously, their context window size imposed a limit on the number of examples that can be shown, making example selection techniques crucial for identifying the maximally effective set of examples. However, the recent advent of Long Context Language Models (LCLMs) has significantly increased the number of examples that can be included in context, raising an important question of whether ICL performance in a many-shot regime is still sensitive to the method of sample selection. To answer this, we revisit these approaches in the context of LCLMs through extensive experiments on 18 datasets spanning 4 tasks. Surprisingly, we observe that sophisticated example selection techniques do not yield significant improvements over a simple random sample selection method. Instead, we find that the advent of LCLMs has fundamentally shifted the challenge of ICL from that of selecting the most effective examples to that of collecting sufficient examples to fill the context window. Specifically, in certain datasets, including all available examples does not fully utilize the context window; however, by augmenting the examples in context with a simple data augmentation approach, we substantially improve ICL performance by 5%.
• 摘要：上下文学习 (ICL) 是一种语言模型根据其输入上下文中提供的示例进行预测的技术。以前，它们的上下文窗口大小限制了可以显示的示例数量，因此示例选择技术对于识别最有效的示例集至关重要。然而，最近出现的长上下文语言模型 (LCLM) 显著增加了可以包含在上下文中的示例数量，这引发了一个重要的问题：在多样本机制中，ICL 的性能是否仍然对样本选择方法敏感。为了回答这个问题，我们通过在涵盖 4 个任务的 18 个数据集上进行大量实验，在 LCLM 的背景下重新审视了这些方法。令人惊讶的是，我们观察到复杂的示例选择技术并没有比简单的随机样本选择方法产生显着的改进。相反，我们发现 LCLM 的出现从根本上将 ICL 的挑战从选择最有效的例子转移到收集足够的例子来填充上下文窗口。具体而言，在某些数据集中，包括所有可用示例并不能充分利用上下文窗口；然而，通过使用简单的数据增强方法增强上下文中的示例，我们大幅提高了 ICL 性能 5%。

Title: Prompting Large Language Models with Rationale Heuristics for Knowledge-based Visual Question Answering

• Authors: Zhongjian Hu, Peng Yang, Bing Li, Fengyuan Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.16936
• Pdf URL: https://arxiv.org/pdf/2412.16936
• Copy Paste: [[2412.16936]] Prompting Large Language Models with Rationale Heuristics for Knowledge-based Visual Question Answering(https://arxiv.org/abs/2412.16936)
• Keywords: language model, llm, prompt
• Abstract: Recently, Large Language Models (LLMs) have been used for knowledge-based Visual Question Answering (VQA). Despite the encouraging results of previous studies, prior methods prompt LLMs to predict answers directly, neglecting intermediate thought processes. We argue that prior methods do not sufficiently activate the capacities of LLMs. We propose a framework called PLRH that Prompts LLMs with Rationale Heuristics for knowledge-based VQA. The PLRH prompts LLMs with Chain of Thought (CoT) to generate rationale heuristics, i.e., intermediate thought processes, and then leverages the rationale heuristics to inspire LLMs to predict answers. Experiments show that our approach outperforms the existing baselines by more than 2.2 and 2.1 on OK-VQA and A-OKVQA, respectively.
• 摘要：最近，大型语言模型 (LLM) 已用于基于知识的视觉问答 (VQA)。尽管先前的研究结果令人鼓舞，但先前的方法促使 LLM 直接预测答案，而忽略了中间思维过程。我们认为先前的方法不能充分激活 LLM 的能力。我们提出了一个名为 PLRH 的框架，该框架提示具有理性启发式方法的 LLM 进行基于知识的 VQA。PLRH 提示具有思路链 (CoT) 的 LLM 生成理性启发式方法，即中间思维过程，然后利用理性启发式方法启发 LLM 预测答案。实验表明，我们的方法在 OK-VQA 和 A-OKVQA 上的表现分别比现有基线高出 2.2 和 2.1 以上。

Title: A Career Interview Dialogue System using Large Language Model-based Dynamic Slot Generation

• Authors: Ekai Hashimoto, Mikio Nakano, Takayoshi Sakurai, Shun Shiramatsu, Toshitake Komazaki, Shiho Tsuchiya
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.16943
• Pdf URL: https://arxiv.org/pdf/2412.16943
• Copy Paste: [[2412.16943]] A Career Interview Dialogue System using Large Language Model-based Dynamic Slot Generation(https://arxiv.org/abs/2412.16943)
• Keywords: language model, llm
• Abstract: This study aims to improve the efficiency and quality of career interviews conducted by nursing managers. To this end, we have been developing a slot-filling dialogue system that engages in pre-interviews to collect information on staff careers as a preparatory step before the actual interviews. Conventional slot-filling-based interview dialogue systems have limitations in the flexibility of information collection because the dialogue progresses based on predefined slot sets. We therefore propose a method that leverages large language models (LLMs) to dynamically generate new slots according to the flow of the dialogue, achieving more natural conversations. Furthermore, we incorporate abduction into the slot generation process to enable more appropriate and effective slot generation. To validate the effectiveness of the proposed method, we conducted experiments using a user simulator. The results suggest that the proposed method using abduction is effective in enhancing both information-collecting capabilities and the naturalness of the dialogue.
• 摘要：本研究旨在提高护理经理职业面试的效率和质量。为此，我们一直在开发一个槽位填充对话系统，该系统在实际面试之前进行预面试以收集有关员工职业的信息，作为准备步骤。传统的基于槽位填充的面试对话系统在信息收集的灵活性方面存在限制，因为对话是基于预定义的槽位集进行的。因此，我们提出了一种方法，利用大型语言模型 (LLM) 根据对话流程动态生成新槽位，实现更自然的对话。此外，我们将溯因推理纳入槽位生成过程，以实现更合适、更有效的槽位生成。为了验证所提方法的有效性，我们使用用户模拟器进行了实验。结果表明，所提出的使用溯因推理的方法可以有效提高信息收集能力和对话的自然性。

Title: Aristotle: Mastering Logical Reasoning with A Logic-Complete Decompose-Search-Resolve Framework

• Authors: Jundong Xu, Hao Fei, Meng Luo, Qian Liu, Liangming Pan, William Yang Wang, Preslav Nakov, Mong-Li Lee, Wynne Hsu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.16953
• Pdf URL: https://arxiv.org/pdf/2412.16953
• Copy Paste: [[2412.16953]] Aristotle: Mastering Logical Reasoning with A Logic-Complete Decompose-Search-Resolve Framework(https://arxiv.org/abs/2412.16953)
• Keywords: language model, llm
• Abstract: In the context of large language models (LLMs), current advanced reasoning methods have made impressive strides in various reasoning tasks. However, when it comes to logical reasoning tasks, major challenges remain in both efficacy and efficiency. This is rooted in the fact that these systems fail to fully leverage the inherent structure of logical tasks throughout the reasoning processes such as decomposition, search, and resolution. To address this, we propose a logic-complete reasoning framework, Aristotle, with three key components: Logical Decomposer, Logical Search Router, and Logical Resolver. In our framework, symbolic expressions and logical rules are comprehensively integrated into the entire reasoning process, significantly alleviating the bottlenecks of logical reasoning, i.e., reducing sub-task complexity, minimizing search errors, and resolving logical contradictions. The experimental results on several datasets demonstrate that Aristotle consistently outperforms state-of-the-art reasoning frameworks in both accuracy and efficiency, particularly excelling in complex logical reasoning scenarios. We will open-source all our code at this https URL.
• 摘要：在大型语言模型（LLM）的背景下，当前的高级推理方法在各种推理任务中取得了令人瞩目的进步。然而，当涉及到逻辑推理任务时，在有效性和效率方面仍然存在重大挑战。这源于这些系统未能充分利用逻辑任务在分解、搜索和解析等推理过程中的固有结构。为了解决这个问题，我们提出了一个逻辑完整的推理框架 Aristotle，它有三个关键组件：逻辑分解器、逻辑搜索路由器和逻辑解析器。在我们的框架中，符号表达式和逻辑规则被全面整合到整个推理过程中，大大缓解了逻辑推理的瓶颈，即降低子任务复杂性、最小化搜索错误和解决逻辑矛盾。在多个数据集上的实验结果表明，Aristotle 在准确性和效率方面始终优于最先进的推理框架，尤其是在复杂的逻辑推理场景中表现出色。我们将在此 https URL 上开源所有代码。

Title: LH-Mix: Local Hierarchy Correlation Guided Mixup over Hierarchical Prompt Tuning

• Authors: Fanshuang Kong, Richong Zhang, Ziqiao Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.16963
• Pdf URL: https://arxiv.org/pdf/2412.16963
• Copy Paste: [[2412.16963]] LH-Mix: Local Hierarchy Correlation Guided Mixup over Hierarchical Prompt Tuning(https://arxiv.org/abs/2412.16963)
• Keywords: prompt
• Abstract: Hierarchical text classification (HTC) aims to assign one or more labels in the hierarchy for each text. Many methods represent this structure as a global hierarchy, leading to redundant graph structures. To address this, incorporating a text-specific local hierarchy is essential. However, existing approaches often model this local hierarchy as a sequence, focusing on explicit parent-child relationships while ignoring implicit correlations among sibling/peer relationships. In this paper, we first integrate local hierarchies into a manual depth-level prompt to capture parent-child relationships. We then apply Mixup to this hierarchical prompt tuning scheme to improve the latent correlation within sibling/peer relationships. Notably, we propose a novel Mixup ratio guided by local hierarchy correlation to effectively capture intrinsic correlations. This Local Hierarchy Mixup (LH-Mix) model demonstrates remarkable performance across three widely-used datasets.
• 摘要：分层文本分类 (HTC) 旨在为每个文本分配层次结构中的一个或多个标签。许多方法将此结构表示为全局层次结构，从而导致图形结构冗余。为了解决这个问题，合并特定于文本的局部层次结构至关重要。然而，现有的方法通常将这种局部层次结构建模为一个序列，专注于显式的父子关系，而忽略兄弟姐妹/同侪关系之间的隐式相关性。在本文中，我们首先将局部层次结构集成到手动深度级别提示中以捕获父子关系。然后，我们将 Mixup 应用于这种分层提示调整方案，以改善兄弟姐妹/同侪关系中的潜在相关性。值得注意的是，我们提出了一种由局部层次相关性指导的新型 Mixup 比率，以有效捕捉内在相关性。这种局部层次混合 (LH-Mix) 模型在三个广泛使用的数据集上表现出色。

Title: On Fusing ChatGPT and Ensemble Learning in Discon-tinuous Named Entity Recognition in Health Corpora

• Authors: Tzu-Chieh Chen, Wen-Yang Lin
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.16976
• Pdf URL: https://arxiv.org/pdf/2412.16976
• Copy Paste: [[2412.16976]] On Fusing ChatGPT and Ensemble Learning in Discon-tinuous Named Entity Recognition in Health Corpora(https://arxiv.org/abs/2412.16976)
• Keywords: language model, gpt, prompt, chat
• Abstract: Named Entity Recognition has traditionally been a key task in natural language processing, aiming to identify and extract important terms from unstructured text data. However, a notable challenge for contemporary deep-learning NER models has been identifying discontinuous entities, which are often fragmented within the text. To date, methods to address Discontinuous Named Entity Recognition have not been explored using ensemble learning to the best of our knowledge. Furthermore, the rise of large language models, such as ChatGPT in recent years, has shown significant effectiveness across many NLP tasks. Most existing approaches, however, have primarily utilized ChatGPT as a problem-solving tool rather than exploring its potential as an integrative element within ensemble learning algorithms. In this study, we investigated the integration of ChatGPT as an arbitrator within an ensemble method, aiming to enhance performance on DNER tasks. Our method combines five state-of-the-art NER models with ChatGPT using custom prompt engineering to assess the robustness and generalization capabilities of the ensemble algorithm. We conducted experiments on three benchmark medical datasets, comparing our method against the five SOTA models, individual applications of GPT-3.5 and GPT-4, and a voting ensemble method. The results indicate that our proposed fusion of ChatGPT with the ensemble learning algorithm outperforms the SOTA results in the CADEC, ShARe13, and ShARe14 datasets, showcasing its potential to enhance NLP applications in the healthcare domain.
• 摘要：命名实体识别传统上是自然语言处理中的一项关键任务，旨在从非结构化文本数据中识别和提取重要术语。然而，当代深度学习 NER 模型面临的一个显著挑战是识别不连续的实体，这些实体通常在文本中是零散的。到目前为止，据我们所知，尚未使用集成学习探索解决不连续命名实体识别的方法。此外，近年来，大型语言模型（如 ChatGPT）的兴起已在许多 NLP 任务中显示出显著的有效性。然而，大多数现有方法主要将 ChatGPT 用作解决问题的工具，而不是探索其作为集成学习算法中集成元素的潜力。在本研究中，我们研究了将 ChatGPT 作为仲裁器集成到集成方法中，旨在提高 DNER 任务的性能。我们的方法将五个最先进的 NER 模型与 ChatGPT 相结合，使用自定义提示工程来评估集成算法的稳健性和泛化能力。我们对三个基准医疗数据集进行了实验，将我们的方法与五个 SOTA 模型、GPT-3.5 和 GPT-4 的单独应用以及投票集成方法进行了比较。结果表明，我们提出的 ChatGPT 与集成学习算法的融合在 CADEC、ShARe13 和 ShARe14 数据集中的表现优于 SOTA 结果，展示了其在医疗领域增强 NLP 应用的潜力。

Title: Robustness of Large Language Models Against Adversarial Attacks

• Authors: Yiyi Tao, Yixian Shen, Hang Zhang, Yanxin Shen, Lun Wang, Chuanqi Shi, Shaoshuai Du
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.17011
• Pdf URL: https://arxiv.org/pdf/2412.17011
• Copy Paste: [[2412.17011]] Robustness of Large Language Models Against Adversarial Attacks(https://arxiv.org/abs/2412.17011)
• Keywords: language model, gpt, llm, prompt
• Abstract: The increasing deployment of Large Language Models (LLMs) in various applications necessitates a rigorous evaluation of their robustness against adversarial attacks. In this paper, we present a comprehensive study on the robustness of GPT LLM family. We employ two distinct evaluation methods to assess their resilience. The first method introduce character-level text attack in input prompts, testing the models on three sentiment classification datasets: StanfordNLP/IMDB, Yelp Reviews, and SST-2. The second method involves using jailbreak prompts to challenge the safety mechanisms of the LLMs. Our experiments reveal significant variations in the robustness of these models, demonstrating their varying degrees of vulnerability to both character-level and semantic-level adversarial attacks. These findings underscore the necessity for improved adversarial training and enhanced safety mechanisms to bolster the robustness of LLMs.
• 摘要：大型语言模型 (LLM) 在各种应用中的部署越来越多，这需要对其抵御对抗性攻击的鲁棒性进行严格评估。在本文中，我们对 GPT LLM 系列的鲁棒性进行了全面的研究。我们采用两种不同的评估方法来评估它们的弹性。第一种方法在输入提示中引入字符级文本攻击，在三个情绪分类数据集上测试模型：StanfordNLP/IMDB、Yelp Reviews 和 SST-2。第二种方法涉及使用越狱提示来挑战 LLM 的安全机制。我们的实验揭示了这些模型的鲁棒性存在显著差异，表明它们对字符级和语义级对抗性攻击的脆弱性程度不同。这些发现强调了改进对抗性训练和增强安全机制以增强 LLM 鲁棒性的必要性。

Title: Reversed Attention: On The Gradient Descent Of Attention Layers In GPT

• Authors: Shahar Katz, Lior Wolf
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.17019
• Pdf URL: https://arxiv.org/pdf/2412.17019
• Copy Paste: [[2412.17019]] Reversed Attention: On The Gradient Descent Of Attention Layers In GPT(https://arxiv.org/abs/2412.17019)
• Keywords: language model, gpt
• Abstract: The success of Transformer-based Language Models (LMs) stems from their attention mechanism. While this mechanism has been extensively studied in explainability research, particularly through the attention values obtained during the forward pass of LMs, the backward pass of attention has been largely overlooked. In this work, we study the mathematics of the backward pass of attention, revealing that it implicitly calculates an attention matrix we refer to as "Reversed Attention". We examine the properties of Reversed Attention and demonstrate its ability to elucidate the models' behavior and edit dynamics. In an experimental setup, we showcase the ability of Reversed Attention to directly alter the forward pass of attention, without modifying the model's weights, using a novel method called "attention patching". In addition to enhancing the comprehension of how LM configure attention layers during backpropagation, Reversed Attention maps contribute to a more interpretable backward pass.
• 摘要：基于 Transformer 的语言模型 (LM) 的成功源于其注意力机制。虽然这种机制在可解释性研究中得到了广泛的研究，特别是通过 LM 前向传递期间获得的注意力值，但注意力的后向传递却在很大程度上被忽视了。在这项工作中，我们研究了注意力后向传递的数学原理，发现它隐式地计算了一个我们称之为“反向注意力”的注意力矩阵。我们研究了反向注意力的属性，并展示了它阐明模型行为和编辑动态的能力。在实验设置中，我们展示了反向注意力直接改变注意力前向传递的能力，而无需修改模型的权重，使用一种称为“注意力修补”的新方法。除了增强对 LM 如何在反向传播过程中配置注意力层的理解之外，反向注意力图还有助于实现更易于解释的后向传递。

Title: A Reality Check on Context Utilisation for Retrieval-Augmented Generation

• Authors: Lovisa Hagström, Sara Vera Marjanović, Haeun Yu, Arnav Arora, Christina Lioma, Maria Maistro, Pepa Atanasova, Isabelle Augenstein
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.17031
• Pdf URL: https://arxiv.org/pdf/2412.17031
• Copy Paste: [[2412.17031]] A Reality Check on Context Utilisation for Retrieval-Augmented Generation(https://arxiv.org/abs/2412.17031)
• Keywords: language model, retrieval-augmented generation
• Abstract: Retrieval-augmented generation (RAG) helps address the limitations of the parametric knowledge embedded within a language model (LM). However, investigations of how LMs utilise retrieved information of varying complexity in real-world scenarios have been limited to synthetic contexts. We introduce DRUID (Dataset of Retrieved Unreliable, Insufficient and Difficult-to-understand contexts) with real-world queries and contexts manually annotated for stance. The dataset is based on the prototypical task of automated claim verification, for which automated retrieval of real-world evidence is crucial. We compare DRUID to synthetic datasets (CounterFact, ConflictQA) and find that artificial datasets often fail to represent the complex and diverse real-world context settings. We show that synthetic datasets exaggerate context characteristics rare in real retrieved data, which leads to inflated context utilisation results, as measured by our novel ACU score. Moreover, while previous work has mainly focused on singleton context characteristics to explain context utilisation, correlations between singleton context properties and ACU on DRUID are surprisingly small compared to other properties related to context source. Overall, our work underscores the need for real-world aligned context utilisation studies to represent and improve performance in real-world RAG settings.
• 摘要：检索增强生成 (RAG) 有助于解决语言模型 (LM) 中嵌入的参数知识的局限性。然而，对 LM 如何利用现实世界场景中不同复杂程度的检索信息的研究仅限于合成上下文。我们引入了 DRUID（检索到的不可靠、不足和难以理解的上下文数据集），其中包含手动注释立场的现实世界查询和上下文。该数据集基于自动索赔验证的典型任务，其中自动检索现实世界证据至关重要。我们将 DRUID 与合成数据集（CounterFact、ConflictQA）进行比较，发现人工数据集通常无法表示复杂多样的现实世界上下文设置。我们表明，合成数据集夸大了在实际检索数据中罕见的上下文特征，从而导致上下文利用率结果被夸大，以我们新颖的 ACU 分数来衡量。此外，虽然以前的研究主要侧重于单例上下文特征来解释上下文利用率，但与其他与上下文源相关的属性相比，单例上下文属性与 DRUID 上的 ACU 之间的相关性出奇地小。总体而言，我们的工作强调了需要进行与现实世界一致的上下文利用率研究，以表示和提高现实世界 RAG 设置中的性能。

Title: MINTQA: A Multi-Hop Question Answering Benchmark for Evaluating LLMs on New and Tail Knowledge

• Authors: Jie He, Nan Hu, Wanqiu Long, Jiaoyan Chen, Jeff Z. Pan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.17032
• Pdf URL: https://arxiv.org/pdf/2412.17032
• Copy Paste: [[2412.17032]] MINTQA: A Multi-Hop Question Answering Benchmark for Evaluating LLMs on New and Tail Knowledge(https://arxiv.org/abs/2412.17032)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: Large language models (LLMs) have demonstrated impressive capabilities in various reasoning tasks but face significant challenges with complex, knowledge-intensive multi-hop queries, particularly those involving new or long-tail knowledge. Existing benchmarks often fail to fully address these challenges. To bridge this gap, we introduce MINTQA (Multi-hop Question Answering on New and Tail Knowledge), a comprehensive benchmark to evaluate LLMs' capabilities in multi-hop reasoning across four critical dimensions: question handling strategy, sub-question generation, retrieval-augmented generation, and iterative or dynamic decomposition and retrieval. MINTQA comprises 10,479 question-answer pairs for evaluating new knowledge and 17,887 pairs for assessing long-tail knowledge, with each question equipped with corresponding sub-questions and answers. Our systematic evaluation of 22 state-of-the-art LLMs on MINTQA reveals significant limitations in their ability to handle complex knowledge base queries, particularly in handling new or unpopular knowledge. Our findings highlight critical challenges and offer insights for advancing multi-hop reasoning capabilities. The MINTQA benchmark is available at this https URL.
• 摘要：大型语言模型 (LLM) 在各种推理任务中表现出令人印象深刻的能力，但在复杂、知识密集型的多跳查询方面面临重大挑战，尤其是涉及新知识或长尾知识的查询。现有的基准往往无法完全解决这些挑战。为了弥补这一差距，我们引入了 MINTQA（新知识和尾部知识的多跳问答），这是一个全面的基准，用于评估 LLM 在四个关键维度上的多跳推理能力：问题处理策略、子问题生成、检索增强生成以及迭代或动态分解和检索。MINTQA 包含 10,479 个用于评估新知识的问答对和 17,887 个用于评估长尾知识的问答对，每个问题都配有相应的子问题和答案。我们对 MINTQA 上 22 个最先进的 LLM 进行了系统评估，发现它们在处理复杂知识库查询方面的能力存在很大局限性，尤其是在处理新知识或不受欢迎的知识方面。我们的研究结果突出了关键挑战，并为提高多跳推理能力提供了见解。MINTQA 基准测试可在此 https URL 上找到。

Title: Shaping the Safety Boundaries: Understanding and Defending Against Jailbreaks in Large Language Models

• Authors: Lang Gao, Xiangliang Zhang, Preslav Nakov, Xiuying Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.17034
• Pdf URL: https://arxiv.org/pdf/2412.17034
• Copy Paste: [[2412.17034]] Shaping the Safety Boundaries: Understanding and Defending Against Jailbreaks in Large Language Models(https://arxiv.org/abs/2412.17034)
• Keywords: language model, llm
• Abstract: Jailbreaking in Large Language Models (LLMs) is a major security concern as it can deceive LLMs to generate harmful text. Yet, there is still insufficient understanding of how jailbreaking works, which makes it hard to develop effective defense strategies. We aim to shed more light into this issue: we conduct a detailed large-scale analysis of seven different jailbreak methods and find that these disagreements stem from insufficient observation samples. In particular, we introduce \textit{safety boundary}, and we find that jailbreaks shift harmful activations outside that safety boundary, where LLMs are less sensitive to harmful information. We also find that the low and the middle layers are critical in such shifts, while deeper layers have less impact. Leveraging on these insights, we propose a novel defense called \textbf{Activation Boundary Defense} (ABD), which adaptively constrains the activations within the safety boundary. We further use Bayesian optimization to selectively apply the defense method to the low and the middle layers. Our experiments on several benchmarks show that ABD achieves an average DSR of over 98\% against various forms of jailbreak attacks, with less than 2\% impact on the model's general capabilities.
• 摘要：大型语言模型 (LLM) 中的越狱是一个主要的安全问题，因为它可以欺骗 LLM 生成有害文本。然而，人们对越狱的工作原理仍然了解不足，这使得制定有效的防御策略变得困难。我们旨在进一步阐明这个问题：我们对七种不同的越狱方法进行了详细的大规模分析，发现这些分歧源于观察样本不足。特别是，我们引入了 \textit{安全边界}，我们发现越狱将有害激活转移到安全边界之外，而 LLM 对有害信息的敏感度较低。我们还发现低层和中层在这种转变中至关重要，而更深的层影响较小。利用这些见解，我们提出了一种名为 \textbf{激活边界防御} (ABD) 的新型防御方法，它可以自适应地将激活限制在安全边界内。我们进一步使用贝叶斯优化将防御方法选择性地应用于低层和中层。我们在多个基准测试上的实验表明，ABD 对各种形式的越狱攻击实现了超过 98% 的平均 DSR，对模型的一般功能的影响不到 2%。

Title: The HalluRAG Dataset: Detecting Closed-Domain Hallucinations in RAG Applications Using an LLM's Internal States

• Authors: Fabian Ridder, Malte Schilling
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.17056
• Pdf URL: https://arxiv.org/pdf/2412.17056
• Copy Paste: [[2412.17056]] The HalluRAG Dataset: Detecting Closed-Domain Hallucinations in RAG Applications Using an LLM's Internal States(https://arxiv.org/abs/2412.17056)
• Keywords: language model, llm, hallucination, prompt
• Abstract: Detecting hallucinations in large language models (LLMs) is critical for enhancing their reliability and trustworthiness. Most research focuses on hallucinations as deviations from information seen during training. However, the opaque nature of an LLM's parametric knowledge complicates the understanding of why generated texts appear ungrounded: The LLM might not have picked up the necessary knowledge from large and often inaccessible datasets, or the information might have been changed or contradicted during further training. Our focus is on hallucinations involving information not used in training, which we determine by using recency to ensure the information emerged after a cut-off date. This study investigates these hallucinations by detecting them at sentence level using different internal states of various LLMs. We present HalluRAG, a dataset designed to train classifiers on these hallucinations. Depending on the model and quantization, MLPs trained on HalluRAG detect hallucinations with test accuracies ranging up to 75 %, with Mistral-7B-Instruct-v0.1 achieving the highest test accuracies. Our results show that IAVs detect hallucinations as effectively as CEVs and reveal that answerable and unanswerable prompts are encoded differently as separate classifiers for these categories improved accuracy. However, HalluRAG showed some limited generalizability, advocating for more diversity in datasets on hallucinations.
• 摘要：在大型语言模型 (LLM) 中检测幻觉对于提高其可靠性和可信度至关重要。大多数研究将幻觉视为与训练期间看到的信息的偏差。然而，LLM 参数知识的不透明性使得理解生成的文本为何看起来毫无根据变得复杂：LLM 可能没有从大型且通常无法访问的数据集中获取必要的知识，或者信息可能在进一步训练期间被更改或矛盾。我们的重点是涉及训练中未使用的信息的幻觉，我们通过使用新近度来确定这些信息是否在截止日期之后出现。本研究通过使用各种 LLM 的不同内部状态在句子级别检测这些幻觉来调查这些幻觉。我们提出了 HalluRAG，这是一个旨在针对这些幻觉训练分类器的数据集。根据模型和量化，在 HalluRAG 上训练的 MLP 检测幻觉的测试准确率高达 75%，其中 Mistral-7B-Instruct-v0.1 的测试准确率最高。我们的结果表明，IAV 检测幻觉的效果与 CEV 一样好，并且表明可回答和不可回答的提示以不同的方式编码为这些类别的单独分类器，从而提高了准确率。然而，HalluRAG 表现出一定的局限性，主张幻觉数据集应更加多样化。

Title: Multi-Agent Sampling: Scaling Inference Compute for Data Synthesis with Tree Search-Based Agentic Collaboration

• Authors: Hai Ye, Mingbao Lin, Hwee Tou Ng, Shuicheng Yan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.17061
• Pdf URL: https://arxiv.org/pdf/2412.17061
• Copy Paste: [[2412.17061]] Multi-Agent Sampling: Scaling Inference Compute for Data Synthesis with Tree Search-Based Agentic Collaboration(https://arxiv.org/abs/2412.17061)
• Keywords: language model, agent
• Abstract: Scaling laws for inference compute in multi-agent systems remain under-explored compared to single-agent scenarios. This work aims to bridge this gap by investigating the problem of data synthesis through multi-agent sampling, where synthetic responses are generated by sampling from multiple distinct language models. Effective model coordination is crucial for successful multi-agent collaboration. Unlike previous approaches that rely on fixed workflows, we treat model coordination as a multi-step decision-making process, optimizing generation structures dynamically for each input question. We introduce Tree Search-based Orchestrated Agents~(TOA), where the workflow evolves iteratively during the sequential sampling process. To achieve this, we leverage Monte Carlo Tree Search (MCTS), integrating a reward model to provide real-time feedback and accelerate exploration. Our experiments on alignment, machine translation, and mathematical reasoning demonstrate that multi-agent sampling significantly outperforms single-agent sampling as inference compute scales. TOA is the most compute-efficient approach, achieving SOTA performance on WMT and a 71.8\% LC win rate on AlpacaEval. Moreover, fine-tuning with our synthesized alignment data surpasses strong preference learning methods on challenging benchmarks such as Arena-Hard and AlpacaEval.
• 摘要：与单智能体场景相比，多智能体系统中推理计算的扩展规律仍未得到充分探索。这项工作旨在通过研究多智能体采样的数据合成问题来弥补这一差距，其中通过从多个不同的语言模型中采样生成合成响应。有效的模型协调对于成功的多智能体协作至关重要。与以前依赖固定工作流的方法不同，我们将模型协调视为一个多步骤的决策过程，为每个输入问题动态优化生成结构。我们引入了基于树搜索的协调代理~(TOA)，其中工作流在顺序采样过程中迭代发展。为了实现这一点，我们利用蒙特卡洛树搜索 (MCTS)，集成奖励模型来提供实时反馈并加速探索。我们在对齐、机器翻译和数学推理方面的实验表明，随着推理计算的扩展，多智能体采样明显优于单智能体采样。TOA 是计算效率最高的方法，在 WMT 上实现了 SOTA 性能，在 AlpacaEval 上实现了 71.8% 的 LC 胜率。此外，使用我们的合成对齐数据进行微调超越了 Arena-Hard 和 AlpacaEval 等具有挑战性的基准上的强偏好学习方法。

Title: SAIL: Sample-Centric In-Context Learning for Document Information Extraction

• Authors: Jinyu Zhang, Zhiyuan You, Jize Wang, Xinyi Le
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.17092
• Pdf URL: https://arxiv.org/pdf/2412.17092
• Copy Paste: [[2412.17092]] SAIL: Sample-Centric In-Context Learning for Document Information Extraction(https://arxiv.org/abs/2412.17092)
• Keywords: language model, llm, prompt
• Abstract: Document Information Extraction (DIE) aims to extract structured information from Visually Rich Documents (VRDs). Previous full-training approaches have demonstrated strong performance but may struggle with generalization to unseen data. In contrast, training-free methods leverage powerful pre-trained models like Large Language Models (LLMs) to address various downstream tasks with only a few examples. Nonetheless, training-free methods for DIE encounter two primary challenges: (1) understanding the complex relationship between layout and textual elements in VRDs, and (2) providing accurate guidance to pre-trained models. To address these challenges, we propose Sample-centric In-context Learning (SAIL) for DIE. SAIL introduces a fine-grained entity-level textual similarity to facilitate in-depth text analysis by LLMs and incorporates layout similarity to enhance the analysis of layouts in VRDs. Additionally, SAIL formulates a unified In-Context Learning (ICL) prompt template for various sample-centric examples, enabling tailored prompts that deliver precise guidance to pre-trained models for each sample. Extensive experiments on FUNSD, CORD, and SROIE benchmarks with various base models (e.g., LLMs) indicate that our method outperforms training-free baselines, even closer to the full-training methods. The results show the superiority and generalization of our method.
• 摘要：文档信息提取 (DIE) 旨在从视觉丰富的文档 (VRD) 中提取结构化信息。以前的全训练方法已经表现出强大的性能，但可能难以推广到看不见的数据。相比之下，无需训练的方法利用强大的预训练模型（如大型语言模型 (LLM)）仅用几个示例即可解决各种下游任务。尽管如此，DIE 的无需训练方法仍面临两个主要挑战：(1) 理解 VRD 中布局和文本元素之间的复杂关系，以及 (2) 为预训练模型提供准确的指导。为了应对这些挑战，我们为 DIE 提出了以样本为中心的上下文学习 (SAIL)。SAIL 引入了细粒度的实体级文本相似性，以促进 LLM 进行深入的文本分析，并结合布局相似性以增强对 VRD 中布局的分析。此外，SAIL 为各种以样本为中心的示例制定了统一的上下文学习 (ICL) 提示模板，从而能够为每个样本提供定制的提示，为预训练模型提供精确的指导。使用各种基础模型（例如 LLM）在 FUNSD、CORD 和 SROIE 基准上进行的大量实验表明，我们的方法优于无训练基线，甚至更接近全训练方法。结果证明了我们方法的优越性和泛化性。

Title: Learning to Adapt to Low-Resource Paraphrase Generation

• Authors: Zhigen Li, Yanmeng Wang, Rizhao Fan, Ye Wang, Jianfeng Li, Shaojun Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.17111
• Pdf URL: https://arxiv.org/pdf/2412.17111
• Copy Paste: [[2412.17111]] Learning to Adapt to Low-Resource Paraphrase Generation(https://arxiv.org/abs/2412.17111)
• Keywords: language model
• Abstract: Paraphrase generation is a longstanding NLP task and achieves great success with the aid of large corpora. However, transferring a paraphrasing model to another domain encounters the problem of domain shifting especially when the data is sparse. At the same time, widely using large pre-trained language models (PLMs) faces the overfitting problem when training on scarce labeled data. To mitigate these two issues, we propose, LAPA, an effective adapter for PLMs optimized by meta-learning. LAPA has three-stage training on three types of related resources to solve this problem: 1. pre-training PLMs on unsupervised corpora, 2. inserting an adapter layer and meta-training on source domain labeled data, and 3. fine-tuning adapters on a small amount of target domain labeled data. This method enables paraphrase generation models to learn basic language knowledge first, then learn the paraphrasing task itself later, and finally adapt to the target task. Our experimental results demonstrate that LAPA achieves state-of-the-art in supervised, unsupervised, and low-resource settings on three benchmark datasets. With only 2\% of trainable parameters and 1\% labeled data of the target task, our approach can achieve a competitive performance with previous work.
• 摘要：释义生成是一项长期存在的 NLP 任务，并借助大型语料库取得了巨大成功。然而，将释义模型转移到另一个领域会遇到领域转移的问题，尤其是在数据稀疏的情况下。同时，广泛使用的大型预训练语言模型 (PLM) 在对稀缺标记数据进行训练时面临过度拟合的问题。为了缓解这两个问题，我们提出了 LAPA，一种通过元学习优化的 PLM 有效适配器。LAPA 在三种相关资源上进行三阶段训练以解决这个问题：1. 在无监督语料库上对 PLM 进行预训练，2. 插入适配器层并在源域标记数据上进行元训练，3. 在少量目标域标记数据上微调适配器。这种方法使释义生成模型首先学习基本的语言知识，然后再学习释义任务本身，最后适应目标任务。我们的实验结果表明，LAPA 在三个基准数据集上的监督、无监督和低资源设置中均达到了最佳水平。仅使用目标任务的 2% 可训练参数和 1% 标记数据，我们的方法就可以实现与之前工作相媲美的性能。

Title: Lies, Damned Lies, and Distributional Language Statistics: Persuasion and Deception with Large Language Models

• Authors: Cameron R. Jones, Benjamin K. Bergen
• Subjects: cs.CL, cs.CY, cs.HC
• Abstract URL: https://arxiv.org/abs/2412.17128
• Pdf URL: https://arxiv.org/pdf/2412.17128
• Copy Paste: [[2412.17128]] Lies, Damned Lies, and Distributional Language Statistics: Persuasion and Deception with Large Language Models(https://arxiv.org/abs/2412.17128)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) can generate content that is as persuasive as human-written text and appear capable of selectively producing deceptive outputs. These capabilities raise concerns about potential misuse and unintended consequences as these systems become more widely deployed. This review synthesizes recent empirical work examining LLMs' capacity and proclivity for persuasion and deception, analyzes theoretical risks that could arise from these capabilities, and evaluates proposed mitigations. While current persuasive effects are relatively small, various mechanisms could increase their impact, including fine-tuning, multimodality, and social factors. We outline key open questions for future research, including how persuasive AI systems might become, whether truth enjoys an inherent advantage over falsehoods, and how effective different mitigation strategies may be in practice.
• 摘要：大型语言模型 (LLM) 可以生成与人类书写的文本一样具有说服力的内容，并且似乎能够选择性地产生欺骗性输出。随着这些系统的广泛部署，这些功能引发了人们对潜在滥用和意外后果的担忧。本综述综合了最近对 LLM 的说服和欺骗能力和倾向的实证研究，分析了这些能力可能产生的理论风险，并评估了拟议的缓解措施。虽然目前的说服效果相对较小，但各种机制可以增加其影响，包括微调、多模态和社会因素。我们概述了未来研究的关键未决问题，包括人工智能系统可能变得多么有说服力，真相是否比谎言具有固有的优势，以及不同的缓解策略在实践中的有效性。

Title: Hate Speech Detection and Target Identification in Devanagari Languages via Parameter Efficient Fine-Tuning of LLMs

• Authors: Rushendra Sidibomma, Pransh Patwa, Parth Patwa, Aman Chadha, Vinija Jain, Amitava Das
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.17131
• Pdf URL: https://arxiv.org/pdf/2412.17131
• Copy Paste: [[2412.17131]] Hate Speech Detection and Target Identification in Devanagari Languages via Parameter Efficient Fine-Tuning of LLMs(https://arxiv.org/abs/2412.17131)
• Keywords: language model, llm
• Abstract: The detection of hate speech has become increasingly important in combating online hostility and its real-world consequences. Despite recent advancements, there is limited research addressing hate speech detection in Devanagari-scripted languages, where resources and tools are scarce. While large language models (LLMs) have shown promise in language-related tasks, traditional fine-tuning approaches are often infeasible given the size of the models. In this paper, we propose a Parameter Efficient Fine tuning (PEFT) based solution for hate speech detection and target identification. We evaluate multiple LLMs on the Devanagari dataset provided by (Thapa et al., 2025), which contains annotated instances in 2 languages - Hindi and Nepali. The results demonstrate the efficacy of our approach in handling Devanagari-scripted content.
• 摘要：在打击网络敌意及其现实后果方面，仇恨言论检测变得越来越重要。尽管最近取得了一些进展，但由于资源和工具稀缺，针对 Devanagari 脚本语言中仇恨言论检测的研究有限。虽然大型语言模型 (LLM) 在语言相关任务中显示出良好的前景，但考虑到模型的大小，传统的微调方法往往不可行。在本文中，我们提出了一种基于参数高效微调 (PEFT) 的仇恨言论检测和目标识别解决方案。我们在 (Thapa et al., 2025) 提供的 Devanagari 数据集上评估了多个 LLM，该数据集包含两种语言（印地语和尼泊尔语）的带注释实例。结果证明了我们的方法在处理 Devanagari 脚本内容方面的有效性。

Title: A Multi-AI Agent System for Autonomous Optimization of Agentic AI Solutions via Iterative Refinement and LLM-Driven Feedback Loops

• Authors: Kamer Ali Yuksel, Hassan Sawaf
• Subjects: cs.CL, cs.AI, cs.ET, cs.MA, cs.NE
• Abstract URL: https://arxiv.org/abs/2412.17149
• Pdf URL: https://arxiv.org/pdf/2412.17149
• Copy Paste: [[2412.17149]] A Multi-AI Agent System for Autonomous Optimization of Agentic AI Solutions via Iterative Refinement and LLM-Driven Feedback Loops(https://arxiv.org/abs/2412.17149)
• Keywords: llm, agent
• Abstract: Agentic AI systems use specialized agents to handle tasks within complex workflows, enabling automation and efficiency. However, optimizing these systems often requires labor-intensive, manual adjustments to refine roles, tasks, and interactions. This paper introduces a framework for autonomously optimizing Agentic AI solutions across industries, such as NLP-driven enterprise applications. The system employs agents for Refinement, Execution, Evaluation, Modification, and Documentation, leveraging iterative feedback loops powered by an LLM (Llama 3.2-3B). The framework achieves optimal performance without human input by autonomously generating and testing hypotheses to improve system configurations. This approach enhances scalability and adaptability, offering a robust solution for real-world applications in dynamic environments. Case studies across diverse domains illustrate the transformative impact of this framework, showcasing significant improvements in output quality, relevance, and actionability. All data for these case studies, including original and evolved agent codes, along with their outputs, are here: this https URL
• 摘要：Agentic AI 系统使用专门的代理来处理复杂工作流程中的任务，从而实现自动化和效率。但是，优化这些系统通常需要进行劳动密集型的手动调整，以优化角色、任务和交互。本文介绍了一个跨行业（例如 NLP 驱动的企业应用程序）自主优化 Agentic AI 解决方案的框架。该系统使用代理进行细化、执行、评估、修改和文档编制，利用由 LLM（Llama 3.2-3B）提供支持的迭代反馈循环。该框架通过自主生成和测试假设来改进系统配置，无需人工输入即可实现最佳性能。这种方法增强了可扩展性和适应性，为动态环境中的实际应用提供了强大的解决方案。不同领域的案例研究说明了该框架的变革性影响，展示了输出质量、相关性和可操作性的显着改进。这些案例研究的所有数据（包括原始和演化后的代理代码及其输出）都在此处：此 https URL

Title: Brain-to-Text Benchmark '24: Lessons Learned

• Authors: Francis R. Willett, Jingyuan Li, Trung Le, Chaofei Fan, Mingfei Chen, Eli Shlizerman, Yue Chen, Xin Zheng, Tatsuo S. Okubo, Tyler Benster, Hyun Dong Lee, Maxwell Kounga, E. Kelly Buchanan, David Zoltowski, Scott W. Linderman, Jaimie M. Henderson
• Subjects: cs.CL, cs.LG, q-bio.NC
• Abstract URL: https://arxiv.org/abs/2412.17227
• Pdf URL: https://arxiv.org/pdf/2412.17227
• Copy Paste: [[2412.17227]] Brain-to-Text Benchmark '24: Lessons Learned(https://arxiv.org/abs/2412.17227)
• Keywords: language model
• Abstract: Speech brain-computer interfaces aim to decipher what a person is trying to say from neural activity alone, restoring communication to people with paralysis who have lost the ability to speak intelligibly. The Brain-to-Text Benchmark '24 and associated competition was created to foster the advancement of decoding algorithms that convert neural activity to text. Here, we summarize the lessons learned from the competition ending on June 1, 2024 (the top 4 entrants also presented their experiences in a recorded webinar). The largest improvements in accuracy were achieved using an ensembling approach, where the output of multiple independent decoders was merged using a fine-tuned large language model (an approach used by all 3 top entrants). Performance gains were also found by improving how the baseline recurrent neural network (RNN) model was trained, including by optimizing learning rate scheduling and by using a diphone training objective. Improving upon the model architecture itself proved more difficult, however, with attempts to use deep state space models or transformers not yet appearing to offer a benefit over the RNN baseline. The benchmark will remain open indefinitely to support further work towards increasing the accuracy of brain-to-text algorithms.
• 摘要：语音脑机接口旨在仅从神经活动中解读一个人想要说的话，恢复瘫痪患者与他人的交流，使他们能够清晰地说话。Brain-to-Text Benchmark '24 和相关竞赛旨在促进将神经活动转换为文本的解码算法的发展。在这里，我们总结了 2024 年 6 月 1 日结束的竞赛的经验教训（前 4 名参赛者也在录制的网络研讨会中介绍了他们的经验）。使用集成方法实现了最大的准确度改进，其中使用经过微调的大型语言模型合并多个独立解码器的输出（所有前 3 名参赛者都使用这种方法）。通过改进基线循环神经网络 (RNN) 模型的训练方式，包括优化学习率调度和使用双音素训练目标，也发现了性能提升。然而，改进模型架构本身被证明更加困难，因为尝试使用深度状态空间模型或变压器似乎尚未提供优于 RNN 基线的优势。该基准将无限期开​​放，以支持进一步提高脑到文本算法的准确性。

Title: Unlocking Cross-Lingual Sentiment Analysis through Emoji Interpretation: A Multimodal Generative AI Approach

• Authors: Rafid Ishrak Jahan, Heng Fan, Haihua Chen, Yunhe Feng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.17255
• Pdf URL: https://arxiv.org/pdf/2412.17255
• Copy Paste: [[2412.17255]] Unlocking Cross-Lingual Sentiment Analysis through Emoji Interpretation: A Multimodal Generative AI Approach(https://arxiv.org/abs/2412.17255)
• Keywords: language model, gpt, llm, chat
• Abstract: Emojis have become ubiquitous in online communication, serving as a universal medium to convey emotions and decorative elements. Their widespread use transcends language and cultural barriers, enhancing understanding and fostering more inclusive interactions. While existing work gained valuable insight into emojis understanding, exploring emojis' capability to serve as a universal sentiment indicator leveraging large language models (LLMs) has not been thoroughly examined. Our study aims to investigate the capacity of emojis to serve as reliable sentiment markers through LLMs across languages and cultures. We leveraged the multimodal capabilities of ChatGPT to explore the sentiments of various representations of emojis and evaluated how well emoji-conveyed sentiment aligned with text sentiment on a multi-lingual dataset collected from 32 countries. Our analysis reveals that the accuracy of LLM-based emoji-conveyed sentiment is 81.43%, underscoring emojis' significant potential to serve as a universal sentiment marker. We also found a consistent trend that the accuracy of sentiment conveyed by emojis increased as the number of emojis grew in text. The results reinforce the potential of emojis to serve as global sentiment indicators, offering insight into fields such as cross-lingual and cross-cultural sentiment analysis on social media platforms. Code: this https URL.
• 摘要：表情符号在在线交流中无处不在，是传达情感和装饰元素的通用媒介。它们的广​​泛使用超越了语言和文化障碍，增进了理解并促进了更具包容性的互动。虽然现有研究对表情符号的理解获得了宝贵的见解，但尚未彻底研究利用大型语言模型 (LLM) 探索表情符号作为通用情绪指标的能力。我们的研究旨在通过跨语言和文化的 LLM 调查表情符号作为可靠情绪标记的能力。我们利用 ChatGPT 的多模态功能探索表情符号各种表示的情绪，并在从 32 个国家/地区收集的多语言数据集上评估表情符号传达的情绪与文本情绪的一致程度。我们的分析表明，基于 LLM 的表情符号传达情绪的准确率为 81.43%，凸显了表情符号作为通用情绪标记的巨大潜力。我们还发现了一个一致的趋势，即随着文本中表情符号数量的增加，表情符号传达情绪的准确性也会提高。结果进一步证实了表情符号作为全球情绪指标的潜力，为社交媒体平台上的跨语言和跨文化情绪分析等领域提供了见解。代码：此 https URL。

Title: LegalAgentBench: Evaluating LLM Agents in Legal Domain

• Authors: Haitao Li, Junjie Chen, Jingli Yang, Qingyao Ai, Wei Jia, Youfeng Liu, Kai Lin, Yueyue Wu, Guozhi Yuan, Yiran Hu, Wuyue Wang, Yiqun Liu, Minlie Huang
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2412.17259
• Pdf URL: https://arxiv.org/pdf/2412.17259
• Copy Paste: [[2412.17259]] LegalAgentBench: Evaluating LLM Agents in Legal Domain(https://arxiv.org/abs/2412.17259)
• Keywords: llm, agent
• Abstract: With the increasing intelligence and autonomy of LLM agents, their potential applications in the legal domain are becoming increasingly apparent. However, existing general-domain benchmarks cannot fully capture the complexity and subtle nuances of real-world judicial cognition and decision-making. Therefore, we propose LegalAgentBench, a comprehensive benchmark specifically designed to evaluate LLM Agents in the Chinese legal domain. LegalAgentBench includes 17 corpora from real-world legal scenarios and provides 37 tools for interacting with external knowledge. We designed a scalable task construction framework and carefully annotated 300 tasks. These tasks span various types, including multi-hop reasoning and writing, and range across different difficulty levels, effectively reflecting the complexity of real-world legal scenarios. Moreover, beyond evaluating final success, LegalAgentBench incorporates keyword analysis during intermediate processes to calculate progress rates, enabling more fine-grained evaluation. We evaluated eight popular LLMs, highlighting the strengths, limitations, and potential areas for improvement of existing models and methods. LegalAgentBench sets a new benchmark for the practical application of LLMs in the legal domain, with its code and data available at \url{this https URL}.
• 摘要：随着法学硕士代理的智能化和自主性不断增强，其在法律领域的潜在应用也日益凸显。然而，现有的通用领域基准测试无法完全捕捉现实世界司法认知和决策的复杂性和细微差别。因此，我们提出了 LegalAgentBench，这是一个专门用于评估中国法律领域法学硕士代理的综合基准测试。LegalAgentBench 包括来自现实世界法律场景的 17 个语料库，并提供了 37 种与外部知识交互的工具。我们设计了一个可扩展的任务构建框架，并仔细注释了 300 项任务。这些任务涵盖多种类型，包括多跳推理和写作，难度级别各不相同，有效反映了现实世界法律场景的复杂性。此外，除了评估最终成功率之外，LegalAgentBench 还在中间过程中结合关键字分析来计算进度率，从而实现更细粒度的评估。我们评估了八种流行的法学硕士，突出了现有模型和方法的优势、局限性和潜在的改进领域。 LegalAgentBench 为法学硕士在法律领域的实际应用设立了新的基准，其代码和数据可在 \url{此 https URL} 上找到。

Title: Assessing Human Editing Effort on LLM-Generated Texts via Compression-Based Edit Distance

• Authors: Nicolas Devatine, Louis Abraham
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.17321
• Pdf URL: https://arxiv.org/pdf/2412.17321
• Copy Paste: [[2412.17321]] Assessing Human Editing Effort on LLM-Generated Texts via Compression-Based Edit Distance(https://arxiv.org/abs/2412.17321)
• Keywords: language model, llm
• Abstract: Assessing the extent of human edits on texts generated by Large Language Models (LLMs) is crucial to understanding the human-AI interactions and improving the quality of automated text generation systems. Existing edit distance metrics, such as Levenshtein, BLEU, ROUGE, and TER, often fail to accurately measure the effort required for post-editing, especially when edits involve substantial modifications, such as block operations. In this paper, we introduce a novel compression-based edit distance metric grounded in the Lempel-Ziv-77 algorithm, designed to quantify the amount of post-editing applied to LLM-generated texts. Our method leverages the properties of text compression to measure the informational difference between the original and edited texts. Through experiments on real-world human edits datasets, we demonstrate that our proposed metric is highly correlated with actual edit time and effort. We also show that LLMs exhibit an implicit understanding of editing speed, that aligns well with our metric. Furthermore, we compare our metric with existing ones, highlighting its advantages in capturing complex edits with linear computational efficiency. Our code and data are available at: this https URL
• 摘要：评估大型语言模型 (LLM) 生成的文本的人工编辑程度对于理解人机交互和提高自动文本生成系统的质量至关重要。现有的编辑距离指标（例如 Levenshtein、BLEU、ROUGE 和 TER）通常无法准确衡量后期编辑所需的工作量，尤其是当编辑涉及大量修改（例如块操作）时。在本文中，我们介绍了一种基于 Lempel-Ziv-77 算法的新型压缩编辑距离指标，旨在量化应用于 LLM 生成文本的后期编辑量。我们的方法利用文本压缩的属性来衡量原始文本和编辑文本之间的信息差异。通过对现实世界的人工编辑数据集进行实验，我们证明了我们提出的指标与实际编辑时间和工作量高度相关。我们还表明 LLM 表现出对编辑速度的隐性理解，这与我们的指标非常吻合。此外，我们将我们的指标与现有指标进行了比较，突出了其在以线性计算效率捕获复杂编辑方面的优势。我们的代码和数据可从以下网址获取：此 https URL

Title: A Dual-Perspective Metaphor Detection Framework Using Large Language Models

• Authors: Yujie Lin, Jingyao Liu, Yan Gao, Ante Wang, Jinsong Su
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.17332
• Pdf URL: https://arxiv.org/pdf/2412.17332
• Copy Paste: [[2412.17332]] A Dual-Perspective Metaphor Detection Framework Using Large Language Models(https://arxiv.org/abs/2412.17332)
• Keywords: language model, llm
• Abstract: Metaphor detection, a critical task in natural language processing, involves identifying whether a particular word in a sentence is used metaphorically. Traditional approaches often rely on supervised learning models that implicitly encode semantic relationships based on metaphor theories. However, these methods often suffer from a lack of transparency in their decision-making processes, which undermines the reliability of their predictions. Recent research indicates that LLMs (large language models) exhibit significant potential in metaphor detection. Nevertheless, their reasoning capabilities are constrained by predefined knowledge graphs. To overcome these limitations, we propose DMD, a novel dual-perspective framework that harnesses both implicit and explicit applications of metaphor theories to guide LLMs in metaphor detection and adopts a self-judgment mechanism to validate the responses from the aforementioned forms of guidance. In comparison to previous methods, our framework offers more transparent reasoning processes and delivers more reliable predictions. Experimental results prove the effectiveness of DMD, demonstrating state-of-the-art performance across widely-used datasets.
• 摘要：隐喻检测是自然语言处理中的一项关键任务，涉及识别句子中的某个单词是否被用作隐喻。传统方法通常依赖于基于隐喻理论隐式编码语义关系的监督学习模型。然而，这些方法的决策过程往往缺乏透明度，从而削弱了其预测的可靠性。最近的研究表明，LLM（大型语言模型）在隐喻检测方面表现出巨大的潜力。然而，它们的推理能力受到预定义知识图的限制。为了克服这些限制，我们提出了 DMD，这是一种新颖的双视角框架，它利用隐喻理论的隐式和显式应用来指导 LLM 进行隐喻检测，并采用自我判断机制来验证上述指导形式的反应。与以前的方法相比，我们的框架提供了更透明的推理过程并提供更可靠的预测。实验结果证明了 DMD 的有效性，在广泛使用的数据集上展示了最先进的性能。

Title: Boosting LLM via Learning from Data Iteratively and Selectively

• Authors: Qi Jia, Siyu Ren, Ziheng Qin, Fuzhao Xue, Jinjie Ni, Yang You
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.17365
• Pdf URL: https://arxiv.org/pdf/2412.17365
• Copy Paste: [[2412.17365]] Boosting LLM via Learning from Data Iteratively and Selectively(https://arxiv.org/abs/2412.17365)
• Keywords: llm
• Abstract: Datasets nowadays are generally constructed from multiple sources and using different synthetic techniques, making data de-noising and de-duplication crucial before being used for post-training. In this work, we propose to perform instruction tuning by iterative data selection (\ApproachName{}). We measure the quality of a sample from complexity and diversity simultaneously. Instead of calculating the complexity score once for all before fine-tuning, we highlight the importance of updating this model-specific score during fine-tuning to accurately accommodate the dynamic changes of the model. On the other hand, the diversity score is defined on top of the samples' responses under the consideration of their informativeness. IterIT integrates the strengths of both worlds by iteratively updating the complexity score for the top-ranked samples and greedily selecting the ones with the highest complexity-diversity score. Experiments on multiple instruction-tuning data demonstrate consistent improvements of IterIT over strong baselines. Moreover, our approach also generalizes well to domain-specific scenarios and different backbone models. All resources will be available at this https URL.
• 摘要：如今的数据集通常由多个来源构建并使用不同的合成技术，因此在用于后训练之前，数据去噪和去重至关重要。在这项工作中，我们建议通过迭代数据选择 (\ApproachName{}) 执行指令调整。我们同时从复杂性和多样性的角度衡量样本的质量。我们不是在微调之前一次性计算复杂性得分，而是强调在微调期间更新此模型特定得分的重要性，以准确适应模型的动态变化。另一方面，多样性得分是在考虑样本信息量的情况下根据样本响应定义的。IterIT 通过迭代更新排名靠前的样本的复杂性得分并贪婪地选择具有最高复杂性多样性得分的样本，将两全其美的优势融为一体。在多个指令调整数据上进行的实验表明，IterIT 在强基线上持续改进。此外，我们的方法还可以很好地推广到特定领域的场景和不同的骨干模型。所有资源都将在此 https URL 上提供。

Title: Interweaving Memories of a Siamese Large Language Model

• Authors: Xin Song, Zhikai Xue, Guoxiu He, Jiawei Liu, Wei Lu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.17383
• Pdf URL: https://arxiv.org/pdf/2412.17383
• Copy Paste: [[2412.17383]] Interweaving Memories of a Siamese Large Language Model(https://arxiv.org/abs/2412.17383)
• Keywords: language model, llm
• Abstract: Parameter-efficient fine-tuning (PEFT) methods optimize large language models (LLMs) by modifying or introducing a small number of parameters to enhance alignment with downstream tasks. However, they can result in catastrophic forgetting, where LLMs prioritize new knowledge at the expense of comprehensive world knowledge. A promising approach to mitigate this issue is to recall prior memories based on the original knowledge. To this end, we propose a model-agnostic PEFT framework, IMSM, which Interweaves Memories of a Siamese Large Language Model. Specifically, our siamese LLM is equipped with an existing PEFT method. Given an incoming query, it generates two distinct memories based on the pre-trained and fine-tuned parameters. IMSM then incorporates an interweaving mechanism that regulates the contributions of both original and enhanced memories when generating the next token. This framework is theoretically applicable to all open-source LLMs and existing PEFT methods. We conduct extensive experiments across various benchmark datasets, evaluating the performance of popular open-source LLMs using the proposed IMSM, in comparison to both classical and leading PEFT methods. Our findings indicate that IMSM maintains comparable time and space efficiency to backbone PEFT methods while significantly improving performance and effectively mitigating catastrophic forgetting.
• 摘要：参数高效微调 (PEFT) 方法通过修改或引入少量参数来优化大型语言模型 (LLM)，以增强与下游任务的一致性。然而，它们可能导致灾难性的遗忘，即 LLM 优先考虑新知识而牺牲了全面的世界知识。缓解此问题的一种有希望的方法是根据原始知识回忆以前的记忆。为此，我们提出了一个与模型无关的 PEFT 框架 IMSM，它将暹罗大型语言模型的记忆交织在一起。具体来说，我们的暹罗 LLM 配备了现有的 PEFT 方法。给定一个传入查询，它会根据预训练和微调的参数生成两个不同的记忆。然后，IMSM 结合了一种交织机制，在生成下一个标记时调节原始记忆和增强记忆的贡献。该框架理论上适用于所有开源 LLM 和现有的 PEFT 方法。我们在各种基准数据集上进行了广泛的实验，使用所提出的 IMSM 评估了流行的开源 LLM 的性能，并与经典和领先的 PEFT 方法进行了比较。我们的研究结果表明，IMSM 保持了与骨干 PEFT 方法相当的时间和空间效率，同时显著提高了性能并有效缓解了灾难性遗忘。

Title: WarriorCoder: Learning from Expert Battles to Augment Code Large Language Models

• Authors: Huawen Feng, Pu Zhao, Qingfeng Sun, Can Xu, Fangkai Yang, Lu Wang, Qianli Ma, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, Qi Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.17395
• Pdf URL: https://arxiv.org/pdf/2412.17395
• Copy Paste: [[2412.17395]] WarriorCoder: Learning from Expert Battles to Augment Code Large Language Models(https://arxiv.org/abs/2412.17395)
• Keywords: language model, gpt, llm
• Abstract: Despite recent progress achieved by code large language models (LLMs), their remarkable abilities are largely dependent on fine-tuning on the high-quality data, posing challenges for data collection and annotation. To address this, current methods often design various data flywheels to gather complex code instructions, enabling models to handle more intricate tasks. However, these approaches typically rely on off-the-shelf datasets and data augmentation from the limited pool of proprietary LLMs (e.g., Claude, GPT4, and so on), which limits the diversity of the constructed data and makes it prone to systemic biases. In this paper, we propose WarriorCoder which learns from expert battles to address these limitations. Specifically, we create an arena for current expert code LLMs, where each model challenges and responds to others' challenges, with evaluations conducted by uninvolved judge models. This competitive framework generates novel training data constructed from scratch, harnessing the strengths of all participants. Experimental results demonstrate that WarriorCoder achieves competitive performance compared to previous methods, even without relying on proprietary LLMs.
• 摘要：尽管大型语言模型 (LLM) 代码最近取得了进展，但它们的卓越能力在很大程度上取决于对高质量数据的微调，这对数据收集和注释提出了挑战。为了解决这个问题，当前的方法通常设计各种数据飞轮来收集复杂的代码指令，使模型能够处理更复杂的任务。然而，这些方法通常依赖于现成的数据集和来自有限专有 LLM 池（例如 Claude、GPT4 等）的数据增强，这限制了构建数据的多样性并使其容易受到系统性偏差的影响。在本文中，我们提出了 WarriorCoder，它从专家战斗中学习以解决这些限制。具体来说，我们为当前的专家代码 LLM 创建了一个竞技场，每个模型都会挑战并回应其他模型的挑战，并由未参与的评判模型进行评估。这个竞争框架生成从头开始构建的新训练数据，利用所有参与者的优势。实验结果表明，即使不依赖专有 LLM，WarriorCoder 也能实现与以前的方法相比具有竞争力的性能。

Title: Just What You Desire: Constrained Timeline Summarization with Self-Reflection for Enhanced Relevance

• Authors: Muhammad Reza Qorib, Qisheng Hu, Hwee Tou Ng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.17408
• Pdf URL: https://arxiv.org/pdf/2412.17408
• Copy Paste: [[2412.17408]] Just What You Desire: Constrained Timeline Summarization with Self-Reflection for Enhanced Relevance(https://arxiv.org/abs/2412.17408)
• Keywords: language model, llm
• Abstract: Given news articles about an entity, such as a public figure or organization, timeline summarization (TLS) involves generating a timeline that summarizes the key events about the entity. However, the TLS task is too underspecified, since what is of interest to each reader may vary, and hence there is not a single ideal or optimal timeline. In this paper, we introduce a novel task, called Constrained Timeline Summarization (CTLS), where a timeline is generated in which all events in the timeline meet some constraint. An example of a constrained timeline concerns the legal battles of Tiger Woods, where only events related to his legal problems are selected to appear in the timeline. We collected a new human-verified dataset of constrained timelines involving 47 entities and 5 constraints per entity. We propose an approach that employs a large language model (LLM) to summarize news articles according to a specified constraint and cluster them to identify key events to include in a constrained timeline. In addition, we propose a novel self-reflection method during summary generation, demonstrating that this approach successfully leads to improved performance.
• 摘要：给定有关某个实体（例如公众人物或组织）的新闻文章，时间线摘要 (TLS) 涉及生​​成一条时间线，总结有关该实体的关键事件。然而，TLS 任务太不明确，因为每个读者的兴趣可能不同，因此没有单一的理想或最佳时间线。在本文中，我们介绍了一项名为受限时间线摘要 (CTLS) 的新任务，其中生成一条时间线，其中时间线中的所有事件都满足一些约束。受限时间线的一个例子涉及泰格·伍兹的法律纠纷，其中只有与他的法律问题相关的事件才会被选中出现在时间线中。我们收集了一个新的人工验证的受限时间线数据集，涉及 47 个实体和每个实体 5 个约束。我们提出了一种方法，该方法采用大型语言模型 (LLM) 根据指定的约束总结新闻文章并对其进行聚类以确定要包含在受限时间线中的关键事件。此外，我们提出了一种在摘要生成过程中进行新颖的自我反思方法，证明这种方法可以成功提高性能。

Title: Measuring Contextual Informativeness in Child-Directed Text

• Authors: Maria Valentini, Téa Wright, Ali Marashian, Jennifer Weber, Eliana Colunga, Katharina von der Wense
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.17427
• Pdf URL: https://arxiv.org/pdf/2412.17427
• Copy Paste: [[2412.17427]] Measuring Contextual Informativeness in Child-Directed Text(https://arxiv.org/abs/2412.17427)
• Keywords: language model, llm
• Abstract: To address an important gap in creating children's stories for vocabulary enrichment, we investigate the automatic evaluation of how well stories convey the semantics of target vocabulary words, a task with substantial implications for generating educational content. We motivate this task, which we call measuring contextual informativeness in children's stories, and provide a formal task definition as well as a dataset for the task. We further propose a method for automating the task using a large language model (LLM). Our experiments show that our approach reaches a Spearman correlation of 0.4983 with human judgments of informativeness, while the strongest baseline only obtains a correlation of 0.3534. An additional analysis shows that the LLM-based approach is able to generalize to measuring contextual informativeness in adult-directed text, on which it also outperforms all baselines.
• 摘要：为了解决在创作儿童故事以丰富词汇量方面存在的一个重要差距，我们研究了如何自动评估故事如何很好地传达目标词汇的语义，这项任务对于生成教育内容具有重大意义。我们激发了这项任务，我们称之为测量儿童故事中的上下文信息量，并为该任务提供了正式的任务定义和数据集。我们还提出了一种使用大型语言模型 (LLM) 自动执行该任务的方法。我们的实验表明，我们的方法与人类对信息量的判断的 Spearman 相关性达到 0.4983，而最强的基线仅获得 0.3534 的相关性。额外的分析表明，基于 LLM 的方法能够推广到测量成人导向文本中的上下文信息量，在这方面它也优于所有基线。

Title: Diving into Self-Evolving Training for Multimodal Reasoning

• Authors: Wei Liu, Junlong Li, Xiwen Zhang, Fan Zhou, Yu Cheng, Junxian He
• Subjects: cs.CL, cs.AI, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.17451
• Pdf URL: https://arxiv.org/pdf/2412.17451
• Copy Paste: [[2412.17451]] Diving into Self-Evolving Training for Multimodal Reasoning(https://arxiv.org/abs/2412.17451)
• Keywords: prompt, chain-of-thought
• Abstract: Reasoning ability is essential for Large Multimodal Models (LMMs). In the absence of multimodal chain-of-thought annotated data, self-evolving training, where the model learns from its own outputs, has emerged as an effective and scalable approach for enhancing reasoning abilities. Despite its growing usage, a comprehensive understanding of self-evolving training, particularly in the context of multimodal reasoning, remains limited. In this paper, we delve into the intricacies of self-evolving training for multimodal reasoning, pinpointing three key factors: Training Method, Reward Model, and Prompt Variation. We systematically examine each factor and explore how various configurations affect the training's effectiveness. Our analysis leads to a set of best practices for each factor, aimed at optimizing multimodal reasoning. Furthermore, we explore the Self-Evolution Dynamics during training and the impact of automatic balancing mechanisms in boosting performance. After all the investigations, we present a final recipe for self-evolving training in multimodal reasoning, encapsulating these design choices into a framework we call MSTaR (Multimodal Self-evolving Training for Reasoning), which is universally effective for models with different sizes on various benchmarks, e.g., surpassing the pre-evolved model significantly on 5 multimodal reasoning benchmarks without using additional human annotations, as demonstrated on MiniCPM-V-2.5 (8B), Phi-3.5-Vision (4B) and InternVL2 (2B). We believe this study fills a significant gap in the understanding of self-evolving training for multimodal reasoning and offers a robust framework for future research. Our policy and reward models, as well as the collected data, is released to facilitate further investigation in multimodal reasoning.
• 摘要：推理能力对于大型多模态模型 (LMM) 至关重要。在缺乏多模态思路链注释数据的情况下，自进化训练（模型从自身输出中学习）已成为一种有效且可扩展的增强推理能力的方法。尽管自进化训练的使用越来越广泛，但对自进化训练的全面了解，特别是在多模态推理的背景下，仍然有限。在本文中，我们深入研究了多模态推理自进化训练的复杂性，确定了三个关键因素：训练方法、奖励模型和提示变化。我们系统地检查了每个因素，并探索了各种配置如何影响训练的有效性。我们的分析为每个因素提供了一套最佳实践，旨在优化多模态推理。此外，我们还探讨了训练过程中的自进化动态以及自动平衡机制对提高性能的影响。经过所有调查，我们提出了多模态推理自我进化训练的最终方案，将这些设计选择封装到一个我们称之为 MSTaR（多模态自我进化推理训练）的框架中，该框架对不同大小的模型在各种基准上都具有普遍有效性，例如，在 5 个多模态推理基准上显著超越了进化前的模型，而无需使用额外的人工注释，如 MiniCPM-V-2.5（8B）、Phi-3.5-Vision（4B）和 InternVL2（2B）上所展示的。我们相信这项研究填补了对多模态推理自我进化训练理解的重大空白，并为未来的研究提供了一个强大的框架。我们发布了政策和奖励模型以及收集的数据，以促进对多模态推理的进一步研究。

Title: A Survey on Multi-Generative Agent System: Recent Advances and New Frontiers

• Authors: Shuaihang Chen, Yuanxing Liu, Wei Han, Weinan Zhang, Ting Liu
• Subjects: cs.CL, cs.MA
• Abstract URL: https://arxiv.org/abs/2412.17481
• Pdf URL: https://arxiv.org/pdf/2412.17481
• Copy Paste: [[2412.17481]] A Survey on Multi-Generative Agent System: Recent Advances and New Frontiers(https://arxiv.org/abs/2412.17481)
• Keywords: language model, llm, agent
• Abstract: Multi-generative agent systems (MGASs) have become a research hotspot since the rise of large language models (LLMs). However, with the continuous influx of new related works, the existing reviews struggle to capture them comprehensively. This paper presents a comprehensive survey of these studies. We first discuss the definition of MGAS, a framework encompassing much of previous work. We provide an overview of the various applications of MGAS in (i) solving complex tasks, (ii) simulating specific scenarios, and (iii) evaluating generative agents. Building on previous studies, we also highlight several challenges and propose future directions for research in this field.
• 摘要：自大型语言模型 (LLM) 兴起以来，多生成代理系统 (MGAS) 已成为研究热点。然而，随着新的相关研究不断涌入，现有的评论难以全面捕捉它们。本文对这些研究进行了全面的调查。我们首先讨论 MGAS 的定义，这是一个涵盖大量先前工作的框架。我们概述了 MGAS 在 (i) 解决复杂任务、(ii) 模拟特定场景和 (iii) 评估生成代理方面的各种应用。在先前研究的基础上，我们还强调了几个挑战并提出了该领域未来的研究方向。

Title: A Silver Bullet or a Compromise for Full Attention? A Comprehensive Study of Gist Token-based Context Compression

• Authors: Chenlong Deng, Zhisong Zhang, Kelong Mao, Shuaiyi Li, Xinting Huang, Dong Yu, Zhicheng Dou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.17483
• Pdf URL: https://arxiv.org/pdf/2412.17483
• Copy Paste: [[2412.17483]] A Silver Bullet or a Compromise for Full Attention? A Comprehensive Study of Gist Token-based Context Compression(https://arxiv.org/abs/2412.17483)
• Keywords: language model, retrieval-augmented generation
• Abstract: In this work, we provide a thorough investigation of gist-based context compression methods to improve long-context processing in large language models. We focus on two key questions: (1) How well can these methods replace full attention models? and (2) What potential failure patterns arise due to compression? Through extensive experiments, we show that while gist-based compression can achieve near-lossless performance on tasks like retrieval-augmented generation and long-document QA, it faces challenges in tasks like synthetic recall. Furthermore, we identify three key failure patterns: lost by the boundary, lost if surprise, and lost along the way. To mitigate these issues, we propose two effective strategies: fine-grained autoencoding, which enhances the reconstruction of original token information, and segment-wise token importance estimation, which adjusts optimization based on token dependencies. Our work provides valuable insights into the understanding of gist token-based context compression and offers practical strategies for improving compression capabilities.
• 摘要：在本文中，我们深入研究了基于要点的上下文压缩方法，以改进大型语言模型中的长上下文处理。我们重点关注两个关键问题：（1）这些方法能多好地替代全注意力模型？（2）压缩会导致哪些潜在的故障模式？通过大量实验，我们表明，虽然基于要点的压缩可以在检索增强生成和长文档问答等任务上实现近乎无损的性能，但它在合成召回等任务中面临挑战。此外，我们确定了三种关键故障模式：边界丢失、意外丢失和沿途丢失。为了缓解这些问题，我们提出了两种有效策略：细粒度自动编码（可增强原始标记信息的重建）和分段标记重要性估计（可根据标记依赖关系调整优化）。我们的工作为理解基于要点标记的上下文压缩提供了宝贵的见解，并提供了提高压缩能力的实用策略。

Title: DRT-o1: Optimized Deep Reasoning Translation via Long Chain-of-Thought

• Authors: Jiaan Wang, Fandong Meng, Yunlong Liang, Jie Zhou
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.17498
• Pdf URL: https://arxiv.org/pdf/2412.17498
• Copy Paste: [[2412.17498]] DRT-o1: Optimized Deep Reasoning Translation via Long Chain-of-Thought(https://arxiv.org/abs/2412.17498)
• Keywords: llm, chain-of-thought, agent
• Abstract: Recently, O1-like models have emerged as representative examples, illustrating the effectiveness of long chain-of-thought (CoT) in reasoning tasks such as math and coding tasks. In this paper, we introduce DRT-o1, an attempt to bring the success of long CoT to neural machine translation (MT). Specifically, in view of the literature books that might involve similes and metaphors, translating these texts to a target language is very difficult in practice due to cultural differences. In such cases, literal translation often fails to convey the intended meaning effectively. Even for professional human translators, considerable thought must be given to preserving semantics throughout the translation process. To simulate LLMs' long thought ability in MT, we first mine sentences containing similes or metaphors from existing literature books, and then develop a multi-agent framework to translate these sentences via long thought. In the multi-agent framework, a translator is used to iteratively translate the source sentence under the suggestions provided by an advisor. To ensure the effectiveness of the long thoughts, an evaluator is also employed to judge whether the translation in the current round is better than the previous one or not. In this manner, we collect tens of thousands of long-thought MT data, which is used to train our DRT-o1. The experimental results on literature translation demonstrate the effectiveness of the DRT-o1. Using Qwen2.5-7B and Qwen2.5-14B as the backbones, the improvement brought by DRT-o1 achieves 7.33~8.26 BLEU and 1.66~3.36 CometScore. Besides, DRT-o1-7B can outperform QwQ-32B-Preview by 7.82 BLEU and 1.46 CometScore, showing its effectiveness. The project is available at this https URL
• 摘要：最近，O1 类模型作为代表性例子出现，说明了长思维链 (CoT) 在数学和编码任务等推理任务中的有效性。在本文中，我们介绍了 DRT-o1，试图将长思维链的成功引入神经机器翻译 (MT)。具体来说，鉴于文学书籍可能涉及明喻和隐喻，由于文化差异，将这些文本翻译成目标语言在实践中非常困难。在这种情况下，直译往往无法有效地传达预期的含义。即使是专业的人工翻译，也必须在整个翻译过程中仔细考虑如何保留语义。为了在机器翻译中模拟 LLM 的长思考能力，我们首先从现有文学书籍中挖掘包含明喻或隐喻的句子，然后开发一个多智能体框架，通过长思考来翻译这些句子。在多智能体框架中，翻译器用于在顾问提供的建议下迭代翻译源句子。为了确保长篇翻译的有效性，我们还聘请了一位评估员来判断本轮翻译是否优于上一轮。通过这种方式，我们收集了数以万计的长篇翻译数据，用于训练我们的 DRT-o1。文学翻译实验结果证明了 DRT-o1 的有效性。使用 Qwen2.5-7B 和 Qwen2.5-14B 作为主干，DRT-o1 带来的改进实现了 7.33~8.26 BLEU 和 1.66~3.36 CometScore。此外，DRT-o1-7B 的表现比 QwQ-32B-Preview 高出 7.82 BLEU 和 1.46 CometScore，证明了其有效性。该项目可从此 https URL 获取

Title: DiffusionAttacker: Diffusion-Driven Prompt Manipulation for LLM Jailbreak

• Authors: Hao Wang, Hao Li, Junda Zhu, Xinyuan Wang, Chengwei Pan, MinLie Huang, Lei Sha
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.17522
• Pdf URL: https://arxiv.org/pdf/2412.17522
• Copy Paste: [[2412.17522]] DiffusionAttacker: Diffusion-Driven Prompt Manipulation for LLM Jailbreak(https://arxiv.org/abs/2412.17522)
• Keywords: language model, llm, prompt
• Abstract: Large Language Models (LLMs) are susceptible to generating harmful content when prompted with carefully crafted inputs, a vulnerability known as LLM jailbreaking. As LLMs become more powerful, studying jailbreak methods is critical to enhancing security and aligning models with human values. Traditionally, jailbreak techniques have relied on suffix addition or prompt templates, but these methods suffer from limited attack diversity. This paper introduces DiffusionAttacker, an end-to-end generative approach for jailbreak rewriting inspired by diffusion models. Our method employs a sequence-to-sequence (seq2seq) text diffusion model as a generator, conditioning on the original prompt and guiding the denoising process with a novel attack loss. Unlike previous approaches that use autoregressive LLMs to generate jailbreak prompts, which limit the modification of already generated tokens and restrict the rewriting space, DiffusionAttacker utilizes a seq2seq diffusion model, allowing more flexible token modifications. This approach preserves the semantic content of the original prompt while producing harmful content. Additionally, we leverage the Gumbel-Softmax technique to make the sampling process from the diffusion model's output distribution differentiable, eliminating the need for iterative token search. Extensive experiments on Advbench and Harmbench demonstrate that DiffusionAttacker outperforms previous methods across various evaluation metrics, including attack success rate (ASR), fluency, and diversity.
• 摘要：大型语言模型 (LLM) 在收到精心设计的输入提示时容易生成有害内容，这种漏洞称为 LLM 越狱。随着 LLM 变得越来越强大，研究越狱方法对于增强安全性和使模型与人类价值观保持一致至关重要。传统上，越狱技术依赖于后缀添加或提示模板，但这些方法的攻击多样性有限。本文介绍了 DiffusionAttacker，这是一种受扩散模型启发的端到端越狱重写生成方法。我们的方法采用序列到序列 (seq2seq) 文本扩散模型作为生成器，以原始提示为条件，并以新颖的攻击损失引导去噪过程。与以前使用自回归 LLM 生成越狱提示的方法不同，这些方法限制了对已生成的令牌的修改并限制了重写空间，而 DiffusionAttacker 则使用 seq2seq 扩散模型，允许更灵活的令牌修改。这种方法在生成有害内容的同时保留了原始提示的语义内容。此外，我们利用 Gumbel-Softmax 技术使扩散模型输出分布的采样过程可微分，从而无需进行迭代标记搜索。在 Advbench 和 Harmbench 上进行的大量实验表明，DiffusionAttacker 在各种评估指标（包括攻击成功率 (ASR)、流畅度和多样性）方面均优于以前的方法。

Title: Behind Closed Words: Creating and Investigating the forePLay Annotated Dataset for Polish Erotic Discourse

• Authors: Anna Kołos, Katarzyna Lorenc, Emilia Wiśnios, Agnieszka Karlińska
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.17533
• Pdf URL: https://arxiv.org/pdf/2412.17533
• Copy Paste: [[2412.17533]] Behind Closed Words: Creating and Investigating the forePLay Annotated Dataset for Polish Erotic Discourse(https://arxiv.org/abs/2412.17533)
• Keywords: language model
• Abstract: The surge in online content has created an urgent demand for robust detection systems, especially in non-English contexts where current tools demonstrate significant limitations. We present forePLay, a novel Polish language dataset for erotic content detection, featuring over 24k annotated sentences with a multidimensional taxonomy encompassing ambiguity, violence, and social unacceptability dimensions. Our comprehensive evaluation demonstrates that specialized Polish language models achieve superior performance compared to multilingual alternatives, with transformer-based architectures showing particular strength in handling imbalanced categories. The dataset and accompanying analysis establish essential frameworks for developing linguistically-aware content moderation systems, while highlighting critical considerations for extending such capabilities to morphologically complex languages.
• 摘要：在线内容的激增迫切需要强大的检测系统，尤其是在当前工具存在很大局限性的非英语环境中。我们提出了 forePLay，这是一个用于色情内容检测的新型波兰语数据集，包含超过 24,000 个带注释的句子，具有涵盖歧义、暴力和社会不可接受性维度的多维分类法。我们的全面评估表明，与多语言替代方案相比，专门的波兰语模型实现了卓越的性能，而基于 Transformer 的架构在处理不平衡类别方面表现出色。该数据集和随附的分析为开发语言感知内容审核系统建立了基本框架，同时强调了将此类功能扩展到形态复杂的语言的关键考虑因素。

Title: Resource-Aware Arabic LLM Creation: Model Adaptation, Integration, and Multi-Domain Testing

• Authors: Prakash Aryan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.17548
• Pdf URL: https://arxiv.org/pdf/2412.17548
• Copy Paste: [[2412.17548]] Resource-Aware Arabic LLM Creation: Model Adaptation, Integration, and Multi-Domain Testing(https://arxiv.org/abs/2412.17548)
• Keywords: language model, llm
• Abstract: This paper presents a novel approach to fine-tuning the Qwen2-1.5B model for Arabic language processing using Quantized Low-Rank Adaptation (QLoRA) on a system with only 4GB VRAM. We detail the process of adapting this large language model to the Arabic domain, using diverse datasets including Bactrian, OpenAssistant, and Wikipedia Arabic corpora. Our methodology involves custom data preprocessing, model configuration, and training optimization techniques such as gradient accumulation and mixed-precision training. We address specific challenges in Arabic NLP, including morphological complexity, dialectal variations, and diacritical mark handling. Experimental results over 10,000 training steps show significant performance improvements, with the final loss converging to 0.1083. We provide comprehensive analysis of GPU memory usage, training dynamics, and model evaluation across various Arabic language tasks, including text classification, question answering, and dialect identification. The fine-tuned model demonstrates robustness to input perturbations and improved handling of Arabic-specific linguistic phenomena. This research contributes to multilingual AI by demonstrating a resource-efficient approach for creating specialized language models, potentially democratizing access to advanced NLP technologies for diverse linguistic communities. Our work paves the way for future research in low-resource language adaptation and efficient fine-tuning of large language models.
• 摘要：本文介绍了一种新方法，使用量化低秩自适应 (QLoRA) 在仅具有 4GB VRAM 的系统上对用于阿拉伯语处理的 Qwen2-1.5B 模型进行微调。我们详细介绍了使用包括 Bactrian、OpenAssistant 和 Wikipedia 阿拉伯语语料库在内的各种数据集将这种大型语言模型适应阿拉伯语领域的过程。我们的方法涉及自定义数据预处理、模型配置和训练优化技术，例如梯度累积和混合精度训练。我们解决了阿拉伯语 NLP 中的特定挑战，包括形态复杂性、方言变化和变音符号处理。超过 10,000 个训练步骤的实验结果显示性能显着提升，最终损失收敛到 0.1083。我们对各种阿拉伯语任务（包括文本分类、问答和方言识别）的 GPU 内存使用情况、训练动态和模型评估进行了全面分析。微调后的模型表现出对输入扰动的鲁棒性，并改进了对阿拉伯语特定语言现象的处理。这项研究展示了一种创建专用语言模型的资源高效方法，有望使不同语言群体能够更自由地使用先进的 NLP 技术，从而为多语言 AI ​​做出贡献。我们的工作为未来低资源语言适应和大型语言模型高效微调的研究铺平了道路。

Title: A Survey of Query Optimization in Large Language Models

• Authors: Mingyang Song, Mao Zheng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.17558
• Pdf URL: https://arxiv.org/pdf/2412.17558
• Copy Paste: [[2412.17558]] A Survey of Query Optimization in Large Language Models(https://arxiv.org/abs/2412.17558)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: \textit{Query Optimization} (QO) refers to techniques aimed at enhancing the efficiency and quality of Large Language Models (LLMs) in understanding and answering queries, especially complex ones in scenarios like Retrieval-Augmented Generation (RAG). Specifically, RAG mitigates the limitations of LLMs by dynamically retrieving and leveraging up-to-date relevant information, which provides a cost-effective solution to the challenge of LLMs producing plausible but potentially inaccurate responses. Recently, as RAG evolves and incorporates multiple components that influence its performance, QO has emerged as a critical element, playing a pivotal role in determining the effectiveness of RAG's retrieval stage in accurately sourcing the necessary multiple pieces of evidence to answer queries correctly. In this paper, we trace the evolution of QO techniques by summarizing and analyzing significant studies. Through an organized framework and categorization, we aim to consolidate existing QO techniques in RAG, elucidate their technological foundations, and highlight their potential to enhance the versatility and applications of LLMs.
• 摘要：\textit{查询优化} (QO) 是指旨在提高大型语言模型 (LLM) 在理解和回答查询方面的效率和质量的技术，尤其是检索增强生成 (RAG) 等场景中的复杂查询。具体而言，RAG 通过动态检索和利用最新的相关信息来减轻 LLM 的局限性，为 LLM 产生合理但可能不准确的响应的挑战提供了一种经济有效的解决方案。最近，随着 RAG 的发展和整合影响其性能的多个组件，QO 已成为一个关键元素，在确定 RAG 检索阶段的有效性方面发挥着关键作用，从而准确地获取正确回答查询所需的多个证据。在本文中，我们通过总结和分析重要研究来追踪 QO 技术的演变。通过有组织的框架和分类，我们旨在巩固 RAG 中现有的 QO 技术，阐明其技术基础，并强调它们在增强 LLM 的多功能性和应用方面的潜力。

Title: ERUPD -- English to Roman Urdu Parallel Dataset

• Authors: Mohammed Furqan, Raahid Bin Khaja, Rayyan Habeeb
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.17562
• Pdf URL: https://arxiv.org/pdf/2412.17562
• Copy Paste: [[2412.17562]] ERUPD -- English to Roman Urdu Parallel Dataset(https://arxiv.org/abs/2412.17562)
• Keywords: prompt
• Abstract: Bridging linguistic gaps fosters global growth and cultural exchange. This study addresses the challenges of Roman Urdu -- a Latin-script adaptation of Urdu widely used in digital communication -- by creating a novel parallel dataset comprising 75,146 sentence pairs. Roman Urdu's lack of standardization, phonetic variability, and code-switching with English complicates language processing. We tackled this by employing a hybrid approach that combines synthetic data generated via advanced prompt engineering with real-world conversational data from personal messaging groups. We further refined the dataset through a human evaluation phase, addressing linguistic inconsistencies and ensuring accuracy in code-switching, phonetic representations, and synonym variability. The resulting dataset captures Roman Urdu's diverse linguistic features and serves as a critical resource for machine translation, sentiment analysis, and multilingual education.
• 摘要：弥合语言差距有助于促进全球发展和文化交流。本研究通过创建一个包含 75,146 个句子对的新型平行数据集，解决了罗马乌尔都语（一种广泛用于数字通信的乌尔都语的拉丁字母改编版）的挑战。罗马乌尔都语缺乏标准化、语音多变且与英语的代码转换使语言处理变得复杂。我们通过采用一种混合方法解决了这个问题，该方法将通过高级提示工程生成的合成数据与来自个人消息组的真实对话数据相结合。我们通过人工评估阶段进一步完善了数据集，解决了语言不一致问题并确保了代码转换、语音表示和同义词多变性的准确性。生成的数据集捕捉了罗马乌尔都语的多种语言特征，是机器翻译、情感分析和多语言教育的重要资源。

Title: LiveIdeaBench: Evaluating LLMs' Scientific Creativity and Idea Generation with Minimal Context

• Authors: Kai Ruan, Xuan Wang, Jixiang Hong, Hao Sun
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.17596
• Pdf URL: https://arxiv.org/pdf/2412.17596
• Copy Paste: [[2412.17596]] LiveIdeaBench: Evaluating LLMs' Scientific Creativity and Idea Generation with Minimal Context(https://arxiv.org/abs/2412.17596)
• Keywords: language model, llm, prompt
• Abstract: While Large Language Models (LLMs) have demonstrated remarkable capabilities in scientific tasks, existing evaluation frameworks primarily assess their performance using rich contextual inputs, overlooking their ability to generate novel ideas from minimal information. We introduce LiveIdeaBench, a comprehensive benchmark that evaluates LLMs' scientific creativity and divergent thinking capabilities using single-keyword prompts. Drawing from Guilford's creativity theory, our framework employs a dynamic panel of state-of-the-art LLMs to assess generated ideas across four key dimensions: originality, feasibility, fluency, and flexibility. Through extensive experimentation with 20 leading models across 1,180 keywords spanning 18 scientific domains, we reveal that scientific creative ability shows distinct patterns from general intelligence metrics. Notably, our results demonstrate that models like QwQ-32B-preview achieve comparable creative performance to top-tier models like o1-preview, despite significant gaps in their general intelligence scores. These findings highlight the importance of specialized evaluation frameworks for scientific creativity and suggest that the development of creative capabilities in LLMs may follow different trajectories than traditional problem-solving abilities.
• 摘要：虽然大型语言模型 (LLM) 在科学任务中表现出了卓越的能力，但现有的评估框架主要使用丰富的上下文输入来评估它们的性能，而忽略了它们从最少的信息中产生新颖想法的能力。我们引入了 LiveIdeaBench，这是一个全面的基准，它使用单个关键字提示来评估 LLM 的科学创造力和发散思维能力。借鉴吉尔福德的创造力理论，我们的框架采用了一组最先进的动态 LLM 来评估四个关键维度上产生的想法：原创性、可行性、流畅性和灵活性。通过对 18 个科学领域的 1,180 个关键字中的 20 个领先模型进行大量实验，我们发现科学创造能力与一般智力指标显示出不同的模式。值得注意的是，我们的结果表明，尽管 QwQ-32B-preview 等模型的一般智力得分存在显著差距，但它们的创造性表现与 o1-preview 等顶级模型相当。这些发现强调了专门的评估框架对科学创造力的重要性，并表明法学硕士创造能力的发展可能遵循与传统解决问题能力不同的轨迹。

Title: Generating Completions for Fragmented Broca's Aphasic Sentences Using Large Language Models

• Authors: Sijbren van Vaals, Yevgen Matusevych, Frank Tsiwah
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.17669
• Pdf URL: https://arxiv.org/pdf/2412.17669
• Copy Paste: [[2412.17669]] Generating Completions for Fragmented Broca's Aphasic Sentences Using Large Language Models(https://arxiv.org/abs/2412.17669)
• Keywords: language model, llm
• Abstract: Broca's aphasia is a type of aphasia characterized by non-fluent, effortful and fragmented speech production with relatively good comprehension. Since traditional aphasia treatment methods are often time-consuming, labour-intensive, and do not reflect real-world conversations, applying natural language processing based approaches such as Large Language Models (LLMs) could potentially contribute to improving existing treatment approaches. To address this issue, we explore the use of sequence-to-sequence LLMs for completing fragmented Broca's aphasic sentences. We first generate synthetic Broca's aphasic data using a rule-based system designed to mirror the linguistic characteristics of Broca's aphasic speech. Using this synthetic data, we then fine-tune four pre-trained LLMs on the task of completing fragmented sentences. We evaluate our fine-tuned models on both synthetic and authentic Broca's aphasic data. We demonstrate LLMs' capability for reconstructing fragmented sentences, with the models showing improved performance with longer input utterances. Our result highlights the LLMs' potential in advancing communication aids for individuals with Broca's aphasia and possibly other clinical populations.
• 摘要：布罗卡失语症是一种失语症，其特征是说话不流畅、费力且断断续续，但理解能力相对较好。由于传统的失语症治疗方法通常耗时费力，而且不能反映现实世界的对话，因此应用基于自然语言处理的方法（如大型语言模型 (LLM)）可能有助于改善现有的治疗方法。为了解决这个问题，我们探索了使用序列到序列 LLM 来完成断断续续的布罗卡失语症句子。我们首先使用基于规则的系统生成合成的布罗卡失语症数据，该系统旨在反映布罗卡失语症语音的语言特征。然后，我们使用这些合成数据，对四个预先训练的 LLM 进行微调，以完成断断续续的句子。我们在合成和真实的布罗卡失语症数据上评估了我们的微调模型。我们展示了 LLM 重建断断续续句子的能力，模型在输入较长的话语时表现出更好的性能。我们的研究结果凸显了法学硕士学位在为布罗卡失语症患者以及其他临床人群提供沟通辅助方面的潜力。

Title: RAGONITE: Iterative Retrieval on Induced Databases and Verbalized RDF for Conversational QA over KGs with RAG

• Authors: Rishiraj Saha Roy, Chris Hinze, Joel Schlotthauer, Farzad Naderi, Viktor Hangya, Andreas Foltyn, Luzian Hahn, Fabian Kuech
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2412.17690
• Pdf URL: https://arxiv.org/pdf/2412.17690
• Copy Paste: [[2412.17690]] RAGONITE: Iterative Retrieval on Induced Databases and Verbalized RDF for Conversational QA over KGs with RAG(https://arxiv.org/abs/2412.17690)
• Keywords: llm, retrieval augmented generation
• Abstract: Conversational question answering (ConvQA) is a convenient means of searching over RDF knowledge graphs (KGs), where a prevalent approach is to translate natural language questions to SPARQL queries. However, SPARQL has certain shortcomings: (i) it is brittle for complex intents and conversational questions, and (ii) it is not suitable for more abstract needs. Instead, we propose a novel two-pronged system where we fuse: (i) SQL-query results over a database automatically derived from the KG, and (ii) text-search results over verbalizations of KG facts. Our pipeline supports iterative retrieval: when the results of any branch are found to be unsatisfactory, the system can automatically opt for further rounds. We put everything together in a retrieval augmented generation (RAG) setup, where an LLM generates a coherent response from accumulated search results. We demonstrate the superiority of our proposed system over several baselines on a knowledge graph of BMW automobiles.
• 摘要：对话式问答 (ConvQA) 是一种便捷的 RDF 知识图谱 (KG) 搜索方法，其中一种流行的方法是将自然语言问题转换为 SPARQL 查询。但是，SPARQL 存在某些缺点：(i) 它对于复杂的意图和对话式问题来说很脆弱，并且 (ii) 它不适合更抽象的需求。相反，我们提出了一种新颖的双管齐下系统，其中我们融合了：(i) 从 KG 自动派生的数据库中的 SQL 查询结果，以及 (ii) KG 事实的言语化文本搜索结果。我们的管道支持迭代检索：当发现任何分支的结果不令人满意时，系统可以自动选择进一步的轮次。我们将所有内容放在检索增强生成 (RAG) 设置中，其中 LLM 从累积的搜索结果中生成连贯的响应。我们在 BMW 汽车知识图谱上展示了我们提出的系统相对于多个基线的优越性。

Title: Understanding the Logic of Direct Preference Alignment through Logic

• Authors: Kyle Richardson, Vivek Srikumar, Ashish Sabharwal
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.17696
• Pdf URL: https://arxiv.org/pdf/2412.17696
• Copy Paste: [[2412.17696]] Understanding the Logic of Direct Preference Alignment through Logic(https://arxiv.org/abs/2412.17696)
• Keywords: language model
• Abstract: Recent direct preference alignment algorithms (DPA), such as DPO, have shown great promise in aligning large language models to human preferences. While this has motivated the development of many new variants of the original DPO loss, understanding the differences between these recent proposals, as well as developing new DPA loss functions, remains difficult given the lack of a technical and conceptual framework for reasoning about the underlying semantics of these algorithms. In this paper, we attempt to remedy this by formalizing DPA losses in terms of discrete reasoning problems. Specifically, we ask: Given an existing DPA loss, can we systematically derive a symbolic expression that characterizes its semantics? How do the semantics of two losses relate to each other? We propose a novel formalism for characterizing preference losses for single model and reference model based approaches, and identify symbolic forms for a number of commonly used DPA variants. Further, we show how this formal view of preference learning sheds new light on both the size and structure of the DPA loss landscape, making it possible to not only rigorously characterize the relationships between recent loss proposals but also to systematically explore the landscape and derive new loss functions from first principles. We hope our framework and findings will help provide useful guidance to those working on human AI alignment.
• 摘要：最近的直接偏好对齐算法 (DPA)，例如 DPO，在将大型语言模型与人类偏好对齐方面表现出巨大的潜力。虽然这促使开发了许多原始 DPO 损失的新变体，但由于缺乏用于推理这些算法底层语义的技术和概念框架，理解这些最新提案之间的差异以及开发新的 DPA 损失函数仍然很困难。在本文中，我们尝试通过用离散推理问题的形式化 DPA 损失来解决这个问题。具体来说，我们问：给定一个现有的 DPA 损失，我们能否系统地得出一个表征其语义的符号表达式？两个损失的语义如何相互关联？我们提出了一种新的形式化方法来表征基于单一模型和参考模型的方法的偏好损失，并为许多常用的 DPA 变体确定了符号形式。此外，我们展示了这种偏好学习的正式观点如何为 DPA 损失格局的规模和结构提供新的见解，不仅能够严格描述近期损失提案之间的关系，而且能够系统地探索格局并从第一原理中得出新的损失函数。我们希望我们的框架和发现能够为那些致力于人类 AI 对齐的人们提供有用的指导。

Title: From Models to Microtheories: Distilling a Model's Topical Knowledge for Grounded Question Answering

• Authors: Nathaniel Weir, Bhavana Dalvi Mishra, Orion Weller, Oyvind Tafjord, Sam Hornstein, Alexander Sabol, Peter Jansen, Benjamin Van Durme, Peter Clark
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.17701
• Pdf URL: https://arxiv.org/pdf/2412.17701
• Copy Paste: [[2412.17701]] From Models to Microtheories: Distilling a Model's Topical Knowledge for Grounded Question Answering(https://arxiv.org/abs/2412.17701)
• Keywords: language model, chain-of-thought
• Abstract: Recent reasoning methods (e.g., chain-of-thought, entailment reasoning) help users understand how language models (LMs) answer a single question, but they do little to reveal the LM's overall understanding, or "theory," about the question's $\textit{topic}$, making it still hard to trust the model. Our goal is to materialize such theories - here called $\textit{microtheories}$ (a linguistic analog of logical microtheories) - as a set of sentences encapsulating an LM's core knowledge about a topic. These statements systematically work together to entail answers to a $\textit{set}$ of questions to both engender trust and improve performance. Our approach is to first populate a knowledge store with (model-generated) sentences that entail answers to training questions and then distill those down to a core microtheory that is concise, general, and non-redundant. We show that, when added to a general corpus (e.g., Wikipedia), microtheories can supply critical, topical information not necessarily present in the corpus, improving both a model's ability to ground its answers to verifiable knowledge (i.e., show how answers are systematically entailed by documents in the corpus, fully grounding up to +8% more answers), and the accuracy of those grounded answers (up to +8% absolute). We also show that, in a human evaluation in the medical domain, our distilled microtheories contain a significantly higher concentration of topically critical facts than the non-distilled knowledge store. Finally, we show we can quantify the coverage of a microtheory for a topic (characterized by a dataset) using a notion of $p$-relevance. Together, these suggest that microtheories are an efficient distillation of an LM's topic-relevant knowledge, that they can usefully augment existing corpora, and can provide both performance gains and an interpretable, verifiable window into the model's knowledge of a topic.
• 摘要：最近的推理方法（例如，思路链、蕴涵推理）可帮助用户理解语言模型 (LM) 如何回答单个问题，但它们几乎无法揭示 LM 对问题主题的整体理解或“理论”，因此仍然很难信任该模型。我们的目标是将这些理论（这里称为微理论（逻辑微理论的语言学类似物））具体化为一组句子，这些句子封装了 LM 对某个主题的核心知识。这些语句系统地协同工作，以包含一组问题的答案，从而建立信任并提高性能。我们的方法是首先用（模型生成的）句子填充知识库，这些句子包含训练问题的答案，然后将它们提炼为简洁、通用且无冗余的核心微理论。我们表明，当微理论被添加到通用语料库（例如 Wikipedia）中时，它们可以提供语料库中不一定存在的关键主题信息，从而提高模型将其答案与可验证知识联系起来的能力（即，展示答案如何系统地由语料库中的文档所包含，完全建立最多 +8% 的答案基础），以及这些建立基础的答案的准确性（绝对值最多 +8%）。我们还表明，在医学领域的人工评估中，我们提炼的微理论包含的主题关键事实的浓度明显高于非提炼的知识库。最后，我们表明我们可以使用 $p$ 相关性概念来量化微理论对主题（由数据集表征）的覆盖范围。总的来说，这些表明微理论是 LM 主题相关知识的有效提炼，它们可以有效地增强现有语料库，既可以提高性能，又可以为模型关于某个主题的知识提供可解释、可验证的窗口。

Title: Knowledge Editing through Chain-of-Thought

• Authors: Changyue Wang, Weihang Su, Qingyao Ai, Yiqun Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.17727
• Pdf URL: https://arxiv.org/pdf/2412.17727
• Copy Paste: [[2412.17727]] Knowledge Editing through Chain-of-Thought(https://arxiv.org/abs/2412.17727)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: Large Language Models (LLMs) have demonstrated exceptional capabilities across a wide range of natural language processing (NLP) tasks. However, keeping these models up-to-date with evolving world knowledge remains a significant challenge due to the high costs of frequent retraining. To address this challenge, knowledge editing techniques have emerged to update LLMs with new information without rebuilding the model from scratch. Among these, the in-context editing paradigm stands out for its effectiveness in integrating new knowledge while preserving the model's original capabilities. Despite its potential, existing in-context knowledge editing methods are often task-specific, focusing primarily on multi-hop QA tasks using structured knowledge triples. Moreover, their reliance on few-shot prompting for task decomposition makes them unstable and less effective in generalizing across diverse tasks. In response to these limitations, we propose EditCoT, a novel knowledge editing framework that flexibly and efficiently updates LLMs across various tasks without retraining. EditCoT works by generating a chain-of-thought (CoT) for a given input and then iteratively refining this CoT process using a CoT editor based on updated knowledge. We evaluate EditCoT across a diverse range of benchmarks, covering multiple languages and tasks. The results demonstrate that our approach achieves state-of-the-art performance while offering superior generalization, effectiveness, and stability compared to existing methods, marking a significant advancement in the field of knowledge updating. Code and data are available at: this https URL.
• 摘要：大型语言模型 (LLM) 在广泛的自然语言处理 (NLP) 任务中表现出卓越的能力。然而，由于频繁重新训练的成本很高，让这些模型与不断发展的世界知识保持同步仍然是一项重大挑战。为了应对这一挑战，出现了知识编辑技术，可以用新信息更新 LLM，而无需从头开始重建模型。其中，上下文编辑范式因其在整合新知识的同时保留模型原有能力的有效性而脱颖而出。尽管具有潜力，但现有的上下文知识编辑方法通常是针对特定任务的，主要侧重于使用结构化知识三元组的多跳 QA 任务。此外，它们依赖于少量提示进行任务分解，这使得它们不稳定，并且在跨不同任务的泛化方面效率较低。为了应对这些限制，我们提出了 EditCoT，这是一种新颖的知识编辑框架，可以灵活高效地跨各种任务更新 LLM，而无需重新训练。 EditCoT 的工作原理是针对给定的输入生成思路链 (CoT)，然后使用基于更新知识的 CoT 编辑器迭代改进此 CoT 过程。我们在多种基准上评估了 EditCoT，涵盖了多种语言和任务。结果表明，与现有方法相比，我们的方法实现了最先进的性能，同时提供了卓越的泛化、有效性和稳定性，标志着知识更新领域的重大进步。代码和数据可从以下网址获取：此 https URL。

Title: Chumor 2.0: Towards Benchmarking Chinese Humor Understanding

• Authors: Ruiqi He, Yushu He, Longju Bai, Jiarui Liu, Zhenjie Sun, Zenghao Tang, He Wang, Hanchen Xia, Rada Mihalcea, Naihao Deng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.17729
• Pdf URL: https://arxiv.org/pdf/2412.17729
• Copy Paste: [[2412.17729]] Chumor 2.0: Towards Benchmarking Chinese Humor Understanding(https://arxiv.org/abs/2412.17729)
• Keywords: gpt, llm, prompt, chain-of-thought
• Abstract: Existing humor datasets and evaluations predominantly focus on English, leaving limited resources for culturally nuanced humor in non-English languages like Chinese. To address this gap, we construct Chumor, the first Chinese humor explanation dataset that exceeds the size of existing humor datasets. Chumor is sourced from Ruo Zhi Ba, a Chinese Reddit-like platform known for sharing intellectually challenging and culturally specific jokes. We test ten LLMs through direct and chain-of-thought prompting, revealing that Chumor poses significant challenges to existing LLMs, with their accuracy slightly above random and far below human. In addition, our analysis highlights that human-annotated humor explanations are significantly better than those generated by GPT-4o and ERNIE-4-turbo. We release Chumor at this https URL, our project page is at this https URL, our leaderboard is at this https URL, and our codebase is at this https URL.
• 摘要：现有的幽默数据集和评估主要集中在英语上，而用于非英语语言（如中文）中文化差异性幽默的资源有限。为了弥补这一差距，我们构建了 Chumor，这是第一个超过现有幽默数据集大小的中文幽默解释数据集。Chumor 来源于若指吧，这是一个类似 Reddit 的中文平台，以分享具有智力挑战性和文化针对性的笑话而闻名。我们通过直接和思路链提示测试了十个 LLM，结果表明 Chumor 对现有 LLM 提出了重大挑战，其准确性略高于随机但远低于人类。此外，我们的分析强调，人工注释的幽默解释明显优于 GPT-4o 和 ERNIE-4-turbo 生成的解释。我们在此 https URL 发布 Chumor，我们的项目页面在此 https URL，我们的排行榜在此 https URL，我们的代码库在此 https URL。

Title: YuLan-Mini: An Open Data-efficient Language Model

• Authors: Yiwen Hu, Huatong Song, Jia Deng, Jiapeng Wang, Jie Chen, Kun Zhou, Yutao Zhu, Jinhao Jiang, Zican Dong, Wayne Xin Zhao, Ji-Rong Wen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.17743
• Pdf URL: https://arxiv.org/pdf/2412.17743
• Copy Paste: [[2412.17743]] YuLan-Mini: An Open Data-efficient Language Model(https://arxiv.org/abs/2412.17743)
• Keywords: language model, llm, long context
• Abstract: Effective pre-training of large language models (LLMs) has been challenging due to the immense resource demands and the complexity of the technical processes involved. This paper presents a detailed technical report on YuLan-Mini, a highly capable base model with 2.42B parameters that achieves top-tier performance among models of similar parameter scale. Our pre-training approach focuses on enhancing training efficacy through three key technical contributions: an elaborate data pipeline combines data cleaning with data schedule strategies, a robust optimization method to mitigate training instability, and an effective annealing approach that incorporates targeted data selection and long context training. Remarkably, YuLan-Mini, trained on 1.08T tokens, achieves performance comparable to industry-leading models that require significantly more data. To facilitate reproduction, we release the full details of the data composition for each training phase. Project details can be accessed at the following link: this https URL.
• 摘要：由于资源需求巨大且涉及的技术流程复杂，大型语言模型 (LLM) 的有效预训练一直具有挑战性。本文详细介绍了 YuLan-Mini 的技术报告。YuLan-Mini 是一个功能强大的基础模型，拥有 24.2 亿个参数，在类似参数规模的模型中实现了顶级性能。我们的预训练方法专注于通过三项关键技术贡献来提高训练效果：精心设计的数据管道将数据清理与数据调度策略相结合，采用稳健的优化方法以减轻训练不稳定性，采用有效的退火方法，结合有针对性的数据选择和长上下文训练。值得注意的是，YuLan-Mini 在 1.08T 令牌上进行训练，其性能可与需要更多数据的行业领先模型相媲美。为了便于复制，我们发布了每个训练阶段数据组成的完整详细信息。项目详细信息可通过以下链接访问：此 https URL。

Title: Deliberation in Latent Space via Differentiable Cache Augmentation

• Authors: Luyang Liu, Jonas Pfeiffer, Jiaxing Wu, Jun Xie, Arthur Szlam
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.17747
• Pdf URL: https://arxiv.org/pdf/2412.17747
• Copy Paste: [[2412.17747]] Deliberation in Latent Space via Differentiable Cache Augmentation(https://arxiv.org/abs/2412.17747)
• Keywords: language model, llm
• Abstract: Techniques enabling large language models (LLMs) to "think more" by generating and attending to intermediate reasoning steps have shown promise in solving complex problems. However, the standard approaches generate sequences of discrete tokens immediately before responding, and so they can incur significant latency costs and be challenging to optimize. In this work, we demonstrate that a frozen LLM can be augmented with an offline coprocessor that operates on the model's key-value (kv) cache. This coprocessor augments the cache with a set of latent embeddings designed to improve the fidelity of subsequent decoding. We train this coprocessor using the language modeling loss from the decoder on standard pretraining data, while keeping the decoder itself frozen. This approach enables the model to learn, in an end-to-end differentiable fashion, how to distill additional computation into its kv-cache. Because the decoder remains unchanged, the coprocessor can operate offline and asynchronously, and the language model can function normally if the coprocessor is unavailable or if a given cache is deemed not to require extra computation. We show experimentally that when a cache is augmented, the decoder achieves lower perplexity on numerous subsequent tokens. Furthermore, even without any task-specific training, our experiments demonstrate that cache augmentation consistently reduces perplexity and improves performance across a range of reasoning-intensive tasks.
• 摘要：通过生成和关注中间推理步骤，使大型语言模型 (LLM) 能够“思考更多”的技术在解决复杂问题方面已显示出良好的前景。然而，标准方法在响应之前立即生成离散标记序列，因此它们会产生显著的延迟成本并且难以优化。在这项工作中，我们证明了冻结的 LLM 可以通过在模型的键值 (kv) 缓存上运行的离线协处理器进行增强。该协处理器使用一组潜在嵌入来增强缓存，旨在提高后续解码的保真度。我们使用来自解码器的语言建模损失在标准预训练数据上训练此协处理器，同时保持解码器本身冻结。这种方法使模型能够以端到端可微分的方式学习如何将额外的计算提炼到其 kv 缓存中。由于解码器保持不变，协处理器可以离线和异步运行，并且如果协处理器不可用或给定的缓存被认为不需要额外的计算，语言模型可以正常运行。我们通过实验表明，当缓存增强时，解码器在众多后续标记上实现的困惑度会降低。此外，即使没有任何特定任务的训练，我们的实验也表明，缓存增强可以持续降低困惑度并提高一系列推理密集型任务的性能。

Title: In Case You Missed It: ARC 'Challenge' Is Not That Challenging

• Authors: Łukasz Borchmann
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.17758
• Pdf URL: https://arxiv.org/pdf/2412.17758
• Copy Paste: [[2412.17758]] In Case You Missed It: ARC 'Challenge' Is Not That Challenging(https://arxiv.org/abs/2412.17758)
• Keywords: llm
• Abstract: ARC Challenge appears more difficult than ARC Easy for modern LLMs primarily due to an evaluation setup that prevents direct comparison of answer choices rather than inherent complexity. Although some researchers have quietly shifted to a more appropriate scheme over the last year, the implications of this change have yet to be widely acknowledged. We highlight this overlooked shift, show how similar evaluation practices falsely imply reasoning deficits in other benchmarks, and demonstrate that fairer methods dramatically reduce performance gaps (e.g. on SIQA) and even yield superhuman results (OpenBookQA). In doing so, we reveal how evaluation shapes perceived difficulty and offer guidelines to ensure that multiple-choice evaluations accurately reflect actual model capabilities.
• 摘要：对于现代 LLM 来说，ARC Challenge 似乎比 ARC Easy 更难，这主要是因为评估设置阻止了对答案选项的直接比较，而不是固有的复杂性。尽管一些研究人员在过去一年中悄悄转向了更合适的方案，但这种变化的影响尚未得到广泛认可。我们强调了这种被忽视的转变，展示了类似的评估实践如何错误地暗示了其他基准中的推理缺陷，并证明了更公平的方法可以显著缩小性能差距（例如在 SIQA 上），甚至产生超人的结果（OpenBookQA）。通过这样做，我们揭示了评估如何影响感知难度，并提供了指导方针，以确保多项选择评估准确反映实际的模型能力。

Title: ResearchTown: Simulator of Human Research Community

• Authors: Haofei Yu, Zhaochen Hong, Zirui Cheng, Kunlun Zhu, Keyang Xuan, Jinwei Yao, Tao Feng, Jiaxuan You
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.17767
• Pdf URL: https://arxiv.org/pdf/2412.17767
• Copy Paste: [[2412.17767]] ResearchTown: Simulator of Human Research Community(https://arxiv.org/abs/2412.17767)
• Keywords: language model, llm, agent
• Abstract: Large Language Models (LLMs) have demonstrated remarkable potential in scientific domains, yet a fundamental question remains unanswered: Can we simulate human research communities with LLMs? Addressing this question can deepen our understanding of the processes behind idea brainstorming and inspire the automatic discovery of novel scientific insights. In this work, we propose ResearchTown, a multi-agent framework for research community simulation. Within this framework, the human research community is simplified and modeled as an agent-data graph, where researchers and papers are represented as agent-type and data-type nodes, respectively, and connected based on their collaboration relationships. We also introduce TextGNN, a text-based inference framework that models various research activities (e.g., paper reading, paper writing, and review writing) as special forms of a unified message-passing process on the agent-data graph. To evaluate the quality of the research simulation, we present ResearchBench, a benchmark that uses a node-masking prediction task for scalable and objective assessment based on similarity. Our experiments reveal three key findings: (1) ResearchTown can provide a realistic simulation of collaborative research activities, including paper writing and review writing; (2) ResearchTown can maintain robust simulation with multiple researchers and diverse papers; (3) ResearchTown can generate interdisciplinary research ideas that potentially inspire novel research directions.
• 摘要：大型语言模型 (LLM) 在科学领域展现出了巨大的潜力，但一个基本问题仍未得到解答：我们能用 LLM 模拟人类研究社区吗？解决这个问题可以加深我们对头脑风暴背后过程的理解，并激发人们自动发现新颖的科学见解。在这项工作中，我们提出了 ResearchTown，这是一个用于研究社区模拟的多智能体框架。在这个框架内，人类研究社区被简化并建模为一个智能体数据图，其中研究人员和论文分别表示为智能体类型和数据类型节点，并根据他们的合作关系进行连接。我们还引入了 TextGNN，这是一个基于文本的推理框架，它将各种研究活动（例如，论文阅读、论文写作和评论写作）建模为智能体数据图上统一消息传递过程的特殊形式。为了评估研究模拟的质量，我们提出了 ResearchBench，这是一个基准，它使用节点掩码预测任务进行基于相似性的可扩展和客观评估。我们的实验揭示了三个主要发现：（1）ResearchTown 可以提供协作研究活动的真实模拟，包括论文写作和评论写作；（2）ResearchTown 可以通过多名研究人员和多样化论文保持稳健的模拟；（3）ResearchTown 可以产生跨学科的研究想法，从而有可能激发新的研究方向。