2024-12-31

Title: GaLore$+$: Boosting Low-Rank Adaptation for LLMs with Cross-Head Projection

• Authors: Xutao Liao, Shaohui Li, Yuhui Xu, Zhi Li, Yu Liu, You He
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.19820
• Pdf URL: https://arxiv.org/pdf/2412.19820
• Copy Paste: [[2412.19820]] GaLore$+$: Boosting Low-Rank Adaptation for LLMs with Cross-Head Projection(https://arxiv.org/abs/2412.19820)
• Keywords: language model, llm
• Abstract: Recent low-rank training methods, such as GaLore, have significantly reduced the memory required to optimize large language models (LLMs). However, these methods often suffer from time-consuming low-rank projection estimations. In particular, the singular value decomposition (SVD) in GaLore can consume more than 80\% of the total training time. To address this issue, we propose GaLore$+$, which uses cross-head low-rank projection to reduce the substantial time consumption in estimating low-rank projections for multi-head attention. In addition, we employ randomized subspace iteration to achieve fast SVD. To further enhance performance, we propose sparsely coded residuals to reduce the errors caused by low-rank approximation on the first- and second-order moments of the optimizers and weight updates. We evaluate GaLore$+$ on arithmetic reasoning and natural language generation datasets. Our experiments demonstrate that GaLore$+$ delivers superior performance while achieving approximately $4\times$ fine-tuning speed compared to vanilla GaLore.
• 摘要：最近的低秩训练方法（例如 GaLore）已大大减少了优化大型语言模型 (LLM) 所需的内存。然而，这些方法通常受到耗时的低秩投影估计的影响。特别是，GaLore 中的奇异值分解 (SVD) 可能消耗超过 80% 的总训练时间。为了解决这个问题，我们提出了 GaLore$+$，它使用跨头低秩投影来减少估计多头注意力低秩投影的大量时间消耗。此外，我们采用随机子空间迭代来实现快速 SVD。为了进一步提高性能，我们提出了稀疏编码残差来减少由优化器一阶和二阶矩和权重更新上的低秩近似引起的误差。我们在算术推理和自然语言生成数据集上评估了 GaLore$+$。我们的实验表明，与 vanilla GaLore 相比，GaLore$+$ 具有更优异的性能，同时实现了大约 $4\times$ 倍的微调速度。

Title: Evaluate Summarization in Fine-Granularity: Auto Evaluation with LLM

• Authors: Dong Yuan, Eti Rastogi, Fen Zhao, Sagar Goyal, Gautam Naik, Sree Prasanna Rajagopal
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.19906
• Pdf URL: https://arxiv.org/pdf/2412.19906
• Copy Paste: [[2412.19906]] Evaluate Summarization in Fine-Granularity: Auto Evaluation with LLM(https://arxiv.org/abs/2412.19906)
• Keywords: llm, prompt
• Abstract: Due to the exponential growth of information and the need for efficient information consumption the task of summarization has gained paramount importance. Evaluating summarization accurately and objectively presents significant challenges, particularly when dealing with long and unstructured texts rich in content. Existing methods, such as ROUGE (Lin, 2004) and embedding similarities, often yield scores that have low correlation with human judgements and are also not intuitively understandable, making it difficult to gauge the true quality of the summaries. LLMs can mimic human in giving subjective reviews but subjective scores are hard to interpret and justify. They can be easily manipulated by altering the models and the tones of the prompts. In this paper, we introduce a novel evaluation methodology and tooling designed to address these challenges, providing a more comprehensive, accurate and interpretable assessment of summarization outputs. Our method (SumAutoEval) proposes and evaluates metrics at varying granularity levels, giving objective scores on 4 key dimensions such as completeness, correctness, Alignment and readability. We empirically demonstrate, that SumAutoEval enhances the understanding of output quality with better human correlation.
• 摘要：由于信息的指数级增长和对高效信息消费的需求，摘要任务变得至关重要。准确客观地评估摘要是一项重大挑战，特别是在处理内容丰富的长篇非结构化文本时。现有方法，如 ROUGE（Lin，2004）和嵌入相似性，通常得出的分数与人类判断的相关性较低，并且也难以直观理解，因此很难衡量摘要的真实质量。LLM 可以模仿人类给出主观评价，但主观分数很难解释和证明。它们可以通过改变模型和提示的语气轻松操纵。在本文中，我们介绍了一种旨在应对这些挑战的新型评估方法和工具，为摘要输出提供更全面、更准确和更可解释的评估。我们的方法（SumAutoEval）提出并评估了不同粒度级别的指标，给出了完整性、正确性、对齐和可读性等 4 个关键维度的客观分数。我们通过实证证明，SumAutoEval 通过更好的人机相关性增强了对输出质量的理解。

Title: HADES: Hardware Accelerated Decoding for Efficient Speculation in Large Language Models

• Authors: Ze Yang, Yihong Jin, Xinhe Xu
• Subjects: cs.CL, cs.AI, cs.AR
• Abstract URL: https://arxiv.org/abs/2412.19925
• Pdf URL: https://arxiv.org/pdf/2412.19925
• Copy Paste: [[2412.19925]] HADES: Hardware Accelerated Decoding for Efficient Speculation in Large Language Models(https://arxiv.org/abs/2412.19925)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have revolutionized natural language processing by understanding and generating human-like text. However, the increasing demand for more sophisticated LLMs presents significant computational challenges due to their scale and complexity. This paper introduces Hardware Accelerated Decoding (HADES), a novel approach to enhance the performance and energy efficiency of LLMs. We address the design of an LLM accelerator with hardware-level speculative decoding support, a concept not previously explored in existing literature. Our work demonstrates how speculative decoding can significantly improve the efficiency of LLM operations, paving the way for more advanced and practical applications of these models.
• 摘要：大型语言模型 (LLM) 通过理解和生成类似人类的文本彻底改变了自然语言处理。然而，对更复杂的 LLM 的需求日益增长，由于其规模和复杂性，带来了巨大的计算挑战。本文介绍了硬件加速解码 (HADES)，这是一种提高 LLM 性能和能效的新方法。我们设计了一种具有硬件级推测解码支持的 LLM 加速器，这是现有文献中以前从未探讨过的概念。我们的工作展示了推测解码如何显著提高 LLM 操作的效率，为这些模型更先进、更实际的应用铺平了道路。

Title: Right vs. Right: Can LLMs Make Tough Choices?

• Authors: Jiaqing Yuan, Pradeep K. Murukannaiah, Munindar P. Singh
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.19926
• Pdf URL: https://arxiv.org/pdf/2412.19926
• Copy Paste: [[2412.19926]] Right vs. Right: Can LLMs Make Tough Choices?(https://arxiv.org/abs/2412.19926)
• Keywords: llm, prompt
• Abstract: An ethical dilemma describes a choice between two "right" options involving conflicting moral values. We present a comprehensive evaluation of how LLMs navigate ethical dilemmas. Specifically, we investigate LLMs on their (1) sensitivity in comprehending ethical dilemmas, (2) consistency in moral value choice, (3) consideration of consequences, and (4) ability to align their responses to a moral value preference explicitly or implicitly specified in a prompt. Drawing inspiration from a leading ethical framework, we construct a dataset comprising 1,730 ethical dilemmas involving four pairs of conflicting values. We evaluate 20 well-known LLMs from six families. Our experiments reveal that: (1) LLMs exhibit pronounced preferences between major value pairs, and prioritize truth over loyalty, community over individual, and long-term over short-term considerations. (2) The larger LLMs tend to support a deontological perspective, maintaining their choices of actions even when negative consequences are specified. (3) Explicit guidelines are more effective in guiding LLMs' moral choice than in-context examples. Lastly, our experiments highlight the limitation of LLMs in comprehending different formulations of ethical dilemmas.
• 摘要：道德困境描述了在涉及相互冲突的道德价值观的两个“正确”选项之间做出选择。我们对 LLM 如何应对道德困境进行了全面的评估。具体来说，我们调查了 LLM 的 (1) 理解道德困境的敏感性、(2) 道德价值选择的一致性、(3) 对后果的考虑，以及 (4) 使其反应与提示中明确或隐含指定的道德价值偏好保持一致的能力。从领先的道德框架中汲取灵感，我们构建了一个数据集，其中包含 1,730 个涉及四对相互冲突的价值观的道德困境。我们评估了来自六个家族的 20 位知名 LLM。我们的实验表明：(1) LLM 在主要价值观对之间表现出明显的偏好，并且优先考虑真相而不是忠诚、社区而不是个人、长期考虑而不是短期考虑。(2) 较大的 LLM 倾向于支持义务论观点，即使指定了负面后果，他们也会保持自己的行动选择。(3) 明确的指导方针比情境中的例子更能有效地指导 LLM 的道德选择。最后，我们的实验强调了法学硕士在理解不同形式的道德困境方面的局限性。

Title: Assessing Text Classification Methods for Cyberbullying Detection on Social Media Platforms

• Authors: Adamu Gaston Philipo, Doreen Sebastian Sarwatt, Jianguo Ding, Mahmoud Daneshmand, Huansheng Ning
• Subjects: cs.CL, cs.SI
• Abstract URL: https://arxiv.org/abs/2412.19928
• Pdf URL: https://arxiv.org/pdf/2412.19928
• Copy Paste: [[2412.19928]] Assessing Text Classification Methods for Cyberbullying Detection on Social Media Platforms(https://arxiv.org/abs/2412.19928)
• Keywords: language model, gpt
• Abstract: Cyberbullying significantly contributes to mental health issues in communities by negatively impacting the psychology of victims. It is a prevalent problem on social media platforms, necessitating effective, real-time detection and monitoring systems to identify harmful messages. However, current cyberbullying detection systems face challenges related to performance, dataset quality, time efficiency, and computational costs. This research aims to conduct a comparative study by adapting and evaluating existing text classification techniques within the cyberbullying detection domain. The study specifically evaluates the effectiveness and performance of these techniques in identifying cyberbullying instances on social media platforms. It focuses on leveraging and assessing large language models, including BERT, RoBERTa, XLNet, DistilBERT, and GPT-2.0, for their suitability in this domain. The results show that BERT strikes a balance between performance, time efficiency, and computational resources: Accuracy of 95%, Precision of 95%, Recall of 95%, F1 Score of 95%, Error Rate of 5%, Inference Time of 0.053 seconds, RAM Usage of 35.28 MB, CPU/GPU Usage of 0.4%, and Energy Consumption of 0.000263 kWh. The findings demonstrate that generative AI models, while powerful, do not consistently outperform fine-tuned models on the tested benchmarks. However, state-of-the-art performance can still be achieved through strategic adaptation and fine-tuning of existing models for specific datasets and tasks.
• 摘要：网络欺凌对受害者的心理产生负面影响，严重加剧了社区的心理健康问题。这是社交媒体平台上普遍存在的问题，需要有效的实时检测和监控系统来识别有害信息。然而，当前的网络欺凌检测系统面临着与性能、数据集质量、时间效率和计算成本相关的挑战。本研究旨在通过调整和评估网络欺凌检测领域中现有的文本分类技术来进行一项比较研究。该研究专门评估了这些技术在识别社交媒体平台上的网络欺凌实例方面的有效性和性能。它侧重于利用和评估大型语言模型（包括 BERT、RoBERTa、XLNet、DistilBERT 和 GPT-2.0）来确定它们是否适用于该领域。结果表明，BERT 在性能、时间效率和计算资源之间取得了平衡：准确率为 95%，精确率为 95%，召回率为 95%，F1 得分为 95%，错误率为 5%，推理时间为 0.053 秒，RAM 使用率为 35.28 MB，CPU/GPU 使用率为 0.4%，能耗为 0.000263 kWh。研究结果表明，生成式 AI 模型虽然功能强大，但在测试基准上的表现并不总是优于微调模型。然而，通过针对特定数据集和任务对现有模型进行战略性调整和微调，仍然可以实现最先进的性能。

Title: Bridging Context Gaps: Enhancing Comprehension in Long-Form Social Conversations Through Contextualized Excerpts

• Authors: Shrestha Mohanty, Sarah Xuan, Jacob Jobraeel, Anurag Kumar, Deb Roy, Jad Kabbara
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.19966
• Pdf URL: https://arxiv.org/pdf/2412.19966
• Copy Paste: [[2412.19966]] Bridging Context Gaps: Enhancing Comprehension in Long-Form Social Conversations Through Contextualized Excerpts(https://arxiv.org/abs/2412.19966)
• Keywords: language model, llm
• Abstract: We focus on enhancing comprehension in small-group recorded conversations, which serve as a medium to bring people together and provide a space for sharing personal stories and experiences on crucial social matters. One way to parse and convey information from these conversations is by sharing highlighted excerpts in subsequent conversations. This can help promote a collective understanding of relevant issues, by highlighting perspectives and experiences to other groups of people who might otherwise be unfamiliar with and thus unable to relate to these experiences. The primary challenge that arises then is that excerpts taken from one conversation and shared in another setting might be missing crucial context or key elements that were previously introduced in the original conversation. This problem is exacerbated when conversations become lengthier and richer in themes and shared experiences. To address this, we explore how Large Language Models (LLMs) can enrich these excerpts by providing socially relevant context. We present approaches for effective contextualization to improve comprehension, readability, and empathy. We show significant improvements in understanding, as assessed through subjective and objective evaluations. While LLMs can offer valuable context, they struggle with capturing key social aspects. We release the Human-annotated Salient Excerpts (HSE) dataset to support future work. Additionally, we show how context-enriched excerpts can provide more focused and comprehensive conversation summaries.
• 摘要：我们专注于提高小组录音对话的理解力，这种对话是将人们聚集在一起的媒介，并提供了一个分享有关重要社会问题的个人故事和经历的空间。解析和传达这些对话信息的一种方法是在后续对话中分享突出显示的摘录。这可以帮助促进对相关问题的集体理解，通过向其他可能不熟悉并因此无法理解这些经历的人群强调观点和经验。然后出现的主要挑战是，从一次对话中摘录并在另一个环境中分享的摘录可能缺少先前在原始对话中引入的关键背景或关键元素。当对话变得更长、主题和共享经验更丰富时，这个问题会更加严重。为了解决这个问题，我们探索了大型语言模型 (LLM) 如何通过提供与社会相关的背景来丰富这些摘录。我们提出了有效的语境化方法，以提高理解力、可读性和同理心。通过主观和客观评估，我们显示出理解力的显着提高。虽然 LLM 可以提供有价值的背景信息，但它们很难捕捉到关键的社交方面。我们发布了人工注释的突出摘录 (HSE) 数据集来支持未来的工作。此外，我们还展示了富含背景的摘录如何提供更有针对性和更全面的对话摘要。

Title: OneKE: A Dockerized Schema-Guided LLM Agent-based Knowledge Extraction System

• Authors: Yujie Luo, Xiangyuan Ru, Kangwei Liu, Lin Yuan, Mengshu Sun, Ningyu Zhang, Lei Liang, Zhiqiang Zhang, Jun Zhou, Lanning Wei, Da Zheng, Haofen Wang, Huajun Chen
• Subjects: cs.CL, cs.AI, cs.DB, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.20005
• Pdf URL: https://arxiv.org/pdf/2412.20005
• Copy Paste: [[2412.20005]] OneKE: A Dockerized Schema-Guided LLM Agent-based Knowledge Extraction System(https://arxiv.org/abs/2412.20005)
• Keywords: llm, agent
• Abstract: We introduce OneKE, a dockerized schema-guided knowledge extraction system, which can extract knowledge from the Web and raw PDF Books, and support various domains (science, news, etc.). Specifically, we design OneKE with multiple agents and a configure knowledge base. Different agents perform their respective roles, enabling support for various extraction scenarios. The configure knowledge base facilitates schema configuration, error case debugging and correction, further improving the performance. Empirical evaluations on benchmark datasets demonstrate OneKE's efficacy, while case studies further elucidate its adaptability to diverse tasks across multiple domains, highlighting its potential for broad applications. We have open-sourced the Code at this https URL and released a Video at this http URL.
• 摘要：我们引入了 OneKE，这是一个基于 Docker 的模式引导知识提取系统，它可以从 Web 和原始 PDF 书籍中提取知识，并支持各种领域（科学、新闻等）。具体来说，我们设计了具有多个代理和一个配置知识库的 OneKE。不同的代理执行各自的角色，从而支持各种提取场景。配置知识库有助于模式配置、错误案例调试和更正，从而进一步提高性能。对基准数据集的实证评估证明了 OneKE 的有效性，而案例研究进一步阐明了它对多个领域中各种任务的适应性，突出了其广泛应用的潜力。我们已在此 https URL 上开源了代码，并在此 http URL 上发布了视频。

Title: STAYKATE: Hybrid In-Context Example Selection Combining Representativeness Sampling and Retrieval-based Approach -- A Case Study on Science Domains

• Authors: Chencheng Zhu, Kazutaka Shimada, Tomoki Taniguchi, Tomoko Ohkuma
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.20043
• Pdf URL: https://arxiv.org/pdf/2412.20043
• Copy Paste: [[2412.20043]] STAYKATE: Hybrid In-Context Example Selection Combining Representativeness Sampling and Retrieval-based Approach -- A Case Study on Science Domains(https://arxiv.org/abs/2412.20043)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) demonstrate the ability to learn in-context, offering a potential solution for scientific information extraction, which often contends with challenges such as insufficient training data and the high cost of annotation processes. Given that the selection of in-context examples can significantly impact performance, it is crucial to design a proper method to sample the efficient ones. In this paper, we propose STAYKATE, a static-dynamic hybrid selection method that combines the principles of representativeness sampling from active learning with the prevalent retrieval-based approach. The results across three domain-specific datasets indicate that STAYKATE outperforms both the traditional supervised methods and existing selection methods. The enhancement in performance is particularly pronounced for entity types that other methods pose challenges.
• 摘要：大型语言模型 (LLM) 展示了在上下文中学习的能力，为科学信息提取提供了一种潜在的解决方案，而科学信息提取通常面临着诸如训练数据不足和注释过程成本高昂等挑战。鉴于上下文示例的选择会显著影响性能，设计一种适当的方法来对有效示例进行采样至关重要。在本文中，我们提出了 STAYKATE，这是一种静态-动态混合选择方法，它将主动学习的代表性采样原理与流行的基于检索的方法相结合。三个领域特定数据集的结果表明，STAYKATE 优于传统的监督方法和现有的选择方法。对于其他方法具有挑战性的实体类型，性能的提升尤为明显。

Title: "My life is miserable, have to sign 500 autographs everyday": Exposing Humblebragging, the Brags in Disguise

• Authors: Sharath Naganna, Saprativa Bhattacharjee, Pushpak Bhattacharyya, Biplab Banerjee
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.20057
• Pdf URL: https://arxiv.org/pdf/2412.20057
• Copy Paste: [[2412.20057]] "My life is miserable, have to sign 500 autographs everyday": Exposing Humblebragging, the Brags in Disguise(https://arxiv.org/abs/2412.20057)
• Keywords: language model, gpt, llm
• Abstract: Humblebragging is a phenomenon where individuals present self-promotional statements under the guise of modesty or complaints. For example, a statement like, "Ugh, I can't believe I got promoted to lead the entire team. So stressful!", subtly highlights an achievement while pretending to be complaining. Detecting humblebragging is important for machines to better understand the nuances of human language, especially in tasks like sentiment analysis and intent recognition. However, this topic has not yet been studied in computational linguistics. For the first time, we introduce the task of automatically detecting humblebragging in text. We formalize the task by proposing a 4-tuple definition of humblebragging and evaluate machine learning, deep learning, and large language models (LLMs) on this task, comparing their performance with humans. We also create and release a dataset called HB24, containing 3,340 humblebrags generated using GPT-4o. Our experiments show that detecting humblebragging is non-trivial, even for humans. Our best model achieves an F1-score of 0.88. This work lays the foundation for further exploration of this nuanced linguistic phenomenon and its integration into broader natural language understanding systems.
• 摘要：谦虚自夸是一种现象，即个人以谦虚或抱怨为幌子进行自我宣传。例如，“哎呀，我不敢相信我被提拔为整个团队的领导。压力太大了！”这样的陈述，在假装抱怨的同时巧妙地强调了一项成就。检测谦虚自夸对于机器更好地理解人类语言的细微差别非常重要，尤其是在情绪分析和意图识别等任务中。然而，计算语言学中尚未研究过这个主题。我们首次引入了自动检测文本中谦虚自夸的任务。我们通过提出谦虚自夸的 4 元组定义来形式化该任务，并评估机器学习、深度学习和大型语言模型 (LLM) 在此任务上的表现，将它们的表现与人类进行比较。我们还创建并发布了一个名为 HB24 的数据集，其中包含使用 GPT-4o 生成的 3,340 个谦虚自夸。我们的实验表明，检测谦虚自夸并非易事，即使对于人类来说也是如此。我们的最佳模型实现了 0.88 的 F1 分数。这项工作为进一步探索这种微妙的语言现象及其融入更广泛的自然语言理解系统奠定了基础。

Title: Comparative Analysis of Listwise Reranking with Large Language Models in Limited-Resource Language Contexts

• Authors: Yanxin Shen, Lun Wang, Chuanqi Shi, Shaoshuai Du, Yiyi Tao, Yixian Shen, Hang Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.20061
• Pdf URL: https://arxiv.org/pdf/2412.20061
• Copy Paste: [[2412.20061]] Comparative Analysis of Listwise Reranking with Large Language Models in Limited-Resource Language Contexts(https://arxiv.org/abs/2412.20061)
• Keywords: language model, gpt, llm
• Abstract: Large Language Models (LLMs) have demonstrated significant effectiveness across various NLP tasks, including text ranking. This study assesses the performance of large language models (LLMs) in listwise reranking for limited-resource African languages. We compare proprietary models RankGPT3.5, Rank4o-mini, RankGPTo1-mini and RankClaude-sonnet in cross-lingual contexts. Results indicate that these LLMs significantly outperform traditional baseline methods such as BM25-DT in most evaluation metrics, particularly in nDCG@10 and MRR@100. These findings highlight the potential of LLMs in enhancing reranking tasks for low-resource languages and offer insights into cost-effective solutions.
• 摘要：大型语言模型 (LLM) 在各种 NLP 任务（包括文本排名）中都表现出显著的有效性。本研究评估了大型语言模型 (LLM) 在资源有限的非洲语言列表重排中的表现。我们在跨语言环境中比较了专有模型 RankGPT3.5、Rank4o-mini、RankGPTo1-mini 和 RankClaude-sonnet。结果表明，这些 LLM 在大多数评估指标中都明显优于传统基线方法（例如 BM25-DT），尤其是在 nDCG@10 和 MRR@100 中。这些发现凸显了 LLM 在增强资源匮乏语言重排任务方面的潜力，并提供了具有成本效益的解决方案的见解。

Title: Extract Information from Hybrid Long Documents Leveraging LLMs: A Framework and Dataset

• Authors: Chongjian Yue, Xinrun Xu, Xiaojun Ma, Lun Du, Zhiming Ding, Shi Han, Dongmei Zhang, Qi Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.20072
• Pdf URL: https://arxiv.org/pdf/2412.20072
• Copy Paste: [[2412.20072]] Extract Information from Hybrid Long Documents Leveraging LLMs: A Framework and Dataset(https://arxiv.org/abs/2412.20072)
• Keywords: language model, llm, prompt
• Abstract: Large Language Models (LLMs) demonstrate exceptional performance in textual understanding and tabular reasoning tasks. However, their ability to comprehend and analyze hybrid text, containing textual and tabular data, remains unexplored. The hybrid text often appears in the form of hybrid long documents (HLDs), which far exceed the token limit of LLMs. Consequently, we apply an Automated Information Extraction framework (AIE) to enable LLMs to process the HLDs and carry out experiments to analyse four important aspects of information extraction from HLDs. Given the findings: 1) The effective way to select and summarize the useful part of a HLD. 2) An easy table serialization way is enough for LLMs to understand tables. 3) The naive AIE has adaptability in many complex scenarios. 4) The useful prompt engineering to enhance LLMs on HLDs. To address the issue of dataset scarcity in HLDs and support future work, we also propose the Financial Reports Numerical Extraction (FINE) dataset. The dataset and code are publicly available in the attachments.
• 摘要：大型语言模型 (LLM) 在文本理解和表格推理任务中表现出色。然而，它们理解和分析包含文本和表格数据的混合文本的能力仍未被探索。混合文本通常以混合长文档 (HLD) 的形式出现，远远超出了 LLM 的标记限制。因此，我们应用自动信息提取框架 (AIE) 使 LLM 能够处理 HLD，并进行实验以分析从 HLD 中提取信息的四个重要方面。得出以下结论：1) 选择和总结 HLD 有用部分的有效方法。2) 简单的表格序列化方法足以让 LLM 理解表格。3) 朴素的 AIE 在许多复杂场景中具有适应性。4) 有用的提示工程可增强 HLD 上的 LLM。为了解决 HLD 中数据集稀缺的问题并支持未来的工作，我们还提出了财务报告数值提取 (FINE) 数据集。数据集和代码在附件中公开提供。

Title: M-MAD: Multidimensional Multi-Agent Debate Framework for Fine-grained Machine Translation Evaluation

• Authors: Zhaopeng Feng, Jiayuan Su, Jiamei Zheng, Jiahan Ren, Yan Zhang, Jian Wu, Hongwei Wang, Zuozhu Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.20127
• Pdf URL: https://arxiv.org/pdf/2412.20127
• Copy Paste: [[2412.20127]] M-MAD: Multidimensional Multi-Agent Debate Framework for Fine-grained Machine Translation Evaluation(https://arxiv.org/abs/2412.20127)
• Keywords: language model, gpt, llm, agent
• Abstract: Recent advancements in large language models (LLMs) have given rise to the LLM-as-a-judge paradigm, showcasing their potential to deliver human-like judgments. However, in the field of machine translation (MT) evaluation, current LLM-as-a-judge methods fall short of learned automatic metrics. In this paper, we propose Multidimensional Multi-Agent Debate (M-MAD), a systematic LLM-based multi-agent framework for advanced LLM-as-a-judge MT evaluation. Our findings demonstrate that M-MAD achieves significant advancements by (1) decoupling heuristic MQM criteria into distinct evaluation dimensions for fine-grained assessments; (2) employing multi-agent debates to harness the collaborative reasoning capabilities of LLMs; (3) synthesizing dimension-specific results into a final evaluation judgment to ensure robust and reliable outcomes. Comprehensive experiments show that M-MAD not only outperforms all existing LLM-as-a-judge methods but also competes with state-of-the-art reference-based automatic metrics, even when powered by a suboptimal model like GPT-4o mini. Detailed ablations and analysis highlight the superiority of our framework design, offering a fresh perspective for LLM-as-a-judge paradigm. Our code and data are publicly available at this https URL.
• 摘要：大型语言模型 (LLM) 的最新进展催生了 LLM-as-a-judge 范式，展示了其提供类似人类判断的潜力。然而，在机器翻译 (MT) 评估领域，当前的 LLM-as-a-judge 方法缺乏学习自动指标。在本文中，我们提出了多维多智能体辩论 (M-MAD)，这是一个基于 LLM 的系统多智能体框架，用于高级 LLM-as-judge MT 评估。我们的研究结果表明，M-MAD 通过以下方式取得了重大进展：(1) 将启发式 MQM 标准解耦为不同的评估维度，以进行细粒度评估；(2) 采用多智能体辩论来利用 LLM 的协作推理能力；(3) 将特定维度的结果综合到最终评估判断中，以确保结果稳健可靠。综合实验表明，M-MAD 不仅优于所有现有的 LLM-as-a-judge 方法，而且即使在使用 GPT-4o mini 等次优模型时，也能与最先进的基于参考的自动指标相媲美。详细的消融和分析凸显了我们框架设计的优越性，为 LLM-as-a-judge 范式提供了全新的视角。我们的代码和数据在此 https URL 上公开提供。

Title: Efficient Multi-Agent Collaboration with Tool Use for Online Planning in Complex Table Question Answering

• Authors: Wei Zhou, Mohsen Mesgar, Annemarie Friedrich, Heike Adel
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.20145
• Pdf URL: https://arxiv.org/pdf/2412.20145
• Copy Paste: [[2412.20145]] Efficient Multi-Agent Collaboration with Tool Use for Online Planning in Complex Table Question Answering(https://arxiv.org/abs/2412.20145)
• Keywords: language model, gpt, llm, agent
• Abstract: Complex table question answering (TQA) aims to answer questions that require complex reasoning, such as multi-step or multi-category reasoning, over data represented in tabular form. Previous approaches demonstrated notable performance by leveraging either closed-source large language models (LLMs) or fine-tuned open-weight LLMs. However, fine-tuning LLMs requires high-quality training data, which is costly to obtain, and utilizing closed-source LLMs poses accessibility challenges and leads to reproducibility issues. In this paper, we propose Multi-Agent Collaboration with Tool use (MACT), a framework that requires neither closed-source models nor fine-tuning. In MACT, a planning agent and a coding agent that also make use of tools collaborate to answer questions. Our experiments on four TQA benchmarks show that MACT outperforms previous SoTA systems on three out of four benchmarks and that it performs comparably to the larger and more expensive closed-source model GPT-4 on two benchmarks, even when using only open-weight models without any fine-tuning. We conduct extensive analyses to prove the effectiveness of MACT's multi-agent collaboration in TQA.
• 摘要：复杂表格问答 (TQA) 旨在回答需要复杂推理（例如多步骤或多类别推理）的问题，这些问题以表格形式表示。以前的方法通过利用闭源大型语言模型 (LLM) 或微调的开放权重 LLM 表现出显著的性能。然而，微调 LLM 需要高质量的训练数据，而这些数据的获取成本很高，并且使用闭源 LLM 会带来可访问性挑战并导致可重复性问题。在本文中，我们提出了使用工具的多智能体协作 (MACT)，这是一个既不需要闭源模型也不需要微调的框架。在 MACT 中，规划智能体和编码智能体也使用工具协作回答问题。我们在四个 TQA 基准上进行的实验表明，MACT 在四个基准中的三个上都优于以前的 SoTA 系统，并且在两个基准上的表现与更大、更昂贵的闭源模型 GPT-4 相当，即使只使用开放权重模型而没有任何微调。我们进行了广泛的分析来证明 MACT 在 TQA 中的多智能体协作的有效性。

Title: Building a Rich Dataset to Empower the Persian Question Answering Systems

• Authors: Mohsen Yazdinejad, Marjan Kaedi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.20212
• Pdf URL: https://arxiv.org/pdf/2412.20212
• Copy Paste: [[2412.20212]] Building a Rich Dataset to Empower the Persian Question Answering Systems(https://arxiv.org/abs/2412.20212)
• Keywords: language model
• Abstract: Question answering systems provide short, precise, and specific answers to questions. So far, many robust question answering systems have been developed for English, while some languages with fewer resources, like Persian, have few numbers of standard dataset. In this study, a comprehensive open-domain dataset is presented for Persian. This dataset is called NextQuAD and has 7,515 contexts, including 23,918 questions and answers. Then, a BERT-based question answering model has been applied to this dataset using two pre-trained language models, including ParsBERT and XLM-RoBERTa. The results of these two models have been ensembled using mean logits. Evaluation on the development set shows 0.95 Exact Match (EM) and 0.97 Fl_score. Also, to compare the NextQuAD with other Persian datasets, our trained model on the NextQuAD, is evaluated on two other datasets named PersianQA and ParSQuAD. Comparisons show that the proposed model increased EM by 0.39 and 0.14 respectively in PersianQA and ParSQuAD-manual, while a slight EM decline of 0.007 happened in ParSQuAD-automatic.
• 摘要：问答系统为问题提供简短、精确和具体的答案。到目前为止，已经为英语开发了许多强大的问答系统，而一些资源较少的语言（如波斯语）则只有少量的标准数据集。在本研究中，为波斯语提供了一个全面的开放域数据集。该数据集称为 NextQuAD，有 7,515 个上下文，包括 23,918 个问题和答案。然后，使用两个预训练的语言模型（包括 ParsBERT 和 XLM-RoBERTa），将基于 BERT 的问答模型应用于该数据集。使用平均 logits 集成了这两个模型的结果。在开发集上的评估显示精确匹配 (EM) 为 0.95，Fl_score 为 0.97。此外，为了将 NextQuAD 与其他波斯语数据集进行比较，我们在 NextQuAD 上训练的模型在另外两个名为 PersianQA 和 ParSQuAD 的数据集上进行了评估。比较表明，所提出的模型在 PersianQA 和 ParSQuAD-manual 中分别将 EM 提高了 0.39 和 0.14，而在 ParSQuAD-automatic 中 EM 略有下降，为 0.007。

Title: AfriHG: News headline generation for African Languages

• Authors: Toyib Ogunremi, Serah Akojenu, Anthony Soronnadi, Olubayo Adekanmbi, David Ifeoluwa Adelani
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.20223
• Pdf URL: https://arxiv.org/pdf/2412.20223
• Copy Paste: [[2412.20223]] AfriHG: News headline generation for African Languages(https://arxiv.org/abs/2412.20223)
• Keywords: llm, prompt
• Abstract: This paper introduces AfriHG -- a news headline generation dataset created by combining from XLSum and MasakhaNEWS datasets focusing on 16 languages widely spoken by Africa. We experimented with two seq2eq models (mT5-base and AfriTeVa V2), and Aya-101 LLM. Our results show that Africa-centric seq2seq models such as AfriTeVa V2 outperform the massively multilingual mT5-base model. Finally, we show that the performance of fine-tuning AfriTeVa V2 with 313M parameters is competitive to prompting Aya-101 LLM with more than 13B parameters.
• 摘要：本文介绍了 AfriHG——一个新闻标题生成数据集，由 XLSum 和 MasakhaNEWS 数据集合并而成，重点关注非洲广泛使用的 16 种语言。我们尝试了两个 seq2eq 模型（mT5-base 和 AfriTeVa V2）和 Aya-101 LLM。我们的结果表明，以非洲为中心的 seq2seq 模型（如 AfriTeVa V2）优于大规模多语言 mT5-base 模型。最后，我们表明，微调具有 313M 个参数的 AfriTeVa V2 的性能与提示具有超过 13B 个参数的 Aya-101 LLM 相媲美。

Title: LLM Reasoning Engine: Specialized Training for Enhanced Mathematical Reasoning

• Authors: Shuguang Chen, Guang Lin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.20227
• Pdf URL: https://arxiv.org/pdf/2412.20227
• Copy Paste: [[2412.20227]] LLM Reasoning Engine: Specialized Training for Enhanced Mathematical Reasoning(https://arxiv.org/abs/2412.20227)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have shown remarkable performance in various natural language processing tasks but face challenges in mathematical reasoning, where complex problem-solving requires both linguistic understanding and mathematical reasoning skills. Existing approaches to address this challenge often rely on ensemble methods and suffer from the problem of data scarcity in target domains. In this work, we present a novel method to enhance LLMs' capabilities in mathematical reasoning tasks. Motivated by the need to bridge this gap, our approach incorporates a question paraphrase strategy, which aims at diversifying the linguistic forms of mathematical questions to improve generalization. Additionally, specialized training objectives are employed to guide the model's learning process, focusing on enhancing its understanding of mathematical concepts and reasoning processes. We conduct experiments on four datasets using different LLMs, and demonstrate the effectiveness of our approach in improving LLMs' performance on mathematical reasoning tasks. Our findings underscore the significance of our methodology in the advancement of large language models and its potential implications for real-world applications that require mathematical reasoning abilities.
• 摘要：大型语言模型 (LLM) 在各种自然语言处理任务中表现出色，但在数学推理方面面临挑战，因为解决复杂问题需要语言理解和数学推理技能。现有的应对这一挑战的方法通常依赖于集成方法，并且存在目标域数据稀缺的问题。在这项工作中，我们提出了一种新方法来增强 LLM 在数学推理任务中的能力。为了弥补这一差距，我们的方法采用了问题释义策略，旨在使数学问题的语言形式多样化，以提高泛化能力。此外，还采用专门的训练目标来指导模型的学习过程，重点是增强模型对数学概念和推理过程的理解。我们使用不同的 LLM 在四个数据集上进行实验，并证明了我们的方法在提高 LLM 在数学推理任务上的表现方面的有效性。我们的研究结果强调了我们的方法在大型语言模型发展中的重要性，以及它对需要数学推理能力的现实世界应用的潜在影响。

Title: ComparisonQA: Evaluating Factuality Robustness of LLMs Through Knowledge Frequency Control and Uncertainty

• Authors: Qing Zong, Zhaowei Wang, Tianshi Zheng, Xiyu Ren, Yangqiu Song
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.20251
• Pdf URL: https://arxiv.org/pdf/2412.20251
• Copy Paste: [[2412.20251]] ComparisonQA: Evaluating Factuality Robustness of LLMs Through Knowledge Frequency Control and Uncertainty(https://arxiv.org/abs/2412.20251)
• Keywords: gpt, llm
• Abstract: The rapid development of LLMs has sparked extensive research into their factual knowledge. Current works claim that LLMs fall short on questions requiring less frequent knowledge. However, their proof is incomplete since they only study the influence of entity frequency, which can not fully represent knowledge frequency. So we introduce ComparisonQA benchmark, containing 283K abstract questions, each instantiated by a pair of high-frequency and low-frequency entities. It ensures a controllable comparison because the difference of knowledge frequency between such a pair is only related to entity frequency. In addition, to avoid possible semantic shortcuts, which is a severe problem of current LLMs study, we design a two-round method for knowledge robustness measurement utilizing both correctness and uncertainty. Experiments reveal that LLMs exhibit particularly low robustness regarding low-frequency knowledge, and GPT-4o is even the worst under this measurement. Besides, we introduce an automatic method to filter out questions with low-quality and shortcuts to form ComparisonQA-Hard. We find that uncertainty effectively identifies such questions while maintaining the data size.
• 摘要：LLM 的快速发展引发了对其事实知识的广泛研究。当前的研究认为 LLM 在需要较少频率知识的问题上存在不足。然而，他们的证明是不完整的，因为他们只研究了实体频率的影响，而实体频率不能完全代表知识频率。因此，我们引入了 ComparisonQA 基准，包含 283K 个抽象问题，每个问题由一对高频和低频实体实例化。它确保了可控的比较，因为这样一对之间的知识频率差异只与实体频率有关。此外，为了避免可能的语义捷径（这是当前 LLM 研究的一个严重问题），我们设计了一种利用正确性和不确定性的两轮知识鲁棒性测量方法。实验表明，LLM 在低频知识方面表现出特别低的鲁棒性，GPT-4o 在这个测量下甚至是最差的。此外，我们引入了一种自动方法来过滤掉质量低下和捷径的问题，形成 ComparisonQA-Hard。我们发现不确定性可以在保持数据量的同时有效地识别此类问题。

Title: Scoring with Large Language Models: A Study on Measuring Empathy of Responses in Dialogues

• Authors: Henry J. Xie, Jinghan Zhang, Xinhao Zhang, Kunpeng Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.20264
• Pdf URL: https://arxiv.org/pdf/2412.20264
• Copy Paste: [[2412.20264]] Scoring with Large Language Models: A Study on Measuring Empathy of Responses in Dialogues(https://arxiv.org/abs/2412.20264)
• Keywords: language model, llm
• Abstract: In recent years, Large Language Models (LLMs) have become increasingly more powerful in their ability to complete complex tasks. One such task in which LLMs are often employed is scoring, i.e., assigning a numerical value from a certain scale to a subject. In this paper, we strive to understand how LLMs score, specifically in the context of empathy scoring. We develop a novel and comprehensive framework for investigating how effective LLMs are at measuring and scoring empathy of responses in dialogues, and what methods can be employed to deepen our understanding of LLM scoring. Our strategy is to approximate the performance of state-of-the-art and fine-tuned LLMs with explicit and explainable features. We train classifiers using various features of dialogues including embeddings, the Motivational Interviewing Treatment Integrity (MITI) Code, a set of explicit subfactors of empathy as proposed by LLMs, and a combination of the MITI Code and the explicit subfactors. Our results show that when only using embeddings, it is possible to achieve performance close to that of generic LLMs, and when utilizing the MITI Code and explicit subfactors scored by an LLM, the trained classifiers can closely match the performance of fine-tuned LLMs. We employ feature selection methods to derive the most crucial features in the process of empathy scoring. Our work provides a new perspective toward understanding LLM empathy scoring and helps the LLM community explore the potential of LLM scoring in social science studies.
• 摘要：近年来，大型语言模型 (LLM) 完成复杂任务的能力越来越强大。LLM 经常用于完成的任务之一是评分，即从某个范围为某个主题分配一个数值。在本文中，我们努力了解 LLM 如何评分，特别是在同理心评分方面。我们开发了一个新颖而全面的框架，用于研究 LLM 在测量和评分对话中回应的同理心方面的有效性，以及可以采用哪些方法来加深我们对 LLM 评分的理解。我们的策略是使用明确且可解释的特征来近似最先进和微调的 LLM 的性能。我们使用对话的各种特征来训练分类器，包括嵌入、动机访谈治疗完整性 (MITI) 代码、LLM 提出的一组明确的同理心子因素以及 MITI 代码和明确子因素的组合。我们的结果表明，仅使用嵌入时，可以实现接近通用 LLM 的性能，而当使用 MITI 代码和 LLM 评分的显式子因素时，经过训练的分类器可以与微调的 LLM 的性能非常接近。我们采用特征选择方法来得出共情评分过程中最重要的特征。我们的工作为理解 LLM 共情评分提供了一个新的视角，并帮助 LLM 社区探索 LLM 评分在社会科学研究中的潜力。

Title: No Preference Left Behind: Group Distributional Preference Optimization

• Authors: Binwei Yao, Zefan Cai, Yun-Shiuan Chuang, Shanglin Yang, Ming Jiang, Diyi Yang, Junjie Hu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.20299
• Pdf URL: https://arxiv.org/pdf/2412.20299
• Copy Paste: [[2412.20299]] No Preference Left Behind: Group Distributional Preference Optimization(https://arxiv.org/abs/2412.20299)
• Keywords: language model
• Abstract: Preferences within a group of people are not uniform but follow a distribution. While existing alignment methods like Direct Preference Optimization (DPO) attempt to steer models to reflect human preferences, they struggle to capture the distributional pluralistic preferences within a group. These methods often skew toward dominant preferences, overlooking the diversity of opinions, especially when conflicting preferences arise. To address this issue, we propose Group Distribution Preference Optimization (GDPO), a novel framework that aligns language models with the distribution of preferences within a group by incorporating the concept of beliefs that shape individual preferences. GDPO calibrates a language model using statistical estimation of the group's belief distribution and aligns the model with belief-conditioned preferences, offering a more inclusive alignment framework than traditional methods. In experiments using both synthetic controllable opinion generation and real-world movie review datasets, we show that DPO fails to align with the targeted belief distributions, while GDPO consistently reduces this alignment gap during training. Moreover, our evaluation metrics demonstrate that GDPO outperforms existing approaches in aligning with group distributional preferences, marking a significant advance in pluralistic alignment.
• 摘要：一群人中的偏好并不统一，而是遵循一定的分布。虽然现有的对齐方法（如直接偏好优化 (DPO)）试图引导模型反映人类偏好，但它们很难捕捉群体中的分布多元偏好。这些方法往往偏向主导偏好，忽视意见的多样性，尤其是在出现相互冲突的偏好时。为了解决这个问题，我们提出了群体分布偏好优化 (GDPO)，这是一个新颖的框架，它通过结合塑造个人偏好的信念概念，将语言模型与群体内的偏好分布对齐。GDPO 使用群体信念分布的统计估计来校准语言模型，并将模型与信念条件偏好对齐，从而提供比传统方法更具包容性的对齐框架。在使用合成可控意见生成和真实世界电影评论数据集的实验中，我们表明 DPO 无法与目标信念分布对齐，而 GDPO 在训练过程中持续缩小了这种对齐差距。此外，我们的评估指标表明，GDPO 在与群体分配偏好保持一致方面优于现有方法，标志着多元化一致性的重大进步。

Title: Understanding the Impact of Confidence in Retrieval Augmented Generation: A Case Study in the Medical Domain

• Authors: Shintaro Ozaki, Yuta Kato, Siyuan Feng, Masayo Tomita, Kazuki Hayashi, Ryoma Obara, Masafumi Oyamada, Katsuhiko Hayashi, Hidetaka Kamigaito, Taro Watanabe
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.20309
• Pdf URL: https://arxiv.org/pdf/2412.20309
• Copy Paste: [[2412.20309]] Understanding the Impact of Confidence in Retrieval Augmented Generation: A Case Study in the Medical Domain(https://arxiv.org/abs/2412.20309)
• Keywords: language model, llm, prompt, retrieval augmented generation
• Abstract: Retrieval Augmented Generation (RAG) complements the knowledge of Large Language Models (LLMs) by leveraging external information to enhance response accuracy for queries. This approach is widely applied in several fields by taking its advantage of injecting the most up-to-date information, and researchers are focusing on understanding and improving this aspect to unlock the full potential of RAG in such high-stakes applications. However, despite the potential of RAG to address these needs, the mechanisms behind the confidence levels of its outputs remain underexplored, although the confidence of information is very critical in some domains, such as finance, healthcare, and medicine. Our study focuses the impact of RAG on confidence within the medical domain under various configurations and models. We evaluate confidence by treating the model's predicted probability as its output and calculating Expected Calibration Error (ECE) and Adaptive Calibration Error (ACE) scores based on the probabilities and accuracy. In addition, we analyze whether the order of retrieved documents within prompts calibrates the confidence. Our findings reveal large variation in confidence and accuracy depending on the model, settings, and the format of input prompts. These results underscore the necessity of optimizing configurations based on the specific model and conditions.
• 摘要：检索增强生成 (RAG) 通过利用外部信息来提高查询的响应准确性，补充了大型语言模型 (LLM) 的知识。这种方法利用其注入最新信息的优势，广泛应用于多个领域，研究人员正致力于理解和改进这一方面，以充分发挥 RAG 在此类高风险应用中的潜力。然而，尽管 RAG 有潜力满足这些需求，但其输出置信度背后的机制仍未得到充分探索，尽管信息的置信度在某些领域（如金融、医疗保健和医学）非常关键。我们的研究重点关注 RAG 在各种配置和模型下对医学领域置信度的影响。我们通过将模型的预测概率视为其输出并根据概率和准确性计算预期校准误差 (ECE) 和自适应校准误差 (ACE) 分数来评估置信度。此外，我们分析提示中检索到的文档的顺序是否校准了置信度。我们的研究结果表明，置信度和准确度会因模型、设置和输入提示的格式而有很大差异。这些结果强调了根据特定模型和条件优化配置的必要性。

Title: HindiLLM: Large Language Model for Hindi

• Authors: Sanjay Chouhan, Shubha Brata Nath, Aparajita Dutta
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.20357
• Pdf URL: https://arxiv.org/pdf/2412.20357
• Copy Paste: [[2412.20357]] HindiLLM: Large Language Model for Hindi(https://arxiv.org/abs/2412.20357)
• Keywords: language model, llm
• Abstract: The advancements in the Large Language Model (LLM) have helped in solving several problems related to language processing. Most of the researches have focused on the English language only, because of its popularity and abundance on the internet. However, a high-performance language model for Hindi and other Indic languages is lacking in the literature. In this work, we have pre-trained two autoregressive LLM models for the Hindi language, namely HindiLLM-Small and HindiLLM-Medium. We use a two-step process comprising unsupervised pre-training and supervised fine-tuning. First, we create a large and high-quality text corpus for unsupervised pre-training. Next, we train a Byte-Pair Encoding, named HindiLLM tokenizer, using the pre-training text data. We then perform training on the unlabeled data, known as the pre-training step, to get the HindiLLM base models. Furthermore, we perform fine-tuning of the HindiLLM base models for different tasks like sentiment analysis, text classification, natural language inference, and multiple choice question-answer on popular labeled datasets to measure the real-world performance. The evaluation shows that the HindiLLM-based fine-tuned models outperform several models in most of the language related tasks.
• 摘要：大型语言模型 (LLM) 的进步有助于解决与语言处理相关的许多问题。由于英语在互联网上非常流行且丰富，因此大多数研究仅集中在英语上。然而，文献中缺乏针对印地语和其他印度语的高性能语言模型。在这项工作中，我们预先训练了两个针对印地语的自回归 LLM 模型，即 HindiLLM-Small 和 HindiLLM-Medium。我们使用一个两步过程，包括无监督预训练和监督微调。首先，我们创建一个大型高质量文本语料库用于无监督预训练。接下来，我们使用预训练文本数据训练一个字节对编码，即 HindiLLM 标记器。然后，我们对未标记数据进行训练（称为预训练步骤），以获得 HindiLLM 基础模型。此外，我们针对情绪分析、文本分类、自然语言推理和热门标记数据集上的多项选择问答等不同任务对 HindiLLM 基础模型进行了微调，以衡量其实际性能。评估表明，基于 HindiLLM 的微调模型在大多数语言相关任务中都优于其他几种模型。

Title: LLM2: Let Large Language Models Harness System 2 Reasoning

• Authors: Cheng Yang, Chufan Shi, Siheng Li, Bo Shui, Yujiu Yang, Wai Lam
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.20372
• Pdf URL: https://arxiv.org/pdf/2412.20372
• Copy Paste: [[2412.20372]] LLM2: Let Large Language Models Harness System 2 Reasoning(https://arxiv.org/abs/2412.20372)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have exhibited impressive capabilities across a myriad of tasks, yet they occasionally yield undesirable outputs. We posit that these limitations are rooted in the foundational autoregressive architecture of LLMs, which inherently lacks mechanisms for differentiating between desirable and undesirable results. Drawing inspiration from the dual-process theory of human cognition, we introduce LLM2, a novel framework that combines an LLM (System 1) with a process-based verifier (System 2). Within LLM2, the LLM is responsible for generating plausible candidates, while the verifier provides timely process-based feedback to distinguish desirable and undesirable outputs. The verifier is trained with a pairwise comparison loss on synthetic process-supervision data generated through our token quality exploration strategy. Empirical results on mathematical reasoning benchmarks substantiate the efficacy of LLM2, exemplified by an accuracy enhancement from 50.3 to 57.8 (+7.5) for Llama3-1B on GSM8K. Furthermore, when combined with self-consistency, LLM2 achieves additional improvements, boosting major@20 accuracy from 56.2 to 70.2 (+14.0).
• 摘要：大型语言模型 (LLM) 在众多任务中展现出了令人印象深刻的能力，但它们偶尔会产生不理想的输出。我们认为这些限制根植于 LLM 的基础自回归架构，该架构本质上缺乏区分理想结果和不理想结果的机制。从人类认知的双重过程理论中汲取灵感，我们引入了 LLM2，这是一个将 LLM（系统 1）与基于过程的验证器（系统 2）相结合的新框架。在 LLM2 中，LLM 负责生成合理的候选，而验证器则提供及时的基于过程的反馈以区分理想输出和不理想输出。验证器使用通过我们的 token 质量探索策略生成的合成过程监督数据进行成对比较损失训练。数学推理基准的实证结果证实了 LLM2 的有效性，例如 Llama3-1B 在 GSM8K 上的准确率从 50.3 提高到 57.8（+7.5）。此外，与自洽性相结合时，LLM2 实现了额外的改进，将 major@20 准确率从 56.2 提升到 70.2 (+14.0)。

Title: Natural Language Fine-Tuning

• Authors: Jia Liu, Yue Wang, Zhiqi Lin, Min Chen, Yixue Hao, Long Hu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.20382
• Pdf URL: https://arxiv.org/pdf/2412.20382
• Copy Paste: [[2412.20382]] Natural Language Fine-Tuning(https://arxiv.org/abs/2412.20382)
• Keywords: language model, llm
• Abstract: Large language model fine-tuning techniques typically depend on extensive labeled data, external guidance, and feedback, such as human alignment, scalar rewards, and demonstration. However, in practical application, the scarcity of specific knowledge poses unprecedented challenges to existing fine-tuning techniques. In this paper, focusing on fine-tuning tasks in specific domains with limited data, we introduce Natural Language Fine-Tuning (NLFT), which utilizes natural language for fine-tuning for the first time. By leveraging the strong language comprehension capability of the target LM, NLFT attaches the guidance of natural language to the token-level outputs. Then, saliency tokens are identified with calculated probabilities. Since linguistic information is effectively utilized in NLFT, our proposed method significantly reduces training costs. It markedly enhances training efficiency, comprehensively outperforming reinforcement fine-tuning algorithms in accuracy, time-saving, and resource conservation. Additionally, on the macro level, NLFT can be viewed as a token-level fine-grained optimization of SFT, thereby efficiently replacing the SFT process without the need for warm-up (as opposed to ReFT requiring multiple rounds of warm-up with SFT). Compared to SFT, NLFT does not increase the algorithmic complexity, maintaining O(n). Extensive experiments on the GSM8K dataset demonstrate that NLFT, with only 50 data instances, achieves an accuracy increase that exceeds SFT by 219%. Compared to ReFT, the time complexity and space complexity of NLFT are reduced by 78.27% and 92.24%, respectively. The superior technique of NLFT is paving the way for the deployment of various innovative LLM fine-tuning applications when resources are limited at network edges. Our code has been released at this https URL.
• 摘要：大型语言模型微调技术通常依赖于大量标记数据、外部指导和反馈，例如人工对齐、标量奖励和演示。然而，在实际应用中，特定知识的稀缺对现有的微调技术提出了前所未有的挑战。在本文中，我们专注于数据有限的特定领域的微调任务，引入了自然语言微调 (NLFT)，首次利用自然语言进行微调。通过利用目标 LM 强大的语言理解能力，NLFT 将自然语言的指导附加到 token 级输出上。然后，使用计算出的概率来识别显着 token。由于 NLFT 有效地利用了语言信息，我们提出的方法显着降低了训练成本。它显著提高了训练效率，在准确性、节省时间和资源节省方面全面超越强化微调算法。此外，在宏观层面上，NLFT 可以看作是 SFT 的 token 级细粒度优化，从而高效替代 SFT 过程而无需预热（而 ReFT 需要使用 SFT 进行多轮预热）。与 SFT 相比，NLFT 不会增加算法复杂度，保持 O(n)。在 GSM8K 数据集上的大量实验表明，仅使用 50 个数据实例，NLFT 的准确率就比 SFT 提高了 219%。与 ReFT 相比，NLFT 的时间复杂度和空间复杂度分别降低了 78.27% 和 92.24%。NLFT 的卓越技术为在网络边缘资源有限的情况下部署各种创新的 LLM 微调应用程序铺平了道路。我们的代码已在此 https URL 上发布。

Title: Multi-Objective Large Language Model Unlearning

• Authors: Zibin Pan, Shuwen Zhang, Yuesheng Zheng, Chi Li, Yuheng Cheng, Junhua Zhao
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.20412
• Pdf URL: https://arxiv.org/pdf/2412.20412
• Copy Paste: [[2412.20412]] Multi-Objective Large Language Model Unlearning(https://arxiv.org/abs/2412.20412)
• Keywords: language model, llm
• Abstract: Machine unlearning in the domain of large language models (LLMs) has attracted great attention recently, which aims to effectively eliminate undesirable behaviors from LLMs without full retraining from scratch. In this paper, we explore the Gradient Ascent (GA) approach in LLM unlearning, which is a proactive way to decrease the prediction probability of the model on the target data in order to remove their influence. We analyze two challenges that render the process impractical: gradient explosion and catastrophic forgetting. To address these issues, we propose Multi-Objective Large Language Model Unlearning (MOLLM) algorithm. We first formulate LLM unlearning as a multi-objective optimization problem, in which the cross-entropy loss is modified to the unlearning version to overcome the gradient explosion issue. A common descent update direction is then calculated, which enables the model to forget the target data while preserving the utility of the LLM. Our empirical results verify that MoLLM outperforms the SOTA GA-based LLM unlearning methods in terms of unlearning effect and model utility preservation.
• 摘要：大型语言模型 (LLM) 领域的机器反学习最近引起了广泛关注，其目的是有效消除 LLM 中的不良行为，而无需从头开始进行完全重新训练。在本文中，我们探讨了 LLM 反学习中的梯度上升 (GA) 方法，这是一种主动降低模型对目标数据的预测概率以消除其影响的方法。我们分析了导致该过程不切实际的两个挑战：梯度爆炸和灾难性遗忘。为了解决这些问题，我们提出了多目标大型语言模型反学习 (MOLLM) 算法。我们首先将 LLM 反学习表述为多目标优化问题，其中交叉熵损失被修改为反学习版本以克服梯度爆炸问题。然后计算一个共同的下降更新方向，这使模型能够忘记目标数据，同时保留 LLM 的效用。我们的实证结果验证了 MoLLM 在去学习效果和模型效用保存方面优于基于 SOTA GA 的 LLM 去学习方法。

Title: Comparative Performance of Advanced NLP Models and LLMs in Multilingual Geo-Entity Detection

• Authors: Kalin Kopanov
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2412.20414
• Pdf URL: https://arxiv.org/pdf/2412.20414
• Copy Paste: [[2412.20414]] Comparative Performance of Advanced NLP Models and LLMs in Multilingual Geo-Entity Detection(https://arxiv.org/abs/2412.20414)
• Keywords: language model, gpt, llm
• Abstract: The integration of advanced Natural Language Processing (NLP) methodologies and Large Language Models (LLMs) has significantly enhanced the extraction and analysis of geospatial data from multilingual texts, impacting sectors such as national and international security. This paper presents a comprehensive evaluation of leading NLP models -- SpaCy, XLM-RoBERTa, mLUKE, GeoLM -- and LLMs, specifically OpenAI's GPT 3.5 and GPT 4, within the context of multilingual geo-entity detection. Utilizing datasets from Telegram channels in English, Russian, and Arabic, we examine the performance of these models through metrics such as accuracy, precision, recall, and F1 scores, to assess their effectiveness in accurately identifying geospatial references. The analysis exposes each model's distinct advantages and challenges, underscoring the complexities involved in achieving precise geo-entity identification across varied linguistic landscapes. The conclusions drawn from this experiment aim to direct the enhancement and creation of more advanced and inclusive NLP tools, thus advancing the field of geospatial analysis and its application to global security.
• 摘要：先进的自然语言处理 (NLP) 方法与大型语言模型 (LLM) 的结合显著增强了从多语言文本中提取和分析地理空间数据的能力，对国家和国际安全等领域产生了影响。本文在多语言地理实体检测的背景下，对领先的 NLP 模型——SpaCy、XLM-RoBERTa、mLUKE、GeoLM——和 LLM，特别是 OpenAI 的 GPT 3.5 和 GPT 4，进行了全面的评估。利用来自英语、俄语和阿拉伯语 Telegram 频道的数据集，我们通过准确率、精确率、召回率和 F1 分数等指标来检查这些模型的性能，以评估它们在准确识别地理空间参考方面的有效性。分析揭示了每个模型独特的优势和挑战，强调了在不同语言环境中实现精确地理实体识别所涉及的复杂性。该实验得出的结论旨在指导更先进、更具包容性的 NLP 工具的增强和创建，从而推动地理空间分析领域及其在全球安全中的应用。

Title: Enhancing Entertainment Translation for Indian Languages using Adaptive Context, Style and LLMs

• Authors: Pratik Rakesh Singh, Mohammadi Zaki, Pankaj Wasnik
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.20440
• Pdf URL: https://arxiv.org/pdf/2412.20440
• Copy Paste: [[2412.20440]] Enhancing Entertainment Translation for Indian Languages using Adaptive Context, Style and LLMs(https://arxiv.org/abs/2412.20440)
• Keywords: language model, llm, prompt
• Abstract: We address the challenging task of neural machine translation (NMT) in the entertainment domain, where the objective is to automatically translate a given dialogue from a source language content to a target language. This task has various applications, particularly in automatic dubbing, subtitling, and other content localization tasks, enabling source content to reach a wider audience. Traditional NMT systems typically translate individual sentences in isolation, without facilitating knowledge transfer of crucial elements such as the context and style from previously encountered sentences. In this work, we emphasize the significance of these fundamental aspects in producing pertinent and captivating translations. We demonstrate their significance through several examples and propose a novel framework for entertainment translation, which, to our knowledge, is the first of its kind. Furthermore, we introduce an algorithm to estimate the context and style of the current session and use these estimations to generate a prompt that guides a Large Language Model (LLM) to generate high-quality translations. Our method is both language and LLM-agnostic, making it a general-purpose tool. We demonstrate the effectiveness of our algorithm through various numerical studies and observe significant improvement in the COMET scores over various state-of-the-art LLMs. Moreover, our proposed method consistently outperforms baseline LLMs in terms of win-ratio.
• 摘要：我们解决了娱乐领域神经机器翻译 (NMT) 这一具有挑战性的任务，其目标是将给定的对话从源语言内容自动翻译成目标语言。此任务有多种应用，特别是在自动配音、字幕和其他内容本地化任务中，使源内容能够覆盖更广泛的受众。传统的 NMT 系统通常孤立地翻译单个句子，而不促进关键元素（例如以前遇到的句子中的上下文和风格）的知识转移。在这项工作中，我们强调了这些基本方面在生成相关且引人入胜的翻译中的重要性。我们通过几个例子证明了它们的重要性，并提出了一种新颖的娱乐翻译框架，据我们所知，这是同类中的第一个。此外，我们引入了一种算法来估计当前会话的上下文和风格，并使用这些估计来生成提示，指导大型语言模型 (LLM) 生成高质量的翻译。我们的方法与语言和 LLM 无关，使其成为一种通用工具。我们通过各种数值研究证明了我们算法的有效性，并观察到 ​​COMET 分数比各种最先进的 LLM 有显著提高。此外，我们提出的方法在胜率方面始终优于基线 LLM。

Title: SAFE-MEME: Structured Reasoning Framework for Robust Hate Speech Detection in Memes

• Authors: Palash Nandi, Shivam Sharma, Tanmoy Chakraborty
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2412.20541
• Pdf URL: https://arxiv.org/pdf/2412.20541
• Copy Paste: [[2412.20541]] SAFE-MEME: Structured Reasoning Framework for Robust Hate Speech Detection in Memes(https://arxiv.org/abs/2412.20541)
• Keywords: chain-of-thought
• Abstract: Memes act as cryptic tools for sharing sensitive ideas, often requiring contextual knowledge to interpret. This makes moderating multimodal memes challenging, as existing works either lack high-quality datasets on nuanced hate categories or rely on low-quality social media visuals. Here, we curate two novel multimodal hate speech datasets, MHS and MHS-Con, that capture fine-grained hateful abstractions in regular and confounding scenarios, respectively. We benchmark these datasets against several competing baselines. Furthermore, we introduce SAFE-MEME (Structured reAsoning FramEwork), a novel multimodal Chain-of-Thought-based framework employing Q&A-style reasoning (SAFE-MEME-QA) and hierarchical categorization (SAFE-MEME-H) to enable robust hate speech detection in memes. SAFE-MEME-QA outperforms existing baselines, achieving an average improvement of approximately 5% and 4% on MHS and MHS-Con, respectively. In comparison, SAFE-MEME-H achieves an average improvement of 6% in MHS while outperforming only multimodal baselines in MHS-Con. We show that fine-tuning a single-layer adapter within SAFE-MEME-H outperforms fully fine-tuned models in regular fine-grained hateful meme detection. However, the fully fine-tuning approach with a Q&A setup is more effective for handling confounding cases. We also systematically examine the error cases, offering valuable insights into the robustness and limitations of the proposed structured reasoning framework for analyzing hateful memes.
• 摘要：模因是分享敏感想法的神秘工具，通常需要上下文知识才能解释。这使得审核多模因具有挑战性，因为现有作品要么缺乏有关细微仇恨类别的高质量数据集，要么依赖于低质量的社交媒体视觉效果。在这里，我们整理了两个新的多模态仇恨言论数据集 MHS 和 MHS-Con，分别捕捉常规和混杂场景中的细粒度仇恨抽象。我们将这些数据集与几个竞争基线进行基准测试。此外，我们引入了 SAFE-MEME（结构化推理框架），这是一种基于思维链的新型多模态框架，采用问答式推理（SAFE-MEME-QA）和分层分类（SAFE-MEME-H）来实现模因中的强大仇恨言论检测。SAFE-MEME-QA 优于现有基线，在 MHS 和 MHS-Con 上分别实现了约 5% 和 4% 的平均改进。相比之下，SAFE-MEME-H 在 MHS 中实现了 6% 的平均改进，而在 MHS-Con 中仅优于多模态基线。我们表明，在常规细粒度仇恨模因检测中，在 SAFE-MEME-H 中微调单层适配器的表现优于完全微调的模型。但是，使用问答设置的完全微调方法对于处理混杂情况更有效。我们还系统地检查了错误情况，为分析仇恨模因所提出的结构化推理框架的稳健性和局限性提供了宝贵的见解。

Title: Counterfactual Samples Constructing and Training for Commonsense Statements Estimation

• Authors: Chong Liu, Zaiwen Feng, Lin Liu, Zhenyun Deng, Jiuyong Li, Ruifang Zhai, Debo Cheng, Li Qin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.20563
• Pdf URL: https://arxiv.org/pdf/2412.20563
• Copy Paste: [[2412.20563]] Counterfactual Samples Constructing and Training for Commonsense Statements Estimation(https://arxiv.org/abs/2412.20563)
• Keywords: language model, llm
• Abstract: Plausibility Estimation (PE) plays a crucial role for enabling language models to objectively comprehend the real world. While large language models (LLMs) demonstrate remarkable capabilities in PE tasks but sometimes produce trivial commonsense errors due to the complexity of commonsense knowledge. They lack two key traits of an ideal PE model: a) Language-explainable: relying on critical word segments for decisions, and b) Commonsense-sensitive: detecting subtle linguistic variations in commonsense. To address these issues, we propose a novel model-agnostic method, referred to as Commonsense Counterfactual Samples Generating (CCSG). By training PE models with CCSG, we encourage them to focus on critical words, thereby enhancing both their language-explainable and commonsense-sensitive capabilities. Specifically, CCSG generates counterfactual samples by strategically replacing key words and introducing low-level dropout within sentences. These counterfactual samples are then incorporated into a sentence-level contrastive training framework to further enhance the model's learning process. Experimental results across nine diverse datasets demonstrate the effectiveness of CCSG in addressing commonsense reasoning challenges, with our CCSG method showing 3.07% improvement against the SOTA methods.
• 摘要：可信度评估 (PE) 在语言模型客观理解现实世界方面发挥着至关重要的作用。大型语言模型 (LLM) 在 PE 任务中表现出色，但由于常识知识的复杂性，有时会产生微不足道的常识错误。它们缺乏理想 PE 模型的两个关键特征：a) 语言可解释：依靠关键词段进行决策，b) 常识敏感：检测常识中细微的语言变化。为了解决这些问题，我们提出了一种新颖的模型无关方法，称为常识反事实样本生成 (CCSG)。通过使用 CCSG 训练 PE 模型，我们鼓励它们关注关键词，从而增强它们的语言可解释性和常识敏感能力。具体而言，CCSG 通过策略性地替换关键词并在句子中引入低级 dropout 来生成反事实样本。然后将这些反事实样本纳入句子级对比训练框架，以进一步增强模型的学习过程。在九个不同数据集上的实验结果证明了 CCSG 在解决常识推理挑战方面的有效性，我们的 CCSG 方法比 SOTA 方法提高了 3.07%。

Title: Towards Neural No-Resource Language Translation: A Comparative Evaluation of Approaches

• Authors: Madhavendra Thakur
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.20584
• Pdf URL: https://arxiv.org/pdf/2412.20584
• Copy Paste: [[2412.20584]] Towards Neural No-Resource Language Translation: A Comparative Evaluation of Approaches(https://arxiv.org/abs/2412.20584)
• Keywords: language model, llm, prompt
• Abstract: No-resource languages - those with minimal or no digital representation - pose unique challenges for machine translation (MT). Unlike low-resource languages, which rely on limited but existent corpora, no-resource languages often have fewer than 100 sentences available for training. This work explores the problem of no-resource translation through three distinct workflows: fine-tuning of translation-specific models, in-context learning with large language models (LLMs) using chain-of-reasoning prompting, and direct prompting without reasoning. Using Owens Valley Paiute as a case study, we demonstrate that no-resource translation demands fundamentally different approaches from low-resource scenarios, as traditional approaches to machine translation, such as those that work for low-resource languages, fail. Empirical results reveal that, although traditional approaches fail, the in-context learning capabilities of general-purpose large language models enable no-resource language translation that outperforms low-resource translation approaches and rivals human translations (BLEU 0.45-0.6); specifically, chain-of-reasoning prompting outperforms other methods for larger corpora, while direct prompting exhibits advantages in smaller datasets. As these approaches are language-agnostic, they have potential to be generalized to translation tasks from a wide variety of no-resource languages without expert input. These findings establish no-resource translation as a distinct paradigm requiring innovative solutions, providing practical and theoretical insights for language preservation.
• 摘要：无资源语言（即数字表示极少或没有数字表示的语言）对机器翻译 (MT) 提出了独特的挑战。与依赖有限但存在的语料库的低资源语言不同，无资源语言通常只有不到 100 个句子可供训练。这项工作通过三个不同的工作流程探讨了无资源翻译的问题：翻译特定模型的微调、使用推理链提示的大型语言模型 (LLM) 的上下文学习以及没有推理的直接提示。以欧文斯谷派尤特 (Owens Valley Paiute) 为例，我们证明无资源翻译需要与低资源场景根本不同的方法，因为传统的机器翻译方法（例如适用于低资源语言的方法）会失败。实证结果表明，尽管传统方法失败了，但通用大型语言模型的上下文学习能力使无资源语言翻译优于低资源翻译方法并可与人工翻译相媲美 (BLEU 0.45-0.6)；具体而言，对于较大的语料库，推理链提示优于其他方法，而直接提示在较小的数据集中表现出优势。由于这些方法与语言无关，因此它们有可能推广到各种无资源语言的翻译任务，而无需专家输入。这些发现将无资源翻译确立为需要创新解决方案的独特范例，为语言保护提供了实践和理论见解。

Title: Controlling Out-of-Domain Gaps in LLMs for Genre Classification and Generated Text Detection

• Authors: Dmitri Roussinov, Serge Sharoff, Nadezhda Puchnina
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.20595
• Pdf URL: https://arxiv.org/pdf/2412.20595
• Copy Paste: [[2412.20595]] Controlling Out-of-Domain Gaps in LLMs for Genre Classification and Generated Text Detection(https://arxiv.org/abs/2412.20595)
• Keywords: language model, gpt, llm, chain-of-thought
• Abstract: This study demonstrates that the modern generation of Large Language Models (LLMs, such as GPT-4) suffers from the same out-of-domain (OOD) performance gap observed in prior research on pre-trained Language Models (PLMs, such as BERT). We demonstrate this across two non-topical classification tasks: 1) genre classification and 2) generated text detection. Our results show that when demonstration examples for In-Context Learning (ICL) come from one domain (e.g., travel) and the system is tested on another domain (e.g., history), classification performance declines significantly. To address this, we introduce a method that controls which predictive indicators are used and which are excluded during classification. For the two tasks studied here, this ensures that topical features are omitted, while the model is guided to focus on stylistic rather than content-based attributes. This approach reduces the OOD gap by up to 20 percentage points in a few-shot setup. Straightforward Chain-of-Thought (CoT) methods, used as the baseline, prove insufficient, while our approach consistently enhances domain transfer performance.
• 摘要：这项研究表明，现代大型语言模型 (LLM，如 GPT-4) 面临着先前对预训练语言模型 (PLM，如 BERT) 的研究中观察到的相同的域外 (OOD) 性能差距。我们通过两个非主题分类任务证明了这一点：1) 体裁分类和 2) 生成文本检测。我们的结果表明，当上下文学习 (ICL) 的演示示例来自一个领域 (例如旅行) 而系统在另一个领域 (例如历史) 上进行测试时，分类性能会显着下降。为了解决这个问题，我们引入了一种方法，可以控制在分类过程中使用哪些预测指标以及排除哪些预测指标。对于这里研究的两个任务，这可确保省略主题特征，同时引导模型关注风格而不是基于内容的属性。这种方法在小样本设置中将 OOD 差距缩小了多达 20 个百分点。作为基线的直接思路链 (CoT) 方法被证明是不够的，而我们的方法则不断提高域传输性能。

Title: NLP-based Regulatory Compliance -- Using GPT 4.0 to Decode Regulatory Documents

• Authors: Bimal Kumar, Dmitri Roussinov
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.20602
• Pdf URL: https://arxiv.org/pdf/2412.20602
• Copy Paste: [[2412.20602]] NLP-based Regulatory Compliance -- Using GPT 4.0 to Decode Regulatory Documents(https://arxiv.org/abs/2412.20602)
• Keywords: language model, gpt, llm
• Abstract: Large Language Models (LLMs) such as GPT-4.0 have shown significant promise in addressing the semantic complexities of regulatory documents, particularly in detecting inconsistencies and contradictions. This study evaluates GPT-4.0's ability to identify conflicts within regulatory requirements by analyzing a curated corpus with artificially injected ambiguities and contradictions, designed in collaboration with architects and compliance engineers. Using metrics such as precision, recall, and F1 score, the experiment demonstrates GPT-4.0's effectiveness in detecting inconsistencies, with findings validated by human experts. The results highlight the potential of LLMs to enhance regulatory compliance processes, though further testing with larger datasets and domain-specific fine-tuning is needed to maximize accuracy and practical applicability. Future work will explore automated conflict resolution and real-world implementation through pilot projects with industry partners.
• 摘要：大型语言模型 (LLM)（例如 GPT-4.0）在解决监管文件的语义复杂性方面显示出巨大的潜力，特别是在检测不一致和矛盾方面。这项研究通过分析与架构师和合规工程师合作设计的包含人工注入的歧义和矛盾的精选语料库来评估 GPT-4.0 识别监管要求中冲突的能力。实验使用精确度、召回率和 F1 分数等指标证明了 GPT-4.0 在检测不一致方面的有效性，结果得到了人类专家的验证。结果凸显了 LLM 在增强监管合规流程方面的潜力，但需要使用更大的数据集和特定领域的微调进行进一步测试，以最大限度地提高准确性和实际适用性。未来的工作将通过与行业合作伙伴的试点项目探索自动冲突解决和实际实施。

Title: Knowledge Editing for Large Language Model with Knowledge Neuronal Ensemble

• Authors: Yongchang Li, Yujin Zhu, Tao Yan, Shijian Fan, Gang Wu, Liang Xu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.20637
• Pdf URL: https://arxiv.org/pdf/2412.20637
• Copy Paste: [[2412.20637]] Knowledge Editing for Large Language Model with Knowledge Neuronal Ensemble(https://arxiv.org/abs/2412.20637)
• Keywords: language model
• Abstract: As real-world knowledge is constantly evolving, ensuring the timeliness and accuracy of a model's knowledge is crucial. This has made knowledge editing in large language models increasingly important. However, existing knowledge editing methods face several challenges, including parameter localization coupling, imprecise localization, and a lack of dynamic interaction across layers. In this paper, we propose a novel knowledge editing method called Knowledge Neuronal Ensemble (KNE). A knowledge neuronal ensemble represents a group of neurons encoding specific knowledge, thus mitigating the issue of frequent parameter modification caused by coupling in parameter localization. The KNE method enhances the precision and accuracy of parameter localization by computing gradient attribution scores for each parameter at each layer. During the editing process, only the gradients and losses associated with the knowledge neuronal ensemble are computed, with error backpropagation performed accordingly, ensuring dynamic interaction and collaborative updates among parameters. Experimental results on three widely used knowledge editing datasets show that the KNE method significantly improves the accuracy of knowledge editing and achieves, or even exceeds, the performance of the best baseline methods in portability and locality metrics.
• 摘要：随着现实世界知识的不断演化，保证模型知识的时效性和准确性至关重要。这使得大型语言模型中的知识编辑变得越来越重要。然而，现有的知识编辑方法面临着参数局部耦合、局部不精确以及跨层动态交互不足等挑战。在本文中，我们提出了一种新的知识编辑方法，称为知识神经元集合（KNE）。知识神经元集合代表一组编码特定知识的神经元，从而缓解了参数局部耦合导致的参数频繁修改问题。KNE 方法通过计算每层每个参数的梯度归因分数来提高参数局部化的精度和准确性。在编辑过程中，只计算与知识神经元集合相关的梯度和损失，并相应地进行误差反向传播，确保参数之间的动态交互和协同更新。在三个广泛使用的知识编辑数据集上的实验结果表明，KNE 方法显著提高了知识编辑的准确性，并在可移植性和局部性指标上达到甚至超过了最佳基线方法的性能。

Title: Align Attention Heads Before Merging Them: An Effective Way for Converting MHA to GQA

• Authors: Qingyun Jin, Xiaohui Song, Feng Zhou, Zengchang Qin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.20677
• Pdf URL: https://arxiv.org/pdf/2412.20677
• Copy Paste: [[2412.20677]] Align Attention Heads Before Merging Them: An Effective Way for Converting MHA to GQA(https://arxiv.org/abs/2412.20677)
• Keywords: language model, llm
• Abstract: Large language models have been shown to perform well on a variety of natural language processing problems. However, as the model size and the input sequence's length increase, the rapid increase of KV Cache significantly slows down inference speed. Therefore GQA model, as an alternative to MHA model, has been widely introduced into LLMs. In this work, we propose a low-cost method for pruning MHA models into GQA models with any compression ratio of key-value heads. Our method is based on $\mathit{L_0}$ masks to gradually remove redundant parameters. In addition, we apply orthogonal transformations to attention heads without changing the model to increase similarity between attention heads before pruning training, in order to further improve performance of the model. Our method can be compatible with rotary position embedding (RoPE), which means the model after training can be fully adapted to the mainstream standard GQA framework. Experiments demonstrate that our strategy can compress up to 87.5% of key-value heads of the LLaMA2-7B model without too much performance degradation, just achieved through supervised fine-tuning.
• 摘要：大型语言模型已被证明在各种自然语言处理问题上表现良好。然而随着模型规模和输入序列长度的增加，KV Cache的快速增加显著减慢了推理速度。因此GQA模型作为MHA模型的替代品被广泛应用于LLM中。在本文中，我们提出了一种低成本的方法将MHA模型剪枝为具有任意压缩比的键值头的GQA模型。我们的方法基于$\mathit{L_0}$ mask逐步去除冗余参数。此外，我们在不改变模型的情况下对注意力头应用正交变换以在剪枝训练前增加注意力头之间的相似性，以进一步提升模型的性能。我们的方法可以兼容旋转位置嵌入（RoPE），这意味着训练后的模型可以完全适配主流的标准GQA框架。实验表明，我们的策略可以压缩高达 LLaMA2-7B 模型的 87.5% 的键值头，而不会造成太多的性能损失，这仅通过监督微调即可实现。

Title: Depression and Anxiety Prediction Using Deep Language Models and Transfer Learning

• Authors: Tomasz Rutowski, Elizabeth Shriberg, Amir Harati, Yang Lu, Piotr Chlebek, Ricardo Oliveira
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.20741
• Pdf URL: https://arxiv.org/pdf/2412.20741
• Copy Paste: [[2412.20741]] Depression and Anxiety Prediction Using Deep Language Models and Transfer Learning(https://arxiv.org/abs/2412.20741)
• Keywords: language model
• Abstract: Digital screening and monitoring applications can aid providers in the management of behavioral health conditions. We explore deep language models for detecting depression, anxiety, and their co-occurrence from conversational speech collected during 16k user interactions with an application. Labels come from PHQ-8 and GAD-7 results also collected by the application. We find that results for binary classification range from 0.86 to 0.79 AUC, depending on condition and co-occurrence. Best performance is achieved when a user has either both or neither condition, and we show that this result is not attributable to data skew. Finally, we find evidence suggesting that underlying word sequence cues may be more salient for depression than for anxiety.
• 摘要：数字筛查和监测应用程序可以帮助提供者管理行为健康状况。我们探索深度语言模型，从 16k 个用户与应用程序交互期间收集的对话语音中检测抑郁、焦虑及其共现。标签来自应用程序收集的 PHQ-8 和 GAD-7 结果。我们发现二元分类的结果范围从 0.86 到 0.79 AUC，具体取决于病情和共现。当用户同时患有两种疾病或两种疾病都没有时，性能最佳，我们表明此结果不是数据偏差造成的。最后，我们发现有证据表明，潜在的词序列线索对抑郁症的影响可能比对焦虑症的影响更大。

Title: Attributing Culture-Conditioned Generations to Pretraining Corpora

• Authors: Huihan Li, Arnav Goel, Keyu He, Xiang Ren
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.20760
• Pdf URL: https://arxiv.org/pdf/2412.20760
• Copy Paste: [[2412.20760]] Attributing Culture-Conditioned Generations to Pretraining Corpora(https://arxiv.org/abs/2412.20760)
• Keywords: language model
• Abstract: In open-ended generative tasks like narrative writing or dialogue, large language models often exhibit cultural biases, showing limited knowledge and generating templated outputs for less prevalent cultures. Recent works show that these biases may stem from uneven cultural representation in pretraining corpora. This work investigates how pretraining leads to biased culture-conditioned generations by analyzing how models associate entities with cultures based on pretraining data patterns. We propose the MEMOed framework (MEMOrization from pretraining document) to determine whether a generation for a culture arises from memorization. Using MEMOed on culture-conditioned generations about food and clothing for 110 cultures, we find that high-frequency cultures in pretraining data yield more generations with memorized symbols, while some low-frequency cultures produce none. Additionally, the model favors generating entities with extraordinarily high frequency regardless of the conditioned culture, reflecting biases toward frequent pretraining terms irrespective of relevance. We hope that the MEMOed framework and our insights will inspire more works on attributing model performance on pretraining data.
• 摘要：在叙事写作或对话等开放式生成任务中，大型语言模型通常表现出文化偏见，知识有限，并为不太流行的文化生成模板化输出。最近的研究表明，这些偏见可能源于预训练语料库中文化代表性的不均衡。这项研究通过分析模型如何根据预训练数据模式将实体与文化联系起来，研究了预训练如何导致有偏见的文化条件生成。我们提出了 MEMOed 框架（来自预训练文档的 MEMOrization）来确定文化的生成是否源于记忆。使用 MEMOed 对 110 种文化的关于食物和衣服的文化条件生成进行研究，我们发现预训练数据中的高频文化产生了更多具有记忆符号的生成，而一些低频文化则没有产生任何生成。此外，无论条件文化如何，该模型都倾向于生成具有极高频率的实体，这反映了对频繁预训练术语的偏见，而与相关性无关。我们希望 MEMOed 框架和我们的见解能够激发更多将模型性能归因于预训练数据的研究。

Title: Disentangling Preference Representation and Text Generation for Efficient Individual Preference Alignment

• Authors: Jianfei Zhang, Jun Bai, Bei Li, Yanmeng Wang, Rumei Li, Chenghua Lin, Wenge Rong
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.20834
• Pdf URL: https://arxiv.org/pdf/2412.20834
• Copy Paste: [[2412.20834]] Disentangling Preference Representation and Text Generation for Efficient Individual Preference Alignment(https://arxiv.org/abs/2412.20834)
• Keywords: language model, llm
• Abstract: Aligning Large Language Models (LLMs) with general human preferences has been proved crucial in improving the interaction quality between LLMs and human. However, human values are inherently diverse among different individuals, making it insufficient to align LLMs solely with general preferences. To address this, personalizing LLMs according to individual feedback emerges as a promising solution. Nonetheless, this approach presents challenges in terms of the efficiency of alignment algorithms. In this work, we introduce a flexible paradigm for individual preference alignment. Our method fundamentally improves efficiency by disentangling preference representation from text generation in LLMs. We validate our approach across multiple text generation tasks and demonstrate that it can produce aligned quality as well as or better than PEFT-based methods, while reducing additional training time for each new individual preference by $80\%$ to $90\%$ in comparison with them.
• 摘要：将大型语言模型 (LLM) 与人类的一般偏好对齐已被证明对于提高 LLM 与人类之间的交互质量至关重要。然而，人类价值观在不同的个体之间本质上是不同的，因此仅仅将 LLM 与一般偏好对齐是不够的。为了解决这个问题，根据个人反馈对 LLM 进行个性化是一个有前途的解决方案。尽管如此，这种方法在对齐算法的效率方面也带来了挑战。在这项工作中，我们引入了一种灵活的个人偏好对齐范式。我们的方法通过将偏好表示与 LLM 中的文本生成分离，从根本上提高了效率。我们在多个文本生成任务中验证了我们的方法，并证明它可以产生与基于 PEFT 的方法一样好甚至更好的对齐质量，同时与它们相比，将每个新个人偏好的额外训练时间减少了 $80\%$ 到 $90\%$。

Title: Are LLMs Really Not Knowledgable? Mining the Submerged Knowledge in LLMs' Memory

• Authors: Xingjian Tao, Yiwei Wang, Yujun Cai, Zhicheng Yang, Jing Tang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.20846
• Pdf URL: https://arxiv.org/pdf/2412.20846
• Copy Paste: [[2412.20846]] Are LLMs Really Not Knowledgable? Mining the Submerged Knowledge in LLMs' Memory(https://arxiv.org/abs/2412.20846)
• Keywords: language model, llm, hallucination
• Abstract: Large language models (LLMs) have shown promise as potential knowledge bases, yet they often struggle with question-answering tasks and are prone to hallucinations. While previous research attributes these issues to knowledge gaps in the model's parameters, our investigation reveals a different phenomenon: LLMs often retain correct knowledge even when generating incorrect answers. Through analysis of model's internal representations, we find that correct answers frequently appear among high-probability tokens despite not being selected as final outputs. Based on this observation, we introduce Hits@k, a new metric to assess knowledge retention independent of expression accuracy. Our extensive experiments demonstrate that LLMs store significantly more knowledge than their QA performance suggests. Building on these findings, we develop SkipUnsure, a method to improve answer accuracy by leveraging detected but unexpressed knowledge. Experiments on both open-domain and specific-domain datasets show consistent improvements, with accuracy gains of up to 11.8% on DBPedia and 6.3% on IMDB, without requiring model retraining.
• 摘要：大型语言模型 (LLM) 已显示出作为潜在知识库的潜力，但它们在问答任务中往往举步维艰，而且容易产生幻觉。虽然先前的研究将这些问题归因于模型参数中的知识差距，但我们的研究揭示了一个不同的现象：即使生成错误的答案，LLM 也经常保留正确的知识。通过分析模型的内部表示，我们发现正确答案经常出现在高概率标记中，尽管没有被选为最终输出。基于这一观察，我们引入了 Hits@k，这是一种独立于表达准确性来评估知识保留的新指标。我们进行了广泛的实验，结果表明 LLM 存储的知识比其 QA 性能所显示的要多得多。基于这些发现，我们开发了 SkipUnsure，这是一种通过利用检测到但未表达的知识来提高答案准确性的方法。在开放域和特定域数据集上进行的实验都显示出持续的改进，DBPedia 上的准确率提高了 11.8%，IMDB 上的准确率提高了 6.3%，而无需重新训练模型。

Title: Enhancing Annotated Bibliography Generation with LLM Ensembles

• Authors: Sergio Bermejo
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.20864
• Pdf URL: https://arxiv.org/pdf/2412.20864
• Copy Paste: [[2412.20864]] Enhancing Annotated Bibliography Generation with LLM Ensembles(https://arxiv.org/abs/2412.20864)
• Keywords: language model, llm
• Abstract: This work proposes a novel approach to enhancing annotated bibliography generation through Large Language Model (LLM) ensembles. In particular, multiple LLMs in different roles -- controllable text generation, evaluation, and summarization -- are introduced and validated using a systematic methodology to enhance model performance in scholarly tasks. Output diversity among the ensemble that generates text is obtained using different LLM parameters, followed by an LLM acting as a judge to assess relevance, accuracy, and coherence. Responses selected by several combining strategies are then merged and refined through summarization and redundancy removal techniques. The preliminary experimental validation demonstrates that the combined outputs from the LLM ensemble improve coherence and relevance compared to individual responses, leading to a 38% improvement in annotation quality and a 51% reduction in content redundancy, thus highlighting the potential for automating complex scholarly tasks while maintaining high-quality standards.
• 摘要：这项研究提出了一种通过大型语言模型 (LLM) 集成来增强带注释书目生成的新方法。具体来说，引入了多个具有不同角色（可控文本生成、评估和摘要）的 LLM，并使用系统方法对其进行了验证，以提高模型在学术任务中的性能。使用不同的 LLM 参数获得生成文本的集成之间的输出多样性，然后由 LLM 充当评判者来评估相关性、准确性和连贯性。然后通过摘要和冗余消除技术合并和细化由几种组合策略选择的响应。初步实验验证表明，与单个响应相比，LLM 集成的组合输出提高了连贯性和相关性，从而使注释质量提高了 38%，内容冗余减少了 51%，从而凸显了在保持高质量标准的同时自动化复杂学术任务的潜力。

Title: DoTA: Weight-Decomposed Tensor Adaptation for Large Language Models

• Authors: Xiaolin Hu, Xiang Cheng, Peiyu Liu, Wei Liu, Jian Luan, Bin Wang, Yong Liu
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.20891
• Pdf URL: https://arxiv.org/pdf/2412.20891
• Copy Paste: [[2412.20891]] DoTA: Weight-Decomposed Tensor Adaptation for Large Language Models(https://arxiv.org/abs/2412.20891)
• Keywords: language model, llm
• Abstract: Low-rank adaptation (LoRA) reduces the computational and memory demands of fine-tuning large language models (LLMs) by approximating updates with low-rank matrices. However, low-rank approximation in two-dimensional space fails to capture high-dimensional structures within the target matrix. Recently, tensor decomposition methods have been explored for fine-tuning LLMs, leveraging their ability to extract structured information. Yet, these approaches primarily rely on random initialization, and the impact of initialization on tensor adaptation remains underexplored. In this paper, we reveal that random initialization significantly diverges from the validation loss achieved by full fine-tuning. To address this, we propose Weight-Decomposed Tensor Adaptation (DoTA), which leverages the Matrix Product Operator (MPO) decomposition of pre-trained weights for effective initialization in fine-tuning LLMs. Additionally, we introduce QDoTA, a quantized version of DoTA designed for 4-bit quantization. Experiments on commonsense and arithmetic reasoning tasks show that DoTA outperforms random initialization methods with fewer parameters. QDoTA further reduces memory consumption and achieves comparable performance to DoTA on commonsense reasoning tasks. We will release our code to support future research.
• 摘要：低秩自适应 (LoRA) 通过使用低秩矩阵近似更新来减少微调大型语言模型 (LLM) 的计算和内存需求。然而，二维空间中的低秩近似无法捕获目标矩阵内的高维结构。最近，张量分解方法已被用于微调 LLM，利用其提取结构化信息的能力。然而，这些方法主要依赖于随机初始化，而初始化对张量自适应的影响仍未得到充分探索。在本文中，我们发现随机初始化与完全微调实现的验证损失存在显著差异。为了解决这个问题，我们提出了权重分解张量自适应 (DoTA)，它利用预训练权重的矩阵积算子 (MPO) 分解来有效初始化微调 LLM。此外，我们引入了 QDoTA，这是 DoTA 的量化版本，专为 4 位量化而设计。在常识和算术推理任务上的实验表明，DoTA 优于参数较少的随机初始化方法。QDoTA 进一步降低了内存消耗，并在常识推理任务上实现了与 DoTA 相当的性能。我们将发布我们的代码以支持未来的研究。

Title: KARPA: A Training-free Method of Adapting Knowledge Graph as References for Large Language Model's Reasoning Path Aggregation

• Authors: Siyuan Fang, Kaijing Ma, Tianyu Zheng, Xinrun Du, Ningxuan Lu, Ge Zhang, Qingkun Tang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.20995
• Pdf URL: https://arxiv.org/pdf/2412.20995
• Copy Paste: [[2412.20995]] KARPA: A Training-free Method of Adapting Knowledge Graph as References for Large Language Model's Reasoning Path Aggregation(https://arxiv.org/abs/2412.20995)
• Keywords: language model, llm, hallucination
• Abstract: Large language models (LLMs) demonstrate exceptional performance across a variety of tasks, yet they are often affected by hallucinations and the timeliness of knowledge. Leveraging knowledge graphs (KGs) as external knowledge sources has emerged as a viable solution, but existing methods for LLM-based knowledge graph question answering (KGQA) are often limited by step-by-step decision-making on KGs, restricting the global planning and reasoning capabilities of LLMs, or they require fine-tuning or pre-training on specific KGs. To address these challenges, we propose Knowledge graph Assisted Reasoning Path Aggregation (KARPA), a novel framework that harnesses the global planning abilities of LLMs for efficient and accurate KG reasoning. KARPA operates in three steps: pre-planning relation paths using the LLM's global planning capabilities, matching semantically relevant paths via an embedding model, and reasoning over these paths to generate answers. Unlike existing KGQA methods, KARPA avoids stepwise traversal, requires no additional training, and is adaptable to various LLM architectures. Extensive experimental results show that KARPA achieves state-of-the-art performance in KGQA tasks, delivering both high efficiency and accuracy. Our code will be available on Github.
• 摘要：大型语言模型（LLM）在各种任务中都表现出色，但它们常常受到幻觉和知识时效性的影响。利用知识图谱（KG）作为外部知识源已成为一种可行的解决方案，但现有的基于 LLM 的知识图谱问答（KGQA）方法往往受限于对 KG 进行分步决策，限制了 LLM 的全局规划和推理能力，或者需要对特定的 KG 进行微调或预训练。为了应对这些挑战，我们提出了知识图谱辅助推理路径聚合（KARPA），这是一个新颖的框架，它利用 LLM 的全局规划能力实现高效、准确的 KG 推理。KARPA 分为三个步骤：使用 LLM 的全局规划功能预先规划关系路径、通过嵌入模型匹配语义相关路径、以及对这些路径进行推理以生成答案。与现有的 KGQA 方法不同，KARPA 避免了逐步遍历，不需要额外的训练，并且适用于各种 LLM 架构。大量实验结果表明，KARPA 在 KGQA 任务中实现了最佳性能，效率和准确性都很高。我们的代码将在 Github 上提供。

Title: Plug-and-Play Training Framework for Preference Optimization

• Authors: Jingyuan Ma, Rui Li, Zheng Li, Lei Sha, Zhifang Sui
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.20996
• Pdf URL: https://arxiv.org/pdf/2412.20996
• Copy Paste: [[2412.20996]] Plug-and-Play Training Framework for Preference Optimization(https://arxiv.org/abs/2412.20996)
• Keywords: language model, llm
• Abstract: Recently, preference optimization methods such as DPO have significantly enhanced large language models (LLMs) in wide tasks including dialogue and question-answering. However, current methods fail to account for the varying difficulty levels of training samples during preference optimization, leading to mediocre performance in tasks with high accuracy requirements, particularly in mathematical reasoning. To address this limitation, we propose a novel training framework, which employs multiple sampling to analyze output distributions, assign different weights to samples, and incorporate these weights into the preference optimization process. This plug-and-play approach enables LLMs to prioritize challenging examples during training, improving learning efficiency. Experimental results demonstrate that our framework integrates seamlessly with various preference optimization methods and achieves consistent improvements in mathematical reasoning tasks.
• 摘要：最近，DPO 等偏好优化方法在对话、问答等广泛任务中显著增强了大型语言模型 (LLM)。然而，当前方法在偏好优化过程中未能考虑训练样本的不同难度级别，导致其在对准确度要求较高的任务中表现平平，尤其是在数学推理中。为了解决这一限制，我们提出了一种新颖的训练框架，该框架采用多重采样来分析输出分布，为样本分配不同的权重，并将这些权重纳入偏好优化过程。这种即插即用的方法使 LLM 能够在训练期间优先考虑具有挑战性的示例，从而提高学习效率。实验结果表明，我们的框架可以与各种偏好优化方法无缝集成，并在数学推理任务中实现持续改进。

Title: Verbosity-Aware Rationale Reduction: Effective Reduction of Redundant Rationale via Principled Criteria

• Authors: Joonwon Jang, Jaehee Kim, Wonbin Kweon, Hwanjo Yu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.21006
• Pdf URL: https://arxiv.org/pdf/2412.21006
• Copy Paste: [[2412.21006]] Verbosity-Aware Rationale Reduction: Effective Reduction of Redundant Rationale via Principled Criteria(https://arxiv.org/abs/2412.21006)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) rely on generating extensive intermediate reasoning units (e.g., tokens, sentences) to enhance final answer quality across a wide range of complex tasks. While generating multiple reasoning paths or iteratively refining rationales proves effective for improving performance, these approaches inevitably result in significantly higher inference costs. In this work, we propose a novel sentence-level rationale reduction training framework that leverages likelihood-based criteria, verbosity, to identify and remove redundant reasoning sentences. Unlike previous approaches that utilize token-level reduction, our sentence-level reduction framework maintains model performance while reducing generation length. This preserves the original reasoning abilities of LLMs and achieves an average 17.15% reduction in generation costs across various models and tasks.
• 摘要：大型语言模型 (LLM) 依赖于生成大量中间推理单元（例如，标记、句子）来提高各种复杂任务的最终答案质量。虽然生成多个推理路径或迭代细化原理已被证明可以有效提高性能，但这些方法不可避免地会导致推理成本显著增加。在这项工作中，我们提出了一种新颖的句子级原理减少训练框架，该框架利用基于可能性的标准、冗长程度来识别和删除冗余的推理句子。与以前利用标记级减少的方法不同，我们的句子级减少框架在减少生成长度的同时保持了模型性能。这保留了 LLM 的原始推理能力，并在各种模型和任务中实现了生成成本平均降低 17.15%。

Title: MapQaTor: A System for Efficient Annotation of Map Query Datasets

• Authors: Mahir Labib Dihan, Mohammed Eunus Ali, Md Rizwan Parvez
• Subjects: cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2412.21015
• Pdf URL: https://arxiv.org/pdf/2412.21015
• Copy Paste: [[2412.21015]] MapQaTor: A System for Efficient Annotation of Map Query Datasets(https://arxiv.org/abs/2412.21015)
• Keywords: language model, llm
• Abstract: Mapping and navigation services like Google Maps, Apple Maps, Openstreet Maps, are essential for accessing various location-based data, yet they often struggle to handle natural language geospatial queries. Recent advancements in Large Language Models (LLMs) show promise in question answering (QA), but creating reliable geospatial QA datasets from map services remains challenging. We introduce MapQaTor, a web application that streamlines the creation of reproducible, traceable map-based QA datasets. With its plug-and-play architecture, MapQaTor enables seamless integration with any maps API, allowing users to gather and visualize data from diverse sources with minimal setup. By caching API responses, the platform ensures consistent ground truth, enhancing the reliability of the data even as real-world information evolves. MapQaTor centralizes data retrieval, annotation, and visualization within a single platform, offering a unique opportunity to evaluate the current state of LLM-based geospatial reasoning while advancing their capabilities for improved geospatial understanding. Evaluation metrics show that, MapQaTor speeds up the annotation process by at least 30 times compared to manual methods, underscoring its potential for developing geospatial resources, such as complex map reasoning datasets. The website is live at: this https URL and a demo video is available at: this https URL.
• 摘要：地图和导航服务（例如 Google 地图、Apple 地图、Openstreet 地图）对于访问各种基于位置的数据至关重要，但它们通常难以处理自然语言地理空间查询。大型语言模型 (LLM) 的最新进展在问答 (QA) 方面显示出前景，但从地图服​​务创建可靠的地理空间 QA 数据集仍然具有挑战性。我们推出了 MapQaTor，这是一款 Web 应用程序，可简化可重现、可跟踪的基于地图的 QA 数据集的创建。凭借其即插即用架构，MapQaTor 可以与任何地图 API 无缝集成，使用户能够以最少的设置收集和可视化来自不同来源的数据。通过缓存 API 响应，该平台可确保一致的事实，即使在现实世界信息不断发展的情况下也能提高数据的可靠性。MapQaTor 将数据检索、注释和可视化集中在一个平台内，提供了一个独特的机会来评估基于 LLM 的地理空间推理的当前状态，同时提高其改善地理空间理解的能力。评估指标显示，与手动方法相比，MapQaTor 将注释过程速度提高了至少 30 倍，凸显了其开发地理空间资源（例如复杂地图推理数据集）的潜力。该网站的网址为：此 https URL，演示视频可从：此 https URL 获取。

Title: Plancraft: an evaluation dataset for planning with LLM agents

• Authors: Gautier Dagan, Frank Keller, Alex Lascarides
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.21033
• Pdf URL: https://arxiv.org/pdf/2412.21033
• Copy Paste: [[2412.21033]] Plancraft: an evaluation dataset for planning with LLM agents(https://arxiv.org/abs/2412.21033)
• Keywords: llm, retrieval augmented generation, agent
• Abstract: We present Plancraft, a multi-modal evaluation dataset for LLM agents. Plancraft has both a text-only and multi-modal interface, based on the Minecraft crafting GUI. We include the Minecraft Wiki to evaluate tool use and Retrieval Augmented Generation (RAG), as well as an oracle planner and oracle RAG information extractor, to ablate the different components of a modern agent architecture. To evaluate decision-making, Plancraft also includes a subset of examples that are intentionally unsolvable, providing a realistic challenge that requires the agent not only to complete tasks but also to decide whether they are solvable at all. We benchmark both open-source and closed-source LLMs and strategies on our task and compare their performance to a handcrafted planner. We find that LLMs and VLMs struggle with the planning problems that Plancraft introduces, and we offer suggestions on how to improve their capabilities.
• 摘要：我们介绍了 Plancraft，这是一个针对 LLM 代理的多模态评估数据集。Plancraft 具有纯文本和多模态界面，基于 Minecraft 制作 GUI。我们包括 Minecraft Wiki 来评估工具使用和检索增强生成 (RAG)，以及 oracle 规划器和 oracle RAG 信息提取器，以消除现代代理架构的不同组件。为了评估决策，Plancraft 还包括一组故意无法解决的示例，这提供了一个现实的挑战，要求代理不仅要完成任务，还要决定这些任务是否可以解决。我们在我们的任务上对开源和闭源 LLM 和策略进行了基准测试，并将它们的性能与手工制作的规划器进行了比较。我们发现 LLM 和 VLM 在解决 Plancraft 引入的规划问题时遇到了困难，并就如何改进它们的能力提出了建议。

Title: GePBench: Evaluating Fundamental Geometric Perception for Multimodal Large Language Models

• Authors: Shangyu Xing, Changhao Xiang, Yuteng Han, Yifan Yue, Zhen Wu, Xinyu Liu, Zhangtai Wu, Fei Zhao, Xinyu Dai
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.21036
• Pdf URL: https://arxiv.org/pdf/2412.21036
• Copy Paste: [[2412.21036]] GePBench: Evaluating Fundamental Geometric Perception for Multimodal Large Language Models(https://arxiv.org/abs/2412.21036)
• Keywords: language model, llm
• Abstract: Multimodal large language models (MLLMs) have achieved significant advancements in integrating visual and linguistic understanding. While existing benchmarks evaluate these models in context-rich, real-life scenarios, they often overlook fundamental perceptual skills essential for environments deviating from everyday realism. In particular, geometric perception, the ability to interpret spatial relationships and abstract visual patterns, remains underexplored. To address this limitation, we introduce GePBench, a novel benchmark designed to assess the geometric perception capabilities of MLLMs. Results from extensive evaluations reveal that current state-of-the-art MLLMs exhibit significant deficiencies in such tasks. Additionally, we demonstrate that models trained with data sourced from GePBench show notable improvements on a wide range of downstream tasks, underscoring the importance of geometric perception as a foundation for advanced multimodal applications. Our code and datasets will be publicly available.
• 摘要：多模态大型语言模型 (MLLM) 在整合视觉和语言理解方面取得了重大进展。虽然现有的基准测试在丰富的现实场景中评估这些模型，但它们往往忽略了偏离日常现实的环境所必需的基本感知技能。特别是几何感知，即解释空间关系和抽象视觉模式的能力，仍未得到充分探索。为了解决这一限制，我们引入了 GePBench，这是一种旨在评估 MLLM 几何感知能力的新型基准测试。大量评估的结果表明，目前最先进的 MLLM 在这些任务中表现出显著的缺陷。此外，我们证明使用来自 GePBench 的数据训练的模型在广泛的下游任务中表现出显着的改进，强调了几何感知作为高级多模态应用基础的重要性。我们的代码和数据集将公开。

Title: Efficient Multi-Task Inferencing with a Shared Backbone and Lightweight Task-Specific Adapters for Automatic Scoring

• Authors: Ehsan Latif, Xiaoming Zhai
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.21065
• Pdf URL: https://arxiv.org/pdf/2412.21065
• Copy Paste: [[2412.21065]] Efficient Multi-Task Inferencing with a Shared Backbone and Lightweight Task-Specific Adapters for Automatic Scoring(https://arxiv.org/abs/2412.21065)
• Keywords: language model
• Abstract: The integration of Artificial Intelligence (AI) in education requires scalable and efficient frameworks that balance performance, adaptability, and cost. This paper addresses these needs by proposing a shared backbone model architecture enhanced with lightweight LoRA adapters for task-specific fine-tuning, targeting the automated scoring of student responses across 27 mutually exclusive tasks. By achieving competitive performance (average QWK of 0.848 compared to 0.888 for fully fine-tuned models) while reducing GPU memory consumption by 60% and inference latency by 40%, the framework demonstrates significant efficiency gains. This approach aligns with the workshops' focus on improving language models for educational tasks, creating responsible innovations for cost-sensitive deployment, and supporting educators by streamlining assessment workflows. The findings underscore the potential of scalable AI to enhance learning outcomes while maintaining fairness and transparency in automated scoring systems.
• 摘要：人工智能 (AI) 与教育的融合需要可扩展且高效的框架，以平衡性能、适应性和成本。本文提出了一种共享主干模型架构来满足这些需求，该架构增强了轻量级 LoRA 适配器，用于特定任务的微调，目标是自动对 27 个互斥任务中的学生答案进行评分。通过实现具有竞争力的性能（平均 QWK 为 0.848，而完全微调模型为 0.888），同时将 GPU 内存消耗降低 60%，推理延迟降低 40%，该框架显示出显着的效率提升。这种方法与研讨会的重点一致，即改进教育任务的语言模型，为成本敏感的部署创造负责任的创新，并通过简化评估工作流程来支持教育工作者。研究结果强调了可扩展人工智能在提高学习成果的同时保持自动评分系统公平性和透明度的潜力。

Title: Exploring and Controlling Diversity in LLM-Agent Conversation

• Authors: KuanChao Chu, Yi-Pei Chen, Hideki Nakayama
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.21102
• Pdf URL: https://arxiv.org/pdf/2412.21102
• Copy Paste: [[2412.21102]] Exploring and Controlling Diversity in LLM-Agent Conversation(https://arxiv.org/abs/2412.21102)
• Keywords: llm, prompt, agent
• Abstract: Diversity is a critical aspect of multi-agent communication. In this paper, we focus on controlling and exploring diversity in the context of open-domain multi-agent conversations, particularly for world simulation applications. We propose Adaptive Prompt Pruning (APP), a novel method that dynamically adjusts the content of the utterance generation prompt to control diversity using a single parameter, lambda. Through extensive experiments, we show that APP effectively controls the output diversity across models and datasets, with pruning more information leading to more diverse output. We comprehensively analyze the relationship between prompt content and conversational diversity. Our findings reveal that information from all components of the prompt generally constrains the diversity of the output, with the Memory block exerting the most significant influence. APP is compatible with established techniques like temperature sampling and top-p sampling, providing a versatile tool for diversity management. To address the trade-offs of increased diversity, such as inconsistencies with omitted information, we incorporate a post-generation correction step, which effectively balances diversity enhancement with output consistency. Additionally, we examine how prompt structure, including component order and length, impacts diversity. This study addresses key questions surrounding diversity in multi-agent world simulation, offering insights into its control, influencing factors, and associated trade-offs. Our contributions lay the foundation for systematically engineering diversity in LLM-based multi-agent collaborations, advancing their effectiveness in real-world applications.
• 摘要：多样性是多智能体通信的一个关键方面。在本文中，我们专注于在开放域多智能体对话的背景下控制和探索多样性，特别是对于世界模拟应用。我们提出了自适应提示修剪 (APP)，这是一种新颖的方法，它使用单个参数 lambda 动态调整话语生成提示的内容以控制多样性。通过大量实验，我们表明 APP 可以有效地控制跨模型和数据集的输出多样性，修剪更多信息可带来更多样化的输出。我们全面分析了提示内容与对话多样性之间的关系。我们的研究结果表明，来自提示所有组成部分的信息通常会限制输出的多样性，其中记忆块的影响最大。APP 与温度采样和 top-p 采样等成熟技术兼容，为多样性管理提供了一种多功能工具。为了解决增加多样性的权衡，例如与省略信息的不一致，我们加入了一个生成后校正步骤，有效地平衡了多样性增强与输出一致性。此外，我们还研究了提示结构（包括组件顺序和长度）如何影响多样性。本研究探讨了多智能体世界模拟中多样性的关键问题，并提供了对其控制、影响因素和相关权衡的见解。我们的贡献为系统地设计基于 LLM 的多智能体协作中的多样性奠定了基础，从而提高了其在实际应用中的有效性。

Title: Facilitating large language model Russian adaptation with Learned Embedding Propagation

• Authors: Mikhail Tikhomirov, Daniil Chernyshev
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.21140
• Pdf URL: https://arxiv.org/pdf/2412.21140
• Copy Paste: [[2412.21140]] Facilitating large language model Russian adaptation with Learned Embedding Propagation(https://arxiv.org/abs/2412.21140)
• Keywords: language model, gpt, llm, chat
• Abstract: Rapid advancements of large language model (LLM) technologies led to the introduction of powerful open-source instruction-tuned LLMs that have the same text generation quality as the state-of-the-art counterparts such as GPT-4. While the emergence of such models accelerates the adoption of LLM technologies in sensitive-information environments the authors of such models don not disclose the training data necessary for replication of the results thus making the achievements model-exclusive. Since those open-source models are also multilingual this in turn reduces the benefits of training a language specific LLMs as improved inference computation efficiency becomes the only guaranteed advantage of such costly procedure. More cost-efficient options such as vocabulary extension and subsequent continued pre-training are also inhibited by the lack of access to high-quality instruction-tuning data since it is the major factor behind the resulting LLM task-solving capabilities. To address the limitations and cut the costs of the language adaptation pipeline we propose Learned Embedding Propagation (LEP). Unlike existing approaches our method has lower training data size requirements due to minimal impact on existing LLM knowledge which we reinforce using novel ad-hoc embedding propagation procedure that allows to skip the instruction-tuning step and instead implant the new language knowledge directly into any existing instruct-tuned variant. We evaluated four Russian vocabulary adaptations for LLaMa-3-8B and Mistral-7B, showing that LEP is competitive with traditional instruction-tuning methods, achieving performance comparable to OpenChat 3.5 and LLaMa-3-8B-Instruct, with further improvements via self-calibration and continued tuning enhancing task-solving capabilities.
• 摘要：大型语言模型 (LLM) 技术的快速发展导致了功能强大的开源指令调整 LLM 的引入，这些 LLM 具有与 GPT-4 等最先进的同类产品相同的文本生成质量。虽然此类模型的出现加速了 LLM 技术在敏感信息环境中的采用，但此类模型的作者并未披露复制结果所需的训练数据，从而使成果成为模型独有的。由于这些开源模型也是多语言的，这反过来又降低了训练特定语言 LLM 的好处，因为提高推理计算效率成为这种昂贵程序的唯一保证优势。词汇扩展和随后的持续预训练等更具成本效益的选项也因缺乏对高质量指令调整数据的访问而受到抑制，因为这是导致 LLM 任务解决能力的主要因素。为了解决限制并降低语言自适应管道的成本，我们提出了学习嵌入传播 (LEP)。与现有方法不同，我们的方法对训练数据大小的要求较低，因为对现有 LLM 知识的影响很小，我们使用新颖的临时嵌入传播程序来强化这些知识，该程序允许跳过指令调整步骤，而是将新的语言知识直接植入任何现有的指令调整变体中。我们评估了 LLaMa-3-8B 和 Mistral-7B 的四种俄语词汇改编，结果表明 LEP 与传统指令调整方法相比具有竞争力，实现了与 OpenChat 3.5 和 LLaMa-3-8B-Instruct 相当的性能，并通过自我校准和持续调整进一步改进，增强了任务解决能力。

Title: Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs

• Authors: Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, Dong Yu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.21187
• Pdf URL: https://arxiv.org/pdf/2412.21187
• Copy Paste: [[2412.21187]] Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs(https://arxiv.org/abs/2412.21187)
• Keywords: llm, chain-of-thought
• Abstract: The remarkable performance of models like the OpenAI o1 can be attributed to their ability to emulate human-like long-time thinking during inference. These models employ extended chain-of-thought (CoT) processes, exploring multiple strategies to enhance problem-solving capabilities. However, a critical question remains: How to intelligently and efficiently scale computational resources during testing. This paper presents the first comprehensive study on the prevalent issue of overthinking in these models, where excessive computational resources are allocated for simple problems with minimal benefit. We introduce novel efficiency metrics from both outcome and process perspectives to evaluate the rational use of computational resources by o1-like models. Using a self-training paradigm, we propose strategies to mitigate overthinking, streamlining reasoning processes without compromising accuracy. Experimental results show that our approach successfully reduces computational overhead while preserving model performance across a range of testsets with varying difficulty levels, such as GSM8K, MATH500, GPQA, and AIME.
• 摘要：OpenAI o1 等模型的出色表现可以归因于它们在推理过程中模拟人类的长期思考的能力。这些模型采用扩展的思路链 (CoT) 过程，探索多种策略来增强解决问题的能力。然而，一个关键问题仍然存在：如何在测试期间智能高效地扩展计算资源。本文首次全面研究了这些模型中普遍存在的过度思考问题，其中过多的计算资源被分配给简单问题，而收益却微乎其微。我们从结果和过程的角度引入了新的效率指标，以评估 o1 类模型对计算资源的合理使用。使用自训练范式，我们提出了缓解过度思考的策略，在不影响准确性的情况下简化推理过程。实验结果表明，我们的方法成功地减少了计算开销，同时在具有不同难度级别的一系列测试集（例如 GSM8K、MATH500、GPQA 和 AIME）中保持了模型性能。