2025-01-09

Title: A Survey on Large Language Models with some Insights on their Capabilities and Limitations

• Authors: Andrea Matarazzo, Riccardo Torlone
• Subjects: cs.CL, cs.AI, cs.LG, cs.NE
• Abstract URL: https://arxiv.org/abs/2501.04040
• Pdf URL: https://arxiv.org/pdf/2501.04040
• Copy Paste: [[2501.04040]] A Survey on Large Language Models with some Insights on their Capabilities and Limitations(https://arxiv.org/abs/2501.04040)
• Keywords: language model, gpt, llm
• Abstract: The rapid advancement of artificial intelligence, particularly with the development of Large Language Models (LLMs) built on the transformer architecture, has redefined the capabilities of natural language processing. These models now exhibit remarkable performance across various language-related tasks, such as text generation, question answering, translation, and summarization, often rivaling human-like comprehension. More intriguingly, LLMs have demonstrated emergent abilities extending beyond their core functions, showing proficiency in tasks like commonsense reasoning, code generation, and arithmetic. This survey paper explores the foundational components, scaling mechanisms, and architectural strategies that drive these capabilities. Emphasizing models like GPT and LLaMA, we analyze the impact of exponential data and computational growth on LLM performance, while also addressing the trade-offs associated with scaling. We also examine LLM applications across sectors, such as healthcare, finance, education, and law, highlighting their adaptability and potential to solve domain-specific challenges. Central to this work are the questions of how LLMs generalize across diverse tasks, exhibit planning, and reasoning abilities, and whether these emergent abilities can be systematically elicited or enhanced. In particular, we provide some insights into the CoT (Chain of Thought) and PoT (Plan of Thought) abilities within LLMs, focusing on how pre-training data influences their emergence. Additionally, we investigate LLM-modulo frameworks that integrate external systems, allowing LLMs to handle complex, dynamic tasks. By analyzing these factors, this paper aims to foster the ongoing discussion on the capabilities and limits of LLMs, promoting their responsible development and application in novel and increasingly complex environments.
• 摘要：人工智能的快速发展，尤其是基于 Transformer 架构的大型语言模型 (LLM) 的发展，重新定义了自然语言处理的功能。这些模型现在在各种语言相关任务中表现出色，例如文本生成、问答、翻译和总结，通常可与人类的理解能力相媲美。更有趣的是，LLM 已经展示了超越其核心功能的新兴能力，在常识推理、代码生成和算术等任务中表现出色。这篇综述论文探讨了推动这些功能的基础组件、扩展机制和架构策略。我们重点介绍了 GPT 和 LLaMA 等模型，分析了指数级数据和计算增长对 LLM 性能的影响，同时也解决了与扩展相关的权衡问题。我们还研究了医疗保健、金融、教育和法律等行业的 LLM 应用，强调了它们的适应性和解决特定领域挑战的潜力。这项工作的核心问题是 LLM 如何在各种任务中推广、展现规划和推理能力，以及这些新兴能力是否可以系统地引出或增强。特别是，我们对 LLM 中的 CoT（思维链）和 PoT（思维计划）能力提供了一些见解，重点关注预训练数据如何影响它们的出现。此外，我们研究了集成外部系统的 LLM 模块框架，使 LLM 能够处理复杂的动态任务。通过分析这些因素，本文旨在促进对 LLM 能力和局限性的持续讨论，促进其在新的、日益复杂的环境中负责任地开发和应用。

Title: "Yeah Right!" -- Do LLMs Exhibit Multimodal Feature Transfer?

• Authors: Benjamin Reichman, Kartik Talamadupula
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.04138
• Pdf URL: https://arxiv.org/pdf/2501.04138
• Copy Paste: [[2501.04138]] "Yeah Right!" -- Do LLMs Exhibit Multimodal Feature Transfer?(https://arxiv.org/abs/2501.04138)
• Keywords: llm, prompt
• Abstract: Human communication is a multifaceted and multimodal skill. Communication requires an understanding of both the surface-level textual content and the connotative intent of a piece of communication. In humans, learning to go beyond the surface level starts by learning communicative intent in speech. Once humans acquire these skills in spoken communication, they transfer those skills to written communication. In this paper, we assess the ability of speech+text models and text models trained with special emphasis on human-to-human conversations to make this multimodal transfer of skill. We specifically test these models on their ability to detect covert deceptive communication. We find that with no special prompting speech+text LLMs have an advantage over unimodal LLMs in performing this task. Likewise, we find that human-to-human conversation-trained LLMs are also advantaged in this skill.
• 摘要：人类交流是一项多方面和多模态的技能。交流需要理解交流的表面文本内容和内涵意图。对于人类来说，学习超越表面层次的交流始于学习言语中的交流意图。一旦人类掌握了口头交流的这些技能，他们就会将这些技能转移到书面交流中。在本文中，我们评估了语音+文本模型和特别强调人与人对话训练的文本模型进行这种多模态技能转移的能力。我们专门测试了这些模型检测隐蔽欺骗性交流的能力。我们发现，在没有特殊提示的情况下，语音+文本 LLM 在执行此任务方面比单模态 LLM 更有优势。同样，我们发现人与人对话训练的 LLM 也具有这项技能的优势。

Title: Multilingual Open QA on the MIA Shared Task

• Authors: Navya Yarrabelly, Saloni Mittal, Ketan Todi, Kimihiro Hasegawa
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.04153
• Pdf URL: https://arxiv.org/pdf/2501.04153
• Copy Paste: [[2501.04153]] Multilingual Open QA on the MIA Shared Task(https://arxiv.org/abs/2501.04153)
• Keywords: language model
• Abstract: Cross-lingual information retrieval (CLIR) ~\cite{shi2021cross, asai2021one, jiang2020cross} for example, can find relevant text in any language such as English(high resource) or Telugu (low resource) even when the query is posed in a different, possibly low-resource, language. In this work, we aim to develop useful CLIR models for this constrained, yet important, setting where we do not require any kind of additional supervision or labelled data for retrieval task and hence can work effectively for low-resource languages. \par We propose a simple and effective re-ranking method for improving passage retrieval in open question answering. The re-ranker re-scores retrieved passages with a zero-shot multilingual question generation model, which is a pre-trained language model, to compute the probability of the input question in the target language conditioned on a retrieved passage, which can be possibly in a different language. We evaluate our method in a completely zero shot setting and doesn't require any training. Thus the main advantage of our method is that our approach can be used to re-rank results obtained by any sparse retrieval methods like BM-25. This eliminates the need for obtaining expensive labelled corpus required for the retrieval tasks and hence can be used for low resource languages.
• 摘要：例如，跨语言信息检索 (CLIR) ~\cite{shi2021cross, asai2021one, jiang2020cross} 可以找到任何语言的相关文本，例如英语 (高资源) 或泰卢固语 (低资源)，即使查询是用不同的、可能资源较少的语言提出的。在这项工作中，我们旨在为这种受限但重要的设置开发有用的 CLIR 模型，在这种设置中，我们不需要任何类型的额外监督或标记数据来进行检索任务，因此可以有效地用于低资源语言。\par 我们提出了一种简单有效的重新排名方法来改进开放式问答中的段落检索。重新排名器使用零样本多语言问题生成模型（这是一种预先训练的语言模型）对检索到的段落重新评分，以计算在检索到的段落条件下输入问题在目标语言中的概率，该段落可能是不同的语言。我们在完全零样本设置中评估我们的方法，不需要任何训练。因此，我们方法的主要优势在于，我们的方法可用于对任何稀疏检索方法（如 BM-25）获得的结果进行重新排序。这样就无需获取检索任务所需的昂贵标记语料库，因此可用于资源匮乏的语言。

Title: Reasoning-Enhanced Self-Training for Long-Form Personalized Text Generation

• Authors: Alireza Salemi, Cheng Li, Mingyang Zhang, Qiaozhu Mei, Weize Kong, Tao Chen, Zhuowan Li, Michael Bendersky, Hamed Zamani
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2501.04167
• Pdf URL: https://arxiv.org/pdf/2501.04167
• Copy Paste: [[2501.04167]] Reasoning-Enhanced Self-Training for Long-Form Personalized Text Generation(https://arxiv.org/abs/2501.04167)
• Keywords: language model, llm
• Abstract: Personalized text generation requires a unique ability of large language models (LLMs) to learn from context that they often do not encounter during their standard training. One way to encourage LLMs to better use personalized context for generating outputs that better align with the user's expectations is to instruct them to reason over the user's past preferences, background knowledge, or writing style. To achieve this, we propose Reasoning-Enhanced Self-Training for Personalized Text Generation (REST-PG), a framework that trains LLMs to reason over personal data during response generation. REST-PG first generates reasoning paths to train the LLM's reasoning abilities and then employs Expectation-Maximization Reinforced Self-Training to iteratively train the LLM based on its own high-reward outputs. We evaluate REST-PG on the LongLaMP benchmark, consisting of four diverse personalized long-form text generation tasks. Our experiments demonstrate that REST-PG achieves significant improvements over state-of-the-art baselines, with an average relative performance gain of 14.5% on the benchmark.
• 摘要：个性化文本生成需要大型语言模型 (LLM) 具备独特的能力，能够从它们在标准训练中通常不会遇到的上下文中学习。鼓励 LLM 更好地利用个性化上下文来生成更符合用户期望的输出的一种方法是指示它们推理用户过去的偏好、背景知识或写作风格。为了实现这一点，我们提出了推理增强型个性化文本生成自训练 (REST-PG)，这是一个训练 LLM 在响应生成期间推理个人数据的框架。REST-PG 首先生成推理路径来训练 LLM 的推理能力，然后采用期望最大化强化自训练根据其自身的高奖励输出迭代训练 LLM。我们在 LongLaMP 基准上评估 REST-PG，该基准由四个不同的个性化长文本生成任务组成。我们的实验表明，REST-PG 比最先进的基线有了显着的改进，基准上的平均相对性能提高了 14.5%。

Title: IOLBENCH: Benchmarking LLMs on Linguistic Reasoning

• Authors: Satyam Goyal, Soham Dan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.04249
• Pdf URL: https://arxiv.org/pdf/2501.04249
• Copy Paste: [[2501.04249]] IOLBENCH: Benchmarking LLMs on Linguistic Reasoning(https://arxiv.org/abs/2501.04249)
• Keywords: language model, llm
• Abstract: Despite the remarkable advancements and widespread applications of deep neural networks, their ability to perform reasoning tasks remains limited, particularly in domains requiring structured, abstract thought. In this paper, we investigate the linguistic reasoning capabilities of state-of-the-art large language models (LLMs) by introducing IOLBENCH, a novel benchmark derived from International Linguistics Olympiad (IOL) problems. This dataset encompasses diverse problems testing syntax, morphology, phonology, and semantics, all carefully designed to be self-contained and independent of external knowledge. These tasks challenge models to engage in metacognitive linguistic reasoning, requiring the deduction of linguistic rules and patterns from minimal examples. Through extensive benchmarking of leading LLMs, we find that even the most advanced models struggle to handle the intricacies of linguistic complexity, particularly in areas demanding compositional generalization and rule abstraction. Our analysis highlights both the strengths and persistent limitations of current models in linguistic problem-solving, offering valuable insights into their reasoning capabilities. By introducing IOLBENCH, we aim to foster further research into developing models capable of human-like reasoning, with broader implications for the fields of computational linguistics and artificial intelligence.
• 摘要：尽管深度神经网络取得了显著进步并得到广泛应用，但它们执行推理任务的能力仍然有限，特别是在需要结构化、抽象思维的领域。在本文中，我们通过引入 IOLBENCH（一种源自国际语言学奥林匹克 (IOL) 问题的新基准）来研究最先进的大型语言模型 (LLM) 的语言推理能力。该数据集包含测试语法、形态、音系和语义的各种问题，所有这些都经过精心设计，以自成体系并独立于外部知识。这些任务挑战模型进行元认知语言推理，需要从最少的例子中推断出语言规则和模式。通过对领先的 LLM 进行广泛的基准测试，我们发现即使是最先进的模型也难以处理语言复杂性的复杂性，特别是在需要组合概括和规则抽象的领域。我们的分析突出了当前模型在语言问题解决方面的优势和持续存在的局限性，为其推理能力提供了宝贵的见解。通过引入 IOLBENCH，我们旨在促进进一步研究开发具有类似人类推理能力的模型，这对计算语言学和人工智能领域具有更广泛的影响。

Title: Multimodal Graph Constrastive Learning and Prompt for ChartQA

• Authors: Yue Dai, Soyeon Caren Han, Wei Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.04303
• Pdf URL: https://arxiv.org/pdf/2501.04303
• Copy Paste: [[2501.04303]] Multimodal Graph Constrastive Learning and Prompt for ChartQA(https://arxiv.org/abs/2501.04303)
• Keywords: language model, llm, hallucination, prompt, chain-of-thought
• Abstract: ChartQA presents significant challenges due to the complex distribution of chart elements and the implicit patterns embedded within the underlying data. In this chapter, we have developed a joint multimodal scene graph for charts, explicitly representing the relationships between chart elements and their associated patterns. Our proposed multimodal scene graph consists of two components: a visual graph and a textual graph, each designed to capture the structural and semantic information within the chart. To unify representations across these different modalities, we introduce a multimodal graph contrastive learning approach that learns unified representations by maximizing similarity between nodes representing the same object across multimodal graphs. The learned graph representations can be seamlessly incorporated into a transformer decoder as a soft prompt. Additionally, given the growing need for Multimodal Large Language Models (MLLMs) in zero-shot scenarios, we have designed Chain-of-Thought (CoT) prompts for MLLMs to reduce hallucinations. We tested both methods on public benchmarks such as ChartQA, OpenCQA, and ChartX, demonstrating improved performance and validating the effectiveness of our proposed methods.
• 摘要：ChartQA 面临着巨大的挑战，因为图表元素的分布很复杂，底层数据中隐含着隐含的模式。在本章中，我们为图表开发了一个联合多模态场景图，明确表示图表元素与其相关模式之间的关系。我们提出的多模态场景图由两个组件组成：一个可视化图和一个文本图，每个组件都旨在捕获图表中的结构和语义信息。为了统一这些不同模态之间的表示，我们引入了一种多模态图对比学习方法，该方法通过最大化多模态图中表示同一对象的节点之间的相似性来学习统一表示。学习到的图表示可以无缝地作为软提示合并到 Transformer 解码器中。此外，鉴于零样本场景中对多模态大型语言模型 (MLLM) 的需求日益增长，我们为 MLLM 设计了思想链 (CoT) 提示以减少幻觉。我们在 ChartQA、OpenCQA 和 ChartX 等公共基准上测试了这两种方法，证明了性能的提高并验证了我们提出的方法的有效性。

Title: LLM4SR: A Survey on Large Language Models for Scientific Research

• Authors: Ziming Luo, Zonglin Yang, Zexin Xu, Wei Yang, Xinya Du
• Subjects: cs.CL, cs.DL
• Abstract URL: https://arxiv.org/abs/2501.04306
• Pdf URL: https://arxiv.org/pdf/2501.04306
• Copy Paste: [[2501.04306]] LLM4SR: A Survey on Large Language Models for Scientific Research(https://arxiv.org/abs/2501.04306)
• Keywords: language model, llm
• Abstract: In recent years, the rapid advancement of Large Language Models (LLMs) has transformed the landscape of scientific research, offering unprecedented support across various stages of the research cycle. This paper presents the first systematic survey dedicated to exploring how LLMs are revolutionizing the scientific research process. We analyze the unique roles LLMs play across four critical stages of research: hypothesis discovery, experiment planning and implementation, scientific writing, and peer reviewing. Our review comprehensively showcases the task-specific methodologies and evaluation benchmarks. By identifying current challenges and proposing future research directions, this survey not only highlights the transformative potential of LLMs, but also aims to inspire and guide researchers and practitioners in leveraging LLMs to advance scientific inquiry. Resources are available at the following repository: this https URL
• 摘要：近年来，大型语言模型 (LLM) 的快速发展改变了科学研究的格局，为研究周期的各个阶段提供了前所未有的支持。本文首次系统地探讨了 LLM 如何彻底改变科学研究过程。我们分析了 LLM 在研究的四个关键阶段所发挥的独特作用：假设发现、实验规划和实施、科学写作和同行评审。我们的评论全面展示了特定任务的方法和评估基准。通过确定当前的挑战并提出未来的研究方向，本调查不仅突出了 LLM 的变革潜力，而且旨在激励和指导研究人员和从业者利用 LLM 推进科学研究。资源可在以下存储库中找到：此 https URL

Title: Who Does the Giant Number Pile Like Best: Analyzing Fairness in Hiring Contexts

• Authors: Preethi Seshadri, Seraphina Goldfarb-Tarrant
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.04316
• Pdf URL: https://arxiv.org/pdf/2501.04316
• Copy Paste: [[2501.04316]] Who Does the Giant Number Pile Like Best: Analyzing Fairness in Hiring Contexts(https://arxiv.org/abs/2501.04316)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) are increasingly being deployed in high-stakes applications like hiring, yet their potential for unfair decision-making and outcomes remains understudied, particularly in generative settings. In this work, we examine the fairness of LLM-based hiring systems through two real-world tasks: resume summarization and retrieval. By constructing a synthetic resume dataset and curating job postings, we investigate whether model behavior differs across demographic groups and is sensitive to demographic perturbations. Our findings reveal that race-based differences appear in approximately 10% of generated summaries, while gender-based differences occur in only 1%. In the retrieval setting, all evaluated models display non-uniform selection patterns across demographic groups and exhibit high sensitivity to both gender and race-based perturbations. Surprisingly, retrieval models demonstrate comparable sensitivity to non-demographic changes, suggesting that fairness issues may stem, in part, from general brittleness issues. Overall, our results indicate that LLM-based hiring systems, especially at the retrieval stage, can exhibit notable biases that lead to discriminatory outcomes in real-world contexts.
• 摘要：大型语言模型 (LLM) 越来越多地被部署在招聘等高风险应用中，但它们在决策和结果不公平方面的可能性仍未得到充分研究，尤其是在生成环境中。在这项工作中，我们通过两个现实任务来检验基于 LLM 的招聘系统的公平性：简历摘要和检索。通过构建合成简历数据集和整理招聘信息，我们调查了模型行为是否因人口统计学群体而异，是否对人口统计学扰动敏感。我们的研究结果表明，大约 10% 的生成摘要中存在种族差异，而只有 1% 存在性别差异。在检索环境中，所有评估的模型都显示出跨人口统计学群体的非均匀选择模式，并且对性别和种族扰动都表现出高度敏感性。令人惊讶的是，检索模型对非人口统计学变化表现出类似的敏感性，这表明公平性问题可能部分源于普遍的脆弱性问题。总体而言，我们的结果表明，基于 LLM 的招聘系统（尤其是在检索阶段）可能会表现出明显的偏见，从而导致现实世界中出现歧视性的结果。

Title: Understanding Before Reasoning: Enhancing Chain-of-Thought with Iterative Summarization Pre-Prompting

• Authors: Dong-Hai Zhu, Yu-Jie Xiong, Jia-Chen Zhang, Xi-Jiong Xie, Chun-Ming Xia
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.04341
• Pdf URL: https://arxiv.org/pdf/2501.04341
• Copy Paste: [[2501.04341]] Understanding Before Reasoning: Enhancing Chain-of-Thought with Iterative Summarization Pre-Prompting(https://arxiv.org/abs/2501.04341)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: Chain-of-Thought (CoT) Prompting is a dominant paradigm in Large Language Models (LLMs) to enhance complex reasoning. It guides LLMs to present multi-step reasoning, rather than generating the final answer directly. However, CoT encounters difficulties when key information required for reasoning is implicit or missing. This occurs because CoT emphasizes the sequence of reasoning steps while overlooking the early extraction of essential information. We propose a pre-prompting method called Iterative Summarization Pre-Prompting (ISP^2) to refine LLM reasoning when key information is not explicitly provided. First, entities and their corresponding descriptions are extracted to form potential key information pairs. Next, we use a reliability rating to assess these pairs, then merge the two lowest-ranked pairs into a new entity description. This process is repeated until a unique key information pair is obtained. Finally, that pair, along with the original question, is fed into LLMs to produce the answer. Extensive experiments demonstrate a 7.1% improvement compared to existing methods. Unlike traditional prompting, ISP^2 adopts an inductive approach with pre-prompting, offering flexible integration into diverse reasoning frameworks. The code is available at this https URL.
• 摘要：思路链 (CoT) 提示是大型语言模型 (LLM) 中用于增强复杂推理的主要范式。它引导 LLM 呈现多步骤推理，而不是直接生成最终答案。然而，当推理所需的关键信息隐含或缺失时，CoT 会遇到困难。发生这种情况的原因是 CoT 强调推理步骤的顺序，而忽略了关键信息的早期提取。我们提出了一种称为迭代总结预提示 (ISP^2) 的预提示方法，用于在未明确提供关键信息时改进 LLM 推理。首先，提取实体及其相应的描述以形成潜在的关键信息对。接下来，我们使用可靠性评级来评估这些对，然后将两个排名最低的对合并为一个新的实体描述。重复此过程，直到获得唯一的关键信息对。最后，将该对与原始问题一起输入 LLM 以产生答案。大量实验表明，与现有方法相比，该方法的效率提高了 7.1%。与传统提示不同，ISP^2 采用预先提示的归纳方法，可灵活地集成到各种推理框架中。代码可在此 https URL 上获取。

Title: SEO: Stochastic Experience Optimization for Large Language Models

• Authors: Jitao Xu, Hongyun Zhou, Lei Shen, Conghui Zhu, Jin Huang, Yitao Duan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.04393
• Pdf URL: https://arxiv.org/pdf/2501.04393
• Copy Paste: [[2501.04393]] SEO: Stochastic Experience Optimization for Large Language Models(https://arxiv.org/abs/2501.04393)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) can benefit from useful experiences to improve their performance on specific tasks. However, finding helpful experiences for different LLMs is not obvious, since it is unclear what experiences suit specific LLMs. Previous studies intended to automatically find useful experiences using LLMs, while it is difficult to ensure the effectiveness of the obtained experience. In this paper, we propose Stochastic Experience Optimization (SEO), an iterative approach that finds optimized model-specific experience without modifying model parameters through experience update in natural language. In SEO, we propose a stochastic validation method to ensure the update direction of experience, avoiding unavailing updates. Experimental results on three tasks for three LLMs demonstrate that experiences optimized by SEO can achieve consistently improved performance. Further analysis indicates that SEO-optimized experience can generalize to out-of-distribution data, boosting the performance of LLMs on similar tasks.
• 摘要：大型语言模型 (LLM) 可以从有用的经验中受益，以提高其在特定任务上的性能。然而，为不同的 LLM 找到有用的经验并不明显，因为不清楚哪些经验适合特定的 LLM。先前的研究旨在使用 LLM 自动查找有用的经验，但很难确保获得的经验的有效性。在本文中，我们提出了随机经验优化 (SEO)，这是一种迭代方法，它通过自然语言中的经验更新来找到优化的特定于模型的经验，而无需修改模型参数。在 SEO 中，我们提出了一种随机验证方法来确保经验的更新方向，避免无效的更新。在三个 LLM 的三个任务上的实验结果表明，经过 SEO 优化的经验可以实现持续的性能提升。进一步的分析表明，SEO 优化的经验可以推广到分布外的数据，从而提高 LLM 在类似任务上的性能。

Title: End-to-End Bangla AI for Solving Math Olympiad Problem Benchmark: Leveraging Large Language Model Using Integrated Approach

• Authors: H.M. Shadman Tabib, Jaber Ahmed Deedar
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.04425
• Pdf URL: https://arxiv.org/pdf/2501.04425
• Copy Paste: [[2501.04425]] End-to-End Bangla AI for Solving Math Olympiad Problem Benchmark: Leveraging Large Language Model Using Integrated Approach(https://arxiv.org/abs/2501.04425)
• Keywords: language model, llm, prompt, retrieval-augmented generation
• Abstract: This work introduces systematic approach for enhancing large language models (LLMs) to address Bangla AI mathematical challenges. Through the assessment of diverse LLM configurations, fine-tuning with specific datasets, and the implementation of Retrieval-Augmented Generation (RAG), we enhanced the model's reasoning precision in a multilingual setting. Crucial discoveries indicate that customized prompting, dataset augmentation, and iterative reasoning improve the model's efficiency regarding Olympiad-level mathematical challenges.
• 摘要：这项研究引入了增强大型语言模型 (LLM) 的系统方法，以解决孟加拉语 AI 数学难题。通过评估不同的 LLM 配置、使用特定数据集进行微调以及实施检索增强生成 (RAG)，我们提高了模型在多语言环境中的推理精度。关键发现表明，定制提示、数据集增强和迭代推理提高了模型在奥林匹克级数学挑战方面的效率。

Title: Hidden Entity Detection from GitHub Leveraging Large Language Models

• Authors: Lu Gan, Martin Blum, Danilo Dessi, Brigitte Mathiak, Ralf Schenkel, Stefan Dietze
• Subjects: cs.CL, cs.DL
• Abstract URL: https://arxiv.org/abs/2501.04455
• Pdf URL: https://arxiv.org/pdf/2501.04455
• Copy Paste: [[2501.04455]] Hidden Entity Detection from GitHub Leveraging Large Language Models(https://arxiv.org/abs/2501.04455)
• Keywords: language model, llm, prompt
• Abstract: Named entity recognition is an important task when constructing knowledge bases from unstructured data sources. Whereas entity detection methods mostly rely on extensive training data, Large Language Models (LLMs) have paved the way towards approaches that rely on zero-shot learning (ZSL) or few-shot learning (FSL) by taking advantage of the capabilities LLMs acquired during pretraining. Specifically, in very specialized scenarios where large-scale training data is not available, ZSL / FSL opens new opportunities. This paper follows this recent trend and investigates the potential of leveraging Large Language Models (LLMs) in such scenarios to automatically detect datasets and software within textual content from GitHub repositories. While existing methods focused solely on named entities, this study aims to broaden the scope by incorporating resources such as repositories and online hubs where entities are also represented by URLs. The study explores different FSL prompt learning approaches to enhance the LLMs' ability to identify dataset and software mentions within repository texts. Through analyses of LLM effectiveness and learning strategies, this paper offers insights into the potential of advanced language models for automated entity detection.
• 摘要：命名实体识别是从非结构化数据源构建知识库时的重要任务。虽然实体检测方法主要依赖于大量训练数据，但大型语言模型 (LLM) 利用 LLM 在预训练期间获得的能力，为依赖零样本学习 (ZSL) 或少样本学习 (FSL) 的方法铺平了道路。具体来说，在没有大规模训练数据的非常特殊的场景中，ZSL/FSL 开辟了新的机会。本文遵循这一最新趋势，并研究在这种情况下利用大型语言模型 (LLM) 自动检测来自 GitHub 存储库的文本内容中的数据集和软件的潜力。虽然现有方法仅关注命名实体，但本研究旨在通过整合存储库和在线中心等资源来扩大范围，其中实体也由 URL 表示。该研究探索了不同的 FSL 提示学习方法，以增强 LLM 识别存储库文本中数据集和软件提及的能力。通过分析 LLM 的有效性和学习策略，本文深入了解了高级语言模型在自动实体检测方面的潜力。

Title: When LLMs Struggle: Reference-less Translation Evaluation for Low-resource Languages

• Authors: Archchana Sindhujan, Diptesh Kanojia, Constantin Orasan, Shenbin Qian
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.04473
• Pdf URL: https://arxiv.org/pdf/2501.04473
• Copy Paste: [[2501.04473]] When LLMs Struggle: Reference-less Translation Evaluation for Low-resource Languages(https://arxiv.org/abs/2501.04473)
• Keywords: language model, llm, prompt
• Abstract: This paper investigates the reference-less evaluation of machine translation for low-resource language pairs, known as quality estimation (QE). Segment-level QE is a challenging cross-lingual language understanding task that provides a quality score (0-100) to the translated output. We comprehensively evaluate large language models (LLMs) in zero/few-shot scenarios and perform instruction fine-tuning using a novel prompt based on annotation guidelines. Our results indicate that prompt-based approaches are outperformed by the encoder-based fine-tuned QE models. Our error analysis reveals tokenization issues, along with errors due to transliteration and named entities, and argues for refinement in LLM pre-training for cross-lingual tasks. We release the data, and models trained publicly for further research.
• 摘要：本文研究了针对低资源语言对的机器翻译的无参考评估，即质量评估 (QE)。句段级 QE 是一项具有挑战性的跨语言理解任务，可为翻译输出提供质量分数 (0-100)。我们全面评估零样本/少样本场景中的大型语言模型 (LLM)，并使用基于注释指南的新提示执行指令微调。我们的结果表明，基于提示的方法优于基于编码器的微调 QE 模型。我们的错误分析揭示了标记化问题以及音译和命名实体导致的错误，并主张改进跨语言任务的 LLM 预训练。我们发布了数据和公开训练的模型以供进一步研究。

Title: rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking

• Authors: Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, Mao Yang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.04519
• Pdf URL: https://arxiv.org/pdf/2501.04519
• Copy Paste: [[2501.04519]] rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking(https://arxiv.org/abs/2501.04519)
• Keywords: language model, llm
• Abstract: We present rStar-Math to demonstrate that small language models (SLMs) can rival or even surpass the math reasoning capability of OpenAI o1, without distillation from superior models. rStar-Math achieves this by exercising "deep thinking" through Monte Carlo Tree Search (MCTS), where a math policy SLM performs test-time search guided by an SLM-based process reward model. rStar-Math introduces three innovations to tackle the challenges in training the two SLMs: (1) a novel code-augmented CoT data sythesis method, which performs extensive MCTS rollouts to generate step-by-step verified reasoning trajectories used to train the policy SLM; (2) a novel process reward model training method that avoids naïve step-level score annotation, yielding a more effective process preference model (PPM); (3) a self-evolution recipe in which the policy SLM and PPM are built from scratch and iteratively evolved to improve reasoning capabilities. Through 4 rounds of self-evolution with millions of synthesized solutions for 747k math problems, rStar-Math boosts SLMs' math reasoning to state-of-the-art levels. On the MATH benchmark, it improves Qwen2.5-Math-7B from 58.8% to 90.0% and Phi3-mini-3.8B from 41.4% to 86.4%, surpassing o1-preview by +4.5% and +0.9%. On the USA Math Olympiad (AIME), rStar-Math solves an average of 53.3% (8/15) of problems, ranking among the top 20% the brightest high school math students. Code and data will be available at this https URL.
• 摘要：我们提出 rStar-Math 来证明小型语言模型 (SLM) 可以与 OpenAI o1 的数学推理能力相媲美甚至超越它，而无需从优秀模型中进行提炼。rStar-Math 通过蒙特卡洛树搜索 (MCTS) 进行“深度思考”来实现这一点，其中数学策略 SLM 在基于 SLM 的过程奖励模型的指导下执行测试时搜索。rStar-Math 引入了三项创新来应对训练两个 SLM 的挑战：(1) 一种新颖的代码增强 CoT 数据合成方法，该方法执行广泛的 MCTS 部署以生成用于训练策略 SLM 的经过验证的逐步推理轨迹；(2) 一种新颖的过程奖励模型训练方法，避免了简单的步骤级分数注释，从而产生了更有效的过程偏好模型 (PPM)；(3) 一种自我进化配方，其中从头开始构建策略 SLM 和 PPM 并迭代进化以提高推理能力。通过 4 轮自我进化，rStar-Math 为 747k 数学问题合成了数百万个解决方案，将 SLM 的数学推理能力提升到了最先进的水平。在 MATH 基准测试中，它将 Qwen2.5-Math-7B 从 58.8% 提高到 90.0%，将 Phi3-mini-3.8B 从 41.4% 提高到 86.4%，分别比 o1-preview 高出 4.5% 和 0.9%。在美国数学奥林匹克竞赛 (AIME) 中，rStar-Math 平均解决了 53.3% (8/15) 的问题，跻身最聪明的高中数学学生的前 20%。代码和数据将在此 https URL 上提供。

Title: OpenOmni: Large Language Models Pivot Zero-shot Omnimodal Alignment across Language with Real-time Self-Aware Emotional Speech Synthesis

• Authors: Run Luo, Ting-En Lin, Haonan Zhang, Yuchuan Wu, Xiong Liu, Min Yang, Yongbin Li, Longze Chen, Jiaming Li, Lei Zhang, Yangyi Chen, Hamid Alinejad-Rokny, Fei Huang
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2501.04561
• Pdf URL: https://arxiv.org/pdf/2501.04561
• Copy Paste: [[2501.04561]] OpenOmni: Large Language Models Pivot Zero-shot Omnimodal Alignment across Language with Real-time Self-Aware Emotional Speech Synthesis(https://arxiv.org/abs/2501.04561)
• Keywords: language model
• Abstract: Recent advancements in omnimodal learning have been achieved in understanding and generation across images, text, and speech, though mainly within proprietary models. Limited omnimodal datasets and the inherent challenges associated with real-time emotional speech generation have hindered open-source progress. To address these issues, we propose openomni, a two-stage training method combining omnimodal alignment and speech generation to develop a state-of-the-art omnimodal large language model. In the alignment phase, a pre-trained speech model is further trained on text-image tasks to generalize from vision to speech in a (near) zero-shot manner, outperforming models trained on tri-modal datasets. In the speech generation phase, a lightweight decoder facilitates real-time emotional speech through training on speech tasks and preference learning. Experiments demonstrate that openomni consistently improves across omnimodal, vision-language, and speech-language evaluations, enabling natural, emotion-rich dialogues and real-time emotional speech generation.
• 摘要：全模态学习的最新进展体现在对图像、文本和语音的理解和生成方面，尽管主要集中在专有模型中。有限的全模态数据集和实时情感语音生成所固有的挑战阻碍了开源进程。为了解决这些问题，我们提出了 openomni，这是一种结合全模态对齐和语音生成的两阶段训练方法，旨在开发最先进的全模态大型语言模型。在对齐阶段，对预先训练的语音模型进行文本图像任务的进一步训练，以（接近）零样本方式从视觉推广到语音，其表现优于在三模态数据集上训练的模型。在语音生成阶段，轻量级解码器通过语音任务训练和偏好学习来促进实时情感语音。实验表明，openomni 在全模态、视觉语言和语音语言评估中持续改进，实现自然、情感丰富的对话和实时情感语音生成。

Title: Quantum-inspired Embeddings Projection and Similarity Metrics for Representation Learning

• Authors: Ivan Kankeu, Stefan Gerd Fritsch, Gunnar Schönhoff, Elie Mounzer, Paul Lukowicz, Maximilian Kiefer-Emmanouilidis
• Subjects: cs.CL, cond-mat.dis-nn, quant-ph
• Abstract URL: https://arxiv.org/abs/2501.04591
• Pdf URL: https://arxiv.org/pdf/2501.04591
• Copy Paste: [[2501.04591]] Quantum-inspired Embeddings Projection and Similarity Metrics for Representation Learning(https://arxiv.org/abs/2501.04591)
• Keywords: language model
• Abstract: Over the last decade, representation learning, which embeds complex information extracted from large amounts of data into dense vector spaces, has emerged as a key technique in machine learning. Among other applications, it has been a key building block for large language models and advanced computer vision systems based on contrastive learning. A core component of representation learning systems is the projection head, which maps the original embeddings into different, often compressed spaces, while preserving the similarity relationship between vectors. In this paper, we propose a quantum-inspired projection head that includes a corresponding quantum-inspired similarity metric. Specifically, we map classical embeddings onto quantum states in Hilbert space and introduce a quantum circuit-based projection head to reduce embedding dimensionality. To evaluate the effectiveness of this approach, we extended the BERT language model by integrating our projection head for embedding compression. We compared the performance of embeddings, which were compressed using our quantum-inspired projection head, with those compressed using a classical projection head on information retrieval tasks using the TREC 2019 and TREC 2020 Deep Learning benchmarks. The results demonstrate that our quantum-inspired method achieves competitive performance relative to the classical method while utilizing 32 times fewer parameters. Furthermore, when trained from scratch, it notably excels, particularly on smaller datasets. This work not only highlights the effectiveness of the quantum-inspired approach but also emphasizes the utility of efficient, ad hoc low-entanglement circuit simulations within neural networks as a powerful quantum-inspired technique.
• 摘要：在过去十年中，表征学习已成为机器学习的一项关键技术，它将从大量数据中提取的复杂信息嵌入到密集向量空间中。除其他应用外，它还是基于对比学习的大型语言模型和高级计算机视觉系统的关键构建块。表征学习系统的核心组件是投影头，它将原始嵌入映射到不同的、通常是压缩的空间中，同时保留向量之间的相似关系。在本文中，我们提出了一种量子启发投影头，其中包括相应的量子启发相似度度量。具体而言，我们将经典嵌入映射到希尔伯特空间中的量子态上，并引入基于量子电路的投影头以降低嵌入维数。为了评估这种方法的有效性，我们通过集成投影头进行嵌入压缩来扩展 BERT 语言模型。我们使用 TREC 2019 和 TREC 2020 深度学习基准，比较了使用量子启发投影头压缩的嵌入与使用经典投影头压缩的嵌入在信息检索任务上的性能。结果表明，我们的量子启发方法相对于经典方法实现了具有竞争力的性能，同时使用的参数减少了 32 倍。此外，当从头开始训练时，它表现出色，尤其是在较小的数据集上。这项工作不仅突出了量子启发方法的有效性，而且还强调了在神经网络中高效、临时的低纠缠电路模拟作为一种强大的量子启发技术的实用性。

Title: Multi-task retriever fine-tuning for domain-specific and efficient RAG

• Authors: Patrice Béchard, Orlando Marquez Ayala
• Subjects: cs.CL, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.04652
• Pdf URL: https://arxiv.org/pdf/2501.04652
• Copy Paste: [[2501.04652]] Multi-task retriever fine-tuning for domain-specific and efficient RAG(https://arxiv.org/abs/2501.04652)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: Retrieval-Augmented Generation (RAG) has become ubiquitous when deploying Large Language Models (LLMs), as it can address typical limitations such as generating hallucinated or outdated information. However, when building real-world RAG applications, practical issues arise. First, the retrieved information is generally domain-specific. Since it is computationally expensive to fine-tune LLMs, it is more feasible to fine-tune the retriever to improve the quality of the data included in the LLM input. Second, as more applications are deployed in the same real-world system, one cannot afford to deploy separate retrievers. Moreover, these RAG applications normally retrieve different kinds of data. Our solution is to instruction fine-tune a small retriever encoder on a variety of domain-specific tasks to allow us to deploy one encoder that can serve many use cases, thereby achieving low-cost, scalability, and speed. We show how this encoder generalizes to out-of-domain settings as well as to an unseen retrieval task on real-world enterprise use cases.
• 摘要：在部署大型语言模型 (LLM) 时，检索增强生成 (RAG) 已变得无处不在，因为它可以解决典型的限制，例如生成幻觉或过时的信息。然而，在构建现实世界的 RAG 应用程序时，会出现实际问题。首先，检索到的信息通常是特定于领域的。由于微调 LLM 的计算成本很高，因此微调检索器以提高 LLM 输入中包含的数据的质量更为可行。其次，随着越来越多的应用程序部署在同一个现实世界系统中，人们无法承担部署单独的检索器的费用。此外，这些 RAG 应用程序通常会检索不同类型的数据。我们的解决方案是针对各种特定领域的任务对小型检索器编码器进行指令微调，以便我们部署一个可以服务于许多用例的编码器，从而实现低成本、可扩展性和速度。我们展示了此编码器如何推广到域外设置以及现实世界企业用例中看不见的检索任务。

Title: Assessing Language Comprehension in Large Language Models Using Construction Grammar

• Authors: Wesley Scivetti, Melissa Torgbi, Austin Blodgett, Mollie Shichman, Taylor Hudson, Claire Bonial, Harish Tayyar Madabushi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.04661
• Pdf URL: https://arxiv.org/pdf/2501.04661
• Copy Paste: [[2501.04661]] Assessing Language Comprehension in Large Language Models Using Construction Grammar(https://arxiv.org/abs/2501.04661)
• Keywords: language model, gpt, llm, prompt
• Abstract: Large Language Models, despite their significant capabilities, are known to fail in surprising and unpredictable ways. Evaluating their true `understanding' of language is particularly challenging due to the extensive web-scale data they are trained on. Therefore, we construct an evaluation to systematically assess natural language understanding (NLU) in LLMs by leveraging Construction Grammar (CxG), which provides insights into the meaning captured by linguistic elements known as constructions (Cxns). CxG is well-suited for this purpose because provides a theoretical basis to construct targeted evaluation sets. These datasets are carefully constructed to include examples which are unlikely to appear in pre-training data, yet intuitive and easy for humans to understand, enabling a more targeted and reliable assessment. Our experiments focus on downstream natural language inference and reasoning tasks by comparing LLMs' understanding of the underlying meanings communicated through 8 unique Cxns with that of humans. The results show that while LLMs demonstrate some knowledge of constructional information, even the latest models including GPT-o1 struggle with abstract meanings conveyed by these Cxns, as demonstrated in cases where test sentences are dissimilar to their pre-training data. We argue that such cases provide a more accurate test of true language understanding, highlighting key limitations in LLMs' semantic capabilities. We make our novel dataset and associated experimental data including prompts and model responses publicly available.
• 摘要：大型语言模型尽管功能强大，但其失败方式却出乎意料且难以预测。由于大型语言模型需要使用大量的网络规模数据进行训练，因此评估其对语言的真正“理解”尤其具有挑战性。因此，我们构建了一个评估系统，利用构​​造语法 (CxG) 系统地评估 LLM 中的自然语言理解 (NLU)，该语法可以深入了解语言元素（称为构造 (Cxns)）所捕获的含义。CxG 非常适合此目的，因为它为构建有针对性的评估集提供了理论基础。这些数据集经过精心构建，包括不太可能出现在预训练数据中的示例，但直观且易于人类理解，从而实现更有针对性和更可靠的评估。我们的实验侧重于下游自然语言推理和推理任务，通过将 LLM 对通过 8 个独特 Cxns 传达的底层含义的理解与人类进行比较。结果表明，虽然 LLM 展示了一些结构信息知识，但即使是包括 GPT-o1 在内的最新模型也难以理解这些 Cxns 所传达的抽象含义，正如测试句子与其预训练数据不同的情况下所证明的那样。我们认为，这种情况可以更准确地测试真正的语言理解能力，凸显了 LLM 语义能力的关键局限性。我们将我们的新数据集和相关实验数据（包括提示和模型响应）公开。

Title: On The Origin of Cultural Biases in Language Models: From Pre-training Data to Linguistic Phenomena

• Authors: Tarek Naous, Wei Xu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.04662
• Pdf URL: https://arxiv.org/pdf/2501.04662
• Copy Paste: [[2501.04662]] On The Origin of Cultural Biases in Language Models: From Pre-training Data to Linguistic Phenomena(https://arxiv.org/abs/2501.04662)
• Keywords: language model
• Abstract: Language Models (LMs) have been shown to exhibit a strong preference towards entities associated with Western culture when operating in non-Western languages. In this paper, we aim to uncover the origins of entity-related cultural biases in LMs by analyzing several contributing factors, including the representation of entities in pre-training data and the impact of variations in linguistic phenomena across languages. We introduce CAMeL-2, a parallel Arabic-English benchmark of 58,086 entities associated with Arab and Western cultures and 367 masked natural contexts for entities. Our evaluations using CAMeL-2 reveal reduced performance gaps between cultures by LMs when tested in English compared to Arabic. We find that LMs struggle in Arabic with entities that appear at high frequencies in pre-training, where entities can hold multiple word senses. This also extends to entities that exhibit high lexical overlap with languages that are not Arabic but use the Arabic script. Further, we show how frequency-based tokenization leads to this issue in LMs, which gets worse with larger Arabic vocabularies. We will make CAMeL-2 available at: this https URL
• 摘要：研究表明，语言模型 (LM) 在非西方语言中运行时对与西方文化相关的实体表现出强烈的偏好。在本文中，我们旨在通过分析几个促成因素来揭示 LM 中与实体相关的文化偏见的起源，包括实体在预训练数据中的表示以及不同语言中语言现象变化的影响。我们引入了 CAMeL-2，这是一个并行的阿拉伯语-英语基准，包含 58,086 个与阿拉伯和西方文化相关的实体以及 367 个掩码的实体自然环境。我们使用 CAMeL-2 进行的评估表明，与阿拉伯语相比，使用英语进行测试时，LM 在不同文化之间的性能差距有所缩小。我们发现，LM 在阿拉伯语中很难处理在预训练中出现频率较高的实体，其中实体可以包含多种词义。这也扩展到与非阿拉伯语但使用阿拉伯语脚本的语言表现出高度词汇重叠的实体。此外，我们展示了基于频率的标记化如何导致 LM 中的这个问题，阿拉伯语词汇量越大，这个问题就越严重。我们将在以下网址提供 CAMel-2：此 https URL

Title: Enhancing Financial VQA in Vision Language Models using Intermediate Structured Representations

• Authors: Archita Srivastava, Abhas Kumar, Rajesh Kumar, Prabhakar Srinivasan
• Subjects: cs.CL, cs.AI, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.04675
• Pdf URL: https://arxiv.org/pdf/2501.04675
• Copy Paste: [[2501.04675]] Enhancing Financial VQA in Vision Language Models using Intermediate Structured Representations(https://arxiv.org/abs/2501.04675)
• Keywords: language model, llm
• Abstract: Chart interpretation is crucial for visual data analysis, but accurately extracting information from charts poses significant challenges for automated models. This study investigates the fine-tuning of DEPLOT, a modality conversion module that translates the image of a plot or chart to a linearized table, on a custom dataset of 50,000 bar charts. The dataset comprises simple, stacked, and grouped bar charts, targeting the unique structural features of these visualizations. The finetuned DEPLOT model is evaluated against its base version using a test set of 1,000 images and two metrics: Relative Mapping Similarity (RMS), which measures categorical mapping accuracy, and Relative Number Set Similarity (RNSS), which evaluates numerical interpretation accuracy. To further explore the reasoning capabilities of large language models (LLMs), we curate an additional set of 100 bar chart images paired with question answer sets. Our findings demonstrate that providing a structured intermediate table alongside the image significantly enhances LLM reasoning performance compared to direct image queries.
• 摘要：图表解释对于可视化数据分析至关重要，但从图表中准确提取信息对自动化模型提出了重大挑战。本研究调查了 DEPLOT（一种模态转换模块，可将图表图像转换为线性化表格）在 50,000 个条形图的自定义数据集上的微调。该数据集包括简单、堆叠和分组条形图，针对这些可视化的独特结构特征。使用 1,000 张图像的测试集和两个指标（相对映射相似度 (RMS)，用于测量分类映射准确性）和相对数集相似度 (RNSS)，用于评估数值解释准确性，根据其基本版本对微调后的 DEPLOT 模型进行评估。为了进一步探索大型语言模型 (LLM) 的推理能力，我们策划了一组额外的 100 张条形图图像与问答集配对。我们的研究结果表明，与直接图像查询相比，在图像旁边提供结构化的中间表可显著提高 LLM 推理性能。

Title: URSA: Understanding and Verifying Chain-of-thought Reasoning in Multimodal Mathematics

• Authors: Ruilin Luo, Zhuofan Zheng, Yifan Wang, Yiyao Yu, Xinzhe Ni, Zicheng Lin, Jin Zeng, Yujiu Yang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.04686
• Pdf URL: https://arxiv.org/pdf/2501.04686
• Copy Paste: [[2501.04686]] URSA: Understanding and Verifying Chain-of-thought Reasoning in Multimodal Mathematics(https://arxiv.org/abs/2501.04686)
• Keywords: language model, llm, chain-of-thought
• Abstract: Chain-of-thought (CoT) reasoning has been widely applied in the mathematical reasoning of Large Language Models (LLMs). Recently, the introduction of derivative process supervision on CoT trajectories has sparked discussions on enhancing scaling capabilities during test time, thereby boosting the potential of these models. However, in multimodal mathematical reasoning, the scarcity of high-quality CoT training data has hindered existing models from achieving high-precision CoT reasoning and has limited the realization of reasoning potential during test time. In this work, we propose a three-module synthesis strategy that integrates CoT distillation, trajectory-format rewriting, and format unification. It results in a high-quality CoT reasoning instruction fine-tuning dataset in multimodal mathematics, MMathCoT-1M. We comprehensively validate the state-of-the-art (SOTA) performance of the trained URSA-7B model on multiple multimodal mathematical benchmarks. For test-time scaling, we introduce a data synthesis strategy that automatically generates process annotation datasets, known as DualMath-1.1M, focusing on both interpretation and logic. By further training URSA-7B on DualMath-1.1M, we transition from CoT reasoning capabilities to robust supervision abilities. The trained URSA-RM-7B acts as a verifier, effectively enhancing the performance of URSA-7B at test time. URSA-RM-7B also demonstrates excellent out-of-distribution (OOD) verifying capabilities, showcasing its generalization. Model weights, training data and code will be open-sourced.
• 摘要：思路链 (CoT) 推理已广泛应用于大型语言模型 (LLM) 的数学推理。最近，在 CoT 轨迹上引入导数过程监督引发了关于增强测试期间扩展能力的讨论，从而提升了这些模型的潜力。然而，在多模态数学推理中，高质量 CoT 训练数据的稀缺阻碍了现有模型实现高精度 CoT 推理，并限制了测试期间推理潜力的实现。在这项工作中，我们提出了一种集成 CoT 提炼、轨迹格式重写和格式统一的三模块综合策略。它产生了一个高质量的多模态数学 CoT 推理指令微调数据集 MMathCoT-1M。我们在多个多模态数学基准上全面验证了训练后的 URSA-7B 模型的最新 (SOTA) 性能。对于测试时间扩展，我们引入了一种数据合成策略，可自动生成过程注释数据集，称为 DualMath-1.1M，侧重于解释和逻辑。通过在 DualMath-1.1M 上进一步训练 URSA-7B，我们从 CoT 推理能力过渡到强大的监督能力。经过训练的 URSA-RM-7B 充当验证器，有效提高了 URSA-7B 在测试时的性能。URSA-RM-7B 还展示了出色的分布外 (OOD) 验证能力，展示了其泛化能力。模型权重、训练数据和代码将开源。

Title: EpiCoder: Encompassing Diversity and Complexity in Code Generation

• Authors: Yaoxiang Wang, Haoling Li, Xin Zhang, Jie Wu, Xiao Liu, Wenxiang Hu, Zhongxin Guo, Yangyu Huang, Ying Xin, Yujiu Yang, Jinsong Su, Qi Chen, Scarlett Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.04694
• Pdf URL: https://arxiv.org/pdf/2501.04694
• Copy Paste: [[2501.04694]] EpiCoder: Encompassing Diversity and Complexity in Code Generation(https://arxiv.org/abs/2501.04694)
• Keywords: llm
• Abstract: Effective instruction tuning is indispensable for optimizing code LLMs, aligning model behavior with user expectations and enhancing model performance in real-world applications. However, most existing methods focus on code snippets, which are limited to specific functionalities and rigid structures, restricting the complexity and diversity of the synthesized data. To address these limitations, we introduce a novel feature tree-based synthesis framework inspired by Abstract Syntax Trees (AST). Unlike AST, which captures syntactic structure of code, our framework models semantic relationships between code elements, enabling the generation of more nuanced and diverse data. The feature tree is constructed from raw data and refined iteratively to increase the quantity and diversity of the extracted features. This process enables the identification of more complex patterns and relationships within the code. By sampling subtrees with controlled depth and breadth, our framework allows precise adjustments to the complexity of the generated code, supporting a wide range of tasks from simple function-level operations to intricate multi-file scenarios. We fine-tuned widely-used base models to create the EpiCoder series, achieving state-of-the-art performance at both the function and file levels across multiple benchmarks. Notably, empirical evidence indicates that our approach shows significant potential in synthesizing highly complex repository-level code data. Further analysis elucidates the merits of this approach by rigorously assessing data complexity and diversity through software engineering principles and LLM-as-a-judge method.
• 摘要：有效的指令调整对于优化代码 LLM、使模型行为与用户期望保持一致以及在实际应用中增强模型性能是必不可少的。然而，大多数现有方法都侧重于代码片段，这些代码片段仅限于特定功能和刚性结构，限制了合成数据的复杂性和多样性。为了解决这些限制，我们引入了一种受抽象语法树 (AST) 启发的基于特征树的新型合成框架。与捕获代码语法结构的 AST 不同，我们的框架对代码元素之间的语义关系进行建模，从而能够生成更细微、更多样化的数据。特征树由原始数据构建，并经过迭代细化以增加提取特征的数量和多样性。此过程可以识别代码中更复杂的模式和关系。通过对深度和广度受控的子树进行采样，我们的框架可以精确调整生成代码的复杂性，支持从简单的函数级操作到复杂的多文件场景的广泛任务。我们对广泛使用的基础模型进行了微调，以创建 EpiCoder 系列，在多个基准测试中在函数和文件级别都实现了最先进的性能。值得注意的是，经验证据表明，我们的方法在合成高度复杂的存储库级代码数据方面表现出巨大的潜力。进一步的分析通过软件工程原理和 LLM-as-a-judge 方法严格评估数据的复杂性和多样性，阐明了这种方法的优点。