2024-12-09

Title: Socio-Emotional Response Generation: A Human Evaluation Protocol for LLM-Based Conversational Systems

• Authors: Lorraine Vanel, Ariel R. Ramos Vela, Alya Yacoubi, Chloé Clavel (IDS, S2A, LTCI)
• Subjects: cs.CL, cs.AI, cs.HC, cs.SI
• Abstract URL: https://arxiv.org/abs/2412.04492
• Pdf URL: https://arxiv.org/pdf/2412.04492
• Copy Paste: [[2412.04492]] Socio-Emotional Response Generation: A Human Evaluation Protocol for LLM-Based Conversational Systems(https://arxiv.org/abs/2412.04492)
• Keywords: language model, llm
• Abstract: Conversational systems are now capable of producing impressive and generally relevant responses. However, we have no visibility nor control of the socio-emotional strategies behind state-of-the-art Large Language Models (LLMs), which poses a problem in terms of their transparency and thus their trustworthiness for critical applications. Another issue is that current automated metrics are not able to properly evaluate the quality of generated responses beyond the dataset's ground truth. In this paper, we propose a neural architecture that includes an intermediate step in planning socio-emotional strategies before response generation. We compare the performance of open-source baseline LLMs to the outputs of these same models augmented with our planning module. We also contrast the outputs obtained from automated metrics and evaluation results provided by human annotators. We describe a novel evaluation protocol that includes a coarse-grained consistency evaluation, as well as a finer-grained annotation of the responses on various social and emotional criteria. Our study shows that predicting a sequence of expected strategy labels and using this sequence to generate a response yields better results than a direct end-to-end generation scheme. It also highlights the divergences and the limits of current evaluation metrics for generated content. The code for the annotation platform and the annotated data are made publicly available for the evaluation of future models.
• 摘要：对话系统现在能够产生令人印象深刻且通常相关的响应。然而，我们无法看到也无法控制最先进的大型语言模型 (LLM) 背后的社会情感策略，这给它们的透明度带来了问题，因此对关键应用的可信度也带来了问题。另一个问题是，当前的自动化指标无法正确评估生成的响应的质量，超出了数据集的基本事实。在本文中，我们提出了一种神经架构，其中包括在响应生成之前规划社会情感策略的中间步骤。我们将开源基线 LLM 的性能与这些相同模型的输出进行了比较，这些模型增强了我们的规划模块。我们还对比了从自动化指标获得的输出和人工注释者提供的评估结果。我们描述了一种新颖的评估协议，其中包括粗粒度一致性评估，以及对各种社交和情感标准的响应的更细粒度注释。我们的研究表明，预测一系列预期策略标签并使用该序列生成响应比直接端到端生成方案产生更好的结果。它还强调了当前生成内容评估指标的分歧和局限性。注释平台的代码和注释数据已公开，以供未来模型评估。

Title: MAG-V: A Multi-Agent Framework for Synthetic Data Generation and Verification

• Authors: Saptarshi Sengupta, Kristal Curtis, Akshay Mallipeddi, Abhinav Mathur, Joseph Ross, Liang Gou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.04494
• Pdf URL: https://arxiv.org/pdf/2412.04494
• Copy Paste: [[2412.04494]] MAG-V: A Multi-Agent Framework for Synthetic Data Generation and Verification(https://arxiv.org/abs/2412.04494)
• Keywords: language model, gpt, llm, agent
• Abstract: Extending the capabilities of Large Language Models (LLMs) with functions or tools for environment interaction has led to the emergence of the agent paradigm. In industry, training an LLM is not always feasible because of the scarcity of domain data, legal holds on proprietary customer data, rapidly changing business requirements, and the need to prototype new assistants. Agents provide an elegant solution to the above by relying on the zero-shot reasoning abilities of the underlying LLM and utilizing tools to explore and reason over customer data and respond to user requests. However, there are two concerns here: (I) acquiring large scale customer queries for agent testing is time-consuming, and (II) high reliance on the tool call sequence (or trajectory) followed by the agent to respond to user queries may lead to unexpected or incorrect behavior. To address this, we propose MAG-V, a multi-agent framework to first generate a dataset of questions that mimic customer queries; and second, reverse-engineer alternate questions from the responses for trajectory verification. Initial results indicate that our synthetic data can improve agent performance on actual customer queries. Furthermore, our trajectory verification methodology, inspired by distant supervision and using traditional machine learning (ML) models, outperforms a GPT-4o judge baseline by 11% accuracy and matches the performance of a GPT-4 judge on our constructed dataset. Overall, our approach is a step towards unifying diverse task agents into a cohesive framework for achieving an aligned objective.
• 摘要：通过为环境交互添加函数或工具来扩展大型语言模型 (LLM) 的功能，导致了代理范式的出现。在行业中，训练 LLM 并不总是可行的，因为领域数据稀缺、专有客户数据的法律保留、业务需求快速变化以及需要为新助手制作原型。代理通过依赖底层 LLM 的零样本推理能力并利用工具探索和推理客户数据并响应用户请求，为上述问题提供了一个优雅的解决方案。然而，这里有两个问题：(I) 获取大规模客户查询以进行代理测试非常耗时，(II) 高度依赖代理响应用户查询所遵循的工具调用序列（或轨迹）可能会导致意外或不正确的行为。为了解决这个问题，我们提出了 MAG-V，这是一个多代理框架，首先生成一个模拟客户查询的问题数据集；其次，从响应中逆向工程替代问题以进行轨迹验证。初步结果表明，我们的合成数据可以提高代理在实际客户查询方面的表现。此外，我们的轨迹验证方法受到远程监督的启发，并使用了传统的机器学习 (ML) 模型，其准确率比 GPT-4o 评判基准高出 11%，并且与 GPT-4 评判员在我们构建的数据集上的表现相当。总体而言，我们的方法是朝着将不同的任务代理统一到一个有凝聚力的框架中以实现一致目标迈出的一步。

Title: Opportunities and Challenges of Large Language Models for Low-Resource Languages in Humanities Research

• Authors: Tianyang Zhong, Zhenyuan Yang, Zhengliang Liu, Ruidong Zhang, Yiheng Liu, Haiyang Sun, Yi Pan, Yiwei Li, Yifan Zhou, Hanqi Jiang, Junhao Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.04497
• Pdf URL: https://arxiv.org/pdf/2412.04497
• Copy Paste: [[2412.04497]] Opportunities and Challenges of Large Language Models for Low-Resource Languages in Humanities Research(https://arxiv.org/abs/2412.04497)
• Keywords: language model, llm
• Abstract: Low-resource languages serve as invaluable repositories of human history, embodying cultural evolution and intellectual diversity. Despite their significance, these languages face critical challenges, including data scarcity and technological limitations, which hinder their comprehensive study and preservation. Recent advancements in large language models (LLMs) offer transformative opportunities for addressing these challenges, enabling innovative methodologies in linguistic, historical, and cultural research. This study systematically evaluates the applications of LLMs in low-resource language research, encompassing linguistic variation, historical documentation, cultural expressions, and literary analysis. By analyzing technical frameworks, current methodologies, and ethical considerations, this paper identifies key challenges such as data accessibility, model adaptability, and cultural sensitivity. Given the cultural, historical, and linguistic richness inherent in low-resource languages, this work emphasizes interdisciplinary collaboration and the development of customized models as promising avenues for advancing research in this domain. By underscoring the potential of integrating artificial intelligence with the humanities to preserve and study humanity's linguistic and cultural heritage, this study fosters global efforts towards safeguarding intellectual diversity.
• 摘要：低资源语言是人类历史的宝贵宝库，体现了文化进化和智力多样性。尽管这些语言意义重大，但它们面临着数据稀缺和技术限制等严峻挑战，这阻碍了对它们的全面研究和保存。大型语言模型 (LLM) 的最新进展为应对这些挑战提供了变革性的机会，使语言、历史和文化研究的创新方法成为可能。本研究系统地评估了 LLM 在低资源语言研究中的应用，涵盖了语言变异、历史文献、文化表达和文学分析。通过分析技术框架、当前方法和道德考虑，本文确定了数据可访问性、模型适应性和文化敏感性等关键挑战。鉴于低资源语言固有的文化、历史和语言丰富性，这项工作强调跨学科合作和定制模型的开发是推动该领域研究的有希望的途径。通过强调人工智能与人文学科相结合以保护和研究人类语言和文化遗产的潜力，这项研究促进了全球维护思想多样性的努力。

Title: Large Language Models in Politics and Democracy: A Comprehensive Survey

• Authors: Goshi Aoki
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2412.04498
• Pdf URL: https://arxiv.org/pdf/2412.04498
• Copy Paste: [[2412.04498]] Large Language Models in Politics and Democracy: A Comprehensive Survey(https://arxiv.org/abs/2412.04498)
• Keywords: language model, llm
• Abstract: The advancement of generative AI, particularly large language models (LLMs), has a significant impact on politics and democracy, offering potential across various domains, including policymaking, political communication, analysis, and governance. This paper surveys the recent and potential applications of LLMs in politics, examining both their promises and the associated challenges. This paper examines the ways in which LLMs are being employed in legislative processes, political communication, and political analysis. Moreover, we investigate the potential of LLMs in diplomatic and national security contexts, economic and social modeling, and legal applications. While LLMs offer opportunities to enhance efficiency, inclusivity, and decision-making in political processes, they also present challenges related to bias, transparency, and accountability. The paper underscores the necessity for responsible development, ethical considerations, and governance frameworks to ensure that the integration of LLMs into politics aligns with democratic values and promotes a more just and equitable society.
• 摘要：生成式人工智能的发展，尤其是大型语言模型 (LLM)，对政治和民主产生了重大影响，在政策制定、政治交流、分析和治理等各个领域都具有潜力。本文调查了 LLM 在政治领域的近期和潜在应用，研究了它们的前景和相关挑战。本文探讨了 LLM 在立法进程、政治交流和政治分析中的应用方式。此外，我们还研究了 LLM 在外交和国家安全背景、经济和社会建模以及法律应用中的潜力。虽然 LLM 提供了提高政治进程效率、包容性和决策能力的机会，但它们也带来了与偏见、透明度和问责制相关的挑战。本文强调了负责任的发展、道德考虑和治理框架的必要性，以确保 LLM 融入政治符合民主价值观并促进更加公正和公平的社会。

Title: A Primer on Large Language Models and their Limitations

• Authors: Sandra Johnson, David Hyland-Wood
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.04503
• Pdf URL: https://arxiv.org/pdf/2412.04503
• Copy Paste: [[2412.04503]] A Primer on Large Language Models and their Limitations(https://arxiv.org/abs/2412.04503)
• Keywords: language model, llm
• Abstract: This paper provides a primer on Large Language Models (LLMs) and identifies their strengths, limitations, applications and research directions. It is intended to be useful to those in academia and industry who are interested in gaining an understanding of the key LLM concepts and technologies, and in utilising this knowledge in both day to day tasks and in more complex scenarios where this technology can enhance current practices and processes.
• 摘要：本文介绍了大型语言模型 (LLM)，并指出了其优势、局限性、应用和研究方向。本文旨在为学术界和业界那些有兴趣了解关键 LLM 概念和技术的人提供帮助，并帮助他们在日常任务和更复杂的场景中运用这些知识，在这些场景中，该技术可以增强当前的实践和流程。

Title: Multi-Bin Batching for Increasing LLM Inference Throughput

• Authors: Ozgur Guldogan, Jackson Kunde, Kangwook Lee, Ramtin Pedarsani
• Subjects: cs.CL, cs.DC, cs.LG, eess.SY
• Abstract URL: https://arxiv.org/abs/2412.04504
• Pdf URL: https://arxiv.org/pdf/2412.04504
• Copy Paste: [[2412.04504]] Multi-Bin Batching for Increasing LLM Inference Throughput(https://arxiv.org/abs/2412.04504)
• Keywords: language model, llm
• Abstract: As large language models (LLMs) grow in popularity for their diverse capabilities, improving the efficiency of their inference systems has become increasingly critical. Batching LLM requests is a critical step in scheduling the inference jobs on servers (e.g. GPUs), enabling the system to maximize throughput by allowing multiple requests to be processed in parallel. However, requests often have varying generation lengths, causing resource underutilization, as hardware must wait for the longest-running request in the batch to complete before moving to the next batch. We formalize this problem from a queueing-theoretic perspective, and aim to design a control policy which is throughput-optimal. We propose Multi-Bin Batching, a simple yet effective method that can provably improve LLM inference throughput by grouping requests with similar (predicted) execution times into predetermined bins. Through a combination of theoretical analysis and experiments, including real-world LLM inference scenarios, we demonstrate significant throughput gains compared to standard batching approaches.
• 摘要：随着大型语言模型 (LLM) 因其多样化功能而越来越受欢迎，提高其推理系统的效率变得越来越重要。批处理 LLM 请求是调度服务器（例如 GPU）上的推理作业的关键步骤，使系统能够通过允许并行处理多个请求来最大化吞吐量。但是，请求的生成长度通常各不相同，导致资源利用不足，因为硬件必须等待批次中运行时间最长的请求完成后才能转到下一个批次。我们从排队论的角度形式化了这个问题，并旨在设计一种吞吐量最优的控制策略。我们提出了多箱批处理，这是一种简单而有效的方法，可以通过将具有相似（预测）执行时间的请求分组到预定的箱中来证明可以提高 LLM 推理吞吐量。通过理论分析和实验（包括现实世界的 LLM 推理场景）的结合，我们证明了与标准批处理方法相比显著的吞吐量提升。

Title: Pragmatic Metacognitive Prompting Improves LLM Performance on Sarcasm Detection

• Authors: Joshua Lee, Wyatt Fong, Alexander Le, Sur Shah, Kevin Han, Kevin Zhu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.04509
• Pdf URL: https://arxiv.org/pdf/2412.04509
• Copy Paste: [[2412.04509]] Pragmatic Metacognitive Prompting Improves LLM Performance on Sarcasm Detection(https://arxiv.org/abs/2412.04509)
• Keywords: language model, gpt, llm, prompt
• Abstract: Sarcasm detection is a significant challenge in sentiment analysis due to the nuanced and context-dependent nature of verbiage. We introduce Pragmatic Metacognitive Prompting (PMP) to improve the performance of Large Language Models (LLMs) in sarcasm detection, which leverages principles from pragmatics and reflection helping LLMs interpret implied meanings, consider contextual cues, and reflect on discrepancies to identify sarcasm. Using state-of-the-art LLMs such as LLaMA-3-8B, GPT-4o, and Claude 3.5 Sonnet, PMP achieves state-of-the-art performance on GPT-4o on MUStARD and SemEval2018. This study demonstrates that integrating pragmatic reasoning and metacognitive strategies into prompting significantly enhances LLMs' ability to detect sarcasm, offering a promising direction for future research in sentiment analysis.
• 摘要：由于措辞的细微差别和上下文依赖性，讽刺检测是情绪分析中的一项重大挑战。我们引入了语用元认知提示 (PMP) 来提高大型语言模型 (LLM) 在讽刺检测方面的表现，它利用语用学和反思的原理帮助 LLM 解释隐含的含义、考虑上下文线索并反思差异以识别讽刺。使用 LLaMA-3-8B、GPT-4o 和 Claude 3.5 Sonnet 等最先进的 LLM，PMP 在 MUStARD 和 SemEval2018 上的 GPT-4o 上实现了最先进的性能。这项研究表明，将语用推理和元认知策略整合到提示中可显著增强 LLM 检测讽刺的能力，为情绪分析的未来研究提供了一个有希望的方向。

Title: Prompting Large Language Models for Clinical Temporal Relation Extraction

• Authors: Jianping He, Laila Rasmy, Haifang Li, Jianfu Li, Zenan Sun, Evan Yu, Degui Zhi, Cui Tao
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.04512
• Pdf URL: https://arxiv.org/pdf/2412.04512
• Copy Paste: [[2412.04512]] Prompting Large Language Models for Clinical Temporal Relation Extraction(https://arxiv.org/abs/2412.04512)
• Keywords: language model, llm, prompt
• Abstract: Objective: This paper aims to prompt large language models (LLMs) for clinical temporal relation extraction (CTRE) in both few-shot and fully supervised settings. Materials and Methods: This study utilizes four LLMs: Encoder-based GatorTron-Base (345M)/Large (8.9B); Decoder-based LLaMA3-8B/MeLLaMA-13B. We developed full (FFT) and parameter-efficient (PEFT) fine-tuning strategies and evaluated these strategies on the 2012 i2b2 CTRE task. We explored four fine-tuning strategies for GatorTron-Base: (1) Standard Fine-Tuning, (2) Hard-Prompting with Unfrozen LLMs, (3) Soft-Prompting with Frozen LLMs, and (4) Low-Rank Adaptation (LoRA) with Frozen LLMs. For GatorTron-Large, we assessed two PEFT strategies-Soft-Prompting and LoRA with Frozen LLMs-leveraging Quantization techniques. Additionally, LLaMA3-8B and MeLLaMA-13B employed two PEFT strategies: LoRA strategy with Quantization (QLoRA) applied to Frozen LLMs using instruction tuning and standard fine-tuning. Results: Under fully supervised settings, Hard-Prompting with Unfrozen GatorTron-Base achieved the highest F1 score (89.54%), surpassing the SOTA model (85.70%) by 3.74%. Additionally, two variants of QLoRA adapted to GatorTron-Large and Standard Fine-Tuning of GatorTron-Base exceeded the SOTA model by 2.36%, 1.88%, and 0.25%, respectively. Decoder-based models with frozen parameters outperformed their Encoder-based counterparts in this setting; however, the trend reversed in few-shot scenarios. Discussions and Conclusions: This study presented new methods that significantly improved CTRE performance, benefiting downstream tasks reliant on CTRE systems. The findings underscore the importance of selecting appropriate models and fine-tuning strategies based on task requirements and data availability. Future work will explore larger models and broader CTRE applications.
• 摘要：目标：本文旨在提示大型语言模型 (LLM) 在少样本和全监督设置中用于临床时间关系提取 (CTRE)。材料和方法：本研究使用四个 LLM：基于编码器的 GatorTron-Base (345M)/Large (8.9B)；基于解码器的 LLaMA3-8B/MeLLaMA-13B。我们开发了完整 (FFT) 和参数高效 (PEFT) 微调策略，并在 2012 i2b2 CTRE 任务上评估了这些策略。我们探索了 GatorTron-Base 的四种微调策略：(1) 标准微调，(2) 使用未冻结的 LLM 进行硬提示，(3) 使用冻结的 LLM 进行软提示，以及 (4) 使用冻结的 LLM 进行低秩自适应 (LoRA)。对于 GatorTron-Large，我们评估了两种利用量化技术的 PEFT 策略——软提示和带有冻结 LLM 的 LoRA。此外，LLaMA3-8B 和 MeLLaMA-13B 采用了两种 PEFT 策略：将带有量化的 LoRA 策略 (QLoRA) 应用于使用指令调整和标准微调的冻结 LLM。结果：在完全监督的设置下，带有未冻结 GatorTron-Base 的硬提示获得了最高的 F1 分数 (89.54%)，比 SOTA 模型 (85.70%) 高出 3.74%。此外，适用于 GatorTron-Large 的两种 QLoRA 变体和 GatorTron-Base 的标准微调分别比 SOTA 模型高出 2.36%、1.88% 和 0.25%。在这种设置下，具有冻结参数的基于解码器的模型优于基于编码器的模型；然而，在少样本场景中，趋势发生了逆转。讨论和结论：本研究提出了新方法，显著提高了 CTRE 性能，使依赖 CTRE 系统的下游任务受益。研究结果强调了根据任务要求和数据可用性选择合适模型和微调策略的重要性。未来的工作将探索更大的模型和更广泛的 CTRE 应用。

Title: Understanding Hidden Computations in Chain-of-Thought Reasoning

• Authors: Aryasomayajula Ram Bharadwaj
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.04537
• Pdf URL: https://arxiv.org/pdf/2412.04537
• Copy Paste: [[2412.04537]] Understanding Hidden Computations in Chain-of-Thought Reasoning(https://arxiv.org/abs/2412.04537)
• Keywords: language model, prompt, chain-of-thought
• Abstract: Chain-of-Thought (CoT) prompting has significantly enhanced the reasoning abilities of large language models. However, recent studies have shown that models can still perform complex reasoning tasks even when the CoT is replaced with filler(hidden) characters (e.g., "..."), leaving open questions about how models internally process and represent reasoning steps. In this paper, we investigate methods to decode these hidden characters in transformer models trained with filler CoT sequences. By analyzing layer-wise representations using the logit lens method and examining token rankings, we demonstrate that the hidden characters can be recovered without loss of performance. Our findings provide insights into the internal mechanisms of transformer models and open avenues for improving interpretability and transparency in language model reasoning.
• 摘要：思路链 (CoT) 提示显著增强了大型语言模型的推理能力。然而，最近的研究表明，即使用填充（隐藏）字符（例如“...”）替换 CoT，模型仍然可以执行复杂的推理任务，这留下了模型内部如何处理和表示推理步骤的问题。在本文中，我们研究了在使用填充 CoT 序列训练的 Transformer 模型中解码这些隐藏字符的方法。通过使用 logit lens 方法分析分层表示并检查 token 排名，我们证明了隐藏字符可以在不损失性能的情况下恢复。我们的研究结果为 Transformer 模型的内部机制提供了见解，并为提高语言模型推理的可解释性和透明度开辟了途径。

Title: Give me Some Hard Questions: Synthetic Data Generation for Clinical QA

• Authors: Fan Bai, Keith Harrigian, Joel Stremmel, Hamid Hassanzadeh, Ardavan Saeedi, Mark Dredze
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.04573
• Pdf URL: https://arxiv.org/pdf/2412.04573
• Copy Paste: [[2412.04573]] Give me Some Hard Questions: Synthetic Data Generation for Clinical QA(https://arxiv.org/abs/2412.04573)
• Keywords: language model, llm, prompt
• Abstract: Clinical Question Answering (QA) systems enable doctors to quickly access patient information from electronic health records (EHRs). However, training these systems requires significant annotated data, which is limited due to the expertise needed and the privacy concerns associated with clinical data. This paper explores generating Clinical QA data using large language models (LLMs) in a zero-shot setting. We find that naive prompting often results in easy questions that do not reflect the complexity of clinical scenarios. To address this, we propose two prompting strategies: 1) instructing the model to generate questions that do not overlap with the input context, and 2) summarizing the input record using a predefined schema to scaffold question generation. Experiments on two Clinical QA datasets demonstrate that our method generates more challenging questions, significantly improving fine-tuning performance over baselines. We compare synthetic and gold data and find a gap between their training efficacy resulting from the quality of synthetically generated answers.
• 摘要：临床问答 (QA) 系统使医生能够快速从电子健康记录 (EHR) 中访问患者信息。然而，训练这些系统需要大量带注释的数据，由于所需的专业知识和与临床数据相关的隐私问题，这些数据受到限制。本文探讨了在零样本设置中使用大型语言模型 (LLM) 生成临床 QA 数据。我们发现，简单的提示通常会导致简单的问题无法反映临床场景的复杂性。为了解决这个问题，我们提出了两种提示策略：1) 指示模型生成与输入上下文不重叠的问题，2) 使用预定义模式总结输入记录以支持问题生成。在两个临床 QA 数据集上进行的实验表明，我们的方法生成了更具挑战性的问题，与基线相比，微调性能显著提高。我们比较了合成数据和黄金数据，发现它们的训练效果之间存在差距，这是由合成生成的答案的质量造成的。

Title: Show, Don't Tell: Uncovering Implicit Character Portrayal using LLMs

• Authors: Brandon Jaipersaud, Zining Zhu, Frank Rudzicz, Elliot Creager
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2412.04576
• Pdf URL: https://arxiv.org/pdf/2412.04576
• Copy Paste: [[2412.04576]] Show, Don't Tell: Uncovering Implicit Character Portrayal using LLMs(https://arxiv.org/abs/2412.04576)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: Tools for analyzing character portrayal in fiction are valuable for writers and literary scholars in developing and interpreting compelling stories. Existing tools, such as visualization tools for analyzing fictional characters, primarily rely on explicit textual indicators of character attributes. However, portrayal is often implicit, revealed through actions and behaviors rather than explicit statements. We address this gap by leveraging large language models (LLMs) to uncover implicit character portrayals. We start by generating a dataset for this task with greater cross-topic similarity, lexical diversity, and narrative lengths than existing narrative text corpora such as TinyStories and WritingPrompts. We then introduce LIIPA (LLMs for Inferring Implicit Portrayal for Character Analysis), a framework for prompting LLMs to uncover character portrayals. LIIPA can be configured to use various types of intermediate computation (character attribute word lists, chain-of-thought) to infer how fictional characters are portrayed in the source text. We find that LIIPA outperforms existing approaches, and is more robust to increasing character counts (number of unique persons depicted) due to its ability to utilize full narrative context. Lastly, we investigate the sensitivity of portrayal estimates to character demographics, identifying a fairness-accuracy tradeoff among methods in our LIIPA framework -- a phenomenon familiar within the algorithmic fairness literature. Despite this tradeoff, all LIIPA variants consistently outperform non-LLM baselines in both fairness and accuracy. Our work demonstrates the potential benefits of using LLMs to analyze complex characters and to better understand how implicit portrayal biases may manifest in narrative texts.
• 摘要：分析小说中人物形象的工具对于作家和文学学者创作和解读引人入胜的故事非常有价值。现有的工具，例如用于分析虚构人物的可视化工具，主要依赖于人物属性的明确文本指标。然而，人物形象往往是隐性的，通过动作和行为而不是明确的陈述来揭示。我们通过利用大型语言模型 (LLM) 来揭示隐含的人物形象，从而解决这一差距。我们首先为这项任务生成一个数据集，该数据集具有比现有叙事文本语料库（如 TinyStories 和 WritingPrompts）更大的跨主题相似性、词汇多样性和叙事长度。然后，我们引入 LIIPA（用于推断人物分析的隐含形象的 LLM），这是一个用于提示 LLM 揭示人物形象的框架。LIIPA 可以配置为使用各种类型的中间计算（人物属性单词列表、思路链）来推断虚构人物在源文本中的形象。我们发现 LIIPA 的表现优于现有方法，并且由于能够利用完整的叙事背景，因此在增加人物数量（描绘的独特人物数量）时更加稳健。最后，我们研究了人物刻画估计对人物人口统计的敏感性，确定了 LIIPA 框架中方法之间的公平性-准确性权衡——这是算法公平性文献中常见的现象。尽管存在这种权衡，但所有 LIIPA 变体在公平性和准确性方面均始终优于非 LLM 基线。我们的工作展示了使用 LLM 分析复杂人物以及更好地理解隐性刻画偏见如何体现在叙事文本中的潜在好处。

Title: Formulation of probability theory problem with subtle condition

• Authors: Rafayel Petrosyan
• Subjects: cs.CL, math.PR
• Abstract URL: https://arxiv.org/abs/2412.04602
• Pdf URL: https://arxiv.org/pdf/2412.04602
• Copy Paste: [[2412.04602]] Formulation of probability theory problem with subtle condition(https://arxiv.org/abs/2412.04602)
• Keywords: gpt, chat
• Abstract: Problems in probability theory prove to be one of the most challenging for students. Here, we formulate and discuss four related problems in probability theory that proved difficult for first to fourth-year undergraduate students whose first language was not English. These examples emphasize how crucial it is to understand the conditions and requirements of the problems precisely before starting to solve them. We discuss the solutions to those problems in detail, complement them with numerical estimations, and link the conditions in the problems to the logical statements in Python programming language. We also tested two widely used chatbots (GPT-4o and Claude 3.5 Sonnet) by checking their responses to these problems.
• 摘要：概率论问题被证明是学生面临的最具挑战性的问题之一。在这里，我们制定并讨论了四个与概率论相关的问题，这些问题对于母语不是英语的一到四年级本科生来说很难。这些例子强调了在开始解决问题之前准确理解问题的条件和要求是多么重要。我们详细讨论了这些问题的解决方案，用数值估计对其进行了补充，并将问题中的条件与 Python 编程语言中的逻辑语句联系起来。我们还测试了两个广泛使用的聊天机器人（GPT-4o 和 Claude 3.5 Sonnet），检查它们对这些问题的回答。

Title: LLM-Align: Utilizing Large Language Models for Entity Alignment in Knowledge Graphs

• Authors: Xuan Chen, Tong Lu, Zhichun Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.04690
• Pdf URL: https://arxiv.org/pdf/2412.04690
• Copy Paste: [[2412.04690]] LLM-Align: Utilizing Large Language Models for Entity Alignment in Knowledge Graphs(https://arxiv.org/abs/2412.04690)
• Keywords: language model, llm, hallucination
• Abstract: Entity Alignment (EA) seeks to identify and match corresponding entities across different Knowledge Graphs (KGs), playing a crucial role in knowledge fusion and integration. Embedding-based entity alignment (EA) has recently gained considerable attention, resulting in the emergence of many innovative approaches. Initially, these approaches concentrated on learning entity embeddings based on the structural features of knowledge graphs (KGs) as defined by relation triples. Subsequent methods have integrated entities' names and attributes as supplementary information to improve the embeddings used for EA. However, existing methods lack a deep semantic understanding of entity attributes and relations. In this paper, we propose a Large Language Model (LLM) based Entity Alignment method, LLM-Align, which explores the instruction-following and zero-shot capabilities of Large Language Models to infer alignments of entities. LLM-Align uses heuristic methods to select important attributes and relations of entities, and then feeds the selected triples of entities to an LLM to infer the alignment results. To guarantee the quality of alignment results, we design a multi-round voting mechanism to mitigate the hallucination and positional bias issues that occur with LLMs. Experiments on three EA datasets, demonstrating that our approach achieves state-of-the-art performance compared to existing EA methods.
• 摘要：实体对齐 (EA) 旨在识别和匹配不同知识图谱 (KG) 中的相应实体，在知识融合和集成中起着至关重要的作用。基于嵌入的实体对齐 (EA) 近年来引起了广泛关注，并由此催生了许多创新方法。最初，这些方法集中于根据关系三元组定义的知识图谱 (KG) 的结构特征来学习实体嵌入。后续方法将实体的名称和属性作为补充信息进行整合，以改进用于 EA 的嵌入。然而，现有方法缺乏对实体属性和关系的深度语义理解。在本文中，我们提出了一种基于大型语言模型 (LLM) 的实体对齐方法 LLM-Align，该方法探索大型语言模型的指令跟踪和零样本能力来推断实体的对齐。LLM-Align 使用启发式方法来选择实体的重要属性和关系，然后将选定的实体三元组输入 LLM 以推断对齐结果。为了保证对齐结果的质量，我们设计了一种多轮投票机制来缓解 LLM 中出现的幻觉和位置偏差问题。在三个 EA 数据集上进行的实验表明，与现有的 EA 方法相比，我们的方法实现了最先进的性能。

Title: Transformers Struggle to Learn to Search

• Authors: Abulhair Saparov, Srushti Pawar, Shreyas Pimpalgaonkar, Nitish Joshi, Richard Yuanzhe Pang, Vishakh Padmakumar, Seyed Mehran Kazemi, Najoung Kim, He He
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.04703
• Pdf URL: https://arxiv.org/pdf/2412.04703
• Copy Paste: [[2412.04703]] Transformers Struggle to Learn to Search(https://arxiv.org/abs/2412.04703)
• Keywords: language model, llm, chain-of-thought
• Abstract: Search is an ability foundational in many important tasks, and recent studies have shown that large language models (LLMs) struggle to perform search robustly. It is unknown whether this inability is due to a lack of data, insufficient model parameters, or fundamental limitations of the transformer architecture. In this work, we use the foundational graph connectivity problem as a testbed to generate effectively limitless high-coverage data to train small transformers and test whether they can learn to perform search. We find that, when given the right training distribution, the transformer is able to learn to search. We analyze the algorithm that the transformer has learned through a novel mechanistic interpretability technique that enables us to extract the computation graph from the trained model. We find that for each vertex in the input graph, transformers compute the set of vertices reachable from that vertex. Each layer then progressively expands these sets, allowing the model to search over a number of vertices exponential in the number of layers. However, we find that as the input graph size increases, the transformer has greater difficulty in learning the task. This difficulty is not resolved even as the number of parameters is increased, suggesting that increasing model scale will not lead to robust search abilities. We also find that performing search in-context (i.e., chain-of-thought) does not resolve this inability to learn to search on larger graphs.
• 摘要：搜索是许多重要任务的基础能力，最近的研究表明，大型语言模型 (LLM) 难以稳健地执行搜索。目前尚不清楚这种无能为力是由于缺乏数据、模型参数不足还是 Transformer 架构的基本限制。在这项工作中，我们使用基础图连通性问题作为测试平台，以生成有效无限的高覆盖率数据来训练小型 Transformer，并测试它们是否可以学习执行搜索。我们发现，当给定正确的训练分布时，Transformer 能够学习搜索。我们通过一种新颖的机械可解释性技术分析 Transformer 学到的算法，该技术使我们能够从训练模型中提取计算图。我们发现，对于输入图中的每个顶点，Transformer 都会计算从该顶点可到达的顶点集。然后，每一层都会逐步扩展这些集合，使模型能够搜索层数呈指数增长的顶点数量。但是，我们发现随着输入图大小的增加，Transformer 在学习任务时会遇到更大的困难。即使参数数量增加，这一困难仍未得到解决，这表明增加模型规模不会带来强大的搜索能力。我们还发现，在上下文中执行搜索（即思路链）并不能解决无法学习在较大图上进行搜索的问题。

Title: BESSTIE: A Benchmark for Sentiment and Sarcasm Classification for Varieties of English

• Authors: Dipankar Srirag, Aditya Joshi, Jordan Painter, Diptesh Kanojia
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.04726
• Pdf URL: https://arxiv.org/pdf/2412.04726
• Copy Paste: [[2412.04726]] BESSTIE: A Benchmark for Sentiment and Sarcasm Classification for Varieties of English(https://arxiv.org/abs/2412.04726)
• Keywords: language model, llm
• Abstract: Despite large language models (LLMs) being known to exhibit bias against non-mainstream varieties, there are no known labeled datasets for sentiment analysis of English. To address this gap, we introduce BESSTIE, a benchmark for sentiment and sarcasm classification for three varieties of English: Australian (en-AU), Indian (en-IN), and British (en-UK). Using web-based content from two domains, namely, Google Place reviews and Reddit comments, we collect datasets for these language varieties using two methods: location-based and topic-based filtering. Native speakers of the language varieties manually annotate the datasets with sentiment and sarcasm labels. Subsequently, we fine-tune nine large language models (LLMs) (representing a range of encoder/decoder and mono/multilingual models) on these datasets, and evaluate their performance on the two tasks. Our results reveal that the models consistently perform better on inner-circle varieties (i.e., en-AU and en-UK), with significant performance drops for en-IN, particularly in sarcasm detection. We also report challenges in cross-variety generalisation, highlighting the need for language variety-specific datasets such as ours. BESSTIE promises to be a useful evaluative benchmark for future research in equitable LLMs, specifically in terms of language varieties. The BESSTIE datasets, code, and models are currently available on request, while the paper is under review. Please email this http URL@unsw.this http URL.
• 摘要：尽管大型语言模型 (LLM) 已知会对非主流变体表现出偏见，但目前尚无已知的带标签的英语情感分析数据集。为了填补这一空白，我们引入了 BESSTIE，这是三种英语变体的情感和讽刺分类基准：澳大利亚英语 (en-AU)、印度英语 (en-IN) 和英国英语 (en-UK)。使用来自两个领域的网络内容，即 Google Place 评论和 Reddit 评论，我们使用两种方法收集这些语言变体的数据集：基于位置和基于主题的过滤。语言变体的母语人士手动用情感和讽刺标签注释数据集。随后，我们在这些数据集上微调了九个大型语言模型 (LLM)（代表一系列编码器/解码器和单/多语言模型），并评估它们在这两个任务上的表现。我们的结果表明，这些模型在内圈变体（即 en-AU 和 en-UK）上的表现始终更好，而 en-IN 的性能显著下降，尤其是在讽刺检测方面。我们还报告了跨语言泛化方面的挑战，强调了对语言变体特定数据集（如我们的数据集）的需求。BESSTIE 有望成为未来公平法学硕士研究的有用评估基准，特别是在语言变体方面。BESSTIE 数据集、代码和模型目前可根据要求提供，而论文正在审查中。请发送电子邮件至此 http URL@unsw.此 http URL。

Title: Ltri-LLM: Streaming Long Context Inference for LLMs with Training-Free Dynamic Triangular Attention Pattern

• Authors: Hongyin Tang, Di Xiu, Lanrui Wang, Xiurui Geng, Jingang Wang, Xunliang Cai
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.04757
• Pdf URL: https://arxiv.org/pdf/2412.04757
• Copy Paste: [[2412.04757]] Ltri-LLM: Streaming Long Context Inference for LLMs with Training-Free Dynamic Triangular Attention Pattern(https://arxiv.org/abs/2412.04757)
• Keywords: language model, llm, long context
• Abstract: The quadratic computational complexity of the attention mechanism in current Large Language Models (LLMs) renders inference with long contexts prohibitively expensive. To address this challenge, various approaches aim to retain critical portions of the context to optimally approximate Full Attention (FA) through Key-Value (KV) compression or Sparse Attention (SA), enabling the processing of virtually unlimited text lengths in a streaming manner. However, these methods struggle to achieve performance levels comparable to FA, particularly in retrieval tasks. In this paper, our analysis of attention head patterns reveals that LLMs' attention distributions show strong local correlations, naturally reflecting a chunking mechanism for input context. We propose Ltri-LLM framework, which divides KVs into spans, stores them in an offline index, and retrieves the relevant KVs into memory for various queries. Experimental results on popular long text benchmarks show that Ltri-LLM can achieve performance close to FA while maintaining efficient, streaming-based inference.
• 摘要：当前大型语言模型 (LLM) 中的注意力机制的二次计算复杂度使得长上下文推理的成本过高。为了应对这一挑战，各种方法都旨在保留上下文的关键部分，以通过键值 (KV) 压缩或稀疏注意力 (SA) 最佳地近似全注意力 (FA)，从而能够以流式方式处理几乎无限长的文本。然而，这些方法难以达到与 FA 相当的性能水平，特别是在检索任务中。在本文中，我们对注意力头模式的分析表明，LLM 的注意力分布表现出很强的局部相关性，自然反映了输入上下文的分块机制。我们提出了 Ltri-LLM 框架，它将 KV 划分为跨度，将它们存储在离线索引中，并将相关 KV 检索到内存中以用于各种查询。在流行的长文本基准上的实验结果表明，Ltri-LLM 可以实现接近 FA 的性能，同时保持高效的基于流的推理。

Title: Foundation Models for Low-Resource Language Education (Vision Paper)

• Authors: Zhaojun Ding, Zhengliang Liu, Hanqi Jiang, Yizhu Gao, Xiaoming Zhai, Tianming Liu, Ninghao Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.04774
• Pdf URL: https://arxiv.org/pdf/2412.04774
• Copy Paste: [[2412.04774]] Foundation Models for Low-Resource Language Education (Vision Paper)(https://arxiv.org/abs/2412.04774)
• Keywords: language model, llm
• Abstract: Recent studies show that large language models (LLMs) are powerful tools for working with natural language, bringing advances in many areas of computational linguistics. However, these models face challenges when applied to low-resource languages due to limited training data and difficulty in understanding cultural nuances. Research is now focusing on multilingual models to improve LLM performance for these languages. Education in these languages also struggles with a lack of resources and qualified teachers, particularly in underdeveloped regions. Here, LLMs can be transformative, supporting innovative methods like community-driven learning and digital platforms. This paper discusses how LLMs could enhance education for low-resource languages, emphasizing practical applications and benefits.
• 摘要：最近的研究表明，大型语言模型 (LLM) 是处理自然语言的强大工具，为计算语言学的许多领域带来了进步。然而，由于训练数据有限，并且难以理解文化差异，这些模型在应用于资源匮乏的语言时面临挑战。研究现在专注于多语言模型，以提高这些语言的 LLM 性能。这些语言的教育也面临着资源和合格教师不足的问题，尤其是在欠发达地区。在这里，LLM 可以带来变革，支持社区驱动的学习和数字平台等创新方法。本文讨论了 LLM 如何加强资源匮乏的语言教育，强调实际应用和好处。

Title: Breaking Event Rumor Detection via Stance-Separated Multi-Agent Debate

• Authors: Mingqing Zhang, Haisong Gong, Qiang Liu, Shu Wu, Liang Wang
• Subjects: cs.CL, cs.MA
• Abstract URL: https://arxiv.org/abs/2412.04859
• Pdf URL: https://arxiv.org/pdf/2412.04859
• Copy Paste: [[2412.04859]] Breaking Event Rumor Detection via Stance-Separated Multi-Agent Debate(https://arxiv.org/abs/2412.04859)
• Keywords: language model, llm, prompt, agent
• Abstract: The rapid spread of rumors on social media platforms during breaking events severely hinders the dissemination of the truth. Previous studies reveal that the lack of annotated resources hinders the direct detection of unforeseen breaking events not covered in yesterday's news. Leveraging large language models (LLMs) for rumor detection holds significant promise. However, it is challenging for LLMs to provide comprehensive responses to complex or controversial issues due to limited diversity. In this work, we propose the Stance Separated Multi-Agent Debate (S2MAD) to address this issue. Specifically, we firstly introduce Stance Separation, categorizing comments as either supporting or opposing the original claim. Subsequently, claims are classified as subjective or objective, enabling agents to generate reasonable initial viewpoints with different prompt strategies for each type of claim. Debaters then follow specific instructions through multiple rounds of debate to reach a consensus. If a consensus is not reached, a judge agent evaluates the opinions and delivers a final verdict on the claim's veracity. Extensive experiments conducted on two real-world datasets demonstrate that our proposed model outperforms state-of-the-art methods in terms of performance and effectively improves the performance of LLMs in breaking event rumor detection.
• 摘要：突发事件发生时，社交媒体平台上谣言的迅速传播严重阻碍了真相的传播。先前的研究表明，缺乏注释资源阻碍了对昨天新闻中未报道的突发事件的直接检测。利用大型语言模型 (LLM) 进行谣言检测具有重大前景。然而，由于多样性有限，LLM 很难对复杂或有争议的问题提供全面的回应。在这项工作中，我们提出了立场分离多智能体辩论 (S2MAD) 来解决这个问题。具体来说，我们首先引入立场分离，将评论分类为支持或反对原始主张。随后，主张被分类为主观或客观，使智能体能够针对每种类型的主张使用不同的提示策略生成合理的初始观点。然后，辩论者按照特定指示通过多轮辩论达成共识。如果没有达成共识，则由评判智能体评估意见并对主张的真实性做出最终裁决。在两个真实数据集上进行的大量实验表明，我们提出的模型在性能方面优于最先进的方法，并有效提高了 LLM 在突发事件谣言检测方面的性能。

Title: EXAONE 3.5: Series of Large Language Models for Real-world Use Cases

• Authors: LG AI Research, Soyoung An, Kyunghoon Bae, Eunbi Choi, Kibong Choi, Stanley Jungkyu Choi, Seokhee Hong, Junwon Hwang, Hyojin Jeon, Gerrard Jeongwon Jo, Hyunjik Jo, Jiyeon Jung, Yountae Jung, Hyosang Kim, Joonkee Kim, Seonghwan Kim, Soyeon Kim, Sunkyoung Kim, Yireun Kim, Yongil Kim, Youchul Kim, Edward Hwayoung Lee, Haeju Lee, Honglak Lee, Jinsik Lee, Kyungmin Lee, Woohyung Lim, Sangha Park, Sooyoun Park, Yongmin Park, Sihoon Yang, Heuiyeen Yeen, Hyeongu Yun
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.04862
• Pdf URL: https://arxiv.org/pdf/2412.04862
• Copy Paste: [[2412.04862]] EXAONE 3.5: Series of Large Language Models for Real-world Use Cases(https://arxiv.org/abs/2412.04862)
• Keywords: language model
• Abstract: This technical report introduces the EXAONE 3.5 instruction-tuned language models, developed and released by LG AI Research. The EXAONE 3.5 language models are offered in three configurations: 32B, 7.8B, and 2.4B. These models feature several standout capabilities: 1) exceptional instruction following capabilities in real-world scenarios, achieving the highest scores across seven benchmarks, 2) outstanding long-context comprehension, attaining the top performance in four benchmarks, and 3) competitive results compared to state-of-the-art open models of similar sizes across nine general benchmarks. The EXAONE 3.5 language models are open to anyone for research purposes and can be downloaded from this https URL. For commercial use, please reach out to the official contact point of LG AI Research: contact_us@lgresearch.ai.
• 摘要：本技术报告介绍了 LG AI Research 开发并发布的 EXAONE 3.5 指令调优语言模型。EXAONE 3.5 语言模型提供三种配置：32B、7.8B 和 2.4B。这些模型具有几项突出的功能：1) 在现实场景中具有出色的指令跟踪能力，在七个基准测试中取得最高分；2) 出色的长上下文理解能力，在四个基准测试中取得最佳性能；3) 与九个通用基准测试中类似规模的最先进的开放模型相比，结果具有竞争力。EXAONE 3.5 语言模型向任何人开放用于研究目的，可以从此 https URL 下载。如需商业使用，请联系 LG AI Research 的官方联系点：contact_us@lgresearch.ai。

Title: Building a Family of Data Augmentation Models for Low-cost LLM Fine-tuning on the Cloud

• Authors: Yuanhao Yue, Chengyu Wang, Jun Huang, Peng Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.04871
• Pdf URL: https://arxiv.org/pdf/2412.04871
• Copy Paste: [[2412.04871]] Building a Family of Data Augmentation Models for Low-cost LLM Fine-tuning on the Cloud(https://arxiv.org/abs/2412.04871)
• Keywords: llm
• Abstract: Specializing LLMs in various domain-specific tasks has emerged as a critical step towards achieving high performance. However, the construction and annotation of datasets in specific domains are always very costly. Apart from using superior and expensive closed-source LLM APIs to construct datasets, some open-source models have become strong enough to handle dataset construction in many scenarios. Thus, we present a family of data augmentation models designed to significantly improve the efficiency for model fine-tuning. These models, trained based on sufficiently small LLMs, support key functionalities with low inference costs: instruction expansion, instruction refinement, and instruction-response pair expansion. To fulfill this goal, we first construct an automatic data collection system with seed datasets generated from both public repositories and our in-house datasets. This system leverages powerful LLMs to expand, refine and re-write the instructions and responses, incorporating quality assessment techniques. Following this, we introduce the training process of our models, which effectively distills task-solving and text synthesis abilities from teacher LLMs. Finally, we demonstrate how we integrate these functionalities into a machine learning platform to support low-cost LLM fine-tuning from both dataset preparation and training perspectives for users. Experiments and an application study prove the effectiveness of our approach.
• 摘要：将 LLM 专门用于各种特定领域的任务已成为实现高性能的关键一步。然而，特定领域的数据集的构建和注释总是非常昂贵的。除了使用优质且昂贵的闭源 LLM API 来构建数据集外，一些开源模型已经变得足够强大，可以在许多场景中处理数据集构建。因此，我们提出了一系列数据增强模型，旨在显著提高模型微调的效率。这些模型基于足够小的 LLM 进行训练，以较低的推理成本支持关键功能：指令扩展、指令细化和指令-响应对扩展。为了实现这一目标，我们首先构建了一个自动数据收集系统，其种子数据集来自公共存储库和我们的内部数据集。该系统利用强大的 LLM 来扩展、细化和重写指令和响应，并结合质量评估技术。接下来，我们介绍了我们模型的训练过程，该过程有效地从教师 LLM 中提炼出任务解决和文本合成能力。最后，我们展示了如何将这些功能集成到机器学习平台中，以便从数据集准备和训练的角度为用户提供低成本的 LLM 微调。实验和应用研究证明了我们方法的有效性。

Title: DEMO: Reframing Dialogue Interaction with Fine-grained Element Modeling

• Authors: Minzheng Wang, Xinghua Zhang, Kun Chen, Nan Xu, Haiyang Yu, Fei Huang, Wenji Mao, Yongbin Li
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.04905
• Pdf URL: https://arxiv.org/pdf/2412.04905
• Copy Paste: [[2412.04905]] DEMO: Reframing Dialogue Interaction with Fine-grained Element Modeling(https://arxiv.org/abs/2412.04905)
• Keywords: language model, llm, agent
• Abstract: Large language models (LLMs) have made dialogue one of the central modes of human-machine interaction, leading to the accumulation of vast amounts of conversation logs and increasing demand for dialogue generation. A conversational life-cycle spans from the Prelude through the Interlocution to the Epilogue, encompassing various elements. Despite the existence of numerous dialogue-related studies, there is a lack of benchmarks that encompass comprehensive dialogue elements, hindering precise modeling and systematic evaluation. To bridge this gap, we introduce an innovative research task $\textbf{D}$ialogue $\textbf{E}$lement $\textbf{MO}$deling, including $\textit{Element Awareness}$ and $\textit{Dialogue Agent Interaction}$, and propose a novel benchmark, $\textbf{DEMO}$, designed for a comprehensive dialogue modeling and assessment. Inspired by imitation learning, we further build the agent which possesses the adept ability to model dialogue elements based on the DEMO benchmark. Extensive experiments indicate that existing LLMs still exhibit considerable potential for enhancement, and our DEMO agent has superior performance in both in-domain and out-of-domain tasks.
• 摘要：大型语言模型 (LLM) 使对话成为人机交互的主要模式之一，导致大量对话日志的积累和对对话生成的需求不断增加。对话生命周期从序言到对话再到尾声，涵盖各种元素。尽管存在大量与对话相关的研究，但缺乏涵盖全面对话元素的基准，阻碍了精确的建模和系统的评估。为了弥补这一差距，我们引入了一个创新的研究任务 $\textbf{D}$对话 $\textbf{E}$元素 $\textbf{MO}$deling，包括 $\textit{元素感知}$ 和 $\textit{对话代理交互}$，并提出了一个新颖的基准 $\textbf{DEMO}$，旨在进行全面的对话建模和评估。受模仿学习的启发，我们进一步构建了基于 DEMO 基准的具有熟练对话元素建模能力的代理。大量实验表明，现有的 LLM 仍然表现出相当大的增强潜力，并且我们的 DEMO 代理在域内和域外任务中都具有卓越的性能。

Title: Large Language Models for Ingredient Substitution in Food Recipes using Supervised Fine-tuning and Direct Preference Optimization

• Authors: Thevin Senath, Kumuthu Athukorala, Ransika Costa, Surangika Ranathunga, Rishemjit Kaur
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.04922
• Pdf URL: https://arxiv.org/pdf/2412.04922
• Copy Paste: [[2412.04922]] Large Language Models for Ingredient Substitution in Food Recipes using Supervised Fine-tuning and Direct Preference Optimization(https://arxiv.org/abs/2412.04922)
• Keywords: language model, llm, prompt
• Abstract: In this paper, we address the challenge of recipe personalization through ingredient substitution. We make use of Large Language Models (LLMs) to build an ingredient substitution system designed to predict plausible substitute ingredients within a given recipe context. Given that the use of LLMs for this task has been barely done, we carry out an extensive set of experiments to determine the best LLM, prompt, and the fine-tuning setups. We further experiment with methods such as multi-task learning, two-stage fine-tuning, and Direct Preference Optimization (DPO). The experiments are conducted using the publicly available Recipe1MSub corpus. The best results are produced by the Mistral7-Base LLM after fine-tuning and DPO. This result outperforms the strong baseline available for the same corpus with a Hit@1 score of 22.04. Thus we believe that this research represents a significant step towards enabling personalized and creative culinary experiences by utilizing LLM-based ingredient substitution.
• 摘要：在本文中，我们通过成分替换来解决食谱个性化的挑战。我们利用大型语言模型 (LLM) 构建一个成分替换系统，旨在预测给定食谱环境中合理的替代成分。鉴于 LLM 很少用于此任务，我们进行了一系列广泛的实验，以确定最佳 LLM、提示和微调设置。我们进一步尝试了多任务学习、两阶段微调和直接偏好优化 (DPO) 等方法。实验是使用公开的 Recipe1MSub 语料库进行的。最佳结果由 Mistral7-Base LLM 在微调和 DPO 后产生。该结果优于同一语料库的强大基线，Hit@1 得分为 22.04。因此，我们相信这项研究代表着通过利用基于 LLM 的成分替换实现个性化和创造性烹饪体验的重要一步。

Title: Probing the contents of semantic representations from text, behavior, and brain data using the psychNorms metabase

• Authors: Zak Hussain, Rui Mata, Ben R. Newell, Dirk U. Wulff
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.04936
• Pdf URL: https://arxiv.org/pdf/2412.04936
• Copy Paste: [[2412.04936]] Probing the contents of semantic representations from text, behavior, and brain data using the psychNorms metabase(https://arxiv.org/abs/2412.04936)
• Keywords: language model, agent
• Abstract: Semantic representations are integral to natural language processing, psycholinguistics, and artificial intelligence. Although often derived from internet text, recent years have seen a rise in the popularity of behavior-based (e.g., free associations) and brain-based (e.g., fMRI) representations, which promise improvements in our ability to measure and model human representations. We carry out the first systematic evaluation of the similarities and differences between semantic representations derived from text, behavior, and brain data. Using representational similarity analysis, we show that word vectors derived from behavior and brain data encode information that differs from their text-derived cousins. Furthermore, drawing on our psychNorms metabase, alongside an interpretability method that we call representational content analysis, we find that, in particular, behavior representations capture unique variance on certain affective, agentic, and socio-moral dimensions. We thus establish behavior as an important complement to text for capturing human representations and behavior. These results are broadly relevant to research aimed at learning human-aligned semantic representations, including work on evaluating and aligning large language models.
• 摘要：语义表征是自然语言处理、心理语言学和人工智能不可或缺的一部分。尽管语义表征通常源自互联网文本，但近年来，基于行为（例如自由联想）和基于大脑（例如 fMRI）的表征越来越受欢迎，这有望提高我们测量和建模人类表征的能力。我们首次系统地评估了源自文本、行为和大脑数据的语义表征之间的相似性和差异性。使用表征相似性分析，我们表明源自行为和大脑数据的词向量编码的信息与源自文本的词向量不同。此外，利用我们的 psychNorms 元数据库以及我们称之为表征内容分析的可解释性方法，我们发现，行为表征在某些情感、行为和社会道德维度上捕捉到了独特的差异。因此，我们将行为确立为文本的重要补充，用于捕捉人类表征和行为。这些结果与旨在学习人类一致的语义表示的研究广泛相关，包括评估和对齐大型语言模型的工作。

Title: Who Speaks Next? Multi-party AI Discussion Leveraging the Systematics of Turn-taking in Murder Mystery Games

• Authors: Ryota Nonomura, Hiroki Mori
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.04937
• Pdf URL: https://arxiv.org/pdf/2412.04937
• Copy Paste: [[2412.04937]] Who Speaks Next? Multi-party AI Discussion Leveraging the Systematics of Turn-taking in Murder Mystery Games(https://arxiv.org/abs/2412.04937)
• Keywords: language model, llm, agent
• Abstract: Multi-agent systems utilizing large language models (LLMs) have shown great promise in achieving natural dialogue. However, smooth dialogue control and autonomous decision making among agents still remain challenges. In this study, we focus on conversational norms such as adjacency pairs and turn-taking found in conversation analysis and propose a new framework called "Murder Mystery Agents" that applies these norms to AI agents' dialogue control. As an evaluation target, we employed the "Murder Mystery" game, a reasoning-type table-top role-playing game that requires complex social reasoning and information manipulation. In this game, players need to unravel the truth of the case based on fragmentary information through cooperation and bargaining. The proposed framework integrates next speaker selection based on adjacency pairs and a self-selection mechanism that takes agents' internal states into account to achieve more natural and strategic dialogue. To verify the effectiveness of this new approach, we analyzed utterances that led to dialogue breakdowns and conducted automatic evaluation using LLMs, as well as human evaluation using evaluation criteria developed for the Murder Mystery game. Experimental results showed that the implementation of the next speaker selection mechanism significantly reduced dialogue breakdowns and improved the ability of agents to share information and perform logical reasoning. The results of this study demonstrate that the systematics of turn-taking in human conversation are also effective in controlling dialogue among AI agents, and provide design guidelines for more advanced multi-agent dialogue systems.
• 摘要：利用大型语言模型 (LLM) 的多智能体系统在实现自然对话方面表现出了巨大的潜力。然而，流畅的对话控制和智能体之间的自主决策仍然是一项挑战。在本研究中，我们重点研究了对话分析中发现的邻接对和话轮转换等对话规范，并提出了一个名为“谋杀之谜智能体”的新框架，将这些规范应用于人工智能智能体的对话控制。作为评估目标，我们采用了“谋杀之谜”游戏，这是一款需要复杂社交推理和信息操纵的推理型桌面角色扮演游戏。在这个游戏中，玩家需要通过合作和讨价还价，根据零碎的信息解开案件的真相。所提出的框架整合了基于邻接对的下一个说话者选择和考虑智能体内部状态的自我选择机制，以实现更自然和更具战略性的对话。为了验证这种新方法的有效性，我们分析了导致对话中断的话语，并使用 LLM 进行了自动评估，并使用为谋杀之谜游戏开发的评估标准进行了人工评估。实验结果表明，实施下一个发言者选择机制显著减少了对话中断，提高了代理共享信息和进行逻辑推理的能力。本研究结果表明，人类对话中话轮转换的系统性在控制人工智能代理之间的对话方面也有效，并为更先进的多代理对话系统提供了设计指南。

Title: C$^2$LEVA: Toward Comprehensive and Contamination-Free Language Model Evaluation

• Authors: Yanyang Li, Tin Long Wong, Cheung To Hung, Jianqiao Zhao, Duo Zheng, Ka Wai Liu, Michael R. Lyu, Liwei Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.04947
• Pdf URL: https://arxiv.org/pdf/2412.04947
• Copy Paste: [[2412.04947]] C$^2$LEVA: Toward Comprehensive and Contamination-Free Language Model Evaluation(https://arxiv.org/abs/2412.04947)
• Keywords: language model, llm
• Abstract: Recent advances in large language models (LLMs) have shown significant promise, yet their evaluation raises concerns, particularly regarding data contamination due to the lack of access to proprietary training data. To address this issue, we present C$^2$LEVA, a comprehensive bilingual benchmark featuring systematic contamination prevention. C$^2$LEVA firstly offers a holistic evaluation encompassing 22 tasks, each targeting a specific application or ability of LLMs, and secondly a trustworthy assessment due to our contamination-free tasks, ensured by a systematic contamination prevention strategy that fully automates test data renewal and enforces data protection during benchmark data release. Our large-scale evaluation of 15 open-source and proprietary models demonstrates the effectiveness of C$^2$LEVA.
• 摘要：大型语言模型 (LLM) 的最新进展已显示出巨大的前景，但它们的评估引起了人们的担忧，尤其是由于缺乏专有训练数据的访问而导致的数据污染问题。为了解决这个问题，我们提出了 C$^2$LEVA，这是一个全面的双语基准，具有系统污染预防功能。C$^2$LEVA 首先提供了涵盖 22 项任务的整体评估，每项任务都针对 LLM 的特定应用或能力；其次，由于我们的无污染任务，我们提供了值得信赖的评估，这由系统污染预防策略确保，该策略可完全自动化测试数据更新并在基准数据发布期间强制数据保护。我们对 15 个开源和专有模型的大规模评估证明了 C$^2$LEVA 的有效性。

Title: KaLM: Knowledge-aligned Autoregressive Language Modeling via Dual-view Knowledge Graph Contrastive Learning

• Authors: Peng Yu, Cheng Deng, Beiya Dai, Xinbing Wang, Ying Wen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.04948
• Pdf URL: https://arxiv.org/pdf/2412.04948
• Copy Paste: [[2412.04948]] KaLM: Knowledge-aligned Autoregressive Language Modeling via Dual-view Knowledge Graph Contrastive Learning(https://arxiv.org/abs/2412.04948)
• Keywords: language model, llm
• Abstract: Autoregressive large language models (LLMs) pre-trained by next token prediction are inherently proficient in generative tasks. However, their performance on knowledge-driven tasks such as factual knowledge querying remains unsatisfactory. Knowledge graphs (KGs), as high-quality structured knowledge bases, can provide reliable knowledge for LLMs, potentially compensating for their knowledge deficiencies. Aligning LLMs with explicit, structured knowledge from KGs has been a challenge; previous attempts either failed to effectively align knowledge representations or compromised the generative capabilities of LLMs, leading to less-than-optimal outcomes. This paper proposes \textbf{KaLM}, a \textit{Knowledge-aligned Language Modeling} approach, which fine-tunes autoregressive LLMs to align with KG knowledge via the joint objective of explicit knowledge alignment and implicit knowledge alignment. The explicit knowledge alignment objective aims to directly optimize the knowledge representation of LLMs through dual-view knowledge graph contrastive learning. The implicit knowledge alignment objective focuses on incorporating textual patterns of knowledge into LLMs through triple completion language modeling. Notably, our method achieves a significant performance boost in evaluations of knowledge-driven tasks, specifically embedding-based knowledge graph completion and generation-based knowledge graph question answering.
• 摘要：通过下一个标记预测预训练的自回归大型语言模型 (LLM) 本质上擅长生成任务。然而，它们在事实知识查询等知识驱动任务上的表现仍然不令人满意。知识图谱 (KG) 作为高质量的结构化知识库，可以为 LLM 提供可靠的知识，有可能弥补其知识的不足。将 LLM 与来自 KG 的显性结构化知识对齐一直是一个挑战；之前的尝试要么未能有效对齐知识表示，要么损害了 LLM 的生成能力，导致结果不尽如人意。本文提出了 \textbf{KaLM}，一种 \textit{知识对齐语言建模} 方法，它通过显性知识对齐和隐性知识对齐的联合目标对自回归 LLM 进行微调以与 KG 知识对齐。显性知识对齐目标旨在通过双视角知识图谱对比学习直接优化 LLM 的知识表示。隐性知识对齐目标侧重于通过三重补全语言建模将知识的文本模式整合到 LLM 中。值得注意的是，我们的方法在知识驱动任务的评估中实现了显著的性能提升，特别是基于嵌入的知识图谱补全和基于生成的知识图谱问答。

Title: PETapter: Leveraging PET-style classification heads for modular few-shot parameter-efficient fine-tuning

• Authors: Jonas Rieger, Mattes Ruckdeschel, Gregor Wiedemann
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.04975
• Pdf URL: https://arxiv.org/pdf/2412.04975
• Copy Paste: [[2412.04975]] PETapter: Leveraging PET-style classification heads for modular few-shot parameter-efficient fine-tuning(https://arxiv.org/abs/2412.04975)
• Keywords: language model
• Abstract: Few-shot learning and parameter-efficient fine-tuning (PEFT) are crucial to overcome the challenges of data scarcity and ever growing language model sizes. This applies in particular to specialized scientific domains, where researchers might lack expertise and resources to fine-tune high-performing language models to nuanced tasks. We propose PETapter, a novel method that effectively combines PEFT methods with PET-style classification heads to boost few-shot learning capabilities without the significant computational overhead typically associated with full model training. We validate our approach on three established NLP benchmark datasets and one real-world dataset from communication research. We show that PETapter not only achieves comparable performance to full few-shot fine-tuning using pattern-exploiting training (PET), but also provides greater reliability and higher parameter efficiency while enabling higher modularity and easy sharing of the trained modules, which enables more researchers to utilize high-performing NLP-methods in their research.
• 摘要：少量学习和参数高效微调 (PEFT) 对于克服数据稀缺和语言模型规模不断增长的挑战至关重要。这尤其适用于专业的科学领域，研究人员可能缺乏专业知识和资源来将高性能语言模型微调到细微任务。我们提出了 PETapter，这是一种新方法，它有效地将 PEFT 方法与 PET 风格的分类头相结合，以提高少量学习能力，而无需通常与完整模型训练相关的大量计算开销。我们在三个已建立的 NLP 基准数据集和一个来自通信研究的真实数据集上验证了我们的方法。我们表明，PETapter 不仅实现了与使用模式利用训练 (PET) 的完整少量微调相当的性能，而且还提供了更高的可靠性和更高的参数效率，同时实现了更高的模块化和易于共享训练模块，这使更多研究人员能够在他们的研究中使用高性能的 NLP 方法。

Title: Steps are all you need: Rethinking STEM Education with Prompt Engineering

• Authors: Krishnasai Addala, Kabir Dev Paul Baghel, Chhavi Kirtani, Avinash Anand, Rajiv Ratn Shah
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.05023
• Pdf URL: https://arxiv.org/pdf/2412.05023
• Copy Paste: [[2412.05023]] Steps are all you need: Rethinking STEM Education with Prompt Engineering(https://arxiv.org/abs/2412.05023)
• Keywords: llm, hallucination, prompt, chain-of-thought
• Abstract: Few shot and Chain-of-Thought prompting have shown promise when applied to Physics Question Answering Tasks, but are limited by the lack of mathematical ability inherent to LLMs, and are prone to hallucination. By utilizing a Mixture of Experts (MoE) Model, along with analogical prompting, we are able to show improved model performance when compared to the baseline on standard LLMs. We also survey the limits of these prompting techniques and the effects they have on model performance. Additionally, we propose Analogical CoT prompting, a prompting technique designed to allow smaller, open source models to leverage Analogical prompting, something they have struggled with, possibly due to a lack of specialist training data.
• 摘要：少量提示和思维链提示在应用于物理问答任务时表现出良好的前景，但由于 LLM 本身缺乏数学能力，因此受到限制，并且容易产生幻觉。通过利用混合专家 (MoE) 模型以及类比提示，我们能够显示与标准 LLM 基线相比改进的模型性能。我们还调查了这些提示技术的局限性及其对模型性能的影响。此外，我们提出了类比 CoT 提示，这是一种提示技术，旨在让较小的开源模型利用类比提示，这可能是由于缺乏专业训练数据而导致他们一直在努力解决的问题。

Title: A Practical Examination of AI-Generated Text Detectors for Large Language Models

• Authors: Brian Tufts, Xuandong Zhao, Lei Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.05139
• Pdf URL: https://arxiv.org/pdf/2412.05139
• Copy Paste: [[2412.05139]] A Practical Examination of AI-Generated Text Detectors for Large Language Models(https://arxiv.org/abs/2412.05139)
• Keywords: language model, gpt, prompt
• Abstract: The proliferation of large language models has raised growing concerns about their misuse, particularly in cases where AI-generated text is falsely attributed to human authors. Machine-generated content detectors claim to effectively identify such text under various conditions and from any language model. This paper critically evaluates these claims by assessing several popular detectors (RADAR, Wild, T5Sentinel, Fast-DetectGPT, GPTID, LogRank, Binoculars) on a range of domains, datasets, and models that these detectors have not previously encountered. We employ various prompting strategies to simulate adversarial attacks, demonstrating that even moderate efforts can significantly evade detection. We emphasize the importance of the true positive rate at a specific false positive rate (TPR@FPR) metric and demonstrate that these detectors perform poorly in certain settings, with TPR@.01 as low as 0\%. Our findings suggest that both trained and zero-shot detectors struggle to maintain high sensitivity while achieving a reasonable true positive rate.
• 摘要：大型语言模型的激增引发了人们对其滥用的担忧，尤其是在 AI 生成的文本被错误地归因于人类作者的情况下。机器生成的内容检测器声称可以在各种条件下和任何语言模型中有效识别此类文本。本文通过评估几种流行的检测器（RADAR、Wild、T5Sentinel、Fast-DetectGPT、GPTID、LogRank、Binoculars）在这些检测器以前从未遇到过的一系列域、数据集和模型上对这些说法进行了批判性评估。我们采用各种提示策略来模拟对抗性攻击，表明即使是适度的努力也可以显着逃避检测。我们强调特定假阳性率（TPR@FPR）指标下的真正阳性率的重要性，并证明这些检测器在某些设置下表现不佳，TPR@.01 低至 0\%。我们的研究结果表明，经过训练和零样本检测器都难以在实现合理的真正阳性率的同时保持高灵敏度。

Title: Explingo: Explaining AI Predictions using Large Language Models

• Authors: Alexandra Zytek, Sara Pido, Sarah Alnegheimish, Laure Berti-Equille, Kalyan Veeramachaneni
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.05145
• Pdf URL: https://arxiv.org/pdf/2412.05145
• Copy Paste: [[2412.05145]] Explingo: Explaining AI Predictions using Large Language Models(https://arxiv.org/abs/2412.05145)
• Keywords: language model, llm
• Abstract: Explanations of machine learning (ML) model predictions generated by Explainable AI (XAI) techniques such as SHAP are essential for people using ML outputs for decision-making. We explore the potential of Large Language Models (LLMs) to transform these explanations into human-readable, narrative formats that align with natural communication. We address two key research questions: (1) Can LLMs reliably transform traditional explanations into high-quality narratives? and (2) How can we effectively evaluate the quality of narrative explanations? To answer these questions, we introduce Explingo, which consists of two LLM-based subsystems, a Narrator and Grader. The Narrator takes in ML explanations and transforms them into natural-language descriptions. The Grader scores these narratives on a set of metrics including accuracy, completeness, fluency, and conciseness. Our experiments demonstrate that LLMs can generate high-quality narratives that achieve high scores across all metrics, particularly when guided by a small number of human-labeled and bootstrapped examples. We also identified areas that remain challenging, in particular for effectively scoring narratives in complex domains. The findings from this work have been integrated into an open-source tool that makes narrative explanations available for further applications.
• 摘要：对于使用 ML 输出进行决策的人们来说，由可解释人工智能 (XAI) 技术（例如 SHAP）生成的机器学习 (ML) 模型预测的解释至关重要。我们探索大型语言模型 (LLM) 将这些解释转换为与自然交流相一致的人类可读叙述格式的潜力。我们解决了两个关键研究问题：(1) LLM 能否可靠地将传统解释转化为高质量的叙述？(2) 我们如何有效地评估叙述解释的质量？为了回答这些问题，我们引入了 Explingo，它由两个基于 LLM 的子系统组成，即叙述者和评分者。叙述者接受 ML 解释并将其转换为自然语言描述。评分者根据准确性、完整性、流畅性和简洁性等一系列指标对这些叙述进行评分。我们的实验表明，LLM 可以生成在所有指标上都获得高分的高质量叙述，尤其是在少量人工标记和引导示例的指导下。我们还发现了一些仍具挑战性的领域，尤其是如何有效地对复杂领域的叙述进行评分。这项工作的成果已被整合到一个开源工具中，该工具使叙述解释可用于进一步的应用。

Title: Findings of the Second BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora

• Authors: Michael Y. Hu, Aaron Mueller, Candace Ross, Adina Williams, Tal Linzen, Chengxu Zhuang, Ryan Cotterell, Leshem Choshen, Alex Warstadt, Ethan Gotlieb Wilcox
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.05149
• Pdf URL: https://arxiv.org/pdf/2412.05149
• Copy Paste: [[2412.05149]] Findings of the Second BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora(https://arxiv.org/abs/2412.05149)
• Keywords: language model
• Abstract: The BabyLM Challenge is a community effort to close the data-efficiency gap between human and computational language learners. Participants compete to optimize language model training on a fixed language data budget of 100 million words or less. This year, we released improved text corpora, as well as a vision-and-language corpus to facilitate research into cognitively plausible vision language models. Submissions were compared on evaluation tasks targeting grammatical ability, (visual) question answering, pragmatic abilities, and grounding, among other abilities. Participants could submit to a 10M-word text-only track, a 100M-word text-only track, and/or a 100M-word and image multimodal track. From 31 submissions employing diverse methods, a hybrid causal-masked language model architecture outperformed other approaches. No submissions outperformed the baselines in the multimodal track. In follow-up analyses, we found a strong relationship between training FLOPs and average performance across tasks, and that the best-performing submissions proposed changes to the training data, training objective, and model architecture. This year's BabyLM Challenge shows that there is still significant room for innovation in this setting, in particular for image-text modeling, but community-driven research can yield actionable insights about effective strategies for small-scale language modeling.
• 摘要：BabyLM 挑战赛是一项社区活动，旨在缩小人类和计算机语言学习者之间的数据效率差距。参赛者竞相在 1 亿字或更少的固定语言数据预算上优化语言模型训练。今年，我们发布了改进的文本语料库以及视觉和语言语料库，以促进对认知上合理的视觉语言模型的研究。参赛作品在针对语法能力、（视觉）问答、语用能力和基础能力等评估任务上进行了比较。参赛者可以提交 10M 字纯文本轨道、100M 字纯文本轨道和/或 100M 字和图像多模态轨道。从 31 份采用不同方法的参赛作品中，混合因果掩蔽语言模型架构的表现优于其他方法。没有参赛作品的表现优于多模态轨道中的基线。在后续分析中，我们发现训练 FLOP 与各项任务的平均性能之间存在密切的关系，并且表现最佳的提交内容提出了对训练数据、训练目标和模型架构的更改。今年的 BabyLM 挑战赛表明，这种设置仍有很大创新空间，尤其是在图像文本建模方面，但社区驱动的研究可以产生关于小规模语言建模有效策略的可行见解。

Title: Multimodal Fact-Checking with Vision Language Models: A Probing Classifier based Solution with Embedding Strategies

• Authors: Recep Firat Cekinel, Pinar Karagoz, Cagri Coltekin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.05155
• Pdf URL: https://arxiv.org/pdf/2412.05155
• Copy Paste: [[2412.05155]] Multimodal Fact-Checking with Vision Language Models: A Probing Classifier based Solution with Embedding Strategies(https://arxiv.org/abs/2412.05155)
• Keywords: language model
• Abstract: This study evaluates the effectiveness of Vision Language Models (VLMs) in representing and utilizing multimodal content for fact-checking. To be more specific, we investigate whether incorporating multimodal content improves performance compared to text-only models and how well VLMs utilize text and image information to enhance misinformation detection. Furthermore we propose a probing classifier based solution using VLMs. Our approach extracts embeddings from the last hidden layer of selected VLMs and inputs them into a neural probing classifier for multi-class veracity classification. Through a series of experiments on two fact-checking datasets, we demonstrate that while multimodality can enhance performance, fusing separate embeddings from text and image encoders yielded superior results compared to using VLM embeddings. Furthermore, the proposed neural classifier significantly outperformed KNN and SVM baselines in leveraging extracted embeddings, highlighting its effectiveness for multimodal fact-checking.
• 摘要：本研究评估了视觉语言模型 (VLM) 在表示和利用多模态内容进行事实核查方面的有效性。更具体地说，我们研究了与纯文本模型相比，加入多模态内容是否会提高性能，以及 VLM 如何很好地利用文本和图像信息来增强错误信息检测。此外，我们提出了一种基于 VLM 的探测分类器解决方案。我们的方法从选定 VLM 的最后一个隐藏层中提取嵌入，并将其输入到神经探测分类器中进行多类真实性分类。通过对两个事实核查数据集的一系列实验，我们证明了虽然多模态可以提高性能，但与使用 VLM 嵌入相比，融合来自文本和图像编码器的单独嵌入可以产生更好的结果。此外，所提出的神经分类器在利用提取的嵌入方面明显优于 KNN 和 SVM 基线，凸显了其对多模态事实核查的有效性。

Title: QueEn: A Large Language Model for Quechua-English Translation

• Authors: Junhao Chen, Peng Shu, Yiwei Li, Huaqin Zhao, Hanqi Jiang, Yi Pan, Yifan Zhou, Zhengliang Liu, Lewis C Howe, Tianming Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.05184
• Pdf URL: https://arxiv.org/pdf/2412.05184
• Copy Paste: [[2412.05184]] QueEn: A Large Language Model for Quechua-English Translation(https://arxiv.org/abs/2412.05184)
• Keywords: language model, gpt, llm, retrieval-augmented generation
• Abstract: Recent studies show that large language models (LLMs) are powerful tools for working with natural language, bringing advances in many areas of computational linguistics. However, these models face challenges when applied to low-resource languages due to limited training data and difficulty in understanding cultural nuances. In this paper, we propose QueEn, a novel approach for Quechua-English translation that combines Retrieval-Augmented Generation (RAG) with parameter-efficient fine-tuning techniques. Our method leverages external linguistic resources through RAG and uses Low-Rank Adaptation (LoRA) for efficient model adaptation. Experimental results show that our approach substantially exceeds baseline models, with a BLEU score of 17.6 compared to 1.5 for standard GPT models. The integration of RAG with fine-tuning allows our system to address the challenges of low-resource language translation while maintaining computational efficiency. This work contributes to the broader goal of preserving endangered languages through advanced language technologies.
• 摘要：最近的研究表明，大型语言模型 (LLM) 是处理自然语言的强大工具，为计算语言学的许多领域带来了进步。然而，由于训练数据有限，并且难以理解文化差异，这些模型在应用于资源匮乏的语言时面临挑战。在本文中，我们提出了 QueEn，这是一种将检索增强生成 (RAG) 与参数高效的微调技术相结合的克丘亚语-英语翻译新方法。我们的方法通过 RAG 利用外部语言资源，并使用低秩自适应 (LoRA) 实现高效的模型自适应。实验结果表明，我们的方法大大超过了基线模型，BLEU 得分为 17.6，而标准 GPT 模型为 1.5。RAG 与微调的结合使我们的系统能够应对资源匮乏的语言翻译挑战，同时保持计算效率。这项工作有助于通过先进的语言技术保护濒危语言的更广泛目标。

Title: ConQRet: Benchmarking Fine-Grained Evaluation of Retrieval Augmented Argumentation with LLM Judges

• Authors: Kaustubh D. Dhole, Kai Shu, Eugene Agichtein
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2412.05206
• Pdf URL: https://arxiv.org/pdf/2412.05206
• Copy Paste: [[2412.05206]] ConQRet: Benchmarking Fine-Grained Evaluation of Retrieval Augmented Argumentation with LLM Judges(https://arxiv.org/abs/2412.05206)
• Keywords: llm, retrieval-augmented generation
• Abstract: Computational argumentation, which involves generating answers or summaries for controversial topics like abortion bans and vaccination, has become increasingly important in today's polarized environment. Sophisticated LLM capabilities offer the potential to provide nuanced, evidence-based answers to such questions through Retrieval-Augmented Argumentation (RAArg), leveraging real-world evidence for high-quality, grounded arguments. However, evaluating RAArg remains challenging, as human evaluation is costly and difficult for complex, lengthy answers on complicated topics. At the same time, re-using existing argumentation datasets is no longer sufficient, as they lack long, complex arguments and realistic evidence from potentially misleading sources, limiting holistic evaluation of retrieval effectiveness and argument quality. To address these gaps, we investigate automated evaluation methods using multiple fine-grained LLM judges, providing better and more interpretable assessments than traditional single-score metrics and even previously reported human crowdsourcing. To validate the proposed techniques, we introduce ConQRet, a new benchmark featuring long and complex human-authored arguments on debated topics, grounded in real-world websites, allowing an exhaustive evaluation across retrieval effectiveness, argument quality, and groundedness. We validate our LLM Judges on a prior dataset and the new ConQRet benchmark. Our proposed LLM Judges and the ConQRet benchmark can enable rapid progress in computational argumentation and can be naturally extended to other complex retrieval-augmented generation tasks.
• 摘要：计算论证涉及为诸如堕胎禁令和疫苗接种等有争议的话题生成答案或摘要，在当今两极分化的环境中变得越来越重要。先进的 LLM 功能可以通过检索增强论证 (RAArg) 为这些问题提供细致入微的、基于证据的答案，利用现实世界的证据进行高质量、有根据的论证。然而，评估 RAArg 仍然具有挑战性，因为对于复杂主题的复杂、冗长的答案，人工评估成本高昂且困难重重。同时，重复使用现有的论证数据集已不再足够，因为它们缺乏来自潜在误导性来源的长篇复杂论证和现实证据，限制了对检索有效性和论证质量的整体评估。为了解决这些差距，我们研究了使用多个细粒度 LLM 评委的自动评估方法，提供比传统的单分数指标甚至之前报道的人工众包更好、更可解释的评估。为了验证所提出的技术，我们引入了 ConQRet，这是一个新的基准，以现实世界的网站为基础，针对有争议的话题，以冗长而复杂的人工论证为特色，可以对检索有效性、论证质量和立足性进行详尽的评估。我们在先前的数据集和新的 ConQRet 基准上验证了我们的 LLM Judges。我们提出的 LLM Judges 和 ConQRet 基准可以促进计算论证的快速发展，并且可以自然地扩展到其他复杂的检索增强生成任务。

Title: Evaluating and Aligning CodeLLMs on Human Preference

• Authors: Jian Yang, Jiaxi Yang, Ke Jin, Yibo Miao, Lei Zhang, Liqun Yang, Zeyu Cui, Yichang Zhang, Binyuan Hui, Junyang Lin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.05210
• Pdf URL: https://arxiv.org/pdf/2412.05210
• Copy Paste: [[2412.05210]] Evaluating and Aligning CodeLLMs on Human Preference(https://arxiv.org/abs/2412.05210)
• Keywords: language model, llm
• Abstract: Code large language models (codeLLMs) have made significant strides in code generation. Most previous code-related benchmarks, which consist of various programming exercises along with the corresponding test cases, are used as a common measure to evaluate the performance and capabilities of code LLMs. However, the current code LLMs focus on synthesizing the correct code snippet, ignoring the alignment with human preferences, where the query should be sampled from the practical application scenarios and the model-generated responses should satisfy the human preference. To bridge the gap between the model-generated response and human preference, we present a rigorous human-curated benchmark CodeArena to emulate the complexity and diversity of real-world coding tasks, where 397 high-quality samples spanning 40 categories and 44 programming languages, carefully curated from user queries. Further, we propose a diverse synthetic instruction corpus SynCode-Instruct (nearly 20B tokens) by scaling instructions from the website to verify the effectiveness of the large-scale synthetic instruction fine-tuning, where Qwen2.5-SynCoder totally trained on synthetic instruction data can achieve top-tier performance of open-source code LLMs. The results find performance differences between execution-based benchmarks and CodeArena. Our systematic experiments of CodeArena on 40+ LLMs reveal a notable performance gap between open SOTA code LLMs (e.g. Qwen2.5-Coder) and proprietary LLMs (e.g., OpenAI o1), underscoring the importance of the human preference alignment.\footnote{\url{this https URL }}
• 摘要：代码大型语言模型 (codeLLM) 在代码生成方面取得了重大进展。大多数以前的代码相关基准测试由各种编程练习以及相应的测试用例组成，被用作评估代码 LLM 性能和功能的通用指标。然而，当前的代码 LLM 专注于合成正确的代码片段，而忽略了与人类偏好的一致性，其中查询应从实际应用场景中采样，模型生成的响应应满足人类偏好。为了弥合模型生成的响应和人类偏好之间的差距，我们提出了一个严格的人工策划的基准测试 CodeArena 来模拟现实世界编码任务的复杂性和多样性，其中 397 个高质量样本涵盖 40 个类别和 44 种编程语言，是从用户查询中精心策划的。此外，我们通过扩展网站上的指令提出了一个多样化的合成指令语料库 SynCode-Instruct（近 20B 个 token），以验证大规模合成指令微调的有效性，其中完全在合成指令数据上训练的 Qwen2.5-SynCoder 可以实现开源代码 LLM 的顶级性能。结果发现基于执行的基准和 CodeArena 之间存在性能差异。我们在 40 多个 LLM 上对 CodeArena 进行了系统性实验，揭示了开放 SOTA 代码 LLM（例如 Qwen2.5-Coder）和专有 LLM（例如 OpenAI o1）之间存在显着的性能差距，强调了人类偏好一致性的重要性。\footnote{\url{this https URL }}

Title: 100% Hallucination Elimination Using Acurai

• Authors: Michael C. Wood, Adam A. Forbes
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.05223
• Pdf URL: https://arxiv.org/pdf/2412.05223
• Copy Paste: [[2412.05223]] 100% Hallucination Elimination Using Acurai(https://arxiv.org/abs/2412.05223)
• Keywords: language model, gpt, llm, hallucination, retrieval-augmented generation
• Abstract: The issue of hallucinations in large language models (LLMs) remains a critical barrier to the adoption of AI in enterprise and other high-stakes applications. Despite advancements in retrieval-augmented generation (RAG) systems, current state-of-the-art methods fail to achieve more than 80% accuracy in generating faithful and factually correct outputs, even when provided with relevant and accurate context. In this work, we introduce Acurai, a novel systematic approach that achieves 100% hallucination-free responses in LLMs by reformatting queries and context data prior to input. Leveraging a deep understanding of LLM internal representations, the importance of noun-phrase dominance, and the role of discrete functional units (DFUs), Acurai ensures alignment between input context and generated output. We validate this method using the RAGTruth corpus, demonstrating its ability to eliminate 100% hallucinations for both GPT-4 and GPT-3.5 Turbo. Acurai sets a new standard for achieving consistent, accurate, and faithful AI responses, marking a significant step forward in the development of trustworthy AI systems.
• 摘要：大型语言模型 (LLM) 中的幻觉问题仍然是企业和其他高风险应用中采用 AI 的关键障碍。尽管检索增强生成 (RAG) 系统取得了进步，但当前最先进的方法在生成忠实且事实正确的输出时无法达到 80% 以上的准确率，即使在提供相关且准确的上下文的情况下也是如此。在这项工作中，我们介绍了 Acurai，这是一种新颖的系统方法，通过在输入之前重新格式化查询和上下文数据，在 LLM 中实现 100% 无幻觉的响应。利用对 LLM 内部表示、名词短语优势的重要性以及离散功能单元 (DFU) 的作用的深刻理解，Acurai 可确保输入上下文和生成的输出之间的一致性。我们使用 RAGTruth 语料库验证了此方法，证明了它能够消除 GPT-4 和 GPT-3.5 Turbo 的 100% 幻觉。 Acurai 为实现一致、准确和忠实的 AI 响应设立了新标准，标志着可信 AI 系统开发向前迈出了重要一步。

Title: BEExformer: A Fast Inferencing Transformer Architecture via Binarization with Multiple Early Exits

• Authors: Wazib Ansar, Saptarsi Goswami, Amlan Chakrabarti
• Subjects: cs.CL, cs.AI, cs.NE
• Abstract URL: https://arxiv.org/abs/2412.05225
• Pdf URL: https://arxiv.org/pdf/2412.05225
• Copy Paste: [[2412.05225]] BEExformer: A Fast Inferencing Transformer Architecture via Binarization with Multiple Early Exits(https://arxiv.org/abs/2412.05225)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) based on transformers achieve cutting-edge results on a variety of applications. However, their enormous size and processing requirements make deployment on devices with constrained resources extremely difficult. Among various efficiency considerations, model binarization and Early Exit (EE) are common effective solutions. However, binarization may lead to performance loss due to reduced precision affecting gradient estimation and parameter updates. Besides, the present early-exit mechanisms are still in the nascent stages of research. To ameliorate these issues, we propose Binarized Early Exit Transformer (BEExformer), the first-ever selective learning transformer architecture to combine early exit with binarization for textual inference. It improves the binarization process through a differentiable second-order approximation to the impulse function. This enables gradient computation concerning both the sign as well as the magnitude of the weights. In contrast to absolute threshold-based EE, the proposed EE mechanism hinges on fractional reduction in entropy among intermediate transformer blocks with soft-routing loss estimation. While binarization results in 18.44 times reduction in model size, early exit reduces the FLOPs during inference by 54.85% and even improves accuracy by 5.98% through resolving the "overthinking" problem inherent in deep networks. Moreover, the proposed BEExformer simplifies training by not requiring knowledge distillation from a full-precision LLM. Extensive evaluation on the GLUE dataset and comparison with the SOTA works showcase its pareto-optimal performance-efficiency trade-off.
• 摘要：基于 Transformer 的大型语言模型 (LLM) 在各种应用上都取得了前沿成果。然而，它们巨大的规模和处理要求使得在资源受限的设备上部署变得极其困难。在各种效率考虑中，模型二值化和提前退出 (EE) 是常见的有效解决方案。然而，二值化可能会导致性能损失，因为精度降低会影响梯度估计和参数更新。此外，目前的提前退出机制仍处于研究的初期阶段。为了改善这些问题，我们提出了二值化提前退出 Transformer (BEExformer)，这是有史以来第一个将提前退出与二值化相结合进行文本推理的选择性学习 Transformer 架构。它通过对脉冲函数的可微二阶近似来改进二值化过程。这使得能够计算与权重的符号和幅度有关的梯度。与基于绝对阈值的 EE 相比，所提出的 EE 机制取决于中间 Transformer 块之间的熵的分数减少和软路由损失估计。虽然二值化导致模型大小缩小了 18.44 倍，但提前退出可将推理过程中的 FLOP 减少 54.85%，甚至通过解决深度网络固有的“过度思考”问题将准确率提高 5.98%。此外，提出的 BEExformer 无需从全精度 LLM 中进行知识提炼，从而简化了训练。对 GLUE 数据集的广泛评估以及与 SOTA 作品的比较展示了其帕累托最优性能效率权衡。

Title: LIAR: Leveraging Alignment (Best-of-N) to Jailbreak LLMs in Seconds

• Authors: James Beetham, Souradip Chakraborty, Mengdi Wang, Furong Huang, Amrit Singh Bedi, Mubarak Shah
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.05232
• Pdf URL: https://arxiv.org/pdf/2412.05232
• Copy Paste: [[2412.05232]] LIAR: Leveraging Alignment (Best-of-N) to Jailbreak LLMs in Seconds(https://arxiv.org/abs/2412.05232)
• Keywords: llm, prompt
• Abstract: Many existing jailbreak techniques rely on solving discrete combinatorial optimization, while more recent approaches involve training LLMs to generate multiple adversarial prompts. However, both approaches require significant computational resources to produce even a single adversarial prompt. We hypothesize that the inefficiency of current approaches stems from an inadequate characterization of the jailbreak problem. To address this gap, we formulate the jailbreak problem in terms of alignment. By starting from an available safety-aligned model, we leverage an unsafe reward to guide the safe model towards generating unsafe outputs using alignment techniques (e.g., reinforcement learning from human feedback), effectively performing jailbreaking via alignment. We propose a novel jailbreak method called LIAR (LeveragIng Alignment to jailbReak). To demonstrate the simplicity and effectiveness of our approach, we employ a best-of-N method to solve the alignment problem. LIAR offers significant advantages: lower computational requirements without additional training, fully black-box operation, competitive attack success rates, and more human-readable prompts. We provide theoretical insights into the possibility of jailbreaking a safety-aligned model, revealing inherent vulnerabilities in current alignment strategies for LLMs. We also provide sub-optimality guarantees for the proposed \algo. Experimentally, we achieve ASR comparable to the SoTA with a 10x improvement to perplexity and a Time-to-Attack measured in seconds rather than tens of hours.
• 摘要：许多现有的越狱技术依赖于解决离散组合优化问题，而较新的方法则涉及训练 LLM 以生成多个对抗性提示。然而，这两种方法都需要大量的计算资源才能产生哪怕一个对抗性提示。我们假设，当前方法的低效率源于对越狱问题的描述不充分。为了解决这一差距，我们从对齐的角度来阐述越狱问题。从可用的安全对齐模型开始，我们利用不安全奖励来引导安全模型使用对齐技术（例如，从人类反馈中进行强化学习）生成不安全输出，从而有效地通过对齐执行越狱。我们提出了一种称为 LIAR（利用对齐来越狱）的新型越狱方法。为了证明我们方法的简单性和有效性，我们采用了一种最佳 N 方法来解决对齐问题。LIAR 具有显着的优势：无需额外训练即可降低计算要求、完全黑盒操作、具有竞争力的攻击成功率以及更人性化的提示。我们从理论上洞察了越狱安全对齐模型的可能性，揭示了当前 LLM 对齐策略中固有的漏洞。我们还为所提出的 \algo 提供了次优保证。通过实验，我们实现了与 SoTA 相当的 ASR，困惑度提高了 10 倍，攻击时间以秒为单位，而不是数十小时。

Title: MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scale

• Authors: Jarvis Guo, Tuney Zheng, Yuelin Bai, Bo Li, Yubo Wang, King Zhu, Yizhi Li, Graham Neubig, Wenhu Chen, Xiang Yue
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2412.05237
• Pdf URL: https://arxiv.org/pdf/2412.05237
• Copy Paste: [[2412.05237]] MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scale(https://arxiv.org/abs/2412.05237)
• Keywords: language model, llm
• Abstract: Open-source multimodal large language models (MLLMs) have shown significant potential in a broad range of multimodal tasks. However, their reasoning capabilities remain constrained by existing instruction-tuning datasets, which were predominately repurposed from academic datasets such as VQA, AI2D, and ChartQA. These datasets target simplistic tasks, and only provide phrase-level answers without any intermediate rationales. To address these challenges, we introduce a scalable and cost-effective method to construct a large-scale multimodal instruction-tuning dataset with rich intermediate rationales designed to elicit CoT reasoning. Using only open models, we create a dataset containing 12M instruction-response pairs to cover diverse, reasoning-intensive tasks with detailed and faithful rationales. Experiments demonstrate that training MLLMs on this dataset significantly improves reasoning capabilities, achieving state-of-the-art performance on benchmarks such as MathVerse (+8.1%), MMMU-Pro (+7%), and MuirBench (+13.3%). Additionally, the model demonstrates notable improvements of up to 4% on non-reasoning-based benchmarks. Ablation studies further highlight the importance of key components, such as rewriting and self-filtering, in the dataset construction process.
• 摘要：开源多模态大型语言模型 (MLLM) 在广泛的多模态任务中显示出巨大的潜力。然而，它们的推理能力仍然受到现有指令调整数据集的限制，这些数据集主要从学术数据集（例如 VQA、AI2D 和 ChartQA）中重新利用。这些数据集针对简单的任务，仅提供短语级答案而没有任何中间原理。为了应对这些挑战，我们引入了一种可扩展且经济高效的方法来构建一个大规模多模态指令调整数据集，该数据集具有丰富的中间原理，旨在引出 CoT 推理。仅使用开放模型，我们创建了一个包含 1200 万个指令-响应对的数据集，以详细而忠实的原理涵盖各种推理密集型任务。实验表明，在这个数据集上训练 MLLM 可显著提高推理能力，在 MathVerse（+8.1%）、MMMU-Pro（+7%）和 MuirBench（+13.3%）等基准测试中实现最先进的性能。此外，该模型在非推理基准上表现出高达 4% 的显著改进。消融研究进一步强调了重写和自我过滤等关键组件在数据集构建过程中的重要性。