2024-10-31

Title: ArxivDIGESTables: Synthesizing Scientific Literature into Tables using Language Models

• Authors: Benjamin Newman, Yoonjoo Lee, Aakanksha Naik, Pao Siangliulue, Raymond Fok, Juho Kim, Daniel S. Weld, Joseph Chee Chang, Kyle Lo
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.22360
• Pdf URL: https://arxiv.org/pdf/2410.22360
• Copy Paste: [[2410.22360]] ArxivDIGESTables: Synthesizing Scientific Literature into Tables using Language Models(https://arxiv.org/abs/2410.22360)
• Keywords: language model
• Abstract: When conducting literature reviews, scientists often create literature review tables - tables whose rows are publications and whose columns constitute a schema, a set of aspects used to compare and contrast the papers. Can we automatically generate these tables using language models (LMs)? In this work, we introduce a framework that leverages LMs to perform this task by decomposing it into separate schema and value generation steps. To enable experimentation, we address two main challenges: First, we overcome a lack of high-quality datasets to benchmark table generation by curating and releasing arxivDIGESTables, a new dataset of 2,228 literature review tables extracted from ArXiv papers that synthesize a total of 7,542 research papers. Second, to support scalable evaluation of model generations against human-authored reference tables, we develop DecontextEval, an automatic evaluation method that aligns elements of tables with the same underlying aspects despite differing surface forms. Given these tools, we evaluate LMs' abilities to reconstruct reference tables, finding this task benefits from additional context to ground the generation (e.g. table captions, in-text references). Finally, through a human evaluation study we find that even when LMs fail to fully reconstruct a reference table, their generated novel aspects can still be useful.
• 摘要：在进行文献综述时，科学家经常会创建文献综述表 - 行表示出版物，列表示模式，即用于比较和对比论文的一组方面。我们可以使用语言模型 (LM) 自动生成这些表格吗？在这项工作中，我们引入了一个框架，该框架利用 LM 将此任务分解为单独的模式和值生成步骤来执行此任务。为了进行实验，我们解决了两个主要挑战：首先，我们通过策划和发布 arxivDIGESTables 来克服缺乏高质量数据集来对表格生成进行基准测试的问题，arxivDIGESTables 是从 ArXiv 论文中提取的 2,228 个文献综述表的新数据集，这些论文总共综合了 7,542 篇研究论文。其次，为了支持对模型生成与人工编写的参考表进行可扩展的评估，我们开发了 DecontextEval，这是一种自动评估方法，尽管表面形式不同，但它可以将具有相同底层方面的表格元素对齐。有了这些工具，我们评估了 LM 重建参考表的能力，发现这项任务受益于为生成打下基础的额外背景信息（例如表格标题、文内引用）。最后，通过一项人工评估研究，我们发现即使 LM 无法完全重建参考表，它们生成的新颖方面仍然很有用。

Title: AAAR-1.0: Assessing AI's Potential to Assist Research

• Authors: Renze Lou, Hanzi Xu, Sijia Wang, Jiangshu Du, Ryo Kamoi, Xiaoxin Lu, Jian Xie, Yuxuan Sun, Yusen Zhang, Jihyun Janice Ahn, Hongchao Fang, Zhuoyang Zou, Wenchao Ma, Xi Li, Kai Zhang, Congying Xia, Lifu Huang, Wenpeng Yin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.22394
• Pdf URL: https://arxiv.org/pdf/2410.22394
• Copy Paste: [[2410.22394]] AAAR-1.0: Assessing AI's Potential to Assist Research(https://arxiv.org/abs/2410.22394)
• Keywords: language model, llm
• Abstract: Numerous studies have assessed the proficiency of AI systems, particularly large language models (LLMs), in facilitating everyday tasks such as email writing, question answering, and creative content generation. However, researchers face unique challenges and opportunities in leveraging LLMs for their own work, such as brainstorming research ideas, designing experiments, and writing or reviewing papers. In this study, we introduce AAAR-1.0, a benchmark dataset designed to evaluate LLM performance in three fundamental, expertise-intensive research tasks: (i) EquationInference, assessing the correctness of equations based on the contextual information in paper submissions; (ii) ExperimentDesign, designing experiments to validate research ideas and solutions; (iii) PaperWeakness, identifying weaknesses in paper submissions; and (iv) REVIEWCRITIQUE, identifying each segment in human reviews is deficient or not. AAAR-1.0 differs from prior benchmarks in two key ways: first, it is explicitly research-oriented, with tasks requiring deep domain expertise; second, it is researcher-oriented, mirroring the primary activities that researchers engage in on a daily basis. An evaluation of both open-source and proprietary LLMs reveals their potential as well as limitations in conducting sophisticated research tasks. We will keep iterating AAAR-1.0 to new versions.
• 摘要：许多研究已经评估了人工智能系统，特别是大型语言模型 (LLM) 在执行日常任务（例如撰写电子邮件、回答问题和生成创意内容）方面的能力。然而，研究人员在利用 LLM 完成自己的工作方面面临着独特的挑战和机遇，例如集思广益研究想法、设计实验以及撰写或审阅论文。在本研究中，我们引入了 AAAR-1.0，这是一个基准数据集，旨在评估 LLM 在三个基本、专业知识密集型研究任务中的表现：(i) EquationInference，根据论文提交中的上下文信息评估方程的正确性；(ii) ExperimentDesign，设计实验来验证研究想法和解决方案；(iii) PaperWeakness，识别论文提交中的弱点；(iv) REVIEWCRITIQUE，识别人工评审中每个部分是否存在缺陷。AAAR-1.0 在两个主要方面不同于之前的基准：首先，它明确以研究为导向，任务需要深厚的领域专业知识；第二，它以研究人员为导向，反映了研究人员每天从事的主要活动。对开源和专有法学硕士的评估揭示了它们在执行复杂研究任务方面的潜力和局限性。我们将继续将 AAAR-1.0 迭代到新版本。

Title: Do Large Language Models Align with Core Mental Health Counseling Competencies?

• Authors: Viet Cuong Nguyen, Mohammad Taher, Dongwan Hong, Vinicius Konkolics Possobom, Vibha Thirunellayi Gopalakrishnan, Ekta Raj, Zihang Li, Heather J. Soled, Michael L. Birnbaum, Srijan Kumar, Munmun De Choudhury
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.22446
• Pdf URL: https://arxiv.org/pdf/2410.22446
• Copy Paste: [[2410.22446]] Do Large Language Models Align with Core Mental Health Counseling Competencies?(https://arxiv.org/abs/2410.22446)
• Keywords: language model, llm
• Abstract: The rapid evolution of Large Language Models (LLMs) offers promising potential to alleviate the global scarcity of mental health professionals. However, LLMs' alignment with essential mental health counseling competencies remains understudied. We introduce CounselingBench, a novel NCMHCE-based benchmark evaluating LLMs across five key mental health counseling competencies. Testing 22 general-purpose and medical-finetuned LLMs, we find frontier models exceed minimum thresholds but fall short of expert-level performance, with significant variations: they excel in Intake, Assessment & Diagnosis yet struggle with Core Counseling Attributes and Professional Practice & Ethics. Medical LLMs surprisingly underperform generalist models accuracy-wise, while at the same time producing slightly higher-quality justifications but making more context-related errors. Our findings highlight the complexities of developing AI systems for mental health counseling, particularly for competencies requiring empathy and contextual understanding. We found that frontier LLMs perform at a level exceeding the minimal required level of aptitude for all key mental health counseling competencies, but fall short of expert-level performance, and that current medical LLMs do not significantly improve upon generalist models in mental health counseling competencies. This underscores the critical need for specialized, mental health counseling-specific fine-tuned LLMs that rigorously aligns with core competencies combined with appropriate human supervision before any responsible real-world deployment can be considered.
• 摘要：大型语言模型 (LLM) 的快速发展为缓解全球心理健康专业人员短缺提供了巨大的潜力。然而，LLM 与基本心理健康咨询能力的契合度仍未得到充分研究。我们推出了 CounselingBench，这是一种基于 NCMHCE 的新型基准，用于评估五项关键心理健康咨询能力的 LLM。通过测试 22 个通用和医学微调的 LLM，我们发现前沿模型超过了最低阈值，但未达到专家级性能，并且存在显著差异：它们在接收、评估和诊断方面表现出色，但在核心咨询属性和专业实践与道德方面却举步维艰。医学 LLM 在准确度方面的表现令人惊讶地不如通用模型，同时产生的论证质量略高，但会犯更多与上下文相关的错误。我们的研究结果凸显了开发心理健康咨询 AI 系统的复杂性，尤其是对于需要同理心和情境理解的能力。我们发现，前沿法学硕士的表现超过了所有关键心理健康咨询能力的最低要求水平，但未达到专家级的表现，而且目前的医学法学硕士并没有显著提高心理健康咨询能力的通才模型。这强调了在考虑任何负责任的实际部署之前，迫切需要专门的、针对心理健康咨询的微调法学硕士，这些法学硕士必须与核心能力严格一致，并结合适当的人工监督。

Title: Scaling LLM Inference with Optimized Sample Compute Allocation

• Authors: Kexun Zhang, Shang Zhou, Danqing Wang, William Yang Wang, Lei Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.22480
• Pdf URL: https://arxiv.org/pdf/2410.22480
• Copy Paste: [[2410.22480]] Scaling LLM Inference with Optimized Sample Compute Allocation(https://arxiv.org/abs/2410.22480)
• Keywords: language model, llm, agent
• Abstract: Sampling is a basic operation in many inference-time algorithms of large language models (LLMs). To scale up inference efficiently with a limited compute, it is crucial to find an optimal allocation for sample compute budgets: Which sampling configurations (model, temperature, language, etc.) do we use? How many samples do we generate in each configuration? We formulate these choices as a learning problem and propose OSCA, an algorithm that Optimizes Sample Compute Allocation by finding an optimal mix of different inference configurations. Our experiments show that with our learned mixed allocation, we can achieve accuracy better than the best single configuration with 128x less compute on code generation and 25x less compute on 4 reasoning tasks. OSCA is also shown to be effective in agentic workflows beyond single-turn tasks, achieving a better accuracy on SWE-Bench with 3x less compute than the default configuration. Our code and generations are released at this https URL.
• 摘要：采样是许多大型语言模型 (LLM) 推理时间算法的基本操作。为了在有限的计算量下有效扩展推理，找到样本计算预算的最佳分配至关重要：我们使用哪些采样配置（模型、温度、语言等）？我们在每种配置中生成多少个样本？我们将这些选择表述为学习问题，并提出 OSCA，这是一种通过找到不同推理配置的最佳组合来优化样本计算分配的算法。我们的实验表明，通过我们学习到的混合分配，我们可以实现比最佳单一配置更好的准确度，代码生成计算量减少 128 倍，4 个推理任务计算量减少 25 倍。OSCA 还被证明在单轮任务之外的代理工作流中有效，在 SWE-Bench 上实现更好的准确度，计算量比默认配置少 3 倍。我们的代码和生成发布在此 https URL 上。

Title: Anticipating Future with Large Language Model for Simultaneous Machine Translation

• Authors: Siqi Ouyang, Oleksii Hrinchuk, Zhehuai Chen, Vitaly Lavrukhin, Jagadeesh Balam, Lei Li, Boris Ginsburg
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.22499
• Pdf URL: https://arxiv.org/pdf/2410.22499
• Copy Paste: [[2410.22499]] Anticipating Future with Large Language Model for Simultaneous Machine Translation(https://arxiv.org/abs/2410.22499)
• Keywords: language model, llm
• Abstract: Simultaneous machine translation (SMT) takes streaming input utterances and incrementally produces target text. Existing SMT methods only use the partial utterance that has already arrived at the input and the generated hypothesis. Motivated by human interpreters' technique to forecast future words before hearing them, we propose $\textbf{T}$ranslation by $\textbf{A}$nticipating $\textbf{F}$uture (TAF), a method to improve translation quality while retraining low latency. Its core idea is to use a large language model (LLM) to predict future source words and opportunistically translate without introducing too much risk. We evaluate our TAF and multiple baselines of SMT on four language directions. Experiments show that TAF achieves the best translation quality-latency trade-off and outperforms the baselines by up to 5 BLEU points at the same latency (three words).
• 摘要：同步机器翻译 (SMT) 接收流式输入话语并逐步生成目标文本。现有的 SMT 方法仅使用已经到达输入的部分话语和生成的假设。受人类口译员在听到未来单词之前预测未来单词的技术启发，我们提出了 $\textbf{T}$translation by $\textbf{A}$nticipating $\textbf{F}$uture (TAF)，这是一种在重新训练低延迟的同时提高翻译质量的方法。其核心思想是使用大型语言模型 (LLM) 来预测未来的源单词并投机性地进行翻译而不会引入太多风险。我们在四个语言方向上评估了我们的 TAF 和 SMT 的多个基线。实验表明，TAF 实现了最佳的翻译质量-延迟权衡，并且在相同延迟（三个单词）下比基线高出多达 5 个 BLEU 点。

Title: Attention Speaks Volumes: Localizing and Mitigating Bias in Language Models

• Authors: Rishabh Adiga, Besmira Nushi, Varun Chandrasekaran
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.22517
• Pdf URL: https://arxiv.org/pdf/2410.22517
• Copy Paste: [[2410.22517]] Attention Speaks Volumes: Localizing and Mitigating Bias in Language Models(https://arxiv.org/abs/2410.22517)
• Keywords: language model, gpt, llm, prompt
• Abstract: We explore the internal mechanisms of how bias emerges in large language models (LLMs) when provided with ambiguous comparative prompts: inputs that compare or enforce choosing between two or more entities without providing clear context for preference. Most approaches for bias mitigation focus on either post-hoc analysis or data augmentation. However, these are transient solutions, without addressing the root cause: the model itself. Numerous prior works show the influence of the attention module towards steering generations. We believe that analyzing attention is also crucial for understanding bias, as it provides insight into how the LLM distributes its focus across different entities and how this contributes to biased decisions. To this end, we first introduce a metric to quantify the LLM's preference for one entity over another. We then propose $\texttt{ATLAS}$ (Attention-based Targeted Layer Analysis and Scaling), a technique to localize bias to specific layers of the LLM by analyzing attention scores and then reduce bias by scaling attention in these biased layers. To evaluate our method, we conduct experiments across 3 datasets (BBQ, Crows-Pairs, and WinoGender) using $\texttt{GPT-2 XL}$ (1.5B), $\texttt{GPT-J}$ (6B), $\texttt{LLaMA-2}$ (7B) and $\texttt{LLaMA-3}$ (8B). Our experiments demonstrate that bias is concentrated in the later layers, typically around the last third. We also show how $\texttt{ATLAS}$ effectively mitigates bias through targeted interventions without compromising downstream performance and an average increase of only 0.82% in perplexity when the intervention is applied. We see an average improvement of 0.28 points in the bias score across all the datasets.
• 摘要：我们探索了在大型语言模型 (LLM) 中，当提供模糊的比较提示时，偏见如何出现的内部机制：输入比较或强制在两个或多个实体之间进行选择，而没有提供明确的偏好背景。大多数缓解偏见的方法都侧重于事后分析或数据增强。然而，这些都是暂时的解决方案，并没有解决根本原因：模型本身。许多先前的研究展示了注意力模块对引导生成的影响。我们认为，分析注意力对于理解偏见也至关重要，因为它可以洞察 LLM 如何将其注意力分散到不同的实体上，以及这如何导致有偏见的决策。为此，我们首先引入一个指标来量化 LLM 对一个实体的偏好。然后，我们提出了 $\texttt{ATLAS}$（基于注意力的目标层分析和缩放），这是一种通过分析注意力分数将偏见定位到 LLM 的特定层，然后通过缩放这些有偏见的层中的注意力来减少偏见的技术。为了评估我们的方法，我们使用 $\texttt{GPT-2 XL}$（1.5B）、$\texttt{GPT-J}$（6B）、$\texttt{LLaMA-2}$（7B）和 $\texttt{LLaMA-3}$（8B）在 3 个数据集（BBQ、Crows-Pairs 和 WinoGender）上进行了实验。我们的实验表明，偏差集中在后面的层中，通常在最后三分之一左右。我们还展示了 $\texttt{ATLAS}$ 如何通过有针对性的干预措施有效地减轻偏差，而不会影响下游性能，并且在应用干预措施时，困惑度平均仅增加 0.82%。我们看到所有数据集的偏差分数平均提高了 0.28 分。

Title: Auto-Intent: Automated Intent Discovery and Self-Exploration for Large Language Model Web Agents

• Authors: Jaekyeom Kim, Dong-Ki Kim, Lajanugen Logeswaran, Sungryull Sohn, Honglak Lee
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.22552
• Pdf URL: https://arxiv.org/pdf/2410.22552
• Copy Paste: [[2410.22552]] Auto-Intent: Automated Intent Discovery and Self-Exploration for Large Language Model Web Agents(https://arxiv.org/abs/2410.22552)
• Keywords: language model, gpt, llm, agent
• Abstract: In this paper, we introduce Auto-Intent, a method to adapt a pre-trained large language model (LLM) as an agent for a target domain without direct fine-tuning, where we empirically focus on web navigation tasks. Our approach first discovers the underlying intents from target domain demonstrations unsupervisedly, in a highly compact form (up to three words). With the extracted intents, we train our intent predictor to predict the next intent given the agent's past observations and actions. In particular, we propose a self-exploration approach where top-k probable intent predictions are provided as a hint to the pre-trained LLM agent, which leads to enhanced decision-making capabilities. Auto-Intent substantially improves the performance of GPT-{3.5, 4} and Llama-3.1-{70B, 405B} agents on the large-scale real-website navigation benchmarks from Mind2Web and online navigation tasks from WebArena with its cross-benchmark generalization from Mind2Web.
• 摘要：在本文中，我们介绍了 Auto-Intent，这是一种无需直接微调即可将预训练的大型语言模型 (LLM) 调整为目标域代理的方法，我们根据经验专注于 Web 导航任务。我们的方法首先以高度紧凑的形式（最多三个单词）无监督地从目标域演示中发现潜在意图。利用提取的意图，我们训练我们的意图预测器，根据代理过去的观察和操作预测下一个意图。具体来说，我们提出了一种自我探索方法，其中将前 k 个可能的意图预测作为提示提供给预训练的 LLM 代理，从而增强决策能力。Auto-Intent 凭借其来自 Mind2Web 的跨基准泛化，在 Mind2Web 的大规模真实网站导航基准和 WebArena 的在线导航任务中显著提高了 GPT-{3.5, 4} 和 Llama-3.1-{70B, 405B} 代理的性能。

Title: Toxicity of the Commons: Curating Open-Source Pre-Training Data

• Authors: Catherine Arnett, Eliot Jones, Ivan P. Yamshchikov, Pierre-Carl Langlais
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.22587
• Pdf URL: https://arxiv.org/pdf/2410.22587
• Copy Paste: [[2410.22587]] Toxicity of the Commons: Curating Open-Source Pre-Training Data(https://arxiv.org/abs/2410.22587)
• Keywords: language model
• Abstract: Open-source large language models are becoming increasingly available and popular among researchers and practitioners. While significant progress has been made on open-weight models, open training data is a practice yet to be adopted by the leading open-weight models creators. At the same time, there researchers are working to make language models safer. We propose a data curation pipeline to reduce harmful outputs by models trained on public domain data. There are unique challenges to working with public domain data, as these sources differ from web text in both form and content. Many sources are historical documents and are the result of Optical Character Recognition (OCR). Consequently, current state-of-the-art approaches to toxicity filtering are often infeasible or inappropriate for open data models. In this paper, we introduce a new fully open-source pipeline for open-data toxicity filtering. Our contributions are threefold. We create a custom training dataset, ToxicCommons, which is composed of texts which have been classified across five different dimensions (racial/origin-based, gender/sex-based, religious, ability-based discrimination, and violence). We use this dataset to train a custom classifier, Celadon, that can be used to detect toxic content in open data more efficiently at a larger scale. Finally, we describe the balanced approach to content filtration that optimizes safety filtering with respect to the filtered data available for training.
• 摘要：开源大型语言模型在研究人员和从业者中越来越受欢迎。虽然开放权重模型已经取得了重大进展，但开放训练数据仍是领先的开放权重模型创建者尚未采用的做法。与此同时，研究人员正在努力使语言模型更安全。我们提出了一个数据管理流程，以减少在公共领域数据上训练的模型产生的有害输出。处理公共领域数据存在独特的挑战，因为这些来源在形式和内容上都不同于网络文本。许多来源是历史文献，是光学字符识别 (OCR) 的结果。因此，当前最先进的毒性过滤方法通常不适用于或不适合开放数据模型。在本文中，我们介绍了一种用于开放数据毒性过滤的全新完全开源流程。我们的贡献有三方面。我们创建了一个自定义训练数据集 ToxicCommons，它由已在五个不同维度（基于种族/出身、基于性别/性别、宗教、基于能力的歧视和暴力）分类的文本组成。我们使用此数据集训练自定义分类器 Celadon，该分类器可用于更有效地在更大范围内检测开放数据中的有害内容。最后，我们描述了一种平衡的内容过滤方法，该方法针对可用于训练的过滤数据优化安全过滤。

Title: Characterizing the Role of Similarity in the Property Inferences of Language Models

• Authors: Juan Diego Rodriguez, Aaron Mueller, Kanishka Misra
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.22590
• Pdf URL: https://arxiv.org/pdf/2410.22590
• Copy Paste: [[2410.22590]] Characterizing the Role of Similarity in the Property Inferences of Language Models(https://arxiv.org/abs/2410.22590)
• Keywords: language model
• Abstract: Property inheritance -- a phenomenon where novel properties are projected from higher level categories (e.g., birds) to lower level ones (e.g., sparrows) -- provides a unique window into how humans organize and deploy conceptual knowledge. It is debated whether this ability arises due to explicitly stored taxonomic knowledge vs. simple computations of similarity between mental representations. How are these mechanistic hypotheses manifested in contemporary language models? In this work, we investigate how LMs perform property inheritance with behavioral and causal representational analysis experiments. We find that taxonomy and categorical similarities are not mutually exclusive in LMs' property inheritance behavior. That is, LMs are more likely to project novel properties from one category to the other when they are taxonomically related and at the same time, highly similar. Our findings provide insight into the conceptual structure of language models and may suggest new psycholinguistic experiments for human subjects.
• 摘要：属性继承是一种现象，即新属性从较高级别的类别（例如鸟类）投射到较低级别的类别（例如麻雀），它为了解人类如何组织和部署概念知识提供了一个独特的窗口。人们争论这种能力是由于明确存储的分类知识还是由于心理表征之间相似性的简单计算而产生的。这些机械假设在当代语言模型中是如何体现的？在这项工作中，我们通过行为和因果表征分析实验研究了语言模型如何执行属性继承。我们发现，在语言模型的属性继承行为中，分类和类别相似性并不相互排斥。也就是说，当语言模型在分类上相关且同时高度相似时，它们更有可能将新属性从一个类别投射到另一个类别。我们的研究结果为语言模型的概念结构提供了见解，并可能为人类受试者提供新的心理语言学实验。

Title: Prove Your Point!: Bringing Proof-Enhancement Principles to Argumentative Essay Generation

• Authors: Ruiyu Xiao, Lei Wu, Yuhang Gou, Weinan Zhang, Ting Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.22642
• Pdf URL: https://arxiv.org/pdf/2410.22642
• Copy Paste: [[2410.22642]] Prove Your Point!: Bringing Proof-Enhancement Principles to Argumentative Essay Generation(https://arxiv.org/abs/2410.22642)
• Keywords: language model
• Abstract: Argumentative essay generation (AEG) aims to generate complete texts on specific controversial topics or debates. Although current AEG methods can generate individual opinions, they often overlook the high-level connections between these opinions. This often leads to the generated results being mired in logical confusion, unable to proof their own arguments effectively. The generated essay may present evidence that contradicts the claims or they may fail to assemble the claims into logical flow. In this paper, we present a unified two-stage framework: Proof-Enhancement and Self-Annotation (PESA) for AEG with a focus on logical enhancement. Specifically, we first construct pseudo-labels for logical information,claims and grounds, using a large language model. We then propose a tree planning approach that introduces proof principles and ensures logical consistency. Extensive experimental results show that, benefiting from proof principle guidance, PESA generates argumentative essays with better logical validity and persuasiveness than strong baseline models.
• 摘要：论证性文章生成 (AEG) 旨在针对特定的争议性话题或辩论生成完整的文本。尽管当前的 AEG 方法可以生成个人观点，但它们往往忽略了这些观点之间的高级联系。这往往导致生成的结果陷入逻辑混乱，无法有效地证明自己的论点。生成的文章可能会提供与主张相矛盾的证据，也可能无法将主张组合成逻辑流。在本文中，我们提出了一个统一的两阶段框架：证明增强和自我注释 (PESA)，用于 AEG，重点是逻辑增强。具体来说，我们首先使用大型语言模型为逻辑信息、主张和理由构建伪标签。然后，我们提出了一种引入证明原理并确保逻辑一致性的树规划方法。大量实验结果表明，在证明原理指导下，PESA 生成的论证性文章比强基线模型具有更好的逻辑有效性和说服力。

Title: Linguistics Theory Meets LLM: Code-Switched Text Generation via Equivalence Constrained Large Language Models

• Authors: Garry Kuwanto, Chaitanya Agarwal, Genta Indra Winata, Derry Tanti Wijaya
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.22660
• Pdf URL: https://arxiv.org/pdf/2410.22660
• Copy Paste: [[2410.22660]] Linguistics Theory Meets LLM: Code-Switched Text Generation via Equivalence Constrained Large Language Models(https://arxiv.org/abs/2410.22660)
• Keywords: language model, llm
• Abstract: Code-switching, the phenomenon of alternating between two or more languages in a single conversation, presents unique challenges for Natural Language Processing (NLP). Most existing research focuses on either syntactic constraints or neural generation, with few efforts to integrate linguistic theory with large language models (LLMs) for generating natural code-switched text. In this paper, we introduce EZSwitch, a novel framework that combines Equivalence Constraint Theory (ECT) with LLMs to produce linguistically valid and fluent code-switched text. We evaluate our method using both human judgments and automatic metrics, demonstrating a significant improvement in the quality of generated code-switching sentences compared to baseline LLMs. To address the lack of suitable evaluation metrics, we conduct a comprehensive correlation study of various automatic metrics against human scores, revealing that current metrics often fail to capture the nuanced fluency of code-switched text. Additionally, we create CSPref, a human preference dataset based on human ratings and analyze model performance across ``hard`` and ``easy`` examples. Our findings indicate that incorporating linguistic constraints into LLMs leads to more robust and human-aligned generation, paving the way for scalable code-switching text generation across diverse language pairs.
• 摘要：代码转换是指在一次对话中交替使用两种或多种语言的现象，它为自然语言处理 (NLP) 带来了独特的挑战。大多数现有研究都侧重于句法约束或神经生成，很少有研究将语言理论与大型语言模型 (LLM) 相结合，以生成自然的代码转换文本。在本文中，我们介绍了 EZSwitch，这是一个新颖的框架，它将等价约束理论 (ECT) 与 LLM 相结合，以生成语言有效且流畅的代码转换文本。我们使用人工判断和自动指标来评估我们的方法，结果表明，与基线 LLM 相比，生成的代码转换句子的质量有显著提高。为了解决缺乏合适评估指标的问题，我们对各种自动指标与人工评分进行了全面的相关性研究，结果表明，当前的指标通常无法捕捉到代码转换文本的细微流畅性。此外，我们创建了 CSPref，这是一个基于人工评分的人类偏好数据集，并分析了“难”和“易”示例中的模型性能。我们的研究结果表明，将语言约束纳入 LLM 可以实现更加稳健、更加符合人类习惯的生成，为跨不同语言对的可扩展代码转换文本生成铺平了道路。

Title: Constructing Multimodal Datasets from Scratch for Rapid Development of a Japanese Visual Language Model

• Authors: Keito Sasagawa, Koki Maeda, Issa Sugiura, Shuhei Kurita, Naoaki Okazaki, Daisuke Kawahara
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.22736
• Pdf URL: https://arxiv.org/pdf/2410.22736
• Copy Paste: [[2410.22736]] Constructing Multimodal Datasets from Scratch for Rapid Development of a Japanese Visual Language Model(https://arxiv.org/abs/2410.22736)
• Keywords: language model
• Abstract: To develop high-performing Visual Language Models (VLMs), it is essential to prepare multimodal resources, such as image-text pairs, interleaved data, and instruction data. While multimodal resources for English are abundant, there is a significant lack of corresponding resources for non-English languages, such as Japanese. To address this problem, we take Japanese as a non-English language and propose a method for rapidly creating Japanese multimodal datasets from scratch. We collect Japanese image-text pairs and interleaved data from web archives and generate Japanese instruction data directly from images using an existing VLM. Our experimental results show that a VLM trained on these native datasets outperforms those relying on machine-translated content.
• 摘要：要开发高性能的视觉语言模型 (VLM)，必须准备多模态资源，例如图像-文本对、交错数据和说明数据。虽然英语的多模态资源很丰富，但非英语语言（例如日语）的相应资源却严重匮乏。为了解决这个问题，我们将日语视为非英语语言，并提出了一种从头开始快速创建日语多模态数据集的方法。我们从网络档案中收集日语图像-文本对和交错数据，并使用现有的 VLM 直接从图像生成日语说明数据。我们的实验结果表明，在这些原生数据集上训练的 VLM 优于依赖机器翻译内容的 VLM。

Title: Beyond Ontology in Dialogue State Tracking for Goal-Oriented Chatbot

• Authors: Sejin Lee, Dongha Kim, Min Song
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.22767
• Pdf URL: https://arxiv.org/pdf/2410.22767
• Copy Paste: [[2410.22767]] Beyond Ontology in Dialogue State Tracking for Goal-Oriented Chatbot(https://arxiv.org/abs/2410.22767)
• Keywords: language model, llm, hallucination, prompt, chat
• Abstract: Goal-oriented chatbots are essential for automating user tasks, such as booking flights or making restaurant reservations. A key component of these systems is Dialogue State Tracking (DST), which interprets user intent and maintains the dialogue state. However, existing DST methods often rely on fixed ontologies and manually compiled slot values, limiting their adaptability to open-domain dialogues. We propose a novel approach that leverages instruction tuning and advanced prompt strategies to enhance DST performance, without relying on any predefined ontologies. Our method enables Large Language Model (LLM) to infer dialogue states through carefully designed prompts and includes an anti-hallucination mechanism to ensure accurate tracking in diverse conversation contexts. Additionally, we employ a Variational Graph Auto-Encoder (VGAE) to model and predict subsequent user intent. Our approach achieved state-of-the-art with a JGA of 42.57% outperforming existing ontology-less DST models, and performed well in open-domain real-world conversations. This work presents a significant advancement in creating more adaptive and accurate goal-oriented chatbots.
• 摘要：面向目标的聊天机器人对于自动执行用户任务（例如预订航班或预订餐厅）至关重要。这些系统的一个关键组件是对话状态跟踪 (DST)，它可以解释用户意图并维护对话状态。然而，现有的 DST 方法通常依赖于固定本体和手动编译的槽值，限制了它们对开放域对话的适应性。我们提出了一种新颖的方法，利用指令调整和高级提示策略来增强 DST 性能，而不依赖于任何预定义的本体。我们的方法使大型语言模型 (LLM) 能够通过精心设计的提示推断对话状态，并包括一种防幻觉机制，以确保在不同的对话环境中准确跟踪。此外，我们采用变分图自动编码器 (VGAE) 来建模和预测后续用户意图。我们的方法达到了最先进的水平，JGA 为 42.57%，优于现有的无本体 DST 模型，并且在开放域真实世界对话中表现良好。这项工作在创建更具适应性、更准确、以目标为导向的聊天机器人方面取得了重大进步。

Title: InjecGuard: Benchmarking and Mitigating Over-defense in Prompt Injection Guardrail Models

• Authors: Hao Li, Xiaogeng Liu, Chaowei Xiao
• Subjects: cs.CL, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2410.22770
• Pdf URL: https://arxiv.org/pdf/2410.22770
• Copy Paste: [[2410.22770]] InjecGuard: Benchmarking and Mitigating Over-defense in Prompt Injection Guardrail Models(https://arxiv.org/abs/2410.22770)
• Keywords: language model, llm, prompt
• Abstract: Prompt injection attacks pose a critical threat to large language models (LLMs), enabling goal hijacking and data leakage. Prompt guard models, though effective in defense, suffer from over-defense -- falsely flagging benign inputs as malicious due to trigger word bias. To address this issue, we introduce NotInject, an evaluation dataset that systematically measures over-defense across various prompt guard models. NotInject contains 339 benign samples enriched with trigger words common in prompt injection attacks, enabling fine-grained evaluation. Our results show that state-of-the-art models suffer from over-defense issues, with accuracy dropping close to random guessing levels (60%). To mitigate this, we propose InjecGuard, a novel prompt guard model that incorporates a new training strategy, Mitigating Over-defense for Free (MOF), which significantly reduces the bias on trigger words. InjecGuard demonstrates state-of-the-art performance on diverse benchmarks including NotInject, surpassing the existing best model by 30.8%, offering a robust and open-source solution for detecting prompt injection attacks. The code and datasets are released at this https URL.
• 摘要：即时注入攻击对大型语言模型 (LLM) 构成严重威胁，可导致目标劫持和数据泄露。即时防护模型虽然防御有效，但也存在过度防御的问题 —— 由于触发词偏差，会将良性输入错误地标记为恶意输入。为了解决这个问题，我们引入了 NotInject，这是一个评估数据集，可系统地测量各种即时防护模型的过度防御情况。NotInject 包含 339 个良性样本，其中富含即时注入攻击中常见的触发词，可进行细粒度评估。我们的结果表明，最先进的模型存在过度防御问题，准确率下降到接近随机猜测水平 (60%)。为了缓解这种情况，我们提出了 InjecGuard，这是一种新颖的即时防护模型，它采用了一种新的训练策略，即免费缓解过度防御 (MOF)，可显着降低对触发词的偏差。 InjecGuard 在包括 NotInject 在内的各种基准测试中表现出色，比现有最佳模型高出 30.8%，为检测即时注入攻击提供了强大且开源的解决方案。代码和数据集在此 https URL 上发布。

Title: MALoRA: Mixture of Asymmetric Low-Rank Adaptation for Enhanced Multi-Task Learning

• Authors: Xujia Wang, Haiyan Zhao, Shuo Wang, Hanqing Wang, Zhiyuan Liu
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.22782
• Pdf URL: https://arxiv.org/pdf/2410.22782
• Copy Paste: [[2410.22782]] MALoRA: Mixture of Asymmetric Low-Rank Adaptation for Enhanced Multi-Task Learning(https://arxiv.org/abs/2410.22782)
• Keywords: llm
• Abstract: Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA have significantly improved the adaptation of LLMs to downstream tasks in a resource-efficient manner. However, in multi-task scenarios, challenges such as training imbalance and the seesaw effect frequently emerge. Mixture-of-LoRA (MoLoRA), which combines LoRA with sparse Mixture-of-Experts, mitigates some of these issues by promoting task-specific learning across experts. Despite this, MoLoRA remains inefficient in terms of training speed, parameter utilization, and overall multi-task performance. In this paper, we propose Mixture of Asymmetric Low-Rank Adaptaion (MALoRA), a flexible fine-tuning framework that leverages asymmetric optimization across LoRA experts. MALoRA reduces the number of trainable parameters by 30% to 48%, increases training speed by 1.2x, and matches the computational efficiency of single-task LoRA models. Additionally, MALoRA addresses overfitting issues commonly seen in high-rank configurations, enhancing performance stability. Extensive experiments across diverse multi-task learning scenarios demonstrate that MALoRA consistently outperforms all baseline methods in both inter-domain and intra-domain tasks.
• 摘要：参数高效微调 (PEFT) 方法（例如 LoRA）以资源高效的方式显著提高了 LLM 对下游任务的适应性。然而，在多任务场景中，训练不平衡和跷跷板效应等挑战经常出现。LoRA 混合 (MoLoRA) 将 LoRA 与稀疏专家混合相结合，通过促进专家之间的特定任务学习来缓解其中一些问题。尽管如此，MoLoRA 在训练速度、参数利用率和整体多任务性能方面仍然效率低下。在本文中，我们提出了非对称低秩适应混合 (MALoRA)，这是一种灵活的微调框架，利用 LoRA 专家之间的非对称优化。MALoRA 将可训练参数的数量减少了 30% 至 48%，将训练速度提高了 1.2 倍，并与单任务 LoRA 模型的计算效率相匹配。此外，MALoRA 解决了高秩配置中常见的过拟合问题，增强了性能稳定性。在不同多任务学习场景中进行的大量实验表明，MALoRA 在域间和域内任务中始终优于所有基线方法。

Title: EvoCodeBench: An Evolving Code Generation Benchmark with Domain-Specific Evaluations

• Authors: Jia Li, Ge Li, Xuanming Zhang, Yunfei Zhao, Yihong Dong, Zhi Jin, Binhua Li, Fei Huang, Yongbin Li
• Subjects: cs.CL, cs.SE
• Abstract URL: https://arxiv.org/abs/2410.22821
• Pdf URL: https://arxiv.org/pdf/2410.22821
• Copy Paste: [[2410.22821]] EvoCodeBench: An Evolving Code Generation Benchmark with Domain-Specific Evaluations(https://arxiv.org/abs/2410.22821)
• Keywords: language model, gpt, llm
• Abstract: How to evaluate Large Language Models (LLMs) in code generation remains an open question. Existing benchmarks have two limitations - data leakage and lack of domain-specific evaluation. The former hurts the fairness of benchmarks, and the latter hinders practitioners from selecting superior LLMs for specific programming domains. To address these two limitations, we propose a new benchmark - EvoCodeBench, which has the following advances: (1) Evolving data. EvoCodeBench will be dynamically updated every period (e.g., 6 months) to avoid data leakage. This paper releases the first version - EvoCodeBench-2403, containing 275 samples from 25 repositories. (2) A domain taxonomy and domain labels. Based on the statistics of open-source communities, we design a programming domain taxonomy consisting of 10 popular domains. Based on the taxonomy, we annotate each sample in EvoCodeBench with a domain label. (3) Domain-specific evaluations. Besides the Pass@k, we compute the Domain-Specific Improvement (DSI) and define LLMs' comfort and strange domains. These evaluations help practitioners select superior LLMs in specific domains and discover the shortcomings of existing LLMs. We evaluate 8 popular LLMs (e.g., gpt-4, DeepSeek Coder) on EvoCodeBench and summarize some insights. EvoCodeBench reveals the actual abilities of these LLMs in real-world repositories. For example, the highest Pass@1 of gpt-4 on EvoCodeBench-2403 is only 20.74%. Besides, we evaluate LLMs in different domains and discover their comfort and strange domains. For example, gpt-4 performs best in most domains but falls behind others in the Internet domain. StarCoder 2-15B unexpectedly performs well in the Database domain and even outperforms 33B LLMs. EvoCodeBench has been released.
• 摘要：如何评估代码生成中的大型语言模型 (LLM) 仍是一个悬而未决的问题。现有的基准测试有两个限制——数据泄露和缺乏领域特定评估。前者损害了基准测试的公平性，后者妨碍了从业者为特定的编程领域选择优秀的 LLM。为了解决这两个限制，我们提出了一个新的基准测试 - EvoCodeBench，它具有以下进步：（1）不断发展的数据。EvoCodeBench 将每隔一段时间（例如 6 个月）动态更新以避免数据泄露。本文发布了第一个版本 - EvoCodeBench-2403，包含来自 25 个存储库的 275 个样本。（2）领域分类法和领域标签。基于开源社区的统计数据，我们设计了一个由 10 个热门领域组成的编程领域分类法。基于分类法，我们用领域标签注释 EvoCodeBench 中的每个样本。（3）领域特定评估。除了 Pass@k，我们还计算了领域特定改进 (DSI)，并定义了 LLM 的舒适域和陌生域。这些评估有助于从业者在特定领域中选择更优秀的 LLM，并发现现有 LLM 的不足之处。我们在 EvoCodeBench 上评估了 8 个流行的 LLM（例如，gpt-4、DeepSeek Coder），并总结了一些见解。EvoCodeBench 揭示了这些 LLM 在现实世界存储库中的实际能力。例如，EvoCodeBench-2403 上 gpt-4 的最高 Pass@1 仅为 20.74%。此外，我们在不同领域评估了 LLM，并发现了它们的舒适域和陌生域。例如，gpt-4 在大多数领域表现最佳，但在互联网领域落后于其他领域。StarCoder 2-15B 在数据库领域出人意料地表现出色，甚至优于 33B LLM。EvoCodeBench 已经发布。

Title: How Well Do Large Language Models Disambiguate Swedish Words?

• Authors: Richard Johansson
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.22827
• Pdf URL: https://arxiv.org/pdf/2410.22827
• Copy Paste: [[2410.22827]] How Well Do Large Language Models Disambiguate Swedish Words?(https://arxiv.org/abs/2410.22827)
• Keywords: language model, prompt
• Abstract: We evaluate a battery of recent large language models on two benchmarks for word sense disambiguation in Swedish. At present, all current models are less accurate than the best supervised disambiguators in cases where a training set is available, but most models outperform graph-based unsupervised systems. Different prompting approaches are compared, with a focus on how to express the set of possible senses in a given context. The best accuracies are achieved when human-written definitions of the senses are included in the prompts.
• 摘要：我们根据瑞典语词义消歧的两个基准对一系列近期的大型语言模型进行了评估。目前，在有训练集的情况下，所有当前模型的准确率都低于最好的监督式消歧器，但大多数模型的表现都优于基于图的无监督系统。我们比较了不同的提示方法，重点关注如何在给定上下文中表达一组可能的含义。当提示中包含人类书写的含义定义时，准确率最高。

Title: Danoliteracy of Generative, Large Language Models

• Authors: Søren Vejlgaard Holm, Lars Kai Hansen, Martin Carsten Nielsen
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.22839
• Pdf URL: https://arxiv.org/pdf/2410.22839
• Copy Paste: [[2410.22839]] Danoliteracy of Generative, Large Language Models(https://arxiv.org/abs/2410.22839)
• Keywords: language model, gpt, llm
• Abstract: The language technology moonshot moment of Generative, Large Language Models (GLLMs) was not limited to English: These models brought a surge of technological applications, investments and hype to low-resource languages as well. However, the capabilities of these models in languages such as Danish were until recently difficult to verify beyond qualitative demonstrations due to a lack of applicable evaluation corpora. We present a GLLM benchmark to evaluate Danoliteracy, a measure of Danish language and cultural competency, across eight diverse scenarios such Danish citizenship tests and abstractive social media question answering. This limited-size benchmark is found to produce a robust ranking that correlates to human feedback at $\rho \sim 0.8$ with GPT-4 and Claude Opus models achieving the highest rankings. Analyzing these model results across scenarios, we find one strong underlying factor explaining $95\%$ of scenario performance variance for GLLMs in Danish, suggesting a $g$ factor of model consistency in language adaption.
• 摘要：生成式大型语言模型 (GLLM) 的语言技术登月时刻不仅限于英语：这些模型也为资源匮乏的语言带来了大量的技术应用、投资和炒作。然而，由于缺乏适用的评估语料库，直到最近，这些模型在丹麦语等语言中的能力都难以通过定性演示进行验证。我们提出了一个 GLLM 基准来评估丹麦语素养，这是衡量丹麦语言和文化能力的指标，涵盖了八个不同的场景，例如丹麦公民身份测试和抽象社交媒体问答。我们发现，这个有限规模的基准可以产生一个与人类反馈相关的稳健排名，相关性为 $\rho \sim 0.8$，其中 GPT-4 和 Claude Opus 模型获得了最高排名。通过分析这些模型在各种场景中的结果，我们发现一个强有力的潜在因素可以解释丹麦语 GLLM 场景性能差异的 $95\%$，这表明语言适应性模型一致性的 $g$ 因子。

Title: Eliciting Critical Reasoning in Retrieval-Augmented Language Models via Contrastive Explanations

• Authors: Leonardo Ranaldi, Marco Valentino, Andrè Freitas
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.22874
• Pdf URL: https://arxiv.org/pdf/2410.22874
• Copy Paste: [[2410.22874]] Eliciting Critical Reasoning in Retrieval-Augmented Language Models via Contrastive Explanations(https://arxiv.org/abs/2410.22874)
• Keywords: language model, llm, hallucination, prompt, retrieval-augmented generation
• Abstract: Retrieval-augmented generation (RAG) has emerged as a critical mechanism in contemporary NLP to support Large Language Models(LLMs) in systematically accessing richer factual context. However, the integration of RAG mechanisms brings its inherent challenges, as LLMs need to deal with potentially noisy contexts. Recent studies have shown that LLMs still struggle to critically analyse RAG-based in-context information, a limitation that may lead to incorrect inferences and hallucinations. In this paper, we investigate how to elicit critical reasoning in RAG via contrastive explanations. In particular, we propose Contrastive-RAG (C-RAG), a framework that (i) retrieves relevant documents given a query, (ii) selects and exemplifies relevant passages, and (iii) generates explanations that explicitly contrast the relevance of the passages to (iv) support the final answer. We show the impact of C-RAG building contrastive reasoning demonstrations from LLMs to instruct smaller models for retrieval-augmented tasks. Extensive experiments demonstrate that C-RAG improves state-of-the-art RAG models while (a) requiring significantly fewer prompts and demonstrations and (b) being robust to perturbations in the retrieved documents.
• 摘要：检索增强生成 (RAG) 已成为当代 NLP 中支持大型语言模型 (LLM) 系统地访问更丰富的事实背景的重要机制。然而，RAG 机制的集成带来了其固有的挑战，因为 LLM 需要处理潜在的噪声背景。最近的研究表明，LLM 仍然难以批判性地分析基于 RAG 的上下文信息，这一限制可能会导致错误的推理和幻觉。在本文中，我们研究如何通过对比解释在 RAG 中引出批判性推理。具体来说，我们提出了对比 RAG (C-RAG)，这是一个框架，它 (i) 根据查询检索相关文档，(ii) 选择和举例说明相关段落，以及 (iii) 生成明确对比段落相关性的解释以 (iv) 支持最终答案。我们展示了 C-RAG 从 LLM 构建对比推理演示以指导较小模型进行检索增强任务的影响。大量实验表明，C-RAG 改进了最先进的 RAG 模型，同时 (a) 需要的提示和演示明显减少，并且 (b) 对检索到的文档中的扰动具有很强的鲁棒性。

Title: Less is More: Pre-Training Cross-Lingual Small-Scale Language Models with Cognitively-Plausible Curriculum Learning Strategies

• Authors: Suchir Salhan, Richard Diehl Martinez, Zébulon Goriely, Paula Buttery
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.22886
• Pdf URL: https://arxiv.org/pdf/2410.22886
• Copy Paste: [[2410.22886]] Less is More: Pre-Training Cross-Lingual Small-Scale Language Models with Cognitively-Plausible Curriculum Learning Strategies(https://arxiv.org/abs/2410.22886)
• Keywords: language model
• Abstract: Curriculum Learning has been a popular strategy to improve the cognitive plausibility of Small-Scale Language Models (SSLMs) in the BabyLM Challenge. However, it has not led to considerable improvements over non-curriculum models. We assess whether theoretical linguistic acquisition theories can be used to specify more fine-grained curriculum learning strategies, creating age-ordered corpora of Child-Directed Speech for four typologically distant language families to implement SSLMs and acquisition-inspired curricula cross-lingually. Comparing the success of three objective curricula (Growing, Inwards and MMM) that precisely replicate the predictions of acquisition theories on a standard SSLM architecture, we find fine-grained acquisition-inspired curricula can outperform non-curriculum baselines and performance benefits of curricula strategies in SSLMs can be derived by specifying fine-grained language-specific curricula that precisely replicate language acquisition theories.
• 摘要：课程学习一直是 BabyLM 挑战赛中提高小规模语言模型 (SSLM) 认知可信度的一种流行策略。然而，它并没有带来比非课程模型更显著的改进。我们评估了理论语言习得理论是否可用于指定更细粒度的课程学习策略，为四个类型学上远距离的语言家族创建按年龄排序的儿童导向语音语料库，以跨语言实施 SSLM 和习得启发课程。通过比较三个客观课程（成长、向内和 MMM）的成功，这些课程在标准 SSLM 架构上精确复制了习得理论的预测，我们发现细粒度的习得启发课程可以胜过非课程基线，并且可以通过指定精确复制语言习得理论的细粒度语言特定课程来获得 SSLM 中课程策略的性能优势。

Title: From Babble to Words: Pre-Training Language Models on Continuous Streams of Phonemes

• Authors: Zébulon Goriely, Richard Diehl Martinez, Andrew Caines, Lisa Beinborn, Paula Buttery
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.22906
• Pdf URL: https://arxiv.org/pdf/2410.22906
• Copy Paste: [[2410.22906]] From Babble to Words: Pre-Training Language Models on Continuous Streams of Phonemes(https://arxiv.org/abs/2410.22906)
• Keywords: language model
• Abstract: Language models are typically trained on large corpora of text in their default orthographic form. However, this is not the only option; representing data as streams of phonemes can offer unique advantages, from deeper insights into phonological language acquisition to improved performance on sound-based tasks. The challenge lies in evaluating the impact of phoneme-based training, as most benchmarks are also orthographic. To address this, we develop a pipeline to convert text datasets into a continuous stream of phonemes. We apply this pipeline to the 100-million-word pre-training dataset from the BabyLM challenge, as well as to standard language and grammatical benchmarks, enabling us to pre-train and evaluate a model using phonemic input representations. Our results show that while phoneme-based training slightly reduces performance on traditional language understanding tasks, it offers valuable analytical and practical benefits.
• 摘要：语言模型通常在默认正字法形式的大量文本上进行训练。然而，这并不是唯一的选择；将数据表示为音素流可以提供独特的优势，从更深入地了解语音语言习得，到提高基于声音的任务的性能。挑战在于评估基于音素的训练的影响，因为大多数基准也是正字法的。为了解决这个问题，我们开发了一个管道，将文本数据集转换为连续的音素流。我们将这个管道应用于 BabyLM 挑战赛中的 1 亿字预训练数据集，以及标准语言和语法基准，使我们能够使用音素输入表示对模型进行预训练和评估。我们的结果表明，虽然基于音素的训练会略微降低传统语言理解任务的性能，但它提供了宝贵的分析和实践优势。

Title: Explainable Behavior Cloning: Teaching Large Language Model Agents through Learning by Demonstration

• Authors: Yanchu Guan, Dong Wang, Yan Wang, Haiqing Wang, Renen Sun, Chenyi Zhuang, Jinjie Gu, Zhixuan Chu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.22916
• Pdf URL: https://arxiv.org/pdf/2410.22916
• Copy Paste: [[2410.22916]] Explainable Behavior Cloning: Teaching Large Language Model Agents through Learning by Demonstration(https://arxiv.org/abs/2410.22916)
• Keywords: language model, llm, agent
• Abstract: Autonomous mobile app interaction has become increasingly important with growing complexity of mobile applications. Developing intelligent agents that can effectively navigate and interact with mobile apps remains a significant challenge. In this paper, we propose an Explainable Behavior Cloning LLM Agent (EBC-LLMAgent), a novel approach that combines large language models (LLMs) with behavior cloning by learning demonstrations to create intelligent and explainable agents for autonomous mobile app interaction. EBC-LLMAgent consists of three core modules: Demonstration Encoding, Code Generation, and UI Mapping, which work synergistically to capture user demonstrations, generate executable codes, and establish accurate correspondence between code and UI elements. We introduce the Behavior Cloning Chain Fusion technique to enhance the generalization capabilities of the agent. Extensive experiments on five popular mobile applications from diverse domains demonstrate the superior performance of EBC-LLMAgent, achieving high success rates in task completion, efficient generalization to unseen scenarios, and the generation of meaningful explanations.
• 摘要：随着移动应用程序的复杂性不断增加，自主移动应用程序交互变得越来越重要。开发能够有效导航和与移动应用程序交互的智能代理仍然是一项重大挑战。在本文中，我们提出了一种可解释行为克隆 LLM 代理 (EBC-LLMAgent)，这是一种新颖的方法，它将大型语言模型 (LLM) 与通过学习演示进行的行为克隆相结合，以创建用于自主移动应用程序交互的智能且可解释的代理。EBC-LLMAgent 由三个核心模块组成：演示编码、代码生成和 UI 映射，它们协同工作以捕获用户演示、生成可执行代码并在代码和 UI 元素之间建立准确的对应关系。我们引入了行为克隆链融合技术来增强代理的泛化能力。对来自不同领域的五个流行移动应用程序进行的大量实验证明了 EBC-LLMAgent 的卓越性能，在任务完成方面取得了高成功率、高效泛化到未见过的场景以及生成有意义的解释。

Title: Multi-Agent Large Language Models for Conversational Task-Solving

• Authors: Jonas Becker
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.22932
• Pdf URL: https://arxiv.org/pdf/2410.22932
• Copy Paste: [[2410.22932]] Multi-Agent Large Language Models for Conversational Task-Solving(https://arxiv.org/abs/2410.22932)
• Keywords: language model, llm, agent
• Abstract: In an era where single large language models have dominated the landscape of artificial intelligence for years, multi-agent systems arise as new protagonists in conversational task-solving. While previous studies have showcased their potential in reasoning tasks and creative endeavors, an analysis of their limitations concerning the conversational paradigms and the impact of individual agents is missing. It remains unascertained how multi-agent discussions perform across tasks of varying complexity and how the structure of these conversations influences the process. To fill that gap, this work systematically evaluates multi-agent systems across various discussion paradigms, assessing their strengths and weaknesses in both generative tasks and question-answering tasks. Alongside the experiments, I propose a taxonomy of 20 multi-agent research studies from 2022 to 2024, followed by the introduction of a framework for deploying multi-agent LLMs in conversational task-solving. I demonstrate that while multi-agent systems excel in complex reasoning tasks, outperforming a single model by leveraging expert personas, they fail on basic tasks. Concretely, I identify three challenges that arise: 1) While longer discussions enhance reasoning, agents fail to maintain conformity to strict task requirements, which leads to problem drift, making shorter conversations more effective for basic tasks. 2) Prolonged discussions risk alignment collapse, raising new safety concerns for these systems. 3) I showcase discussion monopolization through long generations, posing the problem of fairness in decision-making for tasks like summarization. This work uncovers both the potential and challenges that arise with multi-agent interaction and varying conversational paradigms, providing insights into how future research could improve the efficiency, performance, and safety of multi-agent LLMs.
• 摘要：在单一大型语言模型多年来主导人工智能领域的时代，多智能体系统成为对话任务解决的新主角。虽然先前的研究已经展示了它们在推理任务和创造性工作中的潜力，但缺乏对它们在对话范式和单个智能体影响方面的局限性的分析。多智能体讨论在不同复杂程度的任务中的表现如何，以及这些对话的结构如何影响这一过程，仍未确定。为了填补这一空白，这项工作系统地评估了各种讨论范式中的多智能体系统，评估了它们在生成任务和问答任务中的优势和劣势。除了实验之外，我还提出了从 2022 年到 2024 年的 20 个多智能体研究分类法，然后介绍了一个在对话任务解决中部署多智能体 LLM 的框架。我证明了，虽然多智能体系统在复杂的推理任务中表现出色，通过利用专家角色超越了单一模型，但它们在基本任务上却失败了。具体来说，我发现了出现的三个挑战：1) 虽然较长的讨论可以增强推理能力，但智能体无法保持对严格任务要求的一致性，从而导致问题漂移，使得较短的对话对于基本任务更有效。2) 长时间的讨论有导致一致性崩溃的风险，从而引发这些系统的新安全问题。3) 我展示了通过长时间的代际垄断讨论，提出了总结等任务决策公平性的问题。这项工作揭示了多智能体交互和不同对话范式所带来的潜力和挑战，为未来研究如何提高多智能体 LLM 的效率、性能和安全性提供了见解。

Title: Private Synthetic Text Generation with Diffusion Models

• Authors: Sebastian Ochs, Ivan Habernal
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.22971
• Pdf URL: https://arxiv.org/pdf/2410.22971
• Copy Paste: [[2410.22971]] Private Synthetic Text Generation with Diffusion Models(https://arxiv.org/abs/2410.22971)
• Keywords: llm
• Abstract: How capable are diffusion models of generating synthetics texts? Recent research shows their strengths, with performance reaching that of auto-regressive LLMs. But are they also good in generating synthetic data if the training was under differential privacy? Here the evidence is missing, yet the promises from private image generation look strong. In this paper we address this open question by extensive experiments. At the same time, we critically assess (and reimplement) previous works on synthetic private text generation with LLMs and reveal some unmet assumptions that might have led to violating the differential privacy guarantees. Our results partly contradict previous non-private findings and show that fully open-source LLMs outperform diffusion models in the privacy regime. Our complete source codes, datasets, and experimental setup is publicly available to foster future research.
• 摘要：扩散模型生成合成文本的能力有多强？最近的研究表明了它们的优势，其性能达到了自回归 LLM 的水平。但是，如果训练是在差异隐私下进行的，它们是否也能很好地生成合成数据？这里缺少证据，但隐私图像生成的承诺看起来很强大。在本文中，我们通过大量实验解决了这个悬而未决的问题。同时，我们批判性地评估（并重新实现）了以前使用 LLM 生成合成隐私文本的工作，并揭示了一些可能导致违反差异隐私保证的未满足的假设。我们的结果部分与之前的非隐私发现相矛盾，并表明完全开源的 LLM 在隐私制度下的表现优于扩散模型。我们的完整源代码、数据集和实验设置都是公开的，以促进未来的研究。

Title: Bonafide at LegalLens 2024 Shared Task: Using Lightweight DeBERTa Based Encoder For Legal Violation Detection and Resolution

• Authors: Shikha Bordia
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.22977
• Pdf URL: https://arxiv.org/pdf/2410.22977
• Copy Paste: [[2410.22977]] Bonafide at LegalLens 2024 Shared Task: Using Lightweight DeBERTa Based Encoder For Legal Violation Detection and Resolution(https://arxiv.org/abs/2410.22977)
• Keywords: llm
• Abstract: In this work, we present two systems -- Named Entity Resolution (NER) and Natural Language Inference (NLI) -- for detecting legal violations within unstructured textual data and for associating these violations with potentially affected individuals, respectively. Both these systems are lightweight DeBERTa based encoders that outperform the LLM baselines. The proposed NER system achieved an F1 score of 60.01\% on Subtask A of the LegalLens challenge, which focuses on identifying violations. The proposed NLI system achieved an F1 score of 84.73\% on Subtask B of the LegalLens challenge, which focuses on resolving these violations by matching them with pre-existing legal complaints of class action cases. Our NER system ranked sixth and NLI system ranked fifth on the LegalLens leaderboard. We release the trained models and inference scripts.
• 摘要：在本研究中，我们提出了两个系统——命名实体解析 (NER) 和自然语言推理 (NLI)，分别用于检测非结构化文本数据中的违法行为并将这些违法行为与可能受影响的个人关联起来。这两个系统都是基于 DeBERTa 的轻量级编码器，其性能优于 LLM 基线。所提出的 NER 系统在 LegalLens 挑战赛的子任务 A 中获得了 60.01\% 的 F1 分数，该子任务侧重于识别违法行为。所提出的 NLI 系统在 LegalLens 挑战赛的子任务 B 中获得了 84.73\% 的 F1 分数，该子任务侧重于通过将这些违法行为与集体诉讼案件中现有的法律投诉进行匹配来解决它们。我们的 NER 系统在 LegalLens 排行榜上排名第六，NLI 系统排名第五。我们发布了训练好的模型和推理脚本。

Title: \textsc{Long$^2$RAG}: Evaluating Long-Context \& Long-Form Retrieval-Augmented Generation with Key Point Recall

• Authors: Zehan Qi, Rongwu Xu, Zhijiang Guo, Cunxiang Wang, Hao Zhang, Wei Xu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.23000
• Pdf URL: https://arxiv.org/pdf/2410.23000
• Copy Paste: [[2410.23000]] \textsc{Long$^2$RAG}: Evaluating Long-Context \& Long-Form Retrieval-Augmented Generation with Key Point Recall(https://arxiv.org/abs/2410.23000)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: Retrieval-augmented generation (RAG) is a promising approach to address the limitations of fixed knowledge in large language models (LLMs). However, current benchmarks for evaluating RAG systems suffer from two key deficiencies: (1) they fail to adequately measure LLMs' capability in handling \emph{long-context retrieval} due to a lack of datasets that reflect the characteristics of retrieved documents, and (2) they lack a comprehensive evaluation method for assessing LLMs' ability to generate \emph{long-form responses} that effectively exploits retrieved information. To address these shortcomings, we introduce the \textsc{Long$^2$RAG} benchmark and the Key Point Recall (\textit{KPR}) metric. \textsc{Long$^2$RAG} comprises 280 questions spanning 10 domains and across 8 question categories, each associated with 5 retrieved documents with an average length of 2,444 words. \textit{KPR} evaluates the extent to which LLMs incorporate key points extracted from the retrieved documents into their generated responses, providing a more nuanced assessment of their ability to exploit retrieved information. Our dataset and scripts are available at this https URL.
• 摘要：检索增强生成 (RAG) 是一种很有前途的方法，可以解决大型语言模型 (LLM) 中固定知识的局限性。然而，目前用于评估 RAG 系统的基准有两个关键缺陷：(1) 由于缺乏反映检索文档特征的数据集，它们无法充分衡量 LLM 处理 \emph{长上下文检索} 的能力，以及 (2) 它们缺乏一种全面的评估方法来评估 LLM 生成有效利用检索信息的 \emph{长格式响应} 的能力。为了解决这些缺点，我们引入了 \textsc{Long$^2$RAG} 基准和关键点召回 (\textit{KPR}) 指标。\textsc{Long$^2$RAG} 包含 280 个问题，涵盖 10 个领域和 8 个问题类别，每个问题与 5 个检索文档相关，平均长度为 2,444 个单词。 \textit{KPR} 评估 LLM 将从检索到的文档中提取的关键点整合到生成的响应中的程度，从而更细致地评估其利用检索到的信息的能力。我们的数据集和脚本可在此 https URL 上获取。

Title: BUZZ: Beehive-structured Sparse KV Cache with Segmented Heavy Hitters for Efficient LLM Inference

• Authors: Junqi Zhao, Zhijin Fang, Shu Li, Shaohui Yang, Shichao He
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.23079
• Pdf URL: https://arxiv.org/pdf/2410.23079
• Copy Paste: [[2410.23079]] BUZZ: Beehive-structured Sparse KV Cache with Segmented Heavy Hitters for Efficient LLM Inference(https://arxiv.org/abs/2410.23079)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) are essential in natural language processing but often struggle with inference speed and computational efficiency, limiting real-time deployment. The key-value (KV) cache mechanism reduces computational overhead in transformer models, but challenges in maintaining contextual understanding remain. In this paper, we propose BUZZ, a novel KV caching algorithm that leverages structured contextual information to minimize cache memory usage while enhancing inference speed. BUZZ employs a beehive-structured sparse cache, incorporating a sliding window to capture recent information and dynamically segmenting historical tokens into chunks to prioritize important tokens in local neighborhoods. We evaluate BUZZ on four real-world datasets: CNN/Daily Mail, XSUM, Wikitext, and 10-QA. Our results demonstrate that BUZZ (1) reduces cache memory usage by $\textbf{2.5}\times$ in LLM inference while maintaining over 99% accuracy in long-text summarization, and (2) surpasses state-of-the-art performance in multi-document question answering by $\textbf{7.69%}$ under the same memory limit, where full cache methods encounter out-of-memory issues. Additionally, BUZZ achieves significant inference speedup with a $\log{n}$ time complexity. The code is available at this https URL.
• 摘要：大型语言模型 (LLM) 在自然语言处理中必不可少，但通常在推理速度和计算效率方面存在问题，从而限制了实时部署。键值 (KV) 缓存机制减少了 Transformer 模型中的计算开销，但在保持上下文理解方面仍然存在挑战。在本文中，我们提出了 BUZZ，这是一种新颖的 KV 缓存算法，它利用结构化的上下文信息来最大限度地减少缓存内存使用量，同时提高推理速度。BUZZ 采用蜂巢结构的稀疏缓存，结合滑动窗口来捕获最新信息，并动态地将历史标记分割成块，以优先考虑本地邻域中的重要标记。我们在四个真实世界的数据集上评估了 BUZZ：CNN/Daily Mail、XSUM、Wikitext 和 10-QA。我们的结果表明，BUZZ (1) 在 LLM 推理中将缓存内存使用量减少了 $\textbf{2.5}\times$，同时在长文本摘要中保持 99% 以上的准确率，并且 (2) 在相同内存限制下，在多文档问答中的表现比最先进的性能高出 $\textbf{7.69%}$，而全缓存方法会遇到内存不足的问题。此外，BUZZ 实现了显著的推理速度提升，时间复杂度为 $\log{n}$。代码可从此 https URL 获取。

Title: Comparative Analysis of Demonstration Selection Algorithms for LLM In-Context Learning

• Authors: Dong Shu, Mengnan Du
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.23099
• Pdf URL: https://arxiv.org/pdf/2410.23099
• Copy Paste: [[2410.23099]] Comparative Analysis of Demonstration Selection Algorithms for LLM In-Context Learning(https://arxiv.org/abs/2410.23099)
• Keywords: language model, llm
• Abstract: In-context learning can help Large Language Models (LLMs) to adapt new tasks without additional training. However, this performance heavily depends on the quality of the demonstrations, driving research into effective demonstration selection algorithms to optimize this process. These algorithms assist users in selecting the best $k$ input-label pairs (demonstration examples) based on a given test input, enabling LLMs to in-context learn the relationship between the provided examples and the test inputs. Despite all the proposed demonstration selection algorithms, their efficiency and effectiveness remain unclear. This lack of clarity make it difficult to apply these algorithms in real-world scenarios and poses challenges for future research aimed at developing improved methods. This paper revisits six proposed algorithms, evaluating them on five datasets from both efficiency and effectiveness perspectives. Our experiments reveal significant variations in algorithm performance across different tasks, with some methods struggling to outperform random selection in certain scenarios. We also find that increasing the number of demonstrations does not always lead to better performance, and that there are often trade-offs between accuracy and computational efficiency. Our code is available at this https URL.
• 摘要：上下文学习可以帮助大型语言模型 (LLM) 适应新任务而无需额外训练。然而，这种性能在很大程度上取决于演示的质量，这推动了对有效演示选择算法的研究，以优化这一过程。这些算法帮助用户根据给定的测试输入选择最佳的 $k$ 输入标签对（演示示例），使 LLM 能够在上下文中学习提供的示例与测试输入之间的关系。尽管提出了所有演示选择算法，但它们的效率和有效性仍然不清楚。这种缺乏清晰度使得这些算法难以应用于现实世界场景，并对旨在开发改进方法的未来研究构成挑战。本文重新审视了六种提出的算法，从效率和有效性的角度对五个数据集进行了评估。我们的实验表明，算法在不同任务中的性能存在显著差异，某些方法在某些情况下难以超越随机选择。我们还发现，增加演示数量并不总能带来更好的性能，而且准确性和计算效率之间往往存在权衡。我们的代码可在此 https URL 上找到。

Title: Teaching a Language Model to Distinguish Between Similar Details using a Small Adversarial Training Set

• Authors: Chris Achard
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.23118
• Pdf URL: https://arxiv.org/pdf/2410.23118
• Copy Paste: [[2410.23118]] Teaching a Language Model to Distinguish Between Similar Details using a Small Adversarial Training Set(https://arxiv.org/abs/2410.23118)
• Keywords: language model
• Abstract: Language models can achieve high accuracy on natural language tasks such as NLI, but performance suffers on manually created adversarial examples. We investigate the performance of a language model trained on the Stanford Natural Language Inference (SNLI) corpus on a manually created adversarial test set. We then improve the model's performance by fine tuning the model on a small, manually created adversarial training set, designed to help the language model to learn to differentiate between similar words and phrases in the data. We show an increase in accuracy on the adversarial test set (+ 13%) while still maintaining good performance on the original NLI task. We also show an increase in accuracy from 91.2% to 92.9% on the most similar contradictions in the SNLI test set (as judged by cosine similarity).
• 摘要：语言模型可以在 NLI 等自然语言任务上实现高精度，但在手动创建的对抗性示例上性能会受到影响。我们研究了在手动创建的对抗性测试集上训练的语言模型在斯坦福自然语言推理 (SNLI) 语料库上的性能。然后，我们通过在小型手动创建的对抗性训练集上对模型进行微调来提高模型的性能，旨在帮助语言模型学习区分数据中的相似单词和短语。我们发现对抗性测试集的准确率有所提高 (+ 13%)，同时仍在原始 NLI 任务上保持良好的性能。我们还发现，在 SNLI 测试集中最相似的矛盾上（以余弦相似度判断），准确率从 91.2% 提高到了 92.9%。

Title: On Memorization of Large Language Models in Logical Reasoning

• Authors: Chulin Xie, Yangsibo Huang, Chiyuan Zhang, Da Yu, Xinyun Chen, Bill Yuchen Lin, Bo Li, Badih Ghazi, Ravi Kumar
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.23123
• Pdf URL: https://arxiv.org/pdf/2410.23123
• Copy Paste: [[2410.23123]] On Memorization of Large Language Models in Logical Reasoning(https://arxiv.org/abs/2410.23123)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) achieve good performance on challenging reasoning benchmarks, yet could also make basic reasoning mistakes. This contrasting behavior is puzzling when it comes to understanding the mechanisms behind LLMs' reasoning capabilities. One hypothesis is that the increasingly high and nearly saturated performance on common reasoning benchmarks could be due to the memorization of similar problems. In this paper, we systematically investigate this hypothesis with a quantitative measurement of memorization in reasoning tasks, using a dynamically generated logical reasoning benchmark based on Knights and Knaves (K&K) puzzles. We found that LLMs could interpolate the training puzzles (achieving near-perfect accuracy) after fine-tuning, yet fail when those puzzles are slightly perturbed, suggesting that the models heavily rely on memorization to solve those training puzzles. On the other hand, we show that while fine-tuning leads to heavy memorization, it also consistently improves generalization performance. In-depth analyses with perturbation tests, cross difficulty-level transferability, probing model internals, and fine-tuning with wrong answers suggest that the LLMs learn to reason on K&K puzzles despite training data memorization. This phenomenon indicates that LLMs exhibit a complex interplay between memorization and genuine reasoning abilities. Finally, our analysis with per-sample memorization score sheds light on how LLMs switch between reasoning and memorization in solving logical puzzles. Our code and data are available at this https URL.
• 摘要：大型语言模型 (LLM) 在具有挑战性的推理基准上取得了良好的表现，但也可能犯基本的推理错误。这种对比行为在理解 LLM 推理能力背后的机制时令人费解。一种假设是，在常见推理基准上越来越高且接近饱和的表现可能是由于对类似问题的记忆。在本文中，我们使用基于骑士和杰克 (K&K) 谜题的动态生成的逻辑推理基准，通过对推理任务中的记忆进行定量测量，系统地研究了这一假设。我们发现 LLM 可以在微调后插入训练谜题（实现近乎完美的准确性），但当这些谜题受到轻微干扰时就会失败，这表明模型严重依赖记忆来解决这些训练谜题。另一方面，我们表明，虽然微调会导致大量记忆，但它也会持续提高泛化性能。通过扰动测试、跨难度级别可迁移性、探索模型内部以及使用错误答案进行微调的深入分析，我们发现尽管训练数据记忆，LLM 仍能学会推理 K&K 谜题。这一现象表明 LLM 表现出记忆和真正推理能力之间的复杂相互作用。最后，我们对每个样本记忆分数的分析揭示了 LLM 在解决逻辑谜题时如何在推理和记忆之间切换。我们的代码和数据可在此 https URL 上找到。

Title: SciPIP: An LLM-based Scientific Paper Idea Proposer

• Authors: Wenxiao Wang, Lihui Gu, Liye Zhang, Yunxiang Luo, Yi Dai, Chen Shen, Liang Xie, Binbin Lin, Xiaofei He, Jieping Ye
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.23166
• Pdf URL: https://arxiv.org/pdf/2410.23166
• Copy Paste: [[2410.23166]] SciPIP: An LLM-based Scientific Paper Idea Proposer(https://arxiv.org/abs/2410.23166)
• Keywords: language model, gpt, llm
• Abstract: The exponential growth of knowledge and the increasing complexity of interdisciplinary research pose significant challenges for researchers, including information overload and difficulties in exploring novel ideas. The advancements in large language models (LLMs), such as GPT-4, have shown great potential in enhancing idea proposals, but how to effectively utilize large models for reasonable idea proposal has not been thoroughly explored. This paper proposes a scientific paper idea proposer (SciPIP). Based on a user-provided research background, SciPIP retrieves helpful papers from a literature database while leveraging the capabilities of LLMs to generate more novel and feasible ideas. To this end, 1) we construct a literature retrieval database, extracting lots of papers' multi-dimension information for fast access. Then, a literature retrieval method based on semantics, entity, and citation co-occurrences is proposed to search relevant literature from multiple aspects based on the user-provided background. 2) After literature retrieval, we introduce dual-path idea proposal strategies, where one path infers solutions from the retrieved literature and the other path generates original ideas through model brainstorming. We then combine the two to achieve a good balance between feasibility and originality. Through extensive experiments on the natural language processing (NLP) field, we demonstrate that SciPIP can retrieve citations similar to those of existing top conference papers and generate many ideas consistent with them. Additionally, we evaluate the originality of other ideas generated by SciPIP using large language models, further validating the effectiveness of our proposed method. The code and the database are released at this https URL.
• 摘要：知识的指数级增长和跨学科研究的日益复杂给研究人员带来了重大挑战，包括信息过载和探索新想法的困难。GPT-4 等大型语言模型 (LLM) 的进步在增强想法提议方面显示出巨大的潜力，但如何有效利用大型模型进行合理的想法提出尚未得到深入探索。本文提出了一个科学论文想法提出器 (SciPIP)。基于用户提供的研究背景，SciPIP 从文献数据库中检索有用的论文，同时利用 LLM 的功能产生更多新颖可行的想法。为此，1）我们构建了一个文献检索数据库，提取大量论文的多维信息以实现快速访问。然后，提出一种基于语义、实体和引文共现的文献检索方法，根据用户提供的背景从多个方面搜索相关文献。 2）在文献检索之后，我们引入双路径想法提出策略，一条路径从检索到的文献中推断解决方案，另一条路径通过模型头脑风暴产生原创想法。然后我们将两者结合起来，在可行性和原创性之间取得良好的平衡。通过在自然语言处理 (NLP) 领域的大量实验，我们证明了 SciPIP 可以检索与现有顶级会议论文相似的引用，并产生许多与之一致的想法。此外，我们使用大型语言模型评估了 SciPIP 生成的其他想法的原创性，进一步验证了我们提出的方法的有效性。代码和数据库在此 https URL 上发布。

Title: OS-ATLAS: A Foundation Action Model for Generalist GUI Agents

• Authors: Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, Yu Qiao
• Subjects: cs.CL, cs.CV, cs.HC
• Abstract URL: https://arxiv.org/abs/2410.23218
• Pdf URL: https://arxiv.org/pdf/2410.23218
• Copy Paste: [[2410.23218]] OS-ATLAS: A Foundation Action Model for Generalist GUI Agents(https://arxiv.org/abs/2410.23218)
• Keywords: language model, gpt, agent
• Abstract: Existing efforts in building GUI agents heavily rely on the availability of robust commercial Vision-Language Models (VLMs) such as GPT-4o and GeminiProVision. Practitioners are often reluctant to use open-source VLMs due to their significant performance lag compared to their closed-source counterparts, particularly in GUI grounding and Out-Of-Distribution (OOD) scenarios. To facilitate future research in this area, we developed OS-Atlas - a foundational GUI action model that excels at GUI grounding and OOD agentic tasks through innovations in both data and modeling. We have invested significant engineering effort in developing an open-source toolkit for synthesizing GUI grounding data across multiple platforms, including Windows, Linux, MacOS, Android, and the web. Leveraging this toolkit, we are releasing the largest open-source cross-platform GUI grounding corpus to date, which contains over 13 million GUI elements. This dataset, combined with innovations in model training, provides a solid foundation for OS-Atlas to understand GUI screenshots and generalize to unseen interfaces. Through extensive evaluation across six benchmarks spanning three different platforms (mobile, desktop, and web), OS-Atlas demonstrates significant performance improvements over previous state-of-the-art models. Our evaluation also uncovers valuable insights into continuously improving and scaling the agentic capabilities of open-source VLMs.
• 摘要：现有的 GUI 代理构建工作严重依赖于 GPT-4o 和 GeminiProVision 等强大的商业视觉语言模型 (VLM)。从业者往往不愿意使用开源 VLM，因为与闭源 VLM 相比，它们的性能明显滞后，尤其是在 GUI 基础和分布式外 (OOD) 场景中。为了促进该领域的未来研究，我们开发了 OS-Atlas - 一种基础 GUI 动作模型，通过数据和建模方面的创新，在 GUI 基础和 OOD 代理任务方面表现出色。我们投入了大量工程精力来开发一个开源工具包，用于在多个平台（包括 Windows、Linux、MacOS、Android 和 Web）上合成 GUI 基础数据。利用这个工具包，我们发布了迄今为止最大的开源跨平台 GUI 基础语料库，其中包含超过 1300 万个 GUI 元素。该数据集与模型训练创新相结合，为 OS-Atlas 理解 GUI 屏幕截图并推广到未见过的界面提供了坚实的基础。通过对三个不同平台（移动、桌面和网络）的六个基准进行广泛评估，OS-Atlas 展示了比以前最先进的模型显著的性能改进。我们的评估还揭示了持续改进和扩展开源 VLM 代理功能的宝贵见解。

Title: Evaluating Cultural and Social Awareness of LLM Web Agents

• Authors: Haoyi Qiu, Alexander R. Fabbri, Divyansh Agarwal, Kung-Hsiang Huang, Sarah Tan, Nanyun Peng, Chien-Sheng Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.23252
• Pdf URL: https://arxiv.org/pdf/2410.23252
• Copy Paste: [[2410.23252]] Evaluating Cultural and Social Awareness of LLM Web Agents(https://arxiv.org/abs/2410.23252)
• Keywords: language model, llm, prompt, agent
• Abstract: As large language models (LLMs) expand into performing as agents for real-world applications beyond traditional NLP tasks, evaluating their robustness becomes increasingly important. However, existing benchmarks often overlook critical dimensions like cultural and social awareness. To address these, we introduce CASA, a benchmark designed to assess LLM agents' sensitivity to cultural and social norms across two web-based tasks: online shopping and social discussion forums. Our approach evaluates LLM agents' ability to detect and appropriately respond to norm-violating user queries and observations. Furthermore, we propose a comprehensive evaluation framework that measures awareness coverage, helpfulness in managing user queries, and the violation rate when facing misleading web content. Experiments show that current LLMs perform significantly better in non-agent than in web-based agent environments, with agents achieving less than 10% awareness coverage and over 40% violation rates. To improve performance, we explore two methods: prompting and fine-tuning, and find that combining both methods can offer complementary advantages -- fine-tuning on culture-specific datasets significantly enhances the agents' ability to generalize across different regions, while prompting boosts the agents' ability to navigate complex tasks. These findings highlight the importance of constantly benchmarking LLM agents' cultural and social awareness during the development cycle.
• 摘要：随着大型语言模型 (LLM) 扩展到作为传统 NLP 任务之外的实际应用代理，评估它们的稳健性变得越来越重要。然而，现有的基准往往忽略了文化和社会意识等关键维度。为了解决这些问题，我们引入了 CASA，这是一个基准，旨在评估 LLM 代理在两个基于 Web 的任务中对文化和社会规范的敏感性：在线购物和社交讨论论坛。我们的方法评估 LLM 代理检测和适当响应违反规范的用户查询和观察的能力。此外，我们提出了一个全面的评估框架，用于衡量意识覆盖率、管理用户查询的有用性以及面对误导性 Web 内容时的违规率。实验表明，当前的 LLM 在非代理环境中的表现明显优于基于 Web 的代理环境，代理的意识覆盖率不到 10%，违规率超过 40%。为了提高性能，我们探索了两种方法：提示和微调，并发现将这两种方法结合起来可以提供互补的优势——对特定文化数据集进行微调可以显著提高代理在不同地区进行推广的能力，而提示可以提高代理处理复杂任务的能力。这些发现凸显了在开发周期中不断对 LLM 代理的文化和社会意识进行基准测试的重要性。