2024-01-26

Title: EMP: Effective Multidimensional Persistence for Graph Representation Learning

• Authors: Ignacio Segovia-Dominguez, Yuzhou Chen, Cuneyt G. Akcora, Zhiwei Zhen, Murat Kantarcioglu, Yulia R. Gel, Baris Coskunuzer
• Subjects: cs.LG, cs.AI, cs.CG
• Abstract URL: https://arxiv.org/abs/2401.13713
• Pdf URL: https://arxiv.org/pdf/2401.13713
• Copy Paste: [[2401.13713]] EMP: Effective Multidimensional Persistence for Graph Representation Learning(https://arxiv.org/abs/2401.13713)
• Keywords: lora
• Abstract: Topological data analysis (TDA) is gaining prominence across a wide spectrum of machine learning tasks that spans from manifold learning to graph classification. A pivotal technique within TDA is persistent homology (PH), which furnishes an exclusive topological imprint of data by tracing the evolution of latent structures as a scale parameter changes. Present PH tools are confined to analyzing data through a single filter parameter. However, many scenarios necessitate the consideration of multiple relevant parameters to attain finer insights into the data. We address this issue by introducing the Effective Multidimensional Persistence (EMP) framework. This framework empowers the exploration of data by simultaneously varying multiple scale parameters. The framework integrates descriptor functions into the analysis process, yielding a highly expressive data summary. It seamlessly integrates established single PH summaries into multidimensional counterparts like EMP Landscapes, Silhouettes, Images, and Surfaces. These summaries represent data's multidimensional aspects as matrices and arrays, aligning effectively with diverse ML models. We provide theoretical guarantees and stability proofs for EMP summaries. We demonstrate EMP's utility in graph classification tasks, showing its effectiveness. Results reveal that EMP enhances various single PH descriptors, outperforming cutting-edge methods on multiple benchmark datasets.
• 摘要：拓扑数据分析 (TDA) 在从流形学习到图分类的各种机器学习任务中越来越受到重视。 TDA 中的一项关键技术是持久同源性 (PH)，它通过跟踪潜在结构随尺度参数变化的演化来提供数据的独特拓扑印记。目前的 PH 工具仅限于通过单个过滤器参数分析数据。然而，许多场景需要考虑多个相关参数才能更深入地了解数据。我们通过引入有效多维持久性（EMP）框架来解决这个问题。该框架支持通过同时改变多个尺度参数来探索数据。该框架将描述符函数集成到分析过程中，产生高度表达的数据摘要。它将已建立的单一 PH 摘要无缝集成到多维对应物中，例如 EMP 景观、轮廓、图像和表面。这些摘要将数据的多维方面表示为矩阵和数组，与不同的 ML 模型有效地保持一致。我们为EMP摘要提供理论保证和稳定性证明。我们演示了 EMP 在图分类任务中的实用性，展示了其有效性。结果表明，EMP 增强了各种单一 PH 描述符，在多个基准数据集上优于尖端方法。

Title: Conformal Prediction Sets Improve Human Decision Making

• Authors: Jesse C. Cresswell, Yi Sui, Bhargava Kumar, Noël Vouitsis
• Subjects: cs.LG, cs.HC, stat.ML
• Abstract URL: https://arxiv.org/abs/2401.13744
• Pdf URL: https://arxiv.org/pdf/2401.13744
• Copy Paste: [[2401.13744]] Conformal Prediction Sets Improve Human Decision Making(https://arxiv.org/abs/2401.13744)
• Keywords: rag
• Abstract: In response to everyday queries, humans explicitly signal uncertainty and offer alternative answers when they are unsure. Machine learning models that output calibrated prediction sets through conformal prediction mimic this human behaviour; larger sets signal greater uncertainty while providing alternatives. In this work, we study the usefulness of conformal prediction sets as an aid for human decision making by conducting a pre-registered randomized controlled trial with conformal prediction sets provided to human subjects. With statistical significance, we find that when humans are given conformal prediction sets their accuracy on tasks improves compared to fixed-size prediction sets with the same coverage guarantee. The results show that quantifying model uncertainty with conformal prediction is helpful for human-in-the-loop decision making and human-AI teams.
• 摘要：在回答日常问题时，人类会明确表示不确定性，并在不确定时提供替代答案。通过保形预测输出校准预测集的机器学习模型模仿了这种人类行为；更大的集合意味着更大的不确定性，同时提供替代方案。在这项工作中，我们通过向人类受试者提供保形预测集进行预先注册的随机对照试验，研究保形预测集作为人类决策辅助的有用性。具有统计意义，我们发现，当人类获得保形预测集时，与具有相同覆盖率保证的固定大小预测集相比，他们的任务准确性得到了提高。结果表明，通过保形预测量化模型不确定性有助于人在环决策和人类人工智能团队。

Title: Explaining Image Classifiers

• Authors: Hana Chockler, Joseph Y. Halpern
• Subjects: cs.AI
• Abstract URL: https://arxiv.org/abs/2401.13752
• Pdf URL: https://arxiv.org/pdf/2401.13752
• Copy Paste: [[2401.13752]] Explaining Image Classifiers(https://arxiv.org/abs/2401.13752)
• Keywords: agent
• Abstract: We focus on explaining image classifiers, taking the work of Mothilal et al. [2021] (MMTS) as our point of departure. We observe that, although MMTS claim to be using the definition of explanation proposed by Halpern [2016], they do not quite do so. Roughly speaking, Halpern's definition has a necessity clause and a sufficiency clause. MMTS replace the necessity clause by a requirement that, as we show, implies it. Halpern's definition also allows agents to restrict the set of options considered. While these difference may seem minor, as we show, they can have a nontrivial impact on explanations. We also show that, essentially without change, Halpern's definition can handle two issues that have proved difficult for other approaches: explanations of absence (when, for example, an image classifier for tumors outputs "no tumor") and explanations of rare events (such as tumors).
• 摘要：我们以 Mothilal 等人的工作为重点，重点解释图像分类器。 [2021]（MMTS）作为我们的出发点。我们观察到，尽管 MMTS 声称使用 Halpern [2016] 提出的解释定义，但他们并没有完全这样做。粗略地说，哈尔彭的定义有一个必要性条款和一个充分性条款。 MMTS 用一个要求取代了必要性条款，正如我们所表明的，暗示了它。 Halpern 的定义还允许代理人限制所考虑的选项集。正如我们所展示的，虽然这些差异可能看起来很小，但它们可能会对解释产生不小的影响。我们还表明，在本质上无需改变的情况下，Halpern 的定义可以处理其他方法难以解决的两个问题：缺席的解释（例如，当肿瘤的图像分类器输出“无肿瘤”时）和罕见事件的解释（例如如肿瘤）。

Title: NLICE: Synthetic Medical Record Generation for Effective Primary Healthcare Differential Diagnosis

• Authors: Zaid Al-Ars, Obinna Agba, Zhuoran Guo, Christiaan Boerkamp, Ziyaad Jaber, Tareq Jaber
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.13756
• Pdf URL: https://arxiv.org/pdf/2401.13756
• Copy Paste: [[2401.13756]] NLICE: Synthetic Medical Record Generation for Effective Primary Healthcare Differential Diagnosis(https://arxiv.org/abs/2401.13756)
• Keywords: code
• Abstract: This paper offers a systematic method for creating medical knowledge-grounded patient records for use in activities involving differential diagnosis. Additionally, an assessment of machine learning models that can differentiate between various conditions based on given symptoms is also provided. We use a public disease-symptom data source called SymCat in combination with Synthea to construct the patients records. In order to increase the expressive nature of the synthetic data, we use a medically-standardized symptom modeling method called NLICE to augment the synthetic data with additional contextual information for each condition. In addition, Naive Bayes and Random Forest models are evaluated and compared on the synthetic data. The paper shows how to successfully construct SymCat-based and NLICE-based datasets. We also show results for the effectiveness of using the datasets to train predictive disease models. The SymCat-based dataset is able to train a Naive Bayes and Random Forest model yielding a 58.8% and 57.1% Top-1 accuracy score, respectively. In contrast, the NLICE-based dataset improves the results, with a Top-1 accuracy of 82.0% and Top-5 accuracy values of more than 90% for both models. Our proposed data generation approach solves a major barrier to the application of artificial intelligence methods in the healthcare domain. Our novel NLICE symptom modeling approach addresses the incomplete and insufficient information problem in the current binary symptom representation approach. The NLICE code is open sourced at https://github.com/guozhuoran918/NLICE.
• 摘要：本文提供了一种系统方法，用于创建基于医学知识的患者记录，以用于涉及鉴别诊断的活动。此外，还提供了对机器学习模型的评估，该模型可以根据给定的症状区分各种情况。我们使用名为 SymCat 的公共疾病症状数据源并结合 Synthea 来构建患者记录。为了提高合成数据的表达能力，我们使用称为 NLICE 的医学标准化症状建模方法，通过每种情况的附加上下文信息来增强合成数据。此外，还根据合成数据对朴素贝叶斯和随机森林模型进行了评估和比较。本文展示了如何成功构建基于 SymCat 和基于 NLICE 的数据集。我们还展示了使用数据集训练预测疾病模型的有效性结果。基于 SymCat 的数据集能够训练朴素贝叶斯和随机森林模型，分别产生 58.8% 和 57.1% 的 Top-1 准确度分数。相比之下，基于 NLICE 的数据集改进了结果，两个模型的 Top-1 准确度为 82.0%，Top-5 准确度值均超过 90%。我们提出的数据生成方法解决了人工智能方法在医疗保健领域应用的主要障碍。我们新颖的 NLICE 症状建模方法解决了当前二元症状表示方法中信息不完整和不足的问题。 NLICE 代码开源于 https://github.com/guozhuoran918/NLICE。

Title: A Unified Approach to Emotion Detection and Task-Oriented Dialogue Modeling

• Authors: Armand Stricker, Patrick Paroubek
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.13789
• Pdf URL: https://arxiv.org/pdf/2401.13789
• Copy Paste: [[2401.13789]] A Unified Approach to Emotion Detection and Task-Oriented Dialogue Modeling(https://arxiv.org/abs/2401.13789)
• Keywords: language model, gpt, rag
• Abstract: In current text-based task-oriented dialogue (TOD) systems, user emotion detection (ED) is often overlooked or is typically treated as a separate and independent task, requiring additional training. In contrast, our work demonstrates that seamlessly unifying ED and TOD modeling brings about mutual benefits, and is therefore an alternative to be considered. Our method consists in augmenting SimpleToD, an end-to-end TOD system, by extending belief state tracking to include ED, relying on a single language model. We evaluate our approach using GPT-2 and Llama-2 on the EmoWOZ benchmark, a version of MultiWOZ annotated with emotions. Our results reveal a general increase in performance for ED and task results. Our findings also indicate that user emotions provide useful contextual conditioning for system responses, and can be leveraged to further refine responses in terms of empathy.
• 摘要：在当前基于文本的面向任务的对话（TOD）系统中，用户情绪检测（ED）经常被忽视，或者通常被视为单独且独立的任务，需要额外的培训。相比之下，我们的工作表明，无缝统一 ED 和 TOD 建模可以带来互惠互利，因此是值得考虑的替代方案。我们的方法包括通过扩展信念状态跟踪以包括 ED（依赖于单一语言模型）来增强 SimpleToD（一种端到端 TOD 系统）。我们在 EmoWOZ 基准（带有情感注释的 MultiWOZ 版本）上使用 GPT-2 和 Llama-2 评估我们的方法。我们的结果显示 ED 和任务结果的表现普遍提高。我们的研究结果还表明，用户情绪为系统响应提供了有用的情境调节，并且可以用来进一步完善同理心方面的响应。

Title: Automated Root Causing of Cloud Incidents using In-Context Learning with GPT-4

• Authors: Xuchao Zhang, Supriyo Ghosh, Chetan Bansal, Rujia Wang, Minghua Ma, Yu Kang, Saravan Rajmohan
• Subjects: cs.CL, cs.SE
• Abstract URL: https://arxiv.org/abs/2401.13810
• Pdf URL: https://arxiv.org/pdf/2401.13810
• Copy Paste: [[2401.13810]] Automated Root Causing of Cloud Incidents using In-Context Learning with GPT-4(https://arxiv.org/abs/2401.13810)
• Keywords: language model, gpt, llm, code, rag
• Abstract: Root Cause Analysis (RCA) plays a pivotal role in the incident diagnosis process for cloud services, requiring on-call engineers to identify the primary issues and implement corrective actions to prevent future recurrences. Improving the incident RCA process is vital for minimizing service downtime, customer impact and manual toil. Recent advances in artificial intelligence have introduced state-of-the-art Large Language Models (LLMs) like GPT-4, which have proven effective in tackling various AIOps problems, ranging from code authoring to incident management. Nonetheless, the GPT-4 model's immense size presents challenges when trying to fine-tune it on user data because of the significant GPU resource demand and the necessity for continuous model fine-tuning with the emergence of new data. To address the high cost of fine-tuning LLM, we propose an in-context learning approach for automated root causing, which eliminates the need for fine-tuning. We conduct extensive study over 100,000 production incidents, comparing several large language models using multiple metrics. The results reveal that our in-context learning approach outperforms the previous fine-tuned large language models such as GPT-3 by an average of 24.8\% across all metrics, with an impressive 49.7\% improvement over the zero-shot model. Moreover, human evaluation involving actual incident owners demonstrates its superiority over the fine-tuned model, achieving a 43.5\% improvement in correctness and an 8.7\% enhancement in readability. The impressive results demonstrate the viability of utilizing a vanilla GPT model for the RCA task, thereby avoiding the high computational and maintenance costs associated with a fine-tuned model.
• 摘要：根本原因分析 (RCA) 在云服务的事件诊断过程中发挥着关键作用，要求待命工程师识别主要问题并实施纠正措施，以防止未来再次发生。改进事件 RCA 流程对于最大限度地减少服务停机时间、客户影响和手动操作至关重要。人工智能的最新进展引入了 GPT-4 等最先进的大型语言模型 (LLM)，事实证明，它可以有效解决从代码编写到事件管理等各种 AIOps 问题。尽管如此，GPT-4 模型的巨大规模在尝试根据用户数据对其进行微调时带来了挑战，因为对 GPU 资源的需求量很大，并且随着新数据的出现需要不断进行模型微调。为了解决 LLM 微调的高成本问题，我们提出了一种自动根源分析的上下文学习方法，从而消除了微调的需要。我们对 100,000 多个生产事件进行了广泛的研究，使用多个指标比较了几种大型语言模型。结果表明，我们的上下文学习方法在所有指标上均优于之前经过微调的大型语言模型（例如 GPT-3），平均提高了 24.8%，比零样本模型提高了 49.7%，令人印象深刻。此外，涉及实际事件所有者的人工评估显示了其相对于微调模型的优越性，正确性提高了 43.5%，可读性提高了 8.7%。令人印象深刻的结果证明了利用普通 GPT 模型执行 RCA 任务的可行性，从而避免了与微调模型相关的高计算和维护成本。

Title: The Calibration Gap between Model and Human Confidence in Large Language Models

• Authors: Mark Steyvers, Heliodoro Tejeda, Aakriti Kumar, Catarina Belem, Sheer Karny, Xinyue Hu, Lukas Mayer, Padhraic Smyth
• Subjects: cs.LG, cs.AI, cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2401.13835
• Pdf URL: https://arxiv.org/pdf/2401.13835
• Copy Paste: [[2401.13835]] The Calibration Gap between Model and Human Confidence in Large Language Models(https://arxiv.org/abs/2401.13835)
• Keywords: language model, llm
• Abstract: For large language models (LLMs) to be trusted by humans they need to be well-calibrated in the sense that they can accurately assess and communicate how likely it is that their predictions are correct. Recent work has focused on the quality of internal LLM confidence assessments, but the question remains of how well LLMs can communicate this internal model confidence to human users. This paper explores the disparity between external human confidence in an LLM's responses and the internal confidence of the model. Through experiments involving multiple-choice questions, we systematically examine human users' ability to discern the reliability of LLM outputs. Our study focuses on two key areas: (1) assessing users' perception of true LLM confidence and (2) investigating the impact of tailored explanations on this perception. The research highlights that default explanations from LLMs often lead to user overestimation of both the model's confidence and its' accuracy. By modifying the explanations to more accurately reflect the LLM's internal confidence, we observe a significant shift in user perception, aligning it more closely with the model's actual confidence levels. This adjustment in explanatory approach demonstrates potential for enhancing user trust and accuracy in assessing LLM outputs. The findings underscore the importance of transparent communication of confidence levels in LLMs, particularly in high-stakes applications where understanding the reliability of AI-generated information is essential.
• 摘要：为了让大型语言模型（LLM）受到人类的信任，它们需要进行良好的校准，以便它们能够准确地评估和传达它们的预测正确的可能性。最近的工作重点是内部法学硕士置信度评估的质量，但问题仍然是法学硕士如何向人类用户传达这种内部模型置信度。本文探讨了法学硕士回答的外部人类信心与模型的内部信心之间的差异。通过涉及多项选择题的实验，我们系统地检查了人类用户辨别 LLM 输出可靠性的能力。我们的研究重点关注两个关键领域：(1) 评估用户对法学硕士真实信心的看法；(2) 调查定制解释对这种看法的影响。研究强调，法学硕士的默认解释往往会导致用户高估模型的置信度和准确性。通过修改解释以更准确地反映法学硕士的内部置信度，我们观察到用户感知的显着变化，使其与模型的实际置信度更加一致。这种解释方法的调整表明了增强用户信任度和评估法学硕士输出准确性的潜力。研究结果强调了法学硕士中透明地沟通置信度的重要性，特别是在高风险应用中，了解人工智能生成信息的可靠性至关重要。

Title: TPD: Enhancing Student Language Model Reasoning via Principle Discovery and Guidance

• Authors: Haorui Wang (1), Rongzhi Zhang (1), Yinghao Li (1), Lingkai Kong (1), Yuchen Zhuang (1), Xiusi Chen (2), Chao Zhang (1) ((1) College of Computing, Georgia Institute of Technology, (2) Department of Computer Science, University of California, Los Angeles)
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.13849
• Pdf URL: https://arxiv.org/pdf/2401.13849
• Copy Paste: [[2401.13849]] TPD: Enhancing Student Language Model Reasoning via Principle Discovery and Guidance(https://arxiv.org/abs/2401.13849)
• Keywords: language model, llm, prompt, rag, chain-of-thought
• Abstract: Large Language Models (LLMs) have recently showcased remarkable reasoning abilities. However, larger models often surpass their smaller counterparts in reasoning tasks, posing the challenge of effectively transferring these capabilities from larger models. Existing approaches heavily rely on extensive fine-tuning data or continuous interactions with a superior teacher LLM during inference. We introduce a principle-based teacher-student framework called ``Teaching via Principle Discovery'' (TPD) to address these limitations. Inspired by human learning mechanisms, TPD mimics the interaction between a teacher and a student using a principle-based approach. The teacher LLM generates problem-solving instructions and corrective principles based on the student LLM's errors. These principles guide the refinement of instructions and the selection of instructive examples from a validation set. This enables the student model to learn from both the teacher's guidance and its own mistakes. Once the student model begins making inferences, TPD requires no further intervention from the teacher LLM or humans. Through extensive experiments across eight reasoning tasks, we demonstrate the effectiveness of TPD. Compared to standard chain-of-thought prompting, TPD significantly improves the student model's performance, achieving $6.2\%$ improvement on average.
• 摘要：大型语言模型（LLM）最近展示了卓越的推理能力。然而，较大的模型在推理任务中通常会超越较小的模型，这给从较大模型中有效迁移这些功能带来了挑战。现有的方法严重依赖于广泛的微调数据或在推理过程中与优秀的法学硕士老师的持续互动。我们引入了一种基于原则的师生框架，称为“通过原则发现进行教学”（TPD）来解决这些限制。受人类学习机制的启发，TPD 使用基于原则的方法模仿教师和学生之间的互动。法学硕士教师根据法学硕士学生的错误生成问题解决说明和纠正原则。这些原则指导说明的细化以及从验证集中选择指导性示例。这使得学生模型能够从教师的指导和自身的错误中学习。一旦学生模型开始进行推理，TPD 就不需要法学硕士老师或人类的进一步干预。通过八个推理任务的广泛实验，我们证明了 TPD 的有效性。与标准的思维链提示相比，TPD 显着提高了学生模型的表现，平均提高了 6.2%$。

Title: Embedding Attack Project (Work Report)

• Authors: Jiameng Pu, Zafar Takhirov
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2401.13854
• Pdf URL: https://arxiv.org/pdf/2401.13854
• Copy Paste: [[2401.13854]] Embedding Attack Project (Work Report)(https://arxiv.org/abs/2401.13854)
• Keywords: language model
• Abstract: This report summarizes all the MIA experiments (Membership Inference Attacks) of the Embedding Attack Project, including threat models, experimental setup, experimental results, findings and discussion. Current results cover the evaluation of two main MIA strategies (loss-based and embedding-based MIAs) on 6 AI models ranging from Computer Vision to Language Modelling. There are two ongoing experiments on MIA defense and neighborhood-comparison embedding attacks. These are ongoing projects. The current work on MIA and PIA can be summarized into six conclusions: (1) Amount of overfitting is directly proportional to model's vulnerability; (2) early embedding layers in the model are less susceptible to privacy leaks; (3) Deeper model layers contain more membership information; (4) Models are more vulnerable to MIA if both embeddings and corresponding training labels are compromised; (5) it is possible to use pseudo-labels to increase the MIA success; and (6) although MIA and PIA success rates are proportional, reducing the MIA does not necessarily reduce the PIA.
• 摘要：本报告总结了嵌入攻击项目的所有 MIA（成员推理攻击）实验，包括威胁模型、实验设置、实验结果、发现和讨论。目前的结果涵盖了从计算机视觉到语言建模等 6 种人工智能模型上两种主要 MIA 策略（基于损失和基于嵌入的 MIA）的评估。目前正在进行两项关于 MIA 防御和邻域比较嵌入攻击的实验。这些是正在进行的项目。目前MIA和PIA的工作可以概括为六个结论：（1）过拟合的程度与模型的脆弱性成正比； （2）模型中早期的嵌入层不易受到隐私泄露的影响； (3) 更深的模型层包含更多的隶属信息； (4) 如果嵌入和相应的训练标签都受到损害，模型更容易受到 MIA 的影响； (5)可以使用伪标签来增加MIA的成功率； (6) 尽管 MIA 和 PIA 成功率成正比，但减少 MIA 并不一定会减少 PIA。

Title: Inverse Molecular Design with Multi-Conditional Diffusion Guidance

• Authors: Gang Liu, Jiaxin Xu, Tengfei Luo, Meng Jiang
• Subjects: cs.LG, q-bio.BM
• Abstract URL: https://arxiv.org/abs/2401.13858
• Pdf URL: https://arxiv.org/pdf/2401.13858
• Copy Paste: [[2401.13858]] Inverse Molecular Design with Multi-Conditional Diffusion Guidance(https://arxiv.org/abs/2401.13858)
• Keywords: code
• Abstract: Inverse molecular design with diffusion models holds great potential for advancements in material and drug discovery. Despite success in unconditional molecule generation, integrating multiple properties such as synthetic score and gas permeability as condition constraints into diffusion models remains unexplored. We introduce multi-conditional diffusion guidance. The proposed Transformer-based denoising model has a condition encoder that learns the representations of numerical and categorical conditions. The denoising model, consisting of a structure encoder-decoder, is trained for denoising under the representation of conditions. The diffusion process becomes graph-dependent to accurately estimate graph-related noise in molecules, unlike the previous models that focus solely on the marginal distributions of atoms or bonds. We extensively validate our model for multi-conditional polymer and small molecule generation. Results demonstrate our superiority across metrics from distribution learning to condition control for molecular properties. An inverse polymer design task for gas separation with feedback from domain experts further demonstrates its practical utility.
• 摘要：使用扩散模型的逆向分子设计对于材料和药物发现的进步具有巨大的潜力。尽管在无条件分子生成方面取得了成功，但将合成分数和气体渗透率等多种属性作为条件约束集成到扩散模型中仍有待探索。我们引入多条件扩散指导。所提出的基于 Transformer 的去噪模型具有一个条件编码器，可以学习数值和分类条件的表示。去噪模型由编码器-解码器结构组成，在条件表示下进行去噪训练。扩散过程变得依赖于图，以准确估计分子中与图相关的噪声，这与之前仅关注原子或键的边缘分布的模型不同。我们广泛验证了我们的多条件聚合物和小分子生成模型。结果证明了我们在从分布学习到分子特性条件控制等指标方面的优势。气体分离的逆向聚合物设计任务以及领域专家的反馈进一步证明了其实用性。

Title: Unmasking and Quantifying Racial Bias of Large Language Models in Medical Report Generation

• Authors: Yifan Yang, Xiaoyu Liu, Qiao Jin, Furong Huang, Zhiyong Lu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.13867
• Pdf URL: https://arxiv.org/pdf/2401.13867
• Copy Paste: [[2401.13867]] Unmasking and Quantifying Racial Bias of Large Language Models in Medical Report Generation(https://arxiv.org/abs/2401.13867)
• Keywords: language model, gpt
• Abstract: Large language models like GPT-3.5-turbo and GPT-4 hold promise for healthcare professionals, but they may inadvertently inherit biases during their training, potentially affecting their utility in medical applications. Despite few attempts in the past, the precise impact and extent of these biases remain uncertain. Through both qualitative and quantitative analyses, we find that these models tend to project higher costs and longer hospitalizations for White populations and exhibit optimistic views in challenging medical scenarios with much higher survival rates. These biases, which mirror real-world healthcare disparities, are evident in the generation of patient backgrounds, the association of specific diseases with certain races, and disparities in treatment recommendations, etc. Our findings underscore the critical need for future research to address and mitigate biases in language models, especially in critical healthcare applications, to ensure fair and accurate outcomes for all patients.
• 摘要：像 GPT-3.5-turbo 和 GPT-4 这样的大型语言模型为医疗保健专业人员带来了希望，但他们可能会在训练过程中无意中继承偏见，从而可能影响其在医疗应用中的效用。尽管过去很少尝试，但这些偏见的确切影响和程度仍然不确定。通过定性和定量分析，我们发现这些模型往往会为白人预测更高的费用和更长的住院时间，并在具有更高生存率的挑战性医疗场景中表现出乐观的观点。这些偏见反映了现实世界的医疗保健差异，在患者背景的产生、特定疾病与某些种族的关联以及治疗建议的差异等方面都很明显。我们的研究结果强调了未来研究解决和减轻这些问题的迫切需要语言模型中的偏差，特别是在关键的医疗保健应用中，以确保所有患者获得公平和准确的结果。

Title: A comparative study of zero-shot inference with large language models and supervised modeling in breast cancer pathology classification

• Authors: Madhumita Sushil, Travis Zack, Divneet Mandair, Zhiwei Zheng, Ahmed Wali, Yan-Ning Yu, Yuwei Quan, Atul J. Butte
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.13887
• Pdf URL: https://arxiv.org/pdf/2401.13887
• Copy Paste: [[2401.13887]] A comparative study of zero-shot inference with large language models and supervised modeling in breast cancer pathology classification(https://arxiv.org/abs/2401.13887)
• Keywords: language model, gpt, llm, rag
• Abstract: Although supervised machine learning is popular for information extraction from clinical notes, creating large annotated datasets requires extensive domain expertise and is time-consuming. Meanwhile, large language models (LLMs) have demonstrated promising transfer learning capability. In this study, we explored whether recent LLMs can reduce the need for large-scale data annotations. We curated a manually-labeled dataset of 769 breast cancer pathology reports, labeled with 13 categories, to compare zero-shot classification capability of the GPT-4 model and the GPT-3.5 model with supervised classification performance of three model architectures: random forests classifier, long short-term memory networks with attention (LSTM-Att), and the UCSF-BERT model. Across all 13 tasks, the GPT-4 model performed either significantly better than or as well as the best supervised model, the LSTM-Att model (average macro F1 score of 0.83 vs. 0.75). On tasks with high imbalance between labels, the differences were more prominent. Frequent sources of GPT-4 errors included inferences from multiple samples and complex task design. On complex tasks where large annotated datasets cannot be easily collected, LLMs can reduce the burden of large-scale data labeling. However, if the use of LLMs is prohibitive, the use of simpler supervised models with large annotated datasets can provide comparable results. LLMs demonstrated the potential to speed up the execution of clinical NLP studies by reducing the need for curating large annotated datasets. This may result in an increase in the utilization of NLP-based variables and outcomes in observational clinical studies.
• 摘要：尽管监督机器学习在从临床记录中提取信息方面很流行，但创建大型带注释数据集需要广泛的领域专业知识，并且非常耗时。与此同时，大型语言模型（LLM）已经展现出有前景的迁移学习能力。在这项研究中，我们探讨了最近的法学硕士是否可以减少对大规模数据注释的需求。我们整理了包含 769 份乳腺癌病理报告的手动标记数据集，标记为 13 个类别，以比较 GPT-4 模型和 GPT-3.5 模型的零样本分类能力与三种模型架构的监督分类性能：随机森林分类器、带有注意力的长短期记忆网络（LSTM-Att）和 UCSF-BERT 模型。在所有 13 项任务中，GPT-4 模型的表现明显优于或优于最佳监督模型 LSTM-Att 模型（平均宏观 F1 得分为 0.83 vs. 0.75）。在标签之间高度不平衡的任务上，差异更为突出。 GPT-4 错误的常见来源包括来自多个样本的推断和复杂的任务设计。在无法轻松收集大型带注释数据集的复杂任务中，法学硕士可以减轻大规模数据标记的负担。然而，如果法学硕士的使用被禁止，那么使用具有大型注释数据集的更简单的监督模型可以提供可比较的结果。法学硕士展示了通过减少整理大型注释数据集的需求来加速临床 NLP 研究执行的潜力。这可能会导致观察性临床研究中基于 NLP 的变量和结果的使用增加。

Title: No More Distractions: an Adaptive Up-Sampling Algorithm to Reduce Data Artifacts

• Authors: Han Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.13907
• Pdf URL: https://arxiv.org/pdf/2401.13907
• Copy Paste: [[2401.13907]] No More Distractions: an Adaptive Up-Sampling Algorithm to Reduce Data Artifacts(https://arxiv.org/abs/2401.13907)
• Keywords: language model
• Abstract: Researchers recently found out that sometimes language models achieve high accuracy on benchmark data set, but they can not generalize very well with even little changes to the original data set. This is sometimes due to data artifacts, model is learning the spurious correlation between tokens and labels, instead of the semantics and logic. In this work, we analyzed SNLI data and visualized such spurious correlations. We proposed an adaptive up-sampling algorithm to correct the data artifacts, which is simple and effective, and does not need human edits or annotation. We did an experiment applying the algorithm to fix the data artifacts in SNLI data and the model trained with corrected data performed significantly better than the model trained with raw SNLI data, overall, as well as on the subset we corrected.
• 摘要：研究人员最近发现，有时语言模型在基准数据集上取得了很高的准确率，但即使对原始数据集进行很小的改变，它们也不能很好地泛化。有时这是由于数据伪影，模型正在学习标记和标签之间的虚假相关性，而不是语义和逻辑。在这项工作中，我们分析了 SNLI 数据并将此类虚假相关性可视化。我们提出了一种自适应上采样算法来纠正数据伪影，该算法简单有效，并且不需要人工编辑或注释。我们做了一个实验，应用该算法来修复 SNLI 数据中的数据伪影，总体而言，使用校正数据训练的模型以及我们校正的子集上的性能明显优于使用原始 SNLI 数据训练的模型。

Title: A Survey of Deep Learning and Foundation Models for Time Series Forecasting

• Authors: John A. Miller, Mohammed Aldosari, Farah Saeed, Nasid Habib Barna, Subas Rana, I. Budak Arpinar, Ninghao Liu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.13912
• Pdf URL: https://arxiv.org/pdf/2401.13912
• Copy Paste: [[2401.13912]] A Survey of Deep Learning and Foundation Models for Time Series Forecasting(https://arxiv.org/abs/2401.13912)
• Keywords: language model, code
• Abstract: Deep Learning has been successfully applied to many application domains, yet its advantages have been slow to emerge for time series forecasting. For example, in the well-known Makridakis (M) Competitions, hybrids of traditional statistical or machine learning techniques have only recently become the top performers. With the recent architectural advances in deep learning being applied to time series forecasting (e.g., encoder-decoders with attention, transformers, and graph neural networks), deep learning has begun to show significant advantages. Still, in the area of pandemic prediction, there remain challenges for deep learning models: the time series is not long enough for effective training, unawareness of accumulated scientific knowledge, and interpretability of the model. To this end, the development of foundation models (large deep learning models with extensive pre-training) allows models to understand patterns and acquire knowledge that can be applied to new related problems before extensive training data becomes available. Furthermore, there is a vast amount of knowledge available that deep learning models can tap into, including Knowledge Graphs and Large Language Models fine-tuned with scientific domain knowledge. There is ongoing research examining how to utilize or inject such knowledge into deep learning models. In this survey, several state-of-the-art modeling techniques are reviewed, and suggestions for further work are provided.
• 摘要：深度学习已成功应用于许多应用领域，但其在时间序列预测方面的优势却迟迟没有显现出来。例如，在著名的 Makridakis (M) 竞赛中，传统统计或机器学习技术的混合体最近才成为表现最好的选手。随着深度学习的最新架构进步应用于时间序列预测（例如，具有注意力的编码器-解码器、变压器和图神经网络），深度学习已经开始显示出显着的优势。尽管如此，在大流行预测领域，深度学习模型仍然面临挑战：时间序列不够长，无法进行有效训练、不了解积累的科学知识以及模型的可解释性。为此，基础模型（具有广泛预训练的大型深度学习模型）的开发使模型能够在大量训练数据可用之前理解模式并获取可应用于新的相关问题的知识。此外，深度学习模型可以利用大量可用知识，包括利用科学领域知识进行微调的知识图和大型语言模型。目前正在进行研究如何利用这些知识或将这些知识注入深度学习模型。在本次调查中，回顾了几种最先进的建模技术，并提供了进一步工作的建议。

Title: WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models

• Authors: Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, Dong Yu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.13919
• Pdf URL: https://arxiv.org/pdf/2401.13919
• Copy Paste: [[2401.13919]] WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models(https://arxiv.org/abs/2401.13919)
• Keywords: language model, gpt, llm, rag, agent
• Abstract: The advancement of large language models (LLMs) leads to a new era marked by the development of autonomous applications in the real world, which drives innovation in the creation of advanced web-based agents. Existing web agents typically only handle one input modality and are evaluated only in simplified web simulators or static web snapshots, greatly limiting their applicability in real-world scenarios. To bridge this gap, we introduce WebVoyager, an innovative Large Multimodal Model (LMM) powered web agent that can complete user instructions end-to-end by interacting with real-world websites. Moreover, we propose a new evaluation protocol for web agents to address the challenges of automatic evaluation of open-ended web agent tasks, leveraging the robust multimodal comprehension capabilities of GPT-4V. We create a new benchmark by gathering real-world tasks from 15 widely used websites to evaluate our agents. We show that WebVoyager achieves a 55.7% task success rate, significantly surpassing the performance of both GPT-4 (All Tools) and the WebVoyager (text-only) setups, underscoring the exceptional capability of WebVoyager in practical applications. We found that our proposed automatic evaluation achieves 85.3% agreement with human judgment, paving the way for further development of web agents in a real-world setting.
• 摘要：大语言模型（LLM）的进步引领了一个以现实世界中自主应用程序的发展为标志的新时代，这推动了先进的基于网络的代理的创建的创新。现有的 Web 代理通常仅处理一种输入模式，并且仅在简化的 Web 模拟器或静态 Web 快照中进行评估，这极大地限制了它们在现实场景中的适用性。为了弥补这一差距，我们引入了 WebVoyager，这是一种创新的大型多模式模型 (LMM) 支持的 Web 代理，可以通过与现实世界的网站交互来完成端到端的用户指令。此外，我们提出了一种新的网络代理评估协议，利用 GPT-4V 强大的多模态理解能力，解决自动评估开放式网络代理任务的挑战。我们通过从 15 个广泛使用的网站收集现实世界的任务来评估我们的代理，从而创建了一个新的基准。我们表明，WebVoyager 实现了 55.7% 的任务成功率，显着超过了 GPT-4（所有工具）和 WebVoyager（纯文本）设置的性能，凸显了 WebVoyager 在实际应用中的卓越能力。我们发现我们提出的自动评估与人类判断的一致性达到 85.3%，为现实世界环境中网络代理的进一步开发铺平了道路。

Title: LocMoE: A Low-overhead MoE for Large Language Model Training

• Authors: Jing Li, Zhijie Sun, Xuan He, Li Zeng, Yi Lin, Entong Li, Binfan Zheng, Rongqian Zhao, Xin Chen
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2401.13920
• Pdf URL: https://arxiv.org/pdf/2401.13920
• Copy Paste: [[2401.13920]] LocMoE: A Low-overhead MoE for Large Language Model Training(https://arxiv.org/abs/2401.13920)
• Keywords: language model, llm
• Abstract: The Mixtures-of-Experts (MoE) model is a widespread distributed and integrated learning method for large language models (LLM), which is favored due to its ability to sparsify and expand models efficiently. However, the performance of MoE is limited by load imbalance and high latency of All-To-All communication, along with relatively redundant computation owing to large expert capacity. Load imbalance may result from existing routing policies that consistently tend to select certain experts. The frequent inter-node communication in the All-To-All procedure also significantly prolongs the training time. To alleviate the above performance problems, we propose a novel routing strategy that combines load balance and locality by converting partial inter-node communication to that of intra-node. Notably, we elucidate that there is a minimum threshold for expert capacity, calculated through the maximal angular deviation between the gating weights of the experts and the assigned tokens. We port these modifications on the PanGu-Sigma model based on the MindSpore framework with multi-level routing and conduct experiments on Ascend clusters. The experiment results demonstrate that the proposed LocMoE reduces training time per epoch by 12.68% to 22.24% compared to classical routers, such as hash router and switch router, without impacting the model accuracy.
• 摘要：专家混合（MoE）模型是一种广泛应用于大型语言模型（LLM）的分布式集成学习方法，由于其能够有效地稀疏和扩展模型而受到青睐。然而，MoE 的性能受到负载不平衡和 All-To-All 通信的高延迟以及由于专家容量大而导致的相对冗余计算的限制。负载不平衡可能是由于现有的路由策略始终倾向于选择某些专家而导致的。 All-To-All过程中频繁的节点间通信也显着延长了训练时间。为了缓解上述性能问题，我们提出了一种新颖的路由策略，通过将部分节点间通信转换为节点内通信，将负载平衡和局部性结合起来。值得注意的是，我们阐明了专家能力的最小阈值，通过专家的门控权重与分配的令牌之间的最大角度偏差来计算。我们将这些修改移植到基于MindSpore框架的具有多级路由的PanGu-Sigma模型上，并在Ascend集群上进行实验。实验结果表明，与哈希路由器和交换路由器等经典路由器相比，所提出的 LocMoE 将每个 epoch 的训练时间减少了 12.68% 至 22.24%，且不影响模型精度。

Title: Towards 3D Molecule-Text Interpretation in Language Models

• Authors: Sihang Li, Zhiyuan Liu, Yanchen Luo, Xiang Wang, Xiangnan He, Kenji Kawaguchi, Tat-Seng Chua, Qi Tian
• Subjects: cs.LG, cs.IR, q-bio.BM
• Abstract URL: https://arxiv.org/abs/2401.13923
• Pdf URL: https://arxiv.org/pdf/2401.13923
• Copy Paste: [[2401.13923]] Towards 3D Molecule-Text Interpretation in Language Models(https://arxiv.org/abs/2401.13923)
• Keywords: language model, code
• Abstract: Language Models (LMs) have greatly influenced diverse domains. However, their inherent limitation in comprehending 3D molecular structures has considerably constrained their potential in the biomolecular domain. To bridge this gap, we focus on 3D molecule-text interpretation, and propose 3D-MoLM: 3D-Molecular Language Modeling. Specifically, 3D-MoLM enables an LM to interpret and analyze 3D molecules by equipping the LM with a 3D molecular encoder. This integration is achieved by a 3D molecule-text projector, bridging the 3D molecular encoder's representation space and the LM's input space. Moreover, to enhance 3D-MoLM's ability of cross-modal molecular understanding and instruction following, we meticulously curated a 3D molecule-centric instruction tuning dataset -- 3D-MoIT. Through 3D molecule-text alignment and 3D molecule-centric instruction tuning, 3D-MoLM establishes an integration of 3D molecular encoder and LM. It significantly surpasses existing baselines on downstream tasks, including molecule-text retrieval, molecule captioning, and more challenging open-text molecular QA tasks, especially focusing on 3D-dependent properties.
• 摘要：语言模型（LM）极大地影响了不同的领域。然而，它们在理解 3D 分子结构方面的固有局限性极大地限制了它们在生物分子领域的潜力。为了弥补这一差距，我们专注于 3D 分子文本解释，并提出 3D-MoLM：3D 分子语言建模。具体来说，3D-MoLM 通过为 LM 配备 3D 分子编码器，使 LM 能够解释和分析 3D 分子。这种集成是通过 3D 分子文本投影仪实现的，桥接 3D 分子编码器的表示空间和 LM 的输入空间。此外，为了增强3D-MoLM跨模态分子理解和指令跟踪的能力，我们精心策划了一个以3D分子为中心的指令调整数据集——3D-MoIT。通过3D分子文本对齐和以3D分子为中心的指令调整，3D-MoLM建立了3D分子编码器和LM的集成。它显着超越了下游任务的现有基线，包括分子文本检索、分子字幕和更具挑战性的开放文本分子 QA 任务，特别是关注 3D 相关属性。

Title: Adaptive Text Watermark for Large Language Models

• Authors: Yepeng Liu, Yuheng Bu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.13927
• Pdf URL: https://arxiv.org/pdf/2401.13927
• Copy Paste: [[2401.13927]] Adaptive Text Watermark for Large Language Models(https://arxiv.org/abs/2401.13927)
• Keywords: language model, llm, prompt
• Abstract: The advancement of Large Language Models (LLMs) has led to increasing concerns about the misuse of AI-generated text, and watermarking for LLM-generated text has emerged as a potential solution. However, it is challenging to generate high-quality watermarked text while maintaining strong security, robustness, and the ability to detect watermarks without prior knowledge of the prompt or model. This paper proposes an adaptive watermarking strategy to address this problem. To improve the text quality and maintain robustness, we adaptively add watermarking to token distributions with high entropy measured using an auxiliary model and keep the low entropy token distributions untouched. For the sake of security and to further minimize the watermark's impact on text quality, instead of using a fixed green/red list generated from a random secret key, which can be vulnerable to decryption and forgery, we adaptively scale up the output logits in proportion based on the semantic embedding of previously generated text using a well designed semantic mapping model. Our experiments involving various LLMs demonstrate that our approach achieves comparable robustness performance to existing watermark methods. Additionally, the text generated by our method has perplexity comparable to that of \emph{un-watermarked} LLMs while maintaining security even under various attacks.
• 摘要：大型语言模型 (LLM) 的进步引发了人们对 AI 生成文本滥用的日益关注，LLM 生成文本的水印已成为一种潜在的解决方案。然而，生成高质量的水印文本，同时保持强大的安全性、鲁棒性以及在不事先了解提示或模型的情况下检测水印的能力是具有挑战性的。本文提出了一种自适应水印策略来解决这个问题。为了提高文本质量并保持鲁棒性，我们自适应地将水印添加到使用辅助模型测量的高熵令牌分布中，并保持低熵令牌分布不变。为了安全起见并进一步最小化水印对文本质量的影响，我们不使用由随机密钥生成的固定绿/红列表（这可能容易被解密和伪造），而是按比例自适应地放大输出 logits基于使用精心设计的语义映射模型先前生成的文本的语义嵌入。我们涉及各种法学硕士的实验表明，我们的方法实现了与现有水印方法相当的鲁棒性性能。此外，我们的方法生成的文本具有与 \emph{un-watermarked} LLM 相当的复杂性，同时即使在各种攻击下也能保持安全性。

Title: Dynamic Long-Term Time-Series Forecasting via Meta Transformer Networks

• Authors: Muhammad Anwar Ma'sum, MD Rasel Sarkar, Mahardhika Pratama, Savitha Ramasamy, Sreenatha Anavatti, Lin Liu, Habibullah, Ryszard Kowalczyk
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.13968
• Pdf URL: https://arxiv.org/pdf/2401.13968
• Copy Paste: [[2401.13968]] Dynamic Long-Term Time-Series Forecasting via Meta Transformer Networks(https://arxiv.org/abs/2401.13968)
• Keywords: code
• Abstract: A reliable long-term time-series forecaster is highly demanded in practice but comes across many challenges such as low computational and memory footprints as well as robustness against dynamic learning environments. This paper proposes Meta-Transformer Networks (MANTRA) to deal with the dynamic long-term time-series forecasting tasks. MANTRA relies on the concept of fast and slow learners where a collection of fast learners learns different aspects of data distributions while adapting quickly to changes. A slow learner tailors suitable representations to fast learners. Fast adaptations to dynamic environments are achieved using the universal representation transformer layers producing task-adapted representations with a small number of parameters. Our experiments using four datasets with different prediction lengths demonstrate the advantage of our approach with at least $3\%$ improvements over the baseline algorithms for both multivariate and univariate settings. Source codes of MANTRA are publicly available in \url{https://github.com/anwarmaxsum/MANTRA}.
• 摘要：实践中对可靠的长期时间序列预测器的需求很高，但它遇到了许多挑战，例如低计算和内存占用以及针对动态学习环境的鲁棒性。本文提出元变压器网络（MANTRA）来处理动态长期时间序列预测任务。 MANTRA 依赖于快速学习者和慢速学习者的概念，其中快速学习者的集合学习数据分布的不同方面，同时快速适应变化。学习速度慢的人会为学习速度快的人定制合适的表示。使用通用表示变换器层实现对动态环境的快速适应，生成具有少量参数的任务适应表示。我们使用具有不同预测长度的四个数据集进行的实验证明了我们的方法的优势，与多变量和单变量设置的基线算法相比，至少有 $3\%$ 的改进。 MANTRA 的源代码可在 \url{https://github.com/anwarmaxsum/MANTRA} 中公开获取。

Title: Leeroo Orchestrator: Elevating LLMs Performance Through Model Integration

• Authors: Alireza Mohammadshahi, Ali Shaikh, Majid Yazdani
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.13979
• Pdf URL: https://arxiv.org/pdf/2401.13979
• Copy Paste: [[2401.13979]] Leeroo Orchestrator: Elevating LLMs Performance Through Model Integration(https://arxiv.org/abs/2401.13979)
• Keywords: gpt, llm
• Abstract: In this paper, we propose an architecture to harness the collective knowledge of multiple trained LLMs to create a new state-of-the-art. At the core of this framework is a LLM-based orchestrator that is adept at picking the right underlying LLM experts for optimal task execution. Inspired by self-play in reinforcement learning, we created a loop of query generation, orchestration, and evaluation to generate training data for the orchestrator. Our evaluation focused on the MMLU benchmark, employing models with 7B, 13B, and 34B parameters available on Hugging Face. The results demonstrate new state-of-the-art open-source models: Our Leeroo orchestrator achieves performance on par with the Mixtral model while incurring only two-thirds of its cost. Moreover, increasing the allowed cost surpasses Mixtral's accuracy by over 5% at the same cost level, reaching an accuracy of 75.9%. Further enhancements were observed when integrating GPT4 into the underlying model pool. The Leeroo orchestrator nearly matches GPT4's performance at half the cost and even exceeds GPT4's results with a 25% cost reduction. These findings illustrate the potential of our architecture in creating state-of-the-art and cost-effective LLMs by optimizing the synergy between multiple LLMs to achieve superior performance outcomes.
• 摘要：在本文中，我们提出了一种架构，可以利用多个经过培训的法学硕士的集体知识来创建新的最先进技术。该框架的核心是一个基于 LLM 的协调器，它擅长选择合适的底层 LLM 专家来实现最佳任务执行。受到强化学习中自我对弈的启发，我们创建了一个查询生成、编排和评估的循环，为编排器生成训练数据。我们的评估重点是 MMLU 基准，采用 Hugging Face 上提供的具有 7B、13B 和 34B 参数的模型。结果展示了最先进的新开源模型：我们的 Leeroo 编排器实现了与 Mixtral 模型相当的性能，而成本仅为其三分之二。而且，增加允许的成本比相同成本水平下Mixtral的精度提高了5%以上，达到了75.9%的精度。当将 GPT4 集成到底层模型池时，我们观察到了进一步的增强。 Leeroor 编排器以一半的成本几乎与 GPT4 的性能相当，甚至以 25% 的成本降低超过了 GPT4 的结果。这些发现说明了我们的架构在通过优化多个法学硕士之间的协同作用来创建最先进且具有成本效益的法学硕士方面的潜力，以实现卓越的绩效结果。

Title: Towards Consistent Natural-Language Explanations via Explanation-Consistency Finetuning

• Authors: Yanda Chen, Chandan Singh, Xiaodong Liu, Simiao Zuo, Bin Yu, He He, Jianfeng Gao
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.13986
• Pdf URL: https://arxiv.org/pdf/2401.13986
• Copy Paste: [[2401.13986]] Towards Consistent Natural-Language Explanations via Explanation-Consistency Finetuning(https://arxiv.org/abs/2401.13986)
• Keywords: language model, llm, code
• Abstract: Large language models (LLMs) often generate convincing, fluent explanations. However, different from humans, they often generate inconsistent explanations on different inputs. For example, an LLM may generate the explanation "all birds can fly" when answering the question "Can sparrows fly?" but meanwhile answer "no" to the related question "Can penguins fly?". Explanations should be consistent across related examples so that they allow a human to simulate the LLM's decision process on multiple examples. We propose explanation-consistency finetuning (EC-finetuning), a method that adapts LLMs to generate more consistent natural-language explanations on related examples. EC-finetuning involves finetuning LLMs on synthetic data that is carefully constructed to contain consistent explanations. Across a variety of question-answering datasets in various domains, EC-finetuning yields a 10.0% relative explanation consistency improvement on four finetuning datasets, and generalizes to seven out-of-distribution datasets not seen during finetuning (+4.5% relative). Code is available at https://github.com/yandachen/explanation-consistency-finetuning .
• 摘要：大型语言模型 (LLM) 通常会生成令人信服、流畅的解释。然而，与人类不同的是，它们经常对不同的输入产生不一致的解释。例如，法学硕士在回答“麻雀能飞吗？”问题时可能会生成“所有鸟都能飞”的解释。但同时对相关问题“企鹅会飞吗？”回答“不”。相关示例之间的解释应该保持一致，以便人们可以在多个示例上模拟法学硕士的决策过程。我们提出了解释一致性微调（EC-finetuning），这是一种使法学硕士能够对相关示例生成更一致的自然语言解释的方法。 EC 微调涉及对经过精心构建以包含一致解释的合成数据进行 LLM 微调。在各个领域的各种问答数据集上，EC 微调在四个微调数据集上产生了 10.0% 的相对解释一致性改进，并推广到微调期间未见的七个分布外数据集（相对 +4.5%）。代码可在 https://github.com/yandachen/explanation-consistency-finetuning 获取。

Title: Investigate-Consolidate-Exploit: A General Strategy for Inter-Task Agent Self-Evolution

• Authors: Cheng Qian, Shihao Liang, Yujia Qin, Yining Ye, Xin Cong, Yankai Lin, Yesai Wu, Zhiyuan Liu, Maosong Sun
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.13996
• Pdf URL: https://arxiv.org/pdf/2401.13996
• Copy Paste: [[2401.13996]] Investigate-Consolidate-Exploit: A General Strategy for Inter-Task Agent Self-Evolution(https://arxiv.org/abs/2401.13996)
• Keywords: gpt, agent
• Abstract: This paper introduces Investigate-Consolidate-Exploit (ICE), a novel strategy for enhancing the adaptability and flexibility of AI agents through inter-task self-evolution. Unlike existing methods focused on intra-task learning, ICE promotes the transfer of knowledge between tasks for genuine self-evolution, similar to human experience learning. The strategy dynamically investigates planning and execution trajectories, consolidates them into simplified workflows and pipelines, and exploits them for improved task execution. Our experiments on the XAgent framework demonstrate ICE's effectiveness, reducing API calls by as much as 80% and significantly decreasing the demand for the model's capability. Specifically, when combined with GPT-3.5, ICE's performance matches that of raw GPT-4 across various agent tasks. We argue that this self-evolution approach represents a paradigm shift in agent design, contributing to a more robust AI community and ecosystem, and moving a step closer to full autonomy.
• 摘要：本文介绍了调查-巩固-利用（ICE），这是一种通过任务间自我进化来增强人工智能代理的适应性和灵活性的新策略。与现有的专注于任务内学习的方法不同，ICE 促进任务之间的知识转移，实现真正的自我进化，类似于人类的经验学习。该策略动态地研究规划和执行轨迹，将它们整合到简化的工作流程和管道中，并利用它们来改进任务执行。我们在 XAgent 框架上的实验证明了 ICE 的有效性，减少了 80% 的 API 调用，并显着降低了对模型能力的需求。具体来说，当与 GPT-3.5 结合使用时，ICE 在各种代理任务中的性能与原始 GPT-4 的性能相匹配。我们认为，这种自我进化方法代表了代理设计的范式转变，有助于建立更强大的人工智能社区和生态系统，并向完全自治迈进了一步。

Title: ConstraintChecker: A Plugin for Large Language Models to Reason on Commonsense Knowledge Bases

• Authors: Quyet V. Do, Tianqing Fang, Shizhe Diao, Zhaowei Wang, Yangqiu Song
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.14003
• Pdf URL: https://arxiv.org/pdf/2401.14003
• Copy Paste: [[2401.14003]] ConstraintChecker: A Plugin for Large Language Models to Reason on Commonsense Knowledge Bases(https://arxiv.org/abs/2401.14003)
• Keywords: language model, llm, prompt, code
• Abstract: Reasoning over Commonsense Knowledge Bases (CSKB), i.e. CSKB reasoning, has been explored as a way to acquire new commonsense knowledge based on reference knowledge in the original CSKBs and external prior knowledge. Despite the advancement of Large Language Models (LLM) and prompt engineering techniques in various reasoning tasks, they still struggle to deal with CSKB reasoning. One of the problems is that it is hard for them to acquire explicit relational constraints in CSKBs from only in-context exemplars, due to a lack of symbolic reasoning capabilities (Bengio et al., 2021). To this end, we proposed **ConstraintChecker**, a plugin over prompting techniques to provide and check explicit constraints. When considering a new knowledge instance, ConstraintChecker employs a rule-based module to produce a list of constraints, then it uses a zero-shot learning module to check whether this knowledge instance satisfies all constraints. The acquired constraint-checking result is then aggregated with the output of the main prompting technique to produce the final output. Experimental results on CSKB Reasoning benchmarks demonstrate the effectiveness of our method by bringing consistent improvements over all prompting methods. Codes and data are available at \url{https://github.com/HKUST-KnowComp/ConstraintChecker}.
• 摘要：常识知识库推理（CSKB），即CSKB推理，是一种基于原始CSKB中的参考知识和外部先验知识来获取新常识知识的方法。尽管大型语言模型（LLM）在各种推理任务中取得了进步和迅速的工程技术，但它们仍然难以处理 CSKB 推理。问题之一是，由于缺乏符号推理能力，他们很难仅从上下文样本中获得 CSKB 中的显式关系约束（Bengio 等人，2021）。为此，我们提出了**ConstraintChecker**，这是一个通过提示技术来提供和检查显式约束的插件。当考虑一个新的知识实例时，ConstraintChecker 采用基于规则的模块来生成约束列表，然后使用零样本学习模块来检查该知识实例是否满足所有约束。然后将获得的约束检查结果与主要提示技术的输出聚合以产生最终输出。 CSKB Reasoning 基准测试的实验结果通过对所有提示方法带来一致的改进证明了我们方法的有效性。代码和数据可在 \url{https://github.com/HKUST-KnowComp/ConstraintChecker} 获取。

Title: CMMU: A Benchmark for Chinese Multi-modal Multi-type Question Understanding and Reasoning

• Authors: Zheqi He, Xinya Wu, Pengfei Zhou, Richeng Xuan, Guang Liu, Xi Yang, Qiannan Zhu, Hua Huang
• Subjects: cs.CL, cs.AI, cs.MM
• Abstract URL: https://arxiv.org/abs/2401.14011
• Pdf URL: https://arxiv.org/pdf/2401.14011
• Copy Paste: [[2401.14011]] CMMU: A Benchmark for Chinese Multi-modal Multi-type Question Understanding and Reasoning(https://arxiv.org/abs/2401.14011)
• Keywords: language model, gpt, llm
• Abstract: Multi-modal large language models(MLLMs) have achieved remarkable progress and demonstrated powerful knowledge comprehension and reasoning abilities. However, the mastery of domain-specific knowledge, which is essential for evaluating the intelligence of MLLMs, continues to be a challenge. Current multi-modal benchmarks for domain-specific knowledge concentrate on multiple-choice questions and are predominantly available in English, which imposes limitations on the comprehensiveness of the evaluation. To this end, we introduce CMMU, a novel benchmark for multi-modal and multi-type question understanding and reasoning in Chinese. CMMU consists of 3,603 questions in 7 subjects, covering knowledge from primary to high school. The questions can be categorized into 3 types: multiple-choice, multiple-response, and fill-in-the-blank, bringing greater challenges to MLLMs. In addition, we propose a rigorous evaluation strategy called ShiftCheck for assessing multiple-choice questions. The strategy aims to reduce position bias, minimize the influence of randomness on correctness, and perform a quantitative analysis of position bias. We evaluate seven open-source MLLMs along with GPT4-V, Gemini-Pro, and Qwen-VL-Plus. The results demonstrate that CMMU poses a significant challenge to the recent MLLMs.
• 摘要：多模态大语言模型（MLLM）取得了显着的进步，并表现出了强大的知识理解和推理能力。然而，对评估 MLLM 智能至关重要的特定领域知识的掌握仍然是一个挑战。目前针对特定领域知识的多模态基准集中于多项选择题，并且主要以英语提供，这限制了评估的全面性。为此，我们引入了 CMMU，一种用于中文多模态、多类型问题理解和推理的新颖基准。 CMMU由7个科目的3,603个问题组成，涵盖了从小学到高中的知识。题目可分为选择题、多选题、填空题3种类型，给MLLM带来了更大的挑战。此外，我们提出了一种名为 ShiftCheck 的严格评估策略，用于评估多项选择题。该策略旨在减少位置偏差，最小化随机性对正确性的影响，并对位置偏差进行定量分析。我们评估了七个开源 MLLM 以及 GPT4-V、Gemini-Pro 和 Qwen-VL-Plus。结果表明 CMMU 对最近的 MLLM 提出了重大挑战。

Title: Towards Uncertainty-Aware Language Agent

• Authors: Jiuzhou Han, Wray Buntine, Ehsan Shareghi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.14016
• Pdf URL: https://arxiv.org/pdf/2401.14016
• Copy Paste: [[2401.14016]] Towards Uncertainty-Aware Language Agent(https://arxiv.org/abs/2401.14016)
• Keywords: language model, llm, agent
• Abstract: While Language Agents have achieved promising success by placing Large Language Models at the core of a more versatile design that dynamically interacts with the external world, the existing approaches neglect the notion of uncertainty during these interactions. We present the Uncertainty-Aware Language Agent (UALA), a framework that orchestrates the interaction between the agent and the external world using uncertainty quantification. Compared with other well-known counterparts like ReAct, our extensive experiments across 3 representative tasks (HotpotQA, StrategyQA, MMLU) and various LLM sizes demonstrates that UALA brings a significant improvement of performance, while having a substantially lower reliance on the external world (i.e., reduced number of tool calls and tokens). Our analyses provide various insights including the great potential of UALA compared with agent fine-tuning, and underscoring the unreliably of verbalised confidence of LLMs as a proxy for uncertainty.
• 摘要：虽然语言代理通过将大型语言模型置于与外部世界动态交互的更通用设计的核心而取得了有希望的成功，但现有的方法忽略了这些交互过程中的不确定性概念。我们提出了不确定性感知语言代理（UALA），这是一个使用不确定性量化来协调代理与外部世界之间交互的框架。与 ReAct 等其他知名同行相比，我们在 3 个代表性任务（HotpotQA、StrategyQA、MMLU）和各种 LLM 规模的广泛实验表明，UALA 带来了性能的显着提升，同时对外部世界（即外部世界）的依赖大大降低。 ，减少工具调用和令牌的数量）。我们的分析提供了各种见解，包括 UALA 与代理微调相比的巨大潜力，并强调了法学硕士作为不确定性代理的口头信心的不可靠。

Title: Unitxt: Flexible, Shareable and Reusable Data Preparation and Evaluation for Generative AI

• Authors: Elron Bandel, Yotam Perlitz, Elad Venezian, Roni Friedman-Melamed, Ofir Arviv, Matan Orbach, Shachar Don-Yehyia, Dafna Sheinwald, Ariel Gera, Leshem Choshen, Michal Shmueli-Scheuer, Yoav Katz
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.14019
• Pdf URL: https://arxiv.org/pdf/2401.14019
• Copy Paste: [[2401.14019]] Unitxt: Flexible, Shareable and Reusable Data Preparation and Evaluation for Generative AI(https://arxiv.org/abs/2401.14019)
• Keywords: language model, lora, prompt
• Abstract: In the dynamic landscape of generative NLP, traditional text processing pipelines limit research flexibility and reproducibility, as they are tailored to specific dataset, task, and model combinations. The escalating complexity, involving system prompts, model-specific formats, instructions, and more, calls for a shift to a structured, modular, and customizable solution. Addressing this need, we present Unitxt, an innovative library for customizable textual data preparation and evaluation tailored to generative language models. Unitxt natively integrates with common libraries like HuggingFace and LM-eval-harness and deconstructs processing flows into modular components, enabling easy customization and sharing between practitioners. These components encompass model-specific formats, task prompts, and many other comprehensive dataset processing definitions. The Unitxt-Catalog centralizes these components, fostering collaboration and exploration in modern textual data workflows. Beyond being a tool, Unitxt is a community-driven platform, empowering users to build, share, and advance their pipelines collaboratively. Join the Unitxt community at https://github.com/IBM/unitxt!
• 摘要：在生成式 NLP 的动态环境中，传统的文本处理流程限制了研究的灵活性和可重复性，因为它们是针对特定数据集、任务和模型组合量身定制的。不断升级的复杂性（涉及系统提示、特定于模型的格式、指令等）要求转向结构化、模块化和可定制的解决方案。为了满足这一需求，我们推出了 Unitxt，这是一个创新库，用于针对生成语言模型定制文本数据准备和评估。 Unitxt 本身与 HuggingFace 和 LM-eval-harness 等通用库集成，并将处理流程解构为模块化组件，从而可以在从业者之间轻松定制和共享。这些组件包含特定于模型的格式、任务提示和许多其他综合数据集处理定义。 Unitxt-Catalog 集中了这些组件，促进现代文本数据工作流程中的协作和探索。 Unitxt 除了是一个工具之外，还是一个社区驱动的平台，使用户能够协作构建、共享和推进他们的管道。加入 Unitxt 社区：https://github.com/IBM/unitxt！

Title: Accelerating Retrieval-Augmented Language Model Serving with Speculation

• Authors: Zhihao Zhang, Alan Zhu, Lijie Yang, Yihua Xu, Lanting Li, Phitchaya Mangpo Phothilimthana, Zhihao Jia
• Subjects: cs.LG, cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2401.14021
• Pdf URL: https://arxiv.org/pdf/2401.14021
• Copy Paste: [[2401.14021]] Accelerating Retrieval-Augmented Language Model Serving with Speculation(https://arxiv.org/abs/2401.14021)
• Keywords: language model
• Abstract: Retrieval-augmented language models (RaLM) have demonstrated the potential to solve knowledge-intensive natural language processing (NLP) tasks by combining a non-parametric knowledge base with a parametric language model. Instead of fine-tuning a fully parametric model, RaLM excels at its low-cost adaptation to the latest data and better source attribution mechanisms. Among various RaLM approaches, iterative RaLM delivers a better generation quality due to a more frequent interaction between the retriever and the language model. Despite the benefits, iterative RaLM usually encounters high overheads due to the frequent retrieval step. To this end, we propose RaLMSpec, a speculation-inspired framework that provides generic speed-up over iterative RaLM while preserving the same model outputs through speculative retrieval and batched verification. By further incorporating prefetching, optimal speculation stride scheduler, and asynchronous verification, RaLMSpec can automatically exploit the acceleration potential to the fullest. For naive iterative RaLM serving, extensive evaluations over three language models on four downstream QA datasets demonstrate that RaLMSpec can achieve a speed-up ratio of 1.75-2.39x, 1.04-1.39x, and 1.31-1.77x when the retriever is an exact dense retriever, approximate dense retriever, and sparse retriever respectively compared with the baseline. For KNN-LM serving, RaLMSpec can achieve a speed-up ratio up to 7.59x and 2.45x when the retriever is an exact dense retriever and approximate dense retriever, respectively, compared with the baseline.
• 摘要：检索增强语言模型 (RaLM) 展示了通过将非参数知识库与参数语言模型相结合来解决知识密集型自然语言处理 (NLP) 任务的潜力。 RaLM 并非对完全参数化模型进行微调，而是以其低成本适应最新数据和更好的源归因机制而著称。在各种 RaLM 方法中，迭代 RaLM 由于检索器和语言模型之间更频繁的交互而提供了更好的生成质量。尽管有这些好处，迭代 RaLM 通常会由于频繁的检索步骤而遇到很高的开销。为此，我们提出了 RaLMSpec，这是一个受推测启发的框架，它提供了迭代 RaLM 的通用加速，同时通过推测检索和批量验证保留相同的模型输出。通过进一步结合预取、最佳推测步长调度程序和异步验证，RaLMSpec 可以自动充分利用加速潜力。对于朴素迭代 RaLM 服务，对四个下游 QA 数据集上的三种语言模型的广泛评估表明，当检索器是精确密集时，RaLMSpec 可以实现 1.75-2.39x、1.04-1.39x 和 1.31-1.77x 的加速比分别与基线相比的检索器、近似密集检索器和稀疏检索器。对于 KNN-LM 服务，与基线相比，当检索器是精确密集检索器和近似密集检索器时，RaLMSpec 可以分别实现高达 7.59 倍和 2.45 倍的加速比。

Title: Novel Quadratic Constraints for Extending LipSDP beyond Slope-Restricted Activations

• Authors: Patricia Pauli, Aaron Havens, Alexandre Araujo, Siddharth Garg, Farshad Khorrami, Frank Allgöwer, Bin Hu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.14033
• Pdf URL: https://arxiv.org/pdf/2401.14033
• Copy Paste: [[2401.14033]] Novel Quadratic Constraints for Extending LipSDP beyond Slope-Restricted Activations(https://arxiv.org/abs/2401.14033)
• Keywords: rag
• Abstract: Recently, semidefinite programming (SDP) techniques have shown great promise in providing accurate Lipschitz bounds for neural networks. Specifically, the LipSDP approach (Fazlyab et al., 2019) has received much attention and provides the least conservative Lipschitz upper bounds that can be computed with polynomial time guarantees. However, one main restriction of LipSDP is that its formulation requires the activation functions to be slope-restricted on $[0,1]$, preventing its further use for more general activation functions such as GroupSort, MaxMin, and Householder. One can rewrite MaxMin activations for example as residual ReLU networks. However, a direct application of LipSDP to the resultant residual ReLU networks is conservative and even fails in recovering the well-known fact that the MaxMin activation is 1-Lipschitz. Our paper bridges this gap and extends LipSDP beyond slope-restricted activation functions. To this end, we provide novel quadratic constraints for GroupSort, MaxMin, and Householder activations via leveraging their underlying properties such as sum preservation. Our proposed analysis is general and provides a unified approach for estimating $\ell_2$ and $\ell_\infty$ Lipschitz bounds for a rich class of neural network architectures, including non-residual and residual neural networks and implicit models, with GroupSort, MaxMin, and Householder activations. Finally, we illustrate the utility of our approach with a variety of experiments and show that our proposed SDPs generate less conservative Lipschitz bounds in comparison to existing approaches.
• 摘要：最近，半定规划 (SDP) 技术在为神经网络提供准确的 Lipschitz 边界方面显示出了巨大的前景。具体而言，LipSDP 方法（Fazlyab 等人，2019）受到了广泛关注，并提供了可以通过多项式时间保证计算的最不保守的 Lipschitz 上限。然而，LipSDP 的一个主要限制是其公式要求激活函数的斜率限制在 $[0,1]$ 上，从而阻止其进一步用于更通用的激活函数，例如 GroupSort、MaxMin 和 Householder。我们可以将 MaxMin 激活重写为残差 ReLU 网络。然而，将 LipSDP 直接应用于所得的残差 ReLU 网络是保守的，甚至无法恢复众所周知的 MaxMin 激活是 1-Lipschitz 的事实。我们的论文弥补了这一差距，并将 LipSDP 扩展到斜率限制激活函数之外。为此，我们通过利用其底层属性（例如总和保存）为 GroupSort、MaxMin 和 Householder 激活提供新颖的二次约束。我们提出的分析是通用的，并提供了一种统一的方法，用于估计丰富类别的神经网络架构的 $\ell_2$ 和 $\ell_\infty$ Lipschitz 边界，包括非残差和残差神经网络以及隐式模型，具有 GroupSort、MaxMin和家庭激活。最后，我们通过各种实验说明了我们的方法的实用性，并表明与现有方法相比，我们提出的 SDP 生成的 Lipschitz 边界不太保守。

Title: (Chat)GPT v BERT: Dawn of Justice for Semantic Change Detection

• Authors: Francesco Periti, Haim Dubossarsky, Nina Tahmasebi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.14040
• Pdf URL: https://arxiv.org/pdf/2401.14040
• Copy Paste: [[2401.14040]] (Chat)GPT v BERT: Dawn of Justice for Semantic Change Detection(https://arxiv.org/abs/2401.14040)
• Keywords: language model, gpt, chat
• Abstract: In the universe of Natural Language Processing, Transformer-based language models like BERT and (Chat)GPT have emerged as lexical superheroes with great power to solve open research problems. In this paper, we specifically focus on the temporal problem of semantic change, and evaluate their ability to solve two diachronic extensions of the Word-in-Context (WiC) task: TempoWiC and HistoWiC. In particular, we investigate the potential of a novel, off-the-shelf technology like ChatGPT (and GPT) 3.5 compared to BERT, which represents a family of models that currently stand as the state-of-the-art for modeling semantic change. Our experiments represent the first attempt to assess the use of (Chat)GPT for studying semantic change. Our results indicate that ChatGPT performs significantly worse than the foundational GPT version. Furthermore, our results demonstrate that (Chat)GPT achieves slightly lower performance than BERT in detecting long-term changes but performs significantly worse in detecting short-term changes.
• 摘要：在自然语言处理领域，BERT 和（Chat）GPT 等基于 Transformer 的语言模型已成为词汇超级英雄，具有解决开放研究问题的强大能力。在本文中，我们特别关注语义变化的时间问题，并评估它们解决 Word-in-Context (WiC) 任务的两个历时扩展的能力：TempoWiC 和 HistoWiC。特别是，我们研究了 ChatGPT（和 GPT）3.5 等新颖的现成技术与 BERT 相比的潜力，BERT 代表了目前最先进的语义变化建模模型系列。我们的实验代表了评估（聊天）GPT 在研究语义变化中的使用的首次尝试。我们的结果表明 ChatGPT 的性能明显比基础 GPT 版本差。此外，我们的结果表明，(Chat)GPT 在检测长期变化方面的性能略低于 BERT，但在检测短期变化方面的性能明显较差。

Title: Towards Goal-oriented Large Language Model Prompting: A Survey

• Authors: Haochen Li, Jonathan Leung, Zhiqi Shen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.14043
• Pdf URL: https://arxiv.org/pdf/2401.14043
• Copy Paste: [[2401.14043]] Towards Goal-oriented Large Language Model Prompting: A Survey(https://arxiv.org/abs/2401.14043)
• Keywords: language model, llm, prompt
• Abstract: Large Language Models (LLMs) have shown prominent performance in various downstream tasks in which prompt engineering plays a pivotal role in optimizing LLMs' performance. This paper, not as an overview of current prompt engineering methods, aims to highlight the limitation of designing prompts while holding an anthropomorphic assumption that expects LLMs to think like humans. From our review of 35 representative studies, we demonstrate that a goal-oriented prompt formulation, which guides LLMs to follow established human logical thinking, significantly improves the performance of LLMs. Furthermore, We introduce a novel taxonomy that categorizes goal-oriented prompting methods into five interconnected stages and we demonstrate the broad applicability of our framework by summarizing ten applicable tasks. With four future directions proposed, we hope to further emphasize and promote goal-oriented prompt engineering.
• 摘要：大型语言模型 (LLM) 在各种下游任务中表现出了突出的性能，其中即时工程在优化 LLM 性能方面发挥着关键作用。本文并不是对当前提示工程方法的概述，而是旨在强调设计提示的局限性，同时持有期望法学硕士像人类一样思考的拟人化假设。通过对 35 项代表性研究的回顾，我们证明，以目标为导向的提示制定可以引导法学硕士遵循既定的人类逻辑思维，从而显着提高法学硕士的表现。此外，我们引入了一种新颖的分类法，将目标导向的提示方法分为五个相互关联的阶段，并通过总结十个适用的任务来证明我们的框架的广泛适用性。提出了四个未来方向，希望进一步强调和推进目标导向的快速工程。

Title: Ta'keed: The First Generative Fact-Checking System for Arabic Claims

• Authors: Saud Althabiti, Mohammad Ammar Alsalka, Eric Atwell
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.14067
• Pdf URL: https://arxiv.org/pdf/2401.14067
• Copy Paste: [[2401.14067]] Ta'keed: The First Generative Fact-Checking System for Arabic Claims(https://arxiv.org/abs/2401.14067)
• Keywords: llm, lora, rag
• Abstract: This paper introduces Ta'keed, an explainable Arabic automatic fact-checking system. While existing research often focuses on classifying claims as "True" or "False," there is a limited exploration of generating explanations for claim credibility, particularly in Arabic. Ta'keed addresses this gap by assessing claim truthfulness based on retrieved snippets, utilizing two main components: information retrieval and LLM-based claim verification. We compiled the ArFactEx, a testing gold-labelled dataset with manually justified references, to evaluate the system. The initial model achieved a promising F1 score of 0.72 in the classification task. Meanwhile, the system's generated explanations are compared with gold-standard explanations syntactically and semantically. The study recommends evaluating using semantic similarities, resulting in an average cosine similarity score of 0.76. Additionally, we explored the impact of varying snippet quantities on claim classification accuracy, revealing a potential correlation, with the model using the top seven hits outperforming others with an F1 score of 0.77.
• 摘要：本文介绍了 Ta'keed，一个可解释的阿拉伯语自动事实检查系统。虽然现有的研究通常侧重于将主张分类为“真”或“假”，但对为主张可信度（尤其是阿拉伯语）生成解释的探索有限。 Ta'keed 通过基于检索到的片段评估声明的真实性来解决这一差距，利用两个主要组成部分：信息检索和基于 LLM 的声明验证。我们编译了 ArFactEx，这是一个带有手动合理参考的测试黄金标记数据集，用于评估系统。初始模型在分类任务中取得了令人鼓舞的 F1 分数 0.72。同时，系统生成的解释在句法和语义上与黄金标准解释进行比较。该研究建议使用语义相似度进行评估，平均余弦相似度得分为 0.76。此外，我们还探讨了不同片段数量对索赔分类准确性的影响，揭示了潜在的相关性，使用前 7 个命中的模型优于其他模型，F1 得分为 0.77。

Title: Learning under Label Noise through Few-Shot Human-in-the-Loop Refinement

• Authors: Aaqib Saeed, Dimitris Spathis, Jungwoo Oh, Edward Choi, Ali Etemad
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2401.14107
• Pdf URL: https://arxiv.org/pdf/2401.14107
• Copy Paste: [[2401.14107]] Learning under Label Noise through Few-Shot Human-in-the-Loop Refinement(https://arxiv.org/abs/2401.14107)
• Keywords: rag
• Abstract: Wearable technologies enable continuous monitoring of various health metrics, such as physical activity, heart rate, sleep, and stress levels. A key challenge with wearable data is obtaining quality labels. Unlike modalities like video where the videos themselves can be effectively used to label objects or events, wearable data do not contain obvious cues about the physical manifestation of the users and usually require rich metadata. As a result, label noise can become an increasingly thorny issue when labeling such data. In this paper, we propose a novel solution to address noisy label learning, entitled Few-Shot Human-in-the-Loop Refinement (FHLR). Our method initially learns a seed model using weak labels. Next, it fine-tunes the seed model using a handful of expert corrections. Finally, it achieves better generalizability and robustness by merging the seed and fine-tuned models via weighted parameter averaging. We evaluate our approach on four challenging tasks and datasets, and compare it against eight competitive baselines designed to deal with noisy labels. We show that FHLR achieves significantly better performance when learning from noisy labels and achieves state-of-the-art by a large margin, with up to 19% accuracy improvement under symmetric and asymmetric noise. Notably, we find that FHLR is particularly robust to increased label noise, unlike prior works that suffer from severe performance degradation. Our work not only achieves better generalization in high-stakes health sensing benchmarks but also sheds light on how noise affects commonly-used models.
• 摘要：可穿戴技术可以持续监测各种健康指标，例如体力活动、心率、睡眠和压力水平。可穿戴数据的一个关键挑战是获得质量标签。与视频等方式不同，视频本身可以有效地用于标记对象或事件，可穿戴数据不包含有关用户身体表现的明显线索，并且通常需要丰富的元数据。因此，在标记此类数据时，标签噪声可能成为一个越来越棘手的问题。在本文中，我们提出了一种解决噪声标签学习的新颖解决方案，名为“Few-Shot Human-in-the-Loop Refinement”（FHLR）。我们的方法最初使用弱标签学习种子模型。接下来，它使用一些专家修正来微调种子模型。最后，它通过加权参数平均合并种子模型和微调模型，实现了更好的通用性和鲁棒性。我们在四个具有挑战性的任务和数据集上评估我们的方法，并将其与旨在处理噪声标签的八个竞争基线进行比较。我们证明，FHLR 在从噪声标签中学习时取得了显着更好的性能，并大幅达到了最先进水平，在对称和非对称噪声下准确率提高了 19%。值得注意的是，我们发现 FHLR 对于增加的标签噪声特别稳健，这与之前遭受严重性能下降的工作不同。我们的工作不仅在高风险健康传感基准中实现了更好的泛化，而且还揭示了噪声如何影响常用模型。

Title: CompactifAI: Extreme Compression of Large Language Models using Quantum-Inspired Tensor Networks

• Authors: Andrei Tomut, Saeed S. Jahromi, Sukhbinder Singh, Faysal Ishtiaq, Cesar Muñoz, Prabdeep Singh Bajaj, Ali Elborady, Gianni del Bimbo, Mehrazin Alizadeh, David Montero, Pablo Martin-Ramiro, Muhammad Ibrahim, Oussama Tahiri Alaoui, John Malcolm, Samuel Mugel, Roman Orus
• Subjects: cs.CL, cs.AI, cs.LG, quant-ph
• Abstract URL: https://arxiv.org/abs/2401.14109
• Pdf URL: https://arxiv.org/pdf/2401.14109
• Copy Paste: [[2401.14109]] CompactifAI: Extreme Compression of Large Language Models using Quantum-Inspired Tensor Networks(https://arxiv.org/abs/2401.14109)
• Keywords: language model, gpt, llm, chat
• Abstract: Large Language Models (LLMs) such as ChatGPT and LlaMA are advancing rapidly in generative Artificial Intelligence (AI), but their immense size poses significant challenges, such as huge training and inference costs, substantial energy demands, and limitations for on-site deployment. Traditional compression methods such as pruning, distillation, and low-rank approximation focus on reducing the effective number of neurons in the network, while quantization focuses on reducing the numerical precision of individual weights to reduce the model size while keeping the number of neurons fixed. While these compression methods have been relatively successful in practice, there's no compelling reason to believe that truncating the number of neurons is an optimal strategy. In this context, this paper introduces CompactifAI, an innovative LLM compression approach using quantum-inspired Tensor Networks that focuses on the model's correlation space instead, allowing for a more controlled, refined and interpretable model compression. Our method is versatile and can be implemented with - or on top of - other compression techniques. As a benchmark, we demonstrate that CompactifAI alone enables compression of the LlaMA-2 7B model to only $30\%$ of its original size while recovering over $90\%$ of the original accuracy after a brief distributed retraining.
• 摘要：ChatGPT 和 LlaMA 等大型语言模型 (LLM) 在生成人工智能 (AI) 领域迅速发展，但其巨大的规模带来了重大挑战，例如巨大的训练和推理成本、大量的能源需求以及现场部署的限制。传统的压缩方法如剪枝、蒸馏和低秩近似等侧重于减少网络中的有效神经元数量，而量化则侧重于降低各个权重的数值精度，以在保持神经元数量固定的情况下减小模型大小。虽然这些压缩方法在实践中相对成功，但没有令人信服的理由相信截断神经元数量是最佳策略。在此背景下，本文介绍了 CompactifAI，这是一种创新的 LLM 压缩方法，使用受量子启发的张量网络，重点关注模型的相关空间，从而实现更加受控、精细和可解释的模型压缩。我们的方法是通用的，可以与其他压缩技术一起或在其他压缩技术之上实现。作为基准，我们证明仅 CompactifAI 就可以将 LlaMA-2 7B 模型压缩至原始大小的 30%$，同时在短暂的分布式再训练后恢复超过原始精度的 90%$。

Title: FP6-LLM: Efficiently Serving Large Language Models Through FP6-Centric Algorithm-System Co-Design

• Authors: Haojun Xia, Zhen Zheng, Xiaoxia Wu, Shiyang Chen, Zhewei Yao, Stephen Youn, Arash Bakhtiari, Michael Wyatt, Donglin Zhuang, Zhongzhu Zhou, Olatunji Ruwase, Yuxiong He, Shuaiwen Leon Song
• Subjects: cs.LG, cs.AI, cs.AR
• Abstract URL: https://arxiv.org/abs/2401.14112
• Pdf URL: https://arxiv.org/pdf/2401.14112
• Copy Paste: [[2401.14112]] FP6-LLM: Efficiently Serving Large Language Models Through FP6-Centric Algorithm-System Co-Design(https://arxiv.org/abs/2401.14112)
• Keywords: language model, llm, code
• Abstract: Six-bit quantization (FP6) can effectively reduce the size of large language models (LLMs) and preserve the model quality consistently across varied applications. However, existing systems do not provide Tensor Core support for FP6 quantization and struggle to achieve practical performance improvements during LLM inference. It is challenging to support FP6 quantization on GPUs due to (1) unfriendly memory access of model weights with irregular bit-width and (2) high runtime overhead of weight de-quantization. To address these problems, we propose TC-FPx, the first full-stack GPU kernel design scheme with unified Tensor Core support of float-point weights for various quantization bit-width. We integrate TC-FPx kernel into an existing inference system, providing new end-to-end support (called FP6-LLM) for quantized LLM inference, where better trade-offs between inference cost and model quality are achieved. Experiments show that FP6-LLM enables the inference of LLaMA-70b using only a single GPU, achieving 1.69x-2.65x higher normalized inference throughput than the FP16 baseline. The source code will be publicly available soon.
• 摘要：六位量化 (FP6) 可以有效地减小大型语言模型 (LLM) 的大小，并在不同的应用程序中保持一致的模型质量。然而，现有系统不提供对 FP6 量化的 Tensor Core 支持，并且很难在 LLM 推理过程中实现实际的性能改进。在 GPU 上支持 FP6 量化具有挑战性，因为 (1) 位宽不规则的模型权重的内存访问不友好，以及 (2) 权重反量化的运行时开销较高。为了解决这些问题，我们提出了TC-FPx，这是第一个全栈GPU内核设计方案，具有统一的Tensor Core支持各种量化位宽的浮点权重。我们将 TC-FPx 内核集成到现有的推理系统中，为量化的 LLM 推理提供新的端到端支持（称为 FP6-LLM），从而在推理成本和模型质量之间实现更好的权衡。实验表明，FP6-LLM 仅使用单个 GPU 即可实现 LLaMA-70b 的推理，实现比 FP16 基线高 1.69 倍至 2.65 倍的归一化推理吞吐量。源代码将很快公开。

Title: On the Affinity, Rationality, and Diversity of Hierarchical Topic Modeling

• Authors: Xiaobao Wu, Fengjun Pan, Thong Nguyen, Yichao Feng, Chaoqun Liu, Cong-Duy Nguyen, Anh Tuan Luu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.14113
• Pdf URL: https://arxiv.org/pdf/2401.14113
• Copy Paste: [[2401.14113]] On the Affinity, Rationality, and Diversity of Hierarchical Topic Modeling(https://arxiv.org/abs/2401.14113)
• Keywords: code
• Abstract: Hierarchical topic modeling aims to discover latent topics from a corpus and organize them into a hierarchy to understand documents with desirable semantic granularity. However, existing work struggles with producing topic hierarchies of low affinity, rationality, and diversity, which hampers document understanding. To overcome these challenges, we in this paper propose Transport Plan and Context-aware Hierarchical Topic Model (TraCo). Instead of early simple topic dependencies, we propose a transport plan dependency method. It constrains dependencies to ensure their sparsity and balance, and also regularizes topic hierarchy building with them. This improves affinity and diversity of hierarchies. We further propose a context-aware disentangled decoder. Rather than previously entangled decoding, it distributes different semantic granularity to topics at different levels by disentangled decoding. This facilitates the rationality of hierarchies. Experiments on benchmark datasets demonstrate that our method surpasses state-of-the-art baselines, effectively improving the affinity, rationality, and diversity of hierarchical topic modeling with better performance on downstream tasks.
• 摘要：分层主题建模旨在从语料库中发现潜在主题并将其组织成层次结构，以理解具有所需语义粒度的文档。然而，现有的工作很难产生低亲和力、合理性和多样性的主题层次结构，这阻碍了文档的理解。为了克服这些挑战，我们在本文中提出了传输计划和上下文感知分层主题模型（TraCo）。我们提出了一种传输计划依赖方法，而不是早期的简单主题依赖。它限制依赖关系以确保它们的稀疏性和平衡，并规范它们的主题层次结构构建。这提高了层次结构的亲和力和多样性。我们进一步提出了一种上下文感知的解缠解码器。它不是以前的纠缠解码，而是通过解纠缠解码将不同的语义粒度分配到不同级别的主题。这有利于层次结构的合理性。在基准数据集上的实验表明，我们的方法超越了最先进的基线，有效提高了层次主题建模的亲和力、合理性和多样性，在下游任务上具有更好的性能。

Title: True Knowledge Comes from Practice: Aligning LLMs with Embodied Environments via Reinforcement Learning

• Authors: Weihao Tan, Wentao Zhang, Shanqi Liu, Longtao Zheng, Xinrun Wang, Bo An
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2401.14151
• Pdf URL: https://arxiv.org/pdf/2401.14151
• Copy Paste: [[2401.14151]] True Knowledge Comes from Practice: Aligning LLMs with Embodied Environments via Reinforcement Learning(https://arxiv.org/abs/2401.14151)
• Keywords: language model, llm, lora, prompt, agent
• Abstract: Despite the impressive performance across numerous tasks, large language models (LLMs) often fail in solving simple decision-making tasks due to the misalignment of the knowledge in LLMs with environments. On the contrary, reinforcement learning (RL) agents learn policies from scratch, which makes them always align with environments but difficult to incorporate prior knowledge for efficient explorations. To narrow the gap, we propose TWOSOME, a novel general online framework that deploys LLMs as decision-making agents to efficiently interact and align with embodied environments via RL without requiring any prepared datasets or prior knowledge of the environments. Firstly, we query the joint probabilities of each valid action with LLMs to form behavior policies. Then, to enhance the stability and robustness of the policies, we propose two normalization methods and summarize four prompt design principles. Finally, we design a novel parameter-efficient training architecture where the actor and critic share one frozen LLM equipped with low-rank adapters (LoRA) updated by PPO. We conduct extensive experiments to evaluate TWOSOME. i) TWOSOME exhibits significantly better sample efficiency and performance compared to the conventional RL method, PPO, and prompt tuning method, SayCan, in both classical decision-making environment, Overcooked, and simulated household environment, VirtualHome. ii) Benefiting from LLMs' open-vocabulary feature, TWOSOME shows superior generalization ability to unseen tasks. iii) Under our framework, there is no significant loss of the LLMs' original ability during online PPO finetuning.
• 摘要：尽管在众多任务中表现出色，但由于 LLM 中的知识与环境不一致，大型语言模型 (LLM) 常常无法解决简单的决策任务。相反，强化学习（RL）智能体从头开始学习策略，这使得它们始终与环境保持一致，但很难结合先验知识进行有效的探索。为了缩小差距，我们提出了 TWOSOME，这是一种新颖的通用在线框架，它将 LLM 部署为决策代理，通过 RL 与具体环境进行有效交互和协调，而不需要任何准备好的数据集或环境的先验知识。首先，我们使用 LLM 查询每个有效动作的联合概率，以形成行为策略。然后，为了增强政策的稳定性和稳健性，我们提出了两种规范化方法并总结了四个及时的设计原则。最后，我们设计了一种新颖的参数高效训练架构，其中演员和评论家共享一个配备有由 PPO 更新的低秩适配器 (LoRA) 的冻结 LLM。我们进行了大量的实验来评估 TWOSOME。 i) 在经典决策环境 Overcooked 和模拟家庭环境 VirtualHome 中，与传统的 RL 方法 PPO 和即时调整方法 SayCan 相比，TWOSOME 表现出明显更好的样本效率和性能。 ii) 受益于法学硕士的开放词汇特征，TWOSOME 对未见过的任务表现出了卓越的泛化能力。 iii) 在我们的框架下，在线PPO微调过程中LLM的原有能力并没有明显损失。

Title: Agent-based Simulation with Netlogo to Evaluate AmI Scenarios

• Authors: J. Carbo, N. Sanchez, J. M. Molina
• Subjects: cs.AI
• Abstract URL: https://arxiv.org/abs/2401.14153
• Pdf URL: https://arxiv.org/pdf/2401.14153
• Copy Paste: [[2401.14153]] Agent-based Simulation with Netlogo to Evaluate AmI Scenarios(https://arxiv.org/abs/2401.14153)
• Keywords: agent
• Abstract: In this paper an agent-based simulation is developed in order to evaluate an AmI scenario based on agents. Many AmI applications are implemented through agents but they are not compared to any other existing alternative in order to evaluate the relative benefits of using them. The proposal simulation environment developed in Netlogo analyse such benefits using two evaluation criteria: First, measuring agent satisfaction of different types of desires along the execution. Second, measuring time savings obtained through a correct use of context information. So, here, a previously suggested agent architecture, an ontology and a 12-steps protocol to provide AmI services in airports, is evaluated using a NetLogo simulation environment. The present work uses a NetLogo model considering scalability problems of this application domain but using FIPA and BDI extensions to be coherent with our previous works and our previous JADE implementation of them. The NetLogo model presented simulates an airport with agent users passing through several zones located in a specific order in a map: passport controls, check-in counters of airline companies, boarding gates, different types of shopping. Although initial data in simulations are generated randomly, and the model is just an approximation of real-world airports, the definition of this case of use of Ambient Intelligence through NetLogo agents opens an interesting way to evaluate the benefits of using Ambient Intelligence, which is a significant contribution to the final development of them.
• 摘要：本文开发了基于代理的模拟，以评估基于代理的 AMI 场景。许多 AmI 应用程序是通过代理实现的，但并未将它们与任何其他现有替代方案进行比较，以评估使用它们的相对优势。 Netlogo 中开发的提案模拟环境使用两个评估标准来分析这些好处：首先，测量代理在执行过程中对不同类型愿望的满意度。其次，衡量通过正确使用上下文信息所节省的时间。因此，这里使用 NetLogo 模拟环境评估先前建议的代理架构、本体和在机场提供 AmI 服务的 12 步协议。目前的工作使用 NetLogo 模型，考虑到该应用程序域的可扩展性问题，但使用 FIPA 和 BDI 扩展来与我们之前的工作和之前的 JADE 实现保持一致。提出的 NetLogo 模型模拟了一个机场，代理用户穿过地图上按特定顺序排列的多个区域：护照检查、航空公司值机柜台、登机口、不同类型的购物。尽管模拟中的初始数据是随机生成的，并且该模型只是现实世界机场的近似值，但通过 NetLogo 代理使用环境智能的案例​​的定义为评估使用环境智能的好处提供了一种有趣的方法，即对他们的最终发展做出了重大贡献。

Title: Alleviating Structural Distribution Shift in Graph Anomaly Detection

• Authors: Yuan Gao, Xiang Wang, Xiangnan He, Zhenguang Liu, Huamin Feng, Yongdong Zhang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.14155
• Pdf URL: https://arxiv.org/pdf/2401.14155
• Copy Paste: [[2401.14155]] Alleviating Structural Distribution Shift in Graph Anomaly Detection(https://arxiv.org/abs/2401.14155)
• Keywords: code
• Abstract: Graph anomaly detection (GAD) is a challenging binary classification problem due to its different structural distribution between anomalies and normal nodes -- abnormal nodes are a minority, therefore holding high heterophily and low homophily compared to normal nodes. Furthermore, due to various time factors and the annotation preferences of human experts, the heterophily and homophily can change across training and testing data, which is called structural distribution shift (SDS) in this paper. The mainstream methods are built on graph neural networks (GNNs), benefiting the classification of normals from aggregating homophilous neighbors, yet ignoring the SDS issue for anomalies and suffering from poor generalization. This work solves the problem from a feature view. We observe that the degree of SDS varies between anomalies and normal nodes. Hence to address the issue, the key lies in resisting high heterophily for anomalies meanwhile benefiting the learning of normals from homophily. We tease out the anomaly features on which we constrain to mitigate the effect of heterophilous neighbors and make them invariant. We term our proposed framework as Graph Decomposition Network (GDN). Extensive experiments are conducted on two benchmark datasets, and the proposed framework achieves a remarkable performance boost in GAD, especially in an SDS environment where anomalies have largely different structural distribution across training and testing environments. Codes are open-sourced in https://github.com/blacksingular/wsdm_GDN.
• 摘要：图异常检测（GAD）是一个具有挑战性的二元分类问题，因为异常节点和正常节点之间的结构分布不同——异常节点占少数，因此与正常节点相比具有较高的异质性和较低的同质性。此外，由于各种时间因素和人类专家的注释偏好，异质性和同质性可能会在训练和测试数据之间发生变化，本文将其称为结构分布偏移（SDS）。主流方法建立在图神经网络（GNN）的基础上，有利于通过聚合同源邻居来对正常进行分类，但忽略了异常的 SDS 问题并且泛化能力较差。这项工作从特征角度解决了这个问题。我们观察到异常节点和正常节点之间的 SDS 程度有所不同。因此，解决这个问题的关键在于抵制异常的高异质性，同时有利于正常现象从同质性中学习。我们梳理出我们限制的异常特征，以减轻异性邻居的影响并使它们保持不变。我们将我们提出的框架称为图分解网络（GDN）。在两个基准数据集上进行了大量的实验，所提出的框架在 GAD 中实现了显着的性能提升，特别是在 SDS 环境中，异常在训练和测试环境中具有很大不同的结构分布。代码在 https://github.com/blacksingular/wsdm_GDN 中开源。

Title: BayesPrompt: Prompting Large-Scale Pre-Trained Language Models on Few-shot Inference via Debiased Domain Abstraction

• Authors: Jiangmeng Li, Fei Song, Yifan Jin, Wenwen Qiang, Changwen Zheng, Fuchun Sun, Hui Xiong
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.14166
• Pdf URL: https://arxiv.org/pdf/2401.14166
• Copy Paste: [[2401.14166]] BayesPrompt: Prompting Large-Scale Pre-Trained Language Models on Few-shot Inference via Debiased Domain Abstraction(https://arxiv.org/abs/2401.14166)
• Keywords: language model, prompt, rag
• Abstract: As a novel and effective fine-tuning paradigm based on large-scale pre-trained language models (PLMs), prompt-tuning aims to reduce the gap between downstream tasks and pre-training objectives. While prompt-tuning has yielded continuous advancements in various tasks, such an approach still remains a persistent defect: prompt-tuning methods fail to generalize to specific few-shot patterns. From the perspective of distribution analyses, we disclose that the intrinsic issues behind the phenomenon are the over-multitudinous conceptual knowledge contained in PLMs and the abridged knowledge for target downstream domains, which jointly result in that PLMs mis-locate the knowledge distributions corresponding to the target domains in the universal knowledge embedding space. To this end, we intuitively explore to approximate the unabridged target domains of downstream tasks in a debiased manner, and then abstract such domains to generate discriminative prompts, thereby providing the de-ambiguous guidance for PLMs. Guided by such an intuition, we propose a simple yet effective approach, namely BayesPrompt, to learn prompts that contain the domain discriminative information against the interference from domain-irrelevant knowledge. BayesPrompt primitively leverages known distributions to approximate the debiased factual distributions of target domains and further uniformly samples certain representative features from the approximated distributions to generate the ultimate prompts for PLMs. We provide theoretical insights with the connection to domain adaptation. Empirically, our method achieves state-of-the-art performance on benchmarks.
• 摘要：作为一种基于大规模预训练语言模型（PLM）的新颖有效的微调范式，即时调优旨在缩小下游任务与预训练目标之间的差距。虽然即时调整在各种任务中取得了持续的进步，但这种方法仍然存在一个持久的缺陷：即时调整方法无法泛化到特定的少数样本模式。从分布分析的角度，我们揭示了这一现象背后的本质问题是PLM中包含的概念性知识过多以及针对目标下游领域的知识的删减，共同导致PLM错误定位了与目标相关的知识分布。通用知识嵌入空间中的目标领域。为此，我们直观地探索以去偏的方式近似下游任务的未删减的目标域，然后抽象这些域以生成判别性提示，从而为 PLM 提供消除歧义的指导。在这种直觉的指导下，我们提出了一种简单而有效的方法，即BayesPrompt，来学习包含领域判别信息的提示，以抵御与领域无关的知识的干扰。 BayesPrompt 原始地利用已知分布来近似目标域的去偏事实分布，并进一步从近似分布中均匀采样某些代表性特征，以生成 PLM 的最终提示。我们提供与领域适应相关的理论见解。根据经验，我们的方法在基准测试中实现了最先进的性能。

Title: Towards Autonomous Supply Chains: Definition, Characteristics, Conceptual Framework, and Autonomy Levels

• Authors: Liming Xu, Stephen Mak, Yaniv Proselkov, Alexandra Brintrup
• Subjects: cs.AI, cs.MA, eess.SY, math.OC
• Abstract URL: https://arxiv.org/abs/2401.14183
• Pdf URL: https://arxiv.org/pdf/2401.14183
• Copy Paste: [[2401.14183]] Towards Autonomous Supply Chains: Definition, Characteristics, Conceptual Framework, and Autonomy Levels(https://arxiv.org/abs/2401.14183)
• Keywords: lora
• Abstract: Recent global disruptions, such as the pandemic and geopolitical conflicts, have profoundly exposed vulnerabilities in traditional supply chains, requiring exploration of more resilient alternatives. Autonomous supply chains (ASCs) have emerged as a potential solution, offering increased visibility, flexibility, and resilience in turbulent trade environments. Despite discussions in industry and academia over several years, ASCs lack well-established theoretical foundations. This paper addresses this research gap by presenting a formal definition of ASC along with its defining characteristics and auxiliary concepts. We propose a layered conceptual framework called the MIISI model. An illustrative case study focusing on the meat supply chain demonstrates an initial ASC implementation based on this conceptual model. Additionally, we introduce a seven-level supply chain autonomy reference model, delineating a trajectory towards achieving a full supply chain autonomy. Recognising that this work represents an initial endeavour, we emphasise the need for continued exploration in this emerging domain. We anticipate that this work will stimulate further research, both theoretical and technical, and contribute to the continual evolution of ASCs.
• 摘要：最近的全球性破坏，例如大流行和地缘政治冲突，深刻暴露了传统供应链的脆弱性，需要探索更具弹性的替代方案。自主供应链 (ASC) 已成为一种潜在的解决方案，可在动荡的贸易环境中提供更高的可见性、灵活性和弹性。尽管业界和学术界进行了多年的讨论，ASC 仍缺乏完善的理论基础。本文通过提出 ASC 的正式定义及其定义特征和辅助概念来解决这一研究空白。我们提出了一个分层概念框架，称为 MIISI 模型。一个以肉类供应链为重点的说明性案例研究展示了基于此概念模型的初始 ASC 实施。此外，我们引入了七级供应链自治参考模型，描绘了实现完全供应链自治的轨迹。认识到这项工作代表了初步的努力，我们强调需要在这个新兴领域继续探索。我们预计这项工作将刺激理论和技术方面的进一步研究，并有助于 ASC 的不断发展。

Title: How Can Large Language Models Understand Spatial-Temporal Data?

• Authors: Lei Liu, Shuo Yu, Runze Wang, Zhenxun Ma, Yanming Shen
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2401.14192
• Pdf URL: https://arxiv.org/pdf/2401.14192
• Copy Paste: [[2401.14192]] How Can Large Language Models Understand Spatial-Temporal Data?(https://arxiv.org/abs/2401.14192)
• Keywords: language model, llm
• Abstract: While Large Language Models (LLMs) dominate tasks like natural language processing and computer vision, harnessing their power for spatial-temporal forecasting remains challenging. The disparity between sequential text and complex spatial-temporal data hinders this application. To address this issue, this paper introduces STG-LLM, an innovative approach empowering LLMs for spatial-temporal forecasting. We tackle the data mismatch by proposing: 1) STG-Tokenizer: This spatial-temporal graph tokenizer transforms intricate graph data into concise tokens capturing both spatial and temporal relationships; 2) STG-Adapter: This minimalistic adapter, consisting of linear encoding and decoding layers, bridges the gap between tokenized data and LLM comprehension. By fine-tuning only a small set of parameters, it can effectively grasp the semantics of tokens generated by STG-Tokenizer, while preserving the original natural language understanding capabilities of LLMs. Extensive experiments on diverse spatial-temporal benchmark datasets show that STG-LLM successfully unlocks LLM potential for spatial-temporal forecasting. Remarkably, our approach achieves competitive performance on par with dedicated SOTA methods.
• 摘要：虽然大型语言模型 (LLM) 在自然语言处理和计算机视觉等任务中占据主导地位，但利用其能力进行时空预测仍然具有挑战性。顺序文本和复杂的时空数据之间的差异阻碍了这种应用。为了解决这个问题，本文介绍了 STG-LLM，这是一种支持 LLM 进行时空预测的创新方法。我们通过建议解决数据不匹配问题：1）STG-Tokenizer：这种时空图标记器将复杂的图数据转换为捕获空间和时间关系的简洁标记； 2) STG-Adapter：这种简约的适配器由线性编码和解码层组成，弥合了标记化数据和 LLM 理解之间的差距。通过仅微调一小部分参数，它可以有效地掌握 STG-Tokenizer 生成的标记的语义，同时保留 LLM 原有的自然语言理解能力。对不同时空基准数据集的广泛实验表明，STG-LLM 成功释放了 LLM 在时空预测方面的潜力。值得注意的是，我们的方法实现了与专用 SOTA 方法相当的竞争性能。

Title: Parameter-Efficient Conversational Recommender System as a Language Processing Task

• Authors: Mathieu Ravaut, Hao Zhang, Lu Xu, Aixin Sun, Yong Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.14194
• Pdf URL: https://arxiv.org/pdf/2401.14194
• Copy Paste: [[2401.14194]] Parameter-Efficient Conversational Recommender System as a Language Processing Task(https://arxiv.org/abs/2401.14194)
• Keywords: language model, code, rag
• Abstract: Conversational recommender systems (CRS) aim to recommend relevant items to users by eliciting user preference through natural language conversation. Prior work often utilizes external knowledge graphs for items' semantic information, a language model for dialogue generation, and a recommendation module for ranking relevant items. This combination of multiple components suffers from a cumbersome training process, and leads to semantic misalignment issues between dialogue generation and item recommendation. In this paper, we represent items in natural language and formulate CRS as a natural language processing task. Accordingly, we leverage the power of pre-trained language models to encode items, understand user intent via conversation, perform item recommendation through semantic matching, and generate dialogues. As a unified model, our PECRS (Parameter-Efficient CRS), can be optimized in a single stage, without relying on non-textual metadata such as a knowledge graph. Experiments on two benchmark CRS datasets, ReDial and INSPIRED, demonstrate the effectiveness of PECRS on recommendation and conversation. Our code is available at: https://github.com/Ravoxsg/efficient_unified_crs.
• 摘要：会话推荐系统（CRS）旨在通过自然语言对话引出用户偏好，向用户推荐相关项目。先前的工作经常利用外部知识图来获取项目的语义信息、用于对话生成的语言模型以及用于对相关项目进行排名的推荐模块。这种多个组件的组合会遭受繁琐的训练过程，并导致对话生成和项目推荐之间的语义错位问题。在本文中，我们用自然语言表示项目，并将 CRS 制定为自然语言处理任务。因此，我们利用预先训练的语言模型的力量来编码项目，通过对话理解用户意图，通过语义匹配执行项目推荐，并生成对话。作为一个统一的模型，我们的 PECRS（参数高效 CRS）可以在单个阶段进行优化，而不依赖于知识图等非文本元数据。在两个基准 CRS 数据集 ReDial 和 INSPIRED 上进行的实验证明了 PECRS 在推荐和对话方面的有效性。我们的代码位于：https://github.com/Ravoxsg/efficient_unified_crs。

Title: At the junction between deep learning and statistics of extremes: formalizing the landslide hazard definition

• Authors: Ashok Dahal, Raphaël Huser, Luigi Lombardo
• Subjects: cs.LG, physics.geo-ph, stat.AP, stat.ML
• Abstract URL: https://arxiv.org/abs/2401.14210
• Pdf URL: https://arxiv.org/pdf/2401.14210
• Copy Paste: [[2401.14210]] At the junction between deep learning and statistics of extremes: formalizing the landslide hazard definition(https://arxiv.org/abs/2401.14210)
• Keywords: rag
• Abstract: The most adopted definition of landslide hazard combines spatial information about landslide location (susceptibility), threat (intensity), and frequency (return period). Only the first two elements are usually considered and estimated when working over vast areas. Even then, separate models constitute the standard, with frequency being rarely investigated. Frequency and intensity are intertwined and depend on each other because larger events occur less frequently and vice versa. However, due to the lack of multi-temporal inventories and joint statistical models, modelling such properties via a unified hazard model has always been challenging and has yet to be attempted. Here, we develop a unified model to estimate landslide hazard at the slope unit level to address such gaps. We employed deep learning, combined with a model motivated by extreme-value theory to analyse an inventory of 30 years of observed rainfall-triggered landslides in Nepal and assess landslide hazard for multiple return periods. We also use our model to further explore landslide hazard for the same return periods under different climate change scenarios up to the end of the century. Our results show that the proposed model performs excellently and can be used to model landslide hazard in a unified manner. Geomorphologically, we find that under both climate change scenarios (SSP245 and SSP885), landslide hazard is likely to increase up to two times on average in the lower Himalayan regions while remaining the same in the middle Himalayan region whilst decreasing slightly in the upper Himalayan region areas.
• 摘要：最常用的滑坡灾害定义结合了有关滑坡位置（敏感性）、威胁（强度）和频率（重现期）的空间信息。在大面积工作时通常只考虑和估计前两个要素。即便如此，单独的模型构成了标准，而频率很少被调查。频率和强度相互交织并相互依赖，因为较大的事件发生的频率较低，反之亦然。然而，由于缺乏多时相清单和联合统计模型，通过统一的危险模型对此类属性进行建模一直具有挑战性，尚未尝试。在这里，我们开发了一个统一的模型来估计斜坡单元层面的滑坡危险，以解决这些差距。我们采用深度学习，结合极值理论驱动的模型，分析了尼泊尔 30 年来观测到的降雨引发滑坡的清单，并评估了多个重现期的滑坡危险。我们还使用我们的模型进一步探讨截至本世纪末不同气候变化情景下相同重现期的滑坡灾害。我们的结果表明，所提出的模型表现出色，可以用于统一地模拟滑坡灾害。从地貌角度来看，我们发现在两种气候变化情景（SSP245和SSP885）下，喜马拉雅山下部地区的滑坡灾害可能平均增加两倍，而喜马拉雅中部地区保持不变，而喜马拉雅上部地区则略有下降地区。

Title: Explicitly Representing Syntax Improves Sentence-to-layout Prediction of Unexpected Situations

• Authors: Wolf Nuyts, Ruben Cartuyvels, Marie-Francine Moens
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.14212
• Pdf URL: https://arxiv.org/pdf/2401.14212
• Copy Paste: [[2401.14212]] Explicitly Representing Syntax Improves Sentence-to-layout Prediction of Unexpected Situations(https://arxiv.org/abs/2401.14212)
• Keywords: code
• Abstract: Recognizing visual entities in a natural language sentence and arranging them in a 2D spatial layout require a compositional understanding of language and space. This task of layout prediction is valuable in text-to-image synthesis as it allows localized and controlled in-painting of the image. In this comparative study it is shown that we can predict layouts from language representations that implicitly or explicitly encode sentence syntax, if the sentences mention similar entity-relationships to the ones seen during training. To test compositional understanding, we collect a test set of grammatically correct sentences and layouts describing compositions of entities and relations that unlikely have been seen during training. Performance on this test set substantially drops, showing that current models rely on correlations in the training data and have difficulties in understanding the structure of the input sentences. We propose a novel structural loss function that better enforces the syntactic structure of the input sentence and show large performance gains in the task of 2D spatial layout prediction conditioned on text. The loss has the potential to be used in other generation tasks where a tree-like structure underlies the conditioning modality. Code, trained models and the USCOCO evaluation set will be made available via github.
• 摘要：识别自然语言句子中的视觉实体并将其排列在 2D 空间布局中需要对语言和空间的组合理解。布局预测的任务在文本到图像的合成中很有价值，因为它允许对图像进行本地化和受控的修复。在这项比较研究中表明，如果句子提及与训练期间看到的类似的实体关系，我们可以根据隐式或显式编码句子语法的语言表示来预测布局。为了测试组合理解，我们收集了一组语法正确的句子和布局的测试集，描述了在训练期间不太可能看到的实体和关系的组合。该测试集的性能大幅下降，表明当前模型依赖于训练数据中的相关性，并且难以理解输入句子的结构。我们提出了一种新颖的结构损失函数，可以更好地强化输入句子的句法结构，并在以文本为条件的 2D 空间布局预测任务中显示出巨大的性能提升。该损失有可能用于其他生成任务，其中树状结构是调节模式的基础。代码、训练模型和 USCOCO 评估集将通过 github 提供。

Title: Commonsense-augmented Memory Construction and Management in Long-term Conversations via Context-aware Persona Refinement

• Authors: Hana Kim, Kai Tzu-iunn Ong, Seoyeon Kim, Dongha Lee, Jinyoung Yeo
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.14215
• Pdf URL: https://arxiv.org/pdf/2401.14215
• Copy Paste: [[2401.14215]] Commonsense-augmented Memory Construction and Management in Long-term Conversations via Context-aware Persona Refinement(https://arxiv.org/abs/2401.14215)
• Keywords: rag
• Abstract: Memorizing and utilizing speakers' personas is a common practice for response generation in long-term conversations. Yet, human-authored datasets often provide uninformative persona sentences that hinder response quality. This paper presents a novel framework that leverages commonsense-based persona expansion to address such issues in long-term conversation. While prior work focuses on not producing personas that contradict others, we focus on transforming contradictory personas into sentences that contain rich speaker information, by refining them based on their contextual backgrounds with designed strategies. As the pioneer of persona expansion in multi-session settings, our framework facilitates better response generation via human-like persona refinement. The supplementary video of our work is available at https://caffeine-15bbf.web.app/.
• 摘要：记住和利用演讲者的角色是长期对话中生成响应的常见做法。然而，人类创作的数据集通常提供信息不丰富的角色句子，从而影响响应质量。本文提出了一种新颖的框架，利用基于常识的角色扩展来解决长期对话中的此类问题。虽然之前的工作重点是不产生与他人相矛盾的角色，但我们的重点是通过根据上下文背景和设计的策略对其进行细化，将矛盾的角色转化为包含丰富说话者信息的句子。作为多会话环境中角色扩展的先驱，我们的框架通过类人角色细化促进更好的响应生成。我们工作的补充视频可在 https://caffeine-15bbf.web.app/ 上获取。

Title: Assessing the Portability of Parameter Matrices Trained by Parameter-Efficient Finetuning Methods

• Authors: Mohammed Sabry, Anya Belz
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.14228
• Pdf URL: https://arxiv.org/pdf/2401.14228
• Copy Paste: [[2401.14228]] Assessing the Portability of Parameter Matrices Trained by Parameter-Efficient Finetuning Methods(https://arxiv.org/abs/2401.14228)
• Keywords: language model, code
• Abstract: As the cost of training ever larger language models has grown, so has the interest in reusing previously learnt knowledge. Transfer learning methods have shown how reusing non-task-specific knowledge can help in subsequent task-specific learning. In this paper, we investigate the inverse: porting whole functional modules that encode task-specific knowledge from one model to another. We designed a study comprising 1,440 training/testing runs to test the portability of modules trained by parameter-efficient finetuning (PEFT) techniques, using sentiment analysis as an example task. We test portability in a wide range of scenarios, involving different PEFT techniques and different pretrained host models, among other dimensions. We compare the performance of ported modules with that of equivalent modules trained (i) from scratch, and (ii) from parameters sampled from the same distribution as the ported module. We find that the ported modules far outperform the two alternatives tested, but that there are interesting performance differences between the four PEFT techniques. We conclude that task-specific knowledge in the form of structurally modular sets of parameters as produced by PEFT techniques is highly portable, but that degree of success depends on type of PEFT and on differences between originating and receiving pretrained models.
• 摘要：随着训练更大的语言模型的成本不断增加，人们对重用以前学到的知识的兴趣也越来越大。迁移学习方法已经表明，重用非特定任务知识可以如何帮助后续的特定任务学习。在本文中，我们研究了相反的情况：将编码特定于任务的知识的整个功能模块从一个模型移植到另一个模型。我们设计了一项包含 1,440 次训练/测试运行的研究，以测试通过参数高效微调 (PEFT) 技术训练的模块的可移植性，并使用情感分析作为示例任务。我们在广泛的场景中测试可移植性，涉及不同的 PEFT 技术和不同的预训练主机模型等维度。我们将移植模块的性能与（i）从头开始训练的等效模块的性能进行比较，（ii）从与移植模块相同的分布中采样的参数进行训练。我们发现移植的模块远远优于测试的两种替代方案，但四种 PEFT 技术之间存在有趣的性能差异。我们得出的结论是，由 PEFT 技术产生的结构模块化参数集形式的特定任务知识具有高度可移植性，但成功程度取决于 PEFT 的类型以及原始和接收预训练模型之间的差异。

Title: Improving Natural Language Capability of Code Large Language Model

• Authors: Wei Li, Daoguang Zan, Bei Guan, Ailun Yu, Xiaolin Chen, Yongji Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.14242
• Pdf URL: https://arxiv.org/pdf/2401.14242
• Copy Paste: [[2401.14242]] Improving Natural Language Capability of Code Large Language Model(https://arxiv.org/abs/2401.14242)
• Keywords: language model, llm, code, rag
• Abstract: Code large language models (Code LLMs) have demonstrated remarkable performance in code generation. Nonetheless, most existing works focus on boosting code LLMs from the perspective of programming capabilities, while their natural language capabilities receive less attention. To fill this gap, we thus propose a novel framework, comprising two modules: AttentionExtractor, which is responsible for extracting key phrases from the user's natural language requirements, and AttentionCoder, which leverages these extracted phrases to generate target code to solve the requirement. This framework pioneers an innovative idea by seamlessly integrating code LLMs with traditional natural language processing tools. To validate the effectiveness of the framework, we craft a new code generation benchmark, called MultiNL-H, covering five natural languages. Extensive experimental results demonstrate the effectiveness of our proposed framework.
• 摘要：代码大语言模型（Code LLM）在代码生成方面表现出了卓越的性能。尽管如此，现有的大多数工作都侧重于从编程能力的角度提升代码LLM，而其自然语言能力受到的关注较少。为了填补这一空白，我们提出了一个新颖的框架，包括两个模块：AttentionExtractor，负责从用户的自然语言需求中提取关键短语；AttentionCoder，利用这些提取的短语生成目标代码来解决需求。该框架通过将代码法学硕士与传统自然语言处理工具无缝集成，开创了一种创新理念。为了验证该框架的有效性，我们制定了一个新的代码生成基准，称为 MultiNL-H，涵盖五种自然语言。大量的实验结果证明了我们提出的框架的有效性。

Title: Transformers and Cortical Waves: Encoders for Pulling In Context Across Time

• Authors: Lyle Muller, Patricia S. Churchland, Terrence J. Sejnowski
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.14267
• Pdf URL: https://arxiv.org/pdf/2401.14267
• Copy Paste: [[2401.14267]] Transformers and Cortical Waves: Encoders for Pulling In Context Across Time(https://arxiv.org/abs/2401.14267)
• Keywords: language model, gpt, llm, code, chat
• Abstract: The capabilities of transformer networks such as ChatGPT and other Large Language Models (LLMs) have captured the world's attention. The crucial computational mechanism underlying their performance relies on transforming a complete input sequence - for example, all the words in a sentence into a long "encoding vector" - that allows transformers to learn long-range temporal dependencies in naturalistic sequences. Specifically, "self-attention" applied to this encoding vector enhances temporal context in transformers by computing associations between pairs of words in the input sequence. We suggest that waves of neural activity, traveling across single cortical regions or across multiple regions at the whole-brain scale, could implement a similar encoding principle. By encapsulating recent input history into a single spatial pattern at each moment in time, cortical waves may enable temporal context to be extracted from sequences of sensory inputs, the same computational principle used in transformers.
• 摘要：ChatGPT 和其他大型语言模型 (LLM) 等 Transformer 网络的功能已经引起了全世界的关注。其性能背后的关键计算机制依赖于将完整的输入序列转换 - 例如，将句子中的所有单词转换为长“编码向量” - 这允许变压器学习自然序列中的远程时间依赖性。具体来说，应用于该编码向量的“自注意力”通过计算输入序列中单词对之间的关​​联来增强变压器中的时间上下文。我们认为，跨越单个皮质区域或全脑多个区域的神经活动波可以实现类似的编码原理。通过将最近的输入历史记录封装到每个时刻的单个空间模式中，皮层波可以从感官输入序列中提取时间上下文，这与 Transformer 中使用的计算原理相同。

Title: RomanSetu: Efficiently unlocking multilingual capabilities of Large Language Models models via Romanization

• Authors: Jaavid Aktar Husain, Raj Dabre, Aswanth Kumar, Ratish Puduppully, Anoop Kunchukuttan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.14280
• Pdf URL: https://arxiv.org/pdf/2401.14280
• Copy Paste: [[2401.14280]] RomanSetu: Efficiently unlocking multilingual capabilities of Large Language Models models via Romanization(https://arxiv.org/abs/2401.14280)
• Keywords: language model, llm, prompt
• Abstract: This study addresses the challenge of extending Large Language Models (LLMs) to non-English languages, specifically those using non-Latin scripts. We propose an innovative approach that utilizes the romanized form of text as an interface for LLMs, hypothesizing that its frequent informal use and shared tokens with English enhance cross-lingual alignment. Focusing on Hindi, we demonstrate through Hindi-to-English translation and sentiment analysis tasks that romanized text not only significantly improves inference efficiency due to its lower fertility compared to native text but also achieves competitive performance with limited pre-training. Additionally, our novel multi-script prompting approach, which combines romanized and native texts, shows promise in further enhancing task performance. These findings suggest the potential of romanization in bridging the language gap for LLM applications, with future work aimed at expanding this approach to more languages and tasks.
• 摘要：这项研究解决了将大型语言模型 (LLM) 扩展到非英语语言，特别是使用非拉丁文字的语言的挑战。我们提出了一种创新方法，利用罗马化文本形式作为法学硕士的界面，假设其频繁的非正式使用以及与英语的共享标记可以增强跨语言对齐。以印地语为重点，我们通过印地语到英语的翻译和情感分析任务证明，罗马化文本不仅由于其与母语文本相比的生育率较低而显着提高了推理效率，而且还通过有限的预训练实现了有竞争力的性能。此外，我们新颖的多脚本提示方法结合了罗马文本和本地文本，有望进一步提高任务绩效。这些发现表明罗马化在弥合法学硕士申请的语言差距方面具有潜力，未来的工作旨在将这种方法扩展到更多语言和任务。

Title: Topologies of Reasoning: Demystifying Chains, Trees, and Graphs of Thoughts

• Authors: Maciej Besta, Florim Memedi, Zhenyu Zhang, Robert Gerstenberger, Nils Blach, Piotr Nyczyk, Marcin Copik, Grzegorz Kwaśniewski, Jürgen Müller, Lukas Gianinazzi, Ales Kubicek, Hubert Niewiadomski, Onur Mutlu, Torsten Hoefler
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.14295
• Pdf URL: https://arxiv.org/pdf/2401.14295
• Copy Paste: [[2401.14295]] Topologies of Reasoning: Demystifying Chains, Trees, and Graphs of Thoughts(https://arxiv.org/abs/2401.14295)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: The field of natural language processing (NLP) has witnessed significant progress in recent years, with a notable focus on improving large language models' (LLM) performance through innovative prompting techniques. Among these, prompt engineering coupled with structures has emerged as a promising paradigm, with designs such as Chain-of-Thought, Tree of Thoughts, or Graph of Thoughts, in which the overall LLM reasoning is guided by a structure such as a graph. As illustrated with numerous examples, this paradigm significantly enhances the LLM's capability to solve numerous tasks, ranging from logical or mathematical reasoning to planning or creative writing. To facilitate the understanding of this growing field and pave the way for future developments, we devise a general blueprint for effective and efficient LLM reasoning schemes. For this, we conduct an in-depth analysis of the prompt execution pipeline, clarifying and clearly defining different concepts. We then build the first taxonomy of structure-enhanced LLM reasoning schemes. We focus on identifying fundamental classes of harnessed structures, and we analyze the representations of these structures, algorithms executed with these structures, and many others. We refer to these structures as reasoning topologies, because their representation becomes to a degree spatial, as they are contained within the LLM context. Our study compares existing prompting schemes using the proposed taxonomy, discussing how certain design choices lead to different patterns in performance and cost. We also outline theoretical underpinnings, relationships between prompting and others parts of the LLM ecosystem such as knowledge bases, and the associated research challenges. Our work will help to advance future prompt engineering techniques.
• 摘要：近年来，自然语言处理 (NLP) 领域取得了重大进展，特别关注通过创新的提示技术提高大型语言模型 (LLM) 的性能。其中，与结构相结合的即时工程已成为一种有前途的范式，其设计包括思想链、思想树或思想图，其中整体法学硕士推理是由图等结构引导的。正如许多例子所示，这种范式显着增强了法学硕士解决众多任务的能力，从逻辑或数学推理到规划或创意写作。为了促进对这个不断发展的领域的理解并为未来的发展铺平道路，我们设计了有效且高效的法学硕士推理方案的总体蓝图。为此，我们对即时执行管道进行了深入分析，澄清并明确定义了不同的概念。然后，我们构建了结构增强型 LLM 推理方案的第一个分类法。我们专注于识别所利用结构的基本类别，并分析这些结构的表示、使用这些结构执行的算法以及许多其他结构。我们将这些结构称为推理拓扑，因为它们的表示在一定程度上变得空间化，因为它们包含在法学硕士背景中。我们的研究使用提议的分类法比较了现有的提示方案，讨论了某些设计选择如何导致不同的性能和成本模式。我们还概述了理论基础、提示与法学硕士生态系统其他部分（例如知识库）之间的关系以及相关的研究挑战。我们的工作将有助于推进未来的快速工程技术。

Title: ServerlessLLM: Locality-Enhanced Serverless Inference for Large Language Models

• Authors: Yao Fu, Leyang Xue, Yeqi Huang, Andrei-Octavian Brabete, Dmitrii Ustiugov, Yuvraj Patel, Luo Mai
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2401.14351
• Pdf URL: https://arxiv.org/pdf/2401.14351
• Copy Paste: [[2401.14351]] ServerlessLLM: Locality-Enhanced Serverless Inference for Large Language Models(https://arxiv.org/abs/2401.14351)
• Keywords: language model, llm, rag
• Abstract: This paper presents ServerlessLLM, a locality-enhanced serverless inference system for Large Language Models (LLMs). ServerlessLLM exploits the substantial capacity and bandwidth of storage and memory devices available on GPU servers, thereby reducing costly remote checkpoint downloads and achieving efficient checkpoint loading. ServerlessLLM achieves this through three main contributions: (i) fast LLM checkpoint loading via a novel loading-optimized checkpoint format design, coupled with an efficient multi-tier checkpoint loading system; (ii) locality-driven LLM inference with live migration, which allows ServerlessLLM to effectively achieve locality-driven server allocation while preserving the low latency of ongoing LLM inference; and (iii) locality-aware server allocation, enabling ServerlessLLM to evaluate the status of each server in a cluster and effectively schedule model startup time to capitalize on local checkpoint placement. Our comprehensive experiments, which include microbenchmarks and real-world traces, show that ServerlessLLM surpasses state-of-the-art systems by 10 - 200X in latency performance when running various LLM inference workloads.
• 摘要：本文介绍了 ServerlessLLM，这是一种用于大型语言模型 (LLM) 的局部增强型无服务器推理系统。 ServerlessLLM 利用 GPU 服务器上可用的存储和内存设备的大量容量和带宽，从而减少昂贵的远程检查点下载并实现高效的检查点加载。 ServerlessLLM 通过三个主要贡献实现了这一目标：(i) 通过新颖的加载优化检查点格式设计以及高效的多层检查点加载系统来快速加载 LLM 检查点； (ii) 具有实时迁移功能的局部驱动的 LLM 推理，这使得 ServerlessLLM 能够有效地实现局部驱动的服务器分配，同时保持正在进行的 LLM 推理的低延迟； (iii) 位置感知服务器分配，使 ServerlessLLM 能够评估集群中每个服务器的状态，并有效地安排模型启动时间，以利用本地检查点放置。我们的综合实验（包括微基准测试和真实世界跟踪）表明，在运行各种 LLM 推理工作负载时，ServerlessLLM 的延迟性能比最先进的系统高出 10 - 200 倍。

Title: A Comparative Analysis of Noise Reduction Methods in Sentiment Analysis on Noisy Bengali Texts

• Authors: Kazi Toufique Elahi, Tasnuva Binte Rahman, Shakil Shahriar, Samir Sarker, Md. Tanvir Rouf Shawon, G. M. Shahariar
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.14360
• Pdf URL: https://arxiv.org/pdf/2401.14360
• Copy Paste: [[2401.14360]] A Comparative Analysis of Noise Reduction Methods in Sentiment Analysis on Noisy Bengali Texts(https://arxiv.org/abs/2401.14360)
• Keywords: lora
• Abstract: While Bengali is considered a language with limited resources, sentiment analysis has been a subject of extensive research in the literature. Nevertheless, there is a scarcity of exploration into sentiment analysis specifically in the realm of noisy Bengali texts. In this paper, we introduce a dataset (NC-SentNoB) that we annotated manually to identify ten different types of noise found in a pre-existing sentiment analysis dataset comprising of around 15K noisy Bengali texts. At first, given an input noisy text, we identify the noise type, addressing this as a multi-label classification task. Then, we introduce baseline noise reduction methods to alleviate noise prior to conducting sentiment analysis. Finally, we assess the performance of fine-tuned sentiment analysis models with both noisy and noise-reduced texts to make comparisons. The experimental findings indicate that the noise reduction methods utilized are not satisfactory, highlighting the need for more suitable noise reduction methods in future research endeavors. We have made the implementation and dataset presented in this paper publicly available at https://github.com/ktoufiquee/A-Comparative-Analysis-of-Noise-Reduction-Methods-in-Sentiment-Analysis-on-Noisy-Bengali-Texts
• 摘要：虽然孟加拉语被认为是一种资源有限的语言，但情感分析一直是文献中广泛研究的主题。然而，对情感分析的探索很少，特别是在嘈杂的孟加拉语文本领域。在本文中，我们介绍了一个数据集 (NC-SentNoB)，我们手动注释了该数据集，以识别在包含大约 15K 噪声孟加拉文本的现有情感分析数据集中发现的十种不同类型的噪声。首先，给定输入噪声文本，我们识别噪声类型，将其作为多标签分类任务来解决。然后，我们在进行情感分析之前引入基线降噪方法来减轻噪音。最后，我们使用噪声文本和降噪文本评估微调情感分析模型的性能以进行比较。实验结果表明，所采用的降噪方法并不令人满意，突出表明在未来的研究工作中需要更合适的降噪方法。我们已将本文中提出的实现和数据集公开发布于 https://github.com/ktoufiquee/A-Comparative-Analysis-of-Noise-Reduction-Methods-in-Sentiment-Analysis-on-Noisy-Bengali-文本

Title: MoE-Infinity: Activation-Aware Expert Offloading for Efficient MoE Serving

• Authors: Leyang Xue, Yao Fu, Zhan Lu, Luo Mai, Mahesh Marina
• Subjects: cs.LG, cs.PF
• Abstract URL: https://arxiv.org/abs/2401.14361
• Pdf URL: https://arxiv.org/pdf/2401.14361
• Copy Paste: [[2401.14361]] MoE-Infinity: Activation-Aware Expert Offloading for Efficient MoE Serving(https://arxiv.org/abs/2401.14361)
• Keywords: code
• Abstract: This paper presents MoE-Infinity, a cost-efficient mixture-of-expert (MoE) serving system that realizes activation-aware expert offloading. MoE-Infinity features sequence-level expert activation tracing, a new approach adept at identifying sparse activations and capturing the temporal locality of MoE inference. By analyzing these traces, MoE-Infinity performs novel activation-aware expert prefetching and caching, substantially reducing the latency overheads usually associated with offloading experts for improved cost performance. Extensive experiments in a cluster show that MoE-Infinity outperforms numerous existing systems and approaches, reducing latency by 4 - 20X and decreasing deployment costs by over 8X for various MoEs. MoE-Infinity's source code is publicly available at https://github.com/TorchMoE/MoE-Infinity
• 摘要：本文介绍了 MoE-Infinity，这是一种经济高效的专家混合 (MoE) 服务系统，可实现激活感知专家卸载。 MoE-Infinity 具有序列级专家激活跟踪功能，这是一种擅长识别稀疏激活和捕获 MoE 推理的时间局部性的新方法。通过分析这些跟踪，MoE-Infinity 执行新颖的激活感知专家预取和缓存，从而大大减少通常与卸载专家相关的延迟开销，从而提高成本性能。集群中的大量实验表明，MoE-Infinity 的性能优于众多现有系统和方法，将各种 MoE 的延迟减少了 4 - 20 倍，并将部署成本降低了 8 倍以上。 MoE-Infinity 的源代码可在 https://github.com/TorchMoE/MoE-Infinity 上公开获取

Title: TURNA: A Turkish Encoder-Decoder Language Model for Enhanced Understanding and Generation

• Authors: Gökçe Uludoğan, Zeynep Yirmibeşoğlu Balal, Furkan Akkurt, Melikşah Türker, Onur Güngör, Susan Üsküdarlı
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.14373
• Pdf URL: https://arxiv.org/pdf/2401.14373
• Copy Paste: [[2401.14373]] TURNA: A Turkish Encoder-Decoder Language Model for Enhanced Understanding and Generation(https://arxiv.org/abs/2401.14373)
• Keywords: language model, code
• Abstract: The recent advances in natural language processing have predominantly favored well-resourced English-centric models, resulting in a significant gap with low-resource languages. In this work, we introduce the language model TURNA, which is developed for the low-resource language Turkish and is capable of both natural language understanding and generation tasks. TURNA is pretrained with an encoder-decoder architecture based on the unified framework UL2 with a diverse corpus that we specifically curated for this purpose. We evaluated TURNA with three generation tasks and five understanding tasks for Turkish. The results show that TURNA outperforms several multilingual models in both understanding and generation tasks, and competes with monolingual Turkish models in understanding tasks. TURNA is made available at https://huggingface.co/boun-tabi-LMG/TURNA .
• 摘要：自然语言处理的最新进展主要有利于资源丰富的以英语为中心的模型，导致与资源匮乏的语言存在显着差距。在这项工作中，我们介绍了语言模型 TURNA，它是为低资源语言土耳其语开发的，能够完成自然语言理解和生成任务。 TURNA 使用基于统一框架 UL2 的编码器-解码器架构进行了预训练，其中包含我们为此专门策划的多样化语料库。我们使用土耳其语的三个生成任务和五个理解任务来评估 TURNA。结果表明，TURNA 在理解和生成任务中均优于多种多语言模型，并在理解任务中与单语土耳其语模型竞争。 TURNA 可在 https://huggingface.co/boun-tabi-LMG/TURNA 上获取。

Title: Modular Adaptation of Multilingual Encoders to Written Swiss German Dialect

• Authors: Jannis Vamvas, Noëmi Aepli, Rico Sennrich
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.14400
• Pdf URL: https://arxiv.org/pdf/2401.14400
• Copy Paste: [[2401.14400]] Modular Adaptation of Multilingual Encoders to Written Swiss German Dialect(https://arxiv.org/abs/2401.14400)
• Keywords: code
• Abstract: Creating neural text encoders for written Swiss German is challenging due to a dearth of training data combined with dialectal variation. In this paper, we build on several existing multilingual encoders and adapt them to Swiss German using continued pre-training. Evaluation on three diverse downstream tasks shows that simply adding a Swiss German adapter to a modular encoder achieves 97.5% of fully monolithic adaptation performance. We further find that for the task of retrieving Swiss German sentences given Standard German queries, adapting a character-level model is more effective than the other adaptation strategies. We release our code and the models trained for our experiments at https://github.com/ZurichNLP/swiss-german-text-encoders
• 摘要：由于缺乏训练数据以及方言变化，为书面瑞士德语创建神经文本编码器具有挑战性。在本文中，我们以几种现有的多语言编码器为基础，并通过持续的预训练将其适应瑞士德语。对三个不同下游任务的评估表明，只需将瑞士德国适配器添加到模块化编码器即可实现 97.5% 的完全单片适配性能。我们进一步发现，对于在给定标准德语查询的情况下检索瑞士德语句子的任务，采用字符级模型比其他适应策略更有效。我们在 https://github.com/ZurichNLP/swiss-german-text-encoders 发布了我们的代码和为实验训练的模型