2024-01-12

secure

Title: Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training. (arXiv:2401.05566v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2401.05566
• Code URL: https://github.com/anthropics/sleeper-agents-paper
• Copy Paste: [[2401.05566]] Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training(http://arxiv.org/abs/2401.05566)
• Summary:

  Humans are capable of strategically deceptive behavior: behaving helpfully in most situations, but then behaving very differently in order to pursue alternative objectives when given the opportunity. If an AI system learned such a deceptive strategy, could we detect it and remove it using current state-of-the-art safety training techniques? To study this question, we construct proof-of-concept examples of deceptive behavior in large language models (LLMs). For example, we train models that write secure code when the prompt states that the year is 2023, but insert exploitable code when the stated year is 2024. We find that such backdoored behavior can be made persistent, so that it is not removed by standard safety training techniques, including supervised fine-tuning, reinforcement learning, and adversarial training (eliciting unsafe behavior and then training to remove it). The backdoored behavior is most persistent in the largest models and in models trained to produce chain-of-thought reasoning about deceiving the training process, with the persistence remaining even when the chain-of-thought is distilled away. Furthermore, rather than removing backdoors, we find that adversarial training can teach models to better recognize their backdoor triggers, effectively hiding the unsafe behavior. Our results suggest that, once a model exhibits deceptive behavior, standard techniques could fail to remove such deception and create a false impression of safety.

• 摘要：

  人类能够进行战略性欺骗行为：在大多数情况下表现得有帮助，但随后表现得非常不同，以便在有机会时追求替代目标。如果人工智能系统学会了这种欺骗策略，我们是否可以使用当前最先进的安全训练技术来检测并删除它？为了研究这个问题，我们构建了大型语言模型（LLM）中欺骗行为的概念验证示例。例如，我们训练模型在提示指出年份是 2023 年时编写安全代码，但在提示年份是 2024 年时插入可利用代码。我们发现这种后门行为可以持久化，这样就不会被标准删除安全培训技术，包括监督微调、强化学习和对抗性训练（引发不安全行为，然后进行培训以消除它）。后门行为在最大的模型和经过训练以产生欺骗训练过程的思维链推理的模型中最为持久，即使思维链被蒸馏掉，这种持久性仍然存在。此外，我们发现对抗性训练不是删除后门，而是可以教会模型更好地识别其后门触发器，从而有效隐藏不安全行为。我们的结果表明，一旦模型表现出欺骗行为，标准技术可能无法消除这种欺骗并造成安全的错误印象。

Title: Optimized Ensemble Model Towards Secured Industrial IoT Devices. (arXiv:2401.05509v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2401.05509
• Code URL: null
• Copy Paste: [[2401.05509]] Optimized Ensemble Model Towards Secured Industrial IoT Devices(http://arxiv.org/abs/2401.05509)
• Summary:

  The continued growth in the deployment of Internet-of-Things (IoT) devices has been fueled by the increased connectivity demand, particularly in industrial environments. However, this has led to an increase in the number of network related attacks due to the increased number of potential attack surfaces. Industrial IoT (IIoT) devices are prone to various network related attacks that can have severe consequences on the manufacturing process as well as on the safety of the workers in the manufacturing plant. One promising solution that has emerged in recent years for attack detection is Machine learning (ML). More specifically, ensemble learning models have shown great promise in improving the performance of the underlying ML models. Accordingly, this paper proposes a framework based on the combined use of Bayesian Optimization-Gaussian Process (BO-GP) with an ensemble tree-based learning model to improve the performance of intrusion and attack detection in IIoT environments. The proposed framework's performance is evaluated using the Windows 10 dataset collected by the Cyber Range and IoT labs at University of New South Wales. Experimental results illustrate the improvement in detection accuracy, precision, and F-score when compared to standard tree and ensemble tree models.

• 摘要：

  连接需求的增加推动了物联网 (IoT) 设备部署的持续增长，尤其是在工业环境中。然而，由于潜在攻击面数量的增加，这导致了网络相关攻击数量的增加。工业物联网 (IIoT) 设备容易受到各种网络相关攻击，可能对制造过程以及制造工厂工人的安全造成严重后果。近年来出现的一种有前景的攻击检测解决方案是机器学习 (ML)。更具体地说，集成学习模型在提高底层机器学习模型的性能方面显示出了巨大的希望。因此，本文提出了一种基于贝叶斯优化-高斯过程（BO-GP）与基于集成树的学习模型相结合的框架，以提高工业物联网环境中入侵和攻击检测的性能。使用新南威尔士大学 Cyber​​ Range 和 IoT 实验室收集的 Windows 10 数据集来评估所提出的框架的性能。实验结果表明，与标准树和集成树模型相比，检测准确度、精度和 F 分数有所提高。

Title: STAKESURE: Proof of Stake Mechanisms with Strong Cryptoeconomic Safety. (arXiv:2401.05797v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2401.05797
• Code URL: null
• Copy Paste: [[2401.05797]] STAKESURE: Proof of Stake Mechanisms with Strong Cryptoeconomic Safety(http://arxiv.org/abs/2401.05797)
• Summary:

  As of July 15, 2023, Ethererum, which is a Proof-of-Stake (PoS) blockchain [1] has around 410 Billion USD in total assets on chain (popularly referred to as total-value-locked, TVL) but has only 33 Billion USD worth of ETH staked in securing the underlying consensus of the chain [2]. A preliminary analysis might suggest that as the amount staked is far less (11x less) than the value secured, the Ethereum blockchain is insecure and "over-leveraged" in a purely cryptoeconomic sense. In this work, we investigate how Ethereum, or, more generally, any PoS blockchain can be made secure despite this apparent imbalance. Towards that end, we attempt to formalize a model for analyzing the cryptoeconomic safety of PoS blockchain, which separately analyzes the cost-of-corruption, the cost incurred by an attacker, and the profit-from-corruption, the profit gained by an attacker. We derive sharper bounds on profit-from-corruption, as well as new confirmation rules that significantly decrease this upper-bound. We evaluate cost-of-corruption and profit-from-corruption only from the perspective of attacking safety. Finally, we present a new "insurance" mechanism, STAKESURE, for allocating the slashed funds in a PoS system, that has several highly desirable properties: solving common information problem in existing blockchains, creating a mechanism for provably safe bridging, and providing the first sharp solution for automatically adjusting how much economic security is sufficient in a PoS system. Finally, we show that the system satisfies a notion of strong cryptoeconomic safety, which guarantees that no honest transactor ever loses money, and creates a closed system of Karma, which not only ensures that the attacker suffers a loss of funds but also that the harmed parties are sufficiently compensated.

• 摘要：

  截至 2023 年 7 月 15 日，以太坊（PoS）区块链 [1] 链上总资产约为 4100 亿美元（通常称为总价值锁定，TVL）但只有价值 330 亿美元的 ETH 用于确保链的基本共识 [2]。初步分析可能表明，由于质押金额远低于所担保的价值（少 11 倍），因此从纯粹的加密经济学意义上来说，以太坊区块链是不安全且“过度杠杆化”的。在这项工作中，我们研究了如何在存在明显不平衡的情况下确保以太坊，或者更一般地说，任何 PoS 区块链的安全。为此，我们尝试建立一个模型来分析 PoS 区块链的加密经济安全性，该模型分别分析腐败成本（攻击者产生的成本）和腐败利润（攻击者获得的利润） 。我们对腐败利润制定了更严格的界限，以及显着降低这一上限的新确认规则。我们仅从攻击安全的角度来评估腐败成本和腐败利润。最后，我们提出了一种新的“保险”机制 STAKESURE，用于在 PoS 系统中分配削减的资金，该机制具有几个非常理想的特性：解决现有区块链中的常见信息问题，创建可证明安全的桥接机制，并提供第一个自动调整 PoS 系统中足够的经济安全性的尖锐解决方案。最后，我们证明该系统满足强大的加密经济安全概念，保证任何诚实的交易者都不会损失金钱，并创建了一个封闭的 Karma 系统，不仅确保攻击者遭受资金损失，而且还确保受到伤害的人不会遭受损失。当事人得到充分补偿。

Title: Blockchain-based Decentralized Time Lock Machines: Automated Reveal of Time-sensitive Information. (arXiv:2401.05947v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2401.05947
• Code URL: null
• Copy Paste: [[2401.05947]] Blockchain-based Decentralized Time Lock Machines: Automated Reveal of Time-sensitive Information(http://arxiv.org/abs/2401.05947)
• Summary:

  Conditional Information Reveal (CIR) automates the release of information upon meeting specific pre-defined conditions, such as time or location. This paper advances the understanding and implementation of CIR by introducing a new paradigm to highlight the security challenges in CIR design, and proposes a decentralized architecture as a design guideline for secure CIR systems. Furthermore, in the context of time-sensitive data sharing, this paper proposes a practical timed-release cryptography system employing the proposed architecture and a novel verifiable secret sharing scheme. Key achievements of this study include the creation of an open-source prototype for practical deployment and a comprehensive system evaluation that highlights the enhanced security and efficiency of the proposed system. Furthermore, the paper delves into the application of this system in E-voting scenarios, illustrating its capacity to secure and ensure fair electronic voting processes.

• 摘要：

  条件信息披露 (CIR) 可在满足特定的预定义条件（例如时间或位置）时自动发布信息。本文通过引入新的范式来强调 CIR 设计中的安全挑战，促进了对 CIR 的理解和实现，并提出了一种去中心化架构作为安全 CIR 系统的设计指南。此外，在时间敏感数据共享的背景下，本文提出了一种采用所提出的架构的实用的定时发布密码系统和一种新颖的可验证秘密共享方案。这项研究的主要成果包括创建了一个用于实际部署的开源原型以及一个全面的系统评估，强调了所提议系统的增强的安全性和效率。此外，本文还深入研究了该系统在电子投票场景中的应用，展示了其保护和确保公平电子投票过程的能力。

security

Title: Face-GPS: A Comprehensive Technique for Quantifying Facial Muscle Dynamics in Videos. (arXiv:2401.05625v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.05625
• Code URL: null
• Copy Paste: [[2401.05625]] Face-GPS: A Comprehensive Technique for Quantifying Facial Muscle Dynamics in Videos(http://arxiv.org/abs/2401.05625)
• Summary:

  We introduce a novel method that combines differential geometry, kernels smoothing, and spectral analysis to quantify facial muscle activity from widely accessible video recordings, such as those captured on personal smartphones. Our approach emphasizes practicality and accessibility. It has significant potential for applications in national security and plastic surgery. Additionally, it offers remote diagnosis and monitoring for medical conditions such as stroke, Bell's palsy, and acoustic neuroma. Moreover, it is adept at detecting and classifying emotions, from the overt to the subtle. The proposed face muscle analysis technique is an explainable alternative to deep learning methods and a non-invasive substitute to facial electromyography (fEMG).

• 摘要：

  我们引入了一种新颖的方法，该方法结合了微分几何、核平滑和光谱分析，可以从广泛访问的视频记录（例如个人智能手机上捕获的视频记录）中量化面部肌肉活动。我们的方法强调实用性和可访问性。它在国家安全和整形外科方面具有巨大的应用潜力。此外，它还提供中风、贝尔麻痹和听神经瘤等医疗状况的远程诊断和监测。此外，它还擅长检测和分类情绪，从明显的情绪到微妙的情绪。所提出的面部肌肉分析技术是深度学习方法的可解释替代方法，也是面部肌电图（fEMG）的非侵入性替代方法。

Title: The Role of Deep Learning in Advancing Proactive Cybersecurity Measures for Smart Grid Networks: A Survey. (arXiv:2401.05896v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2401.05896
• Code URL: null
• Copy Paste: [[2401.05896]] The Role of Deep Learning in Advancing Proactive Cybersecurity Measures for Smart Grid Networks: A Survey(http://arxiv.org/abs/2401.05896)
• Summary:

  As smart grids (SG) increasingly rely on advanced technologies like sensors and communication systems for efficient energy generation, distribution, and consumption, they become enticing targets for sophisticated cyberattacks. These evolving threats demand robust security measures to maintain the stability and resilience of modern energy systems. While extensive research has been conducted, a comprehensive exploration of proactive cyber defense strategies utilizing Deep Learning (DL) in {SG} remains scarce in the literature. This survey bridges this gap, studying the latest DL techniques for proactive cyber defense. The survey begins with an overview of related works and our distinct contributions, followed by an examination of SG infrastructure. Next, we classify various cyber defense techniques into reactive and proactive categories. A significant focus is placed on DL-enabled proactive defenses, where we provide a comprehensive taxonomy of DL approaches, highlighting their roles and relevance in the proactive security of SG. Subsequently, we analyze the most significant DL-based methods currently in use. Further, we explore Moving Target Defense, a proactive defense strategy, and its interactions with DL methodologies. We then provide an overview of benchmark datasets used in this domain to substantiate the discourse.{ This is followed by a critical discussion on their practical implications and broader impact on cybersecurity in Smart Grids.} The survey finally lists the challenges associated with deploying DL-based security systems within SG, followed by an outlook on future developments in this key field.

• 摘要：

  随着智能电网 (SG) 越来越依赖传感器和通信系统等先进技术来实现高效的能源生成、分配和消耗，它们成为复杂网络攻击的诱人目标。这些不断演变的威胁需要强有力的安全措施来维持现代能源系统的稳定性和弹性。尽管已经进行了广泛的研究，但在文献中仍然很少对利用深度学习 (DL) 的主动网络防御策略进行全面探索。这项调查弥补了这一差距，研究了用于主动网络防御的最新深度学习技术。该调查首先概述相关工作和我们的独特贡献，然后检查 SG 基础设施。接下来，我们将各种网络防御技术分为被动型和主动型两类。我们重点关注基于深度学习的主动防御，我们提供了深度学习方法的全面分类，强调了它们在 SG 主动安全中的作用和相关性。随后，我们分析了当前使用的最重要的基于深度学习的方法。此外，我们还探讨了移动目标防御（一种主动防御策略）及其与深度学习方法的交互。然后，我们概述了该领域使用的基准数据集，以证实这一讨论。{随后对其实际影响以及对智能电网网络安全的更广泛影响进行了批判性讨论。}该调查最终列出了与部署 DL 相关的挑战SG 内部的安全系统，其次是对这一关键领域未来发展的展望。

Title: Securing an Application Layer Gateway: An Industrial Case Study. (arXiv:2401.05961v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2401.05961
• Code URL: null
• Copy Paste: [[2401.05961]] Securing an Application Layer Gateway: An Industrial Case Study(http://arxiv.org/abs/2401.05961)
• Summary:

  Application Layer Gateways (ALGs) play a crucial role in securing critical systems, including railways, industrial automation, and defense applications, by segmenting networks at different levels of criticality. However, they require rigorous security testing to prevent software vulnerabilities, not only at the network level but also at the application layer (e.g., deep traffic inspection components). This paper presents a vulnerability-driven methodology for the comprehensive security testing of ALGs. We present the methodology in the context of an industrial case study in the railways domain, and a simulation-based testing environment to support the methodology.

• 摘要：

  应用层网关 (ALG) 通过将网络划分为不同的关键级别，在保护关键系统（包括铁路、工业自动化和国防应用）方面发挥着至关重要的作用。然而，它们需要严格的安全测试来防止软件漏洞，不仅在网络级别，而且在应用程序层（例如深度流量检查组件）。本文提出了一种用于 ALG 综合安全测试的漏洞驱动方法。我们在铁路领域的工业案例研究背景下介绍该方法，并提供基于模拟的测试环境来支持该方法。

privacy

Title: Developing a Resource-Constraint EdgeAI model for Surface Defect Detection. (arXiv:2401.05355v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.05355
• Code URL: null
• Copy Paste: [[2401.05355]] Developing a Resource-Constraint EdgeAI model for Surface Defect Detection(http://arxiv.org/abs/2401.05355)
• Summary:

  Resource constraints have restricted several EdgeAI applications to machine learning inference approaches, where models are trained on the cloud and deployed to the edge device. This poses challenges such as bandwidth, latency, and privacy associated with storing data off-site for model building. Training on the edge device can overcome these challenges by eliminating the need to transfer data to another device for storage and model development. On-device training also provides robustness to data variations as models can be retrained on newly acquired data to improve performance. We, therefore, propose a lightweight EdgeAI architecture modified from Xception, for on-device training in a resource-constraint edge environment. We evaluate our model on a PCB defect detection task and compare its performance against existing lightweight models - MobileNetV2, EfficientNetV2B0, and MobileViT-XXS. The results of our experiment show that our model has a remarkable performance with a test accuracy of 73.45% without pre-training. This is comparable to the test accuracy of non-pre-trained MobileViT-XXS (75.40%) and much better than other non-pre-trained models (MobileNetV2 - 50.05%, EfficientNetV2B0 - 54.30%). The test accuracy of our model without pre-training is comparable to pre-trained MobileNetV2 model - 75.45% and better than pre-trained EfficientNetV2B0 model - 58.10%. In terms of memory efficiency, our model performs better than EfficientNetV2B0 and MobileViT-XXS. We find that the resource efficiency of machine learning models does not solely depend on the number of parameters but also depends on architectural considerations. Our method can be applied to other resource-constraint applications while maintaining significant performance.

• 摘要：

  资源限制限制了多个 EdgeAI 应用程序只能使用机器学习推理方法，其中模型在云端进行训练并部署到边缘设备。这带来了与异地存储模型构建数据相关的带宽、延迟和隐私等挑战。在边缘设备上进行训练可以克服这些挑战，因为无需将数据传输到另一台设备进行存储和模型开发。设备上训练还提供了对数据变化的鲁棒性，因为可以根据新获取的数据重新训练模型以提高性能。因此，我们提出了一种从 Xception 修改而来的轻量级 EdgeAI 架构，用于资源受限的边缘环境中的设备上训练。我们在 PCB 缺陷检测任务上评估我们的模型，并将其性能与现有的轻量级模型（MobileNetV2、EfficientNetV2B0 和 MobileViT-XXS）进行比较。我们的实验结果表明，我们的模型具有显着的性能，在没有预训练的情况下测试准确率为 73.45%。这与非预训练 MobileViT-XXS (75.40%) 的测试精度相当，并且远优于其他非预训练模型 (MobileNetV2 - 50.05%、EfficientNetV2B0 - 54.30%)。我们的模型在没有预训练的情况下的测试精度与预训练的 MobileNetV2 模型相当 - 75.45％，并且优于预训练的 EfficientNetV2B0 模型 - 58.10％。在内存效率方面，我们的模型表现优于 EfficientNetV2B0 和 MobileViT-XXS。我们发现机器学习模型的资源效率不仅仅取决于参数的数量，还取决于架构方面的考虑。我们的方法可以应用于其他资源受限的应用程序，同时保持显着的性能。

Title: Inferring Intentions to Speak Using Accelerometer Data In-the-Wild. (arXiv:2401.05849v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.05849
• Code URL: null
• Copy Paste: [[2401.05849]] Inferring Intentions to Speak Using Accelerometer Data In-the-Wild(http://arxiv.org/abs/2401.05849)
• Summary:

  Humans have good natural intuition to recognize when another person has something to say. It would be interesting if an AI can also recognize intentions to speak. Especially in scenarios when an AI is guiding a group discussion, this can be a useful skill. This work studies the inference of successful and unsuccessful intentions to speak from accelerometer data. This is chosen because it is privacy-preserving and feasible for in-the-wild settings since it can be placed in a smart badge. Data from a real-life social networking event is used to train a machine-learning model that aims to infer intentions to speak. A subset of unsuccessful intention-to-speak cases in the data is annotated. The model is trained on the successful intentions to speak and evaluated on both the successful and unsuccessful cases. In conclusion, there is useful information in accelerometer data, but not enough to reliably capture intentions to speak. For example, posture shifts are correlated with intentions to speak, but people also often shift posture without having an intention to speak, or have an intention to speak without shifting their posture. More modalities are likely needed to reliably infer intentions to speak.

• 摘要：

  人类具有良好的自然直觉，可以识别另一个人何时有话要说。如果人工智能也能识别说话的意图，那就很有趣了。特别是在人工智能引导小组讨论的情况下，这可能是一项有用的技能。这项工作研究了根据加速度计数据推断成功和不成功的说话意图。选择这种方式是因为它可以保护隐私，并且可以放置在智能徽章中，因此适合野外环境。来自现实生活中的社交网络活动的数据用于训练机器学习模型，旨在推断说话的意图。数据中不成功的意向发言案例的子集被注释。该模型根据成功的说话意图进行训练，并根据成功和不成功的案例进行评估。总之，加速度计数据中有有用的信息，但不足以可靠地捕获说话的意图。例如，姿势的变化与说话的意图相关，但人们也常常在没有说话的意图的情况下改变姿势，或者有说话的意图但没有改变姿势。可能需要更多的方式来可靠地推断说话的意图。

Title: Brave: Byzantine-Resilient and Privacy-Preserving Peer-to-Peer Federated Learning. (arXiv:2401.05562v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.05562
• Code URL: null
• Copy Paste: [[2401.05562]] Brave: Byzantine-Resilient and Privacy-Preserving Peer-to-Peer Federated Learning(http://arxiv.org/abs/2401.05562)
• Summary:

  Federated learning (FL) enables multiple participants to train a global machine learning model without sharing their private training data. Peer-to-peer (P2P) FL advances existing centralized FL paradigms by eliminating the server that aggregates local models from participants and then updates the global model. However, P2P FL is vulnerable to (i) honest-but-curious participants whose objective is to infer private training data of other participants, and (ii) Byzantine participants who can transmit arbitrarily manipulated local models to corrupt the learning process. P2P FL schemes that simultaneously guarantee Byzantine resilience and preserve privacy have been less studied. In this paper, we develop Brave, a protocol that ensures Byzantine Resilience And privacy-preserving property for P2P FL in the presence of both types of adversaries. We show that Brave preserves privacy by establishing that any honest-but-curious adversary cannot infer other participants' private data by observing their models. We further prove that Brave is Byzantine-resilient, which guarantees that all benign participants converge to an identical model that deviates from a global model trained without Byzantine adversaries by a bounded distance. We evaluate Brave against three state-of-the-art adversaries on a P2P FL for image classification tasks on benchmark datasets CIFAR10 and MNIST. Our results show that the global model learned with Brave in the presence of adversaries achieves comparable classification accuracy to a global model trained in the absence of any adversary.

• 摘要：

  联邦学习 (FL) 使多个参与者能够训练全局机器学习模型，而无需共享其私有训练数据。点对点 (P2P) FL 通过消除从参与者聚合本地模型然后更新全局模型的服务器来改进现有的集中式 FL 范例。然而，P2P FL 很容易受到 (i) 诚实但好奇的参与者的攻击，其目标是推断其他参与者的私人训练数据，以及 (ii) 拜占庭参与者，他们可以传输任意操纵的本地模型来破坏学习过程。同时保证拜占庭弹性和保护隐私的 P2P FL 方案研究较少。在本文中，我们开发了 Brave，这是一种在两种类型的对手都存在的情况下确保 P2P FL 的拜占庭弹性和隐私保护属性的协议。我们证明，Brave 通过确定任何诚实但好奇的对手无法通过观察其他参与者的模型来推断他们的私人数据来保护隐私。我们进一步证明 Brave 具有拜占庭弹性，这保证了所有良性参与者都会收敛到一个相同的模型，该模型与没有拜占庭对手训练的全局模型有一定距离的偏差。我们在 P2P FL 上针对基准数据集 CIFAR10 和 MNIST 上的图像分类任务对 Brave 与三个最先进的对手进行了评估。我们的结果表明，在有对手存在的情况下使用 Brave 学习的全局模型所达到的分类精度与在没有任何对手的情况下训练的全局模型相当。

protect

Title: Manipulating Feature Visualizations with Gradient Slingshots. (arXiv:2401.06122v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.06122
• Code URL: https://github.com/dilyabareeva/grad-slingshot
• Copy Paste: [[2401.06122]] Manipulating Feature Visualizations with Gradient Slingshots(http://arxiv.org/abs/2401.06122)
• Summary:

  Deep Neural Networks (DNNs) are capable of learning complex and versatile representations, however, the semantic nature of the learned concepts remains unknown. A common method used to explain the concepts learned by DNNs is Activation Maximization (AM), which generates a synthetic input signal that maximally activates a particular neuron in the network. In this paper, we investigate the vulnerability of this approach to adversarial model manipulations and introduce a novel method for manipulating feature visualization without altering the model architecture or significantly impacting the model's decision-making process. We evaluate the effectiveness of our method on several neural network models and demonstrate its capabilities to hide the functionality of specific neurons by masking the original explanations of neurons with chosen target explanations during model auditing. As a remedy, we propose a protective measure against such manipulations and provide quantitative evidence which substantiates our findings.

• 摘要：

  深度神经网络 (DNN) 能够学习复杂且通用的表示，但是，所学习概念的语义性质仍然未知。用于解释 DNN 学习的概念的常用方法是激活最大化 (AM)，它生成一个合成输入信号，最大限度地激活网络中的特定神经元。在本文中，我们研究了这种方法对对抗性模型操作的脆弱性，并介绍了一种在不改变模型架构或显着影响模型决策过程的情况下操作特征可视化的新方法。我们评估了我们的方法在多个神经网络模型上的有效性，并通过在模型审核期间用选定的目标解释掩盖神经元的原始解释来展示其隐藏特定神经元功能的能力。作为补救措施，我们提出了针对此类操纵的保护措施，并提供了证实我们发现的定量证据。

Title: TOFU: A Task of Fictitious Unlearning for LLMs. (arXiv:2401.06121v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.06121
• Code URL: null
• Copy Paste: [[2401.06121]] TOFU: A Task of Fictitious Unlearning for LLMs(http://arxiv.org/abs/2401.06121)
• Summary:

  Large language models trained on massive corpora of data from the web can memorize and reproduce sensitive or private data raising both legal and ethical concerns. Unlearning, or tuning models to forget information present in their training data, provides us with a way to protect private data after training. Although several methods exist for such unlearning, it is unclear to what extent they result in models equivalent to those where the data to be forgotten was never learned in the first place. To address this challenge, we present TOFU, a Task of Fictitious Unlearning, as a benchmark aimed at helping deepen our understanding of unlearning. We offer a dataset of 200 diverse synthetic author profiles, each consisting of 20 question-answer pairs, and a subset of these profiles called the forget set that serves as the target for unlearning. We compile a suite of metrics that work together to provide a holistic picture of unlearning efficacy. Finally, we provide a set of baseline results from existing unlearning algorithms. Importantly, none of the baselines we consider show effective unlearning motivating continued efforts to develop approaches for unlearning that effectively tune models so that they truly behave as if they were never trained on the forget data at all.

• 摘要：

  基于网络上的大量数据训练的大型语言模型可以记忆和复制敏感或私人数据，从而引起法律和道德问题。忘却或调整模型以忘记训练数据中存在的信息，为我们提供了一种在训练后保护私人数据的方法。尽管存在几种用于这种遗忘的方法，但尚不清楚它们在多大程度上会产生与最初从未学习过要遗忘的数据的模型相当的模型。为了应对这一挑战，我们提出了 TOFU，一项虚构的忘却任务，作为基准，旨在帮助加深我们对忘却的理解。我们提供了一个由 200 个不同的合成作者资料组成的数据集，每个资料由 20 个问答对组成，这些资料的一个子集称为遗忘集，用作忘却的目标。我们编制了一套指标，这些指标共同作用，提供了遗忘效率的整体情况。最后，我们提供了一组来自现有遗忘算法的基线结果。重要的是，我们考虑的基线都没有显示出有效的遗忘数据，可以激励人们继续努力开发遗忘方法，从而有效地调整模型，使它们真正表现得好像从未接受过遗忘数据的训练一样。

Title: Binary Linear Tree Commitment-based Ownership Protection for Distributed Machine Learning. (arXiv:2401.05895v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.05895
• Code URL: null
• Copy Paste: [[2401.05895]] Binary Linear Tree Commitment-based Ownership Protection for Distributed Machine Learning(http://arxiv.org/abs/2401.05895)
• Summary:

  Distributed machine learning enables parallel training of extensive datasets by delegating computing tasks across multiple workers. Despite the cost reduction benefits of distributed machine learning, the dissemination of final model weights often leads to potential conflicts over model ownership as workers struggle to substantiate their involvement in the training computation. To address the above ownership issues and prevent accidental failures and malicious attacks, verifying the computational integrity and effectiveness of workers becomes particularly crucial in distributed machine learning. In this paper, we proposed a novel binary linear tree commitment-based ownership protection model to ensure computational integrity with limited overhead and concise proof. Due to the frequent updates of parameters during training, our commitment scheme introduces a maintainable tree structure to reduce the costs of updating proofs. Distinguished from SNARK-based verifiable computation, our model achieves efficient proof aggregation by leveraging inner product arguments. Furthermore, proofs of model weights are watermarked by worker identity keys to prevent commitments from being forged or duplicated. The performance analysis and comparison with SNARK-based hash commitments validate the efficacy of our model in preserving computational integrity within distributed machine learning.

• 摘要：

  分布式机器学习通过将计算任务委托给多个工作人员来实现对大量数据集的并行训练。尽管分布式机器学习具有降低成本的好处，但最终模型权重的传播往往会导致模型所有权的潜在冲突，因为工作人员很难证实他们对训练计算的参与。为了解决上述所有权问题并防止意外故障和恶意攻击，验证工作人员的计算完整性和有效性在分布式机器学习中变得尤为重要。在本文中，我们提出了一种新颖的基于二叉线性树承诺的所有权保护模型，以有限的开销和简洁的证明来确保计算完整性。由于训练过程中参数的频繁更新，我们的承诺方案引入了可维护的树结构来降低更新证明的成本。与基于 SNARK 的可验证计算不同，我们的模型通过利用内积参数实现高效的证明聚合。此外，模型权重的证明由工作人员身份密钥加水印，以防止承诺被伪造或复制。性能分析以及与基于 SNARK 的哈希承诺的比较验证了我们的模型在分布式机器学习中保持计算完整性的有效性。

defense

Title: Use of Graph Neural Networks in Aiding Defensive Cyber Operations. (arXiv:2401.05680v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2401.05680
• Code URL: null
• Copy Paste: [[2401.05680]] Use of Graph Neural Networks in Aiding Defensive Cyber Operations(http://arxiv.org/abs/2401.05680)
• Summary:

  In an increasingly interconnected world, where information is the lifeblood of modern society, regular cyber-attacks sabotage the confidentiality, integrity, and availability of digital systems and information. Additionally, cyber-attacks differ depending on the objective and evolve rapidly to disguise defensive systems. However, a typical cyber-attack demonstrates a series of stages from attack initiation to final resolution, called an attack life cycle. These diverse characteristics and the relentless evolution of cyber attacks have led cyber defense to adopt modern approaches like Machine Learning to bolster defensive measures and break the attack life cycle. Among the adopted ML approaches, Graph Neural Networks have emerged as a promising approach for enhancing the effectiveness of defensive measures due to their ability to process and learn from heterogeneous cyber threat data. In this paper, we look into the application of GNNs in aiding to break each stage of one of the most renowned attack life cycles, the Lockheed Martin Cyber Kill Chain. We address each phase of CKC and discuss how GNNs contribute to preparing and preventing an attack from a defensive standpoint. Furthermore, We also discuss open research areas and further improvement scopes.

• 摘要：

  在一个日益互联的世界中，信息是现代社会的命脉，定期的网络攻击会破坏数字系统和信息的机密性、完整性和可用性。此外，网络攻击根据目标的不同而有所不同，并且会迅速发展以伪装防御系统。然而，典型的网络攻击展示了从攻击发起到最终解决的一系列阶段，称为攻击生命周期。这些不同的特征和网络攻击的不断发展导致网络防御采用机器学习等现代方法来加强防御措施并打破攻击生命周期。在采用的机器学习方法中，图神经网络因其处理和学习异构网络威胁数据的能力而成为增强防御措施有效性的有前途的方法。在本文中，我们研究了 GNN 在帮助打破最著名的攻击生命周期之一——洛克希德·马丁网络杀伤链的每个阶段中的应用。我们讨论 CKC 的每个阶段，并讨论 GNN 如何从防御的角度帮助准备和防止攻击。此外，我们还讨论了开放的研究领域和进一步改进的范围。

Title: Can We Trust the Unlabeled Target Data? Towards Backdoor Attack and Defense on Model Adaptation. (arXiv:2401.06030v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2401.06030
• Code URL: null
• Copy Paste: [[2401.06030]] Can We Trust the Unlabeled Target Data? Towards Backdoor Attack and Defense on Model Adaptation(http://arxiv.org/abs/2401.06030)
• Summary:

  Model adaptation tackles the distribution shift problem with a pre-trained model instead of raw data, becoming a popular paradigm due to its great privacy protection. Existing methods always assume adapting to a clean target domain, overlooking the security risks of unlabeled samples. In this paper, we explore the potential backdoor attacks on model adaptation launched by well-designed poisoning target data. Concretely, we provide two backdoor triggers with two poisoning strategies for different prior knowledge owned by attackers. These attacks achieve a high success rate and keep the normal performance on clean samples in the test stage. To defend against backdoor embedding, we propose a plug-and-play method named MixAdapt, combining it with existing adaptation algorithms. Experiments across commonly used benchmarks and adaptation methods demonstrate the effectiveness of MixAdapt. We hope this work will shed light on the safety of learning with unlabeled data.

• 摘要：

  模型自适应通过预先训练的模型而不是原始数据来解决分布转移问题，由于其良好的隐私保护而成为流行的范例。现有方法总是假设适应干净的目标域，忽略了未标记样本的安全风险。在本文中，我们探讨了精心设计的中毒目标数据对模型自适应发起的潜在后门攻击。具体来说，我们针对攻击者拥有的不同先验知识提供了两个后门触发器和两种中毒策略。这些攻击取得了很高的成功率，并在测试阶段保持了干净样本上的正常性能。为了防御后门嵌入，我们提出了一种名为 MixAdapt 的即插即用方法，并将其与现有的自适应算法相结合。常用基准和适应方法的实验证明了 MixAdapt 的有效性。我们希望这项工作能够阐明使用未标记数据进行学习的安全性。

attack

Title: Universal Vulnerabilities in Large Language Models: In-context Learning Backdoor Attacks. (arXiv:2401.05949v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.05949
• Code URL: https://github.com/shuaizhao95/iclattack
• Copy Paste: [[2401.05949]] Universal Vulnerabilities in Large Language Models: In-context Learning Backdoor Attacks(http://arxiv.org/abs/2401.05949)
• Summary:

  In-context learning, a paradigm bridging the gap between pre-training and fine-tuning, has demonstrated high efficacy in several NLP tasks, especially in few-shot settings. Unlike traditional fine-tuning methods, in-context learning adapts pre-trained models to unseen tasks without updating any parameters. Despite being widely applied, in-context learning is vulnerable to malicious attacks. In this work, we raise security concerns regarding this paradigm. Our studies demonstrate that an attacker can manipulate the behavior of large language models by poisoning the demonstration context, without the need for fine-tuning the model. Specifically, we have designed a new backdoor attack method, named ICLAttack, to target large language models based on in-context learning. Our method encompasses two types of attacks: poisoning demonstration examples and poisoning prompts, which can make models behave in accordance with predefined intentions. ICLAttack does not require additional fine-tuning to implant a backdoor, thus preserving the model's generality. Furthermore, the poisoned examples are correctly labeled, enhancing the natural stealth of our attack method. Extensive experimental results across several language models, ranging in size from 1.3B to 40B parameters, demonstrate the effectiveness of our attack method, exemplified by a high average attack success rate of 95.0% across the three datasets on OPT models. Our findings highlight the vulnerabilities of language models, and we hope this work will raise awareness of the possible security threats associated with in-context learning.

• 摘要：

  上下文学习是一种弥合预训练和微调之间差距的范式，已在多项 NLP 任务中表现出高效能，尤其是在少量样本设置中。与传统的微调方法不同，上下文学习使预先训练的模型适应未见过的任务，而无需更新任何参数。尽管应用广泛，但情境学习很容易受到恶意攻击。在这项工作中，我们提出了有关此范例的安全问题。我们的研究表明，攻击者可以通过毒害演示上下文来操纵大型语言模型的行为，而无需对模型进行微调。具体来说，我们设计了一种新的后门攻击方法，名为 ICLAtack，针对基于上下文学习的大型语言模型。我们的方法包含两种类型的攻击：中毒演示示例和中毒提示，这可以使模型按照预定义的意图行事。 ICLAttack 不需要额外的微调来植入后门，从而保留了模型的通用性。此外，中毒的例子被正确标记，增强了我们的攻击方法的自然隐蔽性。跨多种语言模型的广泛实验结果（参数大小从 1.3B 到 40B 不等）证明了我们的攻击方法的有效性，OPT 模型上的三个数据集的平均攻击成功率高达 95.0%。我们的研究结果强调了语言模型的漏洞，我们希望这项工作能够提高人们对与上下文学习相关的可能安全威胁的认识。

Title: Combating Adversarial Attacks with Multi-Agent Debate. (arXiv:2401.05998v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.05998
• Code URL: null
• Copy Paste: [[2401.05998]] Combating Adversarial Attacks with Multi-Agent Debate(http://arxiv.org/abs/2401.05998)
• Summary:

  While state-of-the-art language models have achieved impressive results, they remain susceptible to inference-time adversarial attacks, such as adversarial prompts generated by red teams arXiv:2209.07858. One approach proposed to improve the general quality of language model generations is multi-agent debate, where language models self-evaluate through discussion and feedback arXiv:2305.14325. We implement multi-agent debate between current state-of-the-art language models and evaluate models' susceptibility to red team attacks in both single- and multi-agent settings. We find that multi-agent debate can reduce model toxicity when jailbroken or less capable models are forced to debate with non-jailbroken or more capable models. We also find marginal improvements through the general usage of multi-agent interactions. We further perform adversarial prompt content classification via embedding clustering, and analyze the susceptibility of different models to different types of attack topics.

• 摘要：

  虽然最先进的语言模型取得了令人印象深刻的结果，但它们仍然容易受到推理时对抗性攻击，例如红队生成的对抗性提示arXiv:2209.07858。提出的一种提高语言模型生成总体质量的方法是多智能体辩论，其中语言模型通过讨论和反馈进行自我评估arXiv： 2305.14325。我们在当前最先进的语言模型之间实现多代理辩论，并评估模型在单代理和多代理设置中对红队攻击的敏感性。我们发现，当越狱或能力较差的模型被迫与未越狱或能力更强的模型进行辩论时，多智能体辩论可以减少模型毒性。我们还发现通过多智能体交互的普遍使用带来了边际改进。我们进一步通过嵌入聚类进行对抗性提示内容分类，并分析不同模型对不同类型攻击主题的敏感性。

Title: TEN-GUARD: Tensor Decomposition for Backdoor Attack Detection in Deep Neural Networks. (arXiv:2401.05432v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.05432
• Code URL: null
• Copy Paste: [[2401.05432]] TEN-GUARD: Tensor Decomposition for Backdoor Attack Detection in Deep Neural Networks(http://arxiv.org/abs/2401.05432)
• Summary:

  As deep neural networks and the datasets used to train them get larger, the default approach to integrating them into research and commercial projects is to download a pre-trained model and fine tune it. But these models can have uncertain provenance, opening up the possibility that they embed hidden malicious behavior such as trojans or backdoors, where small changes to an input (triggers) can cause the model to produce incorrect outputs (e.g., to misclassify). This paper introduces a novel approach to backdoor detection that uses two tensor decomposition methods applied to network activations. This has a number of advantages relative to existing detection methods, including the ability to analyze multiple models at the same time, working across a wide variety of network architectures, making no assumptions about the nature of triggers used to alter network behavior, and being computationally efficient. We provide a detailed description of the detection pipeline along with results on models trained on the MNIST digit dataset, CIFAR-10 dataset, and two difficult datasets from NIST's TrojAI competition. These results show that our method detects backdoored networks more accurately and efficiently than current state-of-the-art methods.

• 摘要：

  随着深度神经网络和用于训练它们的数据集变得越来越大，将它们集成到研究和商业项目中的默认方法是下载预先训练的模型并对其进行微调。但这些模型的来源可能不确定，因此有可能嵌入隐藏的恶意行为，例如木马或后门，其中输入（触发）的微小变化可能导致模型产生不正确的输出（例如错误分类）。本文介绍了一种新的后门检测方法，该方法使用两种应用于网络激活的张量分解方法。相对于现有的检测方法，这具有许多优点，包括同时分析多个模型的能力、跨各种网络架构工作、不对用于改变网络行为的触发器的性质做出假设，以及可计算性高效的。我们提供了检测管道的详细描述，以及在 MNIST 数字数据集、CIFAR-10 数据集和 NIST TrojAI 竞赛中的两个困难数据集上训练的模型的结果。这些结果表明，我们的方法比当前最先进的方法更准确、更有效地检测后门网络。

Title: SENet: Visual Detection of Online Social Engineering Attack Campaigns. (arXiv:2401.05569v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2401.05569
• Code URL: null
• Copy Paste: [[2401.05569]] SENet: Visual Detection of Online Social Engineering Attack Campaigns(http://arxiv.org/abs/2401.05569)
• Summary:

  Social engineering (SE) aims at deceiving users into performing actions that may compromise their security and privacy. These threats exploit weaknesses in human's decision making processes by using tactics such as pretext, baiting, impersonation, etc. On the web, SE attacks include attack classes such as scareware, tech support scams, survey scams, sweepstakes, etc., which can result in sensitive data leaks, malware infections, and monetary loss. For instance, US consumers lose billions of dollars annually due to various SE attacks. Unfortunately, generic social engineering attacks remain understudied, compared to other important threats, such as software vulnerabilities and exploitation, network intrusions, malicious software, and phishing. The few existing technical studies that focus on social engineering are limited in scope and mostly focus on measurements rather than developing a generic defense. To fill this gap, we present SEShield, a framework for in-browser detection of social engineering attacks. SEShield consists of three main components: (i) a custom security crawler, called SECrawler, that is dedicated to scouting the web to collect examples of in-the-wild SE attacks; (ii) SENet, a deep learning-based image classifier trained on data collected by SECrawler that aims to detect the often glaring visual traits of SE attack pages; and (iii) SEGuard, a proof-of-concept extension that embeds SENet into the web browser and enables real-time SE attack detection. We perform an extensive evaluation of our system and show that SENet is able to detect new instances of SE attacks with a detection rate of up to 99.6% at 1% false positive, thus providing an effective first defense against SE attacks on the web.

• 摘要：

  社交工程 (SE) 旨在欺骗用户执行可能损害其安全和隐私的操作。这些威胁通过使用借口、诱饵、冒充等策略来利用人类决策过程中的弱点。在网络上，SE 攻击包括恐吓软件、技术支持诈骗、调查诈骗、抽奖等攻击类别，这可能会导致敏感数据泄露、恶意软件感染和金钱损失。例如，美国消费者每年因各种 SE 攻击而损失数十亿美元。不幸的是，与软件漏洞和利用、网络入侵、恶意软件和网络钓鱼等其他重要威胁相比，一般的社会工程攻击仍未得到充分研究。现有的少数专注于社会工程的技术研究范围有限，并且主要侧重于测量而不是开发通用防御。为了填补这一空白，我们提出了 SEShield，一个用于在浏览器内检测社会工程攻击的框架。 SEShield 由三个主要组件组成：(i) 一个名为 SECrawler 的自定义安全爬虫，专门用于搜索网络以收集野外 SE 攻击的示例； (ii) SENet，一种基于深度学习的图像分类器，根据 SECrawler 收集的数据进行训练，旨在检测 SE 攻击页面经常引人注目的视觉特征； (iii) SEGuard，一种概念验证扩展，可将 SENet 嵌入到 Web 浏览器中并实现实时 SE 攻击检测。我们对我们的系统进行了广泛的评估，结果表明 SENet 能够检测新的 SE 攻击实例，在 1% 的误报率下检测率高达 99.6%，从而为网络上的 SE 攻击提供有效的第一道防御。

robust

Title: STR-Cert: Robustness Certification for Deep Text Recognition on Deep Learning Pipelines and Vision Transformers. (arXiv:2401.05338v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.05338
• Code URL: null
• Copy Paste: [[2401.05338]] STR-Cert: Robustness Certification for Deep Text Recognition on Deep Learning Pipelines and Vision Transformers(http://arxiv.org/abs/2401.05338)
• Summary:

  Robustness certification, which aims to formally certify the predictions of neural networks against adversarial inputs, has become an integral part of important tool for safety-critical applications. Despite considerable progress, existing certification methods are limited to elementary architectures, such as convolutional networks, recurrent networks and recently Transformers, on benchmark datasets such as MNIST. In this paper, we focus on the robustness certification of scene text recognition (STR), which is a complex and extensively deployed image-based sequence prediction problem. We tackle three types of STR model architectures, including the standard STR pipelines and the Vision Transformer. We propose STR-Cert, the first certification method for STR models, by significantly extending the DeepPoly polyhedral verification framework via deriving novel polyhedral bounds and algorithms for key STR model components. Finally, we certify and compare STR models on six datasets, demonstrating the efficiency and scalability of robustness certification, particularly for the Vision Transformer.

• 摘要：

  鲁棒性认证旨在正式证明神经网络针对对抗性输入的预测，已成为安全关键型应用的重要工具的组成部分。尽管取得了相当大的进步，但现有的认证方法仅限于 MNIST 等基准数据集上的基本架构，例如卷积网络、循环网络和最近的 Transformer。在本文中，我们重点关注场景文本识别（STR）的鲁棒性认证，这是一个复杂且广泛部署的基于图像的序列预测问题。我们处理三种类型的 STR 模型架构，包括标准 STR 管道和 Vision Transformer。我们提出了 STR-Cert，这是 STR 模型的第一个认证方法，通过为关键 STR 模型组件导出新颖的多面体边界和算法来显着扩展 DeepPoly 多面体验证框架。最后，我们在六个数据集上验证并比较 STR 模型，展示鲁棒性认证的效率和可扩展性，特别是对于 Vision Transformer。

Title: Generalized Categories Discovery for Long-tailed Recognition. (arXiv:2401.05352v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.05352
• Code URL: null
• Copy Paste: [[2401.05352]] Generalized Categories Discovery for Long-tailed Recognition(http://arxiv.org/abs/2401.05352)
• Summary:

  Generalized Class Discovery (GCD) plays a pivotal role in discerning both known and unknown categories from unlabeled datasets by harnessing the insights derived from a labeled set comprising recognized classes. A significant limitation in prevailing GCD methods is their presumption of an equitably distributed category occurrence in unlabeled data. Contrary to this assumption, visual classes in natural environments typically exhibit a long-tailed distribution, with known or prevalent categories surfacing more frequently than their rarer counterparts. Our research endeavors to bridge this disconnect by focusing on the long-tailed Generalized Category Discovery (Long-tailed GCD) paradigm, which echoes the innate imbalances of real-world unlabeled datasets. In response to the unique challenges posed by Long-tailed GCD, we present a robust methodology anchored in two strategic regularizations: (i) a reweighting mechanism that bolsters the prominence of less-represented, tail-end categories, and (ii) a class prior constraint that aligns with the anticipated class distribution. Comprehensive experiments reveal that our proposed method surpasses previous state-of-the-art GCD methods by achieving an improvement of approximately 6 - 9% on ImageNet100 and competitive performance on CIFAR100.

• 摘要：

  广义类发现 (GCD) 通过利用从包含已识别类的标记集得出的见解，在从未标记数据集中辨别已知和未知类别方面发挥着关键作用。流行的 GCD 方法的一个显着限制是它们对未标记数据中类别出现的公平分布的假设。与这一假设相反，自然环境中的视觉类别通常表现出长尾分布，已知或普遍的类别比罕见的类别更频繁地出现。我们的研究致力于通过关注长尾广义类别发现（长尾 GCD）范式来弥合这种脱节，该范式反映了现实世界未标记数据集固有的不平衡。为了应对长尾 GCD 带来的独特挑战，我们提出了一种基于两个战略规范化的稳健方法：(i) 一种重新加权机制，可增强代表性较少的尾端类别的突出地位，以及 (ii) 一个类别与预期类别分布一致的先验约束。综合实验表明，我们提出的方法超越了以前最先进的 GCD 方法，在 ImageNet100 上实现了约 6 - 9% 的改进，在 CIFAR100 上实现了具有竞争力的性能。

Title: Wasserstein Distance-based Expansion of Low-Density Latent Regions for Unknown Class Detection. (arXiv:2401.05594v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.05594
• Code URL: null
• Copy Paste: [[2401.05594]] Wasserstein Distance-based Expansion of Low-Density Latent Regions for Unknown Class Detection(http://arxiv.org/abs/2401.05594)
• Summary:

  This paper addresses the significant challenge in open-set object detection (OSOD): the tendency of state-of-the-art detectors to erroneously classify unknown objects as known categories with high confidence. We present a novel approach that effectively identifies unknown objects by distinguishing between high and low-density regions in latent space. Our method builds upon the Open-Det (OD) framework, introducing two new elements to the loss function. These elements enhance the known embedding space's clustering and expand the unknown space's low-density regions. The first addition is the Class Wasserstein Anchor (CWA), a new function that refines the classification boundaries. The second is a spectral normalisation step, improving the robustness of the model. Together, these augmentations to the existing Contrastive Feature Learner (CFL) and Unknown Probability Learner (UPL) loss functions significantly improve OSOD performance. Our proposed OpenDet-CWA (OD-CWA) method demonstrates: a) a reduction in open-set errors by approximately 17%-22%, b) an enhancement in novelty detection capability by 1.5%-16%, and c) a decrease in the wilderness index by 2%-20% across various open-set scenarios. These results represent a substantial advancement in the field, showcasing the potential of our approach in managing the complexities of open-set object detection.

• 摘要：

  本文解决了开放集对象检测 (OSOD) 中的重大挑战：最先进的检测器往往会以高置信度错误地将未知对象分类为已知类别。我们提出了一种新颖的方法，通过区分潜在空间中的高密度区域和低密度区域来有效识别未知物体。我们的方法建立在 Open-Det (OD) 框架之上，为损失函数引入了两个新元素。这些元素增强了已知嵌入空间的聚类并扩展了未知空间的低密度区域。第一个添加是 Class Wasserstein Anchor (CWA)，这是一个细化分类边界的新函数。第二个是光谱归一化步骤，提高模型的稳健性。总之，对现有对比特征学习器 (CFL) 和未知概率学习器 (UPL) 损失函数的这些增强显着提高了 OSOD 性能。我们提出的 OpenDet-CWA (OD-CWA) 方法证明：a) 开集误差减少约 17%-22%，b) 新颖性检测能力增强 1.5%-16%，c) 减少在各种开放场景中，荒野指数提高了 2%-20%。这些结果代表了该领域的重大进步，展示了我们的方法在管理开放集对象检测的复杂性方面的潜力。

Title: Nucleus subtype classification using inter-modality learning. (arXiv:2401.05602v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.05602
• Code URL: null
• Copy Paste: [[2401.05602]] Nucleus subtype classification using inter-modality learning(http://arxiv.org/abs/2401.05602)
• Summary:

  Understanding the way cells communicate, co-locate, and interrelate is essential to understanding human physiology. Hematoxylin and eosin (H&E) staining is ubiquitously available both for clinical studies and research. The Colon Nucleus Identification and Classification (CoNIC) Challenge has recently innovated on robust artificial intelligence labeling of six cell types on H&E stains of the colon. However, this is a very small fraction of the number of potential cell classification types. Specifically, the CoNIC Challenge is unable to classify epithelial subtypes (progenitor, endocrine, goblet), lymphocyte subtypes (B, helper T, cytotoxic T), or connective subtypes (fibroblasts, stromal). In this paper, we propose to use inter-modality learning to label previously un-labelable cell types on virtual H&E. We leveraged multiplexed immunofluorescence (MxIF) histology imaging to identify 14 subclasses of cell types. We performed style transfer to synthesize virtual H&E from MxIF and transferred the higher density labels from MxIF to these virtual H&E images. We then evaluated the efficacy of learning in this approach. We identified helper T and progenitor nuclei with positive predictive values of $0.34 \pm 0.15$ (prevalence $0.03 \pm 0.01$) and $0.47 \pm 0.1$ (prevalence $0.07 \pm 0.02$) respectively on virtual H&E. This approach represents a promising step towards automating annotation in digital pathology.

• 摘要：

  了解细胞通信、共定位和相互关联的方式对于了解人类生理学至关重要。苏木精和伊红 (H&E) 染色普遍可用于临床研究和研究。结肠核识别和分类 (CoNIC) 挑战赛最近在结肠 H&E 染色上对六种细胞类型进行了强大的人工智能标记。然而，这只是潜在细胞分类类型数量的一小部分。具体来说，CoNIC 挑战无法对上皮亚型（祖细胞、内分泌细胞、杯状细胞）、淋巴细胞亚型（B、辅助 T、细胞毒性 T）或结缔细胞亚型（成纤维细胞、基质细胞）进行分类。在本文中，我们建议使用跨模态学习来标记虚拟 H&E 上以前无法标记的细胞类型。我们利用多重免疫荧光 (MxIF) 组织学成像来识别 14 种细胞类型亚类。我们执行风格转移以从 MxIF 合成虚拟 H&E，并将更高密度的标签从 MxIF 转移到这些虚拟 H&E 图像。然后我们评估了这种方法的学习效果。我们在虚拟 H&E 上确定了辅助 T 和祖细胞核的阳性预测值分别为 $0.34 \pm 0.15$（患病率 $0.03 \pm 0.01$）和 $0.47 \pm 0.1$（患病率 $0.07 \pm 0.02$）。这种方法代表了数字病理学自动化注释方面迈出了有希望的一步。

Title: REBUS: A Robust Evaluation Benchmark of Understanding Symbols. (arXiv:2401.05604v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.05604
• Code URL: https://github.com/cvndsh/rebus
• Copy Paste: [[2401.05604]] REBUS: A Robust Evaluation Benchmark of Understanding Symbols(http://arxiv.org/abs/2401.05604)
• Summary:

  We propose a new benchmark evaluating the performance of multimodal large language models on rebus puzzles. The dataset covers 333 original examples of image-based wordplay, cluing 13 categories such as movies, composers, major cities, and food. To achieve good performance on the benchmark of identifying the clued word or phrase, models must combine image recognition and string manipulation with hypothesis testing, multi-step reasoning, and an understanding of human cognition, making for a complex, multimodal evaluation of capabilities. We find that proprietary models such as GPT-4V and Gemini Pro significantly outperform all other tested models. However, even the best model has a final accuracy of just 24%, highlighting the need for substantial improvements in reasoning. Further, models rarely understand all parts of a puzzle, and are almost always incapable of retroactively explaining the correct answer. Our benchmark can therefore be used to identify major shortcomings in the knowledge and reasoning of multimodal large language models.

• 摘要：

  我们提出了一个新的基准，评估多模式大语言模型在画画谜题上的性能。该数据集涵盖 333 个基于图像的文字游戏的原始示例，涵盖电影、作曲家、主要城市和食物等 13 个类别。为了在识别线索单词或短语的基准上取得良好的性能，模型必须将图像识别和字符串操作与假设检验、多步骤推理和对人类认知的理解结合起来，从而对能力进行复杂的多模式评估。我们发现 GPT-4V 和 Gemini Pro 等专有模型的性能明显优于所有其他测试模型。然而，即使是最好的模型，最终准确率也仅为 24%，这凸显了推理方面需要大幅改进的必要性。此外，模型很少理解谜题的所有部分，并且几乎总是无法追溯解释正确的答案。因此，我们的基准可用于识别多模态大语言模型的知识和推理中的主要缺陷。

Title: Evaluating Data Augmentation Techniques for Coffee Leaf Disease Classification. (arXiv:2401.05768v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.05768
• Code URL: null
• Copy Paste: [[2401.05768]] Evaluating Data Augmentation Techniques for Coffee Leaf Disease Classification(http://arxiv.org/abs/2401.05768)
• Summary:

  The detection and classification of diseases in Robusta coffee leaves are essential to ensure that plants are healthy and the crop yield is kept high. However, this job requires extensive botanical knowledge and much wasted time. Therefore, this task and others similar to it have been extensively researched subjects in image classification. Regarding leaf disease classification, most approaches have used the more popular PlantVillage dataset while completely disregarding other datasets, like the Robusta Coffee Leaf (RoCoLe) dataset. As the RoCoLe dataset is imbalanced and does not have many samples, fine-tuning of pre-trained models and multiple augmentation techniques need to be used. The current paper uses the RoCoLe dataset and approaches based on deep learning for classifying coffee leaf diseases from images, incorporating the pix2pix model for segmentation and cycle-generative adversarial network (CycleGAN) for augmentation. Our study demonstrates the effectiveness of Transformer-based models, online augmentations, and CycleGAN augmentation in improving leaf disease classification. While synthetic data has limitations, it complements real data, enhancing model performance. These findings contribute to developing robust techniques for plant disease detection and classification.

• 摘要：

  罗布斯塔咖啡叶病害的检测和分类对于确保植物健康和保持作物高产至关重要。然而，这项工作需要广泛的植物知识并且浪费大量时间。因此，该任务和其他类似任务已成为图像分类领域广泛研究的课题。关于叶病分类，大多数方法都使用更流行的 PlantVillage 数据集，而完全忽略其他数据集，例如 Robusta Coffee Leaf (RoCoLe) 数据集。由于RoCoLe数据集不平衡且样本不多，需要使用预训练模型的微调和多种增强技术。当前的论文使用 RoCoLe 数据集和基于深度学习的方法对图像中的咖啡叶病进行分类，并结合用于分割的 pix2pix 模型和用于增强的循环生成对抗网络 (CycleGAN)。我们的研究证明了基于 Transformer 的模型、在线增强和 CycleGAN 增强在改善叶病分类方面的有效性。虽然合成数据有局限性，但它补充了真实数据，提高了模型性能。这些发现有助于开发强大的植物病害检测和分类技术。

Title: Learn From Zoom: Decoupled Supervised Contrastive Learning For WCE Image Classification. (arXiv:2401.05771v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.05771
• Code URL: null
• Copy Paste: [[2401.05771]] Learn From Zoom: Decoupled Supervised Contrastive Learning For WCE Image Classification(http://arxiv.org/abs/2401.05771)
• Summary:

  Accurate lesion classification in Wireless Capsule Endoscopy (WCE) images is vital for early diagnosis and treatment of gastrointestinal (GI) cancers. However, this task is confronted with challenges like tiny lesions and background interference. Additionally, WCE images exhibit higher intra-class variance and inter-class similarities, adding complexity. To tackle these challenges, we propose Decoupled Supervised Contrastive Learning for WCE image classification, learning robust representations from zoomed-in WCE images generated by Saliency Augmentor. Specifically, We use uniformly down-sampled WCE images as anchors and WCE images from the same class, especially their zoomed-in images, as positives. This approach empowers the Feature Extractor to capture rich representations from various views of the same image, facilitated by Decoupled Supervised Contrastive Learning. Training a linear Classifier on these representations within 10 epochs yields an impressive 92.01% overall accuracy, surpassing the prior state-of-the-art (SOTA) by 0.72% on a blend of two publicly accessible WCE datasets. Code is available at: https://github.com/Qiukunpeng/DSCL.

• 摘要：

  无线胶囊内窥镜 (WCE) 图像中准确的病变分类对于胃肠道 (GI) 癌症的早期诊断和治疗至关重要。然而，这项任务面临着微小病变和背景干扰等挑战。此外，WCE 图像表现出更高的类内方差和类间相似性，从而增加了复杂性。为了应对这些挑战，我们提出了用于 WCE 图像分类的解耦监督对比学习，从显着性增强器生成的放大 WCE 图像中学习鲁棒的表示。具体来说，我们使用统一下采样的 WCE 图像作为锚点，并使用来自同一类的 WCE 图像，特别是它们的放大图像作为正值。这种方法使特征提取器能够在解耦监督对比学习的促进下从同一图像的不同视图中捕获丰富的表示。在 10 个 epoch 内对这些表示进行训练线性分类器，总体准确率高达 92.01%，令人印象深刻，在两个可公开访问的 WCE 数据集的混合上，比之前最先进的技术 (SOTA) 提高了 0.72%。代码位于：https://github.com/Qiukunpeng/DSCL。

Title: CoSSegGaussians: Compact and Swift Scene Segmenting 3D Gaussians. (arXiv:2401.05925v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.05925
• Code URL: null
• Copy Paste: [[2401.05925]] CoSSegGaussians: Compact and Swift Scene Segmenting 3D Gaussians(http://arxiv.org/abs/2401.05925)
• Summary:

  We propose Compact and Swift Segmenting 3D Gaussians(CoSSegGaussians), a method for compact 3D-consistent scene segmentation at fast rendering speed with only RGB images input. Previous NeRF-based 3D segmentation methods have relied on implicit or voxel neural scene representation and ray-marching volume rendering which are time consuming. Recent 3D Gaussian Splatting significantly improves the rendering speed, however, existing Gaussians-based segmentation methods(eg: Gaussian Grouping) fail to provide compact segmentation masks especially in zero-shot segmentation, which is mainly caused by the lack of robustness and compactness for straightforwardly assigning learnable parameters to each Gaussian when encountering inconsistent 2D machine-generated labels. Our method aims to achieve compact and reliable zero-shot scene segmentation swiftly by mapping fused spatial and semantically meaningful features for each Gaussian point with a shallow decoding network. Specifically, our method firstly optimizes Gaussian points' position, convariance and color attributes under the supervision of RGB images. After Gaussian Locating, we distill multi-scale DINO features extracted from images through unprojection to each Gaussian, which is then incorporated with spatial features from the fast point features processing network, i.e. RandLA-Net. Then the shallow decoding MLP is applied to the multi-scale fused features to obtain compact segmentation. Experimental results show that our model can perform high-quality zero-shot scene segmentation, as our model outperforms other segmentation methods on both semantic and panoptic segmentation task, meanwhile consumes approximately only 10% segmenting time compared to NeRF-based segmentation. Code and more results will be available at https://David-Dou.github.io/CoSSegGaussians

• 摘要：

  我们提出了紧凑且快速的 3D 高斯分割（CoSSegGaussians），这是一种仅使用 RGB 图像输入即可快速渲染的紧凑 3D 一致场景分割方法。以前基于 NeRF 的 3D 分割方法依赖于隐式或体素神经场景表示和光线行进体积渲染，这些方法非常耗时。最近的3D Gaussian Splatting显着提高了渲染速度，然而，现有的基于高斯的分割方法（例如：高斯分组）无法提供紧凑的分割掩模，特别是在零样本分割中，这主要是由于直接缺乏鲁棒性和紧凑性造成的当遇到不一致的 2D 机器生成标签时，为每个高斯分配可学习的参数。我们的方法旨在通过浅层解码网络为每个高斯点映射融合的空间和语义上有意义的特征，从而快速实现紧凑且可靠的零镜头场景分割。具体来说，我们的方法首先在 RGB 图像的监督下优化高斯点的位置、协方差和颜色属性。在高斯定位之后，我们将通过非投影从图像中提取的多尺度 DINO 特征提取到每个高斯，然后将其与来自快速点特征处理网络（即 RandLA-Net）的空间特征合并。然后将浅层解码MLP应用于多尺度融合特征以获得紧凑分割。实验结果表明，我们的模型可以执行高质量的零镜头场景分割，因为我们的模型在语义和全景分割任务上都优于其他分割方法，同时与基于 NeRF 的分割相比，仅消耗大约 10% 的分割时间。代码和更多结果将在 https://David-Dou.github.io/CoSSegGaussians 上提供

Title: Sea ice detection using concurrent multispectral and synthetic aperture radar imagery. (arXiv:2401.06009v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.06009
• Code URL: null
• Copy Paste: [[2401.06009]] Sea ice detection using concurrent multispectral and synthetic aperture radar imagery(http://arxiv.org/abs/2401.06009)
• Summary:

  Synthetic Aperture Radar (SAR) imagery is the primary data type used for sea ice mapping due to its spatio-temporal coverage and the ability to detect sea ice independent of cloud and lighting conditions. Automatic sea ice detection using SAR imagery remains problematic due to the presence of ambiguous signal and noise within the image. Conversely, ice and water are easily distinguishable using multispectral imagery (MSI), but in the polar regions the ocean's surface is often occluded by cloud or the sun may not appear above the horizon for many months. To address some of these limitations, this paper proposes a new tool trained using concurrent multispectral Visible and SAR imagery for sea Ice Detection (ViSual\_IceD). ViSual\_IceD is a convolution neural network (CNN) that builds on the classic U-Net architecture by containing two parallel encoder stages, enabling the fusion and concatenation of MSI and SAR imagery containing different spatial resolutions. The performance of ViSual\_IceD is compared with U-Net models trained using concatenated MSI and SAR imagery as well as models trained exclusively on MSI or SAR imagery. ViSual\_IceD outperforms the other networks, with a F1 score 1.60\% points higher than the next best network, and results indicate that ViSual\_IceD is selective in the image type it uses during image segmentation. Outputs from ViSual\_IceD are compared to sea ice concentration products derived from the AMSR2 Passive Microwave (PMW) sensor. Results highlight how ViSual\_IceD is a useful tool to use in conjunction with PMW data, particularly in coastal regions. As the spatial-temporal coverage of MSI and SAR imagery continues to increase, ViSual\_IceD provides a new opportunity for robust, accurate sea ice coverage detection in polar regions.

• 摘要：

  合成孔径雷达 (SAR) 图像是用于海冰测绘的主要数据类型，因为它具有时空覆盖范围，并且能够独立于云和照明条件检测海冰。由于图像中存在模糊信号和噪声，使用 SAR 图像进行自动海冰检测仍然存在问题。相反，使用多光谱图像 (MSI) 可以轻松区分冰和水，但在极地地区，海洋表面经常被云遮挡，或者太阳可能好几个月都不会出现在地平线上方。为了解决其中的一些限制，本文提出了一种使用并发多光谱可见光和 SAR 图像进行海冰检测训练的新工具 (ViSual\_IceD)。 ViSual\_IceD 是一种基于经典 U-Net 架构的卷积神经网络 (CNN)，包含两个并行编码器级，能够融合和串联包含不同空间分辨率的 MSI 和 SAR 图像。将 ViSual\_IceD 的性能与使用串联 MSI 和 SAR 图像训练的 U-Net 模型以及专门在 MSI 或 SAR 图像上训练的模型进行比较。 ViSual\_IceD 优于其他网络，F1 分数比次优网络高 1.60\% 个点，结果表明 ViSual\_IceD 在图像分割过程中使用的图像类型具有选择性。 ViSualIceD 的输出与 AMSR2 无源微波 (PMW) 传感器的海冰浓度产品进行比较。结果凸显了 ViSual\_IceD 是如何与 PMW 数据结合使用的有用工具，特别是在沿海地区。随着 MSI 和 SAR 图像的时空覆盖范围不断增加，ViSual\_IceD 为极地地区稳健、准确的海冰覆盖范围检测提供了新的机会。

Title: Evidence to Generate (E2G): A Single-agent Two-step Prompting for Context Grounded and Retrieval Augmented Reasoning. (arXiv:2401.05787v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.05787
• Code URL: null
• Copy Paste: [[2401.05787]] Evidence to Generate (E2G): A Single-agent Two-step Prompting for Context Grounded and Retrieval Augmented Reasoning(http://arxiv.org/abs/2401.05787)
• Summary:

  While chain-of-thought (CoT) prompting has revolutionized how LLMs perform reasoning tasks, its current methods and variations (e.g, Self-consistency, ReACT, Reflexion, Tree-of-Thoughts (ToT), Cumulative Reasoning (CR)) suffer from limitations like slowness, limited context grounding, hallucination and inconsistent outputs. To overcome these challenges, we introduce Evidence to Generate (E2G), a novel single-agent, two-step prompting framework. Instead of unverified reasoning claims, this innovative approach leverages the power of "evidence for decision making" by first focusing exclusively on the thought sequences (the series of intermediate steps) explicitly mentioned in the context which then serve as extracted evidence, guiding the LLM's output generation process with greater precision and efficiency. This simple yet powerful approach unlocks the true potential of chain-of-thought like prompting, paving the way for faster, more reliable, and more contextually aware reasoning in LLMs. \tool achieves remarkable results robustly across a wide range of knowledge-intensive reasoning and generation tasks, surpassing baseline approaches with state-of-the-art LLMs. For example, (i) on LogiQA benchmark using GPT-4 as backbone model, \tool achieves a new state-of-the Accuracy of 53.8% exceeding CoT by 18%, ToT by 11%, CR by 9% (ii) a variant of E2G with PaLM2 outperforms the variable-shot performance of Gemini Ultra by 0.9 F1 points, reaching an F1 score of 83.3 on a subset of DROP.

• 摘要：

  虽然思想链 (CoT) 提示彻底改变了法学硕士执行推理任务的方式，但其当前的方法和变体（例如，自我一致性、反应、反思、思想树 (ToT)、累积推理 (CR) )) 受到诸如缓慢、有限的背景基础、幻觉和不一致的输出等限制。为了克服这些挑战，我们引入了证据生成（E2G），这是一种新颖的单代理、两步提示框架。这种创新方法不是未经验证的推理主张，而是利用“决策证据”的力量，首先专门关注上下文中明确提到的思维序列（一系列中间步骤），然后将其作为提取的证据，指导法学硕士的输出生成过程具有更高的精度和效率。这种简单而强大的方法释放了思维链（如提示）的真正潜力，为法学硕士中更快、更可靠、更上下文相关的推理铺平了道路。 \tool 在广泛的知识密集型推理和生成任务中取得了显着的成果，超越了最先进的法学硕士的基线方法。例如，(i) 在使用 GPT-4 作为骨干模型的 LogiQA 基准测试中，工具达到了 53.8% 的新准确率，CoT 超出 18%，ToT 超出 11%，CR 超出 9% (ii)带有 PaLM2 的 E2G 变体比 Gemini Ultra 的可变镜头性能高出 0.9 F1 分，在 DROP 子集上达到 83.3 的 F1 分数。

Title: Enhancing Personality Recognition in Dialogue by Data Augmentation and Heterogeneous Conversational Graph Networks. (arXiv:2401.05871v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.05871
• Code URL: https://github.com/fuyahuii/personality-recognition-on-realpersonachat
• Copy Paste: [[2401.05871]] Enhancing Personality Recognition in Dialogue by Data Augmentation and Heterogeneous Conversational Graph Networks(http://arxiv.org/abs/2401.05871)
• Summary:

  Personality recognition is useful for enhancing robots' ability to tailor user-adaptive responses, thus fostering rich human-robot interactions. One of the challenges in this task is a limited number of speakers in existing dialogue corpora, which hampers the development of robust, speaker-independent personality recognition models. Additionally, accurately modeling both the interdependencies among interlocutors and the intra-dependencies within the speaker in dialogues remains a significant issue. To address the first challenge, we introduce personality trait interpolation for speaker data augmentation. For the second, we propose heterogeneous conversational graph networks to independently capture both contextual influences and inherent personality traits. Evaluations on the RealPersonaChat corpus demonstrate our method's significant improvements over existing baselines.

• 摘要：

  个性识别有助于增强机器人定制用户自适应响应的能力，从而促进丰富的人机交互。这项任务的挑战之一是现有对话语料库中说话者的数量有限，这阻碍了稳健的、独立于说话者的个性识别模型的开发。此外，准确地建模对话者之间的相互依赖关系和对话中发言者内部的依赖关系仍然是一个重要的问题。为了解决第一个挑战，我们引入了用于说话者数据增强的人格特质插值。对于第二个，我们提出异构会话图网络来独立捕获上下文影响和固有的个性特征。对 RealPersonaChat 语料库的评估表明我们的方法比现有基线有了显着改进。

Title: LLM-as-a-Coauthor: The Challenges of Detecting LLM-Human Mixcase. (arXiv:2401.05952v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.05952
• Code URL: https://github.com/dongping-chen/mixset
• Copy Paste: [[2401.05952]] LLM-as-a-Coauthor: The Challenges of Detecting LLM-Human Mixcase(http://arxiv.org/abs/2401.05952)
• Summary:

  With the remarkable development and widespread applications of large language models (LLMs), the use of machine-generated text (MGT) is becoming increasingly common. This trend brings potential risks, particularly to the quality and completeness of information in fields such as news and education. Current research predominantly addresses the detection of pure MGT without adequately addressing mixed scenarios including AI-revised Human-Written Text (HWT) or human-revised MGT. To confront this challenge, we introduce mixcase, a novel concept representing a hybrid text form involving both machine-generated and human-generated content. We collected mixcase instances generated from multiple daily text-editing scenarios and composed MixSet, the first dataset dedicated to studying these mixed modification scenarios. We conduct experiments to evaluate the efficacy of popular MGT detectors, assessing their effectiveness, robustness, and generalization performance. Our findings reveal that existing detectors struggle to identify mixcase as a separate class or MGT, particularly in dealing with subtle modifications and style adaptability. This research underscores the urgent need for more fine-grain detectors tailored for mixcase, offering valuable insights for future research. Code and Models are available at https://github.com/Dongping-Chen/MixSet.

• 摘要：

  随着大型语言模型 (LLM) 的显着发展和广泛应用，机器生成文本 (MGT) 的使用变得越来越普遍。这种趋势带来了潜在的风险，尤其是新闻、教育等领域信息的质量和完整性。目前的研究主要针对纯 MGT 的检测，而没有充分解决混合场景，包括人工智能修订的人类书写文本 (HWT) 或人类修订的 MGT。为了应对这一挑战，我们引入了 mixcase，这是一个代表混合文本形式的新颖概念，涉及机器生成和人类生成的内容。我们收集了从多个日常文本编辑场景生成的 mixcase 实例，并组成了 MixSet，这是第一个致力于研究这些混合修改场景的数据集。我们进行实验来评估流行的 MGT 检测器的功效，评估其有效性、鲁棒性和泛化性能。我们的研究结果表明，现有的检测器很难将 mixcase 识别为单独的类或 MGT，特别是在处理细微的修改和风格适应性方面。这项研究强调了对针对混合情况定制的更多细粒度检测器的迫切需求，为未来的研究提供了宝贵的见解。代码和模型可在 https://github.com/Dongping-Chen/MixSet 获取。

Title: Machine Teaching for Building Modular AI Agents based on Zero-shot Learners. (arXiv:2401.05467v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.05467
• Code URL: null
• Copy Paste: [[2401.05467]] Machine Teaching for Building Modular AI Agents based on Zero-shot Learners(http://arxiv.org/abs/2401.05467)
• Summary:

  The recent advances in large language models (LLMs) have led to the creation of many modular AI agents. These agents employ LLMs as zero-shot learners to perform sub-tasks in order to solve complex tasks set forth by human users. We propose an approach to enhance the robustness and performance of modular AI agents that utilize LLMs as zero-shot learners. Our iterative machine teaching method offers an efficient way to teach AI agents over time with limited human feedback, addressing the limit posed by the quality of zero-shot learning. We advocate leveraging the data traces from initial deployments and outputs or annotations from the zero-shot learners to train smaller and task-specific substitute models which can reduce both the monetary costs and environmental impact. Our machine teaching process avails human expertise to correct examples with a high likelihood of misannotations. Results on three tasks, common to conversational AI agents, show that close-to-oracle performance can be achieved with supervision on 20-70% of the dataset depending upon the complexity of the task and performance of zero-shot learners.

• 摘要：

  大型语言模型 (LLM) 的最新进展催生了许多模块化人工智能代理的创建。这些代理使用 LLM 作为零样本学习器来执行子任务，以解决人类用户提出的复杂任务。我们提出了一种方法来增强模块化人工智能代理的鲁棒性和性能，该代理利用法学硕士作为零样本学习者。我们的迭代机器教学方法提供了一种有效的方法，可以在有限的人类反馈的情况下随着时间的推移教授人工智能代理，解决零样本学习质量带来的限制。我们主张利用初始部署的数据跟踪和零样本学习者的输出或注释来训练更小的、特定于任务的替代模型，这可以减少金钱成本和环境影响。我们的机器教学过程利用人类专业知识来纠正极有可能出现错误注释的示例。对话式 AI 代理常见的三个任务的结果表明，根据任务的复杂性和零样本学习者的性能，通过对 20-70% 的数据集进行监督，可以实现接近预言机的性能。

Title: Fast Cerebral Blood Flow Analysis via Extreme Learning Machine. (arXiv:2401.05578v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.05578
• Code URL: null
• Copy Paste: [[2401.05578]] Fast Cerebral Blood Flow Analysis via Extreme Learning Machine(http://arxiv.org/abs/2401.05578)
• Summary:

  We introduce a rapid and precise analytical approach for analyzing cerebral blood flow (CBF) using Diffuse Correlation Spectroscopy (DCS) with the application of the Extreme Learning Machine (ELM). Our evaluation of ELM and existing algorithms involves a comprehensive set of metrics. We assess these algorithms using synthetic datasets for both semi-infinite and multi-layer models. The results demonstrate that ELM consistently achieves higher fidelity across various noise levels and optical parameters, showcasing robust generalization ability and outperforming iterative fitting algorithms. Through a comparison with a computationally efficient neural network, ELM attains comparable accuracy with reduced training and inference times. Notably, the absence of a back-propagation process in ELM during training results in significantly faster training speeds compared to existing neural network approaches. This proposed strategy holds promise for edge computing applications with online training capabilities.

• 摘要：

  我们引入了一种快速、精确的分析方法，利用漫射相关光谱 (DCS) 并应用极限学习机 (ELM) 来分析脑血流 (CBF)。我们对 ELM 和现有算法的评估涉及一套全面的指标。我们使用半无限和多层模型的合成数据集来评估这些算法。结果表明，ELM 在各种噪声水平和光学参数上始终实现更高的保真度，展现出强大的泛化能力并优于迭代拟合算法。通过与计算效率高的神经网络进行比较，ELM 在减少训练和推理时间的情况下获得了相当的精度。值得注意的是，与现有的神经网络方法相比，ELM 在训练过程中没有反向传播过程，因此训练速度明显加快。该策略为具有在线培训功能的边缘计算应用带来了希望。

Title: Enhancing Blood Flow Assessment in Diffuse Correlation Spectroscopy: A Transfer Learning Approach with Noise Robustness Analysis. (arXiv:2401.05580v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.05580
• Code URL: null
• Copy Paste: [[2401.05580]] Enhancing Blood Flow Assessment in Diffuse Correlation Spectroscopy: A Transfer Learning Approach with Noise Robustness Analysis(http://arxiv.org/abs/2401.05580)
• Summary:

  Diffuse correlation spectroscopy (DCS) is an emerging noninvasive technique that measures the tissue blood flow, by using near-infrared coherent point-source illumination to detect spectral changes. While machine learning has demonstrated significant potential for measuring blood flow index (BFi), an open question concerning the success of this approach pertains to its robustness in scenarios involving deviations between datasets with varying Signal-to-Noise Ratios (SNRs) originating from diverse clinical applications and various setups. This study proposes a transfer learning approach, aims to assess the influence of SNRs on the generalization ability of learned features, and demonstrate the robustness for transfer learning. A synthetic dataset with varying levels of added noise is utilized to simulate different SNRs. The proposed network takes a 1x64 autocorrelation curve as input and generates BFi and the correlation parameter beta. The proposed model demonstrates excellent performance across different SNRs, exhibiting enhanced fitting accuracy, particularly for low SNR datasets when compared with other fitting methods. This highlights its potential for clinical diagnosis and treatment across various scenarios under different clinical setups.

• 摘要：

  漫相关光谱 (DCS) 是一种新兴的无创技术，通过使用近红外相干点源照明来检测光谱变化来测量组织血流。虽然机器学习已显示出测量血流指数 (BFi) 的巨大潜力，但有关该方法成功与否的一个悬而未决的问题涉及其在涉及来自不同临床的不同信噪比 (SNR) 的数据集之间存在偏差的场景中的鲁棒性。应用程序和各种设置。本研究提出了一种迁移学习方法，旨在评估信噪比对学习特征泛化能力的影响，并证明迁移学习的鲁棒性。利用具有不同添加噪声水平的合成数据集来模拟不同的 SNR。所提出的网络以 1x64 自相关曲线作为输入并生成 BFi 和相关参数 beta。所提出的模型在不同的信噪比下表现出优异的性能，与其他拟合方法相比，显示出更高的拟合精度，特别是对于低信噪比数据集。这凸显了其在不同临床设置下的各种场景下的临床诊断和治疗的潜力。

Title: Dynamic Indoor Fingerprinting Localization based on Few-Shot Meta-Learning with CSI Images. (arXiv:2401.05711v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.05711
• Code URL: null
• Copy Paste: [[2401.05711]] Dynamic Indoor Fingerprinting Localization based on Few-Shot Meta-Learning with CSI Images(http://arxiv.org/abs/2401.05711)
• Summary:

  While fingerprinting localization is favored for its effectiveness, it is hindered by high data acquisition costs and the inaccuracy of static database-based estimates. Addressing these issues, this letter presents an innovative indoor localization method using a data-efficient meta-learning algorithm. This approach, grounded in the ``Learning to Learn'' paradigm of meta-learning, utilizes historical localization tasks to improve adaptability and learning efficiency in dynamic indoor environments. We introduce a task-weighted loss to enhance knowledge transfer within this framework. Our comprehensive experiments confirm the method's robustness and superiority over current benchmarks, achieving a notable 23.13\% average gain in Mean Euclidean Distance, particularly effective in scenarios with limited CSI data.

• 摘要：

  虽然指纹定位因其有效性而受到青睐，但它受到高昂的数据获取成本和基于静态数据库的估计不准确的阻碍。为了解决这些问题，这封信提出了一种使用数据高效元学习算法的创新室内定位方法。这种方法基于元学习的“学会学习”范式，利用历史定位任务来提高动态室内环境中的适应性和学习效率。我们引入任务加权损失来增强该框架内的知识转移。我们的综合实验证实了该方法相对于当前基准的稳健性和优越性，在平均欧几里德距离方面实现了 23.13% 的平均增益，在 CSI 数据有限的情况下尤其有效。

Title: An experimental evaluation of Deep Reinforcement Learning algorithms for HVAC control. (arXiv:2401.05737v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.05737
• Code URL: https://github.com/ugr-sail/paper-drl_building
• Copy Paste: [[2401.05737]] An experimental evaluation of Deep Reinforcement Learning algorithms for HVAC control(http://arxiv.org/abs/2401.05737)
• Summary:

  Heating, Ventilation, and Air Conditioning (HVAC) systems are a major driver of energy consumption in commercial and residential buildings. Recent studies have shown that Deep Reinforcement Learning (DRL) algorithms can outperform traditional reactive controllers. However, DRL-based solutions are generally designed for ad hoc setups and lack standardization for comparison. To fill this gap, this paper provides a critical and reproducible evaluation, in terms of comfort and energy consumption, of several state-of-the-art DRL algorithms for HVAC control. The study examines the controllers' robustness, adaptability, and trade-off between optimization goals by using the Sinergym framework. The results obtained confirm the potential of DRL algorithms, such as SAC and TD3, in complex scenarios and reveal several challenges related to generalization and incremental learning.

• 摘要：

  供暖、通风和空调 (HVAC) 系统是商业和住宅建筑能源消耗的主要驱动因素。最近的研究表明，深度强化学习（DRL）算法可以超越传统的反应控制器。然而，基于 DRL 的解决方案通常是为临时设置而设计的，缺乏用于比较的标准化。为了填补这一空白，本文在舒适度和能耗方面对几种最先进的 HVAC 控制 DRL 算法进行了关键且可重复的评估。该研究使用 Sinergym 框架检查控制器的鲁棒性、适应性以及优化目标之间的权衡。获得的结果证实了 SAC 和 TD3 等 DRL 算法在复杂场景中的潜力，并揭示了与泛化和增量学习相关的一些挑战。

Title: Revisiting Silhouette: From Micro to Macro Aggregation. (arXiv:2401.05831v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.05831
• Code URL: null
• Copy Paste: [[2401.05831]] Revisiting Silhouette: From Micro to Macro Aggregation(http://arxiv.org/abs/2401.05831)
• Summary:

  Silhouette coefficient is an established internal clustering evaluation measure that produces a score per data point, assessing the quality of its clustering assignment. To assess the quality of the clustering of the whole dataset, the scores of all the points in the dataset are typically averaged into a single value, a strategy which we call as micro-averaging. As we illustrate in this work, by using a synthetic example, this micro-averaging strategy is sensitive both to cluster imbalance and outliers (background noise). To address these issues, we propose an alternative aggregation strategy, which first averages the silhouette scores at a cluster level and then (macro) averages the scores across the clusters. Based on the same synthetic example, we show that the proposed macro-averaged silhouette score is robust to cluster imbalance and background noise. We have conducted an experimental study showing that our macro-averaged variant provides better estimates of the ground truth number of clusters on several cases compared to the typical micro-averaged score.

• 摘要：

  轮廓系数是一种既定的内部聚类评估措施，它为每个数据点生成一个分数，评估其聚类分配的质量。为了评估整个数据集的聚类质量，通常将数据集中所有点的分数平均为一个值，我们将这种策略称为微平均。正如我们在这项工作中所说明的，通过使用综合示例，这种微平均策略对集群不平衡和异常值（背景噪声）都很敏感。为了解决这些问题，我们提出了一种替代聚合策略，该策略首先在集群级别上平均轮廓分数，然后（宏观）平均跨集群的分数。基于相同的合成示例，我们表明所提出的宏观平均轮廓分数对于集群不平衡和背景噪声具有鲁棒性。我们进行了一项实验研究，表明与典型的微观平均得分相比，我们的宏观平均变体可以更好地估计几种情况下的集群的真实数量。

biometric

steal

extraction

Title: MatSAM: Efficient Materials Microstructure Extraction via Visual Large Model. (arXiv:2401.05638v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.05638
• Code URL: null
• Copy Paste: [[2401.05638]] MatSAM: Efficient Materials Microstructure Extraction via Visual Large Model(http://arxiv.org/abs/2401.05638)
• Summary:

  Accurate and efficient extraction of microstructures in microscopic images of materials plays a critical role in the exploration of structure-property relationships and the optimization of process parameters. Deep learning-based image segmentation techniques that rely on manual annotation are time-consuming and labor-intensive and hardly meet the demand for model transferability and generalization. Segment Anything Model (SAM), a large visual model with powerful deep feature representation and zero-shot generalization capabilities, has provided new solutions for image segmentation. However, directly applying SAM to segmenting microstructures in microscopic images of materials without human annotation cannot achieve the expected results, as the difficulty of adapting its native prompt engineering to the dense and dispersed characteristics of key microstructures in materials microscopy images. In this paper, we propose MatSAM, a general and efficient microstructure extraction solution based on SAM. A new point-based prompts generation strategy is designed, grounded on the distribution and shape of materials microstructures. It generates prompts for different microscopic images, fuses the prompts of the region of interest (ROI) key points and grid key points, and integrates post-processing methods for quantitative characterization of materials microstructures. For common microstructures including grain boundary and phase, MatSAM achieves superior segmentation performance to conventional methods and is even preferable to supervised learning methods evaluated on 18 materials microstructures imaged by the optical microscope (OM) and scanning electron microscope (SEM). We believe that MatSAM can significantly reduce the cost of quantitative characterization of materials microstructures and accelerate the design of new materials.

• 摘要：

  准确高效地提取材料显微图像中的微观结构对于探索结构-性能关系和优化工艺参数起着至关重要的作用。基于深度学习的图像分割技术依赖于人工标注，耗时耗力，难以满足模型可迁移性和泛化性的需求。 Segment Anything Model（SAM）是一种大型视觉模型，具有强大的深度特征表示和零样本泛化能力，为图像分割提供了新的解决方案。然而，直接应用SAM在没有人工标注的情况下分割材料显微图像中的微观结构并不能达到预期的结果，因为其原生即时工程难以适应材料显微图像中关键微观结构的密集和分散特征。在本文中，我们提出了 MatSAM，一种基于 SAM 的通用且高效的微观结构提取解决方案。基于材料微观结构的分布和形状，设计了一种新的基于点的提示生成策略。它针对不同的显微图像生成提示，融合感兴趣区域（ROI）关键点和网格关键点的提示，并集成用于材料微观结构定量表征的后处理方法。对于包括晶界和相在内的常见微观结构，MatSAM 实现了优于传统方法的分割性能，甚至优于对光学显微镜 (OM) 和扫描电子显微镜 (SEM) 成像的 18 种材料微观结构进行评估的监督学习方法。我们相信MatSAM可以显着降低材料微观结构定量表征的成本并加速新材料的设计。

Title: Self Expanding Convolutional Neural Networks. (arXiv:2401.05686v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.05686
• Code URL: null
• Copy Paste: [[2401.05686]] Self Expanding Convolutional Neural Networks(http://arxiv.org/abs/2401.05686)
• Summary:

  In this paper, we present a novel method for dynamically expanding Convolutional Neural Networks (CNNs) during training, aimed at meeting the increasing demand for efficient and sustainable deep learning models. Our approach, drawing from the seminal work on Self-Expanding Neural Networks (SENN), employs a natural expansion score as an expansion criteria to address the common issue of over-parameterization in deep convolutional neural networks, thereby ensuring that the model's complexity is finely tuned to the task's specific needs. A significant benefit of this method is its eco-friendly nature, as it obviates the necessity of training multiple models of different sizes. We employ a strategy where a single model is dynamically expanded, facilitating the extraction of checkpoints at various complexity levels, effectively reducing computational resource use and energy consumption while also expediting the development cycle by offering diverse model complexities from a single training session. We evaluate our method on the CIFAR-10 dataset and our experimental results validate this approach, demonstrating that dynamically adding layers not only maintains but also improves CNN performance, underscoring the effectiveness of our expansion criteria. This approach marks a considerable advancement in developing adaptive, scalable, and environmentally considerate neural network architectures, addressing key challenges in the field of deep learning.

• 摘要：

  在本文中，我们提出了一种在训练期间动态扩展卷积神经网络（CNN）的新方法，旨在满足对高效和可持续深度学习模型日益增长的需求。我们的方法借鉴了自扩展神经网络（SENN）的开创性工作，采用自然扩展分数作为扩展标准来解决深度卷积神经网络中过度参数化的常见问题，从而确保模型的复杂性很好根据任务的具体需求进行调整。这种方法的一个显着好处是它的环保性质，因为它消除了训练不同大小的多个模型的必要性。我们采用动态扩展单个模型的策略，有助于提取各种复杂程度的检查点，有效减少计算资源使用和能源消耗，同时还通过从单个训练会话中提供不同的模型复杂性来加快开发周期。我们在 CIFAR-10 数据集上评估我们的方法，我们的实验结果验证了这种方法，证明动态添加层不仅可以保持而且还可以提高 CNN 性能，强调了我们扩展标准的有效性。这种方法标志着在开发自适应、可扩展和环境友好的神经网络架构方面取得了相当大的进步，解决了深度学习领域的关键挑战。

Title: Learning Generalizable Models via Disentangling Spurious and Enhancing Potential Correlations. (arXiv:2401.05752v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.05752
• Code URL: null
• Copy Paste: [[2401.05752]] Learning Generalizable Models via Disentangling Spurious and Enhancing Potential Correlations(http://arxiv.org/abs/2401.05752)
• Summary:

  Domain generalization (DG) intends to train a model on multiple source domains to ensure that it can generalize well to an arbitrary unseen target domain. The acquisition of domain-invariant representations is pivotal for DG as they possess the ability to capture the inherent semantic information of the data, mitigate the influence of domain shift, and enhance the generalization capability of the model. Adopting multiple perspectives, such as the sample and the feature, proves to be effective. The sample perspective facilitates data augmentation through data manipulation techniques, whereas the feature perspective enables the extraction of meaningful generalization features. In this paper, we focus on improving the generalization ability of the model by compelling it to acquire domain-invariant representations from both the sample and feature perspectives by disentangling spurious correlations and enhancing potential correlations. 1) From the sample perspective, we develop a frequency restriction module, guiding the model to focus on the relevant correlations between object features and labels, thereby disentangling spurious correlations. 2) From the feature perspective, the simple Tail Interaction module implicitly enhances potential correlations among all samples from all source domains, facilitating the acquisition of domain-invariant representations across multiple domains for the model. The experimental results show that Convolutional Neural Networks (CNNs) or Multi-Layer Perceptrons (MLPs) with a strong baseline embedded with these two modules can achieve superior results, e.g., an average accuracy of 92.30% on Digits-DG.

• 摘要：

  域泛化（DG）旨在在多个源域上训练模型，以确保它能够很好地泛化到任意未见过的目标域。域不变表示的获取对于 DG 至关重要，因为它们能够捕获数据的固有语义信息，减轻域转移的影响并增强模型的泛化能力。采用样本和特征等多个视角被证明是有效的。样本视角通过数据操作技术促进数据增强，而特征视角则能够提取有意义的泛化特征。在本文中，我们致力于通过解开虚假相关性并增强潜在相关性，迫使模型从样本和特征角度获取域不变表示，从而提高模型的泛化能力。 1）从样本角度，我们开发了频率限制模块，引导模型关注对象特征和标签之间的相关相关性，从而消除虚假相关性。 2）从特征角度来看，简单的尾部交互模块隐式增强了来自所有源域的所有样本之间的潜在相关性，有助于模型获取跨多个域的域不变表示。实验结果表明，嵌入这两个模块的具有强大基线的卷积神经网络（CNN）或多层感知器（MLP）可以取得优异的结果，例如在 Digits-DG 上的平均准确率达到 92.30%。

Title: YOIO: You Only Iterate Once by mining and fusing multiple necessary global information in the optical flow estimation. (arXiv:2401.05879v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.05879
• Code URL: null
• Copy Paste: [[2401.05879]] YOIO: You Only Iterate Once by mining and fusing multiple necessary global information in the optical flow estimation(http://arxiv.org/abs/2401.05879)
• Summary:

  Occlusions pose a significant challenge to optical flow algorithms that even rely on global evidences. We consider an occluded point to be one that is imaged in the reference frame but not in the next. Estimating the motion of these points is extremely difficult, particularly in the two-frame setting. Previous work only used the current frame as the only input, which could not guarantee providing correct global reference information for occluded points, and had problems such as long calculation time and poor accuracy in predicting optical flow at occluded points. To enable both high accuracy and efficiency, We fully mine and utilize the spatiotemporal information provided by the frame pair, design a loopback judgment algorithm to ensure that correct global reference information is obtained, mine multiple necessary global information, and design an efficient refinement module that fuses these global information. Specifically, we propose a YOIO framework, which consists of three main components: an initial flow estimator, a multiple global information extraction module, and a unified refinement module. We demonstrate that optical flow estimates in the occluded regions can be significantly improved in only one iteration without damaging the performance in non-occluded regions. Compared with GMA, the optical flow prediction accuracy of this method in the occluded area is improved by more than 10%, and the occ_out area exceeds 15%, while the calculation time is 27% shorter. This approach, running up to 18.9fps with 436*1024 image resolution, obtains new state-of-the-art results on the challenging Sintel dataset among all published and unpublished approaches that can run in real-time, suggesting a new paradigm for accurate and efficient optical flow estimation.

• 摘要：

  遮挡对甚至依赖全局证据的光流算法提出了重大挑战。我们认为遮挡点是在参考帧中成像但不在下一帧中成像的点。估计这些点的运动非常困难，特别是在两帧设置中。以往的工作仅以当前帧作为唯一输入，无法保证为遮挡点提供正确的全局参考信息，存在计算时间长、遮挡点光流预测精度差等问题。为了兼顾高精度和高效率，我们充分挖掘和利用帧对提供的时空信息，设计环回判断算法以确保获得正确的全局参考信息，挖掘多个必要的全局信息，并设计高效的细化模块融合这些全局信息。具体来说，我们提出了一个YOIO框架，它由三个主要组件组成：初始流量估计器、多个全局信息提取模块和统一细化模块。我们证明，遮挡区域中的光流估计只需一次迭代即可显着改善，而不会损害非遮挡区域的性能。与GMA相比，该方法在遮挡区域的光流预测精度提高了10%以上，occ_out区域超过15%，同时计算时间缩短了27%。这种方法的运行速度高达 18.9fps，图像分辨率为 436*1024，在所有已发表和未发表的可实时运行的方法中，在具有挑战性的 Sintel 数据集上获得了最新的结果，这为精确计算提供了一种新的范式。和高效的光流估计。

Title: Automatic UAV-based Airport Pavement Inspection Using Mixed Real and Virtual Scenarios. (arXiv:2401.06019v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.06019
• Code URL: null
• Copy Paste: [[2401.06019]] Automatic UAV-based Airport Pavement Inspection Using Mixed Real and Virtual Scenarios(http://arxiv.org/abs/2401.06019)
• Summary:

  Runway and taxiway pavements are exposed to high stress during their projected lifetime, which inevitably leads to a decrease in their condition over time. To make sure airport pavement condition ensure uninterrupted and resilient operations, it is of utmost importance to monitor their condition and conduct regular inspections. UAV-based inspection is recently gaining importance due to its wide range monitoring capabilities and reduced cost. In this work, we propose a vision-based approach to automatically identify pavement distress using images captured by UAVs. The proposed method is based on Deep Learning (DL) to segment defects in the image. The DL architecture leverages the low computational capacities of embedded systems in UAVs by using an optimised implementation of EfficientNet feature extraction and Feature Pyramid Network segmentation. To deal with the lack of annotated data for training we have developed a synthetic dataset generation methodology to extend available distress datasets. We demonstrate that the use of a mixed dataset composed of synthetic and real training images yields better results when testing the training models in real application scenarios.

• 摘要：

  跑道和滑行道路面在其预计使用寿命期间会承受高压力，这不可避免地会导致其状况随着时间的推移而恶化。为了确保机场路面状况确保不间断和弹性运行，监测其状况并进行定期检查至关重要。基于无人机的检查由于其广泛的监控能力和降低的成本而最近变得越来越重要。在这项工作中，我们提出了一种基于视觉的方法，使用无人机捕获的图像自动识别路面破损。所提出的方法基于深度学习（DL）来分割图像中的缺陷。深度学习架构通过使用 EfficientNet 特征提取和特征金字塔网络分割的优化实现，利用了无人机嵌入式系统的低计算能力。为了解决训练注释数据的缺乏问题，我们开发了一种合成数据集生成方法来扩展可用的遇险数据集。我们证明，在实际应用场景中测试训练模型时，使用由合成图像和真实训练图像组成的混合数据集可以产生更好的结果。

membership infer

federate

fair

Title: Unveiling the Tapestry of Automated Essay Scoring: A Comprehensive Investigation of Accuracy, Fairness, and Generalizability. (arXiv:2401.05655v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.05655
• Code URL: null
• Copy Paste: [[2401.05655]] Unveiling the Tapestry of Automated Essay Scoring: A Comprehensive Investigation of Accuracy, Fairness, and Generalizability(http://arxiv.org/abs/2401.05655)
• Summary:

  Automatic Essay Scoring (AES) is a well-established educational pursuit that employs machine learning to evaluate student-authored essays. While much effort has been made in this area, current research primarily focuses on either (i) boosting the predictive accuracy of an AES model for a specific prompt (i.e., developing prompt-specific models), which often heavily relies on the use of the labeled data from the same target prompt; or (ii) assessing the applicability of AES models developed on non-target prompts to the intended target prompt (i.e., developing the AES models in a cross-prompt setting). Given the inherent bias in machine learning and its potential impact on marginalized groups, it is imperative to investigate whether such bias exists in current AES methods and, if identified, how it intervenes with an AES model's accuracy and generalizability. Thus, our study aimed to uncover the intricate relationship between an AES model's accuracy, fairness, and generalizability, contributing practical insights for developing effective AES models in real-world education. To this end, we meticulously selected nine prominent AES methods and evaluated their performance using seven metrics on an open-sourced dataset, which contains over 25,000 essays and various demographic information about students such as gender, English language learner status, and economic status. Through extensive evaluations, we demonstrated that: (1) prompt-specific models tend to outperform their cross-prompt counterparts in terms of predictive accuracy; (2) prompt-specific models frequently exhibit a greater bias towards students of different economic statuses compared to cross-prompt models; (3) in the pursuit of generalizability, traditional machine learning models coupled with carefully engineered features hold greater potential for achieving both high accuracy and fairness than complex neural network models.

• 摘要：

  论文自动评分 (AES) 是一项成熟的教育活动，它利用机器学习来评估学生撰写的论文。虽然在这一领域做出了很多努力，但当前的研究主要集中在 (i) 提高 AES 模型对特定提示的预测准确性（即开发特定于提示的模型），这通常严重依赖于使用来自同一目标提示的标记数据； (ii) 评估在非目标提示上开发的 AES 模型对预期目标提示的适用性（即，在交叉提示设置中开发 AES 模型）。考虑到机器学习的固有偏差及其对边缘群体的潜在影响，有必要研究当前 AES 方法中是否存在这种偏差，如果存在，它如何影响 AES 模型的准确性和普遍性。因此，我们的研究旨在揭示 AES 模型的准确性、公平性和普遍性之间的复杂关系，为在现实教育中开发有效的 AES 模型提供实用见解。为此，我们精心挑选了九种著名的 AES 方法，并使用开源数据集的七个指标评估了它们的性能，该数据集包含超过 25,000 篇论文以及有关学生的各种人口统计信息，例如性别、英语学习者状况和经济状况。通过广泛的评估，我们证明：（1）特定提示模型在预测准确性方面往往优于交叉提示模型； （2）与交叉提示模型相比，特定提示模型经常对不同经济状况的学生表现出更大的偏见； （3）为了追求普遍性，传统的机器学习模型加上精心设计的特征，比复杂的神经网络模型具有更大的潜力来实现高精度和公平性。

interpretability

Title: Patchscope: A Unifying Framework for Inspecting Hidden Representations of Language Models. (arXiv:2401.06102v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.06102
• Code URL: null
• Copy Paste: [[2401.06102]] Patchscope: A Unifying Framework for Inspecting Hidden Representations of Language Models(http://arxiv.org/abs/2401.06102)
• Summary:

  Inspecting the information encoded in hidden representations of large language models (LLMs) can explain models' behavior and verify their alignment with human values. Given the capabilities of LLMs in generating human-understandable text, we propose leveraging the model itself to explain its internal representations in natural language. We introduce a framework called Patchscopes and show how it can be used to answer a wide range of research questions about an LLM's computation. We show that prior interpretability methods based on projecting representations into the vocabulary space and intervening on the LLM computation, can be viewed as special instances of this framework. Moreover, several of their shortcomings such as failure in inspecting early layers or lack of expressivity can be mitigated by a Patchscope. Beyond unifying prior inspection techniques, Patchscopes also opens up new possibilities such as using a more capable model to explain the representations of a smaller model, and unlocks new applications such as self-correction in multi-hop reasoning.

• 摘要：

  检查大型语言模型 (LLM) 隐藏表示中编码的信息可以解释模型的行为并验证其与人类价值观的一致性。鉴于法学硕士在生成人类可理解的文本方面的能力，我们建议利用模型本身来解释其自然语言的内部表示。我们介绍了一个名为 Patchscopes 的框架，并展示了如何使用它来回答有关法学硕士计算的各种研究问题。我们表明，基于将表示投影到词汇空间并干预 LLM 计算的先前可解释性方法可以被视为该框架的特殊实例。此外，它们的一些缺点，例如无法检查早期层或缺乏表现力，可以通过 Patchscope 来缓解。除了统一先前的检查技术之外，Patchscopes 还开辟了新的可能性，例如使用功能更强大的模型来解释较小模型的表示，并解锁了新的应用程序，例如多跳推理中的自我校正。

Title: Wavelet-Inspired Multiscale Graph Convolutional Recurrent Network for Traffic Forecasting. (arXiv:2401.06040v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.06040
• Code URL: https://github.com/qqian99/wavgcrn
• Copy Paste: [[2401.06040]] Wavelet-Inspired Multiscale Graph Convolutional Recurrent Network for Traffic Forecasting(http://arxiv.org/abs/2401.06040)
• Summary:

  Traffic forecasting is the foundation for intelligent transportation systems. Spatiotemporal graph neural networks have demonstrated state-of-the-art performance in traffic forecasting. However, these methods do not explicitly model some of the natural characteristics in traffic data, such as the multiscale structure that encompasses spatial and temporal variations at different levels of granularity or scale. To that end, we propose a Wavelet-Inspired Graph Convolutional Recurrent Network (WavGCRN) which combines multiscale analysis (MSA)-based method with Deep Learning (DL)-based method. In WavGCRN, the traffic data is decomposed into time-frequency components with Discrete Wavelet Transformation (DWT), constructing a multi-stream input structure; then Graph Convolutional Recurrent networks (GCRNs) are employed as encoders for each stream, extracting spatiotemporal features in different scales; and finally the learnable Inversed DWT and GCRN are combined as the decoder, fusing the information from all streams for traffic metrics reconstruction and prediction. Furthermore, road-network-informed graphs and data-driven graph learning are combined to accurately capture spatial correlation. The proposed method can offer well-defined interpretability, powerful learning capability, and competitive forecasting performance on real-world traffic data sets.

• 摘要：

  交通预测是智能交通系统的基础。时空图神经网络在交通预测方面表现出了最先进的性能。然而，这些方法没有明确地模拟交通数据中的一些自然特征，例如包含不同粒度或尺度级别的空间和时间变化的多尺度结构。为此，我们提出了一种小波启发图卷积循环网络（WavGCRN），它将基于多尺度分析（MSA）的方法与基于深度学习（DL）的方法相结合。在WavGCRN中，通过离散小波变换（DWT）将流量数据分解为时频分量，构建多流输入结构；然后采用图卷积循环网络（GCRN）作为每个流的编码器，提取不同尺度的时空特征；最后将可学习的 Inversed DWT 和 GCRN 组合起来作为解码器，融合来自所有流的信息以进行流量度量重建和预测。此外，将道路网络信息图和数据驱动图学习相结合，以准确捕获空间相关性。所提出的方法可以在现实世界的交通数据集上提供明确的可解释性、强大的学习能力和有竞争力的预测性能。

explainability

watermark

diffusion

Title: From Pampas to Pixels: Fine-Tuning Diffusion Models for Ga\'ucho Heritage. (arXiv:2401.05520v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.05520
• Code URL: null
• Copy Paste: [[2401.05520]] From Pampas to Pixels: Fine-Tuning Diffusion Models for Ga\'ucho Heritage(http://arxiv.org/abs/2401.05520)
• Summary:

  Generative AI has become pervasive in society, witnessing significant advancements in various domains. Particularly in the realm of Text-to-Image (TTI) models, Latent Diffusion Models (LDMs), showcase remarkable capabilities in generating visual content based on textual prompts. This paper addresses the potential of LDMs in representing local cultural concepts, historical figures, and endangered species. In this study, we use the cultural heritage of Rio Grande do Sul (RS), Brazil, as an illustrative case. Our objective is to contribute to the broader understanding of how generative models can help to capture and preserve the cultural and historical identity of regions. The paper outlines the methodology, including subject selection, dataset creation, and the fine-tuning process. The results showcase the images generated, alongside the challenges and feasibility of each concept. In conclusion, this work shows the power of these models to represent and preserve unique aspects of diverse regions and communities.

• 摘要：

  生成式人工智能已在社会中普及，并在各个领域取得了重大进展。特别是在文本到图像 (TTI) 模型领域，潜在扩散模型 (LDM) 展示了基于文本提示生成视觉内容的卓越能力。本文探讨了 LDM 在代表当地文化概念、历史人物和濒危物种方面的潜力。在本研究中，我们以巴西南里奥格兰德州（RS）的文化遗产为例。我们的目标是帮助人们更广泛地理解生成模型如何帮助捕捉和保护地区的文化和历史特征。本文概述了该方法，包括主题选择、数据集创建和微调过程。结果展示了生成的图像，以及每个概念的挑战和可行性。总之，这项工作展示了这些模型代表和保护不同地区和社区独特方面的力量。

Title: Diffusion Priors for Dynamic View Synthesis from Monocular Videos. (arXiv:2401.05583v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.05583
• Code URL: null
• Copy Paste: [[2401.05583]] Diffusion Priors for Dynamic View Synthesis from Monocular Videos(http://arxiv.org/abs/2401.05583)
• Summary:

  Dynamic novel view synthesis aims to capture the temporal evolution of visual content within videos. Existing methods struggle to distinguishing between motion and structure, particularly in scenarios where camera poses are either unknown or constrained compared to object motion. Furthermore, with information solely from reference images, it is extremely challenging to hallucinate unseen regions that are occluded or partially observed in the given videos. To address these issues, we first finetune a pretrained RGB-D diffusion model on the video frames using a customization technique. Subsequently, we distill the knowledge from the finetuned model to a 4D representations encompassing both dynamic and static Neural Radiance Fields (NeRF) components. The proposed pipeline achieves geometric consistency while preserving the scene identity. We perform thorough experiments to evaluate the efficacy of the proposed method qualitatively and quantitatively. Our results demonstrate the robustness and utility of our approach in challenging cases, further advancing dynamic novel view synthesis.

• 摘要：

  动态新颖视图合成旨在捕捉视频中视觉内容的时间演变。现有的方法很难区分运动和结构，特别是在相机姿势与物体运动相比未知或受到限制的情况下。此外，仅利用参考图像的信息，对给定视频中被遮挡或部分观察到的看不见的区域产生幻觉是极具挑战性的。为了解决这些问题，我们首先使用定制技术在视频帧上微调预训练的 RGB-D 扩散模型。随后，我们将微调模型中的知识提炼为包含动态和静态神经辐射场 (NeRF) 组件的 4D 表示。所提出的管道在保留场景身份的同时实现了几何一致性。我们进行了彻底的实验，以定性和定量地评估所提出方法的有效性。我们的结果证明了我们的方法在具有挑战性的情况下的稳健性和实用性，进一步推进了动态新颖的视图合成。

Title: Object-Centric Diffusion for Efficient Video Editing. (arXiv:2401.05735v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.05735
• Code URL: null
• Copy Paste: [[2401.05735]] Object-Centric Diffusion for Efficient Video Editing(http://arxiv.org/abs/2401.05735)
• Summary:

  Diffusion-based video editing have reached impressive quality and can transform either the global style, local structure, and attributes of given video inputs, following textual edit prompts. However, such solutions typically incur heavy memory and computational costs to generate temporally-coherent frames, either in the form of diffusion inversion and/or cross-frame attention. In this paper, we conduct an analysis of such inefficiencies, and suggest simple yet effective modifications that allow significant speed-ups whilst maintaining quality. Moreover, we introduce Object-Centric Diffusion, coined as OCD, to further reduce latency by allocating computations more towards foreground edited regions that are arguably more important for perceptual quality. We achieve this by two novel proposals: i) Object-Centric Sampling, decoupling the diffusion steps spent on salient regions or background, allocating most of the model capacity to the former, and ii) Object-Centric 3D Token Merging, which reduces cost of cross-frame attention by fusing redundant tokens in unimportant background regions. Both techniques are readily applicable to a given video editing model \textit{without} retraining, and can drastically reduce its memory and computational cost. We evaluate our proposals on inversion-based and control-signal-based editing pipelines, and show a latency reduction up to 10x for a comparable synthesis quality.

• 摘要：

  基于扩散的视频编辑已经达到了令人印象深刻的质量，并且可以按照文本编辑提示转换给定视频输入的全局样式、局部结构和属性。然而，这样的解决方案通常会产生大量的内存和计算成本来生成时间相干的帧，无论是采用扩散反转和/或跨帧注意的形式。在本文中，我们对这种低效率进行了分析，并提出了简单而有效的修改建议，可以在保持质量的同时显着提高速度。此外，我们引入了以对象为中心的扩散（被称为 OCD），通过将计算更多地分配给前台编辑区域来进一步减少延迟，这对于感知质量来说可能更重要。我们通过两个新颖的提议来实现这一目标：i) 以对象为中心的采样，解耦在显着区域或背景上花费的扩散步骤，将大部分模型容量分配给前者，以及 ii) 以对象为中心的 3D 令牌合并，这降低了成本通过在不重要的背景区域融合冗余标记来实现跨帧注意力。这两种技术都很容易适用于给定的视频编辑模型\textit{无需}重新训练，并且可以大大减少其内存和计算成本。我们评估了我们关于基于反转和基于控制信号的编辑管道的建议，结果表明，在同等合成质量的情况下，延迟减少了高达 10 倍。

Title: EraseDiff: Erasing Data Influence in Diffusion Models. (arXiv:2401.05779v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.05779
• Code URL: null
• Copy Paste: [[2401.05779]] EraseDiff: Erasing Data Influence in Diffusion Models(http://arxiv.org/abs/2401.05779)
• Summary:

  In response to data protection regulations and the ``right to be forgotten'', in this work, we introduce an unlearning algorithm for diffusion models. Our algorithm equips a diffusion model with a mechanism to mitigate the concerns related to data memorization. To achieve this, we formulate the unlearning problem as a bi-level optimization problem, wherein the outer objective is to preserve the utility of the diffusion model on the remaining data. The inner objective aims to scrub the information associated with forgetting data by deviating the learnable generative process from the ground-truth denoising procedure. To solve the resulting bi-level problem, we adopt a first-order method, having superior practical performance while being vigilant about the diffusion process and solving a bi-level problem therein. Empirically, we demonstrate that our algorithm can preserve the model utility, effectiveness, and efficiency while removing across two widely-used diffusion models and in both conditional and unconditional image generation scenarios. In our experiments, we demonstrate the unlearning of classes, attributes, and even a race from face and object datasets such as UTKFace, CelebA, CelebA-HQ, and CIFAR10.

• 摘要：

  为了响应数据保护法规和“被遗忘权”，在这项工作中，我们引入了一种扩散模型的遗忘算法。我们的算法为扩散模型配备了一种机制，以减轻与数据记忆相关的问题。为了实现这一目标，我们将遗忘问题表述为双层优化问题，其中外部目标是保留扩散模型对剩余数据的效用。内部目标旨在通过使可学习的生成过程偏离真实的去噪过程来清除与遗忘数据相关的信息。为了解决由此产生的双层问题，我们采用一阶方法，具有优越的实用性能，同时警惕扩散过程并解决其中的双层问题。根据经验，我们证明我们的算法可以保留模型的实用性、有效性和效率，同时在两种广泛使用的扩散模型以及条件和无条件图像生成场景中进行删除。在我们的实验中，我们演示了如何从 UTKFace、CelebA、CelebA-HQ 和 CIFAR10 等人脸和对象数据集中忘记类别、属性甚至种族。

Title: HiCAST: Highly Customized Arbitrary Style Transfer with Adapter Enhanced Diffusion Models. (arXiv:2401.05870v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.05870
• Code URL: null
• Copy Paste: [[2401.05870]] HiCAST: Highly Customized Arbitrary Style Transfer with Adapter Enhanced Diffusion Models(http://arxiv.org/abs/2401.05870)
• Summary:

  The goal of Arbitrary Style Transfer (AST) is injecting the artistic features of a style reference into a given image/video. Existing methods usually focus on pursuing the balance between style and content, whereas ignoring the significant demand for flexible and customized stylization results and thereby limiting their practical application. To address this critical issue, a novel AST approach namely HiCAST is proposed, which is capable of explicitly customizing the stylization results according to various source of semantic clues. In the specific, our model is constructed based on Latent Diffusion Model (LDM) and elaborately designed to absorb content and style instance as conditions of LDM. It is characterized by introducing of \textit{Style Adapter}, which allows user to flexibly manipulate the output results by aligning multi-level style information and intrinsic knowledge in LDM. Lastly, we further extend our model to perform video AST. A novel learning objective is leveraged for video diffusion model training, which significantly improve cross-frame temporal consistency in the premise of maintaining stylization strength. Qualitative and quantitative comparisons as well as comprehensive user studies demonstrate that our HiCAST outperforms the existing SoTA methods in generating visually plausible stylization results.

• 摘要：

  任意风格迁移（AST）的目标是将风格参考的艺术特征注入给定的图像/视频中。现有的方法通常注重追求风格和内容之间的平衡，而忽略了对灵活和定制的风格化结果的巨大需求，从而限制了其实际应用。为了解决这个关键问题，提出了一种新的 AST 方法，即 HiCAST，它能够根据各种语义线索来源显式定制样式化结果。具体来说，我们的模型是基于潜在扩散模型（LDM）构建的，并精心设计以吸收内容和风格实例作为LDM的条件。它的特点是引入了\textit{Style Adapter}，允许用户通过对齐LDM中的多级样式信息和内在知识来灵活地操纵输出结果。最后，我们进一步扩展我们的模型来执行视频 AST。利用新颖的学习目标进行视频扩散模型训练，在保持风格化强度的前提下显着提高跨帧时间一致性。定性和定量比较以及全面的用户研究表明，我们的 HiCAST 在生成视觉上合理的风格化结果方面优于现有的 SoTA 方法。

Title: Efficient Image Deblurring Networks based on Diffusion Models. (arXiv:2401.05907v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.05907
• Code URL: https://github.com/bnm6900030/swintormer
• Copy Paste: [[2401.05907]] Efficient Image Deblurring Networks based on Diffusion Models(http://arxiv.org/abs/2401.05907)
• Summary:

  This article introduces a sliding window model for defocus deblurring that achieves the best performance to date with extremely low memory usage. Named Swintormer, the method utilizes a diffusion model to generate latent prior features that assist in restoring more detailed images. It also extends the sliding window strategy to specialized Transformer blocks for efficient inference. Additionally, we have further optimized Multiply-Accumulate operations (Macs). Compared to the currently top-performing GRL method, our Swintormer model drastically reduces computational complexity from 140.35 GMACs to 8.02 GMacs, while also improving the Signal-to-Noise Ratio (SNR) for defocus deblurring from 27.04 dB to 27.07 dB. This new method allows for the processing of higher resolution images on devices with limited memory, significantly expanding potential application scenarios. The article concludes with an ablation study that provides an in-depth analysis of the impact of each network module on final performance. The source code and model will be available at the following website: https://github.com/bnm6900030/swintormer.

• 摘要：

  本文介绍了一种用于散焦去模糊的滑动窗口模型，该模型以极低的内存使用量实现了迄今为止的最佳性能。该方法名为 Swintormer，利用扩散模型生成潜在的先验特征，有助于恢复更详细的图像。它还将滑动窗口策略扩展到专门的 Transformer 块，以实现高效推理。此外，我们还进一步优化了乘法累加运算 (Mac)。与目前表现最好的 GRL 方法相比，我们的 Swintormer 模型将计算复杂度从 140.35 GMAC 大幅降低到 8.02 GMac，同时还将散焦去模糊的信噪比 (SNR) 从 27.04 dB 提高到 27.07 dB。这种新方法允许在内存有限的设备上处理更高分辨率的图像，显​​着扩展了潜在的应用场景。本文最后进行了消融研究，深入分析了每个网络模块对最终性能的影响。源代码和模型可在以下网站获取：https://github.com/bnm6900030/swintormer。

Title: E$^{2}$GAN: Efficient Training of Efficient GANs for Image-to-Image Translation. (arXiv:2401.06127v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.06127
• Code URL: null
• Copy Paste: [[2401.06127]] E$^{2}$GAN: Efficient Training of Efficient GANs for Image-to-Image Translation(http://arxiv.org/abs/2401.06127)
• Summary:

  One highly promising direction for enabling flexible real-time on-device image editing is utilizing data distillation by leveraging large-scale text-to-image diffusion models, such as Stable Diffusion, to generate paired datasets used for training generative adversarial networks (GANs). This approach notably alleviates the stringent requirements typically imposed by high-end commercial GPUs for performing image editing with diffusion models. However, unlike text-to-image diffusion models, each distilled GAN is specialized for a specific image editing task, necessitating costly training efforts to obtain models for various concepts. In this work, we introduce and address a novel research direction: can the process of distilling GANs from diffusion models be made significantly more efficient? To achieve this goal, we propose a series of innovative techniques. First, we construct a base GAN model with generalized features, adaptable to different concepts through fine-tuning, eliminating the need for training from scratch. Second, we identify crucial layers within the base GAN model and employ Low-Rank Adaptation (LoRA) with a simple yet effective rank search process, rather than fine-tuning the entire base model. Third, we investigate the minimal amount of data necessary for fine-tuning, further reducing the overall training time. Extensive experiments show that we can efficiently empower GANs with the ability to perform real-time high-quality image editing on mobile devices with remarkable reduced training cost and storage for each concept.

• 摘要：

  实现灵活的实时设备图像编辑的一个非常有前途的方向是通过利用大规模文本到图像扩散模型（例如稳定扩散）来利用数据蒸馏来生成用于训练生成对抗网络的配对数据集（GAN）。这种方法显着缓解了高端商用 GPU 通常对使用扩散模型执行图像编辑提出的严格要求。然而，与文本到图像的扩散模型不同，每个精炼的 GAN 专门用于特定的图像编辑任务，需要昂贵的训练工作才能获得各种概念的模型。在这项工作中，我们介绍并提出了一个新的研究方向：从扩散模型中提取 GAN 的过程是否可以显着提高效率？为了实现这一目标，我们提出了一系列创新技术。首先，我们构建一个具有通用特征的基础 GAN 模型，通过微调适应不同的概念，从而无需从头开始训练。其次，我们确定基本 GAN 模型中的关键层，并通过简单而有效的排名搜索过程采用低秩适应 (LoRA)，而不是微调整个基本模型。第三，我们研究微调所需的最少量数据，进一步减少总体训练时间。大量实验表明，我们可以有效地赋予 GAN 在移动设备上执行实时高质量图像编辑的能力，并显着降低每个概念的训练成本和存储成本。

noise learning

data-free

transformer

Title: Spatial-Related Sensors Matters: 3D Human Motion Reconstruction Assisted with Textual Semantics. (arXiv:2401.05412v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.05412
• Code URL: null
• Copy Paste: [[2401.05412]] Spatial-Related Sensors Matters: 3D Human Motion Reconstruction Assisted with Textual Semantics(http://arxiv.org/abs/2401.05412)
• Summary:

  Leveraging wearable devices for motion reconstruction has emerged as an economical and viable technique. Certain methodologies employ sparse Inertial Measurement Units (IMUs) on the human body and harness data-driven strategies to model human poses. However, the reconstruction of motion based solely on sparse IMUs data is inherently fraught with ambiguity, a consequence of numerous identical IMU readings corresponding to different poses. In this paper, we explore the spatial importance of multiple sensors, supervised by text that describes specific actions. Specifically, uncertainty is introduced to derive weighted features for each IMU. We also design a Hierarchical Temporal Transformer (HTT) and apply contrastive learning to achieve precise temporal and feature alignment of sensor data with textual semantics. Experimental results demonstrate our proposed approach achieves significant improvements in multiple metrics compared to existing methods. Notably, with textual supervision, our method not only differentiates between ambiguous actions such as sitting and standing but also produces more precise and natural motion.

• 摘要：

  利用可穿戴设备进行运动重建已成为一种经济且可行的技术。某些方法在人体上采用稀疏惯性测量单元 (IMU)，并利用数据驱动策略来模拟人体姿势。然而，仅基于稀疏 IMU 数据的运动重建本质上充满歧义，这是对应于不同姿势的大量相同 IMU 读数的结果。在本文中，我们探讨了多个传感器的空间重要性，并由描述特定动作的文本进行监督。具体来说，引入不确定性来导出每个 IMU 的加权特征。我们还设计了分层时间变换器（HTT）并应用对比学习来实现传感器数据与文本语义的精确时间和特征对齐。实验结果表明，与现有方法相比，我们提出的方法在多个指标上取得了显着改进。值得注意的是，通过文本监督，我们的方法不仅可以区分坐和站等不明确的动作，而且还可以产生更精确和自然的动作。

Title: FourCastNeXt: Improving FourCastNet Training with Limited Compute. (arXiv:2401.05584v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.05584
• Code URL: null
• Copy Paste: [[2401.05584]] FourCastNeXt: Improving FourCastNet Training with Limited Compute(http://arxiv.org/abs/2401.05584)
• Summary:

  Recently, the FourCastNet Neural Earth System Model (NESM) has shown impressive results on predicting various atmospheric variables, trained on the ERA5 reanalysis dataset. While FourCastNet enjoys quasi-linear time and memory complexity in sequence length compared to quadratic complexity in vanilla transformers, training FourCastNet on ERA5 from scratch still requires large amount of compute resources, which is expensive or even inaccessible to most researchers. In this work, we will show improved methods that can train FourCastNet using only 1% of the compute required by the baseline, while maintaining model performance or par or even better than the baseline.

• 摘要：

  最近，FourCastNet 神经地球系统模型 (NESM) 在 ERA5 再分析数据集上进行训练，在预测各种大气变量方面显示出令人印象深刻的结果。虽然与普通 Transformer 中的二次复杂性相比，FourCastNet 在序列长度上享有准线性时间和内存复杂性，但从头开始在 ERA5 上训练 FourCastNet 仍然需要大量计算资源，这对于大多数研究人员来说是昂贵的甚至是无法访问的。在这项工作中，我们将展示改进的方法，这些方法可以仅使用基线所需计算量的 1% 来训练 FourCastNet，同时保持模型性能或与基线相当甚至更好。

Title: Transforming Image Super-Resolution: A ConvFormer-based Efficient Approach. (arXiv:2401.05633v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.05633
• Code URL: https://github.com/aitical/cfsr
• Copy Paste: [[2401.05633]] Transforming Image Super-Resolution: A ConvFormer-based Efficient Approach(http://arxiv.org/abs/2401.05633)
• Summary:

  Recent progress in single-image super-resolution (SISR) has achieved remarkable performance, yet the computational costs of these methods remain a challenge for deployment on resource-constrained devices. Especially for transformer-based methods, the self-attention mechanism in such models brings great breakthroughs while incurring substantial computational costs. To tackle this issue, we introduce the Convolutional Transformer layer (ConvFormer) and the ConvFormer-based Super-Resolution network (CFSR), which offer an effective and efficient solution for lightweight image super-resolution tasks. In detail, CFSR leverages the large kernel convolution as the feature mixer to replace the self-attention module, efficiently modeling long-range dependencies and extensive receptive fields with a slight computational cost. Furthermore, we propose an edge-preserving feed-forward network, simplified as EFN, to obtain local feature aggregation and simultaneously preserve more high-frequency information. Extensive experiments demonstrate that CFSR can achieve an advanced trade-off between computational cost and performance when compared to existing lightweight SR methods. Compared to state-of-the-art methods, e.g. ShuffleMixer, the proposed CFSR achieves 0.39 dB gains on Urban100 dataset for x2 SR task while containing 26% and 31% fewer parameters and FLOPs, respectively. Code and pre-trained models are available at https://github.com/Aitical/CFSR.

• 摘要：

  单图像超分辨率 (SISR) 的最新进展取得了显着的性能，但这些方法的计算成本对于在资源受限的设备上部署仍然是一个挑战。特别是对于基于 Transformer 的方法，此类模型中的自注意力机制带来了巨大的突破，同时带来了大量的计算成本。为了解决这个问题，我们引入了卷积变换层（ConvFormer）和基于ConvFormer的超分辨率网络（CFSR），它们为轻量级图像超分辨率任务提供了有效且高效的解决方案。具体来说，CFSR利用大核卷积作为特征混合器来取代自注意力模块，以少量的计算成本有效地建模长程依赖性和广泛的感受野。此外，我们提出了一种边缘保留前馈网络，简化为 EFN，以获得局部特征聚合并同时保留更多高频信息。大量实验表明，与现有的轻量级 SR 方法相比，CFSR 可以在计算成本和性能之间实现高级权衡。与最先进的方法相比，例如ShuffleMixer 中，所提出的 CFSR 在 x2 SR 任务的 Urban100 数据集上实现了 0.39 dB 的增益，同时参数和 FLOP 分别减少了 26% 和 31%。代码和预训练模型可在 https://github.com/Atical/CFSR 上获取。

Title: Masked Attribute Description Embedding for Cloth-Changing Person Re-identification. (arXiv:2401.05646v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.05646
• Code URL: null
• Copy Paste: [[2401.05646]] Masked Attribute Description Embedding for Cloth-Changing Person Re-identification(http://arxiv.org/abs/2401.05646)
• Summary:

  Cloth-changing person re-identification (CC-ReID) aims to match persons who change clothes over long periods. The key challenge in CC-ReID is to extract clothing-independent features, such as face, hairstyle, body shape, and gait. Current research mainly focuses on modeling body shape using multi-modal biological features (such as silhouettes and sketches). However, it does not fully leverage the personal description information hidden in the original RGB image. Considering that there are certain attribute descriptions which remain unchanged after the changing of cloth, we propose a Masked Attribute Description Embedding (MADE) method that unifies personal visual appearance and attribute description for CC-ReID. Specifically, handling variable clothing-sensitive information, such as color and type, is challenging for effective modeling. To address this, we mask the clothing and color information in the personal attribute description extracted through an attribute detection model. The masked attribute description is then connected and embedded into Transformer blocks at various levels, fusing it with the low-level to high-level features of the image. This approach compels the model to discard clothing information. Experiments are conducted on several CC-ReID benchmarks, including PRCC, LTCC, Celeb-reID-light, and LaST. Results demonstrate that MADE effectively utilizes attribute description, enhancing cloth-changing person re-identification performance, and compares favorably with state-of-the-art methods. The code is available at https://github.com/moon-wh/MADE.

• 摘要：

  换衣服的人重新识别（CC-ReID）旨在匹配长时间换衣服的人。 CC-ReID 的关键挑战是提取与服装无关的特征，例如面部、发型、体型和步态。目前的研究主要集中在使用多模态生物特征（例如轮廓和草图）来建模身体形状。然而，它并没有充分利用隐藏在原始RGB图像中的个人描述信息。考虑到某些属性描述在换衣服后保持不变，我们提出了一种掩蔽属性描述嵌入（MADE）方法，该方法将个人视觉外观和属性描述统一到CC-ReID。具体来说，处理可变的服装敏感信息（例如颜色和类型）对于有效建模来说是一个挑战。为了解决这个问题，我们掩盖了通过属性检测模型提取的个人属性描述中的服装和颜色信息。然后将屏蔽的属性描述连接并嵌入到各个级别的 Transformer 块中，将其与图像的低级到高级特征融合。这种方法迫使模型丢弃服装信息。在多个 CC-ReID 基准上进行了实验，包括 PRCC、LTCC、Celeb-reID-light 和 LaST。结果表明，MADE 有效地利用了属性描述，增强了换衣服的人重新识别性能，并且与最先进的方法相媲美。代码可在 https://github.com/moon-wh/MADE 获取。

Title: LKCA: Large Kernel Convolutional Attention. (arXiv:2401.05738v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.05738
• Code URL: null
• Copy Paste: [[2401.05738]] LKCA: Large Kernel Convolutional Attention(http://arxiv.org/abs/2401.05738)
• Summary:

  We revisit the relationship between attention mechanisms and large kernel ConvNets in visual transformers and propose a new spatial attention named Large Kernel Convolutional Attention (LKCA). It simplifies the attention operation by replacing it with a single large kernel convolution. LKCA combines the advantages of convolutional neural networks and visual transformers, possessing a large receptive field, locality, and parameter sharing. We explained the superiority of LKCA from both convolution and attention perspectives, providing equivalent code implementations for each view. Experiments confirm that LKCA implemented from both the convolutional and attention perspectives exhibit equivalent performance. We extensively experimented with the LKCA variant of ViT in both classification and segmentation tasks. The experiments demonstrated that LKCA exhibits competitive performance in visual tasks. Our code will be made publicly available at https://github.com/CatworldLee/LKCA.

• 摘要：

  我们重新审视了视觉 Transformer 中的注意力机制和大核卷积网络之间的关系，并提出了一种新的空间注意力，称为大核卷积注意力（LKCA）。它通过用单个大核卷积替换它来简化注意力操作。 LKCA结合了卷积神经网络和视觉变换器的优点，拥有大的感受野、局部性和参数共享。我们从卷积和注意力两个角度解释了LKCA的优越性，为每个视图提供了等效的代码实现。实验证实，从卷积和注意力角度实现的 LKCA 表现出同等的性能。我们在分类和分割任务中对 ViT 的 LKCA 变体进行了广泛的实验。实验表明，LKCA 在视觉任务中表现出有竞争力的表现。我们的代码将在 https://github.com/CatworldLee/LKCA 公开发布。

Title: Surface Normal Estimation with Transformers. (arXiv:2401.05745v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.05745
• Code URL: null
• Copy Paste: [[2401.05745]] Surface Normal Estimation with Transformers(http://arxiv.org/abs/2401.05745)
• Summary:

  We propose the use of a Transformer to accurately predict normals from point clouds with noise and density variations. Previous learning-based methods utilize PointNet variants to explicitly extract multi-scale features at different input scales, then focus on a surface fitting method by which local point cloud neighborhoods are fitted to a geometric surface approximated by either a polynomial function or a multi-layer perceptron (MLP). However, fitting surfaces to fixed-order polynomial functions can suffer from overfitting or underfitting, and learning MLP-represented hyper-surfaces requires pre-generated per-point weights. To avoid these limitations, we first unify the design choices in previous works and then propose a simplified Transformer-based model to extract richer and more robust geometric features for the surface normal estimation task. Through extensive experiments, we demonstrate that our Transformer-based method achieves state-of-the-art performance on both the synthetic shape dataset PCPNet, and the real-world indoor scene dataset SceneNN, exhibiting more noise-resilient behavior and significantly faster inference. Most importantly, we demonstrate that the sophisticated hand-designed modules in existing works are not necessary to excel at the task of surface normal estimation.

• 摘要：

  我们建议使用 Transformer 来准确预测具有噪声和密度变化的点云的法线。以前基于学习的方法利用 PointNet 变体在不同输入尺度上显式提取多尺度特征，然后关注表面拟合方法，通过该方法将局部点云邻域拟合到由多项式函数或多层近似的几何表面感知器（MLP）。然而，将曲面拟合到固定阶多项式函数可能会出现过度拟合或欠拟合的情况，并且学习 MLP 表示的超曲面需要预先生成的每点权重。为了避免这些限制，我们首先统一以前工作中的设计选择，然后提出一个基于 Transformer 的简化模型，为表面法线估计任务提取更丰富、更鲁棒的几何特征。通过大量实验，我们证明了基于 Transformer 的方法在合成形状数据集 PCPNet 和真实室内场景数据集 SceneNN 上实现了最先进的性能，表现出更强的抗噪行为和显着更快的推理速度。最重要的是，我们证明现有作品中复杂的手工设计模块不一定能够出色地完成表面法线估计的任务。

Title: Transformers are Multi-State RNNs. (arXiv:2401.06104v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.06104
• Code URL: null
• Copy Paste: [[2401.06104]] Transformers are Multi-State RNNs(http://arxiv.org/abs/2401.06104)
• Summary:

  Transformers are considered conceptually different compared to the previous generation of state-of-the-art NLP models - recurrent neural networks (RNNs). In this work, we demonstrate that decoder-only transformers can in fact be conceptualized as infinite multi-state RNNs - an RNN variant with unlimited hidden state size. We further show that pretrained transformers can be converted into $\textit{finite}$ multi-state RNNs by fixing the size of their hidden state. We observe that several existing transformers cache compression techniques can be framed as such conversion policies, and introduce a novel policy, TOVA, which is simpler compared to these policies. Our experiments with several long range tasks indicate that TOVA outperforms all other baseline policies, while being nearly on par with the full (infinite) model, and using in some cases only $\frac{1}{8}$ of the original cache size. Our results indicate that transformer decoder LLMs often behave in practice as RNNs. They also lay out the option of mitigating one of their most painful computational bottlenecks - the size of their cache memory. We publicly release our code at https://github.com/schwartz-lab-NLP/TOVA.

• 摘要：

  与上一代最先进的 NLP 模型——循环神经网络 (RNN) 相比，Transformer 在概念上被认为是不同的。在这项工作中，我们证明了仅解码器 Transformer 实际上可以被概念化为无限多状态 RNN——一种具有无限隐藏状态大小的 RNN 变体。我们进一步表明，通过固定隐藏状态的大小，预训练的 Transformer 可以转换为 $\textit{finite}$ 多状态 RNN。我们观察到一些现有的转换器缓存压缩技术可以被构建为这样的转换策略，并引入一种新的策略 TOVA，它比这些策略更简单。我们对多个远程任务进行的实验表明，TOVA 优于所有其他基线策略，同时几乎与完整（无限）模型相当，并且在某些情况下仅使用原始缓存大小的 $\frac{1}{8}$ 。我们的结果表明，变压器解码器 LLM 在实践中通常表现为 RNN。他们还提出了缓解最痛苦的计算瓶颈之一——缓存大小的选项。我们在 https://github.com/schwartz-lab-NLP/TOVA 公开发布我们的代码。

Title: Learning Cognitive Maps from Transformer Representations for Efficient Planning in Partially Observed Environments. (arXiv:2401.05946v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.05946
• Code URL: null
• Copy Paste: [[2401.05946]] Learning Cognitive Maps from Transformer Representations for Efficient Planning in Partially Observed Environments(http://arxiv.org/abs/2401.05946)
• Summary:

  Despite their stellar performance on a wide range of tasks, including in-context tasks only revealed during inference, vanilla transformers and variants trained for next-token predictions (a) do not learn an explicit world model of their environment which can be flexibly queried and (b) cannot be used for planning or navigation. In this paper, we consider partially observed environments (POEs), where an agent receives perceptually aliased observations as it navigates, which makes path planning hard. We introduce a transformer with (multiple) discrete bottleneck(s), TDB, whose latent codes learn a compressed representation of the history of observations and actions. After training a TDB to predict the future observation(s) given the history, we extract interpretable cognitive maps of the environment from its active bottleneck(s) indices. These maps are then paired with an external solver to solve (constrained) path planning problems. First, we show that a TDB trained on POEs (a) retains the near perfect predictive performance of a vanilla transformer or an LSTM while (b) solving shortest path problems exponentially faster. Second, a TDB extracts interpretable representations from text datasets, while reaching higher in-context accuracy than vanilla sequence models. Finally, in new POEs, a TDB (a) reaches near-perfect in-context accuracy, (b) learns accurate in-context cognitive maps (c) solves in-context path planning problems.

• 摘要：

  尽管它们在广泛的任务上表现出色，包括仅在推理过程中揭示的上下文任务，但为下一个标记预测训练的普通变压器和变体（a）没有学习其环境的明确的世界模型，这可以是灵活查询，(b) 不能用于规划或导航。在本文中，我们考虑部分观察环境（POE），其中代理在导航时接收到感知混叠的观察结果，这使得路径规划变得困难。我们引入了一个具有（多个）离散瓶颈的变压器 TDB，其潜在代码学习观察和动作历史的压缩表示。在训练 TDB 来预测给定历史的未来观察之后，我们从其活动瓶颈索引中提取环境的可解释认知图。然后将这些地图与外部求解器配对以解决（受限）路径规划问题。首先，我们表明，在 POE 上训练的 TDB (a) 保留了普通 Transformer 或 LSTM 近乎完美的预测性能，同时 (b) 解决最短路径问题的速度呈指数级增长。其次，TDB 从文本数据集中提取可解释的表示，同时达到比普通序列模型更高的上下文准确性。最后，在新的 POE 中，TDB (a) 达到近乎完美的上下文准确性，(b) 学习准确的上下文认知图，(c) 解决上下文路径规划问题。

generative

Title: An attempt to generate new bridge types from latent space of PixelCNN. (arXiv:2401.05964v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.05964
• Code URL: https://github.com/QQ583304953/Bridge-PixelCNN
• Copy Paste: [[2401.05964]] An attempt to generate new bridge types from latent space of PixelCNN(http://arxiv.org/abs/2401.05964)
• Summary:

  Try to generate new bridge types using generative artificial intelligence technology. Using symmetric structured image dataset of three-span beam bridge, arch bridge, cable-stayed bridge and suspension bridge , based on Python programming language, TensorFlow and Keras deep learning platform framework , PixelCNN is constructed and trained. The model can capture the statistical structure of the images and calculate the probability distribution of the next pixel when the previous pixels are given. From the obtained latent space sampling, new bridge types different from the training dataset can be generated. PixelCNN can organically combine different structural components on the basis of human original bridge types, creating new bridge types that have a certain degree of human original ability. Autoregressive models cannot understand the meaning of the sequence, while multimodal models combine regression and autoregressive models to understand the sequence. Multimodal models should be the way to achieve artificial general intelligence in the future.

• 摘要：

  尝试使用生成人工智能技术生成新的桥梁类型。利用三跨梁桥、拱桥、斜拉桥、悬索桥的对称结构化图像数据集，基于Python编程语言、TensorFlow和Keras深度学习平台框架，构建并训练PixelCNN。该模型可以捕获图像的统计结构，并在给定先前像素的情况下计算下一个像素的概率分布。根据获得的潜在空间采样，可以生成与训练数据集不同的新桥梁类型。 PixelCNN可以在人类原创桥梁类型的基础上有机地组合不同的结构组件，创造出具有一定人类原创能力的新桥梁类型。自回归模型无法理解序列的含义，而多模态模型则结合回归和自回归模型来理解序列。多模态模型应该是未来实现通用人工智能的途径。

Title: GE-AdvGAN: Improving the transferability of adversarial samples by gradient editing-based adversarial generative model. (arXiv:2401.06031v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.06031
• Code URL: https://github.com/lmbtough/ge-advgan
• Copy Paste: [[2401.06031]] GE-AdvGAN: Improving the transferability of adversarial samples by gradient editing-based adversarial generative model(http://arxiv.org/abs/2401.06031)
• Summary:

  Adversarial generative models, such as Generative Adversarial Networks (GANs), are widely applied for generating various types of data, i.e., images, text, and audio. Accordingly, its promising performance has led to the GAN-based adversarial attack methods in the white-box and black-box attack scenarios. The importance of transferable black-box attacks lies in their ability to be effective across different models and settings, more closely aligning with real-world applications. However, it remains challenging to retain the performance in terms of transferable adversarial examples for such methods. Meanwhile, we observe that some enhanced gradient-based transferable adversarial attack algorithms require prolonged time for adversarial sample generation. Thus, in this work, we propose a novel algorithm named GE-AdvGAN to enhance the transferability of adversarial samples whilst improving the algorithm's efficiency. The main approach is via optimising the training process of the generator parameters. With the functional and characteristic similarity analysis, we introduce a novel gradient editing (GE) mechanism and verify its feasibility in generating transferable samples on various models. Moreover, by exploring the frequency domain information to determine the gradient editing direction, GE-AdvGAN can generate highly transferable adversarial samples while minimizing the execution time in comparison to the state-of-the-art transferable adversarial attack algorithms. The performance of GE-AdvGAN is comprehensively evaluated by large-scale experiments on different datasets, which results demonstrate the superiority of our algorithm. The code for our algorithm is available at: https://github.com/LMBTough/GE-advGAN

• 摘要：

  对抗生成模型，例如生成对抗网络（GAN），广泛应用于生成各种类型的数据，即图像、文本和音频。因此，其令人鼓舞的性能催生了白盒和黑盒攻击场景中基于 GAN 的对抗攻击方法。可转移黑盒攻击的重要性在于它们能够在不同的模型和设置中发挥作用，与现实世界的应用程序更加紧密地结合。然而，保持此类方法在可转移对抗样本方面的性能仍然具有挑战性。同时，我们观察到一些增强的基于梯度的可转移对抗攻击算法需要更长的时间来生成对抗样本。因此，在这项工作中，我们提出了一种名为 GE-AdvGAN 的新算法，以增强对抗样本的可转移性，同时提高算法的效率。主要方法是通过优化生成器参数的训练过程。通过功能和特征相似性分析，我们引入了一种新颖的梯度编辑（GE）机制，并验证了其在各种模型上生成可转移样本的可行性。此外，通过探索频域信息来确定梯度编辑方向，GE-AdvGAN 可以生成高度可转移的对抗样本，同时与最先进的可转移对抗攻击算法相比，最大限度地减少执行时间。通过在不同数据集上的大规模实验对GE-AdvGAN的性能进行了综合评估，结果证明了我们算法的优越性。我们算法的代码位于：https://github.com/LMBTough/GE-advGAN

Title: RAVEN: Rethinking Adversarial Video Generation with Efficient Tri-plane Networks. (arXiv:2401.06035v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.06035
• Code URL: null
• Copy Paste: [[2401.06035]] RAVEN: Rethinking Adversarial Video Generation with Efficient Tri-plane Networks(http://arxiv.org/abs/2401.06035)
• Summary:

  We present a novel unconditional video generative model designed to address long-term spatial and temporal dependencies. To capture these dependencies, our approach incorporates a hybrid explicit-implicit tri-plane representation inspired by 3D-aware generative frameworks developed for three-dimensional object representation and employs a singular latent code to model an entire video sequence. Individual video frames are then synthesized from an intermediate tri-plane representation, which itself is derived from the primary latent code. This novel strategy reduces computational complexity by a factor of $2$ as measured in FLOPs. Consequently, our approach facilitates the efficient and temporally coherent generation of videos. Moreover, our joint frame modeling approach, in contrast to autoregressive methods, mitigates the generation of visual artifacts. We further enhance the model's capabilities by integrating an optical flow-based module within our Generative Adversarial Network (GAN) based generator architecture, thereby compensating for the constraints imposed by a smaller generator size. As a result, our model is capable of synthesizing high-fidelity video clips at a resolution of $256\times256$ pixels, with durations extending to more than $5$ seconds at a frame rate of 30 fps. The efficacy and versatility of our approach are empirically validated through qualitative and quantitative assessments across three different datasets comprising both synthetic and real video clips.

• 摘要：

  我们提出了一种新颖的无条件视频生成模型，旨在解决长期的空间和时间依赖性。为了捕获这些依赖性，我们的方法采用了混合显式-隐式三平面表示，其灵感来自于为三维对象表示而开发的 3D 感知生成框架，并采用单一潜在代码来建模整个视频序列。然后从中间三平面表示合成各个视频帧，该中间三平面表示本身是从主要潜在代码导出的。这种新颖的策略将计算复杂性降低了 2 美元（以 FLOP 为单位）。因此，我们的方法有助于高效且时间连贯地生成视频。此外，与自回归方法相比，我们的联合帧建模方法可以减少视觉伪影的产生。我们通过将基于光流的模块集成到基于生成对抗网络（GAN）的生成器架构中，进一步增强了模型的功能，从而补偿了较小生成器尺寸所带来的限制。因此，我们的模型能够以 256\times256$ 像素的分辨率合成高保真视频剪辑，持续时间在 30 fps 的帧速率下延长到超过 5$ 秒。我们的方法的有效性和多功能性通过对三个不同数据集（包括合成视频剪辑和真实视频剪辑）的定性和定量评估进行了实证验证。

Title: Designing Heterogeneous LLM Agents for Financial Sentiment Analysis. (arXiv:2401.05799v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.05799
• Code URL: null
• Copy Paste: [[2401.05799]] Designing Heterogeneous LLM Agents for Financial Sentiment Analysis(http://arxiv.org/abs/2401.05799)
• Summary:

  Large language models (LLMs) have drastically changed the possible ways to design intelligent systems, shifting the focuses from massive data acquisition and new modeling training to human alignment and strategical elicitation of the full potential of existing pre-trained models. This paradigm shift, however, is not fully realized in financial sentiment analysis (FSA), due to the discriminative nature of this task and a lack of prescriptive knowledge of how to leverage generative models in such a context. This study investigates the effectiveness of the new paradigm, i.e., using LLMs without fine-tuning for FSA. Rooted in Minsky's theory of mind and emotions, a design framework with heterogeneous LLM agents is proposed. The framework instantiates specialized agents using prior domain knowledge of the types of FSA errors and reasons on the aggregated agent discussions. Comprehensive evaluation on FSA datasets show that the framework yields better accuracies, especially when the discussions are substantial. This study contributes to the design foundations and paves new avenues for LLMs-based FSA. Implications on business and management are also discussed.

• 摘要：

  大型语言模型 (LLM) 极大地改变了设计智能系统的可能方式，将重点从海量数据采集和新的建模训练转移到人类调整和战略性激发现有预训练模型的全部潜力。然而，由于这项任务的歧视性以及缺乏如何在这种背景下利用生成模型的规范性知识，这种范式转变在金融情绪分析（FSA）中并未完全实现。本研究调查了新范式的有效性，即使用法学硕士而不对 FSA 进行微调。植根于明斯基的心灵和情感理论，提出了一种具有异构 LLM 代理的设计框架。该框架使用 FSA 错误类型的先验领域知识以及聚合代理讨论的原因来实例化专用代理。对 FSA 数据集的综合评估表明，该框架具有更好的准确性，特别是在讨论大量时。这项研究为基于法学硕士的 FSA 奠定了设计基础并铺平了新的途径。还讨论了对业务和管理的影响。

Title: Tuning LLMs with Contrastive Alignment Instructions for Machine Translation in Unseen, Low-resource Languages. (arXiv:2401.05811v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.05811
• Code URL: null
• Copy Paste: [[2401.05811]] Tuning LLMs with Contrastive Alignment Instructions for Machine Translation in Unseen, Low-resource Languages(http://arxiv.org/abs/2401.05811)
• Summary:

  This article introduces contrastive alignment instructions (AlignInstruct) to address two challenges in machine translation (MT) on large language models (LLMs). One is the expansion of supported languages to previously unseen ones. The second relates to the lack of data in low-resource languages. Model fine-tuning through MT instructions (MTInstruct) is a straightforward approach to the first challenge. However, MTInstruct is limited by weak cross-lingual signals inherent in the second challenge. AlignInstruct emphasizes cross-lingual supervision via a cross-lingual discriminator built using statistical word alignments. Our results based on fine-tuning the BLOOMZ models (1b1, 3b, and 7b1) in up to 24 unseen languages showed that: (1) LLMs can effectively translate unseen languages using MTInstruct; (2) AlignInstruct led to consistent improvements in translation quality across 48 translation directions involving English; (3) Discriminator-based instructions outperformed their generative counterparts as cross-lingual instructions; (4) AlignInstruct improved performance in 30 zero-shot directions.

• 摘要：

  本文介绍了对比对齐指令 (AlignInstruct)，以解决大型语言模型 (LLM) 上机器翻译 (MT) 的两个挑战。一是将支持的语言扩展到以前未见过的语言。第二个与缺乏资源语言的数据有关。通过 MT 指令 (MTInstruct) 进行模型微调是应对第一个挑战的简单方法。然而，MTInstruct 受到第二个挑战中固有的微弱跨语言信号的限制。 AlignInstruct 强调通过使用统计单词对齐构建的跨语言鉴别器进行跨语言监督。我们基于最多 24 种未见过的语言对 BLOOMZ 模型（1b1、3b 和 7b1）进行微调的结果表明：（1）法学硕士可以使用 MTInstruct 有效翻译未见的语言； (2) AlignInstruct 导致 48 个涉及英语的翻译方向的翻译质量持续提高； (3) 作为跨语言指令，基于判别器的指令优于生成指令； (4) AlignInstruct 改进了 30 个零样本方向的性能。

Title: Generative Deduplication For Socia Media Data Selection. (arXiv:2401.05883v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.05883
• Code URL: null
• Copy Paste: [[2401.05883]] Generative Deduplication For Socia Media Data Selection(http://arxiv.org/abs/2401.05883)
• Summary:

  Social media data is plagued by the redundancy problem caused by its noisy nature, leading to increased training time and model bias. To address this issue, we propose a novel approach called generative duplication. It aims to remove duplicate text from noisy social media data and mitigate model bias. By doing so, it can improve social media language understanding performance and save training time. Extensive experiments demonstrate that the proposed generative deduplication can effectively reduce training samples while improving performance. This evidence suggests the effectiveness of generative deduplication and its importance in social media language understanding.

• 摘要：

  社交媒体数据因其噪声性质而受到冗余问题的困扰，导致训练时间增加和模型偏差。为了解决这个问题，我们提出了一种称为生成复制的新方法。它的目的是从嘈杂的社交媒体数据中删除重复的文本并减轻模型偏差。通过这样做，它可以提高社交媒体语言理解性能并节省培训时间。大量实验表明，所提出的生成重复数据删除可以有效减少训练样本，同时提高性能。这一证据表明生成式重复数据删除的有效性及其在社交媒体语言理解中的重要性。

Title: An Augmented Surprise-guided Sequential Learning Framework for Predicting the Melt Pool Geometry. (arXiv:2401.05579v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.05579
• Code URL: null
• Copy Paste: [[2401.05579]] An Augmented Surprise-guided Sequential Learning Framework for Predicting the Melt Pool Geometry(http://arxiv.org/abs/2401.05579)
• Summary:

  Metal Additive Manufacturing (MAM) has reshaped the manufacturing industry, offering benefits like intricate design, minimal waste, rapid prototyping, material versatility, and customized solutions. However, its full industry adoption faces hurdles, particularly in achieving consistent product quality. A crucial aspect for MAM's success is understanding the relationship between process parameters and melt pool characteristics. Integrating Artificial Intelligence (AI) into MAM is essential. Traditional machine learning (ML) methods, while effective, depend on large datasets to capture complex relationships, a significant challenge in MAM due to the extensive time and resources required for dataset creation. Our study introduces a novel surprise-guided sequential learning framework, SurpriseAF-BO, signaling a significant shift in MAM. This framework uses an iterative, adaptive learning process, modeling the dynamics between process parameters and melt pool characteristics with limited data, a key benefit in MAM's cyber manufacturing context. Compared to traditional ML models, our sequential learning method shows enhanced predictive accuracy for melt pool dimensions. Further improving our approach, we integrated a Conditional Tabular Generative Adversarial Network (CTGAN) into our framework, forming the CT-SurpriseAF-BO. This produces synthetic data resembling real experimental data, improving learning effectiveness. This enhancement boosts predictive precision without requiring additional physical experiments. Our study demonstrates the power of advanced data-driven techniques in cyber manufacturing and the substantial impact of sequential AI and ML, particularly in overcoming MAM's traditional challenges.

• 摘要：

  金属增材制造 (MAM) 重塑了制造业，提供了复杂设计、最少浪费、快速原型制作、材料多功能性和定制解决方案等优势。然而，其在整个行业的采用面临着障碍，特别是在实现一致的产品质量方面。 MAM 成功的一个关键因素是了解工艺参数和熔池特性之间的关系。将人工智能 (AI) 集成到 MAM 中至关重要。传统的机器学习 (ML) 方法虽然有效，但依赖于大型数据集来捕获复杂的关系，由于创建数据集需要大量的时间和资源，这对 MAM 来说是一个重大挑战。我们的研究引入了一种新颖的惊喜引导顺序学习框架 SurpriseAF-BO，标志着 MAM 的重大转变。该框架采用迭代、自适应学习过程，利用有限的数据对工艺参数和熔池特性之间的动态进行建模，这是 MAM 网络制造环境中的一个关键优势。与传统的机器学习模型相比，我们的顺序学习方法显示出熔池尺寸的预测准确性更高。进一步改进我们的方法，我们将条件表格生成对抗网络（CTGAN）集成到我们的框架中，形成 CT-SurpriseAF-BO。这会产生类似于真实实验数据的合成数据，从而提高学习效率。这一增强功能提高了预测精度，无需额外的物理实验。我们的研究展示了先进数据驱动技术在网络制造中的力量以及顺序人工智能和机器学习的重大影响，特别是在克服 MAM 的传统挑战方面。

large language model

Title: Video Anomaly Detection and Explanation via Large Language Models. (arXiv:2401.05702v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.05702
• Code URL: null
• Copy Paste: [[2401.05702]] Video Anomaly Detection and Explanation via Large Language Models(http://arxiv.org/abs/2401.05702)
• Summary:

  Video Anomaly Detection (VAD) aims to localize abnormal events on the timeline of long-range surveillance videos. Anomaly-scoring-based methods have been prevailing for years but suffer from the high complexity of thresholding and low explanability of detection results. In this paper, we conduct pioneer research on equipping video-based large language models (VLLMs) in the framework of VAD, making the VAD model free from thresholds and able to explain the reasons for the detected anomalies. We introduce a novel network module Long-Term Context (LTC) to mitigate the incapability of VLLMs in long-range context modeling. We design a three-phase training method to improve the efficiency of fine-tuning VLLMs by substantially minimizing the requirements for VAD data and lowering the costs of annotating instruction-tuning data. Our trained model achieves the top performance on the anomaly videos of the UCF-Crime and TAD benchmarks, with the AUC improvements of +3.86\% and +4.96\%, respectively. More impressively, our approach can provide textual explanations for detected anomalies.

• 摘要：

  视频异常检测（VAD）旨在定位远程监控视频时间轴上的异常事件。基于异常评分的方法已经流行多年，但存在阈值复杂度高和检测结果可解释性低的问题。在本文中，我们在VAD框架中装备基于视频的大语言模型（VLLM）进行了开创性的研究，使VAD模型不受阈值限制，并且能够解释检测到的异常的原因。我们引入了一种新颖的网络模块长期上下文（LTC）来缓解 VLLM 在远程上下文建模中的无能。我们设计了一种三阶段训练方法，通过大幅减少对 VAD 数据的需求并降低注释指令调优数据的成本来提高微调 VLLM 的效率。我们训练的模型在 UCF-Crime 和 TAD 基准的异常视频上实现了最佳性能，AUC 分别提高了 +3.86\% 和 +4.96\%。更令人印象深刻的是，我们的方法可以为检测到的异常提供文本解释。

Title: LEGO:Language Enhanced Multi-modal Grounding Model. (arXiv:2401.06071v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.06071
• Code URL: https://github.com/lzw-lzw/lego
• Copy Paste: [[2401.06071]] LEGO:Language Enhanced Multi-modal Grounding Model(http://arxiv.org/abs/2401.06071)
• Summary:

  Multi-modal large language models have demonstrated impressive performance across various tasks in different modalities. However, existing multi-modal models primarily emphasize capturing global information within each modality while neglecting the importance of perceiving local information across modalities. Consequently, these models lack the ability to effectively understand the fine-grained details of input data, limiting their performance in tasks that require a more nuanced understanding. To address this limitation, there is a compelling need to develop models that enable fine-grained understanding across multiple modalities, thereby enhancing their applicability to a wide range of tasks. In this paper, we propose LEGO, a language enhanced multi-modal grounding model. Beyond capturing global information like other multi-modal models, our proposed model excels at tasks demanding a detailed understanding of local information within the input. It demonstrates precise identification and localization of specific regions in images or moments in videos. To achieve this objective, we design a diversified dataset construction pipeline, resulting in a multi-modal, multi-granularity dataset for model training. The code, dataset, and demo of our model can be found at https: //github.com/lzw-lzw/LEGO.

• 摘要：

  多模态大语言模型在不同模态的各种任务中表现出了令人印象深刻的性能。然而，现有的多模态模型主要强调捕获每种模态内的全局信息，而忽略了跨模态感知局部信息的重要性。因此，这些模型缺乏有效理解输入数据的细粒度细节的能力，限制了它们在需要更细致理解的任务中的性能。为了解决这一限制，迫切需要开发能够跨多种模式进行细粒度理解的模型，从而增强其对广泛任务的适用性。在本文中，我们提出了 LEGO，一种语言增强的多模态基础模型。除了像其他多模态模型一样捕获全局信息之外，我们提出的模型还擅长执行需要详细了解输入中的本地信息的任务。它演示了对图像或视频中特定区域的精确识别和定位。为了实现这一目标，我们设计了多样化的数据集构建流程，从而产生用于模型训练的多模式、多粒度数据集。我们模型的代码、数据集和演示可以在 https://github.com/lzw-lzw/LEGO 找到。

Title: TrustLLM: Trustworthiness in Large Language Models. (arXiv:2401.05561v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.05561
• Code URL: https://github.com/HowieHwong/TrustLLM
• Copy Paste: [[2401.05561]] TrustLLM: Trustworthiness in Large Language Models(http://arxiv.org/abs/2401.05561)
• Summary:

  Large language models (LLMs), exemplified by ChatGPT, have gained considerable attention for their excellent natural language processing capabilities. Nonetheless, these LLMs present many challenges, particularly in the realm of trustworthiness. Therefore, ensuring the trustworthiness of LLMs emerges as an important topic. This paper introduces TrustLLM, a comprehensive study of trustworthiness in LLMs, including principles for different dimensions of trustworthiness, established benchmark, evaluation, and analysis of trustworthiness for mainstream LLMs, and discussion of open challenges and future directions. Specifically, we first propose a set of principles for trustworthy LLMs that span eight different dimensions. Based on these principles, we further establish a benchmark across six dimensions including truthfulness, safety, fairness, robustness, privacy, and machine ethics. We then present a study evaluating 16 mainstream LLMs in TrustLLM, consisting of over 30 datasets. Our findings firstly show that in general trustworthiness and utility (i.e., functional effectiveness) are positively related. Secondly, our observations reveal that proprietary LLMs generally outperform most open-source counterparts in terms of trustworthiness, raising concerns about the potential risks of widely accessible open-source LLMs. However, a few open-source LLMs come very close to proprietary ones. Thirdly, it is important to note that some LLMs may be overly calibrated towards exhibiting trustworthiness, to the extent that they compromise their utility by mistakenly treating benign prompts as harmful and consequently not responding. Finally, we emphasize the importance of ensuring transparency not only in the models themselves but also in the technologies that underpin trustworthiness. Knowing the specific trustworthy technologies that have been employed is crucial for analyzing their effectiveness.

• 摘要：

  以 ChatGPT 为代表的大型语言模型 (LLM) 因其出色的自然语言处理能力而受到广泛关注。尽管如此，这些法学硕士提出了许多挑战，特别是在可信度领域。因此，确保LLM的可信度成为一个重要的话题。本文介绍了TrustLLM，这是一项关于法学硕士可信度的综合研究，包括可信度不同维度的原则、主流法学硕士可信度的建立基准、评估和分析，以及对开放挑战和未来方向的讨论。具体来说，我们首先提出了一套涵盖八个不同维度的值得信赖的法学硕士原则。基于这些原则，我们进一步建立了真实性、安全性、公平性、稳健性、隐私性和机器道德等六个维度的基准。然后，我们提出了一项评估 TrustLLM 中 16 个主流法学硕士的研究，其中包含 30 多个数据集。我们的研究结果首先表明，一般来说，可信度和效用（即功能有效性）呈正相关。其次，我们的观察表明，专有法学硕士在可信度方面通常优于大多数开源法学硕士，这引起了人们对广泛使用的开源法学硕士潜在风险的担忧。然而，一些开源法学硕士非常接近专有法学硕士。第三，值得注意的是，一些法学硕士可能会过度校准以表现出可信度，以至于他们错误地将良性提示视为有害提示并因此不响应，从而损害了其效用。最后，我们强调不仅要确保模型本身的透明度，还要确保支撑可信性的技术的透明度。了解已采用的具体可信技术对于分析其有效性至关重要。

Title: POMP: Probability-driven Meta-graph Prompter for LLMs in Low-resource Unsupervised Neural Machine Translation. (arXiv:2401.05596v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.05596
• Code URL: null
• Copy Paste: [[2401.05596]] POMP: Probability-driven Meta-graph Prompter for LLMs in Low-resource Unsupervised Neural Machine Translation(http://arxiv.org/abs/2401.05596)
• Summary:

  Low-resource languages (LRLs) face challenges in supervised neural machine translation due to limited parallel data, prompting research into unsupervised methods. Unsupervised neural machine translation (UNMT) methods, including back-translation, transfer learning, and pivot-based translation, offer practical solutions for LRL translation, but they are hindered by issues like synthetic data noise, language bias, and error propagation, which can potentially be mitigated by Large Language Models (LLMs). LLMs have advanced NMT with in-context learning (ICL) and supervised fine-tuning methods, but insufficient training data results in poor performance in LRLs. We argue that LLMs can mitigate the linguistic noise with auxiliary languages to improve translations in LRLs. In this paper, we propose Probability-driven Meta-graph Prompter (POMP), a novel approach employing a dynamic, sampling-based graph of multiple auxiliary languages to enhance LLMs' translation capabilities for LRLs. POMP involves constructing a directed acyclic meta-graph for each source language, from which we dynamically sample multiple paths to prompt LLMs to mitigate the linguistic noise and improve translations during training. We use the BLEURT metric to evaluate the translations and back-propagate rewards, estimated by scores, to update the probabilities of auxiliary languages in the paths. Our experiments show significant improvements in the translation quality of three LRLs, demonstrating the effectiveness of our approach.

• 摘要：

  由于并行数据有限，低资源语言 (LRL) 在监督神经机器翻译方面面临挑战，这促使人们对无监督方法进行研究。无监督神经机器翻译 (UNMT) 方法，包括反向翻译、迁移学习和基于枢轴的翻译，为 LRL 翻译提供了实用的解决方案，但它们受到合成数据噪声、语言偏差和错误传播等问题的阻碍，这些问题可能会导致大型语言模型 (LLM) 可能会缓解这一问题。 LLM 拥有带有上下文学习 (ICL) 和监督微调方法的先进 NMT，但训练数据不足会导致 LRL 表现不佳。我们认为法学硕士可以通过辅助语言减轻语言噪音，从而改善法学硕士的翻译。在本文中，我们提出了概率驱动的元图提示器（POMP），这是一种采用动态、基于采样的多种辅助语言图来增强法学硕士对 LRL 的翻译能力的新颖方法。 POMP 涉及为每种源语言构建一个有向非循环元图，我们从中动态采样多个路径，以提示法学硕士在训练期间减轻语言噪音并改进翻译。我们使用 BLEURT 指标来评估翻译和反向传播奖励（通过分数估计），以更新路径中辅助语言的概率。我们的实验表明三个 LRL 的翻译质量显着提高，证明了我们方法的有效性。

Title: Scaling Laws for Forgetting When Fine-Tuning Large Language Models. (arXiv:2401.05605v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.05605
• Code URL: null
• Copy Paste: [[2401.05605]] Scaling Laws for Forgetting When Fine-Tuning Large Language Models(http://arxiv.org/abs/2401.05605)
• Summary:

  We study and quantify the problem of forgetting when fine-tuning pre-trained large language models (LLMs) on a downstream task. We find that parameter-efficient fine-tuning (PEFT) strategies, such as Low-Rank Adapters (LoRA), still suffer from catastrophic forgetting. In particular, we identify a strong inverse linear relationship between the fine-tuning performance and the amount of forgetting when fine-tuning LLMs with LoRA. We further obtain precise scaling laws that show forgetting increases as a shifted power law in the number of parameters fine-tuned and the number of update steps. We also examine the impact of forgetting on knowledge, reasoning, and the safety guardrails trained into Llama 2 7B chat. Our study suggests that forgetting cannot be avoided through early stopping or by varying the number of parameters fine-tuned. We believe this opens up an important safety-critical direction for future research to evaluate and develop fine-tuning schemes which mitigate forgetting

• 摘要：

  我们研究并量化了在下游任务上微调预训练大型语言模型 (LLM) 时的遗忘问题。我们发现参数高效微调（PEFT）策略，例如低秩适配器（LoRA），仍然遭受灾难性遗忘的困扰。特别是，当使用 LoRA 微调 LLM 时，我们发现微调性能和遗忘量之间存在很强的逆线性关系。我们进一步获得了精确的缩放定律，该定律表明遗忘随着微调参数数量和更新步骤数量的幂律变化而增加。我们还研究了遗忘对 Llama 2 7B 聊天中训练的知识、推理和安全护栏的影响。我们的研究表明，不能通过提前停止或改变微调​​参数的数量来避免遗忘。我们相信，这为未来的研究开辟了一个重要的安全关键方向，以评估和开发减轻遗忘的微调方案

Title: The Benefits of a Concise Chain of Thought on Problem-Solving in Large Language Models. (arXiv:2401.05618v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.05618
• Code URL: https://github.com/matthewrenze/jhu-concise-cot
• Copy Paste: [[2401.05618]] The Benefits of a Concise Chain of Thought on Problem-Solving in Large Language Models(http://arxiv.org/abs/2401.05618)
• Summary:

  In this paper, we introduce Concise Chain-of-Thought (CCoT) prompting. We compared standard CoT and CCoT prompts to see how conciseness impacts response length and correct-answer accuracy. We evaluated this using GPT-3.5 and GPT-4 with a multiple-choice question-and-answer (MCQA) benchmark. CCoT reduced average response length by 48.70% for both GPT-3.5 and GPT-4 while having a negligible impact on problem-solving performance. However, on math problems, GPT-3.5 with CCoT incurs a performance penalty of 27.69%. Overall, CCoT leads to an average per-token cost reduction of 22.67%. These results have practical implications for AI systems engineers using LLMs to solve real-world problems with CoT prompt-engineering techniques. In addition, these results provide more general insight for AI researchers studying the emergent behavior of step-by-step reasoning in LLMs.

• 摘要：

  在本文中，我们介绍了简洁思维链（CCoT）提示。我们比较了标准 CoT 和 CCoT 提示，以了解简洁性如何影响响​​应长度和正确答案的准确性。我们使用 GPT-3.5 和 GPT-4 以及多项选择问答 (MCQA) 基准对此进行了评估。 CCoT 将 GPT-3.5 和 GPT-4 的平均响应长度减少了 48.70%，同时对问题解决性能的影响可以忽略不计。然而，在数学问题上，带有 CCoT 的 GPT-3.5 会导致性能损失 27.69%。总体而言，CCoT 使每个代币的平均成本降低了 22.67%。这些结果对于使用法学硕士通过 CoT 即时工程技术解决现实世界问题的人工智能系统工程师具有实际意义。此外，这些结果为人工智能研究人员研究法学硕士中逐步推理的涌现行为提供了更普遍的见解。

Title: On Detecting Cherry-picking in News Coverage Using Large Language Models. (arXiv:2401.05650v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.05650
• Code URL: https://github.com/emnlp-cherry/cherry
• Copy Paste: [[2401.05650]] On Detecting Cherry-picking in News Coverage Using Large Language Models(http://arxiv.org/abs/2401.05650)
• Summary:

  Cherry-picking refers to the deliberate selection of evidence or facts that favor a particular viewpoint while ignoring or distorting evidence that supports an opposing perspective. Manually identifying instances of cherry-picked statements in news stories can be challenging, particularly when the opposing viewpoint's story is absent. This study introduces Cherry, an innovative approach for automatically detecting cherry-picked statements in news articles by finding missing important statements in the target news story. Cherry utilizes the analysis of news coverage from multiple sources to identify instances of cherry-picking. Our approach relies on language models that consider contextual information from other news sources to classify statements based on their importance to the event covered in the target news story. Furthermore, this research introduces a novel dataset specifically designed for cherry-picking detection, which was used to train and evaluate the performance of the models. Our best performing model achieves an F-1 score of about %89 in detecting important statements when tested on unseen set of news stories. Moreover, results show the importance incorporating external knowledge from alternative unbiased narratives when assessing a statement's importance.

• 摘要：

  择优挑选是指故意选择支持特定观点的证据或事实，而忽略或歪曲支持相反观点的证据。手动识别新闻报道中精心挑选的陈述实例可能具有挑战性，特别是当反对观点的报道不存在时。这项研究引入了 Cherry，这是一种创新方法，通过查找目标新闻报道中缺失的重要陈述来自动检测新闻文章中精选的陈述。 Cherry 利用对多个来源的新闻报道进行分析来识别择优挑选的情况。我们的方法依赖于语言模型，该模型考虑其他新闻来源的上下文信息，根据语句对目标新闻报道中所涵盖事件的重要性对语句进行分类。此外，这项研究引入了一个专门为挑选检测而设计的新颖数据集，用于训练和评估模型的性能。当对未见过的新闻报道进行测试时，我们表现最好的模型在检测重要陈述方面取得了约 %89 的 F-1 分数。此外，结果表明，在评估陈述的重要性时，结合来自其他公正叙述的外部知识非常重要。

Title: Integrating Physician Diagnostic Logic into Large Language Models: Preference Learning from Process Feedback. (arXiv:2401.05695v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.05695
• Code URL: null
• Copy Paste: [[2401.05695]] Integrating Physician Diagnostic Logic into Large Language Models: Preference Learning from Process Feedback(http://arxiv.org/abs/2401.05695)
• Summary:

  The use of large language models in medical dialogue generation has garnered significant attention, with a focus on improving response quality and fluency. While previous studies have made progress in optimizing model performance for single-round medical Q&A tasks, there is a need to enhance the model's capability for multi-round conversations to avoid logical inconsistencies. To address this, we propose an approach called preference learning from process feedback~(PLPF), which integrates the doctor's diagnostic logic into LLMs. PLPF involves rule modeling, preference data generation, and preference alignment to train the model to adhere to the diagnostic process. Experimental results using Standardized Patient Testing show that PLPF enhances the diagnostic accuracy of the baseline model in medical conversations by 17.6%, outperforming traditional reinforcement learning from human feedback. Additionally, PLPF demonstrates effectiveness in both multi-round and single-round dialogue tasks, showcasing its potential for improving medical dialogue generation.

• 摘要：

  大型语言模型在医学对话生成中的使用引起了广泛关注，重点是提高响应质量和流畅性。虽然之前的研究在优化单轮医疗问答任务的模型性能方面取得了进展，但仍需要增强模型的多轮对话能力，以避免逻辑不一致。为了解决这个问题，我们提出了一种称为从过程反馈中进行偏好学习（PLPF）的方法，它将医生的诊断逻辑集成到法学硕士中。 PLPF 涉及规则建模、偏好数据生成和偏好对齐，以训练模型遵循诊断过程。使用标准化患者测试的实验结果表明，PLPF 将医疗对话中基线模型的诊断准确性提高了 17.6%，优于基于人类反馈的传统强化学习。此外，PLPF 在多轮和单轮对话任务中都表现出了有效性，展示了其改善医学对话生成的潜力。

Title: CAT-LLM: Prompting Large Language Models with Text Style Definition for Chinese Article-style Transfer. (arXiv:2401.05707v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.05707
• Code URL: https://github.com/taozhen1110/cat-llm
• Copy Paste: [[2401.05707]] CAT-LLM: Prompting Large Language Models with Text Style Definition for Chinese Article-style Transfer(http://arxiv.org/abs/2401.05707)
• Summary:

  Text style transfer is increasingly prominent in online entertainment and social media. However, existing research mainly concentrates on style transfer within individual English sentences, while ignoring the complexity of long Chinese texts, which limits the wider applicability of style transfer in digital media realm. To bridge this gap, we propose a Chinese Article-style Transfer framework (CAT-LLM), leveraging the capabilities of Large Language Models (LLMs). CAT-LLM incorporates a bespoke, pluggable Text Style Definition (TSD) module aimed at comprehensively analyzing text features in articles, prompting LLMs to efficiently transfer Chinese article-style. The TSD module integrates a series of machine learning algorithms to analyze article-style from both words and sentences levels, thereby aiding LLMs thoroughly grasp the target style without compromising the integrity of the original text. In addition, this module supports dynamic expansion of internal style trees, showcasing robust compatibility and allowing flexible optimization in subsequent research. Moreover, we select five Chinese articles with distinct styles and create five parallel datasets using ChatGPT, enhancing the models' performance evaluation accuracy and establishing a novel paradigm for evaluating subsequent research on article-style transfer. Extensive experimental results affirm that CAT-LLM outperforms current research in terms of transfer accuracy and content preservation, and has remarkable applicability to various types of LLMs.

• 摘要：

  文本风格迁移在在线娱乐和社交媒体中越来越突出。然而，现有的研究主要集中在单个英语句子内的风格迁移，而忽略了中文长文本的复杂性，这限制了风格迁移在数字媒体领域的更广泛应用。为了弥补这一差距，我们提出了一个中文文章式传输框架（CAT-LLM），利用大型语言模型（LLM）的功能。 CAT-LLM采用了定制的、可插拔的文本风格定义（TSD）模块，旨在全面分析文章中的文本特征，促使法学硕士高效地迁移中文文章风格。 TSD模块集成了一系列机器学习算法，从单词和句子层面分析文章风格，从而帮助LLM在不损害原文完整性的情况下彻底掌握目标风格。此外，该模块支持内部样式树的动态扩展，具有强大的兼容性，并允许后续研究中的灵活优化。此外，我们选择了五篇风格独特的中文文章，并使用 ChatGPT 创建了五个并行数据集，提高了模型性能评估的准确性，并为评估后续文章风格迁移研究建立了一个新的范式。大量的实验结果证实，CAT-LLM在传输准确性和内容保存方面优于当前研究，并且对各种类型的LLM具有显着的适用性。

Title: Zero Resource Cross-Lingual Part Of Speech Tagging. (arXiv:2401.05727v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.05727
• Code URL: null
• Copy Paste: [[2401.05727]] Zero Resource Cross-Lingual Part Of Speech Tagging(http://arxiv.org/abs/2401.05727)
• Summary:

  Part of speech tagging in zero-resource settings can be an effective approach for low-resource languages when no labeled training data is available. Existing systems use two main techniques for POS tagging i.e. pretrained multilingual large language models(LLM) or project the source language labels into the zero resource target language and train a sequence labeling model on it. We explore the latter approach using the off-the-shelf alignment module and train a hidden Markov model(HMM) to predict the POS tags. We evaluate transfer learning setup with English as a source language and French, German, and Spanish as target languages for part-of-speech tagging. Our conclusion is that projected alignment data in zero-resource language can be beneficial to predict POS tags.

• 摘要：

  当没有可用的标记训练数据时，零资源设置中的词性标记可能是低资源语言的有效方法。现有系统使用两种主要的词性标注技术，即预训练的多语言大语言模型（LLM）或将源语言标签投影到零资源目标语言并在其上训练序列标注模型。我们使用现成的对齐模块探索后一种方法，并训练隐马尔可夫模型（HMM）来预测 POS 标签。我们以英语作为源语言，以法语、德语和西班牙语作为词性标记的目标语言来评估迁移学习设置。我们的结论是，零资源语言的投影对齐数据有助于预测 POS 标签。

Title: Probing Structured Semantics Understanding and Generation of Language Models via Question Answering. (arXiv:2401.05777v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.05777
• Code URL: null
• Copy Paste: [[2401.05777]] Probing Structured Semantics Understanding and Generation of Language Models via Question Answering(http://arxiv.org/abs/2401.05777)
• Summary:

  Recent advancement in the capabilities of large language models (LLMs) has triggered a new surge in LLMs' evaluation. Most recent evaluation works tends to evaluate the comprehensive ability of LLMs over series of tasks. However, the deep structure understanding of natural language is rarely explored. In this work, we examine the ability of LLMs to deal with structured semantics on the tasks of question answering with the help of the human-constructed formal language. Specifically, we implement the inter-conversion of natural and formal language through in-context learning of LLMs to verify their ability to understand and generate the structured logical forms. Extensive experiments with models of different sizes and in different formal languages show that today's state-of-the-art LLMs' understanding of the logical forms can approach human level overall, but there still are plenty of room in generating correct logical forms, which suggest that it is more effective to use LLMs to generate more natural language training data to reinforce a small model than directly answering questions with LLMs. Moreover, our results also indicate that models exhibit considerable sensitivity to different formal languages. In general, the formal language with the lower the formalization level, i.e. the more similar it is to natural language, is more LLMs-friendly.

• 摘要：

  最近大型语言模型 (LLM) 功能的进步引发了 LLM 评估的新一轮激增。最近的评估工作倾向于评估法学硕士在一系列任务上的综合能力。然而，自然语言的深层结构理解却很少被探索。在这项工作中，我们研究了法学硕士在人类构建的形式语言的帮助下处理问答任务中的结构化语义的能力。具体来说，我们通过法学硕士的情境学习实现自然语言和形式语言的相互转换，以验证他们理解和生成结构化逻辑形式的能力。对不同规模和不同形式语言的模型进行的大量实验表明，当今最先进的法学硕士对逻辑形式的理解总体上可以接近人类水平，但在生成正确的逻辑形式方面仍然有很大的空间，这表明使用法学硕士生成更多自然语言训练数据来强化小型模型比直接使用法学硕士回答问题更有效。此外，我们的结果还表明模型对不同的形式语言表现出相当的敏感性。一般来说，形式化程度越低的形式语言，即与自然语言越相似，就越适合法学硕士。

Title: Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems. (arXiv:2401.05778v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.05778
• Code URL: null
• Copy Paste: [[2401.05778]] Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems(http://arxiv.org/abs/2401.05778)
• Summary:

  Large language models (LLMs) have strong capabilities in solving diverse natural language processing tasks. However, the safety and security issues of LLM systems have become the major obstacle to their widespread application. Many studies have extensively investigated risks in LLM systems and developed the corresponding mitigation strategies. Leading-edge enterprises such as OpenAI, Google, Meta, and Anthropic have also made lots of efforts on responsible LLMs. Therefore, there is a growing need to organize the existing studies and establish comprehensive taxonomies for the community. In this paper, we delve into four essential modules of an LLM system, including an input module for receiving prompts, a language model trained on extensive corpora, a toolchain module for development and deployment, and an output module for exporting LLM-generated content. Based on this, we propose a comprehensive taxonomy, which systematically analyzes potential risks associated with each module of an LLM system and discusses the corresponding mitigation strategies. Furthermore, we review prevalent benchmarks, aiming to facilitate the risk assessment of LLM systems. We hope that this paper can help LLM participants embrace a systematic perspective to build their responsible LLM systems.

• 摘要：

  大型语言模型（LLM）在解决各种自然语言处理任务方面具有强大的能力。然而，LLM系统的安全保障问题已成为其广泛应用的主要障碍。许多研究广泛调查了法学硕士系统的风险并制定了相应的缓解策略。 OpenAI、Google、Meta、Anthropic等前沿企业也在负责任的LLM方面做出了很多努力。因此，越来越需要组织现有的研究并为社区建立全面的分类法。在本文中，我们深入研究了LLM系统的四个基本模块，包括用于接收提示的输入模块、在广泛语料库上训练的语言模型、用于开发和部署的工具链模块以及用于导出LLM生成内容的输出模块。在此基础上，我们提出了一个全面的分类法，系统地分析了LLM系统每个模块相关的潜在风险，并讨论了相应的缓解策略。此外，我们审查流行的基准，旨在促进法学硕士系统的风险评估。我们希望本文能够帮助LLM参与者以系统的视角来构建他们负责任的LLM系统。

Title: Towards Boosting Many-to-Many Multilingual Machine Translation with Large Language Models. (arXiv:2401.05861v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.05861
• Code URL: https://github.com/gpengzhi/crossconst-llm
• Copy Paste: [[2401.05861]] Towards Boosting Many-to-Many Multilingual Machine Translation with Large Language Models(http://arxiv.org/abs/2401.05861)
• Summary:

  The training paradigm for machine translation has gradually shifted, from learning neural machine translation (NMT) models with extensive parallel corpora to instruction finetuning on pretrained multilingual large language models (LLMs) with high-quality translation pairs. In this paper, we focus on boosting the many-to-many multilingual translation performance of LLMs with an emphasis on zero-shot translation directions. We demonstrate that prompt strategies adopted during instruction finetuning are crucial to zero-shot translation performance and introduce a cross-lingual consistency regularization, XConST, to bridge the representation gap among different languages and improve zero-shot translation performance. XConST is not a new method, but a version of CrossConST (Gao et al., 2023a) adapted for multilingual finetuning on LLMs with translation instructions. Experimental results on ALMA (Xu et al., 2023) and LLaMA-2 (Touvron et al., 2023) show that our approach consistently improves translation performance. Our implementations are available at https://github.com/gpengzhi/CrossConST-LLM.

• 摘要：

  机器翻译的训练范式已逐渐转变，从学习具有广泛并行语料库的神经机器翻译 (NMT) 模型，到对具有高质量翻译对的预训练多语言大语言模型 (LLM) 进行指令微调。在本文中，我们专注于提高法学硕士的多对多多语言翻译性能，重点是零样本翻译方向。我们证明，在指令微调期间采用的即时策略对于零样本翻译性能至关重要，并引入跨语言一致性正则化 XConST，以弥合不同语言之间的表示差距并提高零样本翻译性能。 XConST 不是一种新方法，而是 CrossConST（Gao 等人，2023a）的一个版本，适用于带有翻译指令的 LLM 多语言微调。 ALMA (Xu et al., 2023) 和 LLaMA-2 (Touvron et al., 2023) 的实验结果表明，我们的方法持续提高了翻译性能。我们的实现可在 https://github.com/gpengzhi/CrossConST-LLM 获取。

Title: EpilepsyLLM: Domain-Specific Large Language Model Fine-tuned with Epilepsy Medical Knowledge. (arXiv:2401.05908v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.05908
• Code URL: null
• Copy Paste: [[2401.05908]] EpilepsyLLM: Domain-Specific Large Language Model Fine-tuned with Epilepsy Medical Knowledge(http://arxiv.org/abs/2401.05908)
• Summary:

  With large training datasets and massive amounts of computing sources, large language models (LLMs) achieve remarkable performance in comprehensive and generative ability. Based on those powerful LLMs, the model fine-tuned with domain-specific datasets posseses more specialized knowledge and thus is more practical like medical LLMs. However, the existing fine-tuned medical LLMs are limited to general medical knowledge with English language. For disease-specific problems, the model's response is inaccurate and sometimes even completely irrelevant, especially when using a language other than English. In this work, we focus on the particular disease of Epilepsy with Japanese language and introduce a customized LLM termed as EpilepsyLLM. Our model is trained from the pre-trained LLM by fine-tuning technique using datasets from the epilepsy domain. The datasets contain knowledge of basic information about disease, common treatment methods and drugs, and important notes in life and work. The experimental results demonstrate that EpilepsyLLM can provide more reliable and specialized medical knowledge responses.

• 摘要：

  凭借庞大的训练数据集和海量的计算源，大型语言模型（LLM）在综合能力和生成能力方面表现出色。基于这些强大的LLM，用特定领域的数据集进行微调的模型拥有更专业的知识，因此比医学LLM更实用。然而，现有的微调医学法学硕士仅限于英语语言的一般医学知识。对于特定疾病的问题，模型的响应不准确，有时甚至完全无关，尤其是在使用英语以外的语言时。在这项工作中，我们专注于日语中的癫痫这一特殊疾病，并引入了名为 EpilepsyLLM 的定制法学硕士。我们的模型是通过使用癫痫领域的数据集进行微调技术，从预训练的法学硕士进行训练的。数据集包含疾病的基本信息、常用治疗方法和药物的知识以及生活和工作中的重要注意事项。实验结果表明EpilepsyLLM可以提供更可靠、更专业的医学知识应答。

Title: How Teachers Can Use Large Language Models and Bloom's Taxonomy to Create Educational Quizzes. (arXiv:2401.05914v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.05914
• Code URL: null
• Copy Paste: [[2401.05914]] How Teachers Can Use Large Language Models and Bloom's Taxonomy to Create Educational Quizzes(http://arxiv.org/abs/2401.05914)
• Summary:

  Question generation (QG) is a natural language processing task with an abundance of potential benefits and use cases in the educational domain. In order for this potential to be realized, QG systems must be designed and validated with pedagogical needs in mind. However, little research has assessed or designed QG approaches with the input from real teachers or students. This paper applies a large language model-based QG approach where questions are generated with learning goals derived from Bloom's taxonomy. The automatically generated questions are used in multiple experiments designed to assess how teachers use them in practice. The results demonstrate that teachers prefer to write quizzes with automatically generated questions, and that such quizzes have no loss in quality compared to handwritten versions. Further, several metrics indicate that automatically generated questions can even improve the quality of the quizzes created, showing the promise for large scale use of QG in the classroom setting.

• 摘要：

  问题生成 (QG) 是一项自然语言处理任务，在教育领域具有大量潜在优势和用例。为了实现这一潜力，QG 系统的设计和验证必须考虑到教学需求。然而，很少有研究根据真实教师或学生的意见来评估或设计 QG 方法。本文应用了基于大型语言模型的 QG 方法，其中问题是根据从 Bloom 分类法得出的学习目标生成的。自动生成的问题用于多个实验，旨在评估教师在实践中如何使用它们。结果表明，教师更喜欢使用自动生成的问题编写测验，并且与手写版本相比，此类测验在质量上没有损失。此外，一些指标表明，自动生成的问题甚至可以提高所创建测验的质量，这表明 QG 在课堂环境中大规模使用的前景。

Title: SH2: Self-Highlighted Hesitation Helps You Decode More Truthfully. (arXiv:2401.05930v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.05930
• Code URL: null
• Copy Paste: [[2401.05930]] SH2: Self-Highlighted Hesitation Helps You Decode More Truthfully(http://arxiv.org/abs/2401.05930)
• Summary:

  Large language models (LLMs) demonstrate great performance in text generation. However, LLMs are still suffering from hallucinations. In this work, we propose an inference-time method, Self-Highlighted Hesitation (SH2), to help LLMs decode more truthfully. SH2 is based on a simple fact rooted in information theory that for an LLM, the tokens predicted with lower probabilities are prone to be more informative than others. Our analysis shows that the tokens assigned with lower probabilities by an LLM are more likely to be closely related to factual information, such as nouns, proper nouns, and adjectives. Therefore, we propose to ''highlight'' the factual information by selecting the tokens with the lowest probabilities and concatenating them to the original context, thus forcing the model to repeatedly read and hesitate on these tokens before generation. During decoding, we also adopt contrastive decoding to emphasize the difference in the output probabilities brought by the hesitation. Experimental results demonstrate that our SH2, requiring no additional data or models, can effectively help LLMs elicit factual knowledge and distinguish hallucinated contexts. Significant and consistent improvements are achieved by SH2 for LLaMA-7b and LLaMA2-7b on multiple hallucination tasks.

• 摘要：

  大型语言模型 (LLM) 在文本生成方面表现出了出色的性能。然而，法学硕士仍然饱受幻觉之苦。在这项工作中，我们提出了一种推理时间方法，自我突出的犹豫（SH2），以帮助法学硕士更真实地解码。 SH2 基于一个植根于信息论的简单事实，即对于法学硕士来说，以较低概率预测的标记往往比其他标记提供更多信息。我们的分析表明，法学硕士分配的概率较低的标记更有可能与事实信息密切相关，例如名词、专有名词和形容词。因此，我们建议通过选择概率最低的标记并将它们连接到原始上下文来“突出”事实信息，从而迫使模型在生成之前反复读取和犹豫这些标记。在解码过程中，我们还采用对比解码来强调犹豫带来的输出概率的差异。实验结果表明，我们的 SH2 不需要额外的数据或模型，可以有效地帮助法学硕士获得事实知识并区分幻觉背景。 SH2 对 LLaMA-7b 和 LLaMA2-7b 在多个幻觉任务上取得了显着且一致的改进。

Title: DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models. (arXiv:2401.06066v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.06066
• Code URL: https://github.com/deepseek-ai/deepseek-moe
• Copy Paste: [[2401.06066]] DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models(http://arxiv.org/abs/2401.06066)
• Summary:

  In the era of large language models, Mixture-of-Experts (MoE) is a promising architecture for managing computational costs when scaling up model parameters. However, conventional MoE architectures like GShard, which activate the top-$K$ out of $N$ experts, face challenges in ensuring expert specialization, i.e. each expert acquires non-overlapping and focused knowledge. In response, we propose the DeepSeekMoE architecture towards ultimate expert specialization. It involves two principal strategies: (1) finely segmenting the experts into $mN$ ones and activating $mK$ from them, allowing for a more flexible combination of activated experts; (2) isolating $K_s$ experts as shared ones, aiming at capturing common knowledge and mitigating redundancy in routed experts. Starting from a modest scale with 2B parameters, we demonstrate that DeepSeekMoE 2B achieves comparable performance with GShard 2.9B, which has 1.5 times the expert parameters and computation. In addition, DeepSeekMoE 2B nearly approaches the performance of its dense counterpart with the same number of total parameters, which set the upper bound of MoE models. Subsequently, we scale up DeepSeekMoE to 16B parameters and show that it achieves comparable performance with LLaMA2 7B, with only about 40% of computations. Further, our preliminary efforts to scale up DeepSeekMoE to 145B parameters consistently validate its substantial advantages over the GShard architecture, and show its performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.

• 摘要：

  在大型语言模型时代，专家混合 (MoE) 是一种很有前途的架构，用于在扩展模型参数时管理计算成本。然而，传统的 MoE 架构（如 GShard）激活了 $N$ 专家中的顶级 $K$，在确保专家专业化方面面临着挑战，即每个专家都获得不重叠且有针对性的知识。作为回应，我们提出了 DeepSeekMoE 架构，以实现最终的专家专业化。它涉及两个主要策略：（1）将专家精细分割为$mN$个专家，并从中激活$mK$，从而允许更灵活的激活专家组合； (2) 将$K_s$专家隔离为共享专家，旨在捕获共同知识并减少路由专家中的冗余。从具有 2B 参数的适度规模开始，我们证明 DeepSeekMoE 2B 实现了与 GShard 2.9B 相当的性能，后者的参数和计算量是专家参数和计算的 1.5 倍。此外，DeepSeekMoE 2B 在总参数数量相同的情况下几乎接近其密集对应模型的性能，这设定了 MoE 模型的上限。随后，我们将 DeepSeekMoE 扩展到 16B 参数，并表明它实现了与 LLaMA2 7B 相当的性能，而计算量仅为约 40%。此外，我们将 DeepSeekMoE 扩展到 145B 参数的初步努力一致验证了其相对于 GShard 架构的巨大优势，并显示其性能与 DeepSeek 67B 相当，仅使用 28.5%（甚至可能是 18.2%）的计算量。

Title: Improving Large Language Models via Fine-grained Reinforcement Learning with Minimum Editing Constraint. (arXiv:2401.06081v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.06081
• Code URL: null
• Copy Paste: [[2401.06081]] Improving Large Language Models via Fine-grained Reinforcement Learning with Minimum Editing Constraint(http://arxiv.org/abs/2401.06081)
• Summary:

  Reinforcement learning (RL) has been widely used in training large language models~(LLMs) for preventing unexpected outputs, \eg reducing harmfulness and errors. However, existing RL methods mostly adopt the instance-level reward, which is unable to provide fine-grained supervision for complex reasoning tasks, and can not focus on the few key tokens that lead to the incorrectness. To address it, we propose a new RL method named \textbf{RLMEC} that incorporates a generative model as the reward model, which is trained by the erroneous solution rewriting task under the minimum editing constraint, and can produce token-level rewards for RL training. Based on the generative reward model, we design the token-level RL objective for training and an imitation-based regularization for stabilizing RL process. And the both objectives focus on the learning of the key tokens for the erroneous solution, reducing the effect of other unimportant tokens. The experiment results on mathematical tasks and question-answering tasks have demonstrated the effectiveness of our approach. Our code and data are available at \url{https://github.com/RUCAIBox/RLMEC}.

• 摘要：

  强化学习（RL）已广泛用于训练大型语言模型〜（LLM）以防止意外输出，例如减少危害和错误。然而，现有的强化学习方法大多采用实例级奖励，无法为复杂的推理任务提供细粒度的监督，也无法关注导致错误的少数关键标记。为了解决这个问题，我们提出了一种名为 \textbf{RLMEC} 的新 RL 方法，该方法将生成模型作为奖励模型，在最小编辑约束下通过错误解重写任务进行训练，并且可以为 RL 产生 token 级奖励训练。基于生成奖励模型，我们设计了用于训练的代币级强化学习目标，以及用于稳定强化学习过程的基于模仿的正则化。这两个目标都集中在学习错误解决方案的关键标记，减少其他不重要标记的影响。数学任务和问答任务的实验结果证明了我们方法的有效性。我们的代码和数据可在 \url{https://github.com/RUCAIBox/RLMEC} 获取。

Title: Autocompletion of Chief Complaints in the Electronic Health Records using Large Language Models. (arXiv:2401.06088v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.06088
• Code URL: null
• Copy Paste: [[2401.06088]] Autocompletion of Chief Complaints in the Electronic Health Records using Large Language Models(http://arxiv.org/abs/2401.06088)
• Summary:

  The Chief Complaint (CC) is a crucial component of a patient's medical record as it describes the main reason or concern for seeking medical care. It provides critical information for healthcare providers to make informed decisions about patient care. However, documenting CCs can be time-consuming for healthcare providers, especially in busy emergency departments. To address this issue, an autocompletion tool that suggests accurate and well-formatted phrases or sentences for clinical notes can be a valuable resource for triage nurses. In this study, we utilized text generation techniques to develop machine learning models using CC data. In our proposed work, we train a Long Short-Term Memory (LSTM) model and fine-tune three different variants of Biomedical Generative Pretrained Transformers (BioGPT), namely microsoft/biogpt, microsoft/BioGPT-Large, and microsoft/BioGPT-Large-PubMedQA. Additionally, we tune a prompt by incorporating exemplar CC sentences, utilizing the OpenAI API of GPT-4. We evaluate the models' performance based on the perplexity score, modified BERTScore, and cosine similarity score. The results show that BioGPT-Large exhibits superior performance compared to the other models. It consistently achieves a remarkably low perplexity score of 1.65 when generating CC, whereas the baseline LSTM model achieves the best perplexity score of 170. Further, we evaluate and assess the proposed models' performance and the outcome of GPT-4.0. Our study demonstrates that utilizing LLMs such as BioGPT, leads to the development of an effective autocompletion tool for generating CC documentation in healthcare settings.

• 摘要：

  主诉 (CC) 是患者医疗记录的重要组成部分，因为它描述了寻求医疗护理的主要原因或担忧。它为医疗保健提供者提供关键信息，以做出有关患者护理的明智决策。然而，对于医疗保健提供者来说，记录 CC 可能非常耗时，尤其是在繁忙的急诊科。为了解决这个问题，自动完成工具可以为临床记录提供准确且格式良好的短语或句子，这对于分诊护士来说可能是宝贵的资源。在本研究中，我们利用文本生成技术来开发使用 CC 数据的机器学习模型。在我们提出的工作中，我们训练了一个长短期记忆（LSTM）模型，并对生物医学生成预训练变压器（BioGPT）的三种不同变体进行了微调，即 microsoft/biogpt、microsoft/BioGPT-Large 和 microsoft/BioGPT-Large -PubMedQA。此外，我们还利用 GPT-4 的 OpenAI API，通过合并示例 CC 句子来调整提示。我们根据困惑度得分、修改后的 BERTScore 和余弦相似度得分来评估模型的性能。结果表明，与其他模型相比，BioGPT-Large 表现出优越的性能。在生成 CC 时，它始终实现了 1.65 的非常低的困惑度分数，而基线 LSTM 模型实现了 170 的最佳困惑度分数。此外，我们评估和评估了所提出的模型的性能和 GPT-4.0 的结果。我们的研究表明，利用 BioGPT 等法学硕士，可以开发出一种有效的自动完成工具，用于在医疗保健环境中生成 CC 文档。

Title: Extreme Compression of Large Language Models via Additive Quantization. (arXiv:2401.06118v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.06118
• Code URL: null
• Copy Paste: [[2401.06118]] Extreme Compression of Large Language Models via Additive Quantization(http://arxiv.org/abs/2401.06118)
• Summary:

  The emergence of accurate open large language models (LLMs) has led to a race towards quantization techniques for such models enabling execution on end-user devices. In this paper, we revisit the problem of "extreme" LLM compression--defined as targeting extremely low bit counts, such as 2 to 3 bits per parameter, from the point of view of classic methods in Multi-Codebook Quantization (MCQ). Our work builds on top of Additive Quantization, a classic algorithm from the MCQ family, and adapts it to the quantization of language models. The resulting algorithm advances the state-of-the-art in LLM compression, outperforming all recently-proposed techniques in terms of accuracy at a given compression budget. For instance, when compressing Llama 2 models to 2 bits per parameter, our algorithm quantizes the 7B model to 6.93 perplexity (a 1.29 improvement relative to the best prior work, and 1.81 points from FP16), the 13B model to 5.70 perplexity (a .36 improvement) and the 70B model to 3.94 perplexity (a .22 improvement) on WikiText2. We release our implementation of Additive Quantization for Language Models AQLM as a baseline to facilitate future research in LLM quantization.

• 摘要：

  准确的开放式大语言模型 (LLM) 的出现引发了一场针对此类模型的量化技术的竞赛，这些技术可在最终用户设备上执行。在本文中，我们从多码本量化 (MCQ) 中的经典方法的角度重新审视“极端”LLM 压缩问题——定义为针对极低的位数，例如每个参数 2 到 3 位。我们的工作建立在加性量化（MCQ 系列的经典算法）之上，并使其适应语言模型的量化。由此产生的算法推进了 LLM 压缩的最先进技术，在给定压缩预算的精度方面优于所有最近提出的技术。例如，当将 Llama 2 模型压缩到每个参数 2 位时，我们的算法将 7B 模型量化为 6.93 困惑度（相对于之前最好的工作提高了 1.29，与 FP16 相比提高了 1.81 点），将 13B 模型量化为 5.70 困惑度（a . 36 改进）和 WikiText2 上的 70B 模型到 3.94 困惑度（0.22 改进）。我们发布了语言模型 AQLM 的加性量化的实现作为基准，以促进 LLM 量化的未来研究。

segmentation

Title: AutoVisual Fusion Suite: A Comprehensive Evaluation of Image Segmentation and Voice Conversion Tools on HuggingFace Platform. (arXiv:2401.05379v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.05379
• Code URL: null
• Copy Paste: [[2401.05379]] AutoVisual Fusion Suite: A Comprehensive Evaluation of Image Segmentation and Voice Conversion Tools on HuggingFace Platform(http://arxiv.org/abs/2401.05379)
• Summary:

  This study presents a comprehensive evaluation of tools available on the HuggingFace platform for two pivotal applications in artificial intelligence: image segmentation and voice conversion. The primary objective was to identify the top three tools within each category and subsequently install and configure these tools on Linux systems. We leveraged the power of pre-trained segmentation models such as SAM and DETR Model with ResNet-50 backbone for image segmentation, and the so-vits-svc-fork model for voice conversion. This paper delves into the methodologies and challenges encountered during the implementation process, and showcases the successful combination of video segmentation and voice conversion in a unified project named AutoVisual Fusion Suite.

• 摘要：

  这项研究对 HuggingFace 平台上可用的工具进行了全面评估，适用于人工智能的两个关键应用：图像分割和语音转换。主要目标是确定每个类别中排名前三的工具，然后在 Linux 系统上安装和配置这些工具。我们利用预先训练的分割模型（例如具有 ResNet-50 主干的 SAM 和 DETR 模型）的强大功能来进行图像分割，并利用 so-vits-svc-fork 模型进行语音转换。本文深入探讨了实施过程中遇到的方法和挑战，并展示了视频分割和语音转换在名为 AutoVisual Fusion Suite 的统一项目中的成功结合。

Title: PartSTAD: 2D-to-3D Part Segmentation Task Adaptation. (arXiv:2401.05906v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.05906
• Code URL: null
• Copy Paste: [[2401.05906]] PartSTAD: 2D-to-3D Part Segmentation Task Adaptation(http://arxiv.org/abs/2401.05906)
• Summary:

  We introduce PartSTAD, a method designed for the task adaptation of 2D-to-3D segmentation lifting. Recent studies have highlighted the advantages of utilizing 2D segmentation models to achieve high-quality 3D segmentation through few-shot adaptation. However, previous approaches have focused on adapting 2D segmentation models for domain shift to rendered images and synthetic text descriptions, rather than optimizing the model specifically for 3D segmentation. Our proposed task adaptation method finetunes a 2D bounding box prediction model with an objective function for 3D segmentation. We introduce weights for 2D bounding boxes for adaptive merging and learn the weights using a small additional neural network. Additionally, we incorporate SAM, a foreground segmentation model on a bounding box, to improve the boundaries of 2D segments and consequently those of 3D segmentation. Our experiments on the PartNet-Mobility dataset show significant improvements with our task adaptation approach, achieving a 7.0%p increase in mIoU and a 5.2%p improvement in mAP_50 for semantic and instance segmentation compared to the SotA few-shot 3D segmentation model.

• 摘要：

  我们介绍了 PartSTAD，一种专为 2D 到 3D 分割提升的任务适配而设计的方法。最近的研究强调了利用 2D 分割模型通过少样本自适应实现高质量 3D 分割的优势。然而，以前的方法侧重于调整 2D 分割模型以将域转移到渲染图像和合成文本描述，而不是专门针对 3D 分割优化模型。我们提出的任务适应方法使用 3D 分割的目标函数微调 2D 边界框预测模型。我们引入了用于自适应合并的 2D 边界框的权重，并使用小型附加神经网络来学习权重。此外，我们还结合了 SAM（边界框上的前景分割模型），以改善 2D 分割的边界，从而改善 3D 分割的边界。我们在 PartNet-Mobility 数据集上的实验表明，我们的任务适应方法有了显着改进，与 SotA 少样本 3D 分割模型相比，语义和实例分割的 mIoU 提高了 7.0%p，mAP_50 提高了 5.2%p。