2024-01-11

secure

security

Title: REACT: Autonomous Intrusion Response System for Intelligent Vehicles. (arXiv:2401.04792v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2401.04792
• Code URL: null
• Copy Paste: [[2401.04792]] REACT: Autonomous Intrusion Response System for Intelligent Vehicles(http://arxiv.org/abs/2401.04792)
• Summary:

  Autonomous and connected vehicles are rapidly evolving, integrating numerous technologies and software. This progress, however, has made them appealing targets for cybersecurity attacks. As the risk of cyber threats escalates with this advancement, the focus is shifting from solely preventing these attacks to also mitigating their impact. Current solutions rely on vehicle security operation centers, where attack information is analyzed before deciding on a response strategy. However, this process can be time-consuming and faces scalability challenges, along with other issues stemming from vehicle connectivity. This paper proposes a dynamic intrusion response system integrated within the vehicle. This system enables the vehicle to respond to a variety of incidents almost instantly, thereby reducing the need for interaction with the vehicle security operation center. The system offers a comprehensive list of potential responses, a methodology for response evaluation, and various response selection methods. The proposed solution was implemented on an embedded platform. Two distinct cyberattack use cases served as the basis for evaluating the system. The evaluation highlights the system's adaptability, its ability to respond swiftly, its minimal memory footprint, and its capacity for dynamic system parameter adjustments. The proposed solution underscores the necessity and feasibility of incorporating dynamic response mechanisms in smart vehicles. This is a crucial factor in ensuring the safety and resilience of future smart mobility.

• 摘要：

  自动驾驶和联网车辆正在迅速发展，集成了大量技术和软件。然而，这一进展使它们成为网络安全攻击的有吸引力的目标。随着这一进步，网络威胁的风险不断升级，重点也 从仅仅防止这些攻击转向减轻其影响。当前的解决方案依赖于车辆安全运营中心，在决定响应策略之前分析攻击信息。然而，这个过程可能非常耗时，并且面临可扩展性挑战，以及 车辆连接带来的其他问题。本文提出了一种集成在车辆内的动态入侵响应系统。该系统使车辆能够几乎立即响应各种事件，从而减少与车辆安全运营中心交互的需要。该系统提供了 潜在响应的综合列表、响应评估方法以及各种响应选择方法。所提出的解决方案是在嵌入式平台上实现的。两个不同的网络攻击用例作为评估系统的基础。评估强调了系统的适应性 、快速响应的能力、最小的内存占用以及动态系统参数调整的能力。所提出的解决方案强调了在智能车辆中纳入动态响应机制的必要性和可行性。这是确保未来智能出行的安全性和 弹性的关键因素。

privacy

Title: Source-Free Cross-Modal Knowledge Transfer by Unleashing the Potential of Task-Irrelevant Data. (arXiv:2401.05014v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.05014
• Code URL: null
• Copy Paste: [[2401.05014]] Source-Free Cross-Modal Knowledge Transfer by Unleashing the Potential of Task-Irrelevant Data(http://arxiv.org/abs/2401.05014)
• Summary:

  Source-free cross-modal knowledge transfer is a crucial yet challenging task, which aims to transfer knowledge from one source modality (e.g., RGB) to the target modality (e.g., depth or infrared) with no access to the task-relevant (TR) source data due to memory and privacy concerns. A recent attempt leverages the paired task-irrelevant (TI) data and directly matches the features from them to eliminate the modality gap. However, it ignores a pivotal clue that the paired TI data could be utilized to effectively estimate the source data distribution and better facilitate knowledge transfer to the target modality. To this end, we propose a novel yet concise framework to unlock the potential of paired TI data for enhancing source-free cross-modal knowledge transfer. Our work is buttressed by two key technical components. Firstly, to better estimate the source data distribution, we introduce a Task-irrelevant data-Guided Modality Bridging (TGMB) module. It translates the target modality data (e.g., infrared) into the source-like RGB images based on paired TI data and the guidance of the available source model to alleviate two key gaps: 1) inter-modality gap between the paired TI data; 2) intra-modality gap between TI and TR target data. We then propose a Task-irrelevant data-Guided Knowledge Transfer (TGKT) module that transfers knowledge from the source model to the target model by leveraging the paired TI data. Notably, due to the unavailability of labels for the TR target data and its less reliable prediction from the source model, our TGKT model incorporates a self-supervised pseudo-labeling approach to enable the target model to learn from its predictions. Extensive experiments show that our method achieves state-of-the-art performance on three datasets (RGB-to-depth and RGB-to-infrared).

• 摘要：

  无源跨模态知识转移是一项至关重要但具有挑战性的任务，其目的是将知识从一种源模态（例如 RGB）转移到目标模态（例如深度或红外），而无需访问任务 -由于内存和隐私问题，相关（TR）源数据。最近的一项尝试利用配对的任务无关（TI）数据并直接匹配其中的特征以消除模态差距。然而，它忽略了一个关键线索，即配对的 TI 数据可用于有效估计源数据分布并更好地促进知识向目标模态的迁移。为此，我们提出了一个新颖而简洁的框架，以释放配对 TI 数据的潜力，以增强无源跨模式知识转 移。我们的工作由两个关键技术组成部分支撑。首先，为了更好地估计源数据分布，我们引入了任务无关数据引导模态桥接（TGMB）模块。它基于配对 TI 数据和可用源模型的指导，将目标模态数据（例如红外）转换为类源 RGB 图像，以缩小两个关键差距：1）配对 TI 数据之间的模态间差距； 2）TI和TR目标数据之间的模态内差距。然后，我们提出了一个与任务无关的数据引导知识转移（TGKT）模块，该模块通过利用配对的 TI 数据将知识从源模型转移到目标模型。值得注意的是，由于 TR 目标数据的标签不可用，并且源模型的预测不太可靠，我们的 TGKT 模型采用了自我监督的伪标签方法，使目标模型能够从其预测中学习。大量实验表明，我们的方法在三个数据集（RGB 到深度和 RGB 到红外）上实现了最先进的性能。

Title: Efficient Fine-Tuning with Domain Adaptation for Privacy-Preserving Vision Transformer. (arXiv:2401.05126v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.05126
• Code URL: null
• Copy Paste: [[2401.05126]] Efficient Fine-Tuning with Domain Adaptation for Privacy-Preserving Vision Transformer(http://arxiv.org/abs/2401.05126)
• Summary:

  We propose a novel method for privacy-preserving deep neural networks (DNNs) with the Vision Transformer (ViT). The method allows us not only to train models and test with visually protected images but to also avoid the performance degradation caused from the use of encrypted images, whereas conventional methods cannot avoid the influence of image encryption. A domain adaptation method is used to efficiently fine-tune ViT with encrypted images. In experiments, the method is demonstrated to outperform conventional methods in an image classification task on the CIFAR-10 and ImageNet datasets in terms of classification accuracy.

• 摘要：

  我们提出了一种使用 Vision Transformer (ViT) 保护隐私的深度神经网络 (DNN) 的新方法。该方法不仅使我们能够使用受视觉保护的 图像来训练模型和进行测试，而且还可以避免由于使用加密图像而导致的性能下降，而传统方法无法避免图像加密的影响。域适应方法用于有效地微调加密图像的 ViT。在实验中，该方法在 CIFAR-10 和 ImageNet 数据集上的图像分类任务中被证明在分类精度方面优于传统方法。

protect

defense

attack

Title: Exploring Vulnerabilities of No-Reference Image Quality Assessment Models: A Query-Based Black-Box Method. (arXiv:2401.05217v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.05217
• Code URL: null
• Copy Paste: [[2401.05217]] Exploring Vulnerabilities of No-Reference Image Quality Assessment Models: A Query-Based Black-Box Method(http://arxiv.org/abs/2401.05217)
• Summary:

  No-Reference Image Quality Assessment (NR-IQA) aims to predict image quality scores consistent with human perception without relying on pristine reference images, serving as a crucial component in various visual tasks. Ensuring the robustness of NR-IQA methods is vital for reliable comparisons of different image processing techniques and consistent user experiences in recommendations. The attack methods for NR-IQA provide a powerful instrument to test the robustness of NR-IQA. However, current attack methods of NR-IQA heavily rely on the gradient of the NR-IQA model, leading to limitations when the gradient information is unavailable. In this paper, we present a pioneering query-based black box attack against NR-IQA methods. We propose the concept of \emph{score boundary} and leverage an adaptive iterative approach with multiple score boundaries. Meanwhile, the initial attack directions are also designed to leverage the characteristics of the Human Visual System (HVS). Experiments show our attack method outperforms all compared state-of-the-art methods and is far ahead of previous black-box methods. The effective DBCNN model suffers a Spearman rank-order correlation coefficient (SROCC) decline of $0.6972$ attacked by our method, revealing the vulnerability of NR-IQA to black-box attacks. The proposed attack method also provides a potent tool for further exploration into NR-IQA robustness.

• 摘要：

  无参考图像质量评估 (NR-IQA) 旨在预测与人类感知一致的图像质量分数，而不依赖原始参考图像，作为各种视觉任务的关键组成部分。确保 NR-IQA 方法的稳健性对于不同图像处理技术的可靠比较和推荐中一致的用户体验至关重要。 NR-IQA的攻击方法为测试NR- IQA的鲁棒性提供了强大的工具。然而，目前的NR-IQA攻击方法严重依赖NR-IQA模型的梯度，导致在梯度信息不可用时受到限制。在本文中，我们提出了一种针对 NR-IQA 方法的开创性的基于查询的黑盒攻击。我们提出了\emph{分数边界}的概念，并利用具有多个分数边界的自适应迭代方法。同时，最初的攻击方向也被设计为 利用人类视觉系统（HVS）的特性。实验表明，我们的攻击方法优于所有比较的最先进方法，并且远远领先于以前的黑盒方法。有效的 DBCNN 模型受到我们的方法的攻击，Spearman 排序相关系数 (SROCC) 下降了 0.6972 美元，这揭示了 NR-IQA 容易受到黑盒攻击。所提出的攻击方法还为进一步探索 NR-IQA 鲁棒性提供了有效的工具。

Title: Learning-Based Difficulty Calibration for Enhanced Membership Inference Attacks. (arXiv:2401.04929v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2401.04929
• Code URL: https://github.com/horanshi/ldc-mia
• Copy Paste: [[2401.04929]] Learning-Based Difficulty Calibration for Enhanced Membership Inference Attacks(http://arxiv.org/abs/2401.04929)
• Summary:

  Machine learning models, in particular deep neural networks, are currently an integral part of various applications, from healthcare to finance. However, using sensitive data to train these models raises concerns about privacy and security. One method that has emerged to verify if the trained models are privacy-preserving is Membership Inference Attacks (MIA), which allows adversaries to determine whether a specific data point was part of a model's training dataset. While a series of MIAs have been proposed in the literature, only a few can achieve high True Positive Rates (TPR) in the low False Positive Rate (FPR) region (0.01%~1%). This is a crucial factor to consider for an MIA to be practically useful in real-world settings. In this paper, we present a novel approach to MIA that is aimed at significantly improving TPR at low FPRs. Our method, named learning-based difficulty calibration for MIA(LDC-MIA), characterizes data records by their hardness levels using a neural network classifier to determine membership. The experiment results show that LDC-MIA can improve TPR at low FPR by up to 4x compared to the other difficulty calibration based MIAs. It also has the highest Area Under ROC curve (AUC) across all datasets. Our method's cost is comparable with most of the existing MIAs, but is orders of magnitude more efficient than one of the state-of-the-art methods, LiRA, while achieving similar performance.

• 摘要：

  机器学习模型，特别是深度神经网络，目前是从医疗保健到金融等各种应用程序不可或缺的一部分。然而，使用敏感数据来训练这些模型会引起人们对隐私和安全的担忧。验 证训练模型是否隐私保护的一种方法是成员推理攻击 (MIA)，它允许攻击者确定特定数据点是否是模型训练数据集的一部分。虽然文献中已经提出了一系列 MIA，但只有少数可以在低假阳性率 (FPR) 区域 (0.01%~1%) 内实现高真阳性率 (TPR)。这是 MIA 在现实环境中发挥实际作用时需要考虑的一个关键因素。在本文中，我们提出了一种新的 MIA 方法，旨在显着提高低 FPR 下的 TPR。我们的方法称为基于学习的 MIA 难度校准（LDC-MIA），它使用神经网络分类器根据数据记录的硬度级别来表征数据记录以确定隶属度。实验结果表明，与其他基于难度校准的 MIA 相比，LDC-MIA 可以将低 FPR 下的 TPR 提高高达 4 倍。它还具有所有数据集中最高的 ROC 曲线下面积 (AUC)。我们的方法的成本与大多数现有的 MIA 相当，但比最先进的方法之一 LiRA 的效率高几个数量级，同时实现了类似的性能。

Title: FBSDetector: Fake Base Station and Multi Step Attack Detection in Cellular Networks using Machine Learning. (arXiv:2401.04958v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2401.04958
• Code URL: null
• Copy Paste: [[2401.04958]] FBSDetector: Fake Base Station and Multi Step Attack Detection in Cellular Networks using Machine Learning(http://arxiv.org/abs/2401.04958)
• Summary:

  Fake base stations (FBSes) pose a significant security threat by impersonating legitimate base stations. Though efforts have been made to defeat this threat, up to this day, the presence of FBSes and the multi-step attacks (MSAs) stemming from them can lead to unauthorized surveillance, interception of sensitive information, and disruption of network services for legitimate users. Therefore, detecting these malicious entities is crucial to ensure the security and reliability of cellular networks. Traditional detection methods often rely on additional hardware, predefined rules, signal scanning, changing protocol specifications, or cryptographic mechanisms that have limitations and incur huge infrastructure costs in accurately identifying FBSes. In this paper, we develop FBSDetector-an effective and efficient detection solution that can reliably detect FBSes and MSAs from layer-3 network traces using machine learning (ML) at the user equipment (UE) side. To develop FBSDetector, we created FBSAD and MSAD, the first-ever high-quality and large-scale datasets for training machine learning models capable of detecting FBSes and MSAs. These datasets capture the network traces in different real-world cellular network scenarios (including mobility and different attacker capabilities) incorporating legitimate base stations and FBSes. The combined network trace has a volume of 6.6 GB containing 751963 packets. Our novel ML models, specially designed to detect FBSes and MSAs, can effectively detect FBSes with an accuracy of 92% and a false positive rate of 5.96% and recognize MSAs with an accuracy of 86% and a false positive rate of 7.82%. We deploy FBSDetector as a real-world solution to protect end-users through an Android app and validate in a controlled lab environment. Compared to the existing solutions that fail to detect FBSes, FBSDetector can detect FBSes in the wild in real time.

• 摘要：

  假基站 (FBS) 通过冒充合法基站构成重大安全威胁。尽管我们已努力应对这一威胁，但迄今为止，FBS 的存在以及由此产生的多步攻击 (MSA) 可能会导 致未经授权的监视、敏感信息拦截以及合法用户的网络服务中断。因此，检测这些恶意实体对于确保蜂窝网络的安全性和可靠性至关重要。传统的检测方法通常依赖于额外的硬件、 预定义的规则、信号扫描、不断变化的协议规范或加密机制，这些机制具有局限性，并且在准确识别 FBS 方面会产生巨大的基础设施成本。在本文中，我们开发了 FBSDetector——一种有效且高效的检测解决方案，可以在用户设备 (UE) 侧使用机器学习 (ML) 可靠地从第 3 层网络跟踪中检测 FBS 和 MSA。为了开发 FBSDetector，我们创建了 FBSAD 和 MSAD，这是有史以来第一个高质量和大规模的数据集，用于训练能够检测 FBS 和 MSA 的机器学习模型。这些数据集捕获包含合法基站和 FBS 的不同现实蜂窝网络场景（包括移动性和不同的攻击者能力）中的网络痕迹。组合网络跟踪的容量为 6.6 GB，包含 751963 个数据包。我们的新型机器学习模型专门用于检测FBS和MSA，可以有效检测FBS，准确率为92％，误报率为5.96％，识别MSA的准确 率为86％，误报率为7.82％。我们将 FBSDetector 部署为现实世界的解决方案，通过 Android 应用程序保护最终用户并在受控实验室环境中进行验证。与现有无法检测FBS的解决方案相比，FBSDetector可以实时检测野外的FBS。

robust

Title: ECC-PolypDet: Enhanced CenterNet with Contrastive Learning for Automatic Polyp Detection. (arXiv:2401.04961v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.04961
• Code URL: https://github.com/yuncheng97/ecc-polypdet
• Copy Paste: [[2401.04961]] ECC-PolypDet: Enhanced CenterNet with Contrastive Learning for Automatic Polyp Detection(http://arxiv.org/abs/2401.04961)
• Summary:

  Accurate polyp detection is critical for early colorectal cancer diagnosis. Although remarkable progress has been achieved in recent years, the complex colon environment and concealed polyps with unclear boundaries still pose severe challenges in this area. Existing methods either involve computationally expensive context aggregation or lack prior modeling of polyps, resulting in poor performance in challenging cases. In this paper, we propose the Enhanced CenterNet with Contrastive Learning (ECC-PolypDet), a two-stage training \& end-to-end inference framework that leverages images and bounding box annotations to train a general model and fine-tune it based on the inference score to obtain a final robust model. Specifically, we conduct Box-assisted Contrastive Learning (BCL) during training to minimize the intra-class difference and maximize the inter-class difference between foreground polyps and backgrounds, enabling our model to capture concealed polyps. Moreover, to enhance the recognition of small polyps, we design the Semantic Flow-guided Feature Pyramid Network (SFFPN) to aggregate multi-scale features and the Heatmap Propagation (HP) module to boost the model's attention on polyp targets. In the fine-tuning stage, we introduce the IoU-guided Sample Re-weighting (ISR) mechanism to prioritize hard samples by adaptively adjusting the loss weight for each sample during fine-tuning. Extensive experiments on six large-scale colonoscopy datasets demonstrate the superiority of our model compared with previous state-of-the-art detectors.

• 摘要：

  准确的息肉检测对于早期结直肠癌诊断至关重要。尽管近年来取得了显着的进展，但复杂的结肠环境和边界不清晰的隐匿性息肉仍然给该领域带来严峻的挑战。现有方法要么 涉及计算成本高昂的上下文聚合，要么缺乏息肉的预先建模，导致在具有挑战性的情况下表现不佳。在本文中，我们提出了具有对比学习的增强型 CenterNet（ECC- PolypDet），这是一种两阶段训练\&端到端推理框架，利用图像和边界框注释来训练通用模型，并根据推理分数对其进行微调，以获得最终的鲁棒模型。具体来 说，我们在训练期间进行框辅助对比学习（BCL），以最小化类内差异并最大化前景息肉和背景之间的类间差异，使我们的模型能够捕获隐藏的息肉。此外，为了增强对小息肉的 识别，我们设计了语义流引导特征金字塔网络（SFFPN）来聚合多尺度特征，并设计了热图传播（HP）模块来提高模型对息肉目标的关注。在微调阶段，我们引入了 IoU 引导的样本重新加权（ISR）机制，通过在微调过程中自适应调整每个样本的损失权重来优先考虑硬样本。对六个大规模结肠镜检查数据集的广泛实验证明了我们的模型与以前最 先进的探测器相比的优越性。

Title: Structure from Duplicates: Neural Inverse Graphics from a Pile of Objects. (arXiv:2401.05236v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.05236
• Code URL: https://github.com/tianhang-cheng/sfd
• Copy Paste: [[2401.05236]] Structure from Duplicates: Neural Inverse Graphics from a Pile of Objects(http://arxiv.org/abs/2401.05236)
• Summary:

  Our world is full of identical objects (\emphe.g., cans of coke, cars of same model). These duplicates, when seen together, provide additional and strong cues for us to effectively reason about 3D. Inspired by this observation, we introduce Structure from Duplicates (SfD), a novel inverse graphics framework that reconstructs geometry, material, and illumination from a single image containing multiple identical objects. SfD begins by identifying multiple instances of an object within an image, and then jointly estimates the 6DoF pose for all instances.An inverse graphics pipeline is subsequently employed to jointly reason about the shape, material of the object, and the environment light, while adhering to the shared geometry and material constraint across instances. Our primary contributions involve utilizing object duplicates as a robust prior for single-image inverse graphics and proposing an in-plane rotation-robust Structure from Motion (SfM) formulation for joint 6-DoF object pose estimation. By leveraging multi-view cues from a single image, SfD generates more realistic and detailed 3D reconstructions, significantly outperforming existing single image reconstruction models and multi-view reconstruction approaches with a similar or greater number of observations.

• 摘要：

  我们的世界充满了相同的物体（例如，可乐罐、相同型号的汽车）。当将这些重复项放在一起查看时，它们为我们有效地推理 3D 提供了额外且强有力的线索。受这一观 察的启发，我们引入了重复结构（SfD），这是一种新颖的逆向图形框架，可以从包含多个相同对象的单个图像中重建几何形状、材质和照明。 SfD 首先识别图像中对象的多个实例，然后联合估计所有实例的 6DoF 姿态。随后采用逆图形管道来联合推理对象的形状、材质和环境光，同时遵循跨实例共享几何和材料约束。 我们的主要贡献包括利用对象重复作为单图像逆向图形的稳健先验，并提出用于联合 6-DoF 对象姿态估计的面内旋转稳健运动结构 (SfM) 公式。通过利用单个图像的多视图线索，SfD 生成更真实、更详细的 3D 重建，显着优于现有的单图像重建模型和具有相似或更多观测值的多视图重建方法。

Title: Hierarchical Classification of Transversal Skills in Job Ads Based on Sentence Embeddings. (arXiv:2401.05073v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.05073
• Code URL: null
• Copy Paste: [[2401.05073]] Hierarchical Classification of Transversal Skills in Job Ads Based on Sentence Embeddings(http://arxiv.org/abs/2401.05073)
• Summary:

  This paper proposes a classification framework aimed at identifying correlations between job ad requirements and transversal skill sets, with a focus on predicting the necessary skills for individual job descriptions using a deep learning model. The approach involves data collection, preprocessing, and labeling using ESCO (European Skills, Competences, and Occupations) taxonomy. Hierarchical classification and multi-label strategies are used for skill identification, while augmentation techniques address data imbalance, enhancing model robustness. A comparison between results obtained with English-specific and multi-language sentence embedding models reveals close accuracy. The experimental case studies detail neural network configurations, hyperparameters, and cross-validation results, highlighting the efficacy of the hierarchical approach and the suitability of the multi-language model for the diverse European job market. Thus, a new approach is proposed for the hierarchical classification of transversal skills from job ads.

• 摘要：

  本文提出了一个分类框架，旨在识别职位广告要求和横向技能组合之间的相关性，重点是使用深度学习模型预测个人职位描述的必要技能。该方法涉及使用 ESCO（欧洲 技能、能力和职业）分类法进行数据收集、预处理和标记。分层分类和多标签策略用于技能识别，而增强技术则解决数据不平衡问题，增强模型的稳健性。通过英语特定句子嵌入模 型和多语言句子嵌入模型获得的结果之间的比较显示出接近的准确性。实验案例研究详细介绍了神经网络配置、超参数和交叉验证结果，强调了分层方法的有效性以及多语言模型对 多样化欧洲就业市场的适用性。因此，提出了一种新方法对招聘广告中的横向技能进行层次分类。

Title: Structure-Preserving Physics-Informed Neural Networks With Energy or Lyapunov Structure. (arXiv:2401.04986v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.04986
• Code URL: null
• Copy Paste: [[2401.04986]] Structure-Preserving Physics-Informed Neural Networks With Energy or Lyapunov Structure(http://arxiv.org/abs/2401.04986)
• Summary:

  Recently, there has been growing interest in using physics-informed neural networks (PINNs) to solve differential equations. However, the preservation of structure, such as energy and stability, in a suitable manner has yet to be established. This limitation could be a potential reason why the learning process for PINNs is not always efficient and the numerical results may suggest nonphysical behavior. Besides, there is little research on their applications on downstream tasks. To address these issues, we propose structure-preserving PINNs to improve their performance and broaden their applications for downstream tasks. Firstly, by leveraging prior knowledge about the physical system, a structure-preserving loss function is designed to assist the PINN in learning the underlying structure. Secondly, a framework that utilizes structure-preserving PINN for robust image recognition is proposed. Here, preserving the Lyapunov structure of the underlying system ensures the stability of the system. Experimental results demonstrate that the proposed method improves the numerical accuracy of PINNs for partial differential equations. Furthermore, the robustness of the model against adversarial perturbations in image data is enhanced.

• 摘要：

  最近，人们对使用物理信息神经网络 (PINN) 求解微分方程越来越感兴趣。然而，以适当的方式保存结构，例如能量和稳定性，尚未建立。这种限制可能是 PINN 的学习过程并不总是有效且数值结果可能表明非物理行为的潜在原因。此外，关于它们在下游任务中的应用的研究还很少。为了解决这些问题，我们提出了保留结构的 PINN，以提高其性能并扩大其在下游任务中的应用。首先，通过利用物理系统的先验知识，设计了结构保持损失函数来帮助 PINN 学习底层结构。其次，提出了一种利用结构保留 PINN 进行鲁棒图像识别的框架。这里，保留底层系统的李雅普诺夫结构保证了系统的稳定性。实验结果表明，该方法提高了偏微分方程 PINN 的数值精度。此外，模型针对图像数据中的对抗性扰动的鲁棒性也得到了增强。

Title: ReACT: Reinforcement Learning for Controller Parametrization using B-Spline Geometries. (arXiv:2401.05251v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.05251
• Code URL: null
• Copy Paste: [[2401.05251]] ReACT: Reinforcement Learning for Controller Parametrization using B-Spline Geometries(http://arxiv.org/abs/2401.05251)
• Summary:

  Robust and performant controllers are essential for industrial applications. However, deriving controller parameters for complex and nonlinear systems is challenging and time-consuming. To facilitate automatic controller parametrization, this work presents a novel approach using deep reinforcement learning (DRL) with N-dimensional B-spline geometries (BSGs). We focus on the control of parameter-variant systems, a class of systems with complex behavior which depends on the operating conditions. For this system class, gain-scheduling control structures are widely used in applications across industries due to well-known design principles. Facilitating the expensive controller parametrization task regarding these control structures, we deploy an DRL agent. Based on control system observations, the agent autonomously decides how to adapt the controller parameters. We make the adaptation process more efficient by introducing BSGs to map the controller parameters which may depend on numerous operating conditions. To preprocess time-series data and extract a fixed-length feature vector, we use a long short-term memory (LSTM) neural networks. Furthermore, this work contributes actor regularizations that are relevant to real-world environments which differ from training. Accordingly, we apply dropout layer normalization to the actor and critic networks of the truncated quantile critic (TQC) algorithm. To show our approach's working principle and effectiveness, we train and evaluate the DRL agent on the parametrization task of an industrial control structure with parameter lookup tables.

• 摘要：

  稳健且高性能的控制器对于工业应用至关重要。然而，为复杂的非线性系统推导控制器参数具有挑战性且耗时。为了促进自动控制器参数化，这项工作提出了一种使用深度强 化学习 (DRL) 和 N 维 B 样条几何 (BSG) 的新颖方法。我们专注于参数变化系统的控制，这是一类具有取决于操作条件的复杂行为的系统。对于此类系统， 增益调度控制结构由于众所周知的设计原理而广泛应用于各行业的应用中。为了促进有关这些控制结构的昂贵的控制器参数化任务，我们部署了 DRL 代理。基于控制系统的观察，代理自主决定如何调整控制器参数。我们通过引入 BSG 来映射可能取决于多种操作条件的控制器参数，从而使适应过程更加高效。为了预处理时 间序列数据并提取固定长度的特征向量，我们使用长短期记忆（LSTM）神经网络。此外，这项工作还贡献了与不同于训练的现实环境相关的参与者正则化。因此，我们将 dropout 层归一化应用于截断分位数批评家 (TQC) 算法的参与者和批评家网络。为了展示我们方法的工作原理和有效性，我们使用参数查找表在工业控制结构的参数化任务上训练和评估 DRL 代理。

biometric

steal

Title: Phishing Website Detection through Multi-Model Analysis of HTML Content. (arXiv:2401.04820v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2401.04820
• Code URL: null
• Copy Paste: [[2401.04820]] Phishing Website Detection through Multi-Model Analysis of HTML Content(http://arxiv.org/abs/2401.04820)
• Summary:

  The way we communicate and work has changed significantly with the rise of the Internet. While it has opened up new opportunities, it has also brought about an increase in cyber threats. One common and serious threat is phishing, where cybercriminals employ deceptive methods to steal sensitive information.This study addresses the pressing issue of phishing by introducing an advanced detection model that meticulously focuses on HTML content. Our proposed approach integrates a specialized Multi-Layer Perceptron (MLP) model for structured tabular data and two pretrained Natural Language Processing (NLP) models for analyzing textual features such as page titles and content. The embeddings from these models are harmoniously combined through a novel fusion process. The resulting fused embeddings are then input into a linear classifier. Recognizing the scarcity of recent datasets for comprehensive phishing research, our contribution extends to the creation of an up-to-date dataset, which we openly share with the community. The dataset is meticulously curated to reflect real-life phishing conditions, ensuring relevance and applicability. The research findings highlight the effectiveness of the proposed approach, with the CANINE demonstrating superior performance in analyzing page titles and the RoBERTa excelling in evaluating page content. The fusion of two NLP and one MLP model,termed MultiText-LP, achieves impressive results, yielding a 96.80 F1 score and a 97.18 accuracy score on our research dataset. Furthermore, our approach outperforms existing methods on the CatchPhish HTML dataset, showcasing its efficacies.

• 摘要：

  随着互联网的兴起，我们的沟通和工作方式发生了巨大变化。虽然它开辟了新的机遇，但也带来了网络威胁的增加。网络钓鱼是一种常见且严重的威胁，网络犯罪分子采用欺 骗性方法窃取敏感信息。本研究通过引入一种专门针对 HTML 内容的高级检测模型来解决网络钓鱼的紧迫问题。我们提出的方法集成了用于结构化表格数据的专用多层感知器 （MLP）模型和两个用于分析页面标题和内容等文本特征的预训练自然语言处理（NLP）模型。这些模型的嵌入通过新颖的融合过程和谐地结合在一起。然后将所得的融合嵌入 输入到线性分类器中。认识到用于综合网络钓鱼研究的最新数据集的稀缺性，我们的贡献扩展到创建最新的数据集，并与社区公开共享。该数据集经过精心策划，以反映现实生活中 的网络钓鱼情况，确保相关性和适用性。研究结果强调了所提出方法的有效性，CANINE 在分析页面标题方面表现出卓越的性能，而 RoBERTa 在评估页面内容方面表现出色。两个 NLP 和一个 MLP 模型的融合（称为 MultiText-LP）取得了令人印象深刻的结果，在我们的研究数据集上获得了 96.80 的 F1 分数和 97.18 的准确度分数。此外，我们的方法在 CatchPhish HTML 数据集上的性能优于现有方法，展示了其功效。

extraction

Title: Large Model based Sequential Keyframe Extraction for Video Summarization. (arXiv:2401.04962v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.04962
• Code URL: null
• Copy Paste: [[2401.04962]] Large Model based Sequential Keyframe Extraction for Video Summarization(http://arxiv.org/abs/2401.04962)
• Summary:

  Keyframe extraction aims to sum up a video's semantics with the minimum number of its frames. This paper puts forward a Large Model based Sequential Keyframe Extraction for video summarization, dubbed LMSKE, which contains three stages as below. First, we use the large model "TransNetV21" to cut the video into consecutive shots, and employ the large model "CLIP2" to generate each frame's visual feature within each shot; Second, we develop an adaptive clustering algorithm to yield candidate keyframes for each shot, with each candidate keyframe locating nearest to a cluster center; Third, we further reduce the above candidate keyframes via redundancy elimination within each shot, and finally concatenate them in accordance with the sequence of shots as the final sequential keyframes. To evaluate LMSKE, we curate a benchmark dataset and conduct rich experiments, whose results exhibit that LMSKE performs much better than quite a few SOTA competitors with average F1 of 0.5311, average fidelity of 0.8141, and average compression ratio of 0.9922.

• 摘要：

  关键帧提取旨在用最少的帧数总结视频的语义。本文提出了一种基于大型模型的视频摘要序列关键帧提取，称为LMSKE，它包含以下三个阶段。首先，我们使用大模型“ TransNetV21”将视频切割成连续的镜头，并使用大模型“CLIP2”生成每个镜头内每一帧的视觉特征；其次，我们开发了一种自适应聚类算法，为每个镜头生成候 选关键帧，每个候选关键帧距离聚类中心最近；第三，我们通过每个镜头内的冗余消除进一步减少上述候选关键帧，最后按照镜头顺序将它们连接起来作为最终的顺序关键帧。为了 评估 LMSKE，我们整理了一个基准数据集并进行了丰富的实验，结果表明 LMSKE 的平均 F1 为 0.5311，平均保真度为 0.8141，平均压缩比为 0.9922，比相当多的 SOTA 竞争对手表现得更好。

Title: Entity Recognition from Colloquial Text. (arXiv:2401.04853v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.04853
• Code URL: null
• Copy Paste: [[2401.04853]] Entity Recognition from Colloquial Text(http://arxiv.org/abs/2401.04853)
• Summary:

  Extraction of concepts and entities of interest from non-formal texts such as social media posts and informal communication is an important capability for decision support systems in many domains, including healthcare, customer relationship management, and others. Despite the recent advances in training large language models for a variety of natural language processing tasks, the developed models and techniques have mainly focused on formal texts and do not perform as well on colloquial data, which is characterized by a number of distinct challenges. In our research, we focus on the healthcare domain and investigate the problem of symptom recognition from colloquial texts by designing and evaluating several training strategies for BERT-based model fine-tuning. These strategies are distinguished by the choice of the base model, the training corpora, and application of term perturbations in the training data. The best-performing models trained using these strategies outperform the state-of-the-art specialized symptom recognizer by a large margin. Through a series of experiments, we have found specific patterns of model behavior associated with the training strategies we designed. We present design principles for training strategies for effective entity recognition in colloquial texts based on our findings.

• 摘要：

  从社交媒体帖子和非正式沟通等非正式文本中提取感兴趣的概念和实体是许多领域（包括医疗保健、客户关系管理等）决策支持系统的一项重要功能。尽管最近在训练用于各 种自然语言处理任务的大型语言模型方面取得了进展，但开发的模型和技术主要集中在正式文本上，而在口语数据上表现不佳，口语数据的特点是存在许多独特的挑战。在我们的研 究中，我们专注于医疗保健领域，通过设计和评估几种基于 BERT 的模型微调的训练策略来研究口语文本的症状识别问题。这些策略的特点是基础模型、训练语料库的选择以 及训练数据中术语扰动的应用。使用这些策略训练的性能最佳的模型大大优于最先进的专门症状识别器。通过一系列实验，我们发现了与我们设计的训练策略相关的特定模型行为模 式。根据我们的发现，我们提出了口语文本中有效实体识别的训练策略的设计原则。

Title: HiMTM: Hierarchical Multi-Scale Masked Time Series Modeling for Long-Term Forecasting. (arXiv:2401.05012v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.05012
• Code URL: null
• Copy Paste: [[2401.05012]] HiMTM: Hierarchical Multi-Scale Masked Time Series Modeling for Long-Term Forecasting(http://arxiv.org/abs/2401.05012)
• Summary:

  Time series forecasting is crucial and challenging in the real world. The recent surge in interest regarding time series foundation models, which cater to a diverse array of downstream tasks, is noteworthy. However, existing methods often overlook the multi-scale nature of time series, an aspect crucial for precise forecasting. To bridge this gap, we propose HiMTM, a hierarchical multi-scale masked time series modeling method designed for long-term forecasting. Specifically, it comprises four integral components: (1) hierarchical multi-scale transformer (HMT) to capture temporal information at different scales; (2) decoupled encoder-decoder (DED) forces the encoder to focus on feature extraction, while the decoder to focus on pretext tasks; (3) multi-scale masked reconstruction (MMR) provides multi-stage supervision signals for pre-training; (4) cross-scale attention fine-tuning (CSA-FT) to capture dependencies between different scales for forecasting. Collectively, these components enhance multi-scale feature extraction capabilities in masked time series modeling and contribute to improved prediction accuracy. We conduct extensive experiments on 7 mainstream datasets to prove that HiMTM has obvious advantages over contemporary self-supervised and end-to-end learning methods. The effectiveness of HiMTM is further showcased by its application in the industry of natural gas demand forecasting.

• 摘要：

  时间序列预测在现实世界中至关重要且具有挑战性。值得注意的是，最近人们对时间序列基础模型的兴趣激增，这些模型可以满足各种下游任务的需求。然而，现有的方法常 常忽视时间序列的多尺度性质，而这对于精确预测至关重要。为了弥补这一差距，我们提出了 HiMTM，一种专为长期预测而设计的分层多尺度屏蔽时间序列建模方法。具体来 说，它包含四个组成部分：（1）分层多尺度变换器（HMT），用于捕获不同尺度的时间信息； （2）解耦编码器- 解码器（DED）迫使编码器专注于特征提取，而解码器专注于借口任务； （3）多尺度掩蔽重建（MMR）为预训练提供多级监督信号； （4）跨尺度注意力微调（CSA- FT）以捕获不同尺度之间的依赖关系进行预测。总的来说，这些组件增强了屏蔽时间序列建模中的多尺度特征提取能力，并有助于提高预测精度。我们在 7 个主流数据集上进行了广泛的实验，证明 HiMTM 相对于当代的自监督和端到端学习方法具有明显的优势。 HiMTM在天然气需求预测行业的应用进一步证明了其有效性。

membership infer

federate

Title: Federated Unlearning: A Survey on Methods, Design Guidelines, and Evaluation Metrics. (arXiv:2401.05146v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.05146
• Code URL: null
• Copy Paste: [[2401.05146]] Federated Unlearning: A Survey on Methods, Design Guidelines, and Evaluation Metrics(http://arxiv.org/abs/2401.05146)
• Summary:

  Federated Learning (FL) enables collaborative training of a Machine Learning (ML) model across multiple parties, facilitating the preservation of users' and institutions' privacy by keeping data stored locally. Instead of centralizing raw data, FL exchanges locally refined model parameters to build a global model incrementally. While FL is more compliant with emerging regulations such as the European General Data Protection Regulation (GDPR), ensuring the right to be forgotten in this context - allowing FL participants to remove their data contributions from the learned model - remains unclear. In addition, it is recognized that malicious clients may inject backdoors into the global model through updates, e.g. to generate mispredictions on specially crafted data examples. Consequently, there is the need for mechanisms that can guarantee individuals the possibility to remove their data and erase malicious contributions even after aggregation, without compromising the already acquired "good" knowledge. This highlights the necessity for novel Federated Unlearning (FU) algorithms, which can efficiently remove specific clients' contributions without full model retraining. This survey provides background concepts, empirical evidence, and practical guidelines to design/implement efficient FU schemes. Our study includes a detailed analysis of the metrics for evaluating unlearning in FL and presents an in-depth literature review categorizing state-of-the-art FU contributions under a novel taxonomy. Finally, we outline the most relevant and still open technical challenges, by identifying the most promising research directions in the field.

• 摘要：

  联邦学习 (FL) 支持跨多方协作训练机器学习 (ML) 模型，通过在本地存储数据来促进保护用户和机构的隐私。 FL 不是集中原始数据，而是交换本地精炼的模型参数，以增量方式构建全局模型。虽然 FL 更符合欧洲通用数据保护条例 (GDPR) 等新兴法规，但在这种情况下确保被遗忘权（允许 FL 参与者从学习模型中删除其数据贡献）仍不清楚。此外，人们认识到恶意客户端可能会通过更新将后门注入全局模型，例 如对特制的数据示例生成错误预测。因此，需要一种机制来保证个人即使在聚合后也有可能删除其数据并消除恶意贡献，而不会损害已经获得的“良好”知识。这凸显了新型联合取 消学习（FU）算法的必要性，该算法可以有效地消除特定客户的贡献，而无需进行完整的模型重新训练。这项调查提供了设计/实施高效 FU 方案的背景概念、经验证据和实用指南。我们的研究包括对评估 FL 遗忘的指标进行了详细分析，并提出了深入的文献综述，将最先进的 FU 贡献按照新颖的分类法进行了分类。最后，我们通过确定该领域最有前途的研究方向，概述了最相关且仍然开放的技术挑战。

Title: Relaxed Contrastive Learning for Federated Learning. (arXiv:2401.04928v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.04928
• Code URL: null
• Copy Paste: [[2401.04928]] Relaxed Contrastive Learning for Federated Learning(http://arxiv.org/abs/2401.04928)
• Summary:

  We propose a novel contrastive learning framework to effectively address the challenges of data heterogeneity in federated learning. We first analyze the inconsistency of gradient updates across clients during local training and establish its dependence on the distribution of feature representations, leading to the derivation of the supervised contrastive learning (SCL) objective to mitigate local deviations. In addition, we show that a na\"ive adoption of SCL in federated learning leads to representation collapse, resulting in slow convergence and limited performance gains. To address this issue, we introduce a relaxed contrastive learning loss that imposes a divergence penalty on excessively similar sample pairs within each class. This strategy prevents collapsed representations and enhances feature transferability, facilitating collaborative training and leading to significant performance improvements. Our framework outperforms all existing federated learning approaches by huge margins on the standard benchmarks through extensive experimental results.

• 摘要：

  我们提出了一种新颖的对比学习框架，以有效解决联邦学习中数据异构性的挑战。我们首先分析本地训练期间客户端之间梯度更新的不一致，并建立其对特征表示分布的依赖 性，从而导出监督对比学习（SCL）目标以减轻局部偏差。此外，我们还表明，在联邦学习中天真地采用 SCL 会导致表示崩溃，从而导致收敛缓慢和性能增益有限。为了解 决这个问题，我们引入了一种宽松的对比学习损失，该损失对过度学习施加发散惩罚。每个类中都有相似的样本对。这种策略可以防止折叠表示并增强特征可转移性，促进协作训练 并显着提高性能。通过大量的实验结果，我们的框架在标准基准上大幅优于所有现有的联邦学习方法。

Title: AdaFed: Fair Federated Learning via Adaptive Common Descent Direction. (arXiv:2401.04993v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.04993
• Code URL: null
• Copy Paste: [[2401.04993]] AdaFed: Fair Federated Learning via Adaptive Common Descent Direction(http://arxiv.org/abs/2401.04993)
• Summary:

  Federated learning (FL) is a promising technology via which some edge devices/clients collaboratively train a machine learning model orchestrated by a server. Learning an unfair model is known as a critical problem in federated learning, where the trained model may unfairly advantage or disadvantage some of the devices. To tackle this problem, in this work, we propose AdaFed. The goal of AdaFed is to find an updating direction for the server along which (i) all the clients' loss functions are decreasing; and (ii) more importantly, the loss functions for the clients with larger values decrease with a higher rate. AdaFed adaptively tunes this common direction based on the values of local gradients and loss functions. We validate the effectiveness of AdaFed on a suite of federated datasets, and demonstrate that AdaFed outperforms state-of-the-art fair FL methods.

• 摘要：

  联邦学习 (FL) 是一项很有前景的技术，一些边缘设备/客户端可以通过该技术协作训练由服务器编排的机器学习模型。学习不公平的模型被认为是联邦学习中的一个 关键问题，其中经过训练的模型可能会不公平地使某些设备受益或不利。为了解决这个问题，在这项工作中，我们提出了 AdaFed。 AdaFed 的目标是找到服务器的更新方向，沿着该方向（i）所有客户端的损失函数都在减小； (ii)更重要的是，具有较大值的客户的损失函数以较高的速率下降。 AdaFed 根据局部梯度和损失函数的值自适应地调整这个共同方向。我们在一套联合数据集上验证了 AdaFed 的有效性，并证明 AdaFed 优于最先进的公平 FL 方法。

fair

interpretability

Title: ANGO: A Next-Level Evaluation Benchmark For Generation-Oriented Language Models In Chinese Domain. (arXiv:2401.04898v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.04898
• Code URL: null
• Copy Paste: [[2401.04898]] ANGO: A Next-Level Evaluation Benchmark For Generation-Oriented Language Models In Chinese Domain(http://arxiv.org/abs/2401.04898)
• Summary:

  Recently, various Large Language Models (LLMs) evaluation datasets have emerged, but most of them have issues with distorted rankings and difficulty in model capabilities analysis. Addressing these concerns, this paper introduces ANGO, a Chinese multi-choice question evaluation benchmark. ANGO proposes \textit{Keypoint} categorization standard for the first time, each question in ANGO can correspond to multiple keypoints, effectively enhancing interpretability of evaluation results. Base on performance of real humans, we build a quantifiable question difficulty standard and divide ANGO questions into 9 difficulty levels, which provide more precise guidance for model training. To minimize data leakage impact and fully leverage ANGO's innovative features, we have engineered exclusive sampling strategies and a new evaluation framework that support swift testset iteration. Our experiments demonstrate that ANGO poses a stronger challenge to models and reveals more details in evaluation result compared to existing benchmarks.

• 摘要：

  近年来，各种大型语言模型（LLM）评估数据集不断涌现，但大多存在排名扭曲、模型能力分析困难等问题。针对这些问题，本文介绍了中国多项选择题评估基准 ANGO。 ANGO首次提出\textit{Keypoint}分类标准，ANGO中每个问题可以对应多个关键点，有效增强评估结果的可解释性。基于真人的表现，我们 建立了可量化的问题难度标准，将ANGO问题分为9个难度级别，为模型训练提供更精准的指导。为了最大限度地减少数据泄露的影响并充分利用 ANGO 的创新功能，我们设计了独家采样策略和支持快速测试集迭代的新评估框架。我们的实验表明，与现有基准相比，ANGO 对模型提出了更强的挑战，并且在评估结果中揭示了更多细节。

Title: Transportation Market Rate Forecast Using Signature Transform. (arXiv:2401.04857v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.04857
• Code URL: null
• Copy Paste: [[2401.04857]] Transportation Market Rate Forecast Using Signature Transform(http://arxiv.org/abs/2401.04857)
• Summary:

  Currently, Amazon relies on third parties for transportation marketplace rate forecasts, despite the poor quality and lack of interpretability of these forecasts. While transportation marketplace rates are typically very challenging to forecast accurately, we have developed a novel signature-based statistical technique to address these challenges and built a predictive and adaptive model to forecast marketplace rates. This novel technique is based on two key properties of the signature transform. The first is its universal nonlinearity which linearizes the feature space and hence translates the forecasting problem into a linear regression analysis; the second is the signature kernel which allows for comparing computationally efficiently similarities between time series data. Combined, these properties allow for efficient feature generation and more precise identification of seasonality and regime switching in the forecasting process. Preliminary result by the model shows that this new technique leads to far superior forecast accuracy versus commercially available industry models with better interpretability, even during the period of Covid-19 and with the sudden onset of the Ukraine war.

• 摘要：

  目前，亚马逊依赖第三方进行运输市场费率预测，尽管这些预测的质量很差且缺乏可解释性。虽然准确预测交通市场费率通常非常困难，但我们开发了一种新颖的基于签名的 统计技术来应对这些挑战，并建立了预测和自适应模型来预测市场费率。这项新技术基于签名变换的两个关键属性。首先是它的普遍非线性，它使特征空间线性化，从而将预测问题 转化为线性回归分析；第二个是签名内核，它允许在计算上有效地比较时间序列数据之间的相似性。结合起来，这些属性可以在预测过程中实现高效的特征生成以及更精确地识别季 节性和状态切换。该模型的初步结果表明，即使在 Covid-19 期间和乌克兰战争突然爆发期间，与商业上可用的行业模型相比，这种新技术的预测精度要高得多，并且具有更好的可解释性。

Title: MISS: Multiclass Interpretable Scoring Systems. (arXiv:2401.05069v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.05069
• Code URL: https://github.com/sanoscience/miss
• Copy Paste: [[2401.05069]] MISS: Multiclass Interpretable Scoring Systems(http://arxiv.org/abs/2401.05069)
• Summary:

  In this work, we present a novel, machine-learning approach for constructing Multiclass Interpretable Scoring Systems (MISS) - a fully data-driven methodology for generating single, sparse, and user-friendly scoring systems for multiclass classification problems. Scoring systems are commonly utilized as decision support models in healthcare, criminal justice, and other domains where interpretability of predictions and ease of use are crucial. Prior methods for data-driven scoring, such as SLIM (Supersparse Linear Integer Model), were limited to binary classification tasks and extensions to multiclass domains were primarily accomplished via one-versus-all-type techniques. The scores produced by our method can be easily transformed into class probabilities via the softmax function. We demonstrate techniques for dimensionality reduction and heuristics that enhance the training efficiency and decrease the optimality gap, a measure that can certify the optimality of the model. Our approach has been extensively evaluated on datasets from various domains, and the results indicate that it is competitive with other machine learning models in terms of classification performance metrics and provides well-calibrated class probabilities.

• 摘要：

  在这项工作中，我们提出了一种新颖的机器学习方法，用于构建多类可解释评分系统 (MISS) - 一种完全数据驱动的方法，用于为多类分类问题生成单一、稀疏且 用户友好的评分系统。评分系统通常用作医疗保健、刑事司法和其他领域的决策支持模型，在这些领域中，预测的可解释性和易用性至关重要。先前的数据驱动评分方法，例如 SLIM（超稀疏线性整数模型），仅限于二元分类任务，而对多类域的扩展主要通过“一对一”技术来完成。我们的方法产生的分数可以通过 softmax 函数轻松转换为 类别概率。我们展示了降维和启发式技术，可以提高训练效率并减少最优性差距，这是一种可以证明模型最优性的措施。我们的方法已经在各个领域的数据集上进行了广泛的评估， 结果表明它在分类性能指标方面与其他机器学习模型具有竞争力，并提供了经过良好校准的类别概率。

explainability

watermark

Title: Watermark Text Pattern Spotting in Document Images. (arXiv:2401.05167v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.05167
• Code URL: null
• Copy Paste: [[2401.05167]] Watermark Text Pattern Spotting in Document Images(http://arxiv.org/abs/2401.05167)
• Summary:

  Watermark text spotting in document images can offer access to an often unexplored source of information, providing crucial evidence about a record's scope, audience and sometimes even authenticity. Stemming from the problem of text spotting, detecting and understanding watermarks in documents inherits the same hardships - in the wild, writing can come in various fonts, sizes and forms, making generic recognition a very difficult problem. To address the lack of resources in this field and propel further research, we propose a novel benchmark (K-Watermark) containing 65,447 data samples generated using Wrender, a watermark text patterns rendering procedure. A validity study using humans raters yields an authenticity score of 0.51 against pre-generated watermarked documents. To prove the usefulness of the dataset and rendering technique, we developed an end-to-end solution (Wextract) for detecting the bounding box instances of watermark text, while predicting the depicted text. To deal with this specific task, we introduce a variance minimization loss and a hierarchical self-attention mechanism. To the best of our knowledge, we are the first to propose an evaluation benchmark and a complete solution for retrieving watermarks from documents surpassing baselines by 5 AP points in detection and 4 points in character accuracy.

• 摘要：

  文档图像中的水印文本识别可以提供对经常未探索的信息源的访问，提供有关记录范围、受众甚至有时甚至真实性的重要证据。源于文本识别的问题，检测和理解文档中的水 印也面临着同样的困难——在野外，书写可能有各种字体、大小和形式，这使得通用识别成为一个非常困难的问题。为了解决该领域资源缺乏的问题并推动进一步研究，我们提出了 一个新颖的基准（K-Watermark），其中包含使用 Wrender（一种水印文本模式渲染程序）生成的 65,447 个数据样本。使用人类评分者进行的有效性研究相对于预先生成的带水印文档的真实性得分为 0.51。为了证明数据集和渲染技术的有用性，我们开发了一种端到端解决方案（ Wextract），用于检测水印文本的边界框实例，同时预测所描绘的文本。为了处理这个特定任务，我们引入了方差最小化损失和分层自注意力机制。据我们所知，我们是第 一个提出评估基准和完整的解决方案，用于从检测超过基线 5 个 AP 点、字符准确度超过基线 4 点的文档中检索水印。

diffusion

Title: Diffusion-based Pose Refinement and Muti-hypothesis Generation for 3D Human Pose Estimaiton. (arXiv:2401.04921v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.04921
• Code URL: null
• Copy Paste: [[2401.04921]] Diffusion-based Pose Refinement and Muti-hypothesis Generation for 3D Human Pose Estimaiton(http://arxiv.org/abs/2401.04921)
• Summary:

  Previous probabilistic models for 3D Human Pose Estimation (3DHPE) aimed to enhance pose accuracy by generating multiple hypotheses. However, most of the hypotheses generated deviate substantially from the true pose. Compared to deterministic models, the excessive uncertainty in probabilistic models leads to weaker performance in single-hypothesis prediction. To address these two challenges, we propose a diffusion-based refinement framework called DRPose, which refines the output of deterministic models by reverse diffusion and achieves more suitable multi-hypothesis prediction for the current pose benchmark by multi-step refinement with multiple noises. To this end, we propose a Scalable Graph Convolution Transformer (SGCT) and a Pose Refinement Module (PRM) for denoising and refining. Extensive experiments on Human3.6M and MPI-INF-3DHP datasets demonstrate that our method achieves state-of-the-art performance on both single and multi-hypothesis 3DHPE. Code is available at https://github.com/KHB1698/DRPose.

• 摘要：

  之前的 3D 人体姿势估计 (3DHPE) 概率模型旨在通过生成多个假设来提高姿势准确性。然而，大多数生成的假设与真实姿势有很大偏差。与确定性模型相比， 概率模型过多的不确定性导致单假设预测的性能较差。为了解决这两个挑战，我们提出了一种名为 DRPose 的基于扩散的细化框架，该框架通过反向扩散细化确定性模型的 输出，并通过使用多个噪声的多步细化实现对当前姿态基准更合适的多假设预测。为此，我们提出了可扩展图卷积变换器（SGCT）和用于去噪和细化的姿势细化模块（PRM） 。在 Human3.6M 和 MPI-INF-3DHP 数据集上进行的大量实验表明，我们的方法在单假设和多假设 3DHPE 上均实现了最先进的性能。代码可在 https://github.com/KHB1698/DRPost 获取。

Title: SwiMDiff: Scene-wide Matching Contrastive Learning with Diffusion Constraint for Remote Sensing Image. (arXiv:2401.05093v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.05093
• Code URL: null
• Copy Paste: [[2401.05093]] SwiMDiff: Scene-wide Matching Contrastive Learning with Diffusion Constraint for Remote Sensing Image(http://arxiv.org/abs/2401.05093)
• Summary:

  With recent advancements in aerospace technology, the volume of unlabeled remote sensing image (RSI) data has increased dramatically. Effectively leveraging this data through self-supervised learning (SSL) is vital in the field of remote sensing. However, current methodologies, particularly contrastive learning (CL), a leading SSL method, encounter specific challenges in this domain. Firstly, CL often mistakenly identifies geographically adjacent samples with similar semantic content as negative pairs, leading to confusion during model training. Secondly, as an instance-level discriminative task, it tends to neglect the essential fine-grained features and complex details inherent in unstructured RSIs. To overcome these obstacles, we introduce SwiMDiff, a novel self-supervised pre-training framework designed for RSIs. SwiMDiff employs a scene-wide matching approach that effectively recalibrates labels to recognize data from the same scene as false negatives. This adjustment makes CL more applicable to the nuances of remote sensing. Additionally, SwiMDiff seamlessly integrates CL with a diffusion model. Through the implementation of pixel-level diffusion constraints, we enhance the encoder's ability to capture both the global semantic information and the fine-grained features of the images more comprehensively. Our proposed framework significantly enriches the information available for downstream tasks in remote sensing. Demonstrating exceptional performance in change detection and land-cover classification tasks, SwiMDiff proves its substantial utility and value in the field of remote sensing.

• 摘要：

  随着航空航天技术的最新进步，未标记的遥感图像 (RSI) 数据量急剧增加。通过自我监督学习（SSL）有效利用这些数据在遥感领域至关重要。然而，当前的方法，特别是对比学习（CL）（一种领先的 SSL 方法），在该领域遇 到了特定的挑战。首先，CL经常错误地将具有相似语义内容的地理上相邻的样本识别为负对，导致模型训练过程中的混乱。其次，作为实例级判别任务，它往往忽略非结构化 RSI 固有的细粒度特征和复杂细节。为了克服这些障碍，我们引入了 SwiMDiff，这是一种专为 RSI 设计的新型自监督预训练框架。 SwiMDiff 采用场景范围匹配方法，可以有效地重新校准标签，以将来自同一场景的数据识别为漏报。这一调整使 CL 更适用于遥感的细微差别。此外，SwiMDiff 将 CL 与扩散模型无缝集成。通过实施像素级扩散约束，我们增强了编码器更全面地捕获图像的全局语义信息和细粒度特征的能力。我们提出的框架显着丰富了遥感下游任务可用的信息。 SwiMDiff 在变化检测和土地覆盖分类任务中表现出卓越的性能，证明了其在遥感领域的巨大实用性和价值。

Title: CrossDiff: Exploring Self-Supervised Representation of Pansharpening via Cross-Predictive Diffusion Model. (arXiv:2401.05153v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.05153
• Code URL: null
• Copy Paste: [[2401.05153]] CrossDiff: Exploring Self-Supervised Representation of Pansharpening via Cross-Predictive Diffusion Model(http://arxiv.org/abs/2401.05153)
• Summary:

  Fusion of a panchromatic (PAN) image and corresponding multispectral (MS) image is also known as pansharpening, which aims to combine abundant spatial details of PAN and spectral information of MS. Due to the absence of high-resolution MS images, available deep-learning-based methods usually follow the paradigm of training at reduced resolution and testing at both reduced and full resolution. When taking original MS and PAN images as inputs, they always obtain sub-optimal results due to the scale variation. In this paper, we propose to explore the self-supervised representation of pansharpening by designing a cross-predictive diffusion model, named CrossDiff. It has two-stage training. In the first stage, we introduce a cross-predictive pretext task to pre-train the UNet structure based on conditional DDPM, while in the second stage, the encoders of the UNets are frozen to directly extract spatial and spectral features from PAN and MS, and only the fusion head is trained to adapt for pansharpening task. Extensive experiments show the effectiveness and superiority of the proposed model compared with state-of-the-art supervised and unsupervised methods. Besides, the cross-sensor experiments also verify the generalization ability of proposed self-supervised representation learners for other satellite's datasets. We will release our code for reproducibility.

• 摘要：

  全色（PAN）图像和相应的多光谱（MS）图像的融合也称为全色锐化，其目的是将PAN的丰富空间细节和MS的光谱信息结合起来。由于缺乏高分辨率 MS 图像，可用的基于深度学习的方法通常遵循降低分辨率训练以及降低分辨率和全分辨率测试的范式。当以原始 MS 和 PAN 图像作为输入时，由于尺度变化，它们总是获得次优结果。在本文中，我们建议通过设计一个名为 CrossDiff 的交叉预测扩散模型来探索全色锐化的自监督表示。它有 两个阶段的训练。在第一阶段，我们引入交叉预测借口任务来基于条件DDPM预训练UNet结构，而在第二阶段，UNet的编码器被冻结以直接从PAN和MS中提取空间和 光谱特征，并且只有融合头经过训练以适应全色锐化任务。大量的实验表明，与最先进的监督和无监督方法相比，所提出的模型的有效性和优越性。此外，跨传感器实验还验证了所 提出的自监督表示学习器对其他卫星数据集的泛化能力。我们将发布我们的代码以实现可重复性。

Title: Derm-T2IM: Harnessing Synthetic Skin Lesion Data via Stable Diffusion Models for Enhanced Skin Disease Classification using ViT and CNN. (arXiv:2401.05159v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.05159
• Code URL: null
• Copy Paste: [[2401.05159]] Derm-T2IM: Harnessing Synthetic Skin Lesion Data via Stable Diffusion Models for Enhanced Skin Disease Classification using ViT and CNN(http://arxiv.org/abs/2401.05159)
• Summary:

  This study explores the utilization of Dermatoscopic synthetic data generated through stable diffusion models as a strategy for enhancing the robustness of machine learning model training. Synthetic data generation plays a pivotal role in mitigating challenges associated with limited labeled datasets, thereby facilitating more effective model training. In this context, we aim to incorporate enhanced data transformation techniques by extending the recent success of few-shot learning and a small amount of data representation in text-to-image latent diffusion models. The optimally tuned model is further used for rendering high-quality skin lesion synthetic data with diverse and realistic characteristics, providing a valuable supplement and diversity to the existing training data. We investigate the impact of incorporating newly generated synthetic data into the training pipeline of state-of-art machine learning models, assessing its effectiveness in enhancing model performance and generalization to unseen real-world data. Our experimental results demonstrate the efficacy of the synthetic data generated through stable diffusion models helps in improving the robustness and adaptability of end-to-end CNN and vision transformer models on two different real-world skin lesion datasets.

• 摘要：

  本研究探索利用通过稳定扩散模型生成的皮肤镜合成数据作为增强机器学习模型训练稳健性的策略。合成数据生成在缓解与有限标记数据集相关的挑战方面发挥着关键作用， 从而促进更有效的模型训练。在这种背景下，我们的目标是通过扩展最近成功的少样本学习和文本到图像潜在扩散模型中的少量数据表示来整合增强的数据转换技术。经过优化调整 的模型进一步用于渲染具有多样化和真实特征的高质量皮肤病变合成数据，为现有训练数据提供了有价值的补充和多样性。我们研究了将新生成的合成数据纳入最先进的机器学习模 型的训练流程中的影响，评估其在增强模型性能和对未见过的现实世界数据的泛化方面的有效性。我们的实验结果表明，通过稳定扩散模型生成的合成数据的有效性有助于提高端到 端 CNN 和视觉变换器模型在两个不同的真实皮肤病变数据集上的鲁棒性和适应性。

Title: PIXART-{\delta}: Fast and Controllable Image Generation with Latent Consistency Models. (arXiv:2401.05252v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.05252
• Code URL: null
• Copy Paste: [[2401.05252]] PIXART-{\delta}: Fast and Controllable Image Generation with Latent Consistency Models(http://arxiv.org/abs/2401.05252)
• Summary:

  This technical report introduces PIXART-{\delta}, a text-to-image synthesis framework that integrates the Latent Consistency Model (LCM) and ControlNet into the advanced PIXART-{\alpha} model. PIXART-{\alpha} is recognized for its ability to generate high-quality images of 1024px resolution through a remarkably efficient training process. The integration of LCM in PIXART-{\delta} significantly accelerates the inference speed, enabling the production of high-quality images in just 2-4 steps. Notably, PIXART-{\delta} achieves a breakthrough 0.5 seconds for generating 1024x1024 pixel images, marking a 7x improvement over the PIXART-{\alpha}. Additionally, PIXART-{\delta} is designed to be efficiently trainable on 32GB V100 GPUs within a single day. With its 8-bit inference capability (von Platen et al., 2023), PIXART-{\delta} can synthesize 1024px images within 8GB GPU memory constraints, greatly enhancing its usability and accessibility. Furthermore, incorporating a ControlNet-like module enables fine-grained control over text-to-image diffusion models. We introduce a novel ControlNet-Transformer architecture, specifically tailored for Transformers, achieving explicit controllability alongside high-quality image generation. As a state-of-the-art, open-source image generation model, PIXART-{\delta} offers a promising alternative to the Stable Diffusion family of models, contributing significantly to text-to-image synthesis.

• 摘要：

  本技术报告介绍了 PIXART-{\delta}，这是一种文本到图像合成框架，它将潜在一致性模型 (LCM) 和 ControlNet 集成到先进的 PIXART-{\alpha} 模型中。 PIXART-{\alpha} 因其通过非常高效的训练过程生成 1024px 分辨率的高质量图像的能力而受到认可。 PIXART-{\delta}中 LCM 的集成显着加快了推理速度，只需 2-4 个步骤即可生成高质量图像。值得注意的是，PIXART-{\delta} 在生成 1024x1024 像素图像方面突破了 0.5 秒，比 PIXART-{\alpha} 提高了 7 倍。此外，PIXART-{\delta} 设计为可在一天内在 32GB V100 GPU 上进行高效训练。凭借其 8 位推理能力（von Platen 等人，2023），PIXART-{\delta} 可以在 8GB GPU 内存限制内合成 1024px 图像，大大增强了其可用性和可访问性。此外，结合类似 ControlNet 的模块可以对文本到图像扩散模型进行细粒度控制。我们引入了一种新颖的 ControlNet-Transformer 架构，专为 Transformer 量身定制，可在生成高质量图像的同时实现明确的可控性。作为最先进的开源图像生成模型，PIXART-{\delta} 为稳定扩散模型系列提供了一种有前途的替代方案，为文本到图像的合成做出了重大贡献。

Title: Score Distillation Sampling with Learned Manifold Corrective. (arXiv:2401.05293v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.05293
• Code URL: null
• Copy Paste: [[2401.05293]] Score Distillation Sampling with Learned Manifold Corrective(http://arxiv.org/abs/2401.05293)
• Summary:

  Score Distillation Sampling (SDS) is a recent but already widely popular method that relies on an image diffusion model to control optimization problems using text prompts. In this paper, we conduct an in-depth analysis of the SDS loss function, identify an inherent problem with its formulation, and propose a surprisingly easy but effective fix. Specifically, we decompose the loss into different factors and isolate the component responsible for noisy gradients. In the original formulation, high text guidance is used to account for the noise, leading to unwanted side effects. Instead, we train a shallow network mimicking the timestep-dependent denoising deficiency of the image diffusion model in order to effectively factor it out. We demonstrate the versatility and the effectiveness of our novel loss formulation through several qualitative and quantitative experiments, including optimization-based image synthesis and editing, zero-shot image translation network training, and text-to-3D synthesis.

• 摘要：

  分数蒸馏采样（SDS）是一种最近但已经广泛流行的方法，它依赖于图像扩散模型来使用文本提示来控制优化问题。在本文中，我们对 SDS 损失函数进行了深入分析 ，确定了其公式的固有问题，并提出了一个非常简单但有效的解决方案。具体来说，我们将损失分解为不同的因素，并隔离导致噪声梯度的成分。在最初的配方中，使用高文本指导 来解决噪音，从而导致不必要的副作用。相反，我们训练一个浅层网络来模仿图像扩散模型的时间步相关的去噪缺陷，以便有效地将其分解出来。我们通过几个定性和定量实验证明 了我们新颖的损失公式的多功能性和有效性，包括基于优化的图像合成和编辑、零样本图像翻译网络训练以及文本到 3D 合成。

noise learning

data-free

transformer

Title: DedustNet: A Frequency-dominated Swin Transformer-based Wavelet Network for Agricultural Dust Removal. (arXiv:2401.04750v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.04750
• Code URL: null
• Copy Paste: [[2401.04750]] DedustNet: A Frequency-dominated Swin Transformer-based Wavelet Network for Agricultural Dust Removal(http://arxiv.org/abs/2401.04750)
• Summary:

  While dust significantly affects the environmental perception of automated agricultural machines, the existing deep learning-based methods for dust removal require further research and improvement in this area to improve the performance and reliability of automated agricultural machines in agriculture. We propose an end-to-end trainable learning network (DedustNet) to solve the real-world agricultural dust removal task. To our knowledge, DedustNet is the first time Swin Transformer-based units have been used in wavelet networks for agricultural image dusting. Specifically, we present the frequency-dominated block (DWTFormer block and IDWTFormer block) by adding a spatial features aggregation scheme (SFAS) to the Swin Transformer and combining it with the wavelet transform, the DWTFormer block and IDWTFormer block, alleviating the limitation of the global receptive field of Swin Transformer when dealing with complex dusty backgrounds. Furthermore, We propose a cross-level information fusion module to fuse different levels of features and effectively capture global and long-range feature relationships. In addition, we present a dilated convolution module to capture contextual information guided by wavelet transform at multiple scales, which combines the advantages of wavelet transform and dilated convolution. Our algorithm leverages deep learning techniques to effectively remove dust from images while preserving the original structural and textural features. Compared to existing state-of-the-art methods, DedustNet achieves superior performance and more reliable results in agricultural image dedusting, providing strong support for the application of agricultural machinery in dusty environments. Additionally, the impressive performance on real-world hazy datasets and application tests highlights DedustNet superior generalization ability and computer vision-related application performance.

• 摘要：

  虽然灰尘显着影响自动化农业机械的环境感知，但现有的基于深度学习的除尘方法需要在该领域进一步研究和改进，以提高农业自动化农业机械的性能和可靠性。我们提出了 一种端到端可训练学习网络（DedustNet）来解决现实世界的农业除尘任务。据我们所知，DedustNet 是基于 Swin Transformer 的单元首次在小波网络中用于农业图像除尘。具体来说，我们通过向 Swin Transformer 添加空间特征聚合方案（SFAS）并将其与小波变换、DWTFormer 块和 IDWTFormer 块相结合，提出了频率主导块（DWTFormer 块和 IDWTFormer 块），减轻了Swin Transformer 在处理复杂的灰尘背景时的全局感受野。此外，我们提出了一个跨级信息融合模块来融合不同级别的特 征并有效捕获全局和远程特征关系。此外，我们提出了一种扩张卷积模块来捕获多尺度小波变换引导的上下文信息，它结合了小波变换和扩张卷积的优点。我们的算法利用深度学习 技术有效去除图像中的灰尘，同时保留原始的结构和纹理特征。与现有最先进的方法相比，DedustNet在农业图像除尘方面取得了更优越的性能和更可靠的结果，为农业机 械在粉尘环境下的应用提供了有力的支持。此外，在现实世界的模糊数据集和应用测试上令人印象深刻的性能凸显了 DedustNet 卓越的泛化能力和计算机视觉相关的应用性能。

Title: CTNeRF: Cross-Time Transformer for Dynamic Neural Radiance Field from Monocular Video. (arXiv:2401.04861v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.04861
• Code URL: null
• Copy Paste: [[2401.04861]] CTNeRF: Cross-Time Transformer for Dynamic Neural Radiance Field from Monocular Video(http://arxiv.org/abs/2401.04861)
• Summary:

  The goal of our work is to generate high-quality novel views from monocular videos of complex and dynamic scenes. Prior methods, such as DynamicNeRF, have shown impressive performance by leveraging time-varying dynamic radiation fields. However, these methods have limitations when it comes to accurately modeling the motion of complex objects, which can lead to inaccurate and blurry renderings of details. To address this limitation, we propose a novel approach that builds upon a recent generalization NeRF, which aggregates nearby views onto new viewpoints. However, such methods are typically only effective for static scenes. To overcome this challenge, we introduce a module that operates in both the time and frequency domains to aggregate the features of object motion. This allows us to learn the relationship between frames and generate higher-quality images. Our experiments demonstrate significant improvements over state-of-the-art methods on dynamic scene datasets. Specifically, our approach outperforms existing methods in terms of both the accuracy and visual quality of the synthesized views.

• 摘要：

  我们工作的目标是从复杂动态场景的单眼视频中生成高质量的新颖视图。先前的方法，例如 DynamicNeRF，通过利用时变动态辐射场显示出令人印象深刻的性能 。然而，这些方法在精确建模复杂物体的运动时存在局限性，这可能导致细节渲染不准确和模糊。为了解决这个限制，我们提出了一种基于最近的泛化 NeRF 的新方法，它将 附近的视图聚合到新的观点上。然而，此类方法通常仅对静态场景有效。为了克服这一挑战，我们引入了一个在时域和频域中运行的模块来聚合对象运动的特征。这使我们能够学习 帧之间的关系并生成更高质量的图像。我们的实验证明了动态场景数据集上最先进方法的显着改进。具体来说，我们的方法在合成视图的准确性和视觉质量方面都优于现有方法。

Title: Knowledge-aware Graph Transformer for Pedestrian Trajectory Prediction. (arXiv:2401.04872v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.04872
• Code URL: null
• Copy Paste: [[2401.04872]] Knowledge-aware Graph Transformer for Pedestrian Trajectory Prediction(http://arxiv.org/abs/2401.04872)
• Summary:

  Predicting pedestrian motion trajectories is crucial for path planning and motion control of autonomous vehicles. Accurately forecasting crowd trajectories is challenging due to the uncertain nature of human motions in different environments. For training, recent deep learning-based prediction approaches mainly utilize information like trajectory history and interactions between pedestrians, among others. This can limit the prediction performance across various scenarios since the discrepancies between training datasets have not been properly incorporated. To overcome this limitation, this paper proposes a graph transformer structure to improve prediction performance, capturing the differences between the various sites and scenarios contained in the datasets. In particular, a self-attention mechanism and a domain adaption module have been designed to improve the generalization ability of the model. Moreover, an additional metric considering cross-dataset sequences is introduced for training and performance evaluation purposes. The proposed framework is validated and compared against existing methods using popular public datasets, i.e., ETH and UCY. Experimental results demonstrate the improved performance of our proposed scheme.

• 摘要：

  预测行人运动轨迹对于自动驾驶车辆的路径规划和运动控制至关重要。由于不同环境中人体运动的不确定性，准确预测人群轨迹具有挑战性。对于训练，最近基于深度学习的 预测方法主要利用轨迹历史和行人之间的交互等信息。由于训练数据集之间的差异尚未正确纳入，这可能会限制各种场景的预测性能。为了克服这一限制，本文提出了一种图转换器 结构来提高预测性能，捕获数据集中包含的各个站点和场景之间的差异。特别是，设计了自注意力机制和领域适应模块来提高模型的泛化能力。此外，出于训练和性能评估目的，引 入了考虑跨数据集序列的附加度量。使用流行的公共数据集（即 ETH 和 UCY）对所提出的框架进行了验证并与现有方法进行了比较。实验结果证明了我们提出的方案的性能得到了提高。

Title: EmMixformer: Mix transformer for eye movement recognition. (arXiv:2401.04956v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.04956
• Code URL: null
• Copy Paste: [[2401.04956]] EmMixformer: Mix transformer for eye movement recognition(http://arxiv.org/abs/2401.04956)
• Summary:

  Eye movement (EM) is a new highly secure biometric behavioral modality that has received increasing attention in recent years. Although deep neural networks, such as convolutional neural network (CNN), have recently achieved promising performance, current solutions fail to capture local and global temporal dependencies within eye movement data. To overcome this problem, we propose in this paper a mixed transformer termed EmMixformer to extract time and frequency domain information for eye movement recognition. To this end, we propose a mixed block consisting of three modules, transformer, attention Long short-term memory (attention LSTM), and Fourier transformer. We are the first to attempt leveraging transformer to learn long temporal dependencies within eye movement. Second, we incorporate the attention mechanism into LSTM to propose attention LSTM with the aim to learn short temporal dependencies. Third, we perform self attention in the frequency domain to learn global features. As the three modules provide complementary feature representations in terms of local and global dependencies, the proposed EmMixformer is capable of improving recognition accuracy. The experimental results on our eye movement dataset and two public eye movement datasets show that the proposed EmMixformer outperforms the state of the art by achieving the lowest verification error.

• 摘要：

  眼动（EM）是一种新型的高度安全的生物识别行为方式，近年来受到越来越多的关注。尽管卷积神经网络 (CNN) 等深度神经网络最近取得了可喜的性能，但当前的解决方案无法捕获眼动数据中的局部和全局时间依赖性。为了克服这个问题，我们在本文中提出了一种称为 EmMixformer 的混合变压器来提取时域和频域信息以进行眼动识别。为此，我们提出了一个由三个模块组成的混合块，变压器、注意力长短期记忆（注意力LSTM） 和傅里叶变压器。我们是第一个尝试利用 Transformer 来学习眼球运动中的长期时间依赖性的人。其次，我们将注意力机制融入到 LSTM 中，提出注意力 LSTM，旨在学习短时间依赖性。第三，我们在频域中进行自注意力以学习全局特征。由于这三个模块在局部和全局依赖性方面提供了互补的特征表示，因此所提出的 EmMixformer 能够提高识别精度。我们的眼动数据集和两个公共眼动数据集的实验结果表明，所提出的 EmMixformer 通过实现最低的验证误差而优于现有技术。

Title: HaltingVT: Adaptive Token Halting Transformer for Efficient Video Recognition. (arXiv:2401.04975v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.04975
• Code URL: https://github.com/dun-research/haltingvt
• Copy Paste: [[2401.04975]] HaltingVT: Adaptive Token Halting Transformer for Efficient Video Recognition(http://arxiv.org/abs/2401.04975)
• Summary:

  Action recognition in videos poses a challenge due to its high computational cost, especially for Joint Space-Time video transformers (Joint VT). Despite their effectiveness, the excessive number of tokens in such architectures significantly limits their efficiency. In this paper, we propose HaltingVT, an efficient video transformer adaptively removing redundant video patch tokens, which is primarily composed of a Joint VT and a Glimpser module. Specifically, HaltingVT applies data-adaptive token reduction at each layer, resulting in a significant reduction in the overall computational cost. Besides, the Glimpser module quickly removes redundant tokens in shallow transformer layers, which may even be misleading for video recognition tasks based on our observations. To further encourage HaltingVT to focus on the key motion-related information in videos, we design an effective Motion Loss during training. HaltingVT acquires video analysis capabilities and token halting compression strategies simultaneously in a unified training process, without requiring additional training procedures or sub-networks. On the Mini-Kinetics dataset, we achieved 75.0% top-1 ACC with 24.2 GFLOPs, as well as 67.2% top-1 ACC with an extremely low 9.9 GFLOPs. The code is available at https://github.com/dun-research/HaltingVT.

• 摘要：

  视频中的动作识别由于其较高的计算成本而提出了挑战，特别是对于联合时空视频转换器（Joint VT）而言。尽管它们很有效，但此类架构中过多的代币极大地限制了它们的效率。在本文中，我们提出了 HaltingVT，一种高效的视频转换器，可自适应删除冗余视频补丁标记，它主要由 Joint VT 和 Glimpser 模块组成。具体来说，HaltingVT 在每一层应用数据自适应令牌减少，从而显着降低总体计算成本。此外，Glimpser 模块可以快速删除浅层转换器层中的冗余标记，根据我们的观察，这甚至可能会误导视频识别任务。为了进一步鼓励 HaltingVT 关注视频中与运动相关的关键信息，我们在训练期间设计了有效的运动损失。 HaltingVT 在统一的训练过程中同时获得视频分析能力和令牌停止压缩策略，无需额外的训练过程或子网络。在 Mini-Kinetics 数据集上，我们以 24.2 GFLOP 实现了 75.0% 的 top-1 ACC，以及以极低的 9.9 GFLOP 实现了 67.2% 的 top-1 ACC。该代码可在 https://github.com/dun-research/HaltingVT 获取。

Title: AdvMT: Adversarial Motion Transformer for Long-term Human Motion Prediction. (arXiv:2401.05018v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.05018
• Code URL: null
• Copy Paste: [[2401.05018]] AdvMT: Adversarial Motion Transformer for Long-term Human Motion Prediction(http://arxiv.org/abs/2401.05018)
• Summary:

  To achieve seamless collaboration between robots and humans in a shared environment, accurately predicting future human movements is essential. Human motion prediction has traditionally been approached as a sequence prediction problem, leveraging historical human motion data to estimate future poses. Beginning with vanilla recurrent networks, the research community has investigated a variety of methods for learning human motion dynamics, encompassing graph-based and generative approaches. Despite these efforts, achieving accurate long-term predictions continues to be a significant challenge. In this regard, we present the Adversarial Motion Transformer (AdvMT), a novel model that integrates a transformer-based motion encoder and a temporal continuity discriminator. This combination effectively captures spatial and temporal dependencies simultaneously within frames. With adversarial training, our method effectively reduces the unwanted artifacts in predictions, thereby ensuring the learning of more realistic and fluid human motions. The evaluation results indicate that AdvMT greatly enhances the accuracy of long-term predictions while also delivering robust short-term predictions

• 摘要：

  为了在共享环境中实现机器人与人类之间的无缝协作，准确预测人类未来的运动至关重要。传统上，人体运动预测被视为序列预测问题，利用历史人体运动数据来估计未来的 姿势。从普通的循环网络开始，研究界研究了各种学习人体运动动力学的方法，包括基于图的方法和生成方法。尽管做出了这些努力，实现准确的长期预测仍然是一项重大挑战。在 这方面，我们提出了对抗运动变换器（AdvMT），这是一种集成了基于变换器的运动编码器和时间连续性鉴别器的新颖模型。这种组合有效地在帧内同时捕获空间和时间依赖性 。通过对抗性训练，我们的方法有效地减少了预测中不需要的伪影，从而确保学习更真实、更流畅的人体动作。评估结果表明，AdvMT 极大地提高了长期预测的准确性，同时也提供了稳健的短期预测

Title: Application of Deep Learning in Blind Motion Deblurring: Current Status and Future Prospects. (arXiv:2401.05055v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.05055
• Code URL: https://github.com/visionverse/blind-motion-deblurring-survey
• Copy Paste: [[2401.05055]] Application of Deep Learning in Blind Motion Deblurring: Current Status and Future Prospects(http://arxiv.org/abs/2401.05055)
• Summary:

  Motion deblurring is one of the fundamental problems of computer vision and has received continuous attention. The variability in blur, both within and across images, imposes limitations on non-blind deblurring techniques that rely on estimating the blur kernel. As a response, blind motion deblurring has emerged, aiming to restore clear and detailed images without prior knowledge of the blur type, fueled by the advancements in deep learning methodologies. Despite strides in this field, a comprehensive synthesis of recent progress in deep learning-based blind motion deblurring is notably absent. This paper fills that gap by providing an exhaustive overview of the role of deep learning in blind motion deblurring, encompassing datasets, evaluation metrics, and methods developed over the last six years. Specifically, we first introduce the types of motion blur and the fundamental principles of deblurring. Next, we outline the shortcomings of traditional non-blind deblurring algorithms, emphasizing the advantages of employing deep learning techniques for deblurring tasks. Following this, we categorize and summarize existing blind motion deblurring methods based on different backbone networks, including convolutional neural networks, generative adversarial networks, recurrent neural networks, and Transformer networks. Subsequently, we elaborate not only on the fundamental principles of these different categories but also provide a comprehensive summary and comparison of their advantages and limitations. Qualitative and quantitative experimental results conducted on four widely used datasets further compare the performance of SOTA methods. Finally, an analysis of present challenges and future pathways. All collected models, benchmark datasets, source code links, and codes for evaluation have been made publicly available at https://github.com/VisionVerse/Blind-Motion-Deblurring-Survey

• 摘要：

  运动去模糊是计算机视觉的基本问题之一，受到持续关注。图像内部和图像之间的模糊变化对依赖于估计模糊内核的非盲去模糊技术施加了限制。作为回应，盲运动去模糊应 运而生，旨在在深度学习方法的进步的推动下，在不事先了解模糊类型的情况下恢复清晰详细的图像。尽管该领域取得了长足的进步，但对基于深度学习的盲运动去模糊的最新进展 的全面综合仍然明显缺乏。本文通过详尽概述深度学习在盲运动去模糊中的作用（包括过去六年开发的数据集、评估指标和方法）来填补这一空白。具体来说，我们首先介绍运动模 糊的类型和去模糊的基本原理。接下来，我们概述了传统非盲去模糊算法的缺点，强调了采用深度学习技术进行去模糊任务的优势。接下来，我们根据不同的骨干网络对现有的盲运 动去模糊方法进行分类和总结，包括卷积神经网络、生成对抗网络、循环神经网络和 Transformer 网络。随后，我们不仅阐述了这些不同类别的基本原理，还对它们的优点和局限性进行了全面的总结和比较。在四个广泛使用的数据集上进行的定性和定量实验结果进一步比较了 SOTA 方法的性能。最后，分析当前的挑战和未来的路径。所有收集的模型、基准数据集、源代码链接和评估代码均已在 https://github.com/VisionVerse/Blind-Motion-Deblurring-Survey 上公开发布

Title: MoSECroT: Model Stitching with Static Word Embeddings for Crosslingual Zero-shot Transfer. (arXiv:2401.04821v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.04821
• Code URL: null
• Copy Paste: [[2401.04821]] MoSECroT: Model Stitching with Static Word Embeddings for Crosslingual Zero-shot Transfer(http://arxiv.org/abs/2401.04821)
• Summary:

  Transformer-based pre-trained language models (PLMs) have achieved remarkable performance in various natural language processing (NLP) tasks. However, pre-training such models can take considerable resources that are almost only available to high-resource languages. On the contrary, static word embeddings are easier to train in terms of computing resources and the amount of data required. In this paper, we introduce MoSECroT Model Stitching with Static Word Embeddings for Crosslingual Zero-shot Transfer), a novel and challenging task that is especially relevant to low-resource languages for which static word embeddings are available. To tackle the task, we present the first framework that leverages relative representations to construct a common space for the embeddings of a source language PLM and the static word embeddings of a target language. In this way, we can train the PLM on source-language training data and perform zero-shot transfer to the target language by simply swapping the embedding layer. However, through extensive experiments on two classification datasets, we show that although our proposed framework is competitive with weak baselines when addressing MoSECroT, it fails to achieve competitive results compared with some strong baselines. In this paper, we attempt to explain this negative result and provide several thoughts on possible improvement.

• 摘要：

  基于 Transformer 的预训练语言模型 (PLM) 在各种自然语言处理 (NLP) 任务中取得了显着的性能。然而，预训练此类模型可能会占用大量资 源，而这些资源几乎仅适用于高资源语言。相反，静态词嵌入在计算资源和所需数据量方面更容易训练。在本文中，我们介绍了用于跨语言零样本迁移的静态词嵌入的 MoSECroT 模型缝合，这是一项新颖且具有挑战性的任务，尤其与可使用静态词嵌入的低资源语言相关。为了解决这一任务，我们提出了第一个框架，该框架利用相对表示来为源语言 PLM 的嵌入和目标语言的静态词嵌入构建公共空间。这样，我们可以在源语言训练数据上训练 PLM，并通过简单地交换嵌入层来执行到目标语言的零样本迁移。然而，通过对两个分类数据集的大量实验，我们表明，尽管我们提出的框架在解决 MoSECroT 问题时与弱基线具有竞争力，但与一些强基线相比，它无法实现有竞争力的结果。在本文中，我们试图解释这一负面结果，并就可能的改进提供一些想法。

Title: Real-time and Continuous Turn-taking Prediction Using Voice Activity Projection. (arXiv:2401.04868v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.04868
• Code URL: null
• Copy Paste: [[2401.04868]] Real-time and Continuous Turn-taking Prediction Using Voice Activity Projection(http://arxiv.org/abs/2401.04868)
• Summary:

  A demonstration of a real-time and continuous turn-taking prediction system is presented. The system is based on a voice activity projection (VAP) model, which directly maps dialogue stereo audio to future voice activities. The VAP model includes contrastive predictive coding (CPC) and self-attention transformers, followed by a cross-attention transformer. We examine the effect of the input context audio length and demonstrate that the proposed system can operate in real-time with CPU settings, with minimal performance degradation.

• 摘要：

  展示了实时连续轮流预测系统。该系统基于语音活动投影（VAP）模型，可直接将对话立体声音频映射到未来的语音活动。 VAP 模型包括对比预测编码（CPC）和自注意力变压器，然后是交叉注意力变压器。我们检查了输入上下文音频长度的影响，并证明所提出的系统可以通过 CPU 设置实时运行，并且性能下降最小。

Title: Attendre: Wait To Attend By Retrieval With Evicted Queries in Memory-Based Transformers for Long Context Processing. (arXiv:2401.04881v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.04881
• Code URL: null
• Copy Paste: [[2401.04881]] Attendre: Wait To Attend By Retrieval With Evicted Queries in Memory-Based Transformers for Long Context Processing(http://arxiv.org/abs/2401.04881)
• Summary:

  As LLMs have become capable of processing more complex types of inputs, researchers have recently studied how to efficiently and affordably process possibly arbitrarily long sequences. One effective approach is to use a FIFO memory to store keys and values of an attention sublayer from past chunks to allow subsequent queries to attend. However, this approach requires a large memory and/or takes into the consideration the specific LM architecture. Moreover, due to the causal nature between the key-values in prior context and the queries at present, this approach cannot be extended to bidirectional attention such as in an encoder-decoder or PrefixLM decoder-only architecture. In this paper, we propose to use eviction policies, such as LRA and LFA, to reduce the memory size and adapt to various architectures, and we also propose the Attendre layer, a wait-to-attend mechanism by retrieving the key-value memory (K/V memory) with evicted queries in the query memory (Q memory). As a first step, we evaluate this method in the context length extension setup using the TriviaQA reading comprehension task, and show the effectiveness of the approach.

• 摘要：随着法学硕士已经能够处理更复杂类型的输入，研究人员最近研究了如何高效且经济地处理可能任意长的序列。一种有效的方法是使用 FIFO 内存来存储过去块中关注子层的键和值，以允许后续查询参与。然而，这种方法需要大内存和/或考虑特定的 LM 架构。此外，由于先前上下文中的键值与当前查询之间的因果关系，这种方法无法扩展到双向注意力，例如在编码器-解码器或仅 PrefixLM 解码器架构中。在本文中， 我们提出使用驱逐策略（例如LRA和LFA）来减少内存大小并适应各种架构，并且我们还提出了Attendre层，这是一种通过检索键值内存来等待出席的机制（K/V 内存），并在查询内存（Q 内存）中驱逐查询。第一步，我们使用 TriviaQA 阅读理解任务在上下文长度扩展设置中评估该方法，并展示该方法的有效性。

Title: LogFormer: A Pre-train and Tuning Pipeline for Log Anomaly Detection. (arXiv:2401.04749v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.04749
• Code URL: null
• Copy Paste: [[2401.04749]] LogFormer: A Pre-train and Tuning Pipeline for Log Anomaly Detection(http://arxiv.org/abs/2401.04749)
• Summary:

  Log anomaly detection is a key component in the field of artificial intelligence for IT operations (AIOps). Considering log data of variant domains, retraining the whole network for unknown domains is inefficient in real industrial scenarios. However, previous deep models merely focused on extracting the semantics of log sequences in the same domain, leading to poor generalization on multi-domain logs. To alleviate this issue, we propose a unified Transformer-based framework for Log anomaly detection (LogFormer) to improve the generalization ability across different domains, where we establish a two-stage process including the pre-training and adapter-based tuning stage. Specifically, our model is first pre-trained on the source domain to obtain shared semantic knowledge of log data. Then, we transfer such knowledge to the target domain via shared parameters. Besides, the Log-Attention module is proposed to supplement the information ignored by the log-paring. The proposed method is evaluated on three public and one real-world datasets. Experimental results on multiple benchmarks demonstrate the effectiveness of our LogFormer with fewer trainable parameters and lower training costs.

• 摘要：

  日志异常检测是 IT 运营人工智能 (AIOps) 领域的关键组成部分。考虑到不同域的日志数据，在实际工业场景中，针对未知域重新训练整个网络效率很低。然 而，之前的深度模型仅仅关注于提取同一域内日志序列的语义，导致对多域日志的泛化能力较差。为了缓解这个问题，我们提出了一个基于 Transformer 的统一日志 异常检测框架（LogFormer），以提高跨不同领域的泛化能力，其中我们建立了一个两阶段过程，包括预训练和基于适配器的调整阶段。具体来说，我们的模型首先在源域 上进行预训练，以获得日志数据的共享语义知识。然后，我们通过共享参数将这些知识转移到目标域。此外，还提出了Log-Attention模块来补充log- paring忽略的信息。所提出的方法在三个公共数据集和一个真实世界数据集上进行评估。多个基准的实验结果证明了我们的 LogFormer 的有效性，可训练参数更少，训练成本更低。

Title: T-PRIME: Transformer-based Protocol Identification for Machine-learning at the Edge. (arXiv:2401.04837v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.04837
• Code URL: https://github.com/genesys-neu/t-prime
• Copy Paste: [[2401.04837]] T-PRIME: Transformer-based Protocol Identification for Machine-learning at the Edge(http://arxiv.org/abs/2401.04837)
• Summary:

  Spectrum sharing allows different protocols of the same standard (e.g., 802.11 family) or different standards (e.g., LTE and DVB) to coexist in overlapping frequency bands. As this paradigm continues to spread, wireless systems must also evolve to identify active transmitters and unauthorized waveforms in real time under intentional distortion of preambles, extremely low signal-to-noise ratios and challenging channel conditions. We overcome limitations of correlation-based preamble matching methods in such conditions through the design of T-PRIME: a Transformer-based machine learning approach. T-PRIME learns the structural design of transmitted frames through its attention mechanism, looking at sequence patterns that go beyond the preamble alone. The paper makes three contributions: First, it compares Transformer models and demonstrates their superiority over traditional methods and state-of-the-art neural networks. Second, it rigorously analyzes T-PRIME's real-time feasibility on DeepWave's AIR-T platform. Third, it utilizes an extensive 66 GB dataset of over-the-air (OTA) WiFi transmissions for training, which is released along with the code for community use. Results reveal nearly perfect (i.e. $>98\%$) classification accuracy under simulated scenarios, showing $100\%$ detection improvement over legacy methods in low SNR ranges, $97\%$ classification accuracy for OTA single-protocol transmissions and up to $75\%$ double-protocol classification accuracy in interference scenarios.

• 摘要：

  频谱共享允许同一标准（例如 802.11 系列）或不同标准（例如 LTE 和 DVB）的不同协议在重叠频段中共存。随着这种范例的不断传播，无线系统还必须 不断发展，以在故意扭曲前导码、极低的信噪比和具有挑战性的信道条件下实时识别活动发射机和未经授权的波形。我们通过设计 T-PRIME：一种基于 Transformer 的机器学习方法，克服了在这种情况下基于相关性的前导码匹配方法的局限性。 T-PRIME 通过其注意力机制学习传输帧的结构设计，查看超出前导码范围的序列模式。这篇论文做出了三个贡献：首先，它比较了 Transformer 模型，并证明了它们相对于传统方法和最先进的神经网络的优越性。其次，严格分析了T-PRIME在DeepWave的AIR-T平台上的实时可行性。第三，它利用广泛的 66 GB 无线 (OTA) WiFi 传输数据集进行训练，该数据集与代码一起发布供社区使用。结果显示，在模拟场景下，分类精度近乎完美（即 $>98\%$），在低 SNR 范围内，检测结果比传统方法提高了 $100\%$，OTA 单协议传输的分类精度为 $97\%$，高达干扰场景下双协议分类精度为 75\%$。

generative

Title: Content-Conditioned Generation of Stylized Free hand Sketches. (arXiv:2401.04739v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.04739
• Code URL: null
• Copy Paste: [[2401.04739]] Content-Conditioned Generation of Stylized Free hand Sketches(http://arxiv.org/abs/2401.04739)
• Summary:

  In recent years, the recognition of free-hand sketches has remained a popular task. However, in some special fields such as the military field, free-hand sketches are difficult to sample on a large scale. Common data augmentation and image generation techniques are difficult to produce images with various free-hand sketching styles. Therefore, the recognition and segmentation tasks in related fields are limited. In this paper, we propose a novel adversarial generative network that can accurately generate realistic free-hand sketches with various styles. We explore the performance of the model, including using styles randomly sampled from a prior normal distribution to generate images with various free-hand sketching styles, disentangling the painters' styles from known free-hand sketches to generate images with specific styles, and generating images of unknown classes that are not in the training set. We further demonstrate with qualitative and quantitative evaluations our advantages in visual quality, content accuracy, and style imitation on SketchIME.

• 摘要：

  近年来，手绘草图的识别仍然是一项热门任务。但在军事领域等一些特殊领域，徒手草图很难大规模采样。常见的数据增强和图像生成技术很难生成具有各种手绘草图风格的 图像。因此，相关领域的识别和分割任务受到限制。在本文中，我们提出了一种新颖的对抗性生成网络，可以准确生成各种风格的逼真手绘草图。我们探索了模型的性能，包括使用 从先验正态分布中随机采样的样式来生成具有各种手绘草图风格的图像，将画家的风格与已知的手绘草图分离以生成具有特定风格的图像，以及生成图像不在训练集中的未知类。我 们通过定性和定量评估进一步证明了我们在 SketchIME 上的视觉质量、内容准确性和风格模仿方面的优势。

Title: MISS: A Generative Pretraining and Finetuning Approach for Med-VQA. (arXiv:2401.05163v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.05163
• Code URL: null
• Copy Paste: [[2401.05163]] MISS: A Generative Pretraining and Finetuning Approach for Med-VQA(http://arxiv.org/abs/2401.05163)
• Summary:

  Medical visual question answering (VQA) is a challenging multimodal task, where Vision-Language Pre-training (VLP) models can effectively improve the generalization performance. However, most methods in the medical field treat VQA as an answer classification task which is difficult to transfer to practical application scenarios. Additionally, due to the privacy of medical images and the expensive annotation process, large-scale medical image-text pairs datasets for pretraining are severely lacking. In this paper, we propose a large-scale MultI-task Self-Supervised learning based framework (MISS) for medical VQA tasks. Unlike existing methods, we treat medical VQA as a generative task. We unify the text encoder and multimodal encoder and align image-text features through multi-task learning. Furthermore, we propose a Transfer-and-Caption method that extends the feature space of single-modal image datasets using large language models (LLMs), enabling those traditional medical vision field task data to be applied to VLP. Experiments show that our method achieves excellent results with fewer multimodal datasets and demonstrates the advantages of generative VQA models. The code and model weights will be released upon the paper's acceptance.

• 摘要：

  医学视觉问答（VQA）是一项具有挑战性的多模态任务，其中视觉语言预训练（VLP）模型可以有效提高泛化性能。然而，医学领域的大多数方法将VQA视为答案分类 任务，很难迁移到实际应用场景。此外，由于医学图像的隐私性和昂贵的注释过程，严重缺乏用于预训练的大规模医学图像-文本对数据集。在本文中，我们提出了一种用于医疗 VQA 任务的基于大规模多任务自监督学习的框架（MISS）。与现有方法不同，我们将医学 VQA 视为一项生成任务。我们统一文本编码器和多模态编码器，并通过多任 务学习对齐图像文本特征。此外，我们提出了一种传输和标题方法，该方法使用大语言模型（LLM）扩展单模态图像数据集的特征空间，使这些传统的医学视觉领域任务数据能够 应用于 VLP。实验表明，我们的方法用更少的多模态数据集取得了优异的结果，并展示了生成式 VQA 模型的优势。代码和模型权重将在论文被接受后发布。

Title: InseRF: Text-Driven Generative Object Insertion in Neural 3D Scenes. (arXiv:2401.05335v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.05335
• Code URL: null
• Copy Paste: [[2401.05335]] InseRF: Text-Driven Generative Object Insertion in Neural 3D Scenes(http://arxiv.org/abs/2401.05335)
• Summary:

  We introduce InseRF, a novel method for generative object insertion in the NeRF reconstructions of 3D scenes. Based on a user-provided textual description and a 2D bounding box in a reference viewpoint, InseRF generates new objects in 3D scenes. Recently, methods for 3D scene editing have been profoundly transformed, owing to the use of strong priors of text-to-image diffusion models in 3D generative modeling. Existing methods are mostly effective in editing 3D scenes via style and appearance changes or removing existing objects. Generating new objects, however, remains a challenge for such methods, which we address in this study. Specifically, we propose grounding the 3D object insertion to a 2D object insertion in a reference view of the scene. The 2D edit is then lifted to 3D using a single-view object reconstruction method. The reconstructed object is then inserted into the scene, guided by the priors of monocular depth estimation methods. We evaluate our method on various 3D scenes and provide an in-depth analysis of the proposed components. Our experiments with generative insertion of objects in several 3D scenes indicate the effectiveness of our method compared to the existing methods. InseRF is capable of controllable and 3D-consistent object insertion without requiring explicit 3D information as input. Please visit our project page at https://mohamad-shahbazi.github.io/inserf.

• 摘要：

  我们介绍 InseRF，这是一种在 3D 场景的 NeRF 重建中生成对象插入的新方法。基于用户提供的文本描述和参考视点中的 2D 边界框，InseRF 在 3D 场景中生成新对象。最近，由于在 3D 生成建模中使用了文本到图像扩散模型的强先验，3D 场景编辑方法已经发生了深刻的转变。现有方法在通过样式和外观更改或删除现有对象来编辑 3D 场景时最有效。然而，生成新对象仍然是此类方法的一个挑战，我们在本研究中解决了这个问题。具体来说，我们建议将 3D 对象插入基础为场景参考视图中的 2D 对象插入。然后使用单视图对象重建方法将 2D 编辑提升为 3D。然后，在单目深度估计方法的先验指导下，将重建的对象插入场景中。我们在各种 3D 场景上评估我们的方法，并对所提出的组件进行深入分析。我们在多个 3D 场景中生成对象插入的实验表明，与现有方法相比，我们的方法是有效的。 InseRF 能够进行可控且 3D 一致的对象插入，而不需要明确的 3D 信息作为输入。请访问我们的项目页面：https://mohamad- shahbazi.github.io/inserf。

Title: BELHD: Improving Biomedical Entity Linking with Homonoym Disambiguation. (arXiv:2401.05125v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.05125
• Code URL: null
• Copy Paste: [[2401.05125]] BELHD: Improving Biomedical Entity Linking with Homonoym Disambiguation(http://arxiv.org/abs/2401.05125)
• Summary:

  Biomedical entity linking (BEL) is the task of grounding entity mentions to a knowledge base (KB). A popular approach to the task are name-based methods, i.e. those identifying the most appropriate name in the KB for a given mention, either via dense retrieval or autoregressive modeling. However, as these methods directly return KB names, they cannot cope with homonyms, i.e. different KB entities sharing the exact same name. This significantly affects their performance, especially for KBs where homonyms account for a large amount of entity mentions (e.g. UMLS and NCBI Gene). We therefore present BELHD (Biomedical Entity Linking with Homonym Disambiguation), a new name-based method that copes with this challenge. Specifically, BELHD builds upon the BioSyn (Sung et al.,2020) model introducing two crucial extensions. First, it performs a preprocessing of the KB in which it expands homonyms with an automatically chosen disambiguating string, thus enforcing unique linking decisions. Second, we introduce candidate sharing, a novel strategy to select candidates for contrastive learning that enhances the overall training signal. Experiments with 10 corpora and five entity types show that BELHD improves upon state-of-the-art approaches, achieving the best results in 6 out 10 corpora with an average improvement of 4.55pp recall@1. Furthermore, the KB preprocessing is orthogonal to the core prediction model and thus can also improve other methods, which we exemplify for GenBioEL (Yuan et al, 2022), a generative name-based BEL approach. Code is available at: link added upon publication.

• 摘要：

  生物医学实体链接（BEL）是将实体提及基础到知识库（KB）的任务。完成该任务的一种流行方法是基于名称的方法，即通过密集检索或自回归建模来识别知识库中给定 提及的最合适名称的方法。然而，由于这些方法直接返回知识库名称，因此它们无法处理同音异义词，即不同的知识库实体共享完全相同的名称。这会显着影响它们的性能，特别是 对于同音异义词占大量实体提及的知识库（例如 UMLS 和 NCBI Gene）。因此，我们提出了 BELHD（具有同音词消歧功能的生物医学实体链接），这是一种应对这一挑战的新的基于名称的方法。具体来说，BELHD 建立在 BioSyn（Sung 等人，20 20）模型的基础上，引入了两个关键的扩展。首先，它对知识库进行预处理，其中使用自动选择的消歧字符串扩展同音异义词，从而强制执行唯一的链接决策。其次，我们引入候 选共享，这是一种选择候选进行对比学习的新颖策略，可以增强整体训练信号。对 10 个语料库和 5 种实​​体类型的实验表明，BELHD 在最先进的方法的基础上进行了改进，在 10 个语料库中的 6 个中取得了最佳结果，平均提高了 4.55pp 召回率@1。此外，KB 预处理与核心预测模型正交，因此也可以改进其他方法，我们以 GenBioEL（Yuan 等人，2022）为例，这是一种基于名称的生成 BEL 方法。代码可在以下位置获得：发布时添加的链接。

Title: A Good Score Does not Lead to A Good Generative Model. (arXiv:2401.04856v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.04856
• Code URL: null
• Copy Paste: [[2401.04856]] A Good Score Does not Lead to A Good Generative Model(http://arxiv.org/abs/2401.04856)
• Summary:

  Score-based Generative Models (SGMs) is one leading method in generative modeling, renowned for their ability to generate high-quality samples from complex, high-dimensional data distributions. The method enjoys empirical success and is supported by rigorous theoretical convergence properties. In particular, it has been shown that SGMs can generate samples from a distribution that is close to the ground-truth if the underlying score function is learned well, suggesting the success of SGM as a generative model. We provide a counter-example in this paper. Through the sample complexity argument, we provide one specific setting where the score function is learned well. Yet, SGMs in this setting can only output samples that are Gaussian blurring of training data points, mimicking the effects of kernel density estimation. The finding resonates a series of recent finding that reveal that SGMs can demonstrate strong memorization effect and fail to generate.

• 摘要：

  基于分数的生成模型 (SGM) 是生成建模中的一种领先方法，以其从复杂的高维数据分布生成高质量样本的能力而闻名。该方法在实证上取得了成功，并得到了严格的 理论收敛特性的支持。特别是，如果底层得分函数学得好，SGM 可以从接近真实值的分布中生成样本，这表明 SGM 作为生成模型是成功的。我们在本文中提供了一个反例。通过样本复杂性参数，我们提供了一种可以很好地学习得分函数的特定设置。然而，在这种设置下，SGM 只能输出训练数据点高斯模糊的样本，模仿核密度估计的效果。这一发现与最近的一系列发现相呼应，这些发现表明 SGM 可以表现出很强的记忆效果，但无法生成。

Title: Rethinking Test-time Likelihood: The Likelihood Path Principle and Its Application to OOD Detection. (arXiv:2401.04933v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.04933
• Code URL: https://github.com/XavierXiao/Likelihood-Regret
• Copy Paste: [[2401.04933]] Rethinking Test-time Likelihood: The Likelihood Path Principle and Its Application to OOD Detection(http://arxiv.org/abs/2401.04933)
• Summary:

  While likelihood is attractive in theory, its estimates by deep generative models (DGMs) are often broken in practice, and perform poorly for out of distribution (OOD) Detection. Various recent works started to consider alternative scores and achieved better performances. However, such recipes do not come with provable guarantees, nor is it clear that their choices extract sufficient information.

  We attempt to change this by conducting a case study on variational autoencoders (VAEs). First, we introduce the likelihood path (LPath) principle, generalizing the likelihood principle. This narrows the search for informative summary statistics down to the minimal sufficient statistics of VAEs' conditional likelihoods. Second, introducing new theoretic tools such as nearly essential support, essential distance and co-Lipschitzness, we obtain non-asymptotic provable OOD detection guarantees for certain distillation of the minimal sufficient statistics. The corresponding LPath algorithm demonstrates SOTA performances, even using simple and small VAEs with poor likelihood estimates. To our best knowledge, this is the first provable unsupervised OOD method that delivers excellent empirical results, better than any other VAEs based techniques. We use the same model as \cite{xiao2020likelihood}, open sourced from: https://github.com/XavierXiao/Likelihood-Regret

• 摘要：

  虽然可能性在理论上很有吸引力，但深度生成模型 (DGM) 的估计在实践中经常被破坏，并且在分布外 (OOD) 检测中表现不佳。最近的各种作品开始考虑替代配乐并取得了更好的表现。然而，这些食谱并没有提供可证明的保证，也不清楚他们的选择是否提取了足够的信息。

  < p>我们试图通过对变分自动编码器（VAE）进行案例研究来改变这一点。首先，我们引入似然路径（LPath）原理，推广似然原理。这将信息性汇总统计的搜索范围缩小到 VAE 条件可能性的最小充分统计。其次，引入新的理论工具，例如近本质支持、本质距离和共同Lipschitzness，我们获得了对最小充分统计量的某些精炼的非渐 进可证明的OOD检测保证。相应的 LPath 算法展示了 SOTA 性能，即使使用似然估计较差的简单且小型 VAE。据我们所知，这是第一个可证明的无监督 OOD 方法，它提供了出色的实证结果，优于任何其他基于 VAE 的技术。我们使用与 \cite{xiao2020likelihood} 相同的模型，开源自：https://github.com/XavierXiao/Likelihood-Regret

large language model

Title: Multi-User Chat Assistant (MUCA): a Framework Using LLMs to Facilitate Group Conversations. (arXiv:2401.04883v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.04883
• Code URL: null
• Copy Paste: [[2401.04883]] Multi-User Chat Assistant (MUCA): a Framework Using LLMs to Facilitate Group Conversations(http://arxiv.org/abs/2401.04883)
• Summary:

  Recent advancements in large language models (LLMs) have provided a new avenue for chatbot development, while most existing research has primarily centered on single-user chatbots that focus on deciding "What" to answer after user inputs. In this paper, we identified that multi-user chatbots have more complex 3W design dimensions -- "What" to say, "When" to respond, and "Who" to answer. Additionally, we proposed Multi-User Chat Assistant (MUCA), which is an LLM-based framework for chatbots specifically designed for group discussions. MUCA consists of three main modules: Sub-topic Generator, Dialog Analyzer, and Utterance Strategies Arbitrator. These modules jointly determine suitable response contents, timings, and the appropriate recipients. To make the optimizing process for MUCA easier, we further propose an LLM-based Multi-User Simulator (MUS) that can mimic real user behavior. This enables faster simulation of a conversation between the chatbot and simulated users, making the early development of the chatbot framework much more efficient. MUCA demonstrates effectiveness, including appropriate chime-in timing, relevant content, and positive user engagement, in goal-oriented conversations with a small to medium number of participants, as evidenced by case studies and experimental results from user studies.

• 摘要：

  大型语言模型 (LLM) 的最新进展为聊天机器人的开发提供了新的途径，而大多数现有研究主要集中在单用户聊天机器人上，这些机器人专注于在用户输入后决定回答 “什么”。在本文中，我们发现多用户聊天机器人具有更复杂的 3W 设计维度——“说什么”、“何时”响应以及“谁”回答。此外，我们还提出了多用户聊天助手（MUCA），这是一个基于法学硕士的聊天机器人框架，专门为小组讨论而设计。 MUCA由三个主要模块组成：子主题生成器、对话分析器和话语策略仲裁器。这些模块共同确定合适的响应内容、时间和合适的接收者。为了使 MUCA 的优化过程更容易，我们进一步提出了一种基于 LLM 的多用户模拟器（MUS），可以模仿真实的用户行为。这使得能够更快地模拟聊天机器人和模拟用户之间的对话，从而 使聊天机器人框架的早期开发更加高效。案例研究和用户研究的实验结果证明，MUCA 在与中小数量参与者进行目标导向的对话中展示了有效性，包括适当的插话时机、相关内容和积极的用户参与。

Title: The Impact of Reasoning Step Length on Large Language Models. (arXiv:2401.04925v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.04925
• Code URL: null
• Copy Paste: [[2401.04925]] The Impact of Reasoning Step Length on Large Language Models(http://arxiv.org/abs/2401.04925)
• Summary:

  Chain of Thought (CoT) is significant in improving the reasoning abilities of large language models (LLMs). However, the correlation between the effectiveness of CoT and the length of reasoning steps in prompts remains largely unknown. To shed light on this, we have conducted several empirical experiments to explore the relations. Specifically, we design experiments that expand and compress the rationale reasoning steps within CoT demonstrations, while keeping all other factors constant. We have the following key findings. First, the results indicate that lengthening the reasoning steps in prompts, even without adding new information into the prompt, considerably enhances LLMs' reasoning abilities across multiple datasets. Alternatively, shortening the reasoning steps, even while preserving the key information, significantly diminishes the reasoning abilities of models. This finding highlights the importance of the number of steps in CoT prompts and provides practical guidance to make better use of LLMs' potential in complex problem-solving scenarios. Second, we also investigated the relationship between the performance of CoT and the rationales used in demonstrations. Surprisingly, the result shows that even incorrect rationales can yield favorable outcomes if they maintain the requisite length of inference. Third, we observed that the advantages of increasing reasoning steps are task-dependent: simpler tasks require fewer steps, whereas complex tasks gain significantly from longer inference sequences.

• 摘要：

  思想链（CoT）对于提高大型语言模型（LLM）的推理能力具有重要意义。然而，CoT 的有效性与提示中推理步骤的长度之间的相关性仍然很大程度上未知。为了阐明这一点，我们进行了几次实证实​​验来探索其中的关系。具体来说，我们设计了扩展和压缩 CoT 演示中的基本原理推理步骤的实验，同时保持所有其他因素不变。我们有以下主要发现。首先，结果表明，即使没有在提示中添加新信息，延长提示中的推理步骤也可以显 着增强法学硕士跨多个数据集的推理能力。或者，即使在保留关键信息的情况下缩短推理步骤，也会显着降低模型的推理能力。这一发现强调了 CoT 提示中步骤数量的重要性，并为在复杂的问题解决场景中更好地发挥法学硕士的潜力提供了实用指导。其次，我们还研究了 CoT 的性能与演示中使用的基本原理之间的关系。 令人惊讶的是，结果表明，即使是不正确的理由，如果保持必要的推理长度，也能产生有利的结果。第三，我们观察到增加推理步骤的优势与任务相关：更简单的任务需要更少的步 骤，而复杂的任务可以从更长的推理序列中获得显着的收益。

Title: Can AI Write Classical Chinese Poetry like Humans? An Empirical Study Inspired by Turing Test. (arXiv:2401.04952v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.04952
• Code URL: null
• Copy Paste: [[2401.04952]] Can AI Write Classical Chinese Poetry like Humans? An Empirical Study Inspired by Turing Test(http://arxiv.org/abs/2401.04952)
• Summary:

  Some argue that the essence of humanity, such as creativity and sentiment, can never be mimicked by machines. This paper casts doubt on this belief by studying a vital question: Can AI compose poetry as well as humans? To answer the question, we propose ProFTAP, a novel evaluation framework inspired by Turing test to assess AI's poetry writing capability. We apply it on current large language models (LLMs) and find that recent LLMs do indeed possess the ability to write classical Chinese poems nearly indistinguishable from those of humans. We also reveal that various open-source LLMs can outperform GPT-4 on this task.

• 摘要：

  有人认为，人类的本质，例如创造力和情感，永远无法被机器模仿。本文通过研究一个重要问题对这一信念提出了质疑：人工智能能否像人类一样创作诗歌？为了回答这个问 题，我们提出了ProFTAP，这是一种受图灵测试启发的新颖评估框架，用于评估人工智能的诗歌写作能力。我们将其应用到当前的大型语言模型（LLM）上，发现最近的L LM确实具有写出与人类几乎没有区别的中国古典诗歌的能力。我们还表明，各种开源 LLM 在此任务上的表现都优于 GPT-4。

Title: Bootstrapping LLM-based Task-Oriented Dialogue Agents via Self-Talk. (arXiv:2401.05033v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.05033
• Code URL: null
• Copy Paste: [[2401.05033]] Bootstrapping LLM-based Task-Oriented Dialogue Agents via Self-Talk(http://arxiv.org/abs/2401.05033)
• Summary:

  Large language models (LLMs) are powerful dialogue agents, but specializing them towards fulfilling a specific function can be challenging. Instructing tuning, i.e. tuning models on instruction and sample responses generated by humans (Ouyang et al., 2022), has proven as an effective method to do so, yet requires a number of data samples that a) might not be available or b) costly to generate. Furthermore, this cost increases when the goal is to make the LLM follow a specific workflow within a dialogue instead of single instructions. Inspired by the self-play technique in reinforcement learning and the use of LLMs to simulate human agents, we propose a more effective method for data collection through LLMs engaging in a conversation in various roles. This approach generates a training data via "self-talk" of LLMs that can be refined and utilized for supervised fine-tuning. We introduce an automated way to measure the (partial) success of a dialogue. This metric is used to filter the generated conversational data that is fed back in LLM for training. Based on our automated and human evaluations of conversation quality, we demonstrate that such self-talk data improves results. In addition, we examine the various characteristics that showcase the quality of generated dialogues and how they can be connected to their potential utility as training data.

• 摘要：

  大型语言模型 (LLM) 是强大的对话代理，但将其专门用于实现特定功能可能具有挑战性。指导调优，即根据人类生成的指令和样本响应调整模型（Ouyang et al., 2022），已被证明是一种有效的方法，但需要大量数据样本，而这些数据样本 a) 可能不可用或 b)生成成本高昂。此外，当目标是让法学硕士遵循对 话中的特定工作流程而不是单一指令时，这种成本就会增加。受到强化学习中的自我对战技术和使用法学硕士来模拟人类代理的启发，我们提出了一种通过法学硕士参与各种角色对 话来收集数据的更有效方法。这种方法通过法学硕士的“自言自语”生成训练数据，可以对其进行细化并用于监督微调。我们引入了一种自动化方法来衡量对话的（部分）成功。该 指标用于过滤生成的对话数据，这些数据在 LLM 中反馈用于训练。根据我们对对话质量的自动和人工评估，我们证明此类自言自语数据可以改善结果。此外，我们还研究了展 示生成对话质量的各种特征，以及如何将它们与作为训练数据的潜在效用联系起来。

Title: Generating Diverse and High-Quality Texts by Minimum Bayes Risk Decoding. (arXiv:2401.05054v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.05054
• Code URL: null
• Copy Paste: [[2401.05054]] Generating Diverse and High-Quality Texts by Minimum Bayes Risk Decoding(http://arxiv.org/abs/2401.05054)
• Summary:

  One of the most important challenges in text generation systems is to produce outputs that are not only correct but also diverse. Recently, Minimum Bayes-Risk (MBR) decoding has gained prominence for generating sentences of the highest quality among the decoding algorithms. However, existing algorithms proposed for generating diverse outputs are predominantly based on beam search or random sampling, thus their output quality is capped by these underlying methods. In this paper, we investigate an alternative approach -- we develop diversity-promoting decoding algorithms by enforcing diversity objectives to MBR decoding. We propose two variants of MBR, Diverse MBR (DMBR) and $k$-medoids MBR (KMBR), methods to generate a set of sentences with high quality and diversity. We evaluate DMBR and KMBR on a variety of directed text generation tasks using encoder-decoder models and a large language model with prompting. The experimental results show that the proposed method achieves a better trade-off than the diverse beam search and sampling algorithms.

• 摘要：

  文本生成系统中最重要的挑战之一是产生不仅正确而且多样化的输出。最近，最小贝叶斯风险（MBR）解码因在解码算法中生成最高质量的句子而受到重视。然而，现有的 用于生成不同输出的算法主要基于波束搜索或随机采样，因此它们的输出质量受到这些底层方法的限制。在本文中，我们研究了一种替代方法——通过对 MBR 解码强制执行多样性目标来开发多样性促进解码算法。我们提出了 MBR 的两种变体，即多样化 MBR (DMBR) 和 $k$-medoids MBR (KMBR)，这是生成一组高质量和多样性句子的方法。我们使用编码器-解码器模型和带提示的大型语言模型在各种定向文本生成任务上评估 DMBR 和 KMBR。实验结果表明，所提出的方法比多样化的波束搜索和采样算法实现了更好的权衡。

Title: Aligning Translation-Specific Understanding to General Understanding in Large Language Models. (arXiv:2401.05072v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.05072
• Code URL: null
• Copy Paste: [[2401.05072]] Aligning Translation-Specific Understanding to General Understanding in Large Language Models(http://arxiv.org/abs/2401.05072)
• Summary:

  Although large language models (LLMs) have shown surprising language understanding and generation capabilities, they have yet to gain a revolutionary advancement in the field of machine translation. One potential cause of the limited performance is the misalignment between the translation-specific understanding and general understanding inside LLMs. To align the translation-specific understanding to the general one, we propose a novel translation process xIoD (Cross-Lingual Interpretation of Difficult words), explicitly incorporating the general understanding on the content incurring inconsistent understanding to guide the translation. Specifically, xIoD performs the cross-lingual interpretation for the difficult-to-translate words and enhances the translation with the generated interpretations. Furthermore, we reframe the external tools of QE to tackle the challenges of xIoD in the detection of difficult words and the generation of helpful interpretations. We conduct experiments on the self-constructed benchmark ChallengeMT, which includes cases in which multiple SOTA translation systems consistently underperform. Experimental results show the effectiveness of our xIoD, which improves up to +3.85 COMET.

• 摘要：

  尽管大型语言模型 (LLM) 表现出了令人惊讶的语言理解和生成能力，但它们尚未在机器翻译领域获得革命性的进步。绩效有限的一个潜在原因是法学硕士内部对翻译 特定的理解与一般理解之间的不一致。为了使特定翻译的理解与一般理解保持一致，我们提出了一种新颖的翻译过程xIoD（难词的跨语言解释），明确地将对产生不一致理解的 内容的一般理解纳入指导翻译。具体来说，xIoD 对难以翻译的单词进行跨语言解释，并通过生成的解释来增强翻译。此外，我们重新构建了 QE 的外部工具，以应对 xIoD 在检测困难单词和生成有用解释方面的挑战。我们在自建的基准 ChallengeMT 上进行了实验，其中包括多个 SOTA 翻译系统始终表现不佳的情况。实验结果表明我们的 xIoD 的有效性，最高可提高 +3.85 COMET。

Title: Can ChatGPT Rival Neural Machine Translation? A Comparative Study. (arXiv:2401.05176v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.05176
• Code URL: null
• Copy Paste: [[2401.05176]] Can ChatGPT Rival Neural Machine Translation? A Comparative Study(http://arxiv.org/abs/2401.05176)
• Summary:

  Inspired by the increasing interest in leveraging large language models for translation, this paper evaluates the capabilities of large language models (LLMs) represented by ChatGPT in comparison to the mainstream neural machine translation (NMT) engines in translating Chinese diplomatic texts into English. Specifically, we examine the translation quality of ChatGPT and NMT engines as measured by four automated metrics and human evaluation based on an error-typology and six analytic rubrics. Our findings show that automated metrics yield similar results for ChatGPT under different prompts and NMT systems, while human annotators tend to assign noticeably higher scores to ChatGPT when it is provided an example or contextual information about the translation task. Pairwise correlation between automated metrics and dimensions of human evaluation produces weak and non-significant results, suggesting the divergence between the two methods of translation quality assessment. These findings provide valuable insights into the potential of ChatGPT as a capable machine translator, and the influence of prompt engineering on its performance.

• 摘要：

  受人们对利用大型语言模型进行翻译的兴趣日益浓厚的启发，本文评估了以 ChatGPT 为代表的大型语言模型 (LLM) 与主流神经机器翻译 (NMT) 引擎在将中国外交文本翻译成英语方面的能力。具体来说，我们检查了 ChatGPT 和 NMT 引擎的翻译质量，通过四个自动指标和基于错误类型和六个分析规则的人工评估来衡量。我们的研究结果表明，在不同的提示和 NMT 系统下，自动化指标对 ChatGPT 产生相似的结果，而当提供有关翻译任务的示例或上下文信息时，人类注释者倾向于为 ChatGPT 分配明显更高的分数。自动指标和人类评估维度之间的成对相关性产生的结果较弱且不显着，这表明两种翻译质量评估方法之间存在差异。这些发现为了解 ChatGPT 作为强大的机器翻译器的潜力以及即时工程对其性能的影响提供了宝贵的见解。

Title: Divide and Conquer for Large Language Models Reasoning. (arXiv:2401.05190v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.05190
• Code URL: https://github.com/aimijie/divide-and-conquer
• Copy Paste: [[2401.05190]] Divide and Conquer for Large Language Models Reasoning(http://arxiv.org/abs/2401.05190)
• Summary:

  Large language models (LLMs) have shown impressive performance in various reasoning benchmarks with the emergence of Chain-of-Thought (CoT) and its derivative methods, particularly in tasks involving multi-choice questions (MCQs). However, current works all process data uniformly without considering the problem-solving difficulty, which means an excessive focus on simple questions while insufficient to intricate ones. To address this challenge, we inspired by humans using heuristic strategies to categorize tasks and handle them individually, propose to apply the Divide and Conquer to LLMs reasoning. First, we divide questions into different subsets based on the statistical confidence score ($\mathcal{CS}$), then fix nearly resolved sets and conquer demanding nuanced process ones with elaborately designed methods, including Prior Knowledge based Reasoning (PKR) and Filter Choices based Reasoning (FCR), as well as their integration variants. Our experiments demonstrate that this proposed strategy significantly boosts the models' reasoning abilities across nine datasets involving arithmetic, commonsense, and logic tasks. For instance, compared to baseline, we make a striking improvement on low confidence subsets of 8.72\% for AQuA, 15.07\% for ARC Challenge and 7.71\% for RiddleSense. In addition, through extensive analysis on length of rationale and number of options, we verify that longer reasoning paths in PKR could prevent models from referring infer-harmful shortcuts, and also find that removing irrelevant choices in FCR would substantially avoid models' confusion. The code is at \url{https://github.com/AiMijie/Divide-and-Conquer}

• 摘要：

  随着思想链 (CoT) 及其衍生方法的出现，大型语言模型 (LLM) 在各种推理基准测试中表现出了令人印象深刻的性能，特别是在涉及多项选择问题 (MCQ) 的任务中。然而，目前的工作都统一处理数据，没有考虑解决问题的难度，这意味着过分关注简单问题，而对复杂问题的关注不足。为了应对这一挑战，我们受到人类 使用启发式策略对任务进行分类并单独处理的启发，建议将分而治之应用于法学硕士的推理。首先，我们根据统计置信度得分（$\mathcal{CS}$）将问题划分为不同 的子集，然后修复几乎已解决的集合，并通过精心设计的方法（包括基于先验知识的推理（PKR）和过滤器）克服要求严格的细致入微的过程问题基于选择的推理 (FCR) 及其集成变体。我们的实验表明，这种提出的策略显着提高了模型在涉及算术、常识和逻辑任务的九个数据集上的推理能力。例如，与基线相比，我们在低置信度子集上取得了显着 的改进，AQuA 为 8.72%，ARC Challenge 为 15.07%，RiddleSense 为 7.71%。此外，通过对基本原理长度和选项数量的广泛分析，我们验证了 PKR 中较长的推理路径可以防止模型引用有害的捷径，并且还发现删除 FCR 中不相关的选择将大大避免模型的混乱。代码位于\url{https://github.com/AiMijie/Divide-and-Conquer}

Title: Monte Carlo Tree Search for Recipe Generation using GPT-2. (arXiv:2401.05199v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.05199
• Code URL: null
• Copy Paste: [[2401.05199]] Monte Carlo Tree Search for Recipe Generation using GPT-2(http://arxiv.org/abs/2401.05199)
• Summary:

  Automatic food recipe generation methods provide a creative tool for chefs to explore and to create new, and interesting culinary delights. Given the recent success of large language models (LLMs), they have the potential to create new recipes that can meet individual preferences, dietary constraints, and adapt to what is in your refrigerator. Existing research on using LLMs to generate recipes has shown that LLMs can be finetuned to generate realistic-sounding recipes. However, on close examination, these generated recipes often fail to meet basic requirements like including chicken as an ingredient in chicken dishes. In this paper, we propose RecipeMC, a text generation method using GPT-2 that relies on Monte Carlo Tree Search (MCTS). RecipeMC allows us to define reward functions to put soft constraints on text generation and thus improve the credibility of the generated recipes. Our results show that human evaluators prefer recipes generated with RecipeMC more often than recipes generated with other baseline methods when compared with real recipes.

• 摘要：

  自动食品食谱生成方法为厨师提供了一个创造性的工具来探索和创造新的、有趣的烹饪美食。鉴于大型语言模型（LLM）最近取得的成功，他们有潜力创造出新的食谱，可 以满足个人喜好、饮食限制，并适应冰箱里的食物。关于使用法学硕士生成菜谱的现有研究表明，法学硕士可以进行微调以生成听起来逼真的菜谱。然而，经过仔细检查，这些生成 的食谱通常无法满足基本要求，例如将鸡肉作为鸡肉菜肴的成分。在本文中，我们提出了 RecipeMC，一种使用依赖于蒙特卡罗树搜索 (MCTS) 的 GPT-2 的文本生成方法。 RecipeMC 允许我们定义奖励函数，对文本生成施加软约束，从而提高生成菜谱的可信度。我们的结果表明，与真实菜谱相比，人类评估者更喜欢使用 RecipeMC 生成的菜谱，而不是使用其他基线方法生成的菜谱。

Title: Pre-trained Large Language Models for Financial Sentiment Analysis. (arXiv:2401.05215v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.05215
• Code URL: null
• Copy Paste: [[2401.05215]] Pre-trained Large Language Models for Financial Sentiment Analysis(http://arxiv.org/abs/2401.05215)
• Summary:

  Financial sentiment analysis refers to classifying financial text contents into sentiment categories (e.g. positive, negative, and neutral). In this paper, we focus on the classification of financial news title, which is a challenging task due to a lack of large amount of training samples. To overcome this difficulty, we propose to adapt the pretrained large language models (LLMs) [1, 2, 3] to solve this problem. The LLMs, which are trained from huge amount of text corpora,have an advantage in text understanding and can be effectively adapted to domain-specific task while requiring very few amount of training samples. In particular, we adapt the open-source Llama2-7B model (2023) with the supervised fine-tuning (SFT) technique [4]. Experimental evaluation shows that even with the 7B model (which is relatively small for LLMs), our approach significantly outperforms the previous state-of-the-art algorithms.

• 摘要：

  财经情感分析是指将财经文本内容分为情感类别（例如正面、负面和中性）。在本文中，我们关注财经新闻标题的分类，由于缺乏大量的训练样本，这是一项具有挑战性的任 务。为了克服这个困难，我们建议采用预训练的大型语言模型（LLM）[1,2,3]来解决这个问题。从大量文本语料库中训练出来的法学硕士在文本理解方面具有优势，可以 有效地适应特定领域的任务，同时需要很少量的训练样本。特别是，我们使用监督微调（SFT）技术来调整开源 Llama2-7B 模型（2023）[4]。实验评估表明，即使使用 7B 模型（对于法学硕士来说相对较小），我们的方法也明显优于以前最先进的算法。

Title: CASA: Causality-driven Argument Sufficiency Assessment. (arXiv:2401.05249v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.05249
• Code URL: https://github.com/xxxiaol/casa
• Copy Paste: [[2401.05249]] CASA: Causality-driven Argument Sufficiency Assessment(http://arxiv.org/abs/2401.05249)
• Summary:

  The argument sufficiency assessment task aims to determine if the premises of a given argument support its conclusion. To tackle this task, existing works often train a classifier on data annotated by humans. However, annotating data is laborious, and annotations are often inconsistent due to subjective criteria. Motivated by the probability of sufficiency (PS) definition in the causal literature, we propose CASA, a zero-shot causality-driven argument sufficiency assessment framework. PS measures how likely introducing the premise event would lead to the conclusion, when both the premise and conclusion events are absent. To estimate this probability, we propose to use large language models (LLMs) to generate contexts that are inconsistent with the premise and conclusion, and revise them by injecting the premise event. Experiments on two logical fallacy detection datasets demonstrate that CASA accurately identifies insufficient arguments. We further deploy CASA in a writing assistance application, and find that suggestions generated by CASA enhance the sufficiency of student-written arguments. Code and data are available at https://github.com/xxxiaol/CASA.

• 摘要：

  论证充分性评估任务旨在确定给定论证的前提是否支持其结论。为了解决这一任务，现有的工作通常会根据人类注释的数据来训练分类器。然而，对数据进行注释是费力的， 并且由于主观标准，注释常常不一致。受因果文献中充分性概率（PS）定义的启发，我们提出了 CASA，一种零样本因果驱动的论证充分性评估框架。 PS 衡量当前提事 件和结论事件都不存在时，引入前提事件导致结论的可能性有多大。为了估计这个概率，我们建议使用大型语言模型（LLM）来生成与前提和结论不一致的上下文，并通过注入前 提事件来修改它们。对两个逻辑谬误检测数据集的实验表明，CASA 可以准确识别不充分的论点。我们进一步在写作辅助应用程序中部署 CASA，发现 CASA 生成的建议增强了学生书面论证的充分性。代码和数据可在 https://github.com/xxxiaol/CASA 获取。

Title: INACIA: Integrating Large Language Models in Brazilian Audit Courts: Opportunities and Challenges. (arXiv:2401.05273v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.05273
• Code URL: null
• Copy Paste: [[2401.05273]] INACIA: Integrating Large Language Models in Brazilian Audit Courts: Opportunities and Challenges(http://arxiv.org/abs/2401.05273)
• Summary:

  This paper introduces INACIA (Instru\c{c}\~ao Assistida com Intelig\^encia Artificial), a groundbreaking system designed to integrate Large Language Models (LLMs) into the operational framework of Brazilian Federal Court of Accounts (TCU). The system automates various stages of case analysis, including basic information extraction, admissibility examination, Periculum in mora and Fumus boni iuris analyses, and recommendations generation. Through a series of experiments, we demonstrate INACIA's potential in extracting relevant information from case documents, evaluating its legal plausibility, and generating judicial recommendations. Utilizing a validation dataset alongside LLMs, our evaluation methodology presents an innovative approach to assessing system performance, correlating highly with human judgment. The results highlight INACIA's proficiency in handling complex legal tasks, indicating its suitability for augmenting efficiency and judicial fairness within legal systems. The paper also discusses potential enhancements and future applications, positioning INACIA as a model for worldwide AI integration in legal domains.

• 摘要：

  本文介绍了 INACIA (Instru\c{c}\~ao Assistida com Intelig\^encia Artificial)，这是一个突破性的系统，旨在将大型语言模型 (LLM) 集成到巴西联邦会计法院 (TCU) 的运营框架中）。该系统自动执行案例分析的各个阶段，包括基本信息提取、可受理性审查、Mora 和 Fumus boni iuris 分析以及建议生成。通过一系列实验，我们展示了 INACIA 在从案件文件中提取相关信息、评估其法律合理性以及生成司法建议方面的潜力。通过利用验证数据集和法学硕 士，我们的评估方法提出了一种评估系统性能的创新方法，与人类判断高度相关。结果突显了 INACIA 在处理复杂法律任务方面的熟练程度，表明其适合提高法律体系内的效率和司法公平。该论文还讨论了潜在的增强功能和未来的应用，将 INACIA 定位为法律领域全球人工智能集成的模型。

Title: I am a Strange Dataset: Metalinguistic Tests for Language Models. (arXiv:2401.05300v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.05300
• Code URL: https://github.com/tristanthrush/i-am-a-strange-dataset
• Copy Paste: [[2401.05300]] I am a Strange Dataset: Metalinguistic Tests for Language Models(http://arxiv.org/abs/2401.05300)
• Summary:

  Statements involving metalinguistic self-reference ("This paper has six sections.") are prevalent in many domains. Can large language models (LLMs) handle such language? In this paper, we present "I am a Strange Dataset", a new dataset for addressing this question. There are two subtasks: generation and verification. In generation, models continue statements like "The penultimate word in this sentence is" (where a correct continuation is "is"). In verification, models judge the truth of statements like "The penultimate word in this sentence is sentence." (false). We also provide minimally different metalinguistic non-self-reference examples to complement the main dataset by probing for whether models can handle metalinguistic language at all. The dataset is hand-crafted by experts and validated by non-expert annotators. We test a variety of open-source LLMs (7B to 70B parameters) as well as closed-source LLMs through APIs. All models perform close to chance across both subtasks and even on the non-self-referential metalinguistic control data, though we find some steady improvement with model scale. GPT 4 is the only model to consistently do significantly better than chance, and it is still only in the 60% range, while our untrained human annotators score well in the 89-93% range. The dataset and evaluation toolkit are available at https://github.com/TristanThrush/i-am-a-strange-dataset.

• 摘要：

  涉及元语言自指的陈述（“本文有六个部分。”）在许多领域都很普遍。大型语言模型（LLM）可以处理这种语言吗？在本文中，我们提出了“我是一个奇怪的数据集”， 这是一个解决这个问题的新数据集。有两个子任务：生成和验证。在生成过程中，模型会继续诸如“这句话中的倒数第二个词是”之类的语句（其中正确的延续是“is”）。在验 证中，模型判断诸如“这句话中的倒数第二个词是句子”之类的陈述的真实性。 （错误的）。我们还提供了最小差异的元语言非自引用示例，通过探索模型是否可以处理元语言语言来补充主数据集。该数据集由专家手工制作，并由非专家注释者验证。我们通过 API 测试各种开源 LLM（7B 至 70B 参数）以及闭源 LLM。尽管我们发现模型规模有了一些稳定的改进，但所有模型在两个子任务上，甚至在非自指元语言控制数据上的表现都接近偶然。 GPT 4 是唯一一个始终显着优于随机性的模型，但它仍然只在 60% 的范围内，而我们未经训练的人类注释者的得分在 89-93% 的范围内。数据集和评估工具包可从 https://github.com/TristanThrush/i-am-a-strange-dataset 获取。

Title: Leveraging Print Debugging to Improve Code Generation in Large Language Models. (arXiv:2401.05319v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.05319
• Code URL: null
• Copy Paste: [[2401.05319]] Leveraging Print Debugging to Improve Code Generation in Large Language Models(http://arxiv.org/abs/2401.05319)
• Summary:

  Large language models (LLMs) have made significant progress in code generation tasks, but their performance in tackling programming problems with complex data structures and algorithms remains suboptimal. To address this issue, we propose an in-context learning approach that guides LLMs to debug by using a "print debugging" method, which involves inserting print statements to trace and analysing logs for fixing the bug. We collect a Leetcode problem dataset and evaluate our method using the Leetcode online judging system. Experiments with GPT-4 demonstrate the effectiveness of our approach, outperforming rubber duck debugging in easy and medium-level Leetcode problems by 1.5% and 17.9%.

• 摘要：

  大型语言模型 (LLM) 在代码生成任务方面取得了重大进展，但它们在处理复杂数据结构和算法的编程问题方面的性能仍然不够理想。为了解决这个问题，我们提出了 一种上下文学习方法，引导法学硕士使用“打印调试”方法进行调试，其中包括插入打印语句来跟踪和分析日志以修复错误。我们收集 Leetcode 问题数据集并使用 Leetcode 在线评审系统评估我们的方法。 GPT-4 的实验证明了我们方法的有效性，在简单和中等水平的 Leetcode 问题上，其性能比橡皮鸭调试高出 1.5% 和 17.9%。

Title: How predictable is language model benchmark performance?. (arXiv:2401.04757v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.04757
• Code URL: null
• Copy Paste: [[2401.04757]] How predictable is language model benchmark performance?(http://arxiv.org/abs/2401.04757)
• Summary:

  We investigate large language model performance across five orders of magnitude of compute scaling in eleven recent model architectures. We show that average benchmark performance, aggregating over many individual tasks and evaluations as in the commonly-used BIG-Bench dataset, is decently predictable as a function of training compute scale. Specifically, when extrapolating BIG-Bench Hard performance across one order of magnitude in compute, we observe average absolute errors of 6 percentage points (pp). By contrast, extrapolation for individual BIG-Bench tasks across an order of magnitude in compute yields higher average errors of 18pp. Nonetheless, individual task performance remains significantly more predictable than chance. Overall, our work suggests compute scaling provides a promising basis to forecast AI capabilities in diverse benchmarks, though predicting performance in specific tasks poses challenges.

• 摘要：

  我们在 11 个最新模型架构中研究了跨五个数量级的计算扩展的大型语言模型性能。我们表明，平均基准性能（如常用的 BIG-Bench 数据集中的许多单独任务和评估的汇总）作为训练计算规模的函数是可以很好预测的。具体来说，当在计算中将 BIG-Bench Hard 性能推断出一个数量级时，我们观察到平均绝对误差为 6 个百分点 (pp)。相比之下，在计算量级上对单个 BIG-Bench 任务进行外推会产生 18pp 的较 高平均误差。尽管如此，个人任务的表现仍然比偶然更容易预测。总的来说，我们的工作表明，计算扩展为预测不同基准中的人工智能能力提供了一个有希望的基础，尽管预测特定 任务的性能提出了挑战。

segmentation

Title: Latency-aware Road Anomaly Segmentation in Videos: A Photorealistic Dataset and New Metrics. (arXiv:2401.04942v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.04942
• Code URL: null
• Copy Paste: [[2401.04942]] Latency-aware Road Anomaly Segmentation in Videos: A Photorealistic Dataset and New Metrics(http://arxiv.org/abs/2401.04942)
• Summary:

  In the past several years, road anomaly segmentation is actively explored in the academia and drawing growing attention in the industry. The rationale behind is straightforward: if the autonomous car can brake before hitting an anomalous object, safety is promoted. However, this rationale naturally calls for a temporally informed setting while existing methods and benchmarks are designed in an unrealistic frame-wise manner. To bridge this gap, we contribute the first video anomaly segmentation dataset for autonomous driving. Since placing various anomalous objects on busy roads and annotating them in every frame are dangerous and expensive, we resort to synthetic data. To improve the relevance of this synthetic dataset to real-world applications, we train a generative adversarial network conditioned on rendering G-buffers for photorealism enhancement. Our dataset consists of 120,000 high-resolution frames at a 60 FPS framerate, as recorded in 7 different towns. As an initial benchmarking, we provide baselines using latest supervised and unsupervised road anomaly segmentation methods. Apart from conventional ones, we focus on two new metrics: temporal consistency and latencyaware streaming accuracy. We believe the latter is valuable as it measures whether an anomaly segmentation algorithm can truly prevent a car from crashing in a temporally informed setting.

• 摘要：

  过去几年，道路异常分割在学术界得到了积极探索，并越来越受到业界的关注。背后的原理很简单：如果自动驾驶汽车能够在撞到异常物体之前刹车，那么安全性就会得到提 升。然而，这个基本原理自然需要一个临时通知的设置，而现有的方法和基准是以不切实际的框架方式设计的。为了弥补这一差距，我们贡献了第一个用于自动驾驶的视频异常分割 数据集。由于将各种异常物体放置在繁忙的道路上并在每一帧中对其进行注释既危险又昂贵，因此我们求助于合成数据。为了提高该合成数据集与现实世界应用的相关性，我们训练 了一个以渲染 G 缓冲区为条件的生成对抗网络，以增强照片真实感。我们的数据集由 7 个不同城镇记录的 120,000 个 60 FPS 帧速率的高分辨率帧组成 。作为初始基准测试，我们使用最新的监督和无监督道路异常分割方法提供基线。除了传统指标之外，我们还关注两个新指标：时间一致性和延迟感知流准确性。我们认为后者很有 价值，因为它衡量异常分割算法是否能够真正防止汽车在临时信息环境中发生碰撞。

Title: Enhanced Muscle and Fat Segmentation for CT-Based Body Composition Analysis: A Comparative Study. (arXiv:2401.05294v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.05294
• Code URL: null
• Copy Paste: [[2401.05294]] Enhanced Muscle and Fat Segmentation for CT-Based Body Composition Analysis: A Comparative Study(http://arxiv.org/abs/2401.05294)
• Summary:

  Purpose: Body composition measurements from routine abdominal CT can yield personalized risk assessments for asymptomatic and diseased patients. In particular, attenuation and volume measures of muscle and fat are associated with important clinical outcomes, such as cardiovascular events, fractures, and death. This study evaluates the reliability of an Internal tool for the segmentation of muscle and fat (subcutaneous and visceral) as compared to the well-established public TotalSegmentator tool.

  Methods: We assessed the tools across 900 CT series from the publicly available SAROS dataset, focusing on muscle, subcutaneous fat, and visceral fat. The Dice score was employed to assess accuracy in subcutaneous fat and muscle segmentation. Due to the lack of ground truth segmentations for visceral fat, Cohen's Kappa was utilized to assess segmentation agreement between the tools.

  Results: Our Internal tool achieved a 3% higher Dice (83.8 vs. 80.8) for subcutaneous fat and a 5% improvement (87.6 vs. 83.2) for muscle segmentation respectively. A Wilcoxon signed-rank test revealed that our results were statistically different with p<0.01. For visceral fat, the Cohen's kappa score of 0.856 indicated near-perfect agreement between the two tools. Our internal tool also showed very strong correlations for muscle volume (R^2=0.99), muscle attenuation (R^2=0.93), and subcutaneous fat volume (R^2=0.99) with a moderate correlation for subcutaneous fat attenuation (R^2=0.45).

  Conclusion: Our findings indicated that our Internal tool outperformed TotalSegmentator in measuring subcutaneous fat and muscle. The high Cohen's Kappa score for visceral fat suggests a reliable level of agreement between the two tools. These results demonstrate the potential of our tool in advancing the accuracy of body composition analysis.

• 摘要：

  目的：通过常规腹部 CT 进行身体成分测量，可以对无症状和患病患者进行个性化风险评估。特别是，肌肉和脂肪的衰减和体积测量与重要的临床结果相关，例如心血管 事件、骨折和死亡。本研究评估了用于分割肌肉和脂肪（皮下和内脏）的内部工具与成熟的公共 TotalSegmentator 工具相比的可靠性。

  方法：我们评估了公开的 SAROS 数据集中的 900 个 CT 系列的工具，重点关注肌肉、皮下脂肪和内脏脂肪。 Dice 评分用于评估皮下脂肪和肌肉分割的准确性。由于缺乏内脏脂肪的真实分割，因此利用 Cohen 的 Kappa 来评估工具之间的分割一致性。

  结果：我们的内部工具在皮下脂肪方面的 Dice 提高了 3%（83.8 比 80.8），在肌肉分割方面提高了 5%（87.6 比 83.2）。 Wilcoxon 符号秩检验显示我们的结果在统计上存在差异，p<0.01。对于内脏脂肪，Cohen 的 kappa 分数为 0.856，表明两种工具之间几乎完美一致。我们的内部工具还显示，肌肉体积 (R^2=0.99)、肌肉衰减 (R^2=0.93) 和皮下脂肪体积 (R^2=0.99) 之间具有非常强的相关性，而皮下脂肪衰减 (R^2=0.99) 具有中等相关性。 ^2=0.45）。

  结论：我们的研究结果表明，我们的内部工具在测量皮下脂肪和肌肉方面优于 TotalSegmentator。内脏脂肪的高 Cohen Kappa 分数表明两种工具之间具有可靠的一致性。这些结果证明了我们的工具在提高身体成分分析准确性方面的潜力。