2024-01-23

Title: Make-A-Shape: a Ten-Million-scale 3D Shape Model

• Authors: Ka-Hei Hui, Aditya Sanghi, Arianna Rampini, Kamal Rahimi Malekshan, Zhengzhe Liu, Hooman Shayani, Chi-Wing Fu
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2401.11067
• Pdf URL: https://arxiv.org/pdf/2401.11067
• Copy Paste: [[2401.11067]] Make-A-Shape: a Ten-Million-scale 3D Shape Model(https://arxiv.org/abs/2401.11067)
• Keywords: diffusion, generative
• Abstract: Significant progress has been made in training large generative models for natural language and images. Yet, the advancement of 3D generative models is hindered by their substantial resource demands for training, along with inefficient, non-compact, and less expressive representations. This paper introduces Make-A-Shape, a new 3D generative model designed for efficient training on a vast scale, capable of utilizing 10 millions publicly-available shapes. Technical-wise, we first innovate a wavelet-tree representation to compactly encode shapes by formulating the subband coefficient filtering scheme to efficiently exploit coefficient relations. We then make the representation generatable by a diffusion model by devising the subband coefficients packing scheme to layout the representation in a low-resolution grid. Further, we derive the subband adaptive training strategy to train our model to effectively learn to generate coarse and detail wavelet coefficients. Last, we extend our framework to be controlled by additional input conditions to enable it to generate shapes from assorted modalities, e.g., single/multi-view images, point clouds, and low-resolution voxels. In our extensive set of experiments, we demonstrate various applications, such as unconditional generation, shape completion, and conditional generation on a wide range of modalities. Our approach not only surpasses the state of the art in delivering high-quality results but also efficiently generates shapes within a few seconds, often achieving this in just 2 seconds for most conditions.

Title: UltrAvatar: A Realistic Animatable 3D Avatar Diffusion Model with Authenticity Guided Textures

• Authors: Mingyuan Zhou, Rakib Hyder, Ziwei Xuan, Guojun Qi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.11078
• Pdf URL: https://arxiv.org/pdf/2401.11078
• Copy Paste: [[2401.11078]] UltrAvatar: A Realistic Animatable 3D Avatar Diffusion Model with Authenticity Guided Textures(https://arxiv.org/abs/2401.11078)
• Keywords: diffusion
• Abstract: Recent advances in 3D avatar generation have gained significant attentions. These breakthroughs aim to produce more realistic animatable avatars, narrowing the gap between virtual and real-world experiences. Most of existing works employ Score Distillation Sampling (SDS) loss, combined with a differentiable renderer and text condition, to guide a diffusion model in generating 3D avatars. However, SDS often generates oversmoothed results with few facial details, thereby lacking the diversity compared with ancestral sampling. On the other hand, other works generate 3D avatar from a single image, where the challenges of unwanted lighting effects, perspective views, and inferior image quality make them difficult to reliably reconstruct the 3D face meshes with the aligned complete textures. In this paper, we propose a novel 3D avatar generation approach termed UltrAvatar with enhanced fidelity of geometry, and superior quality of physically based rendering (PBR) textures without unwanted lighting. To this end, the proposed approach presents a diffuse color extraction model and an authenticity guided texture diffusion model. The former removes the unwanted lighting effects to reveal true diffuse colors so that the generated avatars can be rendered under various lighting conditions. The latter follows two gradient-based guidances for generating PBR textures to render diverse face-identity features and details better aligning with 3D mesh geometry. We demonstrate the effectiveness and robustness of the proposed method, outperforming the state-of-the-art methods by a large margin in the experiments.

Title: Exploiting Duality in Open Information Extraction with Predicate Prompt

• Authors: Zhen Chen, Jingping Liu, Deqing Yang, Yanghua Xiao, Huimin Xu, Zongyu Wang, Rui Xie, Yunsen Xian
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2401.11107
• Pdf URL: https://arxiv.org/pdf/2401.11107
• Copy Paste: [[2401.11107]] Exploiting Duality in Open Information Extraction with Predicate Prompt(https://arxiv.org/abs/2401.11107)
• Keywords: generative
• Abstract: Open information extraction (OpenIE) aims to extract the schema-free triplets in the form of (\emph{subject}, \emph{predicate}, \emph{object}) from a given sentence. Compared with general information extraction (IE), OpenIE poses more challenges for the IE models, {especially when multiple complicated triplets exist in a sentence. To extract these complicated triplets more effectively, in this paper we propose a novel generative OpenIE model, namely \emph{DualOIE}, which achieves a dual task at the same time as extracting some triplets from the sentence, i.e., converting the triplets into the sentence.} Such dual task encourages the model to correctly recognize the structure of the given sentence and thus is helpful to extract all potential triplets from the sentence. Specifically, DualOIE extracts the triplets in two steps: 1) first extracting a sequence of all potential predicates, 2) then using the predicate sequence as a prompt to induce the generation of triplets. Our experiments on two benchmarks and our dataset constructed from Meituan demonstrate that DualOIE achieves the best performance among the state-of-the-art baselines. Furthermore, the online A/B test on Meituan platform shows that 0.93\% improvement of QV-CTR and 0.56\% improvement of UV-CTR have been obtained when the triplets extracted by DualOIE were leveraged in Meituan's search system.

Title: MotionMix: Weakly-Supervised Diffusion for Controllable Motion Generation

• Authors: Nhat M. Hoang, Kehong Gong, Chuan Guo, Michael Bi Mi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.11115
• Pdf URL: https://arxiv.org/pdf/2401.11115
• Copy Paste: [[2401.11115]] MotionMix: Weakly-Supervised Diffusion for Controllable Motion Generation(https://arxiv.org/abs/2401.11115)
• Keywords: diffusion
• Abstract: Controllable generation of 3D human motions becomes an important topic as the world embraces digital transformation. Existing works, though making promising progress with the advent of diffusion models, heavily rely on meticulously captured and annotated (e.g., text) high-quality motion corpus, a resource-intensive endeavor in the real world. This motivates our proposed MotionMix, a simple yet effective weakly-supervised diffusion model that leverages both noisy and unannotated motion sequences. Specifically, we separate the denoising objectives of a diffusion model into two stages: obtaining conditional rough motion approximations in the initial $T-T^*$ steps by learning the noisy annotated motions, followed by the unconditional refinement of these preliminary motions during the last $T^*$ steps using unannotated motions. Notably, though learning from two sources of imperfect data, our model does not compromise motion generation quality compared to fully supervised approaches that access gold data. Extensive experiments on several benchmarks demonstrate that our MotionMix, as a versatile framework, consistently achieves state-of-the-art performances on text-to-motion, action-to-motion, and music-to-dance tasks.

Title: Spatial Structure Constraints for Weakly Supervised Semantic Segmentation

• Authors: Tao Chen, Yazhou Yao, Xingguo Huang, Zechao Li, Liqiang Nie, Jinhui Tang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.11122
• Pdf URL: https://arxiv.org/pdf/2401.11122
• Copy Paste: [[2401.11122]] Spatial Structure Constraints for Weakly Supervised Semantic Segmentation(https://arxiv.org/abs/2401.11122)
• Keywords: diffusion
• Abstract: The image-level label has prevailed in weakly supervised semantic segmentation tasks due to its easy availability. Since image-level labels can only indicate the existence or absence of specific categories of objects, visualization-based techniques have been widely adopted to provide object location clues. Considering class activation maps (CAMs) can only locate the most discriminative part of objects, recent approaches usually adopt an expansion strategy to enlarge the activation area for more integral object localization. However, without proper constraints, the expanded activation will easily intrude into the background region. In this paper, we propose spatial structure constraints (SSC) for weakly supervised semantic segmentation to alleviate the unwanted object over-activation of attention expansion. Specifically, we propose a CAM-driven reconstruction module to directly reconstruct the input image from deep CAM features, which constrains the diffusion of last-layer object attention by preserving the coarse spatial structure of the image content. Moreover, we propose an activation self-modulation module to refine CAMs with finer spatial structure details by enhancing regional consistency. Without external saliency models to provide background clues, our approach achieves 72.7\% and 47.0\% mIoU on the PASCAL VOC 2012 and COCO datasets, respectively, demonstrating the superiority of our proposed approach.

Title: Projected Belief Networks With Discriminative Alignment for Acoustic Event Classification: Rivaling State of the Art CNNs

• Authors: Paul M. Baggenstoss, Kevin Wilkinghoff, Felix Govaers, Frank Kurth
• Subjects: cs.LG, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2401.11199
• Pdf URL: https://arxiv.org/pdf/2401.11199
• Copy Paste: [[2401.11199]] Projected Belief Networks With Discriminative Alignment for Acoustic Event Classification: Rivaling State of the Art CNNs(https://arxiv.org/abs/2401.11199)
• Keywords: generative
• Abstract: The projected belief network (PBN) is a generative stochastic network with tractable likelihood function based on a feed-forward neural network (FFNN). The generative function operates by "backing up" through the FFNN. The PBN is two networks in one, a FFNN that operates in the forward direction, and a generative network that operates in the backward direction. Both networks co-exist based on the same parameter set, have their own cost functions, and can be separately or jointly trained. The PBN therefore has the potential to possess the best qualities of both discriminative and generative classifiers. To realize this potential, a separate PBN is trained on each class, maximizing the generative likelihood function for the given class, while minimizing the discriminative cost for the FFNN against "all other classes". This technique, called discriminative alignment (PBN-DA), aligns the contours of the likelihood function to the decision boundaries and attains vastly improved classification performance, rivaling that of state of the art discriminative networks. The method may be further improved using a hidden Markov model (HMM) as a component of the PBN, called PBN-DA-HMM. This paper provides a comprehensive treatment of PBN, PBN-DA, and PBN-DA-HMM. In addition, the results of two new classification experiments are provided. The first experiment uses air-acoustic events, and the second uses underwater acoustic data consisting of marine mammal calls. In both experiments, PBN-DA-HMM attains comparable or better performance as a state of the art CNN, and attain a factor of two error reduction when combined with the CNN.

Title: TreeMIL: A Multi-instance Learning Framework for Time Series Anomaly Detection with Inexact Supervision

• Authors: Chen Liu, Shibo He, Haoyu Liu, Shizhong Li
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.11235
• Pdf URL: https://arxiv.org/pdf/2401.11235
• Copy Paste: [[2401.11235]] TreeMIL: A Multi-instance Learning Framework for Time Series Anomaly Detection with Inexact Supervision(https://arxiv.org/abs/2401.11235)
• Keywords: anomaly
• Abstract: Time series anomaly detection (TSAD) plays a vital role in various domains such as healthcare, networks, and industry. Considering labels are crucial for detection but difficult to obtain, we turn to TSAD with inexact supervision: only series-level labels are provided during the training phase, while point-level anomalies are predicted during the testing phase. Previous works follow a traditional multi-instance learning (MIL) approach, which focuses on encouraging high anomaly scores at individual time steps. However, time series anomalies are not only limited to individual point anomalies, they can also be collective anomalies, typically exhibiting abnormal patterns over subsequences. To address the challenge of collective anomalies, in this paper, we propose a tree-based MIL framework (TreeMIL). We first adopt an N-ary tree structure to divide the entire series into multiple nodes, where nodes at different levels represent subsequences with different lengths. Then, the subsequence features are extracted to determine the presence of collective anomalies. Finally, we calculate point-level anomaly scores by aggregating features from nodes at different levels. Experiments conducted on seven public datasets and eight baselines demonstrate that TreeMIL achieves an average 32.3% improvement in F1- score compared to previous state-of-the-art methods. The code is available at https://github.com/fly-orange/TreeMIL.

Title: Product-Level Try-on: Characteristics-preserving Try-on with Realistic Clothes Shading and Wrinkles

• Authors: Yanlong Zang, Han Yang, Jiaxu Miao, Yi Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.11239
• Pdf URL: https://arxiv.org/pdf/2401.11239
• Copy Paste: [[2401.11239]] Product-Level Try-on: Characteristics-preserving Try-on with Realistic Clothes Shading and Wrinkles(https://arxiv.org/abs/2401.11239)
• Keywords: diffusion, generative
• Abstract: Image-based virtual try-on systems,which fit new garments onto human portraits,are gaining research attention.An ideal pipeline should preserve the static features of clothes(like textures and logos)while also generating dynamic elements(e.g.shadows,folds)that adapt to the model's pose and environment.Previous works fail specifically in generating dynamic features,as they preserve the warped in-shop clothes trivially with predicted an alpha mask by composition.To break the dilemma of over-preserving and textures losses,we propose a novel diffusion-based Product-level virtual try-on pipeline,\ie PLTON, which can preserve the fine details of logos and embroideries while producing realistic clothes shading and wrinkles.The main insights are in three folds:1)Adaptive Dynamic Rendering:We take a pre-trained diffusion model as a generative prior and tame it with image features,training a dynamic extractor from scratch to generate dynamic tokens that preserve high-fidelity semantic information. Due to the strong generative power of the diffusion prior,we can generate realistic clothes shadows and wrinkles.2)Static Characteristics Transformation: High-frequency Map(HF-Map)is our fundamental insight for static representation.PLTON first warps in-shop clothes to the target model pose by a traditional warping network,and uses a high-pass filter to extract an HF-Map for preserving static cloth features.The HF-Map is used to generate modulation maps through our static extractor,which are injected into a fixed U-net to synthesize the final result.To enhance retention,a Two-stage Blended Denoising method is proposed to guide the diffusion process for correct spatial layout and color.PLTON is finetuned only with our collected small-size try-on dataset.Extensive quantitative and qualitative experiments on 1024 768 datasets demonstrate the superiority of our framework in mimicking real clothes dynamics.

Title: Prompt-RAG: Pioneering Vector Embedding-Free Retrieval-Augmented Generation in Niche Domains, Exemplified by Korean Medicine

• Authors: Bongsu Kang, Jundong Kim, Tae-Rim Yun, Chang-Eop Kim
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2401.11246
• Pdf URL: https://arxiv.org/pdf/2401.11246
• Copy Paste: [[2401.11246]] Prompt-RAG: Pioneering Vector Embedding-Free Retrieval-Augmented Generation in Niche Domains, Exemplified by Korean Medicine(https://arxiv.org/abs/2401.11246)
• Keywords: generative
• Abstract: We propose a natural language prompt-based retrieval augmented generation (Prompt-RAG), a novel approach to enhance the performance of generative large language models (LLMs) in niche domains. Conventional RAG methods mostly require vector embeddings, yet the suitability of generic LLM-based embedding representations for specialized domains remains uncertain. To explore and exemplify this point, we compared vector embeddings from Korean Medicine (KM) and Conventional Medicine (CM) documents, finding that KM document embeddings correlated more with token overlaps and less with human-assessed document relatedness, in contrast to CM embeddings. Prompt-RAG, distinct from conventional RAG models, operates without the need for embedding vectors. Its performance was assessed through a Question-Answering (QA) chatbot application, where responses were evaluated for relevance, readability, and informativeness. The results showed that Prompt-RAG outperformed existing models, including ChatGPT and conventional vector embedding-based RAGs, in terms of relevance and informativeness. Despite challenges like content structuring and response latency, the advancements in LLMs are expected to encourage the use of Prompt-RAG, making it a promising tool for other domains in need of RAG methods.

Title: Diffusion Model Conditioning on Gaussian Mixture Model and Negative Gaussian Mixture Gradient

• Authors: Weiguo Lu, Xuan Wu, Deng Ding, Jinqiao Duan, Jirong Zhuang, Gangnan Yuan
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2401.11261
• Pdf URL: https://arxiv.org/pdf/2401.11261
• Copy Paste: [[2401.11261]] Diffusion Model Conditioning on Gaussian Mixture Model and Negative Gaussian Mixture Gradient(https://arxiv.org/abs/2401.11261)
• Keywords: diffusion, generative
• Abstract: Diffusion models (DMs) are a type of generative model that has a huge impact on image synthesis and beyond. They achieve state-of-the-art generation results in various generative tasks. A great diversity of conditioning inputs, such as text or bounding boxes, are accessible to control the generation. In this work, we propose a conditioning mechanism utilizing Gaussian mixture models (GMMs) as feature conditioning to guide the denoising process. Based on set theory, we provide a comprehensive theoretical analysis that shows that conditional latent distribution based on features and classes is significantly different, so that conditional latent distribution on features produces fewer defect generations than conditioning on classes. Two diffusion models conditioned on the Gaussian mixture model are trained separately for comparison. Experiments support our findings. A novel gradient function called the negative Gaussian mixture gradient (NGMG) is proposed and applied in diffusion model training with an additional classifier. Training stability has improved. We also theoretically prove that NGMG shares the same benefit as the Earth Mover distance (Wasserstein) as a more sensible cost function when learning distributions supported by low-dimensional manifolds.

Title: DACR: Distribution-Augmented Contrastive Reconstruction for Time-Series Anomaly Detection

• Authors: Lixu Wang, Shichao Xu, Xinyu Du, Qi Zhu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.11271
• Pdf URL: https://arxiv.org/pdf/2401.11271
• Copy Paste: [[2401.11271]] DACR: Distribution-Augmented Contrastive Reconstruction for Time-Series Anomaly Detection(https://arxiv.org/abs/2401.11271)
• Keywords: anomaly
• Abstract: Anomaly detection in time-series data is crucial for identifying faults, failures, threats, and outliers across a range of applications. Recently, deep learning techniques have been applied to this topic, but they often struggle in real-world scenarios that are complex and highly dynamic, e.g., the normal data may consist of multiple distributions, and various types of anomalies may differ from the normal data to different degrees. In this work, to tackle these challenges, we propose Distribution-Augmented Contrastive Reconstruction (DACR). DACR generates extra data disjoint from the normal data distribution to compress the normal data's representation space, and enhances the feature extractor through contrastive learning to better capture the intrinsic semantics from time-series data. Furthermore, DACR employs an attention mechanism to model the semantic dependencies among multivariate time-series features, thereby achieving more robust reconstruction for anomaly detection. Extensive experiments conducted on nine benchmark datasets in various anomaly detection scenarios demonstrate the effectiveness of DACR in achieving new state-of-the-art time-series anomaly detection.

Title: Long-Term Fair Decision Making through Deep Generative Models

• Authors: Yaowei Hu, Yongkai Wu, Lu Zhang
• Subjects: cs.LG, cs.CY
• Abstract URL: https://arxiv.org/abs/2401.11288
• Pdf URL: https://arxiv.org/pdf/2401.11288
• Copy Paste: [[2401.11288]] Long-Term Fair Decision Making through Deep Generative Models(https://arxiv.org/abs/2401.11288)
• Keywords: generative
• Abstract: This paper studies long-term fair machine learning which aims to mitigate group disparity over the long term in sequential decision-making systems. To define long-term fairness, we leverage the temporal causal graph and use the 1-Wasserstein distance between the interventional distributions of different demographic groups at a sufficiently large time step as the quantitative metric. Then, we propose a three-phase learning framework where the decision model is trained on high-fidelity data generated by a deep generative model. We formulate the optimization problem as a performative risk minimization and adopt the repeated gradient descent algorithm for learning. The empirical evaluation shows the efficacy of the proposed method using both synthetic and semi-synthetic datasets.

Title: A Novel Benchmark for Few-Shot Semantic Segmentation in the Era of Foundation Models

• Authors: Reda Bensaid, Vincent Gripon, François Leduc-Primeau, Lukas Mauch, Ghouthi Boukli Hacene, Fabien Cardinaux
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.11311
• Pdf URL: https://arxiv.org/pdf/2401.11311
• Copy Paste: [[2401.11311]] A Novel Benchmark for Few-Shot Semantic Segmentation in the Era of Foundation Models(https://arxiv.org/abs/2401.11311)
• Keywords: foundation model
• Abstract: In recent years, the rapid evolution of computer vision has seen the emergence of various vision foundation models, each tailored to specific data types and tasks. While large language models often share a common pretext task, the diversity in vision foundation models arises from their varying training objectives. In this study, we delve into the quest for identifying the most effective vision foundation models for few-shot semantic segmentation, a critical task in computer vision. Specifically, we conduct a comprehensive comparative analysis of four prominent foundation models: DINO V2, Segment Anything, CLIP, Masked AutoEncoders, and a straightforward ResNet50 pre-trained on the COCO dataset. Our investigation focuses on their adaptability to new semantic segmentation tasks, leveraging only a limited number of segmented images. Our experimental findings reveal that DINO V2 consistently outperforms the other considered foundation models across a diverse range of datasets and adaptation methods. This outcome underscores DINO V2's superior capability to adapt to semantic segmentation tasks compared to its counterparts. Furthermore, our observations indicate that various adapter methods exhibit similar performance, emphasizing the paramount importance of selecting a robust feature extractor over the intricacies of the adaptation technique itself. This insight sheds light on the critical role of feature extraction in the context of few-shot semantic segmentation. This research not only contributes valuable insights into the comparative performance of vision foundation models in the realm of few-shot semantic segmentation but also highlights the significance of a robust feature extractor in this domain.

Title: Analyzing Task-Encoding Tokens in Large Language Models

• Authors: Yu Bai, Heyan Huang, Cesare Spinoso-Di Piano, Marc-Antoine Rondeau, Sanxing Chen, Yang Gao, Jackie Chi Kit Cheung
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.11323
• Pdf URL: https://arxiv.org/pdf/2401.11323
• Copy Paste: [[2401.11323]] Analyzing Task-Encoding Tokens in Large Language Models(https://arxiv.org/abs/2401.11323)
• Keywords: in-context
• Abstract: In-context learning (ICL) has become an effective solution for few-shot learning in natural language processing. Past work has found that, during this process, representations of the last prompt token are utilized to store task reasoning procedures, thereby explaining the working mechanism of in-context learning. In this paper, we seek to locate and analyze other task-encoding tokens whose representations store task reasoning procedures. Supported by experiments that ablate the representations of different token types, we find that template and stopword tokens are the most prone to be task-encoding tokens. In addition, we demonstrate experimentally that lexical cues, repetition, and text formats are the main distinguishing characteristics of these tokens. Our work provides additional insights into how large language models (LLMs) leverage task reasoning procedures in ICL and suggests that future work may involve using task-encoding tokens to improve the computational efficiency of LLMs at inference time and their ability to handle long sequences.

Title: Prompting Large Vision-Language Models for Compositional Reasoning

• Authors: Timothy Ossowski, Ming Jiang, Junjie Hu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.11337
• Pdf URL: https://arxiv.org/pdf/2401.11337
• Copy Paste: [[2401.11337]] Prompting Large Vision-Language Models for Compositional Reasoning(https://arxiv.org/abs/2401.11337)
• Keywords: generative
• Abstract: Vision-language models such as CLIP have shown impressive capabilities in encoding texts and images into aligned embeddings, enabling the retrieval of multimodal data in a shared embedding space. However, these embedding-based models still face challenges in effectively matching images and texts with similar visio-linguistic compositionality, as evidenced by their performance on the recent Winoground dataset. In this paper, we argue that this limitation stems from two factors: the use of single vector representations for complex multimodal data, and the absence of step-by-step reasoning in these embedding-based methods. To address this issue, we make an exploratory step using a novel generative method that prompts large vision-language models (e.g., GPT-4) to depict images and perform compositional reasoning. Our method outperforms other embedding-based methods on the Winoground dataset, and obtains further improvement of up to 10% accuracy when enhanced with the optimal description.

Title: MedLM: Exploring Language Models for Medical Question Answering Systems

• Authors: Niraj Yagnik, Jay Jhaveri, Vivek Sharma, Gabriel Pila, Asma Ben, Jingbo Shang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.11389
• Pdf URL: https://arxiv.org/pdf/2401.11389
• Copy Paste: [[2401.11389]] MedLM: Exploring Language Models for Medical Question Answering Systems(https://arxiv.org/abs/2401.11389)
• Keywords: generative
• Abstract: In the face of rapidly expanding online medical literature, automated systems for aggregating and summarizing information are becoming increasingly crucial for healthcare professionals and patients. Large Language Models (LLMs), with their advanced generative capabilities, have shown promise in various NLP tasks, and their potential in the healthcare domain, particularly for Closed-Book Generative QnA, is significant. However, the performance of these models in domain-specific tasks such as medical Q&A remains largely unexplored. This study aims to fill this gap by comparing the performance of general and medical-specific distilled LMs for medical Q&A. We aim to evaluate the effectiveness of fine-tuning domain-specific LMs and compare the performance of different families of Language Models. The study will address critical questions about these models' reliability, comparative performance, and effectiveness in the context of medical Q&A. The findings will provide valuable insights into the suitability of different LMs for specific applications in the medical domain.

Title: Causal Generative Explainers using Counterfactual Inference: A Case Study on the Morpho-MNIST Dataset

• Authors: Will Taylor-Melanson, Zahra Sadeghi, Stan Matwin
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.11394
• Pdf URL: https://arxiv.org/pdf/2401.11394
• Copy Paste: [[2401.11394]] Causal Generative Explainers using Counterfactual Inference: A Case Study on the Morpho-MNIST Dataset(https://arxiv.org/abs/2401.11394)
• Keywords: generative
• Abstract: In this paper, we propose leveraging causal generative learning as an interpretable tool for explaining image classifiers. Specifically, we present a generative counterfactual inference approach to study the influence of visual features (i.e., pixels) as well as causal factors through generative learning. To this end, we first uncover the most influential pixels on a classifier's decision by varying the value of a causal attribute via counterfactual inference and computing both Shapely and contrastive explanations for counterfactual images with these different attribute values. We then establish a Monte-Carlo mechanism using the generator of a causal generative model in order to adapt Shapley explainers to produce feature importances for the human-interpretable attributes of a causal dataset in the case where a classifier has been trained exclusively on the images of the dataset. Finally, we present optimization methods for creating counterfactual explanations of classifiers by means of counterfactual inference, proposing straightforward approaches for both differentiable and arbitrary classifiers. We exploit the Morpho-MNIST causal dataset as a case study for exploring our proposed methods for generating counterfacutl explantions. We employ visual explanation methods from OmnixAI open source toolkit to compare them with our proposed methods. By employing quantitative metrics to measure the interpretability of counterfactual explanations, we find that our proposed methods of counterfactual explanation offer more interpretable explanations compared to those generated from OmnixAI. This finding suggests that our methods are well-suited for generating highly interpretable counterfactual explanations on causal datasets.

Title: Enhancing the vision-language foundation model with key semantic knowledge-emphasized report refinement

• Authors: Cheng Li, Weijian Huang, Hao Yang, Jiarun Liu, Shanshan Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.11421
• Pdf URL: https://arxiv.org/pdf/2401.11421
• Copy Paste: [[2401.11421]] Enhancing the vision-language foundation model with key semantic knowledge-emphasized report refinement(https://arxiv.org/abs/2401.11421)
• Keywords: foundation model
• Abstract: Recently, vision-language representation learning has made remarkable advancements in building up medical foundation models, holding immense potential for transforming the landscape of clinical research and medical care. The underlying hypothesis is that the rich knowledge embedded in radiology reports can effectively assist and guide the learning process, reducing the need for additional labels. However, these reports tend to be complex and sometimes even consist of redundant descriptions that make the representation learning too challenging to capture the key semantic information. This paper develops a novel iterative vision-language representation learning framework by proposing a key semantic knowledge-emphasized report refinement method. Particularly, raw radiology reports are refined to highlight the key information according to a constructed clinical dictionary and two model-optimized knowledge-enhancement metrics. The iterative framework is designed to progressively learn, starting from gaining a general understanding of the patient's condition based on raw reports and gradually refines and extracts critical information essential to the fine-grained analysis tasks. The effectiveness of the proposed framework is validated on various downstream medical image analysis tasks, including disease classification, region-of-interest segmentation, and phrase grounding. Our framework surpasses seven state-of-the-art methods in both fine-tuning and zero-shot settings, demonstrating its encouraging potential for different clinical applications.

Title: Exploring Diffusion Time-steps for Unsupervised Representation Learning

• Authors: Zhongqi Yue, Jiankun Wang, Qianru Sun, Lei Ji, Eric I-Chao Chang, Hanwang Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.11430
• Pdf URL: https://arxiv.org/pdf/2401.11430
• Copy Paste: [[2401.11430]] Exploring Diffusion Time-steps for Unsupervised Representation Learning(https://arxiv.org/abs/2401.11430)
• Keywords: diffusion
• Abstract: Representation learning is all about discovering the hidden modular attributes that generate the data faithfully. We explore the potential of Denoising Diffusion Probabilistic Model (DM) in unsupervised learning of the modular attributes. We build a theoretical framework that connects the diffusion time-steps and the hidden attributes, which serves as an effective inductive bias for unsupervised learning. Specifically, the forward diffusion process incrementally adds Gaussian noise to samples at each time-step, which essentially collapses different samples into similar ones by losing attributes, e.g., fine-grained attributes such as texture are lost with less noise added (i.e., early time-steps), while coarse-grained ones such as shape are lost by adding more noise (i.e., late time-steps). To disentangle the modular attributes, at each time-step t, we learn a t-specific feature to compensate for the newly lost attribute, and the set of all 1,...,t-specific features, corresponding to the cumulative set of lost attributes, are trained to make up for the reconstruction error of a pre-trained DM at time-step t. On CelebA, FFHQ, and Bedroom datasets, the learned feature significantly improves attribute classification and enables faithful counterfactual generation, e.g., interpolating only one specified attribute between two images, validating the disentanglement quality. Codes are in https://github.com/yue-zhongqi/diti.

Title: Self-Supervised Bird's Eye View Motion Prediction with Cross-Modality Signals

• Authors: Shaoheng Fang, Zuhong Liu, Mingyu Wang, Chenxin Xu, Yiqi Zhong, Siheng Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.11499
• Pdf URL: https://arxiv.org/pdf/2401.11499
• Copy Paste: [[2401.11499]] Self-Supervised Bird's Eye View Motion Prediction with Cross-Modality Signals(https://arxiv.org/abs/2401.11499)
• Keywords: self-supervised
• Abstract: Learning the dense bird's eye view (BEV) motion flow in a self-supervised manner is an emerging research for robotics and autonomous driving. Current self-supervised methods mainly rely on point correspondences between point clouds, which may introduce the problems of fake flow and inconsistency, hindering the model's ability to learn accurate and realistic motion. In this paper, we introduce a novel cross-modality self-supervised training framework that effectively addresses these issues by leveraging multi-modality data to obtain supervision signals. We design three innovative supervision signals to preserve the inherent properties of scene motion, including the masked Chamfer distance loss, the piecewise rigidity loss, and the temporal consistency loss. Through extensive experiments, we demonstrate that our proposed self-supervised framework outperforms all previous self-supervision methods for the motion prediction task.

Title: Hierarchical Prompts for Rehearsal-free Continual Learning

• Authors: Yukun Zuo, Hantao Yao, Lu Yu, Liansheng Zhuang, Changsheng Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.11544
• Pdf URL: https://arxiv.org/pdf/2401.11544
• Copy Paste: [[2401.11544]] Hierarchical Prompts for Rehearsal-free Continual Learning(https://arxiv.org/abs/2401.11544)
• Keywords: self-supervised
• Abstract: Continual learning endeavors to equip the model with the capability to integrate current task knowledge while mitigating the forgetting of past task knowledge. Inspired by prompt tuning, prompt-based methods maintain a frozen backbone and train with slight learnable prompts to minimize the catastrophic forgetting that arises due to updating a large number of backbone parameters. Nonetheless, these learnable prompts tend to concentrate on the discriminatory knowledge of the current task while ignoring past task knowledge, leading to that learnable prompts still suffering from catastrophic forgetting. This paper introduces a novel rehearsal-free paradigm for continual learning termed Hierarchical Prompts (H-Prompts), comprising three categories of prompts -- class prompt, task prompt, and general prompt. To effectively depict the knowledge of past classes, class prompt leverages Bayesian Distribution Alignment to model the distribution of classes in each task. To reduce the forgetting of past task knowledge, task prompt employs Cross-task Knowledge Excavation to amalgamate the knowledge encapsulated in the learned class prompts of past tasks and current task knowledge. Furthermore, general prompt utilizes Generalized Knowledge Exploration to deduce highly generalized knowledge in a self-supervised manner. Evaluations on two benchmarks substantiate the efficacy of the proposed H-Prompts, exemplified by an average accuracy of 87.8% in Split CIFAR-100 and 70.6% in Split ImageNet-R.

Title: Scalable High-Resolution Pixel-Space Image Synthesis with Hourglass Diffusion Transformers

• Authors: Katherine Crowson, Stefan Andreas Baumann, Alex Birch, Tanishq Mathew Abraham, Daniel Z. Kaplan, Enrico Shippole
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.11605
• Pdf URL: https://arxiv.org/pdf/2401.11605
• Copy Paste: [[2401.11605]] Scalable High-Resolution Pixel-Space Image Synthesis with Hourglass Diffusion Transformers(https://arxiv.org/abs/2401.11605)
• Keywords: diffusion, generative
• Abstract: We present the Hourglass Diffusion Transformer (HDiT), an image generative model that exhibits linear scaling with pixel count, supporting training at high-resolution (e.g. $1024 \times 1024$) directly in pixel-space. Building on the Transformer architecture, which is known to scale to billions of parameters, it bridges the gap between the efficiency of convolutional U-Nets and the scalability of Transformers. HDiT trains successfully without typical high-resolution training techniques such as multiscale architectures, latent autoencoders or self-conditioning. We demonstrate that HDiT performs competitively with existing models on ImageNet $256^2$, and sets a new state-of-the-art for diffusion models on FFHQ-$1024^2$.

Title: In-context Learning with Retrieved Demonstrations for Language Models: A Survey

• Authors: an Luo, Xin Xu, Yue Liu, Panupong Pasupat, Mehran Kazemi
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2401.11624
• Pdf URL: https://arxiv.org/pdf/2401.11624
• Copy Paste: [[2401.11624]] In-context Learning with Retrieved Demonstrations for Language Models: A Survey(https://arxiv.org/abs/2401.11624)
• Keywords: in-context
• Abstract: Language models, especially pre-trained large language models, have showcased remarkable abilities as few-shot in-context learners (ICL), adept at adapting to new tasks with just a few demonstrations in the input context. However, the model's ability to perform ICL is sensitive to the choice of the few-shot demonstrations. Instead of using a fixed set of demonstrations, one recent development is to retrieve demonstrations tailored to each input query. The implementation of demonstration retrieval is relatively straightforward, leveraging existing databases and retrieval systems. This not only improves the efficiency and scalability of the learning process but also has been shown to reduce biases inherent in manual example selection. In light of the encouraging results and growing research in ICL with retrieved demonstrations, we conduct an extensive review of studies in this area. In this survey, we discuss and compare different design choices for retrieval models, retrieval training procedures, and inference algorithms.

Title: Text-to-Image Cross-Modal Generation: A Systematic Review

• Authors: Maciej Żelaszczyk, Jacek Mańdziuk
• Subjects: cs.CV, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.11631
• Pdf URL: https://arxiv.org/pdf/2401.11631
• Copy Paste: [[2401.11631]] Text-to-Image Cross-Modal Generation: A Systematic Review(https://arxiv.org/abs/2401.11631)
• Keywords: self-supervised
• Abstract: We review research on generating visual data from text from the angle of "cross-modal generation." This point of view allows us to draw parallels between various methods geared towards working on input text and producing visual output, without limiting the analysis to narrow sub-areas. It also results in the identification of common templates in the field, which are then compared and contrasted both within pools of similar methods and across lines of research. We provide a breakdown of text-to-image generation into various flavors of image-from-text methods, video-from-text methods, image editing, self-supervised and graph-based approaches. In this discussion, we focus on research papers published at 8 leading machine learning conferences in the years 2016-2022, also incorporating a number of relevant papers not matching the outlined search criteria. The conducted review suggests a significant increase in the number of papers published in the area and highlights research gaps and potential lines of investigation. To our knowledge, this is the first review to systematically look at text-to-image generation from the perspective of "cross-modal generation."

Title: LW-FedSSL: Resource-efficient Layer-wise Federated Self-supervised Learning

• Authors: Ye Lin Tun, Chu Myaet Thwal, Le Quang Huy, Minh N. H. Nguyen, Choong Seon Hong
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.11647
• Pdf URL: https://arxiv.org/pdf/2401.11647
• Copy Paste: [[2401.11647]] LW-FedSSL: Resource-efficient Layer-wise Federated Self-supervised Learning(https://arxiv.org/abs/2401.11647)
• Keywords: self-supervised
• Abstract: Many recent studies integrate federated learning (FL) with self-supervised learning (SSL) to take advantage of raw training data distributed across edge devices. However, edge devices often struggle with high computation and communication costs imposed by SSL and FL algorithms. To tackle this hindrance, we propose LW-FedSSL, a layer-wise federated self-supervised learning approach that allows edge devices to incrementally train one layer of the model at a time. LW-FedSSL comprises server-side calibration and representation alignment mechanisms to maintain comparable performance with end-to-end FedSSL while significantly lowering clients' resource requirements. The server-side calibration mechanism takes advantage of the resource-rich server in an FL environment to assist in global model training. Meanwhile, the representation alignment mechanism encourages closeness between representations of FL local models and those of the global model. Our experiments show that LW-FedSSL has a $3.3 \times$ lower memory requirement and a $3.2 \times$ cheaper communication cost than its end-to-end counterpart. We also explore a progressive training strategy called Prog-FedSSL that outperforms end-to-end training with a similar memory requirement and a $1.8 \times$ cheaper communication cost.

Title: Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs

• Authors: Ling Yang, Zhaochen Yu, Chenlin Meng, Minkai Xu, Stefano Ermon, Bin Cui
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.11708
• Pdf URL: https://arxiv.org/pdf/2401.11708
• Copy Paste: [[2401.11708]] Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs(https://arxiv.org/abs/2401.11708)
• Keywords: diffusion
• Abstract: Diffusion models have exhibit exceptional performance in text-to-image generation and editing. However, existing methods often face challenges when handling complex text prompts that involve multiple objects with multiple attributes and relationships. In this paper, we propose a brand new training-free text-to-image generation/editing framework, namely Recaption, Plan and Generate (RPG), harnessing the powerful chain-of-thought reasoning ability of multimodal LLMs to enhance the compositionality of text-to-image diffusion models. Our approach employs the MLLM as a global planner to decompose the process of generating complex images into multiple simpler generation tasks within subregions. We propose complementary regional diffusion to enable region-wise compositional generation. Furthermore, we integrate text-guided image generation and editing within the proposed RPG in a closed-loop fashion, thereby enhancing generalization ability. Extensive experiments demonstrate our RPG outperforms state-of-the-art text-to-image diffusion models, including DALL-E 3 and SDXL, particularly in multi-category object composition and text-image semantic alignment. Notably, our RPG framework exhibits wide compatibility with various MLLM architectures (e.g., MiniGPT-4) and diffusion backbones (e.g., ControlNet). Our code is available at: https://github.com/YangLing0818/RPG-DiffusionMaster

Title: Unraveling Attacks in Machine Learning-based IoT Ecosystems: A Survey and the Open Libraries Behind Them

• Authors: Chao Liu, Boxi Chen, Wei Shao, Chris Zhang, Kelvin Wong, Yi Zhang
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.11723
• Pdf URL: https://arxiv.org/pdf/2401.11723
• Copy Paste: [[2401.11723]] Unraveling Attacks in Machine Learning-based IoT Ecosystems: A Survey and the Open Libraries Behind Them(https://arxiv.org/abs/2401.11723)
• Keywords: anomaly
• Abstract: The advent of the Internet of Things (IoT) has brought forth an era of unprecedented connectivity, with an estimated 80 billion smart devices expected to be in operation by the end of 2025. These devices facilitate a multitude of smart applications, enhancing the quality of life and efficiency across various domains. Machine Learning (ML) serves as a crucial technology, not only for analyzing IoT-generated data but also for diverse applications within the IoT ecosystem. For instance, ML finds utility in IoT device recognition, anomaly detection, and even in uncovering malicious activities. This paper embarks on a comprehensive exploration of the security threats arising from ML's integration into various facets of IoT, spanning various attack types including membership inference, adversarial evasion, reconstruction, property inference, model extraction, and poisoning attacks. Unlike previous studies, our work offers a holistic perspective, categorizing threats based on criteria such as adversary models, attack targets, and key security attributes (confidentiality, availability, and integrity). We delve into the underlying techniques of ML attacks in IoT environment, providing a critical evaluation of their mechanisms and impacts. Furthermore, our research thoroughly assesses 65 libraries, both author-contributed and third-party, evaluating their role in safeguarding model and data privacy. We emphasize the availability and usability of these libraries, aiming to arm the community with the necessary tools to bolster their defenses against the evolving threat landscape. Through our comprehensive review and analysis, this paper seeks to contribute to the ongoing discourse on ML-based IoT security, offering valuable insights and practical solutions to secure ML models and data in the rapidly expanding field of artificial intelligence in IoT.

Title: EmerDiff: Emerging Pixel-level Semantic Knowledge in Diffusion Models

• Authors: Koichi Namekata, Amirmojtaba Sabour, Sanja Fidler, Seung Wook Kim
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.11739
• Pdf URL: https://arxiv.org/pdf/2401.11739
• Copy Paste: [[2401.11739]] EmerDiff: Emerging Pixel-level Semantic Knowledge in Diffusion Models(https://arxiv.org/abs/2401.11739)
• Keywords: diffusion
• Abstract: Diffusion models have recently received increasing research attention for their remarkable transfer abilities in semantic segmentation tasks. However, generating fine-grained segmentation masks with diffusion models often requires additional training on annotated datasets, leaving it unclear to what extent pre-trained diffusion models alone understand the semantic relations of their generated images. To address this question, we leverage the semantic knowledge extracted from Stable Diffusion (SD) and aim to develop an image segmentor capable of generating fine-grained segmentation maps without any additional training. The primary difficulty stems from the fact that semantically meaningful feature maps typically exist only in the spatially lower-dimensional layers, which poses a challenge in directly extracting pixel-level semantic relations from these feature maps. To overcome this issue, our framework identifies semantic correspondences between image pixels and spatial locations of low-dimensional feature maps by exploiting SD's generation process and utilizes them for constructing image-resolution segmentation maps. In extensive experiments, the produced segmentation maps are demonstrated to be well delineated and capture detailed parts of the images, indicating the existence of highly accurate pixel-level semantic knowledge in diffusion models.

Title: GI-PIP: Do We Require Impractical Auxiliary Dataset for Gradient Inversion Attacks?

• Authors: Yu sun, Gaojian Xiong, Xianxun Yao, Kailang Ma, Jian Cui
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.11748
• Pdf URL: https://arxiv.org/pdf/2401.11748
• Copy Paste: [[2401.11748]] GI-PIP: Do We Require Impractical Auxiliary Dataset for Gradient Inversion Attacks?(https://arxiv.org/abs/2401.11748)
• Keywords: anomaly
• Abstract: Deep gradient inversion attacks expose a serious threat to Federated Learning (FL) by accurately recovering private data from shared gradients. However, the state-of-the-art heavily relies on impractical assumptions to access excessive auxiliary data, which violates the basic data partitioning principle of FL. In this paper, a novel method, Gradient Inversion Attack using Practical Image Prior (GI-PIP), is proposed under a revised threat model. GI-PIP exploits anomaly detection models to capture the underlying distribution from fewer data, while GAN-based methods consume significant more data to synthesize images. The extracted distribution is then leveraged to regulate the attack process as Anomaly Score loss. Experimental results show that GI-PIP achieves a 16.12 dB PSNR recovery using only 3.8\% data of ImageNet, while GAN-based methods necessitate over 70\%. Moreover, GI-PIP exhibits superior capability on distribution generalization compared to GAN-based methods. Our approach significantly alleviates the auxiliary data requirement on both amount and distribution in gradient inversion attacks, hence posing more substantial threat to real-world FL.

Title: Symbrain: A large-scale dataset of MRI images for neonatal brain symmetry analysis

• Authors: Arnaud Gucciardi, Safouane El Ghazouali, Francesca Venturini, Vida Groznik, Umberto Michelucci
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.11814
• Pdf URL: https://arxiv.org/pdf/2401.11814
• Copy Paste: [[2401.11814]] Symbrain: A large-scale dataset of MRI images for neonatal brain symmetry analysis(https://arxiv.org/abs/2401.11814)
• Keywords: anomaly
• Abstract: This paper presents an annotated dataset of brain MRI images designed to advance the field of brain symmetry study. Magnetic resonance imaging (MRI) has gained interest in analyzing brain symmetry in neonatal infants, and challenges remain due to the vast size differences between fetal and adult brains. Classification methods for brain structural MRI use scales and visual cues to assess hemisphere symmetry, which can help diagnose neonatal patients by comparing hemispheres and anatomical regions of interest in the brain. Using the Developing Human Connectome Project dataset, this work presents a dataset comprising cerebral images extracted as slices across selected portions of interest for clinical evaluation . All the extracted images are annotated with the brain's midline. All the extracted images are annotated with the brain's midline. From the assumption that a decrease in symmetry is directly related to possible clinical pathologies, the dataset can contribute to a more precise diagnosis because it can be used to train deep learning model application in neonatal cerebral MRI anomaly detection from postnatal infant scans thanks to computer vision. Such models learn to identify and classify anomalies by identifying potential asymmetrical patterns in medical MRI images. Furthermore, this dataset can contribute to the research and development of methods using the relative symmetry of the two brain hemispheres for crucial diagnosis and treatment planning.

Title: Learning to Approximate Adaptive Kernel Convolution on Graphs

• Authors: Jaeyoon Sim, Sooyeon Jeon, InJun Choi, Guorong Wu, Won Hwa Kim
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.11840
• Pdf URL: https://arxiv.org/pdf/2401.11840
• Copy Paste: [[2401.11840]] Learning to Approximate Adaptive Kernel Convolution on Graphs(https://arxiv.org/abs/2401.11840)
• Keywords: diffusion
• Abstract: Various Graph Neural Networks (GNNs) have been successful in analyzing data in non-Euclidean spaces, however, they have limitations such as oversmoothing, i.e., information becomes excessively averaged as the number of hidden layers increases. The issue stems from the intrinsic formulation of conventional graph convolution where the nodal features are aggregated from a direct neighborhood per layer across the entire nodes in the graph. As setting different number of hidden layers per node is infeasible, recent works leverage a diffusion kernel to redefine the graph structure and incorporate information from farther nodes. Unfortunately, such approaches suffer from heavy diagonalization of a graph Laplacian or learning a large transform matrix. In this regards, we propose a diffusion learning framework, where the range of feature aggregation is controlled by the scale of a diffusion kernel. For efficient computation, we derive closed-form derivatives of approximations of the graph convolution with respect to the scale, so that node-wise range can be adaptively learned. With a downstream classifier, the entire framework is made trainable in an end-to-end manner. Our model is tested on various standard datasets for node-wise classification for the state-of-the-art performance, and it is also validated on a real-world brain network data for graph classifications to demonstrate its practicality for Alzheimer classification.

Title: Self-Labeling the Job Shop Scheduling Problem

• Authors: Andrea Corsini, Angelo Porrello, Simone Calderara, Mauro Dell'Amico
• Subjects: cs.LG, cs.AI, math.CO
• Abstract URL: https://arxiv.org/abs/2401.11849
• Pdf URL: https://arxiv.org/pdf/2401.11849
• Copy Paste: [[2401.11849]] Self-Labeling the Job Shop Scheduling Problem(https://arxiv.org/abs/2401.11849)
• Keywords: self-supervised, generative
• Abstract: In this work, we propose a Self-Supervised training strategy specifically designed for combinatorial problems. One of the main obstacles in applying supervised paradigms to such problems is the requirement of expensive target solutions as ground-truth, often produced with costly exact solvers. Inspired by Semi- and Self-Supervised learning, we show that it is possible to easily train generative models by sampling multiple solutions and using the best one according to the problem objective as a pseudo-label. In this way, we iteratively improve the model generation capability by relying only on its self-supervision, completely removing the need for optimality information. We prove the effectiveness of this Self-Labeling strategy on the Job Shop Scheduling (JSP), a complex combinatorial problem that is receiving much attention from the Reinforcement Learning community. We propose a generative model based on the well-known Pointer Network and train it with our strategy. Experiments on two popular benchmarks demonstrate the potential of this approach as the resulting models outperform constructive heuristics and current state-of-the-art Reinforcement Learning proposals.

Title: The Right Model for the Job: An Evaluation of Legal Multi-Label Classification Baselines

• Authors: Martina Forster, Claudia Schulz, Prudhvi Nokku, Melicaalsadat Mirsafian, Jaykumar Kasundra, Stavroula Skylaki
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.11852
• Pdf URL: https://arxiv.org/pdf/2401.11852
• Copy Paste: [[2401.11852]] The Right Model for the Job: An Evaluation of Legal Multi-Label Classification Baselines(https://arxiv.org/abs/2401.11852)
• Keywords: generative
• Abstract: Multi-Label Classification (MLC) is a common task in the legal domain, where more than one label may be assigned to a legal document. A wide range of methods can be applied, ranging from traditional ML approaches to the latest Transformer-based architectures. In this work, we perform an evaluation of different MLC methods using two public legal datasets, POSTURE50K and EURLEX57K. By varying the amount of training data and the number of labels, we explore the comparative advantage offered by different approaches in relation to the dataset properties. Our findings highlight DistilRoBERTa and LegalBERT as performing consistently well in legal MLC with reasonable computational demands. T5 also demonstrates comparable performance while offering advantages as a generative model in the presence of changing label sets. Finally, we show that the CrossEncoder exhibits potential for notable macro-F1 score improvements, albeit with increased computational costs.

Title: A Review of Physics-Informed Machine Learning Methods with Applications to Condition Monitoring and Anomaly Detection

• Authors: Yuandi Wu, Brett Sicard, Stephen Andrew Gadsden
• Subjects: cs.LG, cs.AI, eess.SY
• Abstract URL: https://arxiv.org/abs/2401.11860
• Pdf URL: https://arxiv.org/pdf/2401.11860
• Copy Paste: [[2401.11860]] A Review of Physics-Informed Machine Learning Methods with Applications to Condition Monitoring and Anomaly Detection(https://arxiv.org/abs/2401.11860)
• Keywords: anomaly
• Abstract: This study presents a comprehensive overview of PIML techniques in the context of condition monitoring. The central concept driving PIML is the incorporation of known physical laws and constraints into machine learning algorithms, enabling them to learn from available data while remaining consistent with physical principles. Through fusing domain knowledge with data-driven learning, PIML methods offer enhanced accuracy and interpretability in comparison to purely data-driven approaches. In this comprehensive survey, detailed examinations are performed with regard to the methodology by which known physical principles are integrated within machine learning frameworks, as well as their suitability for specific tasks within condition monitoring. Incorporation of physical knowledge into the ML model may be realized in a variety of methods, with each having its unique advantages and drawbacks. The distinct advantages and limitations of each methodology for the integration of physics within data-driven models are detailed, considering factors such as computational efficiency, model interpretability, and generalizability to different systems in condition monitoring and fault detection. Several case studies and works of literature utilizing this emerging concept are presented to demonstrate the efficacy of PIML in condition monitoring applications. From the literature reviewed, the versatility and potential of PIML in condition monitoring may be demonstrated. Novel PIML methods offer an innovative solution for addressing the complexities of condition monitoring and associated challenges. This comprehensive survey helps form the foundation for future work in the field. As the technology continues to advance, PIML is expected to play a crucial role in enhancing maintenance strategies, system reliability, and overall operational efficiency in engineering systems.

Title: Feature Denoising Diffusion Model for Blind Image Quality Assessment

• Authors: Xudong Li, Jingyuan Zheng, Runze Hu, Yan Zhang, Ke Li, Yunhang Shen, Xiawu Zheng, Yutao Liu, ShengChuan Zhang, Pingyang Dai, Rongrong Ji
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.11949
• Pdf URL: https://arxiv.org/pdf/2401.11949
• Copy Paste: [[2401.11949]] Feature Denoising Diffusion Model for Blind Image Quality Assessment(https://arxiv.org/abs/2401.11949)
• Keywords: diffusion
• Abstract: Blind Image Quality Assessment (BIQA) aims to evaluate image quality in line with human perception, without reference benchmarks. Currently, deep learning BIQA methods typically depend on using features from high-level tasks for transfer learning. However, the inherent differences between BIQA and these high-level tasks inevitably introduce noise into the quality-aware features. In this paper, we take an initial step towards exploring the diffusion model for feature denoising in BIQA, namely Perceptual Feature Diffusion for IQA (PFD-IQA), which aims to remove noise from quality-aware features. Specifically, (i) We propose a {Perceptual Prior Discovery and Aggregation module to establish two auxiliary tasks to discover potential low-level features in images that are used to aggregate perceptual text conditions for the diffusion model. (ii) We propose a Perceptual Prior-based Feature Refinement strategy, which matches noisy features to predefined denoising trajectories and then performs exact feature denoising based on text conditions. Extensive experiments on eight standard BIQA datasets demonstrate the superior performance to the state-of-the-art BIQA methods, i.e., achieving the PLCC values of 0.935 ( vs. 0.905 in KADID) and 0.922 ( vs. 0.894 in LIVEC).

Title: Claim Detection for Automated Fact-checking: A Survey on Monolingual, Multilingual and Cross-Lingual Research

• Authors: Rrubaa Panchendrarajan, Arkaitz Zubiaga
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.11969
• Pdf URL: https://arxiv.org/pdf/2401.11969
• Copy Paste: [[2401.11969]] Claim Detection for Automated Fact-checking: A Survey on Monolingual, Multilingual and Cross-Lingual Research(https://arxiv.org/abs/2401.11969)
• Keywords: diffusion
• Abstract: Automated fact-checking has drawn considerable attention over the past few decades due to the increase in the diffusion of misinformation on online platforms. This is often carried out as a sequence of tasks comprising (i) the detection of sentences circulating in online platforms which constitute claims needing verification, followed by (ii) the verification process of those claims. This survey focuses on the former, by discussing existing efforts towards detecting claims needing fact-checking, with a particular focus on multilingual data and methods. This is a challenging and fertile direction where existing methods are yet far from matching human performance due to the profoundly challenging nature of the issue. Especially, the dissemination of information across multiple social platforms, articulated in multiple languages and modalities demands more generalized solutions for combating misinformation. Focusing on multilingual misinformation, we present a comprehensive survey of existing multilingual claim detection research. We present state-of-the-art multilingual claim detection research categorized into three key factors of the problem, verifiability, priority, and similarity. Further, we present a detailed overview of the existing multilingual datasets along with the challenges and suggest possible future advancements.

Title: Stereo-Matching Knowledge Distilled Monocular Depth Estimation Filtered by Multiple Disparity Consistency

• Authors: Woonghyun Ka, Jae Young Lee, Jaehyun Choi, Junmo Kim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.12019
• Pdf URL: https://arxiv.org/pdf/2401.12019
• Copy Paste: [[2401.12019]] Stereo-Matching Knowledge Distilled Monocular Depth Estimation Filtered by Multiple Disparity Consistency(https://arxiv.org/abs/2401.12019)
• Keywords: self-supervised
• Abstract: In stereo-matching knowledge distillation methods of the self-supervised monocular depth estimation, the stereo-matching network's knowledge is distilled into a monocular depth network through pseudo-depth maps. In these methods, the learning-based stereo-confidence network is generally utilized to identify errors in the pseudo-depth maps to prevent transferring the errors. However, the learning-based stereo-confidence networks should be trained with ground truth (GT), which is not feasible in a self-supervised setting. In this paper, we propose a method to identify and filter errors in the pseudo-depth map using multiple disparity maps by checking their consistency without the need for GT and a training process. Experimental results show that the proposed method outperforms the previous methods and works well on various configurations by filtering out erroneous areas where the stereo-matching is vulnerable, especially such as textureless regions, occlusion boundaries, and reflective surfaces.

Title: SEDAC: A CVAE-Based Data Augmentation Method for Security Bug Report Identification

• Authors: Y. Liao, T. Zhang
• Subjects: cs.CR, cs.SE
• Abstract URL: https://arxiv.org/abs/2401.12060
• Pdf URL: https://arxiv.org/pdf/2401.12060
• Copy Paste: [[2401.12060]] SEDAC: A CVAE-Based Data Augmentation Method for Security Bug Report Identification(https://arxiv.org/abs/2401.12060)
• Keywords: generative
• Abstract: Bug tracking systems store many bug reports, some of which are related to security. Identifying those security bug reports (SBRs) may help us predict some security-related bugs and solve security issues promptly so that the project can avoid threats and attacks. However, in the real world, the ratio of security bug reports is severely low; thus, directly training a prediction model with raw data may result in inaccurate results. Faced with the massive challenge of data imbalance, many researchers in the past have attempted to use text filtering or clustering methods to minimize the proportion of non-security bug reports (NSBRs) or apply oversampling methods to synthesize SBRs to make the dataset as balanced as possible. Nevertheless, there are still two challenges to those methods: 1) They ignore long-distance contextual information. 2) They fail to generate an utterly balanced dataset. To tackle these two challenges, we propose SEDAC, a new SBR identification method that generates similar bug report vectors to solve data imbalance problems and accurately detect security bug reports. Unlike previous studies, it first converts bug reports into individual bug report vectors with distilBERT, which are based on word2vec. Then, it trains a generative model through conditional variational auto-encoder (CVAE) to generate similar vectors with security labels, which makes the number of SBRs equal to NSBRs'. Finally, balanced data are used to train a security bug report classifier. To evaluate the effectiveness of our framework, we conduct it on 45,940 bug reports from Chromium and four Apache projects. The experimental results show that SEDAC outperforms all the baselines in g-measure with improvements of around 14.24%-50.10%.

Title: Revisiting Demonstration Selection Strategies in In-Context Learning

• Authors: Keqin Peng, Liang Ding, Yancheng Yuan, Xuebo Liu, Min Zhang, Yuanxin Ouyang, Dacheng Tao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.12087
• Pdf URL: https://arxiv.org/pdf/2401.12087
• Copy Paste: [[2401.12087]] Revisiting Demonstration Selection Strategies in In-Context Learning(https://arxiv.org/abs/2401.12087)
• Keywords: in-context
• Abstract: Large language models (LLMs) have shown an impressive ability to perform a wide range of tasks using in-context learning (ICL), where a few examples are used to describe a task to the model. However, the performance of ICL varies significantly with the choice of demonstrations, and it is still unclear why this happens or what factors will influence its choice. In this work, we first revisit the factors contributing to this variance from both data and model aspects, and find that the choice of demonstration is both data- and model-dependent. We further proposed a data- and model-dependent demonstration selection method, \textbf{TopK + ConE}, based on the assumption that \textit{the performance of a demonstration positively correlates with its contribution to the model's understanding of the test samples}, resulting in a simple and effective recipe for ICL. Empirically, our method yields consistent improvements in both language understanding and generation tasks with different model scales. Further analyses confirm that, besides the generality and stability under different circumstances, our method provides a unified explanation for the effectiveness of previous methods. Code will be released.

Title: An Empirical Analysis of In-context Learning Abilities of LLMs for MT

• Authors: Pranjal A. Chitale, Jay Gala, Varun Gumma, Mitesh M. Khapra, Raj Dabre
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.12097
• Pdf URL: https://arxiv.org/pdf/2401.12097
• Copy Paste: [[2401.12097]] An Empirical Analysis of In-context Learning Abilities of LLMs for MT(https://arxiv.org/abs/2401.12097)
• Keywords: in-context
• Abstract: In-context learning (ICL) has consistently demonstrated superior performance over zero-shot performance in large language models (LLMs). However, the understanding of the dynamics of ICL and the aspects that influence downstream performance remains limited, especially for natural language generation (NLG) tasks. This work aims to address this gap by investigating the ICL capabilities of LLMs and studying the impact of different aspects of the in-context demonstrations for the task of machine translation (MT). Our preliminary investigations aim to discern whether in-context learning (ICL) is predominantly influenced by demonstrations or instructions by applying diverse perturbations to in-context demonstrations while preserving the task instruction. We observe varying behavior to perturbed examples across different model families, notably with BLOOM-7B derivatives being severely influenced by noise, whereas Llama 2 derivatives not only exhibit robustness but also tend to show enhancements over the clean baseline when subject to perturbed demonstrations. This suggests that the robustness of ICL may be governed by several factors, including the type of noise, perturbation direction (source or target), the extent of pretraining of the specific model, and fine-tuning for downstream tasks if applicable. Further investigation is warranted to develop a comprehensive understanding of these factors in future research.

Title: The Curious Case of Nonverbal Abstract Reasoning with Multi-Modal Large Language Models

• Authors: Kian Ahrabian, Zhivar Sourati, Kexuan Sun, Jiarui Zhang, Yifan Jiang, Fred Morstatter, Jay Pujara
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.12117
• Pdf URL: https://arxiv.org/pdf/2401.12117
• Copy Paste: [[2401.12117]] The Curious Case of Nonverbal Abstract Reasoning with Multi-Modal Large Language Models(https://arxiv.org/abs/2401.12117)
• Keywords: foundation model
• Abstract: While large language models (LLMs) are still being adopted to new domains and utilized in novel applications, we are experiencing an influx of the new generation of foundation models, namely multi-modal large language models (MLLMs). These models integrate verbal and visual information, opening new possibilities to demonstrate more complex reasoning abilities at the intersection of the two modalities. However, despite the revolutionizing prospect of MLLMs, our understanding of their reasoning abilities is limited. In this study, we assess the nonverbal abstract reasoning abilities of open-source and closed-source MLLMs using variations of Raven's Progressive Matrices. Our experiments expose the difficulty of solving such problems while showcasing the immense gap between open-source and closed-source models. We also reveal critical shortcomings with individual visual and textual modules, subjecting the models to low-performance ceilings. Finally, to improve MLLMs' performance, we experiment with various methods, such as Chain-of-Thought prompting, resulting in a significant (up to 100%) boost in performance.

Title: Anisotropy Is Inherent to Self-Attention in Transformers

• Authors: Nathan Godey, Éric de la Clergerie, Benoît Sagot
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.12143
• Pdf URL: https://arxiv.org/pdf/2401.12143
• Copy Paste: [[2401.12143]] Anisotropy Is Inherent to Self-Attention in Transformers(https://arxiv.org/abs/2401.12143)
• Keywords: self-supervised
• Abstract: The representation degeneration problem is a phenomenon that is widely observed among self-supervised learning methods based on Transformers. In NLP, it takes the form of anisotropy, a singular property of hidden representations which makes them unexpectedly close to each other in terms of angular distance (cosine-similarity). Some recent works tend to show that anisotropy is a consequence of optimizing the cross-entropy loss on long-tailed distributions of tokens. We show in this paper that anisotropy can also be observed empirically in language models with specific objectives that should not suffer directly from the same consequences. We also show that the anisotropy problem extends to Transformers trained on other modalities. Our observations suggest that anisotropy is actually inherent to Transformers-based models.

Title: Single-View 3D Human Digitalization with Large Reconstruction Models

• Authors: Zhenzhen Weng, Jingyuan Liu, Hao Tan, Zhan Xu, Yang Zhou, Serena Yeung-Levy, Jimei Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.12175
• Pdf URL: https://arxiv.org/pdf/2401.12175
• Copy Paste: [[2401.12175]] Single-View 3D Human Digitalization with Large Reconstruction Models(https://arxiv.org/abs/2401.12175)
• Keywords: diffusion, generative
• Abstract: In this paper, we introduce Human-LRM, a single-stage feed-forward Large Reconstruction Model designed to predict human Neural Radiance Fields (NeRF) from a single image. Our approach demonstrates remarkable adaptability in training using extensive datasets containing 3D scans and multi-view capture. Furthermore, to enhance the model's applicability for in-the-wild scenarios especially with occlusions, we propose a novel strategy that distills multi-view reconstruction into single-view via a conditional triplane diffusion model. This generative extension addresses the inherent variations in human body shapes when observed from a single view, and makes it possible to reconstruct the full body human from an occluded image. Through extensive experiments, we show that Human-LRM surpasses previous methods by a significant margin on several benchmarks.

Title: Broiler-Net: A Deep Convolutional Framework for Broiler Behavior Analysis in Poultry Houses

• Authors: Tahereh Zarrat Ehsan, Seyed Mehdi Mohtavipour
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.12176
• Pdf URL: https://arxiv.org/pdf/2401.12176
• Copy Paste: [[2401.12176]] Broiler-Net: A Deep Convolutional Framework for Broiler Behavior Analysis in Poultry Houses(https://arxiv.org/abs/2401.12176)
• Keywords: anomaly
• Abstract: Detecting anomalies in poultry houses is crucial for maintaining optimal chicken health conditions, minimizing economic losses and bolstering profitability. This paper presents a novel real-time framework for analyzing chicken behavior in cage-free poultry houses to detect abnormal behaviors. Specifically, two significant abnormalities, namely inactive broiler and huddling behavior, are investigated in this study. The proposed framework comprises three key steps: (1) chicken detection utilizing a state-of-the-art deep learning model, (2) tracking individual chickens across consecutive frames with a fast tracker module, and (3) detecting abnormal behaviors within the video stream. Experimental studies are conducted to evaluate the efficacy of the proposed algorithm in accurately assessing chicken behavior. The results illustrate that our framework provides a precise and efficient solution for real-time anomaly detection, facilitating timely interventions to maintain chicken health and enhance overall productivity on poultry farms. Github: https://github.com/TaherehZarratEhsan/Chicken-Behavior-Analysis

Title: In-Context Learning for Extreme Multi-Label Classification

• Authors: Karel D'Oosterlinck, Omar Khattab, François Remy, Thomas Demeester, Chris Develder, Christopher Potts
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.12178
• Pdf URL: https://arxiv.org/pdf/2401.12178
• Copy Paste: [[2401.12178]] In-Context Learning for Extreme Multi-Label Classification(https://arxiv.org/abs/2401.12178)
• Keywords: in-context
• Abstract: Multi-label classification problems with thousands of classes are hard to solve with in-context learning alone, as language models (LMs) might lack prior knowledge about the precise classes or how to assign them, and it is generally infeasible to demonstrate every class in a prompt. We propose a general program, $\texttt{Infer--Retrieve--Rank}$, that defines multi-step interactions between LMs and retrievers to efficiently tackle such problems. We implement this program using the $\texttt{DSPy}$ programming model, which specifies in-context systems in a declarative manner, and use $\texttt{DSPy}$ optimizers to tune it towards specific datasets by bootstrapping only tens of few-shot examples. Our primary extreme classification program, optimized separately for each task, attains state-of-the-art results across three benchmarks (HOUSE, TECH, TECHWOLF). We apply the same program to a benchmark with vastly different characteristics and attain competitive performance as well (BioDEX). Unlike prior work, our proposed solution requires no finetuning, is easily applicable to new tasks, alleviates prompt engineering, and requires only tens of labeled examples. Our code is public at https://github.com/KarelDO/xmc.dspy.

Title: LONEStar: The Lunar Flashlight Optical Navigation Experiment

• Authors: Michael Krause, Ava Thrasher, Priyal Soni, Liam Smego, Reuben Isaac, Jennifer Nolan, Micah Pledger, E. Glenn Lightsey, W. Jud Ready, John Christian
• Subjects: cs.CV, astro-ph.IM, physics.space-ph
• Abstract URL: https://arxiv.org/abs/2401.12198
• Pdf URL: https://arxiv.org/pdf/2401.12198
• Copy Paste: [[2401.12198]] LONEStar: The Lunar Flashlight Optical Navigation Experiment(https://arxiv.org/abs/2401.12198)
• Keywords: anomaly
• Abstract: This paper documents the results from the highly successful Lunar flashlight Optical Navigation Experiment with a Star tracker (LONEStar). Launched in December 2022, Lunar Flashlight (LF) was a NASA-funded technology demonstration mission. After a propulsion system anomaly prevented capture in lunar orbit, LF was ejected from the Earth-Moon system and into heliocentric space. NASA subsequently transferred ownership of LF to Georgia Tech to conduct an unfunded extended mission to demonstrate further advanced technology objectives, including LONEStar. From August-December 2023, the LONEStar team performed on-orbit calibration of the optical instrument and a number of different OPNAV experiments. This campaign included the processing of nearly 400 images of star fields, Earth and Moon, and four other planets (Mercury, Mars, Jupiter, and Saturn). LONEStar provided the first on-orbit demonstrations of heliocentric navigation using only optical observations of planets. Of special note is the successful in-flight demonstration of (1) instantaneous triangulation with simultaneous sightings of two planets with the LOST algorithm and (2) dynamic triangulation with sequential sightings of multiple planets.

Title: CheXagent: Towards a Foundation Model for Chest X-Ray Interpretation

• Authors: Zhihong Chen, Maya Varma, Jean-Benoit Delbrouck, Magdalini Paschali, Louis Blankemeier, Dave Van Veen, Jeya Maria Jose Valanarasu, Alaa Youssef, Joseph Paul Cohen, Eduardo Pontes Reis, Emily B. Tsai, Andrew Johnston, Cameron Olsen, Tanishq Mathew Abraham, Sergios Gatidis, Akshay S. Chaudhari, Curtis Langlotz
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2401.12208
• Pdf URL: https://arxiv.org/pdf/2401.12208
• Copy Paste: [[2401.12208]] CheXagent: Towards a Foundation Model for Chest X-Ray Interpretation(https://arxiv.org/abs/2401.12208)
• Keywords: foundation model
• Abstract: Chest X-rays (CXRs) are the most frequently performed imaging test in clinical practice. Recent advances in the development of vision-language foundation models (FMs) give rise to the possibility of performing automated CXR interpretation, which can assist physicians with clinical decision-making and improve patient outcomes. However, developing FMs that can accurately interpret CXRs is challenging due to the (1) limited availability of large-scale vision-language datasets in the medical image domain, (2) lack of vision and language encoders that can capture the complexities of medical data, and (3) absence of evaluation frameworks for benchmarking the abilities of FMs on CXR interpretation. In this work, we address these challenges by first introducing \emph{CheXinstruct} - a large-scale instruction-tuning dataset curated from 28 publicly-available datasets. We then present \emph{CheXagent} - an instruction-tuned FM capable of analyzing and summarizing CXRs. To build CheXagent, we design a clinical large language model (LLM) for parsing radiology reports, a vision encoder for representing CXR images, and a network to bridge the vision and language modalities. Finally, we introduce \emph{CheXbench} - a novel benchmark designed to systematically evaluate FMs across 8 clinically-relevant CXR interpretation tasks. Extensive quantitative evaluations and qualitative reviews with five expert radiologists demonstrate that CheXagent outperforms previously-developed general- and medical-domain FMs on CheXbench tasks. Furthermore, in an effort to improve model transparency, we perform a fairness evaluation across factors of sex, race and age to highlight potential performance disparities. Our project is at \url{https://stanford-aimi.github.io/chexagent.html}.

Title: Less Could Be Better: Parameter-efficient Fine-tuning Advances Medical Vision Foundation Models

• Authors: Chenyu Lian, Hong-Yu Zhou, Yizhou Yu, Liansheng Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.12215
• Pdf URL: https://arxiv.org/pdf/2401.12215
• Copy Paste: [[2401.12215]] Less Could Be Better: Parameter-efficient Fine-tuning Advances Medical Vision Foundation Models(https://arxiv.org/abs/2401.12215)
• Keywords: self-supervised, foundation model
• Abstract: Parameter-efficient fine-tuning (PEFT) that was initially developed for exploiting pre-trained large language models has recently emerged as an effective approach to perform transfer learning on computer vision tasks. However, the effectiveness of PEFT on medical vision foundation models is still unclear and remains to be explored. As a proof of concept, we conducted a detailed empirical study on applying PEFT to chest radiography foundation models. Specifically, we delved into LoRA, a representative PEFT method, and compared it against full-parameter fine-tuning (FFT) on two self-supervised radiography foundation models across three well-established chest radiograph datasets. Our results showed that LoRA outperformed FFT in 13 out of 18 transfer learning tasks by at most 2.9% using fewer than 1% tunable parameters. Combining LoRA with foundation models, we set up new state-of-the-art on a range of data-efficient learning tasks, such as an AUROC score of 80.6% using 1% labeled data on NIH ChestX-ray14. We hope this study can evoke more attention from the community in the use of PEFT for transfer learning on medical imaging tasks. Code and models are available at https://github.com/RL4M/MED-PEFT.