Evaluation of causal understanding in AI models for oral lesion diagnosis: A counterfactual approach
Kioumars Tavakoli Tafti1, Adeetya Patel1, Camille Besombes1, Peter Chauvin1, and Sreenath Madathil1
1. Faculty of Dental Medicine and Oral Health Sciences, 污污草莓视频 University, Montreal, Canada.
Introduction: Oral lesions pose complex clinical challenges and while convolutional neural networks (CNNs) show promise in accurate diagnostics, their lack of interpretability hinders clinical use. Counterfactual scenarios used to evaluate the causal understanding of these models. Furthermore, counterfactual images used to assess the safety and interpretability aspects of CNNs when use for safety critical clinical tasks such as diagnosing oral lesions.
Purpose: To evaluate the causal understanding of two CNN models developed to classify oral lesions, under different counterfactual scenarios.
Methodology: We have previously developed and internally validated two CNN models to identify the oral lesion from intra-oral images. We will generate counterfactual images for 240 images from the test datasets of the models across 6 scenarios (change in color, size, border, anatomical region, absence of lesions, and clinically impossible lesions) using image editing software. In addition, we will create counterfactual images for sixty images from the training data to assess overfitting. We will assess and compare the predictive performance and uncertainty metrics from both models under different counterfactual scenarios.
Results: Our preliminary results show that the comparison between the original images and counterfactual images without any lesions showed an increase on the uncertainty of predicted probabilities. Whereas alterations in lesion characteristics, particularly in size and color, resulted in substantial variations in the model's predictions. Finally, evaluating the model's performance on counterfactual images generated from the training data, consistent predictions with minimal deviation were observed, suggesting potential indications of overfitting.
Conclusions: This study highlights the crucial role of evaluating deep learning models under counterfactual scenarios for ensuring clinical safety. Furthermore, our findings emphasize the need for interpretability-enhancing techniques to refine CNN models, enhancing their clinical applicability in diagnosing oral lesions.
Keywords: Counterfactual approach, Oral lesion