Message

Based on previous work by our group with manual annotation of visible Barrett oesophagus (BE) cancer images, a real-time deep learning artificial intelligence (AI) system was developed. While an expert endoscopist conducts the endoscopic assessment of BE, our AI system captures random images from the real-time camera livestream and provides a global prediction (classification), as well as a dense prediction (segmentation) differentiating accurately between normal BE and early oesophageal adenocarcinoma (EAC). The AI system showed an accuracy of 89.9% on 14 cases with neoplastic BE.

In more detail

This paper follows up on our prior publication on the application of AI and deep learning in the evaluation of BE.1 2 In our initial publications, we developed a computer-aided diagnosis (CAD) model and demonstrated promising performance scores in the classification and segmentation domains during BE assessment.1 2 However, these results were achieved on optimal endoscopic images, which may not mirror the real-life situation sufficiently. To enable the seamless integration of AI-based image classification into the clinical workflow, our previous system was developed further to increase the speed of image analysis for classification and the resolution of the dense prediction, which shows the color-coded spatial distribution of cancer probabilities.1 2 Still based on deep convolutional neural nets (CNNs) and a residual net (ResNet) architecture with DeepLab V.3+, a state-of-the-art encoder–decoder network was adapted.3 To transfer the endoscopic livestream to our AI system, a capture card (Avermedia, Taiwan) was plugged to the endoscopic monitor.

Online supplementary video 1 shows the setting of AI-based BE evaluation in the endoscopy room of the University Hospital Augsburg (figure 1). The AI prediction can be started at any time using either a button on the keyboard or a foot switch. The video clip shows examples of three different patients and the AI predictions at different locations. Note that image freeze is not necessary for the AI analysis, which is randomly done by the system, but was done for evaluation purposes of this study to ensure the same basis for human and AI predictions. As can be seen, a dynamic blue bar at the bottom of the screen provides a continuous prediction of the probability of cancer in real time. In conformity to our previous AI system,2 an image is classified as EAC if the global cancer prediction exceeds 90%.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Seamless integration of AI-based computer-aided diagnosis in the clinical setting. The resulting AI prediction is based on the average predictions of original image and its three flipped variants. Each single prediction results from an ensemble of four independently trained models, each of which uses 90% of the training data available. This procedure increases the robustness but is more time-consuming. Overall, on a desktop with two NVidia TitanX graphics processing units, the AI prediction takes 1.19 and 0.13 s with and without ensembling, respectively.

Online supplementary video 2 shows the user interface (UI) and the application features. The left part of the UI consists of a preview window of the livestream of the endoscopic examination (or optionally a video) and a list of patients examined. The main window shows the probability-based dense prediction. A real-time partial prediction is shown by the dynamic blue bar at the bottom of the screen. Three different visualisations of the results are provided: the Barrett view, the cancer view and the combined view. Additionally, the global prediction of cancer for a particular region of interest is located as a thick blue bar on the lower part of the UI. Finally, online supplementary video 2 sketches briefly the AI method used for global and dense prediction (figure 2). The method is based on DeepLab V.3+, an encoder–decoder artificial neural network.3 Similar to our previous CAD system,2 the encoder is ImageNet-pretrained and based on a ResNet with 101 layers. It successively reduces the spatial resolution of the feature maps in each layer to learn semantic information at a higher abstraction level. The last ResNet layer uses the concept of dilated convolutions followed by a spatial pyramid pooling to allow the integration of larger contexts into the classification.3 For global prediction, two fully connected layers are linked to the encoder path.4 For the dense prediction, the decoder uses the semantic information of the encoder and distributes it to full spatial resolution. Therefore, the encoder neural network is used for classification (global prediction) as well as for segmentation (dense prediction).

• Download figure
• Open in new tab
• Download powerpoint

Figure 2

Encoder–decoder neural network DeepLab V.3+3 with different paths for global and dense prediction,4 respectively.

In our present study, the AI system was trained using a total of 129 endoscopic images from our image database at the University Hospital Augsburg. To validate the system in real time, additional images (36 of early EAC and 26 of normal BE) from 14 patients (4 female and 10 male) were assessed by the AI system concurrently with the endoscopic examination by an expert BE endoscopist. All images were validated by pathological examination of resection specimens (EAC), as well as forceps biopsies (normal BE). The AI system had excellent performance scores in the classification task with a sensitivity and specificity of 83.7% and 100.0%, respectively, and an overall accuracy of 89.9%.

Find more details on Methods and Results online.

Comments

The incidence of EAC in the Western world is still on the rise, and the overall prognosis remains poor.5 BE is the most important risk factor for the development of EAC especially in patients with long-segment BE and intraepithelial neoplasia.6 The early detection of EAC can lead to a substantial improvement in prognosis; however, the endoscopic detection and subsequent characterisation of early EAC in BE is a challenge, even for experienced endoscopists. Early EAC is often flat and difficult to distinguish from the surrounding non-dysplastic Barrett’s mucosa, even with high-definition endoscopes.7 The Bing study group comprising international Barrett experts achieved a sensitivity and specificity of 80% and 89%, respectively, in the diagnosis of high-grade neoplasia and mucosal cancer.8 Endoscopists with less experience in BE evaluation will probably achieve lower performance values. Therefore, the need for more efficient methods of detection and characterisation of early EAC in BE has led to intensive research in the field of AI and CAD, especially with deep neural networks.

To our knowledge, this is the first real-time application of a deep learning AI system in the evaluation and diagnosis of early EAC in BE in a real-life setting. The video demonstrations show the difficulties encountered by expert and non-expert endoscopists during BE assessment. Since the AI system showed a very similar performance compared with the experienced endoscopist (the AI system and the expert disagreed in only one of the examined regions), the AI-based CAD has the potential to improve the quality of BE assessment, especially for non-expert endoscopists. Study limitations include the low number of patients included in the present study. Furthermore, elevated lesions delineated by the subjective impression of the endoscopist were included in our initial study. However, further optimisation with evaluations will follow.