Large-scale medical image annotation with crowd-powered algorithms

Accurate segmentations in medical imagesare the foundations for various clinical applications. Advances in machine learning-based techniques show great potential for automatic image segmentation, but these techniques usually require a huge amount of accurately annotatedreference segmentations for training. The guiding hypothesis of this paper was that crowd-algorithm collaboration could evolve as a key technique in large-scale medical data annotation. As an initial step toward this goal, we evaluated the performance of untrained individuals to detect and correct errors made by three-dimensional (3-D) medical segmentation algorithms. To this end, we developed a multistage segmentation pipeline incorporating a hybrid crowd-algorithm 3- D segmentationalgorithm integrated into a medical imagingplatform. In a pilot study of liver segmentationusing a publicly available dataset of computed tomography scans, we show that the crowdis able to detect and refine inaccurate organ contours with a quality similar to that of experts (engineers with domain knowledge, medical students, and radiologists). Although the crowdsneed significantly more time for the annotationof a slice, the annotationrate is extremely high. This could render crowdsourcing a key tool for cost-effective large-scale medical image annotation.