Two-stage neural-network based prognosis models using pathological image and transcriptomic data: An application in hepatocellular carcinoma patient survival prediction

Pathological images are easily accessible data type with potential as prognostic biomarkers. Here we extend Cox-nnet, a neural network based prognosis method previously used for transcriptomics data, to predict patient survival using hepatocellular carcinoma (HCC) pathological images. Cox-nnet based imaging predictions are more robust and accurate than Cox-PH. Moreover, using a novel two-stage Cox-nnet complex model, we are able to combine pathology image and transcriptomics RNA-Seq data to make impressively accurate prognosis predictions, with C-index close to 0.90 and log-ranked p-value of 4e-21 in the testing dataset. This work provides a new, biologically relevant and relatively interpretable solution to the challenge of integrating multi-modal and multiple types of data, particularly for survival prediction.