Indices for the evaluation of wildfire spread simulations using contemporaneous predictions and observations of burnt area

Methods to objectively evaluate performance are critical for model development. In contrast to recent advances in wildfire simulation, there has been limited attention to evaluating fire model performance. Information to validate fire models is typically limited, commonly to a few perimeter observations at a small number of points in time. We review metrics for comparing two burnt areas at a point in time: observed and predicted. These are compared in an idealised landscape and with a case study evaluating the performance of simulations of an Australian wildfire. We assessed: Shape Deviation Index (SDI), Jaccard's coefficient, F1, Sorensen's Similarity and Area Difference Index (ADI). For decomposing fit into error components (overprediction and underprediction) we assessed the partial indices of SDI and ADI, Precisionand Recall. The various metrics were evaluated for their ability to represent error and their suitability for use in model improvement frameworks. Fire simulation models are of increasing importance to wildfire management.Verification of fire simulation models is necessary before operational use.There are no indices of model performance being used consistently.The properties of various fire model performance indices were evaluated.Performance indices properties should be considered when interpreting results.