Are accident analysis methods fit for purpose? Testing the criterion-referenced concurrent validity of AcciMap, STAMP-CAST and AcciNet

Abstract Systems-based methods such as AcciMap, STAMP-CAST and more recently AcciNet, are used to uncover the multifactorial cause of accidents within complex systems to support the development of new safety interventions. Despite this, there has been a lack of research that has formally examined the reliability and validity of systems-based accident analysis methods. This gap is addressed in this study. The criterion-referenced concurrent validity of AcciMap, STAMP-CAST and AcciNet was compared in a repeated measures design using Signal Detection Theory (SDT). A process of analytical fragmentation was adopted to understand how the individual phases of each method contributed to their overall validity. The results of the overall analyses indicate that the three methods achieved comparable and moderate results. It is concluded that state-of-the-art systems-based accident analysis methods did not produce satisfactory validity results in this study compared to a referent expert analysis. Further research needs to be conducted to demonstrate the efficacy of systems-based accident methods if they are to be used with confidence and work as expected.