HBPFP-DC: A parallel frequent itemset mining using Spark

Abstract The frequent itemset mining (FIM) is one of the most important techniques to extract knowledge from data in many real-world applications. Facing big data applications, parallel and distributed solutions are widely studied. However, the frequent itemset mining process is a continuous iteration process. As an in-memory parallel execution model in which all data will be loaded into memory, Spark is especially beneficial for iterative calculations. In the study, we propose a HBPFP-DC (High Balanced Parallel Fp-Growth Considering Data Correlation) algorithm on the Spark platform. HBPFP-DC uses a newly defined node computation workload estimation model to realize the balanced grouping of the calculation tasks among computing nodes, so that each computing node can achieve a completely asynchronous frequent itemset mining only relying on their respective local projection datasets. And, in order to improve the ‘compression factor’ of the tree structure to boost mining efficiency, we consider the correlation among items when performing the above grouping process. Thereby, network and computing consumption by dividing similar items in the same group are significantly decreased. Finally, extensive experiments demonstrate that our proposed solution is efficient and scalable.