Non-Parametric Message Importance Measure: Storage Code Design and Transmission Planning for Big Data

The storage and the transmission of messages in big data are discussed in this paper, where message importance is taken into account. To this end, we propose to use non-parametric message importance measure (NMIM) as a measure of message importance, which can characterize the uncertainty of random events like Shannon entropy and Rényi entropy. We prove that NMIM sufficiently describes the two key characters of big data, i.e., the rare events finding and the large diversities of events. Based on NMIM, we then propose an effective compressed encoding mode for data storage, and discuss the transmission of messages over some typical channel models with limited message importance loss. Our numerical results show that the proposed strategy occupies less storage space without losing too much important information, and the maximum received entropy rate increases with the increasing of message importance loss until it reaches saturation, which contributes to designing of better practical communication system.