Intelligent Dispatching Lossless Cluster Compression Technology Based on Hadoop Cloud Framework

Aimed at the data sets compression problem in the intelligent dispatching system,using the Hadoop framework and Map/Reduce distributed programming model,a novel method of lossless cluster compression based on cloud framework is proposed.Dictionary coding and statistics coding of lossless compression are compared and classified,with the scheduling machine and monitoring server deployed according to the cluster network configuration of cloud cluster nodes.Lossless compression is incorporated in the cluster data nodes to build a lossless cluster compression experiment environment of dispatching monitoring information.It has been found from the dispatching section measurement that,for the lossless compression of identical section log set,the BZip2 format cluster ratio is better than that of the Deflate and Gzip formats.The test results of different compression formats on the same section sets show that the compression ratio reaches 81.1% when section sets exceed three million,which is an increase of over 30% compared with the traditional lossless compression ratio.