Number extraction in document image is a crucial technique in document image analysis. To simply and efficiently extract the numbers, a mathematical morphology based algorithm is proposed in this paper. Firstly, the square regions containing numbers are labeled by morphological dilation operation using a designed structuring element. Secondly, the square regions are extracted by morphological erosion operation. Thirdly, the inner region of the square regions is extracted through morphological dilation operations. Finally, the numbers are extracted through comparing the extracted and original inner region of the square regions. Experimental results show that the proposed algorithm is efficient.