The application of convolutional neural network (CNN) for classifying optical remote sensing images holds great significance in areas such as maritime management, environmental protection, and more. However, the significant computational demands of CNN pose obstacles to the efficient deployment at the edge. Quantization techniques, as effective model compression methods, have been widely used to reduce computational demands during model deployment. Current uniform low-precision quantization methods suffer from significant accuracy loss. Mixed-precision quantization compresses models while maintaining accuracy, but it introduces computations with multiple bit widths. Due to the introduction of multiple bit-width computations, general-purpose computing platforms are unable to leverage the computational efficiency of quantized network. In this article, a Zynq SoC-based mixed-precision neural network accelerator design is proposed, which can fully leverage its energy efficiency. Experimental results show that the mixed-precision neural network accelerator achieves 4.96 × improvement in inference speed compared with the baseline.