Cloud contamination severely limits the usability of optical remote sensing images and affects many downstream applications. Although diffusion-based models have recently shown strong potential for cloud removal, most existing methods still start from pure Gaussian noise and require long sampling chains, leading to high computational cost and limited efficiency. In this article, we propose PCDiffCR, a cloud-driven diffusion model with a fast sampling strategy for single-pair optical image cloud removal. Unlike conventional diffusion methods, PCDiffCR starts diffusion from the real cloudy image. Partial noise is injected rather than starting from a fully noisy state. We further design a compact subnoise schedule that restricts sampling to the low- and mid-noise regime, enabling the model to preserve useful scene structures while concentrating restoration capacity on cloud-contaminated regions. In addition, a conditional UNet denoiser with a residual attention group and a gated skip refinement module is introduced to enhance cloud-aware feature extraction and suppress cloud-related shallow interference. Experiments on three public datasets show that the proposed method achieves superior or competitive quantitative and perceptual results compared with representative generative adversarial networks -based and diffusion-based baselines, while significantly reducing sampling time. Overall, the proposed PCDiffCR offers a practical diffusion-based framework for efficient and high-quality optical remote sensing cloud removal.