This article introduces a deep learning-based classification framework to accurately detect the fetal anatomical structures in 2-D ultrasound planes, aiming to tackle challenges like low inter-class variations, subtle resemblances among anatomical structures, and redundant information in fetal ultrasound images. The proposed method employs the VGG-19 CNN model as a backbone feature extraction network. The dual attention block that comprises a spatial and channel-aware feature fusion (SCFF) module is placed on top of a backbone network to learn feature interdependence along the channel and spatial dimensions. The attention module strengthens the classification ability of the backbone CNN model to prioritize salient regions by adaptively refining spatial and channel-wise feature activations. The SCFF aids in dynamic recalibration of weights and enhances the sensitivity of the model to subtle anatomical distinctions, thereby improving accuracy and robustness in fetal ultrasound plane identification. Discriminative feature sets are furnished to the extreme learning machine for classifying fetal ultrasound images into various categories. The effectiveness of the proposed method is evaluated using publicly available fetal ultrasound image datasets collected from both high-resource and low-resource settings. The experimental findings demonstrated the supremacy of the proposed approach compared to the state-of-the-art CNN works, which attained an accuracy of 96.11%, 98.20%, and 95.60% on independent and combined datasets, respectively.