Advanced multimodal fusion method for very short-term solar irradiance forecasting using sky images and meteorological data: A gate and transformer mechanism approach

Cloud dynamics are the main factor influencing the intermittent variability of short-term solar irradiance, and therefore affect the solar farm output. Sky images have been widely used for short-term solar irradiance prediction with encouraging results due to the spatial information they contain. At present, there is little discussion on the most promising deep learning methods to integrate images with quantitative measures of solar irradiation. To address this gap, we optimise the current mainstream framework using gate architecture and propose a new transformer-based framework in an attempt to achieve better prediction results. It was found that compared to the classical CNN model based on late feature-level fusion, the transformer framework model based on early feature-level prediction improves the balanced accuracy of ramp events by 9.43% and 3.91% on the 2-min and 6-min scales, respectively. However, based on the results, it can be concluded that for the single picture-digital bimodal model, the spatial information validity of a single picture is difficult to achieve beyond 10 min. This work has the potential to contribute to the interpretability and iterability of deep learning models based on sky images.