Extending machine learning prediction capabilities by explainable AI in financial time series prediction

Prediction with higher accuracy is vital for stock market prediction. Recently, considerable amount of effort has been poured into employing machine learning (ML) techniques for successfully predicting stock market price direction. No matter how successful the proposed prediction model is, it can be argued that there occur two major drawbacks for further increasing the prediction accuracy. The first one can be referred as the black box nature of ML techniques, in other words inference from the predictions cannot be explained. Furthermore, due to the complex characteristics of the predicted time series, no matter how sophisticated techniques are employed, it would be very difficult to achieve a marginal increase in accuracy that would meaningfully offset the additional computational burden it brings in. For these two reasons, instead of chasing incremental improvements in accuracy, we propose utilizing an “eXplainable Artificial Intelligence” (XAI) approach which can be employed for assessing the reliability of the predictions hence allowing decision maker to abstain from poor decisions which are responsible for declining overall prediction performance. If there would be a measure of how sure the prediction model is on any prediction, the predictions with a relatively higher reliability could be used to make a decision while lower quality decisions could be avoided. In this study, a novel two-stage stacking ensemble model for stock market direction prediction based on ML, empirical mode decomposition (EMD) and XAI is proposed. Our experiments have shown that, proposed prediction model supported with local interpretable model-agnostic explanations (LIME) achieved the highest accuracy of 0.9913 when only the most trusted predictions have been considered on KOSPI dataset and analogous successful results have been obtained from five other major stock market indices.