Uncertainty Analysis on Multi-model Ecologies: A Study on Methods to Analyse the Impact of Uncertainties in Multi-model Ecologies and their Application to the Windmaster Model

Our contemporary society consists of increasingly complex systems-of-systems in which technical and social subsystems influence each other. Multi-models reflect this complexity and are useful to analyse these complex socio-technical systems. They consist of connected simulation models which may each have their focus on a subsystem or modelling paradigm. However, these models contain different types of uncertainties influencing the fidelity of the results. Several methods are available to identify and analyse these uncertainties. It is yet unknown if and how existing uncertainty analysis methods can be applied to multi-model ecologies. In this thesis, we aim to provide for an answer to the following research question: “To what extent can we apply existing uncertainty analysis methods to multi-models?” To answer this question, it is important to first identify additional uncertainties in multi-model ecologies compared to single models. Next, we identify and apply methods to analyse these additional uncertainties. As proof of principle, a multi-model is used which focusses on the expansion and decarbonisation of the energy grid in the Port of Rotterdam. There are three dimensions of uncertainty in simulation models: location, level, and nature. The different locations include the conceptual model, the computer model, input data, the technical model implementation, and the processed output data. In multi-models, we found an additional location: the interface. This is where the exchange of parameters between the models takes place. Within the multi-model locations, we identified some aspects that increase uncertainty. Epistemic opacity and computational expense are properties of multi-models that limit the analyst's knowledge of the multi-model and the feasibility of extensive uncertainty analyses. The methods we identified for analysing these types of uncertainties are divided into sensitivity analysis and calibration. Sensitivity analysis quantifies the contribution of specific uncertainties to the overall uncertainty in the model outcome. Well-established methods are extra-trees feature scoring and Sobol. Calibration methods are based on the notion of equifinality. Using a specified likelihood function, they determine parameter values that lead to results with a high likelihood. Monte Carlo Markov Chain (MCMC) methods are often used for calibration purposes. The applicability of these methods depends on the multi-model configuration. To describe these configurations, two limiting archetypes were used: directed graphs, and undirected graphs with feedback mechanisms over the model components. We found that uncertainty analysis of direct graphs can be carried out on both the model components and the whole. For undirected graphs, the research showed only added value in performing an uncertainty analysis on the whole multi-model. Otherwise, the changing context and emerging path dependencies cannot be included. The application of extra-trees feature scoring, Sobol, and MCMC on the case study model showed that methods for uncertainty analysis are applicable on multi-models by including uncertainties on the interface. Furthermore, it is possible to reduce computational costs by factor fixing, distinguishing between deep and stochastic uncertainties, and assessing the convergence of sensitivity indices. Epistemic opacity can be dealt with by performing multiple replications and by including uncertainties related to the technical implementation of the multi-model. MCMC methods are suitable for scenario discovery, which provides insight into parameter values that lead to specific model results. For future research on this topic, it is recommended to apply uncertainty analysis on multi-models with different network structures and a higher number of model components. This research offers a single case study multi-model, being an undirected graph with two model components. The role of uncertainties on the interface, epistemic opacity, and computational expense should then be further investigated in these different configurations. In addition, other uncertainty analysis methods in the context of multi-modelling could be investigated. Especially the application of moment-independent sensitivity analyses could be interesting to investigate further, since multimodal outcomes can arise from the interaction of heterogeneous models.