Methods Vignettes: Sensitivity Analysis

BACK TO METHODS VIGNETTES

Background:

“Sensitivity Analysis (SA) is the study of how the variation in the output of a model (numerical or otherwise) can be apportioned, qualitatively or quantitatively, to different sources of variation, and of how the given model depends upon the information fed into it” (Saltelli 2000, p. 3). In STEPS terms, we can see ‘a model’ to be akin to a form of framing, and as such sensitivity analysis means “linking of alternative framing assumptions with the results which they yield” (Stirling 1999, p.78).

So sensitivity analysis is a systematic approach to help us determine:

  • if a model resembles the system or process under study (read pathway or technology)
  • the factors that mostly contribute to the output variability
  • if there is some region in the space of input factors for which model variation is maximum
  • the model parameters that are insignificant, and that can be eliminated from the final model
  • if and which factors interact with each other

Once a model has been fully explored using sensitivity analysis, we can understand its findings with a much better sense of the contingencies that these are based on.

This has huge relevance when thinking of how models help to characterise and present pathways or parts of them. Often they are used to represent systems with complex dynamics and high levels of uncertainty. Outputs from models can also be communicated in simplistic and singular terms so as to background assumptions about framing, how the ‘system’ has been modelled, and the input variables used.  When we are considering pathways, technologies or decisions made with regard novel technologies, as well as phenomena with associated high levels of uncertainty like climate change or zoonotic diseases, attention to framing assumptions is particularly important.

Sensitivity analysis gives us the quantitative means by which to explore this uncertainty. However, there are many ways to measure sensitivity and, “the type of measure employed, selected on the basis of the context or use one desires to make of SA, has a direct consequence on the outcome of the analysis” (Saltelli 2000, p.5). We give a very introductory overview of quantitative sensitivity analysis here, and then go on to consider whether there are useful lessons we can learn for qualitative interrogation of system framings.


What’s involved?

Because there are many ways to compute a SA, it is important that the approach used is clearly defined and justified.

There are two broad types of SA, within each of which are variations in specific method:

  • ONE-AT-A-TIME (OAT) SA1: Here the aim is to explore the impact of individual aspects of the model on either the final model output of interest (called ‘factor screening’), or partial derivatives within the model (i.e. exploring intermediate steps within the model). Factor screening is important for determining the quality of data used to parameterise (give value(s) to) input variables. Each model input is varied to the same, often quite significant, extent and the consequence of this is then observed in model outputs.  Where model outputs are highly sensitive to particular input variables you need to, if possible, make sure the quality of data used to estimate these inputs is of a high quality. Alternatively, if this is not possible, be very clear about the uncertainty accompanying model inputs and therefore outputs.A second OAT method, called ‘local sensitivity analysis’ involves changing each variable around a small and circumscribed point value to explore how this affects a particular area of the model which is of interest. Values are varied systematically and in the same way rather than in relation to observed variability or uncertainty in that value. One at a time approaches do not explore interactions between elements of the model, and also assume independence between variables.  Nor do they examine sensitivity within the model to inputs, according to our knowledge of the range or probability of their possible values.
  • GLOBAL SA: here the aim is to quantitatively determine how the output uncertainty is linked to uncertainty in input factors. In addition, rather than taking an OAT approach, multiple factors are varied at the same time, allowing for interactions between elements of the model to be explored. Uncertainty in input factors is represented by probability distributions that cover the factor’s known possible values and the probability of taking any value. An SA ‘experiment’ is designed which samples from particular distributions in particular ways to explore what this means for the outputs of interest. The experiment is run multiple times to generate multiple possible output values – often also taking the form of a probability distribution. Global sensitivity analysis is computationally expensive.

A skilled modeller or mathematician will be able to explore SA in depth, and question the robustness of a model in these terms. However, there are also lessons that can be taken from SA in how we might qualitatively explore qualitative models, or simple quantitative models like indicators or indexes. For example, we can question how robust framing assumptions are to changes in circumstance or the broadening out of implicit terms of reference.

We can question what has and has not been included in a framing, and consider the impact on outcomes of broadening inclusion. A qualitative version of factor screening might involve asking stakeholders or ‘experts’ to consider what they think are the most influential variables in any model, and to rank them in order of relative influence. Any differences in this ranking will highlight the variability with which any situation is viewed.


Broadening Out and Opening Up? 

Sensitivity analysis comprises a range of tools and methods that will open up the contingencies and robustness of a model to scrutiny in different ways. However, this analysis will be limited to what is included in a model – sensitivity analysis does not test what could be, but what is. If a model is large or complex, even this can be prohibitive computationally. This, in addition to the need for expertise in conducting an SA, means that the potential for this method to ‘broaden out’ engagement in and inputs to appraisal are limited. The extent to which any given sensitivity analysis is effective at broadening out and opening up a given instance of modelling, will depend on the kinds of questions that govern the choice of sensitivities for exploration. In order to be as effective as possible, it is often the case that this should be subject to a critical or participatory process – which involves forms of interactive modelling which should be considered separately to sensitivity analysis itself.


Fits and Limits

Techniques of sensitivity analysis are of particular interest for STEPS research in areas where quantitative modelling has played an especially influential role. Even under conditions where more ambitiously progressive forms of appraisal prove difficult, it may be possible to achieve significant traction through considering what the performance of sensitivity analysis (either actual or potential) might reveal about the conditionalities inevitably attached to the outcomes of modelling.


References and other useful sources of information:

Saltelli, A. (2000). What is Sensitivity Analysis? Sensitivity Analysis. A. Saltelli, K. Chan and E. M. Scott. Chichester, John Wiley & Sons, Ltd.

Stirling, A. (1999). On science and precaution in the management of technological risk, Volume II, Case studies., European Commission Joint Research Centre, Institute for Prospective Technological Studies. ESTO-IPTS, EUR 19056 EN/2.

Wikipedia: https://en.wikipedia.org/wiki/Sensitivity_analysis


Credit

Material for this vignette was contributed by Dr Becky White.