FORUM
Robustness Analysis
by
Jonathan Rosenhead
Department of Operational Research
London School of Economics & Political Science
London, England
Robustness Analysis is a way of supporting decision making when
there is radical uncertainty about the future. It addresses the seeming
paradox – how can we be rational in taking decisions today if the most
important fact that we know about future conditions is that they are unknowable?
It resolves the paradox by assessing initial decisions in terms of the
attractive future options that they keep open.
While writing this article an academic colleague on another continent
wrote proposing to purchase an air ticket to London for a project meeting
scheduled for the summer of 2003 – ie 1 year ahead. The reason: that his
local currency (in which the project funds are kept) is inflating fast,
so that the ticket will cost much more if purchased later. This can stand
in as a simplified example of the dilemmas that life presents us with.
Should he purchase an inflexible ticket now? Undoubtedly the cheapest option,
but also the one with most exposure to uncertainty. Should he buy now,
but choose a ticket with some form of flexibility for subsequent change
of flight? Should he delay, and buy an inflexible ticket at a later time
when our project uncertainties are less, but when the price will be higher.
This decision needs to be taken now. Not taking a decision is also
a decision, the decision not to purchase. But consider the uncertainties.
The project may move faster or slower than expected, rendering the planned
August meeting untimely. An internal dispute may undermine the project,
or the sponsors may pull the plug. Either of us may be subject to illness,
or family demands, or to competing time priorities for those particular
weeks from other equally valid commitments. And we do not know how fast
or slow the future rate of inflation will be.
To resolve this particular problem will not, I am sure, require a massive
analytical apparatus. But it does illustrate, in the small, the uncertainty-related
issues that can bedevil a wide range of decisions – decisions confronted
by individuals, businesses, public agencies, voluntary associations, governments.
Many of these decisions are of an order of complexity that does merit serious
analytic attention.
The example also points up the organising principle that most sensible
people would use, intuitively, when confronted by such dilemmas: namely,
to explore the future options for action that are left open by the alternative
choices available to them now. What is curious is that OR/MS has almost
entirely neglected this concept of flexibility, remaining largely fixated
on optimisation or methods derived from it.
Flexibility is not the only criterion that is relevant, but it should
be among those that are employed. Employing a number of criteria without
a predefined rule for combining them may seem sloppy and incomplete to
those who think that the task of analysis is to decide the issue. If, however,
we adopt the more modest and practicable aim of providing those who have
the problem with structured information relevant to their decision, this
difficulty evaporates. And indeed it makes especial sense that this information
should, as far as possible, make intuitive sense to those who must use
it.
In this article I will first argue for the wide prevalence of uncertainty
in strategic decision making situations (and hence the potential relevance
of robustness). I will then introduce the basic principles of robustness,
how to specify a problem for robustness analysis, and the calculation of
the robustness score. A commentary suggests how robustness can be appropriately
applied, and there is an indication of the range of practical applications.
Finally, robustness analysis is distinguished from a number of other nearby
methods.
Prevalence of uncertainty
How widespread are the decision situations for which uncertainty is crucial?
I am tempted to wonder whether that question even needs asking in a world
so evidently turbulent in its arrangements. I advance in evidence (just
a sample, and all at the macro level) the collapse of the Soviet Union
(who predicted it?), the dot.com bubble, the Twin Towers, the 2002 bear
market. Clearly there are many decision situations in which uncertainty
does not pay a key role. This is particularly true of repetitive operational
decisions. For these the rate of change of underlying conditions is usually
small compared with the cycle-time of activities, and inherent variability
can be accommodated by probabilistic analysis. However these saving graces
are not usually available in the case of non-routine, more strategic decision
situations.
Businesses are subject to turbulence in the market place, to variations
in regulatory regimes, to new technologies threatening established markets,
to the unpredictable results of R&D or of mineral resource exploration.
Public service providers are vulnerable to the vagaries of governmental
funding, to changing expectations of their clientele, to the organisational
impacts of new technologies. Nation states may experience violent threats
of novel kinds or sources, the impact of decisions by transnational corporations,
the erosion of sovereignty to supranational organisations, forceful demands
for regional autonomy. Grassroots organisations are hit by the backwash
of the decisions of all these more powerful actors. And so on. This is
not an attempt at an exhaustive categorisation of the ways in which uncertainty
permeates our decision environment. Rather it is an attempt to convince
you that uncertainty is significant in particular in the more formative
decisions that social organisations confront.
Principles of robustness analysis
Robustness analysis is applicable when
i) uncertainty is a factor that obstructs confident decision – which
has been discussed above; and
ii) decisions must be or can be staged. - that is, the commitments
made at the first point of decision do not necessarily define completely
the future state of the system. There will be one or more future opportunities
to modify or further define it.
The first element ensures that uncertainty matters. The second ensures
that there is something that we can do about it.
A simple statement of the robustness criterion is that, other things
being equal, an initial commitment should be preferred if the proportion
of desirable future situations that can still be reached once that decision
has been implemented is high. Put still more simply, it is a good thing
to keep your options open.
That is the intuitively sensible proposition that underlies robustness
analysis. Further specification is needed however to transform it into
a systematic methodology that can be applied with some consistency. What
counts as a desirable future situation? How do we count them? How do we
identify which of them are kept open?
Specifying a problem situation for robustness analysis
The first set of elements which must be specified are
- a set of alternative initial commitments to be considered
- (normally) a set of ‘futures’ representative of possible environments
of the system
- a set of relevant possible configurations of the system which the
decisions will modify.
A commitment may be an allocation of resource in a particular decision
domain, or it may comprise an integrated package of such allocations. Commitments
may be those which appear logically possible, or those proposed by stakeholders
with some influence over decision making. The futures, similarly, may be
generated by systematic or more clearly subjective processes, or a mixture
of the two. The configurations may be relevant in the sense that they are
plausible extensions of the directions set by particular initial commitments;
or that they can be expected to perform well in one or more of the identified
futures; or that they have been proposed as a longer term goal by partisans
within the management process.
It is evident that these three elements can be inter-dependent. Configurations
may be generated by thinking about futures; the extrapolation of commitments
may lead to possible configurations; and so on. Specification is often
best achieved in interactive mode with those who are faced with the need
to decide. That is, the analysis is carried out by and under the control
of the relevant management group, with the assistance of one or more consultants.
This and other features place robustness analysis within the family of
Problem Structuring Methods (see Rosenhead and Mingers, 2001).
The three elements above need to be complemented by information of
the following types:
- assessments of the compatibility of each commitment-configuration
pair
- evaluation of the performance of each configuration in each future.
The former, a zero-one assessment, is needed in order to examine the
extent to which options are maintained by particular commitments. The latter
is also carried out on a zero-one basis. Is the predicted performance acceptable
or not?
In cases where configurations consist, in effect, of an aggregation
of the available commitments, compatibility can be directly established.
In other cases there will be a degree of subjectivity in the assessment.
Likewise, for performance evaluation, it may sometimes be possible to agree
a set of multi-dimensional performance measures each with their acceptance
thresholds, and to build a model to predict the values of the measures
for any combination of configuration and future. In such cases the performance
evaluation can be automated. Otherwise it may require discussion among
those with relevant experiential knowledge to establish which performances
are ‘good enough’.
If these two stages need to rely extensively on elicitation rather
than on computation, there is a clear danger of combinatorial escalation
rendering the process infeasible. Groups are not good at rapid and repeated
but thoughtful evaluations of the kind that are required. There is therefor
a strong argument for keeping the dimensions of the problem formulation
as small as possible; and it may be necessary for the group to delegate
the first attempt at one or both of these stages to one of its members,
working with a consultant.
Analysing for robustness
Once these processes of elicitation and evaluation have been carried out,
it is possible to gain a picture of the pattern of flexibility which any
commitment offers, interpreting flexibility to be the future opportunity
to take decisions towards desired goals. The robustness of a commitment
is the ratio of the number of acceptably performing configurations with
which that commitment is compatible, to the total number of acceptably
performing configurations.
Clearly this limits robustness scores to the range (0, 1). A robustness
score of zero indicates that no acceptable options are kept open, while
a robustness of unity means that they all are.
Each commitment now has a robustness score for each future, since a
configuration’s performance will vary across future contexts. Commitments
can thus be assessed for the spread of flexibility they offer both within
and across futures. This process will rarely identify a dominant commitment,
but it will usually eliminate non-contenders, and focus discussion on just
a small number of relatively attractive alternatives. It may also concentrate
attention on those futures which are most crucial to the choice between
these alternatives – raising the question of whether the decision-making
group can exert selective influence on what future does (or does not) materialise.
Some comments
It may be noted that this procedure depends on identifying alternative
futures which the system under consideration may confront. It is a fair
criticism that since the future is infinitely devious, we cannot know that
any of our identified futures will capture the key aspects of the future
that actually happens. Evidently the elicitation process should endeavour
to reduce this risk, for example by selecting a broad range of contrasting
possible future environments. However the approach does not, cannot, require
that this eventual future is actually identified with certainty.
Consider an initial commitment which is the first step to an ‘optimum’
solution in a single predicted future. It will maintain flexibility at
best only by accident. By contrast, a robust commitment will maintain flexibility
over a wide range of conceivable futures. The value of this in a future
which may be outside the range of those considered cannot be rigorously
demonstrated. However it is at least highly plausible that this diversity
of options is more likely to include routes to one or more future configurations
that will perform acceptably in the eventual future context.
In any case the principle advantage of robustness analysis lies more
in its process than in its product. It does not offer a simple decision
rule – “calculate the highest robustness score, and select the commitment
that provides it”. Rather it provides a language in which the logic of
option maintenance can be worked through. Furthermore this language is
accessible also to those without developed quantitative skills. It therefor
opens up for systematic dialogue with and between those who must accept
responsibility for any decision, an uncertainty-based discourse that optimisation-oriented
methods do not provoke.
Applications of robustness
Practical uses of robustness analysis have included
- brewery location
- chemical plant expansion
- hospital location
- regional health planning
- oil field development
- personal educational and career planning
References to most of these will be found in Rosenhead (2001a, 2001b).
Relationship to other approaches
It may be helpful to compare and contrast this notion of robustness analysis
with other related approaches.
Statistical robustness
The term ‘robustness’ is used in statistics to refer to a desirable characteristic
of statistical procedures. One says that a procedure is robust against
some departure from the assumptions of the model when the procedure continues
to work well even when, to a greater or lesser extent, the assumptions
do not hold. Such assumptions, often adopted for ease of computation, might
be that an underlying distribution is Normal, or that the observations
have constant variance. In the case of statistical hypothesis testing,
which approaches most nearly the decision-focussed approach adopted in
this article, a robust test avoids the difficulty of a decision (here between
two hypotheses) resting in too unstable a fashion on a particular assumption.
Bayesians give the term a rather more specific meaning. A Bayesian
application is robust if the posterior distribution for an unknown parameter
is not unduly affected by the choice either of the prior distribution or
of the form of model taken to be generating the data.
In either approach uncertainty, though limited to knowledge about whether
the specific assumptions do in fact hold, clearly lies behind the need
for this concept. It does not of course purport to address other types
of uncertainty, or sequentiality of decisions.
Sensitivity analysis
Sensitivity analysis is a systematic procedure used to explore how an optimal
solution responds to changes in inputs – which are typically either known
values which might vary in the future, or parameters whose values are open
to question. Thus the analysis is based round a prior assumption that optimisation
is centre stage, with uncertainty viewed as a potentially disruptive factor.
The analysis aims at discovering how sensitive the ‘optimal’ solution is
to changes in crucial factors. An insensitive solution is an advantage
and, to add to linguistic confusion, is sometimes termed ‘robust’.
Robustness analysis (after Roy)
This use of the term ‘robustness analysis’ entered the literature some
13 years after it was first introduced in the sense employed in this article.
As with sensitivity analysis this approach seeks to incorporate the real
world experience of uncertainty into the understanding of mathematically
derived results. It differs from sensitivity analysis in two ways. The
first difference is that it aims to handle not only optimisation but a
range of other computational results – eg that a certain solution is feasible,
or that it is near optimal. The second is that its perspective is virtually
the mirror image of that of sensitivity analysis. This is to identify the
domain of points in the solution space for which a particular result continues
to hold. Uncertainty, however, remains attached to parameter values, rather
than to the swathe of intangible uncertainties that may be resistant to
credible quantification. And as with sensitivity analysis, the idea of
exploiting sequentiality to achieve flexibility is absent.
The purpose of this comparison is not to criticise these formulations,
but by distinguishing robustness analysis (in the sense of this article)
from them to clarify its characteristics. Each of them performs functions
which robustness analysis does not attempt, and vice versa.
In conclusion
For a more extended introduction to robustness analysis, see Rosenhead
(2001a, 2001b). Fuller references are available there.
This summary has been couched largely in terms of the practicalities
of decision-making. A more polemical case, but no less legitimate, could
be advanced in the language of sustainable development. To quote Russ Ackoff
(1988)
“The freedom to decide, to make choices, is for me the most important
freedom people of any age can have. But this freedom is empty without alternatives
from which to choose. To deprive future generations of options is a deprivation
of their rights.”
REFERENCES
R.L. Ackoff ‘The Future is now’, Systems Practice 1(1), 1988, pp. 7-9.
J.Rosenhead (2001a) ‘Robustness analysis: keeping your options open’.
In J. Rosenhead and J. Mingers (eds.) Rational Analysis for a Problematic
World Revisited: problem structuring methods for complexity, uncertainty
and conflict, Wiley, Chichester, 2001, pp. 181-207.
J.Rosenhead (2001b) ‘Robustness to the first degree’. In J. Rosenhead
and J. Mingers (eds.) Rational Analysis for a Problematic World Revisited:
problem structuring methods for complexity, uncertainty and conflict, Wiley,
Chichester, 2001, pp. 209-223.
J. Rosenhead and J. Mingers (eds.) Rational Analysis for a Problematic
World Revisited: problem structuring methods for complexity, uncertainty
and conflict, Wiley, Chichester, 2001.
ACKNOWLEDGMENTS
My thanks are due to John Howard and Larry Phillips for advice on statistical
concepts touched on in this article.
EWG-MCDA Newsletter, Fall 2002
[HOME - EWG MCDA]
[Newsletter articles of the EWG - MCDA]