

 
Home  About  News  IGDT  The Campaign  Myths & Facts  Reviews  FAQs  FUQs  Resources  Contact Us 
The rise and rise of voodoo decision making 
Review 52022 (Posted: January 28, 2022; Last update: January 28, 2022)
Reference David R. Johnson and Nathan B. Geldner. Contemporary Decision Methods for Agricultural, Environmental, and Resource Management and Policy. Annual Review of Resource Economics, 11:1941. Year of publication 2019 Publication type Peerreviewed journal article. Downloads https://www.annualreviews.org/doi/10.1146/annurevresource100518094020 Abstract Traditional topdown methods for resource management ask first what future conditions will be, then identify the best action(s) to take in response to that prediction. Even when acknowledging uncertainty about the future, standard approaches (a) characterize uncertainties probabilistically, then optimize objectives in expectation, and/or (b) develop a small number of representative scenarios to explore variation in outcomes under different policy responses. This leaves planners vulnerable to surprise if future conditions diverge from predictions. In this review, we describe contemporary approaches to decision support that address deep uncertainty about future external forcings, system responses, and stakeholder preferences for different outcomes. Many of these methods are motivated by climate change adaptation, infrastructure planning, or natural resources management, and they provide dramatic improvements in the robustness of management strategies. We outline various methods conceptually and describe how they have been applied in a range of landmark realworld planning studies. Reviewer Moshe Sniedovich IFIG perspective This article correctly identified a fundamental flaw in IGDT. However, it missed a golden opportunity to place IGDT in its proper place in the state of the art in decisionmaking under deep uncertainty, and to clarify its lineage.
The article describes IGDT as follows (colors are used for emphasis):
Infogap decision theory, as detailed by BenHaim (2006), contrasts with some of the previously outlined approaches to decision making under deep uncertainty. Rather than focusing on exploration of the range of plausible future SOWs, it focuses on identifying how much future conditions can deviate from a bestguess estimate before outcomes fail to meet requirements.The key conceptual hook of infogap is that decisions should maximize the likelihood of obtaining an acceptable outcome , accounting for uncertainty in both the probability of future SOWs and the utility associated with those SOWs under each strategy. .
Johnson and Geldner (2019, p. 30)This description is based on a myth circulating among some proponents of IGDT. This myth seems to provide the "logic" behind IGDT approach to severe uncertainty, and is often inserted intuitively by users who presume that this is how IGDT actually operates. Apparently, this is done subconsciously, the logic being that otherwise IGDT would make no sense.
It is therefore important to remind the reader that, according to the Father of IGDT (BenHaim 2001, 2006, 2010) the uncertainty that IGDT was designed to deal with is
severe in the sense that
 The uncertainty space is
wide and diverse . In many cases it isunbounded .
 The so called "best estimate" is often a
guess , sometime it is an educatedguess, sometime it is awildguess .
 The uncertainty is
probability, likelihood, plausibility, chance, belief  FREE! And so, contrary to what is claimed in the above quote, "The key conceptual hook of infogap"
cannot possibly be "that decisions shouldmaximize the likelihood of obtaining an acceptable outcome.It is interesting that other scholars expressed similar assessments of IGDT's ability to create an imaginary "likelihood" or "belief" structure:
SuperNatural Power Myth: In some sense IGDT can be viewed as a replacement for probability theory. To wit:
Informationgap (henceforth termed ‘‘infogap’’) theory was invented to assist decisionmaking when there are substantial knowledge gaps and when probabilistic models of uncertainty are unreliable (BenHaim, 2006). In general terms, infogap theory seeks decisions that are most likely to achieve a minimally acceptable (satisfactory) outcome in the face of uncertainty, termed robust satisficing.
Burgman et al. (2008, p. 8)However, if they are uncertain about this model and wish to minimize the chance of unacceptably large costs, they can calculate the robustoptimal number of surveys with equation (5).
Rout et al. (2009, p. 785)Infogap models are used to quantify nonprobabilistic "true" (Knightian) uncertainty (BenHaim 2006). An infogap model is an unbounded family of nested sets, $\mathscr{U}(\alpha,\widetilde{u})$. At any level of uncertainty $\alpha$, the set $\mathscr{U}(\alpha,\widetilde{u})$ contains possible realizations of $u$. As the horizon of uncertainty $\alpha$ gets larger, the sets $\mathscr{U}(\alpha,\tilde{u})$ become more inclusive. The infogap model expresses the decision maker's beliefs about uncertain variation of $u$ around $\tilde{u}$._{ }
Davidovitch and BenHaim (2010, pp. 2678)Reality Check:_{ }
According to BenHaim (2001, 2006, 2010), IGDT is a probability, likelihood, plausibility, chance, belief FREE theory. Hence, its robustness analysis cannot possibly rank decisions according to the likelihood, or chance that they achieve acceptable (satisfactory) outcome in the face of such an uncertainty (except for trivial cases). Furthermore, IGDT cannot possibly express the decision maker's beliefs about uncertain variation of $u$ around $\tilde{u}$.The interesting this is that there are warnings in the IGDT literature itself against such misapplications of the theory. For instance, consider this text that predate the official birth of IGDT :
However, unlike in a probabilistic analysis, $r$ has no connotation of likelihood. We have no rigorous basis for evaluating how likely failure may be; we simply lack the information, and to make a judgment would be deceptive and could be dangerous. There may definitely be a likelihood of failure associated with any given radial tolerance. However, the available information does not allow one to assess this likelihood with any reasonable accuracy.
BenHaim (1994, p. 152)And even the main text on IGDT warns users:
In infogap set models of uncertainty we concentrate on clusterthinking rather than on recurrence or likelihood. Given a particular quantum of information, we ask: what is the cloud of possibilities consistent with this information? How does this cloud shrink, expand and shift as our information changes? What is the gap between what is known and what could be known. We have no recurrence information, and we can make no heuristic or lexical judgments of likelihood.
BenHaim (2006, p. 18)Go figure!
IGDT cannot possibly maximize the likelihood of events associated with its uncertainty because IGDT assumes that the uncertainty is
likelihoodfree .Next consider this text in the article (colors are used for emphasis):
It does, however, have onemajor limitation: It considers only local uncertainty in the neighborhood of the bestestimate state of the world . Thus, it is most useful where a reasonable best estimate exists and there is strong reason to believe increasingly large deviations from the best estimate are decreasingly likely (Hayes et al. 2013). While this does not require the uncertainty to be well parameterized, it is a strong assumption nonetheless (Taleb 2005). If the future SOW is no more likely to be within an arbitrary neighborhood of the best estimate than far away from it, infogap may lead to endorsement of strategies vulnerable to high regret.
Johnson and Geldner (2019, p. 31)The reference to (Hayes et al. 2013) hides a much stronger criticism of IGDT. Here it suffices to note the following:
Satisficing metrics can also be based on the idea of a radius of stability, which has made a recent resurgence under the label of infogap decision theory (BenHaim, 2004; Herman et al., 2015). Here, one identifies the uncertainty horizon over which a given decision alternative performs satisfactorily. The uncertainty horizon 𝛼 is the distance from a prespecified reference scenario to the first scenario in which the prespecified performance threshold is no longer met (Hall et al., 2012; Korteling et al., 2012).However, as these metrics are based on deviations from an expected future scenario, they only assess robustness locally and are therefore not suited to dealing with deep uncertainty (Maier et al., 2016). These metrics also assume that the uncertainty increases at the same rate for all uncertain factors when calculating the uncertainty horizon on a set of axes. Consequently, they are shown in parentheses in Table 2 andwill not be considered further in this article.
McPhail et al. (2018, p. 174)And for a second opinion, they may also wish to consult the article Severe uncertainty and infogap decision theory (Hayes et al. 2013) where they will find these texts (colors are used for emphasis):
Ecologists and managers contemplating the use of IGDT should carefully consider its strengths and weaknesses, reviewed here, and not turn to it as a default approach in situations of severe uncertainty, irrespective of how this term is defined.We identify four areas of concern for IGDT in practice: sensitivity to initial estimates, localized nature of the analysis, arbitrary error model parameterisation and the ad hoc introduction of notions of plausibility.
Hayes et al. (2013, p. 1)Sniedovich (2008) bases his arguments on mathematical proofs that may not be accessible to many ecologists but theimpact of his analysis is profound . It states that IGDT providesno protection against severe uncertainty and that the use of the method to provide this protection is thereforeinvalid.
Hayes et al. (2013, p. 2)The literature and discussion presented in this paper demonstrate that the results of BenHaim (2006) are not uncontested. Mathematical work by Sniedovich (2008, 2010a) identifiessignificant limitations to the analysis. Our analysis highlights a number of other important practical problems that can arise. It is important that future applications of the techniquedo not simply claim that it deals with severe and unbounded uncertainty but provide logical arguments addressing why the technique would be expected to provide insightful solutions in their particular situation.
Hayes et al. (2013, p. 9)Plausibility is being evoked within IGDT in anad hoc manner , and it isincompatible with the theory's core premise, hence any subsequent claims about the wisdom of a particular analysishave no logical foundation . It is therefore difficult to seehow they could survive significant scrutiny in realworld problems . In addition, cluttering the discussion of uncertainty analysis techniques with ad hoc methods should be resisted.
Hayes et al. (2013, p. 609)Next, the issue of the lineage of IGDT in the framework of the state of the art in decisionmaking under uncertainty. The article assigns a lineage of "1,16" to IGDT. Since the article does not provide any detail on how these figures were determined, it is impossible to say much about them. However, any meaningful lineage analysis of IGDT would have discovered the following two important facts that are not discussed in the article.
 IGDT's robustness analysis is a simple application of Wald's famous
Maximin paradigm (circa 1940).
 IGDT robustness model is a reinvention of the well known and well established concept
Radius of Stability (circa 1962).Given all of this, it is not clear at all why some DMDU scholars continue to consider IGDT as a candidate for proper treatment of deep uncertainty.
Readers of this review are encouraged to read Review 22022.
Bibliography and links
Articles/chapters
 Sniedovich M. (2007) The Art and Science of Modeling DecisionMaking Under Severe Uncertainty. Journal of Decision Making in Manufacturing and Services, 1(12), 111136. https://doi.org/10.7494/dmms.2007.1.2.111
 Sniedovich M. (2008) Wald's Maximin Model: A Treasure in Disguise! Journal of Risk Finance, 9(3), 278291. https://doi.org/10.1108/15265940810875603
 Sniedovich M. (2008) From Shakespeare to Wald: Modelling worstcase analysis in the face of severe uncertainty. Decision Point 22, 89.
 Sniedovich M. (2009) A Critique of InfoGap Robustness Model. In Martorell et al. (eds), Safety, Reliability and Risk Analysis: Theory, Methods and Applications, pp. 20712079, Taylor and Francis Group, London.
 Sniedovich M. (2010) A bird's view of infogap decision theory. Journal of Risk Finance, 11(3), 268283. https://doi.org/10.1108/15265941011043648
 Sniedovich, M. (2011) A classic decision theoretic perspective on worstcase analysis. Applications of Mathematics, 56(5), 499509. https://doi.org/10.1007/s104920110028x
 Sniedovich, M. (2012) Black swans, new Nostradamuses, voodoo decision theories and the science of decisionmaking in the face of severe uncertainty. International Transactions in Operations Research, 19(12), 253281. https://doi.org/10.1111/j.14753995.2010.00790.x
 Sniedovich M. (2012) Fooled by local robustness: an applied ecology perspective. Ecological Applications, 22(5), 14211427. https://doi.org/10.1890/120262.1
 Sniedovich, M. (2012) Fooled by local robustness. Risk Analysis, 32(10), 16301637. https://doi.org/10.1111/j.15396924.2011.01772.x
 Sniedovich, M. (2014) The elephant in the rhetoric on infogap decision theory. Ecological Applications, 24(1), 229233. https://doi.org/10.1890/131096.1
 Sniedovich, M. (2016) Wald's mighty maximin: a tutorial. International Transactions in Operational Research, 23(4), 625653. https://doi.org/10.1111/itor.12248
 Sniedovich, M., (2016) From statistical decision theory to robust optimization: a maximin perspective on robust decisionmaking. In Doumpos, M., Zopounidis, C., and Grigoroudis, E. (eds.) Robustness Analysis in Decision Aiding, Optimization, and Analytics, pp. 5987. Springer, New York.
Research Reports
 Sniedovich, M. (2006) What's Wrong with InfoGap? An Operations Research Perspective
 Sniedovich, M. (2011) Infogap decision theory: a perspective from the Land of the Black Swan
Links
 InfoGap Decision Theory
 Voodoo decision making
 Faqs about IGDT
 Myths and Facts about IGDT
 The campaign to contain the spread of IGDT in Australia
 Robust decision making
 Severe uncertainty
 The mighty maximin
 Viva la Voodoo!
 Risk Analysis 101
 IGDT at Los Alamos