Local weather  
Brought to you by
 IF - IG 
International Interdisciplinary Center For
Information and Fact Checking about
Info-Gap Decision Theory

Local time
Home About News IGDT The Campaign Myths & Facts Reviews FAQs FUQs Resources Contact Us
The rise and rise of voodoo decision making

Reviews of publications on Info-Gap decision theory (IGDT)

Review 5-2022 (Posted: January 28, 2022; Last update: January 28, 2022)

Reference David R. Johnson and Nathan B. Geldner. Contemporary Decision Methods for Agricultural, Environmental, and Resource Management and Policy. Annual Review of Resource Economics, 11:19-41.
Year of publication2019
Publication typePeer-reviewed journal article.
Downloads https://www.annualreviews.org/doi/10.1146/annurev-resource-100518-094020
Abstract Traditional top-down methods for resource management ask first what future conditions will be, then identify the best action(s) to take in response to that prediction. Even when acknowledging uncertainty about the future, standard approaches (a) characterize uncertainties probabilistically, then optimize objectives in expectation, and/or (b) develop a small number of representative scenarios to explore variation in outcomes under different policy responses. This leaves planners vulnerable to surprise if future conditions diverge from predictions. In this review, we describe contemporary approaches to decision support that address deep uncertainty about future external forcings, system responses, and stakeholder preferences for different outcomes. Many of these methods are motivated by climate change adaptation, infra-structure planning, or natural resources management, and they provide dramatic improvements in the robustness of management strategies. We outline various methods conceptually and describe how they have been applied in a range of landmark real-world planning studies.
ReviewerMoshe Sniedovich
IF-IG perspective This article correctly identified a fundamental flaw in IGDT. However, it missed a golden opportunity to place IGDT in its proper place in the state of the art in decision-making under deep uncertainty, and to clarify its lineage.

The article describes IGDT as follows (colors are used for emphasis):

Info-gap decision theory, as detailed by Ben-Haim (2006), contrasts with some of the previously outlined approaches to decision making under deep uncertainty. Rather than focusing on exploration of the range of plausible future SOWs, it focuses on identifying how much future conditions can deviate from a best-guess estimate before outcomes fail to meet requirements. The key conceptual hook of info-gap is that decisions should maximize the likelihood of obtaining an acceptable outcome, accounting for uncertainty in both the probability of future SOWs and the utility associated with those SOWs under each strategy. .
Johnson and Geldner (2019, p. 30)

This description is based on a myth circulating among some proponents of IGDT. This myth seems to provide the "logic" behind IGDT approach to severe uncertainty, and is often inserted intuitively by users who presume that this is how IGDT actually operates. Apparently, this is done subconsciously, the logic being that otherwise IGDT would make no sense.

It is therefore important to remind the reader that, according to the Father of IGDT (Ben-Haim 2001, 2006, 2010) the uncertainty that IGDT was designed to deal with is severe in the sense that

And so, contrary to what is claimed in the above quote, "The key conceptual hook of info-gap" cannot possibly be "that decisions should maximize the likelihood of obtaining an acceptable outcome.

It is interesting that other scholars expressed similar assessments of IGDT's ability to create an imaginary "likelihood" or "belief" structure:

Super-Natural Power Myth: In some sense IGDT can be viewed as a replacement for probability theory. To wit:

Information-gap (henceforth termed ‘‘info-gap’’) theory was invented to assist decision-making when there are substantial knowledge gaps and when probabilistic models of uncertainty are unreliable (Ben-Haim, 2006). In general terms, info-gap theory seeks decisions that are most likely to achieve a minimally acceptable (satisfactory) outcome in the face of uncertainty, termed robust satisficing.
Burgman et al. (2008, p. 8)

However, if they are uncertain about this model and wish to minimize the chance of unacceptably large costs, they can calculate the robust-optimal number of surveys with equation (5).
Rout et al. (2009, p. 785)

Info-gap models are used to quantify non-probabilistic "true" (Knightian) uncertainty (Ben-Haim 2006). An info-gap model is an unbounded family of nested sets, $\mathscr{U}(\alpha,\widetilde{u})$. At any level of uncertainty $\alpha$, the set $\mathscr{U}(\alpha,\widetilde{u})$ contains possible realizations of $u$. As the horizon of uncertainty $\alpha$ gets larger, the sets $\mathscr{U}(\alpha,\tilde{u})$ become more inclusive. The info-gap model expresses the decision maker's beliefs about uncertain variation of $u$ around $\tilde{u}$. 
Davidovitch and Ben-Haim (2010, pp. 267-8)

Reality Check: 
According to Ben-Haim (2001, 2006, 2010), IGDT is a probability, likelihood, plausibility, chance, belief FREE theory. Hence, its robustness analysis cannot possibly rank decisions according to the likelihood, or chance that they achieve acceptable (satisfactory) outcome in the face of such an uncertainty (except for trivial cases). Furthermore, IGDT cannot possibly express the decision maker's beliefs about uncertain variation of $u$ around $\tilde{u}$.

The interesting this is that there are warnings in the IGDT literature itself against such misapplications of the theory. For instance, consider this text that pre-date the official birth of IGDT :

However, unlike in a probabilistic analysis, $r$ has no connotation of likelihood. We have no rigorous basis for evaluating how likely failure may be; we simply lack the information, and to make a judgment would be deceptive and could be dangerous. There may definitely be a likelihood of failure associated with any given radial tolerance. However, the available information does not allow one to assess this likelihood with any reasonable accuracy.
Ben-Haim (1994, p. 152)

And even the main text on IGDT warns users:

In info-gap set models of uncertainty we concentrate on cluster-thinking rather than on recurrence or likelihood. Given a particular quantum of information, we ask: what is the cloud of possibilities consistent with this information? How does this cloud shrink, expand and shift as our information changes? What is the gap between what is known and what could be known. We have no recurrence information, and we can make no heuristic or lexical judgments of likelihood.
Ben-Haim (2006, p. 18)

Go figure!

IGDT cannot possibly maximize the likelihood of events associated with its uncertainty because IGDT assumes that the uncertainty is likelihood-free.

Next consider this text in the article (colors are used for emphasis):

It does, however, have one major limitation: It considers only local uncertainty in the neighborhood of the best-estimate state of the world. Thus, it is most useful where a reasonable best estimate exists and there is strong reason to believe increasingly large deviations from the best estimate are decreasingly likely (Hayes et al. 2013). While this does not require the uncertainty to be well parameterized, it is a strong assumption nonetheless (Taleb 2005). If the future SOW is no more likely to be within an arbitrary neighborhood of the best estimate than far away from it, info-gap may lead to endorsement of strategies vulnerable to high regret.
Johnson and Geldner (2019, p. 31)

The reference to (Hayes et al. 2013) hides a much stronger criticism of IGDT. Here it suffices to note the following:

Satisficing metrics can also be based on the idea of a radius of stability, which has made a recent resurgence under the label of info-gap decision theory (Ben-Haim, 2004; Herman et al., 2015). Here, one identifies the uncertainty horizon over which a given decision alternative performs satisfactorily. The uncertainty horizon 𝛼 is the distance from a pre-specified reference scenario to the first scenario in which the pre-specified performance threshold is no longer met (Hall et al., 2012; Korteling et al., 2012). However, as these metrics are based on deviations from an expected future scenario, they only assess robustness locally and are therefore not suited to dealing with deep uncertainty (Maier et al., 2016). These metrics also assume that the uncertainty increases at the same rate for all uncertain factors when calculating the uncertainty horizon on a set of axes. Consequently, they are shown in parentheses in Table 2 and will not be considered further in this article.
McPhail et al. (2018, p. 174)

And for a second opinion, they may also wish to consult the article Severe uncertainty and info-gap decision theory (Hayes et al. 2013) where they will find these texts (colors are used for emphasis):

Ecologists and managers contemplating the use of IGDT should carefully consider its strengths and weaknesses, reviewed here, and not turn to it as a default approach in situations of severe uncertainty, irrespective of how this term is defined. We identify four areas of concern for IGDT in practice: sensitivity to initial estimates, localized nature of the analysis, arbitrary error model parameterisation and the ad hoc introduction of notions of plausibility.
Hayes et al. (2013, p. 1)

Sniedovich (2008) bases his arguments on mathematical proofs that may not be accessible to many ecologists but the impact of his analysis is profound. It states that IGDT provides no protection against severe uncertainty and that the use of the method to provide this protection is therefore invalid.
Hayes et al. (2013, p. 2)

The literature and discussion presented in this paper demonstrate that the results of Ben-Haim (2006) are not uncontested. Mathematical work by Sniedovich (2008, 2010a) identifies significant limitations to the analysis. Our analysis highlights a number of other important practical problems that can arise. It is important that future applications of the technique do not simply claim that it deals with severe and unbounded uncertainty but provide logical arguments addressing why the technique would be expected to provide insightful solutions in their particular situation.
Hayes et al. (2013, p. 9)

Plausibility is being evoked within IGDT in an ad hoc manner, and it is incompatible with the theory's core premise, hence any subsequent claims about the wisdom of a particular analysis have no logical foundation. It is therefore difficult to see how they could survive significant scrutiny in real-world problems. In addition, cluttering the discussion of uncertainty analysis techniques with ad hoc methods should be resisted.
Hayes et al. (2013, p. 609)

Next, the issue of the lineage of IGDT in the framework of the state of the art in decision-making under uncertainty. The article assigns a lineage of "1,16" to IGDT. Since the article does not provide any detail on how these figures were determined, it is impossible to say much about them. However, any meaningful lineage analysis of IGDT would have discovered the following two important facts that are not discussed in the article.

  • IGDT's robustness analysis is a simple application of Wald's famous Maximin paradigm (circa 1940).

  • IGDT robustness model is a reinvention of the well known and well established concept Radius of Stability (circa 1962).

Given all of this, it is not clear at all why some DMDU scholars continue to consider IGDT as a candidate for proper treatment of deep uncertainty.

Readers of this review are encouraged to read Review 2-2022.

Bibliography and links

Articles/chapters

Research Reports

Links

Disclaimer: This site, its contents and style, are the responsibility of its owner the author and do not represent the views, policies or opinions of the organizations he is associated/affiliated with.