Accounting for parametric uncertainty in Markov decision processes

Adam W. Schapaugh, Andrew J. Tyre

Research output: Contribution to journalArticle

6 Scopus citations

Abstract

Markov decision processes have become the standard tool for modeling sequential decision-making problems in conservation. In many real-world applications, however, it is practically infeasible to accurately parameterize the state transition function. In this study, we introduce a new way of dealing with ambiguity in the state transition function. In contrast to existing methods, we explore the effects of uncertainty at the level of the policy, rather than at the level of decisions within states. We use information-gap decision theory to ask the question of how much uncertainty in the state transition function can be tolerated while still delivering a specified expected value given by the objective function. Accordingly, the goal of the optimization problem is no longer to maximize expected value, but to maximize local robustness to uncertainty (while still meeting the desired level of performance). We analyze a simple land acquisition problem, using info-gap decision theory to propagate uncertainties and rank alternative policies. Rather than requiring information about the extent of parameter uncertainty at the outset, info-gap addresses the question of how much uncertainty is permissible in the state transition function before the optimal policy would change.

Original languageEnglish (US)
Pages (from-to)15-21
Number of pages7
JournalEcological Modelling
Volume254
DOIs
StatePublished - Apr 10 2013

    Fingerprint

Keywords

  • Information-gap
  • Markov decision process
  • Reserve selection
  • Stochastic dynamic programming
  • Uncertainty

ASJC Scopus subject areas

  • Ecological Modeling

Cite this