Select Lab Publications


Efficient Solution Algorithms for Factored MDPs (2003)

By: Carlos Guestrin, Daphne Koller, Ronald Parr, and Shobha Venkataraman

Abstract: This paper addresses the problem of planning under uncertainty in large Markov Decision Processes (MDPs). Factored MDPs represent a complex state space using state variables and the transition model using a dynamic Bayesian network. This representation often allows an exponential reduction in the representation size of structured MDPs, but the complexity of exact solution algorithms for such MDPs can grow exponentially in the representation size. In this paper, we present two approximate solution algorithms that exploit structure in factored MDPs. Both use an approximate value function represented as a linear combination of basis functions, where each basis function involves only a small subset of the domain variables. A key contribution of this paper is that it shows how the basic operations of both algorithms can be performed efficiently in closed form, by exploiting both additive and context-specific structure in a factored MDP. A central element of our algorithms is a novel linear program decomposition technique, analogous to variable elimination in Bayesian networks, which reduces an exponentially large LP to a provably equivalent, polynomial-sized one.

One algorithm uses approximate linear programming, and the second approximate dynamic programming. Our dynamic programming algorithm is novel in that it uses an approximation based on max-norm, a technique that more directly minimizes the terms that appear in error bounds for approximate MDP algorithms. We provide experimental results on problems with over 10^{40} states, demonstrating a promising indication of the scalability of our approach, and compare our algorithm to an existing state-of-the-art approach, showing, in some pr oblems, exponential gains in computation time.



Download Information
Carlos Guestrin, Daphne Koller, Ronald Parr, and Shobha Venkataraman (2003). "Efficient Solution Algorithms for Factored MDPs." Journal of Artificial Intelligence Research (JAIR), 19, 399-468. Winner of the 2007 IJCAI-JAIR Best Paper Prize. pdf            
BibTeX citation

@article{Guestrin+al:jair2003factoredmdps,
title = {Efficient Solution Algorithms for Factored MDPs},
author = {Carlos Guestrin and Daphne Koller and Ronald Parr and Shobha Venkataraman},
journal = {Journal of Artificial Intelligence Research (JAIR)},
year = {2003},
volume = {19},
pages = {399-468},
wwwfilebase = {jair2003-guestrin-koller-parr-venkataraman},
wwwtopic = {Factored MDPs},
wwwaward = {Winner of the 2007 IJCAI-JAIR Best Paper Prize}
}



full list