Approximate dynamic programming using Bellman residual elimination and Gaussian process regression Export

@inproceedings{BBethke09, abstract = {This paper presents an approximate policy iteration algorithm for solving infinite-horizon, discounted Markov decision processes (MDPs) for which a model of the system is available. The algorithm is similar in spirit to Bellman residual minimization methods. However, by using Gaussian process regression with nondegenerate kernel functions as the underlying cost-to-go function approximation architecture, the algorithm is able to explicitly construct cost-to-go solutions for which the Bellman residuals are identically zero at a set of chosen sample states. For this reason, we have named our approach Bellman residual elimination (BRE). Since the Bellman residuals are zero at the sample states, our BRE algorithm can be proven to reduce to exact policy iteration in the limit of sampling the entire state space. Furthermore, the algorithm can automatically optimize the choice of any free kernel parameters and provide error bounds on the resulting cost-to-go solution. Computational results on a classic reinforcement learning problem indicate that the algorithm yields a high-quality policy and cost approximation.}, author = {Bethke, B. and How, J. P.}, citeulike-article-id = {6071715}, citeulike-linkout-0 = {http://dx.doi.org/10.1109/ACC.2009.5160344}, citeulike-linkout-1 = {http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=5160344}, day = {10}, doi = {10.1109/ACC.2009.5160344}, journal = {American Control Conference, 2009. ACC '09.}, keywords = {approximate, bellman, dynamic, elimination, gaussian, process, programming, regression, residual}, month = {July}, pages = {745--750}, posted-at = {2009-11-04 22:45:40}, priority = {2}, title = {Approximate dynamic programming using Bellman residual elimination and Gaussian process regression}, url = {http://dx.doi.org/10.1109/ACC.2009.5160344}, year = {2009} }

TY - CONF ID - BBethke09 L3 - citeulike-article-id:6071715 N2 - This paper presents an approximate policy iteration algorithm for solving infinite-horizon, discounted Markov decision processes (MDPs) for which a model of the system is available. The algorithm is similar in spirit to Bellman residual minimization methods. However, by using Gaussian process regression with nondegenerate kernel functions as the underlying cost-to-go function approximation architecture, the algorithm is able to explicitly construct cost-to-go solutions for which the Bellman residuals are identically zero at a set of chosen sample states. For this reason, we have named our approach Bellman residual elimination (BRE). Since the Bellman residuals are zero at the sample states, our BRE algorithm can be proven to reduce to exact policy iteration in the limit of sampling the entire state space. Furthermore, the algorithm can automatically optimize the choice of any free kernel parameters and provide error bounds on the resulting cost-to-go solution. Computational results on a classic reinforcement learning problem indicate that the algorithm yields a high-quality policy and cost approximation. JF - American Control Conference, 2009. ACC '09. EP - 750 TI - Approximate dynamic programming using Bellman residual elimination and Gaussian process regression SP - 745 KW - approximate KW - bellman KW - dynamic KW - elimination KW - gaussian KW - process KW - programming KW - regression KW - residual AU - Bethke, B AU - How, JP PY - 2009/07/10/ UR - http://dx.doi.org/10.1109/ACC.2009.5160344 ER -