Model-Based Exploration in Continuous State Spaces Export

@incollection{citeulike:2234457, abstract = {Modern reinforcement learning algorithms effectively exploit experience data sampled from an unknown controlled dynamical system to compute a good control policy, but to obtain the necessary data they typically rely on naive exploration mechansisms or human domain knowledge. Approaches that first learn a model offer improved exploration in finite problems, but discrete model representations do not extend directly to continuous problems. This paper develops a method for approximating continuous models by fitting data to a finite sample of states, leading to finite representations compatible with existing model-based exploration mechanisms. Experiments with the resulting family of fitted-model reinforcement learning algorithms reveals the critical importance of how the continuous model is generalized from finite data. This paper demonstrates instantiations of fitted-model algorithms that lead to faster learning on benchmark problems than contemporary model-free RL algorithms that only apply generalization in estimating action values. Finally, the paper concludes that in continuous problems, the exploration-exploitation tradeoff is better construed as a balance between exploration and generalization.}, author = {Jong, Nicholas and Stone, Peter}, citeulike-article-id = {2234457}, citeulike-linkout-0 = {http://dx.doi.org/10.1007/978-3-540-73580-9_21}, doi = {10.1007/978-3-540-73580-9_21}, journal = {Abstraction, Reformulation, and Approximation}, keywords = {continuous\_space, model-based, reinforcement}, pages = {258--272}, posted-at = {2008-01-15 10:20:15}, priority = {3}, title = {Model-Based Exploration in Continuous State Spaces}, url = {http://dx.doi.org/10.1007/978-3-540-73580-9_21}, year = {2007} }

TY - CHAP ID - citeulike:2234457 L3 - citeulike-article-id:2234457 N2 - Modern reinforcement learning algorithms effectively exploit experience data sampled from an unknown controlled dynamical system to compute a good control policy, but to obtain the necessary data they typically rely on naive exploration mechansisms or human domain knowledge. Approaches that first learn a model offer improved exploration in finite problems, but discrete model representations do not extend directly to continuous problems. This paper develops a method for approximating continuous models by fitting data to a finite sample of states, leading to finite representations compatible with existing model-based exploration mechanisms. Experiments with the resulting family of fitted-model reinforcement learning algorithms reveals the critical importance of how the continuous model is generalized from finite data. This paper demonstrates instantiations of fitted-model algorithms that lead to faster learning on benchmark problems than contemporary model-free RL algorithms that only apply generalization in estimating action values. Finally, the paper concludes that in continuous problems, the exploration-exploitation tradeoff is better construed as a balance between exploration and generalization. JF - Abstraction, Reformulation, and Approximation EP - 272 TI - Model-Based Exploration in Continuous State Spaces SP - 258 KW - continuous_space KW - model-based KW - reinforcement AU - Jong, Nicholas AU - Stone, Peter PY - 2007/// UR - http://dx.doi.org/10.1007/978-3-540-73580-9_21 ER -