Off-Policy Natural Actor-Critic Export

@techreport{citeulike:2335006, abstract = {Recently-developed Natural Actor-Critic (NAC), which employs natural policy gradient learning for the actor and LSTD-Q(.LAMBDA.) for the critic, has provided a good model-free reinforcement learning scheme applicable to high-dimensional systems. Since NAC is an on-policy learning method, however, past sample sequences cannot be reused for estimating the policy gradient under current policy. Moreover, the control of exploration and exploitation has a large constraint on introducing an exploratory factor. To overcome these problems, we propose an off-policy NAC in this study, in which the policy gradient is estimated by using past system trajectories, and the exploration can be controlled from the outside of the policy optimization. Computer experiments using a snake-like robot simulator show our new method is so effective that the number of required trajectories is much smaller than that by the on-policy method. (author abst.)}, author = {Takeshi, Mori}, citeulike-article-id = {2335006}, keywords = {actor-critic, natural, off-policy}, posted-at = {2008-02-05 13:51:52}, priority = {0}, title = {Off-Policy Natural Actor-Critic}, year = {2005} }

TY - RPRT ID - citeulike:2335006 L3 - citeulike-article-id:2335006 N2 - Recently-developed Natural Actor-Critic (NAC), which employs natural policy gradient learning for the actor and LSTD-Q(.LAMBDA.) for the critic, has provided a good model-free reinforcement learning scheme applicable to high-dimensional systems. Since NAC is an on-policy learning method, however, past sample sequences cannot be reused for estimating the policy gradient under current policy. Moreover, the control of exploration and exploitation has a large constraint on introducing an exploratory factor. To overcome these problems, we propose an off-policy NAC in this study, in which the policy gradient is estimated by using past system trajectories, and the exploration can be controlled from the outside of the policy optimization. Computer experiments using a snake-like robot simulator show our new method is so effective that the number of required trajectories is much smaller than that by the on-policy method. (author abst.) TI - Off-Policy Natural Actor-Critic KW - actor-critic KW - natural KW - off-policy AU - Takeshi, Mori PY - 2005/// ER -