Efficient reinforcement learning in continuous state and action spaces with Dyna and polic

Efficient reinforcement learning in continuous state and action spaces with Dyna and policy

approximation

Shan ZHONG;Quan LIU;Zongzhang ZHANG;Qiming FU

【期刊名称】《中国高等学校学术文摘·计算机科学》 【年(卷),期】2019(013)001

【摘要】Dyna is an effective reinforcement learning (RL) approach that combines

value

function

evaluation

with

model

learning.However,existing works on Dyna mostly discuss only its efficiency in RL problems with discrete action spaces.This paper proposes a novel Dyna variant,called Dyna-LSTD-PA,aiming to handle problems with continuous action spaces.Dyna-LSTD-PA stands for Dyna based on least-squares temporal difference (LSTD) and policy approximation.Dyna-LSTD-PA consists of two simultaneous,interacting processes.The learning process determines the probability distribution over action spaces using the Gaussian distribution;estimates the underlying value function,policy,and model by linear representation;and updates their parameter vectors online by LSTD(λ).The planning process updates the parameter vector of the value function again by using offline LSTD(λ).Dyna-LSTD-PA also uses the Sherman-Morrison formula to improve the efficiency of LSTD(λ),and weights the parameter vector of

the

value

function

to

bring

the

two

processes