官术网_书友最值得收藏!

SARSA

The State-action-reward-state-action (SARSA) algorithm implements an on-policy time differences method, in which the update of the action-value function is performed based on the outcome of the transition from state s to state s' through action a, based on a selected policy π (s, a).

There are policies that always choose the action that provides the maximum reward and non-deterministic policies (ε-greedy, ε-soft, softmax), which ensure an element of exploration in the learning phase.

Greedy is a term used to represent a family of algorithms trying to get a global solution, through excellent local choices.

In SARSA, it is necessary to estimate the action-value function q (s, a), because the total value of a state v (s) (value function) is not sufficient in the absence of an environment model to allow the policy to determine, given a state, which action is best performed. In this case, however, the values are estimated step by step following the Bellman equation with the update parameter v (s), considering, however, in place of a state, the state-action pair.

Being of an on-policy nature, SARSA estimates the action-value function based on the behavior of the π policy, and at the same time modifies the greedy behavior of the policy with respect to the updated estimates from the action-value function. The convergence of SARSA, and more generally of all TD methods, depends on the nature of policies.

The following is a pseudocode for the SARSA algorithm:

Initialize
arbitrary action-value function
Repeat (for each episode)
Initialize s
choose a from s using policy from action-value function
Repeat (for each step in episode)
take action a
observe r, s'
choose a' from s' using policy from action-value function
update action-value function
update s,a

The update rule of the action-value function uses all five elements (st, at, rt + 1, st + 1, at + 1); for this reason, it is called SARSA.

主站蜘蛛池模板: 卢湾区| 环江| 呼图壁县| 手游| 翼城县| 海兴县| 靖安县| 巴楚县| 赫章县| 汝州市| 三明市| 牡丹江市| 金堂县| 泰顺县| 桑日县| 虎林市| 谷城县| 赞皇县| 新津县| 夹江县| 酉阳| 通海县| 宝应县| 边坝县| 石家庄市| 郑州市| 康马县| 雅安市| 咸丰县| 龙海市| 建阳市| 泰兴市| 襄垣县| 景泰县| 奇台县| 盐池县| 榆社县| 体育| 玛曲县| 象州县| 兴山县|