官术网_书友最值得收藏!

Q-learning

Q-learning is one of the most used reinforcement learning algorithms. This is due to its ability to compare the expected utility of the available actions without requiring an environment model. Thanks to this technique, it is possible to find an optimal action for every given state in a finished MDP.

A general solution to the reinforcement learning problem is to estimate, thanks to the learning process, an evaluation function. This function must be able to evaluate, through the sum of the rewards, the optimality/utility or otherwise of a particular policy. In fact, Q-learning tries to maximize the value of the Q function (action-value function), which represents the maximum discounted future reward when we perform actions a in the state s.

Q-learning, like SARSA, estimates the function value q (s, a) incrementally, updating the value of the state-action pair at each step of the environment, following the logic of updating the general formula for estimating the values for the TD methods. Q-learning, unlike SARSA, has off-policy characteristics, that is, while the policy is improved according to the values estimated by q (s, a), the value function updates the estimates following a strictly greedy secondary policy: given a state, the chosen action is always the one that maximizes the value max q (s, a). However, the π policy has an important role in estimating values because, through it, the state-action pairs to be visited and updated are determined.

The following is a pseudocode for a Q-learning algorithm:

Initialize
arbitrary action-value function
Repeat (for each episode)
Initialize s
choose a from s using policy from action-value function
Repeat (for each step in episode)
take action a
observe r, s'
update action-value function
update s

Q-learning uses a table to store each state-action pair. At each step, the agent observes the current state of the environment and, using the π policy, selects and executes the action. By executing the action, the agent obtains the reward Rt+1 and the new state St+1. At this point the agent is able to calculate Q(St, at), updating the estimate.

主站蜘蛛池模板: 金山区| 绥德县| 上犹县| 阳新县| 汤阴县| 武城县| 西乌珠穆沁旗| 田林县| 页游| 临沧市| 昭平县| 凌云县| 抚顺市| 平果县| 吉木萨尔县| 平邑县| 罗城| 灵宝市| 博湖县| 咸丰县| 普格县| 建昌县| 远安县| 九龙县| 绿春县| 房产| 西和县| 铁岭县| 连江县| 芦山县| 中宁县| 理塘县| 犍为县| 新密市| 三台县| 雅江县| 五大连池市| 平果县| 汝城县| 南城县| 洮南市|