官术网_书友最值得收藏!

  • The Reinforcement Learning Workshop
  • Alessandro Palmas Emanuele Ghelfi Dr. Alexandra Galina Petre Mayur Kulkarni Anand N.S. Quan Nguyen Aritra Sen Anthony So Saikat Basak
  • 424字
  • 2021-06-11 18:37:44

Introduction

In the previous chapter, we studied the main elements of Reinforcement Learning (RL). We described an agent as an entity that can perceive an environment's state and act by modifying the environment state in order to achieve a goal. An agent acts through a policy that represents its behavior, and the way the agent selects an action is based on the environment state. In the second half of the previous chapter, we introduced Gym and Baselines, two Python libraries that simplify the environment representation and the algorithm implementation, respectively.

We mentioned that RL considers problems as Markov Decision Processes (MDPs), without entering into the details and without giving a formal definition.

In this chapter, we will formally describe what an MDP is, its properties, and its characteristics. When facing a new problem in RL, we have to ensure that the problem can be formalized as an MDP; otherwise, applying RL techniques is impossible.

Before presenting a formal definition of MDPs, we need to understand Markov Chains (MCs) and Markov Reward Processes (MRPs). MCs and MRPs are specific cases (simplified) of MDPs. An MC only focuses on state transitions without modeling rewards and actions. Consider the example of the game of snakes and ladders, where the next action is completely dependent on the number displayed on the dice. MRPs also include the reward component in the state transition. MRPs and MCs are useful in understanding the characteristics of MDPs gradually. We will be looking at specific examples of MCs and MRPs later in the chapter.

Along with MDPs, this chapter also presents the concepts of the state-value function and the action-value function, which are used to evaluate how good a state is for an agent and how good an action taken in a given state is. State-value functions and action-value functions are the building blocks of the algorithms used to solve real-world problems. The concepts of state-value functions and action-value functions are highly related to the agent's policy and the environment dynamics, as we will learn later in this chapter.

The final part of this chapter presents two Bellman equations, namely the Bellman expectation equation and the Bellman optimality equation. These equations are helpful in the context of RL in order to evaluate the behavior of an agent and find a policy that maximizes the agent's performance in an MDP.

In this chapter, we will practice with some MDP examples, such as the student MDP and Gridworld. We will implement the solution methods and equations explained in this chapter using Python, SciPy, and NumPy.

主站蜘蛛池模板: 巴楚县| 台北市| 彰化市| 陆河县| 青浦区| 宜川县| 南京市| 峨眉山市| 龙里县| 巴中市| 保山市| 无极县| 会东县| 凉城县| 马山县| 合肥市| 黔西县| 如皋市| 京山县| 长乐市| 泊头市| 海晏县| 奉化市| 开封市| 确山县| 西峡县| 吉林市| 嘉善县| 元朗区| 方正县| 霍山县| 天津市| 海淀区| 泰兴市| 穆棱市| 南江县| 南通市| 玉屏| 丹江口市| 连云港市| 凭祥市|