官术网_书友最值得收藏!

  • The Reinforcement Learning Workshop
  • Alessandro Palmas Emanuele Ghelfi Dr. Alexandra Galina Petre Mayur Kulkarni Anand N.S. Quan Nguyen Aritra Sen Anthony So Saikat Basak
  • 424字
  • 2021-06-11 18:37:44

Introduction

In the previous chapter, we studied the main elements of Reinforcement Learning (RL). We described an agent as an entity that can perceive an environment's state and act by modifying the environment state in order to achieve a goal. An agent acts through a policy that represents its behavior, and the way the agent selects an action is based on the environment state. In the second half of the previous chapter, we introduced Gym and Baselines, two Python libraries that simplify the environment representation and the algorithm implementation, respectively.

We mentioned that RL considers problems as Markov Decision Processes (MDPs), without entering into the details and without giving a formal definition.

In this chapter, we will formally describe what an MDP is, its properties, and its characteristics. When facing a new problem in RL, we have to ensure that the problem can be formalized as an MDP; otherwise, applying RL techniques is impossible.

Before presenting a formal definition of MDPs, we need to understand Markov Chains (MCs) and Markov Reward Processes (MRPs). MCs and MRPs are specific cases (simplified) of MDPs. An MC only focuses on state transitions without modeling rewards and actions. Consider the example of the game of snakes and ladders, where the next action is completely dependent on the number displayed on the dice. MRPs also include the reward component in the state transition. MRPs and MCs are useful in understanding the characteristics of MDPs gradually. We will be looking at specific examples of MCs and MRPs later in the chapter.

Along with MDPs, this chapter also presents the concepts of the state-value function and the action-value function, which are used to evaluate how good a state is for an agent and how good an action taken in a given state is. State-value functions and action-value functions are the building blocks of the algorithms used to solve real-world problems. The concepts of state-value functions and action-value functions are highly related to the agent's policy and the environment dynamics, as we will learn later in this chapter.

The final part of this chapter presents two Bellman equations, namely the Bellman expectation equation and the Bellman optimality equation. These equations are helpful in the context of RL in order to evaluate the behavior of an agent and find a policy that maximizes the agent's performance in an MDP.

In this chapter, we will practice with some MDP examples, such as the student MDP and Gridworld. We will implement the solution methods and equations explained in this chapter using Python, SciPy, and NumPy.

主站蜘蛛池模板: 华蓥市| 曲松县| 芮城县| 扶沟县| 建平县| 吉木萨尔县| 桑植县| 莆田市| 柏乡县| 南康市| 景泰县| 保康县| 崇礼县| 巨野县| 天峻县| 郯城县| 南皮县| 固镇县| 兴文县| 根河市| 宣恩县| 平凉市| 阿坝| 南平市| 信宜市| 晋江市| 桂阳县| 香格里拉县| 东港市| 望江县| 五河县| 乌拉特后旗| 宝应县| 青冈县| 泉州市| 营口市| 富平县| 石首市| 密云县| 营山县| 黄冈市|