官术网_书友最值得收藏!

How to do it...

Let's go ahead and implement a random search algorithm with PyTorch:

  1. Import the Gym and PyTorch packages and create an environment instance:
>>> import gym
>>> import torch
>>> env = gym.make('CartPole-v0')
  1. Obtain the dimensions of the observation and action space:
>>> n_state = env.observation_space.shape[0]
>>> n_state
4
>>> n_action = env.action_space.n
>>> n_action
2

These will be used when we define the tensor for the weight matrix, which is size 4 x 2 in size.

  1. Define a function that simulates an episode given the input weight and returns the total reward:
 >>> def run_episode(env, weight):
... state = env.reset()
... total_reward = 0
... is_done = False
... while not is_done:
... state = torch.from_numpy(state).float()
... action = torch.argmax(torch.matmul(state, weight))
... state, reward, is_done, _ = env.step(action.item())
... total_reward += reward
... return total_reward

Here, we convert the state array to a tensor of the float type because we need to compute the multiplication of the state and weight tensor, torch.matmul(state, weight), for linear mapping. The action with the higher value is selected using the torch.argmax() operation. And don't forget to take the value of the resulting action tensor using .item() because it is a one-element tensor.

  1. Specify the number of episodes:
>>> n_episode = 1000
  1. We need to keep track of the best total reward on the fly, as well as the corresponding weight. So, we specify their starting values:
>>> best_total_reward = 0
>>> best_weight = None

We will also record the total reward for every episode:

>>> total_rewards = []
  1. Now, we can run n_episode. For each episode, we do the following:
  • Randomly pick the weight
  • Let the agent take actions according to the linear mapping
  • An episode terminates and returns the total reward
  • Update the best total reward and the best weight if necessary
  • Also, keep a record of the total reward

Put this into code as follows:

 >>> for episode in range(n_episode):
... weight = torch.rand(n_state, n_action)
... total_reward = run_episode(env, weight)
... print('Episode {}: {}'.format(episode+1, total_reward))
... if total_reward > best_total_reward:
... best_weight = weight
... best_total_reward = total_reward
... total_rewards.append(total_reward)
...
Episode 1: 10.0
Episode 2: 73.0
Episode 3: 86.0
Episode 4: 10.0
Episode 5: 11.0
……
……
Episode 996: 200.0
Episode 997: 11.0
Episode 998: 200.0
Episode 999: 200.0
Episode 1000: 9.0

We have obtained the best policy through 1,000 random searches. The best policy is parameterized by best_weight.

  1. Before we test out the best policy in the testing episodes, we can calculate the average total reward achieved by random linear mapping:
 >>> print('Average total reward over {} episode: {}'.format(
n_episode, sum(total_rewards) / n_episode))
Average total reward over 1000 episode: 47.197

This is more than twice what we got from the random action policy (22.25).

  1. Now, let's see how the learned policy performs on 100 new episodes:  
 >>> n_episode_eval = 100
>>> total_rewards_eval = []
>>> for episode in range(n_episode_eval):
... total_reward = run_episode(env, best_weight)
... print('Episode {}: {}'.format(episode+1, total_reward))
... total_rewards_eval.append(total_reward)
...
Episode 1: 200.0
Episode 2: 200.0
Episode 3: 200.0
Episode 4: 200.0
Episode 5: 200.0
……
……
Episode 96: 200.0
Episode 97: 188.0
Episode 98: 200.0
Episode 99: 200.0
Episode 100: 200.0
>>> print('Average total reward over {} episode: {}'.format(
n_episode, sum(total_rewards_eval) / n_episode_eval))
Average total reward over 1000 episode: 196.72

Surprisingly, the average reward for the testing episodes is close to the maximum of 200 steps with the learned policy. Be aware that this value may vary a lot. It could be anywhere from 160 to 200.

主站蜘蛛池模板: 普兰县| 东源县| 大兴区| 县级市| 法库县| 买车| 融水| 大安市| 尚志市| 长岭县| 九龙坡区| 安康市| 无锡市| 舟曲县| 万山特区| 勐海县| 新竹县| 台中县| 罗山县| 东乌珠穆沁旗| 扶风县| 贵阳市| 荆州市| 深圳市| 晋州市| 吉木萨尔县| 辽阳县| 措美县| 浮梁县| 德化县| 巫溪县| 涿州市| 梧州市| 家居| 东宁县| 方正县| 根河市| 温州市| 济宁市| 永寿县| 麻江县|