- PyTorch 1.x Reinforcement Learning Cookbook
- Yuxi (Hayden) Liu
- 361字
- 2021-06-24 12:34:44
How it works...
In Step 2, we calculated the transition probability after k steps, which is the kth power of the transition matrix. You will see the following output:
>>> print("Transition probability after 2 steps:\n{}".format(T_2))
Transition probability after 2 steps:
tensor([[0.6400, 0.3600],
[0.4800, 0.5200]])
>>> print("Transition probability after 5 steps:\n{}".format(T_5))
Transition probability after 5 steps:
tensor([[0.5670, 0.4330],
[0.5773, 0.4227]])
>>> print(
"Transition probability after 10 steps:\n{}".format(T_10))
Transition probability after 10 steps:
tensor([[0.5715, 0.4285],
[0.5714, 0.4286]])
>>> print(
"Transition probability after 15 steps:\n{}".format(T_15))
Transition probability after 15 steps:
tensor([[0.5714, 0.4286],
[0.5714, 0.4286]])
>>> print(
"Transition probability after 20 steps:\n{}".format(T_20))
Transition probability after 20 steps:
tensor([[0.5714, 0.4286],
[0.5714, 0.4286]])
We can see that, after 10 to 15 steps, the transition probability converges. This means that, no matter what state the process is in, it has the same probability of transitioning to s0 (57.14%) and s1 (42.86%).
In Step 4, we calculated the state distribution after k = 1, 2, 5, 10, 15, and 20 steps, which is the multiplication of the initial state distribution and the transition probability. You can see the results here:
>>> print("Distribution of states after 1 step:\n{}".format(v_1))
Distribution of states after 1 step:
tensor([[0.5200, 0.4800]])
>>> print("Distribution of states after 2 steps:\n{}".format(v_2))
Distribution of states after 2 steps:
tensor([[0.5920, 0.4080]])
>>> print("Distribution of states after 5 steps:\n{}".format(v_5))
Distribution of states after 5 steps:
tensor([[0.5701, 0.4299]])
>>> print(
"Distribution of states after 10 steps:\n{}".format(v_10))
Distribution of states after 10 steps:
tensor([[0.5714, 0.4286]])
>>> print(
"Distribution of states after 15 steps:\n{}".format(v_15))
Distribution of states after 15 steps:
tensor([[0.5714, 0.4286]])
>>> print(
"Distribution of states after 20 steps:\n{}".format(v_20))
Distribution of states after 20 steps:
tensor([[0.5714, 0.4286]])
We can see that, after 10 steps, the state distribution converges. The probability of being in s0 (57.14%) and the probability of being in s1 (42.86%) remain unchanged in the long run.
Starting with [0.7, 0.3], the state distribution after one iteration becomes [0.52, 0.48]. Details of its calculation are illustrated in the following diagram:

After another iteration, the state distribution becomes [0.592, 0.408] as calculated in the following diagram:

As time progresses, the state distribution reaches equilibrium.
- Splunk 7 Essentials(Third Edition)
- Practical Ansible 2
- Visual FoxPro 6.0數(shù)據(jù)庫(kù)與程序設(shè)計(jì)
- 80x86/Pentium微型計(jì)算機(jī)原理及應(yīng)用
- 現(xiàn)代機(jī)械運(yùn)動(dòng)控制技術(shù)
- 機(jī)器學(xué)習(xí)流水線實(shí)戰(zhàn)
- Nginx高性能Web服務(wù)器詳解
- 單片機(jī)技術(shù)一學(xué)就會(huì)
- 工業(yè)機(jī)器人維護(hù)與保養(yǎng)
- Deep Reinforcement Learning Hands-On
- Salesforce for Beginners
- Statistics for Data Science
- 在實(shí)戰(zhàn)中成長(zhǎng):Windows Forms開(kāi)發(fā)之路
- Linux系統(tǒng)管理員工具集
- Linux系統(tǒng)下C程序開(kāi)發(fā)詳解