Control processes

An MDP is itself a type of problem called a control process. Broadly speaking, a control process is designed to optimize a value or a set of values within a set of limitations. The following diagram shows a model for a generalized control process:

A real-world example of such an optimization process might be maximizing a company's profit by knowing its gross sales and variable expenses. We might want to keep expenses as low as possible without affecting the sales, and keep the sales as high as possible on a scale that we can manage without incurring too many new expenses. 

An MDP models a set of states, a set of actions that can be taken from each state, and the outcome of taking each action. As the agent making the decisions in an MDP, part of the outcome of each action that we take is under our control, and part of it is random and unknown to us. 

We will not go too deeply into the mathematical details of MDPs, but this information is worth digging into if you want a better understanding of how stochastic processes can be modeled in the context of Bayesian statistics, such as in Monte Carlo problems.

We will link to some useful resources on the mathematics of MDPs at the end of the chapter. For now, it will be sufficient to get an intuitive and visual understanding of them.