Historical averaging

Historical averaging is an approach that takes the average of the parameters in the past and adds this to the respective cost functions of the generator and the discriminator network. It was proposed by Ian Goodfellow and others in a paper mentioned previously, Improved Techniques for Training GANs.

The historical average can be denoted as follows:

In the preceding equation,  is the value of parameters at a particular time, i. This approach can improve the training stability of GANs too.