Autoregressive models

Autoregressive models estimate the conditional distribution of some data  , given some other values of y. For example, in image synthesis, it estimates the conditional distribution of pixels given surrounding or previous pixels; in audio synthesis, it estimates the conditional distribution of audio samples given previous audio samples and spectrograms.

In its simplest linear form, with dependency on the previous time-step only and time-invariant bias term, an autoregressive model can be defined with the following equation:

is a constant term that represents the model's bias,  represents the model's coefficients, Y_t-1 represents the previous output vector, and is assumed to be white noise. The dependency of the current output on the previous output is explicit in this equation.

Although autoregressive models are sequential in nature, given that the training data is available beforehand, they are normally trained in parallel using the teacher-forcing procedure. In this procedure, the model is not conditioned on its output, but on the real output obtained from the training data.

During inference, the model's output must be used, because we do not have access to the correct output – that is, the model must do autoregression on its own output, hence the name autoregressive model.

Autoregressive models have the advantage of being trained with simple and stable maximum likelihood estimates. This simplicity is counterbalanced by the limited capability of autoregressive models to perform inference in parallel, thus potentially requiring long wait times to generate data.

PixelCNN is one of the most famous autoregressive models for image synthesis. You can refer to the following paper for more details: https://www.semanticscholar.org/paper/Conditional-Image-Generation-with-PixelCNN-Decoders-Oord-Kalchbrenner/8e4ab54564fb492dcae9a1e862aedd3e52fb258b.

In the following figure, we show an image of faces generated with PixelCNN:

WaveNet is one of the most famous autoregressive generative models for audio-synthesis. You can refer to the following paper for more details about WaveNet, available at: https://arxiv.org/pdf/1609.03499.pdf. The following diagram describes WaveNet's graph: