官术网_书友最值得收藏!

Training and evaluation in Amazon ML

In the context of Amazon ML, the model is linear regression and the algorithm the Stochastic Gradient Descent (SGD) algorithm. This algorithm has one main meta parameter called the learning rate and often noted , which dictates how much of a new sample is taken into account for each iterative update of the weights. A larger learning rate makes the algorithm converge faster but stabilizes further from the optimal weights, while a smaller learning induces a slower convergence but a more precise set of regression coefficients.

Given a training and a validation dataset, this is how Amazon ML tunes and select the best model:

  • Amazon trains several models, each with a different learning rate
  • For a given a learning rate:
    • The training dataset allows the SGD to train the model by finding the best regression coefficients
    • The model is used on the validation dataset to make predictions
  • By comparing the quality of the predictions of the different models on that validation set, Amazon ML is able to select the best model and the associated best learning rate
  • The held-out set is used as final confirmation that the model is reliable

Usual splitting ratios for the training, validation, and held-out subsets are as follows:

  • Training : 70% validation and held-out 15% each
  • Training : 60% validation and held-out 20% each

Shuffling: It is important to make sure that the predictors and the outcome follow the same distribution in all three subsets. Shuffling the data before splitting it is an important part of creating reliable training, validation, and held-out subsets.

It is important to define the data transformations on the training dataset and apply the transformation parameters on the validation and held-out subsets so that the validation and held-out subsets do not leak information back in the training set.

Take standardization as an example: the standard deviation and the mean of the predictors should be calculated on the training dataset. And these values then applied to standardize the validation and held-out sets. If you use the whole original dataset to calculate the mean and SGD, you leak information from the held-out set into the training set.

A common Supervised Predictive Analytics workflow follows these steps - Let's assume we have an already extracted dataset and that we have chosen a metric to assess the quality of our predictions:

  1. Building the dataset
    • Cleaning up and transforming the data to handle noisy data issues
    • Creating new predictors
    • Shuffling and splitting the data into a training, a validation and a held-out set
  2. Selecting the best model
    • Choosing a model (linear, tree-based, bayesian , ...)
    • Repeat for several values of the meta parameters:
      • Train the model on the training set
      • Assess the model performance on the validation set
  3. Repeat steps 1 and 2 with new data, new predictors, and other model parameters until you are satisfied with the performances of your model. Keep the best model.
  4. Final test of the model on the held-out subset.

In the context of Amazon ML, there is no possibility to choose a model (step 2) other than a linear regression one (logistic regression for classification).

主站蜘蛛池模板: 桓仁| 昌乐县| 卫辉市| 玛纳斯县| 西城区| 定兴县| 定西市| 元谋县| 固阳县| 若羌县| 旌德县| 阿尔山市| 贵南县| 文昌市| 朝阳县| 栾城县| 达孜县| 津市市| 壶关县| 淄博市| 石渠县| 福海县| 获嘉县| 灵石县| 峡江县| 黔江区| 黄龙县| 建平县| 文安县| 南昌市| 德安县| 濮阳县| 福州市| 定襄县| 宁陕县| 枣阳市| 莱西市| 黑龙江省| 攀枝花市| 称多县| 青州市|