Code execution

Let's run this code for h_size of 128, standard deviation of 0.1, and sgd_step of 0.01:

def run(h_size, stddev, sgd_step):
 ...

def main():
run(128,0.1,0.01)

if __name__ == '__main__':
main()

The preceding code outputs the following graph, which plots the steps versus the test and train accuracy:

Let's compare the change in SGD steps and its effect on training accuracy. The following code is very similar to the previous code example, but we will rerun it for multiple SGD steps to see how SGD steps affect accuracy levels.

def run(h_size, stddev, sgd_steps):
    ....
    test_accs = []
    train_accs = []
    time_taken_summary = []
    for sgd_step in sgd_steps:
        start_time = time.time()
        updates_sgd = tf.train.GradientDescentOptimizer(sgd_step).minimize(cost)
        sess = tf.Session()
        init = tf.initialize_all_variables()
        steps = 50
        sess.run(init)
        x  = np.arange(steps)
        test_acc = []
        train_acc = []

        print("Step, train accuracy, test accuracy")


        for step in range(steps):
                # Train with each example
                for i in range(len(train_x)):
                    sess.run(updates_sgd, feed_dict={X: train_x[i: i + 1], 
                              y: train_y[i: i + 1]})

                train_accuracy = np.mean(np.argmax(train_y, axis=1) ==
                                         sess.run(predict, 
                                         feed_dict={X: train_x, y: train_y}))
                test_accuracy = np.mean(np.argmax(test_y, axis=1) ==
                                        sess.run(predict, 
                                        feed_dict={X: test_x, y: test_y}))

                print("%d, %.2f%%, %.2f%%"
                      % (step + 1, 100. * train_accuracy, 100. * test_accuracy))
                #x.append(step)
                test_acc.append(100. * test_accuracy)
                train_acc.append(100. * train_accuracy)
        end_time = time.time()
        diff = end_time -start_time
        time_taken_summary.append((sgd_step,diff))
        t = [np.array(test_acc)]
        t.append(train_acc)
        train_accs.append(train_acc)

Output of the preceding code will be an array with training and test accuracy for each SGD step value. In our example, we called the function sgd_steps for an SGD step value of [0.01, 0.02, 0.03]:

def main():
    sgd_steps = [0.01,0.02,0.03]
    run(128,0.1,sgd_steps)

if __name__ == '__main__':
    main()

This is the plot showing how training accuracy changes with sgd_steps. For an SGD value of 0.03, it reaches a higher accuracy faster as the step size is larger.