build method

The first method we will define for our SimpleCNN class is the build method, which is responsible for building the architecture of our CNN. Our build method takes two pieces of input: the input tensor and the number of classes it should expect:

def build(self, input_tensor, num_classes):
    """
    Builds a convolutional neural network according to the input shape and the number of classes.
    Architecture is fixed.

    Args:
        input_tensor: Tensor of the input
        num_classes: (int) number of classes

    Returns:
        The output logits before softmax
    """

We will first initialize tf.placeholder, called is_training. TensorFlow placeholders are like variables that don't have values. We only pass them values when we actually train the network and call the relevant operations:

with tf.name_scope("input_placeholders"):
    self.is_training = tf.placeholder_with_default(True, shape=(), name="is_training")

The tf.name_scope(...) block allows us to name our operations and tensors properly. While this is not absolutely necessary, it helps us organize our code better and will help us to visualize the network. Here, we define a tf.placeholder_with_default called is_training, which has a default value of True. This placeholder will be used for our dropout operations (since dropout has different modes during training and inference).

Our next step is to define the convolutional layers of our CNN. We make use of three different kinds of layers to create multiple layers of convolutions: tf.layers.conv2d, tf.max_pooling2d, and tf.layers.dropout:

with tf.name_scope("convolutional_layers"):
    conv_1 = tf.layers.conv2d(
        input_tensor,
        filters=16,
        kernel_size=(5, 5),
        strides=(1, 1),
        padding="SAME",
        activation=tf.nn.relu,
        kernel_regularizer=tf.contrib.layers.l2_regularizer(scale=self.beta),
        name="conv_1")
    conv_2 = tf.layers.conv2d(
        conv_1,
        filters=32,
        kernel_size=(3, 3),
        strides=(1, 1),
        padding="SAME",
        activation=tf.nn.relu,
        kernel_regularizer=tf.contrib.layers.l2_regularizer(scale=self.beta),
        name="conv_2")
    pool_3 = tf.layers.max_pooling2d(
        conv_2,
        pool_size=(2, 2),
        strides=1,
        padding="SAME",
        name="pool_3"
    )
    drop_4 = tf.layers.dropout(pool_3, training=self.is_training, name="drop_4")

    conv_5 = tf.layers.conv2d(
        drop_4,
        filters=64,
        kernel_size=(3, 3),
        strides=(1, 1),
        padding="SAME",
        activation=tf.nn.relu,
        kernel_regularizer=tf.contrib.layers.l2_regularizer(scale=self.beta),
        name="conv_5")
    conv_6 = tf.layers.conv2d(
        conv_5,
        filters=128,
        kernel_size=(3, 3),
        strides=(1, 1),
        padding="SAME",
        activation=tf.nn.relu,
        kernel_regularizer=tf.contrib.layers.l2_regularizer(scale=self.beta),
        name="conv_6")
    pool_7 = tf.layers.max_pooling2d(
        conv_6,
        pool_size=(2, 2),
        strides=1,
        padding="SAME",
        name="pool_7"
    )
    drop_8 = tf.layers.dropout(pool_7, training=self.is_training, name="drop_8")

In the preceding table, we have specified the convolutional architecture to have the following sequence of layers:

CONV | CONV | POOL | DROPOUT | CONV | CONV | POOL | DROPOUT

However, you are encouraged to explore different configurations and architectures. For example, you could add batch-normalization layers to improve the stability of training.

Finally, we add the fully-connected layers that lead to the output of the network:

with tf.name_scope("fully_connected_layers"):
    flattened = tf.layers.flatten(drop_8, name="flatten")
    fc_9 = tf.layers.dense(
        flattened,
        units=1024,
        activation=tf.nn.relu,
        kernel_regularizer=tf.contrib.layers.l2_regularizer(scale=self.beta),
        name="fc_9"
    )
    drop_10 = tf.layers.dropout(fc_9, training=self.is_training, name="drop_10")
    logits = tf.layers.dense(
        drop_10,
        units=num_classes,
        kernel_regularizer=tf.contrib.layers.l2_regularizer(scale=self.beta),
        name="logits"
    )

return logits

tf.layers.flatten turns the output of the convolutional layers (which is 3-D) into a single vector (1-D) so that we can pass them through the tf.layers.dense layers. After going through two fully-connected layers, we return the final output, which we define as logits.

Notice that in the final tf.layers.dense layer, we do not specify an activation. We will see why when we move on to specifying the training operations of the network.

Next, we implement several helper functions. _create_tf_dataset takes two instances of numpy.ndarray and turns them into TensorFlow tensors, which can be directly fed into a network. _log_loss_and_acc simply logs training statistics, such as loss and accuracy:

def _create_tf_dataset(self, x, y):
    dataset = tf.data.Dataset.zip((
            tf.data.Dataset.from_tensor_slices(x),
            tf.data.Dataset.from_tensor_slices(y)
        )).shuffle(50).repeat().batch(self.batch_size)
    return dataset

def _log_loss_and_acc(self, epoch, loss, acc, suffix):
    summary = tf.Summary(value=[
        tf.Summary.Value(tag="loss_{}".format(suffix), simple_value=float(loss)),
        tf.Summary.Value(tag="acc_{}".format(suffix), simple_value=float(acc))
    ])
    self.summary_writer.add_summary(summary, epoch)