The TensorFlow graph

One of the more important and powerful features of TensorFlow is its graph. When you define one of the three types of TensorFlow data structures previously described, you automatically add a node and an edge to your graph. Nodes represent operations and edges represent tensors, so if we were to do basic multiplication such as the preceding example, const1 and const2 would represent edges in the graph, tf.multiply would represent a node, and product would represent an outgoing edge from that node. TensorFlow's graph is static, which means we cannot change it at runtime.

Remember, an ANN performs hundreds of computations; computing and interpreting at each step would be extremely compute-intensive. The TensorFlow graph helps solve this problem by creating a graph of all tensors and operations in your network and executing them in your runtime session that we previously described. Adjacent operations are compiled together in the graph for faster computation. On top of that, the graph structure allows us to easily distribute computational tasks across multiple CPUs or GPUs, and the TensorFlow session object automatically manages access to these multiple resources, whether local or in the cloud. This is where TensorFlow gets it flow; data and operations flow through the graph at runtime. 

You can either manually define graphs in TensorFlow, or use it's default graph. When we declared the variables precedingly, we declared them in the default graph that TensorFlow sets up for us when we import it. You can access that default graph with: 

default_graph = tf.get_default_graph()

If we're creating multiple models or processes in the same file however, it's necessary to create multiple graphs. For that, TensorFlow allows us to simply declare a new one:

my_graph = tf.Graph()

with new_graph.as_default():
            x = tf.constant(2)
            y = tf.constant(2)

If you are manually defining multiple graphs, you'll also want to remember to pass those graphs when running a TensorFlow session: 

sess = tf.Session(graph=my_graph)