The order of execution and lazy loading

The nodes in a computation graph are executed in their order of dependency. If node x depends on node y, then x is executed before y when the execution of y is requested. A node is only executed if either the node itself or another node depending on it is invoked for execution. This execution philosophy is known as lazy loading. As the name implies, the node objects are not instantiated and initialized until they are actually required.

Often, it is necessary to control the order of the execution of the nodes in a computation graph. This can be done with the tf.Graph.control_dependencies() function. For example, if the graph has the nodes l, m, n, and o, and we want to execute n and o before l and m, then we would use the following code:

with graph_variable.control_dependencies([n,o]):
  # other statements here

This makes sure that any node in the preceding with block is executed after nodes n and o have been executed.