How referencing objects work – namespaces

Key 1: Interrelations between objects.

The scope is the visibility of a name within a code block. Namespace is mapping from names to objects. Namespaces are important in order to maintain localization and avoid name collision. Every module has a global namespace. Modules store mapping from variable name to objects in their __dict__ attribute, which is a normal Python dictionary along with information to reload it, package information, and so on.

Every module's global namespace has an implicit reference to the built-in module; hence, objects that are in the built-in module are always available. We can also import other modules in the main script. When we use the syntax import module name, a mapping with module name to module object is created in the global namespace of the current module. For import statements with syntax such as import modname as modrename, mapping is created with a new name to module object.

We are always in the __main__ module's global namespace when the program starts, as it is the module that imports all others. When we import a variable from another module, only an entry is created for that variable in the global namespace pointing at the referenced object. Now interestingly, if this variable references a function object, and if this function uses a global variable, then this variable will be searched in the global namespace of the module that the function was defined in, not in the module that we imported this function to. This is possible because functions have the __globals__ attribute that points to its __dict__ modules, or in short, its modules namespace.

All modules that are loaded and referenced are cached in sys.modules. All imported modules are names pointing to objects in sys.modules. Let's define a new module like this with the name new.py:

k = 10 
def foo():
    print(k)

By importing this module in the interactive session, we can see how global namespaces work. When this module is reloaded, its namespace dictionary is updated, not recreated. Hence, if you attach anything new from the outside of the module to it, it will survive reload:

>>> import importlib
>>> import new
>>> from new import foo
>>> import sys
>>> foo()
10
>>> new.foo()
10
>>> foo.__globals__ is sys.modules['new'].__dict__ # dictionary used by namespace and function attribute __globals__ is indeed same
True
>>> foo.__globals__['k'] = 20  # changing global namespace dictionary
>>> new.do   #attribute is not defined in the module
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'new' has no attribute 'do'
>>> foo.__globals__['do'] = 22 #we are attaching attribute to module from outside the module
>>> new.do
22
>>> foo()  # we get updated value for global variable
20
>>> new.foo()
20
>>> importlib.reload(new) #reload repopulates old modules dictionary
<module 'new' from '/tmp/new.py'>
>>> new.do #it didn't got updated as it was populated from outside.
22
>>> new.foo() #variables updated by execution of code in module are updated
10
>>>

If we use the functions that are defined in different modules to compose a class on runtime, such as using metaclasses, or class decorators, this can bring up surprises as each function could be using a different global namespace.

Locals are simple and they work in the way that you expect. Each function call gets its own copy of variables. Nonlocal variables make variables that are defined in the outer scope (not global namespace) accessible to the current code block. In the following code example, we can see how variables can be referenced in enclosed functions.

Code blocks are able to reference variables that are defined in enclosing scopes. Hence, if a variable is not defined in a function but in an enclosing function, we are able to get its value. If, after referencing a variable in an outer scope, we assign a value to this variable in a code block, it will confuse the interpreter in finding the right variable, and we will get the value from the current local scope. If we assign a value to the variable, it defaults to the local variable. We can specify that we want to work with an enclosing variable using a nonlocal keyword:

>>> #variable in enclosing scope can be referenced any level deep
... 
>>> def f1():
...     v1 = "ohm"
...     def f2():
...         print("f2",v1)
...         def f3():
...             print("f3",v1)
...         f3()
...     f2()
... 
>>> f1()
f2 ohm
f3 ohm
>>> 
>>> #variable can be made non-local (variable in outer scopes) skipping one level of enclosing scope
... 
>>> def f1():
...     v1 = "ohm"
...     def f2():
...         print("f2",v1)
...         def f3():
...             nonlocal v1
...             v1 = "mho"
...             print("f3",v1)
...         f3()
...         print("f2",v1)
...     f2()
...     print("f1",v1)
... 
>>> f1()
f2 ohm
f3 mho
f2 mho
f1 mho
>>> 
>>> 
>>> #global can be specified at any level of enclosed function
... 
>>> v2 = "joule"
>>> 
>>> def f1():
...     def f2():
...         def f3():
...             global v2
...             v2 = "mho"
...             print("f3",v2)
...         f3()
...         print("f2",v2)
...     f2()
...     print("f1",v2)
... 
>>> f1()
f3 mho
f2 mho
f1 mho

As variables are searched without any dictionary lookup for the local namespace, it is faster to look up variables inside a function with a small number of variables than to search in a global namespace. On similar lines, we will get a little speed boost if we pull objects that are referenced in loops in a function's local namespace inside a function block:

In [6]: def fun():
   ...:     localsum = sum
   ...:     return localsum(localsum((a,a+1)) for a in range(1000))
   ...: 

In [8]: def fun2():
   ...:     return sum(sum((a,a+1)) for a in range(1000))
   ...: 

In [9]: %timeit fun2()
1000 loops, best of 3: 1.07 ms per loop

In [11]: %timeit fun()
1000 loops, best of 3: 983 μs per loop