The SQL abstract syntax tree

As explained earlier, it doesn't matter if you are using SQL, DataFrame, or Dataset, the Apache Spark SQL parser returns an abstract syntax tree. However, DataFrames or Datasets can be used as starting points. The result of all these methods is again a tree-based structure called ULEP. The following figure is an example of such an AST (abstract syntax tree) adding an attribute coming from a row in a table and two literals together:

So as you can see, at the bottom of the tree (those nodes are also called leaves since they don't have any further nodes connecting to them) we have two integer literals: one and two. On top we have an operation taking those two literals and adding them together. You should note that those literals could also be loaded from a persistent data store. The Add operation virtually turns into another literal (three in this case), which then again is used by another Add operation as one of its inputs. The other input labeled as Attribute(x) now takes the value of x of that particular relation and exposes it as materialized value to the Add operator. So in other words, this AST does nothing else than add the number three to each of the values in column x.