The Dataset API in action

We conclude on Datasets with a final aggregation example using the relational Dataset API. Note that we now have an additional choice of methods inspired by RDDs. So we can mix in the map function known from RDDs as follows:

val dsNew = ds.filter(r => {r.age >= 18}).
      map(c => (c.age, c.countryCode)).
      groupBy($"_2").
      avg()

Let's understand how this works step by step:

1. This basically takes the Dataset and filters it to rows containing clients with ages over 18.
2. Then, from the client object c, we only take the age and countryCode columns. This process is again a projection and could have been done using the select method. The map method is only used here to show the capabilities of using lambda functions in conjunction with Datasets without directly touching the underlying RDD.
3. Now, we group by countryCode. We are using the so-called Catalyst (DSL Domain Specific Language) in the groupBy method to actually refer to the second element of the tuple that we created in the previous step.
4. Finally, we average on the groups that we previously created--basically averaging the age per country.

The result is a new strongly typed Dataset containing the average age for adults by country:

Now we have a quite complete picture of all the first-class citizens of ApacheSparkSQL, as shown in the following figure:

This basically shows that RDD is still the central data processing API where everything else builds on top. DataFrames allow for structured data APIs whereas Datasets bring it to the top with statically-typed domain objects, limited to Scala and Java. Both APIs are usable with SQL or a relational API as we can also run SQL queries against Datasets, as the following example illustrates:

This gives us some idea of the SQL-based functionality within Apache Spark, but what if we find that the method that needed is not available? Perhaps we need a new function. This is where user-defined functions (UDFs) are useful. We will cover them in the next section.