User-defined functions

In order to create user-defined functions in Scala, we need to examine our data in the previous Dataset. We will use the age property on the client entries in the previously introduced client.json. We plan to create an UDF that will enumerate the age column. This will be useful if we need to use the data for machine learning as a lesser number of different values is sometimes useful. This process is also called binning or categorization. This is the JSON file with the age property added:

Now let's define a Scala enumeration that converts ages into age range codes. If we use this enumeration among all our relations, we can ensure consistent and proper coding of these ranges:

 object AgeRange extends Enumeration {
    val Zero, Ten, Twenty, Thirty, Fourty, Fifty, Sixty, Seventy, Eighty, Ninety, HundretPlus = Value
    def getAgeRange(age: Integer) = {
      age match {
        case age if 0 until 10 contains age => Zero
        case age if 11 until 20 contains age => Ten
        case age if 21 until 30 contains age => Twenty
        case age if 31 until 40 contains age => Thirty
        case age if 41 until 50 contains age => Fourty
        case age if 51 until 60 contains age => Fifty
        case age if 61 until 70 contains age => Sixty
        case age if 71 until 80 contains age => Seventy
        case age if 81 until 90 contains age => Eighty
        case age if 91 until 100 contains age => Ninety
        case _ => HundretPlus
      }
    }
    def asString(age: Integer) = getAgeRange(age).toString
  }

We can now register this function using SparkSession in Scala so that it can be used in a SQL statement:

The newly registered function called toAgeRange can now be used in the select statement. It takes age as a parameter and returns a string for the age range: