Installing Spark from binaries

Spark can be either built from the source code or precompiled binaries can be downloaded from http://spark.apache.org. For a standard use case, binaries are good enough, and this recipe will focus on installing Spark using binaries.

Getting ready

All the recipes in this book are developed using Ubuntu Linux but should work fine on any POSIX environment. Spark expects Java to be installed and the JAVA_HOME environment variable to be set.

In Linux/Unix systems, there are certain standards for the location of files and directories, which we are going to follow in this book. The following is a quick cheat sheet:

How to do it...

At the time of writing this, Spark's current version is 1.4. Please check the latest version from Spark's download page at http://spark.apache.org/downloads.html. Binaries are developed with a most recent and stable version of Hadoop. To use a specific version of Hadoop, the recommended approach is to build from sources, which will be covered in the next recipe.

The following are the installation steps:

1. Open the terminal and download binaries using the following command:

   $ wget http://d3kbcqa49mib13.cloudfront.net/spark-1.4.0-bin-hadoop2.4.tgz
   

2. Unpack binaries:

   $ tar -zxf spark-1.4.0-bin-hadoop2.4.tgz
   

3. Rename the folder containing binaries by stripping the version information:

   $ sudo mv spark-1.4.0-bin-hadoop2.4 spark
   

4. Move the configuration folder to the /etc folder so that it can be made a symbolic link later:

   $ sudo mv spark/conf/* /etc/spark
   

5. Create your company-specific installation directory under /opt. As the recipes in this book are tested on infoobjects sandbox, we are going to use infoobjects as directory name. Create the /opt/infoobjects directory:

   $ sudo mkdir -p /opt/infoobjects
   

6. Move the spark directory to /opt/infoobjects as it's an add-on software package:

   $ sudo mv spark /opt/infoobjects/
   

7. Change the ownership of the spark home directory to root:

   $ sudo chown -R root:root /opt/infoobjects/spark
   

8. Change permissions of the spark home directory, 0755 = user:read-write-execute group:read-execute world:read-execute:

   $ sudo chmod -R 755 /opt/infoobjects/spark
   

9. Move to the spark home directory:

   $ cd /opt/infoobjects/spark
   

10. Create the symbolic link:

    $ sudo ln -s /etc/spark conf
    

11. Append to PATH in .bashrc:

    $ echo "export PATH=$PATH:/opt/infoobjects/spark/bin" >> /home/hduser/.bashrc
    

12. Open a new terminal.
13. Create the log directory in /var:

    $ sudo mkdir -p /var/log/spark
    

14. Make hduser the owner of the Spark log directory.

    $ sudo chown -R hduser:hduser /var/log/spark
    

15. Create the Spark tmp directory:

    $ mkdir /tmp/spark
    

16. Configure Spark with the help of the following command lines:

    $ cd /etc/spark
    $ echo "export HADOOP_CONF_DIR=/opt/infoobjects/hadoop/etc/hadoop" >> spark-env.sh
    $ echo "export YARN_CONF_DIR=/opt/infoobjects/hadoop/etc/Hadoop" >> spark-env.sh
    $ echo "export SPARK_LOG_DIR=/var/log/spark" >> spark-env.sh
    $ echo "export SPARK_WORKER_DIR=/tmp/spark" >> spark-env.sh