Installing Spark

Follow these steps to install Spark 2.3.1, compiled with Hadoop 2.7:

1. If you have a Spark 2.0 tar distribution (for example, spark-2.3.1-bin-hadoop2.7.tgz), then copy it into your Linux VM at any location (for example, /opt) using any Windows on Linux file transfer software (FileZilla or WinSCP). Alternatively, you can download the latest binary .tar.gz file from the following Apache Spark link: http://spark.apache.org/downloads.html.

1. Go to the location where you have copied the Spark software package and uncompress it:

cd /opt
tar -xzvf spark-2.3.1-bin-hadoop2.7.tgz

1. Set the environment variable in .bash_profile, as follows:

nano ~/.bash_profile 

1. Add the following lines to the end of the file:

export SPARK_HOME=/opt/spark-2.3.1-bin-hadoop2.7
export PATH=$PATH:$SPARK_HOME/sbin
export PATH=$PATH:$SPARK_HOME/bin

1. Run the following command to update the environment variables in the current session:

source ~/.bash_profile