官术网_书友最值得收藏!

  • Spark Cookbook
  • Rishi Yadav
  • 421字
  • 2021-07-16 13:43:57

Installing Spark from binaries

Spark can be either built from the source code or precompiled binaries can be downloaded from http://spark.apache.org. For a standard use case, binaries are good enough, and this recipe will focus on installing Spark using binaries.

Getting ready

All the recipes in this book are developed using Ubuntu Linux but should work fine on any POSIX environment. Spark expects Java to be installed and the JAVA_HOME environment variable to be set.

In Linux/Unix systems, there are certain standards for the location of files and directories, which we are going to follow in this book. The following is a quick cheat sheet:

How to do it...

At the time of writing this, Spark's current version is 1.4. Please check the latest version from Spark's download page at http://spark.apache.org/downloads.html. Binaries are developed with a most recent and stable version of Hadoop. To use a specific version of Hadoop, the recommended approach is to build from sources, which will be covered in the next recipe.

The following are the installation steps:

  1. Open the terminal and download binaries using the following command:
    $ wget http://d3kbcqa49mib13.cloudfront.net/spark-1.4.0-bin-hadoop2.4.tgz
    
  2. Unpack binaries:
    $ tar -zxf spark-1.4.0-bin-hadoop2.4.tgz
    
  3. Rename the folder containing binaries by stripping the version information:
    $ sudo mv spark-1.4.0-bin-hadoop2.4 spark
    
  4. Move the configuration folder to the /etc folder so that it can be made a symbolic link later:
    $ sudo mv spark/conf/* /etc/spark
    
  5. Create your company-specific installation directory under /opt. As the recipes in this book are tested on infoobjects sandbox, we are going to use infoobjects as directory name. Create the /opt/infoobjects directory:
    $ sudo mkdir -p /opt/infoobjects
    
  6. Move the spark directory to /opt/infoobjects as it's an add-on software package:
    $ sudo mv spark /opt/infoobjects/
    
  7. Change the ownership of the spark home directory to root:
    $ sudo chown -R root:root /opt/infoobjects/spark
    
  8. Change permissions of the spark home directory, 0755 = user:read-write-execute group:read-execute world:read-execute:
    $ sudo chmod -R 755 /opt/infoobjects/spark
    
  9. Move to the spark home directory:
    $ cd /opt/infoobjects/spark
    
  10. Create the symbolic link:
    $ sudo ln -s /etc/spark conf
    
  11. Append to PATH in .bashrc:
    $ echo "export PATH=$PATH:/opt/infoobjects/spark/bin" >> /home/hduser/.bashrc
    
  12. Open a new terminal.
  13. Create the log directory in /var:
    $ sudo mkdir -p /var/log/spark
    
  14. Make hduser the owner of the Spark log directory.
    $ sudo chown -R hduser:hduser /var/log/spark
    
  15. Create the Spark tmp directory:
    $ mkdir /tmp/spark
    
  16. Configure Spark with the help of the following command lines:
    $ cd /etc/spark
    $ echo "export HADOOP_CONF_DIR=/opt/infoobjects/hadoop/etc/hadoop" >> spark-env.sh
    $ echo "export YARN_CONF_DIR=/opt/infoobjects/hadoop/etc/Hadoop" >> spark-env.sh
    $ echo "export SPARK_LOG_DIR=/var/log/spark" >> spark-env.sh
    $ echo "export SPARK_WORKER_DIR=/tmp/spark" >> spark-env.sh
    
主站蜘蛛池模板: 平顺县| 马关县| 安陆市| 德钦县| 阳谷县| 康平县| 皋兰县| 通海县| 北宁市| 建阳市| 陕西省| 荣成市| 秦安县| 兴安盟| 五台县| 伊春市| 芜湖市| 闸北区| 那坡县| 遂昌县| 泸水县| 都兰县| 永宁县| 曲沃县| 兴业县| 寿阳县| 常山县| 拜城县| 南涧| 枣阳市| 湖州市| 台北县| 长治市| 甘洛县| 教育| 河西区| 万安县| 阜城县| 贺州市| 华坪县| 东辽县|