官术网_书友最值得收藏!

Downloading and installing Spark 1.4.1

In the following section, we will go through the Spark installation process in detail. Spark is built on Scala and runs on the Java Virtual Machine (JVM). Before installing Spark, you should first have Java Development Kit 7 (JDK) installed on your computer.

Make sure you install JDK instead of Java Runtime Environment (JRE). You can download it from http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html.

Next, download the latest release of Spark from the project website https://spark.apache.org/downloads.html. Perform the following three steps to get Spark installed on your computer:

  1. Select the package type: Pre-built for Hadoop 2.6 and later and then Direct Download. Make sure you choose a prebuilt version for Hadoop instead of the source code.
  2. Download the compressed TAR file called spark-1.4.1-bin-hadoop2.6.tgz and place it into a directory on your computer.
  3. Open the terminal and change to the previous directory. Using the following commands, extract the TAR file, rename the Spark root folder to spark-1.4.1, and then list the installed files and subdirectories:
    tar -xf spark-1.4.1-bin-hadoop2.6.tgz
      mv spark-1.4.1-bin-hadoop2.6 spark-1.4.1
      cd spark-1.4.1
      ls 

That's it! You now have Spark and its libraries installed on your computer. Note the following files and directories in the spark-1.4.1 home folder:

  • core: This directory contains the source code for the core components and API of Spark
  • bin: This directory contains the executable files that are used to submit and deploy Spark applications or also to interact with Spark in a Spark shell
  • graphx, mllib, sql, and streaming: These are Spark libraries that provide a unified interface to do different types of data processing, namely graph processing, machine learning, queries, and stream processing
  • examples: This directory contains demos and examples of Spark applications

It is often convenient to create shortcuts to the Spark home folder and Spark example folders. In Linux or Mac, open or create the ~/.bash_profile file in your home folder and insert the following lines:

export SPARKHOME="/[Where you put Spark]/spark-1.4.1/"
export SPARKSCALAEX="ls ../spark- 1.4.1/examples/src/main/scala/org/apache/spark/examples/"

Then, execute the following command for the previous shortcuts to take effect:

source ~/.bash_profile

As a result, you can quickly access these folders in the terminal or Spark shell. For example, the example named LiveJournalPageRank.scala can be accessed with:

$SPARKSCALAEX/graphx/LiveJournalPageRank.scala
主站蜘蛛池模板: 台州市| 金平| 黄浦区| 兴隆县| 榆社县| 日土县| 海兴县| 华容县| 孙吴县| 桑日县| 吉林省| 鸡泽县| 广元市| 惠安县| 合水县| 盖州市| 璧山县| 衡阳县| 略阳县| 永清县| 台安县| 吉安市| 崇州市| 伊宁市| 库伦旗| 宝应县| 柳州市| 上思县| 全州县| 新丰县| 铜鼓县| 土默特左旗| 资阳市| 云梦县| 五华县| 泰和县| 伊金霍洛旗| 邮箱| 轮台县| 东兰县| 金门县|