官术网_书友最值得收藏!

Inside the box

So, you have downloaded the latest version of Spark (depending on how you plan on launching Spark) and you have run the standard Hello, World! example....what now?!

Spark comes equipped with five libraries, which can be used separately--or in unison--depending on the task we are trying to solve. Note that in this book, we plan on using a variety of different libraries, all within the same application so that you will have the maximum exposure to the Spark platform and better understand the benefits (and limitations) of each library. These five libraries are as follows:

  • Core: This is the Spark core infrastructure, providing primitives to represent and store data called Resilient Distributed Dataset (RDDs) and manipulate data with tasks and jobs.
  • SQL : This library provides user-friendly API over core RDDs by introducing DataFrames and SQL to manipulate with the data stored.
  • MLlib (Machine Learning Library) : This is Spark's very own machine learning library of algorithms developed in-house that can be used within your Spark application.
  • Graphx : This is used for graphs and graph-calculations; we will explore this particular library in depth in a later chapter.
  • Streaming : This library allows real-time streaming of data from various sources, such as Kafka, Twitter, Flume, and TCP sockets, to name a few. Many of the applications we will build in this book will leverage the MLlib and Streaming libraries to build our applications.

The Spark platform can also be extended by third-party packages. There are many of them, for example, support for reading CSV or Avro files, integration with Redshift, and Sparkling Water, which encapsulates the H2O machine learning library.

主站蜘蛛池模板: 蛟河市| 馆陶县| 长治市| 桃源县| 钦州市| 祁阳县| 治县。| 锡林浩特市| 辽阳市| 南京市| 海晏县| 禹州市| 巴彦淖尔市| 洛隆县| 舞阳县| 保山市| 衢州市| 合肥市| 吉安县| 东乡族自治县| 宝鸡市| 高台县| 通辽市| 洞头县| 和平县| 凤翔县| 灵山县| 鹤峰县| 古丈县| 兴山县| 伊吾县| 台山市| 丰镇市| 嘉峪关市| 普定县| 招远市| 泾川县| 徐闻县| 招远市| 永福县| 台州市|