官术网_书友最值得收藏!

Inside the box

So, you have downloaded the latest version of Spark (depending on how you plan on launching Spark) and you have run the standard Hello, World! example....what now?!

Spark comes equipped with five libraries, which can be used separately--or in unison--depending on the task we are trying to solve. Note that in this book, we plan on using a variety of different libraries, all within the same application so that you will have the maximum exposure to the Spark platform and better understand the benefits (and limitations) of each library. These five libraries are as follows:

  • Core: This is the Spark core infrastructure, providing primitives to represent and store data called Resilient Distributed Dataset (RDDs) and manipulate data with tasks and jobs.
  • SQL : This library provides user-friendly API over core RDDs by introducing DataFrames and SQL to manipulate with the data stored.
  • MLlib (Machine Learning Library) : This is Spark's very own machine learning library of algorithms developed in-house that can be used within your Spark application.
  • Graphx : This is used for graphs and graph-calculations; we will explore this particular library in depth in a later chapter.
  • Streaming : This library allows real-time streaming of data from various sources, such as Kafka, Twitter, Flume, and TCP sockets, to name a few. Many of the applications we will build in this book will leverage the MLlib and Streaming libraries to build our applications.

The Spark platform can also be extended by third-party packages. There are many of them, for example, support for reading CSV or Avro files, integration with Redshift, and Sparkling Water, which encapsulates the H2O machine learning library.

主站蜘蛛池模板: 南岸区| 应城市| 舞钢市| 博乐市| 太康县| 巴彦淖尔市| 鄂伦春自治旗| 锡林郭勒盟| 泽库县| 沅陵县| 棋牌| 白沙| 盱眙县| 神农架林区| 莱芜市| 太康县| 余庆县| 秦皇岛市| 广饶县| 百色市| 广河县| 东乡县| 从江县| 会东县| 红河县| 拉萨市| 禄丰县| 中江县| 安龙县| 平舆县| 七台河市| 宜宾县| 黔西县| 桐柏县| 贺州市| 宁都县| 江北区| 五常市| 昌平区| 威远县| 高邑县|