官术网_书友最值得收藏!

Spark data frame

In Apache Spark, a Dataset is a distributed collection of data. The Dataset is a new interface added since Spark 1.6. It provides the benefits of RDDs with the benefits of Spark SQL's execution engine. A Dataset can be constructed from JVM objects and then manipulated using functional transformations (map, flatMap, filter, and so on). The Dataset API is available only for in Scala and Java. It is not available for Python or R.

A DataFrame is a dataset with named columns. It is equivalent to a table in a relational database or a data frame in R/Python, with richer optimizations. DataFrame is constructed from structured data files, tables in Hive, external databases, or existing RDDs. The DataFrame API is available in Scala, Python, Java, and R.

A Spark DataFrame needs the Spark session instantiated first:

import org.apache.spark.sql.SparkSession 
val spark = SparkSession.builder().appName("Spark SQL").config("spark.some.config.option", "").getOrCreate()
import spark.implicits._

Next, we create a DataFrame from a Json file using the spark.read.json function:

scala> val df = spark.read.json("/home/ubuntu/work/ml-resources
/spark-ml/Chapter_01/data/example_one.json")

Note that Spark Implicits are being used to implicitly convert RDD to Data Frame types:

org.apache.spark.sql
Class SparkSession.implicits$
Object org.apache.spark.sql.SQLImplicits
Enclosing class: SparkSession

Implicit methods available in Scala for converting common Scala objects into DataFrames.

Output will be similar to the following listing:

df: org.apache.spark.sql.DataFrame = [address: struct<city: 
string, state: string>, name: string]

Now we want to see how this is actually loaded in the DataFrame:

scala> df.show
+-----------------+-------+
| address| name|
+-----------------+-------+
| [Columbus,Ohio]| Yin|
|[null,California]|Michael|
+-----------------+-------+
主站蜘蛛池模板: 宁德市| 和平县| 冕宁县| 七台河市| 钦州市| 香港 | 馆陶县| 册亨县| 永清县| 永和县| 克拉玛依市| 陆川县| 宿松县| 密山市| 通化县| 普定县| 宜城市| 铜陵市| 阿拉善盟| 阿城市| 新乡县| 农安县| 永城市| 林甸县| 贵港市| 云龙县| 耒阳市| 武安市| 华阴市| 麻栗坡县| 囊谦县| 丰宁| 波密县| 崇信县| 乳山市| 福州市| 禄劝| 玛多县| 呼和浩特市| 革吉县| 平谷区|