官术网_书友最值得收藏!

ML workflows and Spark pipelines

In this section, we provide an introduction to machine learning workflows, and also Spark pipelines, and then discuss how Spark pipeline can serve as a good tool of computing ML workflows.

After this section, readers will master these two important concepts, and be ready to program and implement Spark pipelines for machine learning workflows.

ML as a step-by-step workflow

Almost all ML projects involve cleaning data, developing features, estimating models, evaluating models, and then interpreting results, which all can be organized into some step by step workflows. These workflows are sometimes called analytical processes.

Some people even define machine learning as workflows of turning data into actionable insights, for which some people will add business understanding or problem definition into the workflows as their starting points.

In the data mining field, Cross Industry Standard Process for Data Mining (CRISP-DM) is a widely accepted workflow standard, which is still widely adopted. And many standard ML workflows are just some form of revision to the CRISP-DM workflow.

ML as a step-by-step workflow

As illustrated in the above picture, for any standard CRISP-DM workflow, we need all the following 6 steps:

  1. Business understanding
  2. Data understanding
  3. Data preparation
  4. Modeling
  5. Evaluation
  6. Deployment

To which some people may add analytical approaches selection and results explanation, to make it more complete. For complicated machine learning projects, there will be some branches and feedback loops to make workflows very complex.

In other words, for some machine learning projects, after we complete model evaluation, we may go back to the step of modeling or even data preparation. After the data preparation step, we may branch out for more than two types of modeling.

主站蜘蛛池模板: 凤城市| 滨州市| 毕节市| 张家川| 鄂温| 资源县| 翁牛特旗| 承德县| 万盛区| 白城市| 沙雅县| 南江县| 阿巴嘎旗| 井冈山市| 禹城市| 比如县| 农安县| 泾川县| 沽源县| 溧水县| 邵阳市| 尼勒克县| 福建省| 平武县| 赤水市| 天柱县| 山阳县| 库车县| 礼泉县| 清水河县| 乌拉特前旗| 三门县| 农安县| 武乡县| 大石桥市| 修水县| 尼勒克县| 施甸县| 靖西县| 宣恩县| 永康市|