官术网_书友最值得收藏!

ML workflows and Spark pipelines

In this section, we provide an introduction to machine learning workflows, and also Spark pipelines, and then discuss how Spark pipeline can serve as a good tool of computing ML workflows.

After this section, readers will master these two important concepts, and be ready to program and implement Spark pipelines for machine learning workflows.

ML as a step-by-step workflow

Almost all ML projects involve cleaning data, developing features, estimating models, evaluating models, and then interpreting results, which all can be organized into some step by step workflows. These workflows are sometimes called analytical processes.

Some people even define machine learning as workflows of turning data into actionable insights, for which some people will add business understanding or problem definition into the workflows as their starting points.

In the data mining field, Cross Industry Standard Process for Data Mining (CRISP-DM) is a widely accepted workflow standard, which is still widely adopted. And many standard ML workflows are just some form of revision to the CRISP-DM workflow.

ML as a step-by-step workflow

As illustrated in the above picture, for any standard CRISP-DM workflow, we need all the following 6 steps:

  1. Business understanding
  2. Data understanding
  3. Data preparation
  4. Modeling
  5. Evaluation
  6. Deployment

To which some people may add analytical approaches selection and results explanation, to make it more complete. For complicated machine learning projects, there will be some branches and feedback loops to make workflows very complex.

In other words, for some machine learning projects, after we complete model evaluation, we may go back to the step of modeling or even data preparation. After the data preparation step, we may branch out for more than two types of modeling.

主站蜘蛛池模板: 亳州市| 西乌珠穆沁旗| 丰顺县| 黄冈市| 犍为县| 渭源县| 昔阳县| 图木舒克市| 页游| 西乌珠穆沁旗| 万全县| 泰宁县| 卢湾区| 麻栗坡县| 儋州市| 孟津县| 公主岭市| 含山县| 精河县| 屏南县| 山西省| 西畴县| 宜都市| 治多县| 通许县| 惠来县| 涿鹿县| 阳西县| 姜堰市| 城口县| 元氏县| 淅川县| 毕节市| 青铜峡市| 永胜县| 比如县| 抚顺县| 安图县| 广饶县| 美姑县| 丰宁|