- Apache Spark Machine Learning Blueprints
- Alex Liu
- 274字
- 2021-07-16 10:39:49
ML workflows and Spark pipelines
In this section, we provide an introduction to machine learning workflows, and also Spark pipelines, and then discuss how Spark pipeline can serve as a good tool of computing ML workflows.
After this section, readers will master these two important concepts, and be ready to program and implement Spark pipelines for machine learning workflows.
ML as a step-by-step workflow
Almost all ML projects involve cleaning data, developing features, estimating models, evaluating models, and then interpreting results, which all can be organized into some step by step workflows. These workflows are sometimes called analytical processes.
Some people even define machine learning as workflows of turning data into actionable insights, for which some people will add business understanding or problem definition into the workflows as their starting points.
In the data mining field, Cross Industry Standard Process for Data Mining (CRISP-DM) is a widely accepted workflow standard, which is still widely adopted. And many standard ML workflows are just some form of revision to the CRISP-DM workflow.

As illustrated in the above picture, for any standard CRISP-DM workflow, we need all the following 6 steps:
- Business understanding
- Data understanding
- Data preparation
- Modeling
- Evaluation
- Deployment
To which some people may add analytical approaches selection and results explanation, to make it more complete. For complicated machine learning projects, there will be some branches and feedback loops to make workflows very complex.
In other words, for some machine learning projects, after we complete model evaluation, we may go back to the step of modeling or even data preparation. After the data preparation step, we may branch out for more than two types of modeling.
- 后稀缺:自動(dòng)化與未來(lái)工作
- AutoCAD快速入門與工程制圖
- 商戰(zhàn)數(shù)據(jù)挖掘:你需要了解的數(shù)據(jù)科學(xué)與分析思維
- Cloud Analytics with Microsoft Azure
- Hands-On Linux for Architects
- 21天學(xué)通Java
- Visual C++編程全能詞典
- 嵌入式操作系統(tǒng)
- Hands-On Data Warehousing with Azure Data Factory
- Excel 2010函數(shù)與公式速查手冊(cè)
- 單片機(jī)技能與實(shí)訓(xùn)
- 人工智能:語(yǔ)言智能處理
- 自動(dòng)化生產(chǎn)線安裝與調(diào)試(三菱FX系列)(第二版)
- ESP8266 Robotics Projects
- Photoshop CS4數(shù)碼照片處理入門、進(jìn)階與提高