書(shū)名： Machine Learning With Go
作者名： Daniel Whitenack
本章字?jǐn)?shù)： 181字
更新時(shí)間： 2021-07-08 10:37:29

Pachyderm jargon

Think about versioning data in Pachyderm kind of like versioning code in Git. The primitives are similar:

Repositories: These are versioned collections of data, similar to having versioned collections of code in Git repositories
Commits: Data is versioned in Pachyderm by making commits of that data into data repositories
Branches: These lightweight points to certain commits or sets of commits (for example, master points to the latest HEAD commit)
Files: Data is versioned at the file level in Pachyderm, and Pachyderm automatically employs strategies, such as de-duplication, to keep your versioned data space efficient

Even though versioning data with Pachyderm feels similar to versioning code with Git, there are some major differences. For example, merging data doesn't exactly make sense. If there are merge conflicts on petabytes of data, no human could resolve these. Furthermore, the Git protocol would not be space efficient in general for large sets of data. Pachyderm uses its own internal logic to perform the versioning and work with versioned data, and the logic is both space efficient and processing efficient in terms of caching.

官术网_书友最值得收藏!

Machine Learning With Go

Pachyderm jargon