- Machine Learning With Go
- Daniel Whitenack
- 181字
- 2021-07-08 10:37:29
Pachyderm jargon
Think about versioning data in Pachyderm kind of like versioning code in Git. The primitives are similar:
- Repositories: These are versioned collections of data, similar to having versioned collections of code in Git repositories
- Commits: Data is versioned in Pachyderm by making commits of that data into data repositories
- Branches: These lightweight points to certain commits or sets of commits (for example, master points to the latest HEAD commit)
- Files: Data is versioned at the file level in Pachyderm, and Pachyderm automatically employs strategies, such as de-duplication, to keep your versioned data space efficient
Even though versioning data with Pachyderm feels similar to versioning code with Git, there are some major differences. For example, merging data doesn't exactly make sense. If there are merge conflicts on petabytes of data, no human could resolve these. Furthermore, the Git protocol would not be space efficient in general for large sets of data. Pachyderm uses its own internal logic to perform the versioning and work with versioned data, and the logic is both space efficient and processing efficient in terms of caching.