Data Version Control

I've used DVC for most of my projects for the past five years. The good things is that it works a lot like git. If your scientists understand branches, commits and diffs, they should be able to understand DVC. The bad thing is that it works like git. Scientists often do not, in fact, understand or use branches, commits and diffs. The best thing is that it essentially forces you to follow Ten Simple Rules for Reproducible Computational Research [1]…

DVC has had the following problems, when I tested it (half a year ago): I gets super slow (waiting minutes) when there are a few thousand files tracked. Thousands files have to be tracked, if you have e.g. a 10GB file per day and region and artifacts generated from it. You are encouraged (it only can track artifacts) if you model your pipeline in DVC (think like make). However, it cannot run tasks it parallel. So it takes a lot…

In the intro video they say it’s “based on git but supports large files”. Are they using Git LFS [1] or did they make something else? And what is their proposed value add over using git directly? Edit: They say a little more about the large file stuff > DVC keeps metafiles in Git instead of Google Docs to describe and version control your data sets and models. DVC supports a variety of external storage types as a remote cache…

Be warned: if you star any of their repos they will find your email, add you to their mailing list and spam you with announcements. Happened to me. Email looks like this: Hi XXXX, I'm Dmitry Petrov - creator of the open-source project DVC Since you were interested in DVC ... spam, spam, spam