Files Are Fraught with Peril

I did work on backup solutions and supported Dropbox, and I think that the author used Dropbox as a reference to support his own concerns about filesystems that are not related in any case to the Dropbox case. From the filesystem point of view, I don't think that Dropbox is so much concerned about you losing your data because of corruption. As, anyway they are supposed to be safe and versioned in the cloud. And I think that their app…

> Pillai et al., OSDI’14 looked at a bunch of software that writes to files, including things we'd hope write to files safely, like databases and version control systems: Leveldb, LMDB, GDBM, HSQLDB, Sqlite, PostgreSQL, Git, Mercurial, HDFS, Zookeeper. They then wrote a static analysis tool that can find incorrect usage of the file API, things like incorrectly assuming that operations that aren't atomic are actually atomic, incorrectly assuming that operations that can be re-ordered will execute in program order…

This situation is so terrible, and its embarrassing that despite decades of work, our industry doesn't seem to be able to solve this problem in any modern operating system. (Edit: Apparently except for windows vista!) It seems obvious to me that we need a transactional filesystem API with methods like: fbegin() fwrite() / rename / etc fcommit() Most of the write calls wouldn't return errors directly. Then the fcommit() call would atomically flush the locally buffered changes to the filesystem…

Always glad to see another Dan Luu article. Also props to Sqlite for being one of the last ones standing when it comes to fighting filesystem crashes. It is truly an incredibly high quality piece of software AND it's free, what a world to live in.