Scalability, but at what cost? (2015)

I've been doing data analysis and machine learning client work for quite some time now and for companies as small as a 3 person startup to advising a department of the Canadian government. Almost always numpy matrix math + cython or C or Java on a single machine is enough. Not always-always; but if you can relax requirements slighly say by accepting a 45 minute lag from new data impacting the total model, or by caching the results of the…

The GraphChi paper from OSDI 2012 made a similar observation: http://select.cs.cmu.edu/publications/paperdir/osdi2012-kyro... From the abstract: "In this work, we present GraphChi, a disk-based system for computing efficiently on graphs with billions of edges. By using a well-known method to break large graphs into small parts, and a novel parallel sliding windows method, GraphChi is able to execute several advanced data mining, graph mining, and machine learning algorithms on very large graphs, using just a…

A flip side of this is that cloud systems can mask performance problems in code by making it seem fast. Badly written code running in a cloud service might seem performant, because those services will magically cover for you by spinning up additional resources. Were the same issues to happen on your own network, you'd be able to address them. Distributed or cloud based systems should be viewed as a tool that can be pulled out to solve specific problems…

I think about this article (and others that came before it) a lot, at least the guidelines given at the end - If you are going to use a big data system for yourself, see if it is faster than your laptop. - If you are going to build a big data system for others, see that it is faster than my laptop. "Faster than your laptop" requires well-written implementation for the laptop, which I wonder - will that be…