Learn how to design large-scale systems

I'm quite tired of everyone wanting to build "large scale systems" and play at being Netflix. The truth of the matter is the vast vast majority of people will never need to do this with their project and instead will just end up making an expensive to maintain mess with way too many moving parts. At least as important as designing something that can scale up is designing something that can scale down. You don't know when the organization's going…

I thought this was going to be about system engineering, but it's just about web scale distributed computing. Disappointing, because I'm digging into systems engineering at the moment and got a bunch of books on it for Christmas, plus thumbing through the SEBoK wiki [1] and associated content. 1: https://www.sebokwiki.org/wiki/Guide_to_the_Systems_Engineer...

I had a look at the "design a crawler" part, since that's something I have experience with. It makes some pretty sketchy assumptions and weird design choices. First in the back-of-the-envelope calculation it overestimates the size of a document by about two orders of magnitude. It's closer to 7.5 KB than 500 KB on average, uncompressed. 500 KB is closer to the size of Homer's Odyssey, which I will state is much longer than most websites. I'm willing to give…

Pretty useful for design interviews and also a general overview. For a more in depth look would recommend the https://dataintensive.net/ book. It's a great deep dive into modern distributed computing and data architectures.