Queues don't fix overload (2014)

First, Little's Law[1] can be mentioned as a formalized version of the "red arrow" in the blog post. It's true you cannot increase subsystem capacity (peak throughput) by adding queues. However, queues can "fix" an overload in one sense by making an engineering tradeoff of increased latency and additional complexity (e.g. SPOF[2]). Peak capacity handling didn't increase but overall server utilization can be maximized because jobs waiting in a queue will eventually soak up resources running at less than 100%…

What queues do is smooth out the mismatch between supply and demand. If the mismatch lasts long enough, the queue will overflow, and then you need to load shed (and you need to plan for what the least bad way of load shedding is). But queues do increase overall throughput, up to a point. If the demand was bursty on short timescales and you only allow a small queue to build before load-shedding, you may be wasting capacity. Increasing the…

Back in the 90s. I was part of a team working R&D project on flexible manufacturing. One of the central concepts was the use of buffers (also called decouplers) between manufacturing stations. These buffers were the space needed to place not finished products while they waited for the next station. Typically a conveyor belt. The main service is that both stations could run at slightly different rhythm, and they would not be causing problems on the other one. Either starving…