$400M in assets to bankrupt in 45-minutes because of a failed deployment (2014)

I'm not sure how automated deployments would have solved this problem. In fact, if anything, it would have magnified the impact and fallout of the problem. Substitute "a developer forgot to upload the code to one of the servers" for "the deployment agent errored while downloading the new binary/code onto the server and a bug in the agent prevented the error from being surfaced." Now you have the same failure mode, and the impact happens even faster. The blame here…

I once shut down an algorithmic trading server by hastily typing (in bash): - Ctrl-r for reverse-search through history - typing 'ps' to find the process status utility (of course) - pressing Enter,....and realizing that Ctrl-r actually found 'stopserver.sh' in history instead. (There's a ps inside stoPServer.sh) I got a call from the head Sales Trader within 5 seconds asking why his GUI showed that all Orders were paused. Luckily, our recovery code was robust and I could restart the…

Amusing personal anecdote -- the Knight debacle caused the market in general to tumble. The week before a coworker of mine -- sure of a market drop but for other reasons -- had bought a raft or puts on the S&P 500. When I saw looking glum at work after the Knight news broke, I asked him what was wrong didn't you make a ton? Yeah, he said, but I can't get out cause my account's with Knight.

This story puts the lie to a couple of canards about HFT: - "It's risk free." Any time you put headless trading code into the market you are risking a catastrophic loss. That risk can be managed to a degree with many layers of programmatic safeties, and other practices like having your operations people look for warning emails the day after you've deployed new code. But the risk is always present. - "It makes the market more unstable." The most…