HN Hall of Fame Weekly email

I test in prod

increment.com Essays & writing Essays & articles Software engineering Candidate
Screenshot of increment.com captured 2026-07-20
Page preview · captured 2026-07-20

Resurfaced independently across 3 calendar years, with breakout response in 2 of them.

submissions
7
submitters
6
observed span
2019–2021
peak thread · 156 comments
301 pts
latest 20+ return · 2021-11-23
301 pts

Submission timeline

2007–2026

One slot for every year since HN launched. Height is that year's peak points; orange marks a 100+ point or 50+ comment breakout. Select a bar to open its strongest thread.

First comments on top threads

HN comment order

After reading a few smug comments, I've concluded that some of the folks in this thread have never worked on an application where the production scale is many, many orders of magnitude greater than preproduction environments. There's no substitution for "testing in prod." If you're not "testing in prod" with your really-big application, you're not testing enough.

So it's a fair point that the fidelity of non-prod environments are inherently limited and you still need a bunch of other stuff like canaries, automated canary analysis, automated rollbacks, zone fault-tolerance, feature-flags, chaos engineering, server and client-side instrumentation, but this is generally 'next-level' stuff when most shops aren't even getting the basics right. For the overwhelming majority of non-prod environments and release processes I've seen they usually have basic problems and would catch…

The first top-level comment from each of the four biggest threads, in HN’s own order. Excerpts are shortened; open a comment for full context.

Breakout years
2

100+ points or 50+ comments

Total points
512

reference only — not used in Hall rules or ranking

Total comments
279

reference only — not used in Hall rules or ranking

Every submission

DateTitle as submittedByPointsComments
2019-08-28I Test in Prodhenrik_w20
2019-08-28I Test in Productioncodesuki20
2019-08-31I test in prod (2019)jrvarela5630
2019-09-04I Test in Prodngaut50
2019-11-09I Test in ProdSparksZilla10
2020-07-19All of us test in production all the time (2019)First breakoutjrvarela56198156
2021-11-23I test in prodBest thread · Latest 20+ point returnrsie_above301123