Cores that don’t count [pdf]

I worked on this problem for the past year at Google. It's a fascinating problem. In my subarea I focused on accelerators (like GPUs) running machine learning training. Many users report problems like "NaN" during training- at some point, the gradients blow up and the job crashes. Sometimes these are caused by specific examples, or numerical errors on the part of the model developer, but sometimes, they are the result of errors from bad cores (during matrix multiplication, embedding lookup…

This is about unstable cores that randomly output incorrect calculation and ways to mitigate it via better hardware testing and duplicating parts of the core that can fail often. I did however thought initially from the title that it's about 1-bit CPUs like the MC14500B Industrial Control Unit (ICU) which is a CMOS one-bit microprocessor designed by Motorola for simple control applications in 1977. It completely lacks an ALU so essentially cannot count, but is designed for PLCs.

Video presentation giving an overview https://www.youtube.com/watch?v=QMF3rqhjYuM

"A deterministic AES mis-computation, which was “self-inverting”: encrypting and decrypting on the same core yielded the identity function, but decryption elsewhere yielded gibberish." Imagine debugging this.