A visual proof that neural nets can approximate any function

Approximate, not compute. The function also must be continuous. NNs are good for approximation / interpolation / extrapolation, which makes them quite useful for certain domains of problems. But of course, it does not make them a kind of universal computing machine (in the computability sense, like universal Turing machines).

There are many caveats to this, esp. that this "fact" has nothing to do with whether training a neural network on a dataset will be useful. There is often no function to find in solving a problem, ie., there is no mapping from ImageSpace -> DogCatSpace. Ie., most things are genuine ambiguitites --- a stick in a water appears bent, indistinguishably from an actually bent stick in some other transparent fluid. Animals solve the problem of the "ambiguity of inference"…

Can't the same be said for Fourier series, which make no claims to be some kind of AI? And likewise humble polynomials: http://en.wikipedia.org/wiki/Stone%E2%80%93Weierstrass_theor...

> Every continuous function in the function space can be represented as a linear combination of basis functions https://en.wikipedia.org/wiki/Basis_function This is basic and obvious math. Does slapping the word 'neural' magically make obvious results 1000% more interesting? Why? Because the word 'neural' carries some of that artificial-intelligence-technology-of-the-future cachet?