Why does a bicycle stand up while rolling and fall down as soon as onward motion ceases?

For a plausible explanation, this and the section on balance here are a good start, but I like the following better:

The bottom of the wheel can have no side motion because it rests on the ground; and since the bottom is constantly becoming the top and the top the bottom, if the upper part of the wheel gets any lateral motion, it is checked by being brought round upon the ground again before the motion has too much influence.

That's taken from "Cycling Art, Energy and Locomotion. A series of remarks on the development of bicycles, tricycles and man motor carriages" by Robert P. Scott (1889). It can be found here. FWIW Mr Scott doesn't seem to find the explanation that I quoted very plausible either. He puts the magic down to the gyroscopic effect of spinning wheels. That's what I was originally taught, but that explanation seems to be discredited these days. The truth seems to be that we learn to steer so that the points where the tyres touch the road stay under the centre of gravity. That's what this is designed to do.

Perhaps what is really astonishing is that people have been building and riding these things for so long without really understanding how they work.