var selected = new List();
while( len(selected) < k ) {
    var newIndex = floor(random() * n);
    if ( items[newIndex] not in selected ) {
        selected.append( items[newIndex] );
    }
}

This may work fine if k is much smaller than n, but when k is large and close to n, this can be disastrously slow. Near the end, most of the items are already selected, so we keep picking items and having to “throw them back”. The alternative of selecting which items to omit (instead of selecting the ones to keep) fails in the same way for small values of k.

Instead, there’s a simple trick that is guaranteed to never require more than n random numbers. As we get to each item, we select it or not with the “correct” probability. Here is the pseudocode:

var selected = new List();
var needed = k;
var available = n;
var position = 0;
while( len(selected) < k ) {
    if( random() < needed / available ) {
        selected.append( items[position] )
        needed -= 1;
    }
    available -= 1;
}

Handy trick… I’ve needed this algorithm quite a few times.

Edward Lorenz had majored in math, but after working as an Air Corp weather forecaster during WWII he found himself working as a meteorologist. In 1961 he was one of the first to seriously apply computer modeling to the prediction of weather. He ran simulations which took hours to calculate, and printed their output every so often. One day he wanted to continue an interrupted run and to save time didn’t want to start the simulation over again from the beginning, so he started it by typing in the numbers from one of the printouts near the end of the previous run.

He didn’t start from the FINAL printout because, of course, there was a danger of typing in a value incorrectly, and he wanted to check against the next couple of printouts in case he’d made a typo. Curiously, the numbers on the subsequent printouts were close, but not exactly right! A normal person would have shrugged their shoulders and ignored it, but Lorenz investigated further.

After some experimentation, he discovered that the discrepancy was due to rounding: he was typing in 0.506 when the computer was storing 0.506127. The result was that the output diverged, slowly at first but eventually giving a completely different result. Here is a sample from his output:

At this point a normal person would have shrugged and just gone back to start from the beginning again, but Lorenz realized there was something deeply wrong about this. The weather measurements that he was working from were not accurate to 6 decimal places — if accuracy at this level mattered, then perhaps all the research he was doing was meaningless! Further investigation showed that he could reproduce the problem, and that it wasn’t due to a error in his program: if there was a bug it lay, not in his program, but in the mathematics behind it!

He tried simplifying his equations, seeking to discover the simplest case that still exhibited the behavior (this is a standard debugging technique used by programmers everywhere). Eventually he got a simple system of just 3 equations that exhibited this property, dubbed “Sensitive Dependence on Initial Conditions”. Under certain circumstances (which apply to weather predictions), tiny variations in input values will grow over time until they completely alter the results of the formula. This discovery was popularized as the “butterfly effect”: a butterfly choosing whether or not to flap its wings could alter weather patterns a few months hence half the world away. This is the reason that computers have gotten hundreds of times faster and bigger, yet weather forecasters still only predict about a week in advance.

Of course, it goes far beyond the weather. Lorenz’s discovery proved to be at the root of an entire field of mathematics (now called “Chaos Theory”) which encompassed many oddities from strange attractors to fractals.

There are several lessons in this story. You have to be willing to question your fundamental assumptions: a tiny difference in the last decimal point turned out to contain a discovery which undermined his entire field of study… and opened up a new one. There’s also a lesson about computers: they can be very helpful, but too much trust can lead you astray. Invalid input, invalid assumptions, invalid equations, odd mathematical chaos… there are many pitfalls. And finally, an experiment that fails may in the end be even more useful than one that succeeds.

So the other day, I had a problem to solve. I had two circles (the initial problem was in 2D, but the solution works for spheres in any number of dimensions). The circles were moving in what I can assume are straight lines, and I want to know when (if ever) they will intersect.

Start with some definitions. I’ll specify the circles with initial position and velocity (vectors, as you can tell from the little arrows over them) and radius. Then each one’s position (position of the center) can be expressed in a simple vector equation as a function of time:

Now here’s where it gets clever. I make a change of frame (a trick I learned from many Physics problems). I’ll consider it from the point of view of circle 1. From that point of view, circle 1 is sitting at the origin, not moving, while the position and velocity of circle 2 can be expressed as a new moving circle which (for lack of imagination) I will call circle 3. While I’m at it, I’ll give circle 3 a radius equal to the sum of that of circles 1 and 2:

Now the problem is reduced to a SINGLE moving circle and we want to know when (if ever) it will touch the origin. Express distance (of the edge of the circle) from the origin and start solving. Double vertical bars is length (of a vector) and other places use the dot product of vectors.

Okay, I’ve simplified that some. Now let’s solve for when it touches. That would be when d(t) = 0.

It’s actually not very messy, but we can make it incredibly neat by defining a few terms:

Giving us this form:

Oh, I think I recognize that one. In fact, I have the solution memorized:

Q.E.D.

I was 13 at the time, and wise to the ways of the world and of math, so I knew immediately that this person was trying to put something over on me. Oh sure, this messy-looking equation might have been the solution to some particular (complicated) homework problem, or it may even have been an intermediate step on the way to a more elegant solution, but there was NO WAY that something this “messy-looking” was actually a meaningful and important piece of mathematics. I’d been going to school for over 8 years now, and every piece of mathematical insight I had seen had been strikingly simple and elegant, never a complicated mess like this.

Well, live and learn I suppose.

The quadratic equation is, of course, the solution to a fundamental question, and it has no other form I know of which is much simpler. The world is just a bit messier than I had believed in those sheltered days.

Since then I have come to believe the following:

(1) The “messy” is far more common than the “simple” or “elegant”. Just as nearly-all functions are not expressible in closed form and nearly-all real numbers are irrational, nearly all problems have very difficult or messy solutions. I’d love to be able to state this precisely, perhaps even prove it, but I’m not quite sure how to go about it.

(2) An astonishingly large portion of the really important and interesting problems have simple and elegant answers. The Navier-Stokes equation is the exception – most things we care about CAN be solved simply. Why this should be is a truly deep question.