Information

Here's how it works. If a probabilistic event has outcomes indexed by i, with probabilities p_i, then each occurrence of the event carries the following amount of information, in bits.

H = − sum ( p_i log₂ p_i )

If, for example, there are N equally likely outcomes, then the amount of information is log₂ N.

To see that the definition makes sense, suppose we generate a random number with eight bits in the computer sense, i.e., a random number between 0 and 255, inclusive. In that case, N = 256, and the event carries log₂ 256 = 8 bits of information, as expected.

As another example, let's think about presidential elections. If there are two candidates, both equally likely, then, as I said in The Problem, each vote carries exactly one bit of information. But what if there are more candidates? Say, four? If the candidates were all equally likely, each vote would carry two bits of information … but that's hardly realistic. If, instead, two candidates polled at 48% and the other two at 2%, each vote would carry about 1.12 bits of information … essentially, still just one.

The other thing that's neat about information is that it has a physical meaning. A system in thermal equilibrium is always moving from one (quantum) state to another, and occupies any particular state with probability e^−E/kT, where E is the energy of the state, T is the temperature, and k is Boltzmann's constant. We can compute the information carried by that probability distribution; and if we multiply by Boltzmann's constant, and divide by log₂ e, we get the entropy of the system … the exact same entropy as in the second law of thermodynamics!

In chemistry, I learned that only differences in entropy could be measured, but later, in physics, I think I read otherwise, that there was some situation in which the actual value of the entropy mattered. Unfortunately, I can't remember the details right now.

My favorite reference for such things is Thermal Physics, but of course there are plenty of others.