In the field of information theory, probability and entropy are not just related—they are two sides of the same coin. Understanding this deep connection is essential for grasping how information is measured, transmitted, and processed in everything from digital communications to machine learning. At its heart, information theory provides a mathematical framework for quantifying uncertainty, and both probability and entropy play starring roles. This article explores their relationship, the mathematics that ties them together, and the practical implications that have shaped modern technology.

The Role of Probability in Information Theory

Probability is the bedrock upon which information theory is built. In this context, probability refers to the likelihood of a specific event, symbol, or message occurring within a set of possible outcomes. For example, consider a binary source that produces bits—0s and 1s. If the source is fair, the probability of a 0 is 0.5 and the probability of a 1 is also 0.5. If the source is biased, those probabilities shift, say to 0.9 for 0 and 0.1 for 1.

A random variable is often used to model such sources. The probability distribution of a random variable assigns a probability to each possible outcome. In information theory, these outcomes are the symbols or messages that a source might emit. The key insight is that the amount of information carried by an event is inversely related to its probability: a rare event carries more information than a common one. For instance, hearing that a normally reliable alarm clock failed to go off is more surprising—and thus carries more information—than hearing that it worked as expected.

This inverse relationship between probability and information content is the foundation of entropy. The less likely an event is, the greater the surprise, and the more bits needed to encode that event efficiently. Thus, probability directly determines the informational weight of each possible outcome.

Understanding Entropy as a Measure of Uncertainty

Entropy, introduced by Claude Shannon in his landmark 1948 paper A Mathematical Theory of Communication, quantifies the average uncertainty—or information content—of a random variable. More precisely, entropy measures the average number of bits required to represent each symbol from a source, assuming optimal encoding. It is a single number that summarizes the "surprise" of an entire probability distribution.

For a discrete random variable X with possible outcomes x₁, x₂, ..., xₙ and probabilities p(x₁), p(x₂), ..., p(xₙ), the entropy H(X) is defined as the expected value of the information content. The self-information of a single outcome x is -log₂ p(x) bits, where the base-2 logarithm gives the amount of information in bits. The entropy is then the weighted average of these self-informations:

H(X) = -∑ p(x) log₂ p(x)

Several important properties follow:

  • Entropy is always non-negative. A deterministic source (where one outcome has probability 1) has zero entropy because there is no uncertainty.
  • Entropy is maximized when all outcomes are equally likely. For example, a fair coin yields 1 bit of entropy per toss, while a biased coin yields less.
  • Entropy is additive for independent random variables, meaning the total entropy of two independent sources equals the sum of their individual entropies.

Entropy thus serves as a fundamental limit on how much a message can be compressed. No lossless compression scheme can compress data to fewer bits than its entropy, on average. This makes entropy the gold standard for measuring information content.

The Mathematical Connection: Shannon's Formula

The formula H = -∑ p(x) log₂ p(x) explicitly ties entropy to probabilities. Each term -p(x) log₂ p(x) represents the contribution of outcome x to the average uncertainty. The logarithm scales the self-information appropriately: less probable events contribute more, but they also occur less often, so the product p(x) log₂ p(x) balances rarity against frequency.

Consider a simple example with two outcomes. Let the probability of outcome A be p and of outcome B be 1-p. The entropy is:

H = -p log₂ p - (1-p) log₂ (1-p)

This function (the binary entropy function) peaks at p = 0.5 with a value of 1 bit, and drops symmetrically to 0 as p approaches 0 or 1. When p = 0.9, the entropy is roughly 0.469 bits, reflecting the lower uncertainty of a biased source.

The Shannon formula can be extended to handle continuous random variables via differential entropy, though that requires careful treatment of units. For discrete variables, the base of the logarithm can be changed, affecting the numerical value but not the conceptual relationship. Base 2 yields bits, base e yields nats, and base 10 yields dits (hartleys).

The deep insight is that entropy is not an arbitrary measure; it emerges uniquely from the axioms that any reasonable measure of information should satisfy, and those axioms revolve around probabilities. Shannon proved that the only function satisfying additivity, monotonicity, and continuity constraints is of the form -K ∑ p(x) log p(x), where K is a constant. Thus, the link between probability and entropy is both necessary and fundamental.

Conditional Entropy and Mutual Information

The connection between probability and entropy extends beyond the simple formula. Conditional entropy H(Y|X) measures the remaining uncertainty about Y given that we already know X. It is defined as the weighted average of entropies over all possible values of X:

H(Y|X) = -∑ p(x) ∑ p(y|x) log₂ p(y|x)

Here, p(y|x) is the conditional probability of Y given X. This shows how knowledge of one variable reduces uncertainty about another. Mutual information I(X;Y) is then the reduction in uncertainty: I(X;Y) = H(Y) - H(Y|X). These derived quantities are all probability-based and form the backbone of modern communication theory.

Similarly, Kullback-Leibler (KL) divergence measures the "distance" between two probability distributions and relates to cross-entropy—a concept widely used in machine learning for loss functions. Cross-entropy H(P, Q) = -∑ p(x) log₂ q(x) measures the average number of bits needed to encode events from a true distribution P using a code optimized for a model distribution Q. Minimizing cross-entropy is equivalent to making Q as close to P as possible, directly linking probability estimation to entropy minimization.

Implications for Data Compression

One of the most immediate practical applications of the probability–entropy relationship is data compression. Lossless compression algorithms such as Huffman coding, arithmetic coding, and Lempel-Ziv variants all rely on accurate estimates of symbol probabilities to assign shorter codewords to more frequent symbols and longer ones to less frequent symbols. Shannon's source coding theorem states that the expected length of any optimal code for a source is bounded below by its entropy. In other words, entropy provides the theoretical limit of compressibility.

For example, in text compression, the probability of each letter in English is known approximately (e is the most common, z the least). An arithmetic encoder uses these probabilities to map an entire message into a single number between 0 and 1, achieving compression rates close to the entropy of English—about 1.5 bits per character, compared to the 8 bits of ASCII. Without the connection between probability and entropy, such schemes would not exist.

Cryptography and Randomness

Cryptographic systems also depend on entropy. The unpredictability of encryption keys—their entropy–determines their resistance to brute-force attacks. A key drawn from a uniform probability distribution over a large space (e.g., 256-bit keys) has maximum entropy and is considered secure. If the key generation process is biased, the entropy drops, and attackers can exploit that bias. Similarly, pseudorandom number generators used in cryptography must produce sequences whose entropy per bit is high; low entropy indicates predictability and weakness.

Machine Learning and Cross-Entropy Loss

In modern machine learning, cross-entropy is a standard loss function for classification problems. When a neural network outputs probabilities for each class, the cross-entropy between the true distribution (a one-hot vector) and the predicted distribution is minimized during training. This is equivalent to maximizing the log-likelihood of the true labels. The gradient of cross-entropy with respect to the model parameters naturally leads to updates that push predicted probabilities toward the actual frequencies of the data. Here, the probability–entropy connection directly drives learning.

Communication Systems and Channel Capacity

Another profound implication is the concept of channel capacity. The maximum rate at which information can be reliably transmitted over a noisy channel is given by the mutual information between the input and the output, which itself depends on input probabilities and the channel's conditional probabilities. By adjusting the input probability distribution, one can maximize the mutual information—and thus the capacity. This interplay between probability and entropy governs the design of modern modems, error-correcting codes, and wireless systems.

Further Reading and References

For those wishing to dive deeper, the following resources are recommended:

Conclusion

The connection between probability and entropy is more than an academic curiosity; it is a cornerstone of modern information science. Probability dictates the uncertainty of individual events, while entropy aggregates that uncertainty into a single, powerful metric. The Shannon entropy formula makes this relationship explicit and leads to critical applications in compression, cryptography, machine learning, and communication. Understanding this link allows engineers and scientists to design systems that efficiently handle, protect, and transmit information. As data continues to grow in volume and importance, the bond between probability and entropy will remain central to how we measure and manage it.