Sivaiah Gudipudi

information theory, the Shannon entropy or information entropy is a measure of the uncertainty associated with a random variable. It quantifies the information contained in a message, usually in bits or bits/symbol. It is the minimum message length necessary to communicate information.

This also represents an absolute limit on the best possible lossless compression of any communication: treating a message as a series of symbols, the shortest number of bits necessary to transmit the message is the Shannon entropy in bits/symbol multiplied by the number of symbols in the original message.

A fair coin has an entropy of one bit. However, if the coin is not fair, then the uncertainty is lower if asked to bet on the next outcome, we would bet preferentially on the most frequent result, and thus the Shannon entropy is lower. A long string of repeating characters has an entropy of 0, since every character is predictable. The entropy of English text is between 1.0 and 1.5 bits per letter.[1]

Equivalently, the Shannon entropy is a measure of the average information content the recipient is missing when he does not know the value of the random variable.

The concept was introduced by Claude E. Shannon in his 1948 paper A Mathematical Theory of Communication.