Information Measures

Information theory is a mathematical theory of communication, based on probability theory. It was introduced by Claude Shannon in his landmark paper A Mathematical Theory of Communications in 1948. He gave a quantitative measure of the amount of information stored in a variable and gave limits of how much information can be transmitted from one place to another over a given communication channel. Both the entropy and the mutual information are important measures of information. The entropy states how much information is needed to determine the outcome of a random variable. The chapter derives a famous result on data processing, called the data-processing lemma, and defines the entropy rate, which is the corresponding entropy measure for random processes. A natural way to define the entropy per symbol for a sequence is by treating the sequence as a multidimensional random variable and averaging over the number of symbols.