Differential Entropy, Entropy Rate, and Maximum Entropy

The entropy associated with continuous random variables is distinct from the entropy of discrete random variables; in fact, it is sufficiently different in its properties that it is given a new name, differential entropy. Treatments of differential entropy are standard in basic information theory texts and in any first course in information theory. So, much of the material in Sect. 3.2 is widely available. The topics of entropy rate, entropy power, and maximum entropy, while usually contained in the same textbooks and introductory courses, are not as extensively emphasized as in this chapter and this book. Further, the developments here are somewhat separated from information theoretic treatments of communications and compression and generalized so that the utility of differential entropy for exploring sequences as in agent learning is more evident.