Information entropy

Information Entropy

Suppose we have a discrete random variable that can take values in . We define the information entropy of to be The higher the entropy the more uniform the random variable.