Kullback–Leibler divergence

Kullback–Leibler divergence

Given two probability distributions over called and . The Kullback–Leibler divergence is the expected value of the log difference between and with the probabilities for each value being given by .