#jensen-inequality

Complete the proof that $D_{\mathrm{KL}}(p \| q) \ge 0$ (assume $p$ and $q$ are positive on the same outcomes). Write the divergence as $E_{x \sim p}\left[-\log \frac{q(x)}{p(x)}\right]$. Because $-\log$ is convex, Jensen's inequality moves the expectation inside the logarithm: $D_{\mathrm{KL}}(p \| q) \ge -\log E_{x \sim p}\left[\frac{q(x)}{p(x)}\right]$. The expectation inside the logarithm equals $\sum_x q(x)$, which is . So the right-hand side equals , and the divergence can never be negative.