Entropy

日付: 3月 13, 2019

### Defined on the **event** of **one** random variable
* Information - The "suprise" when an event is observed <math><semantics><mrow><mi>I</mi><mo>(</mo><mi>x</mi><mo>)</mo><mo>=</mo><mo>−</mo><mi>log</mi><mo>⁡</mo><mi>p</mi><mo>(</mo><mi>x</mi><mo>)</mo></mrow><annotation encoding="application/x-tex">I(x) = -\log p(x)</annotation></semantics></math>I(x)=−logp(x)

### Defined on **one** random variable
* Entropy - The uncertainty of a random variable (or the mean information of the random variable) <math><semantics><mrow><mi>H</mi><mo>(</mo><mi>X</mi><mo>)</mo><mo>=</mo><mo>−</mo><munder><mo>∑</mo><mi>x</mi></munder><mi>p</mi><mo>(</mo><mi>x</mi><mo>)</mo><mi>log</mi><mo>⁡</mo><mi>p</mi><mo>(</mo><mi>x</mi><mo>)</mo><mo>=</mo><msub><mi>E</mi><mi>x</mi></msub><mo>[</mo><mi>I</mi><mo>(</mo><mi>x</mi><mo>)</mo><mo>]</mo></mrow><annotation encoding="application/x-tex">H(X)=-\sum_x p(x)\log p(x) = E_x[I(x)]</annotation></semantics></math>H(X)=−x∑p(x)logp(x)=Ex[I(x)]

### Defined on **two** distributions of **one** random variable
* Cross Entropy <math><semantics><mrow><mi>H</mi><mo>(</mo><mi>p</mi><mo separator="true">,</mo><mi>q</mi><mo>)</mo><mo>=</mo><mo>−</mo><munder><mo>∑</mo><mi>x</mi></munder><mi>p</mi><mo>(</mo><mi>x</mi><mo>)</mo><mi>log</mi><mo>⁡</mo><mi>q</mi><mo>(</mo><mi>x</mi><mo>)</mo></mrow><annotation encoding="application/x-tex">H(p,q)=-\sum_x p(x)\log q(x)</annotation></semantics></math>H(p,q)=−x∑p(x)logq(x)
* Relative Entropy (KL Divergence) - <math><semantics><mrow><mi>K</mi><mi>L</mi><mo>(</mo><mi>p</mi><mi mathvariant="normal">∣</mi><mi mathvariant="normal">∣</mi><mi>q</mi><mo>)</mo><mo>=</mo><munder><mo>∑</mo><mi>x</mi></munder><mi>p</mi><mo>(</mo><mi>x</mi><mo>)</mo><mi>log</mi><mo>⁡</mo><mfrac><mrow><mi>p</mi><mo>(</mo><mi>x</mi><mo>)</mo></mrow><mrow><mi>q</mi><mo>(</mo><mi>x</mi><mo>)</mo></mrow></mfrac><mo>=</mo><mi>H</mi><mo>(</mo><mi>p</mi><mo separator="true">,</mo><mi>q</mi><mo>)</mo><mo>−</mo><mi>H</mi><mo>(</mo><mi>p</mi><mo>)</mo></mrow><annotation encoding="application/x-tex">KL(p||q) = \sum_x p(x) \log \frac{p(x)}{q(x)}=H(p,q)-H(p)</annotation></semantics></math>KL(p∣∣q)=x∑p(x)logq(x)p(x)=H(p,q)−H(p)
* Jensen-Shannon Divergence <math><semantics><mrow><mi>J</mi><mi>S</mi><mo>(</mo><mi>p</mi><mo separator="true">,</mo><mi>q</mi><mo>)</mo><mo>=</mo><mfrac><mn>1</mn><mn>2</mn></mfrac><mo>[</mo><mi>K</mi><mi>L</mi><mo>(</mo><mi>p</mi><mi mathvariant="normal">∣</mi><mi mathvariant="normal">∣</mi><mi>m</mi><mo>)</mo><mo>+</mo><mi>K</mi><mi>L</mi><mo>(</mo><mi>q</mi><mi mathvariant="normal">∣</mi><mi mathvariant="normal">∣</mi><mi>m</mi><mo>)</mo><mo>]</mo></mrow><annotation encoding="application/x-tex">JS(p,q)=\frac{1}{2}[KL(p||m)+KL(q||m)]</annotation></semantics></math>JS(p,q)=21[KL(p∣∣m)+KL(q∣∣m)]

### Defined on the **event** of **two** random variable
* PMI (Pointwise Mutual Information) <math><semantics><mrow><mi>P</mi><mi>M</mi><mi>I</mi><mo>(</mo><mi>x</mi><mo separator="true">;</mo><mi>y</mi><mo>)</mo><mo>=</mo><mi>log</mi><mo>⁡</mo><mfrac><mrow><mi>p</mi><mo>(</mo><mi>x</mi><mo separator="true">,</mo><mi>y</mi><mo>)</mo></mrow><mrow><mi>p</mi><mo>(</mo><mi>x</mi><mo>)</mo><mi>p</mi><mo>(</mo><mi>y</mi><mo>)</mo></mrow></mfrac><mo>=</mo><mi>I</mi><mo>(</mo><mi>x</mi><mo>)</mo><mo>+</mo><mi>I</mi><mo>(</mo><mi>y</mi><mo>)</mo><mo>−</mo><mi>I</mi><mo>(</mo><mi>x</mi><mo separator="true">,</mo><mi>y</mi><mo>)</mo></mrow><annotation encoding="application/x-tex">PMI(x;y)=\log \frac{p(x,y)}{p(x)p(y)}=I(x)+I(y)-I(x,y)</annotation></semantics></math>PMI(x;y)=logp(x)p(y)p(x,y)=I(x)+I(y)−I(x,y)

### Defined on **two** random variable
* Mutual Information <math><semantics><mrow><mi>I</mi><mo>(</mo><mi>X</mi><mo separator="true">;</mo><mi>Y</mi><mo>)</mo><mo>=</mo><mi>H</mi><mo>(</mo><mi>X</mi><mo>)</mo><mo>−</mo><mi>H</mi><mo>(</mo><mi>X</mi><mi mathvariant="normal">∣</mi><mi>Y</mi><mo>)</mo></mrow><annotation encoding="application/x-tex"> I(X;Y) = H(X) - H(X|Y)</annotation></semantics></math>I(X;Y)=H(X)−H(X∣Y)
* Conditional Entropy <math><semantics><mrow><mi>H</mi><mo>(</mo><mi>Y</mi><mi mathvariant="normal">∣</mi><mi>X</mi><mo>)</mo><mo>=</mo><msub><mi>E</mi><mi>x</mi></msub><mo>[</mo><mi>H</mi><mo>(</mo><mi>Y</mi><mi mathvariant="normal">∣</mi><mi>X</mi><mo>=</mo><mi>x</mi><mo>)</mo><mo>]</mo></mrow><annotation encoding="application/x-tex">H(Y|X)=E_x[H(Y|X=x)]</annotation></semantics></math>H(Y∣X)=Ex[H(Y∣X=x)]

KONGROOの基地

このブログを検索

Entropy

コメント

コメントを投稿