EM Algorithm for Mixture Model

We can generate a population for a binomial mixture under the following assumptions: That we have a number n of true cases out of a possible number N total, and that these have a binomial distribution with $\pi_i$ probability of being from each distribution.

This is easier to visualize with a ‘coins in a pot’ example. Say we have a pot filled with two types of coins, each type of coin having its own probability of heads. We pull a number of coins S from the pot, flip each coin N times and record the number of n heads. Under these assumptions, the probability of seeing n heads for each coin would be:

$$ P \left( n \vert N, \Theta \right) = \pi_1 \mbox{Binom} \left( n \vert N, \theta_1 \right) + \pi_2 \mbox{Binom} \left( n \vert N, \theta_2 \right) $$

or more generally for K different types of coins

$$ P \left( n \vert N, \Theta \right) = \sum_{k=1}^{K} \pi_k \mbox{Binom} \left( n \vert N, \theta_k \right) $$

where $\theta_k$ is the probability of heads for each type of coin. Therefor the log-likelihood for our parameters is:

$$ \mathcal{L} \left( \Theta \vert X, Z \right) = ln P \left( X, Z \vert \Theta \right) $$

Where $\Theta_k$ represents the set of $\pi_k, \theta_k$, X represents the set of heads and total coin flips, and Z represents the set of coin proportions and heads proportions. So the Auxiliary Function is:

$$ Q(\Theta, \Theta_o) = E \left[ ln \mathcal{L} \left( \Theta \vert X, Z \right) \vert X, \Theta_o \right] $$

and our expectation is:

$$ E \left[ ln P \left( X, Z, \vert \Theta \right) \vert X, \Theta_o \right] = \sum_{z_i = 1}^K ln P \left( n_i, z_i \vert \Theta \right) \cdot P \left( z_i \vert n_i, \Theta_o \right) $$

where

$$ P \left( z_i = k \vert n_i , \Theta_o \right) = \dfrac{P \left( z_i = k , n_i \vert \Theta_o \right) }{P \left( z_i , n_i \vert \Theta_o \right)} = \dfrac{\pi_{k,o} \mbox{Binom} \left( n_i \vert N_i , \theta_{k,o} \right) }{\sum_{l=1}^{K} \pi_{l,o} \mbox{Binom} \left( n_i \vert N_i , \theta_{l,o} \right) } $$

We can then use the following expressions to update $\pi$ and $\theta$ until convergence.

$$ \pi_m = \dfrac{1}{S} \sum_{i=1}^S P \left( z_i = m \vert n_i , \Theta_{o} \right) $$ $$ \theta_{m,S} = \dfrac{\sum_{i=1}^S n_i \cdot P \left( z_i = m \vert n_i , \Theta_{o} \right)}{\sum_{j=1}^S N_j \cdot P \left( z_j = m \vert n_j , \Theta_{o} \right)} $$