# 3 em algorithm bookmark this page consider the following mixture of two gaussians p 5149171

3. EM Algorithm Bookmark this page Consider the following mixture of two Gaussians: p(x;0) = A1N (2;441,) + 2N (2;M2, oz) This mixture has parameters Q = T1, 12, M1, M2,07,0}}. They correspond to the mixing proportions, means, and variances of each Gaussian. We initialize 0 as = {0.5, 0.5, 6, 7, 1,4}. We have a dataset D with the following samples of x: x(0) = -1, x(1) = 0, x(2) = 4, x(3) = 5,x(4) = 6. We want to set our parameters such that the data log-likelihood l (D; 2) is maximized: argmax Ilog P (260;0). Recall that we can do this with the EM algorithm. The algorithm optimizes a lower bound on the log-likelihood, thus iteratively pushing the data likelihood upwards. The iterative algorithm is specified by two steps applied successively: 1. E-step: infer component assignments from current do = 0 (complete the data) p(y=k ):=p(y=k ); 6.), for k = 1, 2, and i = 0,…,4. 2. M-step: maximize the expected log-likelihood p(2),y=k; 2) 1 (D;o) := p(y=k | 26″) logº (2 p(y=k|x(i)) with respect to 8 while keeping p(y=k 2()) fixed. To see why this optimizes a lower bound, consider the following inequality: log p (z;0) = logŞP (2, 9; 0) = logÇq(yla) P(t, x; 0) p(x,y; 0) (y) [p(x,y; 0) 1 = log Eyq(ul) I a(y) p(x,y; 2) 1 > Eyna(y) log 9 (42) p(x,y; 6) 9 (y) = q (ula) log ? where the inequality comes from Jensen's inequality. EM makes this bound tight for the current setting of O by setting q(y 2) to be py x; 0). Note: If you have taken 6.431x Probability-The Science of Uncertainty, you could review the video in Unit 8: Limit Theorems and Classical Statistics, Additional Theoretical Material, 2. Jensen's Inequality. Likelihood Function What is the log-likelihood of the data lD; given the initial setting of e? Please round to the nearest tenth. Note: You will want to write a script to calculate this, using the natural log (np.log) and np.float64 data types. E-Step What is the formula for pſy=k 2, 6)? Write in terms of Tik, 71, 72, Nk, Ni, and N2 (where Nk=N (X Mk, 02)). M-Step Fixing p (y=k 2,00), we want to update 0 such that our lower bound is maximized. What is the optimal ûn? Answer in terms of 2(1), 2(2), and “kl. Yk2, which are defined to be Yki = p(y=k | 2(); .) (For ease of input, use subscripts instead superscripts, i.e.type x_i for x(). Type gamma_ki for “ki 🙂 What is the optimal ő? Answer in terms of x(+), 2(2), 7ki and Yk2, which are defined as above to be Yki = p(y=k 20);00), and like (Type hatmu_k for îk. As above, for ease of input, use subscripts instead superscripts, i.e. type x_i for x'). Type gamma_ki for ki 🙂 What is the optimal ik? Answer in terms of 7k1 and Yk2, which are defined as above to be Yki = p (y=k 20);00), (As above, type gamma_ki for Vki.) Note: that you must account for the constraint that 111 + 12 = 1 where 71,72 > 0. Note: If you know that some aspect of your formula equals an exact constant, simplify and use this number, i.e. 711 + 721 = 1.