If we think of the interest rate r as a decimal, rather than as a percentage (i.e. we think of "10%" as "0.1" so we don't need to write "r/100" everywhere), then when we apply interest each year, we multiply our investment by 1+r. So after two years, the investment is worth 1200(1+r)^2, and since that equals 1650, we know
1200(1 + r)^2 = 1650
(1 + r)^2 = 165/120 = 11/8 = 1.375
So over the two years, the investment earned 37.5%. We can now rule out three wrong answers instantly. If the investment earned 19% each year, then without any compounding, it would earn 38%. With compounding, it would earn more than that. So there's no way the interest rate is 19% or higher, and A or B must be right. Working out which is right is a bit annoying though, especially because the question is only asking for an approximation, and it turns out the approximation it asks for isn't all that good. We can see which of 1.17^2 or 1.18^2 gets us closer to 1.375 by expanding them, so we only end up needing to square 17 and 18 (or 0.17 and 0.18), which aren't squares you need to know on the GMAT:
(1.18)^2 = (1 + 0.18)^2 = 1^2 + (2)(0.18) + 0.18^2 = 1 + 0.36 + 0.0324 = 1.3924
(1.17)^2 = (1 + 0.17)^2 = 1^2 + (2)(0.17) + 0.17^2 = 1 + 0.34 + 0.0289 = 1.3689
and A is the answer.
Posted from my mobile device