Ax(y) is an operation that adds 1 to y

The question should read:

\(A_x(y)\) is an operation that adds 1 to y and then multiplies the result by x. If x = 2/3, then \(A_x(A_x(A_x(A_x(A_x(x)))))\) is between

(A) 0 and ½
(B) ½ and 1
(C) 1 and 1½
(D) 1½ and 2
(E) 2 and 2½

According to the stem:
\(A_x(x)=(x+1)x=x^2+x\);
\(A_x(A_x(x))=A_x(x^2+x)=(x^2+x+1)*x=x^3+x^2+x\);
\(A_x(A_x(A_x(x)))=A_x(x^3+x^2+x)=(x^3+x^2+x+1)*x=x^4+x^3+x^2+x\);
...

We can see the pattern now, so \(A_x(A_x(A_x(A_x(A_x(x)))))=x^6+x^5+x^4+x^3+x^2+x\).

For \(x=\frac{2}{3}\) we'll get: \((\frac{2}{3})^6+(\frac{2}{3})^5+(\frac{2}{3})^4+(\frac{2}{3})^3+(\frac{2}{3})^2+(\frac{2}{3})\).

So, we have the sum of the 6 terms of the geometric progression with the first term equal to \(\frac{2}{3}\) and the common ratio also equal to \(\frac{2}{3}\).

Now, the sum of infinite geometric progression with common ratio \(|r|<1\), is \(sum=\frac{b}{1-r}\), where \(b\) is the first term. So, if we had infinite geometric progression instead of just 6 terms then its sum would be \(Sum=\frac{\frac{2}{3}}{1-\frac{2}{3}}=2\). Which means that the sum of this sequence will never exceed 2, also as we have big enough number of terms (6) then the sum will be very close to 2, so we can safely choose answer choice D.

Answer: D.

One can also use direct formula.
We have geometric progression with \(b=\frac{2}{3}\), \(r=\frac{2}{3}\) and \(n=6\);

\(S_n=\frac{b(1-r^n)}{(1-r)}\) --> \(S_{6}=\frac{\frac{2}{3}(1-(\frac{2}{3})^{6})}{(1-\frac{2}{3})}=2*(1-(\frac{2}{3})^{6})\). Since \((\frac{2}{3})^{6}\) is very small number then \(1-(\frac{2}{3})^{6}\) will be less than 1 but very close to it, hence \(2*(1-(\frac{2}{3})^{6})\) will be less than 2 but very close to it.

Answer: D.

Hope it's clear.