The outcome of sqrt(x^2-6x+9) depends on the value of x. If x >= 3, then sqrt(x^2-6x+9)=x-3, however if x<3, then sqrt(x^2-6x+9)=3-x. For example, if x=4, then sqrt(x^2-6x+9)=sqrt(16-24+9)=sqrt(1)=1, which is indeed equal to x-3=4-3=1. However, if x=0, then sqrt(x^2-6x+9)=sqrt(0-0+9)=sqrt(9)=3, which is indeed equal to 3-x=3-0=3.
The reason for this simplification is based on the identity sqrt(x^2)=|x|. In this particular case sqrt(x^2-6x+9) = sqrt[(x-3)^2]=|x-3| and the absolute value expression can be rewritten based on the value of x, if x>=3, then |x-3|=x-3, and if x<3, then |x-3|=3-x.
In summary, sqrt(x^2-6x+9) = x-3 if x>=3 and sqrt(x^2-6x+9) = 3-x if x<3. In the question you encountered, it was probably a data sufficiency question where they added the constraint x<3.
Comprehensive GMAT Quantitative Course: GMATQuantum
Free Video Explanations: OFFICIAL GUIDE GMAT 13, 12, 11, 10; QUANT REVIEW 2nd, 1st.