A couple of ways to look at this:
(A+B)(A-B) = 36
Now A+B and A-B are always either both even or both odd, if A and B are integers. Here they multiply to an even number, so they must both be even. So we want two different even numbers that multiply to 36, and while there are more systematic ways to find those numbers, with a number as small as 36 we can just use inspection: they must be 2 and 18. So A+B = 18 and A-B = 2, and adding equations we get 2A = 20 and A = 10.
You could also rewrite the equation like this:
6^2 + B^2 = A^2
Notice that's just the Pythagorean theorem, and if the letters are integers, any solution must be one of the integer Pythagorean triples. We know if we double the lengths of the 3-4-5 triangle, to get 6-8-10, we get an integer solution to Pythagoras, and since this question can only have one right answer (a GMAT question can't have more than one), that must be the only answer to the question, and A = 10.