Hi, there. I'm happy to give my 2 cents on this.

First of all, this is a relatively difficult question for GMAT math. You would only see a question of this difficulty if you were already answering almost all of the math questions correctly.
I can't really suggest much other than trial-and-error. Since 25 is a factor of 75, I figured it might make sense if 25 were one of the three squares. That means the other two squares would have to add up to 50. Well, conveniently, 7^2 = 49, so 7^2 + 1^2 = 50, and 7^2 + 1^2 + 5^2 = 75. The three numbers are 1, 5, and 7, and those have a sum of 13. Answer =
E.
Here, I was really just following my intuition for numbers, which is really just a notch above pure guess-and-check. I don't know the source of this question, but the real GMAT tends to give questions that admit of either a methodical approach or an elegant solution, whereas this more or less requires some guess & check, some poking around in the dark. That's just not the style of questions that the GMAC dishes out.
Please let me know if you have any questions on what I've said here.
Mike

newish here - how come this is labeled sub 600 if it's tough? I found this on a official practice gmat