Is there any direct formula to calculate it?
Thanks in advance !
Not really, but no need for a formula.
Think of, for example, choosing your 3 x 3 square by choosing first its left bottom corner. You have to be sure that the upper right corner will still be inside the big board.
It means that you can choose the bottom left corner anywhere in the bottom left area of 4 x 4. Which means 4 x 4 = 16 possibilities.
Or you can concentrate on any other corner of the 3 x 3 square, and then figure out where that corner can be.
I don't think you need to force here some formula saying 4C1 x 4C1 for choosing those corners...or anything similar.
So, answer D.
PhD in Applied Mathematics
Love GMAT Quant questions and running.