I guess they sit around the table as you mention it in the title of the topic.

So Mary has her own chair, fixed position. Other two females around the table can sit in 2 ways: the first to the left of Mary and the second to the right or vise-versa. Now, if 3 males will sit between them then no two female or two male will sit next to each other (female-male-female-male-female-male). But these males on their spots can also sit in different ways, namely in 3! different ways, so total 2*3!=12.

Answer B.
_________________

Since Mary's position is fixed men can be seated in 3! ways not in 2! ways. Would be easier to draw a circle and count the possible arrangements.
_________________

(For those who always feel that there is some hidden gmat type of surprize) ultra tricky question in a way that special restriction (Mary's fixed place) is not even a restriction. Number of ways these people would sit around the table with Mary going round is repetitive and should be anyways taken out.
Let's fix females first -> according to formula (3-1)!=2!
It would have been 2! for males too, if not the fact that they are sitting in a big circle together with women. Thus we have 3!
3! 2! = 12