In the figure shown, ABCD is a square with side length 5 sqr

There are a lot of ways to do this. You could add the two areas of the quarter-circles ABD and BCD: when you do that, you'll be adding the area of ADFB once, the area of CDEB once, and the area of the shaded region twice. That's just the area of the entire square plus the area of the shaded region. So if you then subtract the area of the square, you'll have the area of the shaded region (this would be easier to demonstrate by drawing diagrams, but hopefully what I mean is clear - it's very much like a Venn diagram question, in which you add the numbers in each of two overlapping groups; in that case you're counting the overlap twice).

The area of each quarter-circle is (1/4)*Pi*(5sqrt(2))^2 = 25*Pi/2. Adding the two quarter-circles' areas gives 25*Pi. Then subtracting the area of the square to get the area of the shaded region gives 25*Pi - (5*sqrt(2))^2 = 25*Pi - 50.

You might also be able to pick the right answer here without really doing any work. The area of the square is 50, and the area of the shaded region looks to be about half the area of the square. That alone gets you to answers B or D immediately. Since you can pretty reliably guess that you'll need to subtract one area from another to get the answer, B almost has to be correct.