The GMAT won't test you on technicalities around the definition of "inscribed" -- unless you know precisely how that word is defined, you wouldn't know if it's possible to inscribe a non-square rectangle in a circle (touching at just two points). You can't actually do that, because of the definition of "inscribed", so the two rectangles here must be squares, but if this were a real GMAT question, it would just tell you both of the quadrilaterals are squares.
It's a pure ratio problem, so we can invent a number - say the edge of the big square is 4. Then the area of that square is 16, and the area of the big circle, which has a radius of 2, is 4π. Subtracting the circle from the square, we get the total area of the four corners (one of which is red), so that total area is 16 - 4π, and the area of one corner is 1/4 of that, or 4 - π. So that's the red-shaded area.
The diameter 4 of the big circle is the diagonal of the small square. Since the diagonal of a square is √2 times an edge of that square, each edge of the small square is 4/√2 = 2√2. So the area of the small square is (2√2)^2 = 8. The small circle has a radius half the length of a side of the small square, so a radius of √2 .Its area is thus 2π. We get the total yellow area by subtracting the area of the small circle from the area of the small square, so the yellow area is 8 - 2π.
Notice now that the yellow area is exactly twice as big as the red area, so 1/3 of the coloured area is red, so that's the probability, if we pick a random point from the coloured regions, that the point is in the red region.
_________________
GMAT Tutor in Montreal
If you are looking for online GMAT math tutoring, or if you are interested in buying my advanced Quant books and problem sets, please contact me at ianstewartgmat at gmail.com