First, we have a 6-8-10 right triangle on the left.
Now, if we start labeling angles, we'll find all the triangles are similar. If you take the point along the bottom where the rectangle meets that line (where the rectangle meets AB), and if we label the angles around that point, we have a 90 degree angle in the middle, since it's an angle in the rectangle. Then if the angle on the left (the one in the 6-8-10 triangle) is x, the angle on the right is 90-x, because the three angles in a straight line add to 180. But now in the lefthand triangle we have now have labeled two angles, 90 and x degrees, and the third angle in the top left corner must be 90-x degrees, because angles in a triangle sum to 180. Similarly we have two angles in the small bottom righthand triangle, 90 and 90-x degrees, and the third angle (at point B) must be x degrees.
So the big right triangle on the left and the small one on the right both have x, 90-x and 90 degree angles, and they're similar. The hypotenuse of the big triangle is 10, and of the small triangle is 5, so the small triangle is exactly half as big (in terms of lengths) as the big one. In the big triangle, the side '6' is opposite the angle x, and in the small triangle, the width of the rectangle is opposite the angle x, so those sides correspond, and the width of the rectangle is 3.
Now if you label angles around the small triangle in the top left, you again find the angles are x, 90-x and 90, so that triangle is similar to the other two, and since its short side is 3, that triangle is exactly identical to the triangle in the bottom left. So it is also a 3-4-5 triangle, where 4 is the length of the downwards sloping edge. We know that downwards sloping line consists of that edge of 4 plus the length of the rectangle, and we also know that line is length 10 because it's the hypotenuse of the 6-8-10 triangle, so the length of the rectangle is 10 - 4 = 6, and since its width is 3, its area is (6)(3) = 18.
It's quite a bit faster to do than it is to explain (at least without a diagram) but it's a good question setup. What is the source?
_________________
GMAT Tutor in Montreal
If you are looking for online GMAT math tutoring, or if you are interested in buying my advanced Quant books and problem sets, please contact me at ianstewartgmat at gmail.com