With these answer choices, I'd definitely just estimate. The side of the octagon is clearly less than 4 (since the octagon makes up only part of the width of the rectangle, which is 4). Only answers D and E are less than 4, since √51 is just slightly bigger than 7. If D were correct, then the side of the octagon would be roughly 3.1 or 3.2, and you can easily see that's impossible just by drawing a picture and seeing what the lengths would look like -- the triangles would need to be nearly flat, and there'd be no room along the side of length 6 for both long legs of the right triangles along with a side of the octagon. So E must be right.
If we need an exact solution, and this seems well beyond GMAT scope: say the side of the octagon is x. If we're cutting out a length of 'b' from the side of length 6, and a length of 'a' from the side of length 4, then we know:
2b + x = 6
2a + x = 4
and subtracting the second equation from the first, 2b - 2a = 2, and b - a = 1. So b = 1 + a.
So now if we focus on the small triangle at the bottom left of the picture, its three sides are a, a+1, and x, where x is the hypotenuse. So by Pythagoras, a^2 + (a+1)^2 = x^2. Since we also know 2a + x = 4, so a = (4-x)/2, we can substitute for a, to get one equation in x:
[(4-x)/2]^2 + [(6-x)/2]^2 = x^2
(4-x)^2 + (6-x)^2 = 4x^2
52 - 20x + 2x^2 = 4x^2
2x^2 + 20x - 52 = 0
x^2 + 10x - 26 = 0
That's only factorable using the quadratic formula, which you'd never need on the GMAT. Doing that, the solutions to ax^2 + bx + c = 0 are:
[ -b ± √(b^2 - 4ac) ] / 2a
so here are
[ -10 ± √(10^2 - 4(1)(-26)) ] / 2 = -5 ± [√(100 + 104)]/2 = -5 ± √51
and since the solution must be positive, the answer is √51 - 5.