Hi Shakandbake,
I really feel for you, specially in such a position where your selection criteria is tighter than many other applicants. I have read somewhere that schools expect some serious excuses when you reject the EA admission offer. If you look at the positive side, this means they are expecting some applicants to do that and, from what I read here, you have a real convincing case to withdraw the application. Also, as an admitted EA applicant, I have seen on my confirmation card (Which I need to return with my reply enclosed) the reply options as either 1. Yes, I accept 2. I decline the offere because... and you need to put your reason. This tells you that they are expecting some candid response as to why you would want to do that, something I believe you won't have problem explaining.
While the contract language gives the school the right to react as they believe appropriate, no one really knows what they will be able to do. But I think schools don't have the time or will to run after students when they have much more important work to do during this time of the year. They will definitely take the deposit which is really hurtful and enough of a punishment on its own
My suggestion: I don't think (I'm not sure about that) the binding nature of the application forbids you from continuing to conduct interviews and hear back from other schools but it may be just an obligation of ultimately joining the school. If so, you should be able to have some good float time until the first deposit deadline (and if it's not enough and it got too serious, pay the first deposit and extend the "deciding" period a few months ahead) to wait and think of what's good for you and your wife. Do not inform them about anything during the next a few days/weeks until you have come down to a final decision.
Lastly, as a rule of thumb, EAs are there to help very committed students get into their dream schools. They're not there to worsen anyone's life
All the best making the best decision for you and your wife. Please keep us posted here