Yale is a great school, and you can do very well with a degree from there. If you allow yourself to descend into the myopic world of ultra-elite or bust, you can easily lose sight of this reality.
Believe it or not there are people in this word that actually get angry when their alma mater loses a rank or two in Joe Magazine's super famous annual Best School publication and get ecstatic if it increases. I suggest their energies are misplaced.
I've got a degree from an "ultra elite" law school. I've had a fairly successful legal career, but I have still never figured out how to gracefully tell a judge or jury where I got my law degree. My success or lack thereof has depended on other things.
Sure, a degree from a super duper elite school may open some doors right out of school, but it is absolutely no guarantee of long term career success and probably not even a great predictor thereof.
And, hey, if you have a normal career with a Stanford MBA does that make you a super duper humongous failure ... given your perceived starting "advantage".
I've met a lot of ranking whores. They tend to fall into two categories: (1) total bores and (2) surprisingly insecure tough guys.
You aren't going to change them. I recommend just ignoring them.