While I see your point, the ultra elites accept many students with less than perfect records. The average GMAT generally around 700 (+/- 20 points) and the average GPA around 3.5. While a GMAT around 700 is not easy, it is certainly easier than a 760 or so. Likewise, a GPA near 3.5 is good but it certainly not flawless and would be considered low for students applying to the top cluster of other types of professional school (e.g. law where the center of the distribution for ultra elite level schools is closer to 3.75).
Accordingly, I would not worry too much about being a late-bloomer since ultra elite programs accept many students with somewhat "blemished" records.