GMAT Prep Software Analysis and What If Scenarios

I've been meaning to post this for a year or two, but to me this experiment proves the exact opposite - it proves that the first ten questions are not especially important, or at least that they're not more important than later questions. That might seem counterintuitive, so I'll explain in more detail, first in a loose way, then in a mathematically precise way.

There's one crucial observation about this experiment: answering the first ten questions incorrectly is just an unspeakably bad performance. If you put a cat in front of the computer, and let it paw away at the keyboard, the cat would normally get 2 right answers in the first 10 questions, just by random guessing. So before looking at the Q38 and drawing a conclusion just from that score, it's important to define expectations. What should we think that test taker's Quant level is? At that point in the test, what would you guess, and how certain would you be? That's the question the algorithm is asking, and with that performance, the algorithm is going to be almost 100% certain that the test taker is below the 200 level, because essentially no test taker ever performs that badly.

To roughly guess what score the test taker ought to get in this experiment, it's important to know how the algorithm works. The algorithm assumes that if a question is at your level, you have a 60% chance to get it right. So if the difficulty levels of your first ten questions were 500, 400, and then eight 300-level questions, you need to get your next twelve 300-level questions right just to prove you're a 300-level test taker. And then you still have a lot of work to do to prove you're above that level. It's impressive that you can even reach a Q38.

The reverse happens when you answer the first 27 questions correctly. By that point you've almost certainly answered fifteen Q50-level questions correctly already, with no wrong answers on anything easier than that, so even with ten wrong answers on Q50-level questions, you still have a 60% hit rate at that level. So while the performance at the end is unusually bad, it's completely understandable that you'd still get a Q50.

Using the actual math behind the algorithm, if the test does proceed as above (500, 400, the rest 300) for the first ten questions, then a true 700-level test taker should have nearly a 99% chance of answering those 300-level questions correctly*. The chance a random 700-level test taker would answer the first ten questions incorrectly is roughly 1 in 6.6 million trillion (6.6 quintillion, or 6.6 * 10^18). It will simply never happen, so of course the algorithm will never give a Q49-Q50 score to a test taker who does that badly early in the test. Even the 300-level test taker only very rarely does that badly - they will only do that about 3 times in 10,000.

Put another way, if you knew before a test that a test taker was either a 300-level, 500-level, or 700-level test taker (with equal probability), and he or she answered the first ten questions on the test incorrectly, at that point in the test the probability would be 1 in 40 trillion that the test taker was 700-level, and 1 in 500,000 that the test taker was even just 500-level. At that point, if 300, 500 or 700 are our only options, you should be willing to bet a small house to win just $1 that the test taker is the 300-level test taker. The odds are astronomically against the test taker being even average. That you can recover from that position to get even a Q38 should indicate that the first ten questions are not weighted any more than the later ones.

It's the fact that there are so many wrong answers on easy questions that makes it so likely the test taker is a very low level test taker -- high level test takers almost never answer easy questions incorrectly. And that's the takeaway test takers should take from this: be careful not to get easy questions wrong, because that will hurt you a lot. It doesn't matter where they are in the test, beginning, middle or end. And if you get a really hard question early in the test, and get it wrong, you shouldn't worry about that at all - that's what you're supposed to do unless you're an 800-level test taker.

* (technical details for people who know about parameter values: here I'm using a=1 and c=0.2, along with b values of 0, -1, -2, for 500, 400 and 300 level, and ability values of -2, 0 and 2 for 300, 500 and 700 level)