Conclusions from GMAT algorithm study based on ESR analyses

To clarify: the opposite of "more heavily weighted" is "less heavily weighted." I do not believe that's what you mean, lest someone reading this interpret it as such. Either way, I must disagree: the "myth" of the first 9-10 questions persists because it's not a myth at all. Despite what the GMAC and others want you to think, both the ESRs and my own anecdotal experience tutoring GMAT students for 15 years have supported the idea that the first 9-10 questions have a larger impact on your score than do any other group of 9-10 questions.

I'm not sure whether we should draw too many firm algorithm-related conclusions from the ESRs, given that for some reason the difficulty level of the questions (Correct / Incorrect / All) is indicated as only 4 average data points, and because of the small sample size / inherently complicated nature of judging the impact of early questions vs. the impact of how many questions the student got wrong in a row (information unavailable), combined with the vague nature of splitting the test into quarters and averaging it all out, rather than providing a question-by-question breakdown. It is likely that question #10 is more impactful than is question #18, but they are in the same quarter of questions, and thus we have no way to differentiate them. Also, every GMAT is different, and we have learned from my study of the IR section that GMAT "raw to scaled" score conversions are not necessarily consistent from test to test.

That being said, I will concede that the ESRs have given us some good insight here, and it's clear that a solid performance across the test can sometimes be as effective as an "excel early in the test" strategy (more on this in a second). Although the GMAT Prep tests use a different algorithm, the results I've seen from real GMATs definitely reinforce the importance of the first 9-12 questions. And in my opinion, so do the limited number ESRs that have been shared on GMAT Club thus far.

Let's use the Quant ESRs as an example. There are 28 Quant questions that count, which means that each quarter of the test consists of 7 real questions and 2 experimental ones (3 in the last quarter).

Here is a screenshot of a student getting off to a great start (100% correct on the first half = 14 out of 14!) and scoring a solid 47 on Quant despite basically falling apart at the end and running out of time (approximately 71% of questions correct overall):

To be more explicit, this person answered 14 of the first 14 (counted) questions correctly, but only 6 of the last 14, and still scored Q47! This, to me, clearly supports the conventional wisdom that getting off to a good start on the the first 1/2 of each GMAT section is indeed the most important requirement for a solid score.



In contrast, here is an example of a student also getting about 71% correct (5 out of every 7), but struggling more at the beginning and thus receiving a lower score:



Here is a screenshot of a student achieving an Q47 score with a more consistent performance, despite a lower percent correct total and struggling in the less-important last quarter of questions (approximately 64% total correct), which shows that you don't have to ace the first 7 questions that count, but that getting at least 5 of them correct (5/7 = 71%) is very helpful. This person managed to leverage his/her relatively low percent correct into a solid score, because of the relatively high difficulty of the questions that he/she had earned in the beginning, and maintained throughout the test.



Here is a student scoring a relatively low Q42 (51%) despite getting approximately 68% of the questions correct and a putting in a generally consistent performance, with slightly more struggles in the important middle half of questions:



Here is another student, who also got about 68% of the questions correct, but who struggled at the beginning and was thus penalized mightily (Q35 = 33%):



You will notice that all of the students whose ESRs I have included here have answered somewhere between 64% and 71% of the total questions correctly, but their scores ranged from 35 to 47. The difference in most cases? Students' performance on the first quarter, and to a lesser degree, their performance on the middle half of the section.

The ESRs do suggest, at the very least, that although earlier questions do still matter more than later ones, the GMAT algorithm might not be quite as "front-loaded" as it used to be. The 2nd and 3rd quarters of questions are also very important, but the 4th quarter, not so much (other than making sure that you take the time to answer all the questions before time runs out, thus avoiding the score penalty).

One of my main concerns with trying to interpret this data is that the "average difficulty" chart is very deceptive, and not nearly as descriptive as it first appears. It appears to chart your entire GMAT journey, but it's really just 4 average data points tied together by straight lines. One aspect of the chart that does stand out, however, is that there is a significant jump in difficulty from the first question (medium difficulty) to the 9th or 10th question, when the student either performs well (earning harder questions) or poorly ("earning" easier questions). The jumps in difficulty at the start of the 3rd and 4th quarters of the test are generally less severe than are the jumps in the first half of the test.

We also see odd features in there, such as the difficulty of incorrectly answered questions being indicated as medium, when the student in fact answered 100% of the questions in that quarter correctly.

He's correct on that, as dabral has already noted. It has to do with using the "percentile correct" number from the ESRs to determine the correct denominator (total number of non-experimental questions) for each quarter of the test. We know this because the GMAC does not provide correct/incorrect information for the experimental questions.

IR has 9 questions that count, and 3 experimental. We can tell that the IR section has exactly 9 non-experimental questions because the "percent correct" always aligns with a denominator of 9.
Quant has 28 questions that count, and 9 experimental. We can tell that the Quant quarters are split into 7 non-experimental questions each because the "percent correct" always aligns with a denominator of 7.
Verbal has 30 questions that count, and 11 experimental. We can tell that the Verbal quarters are split into 7 non-experimental questions (middle two sections) or 8 non-experimental questions (first and last sections) each because the "percent correct" always aligns with a denominator of 7 (2nd and 3rd quarters) or 8 (1st and 4th quarters).

Great stuff, HiLine!! You have zeroed in on at least a couple of important truths about the test.

Even though some people are convinced that GMAC isn't speaking the truth about the importance of the first few questions on a GMAT section, here are some interviews that we conducted with Larry Rudner, former Chief Psychometrician at GMAC:

https://www.gmatclub.com/forum/veritas-prep-resource-links-no-longer-available-399979.html#/talking ... testmaker/

The first video is aptly titled "The Myth of the First 10 Questions." This myth will always persist, I bet, but Dr. Rudner does his best here to put it to rest.

There is one more difference, a big one, which you did not notice.

The difficulty of questions.

A student who answered 10 easy questions incorrectly did not perform as well as a student who answered 10 difficult questions incorrectly. (This is obviously a hypothetical situation and in no way refers to any of the ESR's you posted.)

I did notice it (the change in difficulty level of the questions), but I didn't mention it because there's no way to precisely quantify its effect. But the difficulty of questions is directly tied to one's performance in the first 9-12 questions, which in my opinion the data supports as crucial (see various examples above), so in many ways they are the same issue.

If you don't do well up-front, then you will get easier questions. Doing well on easier questions does not raise your score nearly as much as doing well on medium/hard questions. Thus, the first 1/2 of questions is the most important, because that is where you "earn" the harder questions to follow.

You will notice in the last example that I provided that the student got 43% correct in the first quarter of questions (3 out of 7). He/she then went on to get 6 of the next 7 questions right (86%), but by then it was too late...the difficulty level had dropped too low and the student's score could not fully recover, despite a solid performance on the final 3/4 of the section.

GMAC does not always tell the truth about the test. For years, we were told that experimental ("pretest") questions made up 10-25% of the test. Turns out, they make up 25.6% of the GMAT. It was a white lie, yes, but a lie nonetheless. GMAC also continues to indicate on its website that Verbal and Quant scores range from 0 to 60, when in fact the highest score on either section is a 51.

Seeing that representatives from GMAC have provided incorrect and misleading information about the GMAT in the past, I'm not sure that it's wise to take them at their word when it comes to the scoring algorithm. Instead, I would trust the data first.

Along those lines, it would be more helpful to see actual ESRs that support this contention, rather than vague promises from GMAC.

I have attended many GMAC-led GMAT Test Prep Summits over the years, and every time they give the same answer: The scoring range has always officially been 0-60, but the range that they actually use tops out at 51. It's an artifact of how they originally set the scoring scale vs. what they actually ended up needing to use. No one from GMAC has ever dangled the false hope for test takers that they might somehow attain something higher than a 51.

Tell you what: You trust your data and keep combing through ESRs. We'll keep training students in the higher-order thinking skills needed to actually earn higher scores!

I do all of those things. Call me a multitasker, I guess. ; ) In the meantime, maybe your company can work on paying its GMAT instructors and tutors fairly, and maybe your teachers and tutors can work on actually verifying their GMAT scores.

That's good to know, because the test would be unfair if this weren't the case. Here are a few points others going through this thread could consider:

1. I don't think the questions are "weighted" in "sets" such as the first 10. It's more that the confidence level of the ability estimate changes over the course of the test. Think of it as all the information coming from all the questions (till that point) put together. At the end of the test, the score that a test taker receives is the one with the highest probability out of a range of possible scores.

2. Performance does not depend solely on the difficulty level of a question. There are certain questions that are better able to differentiate between different ability levels, and the information that such questions provide is inherently more valuable than the information provided by questions of the same difficulty level which are not able to differentiate as well.

That something cannot be precisely quantified does not justify the complete omission of it. I'm glad you have incorporated difficulty in the subsequent discussion.



As it is to that in all other questions.



Easier questions are easier to answer by definition. If someone does well on easier questions and not so well on more difficult questions, the algorithm is working well.




Earning harder questions means nothing if you cannot answer these questions correctly. Focusing on the first half of the section at the expense of the second half is only going to result in poor performance and reversion of difficulty level in the second half. The first half of the questions make up half of the battle. The second half is still ahead. And if you're a good student, the second half of the battle is even more fierce than the first.



No it was not too late; rather, the test taker did not perform well enough on easy questions to be challenged with difficult questions. In fact, that the test taker answered only about half of the questions correctly in the first 7 counted questions suggests that he was unlikely to be able to answer many challenging questions anyway.

So now we are being driven to conclude that no part of the exam must be given undue importance or be neglected, rather focus on every question! This is contrary to what I have read on top prep companies websites and many topics in here. So skipping questions (as promoted by certain prep companies and many individuals) is just a sign of moving over inability and nothing tactical as no one knows which questions is experimental.
.
In such a case, would a sound advise be to work on every question and not simply guess and move on?

So in other words, performing well on the first 7 counted questions is extra important, which is what I've been trying to argue all along. That's a very small sample size, and it helps explain why the first part of each section is crucial to achieving a solid GMAT score. Make a few mistakes up-front due to nervousness, carelessness, bad luck, or some other reason, and there is almost no way for your score to recover significantly, no matter how well you perform on the remainder of the section.

Why am I the only person who is including any actual data in my arguments? I would love to see some actual ESRs (with screenshots, please) that support your contentions. We are operating somewhat in the dark here due to the vague nature of the statistics on the ESRs, so I will concede that this issue is far from settled.

You're arguing with a straw man then. No one is saying the first 7 counted questions are not really important. All counted questions are really important.



Doesn't matter where you make careless mistakes. Answering easy questions wrong is going to do a lot of damage.




Your samples are pretty good for the purpose of the discussion. Feel free to bring more to the table if you so wish.

Allow me to clarify, then: getting questions wrong at the beginning hurts your score more than does getting questions wrong at the end. The data from the ESRs supports that conclusion, until I am shown otherwise with actual evidence.

The ESR's that you have posted do not support that conclusion.

What you are doing is comparing two scenarios that are not equivalent.

A candidate that answers 3 questions wrong at the beginning and answers the rest of the questions correctly is going to score less than one that answers 3 questions wrong at the end and answers the rest of the questions correctly, because the latter tackles more challenging questions at the same level of accuracy and thus should be deemed a more competent candidate.

What I am hearing from most of you who disagree with me is that "test-takers who do well on the first quarter of questions are better GMAT students who earn harder questions, and thus deserve better GMAT scores". Overall, of course this is true, but it does not change my point about the unfairly "front-loaded" nature of how GMAT scores are determined.

My point is that 9 questions is not large enough of a sample size to fully determine someone's "initial" GMAT skill level, and hence there is an element of luck involved in scoring well. Answer fewer than 5 out of the first 7 counted questions correctly, and your chances for an elite score are almost zero. Throw in the fact that 2 of those first 9 questions are experimental and don't count toward your score, and this adds even more randomness into the equation. You could answer 7 out of 9 correctly, and the 2 you got wrong were experimental (100% correct), or you could get 7 out of the 9 right, but 2 of the correct answers could have been experimental (5/7 = 71%). This is of course not only true about the first quarter of the test, but the entire GMAT, where slightly more than 25% of the questions are experimental.

Yes, it is true that correct responses (normally--see below) earn harder questions, and consistently answering medium/hard questions correctly is the best path to a great GMAT score. But the student won't even get the chance to try hard questions if the GMAT algorithm makes early, "snap" judgements about his/her skill level, which it does. Anyone who is out there reading this: don't take my word for it--take a look at the ESRs and decide for yourself! The evidence is readily available if you're willing to take a look: https://gmatclub.com/forum/information- ... 21111.html

Any statistician knows that in order to provide an accurate measurement across a large population, you need a large sample size. A score estimate based on a small sample size of GMAT questions is more prone to fluctuations based on lucky / unlucky guesses and other random elements. But perhaps this is exactly what the GMAC wants, so that you will keep paying $250 to take the GMAT again and again.

Also, the first questions are not easy difficulty--they are medium difficulty.

What's interesting about the first 10 questions "myth" is that the GMAC does not directly dispute it...instead it just uses vague, roundabout language such as "all questions count." Yes, all questions count, but that doesn't mean that earlier questions don't affect your score more strongly than do later questions, because in fact they do, and the data from the ESRs supports this.

The excerpt below is taken directly from the OG 2016:

"Myth -vs- FACT–
M - The first 10 questions are critical and you should invest the most time on those.
F – All questions count. It is true that the computer-adaptive testing algorithm uses the first 10 questions to obtain an initial estimate of your ability; however, that is only an initial estimate. As you continue to answer questions, the algorithm self-corrects by computing an updated estimate on the basis of all the questions you have answered, and then administers items that are closely matched to this new estimate of your ability. Your final score is based on all your responses and considers the difficulty of all the questions you answered. Taking additional time on the first 10 questions will not game the system and can hurt your ability to finish the test."

Like the rest of the GMAT, this text requires a close reading. Notice that not even the GMAC disputes that the first 10 questions have a larger impact on your final score than does any other group of 10 questions. Yes, spending too much time on the first 10 can hurt your score if you don't do so correctly (you need to save time for the remaining 27 questions, of course), but this does not change the fact that doing well on those first 10 questions is essentially mandatory for a top GMAT score. GMAT calls it an "initial estimate," but this is simply a euphemism for a snap judgment. Yes, adjustments to one's score continue to be made throughout the test, but the "Average Difficulty" charts on the ESRs support the notion that the largest adjustments in difficulty/scoring occur early in the test, and a close reading indicates that the GMAC does nothing to directly dispute this notion (although it tries its best to imply that this is untrue).

For example, the below performance of (broken down by quarters) 100% / 100% / 57% / 29% on Quant earned a 47 out of 51 (also mentioned in my original post). Not bad for getting more than half of the final 14 counted questions wrong.



To some perfectionistic students, a score of Q47 would be a disappointment. But to many others, it would be a dream come true. This ESR shows that you can essentially turn the GMAT Quant into a 25-question test, take aggressive guesses on the final 12, and still achieve an above-average Quant score.

Should you "invest the most time" on the first 10 questions? Of course not--that's barely 25% of the questions. The GMAC is setting up its own straw man here, to avoid addressing the real issue at hand. You should not spend most of your time on the first 10 questions, but you should spend more time per question on those questions than you do on the remaining ones, given their outsized importance to the scoring algorithm. It is true that for elite scorers, the questions will get harder and harder, and you might suffer at the end due to timing issues, ruining your chance for a 50/51 on Quant, or something above 45 on Verbal...trust me, I've been there. But this only separates the 730 scores from the 770s, or the 750s from the 800s, and all of those scores are great. To crack 700, leveraging the GMAT algorithm is key.

Also, you mentioned in your original post (heading #3) that the difficulty level seemed to drop, despite the student answering questions correctly, and that you weren't sure why.



I think that this passage (also from the OG 2016) should clarify the issue for you:

"Myth -vs- FACT-
M – Getting an easier question means I answered the last one wrong.
F – Getting an easier question does not necessarily mean you got the previous question wrong. To ensure that everyone receives the same content, the test selects a specific number of questions of each type. The test may call for your next question to be a relatively difficult problem-solving item involving arithmetic operations. But, if there are no more relatively difficult problem-solving items involving arithmetic, you might be given an easier item. Most people are not skilled at estimating item difficulty, so don’t worry when taking the test or waste valuable time trying to determine the difficulty of the questions you are answering."

Finally, someone who agrees with me... : ) hiline-gmat-algorithm-studies-223122.html#p1718357

I would also like to point out that those numbers represent a difference of only one correct question in the 1st quarter, yet they resulted in a 4-point score differential! Despite what the GMAC and others might try to tell you, the "myth" of the outsized importance of the first 9-10 questions is anything but.

4/7 counted questions in the 1st quarter = 57%

5/7 counted questions in the 2nd quarter = 71%