Importance of the First 10 Questions - GMAT Prep What If Scenarios

There is no sense in which this is a 'hack'. Anyone capable of getting their first 10 questions right, and every subsequent PS question right (even granted roughly 50% more time to do so) is almost certainly a Q51 level test taker. That person would simply be capping their score at a Q49 or Q50 by guessing at their later DS questions.

Many people seem to believe that, given extra time, test takers could get all of their questions right. This is demonstrably false. Test takers who believe this should simply look back over their diagnostic tests, either timing themselves per question, or using a test that records time per question, and examine their performance on those questions on which they spent the most time. I've looked over hundreds of practice test results, and people tend to have a hit rate in the 30%-40% range on those questions where they spend 4+ minutes. In math, if you're going to see how to solve a problem, you're almost always going to see how to solve it quickly. If you don't see that path to a solution early on, you might never see it, which is why when people spend 5 minutes on a question, their answer ends up being essentially a guess anyway, most of the time.

That's not only true from my own experience, looking at test taker performance. It's also proven to be true in a large scale study:

https://onlinelibrary.wiley.com/doi/pdf ... .tb01905.x

That study gave thousands of test takers two GRE Quant tests, and gave test takers 50% more time on the second test. At the time, GRE Quant scores ranged from 200 to 800. I think most people would expect that, given 50% more time, their score would increase significantly - I'd bet most test takers would guess their GMAT score would go up by 50 to 100 points if they had 50% more time. But that's not what happens. In that study, with lots of extra time, GRE Quant scores increased from an average of 664 to 671, so not even by ten points. And the effect is greatest for low-level test takers, and smallest for high-level ones, so the test taker trying to use the 'hack' you describe is gaining almost nothing from the extra time they'd have, while at the same time sacrificing good answers to a lot of questions.

The above is one reason why the oft-repeated "spend a lot of time on the first ten questions" is such terrible advice. The extra time you spend helps you far less than you might think early in the test, and running short on time will hurt you far more than you might expect (if you believe the person giving this advice) on later questions.

There's a GMAC research report describing precisely how the test behaves in these two scenarios (when it is better to guess to finish the test, and when it is better to leave questions unanswered). In a couple of places (older editions of the OG and an interview with the test developer), they've described the penalty for not finishing the test as a 'proportional' penalty. Based on the one numerical example the OG provided, I always assumed that if you answered, say, 35 out of 37 questions, and left the last two unanswered, they used your score estimate after 35 questions, and multiplied it by 35/37. That's hard to test though, because there's no way to know what your score is after, say, 25 questions, so no way to work out how big the penalty is on any given test. If you answer as you did -- getting the first 21 questions right -- your score is not a Q51 inside the algorithm. Internally, the scoring scale goes beyond Q51 (internally, they track how many standard deviations above average you are, so the scale is theoretically unbounded). Scores above a Q51 are capped at Q51 because that's the top of the scale. So there's no easy way to tell if your score was simply multiplied by 21/31 in your experiment, or if they've softened the penalty a bit for not finishing, or if the proportional penalty is applied to something other than the Q6-Q51 score.

The GMAC research report about guessing vs not finishing concluded that for higher level test takers, it is almost always going to be best to finish the test, even if that means guessing randomly. In part that's because high scores suffer most from proportional penalties. But it's also because higher level test takers who guess at the end of a test are normally guessing at very hard questions, because the test is adaptive. And guessing at hard questions doesn't hurt you much. For a test taker who is below average, a proportional penalty hurts much less, but a guess hurts more, because that guess is much more likely to be on an easy question, and getting easy questions wrong is very harmful to your score. The reason GMAC can't give an exact answer about this question is because the test is not perfectly adaptive, and everyone's test is different - if the Q41 test taker guesses at two questions at the end, and those questions both happen to be easy, those guesses might hurt more than not finishing. That's unlikely though -- usually those questions will be medium-hard -- so guessing will be better most of the time.

From GMAC's own research, the best practice is likely to be this, speaking probabilistically: if you are a higher level test taker (Q40+ say) you should finish the test no matter what you need to do, even if that requires you to guess randomly. If you're somewhere near average (Q27-Q40, say) it probably won't make a big difference, but I'd still suggest trying to finish (though don't panic if you can't - it probably hasn't hurt you much). If you are a below average test taker (Q6-Q27) it almost certainly makes a negligible difference what you do, and in fact it might be better to not finish, if your only choices are 'guess randomly' or 'don't finish the test'. Of course the best thing to do is to finish every question, answering as well as you can, so this only applies to people who find themselves in a test situation where they won't be able to do that.

It depends what you mean by this - if, say, there are more RC questions (proportionally) than before, then this could be true. And because the test is shorter now, in some sense every question matters more than it used to (at least because every question is more likely to count, because the number of experimental questions has been reduced).

But if instead the inference is that each RC question carries a certain weighting, and each SC question carries a certain weighting, and those weightings are different (i.e. if someone were claiming that a random RC question is worth more "points" than a random SC question) then that is not true, and never has been. It will hurt you just as much to get a 500-level RC question wrong as it will to get a 500-level SC question wrong (assuming the other question parameters are identical).

There is a reason, however, that random guessing at RC questions can be more harmful than guessing at SC questions. RC does not adapt by question. If you're doing well, the test might try to give you a hard RC passage. But that just means the questions are hard on average - if you have four questions, they might be 300, 500, 700 and 700 level. If you're guessing at those, you're usually getting a 300-level question wrong. That's a really bad thing to do on an adaptive test. SC will adapt by question, though, so if you're doing well, and guess at an SC question, you'll usually be getting a hard question wrong, and that's not so harmful to your score. So a high level test taker, if forced to guess at one question, is taking less of a risk guessing at one SC question than at one RC, because the RC question will sometimes be easy. But a low level test taker will be taking less of a risk guessing at one RC question than at one SC, because in that case, the RC question is more likely to be a hard question than the SC.

So you'd expect to get lower scores guessing at RC questions and answering everything else perfectly, because you're guessing at more easy questions, not because RC is weighted more heavily than other question types. You should have found that to be true on the old GMAT too, though there will be enough variation from one test to the next in the difficulty of various questions (and in the difficulty level of the ones you guess correctly) that you might need to run several trials to notice a pattern.

One of my concerns about the experiments in this thread is that they give the profoundly misleading impression that the scoring algorithm cares for some reason which question numbers you get right and which you get wrong - I'm afraid this thread can give the impression that the algorithm gives a certain weight to say question #10, and a different weight to question #15. That's simply not true - the algorithm doesn't care about question number at all. It cares about question difficulty. And the difficulty of each specific question varies a lot from test to test.

When a test is full of very hard questions, you can still get a very strong score with many wrong answers, because even a Q49 level test taker is not expected to get Q50-Q51 questions right too often. But when a test is full of easy questions, you need to be almost perfect to get a strong score, because a Q49 level test taker is expected to get a Q29 level question right almost every time. The free official practice tests have a lot of hard questions in the database, so they're often very forgiving of wrong answers. But when you buy the additional tests, you can exhaust the supply of hard questions, and then you can end up taking tests consisting mostly of easy/medium questions. That's what happened in your case: you had a test with lower level questions, and needed to be nearly perfect to get a good score. Notice that doesn't actually affect your score at all -- you got a Q49 just as before -- but it does affect the number of wrong answers you can have and still get a Q49.

On the real test, the question bank will be deep enough that it should feel like your early GMATPrep tests - you'll likely see a lot of hard questions, and you'll be able to get a Q49 or Q50 with several wrong answers.

We can test outcome when one performs exceptionally in some sections but fails abysmally in other section.
The cases such as :
1. SC, CR > 80+, RC<30
2. RC, CR > 80+, SC<30
3. SC, RC> 80+, CR<30

It will be useful for the test takers to know GMAT better.

Great stuff, Bunuel! Cheers to all your valuable contributions to the Quant forums.


I ran a question-type count on the new (online) Verbal section of Exam 1, and here is what I found: 15 SC, 13 RC, 8 CR.

Make of that what you will! Also note that this balance/frequency of question Verbal types on the new GMATPrep tests does NOT appear to reflect their true proportions on the actual GMAT, based on the debriefs we've read so far, which suggest that RC is now the most common question type:



My best guess for the actual number of counted questions on the GMAT is this: 15 PS, 13 DS, 12 RC, 10 SC, 8 CR.

Keep in mind that I only ran one simulation. It is entirely possible that every GMATPrep test now has a slightly different balance of question types, much like the real GMAT itself, due to the random distribution of experimental questions by type.

These analyses also demonstrate that although the GMAT Official Practice Exams are the obviously most realistic in terms of raw material, the scoring on the real GMAT is clearly different. For example, 2 of your demonstrations earned scores of Q50 despite answering 10 questions wrong, which would never happen on the real GMAT: according to 5 years of ESR evidence, for example, we have never seen more than 6 (counted) questions wrong correlate to a Q50. Similarly, you can answer up to 9 questions wrong (in a row!) on GMATPrep and yet still earn a V42. This scenario would never occur on the real GMAT, either: a V42 score correlates to only 2-5 counted questions incorrect.



For more info: https://gmatclub.com/forum/psa-the-scor ... 67046.html

For this reason, to be safe, I would subtract 20-30 points from the average of your GMAT Official Practice Tests in order to estimate your likely score on the real GMAT.

-Brian

Many of the conclusions drawn in this thread are potentially misleading. I'll try to point out where, but it will take me a few posts. Importantly:



This is not really true - the algorithm uses a 3-parameter logistic (IRT) model, something GMAC has confirmed many times in their published reports. Now, anyone designing a 3-parameter IRT test is free to set certain technical parameters how they like, and no one besides the GMAT test developers will know how those have been set, but those parameters mostly have to do with question selection, and not with scoring, and they aren't very important. The other information that is secret, and which prevents anyone from 'reverse engineering' their score from their right and wrong answers on a given test, are the statistics associated with each test question -- the exact difficulty level of each question, for example (and the other two of the three parameters the algorithm uses - what are known as the 'discrimination' and 'pseudo-guessing' parameters for each question). But the scoring algorithm itself is just based on probability theory, and its mathematical basis is explained in countless academic articles, and even on wikipedia.

Knowing the mathematics behind the scoring algorithm will not, however, give a test taker any advantage whatsoever on the GMAT. For that reason (and because you'd need an undergraduate foundation in statistics, probability theory and calculus to even begin to make sense of it) I do not recommend any test taker learn about the algorithm. It seems many people believe, possibly because it is poorly explained in many prep books, that there should be some way to "outsmart" the algorithm, that there might be some strategy like "spend a lot of time on the first ten questions" that will maximize your score. There is no such strategy. It's exactly like a standard multiple choice test: I can tell you the 'algorithm' I will use to grade a standard test ("I will count how many right answers you have"). Knowing that 'algorithm' doesn't help you to do any better on the test. You'd still want to answer the questions correctly that you know how to answer, and not waste time on the ones you don't. The GMAT scoring algorithm is more complicated than 'count your right answers', but the best strategy is the same: answer what you can, and don't invest time when it won't lead to a right answer.

That said, if you do know how the algorithm works mathematically, you can draw certain important conclusions about the consequences of certain response patterns, and so you can draw the most probable interpretations from the studies presented in this thread. So when I have time over the next few days, I'll try to point out where I think people have arrived at incorrect conclusions from these studies.

The GMAT takes absolutely no notice of how long you spend on a question. If it did, I and many other Quant experts would be penalized for answering some questions in five seconds (which is possible for certain question types if you either know the best way to think about them, or you've seen similar problems before). If anyone is telling you otherwise, ignore anything they're saying about the test and find more reliable sources of information, and if you do want to finish the test without leaving questions unanswered, don't leave yourself extra time to do that because you're worried the test might think you're guessing if you answer too quickly. Use all the time you can to solve problems and get right answers.

Brutal. GMAT, in my opinion, doesn't give you enough chances to redeem yourself after you miss an "easy" question, which is likely what happened to you.

This analysis is GOLD!!!
Was continuously scoring V35-V37 in GMAT Prep, Manhattan and Veritas Tests.
Took the GMAT late December and scored a V40.
And I think this was mainly because of my time-management. Did not leave any question, Left a couple of questions in the middle to balance the time.

Can't thank souvik101990 and Bunuel enough.

Here are some estimates based on the hundreds of student ESRs from actual GMATs (not the GMATPrep software, which has a much different scoring algorithm) that I've seen thus far.

VERBAL (30 counted questions per test)

51 = 0 wrong
48 = 1 wrong
47 = 2 or 3 wrong
46 = 3 wrong
42 = 5 wrong
40 = 7-10 wrong
35 = as few as 8 wrong or as many as 12 wrong, depending on where you get them wrong (see below)
etc.

QUANT (28 counted questions per test)

0-2 wrong: 51 (96%)
1-6 wrong: 50 (86%)
4-7 wrong: 49 (75%)
5-8 wrong: 48 (69%)
6-9 wrong: 47 (63%)
6-10 wrong: 46 (60%)
7-11 wrong: 45 (57%)
7-12 wrong: 44 (52%)
etc.

Sources: here and here and here and here

-Brian

GMAC has always been a bit vague about how they assess the penalty for not finishing a section, so no one will be able to give you an exact answer. There are two questions you might be asking: if you want to know "what did the test think my level was when time ran out?" then the answer is probably something like Q50/V36 or Q50/V37. If instead you're wondering "what score would I have gotten if I had guessed at the last few questions instead of letting time expire?" then the answer is probably more like Q49/V34 or Q49/V35 -- it would have helped you to finish, because the penalty for not finishing is fairly severe. But because most of the time you'd be guessing the last few questions incorrectly, it wouldn't change things all that much. If you could improve your pacing earlier in the test though, so you could finish without guessing randomly (usually the best way to do that is by moving on more quickly from the few questions you can't see how to answer) then that alone might get you to the Q50/V36 or Q50/V37 level, which is 710-720.

Suppose the following things were true:

• a test taker knew going into Verbal that he or she would need to guess at about 8 questions in order to finish
• on a fair cross-section of official questions, that test taker had a hit rate of 80% on SC and CR, and a hit rate of 30% on RC

then that test taker will almost certainly do better by guessing at two RC passages, and answering every SC and CR question properly. That would normally produce 4 additional correct answers (vs the 'worst case alternative' of answering all the RC, and guessing at eight SC and CR questions at the end). Difficulty level matters of course, but four additional right answers is pretty much always going to improve your score. I'd expect someone to gain between 1 and 3 Verbal points on average, in this exact situation, by guessing at RC instead of at random questions at the end. And it would be better to guess at the later RC questions, because they're more likely to be higher-level in this situation.

If instead someone only needs to guess at 4-5 questions, then with these hit rates, it very likely makes more sense to guess at just one RC passage, since you wouldn't need to guess at two entire passages to finish on time.

There are three considerations though:

• on a specific test, there's no way to know what strategy will be best without knowing about question difficulty. If you get RC passages that are easier than normal, guessing at them will hurt you more than normal. On some tests, this RC-guessing strategy will not work out in your favour, and there's no way to know when it will and when it won't. On average though, in this precise situation, it should work out favourably;

• I would expect that it's almost never true that a test taker has a true 'hit rate' of 80% on SC and CR, and only 30% on RC, when using a fair (i.e. covering all difficulty levels) sample of official questions. A 30% hit rate is barely better than random guessing. But RC and CR test moderately similar skills. Someone able to answer 80% of official CR questions correctly will, in most cases, be able to do much better than a random guesser at RC. So the above strategy guidance might be worthless in any practical situation. If someone told me they had an 80% hit rate at CR, and a 30% hit rate at RC, I'd want to know how they had measured that. Did they use official questions? Did they control for difficulty level? How big was their sample size? If the hit rates are closer together than 80% and 30%, the above advice could change.

• Regardless, anyone who needs to guess at 5-8 Verbal questions in order to finish is going to have a hard time getting a good Verbal score. If in addition, that person has a 30% hit rate at RC, I'd be surprised if that test taker could exceed a V32 or so no matter what strategy they used (and I would not be surprised if they got a V28 or something like that). Really rather than thinking about how to optimize timing strategy for Verbal, such a test taker would raise their score much more by improving their Verbal speed (which I know is hard to do) and improving their RC accuracy (which is probably going to be achievable with an 80% CR hit rate).

I'd add also that for higher level test takers in Verbal, it is almost never a good idea to guess through an entire RC passage, just because RC doesn't adapt by question, and any passage will have an easier question or two attached to it. Guessing at easier questions is not a good idea for higher level test takers. When a higher level test taker needs to guess in order to finish (and if the test taker is higher level, he or she presumably doesn't need to guess at 4-5 questions) it's usually best to guess at SC and CR near the end of the test, since those are the questions most likely to be difficult, and therefore are the questions that will be least harmful to get wrong.

No, neither of those things is really true. You can't draw conclusions about how the algorithm works in general from a single test. What you can draw conclusions about is how difficult the questions were on that test. So on your test, it's likely the questions early on weren't too difficult. When you get several of those wrong, the algorithm will have quite a bit of evidence that you're not a medium-level test taker. So you probably got easy questions for a while afterwards, and you had to get a lot of those right to recover. If instead your test had a lot of hard questions early on (which can happen), and you got several of them wrong, the test would only think you're probably not a high-level test taker. But if you then got most of your medium-level questions right after that, the algorithm would become persuaded that you are a medium-high or high-level test taker. Question difficulty is what matters, not only hit rate, and question difficulty varies a lot from one test to the next, so what was true on this one test you took may not be true on the next.

And one 500-level (say) Verbal question is "worth" the same as any other 500-level Verbal question. It doesn't matter if it's SC, CR or RC. The question types aren't weighted in any way. There is a difference between RC and the other two types of question, though. RC adapts by passage, not by question -- if the test gives you a "500-level" RC passage, that passage might have two 400-level questions and two 600-level questions. So if you guess at RC questions, or if you get some wrong that you wouldn't normally get wrong, it's more likely you'll be getting easy questions wrong than when you guess at SC or CR (which are more likely to be at your level). And if you get easy questions wrong on an adaptive test, that hurts your score the most. Of course a random RC question is also more likely to be above your level than a random SC/CR question -- RC question difficulty is just going to be more variable than SC/CR difficulty -- so it all depends which specific questions you get right and wrong.

It's still a good idea to try to do well early in the test, as long as you don't consume hours of time doing that. If you're at an above-average level, it's more likely you'll have manageable questions early on than later on, when the test starts challenging you with hard problems. So relax early, don't rush or panic about time, and answer as well as you can. If you do get a lot of RC early, then when you're looking at the clock, account for the fact that you've had a lot of the time-consuming questions already, and don't be alarmed if you see you only have 90-100 seconds per question left. Verbal pacing is a tricky subject and how you should do it depends on how long each question type takes and how much time you typically need to save, but if you finished this test 6 minutes early, it sounds like you can afford to slow down and you probably won't need to worry about pacing too much. Good luck!

bb OmerPelman

In the official GMAT website under the FAQ section of "What are GMAT® Official Practice Exams and Practice Questions?" it is stated that -
"Practice with real GMAT questions from past exams and full-length practice exams that use the same scoring algorithm as the actual exam."

The link - https://www.mba.com/frequently-asked-qu ... -questions

What about a scenario for verbal in that you miss every 3rd question (Get 2 right, miss 1, get 2 right, miss one etc)? (Like Bunnel's Quant Test)

I posted about this question in the previous GMATPrep scenarios thread, but I'll try one more time. To me, this experiment proves the opposite: that early questions are not more important than later ones. But we need to properly define our expectations in advance of a study like this: what score should we reasonably expect a test taker could achieve if they answer their first ten questions incorrectly?

From the conclusion drawn above, and the evidence used to support it (which I'm reading this way: "we get a Q50 with our first 21 right and last 10 wrong, but a Q30 with our last 21 right and first 10 wrong, therefore the first ten questions are especially important"), I'm inferring that some people expect you should get a similar score in those two scenarios. But you very clearly should not get even remotely similar scores in those situations (as I explain below). I find it remarkable you can even get a Q30 if you get your first ten questions wrong.

I'll simplify things a lot, to avoid any complicated math: for each question, the GMAT algorithm knows the probability that a 300-level test taker will answer correctly, and that a 500-level test taker will, and that a 700-level test taker will (and any other level of course). Using those probabilities, when the algorithm knows a test taker's right and wrong answers to many questions, the algorithm can work out using some probability theory what a test taker's most likely ability level is. That's all the scoring algorithm is really doing.

Now, let's ask "how likely is it that a 700-level test taker (so Q47 ish) would answer the first ten questions incorrectly?" We can't answer that precisely without knowing exactly how hard each question is, but since the test is adaptive, say we assume the first three questions are 500, 400, and 300 level, and then the next seven questions are all 300 level. GMAT questions, in the language of test theory, 'discriminate' very well: high level test takers rarely get easy questions wrong, and low level test takers rarely get hard questions right (except by lucky guessing, which the algorithm understands will happen 20% of the time or more). Using standard parameter values, a 700-level test taker will get a 300-level question right roughly 99% of the time. If that's not true for a certain test taker, that test taker simply is not at the 700-level, by definition. So you can already see how astronomically improbable it is that this 700-level test taker will get eight 300-level questions wrong in a row - it's (1/100)^8, or 1 in 10 quadrillion. Throw in the 400 and 500 level questions, then using the math behind the algorithm, the probability a 700-level test takers gets their first ten questions wrong is roughly 1 in 6,600,000,000,000,000,000. To put that in perspective, if one hundred 700-level test takers took the GMAT every single day since the beginning of the universe 14 billion years ago, this would still almost certainly never have happened: it would still be extremely unlikely that a 700-level test taker had answered their first ten questions incorrectly even one time, in 14 billion years. 700-level test takers simply never do this, so why should we expect the algorithm to give a Q47 score to someone who does, no matter how they perform later in the test?

Even the 500-level (Q30 ish) test taker almost never does this. It's 500,000 times more likely, if someone answers the first ten questions incorrectly, that a test taker is a 300-level test taker than a 500-level one, in the scenario I described above. In fact, the 300-level test taker also almost never does this -- even someone guessing completely at random is unlikely to do it -- but if anyone does, it's almost always someone at the absolute bottom of the scoring scale. That you can recover to reach a 500-level after that performance, even after proving to the test that you're worse than a domesticated cat pawing random answers at the keyboard, to me demonstrates that you can recover very successfully from an anomalously bad performance early in the test. Examining more realistic scenarios (while carefully defining expected results in advance) would demonstrate that.

The reverse happens when the test taker answers the first 21 questions correctly. It's extremely rare that even a Q50 level test taker will do this. You're well above a Q51 level if you do. So it shouldn't be surprising that your score doesn't drop too far after that point, even with an unusually bad performance at the end.

For test takers, the consequences are these:

- if you can get right answers early in the test, do - right answers are very valuable;
- if you can't for some questions, don't worry about it. The test can get hard early on, and a wrong answer to a hard early question doesn't hurt you any more than a wrong answer to a hard late question. You're supposed to get hard questions wrong no matter where they are in the test unless you're a top-level test taker;
- do not invest an inordinate amount of time early in the test. That will help you less than you might think early on, but will hurt you a lot later in the test.