GMAT Prep Software Analysis and What If Scenarios

There are exactly 9 (not 13) experimental questions on Quant out of the 37 total questions. 28 are counted. If you happen to get really, really lucky, then maybe 9 of the 12 of the questions you get wrong are experimental. That still means that you will get 3 counted questions wrong, which could perhaps be enough for a Q50 but definitely not enough for a Q51.

It is worth noting that GMAC is now stating on its website that "The practice exams use the same scoring algorithm as the official GMAT® exam."

On the old version of the GMATPrep desktop software, the scoring algorithm was NOT the same as that of the real GMAT, in part because the real GMAT used to have 23 experimental questions, and on the GMATPrep desktop software, every question counted.

Yet, GMAC is now telling us the opposite: that the new online tests use the same scoring algorithm as the real GMAT. However, unless there are also 12 questions on the GMATPrep online tests that are experimental/pretest/unscored, this is not completely true.


Source: GMAC Website

Theoretically, however, there is nothing stopping GMAC from throwing 12 experimentals into the GMATPrep software as well--especially since the software is now online and can be updated at any time. Until we run some controlled, detailed simulations with the new online tests, there is no way to know whether some of the questions count, and some don't. If there truly are 12 unscored questions, then maybe the new GMATPrep software (online version) really DOES now have the same scoring algorithm as the real GMAT.

Or maybe it's just marketing hype. Time will tell...

-Brian

There is no chance that is true. For one thing, if you're a well above average test taker (as you are), your experimental questions will be, on average, easier than your other questions - your experimental questions are selected before you even start your test, and they span the entire difficulty range, so they'll mostly seem easy to a Q48 level test taker. But experimental questions also need to be inserted at random locations in each test so that the data GMAC collects is meaningful, and is not biased in any way. There aren't many experimental questions on the GMAT these days regardless. You also can't see harder questions on one test because of your choice of section order, because then the results for two different test takers in, say, Quant wouldn't be directly comparable. Each section of the test needs to be the same for everyone. Now, it is true that the test can start hard, or start easy, just by luck, and of course the test adapts, so if you're getting off to a good start in your first section, that section can start to get hard quickly. That's probably what happened for you, considering your scores.

No test taker can have a large enough sample of official tests to draw any reliable conclusions about the scoring or question selection algorithm. Test takers won't gain anything by reading them, but there are many published papers about how the algorithm works (including official GMAC research reports, along with academic papers explaining the theory), and anything I've said in this thread or elsewhere about the algorithm comes from those.

Hi Kavya2024,

Those scores are so close that there is effectively no difference between them. Keep working on your concepts, and try not to worry too much about exactly where in your test you ended up getting questions wrong.

You could also go through this post and this post.

Ok here I go, I hope this doesn't get taken down due to being a childish quesiton but:

about CR placement and its affect on score? Most of my wrong in the Verbal Section are CR and they seem pretty darn insignificant. I am still hitting mid 80s percentile in verbal while getting the majority of CR questions wrong

I'm also confused on how to devise a pacing strategy for verbal as I had done with Math by the correctness/guessing based on the number of question.

and I give me corresponding 15minutes OR 8 questions for quant a couple of minutes plus or minus depending on wher ethey fall into that general pace.

For verbal as of now I mentally just give earlier questions some extra time and pace myself through the rest. It seems like verbal is much less rigid when it comes to results and pacing.
How would I devise a pacing strategy for verbal then since, from what I understand, the placement of the SC/RC is random and more important is what comes first?

Awesome work, Bunuel and Vercules!

It's interesting to see that omitting the last seven quant questions is so deadly on GMATPrep, since the fine folks at GMAC insist that there isn't a huge difference between guessing and omitting questions at the end of each section. They published an official blog post on that issue a couple of years ago (https://officialgmat.mba.com/2009/09/17/ ... g/#more-57) and there's also an accompanying research study on the GMAC website (https://www.gmac.com/~/media/Files/gmac/ ... ssWhat.pdf).

Basically, GMAC argues that there isn't really a meaningful difference between guessing and omitting the last five questions on verbal. They also argue that you're better off omitting the last five if you're doing badly on the quant section, and you're better off guessing only if you're a "high ability" test-taker on the quant section. Interesting.

I guess there are a few ways to look at the differences between the GMATPrep results and the GMAC's official statements. It's possible that the GMATPrep software runs a substantially different algorithm than the actual test, but I think it's more likely that the dramatic result in Bunuel's study is the result of an extra couple of questions (omitting seven questions probably does substantially more damage than missing five) and the fact that he is clearly in the "high ability" category.

A large proportion of GMAT Club members will ultimately do well on quant, so it's best for most members to guess at the end of the quant section instead of omitting questions. But on the verbal section--or for test-takers who aren't strong at quant--maybe it doesn't really matter all that much?

WE DO NOT KNOW THE ACTUAL GMAT ALGORITHM.
It is in some ways and it is not in others. The official algorithm is much more complex and has multiple checks/verifications to prevent fraud/cheating.
If you have to skip as many questions, it is a tough situation. Try to get a better handle on your timing.

Hi Bunuel,

May I ask which version of the GMAT Prep Software did you use for this analysis?
And how did you reset the software? Because in v2.2.306 we have to login using our mba.com account so I am not sure whether the reset methods given in another post still work.

Thanks.

2.1.294 version.

In that version (at least) there is a reset button under "Take a practice exam" page.

These kinds of GMATPrep experiments are fine, but it's extremely easy to arrive at incorrect conclusions from them, unless you understand how the scoring algorithm works. Some of the conclusions drawn in this thread are not correct, though in some cases it will take me some time to explain why. I'll try to do so over a few posts in the coming days. For example, here:



There is a GMAC research report which explicitly states that they control for 'answer position' when designing the GMAT. That is, no one answer choice is right more often than another. I've also verified that independently; looking over more than 1000 official questions, each answer choice is correct roughly equally often.

That said, if you look only at a small sample of questions, one answer choice might appear to be correct more often than another, just because of random variance. If you do a binomial probability calculation, then if each answer choice has a 20% chance to be correct on a random GMAT question, it would not be all that improbable for one answer choice to be correct 13 times on one GMAT Quant test (that should happen on about 2% of tests). Since your samples do not appear to be independent (I assume you were using the same GMATPrep test repeatedly, in which case it would not be surprising to get similar results each time because of question repetition), the most likely explanation for your findings is pure random luck.

But if instead you assume that your finding is meaningful, and that, when guessing randomly at every question, C is the most likely answer to be correct, what would that finding mean? It might seem paradoxical, but it means that test takers who need to guess almost certainly should not guess C. If you accept (and all the evidence I have confirms this) that each answer is correct about 20% of the time, then if C is the right answer more often when you guess at every question, that means C is more often the right answer on the absolute easiest questions. For C then to be correct 20% of the time overall, C would need to be correct less often on the harder questions. And those are the questions people need to guess at.

Even if you don't believe each answer is right 20% of the time, knowing which answer is more often correct on 200-level questions is of no help to test takers unless they'll need to guess at 200-level questions, which almost no one needs to do. So it would not be a relevant finding for most test takers anyway. But all of the evidence I've seen suggests to me that no one answer choice is a better guess than another, if you need to guess purely at random. The best strategy in that case is simply to guess as quickly as possible.

A surprising turn of events. In the Official GMAT Monthly this was featured in their FAQ section:

Q: Are the practice GMAT tests on the software indicative of how well you'll do on the actual GMAT?
A: GMATPrep® software uses real GMAT test questions and the actual GMAT CAT® test delivery and scoring algorithm. It should help you to gauge your preparedness for the GMAT exam. Note - the predictive value of these tests may be affected by the extent to which actual testing conditions are adhered to during practice testing sessions. For example, if you were to suspend testing and resume the test later, an option that you have with GMATPrep software but not with the actual GMAT CAT exam, you might score higher on GMATPrep test than you would otherwise.


Previously it was stated/believed that the GMAT Prep and GMAT used different algorithms (a more simplified algorithm was used in GMAT Prep)

I'm almost certain it is not true, and never has been true, that GMATPrep used a different basic scoring algorithm from the real test. I've never heard any official statement that would make me think otherwise, and if there has been one, I'd love to see a link.

The fundamental scoring algorithm used in GMATPrep really has to be the same as is used in the actual GMAT, because the scoring is just based on probability theory, and GMAC isn't free to change the laws of math. There are differences between GMATPrep and the GMAT (for example, GMATPrep does not insert experimental questions in the way the real test does), but those differences don't have to do with the actual scoring of the test.Those differences almost certainly still exist, so I don't think this recent announcement is new information, or signals any change in the GMATPrep tests.

apolo

I know for sure that on Quant you do not need to answer 100% of the questions right to get Q51. I have missed problems on the real GMAT and still got a Q51. And on GMATPrep, one can miss 6 questions and still get a Q51. I don't know how it is on the Verbal section of the GMAT, but my guess would be that it would be the same.

Now, I do know that if you miss say only one question and that happens to be of an easy or medium dificulty then for sure you will not get the highest scaled score on either Quant or Verbal.

Cheers,
Dabral

Took GMATprep Test 3 a few days ago and received a 40 on Verbal (12 questions incorrect) (2 of them in the 1st 10 questions)

Just took Prep 4 and received a V36 (Only 6 questions incorrect)! However, I answered 4 of the first 10 questions incorrectly. After this I do not think I received any questions that were all that challenging.

Therefore there has to be something to the 1st 10 questions myth.

Anyone else see results so skewed?

Thank you for sharing - yes, absolutely have noticed this a number of times taking the GMAT Prep. It does not seem to be a Myth at all.
I think the way that GMAC should explain the first 10 question myth should be different. I think their take should be that - It is impossible to to "trick" or "beat" the test by acing the first 10 questions and then do a poor job on the rest. However, they should also mention that if you screw up the first 10, it is your funeral. At the same time, I can see how they may be reluctant to talk about the statement #2 (screwing up the first 10) without immediately everybody jumping to the conclusion that they MUST do the first 10 right.

The bottom line is that you do need to get as many HARD questions right as possible and the only way to get to them is to do well on the first 10 questions because as you have noticed, you did not get any hard challenging questions down the road after stumbling on a few in the first 10.

Best Regards,
BB

Recently, someone told me that if I am doing good in the real GMAT exam, I would be seeing more of DS questions and less of PS. Is this true? Can some people who have given the exam recently, shed light on the same? Apologies if this is the wrong thread to post my query.

Hi Guys,

Just wanted to know , if these are true for actual GMAT as well ? I have been answering close to 75% of the answers correctly in GMAT Prep but the score is no where close to 600 wondering where should I improve. Will the score be different in the actual GMAT? If it is true for GMAT Prep and not the real GMAT, any other source which you guys can recommend for practice tests and which would give a score something closer to real GMAT.

Awaiting a quick reply as my GMAT is due soon.

Regards