Does CAT randomness in difficulty assignment adversely affect scores?

Question

I am aware that the number of correct answer is not the sole factor in determining scores. What concerns me is the assignment of questions of different difficulties in my recent Princeton Practice Test.
In spite of higher accuracy (Quant) and similar distribution of correct answers (Verbal), the system seems reluctant to release more difficult questions to me in PT9 (compared to my previous personal best), which I attribute my low scores to.

Quantitative: PT9 49pts (35/37) vs PT5 51pts (31/37)

Attachment:

Q49 Princeton PT9 35-37.JPG [ 153.67 KiB | Viewed 1649 times ]

Attachment:

Q51 Princeton PT5 31-37.JPG [ 153.66 KiB | Viewed 1662 times ]

Verbal: PT9 27pts (31/41) vs PT8 40pts (29/41)

Attachment:

V27 Princeton PT9 31-41.JPG [ 141.63 KiB | Viewed 1643 times ]

Attachment:

V40 Princeton PT8 29-41.JPG [ 140.79 KiB | Viewed 1642 times ]

If my interpretation is valid, does such randomness put candidates in unfair disadvantage during actual GMAT CAT?

MartyTargetTestPrep · Answer

Hi gmatretest.

Probably what has happened is that you have been scoring high on the Princeton practice tests and thereby used up most of the hardest questions in the question bank. So there aren't any more to give you.

Obviously the real GMAT is not going to run out of hard questions. So you will not experience similar issues when taking the real test.

mikemcgarry · Answer

Dear gmatretest , I'm happy to respond. :-)

I think MartyMurray already address one very important issue. A few more things to keep in mind. 1) The CAT procedure designed by the folks at GMAC is their own secret proprietary knowledge. Any GMAT CAT systems offered by a private company is that company's best guess on how GMAC designs their CAT. 2) Truly brilliant folks who are world's experts in psychometrics work at GMAC, and they are the ones who oversee the GMAT CAT. I don't think many private companies have comparable expertise. I have tremendous respect for the brilliant folks at MGMAT and their CATs, but I wouldn't trust many other companies to have CATs that are particularly like the GMAT CAT. 3) I will say that one thing I find laughable are the "difficulty" levels listed in your charts. Difficulty levels listed on some arbitrary scale from 1 to 10 or something like this? Really? In most serious psychometric research, difficulty levels are expressed a decimal percents: in other words, a difficulty of 0.43 means that 43% of test takers in the pool get those questions correct. The fact that difficulties are expressed in terms of a kiddie scale here causes me to question what their other psychometric assumptions might have been in designing this CAT. MartyMurray is right. GMAC has a question pool that is truly enormous. You could take 12 GMATs and never see a single repeated question. Running out of questions is not a problem. Here's an article that discusses the GMAT CAT a little further: What is the GMAT CAT (Computer Adaptive Test)? Does all this make sense? Mike :-)

EMPOWERgmatRichC · Answer

Hi gmatretest,

The scoring algorithm on the Official GMAT is far more complicated than most people realize. Since that algorithm is proprietary, no GMAT company has an exact match for it, thus CAT scores can vary a bit based on the 'biases' involved in their respective designs. As such, you really shouldn't be spending your time trying to figure all of that out. A far more useful gauge would be to review each CAT and determine how many questions you SHOULD have gotten correct, but didn't (due to a silly/little mistake). Those mistakes are the things that you have to fix to score at a higher level.

There's clearly a bigger-picture issue here, but you haven't defined what you're *really* interested in; all of this ties into Test Day though, so we should probably start there:

1) How have you scored on each of your practice CATs (including the Quant and Verbal Scaled Scores for each)?
2) What is your goal score?
3) When are you planning to take the GMAT?

GMAT assassins aren't born, they're made,
Rich

gmatretest · Answer

Thanks for sharing

Does it mean I need to forgo Princeton from now on, since I might have exhausted its question banks? I am not sure if gmathopeful90 refers to me. Here is the overall scoring for my PT and also actual GMAT on Feb 26. gmat princeton PT 5-9.JPG Plan to aim for 670 in the Apr 8 test. Will complete 1 PT (Manhattan, Kaplan, Peterson's, and GMAT Prep in standby mode) every 1-2 days until Apr 6, and top up with OG revision if time allows. Full-time work at daytime, therefore unable to fully commit to preparation.

EMPOWERgmatRichC · Answer

Hi gmatretest,

Considering the 'swings' in these CAT scores, there's either some problem with how these CATs function or a problem with your performances (or how you're taking these CATS).

When you took your CATs:
1) Did you take them at home?
2) Did you take them at the same time of day as your Official GMAT?
3) Did you ever do ANYTHING during your CATs that you couldn't do on Test Day (pause the CAT, skip sections, take longer breaks, etc.)? 
4) Did you ever take a CAT more than once?

GMAT assassins aren't born, they're made,
Rich

mikemcgarry · Answer

Dear gmatretest , My friend, it depends on what your needs are. For ordinary question practice, you could still use Princeton. As I have said above, I think the best GMAT CATs from a private company are the MGMAT tests, and you can get full access to them by purchasing any one of their lesson books for math , Sentence Correction , Critical Reasoning , or Reading Comprehension . Of course, the very best estimates are from the official material: GMAT Prep and the related question packs. Finally, you should be aware that many people perform at one level on practice tests and then see a drop, at times a significant drop, on test day. For a deeper understanding of this, and what you can do, see: Lower on the Real GMAT than on Practice Tests Does all this make sense? Mike :-)

gmatretest · Answer

First 4 were taken at home, but time slots vary. Some are on weekend mornings, while some on weekday nights (after work). No pause, no skip, no break. 
The 5th (PT9) was taken in office, AWA-IR-Q in the afternoon and V in the next morning. 
All 5 PTs are not repetitive. 
My Official GMAT was taken on Friday morning, while the upcoming one will be on Friday afternoon.

EMPOWERgmatRichC · Answer

Hi gmatretest,

The extra information that you've provided is quite useful, as it helps to define the fluctuations in your scores - AND what you'll have to do to score higher. In real basic terms, you took your CATs in such a way that you were NOT properly training to face the Official GMAT - so you were not ready to face it on Test Day.

Test Day is a rather specific 'event' - the details are specific and they matter, so you have to train as best as you can for all of them. The more realistic you can make your CATs, the more likely the score results are to be accurate. The more you deviate, the more "inflated" your scores can become - and that's what happened here. By taking the CATs at home, taking them at different times of day, not taking breaks, etc., you weren't properly training for the FULL GMAT 'experience.'

Thankfully, this is a relatively easy set of problems to fix. The big question now is "how long will it take you to properly get 'used to' taking CATs in a way that will mirror what you'll face on Test Day?" You'll certainly need a new set of practice CATs to work with and you have to put in the necessary time to train your brain (and body) for the realistic GMAT. You might also need to invest in some new GMAT training materials.

GMAT assassins aren't born, they're made,
Rich

MartyTargetTestPrep · Answer

From what you said, it does appear that you have exhausted the Princeton question banks.

While going forward those tests might be good sources of practice questions, for representations of the test experience, you need to move onto a new set of tests.

	Manhattan Prep: $99 Bonus Score Higher on Your GMAT Exam. Guaranteed.
	Target Test Prep - Save up to $599 $400 Discount + $99 Bonus GMAT Club Tests
	e-GMAT: 60% Off on GMAT Focus Get to 735+ from any starting point
	Magoosh increases your score, guaranteed - 15% off ALL Magoosh GMAT plans with code GMATCLUB15 + Free GMAT Club Tests.
	Experts' Global - ~~$1720~~ $960 End-to-End GMAT Prep + App Support: 150+ Videos, 2000+ Questions, 15 FLTs +
	GMATWhiz - $299 ~~$499~~ Experience the course through a free trial.
	GMAT Club Tests - Free Included with every course purchase of $149+ View all deals

Does CAT randomness in difficulty assignment adversely affect scores?

Prep Toolkit

Top 5 GMAT Debrief Videos

615 to 715 in 15 Days (Ria's Strategies)

More than Quant/Verbal Abilities: Speed vs. Accuracy

745 with Only Self-Prep and GMAT Club

From 555 to 765: 210-point Improvement

Perfect 805 Score Debrief by Julia

My Rewards