Data: GMAT even adaptive? Same no of questions/diffic level

Question

UPDATE: As requested, I've uploaded finer charts showing how the test seems to adapt. The questions are sorted by order of appearance- I've done a bubble chart for the correct/wrong analysis, and a general trendline : You can see the way the test 'adapts' in terms of difficulty to a V28 scorer and a V47 scorer is just about the same weirdly!

https://imgur .com/a/wfeD1UY
https://imgur.com/a/wfeD1UY

ORIGINAL POST

Sorry for the click bait header, but this is an analysis that I just did. Apologies I couldn't come up with a better looking chart, but here you go.

https://imgur.com/a/YpiwtLe

I took each question from the gmatprep tests, found the link on gmatclub and sourced the difficulty level reported there - This is the interesting pattern that I found

Three different scores, from low to stellar: but almost the same number of 25% difficulty questions to 75%+ difficulty questions.

So is the GMAT, just in essence, a flat paper, that adapts the order of questions to a test taker?

Data from 3 GMAT Prep tests-Verbal Section

Test 1 - gmatprep1: V28 (left 5 questions in the end)
Test 2 - gmatprep2 :V47 (friends score)
Test 3 - gmatprep2: V36 (my score)

Lot of assumptions here, but in effect I've taken the difficulty reported on the forum at face value - and correlated data to it.

P.S: Would help if more of you folks share this, I can send my spreadsheet to you if you need to get started. It's as simple as copy pasting questions and noting down difficulty levels.

How it helped me : I read on Manhattan's book that the scoring algorithm looks at all your answers and finds which difficulty area you get around 50-60% of questions right. I noticed that I got around half the 35-45% difficulty wrong. But my accuracy was weirdly a little better in higher difficulty areas. Based on the hypothesis I'm predicting that is the zone keeping me away from V40. So I'm working on improving my accuracy there, I'll update here how this plan goes!

Posted from my mobile device

Attachments

XIv82gL.png [ 71.77 KiB | Viewed 2274 times ]

bb · Answer

Hi! Interesting analysis. Very cool! That took quite a bit of work to look up difficulty! Very Cool!!! I am pretty sure GMAT is adaptive but I am not going to defend GMAC either

There are a few things that come into play with the algorithm and that's progression of questions, experimental questions, and how many you get in a row right/wrong. You can learn a bit more here: https://gmatclub.com/forum/how-does-the ... 20297.html

sudharshan931 · Answer

Ah a reply from the MR.BB himself!

The header was a bit of a click bait haha, but my point was to show GMAT as a relatively flat test than what people think it is - which the data corroborates kind of. And of course I totally believe that some of the factors like not finishing the test, answering wrong in a row are all heavily penalized

It all started with a small excerpt I read from Manhattan's free book :

—----------------

The GMAT Scoring Algorithm for the Quantitative and Verbal Sections
Tests you took in school were generally based on the percentage of questions correct: the 
more you got right, the higher the score you received. As a result, you have been trained to take 
your time and try to get everything right when you take a test. This general strategy does not 
work on computer-adaptive sections of the GMAT. The Quant and Verbal scores are not based 
on the percentage of questions answered correctly. On the GMAT, most people actually answer 
similar percentages of questions correctly, typically in the 50% to 70% range (even at higher 
scoring levels). 
If test-takers all get a similar percentage correct, how does the GMAT distinguish among differ-
ent performance levels? “Regular” school tests gave everyone the same questions and perfor-
mance was determined based upon who could answer more of those same questions correctly. 
On the Quant and Verbal sections of the GMAT, everybody answers different questions, some 
easier, some harder. You can think of the GMAT as a test that searches for each person’s “60% 
level,” or the difficulty range in which the person is able to answer approximately 60% of the 
questions correctly. (This is not exactly what happens, but it’s a good way to think of the differ-
ence between “regular” tests and computer-adaptive tests.) Your score will be determined by 
the difficulty of the questions that you answer correctly versus the difficulty of those that you 
answer incorrectly.

-----------------------------

Thus this kind of analysis is helping me find out whether this works out as a good hypothesis and also most importantly find out which level of difficulty I should be focusing on in the forum questions on a much finer level!

Posted from my mobile device

GyMrAT · Answer

Hey  sudharshan931

Very interesting analysis.

Correct me if i am wrong, from your chart it seems like the test has a fixed number of questions for each difficulty level that it throws at the test taker, irrespective of whether the question is answered correctly or incorrectly. Hence the deduction that it is a flat test.

Curious to know how the chart will look, if we arrange the questions in the order they appeared on the test. Also highlighting each dot with a color code for Correct/Incorrect answer. Then you can super impose all the three charts in a single chart. It should be good to correlate the progression.

Regards,
GyM

sudharshan931 · Answer

Done! Here you go - the questions are sorted by order I've done a bubble chart for the correct/wrong analysis, and a general trendline : You can see the way the test 'adapts' in terms of difficulty to a V28 scorer and a V47 scorer is just about the same weirdly! 7h78SxW.jpg MJP5TLS.jpg F1GXX6J.jpg https://imgur.com/a/wfeD1UY

bb · Answer

Hi. Thanks for updating this project! It is starting to look more like a pattern.

Usually we say that the GMAT Score depends on the number of hard questions you answer correctly. I see in your first list/diagram you have the total number of questions by percentile, not just the ones that were answered correctly. I would suggest that you consider re-doing/re-working the very first chart with incorrect questions removed from it and you will see the difference. You can also do the same first chart with plotting only incorrect questions. I have no idea if it will be helpful - i am guessing less helpful than the mapping of correct questions only.

GyMrAT · Answer

Hi  sudharshan931

Awesome job bud! Definite pattern in there, wondering why the test throws a easier question than the previous one which was answered correctly or similarly throws a tougher question than the previous one which was answered incorrectly. I think it is because the questions are from a single RC passage. However if they are not RC questions & if it is a string of SC or CR questions, going down or up in difficulty, then that does put a big question mark on the adaptability of the test.

The first quarter & last quarter of V28 is really weird, questions are going down in difficulty even after answering correctly & going up in difficulty after getting answered incorrectly. I can understand if they go up in difficulty, which basically gives the test taker a chance to up the baseline. Not sure about the going down part though.

In the second quarter of V47, i wonder why the test lowered its difficulty, unless ofcourse that was an RC, it seems like the test robbed the test taker of the opportunity to score higher than V47, if instead those questions were of higher difficulty & were answered correctly, the base line could have been raised. Well your friend scoring a V47, needless to say, is going to rock the GMAT.

Thanks for doing this, bud! its good fun! Would love to see what more you come up with.

Cheers,
GyM

	Magoosh increases your score, guaranteed - 15% off ALL Magoosh GMAT plans with code GMATCLUB15 + Free GMAT Club Tests.
	Manhattan Prep: $79 Bonus Score Higher on Your GMAT Exam. Guaranteed.
	Target Test Prep - Save up to $325 $200 Discount + $79 Bonus GMAT Club Tests
	e-GMAT: 60% Off on GMAT Focus Get to 735+ from any starting point
	Experts' Global - ~~$1720~~ $960 End-to-End GMAT Prep + App Support: 150+ Videos, 2000+ Questions, 15 FLTs +
	GMATWhiz - $299 ~~$499~~ Experience the course through a free trial.
	GMAT Club Tests - Free Included with every course purchase of $149+ View all deals

Data: GMAT even adaptive? Same no of questions/diffic level

Prep Toolkit

Top 5 GMAT Debrief Videos

615 to 715 in 15 Days (Ria's Strategies)

More than Quant/Verbal Abilities: Speed vs. Accuracy

745 with Only Self-Prep and GMAT Club

From 555 to 765: 210-point Improvement

Perfect 805 Score Debrief by Julia

My Rewards