It is currently 18 Oct 2017, 02:43

Close

GMAT Club Daily Prep

Thank you for using the timer - this advanced tool can estimate your performance and suggest more practice questions. We have subscribed you to Daily Prep Questions via email.

Customized
for You

we will pick new questions that match your level based on your Timer History

Track
Your Progress

every week, we’ll send you an estimated GMAT score based on your performance

Practice
Pays

we will pick new questions that match your level based on your Timer History

Not interested in getting valuable practice questions and articles delivered to your email? No problem, unsubscribe here.

Close

Request Expert Reply

Confirm Cancel

Events & Promotions

Events & Promotions in June
Open Detailed Calendar

Conclusions from GMAT algorithm study based on ESR analyses

  new topic post reply Question banks Downloads My Bookmarks Reviews Important topics  
Author Message
TAGS:

Hide Tags

8 KUDOS received
Senior Manager
Senior Manager
User avatar
B
Joined: 27 Mar 2016
Posts: 338

Kudos [?]: 111 [8], given: 26

Location: United States (CO)
GMAT 1: 770 Q51 V44
GPA: 3.69
WE: Analyst (Consulting)
Conclusions from GMAT algorithm study based on ESR analyses [#permalink]

Show Tags

New post 02 Aug 2016, 20:01
8
This post received
KUDOS
8
This post was
BOOKMARKED
Thanks to GMAT Club members sharing their ESR's, especially in this thread: information-on-new-gmat-esr-report-beta-221111.html, I now have access to about 40 ESR's. While more reports are definitely needed to help draw robust conclusions, here are a few things that I have noticed and are quite confident about.

1. The distribution of questions by quarter:



-Verbal: 10 (2 exp) - 10 (3 exp) - 10 (3 exp) - 11 (3 exp); 41 questions including 11 experimental
-Quant: 9 (2 exp) - 9 (2 exp) - 9 (2 exp) - 10 (3 exp); 37 questions including 9 experimental

You see, there are A LOT of experimental questions in each section.

2. Sectional weighting



Whether the first 10 questions are more heavily weighted than others has been a subject of debate in our community for a while. I have dug through a lot of the ESR's I have my hands on to draw this important conclusion:

There is no evidence that the first 10 questions are more heavily weighted than the rest of the section. In fact, I think we have nearly sufficient evidence to conclude the opposite: that the first 10 questions are not more heavily weighted.

I have a few reports where the test taker gets several questions wrong in the first quarter but destroys the rest of the section and ends up at the same place as another test taker who scores perfectly in the first quarter but gets more questions wrong later on. It is rare for someone to not do so hot in the first part of the section but then improve significantly in the remainder of the section when the questions tend to get progressively harder if he is doing well. However, it is very common for the opposite scenario to occur. Quite understandably, the more questions you answer correctly, the more difficult the questions tend to get and thus the easier it is to answer subsequent questions incorrectly.

But the bottom line is, don't try too hard in the first quarter at the expense of the rest of the section.

3. Sectional difficulty



This is the conclusion that puzzled me the most:

Sometimes, when you're doing too well, e.g. answering all questions correctly, the difficulty of questions drops.

When you face less difficult questions, each question you answer wrong tends to get more heavily penalized. Perhaps the GMAC wants to make sure you've got all your bases covered so you don't answer hard questions right but easy questions wrong?

4. Sub-sectional scaled score and percentile



Not sure how to post a scaled score vs percentile table without making it look messy, so I'm probably not going to post all percentiles; rather, I'm going to share a few observations I consider interesting:

- Sentence Correction is the hardest sub-section to score perfectly on. A 51 on SC warrants a 99th percentile, whereas the same feat in CR bears a 98th percentile rank, and that in RC would give you a mere 95th percentile.
- Reading Comprehension is the hardest sub-section to score 40+ on. In my sample, there are a few 51's but nothing between 42 and 50.


I apologize for the messy formatting job; this post is a very rough draft and will be continually updated as I have more time for analyses and reformat and as more ESR's become available. If someone that's skilled in formatting can jump in to help with making the content visually appealing, that would save me some time and effort to concentrate on other pieces. :wink:

We need more input. Your contribution to this thread will be super helpful: information-on-new-gmat-esr-report-beta-221111.html

Kudos [?]: 111 [8], given: 26

1 KUDOS received
Senior Manager
Senior Manager
User avatar
B
Joined: 27 Mar 2016
Posts: 338

Kudos [?]: 111 [1], given: 26

Location: United States (CO)
GMAT 1: 770 Q51 V44
GPA: 3.69
WE: Analyst (Consulting)
Conclusions from GMAT algorithm study based on ESR analyses [#permalink]

Show Tags

New post 02 Aug 2016, 20:03
1
This post received
KUDOS

5. Impact of number of questions wrong on scores



These are the rough ranges for the number of questions people have got wrong to achieve a certain score:

Q51: 0-2
Q50: 2-4
Q49: 4-8
Q48: 6-9
Q47: 7-11
Q46: 10
Q45: 10-11

Keep in mind that these are out of the 30 Quant questions that count

V48: 1
V47: 2
V46: 3
V44: 3-5
V42: 4
V41: 4-6
V40: 7
V38: 8-11
V37: 8-9
V36: 9-10
V35: 9-12
V34: 7-11

Keep in mind that these are out of the 28 Verbal questions that count

Kudos [?]: 111 [1], given: 26

Senior Manager
Senior Manager
User avatar
Joined: 31 Mar 2016
Posts: 406

Kudos [?]: 78 [0], given: 197

Location: India
Concentration: Operations, Finance
GMAT 1: 670 Q48 V34
GPA: 3.8
WE: Operations (Commercial Banking)
GMAT ToolKit User Premium Member
Re: Conclusions from GMAT algorithm study based on ESR analyses [#permalink]

Show Tags

New post 02 Aug 2016, 23:05
HiLine how did you conclude that the exam contains experimental questions of 11 or 9 within verbal and quant? Unable to understand that.

Sent from my GT-N7100 using GMAT Club Forum mobile app

Kudos [?]: 78 [0], given: 197

Top Contributor
4 KUDOS received
Senior Manager
Senior Manager
User avatar
S
Status: Professional GMAT Tutor
Affiliations: AB, cum laude, Harvard University (Class of '02)
Joined: 10 Jul 2015
Posts: 404

Kudos [?]: 477 [4], given: 53

Location: United States (CA)
Age: 37
GMAT 1: 770 Q47 V48
GMAT 2: 730 Q44 V47
GMAT 3: 750 Q50 V42
GRE 1: 337 Q168 V169
WE: Education (Education)
Conclusions from GMAT algorithm study based on ESR analyses [#permalink]

Show Tags

New post 03 Aug 2016, 00:10
4
This post received
KUDOS
Top Contributor
1
This post was
BOOKMARKED
HiLine wrote:

There is no evidence that the first 10 questions are more heavily weighted than the rest of the section. In fact, I think we have nearly sufficient evidence to conclude the opposite: that the first 10 questions are not more heavily weighted.

I have a few reports where the test taker gets several questions wrong in the first quarter but destroys the rest of the section and ends up at the same place as another test taker who scores perfectly in the first quarter but gets more questions wrong later on. It is rare for someone to not do so hot in the first part of the section but then improve significantly in the remainder of the section when the questions tend to get progressively harder if he is doing well. However, it is very common for the opposite scenario to occur. Quite understandably, the more questions you answer correctly, the more difficult the questions tend to get and thus the easier it is to answer subsequent questions incorrectly.

But the bottom line is, don't try too hard in the first quarter at the expense of the rest of the section.



First of all, to clarify: the opposite of "more heavily weighted" is "less heavily weighted." I do not believe that's what you mean, lest someone reading this interpret it as such. Either way, I must disagree: the "myth" of the first 9-10 questions persists because it's not a myth at all. Despite what the GMAC and others want you to think, both the ESRs and my own anecdotal experience tutoring GMAT students for 15 years have supported the idea that the first 9-10 questions have a larger impact on your score than do any other group of 9-10 questions.

I'm not sure whether we should draw too many firm algorithm-related conclusions from the ESRs, given that for some reason the difficulty level of the questions (Correct / Incorrect / All) is indicated as only 4 average data points, and because of the small sample size / inherently complicated nature of judging the impact of early questions vs. the impact of how many questions the student got wrong in a row (information unavailable), combined with the vague nature of splitting the test into quarters and averaging it all out, rather than providing a question-by-question breakdown. It is likely that question #10 is more impactful than is question #18, but they are in the same quarter of questions, and thus we have no way to differentiate them. Also, every GMAT is different, and we have learned from my study of the IR section that GMAT "raw to scaled" score conversions are not necessarily consistent from test to test.

That being said, I will concede that the ESRs have given us some good insight here, and it's clear that a solid performance across the test can sometimes be as effective as an "excel early in the test" strategy (more on this in a second). Although the GMAT Prep tests use a different algorithm, the results I've seen from real GMATs definitely reinforce the importance of the first 9-12 questions. And in my opinion, so do the limited number ESRs that have been shared on GMAT Club thus far.

Let's use the Quant ESRs as an example. There are 28 Quant questions that count, which means that each quarter of the test consists of 7 real questions and 2 experimental ones (3 in the last quarter).

Here is a screenshot of a student getting off to a great start (100% correct on the first half = 14 out of 14!) and scoring a solid 47 on Quant despite basically falling apart at the end and running out of time (approximately 71% of questions correct overall):

To be more explicit, this person answered 14 of the first 14 (counted) questions correctly, but only 6 of the last 14, and still scored Q47! This, to me, clearly supports the conventional wisdom that getting off to a good start on the the first 1/2 of each GMAT section is indeed the most important requirement for a solid score.

Image

In contrast, here is an example of a student also getting about 71% correct (5 out of every 7), but struggling more at the beginning and thus receiving a lower score:

Image

Here is a screenshot of a student achieving an Q47 score with a more consistent performance, despite a lower percent correct total and struggling in the less-important last quarter of questions (approximately 64% total correct), which shows that you don't have to ace the first 7 questions that count, but that getting at least 5 of them correct (5/7 = 71%) is very helpful. This person managed to leverage his/her relatively low percent correct into a solid score, because of the relatively high difficulty of the questions that he/she had earned in the beginning, and maintained throughout the test.

Image

Here is a student scoring a relatively low Q42 (51%) despite getting approximately 68% of the questions correct and a putting in a generally consistent performance, with slightly more struggles in the important middle half of questions:

Image

Here is another student, who also got about 68% of the questions correct, but who struggled at the beginning and was thus penalized mightily (Q35 = 33%):

Image

You will notice that all of the students whose ESRs I have included here have answered somewhere between 64% and 71% of the total questions correctly, but their scores ranged from 35 to 47. The difference in most cases? Students' performance on the first quarter, and to a lesser degree, their performance on the middle half of the section.

The ESRs do suggest, at the very least, that although earlier questions do still matter more than later ones, the GMAT algorithm might not be quite as "front-loaded" as it used to be. The 2nd and 3rd quarters of questions are also very important, but the 4th quarter, not so much (other than making sure that you take the time to answer all the questions before time runs out, thus avoiding the score penalty).

One of my main concerns with trying to interpret this data is that the "average difficulty" chart is very deceptive, and not nearly as descriptive as it first appears. It appears to chart your entire GMAT journey, but it's really just 4 average data points tied together by straight lines. One aspect of the chart that does stand out, however, is that there is a significant jump in difficulty from the first question (medium difficulty) to the 9th or 10th question, when the student either performs well (earning harder questions) or poorly ("earning" easier questions). The jumps in difficulty at the start of the 3rd and 4th quarters of the test are generally less severe than are the jumps in the first half of the test.

We also see odd features in there, such as the difficulty of incorrectly answered questions being indicated as medium, when the student in fact answered 100% of the questions in that quarter correctly.
_________________

Harvard grad and 770 GMAT scorer, offering high-quality private GMAT tutoring, both in-person and via Skype, since 2002.

McElroy Tutoring


Last edited by mcelroytutoring on 03 Sep 2017, 12:24, edited 54 times in total.

Kudos [?]: 477 [4], given: 53

Top Contributor
2 KUDOS received
Senior Manager
Senior Manager
User avatar
S
Status: Professional GMAT Tutor
Affiliations: AB, cum laude, Harvard University (Class of '02)
Joined: 10 Jul 2015
Posts: 404

Kudos [?]: 477 [2], given: 53

Location: United States (CA)
Age: 37
GMAT 1: 770 Q47 V48
GMAT 2: 730 Q44 V47
GMAT 3: 750 Q50 V42
GRE 1: 337 Q168 V169
WE: Education (Education)
Conclusions from GMAT algorithm study based on ESR analyses [#permalink]

Show Tags

New post 03 Aug 2016, 00:16
2
This post received
KUDOS
Top Contributor
Senthil7 wrote:
HiLine how did you conclude that the exam contains experimental questions of 11 or 9 within verbal and quant? Unable to understand that.


He's correct on that, as dabral has already noted. It has to do with using the "percentile correct" number from the ESRs to determine the correct denominator (total number of non-experimental questions) for each quarter of the test. We know this because the GMAC does not provide correct/incorrect information for the experimental questions.

IR has 9 questions that count, and 3 experimental. We can tell that the IR section has exactly 9 non-experimental questions because the "percent correct" always aligns with a denominator of 9.
Quant has 28 questions that count, and 9 experimental. We can tell that the Quant quarters are split into 7 non-experimental questions each because the "percent correct" always aligns with a denominator of 7.
Verbal has 30 questions that count, and 11 experimental. We can tell that the Verbal quarters are split into 7 non-experimental questions (middle two sections) or 8 non-experimental questions (first and last sections) each because the "percent correct" always aligns with a denominator of 7 (2nd and 3rd quarters) or 8 (1st and 4th quarters).
_________________

Harvard grad and 770 GMAT scorer, offering high-quality private GMAT tutoring, both in-person and via Skype, since 2002.

McElroy Tutoring

Kudos [?]: 477 [2], given: 53

Expert Post
Veritas Prep Representative
User avatar
Joined: 22 Apr 2004
Posts: 1114

Kudos [?]: 219 [0], given: 57

Location: Southern California
Schools: Kellogg MBA 2004
Re: Conclusions from GMAT algorithm study based on ESR analyses [#permalink]

Show Tags

New post 04 Aug 2016, 10:00
Expert's post
1
This post was
BOOKMARKED
Great stuff, HiLine!! You have zeroed in on at least a couple of important truths about the test.

Even though some people are convinced that GMAC isn't speaking the truth about the importance of the first few questions on a GMAT section, here are some interviews that we conducted with Larry Rudner, former Chief Psychometrician at GMAC:

http://www.veritasprep.com/gmat/talking ... testmaker/

The first video is aptly titled "The Myth of the First 10 Questions." This myth will always persist, I bet, but Dr. Rudner does his best here to put it to rest.
_________________

Scott

Veritas Prep | GMAT Prep | MBA Admissions Consulting | Co-author, Your MBA Game Plan

Save $100 on Veritas Prep GMAT Courses And Admissions Consulting
For a limited time, receive access to five Veritas Prep Computer Adaptive practice tests when you purchase a Veritas Prep GMAT book! Buy Now!
Enroll now. Pay later. Take advantage of Veritas Prep's flexible payment plan options.

Veritas Prep Reviews

Kudos [?]: 219 [0], given: 57

Senior Manager
Senior Manager
User avatar
B
Joined: 27 Mar 2016
Posts: 338

Kudos [?]: 111 [0], given: 26

Location: United States (CO)
GMAT 1: 770 Q51 V44
GPA: 3.69
WE: Analyst (Consulting)
Re: Conclusions from GMAT algorithm study based on ESR analyses [#permalink]

Show Tags

New post 04 Aug 2016, 10:07
mcelroytutoring wrote:

You will notice that all of the students whose ESRs I have included here have answered somewhere between 64% and 71% of the total questions correctly, but their scores ranged from 35 to 47. The difference in most cases? Students' performance on the first quarter, and to a lesser degree, their performance on the middle half of the section.



There is one more difference, a big one, which you did not notice.

The difficulty of questions.

A student who answered 10 easy questions incorrectly did not perform as well as a student who answered 10 difficult questions incorrectly. (This is obviously a hypothetical situation and in no way refers to any of the ESR's you posted.)

Kudos [?]: 111 [0], given: 26

Top Contributor
Senior Manager
Senior Manager
User avatar
S
Status: Professional GMAT Tutor
Affiliations: AB, cum laude, Harvard University (Class of '02)
Joined: 10 Jul 2015
Posts: 404

Kudos [?]: 477 [0], given: 53

Location: United States (CA)
Age: 37
GMAT 1: 770 Q47 V48
GMAT 2: 730 Q44 V47
GMAT 3: 750 Q50 V42
GRE 1: 337 Q168 V169
WE: Education (Education)
Conclusions from GMAT algorithm study based on ESR analyses [#permalink]

Show Tags

New post 04 Aug 2016, 10:20
Top Contributor
HiLine wrote:
mcelroytutoring wrote:

You will notice that all of the students whose ESRs I have included here have answered somewhere between 64% and 71% of the total questions correctly, but their scores ranged from 35 to 47. The difference in most cases? Students' performance on the first quarter, and to a lesser degree, their performance on the middle half of the section.



There is one more difference, a big one, which you did not notice.

The difficulty of questions.

A student who answered 10 easy questions incorrectly did not perform as well as a student who answered 10 difficult questions incorrectly. (This is obviously a hypothetical situation and in no way refers to any of the ESR's you posted.)


I did notice it (the change in difficulty level of the questions), but I didn't mention it because there's no way to precisely quantify its effect. But the difficulty of questions is directly tied to one's performance in the first 9-12 questions, which in my opinion the data supports as crucial (see various examples above), so in many ways they are the same issue.

If you don't do well up-front, then you will get easier questions. Doing well on easier questions does not raise your score nearly as much as doing well on medium/hard questions. Thus, the first 1/2 of questions is the most important, because that is where you "earn" the harder questions to follow.

You will notice in the last example that I provided that the student got 43% correct in the first quarter of questions (3 out of 7). He/she then went on to get 6 of the next 7 questions right (86%), but by then it was too late...the difficulty level had dropped too low and the student's score could not fully recover, despite a solid performance on the final 3/4 of the section.

Image
_________________

Harvard grad and 770 GMAT scorer, offering high-quality private GMAT tutoring, both in-person and via Skype, since 2002.

McElroy Tutoring


Last edited by mcelroytutoring on 04 Aug 2016, 12:21, edited 4 times in total.

Kudos [?]: 477 [0], given: 53

Top Contributor
1 KUDOS received
Senior Manager
Senior Manager
User avatar
S
Status: Professional GMAT Tutor
Affiliations: AB, cum laude, Harvard University (Class of '02)
Joined: 10 Jul 2015
Posts: 404

Kudos [?]: 477 [1], given: 53

Location: United States (CA)
Age: 37
GMAT 1: 770 Q47 V48
GMAT 2: 730 Q44 V47
GMAT 3: 750 Q50 V42
GRE 1: 337 Q168 V169
WE: Education (Education)
Conclusions from GMAT algorithm study based on ESR analyses [#permalink]

Show Tags

New post 04 Aug 2016, 12:08
1
This post received
KUDOS
Top Contributor
quixx23 wrote:
Even though some people are convinced that GMAC isn't speaking the truth about the importance of the first few questions on a GMAT section, here are some interviews that we conducted with Larry Rudner, former Chief Psychometrician at GMAC:

http://www.veritasprep.com/gmat/talking ... testmaker/

The first video is aptly titled "The Myth of the First 10 Questions." This myth will always persist, I bet, but Dr. Rudner does his best here to put it to rest.


GMAC does not always tell the truth about the test. For years, we were told that experimental ("pretest") questions made up 10-25% of the test. Turns out, they make up 25.6% of the GMAT. It was a white lie, yes, but a lie nonetheless. GMAC also continues to indicate on its website that Verbal and Quant scores range from 0 to 60, when in fact the highest score on either section is a 51.

Seeing that representatives from GMAC have provided incorrect and misleading information about the GMAT in the past, I'm not sure that it's wise to take them at their word when it comes to the scoring algorithm. Instead, I would trust the data first.

Along those lines, it would be more helpful to see actual ESRs that support this contention, rather than vague promises from GMAC.
_________________

Harvard grad and 770 GMAT scorer, offering high-quality private GMAT tutoring, both in-person and via Skype, since 2002.

McElroy Tutoring


Last edited by mcelroytutoring on 08 May 2017, 08:32, edited 6 times in total.

Kudos [?]: 477 [1], given: 53

Expert Post
Veritas Prep Representative
User avatar
Joined: 22 Apr 2004
Posts: 1114

Kudos [?]: 219 [0], given: 57

Location: Southern California
Schools: Kellogg MBA 2004
Conclusions from GMAT algorithm study based on ESR analyses [#permalink]

Show Tags

New post 04 Aug 2016, 12:28
mcelroytutoring wrote:
quixx23 wrote:
Even though some people are convinced that GMAC isn't speaking the truth about the importance of the first few questions on a GMAT section, here are some interviews that we conducted with Larry Rudner, former Chief Psychometrician at GMAC:

http://www.veritasprep.com/gmat/talking ... testmaker/

The first video is aptly titled "The Myth of the First 10 Questions." This myth will always persist, I bet, but Dr. Rudner does his best here to put it to rest.


The GMAC also claims that Verbal and Math scores range from 0 to 60, when in fact the highest score on either section is a 51. Seeing that representatives from GMAC have provided incorrect and misleading information about the GMAT in the past, I'm not sure that it's wise to take them at their word when it comes to the scoring algorithm. Instead, I would trust the data first.

Along those lines, it would be more helpful to see actual ESR's that support this contention, rather than vague promises from GMAC.



I have attended many GMAC-led GMAT Test Prep Summits over the years, and every time they give the same answer: The scoring range has always officially been 0-60, but the range that they actually use tops out at 51. It's an artifact of how they originally set the scoring scale vs. what they actually ended up needing to use. No one from GMAC has ever dangled the false hope for test takers that they might somehow attain something higher than a 51.

Tell you what: You trust your data and keep combing through ESRs. We'll keep training students in the higher-order thinking skills needed to actually earn higher scores!
_________________

Scott

Veritas Prep | GMAT Prep | MBA Admissions Consulting | Co-author, Your MBA Game Plan

Save $100 on Veritas Prep GMAT Courses And Admissions Consulting
For a limited time, receive access to five Veritas Prep Computer Adaptive practice tests when you purchase a Veritas Prep GMAT book! Buy Now!
Enroll now. Pay later. Take advantage of Veritas Prep's flexible payment plan options.

Veritas Prep Reviews

Kudos [?]: 219 [0], given: 57

Top Contributor
Senior Manager
Senior Manager
User avatar
S
Status: Professional GMAT Tutor
Affiliations: AB, cum laude, Harvard University (Class of '02)
Joined: 10 Jul 2015
Posts: 404

Kudos [?]: 477 [0], given: 53

Location: United States (CA)
Age: 37
GMAT 1: 770 Q47 V48
GMAT 2: 730 Q44 V47
GMAT 3: 750 Q50 V42
GRE 1: 337 Q168 V169
WE: Education (Education)
Re: Conclusions from GMAT algorithm study based on ESR analyses [#permalink]

Show Tags

New post 04 Aug 2016, 12:49
Top Contributor
quixx23 wrote:

Tell you what: You trust your data and keep combing through ESRs. We'll keep training students in the higher-order thinking skills needed to actually earn higher scores!


I do all of those things. Call me a multitasker, I guess. ; )
_________________

Harvard grad and 770 GMAT scorer, offering high-quality private GMAT tutoring, both in-person and via Skype, since 2002.

McElroy Tutoring

Kudos [?]: 477 [0], given: 53

VP
VP
User avatar
D
Joined: 15 Jul 2015
Posts: 1012

Kudos [?]: 224 [0], given: 4

Location: India
GMAT 1: 780 Q50 V51
GRE 1: 339 Q170 V169
Re: Conclusions from GMAT algorithm study based on ESR analyses [#permalink]

Show Tags

New post 04 Aug 2016, 22:28
HiLine wrote:
There is no evidence that the first 10 questions are more heavily weighted than the rest of the section. In fact, I think we have nearly sufficient evidence to conclude the opposite: that the first 10 questions are not more heavily weighted.
That's good to know, because the test would be unfair if this weren't the case. Here are a few points others going through this thread could consider:

1. I don't think the questions are "weighted" in "sets" such as the first 10. It's more that the confidence level of the ability estimate changes over the course of the test. Think of it as all the information coming from all the questions (till that point) put together. At the end of the test, the score that a test taker receives is the one with the highest probability out of a range of possible scores.

2. Performance does not depend solely on the difficulty level of a question. There are certain questions that are better able to differentiate between different ability levels, and the information that such questions provide is inherently more valuable than the information provided by questions of the same difficulty level which are not able to differentiate as well.
_________________

Profile and Official GMAT score report

Kudos [?]: 224 [0], given: 4

Senior Manager
Senior Manager
User avatar
B
Joined: 27 Mar 2016
Posts: 338

Kudos [?]: 111 [0], given: 26

Location: United States (CO)
GMAT 1: 770 Q51 V44
GPA: 3.69
WE: Analyst (Consulting)
Re: Conclusions from GMAT algorithm study based on ESR analyses [#permalink]

Show Tags

New post 06 Aug 2016, 09:54
mcelroytutoring wrote:

I did notice it (the change in difficulty level of the questions), but I didn't mention it because there's no way to precisely quantify its effect.


That something cannot be precisely quantified does not justify the complete omission of it. I'm glad you have incorporated difficulty in the subsequent discussion. :wink:

mcelroytutoring wrote:

But the difficulty of questions is directly tied to one's performance in the first 9-12 questions,


As it is to that in all other questions.

mcelroytutoring wrote:
If you don't do well up-front, then you will get easier questions. Doing well on easier questions does not raise your score nearly as much as doing well on medium/hard questions.


Easier questions are easier to answer by definition. If someone does well on easier questions and not so well on more difficult questions, the algorithm is working well.


mcelroytutoring wrote:
Thus, the first 1/2 of questions is the most important, because that is where you "earn" the harder questions to follow.


Earning harder questions means nothing if you cannot answer these questions correctly. Focusing on the first half of the section at the expense of the second half is only going to result in poor performance and reversion of difficulty level in the second half. The first half of the questions make up half of the battle. The second half is still ahead. And if you're a good student, the second half of the battle is even more fierce than the first.

mcelroytutoring wrote:
You will notice in the last example that I provided that the student got 43% correct in the first quarter of questions (3 out of 7). He/she then went on to get 6 of the next 7 questions right (86%), but by then it was too late...the difficulty level had dropped too low and the student's score could not fully recover, despite a solid performance on the final 3/4 of the section.


No it was not too late; rather, the test taker did not perform well enough on easy questions to be challenged with difficult questions. In fact, that the test taker answered only about half of the questions correctly in the first 7 counted questions suggests that he was unlikely to be able to answer many challenging questions anyway.

Kudos [?]: 111 [0], given: 26

Senior Manager
Senior Manager
User avatar
Joined: 31 Mar 2016
Posts: 406

Kudos [?]: 78 [0], given: 197

Location: India
Concentration: Operations, Finance
GMAT 1: 670 Q48 V34
GPA: 3.8
WE: Operations (Commercial Banking)
GMAT ToolKit User Premium Member
Conclusions from GMAT algorithm study based on ESR analyses [#permalink]

Show Tags

New post 06 Aug 2016, 10:05
So now we are being driven to conclude that no part of the exam must be given undue importance or be neglected, rather focus on every question! This is contrary to what I have read on top prep companies websites and many topics in here. So skipping questions (as promoted by certain prep companies and many individuals) is just a sign of moving over inability and nothing tactical as no one knows which questions is experimental.
.
In such a case, would a sound advise be to work on every question and not simply guess and move on?

Last edited by Senthil7 on 06 Aug 2016, 10:23, edited 1 time in total.

Kudos [?]: 78 [0], given: 197

Top Contributor
Senior Manager
Senior Manager
User avatar
S
Status: Professional GMAT Tutor
Affiliations: AB, cum laude, Harvard University (Class of '02)
Joined: 10 Jul 2015
Posts: 404

Kudos [?]: 477 [0], given: 53

Location: United States (CA)
Age: 37
GMAT 1: 770 Q47 V48
GMAT 2: 730 Q44 V47
GMAT 3: 750 Q50 V42
GRE 1: 337 Q168 V169
WE: Education (Education)
Conclusions from GMAT algorithm study based on ESR analyses [#permalink]

Show Tags

New post 06 Aug 2016, 10:11
Top Contributor
HiLine wrote:

No it was not too late; rather, the test taker did not perform well enough on easy questions to be challenged with difficult questions. In fact, that the test taker answered only about half of the questions correctly in the first 7 counted questions suggests that he was unlikely to be able to answer many challenging questions anyway.


So in other words, performing well on the first 7 counted questions is extra important, which is what I've been trying to argue all along. That's a very small sample size, and it helps explain why the first part of each section is crucial to achieving a solid GMAT score. Make a few mistakes up-front due to nervousness, carelessness, bad luck, or some other reason, and there is almost no way for your score to recover significantly, no matter how well you perform on the remainder of the section.

Why am I the only person who is including any actual data in my arguments? I would love to see some actual ESRs (with screenshots, please) that support your contentions. We are operating somewhat in the dark here due to the vague nature of the statistics on the ESRs, so I will concede that this issue is far from settled.
_________________

Harvard grad and 770 GMAT scorer, offering high-quality private GMAT tutoring, both in-person and via Skype, since 2002.

McElroy Tutoring


Last edited by mcelroytutoring on 06 Aug 2016, 15:29, edited 2 times in total.

Kudos [?]: 477 [0], given: 53

Senior Manager
Senior Manager
User avatar
B
Joined: 27 Mar 2016
Posts: 338

Kudos [?]: 111 [0], given: 26

Location: United States (CO)
GMAT 1: 770 Q51 V44
GPA: 3.69
WE: Analyst (Consulting)
Re: Conclusions from GMAT algorithm study based on ESR analyses [#permalink]

Show Tags

New post 06 Aug 2016, 10:21
mcelroytutoring wrote:
HiLine wrote:

No it was not too late; rather, the test taker did not perform well enough on easy questions to be challenged with difficult questions. In fact, that the test taker answered only about half of the questions correctly in the first 7 counted questions suggests that he was unlikely to be able to answer many challenging questions anyway.


So in other words, the first 7 counted questions are really important, which is what I've been trying to argue all along.


You're arguing with a straw man then. No one is saying the first 7 counted questions are not really important. All counted questions are really important.

mcelroytutoring wrote:
That's a very small sample size, and it helps explain why the first part of the GMAT is crucial to achieving a solid score. Make a couple of careless mistakes due to nervousness, and there is almost no way for your score to recover.


Doesn't matter where you make careless mistakes. Answering easy questions wrong is going to do a lot of damage.


mcelroytutoring wrote:
Why am I the only person who is including any actual data in my arguments?


Your samples are pretty good for the purpose of the discussion. Feel free to bring more to the table if you so wish. :wink:

Kudos [?]: 111 [0], given: 26

Top Contributor
Senior Manager
Senior Manager
User avatar
S
Status: Professional GMAT Tutor
Affiliations: AB, cum laude, Harvard University (Class of '02)
Joined: 10 Jul 2015
Posts: 404

Kudos [?]: 477 [0], given: 53

Location: United States (CA)
Age: 37
GMAT 1: 770 Q47 V48
GMAT 2: 730 Q44 V47
GMAT 3: 750 Q50 V42
GRE 1: 337 Q168 V169
WE: Education (Education)
Conclusions from GMAT algorithm study based on ESR analyses [#permalink]

Show Tags

New post 06 Aug 2016, 10:24
Top Contributor
Allow me to clarify, then: getting questions wrong at the beginning hurts your score more than does getting questions wrong at the end. The data from the ESRs supports that conclusion, until I am shown otherwise with actual evidence.
_________________

Harvard grad and 770 GMAT scorer, offering high-quality private GMAT tutoring, both in-person and via Skype, since 2002.

McElroy Tutoring


Last edited by mcelroytutoring on 05 Jun 2017, 12:32, edited 1 time in total.

Kudos [?]: 477 [0], given: 53

Senior Manager
Senior Manager
User avatar
B
Joined: 27 Mar 2016
Posts: 338

Kudos [?]: 111 [0], given: 26

Location: United States (CO)
GMAT 1: 770 Q51 V44
GPA: 3.69
WE: Analyst (Consulting)
Re: Conclusions from GMAT algorithm study based on ESR analyses [#permalink]

Show Tags

New post 06 Aug 2016, 10:45
mcelroytutoring wrote:
Allow me to clarify, then: getting questions wrong at the beginning hurts your score more than does getting questions wrong at the end. The data from the ESRs supports that conclusion, until I am shown otherwise...you know, with actual evidence.


The ESR's that you have posted do not support that conclusion.

What you are doing is comparing two scenarios that are not equivalent.

A candidate that answers 3 questions wrong at the beginning and answers the rest of the questions correctly is going to score less than one that answers 3 questions wrong at the end and answers the rest of the questions correctly, because the latter tackles more challenging questions at the same level of accuracy and thus should be deemed a more competent candidate.

Kudos [?]: 111 [0], given: 26

Top Contributor
2 KUDOS received
Senior Manager
Senior Manager
User avatar
S
Status: Professional GMAT Tutor
Affiliations: AB, cum laude, Harvard University (Class of '02)
Joined: 10 Jul 2015
Posts: 404

Kudos [?]: 477 [2], given: 53

Location: United States (CA)
Age: 37
GMAT 1: 770 Q47 V48
GMAT 2: 730 Q44 V47
GMAT 3: 750 Q50 V42
GRE 1: 337 Q168 V169
WE: Education (Education)
Conclusions from GMAT algorithm study based on ESR analyses [#permalink]

Show Tags

New post 06 Aug 2016, 15:14
2
This post received
KUDOS
Top Contributor
HiLine wrote:
The ESR's that you have posted do not support that conclusion.

What you are doing is comparing two scenarios that are not equivalent.

A candidate who answers 3 questions wrong at the beginning and answers the rest of the questions correctly is going to score less than one that answers 3 questions wrong at the end and answers the rest of the questions correctly, because the latter tackles more challenging questions at the same level of accuracy and thus should be deemed a more competent candidate.


What I am hearing from most of you who disagree with me is that "test-takers who do well on the first quarter of questions are better GMAT students who earn harder questions, and thus deserve better GMAT scores". Overall, of course this is true, but it does not change my point about the unfairly "front-loaded" nature of how GMAT scores are determined.

My point is that 9 questions is not large enough of a sample size to fully determine someone's "initial" GMAT skill level, and hence there is an element of luck involved in scoring well. Answer fewer than 5 out of the first 7 counted questions correctly, and your chances for an elite score are almost zero. Throw in the fact that 2 of those first 9 questions are experimental and don't count toward your score, and this adds even more randomness into the equation. You could answer 7 out of 9 correctly, and the 2 you got wrong were experimental (100% correct), or you could get 7 out of the 9 right, but 2 of the correct answers could have been experimental (5/7 = 71%). This is of course not only true about the first quarter of the test, but the entire GMAT, where slightly more than 25% of the questions are experimental.

Yes, it is true that correct responses (normally--see below) earn harder questions, and consistently answering medium/hard questions correctly is the best path to a great GMAT score. But the student won't even get the chance to try hard questions if the GMAT algorithm makes early, "snap" judgements about his/her skill level, which it does. Anyone who is out there reading this: don't take my word for it--take a look at the ESRs and decide for yourself! The evidence is readily available if you're willing to take a look: http://gmatclub.com/forum/information-o ... 21111.html

Any statistician knows that in order to provide an accurate measurement across a large population, you need a large sample size. A score estimate based on a small sample size of GMAT questions is more prone to fluctuations based on lucky / unlucky guesses and other random elements. But perhaps this is exactly what the GMAC wants, so that you will keep paying $250 to take the GMAT again and again.

Also, the first questions are not easy difficulty--they are medium difficulty.

What's interesting about the first 10 questions "myth" is that the GMAC does not directly dispute it...instead it just uses vague, roundabout language such as "all questions count." Yes, all questions count, but that doesn't mean that earlier questions don't affect your score more strongly than do later questions, because in fact they do, and the data from the ESRs supports this.

The excerpt below is taken directly from the OG 2016:

"Myth -vs- FACT–
M
- The first 10 questions are critical and you should invest the most time on those.
F – All questions count. It is true that the computer-adaptive testing algorithm uses the first 10 questions to obtain an initial estimate of your ability; however, that is only an initial estimate. As you continue to answer questions, the algorithm self-corrects by computing an updated estimate on the basis of all the questions you have answered, and then administers items that are closely matched to this new estimate of your ability. Your final score is based on all your responses and considers the difficulty of all the questions you answered. Taking additional time on the first 10 questions will not game the system and can hurt your ability to finish the test."


Like the rest of the GMAT, this text requires a close reading. Notice that not even the GMAC disputes that the first 10 questions have a larger impact on your final score than does any other group of 10 questions. Yes, spending too much time on the first 10 can hurt your score if you don't do so correctly (you need to save time for the remaining 27 questions, of course), but this does not change the fact that doing well on those first 10 questions is essentially mandatory for a top GMAT score. GMAT calls it an "initial estimate," but this is simply a euphemism for a snap judgment. Yes, adjustments to one's score continue to be made throughout the test, but the "Average Difficulty" charts on the ESRs support the notion that the largest adjustments in difficulty/scoring occur early in the test, and a close reading indicates that the GMAC does nothing to directly dispute this notion (although it tries its best to imply that this is untrue).

For example, the below performance of (broken down by quarters) 100% / 100% / 57% / 29% on Quant earned a 47 out of 51 (also mentioned in my original post). Not bad for getting more than half of the final 14 counted questions wrong.

Image

To some perfectionistic students, a score of Q47 would be a disappointment. But to many others, it would be a dream come true. This ESR shows that you can essentially turn the GMAT Quant into a 25-question test, take aggressive guesses on the final 12, and still achieve an above-average Quant score.

Should you "invest the most time" on the first 10 questions? Of course not--that's barely 25% of the questions. The GMAC is setting up its own straw man here, to avoid addressing the real issue at hand. You should not spend most of your time on the first 10 questions, but you should spend more time per question on those questions than you do on the remaining ones, given their outsized importance to the scoring algorithm. It is true that for elite scorers, the questions will get harder and harder, and you might suffer at the end due to timing issues, ruining your chance for a 50/51 on Quant, or something above 45 on Verbal...trust me, I've been there. But this only separates the 730 scores from the 770s, or the 750s from the 800s, and all of those scores are great. To crack 700, leveraging the GMAT algorithm is key.

Also, you mentioned in your original post (heading #3) that the difficulty level seemed to drop, despite the student answering questions correctly, and that you weren't sure why.

HiLine wrote:
This is the conclusion that puzzled me the most:

Sometimes, when you're doing too well, e.g. answering all questions correctly, the difficulty of questions drops.


I think that this passage (also from the OG 2016) should clarify the issue for you:

"Myth -vs- FACT-
M
– Getting an easier question means I answered the last one wrong.
F – Getting an easier question does not necessarily mean you got the previous question wrong. To ensure that everyone receives the same content, the test selects a specific number of questions of each type. The test may call for your next question to be a relatively difficult problem-solving item involving arithmetic operations. But, if there are no more relatively difficult problem-solving items involving arithmetic, you might be given an easier item. Most people are not skilled at estimating item difficulty, so don’t worry when taking the test or waste valuable time trying to determine the difficulty of the questions you are answering."

_________________

Harvard grad and 770 GMAT scorer, offering high-quality private GMAT tutoring, both in-person and via Skype, since 2002.

McElroy Tutoring


Last edited by mcelroytutoring on 03 Jun 2017, 15:20, edited 4 times in total.

Kudos [?]: 477 [2], given: 53

Top Contributor
Senior Manager
Senior Manager
User avatar
S
Status: Professional GMAT Tutor
Affiliations: AB, cum laude, Harvard University (Class of '02)
Joined: 10 Jul 2015
Posts: 404

Kudos [?]: 477 [0], given: 53

Location: United States (CA)
Age: 37
GMAT 1: 770 Q47 V48
GMAT 2: 730 Q44 V47
GMAT 3: 750 Q50 V42
GRE 1: 337 Q168 V169
WE: Education (Education)
Information on NEW GMAT ESR REPORT - BETA - Upload yours here [#permalink]

Show Tags

New post 06 Aug 2016, 15:38
Top Contributor
HKD1710 wrote:
mvictor wrote:
HKD1710 wrote:
Luckily, the snapshots for quant provided here gives a big insight here for the first 10 questions:

HKD1710 (Q45) - 57 57 71 100

mvictor (Q49) --- 71 57 71 100

We should go for more analytics with more data :)



yeap, huge difference!!!
i believe that first 10 questions set the difficulty mode for the whole section :)
compare the difficulty level from your pic with mine.


Yes, "difficulty" decision making of the algorithm is a catch. It seems the algorithm decides everything at the very start of section!


Finally, someone who agrees with me... : ) hiline-gmat-algorithm-studies-223122.html#p1718357

I would also like to point out that those numbers represent a difference of only one correct question in the 1st quarter, yet they resulted in a 4-point score differential! Despite what the GMAC and others might try to tell you, the "myth" of the outsized importance of the first 9-10 questions is anything but.

4/7 counted questions in the 1st quarter = 57%

5/7 counted questions in the 2nd quarter = 71%
_________________

Harvard grad and 770 GMAT scorer, offering high-quality private GMAT tutoring, both in-person and via Skype, since 2002.

McElroy Tutoring

Kudos [?]: 477 [0], given: 53

Information on NEW GMAT ESR REPORT - BETA - Upload yours here   [#permalink] 06 Aug 2016, 15:38

Go to page    1   2    Next  [ 40 posts ] 

Display posts from previous: Sort by

Conclusions from GMAT algorithm study based on ESR analyses

  new topic post reply Question banks Downloads My Bookmarks Reviews Important topics  

Moderator: HiLine



GMAT Club MBA Forum Home| About| Terms and Conditions| GMAT Club Rules| Contact| Sitemap

Powered by phpBB © phpBB Group | Emoji artwork provided by EmojiOne

Kindly note that the GMAT® test is a registered trademark of the Graduate Management Admission Council®, and this site has neither been reviewed nor endorsed by GMAC®.