I am confused between the following two sentences He

Daagh is right as far as he pursues this. The credited answer must make sense if you replace the pronoun with its antecedent. He is also correct that "the student's progress was made in English" is a clause, and so not parallel with "the student's progress in Maths" (though the GMAT is American, and so would use "math").

That's not the only problem with your first option, though. Notice that even if you drop the word "was," so that substitution of the antecedent yields a fine sentence, "what" would still be wrong. Why? Frankly, the grammatical reasons are probably too abstruse to bother about, so I'd just treat this as a diction/idiom-ish question, and say "trust your ear," which advice I almost never give.

In case you're in the mood for abstruse grammatical issues, though... "What" is not an ordinary pronoun like it, it's, they, them, their; nor a relative pronoun, like that, which, , who, whom, when, where; nor even a demonstrative pronoun like this, that, these, those.

Huh? Well, let me give a slightly technical account, and then a simple bottom line.

The slightly technical account: "what" is usually used as what is called an "interrogative pronoun." You use it to ask questions ("What should we order for lunch?") or to describe questions ("Find out what the special at Los Cantaros is.") I believe that I have seen the interrogative "what" used on the GMAT SC in the latter way, but never in a contested part of the question. So we probably don't need to worry about that.

"What" can also be used in some circumstances in which "that which" or "the thing(s) that" would be grammatically correct but awkward ("What I like even better than the sopa is the birria.") I have seen the "what" used on the GMAT SC in this way, but not in a contested part of the question. So we probably don't need to worry about that.

"What" does a few additional jobs in Standard Written English, but those jobs are even less relevant to the GMAT.

The simple bottom line: if the real GMAT gives you a choice between some other pronoun and "what", choose the other pronoun, unless your ear demands otherwise. If you have a terrible ear for this sort of thing, look for some other split first.