My answer could be a bit meta, but I genuinely think that our getting disturbed by the factors around is because there is a disturbance within. My guess is that you are rather confident of your skills in Quant than in Verbal? If that is the case, it explains why the construction sounds don't disturb your Quant score. Your mind is happy to be engaged in Quant (rather than focus on the noise) because it is not intimidated by Quant. On the other hand, if at any point you feel a sense of fear for Verbal, your mind will try to distract you to something else (the noise in your case) because our natural disposition is not to face our fears. So, accept two things 1) you cannot always control your environment or circumstances and if things go out of your control like disturbance in the test room as has been recorded by many GMAT takers, you cannot spiral down. and 2) today you may not feel too confident in Verbal, but tomorrow, the confidence will grow a bit and so on as you as you go with the practice. I know you may want a quick and effective fix to the problem and it might be irritating you to not find one, but the more you force to look for a solution, the more you will get farther from the sense of calm you are looking for in the first place.
Breathe. It will get better. And try meditating for minutes daily if you can. Hope this helps!