Yellow jackets number among the 900 or so species of the

For starters, the official guide explanations were written years -- and often a couple of decades -- after the original question was created. So the OG explanations often aren't a great reflection of the question-writers' intent, and in general, it's worth taking those explanations with a grain of salt.

In this case, the test for whether "which" is correct is twofold:

1) Is the modifier beginning with "which" reasonably close to the noun it modifies?
2) Does the verb in the modifier agree in number with the noun it describes?

With that in mind, take another look at the relevant portion in (C): "Yellow jackets number among the 900 or so species of the world’s social wasps, which means they live in a highly cooperative and organized society." The verb "means" is singular, so we know that the phrase beginning with "which" needs to modify a singular noun. But there aren't any singular nouns that "which" could possibly modify, so (C) is incorrect. (D) has the exact same problem.

And yes, you're correct that in most cases, "which" should modify the nearest noun. That's not a strict rule, though: as long as "which" modifies a nearby noun, it can be OK -- but the modifier needs to actually make sense and adhere to basic subject-verb agreement rules. And in (C) and (D), it simply doesn't.

I hope that helps!