Given its authoritative coverage of other science topics, the textbook

Lets break down the sentence: it starts with a modifier
Given its authoritative coverage of other science topics,
so what sould this modify? the chapter or the book itself? The intended meaning of the sentence is: there is a good book with a bad chapter.
This first part refers to the textbook itself: out A B and D.

C) the textbook contains a surprising and tentative chapter on genetics, which leads
E) the textbook is surprisingly tentative in its chapter on genetics, leading

the main difference between the remaning options is "which leads" VS "leading".
"which" refers to the preceding noun, so C could be written as
(...) surprising and tentative chapter on genetics, which (the genetic) leads (...)- this is clearly wrong.
The genetic cannot lead " one to doubt the author's scholarship in that particular area."
In E we find "leading", an example of COMMA + ING. This form express the results of the preceding clause most of the times, and that's what happens here:
(...) surprisingly tentative in its chapter on genetics, leading one to doubt the author's scholarship (...)- correct

Main point:
D is wrong because the initial modifer refers to the textbook.Given its authoritative coverage of other science topics, the chapter/ the book