During the Renaissance, scholars were uncertain as to the location of Troy, and by the eighteenth century many historians doubted that Troy had ever existed.
(A) doubted that Troy had ever existed (the best option)
Had + past participle denotes action before any ended action. Troy was before the 18th century and historians doubted about him thus it becomes action before any ended action.
(B) doubt that Troy has ever existed (incorrect: tenses error)
The whole passage is a narration which means it should be in the past simple
(C) were in doubt as to the existence of Troy (incorrect: tenses error)
Were in doubt is an exaggeration of doubted and because it is discussing Troy hence there must be past perfect tense.
(D) were doubtful concerning Troy’s existence (incorrect: tenses error)
The same as option C
(E) had doubts about Troy’s ever existing (incorrect: tenses error)
The same as option C. had doubts is an exaggeration of doubted.