Teratomas are unusual forms of cancer because they are composed of tissues such as tooth and bone not normally found in the organ in which the tumor appears.
Option Elimination -
Let's get the basics right first. The key here is to explain "Why" they are unusual. Adverbs are the ones that help explain the action as "why," "where," "how," and "when." "Because" a subordinating conjunction introduces the subordinating clause OR "prepositional" phrase introduced by "in that" - both are adverbial in nature, so either because of or in that is okay here without making much fuss. But if you know the meaning difference here between "because" used for cause and effect and "in that" used to introduce an intrinsic property, it's a plus. Even if you don't know you are okay, both adverbials serve the purpose of explaining "why." "Why" they are unusual. There are enough other issues to eliminate options.
Option elimination -
(A) because they are composed of tissues such as tooth and bone - ambiguous. It's not clear whether "not normally found in the organ in which the tumor appears" modifies "tooth and bone" or "tissues." Wrong.
(B) because they are composed of tissues like tooth and bone that are - even worse, just like A.
(C) because they are composed of tissues, like tooth and bone, tissues - use such as to introduce examples.
(D) in that their composition, tissues such as tooth and bone, is - it conveys a wrong meaning: "their composition is not found in the organ in which the tumor appears." The intended meaning is that "tissues not normally found in the organ in which the tumor appears."
(E) in that they are composed of tissues such as tooth and bone, tissues - ok