Even minor cognitive disabilities such as impulsiveness and inattention can create frustration and cause aggressive behavior in children that, in turn, is likely to lead to the more serious problem of social rejection by their peers.
(A) that, in turn, is likely to lead to the more serious problem of social rejection by their peers Correct choice. "That, in turn, is likely to lead to the more serious problem .. by their peers" is a noun phrase modifier. It modifies the entire preceding complex noun phrase, i.e., aggresive behavior in C that, in turn, is likely to lead to X.
(B) who are likely, in turn, to lead to the more serious problem of social rejection by their peers This option means, children lead to the more serious problem. They do not lead to that. Cognitive disabilities do.
(C) that is likely of leading to the more serious problem, in turn, of social rejection by their peers Like of leading to, in turn, of social rejection. This is just a mess. Reject.
(D) likely to lead, in turn, to social rejection by their peers, a more serious problem ..behavior in children likely to lead to social rejection ... This is a mess too. This means, disabilities create issues in children likely to lead to social rejection. This does not make sense. Reject
(E) likely leading to a more serious problem of social rejection by their peers, in turnThis was a close one and had this option had "a comma" before it, I would have chosen this as the correct answer choice, as then it would have acted as an "Comma + Verbing modifier modifying the entire preceding clause". But, it is not set off by a comma rather modifies the preceding noun "children" which is a wrong modification. Reject