Points to consider..
1. The Modifier, " Whose behaviour...." Modifies birds and should be placed next to it.
2. The verb noticed should be in PAST tense as the non-underlined initial part is in PAST

These errors sufficient to eliminate all but D
Ans D
_________________

Here is my understanding of the sentence:

Museum-goers who glanced up into the building’s five-story atrium often notice birds nesting in the rafters, whose behavior was being studied by the museum’s staff.

The sentence talks about museum goers who glanced up into the building and found something. This sentence is written in past tense and the museum goers glancED and FOUND.
Subject Verb:
1) Museum goers (S)....................Notice (V)
2) Who (S)....................Glanced (V)
3) Behavior(S)..................Was being studied (V)

Pronoun:
1) Who refers to plural noun museum goers
2) Whose refers to plural birds

Modifiers:
1) who glanced up into the building’s five-story atrium : Clearly refers to the museum goers
2) whose behavior was being studied by the museum’s staff: Should refer to birds as it does not make sense to say that rafter's behavior was being studied.

Tense Error:
The sentence is written in past tense. The museum goers glanced and noticed.

Otherwise it could be written like this:
Museum-goers who glance up into the building’s five-story atrium often notice birds,whose behavior is being studied by the museum’s staff, are nesting in the rafters.

This sentence now conveys a fact that is true in the present context that when we will go, we will notice this thing for sure.

I think this makes sense
If not, let me know what is wrong with this construction.

A. notice birds nesting in the rafters, whose behavior was being studied by the museum’s staff
INCORRECT
B. notice birds nesting in the rafters, which were being studied by the museum’s staff
INCORRECT usage of which.
C. notice that birds, whose behavior was being studied by the museum’s staff, are nesting in the rafters
Discussed above. INCORRECT USAGE IN THIS SENTENCE.

D. noticed that birds, whose behavior was being studied by the museum’s staff, were nesting in the rafters
CORRECT
E. noticed that birds, with the behavior that was being studied by the museum’s staff, were nesting in the rafters
Too wordy. Saying "with the behavior"does not add any value. It can be said in a more concise way as in D.


Hope this makes sense.
_________________

KAPLAN OFFICIAL EXPLANATION:

D

There are two problems with this sentence. First, the unchangeable verb glanced is in the past tense, but the underlined notice is in the present tense. Since museum-goers would have noticed the birds at the same time that they glanced up, both verbs must be in the past tense. So notice must become the past tense noticed. On this basis, we can eliminate (A), (B), and (C).

The other problem involves modification. The clause that begins whose behavior is intended to describe the birds, but since it appears (illogically) to be modifying rafters. So the correct choice must place this phrase directly after birds. (D) does this, so it is correct. (E) does so, too, but it incorrectly alters the phrase to read with the behavior.
_________________