Robin wrote 4 different letters to send to 4 different addresses

The reason this is wrong is sort of subtle - but let me try to explain anyways

Let's call the letters A, B, C, D, and E, and let's call the envelopes a, b, c, d, and e. Letter A belongs with envelope a, and so on.

Using your reasoning, you start by choosing a letter to put in the correct envelope. Suppose you choose letter A, and put it in envelope a. Now, the remaining letters are B, C, D, and E, and the remaining envelopes are b, c, d, and e.

Next, you have to put letter B into the WRONG envelope. You're correct that there are 3 ways to do this, so we're at 5*3. Suppose, for this example, that we put letter B into envelope c.

So, now the remaining letters are C, D, and E, and the remaining envelopes are b, d, and e.

Next, we have to put letter C into the wrong envelope. But there aren't just two ways to do this! In fact, all three of the remaining envelopes are "wrong", since envelope c has already been used.

So, your reasoning breaks down when you go from 5*3, to 5*3*2. Since it's possible that the 'right' envelope has already been used up, there may be more 'wrong' envelopes than we thought at any point.

---

Now, how to actually calculate it? I'd argue that this is an unrealistically tough thing to calculate for the GMAT. But one way to do it would be like this:

1. There are five ways to choose a letter to go into the right envelope. Let's say this is letter A in envelope a.

2. Then, we want to know how many ways there are to arrange the remaining four letters in their envelopes so that none of them are in the right envelope.

There are 4! = 24 ways in total to arrange those letters. BCDE, BCED, BDCE, etc.

All of the arrangements where B is first shouldn't be counted, so let's eliminate those 6 arrangements. 24 - 6 = 18.

At this point, I just listed out the other 18 arrangements and crossed off the ones that have a letter in the right envelope:

CBDE
CBED
CDBE
CDEB
CEBD
CEDB

DBCE
DBEC
DCBE
DCEB
DEBC
DECB

EBCD
EBDC
ECBD
ECDB
EDBC
EDCB

It looks like there are 9 good arrangements left. So, the total is 5*9 = 45.