We have ten families of 4, and one family of 2, so there are 42 people in total. If everyone shook hands with everyone else, then every pair of people would produce a handshake, and we have 42C2 = (42)(41)/2! = 861 pairs of people. So if there were no restrictions, the answer would be 861. But we know that within each family, no one shakes hands. In the two-person family, that means 1 handshake doesn't take place, and within each of the ten four-person families, 4C2 = 6 handshakes don't take place. So there are 1 + 60 = 61 handshakes we shouldn't count, and the answer is 861 - 61 = 800.
Or you could do the problem directly. First, just focusing on the families of four (ignoring the family of two completely for now), we have 40 people in total. Each of these 40 people will shake 36 hands, among only the families of four, so within the families of four, we'll have 40*36/2 handshakes (dividing by 2 because we don't want to count each handshake twice, once for each person involved). So among the families of four, we have 720 handshakes in total. But the two hosts shake everyone else's hand, so they're each involved in an additional 40 handshakes, and the answer is 720 + 40 + 40 = 800.