This is a detailed explanation of how to approach such a problem step by step.
Let's start from scratch.
The only ways you can form the mentioned committee are with 3men+3women or 2men+4women
# of ways to select 3 women: 5C3 = 10
# of ways to select 3 men out of 8: 8C3 = 56
6C1 comes from this graph:
Here is the unfavorable outcome whereby A does not want to sit with Y. Let's say A sits with Y, an unfavorable outcome, there are then 6C1 ways of selecting the third person from the remaining 6 men in the group.
Total # of ways of selecting 3 men out of way with the restriction that 2 do not want to sit together is: 8C3 - 6C1 = 56-6 = 50
The same reasoning goes for when you have a committee of 4 women and 2 men
# of ways of selecting 4 women out of 5: 5C4 = 5
# of ways of selecting 2 men out of 8, 2 of which do not want to sit together: Y-A-(X-X-X-X-X-X) --> A does not want to sit with Y, in how many ways can you select 0 persons from the group of 6 men remaining? 1 way. You only want to select 2 men anyways. Therefore, you do not need a third one from the group of remaining 6 men.
nC0 = 1. There is always 1 way of selecting nothing from a group of n objects. Therefore, 8C2 - 6C0 = 28 - 1 = 27
In the end, you add up all favorable outcomes which are 500+135=635