Giving it a try...
n = 7
Now we can form a polygon in 4 ways
1) 3 sides of the polygon are common to the quadrilateral.
we can choose x consecutive sides from an n sided polygon in n ways.
so we can choose 3 consecutive sides from an 7 sided polygon in 7 ways.
Now given any three sides of a polygon there is only 1 way to form a quadrilateral ( by joining th two points not connected)
so total # of quad = 7*1
2) 2 consecutive sides of a polygon are taken as sides of a quadrilateral.
We can select 2 consecutive side of a polygon in 7 ways.
Given two consecutive sides we can have two possible quadrilaterals
As total # of vertex = 7;
3 are linked by the 2 consecutive sides=> remaining = 7-3; 4 other vertex;
2 of these 4 are already connected by the sides of the polygon; => 4-2 = 2 vertex can be connected by diagonals
so total # of quad = = 7*2 = 14
3) 1 side of the polygon is shared by the quad; # of ways in which a side can be selected = 7
Total vertex = 7
vetex on the common side = 2; so remaining = 7-2 =5
2 other vertex are directly linked by the previous 2 to form the polygon; so remaining = 5-2 = 3
we can choose 2 points from these 3 in 3C2 ways = 6
so total # of quad = 7*6 = 42/2 = 21
4)the polygon and quad do not have any side common;
we can choose vertex 1 in 7 ways;
not of the remaing 6 two are connected by sides of polygon;
so remaing vertex = 4
now among these 4 we CANNOT choose 3 non consecutive vertex; so this cmbination does not seem possbile
so total # of quad = 21+14+7 = 42
anyone else? ans?