A security guard at a certain store is instructed to check

Yes, I agree with you. I would think 1 out of every 5 might mean 1 out of every 5 and not 20% of the total customers leaving the store. So if he checks the receipt of the first person who leaves the store, he would not check the receipt of the next 4 people. But since 25 is not an answer, I would then take it to mean (1/5)th of the customers and proceed to get 10C2 (select 2 out of the 10). Seeing that this is not an official question, I wouldn't worry too much about it.
Even in official questions, sometimes ambiguous language might be introduced inadvertently. That is the reason they have "experimental" questions - to weed out every ambiguity before they score you on that question. So rest assured.

If the guard is instructed to select 1 of every 5 receipts and the question is asking how many different sets of customers could be selected out of 10 then the guard will select 2 of the 10 customers to review.

10C2 = 45.

to choose the sample of 2 customers from 10. it can be done 10C2 possible ways.
So the answer is B. 45

should be 10C2. Just the total number of possible combinations of people that the guard could have chosen when he has to check 1 out of 5 or 2 out of every 10. = 45. B.

Bunuel VeritasKarishma, Can you please help with this one? When I read this question, I would think that you select 1 from each 5, so my answer would be 5C1 * 5C1 = 25 -> which isn't even an option. Can you explain how it is 10C2? If you pick the 8th and 9th person through 10C2, doesn't that go against the question?

Thanks a lot, VeritasKarishma for responding. Great! I'm glad my thought process wasn't completely flawed then. I got this question on my GMAT Practice test #3 and that's why I was stuck because it was unlike a GMAC question to have ambiguity. Thanks for clearing that out

If 10 customers leave the store, 2 of them will be checked. Therefore, there are 10C2 = (10 x 9)/2 = 45 different samples possible.

Answer: B