If this is only your first GMAT CAT, I wouldn't worry too much about it. OG questions are pretty reliable. People don't usually say GMATPrep math is harder than Kaplan, though, usually the other way around.
My guess would be that the pressure of the timed test threw you off. How much untimed prep did you do before the first CAT? Did you feel comfortable with the math before that? Did you see a lot of questions in your weak areas on the GMATPrep test? There could be a lot of reasons why it didn't go well.
Don't take another CAT until you've gotten more comfortable with each question type, and work on timing. I would add another book to your collection -- Cracking the GMAT. The strategies for timing really helped me.
You can uninstall GMATPrep and reinstall it to try to get tests without questions you've already seen. It's not foolproof, but worth a shot. You'll probably see a handful of repeats.