Similar thing just happened to me.
I'm using those CATs from GMAT PREP, and the following situation happened:
CAT 1 : V42 - 8 Errors. (Mistakes on 6, 9, 10, 25, 27, 30, 31, 36)
CAT 2 : V41 - 5 Errors. (Mistakes on 8, 13, 16, 25, 41)
As you said, it is indeed a complex algorithm, but a difference of 3 questions being answered correct and a lower score is confusing.
My hypothesis, is that i was given easier questions to answer, but I wouldn't know for sure why.
Additionally, I just checked the difficulty of question 41 (CAT 2) and it was a Sub-600 Question that i got wrong (=/), while Question 41 (CAT1) was a 700-Level Question that I got right. So that could explain the difference in scoring.
However, I don't know why Question 41 in CAT2 was (a lot) easier than Q41 in CAT1.
Go figure...