In short, yes, your GMAT will most likely diminish your chances at HBS. Just out of curiosity, what did you score on your LSAT?
If anything, try boosting your GMAT score to 680 or above and you should have a solid chance. And even then, like you said, it's really just harvard or bust, so if you don't have the time to retake GMAT, just go for it.
I took the LSAT twice my highest score was 170. I hate the GMAT so much more than the LSAT, but I plan on retaking it in August. I was thinking of taking the GRE instead. Since the rankings are based off of the Gmat do you think submitting a GRE score will increase my chances of admission?