I've dealt with about 15 VC firms and have gotten funding for 2 companies (I was founder of one). I'm based in Menlo Park, and have dealt with quite a few of the big players on Sand Hill...well, perhaps spoken with a lot of the big players, but that's starting to split hairs.
My general impression is that to get into VC, you need to be a technology person (hi tech, biotech, MD...basically, some kind of science). This background is paramount, as you'll be investing in these companies and need to deeply understand the technology, not the balance sheets of the companies. Most of the partners I have spoken with have been successful entrepreneurs themselves, or else have been very early players in successful start-ups (i.e. there are a bunch of early Genentech guys running around in biotech VC firms, much like there will eventually be a bunch of early Google guys runnings around in tech VC firms).
That said, from my experiences, your initial assumption was dead on - that being, it will be very hard to get into VC if you're primarly a finance guy...large firm PE, however, is a different story entirely. My impression is that bankers are more-or-less ignored by VCs until they need to liquidate a company - they aren't looked at as potential partners.