There's a lot of time management issues. You do need to prepare appropriately for classes and that means spending time preparing for classes. Juggling that with all the other components of the experience (jobs, networking, socializing, organizing stuff, clubs, etc.) is the hardest part.
IMHO, though, the curriculum per se is not rocket science. No matter how hard it may look, what you'll need to know about Black-Scholes (which is different from what they had to invent back in the day) isn't as tough as other courses I've taken in undergrad. The caveat: I'm an engineer so I don't struggle as much as others with the quant stuff.
However, I believe that if you dedicate appropriate time to learning you should understand what's going on, regardless of your background. The catch is that appropriate time may involve tutoring from 2nd years, learning teams, reading additional materials, etc. It all depends on how fast you can understand new stuff and how much you know going on (specially for the 1st semester).
L.