Your interview will probably be better onsite. They certainly won't expect you to fly-in for an interview, so no points off for doing the interview over the phone, but in-person interviews are almost always better anyway.
So let's say your interview is 5%-10% worse over the phone and the recommendation is (generously) 20% of the application, then that is 1-2% off the overall app. Maybe that helps a lot if you are borderline, but not so much if you are really solid. Whether that is worth it to you to fly-in is a personal decision.
Are you a bay area candidate temporarily abroad? What round are you in?