Here is my honest opinion. I think every human is very biased and therefore your appearance is crucial in order to make this decision. If you got the feedback in the past that u ur voice does not sound so good over the phone, or if u have trouble understanding ppl over the phone or ur voice is too soft or too deep or whatever - then consider taking a day off.
On another hand, if u think that u dont have a strong personal appearance, a kinda desired business-type of look then consider a phone interview. its like u see a very famous radio host in person and get dissapointed by his appearance...
we all sometimes like or dislike someone at first sight and maybe later totally change our opinion, however, all humans including adcoms r very biased and ur the first contact w u can heavily influence their decision
so just pick whatever u think is ur stronger side
just my opinion...