I think as a matter of practicality for me (I'm flying out that evening), I'll probably go with business casual. However, I'll bring a suit just in case (being the paranoid nut like all of us are). That and I tend to take people at their word, and if the letter says "attire for interviews is business casual", I'll go with that.
Put another way, if the letter said, "attire for interviews must include red". I'm sure we'd all wear red shirts (or dresses), and some of us might wear red suits...