Was it racist of me to complain about a non-english speaking professor?

One of my teachers if from japan. He speaks VERY little english. While I have no problems with immigrants at all and feel they deserve jobs based on their qualifications. I can in no way justify nor take a professor that cannot speak clear and fluent english.

I complained to the department head that his teaching style and lack of english was compromising my grade in the course. To paraphrase his response he told me I was being racist and not understanding of the professors situation.

I feel like confronting a serious issue effecting my education isn't racist. Especially when the professors job is to clearly communicate with it's students. I'm paying for his services basically and he is not providing them in a manner I can comprehend.

thoughs? comments? experiences? advice? :(