No, I wouldn't. I try to maintain eye contact with everyone when speaking. I understand that it is considered polite to make eye contact when you are addressing someone but I do understand that just because it's polite doesn't mean that everyone is comfortable doing it.