For a private conversation, assuming that the people on the elevator don't speak Spanish, I think it's fine. Talking on a phone or other in public at all is, however, rude
Does the person switch to spanish because he/she thinks no one will understand? That's kind of stupid seeing as how prevalent spanish speakers are these days. If your conversation is private then wait until you can be alone. Otherwise, you look like a jerk.