It means "Dude, I'm not gay nor am I trying to be gay!"
If two straight guys have beer with one another, one guy might pay for the other guy and say "no homo", which means "I'm not trying to be gay by paying for, and having a, beer with you."
I use it alot to reassure my straight friends I'm not hitting on them.