cold sores-- will they spread to the genitals?

My question is regarding cold sores. I get cold sores on my lips occasionally. I had a beginning of an outbreak, but did not realize it. It was not a full-on blister but the beginning swelling and reddening. I performed oral sex on my partner using a latex condom, but after that we had intercourse. There was no direct contact with the cold sore, but there was saliva. Is it possible that I spread the cold sore virus to my own genitals? If it is possible-- how likely is it?