I have both of them on my computer and I actually think safari is just a little bit faster. Considering that my computer is almost as old as a dinosaur, it doesn't really help that much but if you have a newer computer, it will probably go pretty fast.
Considering it is far less likely to introduce performance killing viri and spyware, I would think actual performance differences are trivial comparatively.