Question for Audiophiles?

I had the volume on my receiver +20.00, and on the first game, the sound coming out of the speakers was great. Not too loud, not too soft; however, Upon placing in another game, I was startled because the sound was too darn loud, and as soon as I heard it (maybe less than 5 seconds), I quickly turned the volume back down. Should I be worried about speaker damage, or if they are not damaged now, should I be fine? I will certainly be more careful next time.