My youngest son has, had a number of injuries over the last few years that have kept him off the bike. A collarbone break which put him out for 12 weeks and a dislocated leg which had him off the bike for another 3 months the first month spent in traction. And that is only just two of the reasons he has had to take time off the bike. He didn't forget what he knew but needed the time to get his strength back.
If you are off with injury you have to allow yourself time to get the stamina and strength back, but if you are just going away and cannot ride as long as you keep a fitness regime up you will be fine.