it definitely would. whether it has caffeine or not, it will affect your swimming performance. even though decaf as no caffeine, it will make you feel more tired after exercising after warm up. don't worry, the whole purpose of warm up is to wake you up and get your muscles ready. but definitely stay away from coffee/sugary stuff, especially right before the swim meet.
p.s. good luck tomorrow!