My horse has learned how to not jump! (funny)?

Well, I've been working my 4 year old gelding to help train him and I found out that he is sluggish. He's not disobediently sluggish, just...slow. He also refuses to canter with someone on him (which I think is from being punished by his scared, previous owner). So I started using poles to organize his pace and liven up his step with great results. This weekend I tried a small jump. I first let him watch my experienced horse jump at the lunge and in fact, he really did watch! He even interrupted our lunging so he could smell the set-up. When it came to his turn he ran into it and knocked the rail down. The first time I wasn't upset and set it back up. A few more run ins and he decided to not attempt at all. Well, realizing he was confused I encouraged him onward. His reaction- Taking his hoof and reaching it over the pole and pushing down to make it fall. He now does this every time! he'll stop and push it down then just look at me. I'm not mad at this (I think it's hilarious) but I would like to eventually jump with him. Any suggestions?
I have tried solid objects and he will jump those. But if he knows he can push it over, he plays with it. He's so funny!