If your permit expires, it does say on the form to return it to HPD. However, when you go in for your next permit (and you know you will,) they pretty much expect you to surrend the old one at that time. I think you'd confuse them terribly if you just walked in and said "Hey, here's my expired permit, see you later." Besides, if you're going down there anyway, why wouldn't you renew?
Just hang onto it. The expired permit police will not come looking for you.