It is not required to do it at hpd, but nice to do to ensure things go smoothly. You mostly need a bill of sale from the current owner. I would match the info on the bill of sale against their ID. After you get the permit, take the bill of sale to HPD with the permit and gun, and you should be good to go. If the current owner didn't register that handgun or it was stolen or something, they could potentially get in a ton of trouble trouble. So it would be stupid of them to sell you a gun that they don't own because it'll come back to them. Again, doing the transaction at HPD is helpful in that HPD will make sure all of that is kosher, then you can hand over the cash.
If he hands over the gun before the 2 weeks are up, and you take it to HPD and try to register at the same time you pick up the permit, you could technically get in trouble. Especially if they close down the left permitting side, and it's obvious you had the gun in your possession.
-----------------
On the flip side, if you dont want to go to HPD, wait until he has the permit and shows it to you. Then do the transaction and fill out a bill of sale. At that point, as far as you're concerned, the transaction is done.