Use a mixed system: Chunks of XP at milestones...##
It's funny: your situation sounds 90% identical to what I've been struggling with for a while, especially the part about forgetting to give out XP for long stretches.
Personally, I've switched primarily to a milestone system: every time the PCs hit an important milestone or made some progress, I dole out a precalculated amount of XP (or even just an entire level at a time). It seems to me that such a system would work for you. Instead of calculating XP, you should break up a level gain into smaller chunks, and go with that.
You say that you make things up on the fly, but such a progression method isn't too difficult to do either--you could split a level-up into smaller chunks, say 1/4 or 1/2 of a level, and dole those out accordingly. You're effectively reducing the XP system from numbers in the thousands to much smaller numbers, but it really reduces the mental overhead for the system. You're not going to get a preparation-free method for dealing with this issue, in my opinion, but this approach should help.
For example, if the players decide to detour into a sidequest, you can simply decide that the sidequest is worth half a level. If the PCs finish a dungeon, that might be worth a full level.
...and bits of XP for fun
At the same time, you can still use small amounts of XP as rewards. When the XP system uses numbers in the tens of thousands, you can hand out 5, 10, or even 100 XP out freely without worrying about unbalancing level-ups, especially if you're using the chunked level progression described above. I've played in games where the DM does this, and it has zero functional effect on the game, except when players like to brag that they're actually 200 xp ahead of others (which is the intended situation, right?).
Incidentally, this also solves a similar problem I had with rapid leveling--I like to throw difficult encounters at my players, but the XP gains were so high that they'd advance far too quickly for my taste. Leveling up by milestones completely eliminates this problem too, and gives you a lot more flexibility in how you run your game.